Better Process Supervision with Bi-directional Rewarding Signals
Authors: Wenxiang Chen, Wei He, Zhiheng Xi, Honglin Guo, Boyang Hong, Jiazheng Zhang, Nijun Li, Tao Gui, Yun Li, Qi Zhang, Xuanjing Huang
Accepted at: acl25

Better Process Supervision with Bi-directional Rewarding Signals

Wenxiang Chen1*, Wei He1*, Zhiheng Xi1*, Honglin Guo1, Boyang Hong1, Jiazheng Zhang1, Nijun Li2, Tao Gui3,4†, Yun Li2, Qi Zhang1,4†, Xuanjing Huang1,4†
1 College of Computer Science and Artificial Intelligence, Fudan University
2 Cognitive AI Lab, Shanghai Huawei Technologies, China
3 Institute of Modern Languages and Linguistics, Fudan University
4 Institute of Trustworthy Embodied Artificial Intelligence, Fudan University
chenwx23@m.fudan.edu.cn, {tgui,qz,xjhuang}@fudan.edu.cn

Abstract

Process supervision, i.e., evaluating each step, is critical for complex large language model (LLM) reasoning and test-time searching with increased inference compute. Existing approaches, represented by process reward models (PRMs), primarily focus on rewarding signals up to the current step, exhibiting a one-directional nature and lacking a mechanism to model the distance to the final target. To address this problem, we draw inspiration from the A* algorithm, which states that an effective supervisory signal should simultaneously consider the incurred cost and the estimated cost for reaching the target. Building on this key insight, we introduce BiRM, a novel process supervision model that not only evaluates the correctness of previous steps but also models the probability of future success. We conduct extensive experiments on mathematical reasoning tasks and demonstrate that BiRM provides more precise evaluations of LLM reasoning steps, achieving an improvement of 3.1% on Gaokao2023 over PRM under the Best-of-N sampling method. Besides, in search-based strategies, BiRM provides more comprehensive guidance and outperforms ORM by 5.0% and PRM by 3.8% respectively on MATH-5001.

1 Introduction

With the rapid development of LLMs, how to supervise them has become a key research challenge, especially for complex tasks like long-term reasoning (Zelikman et al., 2022; OpenAI, 2024b; Wan et al., 2024). Previous work has explored training process supervision models to provide dense supervision on each step (Uesato et al., 2022; Lightman et al., 2024; Wang et al., 2024b), which is intuitively and practically better than outcome supervision models (Cobbe et al., 2021) that only provide sparse signals on the final answer. During test-time, process supervision models can further guide the search of LLMs or perform solution re-ranking by allocating more inference compute (Snell et al., 2024; Brown et al., 2024; Wu et al., 2024).

However, existing approaches, represented by process reward models (PRMs) from OpenAI (Lightman et al., 2024), typically focus on providing one-directional reward signals on the reasoning steps that have already been generated, without consciously considering the probability of future success (Yu et al., 2024a; Zhang et al., 2025). Specifically, while they can accurately distinguish between correct and incorrect steps at the current state (i.e., backward supervision), their ability to identify which partial solution is most likely to reach the correct final answer (i.e., forward supervision) is not guaranteed, leading to sub-optimal performance in guiding effective next-step reasoning (Stroebel et al., 2024; Wang et al., 2025).

To address this challenge, we draw inspira-

![Error-detection accuracy across different steps, where step 1 and steps beyond 15 are truncated for better visualization. We evaluate the process reward model (PRM), value model (VM), and BiRM on PRMBench.](page_1012_670_393_312.png)

Figure 1: Error-detection accuracy across different steps, where step 1 and steps beyond 15 are truncated for better visualization. We evaluate the process reward model (PRM), value model (VM), and BiRM on PRMBench.
tion from the classic A* algorithm, and introduce BiRM, a novel process supervision model that provides bidirectional rewarding signals. Classically, the A* algorithm (Hart et al., 1968) states that an appropriate supervisory signal should take two aspects into account: the cumulative cost up to the current step, and the estimated probability of reaching the target (Zhuang et al., 2024; Wang et al., 2024a). Motivated by this key insight, we redesign the process supervision signals, which should not only assess the correctness of steps taken so far, but also evaluate the future success probability of the partial solution. Specifically, BiRM introduces a value model (VM) head to help model the forward supervision signal (Yu et al., 2024a; Ankner et al., 2024), so that it can estimate both the correctness and success probability of a reasoning prefix/partial solution (Section 4).

To validate our motivation, we conduct a preliminary analysis on PRMBENCH (Song et al., 2025), a benchmark designed to evaluate the capability of process supervision models. We include PRM and VM as baselines, where the former estimates the correctness of partial solutions, and the latter estimates the future success probability. As shown in Figure 1, PRM performs better at detecting error steps in the early stages of reasoning, while VM performs better in the later stages. This indicates that each baseline has limitations, which aligns with the intuition we derive from the A* algorithm. In contrast, BiRM outperforms both of them in all stages, demonstrating the comprehensiveness and effectiveness of our approach.

We then perform extensive experiments on three mathematical reasoning tasks: GSM8K, MATH-500 and Gaokao2023 (Liao et al., 2024) to demonstrate the effectiveness of BiRM across different model series and search strategies. For example, BiRM trained on Qwen2.5-7B-Base achieves a 3.1% improvement on Gaokao2023 over PRM using Best-of-N sampling. Additionally, in beam search with a total sampling size of 100, BiRM further surpasses PRM by 3.8% and ORM by 5.0%.

In summary, our contributions are as follows:

• We draw inspiration from A* algorithm and propose BiRM, a novel process supervision model that provides bidirectional rewarding signals.

• We conduct extensive experiments on math reasoning tasks to demonstrate its effectiveness in solution re-ranking and trajectory searching.

• We present an in-depth analysis and demonstrate that BiRM is orthogonal to existing open-source supervision models, highlighting its robustness and generalization capabilities.

2 Related Work

2.1 Enhancing Mathematical Reasoning Capabilities of LLMs

Mathematical reasoning tasks remain a significant challenge for LLMs (OpenAI, 2024a; Snell et al., 2024). Researchers have conducted extensive studies on both train-time and test-time improvements. At train-time, supervised fine-tuning is a well-established approach. Its core idea is to construct large-scale, high-quality datasets to enhance performance (Liao et al., 2024; Yu et al., 2024b; Tong et al., 2024). On the other hand, experimental results from Openai-o1 (OpenAI, 2024b) and DeepSeek-R1 (DeepSeek-AI et al., 2025) highlight the promising potential of test-time scaling laws. Vanilla sampling methods like Best-of-N sampling (Liu et al., 2025) and search-based strategies such as beam search, A*, and MCTS (Zhuang et al., 2024; Wan et al., 2024; Zhang et al., 2024a) have all achieved remarkable performance by allocating more computational resources at test-time. In this work, we focus on improving LLM’s performance during the test-time phase.

2.2 Process Supervision Models in LLM Reasoning

LLMs can leverage an additional supervision model to achieve accurate test-time reasoning. Mainstream approaches can be divided into outcome reward models (ORMs) and process reward models (PRMs). ORMs are trained with rule-based labeled data and assign one score to the entire solution path (Cobbe et al., 2021; Yu et al., 2024a). This method achieves striking results in reasoning models like Deepseek-R1 but struggles with other tasks where the answers are highly open-ended. On the other hand, PRMs evaluate each intermediate steps in the trajectory, providing more granular reward signals (Lightman et al., 2024; Uesato et al., 2022; Zhang et al., 2025). Depending on the practical implementation, there are several variants of PRMs:
(1) Value Models (VMs, Wang et al., 2024b; Luo et al., 2024) use Monte Carlo estimation to label steps, reducing the burden of manual annotation. The resulting labels represent the probability of future success, essentially making PRMs a type
of value model. (2) Generative Reward Models (Zhang et al., 2024c) leverage the text generation capabilities of LLMs, providing natural language feedback, rather than traditional numerical scores.

3 Motivation

3.1 Task Formulation
Given a mathematical question \( q \), a large language model \( \pi \) generates a sequence of reasoning steps to solve the problem. The complete reasoning trajectory, i.e., chain-of-thought (Wei et al., 2022), can be denoted as \( \tau = \{ s_1, s_2, \ldots, s_m \} \), where \( s_i \) represents the \( i \)-th step and \( m \) is the number of total reasoning steps.

3.2 The Limitations of PRMs
PRMs are typically trained to assign a numerical score to each intermediate reasoning step, evaluating their correctness. For a partial trajectory \( \tau^{[1:t]} = \{ s_1, s_2, \ldots, s_t \} \), PRM can provide an reward score for step \( s_i \):

\[
r(s_i, q) = p(s_i \text{ is correct } | \ q),
\]

where \( r(\cdot) \) represents the process-based reward function provided by PRM. Further, the correctness of the partial trajectory \( \tau^{[1:t]} \) can be expressed as the accumulative correctness reward of all intermediate steps, following Lightman et al. (2024):

\[
\mathcal{R}(\tau^{[1:t]}, q) = p([s_1, s_2, \ldots, s_t] \text{ is correct } | \ q)
= \prod_{i=1}^t p(s_i \text{ is correct } | \ q) = \prod_{i=1}^t r(s_i, q).
\]

This equation highlights the one-directional scoring nature of PRMs, which evaluate whether the sampled trajectory \( \{ s_1, s_2, \ldots, s_t \} \) is correct given the problem \( q \). Instead, for the potential future paths \( \{ s_{t+1}, s_{t+2}, \ldots, s_m \} \) starting from the current state \( s_t \), PRMs lack the capability to provide effective guidance, as Figure 2 illustrates.

3.3 Inspiration from the A* Search Algorithm
To address this limitation, we draw inspiration from the A* algorithm. Originally, A* is a heuristic graph search algorithm designed to find the optimal path (Hart et al., 1968). The key insight from A* is that a good supervision signal should simultaneously consider two aspects: the accumulative cost \( g(n) \) up to the current step and the future cost \( h(n) \) to the target. The final value of a step is given by \( f(n) = g(n) + h(n) \).

In the context of LLM mathematical reasoning, we argue that a good supervision signal should not only consider the correctness of previous steps (i.e., backward supervision) but also model the probability of future success (i.e., forward supervision). On the one hand, PRM can naturally function as \( g(\cdot) \). In other words, PRM can use its one-directional scoring ability to provide rewards for the partial solution up to the current step \( s_t \):

\[
g(s_t) = \operatorname{Agg}(r(s_1), r(s_2), \ldots, r(s_t)) = \mathcal{R}(\tau^{[1:t]}),
\]

where \( \operatorname{Agg} \in \{ \prod, \min, \max, \mathrm{avg} \} \) stands for an aggregation function to summarize the accumulative rewards of all steps from \( s_1 \) to \( s_t \).

On the other hand, to heuristically model the probability of reaching the correct final answer, we seek to utilize a value model (VM) to play the role of \( h(\cdot) \). For the partial solution \( \tau^{[1:t]} \), a forward-looking VM can provide a reliable probability estimation:

\[
h(s_t) = \mathcal{V}(\tau^{[1:t]}, q)
= \mathbb{E}_{\hat{a} \sim \pi(\cdot|\tau^{[1:t]}, q)} [p(\hat{a} \text{ is correct } | \ q)].
\]

Here, \( \hat{a} \) represents the final answer predicted by the LLM \( \pi \), and \( \mathcal{V}(\cdot) \) denotes the estimation of VM for whether the partial trajectory can reach the correct answer. In practical implementations, the VM and PRM share the same model architecture, but differ in the meaning of training labels, which fundamentally trains the VM as a reliable predictive estimator. We will discuss more details in Section 4.2. Finally, the complete value function can be expressed as:

\[
f(s_t) = g(s_t) + \beta \cdot h(s_t),
\]

where the coefficient \( \beta \) balances the importance of the \( g(s_t) \) and \( h(s_t) \) terms. When a step \( s_t \) has a higher \( f(s_t) \) value, it indicates that this step is more promising among multiple candidates, thus contributing to more effective next-step reasoning.

4 BiIRM, a Bidirectional Process Supervision Model

4.1 Training Methodology
For a query \( q \) from the training question set \( Q \), we first sample \( N \) solutions from the generator \( \pi \). Then, we annotate each intermediate step of these solutions, i.e., annotating step-level labels. The resulting dataset \( \mathcal{D} \) for query \( q \) can be formalized as
PRM: \( s_1 \) is correct , \( s_2 \) is correct – must be on the right path!

BiRM: \( s_{2,1} \) is correct, \( s_{2,2} \) is also correct – but the latter seems a more promising path forward.

Figure 2: An example of our proposed BiRM compared with traditional Process Reward Models (PRMs). Given a question \( q \), PRMs only consider the accumulated rewards up to the current step. In contrast, BiRM takes into account two aspects: the correctness rewards received so far and the probability of reaching correct final answers.

\[
\mathcal{D}_q = \{ \tau_i, \{ y_i^1, y_i^2, \ldots, y_i^j, \ldots \} \}_{i=1}^N,
\]
where \( \tau_i \) denotes the \( i \)-th sampled trajectory, and \( y_i^j \) represents the step label for the \( j \)-th step in the \( i \)-th solution. We will introduce more annotation strategies in Section 4.2.

Following Yu et al. (2024a), we implement the vanilla PRM by adding a linear layer for reward prediction after the last hidden layer of the LLM. We also retain the original language modeling head. Formally, a vanilla PRM \( \mathcal{R}(\theta, \phi_R) \) is parameterized by base model parameters \( \theta \) and reward head parameters \( \phi_R \). The training objective of PRM is to minimize the mean squared error (MSE) loss between the predicted reward scores and the binary step-level reward labels. Thus, we have:

\[
\begin{align*}
\mathcal{L}_{\text{PRM}}(\theta, \phi_R) \\
&= \frac{1}{|\mathcal{Q}|} \sum_{q \in \mathcal{Q}} \left[ \mathbb{E}_{\tau \sim \pi(\cdot|q)} \sum_{t=1}^m \left( \hat{r}_{\theta, \phi_R}(s_t, q) - r^t \right)^2 \right],
\end{align*}
\]

where \( \hat{r}(s_t, q) \) represents the predicted reward score for the \( t \)-th step (Equation 1), and \( r^t \) denotes the ground truth step label. \( m \) represents the total number of steps in solution \( \tau \).

Furthermore, to alleviate the one-directional limitation of PRM, we introduce an additional value head to guide process supervision. Specifically, BiRM \( \mathcal{M}(\theta, \phi_R, \phi_V) \) is parameterized by three components: \( \theta \) represents the base model parameters, \( \phi_R \) represents the reward head, and \( \phi_V \) corresponds to the value head. The overall training objective of BiRM is to jointly minimize the discrepancy between the predicted reward score and the reward label, as well as between the value score and the value label. Similar to the vanilla PRM, we employ MSE loss for the value head:

\[
\mathcal{L}_{\text{VM}}(\theta, \phi_V) = \frac{1}{|\mathcal{Q}|} \sum_{q \in \mathcal{Q}} \left[ \mathbb{E}_{\tau \sim \pi(\cdot|q)} \sum_{t=1}^m \left( \hat{\mathcal{M}}_{\theta, \phi_V}(\tau^{[1:t]}, q) - v^t \right)^2 \right],
\]

where \( \hat{\mathcal{M}}_{\theta, \phi_V} \) represents the estimated success probability for the partial solution \( \tau^{[1:t]} \) (Equation 2), and \( v^t \) denotes the value label for \( s_t \).

In this way, the optimized BiRM considers not only the actual accumulative rewards obtained so far, but also the potential of reaching correct final answers (Figure 2). The complete loss function for BiRM can be defined as:

\[
\mathcal{L}_{\text{BiRM}}(\theta, \phi_R, \phi_V) = \mathcal{L}_{\text{PRM}}(\theta, \phi_R) + c \cdot \mathcal{L}_{\text{VM}}(\theta, \phi_V).
\]

We use a coefficient \( c \) to balance the importance of the reward term \( \mathcal{L}_{\text{PRM}} \) and the value term \( \mathcal{L}_{\text{VM}} \).

4.2 Step Label Annotation Strategies

In this section, we discuss our annotation strategies for two kinds of BiRM training labels.
Reward Labels. Reward labels are defined as the correctness of each current step, represented as a binary label. We use the MetaMath dataset (Yu et al., 2024b) as our training data. We first perform supervised fine-tuning on the base model to obtain the generators. Then, we sample 15 rollouts for each query and use Deepseek-V3 (DeepSeek-AI et al., 2024) to annotate the correctness of each step. Detailed annotation procedures and prompts are provided in Appendix B.2.

Value Labels. A key challenge in implementing the value head is to accurately estimate value labels for the partial solution \( \tau^{[1:t]} \). We employ multiple strategies to address this problem.

MC-based estimation (Wang et al., 2024b) is a widely used method for automated labeling, which can be categorized into soft-label and hard-label annotations. Specifically, we sample \( N \) rollouts from an intermediate step in the trajectory. If \( M \) of them are correct, the soft-label for the current step can be defined as: \( \text{label}(s_t) = \frac{M}{N} \). In contrast, the hard-label method suggests that if any of the rollouts reaches the target, then \( \text{label}(s_t) = 1 \).

The essence of Monte Carlo estimation is to assess the potential of reaching correct final answer from the current step and assign this probability to the step label. Thus, for estimating a partial trajectory, we can formally express it as:

\[
\mathcal{V}(\tau^{[1:t]}, q) \approx \frac{1}{N} \sum_{i=1}^N \mathbb{I}(\hat{a}_i \text{ is correct } | \tau^{[1:t]}, q).
\]

As the number of rollouts \( N \) increases, the estimated value label becomes more accurate. Following Wang et al. (2024b), we sample 8 solutions for each intermediate step and analyze the effectiveness of both soft-label and hard-label approaches.

Outcome-supervised estimation (Yu et al., 2024a) states that using the outcome label alone is sufficient to provide probability estimation for each reasoning steps. The underlying idea is that during the training phase, we can replicate the final answer’s correctness label across all intermediate steps. The resulting value model implicitly learns to foresee the future, predicting potential final outcome (i.e. value) for partial solutions. Compared to MC estimation, outcome-supervised estimation has higher data efficiency, but the shortcoming is that the automatically learned estimation in this way is less accurate.

5 Experiments

5.1 Experimental Setup

Tasks. We conduct experiments using three widely used math reasoning datasets: GSM8K (Cobbe et al., 2021), MATH-500 (Lightman et al., 2024), and an out-of-domain (OOD) dataset Gaokao2023 (Liao et al., 2024) to evaluate the generalization ability of BiRM. Besides, we test our method on three base models across different model sizes and families: Qwen2.5-3B-Base (Yang et al., 2024), Qwen2.5-7B-Base (Yang et al., 2024), and Llama3.1-8B-Base (Dubey et al., 2024).

Baselines. To verify the effectiveness of BiRM, we consider a wide range of baselines, including the outcome reward model (ORM, Cobbe et al., 2021), process reward model (PRM, Lightman et al., 2024) and two variants of PRM: Math-Shepherd (Wang et al., 2024b) and ER-PRM (Zhang et al., 2024b). Additionally, we include the results of greedy decoding and rule-based approaches, i.e. Majority Voting. We present more details in Appendix A.1.

Implementation Details. In the SFT phase, we train our generators on the MATH subset of the MetaMath dataset (Yu et al., 2024b) for two epochs, with a learning rate set to \( 1 \times 10^{-5} \). The global batch size is set to 256. In the training phase, we use 225,000 sampled solutions to train BiRM for one epoch based on the generator checkpoint with a learning rate of \( 5 \times 10^{-6} \). More details are provided in Appendix A.2.

Evaluation Metrics. We conduct a comprehensive evaluation of BiRM, considering both vanilla sampling and search strategies. Best-of-N (BoN) sampling is a commonly used evaluation metric for PRMs. It requires the model to score \( N \) candidate solutions, with the highest-scoring solution selected as the final outcome. We also conduct beam search experiments to verify that BiRM can provide more promising guidance for LLM reasoning. In practice, BiRM follows Equation 3, estimating both rewards and values to calculate final scores. A detailed description is provided in Appendix A.3.

5.2 Main Results

BiRM exhibits more comprehensive and superior evaluations in BoN sampling. Table 1 presents a comparison of BoN accuracy across different supervision models on GSM8K, MATH-
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th rowspan="2">Methods</th>
    <th colspan="3">GSM8K</th>
    <th colspan="3">MATH-500</th>
    <th colspan="3">Gaokao2023</th>
  </tr>
  <tr>
    <th>Avg.</th>
    <th>@128</th>
    <th>@256</th>
    <th>@512</th>
    <th>@128</th>
    <th>@256</th>
    <th>@512</th>
    <th>@128</th>
    <th>@256</th>
    <th>@512</th>
  </tr>
  <tr>
    <td rowspan="6">Qwen2.5-3B</td>
    <td>Greedy</td>
    <td>46.8</td>
    <td>——</td>
    <td>73.1</td>
    <td>——</td>
    <td>40.2</td>
    <td>——</td>
    <td>——</td>
    <td>27.0</td>
    <td>——</td>
    <td>——</td>
  </tr>
  <tr>
    <td>Majority Vote</td>
    <td>58.1</td>
    <td>85.1</td>
    <td>85.0</td>
    <td>85.3</td>
    <td>52.5</td>
    <td>53.0</td>
    <td>53.8</td>
    <td>35.8</td>
    <td>36.3</td>
    <td>36.1</td>
  </tr>
  <tr>
    <td>ORM</td>
    <td>58.9</td>
    <td>88.1</td>
    <td>88.1</td>
    <td>88.1</td>
    <td>52.1</td>
    <td>51.8</td>
    <td>52.2</td>
    <td>37.2</td>
    <td>37.0</td>
    <td>35.8</td>
  </tr>
  <tr>
    <td>PRM</td>
    <td>59.9</td>
    <td><b>88.5</b></td>
    <td>88.3</td>
    <td>88.0</td>
    <td>54.6</td>
    <td>54.1</td>
    <td>54.2</td>
    <td><b>37.3</b></td>
    <td>37.1</td>
    <td>37.2</td>
  </tr>
  <tr>
    <td>ER-PRM</td>
    <td>58.8</td>
    <td>88.0</td>
    <td>88.0</td>
    <td>87.7</td>
    <td>52.6</td>
    <td>52.3</td>
    <td>52.0</td>
    <td>36.2</td>
    <td>36.3</td>
    <td>35.8</td>
  </tr>
  <tr>
    <td>Math-Shepherd</td>
    <td>59.0</td>
    <td>87.3</td>
    <td>87.2</td>
    <td>87.0</td>
    <td>53.2</td>
    <td>53.4</td>
    <td>53.8</td>
    <td>36.6</td>
    <td>36.4</td>
    <td>36.1</td>
  </tr>
  <tr>
    <td rowspan="1">BiRM</td>
    <td><b>61.0</b></td>
    <td><b>88.4</b></td>
    <td><b>88.6</b></td>
    <td><b>88.9</b></td>
    <td><b>55.9</b></td>
    <td><b>56.1</b></td>
    <td><b>57.4</b></td>
    <td><b>36.9</b></td>
    <td><b>37.8</b></td>
    <td><b>38.7</b></td>
  </tr>
  <tr>
    <td rowspan="6">Qwen2.5-7B</td>
    <td>Greedy</td>
    <td>52.3</td>
    <td>——</td>
    <td>78.5</td>
    <td>——</td>
    <td>45.0</td>
    <td>——</td>
    <td>——</td>
    <td>33.5</td>
    <td>——</td>
    <td>——</td>
  </tr>
  <tr>
    <td>Majority Vote</td>
    <td>63.6</td>
    <td>88.1</td>
    <td>88.0</td>
    <td>87.8</td>
    <td>57.3</td>
    <td>57.5</td>
    <td>57.6</td>
    <td>45.5</td>
    <td>45.4</td>
    <td>45.2</td>
  </tr>
  <tr>
    <td>ORM</td>
    <td>64.7</td>
    <td>92.0</td>
    <td>91.6</td>
    <td>91.3</td>
    <td>59.6</td>
    <td>59.9</td>
    <td>59.4</td>
    <td>43.6</td>
    <td>43.5</td>
    <td>41.3</td>
  </tr>
  <tr>
    <td>PRM</td>
    <td>66.3</td>
    <td>92.7</td>
    <td>92.8</td>
    <td>92.9</td>
    <td>60.3</td>
    <td>60.1</td>
    <td>58.4</td>
    <td>45.8</td>
    <td>46.2</td>
    <td>47.3</td>
  </tr>
  <tr>
    <td>ER-PRM</td>
    <td>66.2</td>
    <td>92.2</td>
    <td>92.1</td>
    <td>92.2</td>
    <td>59.7</td>
    <td>59.2</td>
    <td>59.0</td>
    <td>47.0</td>
    <td>47.2</td>
    <td>47.3</td>
  </tr>
  <tr>
    <td>Math-Shepherd</td>
    <td>66.3</td>
    <td>92.1</td>
    <td>92.2</td>
    <td>91.7</td>
    <td>60.3</td>
    <td>60.2</td>
    <td>60.4</td>
    <td>46.4</td>
    <td>47.0</td>
    <td>46.5</td>
  </tr>
  <tr>
    <td rowspan="1">BiRM</td>
    <td><b>68.3</b></td>
    <td><b>93.1</b></td>
    <td><b>93.3</b></td>
    <td><b>93.2</b></td>
    <td><b>62.4</b></td>
    <td><b>62.3</b></td>
    <td><b>63.4</b></td>
    <td><b>47.7</b></td>
    <td><b>49.1</b></td>
    <td><b>50.4</b></td>
  </tr>
  <tr>
    <td rowspan="6">Llama3.1-8B</td>
    <td>Greedy</td>
    <td>34.7</td>
    <td>——</td>
    <td>55.7</td>
    <td>——</td>
    <td>31.2</td>
    <td>——</td>
    <td>——</td>
    <td>17.1</td>
    <td>——</td>
    <td>——</td>
  </tr>
  <tr>
    <td>Majority Vote</td>
    <td>46.4</td>
    <td>72.1</td>
    <td>72.0</td>
    <td>72.3</td>
    <td>39.2</td>
    <td>40.2</td>
    <td>41.1</td>
    <td>26.5</td>
    <td>27.2</td>
    <td>27.1</td>
  </tr>
  <tr>
    <td>ORM</td>
    <td>50.3</td>
    <td>84.1</td>
    <td>84.5</td>
    <td>85.0</td>
    <td>41.5</td>
    <td>40.9</td>
    <td>40.8</td>
    <td>25.4</td>
    <td>25.2</td>
    <td>24.9</td>
  </tr>
  <tr>
    <td>PRM</td>
    <td>51.5</td>
    <td>84.1</td>
    <td>84.8</td>
    <td>85.2</td>
    <td>42.5</td>
    <td>42.2</td>
    <td>41.8</td>
    <td>28.2</td>
    <td>27.7</td>
    <td>27.3</td>
  </tr>
  <tr>
    <td>ER-PRM</td>
    <td>50.6</td>
    <td>84.8</td>
    <td>85.3</td>
    <td>85.8</td>
    <td>41.3</td>
    <td>41.0</td>
    <td>40.2</td>
    <td>25.7</td>
    <td>26.1</td>
    <td>24.9</td>
  </tr>
  <tr>
    <td>Math-Shepherd</td>
    <td>51.3</td>
    <td>84.4</td>
    <td>84.9</td>
    <td>85.3</td>
    <td>42.7</td>
    <td>42.9</td>
    <td>43.6</td>
    <td>25.8</td>
    <td>25.8</td>
    <td>26.2</td>
  </tr>
  <tr>
    <td rowspan="1">BiRM</td>
    <td><b>54.1</b></td>
    <td><b>86.1</b></td>
    <td><b>87.2</b></td>
    <td><b>87.8</b></td>
    <td><b>45.4</b></td>
    <td><b>45.4</b></td>
    <td><b>45.6</b></td>
    <td><b>29.4</b></td>
    <td><b>30.0</b></td>
    <td><b>29.6</b></td>
  </tr>
</table>

Table 1: Performance of Best-of-N sampling on GSM8K, MATH-500 and Gaokao2023 with three base models. The accuracy of the BoN solution is utilized as the evaluation metric. The results are reported as the average accuracy across five random seeds. @128, @256, and @512 denote the accuracy with Best-of-128, Best-of-256, and Best-of-512 sampling, respectively. The results of greedy decoding are independent of \( N \) and are listed for comparison purposes. The best results are marked in bold.

500, and the out-of-domain Gaokao2023 dataset. Our observations are as follows: (1) BiRM consistently outperforms vanilla ORM, PRM, and their variants on both GSM8K and MATH-500. For instance, BiRM trained on Llama3.1-8B outperforms PRM on GSM8K by 2.6%, while BiRM based on Qwen2.5-7B achieves an additional 5.0% improvement on MATH-500. (2) BiRM exhibits better generalization ability. Since supervision models are trained solely on the query sets from GSM8K and MATH, Gaokao2023 serves as an out-of-domain (OOD) test set. BiRM-Qwen2.5-7B surpasses the finely labeled Math-Shepherd by 3.9%. In contrast, other supervision methods show fluctuating performance across different base models. (3) As N increases, some supervision methods fail to provide consistent supervision. For example, ORM trained on Qwen2.5-3B shows a decrease on Gaokao2023 from 37.2% to 35.8%. In contrast, BiRM maintains a continuous increase in accuracy. We provide more detailed discussions in Section 6.1.

BiRM demonstrates more meaningful and promising guidance in search-based strategies.

To fully demonstrate the superiority of BiRM’s bidirectional supervision capability, we conduct further experiments under search-based strategies . We run step-level beam search and choose vanilla ORM and PRM as baselines. The detailed algorithm is provided in Appendix A.3. From Table 2, we can conclude that: (1) BiRM achieves the highest accuracy in most cases. For example, on GSM8K, Qwen2.5-7B-BiRM achieves an accuracy of 89.4 at \( K = 8 \), which is a notable improvement over PRM’s 88.1%. (2) As beam size increases, BiRM’s performance continues to improve. On the Llama3.1-8B base model, BiRM outperforms ORM by 2.8% at \( K = 20 \) and achieves a notable 5.0% improvement at \( K = 100 \) in MATH-500 dataset. These results emphasize the valuable bidirectional supervision signals provided by BiRM, which significantly contributes to guiding the LLM toward more successful and promising final answers in solution searching.
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th rowspan="2"># Total Size</th>
    <th colspan="3">GSM8K</th>
    <th colspan="3">MATH-500</th>
    <th colspan="3">Gaokao2023</th>
  </tr>
  <tr>
    <th>ORM</th>
    <th>PRM</th>
    <th><b>BiRM</b></th>
    <th>ORM</th>
    <th>PRM</th>
    <th><b>BiRM</b></th>
    <th>ORM</th>
    <th>PRM</th>
    <th><b>BiRM</b></th>
  </tr>
  <tr>
    <td rowspan="4"><b>Qwen2.5-3B</b></td>
    <td>\( K = 4 \)</td>
    <td><b>83.0</b></td>
    <td>82.1</td>
    <td>82.8</td>
    <td>48.6</td>
    <td>49.3</td>
    <td>50.1</td>
    <td>35.6</td>
    <td>34.9</td>
    <td>36.1</td>
  </tr>
  <tr>
    <td>\( K = 8 \)</td>
    <td>84.6</td>
    <td>83.9</td>
    <td><b>85.1</b></td>
    <td>50.1</td>
    <td>50.9</td>
    <td><b>52.5</b></td>
    <td>36.1</td>
    <td><b>37.9</b></td>
    <td>37.9</td>
  </tr>
  <tr>
    <td>\( K = 20 \)</td>
    <td>86.7</td>
    <td>85.7</td>
    <td><b>86.9</b></td>
    <td>53.0</td>
    <td>54.3</td>
    <td><b>55.0</b></td>
    <td>37.7</td>
    <td>38.4</td>
    <td>39.1</td>
  </tr>
  <tr>
    <td>\( K = 100 \)</td>
    <td>87.5</td>
    <td>85.9</td>
    <td><b>87.6</b></td>
    <td>53.0</td>
    <td>53.9</td>
    <td><b>55.1</b></td>
    <td>38.1</td>
    <td>37.9</td>
    <td><b>39.0</b></td>
  </tr>
  <tr>
    <td rowspan="4"><b>Qwen2.5-7B</b></td>
    <td>\( K = 4 \)</td>
    <td>86.2</td>
    <td>86.5</td>
    <td><b>87.0</b></td>
    <td>55.7</td>
    <td>55.8</td>
    <td><b>57.1</b></td>
    <td>42.8</td>
    <td>44.0</td>
    <td><b>44.5</b></td>
  </tr>
  <tr>
    <td>\( K = 8 \)</td>
    <td>88.6</td>
    <td>88.1</td>
    <td><b>89.4</b></td>
    <td>58.3</td>
    <td>59.1</td>
    <td><b>60.1</b></td>
    <td>44.2</td>
    <td>45.6</td>
    <td><b>46.8</b></td>
  </tr>
  <tr>
    <td>\( K = 20 \)</td>
    <td>90.4</td>
    <td>89.2</td>
    <td><b>90.6</b></td>
    <td>59.1</td>
    <td>61.5</td>
    <td><b>62.3</b></td>
    <td>45.5</td>
    <td>48.1</td>
    <td><b>48.4</b></td>
  </tr>
  <tr>
    <td>\( K = 100 \)</td>
    <td>91.2</td>
    <td>88.4</td>
    <td><b>91.7</b></td>
    <td>60.1</td>
    <td>60.7</td>
    <td><b>62.5</b></td>
    <td>46.8</td>
    <td>48.3</td>
    <td><b>50.0</b></td>
  </tr>
  <tr>
    <td rowspan="4"><b>Llama3.1-8B</b></td>
    <td>\( K = 4 \)</td>
    <td>72.8</td>
    <td>71.7</td>
    <td><b>72.9</b></td>
    <td>38.5</td>
    <td>39.9</td>
    <td><b>40.7</b></td>
    <td>23.9</td>
    <td>25.1</td>
    <td><b>25.4</b></td>
  </tr>
  <tr>
    <td>\( K = 8 \)</td>
    <td>77.4</td>
    <td>75.9</td>
    <td><b>78.3</b></td>
    <td>40.2</td>
    <td>40.1</td>
    <td><b>43.3</b></td>
    <td>25.6</td>
    <td>26.6</td>
    <td><b>27.5</b></td>
  </tr>
  <tr>
    <td>\( K = 20 \)</td>
    <td>81.4</td>
    <td>79.2</td>
    <td><b>81.7</b></td>
    <td>41.5</td>
    <td>42.1</td>
    <td><b>44.3</b></td>
    <td>27.0</td>
    <td>28.6</td>
    <td><b>29.2</b></td>
  </tr>
  <tr>
    <td>\( K = 100 \)</td>
    <td>82.7</td>
    <td>80.3</td>
    <td><b>85.4</b></td>
    <td>41.1</td>
    <td>42.3</td>
    <td><b>46.1</b></td>
    <td>26.2</td>
    <td>29.6</td>
    <td><b>30.7</b></td>
  </tr>
</table>

Table 2: Performance of beam search on GSM8K, MATH-500 and Gaokao2023 with three base models. “# Total Size” stands for total sampling size \( K \) in beam search and we report the best performance among all beam sizes. The results are reported as the average accuracy across three random seeds. The best results are marked in bold.

![Scaling decline phenomenon in Best-of-N sampling. We present the BoN accuracy results across five random seeds. For better visualization, we apply the moving average with a window size of 10.](page_101_1012_627_312.png)

Figure 3: Scaling decline phenomenon in Best-of-N sampling. We present the BoN accuracy results across five random seeds. For better visualization, we apply the moving average with a window size of 10.

<table>
  <tr>
    <th>Models</th>
    <th>Methods</th>
    <th>MATH-500</th>
    <th>Gaokao2023</th>
  </tr>
  <tr>
    <td rowspan="3">Qwen2.5-7B</td>
    <td>+ Outcome</td>
    <td>61.8</td>
    <td>61.1</td>
    <td>46.8</td>
    <td>49.4</td>
  </tr>
  <tr>
    <td>+ MS. (Hard)</td>
    <td>62.1</td>
    <td>62.8</td>
    <td>47.3</td>
    <td>49.7</td>
  </tr>
  <tr>
    <td>+ MS. (Soft)</td>
    <td>62.4</td>
    <td>63.4</td>
    <td>47.7</td>
    <td>50.4</td>
  </tr>
  <tr>
    <td rowspan="3">Llama3.1-8B</td>
    <td>+ Outcome</td>
    <td>44.9</td>
    <td>44.2</td>
    <td>29.0</td>
    <td>29.6</td>
  </tr>
  <tr>
    <td>+ MS. (Hard)</td>
    <td>45.1</td>
    <td>45.4</td>
    <td>29.2</td>
    <td>29.4</td>
  </tr>
  <tr>
    <td>+ MS. (Soft)</td>
    <td>45.4</td>
    <td>45.6</td>
    <td>29.4</td>
    <td>29.6</td>
  </tr>
</table>

Table 3: Different value label annotation strategies for BiRM. “Outcome” stands for Outcome-supervised estimation. “MS. (Hard)” and “MS. (Soft)” represents Math-Shepherd hard and soft estimation respectively.

classifying positive samples as negative. As the sample size increases, this misjudgment becomes more pronounced. Traditional PRMs exhibit a one-directional scoring nature, limiting their ability to evaluate candidates from a comprehensive perspective. In contrast, BiRM estimates both rewards and values, providing more reliable supervision signals.

6.2 Annotation Strategies for Value Labels

As discussed in Section 4.2, we explore various strategies for annotating precise value labels. We aim to demonstrate that our method has good orthogonality with existing annotation strategies.

Table 3 presents the accuracy of BiRM in BoN sampling under different strategies. We can conclude that: (1) More accurate annotations lead to greater improvements. The mash-shepherd soft estimation, which uses the potential success probability of intermediate steps as explicit labels, offers the finest granularity and achieves the best perfor-

6 Analysis and Discussions

6.1 Scaling Decline in BoN sampling

We conduct a further analysis of the scaling decline phenomenon in our main results. The complete BoN accuracy curve, shown in Figure 3, is plotted for \( N \) ranging from 1 to 512. As \( N \) increases, we observe that BiRM shows a consistent improvement. In contrast, the post-verification accuracy of vanilla ORM and PRM plateaus and even declines, which contradicts our intuition learned from the test-time scaling laws (Snell et al., 2024).

We attribute this decline to verifier failures. Imperfect verifiers misrank candidates, erroneously
Figure 4: Performance comparison of ORM, PRM and BiRM under BoN sampling. The base models are open-source RLHFFlow-8B-Deepseek-Data and RLHFFlow-8B-Mistral-Data (Xiong et al., 2024). We follow Equation 3 to calculate the BiRM score at test-time.

mance. In contrast, outcome-supervised estimation, which relies on outcome labels for implicit learning, exhibits greater variability. (2) Even the weakest method, outcome-supervised estimation, shows a notable improvement over PRM. This highlights the flexibility and applicability of BiRM.

6.3 Orthogonality to Existing PRMs
To further demonstrate the generalization ability of our method, we conduct experiments using several existing open-source reward models. We select ORMs and PRMs trained by RLHFFlow (Xiong et al., 2024; Dong et al., 2024) as baselines and reuse the \( N \) sampled solutions they provided for testing. Then we follow Equation 3 to calculate the BiRM scores for BoN sampling.

Experiment results in Figure 4 clearly reveal that BiRM consistently outperforms both ORM and PRM across different values of \( N \), maintaining a consistent upward trend. Furthermore, this trend expands at larger sampling sizes, where BiRM maintains its lead, reaching an accuracy of 57.8% at BoN@256, compared to PRM’s 56.6% and ORM’s 51.4%, respectively. These findings indicate the reliability and generalization of BiRM when using existing open-source reward models.

6.4 Query Scaling or Response Scaling
We also explore a key issue in training supervision models: which matters more, query scaling or response scaling?

We first fix the number of queries and use the original GSM8K and MATH datasets, which contain approximately 15,000 queries. We then test BiRM’s performance with response sizes of 8, 15, and 30. The results in Table 4 reveal that BiRM performs best when the response size is 15 on both datasets. The possible reason is that when the

<table>
  <tr>
    <th># Query</th>
    <th># Resp.</th>
    <th colspan="2">MATH-500</th>
    <th colspan="2">Gaokao2023</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th>@128</th>
    <th>@512</th>
    <th>@128</th>
    <th>@512</th>
  </tr>
  <tr>
    <td rowspan="3">15,000</td>
    <td>×30</td>
    <td>61.3</td>
    <td>61.6</td>
    <td>47.3</td>
    <td>48.3</td>
  </tr>
  <tr>
    <td>×15</td>
    <td>62.0</td>
    <td>63.0</td>
    <td>46.8</td>
    <td>49.4</td>
  </tr>
  <tr>
    <td>×8</td>
    <td>61.3</td>
    <td>61.2</td>
    <td>46.4</td>
    <td>46.8</td>
  </tr>
  <tr>
    <td rowspan="2">7,500</td>
    <td>×15</td>
    <td>59.0</td>
    <td>58.8</td>
    <td>45.4</td>
    <td>44.7</td>
  </tr>
  <tr>
    <td>×30</td>
    <td>57.9</td>
    <td>58.2</td>
    <td>43.4</td>
    <td>42.8</td>
  </tr>
</table>

Table 4: Training data scaling for queries and responses. The base model is Qwen2.5-7B and we use outcome-supervised estimation for simplicity.

number of responses is too low, BiRM cannot learn sufficient and diverse supervision signals. On the other hand, the model struggles with overly similar data patterns per query when # Resp. = 30, leading to overfitting.

Furthermore, we control the total size of the training dataset. Specifically, we conduct experiments with three following settings: 15,000 × 8, 7,500 × 15, and 3,750 × 30. The results demonstrate that BiRM performs best with the 15,000 × 8 configuration. Additionally, we observe that models with fewer queries go through more severe degradation when facing OOD test sets. In the MATH-500 experiments, the gap between the 7,500 × 15 and 3,750 × 30 settings ranges from 0.6% to 1.1%, but this gap significantly widens to 2.0% on the Gaokao2023 benchmark. To sum up, we believe that maintaining an appropriate response size while scaling the number of queries is critical to training process supervision models. We hope this provides valuable insights to the community.

7 Conclusion
In this work, we introduce BiRM, a novel process supervision model for large language models (LLMs), inspired by the A* algorithm. BiRM provides bidirectional supervision signals, evaluating both the correctness of reasoning steps taken so far and the probability of reaching correct answers in the future. Our extensive experiments demonstrate the effectiveness of BiRM across various mathematical reasoning tasks, outperforming existing supervision models like ORM and PRM. Through de-
tained analysis, we highlight the strengths of BiRM in guiding the search process and improving solution re-ranking. We hope that our approach contributes valuable insights to the field of process supervision and opens avenues for future research in enhancing LLM-based reasoning.

Limitations

Our work has some limitations, which we leave for future work to address: (1) High computational cost in test-time searching. In order to improve the performance of LLMs at test-time, we employ vanilla sampling and search-based strategies for solution searching. However, this process requires a significant amount of computational resources. In our work, we use vLLM (Kwon et al., 2023) to alleviate this limitation. Besides, we also observe that search-based strategies sometimes perform worse than repeated sampling due to verifier failures (Yu et al., 2025), even under the same computational budget. We will explore this problem in the future. (2) Generalization across different data patterns and base models. In our experiments, we train our generators and supervision models based on the same base models, ensuring the same data patterns. However, in practical scenarios, an optimal supervision model should be independent of the data pattern and capable of supervising different kinds of reasoning paths. We hope our work provides insights to the community and contributes to the development of more robust and generalized supervision models.

Acknowledgements

The authors wish to thank the anonymous reviewers for their helpful comments. This work was partially funded by National Natural Science Foundation of China (No.62441602,62076069,62206057), Shanghai Rising-Star Program (23QA1400200), and Natural Science Foundation of Shanghai (23ZR1403500).

References

Zachary Ankner, Mansheej Paul, Brandon Cui, Jonathan D. Chang, and Prithviraj Ammanabrolu. 2024. Critique-out-loud reward models. CoRR, abs/2408.11791.

Bradley C. A. Brown, Jordan Juravsky, Ryan Saul Ehrlich, Ronald Clark, Quoc V. Le, Christopher Ré, and Azalia Mirhoseini. 2024. Large language mon-

keys: Scaling inference compute with repeated sampling. CoRR, abs/2407.21787.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. CoRR, abs/2110.14168.

DeepSeek-AI. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowe Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaqiqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaozhi Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948.

DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingx-
uan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang, Qihou Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuan Chen, Shaqoing Wu, Shengfeng Ye, Shengfeng Ye, Shiron Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, and Wangding Zeng. 2024. Deepseek-v3 technical report. CoRR, abs/2412.19437.

Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. 2024. Rhlf workflow: From reward modeling to online rlhf. Preprint arXiv:2405.07863.

Abhimanyu Dubey, Abhinav Jauhari, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravanakumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Bing Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Alionsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esioibu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Illya Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vraneš, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jieca Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasudevanwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, and et al. 2024. The llama 3 herd of models. CoRR, abs/2407.21783.

Peter E. Hart, Nils J. Nilsson, and Bertram Raphael. 1968. A formal basis for the heuristic determination of minimum cost paths. IEEE Trans. Syst. Sci. Cybern., 4(2):100–107.

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP 2023, Koblenz, Germany, October 23-26, 2023, pages 611–626. ACM.

Minpeng Liao, Chengxi Li, Wei Luo, Jing Wu, and Kai Fan. 2024. MARIO: math reasoning with code interpreter output - A reproducible pipeline. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pages 905–924. Association for Computational Linguistics.

Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. Let’s verify step by step. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.

Yantao Liu, Zijun Yao, Rui Min, Yixin Cao, Lei Hou, and Juanzi Li. 2025. Pairwise rm: Perform best-of-n sampling with knockout tournament. arXiv preprint arXiv:2501.13007.

Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, and Abhinav Rastogi. 2024. Improve mathematical reasoning in language models by automated process supervision. CoRR, abs/2406.06592.

OpenAI. 2024a. GPT-4o.

OpenAI. 2024b. Introducing openai o1.

Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. Scaling LLM test-time compute optimally can be more effective than scaling model parameters. CoRR, abs/2408.03314.

Mingyang Song, Zhaochen Su, Xiaoye Qu, Jiawei Zhou, and Yu Cheng. 2025. Prmbench: A fine-grained and challenging benchmark for process-level reward models. arXiv preprint arXiv:2501.03124.

Benedikt Stroebi, Sayash Kapoor, and Arvind Narayanan. 2024. Inference scaling laws: The limits of llm resampling with imperfect verifiers. arXiv preprint arXiv:2411.17501.
Yuxuan Tong, Xiwen Zhang, Rui Wang, Ruidong Wu, and Junxian He. 2024. Dart-math: Difficulty-aware rejection tuning for mathematical problem-solving. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024.

Jonathan Uesato, Nate Kushman, Ramana Kumar, H. Francis Song, Noah Y. Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. 2022. Solving math word problems with process- and outcome-based feedback. CoRR, abs/2211.14275.

Ziyu Wan, Xidong Feng, Muning Wen, Stephen Marcus McAleer, Ying Wen, Weinan Zhang, and Jun Wang. 2024. Alphazero-like tree-search can guide large language model decoding and training. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net.

Chaojie Wang, Yanchen Deng, Zhiyi Lv, Zeng Liang, Jujie He, Shuicheng Yan, and Bo An. 2024a. Q*: Improving multi-step reasoning for llms with deliberative planning. CoRR, abs/2406.14283.

Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. 2024b. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 9426–9439. Association for Computational Linguistics.

Yu Wang, Nan Yang, Liang Wang, and Furu Wei. 2025. Examining false positives under inference scaling for mathematical reasoning. arXiv preprint arXiv:2502.06217.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022.

Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. 2024. Inference scaling laws: An empirical analysis of compute-optimal inference for llm problem-solving. In The 4th Workshop on Mathematical Reasoning and AI at NeurIPS’24.

Wei Xiong, Hanning Zhang, Nan Jiang, and Tong Zhang. 2024. An implementation of generative prn. https://github.com/RLHF10w/RLHF-Reward-Modeling.

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. 2024. Qwen2.5 technical report. CoRR, abs/2412.15115.

Fei Yu, Anningzhe Gao, and Benyou Wang. 2024a. Ovm, outcome-supervised value models for planning in mathematical reasoning. In Findings of the Association for Computational Linguistics: NAACL 2024, Mexico City, Mexico, June 16-21, 2024, pages 858–875. Association for Computational Linguistics.

Fei Yu, Yingru Li, and Benyou Wang. 2025. Scaling flaws of verifier-guided search in mathematical reasoning. arXiv preprint arXiv:2502.00271.

Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2024b. Meta-math: Bootstrap your own mathematical questions for large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.

Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. 2022. Star: Bootstrapping reasoning with reasoning. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022.

Di Zhang, Jianbo Wu, Jingdi Lei, Tong Che, Jiatao Li, Tong Xie, Xiaoshui Huang, Shufei Zhang, Marco Pavone, Yuqiang Li, Wanli Ouyang, and Dongzhan Zhou. 2024a. Llama-berry: Pairwise optimization for o1-like olympiad-level mathematical reasoning. CoRR, abs/2410.02884.

Hanning Zhang, Pengcheng Wang, Shizhe Diao, Yong Lin, Rui Pan, Hanze Dong, Dylan Zhang, Pavlo Molchanov, and Tong Zhang. 2024b. Entropy-regularized process reward model. CoRR, abs/2412.11006.

Lunjin Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. 2024c. Generative verifiers: Reward modeling as next-token prediction. CoRR, abs/2408.15240.

Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. 2025. The lessons of developing process reward models in mathematical reasoning. arXiv preprint arXiv:2501.07301.

Yuchen Zhuang, Xiang Chen, Tong Yu, Saayan Mitra, Victor S. Bursztyn, Ryan A. Rossi, Somdeb Sarkhel, and Chao Zhang. 2024. Toolchain*: Efficient action space navigation in large language models with a* search. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.
A  Experiment Details

A.1 Baselines.

Outcome Reward Model (ORM, Cobbe et al., 2021). The vanilla ORM assigns a score to the entire solution as the final reward. We train ORMs through outcome supervision. Following (Cobbe et al., 2021), we replicate the binary correctness label \( r_t \in \{0, 1\} \) across the entire solution sequence. The reward head is then trained to predict reward scores for each token, enhancing robustness.

Process Reward Model (PRM, Lightman et al., 2024; Uesato et al., 2022). The vanilla PRM assigns scores to each step along a solution path. For training stability , we place the reward label \( r_t \) at the last token of each step. In other words, for t-th step-level sequence, the label vector \( y_t = [0, 0, \ldots, 0, r_t] \).

Math-shepherd PRM (Wang et al., 2024b). Different from the vanilla PRM, Math-Shepherd PRM uses Monte-Carlo Estimation to annotate step labels. This estimation is essentially considered as training a value model (Zhang et al., 2025). In our experiments, we first sample 15 solutions for each query. Then, for each intermediate step, we sample 8 rollouts. We provide a detailed description of this method in Section 4.2.

ER-PRM (Zhang et al., 2024b). Similar to Math-Shepherd PRM, ER-PRM integrates entropy-regularized step labels to train the supervision model. After Monte-Carlo sampling, ER-PRM calculates the label for the t -th step according to the following equation:

\[
\text{label}(s_t) = \frac{1}{\eta} \ln \mathbb{E}_{\tau-[t] \sim \pi} e^{\eta y(\tau)}
\]

where \( \tau \) represents the complete rollout starting from the step \( s_t \), \( \pi \) represents the LLM generator, and \( y(\cdot) \) represents the final correctness of the solution \( \tau \).

A.2 BiRM Training Details

In the BiRM training phase, we collect problems from the original GSM8K and MATH dataset . Then we use LLM generators to sample 15 trajectories per query, resulting in a training set of approximately 225,000 solutions for each base model. We annotate reward and value labels using the Deepseek-V3 (DeepSeek-AI et al., 2024) and Math-shepherd soft-label methods, respectively. We set training labels on the last token of each step, following (Wang et al., 2024b). The coefficient c in Equation 4 is set to 1.0.

A.3 Evaluation Metrics

At test-time, BiRM estimates both reward scores and value scores for partial solutions at the same time. We follow Equation 3 to calculate the final score. The coefficient \( \beta \) for different base models on GSM8K, MATH-500, and Gaokao2023 are set to \( \beta_{Qwen2.5-3B} = \{1.0, 2.5, 2.0\}, \beta_{Qwen2.5-7B} = \{1.5, 3.0, 3.5\}, \beta_{Llama3.1-8B} = \{2.5, 1.0, 3.5\} \) respectively.

Best-of-N Sampling. For a given question q, we sample multiple rollouts from the LLM, resulting in a candidate set of N reasoning paths \( \mathcal{T} = \{ \tau_1, \tau_2, \ldots, \tau_N \} \). Subsequently, an additional supervision model \( \mathcal{R} \) , such as PRM, is used to score each candidate path, yielding \( \mathcal{R}(\tau_i) \), where \( i \in \{1, 2, \ldots, N\} \). The candidate with the highest score represents the most promising solution and is selected as the final output:

\[
\tau^* = \arg \max_{\tau \in \{ \tau_1, \tau_2, \ldots, \tau_N \}} \mathcal{R}(\tau)
\]

Beam Search. We present all search results from the main experiment in Table 5, Table 6, and Table 7. The procedure of the step-level beam search is as follows: We first set the total sampling size K and beam size b ( K should be divisible by b ) . In each round, we only expand one step forward. For a given query, we sample K rollouts in the first round. Then, we use the supervision model \( \mathcal{M} \) to re-rank these candidates and select the top b rollouts for the next step. Starting from the second round, we expand \( \frac{K}{b} \) trajectories for each candidate, getting K candidates in total. We repeat the re-ranking process until a final answer is found or the maximum step count is reached. A detailed pseudocode is provided in 1.

B  Step Label Annotation Details

B.1 Dataset preprocessing

Before the SFT phase, we first preprocess the training data and restructure the delimiters at different levels of granularity. This is because we discover that original solution paths contain numerous meaningless text segments, which hinder the effective learning of process supervision models. Similar findings are reported by (Liao et al., 2024). To address this, we utilize Deepseek-V3 to clean the
MATH subset in the MetaMath dataset, reannotate the delimiters, and ensure that each step is logically complete and meaningful. The prompt template for data preprocessing is shown in Figure 5.

B.2 Reward Label Annotaion

We also use Deepseek-V3 to annotate the correctness of each step (i.e., reward label) in our experiments. The prompt template is provided in Figure 6.

Algorithm 1 Step-Level Beam Search

1: Input: Question \( q \), Total Sampling Size \( K \), Beam size \( b \), Maximum step count \( T \)
2: Output: Best solution path for \( q \)
3: Model: Generator \( \pi \) and BiRM \( \mathcal{M} \)
4: procedure STEPLEVELBEAMSEARCH(q, K, b)
5: Initialize partial solutions \( \mathbb{T} \leftarrow \{ \} \)
6: Sample initial steps \( \{ \tau_1^1, \tau_2^1, \ldots, \tau_K^1 \} \)
7: Estimate scores \( \{ s_1^1, s_2^1, \cdots, s_K^1 \} \) for each step
8: Select top \( b \) scored steps and add to \( \mathbb{T} \)
9: \( t \leftarrow 1 \)
10: while solutions in \( \mathbb{T} \) are not complete and \( t < T \) do
11:     New candidate solutions \( \mathbb{T}_{\text{new}} \leftarrow \{ \} \)
12:     Scores \( S \leftarrow \{ \} \)
13:     for each partial solution \( \tau^{[1:t]} \) in \( \mathbb{T} \) do
14:         for \( i = 1 \) to \( K/b \) do
15:             \( \tau^{[1:t+1]} \sim \pi(\tau^{[1:t]}, q) \)
16:             \( s_i^{[1:t+1]} = \mathcal{M}(\tau_i^{[1:t+1]}, q) \)
17:             \( \mathbb{T}_{\text{new}} \leftarrow \mathbb{T}_{\text{new}} + \tau^{[1:t+1]} \)
18:             \( S \leftarrow S + s_i^{[1:t+1]} \)
19:         end for
20:     end for
21:     \( \mathbb{T}_{\text{new}} \leftarrow \) top \( b \) scored partial solutions in \( \mathbb{T}_{\text{new}} \)
22:     \( \mathbb{T} \leftarrow \mathbb{T}_{\text{new}} \)
23:     \( t \leftarrow t + 1 \)
24: end while
25: return solution with the highest score in \( \mathbb{T} \)
26: end procedure
<table>
  <tr>
    <th rowspan="2">Total Size</th>
    <th rowspan="2">Beam Size</th>
    <th colspan="3">GSM8K</th>
    <th colspan="3">MATH-500</th>
    <th colspan="3">Gaokao2023</th>
  </tr>
  <tr>
    <th>OVM</th><th>PRM</th><th>BiRM</th>
    <th>OVM</th><th>PRM</th><th>BiRM</th>
    <th>OVM</th><th>PRM</th><th>BiRM</th>
  </tr>
  <tr>
    <td rowspan="3">K = 4</td>
    <td>4</td><td>81.50 ± 0.45</td><td>82.11 ± 0.28</td><td>81.96 ± 0.38</td><td>48.60 ± 0.16</td><td>48.13 ± 1.20</td><td>47.77 ± 0.96</td><td>33.85 ± 0.53</td><td>33.16 ± 0.74</td><td>33.07 ± 0.44</td>
  </tr>
  <tr>
    <td>2</td><td>82.97 ± 0.14</td><td>81.53 ± 0.72</td><td>82.76 ± 0.28</td><td>48.27 ± 1.61</td><td>49.27 ± 0.81</td><td>50.07 ± 0.84</td><td>35.15 ± 1.80</td><td>34.55 ± 0.56</td><td>36.10 ± 0.76</td>
  </tr>
  <tr>
    <td>1</td><td>80.82 ± 0.51</td><td>80.57 ± 0.77</td><td>81.93 ± 0.64</td><td>47.60 ± 0.59</td><td>47.80 ± 1.14</td><td>47.67 ± 0.57</td><td>35.58 ± 1.29</td><td>34.89 ± 1.56</td><td>32.47 ± 0.56</td>
  </tr>
  <tr>
    <td rowspan="3">K = 8</td>
    <td>8</td><td>83.70 ± 0.73</td><td>83.65 ± 0.25</td><td>84.41 ± 0.35</td><td>49.33 ± 0.38</td><td>50.13 ± 1.00</td><td>50.07 ± 0.66</td><td>35.93 ± 1.17</td><td>34.63 ± 0.49</td><td>35.06 ± 1.39</td>
  </tr>
  <tr>
    <td>4</td><td>84.61 ± 0.56</td><td>83.93 ± 0.16</td><td>85.11 ± 0.40</td><td>48.87 ± 0.68</td><td>50.87 ± 0.41</td><td>52.33 ± 0.94</td><td>36.10 ± 1.10</td><td>37.92 ± 0.76</td><td>37.92 ± 0.21</td>
  </tr>
  <tr>
    <td>2</td><td>84.10 ± 0.36</td><td>83.17 ± 0.25</td><td>84.00 ± 0.39</td><td>50.07 ± 1.06</td><td>50.33 ± 0.94</td><td>50.67 ± 1.64</td><td>35.32 ± 1.10</td><td>37.58 ± 1.09</td><td>36.97 ± 1.07</td>
  </tr>
  <tr>
    <td rowspan="3">K = 20</td>
    <td>20</td><td>85.27 ± 0.04</td><td>85.65 ± 0.50</td><td>86.13 ± 0.11</td><td>52.11 ± 1.15</td><td>53.20 ± 0.59</td><td>53.33 ± 1.48</td><td>36.54 ± 0.44</td><td>35.67 ± 1.56</td><td>36.10 ± 1.85</td>
  </tr>
  <tr>
    <td>10</td><td>86.73 ± 0.63</td><td>84.66 ± 0.34</td><td>86.91 ± 0.25</td><td>53.00 ± 0.16</td><td>54.27 ± 0.77</td><td>55.00 ± 0.65</td><td>37.66 ± 1.48</td><td>38.35 ± 1.24</td><td>37.23 ± 1.38</td>
  </tr>
  <tr>
    <td>5</td><td>86.23 ± 0.28</td><td>84.86 ± 0.54</td><td>86.28 ± 0.33</td><td>52.20 ± 0.59</td><td>53.40 ± 0.85</td><td>54.27 ± 0.52</td><td>36.88 ± 1.06</td><td>37.49 ± 0.86</td><td>37.58 ± 0.24</td>
  </tr>
  <tr>
    <td rowspan="3">K = 100</td>
    <td>4</td><td>86.20 ± 0.16</td><td>84.76 ± 0.22</td><td>85.04 ± 0.19</td><td>51.73 ± 0.66</td><td>51.80 ± 0.75</td><td>53.60 ± 0.75</td><td>37.49 ± 0.88</td><td>35.41 ± 1.00</td><td>39.05 ± 1.17</td>
  </tr>
  <tr>
    <td>2</td><td>85.32 ± 0.25</td><td>84.74 ± 0.36</td><td>85.19 ± 0.49</td><td>49.00 ± 0.33</td><td>53.00 ± 1.31</td><td>51.80 ± 0.49</td><td>35.67 ± 0.44</td><td>35.84 ± 0.97</td><td>36.62 ± 0.85</td>
  </tr>
  <tr>
    <td>1</td><td>83.60 ± 0.09</td><td>82.56 ± 0.74</td><td>84.23 ± 0.39</td><td>49.00 ± 0.91</td><td>50.67 ± 0.90</td><td>50.87 ± 0.84</td><td>34.29 ± 1.29</td><td>34.46 ± 1.41</td><td>37.06 ± 2.51</td>
  </tr>
  <tr>
    <td rowspan="3">K = 100</td>
    <td>50</td><td>87.29 ± 0.22</td><td>85.87 ± 0.64</td><td>87.34 ± 0.22</td><td>52.87 ± 0.82</td><td>53.87 ± 0.19</td><td>55.13 ± 0.34</td><td>37.06 ± 0.74</td><td>37.40 ± 0.97</td><td>38.96 ± 0.92</td>
  </tr>
  <tr>
    <td>25</td><td>87.54 ± 0.26</td><td>85.52 ± 0.80</td><td>87.64 ± 0.65</td><td>53.00 ± 1.50</td><td>53.20 ± 0.33</td><td>54.73 ± 0.75</td><td>38.10 ± 1.17</td><td>37.75 ± 1.09</td><td>38.18 ± 0.21</td>
  </tr>
  <tr>
    <td>10</td><td>85.90 ± 0.33</td><td>84.51 ± 0.77</td><td>86.71 ± 0.37</td><td>51.27 ± 1.32</td><td>49.80 ± 0.57</td><td>53.40 ± 1.23</td><td>38.01 ± 1.05</td><td>37.92 ± 1.10</td><td>37.40 ± 1.85</td>
  </tr>
</table>

Table 5: Qwen2.5-3B performance of beam search on GSM8K, MATH-500 and Gaokao2023.

<table>
  <tr>
    <th rowspan="2">Total Size</th>
    <th rowspan="2">Beam Size</th>
    <th colspan="3">GSM8K</th>
    <th colspan="3">MATH-500</th>
    <th colspan="3">Gaokao2023</th>
  </tr>
  <tr>
    <th>OVM</th><th>PRM</th><th>BiRM</th>
    <th>OVM</th><th>PRM</th><th>BiRM</th>
    <th>OVM</th><th>PRM</th><th>BiRM</th>
  </tr>
  <tr>
    <td rowspan="3">K = 4</td>
    <td>4</td><td>86.10 ± 0.52</td><td>86.48 ± 0.43</td><td>87.04 ± 0.16</td><td>55.73 ± 1.52</td><td>53.73 ± 1.51</td><td>57.13 ± 1.15</td><td>40.09 ± 0.12</td><td>41.13 ± 1.24</td><td>43.90 ± 0.92</td>
  </tr>
  <tr>
    <td>2</td><td>86.20 ± 0.53</td><td>86.00 ± 0.25</td><td>86.99 ± 0.13</td><td>55.53 ± 0.47</td><td>55.80 ± 1.72</td><td>55.87 ± 0.93</td><td>42.77 ± 0.24</td><td>42.68 ± 0.88</td><td>43.55 ± 0.61</td>
  </tr>
  <tr>
    <td>1</td><td>85.65 ± 1.01</td><td>85.04 ± 0.50</td><td>86.76 ± 0.31</td><td>53.80 ± 1.77</td><td>54.93 ± 1.46</td><td>56.33 ± 1.00</td><td>41.47 ± 0.74</td><td>43.98 ± 0.12</td><td>44.50 ± 1.17</td>
  </tr>
  <tr>
    <td rowspan="3">K = 8</td>
    <td>8</td><td>86.73 ± 0.62</td><td>88.12 ± 0.36</td><td>88.93 ± 0.49</td><td>58.27 ± 0.50</td><td>58.20 ± 0.71</td><td>58.13 ± 0.90</td><td>44.19 ± 0.76</td><td>44.24 ± 0.68</td><td>45.11 ± 0.68</td>
  </tr>
  <tr>
    <td>4</td><td>88.63 ± 0.19</td><td>87.89 ± 0.73</td><td>89.36 ± 0.22</td><td>57.60 ± 0.85</td><td>59.17 ± 1.32</td><td>59.53 ± 1.24</td><td>43.72 ± 0.86</td><td>45.63 ± 1.21</td><td>46.84 ± 0.65</td>
  </tr>
  <tr>
    <td>2</td><td>88.55 ± 0.37</td><td>87.45 ± 0.19</td><td>88.30 ± 0.53</td><td>57.00 ± 0.43</td><td>57.20 ± 1.56</td><td>58.67 ± 1.27</td><td>43.98 ± 1.41</td><td>44.68 ± 0.56</td><td>45.45 ± 0.97</td>
  </tr>
  <tr>
    <td rowspan="3">K = 20</td>
    <td>20</td><td>86.33 ± 0.38</td><td>88.65 ± 0.09</td><td>90.04 ± 0.58</td><td>59.07 ± 0.82</td><td>59.60 ± 0.59</td><td>60.33 ± 0.68</td><td>44.76 ± 1.38</td><td>45.89 ± 1.21</td><td>47.71 ± 0.44</td>
  </tr>
  <tr>
    <td>10</td><td>90.40 ± 0.18</td><td>89.18 ± 0.42</td><td>90.40 ± 0.65</td><td>58.73 ± 1.16</td><td>61.55 ± 1.05</td><td>62.27 ± 1.09</td><td>45.19 ± 0.76</td><td>48.14 ± 0.74</td><td>48.23 ± 0.12</td>
  </tr>
  <tr>
    <td>5</td><td>90.30 ± 0.12</td><td>88.98 ± 0.26</td><td>90.08 ± 0.28</td><td>57.53 ± 0.34</td><td>59.40 ± 0.49</td><td>60.73 ± 0.19</td><td>45.45 ± 0.64</td><td>47.36 ± 1.17</td><td>47.36 ± 1.22</td>
  </tr>
  <tr>
    <td rowspan="3">K = 100</td>
    <td>4</td><td>89.56 ± 0.25</td><td>87.52 ± 0.62</td><td>89.94 ± 0.09</td><td>56.53 ± 0.90</td><td>58.40 ± 1.28</td><td>59.67 ± 0.34</td><td>45.02 ± 0.53</td><td>45.44 ± 0.88</td><td>48.81 ± 2.21</td>
  </tr>
  <tr>
    <td>2</td><td>88.55 ± 0.06</td><td>88.05 ± 0.07</td><td>89.69 ± 0.34</td><td>56.93 ± 0.90</td><td>57.41 ± 1.11</td><td>58.87 ± 0.62</td><td>43.72 ± 1.07</td><td>44.59 ± 0.74</td><td>47.62 ± 0.96</td>
  </tr>
  <tr>
    <td>1</td><td>87.79 ± 0.22</td><td>86.96 ± 0.47</td><td>87.08 ± 0.38</td><td>56.27 ± 0.34</td><td>57.73 ± 1.52</td><td>58.33 ± 0.77</td><td>42.68 ± 1.71</td><td>45.63 ± 0.74</td><td>45.80 ± 0.86</td>
  </tr>
  <tr>
    <td rowspan="3">K = 100</td>
    <td>50</td><td>91.00 ± 0.22</td><td>88.32 ± 0.57</td><td>91.28 ± 0.12</td><td>60.13 ± 0.47</td><td>60.73 ± 0.34</td><td>62.53 ± 0.77</td><td>46.84 ± 0.44</td><td>48.31 ± 0.85</td><td>49.96 ± 0.32</td>
  </tr>
  <tr>
    <td>25</td><td>91.18 ± 0.53</td><td>88.40 ± 0.21</td><td>91.66 ± 0.33</td><td>58.40 ± 1.23</td><td>59.27 ± 0.34</td><td>62.00 ± 0.98</td><td>46.32 ± 0.53</td><td>47.97 ± 0.12</td><td>47.62 ± 0.68</td>
  </tr>
  <tr>
    <td>10</td><td>89.77 ± 0.25</td><td>88.15 ± 0.16</td><td>91.00 ± 0.09</td><td>57.47 ± 1.32</td><td>59.20 ± 1.31</td><td>61.20 ± 0.43</td><td>43.64 ± 0.42</td><td>46.93 ± 0.49</td><td>49.00 ± 0.32</td>
  </tr>
</table>

Table 6: Qwen2.5-7B performance of beam search on GSM8K, MATH-500 and Gaokao2023.

<table>
  <tr>
    <th rowspan="2">Total Size</th>
    <th rowspan="2">Beam Size</th>
    <th colspan="3">GSM8K</th>
    <th colspan="3">MATH-500</th>
    <th colspan="3">Gaokao2023</th>
  </tr>
  <tr>
    <th>OVM</th><th>PRM</th><th>BiRM</th>
    <th>OVM</th><th>PRM</th><th>BiRM</th>
    <th>OVM</th><th>PRM</th><th>BiRM</th>
  </tr>
  <tr>
    <td rowspan="3">K = 4</td>
    <td>4</td><td>71.44 ± 0.36</td><td>71.65 ± 0.33</td><td>71.37 ± 0.41</td><td>37.53 ± 0.66</td><td>38.67 ± 0.50</td><td>38.07 ± 0.68</td><td>23.81 ± 0.65</td><td>24.94 ± 0.21</td><td>23.29 ± 0.86</td>
  </tr>
  <tr>
    <td>2</td><td>72.76 ± 0.41</td><td>71.11 ± 1.11</td><td>72.91 ± 0.80</td><td>38.53 ± 2.22</td><td>39.87 ± 0.68</td><td>40.73 ± 0.52</td><td>23.90 ± 1.10</td><td>25.11 ± 1.36</td><td>26.06 ± 0.68</td>
  </tr>
  <tr>
    <td>1</td><td>70.74 ± 0.16</td><td>68.99 ± 0.67</td><td>71.57 ± 0.38</td><td>36.80 ± 0.75</td><td>39.60 ± 0.85</td><td>39.33 ± 0.94</td><td>23.20 ± 0.74</td><td>24.33 ± 0.74</td><td>24.59 ± 0.65</td>
  </tr>
  <tr>
    <td rowspan="3">K = 8</td>
    <td>8</td><td>76.52 ± 0.45</td><td>75.92 ± 0.20</td><td>76.90 ± 0.53</td><td>39.93 ± 1.48</td><td>39.00 ± 0.59</td><td>41.27 ± 0.09</td><td>25.63 ± 1.07</td><td>25.71 ± 0.56</td><td>26.75 ± 0.56</td>
  </tr>
  <tr>
    <td>4</td><td>77.36 ± 0.47</td><td>75.84 ± 0.54</td><td>78.32 ± 0.55</td><td>40.20 ± 0.49</td><td>40.13 ± 1.64</td><td>43.27 ± 0.57</td><td>25.19 ± 1.29</td><td>26.49 ± 1.53</td><td>27.45 ± 1.56</td>
  </tr>
  <tr>
    <td>2</td><td>75.51 ± 0.49</td><td>73.79 ± 1.02</td><td>76.17 ± 0.04</td><td>39.07 ± 0.50</td><td>39.80 ± 1.85</td><td>41.47 ± 1.23</td><td>25.02 ± 0.96</td><td>26.58 ± 2.04</td><td>26.23 ± 1.18</td>
  </tr>
  <tr>
    <td rowspan="3">K = 20</td>
    <td>20</td><td>79.93 ± 0.22</td><td>79.23 ± 0.48</td><td>80.46 ± 0.29</td><td>41.53 ± 0.84</td><td>41.00 ± 0.75</td><td>44.13 ± 0.19</td><td>26.15 ± 0.74</td><td>25.71 ± 0.21</td><td>27.02 ± 0.44</td>
  </tr>
  <tr>
    <td>5</td><td>79.76 ± 0.37</td><td>76.90 ± 0.35</td><td>81.00 ± 0.20</td><td>40.80 ± 0.71</td><td>42.07 ± 1.09</td><td>43.93 ± 1.00</td><td>27.01 ± 0.37</td><td>28.14 ± 0.98</td><td>28.57 ± 0.73</td>
  </tr>
  <tr>
    <td>4</td><td>79.56 ± 0.26</td><td>76.02 ± 0.36</td><td>81.06 ± 0.64</td><td>40.13 ± 1.55</td><td>42.00 ± 1.07</td><td>44.00 ± 0.43</td><td>24.33 ± 0.32</td><td>28.23 ± 0.12</td><td>26.93 ± 0.80</td>
  </tr>
  <tr>
    <td rowspan="3">K = 100</td>
    <td>2</td><td>77.96 ± 0.60</td><td>75.39 ± 1.04</td><td>79.53 ± 1.08</td><td>39.07 ± 0.90</td><td>39.53 ± 0.50</td><td>42.40 ± 0.66</td><td>26.58 ± 1.41</td><td>26.75 ± 0.85</td><td>26.84 ± 0.12</td>
  </tr>
  <tr>
    <td>1</td><td>76.27 ± 0.80</td><td>73.24 ± 1.02</td><td>78.17 ± 0.87</td><td>39.27 ± 0.82</td><td>40.00 ± 1.82</td><td>41.53 ± 1.36</td><td>25.54 ± 0.24</td><td>27.36 ± 1.44</td><td>27.27 ± 0.56</td>
  </tr>
  <tr>
    <td>50</td><td>82.71 ± 0.11</td><td>80.34 ± 0.88</td><td>85.39 ± 0.52</td><td>41.07 ± 0.50</td><td>43.23 ± 0.66</td><td>46.13 ± 0.98</td><td>26.23 ± 0.20</td><td>29.61 ± 0.37</td><td>30.65 ± 0.37</td>
  </tr>
  <tr>
    <td rowspan="3">K = 100</td>
    <td>25</td><td>82.71 ± 0.61</td><td>78.44 ± 0.43</td><td>84.53 ± 0.60</td><td>40.93 ± 1.32</td><td>42.00 ± 0.71</td><td>44.07 ± 0.68</td><td>25.37 ± 0.61</td><td>28.14 ± 0.86</td><td>29.00 ± 0.61</td>
  </tr>
  <tr>
    <td>10</td><td>81.10 ± 0.46</td><td>77.81 ± 0.93</td><td>83.35 ± 0.70</td><td>39.87 ± 0.77</td><td>40.27 ± 1.06</td><td>45.00 ± 0.16</td><td>25.80 ± 0.32</td><td>27.19 ± 0.86</td><td>29.70 ± 0.80</td>
  </tr>
</table>

Table 7: Llama3.1-8B performance of beam search on GSM8K, MATH-500 and Gaokao2023.
SFT Dataset Preprocessing

You are an expert math examiner, skilled at transforming complex mathematical solution steps into clear formats. Your task is to insert the symbol <step_end> to mark the end of each step in the following math problem's solution. A step should represent a complete statement, structure or calculation process. You must not omit any original content and only insert this symbol at the end of each step. Your output should only include the revised solution, without any additional text.

Figure 5: The prompt template for MetaMath dataset preprocessing.

Reward Label Annotation

You are an expert math examiner. Your task is to review the student's solution and evaluate each step. Mark a step as correct only if it is based on accurate premises and contributes to solving the problem. Mark it as unnecessary if it is logically valid but does not aid in solving the problem. Your judgments should include a very concise analysis of each step and the final judgement.

You must provide your evaluations in JSON format like:
{"step_1": {"analysis": "<concise analysis of the step>", "judgement": "<correct/incorrect/unnecessary>"}, "step_2": {...}, ...}

Below is the question, reference answer, and student's solution that you need to evaluate. Note that the student's solution does not need to match the reference solution exactly.

[Question]
{question}

[Reference Answer]
{answer}

[Solution]
{solution}

Now, provide your evaluations in JSON format.

Figure 6: The prompt template for reward label annotation.

Reviewer paper summaries: This work proposes an improved process reward model, namely BiRM, to evaluates both the correctness of previous steps and the probability of future success. Extensive experiments on three mathematical reasoning datasets show some performance improvements when using Best-of-N sampling and beam search.
Summary of strengths: 1. The motivation of a Bi-directional reward model to consider both the previous process and the future success is reasonable. The proposed reward model seems to perform better than a single PRM or a ORM on some datasets using certain sampling strategy.
Summary of weaknesses: 1. The idea is too simple and lack of sufficient work. 
2. Some results in Table 1 and Table 2 indicate that the improvements are marginal. Besides, the performances of sampling inference strategy should be listed in a form “average +standard deviation” from multiple running times to show their statistical significance.
3. My major concern lies in the necessity of training such a BiRM model instead of simply using a PRM together with a separate VM in the inference process. Further comparison with this baseline maybe involved in experiments to show the necessity of training a PRM head and a VM head sharing other model weights.
Comments, suggestions and typos: The writing could be further improved.

Typos:
In Figure 2, there’s the grammar error in “but the latter seems a more promising path forward”.
Overall assessment scores: 2