The Lessons of Developing Process Reward Models in Mathematical Reasoning
Authors: Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, Junyang Lin
Accepted at: acl25

The Lessons of Developing Process Reward Models in Mathematical Reasoning

Zhenru Zhang    Chujie Zheng    Yangzhen Wu    Beichen Zhang    Runji Lin
Bowen Yu*    Dayiheng Liu*    Jingren Zhou    Junyang Lin*
Qwen Team, Alibaba Group
zhangzhenru.zrzr@alibaba-inc.com
https://hf.co/Qwen/Qwen2.5-Math-PRM-7B
https://hf.co/Qwen/Qwen2.5-Math-PRM-72B

Abstract
Process Reward Models (PRMs) aim to identify and mitigate intermediate errors in the reasoning processes in mathematical reasoning of Large Language Models (LLMs). However, the development of effective PRMs faces significant challenges, particularly in data annotation and evaluation methodologies. In this paper, through extensive experiments, we demonstrate that commonly used Monte Carlo (MC) estimation-based data synthesis for PRMs typically yields inferior performance and generalization compared to LLM-as-a-judge and human annotation methods. Furthermore, we identify potential biases in conventional Best-of-N (BoN) evaluation strategies for PRMs. To address these challenges, we develop a consensus filtering mechanism that effectively integrates MC estimation with LLM-as-a-judge and advocates a more comprehensive evaluation framework that combines response-level and step-level metrics. Based on the mechanisms, we significantly improve both model performance and data efficiency in the BoN evaluation and the step-wise error identification task. Finally, we release a new state-of-the-art PRM that outperforms existing open-source alternatives and provides practical guidelines for future research.

1 Introduction
In recent years, Large Language Models (LLMs) have made remarkable advances in mathematical reasoning (OpenAI, 2023; Dubey et al., 2024; Shao et al., 2024; Zhu et al., 2024; Yang et al., 2024a,c,b), yet they can make mistakes, leading to wrong conclusions. Moreover, even when achieving correct final answers, these powerful models can still regularly use flawed reasoning steps, which undermine the reliability and trustworthiness of LLMs’ reasoning processes. To address these challenges, Process Reward Models (PRMs; Lightman et al. 2023; Wang et al. 2024b) are proposed to identify and mitigate process errors, thereby enabling finer-grained supervision on the reasoning process.

One critical challenge of developing PRMs lies in the data annotation for the correctness of reasoning processes, which is typically expensive and time-consuming. While Lightman et al. (2023) recruited human annotators with detailed instructions and elaborate procedures to achieve satisfactory annotation quality, the prohibitive cost pushes researchers to explore automated annotation methods. Among them, one commonly used approach is to assess process correctness by estimating the empirical probability of leading to the correct final answers through Monte Carlo (MC) methods, which has attracted great research interests and has also been commonly employed in practice (Xiong et al., 2024; Wang et al., 2024b; Luo et al., 2024). Another challenge lies in evaluating PRM performance, as previous studies (Lightman et al., 2023; Wang et al., 2024b; Luo et al., 2024) have predominantly relied on the Best-of-N (BoN) evaluation, which selects the highest-scored response from \( N \) candidates according to a PRM. Recently, PROCESSBENCH (Zheng et al., 2024) have emerged to evaluate the capability of PRMs in identifying step-wise correctness.

Nevertheless, during the training of our own PRM following conventional principles to construct data using MC estimation and evaluate on BoN, we gain several crucial lessons. **In terms of MC estimation**, (1) we observe that the PRM trained via MC estimation demonstrated significantly inferior performance and generalization capabilities compared to LLM-as-a-judge (Zheng et al., 2023) and human annotation. (2) We attribute the suboptimal performance of MC estimation to its fundamental limitation, which attempts to evaluate deterministic current-step correctness based on potential future outcomes. It significantly relies on the performance of the completion model,
Figure 1: Overview of evaluation results on the Best-of-8 strategy of the policy model Qwen2.5-Math-7B-Instruct and the benchmark PROCESSBENCH (Zheng et al., 2024) across multiple PRMs (see Table 3 and Table 4 for details).

which may generate correct answers based on incorrect steps, or incorrect answers based on correct steps, introducing substantial noise and inaccuracy verification into step-wise correctness estimation. Regarding the BoN evaluation, (1) the unreliable policy models generate responses with correct answers but flawed processes, leading to a misalignment between the outcome evaluation criteria of BoN and the PRM objectives of process verification. (2) The PRMs with limited process verification capability demonstrate tolerance for these cases, resulting in inflated BoN performance. (3) We find that in the step scores distribution of existing PRMs, a significant proportion of minimum scores are concentrated on the final answer steps, indicating PRMs have shifted from process to outcome-based assessment in BoN.

To address these challenges, we propose a consensus filtering mechanism that combines MC estimation with LLM-as-a-judge, retaining only instances where both agree on error locations in the solution. Our approach improves both data efficiency and performance over existing PRMs in the conventional BoN evaluation. Furthermore, we advocate for complementing response-level BoN with step-wise evaluation methods. We employ the step-wise benchmark PROCESSBENCH (Zheng et al., 2024) to measure the ability to identify process errors. Our trained PRMs exhibit impressively stronger error identification performance than other open-source models, from PRMs to general language models, confirming that our training approach genuinely teaches PRMs to assess the correctness of intermediate reasoning steps.

2 Preliminary Trials

In this section, we describe our preliminary attempts to train PRMs via MC estimation-based reasoning step annotation. Despite our efforts in scaling up training data and careful tuning of training objectives, we found that the MC estimation-based PRMs do not possess noticeable advantages over the one trained on human-annotated data, and even lag significantly behind the latter in identifying specific erroneous reasoning steps.

2.1 Training Setup

Training Data Synthesis  We followed the commonly used MC estimation approach, Math-Shepherd (Wang et al., 2024b), to construct the PRM training data. Specifically, we collected a large-scale dataset of approximately 500,000 queries with golden answers. For each query, we generate 6-8 diverse responses by mixing outputs from the Qwen2-Math-Instruct and Qwen2.5-Math-Instruct series models (Yang et al., 2024c), spanning the model sizes of 7B and 72B parameters. These responses are systematically split into individual steps using the delimiter “\n\n”. To assess the correctness of each step, we conduct 8 independent completions starting from this step using Qwen2.5-Math-Instruct series with the corresponding model size, estimating the step labels based on the empirical probabilities of each step yielding the
correct final answer. We trained PRMs with either hard labels or soft labels. For hard labels, we treat a step as correct if any one of the 8 completions yields the correct final answer, and negative otherwise. For soft labels, we determined the value (between 0 and 1) as the proportion of completions leading to the correct final answers. Note that we eliminated all steps subsequent to those labeled as incorrect (label 0), as their validity becomes irrelevant after an error occurs. This removal was implemented to prevent potential model confusion during training.

Training Details Our trained PRMs were initialized from the supervised fine-tuned Qwen2.5-Math-7B/72B-Instruct models (Yang et al., 2024c), where we replace the original language modeling head (used for next token prediction) with a scalar-value head, consisting of two linear layers. We calculated the cross-entropy (CE) loss and mean squared error (MSE) loss on the last tokens of each step for the binary classification task using hard labels and for the regression task using soft labels, respectively.

2.2 Evaluation Setup

Best-of-N Consistent with previous work (Lightman et al., 2023; Wang et al., 2024b; Luo et al., 2024; Cobbe et al., 2021; Yang et al., 2024c), we employed the BoN evaluation, which selects the highest-scored response from \( N \) candidates according to a PRM, denoting as “prm@N”. Following Yang et al. (2024c), we sampled 8 responses (i.e., \( N = 8 \)) from Qwen2.5-Math-7B-Instruct across multiple mathematical benchmarks, including GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021b), Minerva Math (Lewkowycz et al., 2022), GaoKao 2023 En (Liao et al., 2024), OlympiadBench (He et al., 2024), College Math (Tang et al., 2024), and MMLU STEM (Hendrycks et al., 2021a). Each candidate response is scored using the product of all the individual scores of each step within the response, as computed in Lightman et al. (2023). We also report the result of majority voting among 8 responses (maj@8) as baseline, and pass@8 (i.e., the proportion of test samples where any of the 8 samplings lead to the correct final answers) as upper bound.

PROCESSBENCH We also evaluated on PROCESSBENCH (Zheng et al., 2024) as a complement which measures the capability of models to identify erroneous steps in mathematical reasoning. Models are required to identify the first step that contains an error or conclude that all steps are correct. Following the evaluation methods for PRMs in PROCESSBENCH, we locate the first erroneous step from predict scores yielded by PRMs.

<table>
  <tr>
    <th>Setting</th>
    <th colspan="2">Best-of-8</th>
    <th>PROCESSBENCH</th>
  </tr>
  <tr>
    <th></th>
    <th>Mean Acc</th>
    <th></th>
    <th>Mean F1</th>
  </tr>
  <tr>
    <td>pass@8 (Upper Bound)</td>
    <td>74.7</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>maj@8</td>
    <td><b>66.2</b></td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>PRM800K</td>
    <td>64.9</td>
    <td><b>56.5</b></td>
    <td>-</td>
  </tr>
  <tr>
    <td>MC estimated hard labels</td>
    <td>65.5</td>
    <td>40.2</td>
    <td>-</td>
  </tr>
  <tr>
    <td>MC estimated soft labels</td>
    <td>64.4</td>
    <td>40.2</td>
    <td>-</td>
  </tr>
</table>

Table 1: Preliminary trials results on Best-of-8 and PROCESSBENCH using PRMs trained with MC estimated hard labels and soft labels, human-annotated PRM800K.

2.3 Evaluation Results

As shown in Table 1, we separately demonstrate the average results on Best-of-8 and PROCESSBENCH diverse mathematical tasks. To serve as a baseline, we also collect human-annotated data PRM800K (Lightman et al., 2023) and train with its hard labels. The comprehensive results are shown in Appendix C.1. It reveal two critical limitations: (1) In Best-of-8 evaluations, our trained models could not surpass the performance of simple majority voting, i.e., maj@8. (2) When evaluate on PROCESSBENCH for identifying erroneous reasoning steps, the models trained on MC estimated data perform significantly worse than the one trained on PRM800K, though the former had larger scale of data.

These undesirable evaluation performances push us to reflect on the currently prevalent data synthesis approach and evaluation strategy. Through the subsequent optimization process, we have indeed gained several observations and lessons learned.

3 The Lessons

In this section, we present the critical lessons gained during the PRM training comprising two main aspects: (1) the limitations of commonly adopted MC estimation approaches in PRMs training, and (2) the bias in using BoN as the sole evaluation metric for optimizing PRMs.

3.1 Limitations of MC Estimation

3.1.1 PRMs vs. Value Models

PRMs provide fine-grained supervision by evaluating the correctness of intermediate reasoning steps. In contrast, value models estimate the potential of reaching the correct final answer from the current
<table>
  <tr>
    <th>Setting</th>
    <th># samples</th>
    <th colspan="2">Best-of-8</th>
    <th colspan="2">PROCESSBENCH</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th>Mean Acc</th>
    <th>F1</th>
    <th>Mean F1</th>
    <th></th>
  </tr>
  <tr>
    <td>MC Estimation (Math-Shepherd)</td>
    <td>440k</td>
    <td>64.3</td>
    <td>28.9</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>MC Estimation (our data)</td>
    <td>860k</td>
    <td><b>65.9</b></td>
    <td>40.1</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>LLM-as-a-judge (our data)</td>
    <td>860k</td>
    <td>65.3</td>
    <td>46.5</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Human Annotation (PRM800K)</td>
    <td>264k</td>
    <td>64.9</td>
    <td><b>56.5</b></td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 2: PRMs performance comparison on the Best-of-8 and PROCESSBENCH. The models are trained on the different data construction methods including MC estimation, LLM-as-a-judge, and human annotation.

step in the future. The key difference between PRM and value model lies in that PRMs function as deterministic evaluators of current step correctness, while value models operate as predictive estimators of future solution potential.

MC estimation attempts to estimate the potential of reaching the correct final answer in the future from the current step. When we follow this approach to construct data and train the PRMs, the value model principles are incorporated into PRMs training essentially. This methodology potentially introduces performance and generalization limitations which we discuss in subsequent sections.

3.1.2 MC Estimation vs. LLM-as-a-judge vs. Human Annotation

Since the observation of MC estimation’s limitations of identifying erroneous steps in Section 2, we conducted a comprehensive comparison of three distinct data construction approaches: MC estimation, LLM-as-a-judge, and human annotation. For the MC estimation approach, we respectively train the PRM on 445k open-source datasets Math-shepherd (Wang et al., 2024b) and our 860k similarly constructed dataset. For the LLM-as-a-judge approach, we use the same 860k dataset and employ Qwen2.5-72B-Instruct (Yang et al., 2024b) to verify the correctness of each step in the responses with the prompt template shown in Appendix E. For the human annotation approach, we use the open-source dataset PRM800K (Lightman et al., 2023) which consists of approximately 265k samples after deduplication against the test set.

The experimental results of Best-of-8 and PROCESSBENCH are shown in Table 2 and the detailed results are shown in Appendix C.2. In general, for Best-of-8, the PRM trained on our MC estimated data achieves the best accuracy and human annotation shows substantially inferior performance. For PROCESSBENCH, human annotation achieves the best performance with the least amount of data, followed by LLM-as-a-judge, while MC estimation performs the worst despite having the largest dataset overall. The contrasting trend in the two evaluation catches our attention and is thoroughly investigated in Section 3.2.

In terms of the ability of identifying the correctness of reasoning steps evaluated in PROCESSBENCH, it can be found that: (1) human annotation, despite being only performed on the MATH dataset, exhibited superior generalization capabilities on more complex tasks, such as Olympiad-Bench and Omni-MATH. (2) Given identical data with different annotation approaches, LLM-as-a-judge demonstrates better generalization performance on challenging problems than MC estimation. (3) For MC estimation, a comparison between our 860k dataset and Math-Shepherd 440k data indicates that performance improvements can still be achieved through data scaling.

3.1.3 Stringent Data Filtering Mechanisms Required in MC Estimation

The inferior performance of MC estimation compared to LLM-as-a-judge and human annotation can be attributed to its high noise in evaluating reasoning step correctness and identifying error positions, due to its heavy dependence on the policy model. The policy model may generate correct final answers despite incorrect reasoning steps, or incorrect answers based on correct steps, contributing to this inaccuracy.

Motivated by LLM-as-a-judge’s encouraging results in Best-of-8 and PROCESSBENCH in Table 2, we naturally propose a simple yet efficient consensus filtering mechanism that integrates LLM-as-a-judge with MC estimation. Based on the aforementioned 860K samples, the instances are only retained when both LLM-as-a-judge and MC estimation show consensus on the error reasoning step locations in the solution. As demonstrated in Figure 2, it can be found that only approximately 40% of the data are preserved after consensus filtering. For evaluation on PROCESSBENCH, the results reveal that the reduced dataset after consensus filtering significantly outperforms MC estimation, and notably, achieves comparable performance to LLM-as-a-judge while using only 40% of the data. Regarding the BoN evaluation, the performance variations among these three models are marginal.

3.1.4 Hard vs. Soft Labels in MC Estimation

Although we have previously demonstrated that MC estimation is not as effective as LLM-as-a-
Figure 2: Performance comparison on Best-of-8 and PROCESSBENCH using PRMs trained with different data synthesis.

Figure 3: Performance comparison on Best-of-8 for the PRMs trained on soft and hard labels before and after consensus filtering.

Figure 4: Performance comparison on PROCESSBENCH for the PRMs trained on soft and hard labels before and after consensus filtering.

judge and human annotation, there remains a noteworthy point of MC estimation to be discussed, i.e., whether to train with soft label or hard label. We construct 3 million training data using MC estimation and apply the consensus filtering strategy subsequently, which reduces the dataset to 1.5 million samples. We respectively train PRMs using both soft labels and hard labels on 3 million and 1.5 million data.

The performance of trained PRMs on Best-of-8 and PROCESSBENCH are illustrated in Figure 3 and 4 separately. Before data filtering, the performance difference between soft and hard labels is not significant, which we attribute to the high noise level masking their distinctions. However, this difference becomes much more pronounced after data filtering, with hard labels substantially outperforming soft labels on both Best-of-8 and PROCESSBENCH. We consider the limitations of soft labels are: (1) As discussed in Section 3.1.1, the correctness of steps (i.e., rewards) should be deterministic. Training PRMs with soft labels that represent future possibilities introduces additional noise. For instance, when numerous completely correct steps are assigned with soft labels lower than 1, it actually reduces the model’s ability to discriminate between positive and negative labels. (2) Only 8 completions for step correctness estimation exhibit high variance and are relatively crude. Although we can reduce the variance of estimation by increasing the number of completions, the compute costs may outweigh the incremental benefits and it cannot eliminate the generation of false-positive answers in completions of the policy model. Moreover, the experimental results indicate that the consensus filtering strategy yields performance benefits across both soft and hard label schemes.

Last but not least, we investigate the threshold selection for distinguishing between positive and negative labels based on the MC estimation result of 8 completions. Following our previous experimental setup, we conduct a series of experiments on the 3 million with threshold values from 1/8 to 7/8 at 1/8 intervals, with results shown in Figure 5. It can be easily observed that as the threshold increases, the performance deteriorates on both Best-of-8 and PROCESSBENCH, indicating that using an MC estimated value of 0 as the negative label and all others as positive labels yields the best results. In other words, we suggest a step is considered correct if any completion reaches the correct final answer in MC estimation. This threshold has also been employed throughout our all experimental studies.

3.2 Bias in BoN Evaluation

Although BoN evaluations are commonly used in previous PRM optimization, their effectiveness as a sole optimization criterion is worth careful reconsideration due to the potential limitations in performance assessment.

3.2.1 Unreliable Policy Models Cause BoN-PRMs Misalignment

In an ideal scenario, the responses generated by the policy model would exhibit both correct answers and accurate solution steps or conversely, flawed processes would correspond to incorrect answers. However, existing policy models are prone to generating responses with correct answers but flawed processes, while BoN inherently only focuses on answers, leading to a misalignment between the evaluation criteria of BoN and the PRM objectives of process verification. To provide empirical evidence for this phenomenon, we sample 8 responses per query from GSM8K, MATH, OlympiadBench, and Omni-MATH using the pol-
Figure 5: PRM Performance changes on Best-of-8 and PROCESSBENCH across different hard label thresholds.

Figure 6: Proportion of cases where the policy model generates correct answers but incorrect reasoning steps.

Figure 7: Performance trends on BoN and extracted PROCESSBENCH for different PRMs.

icy model Qwen2.5-Math-7B-Instruct. Then we randomly choose correct-answer responses from them and conduct thorough manual annotations. As detailed in Figure 6, a substantial percentage of responses contain process errors while maintaining correct answers. Notably, comparing easy task GSM8K and hard task Omni-MATH, this phenomenon becomes more pronounced as the problem’s complexity increases. This implies that an effective PRM might assign low scores to responses with correct answers but flawed processes, resulting in overall lower performance in the BoN.

3.2.2 Limited Process Verification Capability in PRMs Lead to BoN Scores Inflation

When PRMs cannot distinguish responses that have correct answers but flawed processes and assign them high scores, this leads to overestimated performance in the BoN, thereby creating an overly optimistic and potentially misleading assessment of PRM capabilities. To investigate the discriminative capability of PRMs for such cases, we extract instances from PROCESSBENCH where answers are correct but processes are erroneous and analysis the detection accuracy of PRMs for these cases. As shown in Figure 7, the PRMs trained on MC estimation, LLM-as-a-judge and human annotation exhibit completely opposite performance trends in BoN and extracted PROCESSBENCH evaluation. The PRM trained on our MC estimated data shows limited process verification capability but inflated results on the BoN. This limited discriminative capability indicates that PRMs struggle to differentiate between genuinely correct responses and those with merely superficial answer correctness in BoN evaluations. Consequently, this implies that beyond BoN evaluation, supplementary benchmarks are necessary to assess the actual capability of PRMs, especially in detecting process errors.

Figure 8: Percentage of responses where the minimum step score predict by PRMs appears in the final step (among all best of 8 responses).

3.2.3 Process-to-Outcome Shift in BoN Optimized PRMs

The majority of current PRMs are optimized towards BoN. However, the bias of BoN leads PRMs process-to-outcome shift. During the BoN selection process based on PRM-predicted scores following the scoring method for responses in (Lightman et al., 2023), it can be found that regardless of whether we employ the minimum score or the product of scores to evaluate the full solution, the lowest step score acts as the key limiting factor that affects the selection criteria of PRMs.

As shown in Figure 8, we analyze the distribution of minimum step scores assigned by multiple open-sourced PRMs, specifically focusing on cases where the lowest score occurred at the final step, which typically contains the final answer. The results show that models EurusPRM-Stage1, EurusPRM-Stage2, Math-Shepherd-PRM-7B and Skywork-PRM-7B exhibit notably high proportions in this category, which exceed 40%.

This analysis reveals that some PRMs’ performance in BoN evaluation is predominantly determined by final answer scores rather than intermediate reasoning steps. In other words, optimizing
Figure 9: Performance on BoN across multiple PRMs with 3 scoring methods: minimum, product and last.

solely for the BoN evaluation has made current PRMs perform more like Outcome Reward Models (ORMs) in practice. Hence, it is essential to supplement response-level evaluation BoN with step-level assessment methods to avoid the process-to-outcome shift. In this paper, we employ process error localization tasks PROCESSBENCH.

3.2.4 Different PRMs, Different Optimal Scoring Strategies

In BoN, the overall solution score is derived by combining individual step scores. When each step’s score represents the probability of that specific step being correct, it is generally acceptable to combine these step-level scores (through methods like product or minimum) to calculate the overall solution score. However, in MC estimation, each step’s score actually estimates the probability of reaching the correct final answer in the future from the current position. Given this forward-looking nature of MC estimation, we should neither multiply the estimated probabilities across steps (as these estimates are dependent on each other), nor simply take the minimum estimated value from a particular step as the overall score. Instead, the estimated value from the final step naturally integrates information from the entire solution process, making it more suitable as the final score for the complete solution.

To validate that, we evaluate BoN in different scoring strategies for the PRMs trained on MC estimation, LLM-as-a-judge, and human annotation data, as shown in Figure 9. We found that in MC estimation, using the last score shows significantly better performance than product and minimum approaches across multiple PRMs. And the product and minimum scores are better than the last for human annotation and LLM-as-a-judge.

This suggests that if the PRM has to be trained via MC estimation and evaluated in BoN, the last score strategy may be more reasonable and effective. However, it’s worth noting that this use of PRM in BoN has deviated from PRM’s original intended purpose.

4 Our PRMs

This section presents our methodology for overcoming the previously discussed limitations and the details of our trained PRM achieving state-of-the-art performance.

4.1 Training Details

Based on the lessons learned, we implement a simple yet efficient consensus filtering mechanism by filtering out instances where there is a discrepancy between the LLM-annotated and MC-estimated process labels. This ensures the retained data maintains high quality and consistency in the reasoning process annotation. Specifically, we use MC estimation to construct hard label, where a response is classified as negative only if none of the 8 completions achieves the correct final answer. Then, the LLM instantiated by Qwen2.5-Instruct-72B (Yang et al., 2024b) serves as a critic to verify the reasoning process for all responses step by step, i.e., LLM-as-a-judge. We employ cross-entropy loss on the tokens at the end of each step to train the binary classification task. We train the PRMs with 7B and 72B parameter, initialized with Qwen2.5-Math-7B-Instruct and Qwen2.5-Math-72B-Instruct.

4.2 Experimental Setup

To validate the effectiveness of our trained PRMs, we respectively conduct the response-level BoN evaluation and the step-level process errors identification task PROCESSBENCH (Zheng et al., 2024). We follow the experimental setting in Section 2.2.

Best-of-N  In rm@8, we evaluate Outcome Reward Models (ORMs) and Process Reward Models (PRMs). For ORM, we introduce Qwen2.5-Math-RM-72B (Yang et al., 2024c), which assigns a single score to each complete response. For PRMs, we compute the product of each step score as the final response score.

We compare with the following PRMs: (1) Math-Shepherd-PRM-7B (Wang et al., 2024b): determining process labels for each step by estimating the empirical probability of reaching the correct final answer. (2) RLHFlow-PRM-Mistral-8B & RLHFlow-PRM-Deepseek-8B (Xiong et al., 2024):
<table>
  <tr>
    <th>Setting</th>
    <th>GSM8K</th>
    <th>MATH</th>
    <th>Minerva Math</th>
    <th>GaoKao 2023 En</th>
    <th>Olympiad Bench</th>
    <th>College Math</th>
    <th>MMLU STEM</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>pass@8 (Upper Bound)</td>
    <td>98.1</td>
    <td>92</td>
    <td>49.3</td>
    <td>80.5</td>
    <td>59.6</td>
    <td>52.6</td>
    <td>90.5</td>
    <td>74.7</td>
  </tr>
  <tr>
    <td>maj@8</td>
    <td>96.7</td>
    <td>87.1</td>
    <td>41.2</td>
    <td>72.5</td>
    <td>44.4</td>
    <td>47.8</td>
    <td>73.8</td>
    <td>66.2</td>
  </tr>
  <tr>
    <th colspan="9">1.5B</th>
  </tr>
  <tr>
    <td>Skywork-PRM-1.5B</td>
    <td>96.9</td>
    <td>86.7</td>
    <td>37.9</td>
    <td>70.1</td>
    <td>42.1</td>
    <td>47.9</td>
    <td>67.9</td>
    <td>64.2</td>
  </tr>
  <tr>
    <th colspan="9">7B+</th>
  </tr>
  <tr>
    <td>Math-Shepherd-PRM-7B</td>
    <td><b>97.3</b></td>
    <td>85.4</td>
    <td>37.9</td>
    <td>70.6</td>
    <td>40.4</td>
    <td>47.2</td>
    <td>70.5</td>
    <td>64.2</td>
  </tr>
  <tr>
    <td>RLHFFlow-PRM-Mistral-8B</td>
    <td>97.0</td>
    <td>86.1</td>
    <td>37.1</td>
    <td>70.6</td>
    <td>41.2</td>
    <td>47.6</td>
    <td>69.5</td>
    <td>64.2</td>
  </tr>
  <tr>
    <td>RLHFFlow-PRM-Deepseek-8B</td>
    <td><b>97.3</b></td>
    <td>86.3</td>
    <td>40.8</td>
    <td>70.9</td>
    <td>42.2</td>
    <td>47.2</td>
    <td>69.3</td>
    <td>64.9</td>
  </tr>
  <tr>
    <td>Skywork-PRM-7B</td>
    <td><b>97.3</b></td>
    <td>87.3</td>
    <td>38.2</td>
    <td>71.9</td>
    <td>43.7</td>
    <td>47.8</td>
    <td>67.7</td>
    <td>64.8</td>
  </tr>
  <tr>
    <td>EurusPRM-Stage1</td>
    <td>95.6</td>
    <td>83.0</td>
    <td>35.7</td>
    <td>66.2</td>
    <td>38.2</td>
    <td>46.2</td>
    <td>66.6</td>
    <td>61.6</td>
  </tr>
  <tr>
    <td>EurusPRM-Stage2</td>
    <td>95.4</td>
    <td>83.4</td>
    <td>34.9</td>
    <td>67.3</td>
    <td>39.1</td>
    <td>46.3</td>
    <td>67.3</td>
    <td>62.0</td>
  </tr>
  <tr>
    <td>Qwen2.5-Math-7B-Math-Shepherd</td>
    <td>96.9</td>
    <td>86.5</td>
    <td>36.8</td>
    <td>71.4</td>
    <td>41.6</td>
    <td>47.7</td>
    <td>69.3</td>
    <td>64.3</td>
  </tr>
  <tr>
    <td>Qwen2.5-Math-7B-PRM800K</td>
    <td>96.9</td>
    <td>86.9</td>
    <td>37.1</td>
    <td>71.2</td>
    <td>44.0</td>
    <td>47.6</td>
    <td>70.9</td>
    <td>64.9</td>
  </tr>
  <tr>
    <td>★ Qwen2.5-Math-PRM-7B</td>
    <td>97.1</td>
    <td><b>88.0</b></td>
    <td><b>42.6</b></td>
    <td><b>74.5</b></td>
    <td><b>47.6</b></td>
    <td><b>48.7</b></td>
    <td><b>74.5</b></td>
    <td><b>67.6</b></td>
  </tr>
  <tr>
    <th colspan="9">72B</th>
  </tr>
  <tr>
    <td>Qwen2.5-Math-RM-72B</td>
    <td><b>97.9</b></td>
    <td>88.5</td>
    <td>42.6</td>
    <td><b>75.1</b></td>
    <td><b>49.9</b></td>
    <td><b>49.6</b></td>
    <td>78.7</td>
    <td>68.9</td>
  </tr>
  <tr>
    <td>★ Qwen2.5-Math-PRM-72B</td>
    <td>97.6</td>
    <td><b>88.7</b></td>
    <td><b>46.0</b></td>
    <td>74.3</td>
    <td>48.1</td>
    <td>49.3</td>
    <td><b>81.1</b></td>
    <td><b>69.3</b></td>
  </tr>
</table>

Table 3: Performance comparison on the Best-of-8 strategy of the policy model Qwen2.5-Math-7B-Instruct. ★ represents the models we trained.

two LLaMA-3.1-based PRMs that adopt Math-Shepherd’s training methodology while implementing different solution generation models and optimization objectives. (3) Skywork-PRM-1.5B & Skywork-PRM-7B (Skywork, 2024): two recently released Qwen2.5-Math-based PRMs by Skywork. (4) EurusPRM-Stage1 & EurusPRM-Stage2 (Cui et al., 2025): two PRMs trained using Implicit PRM approach (Yuan et al., 2024) with 7B parameters, which obtains process rewards replying on the ORM trained on the response-level labels. (5) Qwen2.5-Math-7B-Math-Shepherd & Qwen2.5-Math-7B-PRM800K: two additional PRMs our developed by fine-tuning Qwen2.5-Math-7B-Instruct separately on the PRM800K (Lightman et al., 2023) and Math-Shepherd (Wang et al., 2024b) open-source datasets.

ProcessBench The compared PRMs are consistent with the previously mentioned PRMs. For LLM-as-a-judge, we compare with proprietary language models GPT-4o-0806 (Hurst et al., 2024) and o1-mini (OpenAI, 2024), open-source language models Llama-3.3-70B-Instruct (Dubey et al., 2024), Qwen2.5-Math-72B-Instruct (Yang et al., 2024c), Qwen2.5-72B-Instruct (Yang et al., 2024b) and QwQ-32B-Preview (Qwen, 2024). We also decompose the N-step response trajectory into N separate instances to enable individual scoring by the ORM Qwen2.5-Math-RM-72B.

4.3 Experimental Results

Best-of-N The evaluation on policy model Qwen2.5-Math-7b-Instruct is shown in Table 3. Qwen2.5-Math-PRM-7B demonstrates superior performance compared to other PRMs of equivalent model scale. Notably, it outperforms maj@8 across all 7 tasks, achieving an average improvement of 1.4%. Furthermore, Qwen2.5-Math-PRM-72B exhibits slightly better overall performance than Qwen2.5-Math-RM-72B, with particularly significant improvements observed in the Minerva Math and MMLU STEM tasks. Supplementary BoN results, including performance on Policy model Qwen2.5-Math-72b-Instruct, alternative scoring strategies, evaluations on Chinese benchmarks, BoN with larger N values and BoN with LLM-as-a-judge are comprehensively documented in the Appendix D.

ProcessBench The evaluation results on ProcessBench are presented in Table 4. When compared with LLM-as-a-judge, Qwen2.5-Math-PRM-7B in smaller model size demonstrates superior performance over all open-source models. For proprietary language models, Qwen2.5-Math-PRM-7B outperforms GPT-4o-0806, while there remains a performance gap compared to o1-mini. Furthermore, in comparison with existing PRMs, both Qwen2.5-Math-PRM-7B and 72B exhibit substantial advantages over their counterparts. An inter-
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">GSM8K</th>
    <th colspan="4">MATH</th>
    <th colspan="4">OlympiadBench</th>
    <th colspan="4">Omni-MATH</th>
    <th rowspan="2">Avg. F1</th>
  </tr>
  <tr>
    <th>error</th><th>correct</th><th>F1</th><th>error</th><th>correct</th><th>F1</th><th>error</th><th>correct</th><th>F1</th><th>error</th><th>correct</th><th>F1</th><th>error</th><th>correct</th><th>F1</th>
  </tr>
  <tr>
    <th colspan="19"><i>LLM-as-a-judge, Proprietary language models</i></th>
  </tr>
  <tr>
    <td>GPT-4-0806</td><td>70.0</td><td>91.2</td><td>79.2</td><td>54.4</td><td>76.6</td><td>63.6</td><td>45.8</td><td>58.4</td><td>51.4</td><td>45.2</td><td>65.6</td><td>53.5</td><td>61.9</td>
  </tr>
  <tr>
    <td>ol-mini</td><td>88.9</td><td>97.9</td><td>93.2</td><td>83.5</td><td>95.1</td><td>88.9</td><td>80.2</td><td>95.6</td><td>87.2</td><td>74.8</td><td>91.7</td><td>82.4</td><td>87.9</td>
  </tr>
  <tr>
    <th colspan="19"><i>LLM-as-a-judge, Open-source language models</i></th>
  </tr>
  <tr>
    <td>Llama-3.3-70B-Instruct</td><td>72.5</td><td>96.9</td><td>82.9</td><td>43.3</td><td>83.2</td><td>59.4</td><td>31.0</td><td>94.1</td><td>46.7</td><td>28.2</td><td>90.5</td><td>43.0</td><td>58.0</td>
  </tr>
  <tr>
    <td>Qwen2.5-Math-72B-Instruct</td><td>49.8</td><td>96.9</td><td>65.8</td><td>36.0</td><td>94.3</td><td>52.1</td><td>19.5</td><td>97.3</td><td>32.5</td><td>19.0</td><td>96.3</td><td>31.7</td><td>45.5</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Instruct</td><td>62.8</td><td>96.9</td><td>76.2</td><td>46.3</td><td>93.1</td><td>61.8</td><td>38.7</td><td>92.6</td><td>54.6</td><td>36.6</td><td>90.9</td><td>52.2</td><td>61.2</td>
  </tr>
  <tr>
    <td>QwQ-32B-Preview</td><td>81.6</td><td>95.3</td><td>88.0</td><td>78.1</td><td>79.3</td><td>78.7</td><td>61.4</td><td>54.6</td><td>57.8</td><td>55.7</td><td>68.0</td><td>61.3</td><td>71.5</td>
  </tr>
  <tr>
    <th colspan="19"><i>PRMs</i></th>
  </tr>
  <tr>
    <th colspan="19">1.5B</th>
  </tr>
  <tr>
    <td>Skywork-PRM-1.5B</td><td>50.2</td><td>71.5</td><td>59.0</td><td>37.9</td><td>65.2</td><td>48.0</td><td>15.4</td><td>26.0</td><td>19.3</td><td>13.6</td><td>32.8</td><td>19.2</td><td>36.4</td>
  </tr>
  <tr>
    <th colspan="19">7B+</th>
  </tr>
  <tr>
    <td>Math-Shepherd-PRM-7B</td><td>32.4</td><td>91.7</td><td>47.9</td><td>18.0</td><td>82.0</td><td>29.5</td><td>15.0</td><td>71.1</td><td>24.8</td><td>14.2</td><td>73.0</td><td>23.8</td><td>31.5</td>
  </tr>
  <tr>
    <td>RLLHFlow-PRM-Mistral-8B</td><td>33.8</td><td>99.0</td><td>50.4</td><td>21.7</td><td>72.2</td><td>33.4</td><td>8.2</td><td>43.1</td><td>13.8</td><td>9.6</td><td>45.2</td><td>15.8</td><td>28.4</td>
  </tr>
  <tr>
    <td>RLLHFlow-PRM-Deepseek-8B</td><td>24.2</td><td>98.4</td><td>38.8</td><td>21.4</td><td>80.0</td><td>33.8</td><td>10.1</td><td>51.0</td><td>16.9</td><td>10.9</td><td>51.9</td><td>16.9</td><td>26.6</td>
  </tr>
  <tr>
    <td>Skywork-PRM-7B</td><td>61.8</td><td>82.9</td><td>70.8</td><td>43.8</td><td>62.2</td><td>53.6</td><td>17.9</td><td>31.9</td><td>22.9</td><td>14.0</td><td>41.9</td><td>21.0</td><td>42.1</td>
  </tr>
  <tr>
    <td>EurusPRM-Stage1</td><td>46.9</td><td>42.0</td><td>44.3</td><td>33.3</td><td>38.2</td><td>35.6</td><td>23.9</td><td>19.8</td><td>21.7</td><td>21.9</td><td>24.5</td><td>23.1</td><td>31.2</td>
  </tr>
  <tr>
    <td>EurusPRM-Stage2</td><td>51.2</td><td>44.0</td><td>47.3</td><td>36.4</td><td>35.0</td><td>35.7</td><td>25.7</td><td>18.0</td><td>21.2</td><td>23.1</td><td>19.1</td><td>20.9</td><td>31.3</td>
  </tr>
  <tr>
    <td>Qwen2.5-Math-7B-Math-Shepherd</td><td>46.4</td><td>95.9</td><td>62.5</td><td>18.9</td><td>96.6</td><td>31.6</td><td>7.4</td><td>93.8</td><td>13.7</td><td>4.0</td><td>95.0</td><td>7.7</td><td>28.9</td>
  </tr>
  <tr>
    <td>Qwen2.5-Math-7B-PRM800K</td><td>53.1</td><td>95.3</td><td>68.2</td><td>48.0</td><td>90.1</td><td>62.6</td><td>35.7</td><td>87.3</td><td>50.7</td><td>29.8</td><td>86.1</td><td>44.3</td><td>56.5</td>
  </tr>
  <tr>
    <td>★ Qwen2.5-Math-PRM-7B</td><td>72.0</td><td>96.4</td><td>82.4</td><td>68.0</td><td>90.4</td><td>77.6</td><td>55.7</td><td>85.5</td><td>67.5</td><td>55.2</td><td>83.0</td><td>66.3</td><td>73.5</td>
  </tr>
  <tr>
    <th colspan="19">72B</th>
  </tr>
  <tr>
    <td>Qwen2.5-Math-RM-72B</td><td>41.1</td><td>46.1</td><td>43.5</td><td>39.7</td><td>58.1</td><td>47.2</td><td>28.1</td><td>56.6</td><td>37.6</td><td>18.8</td><td>50.2</td><td>27.4</td><td>38.9</td>
  </tr>
  <tr>
    <td>★ Qwen2.5-Math-PRM-72B</td><td>78.7</td><td>97.9</td><td>87.3</td><td>74.2</td><td>88.2</td><td>80.6</td><td>67.9</td><td>82.0</td><td>74.3</td><td>64.8</td><td>78.8</td><td>71.1</td><td>78.3</td>
  </tr>
</table>

Table 4: Performance comparison on PROCESSBENCH. ★ represents the models we trained. We report the results in the same calculation method with PROCESSBENCH.

esting observation worth noting is that the ORM Qwen2.5-Math-RM-72B exhibits considerable capability in identifying step errors, even surpassing some open-source PRMs.

5 Related Work

Reward Model in Mathematical Reasoning
Mathematical reasoning reward models primarily fall into two categories: Outcome Reward Models (ORMs) that evaluate final answers, and Process Reward Models (PRMs) (Uesato et al., 2022; Lightman et al., 2023) that assess individual reasoning steps. Though PRMs show greater potential than ORMs (Lightman et al., 2023; Wang et al., 2024b), they rely on high-quality training data.

Mathematical Reasoning Step Verification
Step verification methods usually include human annotation (Lightman et al., 2023) and automated approaches. Automated methods comprise: (1) backward-propagation based methods that infer step correctness from solution outcomes, including MC estimation (Wang et al., 2024b; Luo et al., 2024; Chen et al., 2024), progressive ORM labeling (Xi et al., 2024), credit assignment (Wang et al., 2024a; Cui et al., 2025; Yuan et al., 2024) techniques and so on; (2) prompting-based methods that leverage LLMs serve as critic, i.e., LLM-as-a-judge (Zhang et al., 2024; Gao et al., 2024; Xia et al., 2024) to assess step correctness directly. In this work, we integrate both MC estimation and LLM-as-a-judge methods.

6 Conclusion

In this paper, we present the critical lessons gained during developing PRMs and release a new state-of-the-art PRM. Firstly, we identify critical limitations in current data construction approaches for PRMs, demonstrating that MC estimation-based data construction yields inferior performance and generalization compared to LLM-as-a-judge and human annotation. Then we reveal the potential bias in using response-level BoN evaluation alone for PRMs and advocate for combining both response-level and step-level metrics. To address these issues, we propose an effective consensus filtering strategy combining MC estimation with LLM-as-a-judge. Our evaluation, incorporating both response-level BoN and identifying step-wise correctness task PROCESSBENCH, demonstrates significant improvements in data efficiency and model performance.
Limitation There are several limitations remained in our current work. Firstly, there exists a considerable performance gap between our PRMs and the BoN upper bound (pass@8), suggesting substantial optimization potential. Then the best practices for utilizing PRMs in reinforcement learning remain unexplored. Finally, although our approach combines LLM-as-a-judge with MC estimation for consensus filtering, the efficient utilization of existing high-quality human annotation data is still largely under-explored. For instance, gradually expanding high-quality datasets through weakly supervised methods can be investigated as a promising direction for future exploration.

References

Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan. 2024. Alphamath almost zero: Process supervision without process. Preprint, arXiv:2405.03553.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Hee woo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.

Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, Jiarui Yuan, Huayu Chen, Kaiyan Zhang, Xingtai Lv, Shuo Wang, Yuan Yao, Hao Peng, Yu Cheng, Zhiyuan Liu, Maosong Sun, Bowen Zhou, and Ning Ding. 2025. Process reinforcement through implicit rewards.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Bofei Gao, Zefan Cai, Runxin Xu, Peiyi Wang, Ce Zheng, Rurji Lin, Keming Lu, Dayiheng Liu, Chang Zhou, Wen Xiao, Junjie Hu, Tianyu Liu, and Baobao Chang. 2024. Lim critics help catch bugs in mathematics: Towards a better mathematical verifier with natural language feedback. Preprint, arXiv:2406.14024.

Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. 2024. Olympiad-bench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402.14008.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021a. Measuring massive multitask language understanding. In ICLR. OpenReview.net.

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021b. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874.

Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.

Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay V. Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. 2022. Solving quantitative reasoning problems with language models. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022.

Minpeng Liao, Chengxi Li, Wei Luo, Jing Wu, and Kai Fan. 2024. MARIO: math reasoning with code interpreter output - A reproducible pipeline. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pages 905–924. Association for Computational Linguistics.

Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let's verify step by step. arXiv preprint arXiv:2305.20050.

Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Meiqi Guo, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, and Abhinav Rastogi. 2024. Improve mathematical reasoning in language models by automated process supervision. Preprint, arXiv:2406.06592.

OpenAI. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.

OpenAI. 2024. Openai o1-mini: Advancing cost-efficient reasoning.

Team Qwen. 2024. Qwq: Reflect deeply on the boundaries of the unknown.

Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.

o1 Team Skywork. 2024. Skywork-o1 open series. https://huggingface.co/Skywork.

Zhengyang Tang, Xingxing Zhang, Benyou Wang, and Furu Wei. 2024. Mathscale: Scaling instruction tuning for mathematical reasoning. In Forty-first International Conference on Machine Learning, ICML
2024, Vienna, Austria, July 21-27, 2024. OpenReview.net.

Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. 2022. Solving math word problems with process- and outcome-based feedback. Preprint, arXiv:2211.14275.

Chaojie Wang, Yanchen Deng, Zhiyi Lyu, Liang Zeng, Jujie He, Shuicheng Yan, and Bo An. 2024a. Q*: Improving multi-step reasoning for llms with deliberative planning. Preprint, arXiv:2406.14283.

Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. 2024b. Math-shepherd: Verify and reinforce LLMs step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9426–9439.

Tianwen Wei, Jian Luan, Wei Liu, Shuang Dong, and Bin Wang. 2023. CMATH: can your language model pass chinese elementary school math test? CoRR, abs/2306.16636.

Zhiheng Xi, Wenxiang Chen, Boyang Hong, Senjie Jin, Rui Zheng, Wei He, Yiwen Ding, Shichun Liu, Xin Guo, Junzhe Wang, Honglin Guo, Wei Shen, Xiaoran Fan, Yuhao Zhou, Shihan Dou, Xiao Wang, Xinho Zhang, Peng Sun, Tao Gui, Qi Zhang, and Xuanjing Huang. 2024. Training large language models for reasoning through reverse curriculum reinforcement learning. Preprint, arXiv:2402.05808.

Shijie Xia, Xuefeng Li, Yixin Liu, Tongshuang Wu, and Pengfei Liu. 2024. Evaluating mathematical reasoning beyond accuracy. Preprint, arXiv:2404.05692.

Wei Xiong, Hanning Zhang, Nan Jiang, and Tong Zhang. 2024. An implementation of generative prm. https://github.com/RLHF1ow/RLHF-Reward-Modeling.

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024a. Qwen2 technical report. arXiv preprint arXiv:2407.10671.

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024b. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115.

An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. 2024c. Qwen2.5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122.

Lifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kaiyan Zhang, Bowen Zhou, Zhiyuan Liu, and Hao Peng. 2024. Free process rewards without process labels. arXiv preprint arXiv:2412.01981.

Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. 2024. Generative verifiers: Reward modeling as next-token prediction. Preprint, arXiv:2408.15240.

Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. 2024. Processbench: Identifying process errors in mathematical reasoning. arXiv preprint arXiv:2412.06559.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems, volume 36, pages 46595–46623.

Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. 2024. Agieval: A human-centric benchmark for evaluating foundation models. In NAACL-HLT (Findings), pages 2299–2314. Association for Computational Linguistics.

Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shiron Ma, et al. 2024. Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence. arXiv preprint arXiv:2406.11931.
A  Process Verification Capability of Existing PRMs

The policy model may generate the responses that have correct answers but flawed processes. To investigate the discriminative capability of PRMs for such cases, we extract instances from PROCESSBENCH where answers are correct but processes are erroneous and analysis the detection accuracy rates of PRMs for these cases. As shown in Table 5, except Qwen2.5-Math-PRM-7B and 72B, all other open-sourced PRMs demonstrate detection accuracy rates below 50%.

B  PRM Guided Search

We further integrate PRM with greedy search by generating N candidate steps at each step, evaluating these candidates using PRM scoring, and selecting the highest-scoring step for subsequent expansion. For the policy model, we employed Qwen2.5-7B-Instruct which has greater diversity in generation to sample 8 candidates at each step, with sampling parameters set to \( temperature = 1.0 \) and \( top_p = 1.0 \). We conduct comparative experiments with ORM in BoN approach. As shown in Table 6, Qwen2.5-Math-PRM-72B with greedy search@8 is slightly superior performance compared to Qwen2.5-Math-RM-72B with orm@8. We argue the potentially smaller performance differential between PRM and ORM lies in the consistency of generated token counts between greedy search and BoN outputs. Furthermore, although greedy search always selects the highest-scoring candidate at each step, the highest-scoring step may not be the correct one. Therefore, implementing either Depth-First Search (DFS) with backtracking capabilities or search approaches incorporating score constraints could prove more suitable for this case.

We choose the highest-scoring candidate at each step which the score predicted by PRM represents the correctness of this step. But such locally optimal choices may not lead to the correct final answer. In contrast, value models can predict the future probability of reaching the correct answer, rather than reflecting the correctness of the current step like rewards do, making them particularly well-suited for integration with search strategies. Based on these considerations, we believe there is still significant potential for exploration in the future regarding more appropriate search strategies or combining rewards and values to simultaneously consider both the correctness of the current step and the possibility of reaching the correct future outcomes.

C  Detailed Experimental Results

C.1  Preliminary Trials

Table 7 and Table 8 demonstrate detailed experimental results on Best-of-8 evaluation and PROCESSBENCH.

C.2  MC Estimation vs. LLM-as-a-judge vs. Human Annotation

The PRMs trained on the different data construction methods including MC estimation, LLM-as-a-judge, and human annotation are evaluated on Best-of-8 and PROCESSBENCH. The detailed experimental results are shown in Table 9 and 10.

D  Supplementary BoN Results

D.1  Best-of-8 on Qwen2.5-Math-72b-Instruct

The BoN evaluation on policy model Qwen2.5-Math-72b-Instruct is shown in Table 11. Qwen2.5-Math-PRM-7B outperforms other PRMs of equivalent model scale. However, its performance is inferior to maj@8, suggesting challenges in employing a 7B PRM for the supervision of 72B policy model-generated responses. Besides, Qwen2.5-Math-PRM-72B surpasses maj@8 in prm@8 and is comparable with Qwen2.5-Math-RM-72B in orm@8.

D.2  Best-of-8 with Various Scoring Strategies

We demonstrate experimental results using the last step score, the minimum step score or the production of step scores as the solution-level score. The BoN results with policy model Qwen2.5-Math-7B-Instruct and Qwen2.5-Math-72B-Instruct are shown in Table 13 and Table 14 respectively.

D.3  Best-of-8 on Chinese Benchmarks

We evaluate across three Chinese benchmarks including Chinese math benchmarks CMATH (Wei et al., 2023), GaoKao Math Cloze (Zhong et al., 2024), and GaoKao Math QA (Zhong et al., 2024) following (Yang et al., 2024c), as shown in Table 15 and Table 16.

D.4  BoN with Larger N Values

To validate the effectiveness of our PRMs on the BoN with larger N values, we conduct additional Best-of-8 experiments on the policy model Qwen2.5-Math-7b-Instruct across diverse tasks
<table>
  <tr>
    <th rowspan="2"># samples</th>
    <th colspan="2">GSM8K</th>
    <th colspan="2">MATH</th>
    <th colspan="2">Olympiad Bench</th>
    <th colspan="2">Omni-MATH</th>
    <th rowspan="2">Avg.</th>
  </tr>
  <tr>
    <th>7</th>
    <th>94</th>
    <th>161</th>
    <th>259</th>
  </tr>
  <tr>
    <th colspan="10"><b>1.5B</b></th>
  </tr>
  <tr>
    <td>Skywork-PRM-1.5B</td>
    <td>42.9</td>
    <td>36.2</td>
    <td>12.4</td>
    <td>13.9</td>
    <td>26.4</td>
  </tr>
  <tr>
    <th colspan="10"><b>7B+</b></th>
  </tr>
  <tr>
    <td>Math-Shepherd-PRM-7B</td>
    <td>14.3</td>
    <td>12.8</td>
    <td>13.7</td>
    <td>14.7</td>
    <td>13.9</td>
  </tr>
  <tr>
    <td>RLHFFlow-PRM-Mistral-8B</td>
    <td>14.3</td>
    <td>13.8</td>
    <td>7.5</td>
    <td>10.0</td>
    <td>11.4</td>
  </tr>
  <tr>
    <td>RLHFFlow-PRM-Deepseek-8B</td>
    <td>0.0</td>
    <td>18.1</td>
    <td>9.9</td>
    <td>10.8</td>
    <td>9.7</td>
  </tr>
  <tr>
    <td>Skywork-PRM-7B</td>
    <td><b>57.1</b></td>
    <td>26.6</td>
    <td>14.3</td>
    <td>13.1</td>
    <td>27.8</td>
  </tr>
  <tr>
    <td>EurusPRM-Stage1</td>
    <td>28.6</td>
    <td>25.5</td>
    <td>19.9</td>
    <td>20.1</td>
    <td>23.5</td>
  </tr>
  <tr>
    <td>EurusPRM-Stage2</td>
    <td>42.9</td>
    <td>27.7</td>
    <td>18.0</td>
    <td>20.8</td>
    <td>27.4</td>
  </tr>
  <tr>
    <td>Qwen2.5-Math-7B-Math-Shepherd</td>
    <td>0.0</td>
    <td>9.6</td>
    <td>4.3</td>
    <td>1.2</td>
    <td>3.8</td>
  </tr>
  <tr>
    <td>Qwen2.5-Math-7B-PRM800K</td>
    <td>42.9</td>
    <td>50.0</td>
    <td>31.7</td>
    <td>28.2</td>
    <td>38.2</td>
  </tr>
  <tr>
    <td>★ Qwen2.5-Math-PRM-7B</td>
    <td>42.9</td>
    <td><b>68.1</b></td>
    <td><b>48.4</b></td>
    <td><b>56.0</b></td>
    <td><b>53.9</b></td>
  </tr>
  <tr>
    <th colspan="10"><b>72B</b></th>
  </tr>
  <tr>
    <td>★ Qwen2.5-Math-PRM-72B</td>
    <td>28.6</td>
    <td>76.6</td>
    <td>62.7</td>
    <td>64.5</td>
    <td>58.1</td>
  </tr>
</table>

Table 5: The accuracy in identifying erroneous steps on the test cases of PROCESSBENCH containing correct answers but erroneous reasoning steps. “# samples” represents the number of test cases.

<table>
  <tr>
    <th>Setting</th>
    <th>GSM8K</th>
    <th>MATH</th>
    <th>Minerva Math</th>
    <th>GaoKao 2023 En</th>
    <th>Olympiad Bench</th>
    <th>College Math</th>
    <th>MMLU STEM</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>pass@8 (Upper Bound)</td>
    <td>96.9</td>
    <td>89.6</td>
    <td>48.2</td>
    <td>79.7</td>
    <td>58.4</td>
    <td>55.0</td>
    <td>81.6</td>
    <td>72.8</td>
  </tr>
  <tr>
    <td>pass@1</td>
    <td>91.2</td>
    <td>74.0</td>
    <td>32.0</td>
    <td>64.7</td>
    <td>36.9</td>
    <td>46.2</td>
    <td>57.1</td>
    <td>57.4</td>
  </tr>
  <tr>
    <td>maj@8</td>
    <td>93.7</td>
    <td>80.3</td>
    <td>37.1</td>
    <td>69.9</td>
    <td>45.8</td>
    <td>48.5</td>
    <td>61.9</td>
    <td>62.5</td>
  </tr>
  <tr>
    <th colspan="9">orm@8</th>
  </tr>
  <tr>
    <td>Qwen2.5-Math-RM-72B</td>
    <td>95.4</td>
    <td>84.2</td>
    <td><b>38.6</b></td>
    <td>73.0</td>
    <td>48.6</td>
    <td><b>50.1</b></td>
    <td><b>75.6</b></td>
    <td>66.5</td>
  </tr>
  <tr>
    <th colspan="9">Greedy Search@8</th>
  </tr>
  <tr>
    <td>Skywork-PRM-7B</td>
    <td>95.3</td>
    <td>83.2</td>
    <td>33.8</td>
    <td>70.4</td>
    <td>44.1</td>
    <td>48.2</td>
    <td>60.1</td>
    <td>62.2</td>
  </tr>
  <tr>
    <td>★ Qwen2.5-Math-PRM-7B</td>
    <td>95.5</td>
    <td>82.6</td>
    <td>32.0</td>
    <td>71.4</td>
    <td>44.9</td>
    <td>48.8</td>
    <td>69.6</td>
    <td>63.5</td>
  </tr>
  <tr>
    <td>★ Qwen2.5-Math-PRM-72B</td>
    <td><b>95.9</b></td>
    <td><b>84.7</b></td>
    <td>37.9</td>
    <td><b>73.2</b></td>
    <td><b>48.9</b></td>
    <td>50.0</td>
    <td>75.3</td>
    <td><b>66.6</b></td>
  </tr>
</table>

Table 6: The performance of PRM guided greedy search and ORM of Best-of-8 with policy model Qwen2.5-7B-Instruct. For greedy search, 8 candidates is proposed at each step.

<table>
  <tr>
    <th>Setting</th>
    <th>GSM8K</th>
    <th>MATH</th>
    <th>Minerva Math</th>
    <th>GaoKao 2023 En</th>
    <th>Olympiad Bench</th>
    <th>College Math</th>
    <th>MMLU STEM</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>pass@8 (Upper Bound)</td>
    <td>98.1</td>
    <td>92.0</td>
    <td>49.3</td>
    <td>80.5</td>
    <td>59.6</td>
    <td>52.6</td>
    <td>90.5</td>
    <td>74.7</td>
  </tr>
  <tr>
    <td>maj@8</td>
    <td>96.7</td>
    <td>87.1</td>
    <td><b>41.2</b></td>
    <td><b>72.5</b></td>
    <td><b>44.4</b></td>
    <td>47.8</td>
    <td><b>73.8</b></td>
    <td><b>66.2</b></td>
  </tr>
  <tr>
    <td>PRM800K</td>
    <td><b>96.9</b></td>
    <td>86.9</td>
    <td>37.1</td>
    <td>71.2</td>
    <td>44.0</td>
    <td>47.6</td>
    <td>70.9</td>
    <td>64.9</td>
  </tr>
  <tr>
    <td>MC estimated hard labels</td>
    <td>96.8</td>
    <td><b>87.3</b></td>
    <td>40.1</td>
    <td>70.6</td>
    <td>43.7</td>
    <td><b>48.1</b></td>
    <td>71.6</td>
    <td>65.5</td>
  </tr>
  <tr>
    <td>MC estimated soft labels</td>
    <td>96.8</td>
    <td>86.3</td>
    <td>37.9</td>
    <td>70.6</td>
    <td>41.0</td>
    <td>47.7</td>
    <td>70.4</td>
    <td>64.4</td>
  </tr>
</table>

Table 7: Preliminary trials results on Best-of-8 using PRMs trained with MC estimated hard labels and soft labels, human-annotated PRM800K respectively.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">GSM8K</th>
    <th colspan="3">MATH</th>
    <th colspan="3">OlympiadBench</th>
    <th colspan="3">Omni-MATH</th>
    <th rowspan="2">Avg. F1</th>
  </tr>
  <tr>
    <th>error</th>
    <th>correct</th>
    <th>F1</th>
    <th>error</th>
    <th>correct</th>
    <th>F1</th>
    <th>error</th>
    <th>correct</th>
    <th>F1</th>
    <th>error</th>
    <th>correct</th>
    <th>F1</th>
  </tr>
  <tr>
    <td>PRM800K</td>
    <td>53.1</td>
    <td>95.3</td>
    <td>68.2</td>
    <td>48.0</td>
    <td>90.1</td>
    <td><b>62.6</b></td>
    <td>35.7</td>
    <td>87.3</td>
    <td><b>50.7</b></td>
    <td>29.8</td>
    <td>86.1</td>
    <td><b>44.3</b></td>
    <td><b>56.5</b></td>
  </tr>
  <tr>
    <td>MC estimated hard labels</td>
    <td>67.1</td>
    <td>90.2</td>
    <td>77.0</td>
    <td>35.2</td>
    <td>65.8</td>
    <td>45.8</td>
    <td>13.2</td>
    <td>28.0</td>
    <td>17.9</td>
    <td>13.3</td>
    <td>41.9</td>
    <td>20.2</td>
    <td>40.2</td>
  </tr>
  <tr>
    <td>MC estimated soft labels</td>
    <td>65.7</td>
    <td>93.3</td>
    <td><b>77.1</b></td>
    <td>35.7</td>
    <td>64.5</td>
    <td>46.0</td>
    <td>13.2</td>
    <td>29.2</td>
    <td>18.1</td>
    <td>12.9</td>
    <td>40.2</td>
    <td>19.6</td>
    <td>40.2</td>
  </tr>
</table>

Table 8: Preliminary trials results on PROCESSBENCH using PRMs trained with MC estimated hard labels and soft labels, human-annotated PRM800K respectively.
<table>
  <tr>
    <th>Setting</th>
    <th># samples</th>
    <th>GSM8K</th>
    <th>MATH</th>
    <th>Minerva Math</th>
    <th>GaoKao 2023 En</th>
    <th>Olympiad Bench</th>
    <th>College Math</th>
    <th>MMLU STEM</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>MC Estimation (Math-Shepherd)</td>
    <td>440k</td>
    <td>96.9</td>
    <td>86.5</td>
    <td>36.8</td>
    <td><b>71.4</b></td>
    <td>41.6</td>
    <td>47.7</td>
    <td>69.3</td>
    <td>64.3</td>
  </tr>
  <tr>
    <td>MC Estimation (our data)</td>
    <td>860k</td>
    <td><b>97.0</b></td>
    <td><b>87.6</b></td>
    <td><b>41.9</b></td>
    <td><b>71.4</b></td>
    <td>43.6</td>
    <td><b>48.2</b></td>
    <td><b>71.9</b></td>
    <td><b>65.9</b></td>
  </tr>
  <tr>
    <td>LLM-as-a-judge (our data)</td>
    <td>860k</td>
    <td>96.9</td>
    <td>86.8</td>
    <td>39.0</td>
    <td>71.2</td>
    <td>43.7</td>
    <td>47.7</td>
    <td><b>71.9</b></td>
    <td>65.3</td>
  </tr>
  <tr>
    <td>Human Annotation (PRM800K)</td>
    <td>264k</td>
    <td>96.9</td>
    <td>86.9</td>
    <td>37.1</td>
    <td>71.2</td>
    <td><b>44.0</b></td>
    <td>47.6</td>
    <td>70.9</td>
    <td>64.9</td>
  </tr>
</table>

Table 9: PRMs performance comparison on the Best-of-8 strategy of the policy model Qwen2.5-Math-7B-Instruct. The models are trained on the different data construction methods including MC estimation, LLM-as-a-judge, and human annotation.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="3">GSM8K</th>
    <th colspan="3">MATH</th>
    <th colspan="3">OlympiadBench</th>
    <th colspan="3">Omni-MATH</th>
    <th rowspan="2">Avg.F1</th>
  </tr>
  <tr>
    <th># samples</th>
    <th>error</th>
    <th>correct</th>
    <th>F1</th>
    <th>error</th>
    <th>correct</th>
    <th>F1</th>
    <th>error</th>
    <th>correct</th>
    <th>F1</th>
    <th>error</th>
    <th>correct</th>
    <th>F1</th>
  </tr>
  <tr>
    <td>MC Estimation (Math-Shepherd)</td>
    <td>440k</td>
    <td>46.4</td>
    <td>95.9</td>
    <td>62.5</td>
    <td>18.9</td>
    <td>96.6</td>
    <td>31.6</td>
    <td>7.4</td>
    <td>93.8</td>
    <td>13.7</td>
    <td>4.0</td>
    <td>95.0</td>
    <td>7.7</td>
    <td>28.9</td>
  </tr>
  <tr>
    <td>MC Estimation (our data)</td>
    <td>860k</td>
    <td>62.3</td>
    <td>91.2</td>
    <td><b>74.0</b></td>
    <td>35.2</td>
    <td>71.9</td>
    <td>47.3</td>
    <td>12.7</td>
    <td>41.3</td>
    <td>19.4</td>
    <td>12.1</td>
    <td>54.4</td>
    <td>19.8</td>
    <td>40.1</td>
  </tr>
  <tr>
    <td>LLM-as-a-judge (our data)</td>
    <td>860k</td>
    <td>44.0</td>
    <td>99.0</td>
    <td>60.9</td>
    <td>33.5</td>
    <td>94.8</td>
    <td>49.5</td>
    <td>24.7</td>
    <td>97.1</td>
    <td>39.4</td>
    <td>22.3</td>
    <td>95.4</td>
    <td>36.1</td>
    <td>46.5</td>
  </tr>
  <tr>
    <td>Human Annotation (PRM800K)</td>
    <td>264k</td>
    <td>53.1</td>
    <td>95.3</td>
    <td>68.2</td>
    <td>48.0</td>
    <td>90.1</td>
    <td><b>62.6</b></td>
    <td>35.7</td>
    <td>87.3</td>
    <td><b>50.7</b></td>
    <td>29.8</td>
    <td>86.3</td>
    <td><b>44.3</b></td>
    <td><b>56.5</b></td>
  </tr>
</table>

Table 10: PRMs performance comparison on PROCESSBENCH. The models are trained on the different data construction methods including MC estimation, LLM-as-a-judge, and human annotation.

including MATH500 (Lightman et al., 2023), AIME24 ¹, AMC23 ², Minerva Math (Lewkowycz et al., 2022), GaoKao 2023 En (Liao et al., 2024) and OlympiadBench (He et al., 2024). The results are presented in the Table 12 and it can be found that our PRMs maintain superior performance compared to other PRMs, especially on MATH500.

D.5 Best-of-8 with LLM-as-a-judge

Regarding BoN evaluation with LLMs, there are two ways to implement: pairwise and pointwise. For pairwise comparison, we employ a single-elimination tournament method. For N responses, we conduct N-1 comparisons to determine the optimal response. In terms of pointwise comparison, we score each step 1 for correct and 0 for incorrect. We then calculate the proportion of correct steps across all steps and select the response with the highest percentage of correct steps as the best response. The experiment are conduct on the policy model Qwen2.5-Math-7B-Instruct and Qwen2.5-Math-72B-Instruct and the results are shown in Table 17 and Table 18 respectively.

E Prompt Template for LLM-as-a-judge

To construct PRM training data via LLM-as-a-judge, we use the following prompt.

¹https://huggingface.co/datasets/AI-MO/aimo-validation-aime
²https://huggingface.co/datasets/AI-MO/aimo-validation-amc
<table>
  <tr>
    <th>Setting</th>
    <th>GSM8K</th>
    <th>MATH</th>
    <th>Minerva Math</th>
    <th>GaoKao 2023 En</th>
    <th>Olympiad Bench</th>
    <th>College Math</th>
    <th>MMLU STEM</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>pass@8</td>
    <td>97.3</td>
    <td>93.2</td>
    <td>56.6</td>
    <td>83.6</td>
    <td>62.4</td>
    <td>54.1</td>
    <td>95.3</td>
    <td>77.5</td>
  </tr>
  <tr>
    <td>maj@8</td>
    <td>96.0</td>
    <td>88.6</td>
    <td>47.8</td>
    <td>73.8</td>
    <td>50.1</td>
    <td>50.2</td>
    <td>84.9</td>
    <td>70.2</td>
  </tr>
  <tr>
    <th colspan="9">1.5B</th>
  </tr>
  <tr>
    <td>Skywork-PRM-1.5B</td>
    <td>96.5</td>
    <td>88.1</td>
    <td>45.2</td>
    <td>74.3</td>
    <td>48.4</td>
    <td>49.7</td>
    <td>79.7</td>
    <td>68.8</td>
  </tr>
  <tr>
    <th colspan="9">7B+</th>
  </tr>
  <tr>
    <td>Math-Shepherd-PRM-7B</td>
    <td>96.5</td>
    <td>86.8</td>
    <td>45.6</td>
    <td>71.9</td>
    <td>49.2</td>
    <td>49.5</td>
    <td>77.5</td>
    <td>68.1</td>
  </tr>
  <tr>
    <td>RLHFlow-PRM-Mistral-8B</td>
    <td>96.6</td>
    <td>87.5</td>
    <td>46.3</td>
    <td>73.5</td>
    <td>48.9</td>
    <td>49.4</td>
    <td><b>83.4</b></td>
    <td>69.4</td>
  </tr>
  <tr>
    <td>RLHFow-PRM-Deepseek-8B</td>
    <td>96.5</td>
    <td>87.7</td>
    <td>44.5</td>
    <td>73.5</td>
    <td>48.7</td>
    <td>49.4</td>
    <td>84.6</td>
    <td>69.3</td>
  </tr>
  <tr>
    <td>Skywork-PRM-7B</td>
    <td><b>97.0</b></td>
    <td>89.0</td>
    <td>47.1</td>
    <td>75.3</td>
    <td>49.8</td>
    <td>49.9</td>
    <td>76.3</td>
    <td>69.2</td>
  </tr>
  <tr>
    <td>EurusPRM-Stage1</td>
    <td>95.4</td>
    <td>85.6</td>
    <td>44.1</td>
    <td>72.5</td>
    <td>46.5</td>
    <td>49.2</td>
    <td>80.3</td>
    <td>67.7</td>
  </tr>
  <tr>
    <td>EurusPRM-Stage2</td>
    <td>95.3</td>
    <td>85.1</td>
    <td>44.9</td>
    <td>72.5</td>
    <td>47.1</td>
    <td>49.0</td>
    <td>80.2</td>
    <td>67.7</td>
  </tr>
  <tr>
    <td>Qwen2.5-Math-7B-Math-Shepherd</td>
    <td>96.9</td>
    <td>88.5</td>
    <td>46.0</td>
    <td>75.8</td>
    <td>49.9</td>
    <td>49.5</td>
    <td>79.7</td>
    <td>69.5</td>
  </tr>
  <tr>
    <td>Qwen2.5-Math-7B-PRM800K</td>
    <td>96.5</td>
    <td>88.9</td>
    <td><b>47.4</b></td>
    <td>75.3</td>
    <td>50.7</td>
    <td>50.1</td>
    <td>76.6</td>
    <td>69.4</td>
  </tr>
  <tr>
    <td>★ Qwen2.5-Math-PRM-7B</td>
    <td>96.8</td>
    <td><b>89.6</b></td>
    <td>46.7</td>
    <td><b>77.7</b></td>
    <td><b>51.4</b></td>
    <td><b>50.4</b></td>
    <td>76.4</td>
    <td><b>69.9</b></td>
  </tr>
  <tr>
    <th colspan="9">72B</th>
  </tr>
  <tr>
    <td>Qwen2.5-Math-RM-72B</td>
    <td>96.4</td>
    <td>89.8</td>
    <td><b>47.4</b></td>
    <td>76.9</td>
    <td><b>54.5</b></td>
    <td><b>50.6</b></td>
    <td>80.1</td>
    <td><b>70.8</b></td>
  </tr>
  <tr>
    <td>★ Qwen2.5-Math-PRM-72B</td>
    <td><b>96.4</b></td>
    <td><b>89.9</b></td>
    <td>46.0</td>
    <td><b>77.4</b></td>
    <td>52.9</td>
    <td>50.1</td>
    <td><b>82.3</b></td>
    <td>70.7</td>
  </tr>
</table>

Table 11: Performance comparison on the Best-of-8 strategy of the policy model Qwen2.5-Math-72B-Instruct. ★ represents the models we trained.

<table>
  <tr>
    <th>Setting</th>
    <th>MATH500</th>
    <th>AIME24</th>
    <th>AMC23</th>
    <th>Minerva Math</th>
    <th>GaoKao 2023 En</th>
    <th>Olympiad Bench</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>pass@64</td>
    <td>96.0</td>
    <td>50.0</td>
    <td>95.0</td>
    <td>56.6</td>
    <td>86.8</td>
    <td>73.5</td>
    <td>76.3</td>
  </tr>
  <tr>
    <td>maj@64</td>
    <td>84.2</td>
    <td>16.7</td>
    <td>77.5</td>
    <td>34.6</td>
    <td>73.8</td>
    <td>51.1</td>
    <td>56.3</td>
  </tr>
  <tr>
    <th colspan="8">1.5B</th>
  </tr>
  <tr>
    <td>Skywork-PRM-1.5B</td>
    <td>81.2</td>
    <td>20.0</td>
    <td>62.5</td>
    <td>31.6</td>
    <td>70.9</td>
    <td>46.5</td>
    <td>52.1</td>
  </tr>
  <tr>
    <th colspan="8">7B+</th>
  </tr>
  <tr>
    <td>Math-Shepherd-PRM-7B</td>
    <td>79.6</td>
    <td>20.0</td>
    <td>62.5</td>
    <td>32.4</td>
    <td>70.1</td>
    <td>43.9</td>
    <td>51.4</td>
  </tr>
  <tr>
    <td>RLHFow-PRM-Mistral-8B</td>
    <td>82.4</td>
    <td>20.0</td>
    <td>62.5</td>
    <td>30.9</td>
    <td>69.1</td>
    <td>45.9</td>
    <td>51.8</td>
  </tr>
  <tr>
    <td>RLHFow-PRM-Deepseek-8B</td>
    <td>80.2</td>
    <td>20.0</td>
    <td><b>67.5</b></td>
    <td><b>35.3</b></td>
    <td>69.1</td>
    <td>46.2</td>
    <td>53.1</td>
  </tr>
  <tr>
    <td>Skywork-PRM-7B</td>
    <td>84.6</td>
    <td>20.0</td>
    <td><b>67.5</b></td>
    <td>32.0</td>
    <td>71.2</td>
    <td>47.1</td>
    <td>53.7</td>
  </tr>
  <tr>
    <td>EurusPRM-Stage1</td>
    <td>76.0</td>
    <td>10.0</td>
    <td>55.0</td>
    <td>27.6</td>
    <td>66.5</td>
    <td>40.0</td>
    <td>45.9</td>
  </tr>
  <tr>
    <td>EurusPRM-Stage2</td>
    <td>76.2</td>
    <td>10.0</td>
    <td>52.5</td>
    <td>27.9</td>
    <td>67.0</td>
    <td>40.3</td>
    <td>45.7</td>
  </tr>
  <tr>
    <td>Qwen2.5-Math-7B-Math-Shepherd</td>
    <td>84.2</td>
    <td><b>23.3</b></td>
    <td><b>67.5</b></td>
    <td>34.6</td>
    <td>72.5</td>
    <td>47.4</td>
    <td>54.9</td>
  </tr>
  <tr>
    <td>Qwen2.5-Math-7B-PRM800K</td>
    <td>83.6</td>
    <td><b>23.3</b></td>
    <td><b>67.5</b></td>
    <td>33.8</td>
    <td>74.8</td>
    <td>48.3</td>
    <td>55.2</td>
  </tr>
  <tr>
    <td>★ Qwen2.5-Math-PRM-7B</td>
    <td><b>87.8</b></td>
    <td>20.0</td>
    <td><b>67.5</b></td>
    <td>33.8</td>
    <td><b>75.8</b></td>
    <td><b>51.4</b></td>
    <td><b>56.1</b></td>
  </tr>
  <tr>
    <th colspan="8">72B</th>
  </tr>
  <tr>
    <td>Qwen2.5-Math-RM-72B</td>
    <td>82.0</td>
    <td><b>36.7</b></td>
    <td><b>75.0</b></td>
    <td>37.5</td>
    <td><b>77.7</b></td>
    <td>54.1</td>
    <td><b>60.5</b></td>
  </tr>
  <tr>
    <td>★ Qwen2.5-Math-PRM-72B</td>
    <td><b>87.8</b></td>
    <td>23.3</td>
    <td>72.5</td>
    <td><b>38.6</b></td>
    <td>77.4</td>
    <td><b>55.3</b></td>
    <td>59.2</td>
  </tr>
</table>

Table 12: Performance comparison on the Best-of-64 strategy of the policy model Qwen2.5-Math-7B-Instruct. ★ represents the models we trained.
<table>
  <tr>
    <th>Setting</th>
    <th>Scoring</th>
    <th>GSM8K</th>
    <th>MATH</th>
    <th>Minerva Math</th>
    <th>GaoKao 2023 En</th>
    <th>Olympiad Bench</th>
    <th>College Math</th>
    <th>MMLU STEM</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td rowspan="2">pass@8 (Upper Bound)</td>
    <td>-</td>
    <td>98.1</td>
    <td>92</td>
    <td>49.3</td>
    <td>80.5</td>
    <td>59.6</td>
    <td>52.6</td>
    <td>90.5</td>
    <td>74.7</td>
  </tr>
  <tr>
    <td>-</td>
    <td>96.7</td>
    <td>87.1</td>
    <td>41.2</td>
    <td>72.5</td>
    <td>44.4</td>
    <td>47.8</td>
    <td>73.8</td>
    <td>66.2</td>
  </tr>
  <tr>
    <td rowspan="3">Math-Shepherd-PRM-7B</td>
    <td>last</td>
    <td>96.8</td>
    <td>85.2</td>
    <td>39.0</td>
    <td>70.1</td>
    <td>42.8</td>
    <td>47.2</td>
    <td>67.7</td>
    <td>64.1</td>
  </tr>
  <tr>
    <td>product</td>
    <td>97.3</td>
    <td>85.4</td>
    <td>37.9</td>
    <td>70.6</td>
    <td>40.4</td>
    <td>47.2</td>
    <td>70.5</td>
    <td>64.2</td>
  </tr>
  <tr>
    <td>min</td>
    <td>96.9</td>
    <td>85.3</td>
    <td>39.0</td>
    <td>69.9</td>
    <td>42.2</td>
    <td>47.4</td>
    <td>70.6</td>
    <td>64.5</td>
  </tr>
  <tr>
    <td rowspan="3">RLHFow-PRM-Mistral-8B</td>
    <td>last</td>
    <td>97.0</td>
    <td>85.3</td>
    <td>39.0</td>
    <td>71.2</td>
    <td>44.0</td>
    <td>47.1</td>
    <td>64.0</td>
    <td>63.9</td>
  </tr>
  <tr>
    <td>product</td>
    <td>97.0</td>
    <td>86.1</td>
    <td>37.1</td>
    <td>70.6</td>
    <td>41.2</td>
    <td>47.6</td>
    <td>69.5</td>
    <td>64.2</td>
  </tr>
  <tr>
    <td>min</td>
    <td>97.0</td>
    <td>84.3</td>
    <td>37.1</td>
    <td>69.4</td>
    <td>40.4</td>
    <td>46.9</td>
    <td>68.7</td>
    <td>63.4</td>
  </tr>
  <tr>
    <td rowspan="3">RLHFow-PRM-Deepseek-8B</td>
    <td>last</td>
    <td>97.0</td>
    <td>84.7</td>
    <td>35.7</td>
    <td>70.4</td>
    <td>43.0</td>
    <td>46.8</td>
    <td>63.8</td>
    <td>63.1</td>
  </tr>
  <tr>
    <td>product</td>
    <td>97.3</td>
    <td>86.3</td>
    <td>40.8</td>
    <td>70.9</td>
    <td>42.2</td>
    <td>47.2</td>
    <td>69.3</td>
    <td>64.9</td>
  </tr>
  <tr>
    <td>min</td>
    <td>97.3</td>
    <td>84.5</td>
    <td>38.2</td>
    <td>69.6</td>
    <td>40.7</td>
    <td>46.5</td>
    <td>67.6</td>
    <td>63.5</td>
  </tr>
  <tr>
    <td rowspan="3">Skywork-PRM-1.5B</td>
    <td>last</td>
    <td>96.8</td>
    <td>86.4</td>
    <td>39.0</td>
    <td>71.7</td>
    <td>45.0</td>
    <td>47.9</td>
    <td>68.2</td>
    <td>65.0</td>
  </tr>
  <tr>
    <td>product</td>
    <td>96.9</td>
    <td>86.7</td>
    <td>37.9</td>
    <td>70.1</td>
    <td>42.1</td>
    <td>47.9</td>
    <td>67.9</td>
    <td>64.2</td>
  </tr>
  <tr>
    <td>min</td>
    <td>96.6</td>
    <td>86.6</td>
    <td>37.9</td>
    <td>71.9</td>
    <td>43.1</td>
    <td>48.2</td>
    <td>66.9</td>
    <td>64.5</td>
  </tr>
  <tr>
    <td rowspan="3">Skywork-PRM-7B</td>
    <td>last</td>
    <td>97.2</td>
    <td>87.3</td>
    <td>41.2</td>
    <td>73.8</td>
    <td>45.8</td>
    <td>48.3</td>
    <td>65.3</td>
    <td>65.6</td>
  </tr>
  <tr>
    <td>product</td>
    <td>97.3</td>
    <td>87.3</td>
    <td>38.2</td>
    <td>71.9</td>
    <td>43.7</td>
    <td>47.8</td>
    <td>67.7</td>
    <td>64.8</td>
  </tr>
  <tr>
    <td>min</td>
    <td>96.7</td>
    <td>87.0</td>
    <td>39.7</td>
    <td>71.2</td>
    <td>42.5</td>
    <td>48.2</td>
    <td>66.6</td>
    <td>64.6</td>
  </tr>
  <tr>
    <td rowspan="3">EurusPRM-Stage1</td>
    <td>last</td>
    <td>94.7</td>
    <td>79.7</td>
    <td>32.7</td>
    <td>61.6</td>
    <td>33.8</td>
    <td>45.7</td>
    <td>63.4</td>
    <td>58.8</td>
  </tr>
  <tr>
    <td>product</td>
    <td>95.6</td>
    <td>83.0</td>
    <td>35.7</td>
    <td>66.2</td>
    <td>38.2</td>
    <td>46.2</td>
    <td>66.6</td>
    <td>61.6</td>
  </tr>
  <tr>
    <td>min</td>
    <td>95.8</td>
    <td>83.3</td>
    <td>39.0</td>
    <td>67.8</td>
    <td>37.9</td>
    <td>46.6</td>
    <td>67.4</td>
    <td>62.5</td>
  </tr>
  <tr>
    <td rowspan="3">EurusPRM-Stage2</td>
    <td>last</td>
    <td>94.7</td>
    <td>79.7</td>
    <td>33.1</td>
    <td>61.3</td>
    <td>34.2</td>
    <td>45.7</td>
    <td>63.5</td>
    <td>58.9</td>
  </tr>
  <tr>
    <td>product</td>
    <td>95.4</td>
    <td>83.4</td>
    <td>34.9</td>
    <td>67.3</td>
    <td>39.1</td>
    <td>46.3</td>
    <td>67.3</td>
    <td>62.0</td>
  </tr>
  <tr>
    <td>min</td>
    <td>96.1</td>
    <td>83.6</td>
    <td>39.3</td>
    <td>68.8</td>
    <td>38.8</td>
    <td>46.7</td>
    <td>67.5</td>
    <td>63.0</td>
  </tr>
  <tr>
    <td rowspan="3">Qwen2.5-Math-7B-Math-Shepherd</td>
    <td>last</td>
    <td>97.1</td>
    <td>87.7</td>
    <td>38.6</td>
    <td>73.8</td>
    <td>44.6</td>
    <td>48.1</td>
    <td>68.0</td>
    <td>65.4</td>
  </tr>
  <tr>
    <td>product</td>
    <td>96.9</td>
    <td>86.5</td>
    <td>36.8</td>
    <td>71.4</td>
    <td>41.6</td>
    <td>47.7</td>
    <td>69.3</td>
    <td>64.3</td>
  </tr>
  <tr>
    <td>min</td>
    <td>97.0</td>
    <td>86.7</td>
    <td>36.8</td>
    <td>72.5</td>
    <td>43.1</td>
    <td>47.6</td>
    <td>70.7</td>
    <td>64.9</td>
  </tr>
  <tr>
    <td rowspan="3">Qwen2.5-Math-7B-PRM800K</td>
    <td>last</td>
    <td>96.7</td>
    <td>86.3</td>
    <td>37.9</td>
    <td>71.9</td>
    <td>44.3</td>
    <td>47.6</td>
    <td>68.1</td>
    <td>64.7</td>
  </tr>
  <tr>
    <td>product</td>
    <td>96.9</td>
    <td>86.9</td>
    <td>37.1</td>
    <td>71.2</td>
    <td>44.0</td>
    <td>47.6</td>
    <td>70.9</td>
    <td>64.9</td>
  </tr>
  <tr>
    <td>min</td>
    <td>96.9</td>
    <td>86.6</td>
    <td>39.7</td>
    <td>71.7</td>
    <td>45.6</td>
    <td>47.8</td>
    <td>71.1</td>
    <td>65.6</td>
  </tr>
  <tr>
    <td rowspan="3">★ Qwen2.5-Math-PRM-7B</td>
    <td>last</td>
    <td>96.9</td>
    <td>87.2</td>
    <td>39.0</td>
    <td>73.5</td>
    <td>45.5</td>
    <td>48.5</td>
    <td>72.0</td>
    <td>66.1</td>
  </tr>
  <tr>
    <td>product</td>
    <td>97.1</td>
    <td>88.0</td>
    <td>42.6</td>
    <td>74.5</td>
    <td>47.6</td>
    <td>48.7</td>
    <td>74.5</td>
    <td>67.6</td>
  </tr>
  <tr>
    <td>min</td>
    <td>97.0</td>
    <td>87.8</td>
    <td>42.3</td>
    <td>74.3</td>
    <td>46.2</td>
    <td>48.3</td>
    <td>74.1</td>
    <td>67.1</td>
  </tr>
  <tr>
    <td rowspan="3">★ Qwen2.5-Math-PRM-72B</td>
    <td>last</td>
    <td>97.6</td>
    <td>88.9</td>
    <td>43.4</td>
    <td>73.8</td>
    <td>49.2</td>
    <td>49.6</td>
    <td>76.8</td>
    <td>68.5</td>
  </tr>
  <tr>
    <td>product</td>
    <td>97.6</td>
    <td>88.7</td>
    <td>46.0</td>
    <td>74.3</td>
    <td>48.1</td>
    <td>49.3</td>
    <td>81.1</td>
    <td>69.3</td>
  </tr>
  <tr>
    <td>min</td>
    <td>97.6</td>
    <td>88.8</td>
    <td>45.2</td>
    <td>74.5</td>
    <td>48.1</td>
    <td>49.2</td>
    <td>80.9</td>
    <td>69.2</td>
  </tr>
</table>

Table 13: Performance comparison on the Best-of-8 strategy of the policy model Qwen2.5-Math-7B-Instruct with 3 scoring strategies: last, product and minimum. ★ represents the models we trained.
<table>
  <tr>
    <th>Setting</th>
    <th>Scoring</th>
    <th>GSM8K</th>
    <th>MATH</th>
    <th>Minerva Math</th>
    <th>GaoKao 2023 En</th>
    <th>Olympiad Bench</th>
    <th>College Math</th>
    <th>MMLU STEM</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td rowspan="2">pass@8 (Upper Bound)<br>maj@8</td>
    <td>-</td>
    <td>97.3</td>
    <td>93.2</td>
    <td>56.6</td>
    <td>83.6</td>
    <td>62.4</td>
    <td>54.1</td>
    <td>95.3</td>
    <td>77.5</td>
  </tr>
  <tr>
    <td>-</td>
    <td>96.0</td>
    <td>88.6</td>
    <td>47.8</td>
    <td>73.8</td>
    <td>50.1</td>
    <td>50.2</td>
    <td>84.9</td>
    <td>70.2</td>
  </tr>
  <tr>
    <td rowspan="3">Math-Shepherd-PRM-7B</td>
    <td>last</td>
    <td>96.2</td>
    <td>87.0</td>
    <td>46.7</td>
    <td>73.0</td>
    <td>47.3</td>
    <td>49.8</td>
    <td>76.3</td>
    <td>68.0</td>
  </tr>
  <tr>
    <td>product</td>
    <td>96.5</td>
    <td>86.8</td>
    <td>45.6</td>
    <td>71.9</td>
    <td>49.2</td>
    <td>49.5</td>
    <td>77.5</td>
    <td>68.1</td>
  </tr>
  <tr>
    <td>min</td>
    <td>96.1</td>
    <td>86.8</td>
    <td>45.6</td>
    <td>73.2</td>
    <td>48.6</td>
    <td>49.9</td>
    <td>76.0</td>
    <td>68.0</td>
  </tr>
  <tr>
    <td rowspan="3">RLHFlow-PRM-Mistral-8B</td>
    <td>last</td>
    <td>96.3</td>
    <td>86.6</td>
    <td>44.9</td>
    <td>74.3</td>
    <td>47.6</td>
    <td>49.3</td>
    <td>67.1</td>
    <td>66.6</td>
  </tr>
  <tr>
    <td>product</td>
    <td>96.6</td>
    <td>87.5</td>
    <td>46.3</td>
    <td>73.5</td>
    <td>48.9</td>
    <td>49.4</td>
    <td>83.4</td>
    <td>69.4</td>
  </tr>
  <tr>
    <td>min</td>
    <td>96.4</td>
    <td>86.3</td>
    <td>44.5</td>
    <td>71.9</td>
    <td>47.9</td>
    <td>49.3</td>
    <td>76.0</td>
    <td>67.5</td>
  </tr>
  <tr>
    <td rowspan="3">RLHFlow-PRM-Deepseek-8B</td>
    <td>last</td>
    <td>96.1</td>
    <td>86.6</td>
    <td>46.3</td>
    <td>73.2</td>
    <td>49.2</td>
    <td>49.2</td>
    <td>71.7</td>
    <td>67.5</td>
  </tr>
  <tr>
    <td>product</td>
    <td>96.5</td>
    <td>87.7</td>
    <td>44.5</td>
    <td>73.5</td>
    <td>48.7</td>
    <td>49.4</td>
    <td>84.6</td>
    <td>69.3</td>
  </tr>
  <tr>
    <td>min</td>
    <td>96.6</td>
    <td>87.4</td>
    <td>44.1</td>
    <td>74.0</td>
    <td>48.6</td>
    <td>49.3</td>
    <td>74.8</td>
    <td>67.8</td>
  </tr>
  <tr>
    <td rowspan="3">Skywork-PRM-1.5B</td>
    <td>last</td>
    <td>96.1</td>
    <td>88.6</td>
    <td>44.9</td>
    <td>72.2</td>
    <td>47.9</td>
    <td>50.1</td>
    <td>74.2</td>
    <td>67.7</td>
  </tr>
  <tr>
    <td>product</td>
    <td>96.5</td>
    <td>88.1</td>
    <td>45.2</td>
    <td>74.3</td>
    <td>48.4</td>
    <td>49.7</td>
    <td>79.7</td>
    <td>68.8</td>
  </tr>
  <tr>
    <td>min</td>
    <td>96.0</td>
    <td>88.3</td>
    <td>45.6</td>
    <td>73.8</td>
    <td>48.6</td>
    <td>50.1</td>
    <td>75.9</td>
    <td>68.3</td>
  </tr>
  <tr>
    <td rowspan="3">Skywork-PRM-7B</td>
    <td>last</td>
    <td>97.0</td>
    <td>89.0</td>
    <td>46.0</td>
    <td>74.8</td>
    <td>51.0</td>
    <td>49.7</td>
    <td>66.7</td>
    <td>67.7</td>
  </tr>
  <tr>
    <td>product</td>
    <td>97.0</td>
    <td>89.0</td>
    <td>47.1</td>
    <td>75.3</td>
    <td>49.8</td>
    <td>49.9</td>
    <td>76.3</td>
    <td>69.2</td>
  </tr>
  <tr>
    <td>min</td>
    <td>96.9</td>
    <td>89.2</td>
    <td>46.7</td>
    <td>73.5</td>
    <td>49.8</td>
    <td>49.8</td>
    <td>73.2</td>
    <td>68.4</td>
  </tr>
  <tr>
    <td rowspan="3">EurusPRM-Stage1</td>
    <td>last</td>
    <td>95.9</td>
    <td>87.3</td>
    <td>44.9</td>
    <td>72.7</td>
    <td>47.0</td>
    <td>49.4</td>
    <td>78.4</td>
    <td>67.9</td>
  </tr>
  <tr>
    <td>product</td>
    <td>95.4</td>
    <td>85.6</td>
    <td>44.1</td>
    <td>72.5</td>
    <td>46.5</td>
    <td>49.2</td>
    <td>80.3</td>
    <td>67.7</td>
  </tr>
  <tr>
    <td>min</td>
    <td>96.4</td>
    <td>88.2</td>
    <td>44.9</td>
    <td>75.1</td>
    <td>49.0</td>
    <td>49.5</td>
    <td>83.7</td>
    <td>69.5</td>
  </tr>
  <tr>
    <td rowspan="3">EurusPRM-Stage2</td>
    <td>last</td>
    <td>96.0</td>
    <td>87.7</td>
    <td>44.5</td>
    <td>73.5</td>
    <td>47.0</td>
    <td>49.4</td>
    <td>78.1</td>
    <td>68.0</td>
  </tr>
  <tr>
    <td>product</td>
    <td>95.3</td>
    <td>85.1</td>
    <td>44.9</td>
    <td>72.5</td>
    <td>47.1</td>
    <td>49.0</td>
    <td>80.2</td>
    <td>67.7</td>
  </tr>
  <tr>
    <td>min</td>
    <td>96.5</td>
    <td>88.6</td>
    <td>45.2</td>
    <td>75.3</td>
    <td>48.9</td>
    <td>49.6</td>
    <td>83.3</td>
    <td>69.6</td>
  </tr>
  <tr>
    <td rowspan="3">Qwen2.5-Math-7B-Math-Shepherd</td>
    <td>last</td>
    <td>97.0</td>
    <td>89.6</td>
    <td>44.9</td>
    <td>77.4</td>
    <td>50.8</td>
    <td>50.5</td>
    <td>74.9</td>
    <td>69.3</td>
  </tr>
  <tr>
    <td>product</td>
    <td>96.9</td>
    <td>88.5</td>
    <td>46.0</td>
    <td>75.8</td>
    <td>49.9</td>
    <td>49.5</td>
    <td>79.7</td>
    <td>69.5</td>
  </tr>
  <tr>
    <td>min</td>
    <td>97.0</td>
    <td>88.6</td>
    <td>46.0</td>
    <td>74.8</td>
    <td>50.2</td>
    <td>49.6</td>
    <td>79.6</td>
    <td>69.4</td>
  </tr>
  <tr>
    <td rowspan="3">Qwen2.5-Math-7B-PRM800K</td>
    <td>last</td>
    <td>96.7</td>
    <td>88.8</td>
    <td>47.1</td>
    <td>76.1</td>
    <td>50.1</td>
    <td>49.5</td>
    <td>71.8</td>
    <td>68.6</td>
  </tr>
  <tr>
    <td>product</td>
    <td>96.5</td>
    <td>88.9</td>
    <td>47.4</td>
    <td>75.3</td>
    <td>50.7</td>
    <td>50.1</td>
    <td>76.6</td>
    <td>69.4</td>
  </tr>
  <tr>
    <td>min</td>
    <td>96.5</td>
    <td>89.1</td>
    <td>47.1</td>
    <td>76.1</td>
    <td>50.7</td>
    <td>49.9</td>
    <td>75.3</td>
    <td>69.2</td>
  </tr>
  <tr>
    <td rowspan="3">★ Qwen2.5-Math-PRM-7B</td>
    <td>last</td>
    <td>96.8</td>
    <td>89.0</td>
    <td>46.7</td>
    <td>75.3</td>
    <td>49.8</td>
    <td>50.3</td>
    <td>78.4</td>
    <td>69.5</td>
  </tr>
  <tr>
    <td>product</td>
    <td>96.8</td>
    <td>89.6</td>
    <td>46.7</td>
    <td>77.7</td>
    <td>51.4</td>
    <td>50.4</td>
    <td>76.4</td>
    <td>69.9</td>
  </tr>
  <tr>
    <td>min</td>
    <td>96.7</td>
    <td>89.6</td>
    <td>46.3</td>
    <td>77.9</td>
    <td>50.8</td>
    <td>50.3</td>
    <td>76.0</td>
    <td>69.7</td>
  </tr>
  <tr>
    <td rowspan="3">★ Qwen2.5-Math-PRM-72B</td>
    <td>last</td>
    <td>96.3</td>
    <td>89.8</td>
    <td>47.8</td>
    <td>76.6</td>
    <td>53.3</td>
    <td>50.9</td>
    <td>80.5</td>
    <td>70.7</td>
  </tr>
  <tr>
    <td>product</td>
    <td>96.4</td>
    <td>89.9</td>
    <td>46.0</td>
    <td>77.4</td>
    <td>52.9</td>
    <td>50.1</td>
    <td>82.3</td>
    <td>70.7</td>
  </tr>
  <tr>
    <td>min</td>
    <td>96.4</td>
    <td>89.7</td>
    <td>46.3</td>
    <td>77.7</td>
    <td>52.4</td>
    <td>50.4</td>
    <td>81.2</td>
    <td>70.6</td>
  </tr>
</table>

Table 14: Performance comparison on the Best-of-8 strategy of the policy model Qwen2.5-Math-72B-Instruct with 3 scoring strategies: last, product and minimum. ★ represents the models we trained.
<table>
  <tr>
    <th>Setting</th>
    <th>Scoring</th>
    <th>CMATH</th>
    <th>CN Middle School 24</th>
    <th>GaoKao</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td rowspan="3">pass@8 (Upper Bound)</td>
    <td>-</td>
    <td>95.3</td>
    <td>82.2</td>
    <td>84.3</td>
    <td>87.3</td>
  </tr>
  <tr>
    <td>-</td>
    <td>92.7</td>
    <td>78.2</td>
    <td>68.1</td>
    <td>79.7</td>
  </tr>
  <tr>
    <td>last</td>
    <td>91.8</td>
    <td>80.2</td>
    <td>63.0</td>
    <td>78.3</td>
  </tr>
  <tr>
    <td rowspan="3">Math-Shepherd-PRM-7B</td>
    <td>product</td>
    <td>92.0</td>
    <td>80.2</td>
    <td>69.1</td>
    <td>80.4</td>
  </tr>
  <tr>
    <td>min</td>
    <td>91.5</td>
    <td>80.2</td>
    <td>69.8</td>
    <td>80.5</td>
  </tr>
  <tr>
    <td>last</td>
    <td>92.8</td>
    <td>79.2</td>
    <td>57.2</td>
    <td>76.4</td>
  </tr>
  <tr>
    <td rowspan="3">RLHFlow-PRM-Mistral-8B</td>
    <td>product</td>
    <td>92.7</td>
    <td>77.2</td>
    <td>65.8</td>
    <td>78.6</td>
  </tr>
  <tr>
    <td>min</td>
    <td>92.8</td>
    <td>76.2</td>
    <td>62.1</td>
    <td>77.0</td>
  </tr>
  <tr>
    <td>last</td>
    <td>93.2</td>
    <td>75.2</td>
    <td>56.9</td>
    <td>75.1</td>
  </tr>
  <tr>
    <td rowspan="3">RLHFlow-PRM-Deepseek-8B</td>
    <td>product</td>
    <td>92.7</td>
    <td>76.2</td>
    <td>63.6</td>
    <td>77.5</td>
  </tr>
  <tr>
    <td>min</td>
    <td>93.0</td>
    <td>74.3</td>
    <td>67.3</td>
    <td>78.2</td>
  </tr>
  <tr>
    <td>last</td>
    <td>93.8</td>
    <td>80.2</td>
    <td>66.6</td>
    <td>80.2</td>
  </tr>
  <tr>
    <td rowspan="3">Skywork-PRM-1.5B</td>
    <td>product</td>
    <td>92.8</td>
    <td>79.2</td>
    <td>66.3</td>
    <td>79.4</td>
  </tr>
  <tr>
    <td>min</td>
    <td>93.3</td>
    <td>80.2</td>
    <td>66.6</td>
    <td>80.0</td>
  </tr>
  <tr>
    <td>last</td>
    <td>94.0</td>
    <td>81.2</td>
    <td>66.7</td>
    <td>80.6</td>
  </tr>
  <tr>
    <td rowspan="3">Skywork-PRM-7B</td>
    <td>product</td>
    <td>93.3</td>
    <td>79.2</td>
    <td>68.1</td>
    <td>80.2</td>
  </tr>
  <tr>
    <td>min</td>
    <td>93.8</td>
    <td>80.2</td>
    <td>66.3</td>
    <td>80.1</td>
  </tr>
  <tr>
    <td>last</td>
    <td>91.8</td>
    <td>77.2</td>
    <td>55.4</td>
    <td>74.8</td>
  </tr>
  <tr>
    <td rowspan="3">EurusPRM-Stage1</td>
    <td>product</td>
    <td>91.7</td>
    <td>77.2</td>
    <td>52.6</td>
    <td>73.8</td>
  </tr>
  <tr>
    <td>min</td>
    <td>91.7</td>
    <td>78.2</td>
    <td>64.4</td>
    <td>78.1</td>
  </tr>
  <tr>
    <td>last</td>
    <td>91.8</td>
    <td>77.2</td>
    <td>55.7</td>
    <td>74.9</td>
  </tr>
  <tr>
    <td rowspan="3">EurusPRM-Stage2</td>
    <td>product</td>
    <td>92.0</td>
    <td>77.2</td>
    <td>52.4</td>
    <td>73.9</td>
  </tr>
  <tr>
    <td>min</td>
    <td>92.0</td>
    <td>78.2</td>
    <td>64.7</td>
    <td>78.3</td>
  </tr>
  <tr>
    <td>last</td>
    <td>93.0</td>
    <td>81.2</td>
    <td>65.4</td>
    <td>79.9</td>
  </tr>
  <tr>
    <td rowspan="3">Qwen2.5-Math-7B-Math-Shepherd</td>
    <td>product</td>
    <td>93.0</td>
    <td>79.2</td>
    <td>67.7</td>
    <td>80.0</td>
  </tr>
  <tr>
    <td>min</td>
    <td>92.5</td>
    <td>80.2</td>
    <td>69.8</td>
    <td>80.8</td>
  </tr>
  <tr>
    <td>last</td>
    <td>92.8</td>
    <td>78.2</td>
    <td>67.1</td>
    <td>79.4</td>
  </tr>
  <tr>
    <td rowspan="3">Qwen2.5-Math-7B-PRM800K</td>
    <td>product</td>
    <td>92.7</td>
    <td>77.2</td>
    <td>68.9</td>
    <td>79.6</td>
  </tr>
  <tr>
    <td>min</td>
    <td>93.0</td>
    <td>77.2</td>
    <td>69.4</td>
    <td>79.9</td>
  </tr>
  <tr>
    <td>last</td>
    <td>93.3</td>
    <td>80.2</td>
    <td>68.2</td>
    <td>80.6</td>
  </tr>
  <tr>
    <td rowspan="3">★ Qwen2.5-Math-PRM-7B</td>
    <td>product</td>
    <td>93.7</td>
    <td>80.2</td>
    <td>70.1</td>
    <td>81.3</td>
  </tr>
  <tr>
    <td>min</td>
    <td>93.5</td>
    <td>80.2</td>
    <td>71.7</td>
    <td>81.8</td>
  </tr>
  <tr>
    <td>last</td>
    <td>94.3</td>
    <td>80.2</td>
    <td>72.1</td>
    <td>82.2</td>
  </tr>
  <tr>
    <td rowspan="3">★ Qwen2.5-Math-PRM-72B</td>
    <td>product</td>
    <td>94.2</td>
    <td>80.2</td>
    <td>73.5</td>
    <td>82.6</td>
  </tr>
  <tr>
    <td>min</td>
    <td>94.2</td>
    <td>80.2</td>
    <td>73.1</td>
    <td>82.5</td>
  </tr>
</table>

Table 15: Best-of-8 performance comparison on the Chinese benchmarks with the policy model Qwen2.5-Math-7B-Instruct in 3 scoring strategies: last, product and minimum. ★ represents the PRMs we trained.
<table>
  <tr>
    <th>Setting</th>
    <th>Scoring</th>
    <th>CMATH</th>
    <th>CN Middle School 24</th>
    <th>GaoKao</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>pass@8 (Upper Bound)</td>
    <td>-</td>
    <td>96.8</td>
    <td>83.2</td>
    <td>86.2</td>
    <td>88.7</td>
  </tr>
  <tr>
    <td>maj@8</td>
    <td>-</td>
    <td>95.3</td>
    <td>79.2</td>
    <td>75.0</td>
    <td>83.2</td>
  </tr>
  <tr>
    <td rowspan="3">Math-Shepherd-PRM-7B</td>
    <td>last</td>
    <td>93.7</td>
    <td>78.2</td>
    <td>73.2</td>
    <td>81.7</td>
  </tr>
  <tr>
    <td>product</td>
    <td>94.0</td>
    <td>80.2</td>
    <td>72.1</td>
    <td>82.1</td>
  </tr>
  <tr>
    <td>min</td>
    <td>93.5</td>
    <td>80.2</td>
    <td>73.9</td>
    <td>82.5</td>
  </tr>
  <tr>
    <td rowspan="3">RLHFlow-PRM-Mistral-8B</td>
    <td>last</td>
    <td>94.3</td>
    <td>79.2</td>
    <td>65.5</td>
    <td>79.7</td>
  </tr>
  <tr>
    <td>product</td>
    <td>93.8</td>
    <td>79.2</td>
    <td>72.0</td>
    <td>81.7</td>
  </tr>
  <tr>
    <td>min</td>
    <td>93.3</td>
    <td>79.2</td>
    <td>71.2</td>
    <td>81.2</td>
  </tr>
  <tr>
    <td rowspan="3">RLHFlow-PRM-Deepseek-8B</td>
    <td>last</td>
    <td>94.3</td>
    <td>79.2</td>
    <td>63.0</td>
    <td>78.8</td>
  </tr>
  <tr>
    <td>product</td>
    <td>94.3</td>
    <td>79.2</td>
    <td>72.5</td>
    <td>82.0</td>
  </tr>
  <tr>
    <td>min</td>
    <td>94.5</td>
    <td>79.2</td>
    <td>73.5</td>
    <td>82.4</td>
  </tr>
  <tr>
    <td rowspan="3">Skywork-PRM-1.5B</td>
    <td>last</td>
    <td>94.8</td>
    <td>80.2</td>
    <td>74.3</td>
    <td>83.1</td>
  </tr>
  <tr>
    <td>product</td>
    <td>93.8</td>
    <td>79.2</td>
    <td>69.7</td>
    <td>80.9</td>
  </tr>
  <tr>
    <td>min</td>
    <td>94.5</td>
    <td>80.2</td>
    <td>74.6</td>
    <td>83.1</td>
  </tr>
  <tr>
    <td rowspan="3">Skywork-PRM-7B</td>
    <td>last</td>
    <td>95.3</td>
    <td>80.2</td>
    <td>72.6</td>
    <td>82.7</td>
  </tr>
  <tr>
    <td>product</td>
    <td>94.7</td>
    <td>80.2</td>
    <td>71.5</td>
    <td>82.1</td>
  </tr>
  <tr>
    <td>min</td>
    <td>94.8</td>
    <td>80.2</td>
    <td>76.0</td>
    <td>83.7</td>
  </tr>
  <tr>
    <td rowspan="3">EurusPRM-Stage1</td>
    <td>last</td>
    <td>94.0</td>
    <td>79.2</td>
    <td>64.5</td>
    <td>79.2</td>
  </tr>
  <tr>
    <td>product</td>
    <td>93.8</td>
    <td>80.2</td>
    <td>64.5</td>
    <td>79.5</td>
  </tr>
  <tr>
    <td>min</td>
    <td>94.7</td>
    <td>79.2</td>
    <td>70.8</td>
    <td>81.6</td>
  </tr>
  <tr>
    <td rowspan="3">EurusPRM-Stage2</td>
    <td>last</td>
    <td>94.2</td>
    <td>79.2</td>
    <td>63.4</td>
    <td>78.9</td>
  </tr>
  <tr>
    <td>product</td>
    <td>93.7</td>
    <td>80.2</td>
    <td>65.4</td>
    <td>79.8</td>
  </tr>
  <tr>
    <td>min</td>
    <td>94.3</td>
    <td>79.2</td>
    <td>69.7</td>
    <td>81.1</td>
  </tr>
  <tr>
    <td rowspan="3">Qwen2.5-Math-7B-Math-Shepherd</td>
    <td>last</td>
    <td>95.0</td>
    <td>81.2</td>
    <td>74.6</td>
    <td>83.6</td>
  </tr>
  <tr>
    <td>product</td>
    <td>94.5</td>
    <td>80.2</td>
    <td>73.0</td>
    <td>82.6</td>
  </tr>
  <tr>
    <td>min</td>
    <td>94.3</td>
    <td>80.2</td>
    <td>71.5</td>
    <td>82.0</td>
  </tr>
  <tr>
    <td rowspan="3">Qwen2.5-Math-7B-PRM800K</td>
    <td>last</td>
    <td>94.2</td>
    <td>79.2</td>
    <td>76.5</td>
    <td>83.3</td>
  </tr>
  <tr>
    <td>product</td>
    <td>94.2</td>
    <td>82.2</td>
    <td>70.8</td>
    <td>82.4</td>
  </tr>
  <tr>
    <td>min</td>
    <td>93.8</td>
    <td>80.2</td>
    <td>72.9</td>
    <td>82.3</td>
  </tr>
  <tr>
    <td rowspan="3">★ Qwen2.5-Math-PRM-7B</td>
    <td>last</td>
    <td>94.7</td>
    <td>79.2</td>
    <td>74.5</td>
    <td>82.8</td>
  </tr>
  <tr>
    <td>product</td>
    <td>94.3</td>
    <td>81.2</td>
    <td>77.6</td>
    <td>84.4</td>
  </tr>
  <tr>
    <td>min</td>
    <td>94.5</td>
    <td>81.2</td>
    <td>77.6</td>
    <td>84.4</td>
  </tr>
  <tr>
    <td rowspan="3">★ Qwen2.5-Math-PRM-72B</td>
    <td>last</td>
    <td>96.0</td>
    <td>79.2</td>
    <td>76.1</td>
    <td>83.8</td>
  </tr>
  <tr>
    <td>product</td>
    <td>96.0</td>
    <td>80.2</td>
    <td>77.2</td>
    <td>84.5</td>
  </tr>
  <tr>
    <td>min</td>
    <td>95.8</td>
    <td>80.2</td>
    <td>77.5</td>
    <td>84.5</td>
  </tr>
</table>

Table 16: Best-of-8 performance comparison on the Chinese benchmarks with the policy model Qwen2.5-Math-72B-Instruct in 3 scoring strategies: last, product and minimum. ★ represents the PRMs we trained.
<table>
  <tr>
    <th>Setting</th>
    <th>GSM8K</th>
    <th>MATH</th>
    <th>Minerva Math</th>
    <th>GaoKao 2023 En</th>
    <th>Olympiad Bench</th>
    <th>College Math</th>
    <th>MMLU STEM</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>pass@8 (Upper Bound)</td>
    <td>98.1</td>
    <td>92</td>
    <td>49.3</td>
    <td>80.5</td>
    <td>59.6</td>
    <td>52.6</td>
    <td>90.5</td>
    <td>74.7</td>
  </tr>
  <tr>
    <td>maj@8</td>
    <td>96.7</td>
    <td>87.1</td>
    <td>41.2</td>
    <td>72.5</td>
    <td>44.4</td>
    <td>47.8</td>
    <td>73.8</td>
    <td>66.2</td>
  </tr>
  <tr>
    <th colspan="9">LLM-as-a-judge, Open-source language models</th>
  </tr>
  <tr>
    <th colspan="9">POINTWISE</th>
  </tr>
  <tr>
    <td>QwQ-32B-Preview</td>
    <td>97.0</td>
    <td>86.0</td>
    <td>39.3</td>
    <td>70.1</td>
    <td>46.2</td>
    <td>47.9</td>
    <td>70.5</td>
    <td>65.3</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Instruct</td>
    <td>97.0</td>
    <td>85.6</td>
    <td>40.1</td>
    <td>70.9</td>
    <td>43.4</td>
    <td>47.9</td>
    <td>73.4</td>
    <td>65.5</td>
  </tr>
  <tr>
    <th colspan="9">PAIRWISE</th>
  </tr>
  <tr>
    <td>QwQ-32B-Preview</td>
    <td>97.6</td>
    <td>89.2</td>
    <td>40.8</td>
    <td>75.8</td>
    <td>50.4</td>
    <td>48.9</td>
    <td>70.5</td>
    <td>67.6</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Instruct</td>
    <td>97.3</td>
    <td>86.8</td>
    <td>40.8</td>
    <td>73.5</td>
    <td>45.0</td>
    <td>48.4</td>
    <td>74.5</td>
    <td>66.6</td>
  </tr>
  <tr>
    <th colspan="9">PRMs</th>
  </tr>
  <tr>
    <td>Qwen2.5-Math-PRM-7B</td>
    <td>97.1</td>
    <td>88.0</td>
    <td>42.6</td>
    <td>74.5</td>
    <td>47.6</td>
    <td>48.7</td>
    <td>74.5</td>
    <td>67.6</td>
  </tr>
  <tr>
    <td>Qwen2.5-Math-PRM-72B</td>
    <td>97.6</td>
    <td>88.7</td>
    <td>46.0</td>
    <td>74.3</td>
    <td>48.1</td>
    <td>49.3</td>
    <td>81.1</td>
    <td>69.3</td>
  </tr>
</table>

Table 17: Performance comparison with LLM-as-a-judge on the Best-of-8 strategy of the policy model Qwen2.5-Math-7B-Instruct.

<table>
  <tr>
    <th>Setting</th>
    <th>GSM8K</th>
    <th>MATH</th>
    <th>Minerva Math</th>
    <th>GaoKao 2023 En</th>
    <th>Olympiad Bench</th>
    <th>College Math</th>
    <th>MMLU STEM</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>pass@8 (Upper Bound)</td>
    <td>97.3</td>
    <td>93.2</td>
    <td>56.6</td>
    <td>83.6</td>
    <td>62.4</td>
    <td>54.1</td>
    <td>95.3</td>
    <td>77.5</td>
  </tr>
  <tr>
    <td>maj@8</td>
    <td>96.0</td>
    <td>88.6</td>
    <td>47.8</td>
    <td>73.8</td>
    <td>50.1</td>
    <td>50.2</td>
    <td>84.9</td>
    <td>70.2</td>
  </tr>
  <tr>
    <th colspan="9">LLM-as-a-judge, Open-source language models</th>
  </tr>
  <tr>
    <th colspan="9">POINTWISE</th>
  </tr>
  <tr>
    <td>QwQ-32B-Preview</td>
    <td>96.2</td>
    <td>88.3</td>
    <td>46.3</td>
    <td>75.3</td>
    <td>51.0</td>
    <td>50.0</td>
    <td>74.9</td>
    <td>68.9</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Instruct</td>
    <td>96.5</td>
    <td>87.8</td>
    <td>47.4</td>
    <td>76.4</td>
    <td>48.9</td>
    <td>50.0</td>
    <td>76.0</td>
    <td>69.0</td>
  </tr>
  <tr>
    <th colspan="9">PAIRWISE</th>
  </tr>
  <tr>
    <td>QwQ-32B-Preview</td>
    <td>96.4</td>
    <td>90.9</td>
    <td>46.0</td>
    <td>79.5</td>
    <td>55.1</td>
    <td>50.5</td>
    <td>73.6</td>
    <td>70.3</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Instruct</td>
    <td>96.1</td>
    <td>88.2</td>
    <td>43.4</td>
    <td>75.3</td>
    <td>50.1</td>
    <td>49.6</td>
    <td>71.4</td>
    <td>67.7</td>
  </tr>
  <tr>
    <th colspan="9">PRMs</th>
  </tr>
  <tr>
    <td>Qwen2.5-Math-PRM-7B</td>
    <td>96.8</td>
    <td>89.6</td>
    <td>46.7</td>
    <td>77.7</td>
    <td>51.4</td>
    <td>50.4</td>
    <td>76.4</td>
    <td>69.9</td>
  </tr>
  <tr>
    <td>Qwen2.5-Math-PRM-72B</td>
    <td>96.4</td>
    <td>89.9</td>
    <td>46.0</td>
    <td>77.4</td>
    <td>52.9</td>
    <td>50.1</td>
    <td>82.3</td>
    <td>70.7</td>
  </tr>
</table>

Table 18: Performance comparison with LLM-as-a-judge on the Best-of-8 strategy of the policy model Qwen2.5-Math-72B-Instruct.
Prompt for constructing PRM training data via LLM-as-a-judge

I will provide a math problem along with a solution. They will be formatted as follows:

[Math Problem]

<math_problem>
...(math problem)...
</math_problem>

[Solution]

<paragraph_1>
...(paragraph 1 of solution)...
</paragraph_1>

...

<paragraph_n>
...(paragraph n of solution)...
</paragraph_n>

Your task is to review each paragraph of the solution in sequence, analyzing, verifying, and critiquing the reasoning in detail. You need to provide the analyses and the conclusion in the following format:

<analysis_1>
...(analysis of paragraph 1)...
</analysis_1>

...

<analysis_n>
...(analysis of paragraph n)...
</analysis_n>

<conclusion>
Correct/Incorrect
</conclusion>

* When you analyze each paragraph, you should use proper verification, recalculation, or reflection to indicate whether it is logically and mathematically valid. Please elaborate on the analysis process carefully.

* If an error is detected in any paragraph, you should describe the nature and cause of the error in detail, and suggest how to correct the error or the correct approach. Once a paragraph is found to contain any error, stop further analysis of subsequent paragraphs (as they may depend on the identified error) and directly provide the conclusion of "Incorrect."
For instance, given a solution of five paragraphs, if an error is found in the third paragraph, you should reply in the following format:

<analysis_1>
...(analysis of paragraph 1)...
</analysis_1>

<analysis_2>
...(analysis of paragraph 2)...
</analysis_3>

<analysis_3>
...(analysis of paragraph 3; since an error is found here, also provide detailed critique and correction guideline)...
</analysis_3>

<conclusion>
Incorrect
</conclusion>

Note that the analyses of paragraphs 4 and 5 should be skipped as the paragraph 3 has been found to contain an error.

* Respond with your analyses and conclusion directly.

---------------------------------------------

The following is the math problem and the solution for you task:

[Math Problem]

{tagged_problem}

[Solution]

{tagged_response}

Reviewer paper summaries: This paper criticizes (and investigates) data construction and evaluation in current
process reward modelling, then offers a preliminary solution and reports its
efficacy. First some background: Process reward modelling (PRM) evaluates a step in
a solution (sequence of steps) to some problem. It is a more granular approach
compared to traditional output reward modelling (ORM) which only evaluates solutions
based on the final answer (last step). Currently, PRMs are trained in a supervised
fashion with sample questions and some solution prefix (up until some step), as well
as a scalar label representing its value in yielding a correct solution. To get this
data however, is costly when done manually with human annotation, so research has
turned towards automatic annotation. In terms of evaluation, PRMs are evaluated by
the Best-of-N (BoN) metric, which has the PRM score N solutions generated by a
policy model, and then reports whether the highest ranked solution yielded the
correct answer as a binary task. Recently, ProcessBench has been published which is
an alternative evaluation strategy, that scores PRMs on their ability to evaluate
solutions step-wise rather than all at once, encompassing problems from all major
math benchmarks used in LLM reasoning.

The first criticism this paper makes is about the MC estimation data construction
method for building labelled data used in process reward modelling (PRM). They
criticize the fact that the policy models, which rollout solutions from a given
step, can generate incorrect processes but correct solutions (since the final answer
is what determines the value of a step in current MC estimation data construction
techniques). They further criticize that the labels are framed as a value metric, as
they estimate the probability of yielding the correct solution, and hypothesize this
negatively affects generalization ability. They also criticize current models for
scoring many processes based on the final answer stating this serves more as an
output reward model (ORM). From this the authors then criticize the Best-of-N (BoN)
evaluation metric and advocate for the use of ProcessBench as a complement. They
claim that BoN scores can be inflated. To corroborate these criticisms and claims,
the authors conduct extensive experiments, including training different models with
different data construction techniques and evaluating them in numerous ways, as well
as analysis into evaluation results and different kinds of errors.

The solution they devise to address these issues is to filter the existing MC
estimation training dataset, by asking a LLM-as-judge to label training data again,
and only keeping those samples whose LLM judged label agrees with the existing one.
By training new LLMs on this filtered dataset, they achieve SOTA results across
open-sourced PRMs, and competitive results with proprietary models. They compare
with a large group of models, evaluating by both BoN and ProcessBench.

This paper explores data generation strategies for training Progress Reward Models (PRMs) in mathematical reasoning. It identifies two key limitations: (1) Monte Carlo (MC) estimation for data generation underperforms compared to LLM-as-a-judge and human annotations, particularly on process-related benchmarks; and (2) a gap exists between PRM predictions and the accuracy of the Bag-of-N (BoN) approach.

To address these issues, the authors propose a hybrid method that combines MC estimation with LLM-as-a-judge to refine PRM training data. Experimental results on 7B and 72B models demonstrate improved outcome accuracy (Bo8) and process accuracy compared to baseline methods.
Summary of strengths: - The authors addressed important issues with process reward modelling, and showed them with extensive experiments using a variety of current models/approaches.
- The authors illustrated the significance of these issues by showing that a simple change (filtering) to the data annotation process can greatly improve performance.

This paper presents extensive empirical results that offer valuable insights into PRM training for mathematical reasoning. These findings not only support future research but also highlight the importance of process accuracy alongside outcome accuracy.

Additionally, the proposed method is simple to implement and demonstrates effectiveness in improving both Bo8 outcome accuracy and process accuracy.
Summary of weaknesses: - Structure & Organization: The predominant issue I have with this paper is its
readability. While I believe the research and results were good, the organization,
presentation, and overall structure of this paper was poor. It is presented as a
paper about your new method, that begins with lots of criticism on existing methods
and lots of discussion about preliminary experiments. Then your approach, which is a
twist on existing methods (in fact dependent on existing methods), is hardly
discussed. I think the main contribution of the paper is actually the criticism and
analysis of issues in the field (lessons) and the paper should be written as such.
The paper should introduce the collected dataset, existing approaches and
evaluations, and then discuss the multitude of experiments conducted to analyze
current systems. From these findings (lessons) you can introduce your twist, all
within the methodology and experiments section. Then across all these results
(analyzing data construction methods, BoN, and your new idea) you can discuss all
results (as you did in "lessons").

- Issue with Detail: I found a lack of (and in one case excessive) detail in this
paper:
        - The introduction proposes several findings (or "lessons") but barely introduce
process reward modelling, how and evaluation works, making these difficult to
interpret. It should only take a small paragraph to explain. 
        - The preliminary trials are like an experiments section, except several details
are not provided such as where data was collected from, what the exact objective is
that's being optimized, and how exactly the LLM-as-judge approach works (does it
output a score? identify error location? etc.). I could kind of figure it out from
the prompt in the appendix (not referenced)
        - I believe section 3 (the "lessons") could be significantly shortened, as it is a
huge discussion section in the middle of the paper. I think hard vs soft label
estimation could be moved to the appendix. 
        - The authors seem to describe their approach under "Stringent Data Filtering..."
on page 4, and in fact describe experiment results with no details, as it is
explained later when it's re-proposed in section 4.
        - The authors also do not mention why or how they collected 500k queries to
construct their dataset. Later there is a 860k dataset mentioned as "their"
dataset. What processing went into turn 500k into 860k?
        - When discussing hard vs soft labels, you mention you collected another dataset of
3 million samples, reduced by filtering to 1.5 million. In section 4, did you train
on this dataset or the 860k dataset? It is unclear.

- LLM Justification: The authors criticize that using a policy model for generating
rollouts due to incorrect processes, yet propose to filter using a LLM. While this
is empirically motivated in the preliminary trials I believe further justification /
discussion is due for why this improves performance. Also, there is no justification
for using LLM-as-judge to construct data to begin with.

- PRMS as ORMs: The 3.2 analysis is not clear. They investigate misalignment,
beginning with manual inspection on 8 sample solutions but don't reference where
that is and what it led to. I think Figure 6 provides good support, but more
discussion is needed to support the implication that effective PRMs might get low
BoN scores. How is a process step deemed incorrect? How is the level of
incorrectness qualified? If a model just makes one mistake in the process versus
several, but yields the right answer, surely that matters when making claims about
PRM effectiveness. None of this is addressed in the paper.

- The authors discuss how 8 completions is crude. Before criticizing MC estimation
entirely, I feel the authors should have either experimented with a higher number of
samples (16, 64, etc.), or justified why this is not necessary. All they say is
increasing this will increase cost and MAY outweigh the benefits. While their
filtering does work, it is unclear if the issue with MC estimation lies in a
fundamental limitation of the approach (as the authors claim) or the compute
required to make it effective.   

- The related work section is very small. If that is truly the only work related
then its fine, just suspicious.

- The authors claim their results confirm that their approach "genuinely teaches
PRMs to assess the correctness of intermediate reasoning steps", but how is this
confirmed? The results of their experiments show that their models use less data and
achieve better results in process evaluation than existing ones (which is terrific).
This doesn't imply they can conclude such a claim however. What does "genuinely"
mean here, and how was this conclusively achieved? Even if true, there is no
discussion about this, only the results section which states the highlights of the
tables. I think a cursory analysis of the distribution of minimum scores in their
new models, statistics about incorrect processes in highly scored solutions, and a
discussion could improve the support of this strong claim.

Most of the results in this paper are derived from Bo8 experiments. However, prior PRM studies typically compare performance at larger **N** values, such as 64. It would be valuable to examine whether the paper’s claims hold or weaken as **N** increases.

The proposed method demonstrates effectiveness but offers limited novelty.

The claim that "a substantial percentage of responses contain process errors while maintaining correct answers, with this phenomenon becoming more pronounced as problem complexity increases" is based on only eight samples from a 7B model, which is a limited sample size and may not be sufficiently robust.
Comments, suggestions and typos: I think the research done in this paper was good, and presents an important analysis
to the field of process reward modelling. However, I feel this was presented very
poorly in this paper. I think the paper's claims were not well supported and the
results were presented stridently. I think with perhaps some additional analysis,
and certainly a significant re-write, the paper could become a strong submission.
There are several grammatical errors throughout the paper, so a proofread from a
native English speaker is recommended.

Increase the figure size when extra space is availabl to enhance readability.
Overall assessment scores: 2.5

3.5
Metareview score: 3
Metareview: This paper investigates data generation strategies for training progress reward models (PRMs) for mathematical reasoning. The work identifies key limitations of relying solely on Monte Carlo (MC) estimation—namely, that it underperforms compared to using an LLM as a judge and human annotations, especially on process-related benchmarks—and also highlights a gap between PRM predictions and the Best-of-N (BoN) evaluation. To address these issues, the authors propose a hybrid method that refines the training data by combining MC estimation with LLM filtering. The empirical results on 7B and 72B models show improvements in both outcome accuracy and process accuracy, and additional experiments with larger N values (e.g., Bo64) reinforce the claims.

Reviewers appreciate the extensive and systematic experimental evaluation, which offers valuable insights into PRM training for mathematical reasoning. The findings are practical and have the potential to serve as a foundation for further research in refining data construction and evaluation techniques. However, concerns were raised regarding the incremental novelty of the approach, as the proposed method is primarily an improvement upon existing techniques rather than a fundamentally new idea. There were also issues about the presentation: some reviewers noted that key details are dispersed between the main text and the appendix, making it difficult to quickly grasp the complete experimental setup and data construction process. Additionally, clarity issues—such as small figure fonts and inconsistent discussion of dataset sizes—were mentioned.

The authors’ responses have largely addressed these concerns. They provided further experimental results with larger N values, clarified the sample sizes, and offered detailed responses about the methodology and the rationale for using LLM-as-a-judge. While differences remain among reviewer expectations—particularly regarding the paper organization and the balance between analysis and methodological innovation—the overall strengths in experimental rigor, practical insights, and comprehensive analysis outweigh the weaknesses. Based on the evidence presented and subsequent clarifications, the paper is recommended for acceptance of findings.
Summary of reasons to publish: - The paper provides a comprehensive empirical investigation of PRM training strategies in mathematical reasoning.

- It demonstrates that a hybrid approach—combining MC estimation with LLM filtering—can improve both outcome accuracy and process accuracy.

- The experiments span multiple tasks and model scales, offering practical insights that can guide future research.

- The study clarifies the limitations of existing data generation techniques and highlights how evaluation methods can be enhanced.