Efficient Process Reward Model Training via Active Learning
Authors: Keyu Duan
Zichen Liu
Xin Mao
Tianyu Pang
Changyu Chen
Qiguang Chen
Michael Qizhe Shieh
Longxu Dou
Venue: COLM 2025 (2025)
Keywords: Process Reward Model
Active Learning
TL;DR: We developed a active learning strategy for prm training that achieves new SOTA performance on ProcessBench with merely at most 20\% labeling compared with previous SOTAs.

Abstract: Process Reward Models (PRMs) provide step-level supervision to large language models (LLMs), but scaling up training data annotation remains challenging for both humans and LLMs. 
To address this limitation, we propose an active learning approach, ActPRM, which proactively selects the most uncertain samples for training, substantially reducing labeling costs.
During training, we use the PRM to estimate uncertainty after the forward pass, retaining only highly uncertain data. A capable yet costly reasoning model then labels this data. Then we compute the loss w.r.t. the labels and update the PRM’s weights.
We compare ActPRM vs. vanilla fine-tuning, on a pool-based active learning setting, demonstrating that ActPRM reduce 50\% annotation, but achieving the comparable or even better performance. 
Beyond annotation efficiency, we further advance the actively trained PRM by filtering over 1M+ math reasoning trajectories with ActPRM, retaining 60\% of the data.
A subsequent training on this selected dataset yields a new state-of-the-art (SOTA) PRM on ProcessBench (75.0\%) and PRMBench (65.5\%) compared with same sized models.

Published as a conference paper at COLM 2025
Efficient Process Reward Model Training via Active Learning
Keyu Duan
Zichen Liu
Xin Mao
Tianyu Pang
Changyu Chen
Qiguang Chen
Michael Qizhe Shieh
Longxu Dou
Sea Lab
National University of Singapore
Singapore Management University
Abstract
Process Reward Models (PRMs) provide step-level supervision to large language models (LLMs), but scaling up training data annotation remains challenging for both humans and LLMs. To address this limitation, we propose an active learning approach, ACTPRM, which proactively selects the most uncertain samples for training, substantially reducing labeling costs. During training, we the PRM to estimate uncertainty after the forward pass, retaining only highly uncertain data. A capable yet costly reasoning model then labels this data. Then we compute the loss w.r.t. the labels and update the PRM's weights. We compare ACTPRM vs. vanilla fine-tuning, on pool-based active learning setting, demonstrating that ACTPRM reduce 50% annotation, but achieving the comparable or even better performance. Beyond annotation efficiency, we further advance the actively trained PRM by filtering over 1M+ math reasoning trajectories with ACTPRM, retaining 60% of the data. A subsequent training on this selected dataset yields new state-of-the-at (SOTA) PRM on ProceesBench (75.0%) and PRMBench (65.5%) compared with same sized models! 1.
1 Introduction
Recently, Large Language Models (LLMs) (DeepSeek-AI et al., 2025; Yang et al., 2024; OpenAl et al., 2024b) have shown remarkable advances in mathemat- ical reasoning, yet they can make mistakes during thain-of-thought (CoT) reasoning despite correct final answers(Zheng et al., 2024). To address this challenge, process reward models (Lightman Wang et al., 2024; Zhang et al., 2025) were proposed, aiming to identify process errors and provide finer-grained supervision of the training process.
0.76
0,750
4.8x Cost
0.743
0.74
17.3x Cost
0.735
Average Fl Score
0.72
Model
0.70
al., 2023;
AcPPRM
UniverslPRM
0.68
Qwen25-Mnt-PRNGG
2
2
2
Est. Annotation Cost (Gen. Tokens)
The main challenge in training Process Re- ward Models (PRMs) lies in obtaining fine- grained step-level annotations, which re- main prohibitively expensive. Lightman et al. (2023) pioneered PRM training by em- ploying human experts to label 75K ques- tions at the step level. While their approach
Figure 1:
Average F1 score on Process- Bench (Zheng et al., 2024) versus estimated annotation cost. ACTPRM outperforms prior SOTA models while requiring merely 20% of the annotation costs.
achieved high-quality results (reaching 57.5% on ProcessBench (Zheng et al., 2024)), it does not scale automatically due to the heavy reliance on manual annotation. To reduce human
Work done during the internship in Sea AI Lab. Email: kds.n
*Corresponding authors.
'The code is available at https:/ https:///ithubb.omm siggghtu..co/sslllg///aaaee/e

---

Published as a conference paper at COLM 2025
efforts, Monte Carlo (M.C.) Estimation Methods (Wang et al., 2024; Wei et al., 2024; Luo et al., 2024) were proposed. However, these approaches come with high computational costs (massive rollouts are required for accurate estimation) and struggle to accurately identify the first error step (Zheng et al., 2024). To address this challenge, Owwnn.5-Math-PP.
(Zhang leveraging L.Ms to detect the first error step and filter out unreliable M.C. labels. It significantly boosts the performance of PRR on both ProcessBench (Zheng et al., 2024) and PRRMBench (Song et al., 2025). More recently, UniversalPRM (Tan et al., 2025) relies solely on LLM-as-Judge with ensemble prompting (via majority voting), achieving new SOTA performance on ProcessBench within the same model size. However, the annotation costs are still considerable. We estimate the labeling costs of Qwen2.5-Math-D
et al., 2025) proposed using LLM-as-Judge
(Zhang et al., 2025) and UniversalPRM (Tan et al., 2025) and illustrate them in Figure 1. It shows that Owen-Vath-PRRM7B5 and UniversalPRM consume over 234 and 23 generated tokens, respectively. Refer to Appendix C for estimation strategy.
To reduce annotation costs, we propose ACTPRM, which uses a trained ensemble PRM to identify and select uncertain data for annotation by a high-capability reasoning model. Our approach trains a PRM with ensemble heads for uncertainty estimation. For each reasoning step, we compute the mean and standard deviation of ensemble predictions, identifying uncertain steps when prediction confidence is outside threshold 1 6pred < < 6pred or variation exceeds 6add- We consider a CoT trajectory uncertain if any step up to and including the first predicted error meets these criteria. By annotating only uncertain data and training exclusively on this subset, we significantly reduce labeling costs while maintaining PRM performance.
To validate the effectiveness and efficiency of ACTPRM, we conducted comprehensive experiments in multiple settings:
Pool-based Evaluation (Section 5.1): Using 100K labeled samples, ACTPRM achieved performance comparable to full-data tuning while reducing annotation costs by 50%. It consistently outperformed random selection under identical budget constraints.
One-shot Active Learning (Section 5.2): Starting with our pool-based model, we applied ACTPRM to select uncertain samples from 1M+ unlabeled CoT trajectories from NuminaMath (Li et al., 2024). After annotation and fine-tuning, we achieved new SOTA performance of 75.0% on ProcessBench. As shown in Figure 1, ACTPRM surpasses prior SOTA models with significantly lower costs- outperforming Uni- versalPRM (Tan et al., 2025) by 0.7% using only 20% of its annotation budget and exceeding Qwen2.5-Math-PRMM.
by 1.5% with just 6% of its annotation budget.
Our contributions are summarized as follows: We propose an uncertainty-awaar active learning approach ACTPRM for PRM training that selectively annotates informative rea- soning steps using ensemble-based uncertainty estimation, significantly reducing labeling costs while maintaining performance. @ ACTPRM achieves state-of-the-art results (75.0% on BrocessBench, 65.5% on PRMBench) while requiring only 20% of the annotation budget compared to prior SOTA method UniversalPRM. We release all trained models, datasets, and code to ensure reproducibility and facilitate community adoption.
2 Preliminaries
2.1 Process Reward Models
Problem Formulation. Given a math problem 9 and a corresponding solution trajectory
$ = [$1, .,, where s; denotes i-th step., we require a PRM to identify the correctness of each step until a wrong step is identified. We only label the steps up to and including the first error step following prior works (Lightman et al., 2023; Zheng et al., 2024), since the afterward steps are genuinely difficult to define their correctness. As result, in practice the labels for a solution trajectory are either [1, 1,,,, 1, or [,,,, 0). Then a PRM could be trained using the typical BCE loss:
2

---

Published as a conference paper at COLM 2025
|s| C LCC (s) |s| - yi log (og(Po 9)) + +1 yı) -
1
(1)
where Po is the PRM parameterized by 0 and [1j denotes the steps before 8,. When using PRM for inference, we set threshold å (usually 0.5) to identify the first step that has correctness probability Pa (a(5)]5[a).99 less than d.
PRM Implementation Details. A typical PRM is built upon a pretrained generative LLM by replacing the causal language model head with a binary classification head that outputs the probability of correctness at corresponding token position. In practice, we solely need the prediction at the end of each reasoning step and thus prediction mask is used to mask out the prediction at the other positions.
2.2 Uncertainty Estimation for Classification
Aleatoric Uncertainty. As aforementioned, typical PRM Pa is trained as a binary clas- sification task. The simplest way to measure the uncertainty for it is to use aleatoric uncertainty (Valdenegro-Tows Toro Saromo, 2022):?.
Aleatoric Uncertainty (s(s;) log (s;).
(2)
Epistemic Uncertainty. In addition, ensemble of models is also a common way to estimate epistemic uncertainty (Valdenegro-Tore & Saromo, 2022) by quantifying the disagreement among ensemble models. For example, Liang et al. (2022); Gleave & Irving (2022) use an ensemble of reward models to estimate uncertainty in preference learning. for process reward modeling, we could leverage the standard deviation of the ensemble predictions as the uncertainty estimation:
Epistemic Uncertainty Var((Po(s:))),
(3)
where {Po} } is set of models. It is worth noting that employing an ensemble of heads common strategy mitigate computational costs. We
built upon shared backbone empirically study the combination of aleatoric and epistemic uncertainty and find that they are complementary to each other. Experimental results are shown in Section 5.1.
3 Related Work
Active Learning and Uncertainty Estimation. Active learning has been widely explored in the alignment of L.MMs. Several studies adopt an online bandit formulation, leveraging uncertainty-aware reward models (RMs) for active exploration in response selection (Mehta et al., 2023; Dwaracherla al., 2024; Liu et al., 2024; Melo et al., 2024; Gleave Irving, 2022). For instance, Mehta et al. (2023) and Dwaracherla et al. (2024) use ensemble-based LLM heads to estimate epistemic uncertainty, prioritizing data most informative for preference learning. Similarly, Melo et al. (2024) propose acquisition function combining both entropy (aleatoric uncertainty) and epistemic uncertainty. Our work builds on these ap- proaches, empirically evaluating the role of both uncertainty types in the context of process reward modeling. Beyond active learning, ensemble methods as those in Coste et al. (2024)-have also proven effective mitigating reward hacking (Amodei et al., 2016).
Process Reward Models. Different from outcome rewards (e.g., verifiable re- wards (DeepSeek-AI al., 2025) for mathematical reasoning problems) that assign rewards based on the final outcome, process rewards are assigned based on the intermediate steps of the problem-solving process. For question and corresponding solution with several steps, PRM provides finer-grained rewards for each step. Til current stage, process reward mod- eling contains two categories: (i) Process Reward as Q-values and (ii) Process Reward as Judger. The former one (Wang et al., 2024; Luo et al., 2024; Wei et al., 2024; Li & Li, 2024) regards the process reward as the Q-values of the steps that estimate the probability of the policy model
For simplicity, use Po(s;) the aleatoric probability for the i-th step in the solution trajectory, where the full representation is pp(5i1s1--99) as in Equation
3

---

Published as a conference paper at COLM 2025
Algorithm PRM Active Learning with Cold Start.
1: // The difference with full data tuning is colored.
Input: Ensemble PRM Pg, dataset 'D ((q,s)), uncertainty thresholds à pred and dstd gener-
ative LLM M, batch size B, learning rate !/
for C D do
3: Pe (B) + Forward
4:
B= = [1}
5:
for (9.s) do
6:
if u
(s) V V2g then
Equation 5
7:
B B+ B-Bu((
end if end for
9:
10:
YB f labeling( B)
Labeling using generative LLM Equation 4
11:
C B I (8))) (s,yy)
12:
VoL - Backward(()
13:
0 < 0 - 7 VoL
14:
end for
Output: Pg
to reach the final correct answer. Specifically, they leverage the policy model that generates the solution steps to do Monte Carlo Estimation for each step. The estimated Q-values are used as the process rewards. However, recent works (Zhang et al., 2025; Zheng et al., 2024) show that this kind of process reward modeling suffers from identifying the process errors because it highly depends on the policy model and has large bias with the ground truth distribution. In contrast, the latter one (Lightman et al., 2023; Zhang al., 2025) regards the process reward model as a proxy for identifying the intermediate process errors and the corresponding trained model achieves better performance on several benchmarks (Zheng et al., 2024; Song et al., 2025). In this work, we follow the latter one and regard the process reward a judge that tries identify the first error steps in the solutions if any. In addition, there are other works related to PRM. For example, Yuan et al. (2024) tries to train PRM with fashion outcome reward modeling (ORM). Cheng et al. (2025); Cui al. (2025) proposed RL training frameworks that integrate PRM as finer-grained supervison.
Efficient Process Reward Labeling via Active Learning
Labeling the process rewards for a large-scale dataset is very expensive as it either requires human experts to annotate the correctness of each step for each solution as the previous work (Lightman et al., 2023) leverages highly capable generative models to imitate human experts (Zhang et al., 2025). Even though the latter one automated, is still resource-consuming since the test time scales up with the difficulty of math problems.
To mitigate this issue, we propose to leverage active learning to make the PRM proactively select the data that is most informative to train on. Specifically, we train PRM with ensemble heads to enable uncertainty estimation following Liang et al. (2022); Gleave & Irving (2022). As shown in Algorithm 1, We forward the data candidates to the ensemble PRM (line 3) and estimate the prediction uncertainty for each data point (line 5-6). Then omit the data that the ensemble PRM is confident about (line 7) and only label the other retained data with generative reasoning LLM (line 10). Then, we only backpropagate from the loss of labeled data (line 11). By doing so, we could considerably reduce the labeling cost while maintaining the PRM performance. Now we introduce our two key differences with the original finetuning: ensemble PRM training and uncertain data selection.
Ensemble PRM Training. In this work, we use ensemble of PRMs to estimate the epistemic uncertainty following Gleave & Irving (2022); Liang al. (2022). Specifically, we use shared LLM backbone and build multiple binary classification heads on top of it. In our training, the diversity of ensemble models is ensured by two ways: (i) the random initialization of the head layers and (ii) diversity regularization term (Dwaracherla et al., 2024): div = A Σn= |1 Pimitl l2, where (Φi) are the parameters of the ensemble heads and n is the number of ensemble heads. It is I.2 term penalizing the distance between
4

---

Published as a conference paper at COLM 2025
Listing 1: Pseudo code of uncertainty estimation.
Compute ensemble predictions (num_ensemble, num_step)
score 11m (1m(inut__ids)
means, stds score. mean (8), score.std(C # Per-step statistics Equ. left cristemic_uncertainn
5
stds std_threshold
Equ. right
alearotic.uncertainty (means <= pred_ & (means >= 1
pred_threshold)
Alg. line
uncertainty = any any(eppistemlc_uneertainn
alearotic
the ensemble head parameters and their initial parameters. Our training objective for the ensemble PRM is therefore formulated as follows
C(y,s) = 1 _ CBCE (BCE(c, s(0, Φ¹)
Al10 Pmmallz).
where 8 denotes the backbone parameters and LBCE is from Equation 1, that computes the loss for certain head.
Uncertain Data Selection. Considering a batch of data candidates D {{q,s)), we first for- ward the data to the ensemble PRM Pa to get the ensemble predictions (D) "R" x|s|, for each data (q.s) E 'D, we could determine the hard-value aleatoric and epistemic uncer- tainty with pre-set thresholds. Briefly, the aleatoric (or epistemic) uncertainty is defined uncertainty occurs at any step prior to the first predicted error step; otherwise, it is 0. A formal definition as follows:
E(s)
E(s)
uglea (s)
V max (H(PP(si)), (((((s))) < <6pred) uopis i
(s)
V (o(Po(si)) dsta). i
where u(-) and (-) are the mean and standard deviation of the ensemble predictions among ensemble heads and V denotes the logical 'OR' operation. Moreover, the (s) denotes the first error step the solution trajectory s, defined as {(s) min μ(s;) 8), where the threshold for the correctness, typically set to 0.5. This is because we only care about the correctness of the steps before the first eтror step since it is genuinely difficult to define the correctness of the steps afterwards. For further illustration, we also provide the pseudo code of the uncertainty estimation in Listing 1. By following the uncertainty estimation
strategy, we retain the data that satisfies either Lggee or Lepis as D. Then we could leverage expensive generative LLMs as judger (Zheng et al., 2024) to label the retained data in D.
5 Experiments
In Section 5.1, we first validate ACTPRM in a pool-based active learning setting using 100K labeled samples, including ablation studies on our uncertainty estimation strategy. Based on the optimal configuration, we then scale up to 1M unlabeled samples in Section 5.2, further proving our pipeline's 's efficiency and effectiveness.
5.1 Pool-Based Active Learning
5.1.1
Experimental Settings
To evaluate our active learning e effectiveness, offline setting where ACTPRM iteratively selects the most informative examples from large unlabeled pool as detailed in Algorithm 1. We establish strong baseline by comparing
first conduct experiments in

---

(c) Ablation: number of Published as a conference paper at COLM 2025
0,70
0.67.
Average Soore
0.65
0.60
0.65
0.55
Model
0,005,
F1 Score
0.60
0.005)
0.2
Budget
0.55
F1 Score
0.690
Model
Budget
0.635
Full Tuning
0.600
(a)
Budget
#Heada
Figure 2: (a) Comparison the average Fl score on ProcessBench between ACTPRM and random selection, plotted against the normalized budget positively correlated the number of labeled data instances consumed. The remi-transparent points represent all results in grid searching w.r.t. 6 pred and ōstd For the highlighted ACTPRM curve in the figure, pred 0.95 and ded 0.005. (b) Ablation: uncertainty estimation strategies. ensemble PRM heads.
against full data tuning, where model is trained the complete dataset labeled by single annotator. It is worth noting that as our data is randomly shuffled, the performance of full data tuning at intermediate training steps is essentially equivalent to the performance of random selection with the corresponding budget.
Evaluation Benchmark. We utilize ProcessBench (Zheng et al., 2024) to evaluate the ef- fectiveness of PRMs. The test data in ProcessBench contains intermediate step errors and requires the PRM to identify the first error step. ProcessBench contains four subsets, and report the average Fl score following the original work.
Models. We train ACTPRM based on Diwen2.--Mtth77--naee
Training Dataset. For dataset construction, we randomly select 100K data from Numi- namath (Li et al., 2024) dataset after decontamination against the ProcessBench (Zheng et al., 2024) and PRMBench (Song et al., 2025). We leverage Qween225-Natth-7B--nhhh to generate CoT reasoning trajectories for the selected data and further use QwQ-32B judge to annotate the process correctness for all trajectories following Zhang et al. (2025). For completeness, we provide the prompt template Appendix A.
5.1.2 Experimental Results
ACTPRM achieves comparable performance while reducing annotation costs by 50%. We compare ACTPRM with full data tuning across normalized budget, as illustrated in Figure 2 (a). The results demonstrate that ACTPRM achieves an average F1 score of 0.673 on ProcessBench, matching baseline performance while using only half the annotation budget. Furthermore, ACTPRM consistently outperforms random selection under the same budget constraints. Notably, at 50% budget, ACTPRM surpasses random selection by significant margin of 3.3%. at the end of pool-based active training, ACTPRM achieves better performance of 0.680 on ProcessBench while consuming solely 62.5% budget.
ACTPRM Consistently Outperforms Random Selection Under Diverse 6pred and & As shown in Figure 2 (a), the semi-transparent blue points represent all results of grid searching over 6pred E {0.9, 0.95, 0.97) and & {0.01, 0.005, 0.002, 0.001 One can see that most blue points are above the baseline (gray line) with the same budget, further demonstrating the effectiveness and robustness of ACTPRM.

---

Published as a conference paper at COLM 2025
Ablation Study on Uncertainty Estimation Strategies. We conduct ablation study uncertainty estimation strategies, i.e. using epistemic and aleatoric uncertainty, We selected the best setting (Óstd 0.005, pree = 0.95) searched by grid search as in Figure 2 and ablates epistemic and aleatoric uncertainty by setting std inf and pred 0.5, respectively. As shown illustrated in Figure (b), solely use either epistemic or aleatoric uncertainty under- performs using both, indicating that epistemic and aleatoric uncertainty are complementary to each other.
Ablation Study on Number of Heads for Ensemble PRM. The number of heads for ensemble PRM controls how accurate our estimated epistemic uncertainty is. To find the trade-off between good estimation and computational overhead, we conduct an ablation study regarding and show the results in Figure (c), where we only consider epistemic uncertainty by setting sst = 0.005, 6 pred = 0.5 and report the averaged results with 3 runs. We empirically find that the performance continually grows with the number of heads and converges at about 32.
5.2 Achieving New SOTA Performance on ProcessBench (75.0%) with Solely 6% Annotation Cost.
Obtaining high-quality process supervision la-
Est. Annotation Cost (Gen. Tokai
bels is costly. To demonstrate the efficiency of ACTPRM, we evaluate in one-shot active learning setting. Starting with the model trained in Section 5.1, we select the most uncertain sam-
17:3x
2.0c+10
15,9k
1.5c+10
ples from over 1M+ unlabeled examples and an-
1.0e-10
notate them using a powerful reasoning model.
5.0c=09
estimated labeling costs with those of other real-world datasets for train- ing PRMs, including MathShepherd (Wang et al., 2024), Consensus Filtering (Zhang et al., 2025), and Ensemble Prompting (Tan al., 2025). Since the training data for Consensus Filtering is not publicly available, we estimate costs based on our data statistics. We introduce our estimation strategy in Appendix C.
Figure 3 compares
0.0e+00
Ensmbll
Prompting
Shepherd
Filtering
Figure 3: Estimated annotation costs (gen- erated tokens) comparison between ACT- PRM and popular methods, including Ensemble Prompting (Tan et al., 2025), MathShepherd (Wang et al., 2024) and Consensus Filtering (Zhang et al., 2025).
Training a 1aann---Math-----nttrae on our data, Ensemble Prompting data, MathShepherd data, and Consensus Filtering data yields and Qwen2.5-Math-PRRM in Table 1. We evaluated the performance of models trained on this labeled data on both ProcessBench and PRMBench benchmarks.
ACTPRM, UniversalPRM,
5.2.1 Experimental Settings
Data Filtering with ACTPRM. We used Qwwen2.5MMahh-77Annstuu and Qwen2.5-Math-72B-
Instruct to collect over 1 million (1,061,763) Chain-of-Thought (COT) trajectories from the Numinamath problem set (Li et al., 2024), after decontamination against the test benchmarks.
ACTPRM was then applied to filter out high-confidence (spred 0.95 and ó 0.005 following Section 5.1) data instances that
unnecessary for training, retaining the remaining data for labeling and training. This process resulted in final dataset of 563,030 PRM data points labeled by QwQ-32B, reducing annotation costs by 47.0%.
Models. Obtaining the dataset, we continually train our ACTPRM in Section 5.1 on our filtered dataset. In addition, we empirically find that our retrained data generally useful to other PRMs. Specifically, we also continually train Qwen2.5-Math-PPR.
(the previous SOTA model on ProcessBench) on our constructed data. The resultant model is named ACTPRM-X3.
3X stands for extended version.
7

---

Published as conference paper at COLM 2025
Olympiad Bench
Models
GSM8K
MATH
OmniMath
Average F1
LLM-as-judge
ol-Mini
0.932
0.889
0.872
0.824
0.879
Deepseck-Rl-Distill- QwQ-32B
0.817
0.739
0.659
0.585
0.700
0,871
0.834
0.787
0.771
0,816
Process Reward Models (72B)
Owenn255Matth-PRM-220
0.873
0.806
0.743
0.711
0.783
Process Reward Models (7B+)
Math-ShhepherdPPRM--2 -yven22-Matt-7--Matthhhhpppp
0.479
0.295
0.248
0.238
0.315
0.625
0.316
0.137
0.077
0,289
FurusPRM-Stage22
0.473
0.357
0.212
0.209
0.313
Oween2.5MMatt-7B-PRRMBBB
0.683
0.626
0,507
0.443
0.565
Ensemble-PRM-PRMBON (ours)
0.705
0.630
0.472
0,433
0.560
PURE-PRM-7B
0.690
0.665
0.484
0.459
0.575
Owen2.5-Matt-PRMM-77
0.824
0.776
0,675
0.663
0.735
Universal-PRM
0.858
0.777
0.676
0.664
0.743
ACTPRM (ours)
0.816
0.798
0.714
0.670
0.750
ACTPRM-X (ours)
0.827
0.820
0.720
0.673
0.760
Table 1: Performance comparison on ProcessBench. We report the results in the same calcu- lation method with Process ddenotes the results from Qwen PRM's report (Zhang et al., 2025).
Benchmarks. We use ProcessBench (Zheng et al., 2024) and PRMBench (Song et al., 2025) to evaluate the effectiveness our trained model. Different from ProcessBench that collects intermediate errors from real-world generative models, PRMBench heuristically builds intermediate errors by manipulating correct steps.
Baselines. We compare with the following PRMs: 1 Quen2.5-Math-PRM.. (Zhang al., 2025): This model uses consensus filtering for labeling. It labels 860K data twice using two methods (LLM-as-judge [Zheng al., 2024] and Mathshepherd [Wang et al., 2024]) and filters out 40% of the data where the labels disagree. @ A Swwen..--Mtt-baseee PRM trained on PRMS00K using a two-stage strategy: warming the PRM head and then fine-tuning the entire model. O: EurusPRM-Stagel (Cui et al., 2025): A PRM resulting from the Implicit PRM approach (Yuan et al., 2024), which derives process rewards from an ORM. @ Universal-PRM (Tan et al., 2025): A Qwen2.5-Math- based model trained with data augmentation techniques like ensemble prompting and reverse verification. @ Math--Seppher- PRM--7 (Wang et al., 2024): PRM trained on process labels that estimates hard Q-values for the policy model. @ -uwen.5-Matt-77B-A Shepherd (Zhang et al., 2025): PRM trained 860K data labeled using MathShepherd. Ensemble-PRMMPPMMSOO (ours): a model with ensemble heads trained by ourselves on PRM800K without active learning.
(Cheng et al., 2025):
5.2.2 Experimental Results
ACTPRM and ACTPRM-X achieve new SOTA performance on ProcessBench compared with same size models. The evaluation results on BrochssBench are shown in Table 1. ACTPRM achieves an average F1 score of 0.750, outperforming Swwen2.--Vatt-PRM--75 by margin of 1.5%. Furthermore, ACTPRM-X training based on Sowen25-MMtthPRRM-77 achieves new SOTA performance on ProcessBench with an average F1 of 0.760, outper- forming the second-place model (Universal-PRAe with margin of 1.7% and improve the performance of -aa.nn..--att-PMM-M--- by a significant margin of 2.5%.
QwQ-32B (our PRM label annotator) outperforms all PRMs on ProcessBench. As shown in Table 1, QwQB32B outperforms all PRMs including 72B models. It indicates the reliability of utilizing QwQ-32B PRM label annotator as it provides high empirical upperbound for the training PRMs.
ACTPRM-X achieves new SOTA performance PRMBench, on-par with GPT-4o. We further test our models on PRMBench and show the results in Table 2. As on the leaderboard, ACTPRM achieves the best performance within 7B PRMs and ACTPRM-X achieves new
8

---

Published as a conference paper at COLM 2025
Models
Simlicity
Soundness
Sensitivity
Average
LLM-as-judge
Gemin-200thinkingeex---
0.662
0.718
0.753
0.688
ol-mini
0.646
0.721
0.755
0.688
GPT-4o
0.597
0.709
0.758
0.668
6
Gemfni-2.0-flas--exp
0.627
0.673
0.754
0.660
Process Reward Models (72B)
3
(wen-2.5-Math-PRRM-22
0.546
0.739
0.770
0.682
Process Reward Models (7B+)
0.521
0.710
0.755
0.655
Pure-PRM-7B
0.522
0.702
0.758
0.653
7
ACTPRM (ours)
0.536
0.713
0.752
0.655
5
ACTPRM-X (ours)
0.545
0.727
0.756
0.667
Table 2: Performance comparison on PRMBench. All results of the other models are from
the
0.73
Selected Randonly Selecced
0.72
Average F1 Score
Trahning Loss
0.70
0.69
0.10
ActPRM Selected
Randomly Selected
0.67
1000
1500
2000
Step
Step
Figure 4: ProcessBench performance (left) and training loss (right): ActPRM v.s. random data selection 1M NuminaMath Rollouts.
SOTA performance (0.667), outperforming the other models by large margin of at least 1.2% and on-par with GPT-4o (OpenAl et al., 2024a).
5.2.3 Comparative Experiment with Random Selection
A potential concern is that while ACTPRM achieves state-of-the-art (SOTA) performance on several benchmarks, this success might attributed solely the high quality of our collected data pool, rather than the method itself. To address this, we conducted compar- ative study with random selection. Specifically, we randomly selected 256K data points from our retained dataset as the experimental group. For the control group, randomly selected the same number of data points from the entire data pool (over 1M) and used the same annotator to label any unlabeled data (i.e., data not in the retained set). We then continually train ACTPRM checkpoint, as in Sec 5.1, on both datasets. The results, including performance ProcessBench and training loss, shown in Figure 4.
ACTPRM outperforms random selection of the same amount of data. As illustrated in Figure (left), the model trained on data selected by ACTPRM consistently achieves significantly better results than the model trained on randomly selected data. To further validate this, we compare their training losses in Figure (right). The model trained on AcTPRM-selected data exhibits a consistently higher training loss, with margin of 0.05, suggesting that the data selected by ACTPRM is more challenging and informative, thereby enhancing the learning process.
Conclusion and Future Work
In this work, we address the high annotation costs associated with training Process Reward Models (PRMs) by proposing ACTPRM, uncertainty-aware active learning framework

---

Published as a conference paper at COLM 2025
that selectively annotates the most informative reasoning steps. By leveraging ensemble PRM to estimate uncertainty and strategically labeling only uncertain data, ACTPRM sig- nificantly reduces annotation costs while maintaining competitive performance. Extensive experiments demonstrate that ACTPRM achieves a new state-of-the-at (75.0% on Process- Bench) with merely at most 20% of the labeling budget required by prior methods. Our results highlight the potential of efficient data selection for scalable PRM training, and we commit to releasing all models, datasets, and code foster further research in this direction.
To further enhance PRM's performance, several promising directions can be explored. First, leveraging larger base models and more advanced LLM judges (e.g,, O1-mini) could yield significant improvements. Second, implementing the framework in an online setting would ultimately enable PRM iteratively refine its performance through active learning. Addi- tionally, integrating online PRM training with reinforcement learning frameworks- such as actor-critic methods presents an exciting avenue for research.
Acknowledgement
This project was partly supported by the MOE AcRF Tier 1 grant with grant number 251RE52514.
References
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety, 2016. URL / /1666 06565.
Jie Cheng, Lijun Li, Gang Xiong, Jing Shao, and Yisheng Lv. Stop gamma decay: Min-form credit assignment is all process reward model needs for reasoning, 2025. Notion Blog.
Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward Model Ensembles Help Mitigate Overoptimization, 2024.
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. Xrv preprint ar XrXo:2502.001456 2025.
DeepSeek-Al Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang. Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo,
10

---

Published as a conference paper at COLM 2025
Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijja Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning, 2025. 12948.
URL http://///axiiv.og//////////////
Vikranth Dwaracherla, Seyed Mohammad Asghari, Botao Hao, and Benjamin Van Roy. Efficient exploration for llms. In International Conference on Machine Learning, 2024.
Adam Gleave and Geoffrey Irving. Uncertainty Estimation for Language Reward Models, 2022.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021. URL https: wtgp http:://ammii.orr//b////9...2. 03874.
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. Nu- minamath. project-numina/atie nuwina/aaaaaaaan aimo httpr//ssrrrrieaaoooo///a////pre
(https://gi thub. com/ _aatase... pdf), 2024.
project
Wendi Li and Yixuan Li. Process reward model with q-value rankings. arXiv preprint (xXi::240011128, 2024.
Xinran Liang, Katherine Shu, Kimin Lee, and Pieter Abbeel. Reward Uncertainty for Exploration in Preference-based Reinforcement Learning, 2022.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, llya Sutskever, and Karl Cobbe. Let's Verify Step by Step, 2023.
Zichen Liu, Changyu Chen, Chao Du, Wee Sun Lee, and Min Lin. Sample-efficient alignment for Iims, 2024. URL
01493.
Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Meiqi Guo, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, and Abhinav Rastogi. Improve Mathematical Reasoning in Language Models by Automated Process Supervision, 2024.
Viraj Mehta, Vikramjeet Das, Ojash Neopane, Yijia Dai, Ilija Bogunovic, Jeff Schneider, and Willie Neiswanger. Sample efficient reinforcement learning from human feedback via active exploration. Xiv preprint drXiv:2312.00267, 2023.
Luckeciano C. Melo, Panagiotis Tigas, Alessandro Abate, and Yarin Gal. Deep bayesian active learning for preference modeling in large language models, 2024. URL https: //arxiv.org//bbe. 2042//// 10023.
OpenAI, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Madry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Paino, Alex Renzin, Alex Tachard Passos, Alexander Kirillov, Alexi Christakis, Alexis Conneau, Ali Kamali, Allan Jabri, Allison Moyer, Allison Tam, Amadou Crookes, Amin Tootoochian, Amin Tootoonchian, Ananya Kumar, Andrea Vallone, An- drej Karpathy, Andrew Braunstein, Andrew Cann, Andrew Codispoti, Andrew Galu, Andrew Kondrich, Andrew Tulloch, Andrey Mishchenko, Angela Back, Angela Jiang, Antoine Pelisse, Antonia Woodford, Anuj Gosalia, Arka Dhar, Ashley Pantuliano, Avi Nayak, Avital Oliver, Barret Zoph, Behrooz Ghorbani, Ben Leimberger, Ben Rossen, Ben Sokolowsky, Ben Wang, Benjamín Zweig, Beth Hoover, Blake Samic, Bob McGrew, Bobby Spero, Bogo Giertler, Bowen Cheng, Brad Lightcap, Brandon Walkin, Brendan Quinn, Brian Guarraci, Brian Hsu, Bright Kellogg, Brydon Eastman, Camillo Lugaresi, Carroll Wainwright, Cary Bassin, Cary Hudson, Casey Chu, Chad Nelson, Chak Li, Chan Jun
11

---

Published as a conference paper at COLM 2025
Shern, Channing Conger, Charlotte Barette, Chelsea Voss, Chen Ding, Cheng Lu, Chong Zhang, Chris Beaumont, Chris Hallacy, Chris Koch, Christian Gibson, Christina Kim, Christine Choi, Christine McLeavey, Christopher Hesse, Claudia Fischer, Clemens Win- ter, Coley Czarnecki, Colin Jarvis, Colin Wei, Constantin Koumouzelis, Dane Sherburn, Daniel Kappler, Daniel Levin, Daniel Levy, David Carr, David Farhi, David Mely, David Robinson, David Sasaki, Denny Jin, Dev Valladares, Dimitris Tsipras, Doug Li, Duc Phong Nguyen, Duncan Findlay, Edede Oiwoh, Edmund Wong, Ehsan Asdar, Elizabeth Proehl, Elizabeth Yang, Eric Antonow, Eric Kramer, Eric Peterson, Eric Sigler, Eric Wallace, Eugene Brevdo, Evan Mays, Farzad Khorasani, Felipe Petroski Such, Filippo Raso, Francis Zhang, Fred von Lohmann, Freddie Sulit, Gabriel Goh, Gene Oden, Geoff Salmon, Giulio Starace, Greg Brockman, Hadi Salman, Haiming Bao, Haitang Hu, Hannah Wong, Haoyu Wang, Heather Schmidt, Heather Whitney, Heewoo Jun, Hendrik Kirchner, Henrique Ponde de Oliveira Pinto, Hongyu Ren, Huiwen Chang, Hyung Won Chung, Ian Kivlichan, Ian O'Connell, Ian O'Connell, Ian Osband, Ian Silber, Ian Sohl, Ibrahim Okuyucu, Ikai Lan, Tlya Kostrikov, Sutskever, Ingmar Kanitscheider, Ishaan Gulrajani, Jacob Coxon, Jacob Menick, Jakub Pachocki, James Aung, James Betker, James Crooks, James Lennon, Jamie Kiros, Jan Leike, Jane Park, Jason Kwon, Jason Phang, Jason Teplitz, Jason Wei, Jason Wolfe, Jay Chen, Jeff Harris, Jenia Varavva, Jessica Gan Lee, Jessica Shieh, Ji Lin, Jiahui Yu, Jiayi Weng, Jie Tang, Jieqi Yu, Joanne Jang, Joaquin Quinonero Candela, Joe Beutler, Joe Landers, Joel Parish, Johannes Heidecke, John Schulman, Jonathan Lachman, Jonathan McKay, Jonathan Uesato, Jonathan Ward, Jong Wook Kim, Joost Huizinga, Jordan Sitkin, Jos Kraaijeveld, Josh Gross, Josh Kaplan, Josh Snyder, Joshua Achiam, Joy Jiao, Joyce Lee, Juntang Zhuang, Justyn Harriman, Kai Fricke, Kai Hayashi, Karan Singhal, Katy Shi, Kavin Karthik, Kayla Wood, Kendra Rimbach, Kenny Hsu, Kenny Nguyen, Keren Gu-Lemberg, Kevin Button, Kevin Liu, Kiel Howe, Krithika Muthukumar, Kyle Luther, Lama Ahmad, Larry Kai, Lauren Itow, Lauren Workman, Leher Pathak, Leo Chen, Li Jing, Lia Guy, Liam Fedus, Liang Zhou, Lien Mamitsuka, Lilian Weng, Lindsay McCallum, Lindsey Held, Long Ouyang, Louis Feuvrier, Lu Zhang, Lukas Kondraciuk, Lukasz Kaiser, Luke Hewitt, Luke Metz, Lyric Doshi, Mada Aflak, Maddie Simens, Madelaine Boyd, Madeleine Thompson, Marat Dukhan, Mark Chen, Mark Gray, Mark Hudnall, Marvin Zhang, Marwan Aljubeh, Mateusz Litwin, Matthew Zeng, Max Johnson, Maya Shetty, Mayank Gupta, Meghan Shah, Mehmet Yatbaz, Meng Jia Yang, Mengchao Zhong, Mia Glaese, Mianna Chen, Michael Janner, Michael Mampe, Michael Petrov, Michael Wu, Michele Wang, Michelle Fradin, Michelle Pokrass, Miguel Castro, Miguel Oom Temudo de Castro, Mikhail Pavlov, Miles Brundage, Miles Wang, Minal Khan, Mira Murati, Mo Bavarian, Molly Lin, Murat Yesildal, Nacho Soto, Natalia Gimelshein, Natalie Cone, Natalie Staudacher, Natalie Summers, Natan LaFontaine, Neil Chowdhury, Nick Ryder, Nick Stathas, Nick Turley, Nik Tezak, Niko Felix, Nithanth Kudige, Nitish Keskar, Noah Deutsch, Noel Bundick, Nora Puckett, Ofir Nachum, Ola Okelola, Oleg Boiko, Oleg Murk, Oliver Jaffe, Olivia Watkins, Olivier Godement, Owen Campbell-Moore, Patrick Chao, Paul McMillan, Pavel Belov, Peng Su, Peter Bak, Peter Bakkum, Peter Deng, Peter Dolan, Peter Hoeschele, Peter Welinder, Phil Tillet, Philip Pronin, Philippe Tillet, Prafulla Dhariwal, Qiming Yuan, Rachel Dias, Rachel Lim, Rahul Arora, Rajan Troll, Randall Lin, Rapha Gontijo Lopes, Raul Puri, Reah Miyara, Reimar Leike, Renaud Gaubert, Reza Zamani, Ricky Wang, Rob Donnelly, Rob Honsby, Rocky Smith, Rohan Sahai, Rohit Ramchandani, Romain Huet, Rory Carmichael, Rowan Zellers, Roy Chen, Ruby Chen, Ruslan Nigmatullin, Ryan Cheu, Saachi Jain, Sam Altman, Sam Schoenholz, Sam Toizer, Samuel Miserendino, Sandhini Agarwal, Sara Culver, Scott Ethersmith, Scott Gray, Sean Grove, Sean Metzger, Shamez Hermani, Shantanu Jain, Shengjia Zhao, Sherwin Wu, Shino Jomoto, Shirong Wu, Shuaiqi, Xia, Sonia Phene, Spencer Papay, Srinivas Narayanan, Steve Coffey, Steve Lee, Stewart Hall, Suchir Balaji, Tal Broda, Tal Stramer, Tao Xu, Tarun Gogineni, Taya Christianson, Ted Sanders, Tejal Patwardhan, Thomas Cunninghman, Thomas Degry. Degry, Thomas Dimson, Thomas Raoux, Thomas Shadwell, Tianhao Zheng, Todd Underwood, Todor Markov, Toki Sherbakov, Tom Rubin, Tom Stasi, Tomer Kaftan, Tristan Heywood, Troy Peterson, Tyce Walters, Tyna Eloundou, Valerie Qi, Veit Moeller, Vinnie Monaco, Vishal Kuo, Vlad Fomenko, Wayne Chang, Weiyi Zheng, Wenda Zhou, Wesam Manassra, Will Sheu, Wojciech Zaremba, Yash Patil, Yilei Qian, Yongjik Kim, Youlong Cheng, Yu Zhang, Yuchen He, Yuchen Zhang, Yujia Jin, Yunxing Dai, and Yury Malkov. Gpt-40 system card, 2024a. URL htthtttss////xit www.oogg//////./.ooooooooo.os/// abs/2410 21276.
12

---

Published as a conference paper at COLM 2025
OpenAI, ;, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, Ananya Kumar, Andre Saraiva, Andrea Vallone, Andrew Duberstein, Andrew Kondrich, Andrey Mishchenko, Andy Applebaum, Angela Jiang, Ashvin Nair, Barret Zoph, Behrooz Ghorbani, Ben Rossen, Benjamin Sokolowsky, Boaz Barak, Bob McGrew, Borys Minaiev, Botao Hao, Bowen Baker, Brandon Houghton, Bran- don McKinzie, Brydon Eastman, Camillo Lugaresi, Cary Bassin, Cary Hudson, Chak Ming Li, Charles de Bourcy, Chelsea Voss, Chen Shen, Chong Zhang, Chris Koch, Chris Orsinger, Christopher Hesse, Claudia Fischer, Clive Chan, Dan Roberts, Daniel Kappler, Daniel Levy, Daniel Selsam, David Dohan, David Farhi, David Mely, David Robinson, Dimitris Tsipras, Doug Li, Dragos Oprica, Eben Freeman, Eddie Zhang, Edmund Wong, Eliz- abeth Proehl, Enoch Cheung, Eric Mitchell, Eric Wallace, Erik Ritter, Evan Mays, Fan Wang, Felipe Petroski Such, Filippo Raso, Florencia Leoni, Foivos Tsimpourlas, Fran- cis Song, Fred von Lohmann, Freddie Sulit, Geoff Salmon, Giambattista Parascandolo, Gildas Chabot, Grace Zhao, Greg Brockman, Guillaume Leclere, Hadi Salman, Haiming Bao, Hao Sheng, Hart Andrin, Hessam Bagherinezhad, Hongyu Ren, Hunter Lightman, Hyung Won Chung, Ian Kivlichan, Ian O'Connell, Ian Osband, Ignasi Clavera Gilaberte, Ilge Akkaya, Ilya Kostrikov, Ilya Sutskever, Irina Kofman, Jakub Pachocki, James Lennon, Jean
Jason Wei,Jean Harb, Jerry Twore, Jiacheng Feng, Jiahui Yu, Jiayi Weng, Jie Tang, Jieqi Yu, Joaquin Quiñonnro Candela, Joe Palermo, Joel Parish, Johannes Heidecke, John Hallman, John Rizzo, Jonathan Gordon, Jonathan Uesato, Jonathan Ward, Joost Huizinga, Julie Wang, Kai Chen, Kai Xiao, Karan Singhal, Karina Nguyen, Karl Cobbe, Katy Shi, Kayla Wood, Kendra Rimbach, Keren Gu-Lemberg, Kevin Liu, Kevin Lu, Kevin Stone, Kevin Yu, Lama Ahmad, Lauren Yang, Leo Liu, Leon Maksin, Leyton Ho, Liam Fedus, Lilian Weng, Linden Li, Lindsay McCallum, Lindsey Held, Lorenz Kuhn, Lukas Kondraciuk, Lukasz Kaiser, Luke Metz, Madelaine Boyd, Maja Trebacz, Manas Joglekar, Mark Chen, Marko Tintor, Mason Meyer, Matt Jones, Matt Kaufer, Max Schwarzer, Meghan Shah, Mehmet Yat- baz, Melody Y. Guan, Mengyuan Xu, Mengyuan Yan, Mia Glaese, Mianna Chen, Michael Lampe, Michael Malek, Michele Wang, Michelle Fradin, Mike McClay, Mikhail Pavlov, Miles Wang, Mingxuan Wang, Mira Murati, Mo Bavarian, Mostafa Rohaninejad, Nat McAleese, Neil Chowdhury, Neil Chowdhury, Nick Ryder, Nikolas Tezak, Noam Brown, Ofir Nachum, Oleg Boiko, Oleg Murk, Olivia Watkins, Patrick Chao, Paul Ashboume, Pavel Izmailov, Peter Zhokhov, Rachel Dias, Rahul Arora, Randall Lin, Rapha Gontijo Lopes, Raz Gaon, Reah Miyara, Reimar Leike, Renny Hwang, Rhythm Garg, Robin Brown, Roshan James, Rui Shu, Ryan Cheu, Ryan Greene, Saachi Jain, Sam Altman, Sam Toizer, Sam Toyer, Samuel Miserendino, Sandhini Agarwal, Santiago Hernandez, Sasha Baker, Scott McKinney, Scottie Yan, Shengjia Zhao, Shengli Hu, Shibani Santurkar, Shraman Ray Chaudhuri, Shuyuan Zhang, Siyuan Fu, Spencer Papay, Steph Lin, Suchir Balaji, Suvansh Sanjeev, Szymon Sidor, Tal Broda, Aidan Clark, Tao Wang, Taylor Gordon, Ted Sanders, Tejal Patwardhan, Thibault Sottiaux, Thomas Degry, Thomas Dimson, Tianhao Zheng, Timur Garipov, Tom Stasi, Trapit Bansal, Trevor Creech, Troy Peterson, Tyna Eloundou, Valerie Qi, Vineet Kosaraju, Vinnie Monaco, Vitchyr Pong, Vlad Fomenko, Weiyi Zheng, Wenda Zhou, Wes McCabe, Wojciech Zaremba, Yann Dubois, Yinghai Lu, Yining Chen, Young Cha, Yu Bai, Yuchen He, Yuchen Zhang, Yunyun Wang, Zheng Shao, and Zhuohan wwwpaaawwww....////////aaaaaaaaa abgp 24
Li. Openai ol system card, 2024b. URL https: http///xxxvvvoo//
Mingyang Song, Zhaochen Su, Xiaoye Qu, Jiawei Zhou, and Yu Cheng. PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Models, 2025.
Xiaoyu Tan, Tianchu Yao, Chao Qu, Bin Li, Minghao Yang, Dakuan Lu, Haozhe Wang, Xihe Qiu, Wei Chu, Yinghui Xu, and Yuan Qi. Aurora:automated training framework of universal process reward models via ensemble prompting and reverse verification, 2025. URL https://arxxi hrrg/bs//500/ 11520.
Matias Valdenegro- Toro and Daniel Saromo. A deeper look into aleatoric and epistemic uncertainty disentanglement, 2022. URL https: //arsiv///vvv..o org/ abs/ 220.. 093088.
Peiyi Wang, Lei Li, Zhihong Shao, R. X. Xu, Damai Dai, Yifei Li, Deli Chen, Y. Wu, and Zhifang Sui. Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations, 2024.
13

---

Published as a conference paper at COLM 2025
Xiong Wei, Zhang Hanning, Jiang Nan, and Zhang Tong. An implementation of generative prm., 2024. URL https::/gitiee Cow//LLFooww/R....... Batee//o/////---P---- MeLLf--Rwarrd-Modeiingg
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. Xiv preprint ar 2rXi:22412.51115 2024.
Lifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kaiyan Zhang, Bowen Zhou, Zhiyuan Liu, and Hao Peng. Free process rewards without process labels. ar arviv preprint ar arXiv:2412.01981 2024.
Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The Lessons of Developing Process Reward Models in Mathematical Reasoning, 2025.
Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu, Dayi- heng Liu, Jingren Zhou, and Junyang Lin. ProcessBench: Identifying Process Errors in Mathematical Reasoning, 2024.
14

---

that analyses paragraphs + found to contain an error. Respond with your analyses conclusion directly. Published as a conference paper at COLM 2025
A LLM-as-Judger Prompt Template
For LLM-as-Judger we follow the prompt in Zhang et al. (2025).
I will provide math prablem along with solution. They will be fornatted as follows: [Math Problen]
<math_problen> ... (math problem)...
5
k/math_problen>
[Solution]
<paragraph_12
(..rgggggp 1 solution)...
</paragraph_1>
10
<paregraph_n>
12
(p-aaggggaphhh n of solution)...
</paragraph_n>>
15
Your task is to review each paragraph of the solution in sequence, analyzing, verifying, and critiquing the reasoning in detail. need provide the analyses and the conclusion in the following fornat: <anslysis_
17
19
..-(analysis of paragraph 1)...
20
</analysis_1>
22
<analysis_n>
(analysss paragraph
21
</analysis_n>
kconclusion>
26
torrect/Incarrent
27
</conclusion>
29
When you analyze each paragraph, you should use proper varification, recalculation, or logically mathematically valid. Please reflection indicate whether carefully.
4 elaborate on the analysis process detected in any paragraph, you + error in detail, and suggest how to correct the error or the correct approach. Once paragraph is faund ta contain any error, stop further analysis the identified errar) and directly provide the 4 conclusion of "Incorrect." For instance, given solution of five paragraphs, if an third paregraph, you should reply in following
describe
subsequent
paragraphs (as they may depend
error
31
<analysis_1>
...(analysis paragraph 1)...
33
</analysis_1>>
34
<analysis_2>
... ...(anlyysis of paragraph 2)
36
</analysis_3>>
<analysis_3>
38
.--aaalyssis of paragraph 3; since an error is found here, alss provide detailed critique and - correction guideline)...
39
</analysis_3>
40
sconclusion>
12
</conclusion>
skipped as paragraph
15
46 The following is the math problen and the solution for your task: [Hath Problen]
48
{ [tagged_problen)
[Sclution]
30
(cagged_response)
B More Experiment Results
B.1 Problem diversity is important for Training PRMs
PRMS00K (Lightman et al., 2023) a widely used and human-annotated dataset for PRM training, which contains 800K step-level labels across 75K tree-of-thoughts solutions to 12K MATH (Hendrycks et al., 2021). Our empirical results show that models trained on our dataset (100K samples from 100K diverse questions) consistently and significantly outper-
15

---

Published as a conference paper at COLM 2025
Problem set
CoT Trajectories
ProcessBench F1
PRMS00K
7,500
460,000
0.575
NuminaMath (Random Selected)
100,000
100,000
0.673
Table 3: Comparison between PRM800K and 100K data collected from NutinaMath labeled by Qwen-QwQ.
form those trained on PRM800K (369K samples from only 12K questions) on ProcessBench. These findings suggest that problem diversity plays more crucial role in PRM training than the number of step-level annotations.
C Annotation Cost Estimation
We estimate the labeling cost based on the statistics of our 1M data collected from Numina- Math Li (2024) using Ottett--Matth77nnsstuutc and Qwenn.5-Mathh-2B-Inttt We
introduce the statistics in Table 4.
Value
Source
# Reasoning Steps (5)
8.845
Qwen Models' rollouts
Tokens per Rollout (R)
625.098
Qwen Models' rollouts
Tokens per Critic Response from Judge (C)
1,919.860
Qwen-QwQ's responses LLM-as-Judge
Table 4: Statistics of 1M NuminaMath CoT Trajectories collected by Qwen2.5-Math Models.
In addition to the statistics, we also use N to denote the data number of the dataset and show this statistic of each model's training dataset in Table
Dataset
Labeled Data
ACTPRM
624,000 (labeled two stages)
860,000
Qwen2.5-Math-PPM
860,000
UniversalPRM
690,000
Table 5: Data number of datasets.
Using the statistics, we compute the estimated labeling cost for ACTPRM, Qwen2.5-Math- PRM-Math-shepperd (Zhang et al., 2025), Qwenn.25Math--PMM22hang et al., 2025), Univer- salPRM Tan al. (2025) as follows:
SaMM-MMt--sshpherdd x S x x R/2, where is the number of rollouts per step set in Zhang et al. (2025). We divided by two since the number of tokens for rollouts varies based on the position of reasoning step. For latter reasoning step, it requires less reasoning tokens. As result, the expectation of tokens per rollout should be half of the number of tokens of the complete rollout.
Qwen2.5-Math-PRM, N x x 8 R/2 + C. It used consensus filtering for each data, the cost is both from MathShepherd (S 8 x R/2) and LLM-as-Judge (C).
UniversalPRM: N x C x 4 N S, where 4 is the number of ensemble prompts from the original paper (Tan et al., 2025) and another N is for semantic-based step seperation.
ACTPRM: N x C. We use Qwen-QwQ as Judge and do not include any other operations.
16


Reviews:
[{"review_id": "3l2wwGTYzF", "reviewer": "Reviewer_p2Cc", "summary": "This paper proposes to use active learning for PRMs (Process Reward Models). An ensemble PRM is used. If the mean prediction confidence of the ensemble does not exceed a confidence threshold (0.95 worked best in the paper's results) for any step in a solution trajectory or if the standard deviation exceeds a threshold (0.005 worked best in the paper's results) for any step in a solution trajectory, then the math problem and the solution trajectory is selected to be labeled by a capable but costly LLM (Large Language Model) and used to train the PRM. The diversity of the ensemble models is fostered by using random initialization of the head layers and a diversity regularization term in the loss function for training the ensemble PRM. The proposed new approach is compared with existing approaches on ProcessBench and PRMBench (see Tables 1 and 2).  Experimental results show that the proposed method has relatively strong performance with significantly less data needing to be labeled by the capable but costly LLM. Additional experimental results are reported showing results with different confidence and standard deviation threshold values, {0.9, 0.95, 0.97} and {0.01, 0.005, 0.002, 0.001}, respectively. Also, an ablation study is done checking if the two types of uncertainty are complementary and they are found to be complementary. Also, experimental results are reported with different numbers of heads for the ensemble and it is found that performance improves as the number of heads grows until converging at about 32 heads.\n\nSome parts of the paper were not clearly written. See below sections of review for details and examples.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Seems to be novel if it's indeed the first work to use active learning to improve the data efficiency of training PRM.\n\nNew proposed method seems to be effective in experimental results reported.", "reasons_to_reject": "Some parts of the paper were not clearly written, especially with regard to understanding the contents of the experimental results and how to interpret the results. \n\nSome parts of the evaluation were difficult to understand. For example, for the results in Table 1, it was difficult to determine what the annotation cost differences are between the rows. For example, is it true that the bottom two rows used 563,030 annotations from QwQ-32B?  Then, what did each of the other rows use in terms of how many annotations they got and from what model they got them?  Can some intuition be given for what the cost is in terms of US Dollars or some intuitive measure of computing power or computing time or some appropriate measure(s) for readers that may not be deeply familiar with how much 563,030 annotations from QwQ-32B cost in terms of these more \"practical\" or more \"commonly used\" or more \"intuitive\" measures of cost?  Can similar measure(s) of cost be given for other rows so readers can more easily understand the amounts of the annotation cost savings of the various methods in more meaningful terms?", "questions_to_authors": "See above sections of review. \n\nAlso, I think the ablation study on number of heads for ensemble PRM would make sense to run with both epistemic and aleatoric uncertainty used for sampling since that is the actual best system that is being proposed in which the ensemble would be used. The existing experiment with only epistemic uncertainty considered is perhaps of some theoretical interest, but it could be that in practice the convergence properties are different when both aleatoric and epistemic uncertainty are considered, which is the suggested method, so testing with both would seem to be more practically useful and relevant. \n\nI suggest expanding all acronyms before first use. \n\nI suggest fixing the subscript on the reference to the loss function in line 179. \n\nIn Figure 2a, why does the curve for the ActPRM model stop at Budget = approximately 0.6?  Could it be run out all the way to Budget = 1.0? \n\nIn line 182 the variable n is used. It has been used previously in line 93 as the number of solution steps in a solution trajectory and in line 176 as the number of ensemble heads. I suppose in line 182 it means the number of ensemble heads and most readers will be able to figure this out, but it would be nice to make it clear to readers by perhaps using different variable names for these two different meanings. \n\nIn Algorithm 1, in the \"Input\" section a \"generative LLM M\" is mentioned, but it seems M is never used anywhere in the rest of the algorithm. In line 10, it might make sense to indicate the labeling is done by M. \n\nIn table 2, I guess the column \"Simlicity\" should be \"Simplicity\"?\n\nIt seems that the \\delta_{pred} and \\delta_{std} parameters were set to their best values via running through grid search over runs with a bunch of possible values. Does this give the proposed method an unfair advantage over other competing methods in the experimental results that are reported?  Is it suggested that the values of 0.95 (confidence threshold) and 0.005 (standard deviation threshold) should now always be used with the proposed new method, even for new tasks and applications of the method beyond the datasets used in the paper's experiments?  If not, how should those parameters be set for uses of the proposed method for new tasks and applications beyond the datasets used in the paper's experiments?", "extra_scores": {}}, {"review_id": "FjFZQkblXu", "reviewer": "Reviewer_ZZ8c", "summary": "The paper introduces a novel active learning framework for training Process Reward Models using uncertainty-aware data selection. The proposed method uses ensemble heads to estimate both aleatoric and epistemic uncertainty (rare distinction in ML literature) and only annotate uncertain samples using a strong model as a judge. Reported experiments show that the proposed method achieves SOTA performance on ProcessBench and PRMBench, outperforming existing baselines while using only a small percentage of the annotation budget. The paper includes detailed robustness evaluations, and shows commitment to reproducibility by claiming to release code and models.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper tackles an impactful problem - annotation costs in PRM training are very high, which presents a major challenge in tuning LLMs for multi-step reasoning.\n- The use of both epistemic and aleatoric uncertainty through ensemble-based modeling is effective and is theoretically grounded.\n- Strong empirical results\n- Robust analysis", "reasons_to_reject": "- I see no reasons to reject this paper.", "questions_to_authors": "Have you tried this on problems other than math reasoning? E.g. code generation. I expect it would transfer well, but it would be great to see empirical confirmation.", "extra_scores": {}}, {"review_id": "7uE30keM4f", "reviewer": "Reviewer_YDbb", "summary": "The paper proposes ACTPRM, an active learning framework for training Process Reward Models (PRMs) that significantly reduces annotation costs while maintaining or improving performance. By leveraging ensemble-based uncertainty estimation, ActPRM selectively annotates the most uncertain reasoning steps, achieving a 50% reduction in labeling costs compared to full-data tuning.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper proposes a novel active learning approach (ACTPRM) for PRM training that effectively addresses the critical challenge of high annotation costs.\n2. The experimental results demonstrate significant advantages, achieving state-of-the-art (SOTA) performance on both ProcessBench and PRMBench.\n3. The ablation studies are comprehensive, employing a two-stage experimental design to identify optimal hyperparameters.", "reasons_to_reject": "1. The authors could provide a more coherent explanation and clarification regarding the derivation and definition of the uncertainty score in Section 4.\n2. The writing could be further refined for clarity, particularly in detailing the active learning experimental procedure and highlighting the progressive relationship between Sections 5.1 and 5.2.", "questions_to_authors": "1. In Section 5.1, it remains unclear whether ACTPRM ispre-trained before executing Algorithm 1, which I think is not clearly outlined in paper. If ACTPRM is pre-trained, I hope to see more details. If it is initialized directly from the base model Qwen2.5-Math-7B-Instruct, this raises another question: How much data would be labeled as \"uncertain\"? Intuitively, this proportion should be substantial since the ACTPRM lacks prior knowledge, implying most of the data (100K from Numinamath) would require annotation.\n2. Lines 197 and 205 contain ambiguous phrasing. Active learning should operate on a large unlabeled pool—how should Line 197 be interpreted in this context?\n3. Do authors conduct further analysis of the uncertain data metrics (aleatoric and epistemic)? Why are these specific metrics chosen? Are other factors considered?\n4. I have questions about the initialization method for the ensemble heads. Do authors explore alternative approaches, such as averaging the final layer embeddings across the vocab dimension (a conventional practice)? Given that 32 heads are ultimately selected, could the authors provide insights into why so much heads is necessary?\n5. In Section 5.2, the authors use QwQ-32B to annotate the PRM dataset, yet in the final results, QwQ-32B outperforms all PRMs on ProcessBench. This suggests potential unresolved influence from data distillation via QwQ. Is this an inherent limitation of active learning? A deeper analysis could be valuable.\n6. How should Line 190 be interpreted? Can uncertainty scores determine the correctness of a given step?\n7. Typo Error: Line 179, L_BCE.", "extra_scores": {}}, {"review_id": "8aGlXxzYF7", "reviewer": "Reviewer_hjfr", "summary": "This paper proposes ACTPRM, an active learning method for training Process Reward Models (PRMs) efficiently. Instead of labeling all steps in reasoning trajectories, ACTPRM selectively queries \"uncertain\" examples (identified using ensemble-based epistemic and aleatoric uncertainty) for labeling, dramatically reducing annotation costs. Experiments show ACTPRM matches or exceeds previous state-of-the-art performance (e.g., 75% on ProcessBench) with only 6–20% of the labeling cost.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Reducing annotation costs is critical for scaling PRMs, especially in math reasoning tasks.\n\nThe paper explains ensemble head setup, uncertainty metrics, and selection strategies very cleanly (Algorithm 1 is crisp and actionable).\n\nAcross ProcessBench and PRMBench, ACTPRM and ACTPRM-X achieve new SOTA with dramatically lower labeling costs.\n\nIncludes ablations on uncertainty types, ensemble size, budget-efficiency curves, and comparative studies against random selection.\n\nThe paper is easy to follow, well-structured, and offers thoughtful discussion of limitations and future work.", "reasons_to_reject": "The idea of using ensemble uncertainty for active learning is well-established (e.g., in preference learning, alignment studies). The technical novelty mainly lies in applying this idea to PRM training rather than inventing new uncertainty quantification techniques.\n\nThe method assumes availability of very strong LLMs (e.g., QwQ-32B) for labeling the uncertain examples. In weaker resource settings, ACTPRM's performance may degrade.\n\nThe uncertainty thresholds are tuned once and reused. It’s unclear how sensitive the method is to these choices across very different domains (e.g., non-math CoT).\n\nThe paper could better discuss ethical implications (e.g., reliance on large models, environmental costs of retraining, biases in LLM judges).", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes to reduce the cost of training process reward models by selecting examples to annotate based on the aleatoric and epistemic uncertainty, with the former characterized as P(step) and the latter as V[P(step)] across multiple prediction heads. This apparently makes it possible to attain better performance at significantly lower annotation cost (with annotation performed by a stronger reasoning model).

The reviewers are generally positive on the work, noting the high potential impact, strong motivation, and solid empirical results. Two minor potential concerns are (1) the tuning of the two hyperparameters (and the generality of the values chosen here), and (2) the limitation to settings in which a strong oracle annotator is available.
Decision: Accept