Free Process Rewards without Process Labels
Authors: Lifan Yuan
Wendi Li
Huayu Chen
Ganqu Cui
Ning Ding
Kaiyan Zhang
Bowen Zhou
Zhiyuan Liu
Hao Peng
Venue: ICML 2025 poster (2025)
Keywords: Process Reward Model
Primary area: deep_learning->large_language_models
TL;DR: We derive a new reward representation for outcome reward modeling so that the trained outcome reward model can be repurposed as a process reward model at inference.

Abstract: Different from its counterpart outcome reward models (ORMs), which evaluate the entire responses, a process reward model (PRM) scores a reasoning trajectory step by step, providing denser and more fine-grained rewards. However, training a PRM requires labels annotated at every intermediate step, presenting significant challenges for both manual and automatic data collection. This paper aims to address this challenge. Both theoretically and empirically, we show that an implicit PRM can be obtained at no additional cost, by simply training an ORM on the cheaper response-level labels. The only assumption is to parameterize the outcome reward as the log-likelihood ratios of the policy and reference models rϕ(y) = β log πϕ(y) πref(y) , which can be optimized regardless of the specific choice of loss objectives. In experiments, we instantiate our implicit PRMs with various objectives and evaluate their performance on MATH. We show that our implicit PRM outperforms a strong MCTS-based baseline á la Math-Shepherd (Wang et al., 2023) using less than 1/38 of the training data. Its performance can be further improved with majority voting. We further find that scaling up instructions and responses benefits our implicit PRM, and the latter brings a larger gain. Particularly, we find that our implicit PRM, when instantiated with the cross-entropy (CE) loss, is more data-efficient and can keep improving generation models even when trained with only one response per instruction, the setup that suffers from extreme data scarcity and imbalance. Further, instructions should be relevant to downstream tasks while the diversity of responses does not bring gains. Surprisingly, training on extra Math-Shepherd step labels brings no further improvements to our implicit PRM trained on only outcome data. We hope that our work will encourage a rethinking of PRM training approaches and contribute to making training PRMs more accessible.

Free Process Rewards without Process Labels
Lifan Yuan
Wendi Li
Huayu Chen?
Ganqu Cui
Ning Ding? 2 Kaiyan Zhang?
Bowen Zhou?
Zhiyuan Liu? Hao Peng
Abstract
1aama-.11-8-Inssee
10.6*
+2.9%
Different from its counterpart outcome reward models (ORMs), which evaluate the entire responses, process reward model (PRM) scores reasoning trajectory step by step, providing denser and more fine-grained rewards. However, training a PRM requires labels annotated at every intermediate step. presenting significant challenges for both manual and automatic data collection. This paper aims to address this challenge. Both theoretically and empirically, we show that an Implicit PRM can be obtained at no additional cost, by simply training an ORM on the cheaper response-level labels. The only assumption to parameterize the outcome reward as the log-likelihood ratios of the policy ro(y) and reference models (y) = 33 log Ferly)) which can be optimized regardless of the specific choice of loss objectives. experiments, we train our Implicit PRMs with various objectives and evaluate their performance MATH. Implicit PRMs outperform strong MCTS- based baselines á la Math-Shepher less than 1/38 of the training data. We further find that scaling up instructions and responses benefits our Implicit PRMs, and the latter brings larger gain. Particularly, Implicit PRMs with the cross-entropy (CE) loss is data-efficient, and yields meaningful improvements even trained with only one response per instruction, a setup that suffers from extreme data scarcity and imbalance. We hope that our work will encourage rethinking of PRM training approaches and con- tribute making training PRMs more accessible.
Accuracy (%)
50-
38.6x
45-
40
implicit PRM (Ours. DPO:
Math-Shephherd
implicit (Ours,
10
10
1020
FLOPs
Figure 1:
The x-axis indicates the FLOPs required to collect the data and train the model, and y axis the accuracies of best-of-64 performance. The accuracy is averaged over the best-of-64 accuracies of Diistal--B-Instruct--e et al., 2023), 1--maa..--B--Isttiii Instruct (Meta, 2024) on MATH (Hendrycks et al., 2021). Different dots on the same line indicates models trained with the same approach but on different scales of data. The top- left zone is desirable in this figure, it suggests model can achieve higher performance with less development overhead. Our implicit PRM is much cheaper to train while presenting the best performance under the same budget.
(Jiang and Llama-3.1-700
(Wang et al., 2023) using
1. Introduction
Training on high-quality supervised data has driven the ad- vances in LLMs development (Meta, 2024; Ding et al., 2023; Luo et al., 2024b; Yue et al., 2024; Yuan et al., 2024; Zhang et al., 2024b). Building upon this progress, reward mod- els push the boundaries even further, especially in tasks requiring complex reasoning (Lightman et al., 2023; Wang et al., 2023; Snell et al., 2024). Outcome Reward Models (ORMs), designed to evaluate full responses, have been pri- marily explored, which can be used both reinforcement learning (RL) and inference. However, due to the sparsity of outcome rewards, ORMs often yield suboptimal perfor- mance when reranking responses at inference (Lightman et al., 2023) and struggle with stability and efficiency during RL training (Cao et al., 2024; Chan et al., 2024). This high- lights the growing demand for denser and more fine-grue -ra-gga rewards. Pracess Reward Models (PRMs), evaluating in- termediate steps to provide fine- ane-grained guidance, naturally meet this need. Existing work has shown consistent results
Equal contribution University of Illinois Urbana- Champaign TTinghua University 'Huazhong University of Science and Technology "Shanghai AI Lab. Correspon- dence to: Lifan Yuan <lifan4 illinois.edux, Wendi Li <wendili @ moust.edu.cm en>, Ganqu Cui <cuiganqu @pjlab.org.cmm Ning Ding <dn97 @ msi.lissinnuu..uu.....
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Free Process Rewards without Process Labels
that PRMs outperform ORMs in best-of-N sampling (Wang et al., 2023; Snell et al., 2024) and RL (Setlur al., 2024), and argues that scoring every intermediate step provides better transparency and interpretability (Leike, 2024).
with substantially better trade-offs between accuracy and development overhead, shown in Figure/ 1. Particularly, when integrated into weighted best-of-N, CE stands as the most effective ($??). This makes CE loss appealing in sce- narios where pairwise data is hard to collect, since it can handle unpaired and imbalanced data, and is demonstrated to be less data-consuming than DPO in order for an Implicit PRM with decent performance. Further, we find out that our Implicit PRM benefits from increased training data, and the scale of responses is more impactful than that of instructions ($5. 1). Surprisingly, training on step-level data brings no further improvements to our Implicit PRMs ($C.2). Finally, we observe that, at least for the models and tasks we con- sider, the reference model can be omitted without hurting the model's quality ($5.3.2)) This makes our Implicit PRMs even more appealing, offering improved training efficiency and performance without added inference overhead.
Despite their promise, PRMs are much harder to train than ORMs, since collecting PRM training data requires annotating every intermediate step. To reduce human efforts, automatic annotation approaches have been proposed, where intermediate step is labeled based on its estimated probability of leading to correct outcome. Typically, this is achieved through either sampling massive look-ahead trajectories to estimate or directly training a verifier to predict Q value, both incurring extensive overhead ( (Wang et al., 2023; Lu et al., 2024). For example, collecting step-level data through sampling look-ahead trajectories as Wang et al. (2023) requires 38.8x more FLOPs than training an ORM (64).
Bypassing the need for step labels, implicit PRMs substan- tially lower the data collection and training overhead of building PRMs while delivering stronger performance than existing methods. We hope that our work will encourage rethinking of PRM training approaches and contribute to making training PRMs more accessible.
We argue, from both theoretical and empirical perspectives, that building PRMs can be substantially cheaper than pre- viously realized: a strong PRM can be obtained at no additional cost from training an ORM on the cheaper response-level data, with simple reward parameter- ization. Specifically, by parameterizin the reward the log-likeeln -igllikel ratio of the policy and the reference models firry)). a common practice in DPO (Rafailov et al., 2023) and many of its variants (Azar et al., 2024; Ethayarajh et al., 2024; Chen et al., 2024; Rosset et al., 2024; Wu et al., 2024), a PRM can be automatically learned during ORM training. The process reward is then the same log- oogllikelhhod ratio, but calculated over a partial response. We dub our approach an Implicit PRM since it only re- quries response-leve data and ORM training. Moreover, our insights are agnostic to the specific choice of the train- ing objective, and are applicable to both DPO and all the variants that adopt the same form of implicit reward; it fur- ther extends to other objectives like the Cross-Entropy (CE) loss. This fresh theoretical insight generalizes the conclu- sion from Rafailov et al. (2024) that DPO training enables the model to learn the Q function; practically, our approach particularly well-suited for scenarios where pairwise data is hard to obtain and algorithms like CE loss remain equally applicable, as shown in $5.1.
ro(y)= :llog moly).
2. ORMs vs. PRMs: Dilemma of Performance and Expense
Background ORMs assign sparse rewards ry(y) to the entire response, and no feedback is provided until the last token is generated. In contrast, a PRM assesses the quality of every intermediate step and can provide reward after com- pleting each (Lightman et al., 2023). Given an instruction and an r-step response y with y/ being the t-th step and y<r being the first t 1 steps, PRM receives the concatenation of the instruction and the first reward to the t-th: r r <t, yt). The Q value o (y <t. Bt) indicates the expectation of outcome reward r' conditioned on the observed response y< and current step yt. Lightman et al. (2023) define the process reward as the correctness of each step, while Wang et al. (2023) directly consider Q values as process rewards. We follow Lu et al. (2024) and define process reward as advantages, namely the difference between Q values: r/. == 9 %-- The benefits of adopt- ing advantages process rewards have been discussed by concurrent work (Setlur et al., 2024).
1 steps, and assigns
In experiments, we train our Implicit PRMs on a dataset consisting of 33K math instructions and eight solutions for each, and evaluate them through the best-of-N sampling on MATH (Hendrycks et al., 2021). We explore variants of our Implicit PRMs trained with different training objectives, including DPO, KTO (Ethayarajh et al., 2024), NCA (Chen et al., 2024), and CE ($4.2). All produce strong PRMs, out- performing competitive baselines including our reimplemen- tations of Math- Sath-Shepherd (Wang et al., 2023) and AutoPSV (Lu et al., 2024) and six off-the-shelf open ORMs and PRMs,
PRMs outperform ORMs in both training and inference Both ORMs and PRMs can provide rewards to assess model outputs. The dense step-level rewards from PRMs lead to stable and effective RL training (Cao et al., 2024; Chan et al., 2024), and perform better on reranking responses, with better transparency and interpretability. Also, ORMs are trained on complete responses, but the value model ini-
2

---

Free Process Rewards without Process Labels
tialized from it only receives incomplete responses during RL training. On the contrary, PRMs are intrinsically trained to provide dense rewards given partial responses, thus the re- sulting value models may mitigate out-of-distribute issues that ORMs encounter.
reward parameterization. In other words, a PRM can be inherently derived from the same ORM without any dedicated training, offering better performance than MCTS-base approaches with substantially lower overhead.
Reward parameterization in existing work Current liter- ature typically parameterize rewards by either (1) the linear transformation of hidden states, with the reward model be- ing a sequence classifier (Ouyang et al., 2022; Touvron et al., 2023; Zhu et al., 2023; Cui et al., 2024) or (2) generative logits, with reward models being an auto-regressive L.M and trained to predict the label of partial or complete responses as "good" or "bad" tokens, and sometimes a third "neutral" (Zhang et al., 2024c; Mahan et al., 2024; Lightman et al., 2023; Wang et al., 2023; Luo et al., 2024a).
Training PRMs is substantially more expensive than ORMs Despite its effectiveness, training PRMs more difficult due to challenges in training data collection. To collect training data for PRMs, MCTS is commonly used for automatic step annotation (Wang et al., 2023; Luo et al., 2024a). However, it introduces substantial extra cost. For MCTS-based step label annotation, a policy model will sample N trajectories based on the concatenation of an instruction x and partial response up to step i, each leading to final answer (Wang et al., 2023). E.g., assuming 10-step rollouts and 8 subsequent trajectories for each step as in Wang et al. (2023), total of 10 x 8 = 80 trajectories need to be generated to get step labels for each instruction, which is 80 times more than ORMs. Therefore, the scaling of PRMs is largely limited. Besides the overhead of training data collection, this MCTS approach can lead to suboptimal performance due to the noisy annotation process, as we will show below and in the experiments.
Unfortunately, under either of the two parameterizations, PRMs would require expensive step labels to train. To address this issue, we propose to train an ORM with im- plicit reward modeling, which will automatically enable PRM regardless of the loss functions. Next, we illustrate this in detail:
Proposition 3.1. (Proof in Appendix A) Consider an ORM where the reward is parameterized by the log-likelihood ratio of two causal LMs, i.e. rØly) := 3 log Tully?'
moly] Define
MCTS estimation is not precise either We denote the set of correctness of subsequent trajectories as {c1, . CN) }. each element being 0 or 1. Thereafter, two alternative label estimation strategies are available: (1) Hard Estimation, where step / will be labeled as 1 any rollout is correct and O otherwise: 4,- = max {c1, C2..... }. (2) Soft Estimation, where step is labeled as the proportion of correct answers among all rollouts, namely 4 N' et/N. We refer the ORM used to judge the correctness of rollouts as é, the PRM trained on data from hard estimation as on, and the PRM trained on soft estimation data as &... If on, and é. are perfectly fitted, namely training losses reduced to 0, we have
I's (ycr. Vn) == = Blog [x(llr) Lg is the exponen- tial average of at step I.
d <e.88)) = Blog Bo
(2)
Hence, L represents an exact expectation of outcome re- ward r at step t, i.e., the Q value.
Proposition 3.1 indicates that when modeling ry == 3 log Fsly) train an ORM with the standard pipeline, where / is a hyperparameter, o can implicitly learn a Q function. Hence, process reward r' can be obtained by:
<y(y))
Le (y M+) = max ry),, yivs.
(L)
r 9
4 1-1 = Blogg Jee (y)
y
Se (y<t, v:) To(r)
However, both estimation strategies may be noisy. Specif- ically, 4,, represents the maximum outcome reward ro given Y<t, rather than the expectation, thus overestimating the Q value; For d/,- given the limited capability of the policy model in practice, it can be challenging to sample correct solutions for difficult instructions, suffering from false negative noises and thus underestimating Q.
Notably, this conclusion still holds when y/ represents the t- th token rather than step f. This gives us an inspiring hint: we can indeed obtain PRMs of any granularity simply by collecting response-level data and training an ORM, without any burden of annotating step labels, as shown in Figure 2. The proposition is agnostic to specific choices of the training objective of ORMs. It can be instantiated with different objectives as vanilla ORM training, with the only * difference being substituting the re (y) with 3 log Fwi(y] Particularly, many existing preference learning algorithms have already met our assumption (Rafailov et al., 2023; Azar et al., 2024; Ethayarajh et al., 2024; Chen et al., 2024; Wu et al., 2024).
3. Implicit PRMs For Free Through Reward Parameterization
In this section, we show that an ORM can directly represent an expectation of the outcome reward by itself by simple
3

---

Free Process Rewards without Process Labels
Train as an ORM
Inference as an Implicit PRM
Forward pass on ORM data to obtain implicit reward
Log-likelihood ratio on partial response represents value:
T
x²
Psly) 3 Blog Yoly)
3 Blog
1
Cen fra (1)
y
yf
Derive implicit process reward from Q value:
Train an ORM with vanilla ORM loss (e.g., loss)
flyy
3lg
(26 Vall)
y' 0/t6/2
Outcome label
f/ can be either single token step
Figure 2:
With specifically designed reward representation, train ORM with standard pipeline while automatically enabling an Implicit PRM.
Besides making PRM training more accessible, implicit process reward can be more accurate than those derived from E- and 93, in Eq. 1 (Wang et al., 2023), as indicated by the following proposition:
practical scenarios with unpaired and imbalanced data:
Ty
LCz --1 I.log Blog
Trr
(5)
To To( (Blog
Proposition 3.2. The performance of g is guaranteed by the following conditions: To is bounded by Le and d'y and can reach these bounds with specific values of B. That is,
-(1 11 log
Reference Model One difference between our modeling of rewards and previous ones is the incorporation of a ref- erence model Tea We acknowledge that this comes at an inference cost: to calculate the reward, both the policy and reference model are served, which doubles the inference cost than vanilla PRM. However, it is prevalent in existing preference learning algorithms and works as the KL con- straint to prevent the policy model T&. deviating too far from its starting checkpoint. Moreover, practice, as we will show $5.3.1 that large proportion of the inference overhead in best-of-N sampling comes from the generation model, especially when the generation model is much larger than the reward model. Further, we also show in $5.3.2 that when the Implicit PRM is built from strong model that has undergone preference learning, such as
do Toll) S 4. vi)
maxx roly) Len ylyge
(4)
holds. The left-hand equality is attained as 3 + oo and the right-hand one is attained as B 0.
Proposition 3.2 demonstrates that 4 ranges between the soft-estimated and hard-estimated Q values annotated by MCTS-based approaches. The above bounds suggest that our approach has better accuracy and robustness to noises than
less a problem in
approaches. Specifically, discussed in $2, 4',, overestimates the Q value while de, underestimates Q due false negative noises. Since q/ lies between L,, and 4 it could potentially mitigate both issues and estimate the Q value more accurately. Concurrent work defines our o'3 an entropy regularized process reward and has empirically shown its superiority over q8, and do, on best-of-N sampling (Zhang al., 2024a).
excluding कet leads to little or no accuracy drop. This makes our approach appealing prac- tice since can achieve better accuracy than existing PRMs with exactly the same inference overhead, but substantially lower development overhead.
Connection to Rafailov al. (2024) An intuition similar to Proposition 3.1 has been brought up by Rafailov et al. (2024), which demonstrates that DPO enables models to learn the Q function implicitly, but our insights subsume their conclusion since this property is not limited to the DPO algorithm. For example, given response-leve label I, we can further generalize to cross-entrond entropy (CE) loss handle
4. Experiments
4.1. Setup
Evaluation Following standard practice (Lightman et al., 2023), we evaluate PRMs with best-of-N (BoN) on MATH-

---

Free Process Rewards without Process Labels
500 (Hendrycks et al., 2021). To study the generalizability of the PRMs, we test each PRM using three generation mod- els with different levels of capabilities: Mistral-Instruut--Ok. (Jiang et al., 2023), Llama-3. --B--Instrcct and Llama-3.1- 1- 70B-Instruct (Meta, 2024). For each completion, we apply PRMs to score each step and pick the lowest step reward as the score for overall responses. We also compare the development overhead of the models in terms FLOPs, in- cluding those required in both the automatic data collection and PRM training.
Data', and RLHIFloN-BB-DS-Daaa We note that these off-the-shelf baselines are trained on different instructions and responses, while our two reimplementations are trained on the same data as our Implicit PRM.
4.2. Results
Various implicit reward modeling objectives outperform baselines According to BoN results shown in Table I, all four variants of our Implicit PRMs consistently improve the accuracies of the three different generation models. Among them, DPO achieves an averaged accuracy of 50.4, per- forming better in general, closely followed by NCA with an averaged accuracy of 49.4. CE presents strong performance, despite that it is trained on unpaired and imbalanced data. Specifically, with an averaged accuracy of 48.4, it beats our implemented Math Shah-SShepherd and AutoPSV by 0.6 and 2.7 respectively, and outperforms other open-source reward and RLHFlow- 3B-DS-Data both of which achieves 49.1. This indicates the potential in empowering real-world applications where pairwise data is hard to collect. Nevertheless, according to CE versus CE (Inst.wwise Balanced), it is still beneficial to have balanced positive and negative responses for each instruction in the training dataset, which aligns with conventional understandings on CE as a classification loss. However, comparing CE (Dataset-wise Balanced) to CE, simply balancing the entire dataset by randomly filtering examples of the class with more data can be detrimental.
Training dataset Unless stated otherwise, we adopt the following training setup throughout all experiments: We use math instructions from Ultralnteract (Yuan et al., 2024) and sample eight rollouts per instruction using Llama-3.1- 8B- 8B-Instruct, and then assess rollout correctness with ground truths. We train PRMs based on Llama-3.1 1-8B-Inntruch with 3 = 0.05, which is empirically determined.
models except ---FFloo-----istrrll
Implicit PRM instantiation As demonstrated in $3, our approach can be instantiated with any reward modeling ob- jective with the reward parameterized as ra == ,3 log ne (y) " We explore various objectives that meet the requirements, including DPO (Rafailov et al., 2023), KTO (Ethayarajh al., 2024), NCA (Chen et al., 2024), and the cross-entropy (CE) loss. Please refer to Eq. 5 for the implementation of CE loss. For DPO and NCA, we pair each correct rollout with an incorrect counterpart and train our RM on these response-level pairs, while for KTO and CE loss, we di- rectly train on the unpaired and imbalanced rollouts, which more general in practical scenarios. We also implement two data balanced setup for CE analyze the impact of pair- wise data, i.e. balancing the positive and negative responses simply for the entire dataset, or more strictly for the each each instruction. We denote the two setups as Date Balanced and Instruction-wise Balanceed.
1e20
38.8x
Math-Sneghere
2.5
AutoPSV
Implict PRM (DPO)
2.0
Impict (CE)
1.5
39.5K
1.0
38,8K
0.5
4.3%
4.3x
0.3x
1.0x
42x 0.8x
1.8x
0.0
# Responses Per Instruction
Baselines Our baselines include our implementation of existing methods and off-the-shelf open models. We reimplement Math-Shepherd (Wang et al., 2023) and Au- toPSV (Lu et al., 2024) for fair comparisons, representative algorithms in their categories. Math-Shepherd annotates step labels using MCTS estimations as illustrated in $2. AutoPSV annotates steps with a two-stage strategy. It firsts trains an outcome supervision verifier (OSV) that predicts Q value for each step, then use the OSV annotate step labels. A PRM is obtained by continual training on the OSV with process labels. We also compare to six off-the-shelf ORMs and PRMs, namely EurusRM-7B (Yuan et al., 2024), SkyworkRM-Llama3.1 1-BB (Liu et al., 2024), ArmoRM- Llama3-8B (Wang et al., 2024), Math- Sath-Shephherd11 (the offical release of Wang et al., 2023), RL.Floow SofFllo--B-Miisee 8B-Mistral-
Figure 3: Overhead of developing different PRMs, in terms of FLOPs during data collection and training (lower is bet- ter). The x axis is the number of responses per instruc- tion determining the scale of training data, and the y axis the FLOPs. Our implicit PRM always consumes the least FLOPs, with CE being 38.6x to 38.8x more efficient than Math-Shepherd across different dataset scales.
Our Implicit PRMs reduce the overhead of data collec- tion and training by 38.8x As shown in Figure 3, with
1 9
hntpps//hougiingface..o ctw.co/////..... ow ow/.llaa// Llamm33
-B-PPR--DeepSeee-DDe
5

---

Free Process Rewards without Process Labels
Table 1: Different reward models' best-of-N sampling performance on MATH test set with three different generation models. When completing instructions with a temperature of 0.5, the three generation models' accuracies are 9.6%, 44.6%, and 63.2% respectively.
Mistral-7B-Inst--ol
Llama-3.1-8B-Ins
Llama-3.1-70B-
Type
Reward Model
Pass 1: 9.6
Pass @ 1: 44.6
Pass @ 1: 63.2
Avg.
@4
16
64
@4
@ 16
64
@4
@ 16
@ 64
Open Seerre Reward Models
EurusRM-7B
17.2
21.0
20.4
49.6
51.6
51.8
69.0
69.6
72.2
46,9
ORM
SkyworkRM-Llama3
16.0
19.6
23.4
49.0
50.4
48.2
70.4
72.6
72.0
46.8
ArmoRM-LLama3-8B
16.6
21.0
23.2
47.8
48.6
49.4
70.6
70.8
71.0
46.6
Math-Shepherd-7B
16.0
21.0
20.4
50.0
52.4
52.8
66.4
65.8
6.566
45.6
PRM
RL.HFlo-8BBMM M-------Maal-aait
19.4
25.2
30.2
51.8
52.0
50.6
70.8
71.0
71.2
49.1
RLHFlow-S8 SB----BBDSSS DEB-DS-DDL DS-Data
17.2
23.0
25.2
54.4
54.2
55.8
68.6
70.4
73.0
49.1
Our Implementations
Math-Shepherd
17.6
24.4
26.8
50.0
51.4
52.8
68.6
69.4
68.8
47.8
Baselines
AutoPSV
16.6
20.6
22.2
52.2
51.4
52.2
68.4
65.4
62.4
45.7
DPO
18.6
24.4
28.8
54.0
55.4
57.0
71.8
71.2
72.2
50.4
KTO
15.6
18.4
18.6
49.6
51.8
50.8
72.6
67.0
67.2
45.7
NCA
18.6
23.8
28.0
52.4
53.4
55.2
69.0
73.0
71.6
49.4
Implicit PRM
CE
18.8
24.0
28.0
52.6
54.4
53.0
70.6
67.0
67.2
48.4
CE (Dataset-wise Balanced)
18.0
23.6
27.0
52.6
54.2
52.6
68.6
66.8
67.0
47.8
CE (Inst.-wise Balanced)
17.6
22.6
26.2
52.6
55.2
54.6
69.4
71.2
72.0
49.0
three different training data scales. Math-Shepherd gen- erally costs 38.8x more FLOPSs than the Implicit PRM (CE). Compared Implicit PRM (DPO), the number be- comes 146.5x, 49.9x. and 21.3x under different number of responses per instruction respectively.
improve the performance of our Implicit PRM. The trend particularly clear on Mistral-7BBIIst--00.. and Llama-3.1-8B-Inst but there are also few outliers on Llama-3.1-70B-Ins.
(2) Compared to instructions, scaling up responses seems to be more influential on Implicit PRMs, as reflected by the larger performance variations between the minimum and maximum data setups. Taking a closer look at the response scaling, (3) DPO requires more data to obtain a decent performance than CE. From Figure 5, DPO is under-trained with two responses per instruction, which to the insufficient amount of instructions: two responses may not constitute a pair to train our DPO variant, and thus many instructions can not be used in training. In contrast, CE generally performs better with insufficient data and can always improve different generation model, even when is trained with one response per instruction with pairs, the extreme case of the unpaired setup. This presents huge advantage in real-world data scarcity scenarios.
We plot the scaling trends of the average performance of each method with corresponding number of tokens con- sumed in Figure 1, from which we can clearly see that our Implicit PRMs achieve better performance with much less data collection and training overhead.
be partly attributed
5. Analysis
5.1. Scaling Instructions and Responses Helps
Setup We conduct scaling analysis with DPO and CE on both instructions and responses of the training dataset. For instruction scaling, we randomly downsample 25%, 50%, and 75% instructions to train our Implicit PRM. For response scaling, since DPO can only train on paired responses, train models with 2, 4, and 8 rollouts; while for CE, we also implement training with only one rollout per instruction, the extreme case of unpaired setup.
5.2. PRM Ability Does Not Translate into Policy Performance
Implicit PRM is trained in auto-regressive manner, some- times directly using preference learning algorithms, which are primarily used to improve policy models. Therefore, it reserves the nature as causal LM and can still serve as
Results We present results in Figure and Figure 5 respectively. Takeaways are summarized as follows: (1) Scaling instructions and responses consistently
6

---

Free Process Rewards without Process Labels
Mistral-7B-Inet
Llama-3.1-88
Llama-3. 3.1-700B-Ins
29% Inst
74.0
50% Inst
75% Inst
56.0
72.0
Accuracy (%6
24.0
Inst
Accuracy (%6
52.0
Accuracy
68.0
18.0
48.0
66.0
12.0
46.0
64.0
8.0
2"
2
2
2
2
43.0
2"
2
2
2
62.0
2
2
2
2
2
2"
Candidates BoN Sampling
# Candidates BoN Sampling
Candidates in BoN Sampling
(a) Implicit PRM (DPO).
1istral-7B-Inst-vO.2
Baama-3.1-8B--me
Llama-3.1-7000 3.1-stB-ttt
57.0
28.0
25% Inst
72.0
50% Inst
55.0
Accuracy (%)
24.0
100% Inst
Inst
Accuracy (%)
53.0
Accuracy (%g
70.0
51.0
48.0
16.0
66.0
47.0
12.0
64.0
45.0
2
2
24
2
2
2
2
2
2
2
2
Candidales BoN Sampling
# Candidates BoN Sampling
Candidales in BoN Sampling
(b) Implicit PRM (CE).
Figure 4: Scaling instruction numbers. Our implicit PRM's performance on Mistral-7B-Iie - and Llama-3.1-8B. Instruct scales well with the number of instructions, despite the trend is more complex on Llama-3. Insttuuttt
Table 3: GPU time costs during best-of-N sampling relative to the cost of generation model (%). The overall infer- ence overhead of baselines 70.8%, and 90.9% of that of our implicit PRM, respectively. Namely, the reference model does not double the inference cost in practice, and the extra inference overhead becomes more marginal as the generation model gets larger.
a policy model to solve downstream problems directly. In this section, we test on MATH500 (Hendrycks al., 2021; Lightman al., 2023) to analyze the correlation between their PRM ability and performance policy model.
three test sets 66.6%,
According to Table 2, only traininng with KTO leads to an improvement on MATH500, com- pared to Llama-3.1 1-BB- Instruct. Interestingly, based Table 1, KTO performs the worst as Implicit PRM. In con- trast, DPO and CE, the two algorithms that per- form the best without majority voting and with majority voting setups, respectively, achieve the lowest accuracies. This indicates that PRM ability does not improve the pol- icy model improves, and there can even be unexpected trade-off between the both abilities.
Table 2: Implicit PRMs' per- formance on MATH500 when used to solve the problems directly.
Mistral-7B- Inst-v0.2
Llama-3.1- 8B-Inst
Llama-3.1
Source of Cost
Method
708-Inst
Generation Model
100.0
100.0
100.0
Model
Accuracy
Baselines
33.5
29.4
Llama-3.1-88 1a-aa-311-BB-Istt
45.2
Reward Model
Implicit PRM
201.6
141.7
22.2
+ DPO
25.8
Baselines
200.9
171.1
111.1
Total
+ KTO
46.6
Implicit PRM
301.6
241.7
122.2
+ +NCA
35.6
+ +CE
28.6
pecially when the generation model is much larger than the reward model ($5.3.1). Surprisingly, the reference model can removed at inference in certain cases ($5.3.2).
5.3.1. THE REFERENCE MODEL DOES NOT DOUBLE OVERALL INFERENCE OVERHEAD
Setup We caleulate the time costs of best-of-N sampling MATH500 in practice. The entire process includes (1) generating multiple candidate solutions to the instruction using the generation model, and (2) scoring each candidate using a PRM. We use vLLM (Kwon et al., 2023) to
5.3. Inference Overhead of the Reference Model
Our approach needs of an additional reference model in inference. However, we show that the the reference model does not double overall inference overhead in practice, es-
implement the former and Huggingface Accelerate (Gugger et al., 2022) for the latter.

---

Free Process Rewards without Process Labels
Miissaa-7B-Inst-vO..
Llama-3.1-88-
Llama-3. Llama-3.-770--img 1-700--Inst
Resp
72.0
Reep
Reep
56.0
Accuracy (%6
24.0
Accuracy (%)
Accuracy (%)
52.0
62.0
18.0
48.0
12.0
46.0
$2.0
8.0
2"
2
2
2
2
43.0
2"
2
2
47.0
2
2
2
2
2
2"
Candidates BoN Sampling
Candidates Sampling
Candidates
Sampling
(a) Implicit PRM (DPO).
Mistral-7B-Inst-aa
Llama-3.1-88B 3-88-nntt
Llama-3.1-70B-Hl 1--70---nntt
Resp
Reep
55.0
72.0
24.0
Reep
Accuracy (%)
Resp
Accuracy (%)
70.0
20.0
51.0
Accuracy
68.0
48.0
47.0
12.0
45.0
8.0
29
21
2
2
24
2
43.0
20
2
2
2
24
2
2
62.0
2
2
2
2
2
2
Candidates BoN Sampling
# Candidates Sampling
Candidates BoN Sampling
(b) Implicit PRM (CE). Note that one repsonse per instruction is the extreme case of the unpaired setup.
Figure 5: Scaling responses number for each instruction. Our implicit PRM generally benefits from scaling up the number of responese for each instruction. Particularly, DPO is under-trained with two responses per instruction. This can be partly attributed to the insufficient amount of instructions: two responses may not constitute pair to train our DPO variant, and thus many instructions can not used in training. In contrast, CE generally performs better with insufficient data and can always improve different generation model, even when it is trained with one response per instruction with pairs.
Results We present the GPU time costs A100 80G relative to that of the generation model in Table 3. We find that the inference overhead from generation model takes a large proportion of the total overhead, especially when the generation model is much larger than the reward model. Therefore, the overall inference overhead of baselines on three test sets 66.6%, 70.8%, and 90.9% of that of ours, respectively. It noteworthy that the extra overhead intro- duced by the reference model becomes the generation model b larger, and is almost negligible when Llama-3.. 1uma--31170B.
efficient implementation of proposition by removing the reference model. We examine its effectiveness and explore if we can simply our method to reduce the inference overhead in practice.
Setup this end, we explore two model training config- urations: parameterizing the outcome reward either with or without reference model. We then apply both models to best-of-N sampling and evaluate whether including the reference model has any impact to the performance. We also compare to directly using Llama-3. 3.1- the refer- ence model in our Implicit PRM previous experiments, as the reward model. It serves as controlled baseline without any RM training on our data, but has undergone preference learning (Meta, 2024).
marginal
serves as the generation model.
5.3.2. THE REFERENCE MODEL CAN BE REMOVED AT INFERENCE CERTAIN CASES
We note that our proposition still holds under uniformly distributed reference model, i.e. log नात consland. In best-of-N sampling, only relative scores between steps or responses matter, where the constant log गet can be canceled out, equivalent to exclude the reference model in reward parametrization. Therefore, we derive more
Results Surprisingly, no performance degradation is ob- served when the reference model is ablated in both training and inference, suggesting a more practically efficient variant of our approach. Besides, Llama-3.1 I-IItrurct achieves strong performance too. This potentially explains why the
8

---

Free Process Rewards without Process Labels
Table 4: Ablating reference model in both training and inference. Neither consistently hurts our implicit PRM. More surprisingly, the reference model, Llama-3.1-8P Instruct, already perfroms well on Best-of-N sampling.
Setup
Mistral-7B-Ins-v0..2
Llama-3. I-SB-Inst
Llama-3.
1.taaa...---B--. 1-70B-Isst
Avg.
Train
Inference
@4
@ 16
@64
@4
@
16
@64
@4
@ 16
@64
1amma-3.--B-Inntruce
w/o Ref
14.8
16.2
18.4
49.0
50.4
52.2
69.6
71.0
71.0
45.8
w/ Ref
18.6
24.4
28.8
54.0
55.4
57.0
71.8
71.2
72.2
50.4
+ DPO w/ Ref
w/o Ref
17.8
23.4
27.8
54.2
56.6
57.6
71.6
73.6
73.2
50.6
w/ Ref
17.8
23.4
28.4
54.0
55.2
57.6
70.6
72.0
73.2
50.2
+ DPO w/o Ref
w/o Ref
17.4
22.6
25.6
54.8
56.4
58.2
70.4
73.2
74.0
50.3
reference model can be removed: The reference model is already capable of appropriately assigning high rewards to "good" steps and low ones to "bad" steps. Recall the process reward is Ei Blog m
Implicit Reward Implicit reward has already been widely adopted in preference learning. Most cxisting works focus on applying these algorithms to align models on top of su- pervised fine-tuning (Rafailov al., 2023; Azar et al., 2024; Ethayarajh al., 2024; Chen et al., 2024; Rosset al., 2024; Wu et al., 2024); recent work also tries to leverage the implicit rewards of resulting models as outcome rewards (Lambert et al., 2024; Zhong et al., 2024; Hosseini et al., 2024). Further, following Rafailov et al. (2024), which showed that DPO can automatically learn a Q function, Qiu et al. (2024) devise a self-guided decoding algorithm limited for DPO models leveraging such property. However, despite these applications of adopting DPO models off-the-shelf reward models or Q functions, none of existing work specif- ically targets improving such ability or investigating how to derive decent PRMs upon those off-the-shelf models.
((wllly- (x)) TTe < (y <i). Intu- itively, good step might receive high probabilities by both To and Tref, and therefore lowering its reward; the other hand, bad step might receive low probabilities by both, thereby increasing its reward. This creates confusion to the PRM. We argue that this behavior is actually beneficial during RL training: when the reference model Fee already performs well on certain actions, smaller rewards and consequently smaller policy gradients prevent over-training the policy model V these already-optimized actions. Nevertheless, it is undesired on such inference-tine response selection tasks. This suggests that our Implicit PRM is particularly appealing in practice, since most of the time practitioners will build their PRMs from a strong reference model such as -Jama-s.--BB-Instruee. In such cases, ref can be dropped in inference without hurting the performance the above results suggest, and our approach can achieve stronger performance than baselines with substantially cheaper training, without introducing any additional inference overhead.
7. Conclusion
We start with theoretical proposition demonstrating that parameterizing the outcome reward as the log-likelihood Tsly) ratios of the policy and reference models log Tuy PRM can be intrinsically learned at the same time without any extra training requirements. We discuss its universality to instantiate different training objectives. In experiments, we demonstrate that various implicit reward modeling objec- tives outperform baselines on MATH, with substantially bet- ter trade-offs between accuracy and development overhead, particularly the CE loss. The performance of implicit PRMs can be further improved with majority voting. Further, seal- ing up instructions and responses benefit implicit PRM, with the latter having larger effect, but instructions should be relevant to downstream tasks while the diversity of re- sponses does not bring gains. Surprisingly, training on extra Math-Shepherd step labels brings no further improvements to our implicit PRM trained on only outcome data.
6. Related Work
Complex Reasoning of LLMs Complex reasoning has become key capability of Large Language Models (LLMs) yet remains challenging even to state-of-the- ahr ones (Jimenez et al., 2024; Tian et al., 2024). Various tech- niques have been explored to improve LLMs on reasoning throughout different stages of their lifecycles, such as pre- training (Azerbayev al., 2024; Paster al., 2024: Li et al., 2023), post-training (Luo al., 2024b; Yue al., 2024; Yuan et al., 2024; Meta, 2024; Ouyang al., 2022), and inference (Wei et al., 2022; Fu et al., 2023; Hao et al., 2023; Lightman et al., 2023). Among them, the process reward model. has attracted recent attention for its effectiveness in a variety of settings (Lightman et al., 2023).
9

---

Free Process Rewards without Process Labels
Impact Statement
Hao, S., Gu, Y., Ma, H., Hong. J. J., Wang, Z., Wang. D. Z., and Hu, Z. Reasoning with language model is planning with world model. EMNLP. 2023.
This work aims to make process reward models (PRMs) generally more accessible by reducing training cost, thus promoting the advancement of inference time scaling and reinforcement learning. There may not be specific ethical consequences that are worthy of being particularly discussed here.
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S.. Tang, E., Song, D. X., and Steinhardt, J. Measur- ing mathematical problem solving with the math dataset. ArXiv, 2021.
References
Hosseini, A., Yuan, X., Malkin, N., Courville, A. C., Sor- doni, A., and Agarwal, R. V-star: Training verifiers for self-taught reasoners. COLM, 2024.
Azar, M. G., Rowland, M., Piot, B., Guo, D., Calandriello, D., Valko, M., and Munos, R. A general theoretical paradigm to understand learning from human preferences. International Conference on Artificial Intelligence and Statistics, abs/2310.12055 2024.
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de Las Casas, D., Bressand, F,, Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.- A., Stock, P., Scao, T. L.,, Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7b. ArXiv, abs/23 2023.
Azerbayev, Z., Schoelkopf, H., Paster, K., Santos, M. D., McAleer, S. M., Jiang, A. Q. Deng, J., Biderman, S., and Welleck, S. Llemma: An open language model for mathematics. ICLR, 2024.
Jimenez, E., Yang. J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. Swe-bench: Can language mod- els resolve real-world github issues? ICLR, 2024.
Cao, M., Shu, L., Yu, L., Zhu, Y., Wichers, N., Liu, Y., and Meng, L. Enhancing reinforcement learning with dense rewards from language model critic. In EMNLP, 2024.
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Chan, J., Sun, H., Holt, S., and van der Schaar, M. Dense reward for free in reinforcement learning from human feedback. ICML, 2024.
Chen, H., He, G., Yuan, L., Cui, G., Su, H., and Zhu, J. Noise contrastive alignment of language models with explicit rewards. ArXiv, abs/2402.05369, 2024.
Lambert, N., Pyatkin, V., Morrison, J. D., Miranda, L. V., Lin, B. Y., Chandu, K. R., Dziri, N., Kumar, S., Zick, T., Choi, Y., Smith, N. A., and Hajishirzi, H. Rewardbench: Evaluating reward models for language modeling, ArXiv, abs/2403. 137877 2024.
Cui, G., Yuan, L., Ding, N., Yao, G., He, B., Zhu, W., Ni, Y., Xie, G., Xie, R., Lin, Y., Liu, Z., and Sun, M. Ultrafeedback: Boosting language models with scaled ai feedback. In /CML, 2024.
Leike, J., 2024. URL https://x.com https://x..com//alllkee status /1/2/ 94 0140114 10000000039993--511
Ding, N., Chen, Y., Xu, B., Qin, Y., Zheng, Z., Hu, S., Liu, Z., Sun, M., and Zhou, B. Enhancing chat language mod- els by scaling high-quality instructional conversations. arXiv preprint a 23354:2305.14233, 2023.
Li, R., Allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, Chim, J., Liu, Q.. Zheltonozhskii, E.,, Zhuo, T. Y., Wang, T., Dehaene, O., Davaadorj, M., Lamy- Lairieoi J., Monteiro, J., Shliazhko, O., Gontier, N., Meade, N., Zebaze, A. R. Yee, M.- H., Umapathi, L. K., Zhu, J., Lipkin, B., Oblokulov, M., Wang. Z., Murthy, R., Stillerman, J., Patel, S. S., Abulkhanov, D., Zocca, M., Dey, M., Zhang, 2., Fahmy, N., Bhattacharyya, U., Yu, W., Singh, S., Luccioni, S., Villegas, P., Kunakov, M., Zhdanov, F., Romero. M.. Lee. T., Timor, N., Ding. J., Schlesinger, C., Schoelkopf, H.. Ebert, J., Dao, T., Mishra, M., Gu, A., Robinson, J., Anderson, C. J., Dolan-Gavit,, B., Contractor, D., Reddy, S., Fried, D., Bahdanau, D., Jernite, Y., Ferrandis, C. M., Hughes, S. M., Wolf, T., Guha, A., von Werra, L., and de Vries, H. Starcoder: may the source be with you!
Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kiela, D. Kto: Model alignment as prospect theoretic optimization. ICML, 2024.
Fu, Y., Peng, H., Sabharwal, A., Clark, P., and Khot, T. Complexity prompting for multi-step reasoning. ICLR, 2023.
Gugger, S., Debut, I.., Wolf, T., Schmid, P., Mueller, Z., Mangrulkar, S., Sun, M., and Bossan, B. Ac- celerate: Training and inference at seale made sim- ple, efficient and adaptable. huggi face accelerate, 2022.
com/
TMLR, 2023.
10

---

Free Process Rewards without Process Labels
Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let's verify step by step. ArXiv, 2023.
Rosset, C., Cheng, C.-A., Mitra, A., Santacroce, M., Awadal-
lah, A., and Xie, 'T. Direct nash optimization: Teaching language models to self-improve with general preferences. ArXiv, abs/2404.03715 2024.
Liu, C. Y., Zeng, L., Liu, J., Yan, R., He, J., Wang, C.,, Yan, S., Liu, Y., and Zhou, Y. Skywork-reward: Bag of tricks for reward modeling in llms. arXiv preprint arXiv:2410. 18451, 2024.
Setlur, A. R., Nagpal, C., Fisch, A., Geng, X., Eisenstein, J., Agarwal, R., Agarwal, A., Berant, J., and Kumar, A. Rewarding progress: Scaling automated process verifiers for Ilm reasoning. 2024.
Lu, J., Dou, Z., Wang, H., Cao, Z.. Dai, J., Wan, Y., Huang, Y., and Guo, Z. Autopsy: Automated process-supervised verifier. ArXiv, abs/2405.16800 2024.
Snell, C., Lee, J., Xu, K., and Kumar, A. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters. ArXiv, abs/2408.023 14, 2024.
Luo, L., Liu, Y., Liu, R., Phatale, S., Lara, H., Li, Y., Shu, L., Zhu, Y., Meng, L.., Sun, J., and Rastogi, A. Improve mathematical reasoning in language models by automated process supervision. ArXiv, abs/2406.06592, 2024a.
Tian, M., Gao, L., Zhang, S. D., Chen, X., Fan, C., Guo, X., Haas, R., Ji, P., Krongchon, K., Li, Y., Liu, S., Luo, D., Ma, Y., Tong, H., Trinh, K., Tian, C., Wang, Z., Wu, B., Xiong, Y., Yin, S., Zhu, M., Lieret, K., Lu, Y., Liu, G., Du, Y., Tao, T., Press, O., Callan, J., Huerta, E. A., and Peng. H. Scicode: A research coding benchmark curated by scientists. Arxiv, 2024.
Luo, Z., Xu, C., Zhao, P. Sun, Q.. Geng, X., Hu, W. Tao, C., Ma, J., Lin, Q., and Jiang, D. Wizardcoder: Empowering code large language models with evol-instruct. ICLR, 2024b.
Touvron, H.,, Martin, L., Stone, K. R., Albert, P., Almahairi, A., Babaci, Y.. Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D. M., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn A. S., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I. M., Korenev, A. V., Koura, S., Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y.. Martinet, X., Mihaylov, T., Mishra, P., Molybog. I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X., Tang, B., Taylor, R., Williams, A., Kuan, X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T. Llama 2: Open foundation and fine-tuned chat models. ArXiv, abs/2307. 009288, 2023.
Mahan, D., Phung. D., Rafailov, R., Blagden, C., Lile, N., Castricato, L., Franken, J.-P., Finn, C., and Albalak, A. Generative reward models. 2024.
Meta. Llama 3 model card. Github, 2024. URL https://github comm/mta--ie aelalaaaallii /11ama3// mioob/main/MODE5. CARD.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L. E., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., and Lowe, J. Training language mod- els follow instructions with human feedback. ArXiv, abs/2203.02155. 2022.
Paster, K., Santos, M. D., Azerbayev, Z., and Ba, J. Open- webmath: An open dataset of high-quality mathematical web text, 2024.
Wang, H., Xiong, W., Xie, T., Zhao, H., and Zhang, T. Inter- pretable preferences via multi-objective reward modeling and mixture-of-experts In EMNLP, 2024.
Qiu, Lu, Y. Zeng, Y., Guo, Geng, J., Wang, H., Huang, K., Wu, Y., and Wang, M. Treebon: Enhancing inference- time alignment with speculative tree-search and best-of-n sampling. 2024.
Wang. P., Li, L., Shao, Z., Xu, R., Dai, D., Li, Y., Chen, D., Y. Yu, and Sui, Z. Math-shepah Math-shepherd: Verify and reinforce Ilms step-by-step without human annotations. ArXiv, 2023.
Rafailov, R., Sharma, A., Mitchell, E.,, Ermon, S., Manning, C. D., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. NeurlPS, 2023.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., hsin Chi, H., Xia, F., Le, Q., and Zhou, Chain of thought prompting elicits reasoning in large language models. NeurIPS, 2022.
Rafailov, R., Hejna, J., Park, R., and Finn, C. From r to q": Your language model is secretly a q-fumction. ArXiv, 2024.
Wu, Y., Sun, Z., Yuan, H., Ji, K., Yang. Y., and Gu, Q. Self-play preference optimization for language model alignment. ArXiy, abs/2405.00075. 2024.

---

Free Process Rewards without Process Labels
Yuan, L., Cui, G., Wang, H., Ding, N., Wang, X., Deng, J., Shan, B., Chen, H., Xie, R., Lin, Y., Liu, Z., Zhou, B., Peng, H., Liu, Z., and Sun, M. Advancing IIm reasoning generalists with preference trees. ArXiv, 2024.
Yue, X., Zheng, T., Zhang, G., and Chen, W. Mammoth2: Scaling instructions from the web. NeurIPS, 2024.
Zhang, H., Wang, P., Diao, S., Lin, Y., Pan, R., Dong, H., Zhang, D., Molchanov, P., and Zhang, T. Entropy- regularized process reward model, 2024a.
Zhang, K., Zeng, S., Hua, E., Ding, N., Chen, Z.-R., Ma, Z., Li, H., Cui, G., Qi, B., Zhu, X., et al. Ultramedical: Building specialized generalists in biomedicine. arXiv preprint arXiv:2406.0328 03949, 2024b.
Zhang. L., Hosseini, A., Bansal, H., Kazemi, M., Kumar, A., and Agarwal, R. Generative verifiers: Reward modeling as next-token prediction. 2024c.
Zhong, H., Feng, G., Xiong, W., Zhao, L., He, D., Bian, J.. and Wang, L. Dpo meets ppo: Reinforced token optimization for rlhf. ArXiv, 1e//2404.18922 2024.
Zhu, B., Frick, E.. Wu, T., Zhu, H., and Jiao, J. Starling- 7b: Improving Iim helpfulness & harmlessness with rlaif, November 2023.
12

---

Free Process Rewards without Process Labels
A. Proof of Proposition
Proposition A.1. Consider an ORM where the reward is parameterized by the log-likelihood ratio of two causal LMs, i.e.
roly)= == 3 log Toly), Define o == Ei= 8 Ta(sklyes), "
Follh 43 is the exponential average T& at step t.
(y ve) = 8 log E se tralr)
(6)
Proof. The Proposition can be proven using mathematical induction.
Suppose response y has T tokens.
(1) For Vt < T. if ++1 (y<(+1, B log Barrly (544 ( holds, then (y ve) 3 log Em by (e would also hold.
frott)
(2) At T. (M)): = =r = =3 log Bm. (e (
proof of (1):
Blog Ey (ot -llog E F sem
log E ( e (y-cr+1.1++))
Blog B (y (x 1y
Tee Y<i)
( Vei)
(ve+1 15
log
E marly l1)s))
T M/ y <i)
nea 3/ yr(e)
(T ly
To Ut+ (1 (c Teer (y
log
Thee Tree(Mlllyss
Ther 9/: y
36+1
Blog
( ly Teet (Baa (x
Σ (o(
x=1
To ( y
3 log
Trer y/ ye
proof of (2):
The conclusion is straightforward. Since π is autoregressive, we have
ro(y) = log Tyly) (
(v:ly
To(yilyei)
B log
Σ 3 log
Feat U/ cic))
3 (y
Since ysr y, the expectation Emarlyyyrr can be removed:
3 log En tro = = log rror)) = =ro
B. Frequently Asked Questions
B.I. Why does the implicit reward not include a baseline Z(X)?
Our reward representation does not need a baseline, which differs from the implicit reward defined in Rafailov et al. (2023;
2024). The implicit reward in Rafailov et (2023; 2024) derived from the entropy-regularizer RI. framework, and thus
13

---

Free Process Rewards without Process Labels
(X) must be included to ensure the optimality of the trained policy model. However, this the case in our paper. Instead, aim for reward representation that enables tractable value. We do not target optimal policy in the engropy-regularrizd RL framework; Rather, our reward representation is defined and constructed from scratch, namely, any representation is acceptable as long as it gives tractable way to estimate the Q value and makes Eq. (2) hold. Therefore, we do not need to follow the restrictions as Rafailov et al. (2023; 2024), and our reward representation does not necessarily relate to theirs.
Moreover, a baseline term were added, one can prove that the following equation would hold anymore, with the proof done by simply substituting the new reward representation (with (X)) the right-hand of the equation:
y< 1) (yt ys y< t)
(ys
3 log E (ylys)) [
log
B.2. What are the advantages of CE loss compared to DPO?
As shown in Figure and Figure 6, Implicit PRM with CE loss is more data efficient, while showing better performance when integrated with majority vote, presenting as an appealing alternative in practice as in many scenarios pair-wise data is hard to collect. Also, CE loss only requires one example for forwarding and backwarding which reduces memory overhead in RL training. Therefore, the generalization unpaired losses remains valuable compared pair-wise DPO in more catarronnt settings.
C. Further Experiments
Majority Vate
w/o Majority
w/ Majority
51.8
51.9
51.5
Accuracy (%))
50.4
49.4
48.4
46
45.7
KTO
NCA
DPO
CE
Figure 6: Results with majority voting. We present the averaged best-of-N accuracy across three testsets.
C.1. Incorporating Majority Voting
Our Implicit PRMs can be integrated with majority voting to improve the performance even further. Previously, we apply our Implicit PRMs score each response and pick the response with highest individual score as the final answer. However, when incorporating with majority voting, the scores of responses that lead to the same answer will be aggregated and the answer with the highest aggregated score will be selected the final answer. We present the results averaged over different numbers of candidate solutions per problems across all three generated models in Figure 6.
We observe that our Implicit PRM can successfully adjust voting distributions, and achieves better results than using the Implicit PRM or majority voting separately. Particularly, KTO and CE variants gain the most from the integration, both of which fail to surpass majority voting alone but outperforms it through weighted best-of-N. is also noteworthy that CE loss become the most effective when augmented with majority voting, once again demonstrating its potential.
C.2. Are There Any Other Factors can Improve Implicit PRM Performance?
We consider potential factors that may influence the performance Implicit PRMs, as listed below:
14

---

Free Process Rewards without Process Labels
Table 5: Factors that may affect PRM performance. To our surprise, none of them consistently improve our implicit PRM.
Mistral-7B-IInt-vo00
Llama-3.1-70B-I
Sctup
Avg.
@4
@ 16
@64
@4
@ 16
@64
@4
@ 16
@64
Implicit PRM
18.6
24.4
28.8
54.0
53.4
57.0
71.8
71.2
72.2
49.3
+ UltraFeedback
19.4
24.4
29.0
53.8
55.0
55.8
71.6
70.6
72.2
49.2
+ Ultralnteract (Code)
19.2
24.6
28.0
54.6
54.0
56.8
71.4
70.8
70.0
49.2
+ Dedup.
18.2
22.8
26.8
52.0
53.2
51.6
69.8
69.4
70.4
47.6
+ Base Resp.
17.8
23.2
27.6
54.0
55.0
54.8
71.4
72.4
73.2
48.7
+ Step Label
18.8
25.4
28.8
53.8
54.8
54.6
70.8
71.2
73.0
49.2
Task-irrelevant Instructions We previously only consider math instructions. We now examine if increasing instructions diversity, even if the instructions are irrelevant to downstream tasks, can benefit Implicit PRMs. To this end, we incorporate general instructions from UltraFeedback (Cui et al., 2024) and coding instructions from Ultralnteract (Yuan et al., 2024) into our training dataset. We directly use responses from the original datasets, but for UltaFeeddack we only randomly select one pair for each instruction, instead of using all the pairs.
Response Diversity We first conduct a deduplication on our preference dataset based on 8-gram overlap, aiming to verify repeated responses hurt model performance. We then randomly replace four rollouts per instruction the original training dataset with another four rollouts generated by Llama-3, 1-8B-BBBsst model.
Training Step Labels Our Implicit PRMs do not require step labels for training. However, are interested in exploring whether augmenting them with step labels can further improve their performance. Based on the definition of process labels, we adjust the implicit reward of step by increasing it for positive labels and decreasing it for negative ones. We use the labels obtained from our implemented Math-Shepherd, which has been demonstrated to be strong implementation with step labels of high-quality ($4). We adapt KTO to a step-level version for optimization. Therefore, considering a n-step response with step labels {?2, 12, ...,, }"), we conduct second stage training on our current Implicit PRM explicitly optimize the implicit reward: L.
E log (1* (r))))
Results We present results on Implicit PRM (DPO) in Table 5. In general, none of these factors brings consistent gains. (1) Both adding UltraFeedback and Ultralnteract (code) instructions hurt the performance, with the former suffers more severely. This implies that training instructions deviating from the downstream task could undermine the performance of Implicit PRMs. (2) Regarding response diversity, we observe that the performance of deduplicating responses hurts the performance and close to Implicit PRMs trained on simílar amount of data. This indicates that repeated responses function similarly others and are still beneficial before model performance saturates. Replacing part of original rollouts with those generated by the base model also fails improve performance. (3) Conducting step-level KTO with extra process labels does not bring gains, reinforcing However, one should be cautious about concluding that stepwise labels are generally not helpful due to two factors in our experiments: Firstly, despite our efforts that lead to improved step annotation quality compared to previous work, the MCTS- based approach inevitably introduces noises in the data annotation process, as we discussed $2; Secondly, our choice algorithm may not be optimal. is possible that more advanced PRM data annotation methods and training algorithms can finally integrate information from (noisy) stepwise labels into Implicit PRM.
claim that
already train strong PRM without process label.
15


Reviews:
[{"review_id": "i8vaaPCUfu", "reviewer": "Reviewer_nJ5A", "summary": "Verifiers, such as process reward PRMs and ORMs, evaluate LLMs' partial or full responses, providing feedback and pushing the boundaries of LLMs' ability to solve complex reasoning tasks. . PRMs provide better, fine-grained feedback than ORM based on the nature of their training procedure. However, training PRMs is more challenging than training ORMs because it requires annotating every intermediate step. This paper argues that a strong PRM can be derived at no additional cost from training an ORM. The observation is that using the closed-form solution of the reward model to learn a reward and policy jointly, similar to the DPO method, a PRM can be automatically learned during training. The paper extends this idea beyond DPO-style algorithms by incorporating cross-entropy (non-binary objectives). The paper also experiments with three different LLMs, along with several variants of optimization objectives based on DPO, demonstrating the robustness of this approach. The paper also includes several ablation studies highlighting the importance of increasing the number of responses per instruction and scaling the number of instructions.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback and are glad that you find the method is suitable for the problem and the experiments and analysis are sound. Here are our responses.\n\n> the proof appears to have some issues. Furthermore, the derivation of cross-entropy loss is missing the normalization constant Z(x).\n\n**A**: Thanks for the careful reading, but we believe there are misunderstandings. \n\nAs the reviewer uses different notations and does not mention the line of proof, we would like to kindly ask the reviewer for clarification on the specific steps in the proof, so that we can better address the concern.\n\nRegarding the baseline $Z(X)$ in the cross entropy objective: Please note that our reward representation does not need a baseline $Z(X)$, which is different from the implicit reward defined in [2,3]. The implicit reward in [2,3] is derived from the entropy-regularized RL framework, and a $Z(X)$ must be included to ensure the optimality of the trained policy model. However, this is not the case in our paper. Instead, we aim for a reward representation that enables tractable Q value; we do not target and never claim an optimal policy in the entropy-regularized RL framework. Rather, our reward representation is **defined** and **constructed from scratch**, namely any representation is acceptable as long as it gives a tractable way to estimate Q value and makes Eq. (2) in line 137 hold. Therefore, we do not need to follow the restrictions as [2,3] and our reward representation does not necessarily relate to theirs.\n\nMoreover, if a baseline term $Z(X)$ were added, one can  prove that **the following equation would not hold** anymore, with the proof done by simply substituting the new reward representation (with a $Z(X)$) to the right-hand of the equation: \n\n$$\n\n\\sum_{i=1}^{t} \\beta \\log \\frac{\\pi_\\phi(y_i \\mid \\mathbf{y}_{<i})}{\\pi_\\text{ref}(y_i \\mid \\mathbf{y}_{<i})} = \\beta \\log \\mathbb{E}_{\\pi_\\text{ref}(\\mathbf{y} \\mid \\mathbf{y}_{\\leq t})} \\left[ e^{\\frac{1}{\\beta} r_\\phi(\\mathbf{y})} \\right]\n\n$$\n\nI.e., proposition 3.1 will not consistently hold for all ORM objectives if we follow the reward representation as [2,3], and we won’t be able to find a tractable Q value.\n\nWe plan to add this discussion in the next version.\n\n> How do you manage the normalization constant in the cross-entropy (CE) update?\n\n**A**: Please refer to the response on $Z(X)$ above. As stated in Proposition 3.1, we do not need a baseline in our reward representation.\n\n> The paper concludes that the implicit DPO reward outperforms the other objectives studied. Implicit DPO was originally proposed in [1]. Given this context, it is unclear what novelty this paper offers, as the best algorithm was introduced in previous work.\n\n**A**: First, [1] directly adopts the theory from [3] which only applies to DPO. However, as discussed above, our work also provides a more general proposition with solid theoretical contribution, applying to any ORM objectives including CE loss as long as they use our reward representation, compared to both [1] and [3] that are tailored to DPO. Compared to these previous works, this paper provides a fresh and more general perspective of implicit rewards, which we believe holds significant value.\nSecond, as shown in Figure 5 and Figure 6, Implicit PRM with CE loss is more data efficient than its DPO counterpart while showing better performance when integrated with majority vote, presenting as an appealing alternative in practice as in many scenarios pair-wise data is hard to collect. Also, the CE variant  requires only one example for forwarding and backwarding while the DPO variant has to consider a pair of  examples at the same time. As a result, the CE variant reduces memory overhead in RL training, as observed by a recent work that directly adopts our method (published after the ICML submission deadline) [3]. Therefore, we believe our generalization to unpaired losses holds great theoretical and practical merits compared to previous works. \n\n> In Figure 4, why does increasing the number of scaling instructions hurt performance?\n\n**A:**  We’d like to clarify: for most cases, increasing the number of instructions improves model performance. The only exception is using all instructions to train 8B-sized PRMs and test on Llama-3.1-70B-Inst. As the increased number of instructions come with Llama-3.1-8B-Instruct generated responses, we conjecture that the performance drop can be attributed to PRMs overfitting to responses from the small model and being unable to generalize to larger models. This discussion will be added in the revision.\n\n[1] Treebon: Enhancing inference-time alignment with speculative tree-search and best-of-n sampling Qui et al. 2024\n\n[2] Direct Preference Optimization: Your Language Model is Secretly a Reward Model. Rafailov et al. 2023.\n\n[3] From r to Q∗: Your Language Model is Secretly a Q-Function. Rafailov et al. 2024.", "claims_and_evidence": "Below are the claims and evidence presented in the paper:\n\n- The paper claims that PRM performs better than ORM. They support this claim by referencing other studies in the literature that demonstrate this advantage and conducting experiments to compare both PRM and ORM.\n\n- Another claim is that PRM can be easily learned by optimizing a DPO-style objective, which leverages the closed-form solution of the reward model. The paper provides empirical evidence showing that their implicit PRM performs well in practice compared to ORM and is competitive with traditionally trained PRM.", "methods_and_evaluation_criteria": "Yes, the proposed method and evaluation criteria are suitable for the problem and application at hand. The authors aim to train the PRM more efficiently to address downstream tasks that require complex reasoning, such as math instruction datasets and chat instruction-following tasks.", "theoretical_claims": "I've reviewed the theoretical claims, and the proof appears to have some issues, but I could be wrong. The authors assert that $E_x[f(x) g] = E_x[f(x)] g$, where $g = E_{y<t+1}$ and $f(x) = E_{y_t < y_t}$. This assertion is problematic because $g$ is not a constant. Additionally, the second step of proof (1) seems to have some flaws as well. Furthermore, the derivation of cross-entropy loss is missing the normalization constant Z(x). The reason for DPO to minimize the pairwise loss is to cancel the intractable normalization constant.", "experimental_designs_or_analyses": "The experimental design and analyses are sound. The paper proposes to study their idea across several models to demonstrate its generality. Additionally, the paper provides several key ablation studies to identify where the performance gains are coming from in the proposed approach.", "relation_to_broader_scientific_literature": "The key contribution of the paper relates to the broader scientific literature by focusing on improving the efficiency of training Probabilistic Relational Models (PRM). The authors observe that PRMs outperform Ordinary Relational Models (ORMs), but the drawback is that they are costly to train. Their proposed solution aims to reduce the complexity of PRMs, enabling large language models (LLMs) to learn more efficiently since PRMs provide more informative data.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "None", "questions_for_authors": "Below is list of questions that I have:\n- DPO optimizes pairs to eliminate the intractable computation of the normalization constant. How do you manage the normalization constant in the cross-entropy (CE) update? If you are making assumptions about the normalization constant, what assumption are you making, and why is it feasible?\n\n- The paper concludes that the implicit DPO reward outperforms the other objectives studied. However, implicit DPO was originally proposed in [1], where it was used to learn a PRM and apply it during inference. Given this context, it is unclear what novelty this paper offers, as the best algorithm was introduced in previous work, and the single-sample algorithm has issues with the normalization constant.\n\n- In Figure 4, why does increasing the number of scaling instructions hurt performance?\n\n\n\n[1] Treebon: Enhancing inference-time alignment with speculative tree-search and best-of-n sampling Qui et al. 2024", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "None"}}, {"review_id": "TK9p75lvEq", "reviewer": "Reviewer_kaar", "summary": "This paper introduces a method to create a process reward model (PRM) without the need for expensive step-by-step annotations. The authors propose that an implicit PRM can be derived by training an ORM using only response-level labels, by parameterizing the outcome reward as a log-likelihood ratio between the policy and reference models. Their experiments on MATH show that this implicit PRM performs better than a strong MCTS-based baseline (Math-Shepherd) while using significantly less training data. The model’s performance further improves with majority voting, and scaling up instructions and responses boosts effectiveness, with responses contributing more to improvements.\n\n## update after rebuttal\nI have read the author responses, and my evaluation remains the same. I feel the proposed dense rewards need to be tested as dense rewards for online RL, or at lease beam search at test-time (common in recent literature as I mention below), to know if they are empirically truly effective in improving search efficiency.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments. Here are our responses.\n\n> The true test of PRM would be to run beam search or to use it as dense rewards for online RL.\n\n**A:** We choose best-of-N as our setup because it is a standard practice in recent literature and presents as a valuable approach for inference-time scaling [1,2,3]. Testing PRMs for online RL is of course valuable, but can introduce additional confounding factors and much more overhead. This paper aims to isolate the quality of the reward model, so best-of-N sampling provides relevant and convincing evidence for our claim.\n\nThat being said, we notice that a recent work [4] after the ICML ddl closely resembles the reviewer's suggestion, which applies our implicit PRM in online RL and achieved strong performance and sample efficiency across various benchmarks. We think this proves our approach's potential in online RL.\n\nWe are now running the beam search experiments that the reviewer suggested. They are slow and we will provide update as soon as possible.\n\n[1] Let’s Verify Step by Step. Lightman et al. 2023.\n\n[2] Math-shepherd: Verify and reinforce llms step-by-step without human annotations. Wang et al. 2023.\n\n[3] Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters. Snell et al. 2024.\n\n[4] Process Reinforcement through Implicit Rewards. Cui et al. 2025.\n\n> The gains are small on poorer models like Mistral-7B... It seems that the PRM is not very useful on hard questions.\n\n**A:** This might be  a misinterpretation of Table 1.  In fact, our approach achieves very strong performance on these questions.\n\nWe note that the pass@1 performance of each generation model is highlighted in the header and the caption, and the first column for each model denotes the best-of-4 accuracy and is already much higher than pass@1. We should compare the best-of-N performance to pass@1 to see the absolute gain of each model. \n\nContrary to the reviewer’s observation, the gains on Mistral-7B are actually the largest, with the pass@1 being 9.6% while best-of-64 increased to 28.8% for our DPO model. Compared to our implemented Math-Shepherd and AutoPSV, we achieved significant average improvements of nearly 3% and 5%.\n\nAs for the reviewer’s second point: a 5% average improvement on the MATH benchmark is challenging to achieve: contextualizing it within recent impactful literature [2, 5], a <3% improvement on best-of-64 and 2% improvement compared to baselines is usually considered substantial. Therefore, the average performance gains of our implicit PRM demonstrate its effectiveness on these challenging questions. \n\nWe also add comparisons to ORMs below; both are trained with the same data. The trend is consistent with that in the paper and we observe strong performance of our Implicit PRM.\n||||Mistral-7B-Inst-v0.2|||Llama-3.1-8B-Inst|||Llama-3.1-70B-Inst|Avg.|\n|---|---|---|---|---|---|---|---|---|---|---|\n||@4|@16|@64|@4|@16|@64|@4|@16|@64||\n|ORM|18.6|23.8|26.4|50.8|50.2|52.8|68.8|71.0|69.4|48.0|\n|ImplicitPRM(DPO)|18.6|24.4|28.8|54.0|55.4|57.0|71.8|71.2|72.2|50.4|\n\n[5] Improve mathematical reasoning in language models by automated process supervision. Luo et al. 2024.\n\n> How to optimally scale instructions and responses\n\n**A:** According to our experiments in section 5.1, scaling up instructions from the same tasks to downstream tests and scaling up responses are both is helpful. We did not test data balance for the DPO objective, but we did observe benefits of that for CE loss.\n\n> Does the trained PRM extrapolate\n\n**A:** Yes, Implicit PRM is able to transfer across model families and tests of the same task. For model transfer, all Implicit PRMs in Table 1 are trained from Llama-3.1-8B-Inst with its on-policy rollouts, and existing results have shown their effectiveness on improving Mistral-7B and Llama-3.1-70B-Inst models. For task transfer, we add an experiment to train our model solely on GSM8K and test on MATH as follows, and surprisingly, results show that the overall performance remains comparable to that trained on all instructions, indicating the superior task transferability of Implicit PRM.\n\n||||Mistral-7B-Inst-v0.2|||Llama-3.1-8B-Inst|||Llama-3.1-70B-Inst|Avg.|\n|---|---|---|---|---|---|---|---|---|---|---|\n||@4|@16|@64|@4|@16|@64|@4|@16|@64||\n|ImplicitPRM(DPO)|18.6|24.4|28.8|54.0|55.4|57.0|71.8|71.2|72.2|50.4|\n|ImplicitPRM(DPO on GSM8k)|18.6|24.8|28.2|54.6|54.8|57.0|71.0|71.0|72.6|50.3|\n\n> If Implicit PRM works on other domains\n\n**A:** Though we do not include other tasks in this paper due to our limited capacity and the limited space, after ICML ddl, there are recent works showing that Implicit PRM is helpful in best-of-N sampling on agent tasks [6] and online RL, with substantial gains on coding [4] and function calling tasks [7].\n\n[6] AgentRM: Enhancing Agent Generalization with Reward Modeling. Xia et al. 2025.\n\n[7] Learning to Generate Structured Output with Schema Reinforcement Learning. Lu et al. 2025.", "claims_and_evidence": "The paper claims that 10-30x reduction in the overhead needed to train PRMs, if we instead just train ORMs with re-parameterized rewards as in DPO, and then use the proposed prefix level scores to evaluate intermediate steps. The experiments on Math benchmarks support this claim.", "methods_and_evaluation_criteria": "Yes, the evaluation criteria for BoN follows Lightman et. al.. But more recent works evaluate PRMs with beam search or as dense rewards in RL (see comments below). The benchmarks and models chosen are indeed standard.", "theoretical_claims": "Yes, I checked the validity of Proposition 3.1 in Appendix A.", "experimental_designs_or_analyses": "Yes, the models and benchmarks chosen are valid and sound.", "relation_to_broader_scientific_literature": "The main benefit of the approach proposed in this paper is that the it only requires training an ORM with implicit rewards in order to turn it into a PRM that can score partial generations. In contrast, other prior works like MathShephard, OmegaPRM and PAVs, train PRMs to predict value functions, which requires a larger collection of data, which roughly scales linearly with the sequence length. Though, this paper does not evaluate the trained PRM during beam search or for RL, which is the main way Snell et. al. and Setlur et. al. evaluated and used PRMs to scale inference time compute.", "essential_references_not_discussed": "Improve Mathematical Reasoning in Language Models by Automated Process Supervision Luo et. al.", "other_strengths_and_weaknesses": "Strengths\n- The implicit PRM does not need training or data collection beyond what is needed for an ORM. \n- The analysis in 5.1 seems to suggest that the trained PRMs are indeed data efficient and performance improves as training data is scaled consistently.\n\n\nWeaknesses\n- The paper only evaluates the PRM as an ORM, to re-rank responses during BoN. The true test of PRM would be to run beam search as in Snell et. al. or to use it as dense rewards for online RL, as on Setlur et. al. \n- The gains are small on poorer models like Mistral-7B, and in general, the gains are around 5% when averaged across more performant models on MATH. So, it seems that the PRM is not very useful on hard questions.", "other_comments_or_suggestions": "- L77 typo, parameterizin ---> parameterizing", "questions_for_authors": "- Can the authors provide some discussion on how to optimally scale instructions and responses when collecting the training data for training PRMs? Also, for this, are the samples sampled IID from the base model or balanced across correct and incorrect samples?\n- Does the trained PRM extrapolate, i.e., how well does a PRM train on Mistral7B transfer to LLama and vice-versa, or for the same base model how does it extrapolate from GSM8k --> MATH or vice-versa? \n- Do you think that this has would also work on domains other than MATH, like PRMs for instruction tuning, or coding?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HcVia4bLz2", "reviewer": "Reviewer_JdZn", "summary": "This paper shows that the PRM can be obtained implicitly without additional training by parametarization.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> From the table, the gain delta from the proposed PRM is not that large, and on Mistra-7B is not helping, Besides, the pass@1 is not clear if it helps, would be great to show a curve here.\n\n**A:** This might be  a misinterpretation of Table 1. In fact, our approach achieves very strong performance.\n\n We should compare the best-of-N performance to pass@1(greedy decoding w/o using reward models) to evaluate the effectiveness of the reward models. The pass@1, i.e. best-of-1, of each generation model is highlighted in the header and the caption. The first column for each model denotes the best-of-4 accuracy and is already much higher than pass@1 across the board, confirming the strong performance of our approach. Rows in the “open-source reward models” block  are trained with different and much larger and more engineered data, and therefore are not comparable to ours. Those under “our implementations” are reimplementations of previous works controlling for confounding factors such as the base model and data, for fair comparisons. \n\nContrary to the reviewer’s observation that “on Mistral-7B it is not helping,” the gains on Mistral-7B are actually the largest:  the pass@1 is 9.6% while best-of-64 increased to 28.8% for our DPO model, a 19.2 absolute gain. Improvements are still observed even when we use the 8B-sized PRM to assist 70B-sized generation models, namely in the weak-to-strong setup, evidenced by best-of-64 of 72.2% compared to pass@1 of 63.2%. Importantly, our reported performance improvement on MATH is challenging to achieve: contextualizing it within recent impactful literature [1,2], a <3% improvement on best-of-64 and 2% improvement compared to baselines is usually considered substantial as a significant improvement. We think our Implicit PRMs achieve strong performance in our experiments.\n\nWe also note that why we did not surpass the off-the-shelf baseline RLHFlow-8B-Mistral-Data can be explained by the unfair comparison, due to their advantages from training models using on-policy rollouts. Our evidences include: (1) RLHFlow-8B-Mistral-Data uses **on-policy** rollouts from Mistral-7B while ours use off-policy rollouts from Llama-3.1-8B-Instruct; (2) The same approach with rollouts from DeepSeek models underperforms ours; (3) Our approach outperforms RLHFlow-8B-Mistral-Data on Llama-3.1-8B-Instruct by a large margin.\n\nFollowing the reviewer's suggestion, we plot the curve of pass@N and bes-of-N in this [figures](https://ibb.co/Qvt4GQgM) anonymously. from the figure we can see that our method consistently outperforms baselines when increasing the number of candidate responses (N).\n\n[1] Math-shepherd: Verify and reinforce llms step-by-step without human annotations. Wang et al. 2023.\n\n[2] Improve mathematical reasoning in language models by automated process supervision. Luo et al. 2024.\n\n> Whether this generalize to many other domain.\n\n**A:** Though we do not include other tasks in this paper due to our limited capacity and the limited space, after ICML submission deadline, there are recent works showing that Implicit PRM is helpfull in best-of-N sampling on agent tasks [3], and adopting Implicit PRM for online RL, with brings substantial gains on coding [4] and function calling tasks [5].\n\n[3] AgentRM: Enhancing Agent Generalization with Reward Modeling. Xia et al. 2025.\n\n[4] Process Reinforcement through Implicit Rewards. Cui et al. 2025.\n\n[5] Learning to Generate Structured Output with Schema Reinforcement Learning. Lu et al. 2025.\n\n> Would like to know more about the pass@k from 1 to 16, how the curve looks like, particularly for pass@1, how much it helps. Besides how many runs for this metric, would be great to show mean/std here.\n\n**A:** Thanks for the suggestion. The curve of pass@N and bes-of-N can be found [here](https://ibb.co/Qvt4GQgM).\nFor each experiment we only run for one time since our early experiments show that there is little variance across runs and thus this has been a standard practice mainly due to the cost of the experiments [1,2,6].\n\n[6] Autopsv: Automated process-supervised verifier. Lu et al. 2024.\n\n> The main concern is the empirical gain, if it's really showing the consistent gain. Would be great if this part can be explained more (particularly, why Mistral-7B no gain, and pass@4 for Llama/pass@64 for Llama-70B\n\n**A:** Please see above responses on empirical gains. Particularly, our method has achieved substantial gains (pass@1: 9.6% -> best-of-64: 28.8%) on Mistral-7B rather than no gains, and we only underperform the baseline trained on on-policy Mistral-7B rollouts. Regarding pass@4 for Llama/pass@64 for Llama-70B (actually they are best-of-4 and best-of-64), though we did not achieve the highest accuracy, our performance is very close to the best-performing baseline, only with a difference of 0.4% and 0.8% respectively. We kindly ask the reviewer reevaluate our submission after these clarifications on the empirical gains.", "claims_and_evidence": "From the table, the gain delta from the proposed PRM is not that large, and on Mistra-7B is not helping, Besides, the pass@1 is not clear if it helps, would be great to show a curve here.", "methods_and_evaluation_criteria": "Looks good.", "theoretical_claims": "Looks good, but not an expert to verify the correctness.", "experimental_designs_or_analyses": "Looks reasonable to me, with the ORM and PRM as baselines, and the sufficient base model as well as Reward model to show the ablations and gains.", "relation_to_broader_scientific_literature": "If this generalize to many other domain, would be a big gain for efficient RL scaling for LLM", "essential_references_not_discussed": "Looks comprehensive pile of relevant work.", "other_strengths_and_weaknesses": "The paper is in general a good shape, that not an expert in this domain can still follow the flow easily. Both theoretical and empirical results looks good to me.", "other_comments_or_suggestions": "Would like to know more about the pass@k from 1 to 16, how the curve looks like, particularly for pass@1, how much it helps. Besides how many runs for this metric, would be great to show mean/std here.", "questions_for_authors": "As in comments and above. The main concern is the empirical gain, if it's really showing the consistent gain. Would be great if this part can be explained more (particularly, why Mistral-7B no gain, and pass@4 for Llama/pass@64 for Llama-70B", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GNwGdakDet", "reviewer": "Reviewer_XrJW", "summary": "This paper proposes a new way of training process reward models without expensive fine-grained step-level annotations by training an ORM (reward modeled as log-likelihood ratios of the policy and the reference model) and using it as an implicit PRM. The authors show that their training of the implicit PRM is more data efficient than baseline PRM training strategies while also achieving good performance with Best-of-N sampling on the MATH dataset. Lastly, the paper contains analysis on how the their training strategy scales with different hyperparameters and loss functions for training the implicit PRM.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> [W1] Limited Datasets\n\n**A:** Though we do not include other tasks in this paper due to our limited capacity and the limited space, after ICML submission deadline, there are recent works showing that Implicit PRM is helpful in best-of-N sampling on agent tasks [1], and adopting Implicit PRM for online RL, with brings substantial gains on coding [2] and function calling tasks [3].\n\n[1] AgentRM: Enhancing Agent Generalization with Reward Modeling. Xia et al. 2025.\n\n[2] Process Reinforcement through Implicit Rewards. Cui et al. 2025.\n\n[3] Learning to Generate Structured Output with Schema Reinforcement Learning. Lu et al. 2025.\n\nFollowing reviewer’s suggestion, we also test on other math datasets and results are as follows. The policy model is Llama-3-70B-Instruct, and the baseline is Math-Shepherd in our implementations.\n| Method |  | AMC|(pass@1=36.1)  |   ||AIME|(pass@1=8.9)|\n|--------|:---:|:---:|:---|:---:|---:|:---:|:---|\n|| @4|@16|@64||@4|@16|@64|\n| Math-Shepherd| 39.8|48.2|45.8| |17.8|18.9|17.8|\n| ImplicitPRM(DPO)|41.0|50.6|48.2||20.0|23.3|20.0| \n\n> [W2] Is PRM better than ORM?\n\n**A:** All ORMs in Table 1 are off-the-shelf from HuggingFace and are trained with different data and therefore not comparable to ours. They indeed consume much more data. For example, Eurus-RM-7B uses 287k pairs of data, SkyworkRM uses 465K, ArmoRM uses 1560K, while we only use 263K. Also, ORMs in Table 1 perform well on weak-to-strong settings which increases the average performance, but on Mistral-7B (20.4% vs 28.8% for best-of-64) and Llama-3.1-8B-Instruct (51.8% vs 57.0%), there are significant performance gaps to Implicit PRM.\n\nMoreover, Implicit PRM only needs to forward the response once, with the only difference being log probs are divided into steps and summed up respectively to find the minimum step reward. We acknowledge that Implicit PRM brings extra inference overhead due to the reference model and we have explored this issue in Appendix. As the cost mainly comes from the additional reference model, in Appendix B.3.1, we ablated it in both training and inference and presented results in Table 5. We found that in practice they can be removed at least in this paper’s setup, avoiding the extra compute while maintaining the performance. We also provided explanations in Appendix B.3.1 on why it may be removed. In such cases, Implicit PRM brings negligible overhead during inference compared to an ORM.\n\n> [W3] Fairness of Baseline Comparisons\n\n**A:** Please refer to the above response on how we can reduce the inference overhead of the reference model. Besides, following your suggestion, we present the compute-equivalent comparison between our best-of-4 and their best-of-8/12 as below:\n|Methods ||Mistral-7B-Inst-v0.2|| Llama-3.1-8B-Inst|\n|--------|:---:|:---:|:---|:---:|\n| |  @12|@48|@8|@32 |\n|Math-shepherd |22.2|25.2|51.4| 52.0 |\n||@4|@16|@4|@16|\n|ImplicitPRM(DPO)|18.6| 24.4 |54.0| 55.4|\n\n\n> Given the PRMs provide step-level supervision, did the authors conduct any experiments to show their implicit PRM is more effective than other PRMs or ORMs at refinement [1] or RL training [2]?\n\n**A:** We supplemented comparison to ORMs with the same data as our Implicit PRM and still observed the superiority of Implicit PRM.\n\n||||Mistral-7B-Inst-v0.2|||Llama-3.1-8B-Inst|||Llama-3.1-70B-Inst|Avg.|\n|---|---|---|---|---|---|---|---|---|---|---|\n||@4|@16|@64|@4|@16|@64|@4|@16|@64||\n|ORM|18.6|23.8|26.4|50.8|50.2|52.8|68.8|71.0|69.4|48.0|\n|ImplicitPRM(DPO)|18.6|24.4|28.8|54.0|55.4|57.0|71.8|71.2|72.2|50.4|\n\nFor RL training, a recent work [3] released after ICML submission deadline explored comprehensively how Implicit PRM can improve RL especially on sample efficiency compared to using verifiable outcome rewards only, namely the golden ORM. Results are shown as follows:\n|Method|AIME2024|AMC|MATH-500|MinervaMatch|OlympiadBench|LeetCode|LiveCodeBench|Avg.|\n|---|---|---|---|---|---|---|---|---|\n|GPT-4o|9.3|45.8|76.4|36.8|43.3|58.9|48.8|45.6|\n|Eurus-2-7B-SFT|3.3|30.1|66.2|32.7|29.8|21.7|17.8|28.8|\n|+RLw/GTOnly|20.0|47.0|73.2|36.4|35.4|28.3|26.7|36.9|\n|+RLw/GT+ImplicitPRM|26.7|57.8|79.2|38.6|42.1|33.3|28.6|43.9|\n\n\n[3] Process Reinforcement through Implicit Rewards. Cui et al. 2025.", "claims_and_evidence": "- Training implicit PRM is a more data-efficient approach than conventional PRM training recipes from prior work -- The authors show this successfully and convincingly\n- Implicit PRM is performant and outperforms baselines: When it comes to these empirical results, the paper is lacking in the following ways:\n    - [W1] *Limited Datasets*: All the results in this paper are based on the MATH dataset. However, a key advantage of any reward model is the ability to score generations from different distributions (even within math reasoning). The results would be more convincing if the best-of-N results were shown on several datasets such as GSM-8K, SVAMP, MMLU, AIME, etc. \n   - [W2] *Is PRM better than ORM?*: In table 1, it appears that the Implicit PRM model is on average only a few points better than the other ORMs, so it is unclear why the training the implicit PRM is worthwhile (since training an ORM would be more efficient at inference time).", "methods_and_evaluation_criteria": "In addition to W1 and W2, it is not clear if comparisons of Implicit PRM an other baselines is fair:\n\n- [W3] *Fairness of Baseline Comparisons*: As pointed out in Table 2, at the 7B model scale, the Implicit PRM involves substantially more compute or GPU hours than the baseline, often by a factor comparable to 1-2x generations. In that case, when looking at the results in Table 1, it would be fair to compare the pass@4 performance of Implicit PRM with the pass@8 performance of baselines for Llama3 8B and pass@12 performance for Mistral. Based on the already slim margins by which their method beats baselines, most of the gains could be wiped away in this setting.", "theoretical_claims": "Yes, in Appendix A.", "experimental_designs_or_analyses": "See W1-3.", "relation_to_broader_scientific_literature": "It proposes a new way of training an Implicit PRM without expensive fine-grained step-level annotations, which is more data efficient and performant than baseline PRM training strategies. They also make interesting connections to RL training of LLMs with implicit rewards as used in offline RLHF algorithms like DPO and show versatility in implicit PRM training objectives for paired and unpaired data.", "essential_references_not_discussed": "[1] https://arxiv.org/abs/2402.10963\n[2] https://arxiv.org/abs/2403.04642", "other_strengths_and_weaknesses": "Addressed in W1-3.", "other_comments_or_suggestions": "Found several typos in the main text: L 077 & L 384. Also check W1-3.", "questions_for_authors": "- Given the PRMs provide step-level supervision, did the authors conduct any experiments to show their implicit PRM is more effective than other PRMs or ORMs at refinement [1] or RL training [2]?\n- See the baselines or additional experiments requested in W1-3 above.\n\n[1] https://arxiv.org/abs/2402.10963\n\n[2] https://arxiv.org/abs/2403.04642", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CIz63WRWpq", "reviewer": "Reviewer_QUrp", "summary": "The paper introduces a method to train Process Reward Models (PRMs) without requiring expensive step-level annotations. By parameterizing outcome rewards as the log-likelihood ratio between a policy model and a reference model, PRMs can be implicitly derived from Outcome Reward Models (ORMs) trained on response-level data alone. This approach, validated on mathematical reasoning tasks, outperforms existing PRM methods (e.g., MCTS-based annotations) with significantly lower computational costs.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> 1. The core concepts and theoretical framework have already been established in [1]. Even with the proposed extensions, the theoretical contribution remains marginal.\n\n**A**: We note that the derivation of our work is different from [1] and provides a more general conclusion. [1] is tailored to DPO only and adopts a different reward representation (with a Z(X) baseline), with the implicit reward derived from entropy-regularized RL framework to ensure the optimality of the trained policy model. We aim for a reward representation that enables tractable Q value; we do not target and never claim an optimal policy in the entropy-regularized RL framework. Rather, our reward representation is **defined** and **constructed from scratch**, namely any representation is acceptable as long as it gives a tractable way to estimate Q value and makes Eq. (2) in line 137 hold. Compared to [1], this paper provides a fresh and more general perspective of implicit rewards, which we believe holds significant value. \n\nIn this way, our paper provides a novel and more general theoretical framework which then leads to empirical benefits, e.g. the legibility of a CE loss, which offers an alternative when pairwise data is harder to collect than response-level labels, and scenarios that are more data-scarce. One may also explore other effective objectives beyond DPO and CE within our theoretical framework. Therefore, we believe our generalization to unpaired losses holds great theoretical and practical merits compared to previous works.\n\n[1] From r to Q∗: Your Language Model is Secretly a Q-Function. Rafailov et al. 2024.\n\n> 2. While the authors suggest using cross-entropy (CE) loss to address scenarios with unpaired data, obtaining response-level labels may still pose challenges. Furthermore, the performance of CE loss is generally inferior to that of DPO.\n\n**A**: We agree that in some cases response-level labels are still difficult to collect; our CE objective provides an alternative when such labels are available, and it can utilize the pairwise labels when they are easier to obtain.\n\nAlso, CE loss has its own advantages over DPO. As shown in Figure 5 and Figure 6, Implicit PRM with CE loss is more data efficient, while showing better performance when integrated with majority vote, presenting as an appealing alternative in practice as in many scenarios pair-wise data is hard to collect. Also, CE loss only requires one example for forwarding and backwarding which reduces memory overhead in RL training. Therefore, the generalization to unpaired losses remains valuable compared to pair-wise DPO in more data-constrained settings.\n\n[2] Process Reinforcement through Implicit Rewards. Cui et al. 2025.\n\n\n> 3. Evaluated only on mathematical reasoning; generalizability to code generation or open-ended generation tasks is untested.\n\n**A:** Though we do not include other tasks in this paper due to our limited capacity and the limited space, after ICML submission deadline, there are recent works showing that Implicit PRM is helpful in best-of-N sampling on agent tasks [3], and adopting Implicit PRM for online RL, with brings substantial gains on coding [2] and function calling tasks [4].\n\n[3] AgentRM: Enhancing Agent Generalization with Reward Modeling. Xia et al. 2025.\n\n[4] Learning to Generate Structured Output with Schema Reinforcement Learning. Lu et al. 2025.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "NA", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strengths: \n1. Derives process rewards as log-likelihood ratios between policy and reference models, unifying preference learning (e.g., DPO) with PRM training.\n2. Reduces training FLOPs compared to MCTS-based methods (e.g., Math-Shepherd), making PRMs accessible for resource-limited settings.\n3. Compatible with diverse loss functions (DPO, CE, KTO, NCA), demonstrating flexibility beyond a single algorithm.\n\n\nWeaknesses:\n1. The core concepts and theoretical framework have already been established in [1]. Even with the proposed extensions, the theoretical contribution remains marginal.\n2. While the authors suggest using cross-entropy (CE) loss to address scenarios with unpaired data, obtaining response-level labels may still pose challenges. Furthermore, the performance of CE loss is generally inferior to that of DPO.\n3. Evaluated only on mathematical reasoning; generalizability to code generation or open-ended generation tasks is untested.\n\n\n[1] Your Language Model is Secretly a Q-Function. https://arxiv.org/abs/2404.12358", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a novel method for training implicit Process Reward Models (PRMs) using only outcome labels, without relying on step-by-step process supervision. The approach is inspired by DPO, where the reward function is parameterized as the log-likelihood ratio between the trained and the reference model.
The proposed method offers two main advantages: (1) it achieves better performance compared to Outcome Reward Models (ORMs) on MATH, and (2) it is more data efficient than prior PRM training approaches. These claims are supported by well-designed experiments, and all reviewers express a favorable opinion of the work.

One minor concern, raised by reviewers kaar, XrJW, and QUrp, is that the application of the implicit PRM is currently limited to Best-of-N sampling and has only been evaluated on math datasets. It would be a valuable addition to provide a thoughtful analysis of potential broader applications (e.g., RL training).

Overall, this work introduces a compelling approach to training implicit PRMs without stepwise supervision. I recommend acceptance.
Decision: Accept (poster)