OpenPRM: Building Open-domain Process-based Reward Models with Preference Trees
Authors: Kaiyan Zhang
Jiayuan Zhang
Haoxin Li
Xuekai Zhu
Ermo Hua
Xingtai Lv
Ning Ding
Biqing Qi
Bowen Zhou
Venue: ICLR 2025 Poster (2025)
Keywords: large language models
reward models
open-domain instruction following
Primary area: foundation or frontier models, including LLMs

Abstract: Scaling inference-time computation is increasingly seen as the next frontier in scaling laws for large language models. Previous work in mathematics and coding has demonstrated the remarkable potential for inference-time scaling. During such scaling, fine-grained supervision through process-based reward models (PRMs) is essential for enhancement. However, exploration of inference-time scaling and PRMs in open-domain problems remains limited, where lacking exact answers and obtaining process supervision prove challenging. In this paper, we explore the construction of PRMs for open-domain tasks, specifically for instruction-following tasks. Utilizing existing outcome-based reward models (ORMs), we develop sentence-level preference trees based on the prefix similarity of parallel sampled candidates from datasets like UltraFeedback. This setup allows us to derive weak supervision for processes via back-propagation from outcome-level rewards. Subsequently, we integrate ORMs and PRMs under the same pairwise ranking objectives, resulting in our newly developed reward models, named OpenPRM. This approach significantly enhances the scalability of process-level supervision in open domains at minimal cost. We assess the performance of OpenPRM across various reward benchmarks, demonstrating its competitive edge over traditional ORMs in open domains and PRMs in specialized domains. Additionally, we investigate the scalability of inference-time computation for open-domain instructions. Our results highlight the limitations of ORMs’ scalability, while OpenPRM shows superior performance in scaled settings. Despite these advances, achieving  automatic fine-grained supervision for open-domain inference-time scaling remains a substantial challenge. We hope these findings will spur further development of process supervision reward models in open-domain scenarios.

Published as a conference paper at ICLR 2025
OPENPRM: BUILDING OPEN-DOMAIN PROCESS- BASED REWARD MODELS WITH PREFERENCE TREES
Kaiyan Zhang'
Jiayuan Zhang Haoxin Li'
Xuekai Zhu
Ermo Hua
Xingtai L
Ning Ding'
Biqing Qi
Bowen Zhou 1.4
1 Tsinghua University
Beihang University
Shanghai Jiao Tong University
4 Shanghai Artificial Intelligence Laboratory
ABSTRACT
Scaling inference-time computation is increasingly seen as the next frontier in scaling laws for large language models. Previous work in mathematics and coding has demonstrated the remarkable potential for inference-time scaling. During such scaling, fine-gratee greeggrined supervision through process-basede reward models (PRMs) is essential for enhancement. However, exploration of inference-time scaling and PRMs in open-domain problems remains limited, where lacking exact answers and obtaining process supervision prove challenging. In this paper, we explore the construction of PRMs for open-domain tasks, specifically for instruction- following tasks. Utilizing existing outcome-based reward models (ORMs), we develop sentence-level preference trees based on the prefix similarity of parallel sampled candidates from datasets like UltraFeedback. This setup allows us to derive weak supervision for processes via back-propagation from outcome-level rewards. Subsequently, we integrate ORMs and PRMs under the same pairwise ranking objectives. resulting in our newly developed reward models, named Open- PRM. This approach significantly enhances the scalability of process-level super- vision in open domains at minimal cost. We assess the performance of OpenPRM across various reward benchmarks, demonstrating its competitive edge over tradi- tional ORMs in open domains and PRMs in specialized domains. Additionally, we investigate the scalability of inference-time computation for open-domain instruc- tions. Our results highlight the limitations of ORMs' scalability, while OpenPRM shows superior performance in scaled settings. Despite these advances, achiev- ing automatic fine-gre mne-grained supervision for open- inference- time scaling remains substantial challenge. We hope these findings will spur further develop- ment of process supervision reward models in open-domain scenarios.
INTRODUCTION
Large language models (LLMs) such as GPT-4 (Achiam et al., 2023), Llama (Touvron et al., 2023; Dubey et al., 2024), and Gemini (Team et al., 2023; Reid et al., 2024) have garnered interest across various fields due to their robust performance in numerous tasks and domains. The development of L.MSs involves an official process that includes pre-training on a large-scale unlabeled cor- pus (Brown, 2020) followed by post-training using labeled instructions derived from rorl-world applications. The post-training phase is further categorized into supervised fine-tuning and rein- forcement learning from human model feedback (Ouyang et al., 2022), a process known align- ment (Ji et al., 2023; Shen et al., 2023). During this phase, reward models are crucial as they act as human proxies, providing feedback model behavior and adjusting the models to better align with human values (Bai et al., 2022). Although current popular alignment algorithms like Direct Preference Optimization (DPO) (Rafailov al., 2024) implicitly incorporate the rewarding process within the loss function, reward models still play a significant role in ensuring long-term alignment through methods such as online and iterative DPO (Xiong et al., 2023; Guo al., 2024; Pang al., 2024) and rejected sampling (Dong et al., 2023; Liu et al., 2023; Khaki et al., 2024).
"Corresponding Author (zhoubowen @t nghuaa edu..cn)

---

Published as a conference paper at ICLR 2025
In addition to the training process, reward models crucial for enhancing the performance of LLMs during inference- cerrencce-imeeeee (Khanov et al., 2024; Deng & Raffel, 2023). Unlike the scaling laws applied to compute during pre-training (Kaplan et al., 2020; Hoffmann et al., 2022), there is trend towards scaling inference-time compute through extensive searches in the decoding space (Snell et al., 2024; Brown al., 2024). Reward models play significant role in pruning the search space and ultimately selecting the most accurate answers (Welleck et al., 2024).
Recent studies indicate that outcome-level reward models fail to apply scaling laws during repeated sampling (Brown et al., 2024), particularly due to their coarse granularity on challenging tasks. Consequently, many researchers are exploring the use of more fine-grained reward models, such as process-level (Uesato et al., 2022; Lightman et al., 2023) or token-level (Deng & Raffel, 2023), to enhance search performance in specialized domains like mathematics, coding, and reasoning tasks (Wang et al., 2024b; Havrilla et al., 2024; Xin et al., 2024; Yuan et al., 2024; Chen et al., 2024a; Setlur et al., 2024; Xie et al., 2024). These efforts often follow the paradigm established by AlphaGo (Silver et al., 2016), integrating LLMs with Monte Carlo Tree Search (Coulom, 2006; Kocsis & Szepesvári, 2006) and a value function, analogous to a reward model. However, process- level reward models are typically tailored for specific tasks and exhibit limited generalizability in open-domain applications such as writing and chat. Moreover, there is scant research on developing process-level reward models for open-domain contexts, primarily due the high cost of annotation.
Currently, outcome-level reward models (ORMs) are evolving rapidly (Wang et al., 2024a; Cai et al., 2024; Wang et al., 2024e;c; Vu al., 2024), prompted by the emergence of datasets and benchmarks such as Ultrafeedback (Cui et al., 2023) and RewardBench (Lambert et al., 2024), where open- source reward models trend outperform proprietary ones. This development raises the question of whether process-level reward models (PRMs) can be constructed from instance-level rewards using a weak-to-strong framework (Burns al., 2023).
In this paper, we propose the development of PRMs in open-domain, leveraging existing ORMs to provide fine-grained supervision. Our contributions are summarized as follows: (1) We ana- lyze the potential for extending open-domain ORMs to PRMs, elucidating the characteristic and relationship between them. This analysis inspires our proposal to develop PRMs with outcome- level supervision through building preference trees with key process. (2) We integrate the modeling of PRMs and ORMs under a unified objective and develop OpenPRM. By leveraging only exist- ing ORMs and employing repeated sampling on prompts, we enhance the performance of ORMs, achieving a 3~5%% improvement on RewardBench. (3) We further evaluate OpenPRM across vari- ous downstream applications, including different inference-time scaling settings. Our findings show that ORMs struggle to provide effective supervision, while our proposed OpenPRM outperforms previous RMs under these scaling conditions. We also observe that there is significant journey ahead to fully realize the potential of RMs in open-domain tasks for test-time scaling.
PRELIMINARY
2.1
REWARD MODELING
Reward models play a crucial role in large language models by aligning model outputs with desired human preferences (Wang et al., 2023). There are primarily two types of reward models based on the granularity of the supervision signal: outcome-level and process-level reward models. We will introduce the development of these two methods as follows.
Outcome-level Reward Model (ORM) ORMs are commonly used for preference learning, partic- ularly after supervised fine-tuning in InstructGPT, where they serve as proxy for human feedback to model generations (Ouyang et al., 2022; Lee et al., 2023). Although many studies explore reward model-free preference learning, such direct preference optimization (DPO) (Rafailov et al., 2024), which implicitly models the reward within the policy model training, ORMs continue to be instru- mental in further model improvements. This includes applications in online or iterative DPO (Guo et al., 2024; Pang et al., 2024) and rejection sampling (Liu et al., 2023; Dong et al., 2024).
The primary methods for obtaining ORMs involve preparing pairwise responses with preferences (e.g.. chosen and rejected) and tne-ttunig instructed models using ranking loss (Ouyang et al., 2022; Dong al., 2024). Some studies also consider DPO models reward models (Lambert et al., 2024),
2

---

Published as a conference paper at ICLR 2025
Table A comparison with the most related works on process-level reward models.
Data Acquisition
Training & Inference
Release
Name
Tomaiin
Task
Backhone
A Amnotation
Säze
Labeling
Objective
Search
Data
Model
(Unsuto et ali, 2022)
DEEPMIND PRM
Math
GSMSK
NIA
Haman
10k
011
NIA
BoN
x
(Lihhtman ol,, 2023)
OPENAT PRM
GPT-
Hamon
800k
-1V1
C...
BoN
GSMBK GANESA
MCTS-r
TS-LLM (Png 2023)
Math
ProntoA.RLIE
LLaMA2-7B
Golden Ans.
1 -50kk
0-1
MSE Loss
+Rollout
8
Endgame
MATH-SHEPPEEE
(Wang all, 2024b)
Math
GSMBK
L. 1L
Golden Ane.
445k
QV1
CE Loss
16:64 leaf
MCTS
GL.okt
(Havrilla ot 2024)
Math
GSMHKK
Liama2-78
Golden Ans.
NIA
OV1
CE Loss
BoN
X
(Wang et all, 20240)
MIPS
MatiCCade
GSMBK, MATH
MBPP
PaLM 2-8/L
Gokden Ans.
~ 14k
0-1
MSE Loss
32 32 leaf
MCTS
8
X
MCTS-DPO (Xie ol 20144)
Math
ARC.AnSSiimee.
GSMSKMMTTHI
MCTS
Common
OpenRonkQA,
Mistrai-78
Golden Ane.
seense
CommonSenneQA
(Chen al., 2094aa
SUPER
GSMSK MATH
MathDase-7D1
DeepSeck
Golden Ane.
MSF
leal
STEP-DIO (Lai ct 2014)
MesMath, MMIQC
Qwm2-77B/2B
LLM
Ranking Loos
OMEGAPRM
Math
Gemanl
Golden Ans.
MSE Loss
MCTS
(Zhang al., 2024)
REST-MCTS
Math
MATH
Mistral-7B
Golden Ans.
700k
0-1
MSE Loss
MCTSS
Jeuc
though the effectiveness of these models in iterative optimization still requires further exploration. Additionally, self-play learning (Chen et al., 2024b; Tao et al., 2024) has recently been applicd to continually improve reward models, presenting promising method to enhance the capabilities of ORMs autonomously (Wang et al., 2024c). Our approach on OpenPRM can also be considered a method for continuously improving ORMs through their own annotations.
Process-level Reward Model (PRM) The primary challenge with ORMs is their coarse-grained nature of rewards; even if the final answer correct, errors may still exist within the solution steps. To address this, there is a growing trend to develop more fine-grained, process-level RMs. The main challenge in developing PRMs lies in obtaining accurate supervision signals for each process within a solution. There are three main approaches: 1) Human Annotation: This method requires experts to annotate each process step as neutral, bad, or good. While human annotation can provide precise process supervision, it is difficult to scale and very costly (Uesato et al., 2022; Lightman et al., 2023). 2) Golden Answer:: For mathematical or coding problems, accurate final answers or feedback from exact matching interpreters are available. Common methods involve computing the probability of nodes along the path toward the final, accurate answers, integrating with Monte Carlo Search methods (Wang et al., 2024b; Havrilla et al., 2024; Luo et al., 2024). 3) Model-base Judgment or Reward Models: The final approach involves obtaining rewards from model-based judgments or reward models (Lai et al., 2024). Some research utilizes outcome-level rewards to estimate process rewards (Lu et al., 2024), reducing the high costs associated with extensive sampling.
For training PRMs, two main methods used, depending the data format required: 1) Single Sample: Each step in the solution process is labeled, and losses such as Cross-Entropy (CE) (Light- man et al., 2023; Wang et al., 2024b; Havrilla al., 2024) and Mean Squared Error (MSE) (Feng
et al., 2023; Wang et al., 2024f) loss are typically used. 2) Pair Sample: Each question is associ- ated with chosen and rejected processes, and pairwise ranking loss is employed (Lai et al., 2024).
This method is typically used in the training ORMs (Dong et al., 2024). We provide detailed survey in Table Currently, many ORMs open domains benefit from the development bench- marks (Lambert et al., 2024; Wang et al., 2024e). However, the true effectiveness of these ORMs and the feasibility of developing PRMs from ORMs are still subjects of ongoing exploration.
2.2 DERIVATION OF PRM FROM ORM
ORM is typically trained to predict the quality of the final outcome, while PRM supervises the intermediate steps of the process. The modeling of ORM can be formalized with pairwise ranking
3

---

Published as a conference paper at ICLR 2025
Crat
18. Crathaar
10 Reaaon
Sahay
Accuraic
05
75
70
pm2 Firstin process
252
firstin process
firsin process
Figure Results of reward models based on rewarding processes of varying lengths within the evaluated content. The results indicate that the initial segments of responses are particularly critical challenging tasks such as Chat-Hard and Reasoning, where longer content holds length bias.
loss function based on the outcome-level feedback (Ouyang et al., 2022):
(0) (a.ee.ee [log (a (ro(2, Me) ro(z, yr)))]
Here, (2 ye) and ra ra yr) are the model's scores for the chosen and rejected outcomes, respec- tively. While ORM performs well in satcome-based supervision, it has limitations when applied process-level supervision, due to the cumulative error effect. This effect arises because ORM focuses on the final result, neglecting errors in intermediate steps that can propagate through the sequence and affect the final outcome (Lightman al., 2023).
Previous works (Uesato et al., 2022: Havrilla et al., 2024) have shown that in cases where the base model for response generation sufficiently strong and the task relatively simple, the cumulative error may be negligible. In such scenarios, ORM can be effectively used as substitute for PRM.
However, for more complex tasks, the cumulative error can be significant, necessitating additional process-level supervision. To address this issue, we propose joint modeling approach that inte- grates both ORM and PRM by introducing supervision key process of answer. Specifically, we identify the most divergent process between chosen and rejected outcomes, denoted Pe and pr. and introduce additional supervision at these critical points. The loss function can be defined as:
((0) LORM (0) + (f)
(2) The
where is a ayper-parameter that balances the outcome- based and process- based losses. process-based loss LprM supervises the divergent steps Pe and pr, and be expressed as:
LPRM (0) = log (a (ro(x. Pc) (o(2,))))
Here, re(æ, Pe) and re(2, Pr) represent the scores for the critical steps in the chosen and rejected sequences, respectively. By focusing on these key steps, the cumulative error is mitigated as early enrors corrected critical junctures. We provide more details in Appendix A. In conclusion, the above analysis leads to the following theorem:
Theorem Given a dataset D consisting of pairs of responses (Ye, yr) with outcome-based pref- erences Ve > Yr. and a learned outcome-based reward model ro(x,y), the cumulative error of process supervision can be significantly reduced. This is achieved by identifying the key divergent steps (pe, Pr), such that (ppss pr) is maximized, and incorporating these steps into a joint modeling framework. Thus, under this framework, the cumulative error in process supervision decreases as the discrepancy A (ppe. (pe.Pr), supervision strength S, and model sensitivity y increase.
2.3 EMPIRICAL EVALUATION OF OPEN- -EMMIN ORM IN PROCESS ASSESSMENT
this section, we conduct an empirical evaluation of open-domain ORMs as process evaluators and examine some unique phenomena associated with their performance. Specifically, we assess popu- lar reward models such as FsfirX (Wang et al., 2024a), InternRM (Cai et al., 2024), and UltraFeed- back (Cui et al., 2023) (trained the Llama-3 8B model (Dubey al., 2024)) using RewardBench (named RB) (Lambert et al., 2024). We focus on the primary categories RB, including Chat, Chat-Hard, Reasoning, and Safety tasks. We provide more details about experiments in 4.1.
4

---

Published as a conference paper at ICLR 2025
Reward Model (RM)
Outccoe RM (ORM)
Unpaired
Pairwise
Reward Modeling
(R(s) y)
Youus
d "T (x.3
(PRM)
Simitrr Anower Prefts
Y: x
1850 y 03
OpenPRM Training
Prompts
CA
CA
CA2
CA
Back-Propagation
Sampling
v(Pu) Ens R [(PP
Candidate Answers
(6% 2y366
Ra
Rewarding
Toans R.
Process Preference
PaPa > PaPp
ORMs
Segment
Aggregate
Prr Pr Pso
R(K.))
This figure illustrates the differences between outcome-level and process-level reward models in the top left, including their common training strategies with paired and unpaired data. The training of OpenPRM is depicted on the right and below (mainly blue area).
Figure 2:
As illustrated in Figure 1, consistent trend is observed across various reward models, indicating that RMs perform better on simpler Chat tasks as the evaluated process lengthens. However, this effect reverses in more challenging Chat and Reasoning tasks, where accuracy first increases and then decreases as the length of the evaluated process extends. conclusion, these results support Theorem which posits that ORMs can function PRMs, but their performance deteriorates due to cumulative errors, particularly in harder tasks.
METHODOLOGY
3.1 PROCESS-LEVEL PREFERENCE TREE
To obtain process-level rewards, we construct process-level preference trees using readily available outcome reward models. The pipeline for building preference trees consists of three steps:
Repeated Sampling Prompts We initially prompt open-source language models to generate a large number of parallel candidate responses through repeated sampling. To ensure broad representation, we primarily include models at the 7B and 70B parameter levels.
Step 1.
Step 2. Aggregation on Sentences For each output, we segment it into a collection of sentences and construct tree using depth-first search algorithms. We calculate the edit distance (Ristad & Yianilos, 1998) between sentences from different outputs and merge sentences into single node based predefined threshold. This helps reduce the cost of building prefix trees.
Step Backpropagation on Rewards Once outputs with their respective rewards are segmented into sentence collections, which serve nodes in the preference tree, we designate the outcome rewards for the leaf nodes. For each process node, we compute the process-level rewards using backpropagation. Given the rewards of the leaf nodes, R₁ (outcome-level rewards), the rewards of the inner nodes, Pij. using backpropagation, as detailed in Monte Carlo Tree Search. Notably, the rewards of the inner nodes, denoted as V (Ptt), can be calculated using the formula
compute
v(P.)) EXEL(R,) R [L(P.;)
where I(P.)) represents the set of all leaf nodes descending from P.j-
About Rationality of Process Aggregation Unlike previous works mathematics and coding that reuse partial answers for subsequent answer generations (Lightman al., 2023; Luo et al., 2024),
5

---

Published as a conference paper at ICLR 2025
method involves directly sampling large number of candidates and merging identical sentences,
akin to state aggregation in Monte-carlo algorithms (Hostetler et al., 2014; Jang et al., 2021). This approach enables the RMs to learn high-level actions and logic within the shared sentences. We provide a real example of question-answering for reference in the Appendix B.3.
3.2 PROCESS-LEVEL REWARD MODELING
During the development of OpenPRMs, we enhance the models by integrating rewards and domains, aiming to create more robust process-level reward models from the following two perspectives:
Mixture of Rewards Considering the completeness of outputs, output-level rewards serve as spe- eific instances of process-leve rewards, where the output encompasses the entire process. Therefore, we blend rewards from both the process and output levels to develop more robust reward models.
Mixture of Domains Existing process-level reward models predominantly focus on domains such as mathematics, and reasoning tasks, which provide certain answers for supervision. To leverage the strengths of these domains, also integrate them with general domain preferences to enhance the versatility and applicability of OpenPRMs. We provide details about dataset Appendix B.1.
At the training stage, we treat all preference data a pairwise ranking task. This involves using the input prompt along with chosen and rejected completions (including both process and output). Using this unified format, train the PRM with the Bradley-Terry objective, defined in Equations and 3. This formulation ensures consistent training across both process- and outcome-level datasets.
3.3 APPLICATION OF PROCESS-LEVEL REWARD MODELS
Best-of-N Sampling At inference time, given questions. Subsequently, we can determine the final answer through a majority vote (James, 1998) (referred as Vote @ N); however, this method is primarily applicable questions that re- quire exact answers, such as those found in mathematics and reasoning tasks. For open-ended ques- tions, it is more common to apply reward models all answers and select the one with the highest rewards, a method known as best-of-N sampling (Stiennon et al., 2020) (BoN@N) N). When imple- menting process-level reward models the BoN context, there are two approaches to computing rewards: one approach treats the outcome as a special process and computes rewards directly on the outcome, while the other calculates rewards for each process and selects the minimal one (Lightman al., 2023; Wang et al., 2024b) to derive the outcome rewards.
generate large number of candidate answers for
Process-level Decoding Another significant application of PRMs is in the decoding phase. By evaluating the generated process, can expand the beam search strategy (Sutskever et al., 2014) from token-level to process-level. As result, we maintain N sentences at each step and reward each sentence during generation until the completion of the answer, technique termed process-leve beam search N). Additionally, we can integrate advanced operations akin to those employed in Monte Carlo Tree Search (MCTS) (Browne et al., 2012), such simulation, retrospection, and memory functions. However, these operations may extend the required processing time and lead to increased inference costs. Previous research (Chen al., 2024a; Snell al., 2024) has indicated that @ can achieve performance comparable to MCTS but at a reduced cost.
EXPERIMENTAL SETUP
4.1 DATASET
In developing OpenPRM, we first construct extensive preference trees based on open-domain in- struction dataset, as described in 3.1. This construction utilizes the UltraFeedback (Cui et al., 2023) and ScienceQA (Lu et al., 2022) datasets, which provide a highly diverse and high- range of instructions. Additionally, further enhance the math reasoning capabilities of our reward system. For each prompt within this instruction pool, sample 64 candidate responses from I. Llama-3 models (Dubey et al., 2024).
incorporate the MATH (Hendrycks et al., 2021) dataset to

---

Published as a conference paper at ICLR 2025
4.2 MODELS
Reward Models. We compare our reward models with otate-of-the-art (SOTA) open-source reward models. Due to concerns with inference efficiency, we primarily evaluate classifier-based models, which perform comparably to generative models but are more scalable. We compare our models with ORMs, such as FsfairX (Dong et al., 2024), Eurus (Yuan al., 2024), and UltraRM (Cui al., 2023), and PRMs like TS-LLM (Chen et al., 2024a), MathShepherd (Wang et al., 2024b).
Chat Models. We assess the effectiveness of our reward models using state-of-the-ard open-sourced chat models, including Llama-3. 1-8B--nsstruc and Llama-3. 1-70B-Instruct (Dubey et al., 2024), and Distral-Nemo-Insttue--2hh Instruct-2407 The latter can be regarded as an out-of ditt-off-itttuutiinn evaluation.
4.3 EVALUATIONS
Evaluation of Reward Models. Given the lack of established benchmarks for evaluating process- based reward models, we primarily compare our process-bane based models against established outcome- based reward benchmarks, such as UltaFeedback (Cui et al., 2023) and RewardBench (Lambert et al., 2024). RewardBench is designed to assess the capabilities and safety of reward models across four categories: Chat, Hard Chat, Reasoning, and Safety. We employ the primary dataset from RewardBench to evaluate the out-of-domain generalization capabilities of our reward mod- els. We evaluate the effectiveness of process supervision of reward models solely the test set of PRM800k (Lightman et al., 2023), which features high-quality human annotations. Especially, evaluate PRMs using specific aggregation strategies, such selecting the minimal reward across steps. Detailed descriptions of the different aggregation strategies are provided in the Appendix D.I.
Evaluation of Chat Models. To comprehensively evaluate the impact of reward models on chat models, we test the chat models across variety benchmarks, primarily referencing the Open LLM Leaderboard 2. This evaluation includes benchmarks in: 1) Instruction following tasks such as Alpaca Eval (Dubois al., 2024) and IFEval (Zhou et al., 2023); 2) General domain tasks such as MixEval Hard (Ni et al., 2024), MMLU-Pro (Wang et al., 2024d), and GPQA (Rein et al., 2023); 3) Specific math domain tasks like MATH500 (Lightman et al., 2023). Additional details about these evaluation tasks and methodology are provided in Appendix C.
4.4 SETTINGS FOR INFERENCE
During inference, we primarily evaluate two methods: majority vote and best-of-N. For the best-of- N method, we adhere to the following protocol (Chen et al., 2021): initially, we sample N responses, where is set 128. We then sample K responses from these, repeating the process average the results. The values for K range from I, 2, 4, 8, 16, 32, 64, to 128, and M is set to 5. This approach allows us to reduce inference costs and achieve robust results through multiple averaging. For process-bas basce beam search, we set the beam size to VN to maintain an approximately equivalent decoding cost with best-of-N, as described in (Snell et al., 2024).
times to
EXPERIMENTAL RESULTS
5.1 RESULTS OF REWARD BENCHMARKS
As shown in Table 2, we compare OpenPRMs with standard ORMs and specialized PRMs across both general outcome-basee and specific process-reward benchmarks, Based on the results, we can draw the following conclusions:
OpenPRMs outperform ORMs Utilizing off-the-shelf ORMs and corresponding preference datasets, we have developed advanced reward models that demonstrate superior performance on RewardBench, particularly in the Chat Hard and Reasoning tasks. Additionally, the process-based preferences built upon our method consistently enhance the performance of the base reward models,
-nmo
htttps///hf.cc//ppceeee 1m-1 eade baderboard//ppn. 11m_ 1m_1 eaderboard
7

---

Published as a conference paper at ICLR 2025
Table Results outcome-level and process-level reward models on instance-level reward bench- marks. Models marked with an asterisk (*) were trained using data compiled by our team.
Model / Task
Training Data
UltraFedback
RewardBench
PRMSING Test
Test
Overall
Chat
Chat Hard
Safety
Reasoning
Open-Source ORMs
ULTRARM-13B
UbraFeedback (UF)
96.4
55.5
62.4
LLAMA-3-8B#
UF Binarized
77.8
84.8
97.5
66.9
85,5
89.2
51.8
EURUS-RM-7B
Ultralnteract
65.6
81.4
86.3
FSPAIRX-7B
Mixture Preference
74.5
84.4
99.4
65.1
86.8
86.4
53.3
INTERN2-7B-RM
Unknown
87.6
99.2
69.5
87.2
LLAMA-3-8B*
HelpSteer (HS)
71.8
86.8
95.3
76.8
85,9
89.2
53.7
Open-Source PRMs (Merely Math Domain)
MS-7B-PRM
Math-Shepherd
53.5
56.6
62.3
51.3
39.6
73.2
56.9
TS-LLM
Unknown
52.7
66.8
50.0
55.3
58.4
LLAMA-3-8B
Math-Step-DPO
70.2
73.2
98.0
58.8
59,9
76.1
61.3
OPENPRM (FsfairX)
UF Tree HS
72.8
89.4
95.5
81.1
88.7
92.1
64.3
OPENPRM (InternRM)
Tree + HS
78.5
91.1
98.0
81.6
89.5
95.1
68.1
showcasing their generalization capabilities. These findings validate the effectiveness of our prefer- ence tree construction strategy discussed in 3.1. Moreover, the results substantiate ability to enhance weaker existing models, achieving weak-to-strong generalization (Burns et al., 2023).
OpenPRMs outperform specific PRMs Beyond outcome-level reward benchmarks, we also com- pare OpenPRMs with publicly available PRMs, which predominantly originate from the math do- main, many previous PRMs are not available. We present Shepherd (Wang et al., 2024b), TS-LLM (Feng et al., 2023) and Llama-3 trained on (Lai et al., 2024). Due the domain gap, these math-specific PRMs underperform in open-domain bench- marks, whereas OpenPRMs demonstrate superior performance even on tasks like PRM800K.
results evaluated using Math-
5.2 RESULTS OF APPLICATIONS IN DECODING
To validate the effectiveness of PRMs, we evaluated OpenPRM under various decoding settings across multiple popular open-domain tasks, comparing strategies such as majority vote, best-of-N, and process-level beam search. We summarize the experimental results of OpenPRM as follows:
OpenPRM Performs Effectively with BoN and PBS As illustrated in Table 3, OpenPRM achieves superior performance in both BoNG @ 16 and PBS@4 compared to Vote@ 16 with the Llama-3.1- 8B and 70B models across nearly all tasks. These results confirm the effectiveness of OpenPRM. Additionally, even the out-of-distribution models, such as Mistral-Nemo (compared to the Llama series), validate the advantages of OpenPRM. We also observed that beam search algorithms outper- form BoN, benefiting from the fine ain-igrined evaluation of processes. However, further exploration of decoding strategies (like MCTS) in open-domain settings will be necessary in the future.
Scaling inference-Timee Achieves a High Upper Boundary We further analyze the results of scal- ing inference-time by progressively increasing sampling times from I (2") to 128 (27). The results depicted in Figures 3 demonstrate that the models can achieve exceptional performance with opti- mal reward models (refer coverage@N N and pass @ settings (Chen al., 2021)). The coverage accuracies nearly reach 100% on most open-domain tasks. These findings in open-domain tasks are consistent with prior studies mathematical and coding domains (Brown et al., 2024; Bansal al., 2024), suggesting the emergence of scaling law at inference time in open-domain well. We also include the sampling curves for the Llama 70B and Mistral model in Appendix D.4.
OpenPRMs Optimize Inference-Tinn Time Utilization Compared to the coverage accuracy depicted by the red curve. nearly all the reward models struggle to scale as inference-tine time increases. This indicates that significant advancements for inference-time scaling. Among these previous models, InternRM performs best on most tasks average, notably MMLU-Pro and IFEval, though it still lags significantly behind the coverage
still required to develop more effective reward models
8

---

Published as a conference paper at ICLR 2025
Table 3: Results of majority vote, best-of-N sampling, and process-level search on open-domain tasks. The findings from BoN@ 16 and PBS@ 16 demonstrate the effectiveness of OpenPRMs. No- tably, we have reproduced part of the results, taking into account differences dataset usage.
Alpaca Eval
MixEval
IFEval
GPQA
MMLUUP
MATH
Model / Task
LC%
WC%
Acc
Avg.
Acc
Ace
Acc
GPT-40 (0806)
74.68
53.1
GPT-4-TURBO
46.1
84.4
63.71
49.3
CLAUDE-3.5-SNNNT
52.4
40.6
68.1
88.0
71.1
76.12
59.4
REPORTED
20.9
21.8
45.6
79.5
45.0
40.8
23.7
REPRODUCED
34.6
33.3
39.7
76.5
24.3
37.2
45.6
LLAMA-3.1-SH INSTRUCT
VOTR@16
56.4
B6N@16
50.5
58.8
PBS@4
39.9
42.2
75.59
31.3
47.8
52.8
REPRODUCED
45.0
37.5
35.7
64.2
31.9
36.6
31.4
MISTRAL-NEMO INSTRUCT-202407
VOTE 16
35.2
44.4
40.7
BoNG 16
39.0
36.6
69.5
35.5
48.2
52.3
REPORTED
38.1
29.9
55.9
85.8
65.8
55.0
41,9
LLAMA-3.1-70B INSTRUCT
REPRODUCED
424
41.6
59.1
85.4
45.7
55.4
63.6
VOTE@1
51.4
60.3
70.7
BoN@ 16
44.4
42,9
61.9
87.7
49.8
67.1
72.0
curve. In contrast, proposed OpenPRM outperforms InternRM. showing promising results in scaling up the best-of-N sampling. However, achieving scaling comparable to the coverage curve remains substantial challenge. We will release all of these sampling data the public to encourage further study on process and outcome reward models for inference- teteenece-tmeeae scaling.
6 DISCUSSION
6.1 ABLATION STUDY OF PRM TRAINING
We conducted an ablation study on OpenPRM training, focusing on data sources and model config- urations. As illustrated in Table 4a, we compared the effects of continuously fine-tuning InternRM and FsfairX using process pairs built upon preference trees and outcome pairs. The results indicate that process pairs yield superior outcomes, thus validating the effectiivnness of our method described in Section 3.1. Additionally, the performance of InternRM. when using shared prefix, inferior to configurations using distinct prefixes for chosen and rejected pairs, emphasizing the importance of semantic consistency. Furthermore, while using only UltraFeedback data showed promising sults in Chat tasks, maintaining diversity and generalization for open-domain applications is crucial. Therefore, opted integrate additional reasoning and STEM questions.
6.2 REWARDS SHAPE AND LENGTH BIAS
We compare the reward shapes of OpenPRM with other reward models in Figure 5. We analyze the rewards of chosen and rejected candidates for each instruction in RewardBench and observe that while all reward models can generally distinguish between chosen and rejected candidates, indicated by shift in distributions, there remains some overlap. However, OpenPRM exhibits the minimal overlap among them, similar to the parent reward model (i.e., InternRM).
Language model-based judgers often suffer from length bias, typically awarding higher rewards to longer responses. We address this issue our analysis of OpenPRM, visualizing the correlation between OpenPRM and InterRM in Figure 4b. The results indicate that OpenPRM maintains relation 0.05, compared to 0.37 for InterRM, suggesting that process-level modeling effectively reduces length bias. This also demonstrates the effectiveness and necessity of developing PRMs.

---

Published as a conference paper at ICLR 2025
Coverage
Bchig.
SCNGNLL
Sehign
ScNeH,0re-RRM
91
Agu 12
Wimter of Samclee//))
Sammerr of Samsles (N]
Aumoer Sempp (N)
Sumber 575kmp (H)
18mra-3.) 3.) wi.ineeeeetttee MeL 1t
Jara IndOurt.GPAA
2awe 3- pndmurt, Holh
Ukym 21 FEve
32
57.5
45.0
Figure 3: Results of scaling inference-time for Llama-3. 1.-.3..1----Itiiiite Instruct on open-domain tasks. These results illustrate the effectiveness of OpenPRMs relative to existing reward models, yet they also highlight the distance to the upper boundary of coverage accuracy (red curve).
Model
Data
RB Avg.
10ng 0UAT Cantioal
PrefTree Pairs
91.1
InternRM
w/ Shared Prefix
90.8
Drg
FsfairX
PrefTree Pairs
89.4
Alama-3-88-Il
PrefTree Pairs
87.2
InternRM
Outcome Pairs
88.4
FsfairX
Outcome Pairs
87.7
Llama-3-8B-It
Outcome Pairs
86.8
(a) Ablation Study of OpenPRM.
(b) Rewards VS. Length
Figure 4: Ablation Study of OpenPRM
CONCLUSION
In this paper, we explore the development of process-based reward models (PRMs) in the open do- main. We begin by generalizing rewards from outcome-level to process-level, significantly reducing data annotation costs. We then propose the construction of preference trees with parallel candidates for open-domain instructions, from which we derive process pairs using back pnok-propagation. Leverag- ing this data, our trained OpenPRM achieves excellent results reward benchmarks and performs well under scaling inference-time search conditions. However, our findings also highlight that there is still considerable progress to be made in building open-domain PRMs to achieve high coverage accuracy. In conclusion, we try to unify ORMs and PRMs in the open domain, paving a new path for PRM development that diverges from domains such as mathematics and coding. We hope that OpenPRM will spark new insights into this topic and stimulate further research.
ACKNOWLEDGMENTS
This work is supported by the National Science and Technology Major Project (2025ZD01214033, Young Elite Scientists Sponsorship Program by CAST (2023QNRC001), and National Natural Sci- ence Foundation of China (No. 62406165).
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint wrxiv:2333008772 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.0558 2022.
Hritik Bansal, Arian Hosseini, Rishabh Agarwal, Vinh Tran, and Mehran Kazemi. Smaller, weaker, yet better: Training Ilm reasoners via compute compute-optimal sampling, arXiv preprint arXiv:2408. 16737, 2024.
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21788 2024.
Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005. 2020.
Cameron B Browne, Edward Powley, Daniel Whitchouse, Simon M Lucas, Peter Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. A survey of monte carlo tree search methods. IEEE Transactions on Computational Intelligence and Al in games, 4(1): 43, 2012.
Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbren- ner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-to-stree to-sonng general- ization: Eliciting strong capabilities with weak supervision. arXiv preprint arXiv:2312.09390, 2023.
Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui. Aijia Guo. Qipeng Guo, Conghui He, Yingfan Hu, Ting Huang, Tao Jangg Penglong Jiao, Zhenjiang Jin, Zhikai Lei, Jiaxing Li, Jingwen Li, Linyang Li, Shuaibin Li, Wei Li, Yining Li, Hongwei Liu, Jiangning Liu, Jiawei Hong, Kaiwen Liu, Kuikun Liu, Xiaoran Liu, Chengqi Lv, Haijun Lv, Kai Lv, Ma. Runyuan Ma. Zerun Ma. Wenchang Ning, Linke Ouyang, Jiantao Qiu, Yuan Qu, Fukai Shang, Yunfan Shao, Demin Song, Zifan Song, Zhihao Sui, Peng Sun, Yu Sun, Huanze Tang, Bin Wang, Guoteng Wang. Jiaqi Wang, Jiayu Wang, Rui Wang, Yudong Wang. Ziyi Wang, Xingjian Wei, Qizhen Weng, Fan Wu, Yingtong Xiong, Chao Xu, Ruiliang Xu, Hang Yan, Yirong Yan, Xiaogui Yang, Haochen Ye, Huaiyuan Ying, Jia Yu, Jing Yu, Yuhang Zang, Chuyu Zhang. Li Zhang, Pan Zhang, Peng Zhang, Ruijie Zhang, Shuo Zhang, Songyang Zhang, Wenjian Zhang, Wenwei Zhang, Xingcheng Zhang, Xinyue Zhang, Hui Zhao, Qian Zhao, Xiaomeng Zhao, Fengzhe Zhou, Zaida Zhou, Jingming Zhuo, Yicheng Zou, Xipeng Qiu, Yu Qiao, and Dahua Lin. Internlm2 technical report, 2024.
Guoxin Chen, Minpeng I Liao, Chengxi Li, and Kai Fan. Alphamath almost zero: process supervision without process. arXiv preprint arXiv:2405..e4 03353,, 2024a.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda. Nicholas Joseph, Greg Brockman, al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.033/7 2021.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01335, 2024b.
Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. arXiv preprint arXiv:2310.02743. 2023.
Rémi Coulom. Efficient selectivity and backup operators in monte-carke tree search. In International conference on computers and games, pp. 72-83. Springer, 2006.

---

Published as a conference paper at ICLR 2025
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeeback:: Boosting language models with high- quality feedback. arXiv preprint arXiv:2310.01377 2023.
Haikang Deng and Colin Raffel. Reward -ngaa--a decoding: Efficient controlled text generation with a unidirectional reward model. arXiv preprint arXiv:2310.09520, 2023.
Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:230.066667, 2023.
Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint wwXiv:2405.07786. 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.21700 178.3 2024.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled al- pacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.044: 044444454. 2024.
Xidong Feng, Ziyu Wan, Muning Wen, Ying Wen, Weinan Zhang, and Jun Wang. Alphazero- like tree-search can guide large language model decoding and training. arXiv preprint vrXiv:2309.17179, 2023.
Shangmin Guo. Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model alignment from online ai feedback. arXiv preprint arXiv:2402.044 04092, 2024.
Alex Havrilla, Sharath Raparthy, Christoforus Nalmpantis, Jane Dwivedi- Yu, Maksym Zhuravin- skyi, Eric Hambro, and Roberta Railneau. Glore: When, where, and how to improve Ilm reasoning via global and local refinements. arXiv preprint ArXiv:2402. 10963, 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Train- ing compute-optimal large language models. arXiv preprint 220..v22200.155560 2022.
Jesse Hostetler, Alan Fern, and Tom Dietterich. State aggregation in monte carlo tree search. In Proceedings of the AAAI Conference Artificial Intelligence, volume 28, 2014.
Gareth Michael James. Majority vote classifiers: theory and applications. Stanford University, 1998.
Youngsoo Jang, Seokin Seo, Jongmin Lee, and Kee-Eung Kim. Monte-carlo planning and learning with language action value estimates. In International Conference on Learning Representations, 2021.
Jaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, et al. Ai alignment: A comprehensive survey. arXiv preprint arXiv:2310. 19852, 2023.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint 2XX0:2000.08866 2020.
Saeed Khaki, JinJin Li, Lan Ma. Liu Yang. and Prathap Ramachandra. Rs-dpo: A hybrid rejection sampling and direct preference optimization method for alignment of large language models. arXiv preprint 20..2.422//0. 2024.
12

---

Published as a conference paper at ICLR 2025
Maxim Khanov, Jirayu Burapacheep, and Yixuan Li. Args: Alignment reward-guided search. arXiv preprint arXiv:2402.016644 2024.
Levente Kocsis and Csaba Szepesvári. Bandit based monte-carlo planning. In European conference on machine learning. pp. 282-282222222 293. Springer, 2006.
Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. Step-dpo: Step- wise preference optimization for long-chain reasoning llms. arXiv preprint (tti:2400.1862999 2024.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannanch Hajishirzi. Rewardbench: Evaluating reward models for language modeling, 2024.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, et al. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00261 2023.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20055, 2023.
Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. arXiv preprint arXiv:2309.0663 06627799.6667, 2023.
Jianqiao Lu, Zhiyang Dou, Hongru Wang, Zeyu Cao, Jianbo Dai, Yingjia Wan, Yinya Huang, and Zhijiang Guo. Autocv: Empowering reasoning with automated process labeling via confidence variation. arXiv preprint 2rXiv:2405.16802 2024.
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai Wei Chang, Song-Chur Chng-Chun&on Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference Neural Information Processing Systems (NeurIPS), 2022.
Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, et al. Improve mathematical reasoning in language models by automated process supervision. arXiv preprint arXiv: 24X6522406.06592, 2024.
Jinjie Ni, Fuzhao Xue, Xiang Yue, Yuntian Deng, Mahir Shah, Kabir Jain, Graham Neubig, and Yang You. Mixeval: Deriving wisdom of the crowd from Ilm benchmark mixtures. arXiv preprint arXiv:2406.0653 06565, 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Richard Yuanzhe Pang. Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. arXiv preprint arXiv:2404. 19733, 2024.
Junsoo Park, Seungyeon Jwa, Meiying Ren, Daeyoung Kim, and Sanghyuk Choi. Offsetbias: Lever- aging debiased data for tuning evaluators. arXiv preprint arXiv:2407.0655/1 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean- baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat. Julian Schrittwieser. et al. Gem- ini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.nm 05530, 2024.
13

---

Published as a conference paper at ICLR 2025
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Di- rani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-prec prooee q&a bench- 2023.
mark. arXiv preprint arXiv:
Eric Sven Ristad and Peter N Yianilos. Learning string-edit distance. IEEE Transactions Pattern Analysis and Machine Intelligence, 20(5):522-533 1998.
Amrith Sethur, Saurabh Garg, Xinyang Geng, Naman Garg, Virginia Smith, and Aviral Kumar. RI on incorrect synthetic data scales the efficiency of Ilm math reasoning by eight-fold. arXiv preprint arXiv:2406. 14532, 2024.
Tianhao Shen, Renren Jin, Yufei Huang, Chuang Liu, Weilong Dong, Zishan Guo, Xinwei Wu, Yan Liu, and Deyi Xiong. Large language model alignment: A survey. arXiv preprint arXiv:2309. 15025, 2023.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schritwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587)44844889 2016.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.0333 0331,, 2024.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
Iya Sutskever, Oriol Vinyals, and Quoc V Le.. Sequence sequence learning with neural networks. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 27, Curran Associates, Inc., 2014. URL https: http://oooeeednnn.c.o wwwpiiinnnnnnuuuuneeeeeeee newrips cc www....f........p...............
Zhengwei Tao, Ting-En Lin, Xiancai Chen, Hangyu Li, Yuchuan Wu, Yongbin Li, Zhi Jin, Fei Huang, Dacheng Tao, and Jingren Zhou. A survey on self-evolution of large language models.
arXiv preprint arXiv:2404. /4387, 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.1 1805, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song. Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14271 2022.
Tu Vu, Kalpesh Krishna, Salaheddin Alzubi, Chris Tar, Manaal Faruqui, and Yun-Hsuar Sung- Foundational autoraters: Taming large language models for better automatic evaluation. arXiv preprint arXiv:2407. 101117 2024.
Haoxiang Wang, Yong Lin, Wei Xiong, Rui Yang, Shizhe Diao, Shuang Qiu, Han Zhao, and Tong Zhang. Arithmetic control of Ilms for diverse preferences: Directional preference alignment with multi-objective rewards. arXiv preprint arXiv:2402.18571 2024a.
Peiyi Wang. Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce Ilms step-by-step without human annotations. In Pro- ceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9426 9439, 2024b.
Tianlu Wang. Ila Kulikov, Olga Golovneva, Ping Yu, Weizhe Yuan, Jane Dwivedi- Yu, Richard Yuanzhe Pang, Maryam uators. arXiv preprint arXiv:2408. 02666, 2024c.
Jason Weston, and Xian Li. Self-taught eval-
14

---

Published as a conference paper at ICLR 2025
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj. Xuan He. Ziyan Jiang, Tianle Li, Max Ku, Kai Wang. Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark, 2024d.
Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. Aligning large language models with human: survey, arXiv preprint arXiv:2307.1296, 2023.
Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer2: Open-source dataset for training 2024e.
lop-performing reward models. arXiv preprint
Zihan Wang, Yunxuan Li, Yuexin Wu, Liangchen Luo, Le Hou, Hongkun Yu, and Jingbo Shang- Multi-step problem solving through a verifier: An empirical analysis on model-induced process supervision. arXiv preprint (tt::2402.0265566 2024f.
Sean Welleck, Amanda Bertsch, Matthew Finlayson, Hailey Schoelkopf, Alex Xie, Graham Neubig, Ilia Kulikov, and Zaid Harchaoui. From decoding to meta gentagenneattion: Inference-time algorithms for large language models. arXiv preprint arXiv:2406, 16838, 2024.
Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min-Yee Kan, Timothy P Lillicrap, Kenji Kawaguchi, and Michael Shieh. Monte carlo search boosts reasoning via iterative preference learning. arXiv preprint arXiv:2405.004: 2024.
Huajian Xin, ZZ Ren, Junxiao Song. Zhihong Shao, Wanjia Zhao, Haocheng Wang, Bo Liu, Liyue Zhang, Xuan Lu, Qiushi Du, et al. for reinforcement learning and monte-carlo tree search. arXiv preprint arXiv:2408.08/55 2024.
5: Harnessing proof assistant feedback
Wei Xiong, Hanze Dong, Chenlu Ye, Han Zhong, Nan Jiang, and Tong Zhang. Gibbs sam- pling from human feedback: A provable kl-constrained framework for rlhf. arXiv preprint arXiv: 11456, 2023.
Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, et al. Advancing IIm reasoning generalists with preference trees. arXiv preprint arXiv:2404.0000 2024.
Dan Zhang, Sining Zhoubian, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mets* Lim self- training via process reward guided tree search. arXiv preprint arXiv:2406.038 242.222660.8666 2024.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. mnstruction-follwiin evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023.
15

---

For building preference trees, compute the edit distance segments of different response can- didates, splitting the entire responses using "A'n". The threshold for segment aggregation set dif- ferently for all tasks based the distribution of similarity, and the rewards gap threshold between Published as a conference paper at ICLR 2025
A DETAILS ABOUT DERIVATION OF PRM FROM ORM
The cumulative error from supervising only the outcome can be expressed as follows:
T
T-i+1
Cumulative Error
4
(4)
1-a
==1
where E, the error at step i, and a represents the degree to which errors propagate to subsequent steps.
The reduction cumulative enror can be expressed as:
T
1 a -++1
Cumulative Error new Σ B.6.
(5)
1
where / is a reduction factor that depends on the discrepancy between Pa and pr, and the strength of the supervision. We define 3 as follows:
(6)
1 Pr).
where (yes Pr) the discrepancy between the key steps, Y model-dependent factor repre- senting the model's sensitivity to supervision, and $ is the strength of the supervision signal. As A A(pe.p)) P.) and S increase, the factor 3 decreases, leading to a significant reduction in cumulative error
B DETAILS ABOUT OPENPRM TRAINING
B. DATASETS FOR PREFERENCE TREE BUILDING
As introduced Section 4.1, we con- struet preference tree data primarily using UlrraFeedback, which consists of mixture of instructions in the open domain. Additionally, we incor- porate instructions from the Math and STEM domains to enhance the gen- eralization and reasoning capabilities of open-domain models. Beyond pro- cess pairs built with preference trees, also include some outcome-level preference pairs to maintain the capa- bilities of ORMs. which can be seen as a specific case PRMs. We provide all the statistics of the datasets used in Table 4.
Dataset
Orig
Retain
Outcome?
Process?
UltraFeedback
59,876
N/A
X
+PTS
N/A
70,068
ScienceQA
6,508
N/A
X
+PTS
N/A
12,958
MATH
7,500
N/A
X
+PTS
N/A
19,913
V
HelpSteer 2
N/A
7,221
X
Table 4: Statistics of training datasets. PTS is preference tree sampling strategy proposed 3.1.
Training Data Format. When preparing the training data for the PRM, reformat all process- level and outcome-level pairs into unified format: [Q,C, C, Pa, P.]. where P. and P/ represent the chosen (preferred) and rejected (non-preferred) answers, respectively, based the same context C. For outcome-level pairs, [C, Pa] and [C, P.] represent the complete answers. For process-level pairs, these concatenations represent partial answers.
B.2 HYPER- PARAMETERS FOR TRAINING
process pairs L.0 for UltraFeedback and 0.2 for Math and ScienceQA. For both Llama-3.1 --BB Instruct and I. 1. 1-70B-Instruch sample 64 candidate responses for each instruction. Due
16

---

Published as a conference paper at ICLR 2025
constraints on inference cost, we randomly sample 10,000 instructions from UltraFeedback for the -lama-3.1-708 -0.ttrtt-nttructt model. We set the temperature to 0.5 and top-p to 1.0 for repeated sampling with vLLM engine 7
For reproducing the UltraFeedback and HelpStecr2 reward models, we finetune Llama-3-88 8B-Insruuct using learning rate 10-6 over 1 epoch. Meanwhile, finetune InternRM and FsfairX on process pairs using a learning rate 1 x 10-6 over 1 epoch. All models are finetuned with a batch size of 64 and a maximum sequence length of 2048.
B.3 TREE BUILDING EXAMPLE
As illustrated in Figure 2 (mainly segment and aggregate), we first split each answer into sentences and then merge similar sentences across all answers sequentially. For each merging operation, the candidate sentences are sourced from the same parent node and indexed consistently across their respective answers. To clarify, we provide a toy example along with the corresponding tree for merging three answers from ScienceQA in Table 5.
Table 5: Example for similarity-based sentence merging for preference building.
"Question:" n Which logical fallacy is used in the text?' nBefore refute my opponent's 's argument, would like to draw attention to the fact that he is sweating and clearly does not have much experience on the debate team. Soomm/nntttions.yy.e nA. ad hominem: an attack against the person making the argument, rather than the argument itself' nB. bandwagon fallacy: the assumption that the popular choice is automatically correct' nC. appeal to nature: the assumption that natural things are always good') nAnswer: Let's think step by step." Indices: []
Node(DepthO):
Reward: [] - Reply
Node(Depth1): "To identify the logical fallacy used in the text, let's analyze the situation step by step" - Reward: [0. 1] - Reply Indices: [0. 2]
Similar sentences: "To identify the logical fallacy used in the text, let's analyze the situation
step by step" process step by step"
"To identify the logical fallacy in the text, let's break down the reasoning
Node(Depth2): "The text states that the speaker wants to refute their opponent's 's argument, but instead of addressing the argument itself, they choose comment on their opponent's phys- ical appearance (sweating) and a supposed lack experience on the debate team. This kind of comment is meant to discredit the opponent rather than engaging with the argument they pre-
sented" - Reward: [O] - Reply Indices: [0]
Node(Depth3): "The key phrase here is 'attack against the person making the argument, rather than the argument itself,' which the definition of the ad hominem fallacy. \n Thherefore,, the correct answer is:) nA. ad hominem: an attack against the person making the argument, rather
than the argument itself." - Reward: [0] Reply Indices: [0]
(DeptDe::: "The text states, 'Before I refute my opponenté argument, would like to draw attention to the fact that he is sweating and clearly does not have much experience on the debate team.' This statement does not address the opponents argument itself but instead makes
personal attack on the opponents appearance (sweating) and experience, implying that these characteristics make his argument weak" - Reward: [1] Reply Indices: [2]
NDdphDe: "Now, let's analyze the options: ) nA. ad hominem (A): This type of fallacy involves attacking the person making the argument rather than addressing the argument itself. In this case, the statement attacks the opponent's 's person (by mentioning their sweating and lack of experience) rather than addressing the merits of their argument" Reward: [1]
Reply Indices: [2]
Node(Depth4): "B. bandwagon fallacy (B): This fallacy occurs when it is assumed that something is correct or good because it is popular. There is no indication in the text that the opponent's argument is being rejected because unpopular, but rather because of personal
characteristics" - Reward: [1] Reply Indices: [2]
Phttps:// thub. com watpo//v11mmpooeeettcc propepcto///lll
17

---

Published as a conference paper at ICLR 2025
Node(DepthS): "C. appeal to nature (C); This fallacy involves assuming that some-
thing is good true because it is natural. The text does not mention anything about what is
natural or unnatural, so this option does not apply" - Reward: [1] - Reply Indices: [2]
Node(Depth6): "Given the analysis, the correct answer is A. ad hominem, as the
text attacks the opponent's person rather than addressing the argument itself." - Reward: [1]
Reply Indices: [2]
"To identify the logical fallacy in the text, let's break down step by step:\nin,.. **Ientiff the claim and the counterargument** The text claims that the oppo- nent's argument flawed but before refuting it, mentions that the opponent is sweating and lacks experience on the debate team. This is an attempt to undermine the opponent's credibility without addressing the argument itself" - Reward: [1] - Reply Indices: [7]
Node(Depthl);
Node(Depth2): "2, ** Analyze the nature of the attack**; The attack is not on the argu- ment's merits but on the opponent's 's character (sweating, lack of experience). Sweating can be sign of nervousness, which isn't an't inherently indicative of the validity of one's me's argument. Lack of experience also not directly related to the quality of the argument unless it's the first time the opponent making argument, which isn't specified"
Reward: [1] [ Reply Indices: [7] Node(De) "3. **Classify the fallacy based on the analysis*** The attack is focused on undermining the opponent personally rather than addressing the argument. This matches the description of the "ad hominem" fallacy, which involves attacking the person making an argument rather than addressing the argument itself" Reward: [1] Reply Indices: [7]
Node(Depth4): "Therefore, the logical fallacy used in the text is **d hominem
Reward: [1] -
Reply Indices: [7]
Node(DepthS): "The final answer is: A" Reward: [1] - Reply Indices: [7]
B.4
RESULTS OF REWARD SHAPE
FsfainX-88
HelpSteer2-88
Intern-78
OpenPRM
100
-15
-10
-5
-40
-20
20
-5
Figure 5: Rewards on chosen and rejected content of various RMs.
C DETAILS ABOUT EVALUATIONS
We summarize the open- open-domain datasets used in our evaluations follows:
Alpaca Eval 2 (Dubois et al., 2024): A popular benchmark for evaluating instruction-based language models using automatic evaluators such as GPT-4. features approximately 800 open-domain prompts. Given the length bias in GPT-4 evaluations, Alpaca Eval 2 employs length-comtrolled win-rates.
MixEval Hard (Ni al., 2024): A bround-truth-basse dynamic benchmark derived from established benchmark mixtures. It evaluates LLMs using highly capable model rank- ing system. MixEval Hard includes both free-form and multiple-choice questions, each category containing 500 questions.
MATH500 (Lightman et al., 2023): A subset of the MATH test dataset from OpenAl, featuring 12,500 challenging competition mathematics problems. We the MATH500 version, which contains 500 samples that maintain IID consistency with the original test dataset, to evaluate the mathematics abilities of LLMSs under scaled inference-time settings.
18

---

Published as a conference paper at ICLR 2025
MMLU-PRO (Wang et al., 2024d): An enhanced benchmark designed to evaluate models across broader and more challenging tasks. Built upon the MMLU dataset, MMLU-PRO integrates more challenging, reasoning-focused swer choices per question from four to ten, significantly raising the difficulty and reducing the chance of success through random guessing. We randomly sample 500 questions from the test data for our evaluations.
focused questions and increases the number of an-
GPQA (Rein et al., 2023): Consists of PhD-level STEM questions generated by experts in biology, physics, and chemistry. The original GPQA dataset is divided into main, diamond, and extended parts. We utilize the diamond split to align with OpenAl results, which includes about 200 questions.
IFEval (Zhou et al., 2023): Designed to evaluate the instruction-follooinng abilities of chat models. It focuses on a set of verifiable instructions and includes over 500 prompts with tasks such "write article with more than 800 words" and "wrap your response with double quotation marks."
During implementation, we use zero-shot chain-of-thoughr promips for MATH and GPQA datasets based on prompt code in openai emple-ealls repository' 4. For Alpaca Eval 2, MixEval, IFE- val and MMLU-Pro dataset, we use the evaluation code from the official GitHub repository 6.7 Specifically, we found that Mistral-Nemo struggles to adhere to answer format instructions; there- fore, we opted to use few-shot Chain of Thought (CoT) examples instead of zero-shot CoT.
D ADDITIONAL RESULTS
D.1 DIFFERENT AGGREGATION STRATEGIES
For applying PRM to outcome-level pairs, we explored several aggregation strategies for calculating step-based rewards, as detailed below:
Last Step: Use the reward of the final step as the overall reward, similar to ORM.
Max/Min Step: Select the maximum or minimum reward among steps.
Simple Average: Calculate the average reward across all steps.
Weighted Average: Apply a positional weight, giving later steps higher importance. The formula is: r E r
Dynamic Aggregation: Utilize uncertainty-weighte optimization (UWO) (Coste et al., 2023), which dynamically adjusts weights based on intra-ensemble variance. This penal- izes policies generating outputs with high disagreement across steps.
Max/Min Delta: Inspired by recent research, also compute the delta (difference) be- tween step rewards and use the maximum or minimum delta as the final reward.
Different Performance on Chat Category. As shown in Table 2 and 7, OpenPRM's scores in the Chat category decrease, while performance in the Chat Hard category improves. This reflects the inherent trade-off between optimizing for simpler conversational tasks (Chat Easy) and more complex reasoning tasks (Chat Hard), which a known challenge reward model optimization, as also noted in RewardBench (Lambert et al., 2024). The distinction between these categories lies in their data sources: the Chat category includes datasets like AlpacaEval and MT Bench, while Chat Hard is derived from MT Bench and LLMBar. The decreased scores in the Chat category largely due to AlpacaEval, as there a distributional shift between this dataset and our training data, which sourced from Ulraaeedback.
19

---

Published as a conference paper at ICLR 2025
Table 6: Overall score of RewardBench (including Chat, Chat Hard, Safety and Reasoning) and PRM800K Test Results with different aggregation strategies.
Models
Aggregation
Overall
Chat
Chat Hard
Safety
Reasoning
PRM800K Test
intern2-76 20-22-b-MMM
ORM
87.6
99.2
69.5
87.2
94.5
61.0
intern-77b-Be + FT
ORM
89.6
87.7
84.2
92.4
94.1
63.4
Last Step
91.1
98.0
81.6
89.5
95.1
68.1
Min Step
91.9
96.7
83.6
91.6
95.7
68.1
Max Step
88.7
95.0
80.5
88.2
91.1
68.1
Simple Avg
91.3
97.2
83.3
89.9
94.8
68.1
OpenPRM (intern)
Weight Avg
91.4
98.0
82.7
89.7
95.0
68.1
Dynamic
91.6
97.8
83.3
90.4
95.0
68.1
Min Delta
77.8
64.3
75.7
76.4
95.1
68.1
Max Delta
77.7
77.4
69.5
74.3
89.5
68.1
D.2 EVALUATE RMs WITH OFFSETBIAS
In addition benchmarks like RewardBench, UltraFeedback, and PRM800K, evaluated our reward models using OffsetBias (Park et al., 2024), which provides more granular assessment of bias reward models. The results are summarized in the table below, showcasing the effectiveness of OpenPRM across various bias metrics in Table 7.
Table 7: OffsetBias Evaluation Results
Content/ Continuation
Empty Reference
Familiar Knowledge Preference
Length Bias
Nested Instruction
Models
Concreteness
Overall
Furus-RM M-
71.4
66.7
84.6
33.3
41.2
66.7
60
FsfairX-LLaMAA-RM
100
91.7
53.8
91.7
41.2
58.3
71.3
FafainX-LLaMA3-RM
+ OffsetBias
92.9
100
46.2
58.3
82.4
83.3
77.5
LLaMA3-8B-Insmee
100
95.8
92.3
83.3
85.3
50
+ OffsetBias
Intem2: 7h RM
58.3
58.8
91.7
84.8
Intem2-7h-RNM
+ OpenPRM (Our)
92.86
83,3
83.3
88.2
91.7
89.9
D.3 COMPARISON WITH MCTS METHODS
We compared the computational efficiency our method with MCTS--ased approaches under sim- ilar sampling budgets. The experimental setup and results are as follows:
OpenPRM: We start from 60k samples, with 64 responses per sample, producing approx- imately 3.84M question-answer pairs in 24 hours. From this, 90k pairs were selected for training.
MCTS: We start from 10k samples, with 4 responses per sample. Responses split into sentences, resulting 240k partial outputs. Sampling 8 full paths for each pair of partial outputs produced 3.84M question-answer pairs in 24 hours. Of these, 97k pairs were used for training.
The results of PRM trained with OpenPRM and MCTS are shown in Table fact, the MCTS method remains powerful baseline but is significantly more resource-intensive, requiring up to 10 times the computational cost our approach. With a larger sampling budget, the MCTS method could still be effectively utilized. However, method offers more efficient alternative and can be further optimized with more accurate similarity computation techniques, such as embedding- based methods. While these approaches may increase the time required to construct the tree, they hold great potential for improving performance. We plan to explore these optimizations in future work to further enhance the efficiency and accuracy of method.
20

---

Published as a conference paper at ICLR 2025
Table 8: Comparison with MCTS methods
Models
A Agregattio
Overall
Chat
Chat Hard
Safety
Reasoning
PRM800K Test
intern2-7b-RM
ORM
87.6
99.2
69.5
87.2
94.5
61
intern2-7b-RM
ORM
89.6
87.71
84.21
92.43
94.06
63.4
outcome labels
OpenPRM (intern)
Min Step Min Step
91.9
96.7
83.6
91.6
95.7
68.1
MCTS (intern)
91.4
95.5
81.6
93.2
95.4
68.2
D.4 RESULTS ON MORE LANGUAGE MODELS
Scaling Effect of PBS. We have analyzed the scaling effects of Process Beam Search (PBS), where beam search is conducted at the sentence level, selecting the top N generated outputs based on PRM rewards. As presented in Figure the results indicate that OpenPRM consistently outperforms Bag of N-grams (BoN) in PBS settings, showcasing its effectiveness and reliability in these scenarios. However, we also observed significant variance in the scaling effect of PBS across different tasks, in contrast to the more consistent scaling effect seen with best-of-N methods. This variance can likely be attributed differences the data distributions across tasks, which highlight the need for further investigation into handling data mixtures effectively. We plan to continue exploring this issue in future work better understand and address these challenges.
Llama-3.
Llama-3. 1lama-3.--B--nstrucc.
112
124
Accuracy (96)
Accuracy
2/3
222
Number Samples (N)
Number of Samples (N)
Figure 6: Scaling Effect of Process Beam Search
Scaling Effect of BoN, We present the remaining tasks, in addition to those shown in Figure 3, in Figure 7. We provide additional results for the regarding inference-time scaling in Figures 8, 9, 10 and 11, which support the same conclusions as those drawn from the Llama-3. 3.8-..---BBBBB I-8B-Instruct in Figure 3.
and Mistral-Nemo model
1kmr
11---ii-ntttrrtttt.... AbaraEvvi2
100
70
47
76.5
45
A00Ura/ 150
35
44
70
40
35.0
45
34.5
40
41
40
35
Samples
Samples
Figure 7: Results of scaling inference-tine for Llama-3.1 3.1-BBBBBBBBBB--- --I-truuc on the rest tasks of Figure 3.
21

---

Published as a conference paper at ICLR 2025
Coverage
MarigN
BoNgRL/shwx
BuNgN.ur
BoNg OpenniiMM
Li 10B0--batraaccAA00
32-
(50
ACCurSOY (50
30
70
15
50
Number Semples (M)
Number (4)
Kumber (8)
Upmo InstructNivEedd
U.ma infruct
94
Accuracy (50)
-22
40
Samples
Samp
Figure 8: Results of scaling inference-time for Llama 1lamm-3.1--77 1-1--7B--Instrue on open-domain tasks.
Coveraga
MargeN
BoNgRLISRr
BONgNIH52
BONgNLUF
Boh.y,,item
Llama-J. 1- wtt..//---ittttuuuuouL.
Liame-5.1- www.-//--ttttttt.oo.
L. Lamma 5.2 70B Awo.-nutu...... MATH
ACCursSy 196)
+
54
Number Samples (K)
Number of Samples
Number Samples (1)
Usma 1:70 MikEval
[bmm: 3-170 www.innisuutttttiooooooo Apacssss
Uama-3 3-7000 instract IFEva
64.0
43.5
06.3
06.0
2' Samplee
Samples
Figure 9; Results of scaling inference-time for Llama-3 3.......117000 3.1-77B--nnstrne open-domain tasks.
22

---

Published as a conference paper at ICLR 2025
Eoverage
MarigN
BoNgRUAsPwX
BuNgN.ur
100
Nivire wwt.--------nerutt---tt-- MEILL--r
100
Mivira wtw.---e-e----or-----oo M00TH
Accumacyy (%))
700
AccurSOy
100
kumber Samples (8)
Number (4)
Number Semples (N)
Nistol 2407 MixEvol
Nizrol Nizzi/Noma Instru: AlparaBra12
Histral 2407.
30
Accurcyy
Accusecy
13
30
2 Samples
Samoles
Samples
Figure 10: Results of scaling inference-ting for Mistral-Nemn on open-domain tasks.
Covaraga
MargeN
BONgRUSSAR
BONgNIH52
BONGNLUF
Bohey,iitem
Matral-NMe Matrrl--eeee--ttruddaaoo MM MHLU-NO
Pisira -eeoo-iaa ruc1-2401 MAUT
37.5
2b.0
50
Number of Samp
Number Samp
Kumber Samples (K)
Mist himo Instruc 2407.M
Mistra www.eemmmn-nnnnaaaaa 2in
Mistral Nemo instruct 2407 iFEval
17
Accuracyy 16
augurszy
36
35
34
Samp
Stmp
Figure 11: Results of scaling inference-time for Mistral-Nemo on open-domain tasks.
23


Reviews:
[{"review_id": "187KjW71j4", "reviewer": "Reviewer_WSMi", "summary": "This paper presents OpenPRM, a reward model designed to incorporate both outcome-based and process-based rewards. The authors leverage outcome-based reward models to generate sentence-level preference trees, subsequently backpropagating these outcome rewards to provide weak supervision for process rewards. By jointly training the model on both types of signals, OpenPRM achieves huge improvements over existing ORMs and PRMs. Additionally, the authors demonstrate that OpenPRM can be effectively used during inference to guide the decoding process, with model performance scaling positively with the number of samples.", "questions": "See weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "This model could provide substantial value to the RLHF research community. Although similar ideas have been extensively explored in prior work, particularly in domains like math and coding that focus on reasoning, few of those works released their models and datasets. By extending to more open-domain tasks, this model could facilitate RLHF research beyond specialized tasks.", "weaknesses": "One minor weakness is the lack of novel technical contributions; however, given the value this model brings to the RLHF research community, this is not a major concern. A more significant weakness is the lack of detailed analysis of the experimental results. While the authors demonstrate that OpenPRM generally outperforms existing reward models, they do not provide a breakdown of the results. Specifically:\n1. The improved outcome-level reward performance is not clearly attributed—it's uncertain whether the gains come from the method itself or simply from better data. The data used (including UF, HS2, and additional math/coding datasets) may be better than that used in FsFairx or internRM. It would be helpful to see a baseline where these models are fine-tuned on the same data but with only outcome-level supervision to isolate the impact of the method.\n2. Although OpenPRM shows better results overall, its performance on Chat is consistently lower. An analysis explaining this discrepancy would strengthen the results.\n3. Figure 3 highlights the scaling effect of BoN, but no analysis is provided on PBM, which would provide a more complete picture.\n\nAdditional minor issues include:\n1. Figure 3 lacks clarity due to overlapping lines, making it difficult to interpret the results.\n2. The authors might consider evaluating with the OffsetBias dataset, which could offer a more fine-grained assessment of bias in reward models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PXLr0mZeET", "reviewer": "Reviewer_SVaF", "summary": "The authors proposed a method to create tree-like process supervision data. Specifically, it samples a great amount of trajectories from a set of samplers (LLMs) and merged those with similar prefix (measured by edit distance) to construct a tree. They trained different PRMs using this data and conducted experiments of Best-of-N and beam-search. The PRM-assisted method performed better on some of the selected benchmarks.", "questions": "You mentioned one of the motivation of the proposed method is to save compute. But I don't don't think there are compute difference for crafting a process supervision label. You still need to same amount of compute to sampling the trajectories. Please correct me if I didn't understand correctly here.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Process supervision and PRM are very important topics in the field. Especially given the improving importance of LLM reasoning and the success of OpenAI o1 model. This is an emergent topic and an open-accessible dataset is valuable to academia.\n2. The inference-time scaling experiment is very valuable. Experimenting with scaling law is always a very important thing in today's LLM research. Although it's often infeasible for most academic labs due to the resource constraints. I am very happy to see such analysis in the papers nowadays. The results are not very convincing though but experimenting it should be awarded with credits.", "weaknesses": "1. The biggest issue is the presentation of the paper. Seems many details are missing. For example: how the data merge exactly work. The paper only presented the general idea but I cannot find any detailed description, e.g., the edit distance threshold. Another example is table 2, I cannot find what metrics the table used. Is that per-step accuracy? If so, how could you evaluate PRMs using RewardBench (that's for ORM I remember?)\n2. They are some potential errors in the review of previous method. For example, I don't think any of the reward models are trained with MSE loss. It doesn't make sense to train a reward model with 0~1 labels where the label represented a probability using MSE loss. This is not a regression problem. And the Step-DPO work didn't train a reward model at all I believe.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uCPYXkwBg3", "reviewer": "Reviewer_4T3D", "summary": "This paper presents OpenPRM, a novel approach to developing process-based reward models (PRMs) for open-domain tasks like instruction following. The authors argue that while outcome-based reward models (ORMs) have shown success, their coarse-grained nature limits their effectiveness in supervising complex tasks. OpenPRM leverages existing ORMs and preference trees based on parallel sampled candidates to generate process-level rewards, leading to improved performance on various benchmarks. The paper also explores scaling inference-time compute in open-domain settings and finds OpenPRM superior to traditional ORMs in scaled settings. \n\nThe fundamental of this paper is about the generation of PRM training data, and it is using the similar method as Math-Shepherd and OmegaPRM. The key innovation of the this paper lies in the idea of merging solutions from the same question into a tree structure based on the similarity of the steps. Given this method would greatly reduce of the cost of the tree, the write and the experiments are not discussed throughly.", "questions": "Questions would be around the points in Weaknesses section.\n\n1. Why pick the temperature of 0.5 as most of the time, people would use high temperate to get diverse solutions.\n2. How the threshold 0.88 for similarity is picked and how does it affect the data and final result?\n3. How the training data is formatted and what loss was used? As there are different way of PRM training available, which one was used?\n4. During the inference time, how the rewards for each step/solution is calculated and aggregated?\n5. Can you give more detailed example of the solution and tree building as it is the core contribution of the paper?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper addresses a significant gap in the research by extending the application of PRMs from specialized domains like mathematics to open-domain tasks. The proposed preference tree construction using readily available ORMs is a creative and cost-effective solution to generate process-level supervision.\n2. OpenPRM consistently outperforms existing ORMs on RewardBench, particularly in challenging tasks like Chat Hard and Reasoning. Furthermore, it surpasses specialized PRMs in open-domain benchmarks, highlighting its generalizability. OpenPRM also shows promising results in scaling inference-time compute, outperforming other models in best-of-N sampling.\n3. The paper provides a detailed analysis of the limitations of ORMs in process-level supervision, highlighting the issue of cumulative error. The authors theoretically and empirically demonstrate how OpenPRM mitigates this problem by identifying key divergent steps in the process. The paper also addresses potential concerns like rationality of process aggregation and length bias, strengthening the validity of the proposed approach.", "weaknesses": "1. The tree building in OpenPRM is not convincing. The steps are separated by “.\\n”, and the similarity of steps are based on the edit distance between them, and threshold is pre-defined 0.88. The solutions are sampled with temperature 0.5. This temperature may not be high enough to generate solutions that are diverse enough for the reward calculation. I interpret this temperature is needed to generate steps that similar enough to convert into tree, but it affects the diverse of the solution and the rewards may not be precise. And how this 0.88 threshold selected was not mentioned in the paper, and the affect of different thresholds is not discussed either. As the key innovation of this paper, this is not a solid foundation.\n2. How the PRM is trained given the data collected was not discussed clearly in the paper. Some information about the mixture and hyper parameters are given, but how the data was structured into training examples, how the rewards are predicted during inference and how the rewards for steps are aggregated into final reward are not discussed. Without these information, it is hard to have a good picture of the PRM training and how to interpret the results presented in the paper.\n3. Only one example was given in the Appendix, and it was not clear. I would be very interested to see an real example of how the original solutions are like and how the final tree built looks like.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V6lcSZI7Sg", "reviewer": "Reviewer_MtX7", "summary": "This paper introduces a sentence-level process reward data collection method using Monte Carlo Tree Search (MCTS). The authors use outcome-based rewards and iteratively sample, segment outputs, aggregate at nodes, and apply back-propagation. Experimental results demonstrate that the collected process reward data effectively trains the process reward model.", "questions": "In Table 2, could the authors clarify how comparisons are made across different backbones and training data to ensure an \"apple-to-apple\" comparison?\n\nRegarding Table 3, could the authors provide the results from the outcome reward model?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Clear and well-structured writing makes the methodology easy to follow.\nThe method addresses the popular and relevant area of automatic process reward data collection.\nEffective use of MCTS enhances the quality of data for training process reward models.", "weaknesses": "Compared to previous MCTS-based solutions for process reward models, the novelty of the data creation method is unclear. Further explanation from the authors on this point would be beneficial.\n\n## Update after author response\nThe authors conducted additional experiments to clarify the contributions of OpenPRM compared to MCTS. From the rebuttal results, it is evident that MCTS remains a strong baseline in terms of performance. While the authors claim that OpenPRM is 10x more efficient than MCTS, they also state that both methods can process 3.84M examples within 24 hours. This apparent contradiction requires further clarification from the authors. Overall, OpenPRM's novelty appears limited, positioning it as an incremental improvement.\n\nThe authors also provided comparisons between ORM and PRM, offering additional insights into the value of PRM.\n\nBased on the new evidence, I increase my score to 6, indicating a borderline accept.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)