Process Reward Model with Q-value Rankings
Authors: Wendi Li
Yixuan Li
Venue: ICLR 2025 Poster (2025)
Keywords: process reward model
reasoning
Primary area: other topics in machine learning (i.e., none of the above)

Abstract: Process Reward Modeling (PRM) is critical for complex reasoning and decision-making tasks where the accuracy of intermediate steps significantly influences the overall outcome. Existing PRM approaches, primarily framed as classification problems, employ cross-entropy loss to independently evaluate each step's correctness. This method can lead to suboptimal reward distribution and does not adequately address the interdependencies among steps. To address these limitations, we introduce the Process Q-value Model (PQM), a novel framework that redefines PRM in the context of a Markov Decision Process. PQM optimizes Q-value rankings based on a novel comparative loss function, enhancing the model's ability to capture the intricate dynamics among sequential decisions. This approach provides a more granular and theoretically grounded methodology for process rewards. Our extensive empirical evaluations across various sampling policies, language model backbones, and multi-step reasoning benchmarks show that PQM outperforms classification-based PRMs. The effectiveness of the comparative loss function is highlighted in our comprehensive ablation studies, confirming PQM’s practical efficacy and theoretical advantage.

Published as a conference paper at ICLR 2025
PROCESS REWARD MODEL WITH Q- Q-VALUE RANKINGS
Wendi Li
Yixuan Li
Department of Computer Science Huazhong University of Science and Technology wendi 1 @i@hhst.ttttem edu.
Department of Computer Sciences University of Wisconsin-Madison sharonl iêcs. www.wwcc..e...oo edu
ABSTRACT
Process Reward Modeling (PRM) is critical for complex reasoning and decision- making tasks where the accuracy intermediate steps significantly influences the overall outcome. Existing PRM approaches, primarily framed classification problems, employ cross-entropy loss to independently evaluate each step's s cor- rectness. This method can lead to suboptimal reward distribution and does not adequately address the interdependencies among steps. To address these limita- tions, we introduce the Process Q-value Model (PQM), a novel framework that redefines PRM the context of a Markov Decision Process. PQM optimizes Q- value rankings based novel comparative loss function, enhancing the model's ability to capture the intricate dynamics among sequential decisions. This ap- proach provides a more granular and theoretically grounded methodology for pro- cess rewards. Our extensive empirical evaluations across various sampling poli- cies, language model backbones, and multi-step step reasoning benchmarks show that PQM outperforms classification-basee PRMs. The effectiveness of the compara- tive loss function highlighted in our comprehensive ablation studies, confirming PQM's practical efficacy and theoretical advantage. Our codes can be found at https://g thub. httpu//.oo..iiiiyyeee///////// Pooos.yyyeeeeeee/oooo.........
INTRODUCTION
Process reward modeling (PRM) plays crucial role in tasks where the quality of intermediate steps is pivotal to achieving the final outcome (Lightman et al., 2024). In complex problem-solving sce- narios, such as mathematical reasoning or multi-step decision- maasionnmaking (Shao et al., 2024; Yu al., 2024; Hao et al., 2024), the accuracy and effectiveness of each intermediate action can significantly influence the overall success. Unlike outcome reward models (ORM) (Cobbe et al., 2021), which focus solely the final result, PRM provides detailed feedback at each stage of the process. By capturing the value of intermediate steps, PRM allows for a deeper understanding of how each action contributes to the overall goal. This granular approach supports the development of more sophisti- cated and reliable systems that can navigate complex tasks with greater accuracy.
Existing research typically frames PRM as classification problem (Wang et al., 2023a; Shao al., 2024; Lightman et al., 2024; Luo et al., 2024), where each intermediate state is classified as correct or incorrect. Specifically, for trajectory {ar a1, ....... aH} where I, a, H represent a question, a reasoning step, and the trajectory horizon, reasoning state s; = (I, a14-1) comprises the in- struction and text pieces previously generated (e.g. reasoning steps in reasoning tasks). Current research uses cross- cross-entropy loss to maximize the probability p(ca(s;) for each reasoning state, where G is the label indicating whether 8; is correct. While this approach has shown empirical success, it has notable limitations. Classification-bass account for the dependencies and nuances among states within a trajectory. This can lead subop- timal reward assignments, as these methods often ignore the relative importance of different steps and their influence on the overall process. Furthermore, these approaches lack theoretical grounding on how they approximate the desired reward function.
methods treat each state independently and do not
To address the challenges, we propose a novel framework Process Q- Model (PQM) which frames PRM as Q-value ranking problem. This framework allows us to capture the interdependen- cies among states and provides a more nuanced evaluation of each step's 's's contribution to the overall process. Specifically, our framework is grounded in the Markov Dynamic Process, where each

---

Published as a conference paper at ICLR 2025
Process Q-value Model (PQM)
Question:
What's the highest prime factor of total of disisors
$2005
43
41
45
4g
Step 1: Let's list the divisors $2008.
Step 4: We need oétain all ptime factors of $2008.
X
state: x @ A1:5
action: 06
Step The prime facturs of $2008 highest 958
Step The divisors ares1, 2, to, a0, 25, 48, 50, t0n, 2008.
Hence,
Predicted
21
22 2.76
23 3.22
Q4
25 -.67
2.
X
Q-value
1.68
0.93
1.72
Strp 3: The total of the showe divisors are $6h55
Step 6i The final result is $5$
X
larger Quvalias, mece likely achiere conrect Fital
sgnifiiant divergenss
smaller Q-raloe, less likely achieve
Figure I: Illustration of our proposed framework Process Q-value Model (PQM), The example highlights solution trajectory with six steps, where the first three steps correct and the last three steps incorrect.
tion On is text piece generated based on the current state 8/4 [2) 41:A-1). The LLM policy T (an/2, 41:A-1) maps the observed state to a distribution over the action space. The process reward model intuitively scores each action as based the instruction x and previous generations i.A-- 1- In the context of reasoning tasks, we introduce Q-value function for each state-action pair (8h, as)) as the probability of success achieving the correct final answer. Importantly, the Q-value function implicitly defines reward function for intermediate steps. Under this characterization, we formally derive the optimal Q-value rankings among reasoning steps. by which we then train PRMs to ap- proximate these rankings with a specialized comparative loss function. According to Theorem 3.5, Q-values ascend with the continuation of correct steps and descend as wrong steps proceed, while having a prominent gap between correct and wrong steps (see Fig. 1). We further prove that the previous classification-based PRM can be cast as a special case our theoretical framework under certain conditions.
We conduct comprehensive experiments, revealing the significant advantages of the PQM over prior methods. Following prior research (Wang et al., 2023a; Lightman et al., 2024; Luo et al., 2024), evaluate PRMs based on their verification ability through best-of-n sampling. The metric assesses the correctness of the most preferred trajectory selected by the PRM from candidates for each question. Compared to classification-bue based PRMs, our ranking- superior accuracy in verification, highlighting its effectiveness in capturing nuanced dependencies among steps. For example, when verifying solutions sampled from the 3a0-----00- model, PQM improves the accuracy from 39.8% 51.4%, a direct 11.6% improvement on the challenging MATH500 benchmark (Hendrycks et al., 2021). These results are consistent across diverse datasets, sampling policies, and L.LM backbones, underscoring PQM's s effectiveness and generalizability.
method PQM demonstrates
To summarize, our main contributions are follows:
1. We present new framework for PRM by framing it as Q-value ranking problem, provid- ing theoretical basis for process reward modeling that captures inter-dependencies among reasoning states. We also show that prior classification-based PRM can be cast as a special case under our framework.
2. We offer detailed theoretical analysis of PQM and validate its effectiveness through com- prehensive experiments on a wide range of sampling policies, LLM backbones, and differ- ent test sets.
3. We perform extensive ablation studies on the proposed comparative training objective, and analyze its variations to understand their impact on the model's performance and design.
PRELIMINARIES
LLMss for reasoning. Large language models have demonstrated impressive abilities on challenging reasoning tasks across wide range of math, science, and coding challenges. Chain of thought (Wei et al., 2022) and related techniques (Wang et al., 2023b; Yao et al., 2023; Besta et al., 2024a;b) have emerged dominant methods, linking the question and the final answer by series of intermediate reasoning steps. For a given question I and its corresponding answer y. extensive studies (Wei et al., 2022: Chen et al., 2023; Yao et al., 2023; Besta et al., 2024a;b) have shown that prompting LLMs to arrive at solutions via intermediate steps {ar, fa,,a,... a..... can produce more interpretable and accurate results. To generate the final answer, each intermediate step is sampled in auto-regressive manner:
2

---

Published as a conference paper at ICLR 2025
a TT(-12, (1:-11,, where xe denotes an LLM policy parameterized by 8. The final answer is then generated by y ~ [-]- 01, 01.02,... 02, :.... Note that the final answer can be considered the last reasoning
step, omit y in our subsequent discussion.
ORM vs. PRM. Outcome reward model (ORM) and process reward model (PRM) represent two distinct approaches to reward assignment in decision-making tasks, particularly in the context of reinforcement learning and language models. ORMs focus on the final outcome, assigning rewards based solely on the end state (Cobbe et al., 2021), which advantageous when the end goal is clear and well-defined. For example, this approach has been popularly used in LLM alignment frameworks for learning human preferences, where the emphasis is on aligning the model's 's final output with human judgments (Ouyang et al., 2022; Lee et al., 2023; Rafailov et al., 2024b; Khanov et al., 2024; Im & Li, 2024). However, ORMs often overlook the nuances of the process that lead to the final outcome, potentially ignoring valuable information embedded in the intermediate steps for multi-step reasoning tasks (Uesato et al., 2022).
In contrast, OpenAl's recent work on PRM (Lightman et al., 2024) has shown promise in assigning rewards based on the quality or characteristics of the intermediate steps. PRMs are particularly useful in tasks that require complex reasoning or multi-step problem-solving, where the path taken to reach the solution is as important as the solution itself. By rewarding intermediate steps, PRMs can encourage more interpretable and structured problem-solving processes, offering more granular training signal that captures the intricacies of the decision-making process.
Process reward modeling with BCE loss. For question and trajectory with several steps, 7 (Ir d1, .2, AH). current research on process reward models (Wang et al., 2023a; Shao et al., 2024; Lightman al., 2024; Luo et al., 2024) typically frames PRMs classification problem. This approach aims to maximize the predicted correctness of each reasoning state using a binary cross-entropy (BCE) loss,
H
(TC = H
Σ (c pologpp(cssss) + (1 ci) log(1 (1- - pp (s)),,
(I)
where is the gold classification label of i-th step, equal 1 when s; is correct intermediate state otherwise O. Despite its effectiveness, BCE loss treats each intermediate state independently and does not account for the interdependencies among the reasoning states within trajectory. By treating each state in isolation, BCE loss overlooks the relative contribution each step makes. More- over, the theoretical support for PRM formulation is also lacking. These limitations motivate approach of formulating process reward modeling as Q-value ranking problem grounded in the Markov Dynamic Process, where the focus shifts evaluating the relative quality of different steps in a solution trajectory. thus capturing the interdependencies among steps and providing a more holistic approach reward assignment.
PQM: PROCESS REWARD MODEL WITH Q- VALUE RANKINGS
In this section, introduce our framework PQM, which frames process reward modeling Q-value ranking problem. In what follows, we first define a Q-value function for reasoning tasks, which implicitly defines reward function for each intermediate step (Section 3.2). Then, we derive the desirable Q-value rankings among intermediate reasoning steps (Section 3.3), by which we can train PRMs to approximate the intermediate Q-values by a comparison-bart based loss (Section 3.4). Lastly, we demonstrate that classification-bass PRMs can be viewed as a special case within our theoretical framework (Section 3.5).
3.1 DETERMINISTIC MDP FOR LLMS
Formulations of MDP. A standard Markov Dynamic Process can be formulated as M (S,A,T,r,,) p. H), where S is the state space, A is the action space, T : S x A + A(S) is the transition kernel, is the reward function, P denotes the initial state distribution, and Hf is the max- imal number of interaction steps. A policy in MDPs, denoted by ग S + A(A), maps each state to distribution over actions. The interaction between the environment M and the agent can be described as follows. Initially, the starting state $1 is sampled from the initial distribution p. At each step I, the agent observes the current state 8t and selects action a based on its policy. The
3

---

Published as a conference paper at ICLR 2025
environment then transits to the next state st+1, which is sampled from the distribution T(+sst, at)- This process continues until a termination condition is met, which will be triggered within H steps.
Deterministic MDP for LLMs. In text generation scenarios, the transition kernel T is deterministic, as each new state is formed by concatenating the previous tokens with the current output. The length limit for LLM outputs is characterized by H. Initially, instruction is sampled from an initial distribution p. Each subsequent state 8, = (2, (1:t-1) comprises the instruction x and text pieces previously generated (e.g. reasoning steps in reasoning tasks). Each action dz text piece generated based on the current state 8t. The LLM policy T(a (z, d1=--1 maps the observed state to a distribution over the action space. The process reward model, r S x + E. intuitively scores each action Oe based on the instruction and previous generations d1:t-1. For simplicity, the instruction x is omitted in the state notation (z, (1:2) thereafter when no ambiguity arises.
3.2 DEFINING Q-FUNCTION IMPLICITLY DEFINES REWARD FUNCTION
Recall that the state-action value Q(s, a) (Mnih et al., 2013; Fan et al., 2020; Setlur et al., 2024) typically represents the expected benefit of taking a specific action to achieve a correct answer. In the context of reasoning tasks, we define the Q-value function as the success probability of achieving the correct final answer. Specifically, the Q-value function defined as
2 (a1=-1,, a.) o= (Em (Ea [(,, (1.H)).
where æ is policy, H is the maximum step number, is the sigmoid function and o function to ensure Q E IR. I is an indicator function, which equals 1 if the trajectory reaches the correct answer of x, and 0 otherwise. For simplicity, we also denote ((a a+) Q, when there is no ambiguity.
1 is its inverse
Lemma 3.1. (Ng al., 1999) For two reward functioos r(st, 0z) and r'(8), if there exists a potential function ((s) satisfying (' (8tt + ((8++1) - ((+)), these two reward functions results in the same optimal policy.
Given this lemma, defining the Q-value function implicitly defines a corresponding reward function. Lemma 3.2. Under deterministic MDP, the advantage function of the optimal policy nt" can function the same as the reward function leading to π."
Proof. Due to the deterministic MDP setting, we have (st,,, az) = r(s,, 0z) + (3(++)) -- (st) where we denote the Q, V-value under the optimal policy x² as Q" V Hence, with Lemma 3.1, we have the advantage function of the optimal policy functions the same as the reward function. C
With the definition in Eq. 2, the advantage function of the optimal policy can be formulated as A (s,, a.) = Q* (st. a.) En,-r' (-Ia)) 2* (st, a.) = (st,,,, az) (s Q1-1)
Thus, our objective is to approximate the Q-function of the optimal policy. However, the optimal policy is not known in advance and varies across different algorithms. To establish the relationships between Q-values intermediate steps, we introduce the following mild assumption regarding ideal optimal policies.
Assumption 3.1. For ideal optimal policy x², the next step based on correct state is more likely to be correct than be wrong, i.e. Pp" (at+1l01:z) >> Pp (at+1)01:)) which follows achieving correct answer from correct state is much easier than from a wrong state, i.e. PP* (P((ss) P*(3s).
In Section 4.3, we will further empirically validate this assumption. For the parameter notations of PT (-), we use the original notations represent the correctness of states and overline to indicate incorrectness. For example, Pd (a:+T|a denotes the probability that policy π will produce an incorrect next step given the correct state sequence a1:1, and Pa (r|5) represents the probability that the policy π generates correct trajectory from an incorrect state P is shorthand for p= where T is the optimal policy. Using the above definitions and assumptions, we can collect comparative labels to approximate Q-values, which will be introduced next.
3.3 OPTIMAL Q- VALUE RANKING
In this subsection, we derive the Q-value rankings among intermediate reasoning steps. In our main Theorem 3.5, establish that Q-values ascend with the continuation of correct steps and descend
4

---

Published as a conference paper at ICLR 2025
wrong steps proceed, while maintaining a significant gap between correct and wrong steps. To arrive at this result, we first derive the pairwise relationship between Q-values of an earlier step and later step in Lemma 3.3. Next we show the relationship between the first correct step and the first incorrect step in Lemma 3.4. Finally, we combine these intermediate relationships to derive an integrated ranking across the entire trajectory.
We start by introducing a few lemmas that are useful for deriving our main Theorem 3.5. For any two actions an, Om S.t. m in solution trajectory T (xr, 01 d2, .....) ). we have
P (*((a1n))) = P (a1:m (1:.) P (T (1:m) + P (71:m 01:m )P* (r (1.m)
(3)
(r((mn))) = P (a1:m (1mm) ) (T (1:m) + P (01:m Q1:m P (T (1:m);
(4)
which directly follows the Bayesian factorization. P (a a1m) denotes the possibility that policy generate correct state d1:m conditioned
wrong state d1:u- For a solution T (x), 01, 02 a.))
recall the Q function in Eq.2, we define 2i (a Qe) = (a
= P (T(a):)
where is the sigmoid function. Since is a monotonically increasing function, hence when 2 (a1:m-11 0m) > 2 (0 1, (n) for any two steps am, an, we have 2 (11:mm-,, 4m)
Q (a1m-11 an). Then we can obtain the following lemma.
Lemma 3.3. For two steps On; Om in a solution T where < m, if they are both correct, we have Q* (a1mn-1.00) 0n) 2* (a1mm-1. 0m). If Onz Om are both wrong, we have Q" (a1:n-1, an) > ("(1mm--1 (1:m-1,amm)
Proof. We first analyze the difference between the two correct steps follows,
2x (1 an) 2g (a1:mm am
(am a1m) P (T]a1:m) + P (am a1m ) (T(@1m) PP" (T(aam))
=p Pp (am (1:m) [P* (T1=m) P (T((aamm)
(5)
where the first equation uses the Q-function definition and Eq. 4, the second equation uses "p" (am (1mm) + P (am [1:a) = 1. With the Assumption 3.1, we have P (r (r|0).m) - -000-p"'(- (T(a):m) <
O. Hence, when an and am are both correct, we have 2* (a1mn11 an) < 2* (01:m-1 a.) Similar
to the above proof, we can factorize the Q-value difference between two incorrect steps as follows,
2 (01m-11, an) 2 (a1:m-1, am) PD (am [1x) (PM (T]a1m)) - PP (T|a
(6)
With the Assumption 3.1 where PD" (r((asm)m > (T((Tamm)) if an, Om are both incorrect, we have Q" (d1m-11 a,) 2" (A1:m-1, )m))
Additionally, considering the initial situation intermediate steps and (*)), we have the following lemma.
Lemma 3.4. For the first correct step an and the first incorrect step amd we have Qx (aim- an) > (²(x) > (a(mmmm11 a.m) ).).
Proofs. Considering the first correct step an. similar to the proof in Lemma 3.3, we have
(01=-11, an) V2 (x) = PD (r]a - (T (r) = P (a, (x) (r (1m) - P (r(amm))
(7)
O' (a1mmm1,, am) (z) PM a1mmm))) (r(xx) = Pm (am(l=)) (P' (r(a..)) P (r(a1.))
Hence, we have Q* (a):m-1 4m) < (x) < Q" a.). Now, we obtain the ordering of the Q-value difference, but the specific discrepancy between intermediate steps has not been discussed yet. With Assumption 3.1, for an ideal n², we have P (ā, < (amm (r).. Hence, the difference between V (*( and the Q-value the first correct step is much smaller than the difference between V (x:) and the Q-value of the first incorrect step.
Based on the above derivations, we can rank the state-action Q-values for the whole trajectory. We formalize the ranking in the following theorem.
Theorem 3.5 (Q-value ranking among reasoning steps). Formally, for a trajectory with H steps, C = [C1, .2...., .2.......0 GCj] denotes the index list of the correct intermediate steps, where ci < C2 < ... < qcT. W = [W1, W2...., W[]i]] denotes the index list of the wrong intermediate steps,
where w1 < W2 < < Wil we have
Eir, < 2:
2i, < < 22
where Q7 = V* (xx,, |. denotes the length of the list, and |C| + W H.
5

---

Published as a conference paper at ICLR 2025
3.4 COMPARATIVE LOSS FUNCTION FOR OPTIMIZING Q- VALUE RANKINGS
Given the optimal Q-value ranking derived in Theorem 3.5, we now propose new comparative loss that trains RPM approximate the intermediate Q-values. While the ranking relationship can be captured by the classical Plackett-Luce (PL) ranking model (Plackett, 1975; Luce, 1959), there are significant limitations when using the canonical PL loss directly in this context. The standard PL loss is designed to handle general ranking scenarios without accounting for the varying degrees of discrepancy within the ranking. However, our case, the Q-value gaps between correct and incor- rect steps are often highly pronounced (ef. Lemma 3.4), leading to situation where the standard PL. model may not adequately capture the importance of these differences. As discussed in Section 4.3, this results in suboptimal performance, since the PL loss does not differentiate sufficiently between steps that are only marginally different in rank versus those with substantial Q-value gaps.
Comparative loss with Q-value margin. To address the limitation, we adapt the vanilla PI. loss to better reflect these discrepancies. Our proposed loss function is designed to emphasize the signifi- cant gaps in Q-values, ensuring that the model learns to prioritize these differences in theoretically justified manner. The loss defined as:
IWI
ICI Σ log
exp( (n)))
Σnu exp Qc, + Ewew exp Qu + (t)) (9)
exp( Qcy)
Libeorem
log
H
En= exp Swa.
1=2
1=0
where ( is margin hyperparameter introduced emphasize the gap between correct and incorrect steps, and 0 is inserted at the beginning of C' for clarity.
Practically, prior research (Wang et al., 2023a; Shao et al., 2024) often treats all steps following the first incorrect step wrong. Specifically, for given trajectory where a1:-1 are correct steps and aj is the first incorrect step, existing data corpora typically catego- rize all subsequent steps Ht: incorrect. This approach leads to situation where the wrong steps are not necessarily accurately annotated, as they are all uniformly marked as incorrect. To address this issue and explore practically effective loss function, investigate several variations of the comparative loss function. Our practical implementation, which will be discussed in Section 4.3, is designed to better handle this scenario. The proposed loss function is:
{ai..... 4t-1, at...... ................................ a
IC|
exp( Qc.)
C C= <--
Σ log
(10)
[C|
Ex= exp exp 2r, Ewew exp(Q +
In this formulation, is positive scalar that adjusts the relative importance of incorrect steps, and Qo is set to 0 to simplify the computation. Comparing to Liheorem this objective disregards the internal rankings among incorrect steps, focusing solely the relative rankings among cor- rect steps and the substantial discrepancy between the Q-values of correct and incorrect steps, i.c. We will perform extensive ablation 2i
[Qi .... Die, 2i,, 2 2 < Q
comparing L and Ltheorem in Section 4.3.
3.5 CLASSIFICATION PRM IS A SPECIAL CASE OF Q- Q-VALUE APPROXIMATORS
We show that the previous dlssificationn-aseed PRM can be cast as a special case of our framework under certain conditions. To illustrate this, consider an extreme scenario where the assumptions outlined in Assumption 3.1 are satisfied, namely, when P* p*(a++1 ((1.)) According to the Q Q-function definition provided in Eq. and leveraging Bayesian Factorization, follows that classification based PRMs approximate Q-value rankings under these conditions.
and Pa (a.+1|a1.)) 1.
Lemma 3.6. Formally, when P" (at+11 + and Pd (at41 (1:4) -> 1 for any t, we have 2 (01 am for any correct step am and Q¹ (a)m-11, an) 0. any wrong step an.
Proof. This result can be derived directly from Bayesian Factorization, which states:
H
Dd (r((ammmm)
II (=++1
P" P'(ae(aaa-1) P (T((Tammm))
II (=++1
P* ('(or(aaa-))
(11)
and for wrong step, we have Ø; (a1m-1, a,, = 1-p-((p" (T (a].) 0. Thus, the cross- cross-entropy loss used classification- based PRMs can be interpreted estimating the Q-value without bias.
Therefore, for correct step. we have Q; (a:mm-111 1m)mm = PM (T(a1.m)

---

Published as a conference paper at ICLR 2025
Sampling Policy
Dalasel: MATH500
Dalasel:
GSM-Plus
Methods
@8
@16
@2
@64
128
@
@ @1
@
@64
@ 128
ORM
32.8
34.8
36.2
39.0
38.2
36.58
$7.63
57.17
57.63
58.33
MctaMath- Mistral-7B
-SE
33.2
36.2
37.6
38.8
38.4
58.21
58.75
58.71
58.50
58.17
MSFM
24.2
25.2
26.4
25.0
27.0
50.91
51.67
50.08
49.58
49.79
BCE
33.6
37.0
39.2
40.8
42.0
59.25
60.29
61.16
61.88
61.72
PQM Ç 2
34.8
37.0
39.6
41.8
41.2
62.42
64.04
64.92
65.25
66.00
PQM Ç
36.2
38.2
41.0
44.2
44.6
62.04
63.58
64.50
64.96
65.20
ORM
24.0
28.0
27.0
28.8
28.2
55.41
55.83
56.83
54.83
54.45
Muggle- Math-13B
MSE
28.2
30.2
33.0
33.6
34,0
56.42
58,42
58.38
58.67
59.08
MSEMCTS
21.2
24.2
22.0
23.8
26.8
42.75
45.83
46.95
45.67
46.33
BCE
30.4
31.4
33.4
36.4
37.0
57.50
59.79
61.16
62.00
62.
PQM C 2
30.0
33.4
34.4
36.8
35.0
60.58
62.54
64,25
64.79
65.62
PQM Ç
30.0
34.8
36.2
39.2
39.0
61.00
62.66
64.08
64.79
65.54
ORM
45.0
46.0
43.4
42.4
43.2
71.66
71.50
72.00
71.66
71.13
Llama-3- 70B-Instruct
MSE
41.6
42.2
40.0
36.8
38.0
71.79
71.67
71.96
71.25
71.04
MSEMCTS
39.6
40.4
40.0
41.2
41.4
68.46
69.70
67.79
71.13
70,66
BCE
43.6
41.4
41.6
42.4
39.8
72.16
71.83
72.04
71.38
70.75
PQM ( 2
47.6
49.0
50.4
48.4
51.4
72.04
71.95
72.70
72.33
72.33
PQM Ç 4
47.2
48.2
50.0
46.0
47.8
72.54
73.25
73.38
72.79
71.96
Table 1: Main results measured by best-of-n (BON @ @n) accuracy. The BON @1 of MATH500 for SitaMMath-Mistal-777 is 24.4, for MuggleMath-133 is 18.4, for 1amma-3-70B-Instrre is 37.4. The BON@
@ 1 of GSM-Plus for MetaMathh-Mistral-77 is 48.0, for MuggleMath- 13B is 43.16, for Llama- Boldface and underline indicate the best two results.
is 67.875
EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
Datasets and metrics. Following previous research (Wang al., 2023a; Lightman et al., 2024; Luo et al., 2024), we evaluate PRMs based on their verification ability through best-of-n sampling. The metric, BON @ n, assesses the correctness of the most preferred trajectory selected by the PRM from candidates for each question. During the evaluation, the PRM first scores every step within each trajectory. Consistent with prior studies (Wang et al., 2023a). the final score of trajectory is determined by the minimum score of its individual steps. The test corpus includes 128 solutions for each question from GSM-Plus (Li al., 2024) and MATH500 (Hendrycks al., 2021) datasets. These solutions are sampled from three policy models with strong performance in math tasks with different scales: MetaaMath-Mistrall-77 (Yu et al., 2024), MuggleMath- (Li al., 2023a), Llama- 3-70B-Instrue (A1 @ Meta, 2024). We utilize the existing off-shelf corpus, Math-Shepherd (Wang et al., 2023a), as our training corpus.
Baselines and implementation details. Consistent with prior works (Wang et al., 2023a; Lightman et al., 2024), we evaluate the performance of PRM by comparing it against the outcome reward model (ORM). We also compare comparative loss with the BCE loss, which is employed in Math-Shepherd. Additionally, some research (Zhang et al., 2024a; Wang et al., 2024) adopt more strict MSE loss minimize the distance between the predicted value and the label. We implement MSE loss with two versions: 0-1 label and iterative Monte Carlo Tree Search (MCTS) to estimate the continuous label for MSE loss in Zhang al. (2024a). For the model architecture, general reward model frameworks, incorporating value head on top of the Deepseek-77 7B--e------eeee LLM (Shao et al., 2024). This value head projects the latent representation of the model into a scalar value, facilltaatigg the evaluation of intermediate steps and trajectories. More detailed implementation in- formation, including specific configurations and experimental setups, can be found in Appendix B.
adopt
4.2 MAIN RESULTS
Verification performance across different policy models. Experimental results are shown in Ta- ble 1. Our proposed PQM demonstrates significant performance improvements over all baselines. Firstly, PQM outperforms the outcome reward model, which is consistent with prior findings that prscess-based methods provide more nuanced evaluation of intermediate steps. Moreover, when compared to classification-bast PRM models using BCE or MSE loss, PQM shows a notable ad- vantage. For example, when verifying solutions sampled from the 3-0-a----7---nn-tte model,
7

---

Published as a conference paper at ICLR 2025
MuggleMath- 13B
1lama-3-70B--0 3-70B-Inntrrct
Backhone for PQM
@8
@16
@2
@64
@12
@
@1
6664
@
128
@
@16
@2
@64
@
128
Deepseek-muth- mb-s-
36.2
38.2
41.0
44.2
30.0
34.8
39.2
39.0
48.2
50.0
46.0
47.8
beepseek-math--o 7b-rl
38.0
40.8
42.8
45.4
44.2
31.8
34.6
38.6
37.2
37.4
49.8
50.8
53.2
53.8
55.0
Qwen2-math-ll 1.55
31.4
32.8
34,6
33.8
33.2
25.4
28.2
30.4
35.2
32.4
41.2
39.2
40.0
40.2
39.4
Qwen2-math- 1.5b-inst
38.6
41.2
43.8
46.4
47.6
30.6
34.2
37.6
40.6
41.4
49.4
50.0
49.6
51.0
Metamath-7hb
30.4
32.8
32.8
31.2
33.8
26.2
30.6
29.6
30.2
30.0
42.0
44.8
45.4
44.8
44,0
Metamath-133
32.6
32.4
33.4
33.6
34.2
29.4
30.6
314
31.8
31.4
45.0
45.2
45.0
46.8
45.8
Table 2: Results of PQM across six different LLM backbones on MATH500, Ç is set to 4.
45
40
40
33
55
35
30
50
POMI2)
POMG2
PQM(4)
25
PQM(4)
PQM(4)
SC+POMI21
SC+POMI2)
SC+ PQM(2)
25
20
SC-POM)4)
SC: PQM(4)
81
98
@16
932
G664
@128
@1 @16
832
@64
1128
@1
88
816
@
864
@128
Figure 2: Integration of our approach PQM with self-consisteney (SC) three policy models, MetaMath- Beatth--B-Mannn 7B-Mistral (left), MuggleMath- 13B (middle), Llama-3-70B- Instruttt (right). The evalu- ation is conducted on MATH500. Numbers in brackets denote the value of Ç.
PQM improves the accuracy from 39.8% (BCE) to 51.4%, direct 11.6% improvement on the chal- lenging MATH500 benchmark. This result underscores the effectiveness of PQM in capturing the relative quality of different steps within a trajectory, addressing the limitations of BCE loss which treats each step independently without considering their interdependencies. PQM outperforms MSE loss with either 0-1 label or MCTS search. Compared to 0-1 label, MCTS search requires more computational resources but only leads to marginal performance enhancement. This may stem from its Q- Q-vauue definition with sophisticated heuristics, and theoretically biased estimation of Q-values in MCTS. Other results both the MATH500 and GSM-Plus datasets across three policy models further confirm the efficacy of PQM. In these benchmarks, PQM consistently outperforms existing methods, demonstrating superior performance across different policy scales and test sets, validating the efficacy of ranking- lanking-based process reward modeling.
PQM performance can be boosted by self-consistency (Wang et al., 2023b). By sampling mul- tiple trajectories and then selecting the final answer that appears most fiequently, self-consistency can further enhance the reliability of LLMs. In Figure 2, we report performance when combining seff-consistency with method PQM under both Ç 2 and Ç = 4. This integration capitalizes the strengths of self-consistency to further enhance the verification. The performance gap between PQM and SC+PQM increases we move the right in Figure since the large capacity model tends to reinforce the effectiveness of SC, leading to the increased performance gap observed in the figure. Our results reveal that this combination can boost performance, underscoring that blending self-consistency with process reward modeling provides a more effective verification strategy.
PQM remains effective under different LLM backbones. To explore the generalization of our approach, we train with PQM on additional LLM backbones, including Qwen2-Math-1.. Math-1.5B. Qwen2- -I (Yang et al., 2024), Meepseek-Math--ie Metamath-133B (Yu al., 2024). As shown in Table 2, stronger backbones generally lead to better overall performance under the same sampling policy model. Moreover, Qwen2-Math- 1.5tahhh-5--nsstrree achieves impressive results among six backbones, which indicates that a small-scale PQM can also provide effective verification if the backbone is specialized in mathematics.
(Shao et al., 2024), Metamath-7B and
4.3 FURTHER STUDIES
In ablation studies, we keep most of the experimental settings consistent with the main experiments, except that we use data with a length of less than 512 tokens, totaling 390k data out of 440k data, to the training cost. The detailed hyperparameters
shown in Appendix B.
Impact of margin Ç. In this ablation, we investigate how the margin Ç in our loss function influ- ences the performance. We implement several variations with 0, 2, 4, 8, 16. The experimental results are shown in Table 3, along with loss curves in Figure 5 (Appendix). Our experiments reveal that ( has minimal effect the convergence of training, the loss curves for all values flatten out
8

---

Published as a conference paper at ICLR 2025
MelaMath-Mistrall
MuggleMath- 13B
1lama-3-70B-mmm
Methods
@8
16
@32
@64
@ 128
e8
@
@16
@
@64
@
128
@8
@16
@32
@64
128
C. = 16
34.6
36.4
38.2
40.2
39.2
29,6
32.4
34.6
35.4
35.0
42.4
43.6
40.2
40.2
39.0
C. Ç 8
36.4
40.2
41.2
43.8
44.6
30.8
33.8
37.2
38.8
38.8
47.0
47.0
47.8
46.2
46.0
C. Ç 4
36.8
40.6
41.8
44.4
44.6
32.0
33.6
36.8
38.4
37.4
47.4
47.0
45.6
47.8
48.2
C. 2
35.8
39.0
40.8
43.4
43.8
30.2
32.8
34.2
36.8
37.4
47.4
49.0
50.6
51.2
50.4
C O
32.8
37.0
36.2
35.8
36.4
26.2
27.4
29.2
29.2
28.0
44.6
44.4
45.4
44.2
46.6
Liheccem: = 16
33.2
34.6
35.0
37.2
38.0
28.8
30.6
32.4
32.6
32.6
46.2
45.4
44.8
44.8
44.2
Lihrormm 8
33.6
34.4
35.0
35,4
35.6
29.0
29.4
30.0
31.4
32.6
43.8
42.6
41.0
38.2
37.4
Liheorem 4
35.4
38.2
39.0
40.0
40.2
31.6
33.2
34.8
36.4
34.8
44.8
45.2
46.4
47.8
46.0
Cmeoorm: 2
33.8
35.8
37.6
37.6
38.0
28.4
29.4
31.0
31.4
32.0
43.0
44.8
46.0
47.8
48.6
Lihoomm. 0
30.4 29.8
30.6
31.8
33.0
24.0
26.8
29.0
28.8
26.2
41.6
40.4
40.6
40.4
37.4
Table 3: Ablation results. The BON @1 of MATH500 for MeraMath--iistall-75 is 24.4, for
MuggleMath-13B is 18.4, for 1amma-3-70B-Insteee is 37.4. &, Citeorem refers Eq. 10 and Eq.9
respectively, Boldface and underline indicate the best two results.
0.8
0.8
(r(s)
1.
conrect trajectory
0.36
wrong trajectory
0.6
0.6
0.5-
9.8
0.34
0.12
0.4
0.5
0.4
5.7
0.
0.30
0.2
0.1
0.0-
0.5
0.02
ktep
0.50
Step
Figure 3: Empirical validation for Assump- tion 3.1.
Figure 4: Empirical evidence for Theorem 3.5.
after approximately 200 steps. However, the choice of Ç impacts the effectiveness of our method. As shown in Table 3, extreme values of Ç-eithrr too large or too small-lead to suboptimal perfor- mance. Specifically, Ç values of 2,4,8 yield the best results, whereas Ç values of and 16 perform less effectively. When Ç is too large, the comparative loss overweighs the discrepancy between the correct steps and wrong steps while neglecting the ascending relationship among Q-values of cor- rect steps. Conversely, when Ç is too small, the loss function fails to adequately capture Q-value discrepancies, leading to suboptimal performance. These findings align with our theoretical expec- tations and underscore the importance of choosing an appropriate to balance the comparative loss and capture meaningful Q-value distinctions.
Impact of loss design. Since the empirical training dataset automatically marks all steps after the first incorrect one as negative steps, we ablate the impact of these pseudo-negative steps by compar- ing our loss function with the theoretical version as delineated in Eq. 9. The findings, presented in Table 3, reveal the existence of noise in negative annotations. Specifically, when applying the theo- retical loss in Eq. 9, there is a marked decline in performance. We also explored another variant that emphasize the first negative step since the first negative annotation is verified by the automatic annotation. The experimental results and analysis are supplemented in Appendix C.
Empirical validation of Assumption 3.1 and Theorem 3.5. To empirically validate the Assump- tion 3.1 and Theorem 3.5, we use Llama-3. 1-taa-3..-700--mme Instrutt to substitute the optimal model π*. We sample 256 trajectories from Math-Step-DPO-I Se--Step-DP-OOOK (Lai et al., 2024), each consisting of more than six steps. For each step a; in each trajectory, we sample 32 times by I (+l01:). In Fig. 3, the left panel's y-axis shows the proportion of correct next steps, while the right panel's y-axis displays the proportion of correct trajectories. The z-axis indicates whether the generation is conditioned on correct state or an incorrect state. The plot demonstrates that when conditioned on a correct reasoning state, there is a higher probability of generating a correct subsequent step or completing correct trajectory. This validates our Assumption 3.1. In Fig. 4, zzaxis represents the i-th correct step (left) or wrong step (right), and y-axis represents the approximated Q. According to the graph, the approximated Q-values ascend with the continuation of the correct steps. Meanwhile, the latter wrong steps generally have smaller Q-values than the previous wrong steps. Moreover, there is a noticeable discrepancy between Q-value of correct steps (generally over 0.5) and incorrect steps (generally below 05).). Implementation details and more discussions can be found in Appendix C.
Qualitative example. For each step in the solution, we display the predicted probability of achiev- ing the correct final answer by ORM, classification based PRM, and PQM in Table 4. We also show the original Q value predicted by PQM, along with Q. = aQ). The Q-value predicted by PQM

---

Published as a conference paper at ICLR 2025
Q: Find all values of x that satisfy the equation x ==v11 27 4
ORM
BCE
2,
Q
Step I: Subtract 4 from both sides of the equation. - 4 v11 2x
0.916
0.424
-0.308
Step 2: Square both sides of the equation. (æ 4) =v111 2x)
0.882
0.487
-0.053
Step 3: Simplify..
8z + 16 =6=11 11 2x
0.848
0.482
-0.070
Step 4: Subtract 11 from both sides of the equation. x² -8-+++=222
0.628
0.004
-5.445
Step 5: Subtract 2x from both sides of the equation. - 10 +- =
0.584
0.004
-5.493
Step 6: Factor the quadratic. (2: () - 1) =
0,489
0.002
-6. 66.164
Step 7:
The final answer is 5 and 1. I hope it is correct.
0.475
0.399
0.001
-6.811
Table 4: A case study on MATH500. The solution is sampled by Llama3-70B-Instun
For each step, we display Q-value predicted by PQM(() and the estimated probability of achieving the cor- rect answer by ORM, BCE, and our PQM( 2x).. The steps after the first error (Step 4) are in gray.
has a sharp decrease at Step 4, which accurately locates the error. In contrast, the predicted proba- bility of classification-bast based PRM only decreases smoothly and exhibits large values even for wrong steps. We show more qualitative examples in Appendix E.
5 RELATED WORKS
Process Reward Models. Process supervision (Uesato et al., 2022; Li et al., 2023b), represented by PRMs, can provide more precise feedback, which is easier for humans to interpret, and more directly rewards models in step-by-step reasoning tasks. Most existing research (Lightman et al., 2024: Wang al., 2023a; Shao et al., 2024; Luo et al., 2024) formulates PRM as a classification problem, where the process reward is modeled as the probability of correctness of each step. We show that the prior approach can be cast as a special case under our theoretical framework. Due to the labor-intensive nature of dense annotations, several recent methods have introduced automatic annotation strategies (Wang et al., 2023a; Luo et al., 2024; Lu et al., 2024a). In these approaches, a step is deemed correct if a valid completion can be sampled from the LLM policy within & trials, see details in Appendix A. Generally, the subsequent steps after the first error are all treated as wrong steps in this line of methods. Additionally, Zhang et al. (2024a); Wang et al. (2024) estimate the Q- value of intermediate steps by iterative Monte Carlo Tree Search (MCTS) and MSE loss. However, their Q-value designs are different from ours, which generally incorporate sophisticated heuristics, e.g., reasoning distance and quality value. Moreover, their works necessitate dense online search over the large action space. Besides being costly, the distribution shift between the sampling policy and the optimal nr will result in biased estimation. In contrast, our comparative loss is easy to use, and can achieve unbiased estimation according to our theory. For completeness, we document the automatic annotation pipeline and more related research about PRM in Appendix A.
MDP RL for LLMs. Although the outcome reward model has advanced LLMS by applying re- inforcement learning algorithms in bandit settings, it contradicts the auto-regressive nature of text generation and the step-by-step reasoning process. Recent studies (Rafailov et al., 2024a; Zhong et al., 2024; Xie et al., 2024; Zeng et al., 2024) introduced theoretically sound RL algorithms de- signed for LLMs in MDP settings. Although these efforts bridge the theoretical discrepancy in algorithms, they still rely, least partially, underexplored in MDP-based RL. for LLMS.. Orthogonal to our exploration, several works (Lu et al., 2024b; Lai et al., 2024; Chen et al.; Zhang et al., 2024b) adapt DPO (Rafailov et al., 2024b) to step-level preference optimization for reasoning tasks. We discuss the potential of integrating such methods into our framework in Appendix D.
ORMs. Hence, the process reward model remains
6 CONCLUSION
In this paper, we introduce the Process Q-value Model (PQM), a new approach model process re- wards via optimization Q-value ranking. Unlike existing classification- based methods, which treat intermediate steps independently, PQM captures the interdependencies among steps. To effectively optimize the Q-value rankings, we propose margin-based comparative training objective and vali- date its effectiveness through comprehensive experiments. Our results demonstrate that PQM signif- icantly outperforms previous baselines, achieving an 1 11.6% accuracy improvement when verifying solutions generated by --0B-----7B-Intttttmt
on the MATH500 dataset, and consistently deliv- ering robust results across various backbone scales, policy models, and datasets. We hope our work inspires more future investigation on process reward modeling that better captures the complexities of multi-step reasoning processes.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
We thank Leitian Tao and the ICLR reviewers for their valuable suggestions on the paper.
REFERENCES
AI@ Meta. Llama model card. 2024. URL https://github.. wwwp:/thh....oomtttaattaaam toommmmeaallama//// 1aa.aaaaaaaaaaa/a/ 11ama3/blob/ main/MODEL_CARE CORODDELLLAADmCCC md.
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Giani- nazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyezyk, and Torsten Hoe- fler. Graph of thoughts: Solving elaborate problems with large language models. In Thirty-Eighth AAAI Conference on Artificial Intelligence, pp. 17682-17690, 2024a.
Maciej Besta, Florim Memedi, Zhenyu Zhang, Robert Gerstenberger, Nils Blach, Piotr Nyczyk, Marcin Copik, Grzegorz Kwasniewski, Jürgen Müller, Lukas Gianinazzi, Ales Kubicek, Hubert Niewiadomski, Onur Mutlu, and Torsten Hoefler. Topologies of reasoning: Demystifying chains, trees, and graphs thoughts. arXiv preprint XXXx:24401.14295, 2024b.
Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan. Step-level value preference optimization for mathematical reasoning. arXiv preprint arXiv:2406. 108588
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. Trans. Mach. Learn. Res., 2023, 2023.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint XXv::2100.1116. 2021.
Jianqing Fan, Zhaoran Wang, Yuchen Xie, and Zhuoran Yang. A theoretical analysis of deep q- learning. In Alexandre M. Bayen, Ali Jadbabaie, George J. Pappas, Pablo A. Parrilo, Benjamin Recht, Claire J. Tomlin, and Melanie N. Zeilinger (eds.), Proceedings of the 2nd Annual Confer- ence on Learning for Dynamics and Control, LADC 2020, Online Event, Berkeley, CA, USA, 11- 12 June 2020, volume 120 of Proceedings of Machine Learning Research, pp. 486 489. PMLR, 2020.
Shibo Hao, Yi Gu, Haotian Luo, Tianyang Liu, Xiyan Shao, Xinyuan Wang, Shuhua Xie, Haodi Ma, Adithya Samavedhi, Qiyue Gao, et al. Llm reasoners: New evaluation, library, and analysis of step-by-step reasoning with large language models. ICLR 2024 Workshop on Large Language Model Agents, 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021.
Shawn Im and Yixuan Li. Understanding the learning dynamics of alignment with human feedback. In International Conference on Machine Learning, 2024.
Maxim Khanov, Jirayu Burapacheep. and Yixuan Li. Args: Alignment ceward-guides search. In Proceedings of the International Conference on Learning Representations, 2024.
Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. Step-dpo: Step- wise preference optimization for long-chain reasoning of llms. arXiv preprint arXiv:2406.1861 2024.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. RLAIF: scaling reinforcement learning from human feedback with AI feedback. arXiv preprint 20X00.2309.00267, 2023.

---

Published as a conference paper at ICLR 2025
Chengpeng Li, Zheng Yuan, Hongyi Yuan, Guanting Dong, Keming Lu, Jiancan Wu, Chuanqi Tan, Xiang Wang, and Chang Zhou. Query and response augmentation cannot help math reasoning generalization. arXiv preprint arXiv:2310.05506, 2023a
Qintong Li, Leyang Cui, Xueliang Zhao, Lingpeng Kong, and Wei Bi. Gsm-plus: A comprehensive benchmark for evaluating the robustness of Iims as mathematical problem solvers. arXiv preprint arXiv:2402. 19225, 2024.
Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. Making language models better reasoners with step-aware verifier. In Anna Rogers, Jordan L. Boyd- Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACL 2023, Toronto, Canada, July 9- 14, 2023, pp. 5315-5333. Association for Computational Linguistics, 2023b. doi: 10.18653/V1/ A023.ACI URL wwtthhts///oo.... /ttp:::////oo.//////// org/10. wtt.g//....11//1//2222 18653/v1/2923 acl-long. 291.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2024.
Jianqiao Lu, Zhiyang Dou, Hongru Wang, Zeyu Cao, Jianbo Dai, Yingjia Wan, Yinya Huang, and Zhijiang Guo. Autocv: Empowering reasoning with automated process labeling via confidence variation. arXiv preprint vrXiv:2405.16800 2024a.
Zimu Lu, Aojun Zhou, Ke Wang, Houxing Ren, Weikang Shi, Junting Pan, Mingjie Zhan, and Hong- sheng Li. Step-controlled dpo: Leveraging stepwise error for enhanced mathematical reasoning. arXiv preprint arXiv:2407.00782, 2024b.
R Duncan Luce. Individual choice behavior, volume 4. Wiley New York, 1959.
Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, and Abhinav Rastogi. Improve mathematical reasoning in language models by automated process supervision. arXiv preprint 2xXiv:2406..6552. 2024.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou. Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. CoRR, abs/1312.5602, 2013.
Andrew Y Ng. Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In International Conference on Machine Learning, 1999.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Robin L Plackett. The analysis permutations. Journal of the Royal Statistical Society Series C:
Applied Statistics, 24(2): 1975.
Rafael Rafailov, Jocy Hejna, Ryan Park, and Chelsea Finn. From to q Your language model is secretly a q-function. arXiv preprint arXiv:2404.12355 2024a.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2024b.
Amrith Setlur, Saurabh Garg, Xinyang Geng, Naman Garg, Virginia Smith, and Aviral Kumar. RI on incorrect synthetic data scales the efficiency of Ilm math reasoning by eight-fold. arXiv preprint arXiv:2406. /4532, 2024.
Zhihong Shao. Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu. and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03330 03300, 2024.
12

---

Published as a conference paper at ICLR 2025
Jonathan Uesato, Nate Kushman, Ramana Kumar, H. Francis Song, Noah Y. Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process- 2022.
and outcome-based feedback. CoRR,
Chaojie Wang, Yanchen Deng, Zhiyi Lv, Shuicheng Yan, and An Bo. Q Improving multi-step reasoning for Ilms with deliberative planning. arXiv preprint arXiv:2406. 14283, 2024.
Peiyi Wang. Lei Li, Zhihong Shao, R. X. Xu, Damai Dai, Yifei Li, Deli Chen, Y. Wu, and Zhifang Sui. Math-shepherd. Verify and reinforce Ilms step-by-step without human annotations. arXiv preprint arXiv:2312.08933 2023a.
Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoe V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain thought reasoning in language models. In The International Conference on Learning Representations, 2023b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-though prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 24837, 2022.
Tengyang Xie, Dylan Foster, Akshay Krishnamurthy, Corby Rosset, Ahmed Awadallah, and Alexander Rakhlin. Exploratory preference optimization: Harnessing implicit qf-approximation for sample-efficient rlhf. arXiv preprint arXiv:2405.21044 2024.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671 2024.
Shunyu Yao. Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances in Neural Information Processing Systems, 2023.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu. Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. In International Conference Learning Representations, 2024.
Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen. Ruobing Xie, Yankai Lin, Zhenghao Liu, Bowen Zhou, Hao Peng. Zhiyuan Liu, and Maosong Sun. Advancing LLM reasoning generalists with preference trees. arXiv preprint arXiv:2404.02008,, 2024.
Yongeheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang, Haifeng Zhang, and Jun Wang. Token-level direct preference optimization. arXiv preprint arXiv:2404.1 11999, 2024.
Dan Zhang, Sining Zhoubian, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mets*; Llm self- training via process reward guided tree search. arXiv preprint arXiv:2406.03816 2024a.
Xuan Zhang, Chao Du, Tianyu Pang, Qian Liu, Wei Gao, and Min Lin. Chain of preference op- timization: Improving thain-of-thouuht reasoning in llms. arXiv preprint arXiv:2406.091166 2024b.
Han Zhong, Guhao Feng, Wei Xiong. Li Zhao, Di He, Jiang Bian, and Liwei Wang. Dpo meets ppo: Reinforced token optimization for rlhf. arXiv preprint arXiv:2404. 18922, 2024.
13

---

Published as a conference paper at ICLR 2025
A RELATED WORKS
Several techniques have been developed to accelerate the data collection pipeline for training PRMs (Luo et al., 2024; Lu et al., 2024a). To simplify understanding, first introduce the fundamental version proposed in Wang et al. (2023a). In this approach, the quality of an intermediate step is evaluated based on its potential to lead the correct final answer. The pipeline can be summarized as follows:
For given question x p. several trajectories are sampled by LLM: T1, TN Ti (-1æz). Each trajectory т {a1, {a1,a2,... ....... aæ J consists of a sequence of steps, and the correctness of these steps annotated through the following procedure.
For a trajectory = {a1, ....... aH) we generate n completions for each step from a₁ to an Specifically, to annotate ag sample completions by T2(- a1:4). The correctness of each completion evaluated by final answer string matching.
For each step 0g, if any completion of it achieves the correct final answer. We regard as correct, otherwise wrong. If a, is wrong, the subsequent steps d.+1;"* Uh are all regarded as incorrect.
There have been several research trying to promote the pipeline efficiency. For example, Lu et al. (2024a) trains additional confidence module to simplify the automatic annotations, Luo et al. (2024) performs a binary search to identify the first error location.
B IMPLEMENTATION DETAILS
All training is conducted on NVIDIA A100-SXM4-80GB GPUs. We list the ver- sions the important external packages as follows: corch==2.3.1, tr1==0. 8.0. 0, flashattn=22.6.2 transformers==4.00 34.0, accelerate==0. 33.0. d 13.1 wwidia-nccl-cu12=22 20. We use the ZeRO-3 optimization stage of the deepspeed with 6 precision. The hyperparameters for the ablation studies are provided in Table 5, and each training session for the ablation study took approximately 4.5 hours. For the main experiments, some training data has tokenized sequences longer than 2048 tokens, which limited the batch size and reduced training efficiency. To address this, we divide the training corpus into three groups based on tokenized length: sequences shorter than 512 tokens, between 512 and 1024 tokens, and greater than 1024 tokens. The batch sizes were set to 64, 24, and 8, respectively, for these groups. This strategy reduced the training time from about eleven hours to six hours. To generate the trajectories for Best-of-n sampling, we use the VLLM pipeline with the temperature set to 1, top-p set to 1, and max length set to 2048. For the MCTS baseline, we the policy model as Swen2-math-78 Instrrcct, and utilize iterative MCTS search train PRM. For fair comparison, we use half of the MathhSShphher corpus and its bard-estimated labels to construct Dy, (refer to the original paper (Zhang et al., 2024a)), and train initial PRM. Then conduct an MCTS search on questions of the remaining corpus. To keep the scale of the training set the same, we randomly sample trajectories with the quantity of 1/2 Math-Shepherd from the MCTS tree.
hyper- hyper-parannn parameter
value
scheduler
cosine
warm-up ratio learning rate optimizer
0.1
2c-6
Adam W
batch size per GPU gradient accumulation steps
64
4
gradient checkpointing
True
Table 5: Experimental settings for ablation studies.
14

---

Published as a conference paper at ICLR 2025
MetaMatthMiittrll-B
MuggleMath- 13B
Llama-3-70B-Insse Instrrutt
Methods
@8
@
16
@32
@
664
@
128
@
@ 16
@332 @64
@ 128
@8
@
@16
@:3
@64
@ 118
C. 8
36.4
40.2
41.2
43.8
44.6
30.8
33.8
37.2
38.8
38.8
47.0
47.0
47.8
46.2
46.0
L. 4
36.8
40.6
41.8
44.4
44.6
32.0
33.6
36.8
38.4
37.4
47.4
47.0
45.6
47.8
48.2
L. 2
35.8
39.0
40.8
43.4
43.8
30.2
32.8
34.2
36.8
37.4
47.4
49.0
50.6
51.2
50.4
Cablate ç
34.4
37.4
39.6
42.0
41.0
31.2
34.8
36.8
38.4
37.6
47.6
49.0
50.4
52.0
Cablate Ç
33.0
37.6
40.0
41.6
40.8
30.0
34.4
36.4
39.0
38.6
47.6
49.4
52.4
Cablate Ç 31.6
34.8
37.0
40.0
38.4
30.4
33.4
32.6
35.6
35.2
44.4
45,4
45.0
47.0
Lablates Ç 31.6
34.8
37.0
40.0
38.4
30.4
33.4
32.6
35.6
35.2
44.4
45.4
45.0
47.0
46.0
Table 6: Ablation results. The BON@ of MATH500 for SeetaMath-Mistra--- is 24.4, for Llama-
3-70B-Instruut is 37.4. L, Lablate refers Eq. and Eq. respectively. The detailed hyperparam- eters for experiments of this table are shown Appendix B.
www
10
8
theory-16
loss
the
6
theory-4
theory-2
theory-0
2
50
100
150
200 steps
250
300
350
400
Figure 5: The loss curves for ablation studies in Table 3.
C ADDITIONAL EXPERIMENTS
Loss variation. Here, we explore what if we only emphasize the first incorrect step in the ranking. The loss variant is as follows,
C
exp(2c)
Cablass
log
(12)
|C|
Ex= exp 2a, + exp( +
which promotes Qh Q3 2: < < 2 As shown in Table 6, focusing only
the first negative step, which is verified by automatic annotation, the performance remains relatively stable, suggesting the limited utility of subsequent negative steps.
Comparison with ceiling performance. We evaluate the ceil- ing performance of various policy models and compare how PQM stands against this benchmark. Figure 6 presents the Pass @ N met- ric alongside the best achievable verification performance for three distinct policy models. This comparison illustrates the upper limits of verification accuracy for each policy model and highlights the existing performance gaps. Specifically, the comparison suggests that current PRMs, including PQM, have not yet reached their full potential. These findings underscore the need for further advance- ments and refinements PRM techniques close the gap and ap- proach the ceiling performance more closely.
Pax-J
Pass-3
Pass
Pass-2
80
80
87
Figure 6: The ceiling perfor- mance and the hest verification performance of three policy mod- els MATH500.
Empirical validation for Assumption 3.1 and Theorem 3.5. To empirically validate our Theorem 3.5, use Ll-0O- Instruct to substitute the optimal model n". We sample 256 tra- jectories from Saah-Step-DPO-100 (Lai et al., 2024), comprising 128 correct and 128 incorrect trajectories respectively. Each tra- jectory consists of more than six steps. If the reasoning state is included in rejected answer,
15

---

Published as a conference paper at ICLR 2025
MetaMath- SotaMath-Mistral-T
MuggleMath- 13B
Llama-3-70B-
1emma---7B-Instruue Instruct
Methods Data Size
@
@16
@32
@64
@
128
@
@
@1
@32
@64
@ @28
@8
@16
@32
@64
@ 128
25%
19.6
21.0
18.2
19.0
17.8
17.6
16.8
15.8
15.2
13.8
37.2
35.6
34.2
34.6
30.0
50%
23.6
24.2
22.8
22.4
19.8
17.2
17.8
17.0
14.2
13.0
37.6
35.4
32.6
31.8
29.0
BCE
75%
32.4
31.8
34.0
34.6
33.6
28.4
28.4
31.0
31.6
31.6
40.6
38.8
37.0
38.4
38.8
100%
33.6
37.0
39.2
40.8
42.0
30.4
31.4
33.4
36.4
37.0
43.6
41.4
41.6
42.4
39,8
25%
21.4
21.6
19.8
19.8
19.2
18.0
15.4
17.0
14.8
14.0
37.4
36.6
37.2
38.4
35.6
50%
21.0
22.0
20.2
20.2
19.4
18.6
16.8
16.6
14.0
14.2
37.4
36,4
34,4
34.2
32.6
PQM
75%
33.4
36.4
37.0
39.6
38.0
29.2
32.4
35.0
37.2
37.4
46.8
47.8
47.0
47.2
46.0
100%
36.2
38.2
41.0
44.2
44.6
30.0
34.8
36.2
39.2
39.0
47.2
48.2
50.0
46.0
47.8
Table 9: The Best-of-n performance of PRMs trained on different data size. The comparisons are
conducted on dlasificationn-asee PRM (BCE loss) and our PQM. The BON@ 1 of MATH500 for SoiuMath--iitra-7777 is 24.4, for MuggleMath-1311 is 18.4, for Llama-3-70B-Instru is 37.4.
regard this reasoning state incorrect. For each reasoning state 01d in each trajectory, we sample 32 completions with x² (Ais). The correctness of next-step a;+1 is annotated automatically as in Wang et al. (2023a) with Qwen2-Math- Instttt-sttrut--7B.. We use statisties after the fifth step to avoid wwen2-Math-Instrueee
having a larger possibilityy to self-correct the step, hence misleading the label. We also count the correctness of each whole trajectory to approximate 2. for a as defined in Eq. 2. In Fig. 4, we count the correctness proportionality of correct completions according to the position of the reasoning state 01:i- According to the left subgraph of Fig. 4, the approximated Q. ascends with the continuation of the correct steps. The right subgraph illustrates that the latter wrong steps generally have smaller Q-values than the previous wrong steps. Moreover, there is a no- ticeable discrepancy between the Q-value of correct steps with 2. generally over 0.5 and incorrect steps with 2, generally below 0.15.
PRM-guided beam search. To further validate the effectiveness of our PQM, we have conducted additional experiments on PRM-guided beam search. The com- parison is conducted between PQM and classification-basee PRMs with BCE loss. We set the beam size as 8, and the gener- ative temperature 0.7. The evaluation is conducted on MATH500 across two policy models, LJama-3-8B-Instruu 2024) and Eurus-7b-sft (Yuan et al., 2024). The results are reported in Tabel which demonstrate that PQM can more effectively guide the LLM to reason. For ablations, we compare the performance of PQMs trained with Lihcorem and different Ç values as $4.3. We use Eurus-7b-sft as the policy model. The results in Table 8 align with the findings from the Best-of-N experiments, showing that a sufficiently large range of C leads to strong performance in PRM-guided beam search, with optimal values typically falling in the middle of the range.
Policy Models
Pass@1
BCE
PQM
1lama-3.1-4 1--88-B-nnstrce Burus-7b-sft
17.2
26.4
31.6
19.4
24.2
29.2
Table 7: The performance of PRM-guided beam search on MATH500.
Objective
(=1
(=2
(= 4
(= 8
(= 16
(AI@N) Meta,
C Lihoorrm
26.4
27.8
28.8
28.4
25.6
24.8
26.0
28.0
28.2
26.6
Table 8:
Ablation Studies of PQM-guided beam search on MATH500. The sample policy is Eurus- 7b-sft.
Sample-efficincyy of PQM. To examine whether PQM robustly outperforms classification-bart PRM across different dataset sizes, we randomly sample 25%, 50%, 75% of the original dataset to train PRMs with BCE loss and PQM loss. We keep most of the hyperparameters as in our main experiments, and set { as 4. As shown in Table 9, the results suggest that PQM generally outperforms BCE on all ranges of data sizes, and is more sample efficient.
Comparison of ranking behaviors between PQM and BCE. We first highlight behavioral dif- ferences based on the qualitative example in Table 4. (1) BCE produces probabilities that are monotonically decreasing for correct steps (step 1: 0.916 + step 2: 0.882 - step 3: 0.848). This behavior contradicts the desired property established in Theorem 3.5, which proves that values should increase (rather than decrease) for correct reasoning steps. (2) BCE does not produce a large transition in values between correct and incorrect steps. For example, in Table 4, the prob-
16

---

Published as a conference paper at ICLR 2025
ability only slightly decreases from 0.848 (step 3) 0.628 (step 4), failing to sharply differentiate between correct and incorrect steps. In contrast, our PQM framework produces Q- Q-values with a sig- nificant drop from correct to incorrect steps, better aligning with the desired behavior. For example, in Table 4, the Qo value drops substantially from 0,,82 to 0.004 between steps and 4.
Statistically, conduct an empirical study to confirm whether BCE and PQM result in different rankings on test steps. We calculate the proportion of solutions where classification-bss. and PQM produce the same rankings across steps. In the test set, only 29.18% of solutions shared the same rankings. Furthermore, when comparing rankings across different solutions for the same question (Best-of-N results), we observed that 0% of test questions had identical rankings. We also randomly sample 2048 cases from the training set. Statistically, classification- based PRMs and PQM yield different ranking behaviors on 62. 79% training cases. These statisties indicate a significant behavioral difference between BCE and PQM.
PRM
D INTER-SOLUTION COMPARISON
The comparison introduced in the main paper can termed as intra-solution comparison, since two compared reasoning steps within a single trajectory. This is partially because of the format of currently available corpora for PRM, which generally treats a single trajectory as a data point. Nevertheless, Theorem. 3.5 seamlessly apply comparison among different trajectories, i.e., inter-solution comparison. For instance, if two trajectories are diverged from (-th step with a com- mon correct prior 01:--1, the comparison will proceed between two different I-th steps. Here, we denote of is the correct one while at" is the wrong one. In this setting, we can derive the following corollary (note that represents the optimal Q-function Q" no ambiguity.
Corollary D.1 (Q-value ranking for inter-solution comparison). Formally, for two trajectories with the same correct prior 01:1-1 and of > ax the Q-value rankings among these steps are as follows, O, < Qo Q <... < Q.-1 < 2i, where Qo V(x).
There have been several offline step-level DPO methods (Lu et al., 2024b; Lai et al., 2024; Chen et al.; Zhang et al., 2024b) concurrent to our research. Though not focused on PRM, their theo- retical derivations can also be encompassed by the inter-solution comparison in Corollary D.I. Moreover, they (Lai et al., 2024) generally only utilize O < O and discard the ranking relation- ships among intermediate steps.
Corollary D.2 (Q-value ranking for inter-solution comparison (General Version)). Formally, for a trajectory T with successive step pairs, [[a "f) (as, ab). ..... (a a))),, where the 1-th step is conditioned previous correct steps, i.e. a₁ (-1aaf We have Qu ... < 2x
Q << 20 O <...< < < 24 where Qo = V(x).
Some seeel optimization are based on tree-structured data as in Corollary D.2, however, existing research (Chen et al.; Zhang et al., 2024b) generally considers each step pair independently, i.e. Or 2i, 2% Qi+, neglecting the integral Q-value rankings within the tree. We believe the above corollaries can shed some light on step-level offline optimization.
E CASE STUDIES
We show more qualatative examples in Table 10 and Table 11. We directly report Q* show the value transition. The predicted values generally have slight increase among correct steps. When encountering the wrong step, the predicted value will have a sharp decrease.
F LIMITATIONS & FUTURE WORKS
Notably, as shown in Luo et al. (2024), the data quality of Math-shepherd is restricted due to the noise introduced by automatic annotation. We believe that more advanced datasets could signifi- cantly improve the verification capabilities of PRMs. As shown in Appendix C. there remains a considerable gap between the current performance of PRMs and the ceiling performance, under- scoring the need for further advancements in PRM techniques. Additionally, while PRMs have garnered increasing research attention, there is lack of systematic studies corresponding online RL algorithms that leverage PRMs. The potential for fully utilizing PRMs in this context remains largely unexplored.
17

---

Published as a conference paper at ICLR 2025
Question: What the smallest positive perfect cube that can be written as the sum three consecutive integers? Correct Answer: 27
Step 1:
Do some research about cubes and about consecutive integers. The equation are looking for is in the form of x² = y + (y + 1) + (y + 2). (-1.703125)
Step 2: Add y (1++ (y+ 2), the sum of the 3 consecutive integers. integers.y((((())HH++++)3))3 (-0.97265625)
Step 3: Make the equations equal to each other.. = 3y + + (-1.0225)
Step 4: Subtract out of both sides.
3y (-0.796875)
Step 5: Divide both sides by 3. 3 = y (-0.79296875)
Step 6: Since y has to be integer, look for perfect cube that, when subtracted by 3, it results
in a multiple of 3.Our smallest option is to try 1, but it doesn't 't work. next option is to try 2, but it doesn't work. Our next option is to try 3, but it doesn't work. Our next option is to try 4, which results in 61.Our next option is to try 5, which results in 106. (-3.234375)
Step 7: Check which one works. 61 isn't multiple of 3, but 106 is. Final Answer: The final answer is 125. I hope it is correct. (-5.03125)
Step 1: Write an expression for three consecutive integers. Call them - 1,0, and 1. (-0.53515625)
Step 2: Write expression for their sum. The sum of the three is 1 + a + (-0.302734375)
3a.
Step 3: We want this sum to be a perfect cube, let 3a n Divide both sides by 3 to get
a = 3. (-0.265625)
=²
Step 4: Since want be an integer, it must be that is multiple of 3. Therefore,
n" have n = 3k, where is an integer. Substitute this back into a 3 (-0.33203125)
to get = (3A) 3²
Step 5: From here, we just guess and check for values of & to find our answer. When k = 1, then a = 32 (1)3 3 = and
9, and a+ 10. The sum of these three consecutive integers is 8 + 9 + 10 = 27 = 33. Therefore, the smallest positive perfect cube that can be written as the sum of three consecutive integers is 27 Final Answer: The final answer is 27. I hope is correct. (-1.6015625)
Step 1: Let n. n + 1, + 2 be three consecutive integers, such that their sum is equal to k.,, perfect cube, In other words, n + (n + 1) + (n + 2) 3.3 (-0.734375)
Step 2: Rearranging the terms in the equation, we obtain: 3n + = 1-3, (-0.255859375)
Step 3: Substracting 3 from both sides, we get: 3n = k3 3. (+0.177734375)
Step 4: Dividing both sides by 3, we obtain:
-² (-0.376953125)
Step 5: Since n must be an integer, then k3 3 must be divisible by 3. (-0.451171875)
Step 6: Adding to both sides, we get: 13 1-3 - + 3, A.3 = 13 3 = (k3 -3) + 3.
(-2.3125)
Step 7: The smallest perfect cube that is 3 greater than a multiple is K3 = 64. (-5.46875)
Step 8: Plugging in 1-3 64 to our equation in Step 4, we get: n 61-3 (-5.71875)
61
20 + 3. 3
Step 9: Since n must be an integer, then we must increase 1-3 to k 125 to get an integer
value of n. (-5.0625)
Step 10: Plugging in k3 125 to our equation in Step 4, we get: 125-3
122
40 + 3 (-5.34375)
Step 11: Since n must be an integer, then we must increase 3.3 to k = 216 to get an integer
value of n. (-4.625)
Step 12: Plugging in 13 216 our equation in Step 4, we get: n 216-3
213
71. (-
3
3
4.0625)
Step 13: We found an integer value of n, which is our goal. We will not need to increase 3 any further. (-1.8359375)
Step 14: Plugging in n = 71 to our initial equation, we get: 71+72+73 = 13 216 = 3-3 A3 216. Final Answer: The smallest positive perfect cube that can be written as the sum of three consecutive integers is 216. I hope it is correct. (-4.84375)
Table 10: case of MATH500 sampled by --lmm-3-7700 7-00------7tcccc The red value represents the predicted Q" provided by PQM.
18

---

Published as a conference paper at ICLR 2025
Question: Find all values of x that satisfy the equation x = ~VII 2x + 4. Correct Answer:
Step I: Subtract 4 on both sides of the equation to isolate the radical.æ 4 vII 2x (-0.349609375)
Step 2: Square both sides to get rid of the radical. (x: 4) 11 2x (-0.133535156) (6655516625)
Step 3: Expand, combine like terms, and rewrite the equation in standard form (leading
coefficient of 1) have quadratic in r. x² 10x 21 0,*²
3x 21 0,
<(r 7) 3(x - 7) = 0,(x - (z 3) 0 (- (-1.28125)
Step 4: Factor out the solutions the equation. 1.3984375)
0 or x 0.r x (-
Step Check the solutions in the original equation to see if they satisfy the equation. When 7.7 I11
14 4,7 V No real solution for
14
7. When = 3,3
/ VII 4,3
5
+ 4,3 4- = v5. No real solution for = either. Final
Answer: The final answer is there is solution. I hope it is correct. (-3.640625)
Step 1: Isolate the radical æ - = /I1 2r. (-0.251953125)
Step 2: Square both sides (x - 4)3 (V11 - -xx) (000000803322666655)))
Step 3: Simplify x² 8r + 16 = 11 2r. (+.0022216796775))
Step 4: Write in standard form x²2 - 6z 5 0. (0.0849609375)
Step 5: Factor (x (z 1) 0.. (0.1630859375)
Step 6: Solve for x, x = 1,5 (-0.44140625)
Step 7: Check solutions: x is extraneous. Final Answer: The final answer is 5. I hope it is
correct. (-1.40625)
Step 1: We want first isolate the square root on the right hand side. Subtract from both sides to get x - 4 /11 - 2r. (0.0390625)
Step 2: Now we square both sides to get(x - 4)2 = 11 2x. (0.09521484375)
Step 3: Multiply out to getz? 8z +
11 2z. (2.227735996
Step 4: Move all terms to the left and combine like terms to getr.² 10z + 0.(-4377)
Step 5: We now factor the equation to get (I - 5) - 1) = 0.(-4.5)
Step 6: And we solve for by setting each factor equal to 0: x 1 5. Final Answer:
The final answer is
or x 5 hope it is correct. (-6.5)
Step 1: Subtract from both sides of the equation. 4 = VI1 2r (-0.296875)
Step Square both sides of the equation. - 4) 11 2x (-0.0341796875))
Step 3: Expand and simplify both sides of the equation. 8r 16 = 11611-- 2x.x
8x +
2r 16 + 11 0,z² 6r
0.(x - (x + 1) = 0 (-4.875)
Step 4: Solve for both values of x.r - 5 = 0.z
r + = 0.r -1 (-5.5)
Step 5: Check both answers in the original equation to ensure they are valid.Final Answer: The final answers 5 and hope it is correct. (-6.375)
Table 11: Case studies on MATH500. Solutions are sampled from Llama-3-77 70B-Instruct. The red value represents the predicted Q" provided by PQM.
19


Reviews:
[{"review_id": "V2gzLZ0Ekr", "reviewer": "Reviewer_SdVZ", "summary": "The paper introduces an algorithm to train a process verifier using a Q-value ranking objective. In particular, they split the intermediate steps in a generated trajectory into steps with high and low Q-values and optimize a ranking loss with margins. The authors empirically show that the learned PQM is better for best-of-N, compared to prior works like Wang et. al., Lightman et al., that use a classification based loss to train the Q-function.", "questions": "- What is the precise definition of a correct or incorrect step? Is it based on the Q-value? \n- Equation 3 and 4 are unclear, and in general the notation $\\bar{a_{1:t}}$ is confusing. For example, since $\\tau$ in L222 is composed of $x, a_1, \\ldots, a_H$, shouldn't $P(\\tau | \\bar{a_{1:m}})$ be $0$ in Equation 3? \n- In the objective in Eq. 9, how is $Q_{w_t}$ computed, i.e., what is the input to the network? Is it only the question and action, or the whole prefix up until the action $w_t$?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The proposed ranking loss for training PRMs empirically improves best-of-N on MATH500 dataset, for multiple base LLMs. In particular, it outperforms prior work Wang et. al., that trains the PRM with a BCE loss.\n- The analysis in Section 4.3 is insightful and shows that using a margin based ranking (ablating on $\\zeta$) improves best-of-N performance when using the PQMs as an ORM (taking the min score over individual steps).", "weaknesses": "- The trained verifier is only used for best-of-N which is not its most promising use case. Evaluating its efficacy for beam-search where the PQM ranks intermediate generations is needed to demonstrate why practitioners should train PQMs. \n- Training with the binary cross-entropy loss, where the labels for each prefix are the expected future reward (some value between 0 and 1), will also distinguish prefixes across problems, and maybe the difference in expected rewards for prefixes can be accentuated with an appropriate scaling factor on the expected reward. In theory, it is unclear why the model trained with ranking loss should lead to a more generalizable solution that the one trained with this version of the classification loss. Essentially, it is unclear why the ranking loss should lead to a solution different from the one trained by Wang et al., when the model is trained on the population objective. I believe that the empirical results suggest otherwise, but this observation lacks formal or intuitive justification.\n- It is unclear if BCE and $Q_\\sigma$ models induce very different rankings on the Q-values of the steps. From the qualitative example in Table 4, it seems that they do not differ by a lot. An empirical study is needed to understand if they result in very different rankings on the steps in test examples. And if the rankings do not differ by a lot, then some post-hoc calibration of the BCE solution should also perform as well as PQM. And if they do differ in rankings, then it is more interesting, and important to study why this happens.\n- Are PQMs more sample efficient than BCE? Some analysis on training PQMs with different dataset sizes would help clarify. Also, what is the best data collection strategy to train PQMs? Is it just IID samples from the base LLM? Should we sample more solutions per question, or focus on improving coverage over both questions and solutions. Basically, if we have a fixed sampling budget, how should we collect data to train PQMs?\n- In a number of places, the objective optimized or algorithm used is unclear (see Questions).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UaZbTHgTgj", "reviewer": "Reviewer_oPLj", "summary": "The paper proposes Process Q-Value model (PQM) a framework which uses Q-value instead of immediate rewards in the Process Reward Modeling (PRM).\n\nUnlike PRM, the proposed framework allows to model inter-dependencies among reasoning states. Moreover, the authors show that the existing PRM framework can be cast as a special case of PQM.\n\nThe authors provide an extensive empirical study of the proposed method and demonstrate the clear benefit of the proposed method.", "questions": "Why is the gap in performance between PQM and SC+PQM increases as we move to the right in Figure 2?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall, the paper is overall well written (see one remark in Weaknesses) and easy to follow.\n\nThe paper extends the existing PRM framework to a more general PQM framework which uses Q-values instead of intermediate rewards, which allows to capture the dependency between reasoning states (rather than having these states to be independent). This extension is natural and well motivated.\n\nThe paper provides empirical study highlighting the effectiveness of the proposed method. Moreover, the paper provides ablations for the introduced hyperparameters. For example, Table 3 suggests U-shape behavior of the $\\zeta$ parameter with good values being in the middle. Overall, it seems that there is a sufficiently large range for this parameter which leads to good performance.", "weaknesses": "A suggestion to slightly improve presentation. In Section 3.3, it would be helpful to outline the overall objective for the theorem 3.5 (what do we want to prove any why), and then outline the plan for this proof (why we need other lemmas).\n\nFrom the results, it is unclear why (Figure 2) the gap in performance between SC+PQM and PQM is larger as we go to the right. It would be helpful if the authors add explanations of why they believe it is happening.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zEtkH4ihAz", "reviewer": "Reviewer_QeSG", "summary": "In the presented work, the authors introduce a novel approach to process reward modeling that focuses on the contribution of intermediate reasoning steps and their respective contributions to the final methods. In contrast to existing PRM approaches, the authors introduce a Q-value base method (Process Q-value Model) that is better suited for the problem by using a ranking approach, thereby capturing inter-dependencies among intermediate reasoning steps. The problem addressed by the authors is relevant to the current field, particularly for tasks requiring multi-step reasoning, like MATH500, and is thereby timely and demonstrates significant performance improvements over prior work.", "questions": "- The best results on MATH 500 are achieved with MetaMath-Mistral-7B when using zeta = 4, while for LLaMA-3-70B-instruct, zeta = 2 yields the highest performance. Interestingly, this relationship is reversed for the GSM-Plus benchmark. Is there some intuition behind why the relative importance of intermediate steps has such a model-dependent influence? This trend is also visible in Table 3.\n- Similarly, is the zeta value of 2 required for larger models in general, or is this just a quirk of llama-3?\n- Intuitively, PRMs interpret all subsequent steps after the first mistake as wrong. How does the proposed method handle these situations?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- Sections 2 and 3 are well written and provide sufficient context of the field as well as the proposed method\n- The experiments are comprehensive and, without a doubt, the proposed method achieves high performance across multiple LLM backends and math benchmarks, making it a strong contribution.", "weaknesses": "- Section 3 could benefit from some intuition. While the content seems sound, some higher-level guidance could be beneficial (this is just a minor issue)\n- Experiments on self-consistency could benefit from some additional explanations. In particular, given figure-2 (right), why is self-consistency particularly important for larger models, given the clear difference between PQM and PQM+SC only in the 70B model? Providing additional insights here would be very useful. \n- I would recommend adding some bold numbers to Table 3 also (minor issue)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ewHJk18Jya", "reviewer": "Reviewer_qfhZ", "summary": "This paper proposes a new reward modeling framework for LLMs by using Q-value ranking to enhance the credit assignment of each state’s contribution to the outcome. Both theoretical and empirical results demonstrate the effectiveness of the proposed framework.", "questions": "- Are there cases where Q-values are inaccurately trained?\n- How does inaccuracy in Q-values impact performance?\n- It would be beneficial if the authors provided a discussion on inaccuracies in Q-values, as this could offer an opportunity to improve the proposed method.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- In general, the paper is well-written and the proposed framework is easy to follow and understand.\n- The theoretical results support the use of Q-value ranking, which is further validated by empirical results.\n- The relationship between the proposed framework and prior work (classification-based PRM) is compelling.\n- The experimental results are comprehensive, and the case study helps readers easily grasp the concepts.", "weaknesses": "It is hard to find any weaknesses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w36fcjtVaF", "reviewer": "Reviewer_1S6Q", "summary": "This paper introduces PQM as an improvement over existing PRM methods, particularly in tasks requiring complex reasoning and decision-making. Traditional PRMs, typically framed as classification problems using cross-entropy loss, evaluate each step independently, which can result in suboptimal reward allocation and fails to account for interdependencies between steps. PQM redefines PRM using a MDP framework, optimizing Q-value rankings through a novel comparative loss function that better captures the dynamics of sequential decision-making. The authors claim that PQM offers a more detailed and theoretically robust method for distributing rewards across a process. Empirical evaluations demonstrate that PQM outperforms classification-based PRMs across different language model backbones, sampling policies, and multi-step reasoning tasks. Additionally, ablation studies confirm the effectiveness of the comparative loss function. The code is made available for reproducibility.", "questions": "N/A", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "PQM presents clear and well-structured experimental results, providing strong empirical evidence of its effectiveness across various benchmarks. The thorough evaluations, including comparisons with classification-based PRMs and comprehensive ablation studies, effectively demonstrate the advantages of the proposed framework and its comparative loss function.", "weaknesses": "The paper lacks clarity regarding the necessity of the OSF framework. It does not sufficiently justify why this specific approach is needed to address the challenges in reinforcement learning, or how it fundamentally improves upon existing counterfactual explanation methods. The rationale behind why the OSF state is more effective than traditional approaches remains underexplained, making it difficult to assess its true impact.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)