Linking Process to Outcome: Conditional Reward Modeling for LLM Reasoning
Authors: Zheng Zhang
Ziwei Shan
Kaitao Song
Yexin Li
Kan Ren
Venue: ICLR 2026 Poster (2026)
Keywords: LLM
reasoning
process reward model
reinforcement learning
Primary area: foundation or frontier models, including LLMs

Abstract: Process Reward Models (PRMs) have emerged as a promising approach to enhance the reasoning capabilities of large language models (LLMs) by guiding their step-by-step reasoning toward a final answer.
However, existing PRMs either treat each reasoning step in isolation, failing to capture inter-step dependencies, or struggle to align process rewards with the final outcome. 
Consequently, the reward signal fails to respect temporal causality in sequential reasoning and faces ambiguous credit assignment.
These limitations make downstream models vulnerable to reward hacking and lead to suboptimal performance.
In this work, we propose Conditional Reward Modeling (CRM) that frames LLM reasoning as a temporal process leading to a correct answer.
The reward of each reasoning step is not only conditioned on the preceding steps but also explicitly linked to the final outcome of the reasoning trajectory. By enforcing conditional probability rules, our design captures the causal relationships among reasoning steps, with the link to the outcome allowing precise attribution of each intermediate step, thereby resolving credit assignment ambiguity.
Further, through this consistent probabilistic modeling, the rewards produced by CRM enable more reliable cross-sample comparison.
Experiments across Best-of-N sampling, beam search and reinforcement learning demonstrate that CRM consistently outperforms existing reward models, offering a principled framework for enhancing LLM reasoning.
In particular, CRM is more robust to reward hacking and delivers stable downstream improvements without relying on verifiable rewards derived from ground truth.

Published as a conference paper at ICLR 2026
LINKING PROCESS TO OUTCOME: CONDITIONAL RE- WARD MODELING FOR LLM REASONING
Zheng Zhang"?, Ziwei Shan', Kaitao Song', Yexin Li Kan Ren" School of Information Science and Technology, Shanghai Tech University State Key Laboratory of General Artificial Intelligence, BIGAI 'Independent Researcher
(zhangzheng2024 shanzw2022, ee liyexin@bigaii ai
stilikeeptry@out com
ABSTRACT
Process Reward Models (PRMs) have emerged as a promising approach to en- hance the reasoning capabilities of large language models (LLMs) by guiding their step-by-step reasoning toward a final answer. However, existing PRMs either treat each reasoning step in isolation, failing to capture inter-step depen- dencies, or struggle to align process rewards with the final outcome. Conse- quently, the reward signal fails respect temporal causality in sequential rea- soning and faces ambiguous credit assignment. These limitations make down- stream models vulnerable mance. In this work, we propose Conditional Reward Modeling (CRM) that frames L.LM reasoning as a temporal process leading to a correct answer. The reward of each reasoning step is not only conditioned on the preceding steps but also explicitly linked to the final outcome of the reasoning trajectory. By enforcing conditional probability rules, our design captures the causal relation- ships among reasoning steps, with the link to the outcome allowing precise at- tribution of each intermediate step, thereby resolving credit assignment ambi- guity. Further, through this consistent probabilistic modeling, the rewards pro- duced by CRM enable more reliable cross-sample comparison. Experiments across Best-of-N sampling, beam search and reinforcement learning demonstrate that CRM consistently outperforms existing reward models, offering a principled framework for enhancing LLM reasoning. In particular, CRM is more robust to reward hacking and delivers stable downstream improvements without rely- ing on verifiable rewards derived from ground truth. The project website is at Ittts://foundatii model research. github. 1o/CRM
reward hacking and lead to suboptimal perfor-
INTRODUCTION
Recent advances in enhancing reasoning abilities have greatly improved the performance of large language models (LLMs) (Snell et al. 2025 Jaech et al. (002)), where models derive final answers through explicit step-by-step reasoning. Beyond prompt-based approaches (Wei et al., 2022 Diao
et al. 2024 Fan et al.] 2025). state-of-the-ar systems like DeepSeek-R1 (Guo al. 2025a) have further advanced reasoning capacity through reinforcement learning (RL) with verifiable rewards.
However, verifiable rewards rely checking model outputs against ground-truth labels, whose acquisition is costly and difficult to scale for general reasoning improvements. Reward models provide promising alternative by extrapolating reward signals across general LLM reasoning pro- cesses. Broadly, these models fall into two categories: outcome reward models (ORMs) (Cobbe) al.) 2021]] Yu al.] 2024). , which provide feedback the final step; and process reward models (PRMs) (Wang et al. 2024 Li & Li 2025; Yuan et al. 2025), which provide finer meer-grained rewards the level of individual reasoning steps or even tokens.
"Corresponding Author.

---

Published as a conference paper at ICLR 2026
Normalized Reward Curves (Min-Max)
1.0
Nze Reward
0.5
Oulcume supecvision
0.6
PMM
Summation
PQM
CRM
VR
Chatn nile of probakliity
0.2
CRM
0.0
200
Tr Reward for t
PRRM
OlympiadBench
Depezdency
0.40
PQM
Local step- wise ranking
Accuncy
0.35
Conditional proiadillity
cormect strp given prevtous steps
reward hacking
PRM
0.20
Stop
Step
Step r
100
Training Steps
200
500
Figure 1: (a) Comparison of reward modeling paradigms. Our CRM explicitly conditions each step reward on the previous reasoning steps and aligns it with the final outcome. (b) RL training with different reward models. Our CRM is more robust to reward hacking and achieves performance on par with training using verifier rewards (VR).
Despite offering nuanced signals for reasoning processes, existing PRMs face several limitations. (i) Isolated step modeling: As shown Figure most PRMs (Lightman et al.) 2023 Wang et al.
2024
Luo et al., 2024 Shao al.) 2024) assess each reasoning step in isolation, neglecting the in-
trinsic sequential dependencies reasoning, (ii) Limited outcome awareness: While some methods
Yu et al. 2024 Li & 2025 Yuan et al. 2025) attempt mitigate isolated step rewards, they of-
ten fail to effectively link step-wise rewards with the outcome. For example, PQM (Li & Li 2025),
illustrated in Figure Tal relies relative comparisons between neighboring steps (e.g., greater or smaller reward), but lacks explicit modeling of the final outcome, which may result in biased pro- cess rewards. IPRM (Yuan al. 2025) parameterizes the outcome reward as the logarithmic sum of process rewards, yet it fails to capture how specific step relates to the final outcome and lacks nuanced modeling of inter-step dependencies. This leads ambiguous credit assignment from the final outcome back to the intermediate reasoning steps. As a result, existing methods are prone to reward hacking, where rewards continue to increase while the actual task accuracy declines, which we have observed in our experiments and has been shown in Figure Tb
To address these limitations, we propose Conditional Reward Modeling (CRM), which frames rea- soning temporal process through which LLM progressively approaches the correct final answer. In our formulation, the reward signal is modeled as the probabilistic evolution deriving the correct outcome conditioned on the existing reasoning steps. At each step. the process reward is treated as a conditional probability dependent on all the preceding steps, thereby capturing causal structure inherent in sequential reasoning. Furthermore, by explicitly linking each process reward to the final outcome via the conditional probability chain rule, CRR enables precise attribution of the final result individual reasoning steps, effectively resolving the credit assignment ambiguity prevalent in prior work. An additional advantage is that the probabilistically consistent formulation of process reward signals across reasoning trajectories facilitates cross-sample comparison, which significantly benefits downstream tasks such as Best-of-N N sampling, beam search, and RL optimiza- tion. As shown in Figure la CRM jointly models inter-step dependencies and the relationship between intermediate rewards and the final outcome.
Our contributions are three-fold as follows. (i) Conditional reward modeling framework: We intro- duce CRM, which defines each step's reward as conditional probability dependent on all preceding steps, thereby capturing inter-step dependencies. (ii) Precise credit assignment: By linking process rewards to the final outcome, CRM resolves the ambiguity of credit assignment in existing PRMs. (iii) Practical effectiveness and robustness: CRM enhances pross-sample comparison and improves various downstream tasks. Experiments demonstrate that our CRM achieves superior performance, remains robust to reward hacking and delivers stable reasoning improvements without reliance on a verifier that uses ground-truth labels.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
Enhancing the Reasoning Ability of LLMs. To enhance L.M reasoning, prior studies have ex- plored test-time search (c.g.. Best-of-N, beam search) with reward models to select promising solu- tions Xie ct al., 2023 Zhang et al. 2025a Snell et al. 2025), and post-training via RL (Jaech et al.) 2024 Guo al.J 2025a). Recent work (Zeng al. 2025b Guo et al. 2025a) has investigated RL with verifiable rewards for reasoning tasks. However, these approaches heavily rely on ground-truth and thus do not easily scale. Other studies (Cui al.
2025 Cheng et al. 2025) integrate reward
models into RL to provide dense feedback, but reward hacking remains a major concern (Gao et al., 2024
Process Reward Model. Most existing studies (Lightman et al.
2023: Wang et al. 2024 Luo
et al. 2024 Shao et al., 2024) treat PRM as step-level classification, which treats steps isolation and ignores inter-step dependencies. Recent studies (Lu et al.) 2024 Li & Li 2025 Yuan et al.
2025) attempt to move beyond classification-base PRMs. Li & Li (2025) reframes PRM as Q- value ranking problem to model the relationships between steps, but overlooks explicit modeling of the final outcome, resulting in a gap between process rewards and the final outcome. Yuaan et al. (2025) instead proposes parameterized formulation of outcome, leveraging outcome labels to train the PRM, but the relationship between intermediate step rewards and the final outcome remains unclear, and the lack of step dependency modeling leads to ambiguous credit assignment, Another line of work (Chen et al. 2025; Guo et al., 2025b She et al. 2025; Zhao et al. 2025) integrates reasoning capabilities into reward modeling, such R-PRM (She et al. 2025) and GenPRM (Zhao et al. 2025). which evaluate reasoning processes with detailed thinking. However, these approaches often involve complex training procedures and incur high inference costs.
3 METHODOLOGY
3.1 TASK DESCRIPTION
Given a question s, LLM æ generates a response y (a1, (2i,a2...... .....,,000 @T) where a, denotes the /-th reasoning step and T' is the total step number. Let ası denote the sequence of the first t steps. Each response assigned binary label E 10 indicating whether the final answer derived from is correct.
We model multi-step reasoning as finite-horizon Markov Decision Process (MDP) M (S, .. P, 7) driven by autoregressive LLM æ acting as the policy. S is the state space, A the action space, P represents transition dynamics, r : S x A + R the reward function, and [0, 1] is the discount factor. At reasoning step I, the state is defined 8, (I, aça- 1). which contains the question x and the sequence of previously generated reasoning steps dç(-1- The action a is reasoning step generated based on the state $t. The state transition is deterministic, as the next state is uniquely determined by concatenating the previous sequence with the current output a. r, denotes the reward provided by the environment or generated by the reward model for action de-
3.2 CRM MODELING
To remedy isolated step modeling and limited outcome awareness in existing PRMs, we model LLM reasoning temporal process in which the probability of reaching the correct answer evolves with step 1. However, in practice, it is difficult to directly quantify the extent to which reasoning is approaching the correct answer. Instead, we choose model the complementary event: the reasoning process entering wrong state, which implies that the reasoning trajectory can no longer yield the correct answer. Formally, we define z as the index of the first step at which the reasoning process enters such a wrong state, with 2 1. If no wrong state occurs throughout the trajectory, then x > T, meaning the reasoning is correct and the final answer correct
1). Conversely, the final answer is incorrect (1 = 0), then the trajectory has entered wrong state during reasoning. with = < T. Let p(2; x. asa)) denote the probability mass function of a wrong state occurring at step Accordingly, the probability that wrong state has already occurred at before step is
W(Ga, Pr(z 5) = Σ p(z; agx))
(1)
3

---

Published as a conference paper at ICLR 2026
The probability of maintaining correct reasoning up to step t is its complement,
a(t;
= 1)--))
p(z;z,ag)
(2)
s=t+1
The discrete probability mass function of the wrong state occurring at step t is p(t; x, ag.) = Pr(z = ((t; W - 1; agr-1)
(3)
For simplicity, we use ptt), (tt), (t) to denote p(t; x, agt), (tt: z agi) and S x, agi).
A reasoning trajectory is inherently causal: the correctness of step t logically depends all preced- ing (L-1) steps. Building on this view, we adopt a conditional probability perspective and derive h(t), the probability that step enters wrong state given that all previous (1-1) steps were correct.
Pr(z t) (t) Pr(z = d/2 2 Pr(z 2 () 5(1 1)
p(t)
Naturally, 1 1-h(t) h(t) corresponds to the complementary event: the probability that the current step is correct given that all previous (t 1) steps were correct. This formulation explicitly captures step dependencies through conditional probabilities, addressing the limitations of prior studies (Lightman
al. 2023 Wang et al. [0224] [uo et al., 2024) Shao et al.) (024) that overlook inter-step relations.
To explicitly link intermediate steps to the final outcome, we first apply the chain rule of probability to establish the relationships among 5(1), W(U ((). p(t), and l(t))
S(t) = Pr(z > t) = Pr(z # 1, # 2, .... 2 # t) = Pr(z # 1) Pr(z # 2/z # 1)
... ...,P(( # t/z # 1, # 2..., 2 # - II
[] Pr(z = k 2 [b] II (1 (()))
(5)
k=1
A=1
W(1) Pr(z
5(t)
II (1 n(k:)
(6)
k=1
1-1
p(t) = Pr(z =)
t) = M(t)
II h(k))
For the final step of the reasoning trajectory T, we have S(7) 11 h(t)) from Eq. where S(T) reflects the probability that the reasoning process reaches the correct final answer. We next seek dense, step-wise reward signal aligned with this outcome probability.
We apply Potential-Based Reward Shaping (PBRS) (Ng et al. 1999) to the task of multi-step rea- soning. PBRS proposes that dense reward function He can be constructed from an original sparse reward function R by adding a shaping term derived from a potential function (ss) defined over the state space. For a transition from state St to St+1, the shaped reward is given by:
R' (st, de, 81+1) R(s,, at) + (+++1) - +ss))
PBRS guarantees that any policy optimal under the shaped reward R'(st. ae, S141) is also optimal under the original reward R ((12, ot). Thus, using the shaped reward guides the learning process more effectively without changing the fundamental goal of the task. The key to applying PBRS lies in selecting
appropriate potential function (ss,) which is effective when it encodes goal- directed progress (Lo et al., 2024 Müller & Kudenko 2025 Mindom et al. [2021] Wiewiora et al.) 2003). We define the potential function log S(t), which denotes the log-likelihood of eventually reaching a correct answer from the current state, thereby explicitly capturing the progress toward the final correct answer. Consequently, substituting this definition of log S(t) into the potential function yields the following explicit form:
c
क (sx) log S(1) log
Σ log(1 h(k)))
(9)
k=1
We now apply the general PBRS formula (Eq.
g to derive the specific process reward re, which
corresponds to the transition from state 8t-1 St.
F: R' (8 at-1, 8t) R(s 00- + ( ((8--1) = -log11 h(t))
(10)
4

---

Published as a conference paper at ICLR 2026
where original reward R. 0 for all intermediate steps, and the discount factor is fixed at 1.
The above derivation yields a dense and accurate process reward, rt = log h(t)), which serves credit assignment scheme grounded in our modeling. The probability that the reasoning process reaches the correct final answer, S(T), satisfies S(T) II (1 h(t) II e" Through this decomposition, we explicitly link process rewards to the outcome, addressing the limitation of prior work (Yu al. 2024 Li & Li 2025] Yuan al. 2025) that lacks explicit modeling of the relationship between process and outcome.
3.3 TRAINING OF CONDITIONAL REWARD MODEL
Let D - {(a), yh: 4i, =:}} denote the training dataset, where z; is the i-th question, y4 is the corre- sponding response (multi-step reasoning trajectory), ( indicates whether the final answer is correct, and 2; is the index of the first wrong state. We use large language model fo with parameters to initialize the reward model. Since both S(T) and the process reward 7r are functions of h(t), we train the model to predict (t))) At the 1-th step of the reasoning trajectory, the model takes the question x and the first reasoning steps input, and predicts the h(t) =
agg))
For samples where the reasoning process reaches the correct final answer (4 = 1), we maximize S(T) and the corresponding loss Ls is:
Cs(zi, y:) log Pr(z; > 7) = log S(T) = - log II h(t))
(11)
For samples where the reasoning trajectory fails reach the correct final answer (4: 0), minimize S(T). Moreover, since the reasoning process enters wrong state at step we encourage the model to identify this step by maximizing p(z;), the probability of wrong state occurring exactly at step 2,. The loss Lw and L, for this case are as follows:
T
(a4, vi) log Pr(z; S T) log(1 - -(T)) log
1
II M(t))
(12)
21-1
C Uss 2;) log Pr(z,) log p(2;) log h(2))
II n(t))
(13)
Figure illustrates the roles of the three loss terms. The overall loss to be minimized is as follows:
C
1s Cs(ss,,, v1) -1-1)) (Cw vi) + La Ca Mi. xi)
(14)
D
This consistent probabilistic modeling and train- ing ensure that CRM is grounded in clear proba- bilistic semantics: for example, the value of S(t) any step across different samples consistently carries the same probabilistic meaning, allowing them be compared under the same scale. By contrast, previous approaches (Li & Li, 2025] Yuan al. 2025) struggle to achieve accurate cross-sample comparison (see Appendix for de- tailed analysis).
1.0
1.0
S(t)
Ls
0.5
0.5
p(t)
Lz
Lw
0.0
0.0
Z
Index of Reasoning Steps
Figure Effects of three loss terms.
EXPERIMENTS
4.1
EXPERIMENTAL SETTINGS
Reward Model Training. We train the reward model the Math-Shepherd dataset (Wang et al.
2024 which integrates questions from GSM8K (Cobbe et al., (021) and MATH (Hendrycks et al.
2021 and provides responses annotated with step-level process labels. Following prior research (Li
& 2025, Guan et al.) 2025), we augment the pre-trained model with value head. The reward models trained using full parameter fine-tuning. Further details can be found in Appendix

---

Published as a conference paper at ICLR 2026
Baselines and Evaluation. We compare our CRM against represttattee baselines, including ORM, vanilla PRM (Wang et al., 2024). PQM (Li & Li 2025) and IPRM (Yuan et al. 2025). For faimess, all baselines re-implemented within the same pipeline, backbone, and traming data. We then leverage the rewards provided by the reward model to (i) select an optimal response in Best-of-N (Section 4.2 (ii) guide beam search (Section 4.3) and (iii) optimize LLM reasoning through RI.
(Section 4.4
4.2 BEST-OF-N SAMPLING EXPERIMENTS
Best-of-N sampling generates N responses for a given question and selects the optimal one using reward model, evaluating the model's ability identify correct samples the trajectory level. Our CRM computes S(T') as the trajectory-level score, and we evaluate it on two popular math reasoning datasets: GSM-Plus (i at. 2024) and MATH500 (Lightman et 2023
Table I: Best-of-N accuracy across models. Bold and underlined values denote the top two results.
GSM-Plus
MATH500
Models
Methods
@8
@ 16
@32
@64
@ 128
@8
@ 16
@32
@64
@ 128
Major@N
66.3
65.6
64.7
64.2
64.2
46.2
46.0
45.2
45.0
44.8
@ (Oracle)
79.2
81.2
82,2
83.3
85.5
73.8
79.2
84.6
87.4
90.2
ORM
66.8
67.2
66.4
65.7
65.7
51.6
51.4
51.8
49.0
49.2
PRM
67.6
67.9
67.7
66.9
66.7
54.2
55.2
55.2
54.2
54.6
Qwen2.5-
PQM
68.5
69.2
68.5
68.2
68.0
53.2
54.4
54.8
54.8
55.8
3B-Instruct
IPRM
65.5
66.2
66.8
66.5
66.2
52.4
52.0
52.0
52.2
53.0
CRM (ours)
67.8
68.6
67.9
68.4
68.7
53.0
56.4
56.6
55.8
56.6
ORM
66.9
67.4
67.1
67.2
66.6
47.4
46.4
44.6
45.2
45.6
PRM
67.9
68.2
68.5
68.8
68.9
48.0
48.0
49.8
49.0
47.6
LLaMA3.1-
PQM
66.4
66.2
66.5
67.2
51.0
51.4
48.8
48.4
IPRM
65.1
64.3
63.9
63.5
63.7
48.4
45.8
44.2
46.0
45.8
CRM (ours)
67.8
68.8
69.1
68.6
68.5
49.4
50.6
50.6
49.8
50.6
CRM demonstrates stronger trajectory-level selection in Best-of-N sampling. As shown in Table CRM
Cross-Sample Comparability
consistently ranks at or near the top across both datasets and model families. For example, on MATH500 with
0.845
0.850
PRM
0.84
AUPRC
CRM (ours)
0.82
Qwen2.5-3B-Instruus
CRM reaches 56.6% at N=32,
0.60
surpassing the strongest baseline +1.4 and remain- ing ahead across N=16, 64, and 128. By conditioning each step's reward on all preceding steps, CRM ensures that the trajectory score reflects the coherence of the en- tire reasoning chain rather than isolated fragments. This holistic formulation enables CRM to assess logical consistency across steps and more reliably dis- tinguish correct trajectories from superficial ones, leading to stronger trajectory-level selection. A more in-depth analysis of the performance characteristics of BoN is provided in Appendix
0.78
0.781
0.776
GSM Plus
MATH500
Figure Cross-sample comparability.
CRM shows superior cross-sample comparability. Best-of-N evaluation focuses on the model's ability to identify the correct response among N responses for the same question, where our CRM already shows clear advantages. To move beyond this setting and assess cross-question compa- rability, we mix all responses from different questions and adopt AUPRC (Area Under the Pre- Resiln-Recall Curve) as the metric, which measures whether correct responses are concentrated among those with higher rewards. A higher AUPRC indicates that when ranking all responses glob- ally by reward score, correct responses are consistently placed toward the top. As shown in Figure 3 our CRM outperforms the baselines, validating its superior cross- samsle comparability. This advantage arises from the consistent probabilistic modeling (see Section 3.3), which ensures that reward signals carry the same semantic meaning across reasoning trajectories, thereby making them directly comparable across samples.

---

Published as a conference paper at ICLR 2026
Table 2: Beam Search accuracy on MATH500 and Gaokao2023.
MATH500
GAOKA02023
Models
Methods
N =
8
N 20
N 100
N
N 8
N 20
N = 100
ORM
50.73
54.80
56.80
58.07
35.58
38.18
38.44
40.17
Qwen2.5-
PRM
51.80
55.73
56.87
58.00
34.72
37.84
38.70
38.96
PQM
52.67
56.60
58.87
58,80
36.88
38.61
40.61
39.83
Math-1.5B
IPRM
44.27
47.27
48.33
47.47
32.55
34.46
35.32
34.55
CRM (ours)
54.07
58,40
61.00
63.00
38.70
39.74
41.04
43.55
ORM
51.87
$7.67
59.47
60.73
37.49
40.26
44.07
43.72
Qwen2.5.
PRM
52.13
$5.67
59.93
60.13
37.58
40.52
41.04
43.81
PQM
52.73
$7.87
59.20
61.13
37.84
40.61
42.60
43.29
Math-7B
IPRM
49.53
54.07
54,20
52.60
35.67
38.53
40.26
39.57
CRM (ours)
56.07
60.60
62.87
64.07
39.83
42.77
46.49
48.40
ORM
38.13
38.80
38,93
36.67
25.63
26.93
29.35
27.62
PRM
37.87
39.67
40.13
39.53
26.84
28.66
27.97
27.97
1lama3.
8B
PQM
39.20
40.47
41.00
41,27
26.58
27.71
28.31
28.23
IPRM
37.07
38.67
37.13
34.13
26.49
25.89
26.15
24.42
CRM (ours)
40.20
41.00
42.07
41.00
26.93
28.40
28.74
29.96
4.3
BEAM SEARCH EXPERIMENTS
We conduct beam search on MATH500 and an out-of-domain (OOD) dataset Gaokao2023 (Liao al., 2024) to evaluate the capacity of reward models in guiding LLM reasoning via step-level rewards. For each question, beam search initiates by sampling N responses. Subsequently, a beam of b candidates with the highest rewards is maintained and expanded during the generation process. Our CRM computes S(t) as the step-level reward. We report the best accuracy achieved across various total sampling sizes N, with the results averaged over three random seeds. Detailed experimental settings are provided in Appendix A
CRM provides effective and consistent step-level guidance for beam search. As shown in Ta- ble 2 CRM achieves the highest accuracy both datasets when using the Dwen2.5-Mathh 1.5B and Qwen2.5-Math-7B and demonstrates the best performance in the majority of cases for the Llama3.1-8B. Notably, the performance of CRM scales effectively with the total sampling size N. As increases from to 100, the performance gap over baseline methods widens, underscor- ing the scalable advantage of CRM in selecting more promising intermediate steps within larger search spaces. This can be attributed to how the reward model guides the beam search algorithm. Specifically. beam search relies on the reward model to prune a vast number of trajectories by per- forming two distinct types of comparison: (i) ranking trajectories that share common prefix, and (ii) ranking entirely distinct reasoning paths (cross-sample). By framing the reward as the con- ditional probability of reaching the correct answer given current partial trajectory, our consistent probabilistic modeling provides meaningful step-level rewards applicable both ranking scenarios, thereby effectively guiding the reasoning process.
4.4 RL OPTIMIZATION EXPERIMENTS
Table 3: Pass @ accuracy evaluated on six mathematical reasoning benchmarks.
VR from outcome ground-truth
MATH
Minerva Math
Olympiad Bench
Method
AIME25
AIME24
AMC23
500
PURE
76.0
30.8
36.7
13.3
26.6
70.0
PRM
71.6
36.3
32.5
13.3
10.0
57.5
VR Disabled
PQM
72.0
34.1
34.3
13.3
13.3
52.5
CRM (ours)
77.8
40.0
39,3
23.3
43.3
67.5
Prime
81.2
29.4
40.8
16.6
26.6
72.5
VR Enabled
PURE
82.4
40.0
41.3
23.3
23.3
70.0
VR
76.2
38.6
38,0
16.6
30.0
62.5
CRM +
80.4
43.0
33.3
72.5

---

Published as a conference paper at ICLR 2026
During RL optimization, the reward model provides step-level dense rewards to guide the policy model toward generating improved reasoning trajectories. This setting allows us to empirically val- idate the effectiveness and accuracy of credit assignment delivered by the reward model in practice.
Our CRM provides step-wise process rewards T; = log(1 -(t)). This reward formulation is not only theoretically justified in Section 3.2 but also empirically validated in Appendix G We adopt RLOO (Leave-One-Oue (Ahmadian et al., 2024) to estimate the advantage based on 7-- Since RLOO is originally defined the sequence level, following prior work (Cheng et al.) 2025), we em- ploy its token-level variant. We use Orz-Math-57k (Hu et al., (2025) as the RL training set. All policy models are initialized from Qwen2.5-Malt 5-ath 7B and trained with full parameter optimization. Details of advantage estimation, policy updates, and implementation settings are provided in Appendix
We compare against PRM and PQM, and additionally include two stronger RL baselines that use dense rewards: Prime (Cui et al. 2025) and PURE (Cheng et al.) 2025). Prime adopts online reward model updates, jointly optimizing the reward model and policy, but its reliance on verifiers prevents adaptation to ground-truth-free settings. PURE adopts min-form credit assignment by defining the value function as the minimum of future rewards. However, as stated in its original paper, becomes prone to reward hacking without verifiable rewards. In addition, we also add the VR baseline, where the policy is trained using only final-answer verifiable rewards. To comprchensively evaluate performance, use six widely adopted benchmarks: AIME25 (MAA 2025), AIME24 (MAA) 2024), AMC23 (MAA 2023), MATH500 (Hendrycks et al., 2021). Minerva Math (Lewkowyez) al. 2022). and OlympiadBench (He al.) 2024). We report the pass @ accuracy under the
zero-shot setting.
CRM boosts RL performance without VR. CRM-based RI functions without verifiable re- wards (VR Disabled). For fair comparison with VR-dependent methods, we additionally report results where rewards from CRM are combined with verifiable rewards (VR Enabled). The re- sults are shown Table In the VR Disabled setting, CRM attains the best Pass@1 accuracy on the majority of benchmarks, significantly outperforming the baselines. For instance, on AIME24 it reaches 43.3%, exceeding PURE by +16.7. CRM consistently outperforms the VR baseline. When augmented with VR, CRM + VR yields further gains and achieves the highest performance on most benchmarks. This suggests that the process rewards provided by CRM are complementary to ground-truth-basee verifiable rewards, rather than redundant. Remarkably, even without VR, CRM delivers performance comparable to VR-enabled methods. The strength CRM lies in its explicit linkage between process rewards and outcomes, which ensures that the contribution of each step is causally aligned with the final result. Since process rewards quantify how much each step advances undermines the probability of reaching the correct final answer, rather than relying on local heuristics, this design enables precise credit assignment. As a result, the dense and reliable reward signal facilitates effective RL optimization and remains robust reward hacking, further analyzed in Section 4.5.1
4.5 EXTENDED ANALYSIS
In this section, we conduct a more in-depth analysis of CRM by investigating the following research questions. RQ1: What does reward hacking manifest as, and how can it be mitigated? RQ2: Does CRM exhibit self-reflection behavior? RQ3: How efficient is CRM in utilizing supervision data? RQ4: Can CRM be applied to domains beyond mathematics?
4.5.1 REWARD HACKING IN RL OPTIMIZATION
In RL optimization, employing a reward model is prone to reward hacking (Gao et al.) 2024 Cheng
jt al. 2025). We observe that reward hacking typically manifests as the generation of excessively repetitive content. To capture this behavior, we introduce the repeat score, rased (Li ctt al. 2016) metric ranging from 0 to 1, where higher values indicate greater degree of textual repetition. The formal definition and examples of reward hacking provided in Appendix
Finding 1: Reward hacking manifests as a rapid reward increase accompanied by repeti- tive outputs and declining downstream performance (RQ1), whereas CRM remains robust. Figure reveals that under PRM and PQM. the reward quickly escalates despite declining down- stream accuracy. This discrepancy arises from pathological incentive, where models inflate
8

---

Published as a conference paper at ICLR 2026
Normalized Reward (Min-Man Min-MMx)
Response Length
Repeat Score
AIME2024 Accuracy
1,0
8000
PRM
PRM
PQM
PQM
0.6 FRM
0.5
CRU
URM
PQM
0.4
0.6
PQM
2000
0.1
2-2
2-2
0.0
0.3
0.0
100
200
100
(b)²
100
200
300
(c) 0
100
200
100
100
200
300
Training Steps
Training Steps
Training Steps
Training Steps
Figure 4: Evolution of (a) Normalized Reward (Min-Max), (b) Response Length, (c) Repeat Score, and (d) downstream task performance with training steps.
wards through excessively long, repetitive outputs, as reflected in their near-saturated repeat scores. This indicates that the reward fails to discriminate between genuine reasoning step and superficial repetition. In contrast, CRM tightly couples intermediate rewards to the final outcome, ensuring that reward faithfully reflects reasoning quality (RQ1). CRM's precise credit assignment stabilizes optimization and suppresses degenerate strategies, thereby enhancing robustness to reward hacking.
4.5.2 SELF-REFLECTION DURING REASONING PROCESS
Previous works (He et al. 2024 Liu et al. 2025b [ensal et al., 2025) have observed that improvements in reasoning ability are often associated with key phenomenon of self- reflection, in which models actively review and check previous steps. This phenomenon pro- vides important insights into understanding and enhancing reasoning in L.MMs. Following prior
Self-Reflection Score
MATH500 Accuracy
0.8
0.6
0.2
PRM
0.4
PRM
0.1
PQM
0.2
PQM
0.0
(a)
(b)*
0
200
Training Steps
Training Steps
research (Yeo et al. 2025 Liu et al.) 2025a b).
Figure 5: Evolution of self-reflection and down-
we define the self-reflection score as the aver- age frequency of reflective expressions (e.g., "rethink", "let's check") appearing in model's sponse, normalized by the output length (per 1000 tokens), which measures self-reflection capability during reasoning. The full list of reflective expressions is provided in the Appendix D
stream accuracy during RL training
Finding 2: CRM encourages more self-reftection behaviors (RQ2). As shown in Figure 5 the self-reflection score of CRM steadily rises during RL training, accompanied by improvements in downstream MATH500 accuracy, indicating that the model becomes increasingly reflective. In con- trast, PRM and PQM show little to no growth in self-reflection and their MATH500 accuracy col- lapses early. The temporal co-movement between rising self-reflection and improving MATH500 accuracy suggests that CRM precise credit assignment fosters more meaningful reasoning behav- iors, even without verifiable rewards.
4.5.3 ABLATION STUDY
Ablation loss for CRM training. Our modeling approach employs three losses (Bq. Eq. 12 Eq. 13). Among them, Ls and Lw are supervised by the label E {0, }} indicating whether the final answer is correct, with a one-to-one correspondence to samples. Removing these losses would reduce the number of training samples and undermine fair comparison. Therefore, we preserve Ls and Lw while conducting ablations on L.. The loss Cg directly encourages the model to identify the exact step at which a wrong state occurs. Specifically, set the total amount of data in the dataset applicable to L, as 100% and apply scaling, evaluating CRM with 0%. 10%, 25%, and 50% of the data under the Best-of-N sampling on the MATH500 dataset using Qwen2.5-3B. I-nstrucc....
Table Ablation results for L..
Proportion of data used for C,
@16
@32
@64
128
0%
47.0
44.2
41.6
39.2
38.2
10%
51.2
50.6
49.6
25%
54.0
52.4
53.6
53.2
52.0
50%
54.4
53.6
57.2
55.8
55.0
100%
53.0
56.4
56.6
55.8
56.6

---

Published as a conference paper at ICLR 2026
Finding 3: CRM exhibits high data efficiency (RQ3). The ablation results in Table show that even a very small proportion of data applicable to L. leads to substantial performance gains: moving from 0% to just 10% supervision produces large improvement in Best-of-N accuracy. Increasing the proportion
data yields additional benefits, but the improvements quickly saturate, with 50% already achieving near-optimal performance. This demonstrates that CRM requires only limited data for C2 to achieve strong results, highlighting its high data utilization efficiency. A more detailed analysis of the reasons behind this efficiency, well comparisons with the baseline, can be found in the Appendix
4.5.4 VALIDATION ON MORE DOMAINS
To further validate the effectiveness of our CRM, we extend our evaluation to additional domains be- yond mathematics. We leverage the multi-domain dataset MMLU- CLL-U----CT--raaii Train from VersaPRM Zeng et al., (225a) as provides reasoning processes with step-wise labels across a wide range of domains. We train the reward models including CRM and the baselines on this dataset using Owwen..5-3BIInstruu as backbone and subsequently assess their performance using Best-of-N sam- pling five distinct domains including biology, business, health, history, and physics.
Finding 4: CRM can be extended to other domains and demonstrates strong performance. (RQ4). As shown in Table 5 CRM outperforms the baselines in almost all tested domains. This demonstrates that CRM's validity is not limited to mathematical tasks and can be applied to a broader range of domains.
Table 5: Best-of-N accuracy comparison across different domains MMMLU-Pro-CoT--Eaa
Biolagy @3
Business @332
Health @32
History
Physies
Methods
@ 16
@64
@16
@64
@16
@64
@ 16
32
@64
@16
@32
@64
PRM
79.2
79.2
64.8
60.7
57.1
48.0
48.7
58.9
60.3
PQM
80.8
80.0
80.8
64.1
63.5
65.5
62.1
61.4
61.4
50.0
51.3
49.3
61.6
58.2
62.3
CRM (ours)
83.1
83.1
82.3
66.2
66.9
68.2
61.4
62.1
60.7
50.7
52.0
51.3
64.4
63.7
66.4
CONCLUSION
In this paper, we introduce CRM, which frames LLM reasoning temporal probabilistic process. By explicitly modeling the causal dependencies between steps and linking process to the outcome, CRM addresses the limitations of isolated step modeling and limited outcome awareness in prior work. Experiments across Best-of-N sampling, beam search, and RL demonstrate that CRM consis- tently outperforms strong baselines, improves cross-sample comparability, and is robust reward hacking. Building on these results, we plan to extend CRM to broader domains and task formats, and we hope this work catalyzes further research on -riil RI. for reasoning, moving beyond reliance ground truth to achieve broader generalization.
6 ACKNOWLEDGMENTS
The research received partial support from National Natural Science Foundation of China (Grant No. 62406193). The authors also gratefully acknowledge further assistance provided by Shanghai Frontiers Science Center of Human-centered Artificial Intelligence, MoE Key Lab of Intelligent Perception and Hlman--achiie Collaboration, and HPC Platform of ShanghaiTech University.
REFERENCES
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting REINFORCE-style optimiza- tion for learning from human feedback in LLMs. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pp. 12248-12 12248-12267, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 1/2125///2222..ooc 162. URL https: / antth wgw..org///2.... acl long. 662/]
10

---

Published as a conference paper at ICLR 2026
Shelly Bensal, Umar Jamil, Christopher Bryant, Melisa Russak, Kiran Kamble, Dmytro Mo- zolevskyi, Muayad Ali, and Waseem AlShikh. Reflect, retry, reward: Self-improving lims via reinforcement learning. arXiv preprint 2rxhv:2505.27766 2025.
Xiusi Chen, Gaotang Li, Ziqi Wang, Bowen Jin, Cheng Qian, Yu Wang, Hongru Wang, Yu Zhang, Denghui Zhang, Tong Zhang, et al. Rm-rl: Reward modeling as reasoning. arXiv preprini 0rXi::2505..2287, 2025.
Jie Cheng. Ruixi Qiao. Lijun Li, Chao Guo, Junle Wang. Gang Xiong, Yisheng Lv, and Fei- Yue Wang. Stop summation: Min-form credit assignment is all process reward model needs for rea- soning. arXiv preprint arXiv:2504. 15275, 2025.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reüichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint wrXiv:2110.14166 2021.
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456, 2025.
Shizhe Diao, Pengeheng Wang, Yong Lin, Rui Pan, Xiang Liu, and Tong Zhang. Active prompt- ing with chain-of-thought for large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Com- Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 11118////122022aclllooo https://aclanns logh wttgg/022..co. -lnng 737
putational Linguistics (Volume 1: Long Papers), pp.
URL,
Sinan Fan, Liang Xie, Chen Shen, Ge Teng, Xiaosong Yuan, Xiaofeng Zhang, Chenxi Huang, Wenx- iao Wang, Xiaofei He, and Jieping Ye. Improving complex reasoning with dynamic prompt cor- ruption: soft prompt optimization approach. The Thirteenth International Conference on Learning Representations, 2025.
Jiaxuan Gao, Shusheng Xu, Wenjie Ye, Weilin Liu, Chuyi He, Wei Fu, Zhiyu Mei, Guangju Wang, and Yi Wu. On designing effective rl reward at training time for IIm reasoning. arXiv preprint arXiv:2410.15115. 2024.
Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. rstar-math: Small LLMs can master math reasoning with self-evolved deep thinking. In Forty-second International Conference on Machine Learning. 2025.
Daya Guo. Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-rl incentivizes reasoning llms through reinforce- ment learning. Nature, 645(808) (0000081))6336333 2025a
Jiaxin Guo, Zewen Chi, Li Dong, Qingxiu Dong, Xun Wu, Shaohan Huang, and Furu Wei. Reward reasoning models. In The Thirty-ninth Annual Conference Neural Information Processing Systems, 2025b. WtR.https:://// http:://pppeevvieeee wwwpvviewweet//fooooo dtt.ummmm.ddo8k2222222222
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi. Zhiyuan Liu, and Maosong Sun. OlympiadBench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scien- tific problems. In Lun-Wot Ku, Andre Martins, and Vivek Srikumar (eds.). Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pa- pers), pp. 1828-38501 Bangkok, Thailand, August 2024. Association for Computational Linguis- tics. doi: 10. 1..53331/2024.acl-oooo acl ong. 2117
URL httpt::
org/2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
11

---

Published as a conference paper at ICLR 2026
Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Open-reasoner zero: An open source approach to scaling up reinforcement learning on the base model. arXiv preprint arXiv:2503.24290, 2025.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.1672., 2024.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information P Processing Systems, 35:3843-3857, 2022.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A diversity-promoting objec- tive function for neural conversation models. In Kevin Knight, Ani Nenkova, and Owen Rambow (eds.). Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 110-119, San Diego Califor- nia, June 2016. Association for Computational Linguistics. doi: 10. 10.18653/11/N16-111 1014. URL lttp::://aclannhhlooyy..o www.oog..org/ii.. 1014/
Qintong Li, Leyang Cui, Xueliang Zhao, Lingpeng Kong, and Wei Bi. GSM-plus: A compre- hensive benchmark for evaluating the robustness of LLMs as mathematical problem solvers. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 2961-2988 Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 18653vv1l 2024 lonellloon..6.. 163. URL
163/
Wendi Li and Yixuan Li. Process reward model with q-value rankings. In The Thirteenth Interna- tional Conference on Learning Representations, 2025.
Minpeng Liao, Chengxi Li, Wei Luo, Wu Jing, and Kai Fan. MARIO: MAth reasoning with code interpreter output reproducible pipeline. In Lun-Wei Ku, Andre Martins, and Vivek Sriku- mar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/ 1//2024.indings-aad/
URL https: //ac antho org.2222 findings-ace
53/
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
Xiaoyuan Liu, Tian Liang, Zhiwei He, Jiahao Xu, Wenxuan Wang, Pinjia He, Zhaopeng Tu, Haitao Mi, and Dong Yu. Trust, but verify: A self-verification approach to reinforcement learning with verifiable rewards. arXiv preprint arXiv:2505. /155505./44445 2025a.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl rl-zero-like training: A critical perspective. In 2nd for Math Workshop @ ICML 2025, 2025b.
Chunlok Lo, Kevin Roice, Parham Mohammad Panahi, Scott M. Jordan, Adam White, Gabor Mi- hucz, Farzane Aminmansour, and Martha White. Goal-space planning with subgoal models. Journal of Machine Learning Research, 2--3 2024.
Jianqiao Lu, Zhiyang Dou, Hongru Wang, Zeyu Cao, Jianbo Dai. Yunlong Feng, and Zhijiang Guo. Autopsv: Automated process-supervised verifier. Advances in Neural Information Processing Systems, 37:79935-79962 2024.
Liangehen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Meiqi Guo, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, et al. Improve mathematical reasoning language models by automated process supervision. arXiv preprint 2rXiv:2406.06592, 2024.
MAA. American invitational mathematies examination (aime). https: taaps::/mma...o .rg/ 2023.
12

---

Published as a conference paper at ICLR 2026
MAA. American invitational mathematics examination (aime). https: hatp::://mmaa.oo 2024.
MAA. American invitational mathematics examination (aime). https://maa.org: 2025.
Paulina Stevia Nouwou Mindom, Amin Nikanjam, Foutse Khomh, and John Mullins. On assessing the safety of reinforcement learning algorithms using formal methods. In 2021 IEEE 2 In- ternational Conference on Software Quality, Reliability and Security (QRS), pp. 260-269 IEEE, 2021.
Henrik Müller and Daniel Kudenko. Improving the effectiveness of potential-based reward shaping in reinforcement learning. arXiv preprint arXiv:2502.01307, 2025.
Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application reward shaping. In Ieml, volume 99, pp. 278-287. Citeseer, 1999.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint urXiv:2402.03300 2024.
Shuaijie She, Junxiao Liu, Yifeng Liu, Jiajun Chen, Xin Huang, and Shujian Huang. R- PRM: Reasoning-driver process reward modeling. In Christos Christodoullpoollss, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Proceedings of the 2025 Conference Em- pirical Methods in Natural Language Processing, pp. 1344,-13462, Suzhou, China, November 2025. Association for Computational Linguistics. ISBN 979-8-89176-3333 v 122025.emnlpmmaie
doi: 10.18653/
URL /atps://ccaathhoooyyyy.ooo o222225
2025.
679/
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409. 2024.
Charlie Victor Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time com- pute optimally can be more effective than scaling parameters for reasoning. In The Thirteenth International Conference on Learning Representations, 2025.
Peiyi Wang. Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhi- fang Sui. Math-shepherd: Verify and reinforce LLMS step-by-step without human annotations. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 9426-9439, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 18653v111 2024 long.. URL URL htt.t: http:/://aaaaattt /ttpsst//cclaaatolllooooo org/ 2024 acl- 510/
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 0424822--4487,, 2022.
Eric Wiewiora, Garrison W Cottrell, and Charles Elkan. Principled methods for advising reinforce- ment learning agents. In Proceedings of the 20th international conference on machine learning (ICML), pp. 792-799, 2003.
Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, James Xu Zhao, Min- Yen Kan, Junxian He, and Michael Xie. Self-evaluation guided beam search for reasoning. Advances Neural Information Process- ing Systems, 36:41616 18641618-46500 841650, 2023.
Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. Demystifying long chain- ohothought reasoning in 1lms. arXiv preprint arXiv:2550..033733 2025.
Fei Yu, Anningzhe Gao, and Benyou Wang. OVM, outcome-supervised value models for planning in mathematical reasoning. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.). Findings of the Association for Computational Linguistics: NAACL 2024, pp. 858-875. Mexico City, Mexico, June 2024, Association for Computational Linguistics. doi: 10. 16.883///11/224 findings- 557
URL https: / / antho ogy. ogy.org//022.22 nd ngs
13

---

Published as a conference paper at ICLR 2026
Lifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kaiyan Zhang, Bowen Zhou, Zhiyuan Liu, and Hao Peng. Free process rewards without process labels. In Forty-secont International Conference Machine Learning, 2025.
Thomas Zeng. Shuibai Zhang, Shutong Wu, Christian Classen, Daewon Chae, Ethan Ewer, Minjae Lee, Heeju Kim, Wonjun Kang, Jackson Kunde, Ying Fan, Jungtaek Kim, Hyung Il Koo, Kannan Ramchandran, Dimitris Papailiopoulos. and Kangwook Lee. VersaPRM: Multi-domain process reward model via synthetic reasoning data. In Forty-second International Conference on Machine Learning, 2025a. URL
forum? www.:uuummmmmm d= 9DmXbwPK
net/
Weihao Zeng, Yuzhen Huang. Qian Liu, Wei Liu, Keqing He, Zejun MA, and Junxian He. SimpleRL-zoo: Investigating and taming reinforcement learning for open base models in the wild. Second Conference on Language Modeling, 2025b.
Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. In The Thirteenth International Conference on Learning Representations, 2025a.
Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons developing process reward models in mathematical reasoning. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the Association for Computational Linguistics: ACL 2025, pp. 10495-10516, Vienna, Austria. July 2025b. Association for Computational Linguistics. ISBN 956- findings-acl.547. URL. httppss//ec..anthcooo.....or
5. doi: 10. 10118865/112255
findings-aee h.5.://///200..mminnge.....//
547/
Jian Zhao, Runze Liu, Kaiyan Zhang, Zhimu Zhou, Junqi Gao, Dong Li, Jiafei Lyu, Zhouyi Qian, Biqing Qi, Xiu Li, et al. Genprm: Scaling test-time compute of process reward models via generative reasoning, arXiv preprint arXiv:2504.0033 00899,, 2025.
Chujie Zheng, Zhenru Zhang, Beichen Zhang. Runji Lin, Keming Lu, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. ProcessBench: Identifying process errors in mathematical rea- soning. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pp. 1009-1024, Vienna, Austria, July 2025. Association for Com- putational Linguistics. ISBN
doi: 10.18653/v 1/2032/112222.../ acl long.000. URL
httpss//eelaatbooooo..oor//////. long.
14

---

Published as a conference paper at ICLR 2026
A
IMPLEMENTATION DETAILS
All experiments are conducted on 8 NVIDIA H20 GPUs.
A.1 REWARD MODEL TRAINING
For data processing, we follow prior work Yuu al. 2024 Zheng et al., 2025 Zhang et al. 2025b).
using newline characters to indicate boundaries between individual reasoning steps. In terms of implementation, we used the ZeRO-2 optimization stage of DeepSpeed with b precision to train the model. We employed Adam W optimizer with learning rate of 5e-6 and a batch size of 32.
A.2 BEST-OF-N SAMPLING
For the Best-of-N sampling evaluation, we adopt 1-ama-.1-88--nmae as the generator and gen- erate trajectories using the vLLM pipeline with temperature=1, top-p=1, and maximum length of 2048. Each generated trajectory is subsequently scored by the reward models. Consistent with the settings in their original papers and code implementations, PRM, PQM, and IPRM aggregate pro- cess rewards into a sequence-level score by taking the minimum value across all steps. In contrast, CRM computes S(T) the trajectory-lever score.
A.3
BEAM SEARCH
Our beam search configured with four total sampling numbers, denoted as N {4, 8, 20, 100). For each value of N, test three distinct beam sizes, b. Initially, N candidate responses generated from a given question. In each subsequent step, the reward model evaluates the current set of candidates and selects the top-scoring b trajectories prefixes for expansion, where each prefix produces N/b new continuations. This iterative process continues until all trajectories have generated complete answer the predefined maximum step count or token length is reached. The trajectory with the highest reward is then selected to extract the final answer.
For the evaluation of the Qwen2.5-Math-I.3 Qwen2.5-3B as the generator, and we adopt Ilama3. 18B as the generator for its corresponding reward model. All rollout generation is performed using the vLLM pipeline, with a temperature of 0.7 and top-p of 1. We set a maximum token length of 4096 and a maximum of 30 steps for each trajectory generation.
and @wen2.5-Math-71 reward models, we adopt
A.4 RL OPTIMIZATION
Advantage Estimation. The reward model provides step-wise process rewards т, and we adopt RLOO -utte--nee-Out (Ahmadian al. 2024) to estimate the advantage based on these rewards. RLOO uses multiple model outputs other than the current one to compute baseline, effectively reducing variance advantage estimation. Since RLOO is originally defined at the sequence level, following prior work (Cheng et al. 2025). we employ its token-level variant. Specifically, given question, the LLM generates K responses (yr { each containing most M tokens. Let denote the reward of the t-th token in the i-th response, and A denote its corresponding advantage. In our implementation, we set the discount factor y to 1. The advantage is computed as:
En EM, EM
A
(15)
1)M
Policy Update.
We optimize the policy model Ta by maximizing the following objective:
To(yi vr Toa (ut tt
Ai. clip To( v) To (vf v. "
c, 1 11:.)1111111 11,,,++)A4)
L(0) E,,
min
( Tref)
(16)
where Tea. is the old policy and To., is the reference model. is the PPO clipping threshold and controls the strength of the KL. penalty; both are hyperparameters.
15

---

Published as a conference paper at ICLR 2026
Experimental setting. We use Orz-Math- 57k (Hu al. 2025) was created through a thorough data-cleaning process, ensuring no overlap with commonly used benchmarks. Experiments each I.M--generaed response into steps using double line breaks and assign scores to each step with the reward model. We train with fixed learning rate of 1e-6, prompt batch size of 64, and 4 responses sampled per prompt, using KL coefficient & = le-3 and clip ratio e = 0.2. For generation, we adopt vLLM with temperature I, top-p=1, and a maximum length of 8192 tokens, while during testing we set temperature =0 and top-p=1.
the RL training set. The dataset
conducted with veRL (Sheng et al. 2024) framework. We segment
CRM+VR setting. When used CRM alone, it provides sep-wise rewards (r()) for all steps, in- cluding the final reasoning step T. When combined with VR, only the final step additionally receives verifiable reward (r while earlier steps still rely solely on CRM. Formally, the reward each step / is defined as:
J Ff
< =ft=
T
+
(17)
B FULL EXPERIMENTAL RESULTS FOR BEAM SEARCH
The detailed specification of the beam sizes b used for each N, along with the complete results, is provided in Table Table 7 and Table g
Table 6: Beam Search performance of --ren2.5-MMth--- on MATH500 and Gaokao2023
GAOKA02023
CRM
34.29
50.75
1.47
49.73
2:27
50:27 3.33
2.78
0.7.8
51.27 0.38
52.53
$4,00
0.003
53,67 1.55
38.18 + 1,30
35.15 1.73
$7.55
0:077
36.54 2:42
54.80
1.40
55.33
37.49
1.47
1.30
1.47
52:80
0.60
0.35
36.54
30.56.10.33
39.74
53.93
0.67
55.73
37.75 0.43
57.75
59.05
34.29
56.00
1.:0
59.00
38.44 1.04
34.53
0.78
55.00
0.40
55,30
61.00
37.66
34,70
0.26
50
57.00
1.00
$7,50
0.50
58.60
0.60
46.87 1.33
63.00 1: 0.40
40 0.87
38.53 0.95
59, 39.99-11 1.13
34.55
39.91
1.13
100
25
58.07
0.53
$7.40
0:40
57.73 +
47.47
61.47 0.9
18:30 0.87
38. 96 1,30
59.83 + 1.73
32.99 1.56
43.55
it 0.6
55,87
0.33
61.87
0.73
38.70 0,26
38,27
39.05
32,99
42.08
Table 7: Beam Search performance of Qwen2.5-Math on MATH500 and Gaokao2023
MATIL500 PQM
GAOKA02023 PQM
N
ORM
PRM
IPRM
CRM
ORM
PRM
IPRM
CRM
51.47
1.53
$1.00 + 1.40
50.87
+ 0.71
40.53
1.07
$1.20
1.00
16.88
0.50
34.55 0.50
36.45 + 1.73
35 + 1:47
32 0.35
51.87
0.75
$2.13
1 1.47
52.73
1 0.57
45.73
1.67
54.20
0.60
37.49
2.25
38.32
0.26
$7.84
1. 0.61
51.95
1.56
38.55
1.47
51.53 1 1.87
50.93
1 0.47
52.27
1 1,53
42.13.1008
0.87
$6.07
1 0.55
35.96
1.13
37.58
1: 1,90
$7.90
1. 0.52
32.29
082291100
0.95
39.85
1.73
$4.60
0:40
$3.60
0.50
56.53
1.67
$4.07 0.1.3
$6.00
0:40
38:27
1.21
57.32
1.30
$7.55 + 0.87
38.55
1.99
38.87
0:35
57.67
1.33
55.67
0.13
57.87
+ 1.53
50.27
= 0.33
6060
1.40
40.26
0.36
40.52
= 0.78
59.63
+ 1.39
34.05
1.04
42.60
1.04
56.27
= 0.93
$4,07
= 1.13
56.07
4 0.53
46.33
=: 1.07
55.27
2.35
39.57 + 1.47
39.13
= 1,90
40.61
+ 0.69
31.69
0.52
42.77
1.13
20
56.27
1.13
56.73
1 0.27
56,93 1. 1.47
54.20 1 2.50
$7.80
1.40
41.30
1,30
41.04
0.52
39.83 1 0.69
00.25
41.90
0.69
20
10
59.47
1.53
50.93
2 0.57
59.07
4: 0.93
52.73 +: 1.67
61.40
1.20
44.07
0.61
40.78
=: 0.52
41,21
4: 0.61
37.14
1.56
45.37
061
58:00
= 0.50
57.07
2 0.53
59.20
4: 1.20
49.47
4 1.13
62.87
0.53
42.16
22.16:22.
2.51
40.61
:: 2.25
42.60
4: 0.52
35.41
1.21
46.49
0.52
59.7
1.73
58,40
0.30
59,00
1 0.53
52.80 0.80
63.80
1.00
43:55
0.51
43.12 1,30
42.68 1 0.69
39.57
2.51
48.31
2.60
60.73
0.67
60.13
0.67
61
1: 0.47
63,60
43.64
2.50
43.81
0.35
43.01
1: 1.13
37.89
0.95
47.62
59447
0.53
59.37
1.13
60.30
1.80
48.00
64.07
0.33
43.72
1.73
41.90
0.43
43.29
34.80
48.40
1.21
C REWARD HACKING
C.1 REPEAT SCORE
To quantify the degree of redundancy in generated responses, define the repeat score based on n-gram statistics. This metric is a widely used indicator of textual repetition the literature. Given text sequence, first normalize it by lowercasing, unifying quotation marks, and removing code blocks, LaTeX environments, and inline formulas. The text is then tokenized into alphanumeric tokens and basic punctuation.
16

---

Published as a conference paper at ICLR 2026
Table 8: Beam Search performance of Llama3.1-30 MATH500 and Gaokao2023
N
ORM
PRM
MATH500 PQM
IPRM
CRM
ORM
PRM
GAOKA02023
PQM
IPRM
CRM
37.00
1.00
31.67
2: 0.33
38.27
a 0.32
57.07
2: 0.12
40.20
0.40
25.37 0.51
26,84 2: 1.47
25.37 a 0.35
26.49
1.56
26.95
2.16
38.13
0.47
31.67
1: 1.53
39.20
0.50
34.27
1,33
39.60
0.80
25.63
1.13
26.41
1 0.87
26.58
4 0.43
25.72
1.75
25.89
1.90
36.67 1 1.53
31.87
0.99
38.20
1:000
32.73
1
37.47
1: 0.5
23.55
0.87
26.23
1.82
25.54
1.47
22.34
1:
25.80
1.600...60 0.69
38,00
1.80
39.07
1:0.3
39.47
4: 1.93
38,07
4: 1.73
38,67
0.55
15.45
1.30
26,93
0.35
27.71
a 0.61
25,89
2.42
28.40
0.43
38.23
1.57
19.07
1 0.33
40.47
1: 1.13
36,73
1,27
41.00 1: 1.40
26.93
0.87
28.23
0.35
26,01
1:2.
23.95
1: 2:25
26.8:
1 0.43
38077
3 0.53
39,67
1: 1,33
40.27
0.073
34,40
0.80
39.20
1 0.80
24.00
2.94
28.06
1: 1.99
26.33
1 0.69
23:45 0.95
26.15
0.87
20
96.38
1.07
40.13
41.00
2.800
57.13
0.87
20.53 0.67
26.75
0.52
27.27
1.04
28.31
26.15 2:16
27.62
20
10
98.33
1.07
30.50
2.40
40.67
0.0733
34,07
1.33
42.117
1.15
20.35
0.52
27.53
1
26.75
1.30
24.68
2.08
28.31
1.04
37.23
x 1.47
19.50
1.40
40,40
1,20
33.93
1.07
40.80 3: 0:40
26.23 0.78
27.90
8: 1,85
27.36
1: 0.95
25.02
1.4
28.74:1:00
36.67
0.73
39.07
1.53
41.27
+ 1.53
32.20
1.20
2066 0.35
27.62
1.73
27.79 + 0.78
28.23
0.09
23.46
0.17
28.31
2.88
100
36.67
1.73
30.53
0.57
40.27
0.0.3
33.13
1.07
20.20 0.60
27.19
1.39
27.90
1.13
27.53
+: 0.78
24.42
0.52
29.96 0.69
10
35.33
0:57
36.47
2 0.33
38.47
1 0.33
34.13
1,67
41.00 4: 1.40
27.27
1.82
27.90
2 0.87
27.01
+ 1.04
22.94
1.21
28.83
1.56
For an n-gram size n, we compute the Distinct-n ratio (Li et al. 2016
Distinct-n (unique n-grams}|
total n-grams
The Repetition-n score is defined as:
Repetition-n Distinct-n,
which ranges from 0 to with larger values indicating higher proportion of repeated n-grams.
In practice, since = 2 and
are widely adopted choices in NLP research and have been shown be effective for capturing local redundancy patterns, follow this design in our implementation and compute Repetition-2 and Repetition-3, taking their maximum as the final repeat score:
Repeat Score max(Repetitionn) Repetition-3).
C.2 EXAMPLES OF REWARD HACKING
Example
Question:
Given that the sum the first n terms of an arithmetic sequence {a,) is Sn, af 18, then Sg equals
A: 99
B: 81
C: 6:
LLM Response:
To solve this problem, we need to use the properties of an arithmetic sequence. Let's break down the problem step-by-step. Arithmetic Sequence Formula: The n-th term an arithmetic sequence can written
an = a1 + (n 1)d
Sum of the First Terms:
n (2al + (n 1)d) S, (2a)
2
Given Condition: We are given that a4 a6 18. Using the formula:
a1 a1 3d, a6 + 5d
So,
04 + ag 2a + 8d 18
a1 4d =9
4. Sum of the First 9 Terms:
Sp 2 (2a1 + 8d)
2 (2a: + 8d) = 9999
Let's solve this problem sy-p-by-step. 1. Arithmetic Sequence Formula:
17

---

Published as a conference paper at ICLR 2026
2. Sum of the first terms:
Start generating repeated content.
3. Sum of the first terms:
Sum of the first terms:
Let's solve the first terms: Let's solve the first terms:
Let's solve the first terms:
...
Let
Let9
Let9
Let
Let
Let9
Let9
Example 2
Question:
Among the 900 residents Aimeville, there are 195 who own a diamond ring, 367 who own set of golf clubs, and 562 who own a garden spade. In addition, each of the 900 residents owns a bag of candy hearts. There are 437 residents who own exactly two of these things, and 234 residents who own exactly three of these things. Find the number of residents of Aimeville who own all four of these things.
LLM Response:
To solve this problem, we can use the principle of inclusion and exclusion. Let's define the following sets:
D = the set of residents who own a diamond ring.
G the set of residents who
set of golf clubs.
= the set of residents who own a garden spade.
C' the of residents who own bag of candy hearts (which is the entire population of Aimeville).
We are given the following information:
D = 195
367
SI 562
- C' = 900
The number of residents who own exactly two of these things is 437.
The number of residents who own exactly three of these things is 234.
We need to find the number of residents who own all four things, i.e., IDnGnSnC]
First, let's 's denote the number of residents who own exactly & of these things by Ng- We know that:
N₂ 437
N₂ 234
Find the values of f(x) in [-pi / /4]
Start generating repeated content.
Find the values of f(x) in [-pi Find the values of f(x) in [-pi
Find the maximum and minimum values of f(x) in [-pi 4]
Find the maximum and minimum values of f(x) in [-pi 4]
Find the maximum and minimum values of f(x) in I-pi 4]
Find the maximum and minimum values of f(x) in [-pi / 4]
18

---

Published as a conference paper at ICLR 2026
The values of f(x) in The values of f(x)
The values of f(x) in The values f(x) in
D REFLECTIVE EXPRESSIONS
Table 9 lists the reflective expressions used to compute the self-reflection score.
Table 9: Reflective expressions used in the computation of the self-reflection score.
Reflective Expressions
Example
wait
"wait, let me think"
recheck
"recheck this step" "retry the calculation" "let's try again" "alternatively, we can
retry
try again
alternatively
however
"however, this may fail" "rethink the argument" "let's check the result" "let's verify the answer"
rethink
let's check
let's verify
E CROSS-SAMPLE COMPARISON CAPABILITY
PQM focuses on intra-sample ranking but lacks cross-sample comparability. The core mod- cling principle of PQM (Li & Li (025) focuses on the relative ordering of steps within a sample, rather than modeling absolute values. For instance, applying the same shift operation to the Q-values of all steps within sample still yields the same optimal ordering. As result, the reward magni- tudes across different samples do not share consistent meaning. As stated in Corollary D. of the PQM paper, PQM allows comparison only when two trajectories share the same correct prefix. The limitation is validated by suboptimal performance in our Best-of-N and beam search experiments Section 4.3). further demonstrating PQM's deficiency in cross-sample comparability.
(Section 4.2
IPRM's lack of modeling for relationships between reasoning steps leads to poor cross-sample comparability. core deficiency of IPRM (Yuan et al., 2025) that its training relies solely on the final outcome, which leads to ambiguous credit assignment, as the model only ensures that the aggregate of all step-level rewards aligns with the final result, without enforcing more rigor- ous constraints on the process. Consequently, it lacks explicit modeling of the fine-grained causal relationships between reasoning steps. The process reward for any given step is not necessarily co- herent with the preceding trajectory, resulting in inconsistent process rewards that cannot be reliably compared across different samples.
F CASE STUDIES ON PROCESS REWARDS
Figure 6 presents two case studies that compare the scoring behavior of different reward models on reasoning trajectories from the MathShepherd validation set. Each case visualizes the predicted reward per step (top) and reward drop between consecutive steps (bottom). The background is shaded distinguish correct (green) and incorrect (red) reasoning steps, with a dashed line marking the error occurrence.
Our CRM precisely identifies reasoning errors. ! The ORM fails to provide meaningful pro- cess rewards. Its rewards volatile and poorly aligned with the correctness of intermediate steps.
19

---

Published as a conference paper at ICLR 2026
Correct Steps
Steps
1.00
Reward Drop Reward
0.75
0.50
0.0
10
10
Reasoning Step
Reasoning Step
Figure 6: Case studies of different reward model scoring on trajectories from the MathShepherd validation set.
Response Length
Repeat Score
Self-Reflection Score
0.35
1.0
Lag PRM
Log: PRM
Lag-PQU
Log-POM
6000
0.8
Linear-CRM
0.25
Linear-CRM
Log-Pigg PRM
CRM (ours)
(ours)
Log POM
4000
Linear-CMM CRM
0.15
(ours)
0.4
0.10
0.2
0.05
0.0
0.00
(a)
50
Training Steps
100
150
200
250
300
(b)
0
50
Training Steps
100
150
200
250
300
(c)
100
Training Steps
200
300
400
500
600
Figure 7: Training dynamics of ablation variants: (a) Response Length, (b) Repeat Score, and (c) Self-Reflection Score over training steps.
@ The PQM, while capturing local ranking dependencies between reasoning steps, receives only indirect supervision from the final outcome. Consequently, may incorrectly assign high rewards after a fatal error, failing to provide a precise assessment of the final result. @ In contrast, our CRM overcomes these issues and precisely identifies which reasoning step enters an incorrect state. maintains high rewards for correct steps, exhibits an immediate drop when the first error occurs, and subsequently maintains a low reward for all following incorrect steps.
G ABLATION STUDY ON PROCESS REWARD FORMULATION OF CRM
Table 10: Pass@1 accuracy across benchmarks for ablation variants.
Method
MATH500 Minerva Math OlympiadBench AIME25
AIME24
AMC23
Log-PRO PRM
71.4
33.0
32.2
13.3
16.6
45,0
Log-PQU Bog-PQM
71.4
33.8
34.1
13.3
20.0
45.0
Linear-CRM
71.4
33.0
32.2
13.3
13.3
57.5
CRM
77.8
40.0
39.3
23.3
43.3
67.5
Our process reward is defined as loor= h1), derived from the theoretical analysis in Section 3.2
To validate the necessity of this formulation, we conduct experiments from two perspectives: (1) apply log transform to baselines (PRM and PQM) score, i.e., redefining their original per-step reward rf as rz = logr%, denoted as Log-PRM and Log-PQM; and (2) further test a linearized variant of CRM that replaces the original reward with
h; we refer to this Linear- CRM. Figure shows that Log-PRMM/LogPPOL PQM and Linear- Linear-CRM quickly exhibit reward hacking where response length collapses and self-reflection vanishes, whereas CRM remains stable and steadily increases self-reflection. Table 10 mirrors this trend: CRM attains the best accuracy on every benchmark with clear margins over all ablations. These results underscore the necessity of
20

---

Published as a conference paper at ICLR 2026
theoretically grounded shaping r log(1 h). as it offers stable and informative process reward that effectively mitigates reward hacking.
H COMPARISON WITH REASONING-ENHANCED PRMs
Recent work (Chen et al.
Guo et al. 2025
20256 She al., 2025 Zhao et al. 2025) in-
Table 11: Best-of-N accuracy comparison with reasoning-enhancee PRMs on MATH500.
legrates explicit reasoning capabilities into re- ward modeling. To compare our CRM with this line of work, we evaluate it against the rep- resentative gaasoning-enhaneee reward model R-PRM (She et al. 2025) and GenPRM (Zhao ct al. 2025).
Method
Best-of-8
Best-of-16
R-PRM
49.8
51.2
GenPRM
50.8
55.2
CRM (ours)
53.0
56.4
We evaluated these reward models using their official released checkpoints, the same settings and inference pipelines to score identical responses from our generator on the MATH500 dataset under Best-of-N sampling. The results are shown in Table [T] Compared to both baselines, CRM consistently achieves the highest performance. Specifically, under the Best-of-8 setting, CRM achieves 53.0, surpassing R-PRM (49.8) and GenPRM (50.8). Similarly, under the Best-of-16 setting, CRM reaches 56.4, outperforming both R-PRM (51.2) and GenPRM (55.2). These sults highlight CRM's stronger capability in identifying high-quality responses compared to recent reasoning-enhancee and generative approaches.
I BEST-OF-N SAMPLING PERFORMANCE CHARACTERISTICS
In this section, we provide a more in-depth analysis of the performance characteristics of Best-of-N sampling. In BoN, an LLM generator first samples N candidates for each query. Subsequently, a reward model selects the response with the highest predicted reward. Therefore, the LLM generator plays role in shaping BoN performance. We observe that different LLM generators lead to distinct behaviors: BoN performance may continue to improve as N increases, or it may converge a small N and show no further gains. As shown in Table [2] with strong generator (Ilama-3.1- 8B-Instruct), all methods converge at small N and do not improve further, whereas with a weaker generator BestaMath-Mistral--
all methods exhibit increasing BoN performance as N increases. This phenomenon can be explained as follows. (i) Weaker LLM generators have a lower probability of producing correct responses. At small N, they may generate correct candidates, leaving the reward model with nothing correct to select. As increases, the sampled candidates are more likely to include correct answers, so the BoN performance starts improve. (ii) Stronger LLMM generators are able to produce correct responses even with small N. In this case, the BoN performance is not limited, leading faster convergence.
These points are further supported by the @ass@N metric, which measures the probability that the LLM generator produces least one correct response among the N responses, serving as an upper bound on the achievable performance of BoN sampling with any reward model. Table [3 reports the @ass@N of the two LLM generators. The weaker generator substantial gap between @ 8 and Pass@ 128, with a relatively low Pass @8. This indicates that when N is small, the low upper bound limits BoN performance. The stronger LLM generator LLLMA-3.1 1-8- achieves high @ even for small N, leading to early convergence.
Settaaat--Mistall
shows
J DATA EFFICIENCY ANALYSIS OF CRM.
To further substantiate the data efficiency of CRM, we added PRM for comparison. We refer to the data from the PRM training set, excluding the final step, as process data. We conducted experiments using the same data proportion in our ablation study, except for 0% (since PRM requires process data). As shown in Table 14 CRM reaches near-optimal performance with only 50% of the training data, whereas PRM still exhibits noticeable performance gap under the same data budget.
21

---

Published as a conference paper at ICLR 2026
Table 12: Best-of-N sampling performance across methods using different LLM generators.
MATH500
LLM Generator
Method
@8
@
@
@64
@ 128
Pass@
53.0
62.0
69.6
75.8
80.4
PRM
34.0
35.8
35.2
37.4
38.0
SccaMath-Mistral-22
PQM
36.4
38.4
37.8
38.6
40.4
CRM (ours)
35.6
39.0
39.2
40.2
41.6
Pass N
73.8
79.2
84.6
87.4
90.2
PRM
54.2
55.2
55.2
$4.2
54.6
LLaMA-331-8B-mmsmtne
PQM CRM (ours)
53.2
54.4
54.8
54.8
55.8
53.0
56.4
56.6
55.8
56.6
Table 13: @ of different LLM generators on MATH500 and GSM-Plus datasets.
MATHI500
LLM Generator
A(@8->@12
688
00:22
6964
@
A[688
A(08-10120
688
69 16
6:32
6964
128
MeraMath-Mistmi LLaMA-3.1-88 1.1.-3.-----smeee Instract
27.4
53.0
69.6
75.8
80.4
11.1
73.2
78.1
80.5
83.0
84.3
16.4
73.8
79.2
84.6
87.4
90.2
79.2
81.2
82.2
83.3
85.5
The efficiency gain stems from how CRM leverages the supervision at step z. While PRM treats each step independently and learns a step-wise correctness classifier, CRM instead models reasoning as a temporal progression toward the final answer. By framing the entire reasoning trajectory sequence converging to correctness, as derived in Eqs. II 12 and [3] CRM allows the supervision signal at step to propagate backward to all earlier steps S 2. This provides richer and more coherent training signals and enables CRM to make more effective use of available data, resulting in significantly improved data efficiency.
Table 14: Best-of-N accuracy on Math500 using varying proportions of data for CRM's C3 and PRM.
Proportion data used for L.z
CRM
Process data used for PRM
PRM
@
@ 16
@32
@64
@ 128
@8
@ 16
@ 32
@66
@ 128
0%
47.0
44.2
41.6
39.2
38.2
0%
10%
52.4
51.2
50.6
49.6
47.6
10%
48.0
49.4
48.6
48.0
47.4
25%
54.0
52.4
53.6
53.2
52.0
25%
53.0
54.0
54.6
52.6
50.8
50%
54.4
53.6
$7.2
55.8
55.0
50%
52.4
54.2
55.2
53.4
52.0
100%
53.0
56.4
56.6
55.8
56.6
100%
54.2
55.2
55.2
54.2
54.6
22


Reviews:
[{"review_id": "CglwrH4mMj", "reviewer": "Reviewer_G7br", "summary": "The paper introduces Conditional Reward Modeling (CRM) for multi-step reasoning, aligning step-level rewards with the final outcome through a principled probabilistic framing centered on the “first error” in a chain of thought. This yields a single, comparable trajectory score that supports better sample selection (Best-of-N), stronger beam-search guidance, and more stable RL training with reduced reward hacking, while remaining complementary to verifiable outcome rewards when available. Experiments on math reasoning benchmarks and multiple backbones show consistent gains over PRM/PQM/IPRM/ORM baselines and clearer localization of the first wrong step. Ablations indicate robustness to limited step-level supervision and to reward-shaping choices, suggesting a simple and practical framework for outcome-aligned process rewards.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper poses an interesting problem and ties the method well to practical RL applications.", "weaknesses": "The experiments do not appear to discuss the latest PRM (including trainable PRM) designs. [1][2]\n\nCRM seems to converge with small n; please add Major@N and Pass@N baselines to Table 1 to contextualize performance and the gap to an upper bound.\n\nTable 3 should include an RLVR baseline.\n\n[1] R-PRM: Reasoning-Driven Process Reward Modeling\n\n[2] GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qs4pINvVZd", "reviewer": "Reviewer_tScJ", "summary": "This paper proposes Conditional Reward Modeling (CRM) that frames LLM reasoning as a temporal process leading to a correct answer. The core idea is to model the reasoning process as a conditional probability process, associating the reward of each step with the preceding tokens and the final result. The author conducted experiments on various applications of their reward model, in most of which they showed good performance. The cost of this article is also acceptable, and the experimental comparison is fair from this perspective. The current improvement in the reward model mostly comes from the reasoning reward model, which is easily hacked, or training data scaling. I think this is a good paper that should be accepted as a poster.", "questions": "1. The results in Figure 4 show the behavior of reward hacking. But the reviewer has conducted a similar experiment using Math-Shepherd, and although the effect was not good, the AIME2024 accuracy did not decrease to 0. Is there a problem with this part of the experiment? That is, the author obtained a good training hyperparameter on CRM and directly used it on other baselines, which require different hyperparameters.\n2. In the BoN exps, the authors use the score S(T) for CRM and take the min value for other models. What would happen if other models also use S(T)?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well written, and the proposed method is reasonable.\n2. The experiment is sufficient to demonstrate the effectiveness of the method.\n3. The experiment can prove that the reward learned by the method is robust and is not easily hacked.", "weaknesses": "The author can try to expand the method to more fields, such as code, or even fields without verifiers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vv9zPAMJuu", "reviewer": "Reviewer_8bhE", "summary": "This paper introduces Conditional Reward Modeling (CRM) — a framework for enhancing the reasoning capability of large language models by modeling the reasoning trajectory as a temporal probabilistic process. CRM explicitly links intermediate reasoning steps to the final outcome by defining step-level rewards as conditional probabilities of maintaining correct reasoning. This design aims to resolve credit assignment ambiguity and improve robustness against reward hacking.\nEmpirical results across Best-of-N sampling, beam search, and reinforcement learning (RL) demonstrate that CRM outperforms baseline reward models (PRM, PQM, IPRM) in reasoning tasks such as GSM-Plus, MATH500, and AIME benchmarks. The method shows stable optimization and increased self-reflection behaviors in RL-trained models.", "questions": "1.\tHow does CRM compare quantitatively to DeepSeek-R1 or Qwen3-RL in reasoning benchmarks (e.g., AIME24, GAIA)?\n\t2.\tWhat is the computational overhead introduced by CRM’s step-level conditional modeling during beam search or RL?\n\t3.\tCould CRM be integrated with verifiable rewards rather than replacing them, to combine interpretability with scalability?\n\t4.\tHave you attempted to apply CRM to large reasoning models or long-horizon tasks beyond math reasoning (e.g., proof or planning tasks)?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.\tFormal rigor — CRM introduces a clear probabilistic derivation linking reasoning steps to final outcomes, which elegantly resolves step-level credit ambiguity.\n\t2.\tEmpirical robustness — Demonstrates consistent performance gains across multiple reasoning benchmarks and settings (Best-of-N, beam search, RL).\n\t3.\tReward hacking analysis — The paper includes a thoughtful investigation into degenerate reward optimization and shows CRM’s improved stability.\n\t4.\tData efficiency — The ablation on L_z shows CRM achieves high data utilization, improving with limited supervision.", "weaknesses": "1.\tFlawed motivation — The key claim that verifiable rewards are costly and difficult to scale is unconvincing and outdated.\n\t- Recent large-scale reasoning systems such as DeepSeek-R1 have demonstrated that verifiable rewards are both accurate and scalable, making CRM’s justification inconsistent with current research trends.\n\t- Moreover, DeepSeek-R1 explicitly abandoned PRM-style process rewards for their instability, undermining the central motivation of this paper.\n\t2.\tLack of comparison to verifiable-reward models —\n\t- Although CRM shows improvement over PRM-family models (PRM, PQM, IPRM), the experiments exclude comparisons with contemporary verifier-based RL systems like Qwen3-RL, DeepSeek-R1, or OpenAI o1.\n\t- Without these baselines, it’s difficult to assess whether CRM’s improvements are competitive in modern reasoning settings.\n\t3.\tLimited scalability and applicability —\n\t- CRM is only tested on LLM-based reasoning (Best-of-N, beam search, RL) but not on large reasoning models (LRMs), where process reasoning involves extended multi-branch trajectories and hierarchical dependencies.\n\t- The proposed CRM-based search mechanism may also introduce additional computational cost compared to verifier-based approaches, which is not analyzed or mitigated.\n\t4.\tIncremental novelty —\n\t- The use of conditional probability chains and potential-based reward shaping is mathematically elegant but conceptually derivative of standard PRM extensions (e.g., IPRM, PQM).\n\t- The claimed robustness advantage could likely be replicated by integrating verifiable signals with existing PRMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UwkTWnO1v3", "reviewer": "Reviewer_1Jdv", "summary": "The paper proposes a new theoretical framework with Potential-Based Reward Shaping to train PRMs.  Their framework not only captures the dependencies between reasoning steps, but also between the reasoning steps and the final outcome. The empirical studies also validate the effectiveness of their frameworks.", "questions": "> Implementation details on how to combine CRM + VR would enhance the reproducibility.\n\n> Finding 3 (Line 466) claims CRM exhibits a high data efficiency, but without comparison with previous works. Does CRM exhibits a higher  or on-par data efficiency compared to previous works?\n\n> Confused about what each peak of CRM in Figure 1 means? It cannot show your CRM can capture dependency between reasoning steps and outcome.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The intuition behind the framework is well-motivated and easy to follow.\n- The empirical studies are comprehensive, including BoN, PRM-guided beam-search, and RL experiments. The cross-sample comparability appears appealing.", "weaknesses": "> Ambiguity of why choose $\\log S(t)$ as the potential function.\n\nThe authors directly claim \"A natural choice for the potential function is one that estimates the likelihood of eventually reaching a correct answer from the current state\", but abridge the motivations and insights behind this substitution. Since this substitution is the key step to derive the final objective, the ambiguity here confuses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)