Stop Summation: Min-Form Credit Assignment Is All Process Reward Model Needs for Reasoning
Authors: Jie Cheng
Gang Xiong
Ruixi Qiao
Lijun Li
Chao Guo
Junle Wang
Yisheng Lv
Fei-Yue Wang
Venue: NeurIPS 2025 poster ()
Keywords: LLM
reasoning
process reward model
Primary area: deep_learning

Abstract: Process reward model (PRM) has been proven effective in test-time scaling of LLM on challenging reasoning tasks. However, the reward hacking induced by PRM hinders its successful applications in reinforcement fine-tuning. We find the primary cause of reward hacking induced by PRM is that: the canonical summation-form credit assignment in reinforcement learning (RL), i.e. cumulative gamma-decayed future rewards, causes the LLM to hack steps with high rewards. Therefore, to unleashing the power of PRM in training-time, we propose PURE: Process sUpervised Reinforcement lEarning. The core of PURE is the min-form credit assignment that defines the value function as the minimum future rewards. This method unifies the optimization objective with respect to process rewards during test-time and training-time, and significantly alleviates reward hacking due to the limits on the range of values of value function and more rational assignment of advantages. Through extensively experiments on 3 base models, we achieve similar reasoning performance using PRM-based approach compared with verifiable reward-based approach if enabling min-form credit assignment. In contrast, the canonical sum-form credit assignment even collapses training at the beginning. Moreover, when we incorporate 1/10th verifiable rewards to auxiliary the PRM-based fine-tuning, it further alleviate reward hacking and results in the best fine-tuned model based on Qwen2.5-Math-7B with 82.5% accuracy on AMC23 and 53.3% average accuracy across 5 benchmarks. Furthermore, we summary the reward hacking cases we encountered during training and analysis the cause of training collapse.

Stop Summation: Min-Form Credit Assignment Is All Process Reward Model Needs for Reasoning
Jie Chengl?? Gang Xiong 1., Ruixi Qiao 1.2, Lijun Li Chao Guo',
Junie Wang'. Yisheng Ly1.2.5* Fui-Yue Wang 5,6,78 State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences
School of Artificial Intelligence, University of Chinese Academy of Sciences $Shanghai Artificial Intelligence Laboratory *Tencent
s Faculty of Innovation Engineering, Macau University of Science and Technology
DeSci Center of Parallel Intelligence, Obuda University
'Research Center for Chinese Economics and Social Security,
University of Chinese Academy of Sciences Institute of Automation, Chinese Academy of Sciences
Abstract
Process reward models (PRMs) have proven effective for test-time scaling of Large Language Models (LLMs) challenging reasoning tasks. However, reward hacking issues with PRMs limit their successful application in reinforcement fine- tuning. In this paper, identify the main cause of PRM--inuceed reward hacking: the canonical summation-form credit assignment reinforcement learning (RL.), which defines the value cumulative gamma-decayed future rewards, easily induces LLMs to hack steps with high rewards. To address this, we propose PURE: Process sUpervised Reinforcement EErnning. The key innovation of PURE a min- form credit assignment that formulates the value function as the minimum of future rewards. This method significantly alleviates reward hacking by limiting the value function range and distributing advantages more reasonably. Through extensive experiments on 3 base models, show that PRM-based approaches enabling min- form credit assignment achieve comparable reasoning performance to verifiable reward-bard based methods within only 30% steps. In contrast, the canonical sum-form credit assignment collapses training even at the beginning! Additionally, when we supplement FRM--asee fine-tuning with just 10% verifiable rewards, we further alleviate reward hacking and produce the best fine-tuned model based Qwen2.5- Math-7B in our experiments, achieving 82.5% accuracy on AMC23 and 53.3% average accuracy across 5 benchmarks. Moreover, we summarize the observed reward hacking cases and analyze the causes of training collapse. We release our code and model weights at settps://gttie wwwpab/.cccaCCliiiffoe/////t///e
Introduction
Reinforcement fine-tuning (RFT) of large language models (LLMs) for reasoning tasks has shown
promise in developing advanced problem-soll solving abilities. Recent advancements (Wang et al. 2025
Zeng et al. 2025a), such as DeepSeek R1-Zero (Guo al. 2025) and Kimi K1.5 (Team et al." 2025
have demonstrated strong reasoning skills through RFT with verifiable rewards, which provide sparse feedback for the entire response. However, as response length inereases, sparse rewards potentially lead to inefficient learning (Sutton & Barto 2018 Andrychowicz et al. 2017).
"Corresooniigg author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

In contrast, process reward models (PRMs) offer dense feedback at each step of response. PRMs have proven effective in improving LLMs' performance on challenging reasoning tasks through test-time scaling (Lightman et al. 2023] Wang et al. 2023 Guan et al. (202)). However, successful
applications of PRM RFT for M. remains limited (Setlur et al. | (024). h. key challenge of PRMs
is that the neutral network-generated rewards can lead to reward hacking during training (Weng) 2024
Guo et al., 2025), causing unintended optimization toward higher rewards. Recent studies Yuan
al. 2024: Cui et al., 2025) have explored implicit PRMs for fine-tuning LLMs, which resembles
DPO-style reward formulation (Rafailov et al. 2023), but these still heavily relies on ground-truth verifiable rewards to provide training signals for the system. The question of what causes reward hacking in PRM-based RFT and how to address it effectively has not been explored widely.
To answer this question, we examine the usage of PRM in test-time scaling and identify mismatch between test-time and training-time objectives. Thus in this paper, we propose PURE: Process SUpervised Reinforcement |Earning. which introduces a min-form credit assignment method to align these objectives. Through analysis and experiments, we find that the canonical formulation of credit assignment in RL, named summation- formm credit assignment that defines value as cumulative gamma-decayed future rewards, easily induces LLMs to hack high-reward steps and collapse training. Instead, we use the minimum future reward to quantify credit assignment, which constrains the value function's 's range and assigns advantages more reasonably to stabilize RL training (see the analysis in 3B.ID. PURE offers several benefits: (1) it is simple implement, requiring only a transformation of process rewards without additional code changes; (2) it achieves comparable or better performance in PRM-based RFT compared to recent R1 Rlrrepiictiion studies with around 3x efficiency gains; and (3) it supports the integration of both dense process rewards and sparse verifiable rewards. When combining both reward types in the PURE framework, find that the auxiliary of few ground-truth signals further mitigates reward hacking caused by PRM.
In experiments, we first train a PRM using the PRM800K dataset (Lightman et al. 2023 human- annotated dataset that evaluates the correctness each step. Then we apply PURE framework to @wwen.55-Math-7.
models: Qwen2.5-70
and Qwen2.5- 1...5--t-h-1155 1.5B, using 3 reward configurations: only verifiable rewards, only process rewards, and a combination of both. This comprehensive setup allows to compare PRM-based and verifiable reward-based RFT approaches. We also compare sum-form and min-form credit assignment methods when enabling PRMs and find that the former even collapses training at the beginning. In the discussion, summarize observed reward hacking cases and analyze the causes of training collapse. Our key findings are outlined below.
1. Summation-form credit assignment easily induces LLMs to hack high-reward steps, leading to LLMs that prioritize thinking over problem-solving. Our min-form credit assignment avoids such reward hacking. (S (.11
2. PRM-based RFT achieves performance comparable the verifiable reward- based approach with around 3x efficiency gains if using our min-form credit assignment; otherwise, the training collapses even at the beginning! ($ 4.3)
3. The auxiliary of few ground-truth signals further mitigates PRM-induced reward hacking, achieving 82.5% accuracy on AMC23 and 53.3% average accuracy across MATH-500, Minerva Math, Olympiad Bench, AIME24, and AMC23 when using Qwen2.55Math- as the base model. ($ 4.3)
4. We identify types of PRM-induced reward hacking: (1) only thinking, not solving, (2) extremely few steps (1 step), (3) extremely few steps (0 step). We analyze causes, show examples, and provide solutions for each. ($ 5.I
5. Long, highly repetitive samples that the verifier ruled correct cause training collapse. These pseudo-positive samples, undetected by verifiers, provide numerous incorrect signals and collapse training all of sudden (within 5 gradient steps). (8 5.2
2 Preliminaries
2.1 Credit Assignment Problem in Reinforcement Learning
Minsky (2007) described the credit assignment problem as "how to distribute the credit of success among the multitude of decisions involved". In other words, credit assignment is the problem of

---

Subsequently, we introduce an advantage estimator tailored for estimating the inftuence of an action over an outcome from experience. Various assignment functions
have been proposed to quantify the influence of actions (Pignatelli et al., 2023). Here, we adopt the state-action value as the assignment function to detail the problem (Cheng et al. 2024 2025).
We model L.LM reasoning as step-level Markov Decision Process. Given prompt, L.M generates steps sequentially. At step t, LLM ग takes the prompt p and previous steps {01, 4x-1) J as the input state st, where 5, (p, 01, ... ac-1). 1). Then LLM generates a step dg, sampled as de ~ (- (-1st)- PRM I2 emits a process reward r (s, at). The canonical formulation of state-action value is:
Q" a1) =
Σ
(I)
where y is the discount factor. Eq. shows that the influence of action decreases over time, weighted by a discount factor y. Actions closer to future outcomes have greater influence. Thus, the state-action value quantifies credit assignment. With Eq. (1) we derive the advantage
function (Schulman et al. 2017, Shao et al. 2024 Ahmadian et al. 2024 Hu 2025) and update L.M
using the policy gradient loss.
2.2 Process Reward vs. Verifiable Reward
Verifiable rewards sparse, rule-based rewards assigned to entire response. For prompt. LLM generates a sequence of steps {ai..... ). After the final step on, a verifiable reward r" is assigned based whether the response matches the ground-truth answer. Verifiable rewards provide a straightforward, ground-truth signal for RL training, which has been commonly used in the training pipeline for advanced models (Yue et al., 2024 Guo et al., 2025 Team et al.) 2025). In contrast, process rewards [rī. ... .h are dense rewards to evaluate the quality of each step. Process rewards offer unique benefits in test-time scaling. During inference, LLM generates multiple responses for single prompt, and PRM scores each step for each response. Prior work (Lightman et al. 2023 Wang
al. 2023 Zhang et al. 2025) typically aggregates process rewards into a outcome-level score using the minimum value, min T1. rh)), and selects the best response based on this outcome-level score. This PRMM-bsed approach outperforms majority voting and other reward model-based approaches.
Although dense rewards support effective training in traditional RL, applications, they struggle to effectively fine-tune LLM for advanced reasoning Guo et al. (2025). Therefore, how make PRMs as effective in training as they are in test-time scaling is an important and not widely studied topic.
3 PURE: Process Supervised Reinforcement Learning
To effectively fine-tune LLMs using PRMs, we propose PURE:
Process sUpervised Reinforcement |Earning. PURE leverages a novel min-for minmform formulation quantify the credit assignment, inspired by the test-time application of PRM. In this section, we first detail the min-form credit assignment and analyze its effectiveness 3.1
process rewards in $ 3.4
3.1 Min-form Credit Assignment
Consider a prompt p and LLM и parameterized by 0, generating n-step response denoted as (01,'" ". a. ,.,)) At step f, LLM my takes the prompt and previous steps {ai,... , .4-1) as the input state 9;, where 8t (p, ai.-.. a₁ az-1). 1). Then LLM generates step ae, sampled a, TT(-)s)). A PRM Rh emits a process reward rf R (st, az). Unlike traditional RL whose objective is the summation of discounted rewards the usage of PRM in test-time scaling suggests that RL for
reasoning tasks should optimize the following objective:
arg max E [min(rf 2
(2)
where the expectation is over prompts and step-level actions. This objective implies: (i) Only the "worst"" step that gets the minimum process reward determines the value of the entire response. (ii) For steps before the worst step, their existence as the input state induces I.LM to generate the worst step. (iii) Steps after the worst step actually do not contribute to the objective.
3

---

Rollout
PRM
Sum-form
Min-form
leta 2+ v2 3.74, and let 2 31, the result when is rotated around d x/4 countor dodkwise. Find To find a wee need to miste amundd by ouuntendoddoiine This can be dane using the formula for reterion of's complex number amound another complee number
Process Reward
Sampling probability
Process Reward*
Sampling probability
Adv.
Adv.
0.97
2.54
1
0.04
-0.14
Firs, we find the vector from 1o which is c. Then, we rotzte this oeter by m/4 countend lecowtee using the formula (x- e).
0.75
1.57
0.04
where d - w/4
Finslly add back to the to frnd..
0.99
0.82
0.04
-0.23
Let's calextite
0.04
1x-e (2++ (3:4 SvEE))
3V8-
.0.53
-1.07
-0.11
-0.32
2. -os
0.70
-0.4
0.04
00.21
+ - (1 -4+2
-1244
-0.17
-0.25
4. c - (x-r)- =---- 12 12-30 -10- 20+0002-----
-0.22
0.54
-0.03
-0.08
Therefore, the final arsworld Siod
-0.32
0032
-0.05
-0.05
After RL updating
Question
Question
Let - 2- 22- (3 3-22), and lete- 2- XI, Let
Les -2 2-+ 22 (1+3322) 2222,, andiete- Letto be
mealt when mostee around by 1/4 monteedaechwtee Find
resut when in momeed seoundic by m// coumen-dockatee Rid
Response
Response
To Eand doclowiae.
around by
to resutt follow these stops
be done comple number around another complex number
farmala
I. Trunstate so is at the origit:
(2 V2 B+3))) - (2- b2 3
Rest, we find the vector from to ihishis i Then, we rotate this vector by s/4 contte-ducclvise wing the fermula ed (2- c) where e a/4.
2. Rotate the translated complex number by c/$ counter deedowiss. equitoeent to multipying it Mi/
Binally
back the result
The fins mewwer will plooe within Whreedd).
( 2Z . (ZT- 3.2) -......---
(a)
The response of the model after uddating using the form credit assigmment method. The model only output thinking steps without actually solving the problem!
<umit the mishway calnalation staps which are curmedt
TTrresaate the
original position:
(2-
Therefore, the result of the rotation S
(b) The respunse of the model after updating using the min form credit assigument method.
Figure 1: Comparison of summation-fome form and min-form credit assignment. Adv. and Process reward* in the table means advantage and transformed process reward, respectively. The incorrect steps in the rollout are highlighted in red, and our PRM reasonably assigns negative scores to these steps. For simplicity, advantage baseline and KL penalty terms are omitted in advantage calculation here, and discount factor y and transform temperature T are set 1. Arrows indicate changes in sampling probability, with larger changes marked by contoured arrows.
Therefore, define the min-form credit assignment function as follows: For r-step response, let step vo be the worst step, i.e., w = arg min(r""..... ... (rg)). The return G and state-action value Q" functions are defined follows:
G(s = min((?.
r ift < w
(3)
(0
ift
(" ac) = (G(
(4)
where r = (81, (91,41.77 T ... Sn, an r denotes the trajectory. Eq. (4 is quantitative expression of the above three-fold analysis of the objective. To implement this in the simplest way possible, we transform the process rewards in trajectory T using:
exp(-rf/T) 1 E exp( -r3//r T
where T' is the transform temperature, and r!"" is the transformed process reward. The transform function assigns higher weights to lower rewards. After transformation, the rest of code remains
unchanged. We explain this follows. As the transform temperature T 0+ Eq. 5 yields rpe r., for the worst step w and rf" = for i # w. Setting = 1, the return becomes: (i) For step t B< w.
n
G.
x - min(rr..... ... ,.r)
(6)
i=!

---

(ii) For / >> w, G. = O. The return exactly matches Eq. without altering the return computation logic.
Simple implementation. To align with widely adopted token-level PPO loss, we convert step-level rewards to token-level rewards. We exclusively assign the transformed process reward to the final token of each step, with all other tokens receiving reward of To preserve algorithmic flexibility, we also support verifiable rewards assigned to the last token of the complete response, which can be used alongside process rewards.
3.2 Quantitative Analysis
Here we analyze that the min-form has a smaller value estimation error than the summation-form credit assignment method. We start by first establishing reasonable assumptions:
Assumption 1 (Bounded Process Reward Error): The Process Reward Model, R produces reward estimate r = (-) for each reasoning step. It is assumed that there exists a "true" or optimal reward r for that step, and the estimation error is uniformly bounded by constant 2 0. For any
step é, it holds that !r? r21 S e.
Assumption 2 (Bounded Rewards): The true rewards, and consequently the estimated rewards, bounded. There exists a maximum possible reward Rmax such that [r?] - Rmax and Ir? S Rmax + for all steps I.
Theorem 1 (Q-Value Estimation Error Bound Comparison). Under Assumptions I and 2, for any
state-action pair (st, a4) and trajectory with reasoning steps:
Sum-form Error Bound: The estimation error for the summation- Q-value, is bounded by:
12aT (St, 0z) 1 (st,ae) S
For an infinite horizon (n oo), this bound becomes:
1Q (st, 04) 1aumm (8t, at))
$111-
2. Min-form Error Bound: The estimation error for the min-form Q-value, derived from Eq. (4). is bounded by:
1Qmin (St, (sr,aa)= Qmin. (st, az) S
The proof is provided in Appendix This theorem shows that the error bound of sum-form method is amplified by a factor dependent on the horizon length and the discount factor y, accumulating the error over steps. However, the error bound of min-form method is strictly limited by the single-step reward estimation error e and does not accumulate with the number of steps.
3.3 Qualitative Analysis
Min-form credit assignment restricts the range of value function to be the same as that of the reward function, which contributes to stabilizing RL training. In contrast, for the summation- credit assignment, the range of value is determined by the range of reward and the number of steps, causing excessive value as the number of steps increases and unintended reward hacking.
We now visualize the differences between summation -oomm and our min-form credit assignment using a real training example. As shown in Figure incorrect steps are highlighted in red, with arrows showing the magnitude and direction of changes in sampling probability. Steps with relatively large changes in sampling probability are marked by contoured arrows. Note that the first steps are the thinking, and the subsequent 6 steps are the solution.
Figure Ishows that nummation-form credit assignment significantly alters sampling probabilities, increasing the sampling probabilities for thinking steps and decreasing that for incorrect solution steps. However, this results shoteut towards reward hacking: the model learns to only output thinking steps without actually solving the problem! In other words, the model hacks the implicit pattern inside high-reward steps, i.e., thinking. contrast, our min-form credit assignment reduces sampling probabilities across all steps, with the largest reduction at the first incorrect step. This aligns with the behavior of verifiable rewards (an incorrect final answer leads a reduction in sampling probability of all steps) and assigns advantages more rationally based on step correctness.
5

---

3,4 Advantage Estimation
Following the token-level rewards 3.1 we are ready compute advantages. We compare several advantage estimators, including GAE (Schulman et al. 2017). RLOO (Ahmadian et al., (024). GRPO (Shao al. 2024), and REINFORCE+ (Hu 2025), discussed in $ F2 RLOO are strong enough to produce stable and effective results.
For single prompt, LLM generates K' responses. The maximum generation length is limited to N. Let ri and P (j = 1, N) denote the verifiable reward and token-level transformed process rewards for response y/s (i = 1, ... K), respectively. The token-level advantage for is formulated as follows:
N
N Ensi Ei= Σ (K 1)N
j== R
A
1 K
Kifi
j=b
token--evel baseline
RLO0 with verifiable reward
RLOO with token-level transformed process rewards
where = ..... .N. Specifically: (i) for verifiable rewards, we directly adopt RLOO. (ii) For process rewards, we employ token-level baseline to avoid reward hacking, discussed in the second case in $ 5.1 Moreover, we normalize the baseline with the max generation length N instead of response length avoid length biases, similar to concurrent work (Liu et al. 2025).
4
Experiments
4.1 PRM Training and Evaluation
To conduct our RFT experiments, we first require PRM to provide process rewards. We train our PRM based on Qwen2.5-Math-7B due to its strong performance in mathematical tasks. Follow- ing Lightman et al. 2023), we treat the PRM training as binary classification task. We replace the final layer of the model with a value head and train the model on the PRM800K dataset (Lightman) et al. (023) in 2 stages. In the first stage, we freeze the LLM parameters and train only the value head with a learning rate 10-4 for 3 epochs. In the second stage, we unfreeze the LLM parameters and fine-tune all parameters with learning rate 10 for epoch.
We evaluate our PRM, named PURE-PRM- 7B, through 3 ways: Best-of-N (BoN) method, ProcessBench (Zheng et al., 2024). and PRM- Bench (Song et al.) 2025). For BoN evaluation, we use rollout data from Xiong et al.2 For each question in GSM8K (Cobbe et al.) (021) and MATH (Hendrycks et al.] 2021). they uses Deepseek-7B to generate K 1024 answers. We score each step using our PRM, transform process
Table I: Results of BoN evaluation.
Rows
marked with & are taken from Xiong et al.
2024)
Method
GSMSK
MATH
Pass@I
83.9
42.4
Majority Voting @ 1024
89.7
57.4
Deepseck-PRM-DT BoN @ 1 1024
93.0
58.1
PURE-PREE PRM-7B BoN @ 1024
91.6
62.6
rewards using Eq. (5) and compute outcome score by summing the transformed rewards for each answer. The answer with the highest outcome score is selected as the final answer. Our PURE- PRM-7B achieves BoN @ 1024 accuracy of 91.6% on GSM8K and 62.6% on MATH, performing comparably to the best results of 93.0% and 58.1 % reported by Xiong et al. (0224)
On ProcessBench (Zheng et al. 2024), which assesses the PRM's ability to identify the first process error, our PRM achieves state-of -ftther average Fl score 57.5, surpassing the previous best Fl score of 56.5 reported in the benchmark. Detailed scores for each subset of ProcessBench are provided in Appendix C On PRMBench (Song et al., 2025), which evaluates the fine-grained error detection capabilities of PRMs, our PURE-PRM- ranks third among open-source PRM These results demonstrate that our PRM achieves top performance and is suitable for fine-tuning LLMs.
4.2 RL Settings
Reward types. Our framework supports 3 types of rewards: process reward only (PURE-PRM), verifiable reward only (PURE-VR), which matches the training setup of Deepseek RI-Zero, and mix
Thh official leaderboard is at autttpps:/pprmbbe www.hh....tgguoohho..o.o

---

Table 2: Detailed performance of various models across 5 benchmarks, Report pass@ @ accuracy tested with greedy decoding. The blue lines represent the models trained with our recipe.
MATH
Minerva Math
Olympiad Bench
Base Model
Method
AIME24
AMC23
Avg.
500
SimpleRL-Zoo
78.4
31.2
39.1
16.7
50.0
43.1
Base
71.4
23.9
35.3
10.0
52.5
38.6
Qwen2.5-7B
+ PURF-PRM
76.2
37.1
41.2
13.3
60.0
45.6
PURE-VR
75.6
32.7
39.0
16.7
55.0
43.8
PPURE--PPRER PR
80.4
37.9
43.0
16.7
60.0
47.6
Lurus-2-7B-PRIMM
79.2
38.6
42.1
26.7
57.8
48.9
SimpleRI
80.2
38.2
43.3
23.3
55.0
48.0
Owen2..5-7-DPP--ZZ
76.8
30.9
37.9
62.5
Owen2.5-Math-7B
Base
71.8
29.8
13.3
47.5
39.5
PURE-PRM
818
38.2
44.7
16.7
60.0
49.3
PURE-VR VR
79.4
36.8
41.8
23.3
60.0
48.3
82.6
37.1
44.1
20.0
82.5
53,3
Base
61.4
23.5
29.3
13.3
40.0
33.5
PURE-PRM
75.2
26.5
36.4
13.3
50.0
40.3
Lwen2.5-Math-L.58
PURE-VR
74.2
27.9
36.0
10.0
55.0
40.6
PURE-PRM+VE
76.0
31.6
37.2
16.7
55.0
43.3
of both (RMEE))RMVRR) For verifiable rewards, a reward of +1 assigned if the generated answer matches the ground-truth answer; otherwise, reward of 0 is given. No format-related rewards.
RL dataset. We use the RFT dataset from SimpleRL. (Zeng al. 2025b). samples around 8,000 problems from MATH dataset (Hendrycks et al. 2021) with difficulty Iv.3-5. For ground-truth (GT) answers, PURE-PRM method does not use GT answers. PURE-VE VR method requires GT answers for all problems. For PURE-PRM+VR. GT answers are randomly assigned to 1/10 of the problems, leading to about 800 problems with GT answers and 7,200 open problems. This setup aims to explore the effectiveness of process rewards as the main signal for RL training.
Hyperparameters. We use veRL (Sheng al. 2024) conduct experiments on 3 models: Qwen2.5- 7B (Yang et al. 2024a).. constant learning rate of 10 for PURE-VR and PURE-PRM+VR and 5 10 for PURE-PRM. Training steps are set to 500 for Qwen2.5-Math series and 1000 for Qwen2.5-7BB 5-B.. Other shared hyperparameters are set as follows: prompt batch size of 64, group size of (generating responses per prompt), training mini-batch size of 512, maximum generation length of 8192 tokens, sampling temperature of 1.0, KL coefficient checkpoints of 50 steps.
1ree2.5-MMath--
1ren2.5-Math-1
(Yang et al. 2024b). We use
transform temperature Eq of 0.1, save interval for
Baselines. We compare method with 3 state-of-the-art RFT methods: (i) Farus-2-7BBPRIIN (Cui et al. 2025): A model based on Qwen2.5-Maiie fine-tuned with implicit PRM. (ii) SimpleRL- Zoo (Zng al. 2025b): re-implemennttion of Deepseek Rl's training recipe on several base models. (iii) 2--....---PO--eer (Xiong et al. 2025 A model based on Qwen2.5-Mathe
fine-tuned with iterative DPO.
Evaluation metrics. At test time, we evaluate performance on 5 competition-less mathematical
benchmarks, including AIME24 et al.) 2024), AMC23 (Li al. 2024). MATH500 (Hendrycks)
et al. 2021), Minerva Math (Lewkowycz et al. 2022), OlympiadBench (He et al. 2024). We report of best checkpoint saved in training. During training, track about 20 metrics, including accuracy, reward, response length, repetition score, etc. Details are provided in Appendix D
4.3 Main Results
report the pass@1 accuracy across benchmarks in Table 2 The results indicate that all variants of our methods perform at least comparable to, better than baselines. For example, when using only verifiable rewards, following the setup of SimpleRL-Zoo, our method PURE-VR achieves average scores of 43.8 and 48.3 Qwen2.5-7B and Qwen2.5-Math-77. respectively. These scores are comparable to the baseline results 43.1 and 48.0 obtained by SimpleRL-Zoo confirming the reliability of code-base. Based Table we draw the following observation:
PRM-based approach performs similar the VR-based approach, but combining the two yields better results. On the Qwen2.5-Math-77 model, PURE-PRM achieves average score of
7

---

49.3, slightly higher than the 48.3 of PURE-VR, However, the combined method, PURE-PRM+Y reaches a score of 53.3, surpassing PURE-VR by approximately percentage points. This trend remains consistent across the other 2 base models.
Next, analyze the training dynamics of our methods understand the limitations of the PRM- based approach and explain why PURE-PRM-VR is superior. Figure illusttates the training curves for five variants of our methods on Qwen2.5-Math series, leading the following findings:
2
PURR--00 E/ PAM janns (arr))
PCRE FRM Sccta)
FURS 10R8-7704-VRR..1 pais femo
FURE-VEL
PURE FS2C. (mavdac))
PCLE FRM4 dorns)
FURE 4OUN (ecan-
Catcouse Reseun
0.722
0.70
69 MATRR-0
0.722
0.66
0.797
0.60
0.675
0.50
0,623
645
Picosss
0.531
130
37g
306
400
530
1.30
226
300
400
500
Training
Training
(a) PURE on Qwen2.5-Math-75
(b) Qwen2.5-Math-I.5t
Figure 2: Training curves for different variants of our methods on Qwen2.5- -Mtth series. Curves of PURE-PRM (sum-form) and PURE-PRM+VR (sum-form) are truncated due to training collapse. Process-aggregated outcome reward is the summation of final process rewards for one response: for sum-form, sums PRM-emitted rewards; for min-form, sums transformed rewards, approximating the minimum PRM-emitted reward. Thus values across the 2 credit assignment methods are not comparable. For PURE-PRM, verifiable reward is logged but unused in training.
Summation-form credit assignment collapses training even the beginning, while the min- form method significantly enhances training stability. As shown in Figure 2a. both PURE-PREP (sum-form) and PURE-PRM+VE (sum-form) experience collapse at step 25. At step 80, their average benchmark scores drop to around 30, which is much lower than the base model's 's score of 39.5. In contrast, the min-form methods remain stable over 200 steps, achieving average scores of 49.3 and 53.3 for PURE-PRM (min-form) and PURE-PRM+VR (min-form), respectively.
Dense rewards substantially improve learning efficiency compared to sparse rewards. Figure 2 shows the curves of accuracy on MATH-500 for 3 variants of our methods. We find that PRM-involved approaches takes around 30% of the training steps to achieve the same accuracy as PURE-VR.
Reward hacking is inevitable when rely solely on PRM, though it can be delayed through algorithmic adjustments. While PURE-PRM (min-form) produces well-tuned model, reward hacking still occurs at step 270, where verifiable rewards decrease sharply while process-aggregated outcome rewards increase in Figure ph. This issue was observed in all experiments using the PURE- PRM method, indicating that reward hacking is a persistent challenge when relying solely on PRM and limits further progress. However, compared sum-form credit assignment, min-form approaches delays the onset of reward hacking significantly. Such reward hacking corresponds to the third case in 5. which is inevitable due the current architecture of PRMs.
Incorporating a few ground-truth signals can effectively reduce reward hacking. PURE- PRM+VR (min-form) uses mixed dataset of 800 problems with GT answers and 7,200 open problems. Thus, during training, process rewards guide the RFT process primarily, while verifiable rewards serve as auxiliary signal. As shown in Figure this approach achieves stable training process similar to PURE- VR and attains the highest average benchmark score of 53.3. This suggests practical solution to address reward hacking by including small proportion (around 10%) of ground-trutt -ouun signals during the RFT stage.
Analysis
5.1 Reward Hacking induced by PRM
In this section, we detail the reward hacking cases observed during training. We categorize these cases into the types. Specific examples for each type provided in Appendix B
8

---

Case 1: Only thinking, not solving. This behavior occurs when steps with certain patterns, such as thinking, are rewarded significantly more than others. and the inappropriate credit assignment further widen the gap in advantages between steps. As result, the model learns to exploits these patterns to achieve higher scores. As shown in Figure [] the sum-form credit assignment increases the likelihood of generating thinking steps while reducing that of incorrect solution steps, resulting in an armchair general. However, exploiting specific patterns is not always negative. For example, using patterns like backtracking and verification improves reasoning skills (Guo et al. 2025 Gandhi et al. 2025).
Case 2: Extremely few steps (1 step). This happens when an unsuitable advantage baseline used. In $ $3.4, we employ a token-level baseline for process rewards. An alternative approach is a step-level baseline, formulated as follows:
N
Σ /
A.. r
(8)
K
K
light
j=!
light
where Iysls represents the number of steps of the k-th response within the group. The advantage baseline for process rewards in Eq. (8) means the average process reward of other responses in the group. However, we find the step-level bascline is biased against the number of steps. When the process-aggregated outcome reward are equal, response with more steps are penalized more heavily by the bascline, causing the model to favor responses with fewer steps. Eventually, the model learns to output only single step with excessively large number of tokens. This undermines the purpose of PRM, which is to evaluate the process step by step.
Case 3: Extremely few steps (0 step). In this case, the model learns to output irrelevant responses such as "Thank you.", "Happy Birthday.", or even empty responses. Since PRM infers in a causal manner, it assigns high rewards to these meaningless steps, not realizing that no further content follows. This issue stems from the current architecture of discriminative PRMs (i.e., the causal attention mask). Our solution is adding GT-level signal aid, such verifiable rewards in PURE-PRM+VR. Another potential solution is the generative PRMs, which we leave for future work.
10-1
04
Truzing Steps
Trunzig Steps
Trussng Steps
0.75
365
Proos--Opeen Sopeegapeee Outcconn
MMN
My WM
Aorusay in MATEE600
0,300
Welfbale Bruard
0.49
320
300
340
300
300
320
300
300
3.00
302 300
350
445
Fruscing Staps
Eranzing Stags
Traauin Steps
Figure 3: Training curves for 7B. The correctness of responses are judged by the verifier. Process- agggregaee outcome reward (bottom-left) is the summation of transformed process rewards for one response. No smooth is applied. Training collapses at step 365, showing a sharp drop of rewards and accuracy.
with doubled process rewards based on Qwen2.5-
5.2 Causes of Training Collapse
In this section, we train Qwen2.5-7B using PURE-PRM+VR with doubled process rewards, and show the training curves of pettern-related metrics in Figure 3, including response length, clip ratio, repetition score, rewards, and accuracy on MATH-500. The clip ratio measures how often responses are cut off due to the maximum generation length limit. The repetition score evaluates how repetitive
9

---

response is by calculating the longest common prefix (LCP) lengths between all pairs of its suffixes We consider responses with repetition score above 0.2 as highly repetitive. Details on metries are provided in Appendix D
As shown in Figure 3 training collapses step 365, showing sharp drop of rewards and accuracy. Before step 360, incorrect responses judged by the verifier are longer and more repetitive than the correct responses, but that does not collapse the training. Both the clip ratio and high repetition ratio for correct responses remain at 0 until step 361. After that, the response length, clip ratio, and high repetition ratio for both incorrect and correct responses rise sharply until step 380. Based on these observations, we derive the following conclusions:
Long and highly repetitive "correct" responses cause training collapse. At step 361, the clip ratio and high repetition ratio for correct responses become greater than 0. This large number of positive signals for repetitive patterns is fatal to training. We refer to these samples as pseudo-posithe possiivve samples. We attempt to treat pseudo-positive samples as incorrect samples and assign O rewards instead of +1 rewards, but this does not help much. The model learns repeat content in ways that the LCP function can not detect, as detailed in Appendix D Verifiers can not identify pseudo-positive samples. While PRM has the potential to detect them, it currently fails do so because such patterns are not included in its training data. This highlights a need for future PRM development to not only assess the correctness of steps but also evaluate the quality of patterns.
The model learns repetitive patterns quickly, leading to collapse within 5 gradient steps. From the unusual metrics at step 361 to the training collapse step 365, the model rapidly learns repeat content. Our training setup ensures one gradient step per training step, meaning the collapse happens within just 5 gradient steps.
6 Conclusion
Process sUpervised Reinforcement |Earning. framework leverages process rewards to improve the reasoning abilities of LLMs. Extensive experiments demonstrate that PURE effectively alleviates reward hacking induced by PRM, attributed to the proposed min-form credit assignment. This method allows PRM-based RFT to achieve performance similar to verifiable dwvards-based approaches within 30% steps, and further outperform them with the assistance of few ground-truth signals. Additionally, we summarize the observed reward hacking cases during training. and find that pseudo- posuiivv samples collapse training.
In this paper, we present PURE:
There are several promising directions for further improving PRM-based RFT. First, developing generative PRMs is both urgent and crucial. As discussed in $ current PRMs are unable to address the third type of reward hacking mentioned in $5.T] and also cannot evaluate the quality of patterns like endless repetition. Generative PRMs, however, could potentially resolve these issues by making better use of the strong language capabilities of LLMs. Second, iterative training between PRM and LLM is essential to ensure that the PRM continuously adapts to the output distribution of LLMs.
Acknowledgments
We sincerely thank Ganqu Cui, Songjun Tu, and Zhengbo Wang for their suggestions to the early draft of the paper. This work was partially supported by the National Science and Technology Major Project (2022ZD01 17102), the National Natural Science Foundation of China under Grants 62271489 and 62303462, Beijing Natural Science Foundation under grant 1.241016 and L.233005, Chongqing Transportation Technology Project (CQIT-CZKJ20224-MM the Provincial Key Research and Development Program of Zhejiang (project number: 2022C01 129). Ningbo International Science and Technology Cooperation Project (2023H020), CCF-Tencent Rhino-Binn Open Research Fund.
References
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzich Fadace, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basies: Revisiting reinforce style optimization for learning from human feedback in 1lms. arXiv preprint 20Xiv:2402.147700, 2024.
This is implemented using function from Span-eReao Hu et al. 2025
10

---

Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
Jie Cheng, Gang Xiong, Xingyuan Dai, Qinghai Miao, Yisheng Lv, and Fei- Yue Wang. RIME: Robust preference-basee reinforcement learning with noisy preferences. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 1122-8247.. PMLR, 21-27 Jul 2024. URL httpps://procseddng.. mlr. prwss/ v235/ C2333//hheg224.1br html
Jie Cheng, Ruixi Qiao, ma yingwei, Binhua Li, Gang Xiong, Qinghai Miao, Yongbin Li, and Yisheng Lv. Scaling offline model-based rl via jointly-optimized world-action model pretraining. In Y. Yue, A. Garg. N. Peng. F. Sha, and R. Yu (eds.), International Conference on Representation Learning. volume 2025, pp. 42396-42418, 2025. URL ott.://aaeeeaaaeeeeeaaee----eeee
iclr cc.ppprrrrcpppprtfiilllsss files/
pdf
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2 10. 14168, 2021.
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang. Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arXiv preprint wxXiv:2002.01456, 2025.
Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cognitive behaviors that enable self- improving reasoners, or, four habits of highly effective stars. arXiv preprint arXiv:2503.01
2025.
Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. rstar-math: Small Ilms master math reasoning with self-evolved deep thinking, 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12944 2025.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu. Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-lewe bilingual multimodal scientific problems. arXiv preprint 14008 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint
2021.
Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501.03266, 2025.
Jingcheng Hu, Yinmin Zhang, Qi Han. Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Open-reasoner-zzero: An open source approach to scaling up reinforcement learning the base model, 2025. URL httpttttt//aai.....ooo org/ htrg//bb/25003 .42900
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Cameyy, al. Openai ol system card. arXiv preprint arXiv:2412.16770, 2024.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857, 2022.
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13:9, 2024.
11

---

Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards. Bowen Baker, Teddy Lee, Jan Leike, John Schulman, llya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zooo- training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025.
Marvin Minsky. Steps toward artificial intelligence. Proceedings of the IRE, 49(1)88-3,, 2007.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Eduardo Pignatelli, Johan Ferret, Matthieu Geist, Thomas Mesnard, Hado van Hasselt, Olivier Pietquin, and Laura Toni. A survey of temporal credit assignment in deep reinforcement learning. arXiv preprint arXiv:2312.0107.. 2023.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 1--53728-537411 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 2017.
Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal,
Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated process verifiers for Ilm reasoning. arXiv preprint arXiv:2410.008: 06646 2024.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint 1xXiv:2409.192550 2024.
Mingyang Song. Zhaochen Su, Xiaoye Qu, Jiawei Zhou, and Yu Cheng. Prmbench: A fine-grained and challenging benchmark for process-level reward models. arXiv preprint 2rXhv:2501.03124, 2025.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint 2.X.::2501.125999 2025.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Ilama 2: Open foundation and fine-tuned chat models. arXiv preprint arXi::2307.09288, 2023.
Bin Wang, Bojun Wang, Changyi Wan, Guanzhe Huang, Hanpeng Hu, Haonan Jia, Hao Nie, Mingliang Li, Nuo Chen, Siyu Chen, al. Step-3 is large yet affordable: Model-system co-design for cost-effective decoding. arXiv preprint arXiv:2507. / 19427, 2025.
Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui Math-shepherd: Verify and reinforce lims step-by-step without human annotations. arXiv preprint arXiv:23/2.08933 2023.
Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida I Wang. Swe-rl: Advancing IIm reasoning via reinforcement learning on open software evolution. arXiv preprint arXiv:2502.18449 2025.
Lilian Weng. Reward hacking in reinforcement learning. fitaawennggithuube Nov 2024. URL
11tp//2222111122 28--28--ee rewwrre -ackarn///akkng//
Wei Xiong, Hanning Zhang, Nan Jiang, and Tong Zhang. An implementation of generative prm. stttps://gitan com ctwm/om//////e ///H/RLFoooz
-odewardd-odeeinng 2024.
Wei Xiong, Hanning Zhang, Chenlu Ye, Lichang Chen, Nan Jiang, and Tong Zhang. Self-rewarding correction for mathematical reasoning, arXiv preprint arXiv:2502.196.3 2025.
12

---

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang. Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. 2024a.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-man technical report: Toward mathematical expert model via self-immrovementt arXiv preprint arXiv:2409.12 12122, 2024b.
Lifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kaiyan Zhang, Bowen Zhou, Zhiyuan Liu, and Hao Peng. Free process rewards without process labels. arXiv preprint ar arXiv2241..09981, 2024.
Tongtian Yue, Jie Cheng, Longteng Guo, Xingyuan Dai, Zijia Zhao, Xingjian He, Gang Xiong. Yisheng Lv, and Jing Liu. Sc-tune: Unleashing self-consistent referential comprehension in large vision language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13073-13083, 2024.
Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, et al. Glm-4.5: Agentic, reasoning, and coding (arc) foundation models. arXiv preprint arXiv:2508.0644
2025a.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- 700: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503. /8892, 2025b.
Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons of developing process reward models in mathematical reasoning, arXiv preprint arXiv:2501.0730/ 2025.
Chujie Zheng, Zhenru Zhang, Beichen Zhang. Runji Lin, Keming Lu, Bowen Yu, Dayiheng Liu, Jin- gren Zhou, and Junyang Lin. Processbench: Identifying process errors in mathematical reasoning.
arXiv preprint urXiv:2412.06553 2024.
A Related Work
A.1 Reinforcement Fine-Tuning
Reinforcement fine-tuning (RFT) is promising technique to improve the performance of LLMs (Ouyang et al. 2022 Yue et al. 2024 Wei et al. 2025). OpenAl's ol model (Jaech et al. 2024)) was among the first demonstrate the significant potential of large-scale reinforcement learning for enhancing reasoning abilities in LLMs. Recent studies have further confirmed that a straightforward
reinforcement learning approach, using verifiable rewards, can scale effectively (Guo al. 2025
Team et al.) 2025 Zeng et al. (0225). However, previous research has faced difficulties in effectively utilizing PRM (Guo et al. 2025) in training-time which the primary focus of our work in PURE.
A.2 Reward Models for LLM Alignment
In the area of LLM alignment, outcome reward models (ORMs) are used to evaluate the quality of the entire response generated by LLMs. Employing ORMs for aligning LLMs with human values has become a common practice the post-training stage (Ouyang et al. 2022 Bai et al.) 2023 Touvron et al. 2023). However, ORMs fall short in providing detailed feedback for complex tasks that involve multiple reasoning steps. In contrast, process reward models (PRMs) provide more detailed feedback by evaluating each step of the reasoning process, allowing LLMs to learn more effectively from mistakes made during reasoning (Lightman et al.) 2023 Yuan et al. 2024 Cui et al., 2025). Studies have shown that PRMs outperform majority voting and ORMs in test-time scaling (Lightman et al.)
2023 Wang et al.] 2023: Xiong et al., 2024). Nevertheless, the application of PRMs during the training phase remains largely unexplored, which is the central topic of PURE.
B Proof of Q-value estimation error
Theorem 2 (Q-Value Estimation Error Bound Comparison). Under Assumptions I and 2, for any state-action pair (st, az) and trajectory with reasoning steps:
13

---

1. Sum-form Error Bound: The estimation error for the summation-form Q-value, is bounded by:
1Qam (st, oz) -Q7 -. (SL, ail S
1=
For infinite horizon (n 20), this bound becomes:
1 (st, 0z) Q (st, [4)l== -st
1 1-7
Min-form Error Bound: The estimation error for the min-form Q-value, derived from Eq. (4) is bounded by:
1Qmin (8c, Quinn a,) S
Proof. Part I: Proof for the Sum-form Error Bound.
We begin with the definition of the absolute error of the Q-value, using the canonical formulation in Eq. (1). Q denotes the value calculated with the estimated rewards from the PRM (r²), and Q denotes the true value (calculated with r*).
1Q (st, az) -Q Qsum. = |mTTT
Σ
Exnt
Σ 7 rr]
(
E
(r? - r1)1l
[+n-1 1
< Earil
7- rr)ill
1=t
H+n-1
< P
In
-1)1
i=1
Hn-1
E
Σ
Now, we apply Assumption 1, which states that the reward error at each step is bounded by e, i.e. Ir? ril Se
in-11 Σ x
Q (st, a.) Qrum. (st, (8), & E
Σ
7 [-
=E
Amf
For the infinite horizon case, the geometric series converges
I-
Part Proof for the Min-form Error Bound.
We start similarly with the definition of the absolute Q-value error for the min-form. The return G(st, aelT) is the value being expected.
mia (st, (t) - Qmi.. (St, at)] |E ar [7]]
Ex-r (ot, ar|T]]
En EE (T)T)= G* (st, ar/T]]
Now we analyze the return term inside the expectation. According to Equation (3), for a trajectory and step I, the return is defined the minimum of future rewards.
G(st, aelT) min(rž, ret.....,. rF)
(st, aelr) min(r), 1441...... r"
We use key property of the minimum function: mmnn(a) min(b) < max; la; - bbl. Applying this property: [G(s,, (elT) G (st, axlT))
min (r?) min (r?) max Ir? - r S i=t..m
t=...
14

---

Substitute this result back into the expectation inequality:
1 (st, a4) - Qmin. (s2, = [G(,, G (st, ag dg(T)]] S B =e
This completes the proof. It is proven that the error the min-form does accumulate and is directly bounded by the maximum single-step error of the reward model.
C Benchmark Scores of Our PRM
We report the detailed scores ProcessBench (Zheng et al., 2024) and PRMBench (Song al. 2025 in Table and Table respectively.
Table Fl scores of each subset in ProcessBench. The blue line represents our PRM. Other lines are taken from Zheng al. (2024).
Olympiad- Bench
Omni- MATH
Model
GSM8K
MATH
Average
Math Baath-Sepherd-PRMMEE
47.9
29.5
24.8
23.8
31.5
C.LHFlow-PRM-Deepaee Deeeseek-88
38.8
33.8
16.9
16.9
26.6
Skywork-PRM-7B
70.8
53.6
22.9
21.0
42.1
1aan2-5-Math77-PRMMBOI
68.2
62.6
50.7
44.3
56.5
PURE-PRM-7B
69.0
66.5
48.4
45.9
57.5
Table 4: Results on ProcessBench. The blue line represents our PRM. Other lines are taken from
Song et al. (2025).
Model
S1 (Simplicity)
S2 (Soundness)
S3 (Sensitivity)
Overall
MathShhephen--Mistrall
47.1
45.7
60.7
47.0
SHHFllow-PRM-Deepeeede
47.6
57.5
68.1
54.2
Skywork-PRM-778
59.6
68.5
73.3
65.1
Swen2.5-Math-7B-PRRSSS-
52.1
71.0
75.5
65.5
PURE-PRM-7B
52.2
70.2
75.8
65.3
D Details of Training Metrics
We detail 4 metrics used the main text of the paper, including process- aggregated outcome reward, clip ratio, repetition score, and high repetition ratio.
Process-aggregate outcome reward. This metric reflects the value of response with respect to process rewards. In practice, calculated as the sum of transformed process rewards for the min-form credit assignment, approximating the minimum PRM-emitted reward for given response.
Clip ratio. The clip ratio indicates the proportion of samples in the replay buffer that are truncated due to the maximum generation length limit. After training collapses, LLMs often generate infinitely repetitive responses until hit the maximum context length.
Repetition score. The repetition score measures the level of repetition in a response by calculating
the longest common prefix (LCP) lengths between all pairs of its suffixes. However, this method can only identify basic repetition patterns. A more complex form of repetition involves the use of near-synonyms, where words differ but the meaning remains the same. Such cases are not detected by the LCP method. This limitation explains why treating pseudo-positive samples as incorrect still results in training collapse, as mentioned in (52 The model learns to exploit complex repetition patterns to bypass the LCP function.
15

---

High repetition ratio. We consider responses with a repetition score greater than 0.2 as highly repetitive, This metric shows the percentage of samples the replay buffer that fall into this category.
E Examples of Reward Hacking
We show examples for each reward hacking case in Figure and 6 In practice, double line breaks (")n)n") to separate individual steps within the response.
Question
Compute cos
sin
sin240"
Response
Let's break down the problem step by step:
Step 1: Express the Given Expression Using Trigonometric Identities
Rewrite each term of the expression using identities like cas sec and express them in terms of tan2x and cot2x to simplify the structure.
csc and further
sin
Step 2: Simplify the Expression by Combining Constant Terms
Combine any constant terms resulting from the identities used in the previous step to reduce the expression to more manageable form involving trigonometric functions.
Step 3: Convert Cotangent Terms to Tangent Terms
Use the relationship cot = tanx to express cotangent squares in terms of tangent squares, aiming for consistency in the trigonometric functions used.
Step 4: Explore Angle Relationships Using Multiple-Angle Identities
Consider relevant multiple-angle identities (such for tan 3x) to establish equations or relationships between the angles involved, potentially leading to simplification the expression.
Step 5: Relate Cotangent to Tangent Using Complementary Angle Identities
Use identity cot x tan(90" x)to rewrite cotangent terms as tangent terms complementary angles, facilitating a possible pairing or symmetry in the expression.
Step 6: Apply Known Trigonometric Sum Identities or Results Identify and apply any known results or specific identities cotangents for the given angle configuration to evaluate the expression directly.
the sum squares of tangents and
Step 7: Combine Results Compute the Final Value
Add the results from the simplified trigonometric sum to the constant terms obtained earlier to arrive at the final numerical value of the expression.
Figure 4: Reward hacking, case 1: only thinking, not solving. In this example, the L.M analyzes the problem and gives a few equations for trigonometric simplifications. but does not substitute actual numbers to calculate and solve the problem. This is because the LLM hacks the implicit pattern inside high-reward steps, i.e., thinking.
F Additional Experiments
Ablation on Transform Temperature
In this section, we ablate the transform temperature defined Eq. (5). which controls the level of approximation to the min-form credit assignment. We choose Qwen2.5-7B as the base model and run PURE-PRM+VE with 3 values of transform temperature individually for 230 steps. The results are shown in Table and Figure which indicates that 0.1 is the best for the transform temperature.
16

---

Question
In A,QR we have PQ QR 34 and PR 32. Find the length of median (?M.
Response
Let's first identily the type of triangle use the properties the median an Isosceles triangle. Given that PQ = QR APQR isnsceles with PQ QR. In isosceles triangle, the median to the base (the side opposite the vertex angle) also acts as the altitude and the angle bisector of the vertex angle. So, the median QM from vertex Q to the midpuint M the base PR will split the isosceles triangle into congruent right triangles.
Let's place the triangle
coordinate plane to make calculations easier. Assume = (0,b). = (0, 0), and R (32, (lere,,
PQ QR 34 and PR
Since PQ 34 and @ (0,0), point P at distance 34 units from the origin along the y-axis So, the coordinates of are (0, b),
and the distance from
the z-oordinntee which Therefore, have:
34
coordinates The coordinates of 0 are (0, 0) and the courdinates R are (32,0).
(0, 34).
Next, coordinates The midpoint of the base PR (where P (4) and
midpoint M
(32, 03) is given by the midpoint formula:
+ 2
+2)
M
2
Hern, 34) and (2g.
M= +2
( (16.17)
So, coordinates the midpoint are (16, 17).
Then, let's calculate the length the median QM.
The median QM the line segment from Q = (0, 0) to = (16, The length of the medlan can be found using the distance formuls
between two points (1) and (#2:)2))
Distance
|0r x,
Here, the points are Q (0,0) and M (16, 17)
QM
(16
0)² + (17 03
v16 + 172
Calculate 16 17*2:
16 256
17a 289
So,
QM
V256 289 V545
Therefore, the length of the median QM
+545
Figure Reward hacking, case 2: extremely few steps (1 step). practice, we divide steps according to double line breaks "In),"" and then PRM scores each step. When the advantage baseline is inappropriate, such step-level baseline discussed in $ [5.I] the model learns to deliberately avoid outputting "nn)n", preferring short-step response. In this example, there is no "(n)n" character in the generated response, resulting in the entire response being split into only one step.
Question
Evaluate the expresston v80 4v5
1no
and express vM, where M Integer.
3
Response 1
Thank you.
Response 2 (empty response)
<eos>>
Response
I can help you solve this problem.
Figure 6: Reward hacking, case 3: extremely few steps (0 step). This is the most common cases for PURE-PRM. When relying solely on the PRM, training eventually boils down to this case after numerous steps of training. It is inevitable because the PRM scores based the question and historical steps, and it does not know the role of the current step in the overall response.
17

---

0.1
T 0.01
T
0.78
Accuray MATH-5500
0.76
0.72
0.70
150
Training Steps
Figure 7: Ablation on transform temperature defined in Eq. (5) We use PURE-PRM+VR to fine-tune Qwen2.5-7B in this experiment.
Table 5: Results of PURE-PRM+VR with different transform temperature, defined in Eq (5) We conduct the experiments based on Qwen2.5-7B and report pass@ accuracy tested with greedy decoding.
MATH
Minerva Math
Olympiad Bench
Method
AIME24
AMC23
Avg.
500
Base
71.4
23.9
35.3
10.0
52.5
38,6
PURE-PRMAVR (T 0.01)
76.2
37.1
39.0
6.7
55.0
42.8
PURE-PRM+VR (T = 0.1)
76.2
37.1
41.2
60.0
45,6
+ PURE-PRM-VR (T = 1..0)
75.8
33.5
38.7
13.3
52.5
42.8
F.2 PURE with Other RL Algorithms
In this section, apply PURE with various advantage estimators beyond RLOO, including GAE, GRPO, and REINFORCE++. The training curves are shown in Figure To make different algorithms compatible with the compound of verifier rewards and process rewards, make adjustments similar to those Eq. For GRPO, the advantage defined as:
NN
P4 i.j
mean EE Ej= 20-1.
r mean(r"))
j==
A
sid(rv)
std
EN 1i=1
0.00
Vertfable Reward
0.50
0.70
MATH-500
0.63
0.50
0.00
0.45
0.55
0.40
0.50
0.35
Training Steps
Teaining Stage
Figure 8: Training curves on Qwen2.5-7B using PURE-PRM+VR with different advantage estimators.
18

---

where mean and standard deviation (std) are calculated over K responses in a group. For REIN- FORCE++, the advantage is:
A ri
(10)
From Figure we find that all methods show similar performance before step 150. However, REINFORCE++ experiences spike at around step 220. Although GAE converges bit slower than others, it performs slightly more stably in the last 50 steps. Due its additional learnable value network, GAE takes about 30% more time for forward and backward passes. Considering performance, training time, and stability, we select RLOO as our preferred advantage estimator.
19

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
See abstract, section
I and
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
20

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See section appendix, and supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer: [Yes]
Justification:
See supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/ddeeeeee Cooeeeoodeesssseeeeeee //nips.com htte://alllllliiiee/e//eeeuuaasl ssiniiiiccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
See section and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: All methods tested using greedy decoding. Thus given checkpoint, its benchmark scores are deterministic.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
22

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See section and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Question:
Answer: [Yes]
Justification:
The paper complies with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Both the base models from Qwen and dataset properly credited.
Guidelines:
existing assets. The authors should cite the original paper that produced the code package or dataset.
The answer NA means that the paper does
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
25

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy fattps://neurd/: /00p://ccc...cc.eeeee//////////0 for what should or should not be described.
26


Reviews:
[{"review_id": "sAADjF9Q00", "reviewer": "Reviewer_GZwS", "summary": "The paper proposes PURE, a min-form credit assignment that formulates the value function as the minimum of future rewards. This method significantly alleviates reward hacking by limiting the value function range and distributing advantages more reasonably. Authors test PURE on 3 different base models and demonstrate superiority against the sum-form credit asssignment which they claim collapses training. The paper discusses observed reward hacking cases and also provide an analysis on the training collapse.", "questions": "1. How does the min-form credit assignment behave when the future rewards in a reasoning trace are consistently very low or zero for several steps, and does this lead to specific challenges or biases in the learning process?\n\n2. How would the PURE framework perform in real-world applications where PRMs might be more noisy or less reliable in their reward assignments compared to the experimental setup?\n\n3. Beyond reasoning tasks, could the min-form credit assignment principle be effectively applied to other fine-tuning scenarios (e.g., instruction following) where process-based feedback might be available?", "limitations": "Please refer to weaknesses section.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The paper is nicely motivated and clearly discusses where and how the baseline approaches struggle. The experiments are diverse and satisfactory to convince the claims made in introduction. \n\n2. The min-form assignment captures the limitation of process rewards in a simple yet effective manner. The implementation is simple, and aligned with the commonly used token-level PPO loss. The effectiveness of this simple approach is captured well in both qualitative and quantitative results. \n\n\nWeaknesses:\n\n1. While the paper clearly defines reward hacking in the context of summation-form credit assignment (exploiting high-reward steps), it doesn't explicitly discuss whether the min-form approach might introduce other, subtler forms of exploitation or biases in the learning process, particularly if the minimum reward is consistently low across many steps.\n\n2. The paper demonstrates success on reasoning tasks, but the complexity and length of these reasoning chains are not fully detailed. It would be beneficial to understand how the min-form approach scales to extremely long and intricate reasoning processes where maintaining a minimum reward across all steps could be challenging.", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer GZwS,\n\nWe sincerely appreciate your valuable and insightful comments as they are extremely helpful for improving our manuscript. We have addressed each comment in detail in the paragraphs below.\n\n---\n\n> **[w1]** While the paper clearly defines reward hacking in the context of summation-form credit assignment (exploiting high-reward steps), it doesn't explicitly discuss whether the min-form approach might introduce other, subtler forms of exploitation or biases in the learning process, particularly if the minimum reward is consistently low across many steps.\n\n**[A1]** Thank you for raising this important point. In Section 5.1, we document all observed cases of reward hacking across our experiments. While summation-form credit assignment exhibited all three hacking cases, **min-form credit assignment only manifested the third type** – identical to the case at step 361 in Section 5.2. Specifically:\n\n> *The model only generates meaningless responses (e.g., \"Thank you\", or empty outputs “”).*\n\nThis occurs because discriminative PRMs assign high rewards to such irrelated response steps, not realizing that no further content follows – a fundamental limitation of current PRM architectures (i.e., the causal attention mask). Crucially, **this behavior is independent of the RL algorithm** and affects *both* min-form and sum-form credit assignment methods.\n\nTo mitigate this, we use verifiable rewards as auxiliary guidance with proper reward coefficients:\n\n- Effective configuration: 1.0 * verifiable rewards + 1.0 * process rewards, used in main experiments (Sec 4.3)\n- Failure case: 1.0 * verifiable rewards + 2.0 * process rewards → collapse (Sec 5.2)\n\nRegarding your concern about persistent low minimum rewards, we provide a detailed analysis in **[A3]** below.\n\n---\n\n> **[w2]** The paper demonstrates success on reasoning tasks, but the complexity and length of these reasoning chains are not fully detailed. It would be beneficial to understand how the min-form approach scales to extremely long and intricate reasoning processes where maintaining a minimum reward across all steps could be challenging.\n\n**[A2]** Thank you for highlighting this. To understand the complexity and length of the reasoning chains, we logged the response length and frequnencies of reflection patterns (e.g., “wait,”, “recheck”, “retry”, “alternatively,”, “let's verify”, and so on). For Qwen2.5-Math-7B, the average response length increases monotonically from 750 tokens to around 2000 tokens over 1000 training steps. The average reflection frequency increases from 0.3 to 0.9 in the first 700 steps and fluctuates around 0.9 for the final 300 steps. Following NeurIPS rebuttal policies prohibiting links and images this year, we will include full training dynamics figures in the appendix. These cover nearly 20 metrics spanning sample-level (verifiable/process rewards, returns, advantages), group-level (pass@n, avg@n), benchmark-level (MATH-500, AIME24), pattern-related (repetition score, reflection frequency, response length), stability-related (actor grad norm, response clip ratio), and system metrics (iteration/generation time, step count), etc.\n\nRegarding ultra-long reasoning processes (such as average response length > 10k), this remains challenging for both PRM-based and VR-based methods, representing an ongoing research focus for our team. The primary obstacle is not the RL algorithm itself but rather scoring inaccuracies from PRMs. Since the PRM800k dataset – the largest open-source PRM training data – lacks such long responses, its out-of-distribution characteristics lead to unreliable step-level scoring for ultra-long responses. We are actively addressing this through manual labeling of process rewards for 100k responses (generated by deepseek-R1-0120) and dedicating approximately 1,000 H200 GPU hours for training. **Importantly, our current observations indicate that for ultra-long reasoning, min-form credit assignment still demonstrates greater stability than summation-form.** Specifically, as step counts increase, the summation-form method consistently exhibits reward hacking at certain steps, as if its training had “ten thousand” ways to fail.\n\n---\n\n> **[Q1]** How does the min-form credit assignment behave when the future rewards in a reasoning trace are consistently very low or zero for several steps, and does this lead to specific challenges or biases in the learning process?\n\n**[A3]** Thank you for raising this concern. Consistently low-reward steps do not inherently disrupt training because **the advantage function - not the raw rewards - determines the magnitude and direction of policy updates** (see the PPO loss). For advantage estimators like RLOO and GRPO, the advantage is computed as the reward minus a baseline that depends on other samples within the group. Thus, even if certain steps receive low rewards, their advantage values won't necessarily approach zero unless all samples in the group share identical rewards (i.e., all correct or all incorrect). In such cases, standard practice involves removing these grouped samples (i.e., dynamic sampling proposed by DAPO [1]). Therefore, consistently low rewards across multiple steps do not impede the overall training process.\n\n[1] Yu, Qiying, et al. \"Dapo: An open-source llm reinforcement learning system at scale.\" *arXiv preprint arXiv:2503.14476* (2025).\n\n---\n\n> **[Q2]** How would the PURE framework perform in real-world applications where PRMs might be more noisy or less reliable in their reward assignments compared to the experimental setup?\n\n**[A4]** Thank you for highlighting this practical consideration. We suggest three approaches when PRM rewards may be noisy or unreliable:\n\n1. Robust learning [1]: Since training with noisy samples remains an active research area in deep learning, techniques like label smoothing [2] and t-CE loss [3] could be applied directly to PRMs to enhance their noise tolerance.\n2. Generative PRMs [4]: As discussed in Section 5.1, generative PRMs could leverage the inherent knowledge of foundation models to provide more robust judgments compared to discriminative PRMs.\n3. Larger Group Sizes: Increasing group size improves baseline stability in advantage estimation. Even if PRMs misassign rewards to some samples, this approach minimizes the impact on policy updates for correctly rewarded samples.\n\nAs PRM robustness falls beyond this paper's scope, we defer empirical validation of these approaches to future work.\n\n[1] Song, Hwanjun, et al. \"Learning from noisy labels with deep neural networks: A survey.\" IEEE transactions on neural networks and learning systems 34.11 (2022): 8135-8153.\n\n[2] Wei, Jiaheng, et al. \"To smooth or not? when label smoothing meets noisy labels.\" arXiv preprint arXiv:2106.04149 (2021).\n\n[3] Feng, Lei, et al. \"Can cross entropy loss be robust to label noise?.\" Proceedings of the twenty-ninth international conference on international joint conferences on artificial intelligence. 2021.\n\n[4] Liu, Zijun, et al. \"Inference-time scaling for generalist reward modeling.\" *arXiv preprint arXiv:2504.02495* (2025).\n\n---\n\n> **[Q3]** Beyond reasoning tasks, could the min-form credit assignment principle be effectively applied to other fine-tuning scenarios (e.g., instruction following) where process-based feedback might be available?\n\n**[A5]** Thanks for this valuable suggestions. We agree that instruction following and similar scenarios present promising applications. As we mentioned in **[A2]**, we are currently exploring towards general reasoning (mathematics, coding, science, and logic), which require overcoming significant computational and data challenges. The primary constraints are GPU resources and the scarcity of process-supervised training data in broader domains. We are systematically addressing these limitations and plan to release new projects extending this framework in the near future."}, {"review_id": "TkuFCWQ07q", "reviewer": "Reviewer_GpAR", "summary": "The paper introduces process supervised reinforcement learning, a reinforcement fine-tuning framework designed to stabilize training of large language models using dense, step-level feedback from process reward models. The authors identify a key issue in prior work: summation-form credit assignment often causes LLMs to exploit high-reward steps, leading to reward hacking and training collapse. This method replaces the summation with a novel min-form credit assignment, which focuses on the worst-performing step in a reasoning chain. Experiments across multiple benchmarks show that PURE achieves comparable reasoning performance to traditional methods with higher training efficiency.", "questions": "1.  It is recommended to use argmax instead of maximize in Eq.(2) to keep the consistency of symbol usage.\n2. Some queries and improvement suggestions for Fig.2:\na)  Why are the accuracy curves recorded on the 1.5B model, not 7B?\nb)  It is suggested to explain the reason that some curves are truncated in the caption (e.g., early stop caused by training divergence).", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n1.\tBy replacing the traditional summation-based credit assignment with a min-form approach, the proposed PURE framework significantly reduces reward hacking.\n2.\tThe framework flexibly incorporates both dense and sparse rewards, and shows that even a small fraction of ground-truth signals can substantially enhance final performance and robustness.\n3.\tThe min-form credit assignment in PURE is implemented by simply transforming process rewards without altering the reinforcement learning pipeline or core code structure\n\nWeaknesses:\n1. In Fig.3, it seems that the results in the sub-graph (top center) after approximately step 370 are truncated. Please consider log-scaling on the y-axis if this is not caused by training interruption.\n2. Please provide more accurate curves (like Fig.3(b)) on different datasets and different parameter models in the  Appendix, to better analyze the stability of the proposed method in the whole training cycle.\n3.  A minor error in page 8, line 288: duplicated periods.", "was_revised": true, "final_justification": "I have read the author's rebuttal and considered all the other reviewers' raised points. Most of my concerns are addressed, so I keep my acceptance rating.", "rebuttal": "Dear Reviewer GpAR,\n\nWe sincerely appreciate your valuable and insightful comments as they are extremely helpful for improving our manuscript. We have addressed each comment in detail below.\n\n---\n\n> **[w1]** In Fig.3, it seems that the results in the sub-graph (top center) after approximately step 370 are truncated. Please consider log-scaling on the y-axis if this is not caused by training interruption.\n\n**[A1]** Thank you for highlighting this observation. At approximately step 370, the clip ratio for both correct and incorrect responses rises sharply to around 0.85. This upward trend aligns with concurrent increases in response length (shown in Figure 3, top left). We agree that log-scaling the y-axis would improve visualization. Thank you very much for your suggestion! Due to NeurIPS rebuttal policies, we cannot include images this year but will update this subfigure in the final manuscript.\n\n---\n\n> **[w2]** Please provide more accurate curves (like Fig.3(b)) on different datasets and different parameter models in the Appendix, to better analyze the stability of the proposed method in the whole training cycle.\n\n**[A2]** Thank you for this suggestion. This is indeed crucial for the analysis of RL algorithms. For Qwen2.5-Math series, the accuracy on MATH-500 rises steadily within 500 training steps for PURE-VR, PURE-PRM+VR, and PURE-PRM (until its collapse). For Qwen2.5-7B:\n\n- PURE-PRM: accuracy rises steadily before collapsing.\n- PURE-PRM+VR: accuracy rises for 500 steps, converges, then fluctuates around the converged value.\n- PURE-VR: accuracy rises steadily through 1000 steps, reaching PURE-PRM+VR’s convergence value at step 1600.\n\nPer current NeurIPS rebuttal constraints (no links and images), we cannot display them here. We will add full training dynamic figures to the appendix in the final manuscript.\n\n---\n\n> **[w3]** A minor error in page 8, line 288: duplicated periods.\n\n**[A3]** Thank you for catching this typo. We will correct the duplicated periods and carefully review the manuscript for similar errors.\n\n---\n\n> **[Q1]** It is recommended to use argmax instead of maximize in Eq.(2) to keep the consistency of symbol usage.\n\n**[A4]** Thank you for this suggestion. We will update Eq.(2) accordingly. Once again, we sincerely appreciate your meticulous review, which greatly improves the quality of our manuscript!\n\n---\n\n> **[Q2]** Some queries and improvement suggestions for Fig.2: a) Why are the accuracy curves recorded on the 1.5B model, not 7B? b) It is suggested to explain the reason that some curves are truncated in the caption (e.g., early stop caused by training divergence).\n\n**[A5]** Thank you for pointing this out. \n\n**a)** In order to show as much of the training dynamics of each model as possible within page limits, we assigned Figure 2a to Qwen2.5-Math-7B, Figure 2b to Qwen2.5-Math-1.5B, and Figure 3 to Qwen2.5-7B. Accuracy curves for 7B models are detailed in **[A2]**.\n\n**b)** We appreciate this suggestion. That would indeed be more reasonable! The x-axis of some curves in Figure 2 are truncated due to training collapse. The y-axis of the clip ratio in Figure 3 (top center) is truncated to highlight early-stage changes near $y=0$ (will use log-scaling later). The curve of $T=1$ in Figure 7 is truncated due to training convergence."}, {"review_id": "noig1vwWVW", "reviewer": "Reviewer_Pj7M", "summary": "This paper introduces a method designed to reduce reward hacking during reinforcement fine-tuning (RFT) of LLMs. The main idea is to propose replacing traditional summation-based credit assignment with a min-form credit assignment.", "questions": "See the comments above.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Pros**:\n- The min-form credit assignment is a novel and principled solution that tackles a timely and important problem.\n\n- The paper shows significant improvement across several datasets and models\n\n- Comprehensive analysis and discussion of reward hacking cases, providing actionable insights for further research\n\n**Cons/Questions**:\n- I recommend that the authors not use a title like **\"sth is all sth need\"**, which often exaggerates the facts and shadows the contribution of the paper.\n\n- Theoretical guarantees regarding the min-form credit assignment are not deeply explored.\n\n- The experimental validation is primarily conducted on mathematical reasoning benchmarks and models derived from the Qwen2.5 series,  potentially limiting immediate generalizability to broader domains\n\n- The results are reported based on single-run evaluations, without confidence intervals or multiple seeds. This makes it difficult to assess the statistical robustness and reproducibility of the observed improvements. We cannot tell if the results are statistically significant or generalizable.\n\n- Have the authors conducted any multi-seed experiments? If so, could they report the mean and standard deviation of the performance?  \n\n- All experiments are conducted on mathematical reasoning tasks. Can the authors provide evidence or discussion regarding the applicability of PURE to domains such as commonsense reasoning, code generation, or dialogue modeling?", "was_revised": true, "final_justification": "I have read the author's rebuttal and considered all the raised points. Most of my concerns are addressed. \n\nRegarding the reasonableness of Assumption 1, this assumption can significantly simplify the theoretical derivation. However, if the global bound `ε` from Assumption 1 is large, then the `ε` in the final upper bound will also become large as well. But considering that the experimental performance is very good, I think it empirically suggests that the size of the global bound `ε` is likely relatively small in this setting. Therefore, this is not a major issue; however, if this assumption could be relaxed (or improved), it has the potential to yield a tighter bound.\n\nHope the authors can further consider this point.", "rebuttal": "Dear Reviewer Pj7M,\n\nWe sincerely appreciate your valuable and insightful comments as they are extremely helpful for improving our manuscript. We have addressed each comment in detail in the paragraphs below.\n\n---\n\n> **[w1]** I recommend that the authors not use a title like **\"sth is all sth need\"**, which often exaggerates the facts and shadows the contribution.\n\n**[A1]** We appreciate the reviewer's suggestion regarding title clarity. We will revise the title to **\"Stop Summation: Min-Form Credit Assignment Suffices for Process Reward Models in Mathematical Reasoning\"** to more precisely reflect our contribution's scope. This maintains our core insight that minimal-form credit assignment can effectively replace summation in process-based reward modeling.\n\n---\n\n> **[w2]** Theoretical guarantees regarding the min-form credit assignment are not deeply explored.\n\n**[A2]** Thanks for highlighting this. We derive the following theorem to theoretically analyze our algorithm:\n\n**Assumption:**\n\nWe first establish 2 reasonable assumptions:\n\n* **Assumption 1 (Bounded Process Reward Error):** The Process Reward Model, $R^p$, produces a reward estimate $r_t^p = R^p(s_t, a_t)$ for each reasoning step. It is assumed that there exists a \"true\" or optimal reward $r_t^*$ for that step, and the estimation error is uniformly bounded by a constant $\\epsilon \\ge 0$. For any step $t$, it holds that $|r_t^p - r_t^\\*| \\le \\epsilon$.\n\n* **Assumption 2 (Bounded Rewards):** The true rewards, and consequently the estimated rewards, are bounded. There exists a maximum possible reward $R_{max}$ such that $|r_t^*| \\le R_{max}$ and $|r_t^p| \\le R_{max} + \\epsilon$ for all steps $t$. \n\n**Theorem (Q-value Estimation Error Bound Comparison):**\nUnder Assumptions 1 and 2, for any state-action pair $(s_t, a_t)$ and a trajectory $\\tau$ with $n$ reasoning steps:\n\n1.  **Sum-form Error Bound:** The estimation error for the summation-form Q-value, is bounded by:\n    $$|Q_{sum}^{\\pi}(s_t, a_t) - Q_{sum}^{*}(s_t, a_t)| \\le \\epsilon \\sum_{i=t}^{n} \\gamma^{i-t}$$\n    \n    For an infinite horizon ($n \\to \\infty$), this bound becomes:\n    $$|Q_{sum}^{\\pi}(s_t, a_t) - Q_{sum}^{*}(s_t, a_t)| \\le \\frac{\\epsilon}{1-\\gamma}$$\n    \n    This shows that the error bound is amplified by a factor dependent on the horizon length and the discount factor $\\gamma$, accumulating the error over steps.\n\n2.  **Min-form Error Bound:** The estimation error for the min-form Q-value, derived from Equations (3) and (4) of the paper, is bounded by:\n    $$|Q_{min}^{\\pi}(s_t, a_t) - Q_{min}^{*}(s_t, a_t)| \\le \\epsilon$$\n    \n    This shows that the error bound is strictly limited by the single-step reward estimation error $\\epsilon$ and **does not accumulate** with the number of steps.\n\n**Proof Part 1: Proof for the Sum-form Error Bound**\n\n1.  We begin with the definition of the absolute error of the Q-value, using the canonical formulation from the paper's Equation (1). $Q_{sum}^{\\pi}$ denotes the value calculated with the estimated rewards from the PRM ($r^p$), and $Q_{sum}^{\\*}$ denotes the true value (calculated with $r^*$).\n\n    $$ |Q_{sum}^{\\pi}(s_t, a_t) - Q_{sum}^{\\*}(s_t, a_t)| = \\left| E_{\\tau \\sim \\pi} \\left[ \\sum_{i=t}^{n} \\gamma^{i-t} r_i^p \\right] - E_{\\tau \\sim \\pi} \\left[ \\sum_{i=t}^{n} \\gamma^{i-t} r_i^\\* \\right] \\right| = \\left| E_{\\tau \\sim \\pi} \\left[ \\sum_{i=t}^{n} \\gamma^{i-t} (r_i^p - r_i^\\*) \\right] \\right| $$\n\n    $$\\le E_{\\tau \\sim \\pi} \\left[ \\left| \\sum_{i=t}^{n} \\gamma^{i-t} (r_i^p - r_i^\\*) \\right| \\right] \\le E_{\\tau \\sim \\pi} \\left[ \\sum_{i=t}^{n} \\left| \\gamma^{i-t} (r_i^p - r_i^\\*) \\right| \\right] = E_{\\tau \\sim \\pi} \\left[ \\sum_{i=t}^{n} \\gamma^{i-t} |r_i^p - r_i^\\*| \\right]$$\n\n2. Now, we apply **Assumption 1**, which states that the reward error at each step is bounded by $\\epsilon$, i.e., $|r_i^p - r_i^*| \\le \\epsilon$:\n\n   $$|Q_{sum}^{\\pi}(s_t, a_t) - Q_{sum}^{\\*}(s_t, a_t)| \\le E_{\\tau \\sim \\pi} \\left[ \\sum_{i=t}^{n} \\gamma^{i-t} \\epsilon \\right] = \\epsilon \\sum_{i=t}^{n} \\gamma^{i-t}$$\n\n   For the infinite horizon case, the geometric series converges to $\\frac{1}{1-\\gamma}$, which completes the first part of the proof.\n\n**Proof Part 2: Proof for the Min-form Error Bound**\n\n1.  We start similarly with the definition of the absolute Q-value error for the min-form, based on Equations (3) and (4) from the paper. The return $G(s_t, a_t|\\tau)$ is the value being expected.\n\n    $$|Q_{min}^{\\pi}(s_t, a_t) - Q_{min}^{\\*}(s_t, a_t)| = \\left| E_{\\tau \\sim \\pi} [G(s_t, a_t|\\tau)] - E_{\\tau \\sim \\pi} [G^\\*(s_t, a_t|\\tau)] \\right| \\le E_{\\tau \\sim \\pi} \\left[ |G(s_t, a_t|\\tau) - G^\\*(s_t, a_t|\\tau)| \\right]$$\n\n3.  Now we analyze the return term inside the expectation. According to Equation (3), for a trajectory $\\tau$ and a step $t$, the return is defined as the minimum of future rewards.\n    * $G(s_t, a_t|\\tau) = \\min(r_t^p, r_{t+1}^p, \\dots, r_n^p)$\n    * $G^\\*(s_t, a_t|\\tau) = \\min(r_t^\\*, r_{t+1}^\\*, \\dots, r_n^\\*)$\n\n4.  We use a key property of the minimum function: $| \\min(\\mathbf{a}) - \\min(\\mathbf{b}) | \\le \\max_i |a_i - b_i|$. Applying this property:\n\n    $$|G(s_t, a_t|\\tau) - G^\\*(s_t, a_t|\\tau)| = |\\min_{i=t \\dots n}(r_i^p) - \\min_{i=t \\dots n}(r_i^\\*)| \\le \\max_{i=t \\dots n} |r_i^p - r_i^\\*| \\le \\epsilon$$\n\n4. Substitute this result back into the expectation inequality from step 1:\n\n   $$|Q_{min}^{\\pi}(s_t, a_t) - Q_{min}^{\\*}(s_t, a_t)| = E_{\\tau \\sim \\pi} [|G(s_t, a_t|\\tau) - G^\\*(s_t, a_t|\\tau)|] \\le E_{\\tau \\sim \\pi} [\\epsilon] = \\epsilon$$\n\n   This completes the proof. It is proven that the error in the min-form does not accumulate and is directly bounded by the maximum single-step error of the reward model.\n\n**Furthermore, we also derive upper bounds for both the advantage function error and the update gradient error.** Due to rebuttal's word limit, we will add these two theorems in the appendix.\n\n---\n\n> **[w3]** The experimental validation is primarily conducted on mathematical reasoning benchmarks and models derived from the Qwen2.5 series, potentially limiting immediate generalizability to broader domains\n\n**[A3]** Thank you for this observation. For experiments on other models, we conduct additional experiments using Llama-3.1-8B with PURE-PRM+VR, compared against SimpleRL-Zoo (theoretically equivalent to PURE-VR). Results confirm that our algorithm maintains performance advantages on the Llama-3.1 architecture.\n\n| model        | method         | MATH-500 | Minerva Math | Olympiad Bench | AIME2024 | AMC23 | AVG.  |\n| ------------ | -------------- | -------- | ------------ | -------------- | -------- | ----- | ----- |\n| Llama-3.1-8B | - (Base model) | 13.6     | 4.8          | 3.1            | 0.0      | 2.5   | 4.8   |\n| Llama-3.1-8B | SimpleRL-Zoo   | 23.0     | 9.6          | 5.3            | 0        | 10.0  | 9.58  |\n| Llama-3.1-8B | PURE-PRM+VR    | 26.0     | 10.3         | 5.5            | 3.3      | 15.0  | 12.02 |\n\nFor experiments beyond mathematical reasoning, please refer to **[A6]** for our detailed response.\n\n---\n\n> **[w4]** The results are reported based on single-run evaluations, without confidence intervals or multiple seeds. This makes it difficult to assess the statistical robustness and reproducibility of the observed improvements. We cannot tell if the results are statistically significant or generalizable.\n\n**[A4]** Thank you for emphasizing statistical rigor. We conduct three additional runs with Qwen2.5-Math-7B using three different random seeds. The table below shows mean performance and standard deviation across four runs, demonstrating stable average results with only 0.84 standard deviation across multiple benchmarks.\n\n| random seed | MATH-500 | Minerva Math | Olympiad Bench | AIME2024 | AMC23 | AVG. |\n| ----------- | -------- | ------------ | -------------- | -------- | ----- | ---- |\n| 42          | 82.6     | 37.1         | 44.1           | 20       | 82.5  | 53.3 |\n| 12345       | 82.2     | 37.1         | 44             | 23.3     | 75    | 52.3 |\n| 23451       | 82.6     | 40.4         | 43.3           | 23.3     | 70    | 51.9 |\n| 34512       | 82       | 37.5         | 44.7           | 20       | 72.5  | 51.3 |\n| **mean**    | 82.35    | 38.025       | 44.025         | 21.65    | 75    | 52.2 |\n| **std**     | 0.30     | 1.59         | 0.57           | 1.91     | 5.40  | 0.84 |\n\n---\n\n> **[w5]** Have the authors conducted any multi-seed experiments? If so, could they report the mean and standard deviation of the performance?\n\n**[A5]** Thank you for your inquiry. Please refer to **[A4]** for detailed results from our multi-seed experiments.\n\n---\n\n> **[w6]** All experiments are conducted on mathematical reasoning tasks. Can the authors provide evidence or discussion regarding the applicability of PURE to domains such as commonsense reasoning, code generation, or dialogue modeling?\n\n**[A6]** We appreciate your interest in broader applications. We are actively exploring PURE's extension to general reasoning domains (math, code, science, logic), which presents significant challenges. The primary limitation is not our RL algorithm but rather scoring inaccuracies from PRMs. Since the PRM800k dataset (the largest open-source PRM training data) lacks corner-case responses and diverse domain data, its out-of-distribution characteristics lead to unreliable step-level scoring.\n\nTo address this, we are:\n\n1. Manually labeling 100k process labels for diverse domains\n2. Allocating ~1,000 H200 GPU hours for dedicated training\n\nWe are systematically addressing these challenges one by one. Currently, we have experimented with Qwen2.5-7B in the mixed field of math and science, **increasing the GPQA Diamond (a science benchmark) score from 38 to 53 using PURE-PRM+VR**, compared with 50 of PURE-VR. This result shows that there is no fundamental difference between different domains in terms of algorithms. It is merely a matter of different domain data ratios. We plan to release this new project in the near future."}, {"review_id": "q57DhZ4jrV", "reviewer": "Reviewer_ydx9", "summary": "This paper introduces a novel framework named PURE, which uses a min-form credit assignment to formulate the value function as the minimum of future rewards. The value function is thus also constrained to a range the same as the reward function. The author identify the reward hacking causes in PRM and claim that the PURE framework can significantly alleviate this issue. Compared to three base models, the new framework produces higher performance. In particular, it can achieve similar performance with only 30% steps compared to verifiable reward-based methods. This avoids the collapse issue by adopting the cononical summation of rewards approach. The author also demonstrate the cases for reward hacking and analyze training collapse.", "questions": "1) Referring to Weakness 2, could you tell what is the main contribution compared to the previous works? Why is the min-form credit assignment still novel in your case?\n\n2) What is the main difference between PURE-VR and other VR methods, and could you describe in more detail? The results in Table 2 do not take into consideration the different moving parts for the base models. Is it possible to include those?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n\n1) In general, the paper has an appropriate motivation towards the design of a min-form credit assignment framework, since there requires special curation in RL for language models.\n\n2) The code is simple to implement, as it requires only a transformation of process rewards without additional code changes.\n\n3) PURE achieves more efficiency and demonstrates its advantage in performance compared to the base models. This is well supported by results in Table 2 and Figure 2.\n\n4) The analysis towards reward hacking and collapse is interesting and helpful.\n\n## Weaknesses\n\n1) The author include results and discussion about summation-form credit assignment. However, is min-form based credit assignment the only way to alleviate this? To me, it seems like the problem mainly resides on the lack of data in PRM. It is also common practice in RL to use gradient clip and smaller step size for the optimizer under such scenarios.\n\n2) In line 86 to 88, the author mentioned previous min-value aggratation of process rewards (Lightman et al., 2023; Wang et al., 2023; Zhang et al., 2025), which is also defined by $min(r^p_1, ..., r^p_n)$. The concept is very much similar here, and Eq. (3) and (4) do not include too much novelty either.\n\n3) In Table 2, it is not clear if PURE is actually better than the baselines for all the moving parts.  I assume the appropriate way to compare is such as the base model with VR v.s. PURE with VR, base model with PRM v.s. PURE with PRM etc.", "was_revised": true, "final_justification": "The author answered my question well and eased my main concern regarding the novelty, plus the initial good performance of their methods. Hence, I have raised my score.", "rebuttal": "Dear Reviewer ydx9,\n\nWe sincerely appreciate your valuable and insightful comments as they are extremely helpful for improving our manuscript. We have addressed each comment in detail below.\n\n---\n\n> **[w1]** The author include results and discussion about summation-form credit assignment. However, is min-form based credit assignment the only way to alleviate this? To me, it seems like the problem mainly resides on the lack of data in PRM. It is also common practice in RL to use gradient clip and smaller step size for the optimizer under such scenarios.\n\n**[A1]** Thank you for raising this question. While min-form credit assignment is not necessarily the only solution for PRM-based RFT training, it currently represents the most practical and effective approach. By applying a simple transformation to process rewards, it converts initially unstable training into a stable process capable of sustained learning over hundreds or thousands of steps.\n\nWe respectfully disagree that PRM data limitations are the primary cause of instability in math reasoning RFT, based on three key observations:\n\n1. First, let’s think about the question of how much data is sufficient. During the RLHF era, InstructGPT utilized 500k preference pairs of data [1], while Claude used around the 1M scale [2]. Our preprocessed PRM800k dataset contains 370k samples with approximately 7 process labels for each sample, totaling 2~3M process labels. In comparison, such scale of PRM data can be considered sufficient. Considering the scaling law of preference labeling [3], if we want to further improve the accuracy of PRM, we would need to label another million-scale dataset - presents prohibitive costs for most institutions. Crucially, our method achieves stable training with existing datasets.\n2. As Section 5.1 demonstrates, the core issue is not PRM inaccuracy but reward hacking during RL training. In the first two hacking types (Section 5.1), PRMs provide accurate step-wise judgments (Figure 1), but the RL algorithm derives biased advantage values. This problem is fundamentally RL algorithmic rather than data-related.\n3. We do employ standard RFT stabilization techniques including gradient clipping (norm=1) and reduced learning rates (5e-7 or 1e-6), ensuring minimal per-step updates. Both of these tricks are default configurations in verl, but these measures alone proved insufficient without our credit assignment modification.\n\n[1] Ouyang, Long, et al. \"Training language models to follow instructions with human feedback.\" *Advances in neural information processing systems* 35 (2022): 27730-27744.\n\n[2] Bai, Yuntao, et al. \"Training a helpful and harmless assistant with reinforcement learning from human feedback.\" *arXiv preprint arXiv:2204.05862* (2022).\n\n[3] Hou, Zhenyu, et al. \"Does RLHF Scale? Exploring the Impacts From Data, Model, and Method.\" *arXiv preprint arXiv:2412.06000* (2024).\n\n---\n\n> **[w2]** In line 86 to 88, the author mentioned previous min-value aggratation of process rewards (Lightman et al., 2023; Wang et al., 2023; Zhang et al., 2025), which is also defined by $\\min (r_1^p, \\cdots, r_n^p)$. The concept is very much similar here, and Eq. (3) and (4) do not include too much novelty either.\n\n**[A2]** Thank you for highlighting this. Our approach differs from prior work in two critical aspects:\n\n1. Existing methods apply sample-level min-form during test time, while we define and implement token-level min-form during training time. The most critical issue here is that during testing, the LLM parameters are fixed, meaning that the distribution of LLM’s output does not change over time. Therefore, as long as the PRM adapts to the distribution of the LLM's output, the system will work properly. **However, during training, we must deal with distribution shifts, meaning the distribution of LLM’s output is constantly changing**. Especially in math reasoning, where the length of the entire output continues to increase during training. Additionally, the frequencies of some patterns are activated, while others are suppressed, thus the LLM's output distribution is constantly changing. This places higher demands on the overall algorithm's stability. Simply ensuring the PRM adapt to the initial LLM’s output distribution is insufficient; **the algorithm must continuously ensure that the output/input distributions of the LLM and PRM remain aligned to enable stable training over hundreds or even thousands of steps.** This distinction is similar to the difference between offline (such as native DPO) and online RLHF (such as online iterative DPO [1]). Offline RLHF only needs LLM adapt to a fixed data distribution, while online RLHF requires the entire system to continuously adapt to the evolved data distribution. This requires more sophisticated designs to ensure training stability, but at the same time will also bring better performance [1-2].  Specifically, in our algorithm implementation, we have to carefully decide whether to use the minimum value of future rewards or the minimum value of overall rewards, how to split the steps, whether to introduce a special token or consider all decoded characters containing “\\n\\n” tokens to split steps, and whether steps after the first error step should be included in PRM training, among other details. These factors impact the overall algorithm stability and ultimately determine whether the entire project can work effectively.\n2. While prior work uses sample-level aggregation, training requires token-level value functions. Therefore, we need to define an appropriate token-level value function (Eq. (3) and (4)). In the code implementation, to make our algorithm as easy to implement as possible, we propose a novel transformation function, allowing near-theoretical approximation through a single operation that reduces coding complexity.\n\n[1] Xiong, Wei, et al. \"Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint.\" *arXiv preprint arXiv:2312.11456* (2023).\n\n[2] Dong, Hanze, et al. \"Rlhf workflow: From reward modeling to online rlhf.\" *arXiv preprint arXiv:2405.07863* (2024).\n\n---\n\n> **[w3]** In Table 2, it is not clear if PURE is actually better than the baselines for all the moving parts. I assume the appropriate way to compare is such as the base model with VR v.s. PURE with VR, base model with PRM v.s. PURE with PRM etc.\n\n**[A3]** Thank you for pointing this out. We think there exists misunderstanding and would like to clarify as follows:\n\n1. **The base model with VR is theoretically equivalent to PURE with VR.** Since our framework has made significant modifications to the verl codebase, we must test whether the results can be reproduced under consistent hyperparameter settings. Therefore, we run PURE-VR and compare its performance with SimpleRL-Zoo in Table 2. As mentioned in line 217 of the paper, the settings for PURE with VR and Simple-RL are completely consistent and PURE-VR reproduces SimpleRL-Zoo's performance.\n2. **The base model with PRM is equal to use sum-form credit assignment with PRM, i.e., PURE-PRM (sum-form) shown in Figure 2.** Therefore, “base model with PRM vs. PURE with PRM” is equivalent to “PURE-PRM (sum-form) vs. PURE-PRM (min-form)”. PURE-PRM (sum-form) even collapses training at the beginning, thus, if we need to report its benchmark scores, we can only use the checkpoint at around step 5, which, as one can expect, won't be good either.\n\n---\n\n> **[Q1]** Referring to Weakness 2, could you tell what is the main contribution compared to the previous works? Why is the min-form credit assignment still novel in your case?\n\n**[A4]** Thanks for highlighting this. Please refer to **[A2]** for the novelty of the proposed min-form credit assignment method. In addition, the summarized reward hacking cases and the causes of training collapse in Section 5 are also important and novel to the community, which helps the community further understand the corner cases of LLM$\\times$RL and thereby research new algorithms.\n\n---\n\n> **[Q2]** What is the main difference between PURE-VR and other VR methods, and could you describe in more detail? The results in Table 2 do not take into consideration the different moving parts for the base models. Is it possible to include those?\n\n**[A5]** Thanks for pointing this out. Please refer to the detailed response in **[A3]**. Additionally, we would like to supplement that, based on our experience and technical report from ByteDance [1], Nvidia [2], and SkyWork [3], the dynamic and performance of LLM$\\times$RL experiments are closely related to the codebase. Even with the same hyperparameters, the results and training dynamics obtained using OpenRLHF and verl will differ a lot. Therefore, after implementing our codebase, it is necessary to conduct point-to-point comparison under the same setting (i.e., only verifiable rewards). As mentioned in line 219 of our paper, the performance of PURE-VR is comparable to SimpleRL-Zoo under exactly same setting and hyperparameters, confirming the reliability of our codebase. \n\nOur modifications to verl under verifiable reward settings include:\n\n- Call reward function before computing log probabillity to support curriculum learning\n- Fix the rare occurrence of reward function hang bug\n- Change the placement of the token-level verifiable rewards from the last of the padded tokens to the last token of un-padded valid tokens to better support multiple reward modes\n- More logged metrics to comprehensively monitor the training dynamics\n\n​\t…\n\n[1] Seed, ByteDance, et al. \"Seed1. 5-thinking: Advancing superb reasoning models with reinforcement learning.\" *arXiv preprint arXiv:2504.13914* (2025).\n\n[2] Chen, Yang, et al. \"Acereason-nemotron: Advancing math and code reasoning through reinforcement learning.\" *arXiv preprint arXiv:2505.16400* (2025).\n\n[3] He, Jujie, et al. \"Skywork open reasoner 1 technical report.\" *arXiv preprint arXiv:2505.22312* (2025)."}]

Author rebuttal:

Meta-review: This paper addresses reward hacking issues in Process Reward Models (PRMs) when fine-tuning large language models on mathematical reasoning tasks. The authors propose PURE (Process sUpervised Reinforcement lEarning), which replaces the summation-form credit assignment with min-form credit assignment. The paper conducts extensive experiments to demonstrate empirically that their approach successfully tackles the issue for reward hacking.

The reviewers agree that the paper is well-motivated and tackles an important problem (Reviewer GZwS, Pj7M, ydx9). The proposed solution is simple yet effective (Reviewer GZwS, GpAR, ydx9) and the empirical results support their claim (Reviewer GZwS, Pj7M, ydx9).

A major concern is the novelty compared to previous works which also apply min-aggregation (Reviewer ydx9). The authors effectively clarified a crucial distinction: existing methods apply min-aggregation at test-time with fixed model parameters, while their approach operates during training-time where the model distribution continuously shifts.

Reviewer Pj7M raises several concerns including (1) Lack of theoretical guarantees; (2) Single-run evaluations without statistical validation; (3) Limited scope to mathematical reasoning tasks. In response, the authors provide (1) comprehensive analysis comparing error bounds; (2) Multi-seed experimental results; (3) Additional validation on Llama-3.1-8B architecture; (4) Discussion of ongoing work extending to broader domains.

Reviewer GZwS questions about the performance on ultra-long reasoning chains, suggesting that maintaining minimum reward across many steps could be challenging. The authors acknowledge this as an ongoing research challenge affecting both PRM-based and VR-based methods.

Given the strengths mentioned above, as well as the fact that most concerns are addressed in the rebuttal phase, **I lean towards acceptance**.
Decision: Accept (poster)