Smarter Not Harder: Generative Process Evaluation with Intrinsic-Signal Driving and Ability‑Adaptive Reward Shaping
Authors: Tao He
Rongchuan Mu
Lizi Liao
Yixin Cao
Yang Li
Yijia Luo
Weixun Wang
Ming Liu
Bing Qin
Venue: ICLR 2026 Poster (2026)
Keywords: Generative Process Reward Model
Math Reasoning
Large Reasoning Model
Primary area: foundation or frontier models, including LLMs
TL;DR: We identify key pitfalls for GenPRMs—such as over-reliance on reasoning, exploration suppression, and reward hacking—and propose TP‑GRPO, a RL framework with intrinsic-signal evaluation, thought-level granularity, and ability-adaptive rewards.

Abstract: Large reasoning models (LRMs) have shown strong performance in complex mathematical reasoning when optimized via reinforcement learning (RL). However, conventional outcome-only reward provides sparse feedback, leading to inefficient optimization. In this work, we investigate whether generative process reward models (GenPRMs) can accelerate RL training of LRMs by improving the utilization of reasoning trajectories. We first analyze critical limitations in existing GenPRMs, including their heavy reliance on reasoning ability during correctness judgment, and suppression of exploration as well as vulnerability to reward hacking during reward assignment. To address these limitations, we first propose a novel \textbf{intrinsic-signal-driven evaluation} mechanism, which judges reasoning steps using semantic cues from the solution, thus mitigating extensive dependence on GenPRM. Furthermore, we (i) adopt \textbf{thought-level rewarding granularity} to alleviate over-dense step rewards, and (ii) design a \textbf{difficulty-aware reward formulation} that dynamically balances exploration and exploitation and keeping the optimization target of key tokens to mitigate reward hacking. We integrate these innovations into the process reward-based GRPO, resulting in the proposed \textbf{TP-GRPO} algorithm. Experiments on LRMs with 1.5B and 7B parameters show that TP-GRPO achieves higher improvements while using significantly fewer training samples, and more analyses further confirm the effectiveness of our proposed process evaluation mechanism.

Published as a conference paper at ICLR 2026
SMARTER NOT HARDER: GENERATIVE PROCESS EVALUATION WITH TERIISIC-SIGNAL DRIVING AND ABILITY-ADAPIIVE REWARD SHAPING
Tao Hel Rongchuan Mu Lizi Liao", Yixin Cao', Yang Li Yijia Luo Weixun Wang", Ming Liu!.2i Bing Qin
1 Harbin Institute of Technology. Pengcheng Laboratory, 3 Singapore Management University
4 Fudan University. 5 Shanghai Jiao Tong University, 6 Alibaba Group
theeir wit. en remu ir. edut.edd..cm cn Corresponding author.
Equal contribution.
ABSTRACT
Large reasoning models (LRMs) have shown strong performance in complex math- ematical reasoning when optimized via reinforcement learning (RL). However, conventional outcome-only reward provides sparse feedback, leading to inefficient optimization. In this work, we investigate whether generative process reward mod- els (GenPRMs) can accelerate RL training of LRMs by improving the utilization of reasoning trajectories. We first analyze critical limitations existing GenPRMs, in- cluding their heavy reliance on reasoning ability during correctness judgment, and suppression of exploration as well as vulnerability to reward hacking during reward assignment. To address these limitations, we first propose a novel intrinsic-signal. driven evaluation mechanism, which judges reasoning steps using semantic cues from the solution, thus mitigating extensive dependence on GenPRM. Furthermore, we (i) adopt thought-level rewarding granularity to alleviate over-dense step rewards, and (ii) design difficulty-waree reward formulation that dynamically balances exploration and exploitation and keeping the optimization target of key tokens mitigate reward hacking. We integrate these innovations into the process reward-based GRPO, resulting in the proposed TP-GRPO algorithm. Experiments LRMs with 1.5B and 7B parameters show that TP-GRPO achieves higher im- provements while using significantly fewer training samples, and more analyses further confirm the effectiveness of our proposed process evaluation mechanism.
INTRODUCTION
Reinforcement learning with verifiable rewards (RLVR) has recently achieved notable success in enhancing the reasoning capabilities of large reasoning models (LLMs), represented by DeepSeek R1 (Guo al., 2025) and Kimi k1.5 (Team al., 2025b). These approaches leverage efficient, rule-based outcome rewards (Shao et al., 2024; Yue et al., 2025; Xu et al., 2025) to extend the RL paradigm to complex reasoning tasks. However, existing RLVR methods typically focus solely on sparse answer correctness and overlook the rich semantic signals in intermediate reasoning trajectories. Motivated by the Metacognitive theories (Schraw & Moshman, 1995), outstanding students not only verify answers but also review and refine reasoning process, selectively reinforcing correct paradigms and correcting erroneous steps. This motivates researchers to explore employing process rewards for better exploitation, thereby improving the training efficiency of L.R.M
Process reward models (PRMs) are designed to evaluate the quality of reasoning processes. Early discriminative PRMs (Lightman et al., 2023; Wang et al., 2023b; Zhang et al., 2025; Zheng et al., 2024) partly address the sparsity of outcome rewards but commonly suffer from unstable and subjective step segmentation, poor generalization, and high annotation costs (Guo et al., 2025). Generative PRMs (GenPRMs) have emerged as a more flexible alternative, using strong LRMs to assess reasoning traces via thinking (Khalifa et al., 2025; Feng et al., 2025; She et al., 2025; Liu et al., 2025; Chen et al., 2025). While promising, we find that naïvely applying GenPRMs to reward shaping can introduce critical pitfalls that substantially impact training stability and mislead optimization direction.

---

Published as a conference paper at ICLR 2026
Intrinsic Signal-Driven Evaluation
① Judgment Stage: High dependency on ressoning ahility of GenPRNN
Question
Stop1 Camme Rewood +
Erin, proziem beyond math better than me, and can't find any flaw in his Ingio-mayye can only check the calculations.
Reward +i
Coneet
uvilly vin
GeEPRM
① Rewarding Stage: Static proces rewurds suppres the expleration ability:
Thought-leve Ability- Adapiive Rewarding
Think
see This problem can only be solved this solution templone, and next won'' other app approach.
Braini-1
H Rewarding Stage:
Teo dense process rewards trigue the roward backing:
Answer
Wow, so mony process rewards! Maybe the correct answer isn't that important. earn higher returns through these process rewards.
The worksfow of typical CemPlMS
Actor LRM
Figure I: Illustration of the three target issues across two stages of the prior GenPRM workflow.
We divide the workflow of GenPRM into two stages: judging the correctness of each step; and then, assigning rewards for each step based on the evaluation results. Through empirical analysis, we identify critical challenges in both two stages. A clearer explanation of these issues can be found in Figure/ 1. In the judgment stage, existing GenPRMs primarily judge per-step correctness by re-deriving or simulating reasoning (Feng et al., 2025). Although improving generalization, it implicitly assumes that the reasoning capability of PRM exceeds that of actor LRM. This assumption not only raises the capacity requirements for GenPRM, but also undermines the evaluation reliability when confronted with hard tasks. Morcover, using reasoning to validate reasoning processes may also introduce potential bias, which becomes more pronounced in self-evolution settings where the LRM evaluates its own outputs (He et al., 2025). In the reward assignment stage, the common method of assigning static rewards (+1/-1) for dense steps risks suppressing exploration and triggering reward hacking, Penalizing mistakes during exploring challenging tasks may discourage beneficial trial-and- error behaviors. Besides, dense step rewards can dominate advantage estimation (Experimental results in Figure 5 verifies this issue), incentivizing the model to optimize for process profits maximization rather than final correctness, trap known as reward hacking (Liu et al., 2024).
In response to these challenges, we first propose three guiding principles for GenPRM design: (P1) Decouple evaluation from reasoning to reduce high dependency on the reasoning ability of GenPRM; (P2) Reward in an appropriate granularity to mitigate the bias introduced by dense reward signals; and (P3) Balance exploration and exploitation to ensure that process rewards do not hinder exploratory reasoning. Guided by these principles, we introduce our GenPRM framework. First, in the evaluation stage, we propose an Intrinsic Haarinsie-iinnll-Drivee Evaluation mechanism. This strategy leverages semantic and logical signals inherent in the solution to judge step correctness. This design shifts the heavy reliance from the reasoning ability to PRM's semantic understanding and matching capacities, which constitute fundamental abilities of LLMs.. Second, in the reward assignment stage, we design a Thought-Leve Difficulty-Aware Adaptive Reward mechanism. Instead of rewarding each step, we merge consecutive correct incorrect steps into coherent reasoning segments, which we refer to as "thought". Assigning rewards at the thought level effectively mitigates the issue of dense step rewards. Furthermore, the reward strength is dynamically adjusted according to the LRM's 's current reasoning capability: for difficult tasks, the mechanism automatically reduces the suppression for failed exploration; for easy tasks, more prominent process rewards are used to reinforce correct thoughts and penalize incorrect ones. In addition, our reward design keeps the optimization target of key tokens when introducing process reward, thereby effectively alleviating the issue of reward hacking. Finally, we integrate these mechanisms into the process soward-based GRPO (Shao et al., 2024) and propose our TP-GRPO algorithm.
We validate our approach on the --insSee--R--istil---------- AIME 2024 (Maxwell-Jia, 2024), our 1.5B model trained on only 700 problems achieves a +4.32% improvement, which rises to +5.98% when trained on 1,800 problems: the 7B model trained on 1,070 problems improves by +6.67% While modest in absolute scale, these gains are consistent and achieved in fewer training steps, supporting our hypothesis that reasonable GenPRM improve training efficiency beyond outcome-only rewards. Our main contributions are summarized as follows:
model family (Guo et al., 2025). On
An in-depth analysis of design pitfalls in GenPRM-based process evaluation, identifying three fundamental challenges and proposing three actionable principles for GenPRM design;
Three innovations over challenges for existing GenPRM workflow are introduced: intrinsic-signal-diie -drven evaluation, and thought-leve.. ability reward schemes, further integrated with GRPO to form new RL algorithm, termed TP- TP-GRPO;
2

---

Published as a conference paper at ICLR 2026
Empirical evidence and detailed analysis show that TP-GRPO results in better efficiency and accuracy, underscoring the broader importance of well-designed GenPRM.
2 PRELIMINARY
2.1 PROBLEM FORMULATION
Given a mathematical problem 2, the LRM ma produces a long-form, chain-of-thought output o, also termed Long CoTT, which is formalized as the ordered pair comprising think and answer (Guo et al., 2025). The think encapsulates the full reasoning trajectory. typically spanning multiple rounds of hypothesis formation, exploratory derivation, error correction, and reflection. The answer A is usually a concise summary based on the preceding think, generally retaining only the principal reasoning pathways that substantiate the final result. Assuming consistency between answer and think, we argue that improving reasoning quality primarily hinges on refining the think. Consequently. our process-level evaluation targets the think component rather than the answer. For brevity, unless otherwise specified, we will abuse the term solution to denote think.
2.2 RL FOUNDATION: PROCESS SUPERVISION GRPO
Our method builds on process supervision GRPO, first introduced in DeepSeekMath (Shao et al., 2024). Its optimization objective identical that of outcome supervision GRPO:
JARPO (0) = =Elg ~ P(Q), (o mola (Olz)
1
(I)
min Pr..A clip(es,, 6, 1 + +c) 33.. [ool
G
where Pı, denotes the policy ratio, A., the advantage estimate, the clipping threshold, 3 the KL reg- ularization coefficient, and ret the reference policy. In process supervision GRPO, given G sequences of process-reward signals R (r.iddex(1) 3.... vi where K; represents the number of process rewards in i-th rollout, GRPO first standardize all process rewards:
index(()) J..... (re (nddex(1) GG
index (Ko) (ndex(Ko)) }}.
index(j) mean(It
index(()
std(R)
The per-token advantage then defined as the cumulative sum of standardized process rewards and after the token position:
Ar,
Σ
sindex(f)
index(j)2
where index(j) denotes the token position in the i-th sequence at which the j-th process reward is assigned. This construction refines the advantage estimate using fine- grained, token-level signals.
OUR METHOD: TWO-STAGE THOUGHT-LEVEL GENPRM AND TP-GRPO
In this section, we first introduce novel GenPRM mechanism, which primarily leverages intrinsic signals from the reasoning trajectories for judgment, and adaptively computes rewards to balance exploration and exploitation while mitigating the issue reward hacking $(3.1). To assess its effectiveness, we integrate the designed GenPRM into the training loop of LRM propose a new RL algorithm TP-GRPO S(3.2).
3.1 INTRINSIC SIGNAL-DRIVEN THOUGHT-LEVES GENERATIVE PROCESS EVALU'TIIO
Before evaluation, we segment the think into steps with appropriate semantic granularity. We compare two strategies: heuristic splitting based on "A \n" and segmentation via prompting LRM. Finally, we adopt the method of prompting the LRM for good segmentation effectiveness. The applied prompt presented in Appendix F.

---

Published as a conference paper at ICLR 2026
Question
Question
Question
Merge Steps
Groug Sampling
Token Probubility Change
Correc Thought
++*
outcome reward:
encouraged
Think
Interoal Judgmeent
Think
Error
Roflection Localization
Dense grandarrty
Correct Steps
Correct Thought
+7"
panaltiaa suppress correct tokeess
Grmu Accuracy
Stage I: Intrinsic Inteinsic-signal-dee -igial-drivee step-level process evaluation
Stuge II: Thought-level adilitt--dnptive
Thought rowards: colnect encouraged tokenscttokens penalized
process rewarding mechanism
(a) Our proposed GenPRM method, consisting of: (i) decomposing correctness assessment into three easier sub-tasks, (ii) aggregating steps into thought and employing difficulty-awwe aware adaptive reward.
(b) Illustration of optimization bias induced by dense reward granularity.
Figure 2: Overview of our proposed GenPRM (a) as well as the illustration of optimization bias caused by dense process rewards (b). We design distinct evaluation and reward mechanisms for correct and incorrect solutions; the overview here illustrates only the pipeline for correct solutions.
Building on this decomposition, we propose our GenPRM framework, which be broadly divided into two stages: correctness evaluation and reward assignment. In Stage I, we design a prompt pipeline to judge per-step correctness. Our method leverages semantic and logical signals contained in the solution, thereby reducing the dependency strong reasoning capacity of GenPRM. In Stage II, we introduce thought-level, difficulty-awarr adaptive process reward formulation. This mechanism dynamically balances exploration and exploitation and mitigates reward hacking by keeping the optimization target of key tokens. The overview of our proposed GenPRM is given in Figure 2a. A more detailed illustration is presented by Figure 7 in Appendix F.
3.1.1 STAGE I: INTRINSIC-SIGNAL--DOE SIGNAL-DRIVEN STEP-LEVEL PROCESS EVALUATION
In this subsection, we initiate our exploration of employing generative paradigm judge the correctness of reasoning steps. Our process evaluation mechanism designed around the core principle: Decoupling Evaluation from Reasoning Capacity.
This principle is proposed to solve the 1st Pitfall of cxisting GenPRM paradigm: The quality of current process evaluation mechanisms is fundamentally constrained by the reasoning capabilities of the GenPRM. This requires the GenPRM to identify logical errors within the solution through deliberate thinking, thereby implicitly assuming that GenPRM's reasoning proficiency is least on par with the actor LRM. However, as task complexity increases and the actor LRM's reasoning capability continues advance, the demand the GenPRM escalates accordingly, which poses serious challenge to the long-term effectiveness of the GenPRM as an evaluator.
To solve this challenge, we suggest decoupling evaluation from reasoning by breaking down the evaluation task into sub-tasks completed by other foundational capabilities of LLMs.
By examining Long COT, observe intrinsic semantic cues indicative of the LRM's reasoning state, such as self-reflective statements. These reflections directly indicate errors and their potential causes, thereby providing instructive signals for process evaluation. Building on the design principle and these findings, we design two distinct protocols for correct and incorrect solutions:
Correct Solutions (Identify Effective Steps): For traces with correct final answers, pinpoint ineffective steps via three steps: 1) Reflection Localization: Using semantic comprehension, GenPRM identifies reasoning steps where the LRM realizes its mistakes through self-refiection (e.g., "wait, made a mistake"). 2) Error Source Tracking: During reflection, the LRM typically analyzes the causes of the reflected mistake. Based on the analysis, GenPRM traces back error source and delineates a candidate error interval. 3) Interval Verification: GenPRM applies heuristic rules (e.g., "steps dependent on conclusions from incorrect previous steps deemed erroncous")) ") to judge each step within the eтrоr interval, guided by the reflected error cause.
Incorrect Solutions (Avoid Over-Penalizationl For incorrect solutions, we first propose a conservative hypothesis: all steps in the answer erroneous. Under this hypothesis, think steps
4

---

Published as a conference paper at ICLR 2026
semantically matching the erroneous answer are labeled incorrect, while unmatched steps viewed as uncertain. This design realizes a trade-off: relying solely on the outcome rewards would incur over penalization for underthinking behaviors in the think part, but achieving perfect step-level judgments demands exceptionally high evaluation capability for GenPRM. We propose easy process evaluation requiring only the capacity of semantic alignment for GenPRM.
This evaluation framework is grounded in the following theoretical and practical motivations:
For correct solutions, we leverage reflection signals to trace erroneous steps, based on the self- contained assumption: if the final answer is correct, any erroneous reasoning step must be accompanied by an effective reflection that identifies and corrects the error (either by direct fix or switching reasoning strategies). Otherwise, the error would propagate and mislead the final answer. Under this assumption, enrors and reflections must occur in pairs in correct solutions, enabling reflection-firs attribution strategy that is both theoretically valid and practically reliable.
We further emphasize decoupling evaluation from reasoning. For correct solutions, perform evaluation using intrinsic reflection cues combined with GenPRM's semantic understanding capability; for incorrect solutions, conduct coarse-grained evaluation based solely on semantic matching. This design substantially reduces reliance on GenPRM's 's reasoning capabilities. There are two core motivations: (I) Semantic understanding and matching are more fundamental and stable abilities of LLMs, leading to more robust evaluation results; (2) requiring GenPRM to conduct process evaluation via thinking of inconsistencies between its own reasoning chain and the Long CoT implicitly presupposes stronger and heterogeneous reasoning capabilities. If such capabilities are not satisfied, the evaluation process is likely to introduce systematic bias. By decoupling evaluation from reasoning, we effectively mitigate such risks.
3.1.2 STAGE THOUGHT-LAVEL, ABILITY-ADAPTIEE PROCESS REWARDING MECHANISM
Specific reward values are required after per-step correctness evaluation.
S1: Thought-level Reward Unit through Step Merging
The 2nd Pitfall: Through analyzing the step-level process evaluation results, we observe that think is often decomposed into a large number of steps. process rewards are assigned at the step level, the advantage would be dominated by these process rewards, which could in turn mislead the optimization to incorrect directions. Figure 2b illustrates this effect: although the 4th step correct, the five subsequent incorrect steps result in negative cumulative return. Consequently, this correct step is crroneously penalized during training, highlighting the misleading nature of dense process rewards.
To address the aforementioned issue, revisit the granularity of reward assignment and propose thought-level process reward mechanism. Consecutive steps with the same correctness are merged into a single logical unit- termed thought. For correct solutions, this entails merging consecutive correct/incorrect steps into a correct/incorrect thought; for incorrect solutions, consecutive matching or non-matching steps are merged analogously. The process reward is then assigned the thought level rather than the step level. This design is easy to implement and effectively reduces the density of process rewards. Importantly, we do not blindly filter the reward signals; instead, we retain the minimal set of rewards necessary to correctly guide LRM optimization while maximizing the reduction of redundant rewards. more intuitive comparison is presented Figure 2b.
S2: Difficulty-Aware Adaptive Process Reward Mechanism
We further design two distinct process reward mechanisms based on the solution correctness. We argue that designing effective process rewards for LRMs requires addressing two major Pitfalls:
Static correctness-based process rewards can unintentionally suppress exploration. Although penalties for incorrect attempts lower the probability of unproductive trials, they may also restrict LRM's exploratory capabilities. Process rewards provide more localized learning signals that bias LRM toward exploiting known high-return reasoning paths. However, this also increases the risk of falling into local optima and diminishes further exploration.
Poorly designed process rewards may introduce reward hacking the model may discover ways to improve the process rewards without genuinely improving its reasoning.
5

---

Published as a conference paper at ICLR 2026
To this end, we propose a diffieulty- aware adaptive process reward mechanism with two complemen- tary regimes, defined according to the correctness solutions.
For Correct Solutions: For correct solution og, each correct thought receives +r", and each incorrect thought receives -r", where:
r== aCCG:
0.
Here, acCg is the accuracy of the G sampled solutions for the same problem. When accg = 0, re vanishes, reducing to outcome supervision GRPO (Guo et al., 2025) and prioritizing exploration. When aceg = 1, r = process rewards are strongest to reinforce correct steps and suppress incorrect ones. The hyperparameter a further controls the global scale of the process rewards. By scaling r with accuracy, the model is free to explore facing harder questions while receiving stronger process guidance once the task is easy, thus avoiding both premature suppression of exploration and spurious optimization behaviors.
For Incorrect Solutions:
For an incorrect solution o;, penalties are applied only to thoughts that lead to the erroneous answer, i.e., those in the think matched the answer part. Given the normalized outcome reward rg in Eq. 4, the thought-level process reward rfc is defined as:
Pg. if the i-th thought matches the incorrect answer, otherwise.
Since the solution is incorrect and fg < O, unmatched thoughts, typically other exploratory attempts that are not employed in the answer part, receive a non-negative reward. This prevents penalizing uncertain reasoning steps, thereby mitigating the risk of over-veernett wver-penalization.
We further discuss how our method contributes to mitigating reward hacking in the following section.
3.2 TP-GRPO: GRPO BASED ON THOUGITT-EVEE PROCESS REWARD
Finally, we employ the process evaluation results in process supervision GRPO (Shao al., 2024) and introduce a new algorithm, TP-GRPO. TP-GRPO jointly optimizes the LLM under the guidance of both process and outcome rewards. For the outcome rewards, we begin by normalizing them within each group as in outcome supervision GRPO (Guo et al., 2025):
7 - mean mean((r)))))) ((r))=)) std((r)))=1)
The process rewards are configured as described in Sec 3.1.2, without applying the normalization as in Eq. 2. We explain this design in Appendix C. Following Shao et al. (2024), the advantage of each token og calculated as A,, = Σ = Ti.e' as introduced in Eq 3, where |o:| the number of tokens in Finally, we optimize the LRM following the optimization objective defined in Eq. 1:
To better understand the advantages of the process rewards design, we establish the following theoretical properties from the perspective of advantage estimation in GRPO:
Proposition 1. For a correct solution oi, token Oi,t correct thoughts have advantage A P.. while token og in incorrect thoughts have advantage A -. - r
This proposition shows that tokens within correct thoughts retain the same advantage as using outcome-only rewards, safeguarding against misguided shifts in the optimization objective during RL training which we posit as the primary cause of reward hacking. contrast, tokens within incorrect thoughts receive reduced advantages compared with using outcome-only rewards, effectively weakening the optimization of flawed steps.
Remark:
Proposition 2. For an incorrect solution o,, token e in matched thoughts have advantage A = Po. while token e₁ unmatched thoughts have advantage Ai, 0.
This proposition demonstrates that tokens within matched thoughts retain the same advantage values under outcome-only rewards, thereby achieving the objective mentioned above: introducing process rewards without altering the original training objective for these incorrect tokens during RL training. In contrast, unmatched thoughts receive zero advantages, avoiding wer-penalization since their correctness remains uncertain.
Remark:
The proofs of these propositions presented in Appendix B.

---

Published as a conference paper at ICLR 2026
4 EXPERIMENTS
In this section, we evaluate TP-GRPO on five datasets to investigate the following research questions:
RQ1: How does TP-GRPO perform in enhancing the training efficiency of LRM? ($ 4.2)
RQ2: Does ceatissc-sinna-driiii step-level process evaluation provide advantages over the reasoning-based GenPRM paradigm? (S 4.3.1 & $ 4.3.3)
RQ3: Does dense reward design introduce potential negative effects, and is the proposed step- merge strategy really effective? ($ 4.3.1)
RQ4: Is the proposed difficulty-aware reward mechanism effective in practice? ($ 4.3.1)
RQ5: Are the process rewards designed for correct and incorrect solutions empirically rational? ($ 4.3.2)
Given computational limitations, focus our experiments smaller LRMs.. To meet the format requirement that explicitly includes both think and answer components, we select the DeepSeek-KR Distill-Qwen 1.5B and 7B for experiments. We leave broader studies across more scales and variants to future work.
4.1 EXPERIMENTAL SETTINGS
Baselines We primarily compare TP-GRPO with the reproduction of GRPO using only outcome rewards, aiming to demonstrate the advantage of our proposed process rewards in terms of both performance and efficiency. In addition, we seek to validate the effectiveness of our approach relative to the existing GenPRM methods. To this end, we include two GenPRM-based baselines: (i) an LLM-as-a-judge variant based on Qwen3-32B that evaluates step correctness using prompts derived from Zhao et al. (2025), and (ii) GenPRM-32B model that is pretrained on process evaluation datasets (Zhao et al., 2025). The prompt used for L.MMaas--judde is provided in Table 16 at the end of the Appendix. Besides, we also include results from several open-source baselines trained on DeepSeek-Ri RI--istill- -wenn models as reference and also compare efficiency with them.
Training Data and Evaluation Benchmark We conduct all experiments on DeepScaler-40K (Luo et al., 2025). We follow prior work and evaluate on 5 widely-used math benchmarks: AIME 24 (Maxwell-Jia, 2024), AIME 25 (math ai, 2025), AMC 23 (Li et al., 2024a), MATH-500 (Hendrycks et al., 2021), and Olympiad (He et al., 2024). For experiments with DeepSeek-RR ---i--R----iii-------- 1.5B, we perform 16 independent Pass @ evaluations and report the average (Avg @ 16) as final results. Due to the higher inference cost of DeepSeck-PR -iieeckk-RR-DD RI-Distill-77B -B and limited sizes of AIME 24, AIME 25, and AMC 23, we report Avg@ on these subsets and Pass@ @ on MATH-500 and Olympiad. Besides, we aim to further investigate improvements in training efficiency. To ensure a fair comparison, we design new metric Effic. to measure the training efficiency: Effic. #tmining solutions 10 Here, "Improvement" denotes the reasoning performance gain, and "#solutions" is the total number of solutions used for training. Obviously, a higher Effic.. indicates that the algorithm can achieve greater performance with fewer samples, thereby reflecting higher training efficiency.
Improvement
Hyperparameters All training experiments are conducted using the TRL framework (von Werra et al., 2020) and vLLM backend. We use prompt batch size=5, fixed learning rate=le 6, and sample rollouts per prompt. For on-policy GRPO, we save checkpoints every 50 training steps and perform evaluations accordingly. Owing the efficiency constraints of GenPRM, we train the framework in an off-policy setting. Each iteration consists of: (i) collect rollouts using the latest model sufficient for 50 training steps, (ii) parallel evaluation deploying multiple GenPRMs and (iii) performing multi-step training once evaluation is complete. More details about off-policy pipeline are presented in Appendix G. Further details of the parameter settings are relegated to the Appendix
4.2
MAIN RESULTS
Tables and 2 report the main results with DeepSeck-RI-Disilll Cel-Distill-Qwee- 1.5e and 7B as backbone models, respectively.
Absolute Performance Improvements. On the 1.5B model, TP-GRPO achieves an average improvement of +2.04, with substantial gains on AIME 24 (+4.32) and AIME 25 (+3.13). For
7

---

Published as a conference paper at ICLR 2026
Table Main results on SeepSeek-RI--isiil--wweeee
TP-GRPO achieves higher accuracy than the on-policy salcome-basede GRPO baseline while requiring 5x fewer solutions. "K" means x 1000.
Model
AIME
Avg 16
AIME 25
Avg 16
23
Avg@ LG
MATH-500
Avg 16
Olympiad Avg 16
Avg.
#Solution
Effic.
Model
28.80
22.50
62.90
82.80
43,30
48.06
Outcome Reward-bused RLVR methods
AutoThink (Stage I)
28. 13
19.58
60.39
79.74
40.44
45.66
450.6K
-0.53
Open RS1
30.42
20.83
63.78
83,01
43.91
48.39
57.6K
0.53
B.ll-3-1..5--Previnn
32.50
23.33
66.94
84.19
45.96
50.58
256K
0.57
Adap(Think
33.75
24.38
69.35
84,18
45.04
51.34
614.4K
0.98
AReaL-1 SB-Prreiew(SSSgge 1)
30.83
23.75
66.64
84.33
45.51
50.21
1638.4K
0.13
DeepSealee- 1.5--rvview
43.10
29.38
73.60
87,80
50.00
56.78
>3584K
<0.49
GRPO Replication (850 steps)
32.71
24.58
64.53
82,70
43.67
49.64
34K
4.65
Reward- methods
GRPO with I.I J.M--s---juaee (818 steps)
30.41
24.58
63.32
82.61
43.35
48.85
4.7K
16.8
GRPO with GenPRM-32B (262 stops)
31.45
23.12
63.00
83.30
43.40
48.86
10.4K
7.63
TP-GRPO (140
33.12
25.63
64.01
83.81
43.91
50.10
5.6K
36.43
Table 2: Main results on ---ieee-------iiill---------
TP-GRPO achieves higher accuracy than the on-policy outcome-based GRPO baseline while requiring x fewer solutions. "K" means x 1000.
AIME
AIME 25
AMC
MATH-500
Olympiad Puss@1
Model
Avg.
#Solution
Effic
Avg@5
Avg@5
Avg@5
Pass@1
Base Model
54.00
36.67
81.20
55.11
63.80
Outcome Reward based RLVR methods
AutoThink (Stage 1)
$2.67
32.67
77.59
90.00
54.07
61.40
450.6K
-0,53
ReusonFFuuxFF1
54.67
34.00
80.00
91.60
55.26
63.11
60K
-1.15
Adap[Think
$2.67
38.00
82.17
90.80
56.89
64.11
307. 207.2K
0.10
AReal Rteall-boha-RI...
60.00
44.67
86.02
93.60
60.89
69.04
2048K
0.37
Skywork- ORI-B
69.33
51.33
87.71
93.40
61.33
72.62
>5406.7K
<0.26
GRPO Replication (400 steps)
58.67
38.00
81,20
92.40
56.44
65.34
16K
9.6
Process Reward hased methods
TP-GRPO (214 steps)
60.67
43.33
83.13
92.00
57.04
67.23
8.56K
40.07
7B model, the gain rises +3.43, including +6.67 on AIME 24 and +6.66 on AIME 25. These consistent improvements over backbones validate the effectiveness of our process reward design.
Learning Efficiency over On-Policy GRPO. TP-GRPO also outperforms the on-policy GRPO baseline that relies solely outcome rewards. On the 1.5B model, TP-GRPO surpasses on-policy GRPO trained with 34K solutions (850 steps) using no more than 5.6K solutions (140 steps). Similarly, on the 7B model, TP-GRPO outperforms 16K-solution (400-step) on-policy GRPO run with only 8.56K solutions (214 steps). This highlights a notable improvement learning efficiency enabled by the proposed process reward mechanism.
Comparison with Other GenPRM Methods. On the 1.5B LRM, compare method with two other GenPRMs. Although they achieve slight improvements over the base model, their performance remains far below TP-GRPO. This indicates that process rewards in these methods were not effectively leveraged, likely due to those pitfalls we discuss in the Method section, further highlighting the necessity of designing an appropriate process reward mechanism.
Comparison of Efficiency with Other Methods, Against other perforerfor approaches (e.g.. 1.t.Scaler--.BB-Peee
and Skywork-ORI ORI-)B.. TP-GRPO demonstrates faster performance gains with substantially fewer training samples, despite these models achieving higher final accuracy through much larger budgets. While such comparisons are inherently imperfect due to the non-linear convergence patterns of LM training, our results still show that TP-GRPO matches exceeds competitive baselines with significantly less data.
Training Dynamics. Figure 3 further illustrates the Avg @ 16 curves AIME 24 for 1.5B experiments. Here, "off-policy GRPO" denotes off-policy GRPO variant without process rewards. All baselines are evaluated every 50 training steps. For TP-GRPO, we focus the impact of process rewards; therefore, only solutions with non-zere process rewards are retained. As result, the actual number of training steps per iteration is fewer than 50. Despite this, TP-GRPO's curve
8

---

Published as a conference paper at ICLR 2026
is noticeably steeper, indicating higher training efficiency. This provides strong evidence that process rewards enhance the exploitation of reasoning trajectories for LRM training.
0.5877
2010 Soez Nerg
132
420
polcy caino
02200
020
+
Stag
4
50 200
350
200
250
330
250
400
450
500
200
350
200
256
250
460
Figure 3: The trend of accuracy
Figure 5: Mutual information be- tween advantage and token cor- rectness for TP-GRPO and its variant w/o Step Merge.
(Avg@) 16) on AIME 2024 over training steps for DeepSeek-RI- Distill- Seaill-Qwen-1.5B experiments.
Figure 4: Advantage variance comparison between TP-GRPO and the variant w/o Step Merge.
4.3 ANALYSIS
We also conduct more detailed analyses below. Due to page limitations, more analyses are included in Appendix H.
4.3.1 ABLATION STUDY FOR PROPOSED STRATEGIES
We conduct ablation studies assess the contribution of our proposed three strategies. For the siiinsic-sinaa-driveee evaluation method, we replace it with direct judgment via LLM-as-a-judge. Detailed prompts are presented in Appendix. For the step-merge strategy, we omit merging and assign rewards directly at the step level. For the ability-adaptive process reward mechanism, we fix reward values +1 for correct or uncertain steps and -1 for incorrect steps. Table 3 shows that all three strategies yield clear improvements, confirming the necessity of addressing their respective target issues. Notably, the step-merge strategy yields the largest improvement, despite its remarkably simple implementation. To investigate this reason, we analyze token-level advantage values: (i) the variance across tokens within the same rollout, higher variance implies divergent optimization targets, leading to instability; and (ii) the mutual information between token advantages and the correctness of the corresponding step/thought (simplified as token correctness). correct tokens should have higher advantages. Figure 4 and 5 show that step-level rewards cause large variance and low mutual information. These outcomes arise as dense reward assignment enables consecutive incorrect steps to exert substantial negative impacts, which in turn distort, and even flip, the magnitude and sign of advantages in prior steps, thereby misleading the original optimization direction. By merging consecutive steps with the same correctness and assigning rewards at the thought level, this issue is effectively mitigated.
4.3.2 ABLATION STUDY FOR REWARDS
We design two distinct reward mechanisms for correct and incorrect solutions. To better understand their individual contributions, conduct ablation studies under two settings: one using only process rewards from correct solutions, and the other using those only from incorrect solutions. Results are presented in Table 4. We observe an interesting phenomenon: for AIME 24, using only correct- solution rewards yields better performance, whereas on AIME 25, the opposite holds. This contrast exactly offers compelling illustration of the distinct functions underlying the two reward schemes. Relying exclusively on rewards from correct solutions fails to mitigate the over-penaie peallienaiiotion for valid attempts in incorrect solutions, thereby constraining the model's exploratory capacity and ultimately yielding weak performance on rewards from incorrect answers precludes the targeted reinforcement of useful patterns within correct solutions, leading to fewer gains on both AIME 24 and AIME 25 relative the full TP-GRPO framework. In general, these results show that the proposed process reward mechanism supports effective learning from both correct and incorrect reasoning paths.
the more challenging task AIME 25. In contrast, focusing exclusively

---

Published as a conference paper at ICLR 2026
Table Ablation study on improvement strategies, "Stage I' is the intrinsic-signal-drie -rivenn evaluation. 'S1' step merge strategy, 'S2' refers to difficult--wwar adaptive reward formulation.
Table 4: Ablation study on process rewards. "CS reward' means 'reward for Correct Solutions', while 'IS' means 'Incorrect Solutions'.
Model
AIME 24
AIME 25
AMC 23
Model
AIME 24
AIME 25
TP-GRPO
33.12
25.63
64.01
Base Model
28.80
22.50
w/o Stage
31.04
23.54
63.93
TP-GRPO
33.12
25.63
-w/o Stage II/S1
31.66
22.29
62.19
wwo CS Reward
30.00
25.00
-w/o Stage I1/S2
32.71
22.92
63.47
w/o Reward
32.01
22.29
4.3.3 DIFFERENT LRMs FOR PROCESS EVALUATOR
To evaluate the dependency of our framework on the reason- ing capabilities of the PRM. we further conduct an analysis study. We first conduct experiments using different LLMs as the generative PRM, and then we implement GenPRM using the I.LM-as-a-Judge paradigm with these L.MMs. We select Qwen3- (2025a). whose
51.64
51.33
505
32B, Qwen3-4B and Gemma-3-1251 Team reasoning abilities GPQA Diamond (Rein et al., 2024) are 65.6%. 55.9%, and 40.9%, respectively'. We applied these three GenPRMs to both TP-GRPO and aJude,, with results shown in Table and Figure 6. Each entry is sepa- rated by "/", with the left value for TP-GRPO and the right for LLM-as-a-Judge- The results indicate that despite substan- tial differences in reasoning ability, all GenPRMs consistently improve LRM performance through TP-GRPO. Although per- formance slightly decreases as the evaluator's 's reasoning ability declines, the drop is minimal. In contrast, LLM-as-a- aa-dgee- con- sistently underperforms TP-GRPO, with pronounced decline. Results based on 1armma-3-12B-aa are notably worse than those based on Qwen3-32B and even below the base model. This suggests that existing generative process evaluation methods heavily rely on external GenPRM, especially TP-GRPO consistently delivers stable improvements across different LLM scales (Qwen3-32B vs. Qwen3-4B) and families (Qwen3 Gemma), further validating the effectiveness and robustness of our proposed process evaluation mechanism.
40.50
403
TP-GRPO
40.0
48.75
Owe13332 3-32
Gwon2-48
Gemmi3-12e 128
Model
Figure 6: Performance trend when using different LLMs as evaluators. The x-axis shows different evalua- tors, whose reasoning ability gradu- ally decreases from left to right.
reasoning ability. Moreover,
Table 5: Robustness Analysis with Different LLMs as GenPRMs.
Model
AIME
AIME
AMC
MATH-500
Avg.
Step
Base Model
28.80
22.50
62.90
82.80
49.25
TP-GRPO LLdd-es-aJJddee
-wwnn-32B
33.12/30.41 30.41
25.63 24.58
64.01 63.28
83.81
82.61
51.64
50.22
140 118
Qwwn3-4B
33.33 29.79
24.58 23.54
63.78 63.10
83.62 062//110 81.94
51.33 / 49.59
155 122
-Gemma---33 1-1b-it
32.71 30.20
23.33 / 22.50
64.76 61.25
83.73 81.28
51.13 / 48.75
125 /26
CONCLUSION
This work revisits the design of GenPRM and proposes two fundamental challenges: over-reliance on reasoning capabilities and a suppression of exploratory behavior. To address these, we distilled a set of guiding principles for reward model design, leading robust evaluation mechanism driven by intrinsic signals and a thought-level difficulty-aware adaptive reward formulation that dynamically balances exploration and exploitation. Integrating these components with process supervision GRPO, our proposed TP-GRPO algorithm achieves substantial performance gains over an outcome-only RLVR baseline, even under fewer training steps. Our results demonstrate the efficacy of process reward modeling for improving optimization efficiency for LRMs. Beyond the empirical findings, this study raises broader questions about how to design reward signals that encourage both accurate and diverse reasoning processes. We hope our work serves as foundation for developing more effective and generalizable GenPRM paradigms.
Data from: https: /www. dat earner com/ a -mode1s
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
The research in this article is supported by the National Science Foundation of China (U22B2059, 62276083), Key Research and Development Program of Heilongjiang Province (2022ZX01 A28) and the 5G Application Innovation Joint Research Institute's Project (A003).
ETHICS STATEMENT
This work does not involve human subjects, personal or sensitive data, or experiments that could raise ethical concerns. No new datasets containing private information were collected released. The methods, analyses, and applications presented do not have foreseeable negative societal impacts, nor do they pose risks privacy, security, fairness. We have conflicts of interest, sponsorship- related influence, or legal compliance issues to disclose. All research activities adhered to standard academic integrity practices, and ethics committee approval was required for this study.
8 REPRODUCIBILITY STATEMENT
8.1 IMPLEMENTATION
Our implementation is based on TRL (von Werra et al., 2020), and the complete code is provided in the Supplementary Materials. The training dataset used is the public DeepScaler-40K dataset, which can be found in huggingface requires no preprocessing. For evaluation, we adopt publicly available benchmark datasets, including AIME24, AIME25, AMC23, MATH-500, and OlympiadBench.
8.2 EXPLANATIONS FOR TWO PROPOSITIONS IN SECTION 3.2
Detailed proofs are provided Appendix
REFERENCES
Zachary Ankner, Mansheej Paul, Brandon Cui, Jonathan D Chang, and Prithviraj Ammanabrolu. Critique-out-ouud reward models. arXiv preprint arXiv:2408. h0t::2408.11991,, 2024.
Xiusi Chen, Gaotang Li, Ziqi Wang, Bowen Jin, Cheng Qian, Yu Wang, Hongru Wang, Yu Zhang, Denghui Zhang, Tong Zhang, et al. Rm-rl: Reward modeling as reasoning. arXiv preprint arXiv:2505.023887. 2025.
Zhaorun Chen, Zhuokai Zhao, Zhihong Zhu, Ruiqi Zhang, Xiang Li, Bhiksha Raj, and Huaxiu Yao. Autoprm: Automating procedural supervision for multi-step reasoning via controllable question decomposition. arXiv preprint arXiv:2402.1145: 2024.
Zhangying Feng, Qianglong Chen, Ning Lu, Yongqian Li, Siqi Cheng, Shuangmu Peng, Duyu Tang, Shengcai Liu, and Zhirui Zhang. Is prm necessary? problem- solving implicitly induces prm capability in Ilms. arXiv preprint arXiv:2505.11227. 2025.
Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, al. A survey on alm-as-a-judge. arXiv preprint arXiv:241 1. 15.55444 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning, arXiv preprint arXiv:2501. 12944 2025.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu. Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402. 14008, 2024.
11

---

Published as a conference paper at ICLR 2026
Tao He, Hao Li, Jingchang Chen, Runxuan Liu, Yixin Cao, Lizi Liao, Zihao Zheng, Zheng Chu, Jiafeng Liang, Ming Liu, et al. Breaking the reasoning barrier a survey on Ilm complex reasoning through the lens of self-evolution. In Findings of the Association for Computational Linguistics: ACI. 2025, pp. 7377-7417, 2025.
Dan Hendrycks. Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv: 103.0874, 2021.
Hugging Face. Open rl: A fully open reproduction of deepseek-rl, January 2025. URL https: //gi //github... cow cww.buggiiaaocooooooooo......... www.uugggggggggooo ingface// open-rl. rl.
Nimit Kalra and Leonard Tang, Verdict: A library for scaling judge judge-time time compute. arXiv preprint arXiv:2502.18018, 2025.
Muhammad Khalifa, Rishabh Agarwal, Lajanugen Logeswaran, Jaekyeom Kim, Hao Peng. Moon- tae Lee, Honglak Lee, and Lu Wang. Process reward models that think. arXiv preprint arXiv:2504. 16828, 16828,
2025.
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13(9):9, 2024a.
Jiawei Li, Xinyue Liang, Junlong Zhang, Yizhe Yang, Chong Feng. and Yang Gao. Pspo*: An effective process-superviee policy optimization for reasoning alignment. arXiv preprint arXiv:2411.1166 2024b.
Wendi Li and Yixuan Li. Process reward model with q-value rankings. arXiv preprint arXiv:2410.11287 2024.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
Tianqi Liu, Wei Xiong, Jie Ren, Lichang Chen, Junru Wu, Rishabh Joshi, Yang Gao, Jiaming Shen, Zhen Qin, Tianhe Yu, et al. Rrm: Robust reward model training mitigates reward hacking. arXiv preprint 24t.::2009.131666 2024.
Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu. Inference-time scaling for generalist reward modeling. arXiv preprint arXiv:2504.02444 2025.
Jianqiao Lu, Zhiyang Dou, Hongru Wang, Zeyu Cao, Jianbo Dai, Yuniong Feng, and Zhijiang Guo. Autopsv: Automated process-supervisee verifier. Advances in Neural Information Processing Systems, 37:79935-79968 2024a.
Jannqaa Lu, Zhiyang Dou, Hongru Wang, Zeyu Cao, Jianbo Dai, Yingjia Wan, Yinya Huang, and Zhijiang Guo. Autocv: Empowering reasoning with automated process labeling via confidence variation. arXiv preprint arXiv:2405.166802 2024b.
Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Meiqi Guo, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, et al. Improve mathematical reasoning in language models by automated process supervision. arXiv preprint arXiv:2406.0653 2024.
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li. Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing ol -p-peeee with a 1.5b model by scaling rl http://:///eetttrtttteee ott.aaasaaaaaiino-o-----eeeeeeee --g.::/eeeaaatt-----------a---a- wwwayyy---aagg--aaaaaaaaaaagaaae RL- 81902c1
iotion. www.oonniieeeeeeeieeeeee
2025. Notion Blog.
Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan-Philipe Fränken, Chelsea Finn, and Alon Albalak. Generative reward models. arXiv preprint arXiv:2410. /2832, 2024.
12

---

Published as a conference paper at ICLR 2026
math ai. Aime 2025 dataset, 2025. URL /ttp://ttt////ggggooo./..//////a
1/aime25.
Maxwell-Jia. Aime 2024 dataset, 2024. URL httpp::/hhuggigg.......aaaaaaaaa
Aal-Jia/AAMS _024.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. arXiv preprint arXiv:2501.19392 2025.
Skywork ol Open Series. An implementation of generative prm, 2024a. URL htthttps://githht comm RLHF /ww/ RLHF- MELLE--Rewadhe Rew--d-wddddeeeeeee--------- ling.
Skywork ol Open Series. Skywork ol team, 2024b. URL UKUttpss///uggiigafcc..co// rk.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Julian Michael, and Samuel R Bowman. Gpqa: graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024.
Julien Dirani,
Gregory Schraw and David Moshman. Metacognitive theories. Educational psychology review, 7(4):
351-371, 1995.
Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated process verifiers for Ilm reasoning. arXiv preprint arXiv:2410.08116, 2024.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi. Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits mathemat- ical reasoning in open language models. arXiv preprint 0tt::2402.033000 2024.
Shuaijie She, Junxiao Liu, Yifeng Liu, Jiajun Chen, Xin Huang, and Shujian Huang. R-prm: Reasoning-driven process reward modeling. arXiv preprint arXiv:2503.21295, 2025.
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rívière, et al. Gemma 3 technical report. urXiv preprini 2rXiv:2503. 19786, 2025a.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.125599 2025b.
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-basee feedback. arXiv preprint arXiv:2211. /4275, 2022.
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforcement learning. https::/ / gig.uub.hhh..... com http:hhhggggggcccggeaegaaheaeaee tt,, 2020.
Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. Is chatgpt a good nig evaluator? a preliminary study, arXiv preprint M.Xi::2033.0044. 2023a.
Jun Wang, Meng Fang, Ziyu Wan, Muning Wen, Jiachen Zhu, Anjie Liu, Ziqin Gong, Yan Song, Lei Chen, Lionel M Ni, et al. Openr: An open source framework for advanced reasoning with large language models. arXiv preprint arXiv:2410.09671, 2024a.
Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce lims step-by-step without human annotations. arXiv preprint arXiv:23/2.08935 2023b.
Tianlu Wang, Ili Kulikov, Olga Golovneva, Ping Yu, Weizhe Yuan, Jane Dwivedi- Yu, Richard Yuanzhe Pang, Maryam Fazel- Zaraandi,, Jason Weston, and Xian Li. Self-taught evaluators. arXiv preprint 4408 00262200..02666, 2024b.
13

---

Published as a conference paper at ICLR 2026
Zihan Wang, Yunxuan Li, Yuexin Wu, Liangchen Luo, Le Hou, Hongkun Yu, and Jingbo Shang. Multi-step problem solving through a verifier: An empirical analysis on model-induced process supervision. arXiv preprint arXfv:2402.02668,, 2024c.
Wei Xiong, Wenting Zhao, Weizhe Yuan, Olga Golovneva, Tong Zhang, Jason Weston, and Sain- bayar Sukhbaatar. Stepwiser: Stepwise generative judges for wiser reasoning. arXiv preprint arXiv:2508. 19229, 2025.
Yifei Xu, Tusher Chakraborty, Srinagesh Sharma, Leonardo Nunes, Emre Kıcıman, Songwu Lu, and Ranveer Chandra. Direct reasoning optimization: Lims can reward and refine their own reasoning for open- open-ended tasks. arXiv preprint (ttt::2506..333355 2025.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in Ilms beyond the base model? arXiv preprint arXiv:2504 13837, 2025.
Kaiwen Zha, Zhengqi Gao, Maohao Shen, Zhang-War Wei Hong, Duane S Boning, and Dina Katabi. RI tango: Reinforcing generator and verifier together for language reasoning. arXiv preprint arXiv:2505. 2025.
Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue. Yuxiao Dong, and Jie Tang. Rest-mets*; Llm self-training via process reward guided tree search. Advances in Neural Information Processing Systems, 37:64735-64777 2024a.
Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. arXiv preprint arXiv:2408.15240 2024b.
Yuxiang Zhang, Shangxi Wu, Yuqi Yang, Jiangming Shu, Jinlin Xiao, Chao Kong. and Jitao Sang. ol-coder: an ol replication for coding. arXiv preprint arXiv:2412.00154, 2024c.
Zhenru Zhang, Chujie Zheng. Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons of developing process reward models in mathematical reasoning. arXiv preprint arXiv:2501.07301, 2025.
Jian Zhao, Runze Liu, Kaiyan Zhang, Zhimu Zhou, Junqi Gao, Dong Li, Jiafei Lyu, Zhouyi Qian, Biqing Qi, Xiu Li, et al. Genprm: Scaling test-time compute of process reward models via generative reasoning, arXiv preprint arXiv:2504. 250.000991 2025.
Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu, Dayiheng Liu, Jin- gren Zhou, and Junyang Lin. Processbench: Identifying process errors mathematical reasoning. arXiv preprini arXiv:2412.06533 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging ilm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 06:46595-46622 2023.
Jiaru Zou, Ling Yang, Jingwen Gu, Jiahao Qiu, Ke Shen, Jingrui He, and Mengdi Wang. Reasonflux- prm: Trajectory-aware prms for long chain-of-thouglee reasoning in llms. arXiv preprint Ntfv:2506.18896, 2025.
A THE USE OF LARGE LANGUAGE MODELS (LLMs)
For this paper, LLMs were employed exclusively for textual polishing, without contributing to conceptual ideation.
14

---

Published as a conference paper at ICLR 2026
B PROOF OF PROPOSITION
Proposition 1. For a correct solution o,, token Oi,t in correct thoughts have advantage As, = -. while token Oi,t in incorrect thoughts have advantage Ai,
Proof. The proof is straightforward. After merging steps of the same type, the original think can be decomposed into an alternating sequence correct and incorrect thoughts:
think thought! thought! ineorrcc o ..... thought incorrect thought
Accordingly, the thought-level reward sequence is:
[ Tindex(1)>> (index(2))...>> ['indx((n)] [r", -r
f
where index (j) denotes the index the last token in the j-th thought. The last thought paneet receives only the outcome reward fo.
For tokens in a correct thought thought) the advantage value is computed as:
A correct index - 1) (inddexx()))) Tindex(j) + Timmex(s+1) +11 + ..... + Tn)
(5)
Given the alternating reward pattern, this simplifies to:
+...tr+ (12 (2-3)//2 terms
+ +... +po go.
(n-j)/2 terms
Similarly, for tokens in the incorrect thought thought the corresponding advantage is:
A A [ndex(j-1)<4sinned Tindex(j) + [index(j+1)) + ................................ + Tindex (n)
-.. + 80
(6)
(n-j+1)/2-1 times
(x-j+1)/2 times
80
Proposition 2. For an incorrect solution o;, token ot matched thoughts have advantage Ai,t = f.. while token C2 in unmatched thoughts have advantage Ai,t = 0.
Proof. Following the same reasoning as in Proposition 1, for an incorrect think, we first merge steps of the type and decompose the sequence into alternating matched and unmatched thoughts with respect to the answer. We assign penalty to matched thoughts with reward pinoorrect = fo, and reward unmatched thoughts with inccrreect =0
smanched the advantage of its tokens is computed as: For a matched thought P
A matched
index(j ())okj--iiicck T'inde(j)) (j) + (inde(j+1) + + Vindex(r) (n²)
70 +... (-F9) + ... + (-70) +
(n- f)/2 terms
(n-j)/2 terms
80.
Similarly, for an unmatched thought gunmarchod, the advantage is:
A index - 1) (inke(nde
Ij)dex(j) + Findex(i++)) + ... + Tindex(n)
80 + +.. p0
+ ((-0)) + + (-fo) +
(8)
(a (n-j+1)/2- terns
(m-j+1)/2 terms
0.
15

---

Published as a conference paper at ICLR 2026
C EXPLANATION FOR ADVANTAGE DESIGN
As stated in the paper, we adopt the process-supervised GRPO framework introduced in DeepSeek- Math Shao al. (2024), where the advantage estimator is computed using Eq. 2 and 3:
.indexij) mean Rt)
plndex())
Are
Σ
plndex(j)
std( R)
index())2t
Although this formulation differs from the classical form A Q - V, the normalized term implicitly serves the role of bascline by subtracting the group mean reward and normalizing variance, making A., variance-reduced relative return. Therefore, this relative return aligns with the conceptual purpose of advantage, namely to measure the return of an action relative to the expected return.
Building this formulation, TP-GRPO modifies the advantage estimator by only normalizing outcome rewards:
f1,0 75.0 mc
(10)
std(),,0)
This design is motivated by three considerations:
For correct solutions, we assign a reward of +r to correct thoughts and -r to incorrect thoughts. This construction ensures that the cumulative return for correct thoughts becomes R To. preserving the optimization target as in the outcome-reward-on) setting (as stated in Proposition 1). thereby preventing optimization drift caused by introducing process rewards. Normalizing all rewards would violate Proposition 1 and alter the optimization dynamics.
Normalizing across all rewards, including process-level rewards, would modify the relative scale between correct and incorrect outcome rewards, potentially weakening the role of outcome rewards as the primary global optimization signal. Instead, we aim to retain the original effect of outcome rewards while augmenting them with process-level guidance.
Under the advantage computation defined in Eq. 3, the resulting advantage for correct thoughts becomes = î. (as formally proven in Appendix B), which independent process rewards and benefits from variance reduction due to normalization. This aligns with the intended behavior of an advantage estimator as a relative, variance- variance-reduced training signal.
D RELATED WORKS
D.1 PROCESS REWARD MODELS
Process Reward Models (PRMs) aim to assess the quality of reasoning trajectories by assigning rewards to each evaluation unit unit-lypically a reasoning step step-coopleementing traditional outcome rewards (Li & Li. 2024; Setlur et al., 2024; Zhang al., 2025). Depending the evaluation paradigm, PRMs are typically categorized into discriminative and generative approaches (Zhao al., 2025). Discriminative PRMs learn to predict step-level correctness scores through supervised training (ol Open Series, 2024b;a). OpenAl's PRM800K dataset (Lightman et al., 2023), with large-scale human annotations, has significantly advanced this area. Wang al. (2023b); Luo et al. (2024); Wang et al. (2024c); Lu et al. (2024b;a) explore automatic labeling through simulation. However, discriminative PRMs often suffer from poor generalization, vulnerability reward hacking, and limited interpretability (Guo et al., 2025) due to their reliance on domain specific training and direct score prediction. In contrast, Generative PRMs treat evaluation et al., 2024; Zhao et al., 2025). Rather than predicting rewards directly, they analyze whether each reasoning step correct and use the generation probability of indicative tokens (e.g., "correct", "yes"") rewards (Zhang et al., 2024b; Mahan et al., 2024). These approaches offer greater interpretability and scalability by leveraging LLMs' intrinsic reasoning capabilities (Wang et al., 2023a). Many works adopt LLLM-as-a-Judeee framework (Zheng al., 2023; Gu et al., 2024), prompting LLMs for evaluation without further training (Zheng et al., 2024), and some others enhance evaluation via further training (Wang al., 2024b; Khalifa et al., 2025; Zha et al., 2025; Xiong et al., 2025).
generation task (Ankner
This work does not seek to explore how training might enhance the performance gains GenPRM; nevertheless, believe our framework can achieve additional evaluation improvements via training.
16

---

Published as a conference paper at ICLR 2026
Instead, it targets fundamental issues in the GenPRM workflow that have been overlooked in prior studies. These include whether evaluation should be grounded in reasoning ability or decomposed around other core capabilities, determining suitable reward granularity, and identifying the proper timing for applying process rewards. We hope our findings stimulate broader discussion within the community on these topics. Therefore, our contribution lies primarily in rethinking and innovating the philosophy of generative evaluation and reward design,
D.2 PRMS FOR OPTIMIZING LARGE REASONING MODELS
Prior to the release of DeepSeek RI, process rewards are widely explored as a means to improve the reasoning capabilities of LLMs (Zhang al., 2024a; Wang al., 2024a). Depending the application stage, process rewards are typically employed to guide trajectory pruning during shape optimization
inference (Uesato al., 2022; Kalra & Tang, 2025; Muennighoff al., 2025) during training (Zhang et al., 2024c). In this work, we focus on PRM's role in training. Early approaches such Rest-MCTS* (Zhang al., 2024a) train PRMs to rank reasoning trajectories, selecting high-quality traces for the learning process of reasoning. (Chen et al., 2024) estimates step rewards via question decomposition, but ultimately integrate them into the outcome reward, limiting the benefit of fine tin-grained supervision. A similar issue arises in (Li et al., 2024b), where process rewards are not effectively utilized. While OpenR (Wang et al., 2024a) employs PRMs to guide policy iteration in LLM training, it remains confined to the discriminative PRM paradigm. After the release of DeepSeek R1 (Guo et al., 2025), research attention on PRMs shifted toward Generative PRMs (ol Open Series, 2024a). For instance, ReasonFlux-PRM (Zou al., 2025) estimates process rewards to guide GRPO optimization but still integrates process rewards to outcome ones without fully utilizing the process rewards. To unlock the full potential of PRMs, several key challenges must be addressed, including ambiguous step segmentation, reward hacking, and continual retraining requirements. In response, propose suite of targeted improvements: comprehension- driven evaluation strategy, a thought-level granularity for assessment, and a capacity-adaptive reward mechanism.
E
MORE EXPERIMENT SETTINGS
E. TRAINING SETTINGS
For process-level evaluation, we employ Qwen3-32B as the primary evaluator model. To mitigate the substantial computational cost associated with this step, we implement several heuristic rules to filter out solutions that are unlikely to benefit from fine-grained process rewards:
Solutions that yield correct final answers but have concise reasoning chains (fewer than 4,096 characters) are excluded, as their intermediate steps are presumptively correct.
Solutions that produce incorrect and very short answers (fewer than 256 characters) are also filtered out, as they seldom contain discemible reasoning steps.
Solutions truncated due to exceeding maximum length constraints are similarly discarded.
Due to the application of the aforementioned filtering rules and the fact that some solutions receive no process reward after evaluation, the number of effective training samples in each TP-GRPO iteration is often insufficient to train 50 steps (more details can be found in Appendix G). As result, intervals.
TP-GRPO performs fewer than 50 steps per iteration and does not save checkpoints at fixed 50-step
Our implementation of TP-GRPO, as well as all replicated baseline experiments, built upon the GRPO algorithm. We incorporate two key modifications proposed in DAPO: the Chigh parameter and the use token-level loss. The outcome reward function combines accuracy and format rewards. We extract the answer from " "(boxed{]" and compare it against the golden answer: perfect match yields an accuracy reward of 1, otherwise O. Additionally, we observed collapse when reproducing the on-policy GRPO baseline with B 0. Therefore, following the open-rl implementation (Hugging Face, 2025), we set 8 = 0.04 for the GRPO baseline. Additional hyperparameters and configurations summarized in Table 6.
17

---

Published as a conference paper at ICLR 2026
Table 6: All key configuration hyperparameters used in this paper.
Hyperparameter
Value
Sampling Configuration
system prompt
(problem) | Let's think step by step and output the final answer within (boxed
temperature
1.0
max prompt length
1024
max response length
8192
rollout size
8
Process Evaluation Configuration
default model
Qwen3-32B
backend
VLLM
temperature max tokens
0.7
8192
censorpparallolsszcc(LLLLL
2
max_num_sequi max_num__eqs(VLLLN
512
Training Configuration gradient accumulation steps learning rate
1.0e-06
Ir scheduler type
constant
max steps max iteration num per device train rollout batch size
50
20
5
process reward coefficient a
Gow
0.2
Chigh
0.28
Evaluation Configuration max tokens
32768
temperature
0.6
top P top K
0.95
-1
gpu memory utilization _
1.0
E.2 EVALUATION SETTINGS
All evaluation experiments inperiments--incuudinn those for TP-GRPO and all baselines are conducted using the evaluation codes from the rllm project (Luo et al., 2025).
F PROCESS EVALUATION DETAILS
In this section, we provide the implementation details the process evaluation introduced in Section 3.1.
Overall of our generative process evaluation Pipeline We first revisit the overall process evalua- tion pipeline proposed Section 3.1. As illustrated in Figure 7, the pipeline evaluates solutions in three sequential steps. In the first step, we decompose the reasoning process into individual steps and apply different strategies to determine each step's type based on the correctness of thinking content. In the second step, consecutive correct or incorrect steps grouped together to form thought-level segments. The rationale behind this approach is supported by the results shown Section 4.3.1. In the third step, adaptively assign different rewards to correct and incorrect thought segments based on the model's capability for the given problem, thereby effectively balancing exploitation and exploration during training.
18

---

Published as a conference paper at ICLR 2026
1. Intrinsic Sigmal-driven Step-level Process Evaluation
2. Thought-level Process Evaluation
through Step M Merging
1.1 Evaluation for Correct Solution
Sepp 1: Alight solve the
cthinio
each step given error cause
Step
Though /
Stpp Suppose two sows and four coturea Stop mee stceept....
Step 2
Though 2:
QIO
Identity the first arroi step (error cause) given:
Step-l:
Wal,
mistake
the reflectony
( correct ansuer)
Identify reflection posttion that find out errons
Step 0
Thougl 3: P
</tbinnk
Step 6.
For conrect
solution
1.2 Evaluation Incorrect Solution
3. A.Aiiiyyy- -ddpttaion Process
For incorrect solution
Reward Mechanism
Solutãon:
Ahakk
Stp.
Stop
Stop 3
Strp
-think> A/thiik>
matto
Cormo Solution
Incomect Solution
Cansswers ... Slanswege
DOSSWOTT
Skpp
Saep
Stog
Thonght I:
Thought I:
2
Thonght
Though
Thought 3:
Thought 3:
Problem
On stright line le 100 points and heo point outside the line. What the maxt mum number of isosoilles ttongles that be forned using these 101 points vertices?
Thought kt
Though l:
Figure 7: The pipeline of intrinsic signal-driven thought-level generative process evaluation. The whole process consists of three steps: (1) Intrinsic Signal-driven Step-level Process Evaluation, which evaluates each reasoning step using intrinsic signals from solutions; (2) Thought-level Process Evaluation through Step Merging, which merges continuous steps with the same correctness to enable thought-level evaluation; and (3) Ability-Adaptation Process Reward Mechanism, which calculates rewards based on model capability, balancing exploration and exploitation.
Atomic Logical Actions- Baaoed Decompose Each solution should first be decomposed into a series of steps with appropriate granularity for subsequent processing. Given that model-generate solutions are often lengthy and unstructured, directly segmenting them into content-complete steps with clear boundaries is challenging. Our decomposition begins by splitting the solution into shorter sentences using the "\n),n" nn delimiter. However, we observed in practice that the model often generates ""nn),n" in the middle of process, leading to excessive fragmentation. To address this, we curated list of punctuation marks (detailed in Table 15) that reliably signify the end of a semantically complete, fine-graid graiigr sentence. We then sequentially merge consecutive short sentences until the merged segment terminates with one of these specified punctuation marks. For ease of subsequent processing, then insert sequentially increasing label (e.g., <0>, <I>, ...) at the beginning of each merged sentence. To ensure that each sentence has an appropriate semantic granularity, we define eight atomic logical actions and prompt GenPRM to further merge consecutive sentences based on these actions. Ultimately, for each solution we obtain a series of steps with appropriate quantity and granularity, as well as clear semantic boundaries. The prompt used is shown in Table 17. For solutions marked as incorrect, the corresponding final answer also requires decomposition. As answers are considerably shorter than the full reasoning trajectory, we bypass the initial punctuation- based merging and directly apply the atomic logical action decomposition. Specifically, GenPRM is prompted to enclose each identified step within "<step>" and "</step>" tags. It facilitates the subsequent parsing steps using regular expressions, and mitigates the risk of GenPRM corrupting the original answer (e.g., by rewriting or omitting parts). The prompt for answer decomposition shown in Table 18.
Match-Based Step Type Check for Incorrect Solutions Following the decomposition, each step within an incorrect solution is assessed by performing a semantic match against the corresponding answer (Section 1.1. For solutions that remain excessively long (i.e., exceeding 8 8192 characters), we apply an additional step-level summarization using GenPRM. The prompt used is shown in Table 19. This makes them more concise for subsequent processing. We focus solely on semantic and content similarity, disregarding whether the solution steps themselves are correct or not. The prompt used can be found in Table 20.
Reflection-Based Step Type Check for Correct Solutions For solutions that yield correct final answer, we assess the correctness of each intermediate step using a three-stage methodology: Localization, Traceback, and Verification (Section 3.1.1). First, in the Localization stage. the step-decomposed solution is provided to GenPRM, which is tasked with identifying all reflection segments. To filter out generic validation statements (e.g., "double-checking"). constrain the
19

---

Published as a conference paper at ICLR 2026
Solution
Question
Palicy Model Ni
O1
diought..
thoughte polrrett
thought,
GenPRM Pipeline
Cornet
02
Policy Modd
Dataset
Incorrect Solution A-
Mg
On
Dataset with
thought,
thoouhhty
Solution Sampling
Process Evaluation
Policy Optimization
Figure 8: The overview our proposed efficient off-policy training pipeline. This design ensures full utilization of GPU resources each stage-all all GPUs are either concurrently used for sampling, process evaluation, or parameter updates, avoiding any idle computation.
model to identify only those reflections that pinpoint concrete errors. The prompt for this task is detailed Table 21. Second, during Traceback, we iterate through each identified reflection to trace its corresponding error source, defined the earliest step where the specified error first occureed (see prompt in Table 22). After identifying the source for each reflection, we retain only the one with the earliest index, this represents the ultimate origin of the error. If multiple reflections map to this same earliest source, we select the reflection with the latest index to define the end of the error-reflection span. All steps within this span considered semantically related the source Finally, in the Verification stage, GenPRM classifies each step within the identified span into one of three types: Incorrect, Reflection, and Correct.
Incorrect: A step containing logical or computational enror.
Reflection: step containing a reflective or self-correcting action.
Correct: step that is neither Incorrect nor a Reflection.
The prompt used for this classification task is shown in Table 23. This multi-stage workflow is implemented by parsing the JSON output from each stage and using as input for the subsequent stage.
G OFF-POLICY TRAINING PIPELINE DETAILS
Conventional on on-policy RL training alternates between sampling rollouts and performing policy updates, often on separate sets of GPUs. This approach can lead to significant GPU idling and low overall utilization, policy update hardware waits for the sampling phase to complete. To overcome this inefficiency, we propose an off-policy training framework, depicted in Figure 8, which decouples these phases into three distinct steps:
Solution Sampling:
The policy model is deployed using the vLLMM backend. A batch of prompts sufficient for n update steps (n=50 our experiments) prepared, and the model generates responses for all prompts. The resulting data is saved locally.
The policy model is unloaded from the GPUs, and our GenPRM is deployed, also using the vLLM backend. The process evaluation pipeline, described in Section 3. is then applied to generate process rewards for the data collected in the previous step.
Process Evaluation:
Policy Optimization: Finally, GenPRM is unloaded, and the policy model reloaded for training. The model is then trained on the cffline-collectdd, steps. This completes one full iteration of our framework, and the cycle is repeated until training converges.
data for the target number of
H MORE EXPERIMENTS RESULTS
H.1 HYPERPARAMETER ANALYSTS
We further analyze the hyperparameter a to assess its impact on model performance. Experiments conducted the 1.5B model with (0,0.5, 1, 2, 4), and the results on AIME 2024, AIME
20

---

Published as a conference paper at ICLR 2026
Table 7: Hyperparameter Analysis with different settings.
Model
AIME 24
AIME 25
AMC 23
Avg.
Base Model
28.80
22.50
62.90
38.07
TP-GRPO (a =
30.00
25.00
62.50
39.17
TP-GRPO (a 0.5)
31.46
24.17
63.78
39.80
TP-GRPO (a
33.12
25.63
64.01
40.92
TP-GRPO (a 32.29
25.00
64.46
40.58
TP-GRPO
31.46
23.33
64.38
39.72
2025, and AMC 2023 are summarized Table 7. The results show that a = achieves the best performance. This can be explained by the advantage computation for incorrect thoughts:
= p
p0 aCCG-
(11)
When 1, we have 0 < A < 8o, meaning that incorrect thoughts receive positive but reduced advantage values-effectival decreases, this regularization weakens; when 0, the mechanism reduces outcome-only off- policy GRPO, losing the benefits of process-aware credit assignment for correct solutions. Conversely, when a the penalty intensifies, potentially driving 0, which may over-suppress erroncous reasoning paths and hinder the LRM's ability to engage in productive trial-and-crror exploration.
suppressing their optimization strength relative to correct ones. As a
H.2 ANALYSIS ON OFF-POLICY TRAINING PIPELINE
In this work, we propose an off-policy training pipeline to reduce training time by reducing GPU idling. According the results in Figure 3, evident that off-policy GRPO underperforms on-policy GRPO in both accuracy and training efficiency, indicating that on-policy training is more
conesssennnnnst with conventional training experience. Figure 3 presents a clearer comparison, where on-policy GRPO demonstrates more pronounced improvement trend over training steps. To further investigate the impact update frequency off-policy training, we reduce the update interval of TP-GRPO's off off-policy pipeline from 50 steps to 25 steps (and to step in an extreme case, which essentially corresponds to true on-policy updates). We observe that both AIME 24 and AMC 23 tasks show further performance improvements. In summary, although off-policy training is known to less effective than on-policy training, TP-GRPO trained under the off-policy setting still outperforms the on-policy GRPO baseline. This further validates the effectiveness of our proposed GenPRM mechanism.
Table 8: Analysis study on the impact of off-policy training.
Model
AIME 24
AIME 25
AMC 23
Training Steps
On-policy GRPO
32.71
24.58
64.53
850
Off-policy GRPO
30.83
24.79
62.05
950
TP-GRPO (25 steps)
33.54
23.96
64.53
207
TP-GRPO (50 steps)
33.12
25.63
64.01
140
H.3 TRAINING EFFICIENCY COMPARISON
We evaluated the training efficiency of TP-GRPO against other high- pethhrrrffrinngggg methods, with corresponding hyperparameters summarized in Table 10. Our method demonstrates substantial improvements training efficiency on both the ----Sek-----istll--- Distill- Distill-Qwen-71 models. Relative to the baseline GRPO algorithm, TP-GRPO accelerates training by a factor of to 8 while also achieving superior performance, as detailed in Section 4.2.
1.5B and DeepSeek-Re -
H.4 CASE STUDIES
To intuitively compare the improvements our proposed TP-GRPO algorithm over the original GRPO, selected one problem each from the AIME24 and AIME25. These problems were then
21

---

Published as a conference paper at ICLR 2026
Table 9; Analysis study on the impact of off-policy training.
Model
Traing Steps
Batch Size
Rollout Size
Effic.
DeepSeck- Ri-tts-ll-----lllll--- Meil--Dil---Qe--LLLE
AutoThink (Stage 1)
220
128
-0.53
AdaptThink
300
128
0.53
Open-RS1
100
96
0.57
Still-3-1.5--Prre
250
128
0,98
Ser.L--.S--PreiiwwSSeee 1)
200
1024
0.13
DS-I.SB-Preeiew
1750
128.128.128
8,16.16
<0.49
GRPO Replication
850
5
8
4.65
TP-GRPO
5
36.43
DeepSeek- R1 -Distilll Sestill-Qwen-7ee Qwen-7B
AutoThink (Stage 1)
220
128
-0.53
ReasonFlux-Fi
3750
16
-1.15
AdaptThink
150
128
0.10
AReaL-boba-RL--B
250
512
0.37
Skywork-ORI
1320
256,160
16,32
<0.26
GRPO Replication
400
5
9.6
TP-GRPO
214
5
40.07
Table 10: Training hyperparameters for efficiency comparison.
presented the BrasSSek-RR---isill------ee 1.5B and DeepSeek-RI -DistlllQwenn-77 models, which were trained using both algorithms respectively, to observe their corresponding reasoning processes. model are shown in Table 11 and
The detailed results for the DeepSeck-PR -I-D-stilll-wenn Table 12. while the detailed results for the DeepSeek-PP R----Riiillll-eeeeeree- Table 13 and Table 14. observed from the model's thinking content, the model post-trained with the TP-GRPO algorithm can correct the specific reasoning errors made by the model trained with original GRPO. This provides an intuitive demonstration of the effectiveness of our proposed algorithm.
model are shown in
Case Study for 1.5B Model with GRPO
[Question]
Alice chooses set A of positive integers. Then Bob lists all finite nonempty sets B of positive integers with the property that the maximum element of belongs to Bob's list has 2024 sets. Find the sum of the elements of A. Let's think step by step and output the final answer within \boxed }.
[Model Thinking Content]
Okay, so Alice picks A positive integers. Then Bob lists all finite nonempty sets B of positive integers where the maximum element of B is in A. The total number of such sets B 2024. We need find the sum of the elements A.
So. if A has elements, then the total number of such sets B is the sum from k=l to of 2 Because for each element dg A, the number of subsets where ag is maximum is 2
Therefore, the total number of sets B is 2" since the sum from k=0 to n-1 of is 2" 1
A has elements aj, ag, an, then the total number is 2 -2 + +... 2.. 2024
Now, 2024 can be expressed as a sum of powers of 2. Let's see, 2024 in binary is 11111101100, which is 1024 512 + 256 128 + 64 32 4. Wait, but let me compute that.
Wait, 1024 is 200 512 is 29, 256 25, 128 is 27, 64 26, 32 is 5, 16 is 24, 8 is 23, is 22, 2s 21, is 29, So, 1024 + 512 1536, plus 256 is 1792, plus 128 is 1920, plus 64 is 1984,
22

---

Published as a conference paper at ICLR 2026
plus 32 2016, plus 4 is 2020. Wait, that's 1024 + 512 256 128 + 64 + 32 +4 2020. Hmm, but 2024 is 4 more, so 2020 + 4 is 2024. So, in terms of powers of 2, 2024 20 2 2 2 2 2 222
So, the exponents 0; are 2, 5, 6, 7, 8, 9, 10. Therefore, a; are 3,6,7,8,9,100111 Therefore, the set A is 3,6,7,8,,,10011111 Therefore, the sum of the elements of A is + + +11.
Therefore, the sum of the elements + 8 10 Let me compute that:
is 9.
947 is 16.
16 +8 24.
24 49 is 33.
33 13410 is 43.
43 +11 is 54.
So, the sum is 54.
Therefore, the answer 54.
Final Answer: The sum of the elements of A is
Table I1: Case study of DeepSeek-RI-Dissill---eeeee model incorrectly answered 54, while the correct answer is 55. Due to the model's extensive thinking process, we have excerpted the key steps for clarity. As shown, the error (highlighted by orange) stems from a miscalculation the binary representation of 2024.
with GRPO. For the given problem, the
Case Study for 1.5B Model with TP-GRPO
[Question]
Alice chooses a set A of positive integers. Then Bob lists all finite nonempty sets B of positive integers with the property that the maximum element of B belongs Bob's list has 2024 sets. Find the sum of the elements of A. Let's think step by step and output the final answer within \boxed().
[Model Thinking Content]
Okay, have this problem here where Alice chooses set A of positive integers. Then Bob lists all finite nonempty sets of positive integers with the property that the maximum element of B belongs A. Bob's list has 2024 sets. need to find the sum of the elements of A. Hmm, let me try to break this down step by step.
So for each a in A, the number of such sets is equal to the number of subsets of 1, 2, .... a-l. Because each subset can be combined with form a B with maximum
Therefore, for each a in A, the number of sets B with maximum is 2" Because the number subsets of a with n elements is 2n So this case, n a-l, so it's 2-
So, if A is set of numbers, say A = a1, 02 ..... ag. then the total number of sets B that Bob can list is the sum over each a; in of ii Because each a; contributes 2 sets. Therefore, the total number of sets B is the sum of 2 for all a, in A. So, let me denote that S DA 2a-1 2024.
So, let's factor 2024 into its prime factors to see we can represent as sum of powers of 2.
First, 2024 divided by 2 is 1012, divided by 2 again is 506, divided by 2 again is 253. 253 is not divisible by 2, so the prime factors are 23 253. Now, 253 divided by 11 is 23, because 11 is 253. So, 2024 2 x 11 x 23. So, 2024 8 11 23. Hmm, interesting.
23

---

Published as a conference paper at ICLR 2026
But how does this help me? need to express 2024 as the sum of 2 terms, each which power of 2, and each is positive integer. So, each term is power of 2, and the sum is 2024. So, to find A, need to express 2024 as the sum of distinct powers of 2
So, let me compute the binary representation of 2024, Calculating:
210 1024
2 512
2 256
2 128
26 64
2 = 32
2 = 8
So, summing these up:
1536 1792
1792 128 1920
1920 + 64 1184
1984 +2 2016
2016 8 = 2024
Yes, that adds up correctly. So, 2024 in binary is 11111101000, So, the binary representation has 1s at positions 10,9,8.7,6,5,3. So, that would mean that the exponents in the sum are 10,9,8,7.6,5,33.
So, if the exponents 10,9,8.7,6,5,3, then the corresponding a's 11,10,,,8,7,6444 Therefore, set A is 1,6,7,8,9,10,111. So, the sum of elements in A 4 + Let me compute that:
10 11.
4+6 10
10+7=17
17+8
25
34+ 44
44+11=55 55
So, the sum 55.
Final Answer: The sum of the elements of A is 55
Table 12: Case study of -eepsSee--R--isiil-----e-eeee
with TP-GRPO. Using the same problem from Table 11, also present only the key steps. Post-training with TP-GRPO, the model successfully computes the binary representation of 2024 and yields the correct final answer. We use blue to highlight these steps.
Case Study for 7B Model with GRPO
[Question]
Let A be the set of positive integer divisors of 2025. Let B be a randomly selected subset of A. The probability that B is nonempty with the property that the least common multiple of its elements is 2025 is " where m and n are relatively prime positive integers. Find m Let's think step by step and output the final answer within Aboxed
[Model Thinking Content]
24

---

Published as a conference paper at ICLR 2026
Okay, so have this problem here where I need find the probability that randomly selected subset B of the set A (which consists of all positive integer divisors of 2025) has the property that the least common multiple (LCM) its elements 2025.
I know that 2025 divided by 25 is 81, because 25 times 81 2025. And 81 is 9 squared. so 2025 is 25 times 81, which is 5 squared times 3 to the fourth power. So, 2025 5 Therefore, the prime factorization of 2025 5234.
Now, to find all the positive divisors of 2025, can use the exponents of its prime factors. The number of divisors is given by multiplying one more than each of the exponents in the prime factorization. So here, for 52, the exponent is 2, we have 2 1 3. For 3 the exponent is 4, so 4 + 1 5. Therefore, the total number of divisors x 5 = 15. So, set A has 15 elements.
Next, we need to consider all possible subsets B of A, and find the probability that the LCM of the elements in B is 2025. Since we' re dealing with subsets, the total number of subsets is 2 because each element can either be included or excluded. However, the problem specifies that B is nonempty set, so we have to subtract 1 for the empty set. Therefore, the total number possible nonempty subsets is 2 1.
Wait, actually, let me think. The problem says "a randomly selected subset of A." It doesn't 't specify whether itš nonemply or not. But then, it goes on to specify that B is a nonempty set with the property that LCM of its elements is 2025. So, perhaps the probability is considering all subsets, including the empty set, but the event we're we're interested in only when B is nonempty. Hmm, the wording is bit ambiguous.
Looking back at the problem: "The probability that B is a nonempty set with the property that the least common multiple of its elements is 2025." So, it's given that B is a nonempty set, so we can consider the total number of subsets as 215 1, since we exclude the empty set. Therefore, the probability is (number of desined subsets).
(215-1)
Given that 2025 35 the LCM of a subset B is 2025 if and only if the subset contains elements that cover all the prime factors with their maximum exponents.
By inclusion-exclusion this number is U ||| So, |A| the number of subsets without 81. Since 81 is one specific element, the number of subsets without 81 is 214
B + A Bl.
Similarly, [B] is the number of subsets without 25, which is also 2
IAn BJ is the number of subsets without both 81 and 25, which is 2/3 Therefore, the number of subsets containing at least one 81 and at least one 25 is:
215 211-221__1 224 4 213
Let me compute this:
215 32768
214 = 16384
213 8192
So, 32768 16336-444- 16384 8192.
Compute step by step:
32768 - 16384 16384
16384 16384 0 8192 = 8192
So. the number of subsets containing at least one 81 and at least one 25 is 8192.
Therefore, the probability (21511
32767
Now, we need to simplify this fraction. Let's see if 8191 and 32767 have any common factors.
First, let me factorize 32767. I know that 32767 is 2 - 1. which is 32768 = 32767. 32767 is a Mersenne number, 215 1. Let's see it's prime. Well, 25 is 32768, so 32767 is 32768 1. Let check its divisibility.
327677 74681 = 32767? Let's s check: 7 x 4000 = 28000, 7 681 4767, so 28000 4767 32767. Yes, so 32767 7 4681.
Now, let's factor 4681. Let me check if 4681 is divisible by 7: 7 x 668 = 4676, 4681 - 4676
25

---

Published as a conference paper at ICLR 2026
5, so Next, 11: 6 5, which is not divisible by 11. 13: 4681 13. 13 x 360
4680, so 4681 - 4680 = 10 no. 17: x 275 = 4675, 4681 4675 6, not divisible. 19: 19 1 no. 23: 23 x 203 4669, 4681 4669 12, no. 29: 29 x 161
246 4674,
-4669, same above. 31: 31 x 151 4681? Let's compute 31 150 = 4650, plus 31 =
4681. Yes! So, 4681
151.
Therefore, 32767 = 31 x 151.
Now, let's check if 8191 is divisible by any of these primes: 71 151.
8191 7: 7 x 1170 8190, so 8191 8190 s0 remainder 1. Not divisible by 7.
8191 31: Let's sec, x 8191 151: 151 x 54 8154, 8191 - 8154 37, which is less than 151, not divisible.
8191 8184 = 7, so remainder 7. Not divisible by 31.
Therefore, 8191 and 32767 share no common factors besides 1, so the fraction 8191 is 32767
already in its simplest form.
Therefore, m 8191, 32767, and m + 40958.
I think that's s solid. don't t see any mistakes in the reasoning.
Final Answer: 40958
Table 13: Case study of DeepSeek-R -i--iiistl-Qeenn with GRPO. For the given problem, the model incorrectly answered 40958, while the correct answer is 237. Due to the model's extensive thinking process, we have excerpted the key steps for clarity. shown, the error (highlighted by orange) mainly from miscaleulation in the number of corresponding subsets.
Case Study for 7B Model with TP-GRPO
[Question]
Let A be the set of positive integer divisors of 2025. Let B be randomly selected subset of A. The probability that B is nonempty set with the property that the least common multiple of its elements is 2025 is n where and relatively prime positive integers. Find m + n. Let's think step by step and output the final answer within \boxed }.
[Model Thinking Content]
Okay, so have this problem here where need to find the probability that a randomly selected subset B of the set A (which consists of all positive integer divisors of 2025) has the property that the least common multiple (LCM) its elements is 2025.
...
is the set of all positive integer divisors of 2025. So, let me factorize 2025 to find out square. Let check: 45 squared is 2025 because 45 45 is 2025. So, 2025 = 452. But itself can broken down further: 45 9 5, and 9 is 32, So, 2025 is 3 x 52. Therefore, the prime factorization of 2025 is 31 x 52.
its divisors. 2025 is number I might recognize
Now, to find the number of positive divisors, can use the formula for the number of divisors based on prime factorization. If a number N has the prime factorization N Pt *p *p. then the number of positive divisors is (a, 1)(a, + ()...aa, + 1). So, for 2025, which is 3 x 52, the number of divisors (4 11) (2 + = x = 15. So, set A has 15 elements.
Next, I need to consider the power set of A, which is all possible subsets of A, including the empty set and A itself. The number of subsets set with elements is 2n, Since A has 15 elements, the power set has 25 subsets. Therefore, the total number of possible subsets B is 32768.
Now, the problem is asking for the probability that randomly selected subset B has LCM of 2025. So, need to find how many such subsets exist where the LCM of their elements is 2025, and then divide that by the total number of subsets, which is 32768.
Given that 2025 is 34 52, the LCM of a subset will be 2025 if and only if in the subset,
26

---

Published as a conference paper at ICLR 2026
there is at least one number with 34 and at least one number with 52. Wait, that correct? Or it sufficient that the maximum exponent of 3 in the subset is 4 and the maximum exponent of 5 is 2? Yes, exactly. So, for the LCM be 2025, the subset highest power of 3 in the subset is 3 and the highest power of 5 is 5". So, in other words, the subset must contain at least one multiple of 3* (which is 81) and at least one multiple of 5 (which is 25).
include numbers such that the
Therefore, the number of subsets with LCM 2025 is equal to the number of subsets that include at least one multiple of 81 and at least one multiple of 25, So, to compute this, we can use the principle of inclusion-exclusion.
First, let's denote:
Let S be the set A, which has 15 elements.
Let X be the of subsets of S that include least one multiple of 81.
Let Y be the set of subsets of S that include at least one multiple of 25.
We need find [X Y|, which is the number of subsets that include least one multiple of 81 and at least one multiple of 25.
By the principle of mnclusion-excluiiin
XX nnn |X]] + YY| [X Y So, let's compute each term.
Total subsets: 215
Subsets without any multiple of 81: So, we need count the number of subsets that do not include 81. 2025 34 5", so the divisors are of the form 3 3 where a is 0,1,2.3,4 and is 0,1.2. The number of divisors is 15. The number of divisors that are multiples of 81: 81 is 34, so any multiple of 81 must have a=4. So, the number of such divisors is equal to the number of possible b's, which are 0,1,2. So, there are 3 divisors that are multiples of 81: 81 =1 81, 81 5 405, 81 25 2025.
Therefore, the number of elements not divisible by 81 is 15 3 = 12. Therefore, the number of subsets without any multiple of 81 is 2
Similarly, subsets without any multiple of 25: 25 is 52. So, the divisors that are multiples of 25 are those with b=2. The number of such divisors equal to the number of a's, which
are 0,1,4,3,4 So, that's 5 divisors: 1, 5, 25, 75, 225, 2025? Wait, no: 1 is 3 59, 5 30 x 57, 25 is 30 x 52, 75 is 33 52, 225 3 x 52, 2025 34 x 52. Wait, that's divisors. Wait, but 2025 is 34 5", so for b=2, can be 0,1,233,4: that's 5 divisors. Wait, 39 52,3 52, 52,....? 3 x 52: that's divisors. Wait, why did think 6? Hmm, maybe confused with 2025's total divisors.
Wait, no, the number of divisors where b-2 5 because a can be 0,1,2,3,4. So, 5 divisors. Therefore, the number of divisors not divisible by 25 is 15 5 10. Therefore, the number of subsets without any multiple of 25 is 210
Now, the subsets that include neither any multiple of 81 nor any multiple of 25: So, these are subsets that do not contain any of the multiples of 81 and do not contain any of the multiples of 25. So, how many such elements are there? Total elements: Number of elements that are multiples of 81 or 25: Let's compute this using inclusion- exclusion. Number of multiples of 81:3, multiples of 25:5, and multiples of both 81 and 25: Well, 81 and 25 are coprime, so their least common multiple is 81 x 25 = 2025. So, only 2025 is multiple of both 81 and 25. Therefore, the number of elements that are multiples of 81 or 25 is 3 3- Therefore, the number of elements that are neither multiples of 81 nor multiples of 25 is 15 8.
Therefore, the number of subsets that include neither any multiple of 81 nor any multiple of 25 is 2².
Therefore, putting all together:
[X NYl Total subsets subsets without 81 subsets without 25 + subsets without both 81 and 25
Which is:
27

---

Published as a conference paper at ICLR 2026
X Tnnn Y / = 21 212 2210 28
Let compute these values step by step.
First, 21 -32768
212 4096
2 = 1024
2 =226 So, substituting:
[X n XY = 32768 4096 1024 256
Let compute this step by step:
First, 32768 -4096 = 28672
Then, 28672 - 1024 27648
Then, 27648 + 256 27904
So, XX nY] 27904
Therefore, the number of subsets B where LCM(B) = 2025 is 27904.
Therefore, the probability 32768 Now, we need to reduce this fraction to its lowest terms. Let's find the greatest common divisor (GCD) of 27904 and 32768.
27901
First, let's note that both numbers are divisible by 16: 27904 + 16 = 1744, 32768 16 = 2048.
So, 1744 and 2048.
Again, both are divisible by 16: 1744 16= 109, 2048 + 16 128.
So. 109 and 128. Now, 109 is prime number (since it's not divisible by 2,3,5,7,11;
V109 10.4, so primes up to 11 are tested). 128 is 27. Since 109 is prime and doesn't divide 128, the GCD is Therefore, the reduced fraction is 1744 Wait, no: wait, 27904 16 1744, 32768 16 =
2048. Then, 1744 + 16 109, 2048 + 16 128. So, the reduced fraction 128 Wait, but 1744 +14=10977 Wait, 16 (6110=17744? Let me compute 16 100=1600, 16*9=144, so 1600+144=1744, Yes. correct. Similarly. 16* 128=2048, which is correct. So, 109 and 128 are coprime, so the reduced fraction is 109 Therefore, the probability is 109 Thus, m + n=109 128 237.
109
128
128 m=109, n=128.
Wait, hold but let me confirm the calculations more because 27904 divided by 109 which is correct.
32768 reduces by 16 twice, leading to 128
Therefore, the probability 27904
109
m + 237.
32768
128
Thus, I think that is the correct answer. Final Answer: 237
Table 14: Case study of DeepSeek-RR -Iit---I---itilll-e------ with TP-GRPO. Using the same problem from Table 13, we also present only the key steps. Post-training with TP-GRPO, the model successfully computes the number of corresponding subsets and yields the correct final answer. We use blue to highlight these steps.
Punctuation Marks List
[.,
!, ?]
....
Table 15: The punctuation marks list use to merge consecutive pre-split sentences.
28

---

Published as a conference paper at ICLR 2026
Prompt for L.L.- ---M-as---judge
I will provide math problem along with a solution. They will be formatted as follows: [Math Problem]
cmath_ppoblent
(..(math problem)...
[Solution]
<step_1> 1
.s.(step 1 of solution)...
</step_11
<step_n>
(step n of solution)...
</step_n>
Your task is review each step of the solution in sequence, analyzing, verifying, and critiquing the reasoning in detail. You need to provide the analyses and the conclusion the following format:
<analysis_
(analyssss of step 1)...
</analysis 1>
<analysis_ n >
...aallyssis of step n)...
</analysis_n n>
<conclusion
Correct/Incorrect
</conclusion>
When you analyze each paragraph, you should use proper verification, recalculation, or reflection indicate whether it is logically and mathematically valid. Please elaborate on the analysis process carefully.
If an enror is detected in any step, you should describe the nature and cause of the error in detail, and suggest how to correct the eтror or the correct approach. Once a step is found to contain any identified error) and directly provide the conclusion of "Incorrect." For instance, given a solu- tion of five steps, if an error found in the third step, you should reply in the following format:
stop further analysis of subsequent steps (as they may depend on the
<analysis_ 1
..nallsssss of step 1)...
</analysis >>
<analysis_2 2
.analyssis of step 2)...
</analysis_24 2
<analysis_3
..naallyssi of step 3; since error is found here, also provide detailed critique and correction
29

---

Published as a conference paper at ICLR 2026
guideline)...
</analysis_3 >
<conclusion >
Incorrect
</conclusion
Note that the analyses of steps 4 and 5 should be skipped as the step 3 has been found to contain an error.
* Respond with your analyses and conclusion directly
The following is the math problem and the solution for you task:
[Math Problem] {question)
[Solution] !decomposed biceeeddd_thii_iseeeee think_steps/
Table 16: The prompt used for LLM-as-a-judge.
30

---

Published as a conference paper at ICLR 2026
Prompt for Decomposing Solution
You need to read the problem-solving process below, which divided into some steps using line breaks, with each step labeled by tag "ax".
[Pre Split Format Thought] {format solution)
Then, you are required to combine consecutive steps into several logically independent sections. Note that each independent section must belong one of the eight action categories listed below:
(1) Problem Analysis:
Decompose the original problem, clarify known conditions and
objectives
(2) Subtask Definition: Establish phased goals (e.g.. equation construction, variable
definition)
(3) Calculation Solving: Perform numerical operations symbolic derivations
(4) Reflection Check: Verify logical/computational errors
(5) Answer Validation: Confirm correctness of final/intermediate conclusions
(6) Error Correction: Rectify identified issues
Strategy Shift: Switch alternative problem-solving approach
(8) Final Conclusion: Summarize and present ultimate answer
**utput Specifications
Maintain original thought chain order when outputting sections
Section format: estart x- <end Concise explanation the reasons for combining these steps together(max 15 words)
Must satisfy: start end < nert_start
* **Special Constraints**
If several consecutive steps involving calculations, enumerations, proofs, or similar processes working toward the same objective-such solving the same equation, proposition, or problem-please group them together into a single section without splitting them apart!
If several consecutive steps are reflections on or verifications for the same issue, combine them into single section and do not separate them!
** Exxmple output**
<0 x2 > >'These three steps aim to analyze the question
<5 These three steps is computing the same equation
<6 <6 > This step aims draw the final answer
Table 17: The prompt used to decompose preprocessed solutions. We define eight atomic logical actions and use them to guide the step-wise decomposition.
31

---

Published as a conference paper at ICLR 2026
Prompt for Decomposing Answer
You will be presented with the following content:
[Math Problem] {problem}
[Answer]
(answer of the problem}
*Taask Requirements**
Insert "<step>" and "</step>" into the original answer to break down the above answer into several atomic reasoning steps. Each atomic reasoning step belongs only one of the following actions:
1. **Problem *nalysis** Analyze the original problem, organize the conditions, and deepen understanding of the problem.
2. **Set Subtask Goals*#; Clearly define the specific goal to be solved next, such setting up an equation.
3. ** *Calculating Solving** Calculate variable solve equation.
""Seff-Reflection*** Reflect on potential mistakes and assess whether previous thoughts contain mistakes.
Veriff Correctness* Validate whether the final answer or intermediate conclusions are correct.
6. **Eroor Correction**; Rectify identified mistakes.
** **Switch Thinking**; Change different thought for solving the problem.
8. Arrive at the Final Answer**; Conclude with the final answer.
#*Uupput #*rmat*#
The breakdown steps should not be too coarse; each step should belong to only one type
of action.
2. Separate the reasoning steps by inserting "<step>" and "</step>".
3. Besides inserting the "<step>" and "</step>" tags, do not change the original solution.
4. Do not alter the original solution except for inserting the "<step>" and "</step>" tags;
keep the original solution complete.
5. You only need to reply with the complete solution, without outputing any additional content.
Table 18:
The prompt used to decompose answer. We used the same set of eight atomic logical actions as detailed in Table 17, with only minor modifications to the phrasing.
32

---

Published as a conference paper at ICLR 2026
Prompt for Summarizing Solution
You will be presented with the following two contents:
[Math Problem] (problem)
[Solution]
(solution in list format, each element is step)
This thought process includes len(solution) Steps in total. Your task is to summarize the main content of these all Steps in the solution.
Make sure each summary clearly describes the key points and the intermediate conclusion of the respective step in the thought process.
Finally, your output should be in standard JSON format: json
('Toought Step 0': Summary Step 0',
Thought Step 1': Summary of Step 1', .
Table 19: The prompt used to summarize each step of a solution. Note that the input solution has already been decomposed into steps in list format.
33

---

Published as a conference paper at ICLR 2026
Prompt for Matching Solution and Answer
You will be given the following three sections:
[Math Problem]
{problem)
[Summarized Solution]
(summarized solution a list format, each element a summarized step)
[Answer]
(answer of the problem in a list format, each element a step answer]
Tassk Requirements
1. Establish a mapping relationship between all these solution steps and the answer steps.
2. For each answer step, find all solution steps whose summaries match the content of the current solution step summary.
3.
Allow one-to-many or many-to-one correspondence.
Maatching Criteria**
The Answer Step should summarize the corresponding Solution Step, or be refined articulation of it. If the LAST step the Solution includes the "final answer" from the Answer, it should at least match the LAST step of the Answer.
**utput Requirements
Analysis Report:
Explain the matching basis for each solution step in order.
Only record clear content correspondences.
Mapping Table (JSON format):
[Thought_Step__umbee) ...)
Unmatched thought process steps are not included.
Example:
jsson
("O":
[1, 31, "1": [5]}
Table 20: The prompt used match each answer step with its corresponding solution step(s). Note that both the solution and the answer have already been decomposed into steps in list format.
34

---

Published as a conference paper at ICLR 2026
Prompt for Locating Reflection
** #IIstruction:
You will receive the following two pieces of content: [Math Problem] {problem)
(Decomposed Solution] (decomposed solution list format, each element step)
Note that this thought process has been broken down into multiple steps, documenting the complete path from the initial attempt to the final correct solution. It may include attempts with incorrect reasoning, reflections on those attempts, and subsequent corrections.
Your Te
Carefully read and analyze the provided thought process identify all segments where 'an error in previous reasoning steps was discovered through reflection or verification." For each identified segment, complete the following tasks:
1. Record the Step Index: Indicate the step number where this segment occurs.
Identify the Fragment: Clearly specify the exact content of these segments.
3. Summarize the Cause: Analyze and summarize the reason why the error occurred.
**Ottput Format Requirements:
Return your results in the following JSON format: json
("Reflection 1": ("Step Index": "The step number where this segment occurs", "Identified Fragment": "The exact content of the segment where an error in previous reasoning steps was discovered through reflection or veri perification", "Reason": "A summary of the reason why the occurred"),
#*Notes:
The steps you find MUST have identified a previous ERROR through reflection. If this step ONLY involves double-checking previous calculation to verify its correctness, without finding a specific error, then ignore this step.
2. If there are qualifying segments in the thought process, return an empty JSON object: "()".
Table 21:
The prompt used to find reflection segments within a correct solution. Note that the input solution has already been decomposed into steps in list format.
35

---

Published as a conference paper at ICLR 2026
Prompt for Locating Mistake Source
You will see math problem and pre-decomposed thought process for solving this math problem:
[Math Problem]
{problem)
(Decomposed Solution]
(decomposed solution list format, each element is step)
This solution records the complete mental journey from initial attempts to the final correct answer, including trials of incorrect approaches, reflections, and corrections.
The following paragraph explicitly highlights the discovery of mistake through reflection: [Identified Fragment] (reflection identified fragment]
And the reason of the mistake summarized following: [Reason]
(reflection reason]
Your task is to identify the earliest step in the thought process that is responsible for this mistake.
TTaask Requirements**
-The step you find MUST itself actually contain logical or calculation errors. You need to verify the step you have identified CAREFULLY.
Before stating the earliest step you believe is responsible for this mistake, you need to carefully think and provide your detailed analysis.
Finally, based your analysis, output the earliest thought step and step number that you believe responsible for the mistake, and provide your reason.
#*Otpput Format Requirements:**
Return the final results in the following JSON format:
json
"The earliest step content that is responsible for this mistake", "Step Index"; "The step number of this thought step", "Reason": "The reason you believe this step is responsible for this mistake")
("Earliest Step":
Table 22: The prompt used to find the error source based on reflection segment. This prompt utilizes information returned from the prompt in Table 21.
36

---

Published as a conference paper at ICLR 2026
Prompt for Reasoning All Mistakes
You will be presented with the following content:
A math problem:
[Math Problem]
(problem]
A decomposed thought process for solving the problem:
[Decomposed solution]
[decomposed solution in list format, each element is step)
This thought process does not directly lead to the correct answer but includes attempts with incorrect reasoning, reflections, and corrections before arriving at the right solution.
In Step <reflection start step index> of the thought process, critical mistake occurs: [Farliest Step]
[reflection earliest step)
This mistake is identified through reflection in Step <reflection step index>:
[ldentified Fragment]
(reflection identified fragment]
**Task Description*
Your task is identify the types of thinking steps within the range from step <reflection start step index > to step<reflection step index> that meet the following definitions. You need check each definition in sequence. Once definition is met, immediately label it with the corresponding step type and stop checking further.
Reflection Steps: A step that involves reflecting on or verifying the error and its resulting subsequent mistakes.
Incorrect Step: A step is considered incorrect if: It contains mistakes itself or it continues the analysis based on the error from above steps without introducing a new, correct approach or making proper correction.
*Speciffic Requirements
For each thinking step, carefully analyze and explain your reason before determining its type.
The final output format should strictly adhere to standard JSON, for example: json
("Step k"; ("Analysis"; "The specific analysis for Step k before determining its type", "Step Type": ("Incorrect" "Reflection" / "Correct")).....
** **
Ensure that your analysis of each step is well-reasoned and avoids subjective assumptions.
Table 23: The prompt used determine the category of each step within the range. This prompt utilizes information returned from the prompts in Table 21 and Table 22.
37


Reviews:
[{"review_id": "7JKp8OyMcT", "reviewer": "Reviewer_ebqn", "summary": "This paper begins with an in-depth analysis of the challenges inherent in Generative Process Reward Model (GenPRM) based process evaluation. To mitigate these challenges, the authors propose a novel generative process evaluation mechanism. This mechanism features an intrinsic-signal-driven evaluation (judging reasoning steps based on semantic information) and thought-level, ability-adaptive reward schemes. The mechanism is integrated with the GRPO algorithm to form a new RL algorithm, termed TP-GRPO. Experimental results on mainstream reasoning benchmarks demonstrate that TP-GRPO achieves superior training efficiency and accuracy compared to baselines.", "questions": "Overall, this paper is well-written, well-motivated, and the proposed method is technically sound. The empirical results are promising, demonstrating clear improvements over strong baselines, even if they do not surpass all current SOTA methods. In order to maintain my rating, I would like the authors to address the points in the weaknesses above.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and presents the topic clearly.\n\n2. The proposed method is well-motivated and appears technically sound.\n\n3. The empirical evaluation is promising. TP-GRPO is shown to outperform the GRPO baseline and most existing outcome reward-based RLVR methods in terms of both training efficiency and accuracy.", "weaknesses": "1. Notable performance gap remains when comparing TP-GRPO to state-of-the-art models with much larger training budgets (e.g., DeepScaler-1.5B-Preview and Skywork-OR1-7B). The paper does present initial scaling trend (i.e., Fig 3), but the experiment appears to be conducted with relatively low training budgets. This limited scope makes it difficult to conclusively determine whether the advantages of TP-GRPO will persist, widen, or saturate as model scale and training budgets increase significantly.\n\n2. The introduction clearly articulates several \"design pitfalls\" of existing GenPRM-based process evaluation. While the paper does provide some analyses related to these points, the insights are somewhat scattered throughout the experimental section rather than being presented cohesively. The paper would be significantly strengthened if this analysis were consolidated and made more explicit, clearly demonstrating how the proposed method directly mitigates each of the identified pitfalls.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DSfRCkw4Mt", "reviewer": "Reviewer_iXx8", "summary": "This paper revisits GenPRM for RL of large reasoning models by using intrinsic semantic cues in the solution trajectory to judge step correctness (locating reflections, tracing the earliest error, and interval labeling), merging consecutive steps with the same correctness into “thoughts,” and applying thought-level, ability-adaptive rewards; on incorrect solutions, only thoughts semantically aligned with the wrong answer are penalized. Combined with GRPO as TP-GRPO, the method attains higher or comparable accuracy with far fewer training solutions on DeepSeek-R1-Distill-Qwen 1.5B/7B (e.g., 5.6K vs 34K; 8.56K vs 16K) and introduces the Effic. metric to quantify improved sample efficiency.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The research question and proposed method are interesting, e.g., alleviating GenPRM’s reasoning burden and the observation on dense step-wise rewards.", "weaknesses": "The method's reliance on the LRM's own capacity for self-reflection and annotation is a fundamental limitation. This approach is inherently self-limiting, as any improvement is capped by the model's existing capabilities and the task's difficulty, hindering the acquisition of novel skills. The poor performance in Table 1 exemplifies this concern.\n\nThe experimental setup is questionable. The RL training is remarkably inefficient, yielding only a 2-4% accuracy gain over 400-1000 iterations—a potential artifact of an undersized group size for GRPO that undermines the paper's credibility. Furthermore, the efficiency comparisons in Tables 1 and 2 are inequitable due to mismatched hyperparameters (e.g., group_size, batch_size) across baselines, and a performance comparison at convergence is conspicuously absent.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V0wZbQPvdh", "reviewer": "Reviewer_N8ji", "summary": "This paper proposes a novel reinforcement learning framework for large reasoning models (LRMs) that enhances reasoning efficiency via process reward modeling (PRM). The authors identify key limitations in existing generative process reward models (GenPRMs), notably their dependence on reasoning ability, over-dense reward signals, and susceptibility to reward hacking. To overcome these issues, the paper introduces an Intrinsic-Signal-Driven Evaluation method and a Thought-Level Ability-Adaptive Reward Mechanism, which together form the proposed TP-GRPO algorithm. Specifically, thought-level\n\nExperimental results on the DeepSeek-R1-Distill-Qwen (1.5B and 7B) models demonstrate significant improvements in training efficiency while maintaining comparable performance. Also, the proposed techniques are less sensitive to the GenPRM's reasoning ability, successfully mitigating the reasoning dependency issue.", "questions": "* In Figure 3, the accuracy fluctuates significantly and is not monotonically increasing. I am wondering if this is still the case under different random seeds and whether the conclusion still holds.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Novelty: The techniques of (1) reflection localization for determining the granularity of thoughts and (2) thought-level advantage adjustment are new to me. The paper also introduces several useful methods in Sections 3.1.1 and 3.1.2.\n\n2. Quality: The paper accurately pinpoints three major weaknesses in current GenPRM designs: excessive dependency on reasoning strength, over-dense reward granularity, and reward hacking. The analysis is precise and well-supported by empirical and conceptual insights.", "weaknesses": "1. Clarity: It is better to highlight the contribution of the paper with a more concise title, such as \"Improving Reasoning Efficiency via Thought-Level Reflective Reward Shaping.\" The terms \"intrinsic-signal\" and \"ability-adaptive\" are somewhat misleading, as the former can be easily confused with the conventional term \"intrinsic reward\" [1], and the latter could be replaced with more commonly used terms such as \"difficulty-aware,\" given that \"ability\" sometimes refers to more general categories of skills like coding, math, tool use, etc.\n\n2. Significance, Quality: In Tables 2 and 3, TR-GRPO is compared only with methods based on outcome reward models (ORMs), and additional comparisons with other PRM baselines are needed, such as [2][3][4]. It is also recommended to include comparisons with stronger ORM methods, such as those in [5].\n\n3. Novelty: The thought-level idea is similar to the one in [4]. It is worth discussing the differences between the two works and making an empirical comparison if applicable.\n\n4. Quality: The effectiveness of TR-GRPO would be more evident if the comparisons in Tables 2 and 3 were made fair by using the same budget or by measuring the number of samples needed to achieve the same overall accuracy.\n\nI think the paper provides many useful techniques, but also has obvious issues, as mentioned above. I would be happy to consider raising my score if the aforementioned concerns are well addressed.\n\n### References\n\n[1]: Pathak, Deepak, et al. \"Curiosity-driven exploration by self-supervised prediction.\" International conference on machine learning. PMLR, 2017.\n\n[2]: Zhao, Jian, et al. \"Genprm: Scaling test-time compute of process reward models via generative reasoning.\" arXiv preprint arXiv:2504.00891 (2025).\n\n[3]: Zhang, Hanning, et al. \"Entropy-regularized process reward model.\" TMLR, 2024.\n\n[4]: Xiong, Wei, et al. \"Stepwiser: Stepwise generative judges for wiser reasoning.\" arXiv preprint arXiv:2508.19229 (2025).\n\n[5]: Chen, Minghan, et al. \"Seed-grpo: Semantic entropy enhanced grpo for uncertainty-aware policy optimization.\" arXiv preprint arXiv:2505.12346 (2025).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Cxy1FkIVja", "reviewer": "Reviewer_LpBz", "summary": "This paper tries to address key challenges in incorporating generative process reward models (GenPRMs) into RLVR. The identified challenges are: (1) the requirement for a GenPRM with strong reasoning ability; (2) the risk of reward hacking from over-densified rewards; and (3) the potential discouragement of policy exploration. The authors tackle challenge (1) by using semantic matching to locate correct and incorrect steps. They tackle challenge (2) by merging continuous steps with the same correctness into a larger block. They tackle challenge (3) by using an adaptive process reward assignment. By incorporating the process rewards provided by their framework into GRPO, they proposed TP-GRPO. The main results are evaluated on mathematical reasoning benchmarks.", "questions": "- At line 248, the paper states that \"$\\text{acc}_G$ is the accuracy of the G sampled solutions for the same problem.\" The definition of $\\text{acc}_G$ is confusing. Is it the average accuracy across G sampled solutions for the same problem or something else?\n\n- Also at line 248, the paper states that \"when $\\text{acc}_G=0$, $r^c$ vanishes, reducing to outcome supervision\" I am a little bit confused about why  $r^c=0$ would cause it to reduce to outcome supervision.\n\n- Could $\\text{acc}_G$ also be applied to the computation of $r^{ic}_i$ to achieve adaptive penalization according to the difficulty of the problem?\n\n- I am confused about the definition of the advantage function in this paper. According to Equation 3 in the paper, it is more like the definition of returns (or cumulative rewards). According to the definition in [1], the advantage function measures whether an action is better or worse than the policy's default (or expected) behavior. I hope the authors will clear this up for me.\n\n[1] High-Dimensional Continuous Control Using Generalized Advantage Estimation, ICLR 2016.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "- The paper is clearly written and easy to follow.\n\n- By identifying three crucial challenges for GenPRMs, this paper provides meaningful guidance for future research directions in the field.\n\n- The idea of using semantic alignment for process evaluation is intuitively sound, and it could potentially reduce the difficulty of this task for GenPRM.", "weaknesses": "**Lack of Evidential Support**\n\nThe paper makes several assertions that lack sufficient supporting evidence. For instance:\n\n- At line 236, the paper states that \"static rewards can unintentionally suppress exploration.\" At line 248, the paper states that \"when $\\text{acc}_G=0$, $r^c$ vanishes, reducing to outcome supervision GRPO and prioritizing exploration.\" Why static rewards would suppress exploration? And why, when reduced to outcome supervision, is exploration prioritized? Empirical or theoretical evidence is needed to support these justifications.\n\n- At line 220, the paper states that \"If process rewards are assigned at the step level ... which could in turn mislead the optimization to incorrect directions.\" This conclusion requires further justification. For example, if the process reward is assigned correctly at the step level, would it still mislead the optimization direction?\n\n**Strong Assumptions and Limited Application Scenarios**\n\n- At line 203, the \"Identify Effective Steps\" component depends heavily on the LRM capacity for accurate self-reflection, a capability not universal among all models. How can the proposed method be applied to those LRMs without self-reflection behavior?\n\n- At line 205, the paper states \"During reflection, the LRM typically analyzes the causes of the reflected mistake\". This assumes that LRM can correctly diagnose the cause of a mistake during reflection. The case of incorrect self-diagnosis is not addressed.\n\n- Even if the LRM has self-reflection and can correctly identify the location of a previous error, the requirement for the GenPRM to identify \"steps dependent on... incorrect previous steps\" (line 207) might still raise concerns about its need for non-trivial reasoning capabilities.\n\n- At line 211, the hypothesis that \"all steps in the answer are erroneous\" does not generally hold. As established by previous work [1], incorrect trajectories often contain valid reasoning steps before the first error occurs. \n\n[1] Let's Verify Step by Step, arXiv 2023.\n\n**Incomplete Experimental Setting**\n\nTo robustly support the claim that the evaluation scheme has \"low reasoning requirements on the PRM,\" additional settings should be added: using models like Qwen3-32B, Qwen3-4B, and Gemma-3-12B-it as the GenPRMs without applying the proposed \"reducing reasoning requirements\" methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)