ReasonFlux-PRM: Trajectory-Aware PRMs for Long Chain-of-Thought Reasoning in LLMs
Authors: Jiaru Zou
Ling Yang
Jingwen Gu
Jiahao Qiu
Ke Shen
Jingrui He
Mengdi Wang
Venue: NeurIPS 2025 poster ()
Keywords: Process Reward Models
Inference Scaling
LLMs
Reasoning Trajectories Selection
Primary area: deep_learning

Abstract: Process Reward Models (PRMs) have recently emerged as a powerful framework for supervising intermediate reasoning steps in large language models (LLMs). Previous PRMs are primarily trained on model final output responses and struggle to evaluate intermediate thinking trajectories robustly, especially in the emerging setting of trajectory–response outputs generated by frontier reasoning models like Deepseek-R1.  In this work, we introduce ReasonFlux-PRM, a novel trajectory-aware PRM explicitly designed to evaluate the trajectory-response type of reasoning traces. ReasonFlux-PRM incorporates both step-level and trajectory-level supervision, enabling fine-grained reward assignment aligned with structured chain-of-thought data. We adapt ReasonFlux-PRM to support reward supervision under both offline and online settings, including (i) selecting high-quality model distillation data for downstream supervised fine-tuning of smaller models, (ii) providing dense process-level rewards for policy optimization during reinforcement learning, and (iii) enabling reward-guided Best-of-N test-time scaling.  Empirical results on challenging downstream benchmarks such as AIME, MATH500, and GPQA-Diamond demonstrate that ReasonFlux-PRM-7B selects higher quality data than strong PRMs (e.g., Qwen2.5-Math-PRM-72B) and human-curated baselines. Furthermore, ReasonFlux-PRM-7B yields consistent performance improvements, achieving average gains of 12.1\% in supervised fine-tuning, 4.5\% in reinforcement learning, and 6.3\% in test-time scaling. We also release an efficient ReasonFlux-PRM-1.5B for resource-constrained applications and edge deployment. Our code and models are released at https://github.com/Gen-Verse/ReasonFlux.

ReasonFlux-PRM:: Trajectory-Aware PRMs for Long Chain-of-Thouge Reasoning in LLMs
Jiaru Zou'r, Ling Yang2.4+,, Jingwen Gu Jiahao Qiu²,
Ke Shen", Jingrui He', Mengdi Wang
UICC Princeton University CComell University *ByteDance Seed
n Code: ReasonFlux-PRM-CCO
Models: MassonFlux-PRM-1.5511
Abstract
Process Reward Models (PRMs) have recently emerged as a powerful framework for supervising intermediate reasoning steps in large language models (LLMs). Previous PRMs are primarily trained on model final output responses and struggle to evaluate intermediate thinking trajectories robustly, especially the emerg- ing setting of trajectory-response outputs generated by frontier reasoning mod- els like Deepseek- R1. this work, we introduce ReasonFFux-PRMM, a novel trajectory-awarr PRM explicitly designed to evaluate the rrajectory-responss type reasoning traces. ReasonFlux-PRM. incorporates both step-level and trajectory- level supervision, enabling fine-grained reward assignment aligned with structured chain-of- of-thhught data. We adapt ReasonFlux- PRM to support reward supervision under both offline and online settings, including (i) selecting high-quality model distillation data for downstream supervised fine-tuning of smaller models, (ii) pro- viding dense process-level rewards for policy optimization during reinforcement learning, and (iii) enabling reward-guided Best-of-N test-time scaling. Empirical results on challenging downstream benchmarks such as AIME, MATH500, and GPQA-Diamond demonstrate that
selects higher quality data and human-curated baselines. Furthermore, ReasonFlux -RM--BM yields consistent performance improvements, achieving average gains of 12. 1% in supervised fine-tuning, 4.5% in reinforcement learning, and 6.3% in test-time scaling. We also release an efficient ReasonFlux- PRM-1.5B for resource-constrained applications and edge deployment.
than strong PRMs (c.g., -aa.n..5--a--PMMM----
AM239
GhoA-Diammn
GPFO
Oone Heporyy
Owerz (-80))--900--7
Mads RM
Stap
Mambbar Soltion (1)
ReasonFlux-PPen Figure I: Overview of ReasonFFlux-PRMt P PPM is designed to provide general-purpose reward supervision across multiple application scenarios. Left: Offline selection of high-quality distilled engjetoryy-esppose data to enhance downstream supervised fine-tuning of smaller models.
Middle: Online reward modeling integrated into GRPO-based policy optimization. Right: Reward- guided Best-of-N test-time scaling to improve inference-ting performance.
* Equal contribution
Corresponding authors
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Trajectory-Responee Data
Question:
Is the Fourier transform of the signal imaginary?
2tt
2nt
stn(snat)
xx00
dip
otherwise
Thinking Trajectories:
see definition the signal definition The Pourier transform properties suggests that Break the exponential
user askdnng whether the Fourier trunsform given signal imagleary
into sine termss, we have Recall from the previous dective integral expression... Let me revisit the given term for (f).. Wait the function add... Recall the definition imaginary and expression the Fourter transform Imaginary.
PRM
Final Respomes
step- by step.
Stop 1: Fourter transform of signal is defined
Step W\boxed(0.71 Step Wboxedd0 85] Stop 3: Wboxedjo. 92)
Step 2: We use the idmniity as
Sup 3: Subtiituting into the integral...
PRM
Based on the following steps, the result is purely imaginary.
Figure 2: Illustration of the Trajectory- Trajecooy-Respoonee Data generated by Deepseck-RI. Existing PRMs can assign appropriate scores to final responses but often struggle to evaluate intermediate reasoning trajectories accurately.
Introduction
Process Reward Models [49, 79, 24] have recently emerged as powerful framework for providing process-level supervision large language models (LLMs) reasoning process, particularly for complex domains such as mathematical problem solving [35, 27, 32, 49, 44]. Given question and the corresponding model's final response, PRMs verify the reasoning step-by-sted and assign fine- grained rewards to each step of the response. Prior studies have leveraged PRMs in both post-training stages [56, 8). including providing dense rewards for online reinforcement learning (RL) [5], and reward-guided inference-time scaling [18, 80).
However, existing PRMs are primarily trained and applied -ener-ggeeeeted final responses, typi- cally presented in an explicit and organized stcy-by-step chain-of-thouge (CoT) format. Concurrently, with recent advancements in frontier reasoning models such as OpenAl-ol [16] and Deepseek-R1 [6], these models have increasingly adopted trajectory-responss format of output: a lengthy, com- prehensive, and less organized intermediate thinking trajectory, followed by a concise, step-by-step final response conditioned on the prior thinking (as illustrated in Figure 2). Such trajectory- response pairs have been widely distilled and acquired from large reasoning models to support downstream training of smaller models, enabling them to emulate the reasoning capabilities of larger models to first think then produce coherent, extended CoT rationales [40, 72, 68). The increasing utilization of vajectoryy-espone data raises an important question: Can PRMs provide supervision not only to the final responses of large reasoning models, but also to their intermediate thinking trajectories?
Addressing this question first presents a challenge of how assign informative and correct rewards to the model intermediate thinking trajectories. Unlike final responses, these trajectories are typically treated as silver-standard data [73], automatically generated by large reasoning models without rigorous quality control or standardized verification criteria, making their evaluation inherently noisy and less reliable. To address this, we first revisit several state-of-the-art PRMs and evaluate wajectory--espooe
their performance to robustly supervise model thinking trajectories and can degrade downstream training on such data. We further find that this degradation stems primarily from two key issues: an structural and formatting mismatch between intermediate thinking trajectories and final responses, and the lack of trajensory-resposee data with assigned rewards during PRMs training.
pairs. Our analysis reveals that existing PRMs struggle
Motivated by these observations, we propose a new trajectory- PRM, namely ReasonFlux- PRM. which incorporates both step-level and trajectory-level supervision to better align the models' middle thinking trajectories with their final responses. ReasonFlux-PRR dataset of high-quality trajectory- pairs covering math and science reasoning. Unlike existing PRMs, ReasonFlux-PRee explicitly tailored to intermediate thinking processes by providing fine- grained rewards supervision signals for each step within the thinking trajectory. We further
trained on a 10k curated

---

adapt ReasonFlux-PRM, for more general reward modeling scenarios, as illustrated in Figure 1. In offline settings, ReasonFlux-PPe PPMM assigns scores filter high-quality trajectory-restt response pairs, facilitating effective training data curation for downstream supervised fine-tuning of smaller models. In online settings, ReasonFlux- grained supervision signals during policy optimization, such as GRPO [51]. Moreover, ReasonFlux PRM facilitates test-time scaling by evaluating multiple generated responses and selecting the most promising one via reward-guider Best-of-N strategy.
integrated into reward modeling process to provide fine-
In summary, our main contributions are:
In-Depth Trajectory-Respon Response Data Analysis in Long-Co'I Reasoning. We identify, formulate, and analyze the problem of adapting several existing PRMs to supervise both models' interme- diate reasoning trajectories and their final responses, motivated by the increasing prevalence of eajjetooy-responns distillation data in downstream post-training and test-time scaling.
Trajectory-aware Reward Modeling for Data Selection, RL and Test-Time Scaling. We introduce ReasonFlux-PRM, a trajectory-aware process reward model that incorporates both step-level and trajectory-levet supervision, enabling fine-grained reward assignment for model thinking trajectories. ReasonFlux-PPe can be integrated into both offline and online workflows for more generalized purposes, including offline selection of high-quality training data, online policy optimization in RL training, and test-time scaling.
Extensive Downstream Evaluations. Across extensive evaluations on challenging reasoning benchmarks, ReasonFFlxx-PRe PRM demonstrates superior data selection quality at smaller model scales, with RcaasonFlux-PRMM-8 outperforming strong baselines such as 1ren2.5-Mathh M 72B [79] and datasets curated by human experts. On tasks such AIME [38, 37J. MATH500 [13], and GPQA-Diamond [46], ReasonFlux-PRMMPT achieves notable average accuracy improvement of 12.1% during supervised fine-tuning, 4.5% during reinforcement learning, and 6.3% during inference test-time scaling.
Preliminaries
Trajectory-Respont Data. Let f-)acte-) denote an oracle model, such as Deepseck-R1, capable of producing structured reasoning traces. Given complex input prompt x, the oracle generates sequence of intermediate thinking steps followed by a final response. We represent each instance of such data tuple (s, a), where intermediate steps, and a = (a1, .2,..., ,T) denotes the final response, which can also be structured as thain-of-tthouhh trace with 7 formatted and organized steps. For large reasoning models, we assume that both 8 and a consist of I'' reasoning steps. This structural alignment reflects the modeling assumption that the final output trace generated in a step-by-step manner, strictly conditioned on the preceding intermediate reasoning steps s. Both the thinking trajectory and final response are generated regr-regressyvely by the oracle model, i.e.,
(81, 82, ST) denotes thinking trajectory consisting of T
8y ~ Soracte (v. 821); 0g ~ Soracte 8, a<t);
where 8c. = (81,,..,, 8)-1) and act (a1,..., at-1) denote the reasoning and answer histories up step respectively. In the vajectory-responee outputs distillation setting, the full supervision target instance can be constructed as the concatenation of thinking trajectories and the final response, i.e.,
y + a.
Process Reward Modeling, Given a trajectory-answer pair (s, a). where both s = .. ST) and (aI...., ) structured as reasoning traces, the goal of a process reward model is to evaluate each intermediate reasoning step s, E s with respect to its utility in achieving a correct and coherent final response. We first define reference reward function R,t that provides step-level supervision:
T Rye((st 2, S<t,a).
(2)
where (ee(()) scores the 1-th step conditioned on the input ar the prior thinking trajectory steps, and the full final response a. The total reward for the trajectory is then computed by aggregating the step-by-step scores:
R A(TI, T2, .2. rT),
where A(-) denotes an aggregation function such as MEAN and SUM. The training objective for PRMs is to learn a scoring function Rg(.), parameterized by o, that approximates the reference
3

---

Deesseck.ft
Deepseek-81 Flush Thinking
Thinking
Dendle
Denaity
Reward Score
over 1,000 snnecttry-responne pairs distilled from Deepseek-RI R1 and the Gemini Flash Thinking API. Left: Distribution of scores computed over thinking trajectories. Right: Distribution scores based on final responses.
Figure 3: Score distributions rewarded by
reward for each step. This formulated as minimizing the discrepancy between predicted and reference rewards over training dataset D ((x ( ri:T) } where ri denotes the target reward for step s Formally, the training objective can be written as:
(t)
N T
min MΣΣN (Ro(s)) 1=1
|0 o2 r(
Existing PRMs Are Not Prepared for Rewarding Thinking Trajectories
To examine whether existing frontier PRMs can directly applied to reward the trajectory-response data, we first conduct preliminary study to investigate two key questions:
RQ1: Can PRMs distinguish the quality of thinking trajectories distilled from different oracle models?
RQ2: What is the effectiveness of using the PRM-selected rajectory-response data on the downstream fine-tuning of smaller models?
For brevity, defer detailed experimental setups Appendix A.1. To investigate RQ1, evaluate the Qwen2.5-Math-PPM.
PRM model on 1 1,000 sampled problems in s s1k [40] with trajectory- response traces generated by Google Flash Thinking API [7] and Deepseek-RI [6], respectively. For each data trace, we apply the PRM model to compute the step-level rewards (spitted by "\n\n"), and then aggregate these rewards by taking the (left) compares the distribution of PRM scores across the two oracle models. The histogram shows significant overlap the score distributions, though Deepseek-F RI traces tend receive higher rewards on average, with longer tail toward high-reward regions (e.g., scores above 0.6), The results suggest that while Qwen2.5 -aae2.5---th------ two sources, its discriminative ability remains limited.
to obtain final trajectory-level reward. Figure
captures some signal for differentiating between the
Takeaway 1
Several existing PRMs exhibit limitations in distinguishing reasoning traces distilled from different oracle models and often struggle clearly separate high- and low-quality model thinking trajectories.
Next, to investigate RQ2, we evaluate the performance using the PRM-selected data on the down- stream supervised fine-tuning of smaller models. We apply four different PRMs assign a reward score to each of the 59K raw trajectory-respont the same mean aggregation over step-level rewards to compute a trajectory-Ieve score. Based on these scores, we rank all traces and select the top 1,000 samples from each PRM as fine-tuning dataset for downstream small model. For better comparison, we also adopt the direct set of 1K human-curated examples in s1k [40]. Table I presents the accuracy of the fine-tuned Qwen2.5- Ins-rutt on four challenging downstream tasks. We observe that all PRM-selected training sets underperform significantly compared to the human-curated bascline, suggesting that existing PRMs
traces generated by Gemini [7] in s1 [40], using

---

Table 1: Performance of Swen2.5-14B-Instrre on four challenging reasoning tasks after fine-
tuning on the trajectory-respone data selected by four different PRMs. We also compare the SFT performance with using randomly sampled data (Ik from 59k) and the slk human-curated data [40].
SFT Data Source
AIME24
AIME25
MATH500
GPQA-Diamond
Random
16.7 (4 16.6)
20.0 ( 13.3)
68.4 (L 10.4)
34.8 (4 6.6)
Mauh--Shepherd-RRMM-78
13.3 (4 20.0)
6.7 26.6)
67.8 (L 11.0)
33.3 8.1)
Skywork-PRM-7B
13.3 20.0)
13.3 (1 20.0)
71.8 (1 7.0)
37.9 3.5)
Owen2.5-Math-PRM..
26.7 6.6)
20.0 13.3)
73.2 (.)
39.4 2.0)
Qwen2.5-Math-PRRM-Z5
33.3 0.0)
26.7 6.6)
77.0 (.8)
39.4 (20.)
on model responses
36.7 (14)
26.7 6.6)
77.8 (L 1.0)
40.9 (.055)
Human-curated (s1k)
33.3
33.3
78.8
41.4
not yet sufficiently calibrated to identify high-quality trajectory-response data, and can even degrade downstream model performance by selecting suboptimal or misaligned training samples.
Takeaway 2
Direct reliance on current PRMs for trajectory-resee response selection can yield misaligned training data, which in turn diminishes the effectiveness of downstream supervised fine-tuning for smaller models.
As most existing PRMs are trained on reasoning traces derived from model final output responses rather than intermediate thinking trajectories [79, 10], we take closer look the distinctions between genuine thinking trajectories and post-hoc generated responses. As we detailed in the Appendix A.2, these types of data exhibit several fundamental differences: (i) Thinking trajectories often include branching, where the model revisits earlier steps, explores alternative paths, and revises prior assumptions--betee behavior rarely observed in the linear and polished structure of final responses. (ii) Thinking trajectories tend to exhibit weaker global coherence across steps, as each step is often locally focused and not optimized for narrative continuity.
To further validate that the performance degradation of existing PRMs stems from the aforementioned data mismatch, conduct additional experiment in which Sreen..5-MtthPRRMM..
is applied to score each data instance based solely on the model response, rather than the middle thinking trajectories. As shown in Figure (right), the PRM produces relatively clearer separation score distributions between the two oracle models. Also as shown in Table (row: on model responses), the performance drop reduced when training on PRM-selected data based final responses, suggesting that existing PRMs are better aligned with redeelreepoossellvee supervision.
Takeaway 3
Thinking trajectories instinctively differ from final responses, and existing PRMs are more accustomed scoring final outputs than intermediate reasoning steps.
Motivation on ReasonFlux-PRM. Our findings above highlight the need for more general reward model that can effectively evaluate both intermediate model thinking trajectories and final responses. As thinking trajectories become integral supervised and RL-based fine-tuning, existing PRMs, trained primarily on final responses, struggle to provide reliable supervision. To address this, we propose a new thinking- aware PRM tailored the trajectory- respooss data supervision.
4 ReasonFlux-PRM
In this section, we introduce ReasonFlux- PRMM, a trajectory-aware process reward model, as illustrated in Figure 4. We first present new reward design tailored for thinking trajectories in Section 4. which incorporates both step-level and trajectory-Ievel signals to reflect line-grained and holistic reasoning quality. We then elaborate how ReasonFFux-PRRn is applied in more general reward supervision setting Section 4.2, covering both offline data selection and online reward modeling.
5

---

ReasonFlux-PRM Training
ReasonFluz-PRM Inference
Training Data Curation
Offline Setting
Distiled Teaestory
Downebmam
Thinking Trpectories
Final Responic
Rusporss Pairs
Trairing
Qqueetion
Highouatty
TrajectoryRResonee Data
Reward Design
Online Setting
Quality
Reward
Eper LAM
1. RL Training
Coherance
Thinking Trajectertes
Varty
Guided Templater
Recoardllu PRM
Jonno
Reward
Test-TTme-Seallig
Alignment
Gensrate
θ Policy Model
Instruct
Scancor9
Reward
Responses
Step-level reward
Trajectory-evvel rewerd
Ressandiaux-Pe PRM
Figure 4: Illustration of the overall method design. ReasonFlux PPMMo is trained trajeotory-repponse data pairs with a novel reward design that integrates both step-level and trajectory-leve signals. As a general-purpose PRM, ReasonFlux-PRM, supports both offline data selection for small models SFT and online reward modeling including policy optimization in RL training and test-time scaling.
4.1 How Should We Define Process Rewards and Why?
We first propose new reward design train ReasonFFux-PRMM from the trajectory-responee data. Our formulation integrates both step-level and trajectory-Ievel rewards to better address the discrepancy between intermediate thinking trajectories and final responses, and to align ReasonFlux- PRM with the underlying thinking process through more targeted reward signals during training.
Step-level reward for thinking trajectories. As discussed in Section 3, we observe that thinking trajectories are often more complex than final responses, frequently involving branching logic, self- corrections, and redundant reasoning. To better align these two, incorporate straightforward alignment score yalign that measures the semantic similarity between each step in the intermediate thinking trajectories and each step in the final response dç:
(5) where & is pretrained encoder and sim(.,-) denotes cosine similarity. This alignment score uses the final response as a learning signal for earlier thinking trajectories, encouraging those that are topically relevant to the final response and penalizing hallucinated off-topic content.
aalign s (st), ((a)), rt
Concurrently, to avoid over-penalizing complex yet meaningful thinking trajectory steps that may not be semantically aligned with the final response, we incorporate a complementary quality score qual Inspired by the LLM--s---jjude paradigm [84, 47], employ a strong expert model (e.g.. GPT-40) as judge J to evaluate the logical soundness of each step in context:
r( qual
J(st x, a).
(6)
The quality score is designed capture deeper aspects inside reasoning traces, including step correctness, internal coherence, and progression toward the final response.
In addition to alignment with the final model output and logical step quality, we apply step-by-step coherence score r to ensure contextual compatibility between adjacent reasoning steps using a contrastive mutual information formulation. Specifically, we model the coherence between each thinking trajectory step s, and its predecessor s;-1 by contrasting their embedding similarity against A negative samples drawn from unrelated trajectories:
coh llog exp(sim(0(sr-1)) ()) ((s:))/T)
EveN wxwpxxxppppxxxxpppppp (8)))))
where the temperature parameter. By penalizing incoherent transitions or topic shifts, the coherence score encourages each step to be semantically and logically consistent with its immediate predecessor while remaining distinct from unrelated or disjoint reasoning steps. Finally, to aggregate the alignment, quality, and coherence scores into unified reward signal, apply softmax- based

---

weighting over the three components:
Step
softmax (rath t
(8)
E (alig, qua, coh)
While the step-level rewards offer fine-grained su- pervision on the completeness and coherence of individual reasoning steps, they might not fully assess whether the overall problem-solving strategy encoded in model's thinking trajectory is reliably leads correct solutions, derived from the final response. We thus introduce camplate-guided trajectory-level reward to evaluate each trajoose data at higher level of abstraction [69, 68].
Template-Guided Trajectory-leve Reward.
Specifically, given an input problem æ and the distilled trajectory-responee y = 8 $ a, we employ strong expert LLM (e.g.. GPT-4o) verifier v. The verifier processes the complete output and extracts a reasoning template T, which captures the high-level strategy underlying the original trajectory-response trace. By abstracting the high-level strategy, the template provides structured guide for subsequent reasoning. The detailed prompt used for template generation is provided in Appendix B. Next, a policy model ne is conditioned on the extracted template and tasked with solving the input problem x by strictly adhering to the prescribed template T. The model generates N chain-of- of-thought responses as follows:
yd ..... (N)
TT(- x, T).
Then, we define the trajectory-level reward linal the average correctness of the generated responses:
N
T fimal
Σ N
1
1(y0) is correct).
(9)
The template- guided trajectory-level reward evaluates whether the high-level reasoning strategy can be generalized and executed by the policy model independent of the low-level execution in the original trace.
Joint Training Objective. To fully leverage both step-level and trajectory- supervision signals, we integrate the previously defined rewards and propose the following joint training objective:
T
1 Livtal >aepp T Catep (Ro(st I. 8ige, u), (((P)) + Aanal Limal (Ro(I, v), fina))
(10)
where we adopt mean squared error (MSE) as the loss function for both the step and trajectory reward supervision, and Asep and Agmma are tunable parameters to balance the relative contributions of fine-grained step supervision and high-level strategic feedback. We train ReasonFlux-PRM, with this joint objective as the practical surrogate for the optimization objective in Eq. to align with both token-level and trajectory-leve reward signals, thereby enabling the supervision effectiveness on the trajectory-resonss data.
4.2 Offline Data Selection and Online Reward Modeling
We elaborate on the utilities of ReasonFlux-PPi -PRM from two perspectives: (i) Offline trajectory- response data selection, where ReasonFlux-PRM. is used to identify and select high-quality reasoning traces for downstream supervised fine-tuning and reinforcement learning: and (ii) Online reward modeling, where ReasonFlux- provides token-level and trajectory-level reward signals during RL training, and enables efficient reward estimation for test-time scaling.
Offtine Data Selection. For offline data selection, ReasonFlux-PRM assigns each trajectory-response pair (z, y @ a) step-level reward sequence { Jt=1 for each reasoning steps and trajectory- level reward plinal The overall score is computed as:
Σ
Pt
+ pfinal
(11)
T
where a balances the contributions of local and global reward signals. The aggregated score applied to filter samples for later downstream supervised fine-tuning of smaller models.
7

---

Online Reward Modeling, We first leverage ReasonFlux-PRM to produce composite reward signal that guides policy optimization through process-level supervision during reinforcement learning. Specifically, during the RI. training, we incorporate ReasonFFlx-PRM into the Group Relative Policy Optimization (GRPO) [.1]. By default, GRPO optimizes for the outcome-level reward Youts which reflects the task accuracy of the policy xe each training sample. To incorporate process-level supervision from ReasonFlux-PRRe PRM, we augment this reward with the PRM-based reward in Eq. 11. Given input and sampled response y ~ TT(- x)), the new composite reward used for policy training after incorporating ReasonFlux PRMM then becomes:
new (1 3) x Fout B.
(12)
where 3 controls the relative weight of supervision from i. With a total of G group size (i.e., number of sampled responses per input), we proceed with group-normalized advantage estimation as:
mean (now GG
Inew
Anew
(13)
std( ( new
j==1
With the ReasonFlux-Pai derived advantage term Anew, we then update the GRPO objective by:
molsie B Bloce)
Teencells JeenseEhes.PM..GRU PRM (RFO E. 1o. (ln))
min
naw (MG4 (iss) Arcws
mm1
clip
1 e, +c) Arcn 6Da Hwel)
(Blie Wi, Ml,cs)
(14)
Note that ReasonFlux-PPe can be seamlessly integrated into other online RL policy optimization algorithms such as PPo [48] and + [15] by replacing the reward signal with ReasonFlux- PRM 's composite rewards.
Reward-guider Test-Time Scaling. During inference, we further apply ReasonFlux-PRi into uss-ttmm-scallig strategies such as Best-of-N to identify the most promising output from set of generated candidates. For each new input question and its corresponding set of sampled model responses, ReasonFlux-PRM. assigns a score to each response based on the formulation in Eq. 11, and selects the response with the highest score the final output.
Empirical Evaluations
We empirically evaluate ReasonFlux-PRM. focusing on two core applications: (i) Offline data selection, where ReasonFFlx--PRe identifies high-quality reasoning traces improve supervised fine-tuning; and (ii) Online reward modeling, where ReasonFlux-PRM, offers reward signals for Best-of-N decoding strategy in test-time scaling and GRPO-based policy optimization.
Benchmarks. We evaluate ReasonFlux-PRA. on four representative and challenging reasoning benchmarks, including MATH500 [13]. diverse set of 500 mathematical problems of varying difficulty; AIME24 [38], consisting of 30 problems from the 2024 American Invitational Mathemat-
ics Examination (AIME); AIME25, which includes 15 problems from the 2025 AIME [37]; and GPQA-Diamond [46], a benchmark of 198 PhD-level science questions to assess advanced scientific reasoning.
Implementation Details. We train ReasonFlux-PP PRM using two off-the-shelf base models, Qwen2.5- 1. I.5B-Instruct and 20wen2.5-7B-Instruuct [54], resulting in ReasonFFux--PRee PPM--1-- and ReasonFlux- PRM-7B, respectively. The training data is primarily sourced from the public trajectory-respont reasoning traces such as OpenThought 1 14K [53]. All experiments are conducted on 8 A100 GPUs. Additional experimental setups including ReasonFlux-PPee training details and downstream tasks model configurations are provided in Appendix C.
Baselines and Models. For offline data selection, we compare ReasonFlux- PRMM with the four frontier PRMs introduced in Section 3, using Qwen2.5- 11B-5-11B-BB 141--4BBnsss 14--tsttutt6 54] as the generator model for standard supervised fine-tuning evaluations. For online reward modeling, constrained by compu- tational resources, we primarily use 7B-scale models as policy models for reinforcement learning, including Qwen2.5-7B and Deepseek-R adopt Qwen2.5-144 14B as the generator model to evaluate inference-time performance.
[6]. For test-time Best-of-N scaling, we
8

---

Table 2: Offlin Data Selection Comparison. We fine-tune the generator model Qwen2.5-144 -4B-Instr using data selected and additional baselines. The high- est performance of the generators trained on each data source bold. DeassonFlux-PRM-DD achieves better performance than the strongest human-curated baseline.
Deepseck-A1
by ReasonFlux-PRMM-
Gertin Thanking
SFT Data Source
AIME24
ADME25
MATH500
OPQA--Diaond
Human curated (sik)
33.3
33.3
78.8
41.4
00
04
16.7 (1 16.6)
20.0
13.3)
68.4
(1 6.6)
Muth-Shephend-PRMM P-M888
13.3 (0.00)
20.0)
6.7 26.6)
67 11.0)
33.3 8.1)
Figure 5: Score distributions rewarded by ReasonFlux- PRMM--------- on Deepseek- RI and Gemini over 1000 trajectory- response data.
Skywork- PRMM-PRM-7
13.3 (0.00)
[3.3
71.8
7.07
37.9 3.5)
Qwen2.5. MMMh-
26.7 (1.66)
20.0 13:3)
(U.6())).))
39.4 (.00
Qwen2. S- -att---PMM-22
33.3 0.0)
26.7 (1 6.6)
77.0 1.8)
39. (1 2.0)
model resporsess
(1 3.4)
() 6.6)
77.8 1.0)
(1 0.5)
ReasonFlu-PPMM-77 40.0 6.7) 333 (1 0.0)
84.8 (T 6.0)
47.5 (1 6.1)
Table 3: Performance of PRMs reward signals in policy optimization. For each of the two policy models, i.e. SeasSee--R--Distil----- -I-Distill-Qwen--1 and Qwen2.5-7B-Insttere we run GRPO with three different reward signals: entirely rule-based latter two non-rule- based rewards are factored into the overall reward signal according to Eq. 12. We report the mean accuracy averaged over 16 independent runs.
Twen2.5-Math 1--Math-PRR-77B and ReasonFlux-PPR
The
Policy Model
Reward Signal Source
AIME24
AIME25
MATH500 GPQA-Diamonn
Rulc-hased
12,9
11.1
73.6
32.7
@wwn2.5-Math-PRM..
12.9
13.3
74.8
32.4
ReasonFlux-PRM-75
16.3
17.1
77.2
34,9
Rule-hased
50.2
38.3
89.6
47.1
DeepSeek-R1 I--iistil-Qwenne
Qwen2.5-Math-PRM-P7
51.2
40.8
92.8
49.1
ReasonFFlux-PRMM225
54.6
44.2
94.8
51.6
5.1 Offline Data Selection
with training data selected by different strategies, including ReasonFlux-PRMMTTH baseline PRMs, and human-curated examples. Notably, ReasonFlux-PRM-77 outperforms the high-quality human-curated slk dataset. Specifically, our model achieves a 6.0% gain on MATH500 and a 6.1% improvement on GPQA- Diamond relative the human-curated baseline. We also plot the score distribution over the 1,000 trajentory-responee pairs generated by Deepseek-Ri R1 and Gemini, as shown in Figure 5. The clearly separated score distributions in the figure demonstrate that MiwsonnFux-PRRM--- effectively distinguishes between the trajectory- reliable reward signal for high-qualit, data selection.
Table presents the supervised fine-tuning results of Qwen2.5-14BB-nns..
quality generated by different models, providing
5.2 Online Reward Modeling
Reward Signal for RL training. Figure 6 and Table present the training dynamics and downstream reasoning performance after incorporating different reward signals into policy optimization via GRPO. We evaluate two 7B-scale policy models: Swven2.5-7BIInstrue and
---SSekk----iisilll -Been.ll For each model, we compare three reward signal sources: fully rule-based heuristic following the original GRPO approach, models and all evaluated tasks, CeasonnFux--PRMM. PRM-7B consistently delivers superior gains over both the rule-based and prior PRM-bbsed reward signals. On (wen2.5--BBIInsrruct ReasonFlux-PRM-- improves performance by 3,4% on AIME24 and 5.8% on AIME25 relative to the rule-based baseline. On the stronger DeepSeek-RI-Distilll raising MATH500 accuracy from 89.6% to 94.8% and GPQA-Diamond from 47. 1 1% to 51.6%. In addition, when directly comparing ReasonFFux-PRMM observe consistent improvements. For example, a 3.8% gain on AIME25 with Sween..--7B-Instmn and 2.5% gain on GPQA GPQA-Diamond with --insSee------sii---O----- that the high-quality learned reward signals from ReasonFlux-PRa substantially enhance policy
5-tt2..5-Mth--RMM and ReasonFFlux-PRMM77B Across both policy
model, ReasonFlux -RM-- further advances results,
against Qwen2.5-Math Mar.2---Mat--PPM-7B... we
These results demonstrate

---

(8)
(b)
Weldeero Arrurer
Training Reward
0.3
(C)
0.2
GRPO (Rule-based)
0.1
ReasonFlux PRM-78
0 20
40
60
80
100
120
140
160
18
Step
Stop
Figure 6: Training dynamics of GRPO policy optimization using ReasonFlux-PRM-I reward signals and Qwen2.5-7B-Ing -B-Isrrrttt as the policy model. (a) Training reward vs. step: We com- pare the training reward evolution across original rule-based GRPO, Swen2.5-MMathPRRM-T7 and ReasonFlux-PRM-755 (b) Validation accuracy vs. step: We report the validation accuracy during training with ReasonFlux-Pee Flux-PRM-7B;; (c) Response length step: We report the evolution of generated response lengths over training steps with ReasonFlux-PRM-73
optimization, outperforming both heuristic and strong PRM baselines, and ultimately yielding more capable reasoning models through RL training.
Additional Performance Analyses. We leave further performance analyses ReasonFlux PRMM and case studies in Appendix D and Appendix E.
6 Related Works
Process Reward Models. Process Reward Models (PRMs) [27, 55, 78, 50, 36, 65] provide
step-level supervision for model reasoning answers, assigning intermediate rewards to each reasoning step [79, 60, 33, 4, 42]. Existing PRMs, such as Math-Shepherd [59], Skywork-PRN. [10], and Qwen2.5-Math-PM MMM series [79], are trained either human-annotated rewards [27] synthesized supervision signals [35] to provide fine-grained step-level rewards for model-generated reasoning solutions across different tasks such math problem solving [38, 34], science reasoning [46], and programming [12]. More recent work such as Think-PRe PRM [18] introduces a generative PRM to produce long CoT verification. Prior works have integrated PRMs reward signals during training [56, 49, 23, 8, 85, 82], such as step-b verified online RL policy optimization [8, 5, 86, 75, 29] or iterative generator improvement through verifier-
self self-training [14]. Others apply PRMs during inference-time scaling [52, 80, 18, 52, 87, 70, 68, 69] by integrating the models with step-level search and decoding strategies, including beam search [52], reward-guided tree search [62, 43], Best- of-N sampling [28], etc. However, since current PRMs are mostly trained on model-generater final solutions, they struggle to provide effective reward supervision for the internal reasoning trajectories produced by large reasoning models [6, 26) prior to generating final answers. To address this, we design new trajectory-aware PRM specifically aimed providing reward supervision for such trajectory-response response formatted long CoT data. We leave additional related works in Appendix F.
7 Conclusion
We present ReasonFlux-PRAt a trajectory-aware PRM that delivers fine-grained step-level and trajectory-Iever supervision for trajectory-responss long chain-of-thought reasoning traces. Through extensive empirical evaluations, ReasonFlux-PRM consistently improves downstream model perfor- mance across multiple challenging benchmarks and application settings. Specifically, ReasonFlux- PRM surpasses strong baselines and human curated data offline training data selection, enhances policy optimization during reinforcement learning via dense process-level rewards, and demonstrates superior test-time scaling in Best-of-N inference. Our results highlight the importance of trajectory- aware reward modeling for supervising model intermediate reasoning processes. The discussion of limitations and broader impacts is provided in Appendix G.
10

---

References
[1] Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, selection for language models. arXiv preprint urXiv:2402.16877 2024.
survey on data
[2] Zachary Ankner, Mansheej Paul, Brandon Cui, Jonathan D Chang, and Prithviraj Ammanabrolu. Critique-out-loud reward models. arXiv preprinl arXiv:2408.1 1791, 2024.
[3] Yihan Cao, Yanbin Kang, Chi Wang, and Lichao Sun. Instruction mining: Instruction data selection for tuning large language models, 2024.
[4] Xiusi Chen, Gaotang Li, Ziqi Wang, Bowen Jin, Cheng Qian, Yu Wang, Hongru Wang, Yu Zhang, Denghui Zhang, Tong Zhang, Hanghang Tong, and Heng Ji. Rm-rl: Reward modeling as reasoning, 2025.
[5] Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456 2025.
[6] DeepSeek-AI. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, 2025.
[7] Google, Gemini Flash Thinking API. Accessed via the Google Cloud Platform, 2025. Accessed on May 15, 2025.
[8] Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. rstar-math: Small Ilms can master math reasoning with self-evolved deep thinking. arXiv preprint arXiv:2501.0455 2025.
[9] Etash Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smyrnis, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, Ashima Suvarna, Benjamin Feuer, Liangyu Chen, Zaid Khan, Eric Frankel, Sachin Grover, Caroline Choi, Niklas Muennighoff, Shiye Su, Wanjia Zhao, John Yang. Shreyas Pimpalgaonkar, Kartik Sharma, Charlie Cheng-Jie Ji, Yichuan Deng, Sarah Pratt, Vivek Ramanujan, Jon Saad-Falcon, Jeffrey Li, Achal Dave, Alon Albalak, Kushal Arora, Blake Wulfe, Chinmay Hegde, Greg Durrett, Sewoong Oh, Mohit Bansal, Saadia Gabriel, Aditya Grover, Kai-Wei Chang, Vaishaal Shankar, Aaron Gokaslan, Mike A. Merrill, Tatsunori Hashimoto, Yejin Choi, Jenia Jitsev, Reinhard Heckel, Maheswaran Sathiamoorthy, Alexandros G. Dimakis, and Ludwig Schmidt. Openthoughts: Data recipes for reasoning models, 2025.
[10] Jujie He. Tianwen Wei, Rui Yan, Jiacai Liu, Chaojic Wang. Yimeng Gan, Shiwen Tu, Chris Yuhao Liu, Liang Zeng, Xiaokun Wang. Boyang Wang, Yongcong Li, Fuxiang Zhang, Jiacheng Xu, Bo An, Yang Liu, and Yahui Zhou. Skywork-ol open series. https: //huggingface. co /kyywok,, November 2024.
[11] Xinrui He, Yikun Ban, Jiaru Zou, Tianxin Wei, Curtiss Cook, and Jingrui He. LLM-forest: Ensemble learning of LLMs with graph-augmented prompts for data imputation. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Findings of the Association for Computational Linguistics: ACL 2025, pages 6921-6936, Vienna, Austria, July 2025. Association for Computational Linguistics.
[12] Xinyi He, Jiaru Zou, Yun Lin, Mengyu Zhou, Shi Han, Zejian Yuan, and Dongmei Zhang. CoCoST: Automatic complex code generation with online searching and correctness testing. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 14433-19451, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
[13] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021.

---

[14] Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishabh Agarwal. V-star: Training verifiers for self-taught reasoners, arXiv preprint
arXiv:2402006457, 2024.
[15] Jian Hu. Reinforee++: A simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501.03200 2025.
[16] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed Bl-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Cameyy, et al. Openai ol system card. arXiv preprint arXiv:2412.16720, 2024.
[17] Angelos Katharopoulos and François Fleuret. Not all samples are created equal: Deep learning with importance sampling. In International conference machine learning, pages 2525-2534 PMLR, 2018.
[18) Muhammad Khalifa, Rishabh Agarwal, Lajanugen Logeswaran, Jaekyeom Kim, Hao Peng, Moontae Lee, Honglak Lee, and Lu Wang. Process reward models that think, 2025.
[19] Gaotang Li, Ruizhong Qiu, Xiusi Chen, Heng Ji, and Hanghang Tong. Beyond log likelihood: Probability-baseed objectives for supervised fine-tuning across the model capability continuum. urXiv:2510.0052:0 2025.
[20] Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for "mind" exploration of large scale language model society, 2023.
[21] Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang, Shaohan Huang. Xiaolong Huang. Zeqiang Huang. Dongdong Zhang, al. Synthetic data (almost) from scratch: Generalized instruction tuning for language models. arXiv preprint arXiv:2402. 13064, 2024.
[22] Jia LI, Edward Beeching. Lewis Tunstall, Ben Lipkin, Roman Soletskyi. Shengyi Costa Huang. Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. Numinamath. [https: hnttps:/nuuggingface hot.a///..//NN///Miaaaaaaaaa---. hittss//hhhhttttpps////hhhbbb ctm/ eeject-numinne htt/aammmmmrrreeeee htt.://ggggsssssrrrroooo///////p 2024.
hatalnaaadaattttppae pdf),
[23] Wendi Li and Yixuan Li. Process reward model with q-value rankings. arXiv preprint arXiv:2410. 1287, 2024.
[24] Wendi Li and Yixuan Li. Process reward model with v-vllue rankings. In The Thirteenth International Conference on Learning Representations, 2025.
[25] Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d' Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-Ievel code generation with alphacode. arXiv preprint xrXiv:2203.07814 2022.
[26] Zhuofeng Li, Haoxiang Zhang, Seungju Han, Sheng Liu, Jianwen Xie, Yu Zhang, Yejin Choi, James Zou, and Pan Lu. In-the-flow agentic system optimization for effective planning and tool use, 2025.
[27] Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2024.
[28) Runze Liu, Junqi Gao, Jian Zhao, Kaiyan Zhang, Xiu Li, Biqing Qi, Wanli Ouyang, and Bowen Zhou. Can 1b II surpass 405b Ilm? rethinking compute-optimal test-time scaling. arXiv preprint 06700.00073..... 2025.
12

---

[29] Tianqi Liu, Wei Xiong, Jie Ren, Lichang Chen, Junru Wu, Rishabh Joshi, Yang Gao, Jiaming Shen, Zhen Qin, Tianhe Yu, Daniel Sohn, Anastasiaa Makarova, Jeremiah Liu, Yuan Liu, Bilal Piot, Abe Ittycheriah, Aviral Kumar, and Mohammad Saleh. Rrm: Robust reward model training mitigates reward hacking, 2025.
[30] Ziche Liu, Rui Ke, Yajiao Liu, Feng Jiang, and Haizhou Li. Take the essence and discard the dross: A rethinking on data selection for fine-tuning large language models. arXiv preprint arXiv:2406. 14115, 2024.
[31] Ziche Liu, Rui Ke, Yajiao Liu, Feng Jiang. and Haizhou Li. Take the essence and discard the dross: A rethinking on data selection for fine-tuning large language models, 2025.
[32] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-We Wei Chang. Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts, 2024.
[33] Pan Lu, Bowen Chen, Sheng Liu, Rahul Thapa, Joseph Boen, and James Zou. Octotools: An agentic framework with extensible tools for complex reasoning, 2025.
[34] Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, Yansong Tang, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct, 2025.
[35] Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Meiqi Guo, Harsh Lara, Yunxuan Li. Lei Shu, Yun Zhu, Lei Meng, et al. Improve mathematical reasoning in language models by automated process supervision. arXiv preprint arXiv:2406.06592, 2024.
[36] Chengqi Lyu, Songyang Gao, Yuzhe Gu, Wenwei Zhang, Jianfei Gao, Kuikun Liu, Ziyi Wang, Shuaibin Li, Qian Zhao, Haian Huang, et al. Exploring the limit of outcome reward for learning mathematical reasoning. arXiv preprint arXiv:2502.0678. 2025.
[37] math ai. AIME 2025 dataset. https://hanghhnneee..coaatttttts 2025. Accessed: 2025-05-15.
[38] Maxwell-Jia. AIME 2024 dataset. http:://nnngiiggaeeecc////t/ttts AIME_2024, 2024. Accessed: 2025-05-15.
[39] Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 36:50358-555 -03766, 2023.
[40] Niklas Muennighoff, Zitong Yang. Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannanch Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling, 2025.
[41] Jiahao Qiu, Yinghui He, Xinzhe Juan, Yimin Wang, Yuhan Liu, Zixin Yao, Yue Wu, Xun Jiang, Ling Yang, and Mengdi Wang. Emoagent: Assessing and safeguarding human-ai interaction for mental health safety, 2025.
[42] Ruizhong Qiu, Gaotang Li, Tianxin Wei, Jingrui Hc, and Hanghang Tong. Saffron-l: 1: Safety inference scaling, 2025.
[43] Ruizhong Qiu, Zhiqing Sun, and Yiming Yang. DIMES: A differentiable meta solver for combinatorial optimization problems. In Advances in Neural Information Processing Systems, volume 35, pages 25531-25544 2022.
[44] Ruizhong Qiu, Zhe Xu, Wenxuan Bao, and Hanghang Tong. Ask, and it shall be given: On the Turing completeness of prompting. In 13th International Conference Learning Representations, 2025.
[45] Ruizhong Qiu, Weiliang Will Zeng, Hanghang Tong. James Ezick, and Christopher Lott. How efficient L.M--geneatedd code? rigorous & high-standard benchmark. In 13th International Conference Learning Representations, 2025. 2025.
13

---

[46] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel Bowman. Gpqa: A graduate-level google-pros proof q&a benchmark, 2023.
[47] Swarnadeep Saha, Xian Li, Marjan Ghazvininejad, Jason Weston, and Tianlu Wang. Learning to plan & reason for evaluation with thinking saaiing-lmm-ss llm-al-a judgee. arXiv preprint 1xXi::550111899 2025.
[48] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimization algorithms. arXiv preprint
06347, 2017.
[49] Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated process verifiers for Ilm reasoning. arXiv preprint arXiv:2410.08146, 2024.
[50] Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein. Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated process verifiers for IIm reasoning. arXiv preprint 24Xiv:2410.08146, 2024.
[51] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024.
[52] Charlic Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute opti- mally can be more effective than scaling model parameters. arXiv preprint 40Xiv:2408.03374, 2024.
[53] ThenThouuhts Team. Open Thoughts. https://penn-thouhtssaas January 2025.
[54] Qwen Team. Qwen2.5: A party of foundation models, September 2024.
[55] Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process- and outcome-based feedback, 2022.
[56] Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:22/1.1422 2022.
[57] Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. TRL: Transformer Reinforee- ment Learning.
[58] Jiachen Tianhao Wang, Tong Wu, Dawn Song, Prateek Mittal, and Ruoxi Jia. Greats: Online selection of high-quality data for 1lm training in every iteration. Advances in Neural Information Processing Systems, 37:131 197-13123, 2024.
[59] Peiyi Wang, Lei Li, Zhihong Shao, R. X. Xu, Damai Dai, Yifei Li, Deli Chen, Y. Wu, and Zhifang Sui. Math Math-shepherd: Verify and reinforce llms step-by-step without human annotations, 2024.
[60] Yinjie Wang, Ling Yang. Ye Tian, Ke Shen, and Mengdi Wang. Co-evolving Ilm coder and unit tester via reinforcement learning. arXiv preprint arXiv:2506.031.0 2025.
[61] Minghao Wu, Thuy-Trang Vu, Lizhen Qu, and Gholamreza Haffari. The best of both worlds: Bridging quality and diversity in data selection with bipartite graph, 2024.
[62] Yangzhen Wu, Zhiqing Sun. Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for moblem-solving with language models, arXiv preprint arXiv:2408.000 00724, 2024.
[63] Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning. arXiv preprint arXiv:2402.0433 2024.
14

---

[64] Sang Michael Xie, Hieu Pham. Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoe Le. Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. Advances in Neural Information Processing Systems, 36:69798-6999 69818, 2023.
[65] Wei Xiong, Wenting Zhao, Weizhe Yuan, Olga Golovneva, Tong Zhang. Jason Weston, and Sainbayar Sukhbaatar. Stepwiser: Stepwise generative judges for wiser reasoning, 2025.
[66] Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. A survey on knowledge distillation of large language models. arXiv preprint arXiv:2402. 13116, 2024.
[67] An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang. Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report. arXiv preprint arXiv:2407. 10671, 2024.
[68] Ling Yang, Zhaochen Yu, Bin Cui, and Mengdi Wang. Reasonflux: Hierarchical IIm reasoning via scaling thought templates. arXiv preprint arXiv:2502.0677. 2025.
[69] Ling Yang, Zhaochen Yu, Tianjun Zhang, Shiyi Cao, Minkai Xu, Wentao Zhang, Joseph E Gonzalez, and Bin Cui. Buffer of thoughts: Thought-augmmented reasoning with large language models. Advances in Neural Information Processing Systems, 37:113519-113544 2024.
[70] Wenkai Yang, Shuming Ma, Yankai Lin, and Furu Wei. Towards thinking-optimal scaling of test-time compute for Ilm reasoning. arXiv preprint arXiv:2502. 18080, 2025.
[71] Xiyuan Yang, Wenke Huang, and Mang Ye. Fedas: Bridging inconsistency in personalized federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11986-11995,
June 2024.
[72] Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chem, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning. arXiv preprini arXiv:2502.03387 2025.
[73] Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. Demystifying long chain-of-thoughh reasoning llms. arXiv preprint arXiv:2502.03377, 2025.
[74] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprini arXiv:2503. 14476, 2025.
[75] Zhaochen Yu, Ling Yang, Jiaru Zou, Shuicheng Yan, and Mengdi Wang. Demystifying reinforcement learning in agentic reasoning. arXiv preprint arXiv:2510.11 2025.
[76] Murong Yue, Wenlin Yao, Haitao Mi, Dian Yu, Ziyu Yao, and Dong Yu. Dots: Learning to reason dynamically in lims via optimal reasoning trajectories search. arXiv preprint arXiv:2410.0388 2024.
[77] Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat. When scaling meets 1mm finetuning: The effect of data, model and finetuning method. arXiv preprint arXiv:2402.1 /7193, 2024.
[78) Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mets*: Llm self-training via process reward guided tree search. arXiv preprint 2rxiv:2406.03816,
2024.
[79] Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons developing process reward models in mathematical reasoning. arXiv preprint arXiv: 2025.
15

---

[80] Jian Zhao, Runze Liu, Kaiyan Zhang, Zhimu Zhou, Junqi Gao, Dong Li. Jiafei Lyu, Zhouyi Qian, Biqing Qi, Xiu Li, and Bowen Zhou. Genprm: Scaling test-time compute of process reward models via generative reasoning. 2025.
[81] Kai Zhao, Yanjun Zhao, Jiaming Song, Shien He, Lusheng Zhang, Qiang Zhang, and Tianjiao Li. Saber: Switchable and balanced training for efficient Ilm reasoning, 2025.
[82] Yanjun Zhao, Sizhe Dang, Haishan Ye, Guang Dai, Qian, and Ivor W. Tsang. Second-order fine-tuning without pain for Imss: hessian informed zeroth-order optimizer, 2025.
[83]
Yanjun Zhao, Tian Zhou, Chao Chen, Liang Sun, Yi Qian, and Rong Jin. Sparse-vq transformer: An ffn-free framework with vector quantization for enhanced time series forecasting, 2024.
[84] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, al. Judging lim-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595546666 2023.
[85] Jiaru Zou, Yikun Ban, Zihao Li, Yunzhe Qi, Ruizhong Qiu, Ling Yang, and Jingrui He. Transformer copilot: Learning from the mistake log in Ilm fine-tuning. arXiv preprint arXiv:2505.16270, 2025.
[86] Jiaru Zou, Soumya Roy, Vinay Kumar Verma, Ziyi Wang, David Wipf, Pan Lu, Sumit Negi, James Zou, and Jingrui He. Tattoo: Tool-grounded thinking prm for test-time scaling in tabular reasoning. arXiv preprini 17.02225 2025.
[87] Jiaru Zou, Qing Wang, Pratyush Thakur, and Nickvash Kani. Stem-pom: Evaluating language models math-symbol reasoning in document parsing. arXiv preprint arXiv:2411.0038. 2024.
[88] Jiaru Zou, Mengyu Zhou, Tao Li, Shi Han, and Dongmei Zhang. Promptintern: Saving inference costs by internalizing recurrent prompt during large language model fine-tuning, arXiv preprint arXiv:2407.02211, 2024.
16

---

Table of Contents
A Details the Preliminary Study in Section 3
18
Preliminary Study Setups
18
A.2 Difference between Model Thinking Trajectories and Final Responses
18
B Template guided trajectory-level reward design
23
C Additional Experimental Setups
24
C.1 ReasonFlux-PRM Training
24
C.2 Downstream Tasks
24
D Additional Analyses
25
D.I Best-of-N of ReaSonFlux-PRM in Test-Time Scaling,
25
D.2 Scaling up ReasonFlux- PRM Model Size on Policy Optimization
25
D.3 End-to-End Training with ReasonFlux- (SFT+RL)
25
D.4 Efficiency Analyses
26
D.5 Ablation Study
27
E Case Study on ReasonFlux-PRM
27
Additional Related Works
28
Limitations and Broader Impacts
28
17

---

A Details on the Preliminary Study in Section 3
A.1 Preliminary Study Setups
Process Reward Models. We evaluate four ofate-of-the-art process reward models for scoring the quality of the thinking trajectories data: Mathh-Shepherd-PRRM-BB [59], Skywork-PRM-7B [10], Sren2,5-Math-PRNM [79], and Qween255-Math-PRM-225 [79]. The details description for each model is shown below:
Math-Shepherd-PRRR PRMM88 [59]: A 7B PRM based on Mistral, trained with data auto- generated from Mistral-7B fine-tuned on MetaMath. emphasizes verification of step-level reasoning through process-level rewards without human annotations.
Skywork-PRM-P [10]: A PRM built on Qwen2.5- Maaath--BBBa 7 Instrutttt and trained on data derived from LLaMA-2 fine-tuned on math tasks. shows strong generalization for verifying reasoning trajectories across models and supports efficient TTS with low inference overhead.
Qwen2.5-Math-PRMM-77 [79]: Trained on
1ww.----aah-7--Istteee using data from the Qwen2.5-Matt series, this PRM offers robust step-by-sted reward signals and high compatibility with Qwen family models, demonstrating superior supervision ability TTS tasks among 7B-scale PRMs.
1ann2-5-MMth-PPMM-22 [79]: A high-capacity verifier trained on Dwen2.5-Math-722 Instruct and Qwen-generated data. It achieves state-of-the-ar process supervision and excels in guiding both sampling- and search- base TTS strategies across range of mathematical reasoning tasks.
Data Sources. For the data sources, we follow slk [40] to use its collected datasets consisting of 59K raw model thinking trajectories distilled from the Google Gemini Flash Thinking API [7], along with 1K human-curated samples from the same source and an additional IK curmand samples from Deepscek-RI [6]. These trajectories span broad range of topics, including math and scientific reasoning. For downstream tasks, we choose 4 challenging benchmarks including: AIME24 [38], AIME25 [37], MATH500 [13], and GPQA-Diamond [46].
Training Details in RQ2. As the downstream generator, we adopt Qwen2.5- I4B-rnstt as our base model for fine-tuning evaluation. We perform supervised fine-tuning on the 1eeeen..514B--nsiiii model using various 1, 1,000-sample training datasets, each selected either by different PRM-based rankings or curated by human annotators in slk [40]. We fine-tune the model for 5 epochs using a learning rate of - weight decay of 1e-1, and a maximum sequence length of 32,768. All experiments are conducted on server node with 8 A100-80G GPUs.
A.2 Difference between Model Thinking Trajectories and Final Responses
As we mentioned in Section 3, there are two key difference of the data between model thinking trajectories and final responses:
Branching steps across thinking trajectories: For instance, the thinking trajectories might initially assume an incorrect variable assignment in math problem, detect the inconsistency, and backtrack to re-derive the solution.
Weaker global coherence across steps: This manifests in speculative or uncertain state- ments (e.g., "if we assume X, then Y"") that may not resolve within the same reasoning path, and in disjointed or redundant logic, such as repeating subgoals or prematurely concluding without fully integrating prior steps. In contrast, final responses are typically globally fluent and logically unified, aiming to deliver a streamlined and conclusive answer.
In Examples I and below, we provide detailed data examples that highlight the distinctions between model thinking trajectories and final output responses. These examples showease how the thinking trajectories often contain exploratory, speculative, partially resolved reasoning steps, in contrast to the more polished and coherent final outputs. To better illustrate these differences, we highlight all branching steps, backtracking behaviors, and internal deliberations in blue, emphasizing the non-linear and less deterministic nature of the model's intermediate reasoning process.
18

---

Example 1
Question: Two capacitors with capacitance values C1 2000 + 10 pI and C2 3000 + 15 pI are connected series. The voltage applied across this combination What is the percentage error in the calculation of the energy stored in this combination of capacitors?
5.00 0.02 V.
Thinking Trajectories:
Let C1 and C2 be the capacitance values of the capacitors, and AC1, AC2 be their absolute errors.
Given: C1 2000 + pF C2 3000 + 15 pF Voltage applied across combination, V
5.00 + 0.02 V When capacitors are connected in series, the equivalent capacitance Cop is given by:
- C Ca
CiO₂
Ce
Ci + C2
Let me double check, initially thought the formula was Cs + C2. that's for parallel combination. Since it's series, we should use the reciprocal formula. Calculate the nominal value:
2000 x 3000 6 x 10 Cos
1200 pF
2000 - 3000
5000
To find the error in Cau,
V
(aCoy ACi)
AC;)
)² "+(OC: AC2
(8Cm
AC
(oc1
802
Partial derivatives:
80m
C2
d0a
C²
8C1
(C1 + C2)2
0C2
(C1 + C2)
Evaluate:
25
25
9
2
4
2
ACo
10
+
x
15
V 990
602
25
25
25
/8100 + 3600
V11700 = 4.33 pF
25
25
Hmm, but is this the only way? Mayhe logarithmic differentiation would give cleaner expression? Let me try an alternative method:
1 Ceg
dCog
dC1
dC2
C1
C2
C2
C2
So:
dC
dC'
dC'
AC
ACi
Coy
C
Ci
C
Ceg
C
This seems messier, maybe it is generalizable. Now let me compute it:
3000 10
2000 0.003
0.002 0.005 ACm 0.005 1200 pF
2000 5000
3000 5000
Now compute energy:
F 2 C2
Let double-check: voltage is squared here, so the error contribution from voltage gets doubled in fractional terms. Fractional error:
AE
ACay
AV
2
0.005 + 2 0.004 = 0.013 = Percentage emoor = 1.3%
E
Coy
V
Alternative verification might help... I might try distribute the weight of errors proportionally:
ACag
ACi
C2
AC2
C. 0.005 0.6 0.005 0.4 0.005
Ceg
Ci
O2
C2
Ci C2
Let Final check:
11)
AE
ACm Cog
AV
In E
In Cog + 21
+ 22
F.
V
19

---

The final answer
1.3%
Final Response:
Step I: Calculate the Equivalent Capacitance
When capacitors connected in series, the equivalent capacitance Coy is given by:
Ci+C2
C₁C₂
Cos C1 C2
Cog
C1
C2
C1C₂
2000
3000
6 x 10
Cog
1200 pF
2000
3000
5000
Step 2: Calculate the Error in Ceq
Using the formula for propagation of uncertainty:
3C 801
aCey 8C2
AC
AC1
AC2
Compute the partial derivatives:
8Cgg
C2 (C1 + C2)
dGe 8C2
C² (C1 + C2)
8C1
Substitute the given values:
ac
3000
0Cey
20002
4
aC1
50002
25'
aC2
5000
25
Now compute the error:
4
90
60
150
A
10
15
+
6 pl
25
25
25
25
25
Step 3: Compute the Fractional Error C.
ACes
0.005
Cen
1200
Step 4: Energy Stored in the Capacitor System The energy stored in capacitor is given by:
E
Cop 12
2
Step 5: Propagate Error in Energy Calculation The fractional error in energy E is:
AE
ACa
AV
2.
Coq
Given:
ACa
AV
0.02
0.005,
0.004
Cen
V
5.00
Substitute into error formula:
AE
0.005
+2 2 0.004 = 0.005 + 0.008 = 0.013
E
Step 6: Convert to Percentage Error
E
Percentage error
100%
0.013 100%
1.3%
Final Answer: The percentage in the energy stored
1.3%
20

---

Example 2
One base of trapezoid 100 units longer than the other base. The segment that joins the midpoints of the legs divides the trapezoid into two regions whose areas are in the ratio 3. Let the length of the segment, parallel to the bases, that divides the trapezoid into two regions of equal area.
Question:
x²a
Find the greatest integer that does not exceed
100'
Thinking Trajectories:
The problem asks for the value of |=2/100) /100], where a is the length of segment parallel to the bases that divides the trapezoid into two regions of equal area. We are given information about the lengths of the bases and the ratio of the areas created by the segment joining the midpoints of the legs.
Let the lengths of the bases be by and b2, with di b2. We are told that:
bi b2 + 100
Let the height of the trapezoid be h. The midline of the trapezoid has length:
11 b2
2
It divides the trapezoid into two smaller trapezoids, each of height h/2. The area of upper trapezoid
is:
h (b) + 3b22
A1 2 (b2 + m)
2 8
The area of the lower trapezoid is:
A2 (b1 + m) 8 (361 + b2)
We are told that the ratio of areas is 2: 3 (upper to lower), so:
A
b + 36g
2
3(b, + 3b2) = 2(3b + b2) = 3b1 + 9b2 = 6b, + 2bg + 7b2 = 3b1 +
1₂
3b, + b2
3
Substitute
= by + 100:
7b2 3(b2 + 100) = 7b2 362 + 300 = 4b2 300 = b2 = 75 = b2 = 175
Let also test the other case just in case it's typo, what the area ratio was actually 3 2?
01
3b2
3
2(b + 3b2) 3(3b: b2) 2b 6d2 9h:
3h2 3b2 7h1
3b b2
2
362 7(bz + 100) 3b2 762 + 700 4b2
700 + b2 175
That's invalid as the length can't negative. the correct ratio 3 and b2 75, 01 175. Now consider a segment of length æ, parallel to the bases, that divides the trapezoid into two equal-area area regions. Let the height from by up to this segment be h'. The total area is:
1 (6: + b2)h =
1 (175 + 75) 125h
Awaal
2
We want:
125h
(b2 z)h'
= (b2 + æ)h' = 125h
2
2
We also know from similar triangles:
b2
bi
bj
h'
by
W
h
lt
by ba
Substitute into the area equation:
(b2 + x)) h
r
b2
125h = (b2 + [x b2) = 125(b1 b2)
b b2
2 b2 125(b,
b2) = x² = 125 .2)) + b2
Substitute 175, by 75:
x²
18125
x² 125(100) + 752 12500 + 5625 18125 +
= 181
100
100
21

---

also saw a different derivation leads directly to this result. me verify it:
2n.²
b
bj
62
175
75
30625 + 5625
18125
x²a
181
2
2
100
Final Response:
Step 1: Set up variables and the area ratio condition
Let the lengths
two bases the trapezoid be and ba. with b2. are given that
b1 b2 + 100.
The segment that joins the midpoints of the legs (the midline) divides the trapezoid into two smaller trapezoids with heights h/2. The lengths of the parallel sides of the upper trapezoid are b2 and m t12 and area is
A (b: + 3b2).
The lengths the parallel sides of lower trapezoid are m b, area
A
(3b1 + b2)-
The ratio of the areas is A A2 (b) 3b2) (3b1 + b2). Given that this ratio is 3 If & b-t2
4by 3: then 36, 962 6b; + 262 so 7b2 = 36, Substituting b = b2 + 100, we get
7b2 3(b2 + 100),
so 4b2 300, = and b1 175. Step 2: Verify the alternate area ratio leads to contradiction
If 61+362 2: we get 2b, + 6b2 961 3b2, so 3b2 7b, which leads to negative value for b2, 36,
which is impossible. The lengths of the bases are b1 = 175 and b2 = 75.
Step 3: Express the condition for equal-area division
Let the segment of length parallel to the bases divide the trapezoid into two regions of equal area. The square the length of such segment is given by
6
2
1752
+5
+
30625 + 5625
36250
18125.
2
2
2
Step 4: Compute the final result
We need to find greatest integer that does not exceed x /100.
18125
181.25.
100
100
The greatest integer that does not exceed 181.25 is
[181.25] 181.
Final Answer: The final answer 181
22

---

B
Template guided trajectory-level reward design
Prompt Design of the Template ILLM-as-a-Verifier
You are given long chain-of-thought (CoT) response to a challenging math problem. Your task is to summarize the response into a structured sequence of reasoning steps that can serve as clear and guided template for use by a smaller model.
Problem: [problem]
Response: [response]
Instructions:
Please summarize the response concise list of reasoning steps, each capturing distinet part of the thought process. These may include restating the problem, defining variables, constructing mathematical models, performing calculations, verifying results, handling different cases, correcting mistakes, and drawing the final conclusion. Focus on preserving the logical flow while keeping each step clear and concise.
Here are few template examples you should strictly follow:
[Template Example
[Template Example 2]
[TTmmlate Example
Write your answer below.
23

---

C Additional Experimental Setups
C.1 ReasonFUlx-PRM Training
For ReasonFlux-PPe training, we initialize from the fff-the-shelf Qwen2.5-1.5B-Instree and Dwen2.5-7B-Insttren models [54], serving as our 1 1.5B-scale and 7B-scale ReasonFlux-Pi back- bones. We then train ReasonFlux-PRM on the OpenThoughts- 14K [53] collection of datasets containing rich, model-generated thinking trajectories paired with their corresponding final responses.
OpenThoughts- 14k publicly available synthetic reasoning dataset comprising 114,000 high- mathematics, science, code, and puzzles. Each example includes problem statement, thinking trajectory generated by the Deepseek-RI, and corre- sponding model response. The dataset was constructed by curating prompts from existing datasets, such as A--MO/NninaMMthh-C
quality examples across four domains:
[22] for math, DeepMind/code-come contesssssssssss/seeeooo [25] for code, and camel- ai/chemistry [20] for science. We utilize the model-generated thinking trajectories and final responses from the datasets as raw training data. Subsequently, we assign step-level and trajectory- rewards based on our specific reward design, as detailed in Section 4.1.
We follow our detailed description in Section 4.1 to train with the step-level reward. In addition, to train with the template- emplate-guided guided trajectory-level reward, we first randomly sample 1000 problem- response samples from OpenThoughts 14k, and prompt GPT-4o extract the reasoning template from each CoT sample using the prompt in Section B. For each cooblem-template pair, we choose Owen2.5--B-Inntruuc as our generator Ty and generate N responses that attempt solve the problem while adhering to the reasoning template. The ground truth trajectory-level reward is then computed as the average accuracy across the 5 responses, as shown in Eq. We then combine the step-level and trajectory- level rewards to obtain the ground truth reward values for the 1000 samples, and train ReasonFlux- -RMM to learn these reward values using the joint training objective Eq. 10. To train our reward model, we use a learning rate of le-5 and train for 3 epochs.
C.2 Downstream Tasks
For oflline data selection and subsequent supervised fine-tuning, we follow the exact experimental setup described in Appendix A.1 to ensure fair comparison with baseline models. Specifically, we pair in OpenThoughts- I 14k using the
begin by assigning reward scores to each designated reward model. We then rank all samples based on their aggregated reward scores and select the top 1,000 examples to serve as the training set for downstream fine-tuning.
For online policy optimization, we use a training dataset comprising 10k competition-level math- ematical reasoning problems collected from MATH [13] and the DAPO [74] training set. These training data contains math problems spanning wide range of topics, including algebra, geometry, probability, and precalculus. Our GRPO training framework is built on the original Hugging Face GRPO Trainer [57]. We train with a batch size of 32, generating 6 samples per prompt, and run training for epochs. As mentioned above, the vanilla GRPO relies on a rule-based reward that evaluates only the correctness of the final answer. On the other hand, we replace the rule- based reward with the learned reward signal obtained by passing the training prompt and the policy model's output through ReasonFlux-Ba PRM.
For the Best-of-N test-time scaling experiments, use Qwen2.5- 10--11B--sstte Instruct as the generator model. Given an input problem v, the generator produces N candidate reasoning trajectories using nucleus sampling with temperature T' 0.3, where N {2, 4, 8, 16). Each candidate trajectory is then scored by ReasonFFux-PPR PRMM which provides a scalar reward reflecting the trajectory's 's quality in terms of correctness, coherence, and reasoning structure. The final output selected as the trajectory with the highest ReasonFlux-Pei PRM assigned reward. We evaluate performance by measuring final-answer accuracy over the selected outputs.
24

---

D
Additional Analyses
PannorrPa HMN 16
Dwerdd Matro HMM
Saywor 7B
Maurty
CPQA-Dia
32-
Accurcy ONC
15
18
21
2
Number Solutions (M)
Number Solutions [M]
Number Sotutions (8)
Figure 7: Test-time performance of Best-of-N selection using PrassonFlu-PRMM--1 Qwen2.5-Math- PRM-72B, and Skywork- PRMM-BB across reasoning tasks. We also report results using the majority voting method.
D.1 Best-of-N of ReasonFlux-PRM Test-Time Scaling.
In Figure 7, we present Best-of-N selection results using ReasonFlux-PRM-7B and baseline PRMs across four reasoning tasks. For the generator model, we use the fine-tuned Qwen2.5- 1122255-11--B- 14s-Inst with the same checkpoint in Section 5.1. KeasonFlux-PRMM- racysuggest that larger reward models yield more accurate and informative signals for RL, thereby facilitatingor flat returns with increased sampling, ReasonFlux-PRMM77 maintains strong upward trend, demonstrating superior ability to identify high-quality reasoning traces.
consistently leads to greater accu-
D.2 Scaling up ReasonFlux-PPe Model Size on Policy Optimization
Table 4: Scaling Effects of ReasonFlux-PRR, model size on GRPO online policy optimization performance. Larger ReasonFlux-PRM. reward models (7B vs. 1.5B) consistently yield better downstream performance MATH500 and GPQA-Diamond across both Qwen2.5-7BBInsttue and DeepSeek-RI-Dissilll 7 policy models.
Policy Model
ReasonFlux-PRM Size MATH500 GPQA-Diamond
1.5B
73.8
30.8
@wee2.5--B-Innstee
7B
77.6
34.3
1.5B
90.4
48.5
DeepSeek- RI- RI-Distill Swiill--ee--BBB
7B
93.8
51.5
To investigate the impact of reward model capacity, we vary the size of the ReasonFlux-PRM model used provide rewards for GRPO-based policy optimization. As shown in Table using larger ReasonFlux-PPi model consistently improves performance across both policy models, Qwen2.5- 7B-Instruct and -ere-Seek-R--iiill----e---- to 7B leads to 3.8% gain on MATH500 and 3.5% on Likewise, for DeepSeek-F R Qwen- we observe 3.4% improvement on MATH500 and 3.0% on GPQA-Diamond These results indicate that larger reward models provide more accurate and informative signals for RL, thereby enabling stronger policy optimization.
Specifically, scaling ReasonFlux PRR from 1.5B
for Qwen2.5-77 -----Insttuctt
D.3 End-to-End Training with ReasonFlux- (SFT+RL)
As supervised fine- finettuning followed by reinforcement learning (SFT+RL) has become dominant paradigm for aligning large language models with reasoning-intensse tasks, we are motivated to evaluate if ReasonFlux-PRM. can serve as ceneral-purposs reward model to be effectively applied across both stages of training. Table presents comparative analysis on the 5-7en225-7B-nssttruuc policy model, where we evaluate two checkpoints: (i) the original backbone model, and (ii) the same model after SFT on ReasonFlux-PRMM- selected data over the 59K raw data in [40]. Both checkpoints then further optimized with different reward signal sources during RL. The results
25

---

Table 5: Effect of ReasonFlux-PRM- selected supervised fine-tuning on downstream RL. We compare the original backbone model (Checkpoint 1) and the model fine-tuned on Ik ReasonFlux- PRM-7B selected data (Checkpoint 2), each evaluated under different reward signal sources.
Policy Model (Qwen2.5-7B-Instrucc)
Reward Signal Source
MATH500
Rule- Rule-based
74.0
Checkpoint I:
Original backbone model
Qwen2.5-Matte --thh-----------77
75.4
ReasonFlux-PRMM-B5
77.0
Rule-based
84.8
Checkpoint 2: SFT Ik ReasonFlux-PRM-T7 selected data
87.6
BissonFlux-PRMM-11
89.8
8F7 Mwenn5-11--hauuuc
SF S 144-irustt
Accuracy on MATr5000 (%)
60-
Accuracy on MATHSCO
90-
80
Qperd
Poosu Fax HiN 1.98
20
Puman sovected bill
Skywere-Rou-77
Owts (96)
1000
Number of Training Semples
5000
10000
590000
155
Parameter Size of PRMs
78
728
Figure 8: Effeciency Analyses on ReasonFlux-PRM-T7 Left: Accuracy on MATH500 improves steadily the number of ReasonFlux both human-selected (1k) and full raw data (59k) baselines with fewer total training instances. Right: ReasonFlux-PRM-I achieves higher accuracy than other PRMs under 7B scale and even larger 72B scale parameter size.
selected training samples increases, outperforming
demonstrate that ReasonFlux--PRN PRM-7B consistently improves downstream performance at SFT and RL stages. We also observe that across all reward signal sources, fine-tuning 1k ReasonFFlux-RR. 7B selected data consistently improves performance over the original backbone model. Notably, the guided
combination of ReasonFlux-PRMN PR-M--B based supervised fine-tuning and CeasonFlux-PRMM- reinforcement learning yields the highest MATH500 accuracy of 89.8%, with a significant 12.8% accuracy improvement compared to the original backbone model (77.0%). These results highlight the end-to-end effectiveness of ReasonFlux-PPe as a general reward model for enhancing reasoning capabilities throughout the full training pipeline.
D.4 Efficiency Analyses
In this section, we evaluate the efficiency of BimtonFluxPPRM-777 in both offline data selection for SFT and online RL settings by comparing the training performance and overhead under different data and reward supervision strategies.
Supervised Fine tne-tuning
Pallcy Optimization (RL)
As shown in Figure 8, the data selected by ReasonFlux- PRM-7B reduces the amount training data required while achieving superior model performance. When fine- tuning Qwen2.5-14B-- InBtrurtt on only samples selected by PaassoFFux-PPMM--1
the model outperforms the base- line trained on 59k raw trajectories by substantial margin on MATH500. This highlights ReasonFlux-Pre -RM 's ability to identify high-quality, informative samples that yield greater performance per data point. The result aligns with recent findings on the power of curated supervision in data- efficient post-training, and further shows that ReasonFlux- PRM-7B can outperform even numan-selected samples under similar data scales.
Figure 9: Time overhead of ReasonFlux- PRM- during SFT and RL stages. For SFT, we compare the training time using 1 selected samples versus the full 59k raw data. For RL training, we evaluate the overall time with/without incorporat- ing ReasonFlux-PRM-773
We further investigate the overhead of incorporating ReasonFlux-PRM into policy optimization using the GRPO framework. As shown in the right
26

---

Table 6: Ablation study on the a parameter.
Table Ablation study on the parameter.
AIME24
AIME25
MATH500
8
AIME24
AIME25
MATH500
0.1
26.7
6.7
81.2
0.1
10.0
6.7
73.6
0.8
40.0
33.3
83.6
0.3
13.3
13.3
74.4
1.0
33.3
33.3
84.8
0.5
13.3
6.7
75.2
1.5
33.3
40.0
83.2
0.8
20.0
16.7
76.8
panel of Figure 9, although ReasonFlux-PRMM. introduces additional computation for step- and trajectory-level reward modeling, the increase in total training time remains moderate compared to standard GRPO. Crucially, this additional cost leads to consistent improvements downstream reasoning performance, and offline settings above demonstrate that ReasonFlux- across both SFT and RL regimes, but does with minimal computational overhead, achieving superior efficiency in reasoning-centric fine-tuning and optimization pipelines.
demonstrated in main experiments. Our experiments both online
not only improves model performance
D.5 Ablation Study
Ablation As described in Eq. 11, the parameter controls the balance between step-level rewards and the trajectory-level reward during ReasonFlux-PRM' reward aggregation. To assess the impact of this weighting, conduct ablation study by varying to select offline fine-tuning data accordingly. The @wwe2.55-14BInnse
0.8, 1.0, 1.5), and
use ReasonFlux-PRMM.. model is then fine-tuned the top 1,000 selected examples and evaluated across AIME24, AIME25, and MATH500. As shown in Table 6, performance improves when more weight is placed on the trajectory-level reward. Notably, a 1.0 achieves the best result MATH500, while 1.5 yields the highest accuracy on AIME25. These results suggest that combining both local (step-level) and global (trajectory-Ievel reward signals is essential, and that moderate emphasis on trajectory-level reasoning is particularly beneficial for complex tasks. We also observe that the optimal value of a may influenced by the underlying data distribution. As part of future work, we plan to make a learnable by introducing a lightweight neural module that dynamically adapts the weight between step-level and trajectory-
rewards based on the characteristics of each input sample.
Ablation on 3. In Eq. 12, we introduce 3 as a weighting coefficient to balance the original rule-based GRPO reward and the process-level reward provided by ReasonFFux-PRM--
To understand its influence, we conduct an ablation study by varying 8 E 1, 0.3, 0.5, 0.8} and applying GRPO with ReasonFlux-PRN.
reward integration on the Qwen2.5- ----truuutt policy model. As shown in Table 7, we evaluate the resulting models across AIME24, AIME25, and MATH500. The performance consistently improves with increasing B, indicating the effectiveness of ReasonFlux-PPe level supervision. The highest gains are achieved at 8 = 0.8, which yields 20.0% accuracy on AIME24, 16.7% on AIME25, and 76.8% on MATH500. The result demonstrates that a stronger emphasis on ReasonFlux- PRM rewards leads to more effective RL training.
process-
E
Case Study on ReasonFFlx-PRM
In Case Studies and 2 below, we present two responses to the same mathematical problem, one incorrect and one correct. For each response, we show the step-level and trajectory-level rewards assigned by ReasonFFlxxPPRM-77 In the incorrect response, ReasonnFlux-PRM-77 effectively identi- fies the point which the reasoning begins deviate by assigning significantly lower rewards to the subsequent steps. In contrast, the correct response receives consistently high rewards throughout the trajectory. Our study on the example suggests that ReasonFlux- distinguishing between correct and incorrect solutions but also of pinpointing the precise step where an error reasoning occurs.
is not only capable of
27

---

F Additional Related Works
Offfine Data Selection for CoT Reasoning at Scale. The quality of data has proven pivotal in the model training process [39, 77, 83, 11]. Recent studies further demonstrate that small subsets of high-quality data can outperform much larger unfiltered datasets in enhancing model reasoning abilities [41] during post-training stages such supervised fine-tuning [30, 71, 40, 72, 1, 45, 19]. In contrast to online batch data selection methods [17, 58], which select samples based on updated model signals such as gradient norms or maximum sample loss during training, offline data selection approaches aim to select data once prior to the model training process. Motivated by the need for efficiency at scale, recent works have increasingly explored offline data selection as a means of curating high-quality datasets for LLMs training. Beyond simple rejection sampling, these approaches either train an additional model for data selection [64, 63], or adaptively select data based on natural language quality indicators [3], dataset diversity [61], model-specific quality labels [31, 88). More recently, model distillation [21, 66] has been widely adopted to leverage longer reasoning traces distilled from large-scale reasoning models as training data for improving the capabilities of downstream smaller models. Methods such as sl [40], LIMO [72] and ReasonFlux [68] adapt smaller subsets of human- selnaned high-quality distilled data, enabling smaller models to perform better sophisticated reasoning tasks compared to training on much larger quantities of raw distilled data. Building on these insights, instead of incurring additional computational costs by focusing solely training data selection, our work extends the applicability of process reward models from traditional reward supervision to offline data selection, particularly in the context of raw model-distilled chain- of-thought reasoning trajectories [76, 9, 81]. Leveraging the step-by-step supervision capability of PRMs, we utilize them as metric select high-quality reasoning traces from "silver" distilled data [73], with the goal of improving downstream post-training performance for smaller models.
G Limitations and Broader Impacts
Limitations While Reason demonstrates robust improvements in reasoning quality across diverse mathematical and scientific benchmarks, limitation lies in its reliance on high-quality trajectory-respons. pairs for training. Although ReasonFlux-PRe PRM is designed to be general-pon general-purpose and adaptable, its current instantiation tailored toward domains with well-structured logic (e.g., math and science), where reward signals based on alignment, quality, and coherence can be meaningfully computed. Applying ReasonFlux-PRe in more open-ended tasks such as commonsense dialogue or code generation may require redefining trajectory-leve reward criteria and further tuning of the reward decomposition strategy. However, these domains fall outside the scope of this work and do not detract from the validity or strength of our conclusions in the reasoning- focused setting.
Broader Impact This paper introduces ReasonFlux-Pie -PRM- trajectory-aware reward modeling framework that advances L.LM reasoning by providing fine-grained supervision over both intermediate thinking steps and high-level solution strategies. This improves LLM alignment, interpretability, and efficiency in structured reasoning tasks such as math and science, while supporting both supervised fine-tuning and reinforcement learning. While our work has broad implications, we do not foresee significant negative impacts or societal concerns that warrant specific emphasis.
28

---

Case Study 1
what the greatest possible value of (a+ (?-
Question: 1 and و
Thinking Trajectories:
The problem asks for the greatest possible value of the expression a
1 5
given the ranges
for and b.
[Step-level reward: 0.997]
First, let's expand the expression:
(0))
= =
1 1/ =a. -a b
a
+
[Step-level reward: 0.983]
Now need to find the maximum value of -a² + 8 given the constraints -6 S -2 and b S 5. [Step-level reward: 0.996]
Consider the terms -a² and 1 separately, Since -6 a S -2, we have 0 S (--))))( 36. Therefore, -36 36 < -a² x -4
52
[Step-level reward: 0.503]
Since 3 < S 5, have [Step-level reward: 0.895]
1² < 25. Therefore, 25 S 1 62 1 9-
We want to maximize -a² + t. This is equivalent minimizing -a² and maximizing y [Step-level reward: 0.297]
(This the place where thinking trajectory went wrong, because order to maximize a² + g2. a² needs be maximized, not minimized. ReasonFlux- PRM assigns low reward starting from this segment within the thinking trajectory, which indicates that ReasonFlux: MRX is able to identify ervors in reasoning traces.)
The minimum value of -a is -36.
The maximum value of μz
(Step-level reward: 0.238]
(The step-level rewards drop significantly since the last wrong step.)
the greatest possible value of -a²
36
Let's calculate this value: -36
[Step-level reward: 0.280]
Final Response:
Given the ranges 6 < < -2 and b 5. We want find the greatest possible value of the
expression (a+1)
[Step-level reward: 0.995]
First, we expand the expression: (a :) (1. a : a) a - = - a² uz
a² +
[Step-level reward: 0.997]
Now we need to find the maximum value of -a² 2 given the constraints for and b. Consider the
term -a². Since -6 the values -a² range from 336 to -4.
-2, the values of a range from (-6)2 36 to (-2)² Therefore,
[Step-level reward: 0.988]
Consider the term gr. Since 3 < < 5, the values of 6² range from 3 99 52 25. Therefore, the values of range from 25 to
29

---

[Step-level reward: 0.897]
To maximize -a² + g2 we need to maximize &2 and minimize -a". The maximum value of is g 62
which occurs when = 3. The minimum value of -a² ² 336, which occurs when = -6. [Step-level reward: 0.108]
(The low score here aligns with the low in the model thinking trajectories.)
Thus, the greatest possible value -a² +
1 is 52
-36
+ 9*
36 x
324
1
324
323
36
+
9
9
9
9
323
Final Answer: The final answer
9
[Step-level reward: 0.319]
(Trajeccory-level reward: 0.197]
(As the final answer incorrect, ReasonFlux-PRt assigns low trajectory-level reward the overall trajectory-respone data.)
30

---

Case Study 2
Question: - and E & what the greatest possible value of (a+ (?-
Thinking Trajectories:
The problem asks for the greatest possible value of the expression a 5/ )) given the ranges
for and b.
[Step-level reward: 0.998]
Let E = + 3) ( - a). We can expand the expression:
E (1) + + a-a))) (
E
E = aa²
[Step-level reward: 0.985]
need to find maximum value of E given constraints -6 a and 566 5. [Step-level reward: 0.998]
Since E a 3. need minimize a² and maximize 82 [Step-level reward: 0.974]
For a, the range is 6 S a S 2. The minimum value of a² occurs when is closest to 0. The values of a are negative, so the minimum value of a² occurs when is the least negative, which is -2. When = 22 a² (-2)² 4. [Step-level reward: 0.962]
For b, the range is 3 < b < 5. The maximum value of f occurs when b is the smallest.
When b= 3. 822 32
[Step-level reward: 0.985)
So, the maximum value of F. occurs when a² is minimized and & is maximized. Maximum E -a² [Step-level reward: 0.818]
4
To add these values, we need a common denominator: Maximum E = -4
38
38-1
[Step-level reward: 0.815] Let's double check calculations.
The expression is (a + b) ( a).
Expand expression: a + a (-a) (-aa) (-a)
a² 3 E
-a- [Step-level reward: 0.980]
We want to maximize -a² + +
-a² 5. 0. Since a² > 0,
Since 6² > 0, 전 0.
The maximum value of
g. which occurs when = 3.
We need to find minimum value of a² in range 1-6.-2)
The minimum value af occurs at the boundary of the interval, which is a -2
When = 2, a² (-2)² 4.
[Step-level reward: 0.927]
We need to maximize f the range [3, 5).
The maximum value of i accurs the boundary of the interval, which is 3.
When b = 3, 82 [Step-level reward: 0.989)
31

---

So the maximum value of the expression is -a² 12
-4+
36
35
Final Respouse:
Given the ranges for a and -6 a a -23
[Step-level reward: 0.958)
(a+ -a)
We want to find the greatest possible value of the expression
[Step-level reward: 0.999]
First, expand the expression:
a
a
b
+
(-a) +
+ (-a)
a
a
a + b
IStep-level reward: 0.9981
To find the greatest possible value of this expression, we need to minimize a² and maximize 1 [Step-level reward: 0.987]
82.
For term 2 we want to minimize a², Since a² 2 for real numbers a, the minimum value a² occurs when a = 0. However, the given range for a is -2,, so a = is not within the range. The minimum value of a² in the given range occurs at the boundary of the interval, which is a = -2. When 2, a² (-2)² 4.
[Step-level reward: 0.967]
For the term f we want to maximize g Since b² > for all real numbers & O, the maximum value of fa occurs when 6² is the smallest. The smallest value of 6² the given range occurs when b the largest in absolute value within the interval [3, 5], which is When 3, 6² 3 9-
[Step-level reward: 0.667]
Now, we substitute the minimum value of a² and maximum value of into expanded expression:
1
= 4+
9
To combine these terms, find common denominator:
x
36
36
35
4
9
9
9
9
9
9
9
[Step-level reward: 0.993]
Thus, the greatest possible value of the given expression is [Step-level reward: 0.988]
35
35
Final Answer: The final answer
[Step-level reward: 0.999]
(Trajectory-level reward: 0.949]
(As the final answer is correct, ReasonFlux-PRS assigns a high trajectory-leww reward to the overall trajectory- data.)
32

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our claims are summarized properly in our Introduction section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Limitation discussion is included in the Appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texx system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess..
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
33

---

Justification:
We provide detailed assumptions and complete proof in the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
The paper provides detailed descriptions of the method implementation, training details, and experimental setup in the Experiment Section and Appendix.
Justification:
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
34

---

Answer: [Yes]
Justification:
Our code implementation is submitted along with the manuscript.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /niiss wwtppcccpppppaeeeccccecccccceeee / siiissssiiiiiyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Experiment details are provided detail in the Appendix. Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Each experiment's results reported as the average three independent experimental runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
35

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The experimental details are provided in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //neurip///uri..... tttpss//aarrss.cp/bb..tc// wettthhhsccceeeleeceee ines?
Answer: [Yes]
Justification:
We comply with the Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We include the broader impact discussion in the Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
36

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not observe such risks of misuse in this paper.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All existing resources are properly cited in this paper.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
37

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release any real- dataset. Other code implementations are detailed as supplementary material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification: No human objects are involved in this paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
38

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification:
The LLM usage is described in detail in this paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy (https: http://eeeripp..pe cc. cceeee/eeeeeeeeee22255555555 for what should or should not be described.
39


Reviews:
[{"review_id": "dk6RO3kWa5", "reviewer": "Reviewer_Wmgb", "summary": "This paper identifies the challenge of supervising long, unstructured \"thinking trajectories\" from advanced LLMs, a gap left by existing PRMs focused on shorter, structured reasoning. The paper propose a new Process Reward Model, TAP, which combines alignment, quality, and coherence scores to provide fine-grained rewards for these long trajectories. The goal is to use this reward model to improve the complex reasoning abilities of frontier models through reinforcement learning.", "questions": "The related work section would benefit from discussing more recent and relevant Reasoning-PRM evaluation paradigms [1,2].\n[1] R-PRM: Reasoning-Driven Process Reward Modeling\n[2] Inference-Time Scaling for Generalist Reward Modeling\n\nHyperparameter Analysis:  Given the monotonic improvement shown in Table 5, have you try an experiment for β=1 (pure reward from PRM)\n\nline318 should be Appendix F", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "# Strengths\nThe paper's main strength is its excellent problem formulation. It correctly identifies a critical and underexplored area: supervising the raw, lengthy \"thinking trajectories\" of frontier models, moving beyond the limitations of PRMs.\n\n# Weaknesses\n* Methodological Complexity vs. Ablation: The method's complexity, with its multiple reward components and encoders, is not justified by a proper ablation study. It is unclear if all parts are necessary, making the design feel over-engineered.\n* Counter-intuitive Training Data Size: The claim that the PRM is trained on only 1,000 samples (Appendix C.1) is counter-intuitive and lacks justification. \n* Questionable Experiment Setup:\n  + It is not explained how the Qwen2.5-PRM baseline, with its 4096-token context limit, evaluated long trajectories.\n  + Strange RL Results: The reinforcement learning results in Table 3 appear flawed. The performance of baselines, especially the rule-based reward, is significantly underestimated, eg. refer to [1].\n  + Suboptimal Best-of-N Evaluation: The Best-of-N evaluation uses a restrictive sampling temperature (T=0.3), limiting diversity (commonly using 0.6~1). Majority Voting seems to be underestimated. Also recomend to include the upper-bound baseline like Pass@N.", "was_revised": true, "final_justification": "Author's detailed response address some of my concern, i have changed the overall score to 4", "rebuttal": "We appreciate the reviewers' efforts and detailed feedback. We address each of your concerns below.\n\n---\n## **[W.1] Methodological Complexity vs. Ablation.**\nThank you for the insightful question. For better elaboration of our TAP design:\n\n**1. Motivation on TAP Method Design.** \n- In `Section 3`, we first thoroughly evaluate existing PRMs and identify several key shortcomings from our preliminary experiments. To address these issues, we designed TAP with targeted components, each aimed at mitigating specific limitations of prior PRMs.\n- As an illustration, we integrate (i) **LLM-based scoring** to provide fine-grained logical correctness at the step level; (ii) **Template generation** to ensure trajectory-level reward supervision; and (iii) **Trajectory decoding** for practical integration into RL and test-time inference.\n\n**2. Ablations on TAP Method Components.**\n\n**[Ablation Study 1]** TAP's step-level rewards choices: alignment, quality, and coherence ($r^{align}, r^{qual}, r^{coh}$).\n\n**Setups:** We follow training setups in `Section 5` and `Appendix C.1` for the ablation studies. For downstream evaluation, we choose offline data selection and report the SFT performance.\n|Method|MATH500|GPQA-Diamond|\n|-|-|-|\n|w/o $r^{align}$|82.4|43.4|\n|w/o $r^{qual}$|81.8|41.9|\n|w/o $r^{coh}$|84.4|46.0|\n|**TAP (ours)**|**84.8**|**47.5**|\n\nThe results show that:\n- Removing the quality and coherence scores during TAP training leads to noticeably suboptimal performance.\n- While the alignment score has relatively less impact compared to the other two-step reward supervision.\n- Motivated by these ablations, we plan to explore a lightweight version of TAP in the future that includes only selected step-level rewards tailored to specific application needs.\n\n**[Ablation Study 2]** TAP's step-level and trajectory-level rewards ($r^{step}, r^{final}$).\n\nWe follow the same experimental setup as above to train different TAP variants, excluding either step-level or trajectory-level rewards. We increase the training data for trajectory-level rewards from 1k to 10k for better training.\n|Method|MATH500|GPQA-Diamond|\n|-|-|-|\n|w/o $r^{final}$|82.8|43.9|\n|w/o $r^{step}$|83.4|44.9|\n|**TAP (ours)**|**84.8**|**47.5**|\n\nOur ablation study shows that removing either the trajectory-level ($r^{final}$) or step-level ($r^{step}$) reward degrades performance, with trajectory-level rewards having a greater impact. This underscores the necessity of jointly training TAP with both reward signals.\n\nWe will incorporate our new ablations into the revision to better demonstrate the contribution of each component in TAP.\n\n---\n## **[W.2] Training Data Size.**\n\nWe apologize for the confusion regarding TAP's training data size. As we stated in `Appendix C.1`:\n\n> Line 407: We train TAP on the OpenThoughts-114K collection of datasets.\n\n> Line 419: In addition to training with the template-guided trajectory-level reward, we first randomly sample 1000 problem-response samples from OpenThoughts-114k. \n\nTAP is trained on the **full 114K data**, while **the 1,000 samples are used for trajectory-level reward training**, since:  \n- Our experiments show that 1k samples suffice for robust trajectory-level training.  \n- Trajectory-level reward signals are more computationally expensive to obtain than step-level rewards.\n\nWe will refine our description of TAP's training settings to improve readability and clarity.\n\n---\n## **[W.3.1] Explanation on Qwen2.5-PRM Baseline.**\n\nThank you for the question. The baseline PRMs, Qwen2.5-Math-PRM-7B and Qwen2.5-Math-PRM-72B, indeed support a maximum context length of 4096. \n\nFor the evaluated trajectory-response data **over 4096 tokens**:\n- We first separate the data into thinking trajectories and final responses, respectively. \n- For each part, the PRM computed step-level rewards by segmenting steps (using \"\\n\\n\" separators).\n- Then we leverage the PRMs to assign step rewards for them separately and then average the entire step rewards as the final reward. \n- These step-level scores were then aggregated (via mean) together.\n- Note that if either the trajectory or response individually exceeded 4096 tokens, we truncated the excess tokens to fit within the PRM’s context window, which we found to be a necessary preprocessing step for compatibility with Qwen2.5-Math-PRM.\n\nWe also **highlight** that, unlike existing PRMs limited to scoring final responses, TAP extends supervision to long CoT data by incorporating both intermediate thinking trajectories and final responses, better aligning with frontier reasoning models.\n\nWe thank the reviewer and will incorporate the detailed settings above into our paper to improve clarity.\n\n---\n## **[W.3.2] RL Results Justification.**\n\nThank you for the careful review and the helpful reference [1]. In response to the TAP's performance on RL policy optimization:\n\n**1. Updated RL Performance on TAP:**  \nIn our early-stage experiments, due to submission time constraints, we settled the context length of the policy model (DeepSeek-R1-Distill-Qwen-7B) to a reduced value of 2800. We recognize that this could potentially hinder the generation.\n\nTo address this issue, we **reset the context length to 16,384** and conducted experiments in `Section 5.2`. We also included an extra downstream task, GPQA-Diamond. The new updated results are reported below:\n|Policy Model|Reward Signal Source|AIME24|AIME25|MATH500|GPQA-Diamond|\n|-|-|-|-|-|-|\n|**Qwen2.5-7B-Instruct**|Rule-based|12.9|11.1|73.6|32.7|\n||Qwen2.5-Math-PRM-7B|12.9|13.3|74.8|32.4|\n||**TAP**|**16.3**|**17.1**|**77.2**|**34.9**|\n|**DeepSeek-R1-Distill-Qwen-7B**|Rule-based|50.2|38.3|89.6|47.1|\n||Qwen2.5-Math-PRM-7B|51.2|40.8|92.8|49.1|\n||**TAP**|**54.6**|**44.2**|**94.8**|**51.6**|\n\n**2. TAP on Larger Policy Model:**\nWe also conducted additional experiments of TAP on larger policy models right after our submission. We present the updated experiments for the reviewer's reference:\n\n**Setups:** We follow the same experimental setups as `Section 5.2` to run GRPO.\n- **Larger Policy Model**: Qwen-2.5-14B-Instruct and Qwen-2.5-32B-Instruct.\n- **Larger PRM Baseline**:  Qwen2.5-Math-PRM-72B.\n\n|Policy Model|Reward Signal Source|AIME24|AIME25|MATH500|\n|-|-|-|-|-|\n|**Qwen2.5-14B-Instruct**| Rule-based |36.7|26.7|84.6|\n||Qwen2.5-Math-PRM-72B|36.7|33.3|85.2|\n||**TAP**|**43.3**|**40.0**|**86.8**|\n|**Qwen2.5-32B-Instruct**| Rule-based |53.3|46.7|92.4|\n||Qwen2.5-Math-PRM-72B|50.0|46.7|92.2|\n||**TAP**|**56.7**|**53.3**|**95.2**|\n\n- **Consistent Conclusion with Original Submission:** Our experiments across 7B, 14B, and 32B policy models affirm our original claim in `Section 5.2` regarding the effectiveness of TAP on RL.\n\nWe will incorporate the new experiments above and the related work [1] mentioned by the reviewer into our paper revision to improve clarity.\n\n---\n## **[W.3.3] Best-of-N Evaluation.**\n\nWe thank the reviewer for the helpful suggestion. In response:\n- We conduct new experiments below using a proper temperature setting of $T=0.7$.\n- We add the upper-bound pass@N and the majority@N metrics with $N = 8$.\n|Method|AIME24|MATH500|GPQA-Diamond|\n|-|-|-|-|\n|TAP Accuracy@8 (T=0.3)|46.7|91.2|55.6|\n|TAP Accuracy@8 (T=0.7)|46.7|91.4|56.6|\n|Majority@8 (T=0.7)|40.0|87.8|51.0|\n|TAP pass@8 (T=0.7)|60.0|93.6|59.1|\n\nFrom the results on the fine-tuned Qwen2.5-14B-Instruct policy model:  \n- Increasing $T$ from 0.3 to 0.7 yields improved performance for the policy model.\n- Consistent with the results in `Section 5.2`, TAP shows better improvements over the updated majority voting baseline and the new pass@8 metric.\n\nWe hope our new experiments provide a more comprehensive best-of-N evaluation of TAP and will incorporate results into our paper.\n\n---\n## **[Q.1] Additional Related Works.**\nWe thank the reviewer for the suggestion and agree that incorporating these recent works on reasoning-focused PRMs strengthens our related work section.\n\nAs a **brief discussion** on the relationship between TAP and each work:\n- **Relation to R-PRM [1]:** R-PRM improves step-level reward modeling via reasoning-driven analysis and preference optimization. TAP enables **offline data selection from a trajectory-aware perspective** and focuses more specifically on the **trajectory–response form of long CoT data**.\n- **Relation to DeepSeek-GRM [2]:** DeepSeek‑GRM uses a point‑wise GRM paradigm for inference‑time scaling. While TAP also supports best‑of‑N selection, we employ a different training and reward design, providing more dense process‑level feedback rather than only scalar final‐answer scores.\n\nWe thank the reviewer for the suggestions and will revise our related work section to **include a detailed discussion of these two works and cite them appropriately** in our revision.\n\n---\n## **[Q.2] Hyperparameter Analysis on $\\beta=1$.**\n\nThank you for the question. We indeed conduct experiments **with $\\beta=1$**, and the updated experiment results is shown below:\n\n|β|AIME24|AIME25|MATH500|\n|-|-|-|-|\n|0.1|10.0|6.7|73.6|\n|0.3|13.3|13.3|74.4|\n|0.5|13.3|6.7|75.2|\n|0.8|20.0|16.7|76.8|\n|1.0|20.0|16.7|76.6|\n\n- As the reviewer noted, increasing $\\beta$ from 0.1 to 0.8 yields a clear monotonic gain, demonstrating that **stronger emphasis on TAP’s process reward steadily boosts policy performance**.\n- At $\\beta=1.0$, we observe several tasks plateau. We attribute this to **the need for outcome‑level anchoring**:  By relying exclusively on the learned PRM reward at  $\\beta=1.0$, we drop the direct correctness signal that **anchors learning to the final-task objective.** Losing such direct correctness feedback might potentially introduce noise or bias.\n- Our ablations indicate that **$\\beta=0.8$ is the “sweet spot”**, offering the best trade‑off between TAP's rich trajectory‑level supervision and enough outcome reward to stabilize learning.\n\n## **References:**\n[1] R-PRM: Reasoning-Driven Process Reward Modeling.\\\n[2] Inference-Time Scaling for Generalist Reward Modeling."}, {"review_id": "UdqCD26sM5", "reviewer": "Reviewer_wGca", "summary": "This paper proposes a new process reward model (PRM) for improving reasoning, especially on hard math problems. They identify an important limitation of existing process reward models: that they focus on scoring only the final response and not on the intermediate reasoning trajectories. They set up an experiment where they show that existing process reward models cannot distinguish reliably between the intermediate reasoning trajectories from different models, even when the PRM can distinguish between final responses. \nThey then define TAP which is a trajectory aware process reward model. TAP combines a number of separate step level rewards and a whole trajectory reward together. The step level rewards include: alignment scores between trajectory steps and final answers, LLM-as-a-judge of step quality, and a coherence between steps score. The whole trajectory reward takes the trajectory and summarises it with a strong LLM and then provides this as a prompt to an LLM, and evaluates if the LLM gets the question correct with this summary. They show that TAP outperforms baseline PRMs for offline data selection of fine-tuning data, online reinforcement learning with GRPO for smaller reasoning models, and for test time n-best list reranking.", "questions": "In strengths and weaknesses", "limitations": "Limitations in Appendix which was in supplementary materials and I did not read it.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\t1) The paper is clear and well written. \n\t2) The problem is important: improving reasoning is currently one of the most interesting and impactful areas of AI research\n\t3) They have identified an important drawback of current PRMs. Although the final answer is clearly important, the trajectories are valuable and should be part of the reward. Good trajectories will lead to better answers even if they contain self corrections, redundant reasoning and branching logic. \n\t4) The have suggested a significantly novel approach to rewarding both trajectories and answers in one reward model. \n\t5) The have demonstrated that TAP outperforms baselines across 3 very different uses of a reward model: data selection, reinforcement learning, and n-best list reranking. \n\nWeaknesses:\n\t1) I am not completely convinced that TAP leads to significant improvements over the baseline reward models. I am slightly concerned that for reinforcement learning they used one of the smaller models (7B). RL is the most important application of a PRM and it would have been nice to show improvements over the 14B or the 72B baseline which were used in other experiments in this paper. The data selection experiment again was good, but why stop at 1000 examples? That being said - the experiments cover 3 good test sets and 3 diverse tasks and show improvements on all three so I do think TAP works - the question remaining for me is - how well? \n\t2) (not major) I don't think that the score distribution graphs tell us as much as is claimed. The score distributions between Gemini and Deepseek are the same for trajectories and different for final answers yes, but the authors should at least discuss if these  distribution differences are meaningful. Are the Gemini answers really so much worse? This is not made clear or justified \n\t3) (minor) It is odd that they report mostly the same table of results in Table 1 and then again in Table 2. Perhaps they should combine into one table and reorder discussion somewhat to reflect this.", "was_revised": true, "final_justification": "A good paper with interesting results", "rebuttal": "Thank you for your thorough comments and suggestions! Below, we address each of your concerns.\n\n---\n## **[W.1] Scalability on Larger Policy Models &  Data Selection Examples.**\nWe thank the reviewer for the thoughtful questions regarding our experiments. In response:  \n1. We first present **new RL experiments** incorporating TAP on **14B and 32B policy models**. \n2. We then **justify our intention of using 1,000 examples** in the data selection experiments.  \n3. We further **expand the training data size to 3k and 5k** to demonstrate the effectiveness and scalability of our method.\n\n**1. New RL Experiments with Larger Policy Models.**\\\nWe have conducted additional experiments on larger policy models right after the original submission. We present the updated experiments below.\n\n**Setups:** We follow the same experimental setups as `Section 5.2` to run GRPO. For new add-ups:\n- Larger Policy Model: We integrate TAP into larger policy models, including **Qwen-2.5-14B-Instruct**, **Qwen-2.5-32B-Instruct**.\n- Larger PRM Baseline: We compare TAP with a larger PRM baseline, including the **Qwen2.5-Math-PRM-72B.**\n\n**Results:**\n|Policy Model|Reward Signal Source|AIME24|AIME25|MATH500|\n|-|-|-|-|-|\n|**Qwen2.5-14B-Instruct**| Rule-based |36.7|26.7|84.6|\n||Qwen2.5-Math-PRM-72B|36.7|33.3|85.2|\n||**TAP (ours)**|**43.3**|**40.0**|**86.8**|\n|**Qwen2.5-32B-Instruct**| Rule-based |53.3|46.7|92.4|\n||Qwen2.5-Math-PRM-72B|50.0|46.7|92.2|\n||**TAP (ours)**|**56.7**|**53.3**|**95.2**|\n\n- From the table above, TAP outperforms baselines, including rule-based methods and Qwen2.5-Math-PRM-72B, **across both 14B and 32B policy models**. \n- This demonstrates that **TAP scales effectively to different sizes (7B, 14B, and 32B) policy models** and delivers greater performance gains compared to even larger existing PRMs.\n\nOur results above show a consistent improvement trend of TAP across scales. Due to rebuttal time and resource limits, we leave additional experiments on larger models (e.g., 72B-scale policy models) to future work.\n\n**2. Rationale for 1,000 Examples in Data Selection.**\\\nThe **primary reason** for using 1,000 examples in offline data selection is to **ensure a fair comparison with the strongest baseline (i.e., the 1k high-quality human-curated data in s1k[1])**.\n- s1k [1] shows that a small set of 1,000 human-expert-selected CoT examples can outperform much larger amounts of raw data.  \n- Given the high cost of human annotation, we explore **a new perspective** of applying PRMs for automated high-quality data selection. **Under the same 1,000-example experiment condition**, we evaluate whether TAP can efficiently outperform human-curated data in improving downstream SFT performance for smaller models (as shown in Table 2).\n\n\n**3. Scaling Training Data to 3k and 5k for Better Validation.**\\\nTo better address the reviewer’s question, we extend the data selection experiment in `Section 5.1` by applying TAP to select **3k and 5k samples** (from the same pool of 59k raw data traces) and fine-tune the generator (Qwen2.5-14B-Instruct) under the identical setup. \n\n- The results are reported below:\n    |SFT Data Size (TAP)|AIME24|MATH500|GPQA-Diamond|\n    |-|-|-|-|\n    |1k|40.0|84.8|47.5|\n    |3k|40.0|85.4|49.0|\n    |5k|43.3|86.2|49.5|\n- The results above show that **increasing the size of TAP-selected data generally improves downstream performance**, with a clear upward trend across all three benchmarks. \n\n\nWe thank the reviewer for these insightful questions and will incorporate the suggestions and new experimental results into our paper.\n\n---\n## **[W.2 (not major)] Deeper Analysis on Score Distribution.**\n\nWe appreciate the reviewer's observation and agree that clarifying the meaning of the score distributions is important. Below,\n \n1. We **first** present a data quality analysis comparing DeepSeek-R1 and Gemini Flash Thinking, demonstrating that DeepSeek-R1 indeed generates higher-quality thinking trajectories and thus merits higher process rewards. \n2. We **then** discuss why our observation, as shown in **Figure 2** of the paper, is meaningful and how it motivated us to design our PRM.\n\n\n**1. Analyses on the Quality of Generated Thinking Trajectories.**\\\nWe first follow [1] and utilize **1,000 thinking trajectories CoT data distilled separately from DeepSeek-R1 and Gemini Flash Thinking API**. We then perform supervised fine-tuning on a downstream model (**Qwen-2.5-14B**) using each of the two 1k datasets, while keeping all other settings identical. Finally, we evaluate the resulting models on **four downstream tasks** (same as **Table 1** of our paper) and report the results below:\n| SFT Data Source | AIME 24 | AIME 25 | MATH 500 | GPQA-Diamond|\n|-|-|-|-|-|\n|1k instances from **Gemini Flash Thinking**|33.3|33.3|78.8|41.4|\n|1k instances from **DeepSeek-R1**|43.3|40.0|80.2|44.9|\n\n- From the results, we observe a substantial performance gap between the two training data sources. Fine-tuning with distilled CoT data from **DeepSeek-R1** significantly improves downstream accuracy, **indicating that DeepSeek-R1 indeed produces higher-quality thinking trajectories**. \n- We further conducted a closer examination and case studies of Gemini’s “Flash Thinking” traces and found that they are noisier and more verbose than traces from DeepSeek-R1, often containing exploratory but ultimately irrelevant and incorrect reasoning steps.\n- Our observation is also consistent with findings from prior works on reasoning traces, such as [2, 3].\n\n\n**2. Discussion on Why Distribution Differences are Meaningful.**\n- From **Figure 2 (left)** of our paper, we observe that on thinking trajectories, existing PRMs' output score distributions between DeepSeek-R1 and Gemini Flash Thinking are **largely overlapped**. \n- Given that DeepSeek-R1 generally produces higher-quality thinking traces, as demonstrated in our analyses above, this highlights a **key limitation of current PRMs**: their inability to reliably distinguish and reward high-quality thinking trajectories.\n- Motivated by the observations above, we argue that a comprehensive PRM should not only assess final answers but also **provide robust rewards on thinking trajectories, enabling the selection of complete high-quality CoT traces for downstream applications**. As demonstrated in **Figure 4**, our TAP effectively differentiates the quality of thinking trajectories from different sources (i.e., via the score distribution differences), which in turn leads to improved downstream SFT performance.\n\n\nWe thank the reviewer for the insightful question and will incorporate the above explanations into our paper to improve clarity.\n\n---\n## **[W.3 (minor)] Tables Results Formatting.**\nWe thank the reviewer for the helpful suggestion! Our intention in reporting results in both **Table 1** and **Table 2** was to serve distinct purposes:\n\n- **Table 1 (Problem Diagnosis):** Reports the sub-optimal performance of existing PRMs to highlight their limitations and motivate our work.  \n- **Table 2 (Solution Efficacy):** Compares our proposed method against other PRMs and human-curated baselines to demonstrate its effectiveness.\n\nWe will incorporate the reviewer’s suggestion to reorder our discussion for improved clarity and readability of the paper.\n\n## **References:**\n[1] s1: Simple test-time scaling.\\\n[2] LIMO: Less is More for Reasoning.\\\n[3] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning."}, {"review_id": "8fsjM8PXVo", "reviewer": "Reviewer_N1UW", "summary": "This paper introduces TAP, a reward modeling framework designed to assess the quality of the reasoning trajectories. The authors argue that conventional PRMs, only trained on final outputs, are not suitable for evaluating the trajectories. TAP incorporates both step-level rewards and trajectory-level rewards to tackle this challenge. TAP is shown to outperform existing PRMs.", "questions": "* Why does scoring based on final responses perform similarly to PRMs trained on final outputs? Doesn’t this go against the main claim that trajectory-level supervision is essential?\n* Appendix D.1 You say TAP has low overhead—can you show reward computation time alone? Is it just that other parts dominate total cost?\n* Appendix D.2 You show that trajectory-level reward helps, but is step-level reward really needed? Especially r_coh—why assume adjacent steps must be similar? Can you show ablations using only step-level or only trajectory-level rewards?", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength\n* The paper tackles a timely and significant gap in PRM research, aligned with the trajectory-based outputs of modern frontier LLMs.\n* TAP consistently outperforms prior PRMs and even human-curated datasets.\n\nWeakness\n* Despite claims in Appendix D.1, TAP’s design—especially the LLM-based scoring, template generation, and trajectory decoding—seems over-engineered for many practical applications.\n* Insufficient ablation analysis. While the authors analyze α (the trajectory reward weight), there’s no clear ablation showing the effect of removing each component of the reward (e.g., removing r_coh or r_align entirely).\n* Although Table 1 is used to argue that conventional PRMs are miscalibrated on reasoning trajectories, the support is not entirely convincing—performance gaps are modest, and the diagnosis lacks granularity.\n* TAP can only be applied post-hoc after a full trajectory is generated, limiting its use for online reranking or early stopping.", "was_revised": true, "final_justification": "Thanks to the authors for their detailed explanations. While I still think that the method is a bit excessive (e.g., minimal influence of $r^{coh}$), my other concerns are resolved. Specifically, I was unsure how to explain Table 1, due to the modest gap between the PRM and the human-curated data. The authors' clarification in [Q.1] has provided a clear understanding of the table. Consequently, I have decided to maintain my current score.", "rebuttal": "Thank you for your thorough feedback! Below, we address each of the concerns.\n\n---\n## **[W.1] TAP's Method Design.**\n> \"TAP’s design—especially the LLM-based scoring, template generation, and trajectory decoding—seems over-engineered for many practical applications.\"\n\nThank you for the insightful question. While we acknowledge that TAP incorporates multiple design components and may appear \"over-engineered\" at first glance, we would like to justify our motivation and design choices for the following reasons:\n\n- In `Section 3`, we thoroughly evaluate existing PRMs and identify several key shortcomings from our preliminary experiments. To address these issues, we designed TAP with targeted components, each aimed at mitigating specific limitations of prior PRMs.\n- As an illustration, we integrate (i) **LLM-based scoring** to provide Fine-grained logical correctness at the step level; (ii) **Template generation** to ensure trajectory-level reward supervision; and (iii) **Trajectory decoding** for practical integration into RL and test-time inference.\n\nIn our following replies to the reviewer's questions, we will conduct new ablation studies to verify the effectiveness of each component in TAP’s design. \n\n---\n## **[W.2 & Q.3] Ablation Analyses on TAP.**\n> \"Is step-level reward really needed? Can you show ablations using only step-level or only trajectory-level rewards?\"\n\nThank you for asking about the effectiveness of each component in TAP. To better illustrate our design choices, we present several ablation studies below.\n\n**1. Ablations on TAP's step-level rewards choices: alignment, quality, and coherence ($r^{align}, r^{qual}, r^{coh}$).** \n\n**Setups:** We follow the same training setups in `Section 5` and `Appendix C.1` for the ablation studies. For downstream evaluation, we choose offline data selection and report the SFT performance.\n|Method|MATH500|GPQA-Diamond|\n|-|-|-|\n|w/o $r^{align}$|82.4|43.4|\n|w/o $r^{qual}$|81.8|41.9|\n|w/o $r^{coh}$|84.4|46.0|\n|**TAP (ours)**|**84.8**|**47.5**|\n\nThe results show that:\n- Removing the quality and coherence scores during TAP training leads to noticeably suboptimal performance.\n- while the alignment score has relatively less impact compared to the other two-step reward supervision.\n- Motivated by these ablations, we plan to explore a lightweight version of TAP in the future that includes only selected step-level rewards tailored to specific application needs.\n\n\n\n**2. Ablations on TAP's step-level and trajectory-level rewards ($r^{step}, r^{final}$).** \n\nWe follow the same experimental setup as above to train different TAP variants, each excluding either step-level or trajectory-level rewards.\n\n**Note:** We increase the training data for trajectory-level rewards from 1k to 10k for training stability.\n|Method|MATH500|GPQA-Diamond|\n|-|-|-|\n|w/o $r^{final}$|82.8|43.9|\n|w/o $r^{step}$|83.4|44.9|\n|**TAP (ours)**|**84.8**|**47.5**|\n\nOur ablation study here shows that removing either the trajectory-level ($r^{final}$) or step-level ($r^{step}$) reward degrades performance, with trajectory-level rewards having a greater impact. This underscores the necessity of jointly training TAP with both reward signals.\n\n\nWe thank the reviewer for the suggestion and will incorporate our new ablations into the revision to better demonstrate the contribution of each component in TAP.\n\n---\n## **[W.3] Analysis on the Shortcomings of Conventional PRMs.**\n>\"Although Table 1 is used to argue that conventional PRMs are miscalibrated on reasoning trajectories, the support is not entirely convincing—performance gaps are modest, and the diagnosis lacks granularity.\"\n\nWe thank the reviewer for raising this question. In response:\n\n**1. More Detailed Interpretation on Table 1.**\n- As part of our preliminary experiments, Table 1 serves as one component of our diagnosis of existing PRMs’ shortcomings on the trajectory-response data.\n- Although larger PRMs, such as Qwen2.5-Math-PRM-72B, show a relatively modest gap compared to the human-curated baseline, smaller PRMs like Skywork-PRM-7B and Math-Shepherd-PRM-7B exhibit significant gaps and even fail to outperform random selection.\n\n**2. Additional Analyses and Diagnosis on Existing PRMs.** \\\nWe kindly point out that our argument on conventional PRMs is miscalibrated on reasoning trajectories, which has also been supported by multiple other complementary analyses:\n- **Distributional Evidence:** In `Section 3`, we show that existing PRMs (e.g., Qwen2.5-Math-PRM-72B) exhibit significant overlap in score distributions between trajectories distilled from stronger (DeepSeek-R1) vs. weaker (Gemini) oracles. \n- **Granular Diagnostics in Appendix:** In `Appendix A.2`, we supplement Table 1 with additional analyses highlighting the **formatting mismatch between thinking trajectories and final responses.**.\n\nWe thank the reviewer for the question and will update our paper with a more detailed diagnostic analysis to improve both clarity and logical flow.\n\n---\n## **[W.4] Justifications on Online Reranking and Early Stopping.**\n> \"TAP ... limiting its use for online reranking or early stopping.\"\n\nWe appreciate the reviewer's observation. While TAP is indeed applied post-hoc on the trajectory-response data in our current design, we would like to clarify that:\n\n**1. Consistent with common PRM's objective.**\\\nIn `Section 2 (Preliminaries)`, we explicitly define the problem formulation of process reward modeling on trajectory-response data. **This objective is consistent with the formulations adopted by all other existing PRMs cited in our paper**.\n\n**2. Primary Objective of TAP on General Applications.**\\\nAs the main focus of our paper, TAP is designed as a **general-purpose PRM** to support high-quality data selection for offline training, reward modeling for online policy optimization in RL, and test-time scaling.\n\n**3. Extension on Online Reranking and Early Stopping.**\\\nWe kindly interpret the reviewer’s intention as suggesting that TAP should provide robust process rewards during **token-level generation**. While current PRMs like TAP are primarily designed for **sequence-level process supervision**, we agree that extending TAP to support token-level supervision, including online reranking and early stopping, **is a promising research direction**. We will incorporate such studies in future work.\n\n---\n## **[Q.1] Trajectory-level Supervision.**\n\nWe thank the reviewer for the insightful question. In Table 1, we observed that existing PRMs trained only on final responses perform better when scoring final responses rather than full trajectories. \n\nWe argue that this observation highlights the importance of trajectory-level supervision for the following reasons:\n\n>\"Doesn’t this go against the main claim that trajectory-level supervision is essential?\"\n\n**1. Importance of Trajectory-Response CoT Data.**\nAs current advanced large reasoning models mostly adopt the output format of thinking trajectories followed by final responses, and these full reasoning chains are often used as training data for downstream distillation, providing supervision only on the final response is insufficient and incomplete. \n\n>\"Why does scoring based on final responses perform similarly to PRMs trained on final outputs?\"\n\n**2. Suboptimal Performance without Trajectory-level Supervision.**\nIn Table 1, existing PRMs perform better when used to select only final responses compared to selecting entire trajectory-response data. However, regardless of selection based on entire trajectory-response or response-only, there remains a performance gap between PRM-selected data and the human-curated baseline.\n\nIn addition, as analyzed in `Appendix A.2`, thinking trajectories differ significantly from final responses, which makes it challenging for existing PRMs to provide reliable rewards on these intermediate trajectory steps.\n\n**3. Empirical Validation of TAP with the Trajectory-level Reward.**\nTo address the problem mentioned above, we design TAP with the template-guided trajectory-level reward to access that **whether the overall problem-solving strategy encoded in the model’s thinking trajectory reliably leads to correct solutions in final responses.**\n\nAs later shown in Table 2, TAP outperforms not only existing PRMs applied to model final responses (row: \"on model responses\") but also the strongest human-curated baseline.\n\n---\n## **[Q.2] Computation Overhead of TAP.**\n> \"Appendix D.1 You say TAP has low overhead—can you show reward computation time alone? Is it just that other parts dominate the total cost?\"\n\nWe thank the reviewer for asking about the computation overhead of our method. In response:  \n\n**1. Time Overhead Report on Reward Computation**\n\nWe provide a time overhead report, including both the reward computation and the total time overhead.\n\n**Setup:** We run TAP-enhanced GRPO under the GPU settings in our paper and report the average time per step over 500 training steps. The results are shown below:\n|Base Model|Method|Avg. Reward Computation Time/Step (s)| Avg. Total time/Step (s)| Percentage|\n|-|-|-|-|-|\n|Qwen2.5-7B-Instruct|TAP|7.42|183.75| 4.0%|\n|DeepSeek-R1-Distill-Qwen-7B| TAP|11.45|484.71|2.4%|\n\nThe table shows that  TAP’s reward computation takes **< 5%** of the overall training time per step. This demonstrates that **incorporating TAP into RL policy optimiation adds marginal overhead to the overall runtime**.\n\n**2. Detailed Explanation of TAP’s High Efficiency**  \nIn `Section D.1`, we attribute TAP’s efficiency to two key factors:  \n\n- **Data Efficiency:** TAP reduces training data size by offline selecting high-quality examples (e.g., 1k TAP-selected data outperforms both raw 59k data and human-curated 1k data). This significantly lowers both human annotation costs and overall SFT training time.  \n- **Model Efficiency:** TAP uses far fewer parameters while achieving superior reward modeling performance compared to 7B and even 72B PRMs, thereby reducing deployment and inference costs."}, {"review_id": "OwuAPM36Fg", "reviewer": "Reviewer_vTjq", "summary": "The paper proposes TAP (Trajectory-Aware Process Reward Model), a new framework for evaluating and supervising intermediate reasoning steps in LLM. Unlike existing PRMs focused on final outputs, TAP assigns fine-grained rewards to entire reasoning trajectories using step-level signals and a trajectory-level check via a proxy policy model. The proposed model could be used in both offline and online reward supervision. It achieves over 22% accuracy gains on hard reasoning benchmarks like AIME and GPQA compared to prior reward models.", "questions": "Same as the Weaknesses section:\n\n\n1. The paper uses GPT-4o as the expert model. Could the authors provide an estimate of the associated cost, and discuss whether it could be replaced with an open-source model? Would that affect performance?\n2. DeepSeek-R1 reported that PRMs did not work well in their setting, whereas the proposed method appears effective. Could the authors provide further analysis on this discrepancy?\n3. I am also curious whether the performance gain from this method, compared to rule-based judges, increases or decreases as model size scales up.\n4. I suggest the authors add experiments comparing their method to step-wise LLM-as-judge baselines.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n1. The paper is well-structured and the motivation is clear.\n2. It advances PRM research from post-hoc answer checking to process-aware supervision; it demonstrates that well-scored trajectories can outperform human curation and enhance reinforcement learning.\n3. The proposed method achieves strong performance in both offline and online reward supervision settings.\n\n## Weaknesses\n\n1. The paper uses GPT-4o as the expert model. Could the authors provide an estimate of the associated cost, and discuss whether it could be replaced with an open-source model? Would that affect performance?\n2. DeepSeek-R1 reported that PRMs did not work well in their setting, whereas the proposed method appears effective. Could the authors provide further analysis on this discrepancy?\n3. I am also curious whether the performance gain from this method, compared to rule-based judges, increases or decreases as model size scales up.\n4. I suggest the authors add experiments comparing their method to step-wise LLM-as-judge baselines.", "was_revised": true, "final_justification": "In this paper, the authors propose TAP (Trajectory-Aware Process Reward Model), a new framework for evaluating and supervising intermediate reasoning steps in LLM. Unlike existing PRMs focused on final outputs, TAP assigns fine-grained rewards to entire reasoning trajectories using step-level signals and a trajectory-level check via a proxy policy model. \n\nDuring the rebuttal, the authors have shown their method's cost is cheap and also could use the open-source model to replace. Furthermore, they conduct some experiments on the 14B and 32B model, which shows the effectiveness of the proposed TAP. Based on these strengths, I decide to raise my score.", "rebuttal": "We sincerely thank the reviewer for your constructive suggestions! We address each of your concerns below.\n\n---\n## **[Q.1] Cost analysis of GPT-4o & Open-source Alternatives.**\n\nThank you for the insightful question regarding expert model cost. **Our trajectory-level reward method is model-agnostic and can be seamlessly applied with any expert model, including open-source alternatives.** As a reply:  \n\n1. We first provide the associated cost of using GPT-4o as the expert model.  \n2. We then demonstrate the effectiveness of our method when adopting two additional open-source models as the expert.  \n\n**1. GPT-4o Cost.**\n||Token Count|Price (per 1 mil tokens)|Cost|\n|-|-|-|-|\n|Input|3,589,427|$2.5|$8.97|\n|Output|343,864|$10|$3.44|\n|Total|3,933,291|-|**$12.41**|\n\nThe table above reports the primary cost of using GPT-4o as the expert model. Our total cost on the expert model is **less than $15**. We highlight that our \"template-guided\" trajectory reward method is **cost-efficient** for the following reasons:\n- In our TAP model training, we utilize only a relatively small dataset (**1,000 samples**) to prompt GPT-4o for template generation, which is sufficient to demonstrate effectiveness in downstream reward modeling.  \n- The expert model GPT-4o is used solely to **generate concise, structured templates** (typically under 500 tokens), significantly reducing token costs compared to directly leveraging the expert model to produce full-length CoT reasoning traces and solutions.\n\n**2. Effectiveness of Open-source Alternatives.**  \nRegarding performance impact, as noted in Lines 190–197, the policy model uses templates generated by the expert model to produce $N$ responses, which are then used to compute trajectory rewards. \n\nConsequently, the choice of expert model will influence TAP training quality: **a more powerful and reliable expert model provides more robust trajectory reward signals during PRM training, leading to stronger performance.**\n\nTo better address the reviewer's question, **we newly adopt DeepSeek-R1 and QwQ-32B as the common open-source expert models for the 1,000 template generation**. We follow the same training setups in our paper. We report the new comparison results on offline data selection and SFT:\n|Expert Model|MATH500|GPQA-Diamond|\n|-|-|-|\n|GPT-4o| 84.8|47.5|\n|QwQ-32B|84.2| 46.0|\n|DeepSeek-R1|86.4| 49.5|\n\n- From the table, we observe that TAP remains effective when using open-source expert models. Notably, as DeepSeek-R1 is more powerful on reasoning tasks, adopting it as the expert model surpasses the original GPT-4o with higher quality trajectory rewards. \n- Thus, **TAP’s effectiveness is not tied to a single proprietary expert model, and it can generalize well to other high-quality open-source alternatives**.\n\n---\n## **[Q.2] Justification on DeepSeek-R1 Findings.**\nWe thank the reviewer for asking this insightful question. We justify from three key perspectives:\n\n**1. More advanced PRM Design:**\\\nIn DeepSeek-R1 [1], the authors noted that several early-stage PRMs (e.g., [2,3]) are ineffective for complex reasoning tasks. However, our PRM is designed and trained fundamentally different from the following:\n- **Effectiveness on Long CoT Data.** The adopted PRMs in [1] were primarily trained and designed for simpler tasks, where the PRM training data typically contained short outputs with minimal step-level information. In contrast, **DeepSeek-R1** is trained to generate much longer and more complex CoT outputs. As a result, these earlier PRMs often failed when required to provide step-level reward supervision for such long CoT outputs.\n\n    In contrast, **TAP is specifically designed and trained on advanced reasoning tasks with much longer and more complex CoT data (OpenThoughts-114K)**, which includes not only models’ final responses but also rich intermediate thinking trajectories.\n    \n- **Overlook of Intermediate Thinking Trajectories.** In our preliminary studies in `Section 3`, we found that models’ intermediate thinking trajectories are integral for effective process supervision. However, existing PRMs largely overlook this rich intermediate information and fail to provide reliable supervision on these trajectories. \n\n    In contrast, **TAP is the first trajectory-aware PRM to explicitly incorporate models’ intermediate thinking trajectories**, enabling it to provide robust and fine-grained process rewards throughout the reasoning process.\n\n- **Process Reward Design.** The earlier PRMs in [1] often failed to provide fine-grained step-level rewards, which led to issues such as reward hacking.\n\n    In contrast, **TAP** provides both step-level and trajectory-level reward supervision, capturing both fine-grained and holistic reasoning quality in long CoT data. This dual-level design offers a more precise and reliable optimization objective for complex reasoning tasks.\n\n\n**2. Broader applicability on TAP:**\\\nWe also want to highlight that the core idea and contribution of TAP is to **support more general and comprehensive reward modeling scenarios**. Beyond its application in **RL policy optimization** as mentioned in [1], TAP also provides process reward supervision for tasks such as **offline data selection** and **inference-time scaling**.\n\n\n\n**3. Evolution of the process reward-modeling:**\nLastly, we want to briefly mention that several recent works [4,5,6] have shown that richer process-reward signals lead to improved performance. This suggests that more advanced PRMs can potentially overcome the limitations of earlier, outdated PRMs used in prior studies.\n\n**References:**\\\n[1] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. \\\n[2] Solving math word problems with process- and outcome-based feedback.\\\n[3] Math-shepherd: Verify and reinforce LLMs step-by-step without human annotations. \\\n[4] Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning. \\\n[5] Inference-Time Scaling for Generalist Reward Modeling. \\\n[6] Process Reinforcement through Implicit Rewards.\n\n---\n## **[Q.3] Scaling Trend of Performance Gain vs. Model Size.**\n\nThank you for the questions regarding the scalability of our method. We have conducted **additional experiments on larger policy models** right after our original submission. \n\nBelow, we show that TAP is scalable to larger policy models with improved reward modeling:\n\n- **Experiment Setups:** We follow the same experimental setup as in `Section 5.2` to run GRPO. As an extension, we integrate TAP into larger policy models, including **Qwen-2.5-14B-Instruct** and **Qwen-2.5-32B-Instruct**.\n- **New Results:**\n    |Policy Model|Reward Signal Source|AIME24|AIME25|MATH500|\n    |-|-|-|-|-|\n    |**Qwen2.5-14B-Instruct**| Rule-based |36.7|26.7|84.6|\n    ||**TAP (ours)**|**43.3**|**40.0**|**86.8**|\n    |**Qwen2.5-32B-Instruct**| Rule-based |53.3|46.7|92.4|\n    ||**TAP (ours)**|**56.7**|**53.3**|**95.2**|\n\n\n    From the table above, TAP outperforms the rule-based judges baseline across both 14B and 32B policy models. \n    \n    Our new experiment demonstrates that **TAP scales effectively to different sizes (7B, 14B, and 32B) policy models** and **delivers consistent performance gains**.\n    \n- We thank the reviewer for their question and will ensure to incorporate our new experiment results into our paper for a better empirical demonstration of our method.\n\n---\n## **[Q.4] Additional Comparison to Step-wise LLM-as-judge.**\nWe thank the reviewer for the helpful suggestion. In response:\n- **Existing Step‑wise Baselines:** We first want to highlight that several PRM baselines compared in our paper, such as Qwen2.5-Math-PRM-72B [1], already employ step‑wise LLM‑as‑judge signals during training.\n- **TAP's Novelty:** Unlike PRM baselines directly utilizing LLM-as-judge to obtain reward signals, **TAP leverages an expert LLM as the verifier to enforce trajectory‑level coherence via template alignment.**\n- **Additional Experiments:** We conduct additional experiments to **compare TAP with a direct step-wise LLM-as-judge baseline** on two test time scaling tasks.\n    - **Experimental Setup:** Following prior works [2,3], we use GPT-4o as the judge LLM to score and rerank $N = 16$ candidate responses using pure step-wise LLM-as-judge aggregation (i.e., averaging GPT-4o judged quality scores across all steps). Additional setups on best-of-N TTS are the same as `Section 5.2` in our paper.\n    - **Results:**\n        |Method|GPQA-Diamond@16|MATH500@16|\n        |-|-|-|\n        |LLM-as-Judge|51.5|90.4|\n        |**TAP**|**56.6**|**92.2**|\n        \n        From the results, TAP consistently outperforms pure step‑wise LLM-as-Judge baseline on both GPQA-Diamond and MATH500.\n\nDue to the rebuttal time constraints, we present a simplified baseline here. We will incorporate these results into our manuscript and evaluate more advanced LLM‑as‑judge variants in future work.\n\n\n**References:**\\\n[1] The Lessons of Developing Process Reward Models in Mathematical Reasoning. \\\n[2] Let's Verify Step by Step. \\\n[3] Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena."}]

Author rebuttal:

Meta-review: The paper introduces TAP (Trajectory-Aware PRM), a novel reward modeling framework designed to evaluate multi-step reasoning trajectories of large language models. Unlike existing PRMs focusing on final responses, TAP incorporates both step-level (alignment, quality, coherence) and trajectory-level supervision to provide fine-grained feedback. The authors demonstrate TAP’s effectiveness in three applications: offline data selection for distillation, online reinforcement learning, and test-time reranking via best-of-N strategies. Experiments on AIME and GPQA benchmarks show a 22.7% performance improvement over baseline PRMs, validating its utility for supervising complex reasoning in frontier models like OpenAI-o1 and DeepSeek-R1. 

Reviewers acknowledge the paper’s timely and impactful contribution to PRM research. The strong problem formulation addresses an underexplored gap in supervising lengthy, unstructured reasoning trajectories from frontier LLMs. The framework’s versatility across offline/online settings and its consistent outperformance of baselines, including human-curated data, are highlighted as key strengths. The significance of advancing PRM research from final-output focus to process-aware supervision is also emphasized.

On the other hand, reviewers point out several weaknesses. Critiques include insufficient ablation studies to justify the complexity of TAP’s components, and limited scalability analysis. The RL experiments’ focus on smaller models and the restrictive sampling temperature in best-of-N evaluation raise questions about broader applicability (Reviewer wGca, Wmgb). Some reviewers request cost analysis for GPT-4o dependency and comparisons to step-wise LLM-as-judge baselines. 

In general, the paper tackles a timely and important problem with a novel framework that shows promising results across multiple applications. While it has limitations needing revision, its strengths in problem relevance, methodological innovation, and empirical performance outweigh the weaknesses. Thus, the paper is recommended for acceptance with minor revisions.

During the rebuttal, the authors demonstrated that their method is cost-effective and can be replaced with an open-source model. Furthermore, they conduct some experiments on larger policy models, and ablations on TAP method components, which shows the effectiveness of the proposed TAP. The effective rebuttal leads to the increase of average rating.
Decision: Accept (poster)