Title: ReasonFlux-PRM: Trajectory-Aware PRMs for Long Chain-of-Thought Reasoning in LLMs
Venue: NeurIPS 2025
Accepted: yes
Keywords: Process Reward Models

Inference Scaling

LLMs

Reasoning Trajectories Selection
Abstract: Process Reward Models (PRMs) have recently emerged as a powerful framework for supervising intermediate reasoning steps in large language models (LLMs). Previous PRMs are primarily trained on model final output responses and struggle to evaluate intermediate thinking trajectories robustly, especially in the emerging setting of trajectory–response outputs generated by frontier reasoning models like Deepseek-R1.  In this work, we introduce ReasonFlux-PRM, a novel trajectory-aware PRM explicitly designed to evaluate the trajectory-response type of reasoning traces. ReasonFlux-PRM incorporates both step-level and trajectory-level supervision, enabling fine-grained reward assignment aligned with structured chain-of-thought data. We adapt ReasonFlux-PRM to support reward supervision under both offline and online settings, including (i) selecting high-quality model distillation data for downstream supervised fine-tuning of smaller models, (ii) providing dense process-level rewards for policy optimization during reinforcement learning, and (iii) enabling reward-guided Best-of-N test-time scaling.  Empirical results on challenging downstream benchmarks such as AIME, MATH500, and GPQA-Diamond demonstrate that ReasonFlux-PRM-7B selects higher quality data than strong PRMs (e.g., Qwen2.5-Math-PRM-72B) and human-curated baselines. Furthermore, ReasonFlux-PRM-7B yields consistent performance improvements, achieving average gains of 12.1\% in supervised fine-tuning, 4.5\% in reinforcement learning, and 6.3\% in test-time scaling. We also release an efficient ReasonFlux-PRM-1.5B for resource-constrained applications and edge deployment. Our code and models are released at https://github.com/Gen-Verse/ReasonFlux.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes TAP (Trajectory-Aware Process Reward Model), a new framework for evaluating and supervising intermediate reasoning steps in LLM. Unlike existing PRMs focused on final outputs, TAP assigns fine-grained rewards to entire reasoning trajectories using step-level signals and a trajectory-level check via a proxy policy model. The proposed model could be used in both offline and online reward supervision. It achieves over 22% accuracy gains on hard reasoning benchmarks like AIME and GPQA compared to prior reward models.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. The paper is well-structured and the motivation is clear.\n2. It advances PRM research from post-hoc answer checking to process-aware supervision; it demonstrates that well-scored trajectories can outperform human curation and enhance reinforcement learning.\n3. The proposed method achieves strong performance in both offline and online reward supervision settings.\n\n## Weaknesses\n\n1. The paper uses GPT-4o as the expert model. Could the authors provide an estimate of the associated cost, and discuss whether it could be replaced with an open-source model? Would that affect performance?\n2. DeepSeek-R1 reported that PRMs did not work well in their setting, whereas the proposed method appears effective. Could the authors provide further analysis on this discrepancy?\n3. I am also curious whether the performance gain from this method, compared to rule-based judges, increases or decreases as model size scales up.\n4. I suggest the authors add experiments comparing their method to step-wise LLM-as-judge baselines.\n\nQUESTIONS:\nSame as the Weaknesses section:\n\n\n1. The paper uses GPT-4o as the expert model. Could the authors provide an estimate of the associated cost, and discuss whether it could be replaced with an open-source model? Would that affect performance?\n2. DeepSeek-R1 reported that PRMs did not work well in their setting, whereas the proposed method appears effective. Could the authors provide further analysis on this discrepancy?\n3. I am also curious whether the performance gain from this method, compared to rule-based judges, increases or decreases as model size scales up.\n4. I suggest the authors add experiments comparing their method to step-wise LLM-as-judge baselines.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces TAP, a reward modeling framework designed to assess the quality of the reasoning trajectories. The authors argue that conventional PRMs, only trained on final outputs, are not suitable for evaluating the trajectories. TAP incorporates both step-level rewards and trajectory-level rewards to tackle this challenge. TAP is shown to outperform existing PRMs.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n* The paper tackles a timely and significant gap in PRM research, aligned with the trajectory-based outputs of modern frontier LLMs.\n* TAP consistently outperforms prior PRMs and even human-curated datasets.\n\nWeakness\n* Despite claims in Appendix D.1, TAP’s design—especially the LLM-based scoring, template generation, and trajectory decoding—seems over-engineered for many practical applications.\n* Insufficient ablation analysis. While the authors analyze α (the trajectory reward weight), there’s no clear ablation showing the effect of removing each component of the reward (e.g., removing r_coh or r_align entirely).\n* Although Table 1 is used to argue that conventional PRMs are miscalibrated on reasoning trajectories, the support is not entirely convincing—performance gaps are modest, and the diagnosis lacks granularity.\n* TAP can only be applied post-hoc after a full trajectory is generated, limiting its use for online reranking or early stopping.\n\nQUESTIONS:\n* Why does scoring based on final responses perform similarly to PRMs trained on final outputs? Doesn’t this go against the main claim that trajectory-level supervision is essential?\n* Appendix D.1 You say TAP has low overhead—can you show reward computation time alone? Is it just that other parts dominate total cost?\n* Appendix D.2 You show that trajectory-level reward helps, but is step-level reward really needed? Especially r_coh—why assume adjacent steps must be similar? Can you show ablations using only step-level or only trajectory-level rewards?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new process reward model (PRM) for improving reasoning, especially on hard math problems. They identify an important limitation of existing process reward models: that they focus on scoring only the final response and not on the intermediate reasoning trajectories. They set up an experiment where they show that existing process reward models cannot distinguish reliably between the intermediate reasoning trajectories from different models, even when the PRM can distinguish between final responses. \nThey then define TAP which is a trajectory aware process reward model. TAP combines a number of separate step level rewards and a whole trajectory reward together. The step level rewards include: alignment scores between trajectory steps and final answers, LLM-as-a-judge of step quality, and a coherence between steps score. The whole trajectory reward takes the trajectory and summarises it with a strong LLM and then provides this as a prompt to an LLM, and evaluates if the LLM gets the question correct with this summary. They show that TAP outperforms baseline PRMs for offline data selection of fine-tuning data, online reinforcement learning with GRPO for smaller reasoning models, and for test time n-best list reranking.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\t1) The paper is clear and well written. \n\t2) The problem is important: improving reasoning is currently one of the most interesting and impactful areas of AI research\n\t3) They have identified an important drawback of current PRMs. Although the final answer is clearly important, the trajectories are valuable and should be part of the reward. Good trajectories will lead to better answers even if they contain self corrections, redundant reasoning and branching logic. \n\t4) The have suggested a significantly novel approach to rewarding both trajectories and answers in one reward model. \n\t5) The have demonstrated that TAP outperforms baselines across 3 very different uses of a reward model: data selection, reinforcement learning, and n-best list reranking. \n\nWeaknesses:\n\t1) I am not completely convinced that TAP leads to significant improvements over the baseline reward models. I am slightly concerned that for reinforcement learning they used one of the smaller models (7B). RL is the most important application of a PRM and it would have been nice to show improvements over the 14B or the 72B baseline which were used in other experiments in this paper. The data selection experiment again was good, but why stop at 1000 examples? That being said - the experiments cover 3 good test sets and 3 diverse tasks and show improvements on all three so I do think TAP works - the question remaining for me is - how well? \n\t2) (not major) I don't think that the score distribution graphs tell us as much as is claimed. The score distributions between Gemini and Deepseek are the same for trajectories and different for final answers yes, but the authors should at least discuss if these  distribution differences are meaningful. Are the Gemini answers really so much worse? This is not made clear or justified \n\t3) (minor) It is odd that they report mostly the same table of results in Table 1 and then again in Table 2. Perhaps they should combine into one table and reorder discussion somewhat to reflect this.\n\nQUESTIONS:\nIn strengths and weaknesses\n\nLIMITATIONS:\nLimitations in Appendix which was in supplementary materials and I did not read it.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper identifies the challenge of supervising long, unstructured \"thinking trajectories\" from advanced LLMs, a gap left by existing PRMs focused on shorter, structured reasoning. The paper propose a new Process Reward Model, TAP, which combines alignment, quality, and coherence scores to provide fine-grained rewards for these long trajectories. The goal is to use this reward model to improve the complex reasoning abilities of frontier models through reinforcement learning.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\nThe paper's main strength is its excellent problem formulation. It correctly identifies a critical and underexplored area: supervising the raw, lengthy \"thinking trajectories\" of frontier models, moving beyond the limitations of PRMs.\n\n# Weaknesses\n* Methodological Complexity vs. Ablation: The method's complexity, with its multiple reward components and encoders, is not justified by a proper ablation study. It is unclear if all parts are necessary, making the design feel over-engineered.\n* Counter-intuitive Training Data Size: The claim that the PRM is trained on only 1,000 samples (Appendix C.1) is counter-intuitive and lacks justification. \n* Questionable Experiment Setup:\n  + It is not explained how the Qwen2.5-PRM baseline, with its 4096-token context limit, evaluated long trajectories.\n  + Strange RL Results: The reinforcement learning results in Table 3 appear flawed. The performance of baselines, especially the rule-based reward, is significantly underestimated, eg. refer to [1].\n  + Suboptimal Best-of-N Evaluation: The Best-of-N evaluation uses a restrictive sampling temperature (T=0.3), limiting diversity (commonly using 0.6~1). Majority Voting seems to be underestimated. Also recomend to include the upper-bound baseline like Pass@N.\n\nQUESTIONS:\nThe related work section would benefit from discussing more recent and relevant Reasoning-PRM evaluation paradigms [1,2].\n[1] R-PRM: Reasoning-Driven Process Reward Modeling\n[2] Inference-Time Scaling for Generalist Reward Modeling\n\nHyperparameter Analysis:  Given the monotonic improvement shown in Table 5, have you try an experiment for β=1 (pure reward from PRM)\n\nline318 should be Appendix F\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-05"}