Title: Process vs. Outcome Reward: Which is Better for Agentic RAG Reinforcement Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Process-Supervision

Agentic RAG
Abstract: Retrieval-augmented generation (RAG) enhances large language models (LLMs) by integrating external knowledge, yet traditional RAG systems struggle with static workflows and limited adaptability for complex, multistep reasoning tasks. Agentic RAG systems, such as DeepResearch, address these issues through dynamic retrieval, iterative context refinement, and adaptive workflows. However, recent methods like Search-R1, which rely on outcome-based reinforcement learning, face challenges such as low exploration efficiency, gradient conflict, and sparse reward signals. To tackle these limitations, we introduce ReasonRAG, a novel method that leverages RAG-ProGUIDE—a high-quality dataset providing fine-grained, process-level rewards for query generation, evidence extraction, and answer generation. By employing process-supervised reinforcement learning, ReasonRAG enhances LLMs’ autonomous capabilities in search, query generation, evidence extraction, and answer synthesis. Experimental results show that ReasonRAG, utilizing RAG-ProGUIDE, outperforms existing approaches like Search-R1 and traditional RAG systems, achieving superior performance on five benchmark datasets with only 5k training instances—significantly fewer than the 90k required by Search-R1. Our code is available at https://github.com/Applied-Machine-Learning-Lab/ReasonRAG.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates reward design for agentic RAG. It introduces ReasonRAG, which uses Monte-Carlo Tree Search (MCTS) plus a “Shortest-Path Reward Estimation” (SPRE) signal to build a 5k-query process-level dataset (RAG-ProGuide). A policy trained with DPO on these preference pairs outperforms the outcome-level baseline Search-R1 on five QA benchmarks while using less data.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. The paper addresses an important problem in agentic retrieval-augmented generation—how to provide effective reward beyond sparse final answer. \n2. The proposed model outperforms a 90k-sample outcome-based baseline across five QA benchmarks with only 5k sample.\n\nWeakness:\n1. Component-wise contribution unclear: The proposed system integrates several components—MCTS, SPRE, and DPO—but their relative importance remains unclear. An ablation study removing or replacing each module (e.g., substituting MCTS with naive sampling) would help clarify which elements are essential to performance.\n\n2. Penalty coefficient \\alpha is under-analyzed: Equation 1 includes a tunable penalty weight \\alpha to encourage shorter paths. However, the paper does not analyze its effect on learning dynamics or final performance. Could the authors provide experiments or sensitivity analysis on different values of \\alpha?\n\n3. Process vs. outcome comparison not controlled: Search-R1 differs in model scale and data quantity. To isolate reward type effects, would it be possible to train a DPO model on outcome-based preferences from the same 5k questions?\n\nQUESTIONS:\n1. Effect of data selection on sample efficiency: The method achieves strong results using only 5k queries, compared to 90k used in Search-R1. This raises the question: to what extent does the performance stem from careful data selection rather than reward formulation? How were these 5k examples chosen, and could biased or curated selection influence the outcome?\n\n2. Does SPRE provide sufficiently discriminative reward signals for complex queries where multiple reasoning paths may be valid? Are there failure cases where SPRE assigns similar scores to semantically different strategies?\n\n3. How reliable are the preference labels used for DPO? Were the \"better\" and \"worse\" trajectories chosen purely via SPRE scores?\n\nLIMITATIONS:\nSee weakness.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper examines the impact of ORM and PRM on RAG FT by creating an MCTS‑sampled dataset, RAG‑ProGuide, and training the inference model, ReasonRAG, via DPO FT. The proposed workflow demonstrates innovation in exploration diversity, iteration‑count distribution, and performance on downstream datasets. Furthermore, the authors compare ReasonRAG’s downstream metrics after ORM versus PRM FT across multiple datasets, providing preliminary evidence that PRM outperforms ORM and offering valuable insights for RAG researche\n\nSTRENGTHS AND WEAKNESSES:\n> Strengths:\n\n1.The paper’s premise—comparing the impact of PRM versus ORM on agentic RAG FT is both novel and of practical significance to the RAG research community.\n\n2.The experimental evaluation is thorough, leveraging a diverse set of datasets and RAG frameworks to assess exploration diversity, iteration‑count distributions, and test‑set EM performance; these comprehensive results effectively demonstrate the advantages of ReasonRAG.\n\n> Weaknesses: \n\n1.From a workflow perspective, an MCTS‑based RAG system has already been proposed in AirRAG (https://arxiv.org/pdf/2501.10053). The main contribution of this paper—using a UCB‑based stepwise selection of positive and negative examples for DPO FT offers limited novelty.\n\n2.In terms of training loss, although the paper’s focus is on comparing PRM and ORM for agentic RAG FT, it applies identical DPO weighting to both procedural decision correctness and final‑output correctness. The manuscript does not explore how varying these weightings affects training outcomes, nor does it investigate which aspect has a greater impact on RAG system performance.\n\n3.Figure 3 contains incorrect annotations, and its caption does not clearly explain the figure’s intended meaning.\n\n4.The paper does not specify which large language models were used at each baseline; if different LLMs were employed, the reliability of the results would be compromised.\n\nQUESTIONS:\n1. Given that the primary focus of your paper is the comparative impact of PRM versus ORM on LLM FT, it is advisable to enrich the PRM experiments, for example, by assigning distinct weights to procedural correctness and outcome correctness and systematically evaluating how these weightings affect downstream performance metrics.\n\n2. The set of baselines should concentrate on RAG frameworks that have undergone comparable ORM or PRM FT. Incorporating additional relevant baselines, such as RAG‑Gym, would further bolster the robustness and persuasiveness of your evaluation.\n\n3. The axis labels and their semantic descriptions in certain figures (e.g., Figure 3) lack sufficient clarity. A careful revision of these annotations is necessary to ensure that the figure’s intent and interpretation are unambiguous.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 1, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an algorithm for computing process-supervised rewards in agentic retrieval-augmented generation (RAG) tasks. The proposed reward formulation is integrated with reinforcement learning to train a RAG agent. Experiments on several public benchmarks demonstrate the effectiveness of the approach.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThe paper is clearly written, and the idea of using Monte Carlo Tree Search (MCTS) to compute process-supervised rewards is interesting. The experimental results are promising and suggest the approach is effective.\n\nWeaknesses:\nWhile the results are strong, the use of MCTS introduces additional sampling overhead. It remains unclear whether the performance gains stem primarily from the increased number of generated samples or from the specific reward computation method itself. A more controlled analysis would help clarify this distinction.\n\nMoreover, the datasets used in their experiments involve no more than five steps per episode. It remains unclear how the computational cost and accuracy would scale as the number of steps increases.\n\nQUESTIONS:\n1. What are the computational costs of the different algorithms listed in Tables 2 and 3? A comparison in terms of runtime or sample efficiency would help contextualize the performance gains.\n\n2. How do computational cost and accuracy vary with the number of steps in a sample? Since the current datasets contain at most five steps, it would be valuable to understand how the method scales with longer reasoning trajectories.\n\n3. Have you tried using different base models? It would be helpful to know whether the observed improvements are consistent across models of varying sizes or architectures.\n\nLIMITATIONS:\nWhile the results are strong, the use of MCTS introduces additional sampling overhead. It remains unclear whether the performance gains stem primarily from the increased number of generated samples or from the specific reward computation method itself. A more controlled analysis would help clarify this distinction.\n\nMoreover, the datasets used in their experiments involve no more than five steps per episode. It remains unclear how the computational cost and accuracy would scale as the number of steps increases.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the limitations of outcome supervised RL for RAG by introducing a process supervised framework called ReasonRAG. Unlike prior methods that only reward the final answer’s correctness, ReasonRAG leverages fine grained, stepwise rewards constructed via their Shortest Path Reward Estimation algorithm (SPRE). They also employ Monte Carlo Tree Search to explore and annotate intermediate reasoning trajectories. The resulting dataset (5 k questions, 13 k preference pairs) is called RAG-Proguide which is used to optimize the agentic RAG policy via DPO. Experiments were done on five QA benchmarks  and show that ReasonRAG trained on only 5k queries, outperforms Search-R1 baseline (90 k queries) in both EM and F1 while using less GPU compute\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n1. The introduction of SPRE to estimate process‐level rewards by penalizing unnecessarily long reasoning paths is a novel contribution that directly addresses sparse and delayed rewards in RL for RAG.\n2. Mixing MCTS and SPRE to label data by exploring and minimizing the cost of human annotation is an interesting approach. \n3. Proposed method is definitely compute efficient compared to Search-R1 in terms of GPU hours and the statistical significance test in the  table 2 is convincing. \n\nWeaknesses: \n1. Experiments use a Qwen2.5-7B-Instruct model only. It is uncertain whether ReasonRAG’s benefits translate to smaller or less capable LLMs. \n2. Although MCTS is used offline for annotation, the inference algorithm still involves iterative reasoning loops which may incur latency in real‐time applications. An analysis of runtime cost per query would strengthen practicality claims.\n3. While the authors claimed that they used 2 OOD datasets (unseen) it is not very convincing see the gains are marginal over the search-r1 in those 2 datasets (Bamboogle, MusiQue). And No recent baselines like search-O1. \n4. The base & SFT performance of the ReasonRAG is closer or same as standard RAG (table-3 row 1-2 and table 2 zero shot section) this requires further analysis on what exactly is happening.\n\nQUESTIONS:\n1. To prove the generality of the Method it will be interesting if the authors can perform experiment using a simple llama-3.1 or 3.3 model against search-o1 and search-r1 on 2 unseen datasets only. This will strength the papers claims. And take care of another baseline (i.e search-o1) . \n2. Giving the run time cost, speed , searches done and tokens for the inference will significantly help the paper.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}