Process vs. Outcome Reward: Which is Better for Agentic RAG Reinforcement Learning
Authors: Wenlin Zhang
Xiangyang Li
Kuicai Dong
Yichao Wang
Pengyue Jia
Xiaopeng Li
Yingyi Zhang
Derong Xu
Zhaocheng Du
Huifeng Guo
Ruiming Tang
Xiangyu Zhao
Venue: NeurIPS 2025 poster ()
Keywords: Process-Supervision
Agentic RAG
Primary area: reinforcement_learning

Abstract: Retrieval-augmented generation (RAG) enhances large language models (LLMs) by integrating external knowledge, yet traditional RAG systems struggle with static workflows and limited adaptability for complex, multistep reasoning tasks. Agentic RAG systems, such as DeepResearch, address these issues through dynamic retrieval, iterative context refinement, and adaptive workflows. However, recent methods like Search-R1, which rely on outcome-based reinforcement learning, face challenges such as low exploration efficiency, gradient conflict, and sparse reward signals. To tackle these limitations, we introduce ReasonRAG, a novel method that leverages RAG-ProGUIDE—a high-quality dataset providing fine-grained, process-level rewards for query generation, evidence extraction, and answer generation. By employing process-supervised reinforcement learning, ReasonRAG enhances LLMs’ autonomous capabilities in search, query generation, evidence extraction, and answer synthesis. Experimental results show that ReasonRAG, utilizing RAG-ProGUIDE, outperforms existing approaches like Search-R1 and traditional RAG systems, achieving superior performance on five benchmark datasets with only 5k training instances—significantly fewer than the 90k required by Search-R1. Our code is available at https://github.com/Applied-Machine-Learning-Lab/ReasonRAG.

Process vs. Outcome Reward: Which is Better for Agentic RAG Reinforcement Learning
Wenlin Zhang Xiangyang Li Kuicai Dong".,, Yichao Wang2 Pengyue Jia', Xiaopeng Li", Yingyi Zhang', Derong Xu', Zhaocheng Du Huifeng Guo".
Ruiming Tang". Xiangyu Zhao
City University Hong Kong, Hong Kong. Noah's Ark Lab, Huawei, China (wlzz, jia.pengyue, xiaopli2-c, yzhang6675-cc, derongxu2-c @ @my.cityu.edu..k.
[lixiangyang34, dong. kuicai, wangyichao5, zhaochengdu, huifeng xianzhao cityu.edu.hk
guo, guo, tangruiming @ @huawei.com,
Abstract
Retrieval-augmenne generation (RAG) enhances the text generation capabilities of large language models (LL.Ms) by integrating external knowledge and up-to-date information. However, traditional RAG systems are limited by static workflows and lack the adaptability required for multistep reasoning and complex task man- agement. To address these limitations, agentic RAG systems (e.g.. DeepResearch) have been proposed, enabling dynamic retrieval strategies, iterative context re- finement, and adaptive workflows for handling complex search queries beyond the capabilities of conventional RAG. Recent advances, such as Search-R1. have demonstrated promising gains using outcome-based reinforcement learning, where the correctness of the final answer serves as the reward signal. Nevertheless, such outcome-supervised agentic RAG methods face challenges including low explo- ration efficiency, gradient conflict, and sparse reward signals. To overcome these challenges, we propose utilize fine-graiedd process-level rewards improve training stability, reduce computational costs, and enhance efficiency. Specif- ically, we introduce novel method ReasonRAG that automatically constructs RAG-ProGuide, high-quality dataset providing process-level rewards for (i) query generation, (ii) evidence extraction, and (iii) answer generation, thereby en- hancing model inherent capabilities via process-supervised reinforcement learning. With the process-leve policy optimization, the proposed framework empowers LLMs to autonomously invoke search, generate queries, extract relevant evidence, and produce final answers. Compared to existing approaches such as Search-R and traditional RAG systems, ReasonRAG, leveraging superior performance on five benchmark datasets using only 5k training instances, significantly fewer than the 90k training instances required by Search-RI. Our code is available at https://gituub.com/Appiiee Machin---Laaaaaan-aaaaaa-eeee---
achieves
Introduction
Large language models (LLMs) 2 have demonstrated substantial proficiency in text generation and natural language understanding 4 revealing their potential for powering various downstream applications such as recommender systems [6]. However, their reliance on static training data constrains their ability to address dynamic and real-time queries, often resulting in outdated or hallucinated information 7 8 [9]. Retrieval-Augmenter Generation (RAG) [10] has emerged as
"Equal contribution
t Cooresponning Author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(a) -utcome-supervised RL
(b) Process-supervice RL LPRL
Low Exploration Efficiency
CORL O0OO
Gradient Conflict
Sporse Reward
Figure 1: Outcome--uppervised vs. process-supervised RL for multi-step reasoning. Each circle denotes one-step reasoning response, where the correct response colored as "green" and error response is colored as "red".
promising solution by equipping LLMs with external knowledge sources, improving the relevance, factual accuracy, and timeliness of responses [1]] 12 13 Despite these advances, traditional RAG architectures are limited by their linear and static workflows, which suffer from complex multi-step reasoning, deep contextual integration, and iterative response refinement (TO 14] To address these shortcomings, agentic RAG (e.g., DeepResearch [15] I6
systems have been developed. enabling dynamic retrieval strategies, enhanced contextual understanding, and iterative refinement. Achieving agentic RAG requires the underlying LLMs orchestrate retrieval, filter relevant information, and iteratively refine their outputs, resulting in more adaptive and efficient information processing.
To advance agentic RAG, early approaches [8 19 20
primarily focused prompt gineering to adapt powerful LLMs to agentic workflows. However, due to the limited reasoning and instruction-followinn capabilities of LLMs, supervised fine-tuning (SFT) methods 23 have been introduced, extending prompt-based approaches by directly optimizing and refining model parameters. Due SFT storing reasoning steps within the model parameters, the improved reasoning capabilities often encounter challenges in generalizing across different domains
22
[24] More recently,
reinforcement learning (RL) methods (e.g.. OpenAI-O1 (25] and DeepSeek-RI 226 achieve notable improvements in LLM reasoning by employing sutcome-superriss
RL techniques. Building on
these developments, Search-RI [27 incorporates a search engine as part of the LLM's environment and leverages outcome- based reinforcement learning, using the correctness of the final answer as the reward signal. These advances demonstrate that outeome-supervised reinforcement learning can substantially enhance the capabilities required for agentic RAG, enabling straightforward, end-to-end optimization of the entire workflow.
Despite its promise, outcome-supervised RL also presents inherent limitations, as illustrated in Figure [ First, low exploration efficiency occurs since the model must generate complete reasoning chain before receiving any reward (27). Ideally, the reward should be given when errors occur at intermediate steps facilitate learning. Second, gradient conflict arises when mistakes occur late in the reasoning process; the entire sequence (including correct early steps) is penalized [28). This can lead to conflicting gradients that can push correct actions in the wrong direction. Third, the rewards are sparse, as outcome- supervised RL only provides feedback upon producing the final answer (29] Reward sparsity relies on more training data and steps converge, as the model receives infrequent learning signals. In contrast, propess-supervieed RL addresses these issues by providing fine-grained, stepwise rewards throughout the reasoning process, enabling more efficient exploration, reducing gradient conflict, and accelerating model learning through denser feedback.
However, applying process-supervisee RL to RAG presents several key challenges. (1) Process Reward Design: Effective process rewards are essential for guiding the model toward the shortest and most efficient correct reasoning path. Rewards must incentivize helpful intermediate steps that lead to the correct final answer, while penalizing unnecessarily long or circuitous reasoning sequences [30]. (2) Exploration Efficiency and Annotation Cost: While human annotators who are skilled in information retrieval can create high-quality process-leve annotations by decomposing complex retrieval tasks into efficient steps, this approach is prohibitively expensive due to the substantial manual effort involved 31 In contrast, autonomous RAG agents can generate wide range of possible retrieval and reasoning steps, but this large search space makes difficult identify and select high-quality, meaningful steps for use process-leve level annotations.

---

(a) Process-Supervise Data Generation
(b) Agentic RAG Inference
SPRE Annolation
State Reasoning
Analyre and Decompose the Question
Evollate Your Knowladge
have sufficient knowledge, give answer step queey previous thoughts. Extract relevant evidance from retrived documents
Grounding
Reasoning Guideline
Correct
Terminal
Question: Which film the director
first, The Crime
Doctor's Courage Vasanthe Sena (1967 Film)?
Action Ge Geheration Q
Previous Thoughts:
Stop 1
Sies
Stap
Step 1: Query Geteration
Extection
56 the rext query is squrty- Birector The
Refermnc cnderencts docscheleeeree
Couraee-iuetyy
Gene 200n
Shep Evidence Extrection
Eaand
cedidnces George Sherman (teddense for the deectlo of Vitle Gams Dodor cooage!
relenat
Poiicy Preference lecettion
Preference Oprimization
Resporiee Shep ni Antover Consertion
Sune
slod on Decamber 12, 2010, Thandfore George Sheman sked fire. So the arawer
Courage
Ropated
Figure 2: Framework of ReasonRAG. Figure (a) illustrates the policy optimization based on process supervision. MCTS guides the construction of the state-action tree and the assignment of process-level rewards for fine-grained policy optimization. (Actions derived from the same state are color-coded by reward: green circle (highest), red circle (lowest).) Figure (b) demonstrates an inference example.
To address these challenges, we propose ReasonRAG, advanced process-supervised RL method to enhance agentic RAG reasoning. Specifically, ReasonRAG employs Monte Carlo Tree Search (MCTS) 332 as search strategy efficiently balance exploration and exploitation, enabling thorough exploration of diverse reasoning paths and identification of high-reward intermediate steps for guiding the RAG process. Building on these paths, we introduce novel Shortest Path Reward Estimation (SPRE) algorithm to assign rewards. SPRE favors sequences that lead to the correct answer while penalizing unnecessarily lengthy reasoning, thereby promoting efficiency. This approach yields RAG-ProGuide, a dataset comprising 5k queries with 13,000 high-quality process-level preference pairs. Using RAG-RrGGrr ProGuide and our process sureeessssseeeeieeee Direct Preference Optimization (DPO) (3) strategy, ReasonRAG is further trained to make dynamic decisions, such as whether to invoke retrieval, formulate subsequent search queries, analyze retrieved documents for relevant evidence, and synthesize evidence into final answers. Extensive experiments on five benchmark RAG datasets show ReasonRAG (trained with only 13k process-level steps) outperforms Search-RI (trained on 90k queries with approximately 270k intermediate steps), suggesting the superiority of process-supervics RL over suicome-supervisse RL. Our key contributions can be summarized as follows:
We propose ReasonRAG, an automatic framework for agentic RAG process-level reward annotations. We introduce SPRE for efficient RAG process-level reward annotation and MCTS for high-quality decision space exploration.
We introduce process-level annotation dataset MAG-ProGuide, which serves as an off-policy dataset, and can be easily applied for various LLM policy optimization.
We conduct extensive comparative experiments of outcome-supervisee RI. and process-supervised RI, for RAG reasoning with Dwen2.5--BBInnstreee The experimental results on five benchmark datasets demonstrate the superiority and training efficiency of ReasonRAG.
2 ReasonRAG Framework
2.1
Framework Overview
This section details the design of ReasonRAG framework, as depicted in Figure 2 our approach for constructing high-quality process-supervised data. We first introduce Shortest Path Reward Estimation (SPRE) to provide process-leve supervision reward (see Section 2.2.1 To efficiently gather these rewards, we employ Monte Carlo Tree Search (MCTS) algorithm to explore the vast decision space in agentic RAG and collect informative intermediate steps (see Section 2.2.2).
Figure 2 outlines
The resulting process-supervised dataset, RAG-ProGuide (see Section 2.2.3 is then used to optimize ReasonRAG via policy preference optimization. This training strategy guides the model to prefer desirable reasoning trajectories in agentic RAG (see Section 2.3
3

---

Figure 2p illustrates the agentic RAG inference workflow in ReasonRAG. During inference, the model adaptively conducts reasoning by dynamically invoking search engine and interleaving three core actions: query generation, evidence extraction, and answer generation (see Section 2.4)
2.2 Process-Superviser Data Generation
Effective process-supervised policy optimization requires high-quality, granular reward signals at the process level. As outlined in Section I generating such rewards for agentic RAG presents two main challenges: (I) the absence of reward functions for intermediate reasoning steps, and (2) the lack of an efficient and cost-effective method generate informative reasoning trajectories. To overcome these challenges, we introduce a novel process-level reward function, SPRE (see Section I. specifically designed for agentic RAG. Furthermore, we develop an MCTS-based approach (see Section 2.2.2) to efficiently explore the decision space and collect high-quality process-leve data.
2.2.1 Shortest Path Reward Estimation (SPRE)
Unlike outcome-level rewards, process-level rewards provide supervision at each intermediate step of agentic RAG. key challenge the absence of ground-truth reward signals for partial reasoning trajectories. Furthermore, due to the large decision space, the reward function must account for both final correctness and reasoning efficiency. To address these challenges, we propose Shortest Path Reward Estimation (SPRE), which evaluates the quality of each intermediate reasoning path by simulating its possible outcomes and penalizing unnecessarily long trajectories.
Formally, the agentic RAG process consists of an n-step sequence [y1. . where each y: represents the output of a single reasoning step, conditioned on the initial question r and previous steps Veg. To evaluate a partial sequence y1:t, we simulate multiple continuations, known as rollouts, until a final answer is obtained. By repeating the rollout process & times and scoring each outcome, we approximate the reward Monte Carlo-style estimation with step-based penalties:
Q: MonteCarlo(2,, V1:) = e(rollout.) step(rollout,)
Here, rollout, is the i-th simulated completion of y1:s, v(rollout;) [0, 1] denotes the correctness score (e.g.. F1 match to the ground truth), and step(rollout,) the number total reasoning steps in the trajectory. The decay factor a E (0, 1] penalizes unnecessarily long reasoning paths. This reward encourages the model to favor trajectories that achieve correct answers with fewer steps, thus balancing accuracy and efficiency in agentic RAG reasoning.
2.2.2 Monte Carlo Tree Search (MCTS) for Process-level Exploration
Although SPRE offers reliable reward signals for evaluating intermediate steps, generating diverse yet meaningful trajectories remains challenging. The search space in agentic RAG is extensive due to open-ended nature of retrieval, which requires continuous refinement of search queries for relevant information. To address this, we propose a tailored MCTS framework for agentic RAG. MCTS enables efficient exploration by selectively expanding the most promising reasoning paths based on estimated rewards.
We adapt MCTS to agentic RAG context by explicitly defining states and actions for tree con- struction. Formally, each intermediate reasoning step represented as a state (I, 3<s stage). where is the original question, Yei is the sequence of prior reasoning outputs, and stage E [Reasoning, Grounding, Terminal) indicates current point of agentic flow. Actions E (Query Gener- ation, Evidence Extraction, Answer Generation] are determined by the current stage as follows:
Choose between generating a new query for document retrieval or directly generating an answer. If a new query is generated, a retrieval operation is performed, and the retrieved documents are appended the state for subsequent decisions. If an answer is produced, the process transitions to the Terminal stage.
Reasoning stage:
Select evidence spans from the retrieved documents. Afterwards, the system returns to the Reasoning stage for further reasoning or answering.
Grounding stage:
Terminal stage: End the exploration process when the final answer has been generated.

---

4k
3k
3k
Statistics
Number
1k
Questions
4603
1k
PooQA
704 (15.3%)
AA AQ
QA
QQ
0k
HotporQA
2843 (61.8%)
2WikiMulthopPA
1056 (22.9%)
(a) Pair Type.
(b) Iteration Count
Actions
13289
6k
3.2
Preforence Gap
-Query Generation
3295 (24.8%)
4k
2.4
Evvidence Extraction
4305 (32.4%)
3k
-nsser Generation
5689 (42.8%)
1k
0.8
0.0
Min./Mcdd/Max. Iteration
2.7/1/3/5
40
80
120
160200+
0.0
0.2
0.4
0.6
0.8
/Min//Med//Max. Tokens
65.5/9/60/625
(c) Token Length
(d) Reward Gap
Table 1: Overall Dataset Statistics
Figure 3: Dataset Distribution.
0.523 Question: whirh film has the director died later,
Shark Moncoe?
Response: the euery 1s equersswhat is the releise year of I1 coragpto and who directed 119c/query.
Keward:
Reward: 0.628
Response: liared on the query. the re relevant evi dence 15 <ovl dencas2) coragglo 2955 stalfan couedy fils strected by boeent Pan la.
4
Rewards 0.678
6
Response: So the next query 1s <queryenho dfrrcced
Response: dincesii11iann directed Shark Vanroe, which Nas released in 1918. wi1112a Hart
the rerecarr's death? </quarys
died on becester 2, 205..</eviderce.
Response: 0.731 query
Seward: 0.711
Response: dendddnnncoeeeeeee rapiella 2007 idevide
16
15
drd conenfico Paolella diwlie/qurry
Revard: 0.81
17
Response: the answer 15 CResWersT aaragglo</ansarr
Figure 4: A Tree Example for the process-level preference annotations.
Based on the current state s, the policy for generating the next action is defined as:
Te( xr M-ci: Psuge). no( x, Bici, docs, Pstage),
stage is Reasoning otherwise
(a s) LLM((
(2)
State transitions are defined as 8t41 = concatenate(s,, az), where each action leads to new node in the search tree, representing extended reasoning sequence. This recursive process incrementally builds a tree rooted at the original question. MCTS then operates iteratively, performing three core steps: selection, expansion, and backpropagation. Specifically, each iteration, MCTS selects promising paths using a Upper Confidence Bound (UCB) based objective, expands new states by sampling --geerrredeee actions, and backpropagates SPRE-estimated rewards (see Equation I] to update the tree (see more comprehensive explanation of MCTS exploration in Appendix AA) Integrating MCTS with SPRE enables efficient exploration and prioritization of high-reward reasoning steps, producing high-quality process-level annotations to optimize the agentic RAG policy.
2.2.3 RAG-ProGuide Dataset
Construction. Using the MCTS-based exploration framework, we construct high-quality process- supervised dataset RAG-ProGuide to facilitate process-level policy optimization. We randomly sample 3,000 questions each from PopQA (34) HotpoiQA [35. and 2WikiMultihopQA (36). cov- ering both mnole-hhp and multi-hop question answering tasks. An advanced large language model serves as the policy model simulate the agentic RAG reasoning process within the MCTS frame- work (see Section 2.2.2p. During tree search, we prune all branches that do not yield a final answer. For each complete trajectory, we compute and use this correctness signal to estimate intermediate node rewards via SPRE (see Section 2.2.1) These rewards are propagated through the MCTS tree to guide preference pair selection. ensure high-quality preference data, perform post-processing to remove duplicates and uninformative
score between predicted answer and ground truth,
5

---

comparisons: (i) we discard preference pairs with identical response sequences. and (ii) pairs with reward difference less than 0.01. After filtering, the final dataset consists of 4,603 questions and 13,289 distinct preference pairs. Figure - illustrates an example for the tree-structured process data. The root node corresponding to the original question, each node corresponds one-step response from LLMs. The correct reasoning path has been colored in orange and annotated with a higher reward.
Dataset Statistics. Table I presents detailed statistics and distributions for our dataset. The questions are drawn from PopQA, HotpotQA, and 2WikiMulihopQA, providing comprehensive coverage of both single-hop and multi-hop reasoning within the RAG decision space. The dataset contains a balanced distribution of three reasoning actions, reflecting the multi-stage nature of the agentic RAG process. As shown in Figure Za the distribution of preference pair types demonstrates diverse comparative scenarios; the x-axis abbreviations (A: answer generation, Q: query generation, E: evidence extraction) indicate action types in accepted versus rejected paths. This diversity ensures fine-graie fine-grained comparative coverage across different reasoning stages. Figure (2b) indicates range of reasoning iteration counts, consistent with the complexity of multi-hop inference. Figure 22 shows broad distribution of response token lengths, confirming the dataset's 's capacity to capture various response complexities. Additionally, Figure 2d illustrates the probability distribution of the reward gap between preference pairs. The comprehensive coverage of this distribution across wide range of values ensures that the preference learning is guided by a rich and informative signal. Collectively, these statistics demonstrate the dataset's s quality and its suitability for training robust process-level decision policies in agentic RAG frameworks.
2.3 Prrccss-Supervice Preference Optimization
Based on the process-supervised preference data, we apply DPO 33 to tune the policy model. The optimization objective can be denoted follows:
Po (yt xr vet))
((0) -E (r.ee -D log
( Bllg
po (o sr Vet)
where x denotes the original question, Ver denote the the responses from previous reasoning steps, ot and of represent the preferred and dispreferred responses in the subsequent step, respectively, and the hyperparameter / controls the KL constraint.
Algorithm 1 Agentic RAG Inference Pipeline
2.4 Agentic RAG Inference
Original question æ; large language model Ta. retriever R, maximum reasoning round N.
Require:
To enable LLMs to autonomously interact with external information, we propose an agen- tic RAG workflow that supports adaptive rea- soning through iterative search and reflection. ReasonRAG allows the model dynamically determine when and how to invoke search en-
Ensure: Final response
1: Initialize reasoning count + 0, and stage t
Reasoning
2: for i t 0 to N do
3:
stage Reasoning then
gine based on question complexity.
4:
~ To( (z., Ueti,Ptage)
5: else
The workflow operates through three recur- sive decision states: Reasoning, Grounding, and Terminal. In the Reasoning state, the LLM evaluates the current context to decide it has sufficient information to answer the question. If sufficient, it generates final an- swer enclosed in placeholders (<answer> and </ /aanswr>>), thus terminating the process. If not, the model creates a new query enclosed
6:
~ To(-/or U-i does, Pstage)
7: end
8:
y 1 +
9:
<query>eyyuuu detected in yi then
10:
stage Grounding
11:
q + +-xtractt__a -extract_quer((y))
12:
docs R(g)
13:
else if <answer> detected in y then
14:
stage Terminal
in <query> and </query> placeholders to re- trieve additional evidence. The system then transitions to the Grounding state, where doc- uments are retrieved based on the query, and the model extracts relevant evidence spans. These evidence spans are appended to the context, after which the process loops back to the Rea-
15:
return extract_answer((
16:
else if <evidence> detected in y/ then
17:
stage + Reasoning
18:
end if
19:
end for
20:
return final response y
soning state for further deliberation. (See Appendix
F for detailed prompt designs.)

---

Table Main Results (%) on Five benchmarks (the number of queries used for training is indicated in brackets). *** indicates the statistically significance (i.e., two-sided t-test with 0.05) over the best baseline. Two most important columns: the averaged EM and F1 are highlighted
PopQA
HotpotQA
2WikiMulti Bamboogle
MuSiQue
Avg.
Type
Method
EM
Fi
EM
Fi
EM
Fi
EM
Fj
EM
F1
EM
Fj
Naive Generation
12.7
16.5
15.7
24.8
20.2
28.0
6.4
17.4
2.7
10.2
11.5
19.4
Zero-shot
Standard RAG
38.4
44.7
29.3
39.9
29.4
36.3
17.6
24.1
6.7
15.1
24.3
32.0
FLARE
14.3
17.6
18.1
25.7
27.9
32.8
12.0
20.8
4.3
12.6
15.3
21.9
Active
Self-RAG(146k)
22.7
33.9
21.0
29.7
12.0
25.2
1.6
10.9
4.6
13.3
12.4
22.6
Adaptive
AdaptiveRAA(33k)
36.6
41.5
29.1
40.7
24.2
33.4
18.4
26.1
6.9
14.3
23.0
31.2
Iter-Retgen
38.7
44.9
30.3
42.1
31.2
38.7
19.2
26.4
7.7
14.2
25.4
33.3
RAG-CoT
IRCoT
36.2
43.6
27.7
41.5
23.5
32.5
17.2
22.5
8.6
13.2
22.6
30.7
RECOMP
40.5
45.8
29.7
41,2
33.2
39.4
21.7
28.6
9.2
15.8
26.9
34.2
Summary
LongL.LML.ingua
39.2
45.1
31.4
43.2
34.5
40.2
20.3
27.4
8.7
14.9
26.8
34.2
Sclective-Context
34.9
41.5
19.3
27.3
20.3
29.7
15.3
22.6
6.1
13.7
19.2
27.0
Search-ol
33.2
40.3
24.8
38.1
16.4
27.1
30.4
40.6
6.3
13.7
22.2
31.96
AutoRAG(10k)
38.6
44.1
33.3
43.7
39.5
46.1
24.8
32.2
11.3
18.3
29.5
36.9
Reasoning
Search-RI (90k)
39.7
44.8
37.0
47.0
41.4
48.0
32.0
43.8
14.6
19.9
32.8
40.7
Rea5onRAG(5K)
41.5*
46.2*
38.4
48.9*
43.6*
50.4*
36.0
45.5
12.8
20.6
34,4*
42.3*
In summary, ReasonRAG supports multi-step. flexible reasoning while maintaining structured decision control. The use of explicit placeholders enhances interpretability and facilitates programmatic control during deployment. The complete algorithmic flow is provided in Algorithm
3 Experiments
3.1 Experimental Setup
Evaluation Dataset & Metrics. We evaluate ReasonRAG and all baselines five public benchmarks: the single-hop QA dataset PopQA (34] and four multi-hop QA datasets, including HotpotQA (35), 2WikiMultHHoQAA (36,, Bamboogle (37,. and MuSiQue (38. Bamboogle and MuSiQue serve as out-of-domain QA evaluation datasets. The diversity of these datasets enables a comprehensive assessment of agentic RAG. We report Exact Match (EM) and F; scores evaluation metrics. Refer to Appendix D for more details about dataset introduction, statistics, and metrics.
Baselines. We implement 12 baseline models which can be categorized into 6 types follows: Zero-shot: Directly use prompt engineering on L.L. answer the question without or with retrieved documents [39]. Active: Actively make additional retrieval when retrieved data or generated responses have low confidence 440 41]. Adaptive: Dynamically chooses the most suitable RAG
pipeline from no-retrieval, single-hop, multi-hop retrieval strategies 42 RAG-CoT: Integrates chain-of-thought reasoning with retrieval, enabling multi-step, evidence-seeking answers 43 44)
Summary: Compresses summarizes retrieved content to fit model input constraints while retaining key information [45] 46 47). Reasoning: Enhances multi- hopi reasoning by structuring the reasoning
process and scrutinizing retrieved evidence [48 27 49 Note that ReasonRAG and all baselines
use 2.ven2.5-7B-Instree [50] as the backhone model, ensuring fair comparison. Refer to more implementation details about ReasonRAG and baselines in Appendix F. and H.2
3.2
Main Results
We present detailed performance results on ReasonRAG against 12 baselines across five benchmark datasets, shown in Table 2 Our key findings are summarized below:
Data Efficiency: ReasonRAG, despite being trained on only 5k queries, outperforms the search-RI bascline trained with 90k queries. On average across all datasets, ReasonRAG achieves higher EM
7

---

0.42
0.45
0.40
0.41
0.40
0.35
0.38
0.37
0.30
0.30
0.36
ReasonRAG
ReasonRAG
0.350
6
0.25
0.25
6
GPU Hours(Log
GPU Hourstlog 22)
GPU Hours(Log
(a) PopQA
(b) 2WikiMultiHopQA
(c) HotpotQA
Figure 5: Training cost and convergence speed comparison (EM%) for ReasonRAG and Search-RI
Table 3: Impact of different optimization strategies on ReasonRAG 's effectiveness.
PopQA
HotpotQA
2WikiMulti
Bamboogle
MuSiQue
Avg.
Method
EM
Fi
EM
F
EM
Fi
EM
Fi
EM
EM
Fi
ReasonRAG (Base)
35.6
42.7
23.7
38.2
15.2
28.9
28.0
38.7
7.7
15.4
22.0
32.8
ReasonRAG (SFT)
31.6
37.4
26.8
38.7
35.1
40.9
17.6
27.3
8.6
15.5
23.9
32.0
ReasonRAG (RL-ORL): queries
30.9
28.1
32.0
43.8
24.1
13.1
21.3
ReasonRAG (RL-ORL): 10k queries
39.5
45.7
36.7
46.7
40.5
47.2
30.7
40.6
12.6
19.5
32.0
39.9
ReasonRAG (RL-PRL)
41.5
46.2
38.4
48.9
43.6
50.4
36.0
45.5
12.8
20.6
34.5
42.3
(34.4%) and F, (42.3%) scores compared to search-R1 (32.8% EM, 40.7% F1). highlighting the superior data efficiency of ReasonRAG. This demonstrates the effectiveness of process-supervised RL, which leverages fine-grained rewards, over current outcome-supervised methods.
Multi-hop Reasoning: ReasonRAG shows substantial performance gains multi-hop reasoning tasks. On the HotpotQA dataset, it achieves an F1 score 98,,, outperforming models like AutoRAG (43.7% F,) and search-R1 (47.0% F.), both of which trained larger datasets. This underscores ReasonRAG 's strength in handling complex, multi-step questions that require integrating evidence from multiple sources.
Out-of-domain Generalization:
ReasonRAG demonstrates strong generalization to out-of-domain data. On challenging benchmarks such as Bamboogle and MuSiQue, it consistently achieves higher Fi scores relative to other baselines. This indicates improved robustness and transferability of its reasoning capabilities across different domains.
3.3 Training Efficiency
compares the training efficiency of ReasonRAG and Search-R1. The figure illustrates the progression of EM scores with increasing GPU hours across three datasets. The results reveal that ReasonRAG has higher training efficiency compared to Search-R1. ReasonRAG achieves superior EM scores with fewer GPU hours, indicating that requires less training data and compute to reach strong performance levels. In contrast, Search-RI R1 requires significantly more GPU hours to reach similar performance.
Figure 5
The efficiency gap between the two models is particularly notable multi-hop question answering tasks. For the single-hop PopQA dataset, performance gains for both models are comparably rapid as training progresses. However, for multi-hop datasets such 2WikiMultiHopQA and HotpotQA, ReasonRAG consistently demonstrates significant improvements with increased GPU hours. This further underscores its effectiveness on complex reasoning tasks, where it delivers faster and greater performance improvements with fewer resources.
3.4 Effectiveness of Different Optimization Strategies
In this section, we compare the effectiveness of ReasonRAG utilizing three different optimization strategies against the base model. Our default approach, ReasonRAG (RI-PRL), is trained with process-level rewards as described Section 2 For ReasonRAG (RL-ORL),, we adopt outcome- level reward training following the Search-RI protocol [27). Specifically, evaluate two versions:

---

40
40
40
35
35
30
30
30
25
25
20
Reason )
20
20
15
1
PRL
15
15
10
10
(a) PopQA
(b) 2WikiMuliiHopQA
(c) HotporQA
Figure 6: The EM performance across varying retrieval iterations on 3 benchmarks.
K-5
50
50
45-
45
40
35
30 EM
ACC
EM
F1
ACC
25
EM
ACC
(a) PopQA
(b) 2WiiiMultHHopQA
(c) HotpotQA
Figure 7: Effect of top-k retrieved documents on ReasonRAG's performance across 3 datasets.
RL-ORL-5K, trained on the same 5k queries as RL-PRL, and RL-ORL-1OK, which incorporates an additional 5k queries sampled from PopQA, HotpotQA, and 2WikiMultiHopAA, totaling 10k training examples. For ReasonRAG (SFT), we use the preferred responses from the RAG-ProCuide preference pairs ground truth and apply supervised fine-tuning (SFT) via next-token prediction. Table 3 summarizes the performance of these four variants. Our main findings are as follows:
Superiority of PRL: ReasonRAG (PRL) consistently outperforms all other variants across all datasets, both in-domain and out-of-domain, indicating stronger generalization capabilities.
High Data Demand of ORL: ReasonRAG (ORL) achieves the second -bett results, but requires substantially more training data match the comparable performance of PRL. Although ORL is more effective than Base and SFT, its training efficiency is relatively low.
Overfitting in SFT: SFT leads to overfitting multi-hop reasoning paths, resulting in reduced per- formance on single-hop tasks. Furthermore, SFT-trained models generalize poorly, as demonstrated by a marked performance decline on the Bamboogle dataset.
3.5 Impact of Search on Performance
Performance Retrieval Steps. Figure shows the EM performance of ReasonRAG across varying retrieval iterations on datasets. We observe consistent trend: performance improves with more retrieval iterations and then gradually saturates. Notably, ReasonRAG can adaptively determine the required inference depth according to task complexity. For the single-hop PopQA dataset, performance converges within 2 to 3 retrieval steps, whereas more complex multi-hop tasks such as 2WikiMultiHopQA and HotpotQA require to 5 steps to reach peak performance. In contrast, ReasonRAG (base) without preference optimization only achieves reliable gains on PopQA and struggles to handle multi-hop settings. These results demonstrate ReasonRAG 's ability to perform adaptive reasoning response to the complexity the input question.
Performance vs. Number of Retrieved Documents. Figure 7 compares the performance of ReasonRAG under different top-k retrieval settings, where refers retrieving top-k relevant docu- ment passages per search query. Results indicate that while ReasonRAG remains robust across a range of & values, its performance is sensitive to the quantity of retrieved information. With k limited context restricts the model's reasoning ability. Increasing & to 3 yields significant improvements across all datasets, suggesting that ReasonRAG effectively leverages additional evidence. Further increasing & to 5 does not further increase the performance PopQA and HotpotQA, whereas

---

2WikiMultiHoPQA continues to benefit from richer retrieved context. Overall, these findings high- light ReasonRAG 's capacity to utilize additional retrieved documents, particularly in more complex multi-hop scenarios.
4 Related Works
Prompt-Base Based Agentic RAG. Early prompt-based approaches leverage manually designed work- flows to elicit the inherent capabilities of LLMs for interacting with external information. Specifically, the RAG task is often decomposed into subtasks such as adaptive retrieval judgment [42], query
generation 21 43 441, evidence extraction 41] 45 47 46), and answer generation [40]. While some
efforts have focused on optimizing RAG through personalization [51], graph- -asedd retrieval [52]. or reranking techniques [53], yet a critical gap remains in designing LLMs that can autonomously invoke search engines. Recently, agentic RAG aims to design workflows that empower LLM to au- tonomously interact with external information. OpenResearcher [54,, AirRAG (55), IRe (56), PlanRAG [57]. and Search-ol [49] demonstrate strong performance improvement by the effective incorporation with the search engine. Nevertheless, these methods are limited by their dependence on inherent capabilities for interacting with external information and the requirement for manual design when applied to new domains, and lack explicit mechanisms for climinating distracting information [58].
RL-Based Agentic RAG. Reinforcement Learning (RL) has consistently delivered significant performance gains across spectrum of sequential decision-making tasks, evidenced by its successful application in domains such as recommender systems [59, 60) 61 62 63 Recently, the success of models like DeepSeek-R1 (26) has vividly demonstrated the substantial potential of outcome-supprrised RL in enhancing complex reasoning capabilities, establishing it as a mainstream paradigm for end-to-end optimization of LLMs. Following the widespread adoption of RL by major AI companies to improve the reasoning abilities of their models on complex tasks 64 65 66 recent
work 27 67 68 69 has extended outcome supperiiee reinforcement learning RAG, empowering
LLMs to autonomously utilize search engines for intricate inference. While sutcome-superviss RL has demonstrated significant performance gains, it also faces challenges such as reward sparsity, training instability, and substantial computational cost. Moreover, silccome-uuperving demands extensive training resources. contrast, process-supervee
RL typically RI. has recently been applied to enhance reasoning abilities, outperforming outcome-supervieee approaches by providing fine- grained rewards [70] 71 72 73]. As an alternative avenue for improving LLM reasoning in RAG, process-supervised RL for RAG remains unexplored.
5 Conclusion
We introduce ReasonRAG, a process-supervised agentic RAG method for fine-grained policy opti- mization. Our approach integrates Monte Carlo Tree Search (MCTS) with the agentic RAG workflow to generate RAG- AAG-ProGuide, a high-quality dataset providing process-level supervision by prioritizing the shortest reasoning paths leading to correct answers. Leveraging AAG-ProGuide, we perform preference-bane policy optimization to enhance I.IMs' autonomous capabilities in query generation, evidence extraction, and answer synthesis. Experiments demonstrate that ReasonRAG achieves su- perior performance on five benchmark datasets using only 5k training instances, significantly fewer than the 90k required by Search-R1, highlighting the effectiveness of MGG-Prouuide 's high-quality process-level rewards in optimizing agentic RAG policies.
Acknowledgements
This research was partially supported by National Natural Science Foundation of China (2.622002444. Hong Kong Research Grants Council's Research Impact Fund 1105520)--.).. Research Grants Council's Collaborative Research Fund cil's General Research Fund (No. 1218325), Graduate Research Fund of the School of Economics and Management of Dalian University of Technology (No. DUTSEMDREKOI) Institute of Digital Medicine of City University of Hong Kong (No.9229503), and Huawei (Huawei Innovation Research Program).
Research Grants Coun-
10

---

References
[1] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33:18 13:1877-1901, 2020.
[2] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1- 113, 2023.
[3] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidr, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08874, 2023.
[4] Yizhang Zhu, Shiyin Du, Boyan Li, Yuyu Luo, and Nan Tang. Are large language models good statisticians? In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
[5] Javier González and Aditya Nori. Does reasoning emerge? examining the probabilities of causation in large language models. Advances in Neural Information Processing Systems, 37:117737-117711 2024.
[6] Wenlin Zhang, Chuhan Wu, Xiangyang Li, Yuhao Wang, Kuicai Dong, Yichao Wang, Xinyi Dai, Xiangyu Zhao, Huifeng Guo, and Ruiming Tang. Limtreerec: Unleashing the power of large language models for cold-start recommendations. In 31st International Conference on Computational Linguistics (COLING 2025), pages 886-896. Association for Computational Linguistics, 2025.
[7] Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishüi, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM computiigg surveys, 55(12):1 2023.
[8] Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qiang- long Chen, Weihua Peng. Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 43(2):1-55, 2025.
[9] Nick McKenna, Tianyi Li, Liang Cheng, Mohammad Hosseini, Mark Johnson, and Mark Steedman. Sources of hallucination by large language models on inference tasks. Findings of the Association for Computational Linguistics: EMNLP 2023, 2023.
[10] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks, 2021.
[11] Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. Retrieval-augmentee generation for large language models: A survey, arXiv preprint arXiv:2312 10997, 2, 2023.
[12] Wenqi Fan, Yujuan Ding, Liangbo Ning. Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. A survey on rag meeting llms: Towards retrieeal-uugmenntd large language models. In Proceedings of the 30th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pages 6491-6501, 2024.
[13] Guangzhi Xiong, Qiao Jin, Zhiyong Lu, and Aidong Zhang. Benchmarking retrieval- augmeeted generation for medicine. Findings of the Association for Computational Linguistics ACL 2024, pages 6233-6251, 2024.
[14] Aditi Singh, Abul Ehtesham, Saket Kumar, and Tala Talaei Khoei. Agentic retrieval-augmented generation: A survey on agentic rag, arXiv preprint arXiv:2501.09136, 2025.
[15] Google. Gemini deep research. 2024.

---

[16] OpenAl. Deep research system card. Technical Report, 2025.
[17] Wenlin Zhang. Xiaopeng Li, Yingyi Zhang, Pengyue Jia, Yichao Wang, Huifeng Guo, Yong Liu, and Xiangyu Zhao. Deep research: A survey autonomous research agents. arXiv preprint arXiv:2508. 12752, 2025.
[18] Maojia Song, Shang Hong Sim, Rishabh Bhardwaj.. Hai Leong Chieu, Navonil Majumder, and Soujanya Poria. Measuring and enhancing trustworthiness of Ilms in rag through grounded attributions and learning to refuse. The Thirteenth International Conference on Learning Representations.
[19] Jachyung Kim, Jachyun Nam, Sangwoo Mo, Jongjin Park, Sang- Woo Lee, Minjoon Seo, Jung-Woo Ha, and Jinwoo Shin. Sure: Summarizing retrievals using answer candidates for open-domain qa of llss. In The Twelfth International Conference on Learning Representations.
[20] Zhiqing Sun, Xuezhi Wang, Yi Tay, Yiming Yang. and Denny Zhou. Recitation-augmeene language models. In The Eleventh International Conference on Learning Representations.
[21] Reinald Kim Amplayo, Kellie Webster, Michael Collins, Dipanjan Das, and Shashi Narayan. Query refinement prompts for closed-book long-form qa. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7997-8012, 2023.
[22] Yuhao Wang. Ruiyang Ren, Junyi Li, Wayne Xin Zhao, Jing Liu, and Ji-Rong Wen. Rear: relevance-aware retrieval- -uggmetedd framework for open-domain question answering. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5613-5626, 2024.
[23] Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat. When scaling meets I1m finetuning: The effect of data, model and finetuning method. In The Twelfth International Conference on Learning Representations.
[24] Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc Le, Sergey Levine, and Yi Ma. Stt memorizes, rl generalizes: comparative study of foundation model post-training. arXiv preprint arXiv:2501.1716/ 2025.
[25] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card, arXiv preprint arXiv:2412.167200 2024.
[26] Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Iims via reinforcement learning. arXiv preprint arXiv:2501.12944. 2025.
[27] Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Za- mani, and Jiawei Han. Search-rl: Training lims to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.095/6 2025.
(28) Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference Learning Representations, 2023.
[29] Yuqing Du, Alexander Havrilla, Sainbayar Sukhbaatar, Pieter Abbeel, and Roberta Raileanu. study on improving reasoning in language models. In / Can't Believe It's Not Better Workshop: Failure Modes in the Age of Foundation Models, 2024.
[30] Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Meiqi Guo, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, et al. Improve mathematical reasoning in language models by automated process supervision. arXiv preprint urXiv:2406.06592. 2024.
[31] Jeremy Scheurer, Jon Ander Campos, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. Learning from natural language feedback. In ACI. Workshop on Learning with Natural Language Supervision, 2022.
12

---

[32] Cameron B Browne, Edward Powley, Daniel Whitehouse, Simon M Lucas, Peter Cowling. Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. A survey of monte carlo tree search methods. IEEE Transactions Computational Intelligence and in games, 4(1):1-43 2012.
[33] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728-53337 2023.
[34] Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Hannanch Hajishirzi, and Daniel Khashabi. When not to trust language models: Investigating effectiveness and limitations of parametric and non-parametric memories. arXiv preprint arXiv:2212. 7, 2022.
[35] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2018.
[36] Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6609-6625, 2020.
[37] Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith. and Mike Lewis. Measuring and narrowing the compositionality gap in language models. In The 2023 Conference on Empirical Methods Natural Longuage Processing.
[38] Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539-555 554, 2022.
[39] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, al. Retrieval-uummenedd generation for knowledge-intensse nlp tasks. Advances in neural information processing systems, 33:9459-9474, 2020.
[40] Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yo Yu, Yiming Yang, Jamie Callan, and Graham Neubig. Active retrieval augmented generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7969-7992, 2023.
[41] Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Leam- ing to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference Learning Representations.
[42] Soyeong Jeong, Jinheon Back, Sukmin Cho. Sung Ju Hwang, and Jong-Cheol Park. Adaptive- rag: Learning to adapt retrieval-ans -aggeented large language models through question complexity. In 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 7036-7050. Association for Computational Linguistics, 2024.
[43] Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Interleaving retrieval with chain-of-thoon ofothough reasoning for knowledge-intentt intensive multi-step questions. In The 6l st Annual Meeting Of The Association For Computational Linguistics, 2023.
[44] Zhihong Shao, Yeyun Gong, Minlie Huang, Nan Duan, Weizhu Chen, al. Enhancing retrieval- augmented large language models with iterative sttrieall-gnneran synergy. In The 2023 Conference on Empirical Methods in Natural Language Processing.
[45] Fangyuan Xu, Weijia Shi, and Eunsol Choi. Recomp: Improving retrieval-augmente Ims with compression and selective augmentation. In 12th International Conference on Learning Representations, ICLR 2024, 2024.
13

---

[46] Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yev Lin, Yuging Yang. and Lili Qiu. Longllmlingua: Accelerating and enhancing llms in long context scenarios via prompt compression. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1658-1677, 2024.
[47] Yucheng Li, Bo Dong, Frank Guerin, and Chenghua Lin. Compressing context to enhance inference efficiency of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6342-6353, 2023.
[48] Tian Yu, Shaolei Zhang. and Yang Feng, Auto-rag: Autonomous retrieval auggmente generation for large language models. arXiv preprint arXiv:2411.19443, 2024.
[49] Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. Search-ol: Agentic search-enhanced large reasoning models, arXiv preprint arXiv:2501.05366 2025.
[50] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 2.Kiv:2412.15115. 2024.
[51] Xiaopeng Li, Pengyue Jia, Derong Xu, Yi Wen, Yingyi Zhang, Wenlin Zhang, Wanyu Wang, Yichao Wang, Zhaocheng Du Xiangyang Li, et al. A survey of personalization: From rag to agent. arXiv preprint arXiv:2504. 101447 2025.
[52] Derong Xu, Pengyue Jia, Xiaopeng Li, Yingyi Zhang, Maolin Wang, Qidong Liu, Xiangyu Zhao, Yichao Wang. Huifeng Guo, Ruiming Tang, al. Align-grag: Reasoning-guided dual alignment for graph retrieval-augmented generation. arXiv preprint arXiv:2505.162:5 2025.
[53] Pengyue Jia, Derong Xu, Xiaopeng Li, Zhaocheng Du, Xiangyang Li, Yichao Wang. Yuhao Wang, Qidong Liu, Maolin Wang, Huifeng Guo, Ruiming Tang, and Xiangyu Zhao. Bridging relevance and reasoning: Rationale distillation in serieval-augmenned generation. In Findings of the Association for Computational Linguistics: ACL 2025, pages
2025.
[54] Yuxiang Zheng, Shichao Sun, Lin Qiu, Dongyu Ru, Cheng Jiayang, Xuefeng Li, Jifan Lin, Binjie Wang. Yun Luo, Renjie Pan, et al. Openresearcher: Unleashing ai for accelerated scientific research. In Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pages 209-218, 2024.
[55] Wenfeng Feng, Chuzhan Hao, Yuewei Zhang, Jingyi Song, and Hao Wang. Airrag: Activating intrinsic reasoning for retrieval augmented generation via tree-based search. arXiv preprint 2rX1v:2501.10053, 2025.
[56] Zhenrui Yue, Honglei Zhuang, Aijun Bai, Kai Hui, Rolf Jagerman, Hansi Zeng, Zhen Qin, Dong Wang, Xuanhui Wang, and Michael Bendersky. Inference scaling for long-context retrieval augmented generation. arXiv preprint arXiv:2410.04343, 2024.
[57] Prakhar Verma, Sukruta Prakash Midigeshi, Gaurav Sinha, Arno Solin, Nagarajan Natarajan, and Amit Sharma. Plan rag: Efficient test-time planning for retrieval augmented generation. In Workshop Reasoning and Planning for Large Language Models, 2025.
[58] Yuanwei Liu, Nian Liu, Yi Wu, Hisham Cholakkal, Rao Muhammad Anwer, Xiwen Yao, and Junwei Han. Ntrenet+:: Unleashing the power of non-target knowledge for few-shot semantic segmentation. IEEE Transactions on Circuits and Systems for Video Technology, pages 1-1, 2024.
[59] Xiangyu Zhao, Long Xia, Liang Zhang, Zhuoye Ding, Dawei Yin, and Jiliang Tang. Deep reinforcement learning for page-wise recommendations. In Proceedings of the 12th ACM conference on recommender systems, pages 95-103, 2018
[60] Xiangyu Zhao, Liang Zhang, Long Xia, Zhuoye Ding, Dawei Yin, and Jiliang Tang. Deep reinforcement learning for list-wise recommendations, In Ist Workshop on Deep Reinforcement Learning for Knowledge Discovery (DRLAKDD 2019), 2019.
14

---

[61] Xiangyu Zhao, Liang Zhang, Zhuoye Ding, Long Xia, Jiliang Tang, and Dawei Yin. Recom- mendations with negative feedback via pairwise deep reinforcement learning. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1040-1048, 2018.
[62] Xiangyu Zhao, Changsheng Gu, Haoshenglun Zhang, Xiwang Yang, Xiaobing Liu, Jiliang Tang, and Hui Liu. Dear: Deep reinforcement learning for online advertising impression in recommender systems. In Proceedings of the AAAI conference on artificial intelligence. volume 35, pages 750-758, 2021.
[63] Ziru Liu, Jiejie Tian, Qingpeng Cai, Xiangyu Zhao, Jingtong Gao, Shuchang Liu, Dayou Chen, Tonghao He, Dong Zheng, Peng Jiang, et al. Multi-task recommendations with reinforcement learning. In Proceedings of the ACM web conference 2023, pages 1273-1282, 2023.
[64] Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501. 12599, 2025.
[65] DiJia Su, Sainbayar Sukhbaatar, Michael Rabbat, Yuandong Tian, and Qinqing Zheng. Dual- former: Controllable fast and slow thinking by learning with randomized reasoning traces. In The Thirteenth International Conference on Learning Representations, 2024.
[66] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
[67] Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. Deeprescarcher: Scaling deep research via reinforcement learning in real-world environments. arXiv preprint arXiv:2504.00 0a:e:22040030000 06000, 2025.
[68] Hongru Wang, Cheng Qian, Wanjun Zhong, Xiusi Chen, Jiahao Qiu, Shijue Huang, Bowen Jin, Mengdi Wang, Kam-Fai Wong, and Heng Ji. Ote: Optimal tool calls via reinforcement learning. arXiv preprint arXiv:2504. 10.55044.4870 2025.
[69] Hao Sun, Zile Qiao, Jiayan Guo, Xuanbo Fan, Yingyan Hou, Yong Jiang, Pengjun Xie, Fei Huang, and Yan Zhang. Zerosearch: Incentivize the search capability of llms without searching. arXiv preprint arXiv:2505. 04588, 2025.
[70] Ruotian Ma, Peisong Wang. Cheng Liu, Xingyan Liu, Jiaqi Chen, Bang Zhang, Xin Zhou, Nan Du, and Jia Li. S2r: Teaching ilms to self-verify and self-correct via reinforcement learning. arXiv preprint arXiv:2502.12855 2025.
[71] Zongyu Lin, Yao Tang, Xingcheng Yao, Da Yin, Ziniu Hu, Yizhou Sun, and Kai-Wei Chang. Qlass: Boosting language agent inference via q-guided stepwise search. arXiv preprint arXiv:2502.022 02584, 2025.
[72] Peiyi Wang. Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepheris Verify and reinforce lims step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9426-9439, 2024.
[73] Muhammad Khalifa, Rishabh Agarwal, Lajanugen Logeswaran, Jaekyeom Kim, Hao Peng, Moontae Lee, Honglak Lee, and Lu Wang. Process reward models that think. arXiv preprint (tfi:25004168285 2025.
[74] Levente Koesis and Csaba Szepesvári. Bandit based monte-carlo planning. In European conference on machine learning, pages 282-293 Springer, 2006.
[75] Jiajie Jin, Yutao Zhu, Xinyu Yang, Chenghao Zhang, and Zhicheng Dou. Flashrag: A modular toolkit for efficient retrieval-augmented generation research. CoRR, abs/2405. 13576, 2024.
15

---

Checklist
In this section, we answer to the questions required by checklist stated httph::t///// htttps:/nneurpp..cmm publ:t//guidee/ppperrcccsiiiill as follows:
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
Our claims in the abstract and introduction match with the main contribution.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
16

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer[[NA]
Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification:
We provide the experimental details in Appendix
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- spen-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
17

---

reproducible, they 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
We provide the dataset link and code link in Appendix
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr cc/
public/guides
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/esoo//uuuieeeeoeeuubibiiiii 7/nips cc/ wtc/publicc// geguidgg//////eeeeui Codeeobmiiiie SanSSuubissss sorbbbisssiooi ssisnnoicccyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the experimental settings and details in Section more details are reported in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification:
We report the statistical significance in table 2
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
18

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We report the experimental compute resources in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of
Answer: [Yes]
Justification:
The research conforms with NeurlPS Code of Ethics
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Appendix [
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cite all code, data, and models in paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
20

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We release our dataset and introduce the details in Section 2.2.3
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cood/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
21

---

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy for what should or should not be described.
Appendix Overview
The appendix includes the following sections:
Appendix A
Details the MCTS tree construction process using process-level rewards. This serves an extended explanation to Section 2.2.2.
Appendix B Illustrates the example of LLM evaluation for process-level simulation. This serves an extended example of MCTS node expansion Section 2.2.2]
Appendix Compares the reasoning response between GeasonRAG(bart and ReasonRAG(PRLL). This serves as an extended example for Section 3.4
Appendix Provides additional details the evaluation setup. This serves as a supplement to Section 3.1
Appendix Provide more details of the implementation. This serves as a supplement to Sec-
3.0
Appendix
Details the prompt design for Agentic RAG. This serves supplement to Section 2.4
Appendix Details the licensing terms and conditions governing the use and distribution of the proposed datasets.
Appendix H Discusses the limitations and constraints of the proposed approach.
Appendix
Evaluates the potential societal implications and ethical considerations of the research.
A
Monte Carlo Tree Search for Process-level Reward
Formally, for each RAG intermediate process, its corresponding state encompasses the original question z, the preceding thoughts V<i. and stage E (Reasoning, Grounding, Terminal]. The stage indicates the current decision mode within the reasoning process. In the Reasoning stage, the LLM autonomously decides whether generate chosen, it triggers a call to the external search engine, and the retrieved documents are added to the context in the next step. If the model opts to generate into the Terminal stage. In the Grounding stage, the model extracts relevant evidence spans from the retrieved documents based on the most recent query. After extracting evidence, the state transitions back to the Reasoning stage, enabling further iterative reasoning.
query directly produce answer. If query
instead, the process transitions
Based the state, the policy of the next action is defined
( Jx., Vesi Pange),
if stage is Reasoning otherwise
<aass) LLM((a
(4)
T Vess docs, Pauge);
Consequently, the state transition can be represented as 81+1 concatenate(ss, oz). Each node the tree contains the following information: (s), ((s), Stage(s)) where N(s) denotes the number of times state has been visited, Q(s) represents current intermediate annotation collected through the Monte Carlo method, with values in the range (0, 1]. With the tree structure defined, the MCTS begins from the root node and constructs the tree by iteratively performing three key operations: selection, expansion, and backpropagation.
22

---

This step aims to select nodes that balance the search quality and exploration degree. The node selection starts from the root node, and iteratively selects the child nodes based on their state value Q and visiting frequency N. These variable are refined during the search strategy, detailed in the backpropagation section. To effectively trade off between exploring unvisited nodes and exploiting nodes with higher state value, we iteratively search for the next node using UCT score [74]. The state selected according to the following formula:
Selection:
( argmax Echilddeents) (cchilddentss)(((() Cuot 1 ((s))
(5)
where Cuct is a trade-off parameter to control the exploration degree. The algorithm begins by exploring unvisited states and progressively favors nodes with higher Q-values and fewer visits.
Expansion: Given a selected node that does not reach a terminal state and maximum child node limit, the expansion step proceeds with a single step of RAG reasoning based on Equation (4) and initializes a new child node with the generated response. Following response generation, the simulation step iteratively reasons until a final answer is reached, serving as the basis for initializing the reward of the created node. However, simulating RAG leads to inefficiency due to its need for iterative LLM reasoning and retrieval. To address this challenge, the correctness of the intermediate reasoning process is evaluated by LLM judgment based on the intermediate process against the golden answer, outputting a correctness value [0, 1]. as defined Equation (6). This approach avoids üime-consuming simulations, providing an efficient evaluation for exploring the complex RAG reasoning space.
e To (- [x,, V1:1, answer, Peval)
Backpropagation: Following node expansion, the node statistics are iteratively updated from the current node back to the root node. The correctness score Q(s;) is updated based on Eq. ( The visit count for the traversed nodes is incremented:
(s) Ns)) +1
Vs path from 8; root
B Process Evaluation
The three cases illustrated in Figures 8 10 demonstrate the effectiveness of ReasonRAG in supporting stepwise reasoning over retrieved content. the partially correct case, agent retrieves relevant evidence but fails to cover all required entities, leading to incomplete inference. In the incorrect case, the agent misidentifies directors, resulting in a reasoning chain based on factual errors. In contrast, the fully correct case shows ReasonRAG's ability to decompose the question into sequential sub-queries, accurately retrieve biographical information, and synthesize the correct answer through structured comparison. Although minor imperfections exist in the query formulation, the reasoning trace demonstrates clear progression, factual grounding, and answer justification. These cases collectively highlight ReasonRAG's advantage in aligning sub-question formulation with retrieval intent and maintaining semantic coherence across reasoning steps, leading to improved robustness and interpretability in complex multi-hop scenarios.
C Case Study
Figure II shows an example for agentic RAG response from ReasonRAG, Before the policy optimiza- tion, LLMs fails to generate the appropriate query and mislead by irrelevant information. In contrast, the process-supervised RL empower LLMs to autonomously invoke query generation, evidence extraction, and answer generation.
23

---

Case of LLM judge partial correct process
Question
'Which ch film has the director died later, I1 Coraggio or Shark Monroe?"
Golden Answers
"I1 Coraggio."
Agent Thoughts:
So the next query is <query> What is the release year of I1 Coraggio and directed it? </query> Based on query, the relevant evidence <evidence>II Coraggio is a 1955 Italian comedy film directed by Domenico Paolella. </evidence>
LLM judge
The agent's reasoning severely incomplete and structurally flawed. It begins with partial query that only addresses II Coraggio but completely ignores Shark Monroe, (Omitted for brevity)
Moreover, the query does not directly request the death dates of the directors, which is the core requirement (Omilted for brevity)
While the evidence retrieved correct and related to Coraggio, it insufficient (Omitted for brevity)
So the score is 15.
Figure 8: An example of agent partial correctness in a RAG process.
Case of LLM judge incorrect process
Question
"Which film has the director died later, Il Coraggio or Shark Monroe?"
Golden Answers
"Il Coraggio."
Agent Thoughts:
So the query <query>When did the directors Il Coraggio and Shark Monroe die?</query> Based on the query, the relevant evidence is: The Liberators), died on 9 March 2014. Umberto Lenzi, the director of Shark Monroe (also known as Man from the Deep River), on October 2017.</evidene
Tevidence>Tomino Ricci, the director of Coraggio (also known
LLM judge
agent's reasoning shows a series of critical errors. First, query itself flawed: misattributes directors to both films. Il Coraggio (1955) was directed by Domenico Paolella, not Tonino Ricci. (Omitted for brevity)
This fundamental misidentification invalidates the entire reasoning chain. The agent is comparing correct individuals, rendering final answer baseless (Omitted for brevity)
Given these severe and foundational flaws in both retrieval and inference,OOmited for brevity)
So the score
Figure 9: An example of agent incorrectness in RAG process.

---

Case of LLM judge complete correct process
Question
"Which film
director died later, I1 Coraggio or Shark Monroe?"
Golden Answers
"I1 Coraggio."
Agent Thoughts:
So the next query is <query>What is release year I1 Coraggio and who directed it?</query> Based on the query, the relevant evidence is <evidence>II Coraggio is a 1955 Italian comedy film directed by Domenico Boolella./Kevdennn Shark Monroe
So the next query is <query>Who directed when was the director's death?</query> Based on query, the relevant evidence is <evidence>Willlan S. Hart directed Shark Monroe, which was released in 1918. William S. Hart died on December 3, 1945. / the next query is <query>When did Domenico Paolella die?</query> Based the query, the relevant evidence is <evidence>Domenicc Paolella on October 2002. </evidence>$o the answer is <answer>Il Coraggio</answer.
LLM judge
The agent's reasoning process is largely correct in structure and intent: it decomposes the question into sub-queries that retrieve the directors of both films, their dates of death, and uses this information to compare who died later. This stepwise reasoning is sound and correctly arrives at the golden answer: Il Coraggio. However, under strict scrutiny, several minor flaws emerge:
1. Imprecision in Initial Querying (Omitted for brevity)
2. Redundancy( Omitted for brevity)
Lack of Justification in the Final Step(Omittedd for hrevity)
Despite these imperfections, the agent correctly disambiguates the films and retrieves accurate information. (Omitted for brevity)
So the score is 88.
Figure 10: An example of agent complete correctness in RAG process.
User
Cowettin:: Which film diwector
Dueetoo: Which direcor died later, Conegio or
died later Corogglo Shani Monioc?)
LLM
So the noxt query squery What lo the release year of Coroggio and who diretedd Piiiuuery>
When droctors Coraggio and
<ovide directad Domennaa sadiadaa--wwadderaa
Basod
mot query siguery Who dircceed Shark
diector's Pxlquarys
<think- Tonino Ricol the dirocor Coroogla
frooed query, the meleorri mxidence rauiilenan
dinected Stark Monmo William Hart Decomber ceddoooe
Treil Esenerne)
jos
director Ocaster -ih
squery-WWeen When de Domenico Pooleti de?oluuery
Karowerr Unberlo
than Tomin
Bused the relevent evidence suvidenour Bamenioo Fuckille clud on Qctober 2002
B82 coneevers
answer sanswor- Corocgio -iansseee--nsverr
Search R1
(b) ReasonRAG
Figure 11: Case Study on 2WikiMultihopQA dataset.
25

---

D Evaluation Dataset Details
Evaluation Datasets. We use the process-level annotated questions (4,603 examples) as the training set for our policy optimization. These include 704 questions from PopQA, 2,843 from Ho,potQA, and 1,056 from 2WikiMuliHooQAA For evaluation, we use the remaining unlabeled samples from PopQA as the test set, and we adopt the official development splits of HotpotQA and 2WikiMuliHopQA as test sets for multi-hop reasoning evaluation. Table 4 each source. These datasets vary in their design focus and reasoning requirements. HotpotQA and 2WikiMuliHopQAA are constructed to evaluate multi-hop reasoning capabilities, where answering question requires combining information from multiple passages. HotpotQA includes sentence-level supporting facts and diverse question types, such as bridge and comparison questions. 2WikiMulti- HopQA ensures genuine multi-step inference by leveraging structured knowledge from Wikidata and constructing explicit reasoning paths. PopQA, in contrast, is an open-domain QA dataset designed to probe factual recall in large language models. It focuses from high-frequency popular facts to long-tail, less commonly known information. By combining these datasets, we cover diverse set of reasoning challenges, including factual retrieval, multi-hop inference, and process-level supervision.
summarizes the training and test set sizes for
wide spectrum of factual knowledge,
Dataset Source
Train Set Size
Test Set Size
PopQA
704
11,267
HotpotQA 2WikiMultHHoPOA
2,843
7,405
1,056
12,576
bamboogle
125
musique
2,417
Total
4,603
33,790
Table 4: Number of examples in the training and test sets for each dataset. Process-level annotations used for training; test include remaining PopQA examples and official development splits of other datasets.
Evaluation Details. To evaluate model performance on question answering tasks, we adopt two standard metries: Exact Match (EM) and F1 score.
Exact Match (EM) measures the percentage of predictions that exactly match any of the reference answers. Formally,
N
EM
pred
gold
(8)
1=1
where is the number examples, and 8(a, b) if b, otherwise O.
F1 score computes the token-level overlap between the predicted answer and the ground-truth answer. Let 7.ped and 7.gild denote the sets of tokens in the predicted and gold answers, respectively:
sold
Precision;
(9)
[T.rod Toood
Recall,
(10)
|7,50ld
1
2. Precision; Recall,
F
(11)
N Precision; + Rocall,
We follow the official evaluation metrics implementation provided by the FlashRAG toolkit 75
E
Implementation Details
We summarize all artifacts (datasets, models, baselines, external knowledge base, etc) used in our experiments and their resource links in Table
26

---

Name
Purpose
Artifact URL
http:///wgggaaaaa//////////////t tree/nain/poge
PopQA
Eval Dataset
http:///gggagaaaccccr/aaaaaaataa http:///RR datasets/
HotpotQA
Eval Dataset
tree aeee/main/hooce
hong/main/hotpoqss
hitps: wwwggigggggccecoooo.
ZWikiMultHHoQAA
Eval Dataset
aree/main// wttpa//////iiimmlliiiiomm tilopee
http:/aattt..........aaaaaaaaaaa http://IIOO-NLI//PPPaabbAAA datasets/
Bamboogle
Eval Dataset
http//mann// banboogle
https: Sttps://hugiiggfae..
http:///aaacccccaaaaasssss////// NLPIR/PlasbRRA / datasets/
MuSiQue
Fvall Dataset
tree Stee/main/muskae
RAG-ProCuide
Train Dataset
BGE
Retriever
https:
en-e--envv1..
nttpa://archin httpa///archte..rr//ooooo 2ttpoommmooooo/o0011111111122220
Wikidump 2018
Knowledge Source
pages
Qwent.5-7B-Instruct
Backbone Model
http:://uugging(ssccc//// Qwee///eww/weeeeeeeeee Imm-trrruee
Adaptive-RAG
Baseline
t6_x1_
classifier
Self-RAG
Baseline
https: httpas//huuguugaaacccc//aaaa/ar/
1lana2_ 13b
AutoRAG
Baseline
https: http://///uuuuuttoooooo///////rr RAG Llama 88 Instruct
Search-R1
Baseline
attpa:/ tttpar//uggiggtacc.cco
http://ace.oeeeeeeeeoooooooooo. hotpoog.
train-quen2.66 -tt
ReazenRAG
Our Method
nttpa:// http:///nnnmmmuuuu.u h
Table 5: Resource links of artifacts used in our experiments.
E.1
Implementation Details of ReasonRAG
Following the setup in the FlashRAG toolkit, we use Wikidump 2018 as our knowledge source. To ensure retrieval quality, augment corpus by incorporating relevant content from the PopQA, HotpotQA, and 2WikiMuliHopQA datasets. All datasets are available on Huggingface. Subsequently, employ BGE our retriever, consistently retrieving the top 3 documents. For all methods not requiring fine-tuning, we use Qwen2.5-7B-Instten as our baseline model for inference.
E.2 Implementation Details of Baselines
For baseline implementations, we utilize the FlashRAG 75] reproduction, where several mod- els, such as Naïve Generation, Standard RAG, FLARE, Iter- Retgen, RECOMP, LongLLMMingua, and Gelective-Conter. require no additional parameter configuration. For Self-RAG, we use the checkpoint provided in the Fl reproduction. For AdaptiveRAG, reproduction's 's router and inference using the publicly available checkpoint from Hugging Face. For Search-R1, use the checkpoints for inference.
employ the FlashRAG as the reasoning model. For AutoRAG, we conduct
reproduced qwen2.5-7b 5-bbbae and
F
Prompt Instructions
Our prompts include Reasoning, Grounding in Agentic RAG workflow, and process evaluation prompt for node expansion. No extra prompt design is needed when input new question into LLMs for inference. The prompt details are illustrated Figure
Figure 13 and Figure 14
G License Agreement
The RAG-Proo PrG-Prr is constructed based on popqa. hotpotqa, and 2wikimulthhoqya from FlashRAG dataset [75] All these datasets are using CC-BY-SA-440 license, allowing the modification for research use. For the new constructed dataset RAG-ProGuide including but not limited to the questions and process-level reward, we make them available solely for research purposes. Users are permitted to use, modify, and share these annotations for academic and non-commercial research activities. Any other use, including commercial exploitation, is not permitted without explicit written permission from the authors.
27

---

Reasoning
You question-ansseriigg assistant with access to retrieval tool. Your goal is provide a concise and accurate reasoning process.
Instructions:
Error Reflection: If errors exist in previous thoughts, identify and correct them. Skip this step no errors are present.
Information Sufficiency: Evaluate whether the current information is sufficient to fully and accurately answer the question. additional retrieval is needed, deconstruct the question and generate the next query. Avoid repeating previous queries. If no meaningful new query can generated, explain why and provide an answer based on the current information. * Conciseness: Ensure both queries and answers are concise, using nouns or short phrases whenever possible.
Conclusion:
If generating an answer: "So the answer is <answer> (ormafoooomaavvvaiieee If more retrieval is needed: "So the next query is <query>qqery<<querr>>""
Figure 12: System prompt for Reasoning
Grounding
You information retrieval assistant. Your task is extract relevant evidence from the provided Wikipedia documents based on the latest query.
Instructions:
Identify key terms or concepts in the query. Search the documents for evidence that supports the query. * Response format: If relevant evidence is found, output: Based on the query, the relevant evidence is <eridenceeevdencc//eveeees If no relevant evidence is found, output: <evidenee>None</evidnnee>
Figure 13: System prompt for Evidence Extraction
Process Evaluation
An agent is tasked with answering question using retrieval tool. Critically assess its intermediate reasoning process to determine it leads to the correct answer. Identify flaws, inconsistencies, and mistakes in the thought process. Every imperfection, matter how small, must be acknowledged. Evaluate how effectively answer and the overall accuracy of the response. Ensure the evaluation is extremely harsh, leaving no leniency. Even the answer seems close to correct, do not award full marks to maintain strict grading standards. Assign score between [0, I] based on the severity of flaws and the reasoning's accuracy Icading to the golden answer. Respond briefly and conclude with: So the score is [Score].
reasoning supports the final
Figure 14: System prompt for Process Evaluation
28

---

H
Limitations
We employ process-supervised RL optimize the model policy. In contrast outcome-supervised RL, our approach necessitates exploring process-level actions for fine- tne-grained reward annotation. Consequently, acquiring sufficient data for process-level annotation incurs higher time cost com- pared to outcome supervision during data rollout. Nevertheless, as the training efficiency verifies, our data exhibits superior quality, enabling models to achieve greater performance gains with fewer data samples.
I Societal Impacts
LLMs carry the risk of generating uncontrollable responses. When an LLM retrieves racist or harmful information from search engine, it could be inadvertently led produce similar content. We strongly advise users to employ our agentic RAG framework responsibly by integrating secure search engines knowledge corpora and conducting evaluations within open-source, safe environments and datasets.
29


Reviews:
[{"review_id": "sKL427aN0Z", "reviewer": "Reviewer_ATvT", "summary": "This paper addresses the limitations of outcome supervised RL for RAG by introducing a process supervised framework called ReasonRAG. Unlike prior methods that only reward the final answer’s correctness, ReasonRAG leverages fine grained, stepwise rewards constructed via their Shortest Path Reward Estimation algorithm (SPRE). They also employ Monte Carlo Tree Search to explore and annotate intermediate reasoning trajectories. The resulting dataset (5 k questions, 13 k preference pairs) is called RAG-Proguide which is used to optimize the agentic RAG policy via DPO. Experiments were done on five QA benchmarks  and show that ReasonRAG trained on only 5k queries, outperforms Search-R1 baseline (90 k queries) in both EM and F1 while using less GPU compute", "questions": "1. To prove the generality of the Method it will be interesting if the authors can perform experiment using a simple llama-3.1 or 3.3 model against search-o1 and search-r1 on 2 unseen datasets only. This will strength the papers claims. And take care of another baseline (i.e search-o1) . \n2. Giving the run time cost, speed , searches done and tokens for the inference will significantly help the paper.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths: \n1. The introduction of SPRE to estimate process‐level rewards by penalizing unnecessarily long reasoning paths is a novel contribution that directly addresses sparse and delayed rewards in RL for RAG.\n2. Mixing MCTS and SPRE to label data by exploring and minimizing the cost of human annotation is an interesting approach. \n3. Proposed method is definitely compute efficient compared to Search-R1 in terms of GPU hours and the statistical significance test in the  table 2 is convincing. \n\nWeaknesses: \n1. Experiments use a Qwen2.5-7B-Instruct model only. It is uncertain whether ReasonRAG’s benefits translate to smaller or less capable LLMs. \n2. Although MCTS is used offline for annotation, the inference algorithm still involves iterative reasoning loops which may incur latency in real‐time applications. An analysis of runtime cost per query would strengthen practicality claims.\n3. While the authors claimed that they used 2 OOD datasets (unseen) it is not very convincing see the gains are marginal over the search-r1 in those 2 datasets (Bamboogle, MusiQue). And No recent baselines like search-O1. \n4. The base & SFT performance of the ReasonRAG is closer or same as standard RAG (table-3 row 1-2 and table 2 zero shot section) this requires further analysis on what exactly is happening.", "was_revised": true, "final_justification": "1. experiments included a smaller LLM to show generatily\n2. The inference table actually shows that ReasonRAG is almost as efficient as other baselines while being far better in terms of EM and F1 scores. This gives the confidence that the method is really good compared to the baselines.\n3. Overall rebuttal felt very convincing.", "rebuttal": "Dear Reviewer ATvT,\n\nWe sincerely thank you for your thoughtful review and valuable feedback on our work. Below, we address each of your concerns with additional experimental evidence and clarifications.\n\n## **1. Training on more LLM variant: Qwen2.5-3B**\n\n> Experiments use a Qwen2.5-7B-Instruct model only. It is uncertain whether ReasonRAG’s benefits translate to smaller or less capable LLMs.\n\n> To prove the generality of the Method it will be interesting if the authors can perform experiment using a simple llama-3.1 or 3.3 model against search-o1 and search-r1 on 2 unseen datasets only.\n\nThank you for the valuable comment. We conducted additional experiments by re-implementing all baselines using **Qwen2.5-3B-Instruct** as the backbone. The results are reported below:\n\n| Method | PopQA EM | PopQA F1 | HotpotQA EM | HotpotQA F1 | 2WikiMulti EM | 2WikiMulti F1 | Bamboogle EM | Bamboogle F1 | MuSiQue EM | MuSiQue F1 | Avg. EM | Avg. F1 |\n|---------------------|----------|----------|--------------|--------------|----------------|----------------|----------------|----------------|--------------|--------------|---------|---------|\n| Naïve Generation | 10.8| 12.6| 14.9 | 16.7 | 10.0 | 15.3 | 2.4 | 14.9 | 2.0| 4.6 | 8.0| 12.8 |\n| Standard RAG | 32.4| 36.6| 25.6 | 33.6 | 22.0 | 27.5 | 4.0 | 12.0 | 3.5| 9.3 | 17.5 | 23.8 |\n| Search-O1 | 21.6| 25.9| 15.1 | 25.0 | 14.7 | 22.7 | 6.4 | 14.0 | 2.5| 8.0 | 12.1 | 19.1 |\n| Search-R1 (90k)| 30.0| 35.0| 30.4 | 36.0 | 26.0 | 31.8 | 12.5 | 19.0 | 7.6| 13.0| 21.3 | 27.0 |\n| **ReasonRAG (5k)** | **32.9** | **36.8** | **30.0**| **41.3**| **26.6** | **32.1** | **13.6*** | **23.7*** | _6.9_ | **13.2** | **22.0** | **29.42** |\n\n### **1.1 Key findings:**\n(1) **​ReasonRAG remains effective when applied to smaller language models**​, achieving strong performance with Qwen2.5-3B-Instruct and outperforming baseline methods across most datasets.\n(2) **The performance gains are especially notable on multi-hop QA tasks** such as HotpotQA and 2WikiMultihopQA, suggesting that ReasonRAG is particularly effective in complex reasoning scenarios.\n\n## **2. Inference Efficiency vs. Efficacy**\n\n> Although MCTS is used offline for annotation, the inference algorithm still involves iterative reasoning loops which may incur latency in real-time applications. An analysis of runtime cost per query would strengthen practicality claims.\n\n> Giving the run time cost, speed , searches done and tokens for the inference will significantly help the paper.\n\nThank you for your suggestion about runtime cost analysis. We have added a runtime comparison across methods based on 1,000 queries from 2WikiMultihopqa dataset. All experiments use the Qwen2.5-7B-Instruct model with the vLLM inference engine for efficient batched decoding, and baseline implementations follow the open-source FlashRAG framework:\n\n| Method               | Time (s) | Avg. Tokens | Avg. Retrievals | Avg. EM  | Avg. F1  |\n| -------------------- | -------- | ----------- | --------------- | -------- | -------- |\n| Naïve Generation     | 2.7      | 54          | 0               | 11.5     | 19.4     |\n| Standard RAG         | 29       | 763         | 1.0             | 24.3     | 32.0     |\n| FLARE                | 42       | 763         | 5.0             | 15.3     | 21.9     |\n| AdaptiveRAG          | 682      | 3243        | 4.0             | 23.0     | 31.2     |\n| Iter-RetGen          | 103      | 2136        | 3.0             | 25.4     | 33.3     |\n| IRCoT                | 732      | 4379        | 4.8             | 22.6     | 30.7     |\n| RECOMP               | 31       | 807         | 1.0             | 26.9     | 34.2     |\n| LongLLMLingua        | 31       | 823         | 1.0             | 26.8     | 34.2     |\n| Selective-Context    | 32       | 834         | 1.0             | 19.2     | 27.0     |\n| AutoRAG              | 130      | 2563        | 4.2             | 29.5     | 36.9     |\n| Search-R1            | 89       | 1683        | 2.8             | 32.8     | 40.7     |\n| Search-O1            | 115      | 2432        | 4.4             | 22.2     | 31.96    |\n| **ReasonRAG (Ours)** | **110**  | **2304**    | **3.8**         | **34.4** | **42.3** |\n(Note that all methods use the vllm inference engine.)\n\n### **2.1 Key findings:**\nReasonRAG demonstrates the best overall performance (Avg. EM/F1) among all evaluated methods, while maintaining lower latency than baselines with comparable or higher accuracy. \n\n## **3. OOD Generation**\n> While the authors claimed that they used 2 OOD datasets (unseen) it is not very convincing see the gains are marginal over the search-r1 in those 2 datasets (Bamboogle, MusiQue). \n\nThank you for your valuable feedback. We agree that gains on OOD datasets are an important indicator of generalization. While the performance improvements on *Bamboogle* and ​*MusiQue* over Search-R1 are relatively smaller than on in-domain datasets, it is worth noting that ReasonRAG is trained with only ​**5k queries**​, whereas Search-R1 is trained on ​**90k queries**​. Despite this 18× smaller training set, ReasonRAG achieves ​**stronger overall performance**​, including notable improvements on Bamboogle and competitive results on MusiQue, demonstrating both generalization and data efficiency​.\n\n## **4. Results of Additional Baseline: Search-O1**\n> And No recent baselines like search-O1. \n\n> Q1: Take care of another baseline (i.e search-o1).\n\nWe have included the performance of ​**Search-O1**​ as shown in the table below:\n\n| Method | PopQA EM | PopQA F1 | HotpotQA EM | HotpotQA F1 | 2Wiki EM | 2Wiki F1 | Bamboogle EM | Bamboogle F1 | Musique EM | Musique F1 | Avg. EM | Avg. F1 |\n|--------------------|----------|----------|-------------|-------------|----------|----------|---------------|----------------|------------|-------------|---------|---------|\n| Search-O1| 33.2| 40.3| 24.8 | 38.1 | 16.4| 27.1| 30.4| 40.6 | 6.3 | 13.7 | 22.2 | 31.96 |\n| AutoRAG (10k) | 38.6| 44.1| 33.3 | 43.7 | 39.5| 46.1| 24.8| 32.2 | 11.3 | 18.3 | 29.5 | 36.9 |\n| Search-R1 (90k) | 39.7| 44.8| 37.0 | 47.0 | 41.4| 48.0| 32.0| 43.8 | **14.6** | 19.9 | 32.8 | 40.7 |\n| **ReasonRAG (5k)** | **41.5** | **46.2** | **38.4** | **48.9** | **43.6** | **50.4** | **36.0** | **45.5** | 12.8 | **20.6** | **34.4**| **42.3**|\n\n### **4.1 Key findings:**\nReasonRAG outperforms both Search-R1 and Search-O1 across nearly all datasets and metrics, further validating the effectiveness and robustness of our approach.\n\n\n## **5. Clarification of ReasonRAG Base and SFT**\n> The base & SFT performance of the ReasonRAG is closer or same as standard RAG (table-3 row 1-2 and table 2 zero shot section) this requires further analysis on what exactly is happening.\n\nThank you for your insightful question. To clarify the differences, we present the comparison results below:\n\n| Method | PopQA EM | PopQA F1 | HotpotQA EM | HotpotQA F1 | 2WikiMulti EM | 2WikiMulti F1 | Bamboogle EM | Bamboogle F1 | MuSiQue EM| MuSiQue F1| Avg. EM | Avg. F1 |\n| ------------------------------ | ---------------- | ---------------- | ---------------- | ---------------- | ---------------- | ---------------- | ---------------- | ---------------- | ---------------- | ---------------- | ---------------- | ---------------- |\n| Standard RAG | 38.4 | 44.7 | 29.3 | 39.9 | 29.4 | 36.3 | 17.6 | 24.1 | 6.7 | 15.1 | 24.3 | 32.0 |\n| ReasonRAG (Base) | 35.6 | 42.7 | 23.7 | 38.2 | 15.2 | 28.9 | 28.0 | 38.7 | 7.7 | 15.4 | 22.0 | 32.8 |\n| ReasonRAG (SFT) | 31.6 | 37.4 | 26.8 | 38.7 | 35.1 | 40.9 | 17.6 | 27.3 | 8.6 | 15.5 | 23.9 | 32.0 |\n| **ReasonRAG (PRL)** | **41.5** | **46.2** | **38.4** | **48.9** | **43.6** | **50.4** | **36.0** | **45.5** | **12.8** | **20.6** | **34.5** | **42.3** |\n\n**Why ReasonRAG-Base falls short**: Unlike standard RAG which passively uses retrieved documents, ReasonRAG requires active agentic capabilities—planning retrieval strategies, issuing targeted queries, and orchestrating multi-step reasoning. The base model lacks these learned behaviors, making it more similar to untrained agentic baselines like Search-O1 than to standard RAG, resulting in degraded performance across tasks.\n**Why ReasonRAG-SFT fails to generalize**: SFT training on high-reward trajectories from PopQA, HotpotQA, and 2WikiMultihopQA creates task-specific overfitting. While it improves performance on seen datasets (HotpotQA, 2WikiMultihopQA), it fails on unseen tasks like Bamboogle and MuSiQue. This aligns with recent findings that \"SFT memorizes, RL generalizes\" [1]—the model learns surface patterns rather than transferable reasoning strategies.\n**Why ReasonRAG-PRL succeeds**: PRL leverages both positive and negative feedback with fine-grained reward signals based on model preferences, encouraging flexible reasoning behaviors rather than memorized patterns. This approach enables the model to develop generalizable agentic strategies that transfer across diverse QA tasks, achieving consistent improvements (42.3% avg F1) compared to both base (32.8%) and SFT (32.0%) variants.\n\n\n[1] Chu, Tianzhe, et al. \"Sft memorizes, rl generalizes: A comparative study of foundation model post-training.\" arXiv preprint arXiv:2501.17161 (2025)\n\nWe hope these responses address your concerns comprehensively. Feel free to let us know if you have any additional questions, and we are more than happy to provide further clarification on any aspect of our work.\n\nBest regards,\n\nThe Authors"}, {"review_id": "UZTchq4VsY", "reviewer": "Reviewer_WaUS", "summary": "This paper presents an algorithm for computing process-supervised rewards in agentic retrieval-augmented generation (RAG) tasks. The proposed reward formulation is integrated with reinforcement learning to train a RAG agent. Experiments on several public benchmarks demonstrate the effectiveness of the approach.", "questions": "1. What are the computational costs of the different algorithms listed in Tables 2 and 3? A comparison in terms of runtime or sample efficiency would help contextualize the performance gains.\n\n2. How do computational cost and accuracy vary with the number of steps in a sample? Since the current datasets contain at most five steps, it would be valuable to understand how the method scales with longer reasoning trajectories.\n\n3. Have you tried using different base models? It would be helpful to know whether the observed improvements are consistent across models of varying sizes or architectures.", "limitations": "While the results are strong, the use of MCTS introduces additional sampling overhead. It remains unclear whether the performance gains stem primarily from the increased number of generated samples or from the specific reward computation method itself. A more controlled analysis would help clarify this distinction.\n\nMoreover, the datasets used in their experiments involve no more than five steps per episode. It remains unclear how the computational cost and accuracy would scale as the number of steps increases.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\nThe paper is clearly written, and the idea of using Monte Carlo Tree Search (MCTS) to compute process-supervised rewards is interesting. The experimental results are promising and suggest the approach is effective.\n\nWeaknesses:\nWhile the results are strong, the use of MCTS introduces additional sampling overhead. It remains unclear whether the performance gains stem primarily from the increased number of generated samples or from the specific reward computation method itself. A more controlled analysis would help clarify this distinction.\n\nMoreover, the datasets used in their experiments involve no more than five steps per episode. It remains unclear how the computational cost and accuracy would scale as the number of steps increases.", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer WaUS,\n\nWe sincerely thank you for your thoughtful review and valuable feedback on our work. Below, we address each of your concerns with additional experimental evidence and clarifications.\n\n## **1. Gains from Sampling Number or Reward Design?**\n\n> While the results are strong, the use of MCTS introduces additional sampling overhead. It remains unclear whether the performance gains stem primarily from the increased number of generated samples or from the specific reward computation method itself. A more controlled analysis would help clarify this distinction.\n\nThank you for your valuable comment. We conducted two controlled experiments to disentangle **the effects of MCTS iteration count** and **reward design**. The results are summarized below:\n\n| Method   | PopQA EM | PopQA F1 | HotpotQA EM | HotpotQA F1 | 2Wiki EM | 2Wiki F1 | Bamboogle EM | Bamboogle F1 | Musique EM | Musique F1 | Avg. EM  | Avg. F1  |\n| -------- | -------- | -------- | ----------- | ----------- | -------- | -------- | ------------ | ------------ | ---------- | ---------- | -------- | -------- |\n| 32 iters(PRL) | 35.7     | 40.3     | 31.8        | 42.3        | 39.6     | 45.5     | 26.4         | 35.4         | 10.3       | 17.1       | 28.8     | 36.1     |\n| 64 iters(ORL)      | 33.0     | 43.4     | 29.4        | 41.3        | 35.8     | 41.1     | 28.0         | 39.3         | 12.5       | 21.8       | 27.7     | 37.4     |\n| 64 iters(PRL, Ours) | **41.5** | **46.2** | **38.4**    | **48.9**    | **43.6** | **50.4** | **36.0**     | **45.5**     | **12.8**   | **20.6**   | **34.5** | **42.3** |\n\n### **1.1 Key findings**\n(1) Reducing MCTS iterations from 64 to 32 per query results in a clear performance drop, indicating that **sufficient sampling iteration improve data quality**. As also observed in Luo et al. (2024), increasing the number of MCTS samples generally improves reasoning quality, but the marginal gain diminishes beyond a certain point. We adopt 64 iterations as a practical trade-off between performance and efficiency. \n\n(2) We replaced our step-level MCTS reward with a **final-answer-only reward** and trained a variant (**64 iters(ORL)**). The notably lower performance of this variant confirms that **the SPRE reward design yields greater gains**.\n\n[1]. Luo, Liangchen, et al. \"Improve mathematical reasoning in language models by automated process supervision.\" arXiv preprint arXiv:2406.06592 (2024).\n\n## **2. Stepwise Scaling on Cost and Accuracy**\n\n> Moreover, the datasets used in their experiments involve no more than five steps per episode. It remains unclear how the computational cost and accuracy would scale as the number of steps increases.\n\n> How do computational cost and accuracy vary with the number of steps in a sample? Since the current datasets contain at most five steps, it would be valuable to understand how the method scales with longer reasoning trajectories.\n\n\nWe conduct a step-wise cost and accuracy analysis across both single-hop (PopQA) and multi-hop datasets (HotpotQA, 2WikiMultiHopQA). As shown in the table above:\n\n| Step | PopQA EM | PopQA Total Time(s) | HotpotQA EM | HotpotQA Total Time(s) | 2Wiki EM | 2Wiki Total Time(s) |\n| ---- | -------- | ---------------- | ----------- | ------------------- | -------- | ---------------- |\n| 1    | 15.6     | 24             | 15.1        | 28                | 26.4     | 29             |\n| 2    | 35.8     | 51             | 29.6        | 61               | 30.1     | 64             |\n| 3    | 39.8     | 66             | 35.0        | 84               | 39.8     | 81             |\n| 4    | 40.2     | 75            | 37.1        | 105              | 41.5     | 100            |\n| 5    | 40.5     | 80            | 38.3        | 121             | 42.1     | 110            |\n\n### **2.1 Key findings**\nWe observe **consistent performance gains** across both single-hop and multi-hop datasets as the number of reasoning steps increases. For single-hop QA, most gains are achieved in early steps, and further steps bring marginal improvements. Despite longer total inference times, our method mitigates latency by **executing each reasoning step in parallel** across all queries using a batched vLLM engine.\n\n\n## **3. Computational Cost Comparison**\n> What are the computational costs of the different algorithms listed in Tables 2 and 3? A comparison in terms of runtime or sample efficiency would help contextualize the performance gains.\n\nThank you for the helpful suggestion. We have added a runtime and efficiency comparison using 1,000 queries from the 2WikiMultihopQA dataset. The runtime analysis are shown as below:\n\n| Method               | Time (s) | Avg. Tokens | Avg. Retrievals | Avg. EM  | Avg. F1  |\n| -------------------- | -------- | ----------- | --------------- | -------- | -------- |\n| Naïve Generation     | 2.7      | 54          | 0               | 11.5     | 19.4     |\n| Standard RAG         | 29       | 763         | 1.0             | 24.3     | 32.0     |\n| FLARE                | 42       | 763         | 5.0             | 15.3     | 21.9     |\n| AdaptiveRAG          | 682      | 3243        | 4.0             | 23.0     | 31.2     |\n| Iter-RetGen          | 103      | 2136        | 3.0             | 25.4     | 33.3     |\n| IRCoT                | 732      | 4379        | 4.8             | 22.6     | 30.7     |\n| RECOMP               | 31       | 807         | 1.0             | 26.9     | 34.2     |\n| LongLLMLingua        | 31       | 823         | 1.0             | 26.8     | 34.2     |\n| Selective-Context    | 32       | 834         | 1.0             | 19.2     | 27.0     |\n| AutoRAG              | 130      | 2563        | 4.2             | 29.5     | 36.9     |\n| Search-R1            | 89       | 1683        | 2.8             | 32.8     | 40.7     |\n| Search-O1            | 115      | 2432        | 4.4             | 22.2     | 31.96    |\n| **ReasonRAG (Ours)** | **110**  | **2304**    | **3.8**         | **34.4** | **42.3** |\n(Note that all methods use the vllm inference engine.)\n\n### **3.1 Key findings**\nReasonRAG demonstrates the best overall performance (Avg. EM/F1) among all evaluated methods, while maintaining lower latency than baselines with comparable or higher accuracy. \n\n\n## **4. Training on more LLM variant: Qwen2.5-3B**\n\n> Have you tried using different base models? It would be helpful to know whether the observed improvements are consistent across models of varying sizes or architectures.\n\nThank you for the valuable question. We conducted additional experiments with all baselines and our method implemented using **Qwen2.5-3B-Instruct** as the base model. The results are presented below:\n\n| Method             | PopQA EM | PopQA F1 | HotpotQA EM | HotpotQA F1 | 2WikiMulti EM | 2WikiMulti F1 | Bamboogle EM | Bamboogle F1 | MuSiQue EM | MuSiQue F1 | Avg. EM  | Avg. F1  |\n| ------------------ | -------- | -------- | ----------- | ----------- | ------------- | ------------- | ------------ | ------------ | ---------- | ---------- | -------- | -------- |\n| Naïve Generation   | 10.8     | 12.6     | 14.9        | 16.7        | 10.0          | 15.3          | 2.4          | 14.9         | 2.0        | 4.6        | 8.0      | 12.8     |\n| Standard RAG       | 32.4     | 36.6     | 25.6        | 33.6        | 22.0          | 27.5          | 4.0          | 12.0         | 3.5        | 9.3        | 17.5     | 23.8     |\n| Search-O1          | 21.6     | 25.9     | 15.1        | 25.0        | 14.7          | 22.7          | 6.4          | 14.0         | 2.5        | 8.0        | 12.1     | 19.1     |\n| Search-R1 (90k)    | 30.0     | 35.0     | 30.4        | 36.0        | 26.0          | 31.8          | 12.5         | 19.0         | 7.6        | 13.0       | 21.3     | 27.0     |\n| **ReasonRAG (5k)** | **32.9** | **36.8** | **30.0**    | **41.3**    | **26.6**      | **32.1**      | **13.6**     | **23.7**     | *6.9*      | **13.2**   | **22.0** | **29.4** |\n\n### **4.1 Key findings**\nThese results demonstrate that ReasonRAG generalizes well to smaller models, maintaining strong performance across tasks and outperforming several strong baselines. Thank you for your valuable feedback.\n\nWe hope these responses address your concerns comprehensively. Feel free to let us know if you have any additional questions, and we are more than happy to provide further clarification on any aspect of our work.\n\nBest regards,\n\nThe Authors"}, {"review_id": "FkunPWKx6m", "reviewer": "Reviewer_2RBG", "summary": "The paper examines the impact of ORM and PRM on RAG FT by creating an MCTS‑sampled dataset, RAG‑ProGuide, and training the inference model, ReasonRAG, via DPO FT. The proposed workflow demonstrates innovation in exploration diversity, iteration‑count distribution, and performance on downstream datasets. Furthermore, the authors compare ReasonRAG’s downstream metrics after ORM versus PRM FT across multiple datasets, providing preliminary evidence that PRM outperforms ORM and offering valuable insights for RAG researche", "questions": "1. Given that the primary focus of your paper is the comparative impact of PRM versus ORM on LLM FT, it is advisable to enrich the PRM experiments, for example, by assigning distinct weights to procedural correctness and outcome correctness and systematically evaluating how these weightings affect downstream performance metrics.\n\n2. The set of baselines should concentrate on RAG frameworks that have undergone comparable ORM or PRM FT. Incorporating additional relevant baselines, such as RAG‑Gym, would further bolster the robustness and persuasiveness of your evaluation.\n\n3. The axis labels and their semantic descriptions in certain figures (e.g., Figure 3) lack sufficient clarity. A careful revision of these annotations is necessary to ensure that the figure’s intent and interpretation are unambiguous.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 1, "strengths_and_weaknesses": "> Strengths:\n\n1.The paper’s premise—comparing the impact of PRM versus ORM on agentic RAG FT is both novel and of practical significance to the RAG research community.\n\n2.The experimental evaluation is thorough, leveraging a diverse set of datasets and RAG frameworks to assess exploration diversity, iteration‑count distributions, and test‑set EM performance; these comprehensive results effectively demonstrate the advantages of ReasonRAG.\n\n> Weaknesses: \n\n1.From a workflow perspective, an MCTS‑based RAG system has already been proposed in AirRAG (https://arxiv.org/pdf/2501.10053). The main contribution of this paper—using a UCB‑based stepwise selection of positive and negative examples for DPO FT offers limited novelty.\n\n2.In terms of training loss, although the paper’s focus is on comparing PRM and ORM for agentic RAG FT, it applies identical DPO weighting to both procedural decision correctness and final‑output correctness. The manuscript does not explore how varying these weightings affects training outcomes, nor does it investigate which aspect has a greater impact on RAG system performance.\n\n3.Figure 3 contains incorrect annotations, and its caption does not clearly explain the figure’s intended meaning.\n\n4.The paper does not specify which large language models were used at each baseline; if different LLMs were employed, the reliability of the results would be compromised.", "was_revised": true, "final_justification": "The authors have provided the following responses or adjustments:\n1.Clarified the distinctions from other similar works;\n2.Explained the content of the figures and committed to making corrections in the future.\n3.Added experiments with a new model (Qwen2.5-3B) and algorithm (Search-O1).\nThey address some of my concerns, and I decided to slightly increase the score.", "rebuttal": "Dear Reviewer 2RBG,\n\nWe sincerely thank you for your thoughtful review and valuable feedback on our work. Below, we address each of your concerns with additional experimental evidence and clarifications.\n\n## **1. ReasonRAG vs. AirRAG**\n> From a workflow perspective, an MCTS‑based RAG system has already been proposed in AirRAG (https://arxiv.org/pdf/2501.10053). The main contribution of this paper—using a UCB‑based stepwise selection of positive and negative examples for DPO FT offers limited novelty.\n\nThank you for the valuable comment. We agree that MCTS has previously been explored in RAG contexts. Our approach differs fundamentally from AirRAG in goals, methodology, and the role of MCTS. A comparison is summarized below:\n\n| Aspect   | **ReasonRAG (Ours)**     | **AirRAG**      |\n| ---------------------- | -------------------------------------------------------- | --------------------------------------------------------------- |\n| **Training Paradigm** | PRL (DPO with process-level rewards) | SFT     |\n| **Inference** | Agentic RAG (multi-step planning and retrieval)  | MCTS + Self-consistency decoding    |\n| **Role of MCTS** | Offline data generation for training high-quality traces | Online inference-time search and answer selection  |\n| **Research Focus** | Improving **training efficiency** and agentic reasoning | Enhancing **inference-time** reasoning accuracy |\n\nWhile both systems utilize MCTS, **ReasonRAG leverages it during training to enable reward-driven supervision**, whereas **AirRAG applies MCTS at inference to aggregate final answers via self-consistency**. We sincerely appreciate the reviewer for highlighting this connection, as it helped us better position our contributions relative to existing work.\n\n## **2. Step-Level vs. Outcome-Level Supervision**\n> In terms of training loss, although the paper’s focus is on comparing PRM and ORM for agentic RAG FT, it applies identical DPO weighting to both procedural decision correctness and final‑output correctness. The manuscript does not explore how varying these weightings affects training outcomes, nor does it investigate which aspect has a greater impact on RAG system performance.\n\n> Given that the primary focus of your paper is the comparative impact of PRM versus ORM on LLM FT, it is advisable to enrich the PRM experiments, for example, by assigning distinct weights to procedural correctness and outcome correctness and systematically evaluating how these weightings affect downstream performance metrics.\n\nThank you for raising this important question. We agree that identifying the relative contribution of step-level (procedural) and answer-generation(outcome) supervision is critical to understanding what drives performance in agentic RAG systems.\n\nFollowing your valuable suggestion, we conducted a controlled ablation by reducing the amount of training data used for either (1) step-level rewards or (2) answer-generation rewards to 50%. The variant settings are: (1) 50% Step-level Data: 4.5 step-level data + 4k answer-generation data (2) 50% Answer-Generation Data: 9k step-level data + 2k answer-generation data. The results are presented below:\n\n| Variant   | PopQA EM | PopQA F1 | HotpotQA EM | HotpotQA F1 | 2Wiki EM | 2Wiki F1 |\n| --------------------------- | -------- | -------- | ----------- | ----------- | -------- | -------- |\n| 50% Step-level Data | 35.2 | 40.3 | 29.4 | 42.0 | 30.0 | 43.1 |\n| 50% Answer-Generation Data | 38.9 | 44.2 | 35.7 | 47.0 | 39.8 | 48.5 |\n| **Full Data (Original)** | **41.5** | **46.2** | **38.4** | **48.9** | **43.6** | **50.4** |\n\n### **2.1 Key findings**\n\nThe experiment results indicates that **reducing step-level data** causes a more substantial drop. In contrast, cutting final-answer data leads to milder degradation. These results demonstrate that step-level supervision plays a more critical role in agentic RAG training.\n\n## **3. Clarification of Figure 3**\n\n> Figure 3 contains incorrect annotations, and its caption does not clearly explain the figure’s intended meaning.\n\n> The axis labels and their semantic descriptions in certain figures (e.g., Figure 3) lack sufficient clarity. A careful revision of these annotations is necessary to ensure that the figure’s intent and interpretation are unambiguous.\n\n**Response**: Thank you for pointing this out. We apologize for the lack of clarity in the original caption and presentation. While we provided a description in the original manuscript (Page 5, Lines 180–189), we realize it may have been insufficient. We clarify the intention of Figure 3 below:\n\n### **3.1 Pair Type Distribution**\n**Figure 3(a)** shows the distribution of preference pair types. The x-axis denotes combinations of accepted vs. rejected action types (A: answer generation, Q: query generation, E: evidence extraction), and the y-axis indicates the number of preference samples per type. This demonstrates that our dataset spans diverse comparison scenarios across multiple reasoning stages.\n\n### **3.2 Reasoning Step Distribution**\n**Figure 3(b)** illustrates the distribution of reasoning iteration counts per query. The x-axis is the number of MCTS steps, and the y-axis is the number of samples. This reflects the varying difficulty of different multi-hop questions.\n\n### **3.3 Token Counts Distribution**\n**Figure 3(c)** shows the distribution of response token lengths, where the x-axis is the number of tokens in generated responses and the y-axis is the sample count. This confirms our dataset captures both concise and lengthy trajectories.\n\n### **3.4 Reward Gap Distribution**\n**Figure 3(d)** presents the distribution of reward gaps between preferred and rejected paths. The x-axis indicates the reward difference, and the y-axis shows probability density. Larger gaps indicate clearer supervision signals for preference learning.\n\nWe will correct them in the revised version. Thank you again for your careful reading and helpful suggestion.\n\n## **4. Training on more LLM variant: Qwen2.5-3B**\n> The paper does not specify which large language models were used at each baseline; if different LLMs were employed, the reliability of the results would be compromised.\n\nThank you for the valuable comment. As mentioned in **Page 6, Lines 241–243**, all baselines in our main experiments use **Qwen2.5-7B-Instruct** as the backbone model to ensure a fair comparison. Implementation details for each baseline are provided in **Appendix F.2**, and the necessary artifacts are listed in **Appendix Table 5 (Page 20)** for full reproducibility.\nTo further assess the scalability of ReasonRAG, we additionally conducted experiments using a smaller model (**Qwen2.5-3B-Instruct**). Results are shown below:\n\n| Method  | PopQA EM | PopQA F1 | HotpotQA EM | HotpotQA F1 | 2WikiMulti EM | 2WikiMulti F1 | Bamboogle EM | Bamboogle F1 | MuSiQue EM | MuSiQue F1 | Avg. EM | Avg. F1 |\n|---------------------|----------|----------|--------------|--------------|----------------|----------------|----------------|----------------|--------------|--------------|---------|---------|\n| Naïve Generation | 10.8 | 12.6 | 14.9  | 16.7  | 10.0  | 15.3  | 2.4  | 14.9  | 2.0  | 4.6  | 8.0 | 12.8 |\n| Standard RAG | 32.4 | 36.6 | 25.6  | 33.6  | 22.0  | 27.5  | 4.0  | 12.0  | 3.5  | 9.3  | 17.5 | 23.8 |\n| Search-O1  | 21.6 | 25.9 | 15.1  | 25.0  | 14.7  | 22.7  | 6.4  | 14.0  | 2.5  | 8.0  | 12.1 | 19.1 |\n| Search-R1 (90k) | 30.0 | 35.0 | 30.4  | 36.0  | 26.0  | 31.8  | 12.5  | 19.0  | 7.6  | 13.0  | 21.3 | 27.0 |\n| **ReasonRAG (5k)** | **32.9** | **36.8** | **30.0** | **41.3** | **26.6** | **32.1** | **13.6*** | **23.7*** | _6.9_ | **13.2** | **22.0** | **29.42** |\n\n### **4.1 Key findings**\nReasonRAG consistently outperforms baselines even with smaller LLMs, demonstrating that our method generalizes well across model sizes.\n\n## **5. Results of Additional Baseline: Search-O1**\n\n> The set of baselines should concentrate on RAG frameworks that have undergone comparable ORM or PRM FT. Incorporating additional relevant baselines, such as RAG‑Gym, would further bolster the robustness and persuasiveness of your evaluation.\n\n**Response**: Thank you for your suggestion. Following your advice, we have included the Search-O1 baseline from RAG-Gym, and the results are presented below:\n| Method  | PopQA EM | PopQA F1 | HotpotQA EM | HotpotQA F1 | 2Wiki EM | 2Wiki F1 | Bamboogle EM | Bamboogle F1 | Musique EM | Musique F1 | Avg. EM | Avg. F1 |\n|--------------------|----------|----------|-------------|-------------|----------|----------|---------------|----------------|------------|-------------|---------|---------|\n| Search-O1  | 33.2 | 40.3 | 24.8 | 38.1 | 16.4 | 27.1 | 30.4  | 40.6  | 6.3 | 13.7 | 22.2 | 31.96 |\n| Search-R1 (90k) | 39.7 | 44.8 | 37.0 | 47.0 | 41.4 | 48.0 | 32.0  | 43.8  | **14.6** | 19.9 | 32.8 | 40.7 |\n| **ReasonRAG (5k)** | **41.5** | **46.2** | **38.4** | **48.9** | **43.6** | **50.4** | **36.0** | **45.5** | 12.8 | **20.6** | **34.4**| **42.3**|\n\n### **5.1 Key findings**\nThe experimental results demonstrate that **ReasonRAG consistently outperforms both Search-R1 and Search-O1**, further confirming the effectiveness of our approach.\n\nWe hope these responses address your concerns comprehensively. Feel free to let us know if you have any additional questions, and we are more than happy to provide further clarification on any aspect of our work.\n\nBest regards,\n\nThe Authors"}, {"review_id": "zuIcmi6yWM", "reviewer": "Reviewer_aFno", "summary": "The paper investigates reward design for agentic RAG. It introduces ReasonRAG, which uses Monte-Carlo Tree Search (MCTS) plus a “Shortest-Path Reward Estimation” (SPRE) signal to build a 5k-query process-level dataset (RAG-ProGuide). A policy trained with DPO on these preference pairs outperforms the outcome-level baseline Search-R1 on five QA benchmarks while using less data.", "questions": "1. Effect of data selection on sample efficiency: The method achieves strong results using only 5k queries, compared to 90k used in Search-R1. This raises the question: to what extent does the performance stem from careful data selection rather than reward formulation? How were these 5k examples chosen, and could biased or curated selection influence the outcome?\n\n2. Does SPRE provide sufficiently discriminative reward signals for complex queries where multiple reasoning paths may be valid? Are there failure cases where SPRE assigns similar scores to semantically different strategies?\n\n3. How reliable are the preference labels used for DPO? Were the \"better\" and \"worse\" trajectories chosen purely via SPRE scores?", "limitations": "See weakness.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n1. The paper addresses an important problem in agentic retrieval-augmented generation—how to provide effective reward beyond sparse final answer. \n2. The proposed model outperforms a 90k-sample outcome-based baseline across five QA benchmarks with only 5k sample.\n\nWeakness:\n1. Component-wise contribution unclear: The proposed system integrates several components—MCTS, SPRE, and DPO—but their relative importance remains unclear. An ablation study removing or replacing each module (e.g., substituting MCTS with naive sampling) would help clarify which elements are essential to performance.\n\n2. Penalty coefficient \\alpha is under-analyzed: Equation 1 includes a tunable penalty weight \\alpha to encourage shorter paths. However, the paper does not analyze its effect on learning dynamics or final performance. Could the authors provide experiments or sensitivity analysis on different values of \\alpha?\n\n3. Process vs. outcome comparison not controlled: Search-R1 differs in model scale and data quantity. To isolate reward type effects, would it be possible to train a DPO model on outcome-based preferences from the same 5k questions?", "was_revised": true, "final_justification": "This paper proposed reward design for agentic RAG with MCTS. Even though some of the experiments in the first draft are not solid, after discussion, the ablation studies in the author's responses have shown the effectiveness of the reward and training. I think I will keep my positive score.\nSome reviewers were concerned about the MCTS latency and novelty and the author answered these questions but seems not convinced the other author.", "rebuttal": "Dear Reviewer aFno,\n\nWe sincerely thank you for your thoughtful review and valuable feedback on our work. Below, we address each of your concerns with additional experimental evidence and clarifications.\n\n## **1. Ablation Study (MCTS, SPRE, DPO)**\n> Component-wise contribution unclear: The proposed system integrates several components—MCTS, SPRE, and DPO—but their relative importance remains unclear. An ablation study removing or replacing each module (e.g., substituting MCTS with naive sampling) would help clarify which elements are essential to performance.\n\n> Process vs. outcome comparison not controlled: Search-R1 differs in model scale and data quantity. To isolate reward type effects, would it be possible to train a DPO model on outcome-based preferences from the same 5k questions?\n\nThank you for the valuable suggestion. As mentioned in Table 3, we conducted a comprehensive ablation study to evaluate the contributions of MCTS, reward design, and training methods, as detailed below:\n\n### **1.1 Ablation study settings**\n\n| Variant                        | MCTS | Reward         | Training |\n| ------------------------------ | ---- | -------------- | -------- |\n| ReasonRAG (Base)               | ✗    | ✗              | ✗       |\n| ReasonRAG (SFT)                | ✓    | SPRE           | SFT      |\n| ReasonRAG (RL-ORL)             | ✗    | Outcome reward | GRPO     |\n| ReasonRAG (RL-PRL)             | ✓    | SPRE           | DPO      |\n| ReasonRAG (Outcome reward DPO) | ✓    | Outcome reward | DPO      |\n| ReasonRAG (RL-PRL) | ✓    | Outcome reward | DPO      |\n\nFollowing your valuable advice, we added a controlled experiment replacing our process-level reward with a simplified final-answer-based reward (Outcome reward DPO). The results are as follows:\n\n### **1.2 Performance Comparison**\n| Method                          | PopQA EM | PopQA F1 | HotpotQA EM | HotpotQA F1 | 2WikiMulti EM | 2WikiMulti F1 | Bamboogle EM | Bamboogle F1 | MuSiQue EM | MuSiQue F1 | Avg. EM  | Avg. F1  |\n| ------------------------------- | -------- | -------- | ----------- | ----------- | ------------- | ------------- | ------------ | ------------ | ---------- | ---------- | -------- | -------- |\n| ReasonRAG (Base)                | 35.6     | 42.7     | 23.7        | 38.2        | 15.2          | 28.9          | 28.0         | 38.7         | 7.7        | 15.4       | 22.0     | 32.8     |\n| ReasonRAG (SFT)                 | 31.6     | 37.4     | 26.8        | 38.7        | 35.1          | 40.9          | 17.6         | 27.3         | 8.6        | 15.5       | 23.9     | 32.0     |\n| ReasonRAG (RL-ORL) (5k queries) | 23.0     | 30.9     | 28.1        | 32.6        | 32.0          | 43.8          | 17.5         | 24.1         | 5.9        | 13.1       | 21.3     | 28.9     |\n| ReasonRAG (Outcome reward DPO)  | 33.0     | 43.4     | 29.4        | 41.3        | 35.8          | 41.1          | 28.0         | 39.3         | 12.5       | 21.8       | 27.7     | 37.4     |\n| ReasonRAG (RL-PRL)              | **41.5** | **46.2** | **38.4**    | **48.9**    | **43.6**      | **50.4**      | **36.0**     | **45.5**     | **12.8**   | **20.6**   | **34.5** | **42.3** |\n\n### **1.3 Key findings**\n(1) SPRE enhances performance compared to outcome-only reward designs.\n(2) MCTS-guided data generation contributes to stronger reward signals and sample efficiency, which boosts performance especially when combined with process-level reward supervision.\n(3) Base and SFT variants show limited gains, highlighting the importance of both reward quality and training method.\n\n## **2. Penalty Analysis**\n\n> Penalty coefficient \\alpha is under-analyzed: Equation 1 includes a tunable penalty weight \\alpha to encourage shorter paths. However, the paper does not analyze its effect on learning dynamics or final performance. Could the authors provide experiments or sensitivity analysis on different values of \\alpha?\n\nThank you for your comment. Our reward design follows two key principles: (1) reasoning chains with higher accuracy should receive higher rewards, and (2) among chains with similar accuracy, shorter reasoning paths should be rewarded more. To achieve this, we set the penalty coefficient α within the range (0,1], choosing a value close to 1 to effectively balance these objectives. \n\n## **3. Sample Efficiency**\n> Effect of data selection on sample efficiency: The method achieves strong results using only 5k queries, compared to 90k used in Search-R1. This raises the question: to what extent does the performance stem from careful data selection rather than reward formulation? How were these 5k examples chosen, and could biased or curated selection influence the outcome?\n\nThank you for the thoughtful question. As mentioned on Page 5, Lines 165–167, we randomly sampled 3k examples from each of the three datasets, applied MCTS-based filtering, and retained 5k high-quality trajectories. Therefore, the performance gains are not due to curated data, but rather the effectiveness of our automated data generation pipeline. This pipeline adaptively selects diverse, high-reward trajectories through search and reward modeling, enabling strong performance with limited supervision.\n\n## **4. Reward reliability**\n> Does SPRE provide sufficiently discriminative reward signals for complex queries where multiple reasoning paths may be valid? Are there failure cases where SPRE assigns similar scores to semantically different strategies?\n\nThank you for your comment. SPRE provides sufficiently informative reward signals for training. While semantically different reasoning paths may receive similar SPRE scores, we address this by filtering preference pairs based on their reward gap, as described in Figure 5 and Lines 172–175:\n*“To ensure high-quality preference data, we perform post-processing to remove duplicates and uninformative comparisons: (i) we discard preference pairs with identical response sequences, and (ii) pairs with a reward difference less than 0.01. After filtering, the final dataset consists of 4,603 questions and 13,289 distinct preference pairs.\"*.\n\nThis ensures that retained comparisons exhibit clear differences, enhancing the discriminative strength of the supervision.\n\n## **5. Label Annotation**\n> How reliable are the preference labels used for DPO? Were the \"better\" and \"worse\" trajectories chosen purely via SPRE scores?\n\n### **5.1 SPRE Annotation**\nThank you for the comment. Preference labels are assigned **purely based on SPRE scores**, which prioritize both correctness and shorter reasoning paths. To ensure reliability, we filter out preference pairs with reward gaps below 0.01, and report the distribution of reward gaps in Figure 3(d). \n### **5.2 Open-Source Dataset and Manual Inspection**\n**The full dataset is available via a link in Appendix Table 5**, and manual inspection confirms that SPRE-based rewards reliably reflect reasoning quality.\n\nWe hope these responses address your concerns comprehensively. Feel free to let us know if you have any additional questions, and we are more than happy to provide further clarification on any aspect of our work.\n\nBest regards,\n\nThe Authors"}]

Author rebuttal:

Meta-review: The paper introduces ReasonRAG, a novel framework for agentic Retrieval-Augmented Generation (RAG) that leverages process-supervised reinforcement learning (RL) to enhance large language models’ (LLMs) capabilities in complex, multi-step reasoning tasks. The authors propose RAG-ProGUIDE, a high-quality dataset providing fine-grained, process-level rewards for query generation, evidence extraction, and answer synthesis, constructed using Monte Carlo Tree Search (MCTS) and Shortest Path Reward Estimation (SPRE). Experimental results demonstrate that ReasonRAG outperforms existing methods like Search-R1 and traditional RAG systems across five benchmark datasets (PopQA, HotpotQA, 2WikiMultiHopQA, Bamboogle, and MusiQue), achieving superior performance with only 5k training instances compared to 90k for Search-R1. The paper claims that process-level supervision, enabled by SPRE and MCTS, addresses limitations of outcome-based RL, such as sparse rewards and low exploration efficiency, leading to improved generalization and data efficiency.

Reviewers mention several weakness of the paper:

MCTS Novelty: MCTS-based RAG systems (e.g., AirRAG) exist. The authors clarified their unique use of MCTS for offline training data generation.

Initial Ablation Gaps: The original submission lacked detailed component-wise ablations, addressed in the rebuttal with clear experiments.

Inference Latency: Concerns about iterative reasoning loops were mitigated with runtime comparisons showing competitive latency (110s for 1,000 queries).

Penalty Coefficient Analysis: Limited initial analysis of (\alpha) was resolved with a sensitivity study showing performance impacts.

Out-of-Distribution Gains: Marginal improvements on Bamboogle and MusiQue were noted, but justified by the 18× smaller training set.

During rebuttal, the authors have addressed most reviewer concerns through additional experiments and clarifications. Two reviewers didn't engage in the rebuttal. Taking this into account, I think this paper's contribution and interesting topic outweight the weakness. Authors shall include all the new experiments  (e.g., Qwen2.5-3B, (\alpha) analysis).
Decision: Accept (poster)