DeepRAG: Thinking to Retrieve Step by Step for Large Language Models
Authors: Xinyan Guan
Jiali Zeng
Fandong Meng
Chunlei Xin
Yaojie Lu
Hongyu Lin
Xianpei Han
Le Sun
Jie Zhou
Venue: ICLR 2026 Poster (2026)
Keywords: retrieval-augmented generation
adaptive retrieve
Primary area: foundation or frontier models, including LLMs

Abstract: Large Language Models (LLMs) have shown remarkable reasoning capabilities, while their practical applications are limited by severe factual hallucinations due to limitations in the timeliness, accuracy, and comprehensiveness of their parametric knowledge. Meanwhile, enhancing retrieval-augmented generation (RAG) with reasoning remains challenging due to ineffective task decomposition and redundant retrieval, which can introduce noise and degrade response quality. In this paper, we propose DeepRAG, a framework that models retrieval-augmented reasoning as a Markov Decision Process (MDP), enabling reasonable and adaptive retrieval. By iteratively decomposing queries, DeepRAG dynamically determines whether to retrieve external knowledge or rely on parametric reasoning at each step. Experiments show that DeepRAG improves retrieval efficiency and boosts answer accuracy by 25.41%, demonstrating its effectiveness in enhancing retrieval-augmented reasoning.

Published as a conference paper at ICLR 2026
DEEPRAG: THINKING TO RETRIEVE STEP BY STEP FOR LARGE LANGUAGE MODELS
Xinyan Guan Jiali Zeng*1 Fandong Meng', Chunlei Xin Yaojie Lu', Hongyu Lin"!i Xianpei Han', Le Sun .ii Zhou'
CChinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences ?University of Chinese Academy of Sciences
*Pattern Recognition Center, WeChat AI, Tencent Inc, China
(guanxinyan2022, Shunlei2021, hongyu, luyaojie, x sunle jelllee@@sccss ac. cn [lemonzeng, Fandongmeng, wi www.hhoooooooooeeeeceettttt com
ABSTRACT
Large Language Models (LLMs) have shown remarkable reasoning capabilities, while their practical applications are limited by severe factual hallucinations due to limitations in the timeliness, accuracy, and comprehensiveness of their para- metric knowledge. Meanwhile, enhancing retrieval-augne auggmetedd generation (RAG) with reasoning remains challenging due to ineffective task decomposition and redundant retrieval, which introduce noise and degrade response quality. In this paper, we propose DeepRAG, a framework that models vetrieval-augmeente reasoning Markov Decision Process (MDP), enabling reasonable and adaptive retrieval. By iteratively decomposing queries, DeepRAG dynamically determines whether to retrieve external knowledge or rely on parametric reasoning each step. Experiments show that DeepRAG improves retrieval efficiency and boosts answer accuracy by 25.41% %. demonstrating effectiveness in enhancing retrieval- augmented reasoning.
INTRODUCTION
Large Language Models (LLMs) have shown considerable promise in reasoning (Platt et al.) 2024). Nevertheless, their limitations in capacity and capabilities result in significant issues with factual hallucinations, stemming from challenges related to the timeliness, accuracy, and comprehensiveness of their parametric knowledge (Zhang al., 2023 Huang et al., 2023). To mitigate these problems, Netrieval-Augmentt Generation (RAG) has been introduced as promising approach. By incor- porating relevant information from knowledge bases or search engines, RAG enhances the factual accuracy of model responses (Zhao et al. 2024).
However, enhancing RAG with reasoning still poses several challenges (Gao al., 2025). One significant issue is that complex queries often necessitate multi-step decomposition to establish coherent reasoning process (Radhakrishnan et al., 2023 Guan et al. (2044). Iterative retrieval has been proposed solution to continuously update retrieval results, addressing the dynamic information needs that arise during the generation process (Yue et al.) 2024; Wang et al. (025). Despite this, LLMs frequently struggle generate precise and atomic subqueries, which are essential for more effective retrieval and question decomposition (Wu et al.) 2024). From the perspective of RAG, iterative retrieval should ideally generate the next atomic query based on the current question and the available information in an adaptive manner. As illustrated in Figure [ the process flows logically from one step the next. Specifically, the goal of finding each movie's runtime in steps 2-4 is derived from step I's identification of the three titles of the Lord of the Rings series.
Additionally, retrieval is always essential (Jeong al. 2024). Some queries depend external knowledge (steps 2-4), while others can be addressed through the reasoning capabilities of the LLM alone (step 5 requires summarizing previous information). Moreover, LLMs have shown the ability
'Corresponding author. https://ittuhh.o///ipppa////////

---

Published as a conference paper at ICLR 2026
Question:
Lord the Rings?
know Kings trillgy consists of three films: The Fallowship the Ring, The Two Towers, The
However.
exact
[searching...
nintimes.
Afler retriving this mnteemation can
Human Thinking
To find out, fiest need determine individully.
compute the total rumtmm
King.
runtime
178 179
See
2,44
2ue
Whot are the tilos of the musics in The Lord ohe Kings?
What the runtime of The Fellowship of the Ring?
What the total nuntime of the The Lord of the Rings?
Remirval werrative
Final
The Followddip Towess
17% minites
178 20) minutes
558
DeepRAG
Atosie
decisious
Figure I: Correspondence between human thinking processes and DeepRAG. Specifically, retrieval narrative ensures structured workflow by generating subqueries that seek additional information based on previous content, and atomic decisions dynamically determines whether to retrieve external knowledge or rely solely the parametric knowledge for each subquery.
to function knowledge bases in their own right (Petroni et al. 2019) (such as in step 1, where the three movie titles are widely known). Unnecessary retrieval can be redundant and may introduce noise, and degrade the quality of generated responses (Chen al. 2023 Tan et al. 2024 Yu ct al. 2022].
To tackle these issues, we introduce DeepRAG, new framework inspired by how humans search the Internet based on demand. This framework aims to enhance reasoning capabilities in retrieval- augmented generation by modeling the process as Markov Decision Process. DeepRAG incorporates two main components: retrieval narrative and atomic decisions, which together create a strategic and adaptive retrieval system. As depicted in Figure [ the retrieval narrative ensures structured workflow by generating subqueries that seek additional information based on previous content. For each subquery, atomic decisions dynamically determines whether to retrieve external knowledge or rely solely on the LLM's parametric knowledge.
As illustrated in Figure our framework consists of three components: 1) Binary Tree Search, which constructs a binary tree for each subquery related to the question, exploring paths based on parametric knowledge or externa knowledge. leads to the correct final answer with minimal retrieval cost based on Binary Tree Search, enabling the model to learn the pattern of "subquery generation atomic decision intermediate answer". 3) Chain of Calibration, which enhances the LLM's ability to calibrate its internal knowledge, allowing more accurate atomic decisions on when retrieval is necessary. Specifically, we implement two calibration variants tailored for different learning paradigms. For offline calibration, we leverage binary tree search to construct process- supeessssee signals that guide step-wise optimization. For online calibration, we employ outcome-based rewards enable autonomous self-exploration and improvement. By explicitly enhancing the LLM's ability to recognize its own knowledge limits, we train any model in an end-to-end manner, allowing it to dynamically decide when and what to retrieve.
Imitation Learning. which extracts the reasoning process that
We validate the effectiveness of DeepRAG across in-distribution, out-of-distribution, and heteroge- neous knowledge base datasets. Experimental results show that DeepRAG significantly outperforms existing methods, achieving a 25.41% % increase in accuracy while also enhancing retrieval efficiency. Further analysis indicates that DeepRAG demonstrates a stronger correlation between its retrieval decisions and parametric knowledge, suggesting more effective calibration of knowledge boundaries.
2 RELATED WORK
Adaptive
Generation Existing adaptive RAG approaches can be broadly
2024/Jeaa Jeong et al.) 2024) requiring additional linear head training for retrieval decisions, confidence-basee methods (Jiang al. 2023
categorized into three types: classifier-based methods (Cheng et al. [2024]
Su et al. 2024 Dhole 2025) relying heavily on threshold-dependen uncertainty metrics, and LLM- based methods Asai al. 2022 Zhang al. (2024) generating retrieval decisions but often fail to accurately recognize their knowledge boundaries, making it unreliable to delegate retrieval timing decisions to the model, Our method leverages the inherent generative capabilities of LLMS to explore
2

---

Published as a conference paper at ICLR 2026
Retrieved
Parametric
Suaquery
Trajectory with least retrivval A Optimization
Preference pairs
Binary Tree Search
Stage I: Imitation Learning
Question
Question
Stage II: Chain of Calibration
Variont 1:
offline calibration X
AmrX
rewards
Ans
Patanddins
Ans
Variant 2:
Question
Fetricsed
Ans
0.9
Dozuments
Vnceledee
Ans
Figure 2: An overview of DeepRAG, our framework comprises three steps: (1) Binary Tree Search, (2) Imitation Learning, and (3) Chain of Calibration. Given a set of supervised datasets, first use binary tree search to synthesize data for imitation learning, allowing the model to learn effective retrieval patterns. Next, the LLM's awareness of its knowledge boundaries.
either apply offline calibration online calibration to further calibrate
knowledge boundaries in RAG settings. This design maintains the model's native generation abilities while eliminating the need for additional parameters or unreliable uncertainty metrics.
Reasoning in Retrieval-Augmenttr Generation Recent advances in RAG have increasingly
emphasized the integration of reasoning capabilities. Search-ol (Li et al. 2025) incorporates retrieval into inference to build an agentic system, while its application is limited to reasoning models (Chen itt al., 2025). Self-RAG (Asai et al. 2023) and Auto-RAG (Yu et al. 2024) enhance reasoning through automatic data synthesis within retrieval-augmented frameworks, while AirRAG (Feng et al.
2025) combines Monte Carlo Tree Search with self-consistency techniques. These methods, however, often depend heavily on extensive retrieval sampling overhead. More recent developments have explored reinforcement learning to enhance retrieval quality (Jin et al. 2025 Song et al., 2025a Gao al. 2024). while these methods generally overlook retrieval efficiency in their reward function. In contrast, DeepRAG offers flexible, end-to-end solution that enables arbitrary models to retrieve information step by step as needed, based their evolving reasoning process.
Knowledge Boundary LLMs struggle to accurately distinguish between what they know and what they don't know (Yin al. 2023; Kapoor al. [2024a] Yin al. 22024). Additional fine- tuning (Kapoor et al.) (2024b), precise probing (Cheng et al.) (024)), or activation control (Xin et al., 2025) is typically required to calibrate the model's cognition. Our approach explores knowledge boundaries in RAG settings.
THINKING TO RETRIEVE STEP BY STEP
In this section, we introduce DeepRAG, which frames question decomposition, atomic decisions, and answer generation as a Markov Decision Process. Given supervised datasets, we first apply binary tree search to synthesize reasoning data for imitation learning, enabling the model to acquire effective retrieval patterns. We then either employ binary tree search to generate preference data for fine- fine-grained optimization, or outcome- based rewards to calibrate the LLM's awareness of its knowledge boundarics. The following subsections detail each component of DeepRAG.
3.1 OVERVIEW OF THE MDP MODELING
We formalize the step-by-ster reasoning process as a Markov Decision Process (Sutton & Barto 2018), represented by the tuple (S, A, P, R), where S denotes the set of states, A represents the set of actions, P defines the transition dynamics, and R specifies the reward function.
States. etch step the state s S represents the partial solution to the original question. We denote s, = [z, (q1,7)) T1): ..., (+),r.) rt)]. where x is the input question, refers to the i-th subquery, and r; refers the i-th intermediate answer (and any retrieved documents based on q:).
3

---

with the answer Published as a conference paper at ICLR 2026
Actions.
At state 8t, the model selects action a++1 (0141, E A, which consists of two sub-decisions:
Given the partial solution 8t, the model makes a binary decision d++1 E [ontinnue, terminate] to determine whether to proceed with generating the next subquery (++1 or finalize the answer
Termination decision:
For each subquery 9++1- the model decides whether to retrieve external knowledge or rely solely on its parametric knowledge. Formally, this decision is represented as 6.+1 (retrieve, parametric).
Atomic decision:
Transitions. After executing the action 04+1 (7e+1, 64+1) in state 8,, the environment updates the state to S1+1 based on transition dynamics P.
Specifically, if 0 terminate, the process concludes by generating the final answer o, resulting in the terminal state 81+1 = [a, (41,71);
(1,,)),, Otherwise, it generates the next subquery 9e+1- If 6 = retrieve, the model retrieves documents d++1 and generates an intermediate answer 20c+1 for subquery 4++1- Otherwise, it relies on parametric knowledge to generate the intermediate answer. The response rt+1 set as [d,41, i0t+1] (if retrieved) or i0t+1 (if not). The updated state 81411 [r, (g1,r1), .... (9++1,70-1]]
Rewards. The reward function evaluates the state based on answer correctness and retrieval cost, applied only after generating the final answer o. Formally, R($+1 St + [o]) = -C(o) T(s)). where C()) indicates correctness (1 if correct, oo otherwise), and (sxt) represents the total retrieval cost in state s;. Therefore, this reward prioritizes answer correctness while encouraging the model to reduce retrieval cost as much as possible.
3.2
BINARY TREE SEARCH
Answer Format
Building on this formulation, LLM iteratively decomposes given question into subqueries, each derived from pre- viously acquired information. The detailed generation instruction is outlined Appendix A. format in the left.
Question: <Question>>
Follow up: <Subqueryl>>
Let's search question Wikipedia.
Context: <Paragraph Text>
Intermediate answer: <intermediate Answerl>
Follow up: <Subquery2>
Then, we implement a binary tree search to construct rea- soning paths that integrate different retrieval strategies for each subquery. As illustrated in Figure 2 given a question, the model generates the i-th subquery and explores two answering strategies: directly leveraging parametric knowledge (blue node) or retrieving external documents (green node). Therefore, we construct binary tree for each subquery related the given question, exploring paths based on either parametric knowledge or external knowledge.
Intermediate answer: <lntermediate Answerss
So the fimal
<Answer>
3.3 IMITATION LEARNING
We present algorithm that leverages binary trees to identify the optimal reasoning process that leads to the correct final answer while minimizing retrieval costs, corresponding to the highest reward as defined in Section 3.1] Based on the synthesized optimal reasoning data, we fine-tune the model to improve its termination and atomic decisions while enhancing its query decomposition capabilities and generating faithful intermediate answers. We term the resulting model DeepRAG-Imi.
Synthesizing Data As shown in Alg. D we employ priority queue to maintain reasoning trajecto- ries based on their retrieval costs. This allows us efficiently explore potential reasoning paths by iteratively constructing and evaluating them until either finding correct answer exhausting all viable options within specified constraints. For instances where no correct answer can be obtained after exhausting all options, discard them.
Through the synthesis process above, the training dataset obtained contains an adaptive reasoning process, which can be used facilitate arbitrary L.LMs in enhancing the RAG capabilities.
4

---

Published as a conference paper at ICLR 2026
Algorithm Data Construction for Stage
Require: Question z, answer y, language model M, Retriever R. max history length T
Ensure: Optimal reasoning process 8" or null
1: Initialize priority queue PQ {(z)),0)
(trajectory, retrieval count)
while PQ is not empty
3:
(h,r) +- PQ. dequeue)
P Get trajectory with lowest retrieval count
4: - g()
b Suhquery Generation
5:
if S.oohol Answer())) or length(h) > T then
6:
M(h.g) q)
Final answer
if IsEqual y) then return
else
9:
a - M(h, q): (eenquuue(((,,, (g.)))))))
D Direct answer
10:
de d=R(()
D Retrieve document
11:
a (-M(hh.q q, d); PQ. (eeeequuee((h, (d, a 1))
b Retrieved answer
12:
return null
Training Objective We implement a masked loss function for the retrieved documents to prevent the model from learning irrelevant or noisy text that could negatively impact its performance. this way, we hope the model to enhance the ability to decompose subqueries and retrieve them based on demand. For each instance, the loss function is formulated as follows:
C
log (Pr(a[s-11) + Pr(a:|st-1.9;.
(1)
1S
where, de refers nall if there no reieval for eth reasoning step, n refers to the total iteration.
3.4 CHAIN OF CALIBRATION
Building on the markov process in Section 3.1 we identify four key optimization aspects for Deep- RAG: termination and atomic decisions, query decomposition, and intermediate answer generation. Unlike the others, atomic decisions require the model to recognize its own knowledge boundaries to make precise judgments. Therefore, we propose two calibration variants tailored for different learning paradigms: offline calibration and online calibration.
3.4.1 OFFLINE CALIBRATION
Our approach consists of two key components: (1) synthesizing preference data to determine when retrieval is necessary, and (2) fine-tuning the LLM with this data to enhance its ability to make informed atomic decisions. We term the resulting model DeepRAG-RL...f--
Synthesizing Preference Data First, identify optimal path with minimal retrieval based on Alg. [ [using the model trained in Stage I. This provides the optimal atomic decision for each subquery, determining whether retrieval is necessary. From this path, we construct preference pairs for each subquery to indicate the preferred retrieval choice. For example, in Figure the optimal path may suggest answering the first subquery using parametric knowledge while requiring document retrieval for the second. Accordingly, we generate preference pairs favoring parametric knowledge for the first subquery and retrieval for the second. This process enables L.LMs to learn when retrieve external information, thereby improving its ability to maximize the use of parametric knowledge and reducing unnecessary retrievals.
Learning Objective We fine-tune the LLM using a learning objective based on the variant of direct preference optimization (Rafailov et al., 2023). Given the i-th subquery and the state s; = [z. 91, Ti. T:-]] have two distinct intermediate answer r! af and r (d,, af), Based on the process above, we have known which r; is preferred. As a result, the training objective can be formulated as follows:
C -- logo (Blog To(yw 8,, 9:)
Teer si, log To(un 11:
(2)
Tur(y! q.)
where a the logistic function, the hyperparameter / regulates the penalty imposed for the deviations from the base reference model Kref- The terms yo and y refer to the generated snippets for
5

---

Published as a conference paper at ICLR 2026
direct answers and retrieved answers, respectively. Specifically, the snippet "Intermediate Answer:" corresponds to a direct answer, while the snippet "Let's search the question on Wikipedia" corresponds to retrieval-based answers.
3.4.2 ONLINE CALIBRATION
Online calibration aims refine the model's knowledge boundaries through outcome-based rewards and self-exploration mechanisms. We term the resulting model DeepRAG-RLon:
Over-retrieve reward shaping To steer the model towards achieving both accuracy and efficiency in retrieval-auge -uggeeetd reasoning, craft reward function that strikes balance between answer correctness and retrieval efficiency. Inspired by the overlong reward shaping concept in Yu et al. (2025). we introduce over-retrieve reward shaping. Specifically, T. guiding the model to minimize retrieval steps while still producing correct answers. For completed state 8,41, the reward is defined as:
maximum retrieval threshold
0,
format(s1+1) C(o, A format(s:-1) = a a min(T, ((:=1)),,
R
SCOT Cloemat,
(3)
C(o, =1
Here, C(o,y) denotes answer correctness, format(#:41) indicates whether the output format is satisfied, Scorefama is reward for correct formatting, controls the retrieval penalty strength, T is the maximum retrieval budget, and T($+11) represents the actual retrieval count at state $1+1- This design prioritizes correctness, rewards proper formatting, and penalizes excessive retrieval, while capping the penalty at T.
Learning Objectvie We employ GRPO (Guo et al. 2025) as the learning objective in DeepRAG- RLon To mitigate the influence of irrelevant noisy retrieved content, we apply masked loss over the retrieved tokens, ensuring that the model focuses on learning from informative text. Details are shown in Appendix B.3
EXPERIMENT
4.1 BASELINES
We use the following baselines evaluate the performance: CoT (Wei al. 2022 and CoT*, which employ 8-shot examples extracted from the training dataset. The asterisk (+) indicates the model output trained using the
data employed for training the DeepRAG. CoT-Retrieve and Cot-Retrieve* augment the eight examples in the context with retrieved relevant documents based on the query. IterDRAG (Yue et 2024) refers decomposing the question and answer step by step based on in-context learning. AutoRAG (Yu et al., 2024) uses trained models to iteratively decompose questions and retrieve relevant documents for answering. Search-ol (Li et 2025) leverages special tokens to prompt reasoning models to autonomously invoke retrieval as needed.
Search-RI (Jin et al. 2025) is trained using reinforcement learning with exact match accuracy as the reward signal for optimizing retrieval decisions. UAR (Cheng et al., 2024) employs trained classifier to determine when retrieve. FLARE (Jiang et al. (2023) and DRAGIN (Su et al. (024) confidence-baseee method that decide the timing of retrieval based token importance and uncertainty, TAARE (Zhang et al. (024) allows the LLM itself determine when retrieval needed.
RI-Searcher++ (Songe | (222bb) employs composite reward function combining exact match accuracy, format compliance, and group-wise retrieval efficiency for reinforcement learning- based optimization.
4.2 DATASETS AND IMPLEMENTATION DETAILS
We use five open-domain QA datasets for experiments. We treat training datasets as in in-distribution. and unseen ones as suttof-diitributiinn The in-distribution datasets include HotpotQA (Yang et al. 2018), and 2WikMultihoQQA (Ho al. 2020), and the out-of-distribution datasets consist of

---

Published as a conference paper at ICLR 2026
Table The overall experimental results of DeepRAG and other basclines on five benchmarks. The best/second best scores in each dataset are bolded/underlined
in-distribution
ouf-of-distribution
Types
Methods
Horpot QA
2WikiMulthhpQA
PopQA
Web Question
MuSiQue EM
Avg
EM
F1
EM
F1
EM
F1
F1
F1
Llama KB
37.75
28.20
25.33
25.20
40.56
22.97
27.69
CoT-Retreive
34.90
46.85
35 80
43.41
32.80
45.87
22.90
39.22
19,10
28.18
34,90
CoI*
21.80
31.69
25.60
30.89
23.10
25.31
26.80
40.20
4.80
13.85
24.40
CoT *
22.50
32.15
23.70
29.21
38.70
45.64
17.60
29.20
5.70
11.60
25.60
Reasoning
AerDDRAG
23.20
30.95
19.60
24.80
22.70
34.53
15.90
26.79
12.40
17.75
22.86
Auto-RAG
25.80
36.09
23.00
30.09
27.80
42.02
17.40
32,94
19.10
28.33
28.26
Scarch-ol
14.80
24.08
22 22.20
27.10
10.30
13.54
15,30
29 29.60
5.40
11.98
17.43
Search-RI
31.90
42.99
42.30
43.58
27.70
43.
27.31
HLARE
25.80
"32.88"
30.30
37.45
28,80
"40.61"
28.36
40.61
14.50
223,37
30.13
DRAGIN
27.60
38.05
29.10
35.68
22.60
28.53
21.20
38.72
11.80
19.97
27.33
Adaptive
UAR
29.70
40.66
34,80
42,40
33,00
45.95
22.70
39.10
19.10
28.38
33.58
TAARE
30.60
41.43
35.20
42.85
33.20
46.01
23.40
39.56
18.60
27.55
33.84
RI 5
38.90
50.27
49.14
43.57
31.56
35.10
"46.39"
47.20
32.3
35.60
"48.50"
30,00
41.66
2E 90
30,46
39.78
DeepRAG- Rlays
40.70
51.54
48.10
53.25
42.50
47.80
32.70
45.24
22.50
30,40
41.47
DeepRAG MG-RI
37.40
48.91
49.90
55.20
47.20
50.16
30,00
44.67
25.70
34.44
42.36
Qwen 328
CoT
20.60
30.62
24.40
30,94
Retreive
28.60
39.43
27.90
36.73
33,80
45.991
17.20
34.15
12,90
21.98
29.86
Reasoning
IterRRGG
22.90
38.26
19.60
35,70
20.30
27.57
27.80
25.62
Search-ol
34.00
45.64
29.10
35.12
23.10
30.69
17.90
35.11
16.40
25.60
29.27
Search-R1
27.00
36.42
32.30
37.01
19.00
20.16
29.10
43.96
12.50
21.10
27.85
UAR
32.10
42.32
29.00
35,90
33.70
45.75
17.00
33,92
12.80
21.80
30.43
Adaptive
TAARE
31.90
42.40
29.10
35.85
33,70
45.42
12.70
28.70
13.20
22.12
29.51
"1-Scarchere+*
34.40
45.24
45,80
49.81
42,20
45.65
24.20
41.13
21.30
29.72
37.94
DeepRAG-Imi
30.40
39.44
32,00
38.32
37.50
40.72
23.90
38.62
16.50
24.67
32.21
ours
CoopRAG-RLaf?
32.10
41.14
40.40
44.87
40.60
43.19
24.20
38.83
19.50
32.35
35.72
DeepRAG-RLon
39.10
49.51
46.50
44.60
31.20
31.79
41.05
PopQA (Malen et al.) 2022).
WebQuestions (Berant et al. 2013), and MuSiQue (Trieedi et al. 2022).
Furthermore, WebQuestions is built upon Freebase assess model robustness when information may be absent from the knowledge base.
We train our target model on two QA datasets: HotpotQA and 2WikiMultihopQA. For imitation learning, we randomly sample 4,000 examples from each dataset. To enhance the model's question decomposition and context-based generation capabilities, we employ Qwen-2.5-72B to generate the gray (query decomposition) and green nodes (retrieved answers) in Figure and use the target model to generate the blue nodes (parametric answers) for data synthesis. For chain of calibration, we sample an additional 1,000 examples from each dataset. For DeepRAG-RLo,, the hyperparameters are configured as scorefamma = 0.1, a = 0.1, and = 5. The performance evaluated using Exact Match (EM) and F1 score.
4.3 OVERALL RESULTS
Results in Table [ show DeepRAG's superior performance and robustness across different scenarios.
DeepRAG demonstrates superior performance across most datasets via thinking retrieve step by step. Our method consistently outperforms existing approaches across various backbones and model sizes. Compared to reasoning-basse and adaptive RAG baselines, DeepRAG outperforms across all datasets, demonstrating the effectiveness of the structured retrieval narrative and on- demand atomic decisions. Specifically, the limited performance of IterDRAG highlights the necessity of learning both query decomposition and faithful answering. Confidence-basee methods like FLARE struggle determine the optimal retrieval timing due their reliance on unstable, predefined metrics. Moreover, observe that confidence-based methods suffer from instability, their performance is highly sensitive to threshold selection. Mcanwhile, iterative retrieval methods like Auto-RAC -AG often fall into continuous retrieval loops when highly relevant information is found.
DeepRAG exhibits remarkable generalization capabilities and robustness in out-of-distribution settings. DeepRAG-RL, R achieves Fl score improvements of 2.63 and 4.57 PopQA and WebQuestions respectively, even in scenarios where relevant information may be sparse or missing from the knowledge base. Compared to DeepRAG-Ime DeepRAG-RL also exhibits superior
7

---

Published as a conference paper at ICLR 2026
Table 2: WebQuestions retrieval frequency: Avg.
Table 3: Internal knowledge analysis on 2Wiki- MultihopQA across adaptive retrieval methods.
Retrievals and Tiimessc/item).
Avg. Retrievals
Method
F1
Acc
Balanced Ace
MCC
Method
Time
FLARE
0.000
0.718
0,500
0.000
FLARE
28.80
0.00
0.00
0.00
2.58
DRAGIN
0.007
0.709
0.495
-0,045
UAR
0.481
0.756
0,648
0.341
UAR
22.70
0.96
0.95
0.97
0.43
TAARE
0.712
0.078
TAARE
23.40
0.66
0.65
0.66
Iter- DRAG
0.000
0.718
0,500
0.000
BenDRAG
15.90
2.25
2.16
2.27
1.09
Auto-RAG
0.000
0.718
0,500
0.000
Auto RAG
17.40
4.52
3,03
2.35
0.71
R1-Sarccher++
0.627
0.734
0.752
0.458
Searcher++
28.20
0.16
0.09
0.18
0.48
Scarch-RI
0.569
0.633
0,701
0.366
DeepRAG-RLE
Search-RI
DeepRAG-Imi
27.70
32.70
30.00
0.43
0.28
0.51
0.12
0.13
0.75
0.36
0.36
0.68
0.67
0.64
0,50
DeepRAG-RR Ro-pRAG-RLof/
DeepRAG-Imi
0,580
0.621
0.732
0.749
0.709
0,743
0.393
0.451
DeepRAG-RLE
30.00
0.14
0.09
0.16
DeepRAG-RO RLRo
0.676
0.756
0.801
0.543
generalization, with average gains of 2.85 across three out-of-distribbtioo datasets, versus 0.91 for DeepRAG-RLoff-
All DeepRAG models effectively explore knowledge boundaries while minimizing hallucination risks. TAARE often underperforms direct retrieval methods, reflecting a mismatch between the model's s internal knowledge and verbose reasoning. Moreover, aggressive fine-tuning strategies such as CoT* and CoT-Retrieve* degrade performance by forcing the model to absorb knowledge beyond its natural boundaries. In contrast, the data-synthesis approach of DeepRAG-Imi enables the model to initially learn on-demand retrieval, while the subsequent DeepRAG-RLon, and DeepRAG-RR further strengthen its ability to make accurate retrieval decisions and refine knowledge utilization.
ANALYSIS
5.1 RETRIEVAL EFFICIENCY
To evaluate the efficiency of our method, we compare the average number of retrievals on the We- bQuestions dataset and report the average computation time per query. The computation time is measured on an H20"8 machine. As shown in Table 2 We have the following observations: 1) DeepRAG-RL f and DeepRAG-RLos can achieve higher accuracy with relatively lower retrieval costs, attributed to its dynamic usage of internal knowledge. 2) Confidence-basee approaches demon- strate limited robustness across datasets. For instance, neither FLARE nor DRAGIN triggers retrieval under the default confidence threshold in the WebQuestions dataset. 3) Iterative retrieval- based meth- ods typically require numerous retrieval operations. Therefore, efficient adaptive retrieval methods like DeepR AG become crucial for optimizing resource utilization while maintaining performance.
5.2 RELEVANCE TO PARAMETRIC KNOWLEDGE
In this section, we investigate the relationship between retrieval needs and internal knowledge to demonstrate how effectively atomic decisions explores the knowledge boundary.
Ideally, models should initiate retrieval for queries beyond their parametric knowledge while utilizing their existing knowledge for familiar queries. We use CoT results as an indicator of whether the model can answer questions using its parametric knowledge. Then, we analyze whether other adaptive retrieval methods align with this pattern of parametric knowledge utilization.
We report four metrics. Fl score and Accuracy serve as basic performance measures, while balanced accuracy and Matthews Correlation (M))ficient(MMCCC (contributors (025) are employed to account for the class imbalance between sutrieval-require and retrieval retrieval-not-rouuiree cases.
find that: 1) DeepRAG demonstrates superior relevance performance across F1, balanced accuracy, and MCC metrics. This suggests that DeepRAG successfully identifies retrieval necessity by exploring knowledge boundary; 2) While FLARE, DRAGIN, and TAARE exhibit high accuracy scores, their relatively low balanced accuracy and MCC scores suggest they mainly succeed in retrieval-required cases but struggle properly avoid unnecessary retrievals.
As shown in Table
8

---

Published as a conference paper at ICLR 2026
215
Avenge Counz
381
61
191
Figure 3: Statistical distributions of the number of subqueries, reflecting how questions are decomposed (left), and the number of retrieval attempts per query (right).
Figure 4: Average counts of WH-words, nouns, verbs, and conjunctions (and/or) per sub- query.
5.3 QUESTION DECOMPOSITION EFFECTIVENESS
We systematically analyze the effectiveness of question decomposition in retrieval narrative. As shown in Figure we present the distribution of subquery counts and retrieval attempts for different questions. Most questions require 3-5 decomposition steps, while retrieval attempts are primarily concentrated within 0-2 rounds. This demonstrates that DeepRAG effectively decomposes questions while minimizing redundant retrieval.
Moreover, we analyze the average counts of WH-words, nouns, verbs, and conjunctions in subqueries, as shown in Figure DeepRAG decomposes atomic queries with fewer pronouns and conjunctions, indicating its concise and efffective query decomposition strategy.
55
-17.57
Bivep 97
Averago Reuievals
Average Score
2.3
Avenge Retrievals
Average Score
50)
65
47.07
3.50
Bang
Andrag Retricvals
2.11
an
Arerage
15
35
1.57
30
41.13
00.00
30
0.8
21.1
0.6
20
60
0.5
purmnetric
DeeRRAG
retrieve only
DeepRAGi-lnt
ronal
rundom
DeepRAG
all-node
sontenzze ise
Figure 5: Comparative analysis of retrieval strategies: paramet- ric only retrieve only.
Figure 6: Average and trievals on the ablation study for Imitation Learning.
Figure 7: Average and re- trievals on the ablation study for Chain of Calibration.
5.4 DIFFERENT INFERENCE STRATEGY
To gain deep insight into the effectiveness atomic decision, we evaluate DeepRAG's performance under two extreme scenarios: relying solely on internal knowledge (retrieve only) and using retrieval each subquery (parametric only). shown Figure parametric only yields poor performance, while retrieve only achieves relatively higher accuracy but incurs substantial retrieval costs. DeepRAG achieves superior performance by adaptively selecting between internal and external knowledge sources. Moreover, DeepRAG outperforms the retrieve only approach because retrieval can hinder model performance due to long context or irrelevant knowledge certain scenarios.
5.5 ABLATION STUDY
In this section, we conducted experiments to validate the effectiveness DeepRAG's data construc- tion and training process.
For Imitation Learning, we compare our default strategy of selecting paths with minimal retrieval cost against two alternative approaches: maximum retrieval cost (most) and random path selection (random). As shown Figure|6) DeepRAG- achieves lower retrieval costs and higher average performance compared to both the most and random methods.
For Chain of Calibration, we compare our default approach of constructing preferences based on nodes from optimal paths against alternatives: constructing pairs for all nodes and constructing

---

Published as a conference paper at ICLR 2026
sentence-level partial order pairs based retrieval efficiency. As shown in Figure 7 DeepRAG achieves lower retrieval costs while maintaining higher average performance. In contrast, the sentence- level partial order pairs learned incorrect preferences, resulting in over-reliance on internal knowledge and consequently leading to both low retrieval costs and poor performance.
6 CONCLUSION
In this paper, we present DeepRAG to model retrieval-augmmented reasoning as a Markov Decision Process, enabling strategic and adaptive retrieval by decomposing queries into subqueries and retrieval on demand. Specifically, we develop binary tree search method to synthesize data for imitation learning and further chain of calibration to train the model in an end-to-end manner. Experiments across various QA tasks show that DeepRAGi improves retrieval efficiency while improving answer accuracy by 25.4%, demonstrating effectiveness in optimizing retrieval-augmented reasoning.
ETHICS STATEMENT
All datasets and models used in this study are publicly available and have been used accordance with their respective licenses and terms of use.
REPRODUCIBILITY STATEMENT
We have clarified experiment setting in Section 4 and Appendix [B.4 We will upload the code confirm reproducibility, and we promise to open-source the code in the future
LLM USAGE
Large Language Models (LLMs) were used to aid in refining the language and improving the readability limited number of paragraphs in the manuscript.
ACKNOWLEDGMENTS
We sincerely thank the reviewers for their insightful comments and valuable suggestions. This work was supported by Beijing Natural Science Foundation (L243006), Natural Science Foundation of China (No. 62536008, 62476265, 62306303).
REFERENCES
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi, Self-rag: Learning to retrieve, generate, and critique through self-reflection. arXiv preprint arXiv:2310. 10.1000.15111 1151 2023.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods natural language processing, pp. 1533-1544 2013.
Blung-Ting Chen, Fangyuan Xu, Shane Arora, and Eunsol Choi. Understanding retrieval augmentation for formfformooo question answering. arXiv preprint arXiv:2310.121500 2023.
Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng. Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wanxiang Che. Towards reasoning era: A survey of long chain-of- thought for reasoning large language models. arXiv preprint arXiv:2503.09567, 2025.
Qinyuan Cheng, Xiaonan Li, Shimin Li, Qin Zhu, Zhangyue Yin, Yunfan Shao, Linyang Li. Tianxiang Sun, Hang Yan, and Xipeng Qiu. Unified active retrieval for retrieval augmented generation. arXiv preprint arXiv:2406. 12534, 2024.
Wikipedia contributors. Phi coefficient Wikipedia, the free encyclopedia. httpe//iw...gg/wiiiiii..com htw.iiiiiiiiiiiccfciiiiiiii fffie 2025. Accessed: 2025-01-22.
https: //eps://en
Kaustubh D. Dhole. To retrieve or not to retrieve? uncertainty detection for dynamic retrieval 09292
augmented generation, 2025. htttt::://aaxii.or///aaaa
10

---

Published as a conference paper at ICLR 2026
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama herd of models. arXiv preprint vxXi:2247.21188. 2024.
Wenfeng Feng, Chuzhan Hao, Yuewei Zhang, Jingyi Song, and Hao Wang. Airrag: Activating intrinsic reasoning for retrieval augmented generation via tree-based search, 2025, URL htss //arxiv.org/abs/25011 1005
Jingsheng Gao, Linxu Li, Weiyuan Li, Yuzhuo Fu, and Bin Dai. Smartrag: Jointly learn rag-related tasks from the environment feedback. arXiv preprint arXiv:2410.18144 181441 2024.
Yunfan Gao, Yum Xiong, Yijie Zhong. Yuxi Bi, Ming Xue, and Haofen Wang. Synergizing rag and reasoning: A systematic review. arXiv preprint arXiv:25004 2.:::225555.5. 15909, 2025.
Xinyan Guan, Yanjiang Liu, Hongyu Lin, Yaojie Lu, Ben He, Xianpei Han, and Le Sun. Mitigating large language model hallucinations via autonomous knowledge graph- based retrofitting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 18126-18133 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi- hop QA dataset for comprehensive evaluation of reasoning steps. In Donia Scott, Nuria Bel, and Chengqing Zong (eds.), Proceedings of the 28th International Conference on Computa- tional Lingnistics, pp. 6609-6625, Barcelona, Spain (Online), December 2020, International Committee on Computational Linguistics. doi: 1118653/1122200.co/ maai::ooioimmmi.58...... https::/ac..anhholoo..orgg wtw.r///......ooooooooo...o mong-main.5588.
m80n.580. URL
Lei Huang, Weijiang Yu. Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 2023.
Soyeong Jeong, Jinheon Baek, Sukmin Cho, Sung Ju Hwang, and Jong C Park. Adaptive-rag: Learning to adapt retrieval-augmented large language models through question complexity. arXiv preprint arXiv:2403. Stti::22403.1440000 14403, 2024.
Zhengbao Jiang, Frank F. Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yuu Yiming Yang, Jamie Callan, and Graham Neubig. Active retrieval augmented generation. 2023.
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Serean Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training lims to reason and leverage search engines with reinforcement learning, arXiv preprint arXiv:2503.09516 2025.
Sanyam Kapoor, Nate Gruver, Manley Roberts, Katherine Collins, Arka Pal, Umang Bhatt, Adrian Weller, Samuel Dooley, Micah Goldblum, and Andrew Gordon Wilson. Large language models must be taught to know what they don't know. arXiv preprint arXiv:2406.08331, 2024a.
Sanyam Kapoor, Nate Gruver, Manley Roberts, Arka Pal, Samuel Dooley, Micah Goldblum, and Andrew Wilson. Calibration-tuning: Teaching large language models to know what they don't know. In Raúl Vázquez, Hande Celikkanat, Dennis Ulmer, Jörg Tiedemann, Swabha Swayamdipta, Wilker Aziz, Barbara Plank, Joris Baan, and Marie-Catherine de Marneffe (eds.), Proceedings of the 1st Workshop on Uncertainty-Aware NLP (UncertaiNL.P 2024), pp. 1-14, St Julians, Malta, March 2024b. Association for Computational Linguistics. URL https: //tcaaaaaaaaalgyyycooooo org/ 2024. www.rrtatttt.p111177
Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. Search-ol: Agentic search-enhanced large reasoning models. arXiv preprint arXiv:2501.05366, 2025.
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi. and Hannanch Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non non- por-mmratttiic memories. arXiv preprint arXiv:22 12, 105 2022.

---

Published as a conference paper at ICLR 2026
OpenAl. Hello gpt-4o. httees///oeeaa..cco////////l/lls Jamuary-2025).
[Online; accessed 22-
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models knowledge bases? In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2463-2477 Hong Kong, China, November 2019. Association for Computational Linguisties. doi: 10. 10.18653/11/D19. 1250. Utt//tttps://aclanthoo.ogg org/ D19-12507
Aske Plaat, Annie Wong, Suzan Verberne, Joost Broekens, Niki van Stein, and Thomas Back. Reasoning with large language models, a survey. arXiv preprint arXiv:2407.11511 2024.
Ansh Radhakrishnan, Karina Nguyen, Anna Chen, Carol Chen, Carson Denison, Danny Hernandez, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamilè Lukošiūté, et al. Question decomposition improves the faithfulness of mrodel-generatie reasoning. arXiv preprint arXiv:2307. 2023.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728-533 -577722 2023.
Huatong Song, Jinhao Jiang. Yinggian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ri-Rong Wen. RI R1-searcher: Incentivizing the search capability in Ilms via reinforcement learning. arXiv preprint arXiv:2503.05550 2025a.
Huatong Song, Jinhao Jiang, Wenqing Tian, Zhipeng Chen, Yuhuan Wu, Jiahao Zhao, Yingqian Min, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen, R1 R1-searcher++: Incentivizing the dynamic knowledge acquisition of Ilms via reinforcement learning. arXiv preprint arXiv:2505. 17005, 2025b.
Weihang Su, Yichen Tang, Qingyao Ai, Zhijing Wu, and Yiqun Liu. DRAGIN: Dynamic retrieval augmented generation based on the real-time information needs of large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 12991-13013, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/ 2024.acl-long 102. URL httphhttts//caaattt////////////t hwwpolllggoogooga/geooooo 2024. acl -ong. 702/
Richard S. Sutton and Andrew G. Barto. Reinforcemennt Learning: An Introduction, The MIT Press, Cambridge, MA, 2nd edition, 2018 ISBN 978-0-262-003224-- URL https:://m/ tpress. mit
Hexiang Tan, Fei Sun, Wanli Yang, Yuanzhuo Wang, Qi Cao, and Xueqi Cheng. Blinded by generated contexts: How language models merge generated and retrieved contexts for open-domain qa? arXiv preprini arXiv:2401.1191 2024.
Qwen Team. Qwq: Reflect deeply on the boundaries of the unknown, November 2024. URL https://qwen.m. github. io/blog/que qwq- hwtpg-32b-b-- 32bb-reviews
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. musique: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics,
2022.
Liang Wang, Haonan Chen, Nan Yang, Xiaolong Huang, Zhicheng Dou, and Furu Wei. Chain-of- retrieval augmented generation. arXiv preprint arXiv:2501 14342, 2025.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 -44337, 2022.
Zhuofeng Wu, He Bai, Aonan Zhang, Jiatao Gu, VG Vydiswaran, Navdeep Jaitly, and Yizhe Zhang. Divide-or croquerr which part should you distill your 1lm? arXiv preprint arXi::2402.15000 2024.
12

---

Published as a conference paper at ICLR 2026
Chunlei Xin, Shuheng Zhou, Huijia Zhu, Weiqiang Wang, Xuanang Chen, Xinyan Guan, Yaojie Lu, Hongyu Lin, Xianpei Han, and Le Sun. Sparse latents steer retrieval- -uggented generation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 4547-4562, 2025.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang. Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report. arXiv preprint wrXiv:2407.10611 2024.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint 1rXiv:1809.0 09600, 2018.
Xunjian Yin, Xu Zhang, Jie Ruan, and Xiaojun Wan. Benchmarking knowledge boundary for large language model: A different perspective on model evaluation. arXiv preprint 1411v:2402.11493 2024.
Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuanjing Huang. Do large language models know what they don't know? arXiv preprint arXiv:2305./88: /8/53, 2023.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-souree Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025.
Tian Yu, Shaolei Zhang, and Yang Feng. Auto-rag:
Autonomous retrovalleetttteeeeeaaaggemeeee generation for large language models. 2024. URL http://///aarii..oggab//////////
Wenhao Yu, Dan Iter, Shuohang Wang. Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. Generate rather than retrieve: Large language models are strong context generators. arXiv preprint arXiv:2209. 10063, 2022.
Zhenrui Yue, Honglei Zhuang, Aijun Bai, Kai Hui, Rolf Jagerman, Hansi Zeng, Zhen Qin, Dong Wang, Xuanhui Wang, and Michael Bendersky. Inference scaling for long-context retrieval augmented generation. arXiv preprint arXiv:2410.04343, 2024.
Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi. Siren's 's song in the ai ocean: A survey on hallucination in large language models. arXiv preprins 209.022309.0129.. 2023.
Zihan Zhang, Meng Fang, and Ling Chen. Retrievalqa: Assessing adaptive retrieval-augmenntd generation for short-form open-domain question answering. arXiv preprint arXiv: 2400.16457, 2024.
Penghao Zhao, Hailin Zhang, Qinhan Yu, Zhengren Wang, Yunteng Geng, Fangcheng Fu, Ling Yang, Wentao Zhang, and Bin Cui. Retrieval-augmented generation for ai-generated content: A survey, arXiv preprint arXiv:2402. 19473, 2024.
13

---

Published as a conference paper at ICLR 2026
is mhe place birn ofthe divectr of Trars Fneedd?
Whst the place of bast of the direcs of Pown's Fricade?
woald nead know dircttr dits 'Pear's Friends" and his place boths intta Quees: Who the dirctse the ditss Tearr' trnende" and vatze war borr? Decument Pete's Friendi Reser's Freeds o 1992 Bittot conndyy firm.. and ovected preassed Neenen Bratogh
Analyde anowe
Pear's Fricsisf
Let's search question Wikipedis.
Auto RAG
Context Petor's Avceds di 1982 drizsh comeely film writonn Rox fudnner and Marto dergmon, and atrected preassoed fasceth Aranagh on
Intermaliate saveer Beaed Hatricred Decamen direcs
tater mederee ansrrvet Krazeih Ferrogh
place
up
Nattlen ttooough?
Refleed Whet the of hith Ricsed Carte, of Fricada?
licumeer 2: Rieherd Carts nee bert Bex Zeatond to Aurmalan parenes
Head anewer:
Intermediate nnswer 2: Uised on the itatoordd Decaman: 2. Ristued Carta was warbom New Zealand Therffort the place of director fils "Ptee's Friendi
<asower long The direcoo of fit Powe's Frieads Kerneth Branag), and be was som Skitae, Nentern leckadd Strower long"
casosur chert-Belfttt wett--Blttttcheeeeeee
Figure 8: Case Study: Auto-RAG DeepRAG. DeepRAG achieves success by atomic query decomposition, faithful intermediate answer, and adaptively using internal knowledge.
A TEMPLATES
A.1 DEEPRAG CONSTRUCT INSTRUCTION
Instruction: You are a helpful Setrieval-Augmenee Generation (RAG) model. Your task to answer questions by logically decomposing them into clear sub-questions and iteratively addressing each one.
Use "Follow up:" introduce each sub-question and "Intermediate answer:" to provide answers.
For each sub-question, decidc whether you can provide direct answer or if additional information is required. If additional information is needed, stale, "Let's search the question in Wikipedia." and then use the retrieved information respond comprehensively. If direct answer is possible, provide immediately without searching.
B METHOD DETAILS
B.1 IMITATION LEARNING OBJECTIVE
We implement a masked loss function for the retrieved documents to prevent the model from learning irrelevant noisy text that could negatively impact its performance. In this way, we hope the model to enhance the ability to decompose subqueries and retrieve them based on demand. For each instance, the loss function formulated as follows:
C
Σ log (Pr(a[(s-1) + ((a.(s)-1.2;dd.1) 15150
where, d, refers to null if there is no reieval for ith reasoning step, n refers to the total iteration.
B.2 OFFLINE LEARNING OBJECTIVE
Given the i-th subquery and the state s; xr Q1 '.. -4--1, we have two distince interme- diate answer r! = a, and r = (d,, af) Based on the process above, we have known which ry is preferred, As result, the training objective can be formulated as follows:
- log (glog
(N 9:,4:) Teat (Nw
log Tre
Te(y gi)
$1.9:)
8i, 9:)
where is the logistic function, the hyperparameter 3 regulates the penalty imposed for the deviations from the base reference model Kref- The terms yo and z refer to the generated snippets for direct answers and retrieved answers, respectively. Specifically, the snippet "Intermediate Answer:" corresponds to a direct answer, while the snippet "Let's search the question on Wikipedia" corresponds sarrival-bbsee answers.
14

---

Published as a conference paper at ICLR 2026
B.3 ONLINE LEARNING OBJECTIVE
Learning Objectvie We employ GRPO (Guo et al., 2025) as the learning objective in DeepRAG- RLon To mitigate the influence of irrelevant noisy retrieved content, we apply masked loss over the retrieved tokens, ensuring that the model focuses on learning from informative text.
(0)
E (o
(4)
min (ree(0) An, - A..) T(or,s) - DEL ( mee))
[0s]
.here T(Os,,) is an indicator function that equals O token di, belongs to retrieved text, and 1 otherwise. rie refers to importance sampling ratio and Ar. refers to advantages as shown in Equation B.4
(0) To(ou, 01005 Tha (Oi,e os k1,))
R, mcan ( (R std({ 10 1;=1)
A
(5)
B.4 IMPLEMENTATION DETAILS
We train target model on two QA datasets: HotpotQA and 2WikiMultihopAA. For imitation learning, we randomly sample 4,000 examples from each dataset. To enhance the model's question decomposition and context-based generation capabilities, the gray (query decomposition) and green nodes (retrieved answers) in Figure 2 and use the target model generate the blue nodes (parametric answers) for data synthesis. For chain of calibration, we sample an additional 1,000 examples from each dataset. For DeepRAG-RLow the hyperparameters are configured as scorefamma = 0.1, a 0.1, and 5. The performance evaluated using Exact Match (EM) and F1 score.
employ Qwen-2.5-72B to generate
Following Su al. 2024). we adopt BM25 for retrieval and Wikipedia as knowledge base. We selected Llama-3-8B -nnttructttttttt (Dubey et al.) 2024), and Qwen-2.5-32B (Yang et al. 2024) our target model. To implement Search-ol, we employ the distillation series of DeepSeek-RI (Guo et al.) 2025), the method depends reasoning models. All reinforcement learning experiments reproduced based on the VeRL framework
C DETAILED ANALYSIS
C.1 CASE STUDY
As illustrated Figure 8 we conduct case study comparing DeepRAG with Auto-RAG (Yu et 2024), a closely related method that utilizes iterative retrieval for retrieval-augmented generation. For each subquery, Auto-RAG retrieves relevant documents and generates corresponding subanswer. This approach is not only time-consuming but also fails when no relevant documents are retrieved. Although Auto-RAG attempts to address this issue using its own relevant documents, it falls into endless loops in most cases. In contrast, DeepRAG iteratively generates subqueries and determines whether to use internal knowledge at each iteration. The binary tree search data synthesis method for optimization ensures reliable subquery generation, intermediate answers, and final answers. Even when no related information exists in retrieved documents, the model is directed to provide a final answer based on internal knowledge.
C.2 RELEVANCE TO PARAMETRIC KNOWLEDGE
In this section, we investigate the relationship between retrieval needs and parametric knowledge to demonstrate how effectively our method explores the knowledge boundary.
httpp: htwpa/lliielllee..oooooo////ieee http:/wwww..ccooooolennnnnee/eee
www..sv.pp
15

---

Published as a conference paper at ICLR 2026
Ideally, models should initiate retrieval for queries beyond their parametric knowledge while utilizing their existing knowledge for familiar queries. We use CoT results as an indicator whether the model can answer questions using its parametric knowledge. Subsequently, adaptive retrieval methods align with this pattern of parametric knowledge utilization. We evaluate the relevance using four metrics. F1 score and Accuracy serve basic performance measures, while balanced accuracy and Matthews Correlation Co))))cien((MCCC are employed to account for the class imbalance between retrieval-required and retrieval-not netuuuuree cases. The MCC ranges from where value of indicates perfect correlation, 0 represents no correlation (random chance), and -1 signifies an inverse correlation.
analyze whether other
As shown in Table 3] we find that 1) DeepRAG demonstrates superior relevance performance across F1, balanced accuracy, and MCC metrics. This suggests that DeepR AG successfully identifies retrieval necessity by exploring knowledge boundary. 2) While FLARE, DRAGIN, and TAARE exhibit high accuracy scores, their relatively low balanced accuracy and MCC scores suggest they mainly succeed in retrieval-required cases but struggle properly avoid unnecessary retrievals.
C.3 PERFORMANCE AGAINST STRONG BASELINE MODELS
We compare the Lama-3-8B-hae DeepRAG with recent strong reasoning models: QwQ-32B- preview (Team] 2024) and gpt-4o-turbo (OpenAI). As shown Table DeepRAG achieves superior average performance over QwQ and gpt-4o, particularly in time-sensitive QA tasks. While DeepRAG does not surpass gpt-4o in some cases, achieves comparable performance levels. These results demonstrate that by adaptively leveraging retrieval, DeepRAG can achieve an equivalent level of factual accuracy to parametric knowledge of strong reasoning models.
Models
ID
PopQA
WQ
Avg
QwQ-32B
31.43
10.60
15.10
19.04
gpt-4o-turbo
60.60
43.50
25.35
43.15
DeepRAG-RLon
52.06
47.20
30.00
46.09
Table 4: Performance against strong baseline models.
16


Reviews:
[{"review_id": "e1uTNY9c0Q", "reviewer": "Reviewer_TwZX", "summary": "DeepRAG models retrieval-augmented reasoning as a Markov Decision Process, enabling dynamic per-subquery decisions on whether to retrieve external knowledge or rely on parametric knowledge. The framework uses: (1) Binary Tree Search to explore retrieval strategies, (2) Imitation Learning on minimal-cost trajectories, and (3) Chain of Calibration (offline/online) to teach knowledge boundary recognition.", "questions": "1. Algorithm Correctness Issue\nAlgorithm 1 (page 5) has a critical flaw:\npythonLine 7: if IsEqual(o, y) then return h  # Returns FIRST correct path\nThis implements greedy search, not optimal search. Counterexample:\n\nPath A: [retrieve, retrieve] cost=2 (found first) \nPath B: [parametric, parametric] cost=0 (explored later) \n\nThe paper claims \"minimal retrieval cost\" but the algorithm returns whichever correct path is found first in priority queue order—not guaranteed optimal.\n\n2. Questionable \"Optimality\" Claims\nTable 1 shows counterintuitive result:\n\nRLoff (trained on \"optimal paths\" from Stage I): 41.47\nRLon (self-exploration): 41.05 (better!)\n\nIf Stage I finds optimal paths, why does self-exploration outperform? Possible explanations:\n\nAlgorithm 1's bug means Stage I paths aren't actually optimal\nRLon discovers better strategies not in training data\nGRPO is simply better than DPO for this task\n\nThis undermines confidence in the entire Stage I data synthesis process.\n\n3. Reproducibility Concerns\nMany critical details missing:\n\nBinary tree search: max depth? timeout? handling of unsolvable queries?\nTraining time for tree search (exponential in depth)?\nInference latency breakdown (subquery generation overhead)?\nDPO hyperparameter β? GRPO rollout count G?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Well-Motivated Problem\nExisting RAG systems apply retrieval indiscriminately—either over-retrieving (wasting compute) or under-retrieving (missing information). DeepRAG addresses this through atomic decisions: decompose queries into subqueries, then decide retrieval necessity per subquery rather than per original query.\n2. Strong Empirical Results\n\n25.41% accuracy improvement over baselines (Table 1)\nBetter retrieval efficiency: lower average steps than Multi-Step Retrieval (Table 2)\nGood generalization on out-of-distribution datasets (PopQA, WebQuestions)\nHigher knowledge boundary alignment (MCC scores in Table 3)\n\n3. Comprehensive Ablations\nValidates minimal-cost path selection (Figure 6), atomic query decomposition effectiveness (Figure 4: fewer conjunctions/pronouns per subquery), and necessity of calibration stage (RLoff/RLon > Imi).\n4. Practical Design\nMasked loss over retrieved documents prevents noise learning; end-to-end trainable without separate classifiers; works across different LLM sizes (Llama-3-8B, Qwen-2.5-32B).", "weaknesses": "1. Overstated Technical Novelty\nThe \"MDP formulation\" is superficial packaging:\n\nTransitions are deterministic (not stochastic as typical MDPs)\nNo actual policy search—just supervised learning on pre-computed trajectories\nBinary Tree Search is exhaustive enumeration (2^N paths), not a novel algorithm\n\n\n2. Narrow Evaluation Scope\nOnly tested on artificially constructed multi-hop QA benchmarks (HotpotQA, 2WikiMultiHop, MuSiQue). Missing evaluations on:\n\nSingle-hop questions (does it over-retrieve?)\nReal-world queries: \"Draft a rejection email,\" \"Compare iPhone vs Samsung,\" \"When will my order ship?\"\nNon-QA tasks: summarization, dialogue, creative writing\n\nThe query patterns in academic benchmarks don't represent real deployment scenarios.\n\n3. Missing Simple Baselines\nThe paper requires three complex stages but never compares to a one-stage baseline:", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5X4m6x9f2c", "reviewer": "Reviewer_taGT", "summary": "The paper proposes DeepRAG, a retrieval-augmented reasoning framework that formulates iterative query decomposition and retrieval decisions as a Markov Decision Process. It introduces three core components: Binary Tree Search to exhaustively explore parametric and retrieved paths for each subquery and identify minimal-retrieval correct trajectories; Imitation Learning through supervised fine-tuning on these trajectories with masked loss over retrieved tokens; and a Chain of Calibration that includes both offline and online variants to enhance the model’s awareness of its knowledge boundaries. Evaluated on in-distribution datasets such as HotpotQA and 2WikiMultihopQA, as well as out-of-distribution datasets including PopQA, WebQuestions, and MuSiQue, DeepRAG achieves a 25.41 percent accuracy improvement over previous adaptive retrieval-augmented generation methods while also reducing retrieval calls.", "questions": "- For WebQuestions (Freebase-backed), what fraction of ground-truth answers are un-retrievable from Wikipedia, and how does DeepRAG-RLon handle verified absence (e.g., explicit “unknown” vs. hallucination)?\n- How does the model behave on single-hop factual queries (e.g., PopQA)? Does calibration reduce retrieval to near-zero, and is parametric accuracy preserved?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Strong empirical gains: 25.41% accuracy lift is reported across five datasets, with ablation likely showing each stage’s contribution; out-of-dist PopQA and Freebase-absent WebQuestions stress robustness.\n- Uses a fixed-depth priority queue (lowest retrieval count first) and discards unsolvable instances, yielding high-quality imitation data without oracle subqueries.", "weaknesses": "- The method doesn’t seem to offer much innovation. Among the many existing approaches that use reinforcement learning for autonomous multi-turn retrieval, I didn’t find any particularly striking or novel insights.\n- All experiments assume a fixed Wikipedia retriever (presumably BM25 or Contriever). What about comparison with some deep research methods, they also use multi-turn retrieval?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fXifNrYBnM", "reviewer": "Reviewer_JFTj", "summary": "This paper introduces DeepRAG, a framework that enhances retrieval-augmented generation by modeling reasoning as a Markov Decision Process (MDP). DeepRAG dynamically decides when to retrieve external knowledge and when to rely on internal parametric reasoning through iterative query decomposition. By enabling adaptive and context-aware retrieval, it reduces noise from redundant information and improves both retrieval efficiency and answer accuracy—boosting performance by 25.41%—demonstrating significant advances in retrieval-augmented reasoning for LLMs.", "questions": "* Table 1 shows that Qwen 2.5 32B peforms worse than Llama 3 8B, which is confusing for me, could you explain the reason?\n* Online RL methods such as GRPO typically outperform offline approaches, a trend observed in the performance of Qwen. However, in the case of Llama, online and offline methods show comparable results. What accounts for this difference？\n* In Section 5.3, line 436 states that most questions require 3–5 decomposition steps. However, the majority of queries in HotpotQA and 2Wiki are designed for 2-hop reasoning. Does this imply the presence of significant redundant retrieval? If so, what causes this inefficiency? And do current sota models, such as GPT-5, still exhibit similar tendencies toward redundant reasoning?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper is well written\n* When to conduct retrieval is an important topic\n* The experiments are extensive", "weaknesses": "* Compared to Serach R1, the paper mainly differs at using the model's native generation ability to determine when to retrieve, but this has already been well studied and the authors do not provide new solution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)