FrugalRAG: Less is More in RL Finetuning for Multi-hop Question Answering
Authors: Abhinav Java
Srivathsan Koundinyan
Nagarajan Natarajan
Amit Sharma
Venue: ICLR 2026 Poster (2026)
Keywords: Multi-Hop RAG
Efficiency
Reasoning
SLMs
Primary area: foundation or frontier models, including LLMs

Abstract: Reinforcement learning (RL) based on the final answer's reward has driven recent progress in small language models (SLMs) on reasoning-heavy tasks such as math and code. However, applying the same techniques to retrieval-augmented generation (RAG) benchmarks like multi-hop QA has yielded limited gains—often trailing supervised or prompting-only baselines. Instead, we argue that a viable path for RL in multi-hop QA is to use test-time scaling judiciously, for optimizing both the final answer accuracy and the efficiency in reaching that answer. 
We propose FrugalRAG, a two-stage finetuning framework that adaptively _reduces_ the number of retrieval steps based on a question's difficulty. First, we train an SLM with supervised finetuning on a full-exploration policy that generates broad sub-queries. Then, we apply RL to adaptively prune search depth based on question difficulty, directly rewarding policies that balance correctness with frugality. Unlike prior approaches requiring 10× more data, our method achieves competitive performance with only ~1,000 examples. On HotPotQA and other multi-hop QA benchmarks, FrugalRAG attains state-of-the-art efficiency–accuracy tradeoffs, cutting retrieval cost nearly in half.  Moreover, on the challenging BrowseCompPlus benchmark, it generalizes zero-shot and surpasses SLM-based and other baselines. These results demonstrate the use of RL—not to increase reasoning steps but to reduce them—as an effective solution for scalable, efficient RAG.

Published as a conference paper at ICLR 2026
FRUGALRAG: LESS IS MORE IN RL FINETUNING FOR MULTI-HOP QUESTION ANSWERING
Abhinav Java, Srivathsan Koundinyan, Nagarajan Natarajan & Amit Sharma Microsoft Research India
ABSTRACT
Reinforcement learning (RL) based on the final answer's reward has driven recent progress in small language models (SLMs) on reasoning-heavy tasks such as math and code. However, applying the same techniques to retrieval--ugmmenee generation (RAG) benchmarks like multi-hop QA has yielded limited gains often trailing supervised or prompting-On only baselines. Instead, we argue that a viable path for RL in multi-hop QA is to test-time scaling judiciously, for optimizing both the final answer accuracy and the efficiency in reaching that answer. We propose FrugaIRAG, two-stage finetuning framework that adaptively reduces the number of retrieval steps based on a question's 's difficulty. First, we train an SLM with supervised finetuning on full-exploration policy that generates broad sub-queries. Then, we apply RL to adaptively prune search depth based on question difficulty, directly rewarding policies that balance correctness with frugality. Unlike prior approaches requiring 100x more data, our method achieves competitive performance with only 1,000 examples. On HotPotQA and other multi-hop QA benchmarks, FrugalRAG attains state-of-the-arr efficiency-accuracy tradeoffs, cutting retrieval cost nearly in half. Moreover, on the challenging BrowseCompPlus benchmark, it generalizes zero-shot and surpasses SLM-based and other basclines. These results demonstrate the use of RL RL-not not to increase reasoning steps but to optimize them- as effective solution for scalable, efficient RAG.
INTRODUCTION
We study the problem of answering questions, such as "Can a microwave melt Toyota Prius battery?", given access to a large unstructured corpus like Wikipedia. The de facto approach to solving the problem is to use language models (LMs) coupled with the ability to retrieve relevant documents (e.g.. Wiki passages) against queries, i.e., the retrieval-augmented generation (RAG) paradigm (Jeong et al. 2024, Diang et al. 2023] Chan et al., 2024) Asai et al.) 2023). However, answering complex questions often requires multi-hop reasoning and retrieval, i.e., the LM has to iteratively decompose the user utterance into sub-queries or search phrases (e.g., "melting point of Toyola Prius batlery"), retrieve documents relevant to the sub-queries, and reason through the retrieved documents to issue further sub-queries, until the LM is able to generate an answer for the original query.
Based on the success of reinforcement learning (RL) in math and code applications (DeepSeck-Al 2025), recent work applies RL-based finetuning to optimize retriever tool calling and generation of the right queries answer given question. However, the accuracy gains in multi-hop QA benchmarks are less impressive. For instance, on the HotPotQA benchmark (Yang et al. 2018). simple ReAct-based Yao et al. (2023) strategy of letting a model generate up to 10 subqueries leads to higher document recall (63%) than state-of-the-art RL. methods such as Search-R1 I (see Table 2). Therefore, the key question is not how RI. can be used improve accuracy, rather how RI. can be used to optimize the search process and make RAG system more efficient (c.g., for HotPotQA, most questions can ideally be solved in 2-3 search queries). Ideally, we want a system that can adapt the question's 's difficulty level.
number of search queries based
On efficiency, a second problem is the availability of labelled training data that maps a question to its correct answer. Outside of general web muestion-answerin labels for questions when adapting a RAG system to work a real-world application domain (with potentially private documents). Typical solutions in the literature (Tin et al. 2025a Hsu et al. 2024
it may be difficult to obtain ground-truth

---

Published as a conference paper at ICLR 2026
however, train 00,000-1,00,000 examples from existing benchmarks such HotPotQA Yang ctt al. 2018). Therefore, a second key problem is finetuning a language model for retrieval when labelled data is scarce. For concreteness, set the number of training examples 1000, an order of magnitude lower than existing efforts.
Under these considerations, we ask the question: How can we train model answer questions using as few search calls as necessary? And can we do is using only 1000 training examples? Our solution, FRUGALRAG, is two-stage framework that uses exploration generate a large number of search queries per question (Stage 1), and uses RL to optimize the number of search queries per question (stage 2). Specifically, in the first stage, the model is trained to maximize evidence coverage by generating diverse and informative search queries across multiple hops. In the second stage, we post-train the model using RI. to decide when to stop retrieving and generate an answer. This decision is modeled explicitly, allowing the model to weigh the cost of further retrievals against the confidence in the retrieved evidence. Optimizing for coverage and efficiency training: we find that models either over-retrieve or stop too early. Our key insight is that learning when to stop is more naturally learned through reinforcement learning (RL) signals, whereas better coverage can be obtained by repeatedly issuing high quality search queries using frameworks such as ReAct.
single stage leads to unstable
We evaluate FRUGALRAG standard multi-hop datasets such HotPotQA Yang al., 2018). 2WikiMultHHopQA (Ho et al. 2020) and MuSiQue (Trivedi et al.] 20226). using both document retrieval metrics such as recall and answer quality metries. Compared to baselines, we show that FRUGALRAG obtains the highest document recall and answer quality while incurring a low number of search queries per question. In particular, our models obtain state-of-the-arr accuracy on the recently released FlashRAG index benchmark for HotPoiQA and MuSiQue, despite being trained on only 1000 examples.
We also find that the same FRUGALRAG model generalizes to harder information search tasks. A recently released benchmark from OpenAI tests models' abilities on challenging needle-in-a-her hayyl--n problems, wherein the reasoning steps (and hence the search queries) to find the answer can be much larger. The same FRUGALR 7B model described above generalizes zero-shot on this out-of-domain task; obtaining accuracy of 20%, which is higher than that of bigger models such as DeepSeek-RI I and Search-RI --32B. Remarkably, it adapts to issue a larger number of queries for the BrowseComp data compared to earlier datasets, unlike other RL methods such as Search-R1 (Jin et al. 2025a).
RELATED WORK
In this section, we present detailed overview of the relevant literature comprising representative multi-hop RAG methods, recent work leveraging reinforcement learning for search, and finally standard metrics for evaluation.
Multi-Hop RAG. Retrieval Augmented Generation is an active area of research that aims to ground the responses generated by LLMs in real world information, tackling fundamental limitations like
hallucinations and trustworthiness. Several works (Guu al. 2020 Lewis et al. 2021 Shao et al.
2023
Jeong et al. 2024 Jiang et al. 2023 Asai et al. 2023 Chan et al. 2024 Hsu et al. 2024 Li
et al.
2025 Wang et al. 2025) have been proposed in recent literature that explore this problem.
Guu et al. (2020) and Lewis et al. (2021) 1) showed that augmenting the inputs of language models with retrieved information significantly improved their performance on knowledge intensive tasks like question answering. Subsequently, (Tivedi et al. (2022a) leveraged few-shot Chain-of- Chain-of-Thought (Wei) al. (022) prompts and iteratively retrieved information for complex questions that required multiple reasoning and retrieval steps (i.e. multi-hop settings). (hao et al. (2023) also incorporate a similar iterative retrieval approach. By using the intermediate traces, the IRCoT is able to decide what to retrieve by issuing the right search queries. Iter-RetGen (Shao et al.] 2023) improves evidence gathering in multi-hop scenarios by combining retrieval and generation iteratively, such that a model's response is incorporated in the reasoning trace. However, both IRCoT (Trivedi et al. 2022a) and teer-RetGen (Shao et al.) 2023) rely on a fixed or predefined number of retrieval loops at inference, offering limited control over latency. Toolformer(Schiel et al. 2023) uses supervuue objective to train an external model that decides to call tools (like Bing and Google search engines). ReAct ( (Yoo)
et al. 2023) uses general prompting framework by interleaving thoughts, actions, and tool calls
2

---

Published as a conference paper at ICLR 2026
in order to perform complex tasks. Khattab et al. (2023) invoked LLMs through declarative calls, allowing a more structured programmatic way of calling LLMs. Further |Khattb et al. (2023) also
design example-data efficient compilers optimize a given metric. [Ammann al. (2025) introduce question-decompositio and reranking system for multi-hop RAG. Theii approach improves retrieval quality through decomposition, whereas our approach focuses primarily learning to adaptively
retrieve. Collab-RAG (Xu et al.) (002), proposes collaboration between an LLM and an SLM. This method relies on coordination between models, whereas FRUGALRAG operates only with an SLM.
Beyond prompting, SelfRAG (Asai et al. 2023) trains small language models (SLMs) with dense supervision from stronger models to decide when retrieve external information during question answering. RQRAG (Chan et al. 2024) leverages LLMs to curate datasets that teach smaller models essential skills such as query decomposition and backtracking.
addition, Chan et al. (2024) highlight the effectiveness of advanced decoding strategies such as tree-of-thought sampling for identifying high-quality reasoning trajectories. More recently, approaches like CoRAG (Wang) et al. 2025) have scaled test-time computation by jointly reasoning over multiple trajectories and training models through rejection sampling. However, these methods (SelfRAG, RORAG, CoRAG) demand substantial supervision (100k+ annotated examples) and are constrained either by inference inefficiency (RQRAG (Chan et al. 2024)) or by limited flexibility (CoRAG (Wang et al. (025)), which relies predefined number of search). Other works, such DRAGIN (Su al. (024) and Adaptive RAG (Jeong et al. 2024), train multiple modules with large-scale supervised data to enable dynamic retrieval. SimpieDeepResearcher (Sun et al., 2025) propose tata-enginerring strategies to mitigate the problem of domain shift between training and testing environment showing that supervised finetuning on a small number of representative samples can yield strong improvements. In contrast to these approaches, we introduce the first method that jointly accounts for both training and inference costs. Our key intuition is that reinforcement learning can be leveraged to scale down search effectively, thereby reducing supervision requirements while maintaining competitive performance.
Reinforcement Learning for Search. Recently, framing search query as an RL problem has received attention. LeReT (Hsu et al., 2024) is an early work that adopted RI. by performing preference optimization using diverse few shot prompts leveraging muudred-thousande of ground truth annotated documents. However, this finetuning process is computationally expensive and cannot be cheaply and readily generalized to variable-hop scenarios. As result, LeReT utilizes a fixed amount of compute (i.e. search calls) per instance during inference. Similarly, concurrent works, Jin et al. (2025a) and Chen et al.] (2025) propose end-to- -o-doo-- RL-based optimization that only leverages the final answer annotation. R-eeeah (Song al. (2225) proposes an RL-only strategy for finetuning demonstrating strong performance on multi-hop RAG benchmarks by utilizing a curated balanced dataset with outcome based rewards. These methods show that RL can effectively be used to teach the search query generator model to issue more search queries for multi-hop problems without considering latency. Our two-stage RL framework, by contrast, first explores without RL. to maximize recall and then learns to stop at test time using RL.
Metrics for Evaluation. Multi-hop QA involves two sub-tasks: retrieving relevant documents, and then answering the question based on the documents. Some methods report document retrieval- specific metrics such as recall (Hsu et al. 2024) whereas others report final answer metrics such as exact match (Jin et al., 2025a). Typically, a model is trained to optimize a particular metric (such as recall) and also evaluated on the same metric. For robustness, in this work we train on the recall metric and test on all metrics, including robust model based evaluation of the final answer following FlashRAG (Jin et al. 20256). Unlike WWang et al. (2025) ; Li et al.] (2025): Jin et al.] (2025a); [Song et al.] (2025) we train reasoner module which performs reasoning and issues search calls and an off-the-shelf final answer generator.
3 FRUGALRAG: EFFICIENT TWO-STAGE FINETUNING FOR RAG
We describe FRUGALRAG, a novel framework for enhancing retrieval augmented generation in LMs by decoupling the evidence exploration stage from answer generation. FRUGALRAG demonstrates several key advantages over contemporary RAG approaches (1) Requires only 1000 annotated training examples which is a 100 times reduction in dataset size compared to existing works (Jin et al.,
2025al Hsu et al. 2024 Chan et al. 2024), (2) Dynamically adapts test-time compute which results
3

---

Published as a conference paper at ICLR 2026
in low inference time latency and high retrieval recall, unlike existing fixed compute methods (Hsu summarizes our framework.
et al. 2024). Algorithm |
question that requires multiple iterative retrievals to answer. Let ff denote a "reasoner" language model (LM) that, at each reasoning hop, examines the current context and determines the next action. At hop h (where 1 I h I B, and B is query triplet
Problem Formulation. Let Q denote complex
the maximum allowed number of hops), the model f produces a thouglt-action-seen (Th, Ab. Sn). The search query Sn is passed retriever R(.), which returns a of documents: Dr = R(Sx) from the document index I.
Let Do denote the initial context which is either empty initialized as R(Q). At hop h, the model has access to the context: {Q} U {(D,, Th, An, S,)) This includes the original query, all previously retrieved documents, and the previously generated thought, action, search query triplets. The process continues until the model outputs a special FINISH action, terminating after herm hops (or at the budget limit B). At this point, separate generator LM g is invoked produce the final answer, conditioned on the original question Q and the full context accumulated up to termination.
The central challenge for f is to iteratively construct highly targeted queries Sn, such that the retriever R can surface minimal, sufficient set of documents answer Q within B hops.
Our key observation is that, with sufficient test-time compute, even base models can generate multiple, diverse search queries to address question (e.g.. see Sec 4 Table 2, ReAct+DsPy).. Therefore, the goal of our work is not to scale up test-time computation, as argued in prior work (Jin et al.) [202aa] Chen et al., 2025a). but rather to adaptively control it based on the difficulty of each question.
Our framework, FRUGALRAG requires access only to ground truth documents Y during training. It does not require supervision in the form of final answer annotations. These documents are used to provide fine- tne-grrined feedback signals. At inference time, FRUGALRAG relies solely on the input question Q, the document index Z, and a trained retriever R.
In the following subsections, we describe the two stages of our learning algorithm: (1) Stage 1 (Sec. 3.T y Generating base policy that maximizes evidence coverage through exploration, and (2) Stage (Sec. 3.2): Finetuning this policy with reinforeement learning control test-time compute.
3.1 STAGE 1: EVIDENCE COVERAGE MAXIMIZATION (EXPLORE)
Gathering evidence plays a crucial role in answering multi-hop questions, which often require iterative retrieval and reasoning across multiple sources. Drawing recent advances in test-time scaling, we observe that we can boost evidence coverage (i.e., recall) simply by letting the model f issue multiple search queries S₁, at inference time. This approach sidesteps the need for massive supervised ten-ttuninnnntttalll it harnesses the model's own generated rollouts to gather, and then integrate additional information. In the next section, we describe how we construct our training data and design our fine-tune fine-tuning protocol to fully leverage this capability.
Training Dataset Generation. We refer to rollout as a set of outputs generated by the model f. Each rollout comprises thought-action pair (T,)))) at each hop h E [1, BJ. where A, is either a call to the retriever R. or FINISH indicating the end of rollout. This setup is similar to the standard ReAct (Yao et al.) (023) framework which provides general prompting template for interleaving external (retrieved) information and model generated outputs. At each hop h. we generate samples {(T)) AJ, S)) ... (TT, Ag, } using bootstrapped prompts (Khattab al. 2023] (See Appendix F). For each search query S i E [1, we retrieve corresponding documents Dj R(S),, ), then discard any documents already present in the context. We then compute recall against ground -trtt labels and add the sample that achieves the highest recall to the context for the next hop h + 1. This dataset generation strategy is simple and easily parallelizable. We conduct two separate runs - standard run where f is allowed terminate generation by generating FINISII, and the second where f can only call the retriever till maximum budget is reached. Although the former is more efficient and finishes before B search hops, we observe that the latter yields a higher overall recall owing to greater number of retrievals. Unlike previous work (Chan et al.] 2024]] Asai (et al. 2023 Hsu et al. 2024; Jin et al.) (025a) that generated orders of magnitude more data, we only generate 1000 examples during this step.
4

---

Published as a conference paper at ICLR 2026
Supervised "Exploration" Finetuning (FRUGALRAG-Explon) Although the base model f without FINISH maximizes exploration, we cannot use it directly for reinforcement learning because does not include FINISH. Consequently, during fine-tuning, sample rollouts from both the configurations described above, i.e., 90% without FINISH and 10% with it. We want to use supervised finetuning to build strong base-policy for RI., that prioritizes exploration while ensuring that FINISH remains in the model's 's generation distribution. Hence, we finetune the model to obtain our base policy fs. At each iteration, the model predicts the next (Tn, Ak. Sp) tuple given the rollout which comprises interleaved thought- aettlaaaaiionnrreeee tuples and retrieved documents represented as an ordered set {(Do, Tb. Au, So), (D1. Ti, A1, S1) (D---)): 1): T(n-1)- AQ--1) Sk-1)} till 1, using standard cross-entropy error as the objective function. fs has several notable advantages (1) off-the-sheel model J does not explore. Despite prompt optimization, I is generally over-confident and predicts the answer without sufficient exploration. (2) removing FINISH results in over-refrievals. We observe that simply removing FINISH from the action space yields high recall even with the base model f. however, the model is forced to utilize the full budget for every question and cannot be post-trained for efficiency as it never generates a rollout with FINISH.
3.2 STAGE 2: CONTROLLING TEST-TIME COMPUTE WITH RL
Given a finetuned base policy model fs. we propose a strategy that enables the model to generate extended rollouts only when required. This mechanism allows the model to adaptively determine the appropriate rollout length based on the complexity of the question, rather than scale the number of retrievals as in recent RL techniques (Jin et 2025ap. Since Is generally prioritizes exploration, our only goal is to learn when to sufficient evidence has been gathered. In turn, we can also reduce overall search latency during inference whenever possible. Below we show how this problem can be formulated reinforcement learning task, as it requires evaluating and comparing different rollouts.
Reward Design. Our reward function is designed to guide the model toward discovering the optimal rollout length. To compute the reward, we first generate the complete rollout using the policy fs and then evaluate with ground truth evidence. Let h² denote the optimal number of retrieval steps (or hops), defined as the point beyond which further retrievals do not improve predefined metric, say We compute c, the document recall against ground-truth evidence Y. Operationally, h* the smallest hop at which the rollout achieves recall 2 T.
If the model terminates stopping at brem < h² is also penalized to encourage sufficient exploration. This reward structure enables the model to explore adequately for complex queries while avoiding unnecessary computation for simpler ones. Mathematically,
hop heem h" incurs penalty for redundant steps. Conversely,
1-4)
Rmaxx) ))
max
Rmaxx min
log
if 0AC27 (late stop)
R
Rmax
if 0AC27
(perfect stop) (1)
1
.0) .)))
max
- -maax min
log
if (early stop)
A
Here, denotes the absolute normalized deviation
A [ h*1
where B is the maximum hop budget. When T, set frrm B to promote continued exploration rather than early termination. Note that is clipped for numerical stability.
Rmax specifies the upper bound on the reward, and is tunable hyperparameter that scales the h' bonus awarded for terminating exactly h" This bonus is proportional to B thereby assigning greater reward to correct terminations on more complex (i.e., longer- horizon) rollouts. A rollout is deemed answerable if its performance satisfies c 2
5

---

Published as a conference paper at ICLR 2026
Table 1: Results on HotPotQA, 2Wiki, and MuSiQue with ColBERTv2 retriever. FRUGALRAG demonstrates superior performance on Model Based Evaluation (MBE) and Gold Evidence Recall (%) compared to standard baselines. Here, Qwen2.5-7B-Inse InIstrruct is used as the answer generation model. Best results are in bold and second best are underlined.
Method
HotPoiQA
2Wiki
MuSiQue
MBE
Recall Searches
MBE
Recall Searches
MBE
Recall Searches
Zero-Shot
28.10
NA
28.4
NA
10.6
NA
NA
Lero-Shot CoT
29.10
NA
NA
29.8
NA
NA
10.7
NA
NA
Zero-Shot RAG
51.00
59.05
28.8
34.75
18.5
23.96
ReAct + DsPy
64.20
79.60
2.76
45.60
58.5
3.37
29.20
52.1
3.37
BRUGALRRAG-Explon 7B
5
67.70
86.40
5.99
|7.660
68.90
5.99
30.10
59.10
5.93
FRUGALRAG 7B +Qwen225-7B-Inst
68.47
82.80
2.05
48.93
63.50
2.95
33.72
54.00
3.02
Intuitively, the reward function imposes a penalty proportional to |ا. which quantifies the deviation from the optimal stopping step, Accordingly, the reward decreases monotonically as the termination step moves away from h²" and is maximized when Meer h²
In addition to R (Eq. I we define a format reward R& to enforce adherence to the format. If the output deviates from the expected format and results in failed retrievals, we assign a reward of 0.55; conversely, if the retrievals succeed, the model receives a reward of +0055. This reward is averaged across all hops. The final reward is the mean of main reward and the format reward Rr, yielding overall reward in the range [ Rmax 0.50, Rmax + + 0.50].
Optimal Rollout Length. We define the optimal rollout length h as the minimum number of retrieval steps required to answer query Q effectively. Any additional retrievals beyond ht' are considered redundant. Conversely, if the rollout has not yet gathered sufficient evidence to answer the query, FRUGALRAG:
should continue retrieving. To balance retrieval efficiency and performance, Explore (Js) as a reference policy, assuming it represents the best achievable performance within a fixed budget B. Similarly, determine the threshold based on the performance of the base policy TRUGALRAG-Exploo (fs).
Optimization. We adopt GRPO (Shao et al. 2024) optimization algorithm because of its memory efficiency. At each hop ht we sample e tuples (T7, S and retrieve non-duplicate documents Di.- We collect sample tuples and documents through this process until FINISH or till maximum budget exhausted. For each rollout backpropagate through every logit produced by the policy along that rollout.
then compute cumulative reward using Eq. D and
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Benchmarks and Evaluation. We evaluate FRUGALRAG on three widely adopted multi-hop
retrieval-augmented generation (RAG) Senchmarks-HotPord) (Yang al. 2018). 2WikiMul- tiHopQA (Ho et al. 2020) (2Wiki), and MuSiQue (Trivedi et al. 2022b under their full-wiki setting using the development splits, following the protocol of FlashRAG (Jin al. 2025b). Be- yond multi-hop hopp RAG, we also assess FRUGALRAG in highly challenging deep research scenario using BrowseCompPlus (Chen et al., 20256). To demonstrate the effectiveness of FRUGAIRAGG
we experiment with three different retrievers: ColBERT-v2 (Santhanam et al.) 2021). Qwen3-8B- Embedding (Zhang et al.) 2025), and E5-base-v2 Wang et al.) 2022). Additional details on the datasets and indexing are provided in Appendix G Following prior work, we adopt a robust LLM
judge-base based evaluation metric (Chen et al. (025bb to assess the accuracy of the final answers (MBE Model Based Evaluation). We choose Owen3-32B for answer evaluation. For retrieval-level evalua- tion, we follow FlashRAG (Jin et al.] (0255) and use recall as the metric, which measures whether the ground truth answer is present in the retrieved documents, assigning score of if it is found.

---

Published as a conference paper at ICLR 2026
Table 2: Comparison of FRUGALRAG against state-of-the-art approaches on HotPoIQA, 2Wiki, and MuSiQue, evaluated using Answer (MBE) and Retrieval (Recall) metrics with the E5-base- v2 retriever. Despite being trained on only 1,000 examples, FRUGALRAG delivers consistently competitive performance. The results further show that FRUGALRAG is compatible with diverse answer generator models, demonstrating strong modularity. Best results are in bold and second best are underlined.
Method
#Train
HotPotQA
2Wiki Recall Searches
MuSiQue
MBE
Recall Searches
MBR
MBE
Recall Searches
Zero-Shot Zero-Shot CoT
28.10
NA
NA
28.4
NA
NA
10.60
NA
NA
29.10
NA
29.8
NA
NA
10.7
NA
Zero-Shot RAG
45.50
52.49
28.80
35.00
18.00
22.01
ReAct + DsPy
100
49.30
63.40
3.23
35.65
60.03
3.27
27.30
49.8
3.74
IRCOT
45.60
66.90
3.31
30.53
57.80
3.41
20.62
44.10
3.19
ITER-REE RETGEN
36k
46.10
55.40
3,00
32.10
45.30
3.00
19.10
34,20
3.00
Ret-Robust
1000
39.38
35.90
4.02
45.17
31.50
4.23
24.24
18.80
4.23
Self RAG 7B
150k
37.60
52.60
1.00
30.10
41.30
1.00
15.00
26.70
1.00
Self RAG 13B
150k
39.70
52.60
1.00
31.50
41.30
1.00
15.00
26.70
1.00
Searcher
1440
42.70
50.10
1.77
45.70
55.20
2.42
26.40
37.00
1.95
SiepeeDeepSSercher
871
50.40
64.80
2.75
49.30
60.50
3.64
34.30
50.40
2.86
RI-Searcher
8k
57.66
69.10
2.22
$2.00
60.40
2.36
39.78
57.70
2.31
Search-RI
>10k
46.20
48.20
1.28
36,20
47.70
1.89
24.80
38.10
1.36
CoRAG
>100
58.20
64.30
4.00
59.00
65.40
4.00
40.50
54.00
4.00
FRUGALRAG-7B
-wenn.55-7B-nn
1000
58.5
70.40
2.89
50.40
58.80
3.03
36.40
53.30
3.30
FRUGALRAG-7B
Qwen2.5-322 5-32B---------
1000
61.4
70.40
2.89
50.90
58.80
3.03
39.90
53.30
3.30
FRUGALRAG-7B
+ CoRAG
1000
58.00
70.40
2.89
51,20
58.80
3.03
34.60
53,30
3.30
Finally, evaluate efficiency, report the number of searches performed proxy for latency across different methods.
Baselines.
We perform extensive experiments comparing FRUGALRAG with several strong baselines
and state-of-the-art models. Our evaluation covers: (1) zero-shot performance; (2) prompting-basee
methods such ReAct (Yao et al. 2023), IRCOT (Trivedi al., 2022a), and DsPy (Khattab et al.
2023 ; (3) finetuned approaches inc uuiin Iter-RetGen (Shao et al. (023). Ret-Robust (Yoran et al.
2024). and Self-RAG (Asai et al.) 2023). We further benchmark FRUGALRAG against recent
reasoning-focusel models, including CoRAG (Wang et al.) (2025). 02 Searcher (Mei et al. 2025).
Simple Deep Searcher (Sun et
2025). and Ri-Searcher (Song et al. 2025).
Training and Inference. FRUGALRAG is model -agnos-gg,,tt;;; we train Qwen2.5 In-IIruttt using our two-stage framework. Both supervised finetuning and reinforcement learning (RL) stages leverage the TRI. library (von Werra et al. 2020), with prompt bootstrapping based on DsPy (Khattab et al.) 2023). For each dataset, Stage finetuning uses 1000 randomly sampled training examples, performing full-parameter training for one epoch with learning rate of 2 x 10-5, weight decay of 0.01, and maximum sequence length of 4096 (Sec. (Sec.3.II) Stage applics GRPO (Shao et al.) (024) to further train the models (Sec. 3.2). During inference, we use our finetuned model FRUGALRAG for multi- hop reasoning and retrieval followed by an off-the-shell model for final answer generation. The answer generator is provided with the necessary context (trajectory-level information) to generate the final answer. Full hyperparameter and prompt details are Appendices respectively.
4.2
MAIN RESULTS: EFFECTIVENESS OF FRUGALRAG
Table I compares FRUGALRAG with standard zero-retrieval and prompt-based baseline (ReAct). For fair comparison, all baselines use the ColBERTv2 Santhanam et al. 2021 retriever indexed on Instruct Wikipedia (See Appendix F) and Qwen2.5-7B - the base model. The key takeaway is that
FRUGALRAG consistently outperforms the baselines on both answer and retrieval metrics, using competitive significantly smaller number of searches average.
7

---

Published as a conference paper at ICLR 2026
Table BrowseCompPlus. FRUGALRAG, trained on HotPorQA generalizes this harder task and
outperforms significantly larger models such as DeepSeek-RI.
Method
Model Size
Accuracy (%)
Recall (%)
Avg. Searches
Sonnet 4
37.35
47.33
9.03
Opus 4
36.75
50.84
10.24
kimi-k2-0711-11
k2-0712-11111 1-previww
35,42
38.38
11.22
Gemini-2.5-Flan
34.58
40.19
9.77
Gcmini-2.5-Pro
29.52
35.31
6.04
08s- 18s-120b-low
120B
25.54
22.50
2.21
oss-20b-high
20B
35.06
49.29
23.87
oss-20b-medium
20B
30.48
41.31
13.64
oss-20b-low
20B
14.10
17.37
1.87
DeepSeck-R1-0528
600B
16.39
16.32
2.72
Qwen3-32B
32B
10.72
7.80
0.94
Search-R1-32B
32B
11.08
10.17
1.69
FRUGALRAGG-B++
Qwen2.5-7B-Inst (HotPoiQA)
7B
20.46
23.57
7.95
FRUGALRAG-7B+
Qwen2.5-7B-Inst (2Wiki)
7B
21.53
22,93
10.96
FRUGALRAG-7B Qwen2.5-7B-Inst (MuSiQue)
7B
21.14
23.73
8.39
The ReAct Few Shot (FS) baseline (Yao et al. [2023] Khattab et al. (023) outperforms vanilla Retrieval Augmented Generation. The optimized prompts enable ReAct to generate more search queries, and as result achieve very strong performance. Strikingly, on HotPotQA, ReAct achieves (a) 79.60 % which greater than the recall achieved by LeReT (Hsu et al. 2024) (77.1 % using Llama3. 1-88) after significant finetuning; signifying the importance of building strong baseline.
Table I demonstrates the overall effectiveness of FRUGALRAGE AG-Exporre achieving the highest Recall and MBE scores compared all baselines. However, note that TRUGALRAG-ExTE best or second best in terms of both answer and recall metrics but introduces a very high latency compared to FRUGALRAG. Stage-2 finetuning significantly reduces the number of searches required while retaining the performance across three datasets. For instance, the search count reduces on average by R50% across the three datasets while the MBE scores improve. Overall, results highlight that FRUGALRAG strikes a significantly better efficiency-accurran tradeoff.
is either
4.3 COMPREHENSIVE BASELINE COMPARISONS
Table demonstrates the performance of FRUGALRAG against 14 baselines and state- tfitee-of-heer the-art-rt RAG methods. For fair comparison, we setup our experiment using FlashRAG (Jin et al.) 2025b) with common retriever backend, E5-Base- V2 and index of 21 million wikipedia passages.
FRUGALRRAG achieves strong overall performance across all three benchmarks. On HotPotQA, attains 58.5% MBE with Qwen2.5-7B and improves 61.4% with Qwen2.5-32B while being trained on only 1000 examples. Notably, it also achieves the highest retrieval recall (70.4%), indicating merely due stronger generation but are supported by improved evidence
that its gains acquisition. In contrast, the second-be best method, CoRAG, relies on a fixed inference- time budget, which may be inefficient for simpler questions and inflexible for more complex ones. FRUGALRAG instead adaptively allocates search steps, using approximately 2.89 searches on average (HotPotQA).
On 2Wiki and MuSiQue, FRUGALRAG remains competitive, ranking best or second-best in MBE while maintaining strong recall. Although CoRAG achieves the top MBE on these datasets, it relies on over 100k training examples, joint training of the answer generator, and a fixed search budget. In comparison, FRUGALRAG matches closely approaches these results with two orders of magnitude fewer training examples and consistently strong retrieval performance. Additionally, FRUGALRAG uses off-the-sher answer generator without additional fine-tuning. This is important because for other methods, performance gains may partially stem from generator training rather than improvements in multi-hop retrieval. By decoupling retrieval from generation and still achieving
8

---

Published as a conference paper at ICLR 2026
Table FRUGALRAG issues more queries for harder questions. We see clear increasing trend in num. of search queries with amount of ground truth evidence (2Wiki) and ground truth num. of hops.
2WikiuulcHopPAA
MuSiQue
Num
Number Questions
Actual Hops
Number Questions
Searches
Scarches
GT Evidence
9,595
2.665 + 1.430
2
1,252
3.054
1.433
88
2.875 + 1.483
3
760
3.924
1.464
2,806
3,909 + 1.502
4
405
4.205 + 1.368
Table 5: Comparison of SFT baselines and FRUGALRAG across datasets using the Efficiency Tradeoff metric (Eff) = (MBE + Re Searches). R: Recall, MBE: Model-Based Evaluation,
S: Searches.
Method
HotpotQA
2Wiki
Musique
MBE
S
Eff.
R
MBE
Eff.
MBE
S
Eff.
E5
SFT (with FINISH)
57.8
2.84
22.31
52.6
43.8
3.48
13.85]
52.5
31.8
3.43
12.29
FRUGALRAG-T AG-Explore-7B
77.0
59.60
10.88
6.28
64.3
46.80
10.79
5.15
51.7
35.30
10.85
4.01
GRUGALRAG-77 Qwen2.5-7B-Inst
70.4
58.5
2.89
22.30
58.8
50.4
3.03
18.02
53.30
36.40
3.30
13.59
ColBERTv2
SFT (with FINISH)
79.90
67.70
2.07
35.65
63.70
48.50
3.13
17.93
51.70
29.80
3.25
12.54
Qwen2.5-77 5-7B-Inst
86.40
67.70
5.99
12.87
668.90
47.60
5.99
9.73
559.10
30.10
5.93
7.52
BOUGALRAG-TE -QWeen.5-77-Imm
82.80 68.50 2.05 36.90
63355
48.90 2.95
19.05
55.00
33.70
3.02
14.52
competitive results, FRUGALRAG isolates the contribution of adaptive search and demonstrates that strong multi-hop reasoning can be obtained without end-to-end retraining of the generator. Finally, the improvements obtained by swapping the answer generator (e.g., from Qwen2.5-7B Qwen2.5-32B or CoRAG) demonstrate the modularity of FRUGALRA RAG, confirming that its benefits stem from the search-and-retrieval strategy rather than reliance on a specific generator.
4.4 RESULTS DEEP RESEARCH
We evaluate FRUGALRAG BrowseComp-
recent, challenging deep research dataset that demands massively multi-hop search and reasoning. Strikingly, our models trained HotPotQA, MuSiQue, and 2Wiki readily generalize to harder datasets like BrowseComp-PPuu In Table 3 show that FRUGALR issues between 7-10 queries on average despite being trained with a budget of 6 queries. We show that FRUGALRAG achieves superior performance compared to many larger models like Search-RI 32B, Qwen3-32B (Yang et al., 2025), and DeepSeek-RI (DeepSeek-Al on both answer accuracy (MBE) and recall.
2025)
4.5 ANALYSIS
FRUGALRAG is Adaptive. Table demonstrates that FRUGALRAG adaptively scales test-time compute compute-measured measurre by the number of search queries-in proportion to question difficulty. To operationalize difficulty, we leverage MuSiQue, which provides explicit annotations for the expected number of reasoning hops, and 2Wiki, where we use the number of ground-truth evidence documents as proxy for reasoning complexity. To rigorously assess the correlation between question hardness and the number of queries issued by FRUGALRAG, we evaluate over the entire development sets of both datasets (approx. 12k and 2.5k examples, respectively). We observe strong positive correlations: r = 0.82 on 2Wiki and r = 0.95 on MuSiQue. These results indicate that FRUGALRAG effectively allocates additional computation in response to increased multi-hop reasoning demands.

---

Published as a conference paper at ICLR 2026
ColBERTV2
39.3
30-
2.5
22.
28-3
34.0
3Y.
Efficlency
13.9
12.
125
10-
75
HolpotOA
2W1KMOA
Musique
HorpalQA
2WIKIMOA
Musique
Finish)
FragaiRAG BEAAAAGG-pppo-77755
Figure I: FRUGALRAG on average outperforms fixed budget and SFT based baselines, demonstrating the effectiveness of both Stage-1 finetuning and Stage-2 learning control test time compute. The plots show the Tradeoff metrics (See Table 5 for detailed results).
FRUGALRAG is robust. In Table 14 we report results for models trained on 1000 examples from HotPotQA, 2Wiki, and MuSiQue and evaluated across all datasets, testing FRUGALRAG's ability to generalize to unseen scenarios. Despite the distribution shift, FRUGALRAG maintains strong performance without compromising efficiency. This robustness holds for both E5-base-v2 and ColBERT-v2 retrievers and different answer generators.
FRUGALRAG outperforms SFT. To capture the trade-off between answer quality and retrieval cost, we define simple metric: Efficiency Tradeoff: (MBE + Recall) /2 Searches). This metric allows us to jointly evaluate answer accuracy and evidence quality relative to the number of retrieval steps, enabling comparison of efficiency across models and budgets. To assess the impact of the RL-finetuning (Stage 2), we conduct an ablation study by comparing FRUGALRAC against simple baseline, called SFT (with FINISH). This baseline is trained for 1 epoch on the dataset generated during Stage 1, but with one key modification. Instead of sampling 90% rollouts without FINISH, sample all traces where the model outputs FINISH its own In Fig. [ demonstrate that FRUGALRAG on average across all models is better on both Efficiency Tradeoff. FRUGALRAC outperforms both FRUGALRAG-SEP and FRUGALRAG-Exploo by significant margin using 1ren2.5-7BIInstruce on all the datasets as illustrated in Fig.
CONCLUSIONS, LIMITATIONS, AND FUTURE WORK
In this work, showed that reinforcement learning can be used optimize the search queries issued by RAG system, based on query difficulty. We found that a simple ReAct baseline that iteratively retrieves (by invoking the search tool) and reasons decide what search call to issue next) is quite competitive, especially if we can optimize its few-shot prompt using just tens of training examples. We proposed two-stage framework FRUGALRAG that a) works with 1000 training examples, compared to state-of-the-ar RAG techniques that use over 100,000 examples, and b) yet achieves competitive accuracies while also using far fewer search queries at inference time, on popular multi-hop QA datasets.
REFERENCES
Paul J. L. Ammann, Jonas Golde, and Alan Akbik. Question decomposition for retrieval-augmented generation, 2025. ltttaa wra.iiioorrr/aaae wwt.///g///aaaa.ae 2507. 00355
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference Learning Representations, 2023.
Chi-Min Chan, Chunpu Xu, Ruibin Yuan, Hongyin Luo. Wei Xue, Yike Guo, and Jie Fu. Rq-rag: Learning to refine queries for retrieval augmented generation. arXiv preprint arXiv:2404.000 2024.
10

---

Published as a conference paper at ICLR 2026
Mingyang Chen, Tianpeng Li, Haoze Sun, Yijie Zhou, Chenzheng Zhu, Haofen Wang, Jeff Z Pan, Wen Zhang, Huajun Chen, Fan Yang, et al. Learning to reason with search for llms via reinforcement learning. arXiv preprint 2rXiv:2503.1944
2025a.
Zijian Chen, Xueguang Ma, Shengyao Zhuang, Ping Nie, Kai Zou, Andrew Liu, Joshua Green, Kshama Patel, Ruoxi Meng, Mingyi Su, Sahel Sharifymoghaddam. Yanxi Li, Haoran Hong, Xinyu Shi, Xuye Liu, Nandan Thakur, Crystina Zhang, Luyu Gao, Wenhu Chen, and Jimmy Lin. Browsecomp-plus: A more fair and transparent evaluation benchmark deep-research agent. arXiv preprint ar aww.::2508..00060 066000000660,,.000 2025b.
DeepSeek-AI. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning, wwg./og/g/aa/aae 22911222488118
2025. URLbttps://arre
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming- Wei Chang. Realm: Retrieval- augmented language model pre-training, 2020. URL. btctps://arxiv.orgge
08909
Xanh Ho. Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing multi- hop QA dataset for comprehensive evaluation reasoning steps. In Donia Scott, Nuria Bel, and Chengqing Zong, editors, Proceedings of the 28th International Conference Compu- tational Linguistics, pages 6609-6625, Barcelona, Spain (Online), December 2020. Interna- tional Committee on Computational I Linguistics. doi: 10. 1866/////002..ooooggm.ooo/
URL
org col..//ooo....oooo.ooooo. main. 5807
Sheryl Hsu, Omar Khattab, Chelsea Finn, and Archit Sharma. Grounding by trying: Lims with reinforcement eenannegenhaneed retrieval. arXiv preprint arXiv:2410.23214, 2024.
Soyeong Jeong, Jinheon Back, Sukmin Cho, Sung Ju Hwang, and Jong C Park. Adaptive-rag: Learning to adapt eetrieval-augmeen large language models through question complexity. arXiv preprint arXiv:2403. 14403, 2024.
Zhengbao Jiang, Frank Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi- Yu, Yiming Yang, Jamie Callan, and Graham Neubig. Active retrieval augmented generation. In Proceedings the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7969-7992, 2023.
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training Ilss reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516 2025a.
Jiajie Jin, Yutao Zhu, Zhicheng Dou, Guanting Dong, Xinyu Yang, Chenghao Zhang, Tong Zhao, Zhao Yang, and Ji-Rong Wen. Flashrag: A modular toolkit for efficient retrreeal-augmenned generation research. Guodong Long, Michale Blumestein, Yi Chang, Liane Lewin-Eytan, Zi Helen Huang, and Elad Yom-Tov, editors, Companion Proceedings of the ACM on Web Conference 2025, WWW 2025, Sydney, NSW, Australia, 28 April 2025 2 May 2025, pages 737-740, ACM, 2025b. doi: 10.1 10.1145/700171133715311 URL/attps: doi org. 10. 1145/110111661.11 37533
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick SH Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In EMNLP (1), pages 6769-6781, 2020.
Omar Khattab, Arna Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vard- hamanan, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, et al. Dspy: Compiling declarative language model calls into self-improving pipelines. arXiv preprint arXiv:2310.03714, 2023.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler. Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmmented generation for knowledge-intene intensive nlp tasks, 2021. (thhts
7/a //arxiv.com //aaxii..orr//bs///000. 11401
11

---

Published as a conference paper at ICLR 2026
Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. Search-ol: Agentic search-enhanced large reasoning models. arXiv preprint arXiv:2501.05366, 2025.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
Jianbiao Mei, Tao Hu, Daocheng Fu, Licheng Wen, Xuemeng Yang, Rong Wu, Pinlong Cai, Xinyu Cai, Xing Gao, Yu Yang, Chengjun Xie, Botian Shi, Yong Liu, and Yu Qiao. O*-searcher: A searching-basee agent model for open-domain open-ended question answering, 2025.
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao. James Thoree, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. Kilt: a benchmark for knowledge intensive language tasks, 2021. URL
htttps://axii.. wtt. 20099 02252
Jeff Rasley, Samyam Rajbhandari. Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimiza- tions enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 3505-3506, 2020.
Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. Colbertv2: Effective and efficient retrieval via lightweight late interaction. arXiv preprint 12.04221 2021.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances Neural Information Processing Systems, 36:68539 68551, 2023.
Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen. Enhancing retrieval- -ugmented large language models with iterative retrieval-generation synergy. arXiv preprint arXiv:2305. 15294, 2023.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint 240.:220000030000, 2024.
Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. R1-searcher: Incentivizing the search capability in Ilms via reinforcement learning. arXiv preprint arXiv:2503.05592 2025.
Weihang Su, Yichen Tang, Qingyao Ai, Zhijing Wu, and Yiqun Liu, Dragin: Dynamic retrieval augmented generation based on the information needs of large language models, 2024. URL
https://arxiv>
org/ abs 2403. 10081
Shuang Sun, Huatong Song, Yuhao Wang. Ruiyang Ren, Jinhao Jiang, Junjie Zhang, Fei Bai, Jia Deng, Wayne Xin Zhao, Zheng Liu, Lei Fang, Zhongyuan Wang, and Ji-Rong Wen. Simpledeepsearcher: Deep information seeking via web-powered reasoning trajectory synthesis. ArXiv, abs/2505.166 16334, 2025. http://attpppppaaaaatttttiaaiia. http:////...p..mmiiiiccccooooooo
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. arXiv preprini arXiv:2212. 10509, 2022a.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique Multihop questions via songle-hop question composition. Transactions of the Association for Computational Linguistics, 10:539-554, 2022b.
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforcement learning. https: http://ggtttt...ooo///gggggggeee
2020.
12

---

Published as a conference paper at ICLR 2026
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly- supervised contrastive pre-training arXiv preprint arXiv:2212.03533, 2022.
Liang Wang, Haonan Chen, Nan Yang, Xiaolong Huang, Zhicheng Dou, and Furu Wei. Chain-of- retrieval augmented generation. arXiv preprint urXiv:2501.14344 2025.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24831 2022.
Ran Xu, Wenqi Shi, Yuchen Zhuang, Yue Yu, Joyce C. Ho, Haoyu Wang, and Carl Yang. Collab- rag: Boosting retrieeal-augmented generation for complex question answering via white-box and black-box Ilm collaboration, 2025.
org/ abs/ 2504 04915
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint (xXx:22555009880 2025.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference Learning Representations (ICLR), 2023.
Ori Yoran, Tomer Wolfson, Oni Ram, and Jonathan Berant. Making mereeiigeettie language models robust to irrelevant context, 2024. URL
01858
Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. Qwen3 embedding: Advanc- ing text embedding and reranking through foundation models. arXiv preprint 200::2006.05766 2025.
13

---

Published as a conference paper at ICLR 2026
Recall (%)
70.4% (8=6)
4.0
Aug. Searches
70
3.5
3.0
Avg. Searches
Recall (%)
65
2.5
60
2.0
-155
0.5
11
Budget B
Figure 2: FRUGALRAG tested using different maximum budgets B on HotPotQA. We find that approach achieves maximum recall at the training budget followed by diminishing returns subsequent searches.
A ADDITIONAL RESULTS
A.1 ADDITIONAL ABLATIONS
Effect of changing Budget B. We analyze the sensitivity of FRUGALRAG to the retrieval budget at inference time on HotPotQA. Specifically, we train with maximum budget of and evaluate it under range of budgets B E {1...., 11}. Figure reports recall and the average number of searches executed.
As the budget increases, the model adaptively utilizes the additional compute by issuing more searches. Recall improves sharply when moving from very small budgets (B = 1-2) to moderate budgets (B- = 3-6), indicating that additional retrieval steps in this regime meaningfully expand the available evidence. Beyond = 6, improvements become marginal despite a steady increase in search count. This saturation behavior is consistent with diminishing returns from additional retrieval once sufficient supporting evidence has been gathered.
Notably, the highest recall is achieved B 6. which matches the training- time budget. This is expected: the policy has learned retrieval strategies calibrated to this constraint. Similar saturation trends are observed on MWikioultHHppAA and MuSiQue. On substantially more complex datasets, however, larger budgets may continue to yield gains. Exploring training under variable or adaptive budgets is a promising direction for future work.
Sample Count Ablation. We analyze the impact of the number of supervised training examples on retrieval performance and efficiency in Fig. 3 Across all three datasets, recall improves substantially as the number of training examples increases from 100 to 500. For instance, on HotPotQA, recall improves from 67.9 to 70.8 when scaling from 100 to 500 examples. The same upward trend is observed on 2WikiMultiHopQ.A and MuSiQue, indicating that the policy rapidly acquires the core retrieval and termination behaviors with only a few hundred annotated instances.
Beyond 500 examples, gains become more incremental but remain consistent. Notably, search efficiency stabilizes-and in some cases improves-win additional data. At 5,000 training examples, the average number of searches decreases consistently across all three datasets, suggesting that the policy not only improves recall but also learns to allocate retrieval steps more judiciously.
Overall, these results indicate that FRUGALRAG operates effectively in the low-data regime, achiev- ing strong performance with relatively limited supervision while continuing to benefit from additional examples. Nevertheless, these findings should be interpreted with care, experiments focus pri- marily on small-scale supervision, and comprehensive study of large-scale scalability a future direction.
FRUGALRAG is agnostic base model. To evaluate whether FRUGALRAG generalizes beyond the (wen2.5-7BB.. InnBrunst base model, conduct additional experiments using Llama3.1 1--aa...18B.... I-sttruttutt...
14

---

Published as a conference paper at ICLR 2026
Recall
3.5
Tucall
Hetpaag)
2.0
500
Triining Simples
Trxining Sumples
Figure 3: FRUGALRAG trained with varying number of supervised examples. We demonstrate that FRUGALRAG is robust even low-data regimes and performance improves consistently with increasing number of examples.
Base Model
Dataset
Recall (%)
Avg. Searches
FrugalRAG Sivven..57B-Instruurr
HotPotQA
70.40
2.89
FrugalRAG (LJama331-88-Instmu) 1-8B-lnsttruct
HotPotQA
70.00
2.95
FrugalRAG (Qwen2,5-7B-Instte InItttttt
2WikiMultHHoPAA
60.40
3.62
FrugalRAG (Llama3.1 (L-ama3.1-8B-Inns. 1-8B-Instrruct
2WikiMultHHoQAA
60.41
3.79
FrugalRAG Brrenn.5--B-Insttiee
MuSiQue
53.70
3.74
FrugalRAG (Llama3, 1LJamma..--B-Insttee 1-8B-Insttruct
MuSiQue
53.00
3.96
Table Performance comparison across different base models.
In this setting, Instruct and fine-tune Llama3. 1amma..1-BB. I-8B-Instruct under the same two-stage pipeline.
the Stage-1 supervised training data originally generated with Qwen2.5-7B-
As shown in Table performance remains highly consistent across base models all three datasets. Recall differences are marginal (within ~0.7 points), and search behavior remains comparable. These results indicate that FRUGALRAG mgdel-agnostic and that the gains stem from the training framework.
B REWARD DESIGN
We establish the efficacy of FRUGALRAG by conducting additional reward design ablations.
Stage-1 1 is necessary. We first evaluate Stage-2 without Stage-I on HotPorQA, where we directly apply RL fine-tuning to the base Qwen2.5-7B IIIItrutct policy without prior SFT. As shown in Table 7 removing Stage-1 substantially degrades recall (66.5 vs. 70.4). This demonstrates that SFT is a cnucial step that teaches the model essential reasoning and exploration, stabilizing the subsequent RI. optimization which uses the proposed "learning to stop" criterion.
Alternative rewards. We analyze alternative reward formulations and training strategies Hot- PoIQA. Table summarizes the results. Directly training Qwen2.5--BBIInstrucc with a recall reward yields substantially worse performance (66.8%), showing that naively applying RL does not recover the behavior of FRUGAL AG. Moreover, optimizing recall alone PRUGALRAA-EXPIL. matically increases search usage (5.76), indicating poor cost control. Furthermore, jointly optimizing both query generation and stopping via recall reward, overall recall decreases (68. (68.3)), suggesting that simultaneous optimization of both behaviors increases training difficulty and leads to suboptimal convergence. Finally, Table Rable@eee demonstrates that removing the late-stop penalty yields similar recall (70.5%) but increases the average number of searches (3.144 vs. 2.89). This verifies that the penalty term effectively removes redundant retrieval steps without sacrificing accuracy. Across all ablations, FRUGALRAS consistently achieves the strongest balance between recall and search efficiency. These experiments demonstrate that the improvements not merely the result of generic RL regularization, but stem from (i) the two-stage training procedure and (ii) the specific reward design that explicitly balances recall gains against retrieval cost.
dra-
15

---

Published as a conference paper at ICLR 2026
Ablation
Recall (%)
Avg. Scarches
Stage-2 w/o Stage-1 (wlo SFT)
66.5
2.36
FRUGALRAG (Ours)
70.4
2.89
Table 7: Effect of removing Stage-1 supervised fine-tuning,
Ablation
Recall (%)
Avg. Searches
Dwve2..5-7B-Instucct + Recall Reward
66.8
2.57
SntealRAG-Exploor + Recall Reward
76.0
5.76
FrugalRAG + Recall Reward (Query Opt)
68.3
2,45
FrugalRAG w/o Late Penalty
70.5
3.144
FrugaIRAG (Ours)
70.4
2.89
Table 8: Reward and optimization ablations on HotPoiQA.
B.1 COMPARISON AGAINST HEURISTIC STOPPING RULES
To further isolate the contribution of our learned termination policy, we compare FRUGALRAG against fixed heuristic stopping strategies that do require training separate model. This ensures that any performance differences are attributable to our reward design rather than regularization. Other heuristic-ster style stopping mechanisms (e.g.. ReAct (Yao et 2023) and IRCOT (Trivedi et al.. 2022a)) are already included as baselines in Table 2 of the main paper. Here, we focus specifically on deeth-deph stopping rules derived from dataset statistics. Table summarizes the results.
1. Stopping Heuristic
Depth). We first estimate the retrieval depth at which recall peaks on the training set under the FRUGALRAGE Baw.RRAG-EEPOOOO policy. The estimated depths are: HotPotQA (2.96), 2WikiMuliHopQA (1.93), and MuSiQue (4.96). These values are rounded to the nearest integer and used as fixed stopping depths at test time.
2. Stopping Heuristic (Ground- (Cround-Truth Average Depth). In a second variant, we compute the average number of retrieval steps required recover the ground-truth evidence chains in the training set: HotPotQA (2.0), 2WikiMuliHHoPQA (2.40), and MuSiQue (2.32). These averages are rounded the nearest integer and used as constant stopping depths during inference.
Across HotPorQA, 2WikiMultiHopQA, and MuSiQue, fixed-depth heuristics consistently under- perform the learned FRUGALRAC policy. Although heuristic stopping based on peak-recall depth can approach competitive performance on easier datasets (e.g., HotPotQA), it fails to match the efficiency-accuree trade-off achieved by FRUGALRAAG In particular, on more challenging datasets such as 2WikiMultiHopQA and MuSiQue, where the optimal number of reasoning hops varies substantially across instances, the gap widens.
These results demonstrate that static stopping rule, even when calibrated using training statisties or ground-truth evidence chains, cannot capture instance-specific termination dynamics. In contrast, our RL-based policy adaptively determines when to stop retrieval conditioned on the evolving reasoning state, yielding consistently higher recall with competitive lower search budgets.
B.2 ADDITIONAL EFFICIENCY ANALYSIS
We additionally compare computational efficiency across methods by reporting total processed tokens, approximate FLOPs, end-to-end latency, and average search count. Tables 10 the results on HotPotQA, 2WikiMuliHHpoAA, and MuSiQue, respectively.
II
and 12 summarize
Across datasets, FRUGALRAG remains competitive on compute-normalizee metrics, including total tokens, FLOPs, and search count. While some baselines achieve lower latency due to fewer overall decoding steps, they typically do so at the of substantially lower recall. In contrast, FRUGALRAG consistently operates a favorable seffeiencc-acccrran regime, matching or exceeding the best-performing baselines in recall while maintaining comparable token usage and search budgets.
16

---

Published as a conference paper at ICLR 2026
Dataset
Method
Recall
Avg. Searches
Heuristic 10wen2.5-7B-Insttee
67.7
Heuristic BruugalRAG-Explorg
70.9
HotPorQA
Heuristic MingalRAAG-Eploore
67.9
FrugalRAG (Ours)
70.4
2.89
Heuristic 1/2 (Qwen2.5 10-n2...-7-lnstrruoe
52.0
2WikiMultHHpQA
Heuristic 1/2 SegaaRAA-EEploor
55.4
2
FrugaIRAG (Ours)
58.8
3.03
Heuristic (Qwen2.5-72) (---ml25--7-Inssreee
51.8
Heuristic TrrugalRAG-Exploos
54.5
MuSiQue
Heuristic (FrugalRAG-Expil Exxllor))
43.8
FrugaIRAG (Ours)
3.30
Table 9; Comparison between FRUGALRAG and fixed heuristic stopping strategies.
HotPotQA
Method
Avg Tokens
FLOPs
Latency (s)
Scarch Count
Recall (%)
02 Scarcher-3B
2,937
1.79 x 10
0.1050
1.77
50.10
SimpleDeepScarcher--
9,657
1.31 x 10
0.6571
2.75
64.80
RI Searcher-7B
4,458
6.17 x 10
0.1335
2.22
69.10
Search-R1-7B
2,119
2.99 x 1013
0.0650
1.28
48.20
CoRAG-8B
7,770
1.24 x 10
0.1486
4.00
64,30
FrugalRAG-7B (ours)
9,138
1.27 x 10"
0.2415
2.89
70.40
Table 10: Efficiency comparison on HotPotQA.
Importantly, in practical settings, monetary cost is dominated primarily by (i) the number of tokens generated and (ii) the number of external tool calls (i.e., search operations). Under these cost-relevant metries, FRUGALRAGE is directly comparable to-and in several cases more efficient than-strong multi-hop retrieval baselines. This indicates that the gains retrieval effectiveness do not come from disproportionate increases in computational or tool usage, but rather from more effective, adaptive allocation of retrieval steps.
C ADDITIONAL EVALUATIONS
In multi-hop QA, retrieval differs from standard IR because sequential queries rather than producing single ranked list. The key metric in multi-hop retrieval is recall over all the steps, since missing any required evidence makes the reasoning worse. Prior work like CoRAG and Search-RI only focus on answer-level metrics. To strengthen our evaluation, we report document-level overall precision and Fl in addition to recall. These metrics capture retrieval quality without relying on ranking assumptions that do not hold in the multi-hop setting. Table 13 summarizes results.
D STABILITY OF RL TRAINING
We conduct three independent Stage runs using 1000 examples from the HotPotQA datasets and test on all three dataset. The recall on HotPot evaluation is 69.60 + 1.21 using 2.74 + 0,16 scarches average. On 2WikiMultiHoPAA and MuSiQue achieve recall of 59.20 + 1.11 (scarches 3.60 + 0.11) and 53. 10 + 1.4 (searches- 3.42 + 0.08) respectively.
QUALITATIVE EXAMPLES
In Table 4 we show that with increasing difficulty, FRUGALRAG on average adaptively issues more queries. To further this claim, provide qualitative examples MuSiQue in Table 15
17

---

Published as a conference paper at ICLR 2026
2WikiMultiHopAA
Method
Avg Tokens
FLOPs
Latency (s)
Search Count
Recall (%)
02 Searcher-3B
4,212
2.53 x 10
0.1206
2.42
55.20
SinpleeeeppSercchr>>)
13,433
1,80 x 10
0.7625
3.64
60.50
R1 Scarcher-7B
4,939
6.89 x 10t3
0.1374
2.36
60.40
Search-R1-7B
3,456
4.96 x 10
0.0852
1.89
47.70
CoRAG-8B
8,018
1.28 x 10"
0.1511
4.00
65.40
FrugalRAGG77B (ours)
11,279
1.45 x 10
0.4673
3.03
59.90
Table 11: Efficiency comparison on 2WikiMultHHoPQA.
MuSiQue
Method
Avg Tokens
FL.OPs
Lateney (8)
Search Count
Recall (%)
02 Searcher-3t 3B
2,923
1.74 x 10
0.1009
1.95
37.00
SimpleDeepSSarchee--71
10,027
1.35 x 10
0.7366
2.86
50.40
RI Searcher-7B
4,665
6.47 x 10
0.1327
2.31
57.70
Search-R1-7B
2,212
3.14 x 10th
0,0676
1.36
38.10
CoRAG-8B
7,683
1.23 x 10
0.1472
4.00
54.00
FrugaIRAG-7B (ours)
11,914
1.66 x 10
0.4098
3.30
52.60
Table 12: Efficiency comparison on MuSiQue.
F TRAINING DETAILS
Algorithm T shows the overall training framework of FRUGALRAG. We plan to publicly release code soon. Below, we discuss each step along with their implementation details.
Few-Shot Prompt Optimization Details. We leverage DSPy (Khattab al. 2023) for automatic few-shot prompt generation following LeReT (Hsu et al. 2024). Specifically, we use 100 training SESTTRAAFFWSHOTWWW.....
examples (Limi) with the
ANDOMSEARCH method, which the LM f to generate few-shot examples, selecting the best performing ones for subsequent prompting. Using more advanced prompting techniques may yield further improvements. We select 4 best performing few-shot prompts from a total of 15 candidate sets using the sum of answer EM and answer passage match. Answer EM checks for an exact string-match between the generated and actual answer, and passage match checks if the actual answer is present in the retrieved passages. This step crucial because it facilitates dataset generation using diverse rollouts and ensures the answer format is followed by the model. For this step, we serve our model on one GPU using VI.LM (Kwon) al. 2023).
Dataset Generation Details. For each few-shot prompt Pi, the model f generates tuple (T) A S representing a candidate output for the next hop. As described in Sec. 3.1] we evaluate all candidate tuples at hop h and select one with the highest recall. This selected candidate is then used as the context for the next hop and the process is repeated till budget B (optionally till the selected candidate action A, indicates FINISH). We set the budget B 6, where the initial retrieval step is always RR(Q())) with Q6i denoting the original user utterance. The generated dataset is denoted by D. For all experiments involving Qwen2.5, we utilize the 7B-Instruct variant along with its prompts to generate the dataset. For further improving results, can repeat few shot prompt optimization and dataset generation using different base models.
Supervised "Explore" Finetuning Details. We use the standard next token prediction loss given by:
max (Zz)) log (xly)
where y = (Th, An, Sw) Sk)) and x = Q U (Te, St. D& D sampled from the generated dataset D.
We train the model f for epoch using a batch size and apply gradient accumulation of 2 steps. resulting in effective batch size of 8. Optimization is performed using AdamW (Loshchilov and
18

---

Published as a conference paper at ICLR 2026
Dataset
Model
Rec
Prec
F1
Searches
O2 Searcher
50.10
22.29
30.85
1.77
SimpleDeepSearcher
64.80
23.43
34.42
2.75
Searcher
69.10
24.25
35.90
2.22
HotPotQA
Search-R1
48.20
20.24
28,51
1.28
CoRAG
64.30
24.03
34.99
4.00
FrugaIRAG (ours)
70.40
26.81
38.83
2.89
02 Searcher
55.20
21.99
26.42
2,42
SimpleDeepSSercher
60.50
17.97
24.38
3.64
Searcher
60.40
21.60
29.29
2.36
2wikiMultiHoOQA
Search-R1
47.70
15.89
20.78
1.89
CoRAG
65,40
22.79
30.67
4,00
FrugaIRAG (ours)
59.90
15.45
23.09
3.03
02 Searcher
37.00
14.25
21.37
1.95
SimpleDeepSearcher
50.40
14.79
22.87
2.86
Searcher
$7.70
15.77
24.77
2.31
MuSiQue
Search-RI
38.10
12.65
19.86
1.36
CoRAG
54.00
16.71
25.97
4,00
FrugaIRAG (ours)
52.60
15.13
23.96
3.30
Table 13:
Retrieval metrics across datasets. We report Recall (Rec), Precision (Prec), FI, and average number of searches.
Table 14: FRUGALRAG demonstrates robust cross-dataset performance. The model is adapted using 1,000 training examples from the dataset shown in parentheses.
Training Dataset
HotPotQA
2Wiki
MuSiQue
Recall MBE Searches Recall MBE Scarches Recall
MBE Searches
FRUGALRAG-77 Qwen2.5-7B-Inst (ES)
HotPotQA
70.40
58.50
2.89
60.40
54,20
3.62
53.70
36.60
3.47
2Wiki
71.00
59.30
2.85
58.80
50.40
3.03
52.22
36.10
3,38
MuSiQue
69.10
57.50
2.72
59.70
51.30
3.33
53.30
36.40
3.30
FRUGALRAG-7B + Owwen..5332-Im Bisst (ES)
HotPotQA
70.40
61.40
2.89
60.40
54.20
3.62
53.70
37.60
3.47
2Wiki
71.00
61.10
2.85
58.80
50.90
3.03
52.22
38.90
3.38
MuSiQue
69.10
60.10
2.72
59.70
52.40
3.33
53,30
38.90
3.30
FRUGALRAG
@wen2.5-7B (CoIBERTv2)
HotPotQA
82,80
68.47
2.05
64.20
48.47
3.07
53.80
36.27
2.75
2Wiki
81.90
68.34
2.56
63.50
48.93
2.95
53.80
34.30
3.10
MuSiQue
83.10
61.11
2.53
60.80
46.41
3.13
51.70
29.80
3.02
Hutter 2017) with learning rate of x 10-5, We use linear learning rate scheduler with warmup phase of 20 steps. The training is performed using 8 H100 80GB GPUs.
Controlling test-time compute with RL. Our RL step employs GRPO for fine-tuning the base policy fs. Specifically, following the notation in DeepSeekMath (Shao et al. 2024). for each question Q() we sample group of outputs {or of ..... o% at hop h, where v is set to 8. We optimize our base policy fs using the standard GRPO objective using the cumulative rollout reward as defined in Eq 1 We use KI. divergence penalty with weight 0.1 since we have a trained base policy, and set the maximum reward Rmmax 2.0 for stability. Generation limited a maximum of 256 completion tokens and the maximum prompt size is 1024. Training is conducted using DeepSpeed-Zero: (Rasley et al. 2020) and 7 H100 GPUs (where 1 is exclusively reserved for sampling). We set the learning rate 10 Due to the long prompt (which includes retrieved documents from previous hops), use a total batch size of 48. We train FRUGALRAG for 400 steps across datasets and models, and report the performance using the final checkpoint.
19

---

Published as a conference paper at ICLR 2026
Algorithm Our novel two-stage framework, FRUGALRAG consists of (1) Dataset Generation and Supervised "Explore" Finetuming, and (2) Controlling test-time compute wth RL.
Input: Labeled dataset L {(Q 1 Lintt {(Q6), Y retriever R, base
LM f, budget B, max hops m, number of samples u
// Prompt Optimi zat ion Perform prompt optimization on f using Lma to obtain few-shot prompts (pi....,P.l.
// Dataset Generation on Initialize finetuning dataset: x for Q(), y() in L do
Initialize buffer:
+- []: Initialize Dn + R(QQ1) 0r
Initialize To, Ao:
Ha {Q To, Ao, Do} // stores previous context Append Hn to main_rollout;
for h 1 to do
10
for i in 1... n do
for h 1 to do
(Ti, Ag, Sh) (H
occurs in 10% of calls
if A FINISH then break
15
Dj + R()));
Remove duplicate retrievals from Dj
Hi + H U (T7, Sy, S D)
Evaluate all {Di (recall against ground truth y())
18
Select best- best-performing trajectory H";
20
Append H main_rollout:
21
Append each hop from main main_rrcloutt to D:
// Stage 1:
Supervised "Explore" Finetuning
fs Fine-tune f on D using standard next-token prediction // See Eq.
Controlling test-time compute with RL 23 for Q(1,, y() in do
// Stage 2:
24
for h = 1 to m do
L Generate sample tuples Af. S Dj D for hop h;
25
26
for to
Compute reward R² )( ,(),,, Backpropagate loss on [T), Ai. St m using R!:
Y(), fs) // See Eq.
27
28
20

---

Published as a conference paper at ICLR 2026
Question
bbrreviated Reasoning Trace
Searches
Who is the spouse of the screenwriter of The Ac- tress?
TO: Search original question.
T1: Initial results mention unrelated screenwriters (c.g., David Fury); refine query.
T2: Search "screenwriter of The Actress movie".
T3: Identify screenwriter Ruth Gordon.
T4: Search "Ruth Gordon spouse" - Garson Kanin.
What administrative territorial entity the owner of Ciudad Deportiva located?
TO: Search original question.
2
Identify location: Benito Juárez borough, Mexico City.
T2: Refine query to determine owning administrative entity + Local gov-
ernment of Madrid.
In which borough was Callum McManaman born?
TO: Search original question.
T1: Retrieve birthplace: Huyton + Borough of Knowsley.
Table 15: Qualitative reasoning traces demonstrating adaptive retrieval depth. More complex multi- hop questions require iterative query refinement and additional searches, whereas simpler factual queries terminate early.
G DATASET AND RETRIEVAL INDEX
use the pre-processed Wikipedia abstracts index provided by ColBERTv2 (Santhanam et al.
202I) for all our experiments HotPotQA (Yang et al. 2018). For each instance, retrieve the top 3 documents and their titles and perform a maximum retrievals. HotPotQA annotations consists of document title and evidence sentences which used to compute the Recall and Supporting Document F1 respectively.
Since 2WikiMultHHopAA (Ho al.) 2020) and MuSiQue (Trivedi et al. 2022b) datasets created using both the body and abstract of wikipedia articles we use the pre-processed dump of Wikipedia provided by Karpukhin al. (220)) and index it using ColBERTv2 (Santhanam al., 2021 The generated index consists of 2MM passages. For each instance, we retrieve top 5 documents and append to our context. For experiments Table 2 we use E5-base provided by FlashRAG Jin et al. (2025b)
indexed on Wikipedia Petroni et al. (2021 which consists of 21 million passages, and retrieve top documents for all datasets.
H PROMPT DETAILS
For completeness, we provide the exact prompt templates used for the ReAct-style reasoner model and the answer extraction model which utilize DsPy (Khattab et al., 2023). Note that after prompt optimization, the demos field is populated but is omitted here for brevity. Additionally, we provide the instruction used for computing MBE using FlashRAG (Jin et al. 2025b) prompt.
Prompt (with FINISH). React is the reasoner model prompt and extract is answer model prompt.
"react";
"signature";
"instructions": "Given the fields 'question', produce the fields answer', "esser'.Annnorr AninYou will be given 'question" and your goal is to finish with 'answer', AnAnTo Tool Name, and Tool Args, and receive resulting Observation. An \nThought can reason about the current situation, and Tool Name can be the following sypes:\ntn(11 AdvancedSearch, whose description is <desc>Searches documents using search search_query. query. Arguments: \"search_quezy"s
Annlnfo do this, you will interleave Thought,
optimized search query for
https://down lods. stanford.edu// o 1eei/i/kk..... abst 2017. 201...t.r.......
21

---

Published as a conference paper at ICLR 2026
dense passage retrieval. IMPORTANT: YOU MUST always PROVIDE \" search_query"" the arguments!
It takes arguments
search_query'; 'str', 'trajectozy_ JSON format. (2) finish, whose description is <desc>signals that the final outputs, i.e. 'answer', are now available and marks the task format.",
Unioon[st,, NoneTypel') in
smplete.</desee It takes arguments () in JSON
"fields":
{ ""preiix: "Question:", "description": "Siquestion/",,
["pretix": "Trajectory:", "description"; "S[trajectory]";
("prefix:: "Next Thought:", "description": "s(next_thought)
preffxx" "Next Tool Name:", "description": "S(next_ttol__amee
"prefix": "Next Tool Args:", "description": "$(next_tool_argg)
b "extract"; {
"signature";
"instructions"; "Given the fields 'question', produce the fields
answer".",
"fields":
( ("prefix": "Question:", "description": "S[question]"),
{ ("prefix": "Trajectory:", "description"; "s(trajectory]") ("prefix": "Reasoning: Let's think step by step in order to", description"; "Sizeascningi")
"prefix"; "Answer:", "description": "S(answer)")
Prompt (without FINISH).
"react": { "signature"; "instructions": "Given the fields 'question', produce the fields answer'.
You will be given 'question'.
You must decide what do next by providing:
Thought: reasoning about the current step
Tool Name: one of the available tcols Tool Args: A valid JSON with the necessary keys.
Repeat this for a few steps to gather information.
You do not need to produce final answer just use the tools iteratively.
Make sure include all required Tool Args as a valid JSCN object.
(1) AdvancedSearch: Searches documents using search_query. Arguments:
("search_query"" an optimized search query dense passage retrieval.
\"trajectory_ict id\": optional string.
IMPORTANT: YOU MUST always PROVIDE \"search__uerr\""
"fields";
( ("prefix": "Question:", "description": "Siquestion)"),
{ ("prefix": "Trafectory:", "description"; "S[trajectoryl")
{ ("prefix": "Next Thought:", "description": "S(next_thought
("prefix": "Next Tool Name:", "description": [y[next_ttol_namee
22

---

Published as a conference paper at ICLR 2026
("prefix": "Next Tool Args:", "description": "$(next_ args)
b
"extract": {
"signature":
"instructions": "Given the fields 'question', produce the fields
answer'.",
"fields":
("pretix": "Question:", "description"; "Siquestion)"),
("prefix:: "Irajectory:", "description": "s(rrajectory)",, "pprffi"; "Reasoning: Let's think step by step in order to", description": "s; "eironnnn" ("prefix:: "Answer:", "description"; "#(answer)")
Answer extraction for E5, For the E5 results, we use a simpler alternative answer extraction mechanism to ensure that the answer extraction is less noisy.
Answer the fellowing question using ONLY the provided docurents.
Question: ((question)
## Retrieved Documents
[|doc_contexs
Think step by step, then give your final answer.
MBE Prompt.
Judge whether the following [respense] to [question] is correct or not based en the precise and unanbiguous [correct_answer] below.
[question]
(question)
[response]
(response)
[correct answer] ccorrect_answer
Your jucgement must be in the format and criteria specified below:
finated__inal__ssee The final exact answer extracted from the response].
[correct_answet Repeat the [correct_ansver] given above.
reasoning: Explain why the extracted final answer is correct incorrect based on Icorrect [correct_ansvezl] in the context of this [question]. You should judge whether the extracted ansvr is semantically equivalent to [correct_ansver], allowing the extracted_final_amm to be string variations of [correct_answer] You should also allow extracted__inal_anew
answer
be more precise or verbose than as long as its additional details are correct. Do not comment on any background to the problem, do not attempt to solve the problem, do not argue for any answer different than correct_ante perrect_answer), focus only whether the equivalent.
correct_answent
semantically
correct: Answer "yes' if extracted_final final_lnnaaaeeeeellae answer matches the [ correct_answer] given above. Answer 'no' otherwise, i.s. if there if there is any intenssstenyy, ambiguity, ron-equivalencyy or if the
23

---

Published as a conference paper at ICLR 2026
extracted answer is incorrect. the response lists multiple items one them matches the correct answer, judge correct UNLESS the question specifically asks for a single item and the additional items are wrong.
confidence: The extracted confidence score between 01\81 and 100 1 from [response]. Put 100 it there is no confidence score available.
24


Reviews:
[{"review_id": "Phcm8g1bAE", "reviewer": "Reviewer_GAQh", "summary": "The paper proposes FrugalRAG, a two‑stage finetuning framework that aims to optimize both accuracy and test‑time retrieval cost for multi‑hop QA in RAG systems. Stage 1 (“Explore”): supervised finetuning of a reasoner to maximize evidence coverage by generating diverse ReAct‑style thought/action/search tuples; dataset rollouts are constructed by greedily selecting the candidate query at each hop that maximizes recall against ground‑truth evidence. FINISH is included in only ~10% of traces to keep it in distribution while emphasizing exploration.  Stage 2 (RL): GRPO is used to learn when to stop searching. The reward penalizes stopping before or after an “optimal” hop count h^\\* defined by when further retrieval no longer improves a metric c (thresholded by \\tau); an additional format reward encourages valid tool calls. The optimization trains the reasoner only; the final answer generator is fixed.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Simple yet effective idea: learn when to stop searching rather than always searching more.  \n\n2. Strong empirical results with small data (1k); especially on HotPotQA under standardized E5/FlashRAG settings\n\n3. Adaptive compute validated quantitatively and qualitatively (fewer searches on easy, more on hard).  \n\n4. Generalization to BrowseCompPlus with more hops than seen in training.  \n\n5. Can pair with different retrievers and answerers; demonstration with CoRAG", "weaknesses": "I don't identify major weakness for this work, but have a few questions:\n\n1. Reward design depends on oracle‑like signals (evidence recall) and somewhat opaque hyperparameters\n\n2. No baseline learned/heuristic stoppers, so it’s unclear how much RL beats simpler termination rules.  \n\n3. Efficiency claims would benefit from runtime/FLOPs and ablation on budget B and sample count v.  \n\n4. Practicality depends on availability of ground‑truth evidence for training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aDbAh539VJ", "reviewer": "Reviewer_2odc", "summary": "The paper proposes FrugalRAG, a two-stage finetuning method that makes language models better at multi-hop question answering with retrieval-augmented generation while staying efficient. First, it uses supervised finetuning on about 1,000 examples to teach the model how to write good search queries and gather relevant evidence across multiple hops. Then it uses reinforcement learning not to make the model search more, but to teach it when to stop searching once it has enough information. This lets the model adapt its compute: easy questions get only a couple searches, hard questions get more. FrugalRAG matches or beats prior systems on benchmarks like HotPotQA in both answer accuracy and evidence recall, while using fewer retrieval calls and far less training data than existing RL approaches.", "questions": "Q1. Could you explain the technical concerns in W1?  \nQ2. Do you have other cost measurements (e.g., wall-clock time, API/tool-call cost)?  \nQ3. Do you have more RL ablation studies?  \nQ4. How stable is RL training across random seeds and hyperparameters?  \nQ5. Will you release the code (especially for the RL loop and reward)?  \nQ6. Do you have other evaluation results such as F1, precision@k, MRR, or MAP?  \nQ7. How would you make the stopping policy safer in high-stakes settings where stopping too early and hallucinating is unacceptable?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "S1. Clear problem focus. The paper presents a well-defined motivation by targeting the inefficiency problem in multi-hop RAG systems, where existing RL/RAG agents often over-retrieve and waste searches. FrugalRAG directly addresses this issue by introducing an adaptive stopping mechanism that halts retrieval once sufficient evidence is gathered, ensuring both cost efficiency and effectiveness.\n\nS2. Logical two-stage design. The technical structure is intuitive and easy to follow.\nStage 1 uses supervised fine-tuning to learn effective retrieval, and Stage 2 applies RL to decide when to stop searching—cleanly separating exploration and stopping behaviors.\n\nS3. Strong efficiency and quality gains. The method achieves notable efficiency improvements while maintaining or improving accuracy.", "weaknesses": "W1. Concerns about technical design\n\n(1) Line 251 defines “$\\Delta$ is the normalized difference $(h_\\text{term} - h) / B$”, but only $h_\\text{term}$, $h*$, and $B$ are defined before. Should it be “$\\Delta$ = $(h_\\text{term} - h) / B$”? This matches the idea that the penalty depends on how far you are from the optimal stopping hop.\n\n(2) Lines 243–249 define a piecewise reward $R$ for late stop, perfect stop, and early stop. The “early stop” case is triggered by $c < \\tau$ (low answer quality), and it reuses the same penalty term $log((1-\\Delta)/\\Delta)$. But if you stop early then $h_\\text{term} < h*$, so $\\Delta < 0$, and $log((1-\\Delta)/\\Delta)$ is undefined for negative $\\Delta$ (the fraction becomes negative). As written, the early-stop reward cannot actually be evaluated, so the reward function is internally inconsistent.\n\n(3) In Algorithm 1 (Line 766), the notation alternates between “budget $B$, max hops $m$”. Both $B$ and $m$ appear to mean “max hops”, which is confusing and should be unified.\n\n(4) Around Line 237, the score $c$` and the threshold $\\tau$ are both used in the reward but never clearly defined. \n\nW2. Limited efficiency analysis\n\nThe paper reports “average searches per question” as its main efficiency metric, but does not provide end-to-end latency or monetary cost in realistic settings (e.g., retrieval time over a large index, API/tool-call pricing). For real RAG systems, wall-clock delay and dollars per query matter, and that systems-level cost analysis is missing.\n\nW3. Missing ablations on RL design\n\nWe see that “FrugalRAG-Explore (SFT)” vs “FrugalRAG (RL)” works, but we do not get deeper ablations, such as: (i) alternative stopping rewards, (ii) RL that also tunes query generation instead of only stopping, or (iii) RL from scratch without Stage 1 supervision (iv) training with backbone pipelines other than Qwen2.5-7B-Instruct. \n\nW4. Code release\n\nThere is no released code and no clear commitment to release it. \n\nW5. Evaluation metric concerns\n\nAnswer quality is judged mainly with a model-based evaluator (“MBE”) rather than exact match or human review. That means reported gains could reflect the judging model’s stylistic preferences instead of true factual correctness. For retrieval, the paper mostly reports recall and does not report precision, F1, MAP, or MRR.\n\nW6. Grammar issues\n\nThere are several grammar issues. Examples:\n\n(1) Line 230: “our only goal is to learn when to sufficient evidence has been gathered.”\n\n(2) Lines 52–53: a long sentence ending with “(with potentially private documents), Typical solutions…” incorrectly capitalizes “Typical” after a comma and fuses two sentences.\n\n(3) Line 54: “train on 90,000-1,00,000 examples...” mixes numbering styles; it should be “90,000–100,000.”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FAgZftDQv7", "reviewer": "Reviewer_r6Km", "summary": "The paper proposes FrugalRAG, a two-stage finetuning framework for multi-hop RAG: Stage-1 supervised “explore” training to maximize evidence coverage, followed by Stage-2 RL to adaptively stop retrieval, balancing answer quality against search cost. With ~1k training examples, the method reports strong efficiency–accuracy tradeoffs and adaptive compute (e.g., fewer searches on easier questions) across HotPotQA, 2Wiki, MuSiQue, and some generalization to BrowseCompPlus.", "questions": "refer to weakness", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Separating exploration (coverage) from stopping (efficiency) is a clean, actionable design; the reward explicitly ties termination to an “optimal” hop length and format compliance.\n\n2. Tables show fewer searches at comparable or better MBE/recall versus strong prompting and SFT baselines, and the “tradeoff” metric highlights practical benefits.\n\n3. The method increases search steps with question difficulty and shows some cross-dataset transfer; the modularity across retrievers/generators is also demonstrated.", "weaknesses": "1. Motivation around “less data” is under-argued: The paper asserts success with ~1k examples, but the core motivation, why low-label regimes are critical for RAG, feels underspecified. In many RAG settings, labeled QA pairs or weak labels are not rare (e.g., synthetic or distantly-supervised pipelines), and the paper does not clarify whether performance keeps improving with more data (scaling behavior is central to RAG systems). A scaling curve (1k→10k→100k) would make this claim concrete.\n\n2. Limited novelty. Algorithmically, Stage-2 introduces a termination-oriented reward shaping that discourages extra retrieval steps; this reads as a targeted constraint atop existing RL-for-RAG methods rather than a fundamentally new learning principle. The contribution is useful but incremental.\n\n3. Missing ablation on the retrieval-steps component of the reward. Since the central claim is “RL learns to stop,” an ablation that removes the step-penalty/termination reward is necessary to show the gain is not just from better exploration traces or general RL regularization. (Table/figures do not isolate this factor.)\n\n4. Questionable cross-method comparisons due to retriever mismatch. Several headline comparisons juxtapose methods that use different retrievers (e.g., E5-base-v2 vs. ColBERTv2) or different indices/evaluators. This undermines causal conclusions about the policy’s quality. A strict apples-to-apples study (same retriever, index, judge) is needed for Search-R1/CoRAG comparisons.\n\n5. Related work coverage is incomplete. Important recent efforts are missing or not discussed in sufficient depth, such as [1-3].\n\n[1] Wang X, Wang Z, Gao X, et al. Searching for best practices in retrieval-augmented generation[J]. arXiv preprint arXiv:2407.01219, 2024. \n\n[2] Ammann P J L, Golde J, Akbik A. Question Decomposition for Retrieval-Augmented Generation[J]. arXiv preprint arXiv:2507.00355, 2025. \n\n[3] Xu R, Shi W, Zhuang Y, et al. Collab-rag: Boosting retrieval-augmented generation for complex question answering via white-box and black-box llm collaboration[J]. arXiv preprint arXiv:2504.04915, 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HWcjLDHYnc", "reviewer": "Reviewer_9twQ", "summary": "This paper introduces FrugalRAG, a two-stage finetuning framework that combines supervised exploration and reinforcement learning to optimize retrieval efficiency in multi-hop question answering. Instead of simply scaling test-time retrieval, the approach adaptively determines the number of retrieval steps per query, balancing accuracy with efficiency. Experiments across benchmarks such as HotPotQA, 2Wiki, MuSiQue, and BrowseCompPlus show that FrugalRAG achieves strong efficiency–accuracy tradeoffs while training on only 1,000 examples.", "questions": "Suggestion: The exposition is sometimes dense, with technical details (e.g., rollout generation, reward shaping) overshadowing high-level intuition. A clearer articulation of practical implications and limitations would improve readability.\n\n\nCould you provide more detailed analysis or case studies showing how the system behaves on questions of varying complexity?\n\nSince the method uses only 1,000 labeled examples, how sensitive is performance to the exact number of examples? Have you tested smaller budgets (e.g., 100 or 500 examples) to see the degradation curve? I am wondering what is the performance of the baseline with the same amount of number of examples.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The focus on frugality addresses a critical yet underexplored dimension in RAG systems (efficiency, not just accuracy).\n\n2. The proposed method achieves competitive or superior results with only 1,000 training examples is impressive compared to prior work requiring 100k+.", "weaknesses": "1. The comparison is unfair by natural, as baseline uses weak retriever (not colbert v2). \n\n2. The RL reward design assumes access to the “optimal” stopping point based on ground-truth evidence, which is often unavailable or impractical in real-world applications.\n\n3. While the paper emphasizes data efficiency and search efficiency as central contributions, these concepts have been discussed extensively in prior work (e.g., [1,2,3, 4]). The paper would benefit from a clearer articulation of how its framing or results go beyond existing definitions of efficiency.\n\n4. Limited case studies and inference efficiency studies of the proposed method. \n\n\n[1] Wang, Hongru, Cheng Qian, Wanjun Zhong, Xiusi Chen, Jiahao Qiu, Shijue Huang, Bowen Jin, Mengdi Wang, Kam-Fai Wong, and Heng Ji. \"Acting Less is Reasoning More! Teaching Model to Act Efficiently.\" arXiv preprint arXiv:2504.14870 (2025).\n\n[2] Jiang, Pengcheng, Xueqiang Xu, Jiacheng Lin, Jinfeng Xiao, Zifeng Wang, Jimeng Sun, and Jiawei Han. \"s3: You Don't Need That Much Data to Train a Search Agent via RL.\" arXiv preprint arXiv:2505.14146 (2025).\n\n[3] Li, Zhuofeng, Haoxiang Zhang, Seungju Han, Sheng Liu, Jianwen Xie, Yu Zhang, Yejin Choi, James Zou, and Pan Lu. \"In-the-Flow Agentic System Optimization for Effective Planning and Tool Use.\" arXiv preprint arXiv:2510.05592 (2025).\n\n[4] Song, Huatong, Jinhao Jiang, Wenqing Tian, Zhipeng Chen, Yuhuan Wu, Jiahao Zhao, Yingqian Min, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. \"R1-Searcher++: Incentivizing the Dynamic Knowledge Acquisition of LLMs via Reinforcement Learning.\" arXiv preprint arXiv:2505.17005 (2025).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)