Chain-of-Retrieval Augmented Generation
Authors: Liang Wang
Haonan Chen
Nan Yang
Xiaolong Huang
Zhicheng Dou
Furu Wei
Venue: NeurIPS 2025 poster ()
Keywords: retrieval-augmented generation
scaling law
multi-hop reasoning
Primary area: applications
TL;DR: This paper introduces an approach for training o1-like RAG models that retrieve and reason over relevant information step by step before generating the final answer.

Abstract: This paper introduces an approach for training o1-like RAG models that retrieve and reason over relevant information step by step before generating the final answer. Conventional RAG methods usually perform a single retrieval step before the generation process, which limits their effectiveness in addressing complex queries due to imperfect retrieval results. In contrast, our proposed method, CoRAG (Chain-of-Retrieval Augmented Generation), allows the model to dynamically reformulate the query based on the evolving state. To train CoRAG effectively, we utilize rejection sampling to automatically generate intermediate retrieval chains, thereby augmenting existing RAG datasets that only provide the correct final answer. At test time, we propose various decoding strategies to scale the model's test-time compute by controlling the length and number of sampled retrieval chains. Experimental results across multiple benchmarks validate the efficacy of CoRAG, particularly in multi-hop question answering tasks, where we observe more than $10$ points improvement in EM score compared to strong baselines. On the KILT benchmark, CoRAG establishes a new state-of-the-art performance across a diverse range of knowledge-intensive tasks. Furthermore, we offer comprehensive analyses to understand the scaling behavior of CoRAG, laying the groundwork for future research aimed at developing factual and grounded foundation models.

Chain-of-Retrieval Augmented Generation
Liang Wang** Haonan Chen
Zhicheng Dou' Furu Wei Microsoff Research 'Remmin University of China
Nan Yang Xiaolong Huang
httss//aka.mssGGeerraAAA
Abstract
This paper introduces approach for training ol-like RAG models that retrieve and reason over relevant information step by step before generating the final answer. Conventional RAG methods usually perform single retrieval step before the generation process, which limits their effectiveness in addressing complex queries due to imperfect retrieval results. In contrast, proposed method, CoRAG Chain-of--Retievee Augmented Generation). allows the model to dynamically reformulate the query based on the evolving state. To train CoRAG effectively, we utilize rejection sampling to automatically generate intermediate retrieval chains, thereby augmenting existing RAG datasets that only provide the correct final answer. At test time, we propose various decoding strategies to scale the model's s test-time compute by controlling the length and number of sampled retrieval chains. Experimental results across multiple benchmarks validate the efficacy of CoRAG, particularly in multi-hop question answering tasks, where we observe more than 10 points improvement in EM score compared to strong baselines. On the KILT benchmark, CoRAG establishes new state-of of-the-art performance across diverse range of knowledge nhowledg-inntenivee tasks. Furthermore, we offer comprehensive analyses to understand the scaling behavior of CoR AG, laying the groundwork for future research aimed at developing factual and grounded foundation models.
CoRAG MusiQue
Where clid the star of Dark Hazard study?
What was the name of the star of Dark Hazard? Edwerd Robinaon
Q
decomposition
Where did Edwerd Robinson go to collage? No relevant information found. retrieval failure
Q
college
Robinson attend?
Q
City College of New York. reforuuaation
Our
Previous SofTA
frontier
City College New York
Avg. Tokenss
(a)
(b)
Figure (a) Test-time scaling behavior of CoRAG. Increased token budget leads to consistent performance improvements. (b) An example of CoRAG on the MuSiQue dataset. It learns to decompose the complex query and conduct query reformulation when encountering retrieval failure.
"Corresoonencce to wangliang @ microsoft.com
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
eetrieval-augmentte generation (RAG) [20] is one of the core techniques in enterprise applications, necessitating the integration of large foundation models with proprietary data sources to produce responses that are both grounded and factual. Conventionally, foundation models are trained on large-scale datasets comprising trillions of tokens and remain frozen post-deployment. Nonetheless, these models frequently struggle to memorize long-tai tail factual knowledge or may hallucinate false claims, resulting in unreliable responses in real-world scenarios. RAG mitigates this challenge by augmenting the generation process with retrieved information, thereby improving the trustworthiness of model-generated content and facilitating the incorporation of up-to-date information.
Contemporary RAG systems typically employ a sequential pipeline of retrieval and generation, wherein the retrieved information serves as additional input to the generative model. The effectiveness of RAG systems predominantly relies on the quality of the retrieved information. Retrieval models are engineered for efficiency to ensure scalability to large corpora. For instance, dense retrievers [18, 35] commonly utilize a bi-encoder architecture to compress documents and queries into fixed-size vector representations. This architectural choice permits the use of fast approximate nearest neighbor search algorithms but simultancously constrains the expressive capacity of retrieval models to handle complex queries. Furthermore, in multi-hop reasoning tasks, it is often unclear what information should be retrieved initially; decisions must be made based on the progressively evolving state of the reasoning process.
To break the bottleneck of retrieval quality, we propose a framework that dynamically retrieves relevant information and plans subsequent retrieval steps based on the current state. By adjusting the number retrieval steps at test time, our model can explore various aspects of the query and experiment with different query rewriting strategies when the retriever does not yield useful information. This paradigm mirrors the human problem solving process, where we iteratively seek information to address complex questions. An example is illustrated in Figure 1.
Rather than solely relying on the model's in-context learning capability [42] or distillation from proprietary models [1], we advocate for explicitly training language models to retrieve step by step. To this end, we utilize rejection sampling [43, 5] to augment existing RAG datasets with intermediate retrieval chains. Open-source language models are then fine-tuned on these augmented datasets using standard next-token prediction objectives. To examine the scaling behavior of our model, we propose various test-time decoding strategies, including greedy decoding, best-of-N sampling, and tree search. Diverse decoding strategies and hyperparameter configurations can employed to control test-time token consumption and the frequency of retriever calls.
Our empirical evaluation demonstrates that CoRAG substantially surpasses strong baselines in QA tasks that require multi-hop reasoning, where retrievers frequently struggle to recall all necessary information single retrieval step. Across diverse decoding strategies, the Pareto frontier approxi- mately adheres to log-linear relationship between total token consumption and model performance, although the coefficients differ across datasets.
On the KILT benchmark [27], which encompasses a more diverse array of tasks, new state-of-the-art scores are achieves on the hidden test sel for nearly all tasks. Additionally, we uncover that CoRAG exhibits varied scaling behaviors across different task types. For datasets such as NQ [19], where state-of- of-te-of-thh-art retrievers already achieve high recall, the benefits of test-time scaling are often marginal. This suggests the potential for dynamically allocating test-time compute based on the complexity of the query and the quality of the retriever. Upon further analysis, we find that CoRAG can effectively decompose complex queries and perform flexible query reformulation to improve the quality of the generated responses. It also shows robustness against retrievers of varying quality. We posit that CoRAG represents promising avenue for future research in the RAG domain, with the potential to mitigate hallucination in model-generated content. Our code, data and trained models are available at https: //t://gtttt /gittub.. cow/micmm/ca.cceccccc.acceeo httpf//cooooooo//oooo/ tttpee//eeeememmeeee htt///aann/oooo/ea /orag.
2 Related Work
Retrieval-Augmented Generation (RAG) integrates information retrieval techniques with genera- tive models to enhance the quality and factual accuracy of generated content [20, 21J. By equipping
LLMs with the ability to browse the web [26], RAG systems can access real-time data, thereby

---

providing responses that are both up-to-date and grounded. The relevance and quality of the retrieved information are pivotal for the efficacy of RAG systems. substantial body of recent research has concentrated on developing better general-purpose text embeddings [18, 35]. Nevertheless, text embeddings frequently face limitations in addressing complex queries due to their reliance on fixed-size vector representations for efficiency purposes.
To mitigate this constraint, contemporary research has extended the conventional paradigm of a single retrieval step followed by generation, advancing to multi-step iterative retrieval and generation [6]. FLARE [13] prompts an LLM to actively determine when and what to retrieve during the generation process.
[30] proposes to interleave
generation with generation-auggeenee tasks. Similarly, IRCoT [33] employs a chain-of- tf-tuugghtt methodology, which recursively refines the reasoning thought for subsequent retrieval steps. Self-RAG [1] empowers L.MS to adaptively retrieve, generate, and critique through self-reflection, thus improving factual accuracy and citation precision in open-domain QA and long-form generation tasks. Auto-RAG [41] utilizes heuristic rules and exact answer matching to construct intermediate retrieval steps, yet its performance remains significantly below that state-of-the-art models. AQA [3] learns to reformulate questions using reinforcement learning but only focuses on single-hop QA tasks. In this study, rather than exclusively on few-shot prompting or distillation from proprietary models, we propose a novel approach to explicitly train LLMs to iteratively retrieve and reason over relevant information.
retrieval, demonstrating enhancements in multi-hop QA
Scaling Test-time Compute Instead of prompting LLMs to directly generate the final answer, Chain- of-Thought (CoT) [36] demonstrates that letting the model to think step by step can drastically improve the performance on mathematical reasoning tasks. Tree-of-Thought (ToT) [40] extends the idea of CoT by adopting a tree structure, allowing the model to explore the search space more comprehensively. To further enhance the reasoning capabilities of LLMs, STaR [43] proposes to leverage bootstrapping techniques to generate intermediate states for training. OpenAl ol [12] conducts large-scale reinforcement learning and exhibits promising test-time scaling behaviors on advanced reasoning datasets, but the technical details are not publicly available. drawback of these methods is the increased token consumption, which consequently increases the response latency.
In the realm of RAG, test-time compute can be increased by retrieving more documents or performing additional retrieval steps. LongRAG [14] posits that RAG performance can be enhanced by integrating long-context L.MSs with more retrieved documents. In contrast, IterDRAG [42] empirically examines the test-time scaling law through few-shot prompting and iterative retrieval for up to 5M tokens. Search-ol [22] combines the open-source QwQ model [37] with active search from Bing, achieving tasks. Concurrent works such as Search-RI [15] train LLMs to use retrieval as tool via reinforcement learning. Our work extends the study of test-time scaling in RAG to targeted fine-tuning paradigm under diverse decoding strategies.
competitive results on knowledge-intene
3 Methodology
The CoRAG framework is illustrated in Figure 2. The "Current State" denotes the input context and instructions provided to the LLM, while the "Next Action" refers to the I.L.M output responding to the given instruction. this section, we describe the key components of CoRAG, including retrieval chain generation through rejection sampling, model training with augmented datasets, and strategies for scaling test-time compute.
3.1 Retrieval Chain Generation
Most RAG datasets only come with a query Q and the corresponding final answer A, without providing intermediate retrieval steps. We propose an automated method for generating retrieval chains through rejection sampling. Each sampled chain consists of a sequence of sub-queries Q = {Q1, Q 2,, } and the corresponding sub-answers ALL where L is a predetermined maximum chain length. The sub-query Q: = LLM(Q Aci, Q) is generated by sampling LLM based on the query Q and the preceding sub-queries and sub-answers. To generate the sub-answer A,,
first retrieve the top-k most relevant documents D i:k using text retriever with Q: as the search
3

---

LLM
retriever
answer
chosen state
discarded state
Query
Trarninn
Current State
LLM
Next Action
query
sub-quary
query
query
query
sub-answer
aubb-answer
quory n
sub-query
sub-answer 'n'
rollouts
rolloute
output
output
output
Greedy
Best-of-N
Tree Search
P(Ans Chain 1)
PIAns | Chain 21
P/Ans Chain 3)
Rejection Sampl ing
Inference
Figure 2: Overview of CoRAG. Rejection sampling is utilized to augment QA-only datasets with retrieval chains. Each chain starts with the original query, followed by a sequence sub-queries and sub-answers. An open-source LLM is then fine-tuned to predict the next action based on the current state. During inference, multiple decoding strategies are available to control the test-time compute.
query, and subsequently prompt an LLM to yield the answer A; = LLM(Q; D This procedure iterated until the chain reaches the maximum length T. or A; matches the correct answer A.
To assess the quality of retrieval chain, we calculate the log-likelihood of the correct answer log P(AIQ, Qu., AL.L) conditioned the chain information. The retrieval chain with the highest log-likelihood score is selected to augment the original QA-only dataset.
3.2
Training
Each training instance in the augmented dataset is represented as tuple (Q, QI.L. Ar:)), accom- panied by the corresponding top-k retrieved documents for the query Q and each sub-query. We fine-tune an LLM on the augmented dataset using the standard next-token prediction objective within unified multi-task learning framework.
The model is simultaneously trained on three tasks: next sub-query prediction, sub-answer prediction, and final answer prediction. We employ the same prompt templates utilized in the retrieval chain generation process, with the exception that we also incorporate the top retrieved documents Diak for the original query Q input for the final answer prediction task.
Luub_query = log P(Q:QQ, Ac), [1,
Liub answer log P(A,),, (1 L]
Liual answer = log P(AIQ, Q1. ALL, D1.))
The cross-entropy loss is computed only for the target output tokens. As reuse the prompt templates for both data generation and model training, a fine-tuned model can be utilized for the next round of rejection sampling in iterative manner.
3.3 Test-time Scaling
Given a trained CoRAG model, we propose several decoding strategies to control the trade-off between model performance and test-time compute. The test-time compute is measured by the

---

total number of token consumptions, excluding the retrieval costs. Unlike previous approaches that consider only prompt tokens [42] or generated tokens [12], we account for both. To simplify further discussion, the prompt tokens are treated equally as the generated tokens, despite prompt tokens typically being less expensive due to prefix caching and computation parallelism of the prefilling
stage.
Greedy Decoding This strategy utilizes greedy decoding generate sub-queries and their corre- sponding sub-answers sequentially. The final answer is generated using the same prompt template as employed during the training phase.
Best-of-N Sampling This method involves sampling N retrieval chains with a temperature 0.7, subsequently selecting the best chain to generate the final answer. As the ground truth answer is not available at test time, we instead calculate the conditional log-likelihood of "No relevant information found" penalty score for each chain. The retrieval chain with the lowest penalty score chosen.
Tree Search We implement breadth-firs search (BFS) variant with retrieval chain rollouts. At each step, the current state is expanded by sampling several sub-queries. For each expanded state, we perform multiple rollouts, and then compute the average penalty score of these rollouts. The state with the lowest average penalty score is retained for further expansion.
To control the test-time compute, the maximum length of the retrieval chain can be adjusted across all decoding strategies. For best-of-N sampling, the number of sampled chains N offers an alternative option to scale the test-time compute. tree search, the number of rollouts and expansion size are two additional hyperparameters.
4 Experiments
4.1 Setup
Data and Evaluation We evaluate CoRAG utilizing two sets of benchmarks: (1) a collection of multi-hop QA datasets, including 2WikiMultihoPQA [8], HotpotQA [39], Bamboogle [28], and MuSiQue [32]; (2) the KILT benchmark [27], which encompasses a broad spectrum of knowledge- intensive tasks. The multi-hop QA datasets serve to evaluate the model's ' capacity to perform multi-hop reasoning, whereas the KILT benchmark assesses the framework's ability to generalize across more diverse tasks. For each training dataset, we prompt the open- open-source Llama-3. 1. /-8B- Instruct model to perform rejection sampling, unless specified otherwise. We utilize E5-large [34] as the text retriever for intermediate retrieval steps. The retrieval corpus is the English Wikipedia provided by KILT, comprising approximately 36 million passages [25]. The selected retrieval chains are employed to augment the original QA-only datasets for subsequent model training.
Regarding evaluation metrics, we report the exact match (EM) and Fl scores [29] for the multi-hop QA datasets. For the KILT benchmark, we submit the model's predictions to the official evaluation server and report the downstream metrics on the hidden test set. To adhere to the leaderboard submission policy, we report public validation set results when conducting ablation studies on the KILT benchmark.
Note that while HotpotQA and MuSiQue maintain public leaderboards, these adopt either simplified reading comprehension setting abstract-only retrieval configuration. Conse- quently, the leaderboard results are not directly comparable to our open-domain QA evaluation setting.
Model Training We conduct full-parameter fine-tuning on the augmented datasets, initializing from the Llama-3.1 --8B-Instruct checkpoint. Two separate models are trained: one for the multi-hop QA datasets and another for the KILT benchmark. The compiled multi-hop QA dataset comprises 125k training instances, whereas the KILT benchmark includes 660k instances after sub-sampling. The model is fine-tuned for 1 epoch with a maximum sequence length of 3k tokens. For the KILT benchmark, we fine-tune E5-Mistral retriever [35] and Ran re-ranker [24] the respective training set to boost the ranking quality.
Further implementation details are provided in Appendix A.
5

---

Table 1: Results on multi-hop QA datasets. We report the performance of CoRAG-8B using various decoding strategies and retrieval chain lengths L. The "Fow-shot w/o Retrieval" configuration utilizes only QA pairs without retrieval augmentation. Both DRAG and IterDRAG are based on Gemini 1.5 Flash [31], while Search-ol-33 based on QwQ [37] and the Bing Search API.
2WikiQA
HotpotQA
Bamboogle
MuSiQue
EM
EM
EM
EM
Feo-shot w/o Retrieval
3-shot Llama-3.1-8B-Inst
27.6
32.1
20.8
28.8
17.6
21.3
3.4
9.7
3-shot GPT-4o
39.5
47.3
38.2
51,2
49.6
61.5
15.8
27.2
w/ Retrieval
3-shot Llama-3.1 1aama-3.118B-mme
30.7
39.9
34.1
46.6
28.0
37.3
7.7
15.4
3-shot GPT-4o
49.0
56.2
45.8
59.4
53.6
63.8
15.7
25.8
Self-RAG-7B
12.2
24.1
16.6
29.4
5.6
16.8
4.6
13.2
ITER-RETGEN
35.5
47.4
45.1
60.4
40.0
50.7
26.1
42.0
DRAG (32k)
45.9
53.7
46.9
60.3
48.8
59.2
15.4
26.0
IterDRAG (32k)
44.3
54.6
38.3
49.8
46.4
56.2
12.5
23.1
Search-ol-23 --BBB
58.0
71.4
45.2
57.3
56.0
67.8
16.6
28.2
Fine-tuned I. Llama-8B w/ turge
55.1
60.7
50.3
63.5
40.8
53.7
17.4
28.1
CoRAG-8B (Ours)
b I==1, greedy
56.5
62.3
50.1
63.2
37.6
51.4
18.6
29.3
D L=6, greedy
70.6
75.5
54.4
67.5
48.0
63.5
27.7
38.5
b L=6, best-of-4
71.7
76.5
55.3
68.5
51.2
63.1
28.1
39.7
P L=6, tree search
71.7
76.4
55.8
69.0
48.8
64.4
29.0
40.3
b L=10, best-of-8
72.5
77.3
56.3
69.8
54.4
68.3
30.9
42.4
4.2 Main Results
Multi-hop QA In Table present comparative analysis of CoRAG-8B against several models, including few-shot 1ama-3.1-8B-Instrul [5], GPT-4o [10], Self-RAG-7B [1], ITER-RETGEN [30], DRAG, IterDRAG [42], and Search-ol-32B [22]. For fair comparison,
also include a fine- tuned Llama-8B baseline utilizing the ES-large retriever, which is fine-tuned on the same datasets as CoRAG-SB but without retrieval chain augmentation. CoRAG-8B substantially surpasses all baselines, with the exception of the Bamboogle dataset, despite being based on a weaker LLM compared to Search-ol-322 -3-3 and IteDDRAG. Conversely, we recognize that fine-tumie tinettuiig on multi-hop QA datasets creates an advantage for CoRAG-SB, compared to the few-shot setting for DRAG and IterDRAG.
The Bamboogle dataset comprises only 125 instances, resulting in considerable variance in perfor- mance across different runs. Certain questions within Bamboogle necessitate access to knowledge more recent than the Wikipedia dump used for retrieval. Systems like Search-ol-32B. which rely on commercial search engines, possess an advantage in this regard.
KILT Benchmark We present several strong systems on the KILT benchmark in Table 2, including KILT-RAG [27]. SEAL [2], Atlas 1BB [11], RA-DIT 65B [23], and FiD with RS [9]. For submission to the KILT Icaderboard, we choose the best decoding configuration for each task based on the public validation set. The results of different decoding strategies detailed in Appendix Table 7. Our CoRAG-8B model achieves a new state-of-the-ar performance across all tasks, with the exception of FEVER, where it marginally trails behind larger model with 11B parameters.
4.3 Scaling Test-Time Compute
In alignment with OpenAl ol [12], our model allows for scaling test-time compute to potentially achieve better performance without updating model weights. There are multiple ways to control the test-time compute. In Figure 3, concentrate two factors: the retrieval chain length I. and the number of sampled chains N for best-of-N sampling. Greedy decoding is a special instance of best-of-N sampling with N and the temperature set to 0.

---

Table 2: The downstream results on the hidden test set of the KILT benchmark. All scores are sourced directly from the official leaderboard, with the exception that "RA-DIT 65B" is from the original paper [23]. "Previous Best" refers to the highest score for each task on the public KILT leaderboard as of January 10, 2025.
Entity Linking
Slot Filling
Open QA
Fact
System
AIDA
WnWi
WnCw
T-REx
zsRE
NQ
HoPo
TQA
FEVER
KILT-RAG
72.6
48.1
47.6
59.2
44.7
44.4
27.0
71.3
86.3
SEAL
83.6
74.6
53.7
40.5
70.9
89.5
Atlas-11B
90.6
85.1
80.8
61.3
50.6
84.0
93.5
RA-DIT 65B
80.5
72.8
78.1
43.5
36.6
72.8
86.9
FiD with RS
85.2
83.7
61.2
39.1
84.6
92.2
Previous Best"
90.6
87.4
71.2
87.7
85.3
62.3
50.6
84.6
93.5
CoRAG-SB (Ours)
93.9
88.2
76.7
88.0
87.2
63.1
60.6
88.3
93.1
2WiciMulthopOOA
HotpotDA
Bamboogle
MusiQque
72
55-
55
64
22
greedy
greedy bct
greedy
20
greedy
58
bost
bext-26-
56
20
10
10
72.5
73.0
07.5
0..0
53
825
bett-of4
battif-
bestofld
bolold
the searc?
50
free search
tree search pantoo honder
The SearCT 2arto Render
partlo
partlo frontkr
15k
120c
Tokent
Figure 3: Scaling test-time compute on multi-hop QA datasets. The Pareto frontier is in the form of a x log(x + b) + fitted on the Pareto optimal points. A point is considered Pareto optimal no other point achieves higher EM score with less token consumption. The metric "# Avg. Tokens" represents the average number of tokens consumed per test instance, summing up both the prompt and generated tokens.
We observe that increasing the retrieval chain length L. results in substantial performance improve- ments when is small, but the gains diminish as L. increases. This observation aligns with intuition that longer chains can encapsulate more reasoning steps and allows for trial-and-error explo- ration various query rewriting strategics. Several examples are provided in Appendix Table 11. Conversely, increasing N for best-of-N sampling yields mixed effects depending the dataset. For the most challenging dataset, MuSiQue, terms of EM score, larger N enhances performance, smaller N suffices. We defer the
whereas for the less challenging dataset, 2WikiMultihopQA further exploration of tree search to future work, as it is considerably more computationally expensive than greedy decoding and best-of-N sampling.
The Pareto frontier between EM score and token consumption approximately follows log-linear trajectory for up to 128k tokens, although the scaling behavior varies across different datasets. This observation assists practitioners in making informed decisions regarding the allocation of test-time compute based the quality requirements. It is important to note that make several simplifications

---

in this scaling study, such as treating the prompt tokens equivalently to the generated tokens and ignoring the retrieval costs, more rggorous analysis could take these factors into account.
Analysis
Table 3: Ablation study results. "Iterative training" employs trained CoRAG model for another round of rejection sampling. "Distill from GPT-4o" leverages the GPT-4o model to generate retrieval chains. "Weak-to-strong Generalization" utilizes weaker LLMs for retrieval chain generation while using stronger LLMs (Llama-3, 1-8t-Inst) for training. "Different Retrievers" replaces the text retriever at test time.
2WikiQA
HotpotQA
Bamboogle
MuSiQue
EM
F1
EM
F1
EM
F1
EM
F1
CoRAG-SB (L=6, greedy)
70.6
75.5
54.4
67.5
48.0
63.5
27.7
38.5
> iterative training distill from GPT-4o
72.2
76.9
53.4
66.5
45.6
60.9
26.6
37.6
75.1
79.5
56.6
70.2
51.2
67.0
28.2
38.5
Weak-to-strong Generalization w/ Llama-3.2- B-Ist..
59.3
64.2
50.3
63.6
40.8
51.6
22.3
32.7
w/ Llama-3.2-3 3.2-ma-3.---B-Inst.
69.9
74.0
53.9
67.3
45.6
59.8
25.2
36.0
Different Retrievers
E5-base w/o chain-of-retrieval
53.1
58.9
47.9
61.1
38.4
52.7
15.8
26.4
b L=6, best-of-4
70.8
75.4
53.0
66.2
47.2
59.8
26.3
37.6
BM25 w/o chain-of-retrieval
49.1
55.3
46.9
60.3
36.8
48.6
14.3
24.8
> L=6, best-of-4
62.6
67.7
51.6
64.7
37.6
52.5
23.5
33.0
5.1 Iterative Rejection Sampling
Our framework facilitates self-improvement through iterative training, akin to the iterative rejection sampling employed in LLM post-training [5]. By utilizing the same prompt templates for both data generation and model training, trained CoRAG model generate new sets of retrieval chains. However, the results in Table 3 are mixed, showing performance improvements on the 2WikiMultihopQA dataset but slight declines other datasets. This indicates that instruction-tuned LLMs already possess a strong ability to generate high-quality retrieval chains.
5.2 Robustness and Generalization
Different Retrievers We further investigate the influence of various text retrievers at test time. Instead of using the E5-large dense retriever, we substitute it with two weaker alternatives in plug-and-play fashion: E5-base and BM25. Across all datasets, we observe consistent performance gains when investing more test-time compute, although stronger retrievers continue outperform in terms absolute performance. Improvements to text retriever quality represent an orthogonal dimension that further amplify CoRAG's performance gains.
Weak- fook-too-ttong Generalization Due to the need of repeated sampling and autoregressive generationn the retrieval chain generation process costs more GPU hours than the model training. To mitigate this cost, one strategy is to employ weaker LLMs for retrieval chain generation and subsequently fine-tune stronger LLMs the augmented datasets, similar to the weak-to-strong generalization setting [4].
The results in Table demonstrate that utilizing Llama-31 3B achieves very close performance compared the 8B model, whereas Llama- exhibits a noticeable performance drop. Manual inspection re- veals that the 1B model frequently struggles to follow the given instructions, resulting in sub-optimal retrieval chains. Employing weaker LLMS also lowers the barrier to adopting more computationally expensive tree search strategies during data generation, which show great potential in mathematical reasoning tasks [7]. In contrast, distilling from a stronger model like GPT-4o yields a further perfor- mance boost, indicating that the quality of the retrieval chains is crucial for the final performance.

---

5.3 Does Chain-of-Rettievall Always Help?
NQ
95
65
94
90
Accuracy
93
89
63
92
88
62
91
greedy
greedy
greedy
of
60
90
best-ofe 8f
best of
best of-8
Chain Length
Chain Length
Chain Length
Figure 4: Scaling test-time compute across three datasets from the KILT benchmark. We report scores on the public validation set.
Multi-hop QA datasets are specifically designed to evaluate complex reasoning capabilities and are expected benefit from the ofetinol-rreriieval mechanism. Table 1 presents empirical evidence supporting this assertion. In contrast, for tasks that a single retrieval step is typically sufficient, the advantage tends be marginal, as demonstrated in Figure 4. Datasets such as NQ [19] and TriviaQA [17] are known for their (mostly) single-hop nature. This phenomenon implies that decoding strategies should be adaptive based on the complexity of the query. Additional results on the full KILT benchmark are listed in Appendix Table 7, where similar observations for other task types also hold.
5.4 Learning Stop at Test Time
Instead of always performing L retrieval steps, we explore model variant that learns stop at test time. After each retrieval step, the model prompted to predict whether the information gathered thus far suffices to answer the query. Note that this prompt itself also incurs token consumption and additional cost. The decoding space is constrained to two tokens: "Yes" and "No". If the decoded output is "Yes". no further sub-queries are generated. By adjusting the logit bias of the "Yes" token, stopping behavior.
MuSiQue (Greedy Decoding)
Token Consumption
-6.8k
EN Score
L-0
La0
5.9k
control the early
20-
4.9k
During the training phase, an additional loss term added for the stop prediction task. The target output is "Yes" if the current retrieval chain encompasses the prefix that maximizes the likelihood of the final answer, and "No" oth- erwise. The associated prompt template in Appendix Section D.
3.9k
-2.0
-1.5
-10
-2.5
3.0
0.5
1.0
1.5
20
Early Stop Logit Blas
Figure 5: Learning to stop at test time. Larger logit bias values result in earlier stopping. = correspond to always performing 6 retrieval steps, while L = indicate no intermediate retrieval steps.
In Figure 5, we illustrate how the performance varies along with the token consumption on the MuSiQue dataset. While early stopping can save some amount of token quota, it comes at the cost of performance degradation. The optimal configuration depends on the dataset characteristics and the quality expectations.

---

5.5 Does CoRAG Learn to Retrieve Better?
To evaluate whether CoRAG improves retrieval quality beyond just answer accuracy, we measure retrieval recall across multiple datasets. We report Recall @K metrics for A: {10, 20, 100), compar- ing standard retrieval using tagge against our approach. We follow the evaluation protocol from DPR [18] for caleulating recall based answer matches, not all datasets provide gold supporting paragraphs. For CoRAG, we utilize reciprocal rank fusion to merge multiple retrieval results from the chain into single ranked list, from which recall is calculated.
Table 4: Retrieval recall comparison between standard retrieval and CoRAG across multi-hop QA datasets.
R@ 10
R@20 20
R@
100
HotpotQA w/ EStage CoRAG
59.1
65.2
76.8
72.1
76.7
84.3
2WikiMultiHopQA w/ ESturge
54.9
62.1
74.6
CoRAG
81.4
84.8
88.8
Bamboogle w/ J5re w/ CoRAG
31.2
40.0
57.6
59.2
68.0
75.2
MuSiQue w/ EStarge
29.0
36.5
52.7
w/ CoRAG
47.1
54.6
68.4
The results Table 4 demonstrate that CoRAG consistently improves recall across all datasets and recall thresholds. The improvements particularly pronounced more challenging datasets like MuSiQue and Bamboogle, where single-step retrieval struggles most. This indicates that CoRAG's iterative query reformulation and decomposition strategy effectively addresses the limitations of traditional dense retrieval, enabling the model to gather more relevant information through multiple retrieval steps.
Conclusion
In this work, we introduce CoRAG, a framework that teaches LLMs conduct iterative retrieval and reasoning to answer complex queries. The intermediate retrieval chains are automatically generated via rejection sampling, therehy alleviating the need for manual annotation. test time, we offer multiple decoding strategies manage the trade-off between performance and compute. Our experiments demonstrate that CoRAG-SB achieves state-of-the-an performance on both multi- hop QA datasets and the KILT benchmark, surpassing many baselines built with larger LLMs. A comprehensive analysis conducted to understand scaling behavior and generalization capability, In the future, intend to extend CoRAG to more challenging and economically valuable RAG tasks, advancing towards building factual and trustworthy Al systems.
7 Limitations and Broader Impacts
This study primarily investigates RAG tasks characterized by short and easy-to-verify answers, such multi-hop QA and entity linking. However, real-world applications often necessitate addressing more complex tasks that demand generating long-form outputs. A significant challenge long-form generation lies in the absence of robust evaluation metrics within the current research landscape.
Regarding broader impacts, the proposed framework aims to improve the factuality and groundedness of language model outputs. It is anticipated that this work can facilitate efficient and effective information retrieval for users. Nevertheless, the inherent risk of hallucination persists and warrants careful monitoring in practical deployments.
References
[1] Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learn- ing to retrieve, generate, and critique through self-reflection. In The Twelfth International
10

---

Conference on Learning Representationss ICLR 2024, Vienna, Austria, May 7-11, 2024. Open- Review.net, 2024. URL htttt::/ met//ooruumnddddSSylooo....
[2] Michele Bevilacqua, Giuseppe Ottaviano, Patrick S. H. Lewis, Scott Yih, Sebastian Riedel, and Fabio Petroni. Autoregressive search engines: Generating substrings as document iden- tifiers. Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 December 9. 2022, 2022. URL http: //papers mips. cc/ http/ppper__ffle wwwles/ppper/2000 Conference html.
hash/
[3] Christian Buck, Jannis Bulian, Massimiliano Ciaramita, Wojciech Gajewski, Andrea Gesmundo, Neil Houlsby, and Wei Wang. Ask the right questions: Active question reformulation with 07830, 2017.
reinforcement learning. arXiv preprint
[4] Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschen- brenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, Ilya Sutskever, and Jeffrey Wu. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. In Forty- first International Conference on Machine Learning, ICML. 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https: /ttp:s://ppaaee openrevieve ew. wwwprrvvieeeeeeeffruueeeee http///rrnndggglgnmmmm.....
[5] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. ArXiv preprint, abs/2407.21783, 2024. URL
www.aioww.......a..........ooooe wtrggabb/24000
21783.
[6] Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai. Jiawei Sun, Qianyu Guo, Meng Wang, and Haofen Wang. Retrieval-augmentt generation for large language models: A survey. ArXiv preprint, abs/2312.10997, 2023. URL https://arxta org/abs 2312. 10997.
[7] Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. rstar-math: Small llms can master math reasoning with self-evolved deep thinking. ArXiv preprint, 0e//22011045111 2025. URL bttps://arxivv org/ abs/2501 04519.
[8] Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps. Donia Scott, Nuria Bel, and Chengqing Zong, editors, Proceedings of the 28th International Conference Computational Linguistics, pages 6609-6625, Barcelona, Spain (Online), 2020. International Committee on Computational Linguistics. doi: 10.18653/w1/2220.oolli maw.:/0ooooiimmmm.8........ https:// ac /stps://aclanttaee ha//a/ccaaaatoogggyy.o org/ cowing-maaaaaa m00..500..
main.580. URL
[9] Sebastian Hofstätter, Jiecao Chen, Karthik Raman, and Hamed Zamani. Multi-task retrieval- augmented text generation with relevance sampling. ArXiv preprint, abs/2207.03030 2022. URL
otg/a/ss htwpaaabbe/220020 03030..
[10] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. ArXiv preprint, abs/2410.21276, 2024. URL https: //t::v////ivvv/// hrrg/bss/444 21276.
[11] Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi- Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave Atlas: Few-shot learning with retrieval augmented language models. Journal of Machine Learning Research, 24(251):
1 1-43, 2023.
[12] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. ArXiv preprint, abs/2412. 16720, 2024. URL. ptattps://arivv
16720.
[13] Zhengbao Jiang, Frank Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Wos Yu, Yiming Yang, Jamie Callan, and Graham Neubig. Active retrieval augmented generation. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7969-7992, Singapore, 2023. Association
11

---

for Computational Linguistics. doi: 10. 10.18653/112023.mmlppp ma..........mmm.-m..9999999 495. URL https://
aclan aclanthology. satthooogg.oorg2223.... 2ttp/2223.mmmmmmmmamm... emnlp-main 495.
[14] Ziyan Jiang, Xueguang Ma, and Wenhu Chen. Longrag: Enhancing retrieval- auggeee-d
generation with long-contexl llms. ArXiv preprint, abs/2406.15319, 2024. URL https:
/ //axxiv.rrg/ hts//bb//// ab////bb/222.... 2406.420620 1440111319.
[15] Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Za- mani, and Jiawei Han. Search-rl: Training Ilms to reason and leverage search engines with reinforcement learning. arXiv preprint XxXx:25533009//6 2025.
[16] Jiajie Jin, Yutao Zhu, Xinyu Yang, Chenghao Zhang, and Zhicheng Dou. Flashrag: A modular toolkit for efficient retrieval-augmentee generation research. ArXiv preprint, abs/2405. 13576, 2024. URL https: / arxivv. org./.ooooooooooooo..oo org/ wtw.///...... 13576.
[17] Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min- Yen Kan, editors, Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601-1611, Vancouver, Canada, 2017. Association for Computational Linguistics. doi: org/P17-1147.
URL https: https////ccaaaaoa/aaooee
[18] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781, Online, 2020. Association for Computational Linguistics. doi: 10.18653/v 005553/w11222..mm//// waw.emmmmmmmmmmmmm01-- main.550. URL https: http:///aaaaathhooooooooo
2rgg/2000... emnlp-main.550 550.
[19] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452 466, 2019. doi: 10.1162/tacl a 00276. URL https: ac lwtlaaaanntlllyooooooooooo wwwpeey..rrgggg---// Q11--Q161-11100
[20] Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrievallaugmmnned generation for phowledge-intensing NLP tasks. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrlPS 2020, December 6- 12, 2020, virtual, 2020. URL https: http:::::///.oredeings//.. wtwpeeeediinn..eerripppeee www.ieuuri.... wwturiss.ccpapapee http:/aappe/2///aa/e/ep 1tt.:/4881155Abbraaattea html.
[21] Xiaoxi Li, Jiajie Jin, Yujia Zhou, Yuyao Zhang, Peitian Zhang, Yutao Zhu, and Zhicheng Dou. From matching to generation: A survey on generative information retrieval. ArXiv preprint, abs////////44 14851.
14852, 2024. URL https: //trx//aaiiive//aa
[22] Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang. Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. Search-ol: Agentic search-
search-enhanced enhanced large reasoning models. ArXiv preprint, abs/2501.05364 2025. URL https: /wwp//aa/vvv./////////////...av. wwg/abs/////ooooooooo..ooceseere 2201. 05366.
[23] Xi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi, Maria Lomeli, Richard James, Pedro Rodriguez, Jacob Kahn, Gergely Szilvasy. Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. RA-DIT: retreeval-agmmented dual instruction tuning. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https::////// http::::///ppreeeeee http////ppeeeeeeee//e/////e net htlffformmmid 2d-umidd-20Tuuuugg.
[24] Xueguang Ma. Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. Fine-tuning llama for suati-stage text retrieval. Grace Hui Yang. Hongning Wang. Sam Han, Claudia Haufff Guido Zuccon, and Yi Zhang, editors, Proceedings of the 47th International ACM SIGIR Conference
12

---

on Research and Development in Information Retrieval. SIGIR 2024, Washington DC, USA, July 14-18, 2024, pages 2421-2425. ACM, 2024. doi: 10.1145/322677233679991 URL 44445/63667772.36755555
https://doi.orge:
[25] Jean Maillard, Vladimir Karpukhin, Fabio Petroni, Wen-tau Yih, Barlas Oguz, Veselin Stoyanov, and Gargi Ghosh. Multi-task retrieval for knowledge-intene intensive tasks. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Pupers), pages 1098-1111, Online, 2021. Association for Computational Linguisties. doi: 10.18653/v1/2011 aacl-long.89, URL https: / //ac/ /acllantholgyy. org/ 2201/ acl-long.B0. 89.
[26] Reiichiro Nakano, Jacob Hilton, Suchir Balaji. Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answerrigg with human feedback. ArXiv preprint, abs/21 12.09332, 2021. URL https://arxiv. /rg/ abs 2212.. 09332.
[27] Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. KILT: a benchmark for knowledge intensive language tasks. In Kristina Toutanova, Anna Rumshisky, Luke Zeulemoyer. Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages for Computational Linguistics. doi: 10.18653/v1/22
Online, 2021. Association http://///2...alllleeemm moedd-mmiin.0000
URL https://
aclan aclanthology. org/2021 wte22222anaall...emmma main. 200.
[28] Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5687-5711, Singapore, 2023. Association for Computational Linguistics. doi: 10.18653/ URL https : / //acclaatitnn aclanthology. org/2023. findings cereddnnnseellpp....
v1/2023. 11/2023.inndings 378.
[29] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Jian Su, Kevin Duh, and Xavier Carreras, editors, Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2392,, Austin, Texas, 2016. Association for Computational Linguistics. httttps:/accattthatt
doi: 10.18653/v1/D16- 1264. URL https: : /aa//claantttt /aacaaaaannooooyoooooooco www.://-gg///11/ 1264.
[30] Zhihong Shao, Yeyun Gong. Yelong Shen, Minlic Huang, Nan Duan, and Weizhu Chen. Enhancing retgenedl-augmeeteed large language models with iterative -erriaval-eenerationn synergy. In Houda Bouamor, Juan Pino, and Kalika Bali, editors. Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9248 9274, Singapore, 2023. Association for Computational Linguistics. doi: 10. 10.18653/11/2033. //aclanthology.. .oggy.org org/ 2023.findinge.. emnlp. 620.
http:/222..nnnngeeemmmem. findings-emnlp.622 URL https:
[31] Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. ArXiv preprint, abs/2403.055300, 2024. URL
https://arxiv http:///xxivvor///////// abs/2403. 05533..
[32] Harsh Trivedi, Niranjan Balasubramanian. Tushar Khot, and Ashish Sabharwal. MuSiQue: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539-554, 2022. doi: 10. 10.1 a 00475. URL https:
/ l/aclanthology.oom .rg. wwtp/022..oco/.e.e http:/2222tacc11.333
[33] Harsh Trivedi, Niranjan Balasubramanian. Tushar Khot, and Ashish Sabharwal. Interleaving retrieval with of-inn reasoning for mooleedg-intensive multi-step questions. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 10034, Toronto, Canada, 2023. Association for Computational Linguistics. doi: 10.18653/v 1/202 1o1- URL https: bettts://aclanttaae aclaahhooootoooocccc org/ 22gg/222...2 acl- acl-long.555
13

---

[34] Liang Wang, Nan Yang. Xiaolong Huang, Binxing Jiao, Linjun Yang. Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-superrised contrastive pre-training, ArXiv preprint, abs/2212.03533, 2022. https: / /rrivv...v.. org/ abs /212. 03533.
[35] Liang Wang. Nan Yang. Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Improving text embeddings with large language models. ArXiv preprint, abs/2401.00m 000668, 2024. URL https: / /arxiv..vvvv org/ aa/aabbs224000 .00368.
[36]
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc Le, and Denny Zhou. Chain-of-though prompting elicits reasoning large language models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022. New Orleans, LA, USA, November 28 December 9. 2022, 2022. URL http::/ nips cc http:ppapeeeeeeeaaeeeaapaa wttpe//ppppe//////// hash
Abstraattbbb---ttttttt-----a Conf Confrrecce--oeeeeeeeee html.
[37] An Yang. Baosong Yang. Beichen Zhang. Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. ArXiv preprint,
abs/2412.1511 2024. URL https:://////. //tp///aavvvv./.e///a/eeeeeaaava org/ barrg/abs/// abs/2412.. 15115.
[38] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388 2025.
[39] Zhilin Yang. Peng Qi. Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Ellen Riloff, David Chiang. Julia Hockenmaier, and Jun ichi Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 23800, Brussels, Belgium, 2018. Association for Computational Linguisties. doi: (0.18653/v1/) ID11 8- 1259. URL https: /awpa//olaaaoloooo.oacoo org/Do8- 1259.
[40] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 2//22/222//////
10- - 16, 2023, 2023. URL http: / //ppperspppppp/ppeeee nips cc/ wecpppprr.... llaperrfiles.///////lppppppapeee 271ab9922b8d1 wtt.ed//7aaa4445555/c/a// Aeee-ratstttteeeeeeee Conf .html.
[41] Tian Yu, Shaolei Zhang, and Yang Feng. Auto-rag: Autonomous retrieval-auggmenté generation for large language models. ArXiv preprint, abs/241 2024. URL https://arxix org/ abs/ 2211.241.111 19443.
[42] Zhenrui Yue, Honglei Zhuang, Aijun Bai, Kai Hui, Rolf Jagerman, Hansi Zeng. Zhen Qin, Dong Wang, Xuanhui Wang, and Michael Bendersky. Inference scaling for long-context retrieval augmented generation. ArXiv preprint, abs/2410.04343, 2024. URL https:://///// htttpps///rxii....// org/ abs/2410. 04343.
[43] Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. Star: Bootstrapping reasoning with reasoning. In Sanmi Koyejo, Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022. http: //wppess.ppppppp..eeeesrree nips. cc/ cc/ hatppppper__re files/illeeeeel wwtpss//pp///////eeeeeeeeeeee (apper/2022 /022/ haaa///588a172004166666776666662
ConerracccCooeeeeee erence html.
A Implementation Details
Rejection Sampling For each training instance, we sample up to 16 retrieval chains, with the maximum length randomly selected from the interval [1, 5). The sampling temperature is set to 0.7 for sub-query generation and 0 for sub-answer generation. Chain generation terminated the sub-answer matches the correct answer if the average conditional log- likeelooooo of the correct
14

---

answer exceeds - -0005. For each sub-query. we utilize the E5-large retriever to retrieve the top-5 most relevant documents from the KILT version of the Wikipedia corpus [25]. This corpus comprises 36 million passages.
Table 5: Hyperparameters for training CoRAG.
Multi-hop QA
KILT Benchmark
Initialization
-lama-..18B--nstiiee
Learning rate
x 10 6
10-5
Batch size
256
1024
Epoch
1
1
Warmup steps
100
100
# Training samples
125k
660k
# Retrieved passages
20
20
Max sequence length
3072
3072
Table 6: Statistics of the datasets used for multi-hop QA training.
2WikiMultihopQA
HotpotQA
Bamboogle
MuSiQue
# Training Samples
15, 000
90, 447
19, 938
# Validation Samples
176
7, 7,405
125
2, 417
Multi-Hop QA Training Hyperparameters The training set is the union of the 2WikiMultihopQA, HotpotQA, and MuSiQue datasets, comprising a total of 125k samples as shown in Table 6. The Bamboogle dataset, consisting of only 125 questions, is reserved for evaluation only. Additional hyperparameters are detailed in Table To balance the three loss terms in Section 3.2, we set a sample ratio of 0.2 for both the sub-query and sub-answer generation tasks; this ratio is also applied to the KILT training.
KILT Training Hyperparameters We utilize the official training set of the KILT benchmark, omitting the ELI5 and WoW datasets due to the lack of reliable evaluation metrics. To balance the task distribution, only select 100k samples for large datasets like T-REx and Zero-Shot RE. In accordance with the benchmark's 's guidelines, we also add 100k samples from the BLINK dataset for entity linking.
Rather than using off-the-shelff retrievers, we fine-tune an E5-Mistral retriever following Wang et al., and RankLLaMA re-ranker following Ma et al.. We adhere to the exact training hyperparameters outlined in the original papers, except that the training data is replaced with the KILT training set. For training the RanKLLaMA re-ranker, the backbone is initialized with the Jama-338B-Bas model, as opposed to Llama-2, to enhance performance. Retrieval and re-ranking scores are presented in Table 8.
All training jobs are conducted using 8 A100 GPUs. The multi-hop QA task requires less than hours of training, whereas the KILT training takes approximately 30 hours. When submitting to the KILT leaderboard, we select the optimal decoding strategy for each task based on validation set performance.
Decoding Strategies the context of best-of-N sampling, the temperature set 0.7 for sub-query generation. For sub-answer generation and final answer prediction, the temperature is always set to 0 across all decoding strategies. Regarding tree search, we set the expansion size to 4 and the number of rollouts to 2. Given that tree search incurs significantly higher token consumption compared to other decoding strategies, we limit the rollouts maximum of steps for each expansion. To avoid the model from generating repetitive sub-queries endlessly, any generated sub-query identical to previous ones discarded.
Evaluation For multi-hop QA tasks, we evaluate the performance using the exact match (EM) and F1 scores [18]. For Self-RAG-7B, we reproduce the results utilizing the FlashRAG [16] toolkit with the official checkpoint released by the authors.
15

---

For the KILT benchmark, we employ the official evaluation scripts provided by the organizers. For Open QA tasks, the main evaluation metric the EM score, while other task types are evaluated using accuracy scores. The KILT benchmark also offers a variant of the evaluation protocol that requires the model not only to generate the correct answer but also to provide the correct supporting evidence. However, our method spreads the evidence documents across the retrieval chain, rendering challenging to conform to such an evaluation protocol.
B Additional Results
Table 7: Downstream results on the public validation set of the KILT benchmark.
Entity Linking
Slot Filling
Open QA
Fact
System
AIDA
WnWi
WnCw
T-REx
zsRE
NQ
HoPo
TQA
FEVER
CoRAG-SB (Ours)
D L=1, greedy
90.4
86.0
76.8
87.0
82.1
62.5
56.4
88.4
91.4
L= greedy
92.7
87.4
75.8
86.6
83.8
63.2
59.1
88.6
93.8
L=6, best-of-4
92.5
87.4
75.8
86.3
83.5
62.6
59.6
88.7
93,9
b 6=6, tree search
91.8
86.8
75.5
86.4
83.0
62.4
59.9
88.9
93.9
Table 8: Retrieval results (R-Precision) on the public validation set of the KILT benchmark. For re-ranking, the top-100 candidates from the fine-tuned retriever input.
Entity Linking
Slot Filling
Open QA
Fact
System
AIDA
WnWi
WnCw
T-REx
zsRE
NQ
HoPo
TQA
FEVER
Fine-tuned misistall
92.9
86.7
76.0
80.5
95.3
77.7
66.7
78,9
90.9
w/ re-ranking
93.3
88.0
77.1
83.2
97.6
78.2
78.2
81.5
92.3
Different Decoding Strategies the KILT Benchmark In Table we present the results of various decoding strategies applied to the validation set of the KILT benchmark. Given that most tasks within the KILT benchmark are much easier for strong dense retrievers compared multi-hop QA, the disparity in performance across different decoding strategies is less pronounced. This observation underscores the necessity of developing a system capable of adaptively selecting the optimal decoding strategy to effectively balance the trade-off between performance and test-time compute.
Damboogle
76
32
74
58
46
70
54
26
68
42
Le6,
50
Lef,
Led,
Rejection Sampting
Rejection Sampligg Size
Rejection Sampling
Rejection Sampting
Figure 6: Scaling rejection sampling compute for training data generation. We vary the number of sampled chains from 4 to while maintaining other hyperparameters fixed.
Scaling Compute for Training Data Generation Within our proposed framework, rather than investing more compute test time, we can scale the compute for retrieval chain generation during rejection sampling. By increasing the number of sampled chains, we may identify better chains that contribute to higher-quality training data. However, as illustrated in Figure 6, no definitive trend emerges indicating that increasing the number of sampled chains always leads to better performance. Conversely, the training loss consistently decreases as we scale up rejection sampling, suggesting that the training data becomes less noisy and easier fit. We hypothesize that the majority of sampled
16

---

chains are already of high quality and that LM fine-tuning exhibits considerable robustness to noisy training data.
Bamboogle
MuSiQue
4.2
40
30
34
greedy best of-4
greedy
greedy
10-
gready
best-of-4
best-2f4
best of
bestal-
bast-of-
bast-sf-5
Sast-d-8
Clam Langth
Chain Langth
Chain Langth
Chain Length
20
12
35
best
bestaff
baskef-
pares Ronder
cartts fioe
23
pornto freeder
parnto fronter
beak-eld
96
18K
3IK
542
120
36
30k
641
122:
41
6K
SIK
54k
120c
120k
Aug.
Ang.
Figure 7: Scaling test-time compute on multi-hop QA datasets with Llama-33 3.1--B-Insturu-cc No fine-tuning performed on the model weights.
Scaling Test-Time Compute without Model Fine-Tuning In Figure 7, we present the scaling results on multi-hop QA datasets using the Llama-3.1 IS--Isst model directly without any fine- tuning. The scaling curves are similar those observed in Figure 3, but the absolute performance is significantly lower, indicating that targeted fine-tuning is essential for improving the scaling upper bound.
2WlkMMuiPPoQAA
HotpotQA
Bamboogge
MuSiQue
36.5
30.0
52
29.5
20.0
55.5
300
28.5
28.0
40
$4.5
34.0
bust-3f4
0.2
0.5
8.7
1.0
1.2
1.5
0.
0.5
0.7
1.5
1.5
3.2
0.5
0.7
3.0
1.2
1.5
0.2
0.5
07
3.0
3.2
1.5
Jumperature
kmoerature
Temperatirn
Temporature
Figure 8: Effects of varying the sampling temperature on multi-hop QA datasets.
Effeets of Sampling Temperature best-of-N sampling, the sampling temperature controls diversity and quality trade-off in the generated retrieval chains. A higher temperature results in more diverse chains, albeit with the potential introduction of increased noise. Figure illustrates the lack of consistent conclusion regarding the impact of sampling temperature on performance. For the MuSiQue and HotpotQA datasets, a lower temperature generally yields superior results, whereas for the 2WikiMultihopQA dataset, medium temperature leads to the best performance. As result, stick to a temperature of 0.7 for both rejection sampling and test-time decoding for simplicity.
Extension to Other Model Families To demonstrate that our CoRAG framework is agnostic to model families and not limited to Llama-based architectures, we conduct experiments using Qwen3- 4B and Qwen3-8B [38] models following the same training procedure, As shown in Table 9, CoRAG consistently outperforms the baseline fine-tuned models across all datasets and model sizes, with improvements of over 10 EM points average. This validates that the chain-of-retrieval mechanism
17

---

Table 9: Extension to the Qwen3 model families.
2WikiQA
HotpotQA
Bamboogle
MuSiQue
EM
F1
EM
Fl
EM
F1
EM
Fl
Fine-tuned Qwen3-4B w/ EStare CoRAG-Qwen3-4B (L=6, greedy)
49.3
55.3
45.0
57.9
32.8
43.1
13.4
23.8
69.3
74.1
51.6
64.2
49.6
62.5
24.0
34.5
Fine-tuned Qwen3-8B w/ ESarge
52.1
57.9
47.1
60.0
33.6
47.6
15.3
26.3
CoRAG--Qen3-8B (L=6, greedy)
70.0
74.8
52.8
66.0
49.6
63.7
25.2
35.9
broadly applicable across different model architectures and confirms the generalizability of our approach beyond specific model families.
Case Analysis Table I presents several model predictions on the validation of the HotpotQA dataset. We compare the performance of RAG without chain-of-retrieva against CoRAG. CoRAG effectively decompose the complex multi-hop queries into a sequences of simpler sub-queries and dynamically conducts query reformulation when the retrieved information proves unhelpful. In the fourth example, the model initially hallucinates some incorrect information but subsequently self-corrects by verifying the poet's name and country of origin through additional retrieval steps.
C Statistical Significance
We compute the 95% confidence intervals for our main results in Table and 2 using the bootstrap resampling method. On all datasets except the Bamboogle dataset, observe that the performance of CoRAG-8B is significantly better than the baselines.
D Prompts
Table 10: Task descriptions for each dataset.
Dataset
Task Description
HotpotQA / 2WikiMulti-
answer multi-hop questions
hopQA
NQ
answer natural questions from Google search
AidaYago 2 WnWi WnCw Blink
link the mention surrounded by [START_ and [END_ENT] to the title the correct Wikipedia page
FEVER
verify if claim is supported or refuted
T-REx Shoto-Shot RE
given head entity and relation separated by [SEP], find the correct tail entity, return the title of its Wikipedia page
Trivia QA
answer trivia questions
MuSiQue Bamboogle
answer multi-hop hop questions
All prompts are listed in this section. Texts highlighted in blue denote placeholders to filled in.
The "task description" field is datasct-specific, with corresponding values detailed in Table 10.
18

---

Table 11: Examples from the validation set of the HotpotQA dataset. For conciseness, all retrieved documents at each step are omitted. Correct answers are highlighted in blue, while incorrect answers are highlighted in red.
Query: What wrestling team Mathew Thomas Rehwoldt a part of?
RAG without Chain-of- Retrr-rvaa
Final Answer: WWF X
CoRAG (Ours)
Sub-query I: What is Mathew Thomas Rehwoldt's 's profession?
Sub-answer 1: No relevant information found.
Sub-query 2: What is Mathew Thomas Rehwoldt's 's name in the wrestling industry?
Sub-answer 2: Aiden English
Sub-query 3: What wrestling team is Aiden English part of?
Sub uub-answer33 The Vaudevillains
Final Answer: The Vaudevillains
Query: How many months apart are Johan Mjällby and Ncil Lennon in age?
RAG without Chain-of- Chatn-of-Retrivval Retrieval Final Answer: two months X
CoRAG (Ours)
Sub-query I: What is Johan Mjällby's birthdate?
Sub-answer 1: February 1971
Sub-query 2: What is Neil Lennon's birthdate?
Sub-answer 2: 25 June 1971
Sub-query 3: What is the difference in months between 9 February 1971 and 25 June 1971?
Sub-answer 3: 4 months
Final Answer: /
Thomas Parker Sanborn was model for the protagonist in the only novel by poet originally from what country?
Query:
RAG without Chain-of-Retriewe Retrieeal Final Answer: Italian X
CoRAG (Ours)
Sub-query 1: What country the poet originally from?
Sub-answer Hungary. Romania, Switzerland, China, Ukraine, France, Turkey. Sub-query 2: What is the name of the poct who wrote the only novel?
Sub-answer 2: Emily Brontë
Sub-query 3: What country is the poet who wrote the only novel originally from?
Sub-answer 3: United States
Sub-query 4: What the of the poct who wrote the only novel that Thomas Parker Sanborn was model for?
Sub-answer 4: George Santayana
Sub-query 5: What poct wrote the only novel that Thomas Parker Sanborn was a model for?
Sub-answer S: George Santayana
Sub-query 6: What country the poet George Santayana originally from?
Sub-answer Spain.
Final Answer: Spain
Query: Which university is further North, University of New Hampshire Stony Brook University?
RAG without Chain-of-Reed Rettrevall
Final Answer: University of New Hampshire X
CoRAG (Ours)
Sub-query 1: What is the location of the University of New Hampshire?
Sub-answer 1: Durham, New Hampshire.
Sub-query 2: What is location of Stony Brook University?
Sub-answer 2: Stony Brook, NY.
Sub-query 3: What is the latitude of Durham, New Hampshire?
Sub-answer 3: No relevant information found.
Sub-query 4: What is the latitude of Stony Brook, NY?
Sub-answer No relevant information found.
Final Answer: University of New Hampshire
19

---

Prompt: Sub-query Generation
You are using search engine to answer the main query by iteratively searching the web. Given the following intermediate queries and answers, generate new simple follow-up question that help answer the main query. You may rephrase or decompose the main query when previous answers are not helpful. Ask simple follow-up questions only as the search engine may not understand complex questions.
# Previous intermediate queries and answers (nteermediate queries and answers)
## Task description (task description)
## Main query to answer (query)
Respond with simple follow-up question that will help answer the main query, do not explain yourself output anything else.
Prompt: Intermediate Answer Generation
Given the following documents, generate appropriate answer for the query. DO NOT hallucinate any information, only use the provided documents to generate the answer. Respond "No relevant information found" if the documents do not contain useful information.
## Documents
(retrieved documents)
## Query
(sub-query)
Respond with a concise answer only, do not explain yourself or output anything else.
Prompt: Final Answer Generation
Given following intermediate queries answers, generate a final answer the main query by combining relevant information. Note that intermediate answers are generated by an LLM and may not always be accurate.
## Documents (retrieved documents)
## Intermediate queries and answers (intermediate queries and answers)
## Task description (task description
## Main query (query)
Respond with appropriate answer only, do not explain yourself output anything else.
20

---

Prompt: Learning to Stop
Given the following intermediate queries and answers, judge whether you have enough information to answer the main query. you believe you have enough information, respond with "Yes", otherwise respond with "No".
## Intermediate queries and answers (intermediate queries and answers)
## Main query (query)
Respond with "Yes" or "No" only, do explain yourself output anything else.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of work in Section 7.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

Justification:
No theoretical results are provided in the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Implementation details are provided in Section A.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer: [Yes]
Justification: We include implementation details in the paper and will release the code and data after publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. ec/
Policy) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / /nips. ccc/publcc//ee Seeiidds/CodeSuulei ssisssoliccyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Section 4.1 and A.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We report confidence intervals in Section C.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide details on compute resources in Section A.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: pettps:/neeure http:///eeruue.eeeee tttpp////errppp..ppppbtttt////// wwwwhhhhhhieellnneerr
Answer: [Yes]
Justification:
We have reviewed the NeurIPS Code of Ethics and our work conforms to it.
Guidelines:
The answer NA means that the authors have reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please see Section 7.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No safeguards are needed for our work as it does not involve high-risk data or models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All assets used in this paper properly credited and the licenses respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
26

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The documentation is provided in the supplemental material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing or human subjects were involved in this research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
No human subjects were involved in this research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
27

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https (https:// //tp://iss...eeeeee.e http//uuris..ceeoeeeeeeeeeeeeeee wwtp///oeeeeennce//2////111/ for what should or should be described.
28


Reviews:
[{"review_id": "sDfiXTYmpy", "reviewer": "Reviewer_ovoL", "summary": "The paper proposes CoRAG (Chain-of-Retrieval Augmented Generation), a framework that teaches language models to iteratively retrieve, reason over, and reformulate queries before producing an answer. Instead of the single pre-retrieval step used in standard RAG systems, CoRAG creates multi-step “retrieval chains”; these chains are constructed with rejection sampling using an instruction-tuned Llama-3.1-8B model, turning ordinary QA datasets into training data that include intermediate sub-queries, retrieved passages, and sub-answers. A single model is then fine-tuned on three intertwined tasks—next sub-query prediction, sub-answer generation, and final answer generation—so it can decide, at inference time, how many retrieval steps to take and which decoding strategy (greedy, best-of-N, or limited tree search) best balances accuracy against token budget. Experiments on four multi-hop QA datasets show double-digit EM gains over strong baselines, and on the full KILT benchmark an CoRAG-8B model sets new state-of-the-art scores on nearly every task. The authors also conducted comprehensive ablations, demonstrating the effectiveness of each design choice.", "questions": "- Not sure if I missed this but what model is used during the rejection sampling stage?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n- The proposed training framework -- CoRAG, is technically sound, intuitive, simple, yet quite effective.\n- Strong results on a set of multihop reasoning datasets and the CoRAG-8B model achieves a new state-of-the-art performance across all tasks, with the exception of FEVER on KILT benchmark.\n- Comprehensive analysis and ablations, clearly showing the how each design choice is made in the framework, which brings insights to the research community.\n- The calculation the conditional log-likelihood of “No relevant information found” at test time as a penalty score for each chain is smart and works well.\n\n### Weaknesses\n- From my perspective, I don't see major weaknesses of this paper. Here are some of my thoughts that could make the paper more solid: 1) The model is simultaneously trained on three tasks. It would bring more insights to the audience to conduct ablations over the training objective. 2) I would also like to see more experiments regarding the rejection sampling stage, e.g., how different models choices affect the final performance? How about other design choices compared to rejection sampling such as distilling from larger models or using larger models as a judge?", "was_revised": true, "final_justification": "- It's a technically solid paper and the authors have addressed my comments in their rebuttal. I keep my score unchanged.", "rebuttal": "Thank you for your encouraging feedback!\n\n>\"what model is used during the rejection sampling stage?\"\n\nWe use the off-the-shelf instruct model (e.g., Llama-3.1-8B-Instruct) for rejection sampling.\nFor iterative training, we use the model trained in the previous iteration.\nThis approach allows the model to generate its own training data in a bootstrapping manner.\n\n>\"It would bring more insights to the audience to conduct ablations over the training objective.\"\n\nThanks for the suggestion! One factor we studied is the data sampling ratio for $L_{\\text{subquery}}$ and $L_{\\text{subanswer}}$. We found that a sampling ratio of 0.2 works well and accelerates training. Increasing the sampling ratio to 1.0 resulted in only a negligible performance drop, as shown in the table below. Both models were evaluated with greedy decoding and a maximum of 6 retrieval steps. We will include this ablation in the revised paper.\n\n| **Model**             | **2WikiQA** |      | **HotpotQA** |      | **Bamboogle** |      | **MuSiQue** |      |\n|-----------------------|:-----------:|:----:|:------------:|:----:|:-------------:|:----:|:-----------:|:----:|\n|                       |     EM      |  F1  |      EM      |  F1  |      EM       |  F1  |     EM      |  F1  |  \n| CoRAG-8B (default)    |    70.6     | 75.5 |     54.4     | 67.5 |     48.0      | 63.5 |    27.7     | 38.5 |\n| w/ sampling ratio 1.0 |    69.9     | 74.8 |     53.9     | 67.0 |     48.8      | 63.2 |    26.0     | 36.5 |\n\n> \"I would also like to see more experiments regarding the rejection sampling stage, e.g., how different models choices affect the final performance? How about other design choices compared to rejection sampling such as distilling from larger models or using larger models as a judge?\"\n\nIn Table 3, we present an ablation study comparing the use of smaller LLMs (Llama-3.2-1B and Llama-3.2-3B) with GPT-4o for rejection sampling. The results align with expectations: larger models consistently lead to better performance. Using larger models as a judge is an interesting idea that we will explore in future work."}, {"review_id": "ezm5U6dz3P", "reviewer": "Reviewer_gW6e", "summary": "This paper proposes a novel framework named CoRAG for training and inference in RAG. The key idea is to perform multi-step retrieval via intermediate query reformulation and reasoning. The authors use rejection sampling to automatically generate training data with retrieval chains from QA-only datasets and train LLMs to predict sub-queries, sub-answers, and final answers. At test time, they explore several decoding strategies (e.g., greedy, best-of-N, tree search) to balance performance and efficiency. It achieves good results across multi-hop QA datasets and the KILT benchmark.", "questions": "Questions:\n\nQ1. Following W2. What is the retrieval accuracy (e.g., support fact recall) of CoRAG compared to strong retrievers? \n\nQ2. Following W3, Does the proposed framework improve performance when applied to other LLMs beyond LLaMA-3?\n\nQ3. Is there any human evaluation to verify the validity of generated sub-questions and sub-answers? \n\nQ4. In the ablation study on retrievers (Line 234), performance gains are attributed to stronger retrievers. Can the authors isolate the contribution of the CoRAG mechanism from that of the retrieval encoder?", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths: \n\n1. The methodology is clearly described, and the system components are coherently integrated.\n\n2. The test-time compute scaling analysis is well done and offers practical value for real-world deployment.\n\n\nWeaknesses:\n1. In Table 1, most baselines compared are designed for zero-shot or few-shot prompting, whereas the proposed CoRAG is fine-tuned for multi-hop QA task.  The only fine-tuned baseline is a standard LLaMA-8B model, which is not specifically optimized for multi-hop QA task. This discrepancy makes it challenging to draw clear conclusions from the unfair comparison. Including stronger fine-tuned baselines, especially encoder-based models optimized for multi-hop QA, would help contextualize CoRAG’s performance. For instance, CoRAG achieves an F1 of 69.8 on HotpotQA, whereas leaderboard models exceed 80.5 [1]. Similarly, on MuSiQue, CoRAG’s best F1 is 42.4, while top models score above 78.4 [2]. The full wiki setting is close to open-domain setting. \n[1] https://hotpotqa.github.io/\n[2] https://leaderboard.allenai.org/musique_ans/submissions/public\n\n2. Since one of the paper’s key ideas is improving retrieval quality through dynamic retrieval, it would be helpful to include standard retrieval metrics (e.g., recall@k or supporting fact accuracy) to better assess the effectiveness of the proposed method in retrieving supporting paragraphs, which is missing from the paper.\n\n\n3. The current experiments are conducted using one backbone. Including results with additional model backbones could strengthen the case for generalizability and robustness of the proposed approach.\n\n4. The idea of multi-step, dynamic retrieval has been extensively studied in prior work, such as IRCoT, Self-RAG, ITER-RETGEN, and others.", "was_revised": true, "final_justification": "Thank you for your response, which partially addresses my concerns regarding retrieval, so I have raised scores accordingly. However, my key concern about the unfair comparison noted in the weaknesses remains: it is still not fair to compare your fine-tuned model to other zero/few-shot LLMs, or to other fine-tuned models without retrieval.", "rebuttal": "Thanks for your constructive feedback!\n\n>\"CoRAG achieves an F1 of 69.8 on HotpotQA, whereas leaderboard models exceed 80.5 [1]. Similarly, on MuSiQue, CoRAG’s best F1 is 42.4, while top models score above 78.4 [2]\"\n\n**The leaderboard scores you cited for HotpotQA and MuSiQue are based on distinct and easier settings, making them not directly comparable to our results.** Specifically:\n* The MuSiQue-Ans leaderboard and HotpotQA Distractor setting provide gold paragraphs within the context, simplifying the task to reading comprehension. In contrast, our work focuses on an open-domain QA setting, where no gold paragraphs are assumed.\n* The HotpotQA Full-Wiki setting, by design, only requires retrieval from the abstracts of Wikipedia articles. Our approach does not incorporate this specific prior knowledge, as it does not generalize.\n\nWe will clarify these differences in the paper to provide a more accurate context for our evaluations.\n\n>\"it would be helpful to include standard retrieval metrics (e.g., recall@k or supporting fact accuracy) to better assess the effectiveness of the proposed method in retrieving supporting paragraphs\"\n\nWe have incorporated retrieval metrics for 9 KILT tasks in Table 7 of Appendix B. For multihop QA tasks, we calculated answer recall@{10, 20, 100} for your reference:\n\n|                       | R@10     | R@20     | R@100    |\n|-----------------------|----------|----------|----------|\n| HotpotQA              | 59.1     | 65.2     | 76.8     |\n| &nbsp;&nbsp; w/ CoRAG | **72.1** | **76.7** | **84.3** |\n| 2WikiMultiHopQA       | 54.9     | 62.1     | 74.6     |\n| &nbsp;&nbsp; w/ CoRAG | **81.4** | **84.8** | **88.8** |\n| Bamboogle             | 31.2     | 40.0     | 57.6     |\n| &nbsp;&nbsp; w/ CoRAG | **59.2** | **68.0** | **75.2** |\n| MuSiQue               | 29.0     | 36.5     | 52.7     |\n| &nbsp;&nbsp; w/ CoRAG | **47.1** | **54.6** | **68.4** |\n\nAs demonstrated, CoRAG consistently and significantly improves recall across all datasets, underscoring its effectiveness in retrieving relevant information.\n\nNotes: We followed DPR [1] in calculating recall based on answer matches, as not all datasets provide gold supporting paragraphs. For CoRAG, we utilized reciprocal rank fusion to merge multiple retrieval results into a single ranked list, from which recall was then calculated. For baselines, we use $E5_{\\text{large}}$ to retrieve the top 100 passages.\n\n[1] Karpukhin, V., Oğuz, B., Min, S., Lewis, P., Wu, L.Y., Edunov, S., Chen, D., & Yih, W. (2020). Dense Passage Retrieval for Open-Domain Question Answering. ArXiv, abs/2004.04906.\n\n>\"Does the proposed framework improve performance when applied to other LLMs beyond LLaMA-3?\"\n\nOur proposed method is designed to be model-agnostic and can be seamlessly applied to various LLMs beyond LLaMA-3. For instance, we have successfully applied CoRAG to the Qwen3 family of models, observing similar performance improvements:\n\n| Model                                 | 2WikiQA |      | HotpotQA |      | Bamboogle |      | MuSiQue |      |\n|---------------------------------------|:-----------:|:----:|:------------:|:----:|:-------------:|:----:|:-----------:|:----:|\n|                                       |     EM      |  F1  |      EM      |  F1  |      EM       |  F1  |     EM      |  F1  |  \n| Fine-tuned Qwen3-4B w/ $E5_{\\text{large}}$ |    49.3     | 55.3 |     45.0     | 57.9 |     32.8      | 43.1 |    13.4     | 23.8 |\n| CoRAG-Qwen3-4B (L=6, greedy)          |  **69.3**   | **74.1** |     **51.6**     | **64.2** |     **49.6**      | **62.5** |    **24.0**     | **34.5** |\n|                                       |             |      |              |      |               |      |             |      |\n| Fine-tuned Qwen3-8B w/ $E5_{\\text{large}}$ |    52.1     | 57.9 |     47.1     | 60.0 |     33.6      | 47.6 |    15.3     | 26.3 |\n| CoRAG-Qwen3-8B (L=6, greedy)          |    **70.0**     | **74.8** |     **52.8**     | **66.0** |     **49.6**      | **63.7** |    **25.2**     | **35.9** |\n\n>\"The idea of multi-step, dynamic retrieval has been extensively studied in prior work, such as IRCoT, Self-RAG, ITER-RETGEN, and others.\"\n\nWe acknowledge that multi-step retrieval has been explored in prior works as you mentioned. However, to our knowledge, none of the existing literature has investigated the test-time scaling of RAG within a fine-tuning setting. We believe this is a crucial aspect for emerging applications like Deep Research.\n\nAn apt analogy would be the distinction between zero-shot CoT prompting and o1-like training for long CoT: while LLMs can generate CoT in a zero-shot manner, explicitly training LLMs to produce longer and higher-quality CoT traces still yields significant benefits.\n\n>\"Is there any human evaluation to verify the validity of generated sub-questions and sub-answers?\"\n\nOur primary evaluation relies on automatic task metrics. However, for deeper analysis, we also conduct manual inspections of generated sub-questions and sub-answers. Representative examples are provided in Table 9 of the Appendix and in the supplementary material.\n\n>\"In the ablation study on retrievers (Line 234), performance gains are attributed to stronger retrievers. Can the authors isolate the contribution of the CoRAG mechanism from that of the retrieval encoder?\"\n\n**The ablation results concerning retrievers illustrate that stronger retrievers contribute to improved performance as an orthogonal factor, rather than being the primary source of CoRAG's gains.** In our main results (Table 1), we maintained a consistent retriever (E5$_{\\text{large}}$) across various decoding strategies and baselines, including SelfRAG and fine-tuned LLaMA-8B, to isolate and demonstrate the specific contributions of the CoRAG mechanism itself."}, {"review_id": "DA9hRR87Gt", "reviewer": "Reviewer_wBfm", "summary": "The paper introduces Chain-of-Retrieval Augmented Generation (CoRAG), a method that decomposes or reformulates a query into subqueries that the model can more effectively address. Training is conducted iteratively with rejection sampling, and the authors show how test-time scaling further enhances accuracy. CoRAG achieves strong performance, outperforming several baselines and setting a new state of the art on the KILT benchmark. The paper also offers thoughtful analysis and ablations, exploring key aspects such as the role of rejection sampling, the impact of using different retrievers or subquery generation models, and whether the model can learn to decide when to terminate the retrieval chain.", "questions": "It would be helpful to better understand the failure modes of the proposed approach. For instance, was there any analysis on the relevance or utility of the generated subqueries—e.g., what percentage of the chain consists of subquestions that are actually helpful or on-topic? Given that performance improves with increased generation length, one might assume that longer chains could also introduce noise or irrelevant steps. Did the authors observe any failure cases where longer chains hurt performance, or where irrelevant subquestions derailed the retrieval process?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "*Strengths*\n\n- The paper is well-written and easy to follow.\n\n- The proposed method achieves strong results across several tasks, and the accompanying analysis offers valuable insights that are likely to benefit the community.\n\n- The test-time scaling experiments are particularly interesting and add depth to the empirical evaluation.\n\n*Weaknesses*\n\n- A central claim of the paper appears to be that training a model to decompose queries via rejection sampling leads to performance gains. However, the results suggest that rejection sampling had limited impact, which weakens this contribution. It would have been valuable to explore this direction further—e.g., through preference learning or alternative training approaches.\n\n- Some relevant (albeit slightly older work was missed)  eg [1]\n\n\n[1] Buck, Christian, et al. \"Ask the right questions: Active question reformulation with reinforcement learning.\" arXiv preprint arXiv:1705.07830 (2017).\n\n\nOverall, I think this is a decent contribution and I recommend acceptance", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback and constructive suggestions!\n\n> \"the results suggest that rejection sampling had limited impact, which weakens this contribution. It would have been valuable to explore this direction further—e.g., through preference learning or alternative training approaches.\"\n\n**Our findings indicate that the choice of LLM for rejection sampling significantly impacts performance (Table 3), whereas merely increasing the number of sampled trajectories yielded limited gains (Figure 6 in Appendix B).** We hypothesize that the diversity and quality of sampled trajectories are currently constrained by the inherent capabilities of the LLM. We agree that exploring approaches like online rejection sampling or reinforcement learning, as you suggested, holds substantial promise for addressing this limitation, and we plan to investigate these directions in our future work.\n\n> \"Did the authors observe any failure cases where longer chains hurt performance, or where irrelevant subquestions derailed the retrieval process?\"\n\nYes, we indeed observed common failure patterns. A prominent one is the generation of sub-queries that are mere paraphrases of previous ones. The model sometimes persists in asking similar questions even when the retriever fails to find relevant information, rather than exploring alternative avenues. This issue is particularly pronounced with smaller models such as Llama-3.2-1B. We will provide a more detailed analysis of these failure cases in the final version of the paper.\n\n> \"Some relevant (albeit slightly older work was missed) eg [1]\n[1] Buck, Christian, et al. \"Ask the right questions: Active question reformulation with reinforcement learning.\" arXiv preprint arXiv:1705.07830 (2017).\"\n\nThanks for the pointer! \nWe agree that reinforcement learning is a crucial area for developing more effective RAG agents.\nWe will discuss this work in the related work section."}, {"review_id": "GxCM6rfAkf", "reviewer": "Reviewer_PDbv", "summary": "This paper introduces an approach for training o1-like RAG models called CoRAG. It combines the RAG with Chain-of-Thought during training. CoRAG first performs rejection sampling with log-likelihood as the signal of the correction, then training for the next sub-query prediction, sub-answer prediction, and final answer prediction. This method also introduces inference strategies: Greedy Decoding, Best-of-N Sampling, and Tree Search. Experiments show good performance.", "questions": "I am a little confused about the terms \"Current State\" and \"Next Action\" in Figure 1. How many times will the LLM be trained? Does the \"State\" refer to the token?\n\nSome recent papers have proposed R1-Zero-like RAG methods. It would be interesting to compare them. I am aware that they should be considered contemporary work, and this will not influence my evaluation of this paper.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "The paper is well-written and easy to follow.\n\nThe main contribution is to put RAG in o1-like training. It is interesting to combine the o1-training manner and RAG. However, the only novelty seems to be \"add RAG\" (please let me know if the authors believe this is incorrect). The method is also straightforward, but a simple method could also be a good method.\n\nExperiments are conducted on two sets of benchmarks with several datasets, which demonstrates its strong performance as well as good robustness and generalization.\n\nIn summary, the method is straightforward but useful, as demonstrated by experiments. I am slightly positive about the paper.", "was_revised": true, "final_justification": "Good writing and experiments. The method is straightforward, and I am a little concerned about the novelty. So my rating is 4.", "rebuttal": "Thank you for your positive feedback and constructive suggestions!\n\n>\"I am a little confused about the terms \"Current State\" and \"Next Action\" in Figure 1. How many times will the LLM be trained? Does the \"State\" refer to the token?\"\n\nIn Figure 1, \"Current State\" denotes the input context and instructions provided to the LLM. The \"Next Action\" refers to the LLM output responding to the given instruction. \nTo illustrate, consider sub-query generation: the \"Current State\" encompasses the original query along with all historical sub-queries and their corresponding answers, while the \"Next Action\" is the newly generated sub-query. This conceptualization similarly applies to both sub-answer generation and final answer generation. For detailed examples of our prompt templates, please refer to Page 19 of the manuscript.\n\nTo clarify the training process, the LLM is trained only once. We mix the training data for all three tasks, enabling simultaneous multi-task learning.\n\n>\"Some recent papers have proposed R1-Zero-like RAG methods. It would be interesting to compare them. I am aware that they should be considered contemporary work, and this will not influence my evaluation of this paper.\"\n\nWe appreciate your suggestion! We have incorporated a brief discussion of Search-R1 in the related work section. RL holds great promises for agentic systems; however, it is also well-known for high computational demands and training instability. Given the emergence of numerous concurrent works applying RL to RAG in recent months, we intend to substantially expand upon this discussion in the final version of our paper."}]

Author rebuttal:

Meta-review: This paper introduces CoRAG (Chain-of-Retrieval Augmented Generation). CoRAG is a  framework for training RAG systems to perform multi-step retrieval to answer complex queries.

The reviewers noted the following strengths of the paper:
* The paper is well-written and the proposed method clearly explained
* The proposed framework is novel and technically sound
* CoRAG achieves strong empirical results, establishing a new SOTA on the KILT benchmark as well as showing significant gains on multi-hop QA tasks
* The paper presents a comprehensive analysis of test-time compute scaling that demonstrates a trade-off between performance and token consumption

The reviewers also identified a number of weakness with the paper, including:
* There were some initial concerns about unfair experimental conditions with respect to few-shot baselines and leaderboard scores
* The novelty of the approach was also questioned, as the high-level concept of multi-step retrieval was previously explored

The authors provided convincing rebuttal that adequately addressed the primary concerns raised by the reviewers. The rebuttal clarified that the leaderboard comparisons were indeed not applicable due to differences in task settings. It highlighted the paper does include a direct and hence more fair comparison to a fine-tuned baseline, which CoRAG outperforms. Furthermore, the rebuttal included additional experimental evaluations focused on retrieval recall and model generalization, helping to strengthen the paper. The consensus among the reviewers is positive, as the initial weaknesses raised were largely resolved. 

Overall, this is a high-quality paper with strong results. Therefore, the final decision is to accept this paper.
Decision: Accept (poster)