Title: Chain-of-Retrieval Augmented Generation
Venue: NeurIPS 2025
Accepted: yes
Keywords: retrieval-augmented generation

scaling law

multi-hop reasoning
Abstract: This paper introduces an approach for training o1-like RAG models that retrieve and reason over relevant information step by step before generating the final answer. Conventional RAG methods usually perform a single retrieval step before the generation process, which limits their effectiveness in addressing complex queries due to imperfect retrieval results. In contrast, our proposed method, CoRAG (Chain-of-Retrieval Augmented Generation), allows the model to dynamically reformulate the query based on the evolving state. To train CoRAG effectively, we utilize rejection sampling to automatically generate intermediate retrieval chains, thereby augmenting existing RAG datasets that only provide the correct final answer. At test time, we propose various decoding strategies to scale the model's test-time compute by controlling the length and number of sampled retrieval chains. Experimental results across multiple benchmarks validate the efficacy of CoRAG, particularly in multi-hop question answering tasks, where we observe more than $10$ points improvement in EM score compared to strong baselines. On the KILT benchmark, CoRAG establishes a new state-of-the-art performance across a diverse range of knowledge-intensive tasks. Furthermore, we offer comprehensive analyses to understand the scaling behavior of CoRAG, laying the groundwork for future research aimed at developing factual and grounded foundation models.


Peer reviews:
comments: SUMMARY:
This paper introduces an approach for training o1-like RAG models called CoRAG. It combines the RAG with Chain-of-Thought during training. CoRAG first performs rejection sampling with log-likelihood as the signal of the correction, then training for the next sub-query prediction, sub-answer prediction, and final answer prediction. This method also introduces inference strategies: Greedy Decoding, Best-of-N Sampling, and Tree Search. Experiments show good performance.

STRENGTHS AND WEAKNESSES:
The paper is well-written and easy to follow.

The main contribution is to put RAG in o1-like training. It is interesting to combine the o1-training manner and RAG. However, the only novelty seems to be "add RAG" (please let me know if the authors believe this is incorrect). The method is also straightforward, but a simple method could also be a good method.

Experiments are conducted on two sets of benchmarks with several datasets, which demonstrates its strong performance as well as good robustness and generalization.

In summary, the method is straightforward but useful, as demonstrated by experiments. I am slightly positive about the paper.

QUESTIONS:
I am a little confused about the terms "Current State" and "Next Action" in Figure 1. How many times will the LLM be trained? Does the "State" refer to the token?

Some recent papers have proposed R1-Zero-like RAG methods. It would be interesting to compare them. I am aware that they should be considered contemporary work, and this will not influence my evaluation of this paper.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-06-25

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Chain-of-Retrieval Augmented Generation (CoRAG), a method that decomposes or reformulates a query into subqueries that the model can more effectively address. Training is conducted iteratively with rejection sampling, and the authors show how test-time scaling further enhances accuracy. CoRAG achieves strong performance, outperforming several baselines and setting a new state of the art on the KILT benchmark. The paper also offers thoughtful analysis and ablations, exploring key aspects such as the role of rejection sampling, the impact of using different retrievers or subquery generation models, and whether the model can learn to decide when to terminate the retrieval chain.\n\nSTRENGTHS AND WEAKNESSES:\n*Strengths*\n\n- The paper is well-written and easy to follow.\n\n- The proposed method achieves strong results across several tasks, and the accompanying analysis offers valuable insights that are likely to benefit the community.\n\n- The test-time scaling experiments are particularly interesting and add depth to the empirical evaluation.\n\n*Weaknesses*\n\n- A central claim of the paper appears to be that training a model to decompose queries via rejection sampling leads to performance gains. However, the results suggest that rejection sampling had limited impact, which weakens this contribution. It would have been valuable to explore this direction further—e.g., through preference learning or alternative training approaches.\n\n- Some relevant (albeit slightly older work was missed)  eg [1]\n\n\n[1] Buck, Christian, et al. \"Ask the right questions: Active question reformulation with reinforcement learning.\" arXiv preprint arXiv:1705.07830 (2017).\n\n\nOverall, I think this is a decent contribution and I recommend acceptance\n\nQUESTIONS:\nIt would be helpful to better understand the failure modes of the proposed approach. For instance, was there any analysis on the relevance or utility of the generated subqueries—e.g., what percentage of the chain consists of subquestions that are actually helpful or on-topic? Given that performance improves with increased generation length, one might assume that longer chains could also introduce noise or irrelevant steps. Did the authors observe any failure cases where longer chains hurt performance, or where irrelevant subquestions derailed the retrieval process?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel framework named CoRAG for training and inference in RAG. The key idea is to perform multi-step retrieval via intermediate query reformulation and reasoning. The authors use rejection sampling to automatically generate training data with retrieval chains from QA-only datasets and train LLMs to predict sub-queries, sub-answers, and final answers. At test time, they explore several decoding strategies (e.g., greedy, best-of-N, tree search) to balance performance and efficiency. It achieves good results across multi-hop QA datasets and the KILT benchmark.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\n1. The methodology is clearly described, and the system components are coherently integrated.\n\n2. The test-time compute scaling analysis is well done and offers practical value for real-world deployment.\n\n\nWeaknesses:\n1. In Table 1, most baselines compared are designed for zero-shot or few-shot prompting, whereas the proposed CoRAG is fine-tuned for multi-hop QA task.  The only fine-tuned baseline is a standard LLaMA-8B model, which is not specifically optimized for multi-hop QA task. This discrepancy makes it challenging to draw clear conclusions from the unfair comparison. Including stronger fine-tuned baselines, especially encoder-based models optimized for multi-hop QA, would help contextualize CoRAG’s performance. For instance, CoRAG achieves an F1 of 69.8 on HotpotQA, whereas leaderboard models exceed 80.5 [1]. Similarly, on MuSiQue, CoRAG’s best F1 is 42.4, while top models score above 78.4 [2]. The full wiki setting is close to open-domain setting. \n[1] https://hotpotqa.github.io/\n[2] https://leaderboard.allenai.org/musique_ans/submissions/public\n\n2. Since one of the paper’s key ideas is improving retrieval quality through dynamic retrieval, it would be helpful to include standard retrieval metrics (e.g., recall@k or supporting fact accuracy) to better assess the effectiveness of the proposed method in retrieving supporting paragraphs, which is missing from the paper.\n\n\n3. The current experiments are conducted using one backbone. Including results with additional model backbones could strengthen the case for generalizability and robustness of the proposed approach.\n\n4. The idea of multi-step, dynamic retrieval has been extensively studied in prior work, such as IRCoT, Self-RAG, ITER-RETGEN, and others.\n\nQUESTIONS:\nQuestions:\n\nQ1. Following W2. What is the retrieval accuracy (e.g., support fact recall) of CoRAG compared to strong retrievers? \n\nQ2. Following W3, Does the proposed framework improve performance when applied to other LLMs beyond LLaMA-3?\n\nQ3. Is there any human evaluation to verify the validity of generated sub-questions and sub-answers? \n\nQ4. In the ablation study on retrievers (Line 234), performance gains are attributed to stronger retrievers. Can the authors isolate the contribution of the CoRAG mechanism from that of the retrieval encoder?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes CoRAG (Chain-of-Retrieval Augmented Generation), a framework that teaches language models to iteratively retrieve, reason over, and reformulate queries before producing an answer. Instead of the single pre-retrieval step used in standard RAG systems, CoRAG creates multi-step “retrieval chains”; these chains are constructed with rejection sampling using an instruction-tuned Llama-3.1-8B model, turning ordinary QA datasets into training data that include intermediate sub-queries, retrieved passages, and sub-answers. A single model is then fine-tuned on three intertwined tasks—next sub-query prediction, sub-answer generation, and final answer generation—so it can decide, at inference time, how many retrieval steps to take and which decoding strategy (greedy, best-of-N, or limited tree search) best balances accuracy against token budget. Experiments on four multi-hop QA datasets show double-digit EM gains over strong baselines, and on the full KILT benchmark an CoRAG-8B model sets new state-of-the-art scores on nearly every task. The authors also conducted comprehensive ablations, demonstrating the effectiveness of each design choice.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- The proposed training framework -- CoRAG, is technically sound, intuitive, simple, yet quite effective.\n- Strong results on a set of multihop reasoning datasets and the CoRAG-8B model achieves a new state-of-the-art performance across all tasks, with the exception of FEVER on KILT benchmark.\n- Comprehensive analysis and ablations, clearly showing the how each design choice is made in the framework, which brings insights to the research community.\n- The calculation the conditional log-likelihood of “No relevant information found” at test time as a penalty score for each chain is smart and works well.\n\n### Weaknesses\n- From my perspective, I don't see major weaknesses of this paper. Here are some of my thoughts that could make the paper more solid: 1) The model is simultaneously trained on three tasks. It would bring more insights to the audience to conduct ablations over the training objective. 2) I would also like to see more experiments regarding the rejection sampling stage, e.g., how different models choices affect the final performance? How about other design choices compared to rejection sampling such as distilling from larger models or using larger models as a judge?\n\nQUESTIONS:\n- Not sure if I missed this but what model is used during the rejection sampling stage?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-07"}