Title: Search and Refine During Think: Facilitating Knowledge Refinement for Improved Retrieval-Augmented Reasoning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Language Model

Reinforcement Learning

Retrieval-Augmented Reasoning
Abstract: Large language models have demonstrated impressive reasoning capabilities but are inherently limited by their knowledge reservoir.
Retrieval-augmented reasoning mitigates this limitation by allowing LLMs to query external resources, but existing methods often retrieve irrelevant or noisy information, hindering accurate reasoning.
In this paper, we propose **AutoRefine**, a reinforcement learning post-training framework that adopts a new "search-and-refine-during-think" paradigm.
AutoRefine introduces explicit knowledge refinement steps between successive search calls, enabling the model to iteratively filter, distill, and organize evidence before generating an answer.
Furthermore, we incorporate tailored retrieval-specific rewards alongside answer correctness rewards using group relative policy optimization.
Experiments on single-hop and multi-hop QA benchmarks demonstrate that AutoRefine significantly outperforms existing approaches, particularly in complex, multi-hop reasoning scenarios.
Detailed analysis shows that AutoRefine issues frequent, higher-quality searches and synthesizes evidence effectively.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method to enhance retrieval-augmented generation (RAG) by allowing a large language model (LLM) to first think before formulating a search query, and then generate an answer based on the retrieved documents. Building upon prior work such as Search-R1, the authors introduce a novel refine step: after retrieval, the LLM processes the raw documents into a refined form, aiming to prevent misleading information from affecting the final answer. The entire pipeline is trained using reinforcement learning (RL), with a retrieval reward that encourages fetching documents containing the correct answer. Experimental results demonstrate that both the refine step and the retrieval reward contribute to improved performance.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n- The proposed method is simple yet effective.\n- The paper is well-written and easy to follow, with clear analyses of the method's components.\n- The experimental results support the effectiveness of the proposed additions (refinement and reward design).\n\n## Weaknesses\n\n- **Incremental Contribution.** The main idea appears to be an incremental improvement over Search-R1 [1], which already introduced the paradigm of reasoning-before-retrieval trained via RL. The notion that irrelevant context can harm model performance is not new [2], and several previous works have proposed techniques to mitigate this by extracting only relevant information [3]. The contribution of this paper lies primarily in combining document refinement with the Search-R1-style framework and demonstrating its effectiveness, but this alone may be insufficient to warrant acceptance unless further novelty is established.\nTo better establish the novelty, the authors should articulate the unique challenges or benefits of applying document refinement within a Search-R1-style reasoning-then-retrieval framework, especially in contrast to naive RAG or prior refinement approaches used in more traditional pipelines. Clarifying what makes refinement in this setting particularly necessary or non-trivial (e.g., interaction with the RL objective, refinement timing, or complexity from prior reasoning steps) would significantly strengthen the contribution.\n- **Unclear Justification for RL.** In the Introduction, the authors state that “SFT often fails to generalize retrieval behaviors beyond the training distribution.” However, there appears to be no empirical evidence in the paper supporting this claim, nor do the cited references directly address RAG settings. The necessity of RL for both query generation and refinement should be better motivated. Could supervised fine-tuning (SFT) or instruction tuning suffice for learning the refinement step?\n- **Misleading Title.** The part of title “Autonomous Retrieval-Augmented Reasoning of LLMs” is somewhat misleading. The core ideas behind autonomous retrieval-augmented reasoning were already developed in Search-R1, while this paper primarily introduces a refinement module and retrieval reward. The current title may give the impression that this paper introduces the full framework from scratch.\n\nQUESTIONS:\n1. Regarding the retrieval reward: how does the method handle cases where (a) the original documents do not contain the correct answer, or (b) the answer is present in the original documents but lost during refinement? Are these two scenarios treated equivalently during training?\n2. In Section 1, the paper claims that SFT struggles to generalize retrieval behaviors to out-of-distribution (OOD) scenarios. Can the authors provide empirical evidence or prior work that supports this claim, especially in the context of retrieval-augmented generation?\n3. Is RL truly necessary for learning the refinement step? Have the authors compared against a supervised or prompted alternative (e.g., using gold summaries or instruction-tuned additional refiners) to validate that RL offers a meaningful advantage?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problem of training an LLM to issue search queries to enable RAG. Based on the observation that prior works have mostly focused on eliciting search skills in the LM via SFT, the proposed method uses RL to train a reader model. While the proposed paper is not the first to use RL for training a reader, they make two key contributions:\n1. the proposed method, AutoRefine, flexibly summarizes (“refines”) retrieved documents in addition to retrieving, akin to https://arxiv.org/abs/2311.08377\n2. the proposed method uses process rewards for retrieval\n\nOn a variety of question answering benchmarks, the proposed method is shown to be very promising.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The experiments in this paper are very well-chosen and give a complete picture of the proposed method.\n- The empirical support for the efficacy of this method is strong for multi-hop QA. \n\nWeaknesses:\n- It seems that the two methodological contributions of this paper (\"Refine\" and \"Process Rewards for Search\") each help on different benchmarks. This suggests that potentially these two methods are actually more targeted. I was hoping to see more discussion of this.\n- The retriever reward uses a binary reward based on term matching of the answer with the ground truth. This seems to effectively be training the model to prefer exact match queries, which may not be globally optimal. I'd like to see more discussion of the possible risks of this approach.\n- As mentioned by the authors in the limitation section, the proposed method is only demonstrated with 3B models, which are a bit smaller than standard for readers in a RAG system these days (8B has become the norm in academic work). It's likely that 7B or 14B models will benefit less from carefully-designed inductive biases such as the ones proposed in this paper. In any case, this does not invalidate the value of the paper.\n\nQUESTIONS:\n- The retriever reward uses a binary reward based on term matching of the answer with the ground truth. This seems to effectively be training the model to prefer exact match queries, which may not be globally optimal. Is this a legitimate concern?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a method for enhancing reinforcement learning (RL) training in retrieval-augmented question answering through evidence compression. It introduces soft rewards designed to promote faithful answer generation based on retrieved context. The approach is evaluated on diverse Wikipedia-based short-form single- and multi-hop open-ended QA tasks, demonstrating performance gains compared to existing baselines.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n1. The proposed method demonstrates improvements over multiple baselines, such as Search-R1 and ReSearch, by a relative margin over 20% in hard multi-hop tasks.\n\n\n### Weaknesses\n1. The paper appears to be an incremental development over Search-R1[1] and Search-o1[2]. It is established in the recent works, such as Search-o1[2] (Figure 2, \"Get concise information and continue coherent reasoning.\") and FILCO[3], that evidence compression plays critical role to avoid noise generate from imperfect retrieval. \n\n2. Though the focus of the paper is on evidence compression, no strong baseline is chosen related to it, such as FILCO[3] and FaviComp[4]. Moreover, the Related Work section completely overlooks all evidence compression literature. Moreover, important details of the baselines are missing, such as which model size and algorithm (For Search-R1, PPO or GRPO?) was chosen for comparison.\n\n3. Model scale and scalability: The paper does not discuss model scale’s impact on generalization or scalability. The choice of GRPO over PPO for Search-R1 is unclear, as PPO is a stronger baseline for this task in Search-R1.\n\n4. Limited applicability: The soft-reward mechanism is restricted to short-form generations, with no exploration of its utility for long-form tasks (e.g., Self-RAG). This limits the method’s real-world relevance. It is also unclear  what is the  latency of  the method over baselines for use experience.\n\n5. Statistical rigor: Error bars or standard deviations are omitted, such as in Table 1 and Figure  4, despite the checklist mentioning them to validate result significance. This weakens the reliability of the reported improvements.\n\n\nReferences:\n\n[1] Jin, Bowen, et al. \"Search-r1: Training llms to reason and leverage search engines with reinforcement learning.\" arXiv preprint arXiv:2503.09516 (2025).\n\n[2] Li, Xiaoxi, et al. \"Search-o1: Agentic search-enhanced large reasoning models.\" arXiv preprint arXiv:2501.05366 (2025).\n\n[3] Wang, Zhiruo, et al. \"Learning to filter context for retrieval-augmented generation.\" arXiv preprint arXiv:2311.08377 (2023).\n\n[4] Jung, Dongwon, et al. \"Familiarity-aware evidence compression for retrieval augmented generation.\" arXiv preprint arXiv:2409.12468 (2024).\n\n[5]  Asai, Akari, et al. \"Self-rag: Learning to retrieve, generate, and critique through self-reflection.\" The Twelfth International Conference on Learning Representations. 2023.\n\nQUESTIONS:\n1. Hyperparameter justification: The paper lacks explicit rationale for key hyperparameters (e.g., choice of E5 as retriever, 3 documents fetched during training). If informed by prior work, this should be clearly stated to demonstrate scientific grounding.\n\n2. Reward function ablation: The reward function’s design (e.g., 0.1 as partial reward, no partial reward for correct final answers) is not analyzed. This omission limits understanding of the method’s effectiveness and trade-offs.\n\n3.  Training configuration ambiguity: The choice of maximum search calls (5) and whether training settings align with Search-R1 are unclear. Section B.1 and Table 3 do not provide sufficient details to verify reproducibility or consistency with prior work.\n\n4. Data split specification: The dev split of Hotpot-QA (distractor vs. fullwiki) is not explicitly stated, creating ambiguity about the evaluation’s scope and fairness.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces AutoRefine, an RL post-training framework that RAG by adding an explicit “search-and-refine-during-think” reasoning loop. In each reasoning cycle the model (i) plans (\\<think\\>), (ii) issues a search query (\\<search\\>), (iii) receives documents, (iv) refines those documents to distill the most relevant facts (\\<refine\\>), and (v) eventually produces an answer (\\<answer\\>). In this reasoning process, the authors introduced two rewards: 1) the outcome reward (whether the final answer is correct) and 2) the retrieval reward, which is measured based on the quality of refined documents within \\<refine\\>\\</refine\\> blocks. The model is trained end2end using GRPO. To demonstrate the effectiveness of the proposed method, the authors conducted experiments across a set of sing-hop and multi-hop datasets. The strong empirical results show that AutoRefine is effective in identifying and addressing knowledge gaps via multi-turn, high-quality search queries.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- The proposed \"search-and-refine-during-think\" approach is technically sound and well-presented.\n- Strong empirical results over a set of single-hop and multi-hop datasets. It not only outperforms the baselines but also the strong results in the literature.\n- The experiments are comprehensive, including thorough ablations showing the effectiveness of the framework design.\n\n### Weaknesses\n\nThe major concern I have regarding this paper is its similarity to the Search-R1 paper. The core architecture, prompt template, RL framework, datasets, and evaluation metrics of AutoRefine mirror those of Search-R1. The new elements introduced in AutoRefine are (i) inserting a <refine> block in which the model summarizes the retrieved passages and (ii) adding a “retrieval-specific” reward that checks whether the gold answer appears inside that block. Search-R1 could adopt these two tweaks with minimal changes. Although I admit that making those two changes brings significant gains over multi-hop reasoning datasets, I am not fully convinced that it is worth a NeurIPS publication. From my perspective, exploring more diverse experimental settings could make the paper much stronger. For example, if the ground-truth is given as a long and complex answer, how to compute the retrieval reward, and how does that affect the final performance?\n\nQUESTIONS:\nn/a\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-05"}