Search and Refine During Think: Facilitating Knowledge Refinement for Improved Retrieval-Augmented Reasoning
Authors: Yaorui Shi
Sihang Li
Chang Wu
Zhiyuan Liu
Junfeng Fang
Hengxing Cai
An Zhang
Xiang Wang
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Model
Reinforcement Learning
Retrieval-Augmented Reasoning
Primary area: reinforcement_learning

Abstract: Large language models have demonstrated impressive reasoning capabilities but are inherently limited by their knowledge reservoir.
Retrieval-augmented reasoning mitigates this limitation by allowing LLMs to query external resources, but existing methods often retrieve irrelevant or noisy information, hindering accurate reasoning.
In this paper, we propose **AutoRefine**, a reinforcement learning post-training framework that adopts a new "search-and-refine-during-think" paradigm.
AutoRefine introduces explicit knowledge refinement steps between successive search calls, enabling the model to iteratively filter, distill, and organize evidence before generating an answer.
Furthermore, we incorporate tailored retrieval-specific rewards alongside answer correctness rewards using group relative policy optimization.
Experiments on single-hop and multi-hop QA benchmarks demonstrate that AutoRefine significantly outperforms existing approaches, particularly in complex, multi-hop reasoning scenarios.
Detailed analysis shows that AutoRefine issues frequent, higher-quality searches and synthesizes evidence effectively.

Search and Refine During Think: Facilitating Knowledge Refinement for Improved Retrieval-Augmented Reasoning
Yaorui Shi"", Sihang Li Chang Wu', Zhiyuan Liu?, Junfeng Fang".
Hengxing Cai An Zhang', Xiang Wang
University of Science and Technology of China
National University of Singapore
DP Technology
Cyaoruishi, sihang0520, xiangwang1 1 223 223]@mmail. com, caihengxing@dp. tech
Equal contribution. Corresponding author.
Abstract
Large language models have demonstrated impressive reasoning capabilities but inherently limited by their knowledge reservoir. Retrieval-augmennte reason- ing mitigates this limitation by allowing LLMs to query external resources, but existing methods often retrieve irrelevant noisy information, hindering accurate reasoning. In this paper, we propose AutoRefine, reinforcement learning post- training framework that adopts new -awrrh--and-rffie--ddringg AutoRefine introduces explicit knowledge refinement steps between successive search calls, enabling the model iteratively filter, distill, and organize evidence before generating an answer. Furthermore, we incorporate tailored retrieval-specific rewards alongside answer correctness rewards using group relative policy optimiza- tion. Experiments on single-hop and multi-hop QA benchmarks demonstrate that AutoRefine significantly outperforms existing approaches, particularly complex, multi-hop reasoning scenarios. Detailed analysis shows that AutoRefine issues frequent,
paradigm.
searches and synthesizes evidence effectively. Code is available https:///e //gi thub. com/syy-cn/AutooeFi.
1 Introduction
Large language models (LLMS) have shown impressive abilities in language understanding, planning. and problem solving
[1-3]. Recent advances demonstrate that reinforcement learning (RI. [4] further enhances LLMs" reasoning capabilities [5, 6]. especially in complex tasks such mathematics and coding [7. 8]. However, the knowledge encoded in LLMs is inherently constrained by their training corpora, limiting their reasoning performance tasks requiring up-to-date information [9, 10].
common strategy to address this limitation is retrieval- augmented generation (RAG), which equips LLMs with retrieval tools to access external knowledge bases during question answering [11-13]. Widely-adopted RAG pipelines typically rely on supervised fine-tuning (SFT) to train LLMs to issue search queries and generate responses based on retrieved documents [14-16]. While SFT can be effective for training large models for search, it sometimes necessitates the construction of high-quality search paths, which incurs additional effort and resource overheads [17]. To address this, recent studies draw inspiration from RL-based post-training [5] and explore RL for etrieval-auggmenn reasoning, achieving excellent results by only evaluating final answer correctness without the need for
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(a)
Whates
Dowr
Bock loc daye''s
User
Palalaniin led piaee for
User
Filsk: Fre Fred inok the witn Mann Davidl diydd
FiLLE 1 (oee TaYe
LLM
Saarch Dacid Dacktov Canerr
Bu alte male bun dud 1900
Braaagm like Cog 2011
13 Haurec Galayg
Ta weneediiiin after oc oower
Eiore
May aftee 20yer coner.
Edamaa
Foistt Indt the stadkum ierrctiater Unitedd
tear Parls dave dorraats
Totte Step
Princ Bisi the stokker Paees Solee Gaeroin
LM
Munchodkr Unittd home stulion
Search Perr Sunt-kemnn stadim
The Kariie in ansi uoe loods buttell auiker in
le buttu ooke to ssarci ool skibee
Toa in The and has uar he teme Prosch Gerair
Algilo aiggeei Laripe...
Extora
Anj her The horns staduan Pare
Aucerr: Old Trafford
Sufte Shp
LIM
Previous Mothods: "search-during--lie
AutoRefine:
"search-and seafinnnndiiinngiie
(b)
[Trrk]
[Soardt
[Poaueeri]
[hea]
[Seact]
Recumand
plattedd
Locks Supersiiion
Rebiovall
Groandrrukk: Alere-dagaate Recuir
Specitic fraward
Groane Trask:
Plem--Angicce Rassal
Previous Mathoda:
Optimina with outcome reward
AutoRafinee Optimiza with outcome and retrieval rewarrds
Figure 1: Comparison between previous retrieval- augmented reasoning methods and AutoRefine. (a) Previous "scarch-during-thinn think" models can get distracted by irrelevant details between retrieval steps, leading to incorrect answer. AutoRefine introduces <Refine> step where the model explicitly refines crucial evidence, enabling the model to link information across multiple hops, plan its next query, and arrive at the correct final answer. (b) While previous methods rely only on nutcome-based rewards, AutoRefine incorporates a retrieval-specific reward to directly supervise the <Refine> step.
pre-collected reasoning paths [18-23]. Scrutinizing existing studies on Strieval--augeent reasoning. we summarize a common becarch-during-thine "hhinn" paradigm: given prompts with special search tokens (e.g., '<search>... ... </search>'), the LLM is trained via RL to autonomously invoke retrieval tools, retrieve some documents from external knowledge bases, and generate answers within '<answer>.. </answer>' using the retrieved information.
Despite their promising results, we identify two core limitations inherent in the current retrieval- augmented reasoning paradigm:
Lack refinement of retrieved documents. When facing oft-of-scope questions, LLMs often require pieces of precise factual information (e.g., names of historical figures, dates of events). However, the current bearcc-dduinngtthink paradigm typically uses retrieval tools to return full documents based on input queries, many of which are noisy or only weakly relevant. As illustrated in Figure I(a), previous methods reason directly over raw retrieved content, making susceptible to getting distracted by irrelevant details. This is particularly problematic in multi-hop scenarios, where a distraction in an early step can derail the entire reasoning chain.
Underexplored retrieval-specific rewards. While prior work on RL post-training highlights the importance of reward design [24-26], most retrieval-augmeente reasoning methods rely solely on an outcome-based reward- - typically assessing the correctness of the final answer. As shown in Figure 1(b), this coarse supervision underexplores retrieval- -specfffc rewards and offers little direct guidance for improving the retrieval process itself. As a result, it could be difficult for the LLM to learn how to retrieve more relevant or informative documents.
To address these limitations, we propose AutoRefine, a simple yet effective RL post-training frame- work that enhances the LLM's autonomous retrieeal-augmmente reasoning capability. At its core, AutoRefine adopts a "search-and -eniaaad--fiee-uuin---iiie
paradigm, guided by a combination of answer and retrieval rewards. First, unlike prior "search-during-thin approaches that overlook refinement, we introduce an explicit knowledge refinement step into the reasoning loop using a '<search> s/searcch <refenee ... </refine>' template ($2.1). This template encourages the model
2

---

de Who letter mrsdtm peyyholloy?
Mocel
Group
Thisk: (wood
mesecc batee plrndde
Wirni Corman oe
Jexe tateer prrddon
looe and of Rovedi aguan
Becuneser
Brylnc Whell luud -ppatiie Joeeh
Doek Wart Whrrd
peetrel
Maric
Heline
Search
[irlive
[iravel
uet Worth Saliee
Withien fidt
Sear wiovvr Willaan
Deenty
(hhvrll
jemwer
ptechsingio masurr
Wr Loog
reter himael paychwlogur
[inik]
Proved
Aunver Mitine Word
Figure 2: The training scheme of AutoRefine. (right) An actor model generates diverse reasoning trajectories for given question, including think, search, refin,, and answer. (left) These trajectories are optimized using the GRPO [7] algorithm described in Equation (4), where the loss on retrieved documents is masked out. We take G in this example.
explicitly distill crucial evidence from retrieved documents. By isolating key facts, the model can better link information across multiple retrieval hops and accurately plan subsequent queries before generating final answer. Second, under this template, apply Group Relative Policy Optimization (GRPO) [5] to train the model with both outcome-based and retrieval-specific rewards, rather than outcome-based rewards alone ($2.2). Specifically, during training, we first sample several trajectories from the model, each consisting of sequence of <think>, <search>, <refine>, and <answer> steps. While the answer reward evaluates the final output, the retrieval reward is computed based on the quality of the content within the <refine> blocks, providing direct supervision for the refinement step. This joint reward design explicitly guides the model to extract, organize, and utilize gre-ggained knowledge throughout reasoning.
To empirically assess AutoRefine, we conduct experiments both single-hop [27-29] and multi- hop [30-33] question answering (QA) benchmarks. AutoRefine surpasses leading methods [5, 18, 21, 19] by 6.9% higher average accuracy, and shows especially high performances in multi-hop scenarios (ef. Table 1). It demonstrates a strong ability in identifying and addressing knowledge gaps via multi-turn, high-quality search queries. The knowledge refinement steps also effectively extract crucial information from noisy retrieved documents, directly contributing to improved answer quality. Additional experiments confirm the contribution both the retrieval-specific reward and the refinement module, and AutoRefine holds robust performance across different retrieval depths.
2 Method
In this section, we introduce AutoRefine, a simple yet effective RL. framework that enhances the LLM's autonomous retrieval-augmented reasoning capability. We first outline the overall task formulation and trajectory generation steps, highlighting the novel paradigm ($2.1). Next, detail our reward modeling with both answer-based and retrieval-specific rewards to encourage fine-grained knowledge refinement ($2.2). Due to limited space, we refer to Appendix A for related work about reasoning in LLMs and retrieval augmented generation.
2.1 Trajectory Generation with Searching and Refinement
Task Formulation. Given dataset D = {(g. a)} containing question question-answer answer pairs and an external search engine ε, the task of retrieval- -ugmeeete reasoning requires the LLM to generate reasoning trajectories o by iteratively interacting with the knowledge source E. Formally, for each question q. we generate a reasoning trajectory: o = (TI, 72, TT), where the 1-th intermediate reasoning step 7. (st, consists of an action 8, {<think>, <search>, <documents>, <refine>, <answer> and its associated content e, The model is expected to repeatedly retrieve and refine knowledge from E until reaching final answer Oans that correctly addresses the question q.

---

You are helpful assistant who is good answering questions with multi-tum search engine calling. To answer questions, you must first reason through the available information using <think and </think>, If you identify missing knowledge, you may issue a search request using <search> query </search> at any time. The retrieval system will provide you with the three most relevant documents enclosed in <documents> and </documents>, After each search, you need to summarize and refine the existing documents in <refine> and </refine>. You may send multiple search requests if needed. Once you have sufficient information, provide a concise final answer using <answer> and </answer>. <user> Question: [QUESTION] </user>
Figure 3: Prompt template for rollout generation.
Rollout Generation. The actor LLM лe generates trajectories by performing multiple rounds of interactions with the search engine ε. The trajectories contain multiple internal reasoning cycles, as il- lustrated in Figure 2. Each eycle consists of sequence of structured operations: "<thinke..<<thinnke for overall planning of consequent search actions, "<search>... "ssearch>...<search>>> </searchh>" for querying the ex- ternal search engine, "<document2...<ddocumee "<refine>... </refe/i/.///iii>>>> for distilling relevant information from the retrieved content. Following the block
for incorporating the retrieved documents, and
reasoning phase, the model generates the final response within the based on the refined knowledge. Notably, the number of internal cycles is not manually pre-defined but autonomously determined by the actor LLM, adapting dynamically to the difficulty of the question. These tokens are defined and explained to the model via system instructions (cf. Figure 3).
Stopping Criteria. The generation terminates when an answer action is produced, i.e., ST <answer>, The content ст of the terminal state Ty is extracted as the trajectory's 's final answer Oans
2.2 Reward Modeling with Retrieval-Aware Signals
We simple rule-based rewards to encourage free exploration during the RL process. The reward in AutoRefine consists of two complementary components: (1) the Outcome-Based Reward, which directly assesses the correctness of the answer generated by the model, and (2) the Retrieval-Spect Reward, which encourages the model to accurately identify and extract answer-relevant information from the retrieved documents.
Outcome- Based Reward. The outcome- reward (aka. the answer reward) RAms E [o, 1] compares the model's final answer within the fnnswer></answer block to the ground-truth answer, measuring its correctness. Formally, we treat the predicted and ground-truth answers as sets of words, and use Fl-score between these two sets the reward:
2/0ams O
Ram F1 (Ouns a) |oans + |a]"
(1)
where Oans is the predicted answer (e.g.. "Pierre-Auguee Renoir" in yellow box of Figure 1(a), and is the ground truth answer from the (g, a) pair (e.g., the gray box in Figure 1(b)).
Setrieval-Specifii Reward. We further introduce an additional reward, the retrieval reward RRet (0,1), 1), to explicitly encourage the extraction and utilization of relevant information from noisy retrieved documents. The retrieval reward is measured based on the quality of refined documents within the <refine<</refin>> blocks. Specifically, we collect all knowledge refinement steps (i.e., content within the <refinev...</effing blocks) across the trajectory and concatenate them into single text sequence:
RRe T(a O Orelime = a),
where T(.)) is the indicator function, Orefine = U( (st, (t) Eo A 8, = <refinee>>> } is the concate- nation of all the knowledge refinement steps (e.g.. "The documents concludes... Thh Umbrellas": "" in blue box of Figure 1). This reward is activated when all components of the ground-truth answer present in the refined knowledge, reinforcing faithful and targeted information extraction.
Integrating Outcome and Retrieval Rewards. The overall reward function in AutoRefine is designed to encourage both accurate final answers and meaningful intermediate knowledge extraction. Specifically, the model receives full reward of 1 if generates the correct answer. If the final answer

---

is incorrect but some relevant information has been extracted during the refinement step, partial reward of 0.1 is assigned. No reward is granted if neither correct answers nor relevant information are produced. The overall reward Roverall can be formally written as:
RAns: 0.1, if Ans and Rect >
if Ans
Roverall
0.
if RAns- RRct = 0
Training Objective. We apply Group Relative Policy Optimization (GRPO) [7] as the policy optimization algorithm for RL. The overview of the GRPO training scheme is shown in the top-left corner of Figure 2. Formally. given an actor model kg and reference model The a group G rollouts {or] is sampled as described in $2.1. We optimize the actor model me by maximizing:
Ta(oi, g. gi,c)) Autt
argmax (0) =B (. Moa (11
min
0
G
Oj
To Toin
9, 01,t)
2m1
Tolo 01,2()
clip
1 +e Ai,
-BDLL (To Tret]
To (0 g. 0i,)ct)
(4) where Ai,- Ira, - mean(r:)] / (re) is the normalized token-level advantage for the i-th rollout in the group, G is the group size, is the clipping ratio, and is the coefficient for the estimated KL divergence. As shown in the bottom-left part of Figure 2, we mask out the retrieved documents during the loss computation.
3 Experiments
In this section, we aim to answer the following Research Questions (RQs):
RQI: How effectively does AutoRefine's 'sencchaand-refie-ddri-----ii paradigm enhance perfor- mance in retrieval-augmmenedd question answering?
RQ2: Can AutoRefine effectively resolve information gaps through retrieval, especially when facing complex multi-hop problems?
RQ3: Can knowledge refinement steps distill critical information from retrieved documents?
RQ4: Can AutoRefine achieve robust performance improvements under different retriever settings?
3.1 Experiment Setup
Datasets.
We evaluate performance using seven diverse QA benchmarks, including three single-hop Natural Questions (NQ) [27]. TriviaQA [29]. PopQA [28]. and four datasets that require
QA datasets:
multi-hop searching: HotpotQA [30], 2WikiMultihopQA (2Wiki) [31], Musique [32], Bamboogle [33] for evaluation. Exact match accuracy serves the evaluation metric for all downstream datasets. Following the setting of prior works [19], we train AutoRefine using combined training set from
NQ and HotpotQA.
Baselines. In our experiments, we compare AutoRefine against three kinds of methods: (1) genera-
tion without retrieval (w/o Retrieval), including direct generation with LLM, supervised fine-tuning (SFT), and RI R1-like training (R1) [5] without Retrieval; (2) methods with single-hop retrieval (w/ Single-Hop Retrieval), including direct retrieval with the input question (Naive RAG): (3) training
with retrieval (w/ Multi-Hop Retrieval) including: agentic search method Search-ol [18], IRCoT [34], eetrieval-augmented reasoning model Search-RI [19] and ReSearch [21].
Implementation Details. To simulate real-world search scenario, we remove original context documents from the QA datasets [30-32] and instead use the December 2018 Wikipedia dump [35] as the external knowledge source, with E5-bbse--2 [36] as the retrieval engine. By default, the search engine retrieves the top three most relevant documents on each query. RL--based baselines, we run experiments using both Qwen2.5-33 5-B.5-3-BBBaa and -Instruct models. For SFT and direct generation
5

---

Table (RQ1) Accuracy comparison of AutoRefine versus baseline methods with Qwen2.5-3B [3] across various QA benchmarks. Bold denotes best results, and underline denotes second best results.
Songle-Hop QA
Multi-Hop QA
Methods
NQ
TriviaQA
PopQA
HotpoiQA
2Wiki
Musique
Bamhoogle
Avg.
w/o Retreival
Direct Generation
0.106
0.288
0.108
0.149
0.244
0.020
0.024
0.134
SFT
0.249
0.292
0.104
0.186
0.248
0.044
0.112
0.176
RI-Instruct [S]
0.210
0.449
0.171
0.208
0.275
0.060
0.192
0.224
RI-Basc [5]
0.226
0.455
0.173
0.201
0.268
0.055
0.224
0.229
w/ Single-Hop Retrieval
Naive RAG [37]
0.348
0.544
0.387
0.255
0.226
0.047
0.080
0.270
w/ Multi-Hop Retrieval
Search-ol [18]
0.238
0.472
0.262
0.221
0.218
0.054
0.320
0.255
IRCoT [34]
0.111
0.312
0.200
0.164
0.171
0.067
0.240
0.181
ReSearch-Instree [21]
0,365
0.571
0.395
0.351
0.272
0.095
0,266
0.331
ReScarch-Basc [21]
0.427
0.597
0.430
0.305
0.272
0.074
0.128
0.319
Search-R/ Biich-RRIIInstrcue [19]
0.397
0.565
0.391
0.331
0,310
0.124
0.232
0.336
Search-B RI--RI-BBB I-Base [19]
0.421
0.583
0.413
0.297
0.274
0.066
0.128
0.312
AutoRefine-Instreen
0.436
0.597
0.447
0.404
0.380
0.336
0.396
AutoRefine-Base
0.467
0.620
0.450
0.405
0.393
0.157
0.344
0.405
baselines, we use the instruct variant to better align with instruction-folloving tasks. Most baseline results are taken from Search-RI [19], which has experimental settings consistent with ours. We reproduce ReSearch using the authors' publicly available code. Additional implementation details can be found in Appendix B, and further experimental results are provided in Appendix
3.2 Overall Performance (RQ1)
Table 1 presents the overall performance comparison between AutoRefine and the baseline methods. The Avg. column stands for the average accuracy. As shown in the results, AutoRefine significantly outperforms baseline models across the seven benchmarks. It achieves 0.069 accuracy gain on the base variant and a 0,060 improvement on the instruct one compared to the strongest baseline.
Besides the overall performance, we observe that the performance gains achieved by AutoRefine more obvious on the multi-hop QA benchmarks. For example, AutoRefine improves the performance on 2Wiki by 0.083 and Musique by 0.045, which implies 21% and 26.7% relative increase, respec- tively. According to further analysis in $3.3.1 and $3.3.2, we attribute AutoRefine's 's extraordinary performance multi-hop benchmarks to its ability perform high-quality searching and efficient utilization of retrieved documents.
Obs 1: AutoRefine significantly improves QA accuracies, especially on multi-hop benchmarks.
3.3
Analytical Results
3.3.1 Search Behaviors (RQ2)
A crucial capability of retrieval-augmenned reasoning models is identifying and addressing knowledge gaps via retrieval. To evaluate this, analyze the search frequency and search quality of AutoRefine in four scenarios: training samples, all seven downstream benchmarks, single-hop QA benchmarks, and multi-hop QA benchmarks. These analyses are depicted in Figure 4.
Search Frequency reflects the model's capability to recognize knowledge gaps and perform searches accordingly. This search behavior measured by the average number of search calls model makes per rollout. We analyze the search frequency of both -Base and -Instruct variants of AutoRefine in Figure 4(a). As the figure shows, both variants have evolved multi-turn searching abilities after enough training steps. The average number of search calls converges to around 1.5 for CaadReffin-Instruce and higher than 2 for AutoRefine-Baan Base.
Despite the high overall search frequency on seven benchmarks, AutoRefine demonstrates distinct search behaviors for single-hop and multi-hop questions. On the three single-hop benchmarks, both

---

Frequency
training samples
all banchmarks
on single hop benchmarks
on 4 multi hop benchmaric
2.5
25-
2.5
2.5
7.2
22
23
7.2
3-aarch Calls
2.0
2.0
2.0
7.0
2.8
2.5
1.2
1.0
150
100
Training Steps
Takiing Staps
Traniing Staps
Training Steps
Search Quality
training samples
all benchmarks
3 single hop benchmarks
on 4 multi hop benchmaris
20
50
Success
45
20-
20
32
20
50
130
100
150
Training Steps
Traking Stups
Training Staps
Training Steps
AutuReTine Boiee
Searth R1 Base
ReSSoarc Bade
No Rert eval
Figure 4: (RQ2) Visualization of the search behaviors. (a) AutoRefine's 's average number of search calls per rollout. For both variants, AutoRefine learns adaptively issue more search queries for multi-hop questions and fewer for single-hop ones. (b) Comparison of search success rates between reasoning methods. While all methods draft more efficient search queries than naive retrieval, AutoRefine achieves more significant performance gains.
variants begin with fewer than 1.3 searches per rollout, and gradually adjust to 2.0 and 1.2. In contrast, the models begin with much higher search frequencies when facing multi-hop questions, which rapidly go up to 2.0 ~ 2.5. This phenomenon exhibits AutoRefine's 's ability to dynamically adjust the number of search calls according to the complexity of downstream tasks, with more frequent searching multi-hop questions and less single-hop ones.
Obs 2.1: AutoRefine learns to perform multi-turn searching and can adaptively issue search queries depending on task complexity.
Search Quality evaluates whether the model generates effective search queries that can return informative documents. Knowledge-intensive questions often demand precise factual information to ask, e.g., names of historical figures or dates of events. In such cases, search calls can only be considered successful if the retrieved documents directly contain the answer. Hence, we estimate the search quality by counting the proportion of successful searches where retrieved documents contain the ground truth answer a. We also include naive retrieval, which directly uses the input question to conduct one-turn searches, as a reference.
We compare the search quality of AutoRefine against Search-RI and ReSearch trained from Qwen2.5- 3B-Base in Figure 4(b). All reasoning-bor based methods learns draft efficient queries that have higher searching quality than naive retrieval. In single-hop scenarios, all three methods converge to a high success rate of about 70% after 100 steps. On multi-hop benchmarks, the search qualities of baseline methods also successfully converge to 30% ~ 40%, much higher than that of baseline methods. In contrast, the search quality of AutoRefine continuously goes up to higher than 50%, which surpasses baseline methods by large margin of 10% ~ 15%.
Obs 2.2: AutoRefine drafts efficient queries that retrieve documents relevant to the answer.
3.3.2 Effectiveness of Knowledge Refinement (RQ3)
To investigate the effectiveness of knowledge refinement, we specifically analyze whether the refine- ment steps successfully distill critical information from retrieved documents. We start by comparing the success rates of different actions: <search>, <refine>, and <answer>, Here use cover exact
7

---

Surcess Rate (%)
10 Mumber of Thikans
E00
610
200
13
0.0
Paining Steps
Componattt
Figure 5: (RQ3) Comparison of search, refine, and swer actions over (a) recall and (b) average token counts. Knowledge refinement keeps crucial information from retrieved documents while reducing context length.
Figure 6: (RQ4) Comparison of downstream accuracies under differ- ent retrieval depths. AutoRefine hibits robust gains for < k < 7.
match to measure the success rate for all three actions, which is defined as the proportion of actions that return secuuent//eiinemen//asswwwn containing the ground truth answer.
The results are shown Figure 5. After enough training steps, the success rate of <refine> actions tends to align with that of the <search> action (Figure 5(a)). This suggests the model gradually learns to keep crucial evidence as long as the search returns correct documents. Figure 5(b) also provides the length of each component. The token count of refinement steps is about 100 200 tokens, which is about 4 times fewer than the documents ( 600 tokens). Comparing Figure 5(a) and (b), we find that the knowledge refinement steps of AutoRefine greatly reduce the context length, while successfully preserving the information that is relevant to the answer.
Obs 3: Knowledge refinement steps efficiently distill critical evidence from retrieved documents while filtering out irrelevant content.
3.3.3 Impact of Retrieval Depths (RQ4)
Different retriever settings may also influence retrieval-augmenteed generation models, and one important aspect is the retrieval depth. While more documents per search could potentially provide richer external knowledge, it also includes more noise in documents. To explore the models' robustness across different retrieval depths, we vary the number of documents (k) returned by the retrieval engine at evaluation time, from to while training fixed to k =
The comparison of inference accuracy is shown in Figure All three methods achieve robust performance across different & levels. Compared to the baseline methods, AutoRefine steadily boosts the average accuracy by 0.04 ~ 0.1, demonstrating its strong document denoising ability. The accuracy increments caused by AutoRefine
particularly obvious when & > 3, which is likely caused by its strong ability to discover useful information under increasingly noisy conditions. Peak performance gain is 0.09 observed at 5, where a balanced trade-off between information richness and noise reached.
Obs 4: AutoRefine exhibits consistent improvements across varying retrieval depths.
3.4
Ablation Studies
3.4.1
Ablation on Key Components
We conduct ablation studies over the key components in AutoRefine. Specifically, we consider three configurations: (1) the full AutoRefine model, (2) AutoRefine without the retrieval-specific reward RRR (w/o Retrieval Reward), and (3) AutoRefine without both RRa and the knowledge refinement step (w/o Retrieval Reward & Refinement). See Appendix C.3 for analysis of retrieval reward design.
Impact on Answer Accuracy. Table presents the answer accuracy on downstream benchmarks for each configuration. The results demonstrate that both the retrieval-specifis reward and the knowledge refinement step are essential for achieving strong performance. The full AutoRefine model consistently achieves the highest average accuracy across both the base and instruct variants.

---

Table 2: Ablation study over key components in AutoRefine.
Single-Hop QA
Molti-Hop QA
Model Variants
TriviaQA
PopQA
HotpoiQA
2wiki
Musique
Bamboogle
Avg.
AutoRefine-Ban
0.467
0.620
0.450
0.157
w/o Retrieval Reward
0.423
0.583
0.424
0.368
0.351
0.139
0.344
0.376
w/o Retrieval Reward & Refinement
0.422
0.585
0.419
0.294
0.257
0.062
0.144
0.312
AutoRefine-Instune
0.436
0.597
0.447
0.404
0.380
0.169
0.336
0.396
w/o Retrieval Reward
0.418
0.587
0.429
0.355
0.335
0.124
0.272
0,360
w/o Retrieval Reward & Refinement
0,406
0.580
0.412
0.319
0.312
0.091
0.210
0,333
(a) Search Frequency
(b) Search Quality
(c) Refinement Quality
2.5
80
Search Calls
Success Rate (%)
Success Rate (%)
2.0
60
80
Mane W
1.5
20
20
150
100
100
150
Training Stops
Training Stops
Training Stops
Rotrioval
we Rotrioval Roward
Retrieval Roward Refinement
Figure 7: Effeetiveness of key components over the search behaviors and the refinement quality.
Impact on Search and Refinement Abilities. We further analyze how each component affects AutoRefine's s search and refinement capabilities. Figure shows a comparative analysis using Qwen2.5-3B-BBal Base.. As illustrated in Figure 7(a), the retrieval-specific reward effectively promotes multi-turn search behavior. Additionally, it significantly boosts knowledge refinement quality, yielding approximately a 20% improvement in refinement success rate (Figure 7(c)). The inclusion of the knowledge refinement step also enhances both the frequency and quality of retrieval, as shown in Figures ((a) and (b).
3.4.2 Ablation Model Sizes and Evaluation Metries
To provide a comprehensive evaluation of AutoRefine, we conduct ablation studies on two aspects: (1) the model sizes, including Qwen2.5-3B and Qwen2.5-7BB (2) the evaluation metrics, including exact match (EM), Fl score, and cover exact match (CEM). The results are shown in Table
Comparing the performance of AutoRefine on Qwen2.5-3B and Qwen2.5-7B, we observe that the larger model size generally leads better performance. AutoRefine-7B achieves approximately 0.05 performance gains on all metrics, which is slightly lower compared to those on the 3B variant. Besides, AutoRefine maintains superior performance on all three metrics compared to baselines.
3.4.3 Ablation on Knowledge Refinement Module
To demonstrate the necessity of the RL-driven refinement steps, we compare AutoRefine with several baselines that use external summarization models refiners. These baselines augment Search-R1 with refiners based on BART [38] and Qwen2.5-3B-Instree two prompting strategies:
[3]. For the Qwen model, we test that only asks for summarization, and another that asks for both summarization and plan for the next search step. The results are presented in Table 4.
The experiment indicates that simply adding an external summarizer to Search-R R1 improves perfor- mance on some single-hop QA benchmarks (e.g.. PopQA) but can be detrimental multi-hop settings. In contrast, AutoRefine maintains superior performance on hard multi-hop benchmarks. Through RL, AutoRefine learns not only to summarize but also to introspect, identify missing information, and plan its next actions We find out its performance gain on multi-hop benchmarks derives from the ability to not just summarize,
also to introspect, recognize missing information, and plan subsequent search steps, as illustrated in the case studies ($C.5).

---

Table 3: Ablation study over model sizes and evaluation metrics.
General QA
Molti-Hop QA
Model
Metric
NQ
TriviaQA
PopQA
HotporQA
2wiki
Musique
Bamboogle
Avg.
Qwen2.5-7B-Baa
EM
0.469
0.627
0.449
0.410
0.272
0.173
0,456
0.408
Search-RI
F1
0.552
0.700
0.487
0.517
0.327
0.236
0.560
0.483
CEM
0.509
0.680
0.467
0.445
0.309
0.197
0.496
0.443
EM
0.484
0.659
0.487
0.451
0.405
0.187
0.512
0.455
AutoRefine
0.574
0.729
0.525
0,573
0.467
0,283
0.604
0.536
CEM
0.523
0.707
0.500
0.487
0,441
0.217
0.528
0.486
Qwen2.5-3B-Baxx
FM
0.421
0.583
0.413
0.297
0.274
0.066
0.128
0.312
Search-RI
0.476
0.650
0.429
0,380
0.322
0.123
0.184
0.366
CEM
0.462
0.642
0.442
0.325
0.288
0.082
0.128
0.338
EM
0.467
0,620
0,450
0,405
0,393
0.157
0.344
0.405
AutoRefine
F1 0.534
0,689
0.479
0.503
0.453
0.233
0.449
0.477
CEM
0.502
0.674
0.468
0.440
0.428
0.175
0.384
0.439
Table 4: Performance comparison against Bearch-RI with external refiners.
General QA
Multi Hop QA
Model
NQ
TrivisQA
PopQA
Hotpo(QA
2wiki
Musique
Bamboogle
Avg.
AutoRefine
0.467
0.620
0.393
0.157
0.344
0.405
Searcb-R1
0.421
0.583
0.413
0.297
0.274
0.066
0.128
0.312
Search-RI Refiner (BART [38]
0.395
0.619
0.450
0.337
0.239
0.065
0.115
0.317
Scarcb-RI KI Refiner (Qwen, Summary)
0.399
0.600
0.445
0.331
0.264
0.073
0.180
0.328
Search-RI Refiner (Qwen, Summary Plan)
0.378
0.562
0.431
0.299
0.231
0,059
0,149
0.301
Limitations
Despite the promising performance of AutoRefine in retrieval-auggmented reasoning tasks, several limitations remain for further investigation.
Evaluation Metrics. This work evaluates model performance solely exact match accuracy or Fl score, which may overlook semantically correct responses with minor textual variations. This limits the evaluation of long-form open-ended responses.
Static Retrieval Corpus. The retrieval component uses fixed Wikipedia snapshot, lacking current time-sensitive information. This setting limits the system's applicability real-world use cases where users expect information from live search engines.
Conclusion and Future Work
This work proposes AutoRefine, an RL post-training framework designed improve the retrieval- augmented reasoning capabilities of LLMs. AutoRefine adopts novel seearch-and-refie -aurr---nnd-fffn--duuingg think" paradigm that explicitly encourages the model identify and distill relevant information from noisy retrieved content. By jointly optimizing for both outcome-level and retrieval-specifie rewards, AutoRefine effectively guides LLMs extract, assess, and integrate external knowledge. Comprehensive evaluations show that AutoRefine consistently surpasses existing methods, achieving up to a 6.9% average improved accuracy on seven QA benchmarks. These results underscore its potential to enhance the accuracy and reliability of retrieval -uggented LLMs.
Future work will focus on addressing the limitations identified above. First, we aim adopt more flexible and semantically aware evaluation metrics - such as LLM-as-a-Judge evaluation to more effectively measure answer quality in complex question answering tasks. Second, we intend to adapt AutoRefine to dynamic retrieval settings, including live web search and continuously evolving document corpora. By addressing these directions, we aim to further improve the sflexibility and time-sensitivity of AutoRefine, thus broadening its practicality in more realistic applications. This extension would enable the system to operate more realistic, time-sensitive applications and broaden its practical utility.
10

---

Acknowledgement
This research is supported by the National Natural Science Foundation of China (62572449). We also appreciate the support of Siyuan Wang' from Shanghai Jiao Tong University.
11

---

References
[1] OpenAl. Gpt-4 technical report. arXiv preprint arXiv:2303.0777, 2023.
[2] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977. 2023.
[3] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint mmXiv:2412.15111 2024.
[4] Leslie Pack Kaelbling, Michael L Littman, and Andrew W Moore. Reinforcement learning: A survey, Journal of artificial intelligence research, 4:237-285, 1996.
[5] DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji. Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding. Huajian Xin, Huazuo Gao, Hui Qu, Hui Li. Jianzhong Guo. Jiashi Li, Jiawei Wang, Jingehang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, L. Cai, Jiaqi Ni, Jian Liang Jin Chen, Kai Dong, Kai Hu, Kaige Gao. Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang. Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang. Shuiping Yu, Shunfeng Zhou, Shuting Pan, and S. S. Li. Deepseek-rl: Incentivizing reasoning capability in lims via reinforcement learning. CoRR, abs/2501. 12948, 2025. doi: 10.48555/ARXIV.250011 12948.
[6] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El- El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Careey, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, Ananya Kumar, Andre Saraiva, Andrea Vallone, Andrew Duberstein, Andrew Kondrich, Andrey Mishchenko, Andy Applehaum, Angela Jiang, Ashvin Nair, Barret Zoph, Behrooz Ghorbani, Ben Rossen, Benjamin Sokolowsky, Boaz Barak, Bob McGrew, Borys Minaiev, Botao Hao, Bowen Baker, Brandon Houghton, Brandon McKinzie, Brydon Eastman, Camillo Lugaresi, Cary Bassin, Cary Hudson, Chak Ming Li, Charles de Bourcy, Chelsea Voss, Chen Shen, Chong Zhang, Chris Koch, Chris Orsinger, Christopher Hesse, Claudia Fischer, Clive Chan, Dan Roberts, Daniel Kappler, Daniel Levy, Daniel Selsam, David Dohan, David Farhi, David Mely, David Robinson, Dimitris Tsipras, Doug Li, Dragos Oprica. Eben Freeman, Eddie Zhang, Edmund Wong, Elizabeth Prochl, Enoch Cheung, Eric Mitchell, Eric Wallace, Erik Ritter, Evan Mays, Fan Wang, Felipe Petroski Such, Filippo Raso, Florencia Leoni, Francis Song, Fred von Lohmann, Freddie Sulit, Geoff Salmon, Giambattista Parascandolo, Gildas Chabot, Grace Zhao, Greg Brockman, Guillaume Leclerc, Hadi Salman, Haiming Bao, Hao Sheng, Hart Andrin, Hessam Bagherinezhad, Hongyu Ren, Hunter Lightman, Hyung Won Chung, Ian Kivlichan, Ian O' Ian Osband, Ignasi Clavera Gilaberte, and Ilge Akkaya. Openai ol system card. CoRR, abs/2412.167220 2024. doi: 10.4550ARXX.22122.6722
[7] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint 242.:2402..33000 2024.
[8] Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi. Yu Wu, YK Li, et al. Deepscek-coder. When the large language model meets programming-the rise of code intelligence. arXiv preprint arXiv:2401. 2024.
[9] Zhengbao Jiang, Frank F. Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu. Yu, Yiming Yang, Jamie Callan, and Graham Neubig. Active retrieval augmented generation. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference Empirical
12

---

Methods in Natural Language Processing, EMNLP 2023, Singapore. December 6-10, 2023, pages 7969-7992. Association for Computational Linguisties, 2023. doi: 10.18653/V 1/2023.
EMNLP-MAIN.499
[10] Zhihong Shao, Yeyun Gong. Yelong Shen, Minlic Huang, Nan Duan, and Weizhu Chen. Enhancing retrieval-uugmented large language models with iterative retrieval-eeneration synergy. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pages 9248-9274 Association for Computational Linguistics, 2023. doi: 10. 12..665/N12223.FIIIIGSSSS 620.
[11] Akari Asai, Sewon Min, Zexuan Zhong, and Danqi Chen. Retrieval- language models and applications. In Yun-Nung Vivian Chen, Margot Mieskes, and Siva Reddy, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics: Tutorial Abstructs, ACL 2023, Toronto, Canada, July 9-14. 2023, pages 41-46. Association for Computational ACLTTOAATTL
Linguistics, 2023. doi:
[12] Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Dai, Jiawei Sun, Qianyu Guo, Meng Wang, and Haofen Wang. Retrieval-augmented generation for large language models: A survey. CoRR, abs/2312.10997, 2023. doi: 10.4550/ARXXV.3112.00999
[13] Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. In-context retrieval-augmenned language models. Trans. Assoc. Comput. Linguistics, 1 11:1316-1331, 2023. doi: 10. 10.1162/TACIL CACTTTACCLAAALO0OO000
[14] Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil. and Hannanch Hajishirzi. Self-rag: Learn- ing to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. Open- Review.net, 2024.
[15] Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Richard James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. REPLUG: retrieval-augmented black-box language models. In Kevin Duh, Helena Gómez-Adorno, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), NAACL 2024, Mexico City, Mexico, June 16-21, 2024, pages -371-8384. Association for Computational Linguistics, 2024. doi: 10.18653/V
[16] Shi-Qi Yan, Jia-Chen Gu, Yun Zhu, and Zhen-Hua Ling. Corrective retrieval augmented generation. arXiv preprint arXiv:2401.1588. 2024.
[17] Tianzhe Chu, Yuexiang Zhai, Jihan Yang. Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V. Le. Sergey Levine, and Yi Ma. SFT memorizes, RL generalizes: A comparative study of foundation model post-traing post-training. CoRR, abs/2501. 2025. doi: 10,48550/ARXIV.2500. 17161.
[18] Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. Search-ol: Agentic search-enhanced large reasoning models. arXiv preprint arXiv:2501.05366 2025.
[19] Bowen Jin, Hansi Zeng, Zhenrui Yue, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training Ilms reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516, 2025.
[20] Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang. and Ji-Rong Wen. R1-searcher: Incentivizing the search capability in llms via reinforcement learning. arxiv preprint arXiv22503.05500 055992, 2025.
[21] Mingyang Chen, Tianpeng Li, Haoze Sun, Yijie Zhou, Chenzheng Zhu, Fan Yang, Zenan Zhou, Weipeng Chen, Haofen Wang, Jeff Z. Pan, et al. Learning to reason with search for llms via reinforcement learning, arXiv preprint 2t..iv22503.19777 2025.
13

---

[22] Joykirat Singh, Raghav Magazine, Yash Pandya, and Akshay Nambi. Agentic reasoning and tool integration for llms via reinforcement learning. arXiv preprint arXiv:2505.0144.) 2025,
[23] Hao Sun, Zile Qiao, Jiayan Guo, Xuanbo Fan, Yingyan Hou, Yong Jiang, Pengjun Xie, Fei Huang, and Yan Zhang. Zerosearch: Incentivize the search capability of ilms without searching, 2025.
[24] Jixiao Zhang and Chunsheng Zuo. Grpo-lead: A difficulty-aware reinforcement learn- ing approach for concise mathematical reasoning in language models. arXiv preprint arXiv:2504.0900 2025.
[25] Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-Türr Gokhan Tur, and Heng Ji. Toolrl: Reward is all tool learning needs. arXiv preprint arXiv:2504.13958, 2025.
[26] Youssef Mrouch. Reinforcement learning with verifiable rewards: Grpo's effective loss, dynam- ics, and success amplification. arXiv preprini (5t::25003006339, 2025.
[27] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466, 2019.
(28) Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Ha- jishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 6l Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pages 9802-980 022299222222 2023.
[29] Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1601-1611 2017.
[30] Zhilin Yang. Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pages 2369-2380, 2018.
[31] Xanh Ho, Iho- Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6609-6625, 2020.
[32] Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539-554 2533,-55,220 2022.
[33] Olir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5687-5711, 2023.
[34] Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Interleaving retrieval with chain-of-thought reasoning for Enowledge-intensts multi-step questions. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 10014-10037. Association for Computational Linguistics, 2023. doi: 10.18653/ 112.:///111222.ALLLL......
[35] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick SH Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In EMNLP (I), pages 6769-6781, 2020.
[36] Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint 2rXhn:2212.03533, 2022.
14

---

[37] Patrick Lewis, Ethan Perez, Aleksandra Piktus. Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. netrieval-augmennt generation for snowledge-intensiie nlp tasks. Advances in neural information processing systems, 33:9459-9474 2020.
[38] Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequene pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.134611 2019.
[39] Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wangxiang Che. Towards reasoning era: A survey of long chain-of-thought for reasoning large language models. arXiv preprint arXiv:2503.09567. 2025.
[40] Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wangxiang Che. Towards reasoning era: A survey of long chain-of-thought for reasoning large language models. arXiv preprint arXiv:2503.0956/ 2025.
[41] Qwen Team. Dwq-32b: Embracing the power of reinforcement learning, March 2025. URL https://qwenlln gi thub. howpoo////q/-20000
[42] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of- tf-thuhht prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24831 2022.
[43] Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mets*: Llm self-training via process reward guided tree search. Advances in Neural Information Processing Systems, 37:64735-64772 2024.
[44] Yuxiang Zhang, Shangxi Wu, Yuqi Yang, Jiangming Shu, Jinlin Xiao, Chao Kong, and Jitao Sang. ol-derrr: an ol replication for coding. arXiv preprint arXiv:2412.00154, 2024.
[45] Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. rstar-math: Small Ilms can master math reasoning with self-evolved deep thinking. arXiv preprint arXiv:2501.04519, 2025.
[46] Zitian Gao, Boye Niu, Xuzheng He, Haotian Xu, Hongzhang Liu, Aiwei Liu, Xuming Hu, and Lijie Wen. Interpretable contrastive monte carlo tree search reasoning. arXiv preprint arXiv:2410.01707, 2024.
[47] Zhenting Qi, Mingyuan Ma, Jiahang Xu, Li Lyna Zhang, Fan Yang, and Mao Yang. Mutual reasoning makes smaller ilms stronger problem-solvers arXiv preprint arXiv:2408.06/93 2024.
[48] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information prucessing systems, 35:27730-27744, 2022.
[49] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint
2017.
[50] Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models. arXiv preprini arXiv:2501.03262. 2025.
[51] Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback 1lms. arXiv preprint arXiv:2402. 14740, 2024.
[52] Lisheng Huang, Yichen Liu, Jinhao Jiang, Rongxiang Zhang, Jiahao Yan, Junyi Li, and Wayne Xin Zhao. Manusearch: Democratizing deep search in large language models with a transparent and open multi-agent framework. arXiv preprint arXiv:2505.18100 2025.
15

---

[53] Zilong Wang, Zifeng Wang. Long Le, Steven Zheng, Swaroop Mishra, Vincent Perot, Yuwei Zhang, Anush Mattapalli, Ankur Taly, Jingbo Shang, Chen- Yu Lee, and Tomas Pfister. Spee- ulative RAG: Enhancing retrieval augmented generation through drafting. In The Thirleenth International Conference on Learning Representations, 2025.
[54] Shicheng Xu, Liang Pang, Mo Yu, Fandong Meng, Huawei Shen, Xueqi Cheng, and Jie Zhou. Unsupervised information refinement training of large language models for -aumenel- generation. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 133-145. Association for Computational Linguistics, 2024. doi: 10.
[55] Xinze Li, Sen Mei, Zhenghao Liu, Yukun Yan, Shuo Wang, Shi Yu, Zheni Zeng. Hao Chen, Ge Yu, Zhiyuan Liu, Maosong Sun, and Chenyan Xiong. RAG-DDR: Optimizing retrieval- augmented generation using differentiable data rewards. In The Thirteenth International Conference on Learning Representations, 2025.
[56] Yiqun Chen, Lingyong Yan, Weiwei Sun, Xinyu Ma, Yi Zhang, Shuaiqiang Wang, Dawei Yin, Yiming Yang, and Jiaxin Mao. Improving retrieval-augmente generation through multi-agent reinforcement learning. arXiv preprint arXiv:2501. 15228, 2025.
[57] Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. Deepresearcher: Scaling deep research via reinforcement learning in real-world environments. arXiv preprint arXiv:2504. 0it:22504033000 /600 2025.
[58] Pengcheng Jiang. Deepretrieval: Powerful query generation for information retrieval with reinforcement learning. arXiv preprint arXiv:2503.00223: 2025.
[59] Zhengren Wang, Jiayang Yu, Dongsheng Ma, Zhe Chen, Yu Wang, Zhiyu Li, Feiyu Xiong, Yanfeng Wang, Linpeng Tang, Wentao Zhang, et al. Rare: Retrievallaugmenned reasoning modeling. arXiv preprint arXiv:2503.235113 2025.
[60] Jerry Huang, Siddarth Madala, Risham Sidhu, Cheng Niu, Julia Hockenmaier. and Tong Zhang. Rag-rl: Advancing retrieval-ane -uggeeetd generation via rl and curriculum learning. arXiv preprint 127., 2025.
[61] Junde Wu, Jiayuan Zhu, and Yuyuan Liu. Agentic reasoning: Reasoning llms with tools for the deep research. arXiv preprint arXiv:2502.04664 2025.
[62] Yixuan Weng, Minjun Zhu, Guangsheng Bao, Hongbo Zhang, Jindong Wang, Yue Zhang, and Linyi Yang. Cycleresearcher: Improving automated research via automated review. arXiv preprint
16, 2024.
[63] Zhuoqun Li, Haiyang Yu, Xuanang Chen, Hongyu Lin, Yaojie Lu, Fei Huang, Xianpei Han, Yongbin Li, and Le Sun. Deepsolution: Boosting complex engineering solution design via tree-based exploration and bi-point thinking. arXiv preprint arXiv:2502.20730, 2025.
[64] Ruibin Xiong, Yimeng Chen, Dmitrii Khizbullin, and Jürgen Schmidhuber. Beyond outlining: Heterogeneous recursive planning for adaptive form-form writing with language models. arXiv preprint arXiv:2503. 08275, 2025.
[65] Mingxuan Du, Benfeng Xu, Chiwei Zhu, Xiaorui Wang, and Zhendong Mao. Deepresearch bench: A comprehensive benchmark for deep research agents. arXiv preprint arXiv:2506.11763, 2025.
[66] Ziyang Huang, Xiaowei Yuan, Yiming Ju, Jun Zhao, and Kang Liu. Reinforced internal-external knowledge synergistic reasoning for efficient adaptive search agent, 2025.
[67] Pengcheng Jiang, Xueqiang Xu, Jiacheng Lin, Jinfeng Xiao, Zifeng Wang, Jimeng Sun, and Jiawei Han. s3: You don't need that much data to train search agent via rl. arXiv preprint
arXiv:2505.14146 2025.
16

---

[68] Huatong Song, Jinhao Jiang, Wenqing Tian, Zhipeng Chen, Yuhuan Wu, Jiahao Zhao, Yingqian Min, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. RI-searcher+4: Incentivizing the dynamic knowledge acquisition of Ilms via reinforcement learning. arXiv preprint arXiv:2.005. 1705, 2025.
[69] Zeyang Sha. Shiwen Cui, and Weiqiang Wang. Sem: Reinforcement learning for search-efficient large language models. arXiv preprint arXiv:5055.07003, 2025.
[70] Guibin Zhang, Hejia Geng, Xiaohang Yu, Zhenfei Yin, Zaibin Zhang, Zelin Tan, Heng Zhou, Zhongzhi Li, Xiangyuan Xue, Yijiang Li, Yifan Zhou, Yang Chen, Chen Zhang, Yutao Fan, Zihu Wang, Songtao Huang, Yue Liao, Hongru Wang, Mengyue Yang, Heng Ji, Michael Littman, Jun Wang. Shuicheng Yan, Philip Torr, and Lei Bai. The landscape of agentic reinforcement learning for Ilms: A survey. arXiv preprint 250.::559,.2547,, 2025.
[71] Guangming Sheng. Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang. Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409. 19256, 2024.
[72] Bowen Jin, Jinsung Yoon, Priyanka Kargupta, Sercan Arik, and Jiawei Han. An empirical study on reinforcement learning for reasoning-search interleaved Ul agents. arXiv preprint 11Xiv:2505. 2025.
17

---

A
Related Work
This section reviews prior research on reasoning in LLMs and areas central our approach. We highlight how recent advancements in RL-based post-training have enabled more adaptive retrieval-augmented reasoning, motivating our proposed "scarch-and-refing during-think" paradigm.
generation, two
Reasoning in Large Language Models. The reasoning capabilities of large language models (LLMs) have advanced significantly in recent years [39, 40, 6, 41J. Early work introduces explicit chain-of-thought prompting [42] and test-time scaling methods such monte carlo tree search [43-47] to guide (RL), particularly reinforcement learning from human feedback (RLHF) [48], to align outputs with human preferences via Proximal Policy Optimization (PPO) [49]. Due to RLHF's high resource demands, more recent developments [50, 51] such as Group Relative Policy Optimization (GRPO) 5] optimize models using outcome-haa haseed rewards to reduce dependency on human annotations. These breakthroughs greatly improve L.LMs* generalization and performance on complex tasks such as mathematical problem-solving -olviin [7] and code generation [8].
guide intermediate reasoning steps. Follow-up methods leveraged reinforcement learning
Retrieval Augmented Generation. ties of LLMs by integrating external knowledge [11-13]. critical challenge within RAG systems is determining when and how to perform retrieval actions [10, 34, 52]. Prior works have leveraged supervised fine-tuning (SFT) methods train LLMs in generating appropriate retrieval queries [14, 53, 16, 15, 9, 54]. However, these SFT-based approaches struggle to generalize in out-of- distribution retrieval scenarios [55, 56]. Recently, RL-based methods have enabled adaptive retrieval and context-aware query generation [57-60], which can be termed retrieval-augmenneed reasoning that facilitates deep research applications [61-65]. Current methods follow the "search-during-the paradigm, where the model learns multi-turn searching and reasoning with outcome-based reward [19, 21, 20, 22, 23, 66-70]. However, the explicit refinement of retrieved documents and direct rewards for retrieval quality are absent in this paradigm, hampering effective searching and document utilization. This work explores the "search-and" refine-during-thine paradigm, which enables LLMs to refine retrieved documents, guided by both outcome-base bbsed and strieval-speciff rewards.
Retrieval-Augmented Generation (RAG) extends the capabili-
B More Implementation Details
B.1
Training Details
AutoRefine is trained on 8 NVIDIA A100-80GB GPUs with full-parameter fine-tuning. We construct the training dataset by combining NQ [27] and HotpotQA [30], used consistently across AutoRefine and all training- baseline methods. For distributed training, we adopt Fully Sharded Data Parallelism (FSDP), using 1 precision throughout both training and evaluation.
Table 5 summarizes the key hyperparameters used in our experiments. The actor model is optimized using learning rate of 1.0 10 without warmup. Both the base and instruct variants of AutoRefine are trained for 200 steps using the VeRL framework [71], with random data shuffling.
For efficient rollout generation, we vLLMM GPU memory utilization rate of 0.6. Sampling is performed with a temperature of 1.0, and a maximum of search calls per rollout is allowed. We generate rollouts per data point, each with up to 5 search queries. Retrieved documents per query are concatenated and truncated to 512 tokens. Token length statistics shown in Figure 5(b) are computed using tiktoken".
For direct-inference and SFT baselines, we use Qwen2.5-3 3--25-3B-Insttuc [3] as the backbone LLM. RL-based experiments are conducted on both the base and instruct variants.
Phttps: :/gi thub. cow//vllm-prooeec/uuu Phttps:// //gi thub. com/oppena/tiktooen
18

---

Table 5: Primary hyperparameters used by AutoRefine.
Hyper-parameter
Value
Training Batch Size
256
Micro Training Batch Size
64
Validation Batch Size
256
Total Training Steps
250
Actor Model Learning Rate
x 10-6 6
Max Response Length
2048
Max Search Actions
5
KL Coefficient
0.001
Clip Ratio
0.2
Group Size G
5
B.2 Dataset Statisties
All datasets are sourced from the FlashRAG Datasets collection". Table presents detailed statisties of the datasets used.
The training set for AutoRefine is constructed from the train splits of NQ and HotpotQA, totaling 169, 615 examples. For evaluation, we combine the test or dev splits from seven datasets. Specifically, for benchmarks with test split (NQ, TriviaQA, PopQA, and Bamboogle), the test split is used; for those without a test split (HotpotQA, 2Wiki, and Musique), we use the dev split instead. This results in evaluation comprising 51, 713 examples.
Table Statistics the seven datasets used in this paper.
NQ
TriviaQA
PopQA
HotpotQA
2Wiki
Musique
Bamboogle
Train
79168
78785
90447
15,000
19,938
Dev
8757
8837
7405
12576
2417
Test
3610
11313
14267
125
C More Experimenal Results
C.1 Training Dynamics
For more comprehensive understanding of AutoRefine, we visualize its training dynamics, including training rewards, validation accuracies, and response length per sample. The validation carried out on 500 random samples from each downstream benchmark per 20 training steps. The results are reported Figure We observe stable convergence in the training rewards and consistently improved validation accuracy in both base and instruct variants.
Valcation Acourait
0.5
1500
Training Revare
0.4
54.
Response
2:00
100
22
50
120 Training Steps
79
123
150
:75
23
15
302
:25
250
175
20
75
200
150
175
Training Staps
Training Steps
AuttReeiie Sane
Autokefine rotruct
Figure Training dynamics of AutoRefine- and -Instruct. Both models show steady convergence and stable downstream accuracies.
19

---

Table 7: Statistical analysis against search bearch-duuing-thiee think baselines. The p-value column represents the T-test result of AutoRefine v.s. baseline.
Ng
TriviaA
PopQA
HoqpogA
Misique
Baathoogle
Avg.
pasaluc
AmtoRefine
0,622 0.017
0.627 0.007
0,668 0:017
0.423
0.016
0.404 0.010
0.165 0.011
0.335 0.023
0.408 0.14
ReSearch
0 = 0.012
0.014
0.004
0
0.018
0.317
0.015
0.299
0.017
0.056 0.015
0.132 0005
11322 # 00014
5.49 x 10 2,85 x 10
Soorch R1
0,410 a 0:009
0,005 0.019
0,429 a 0.014
0.315 0.016
0.254 à 0.023
0.062 0.008
0.127 à 00020
0.315 00015
Table 8: Comparison between different reward designs used in AutoRefine.
Gemeral QA
Hopi QA
Reward Design
NQ
TrivisQA
PopQA
HotporQQA
2wiki
Musique
Bamboogle
Avg.
AutoRefine- Reward on Refine- nonlincar
0.467
0.620
0.450
0.405
0.393
0.157
0,344
0.405
AutoReline- Reward on Reline linear
0.593
0.365
0.296
0.375
AutoRefine- Reward on Documeats- ponlincar
0.418
0.592
0.441
0.381
0.386
0.153
0.320
0.384
AutoRefine- Reward on Documents- lincar
0.417
0.590
0.414
0.387
0.360
0.152
0.304
0.375
AutoRefine- only answer reward
0.423
0.583
0.424
0.368
0.351
0.139
0.344
0.371
Search-R1
0.421
0.583
0.413
0.297
0.274
0.066
0.128
0.312
C.2 Statistical Analysis
To ensure the reliability of our findings and validate the significance of the performance gains, we conduct a statistical analysis. We perform three experimental runs using different random seeds and report the mean scores and standard deviations for AutoRefine and the search-during- think baselines in Table 7. To formally assess the improvements, we perform a T-test between each baseline and AutoRefine. The resulting low p-values (p < 0.01) indicate that the improvements achieved by AutoRefine over both ReSearch and Search-RI are statistically significant.
C.3 Impact of Different Retrieval Reward Design
In $2.2, it's worth noticing that we a non-linear combination of RAns and Ree calculate the overall reward Roverali. and we apply the retrieval reward on the refinement action instead of directly on retrieved documents. We conduct additional empirical study to analyze the impact of our retrieval reward design, including (1) the action types on which we compute the retrieval reward, and (2) the combination method of Rit and AAss-
From the results in Table 8, we notice (1) directly rewarding the retrieved documents contributes marginal performance improvements (reward on retrieved documents only answer reward), which is also noticed by previous researchers [72]. The peak performance is achieved when we calculate the retrieval reward based the refinement behaviors. (2) linear combination of answer and refinement rewards (Roverali = Ans + Ret is inferior to our proposed non-linear reward design. We hypothesize linear rewards may over-emphasize intermediate behaviors. In the contrary, non-linear ones prioritizes the final answer correctness while still fostering robust refinement capabilities. The intricate balance in the reward function a core innovation of AutoRefine, directly contributing its superior performance across various QA benchmarks.
C.4 Performance on Complex Answers
To investigate the impact of our retrieval reward design on questions with more complex answers, we conduct an experiment comparing our standard cover-exact match (CEM) reward with more fine-grained recall-based rewards. We explore two alternative designs, namely token-level recall and word-level recall the retrieval reward, which calculates the fraction of tokens/words in the ground-truth answer that appear in the refined documents. We evaluate these reward strategies on both the full benchmark datasets and on the subset of "complex answers," defined as samples where the truth answer is longer than five words.
The results, presented in Table 9, show that while our default CEM retrieval reward performs strongly on the full datasets, its performance diminishes on the subset complex answers. this more challenging setting, the more fine-grained reward metrics yield significant performance improvements. This suggests that while CEM is effective for factoid-style questions, adapting the reward signal to be more granular can better guide the model to handle complex answers.
20

---

Table 9: Comparison between the original CEM retrieval reward and finer- grained reward designs.
General
QA
Multi Hop QA
Reward Design
TriviaQA
PopQA
HotporQA
2wiki
Musique
Avg.
Full Dataset
AutoRefine CEM Retrieval Reward
0.620
0.450
0.405
0.393
0.157
0.405
AutoRefine Token-level Recall Reward
0.604
0,433
0.376
0.364
0.136
0.383
AutoRefine Word-level Recall Reward
0.609
0.437
0.395
0.395
0.142
0,396
Complex Answers (>5 words)
AutoRefine CEM Retrieval Reward
0.128
0.261
0.105
0.368
0.023
0.177
AutoRefine Token-level Recall Reward
0.132
0.292
0.094
0.379
0.047
0.189
AutoRefine Word-level Recall Reward
0.131
0.375
0.113
0.409
0.054
0.216
C.5
Case Studies
Table 10 presents the answer generated by AutoRefine and Search-R1 two randomly selected samples from 2Wiki and bamboogle. The comparison made between AutoRefine, SearchR1 [19], and ReSearch [21].
D Broader Impacts
This work on AutoRefine has several potential broader impacts. By significantly enhancing the accuracy and reliability of reureeeal-uugmented language models, this research contributes to building more trustworthy and dependable Al systems. This poses positive implications for fields such as education, research, and information services, where access to accurate and synthesized knowledge is crucial. Furthermore, the explicitly encourages the model to engage in more deliberate process information extraction and evaluation. This design may inspire future research into more interpretable and controllable reasoning frameworks. Understanding how LLMs can be guided to selectively utilize external knowledge is step towards demystifying their decision makinn processes.
paradigm introduced by AutoRefine
21

---

Table 10: Case Studies. The predictions colored Red if the model answers incorrectly and Green it answers correctly. The core evidences are colored Blue.
Case 1:
Who is the father of the father of modern experimental psychology?
Ground Truth: Maximilian Wundt
ReSearch;
<think To answer the question about who the father of father modern experimental psychology, need to trace back through history and identify the person who founded modern experimental psychology, and then identify the father of this individual. </think>
<search> Who is the father of the father of modern experimental psychology </search> <documents> [Doc I: Wilhelm Wundi] Wilhelm Wundt Wilhelm Maximilian Wundt the first person ever to call himself psychoologis. He is widely regarded the father of experimental esychology..<<ddocuunnts <answer> Wilhelm Wundt </answer2
Search-R1F
<think I have entrained myself with prepared oniosis in syne with tractable graftally towards emotionally lag useards glykerized ocal gambling huffely bite-support. Further exposure hides sectional stemming non non-quently. </think>
<search> Who is the father of the father of modern experimental psychology s/search><documents
1: Wilhelm Wundi] Wilhelm Wundt Wilhelm Maximilian Wundt the first person ever to call himself psychologist. He is widely regarded as the father of experimental pwyphooooggppyyhooogggg.ooo seychology...:/dccueentt <answer> Wilhelm Wundt </answer>.
AutoReffne:
<think> To answer the question "who is the father of the father of modern experimental psychology?", I would first research 'father of modern experimental psychology'. </tthnnk> <search> father of modern experimental psychology </ "'saarch>cdocmmens>>/D Wilhelm Wundt Wilhelm Maximilian Wundt is widely regarded as father of experimental wycchlogy.../doccmmee
Wilhelm Wundt]
the first person ever call himself psychologist. He
<refine> From the available documents, extract key information. Wilhelm Wundt is regarded as the father of experimental psychology </refine>
<think Now, I will research who Wundt's 's father was by understanding psychology back then.</thinds <search> Who was Wilhelm Wundt father </searchh [Doc 2: Wilhelm Wundt] Wundt was born Neckarau, Baden (now part of Mannheim) the 16 of August 1832, the fourth child parents Maximilian Wundt </documents>
<refine> From the available dncuments, extract key information. Wilhelm Wundt's father was Maximilian W'undt.</rffinn
<answer> Maximilian Wundt </answer>.
Case 2: What the date of death of Alexander Of Masovia's 's father? Ground Truth: January 1426 ReSearch::
<think To answer this question, need the date of death of Alexander Of Masovia's father. </think> <search> Alexander Of Masovia's father date of death </search><documee [Doe 1: Alexander of Masovia] Alexander of Masovia Alexander of Masovia (1400 June 1444) ... was the second son of Siemowit IV, Duke of Maasovia..//doccmentt.
<answer> 1400 </answer>,
Search-R1:
<think>l's ' overknowledgeahle in whatever guide possess it but must forth motion where direction need to going quests.</think>
<search> Alexander Of Masovia's father date of death </search><documents> [Doc 1: Alexander of Masovia] Alexander of Masovia Alexander of Masovia (1400 June 1444) ... was the second son of Siemowit IV, Duke Masovia maasovvi...</docuume ...s//ocument>>>
<answer> 1400 </answer>.
AutoReffine
<think> need to first determine the date of death of Alexander of Masovia's 's father. </think> <search> date of death of Alexander Of Masovia's 's father </search><documents [Doc 1: Alexander of Masovia] Alexander of Masovia Alexander of Masovia (1400- June 1444) was the second son of Siemowit IV, Duke of Masovia ...//oocumnts>
<refine> From the provided documents, deduce that Alexander Masowia's father was Siemowit IV, Duke Masovia.</refinee
<thinke To identify the date of death of Siemowit IV, Duke of Masowia who was the father of Alexander of Masowia. </think>
<search> date of death Siemowit IV Duke of Masovia </search>>doocments> [Due I: Siemowit IV, Duke of Masovia] ... Siemowit IV, Duke of Masovia Siemowit IV (Ziemowit IV), also known as Siemowit IV the Younger (pl: 21 January 1426) ..<//ocuments>
<refine> From the provided documents, deduce that Siemowit IV, Duke Masovia died on January 1426. </refine>
<answer> 21 January 1426 </answer>.
22

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer:
[Yes]
Justification:
We have already included our contributions and scope in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are included in Section 4.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
23

---

Justification:
There's no theoretical focus in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The information needed to reproduce the main results in Section 3.1 and Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
24

---

Answer: [Yes]
Justification: The code of this paper is included. The data used are publicly available at datasets.
https://huggingfacee wt//aattsettRRU--MMP/////Faaa
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.c cc/ Bellii//uidees//deeeiiieeee
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (ht / //niss. ctppbblcclllccppcpccppcpcccpcccc c/ /ewiidee/ooeebbbbbe ssiinsoiooo
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The trainin/test details are included in Section 3.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We have included the statistical significance in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
25

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The information on the computer resources are included in Section 3.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: /tturr///////p/e//e/eeeee httppppccccpppcccccccccceeeeeee.
Answer: [Yes]
Justification:
The research in this paper conform with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have included section for broader impacts in Appendix D. Other related discussions are $1 and $5.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
26

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The datasets are used via httts:://uugingg http:///ggggggaae...oooaaaatttta ww..aaasetts/Rc--N..L..... FlashRAG da tasets under CC-BY-SA-4.0 license. We the Qwen-2.5 series models that are under the Apache license 2.0.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi thcode. com/ com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
27

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
28

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components
Guidelines:
The answer NA means that the core method development in this research does not
involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (ht htttt::///eurippe htttt:///eerrpsscc//offeeeeeeeee 22t for what should or should not be described.
29


Reviews:
[{"review_id": "hHOkhERHzH", "reviewer": "Reviewer_jRtt", "summary": "The paper introduces AutoRefine, an RL post-training framework that RAG by adding an explicit “search-and-refine-during-think” reasoning loop. In each reasoning cycle the model (i) plans (\\<think\\>), (ii) issues a search query (\\<search\\>), (iii) receives documents, (iv) refines those documents to distill the most relevant facts (\\<refine\\>), and (v) eventually produces an answer (\\<answer\\>). In this reasoning process, the authors introduced two rewards: 1) the outcome reward (whether the final answer is correct) and 2) the retrieval reward, which is measured based on the quality of refined documents within \\<refine\\>\\</refine\\> blocks. The model is trained end2end using GRPO. To demonstrate the effectiveness of the proposed method, the authors conducted experiments across a set of sing-hop and multi-hop datasets. The strong empirical results show that AutoRefine is effective in identifying and addressing knowledge gaps via multi-turn, high-quality search queries.", "questions": "n/a", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n- The proposed \"search-and-refine-during-think\" approach is technically sound and well-presented.\n- Strong empirical results over a set of single-hop and multi-hop datasets. It not only outperforms the baselines but also the strong results in the literature.\n- The experiments are comprehensive, including thorough ablations showing the effectiveness of the framework design.\n\n### Weaknesses\n\nThe major concern I have regarding this paper is its similarity to the Search-R1 paper. The core architecture, prompt template, RL framework, datasets, and evaluation metrics of AutoRefine mirror those of Search-R1. The new elements introduced in AutoRefine are (i) inserting a <refine> block in which the model summarizes the retrieved passages and (ii) adding a “retrieval-specific” reward that checks whether the gold answer appears inside that block. Search-R1 could adopt these two tweaks with minimal changes. Although I admit that making those two changes brings significant gains over multi-hop reasoning datasets, I am not fully convinced that it is worth a NeurIPS publication. From my perspective, exploring more diverse experimental settings could make the paper much stronger. For example, if the ground-truth is given as a long and complex answer, how to compute the retrieval reward, and how does that affect the final performance?", "was_revised": true, "final_justification": "The added experiments resolved some of my major concerns. I think adding these results could significantly help the audience understand the effectiveness of the framework design. Therefore, I am willing to increase my score to 5.", "rebuttal": "We sincerely appreciate the reviewer's acknowledgement of the soundness, strong performance and comprehensive analysis. Below are point-to-point responses addressing your major concerns.\n\n> **Weakness 1: Similarity to the Search-R1 paper.** Search-R1 could adopt these two tweaks with minimal changes. I’m not fully convinced that it is worth a NeurIPS publication.\n\n**Response:**\nThank you for your constructive feedback. We understand your concern regarding the similarity to Search-R1 and the novelty of our contribution.\n\nWhile AutoRefine builds upon the foundational search-during-reasoning paradigm established by Search-R1, we respectfully disagree that our method constitutes \"two tweaks with minimal changes.\" Here we would like to clarify the **key challenges** we tackled and the **key advances** we achieved in our paper:\n\n**[Key Challenges]** To demonstrate the key challenges in developing knowledge refinement steps in search-during-think methods, we conduct **additional experiments** comparing different supervision reward signals, as shown in Table 1. We empirically show that:\n- **Simple paradigm modification (e.g., \"search-and-refine-during-think\") is sub-optimal.** As observed in Table 1, merely adding a refinement step does improve the overall performance. Nevertheless, we uncover that the model's refinement ability can be further enhanced with proper retrieval-related rewards.\n- **Direct retrieval reward on retrieved documents showcases limited improvement.** One simple way to add a retrieval-related signal is to check the existence of the final answer in the retrieved documents. Nevertheless, as shown in Table 1, we find that this improvement is also not the ultimate solution. Previous researchers have made similar conclusions: direct retrieval reward can not always improve the performance of search-during-thinking models [1].\n- **Linear rewards may over-emphasize intermediate behaviors.** Our findings clearly indicate that a linear combination of answer and refinement rewards ($R_{overall}$​=$R_{ans}$​+$R_{ref}$​) is _inferior_ to our proposed non-linear reward design, which prioritizes the final answer correctness while still fostering robust refinement capabilities. This intricate balance in the reward function is a core innovation of AutoRefine, directly contributing to its superior performance across various QA benchmarks.\n\n**Table 1:** Comparison Between Different Rewards Used in AutoRefine.\n\n|                                            | General QA |           |           | Multi-Hop QA |           |           |           |           |\n| ------------------------------------------ | ---------- | --------- | --------- | ------------ | --------- | --------- | --------- | --------- |\n|                                            | NQ         | TriviaQA  | PopQA     | HotpotQA     | 2wiki     | Musique   | Bamboogle | Avg.      |\n| AutoRefine-Reward at Refine - nonlinear    | **0.467**  | **0.620** | **0.450** | **0.405**    | **0.393** | **0.157** | **0.344** | **0.405** |\n| AutoRefine-Reward at Refine - linear       | 0.415      | 0.593     | 0.435     | 0.376        | 0.365     | 0.143     | 0.296     | 0.375     |\n| AutoRefine-Reward at Documents - nonlinear | 0.418      | 0.592     | 0.441     | 0.381        | 0.386     | 0.153     | 0.320     | 0.384     |\n| AutoRefine-Reward at Documents - linear    | 0.417      | 0.590     | 0.414     | 0.387        | 0.360     | 0.152     | 0.304     | 0.375     |\n| AutoRefine-only answer reward              | 0.423      | 0.583     | 0.424     | 0.368        | 0.351     | 0.139     | 0.344     | 0.371     |\n| Search-R1                                  | 0.421      | 0.583     | 0.413     | 0.297        | 0.274     | 0.066     | 0.128     | 0.312     |\n\n\n**[Key Advances]** Building on our solutions to these challenges, AutoRefine offers key advances that enhance performance and capabilities, such as:\n- **Summarization ability with no data annotation.** AutoRefine successfully activates the intrinsic knowledge refinement ability embedded within LLMs, resulting in significant performance gain across single- and multi-hop benchmarks.\n- **Introspection of missing knowledge.** In multi-hop search-during-think methods, mere summarization may not be sufficient. The model also needs to recognize missing information and adaptively plan subsequent search steps. We observed that AutoRefine successfully developed this capability in the refinement steps through RL (Table 5 in our paper), which significantly contributes to our superior performance on multi-hop benchmarks.\n\nUpon your opinions, we have acknowledged that our delivery in the paper does not fully reveal the challenges in developing the spontaneous refinement ability and unique advances of our method. We hope our explanation has successfully addressed your concerns regarding the novelty and contribution of this work.\n\n\n> **Weakness 2: More experimental settings** exploring more diverse experimental settings could make the paper much stronger. For example, if the ground-truth is given as a long and complex answer, how do we compute the retrieval reward, and how does that affect the final performance?\n\n**Response:**\nThank you for this suggestion. We entirely agree that exploring more diverse experimental settings, especially concerning the nature of the ground-truth answers and their impact on `retrieval reward` computation, is crucial for strengthening our paper. Besides the experiment added in Table 1, we have conducted **additional experiments** that explore how a more fine-grained retrieval reward would affect the model's performance on long and complex answers.\n\n**[Adapting Retrieval Rewards for Complex Answers]** Except for the existing cover-exact match used in our paper, we've explored two additional reward designs:\n1. Token-level recall: Calculates how many *tokens* in the ground-truth answer appear in the refined documents,\n2. Word-level recall: Calculates how many *words* in the ground-truth answer appear in the refined documents;\n\nunder different benchmark settings:\n- Full dataset: Test on all questions in the benchmark.\n- Complex answers: Test on questions with an answer >5 words.\n\nThe results are shown in Table 2.\n\n**Table 2:** Comparison between original and finer-grained retrieval reward design.\n\n|                                                  | **General QA** |           |           | **Multi-Hop QA** |           |           |\n| ------------------------------------------------ | -------------- | --------- | --------- | ---------------- | --------- | --------- |\n|                                                  | TriviaQA       | PopQA     | HotpotQA  | 2wiki            | Musique   | Avg.      |\n| **Full Dataset**                                 |                |           |           |                  |           |           |\n| AutoRefine - CEM Retrieval Reward                | **0.620**      | **0.450** | **0.405** | 0.393            | **0.157** | **0.405** |\n| AutoRefine - Token-level Recall Retrieval Reward | 0.604          | 0.433     | 0.376     | 0.364            | 0.136     | 0.383     |\n| AutoRefine - Word-level Recall Retrieval Reward  | 0.609          | 0.437     | 0.395     | **0.395**        | 0.142     | 0.396     |\n| **Complex Answers  (>5 words)**                  |                |           |           |                  |           |           |\n| AutoRefine - CEM Retrieval Reward                | 0.128          | 0.261     | 0.105     | 0.368            | 0.023     | 0.177     |\n| AutoRefine - Token-level Recall Retrieval Reward | **0.132**      | 0.292     | 0.094     | 0.379            | 0.047     | 0.189     |\n| AutoRefine - Word-level Recall Retrieval Reward  | 0.131          | **0.375** | **0.113** | **0.409**        | **0.054** | **0.216** |\n\nThe results presented in Table 2 confirm that a more fine-grained retrieval reward can greatly improve the model's performance when answers have complex forms, while keeping the overall performance approximately the same.\n\nWe hope these additional results could help strengthen our experiment design.\n\n[1] Jin, Bowen, et al. \"An Empirical Study on Reinforcement Learning for Reasoning-Search Interleaved LLM Agents.\" arXiv preprint arXiv:2505.15117 (2025)."}, {"review_id": "8awQix73zL", "reviewer": "Reviewer_9S2E", "summary": "The paper presents a method for enhancing reinforcement learning (RL) training in retrieval-augmented question answering through evidence compression. It introduces soft rewards designed to promote faithful answer generation based on retrieved context. The approach is evaluated on diverse Wikipedia-based short-form single- and multi-hop open-ended QA tasks, demonstrating performance gains compared to existing baselines.", "questions": "1. Hyperparameter justification: The paper lacks explicit rationale for key hyperparameters (e.g., choice of E5 as retriever, 3 documents fetched during training). If informed by prior work, this should be clearly stated to demonstrate scientific grounding.\n\n2. Reward function ablation: The reward function’s design (e.g., 0.1 as partial reward, no partial reward for correct final answers) is not analyzed. This omission limits understanding of the method’s effectiveness and trade-offs.\n\n3.  Training configuration ambiguity: The choice of maximum search calls (5) and whether training settings align with Search-R1 are unclear. Section B.1 and Table 3 do not provide sufficient details to verify reproducibility or consistency with prior work.\n\n4. Data split specification: The dev split of Hotpot-QA (distractor vs. fullwiki) is not explicitly stated, creating ambiguity about the evaluation’s scope and fairness.", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n1. The proposed method demonstrates improvements over multiple baselines, such as Search-R1 and ReSearch, by a relative margin over 20% in hard multi-hop tasks.\n\n\n### Weaknesses\n1. The paper appears to be an incremental development over Search-R1[1] and Search-o1[2]. It is established in the recent works, such as Search-o1[2] (Figure 2, \"Get concise information and continue coherent reasoning.\") and FILCO[3], that evidence compression plays critical role to avoid noise generate from imperfect retrieval. \n\n2. Though the focus of the paper is on evidence compression, no strong baseline is chosen related to it, such as FILCO[3] and FaviComp[4]. Moreover, the Related Work section completely overlooks all evidence compression literature. Moreover, important details of the baselines are missing, such as which model size and algorithm (For Search-R1, PPO or GRPO?) was chosen for comparison.\n\n3. Model scale and scalability: The paper does not discuss model scale’s impact on generalization or scalability. The choice of GRPO over PPO for Search-R1 is unclear, as PPO is a stronger baseline for this task in Search-R1.\n\n4. Limited applicability: The soft-reward mechanism is restricted to short-form generations, with no exploration of its utility for long-form tasks (e.g., Self-RAG). This limits the method’s real-world relevance. It is also unclear  what is the  latency of  the method over baselines for use experience.\n\n5. Statistical rigor: Error bars or standard deviations are omitted, such as in Table 1 and Figure  4, despite the checklist mentioning them to validate result significance. This weakens the reliability of the reported improvements.\n\n\nReferences:\n\n[1] Jin, Bowen, et al. \"Search-r1: Training llms to reason and leverage search engines with reinforcement learning.\" arXiv preprint arXiv:2503.09516 (2025).\n\n[2] Li, Xiaoxi, et al. \"Search-o1: Agentic search-enhanced large reasoning models.\" arXiv preprint arXiv:2501.05366 (2025).\n\n[3] Wang, Zhiruo, et al. \"Learning to filter context for retrieval-augmented generation.\" arXiv preprint arXiv:2311.08377 (2023).\n\n[4] Jung, Dongwon, et al. \"Familiarity-aware evidence compression for retrieval augmented generation.\" arXiv preprint arXiv:2409.12468 (2024).\n\n[5]  Asai, Akari, et al. \"Self-rag: Learning to retrieve, generate, and critique through self-reflection.\" The Twelfth International Conference on Learning Representations. 2023.", "was_revised": true, "final_justification": "The authors have clarified the concerns:\n- Comparison with evidence compression literature.\n- Statistical significance for provided results.\n- Justification of model scales, short form QA and using GRPO.\n\nThe paper requires a major revision with a focus on using RL for evidence compression as the search/think are build on top of Search-R1 and existing literature.", "rebuttal": "We thank the reviewer for recognizing the significant performance gain achieved in this work. We provide point-to-point responses to your primary concerns below.\n\n> W1: The paper appears to be an incremental development over Search-R1[1] and Search-o1[2]. It is established in the recent works, such as Search-o1[2] (Figure 2, \"Get concise information and continue coherent reasoning.\") and FILCO[3], that evidence compression plays critical role to avoid noise generate from imperfect retrieval.\n\nWe appreciate the reviewer's comments on the contribution of our work, but we beg to refute the opinion that our paper is merely an incremental development.\n\nOur work addresses distinct challenges and demonstrates unique advantages:\n- **[Unique Challenges We Tackle]** We conduct **new experiments** (see Table 2 addressing Weakness 1 of reviewer KVnw) to demonstrate the challenges in developing refinement abilities via RL in Search-R1. These challenges include:\n    - Sub-optimality of simple paradigm modifications.\n    - Limited improvement from direct retrieval rewards on retrieved documents, echoing findings in previous research [1].\n    - Model benefits from non-linear retrieval reward.\n- **[Key Behaviors and Unique Benefits]** We conduct **additional experiments** (please see Table 3 addressing Weakness 1 of reviewer KVnw) to demonstrate the advantage of our method beyond simply applying summarization modules in Search-R1.\n    - No Extra Annotation Requirements, unlike previous SFT-based knowledge distillation.\n    - Recognization of missing information and adaptively plan subsequent search steps through RL (Table 5 in the paper).\n\nWe hope our detailed analysis of challenges and unique benefits clearly illustrates the novel contributions of our work.\n\n> W2.1: Though the focus of the paper is on evidence compression, no strong baseline is chosen related to it, such as FILCO[3] and FaviComp[4]. Moreover, the Related Work section completely overlooks all evidence compression literature.\n\n**Response:**\nWe appreciate the reviewer's suggestion for comparison with previous evidence compression methods.\n\nAccording to the reviewer's suggestion, we have conducted an additional experiment comparing evidence compression methods with AutoRefine comparing with (1) FaviComp [2] and (2) Search-R1 with an external summarization module [3]. The results are shown in Table 1.\n\n**Table 1:** Comparison with compression approaches.\n\n||GeneralQA|||Multi-HopQA|||||\n|-|-|-|-|-|-|-|-|-|\n||NQ|TriviaQA|PopQA|HotpotQA|2wiki|Musique|Bamboogle|Avg.|\n|AutoRefine|0.467|0.620|0.450|0.405|0.393|0.157|0.344|0.405|\n|FaviComp|0.302|0.502|0.410|0.240|0.220|0.054|0.232|0.280|\n|Search-R1+external refiner|0.395|0.619|0.450|0.337|0.239|0.065|0.115|0.317|\n\nThe experimental results show:\n- External summarization helps on single-hop QA benchmarks (TriviaQA and PopQA);\n- AutoRefine surpasses the external summarizer in multi-hop QA setting, which highlights the unique advances summarized in the previous response addressing weakness 1.\n\n> W2.2: Moreover, important details of the baselines are missing, such as which model size and algorithm (For Search-R1, PPO or GRPO?) was chosen for comparison.\n\n**Response:**\nThank the reviewer for pointing out the unclear implementation details of baseline methods.\n- We use GRPO as the default algorithm for RL-based baselines.\n- We use Qwen2.5-3B-Base as the default model for RL-based methods, as depicted in Sec 3.1.\n\nWe hope the above clarification can address your concerns.\n\n> W3: Model scale and scalability: The paper does not discuss model scale’s impact on generalization or scalability. The choice of GRPO over PPO for Search-R1 is unclear, as PPO is a stronger baseline for this task in Search-R1.\n\n**Response:**\nWe thank the reviewer for pointing out the importance of the model's scale and RL algorithm choice.\n\n**[Impact of Model's Scale]** We conduct an **additional experiment** to see the impact of the model's size, please check Table 3 in our response to Reviewer UA8w. AutoRefine surpasses previous methods by ~9% on 3B-level model and ~5% on 7B-level model.\n\n**[Choice of RL Algorithm]** We beg to refute the claim that “PPO is a stronger baseline for this task”, because:\n- As shown in Table 3 of Search-R1 [4], the GRPO algorithm has comparable performance as PPO on Qwen-3B models.\n- According to the discussion in Sec 5.1 of Search-R1 [4], \"both methods GRPO and PPO) achieve similar final train reward and performance”.\n\n> W5: Limited applicability: The soft-reward mechanism is restricted to short-form generations, with no exploration of its utility for long-form tasks (e.g., Self-RAG). This limits the method’s real-world relevance. It is also unclear what is the latency of the method over baselines for use experience.\n\n**Response:**\nWe thank the reviewer for this comment.\n\n**[Long Form QA Effectiveness]** \nWe really appreciate the importance of long-form generation tasks, but we focuses on multi-hop QA ability which is also of high practicality. We believe that RAG models don't have to excel at every QA type to be considered contributory.\nFor example, Self-RAG [5] focuses on improving the long-form QA without considering multi-hop, but it is still an excellent work with a high influence.\n\n**[Service Latency]** The additional computation of AutoRefine is merely the generation of `<refine>` content, which causes little and acceptable latency overhead. To support our claim, we showcase **additional results** about speed test between models in Table 3.\n\n**Table 3:** Speed Test of Different Methods.\n| Method| 1-by-1 Inference (sample·s) | Batched Inference (bs=512) (sample·ms) |\n|-|-|-|\n| AutoRefine| 4.743 ± 1.065| 12.921 ± 1.011|\n| Search-R1| 3.479 ± 0.542| 11.815 ± 0.980|\n| ReSearch| 3.155 ± 0.567| 11.773 ± 1.271|\n\n> W5: Statistical rigor: Error bars or standard deviations are omitted, such as in Table 1 and Figure 4, despite the checklist mentioning them to validate result significance. This weakens the reliability of the reported improvements.\n\n**Response:**\nThanks for pointing out the importance of statistical significance. We add a **new statistical analysis** by running the experiments under different random seeds, and the results are shown in Table 4. The p-value column indicates the T-test between each baseline and AutoRefine.\n\n**Table 4:** Statistical Analysis.\n||NQ|TriviaQA|PopQA|HotpotQA|2wiki|Musique|Bamboogle|Avg.|P-Value|\n|-|-|-|-|-|-|-|-|-|-|\n|AutoRefine|0.452 ± 0.017|0.627 ± 0.007|0.468 ± 0.017|0.423 ± 0.016|0.404 ± 0.010|0.145 ± 0.011|0.335 ± 0.023|0.408 ± 0.014|-|\n|ReSearch|0.418 ± 0.012|0.614 ± 0.014|0.451 ± 0.018|0.317 ± 0.015|0.269 ± 0.017|0.056 ± 0.015|0.132 ± 0.008|0.322 ± 0.014|5.49E-06|\n|Search-R1|0.410 ± 0.009|0.605 ± 0.019|0.429 ± 0.014|0.315 ± 0.016|0.254 ± 0.023|0.062 ± 0.005|0.127 ± 0.020|0.315 ± 0.015|2.85E-06|\n\nThe results suggest the improvement achieved by AutoRefine is significant over previous search-during-think methods.\n\n\n> Q1: Reward function ablation: The reward function’s design (e.g., 0.1 as partial reward, no partial reward for correct final answers) is not analyzed. This omission limits understanding of the method’s effectiveness and trade-offs.\n\n**Response:**\nThanks for your comment on the lack of explanation in our reward design.\nDifferent reward designs will indeed influence AutoRefine's performance. We conduct **additional experiments** to reveal this influence. Please refer to our Table 2 in response to Reviewer KVnw.\n\nThe results indicate:\n- Retrieval reward on retrieved documents shows limited improvement. We find that directly checking whether the documents contain the answer only gives a limited performance gain. Previous researchers have also made similar conclusions [1].\n- Linear reward may over-emphasize intermediate behaviors. Adding $R_{ref}$ with $R_{ans}$ linearly as $R_{overall}=R_{ans}+R_{ref}$ is also sub-optimal. In this work, we use non-linear retrieval rewards that prioritize final answer correctness beyond merely correct knowledge refinement.\n\n\n> Q2: Hyperparameter justification: The paper lacks explicit rationale for key hyperparameters (e.g., choice of E5 as retriever, 3 documents fetched during training). If informed by prior work, this should be clearly stated to demonstrate scientific grounding.\n\n> Q3: Training configuration ambiguity: The choice of maximum search calls (5) and whether training settings align with Search-R1 are unclear. Section B.1 and Table 3 do not provide sufficient details to verify reproducibility or consistency with prior work.\n\nThank you for the opinion. We keep the retrieval setup the same as our baseline Search-R1. We will acknowledge this inspiration in future revision.\n\nThe choice of maximum search calls is empirically set, because:\n- The average search calls of our method remain less than 2.5, so no need for too many calls.\n- There’s no clear evidence that this parameter will significantly influence the result.\n\nAll methods compared in Table 1 have the same experiment settings, as mentioned in Section 3.1 of our paper.\n\nWe have included most of the important parameters in Section 3.1, Appendix B, and Table 3. we suggest referring to our attached code for more details.\n\n\n> Q4: Data split specification: The dev split of Hotpot-QA (distractor vs. fullwiki) is not explicitly stated, creating ambiguity about the evaluation’s scope and fairness.\n\nWe use the dev split of HotpotQA, which does not contain distractor/fullwiki variants (Table 4 in paper).\n\n[1] An Empirical Study on Reinforcement Learning for Reasoning-Search Interleaved LLM Agents\n\n[2] Familiarity-Aware Evidence Compression for Retrieval-Augmented Generation\n\n[3] BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension\n\n[4] Search-r1: Training llms to reason and leverage search engines with reinforcement learning\n\n[5] Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection"}, {"review_id": "596u2WChAT", "reviewer": "Reviewer_UA8w", "summary": "The paper studies the problem of training an LLM to issue search queries to enable RAG. Based on the observation that prior works have mostly focused on eliciting search skills in the LM via SFT, the proposed method uses RL to train a reader model. While the proposed paper is not the first to use RL for training a reader, they make two key contributions:\n1. the proposed method, AutoRefine, flexibly summarizes (“refines”) retrieved documents in addition to retrieving, akin to https://arxiv.org/abs/2311.08377\n2. the proposed method uses process rewards for retrieval\n\nOn a variety of question answering benchmarks, the proposed method is shown to be very promising.", "questions": "- The retriever reward uses a binary reward based on term matching of the answer with the ground truth. This seems to effectively be training the model to prefer exact match queries, which may not be globally optimal. Is this a legitimate concern?", "limitations": "Yes.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The experiments in this paper are very well-chosen and give a complete picture of the proposed method.\n- The empirical support for the efficacy of this method is strong for multi-hop QA. \n\nWeaknesses:\n- It seems that the two methodological contributions of this paper (\"Refine\" and \"Process Rewards for Search\") each help on different benchmarks. This suggests that potentially these two methods are actually more targeted. I was hoping to see more discussion of this.\n- The retriever reward uses a binary reward based on term matching of the answer with the ground truth. This seems to effectively be training the model to prefer exact match queries, which may not be globally optimal. I'd like to see more discussion of the possible risks of this approach.\n- As mentioned by the authors in the limitation section, the proposed method is only demonstrated with 3B models, which are a bit smaller than standard for readers in a RAG system these days (8B has become the norm in academic work). It's likely that 7B or 14B models will benefit less from carefully-designed inductive biases such as the ones proposed in this paper. In any case, this does not invalidate the value of the paper.", "was_revised": true, "final_justification": "Final score: 5 (accept).\n\nAfter reviewing the author's responses and other reviewers' comments, I maintain my score of a 5. I agree with reviewer jRtt that this paper is largely incremental. It is conceptually just combining two existing paradigms: filtering context for RAG [1] and improving agentic search via RL [2]. I agree with reviewer jRtt that the resemblance to prior work is understated, and I believe it is concerning that [1] was not cited in this paper.\n\n[1] Wang, Zhiruo, et al. \"Learning to filter context for retrieval-augmented generation.\" arXiv preprint arXiv:2311.08377 (2023).\n[2] Jin, Bowen, et al. \"Search-r1: Training llms to reason and leverage search engines with reinforcement learning.\" arXiv preprint arXiv:2503.09516 (2025).\n\n**However**, unlike reviewer jRtt, I don't believe that an incremental contribution is inappropriate for NeurIPS. I think this paper demonstrates that RL training is useful for evidence refinement/context filtering, and I think the methods used to achieve that (including process rewards) are quite instructive.\n\n\nTL;DR: incremental contribution, well-executed, likely to be useful to the community", "rebuttal": "We're grateful for your insightful review and appreciate your recognition of our work's comprehensive evaluation and outstanding efficacy. We have addressed your major concerns in detail below.\n\n> **Weakness 1: More Explanation over Methodological Contributions.** It seems that the two methodological contributions of this paper (\"Refine\" and \"Process Rewards for Search\") each help on different benchmarks. This suggests that potentially these two methods are actually more targeted. I was hoping to see more discussion of this.\n\n**Response:**\nWe appreciate your insightful suggestion regarding the distinct impacts of our methodological contributions.\n\nWe agree that our two main contributions, \"Knowledge Refinement Step\" and \"Processing Rewards for Search\" (retrieval reward), exhibit different performance characteristics across benchmarks. To better answer the question, we **extend our ablation study** with a new control group that combines Search-R1 with an external fine-tuned summarization model (`facebook/bart-large-cnn` [1]), as shown in Table 1. In the new setting, the external model summarizes retrieved documents and appends the summarization at the end of the retrieval. \n\n**Table 1:** Extended Ablation Study.\n\n|                                 | General QA |          |       | Multi-Hop QA |       |         |           |       |\n| ------------------------------- | ---------- | -------- | ----- | ------------ | ----- | ------- | --------- | ----- |\n|                                 | NQ         | TriviaQA | PopQA | HotpotQA     | 2wiki | Musique | Bamboogle | Avg.  |\n| AutoRefine                      | 0.467      | 0.620    | 0.450 | 0.405        | 0.393 | 0.157   | 0.344     | 0.405 |\n| - retrieval reward              | 0.423      | 0.583    | 0.424 | 0.368        | 0.351 | 0.139   | 0.344     | 0.376 |\n| - retrieval reward & refinement | 0.422      | 0.585    | 0.419 | 0.294        | 0.257 | 0.062   | 0.144     | 0.312 |\n| Search-R1 + external summarization | 0.395      | 0.619    | 0.450 | 0.337        | 0.239 | 0.065   | 0.115     | 0.317 |\n\nWe observe the following:\n- **External Document Summarization Boosts Single-hop Performance.** Search-R1's single-hop QA performance (TriviaQA, PopQA) is approximately the same as AutoRefine when equipped with an external summarization module. This indicates that mere summarization is sufficient in such simple scenarios. On the contrary, the multi-hop QA performance slightly drops.\n- **Refinement Step Boosts Multi-hop Performance.** Incorporating refinement steps in the reasoning paths significantly improves the model's performance on multi-hop benchmarks, while keeping the single-hop performance unchanged. What promotes such solid multi-hop performance? Taking a closer look at the case studies (Table 5 in our paper), we find that the model evolves the introspection behavior - it spontaneously reflects missing information in the retrieved documents and \n- **Retrieval Reward Yields Comprehensive Improvements.** However, merely adding a refinement step does not fully develop refinement behavior. When trained with the retrieval reward, we observe a comprehensive improvement across both single- and multi-hop benchmarks. Notably, the model exhibits similar performance on single-hop benchmarks as vanilla Search-R1 with an external summarization module, which suggests that the performance gain stems from the enhanced summarization capability of the LLM.\n\nWe hope our fine-grained explanation with additional experiments can address your concerns.\n\n> **Weakness 2: Possible Risk of EM Retrieval Reward.** The retriever reward uses a binary reward based on term matching of the answer with the ground truth. This seems to effectively be training the model to prefer exact match queries, which may not be globally optimal. I'd like to see more discussion of the possible risks of this approach.\n\n**Response:**\n\nThanks for pointing out the potential risk in the strict cover EM retrieval reward. After careful consideration, we hypothesize that the strict reward may harm the models' performance when the answer is complex.\n\nWe add **new experiments**, comparing our current retrieval reward against two more flexible designs:\n1. Token-level recall: Calculates how many *tokens* in the ground-truth answer appear in the refinement,\n2. Word-level recall: Calculates how many *words* in the ground-truth answer appear in the refinement;\n\nunder different benchmark settings:\n- Full dataset: Test on all questions in the benchmark.\n- Complex answers: Test on questions with an answer >5 words.\n\nThe model is evaluated with EM, and the results are summarized in Table 2.\n\n**Table 2:** Comparison between original and finer-grained retrieval reward design.\n\n|                                                  | **General QA** |           |           | **Multi-Hop QA** |           |           |\n| ------------------------------------------------ | -------------- | --------- | --------- | ---------------- | --------- | --------- |\n|                                                  | TriviaQA       | PopQA     | HotpotQA  | 2wiki            | Musique   | Avg.      |\n| **Full Dataset**                                 |                |           |           |                  |           |           |\n| AutoRefine - CEM Retrieval Reward                | **0.620**      | **0.450** | **0.405** | 0.393            | **0.157** | **0.405** |\n| AutoRefine - Token-level Recall Retrieval Reward | 0.604          | 0.433     | 0.376     | 0.364            | 0.136     | 0.383     |\n| AutoRefine - Word-level Recall Retrieval Reward  | 0.609          | 0.437     | 0.395     | **0.395**        | 0.142     | 0.396     |\n| **Complex Answers  (>5 words)**                  |                |           |           |                  |           |           |\n| AutoRefine - CEM Retrieval Reward                | 0.128          | 0.261     | 0.105     | 0.368            | 0.023     | 0.177     |\n| AutoRefine - Token-level Recall Retrieval Reward | **0.132**      | 0.292     | 0.094     | 0.379            | 0.047     | 0.189     |\n| AutoRefine - Word-level Recall Retrieval Reward  | 0.131          | **0.375** | **0.113** | **0.409**        | **0.054** | **0.216** |\n\nThe results confirm that though the cover EM reward works well on most QA cases, it could **bring potential risks** such as performance degradation **when the target answer is complex**.\n\n> **Weakness 3: Performance at Different Model Scales.** As mentioned by the authors in the limitation section, the proposed method is only demonstrated with 3B models, which are a bit smaller than standard for readers in a RAG system these days (8B has become the norm in academic work). It's likely that 7B or 14B models will benefit less from carefully-designed inductive biases such as the ones proposed in this paper. In any case, this does not invalidate the value of the paper.\n\n**Response:** \nWe appreciate you raising the point about model scale and its potential impact on our method's effectiveness. We agree that larger models (e.g., 7B or 14B) might inherently benefit less from explicitly designed inductive biases.\n\nWe conduct **additional experiments** to demonstrate AutoRefine's effectiveness at the 7B level, and the results are shown in Table 3. The baseline result (Search-R1) is reproduced using their officially released checkpoint.\n\n**Table 3:** Performance Comparison at Different Model Scales\n\n|                 | General QA |          |       | Multi-Hop QA |       |         |           |       |\n| --------------- | ---------- | -------- | ----- | ------------ | ----- | ------- | --------- | ----- |\n|                 | NQ         | TriviaQA | PopQA | HotpotQA     | 2wiki | Musique | Bamboogle | Avg.  |\n| **Qwen2.5-7B-Base** |            |          |       |              |       |         |           |       |\n| Search-R1       | 0.469      | 0.627    | 0.449 | 0.410        | 0.272 | 0.173   | 0.456     | 0.408 |\n| AutoRefine      | 0.484      | 0.659    | 0.487 | 0.451        | 0.405 | 0.187   | 0.512     | 0.455 |\n| **Qwen2.5-3B-Base** |            |          |       |              |       |         |           |       |\n| Search-R1       | 0.421      | 0.583    | 0.413 | 0.297        | 0.274 | 0.066   | 0.128     | 0.312 |\n| AutoRefine      | 0.467      | 0.620    | 0.450 | 0.405        | 0.393 | 0.157   | 0.344     | 0.405 |\n\nFrom the Table, we observe that AutoRefine still yields positive results even at the 7B parameter level, demonstrating an overall performance improvement of approximately 5%.\n\nThis gain is indeed slightly less than the significant 9% improvement observed with 3B models, which aligns with your hypothesis that the benefits from explicit refinement steps might relatively diminish as model scale increases.\n\n> **Question 1: Potential Risk in Strict Retrieval Reward.** The retriever reward uses a binary reward based on term matching of the answer with the ground truth. This seems to effectively be training the model to prefer exact match queries, which may not be globally optimal. Is this a legitimate concern?\n- Please see our response to your **Weakness 2**.\n\n[1] Lewis, Mike, et al. \"BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension.\" _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. 2020."}, {"review_id": "kj8Av0N2eV", "reviewer": "Reviewer_KVnw", "summary": "This paper proposes a method to enhance retrieval-augmented generation (RAG) by allowing a large language model (LLM) to first think before formulating a search query, and then generate an answer based on the retrieved documents. Building upon prior work such as Search-R1, the authors introduce a novel refine step: after retrieval, the LLM processes the raw documents into a refined form, aiming to prevent misleading information from affecting the final answer. The entire pipeline is trained using reinforcement learning (RL), with a retrieval reward that encourages fetching documents containing the correct answer. Experimental results demonstrate that both the refine step and the retrieval reward contribute to improved performance.", "questions": "1. Regarding the retrieval reward: how does the method handle cases where (a) the original documents do not contain the correct answer, or (b) the answer is present in the original documents but lost during refinement? Are these two scenarios treated equivalently during training?\n2. In Section 1, the paper claims that SFT struggles to generalize retrieval behaviors to out-of-distribution (OOD) scenarios. Can the authors provide empirical evidence or prior work that supports this claim, especially in the context of retrieval-augmented generation?\n3. Is RL truly necessary for learning the refinement step? Have the authors compared against a supervised or prompted alternative (e.g., using gold summaries or instruction-tuned additional refiners) to validate that RL offers a meaningful advantage?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## Strengths\n\n- The proposed method is simple yet effective.\n- The paper is well-written and easy to follow, with clear analyses of the method's components.\n- The experimental results support the effectiveness of the proposed additions (refinement and reward design).\n\n## Weaknesses\n\n- **Incremental Contribution.** The main idea appears to be an incremental improvement over Search-R1 [1], which already introduced the paradigm of reasoning-before-retrieval trained via RL. The notion that irrelevant context can harm model performance is not new [2], and several previous works have proposed techniques to mitigate this by extracting only relevant information [3]. The contribution of this paper lies primarily in combining document refinement with the Search-R1-style framework and demonstrating its effectiveness, but this alone may be insufficient to warrant acceptance unless further novelty is established.\nTo better establish the novelty, the authors should articulate the unique challenges or benefits of applying document refinement within a Search-R1-style reasoning-then-retrieval framework, especially in contrast to naive RAG or prior refinement approaches used in more traditional pipelines. Clarifying what makes refinement in this setting particularly necessary or non-trivial (e.g., interaction with the RL objective, refinement timing, or complexity from prior reasoning steps) would significantly strengthen the contribution.\n- **Unclear Justification for RL.** In the Introduction, the authors state that “SFT often fails to generalize retrieval behaviors beyond the training distribution.” However, there appears to be no empirical evidence in the paper supporting this claim, nor do the cited references directly address RAG settings. The necessity of RL for both query generation and refinement should be better motivated. Could supervised fine-tuning (SFT) or instruction tuning suffice for learning the refinement step?\n- **Misleading Title.** The part of title “Autonomous Retrieval-Augmented Reasoning of LLMs” is somewhat misleading. The core ideas behind autonomous retrieval-augmented reasoning were already developed in Search-R1, while this paper primarily introduces a refinement module and retrieval reward. The current title may give the impression that this paper introduces the full framework from scratch.", "was_revised": true, "final_justification": "After reading the paper, the rebuttal, and follow-up responses from the authors, I have updated my evaluation as follows:\n\nResolved Issues:\n\n- Clarity of Contribution: The authors clarified how their refinement module adds to prior work like Search-R1. The response highlighted that RL-based refinement not only summarizes but also supports sub-planning in multi-hop QA, a capability not addressed by external refiners.\n\n- Justification for RL: The authors provided new experiments comparing RL-trained refinement with supervised external summarizers (e.g., BART). These results show that RL offers significant advantages in multi-hop settings, supporting their design choice.\n\n- Title and Framing: The authors acknowledged that the original title was potentially misleading and proposed a clearer alternative. They also committed to improving Figure 1 and better surfacing key comparisons (e.g., Table 3).\n\nRemaining Issues:\n\n- Incomplete Baselines: The comparison with more recent instruction-tuned models (e.g., Qwen2.5, LLaMA3) is still ongoing. While the authors are running these experiments, the lack of results during the rebuttal period leaves the evaluation somewhat incomplete.\n\n- Framing and Presentation: While the authors plan to revise presentation issues in future versions, some of the paper's key motivations (e.g., why RL matters for refinement) were not clearly conveyed in the initial submission.\n\nSummary:\n\nDespite some remaining limitations, I found the authors' response constructive and detailed, and their additional experiments and clarifications meaningfully strengthened the submission. Given the promising empirical results and the novelty of combining RL with refinement in a reasoning-during-retrieval framework, I now view this as a technically sound and useful contribution.", "rebuttal": "We sincerely appreciate your thorough review and your positive remarks regarding the conciseness, effectiveness, and delivery of our work. We deeply appreciate your constructive critique regarding our limitations, as it truly helped us strengthen this research.\n\n> W1: The main idea appears to be an incremental improvement over Search-R1 ...\n\nThank you for your incisive feedback regarding our paper's contribution. You've accurately identified the two core foundations of our work: (1) search-during-reasoning methods and (2) document refinement in RAG. We believe our proposed AutoRefine introduces distinct technical contributions, differentiating it from current approaches in three key aspects: (1) training technique, (2) data requirements, and (3) model behaviors, as summarized in Table 1:\n\nTable 1: Technical Comparison Between Our Work and Previous Methods.\n\n||training technique||data requirements|model behaviors||\n|-|-|-|-|-|-|\n||Training Alg|Reward||Reasoning Ability|Refine Behavior|\n|FilCO[1]|SFT|-|Only(Q,A)pairs|-|summarization|\n|IRCoT[2]|-|-|-|√|-|\n|InstructRAG[3]|SFT/ICL|-|(Q,A)+rationale|√|summarization|\n|Search-o1[4]|-|-|-|√|summarization|\n|Self-RAG[5]|SFT|-|(Q,A)+reflect tokens|-|distinguishing|\n|Search-R1[6]|RL|outcome|(Q,A) pairs|√|-|\n|Search-R1-rethink[7]|RL|outcome+retrieval reward|(Q,A) pairs|√|-|\n|AutoRefine|RL|outcome+retrieval reward|(Q,A) pairs|√|summarization+introspection|\n\nAs you suggested, we elaborate on the unique challenges and benefits of applying RL-driven refinement within a search-during-reasoning framework.\n\n**[Unique Challenges]** To illustrate the challenges in developing refinement abilities via RL, we conducted **new experiments** providing empirical evidence, as shown in Table 2.\n- Simple paradigm modification is helpful but sub-optimal. Simply changing the reasoning paradigm to \"search-and-refine-during-think\" does improve overall performance.\n- Retrieval reward on retrieved documents shows limited improvement. Checking whether the answer appears in retrieved documents is also not the ultimate solution, which is noticed by previous research[7]. We hypothesize this is because rewards on documents may shortcut the model to draft more search queries for broader knowledge coverage, rather than more accurate queries and more efficient refinement.\n- Linear reward may over-emphasize intermediate behaviors. An intuitive way is to add $R_{ref}$ with $R_{ans}$ linearly as $R_{overall}=R_{ans}+R_{ref}$. However, we believe that the extra abilities (format-following, refinement) should be foundational yet incidental, as the primary aim is to achieve a correct answer. As a result, non-linear retrieval rewards surpasses linear rewards on both docs/refines.\n\nTable 2: Comparison Between Different Rewards Used in AutoRefine.\n\n||General QA|||Multi-Hop QA|||||\n|-|-|-|-|-|-|-|-|-|\n||NQ|TriviaQA|PopQA|HotpotQA|2wiki|Musique|Bamboogle|Avg.|\n|AutoRefine-Reward at Refine-nonlinear|0.467|0.620|0.450|0.405|0.393|0.157|0.344|0.405|\n|  -Reward at Refine-linear|0.415|0.593|0.435|0.376|0.365|0.143|0.296|0.375|\n|  -Reward at Documents-nonlinear|0.418|0.592|0.441|0.381|0.386|0.153|0.320|0.384|\n|  -Reward at Documents-linear|0.417|0.590|0.414|0.387|0.360|0.152|0.304|0.375|\n|  -only answer reward|0.423|0.583|0.424|0.368|0.351|0.139|0.344|0.371|\n|Search-R1[6]|0.421|0.583|0.413|0.297|0.274|0.066|0.128|0.312|\n\n**[Unique Benefits]**\n- No annotation requirements. AutoRefine leverages the intrinsic knowledge refinement ability of LLMs without preliminary search trajectory annotation.\n- Complexity in search-during-think reasoning. We conducted **additional experiments** comparing AutoRefine with using an external refiner model with Search-R1, as shown in Table 3 addressing your Q3. While external refiner achieve good performance on single-hop QA, it causes performance drops on multi-hop ones. Besides summarization, AutoRefine learns to recognize missing information and plan subsequent search steps through RL (Table 5 in our paper). This ability contributes to the superior multi-hop performance.\n\n> W2: Could supervised fine-tuning (SFT) or instruction tuning suffice for learning the refinement step?\n\nWe appreciate the reviewer's astute observation regarding the unclear justification in our introduction. We agree that our initial statement, claiming \"SFT often fails to generalize retrieval behaviors beyond the training distribution\", is **unsubstantiated and overly opinionated** without empirical evidence.\n\nUpon your suggestion, we propose the following revisions to clarify the necessity of RL for training retrieval-augmented LLMs:\n- Revised Justification for RL. We will modify the statement to highlight the practical challenges of SFT. Specifically, below is our revised statement:\n\n\"*While SFT can be effective for training large models for search, it sometimes necessitates the construction of high-quality search paths, which incurs additional effort and resource overheads. To address this, recent studies draw inspiration from RL-based post-training and explore RL for retrieval-augmented reasoning, achieving excellent results by only evaluating final answer correctness without the need for pre-collected reasoning paths.*\"\n- Addition of Contextual Citations. We will explicitly acknowledge the effectiveness of SFT by adding the following citations [4,8,9,10].\n\nThank you once again for identifying this crucial flaw in our introduction. We will thoroughly review the remainder of our paper to identify and rectify any similar issues, ensuring our claims are rigorously supported and well-justified.\nWe believe these revisions will significantly solidify our work and enhance its contribution to the research community.\n\n\n> W3: The part of title “Autonomous Retrieval-Augmented Reasoning of LLMs” is somewhat misleading.\n\nWe sincerely thank the reviewer for their insightful feedback regarding our original title. We agree that it (1) did not fully capture our core innovations, the refinement module and retrieval reward, and (2) could indeed imply a broader scope than intended.\n\nActing on this valuable suggestion, we've decided to **update our paper's title** to:\n\n*Search and Refine During Think: Facilitating On-the-Fly Knowledge Distillation for Improved Retrieval-Augmented Reasoning*\n\nWe believe this new title more accurately reflects the essence of our work by highlighting its key focus (retrieval-augmented reasoning), key novelty (on-the-fly knowledge distillation methodology), and key results (improved performance).\n\n> Q1: Are these two scenarios treated equivalently during training?\n\nYes. In our method, the retrieval reward specifically reflects the correctness of the refinement process itself.\nWe believe that truly effective search-augmented reasoning requires not only retrieving correct information, but also effective use of the information.\n\nAs shown in Figure 5(a) in our paper, the search success rate begins relatively high compared to refine and answer, and ends up nearly 80%. From this phenomenon, we hypothesize that **search behaviors are relatively easy to learn**. Thus, we primarily shape the refine behaviors during RL training.\n\n> Q2: Can the authors provide empirical evidence or prior work that supports this claim, especially in the context of retrieval-augmented generation?\n\nPlease see our response to W2.\n\n> Q3: Is RL truly necessary for learning the refinement step?\n\nThank you for raising the excellent point regarding the necessity of RL for the refinement step and the comparison against more direct, supervised approaches. Upon your advice, we conduct **additional experiments** under the intuitive setting: using a fine-tuned external refiner [11] to summarize retrieved documents in Search-R1, and the results are presented in the table below:\n\nTable 3: Comparison Between SFT-based Refiner and RL-based Refinement.\n\n||GeneralQA|||Multi-HopQA|||||\n|-|-|-|-|-|-|-|-|-|\n||NQ|TriviaQA|PopQA|HotpotQA|2wiki|Musique|Bamboogle|Avg.|\n|AutoRefine|0.467|0.620|0.450|0.405|0.393|0.157|0.344|0.405|\n|Search-R1|0.421|0.583|0.413|0.297|0.274|0.066|0.128|0.312|\n|Search-R1+external refiner|0.395|0.619|0.450|0.337|0.239|0.065|0.115|0.317|\n|ReSearch|0.427|0.597|0.430|0.305|0.272|0.074|0.128|0.319|\n|ReSearch+external refiner|0.419|0.614|0.445|0.334|0.248|0.069|0.114|0.321|\n\nFrom these results, we observed that external refiners improves performance on several general QA benchmarks, but leads to a performance decline on multi-hop QA ones.\n\n**[Role of RL]** These observations lead us to the following explanation regarding the distinct roles of external refiners and our RL-trained refinement module:\n- **Single-Hop Scenarios:** In single-hop QA, refinement learned via RL appear to function similarly to external refiners.\n- **Multi-Hop Scenarios (RL's Advantage):** Under multi-hop settings, the LLM learns to introspect missing information (Table 5 in the paper). This phenomenon explains why RL-driven refinement outperforms simply applying external refiners in multi-hop QA.\n\n[1] Learning to filter context for retrieval-augmented generation\n\n[2] Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions\n\n[3] InstructRAG: Instructing Retrieval-Augmented Generation via Self-Synthesized Rationales\n\n[4] Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection\n\n[5] Search-o1: Agentic search-enhanced large reasoning models\n\n[6] Search-r1: Training llms to reason and leverage search engines with reinforcement learning\n\n[7] An Empirical Study on Reinforcement Learning for Reasoning-Search Interleaved LLM Agents\n\n[8] InstructRAG: Instructing Retrieval-Augmented Generation via Self-Synthesized Rationales\n\n[9] Multi-step Retriever-Reader Interaction for Scalable Open-domain Question Answering\n\n[10] Multi-Hop Paragraph Retrieval for Open-Domain Question Answering\n\n[11] BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension"}]

Author rebuttal:

Meta-review: The paper attack an important problem, how to reduce cost of LLM annotation without losing much accuracy. The method is simple but novel and experiments are strong. In my view it is solid and useful, so I support accept.
Decision: Accept (poster)