DeepDiver: Adaptive Web-Search Intensity Scaling via Reinforcement Learning
Authors: Wenxuan Shi
Haochen Tan
Chuqiao Kuang
Xiaoguang Li
Hanting Chen
Xiaozhe Ren
Yasheng Wang
Lu Hou
Lifeng Shang
Venue: NeurIPS 2025 spotlight ()
Keywords: Agentic Reinforcement Learning
Retrieval Augmented Generation
Search Intensity Scaling
Data Synthesis
Primary area: deep_learning
TL;DR: Our paper introduces WebPuzzle, a novel dataset boosting LLMs' real-world info-seeking capability, and DeepDiver, an RL-based framework enabling dynamic Search Intensity Scaling for iterative evidence gathering.

Abstract: Information seeking demands iterative evidence gathering and reflective reasoning, yet large language models (LLMs) still struggle with it in open-web question answering. Existing prompting and supervised fine-tuning (SFT) methods remain fixed by prompt rules or training corpora, and are usually benchmarked only on well-structured wiki sources, limiting real-world adaptability. We introduce $\textbf{WebPuzzle}$, a 24k-sample training and 275-sample test benchmark that evaluates information seeking on the live internet, across both wiki and open-domain queries. Leveraging 7k WebPuzzle instances, we develop $\textbf{DeepDiver}$, a reinforcement-learning (RL) framework that cultivates $\textbf{Search Intensity Scaling (SIS)}$—an emergent ability to escalate search frequency and depth instead of settling on overconfident, under-evidenced answers. With SIS, Qwen2.5-7B-Instruct and Pangu-7B-Reasoner attain performance on real-web tasks comparable to the 671B-parameter DeepSeek-R1. We detail DeepDiver’s curriculum from cold-start SFT to a well designed RL procedure, and show that its seeking policy generalized from closed-ended queries to open-ended generation such as long-form writing. Our results advance adaptive information seeking in LLMs and provide a rigorous benchmark for future work.

DeepDiver: Adaptive Web-Search Intensity Scaling via Reinforcement Learning
Wenxuan Shi, Haochen Tan ; Chugiao Kuang, Xiaoguang Li, Hanting Chen, Xiaozhe Ren, Yasheng Wang, Lu Hou, Lifeng Shang
Huawei Language Model Lab
{wenxuan. shi, haochen. tann@euaweiicom
Abstract
Information seeking demands iterative evidence gathering and reflective reasoning. yet large language models (LLMs) still struggle with it in open-web question answering. Existing prompting and supervised fine-tuning (SFT) methods remain fixed by prompt rules or training corpora, and are usually benchmarked only on well- structured wiki sources, limiting real-world adaptability. We introduce WebPuzzle, a 24k-sample training and 275-sample test benchmark that evaluates information seeking on the live internet, across both wiki and open-domain queries. Leveraging 7k WebPuzzle instances, we develop DeepDiver, a wewdfforeemmt--laaaaiii (RL) framework that cultivates Search Intensity Scaling (SIS) an emergent ability to escalate search frequency and depth instead of settling on overconfident, under- evidenced answers. With SIS, Qwen2.5-7B-Inns Instruct and performance on real-web tasks comparable to the 67 B-parmmeetrr DeepSeek-R R1. We detail DeepDiver's curriculum from cold-start SFT to well designed RL procedure, and show that its seeking policy generalized from closed-ended queries to open-ended generation such as long-form writing. Our results advance adaptive information seeking in LLMs and provide rigorous benchmark for future work.
7B-Reasoner attain
Introduction
Information seeking (34] is a fundamental cognitive skill that involves iterative evidence gathering. reflective reasoning, and the resolution of conflicting information. Despite significant advancements in behaviors.
artificial intelligence, LLMs continue to struggle with replicating such information-seekinn snowledge-intensiig question answering. central challenge for L.Mss, requires robust capability for information seeking. Current models often fail to determine when and what information to seek, verify the relevance of evidence, and reason effectively over noisy or conflicting contexts.
Iterative Aatrivall-Augmeneer Generation (RAG) 15] frameworks have been proposed to address these challenges by alternating between retrieval and reasoning. Existing approaches generally fall into two categories: prompting based and task-specific supervised fine-tuning (SFT). Prompting- based methods leverage predefined rules or in-context learning (ICL) 3 forcing the LLM follow
16]. However, the fixed reasoning flow provided in the prompts limits their adaptability to complex, dynamic problems. In contrast, supervised fine-tuning methods train models to improve retrieval and reasoning capabilities [1) generally yielding better performance. However, these methods often internalize inference patterns tied to the training corpus, restricting generalization to more dynamic or unseen situations.
a specific pipeline to complete complex tasks [[]] 23
32
42
Recently, reinforcement learning (RL) 26 [K] has been applied enhance inference-time reasoning
in LLMs, enabling iterative refinement and exploration of reasoning 6 31 19). Several studies
'Equal contribution.
Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

have integrated RL into iterative RAG frameworks, encouraging models to explore diverse reasoning
paths and rewarding accurate outcomes [22 25 45]. However, these works predominantly train and evaluate their methods on well-structured datasets such as HotpotQA [39, which are based on corpora like Wikipedia. such settings, many tasks can be effectively solved using the LLMs' internal knowledge, and the introduced search environments are "clean," containing minimal noise or conflicting information. In contrast, real-world search environments are inherently more complex-characternne by noisy, inconsistent, and unreliable sources. This discrepancy limits the generalizability of the reported "incentivized search capabilities" to more realistic, open-ended iiformationnseeking scenarios.
To investigate RL-guided LLM behaviors in more realistic. open-domain scenarios, we introduce WebPuzzle, dataset designed to evaluate iooomatiion-seeinni capabilities in real-world search environments. WebPuzzle contains 24k training samples and 275 numan-annotated test examples, covering tasks solvable with Wikipedia content as well as broader open-domain queries extracted from open-web environment. Even Wikipedia subset are rigorously validated require external retrieval, ensuring realistic assessment of LLMs' search behaviors. Along with WebPuzzle, we introduce DeepDiver, an RL-driven search and reasoning framework trained on this dataset. DeepDiver interacts with real-world search engines, continuously refining and denoising retrieved documents to provide accurate answers. A key innovation of DeepDiver is the emergent capability of search intensity scaling (SIS), which dynamically scales up the search frequency and depth as information demands increase. This enables LLMs to tackle more complex, information-intensie open-web environment. Together, WebPuzzle and DeepDiver provide comprehensive framework for developing and examining information seeking ability of L.Mss, offering a promising approach for Bhowledge-intensiie problem solving.
problems under
Through systematic empirical analysis, we identify critical factors that influence model behavior, including the search intensity, the training environment, and the generalization capabilities. Our analysis reveals several key insights: (1) DeepDiver exhibits exceptional infoommttion-seeiing ability via adaptive SIS, where the depth and frequency of searching proportional to both problem difficulty and the model's performance. (2) Compared to the "clean" Wiki-based environment, WebPuzzle and real-world search settings better support complex reasoning bcahviours, guiding LLMs to actively supplement evidence, resolve conflicts, verify content, and reflect for self-correction. (3) RL training significantly enhances the generalization capability of LLMss, enabling the transition from closed-ended to open-ended problems. In conclusion, our method underscores the potential of reinforcement learning to foster emergent adaptive search behaviors specificilsllllllllly...... search intensity scaling-in LLMs. This significantly enhances their ability to perform adaptive, verifiable, and scalable information seeking, providing a promising direction for future advancements in knowledge- intensive problem solving.
2
Preliminaries
2.1
Iterative RAG
We formulate the iterative Oetrieval-Auugeentt Generation (RAG) framework for question answering. Given a question q, the model iteratively performs reasoning and retrieval to produce an answer.
At each iteration t E {1, T}, the model maintains a reasoning history H--1 (q, di),... (re-1, dr-1)}. where r; represents the intermediate CoT generated at round
i, denotes search queries, and d; denotes retrieved documents from web search.
At round I, conditioned on history H4-1, the model first generates intermediate reasoning r; ~ p(ra H(-1) to analyze the current status. Then, based the reasoning r,, the model selects one of two actions: (1) Search: generate additional queries s; p(5t H4-1, and retrieve supporting documents d, Retrieval(s)) (2) Answer: finalize the answer p(a H4-1;7r) to question g. This iterative
process continues until the model chooses the answer action, resulting in final answer that is well supported by retrieved evidence and explicit reasoning steps.

---

H
conflict
H2
-5t-1
H
conffict
D
Reason
Reason
Search
Reason
Reason
Search
Search
Search
H++1
Hj+1
He:1
Heis
(a)
(b)
(c)
(d)
Figure I: Illustration four key ceermaation-seeiingg behaviors: (a) Evidence Gathering
& Supple-
ments (b) Conflict Resolution (c) Verification & Denoising and (d) Reflection & Correction.
2.2 Information Seeking Behaviour
We define information seeking behaviour within iterative RAG frameworks structured decision- making process: at each iteration the model adopts specific strategies to resolve uncertainties, improve evidence quality, and enhance the overall reliability of answers. Formally, at iteration conditioned on the reasoning history H4-1 and current intermediate reasoning the model exhibits several strategies guide its search and reasoning processes.
Inspired by the findings Gandhi et al. 77. we categorize these strategies into four types of informa- tion seeking behaviours: (1) Evidence Gathering & Supplements, where the model actively seeks to fill identified knowledge gaps by formulating targeted queries s; and retrieving supporting documents d, formally represented as (st. d,) ~ pl st. de H4-1, where d, = Retrieval s,). This strategy exemplified by traditional question-ani anwwerrin datasets such 2Wiki (9. HotpotQA [39], and FRAMES [4]; (2) Conflict Resolution, where the model reasons about inconsistencies and evaluates competing claims when retrieved information contains contradictions; (3) Verification & Denoising, where the model cross-checks facts and isolates trustworthy information from noisy or irrelevant retrieved content; and (4) Reflection & Correction, where the model periodically re-assesses its reasoning trajectory, revisits earlier assumptions, and explicitly corrects previous reasoning steps for iterative refinement. The latter three behaviours can be represented generally as generating reasoning steps r. p(r: H4-1, dz), where the specific conditions (e.g. presence of contradiction, noise, or previous mistakes) differ according to each behaviour.
Existing works adopting the wiki-based datasets, limiting their scope to structured and well-organized knowledge bases, and thus predominantly emphasize "Evidence Gathering & Supplements". To prove this observation, show an detailed analysis in Appendix A.2 In contrast, proposed WebPuzzle and the real-world searching environment explicitly necessitates employing all four behaviours, thereby reflecting more comprehensive and realistic scenario for real-world problem-solving with web-searching. More details about WebPuzzle will be included in section 3.1
3 Method
this section, we discuss the details of our approach. We begin by introducing WebPuzzle, a dataset designed to address real-world reasoning and search challenges. Next, describe DeepDiver, reinforcement learning- based training framework aimed at enhancing LLMs with robust capabilities introduced in section 2.2
3.1 WebPuzzle
Unlike existing open-domain QA datasets based on Wikipedia [39 21
where LLMs often perform
well using only internal knowledge, we introduce WebPuzzle, dataset designed evaluate LLMs' ability to locate and reason over noisy, scattered information on the open web. Figure iillssrates our data synthesis and curation processes.
Candidate Data Generation We collect candidate data from Wiki-corpus and real-user queries with retrieved webpages from our deployed smart assistant service. Our generation involves two

---

Open Damain Webpages
Crocs Page Info Synthesis
Page (10k)
Open (7.5k)
CIA Generation
Riddle Generation
Wili Riddle [6k)
Wiki Pages
Entity Selection
Context Hiding
WebPuzzle Full Set
Deepßiver
Easy (8k)
Fany (200)
Training
WebPuzzle (21kk
Passgd k-based Difficulty Tagging
WebPuzzle
Med. (5 5k)
Med. 4kj
Sampling for DeepDiver
Hard (3k)
Hann (7.4k)
pass@k- 0
(3.5
NAA (1k)
Figure 2: WebPuzzle pipeline. Above: Candidate Generation: Wiki and open-web pages yield QA pairs via (i) Cross-Page QA and (ii) Riddle pipelines, grouped as Cross-Page QA, Open Riddle, and Wiki Riddle. Below: Difficully Tagging: Each sample is tagged (easy/medium/hhrd)) for adaptive mixing in RI; DeepDiver is trained on a curated 7k-sample mix.
approaches: (1) Cross-page question generation, where LLM extracts facts from web pages to generate "inverted" questions, answers and checklists (29 14 applied only to open web pages as Wiki-corpus tends to produce overly-simple questions; and (2) Riddle creation, where the LLM selects distinctive entity attributes and applies obfuscation or generalization to create challenging problems, with original entities as labels. Examples appear in Appendix [.I]] More quality assurance
protocols are shown in Appendix D.3
Difficulty Assessment To ensure stable RL training with consistent reward signals, tag each problem's ' difficulty level, enabling a data mixture strategy that prevents all-zero rewards which could lead to training collapse. For each problem, we test DeepSeek-RI four times, using the number of correct answers to determine difficulty. The formal definition appears in Appendix D.4] with the statistics of the dataset presented in Table and tagging workflow in Appendix E.6
Test Set Annotation Unlike the training set which used LLM labeling, our test set was manually annotated by 5 human experts using open-web search engine. From 500 seed samples, experts followed the principles in Appendix D.2 to ensure meaningful evaluation of LLMs' information- seeking behaviors. Through iterative annotation, finalized 275 samples for testing.
3.2 DeepDiver
Building upon the WebPuzzle, we showcase its efficacy within a RL framework designed to explore the information-seekin behavior of LLMSs. In this section, we present our method, DeepDiver. DeepDiver ultilize the procedure of cold-start supervised fine-tuning (SFT) followed by reinforcement learning (RL), while incorporates a carefully designed reward assignment and scheduling mechanism to maintain stable RL training.
Initialization of Reasoning and Searching To equip DeepDiver with essential reasoning and searching capabilities for WebPuzzle, we implement a cold-start supervised fine-tuning process using diverse data: 2,000 WebPuzzle samples across difficulty levels, 300 real-user questions from deployed smart assistant, 2,200 general reasoning problems from recent studies [10, 17
35 37 43),
and 1,000 real-user queries concatenated with retrieved documents. This dataset distills responses from DeepSeek-R1, establishing DeepDiver's foundational abilities to iteratively search and reasoning over retrieved documents. The distillation prompt configuration is detailed in Appendix E.6
GRPO With Iterative RAG After SFT, we enhance DeepDiver by extending GRPO 24 with iterative RAG. As shown in Figure 3 the model iteratively performs reasoning and searching until reaching an acceptable answer, following the pipeline in Section 2.1 We apply a loss mask to distinguish model- neder-generated from externaly retrieved tokens, with GRPO updating parameters based solely on model-generater content.
Extra Search Call Rewards Beyond standard format and accuracy rewards in GRPO, we introduce an extra reward to encourage search engine use for complex problems. When no search-free rollouts solve problem but at least one search-enabled rollout succeeds, assign additional reward of

---

Answer After Reason Search Loap
Loop
ground truth
query
Search
Grader
reasoning
BRSWEE
Update
dg
AO
Arnurory Reward
a2
Extra Search Keward
LLM
RG
Scheduler
(b)
Figure 3: DeepDiver overview. (a) Rollout Generation: DeepDiver iteratively reasons, retrieves evidence, and answers WebPuzzle queries, then receives rewards based on comparison with ground truth. (b) RL Updates: Retrieved text masked during loss calculation, and the LLM is refined via GRPO using advantages A; derived from rewards r,,
1.0 to the successful search- solutions. This ensures the model learns to leverage external tools when necessary. The formal definition appears in Appendix E.3
Loose and Striet Rewards Our reward function employs LLM-- graders in a two-stage training approach that transitions from loose to strict grading: the loose grader assigns scores from 1 to 10 (scores 2 6 yield 1.0 reward). partieularly benefiting early training as shown Section[5.4] The strict grader conducts three evaluation rounds, requiring at least 2 of 3 positive judgments. Both grader definitions appear Appendix E.2
Experiments
4.1
Setup
Data Mixture and Selection Due to computational constraints and capability limits of the 7B model, we train DeepDiver on a carefully selected mixture of 7k WebPuzzle samples rather than the full dataset. We evenly split these into 2k samples for cold-start SFT (Section 3.2) and 5k for RL training. This mixture strategy balances computational efficiency and model effectiveness. Detailed statistics appear in T
Benchmark Datasets and Baseline Models We evaluate performance using closed-ended Chinese benchmarks including C-simpleQA-5000 33 8. FRAMES-Zh-230 [14, BamBoogle-zh-71 21] and
our proposed WebPuzzle (detailed in Appendix E.4) For trainable baselines, we use Qwen2.5-7B-
Instruct [30] and 7e-aaseeeR (28 as backbone models. Training- baselines include QwQ 32B (3T) GPT-4o 20 and DeepSeek-RI [6). We evaluate methods including Prompted without Web Search, Prompted with Iterative RAG, and R1-Distillation (detailed in Appendix E.5). Our evaluation uses the strict grader from Section 3.2, which considers both reference answers and checklists for more robust assessment than conventional Ludgge---judge [4] approaches. Grader details appear in Appendix E.2
4.2 How does DeepDiver's Performance Compare to Baselines?
Our proposed DeepDiver demonstrates substantial improvements over iisiilatiionbbssee methods and achieves performance comparable state-of-the-ar models such as DeepSeek-RI and QwQ As shown in Table [] Qwen powered by DeepDiver achieves a 10-point improvement over the cold-start model on WebPuzzle, reaching 37.6 accuracy. DeepDiver-Qwen22.55 the RI Riddiiiileed model (37.6 versus 29.8), highlighting the effectiveness of our RL training pipeline.
also outperforms
5

---

Open- Web Problems
Wiki-based Problems
WehPuzzle
C-SimpleQA-500
FRAMES-2h-230
BamBoogle-zh-71
Prompted without Web Search (Traming-free)
Qwen2.5-7B-Ins.
744
28.4
19.7
Pangu-7B-Reasone
28
15.0
36.3
20.4
27.2
GPT-4o 20
14.2
61.8
51.7
52.6
QwQ-32B B1
21.9
51.3
36.5
54.5
DeepSeek-RI
i
32,7
74.6
63.8
73.2
Prompted with lterative RAG (Treining-free)
17.0
Qwen2.5-7B-Ing
(2.24)
(1.9)
(2.74)
(2.13)
Pangu-7B-Reasone
20.4
61.7
30.9
45.5
(3.80)
(1.87)
(2.09)
(2.41)
27.1
81.0
58.6
71.4
GPT-4a
(1.39)
(1.24)
(1.56)
(1.29)
31.4
79.0
50.4
73.2
QwQ-32B
(0.95)
(0.94)
(0.98)
(0.88)
37.1
84.8
65.8
79.3
DeepSeck-R1
(1.48)
(1.17)
(1.31)
(1.23)
Training with Qwen2.55778 Insturct Series
27.9
75.5
35.1
Cold- Start-SFT
(1.85)
(1.35)
(1.73)
(1.24)
29.8
78.7
40.1
52.6
RI-Distill
(1.75)
(1.32)
(1.56)
(1.34)
37.6
81.9
44.5
Deen.iver-Qwen255-88
(2.51)
(1.90)
(2.57)
(2.07)
Training with Pangu-7B-Reasoner Series
30.3
78.1
38.4
59.2
Sol
(1.84)
(1.37)
(1.84)
(1.49)
30.7
80.0
41.7
53.5
R1-Distill
(1.77)
(1.34)
(1.83)
38.1
83.7
52.3
69.5
DeepDiver-Pangu-7B
(2.89)
(2.61)
(3.05)
(2.72)
Table 1: Numbers in O indicate average search call rounds per example, with each round potentially involving 1-5 search queries. Results show average accuracy across 3 runs.
07
Call Round
Phase Change (Lonse Striet)
LLM Grader Reward
Tool Call Roand
6.1
Figure 4: Correlation between reward value and the number of search calls across training phases. The increase in the number of search engine calls is accompanied by rise in training rewards.
SeepDiver-Pangu- shows similar improvements. While R Rd-ditilledd Pangu-7B quickly hits perfor- mance bottlenecks (dropping 5.7 points Bamboggle compared the cold-start model), DeepDiver- Pangu-7B breaks through these limitations, showing substantial improvements across all benchmarks and achieving exceptional performance (38.1) on WebPuzzle. In conclusion, both Prras-eBrr and Qwen DeepDiver demonstrate competitive performance against tigh-performing models like DeepSeck-RR R1 and QwQ. This highlights DeepDiver's capability to effectively search for and reason over relevant information, and solve complex reasoning tasks through search intensity scaling.
4.3 What is the Relationship between Search Intensity and Performance?
Search intensity is strongly correlated with performance improvements, as increases in search frequency and depth during the RI. phase consistently lead to better oulcomes. FigureTTillustrat illustrates this relationship during the training phase, showing a clear trend: as search engine calls increase, so do training rewards. For testing results, despite SFT's progress compared with the prompting methods,

---

Deepilivrr DeepScek-Rt -Promppinggg
Deepliver QwQ-Prompting
DeepDiver CrT4o-Prompting
3.0
465
2,5
36.0
+2
Accuracy (1g
31.6
20/4
+20
1/4
22.8
164
173
116
WebPazzle-Fe 150 PRAMES 2h 56 BemBoogle
WebPazzle 154 FRAMES ab 111 BemBoogle 22
WebPazzle 209 FRAMES 2b 30 BamBloggle 26
Depthrettema Acarcyy
(pel--harqen Accuncy
Doup erjounn Toel Call Iloand
Gect--harpizz Roond
Depdickll (-Proggiing Acorscy
GPT Orrto-Prinn Priegting Aearney
Deep Sevls Prirpting That Call Roaad
GPT Printing Threl Roweal
Figure 5: The comparison after removing cases answered correctly through internal knowledge.
the model faces performance bottleneck to adapt to more challenging problems, still lagging behind off- tif-the-shelf APIs with large margin. In contrast, our RL-based DeepDiver Siaverr-wee2.5--1 promotes higher search intensity with average of 2.51 search and reasoning rounds, substantially higher than the SFT model's 1.75. Similar gains appear in SeepDiver-Pang--20 where increased search rounds (1.84 - 2.89) correspond to performance improvements (30.3 38.1). This searching intensity scaling enables models to explore and verify more relevant information, enhancing their ability to tackle complex problems.
4.4 Can DeepDiver Generalize from Open-web Training to OOD Wiki-based Problem?
Training with WebPuzzle, DeepDiver demonstrates strong generalization capabilities and perfor- mance improvements on Wiki-based problems. DeepDiver shows impressive generalization Wiki- based benchmarks despite not being specifically trained for these tasks. Both DeepDiver-Qwen2.55 and DeepDiver-Pangee
significantly outperform their distilled variants and demonstrate substantial improvements over cold-start models. While DeepSeek-R performs well on Wiki-based problems without web search, it shows modest gains when combined with iterative RAG pipeline. This suggests DeepSeek-R1 R1 may have already internalized the necessary knowledge for Wiki-based problems, highlighting the importance of our proposed WebPuzzle benchmark. We further investigate this hypothesis through isolated tests on information seeking and verification in section 5.1
Analysis
This section focuses on the Dwwn2.5-7BBIInsee model, simpler model comparing with the Pangu- 7B- IB-Reasoner. We analyze several key aspects, including isolated evaluations of information-seekin behavior, comparisons with concurrent related work, the design of the reward function, and the model's generalization to open-ended problems. Additional analyses-such as the relationship between search intensity and problem difficulty, statistics of information-seeking behavior across different training and testing environments, comparisons between human and DeepDiver performance, and detailed case studies-ae provided in Appendix A
5.1 Isolation Testing of Information-Seeking
While DeepDiver lags behind models such as QwQ and DeepSeek-RI on certain datasets in Section 4 primary focus is investigating informmtiion-eeeinng behavior rather than knowledge memorization. This raises question: When isolating evaluation to focus purely on information seeking ability, how does DeepDiver compare to strong baselines?
Setup conduct pairwise comparisons between DeepDiver and each baseline. For each pair, we perform A: 3 tests without web search to evaluate whether problems can be solved using internal knowledge alone. We calculate the pass&t rate to filter out problems solvable by both models, then analyze accuracy the remaining problems with the iterative RAG pipeline.
7

---

Results DeepDiver exhibits exceptional information-seekin capabilitiess comparable to all baselines on problems that cannot be solved by internal knowledge alone. While our 7B DeepDiver initially trails behind 671B baselines in full-set tests, results shift when isolating information-seeking behavior. As Figure 5 shows, on problems challenging even for larger models, DeepDiver demonstrates competitive performance across all benchmarks. Notably, it outperforms DeepSeek-RI R1 across all domains, with a -point lead on WebPuzzle. This suggests our 7B model's limitations in full-dataset performance stem primarily from its smaller size limiting internal knowledge. However, when tackling problems requiring external information search and verification, DeepDiver's 's information-seet -eekingg capability demonstrates strength in addressing real-world open-web problems.
5.2 Comparisons with Wiki-based Methods
To highlight wiki-based training environments' limitations, we compare DeepDiver with prior wiki- based methods. Despite being trained entirely in Chinese, we evaluate DeepDiver on English benchmarks with English search engines to demonstrate its robustness and generalizability.
Open-Wel Web Probtems
Wiki-based Problems
WebPuzzle-en
BamBoogle
FRAMES
HotpotQA
RI-Searcher 25
13.7 (1.9)
46.7 (2.0)
25.3 (1.9)
57.9 (2.3)
DeepResearcher 45
15.0 (7:5)
53.9 (7.1)
33.6 (7.2)
56.6 (4.4)
DeepDiver-Qwen
26.1 (14.7)
56.8 (9.1)
32.0 (14.2)
58.4 (10.4)
Table 2: The comparison results with relevant works on the English evaluation dataset using English search engine environment. The number in () indicates the average number of search queries invoked.
Setup We use RI-Searcher [25] and DeepResearcher 445 as baselines-bool trained in English using Wiki-based corpora. Search engine settings appear in Appendix [E.8] For evaluation, translate WebPuzzle into English via Qwen Max [30], use the full Bamboogle dataset [21] (125 examples), and randomly sample 300 examples from FRAMES [14] and HotpotQA (39]. For fairness, we report accuracy based on the average judgment across all methods.
Results Despite the language gap, DeepDiver-trained in real-world Chinese internet setting using WebPuzzle queries-outperfonss Wiki-based baselines on most tasks, underscoring the strength of SIS. As shown in Table DeepDiver significantly outperforms DeepRescarchers WebPuzzle-en with 11.1 point leads, while maintaining strong results on Wiki-based datasets despite no English training for information-ssekking We attribute this to DeepDiver's use of SIS, which enables intensive information retrieval and verification rather than relying on limited internal knowledge or language- specific constraints. R1-Searcher and DeepResearcher make significantly fewer search calls than DeepDiver due to their "cleaner" and more constrained training environments, leading to poorer real-world performance when facing the noise and complexity of open information-seeking tasks. For additional results, including individual judge assessments, see Table
5.3 Emergence of the SIS
natural concern that SIS could be an artifact of reward shaping rather than genuine behavior that emerges from training in real web environment. To address this concern, we analyze whether the extra search-call reward introduced Section E.3 consistently encourages the model to prefer search over no-search when both solve the task.
Setup Theoretically, recall the bonus is only awarded when at least one search-enabled rollout succeeds and no search-free rollout succeeds for the same prompt group, i.e., it should not reward search when both search and no-search solve the task. Consequently, during training, we tracked the frequency of the extra bonus. Every 10 steps (448 trajectories per step), counted how often the bonus fired and compared this against the evolving search intensity (average number of tool-use rounds per query).
Results SIS is not merely reward-shaped but emergent behavior developed during RL training, our reward design serves as a transient early scaffold (not persistent incentive) and differs funda- mentally from existing works that explicitly encourages the tool calls. As shown in Figure 6 the
8

---

Search Rewards Distribution Across Different Step Ranges
3.1%
Num
2.3%
100
1.1%
1.0%
0.6%
0.1%
0--9
10-19
20-29
30-39
60-69
70-80
Step Range
Figure 6: The count and percentage of trajectories triggering the special auxiliary search reward (value=3.0) across training step ranges. The trigger frequency drops sharply (4.5% 0-9 steps to 0.1 % in 70-80 steps) and fades after step 30, confirming the reward's transient role and supporting SIS as an emergent behavior.
(socuracy)
FRAMES d5-230 (accaraes)
(trrBreglodh-ii (acumaey)
FRAMES-B Gool alll round) DamikoopkorhhT1 (boal all mund)
592
275
2017
50-
43.5
2
44.1
60
2.00
00 Ca
30-
29.1
130
30.2
20
146
1.50
125
step80-loose
1ep110-loose
step1 20-siit
Figure 7: Training results with different reward functions show that looser reward function stabilizes initial RI. training, while a stricter reward function helps overcome later bottlenecks.
its trigger percentage drops from 4.5% (198 occurrences) in steps 0-9 to 0.1 % (6 occurrences) in steps 70-80. Two key observations confirm its transient role: (1) Even the earliest training phase, only small fraction of trajectories (4.5% received the reward, ruling out "over-rewarding". (2) After step 30, the trigger percentage remains = 1.1%, indicating the reward is essentially inactive mid-to-late training. More importantly, Figure 4 shows that tool-use rounds grow sharply during steps 80-120-well after the auxiliary reward Tades. This confirms SIS is not reward-driven "phenomenon" but an emergent behavior: the model proactively leverages external tools to compensate for internal knowledge limitations, even without direct incentives.
special search reward exhibits a clear phasing-out trend:
5.4 Tolerance of the Reward Function
During ReepDiver-Qwen7ee s RL training, we observed a reward plateau after approximately 80 optimization steps. We investigated potential factors including learning rate scheduler, exploration diversity, environmental instability, and gradient issues, but found no obvious problems. We therefore focused the reward function design potential cause of the performance plateau.
Setup Starting from checkpoints obtained after 80 optimization steps, we compared DeepDiver's performance under continued training with two different reward functions: the loose and strict

---

rewards introduced in Section 3.2 performance on WebPuzzle test sets, analyzing accuracy and search intensity trends.
Both guided continued training from steps 80 to 120. We evaluated
Results A looser reward functionn stabilizes the initial training phase of RI,, while a stricter reward function helps overcome bottlenecks in the later stages. Our results show that looser reward function stabilizes early RL training, but continuing with it doesn't always lead to improvements. As Figure 7 shows, when transitioning from loose rewards (first 80 steps) stricter rewards, we observed nearly 9-point performance increase on WebPuzzle (from 29.1 1 to 37.6), compared to almost no improvement when continuing with loose rewards. On FRAMES zh-230, continued training with loose rewards caused a sharp 7-point performance drop, while the stricter reward function continued driving performance upward.
5.5 Generalization to Open-ended Problems
DeepDiver trained exclusively on closed-ended WebPuzzle problems, adaptively scaling search intensity based on complexity. We investigate whether these capabilities can generalize to open-ended tasks like long-form writing.
Setup We evaluate DeepDiver on ProxyQA [27] against R1-Distilled baselines. Since DeepDiver generates Chinese responses, we translate all ProxyQA meta-questions and sub-questions for evalua- tion. Testing prompt and evaluator config configuration follow the original study. We analyze generalization benefits gained through RL training compared to distillation.
Results RL training significantly enhances the generalization capability of LLMs, enabling transition from closed-ended to open-ended problems and demonstrating strong adaptability to long-form writing tasks, As shown in Table 6 our RL-guided DeepDiver achieves 32.72%, outperforming the RI R1-distilled model by 9.47 percentage points. This suggests RL training enables more effective information seeking and validation in open-web environments, resulting in more comprehensive responses. Additionally, DeepDiver's 's response length and search queries substantially higher than the distilled model's, 's providing evidence that search intensity scaling encourages active information acquisition for more comprehensive answers.
Related Work
Prompting- braep thought pipelines (1 enbbble zero- few-shot question answering, yet their fixed templates rarely adapt retrieval depth to unforeseen information gaps. Supervised aieettnnig (SFT) improves the synergy between retrieval and generation 2 41) but can overfit corpus-specific inference patterns, hindering transfer to noisy settings. Reinfoncement-leariig (RL) methods let L.MSs decide when and what search, achieving sfate-of-the--rt results curated benchmarks such HotpotQA [12 25 4
in-context learning and retrieval-augmentt chain-of-
[5] 39
yet they remain evaluated mostly in "clean" Wikipedia-style environments. Beyond these
directions, tool-augmented agents that interleave reasoning with web search [18 40 38) similarly
demonstrate promise but still rely on limited test beds, underscoring the need for benchmarks that reflect real-world, noisy iiformationnseekinng scenarios. Additional introduction of the related works
are shown in Appendix B
7 Conclusion
We conducted a comprehensive investigation into various aspects infformation-seekinn behavior in LLMs for solving real- real-world, wowleege-intensive problems. Our findings indicate that an RL-driven framework, when combined with open-web search engines, enables LLMs to scale search intensity and adapt to tasks of varying difficulty levels. We introduced WebPuzzle, a large-scale dataset designed specifically for developing and testing LLMs' information-seekinne behavior, and DeepDiver, a 7B parameter LLM powered by WebPuzzle, which demonstrates competitive performance when compared to the 671B DeepSeek-RI model on knowledge-intensive tasks. Additionally, we explored key factors influencing RL training and the behavior of LLMs. Our work empowers LLMs to spontaneously adapt their seeking behavior, contributing to advancements in the field and providing extensive insights into the information-scckinn capabilities of LLMs in real-world tasks.
10

---

References
[1]
Anthropic. Introducing the model context protocol. November 2024. URL https://www /ttp:////o//////oooooooooottte//
Accessed: 2025-04-13.
[2] Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannanch Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection, 2023. URL https://arxiv org/
abs/2310.11511
[3] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020. htttt::////attti org/ ab00 2005. 14165
[4] Mingyang Chen, Tianpeng Li, Haoze Sun, Yijie Zhou, Chenzheng Zhu, Haofen Wang, Jeff Pan. Wen Zhang, Huajun Chen, Fan Yang, Zenan Zhou, and Weipeng Chen. Research: Learning to reason with search for Ilms via reinforcement learning, 2025. URL /tttts:://aaati org/
abs/2503. 19470
[5] Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang. Zhaopeng Tu, Haitao Mi, and Dong Yu. Do not think that much for 2+3=? on the overthinking of ol-like Ilms, 2025. URL.
/ttps://arxiv.. org/ 22122 21187
[6] AL., Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang. Jingehang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li. Miaojun Wang. Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang. Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li. Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei. Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingehao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen. Xiaowen Sun, Xiaoxiang Wang, Xinnan Song. Xinyi Zhou, Xianzu Wang. Xinxia Shan, Y. K. Li, Y. Q. Wang. Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang. Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma. Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in 1lms via reinforcement learning, 2025. URL htttps////aax wwwerrvvvo.g/g// www./bg.///....../..../..eeeee// /501 12948
[7] Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D. Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars, 2025. htLhstts::. //tpx//axvvv.///////a/ org/ aws//b/s.... 013007
11

---

[8] Yancheng He, Shilong Li, Jiaheng Liu, Yingshui Tan. Weixun Wang. Hui Huang. Xingyuan Bu, Hangyu Guo. Chengwei Hu, Boren Zheng, Zhuoran Lin, Xuepeng Liu, Dekai Sun, Shirong Lin, Zhicheng Zheng, Xiaoyong Zhu, Wenbo Su, and Bo Zheng. Chinese simpleqa: A chinese factuality evaluation for large language models, 2024. URL https://arxiiii org wrg.ab/////// 07140
[9] Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps, 2020. URL https
7/arxiv. org/ farg/abu abs/ /201..21..... 01060
[10] Jinhao Jiang, Zhipeng Chen, Yingqian Min, Jie Chen, Xiaoxue Cheng, Jiapeng Wang, Yiru Tang, Haoxiang Sun, Jia Deng, Wayne Xin Zhao, Zheng Liu, Dong Yan, Jian Xie, Zhongyuan Wang, and Ji-Rong Wen. Enhancing Ilm reasoning with seward-guide tree search. arXiv preprini arXiv:2411.11666 2024.
[11] Zhengbao Jiang, Frank F. Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi- Yu, Yiming Yang, Jamie Callan, and Graham Neubig. Active retrieval augmented generation, 2023. URL https:/ arxiv.iiivv..... org/ abs 2als//335..0ee 06983
[12] Bowen Jin, Hansi Zeng, Zhenrui Yue, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training Iims to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.095/6. 2025.
[13] L. P. Kaclbling, M. L. Littman, and A. W. Moore. Reinforcement learning: A survey, 1996. URL https: org/abs/////oooooooooo..oooeeer/ abs/ cs/960500
[14] Satyapriya Krishna, Kalpesh Krishna, Anhad Mohananey, Steven Schwarez, Adam Stambler, Shyam Upadhyay, and Manaal Faruqui. Fact, fetch, and reason: A unified evaluation of retrieval-augmented generation, 2024. URL httpt://///aaiii wwaaxivv.or//re httgg/bbs/4409// 12941.
[15] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmeente generation for knowledge-intensive nlp tasks, 2021. URL 11401
[16] Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. Search-ol: Agentic search-enhanced large reasoning models, 2025. URL httpaaawwww...ra/////..1..11 05366
[17) Yinggian Min, Zhipeng Chen, Jinhao Jiang. Jie Chen, Jia Deng. Yiwen Hu, Yiru Tang, Jiapeng Wang, Xiaoxue Cheng, Huatong Song, Wayne Xin Zhao, Zheng Liu, Zhongyuan Wang, and Ji-Rong Wen. Imitate, explore, and self-improve: A reproduction report on slow-thinking reasoning systems. arXiv preprint arXiv:2412.09413 2024.
[18] Relichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browser-assisted question-answering with human feedback, 2022. URL http://wwww...r////a///111.2/..0
[19] OpenAl. Learning to reason with 1lms, 2024. - to- www.rressontth--l-e
URL https://openail ctt.iieeoom//iii.....e/a
[20] OpenAl. Hello got-4o. httpps///ppeaaa..commnneee//////
2024.
[21] Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models, 2023. URL https //arxiv. bbgg/bbs//ie /2100. 03350
[22] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL httpst:s::/ pestps://arxia //arxiv org/abs/ 1adbs/100/ 06347
12

---

[23] Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen. En- hancing retrieval-ane -uggenttd large language models with iterative retrieval- -enerratonn synergy. 2023. URI.httpe //tp:///araaaa //axxviivvv.. org/ absaaa/s../// 15294.
[24] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL (tRlpsttts:// paatps::/arxivy org/ /bs/
2402.03300)
[25] Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang. and Ji-Rong Wen. R R1-searcher: Incentivizing the search capability in llms via reinforcement learning, 2025. URL
org/ hwngg/abs/2/e abdp//a/////0/.../ 05592
[26] Richard S. Sutton and Andrew G. Barto. Reinfforcement Learning: An Introduction. The MIT Press, second edition, 2018. URL http:///inccappeeecdeeesseeettts www/booookkhhebooooooooooooooo-e 2nd html
[27] Haochen Tan, Zhijiang Guo. Zhan Shi, Lu Xu, Zhili Liu. Yunlong Feng, Xiaoguang Li, Yasheng Wang, Lifeng Shang, Qun Liu, and Linqi Song. Proxyqa: An alternative framework for evaluating fong-form text generation with large language models, 2024. URL org/ arg/abs/2400 2aas/2241... 14012
[28] Ychui Tang, Fangeheng Liu, Yunsheng Ni, Yuchuan Tian, Zheyuan Bai, Yi-Qi Hu, Sichao Liu, Shangling Jui, Kai Han, and Yunhe Wang. Rethinking optimization and architecture for tiny language models. arXiv preprint arXiv:240..02791 2024.
[29] Yixuan Tang and Yi Yang. Multihop-ragg Benchmarking retrieval- generation for multi-hop queries, 2024.
[30] Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL https:// quenlm. qwenlm.github.ue www.ggtthhu..... wtt.gii......oogg///oo qwtp///wen2//////
[31] Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025. URL https://qwennm... hwt.:lmm.ithubbb...m wtt.:hbb....oooogggg--3-30 quq-3/////
[32] Harsh Trivedi, Niranjan Balasubramanian. Tushar Khot, and Ashish Sabharwal. Interlcaving retrieval with chain-of- reasoning for tnowledge-intensive multi-step questions, 2023. URL htttpp::/aaxxv.orr/////////
2259.10509
[33] Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. Measuring short-form factuality in large language models, 2024. htRpsttss: / //wpxvvvvvvv///////////aa///aaaa www./ogg//////////oo/.oooeeeeeet 0abb///41..440
[34] Thomas D. Wilson, Models in information behaviour research. Journal of Documentation, 55 (3):249-270, 1999. doi: 10. 101100//EUM000000000007111 URL https://doi. org 1trg/10.10 1108/ EUM0000000007145
[35] Guohai Xu, Jiayi Liu, Ming Yan, Haotian Xu, Jinghui Si, Zhuoran Zhou, Peng Yi, Xing Gao, Jitao Sang, Rong Zhang, Ji Zhang, Chao Peng. Fei Huang, and Jingren Zhou. Cvalues: Measuring the values of chinese large language models from safety to responsibility, 2023.
[36] An Yang, Baosong Yang, Binyuan Hui, Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan. Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report. arXiv preprint arxiv:2407.10671, 2024.
[37] Jianxin Yang. http:://gtttabo/////gaaaaaaaaaaa
2023.
13

---

[38] Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge. Xiu Li, and Ying Shan. Gpt4tools: Teaching large language model to use tools via self ceff-instruction, 2023, URL https://arxiv org/ hts/abs/23000 18752
[39] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhut- dinov, and Christopher D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering, 2018. URL https: pettps//arriv org/ htt///bs/180// htt.//8899000000 09600
[40] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models, 2023. URL https //arxiv.org/abs/ /2100. 03629
[41] Tian Yu, Shaolei Zhang, and Yang Feng, Auto-rag: Autonomous retroooalgeetiiiiiaacaanmmmeee generation for large language models, 2024. URL https:://txiiiii htttts///aaxii..or//a////////// 19443
[42] Zhenrui Yue, Honglei Zhuang, Aijun Bai, Kai Hui, Rolf Jagerman, Hansi Zeng, Zhen Qin, Dong Wang, Xuanhui Wang, and Michael Bendersky. Inference scaling for long-context retrieval augmented generation, 2025. URL httpss://axii:> www./ogg///////oooooo/.......e// abaaab//// /2200 04343
[43] Ge Zhang, Yemin Shi, Ruibo Liu, Ruibin Yuan, Yizhi Li, Siwei Dong. Yu Shu, Zhaoqun Li, Zekun Wang, Chenghua Lin, et al. Chinese open instruction generalist: A preliminary release, 2023.
[44] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang. Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. URL URL https://arxii org/ abs/ (iee2366000685.
[45] Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. Deepresearcher: Scaling deep research via reinforcement learning real-world environments. arXiv preprint arXiv:2504.03/ 160, 2025.
14

---

Appendix
A Further Analysis
A.1 Search Intensity Difficulty
To showcase DeepDiver's 's ability to dynamically adjust search intensity based on problem complexity, we examine the relationship between search intensity and accuracy across fractions with varying difficulty levels in the WebPuzzle. This analysis follows the experimental setup outlined in Section and compares our proposed DeepDiver with the DeepSeek-RI baseline.
WebPuzzle
Methods
Cross-Page QA-130
Open& Wiki Reddle-145
Easy&Medium-96
Hard& Hard&Outliers-17
DeepSeek-ER R1 (w/o search)
32.6 (0.00)
32,9 (0.00)
53.5 (0.00)
21.6 (0.00)
DeepSeek-RP RI Iterative RAG
43.8 (1.31)
31.0 (1.64)
61.1 (1.30)
24.2 (1.58)
Owen7b-Ins-RI-Dine In----iiitt
37.2 (1.49)
23.2 (1.99)
45.2 (1.62)
21.6
DiepDiver (Ours)
47.4 (2.35)
28.8 (2.65)
55.6 (2.34)
27.9 (2.60)
Table 3:
The performance of different subsets in WebPuzzle. The number in () indicates the average number of search call rounds on the subset.
Results DeepDiver demonstrates significant benefits from adaptive search intensity scaling, where search intensity is proportional both problem difficulty and the L.M.'s 's performance. shown in table[]] across all difficulty levels, both DeepDiver and the baseline models show an increasing number of search call rounds as problem complexity rises. However, DeepDiver consistently consumes more search calls, which translates into better performance. In particular, when compared to DeepSeek-RI, DeepDiver outperforms it in the hard and outlier fractions by a large margin. Specifically, DeepDiver achieves notable 3.7-point performance leading, driven by average of 2.6 search rounds compared to DeepSeek-R1 R1's 1.59. This demonstrates that DieeDiver, empowered by open-web environment and reinforcement learning, exhibits superior performance more complex problems.
interesting observation arises when examining the performance of DeepSeck-RI on the Wiki Riddle fraction. Although equipped with the iterative RAG pipeline, DeepSeek-RI experiences 1.9-point performance drop (from 32.9 31.0). We hypothesize that this decline is due knowledge conflicts between the pre-trained internalized Wiki corpus and the real-world open-web environment, which introduces confusion and hallucination that hinders the model's s ability to correctly answer the question. These results further validate the effectiveness of the WebPuzzle and DeepDiver.
A.2 Statisties of Information-Seeking Behaviors
To further analyze the differences between our WebPuzzle dataset and wiki-based datasets, well as to conduct an in-depth investigation into the behavior of our DeepDiver model, we computed the proportions of different information-seeking behaviors (defined in section 2.2) across various models on multiple datasets. The detailed results are presented in Table
Methods
WebPuzzle-en
BamBoogle
FRAMES
HotpotQA
R1 R1-Searcher
0.04
0.03
0.04
0.02
Reflection & Correction
DeepRescarcher
0.20
0.11
0.13
0.07
DeepDiver-Qwen
0.45
0.25
0.28
0.27
RI-Searcher
0.06
0.02
0.02
0.02
Conflict Resolution
DeepResearcher
0.16
0.07
0.08
0.06
DeepDiver-Qwen
0.31
0.23
0.32
0.18
R1-Scarcher
0.17
0.18
0.13
0.10
Verification & Denoising
DeepResearcher DeepDiver-Qwen
0.32
0.15
0.24
0.18
1.72
1,80
1.60
1.54
Table 4: Behavior statistics of multiple models on our WebPuzzle dataset and several wiki-based
datasets. Each value the table represents the average occurrence count of rollout-level behavior.
15

---

Setup We developed an automatic pipeline based on GPT-4o [20] to identify and count the oc- currences of different behaviors in the reasoning chains of model outputs. We primarily focused on three behaviors: Reflection & Correction, Conflict Resolution, and Verification & Denoising. while omitting the statistics for Evidence Gathering & Supplements due to its overly fundamental nature. Detailed prompt specifications can be found in Figure T6. The baseline models, evaluation benchmarks, and analysis setup align with those in Experiment 5.2
The WebPuzzle proves to be more challenging compared to wiki-based datasets, requiring more complex cirrmattion-eeeiiie behaviors. Additionally, DeepDiver's reasoning chains exhibit richer and more diverse patterns. As shown in tabel[4] The behavioral statistics of WebPuzzle-or surpass those of wiki-based datasets across nearly all models, particularly in Reflection & Correction and Conflict Resolution behaviors. Notably, for the HotpotQA dataset the primary training data for both baseline models- we observe substantial reduction the proportion of complex behaviors required compared to WebPuzzle when evaluating the same model, further highlighting the impor- tance of more open-web training data. Additionally, DeepDiver, trained on WebPuzzle with real search engine integration, demonstrates richer and more sophisticated information-sseking behaviors compared to various baseline models.
Results
A.3 Human DeepDiver
To assess the difficulty of WebPuzzles and compare human performance with that of DeepDiver, we conducted human testing on a subset of the WebPuzzle evaluation dataset. Human performance was evaluated by tracking the number of web-search queries and web pages browsed during the problem-solving process, which were then compared with the performance of DeepDiver.
Setup Five human experts, who were not involved in the annotation of the evaluation sets, partici- pated in the human evaluation. Each expert was tasked with answering 5 problems, creating a subset of the evaluation set. During testing, the number of search queries and web pages browsed were recorded. Consistent with the experimental setup, we report the average accuracy of the human evalu- ators and compare it against DeepSeek-RR and our proposed DeepDiver highlight the challenges posed by WebPuzzle.
WebPuzzle
Accuracy
# Search Rounds
# Search Queries
# of Page Browsed
Human
44.0
6.16
9.28
GPT-4o
30.7
1.47
5,28
QwQ-32B
27.0
0.95
3.77
RI-Distill
38.7
1.81
6.79
DeepSeek-R1
26.7
1.94
7.37
DeepDiver 40.0
2.68
10.69
Table 5: Human evaluation results for 25 randomly sampled questions from WebPuzzle evaluation dataset, compared with baseline models and DeepDiver.
Results WebPuzzle presents significant challenges, even even human experts, who face difficulties with numerous searches and reasoning steps. As shown in Table human evaluators achieved an accuracy rate of 44.0%, requiring an average of 6. 16 search queries and browsing 9.28 web pages solve the problems. comparison, DeepSeek-RI made 7.37 search queries with 1.94 rounds of searching. Despite DeepSeek-RI R1 performing the most search rounds among all baselines and humans, it achieved relative lower accuracy of 38.7%. In contrast, our proposed DeepDiver conducted 10.6 search queries across 2.68 rounds, significantly surpassing the other methods in terms of search effort. This additional effort allowed DeepDiver to more thoroughly collect and verify evidence, leading to a 40.0% accuracy rate, which is closer to human performance. These results suggest that DeepDiver follows more comprehensive cormattion-sekknng process, better aligning with the approach taken by human evaluators.
16

---

A.4 Case Study
In this section, we conduct an error analysis and case study on the response generated from DeepSeek- RI, RI Rd-iistileed ob and our DeepDiver. We explain the reason why the DeepDiver outperforms the RI-distilled model and show competitive performance compared with the DeepSeek-rl using one typical example.
Results DeepDiver demonstrates exceplional information-seeking ability while incorporating
correction on reasoning history and retrieved documents, providing a more robust and aduptable solution for overcoming the limitations of flawed internal knowledge. Specifically, as shown in Table DeepSeek-R leverages its rich internal knowledge to quickly narrow the exploration scope, consistently demonstrating the ability to list the correct answer among candidate options in the first round, showcasing remarkable knowledge retention. Taking advantage of this, R can focus more on verifying whether candidate answers satisfy all constraints when designing search queries, allowing it to find the correct answer in fewer rounds. However, the R Risistileed Qwen-7B 7B attempts to mimic DeepSeek-R1's behavior but lacks error correction when internal knowledge is flawed. Specifically, the RI-distilled Qwen suggests "Nico Hülkenberg" the first round of searching and reasoning, but fails resolve conflicting conditions in subsequent rounds due limited internal knowledge, ultimately producing a faulty answer.
In contrast, without RI-level internal knowledge, DeepDiver compensates by increasing search intensity to acquire more relevant external documents. This results in 7 generated search queries across rounds and 2 to explore diverse documents, rather than relying on limited internal knowledge. Furthermore, in rounds 2-3, DeepDiver encounters potential candidate, "Chaz Mostert," but allocates only query for validation, compared to 3 queries for continued exploration. This persistent exploration enables the model to identify the correct answer by round 3. The search and reasoning strategy, empowered by SIS, aids DeepDiver in delivering correct and acceptable answer.
A.5 Results on ProxyQA
We evaluate DeepDiver ProxyQA 27 against R1 R1-Distilled baselines. Since DeepDiver generates Chinese responses, we translate all ProxyQA meta-questions and sub-questions for evaluation. Testing prompt and evaluator configuration follow the original study. We analyze generalization benefits gained through RL training compared to distillation.
ProxyQA
Acc.
# rounds
# searches
response length
RI-Distill
23.25%
1.76
7.39
590.31 1971.58
DeepDiver 32.72%
2.27
10.54
Table 6: Results on ProxyQA. The results include accuracy rate, number of search calls, rounds of search, the number of search queries, and response length.
B Additional Related Works
Prompting-basee strategies. Few-shot in-context learning 3 allows frozen LLMs to imitate reasoning patterns from exemplars, while RAG with CoT prompts [11 32, 23 interleave search queries with intermediate thoughts to inject fresh evidence. These methods require no training, but their step counts and query formats are anchored to the prompt, so the model cannot escalate effort when initial evidence is missing. As result, they often stop searching too early or hallucinate unsupported facts hard, open-web questions.
Supervised fine-tuning (SFT). Retrieval-augnee augmented generation systems fine-tuned gold pas- sages-e.g.. Self-RAG and Auto-RAG 2 -eam to quote and merge the external text into answers, reducing hallucinations inference paradigm tied the training corpus; performance drops when pages are noisy, multilingual, partially missing, which is common in open-web internet environment.
Wikipedia benchmarks. Nevertheless, SFT tends to overfit the
17

---

Reinforcement learning (RL). RL integrated iterative RAG pipeline offers principled way for LLM agents to decide when and what search. Recent work trains search-capable agents with reward shaping and curricula, achieving tfate-of-the-are accuracy on HotpotQA and similar wiki- based corpus [2
45 Yet these studies use closed Wikipedia environments, where every answer is guaranteed to exist and pages are clean.
Tool-augmented agents. ReAct-style frameworks combine chain-of-thoughee with executable ac- tions, letting models call a browser other tools between reasoning steps [18 40 38). They excel at citing fresh evidence and correcting themselves trajectcy, but they still struggle with irrelevant pages and and hard to dealing with the real-world open-web environment.
Group Relative Policy Optimization Group Relative Policy Optimization (GRPO) 24 a reinforcement learning algorithm designed enhance the efficiency of Proximal Policy Optimization (PPO) [2 by eliminating the need for critic network. Specifically, GRPO samples multiple outputs from previous policy for given prompt and computes their average reward to baseline. The advantage of each output is defined relative to this baseline, resulting in positive advantages for outputs exceeding the baseline and negative advantages otherwise. Formally, define the GRPO objective as follow:
dynamic
TT(o))
To(us) Te (o,) e,1 (t)a))) BTx (To ll moa)
JGRPO (0) = =E Σ min
As clip
(1)
Tenu (o.)
where the relative advantage A; is computed as:
A, = " mean((r F2., std( 72. .2....,.a))
(2)
GRPO retains PPO's clipping strategy and incorporates KL-divergence term for regularization to ensure the stable training of RL.
C Discussion: Limitations and Extensions
In this work, we aimed to provide a deeper understanding of the information-seeking behavior of large language models (LLMs) with respect to real-world open-web challenges. While we observed that DeepDiver outperforms multiple baselines in several cases, we cannot claim that DeepDiver the most optimized solution for enabling LLMS to solve complex problems in broad sense. Below, we outline several limitations our work and potential areas for future exploration.
Curation of WebPuzzle We argue that DeepSeek-RI excels at solving problems by leveraging internal knowledge rather than extensively exploring the internet. As a result, we designed the WebPuzzle to assess and test the LLM's information-scekin abilities. However, over time, there a possibility that cutting-edge LLMS will internalize the knowledge sources used in WebPuzzle, which could lead scenarios similar to those in Wiki-based problems. Moreover, the curation pipeline heavily relies on the utilization of DeepSeek-R11 introducing some potential bias in the testing process. Therefore, developing effective benchmarks to assess the evolving capabilities of LLMs will remain an ongoing challenge.
RL-Driven Open-ended Problems Solving We demonstrate that DeepDiver, trained on closed- ended open-web problems, exhibits strong generalization capabilities when applied to open-ended tasks, such as long-form writing. However, developing more effective RL-driven framework that directly enhances the model's ability to solve these problems remains an open challenge. Since the RL framework relies on a stable reward signal, the lack of reliable metrics to assess open-ended content complicates the task of defining such signals. As result, designing framework that can handle both open-ended and closed-ended problems remains a key area for future exploration. One potential approach could following the approaches of ProxyQA, involving crafting sub-questions in an online way during RI training, which may offer promising direction for enhancing the model's performance open-ended tasks.
18

---

The Border Between Cold Start SFT and RL DeepDiver is powered by cold-start SFT and RL pipeline, which first initializes the model's capabilities through SFT training, followed by RL training. However, there is no established guideline on the optimal extent to which SFT should be conducted before transitioning to RI. training. This presents a challenge, as maintaining stable and effective RL training requires researchers to experiment with various proportions and combinations of SFT samples, while continuously monitoring the RL training process. Future improvements could involve an adaptive pipeline, allowing the LLM to dynamically switch from SFT to RL training when necessary.
The Extension of Tool Usage Search engines are commonly viewed as tools that LLMs can utilize to enhance their capabilities. In our study, we focused solely investigating the reasoning and searching behavior LLMs with the aid of search engines. This limits the scope of our work, as there various other tools, such as those compatible with the Model Context Protocol (MCP) M. which could also contribute to improving reasoning and searching processes in knowledge-intensinn tasks. Future research could expand this by considering the integration of additional tools to further enhance the performance LLMs such tasks.
Scalability with Respect to Model Size and Sequence Length Due to computational constraints, the experiments presented in this report limited to 7B model with maximum sequence length of 20k tokens. This limitation restricts the generalizability of our findings to larger models (e.g.. 13B, 32B) and longer sequence lengths (e.g.. 32k, 64k), and also limits exploration of DeepDiver's upper performance boundaries. Currently, DeepDiver's performance is predominantly constrained by both model size and response length. Extending the training and evaluation encompass these configurations in future work would provide valuable insights into the model's scaling behavior. Such an extension would help assess DeepDiver's performance, robustness, and applicability across different model sizes and sequence lengths, offering more comprehensive understanding of its real-world potential.
Problem of Over-searching Prior work has shown that reasoning LLMs trained in RL environments often suffer from "overthinking" [5]. generating excessively long reasoning sequences sometimes spanning thousands of tokens even for simple questions. We observe similar phenomenon in our DeepDiver model, which invokes significantly more search calls compared to other baselines, even when evaluated on simple tasks. This over-searching behavior highlights an inefficiency that future work should aim to mitigate. A promising direction would be to develop methods that reduce both the search and reasoning overhead LLM-based systems.
D Detailed Dataset Curation
D.1 Examples of the WebPuzzle
In this section, we show two examples of the WebPuzzle sampled from the cross pross-page question and riddles respectively.
Example Cross-page Question
31 -
12.971125500 125cm22 thil nt A 2e BE 4
Translation: On October 31st, 2024, how much does an ordinary adult tourist with height of 125 cm need to for the night ticket of the Halloween Happy Valley Shenzhen?
Solution: 149 RMB
19

---

Example of Riddle
SHERKTANEBARNTTTIISSS AA 1 AE ChatGPT T F. 12
18
Translation: A Chinese social media app unexpectedly gained great deal of attention due to U.S. government ban. Users refer to themselves as "refugees" and have migrated to this platform. one point, the app ranked first downloads the Apple App Store, surpassing ChatGPT. Its parent company is headquartered in Shanghat. Which app is this?
Solution:
/ Tv Rednote Xiaohongshu
D.2 Principles Annotate the Evaluation Set
The experts were required to adhere to the following principles: (1) The answer to the posed question must be definitive and unique. (2) The answer should derived from internet search results, not from common-sense knowledge. (3) Answering the question should involve multiple searches, thorough reasoning, and validation, rather than a simple query. (4) The answer must be accessible and solvable through sufficient rounds of searching and reasoning.
D.3 Quality Assurance of WebPuzzle
We prioritize recently updated pages fully covered by LLMs' knowledge cutoffss. emphasizing open-web searches for problem-solving We exclude offensive, politically sensitive, ethically con- cerning, or NSFW content using an LLM-based filter. Ambiguous, controversial, multiple-choice and boolean questions are removed to prevent answer hacking. We also eliminated unsolvable problems to ensure low error rates, ultimately collecting 24k WebPuzzle training samples.
D.4 Diffculty Level Tagging
Formally, let Noorect denote the number of correct answers out of the 4 tests. The difficulty level D of problem can be determined as:
easy
if Nenreet
medium
if N Veuract
3
D
(3)
hard
if N conrect
outlier
if Neomaat
E Detailed Experimental Setup
E.1 Statistic of WebPuzzle
Training Data Num
Evaluation Data Num
Data Category
Easy
Medium
Hard
Outliers
ALL
Easy
Medium
Hard
Outliers
ALL
Cross-Page QA
2553
2451
1404
3970
4500
5
27
130
Open& Wiki Riddle
5566
2956
1409
3375
2500
0
33
145
Total Set
8119
5407
2813
7345
23684
5
60
119
275
Table 7: Data statistics the full WebPuzzle dataset. Problems in WebPuzzle are labeled as easy, medium, or hard, and outliers refer to cases with pass@4 0.
E.2 Reward Defination
Formally, Let 9, denote the similarity score assigned by the looser grader for the i-th response. The reward E, assigned by the looser grader is defined as:
20

---

1.0 if T 2 6, &
0.0 otherwise.
For the stricter grader, the response undergoes three rounds of evaluations, each providing binary judgment, J.eh 0, 1, where & 2,2 represents corresponding round. The reward & is assigned only if at least two out of the three rounds agree that the response is semantically equivalent to the reference. The stricter grader's 's reward mechanism is defined as:
if Ei=1 J. 2,
&, 1
(5)
otherwise.
the strict grader evaluates the generated response by comparing the reference answer and the checklists over three rounds. In each round, the evaluator determines whether the model's output matches the reference answer and aligns with the requirements specified in the checklists. the evaluator deems the response correct in at least two out of the three rounds, the result is classified as correct; otherwise, it considered incorrect. The accuracy rate, derived from this evaluation process, is reported as the primary metric in our results.
E.3 Extra Tool Call Rewards Assignment
Formally, the triggering condition for extra rewards defined as:
GiEG = C 0, 3j E G such that Sj = and Cj 1,
(6)
where G is the group of rollouts, S, indicates whether the i-th rollout uses a search engine, and C, indicates success. The extra reward is assigned only when search-free rollouts succeed and at least one search-enabled rollout succeeds. In such cases, a +1.0 reward added to successful search-enabled rollouts.
Formally, we apply the extra reward with:
1.0
if = and C₁ = 1,
& 0.0 otherwise.
E.4 Benchmarks
We evaluate the performance of models using the following benchmarks:
Our proposed novel, web-based assess models' deep information-seekkngg abilities within real-world web environment. WebPuzzle serves as an in-domain task that evaluates a model's capacity to extract and process information from the web.
WebPuzzle:
benchmark designed to
C-SimpleQA-500: A randomly sampled subset of 500 instances from C-SimpleQA [33]. C-SimpleQA is Chinese-translatee version designed assess the factuality language models. While not explicitly designed for complex or real-time question answering, we utilize C-SimpleQA explore the impact of web search scaling on simpler information- seeking tasks.
FRAMES-Zh-230: A subset of the FRAMES [4] benchmark with 230 samples that requires multi-hop information-seeking The queries are translated into Chinese, and annotators verify whether the golden answer can be retrieved via interactions with our web search API. Only cases where the golden answer is reachable are included in the evaluation.
Bamboogle-zh-71: A subset of the Bamboogle (1 benchmark consists of 71 samples, processed similarly to the FRAMES-ZL 2h--200 dataset.
E.5 Baseline Methods
The following baseline methods evaluated:
21

---

Prompted without Web Search: In this setup. LLMs respond to problems based on single round of prompting without web access. The model generates answers based solely on its pre-
existing knowledge. We include off-the-shelf L.MMs such as Qwen2.5-7B: 2012255-77--nstruee [30 36
QwQ-32B (31), GPT-4o
(0), DeepSeek-RI
and Mingu-BB-Reasone [28]
Prompted with Iterative RAG: In this approach, LLMs answer problems through multiple rounds of retrieval and reasoning, using real-world open-web search engine. The baselines from the "Prompted without Web Search Methods" are tested using the same iterative RAG workflow in our approach (see Appendix E.6 for implementation details).
We performed SFT on Swven25--B-Inntruue and Pangu-7B Pangu-7B-Reasone 1B-Reasoner using a combined dataset that integrates both the cold-start data (introduced in Section 3.2) and the dataset used during the RL training phase of DeepDiver. Both models are distilled using the responses generated by DeepSeek-RI.
RI-Distillation:
E.6 Iterative RAG Prompting
In our experimental framework, which includes WebPuzzle data tagging (Section 3.1 cold start data construction (Section 3.2). DeepSeek-RI distillation, and multiple iterative RAG baselines (Section 4.1]. we utilize prompt-based approaches to implement iterative RAG. We have specifically designed a prompt where, in each round, we evaluate the model's response decide whether to terminate the process. If retrieval is triggered, the retrieved results are concatenated in the next user turn to facilitate continued reasoning and searching. For detailed description of our prompt design, please refer to Figure 12
E.7 Grader Details
During the training of the RL model, both a loose grader and strict grader are involved. The loose grader performs single evaluation and provides a score ranging from 10, with or above considered correct. This loose grader is used in the early stages of training to enhance training signals and ensure stability. In contrast, the strict grader performs three rounds of validation using different prompts, each producing binary classification of "correct" or "incorrect." The final result determined by majority voting across the three validations. This grader is used in the later stages of training to prevent model manipulation and further improve performance. Both graders are implemented base on qwen-turbo AP For details on the specific grader prompts, please refer to Figure 14 and 15
of 6
E.8
Implementation Details
During training, each data sample undergoes 14 rollouts with a sampling temperature of 0.9. We employ batch size of 32 and a learning rate of le-6, training for single epoch with KL divergence coefficient of 0.001. The maximum number of tool call round is set to For online search, we utilize the Bocha search engine for Chinese scenario and LangSeareh for English scenario, retaining only the top 2 results per search query to ensure efficiency.
Training Data Num
Evaluation Data Num
Data Category
Easy
Medium
Hard
Outliers
ALL
Easy
Medium
Hard
Outliers
ALL
Cross- Page QA
200
2200
1300
800
4500
54
27
44
130
Open& Wiki Riddle
1200
1100
200
2500
37
33
75
145
Total Set
200
3400
2400
1000
7000
91
60
119
275
Table 8: Data statistics of the WebPuzzle training and evaluation sets uesd in our Experiment. Problems are labeled easy, medium, or hard, and outliers refer to cases with pass@4 0.
lttsss://open..bccaat com/
httpe:/llanssearccccom
22

---

Methods
WebPuzzle-en
BamBoogle
FRAMES
HotpotQA
R1-Searcher
25
6.5
44.8
24.0
56.7
Our LLM Judge
DeepResearcher
45
5.5
51.2
32.3
53.7
DeepDiver-Qwar
21.1
61.6
36.0
64.3
R1-ceareeer
24.3
RI-Searcher LLM Judge
DeepResearcher
32.8
DeepDiver- -wenen--Qeen
43.2
22.4
44.0
DeepResearcher
R1 R1-Searcher
21.1
48.8
27.7
62.3
DeepReseurcher
24.0
58.9
35.8
61.2
LLM Judge
DeepDiver- -weeer-Qeen
41.1
65.6
37.5
67.0
RI- R1-Searcher
13.7 (1.9)
46.7 (2.0)
25.3 (1.9)
57.9 (2.3)
Average
DeepReseurcher
15.0 (7.5)
53.9 (7.1)
33.6 (7.2)
56.6 (4.4)
DeepDiver-Qwen 26.1 (14.7)
56.8 (9.1)
(14.2)
58.4 (10.4)
Table 9: The full comparison results with relevant works the English evaluation dataset using English search engine environment. We took the average of the LLM Judge's results from three works evaluate the accuracy. The number in indicates the average number of search queries invoked.
F
Case Study Examples
In this section, we present a case study on the WebPuzzle, comparing the performance of our proposed DeepDiver model with the RI-distilled model and DeepSeek-RI. The results are shown in Table I0
F.1 ProxyQA Case
In this section, we present an example of DeepDiver's results on ProxyQA. We compare the perfor- mance of DeepDiver-QQwe22.5-77 with its R1-distilled variant. The original R1 model only requires a single round reasoning due to its strong internal knowledge. However, as shown in Figure 9 the distilled model replicates the single round of reasoning without possessing the same internal knowledge. This lack of depth leads to overconfidence, causing the model to prematurely provide an answer after just one round of reasoning. In contrast, Figure demonstrates that our proposed DeepDiver consistently searches and reasons over multiple rounds to gather the necessary knowledge thorough and comprehensive output.
required to answer the problem, resulting
23

---

Problem
Solution
#
RI-
<thinks ...ff ...9% Ett.
Prompting
[H 73
<search:: |"F1R
BA", "WW"WECC 26 $164$$$F11
Search Results: ...corecct answer retrreeeed
<thinks; 25. HI THE
<searcho: (- 35 2F "2019
"FIS PESWERC 26
12/0. (1.44111
Search Results: (correct answer involved)
<thinks: ...44 1E 2008
canswers: 1.
<thinks: BO MUST
Distill
<searche:
"10 2244 1626
Search Results: (correctt answer not included)
<thinks: -..1E--- HOts M EATRE SF TE
['20144
Seurch Results: (eooreettt answer not included))
<think>
7-th M. FJ]]
1i 1: N0
TEBA MERAR J9 JO PEEAA 18838,
<answers:
...
DeepDiver
<thinks:
<searchs: ['IR
"E 4:07 2F
#F26
Search Results:
(corret answer included)
<think>:
22 Fuele Hir Fed 18 RE
Arise Racing GT $ Chaz
Mosten Mosten
<searcho: [22021-2025 BB
"Chaz Mostent
Search Results:
<think>:
de 2 02 2025
Me Chaz Moster
26 in
<wearchs: 18"1]
C"Chaz Mosteri F1EM01
2002 2025
Search Results; ...I 1 F4
22222 42
3e W323 FI 1000 A
(1000 Miles of Sebring) _.
retrieved by "26 #
<thinks:
canswer>: A. $450
Blue: Exploring possibilities and widening scope. Orange: Narrowing down to incorrect candidates. Green: Narrowing down to right answer. Iulies. Faulty conflict resolation.
Table 10: DeepDiver leverages the SIS to compensate for gaps in flawed internal knowledge. Rather than prematurely providing an answer based on unverified or insufficient evidence, DeepDiver continues to intensify its search process, completing a total of 12 searches, progressively refining its findings until it identifies an acceptable answer.
24

---

Problem
Throughout his career, he broke the record the youngest driver to points Formula One history. He competed for thrse different top Fl teams and underwent two special scat swap arrangements.
At the German Grand Prix, be helped midlield team secure its podium linish since 2008 After transitioning to endurance racing, he pillted the 26.26 car and crafted new history across multiple events. Who driver?
Solution
Daniil Kvyat
RI-
<think>: ... -.Verstappenholds the youngest points Gcord..Holleebeegg 's age when scoring points in 2010 didn't didn't break the record...Kvyee Kvyat exclusion required (Scuderia Tom Rosso being midfield but didn't drive 26 endurance)...
Prompting
<search>:: ['Fl youngest woints-scorrin driver record", "Drivers with three Fl teams German GP podium", "2019 German
Grand Prix podium teams". "WEC 26 car drivers Fl background"]
Search Resulls: ...correct answer retrieved by "2019 German Grand Prix podium teams"
<think>: ... Veesttpppen excluded... wwr.ad......yt...oo...... Konsee..o mlexiedd....eexuddee Albon...Noed confimation of other driver
asuuciations.
<searcho: ['Kvyat number of Fl teams served", "2019 German GP Toro Rosso midfield status", "Fl drivers WFC 26 car",
"Daniil Kvyat endurance macng"]]
Search Results: ...orett answer involved)
<think>: ..yat fulfilling "first German GP podium by midfield team post-2008"... Otherr candidates like Hulken-
berg meet all conditions.
canswer>: ...Daniil Kvyat
Qwen7d-RI-
<think>: Initial assessment suggests it may be Nico Hülkeaberg....
Distill
<search>: |YYoungesi point scoring driver in F1 history", "Nico Hülkenneerg seat swap history". ". "2014 German Grand Prix
Sauber Nolson Piquet", "24 Hours Le Mans Car 26 Hülkknberg"]
Search Results: ...(rrreect answer is not included)
<think>: ...Nico Hülkennergg never achieved a podium finish, confficting with the German GP third thind-place claim...
<searcho: ['2014 German Grand Sauber team results", "Youngest point-scoring driver in history", Hours Le
Mans Car 26 driver Hülkenberg']
Search Results: ...rrret answer not included)
<think>: Contradiction resolution: F1 historical data showa the 2014 German GP's 's best midfield team result was Mussa's 's third place (Fervari), white Sauber's optimal result race was Hilkkenber'' 's third place. This likely stems from misinter- pretation of "midfield teams" as "non top teams", requiring explicir clarification in the response.
<answer>: ...Nico Hülkenberg X...
DoepDiver
<think>: ...Verify there exists a driver mecting all criteria...
esearchs: ("Youngest H1 points record holder", "Fl seat changes across three teams". "Midtield team podium 2008
German GP", "Endurance 26 car driver")
Search Results: ...(crrect answer not included)
<think>: ...Antonllli Verstappen relate youth records, bui require more specific information... is driven
by Chaz Mostert...
<search>: ("2023-2025 German midfield team podium". ", "No.26 car Fl eadurance racing". "FI seat changes across three teams". "Chaz Mostert racing career"]
Search Results:
<think>: ... Neriify.,.haaz Mostert's participation with three F1 teams, seat changes, and No.26 car usage during 2023-2025. Potential unretrieved German GP midlield team podium record requires supplementary search of 2025 F- German GP official results.
esearcho:: ['Chaz Mostert FI history", "2023-2025 German GP midfield team podium", "No.26 car drivers", seat changes three teams"]
Search Results: ..[[14] Webpage title:... Kvyat represented the Hezeburg team three times in the 2022 NASCAR Cup Scries, and his car used the same number 26 in Formula I... In the scason opener, the Schring 1000-mile endurance race. Kvyat finished third and earned his first WEC podium....(corree answer is retrieved by "No.26 car F1l drivers")
<thinko: ...Daniil Kvyat matches No.26 cudurance racing characteristic...
<answer>: Damiil Kvyat v ...
Blue: Exploring possibilities widening scope. Orange: Narrowing down incorrect candidates. Green; Narrowing down to right answer.
holles. Faulty conffict resolation.
Table 10: DeepDiver leverages the SIS to compensate for gaps in flawed internal knowledge. Rather than prematurely providing an answer based on unverified or insufficient evidence, DeepDiver continues to intensify its search process, completing a total of 12 searches, progressively refining its findings until it identifies an acceptable answer.
25

---

Reasoning and Search Rounds Proxy QA DeepDiver
Pruny Question:
ELA AO
Response:
BiNaga Viper H
Bihund au
HE di
# R. # Wewkea FE
49
HE
ROUND
fth, Naga ViperSR FHR M. AOL
Naga Viper 16 THE 2E
4. M
33 BR aRa Viper
IR.. 101222 If TE META MESH
BE
ROUND 2: 12023
HOUAAE. Naga Viper Eilfi
# #e Ht. 202344 de Ve: ME BALL 2:269 SHU
ai + Ae H1.2222077. #AAAAI
2.
P (17 VipertH ME "RAN (E 2022 PO MI Te (F
ne
4. 68 M #
14 [+133]1111111 171 MA A9
ROUND [2024 AE
FT.
Figure 8: DeepDiver's reasoning process on ProxyQA involves expanding its search beyond a narrow scope explore additional knowledge when faced with open-ended, challenging problems, improving its generalization through SIS-incentivized behavior.
Reasoning and Search Rounds of Proxy QA R1-Distill
Proxy Question:
Response:
ETTE IE 2 S0th 2F.
[El
ITAM) RUIL
SE.
#1oRttt # -
4.
(MICAPS #1 (Amal 18
(88)
SR AE Bn HOAEER REERRA Jn FO 2F AARAEE (16577-325
ROUND I: [1 didde EE RROVEE DEER H9, BABLECAPSS A WAEKAA
Figure 9: The RI-distilled model on ProxyQA mimics DeepSeek R1's behavior but shows overconfi- dence in its flawed internal knowledge, resulting in just one round of reasoning and searching.
26

---

Answer Rounds of ProxyQA R1-Distill
Pruxy Question:
Response: B
P RA Be
(SHU) 2011
A CurrieFF
2E / Mi SA T
S IN 3
(capsaricin) 38 3E 4 5E 32
H (20-30 1
dE AEL FF [5]
Curriciifiil
1R (2.
#
AO 13,00
16), OF of
Figure 10: With limited reasoning and searching, the R1 R1-distilled model produces output that lacks informativeness, comprehensiveness, and depth.
27

---

Answer Rounds of ProxyQA DeepDiver
Proxy Question:
#2024
-
2024
X th
Pepper
(Ed Currie) TF
S2H CEAMEE SN (2 fo 96- BE
(2207SHHU)
: (Carolina Reaper
2
THASEE SHU.
2 a
203E Ri 11
Naga Viper 7 M AE
#:
X
NE 4p:
AA x121110))
109 [7]. &
2 WO.. 5g BERC frttaad)
[14]. IL
3.
# 5
BR 14
CHARE 097096
F [23][24]
(140%
3 18 6E 64,
#
131
* Wi: 1mg/kg11/88 18 1600755H1U
#t[24][23]
4
#A काा 1: M.H60.0077
(90-1205k)
(0) [7][12]-
f
A
[]]]]))][266
TRPVIE
2021 OL
DDvid IE
Juliusi N. #N
X [_]]]]]]]
HC72%) FEF..
1E [24]
I
X To 20234F
1000
#6 :
135 0
2024 F...
*
(>200]]SH()) TT
Figure 11: After several rounds of reasoning and searching. DeepDiver generates result that covers various aspects of the given question, delivering extensive and comprehensive content with a wealth of knowledge.
28

---

Iterative RAG Prompt
TA T
#* 4 13 3a
16 + <<thinkings
1. F0 3
11
2. 12 #0 s8 7 BA
THE E00 t
3. * X AA
2 A4
4. 11
5. J (5
AT. 11 SU
BH Alt Ro AO
6. 11 0:: IEX. I
16 (#
1. (i 10 | web_searchl| 'scarch search__ueeries". [
fi.."....J.. 1
2. :
P A0 19 FO/F P Et.
30:
(E 3
X (A0 3 X28)) WA B
A
2
& TE SE $
オ 13
#* th $ F
7 T E #n B0 E 360 (
Squery [ [
Figure 12: The prompt we designed to implement iterative RAG, which is used in WebPuzzle data tagging (Section B.I.. cold start data construction (Section 3.2.. DeepSeek-R distillation, and multiple iterative RAG baselines (Section 4.1
29

---

Taugh
Cara 231
1.50
The detailed training curve of the 1oepDiiver-we2225--7-------- model shows the trending of key metrics throughout the training process. These metrics include reward, tool call frequency, response length, reward standard deviation, gradient norm, KL divergence, and response completion rate.
Figure 13:
30

---

Loose Grader Prompt
#* 
1.
2.
3. 30 h2g.
30 JA X
4. - 14 + MA3h 5 11 1E
5. 14
6. 2
A
7. 30 88.
(1)
20 TOS BI (2) S
# 30 -
10th: 22 BEL. M
8-9/h:
BYY MAN
6-75 I
5 (1
XX. 1
2-45h: 1E (7) BE 11 IT
14:: #1 Si
| [4]
$query
$ サ K
Ssolution
||| |f
lchecklist
C A
|f
Sresponse
## 1 11
$ F
A.. CLR
34
$- SAi
$ js
tr
$ jsonfit II!:
" '" $ 1-10.
Figure 14: The prompt used for the loose grader, which assigns a score between and 10 based on a single evaluation. A score of 6 or higher is considered correct. The design aims to enhance training signals and stabilize the learning process in the early training stage.
31

---

Strict Grader Prompt
Strict Grader Prompt I:
#chheckkist
Squery
Ssolution
hheekkistt Schecklist
Sresponse
. F:
1. S Pr
2. H /5r fc
MR A Di A0
3. (7 195 Si.
ー
レ 8 " RRE PRAEE BE EO
F 10
, 17 :1 "T P
"'
Strict Grader Prompt 2:
1
12 P MR Bchhhkkk [u]
M M
# T
A Squery
$1 F:
1. # 0
2. E &F
3. 59 1E
28 2500, 10
Striet Grader Prompt 3:
Rccheckliss AEEE. #
H
Squery
F:
1.
x|e > Wi iR S0 d0 10
2. (5 te #checklistRf [0 L AT
3. R T S TE
Figure 15: Different prompts used in the three rounds of validation for the strict grader. Each round of validation generating binary classification of "correct" or "incorrect." The final result determined by majority voting across the three evaluations. This grader is employed in the later stages of training further enhance performance.
32

---

Behavoir Statistic Prompt
Reflection & Correction Prompt:
F
R E
Squery [A #1] 1
Scot
Ssolution
| A0 2F -
3 E & Correctionf 30 :
TE # S
it Reffeccton & Conrectio
27 # #=cconnt>c/coontsff
Conflict Resolution Prompt:
F Conflict (1 WEAKEEE
# म - F. 1 24 64 6
お Conflict Resolution
M # 76 i0 it,
# te /ccuuuoooo
Verification & Denoising Prompt:
Verifficaaaoooooo & Denoising t/ ハ # A 1
i | X Verification & Denoising St -
El Hl &6 & /1 79 A CE
H H B
Figure 16: Prompts used for automatically evaluating the occurrence counts of different behaviors in model outputs.
33

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, the abstract and introduction accurately summarize the paper's key findings, including the experimental results and their robustness and generalizability.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Yes, we thoroughly discuss the limitations of our work and potential solutions
in the Discussion section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
34

---

Answer:
[NA]
Justification:
There are no new theorems, formulas, and proofs involved.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Yes, the paper provides detailed step-by-step descriptions of the dataset curation (Section B.1). methodology (Section 3.2). and experiments (Section 4 to ensure
reproducibility of the main results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
35

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification: The paper contains experimental results that rely on proprietary code and data that are currently undergoing internal review process for open-source release approval. While we cannot provide open access the code and data at submission time, we plan to release them once the review process is completed. This restriction is in line with the NeurIPS guidelines, which acknowledge that there may be legitimate reasons why code cannot be made available immediately. We have included detailed descriptions of our methods, algorithms, and experimental procedures in the paper and supplementary materials ensure the scientific contribution is clear and to allow for conceptual reproducibility of our work.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/ public/guider httee//dddee//eeeuuiiiieeee
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results. See the NeurIPS code and data submission guidelines (https httpee///ooodeueaaassisonooooooo
http:///ccc.o..ggaiaooo///o//aaa ccsppccc
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, Sections 3.1 and Appendix D provide comprehensive details on data selection, experimental setup, hyperparameters, and implementation to ensure full reproducibility of the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
36

---

Justification: While our paper does not include formal error bars, confidence intervals, or statistical significance tests, we did take steps to ensure reproducibility and reliability of our experimental results. Specifically, each experiment was conducted with 3 independent runs using different random seeds, and we reported the average scores in our results. We acknowledge that a more comprehensive statistical analysis with properly defined error bars would strengthen our findings. For future versions or camera-ready preparation (if accepted), plan to expand our statistical reporting to include standard deviations or confidence intervals for all key experimental results, clearly stating the calculation methods and assumptions.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative етror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We introduced the computation resources and requirements in details in Section
E.8
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
The paper is fully conform with the NeurlPS code of ethics.
Guidelines:
37

---

The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
To avoid potential ethical and societal impact related issues, we carefully checked all questions in datasets in multiple aspects, as discussed in Section D.3 We try to guarantee that all samples do not involve any offensive, conder-biased or political content, and any other ethical issues. The source code will be released with instructions to support correct use. The baseline model we tested are all open-sourced LLMs and public APIs, these previous works have already considered the societal impact issues when creating the models.
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided relcasing unsafe images.
38

---

We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All the code dataset and benchmarks and related basclins are well cited and introduced in Appendix E.5 Appendix E..4 and Section
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [No]
While we have provided detailed documentation and descriptions of our new dataset and models within the paper itself (including details on collection methodology, properties, limitations, and intended uses), we are unable to provide the actual assets alongside the submission due to the ongoing internal review process for open-source release approval. The paper contains comprehensive documentation that would typically accompany these assets, but the assets themselves cannot be uploaded submission time. We plan to release these assets with complete documentation once the review process is completed.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
39

---

Justification: We conduct the human evaluation and annotation, all the details including the guideline and instructions are performend in Section A.3 and Appendix D.2
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: We do invovle human subjects, all annotators and evaluators are recruited by a company.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [Yes]
Justification: Yes, our paper transparently documents all instances of LLM usage usage-whether whether proprietary or open-source- -inruccinncuuinngggc detailed prompts and implementation specifies. This ensures reproducibility and clarifies their role in our methodology.
Guidelines:
The answer NA means that the core method development in this research does not involve L.MMs any important, original, non-standard components.
Please refer to LLM policy for what should or should not be described.
40


Reviews:
[{"review_id": "vp15upCdp3", "reviewer": "Reviewer_5N3z", "summary": "The paper investigates LLMs capability for open-web retrieval and proposes letting the model learn to perform proactive, multi-step searches when solving open-domain questions. The authors release WebPuzzle (24k training examples; 275 human-curated test questions), a dataset explicitly designed to measure an LLM’s ability to retrieve and reason over information in a realistic, noisy and fragmented web environment. They further introduce the DeepDiver framework: after a SFT warm-start, the system is refined with reinforcement learning (GRPO) to cultivate a policy that decides when and how much to search. A key claimed feature is the emergent SIS ability. Experiments with 7B parameter base models show that DeepDiver nearly matches R1 on real-time web QA benchmarks, while raising the average number of retrieval rounds from ~1.7 to > 2.5. The paper also analyses the training curriculum and the strategy’s generalisation to tasks such as open-ended long-form generation.", "questions": "1. Most of the training samples in the WebPuzzle dataset are synthesized through LLM. How do the authors ensure that these synthetic problems realistically reflect the complexity of real user search tasks?\n2. DeepDiver’s RL training uses complex reward design and training scheduling. Can the authors share the sensitivity of key training settings?", "limitations": "n/a", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Pros:\n\n1. The paper constructs a new WebPuzzle dataset to evaluate knowledge retrieval and reasoning in an open network environment.\n\n2. The overall scores of Qwen2.5-7B and Pangu-7B trained by DeepDiver are almost equal to or even partially surpass those of the DeepSeek-R1 model, which has two orders of magnitude larger parameters.\n\nCons:\n1. Although the paper claims that SIS is a major innovation, this capability is essentially a model behavior promoted by reward design, and is conceptually an extension of existing ideas. Using reinforcement learning to optimize LLM retrieval decisions is not new. In recent years, there have been works that use RL to let the model learn when to retrieve and how many steps to retrieve. The main contribution of the author's method is that it is applied to more complex environments and the number of model retrieval times is observed to increase. This is more of a phenomenon report rather than a new algorithmic principle.\n\n2. The paper repeatedly describes SIS as a feature that emerges automatically after RL training, but in fact the authors explicitly add rewards to the model to encourage multiple searches. Calling this behavior \"emergent\" under the influence of rewards is a bit exaggerated and may mislead readers, as if the model has spontaneously evolved this skill.", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Reviewer 5N3z**\n\n---\n\nThank you very much for your thoughtful and constructive feedback. We truly appreciate your careful reading of our work and your insightful comments regarding the novelty of SIS and the reward design. Please allow us to clarify and expand on the points you raised.\n\n> Although the paper claims that SIS is a major innovation, this capability is essentially a model behavior promoted by reward design, and is conceptually an extension of existing ideas. Using reinforcement learning to optimize LLM retrieval decisions is not new. In recent years, there have been works that use RL to let the model learn when to retrieve and how many steps to retrieve. The main contribution of the author's method is that it is applied to more complex environments and the number of model retrieval times is observed to increase. This is more of a phenomenon report rather than a new algorithmic principle.\n>\n> The paper repeatedly describes SIS as a feature that emerges automatically after RL training, but in fact the authors explicitly add rewards to the model to encourage multiple searches. Calling this behavior \"emergent\" under the influence of rewards is a bit exaggerated and may mislead readers, as if the model has spontaneously evolved this skill.\n> \n\n**Response:**\n\nWe apologize if our explanation was unclear and led to the impression that our approach is merely a restatement of prior work or that the behavior we observe is entirely driven by reward shaping.\n\nWhile it is true that our method includes an auxiliary reward to encourage search actions, our primary objective is to **mitigate the model’s initial reluctance to use external tools**, which we observed to be a consistent bias during early training. Specifically, in the early phases, the model tends to over-rely on internal knowledge—even when external retrieval is necessary (for example the questions about latest news). The auxiliary reward is thus introduced to encourage exploration of retrieval behaviors during this critical learning phase.\n\nFormally, the condition for extra rewards is defined as:\n\n$$\n\\forall i \\in G, \\mathcal{S}_i = 0 \\implies \\mathcal{C}_i = 0, \\quad \\exists j \\in G \\text{ such that } \\mathcal{S}_j = 1 \\text{ and } \\mathcal{C}_j = 1,\n$$\n\nwhere G is the group of rollouts, S_i indicates whether the i-th rollout uses a search engine, and C_i indicates success.\n\nWe apply the extra reward with\n\n$$\n\\mathcal{E}_i =\n\\begin{cases}\n1.0 & \\text{if } \\mathcal{S}_i = 1 \\text{ and } \\mathcal{C}_i = 1, \\\\\n0.0 & \\text{otherwise}.\n\\end{cases}\n$$\n\nThis means that when no **search-free** rollouts solve a problem but **at least** one search-enabled rollout succeeds, we assign an additional reward of 1.0 to the successful search-enabled solutions.\n\nThis design **differs from prior work** such as R1-Searcher, which provides persistent additional rewards (+0.5) for using external tools. In contrast, **our reward scheme does not prefer search-based over search-free solutions when both yield correct results**. This ensures that the model does not develop a dependency on tool use, but instead learns to call tools only when beneficial.\n\nTo better understand the effect of our reward scheme, we analysed the frequency of the special search-call reward (value = 3.0) throughout training. Every 10 training steps (each step producing 448 trajectories), we measured how often this reward was triggered. As shown below, the reward is naturally **phased out over time**, confirming its role as a **transient scaffold** in early learning:\n\n| Step Range | count | Percentage |\n| --- | --- | --- |\n| 0–9 | 198 | 4.5% |\n| 10–19 | 140 | 3.1% |\n| 20–29 | 104 | 2.3% |\n| 30–39 | 49 | 1.1% |\n| 40–49 | 42 | 0.9% |\n| 50–59 | 43 | 1.0% |\n| 60–69 | 27 | 0.6% |\n| 70-80 | 6 | 0.1% |\n\nFrom this, we highlight two key observations:\n\n1. Even in the earliest stages, only a small fraction of trajectories receive the additional reward.\n2. The reward is **naturally phased out** over time and becomes essentially inactive after approximately 30 steps.\n\nDespite this, as shown in **Figure 4** of our manuscript, the number of tool-use rounds continues to grow significantly, with a particularly sharp increase observed during steps 80–120—**well after the auxiliary reward is no longer in effect**. This pattern supports our view that **SIS** is **not merely a result of explicit reward shaping**, but rather an **emergent behavior** developed by the model during RL training. The model learns to leverage external tools to compensate for the limitations of its internal knowledge—**a behavior that persists and strengthens even in the absence of direct incentives**.\n\nWe hope this clarifies the motivation and effect of our reward design, and why we consider SIS to be an emergent and meaningful outcome of our method.\n\n---\n\n> Most of the training samples in the WebPuzzle dataset are synthesized through LLM. How do the authors ensure that these synthetic problems realistically reflect the complexity of real user search tasks?\n> \n\nThank you for this important and insightful question. We fully agree that ensuring the realism and relevance of synthetic data is critical, particularly when modeling open-domain search and reasoning tasks. We address this concern through three complementary perspectives:\n\n1. **From the Perspective of Data Curation and Generation**\n    \n    WebPuzzle are not generated from templated rules or knowledge bases, but rather through **LLMs grounded in open-web content**. \n\n    - Cross-page QA: LLMs extract real web pages and then synthesize inverted, multi-hop questions that require reasoning across multiple sources. This goes beyond conventional Wikipedia-based QA datasets and aims to reflect the structure and ambiguity of real-world web search tasks.\n    - Riddle-style Question: LLMs are prompted to create riddles or obfuscated queries based on unique attributes of entities. This introduces indirect phrasing and semantic ambiguity, simulating the kinds of vague queries users often make online.\n\n2. **From the Perspective of Human Evaluation**\n   \n    To assess and improve realism, we conducted **manual filtering** during test set construction. From an initial pool of 500 synthetic examples, human annotators reviewed each item to:\n    \n    - Eliminate samples that **cannot be solved within the Chinese internet environment**.\n    - Remove any questions that **do not adequately reflect the complexity** or ambiguity of real user queries.\n    \n    This resulted in a curated test set of 275 examples. Notably, most exclusions were due to regional search constraints, while only a small portion were rejected for lacking sufficient complexity. This suggests that the majority of LLM-generated queries do capture key aspects of real-world difficulty.\n    \n3. **From the Perspective of Model Generalization**\n    \n    We also validate the quality of WebPuzzle indirectly through **cross-dataset generalization**. Specifically, after training on WebPuzzle, our DeepDiver demonstrates strong performance on **Bamboogle**—a manually crafted dataset designed for evaluating real-world internet search capability. We believe this strong transfer supports the conclusion that WebPuzzle enhances model reasoning in ways that generalize beyond synthetic tasks.\n\n---\n\n> DeepDiver’s RL training uses complex reward design and training scheduling. Can the authors share the sensitivity of key training settings?\n> \n\n**Response:**\n\nWe are happy to share the insights we have gathered through extensive experimentation, including many unsuccessful trials.\n\n1. **Loose-to-Strict Reward Scheduling**\n    \n    Initially, we employed a strict reward rubric based on LLM-as-Judge assessments. However, we quickly discovered that such a reward scheme led to a high prevalence of zero-reward episodes, preventing effective gradient propagation. Specifically, when the model’s early outputs were consistently poor, the computed advantage:\n    \n\n$$\n\\hat{A}_{i,t} = \\frac{r_i - \\mathrm{mean}(r)}{\\mathrm{std}(r)},\n$$\n\ncollapsed due to all-zero rewards triggers the zero adavantage. We also attempted curriculum learning by filtering for easier examples in early training, but this did not overcome the zero-reward issue.\n\nTo address this, we designed a **loose reward** function that relaxed evaluation criteria, allowing the model to receive credit for partially correct or approximately matched outputs. This approach helped bootstrap the model’s learning. Once stable training progress was achieved, we transitioned to the **strict** reward rubric to further refine and push performance beyond this initial plateau.\n\n1. **Auxiliary Tool-Use Reward**\n    \n    As discussed above, we found that LLMs exhibit a strong early bias toward using internal knowledge. Our auxiliary reward is designed solely to counteract this reluctance during early training. It is deactivated naturally during the middle training stage.\n    \n2. **Cold-Start Data Selection**\n    - We experimented with prompt prefilling strategies that concat phrases such as *“Wait, I need to search again to verify my conclusion”*. While this led to slight gains after SFT (around +0.5–1.5 points), the improvements did not persist after RL. Post-hoc analysis confirmed that such “s1-like” patterns were rarely used after RL, suggesting that forced prompting **encourages the search** does not translate into sustainable behaviors and performance gain.\n    - We also explored cold-start fine-tuning on the WebPuzzle dataset alone. However, without mixing in general reasoning datasets, the model’s performance was significantly worse. We found that exposure to diverse, long-context reasoning tasks during cold start  helped bootstrap the model’s ability to reason and reflect.\n    \n    We hope these insights provide a clearer view into our training strategy and its sensitivity. We thank the reviewer again for encouraging us to share these details."}, {"review_id": "VmRnZwu8Hv", "reviewer": "Reviewer_ucvZ", "summary": "The paper studies the task of question answering via web information retrieval using Large Language Models. The authors provide a new benchmark dataset, WebPuzzle, testing the LLMs ability to navigate and reason about noisy information from websites. Using this, they train DeepDiver, a 7B LLM that reaches similar performance to much larger models on such task. For this, they propose a tailored RL pipeline to incentivize search intensity and generalize to new domains. Moreover, they analyze various evaluation aspects such as the relationship between search intensity and problem difficulty.", "questions": "- From my understanding, only 2k samples (demonstrations?) were used for SFT, while 5k samples (prompts?) for RL. Did the authors ablate the chosen split? In particular, I wonder how would the performance vary as a function of the amount of SFT data. E.g. what would have happened with a single SFT phase on 7k samples.\n- Related to the above, online RL (via GRPO) seems crucial for performance but also the main computational bottleneck to scale the approach to more samples. Did the authors consider offline baselines (such as DPO) for sample efficiency?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strenghts: \n- The paper studies a relevant and significant problem, the contributions are clear and the overall exposition is good. \n- The WebPuzzle dataset looks like a relevant benchmark that looks complex enough to separate and differentiate different LLM agents and approaches\n- DeepDiver looks like a strong LLM agent for WebSearch\n- The explanation and ablations of the RL method used for training DeepDiver is a valuable contribution to the community\n- Experiments' analyses are somewhat extensive and informative.\n\nWeaknesses:\n- I think the authors could have given more details on the used RL algorithm and training procedure, with training metrics and ablations on the choices made.", "was_revised": true, "final_justification": "The paper presents a novel relevant benchmark dataset and a performant LLM agent for the Web search task. The overall training pipeline is described, making it a nice contribution to the community.", "rebuttal": "We sincerely thank the reviewer for the thoughtful and constructive feedback. Below, we address each of the points raised in detail.\n\n---\n\n> I think the authors could have given more details on the used RL algorithm and training procedure, with training metrics and ablations on the choices made.\n> \n\n**Response:**\n\nThank you for pointing this out. We have provided the key hyperparameters used during RL training in Appendix E.8, now summarized below for clarity:\n\n| Parameter | Value |\n| --- | --- |\n| Rollouts per Sample | 14 |\n| Sampling Temperature | 0.9 |\n| Batch Size | 32 |\n| Learning Rate | 1e-6 |\n| Number of Epochs | 1 |\n| KL Divergence Coefficient | 0.001 |\n| Max Tool Call Rounds | 7 |\n| Training Hardware | 1 machine with 8 × H20 GPUs |\n| Chinese Search Engine | Bocha |\n| English Search Engine | LangSearch |\n| Search Results Retained | Top 2 per query |\n\nOur choice of hyperparameters was informed by prior research, practical heuristics from relevant blogs, and official TRL guidelines and recipes. We evaluated multiple configurations and selected the ones that aligned best with trends in existing literature. For example, based on the existing experience of the community, the learning rate around 1e-6 [1,2], and the trend for the kl coefficient is to decrease gradually or even be omitted [3,4].\n\n---\n\n> From my understanding, only 2k samples (demonstrations?) were used for SFT, while 5k samples (prompts?) for RL. Did the authors ablate the chosen split? In particular, I wonder how would the performance vary as a function of the amount of SFT data. E.g. what would have happened with a single SFT phase on 7k samples.\n> \n\n**Response:**\n\nWe did in fact evaluate a baseline trained with the full 7k r1 teacher dataset using SFT, which we denote as **R1-Distill** in Table 1. As detailed in Appendix E.5, this model uses the entire 2k + 5k sample set in a single supervised fine-tuning stage.\n\nCompared to prior works that often compare their works with prompting-based methods, or RFT self-training baselines after the cold-start checkpoints, our R1-Distill model represents a stronger baseline by leveraging full teacher supervision. While it performs well, it still falls short of the RL-trained model—particularly on out-of-distribution tasks such as **Bamboogle**, as discussed in Section 4.4.\n\nRegarding the use of 2k samples for cold start and whether we experimented with other split ratios, we did not conduct such experiments. This is because the experimental cost of RL training is relatively high. The choice of cold-start data size was primarily based on experiences from previous literature [5], which suggested a data volume in the thousands, and then made our decision based on our overall data volume.\n\n---\n\n> Related to the above, online RL (via GRPO) seems crucial for performance but also the main computational bottleneck to scale the approach to more samples. Did the authors consider offline baselines (such as DPO) for sample efficiency?\n> \n\n**Response:**\n\nWe agree that offline methods like DPO offer a promising direction for improving sample efficiency. However, applying DPO to our multi-turn tool-use task poses specific challenges. Effective supervision requires carefully curated positive–negative pairs that capture not only correctness of the final answer, but also proper reasoning, tool invocation, and formatting. \n\nTo explore this, we created 5k r1 examples consisting of correct–incorrect output pairs and used them to train **Qwen2.5-7B-Instruct** via DPO. However, the resulting model often failed to follow the required formatting for reasoning and tool usage.\n\nWe also applied DPO starting from the cold-started Qwen2.5-7B-instruct checkpoint, but did not observe improvements over the base model. We hypothesize two main reasons for this:\n\n1. The DPO training signal, which is applied at the sequence level, may be too coarse for our long (22k-token) and complex tasks that require step-by-step tool usage.\n2. Due to time constraints of the rebuttal period, we did not have the opportunity to explore alternative DPO configurations or tune the relevant hyperparameters, and took more fine-grained DPO algorithm implementation. For example when the LLM is sampled multiple times with various correct and incorrect samples, we didn't take into care consideration that choose the most suitable positive-negative pairs (using tricks such as ranking, classification etc. or using synthesised negative samples for more a  contrastive learning objective), but simply randomly select one correct and one negative sample for the training.\n\nWe acknowledge the potential of DPO and plan to explore it more thoroughly in the revision version. We greatly appreciate the reviewer’s insightful suggestion.\n\n---\n\n**Reference**\n\n[1] Search-r1: Training llms to reason and leverage search engines with reinforcement learning.\n\n[2] R1-searcher: Incentivizing the search capability in llms via reinforcement learning.\n\n[3] DAPO: An open-source llm reinforcement learning system at scale.\n\n[4] Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild.\n\n[5] Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning."}, {"review_id": "Wj85Ud5EVc", "reviewer": "Reviewer_2jv9", "summary": "This paper investigates how to enable LLMs to perform adaptive, evidence-seeking behavior in open-domain question answering. To support this, the authors introduce WebPuzzle, a benchmark comprising both Wikipedia and open-web queries designed to elicit complex reasoning behaviors under noisy conditions in Chinese. They propose DeepDiver, a RL framework that trains LLMs to interleave reasoning and search, guided by a two-stage process involving supervised initialization and GRPO-based fine-tuning. A key emergent behavior, Search Intensity Scaling (SIS), allows models to dynamically adjust search depth based on task difficulty. Experiments show that DeepDiver-equipped 7B models outperform prompting and distillation baselines, approaching the performance of much larger models on multiple QA tasks.", "questions": "What’s the rationale behind not explicitly modeling the search?\n\nIn practice, how much impact would the search latency create with increasing search depth?", "limitations": "yes in appendix", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strength\n- The paper tackles an important and challenging problem of adaptive information seeking in real-world, noisy environments where static prompting and closed-domain SFT approaches fall short. The paper demonstrates a novel emergent behavior—Search Intensity Scaling (SIS)—where models increase search frequency and reasoning steps based on task difficulty, leading to substantial performance gains over prompting and distillation baselines.\n- The authors introduce WebPuzzle, a new benchmark with 24k training and 275 expert-annotated test samples. It explicitly stresses noisy retrieval, ambiguity resolution, and reflective reasoning, offering a more realistic alternative to clean Wikipedia-style datasets.\n- The DeepDiver framework is well-engineered, incorporating cold-start SFT, GRPO, and curriculum-aware reward shaping to stabilize and scale reinforcement learning on real-web tasks.\n- The authors provide thorough empirical evidence, including accuracy on multiple Chinese QA benchmarks, information-seeking behavior counts, reward ablations, generalization English benchmarks etc.\n\nWeakness\n- The code, data, and models are not released, citing the internal review process in the checklist. This limits verification and reuse for now, and the release is not guaranteed. \n- The test set size of 275 examples in WebPuzzle is relatively small for drawing strong statistical conclusions, especially given the complexity and variability of open-web search tasks.\n- The reward function design is highly heuristic, i.e., manual engineering of multiple components, including staged loose-to-strict grading and auxiliary reward triggers.\n- It is hard to understand the benefits of DeepDiver over existing agentic tool-augmented approaches, such as ReAct or WebGPT-style agents, which are not included as comparison systems in this work.", "was_revised": true, "final_justification": "I've read authors' rebuttal, and continue to support the acceptance of this work.", "rebuttal": "Thank you for your thoughtful questions, we hope we could address these concerns with our below responses.\n\n---\n\n> The code, data, and models are not released, citing the internal review process in the checklist. This limits verification and reuse for now, and the release is not guaranteed.\n> \n\n**Response:**\n\nWe appreciate the reviewer’s concern. Both the code and data are **fully prepared for release**, and our internal review process has already been completed. We plan to make them publicly available **immediately following the paper’s acceptance announcement**.\n\nWe would have been happy to release them via an anonymous repository during the rebuttal phase to facilitate transparency and reproducibility. However, **conference policy restricts releasing any links during the rebuttal periods**, and we aim to fully comply with these guidelines.\n\nWe are fully committed to open-sourcing all artifacts to support verification and reuse.\n\n---\n\n> The test set size of 275 examples in WebPuzzle is relatively small for drawing strong statistical conclusions, especially given the complexity and variability of open-web search tasks.\n> \n\n**Response:**\n\nWe acknowledge the limited size of our benchmark. However, this is primarily due to the fact that all samples are **manually annotated and iteratively verified**, which is both time-consuming and labor-intensive. As shown in Section A.3, solving each problem requires an average of **9.28 webpages browsed per query**, and even under these conditions, human annotators only achieve **44.0% accuracy**.\n\nDuring the rebuttal period, we evaluated several advanced research agents on the 25 samples presented in Section A.3. The results are summarized below:\n\n| Model | **Accuracy** |\n| --- | --- |\n| **Human** | 44.0 |\n| **GPT-4o (Web Search Enabled)** | 60.0 |\n| **ChatGPT Deep Research** | **80.0** |\n| **Gemini Deep Research** | 52.0 |\n| **OpenHands Cloud** | 21.3 |\n| **DeepDiver** | 40.0 |\n\nAs the table shows, **ChatGPT Deep Research** significantly outperforms both humans and other automated systems on this subset. However, solving each query typically takes **over 10 minutes**, even for state-of-the-art models. This highlights the difficulty of the task and the cost of high-quality annotation—making a benchmark size of **275 samples** a practical and acceptable compromise.\n\nMoreover, we note that several widely-used benchmarks—such as **WSC273**, **CB (SuperGLUE)**, **HumanEval**, and various subtasks in **BBH**—also contain fewer than 275 examples, reinforcing the precedent for using datasets of this scale in challenging reasoning tasks.\n\nThat said, we agree with the reviewer on the importance of statistical robustness. In the revised version, we will include **confidence intervals** (CI) for key metrics to better quantify variability. We also plan to incorporate **pass@k** and **avg@k** metrics with a number of evaluation trials significantly larger than *k*, to provide a more comprehensive view of model performance.\n\nThank you again for this valuable suggestion.\n\n---\n\n> The reward function design is highly heuristic, i.e., manual engineering of multiple components, including staged loose-to-strict grading and auxiliary reward triggers.\n> \n\n**Response:**\n\nThank you very much for this excellent question. Reward design for reinforcement learning in open and complex information-seeking tasks remains a relatively new challenge. Our reward design draws not only on experiences from previous works but, more importantly, on adjustments made in response to \"reward hacking\" phenomena observed in our experiments.\n\n1. **Loose-to-Strict Reward Scheduling**\n    \n    Initially, we employed a strict reward rubric based on LLM-as-Judge assessments. However, we quickly discovered that such a reward scheme led to a high prevalence of zero-reward episodes, preventing effective gradient propagation. Specifically, when the model’s early outputs were consistently poor, the computed advantage:\n    \n\n$$\n\\hat{A}_{i,t} = \\frac{r_i - \\mathrm{mean}(r)}{\\mathrm{std}(r)},\n$$\n\ncollapsed due to all-zero rewards triggers the zero adavantage. We also attempted curriculum learning by filtering for easier examples in early training, but this did not overcome the zero-reward issue.\n\nTo address this, we designed a **loose reward** function that relaxed evaluation criteria, allowing the model to receive credit for partially correct or approximately matched outputs. This approach helped bootstrap the model’s learning. Once stable training progress was achieved, we transitioned to the **strict** reward rubric to further refine and push performance beyond this initial plateau.\n\n1. **Auxiliary Tool-Use Reward**\n    \n    Our motivation for introducing **extra** **search call rewards** is not to encourage excessive use of the search tool, but rather to **de-bias the model’s early reluctance to use external tools**. In the initial phase of training, we observed that the model **tends to over-rely on its internal knowledge and avoids invoking external tools**, even when they would be helpful. The additional reward is designed to encourage exploration of retrieval actions during this early phase.\n    \n    Importantly, in contrast to R1-searcher, **our reward design does not give higher returns to search-based solutions when the internal knowledge is sufficient**. Instead, **we treat search-free and search-based correct trajectories equally**, which helps the model avoid unnecessary search calls once it becomes more capable.\n    \n    To better understand the effect of our reward scheme, we analyzed the frequency of the extra search-call reward (value = 3.0) throughout training. Every 10 training steps (each step producing 448 trajectories), we measured how often this reward was triggered. As shown below, the reward is naturally **phased out over time**, confirming its role as a **transient scaffold** in early learning:\n    \n    | Step Range | Reward = 3.0 (count) | Percentage |\n    | --- | --- | --- |\n    | 0–9 | 198 | 4.5% |\n    | 10–19 | 140 | 3.1% |\n    | 20–29 | 104 | 2.3% |\n    | 30–39 | 49 | 1.1% |\n    | 40–49 | 42 | 0.9% |\n    | 50–59 | 43 | 1.0% |\n    | 60–69 | 27 | 0.6% |\n    | 70–80 | 6 | 0.1% |\n    \n    This trend shows that the extra search reward is **adaptive**, mainly active in the early training stages to de-bias the initial underuse of retrieval, and naturally diminishing as the model matures.\n    \n    We will clarify this behavior and the motivation in the revised version of the paper.\n    \n\n---\n\n> It is hard to understand the benefits of DeepDiver over existing agentic tool-augmented approaches, such as ReAct or WebGPT-style agents, which are not included as comparison systems in this work.\n> \n\n**Response:**\n\nOur framework is indeed aligned with the **ReAct paradigm**. Each query is handled through a single trajectory that alternates between *reasoning* and *acting*—i.e., thinking, issuing a search query, processing the retrieved results, and planning the next step—following the core principles of ReAct.\n\nIn fact, all tool-augmented baselines compared in our paper are **built on top of the ReAct framework**, including systems that perform multi-step reasoning and tool invocation. Our contribution lies in extending this framework with improved **search control**, **tool-use precision**, and **learning signals from reward models**, enabling better performance on complex queries that require deep research.\n\nWhile WebGPT-style agents focus on browsing based instead of searching / RAG agent, therefore not included in our baselines.\n\n---\n\n> What’s the rationale behind not explicitly modeling the search?\n> \n\nThank you for the insightful question. We distinguish between two types of tools in agent environments: those that can be effectively simulated by LLMs, and those that cannot.\n\nTools like software applications or APIs often involve **deterministic interaction patterns** and do not require **real-time factual grounding**. These can be simulated by LLMs to teach agents how to operate them.\n\nSearch engines, however, are fundamentally different. While the interaction logic—such as issuing queries or selecting snippets—can be simulated, the retrieved content cannot. Search results depend on real-time, external, and verifiable information that lies beyond the model’s parameters. Simulating such content would result in fabricated or outdated information, defeating the purpose of using a search engine.\n\nCrucially, training robust agents requires more than just learning to issue queries—it requires iteratively **verifying information, resolving conflicts across sources, and reasoning over real-world content**. These abilities depend on exposure to authentic, up-to-date information, which cannot be reliably simulated.\n\nIn summary, although the interface behavior of search tools can be mimicked, the content’s factuality and variability make real search access essential for training agents with strong reasoning and verification capabilities.\n\n---\n\n> In practice, how much impact would the search latency create with increasing search depth?\n> \n\n**Response:**\n\nSearch latency primarily depends on **concurrency** during execution. For example, with fewer than 200 concurrent requests, the Bocha search API typically returns results within **9 seconds**. However, at higher levels of concurrency, latency may increase due to **queuing or retry mechanisms** on the API side.\n\nAs search depth increases, **inference latency** can also rise, particularly in multi-turn ReAct-style trajectories. To mitigate this, we leverage **prefix caching** during decoding, which is especially effective in scenarios involving repetitive context patterns. In our tests, prefix caching improved inference **throughput by approximately 50%**, helping reduce latency as reasoning chains grow longer."}, {"review_id": "vcjoJZx8P8", "reviewer": "Reviewer_rFPC", "summary": "The paper explores the task of open-web question answering to evaluate the ability of a model / agent on real-world information gathering and reasoning. The paper has two key contributions: 1) Webpuzzle which provides a more challenging real world benchmark (with both training and test sets) for real-world open-web question answering. 2) Deep-diver - an RL based framework which encourages higher search frequency and depth (by reward shaping i.e. rewarding GRPO rollouts which give correct answer while using search). Quantiative results show that Deep-diver performs favorably as compared to prior open-weight models.", "questions": "* In Sec. 3.2, the authors mention for cold start they use: \"300 real-user questions from our deployed smart assistant\". Can the authors please provide details on this?\n* For iterative RAG mentioned in Sec. 3 for GRPO, what is the exact scaffold used for inference and training (e.g., basic ReACT loop, or more complex iterative multi-step scaffoldsOpenHands etc)\n* The use of extra search call rewards is interesting, but did the authors explore other simpler alternatives such as just providing a small reward for solving with search etc.\n* What is the maximum number of steps in each rollout during training and inference?\n* For closed-source models in Table 1 only gpt4o is evaluated, can the authors also provide results with better models such as o3 or claude-4-sonnet. This will better help understand the performance on the proposed test set. \n* Since it is a new benchmark, thorough evaluation across different baselines (including those with better scaffolds such as OpenHands) can help the reader better understand the difficulty of the proposed benchmark.\n* Also can the authors provide more details on RL training, such as max context, max steps, the training algorithm, use of overlong filtering etc for GRPO.\n* Finally, have the authors tried to compare performance of RL model trained on Webpuzzle with RL model trained on previous benchmarks and training datasets?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n* The paper proposes a challenging Webpuzzle dataset for evaluating information seeking capabilities of agents in real-world scenarios.\n*  The use of extra search call rewards in Sec. 3 is interesting.\n* The paper has some nice insights with regards to correlation of agent behavior with final performance (such as increase in search frequency)\n* The results in Sec. 5.2, for cross-lingual generalization are also interesting\n\nWeaknesses:\n* In Sec. 3.2, the authors mention for cold start they use: \"300 real-user questions from our deployed smart assistant\". Can the authors please provide details on this?\n* For iterative RAG mentioned in Sec. 3 for GRPO, what is the exact scaffold used for inference and training (e.g., basic ReACT loop, or more complex iterative multi-step scaffoldsOpenHands etc)\n* The use of extra search call rewards is interesting, but did the authors explore other simpler alternatives such as just providing a small reward for solving with search etc.\n* What is the maximum number of steps in each rollout during training and inference?\n* For closed-source models in Table 1 only gpt4o is evaluated, can the authors also provide results with better models such as o3 or claude-4-sonnet. This will better help understand the performance on the proposed test set. \n* Since it is a new benchmark, thorough evaluation across different baselines (including those with better scaffolds such as OpenHands) can help the reader better understand the difficulty of the proposed benchmark.", "was_revised": true, "final_justification": "The authors have addressed most of my concerns. I will therefore raise the score.", "rebuttal": "We appreciate your thoughtful review and interest in our work. Below, we clarify the concerns raised and will incorporate the suggested improvements in the final version.\n\n---\n\n> In Sec. 3.2, the authors mention for cold start they use: \"300 real-user questions from our deployed smart assistant\". Can the authors please provide details on this?\n\n**Response:**\n\nThe *\"300 real-user questions from our deployed smart assistant\"* refer to **open-ended** queries collected from internal team members interacting with our in-house assistant system, which uses DeepSeek-V3 and our internal search backend. Unlike WebPuzzle's close-ended QA format, these queries reflect diverse real-world information needs, such as:\n\n* *Should I buy an electric or gasoline car in 2025?*\n\n* *Any flagship TWS headphones worth recommending?*\n\n* *What’s your opinion on xxx event? What impact might it have?*\n\nThese examples cover current events, product comparisons, and opinion-based analysis. We included them in training to **avoid overfitting to narrow, factoid-style QA tasks.**\n\n---\n\n> For iterative RAG mentioned in Sec. 3 for GRPO, what is the exact scaffold used for inference and training (e.g., basic ReACT loop, or more complex iterative multi-step scaffolds like OpenHands)?\n\n**Response:**\n\nWe apologize for the ambiguity in our original description.\n\nOur iterative RAG setup for GRPO uses a **basic ReAct-style loop**, implemented in-house. The inference pattern follows the standard *thought → search → thought →  … → answer* structure, closely aligned with the original ReAct formulation.\n\nFor training, we combine this ReAct-style reasoning loop with the Hugging Face TRL library. GRPO and other RL algorithms are implemented via TRL, while the agent orchestration is handled by our own codebase.\n\n---\n\n> The use of extra search call rewards is interesting, but did the authors explore other simpler alternatives such as just providing a small reward for solving with search etc.?\n\n**Response:**\n\nThank you for the insightful question.\n\nYes, we experimented with simpler reward schemes like that in the R1-searcher baseline, where a small bonus is given for solving with search. However, this led to **over-searching**—models learned to invoke the search tool even when unnecessary.\n\nOur design of the **extra search call reward** specifically addresses the **opposite problem** observed in early training: models were overly reliant on internal knowledge and hesitant to use external tools.\n\nCrucially, **correct answers with and without search receive equal terminal rewards**, avoiding long-term bias toward tool use. As shown in the table below, the frequency of this reward **naturally decays** during training, functioning as a **transient scaffold**:\n\n| Step Range |  Count | Percentage |\n| ---------- | ------------------ | ---------- |\n| 0–9        | 198                | 4.5%       |\n| 10–19      | 140                | 3.1%       |\n| 20–29      | 104                | 2.3%       |\n| 30–39      | 49                 | 1.1%       |\n| 40–49      | 42                 | 0.9%       |\n| 50–59      | 43                 | 1.0%       |\n| 60–69      | 27                 | 0.6%       |\n| 70–80      | 6                  | 0.1%       |\n\nThis confirms the reward’s **temporary role** in overcoming early underuse of retrieval, without promoting long-term overdependence. \n\n---\n\n> What is the maximum number of steps in each rollout during training and inference?\n> \n\n**Response:**\n\nWe cap each rollout at a maximum of 7 reasoning steps, where each step follows a thought → search cycle in the ReAct-style loop.\n\nDuring each step, the model can issue multiple search queries in parallel, rather than being limited to one per step. This results in roughly 20–28 total queries per full rollout.\n\n---\n\n> For closed-source models in Table 1 only gpt4o is evaluated, can the authors also provide results with better models ....\n>\n>Since it is a new benchmark, thorough evaluation across different baselines ....\n> \n\n**Response:**\n\nThank you for the valuable suggestion. As recommended, we conducted additional evaluations on more advanced closed-source models, including **Claude-4-Sonnet**, **Gemini 2.5 Pro**, and **OpenAI’s o3 model**.\n\nWe evaluated the models' *internal knowledge only* performance on our benchmark, with no retrieval support.\n\n**Accuracy by Category**\n\n| Category | o3 | claude-4-sonnet | claude-4-sonnet-thinking | gemini-2.5-pro |\n| --- | --- | --- | --- | --- |\n| **riddles** | **35.86%**  | 15.86%  | 24.14%  | 28.28%  |\n| **CrossQA** | 33.08% | 20.0%  | 20.77%  | **36.15%**  |\n| **Overall** | **34.55%**  | 17.82%  | 22.55%  | 32.00%  |\n\n**Accuracy by Difficulty**\n\n| Level | o3 | claude-4-sonnet | claude-4-sonnet-thinking | gemini-2.5-pro |\n| --- | --- | --- | --- | --- |\n| **medium** | **50.55%**  | 28.57%  | 39.56%  | **50.55%**  |\n| **hard** | **30.00%**  | 11.67%  | 15.0%  | 26.67 |\n| **Extreme Hard** | **25.21%**  | 11.76%  | 13.45% | 20.17%  |\n| **easy** | 20.00%  | **40.00%**  | 20.00% | 40%  |\n\n**Observation**: o3 and Gemini perform competitively with DeepSeek-R1 (32.7%), with o3 excelling on riddles and Gemini on CrossQA. Gemini, however, struggles more with harder tasks.\n\nWe also tested Claude-4-Sonnet and o3 with our ReAct-style iterative RAG scaffold:\n\n**Accuracy by Category**\n\n| Category | o3 | claude-4-sonnet | claude-4-sonnet-thinking |\n| --- | --- | --- | --- |\n| **riddles** | 38.6% (1.05) | **41.4% (2.19)** | 40.7% (2.35) |\n| **CrossQA** | 53.8% (0.96) | 48.5% (1.48) | **58.5% (1.49)** |\n| **Overall** | 45.8% (1.01) | 44.7% (1.85) | **49.1% (1.95)** |\n\n**Accuracy by Difficulty**\n\n| Level | o3 | claude-4-sonnet | claude-4-sonnet-thinking |\n| --- | --- | --- | --- |\n| **medium** | **63.7% (0.62)** | 57.1% (1.66) | 62.6% (1.68) |\n| **hard** | 50.0% (1.0) | 50.0% (1.82) | **55.0% (1.98)** |\n| **Extreme Hard** | 29.4% (1.31) | 30.3% (2.05) | **35.3% (2.15)** |\n| **easy** | 60.0% (1.0) | **100.0 %(1.20)** | 60.0% (1.40) |\n\n*[Numbers in () indicate average search call rounds per example]*\n\n**Observation**: While Claude models lag behind o3 in the no-search setting, they gain 30+ points with retrieval, showing they are agent-compatible and effectively utilize external tools.\n\n**Evaluation of Off-the-Shelf Agent Products**\n\nWe evaluated several commercial agent products and open frameworks on a **subset** of our benchmark (described in Appendix A.3: *Human vs. DeepDiver*), including:\n\n- **ChatGPT Deep Research Mode**\n- **Gemini Deep Research**\n- **OpenHands Cloud (open-source)**\n\n**Human vs. DeepDiver & Other Framework**\n\n| Model | **Accuracy** |\n| --- | --- |\n| **Human** | 44.0 |\n| **GPT-4o (Web Search Enabled)** | 60.0 |\n| **ChatGPT Deep Research** | **80.0** |\n| **Gemini Deep Research** | 52.0 |\n| **OpenHands Cloud** | 21.3 |\n| **DeepDiver** | 40.0 |\n\n**Insights:**\n\nChatGPT Deep Research outperforms all others, including humans, but requires ~10 minutes per query, making it impractical for most real-time use cases. OpenHands Cloud underperforms mainly due to CAPTCHA issues blocking search/browse access.\n\n---\n\n> Finally, have the authors tried to compare performance of RL model trained on Webpuzzle with ...\n> \n\n**Response:**\n\nThank you for the thoughtful question.\n\nYes, as presented in Section 5.2, we compared our DeepDiver with models trained on prior datasets:\n\n- **R1-Searcher**: trained on a Wikipedia-based search and data.\n- **DeepResearcher**: trained with open-Internet search but still based on Wikipedia corpora.\n\nDespite being trained entirely in Chinese, DeepDiver was evaluated on English benchmarks and search engines to test its cross-lingual generalization:\n\n| Model | WebPuzzle-en | BamBoogle | FRAMES | HotpotQA |\n| --- | --- | --- | --- | --- |\n| R1-Searcher | 13.7 (1.9) | 46.7 (2.0) | 25.3 (1.9) | 57.9 (2.3) |\n| DeepResearcher  | 15.0 (7.5) | 53.9 (7.1) | **33.6** (7.2) | 56.6 (4.4) |\n| DeepDiver-Qwen | **26.1** (14.7) | **56.8** (9.1) | 32.0 (14.2) | **58.4** (10.4) |\n\n**Key Observations:**\n\n- **DeepDiver** outperforms both prior methods on **WebPuzzle** by a great margin.\n- It also performs competitively on **wiki-based benchmarks**, despite **never being trained on English corpora**.\n\nWe attribute this to **SIS**, which encourages more robust and tool-driven information gathering and verification.\n### Behavioral Comparison: Information-Seeking Patterns\n\nAs detailed in Appendix A.2, we compared the information-seeking behavior of each model using metrics from Section 2.2:\n\n| Methods | Model | WebPuzzle-en | BamBoogle | FRAMES | HotpotQA |\n| --- | --- | --- | --- | --- | --- |\n| **Reflection & Correction** | R1-Searcher | 0.04 | 0.03 | 0.04 | 0.02 |\n|  | DeepResearcher | 0.20 | 0.11 | 0.13 | 0.07 |\n|  | DeepDiver-Qwen | **0.45** | **0.25** | **0.28** | **0.27** |\n| **Conflict Resolution** | R1-Searcher | 0.06 | 0.02 | 0.02 | 0.02 |\n|  | DeepResearcher | 0.16 | 0.07 | 0.08 | 0.06 |\n|  | DeepDiver-Qwen | **0.31** | **0.23** | **0.32** | **0.18** |\n| **Verification & Denoising** | R1-Searcher | 0.17 | 0.18 | 0.13 | 0.10 |\n|  | DeepResearcher | 0.32 | 0.15 | 0.24 | 0.18 |\n|  | DeepDiver-Qwen | **1.72** | **1.80** | **1.60** | **1.54** |\n\nThese findings reinforce two important points:\n\n1. **WebPuzzle is more challenging** than traditional wiki-based datasets, requiring more complex reasoning and tool usage.\n2. **DeepDiver** exhibits richer agent behaviors—iterative verification, reflection, and conflict handling—critical for open-domain performance.\n\n---\n\n> Also can the authors provide more details on RL training, such as max context, max steps, the training algorithm, use of overlong filtering etc for GRPO.\n> \n\n**Response:** \n\nqueries_per_step: 32\n\ngroup_size: 14\n\ntemperature: 0.9\n\nkl_coef: 0.001\n\nmax_tool_call_rounds: 7\n\nmax_length: 22k\n\noverlong_reward: 0\n\n---\n\nThank you for your thoughtful question! We will revise our manuscripts according to your precious comments!"}]

Author rebuttal:

Meta-review: This paper proposed a novel way to scale up the performance of search-based tasks in LLMs. It proposed a new dataset and evaluation metrics called WebPuzzle, which is carefully designed to test and induce model's effective use of search as a tool in RL training. The result shows good scaling of search behavior and competitive results.

Paper strength

Novel method that scales well in RL.
Thorough evaluation to show the scaling behavior.
Release of a new training and evaluation set WebPuzzle, which could be valuable to the broader community.

Paper weakness

Most of the weaknesses raised by the reviewers have been addressed in the rebuttal phase.
The authors acknowledged that the size of the evaluation dataset is on the smaller side, which is limited by cost of curation.

One Reviewer didn't respond to the rebuttal (rating=4). Upon close inspection, AC thinks all the concerns in the original review have been adequately addressed.

AC recommends acceptance of the paper based on the importance of the task, contribution of novel datasets, and nice thorough experiments to show the effectiveness.
Decision: Accept (spotlight)