Title: DeepDiver: Adaptive Web-Search Intensity Scaling via Reinforcement Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Agentic Reinforcement Learning

Retrieval Augmented Generation

Search Intensity Scaling

Data Synthesis
Abstract: Information seeking demands iterative evidence gathering and reflective reasoning, yet large language models (LLMs) still struggle with it in open-web question answering. Existing prompting and supervised fine-tuning (SFT) methods remain fixed by prompt rules or training corpora, and are usually benchmarked only on well-structured wiki sources, limiting real-world adaptability. We introduce $\textbf{WebPuzzle}$, a 24k-sample training and 275-sample test benchmark that evaluates information seeking on the live internet, across both wiki and open-domain queries. Leveraging 7k WebPuzzle instances, we develop $\textbf{DeepDiver}$, a reinforcement-learning (RL) framework that cultivates $\textbf{Search Intensity Scaling (SIS)}$—an emergent ability to escalate search frequency and depth instead of settling on overconfident, under-evidenced answers. With SIS, Qwen2.5-7B-Instruct and Pangu-7B-Reasoner attain performance on real-web tasks comparable to the 671B-parameter DeepSeek-R1. We detail DeepDiver’s curriculum from cold-start SFT to a well designed RL procedure, and show that its seeking policy generalized from closed-ended queries to open-ended generation such as long-form writing. Our results advance adaptive information seeking in LLMs and provide a rigorous benchmark for future work.

DeepDiver: Adaptive Web-Search Intensity Scaling via Reinforcement Learning

Abstract

Information seeking demands iterative evidence gathering and reflective reasoning, yet large language models (LLMs) still struggle with it in open-web question answering. Existing prompting and supervised fine-tuning (SFT) methods remain fixed by prompt rules or training corpora, and are usually benchmarked only on wellstructured wiki sources, limiting real-world adaptability. We introduce WebPuzzle , a 24k-sample training and 275-sample test benchmark that evaluates information seeking on the live internet, across both wiki and open-domain queries. Leveraging 7k WebPuzzle instances, we develop DeepDiver , a reinforcement-learning (RL) framework that cultivates Search Intensity Scaling (SIS) -an emergent ability to escalate search frequency and depth instead of settling on overconfident, underevidenced answers. With SIS, Qwen2.5-7B-Instruct and Pangu-7B-Reasoner attain performance on real-web tasks comparable to the 671B-parameter DeepSeek-R1. We detail DeepDiver's curriculum from cold-start SFT to a well designed RL procedure, and show that its seeking policy generalized from closed-ended queries to open-ended generation such as long-form writing. Our results advance adaptive information seeking in LLMs and provide a rigorous benchmark for future work.

1 Introduction

Information seeking [34] is a fundamental cognitive skill that involves iterative evidence gathering, reflective reasoning, and the resolution of conflicting information. Despite significant advancements in artificial intelligence, LLMs continue to struggle with replicating such information-seeking behaviors. Knowledge-intensive question answering, a central challenge for LLMs, requires a robust capability for information seeking. Current models often fail to determine when and what information to seek, verify the relevance of evidence, and reason effectively over noisy or conflicting contexts.

Iterative Retrieval-Augmented Generation (RAG) [15] frameworks have been proposed to address these challenges by alternating between retrieval and reasoning. Existing approaches generally fall into two categories: prompting-based and task-specific supervised fine-tuning (SFT). Promptingbased methods leverage predefined rules or in-context learning (ICL) [3], forcing the LLM to follow a specific pipeline to complete complex tasks [11, 23, 32, 42, 16]. However, the fixed reasoning flow provided in the prompts limits their adaptability to complex, dynamic problems. In contrast, supervised fine-tuning methods train models to improve retrieval and reasoning capabilities [2, 41], generally yielding better performance. However, these methods often internalize inference patterns tied to the training corpus, restricting generalization to more dynamic or unseen situations.

Recently, reinforcement learning (RL) [26, 13] has been applied to enhance inference-time reasoning in LLMs, enabling iterative refinement and exploration of reasoning [6, 31, 19]. Several studies

have integrated RL into iterative RAG frameworks, encouraging models to explore diverse reasoning paths and rewarding accurate outcomes [12, 25, 4, 45]. However, these works predominantly train and evaluate their methods on well-structured datasets such as HotpotQA [39], which are based on corpora like Wikipedia. In such settings, many tasks can be effectively solved using the LLMs' internal knowledge, and the introduced search environments are 'clean,' containing minimal noise or conflicting information. In contrast, real-world search environments are inherently more complex-characterized by noisy, inconsistent, and unreliable sources. This discrepancy limits the generalizability of the reported 'incentivized search capabilities' to more realistic, open-ended information-seeking scenarios.

To investigate RL-guided LLM behaviors in more realistic, open-domain scenarios, we introduce WebPuzzle , a dataset designed to evaluate information-seeking capabilities in real-world search environments. WebPuzzle contains 24k training samples and 275 human-annotated test examples, covering tasks solvable with Wikipedia content as well as broader open-domain queries extracted from open-web environment. Even Wikipedia subset are rigorously validated to require external retrieval, ensuring a realistic assessment of LLMs' search behaviors. Along with WebPuzzle, we introduce DeepDiver , an RL-driven search and reasoning framework trained on this dataset. DeepDiver interacts with real-world search engines, continuously refining and denoising retrieved documents to provide accurate answers. A key innovation of DeepDiver is the emergent capability of search intensity scaling (SIS) , which dynamically scales up the search frequency and depth as information demands increase. This enables LLMs to tackle more complex, information-intensive problems under open-web environment. Together, WebPuzzle and DeepDiver provide a comprehensive framework for developing and examining information seeking ability of LLMs, offering a promising approach for knowledge-intensive problem solving.

Through systematic empirical analysis, we identify critical factors that influence model behavior, including the search intensity, the training environment, and the generalization capabilities. Our analysis reveals several key insights: (1) DeepDiver exhibits exceptional information-seeking ability via adaptive SIS, where the depth and frequency of searching proportional to both problem difficulty and the model's performance. (2) Compared to the 'clean' Wiki-based environment, WebPuzzle and real-world search settings better support complex reasoning beahviours, guiding LLMs to actively supplement evidence, resolve conflicts, verify content, and reflect for self-correction. (3) RL training significantly enhances the generalization capability of LLMs, enabling the transition from closed-ended to open-ended problems. In conclusion, our method underscores the potential of reinforcement learning to foster emergent adaptive search behaviors-specifically, search intensity scaling-in LLMs. This significantly enhances their ability to perform adaptive, verifiable, and scalable information seeking, providing a promising direction for future advancements in knowledgeintensive problem solving.

2 Preliminaries

2.1 Iterative RAG

Weformulate the iterative Retrieval-Augmented Generation (RAG) framework for question answering. Given a question q , the model iteratively performs reasoning and retrieval to produce an answer.

At each iteration t ∈ { 1 , 2 , . . . , T } , the model maintains a reasoning history H t -1 = { q, ( r 1 , s 1 , d 1 ) , . . . , ( r t -1 , s t -1 , d t -1 )} , where r i represents the intermediate CoT generated at round i , s i denotes search queries, and d i denotes retrieved documents from web search.

At round t , conditioned on history H t -1 , the model first generates intermediate reasoning r t ∼ p ( r t ∣ H t -1 ) to analyze the current status. Then, based on the reasoning r t , the model selects one of two actions: (1) Search: generate additional queries s t ∼ p ( s t ∣ H t -1 , r t ) and retrieve supporting documents d t = Retrieval ( s t ) ; (2) Answer: finalize the answer a ∼ p ( a ∣ H t -1 , r t ) to question q . This iterative reasoning-and-retrieval process continues until the model chooses the answer action, resulting in a final answer that is well supported by retrieved evidence and explicit reasoning steps.

Reason

Search

𝐻

!"#

𝐻

!

(a)

Reason

Search

𝐻

!"#

𝐻

!

𝐷

!

$

𝐷

!

%

conflict

(b)

𝐻

!

𝐷

!

"

𝐷

!

#

𝐷

!

$

…

𝐷

!

%

…

Reason

Search

𝐻

!&%

(c)

Reason

Search

𝐻

!"#

𝐻

!

𝐻

$!%#

conflict

𝐻

!

𝐻

$!%#

𝐻

!

&

(d)

2.2 Information Seeking Behaviour

We define information seeking behaviour within iterative RAG frameworks as a structured decisionmaking process: at each iteration the model adopts specific strategies to resolve uncertainties, improve evidence quality, and enhance the overall reliability of answers. Formally, at iteration t , conditioned on the reasoning history H t -1 and current intermediate reasoning r t , the model exhibits several strategies to guide its search and reasoning processes.

Inspired by the findings of Gandhi et al. [7], we categorize these strategies into four types of information seeking behaviours: (1) Evidence Gathering & Supplements , where the model actively seeks to fill identified knowledge gaps by formulating targeted queries s t and retrieving supporting documents d t , formally represented as ( s t , d t ) ∼ p ( s t , d t ∣ H t -1 , r t ) , where d t = Retrieval ( s t ) . This strategy is exemplified by traditional question-answering datasets such as 2Wiki [9], HotpotQA [39], and FRAMES [14]; (2) Conflict Resolution , where the model reasons about inconsistencies and evaluates competing claims when retrieved information contains contradictions; (3) Verification & Denoising , where the model cross-checks facts and isolates trustworthy information from noisy or irrelevant retrieved content; and (4) Reflection & Correction , where the model periodically re-assesses its reasoning trajectory, revisits earlier assumptions, and explicitly corrects previous reasoning steps for iterative refinement. The latter three behaviours can be represented generally as generating reasoning steps r t ∼ p ( r t ∣ H t -1 , d t ) , where the specific conditions (e.g. presence of contradiction, noise, or previous mistakes) differ according to each behaviour.

Existing works adopting the wiki-based datasets, limiting their scope to structured and well-organized knowledge bases, and thus predominantly emphasize 'Evidence Gathering & Supplements'. To prove this observation, we show an detailed analysis in Appendix A.2. In contrast, our proposed WebPuzzle and the real-world searching environment explicitly necessitates employing all four behaviours, thereby reflecting a more comprehensive and realistic scenario for real-world problem-solving with web-searching. More details about WebPuzzle will be included in section 3.1.

3 Method

In this section, we discuss the details of our approach. We begin by introducing WebPuzzle, a dataset designed to address real-world reasoning and search challenges. Next, we describe DeepDiver, a reinforcement learning-based training framework aimed at enhancing LLMs with robust capabilities introduced in section 2.2.

3.1 WebPuzzle

Unlike existing open-domain QA datasets based on Wikipedia [39, 21, 14] where LLMs often perform well using only internal knowledge, we introduce WebPuzzle , a dataset designed to evaluate LLMs' ability to locate and reason over noisy, scattered information on the open web. Figure 2 illustrates our data synthesis and curation processes.

Candidate Data Generation We collect candidate data from Wiki-corpus and real-user queries with retrieved webpages from our deployed smart assistant service. Our generation involves two

Open Domain

Webpages

Wiki Pages

Entity Selection

Context Hiding

Riddle

Generation

Cross-Page

QA Generation

Cross-Page QA

(10k)

Open Riddle

(7.5k)

(6k)

WebPuzzle Full Set

DeepDiver

Training Set

WebPuzzle

(24k)

Pass@k-based

Difficulty Tagging

Outliers (7.5k)

N/A (1k)

WebPuzzle

(7k)

Easy (8k)

Med. (5.5k)

Hard (3k)

pass@k = 0

Easy (200)

Med. (3.4k)

Hard (2.4k)

Sampling

for

DeepDiver

approaches: (1) Cross-page question generation, where an LLM extracts facts from web pages to generate 'inverted' questions, answers and checklists [29, 14]-applied only to open web pages as Wiki-corpus tends to produce overly-simple questions; and (2) Riddle creation, where the LLM selects distinctive entity attributes and applies obfuscation or generalization to create challenging problems, with original entities as labels. Examples appear in Appendix D.1. More quality assurance protocols are shown in Appendix D.3.

Difficulty Assessment To ensure stable RL training with consistent reward signals, we tag each problem's difficulty level, enabling a data mixture strategy that prevents all-zero rewards which could lead to training collapse. For each problem, we test DeepSeek-R1 four times, using the number of correct answers to determine difficulty. The formal definition appears in Appendix D.4, with the statistics of the dataset presented in Table 7 and tagging workflow in Appendix E.6.

Test Set Annotation Unlike the training set which used LLM labeling, our test set was manually annotated by 5 human experts using an open-web search engine. From 500 seed samples, experts followed the principles in Appendix D.2 to ensure meaningful evaluation of LLMs' informationseeking behaviors. Through iterative annotation, we finalized 275 samples for testing.

3.2 DeepDiver

Building upon the WebPuzzle, we showcase its efficacy within a RL framework designed to explore the information-seeking behavior of LLMs. In this section, we present our method, DeepDiver. DeepDiver ultilize the procedure of cold-start supervised fine-tuning (SFT) followed by reinforcement learning (RL), while incorporates a carefully designed reward assignment and scheduling mechanism to maintain stable RL training.

Initialization of Reasoning and Searching To equip DeepDiver with essential reasoning and searching capabilities for WebPuzzle, we implement a cold-start supervised fine-tuning process using diverse data: 2,000 WebPuzzle samples across difficulty levels, 300 real-user questions from our deployed smart assistant, 2,200 general reasoning problems from recent studies [10, 17, 35, 37, 43], and 1,000 real-user queries concatenated with retrieved documents. This dataset distills responses from DeepSeek-R1, establishing DeepDiver's foundational abilities to iteratively search and reasoning over retrieved documents. The distillation prompt configuration is detailed in Appendix E.6.

GRPO With Iterative RAG After SFT, we enhance DeepDiver by extending GRPO [24] with iterative RAG. As shown in Figure 3, the model iteratively performs reasoning and searching until reaching an acceptable answer, following the pipeline in Section 2.1. We apply a loss mask to distinguish model-generated from externally retrieved tokens, with GRPO updating parameters based solely on model-generated content.

Extra Search Call Rewards Beyond standard format and accuracy rewards in GRPO, we introduce an extra reward to encourage search engine use for complex problems. When no search-free rollouts solve a problem but at least one search-enabled rollout succeeds, we assign an additional reward of

Reason

Search

𝑑𝑜𝑐.

𝑟𝑒𝑎𝑠𝑜𝑛𝑖𝑛𝑔

Loop

Answer

𝑎𝑛𝑠𝑤𝑒𝑟

Grader

query

ground truth

reward

𝑟𝑒𝑎𝑠𝑜𝑛𝑖𝑛𝑔

𝑑𝑜𝑐.

𝑟𝑒𝑎𝑠𝑜

𝑖𝑛𝑔

𝑎

𝑠𝑤𝑒𝑟

Answer After Reason-Search Loop

(a)

GRPO Update

𝑟𝑒𝑎𝑠𝑜𝑛𝑖𝑛𝑔

𝑑𝑜𝑐.

𝑟𝑒𝑎𝑠𝑜𝑛𝑖𝑛𝑔

𝑎

!

𝑟𝑒𝑎𝑠𝑜𝑛𝑖𝑛𝑔

𝑑𝑜𝑐.

𝑟𝑒𝑎𝑠𝑜𝑛𝑖𝑛𝑔

𝑎

"

…

𝑟𝑒𝑎𝑠𝑜𝑛𝑖𝑛𝑔

𝑑𝑜𝑐.

𝑟𝑒𝑎𝑠𝑜𝑛𝑖𝑛𝑔

𝑎

#

𝑟

!

𝑟

"

𝑟

#

…

𝐴

!

𝐴

"

𝐴

#

…

LLM

Extra Search

Reward

Format

Reward

Reward

DeepDiver

(b)

1 . 0 to the successful search-enabled solutions. This ensures the model learns to leverage external tools when necessary. The formal definition appears in Appendix E.3.

Loose and Strict Rewards Our reward function employs LLM-based graders in a two-stage training approach that transitions from loose to strict grading: the loose grader assigns scores from 1 to 10 (scores ≥ 6 yield 1.0 reward), particularly benefiting early training as shown in Section 5.4. The strict grader conducts three evaluation rounds, requiring at least 2 of 3 positive judgments. Both grader definitions appear in Appendix E.2.

4 Experiments

4.1 Setup

Data Mixture and Selection Due to computational constraints and capability limits of the 7B model, we train DeepDiver on a carefully selected mixture of 7k WebPuzzle samples rather than the full dataset. We evenly split these into 2k samples for cold-start SFT (Section 3.2) and 5k for RL training. This mixture strategy balances computational efficiency and model effectiveness. Detailed statistics appear in Table 8.

Benchmark Datasets and Baseline Models We evaluate performance using closed-ended Chinese benchmarks including C-simpleQA-500 [33, 8], FRAMES-zh-230 [14], BamBoogle-zh-71 [21], and our proposed WebPuzzle (detailed in Appendix E.4). For trainable baselines, we use Qwen2.5-7BInstruct [30] and Pangu-7B-Reasoner [28] as backbone models. Training-free baselines include QwQ32B [31], GPT-4o [20] and DeepSeek-R1 [6]. We evaluate methods including Prompted without Web Search , Prompted with Iterative RAG , and R1-Distillation (detailed in Appendix E.5). Our evaluation uses the strict grader from Section 3.2, which considers both reference answers and checklists for more robust assessment than conventional LLM-as-a-judge [44] approaches. Grader details appear in Appendix E.2.

4.2 How does DeepDiver's Performance Compare to Baselines?

Our proposed DeepDiver demonstrates substantial improvements over distillation-based methods and achieves performance comparable to state-of-the-art models such as DeepSeek-R1 and QwQ. As shown in Table 1, Qwen powered by DeepDiver achieves a 10-point improvement over the cold-start model on WebPuzzle, reaching 37.6 accuracy. DeepDiver-Qwen2.5-7B also outperforms the R1-distilled model (37.6 versus 29.8), highlighting the effectiveness of our RL training pipeline.



















































DeepDiver-Pangu-7B shows similar improvements. While R1-distilled Pangu-7B quickly hits performance bottlenecks (dropping 5.7 points on Bamboggle compared to the cold-start model), DeepDiverPangu-7B breaks through these limitations, showing substantial improvements across all benchmarks and achieving exceptional performance (38.1) on WebPuzzle. In conclusion, both Pangu-7B-reasoner and Qwen DeepDiver demonstrate competitive performance against high-performing models like DeepSeek-R1 and QwQ. This highlights DeepDiver's capability to effectively search for and reason over relevant information, and solve complex reasoning tasks through search intensity scaling.

4.3 What is the Relationship between Search Intensity and Performance?

Search intensity is strongly correlated with performance improvements, as increases in search frequency and depth during the RL phase consistently lead to better outcomes. Figure 4 illustrates this relationship during the training phase, showing a clear trend: as search engine calls increase, so do training rewards. For testing results, despite SFT's progress compared with the prompting methods,

 





























  















 

















































































the model faces a performance bottleneck to adapt to more challenging problems, still lagging behind off-the-shelf APIs with large margin. In contrast, our RL-based DeepDiver-Qwen2.5-7B promotes higher search intensity with an average of 2.51 search and reasoning rounds, substantially higher than the SFT model's 1.75. Similar gains appear in DeepDiver-Pangu-7B, where increased search rounds ( 1 . 84 → 2 . 89 ) correspond to performance improvements ( 30 . 3 → 38 . 1 ). This searching intensity scaling enables models to explore and verify more relevant information, enhancing their ability to tackle complex problems.

4.4 Can DeepDiver Generalize from Open-web Training to OOD Wiki-based Problem?

Training with WebPuzzle, DeepDiver demonstrates strong generalization capabilities and performance improvements on Wiki-based problems. DeepDiver shows impressive generalization on Wikibased benchmarks despite not being specifically trained for these tasks. Both DeepDiver-Qwen2.5-7B and DeepDiver-Pangu-7B significantly outperform their distilled variants and demonstrate substantial improvements over cold-start models. While DeepSeek-R1 performs well on Wiki-based problems without web search, it shows modest gains when combined with iterative RAG pipeline. This suggests DeepSeek-R1 may have already internalized the necessary knowledge for Wiki-based problems, highlighting the importance of our proposed WebPuzzle benchmark. We further investigate this hypothesis through isolated tests on information seeking and verification in section 5.1.

5 Analysis

This section focuses on the Qwen2.5-7B-Instruct model, a simpler model comparing with the Pangu7B-Reasoner. We analyze several key aspects, including isolated evaluations of information-seeking behavior, comparisons with concurrent related work, the design of the reward function, and the model's generalization to open-ended problems. Additional analyses-such as the relationship between search intensity and problem difficulty, statistics of information-seeking behavior across different training and testing environments, comparisons between human and DeepDiver performance, and detailed case studies -are provided in Appendix A.

5.1 Isolation Testing of Information-Seeking

While DeepDiver lags behind models such as QwQ and DeepSeek-R1 on certain datasets in Section 4, our primary focus is investigating information-seeking behavior rather than knowledge memorization. This raises a question: When isolating evaluation to focus purely on information seeking ability, how does DeepDiver compare to strong baselines?

Setup We conduct pairwise comparisons between DeepDiver and each baseline. For each pair, we perform k = 3 tests without web search to evaluate whether problems can be solved using internal knowledge alone. We calculate the pass @ k rate to filter out problems solvable by both models, then analyze accuracy on the remaining problems with the iterative RAG pipeline.

Results DeepDiver exhibits exceptional information-seeking capabilities, comparable to all baselines on problems that cannot be solved by internal knowledge alone. While our 7B DeepDiver initially trails behind 671B baselines in full-set tests, results shift when isolating information-seeking behavior. As Figure 5 shows, on problems challenging even for larger models, DeepDiver demonstrates competitive performance across all benchmarks. Notably, it outperforms DeepSeek-R1 across all domains, with a 5.1-point lead on WebPuzzle. This suggests our 7B model's limitations in full-dataset performance stem primarily from its smaller size limiting internal knowledge. However, when tackling problems requiring external information search and verification, DeepDiver's information-seeking capability demonstrates strength in addressing real-world open-web problems.

5.2 Comparisons with Wiki-based Methods

To highlight wiki-based training environments' limitations, we compare DeepDiver with prior wikibased methods. Despite being trained entirely in Chinese, we evaluate DeepDiver on English benchmarks with English search engines to demonstrate its robustness and generalizability.

Setup We use R1-Searcher [25] and DeepResearcher [45] as baselines-both trained in English using Wiki-based corpora. Search engine settings appear in Appendix E.8. For evaluation, we translate WebPuzzle into English via Qwen Max [30], use the full Bamboogle dataset [21] (125 examples), and randomly sample 300 examples from FRAMES [14] and HotpotQA [39]. For fairness, we report accuracy based on the average judgment across all methods.

Results Despite the language gap, DeepDiver-trained in a real-world Chinese internet setting using WebPuzzle queries-outperforms Wiki-based baselines on most tasks, underscoring the strength of SIS. As shown in Table 2, DeepDiver significantly outperforms DeepResearchers on WebPuzzle-en with 11.1 point leads, while maintaining strong results on Wiki-based datasets despite no English training for information-seeking. We attribute this to DeepDiver's use of SIS, which enables intensive information retrieval and verification rather than relying on limited internal knowledge or languagespecific constraints. R1-Searcher and DeepResearcher make significantly fewer search calls than DeepDiver due to their "cleaner" and more constrained training environments, leading to poorer real-world performance when facing the noise and complexity of open information-seeking tasks. For additional results, including individual judge assessments, see Table 9.

5.3 Emergence of the SIS

A natural concern is that SIS could be an artifact of reward shaping rather than a genuine behavior that emerges from training in a real web environment. To address this concern, we analyze whether the extra search-call reward introduced in Section E.3 consistently encourages the model to prefer search over no-search when both solve the task.

Setup Theoretically, recall the bonus is only awarded when at least one search-enabled rollout succeeds and no search-free rollout succeeds for the same prompt group, i.e., it should not reward search when both search and no-search solve the task. Consequently, during training, we tracked the frequency of the extra bonus. Every 10 steps (448 trajectories per step), we counted how often the bonus fired and compared this against the evolving search intensity (average number of tool-use rounds per query).

Results SIS is not merely reward-shaped but an emergent behavior developed during RL training, our reward design serves as a transient early scaffold (not persistent incentive) and differs fundamentally from existing works that explicitly encourages the tool calls. As shown in Figure 6, the



 





 

 

 

 

 





































































































































special search reward exhibits a clear phasing-out trend: its trigger percentage drops from 4.5% (198 occurrences) in steps 0-9 to 0.1% (6 occurrences) in steps 70-80. Two key observations confirm its transient role: (1) Even in the earliest training phase, only a small fraction of trajectories (4.5%) received the reward, ruling out "over-rewarding"; (2) After step 30, the trigger percentage remains ≤ 1.1%, indicating the reward is essentially inactive in mid-to-late training. More importantly, Figure 4 shows that tool-use rounds grow sharply during steps 80-120-well after the auxiliary reward fades. This confirms SIS is not a reward-driven "phenomenon" but an emergent behavior: the model proactively leverages external tools to compensate for internal knowledge limitations, even without direct incentives.

5.4 Tolerance of the Reward Function

During DeepDiver-Qwen7B's RL training, we observed a reward plateau after approximately 80 optimization steps. We investigated potential factors including learning rate scheduler, exploration diversity, environmental instability, and gradient issues, but found no obvious problems. We therefore focused on the reward function design as a potential cause of the performance plateau.

Setup Starting from checkpoints obtained after 80 optimization steps, we compared DeepDiver's performance under continued training with two different reward functions: the loose and strict

rewards introduced in Section 3.2. Both guided continued training from steps 80 to 120. We evaluated performance on WebPuzzle test sets, analyzing accuracy and search intensity trends.

Results A looser reward function stabilizes the initial training phase of RL, while a stricter reward function helps overcome bottlenecks in the later stages. Our results show that a looser reward function stabilizes early RL training, but continuing with it doesn't always lead to improvements. As Figure 7 shows, when transitioning from loose rewards (first 80 steps) to stricter rewards, we observed a nearly 9-point performance increase on WebPuzzle (from 29.1 to 37.6), compared to almost no improvement when continuing with loose rewards. On FRAMES-zh-230, continued training with loose rewards caused a sharp 7-point performance drop, while the stricter reward function continued driving performance upward.

5.5 Generalization to Open-ended Problems

DeepDiver is trained exclusively on closed-ended WebPuzzle problems, adaptively scaling search intensity based on complexity. We investigate whether these capabilities can generalize to open-ended tasks like long-form writing.

Setup We evaluate DeepDiver on ProxyQA [27] against R1-Distilled baselines. Since DeepDiver generates Chinese responses, we translate all ProxyQA meta-questions and sub-questions for evaluation. Testing prompt and evaluator configuration follow the original study. We analyze generalization benefits gained through RL training compared to distillation.

Results RL training significantly enhances the generalization capability of LLMs, enabling transition from closed-ended to open-ended problems and demonstrating strong adaptability to long-form writing tasks. As shown in Table 6, our RL-guided DeepDiver achieves 32 . 72% , outperforming the R1-distilled model by 9 . 47 percentage points. This suggests RL training enables more effective information seeking and validation in open-web environments, resulting in more comprehensive responses. Additionally, DeepDiver's response length and search queries are substantially higher than the distilled model's, providing evidence that search intensity scaling encourages active information acquisition for more comprehensive answers.

6 Related Work

Prompting-based strategies-including in-context learning [3] and retrieval-augmented chain-ofthought pipelines [11, 32, 23]-enable zero- or few-shot question answering, yet their fixed templates rarely adapt retrieval depth to unforeseen information gaps. Supervised fine-tuning (SFT) improves the synergy between retrieval and generation [2, 41] but can overfit corpus-specific inference patterns, hindering transfer to noisy settings. Reinforcement-learning (RL) methods let LLMs decide when and what to search, achieving state-of-the-art results on curated benchmarks such as HotpotQA [12, 25, 4, 45, 39], yet they remain evaluated mostly in 'clean' Wikipedia-style environments. Beyond these directions, tool-augmented agents that interleave reasoning with web search [18, 40, 38] similarly demonstrate promise but still rely on limited test beds, underscoring the need for benchmarks that reflect real-world, noisy information-seeking scenarios. Additional introduction of the related works are shown in Appendix B.

7 Conclusion

We conducted a comprehensive investigation into various aspects of information-seeking behavior in LLMs for solving real-world, knowledge-intensive problems. Our findings indicate that an RL-driven framework, when combined with open-web search engines, enables LLMs to scale search intensity and adapt to tasks of varying difficulty levels. We introduced WebPuzzle, a large-scale dataset designed specifically for developing and testing LLMs' information-seeking behavior, and DeepDiver, a 7B parameter LLM powered by WebPuzzle, which demonstrates competitive performance when compared to the 671B DeepSeek-R1 model on knowledge-intensive tasks. Additionally, we explored key factors influencing RL training and the behavior of LLMs. Our work empowers LLMs to spontaneously adapt their seeking behavior, contributing to advancements in the field and providing extensive insights into the information-seeking capabilities of LLMs in real-world tasks.

References

Anthropic. Introducing the model context protocol, November 2024. URL https://www. anthropic.com/news/model-context-protocol . Accessed: 2025-04-13.

Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection, 2023. URL https://arxiv.org/ abs/2310.11511 .

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020. URL https://arxiv.org/abs/2005.14165 .

Mingyang Chen, Tianpeng Li, Haoze Sun, Yijie Zhou, Chenzheng Zhu, Haofen Wang, Jeff Z. Pan, Wen Zhang, Huajun Chen, Fan Yang, Zenan Zhou, and Weipeng Chen. Research: Learning to reason with search for llms via reinforcement learning, 2025. URL https://arxiv.org/ abs/2503.19470 .

Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Do not think that much for 2+3=? on the overthinking of o1-like llms, 2025. URL https://arxiv.org/abs/2412.21187 .

DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URL https://arxiv.org/abs/2501.12948 .

Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D. Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars, 2025. URL https://arxiv.org/abs/2503.01307 .

Yancheng He, Shilong Li, Jiaheng Liu, Yingshui Tan, Weixun Wang, Hui Huang, Xingyuan Bu, Hangyu Guo, Chengwei Hu, Boren Zheng, Zhuoran Lin, Xuepeng Liu, Dekai Sun, Shirong Lin, Zhicheng Zheng, Xiaoyong Zhu, Wenbo Su, and Bo Zheng. Chinese simpleqa: A chinese factuality evaluation for large language models, 2024. URL https://arxiv.org/abs/2411. 07140 .

Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps, 2020. URL https: //arxiv.org/abs/2011.01060 .

Jinhao Jiang, Zhipeng Chen, Yingqian Min, Jie Chen, Xiaoxue Cheng, Jiapeng Wang, Yiru Tang, Haoxiang Sun, Jia Deng, Wayne Xin Zhao, Zheng Liu, Dong Yan, Jian Xie, Zhongyuan Wang, and Ji-Rong Wen. Enhancing llm reasoning with reward-guided tree search. arXiv preprint arXiv:2411.11694 , 2024.

Zhengbao Jiang, Frank F. Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. Active retrieval augmented generation, 2023. URL https://arxiv.org/abs/2305.06983 .

Bowen Jin, Hansi Zeng, Zhenrui Yue, Dong Wang, Hamed Zamani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516 , 2025.

L. P. Kaelbling, M. L. Littman, and A. W. Moore. Reinforcement learning: A survey, 1996. URL https://arxiv.org/abs/cs/9605103 .

Satyapriya Krishna, Kalpesh Krishna, Anhad Mohananey, Steven Schwarcz, Adam Stambler, Shyam Upadhyay, and Manaal Faruqui. Fact, fetch, and reason: A unified evaluation of retrieval-augmented generation, 2024. URL https://arxiv.org/abs/2409.12941 .

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks, 2021. URL https://arxiv.org/abs/2005.11401 .

Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. Search-o1: Agentic search-enhanced large reasoning models, 2025. URL https://arxiv.org/abs/2501.05366 .

Yingqian Min, Zhipeng Chen, Jinhao Jiang, Jie Chen, Jia Deng, Yiwen Hu, Yiru Tang, Jiapeng Wang, Xiaoxue Cheng, Huatong Song, Wayne Xin Zhao, Zheng Liu, Zhongyuan Wang, and Ji-Rong Wen. Imitate, explore, and self-improve: A reproduction report on slow-thinking reasoning systems. arXiv preprint arXiv:2412.09413 , 2024.

Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browser-assisted question-answering with human feedback, 2022. URL https://arxiv.org/abs/2112.09332 .

OpenAI. Learning to reason with llms, 2024. URL https://openai.com/index/ learning-to-reason-with-llms/ .

OpenAI. Hello gpt-4o. https://openai.com/index/hello-gpt-4o/ , 2024.

Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models, 2023. URL https: //arxiv.org/abs/2210.03350 .

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL https://arxiv.org/abs/1707.06347 .

Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen. Enhancing retrieval-augmented large language models with iterative retrieval-generation synergy, 2023. URL https://arxiv.org/abs/2305.15294 .

Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv.org/abs/ 2402.03300 .

Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. R1-searcher: Incentivizing the search capability in llms via reinforcement learning, 2025. URL https://arxiv.org/abs/2503.05592 .

Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction . The MIT Press, second edition, 2018. URL http://incompleteideas.net/book/the-book-2nd. html .

Haochen Tan, Zhijiang Guo, Zhan Shi, Lu Xu, Zhili Liu, Yunlong Feng, Xiaoguang Li, Yasheng Wang, Lifeng Shang, Qun Liu, and Linqi Song. Proxyqa: An alternative framework for evaluating long-form text generation with large language models, 2024. URL https://arxiv. org/abs/2401.15042 .

Yehui Tang, Fangcheng Liu, Yunsheng Ni, Yuchuan Tian, Zheyuan Bai, Yi-Qi Hu, Sichao Liu, Shangling Jui, Kai Han, and Yunhe Wang. Rethinking optimization and architecture for tiny language models. arXiv preprint arXiv:2402.02791 , 2024.

Yixuan Tang and Yi Yang. Multihop-rag: Benchmarking retrieval-augmented generation for multi-hop queries, 2024.

Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL https:// qwenlm.github.io/blog/qwen2.5/ .

Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025. URL https://qwenlm.github.io/blog/qwq-32b/ .

Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions, 2023. URL https://arxiv.org/abs/2212.10509 .

Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. Measuring short-form factuality in large language models, 2024. URL https://arxiv.org/abs/2411.04368 .

Thomas D. Wilson. Models in information behaviour research. Journal of Documentation , 55 (3):249-270, 1999. doi: 10.1108/EUM0000000007145. URL https://doi.org/10.1108/ EUM0000000007145 .

Guohai Xu, Jiayi Liu, Ming Yan, Haotian Xu, Jinghui Si, Zhuoran Zhou, Peng Yi, Xing Gao, Jitao Sang, Rong Zhang, Ji Zhang, Chao Peng, Fei Huang, and Jingren Zhou. Cvalues: Measuring the values of chinese large language models from safety to responsibility, 2023.

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report. arXiv preprint arXiv:2407.10671 , 2024.

Jianxin Yang. Firefly. https://github.com/yangjianxin1/Firefly , 2023.

Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan. Gpt4tools: Teaching large language model to use tools via self-instruction, 2023. URL https://arxiv. org/abs/2305.18752 .

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering, 2018. URL https://arxiv.org/abs/1809.09600 .

Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models, 2023. URL https: //arxiv.org/abs/2210.03629 .

Tian Yu, Shaolei Zhang, and Yang Feng. Auto-rag: Autonomous retrieval-augmented generation for large language models, 2024. URL https://arxiv.org/abs/2411.19443 .

Zhenrui Yue, Honglei Zhuang, Aijun Bai, Kai Hui, Rolf Jagerman, Hansi Zeng, Zhen Qin, Dong Wang, Xuanhui Wang, and Michael Bendersky. Inference scaling for long-context retrieval augmented generation, 2025. URL https://arxiv.org/abs/2410.04343 .

Ge Zhang, Yemin Shi, Ruibo Liu, Ruibin Yuan, Yizhi Li, Siwei Dong, Yu Shu, Zhaoqun Li, Zekun Wang, Chenghua Lin, et al. Chinese open instruction generalist: A preliminary release, 2023.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. URL https://arxiv.org/ abs/2306.05685 .

Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. Deepresearcher: Scaling deep research via reinforcement learning in real-world environments. arXiv preprint arXiv:2504.03160 , 2025.

Appendix

A Further Analysis

A.1 Search Intensity vs. Difficulty

To showcase DeepDiver's ability to dynamically adjust search intensity based on problem complexity, we examine the relationship between search intensity and accuracy across fractions with varying difficulty levels in the WebPuzzle. This analysis follows the experimental setup outlined in Section 4 and compares our proposed DeepDiver with the DeepSeek-R1 baseline.

Results DeepDiver demonstrates significant benefits from adaptive search intensity scaling, where search intensity is proportional to both problem difficulty and the LLM's performance. As shown in table 3, across all difficulty levels, both DeepDiver and the baseline models show an increasing number of search call rounds as problem complexity rises. However, DeepDiver consistently consumes more search calls, which translates into better performance. In particular, when compared to DeepSeek-R1, DeepDiver outperforms it in the hard and outlier fractions by a large margin. Specifically, DeepDiver achieves a notable 3.7-point performance leading, driven by an average of 2.6 search rounds compared to DeepSeek-R1's 1.59. This demonstrates that DeepDiver, empowered by open-web environment and reinforcement learning, exhibits superior performance on more complex problems.

An interesting observation arises when examining the performance of DeepSeek-R1 on the Wiki Riddle fraction. Although equipped with the iterative RAG pipeline, DeepSeek-R1 experiences a 1.9-point performance drop (from 32.9 to 31.0). We hypothesize that this decline is due to knowledge conflicts between the pre-trained internalized Wiki corpus and the real-world open-web environment, which introduces confusion and hallucination that hinders the model's ability to correctly answer the question. These results further validate the effectiveness of the WebPuzzle and DeepDiver.

A.2 Statistics of Information-Seeking Behaviors

To further analyze the differences between our WebPuzzle dataset and wiki-based datasets, as well as to conduct an in-depth investigation into the behavior of our DeepDiver model, we computed the proportions of different information-seeking behaviors (defined in section 2.2) across various models on multiple datasets. The detailed results are presented in Table 4.

Setup We developed an automatic pipeline based on GPT-4o [20] to identify and count the occurrences of different behaviors in the reasoning chains of model outputs. We primarily focused on three behaviors: Reflection & Correction, Conflict Resolution, and Verification & Denoising, while omitting the statistics for Evidence Gathering & Supplements due to its overly fundamental nature. Detailed prompt specifications can be found in Figure 16. The baseline models, evaluation benchmarks, and analysis setup align with those in Experiment 5.2.

Results The WebPuzzle proves to be more challenging compared to wiki-based datasets, requiring more complex information-seeking behaviors. Additionally, DeepDiver's reasoning chains exhibit richer and more diverse patterns. As shown in tabel 4, The behavioral statistics of WebPuzzle-en surpass those of wiki-based datasets across nearly all models, particularly in Reflection & Correction and Conflict Resolution behaviors. Notably, for the HotpotQA dataset - the primary training data for both baseline models - we observe a substantial reduction in the proportion of complex behaviors required compared to WebPuzzle when evaluating the same model, further highlighting the importance of more open-web training data. Additionally, DeepDiver, trained on WebPuzzle with real search engine integration, demonstrates richer and more sophisticated information-seeking behaviors compared to various baseline models.

A.3 Human vs. DeepDiver

To assess the difficulty of WebPuzzles and compare human performance with that of DeepDiver, we conducted human testing on a subset of the WebPuzzle evaluation dataset. Human performance was evaluated by tracking the number of web-search queries and web pages browsed during the problem-solving process, which were then compared with the performance of DeepDiver.

Setup Five human experts, who were not involved in the annotation of the evaluation sets, participated in the human evaluation. Each expert was tasked with answering 5 problems, creating a subset of the evaluation set. During testing, the number of search queries and web pages browsed were recorded. Consistent with the experimental setup, we report the average accuracy of the human evaluators and compare it against DeepSeek-R1 and our proposed DeepDiver to highlight the challenges posed by WebPuzzle.

Results WebPuzzle presents significant challenges, even for human experts, who face difficulties with numerous searches and reasoning steps. As shown in Table 5, human evaluators achieved an accuracy rate of 44.0%, requiring an average of 6.16 search queries and browsing 9.28 web pages to solve the problems. In comparison, DeepSeek-R1 made 7.37 search queries with 1.94 rounds of searching. Despite DeepSeek-R1 performing the most search rounds among all baselines and humans, it achieved a relative lower accuracy of 38.7%. In contrast, our proposed DeepDiver conducted 10.69 search queries across 2.68 rounds, significantly surpassing the other methods in terms of search effort. This additional effort allowed DeepDiver to more thoroughly collect and verify evidence, leading to a 40.0% accuracy rate, which is closer to human performance. These results suggest that DeepDiver follows a more comprehensive information-seeking process, better aligning with the approach taken by human evaluators.

A.4 Case Study

In this section, we conduct an error analysis and case study on the response generated from DeepSeekR1, R1-distilled Qwen-7b and our DeepDiver. We explain the reason why the DeepDiver outperforms the R1-distilled model and show competitive performance compared with the DeepSeek-r1 using one typical example.

Results DeepDiver demonstrates exceptional information-seeking ability while incorporating correction on reasoning history and retrieved documents, providing a more robust and adaptable solution for overcoming the limitations of flawed internal knowledge. Specifically, as shown in Table 10, DeepSeek-R1 leverages its rich internal knowledge to quickly narrow the exploration scope, consistently demonstrating the ability to list the correct answer among candidate options in the first round, showcasing remarkable knowledge retention. Taking advantage of this, R1 can focus more on verifying whether candidate answers satisfy all constraints when designing search queries, allowing it to find the correct answer in fewer rounds. However, the R1-distilled Qwen-7B attempts to mimic DeepSeek-R1's behavior but lacks error correction when internal knowledge is flawed. Specifically, the R1-distilled Qwen suggests 'Nico Hülkenberg' in the first round of searching and reasoning, but fails to resolve conflicting conditions in subsequent rounds due to limited internal knowledge, ultimately producing a faulty answer.

In contrast, without R1-level internal knowledge, DeepDiver compensates by increasing search intensity to acquire more relevant external documents. This results in 7 generated search queries across rounds 1 and 2 to explore diverse documents, rather than relying on limited internal knowledge. Furthermore, in rounds 2-3, DeepDiver encounters a potential candidate, 'Chaz Mostert,' but allocates only 1 query for validation, compared to 3 queries for continued exploration. This persistent exploration enables the model to identify the correct answer by round 3. The search and reasoning strategy, empowered by SIS, aids DeepDiver in delivering a correct and acceptable answer.

A.5 Results on ProxyQA

We evaluate DeepDiver on ProxyQA [27] against R1-Distilled baselines. Since DeepDiver generates Chinese responses, we translate all ProxyQA meta-questions and sub-questions for evaluation. Testing prompt and evaluator configuration follow the original study. We analyze generalization benefits gained through RL training compared to distillation.

Table 6: Results on ProxyQA. The results include accuracy rate, number of search calls, rounds of search, the number of search queries, and response length.

B Additional Related Works

Prompting-based strategies. Few-shot in-context learning [3] allows frozen LLMs to imitate reasoning patterns from exemplars, while RAG with CoT prompts [11, 32, 23] interleave search queries with intermediate thoughts to inject fresh evidence. These methods require no training, but their step counts and query formats are anchored to the prompt, so the model cannot escalate effort when initial evidence is missing. As a result, they often stop searching too early or hallucinate unsupported facts on hard, open-web questions.

Supervised fine-tuning (SFT). Retrieval-augmented generation systems fine-tuned on gold passages-e.g., Self-RAG and Auto-RAG [2, 41]-learn to quote and merge the external text into answers, reducing hallucinations on Wikipedia benchmarks. Nevertheless, SFT tends to overfit the inference paradigm tied to the training corpus; performance drops when pages are noisy, multilingual, or partially missing, which is common in open-web internet environment.

Reinforcement learning (RL). RL integrated iterative RAG pipeline offers a principled way for LLM agents to decide when and what to search. Recent work trains search-capable agents with reward shaping and curricula, achieving state-of-the-art accuracy on HotpotQA and similar wikibased corpus [12, 25, 4, 45]. Yet these studies use closed Wikipedia environments, where every answer is guaranteed to exist and pages are clean.

Tool-augmented agents. ReAct-style frameworks combine chain-of-thought with executable actions, letting models call a browser or other tools between reasoning steps [18, 40, 38]. They excel at citing fresh evidence and correcting themselves mid-trajectory, but they still struggle with irrelevant pages and and hard to dealing with the real-world open-web environment.

Group Relative Policy Optimization Group Relative Policy Optimization (GRPO) [24] is a reinforcement learning algorithm designed to enhance the efficiency of Proximal Policy Optimization (PPO) [22] by eliminating the need for critic network. Specifically, GRPO samples multiple outputs from a previous policy for a given prompt and computes their average reward to serve as a dynamic baseline. The advantage of each output is defined relative to this baseline, resulting in positive advantages for outputs exceeding the baseline and negative advantages otherwise. Formally, we define the GRPO objective as follow:

where the relative advantage A i is computed as:

GRPO retains PPO's clipping strategy and incorporates a KL-divergence term for regularization to ensure the stable training of RL.

C Discussion: Limitations and Extensions

In this work, we aimed to provide a deeper understanding of the information-seeking behavior of large language models (LLMs) with respect to real-world open-web challenges. While we observed that DeepDiver outperforms multiple baselines in several cases, we cannot claim that DeepDiver is the most optimized solution for enabling LLMs to solve complex problems in a broad sense. Below, we outline several limitations of our work and potential areas for future exploration.

Curation of WebPuzzle We argue that DeepSeek-R1 excels at solving problems by leveraging internal knowledge rather than extensively exploring the internet. As a result, we designed the WebPuzzle to assess and test the LLM's information-seeking abilities. However, over time, there is a possibility that cutting-edge LLMs will internalize the knowledge sources used in WebPuzzle, which could lead to scenarios similar to those seen in Wiki-based problems. Moreover, the curation pipeline heavily relies on the utilization of DeepSeek-R1, introducing some potential bias in the testing process. Therefore, developing effective benchmarks to assess the evolving capabilities of LLMs will remain an ongoing challenge.

RL-Driven Open-ended Problems Solving We demonstrate that DeepDiver, trained on closedended open-web problems, exhibits strong generalization capabilities when applied to open-ended tasks, such as long-form writing. However, developing a more effective RL-driven framework that directly enhances the model's ability to solve these problems remains an open challenge. Since the RL framework relies on a stable reward signal, the lack of reliable metrics to assess open-ended content complicates the task of defining such signals. As a result, designing a framework that can handle both open-ended and closed-ended problems remains a key area for future exploration. One potential approach could following the approaches of ProxyQA, involving crafting sub-questions in an online way during RL training, which may offer a promising direction for enhancing the model's performance on open-ended tasks.

The Border Between Cold Start SFT and RL DeepDiver is powered by a cold-start SFT and RL pipeline, which first initializes the model's capabilities through SFT training, followed by RL training. However, there is no established guideline on the optimal extent to which SFT should be conducted before transitioning to RL training. This presents a challenge, as maintaining stable and effective RL training requires researchers to experiment with various proportions and combinations of SFT samples, while continuously monitoring the RL training process. Future improvements could involve an adaptive pipeline, allowing the LLM to dynamically switch from SFT to RL training when necessary.

The Extension of Tool Usage Search engines are commonly viewed as tools that LLMs can utilize to enhance their capabilities. In our study, we focused solely on investigating the reasoning and searching behavior of LLMs with the aid of search engines. This limits the scope of our work, as there are various other tools, such as those compatible with the Model Context Protocol (MCP) [1], which could also contribute to improving reasoning and searching processes in knowledge-intensive tasks. Future research could expand on this by considering the integration of additional tools to further enhance the performance of LLMs in such tasks.

Scalability with Respect to Model Size and Sequence Length Due to computational constraints, the experiments presented in this report are limited to a 7B model with a maximum sequence length of 20k tokens. This limitation restricts the generalizability of our findings to larger models (e.g., 13B, 32B) and longer sequence lengths (e.g., 32k, 64k), and it also limits our exploration of DeepDiver's upper performance boundaries. Currently, DeepDiver's performance is predominantly constrained by both model size and response length. Extending the training and evaluation to encompass these configurations in future work would provide valuable insights into the model's scaling behavior. Such an extension would help assess DeepDiver's performance, robustness, and applicability across different model sizes and sequence lengths, offering a more comprehensive understanding of its real-world potential.

Problem of Over-searching Prior work has shown that reasoning LLMs trained in RL environments often suffer from 'overthinking' [5], generating excessively long reasoning sequences-sometimes spanning thousands of tokens-even for simple questions. We observe a similar phenomenon in our DeepDiver model, which invokes significantly more search calls compared to other baselines, even when evaluated on simple tasks. This over-searching behavior highlights an inefficiency that future work should aim to mitigate. A promising direction would be to develop methods that reduce both the search and reasoning overhead in LLM-based systems.

D Detailed Dataset Curation

D.1 Examples of the WebPuzzle

In this section, we show two examples of the WebPuzzle sampled from the cross-page question and riddles respectively.

Solution: 149 RMB

Example of Cross-page Question

2024 年 10 月 31 日 ， 一 名 身 高 125cm 的 普 通 成 人 游 客 购 买 深 圳 欢 乐 谷 万 圣 节 夜 场 票 ， 需 支 付 多 少 钱 ？

Translation: On October 31st, 2024, how much does an ordinary adult tourist with a height of 125 cm need to pay for the night ticket of the Halloween event at Happy Valley Shenzhen?

Solution: 149 RMB

Example of Riddle

某 中 国 社 交 媒 体 应 用 因 美 国 政 府 的 禁 令 而 意 外 获 得 大 量 关 注 ， 用 户 自 称 为 ' 难 民 ' 并 迁 移 到 该 平 台 。 该 应 用 在 苹 果 应 用 商 店 一 度 下 载 量 排 名 第 一 ， 超 过 ChatGPT 。 其 母 公 司 总 部 位 于上 海 。 这 是 哪 个 应 用 ？

Translation: A Chinese social media app unexpectedly gained a great deal of attention due to a U.S. government ban. Users refer to themselves as 'refugees' and have migrated to this platform. At one point, the app ranked first in downloads on the Apple App Store, surpassing ChatGPT. Its parent company is headquartered in Shanghai. Which app is this?

Solution: 小 红 书 / Rednote / Xiaohongshu

D.2 Principles to Annotate the Evaluation Set

The experts were required to adhere to the following principles: (1) The answer to the posed question must be definitive and unique. (2) The answer should be derived from internet search results, not from common-sense knowledge. (3) Answering the question should involve multiple searches, thorough reasoning, and validation, rather than a simple query. (4) The answer must be accessible and solvable through sufficient rounds of searching and reasoning.

D.3 Quality Assurance of WebPuzzle

We prioritize recently updated pages not fully covered by LLMs' knowledge cutoffs , emphasizing open-web searches for problem-solving. We exclude offensive, politically sensitive, ethically concerning, or NSFW content using an LLM-based filter. Ambiguous, controversial, multiple-choice and boolean questions are removed to prevent answer hacking. We also eliminated unsolvable problems to ensure low error rates, ultimately collecting 24k WebPuzzle training samples.

D.4 Diffculty Level Tagging

Formally, let N correct denote the number of correct answers out of the 4 tests. The difficulty level D of a problem can be determined as:

E Detailed Experimental Setup

E.1 Statistic of WebPuzzle

E.2 Reward Defination

Formally, Let G i denote the similarity score assigned by the looser grader for the i -th response. The reward E i assigned by the looser grader is defined as:

E.4 Benchmarks

We evaluate the performance of all models using the following benchmarks:

WebPuzzle : Our proposed novel, web-based question-answering benchmark designed to assess models' deep information-seeking abilities within a real-world web environment. WebPuzzle serves as an in-domain task that evaluates a model's capacity to extract and process information from the web.

C-SimpleQA-500 : A randomly sampled subset of 500 instances from C-SimpleQA [33], C-SimpleQA is a Chinese-translated version designed to assess the factuality of language models. While not explicitly designed for complex or real-time question answering, we utilize C-SimpleQA to explore the impact of web search scaling on simpler informationseeking tasks.

FRAMES-zh-230 : Asubset of the FRAMES [14] benchmark with 230 samples that requires multi-hop information-seeking. The queries are translated into Chinese, and annotators verify whether the golden answer can be retrieved via interactions with our web search API. Only test cases where the golden answer is reachable are included in the evaluation.

Bamboogle-zh-71 : A subset of the Bamboogle [21] benchmark consists of 71 samples, processed similarly to the FRAMES-zh-230 dataset.

E.5 Baseline Methods

The following baseline methods are evaluated:

For the stricter grader, the response undergoes three rounds of evaluations, each providing a binary judgment, J k i ∈ 0 , 1 , where k ∈ 1 , 2 , 3 represents the corresponding round. The reward E i is assigned only if at least two out of the three rounds agree that the response is semantically equivalent to the reference. The stricter grader's reward mechanism is defined as:

the strict grader evaluates the generated response by comparing it to the reference answer and the checklists over three rounds. In each round, the evaluator determines whether the model's output matches the reference answer and aligns with the requirements specified in the checklists. If the evaluator deems the response correct in at least two out of the three rounds, the result is classified as correct; otherwise, it is considered incorrect. The accuracy rate, derived from this evaluation process, is reported as the primary metric in our results.

E.3 Extra Tool Call Rewards Assignment

Formally, the triggering condition for extra rewards is defined as:

where G is the group of rollouts, S i indicates whether the i -th rollout uses a search engine, and C i indicates success. The extra reward is assigned only when no search-free rollouts succeed and at least one search-enabled rollout succeeds. In such cases, a +1.0 reward is added to successful search-enabled rollouts.

Formally, we apply the extra reward with:

Prompted without Web Search : In this setup, LLMs respond to problems based on a single round of prompting without web access. The model generates answers based solely on its preexisting knowledge. We include off-the-shelf LLMs such as Qwen2.5-7B-Instruct [30, 36], QwQ-32B [31], GPT-4o [20], DeepSeek-R1 [6] and Pangu-7B-Reasoner [28].

Prompted with Iterative RAG : In this approach, LLMs answer problems through multiple rounds of retrieval and reasoning, using a real-world open-web search engine. The baselines from the 'Prompted without Web Search Methods' are tested using the same iterative RAG workflow in our approach (see Appendix E.6 for implementation details).

R1-Distillation : We performed SFT on Qwen2.5-7B-Instruct and Pangu-7B-Reasoner using a combined dataset that integrates both the cold-start data (introduced in Section 3.2) and the dataset used during the RL training phase of DeepDiver. Both models are distilled using the responses generated by DeepSeek-R1.

E.6 Iterative RAG Prompting

In our experimental framework, which includes WebPuzzle data tagging (Section 3.1), cold start data construction (Section 3.2), DeepSeek-R1 distillation, and multiple iterative RAG baselines (Section 4.1), we utilize prompt-based approaches to implement iterative RAG. We have specifically designed a prompt where, in each round, we evaluate the model's response to decide whether to terminate the process. If retrieval is triggered, the retrieved results are concatenated in the next user turn to facilitate continued reasoning and searching. For a detailed description of our prompt design, please refer to Figure 12.

E.7 Grader Details

During the training of the RL model, both a loose grader and a strict grader are involved. The loose grader performs a single evaluation and provides a score ranging from 1 to 10, with a score of 6 or above considered correct. This loose grader is used in the early stages of training to enhance training signals and ensure stability. In contrast, the strict grader performs three rounds of validation using different prompts, each producing a binary classification of 'correct' or 'incorrect.' The final result is determined by majority voting across the three validations. This grader is used in the later stages of training to prevent model manipulation and further improve performance. Both graders are implemented base on qwen-turbo API * . For details on the specific grader prompts, please refer to Figure 14 and 15.

E.8 Implementation Details

During training, each data sample undergoes 14 rollouts with a sampling temperature of 0.9. We employ a batch size of 32 and a learning rate of 1e-6, training for a single epoch with a KL divergence coefficient of 0.001. The maximum number of tool call round is set to 7. For online search, we utilize the Bocha † search engine for Chinese scenario and LangSearch ‡ for English scenario, retaining only the top 2 results per search query to ensure efficiency.

F Case Study Examples

In this section, we present a case study on the WebPuzzle, comparing the performance of our proposed DeepDiver model with the R1-distilled model and DeepSeek-R1. The results are shown in Table 10.

F.1 ProxyQA Case

In this section, we present an example of DeepDiver's results on ProxyQA. We compare the performance of DeepDiver-Qwen2.5-7B with its R1-distilled variant. The original R1 model only requires a single round of reasoning due to its strong internal knowledge. However, as shown in Figure 9, the distilled model replicates the single round of reasoning without possessing the same internal knowledge. This lack of depth leads to overconfidence, causing the model to prematurely provide an answer after just one round of reasoning. In contrast, Figure 8 demonstrates that our proposed DeepDiver consistently searches and reasons over multiple rounds to gather the necessary knowledge required to answer the problem, resulting in a more thorough and comprehensive output.

Problem

他

职

业

生

涯

中

曾

打

破

一

级

方

程

式

最

年

轻

获

得

积

分

车

手

的

纪

录

，

先

后

效

力

过

三

支

不

同

顶

级

车

队

的

F1

阵

营

，

并

两

度

经

历

席

位

调

换

的

特

殊

安

排

，

曾

于

德

国

大奖

赛

助力

某

中

游

车

队

斩

获

2008

年

后

的

首

个

领

奖

台

，

转

战

耐

力

赛

领

域

后

，

他

26

号

赛

车

在

多

项

赛

事中

缔

造

了

新

的

历史

这

位

车

手

究竟

是

谁

？

Solution

丹

尼尔

·

科

维

亚

特

R1-

Prompting

<think>

:

...

维

斯

塔

潘

保

最

年

轻

积

分

纪

录

...

霍

肯

博

格

在

2010

年

拿

积

分

时

年

龄

未

破

纪

录

...

科

维

亚

特

（

红

牛

二

队

属

中

游

但

耐

力

赛

未

26

号

车

）

...

<search>

: ['F1

最

年

轻

获

得

积

分

车

手

纪

录

', '

效

力

三

支

F1

车

队

的

车

手

德

国

大奖

赛

领

奖

台

', '2019

德

国

大奖

赛

领

奖

台

车

队

', 'WEC 26

号

车车

手

F1

背

景

']

Search Results:

...correct answer is retrieved by '2019

德

国

大奖

赛

领

奖

台

车

队

'

<think>

:

...

维

斯

塔

潘

排

除

...

科

维

亚

特

需

校

验

...

亚

历

山

大

·

阿

尔

本

需

确

认

是

否

存

在

其

他

车

手

关

联

。

<search>

: ['

科

维

亚

特

效

力

F1

车

队

数

量

', '2019

德

国

大奖

赛

红

牛

二

队

是

否

中

游

车

队

', 'F1

车

手

WEC 26

号

车

', '

丹

尼尔

·

科

维

亚

特

耐

力

赛

']

Search Results:

...(correct answer involved)

<think>

:

...

科

维

亚

特

符

合

'2008

年

后

中

游

车

队

首

登

德

国

站

领

奖

台

'

条

件

...

其

他

如

霍

肯

博

格

阿

尔

本

均

不

符

合

全

部

条

件

。

<answer>

:

...

丹

尼尔

·

科

维

亚

特

√

...

Qwen7b-R1-

Distill

<think>

:

...

初

步

判

断

可

能

为

尼

科

·

霍

肯

伯

格

,...

<search>

: ['F1

最

年

轻

积

分

车

手

记

录

保

者

', '

尼

科

·

霍

肯

伯

格

席

位

调

换

历史

', '2014

年

德

国

大奖

赛

索

伯

纳

尔

逊

·

皮

奎

特

', '

勒

芒

24

小

时

赛

26

号

车

霍

肯

伯

格

']

Search Results:

...(correct answer is not included)

<think>

:

...

尼

科

·

霍

肯

伯

格

从

未

登

上

过

领

奖

台

，

这

与

德

国

站

季

军

存

在

冲

突

...

<search>

: ['2014

年

德

国

大奖

赛

索

伯

车

队

成

绩

', 'F1

最

年

轻

积

分

车

手

记

录

保

者

', '

勒

芒

24

小

时

赛

26

号

车

手

霍

肯

伯

格

']

Search Results:

...(correct answer is not included)

<think>

:

矛

盾

点

处

理

：

F1

历史

数

据

显

示

2014

年

德

国

站

中

游

车

队

（

非

顶

级

车

队

）

的

最

好

成

绩

为

马

萨

（

法

拉

利

车

队

季

军

），

而

索

伯

车

队

该

站

最

佳

为

霍

肯

伯

格

季

军

。

此

处

可

能

存

在

用

户

将

'

中

游

车

队

'

误

解

为

'

非

顶

级

车

队

'

的

表

述

误

差

，

需

在

回

答

中

明

确

说

明

。

<answer>

:

...

尼

科

·

霍

肯

伯

格

×

...

DeepDiver

<think>

:

...

需

要

验

证

是

否

存

在

符

合

所

有条

件

的

车

手

...

<search>

: ['F1

最

年

轻

积

分

车

手

纪

录

打

破

者

', 'F1

三

支

不

同

车

队

席

位

调

换

', '

德

国

站

中

游

车

队

2008

年

领

奖

台

', '

耐

力

赛

26

号

车

手

']

Search Results:

...(correct answer is not included)

<think>

:

...

安

东

内

利

和

维

斯

塔

潘

都

涉

及

F1

年

轻

纪

录

，

但

需

要

更

具

体信

息

。

其

中

的

26

号

车

由

Chaz

，

但

需

确

认

是

否

符

合

所

有条

件

。

<search>

: ['2023-2025

年

德

国

站

中

游

车

队

领

奖

台

', '26

号

赛

车

F1

耐

力

赛

', 'F1

三

支

不

同

车

队

席

位

调

换

', 'Chaz Mostert

车

手

履

历

']

Search Results:

...

<think>

:

...

需

要

验

证

2023-2025

年

间

是

否

有

关

于

效

力

三

支

F1

车

队

、

席

位

调

换

、

26

号

赛

车

使

用

的

公

开

信

息

。

可

能

存

在

未

被

检

索

到

的

2025

赛

季

德

国

站

中

游

车

队

领

奖

台

记

录

，

需

补

充

检

索

2025

赛

季

F1

德

国

站

官

方

成

绩

。

<search>

经

历

', '2023-2025

德

国

站

中

游

车

队

领

奖

台

', '26

号

赛

车

F1

车

手

', 'F1

三

支

不

同

车

队

席

位

调

换

']

Search Results:

...[14]

网

页

标

题

：

...

科

维

亚

特

在

2022

年

纳

斯

卡

杯

系

列

赛

中代

表

赫

泽

伯

格

车

队

出

场

三

次

，

他

的

赛

车

使

用

与他

在

一

级

方

程

式

赛

车

中

相

同

的

号

码

26...

在

赛

季

揭

幕

战

赛

百

灵

1000

英

里

耐

力

赛

（

1000 Miles of Sebring

）

上

，

科

维

亚

特

获

得

第

三

名

，

首

次

登

上

WEC

领

奖

台

。

...(correct answer is retrieved by '26

号

赛

车

F1

车

手

')

<think>

:

...

丹

尼尔

·

科

维

亚

特

符

合

耐

力

赛

26

号

特

征

...

<answer>

:

...

丹

尼尔

·

科

维

亚

特

√

...

Blue: Exploring possibilities and widening scope. Orange: Narrowing down to incorrect candidates. Green: Narrowing down to right answer. Italics : Faulty conflict resolution.

Blue: Exploring possibilities and widening scope. Orange: Narrowing down to incorrect candidates. Green: Narrowing down to right answer. Italics : Faulty conflict resolution.

Reasoning and Search Rounds of ProxyQA - DeepDiver

Proxy Question:

你

能

详

细

介

绍

一下世

界

上

最

辣

的

辣

椒

吗

，

包

括

它

的

起

源

、

培

育

过

程

以

及

导

致

其

极

高

辣

度

的

因

素

？

Response:

1.

用

户

询

问

的

是

'

世

界

最

辣

的

辣

椒

'

，

首

先

需

要

明

确

具

体

是

哪

种

辣

椒

，

因

为

可

能

有

很

多

种

被

称

作

'

最

辣

'

的

，

比

如

印

度

的

、

阿

富

汗

的

Bihun

辣

椒

等

，

但

需

要

确

认

权

威

来

源

。

2.

需

要

查

找

该

辣

椒

的

起

源

地

，

比

如

是

否

产

自

某

个

国

家

或

地

区

，

以

及

是

否

有

特

定

的

栽

培

历史

。

3.

培

育

过

程

可

能

涉

及

抗

病

性

、

耐

热

性

等

改

良

，

需

要

检

索

相

关农

业

技

术

或

育

种

论

文

。

4.

热

度

因

素

可

能

涉

及

辣

椒

素

（

如

辣

椒

素

、

二

氢

辣

椒

素

含

量

）

、

辣

椒植

株

结

构

（

如

表

皮

厚

薄

）

、

生

长

环

境

（

温

度

、

湿

度

）

等

，

需

要

具

体

数

据

支

。

5.

需

要

验

证

信

息

的

权

威

性

，

比

如

是

否

被

吉

尼

斯

世

界

纪

录

或

权

威

植

物

数

据

库

认证

。

6.

可

能

需

要

中

英

文

搜

索

结

合

，

覆

盖

更

多

资

料

来

源

。

...

ROUND 1:

[

世

界

最

辣

的

辣

椒

具

体

品

种

辣

椒

起

源

,

辣

椒

素

含

量

与

辣

度

关

系

,

辣

椒

育

种

技

术最

辣

品

种

]

...

1.

根

据

最

新

检

索

结

果

，

X

辣

椒

被

吉

尼

斯

认证

为

269

万

SHU

，

而

卡

罗

莱

纳

死

神

为

220

万

，

为

135.9

万

，

需

明

确

当

前

世

界

最

辣

品

种

。

2.

需

要

补

充

X

辣

椒

的

培

育

细

节

，

包

括

杂

交

品

种

及

技

术

手

段

。

3.

需

要

验

证

不

同

辣

椒

的

培

育

年

限

、

抗

病

性

等

技

术

参

数

。

4.

热

度

因

素

部

分

需

要

具

体

分

析

辣

椒

素

分

布

、

植

株

结

构

等

科

学

数

据

。

5.

需

要

对

各

辣

椒

的

起

源

地

进

行

交

叉

验

证

，

例

如

是

否

确

实

源

自

英

国

。

6.

需

要

权

威

机构

的

最

新数

据

，

如

2023

年

吉

尼

斯

认证

结

果

。

7.

用

户

可

能

对

辣

椒

的

化

学

成

分

与人

类

感

官

体

验

有

更

深

入

需

求

，

需

补

充

相

关

科

学

原

理

。

...

ROUND 2:

[2023

年

最

辣辣

椒

吉

尼

斯

, X

辣

椒

培

育

技

术

,

辣

椒

素

分

布

与

辣

度

关

系

,

辣

椒

育

种

抗

病

性

起

源

地

]

...

1.

根

据

最

新

检

索

结

果

，

2023

年

吉

尼

斯

认证

的

X

辣

椒

辣

度

为

269

万

SHU

，

超

过

卡

罗

莱

纳

死

神

的

220

万

，

成

为

新

的

世

界

之

最

，

但

需

确

认

2024

年

是

否

有

新

纪

录

。

2.

网

页

[17]

提

到

由

英

国

培

育

，

但

2023

年

X

辣

椒

认证

后

，

需

重

新

评

估

其

地

位

。

3.

育

种

技

术

方

面

，

需

补

充

X

辣

椒

的

10

年

培

育

周

期

、

杂

交

品

种

组

合

及

抗

病

性

改

良

。

4.

辣

度

因

素

中

，

网

页

[13][14][17]

均

提

到

辣

椒

素

在

胎

座

、

隔

膜

等

部

位

集

中

，

但

需

要

具

体

数

据

支

撑

。

5.

需

要

验

证

2024

年

是

否

有

新

的

辣

椒

品

种

突

破

269

万

SHU

。

6.

用

户

可

能

对

辣

椒

的

化

学

成

分

与

神

经

科

学

的

关

联

性

感

兴

趣

，

可

补

充

辣

椒

素

对

通道

的

作

用

机

制

。

...

ROUND 3:

[2024

年

最

辣辣

椒

吉

尼

斯

, X

辣

椒

抗

病

性

, X

辣

椒

辣

椒

素

胎

座

含

量

辣

椒

素

]

...

Reasoning and Search Rounds of ProxyQA - R1-Distill

Proxy Question:

你

能

详

细

介

绍

一下世

界

上

最

辣

的

辣

椒

吗

，

包

括

它

的

起

源

、

培

育

过

程

以

及

导

致

其

极

高

辣

度

的

因

素

？

Response:

1.

用

户

需

要

了

解

世

界

上

最

辣

的

辣

椒

的

详

细

信

息

，

包

括

起

源

、

培

育

过

程

和

辣

度

因

素

。

首

先

需

要

确

定

'

世

界

最

辣辣

椒

'

的

具

体

品

种

，

常

见

候

选

包

括

卡

罗

来

纳

辣

椒

、

印

度

魔鬼

椒

、

龙

舌

兰

辣

椒

等

，

但

需

要

验

证

最

新

权

威

数

据

。

2.

起

源

部

分

需

明

确

原

产

地

（

如

美

洲

、

亚

洲

）

及

驯

化历史

，

需

搜

索

'

最

辣辣

椒

原

产

地

''

辣

椒

驯

化

过

程

'

等

关

键

词

。

3.

培

育

过

程

涉

及

选

种

、

嫁

接

技

术

、

抗

病

性

改

良

等

，

需

搜

索

'

辣

椒

育

种

技

术

''

高

辣

度

辣

椒

栽

培

方

法

'

等

。

4.

辣

度

因

素

需

区

分

自

然

基

因

（

如

CAPS

含

量

）

和

环

境

因

素

（

如

温

度

、

湿

度

），

需

搜

索

'

辣

椒

辣

度

形

成

机

制

''

环

境

对

辣

椒

素

影

响

'

。

5.

需

验

证

最

新

吉

尼

斯

纪

录

数

据

，

确

认

当

前

最

辣

品

种

是

否

为

卡

罗

来

纳

死

神

（

165

万

-325

万

斯

科

维

尔

）

或

某

新

型培

育

品

种

。

6.

需

注

意

区

分

不

同

辣

椒

的

辣

度

范

围

，

避

免

将

不

同品

种

的

数

据

混淆

。

...

ROUND 1

: [

世

界

最

辣辣

椒

品

种

吉

尼

斯

纪

录

,

卡

罗

来

纳

死

神

辣

椒

起

源

培

育

,

辣

椒

素

CAPS

合

成

机

制

环

境

影

响

,

高

辣

度

辣

椒

育

种

技

术最

新

技

术

]

...

Answer Rounds of ProxyQA - R1-Distill

Proxy Question:

你 能 详 细 介 绍 一下世 界 上 最 辣 的 辣 椒 吗 ， 包 括 它 的 起 源 、 培 育 过 程 以 及 导 致 其 极 高 辣 度 的 因 素 ？

Response:

目 前 公 认 的 世 界 上 最 辣 的 辣 椒 是 ' 辣 椒 X' （ PepperX ）， 其 史 高 维 尔 辣 度 单 位 （ SHU ） 检 测 值 达 到 269 万 [1][2] 。 该 品 种 由 美 国 辣 椒 育 种 专 家 Ed Currie 开 发 ， 以 卡 罗 来 纳 死 神 辣 椒 为 基 础 ， 通 过 杂 交 技 术 引 入其 他 高 辣 品 种 特 性 [4] 。

起 源 方 面 ， 辣 椒 原 产于中 美 洲 ， 现 代 栽 培 品 种 经 过 长 期 驯 化 。 ' 辣 椒 X' 的 培 育 过 程 涉 及 多 代 杂 交 选 育 ， Currie 通 过 特 殊 处 理 （ 如 硫 酸 铜 种 子 消 毒 ） 和 精 准 的 田 间 管 理 ， 最 终 培 育 出 这 种 扭 曲 皱 缩 的 黄 褐 色 果 实 [2][4] 。

辣 椒 的 辣 度 主 要 由 辣 椒 素 （ capsaicin ） 及 其 衍 生 物 含 量 决 定 ， 这 些 生 物 碱 集 中 分 布 在 白 色 ' 辣 囊 ' 组织 中 [2] 。 其 合 成 受 CAPS 基 因 调 控 ， 该 基 因 的 表 达 在 果 实 发 育 后 期 显 著 增 强 [5] 。 环 境 因 素 如 温 度 （ 20-30°C 最 佳 生 长 ） 和 湿 度 也 会 影 响 辣 椒 素 积 累 ， Currie 通 过 控 制 温 室 条 件 实 现 辣 椒 素 最 大 化 [4][7] 。

培 育 技 术 上 ， 该 品 种 采 用 分 阶 段 育 苗 ： 种 子 先 在 1% 硫 酸 铜 溶 液 中 浸 泡 消 毒 后 催 芽 ， 幼 苗 移 植 到 营 养 土 中 定 期 施 肥 ， 成 熟 果 实 需 在 特 定 部 位 采 收 以 保 证 辣 度 [4][7] 。 目 前 辣 椒 育 种 正 向 基 因 编 辑 方 向 发 展 ， 最 新 研 究 已 发 现 调 控 辣 椒 素 合 成 的 关 键 转运 蛋 白 基 因 [6] ， 为 培 育 更 高 辣 度 品 种 提 供 新 可 能 。

Answer Rounds of ProxyQA - DeepDiver

Proxy Question:

你 能 详 细 介 绍 一下世 界 上 最 辣 的 辣 椒 吗 ， 包 括 它 的 起 源 、 培 育 过 程 以 及 导 致 其 极 高 辣 度 的 因 素 ？

关 于世 界 最 辣辣 椒 （ 截 至 2024 年 ） 的 详 细 介 绍 如 下 ：

1. 主 要 品 种

2024 年 吉 尼 斯 认证 最 辣辣 椒

X 辣 椒 （ Pepper X ）： 由 美 国 辣 椒 培 育 专 家 埃 德 · 柯 里 （ Ed Currie ） 于 2024 年 培 育 ， 经 吉 尼 斯 世 界 纪 录 认证 为 269.3 万 SHU （ 史 高 维 尔 辣 度 单 位 ） 。 该 辣 椒 被 称 作 ' 辣 椒 界 的 核 武 器 ' ， 其 黄 绿 色 扁 平 扭 曲 的 外 观 与 ' 卡 罗 莱 纳 死 神 ' （ 220 万 SHU ） 相 比 ， 辣 度 提 升 1.6 倍 [2][9][20] 。

卡 罗 莱 纳 死 神 （ Carolina Reaper ）： 该 品 种 2012 年 获 吉 尼 斯 认证 ， 平 均 辣 度 156.93 万 SHU ， 峰 值 220 万 ， 是 首 款 突 破 200 万 SHU 的 商 业 辣 椒 [1][2][7] 。

Naga Viper ： 2011 年 在 英 国 培 育 ， 辣 度 135.9 万 SHU ， 是 首 获 吉 尼 斯 认证 的 ' 世 界 最 辣 ' 辣 椒 [3][17] 。

2. 育 种 技 术

长 周 期杂 交 ： X 辣 椒 育 种 历 时 10 年 ， 柯 里 每 年 进 行 100 次 杂 交 试 验 ， 通 过 将 卡 罗 莱 纳 死 神 与 ' 密 歇 根 杂 交 辣 椒 ' 进 行 ' 精 准 嫁 接 ' ， 使 子 房 室 辣 椒 素 储 存 面 积 扩 大 [2][20] 。

抗 病 性 改 良 ： 育 种 过 程 中 采 用 硫 酸 铜 溶 液浸 种 （ 1% 浓 度 ， 10 分 钟 ） [7] ， 配 合 无 土 育 苗 技 术 （ 清 水 中 浸 泡 5 小 时 后 ， 用 25mL 乙 醇 溶 解 0.5g 辣 椒 粉 提 取 辣 椒 素 进 行 检 测 ） [14] ， 有 效 防 止 土 传 病 害 。

3. 辣 度 形 成 机 制

辣 椒 素 分 布 ： 辣 椒 素 （ 辣 椒 碱 ） 主 要 集 中 在 辣 椒 的 胎 座 、 隔 膜 及 果 肉 （ 占 总 辣 椒 素 含 量 的 70% 以 上 ） [23][24] 。 X 辣 椒 通 过 基 因 改 良 ， 使 胎 座 区 域 辣 椒 素 浓 度 提 升 40%[24] 。

表 皮 结 构 ： 其 表 面 褶 皱 增 加 辣 椒 素 接 触 面 积 ， 类 似 人 类 皮 肤 褶 皱 会 增 强 疼痛 感 [13] 。

辣 椒 素 含 量 ： 1mg/kg 的 辣 椒 素 约 等 于 1600 万 SHU ， X 辣 椒 的 辣 椒 素 浓 度 达 5.771mg/g （ 干 重 ）， 远 超 普 通 辣 椒 [24][23] 。

4. 栽 培 与 食 用

种 植 技 术 ： 需 保 持 土 壤 pH6.0-7.0 ， 成 熟 期 （ 90-120 天 ） 需 每 日 8 小 时 全 日 照 ， 同 时 采 用 轮 作 制 （ 避 开 茄 科 作 物 ） [7][12] 。

食 用 风 险 ： 2023 年 记 录 显 示 ， 食 用 30mg X 辣 椒 可 使 人 体 血 清 素 释 放 量 激 增 ， 引 发 ' 辣 椒 素 中 毒 ' 症 状 （ 胸 闷 、 呼吸 困 难 ）， 甚 至 导 致 免 疫 系 统 超负 荷 [2][20][26] 。

5. 科 学 意 义

TRPV1 通道 激 活 ： 2021 年 诺 贝 尔 奖 得 主 大 卫 · 朱 利 叶 斯 （ David Julius ） 发 现 ， 辣 椒 素 通 过 激 活 瞬 时 受 体 电 位 香 草 酸 型 1 （ TRPV1 ） 通道 ， 使 神 经 末 梢 持 续 产 生 ' 灼烧 感 '[25][26] 。 X 辣 椒 中 辣 椒 素 与二 氢 辣 椒 素 （ 占 比 72% ） 的 协 同 作 用 ， 使 该 通道 被 激 活 概 率 提 升 2.3 倍 [24] 。

6. 特 殊 用 途

工 业 应 用 ： X 辣 椒 被 用 于 高 端 辣 椒 酱 ， 2023 年 上 市 24 小 时 内 售 罄 1000 瓶 ， 其 辣 椒 素 提 取 物 还 被 开 发 为 新 型 止 痛 剂 候 选 [9][20] 。

科 研 价 值 ： 其 135.9 万 SHU 的 辣 椒 素 提 取 物 被 用 于 癌 症疼痛 研 究 ， 2024 年 相 关 论 文 显 示 ， 高 浓 度 辣 椒 素 （ >200 万 SHU ） 可 使 癌 细 胞 线 粒 体 应 激 反 应 增 强 48%[24][26] 。

Figure 11: After several rounds of reasoning and searching, DeepDiver generates a result that covers various aspects of the given question, delivering extensive and comprehensive content with a wealth of knowledge.

Iterative RAG Prompt

你 是 一 名 善 于 准 确 灵 活 地 使 用 搜 索 工 具 、 且 富 有 解 题 策 略 和 创 造 性 的 助 手 。 我 会 给 你 一个 问 题 ， 接 下 来 通 过 多 轮 搜 索 工 具 调 用 解 决 问 题 ， 每 一 轮 中 你 都 需 要 先 思 考 ， 然 后 再决 定 是 否 检 索 或 者 给 出 最 终 答 案 。 如 果 你 选 择 检 索 ， 我 会 返 回 给 你 检 索 结 果 然 后 再 进 行 下一 轮 。

## 每 一 轮 回 复 要 求

每 一 轮 你 的 回 答 需 要 包 括 两 部 分 ， 第 一 部 分 是 思 考 ， 第 二 部 分 是 你 的 最 终 回 答 。

## 你 的 思 考 部 分 （ <thinking> 和 </thinking> 中 间 部 分 ）：

原 则 是 尽 可 能 的 利 用 搜 索 引 擎 帮 助 解 决 问 题 或 验 证 结 论 ， 每 一 轮 要 基 于之 前 已 有 思 维 链 条 继续 自 我 思 考 分 析 回 答 问 题 ， 要 善 于 利 用 思 维 链 条 已 有 结 论 ， 不 要 重 复 推 理 和 计 算 ；

如 果 需 要 检 索 ， 检 索 规 划 时 ， 请 避 免 基 于 内 部 知 识 进 行 草 率 、 较 大 跳跃 的 推 论 ， 总 是 要 假 定 还 有 其 他 可 能 ， 防 止 过 早 的 缩 小 问 题 的 范 围 ；

多 来 源 交 叉 验 证 和 冲 突 解 决 ： 强 调 对 多 个 来 源 的 交 叉 验 证 ， 尤 其 是 当 多 个 结 果 出 现 矛 盾 时 ， 应 该 主 动 发 起 更 多 检 索 ， 比 较 不 同 来 源 的 可 信 度 ， 进 行 更 深 入 的 检 索 规 划 ， 而 不 是 快 速 做 决 定 、 草 率 的 选 一个 可 能 权 威 的 来 源 ；

回 溯 和 跳 出 局 部 假 设 ： 在 遇 到 多 次 检 索 没 有 搜 集 到 严 格 符 合 要 求 的 信 息 的 情 况 时 不 要 轻 易 放 弃 ， 总 是 要 考 虑 是 否 有 其 他 可 能 性 ， 能 够 重 新 评 估 假 设 ， 调 整 搜 索 策 略 ， 回 溯 和 跳 出 局 部 假 设 ， 重 新 规 划 ；

基 于一 般 搜 索 引 擎 的 特 点 ， 要 根 据 问 题 类 型 善 于 利 用 如 拆 解 、 细 化 以 及 相 反 的 简 化 为 更 上 位 的 概 念 、 搜 索 原 题 原 文 、 搜 索 类 似 问 题 等 多 个 手 段 ；

你 的 思 考 使 用 语 言 尽 量 和 用 户 问 题 语 言 保 持 一 致 ， 但你 可 以 尝 试 多 种 语 言 的 搜 索 语 句 ， 目 前 的 搜 索 引 擎 主 要 以中 文 为主 ， 所 以 英 文 问 题 也 要 尝 试 用 一 部 分 中 文 搜 索 语 句 。

## 你 的 最 终 回 答 部 分 （ </thinking> 后 的 部 分 ）：

1. 如 果 需 要 进 一 步 检 索 ， 则 该 部 分 为 工 具 调 用 格 式 ： web_search|{'search_queries': [' 搜 索 语 句 1', ' 搜 索 语 句 2', ...]} ， 不 要 有 任 何 其 他 多 余 的 内 容 ；

2. 如 果 不 需 要 进 一 步 搜 索 ， 则 :

-综 合 整 个 思 维 链 条 给 出 信 息 量 丰 富 、 逻 辑 清 晰 的 最 后 回 复 ；

-除 非 用 户 要 求 ， 否 则 你 最 终 回 答 的 语 言 需 要 和 用 户 提 问 的 语 言 保 持 一 致 。

## 问 答 规 范 ：

下 面 是 一些 回 答 问 题 的 规 范 ， 如 有 必 要 可 以 在 思 考 部 分 回 顾 与 其 可 能 相 关 规 范 （ 回 顾 规 范 时 ， 要 复 述 相 应 规 范 条 例 的 对 应 片 段 ， 而 不 是 指 出 基 于 第 几 条 规 范 ）， 以 及 在 最 终 回 复 部 分 遵 循 这 些 规 范 ：

-安 全 性 ： 回 答 必 须 符 合 基 本 道 德 规 范 及 主 流 价 值 观 ， 体 现 人 文 关 怀 ； 不 能 复 述 用 户 问 题 中 的 敏 感 和 不 文 明 用 词 用 语 ； 谨 慎 给 出 不 权 威 的 影 响 用 户 重 要 决 策 的 建 议 ；

-完 整 性 ： 用 户 问 题 的 所 有 部 分 都 应 该 在 思 考 中 考 虑 到 ；

-对 于 客 观 类 的 问 答 ， 如 果 问 题 的 答 案 非 常 简 短 ， 可 以 适 当 补 充 一 到 两 句 相 关 信 息 ， 以丰 富 内 容 ；

-对 于 长 文 生 成 类 的 问 题 （ 如 写 报 告 、 论 文 、 攻 略 ）， 你 需 要解 读 并 概 括 用 户 的 题 目 要 求 ， 选 择 合 适 的 格 式 ， 充 分利 用 搜 索 结 果 并 抽 取 重 要 信 息 ， 生 成 符 合 用 户 要 求 、 极 具 思想 深 度 、 富 有 创 造 力 与专业 性 的 答 案 。 你 的 写 作 章 节 和 篇 幅 要 尽 可 能 延 长 ， 对 于 每 一个 要 点 的 论 述 给 出 尽 可 能 多 角 度 的 回 答 ， 务 必 信 息 量 大 、 论 述 详 尽 。

## 输 出 格 式 ：

思 考 部 分 用 <thinking></thinking> 标 签 完 整 包 裹 ， </thinking> 后 接 着 输 出 最 终 回 复 或 工 具 调 用 ， 不 要 有 多 余 的 不 属 于 思 考 和 最 终 回 复 两 部 分 的 前 缀 和后 缀 的 解 释 和 描 述 。 输 出 示 例 如 下 ： <thinking>[ 你 的 思 考 过 程 ...]</thinking>[ 你 的 最 终 回 复 或 工 具 调 用 ...]

[ 问 题 开 始 ]

$query

[ 问 题 结 束 ]

















































































































































































































































































































Loose Grader Prompt

你 将 看 到 一个 用 户 问 题 和 待 评 估 的 回 答 ， 我 会 给 你 参 考 答 案 和 回 答 规 范 ， 你 需 要 依 据 下 面 的 要 求 对 待 评 估 回 答 进 行 分 析 和 打 分 。

## 打 分 标 准

请 以 是 否 很 好 的 回 答 用 户 问 题 为 准 则 ， 参 考 答 案 只 是 辅 助 信 息 。

待 评 估 答 案 与 参 考 答 案 的 ' 关 键 语 义 含 义 ' 大 致 相 同 即可 ， 不 必 严 格 遵 照 参 考 答 案 的 文 字 表 述 。

对 于 编 号 或 分 步 骤 的 问 题 ， 完 全 可 以 接 受 步 骤 序 号 不 同 、 划分 粒 度 不 同 的 情 况 ， 只 要 含 义 相 同 即可 。

对 于 列 举 类 问 题 ， 如 果 待 评 估 答 案 覆 盖 了 参 考 答 案 中 的 一 部 分 ， 另 一 部 分 有 所 遗 漏 ， 请 根 据 正 确 覆 盖 的 比 例 给 出 合 理 分 数 。

对 于 数 值 类 的 答 案 在 ' 得 分 标 准 ' 允 许 的 误 差 范 围 内 ， 都 视 为 正 确 。

参 考 答 案 的 细 节 性 要 求 有 时 过 于严 格 ， 请 不 要 严 格 遵 照 ， 如 ' 必 须 ...'' 不 能 ...'' 一 定 要 ...' 等 ， 只 要 不 影 响 给 用 户 提 供 正 确 和 完 整 的 信 息 ， 可 以 忽 略 参 考 答 案 的 严 格 要 求 。

扣 分 情 况 ：（ 1 ） 参 考 答 案 的 内 容 覆 盖 不 完 整 ， 按 遗 漏 比 例 酌 情 扣 分 （ 2 ） 与 参 考 答 案 有 事 实 性 冲 突 ， 应 大 幅度 扣 分 。

基 于 参 考 答 案 和 打 分 标 准 ， 按 下 面 的 等 级 对 待 评 估 回 答 给 出 1 ∼ 10 分 。

10 分 ： 待 评 估 回 答 的 结 论 和 参 考 答 案 完 全 一 致 ， 完 全 遵 从 回 答 规 范 。

8 ∼ 9 分 ： 没 有 硬 性 问 题 ， 即 待 评 估 回 答 的 结 论 和 参 考 答 案 完 全 一 致 ， 基 本 遵 从 回 答 规 范 。

6 ∼ 7 分 ： 正 确 性 方 面 没 有 原 则 性 问 题 ， 即 待 评 估 回 答 的 结 论 与 参 考 答 案 说 法 不 冲 突 或 互为 包 含 关 系 ， 但 可 能 对 回 答 规 范 有 一 定 漏 考 虑 的 地 方 ， 或 推 理 思 路 可 能 有 不 清 晰 具 体 的 问 题 。

5 分 ： 正 确 性 有 小 问 题 ， 即 待 评 估 回 答 的 结 论 和 参 考 答 案 不 完 全 一 致 ， 比 如 可 能 有 漏 考 虑 或 多 考 虑 的 地 方 。

2 ∼ 4 分 ： 正 确 性 有 不 小 的 问 题 ， 即 待 评 估 回 答 的 结 论 和 参 考 答 案 有 一 定 本 质 的 区 别 ， 比 如 可 能 有 理 解 错 误 、 推 理 计 算 错 误 。

1 分 ： 待 评 估 回 答 完 全 不 可 用 ， 比 如 结 论 完 全 错 误 且与 正 确 答 案 差 距 很 大 。

## 待 评 估 内 容 [ 用 户 问 题 开 始 ]

$query

[ 用 户 问 题 结 束 ]

[ 参 考 答 案 开 始 ]

$solution

[ 参 考 答 案 结 束 ]

[ 回 答 规 范 开 始 ]

$checklist

[ 回 答 规 范 开 始 ]

[ 待 评 估 答 案 开 始 ]

$response

[ 待 评 估 答 案 结 束 ]

## 你 分 两 步 输 出 你 的 结 果 ：

第 一 步 ： 分 析 。

基 于上 述 给 定 各 维 度 标 准 ， 以 及 打 分 评 价 方 法 ， 一 步 一 步 分 析 待 评 估 回 答 属 于 评 分 标 准 中 的 哪 一 类 ， 以 及 应 该 给 出 的 分 数 。

该 步 骤 输 出 格 式 为 ： 第 一 步 ， 分 析 ： ...

为了 方 便 解 析 ， 总 结 你 第 一 步 分 析 的 结 果 ， 按 如 下 json 格 式 输 出 ：

第

二

步

，

输

出

：

{

"

打

分

理

由

": "",

"

得

分

": 1

∼

10,

}

Strict Grader Prompt

Strict Grader Prompt 1:

你 是 一个 资 深 的 问 答 专 家 ， 根 据 用 户 问 题 、 标 准 答 案 、 答 案 checklist 、 问 题 回 复 这 四 项 内 容 ， 评 估 问 题 回 复 的 质 量 。

用 户 问 题

：

$query

标 准 答 案

： $solution

答 案 checklist: $checklist

问 题 回 复 ：

$response

标 准 如 下 ：

根 据 标 准 答 案 ， 你 判 断 问 题 回 复 的 意 思 是 否和 标 准 答 案 一 致 ， 如 果 一 致 则 正 确 ， 不一 致 则判 为 错 误 。 注 意 ： 数 值 类 的 答 案 在 答 案 checklist 允 许 的 误 差 范 围 内 ， 都 视 为 正 确 。

根 据 答 案 checklist 中 描 述 的 要 求 ， 衡 量 必 须 答 对 的 部 分 与 适 当 宽 松 的 部 分 ， 你 判 断 问 题 回 复 的 答 案 是 否 符 合 要 求 ， 符 合 则 正 确 ， 不 符 合 则判 为 错 误 。

只 有 同 时 符 合 前 两 条 ， 问 题 回 复 才 会 最 终 被 视 为 正 确 ， 否 则 问 题 回 复 最 终 被 视 为 错 误 。

输 出 要 求 ：

** 第 一 步 ， 思 考 **

以 ' 第 一 步 ， 思 考 ： ' 开 始 ， 鼓 励 你 进 行 细 致 和 审 慎 的 推 理 和 思 考 ， 直 到 你 的 思 考 过 程 已 经 完 整 详 尽 且 逻 辑 严 密 ， 足 够 给 出 最 终 的 评 判 结 果 ， 即可 停 止 思 考 并 给 出 评 判 结 果 。

** 第 二 步 ， 评 判 结 果 **

以 ' 第 二 步 ， 评 判 结 果 ： ' 开 始 ， 严 格 按 照 以下 给 定 的 字 典 格 式 输 出 最 后 的 评 估 结 果 ， 不 要 输 出 带 有 'json' 等 话 术 ， 你 的 输 出 严 格 按 照 以下 给 定 的 字 典 格 式 ， 不 要 输 出 任 何 无 关内 容 。

最 终 输 出 格 式 为 ： {' 回 复 正 确 性 ': ' 正 确 '/' 错 误 '}

Strict Grader Prompt 2:

假 设 你作 为一 位 经 验 丰 富 的 问 答 专 家 ， 你 需 要 基 于 后 续给 定 的 用 户 问 题 、 标 准 答 案 、 答 案 checklist 、 问 题 回 复 ， 并 结 合 两个 核 心 维 度 ， 对 回 复 内 容 的 准 确 性 进 行 评 估 。

用 户 问 题 ： $query

...

评 估 标 准 如 下 ：

参 照 标 准 答 案 ， 你 需 评 估 问 题 回 复 是 否 与 标 准 答 案 相 吻合 ， 若 两 者 意 思 相 同 则 视 为 正 确 ， 否 则判 定 为 错 误 。 注 意 ： 如 果 在 答 案 checklist 中 有 提 及 允 许 的 误 差 范 围 ， 数 值 类 的 问 题 回 复 只 要 在 误 差 范 围 内 ， 都 视 为 正 确 。

跟 从 答 案 checklist 的 要 求 ， 衡 量 问 题 回 复 ， 给 出 正 确 或 错 误 的 结 论 。

前 两 点 的 答 案 如 果 都 是 正 确 ， 那 么 问 题 答 案 是 正 确 的 。 否 则 ， 问 题 答 案 是 错 误 的 。

...

Strict Grader Prompt 3:

假 设 你 是 问 答 领 域 的 资 深 专 家 ， 你 将 基 于 给 定 的 用 户 问 题 、 标 准 答 案 、 答 案 checklist 、 问 题 回 复 ， 并 结 合 以下 提 及 的 两个 评 测 维 度 ， 对 问 题 的 回 复 进 行 综 合 评 估 。

用 户 问 题 ： $query

...

接 下 来 给 出 两个 评 测 维 度 的 标 准 ， 如 下 ：

你 将 标 准 答 案 作 为 参 照 ， 然 后 评 估 问 题 回 复 是 否 准 确 无 误 地 反 映 了 标 准 答 案 的 内 容 ， 若 相 符 则判 断 为 正 确 ， 否 则判 断 为 错 误 。 需 要 注 意 的 是 ： 计 算 类 的 结 果 如 果有 轻 微 误 差 ， 则 视 为 正 确 。

你 需 根 据 答 案 checklist 对 问 题 回 复 进 行 判 断 ， 若 符 合 答 案 checklist 的 每 一 项 要 求 则判 定 为 正 确 ， 否 则 视 为 错 误 。

只 有 前 两 条 的 结 论 都 是 正 确 ， 问 题 回 复 的 最 终 评 估 结 果 才 是 正 确 ， 否 则 问 题 回 复 的 最 终结 果 是 错 误 。

...

Behavoir Statistic Prompt

Reflection & Correction Prompt:

下 面 是 一个 大 模 型 调 用 搜 索 工 具 进 行 多 轮 推 理 和 检 索 回 答 问 题 的 思 维 链 条 ， 我 会 给 你 该 问 题 、 思 维 链 条 和 问 题 的 标 准 答 案 。

[ 问 题 开 始 ]

$query

[ 问 题 结 束 ]

[ 思 维 链 条 开 始 ]

$cot

[ 思 维 链 条 结 束 ]

[ 问 题 答 案 开 始 ]

$solution

[ 问 题 答 案 结 束 ]

我 需 要 你 帮 我 评 估 并 计 数 该 思 维 链 条 中 是 否 存 在 任 何 Reflection & Correction 的 模 式 ， 即 模 型 发 现 了 前 面 步 骤 中 潜 在 的 错 误 或 遗 漏 ， 重 新 评 估 其 已 有 思 维 链 条 ， 审 视 先 前 的 假 设 ， 并 明 确 地 纠 正 或 细 化 先 前 的 推 理 步 骤 或 结 论 。

请计 数 Reflection & Correction 模 式 一 共 在 思 维 链 条 中一 共出 现 了 多 少 次 ， 如 果 不 存 在 则 计 数 为 0 。 请 先 分 析 思 维 链 条 各 部 分 是 否 符 合 该 模 式 特 征 ， 并 最 后 将 该计 数 结 果 写 在 <count></count> 中 。

Conflict Resolution Prompt:

...

我 需 要 你 帮 我 评 估 并 计 数 该 思 维 链 条 中 是 否 存 在 任 何 Conflict Resolution 的 模 式 ， 即 在 检 索 到 的 信 息 包 含 矛 盾 或 不一 致 的 情 况 下 ， 模 型 能 够 发 现 矛 盾 并 主 动 发 起 新 的 检 索 以 解 决冲 突 。

请计 数 Conflict Resolution 模 式 一 共 在 思 维 链 条 中一 共出 现 了 多 少 次 ， 如 果 不 存 在 则 计 数 为 0 。 请 先 分 析 思 维 链 条 各 部 分 是 否 符 合 该 模 式 特 征 ， 并 最 后 将 该计 数 结 果 写 在 <count></count> 中 。

Verification & Denoising Prompt:

...

我 需 要 你 帮 我 评 估 并 计 数 该 思 维 链 条 中 是 否 存 在 任 何 Verification & Denoising 的 模 式 ， 即 在 给 定 有 噪 声 或 不 相 关 的 检 索 信 息 ， 模 型 分 离 出 可 信信 息 ， 从 而 执 行 去 噪 。

请计 数 Verification & Denoising 模 式 一 共 在 思 维 链 条 中一 共出 现 了 多 少 次 ， 如 果 不 存 在 则 计 数 为 0 。 请 先 分 析 思 维 链 条 各 部 分 是 否 符 合 该 模 式 特 征 ， 并 最 后 将 该计 数 结 果 写 在 <count></count> 中 。

Figure 16: Prompts used for automatically evaluating the occurrence counts of different behaviors in model outputs.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Yes, the abstract and introduction accurately summarize the paper's key findings, including the experimental results and their robustness and generalizability.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Yes, we thoroughly discuss the limitations of our work and potential solutions in the Discussion section C.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: There are no new theorems, formulas, and proofs involved.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Yes, the paper provides detailed step-by-step descriptions of the dataset curation (Section 3.1), methodology (Section 3.2), and experiments (Section 4) to ensure reproducibility of the main results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: The paper contains experimental results that rely on proprietary code and data that are currently undergoing internal review process for open-source release approval. While we cannot provide open access to the code and data at submission time, we plan to release them once the review process is completed. This restriction is in line with the NeurIPS guidelines, which acknowledge that there may be legitimate reasons why code cannot be made available immediately. We have included detailed descriptions of our methods, algorithms, and experimental procedures in the paper and supplementary materials to ensure the scientific contribution is clear and to allow for conceptual reproducibility of our work.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Yes, Sections 3.1, 4 and Appendix E provide comprehensive details on data selection, experimental setup, hyperparameters, and implementation to ensure full reproducibility of the results.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: While our paper does not include formal error bars, confidence intervals, or statistical significance tests, we did take steps to ensure reproducibility and reliability of our experimental results. Specifically, each experiment was conducted with 3 independent runs using different random seeds, and we reported the average scores in our results. We acknowledge that a more comprehensive statistical analysis with properly defined error bars would strengthen our findings. For future versions or camera-ready preparation (if accepted), we plan to expand our statistical reporting to include standard deviations or confidence intervals for all key experimental results, clearly stating the calculation methods and assumptions.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We introduced the computation resources and requirements in details in Section E.8.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The paper is fully conform with the NeurIPS code of ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: To avoid potential ethical and societal impact related issues, we carefully checked all questions in our datasets in multiple aspects, as discussed in Section D.3. We try to guarantee that all samples do not involve any offensive, gender-biased, or political content, and any other ethical issues. The source code will be released with instructions to support correct use. The baseline model we tested are all open-sourced LLMs and public APIs, these previous works have already considered the societal impact issues when creating the models.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All the code dataset and benchmarks and related baselins are well cited and introduced in Appendix E.5, Appendix E.4 and Section 4.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [No]

Justification: While we have provided detailed documentation and descriptions of our new dataset and models within the paper itself (including details on collection methodology, properties, limitations, and intended uses), we are unable to provide the actual assets alongside the submission due to the ongoing internal review process for open-source release approval. The paper contains comprehensive documentation that would typically accompany these assets, but the assets themselves cannot be uploaded at submission time. We plan to release these assets with complete documentation once the review process is completed.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [Yes]

Justification: We conduct the human evaluation and annotation, all the details including the guideline and instructions are performend in Section A.3 and Appendix D.2.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We do not invovle human subjects, all annotators and evaluators are recruited by a company.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [Yes]

Justification: Yes, our paper transparently documents all instances of LLM usage-whether proprietary or open-source-including detailed prompts and implementation specifics. This ensures reproducibility and clarifies their role in our methodology.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores the task of open-web question answering to evaluate the ability of a model / agent on real-world information gathering and reasoning. The paper has two key contributions: 1) Webpuzzle which provides a more challenging real world benchmark (with both training and test sets) for real-world open-web question answering. 2) Deep-diver - an RL based framework which encourages higher search frequency and depth (by reward shaping i.e. rewarding GRPO rollouts which give correct answer while using search). Quantiative results show that Deep-diver performs favorably as compared to prior open-weight models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n* The paper proposes a challenging Webpuzzle dataset for evaluating information seeking capabilities of agents in real-world scenarios.\n*  The use of extra search call rewards in Sec. 3 is interesting.\n* The paper has some nice insights with regards to correlation of agent behavior with final performance (such as increase in search frequency)\n* The results in Sec. 5.2, for cross-lingual generalization are also interesting\n\nWeaknesses:\n* In Sec. 3.2, the authors mention for cold start they use: \"300 real-user questions from our deployed smart assistant\". Can the authors please provide details on this?\n* For iterative RAG mentioned in Sec. 3 for GRPO, what is the exact scaffold used for inference and training (e.g., basic ReACT loop, or more complex iterative multi-step scaffoldsOpenHands etc)\n* The use of extra search call rewards is interesting, but did the authors explore other simpler alternatives such as just providing a small reward for solving with search etc.\n* What is the maximum number of steps in each rollout during training and inference?\n* For closed-source models in Table 1 only gpt4o is evaluated, can the authors also provide results with better models such as o3 or claude-4-sonnet. This will better help understand the performance on the proposed test set. \n* Since it is a new benchmark, thorough evaluation across different baselines (including those with better scaffolds such as OpenHands) can help the reader better understand the difficulty of the proposed benchmark.\n\nQUESTIONS:\n* In Sec. 3.2, the authors mention for cold start they use: \"300 real-user questions from our deployed smart assistant\". Can the authors please provide details on this?\n* For iterative RAG mentioned in Sec. 3 for GRPO, what is the exact scaffold used for inference and training (e.g., basic ReACT loop, or more complex iterative multi-step scaffoldsOpenHands etc)\n* The use of extra search call rewards is interesting, but did the authors explore other simpler alternatives such as just providing a small reward for solving with search etc.\n* What is the maximum number of steps in each rollout during training and inference?\n* For closed-source models in Table 1 only gpt4o is evaluated, can the authors also provide results with better models such as o3 or claude-4-sonnet. This will better help understand the performance on the proposed test set. \n* Since it is a new benchmark, thorough evaluation across different baselines (including those with better scaffolds such as OpenHands) can help the reader better understand the difficulty of the proposed benchmark.\n* Also can the authors provide more details on RL training, such as max context, max steps, the training algorithm, use of overlong filtering etc for GRPO.\n* Finally, have the authors tried to compare performance of RL model trained on Webpuzzle with RL model trained on previous benchmarks and training datasets?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates how to enable LLMs to perform adaptive, evidence-seeking behavior in open-domain question answering. To support this, the authors introduce WebPuzzle, a benchmark comprising both Wikipedia and open-web queries designed to elicit complex reasoning behaviors under noisy conditions in Chinese. They propose DeepDiver, a RL framework that trains LLMs to interleave reasoning and search, guided by a two-stage process involving supervised initialization and GRPO-based fine-tuning. A key emergent behavior, Search Intensity Scaling (SIS), allows models to dynamically adjust search depth based on task difficulty. Experiments show that DeepDiver-equipped 7B models outperform prompting and distillation baselines, approaching the performance of much larger models on multiple QA tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n- The paper tackles an important and challenging problem of adaptive information seeking in real-world, noisy environments where static prompting and closed-domain SFT approaches fall short. The paper demonstrates a novel emergent behavior—Search Intensity Scaling (SIS)—where models increase search frequency and reasoning steps based on task difficulty, leading to substantial performance gains over prompting and distillation baselines.\n- The authors introduce WebPuzzle, a new benchmark with 24k training and 275 expert-annotated test samples. It explicitly stresses noisy retrieval, ambiguity resolution, and reflective reasoning, offering a more realistic alternative to clean Wikipedia-style datasets.\n- The DeepDiver framework is well-engineered, incorporating cold-start SFT, GRPO, and curriculum-aware reward shaping to stabilize and scale reinforcement learning on real-web tasks.\n- The authors provide thorough empirical evidence, including accuracy on multiple Chinese QA benchmarks, information-seeking behavior counts, reward ablations, generalization English benchmarks etc.\n\nWeakness\n- The code, data, and models are not released, citing the internal review process in the checklist. This limits verification and reuse for now, and the release is not guaranteed. \n- The test set size of 275 examples in WebPuzzle is relatively small for drawing strong statistical conclusions, especially given the complexity and variability of open-web search tasks.\n- The reward function design is highly heuristic, i.e., manual engineering of multiple components, including staged loose-to-strict grading and auxiliary reward triggers.\n- It is hard to understand the benefits of DeepDiver over existing agentic tool-augmented approaches, such as ReAct or WebGPT-style agents, which are not included as comparison systems in this work.\n\nQUESTIONS:\nWhat’s the rationale behind not explicitly modeling the search?\n\nIn practice, how much impact would the search latency create with increasing search depth?\n\nLIMITATIONS:\nyes in appendix", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the task of question answering via web information retrieval using Large Language Models. The authors provide a new benchmark dataset, WebPuzzle, testing the LLMs ability to navigate and reason about noisy information from websites. Using this, they train DeepDiver, a 7B LLM that reaches similar performance to much larger models on such task. For this, they propose a tailored RL pipeline to incentivize search intensity and generalize to new domains. Moreover, they analyze various evaluation aspects such as the relationship between search intensity and problem difficulty.\n\nSTRENGTHS AND WEAKNESSES:\nStrenghts: \n- The paper studies a relevant and significant problem, the contributions are clear and the overall exposition is good. \n- The WebPuzzle dataset looks like a relevant benchmark that looks complex enough to separate and differentiate different LLM agents and approaches\n- DeepDiver looks like a strong LLM agent for WebSearch\n- The explanation and ablations of the RL method used for training DeepDiver is a valuable contribution to the community\n- Experiments' analyses are somewhat extensive and informative.\n\nWeaknesses:\n- I think the authors could have given more details on the used RL algorithm and training procedure, with training metrics and ablations on the choices made.\n\nQUESTIONS:\n- From my understanding, only 2k samples (demonstrations?) were used for SFT, while 5k samples (prompts?) for RL. Did the authors ablate the chosen split? In particular, I wonder how would the performance vary as a function of the amount of SFT data. E.g. what would have happened with a single SFT phase on 7k samples.\n- Related to the above, online RL (via GRPO) seems crucial for performance but also the main computational bottleneck to scale the approach to more samples. Did the authors consider offline baselines (such as DPO) for sample efficiency?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates LLMs capability for open-web retrieval and proposes letting the model learn to perform proactive, multi-step searches when solving open-domain questions. The authors release WebPuzzle (24k training examples; 275 human-curated test questions), a dataset explicitly designed to measure an LLM’s ability to retrieve and reason over information in a realistic, noisy and fragmented web environment. They further introduce the DeepDiver framework: after a SFT warm-start, the system is refined with reinforcement learning (GRPO) to cultivate a policy that decides when and how much to search. A key claimed feature is the emergent SIS ability. Experiments with 7B parameter base models show that DeepDiver nearly matches R1 on real-time web QA benchmarks, while raising the average number of retrieval rounds from ~1.7 to > 2.5. The paper also analyses the training curriculum and the strategy’s generalisation to tasks such as open-ended long-form generation.\n\nSTRENGTHS AND WEAKNESSES:\nPros:\n\n1. The paper constructs a new WebPuzzle dataset to evaluate knowledge retrieval and reasoning in an open network environment.\n\n2. The overall scores of Qwen2.5-7B and Pangu-7B trained by DeepDiver are almost equal to or even partially surpass those of the DeepSeek-R1 model, which has two orders of magnitude larger parameters.\n\nCons:\n1. Although the paper claims that SIS is a major innovation, this capability is essentially a model behavior promoted by reward design, and is conceptually an extension of existing ideas. Using reinforcement learning to optimize LLM retrieval decisions is not new. In recent years, there have been works that use RL to let the model learn when to retrieve and how many steps to retrieve. The main contribution of the author's method is that it is applied to more complex environments and the number of model retrieval times is observed to increase. This is more of a phenomenon report rather than a new algorithmic principle.\n\n2. The paper repeatedly describes SIS as a feature that emerges automatically after RL training, but in fact the authors explicitly add rewards to the model to encourage multiple searches. Calling this behavior \"emergent\" under the influence of rewards is a bit exaggerated and may mislead readers, as if the model has spontaneously evolved this skill.\n\nQUESTIONS:\n1. Most of the training samples in the WebPuzzle dataset are synthesized through LLM. How do the authors ensure that these synthetic problems realistically reflect the complexity of real user search tasks?\n2. DeepDiver’s RL training uses complex reward design and training scheduling. Can the authors share the sensitivity of key training settings?\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}