Scent of Knowledge: Optimizing Search-Enhanced Reasoning with Information Foraging
Authors: Hongjin Qian
Zheng Liu
Venue: NeurIPS 2025 spotlight ()
Keywords: Complex knowledge discovery
Search-Enhanced Reasoning
Retrieval-Augmented Generation
Primary area: applications
TL;DR: We introduce InForage, an RL framework that enables LLMs to perform adaptive, multi-step retrieval by rewarding informative intermediate steps.

Abstract: Augmenting large language models (LLMs) with external retrieval has become a standard method to address their inherent knowledge cutoff limitations. However, traditional retrieval-augmented generation methods employ static, pre-inference retrieval strategies, making them inadequate for complex tasks involving ambiguous, multi-step, or evolving information needs. Recent advances in test-time scaling techniques have demonstrated significant potential in enabling LLMs to dynamically interact with external tools, motivating the shift toward adaptive inference-time retrieval.
Inspired by Information Foraging Theory (IFT), we propose InForage, a reinforcement learning framework that formalizes retrieval-augmented reasoning as a dynamic information-seeking process. Unlike existing approaches, InForage explicitly rewards intermediate retrieval quality, encouraging LLMs to iteratively gather and integrate information through adaptive search behaviors. To facilitate training, we construct a human-guided dataset capturing iterative search and reasoning trajectories for complex, real-world web tasks. Extensive evaluations across general question answering, multi-hop reasoning tasks, and a newly developed real-time web QA dataset demonstrate InForage's superior performance over baseline methods. 
These results highlight InForage's effectiveness in building robust, adaptive, and efficient reasoning agents. 
We provide all codes and datasets in the supplementary materials.

Scent of Knowledge: Optimizing Search-Enhanced Reasoning with Information Foraging
Hongjin Qian', Zheng Liul:2+
Beijing Academy of Artificial Intelligence 2 Hong Kong Polytechnic University {chienghj. zhengl iul 1ul com
Abstract
Augmenting large language models (LLMs) with external retrieval has become a standard method to address their inherent knowledge cutoff limitations. However, traditional eerrieal--augmened generation methods employ static, pre-inference retrieval strategies, making them inadequate for complex tasks involving ambigu- ous, multi-step, or evolving information needs. Recent advances in test-time scaling techniques have demonstrated significant potential in enabling LLMs to dynamically interact with external tools, motivating the shift toward adaptive inference-tinee retrieval. Inspired by Information Foraging Theory (IFT), we propose InForage, reinforcement learning framework that formalizes retrieval- augmented reasoning dynamic information-secktn approaches, InForage explicitly rewards intermediate retrieval quality, encouraging LLMs to iteratively gather and integrate information through adaptive search behav- iors. To facilitate training, we construct human-guided dataset capturing iterative search and reasoning trajectories for complex, real-world web tasks. Extensive evaluations across general question answering, multi-hop reasoning tasks, and a newly developed real-time web QA dataset demonstrate InForage's superior perfor- mance over baseline methods. These results highlight InForage's 's effectiveness in building robust, adaptive, and efficient reasoning agents. We provide all codes and datasets in the supplementary materials as well as in this repository
process. Unlike existing
Introduction
Augmenting large language models (LLMs) with external knowledge retrieved via search tools common approach address their inherent knowledge cutoff limitation (Lewis et al. 2020a Zhao et al., 2024a). Existing methods typically apply static, pre-inference retrieval strategy by concatenating retrieved information into the input prompt, enabling the LLM to generate answers based on the provided external knowledge [Gao et al. 2024]. However, this approach often lacks adaptiveness, especially for complex tasks in which users' information needs are ambiguous, rationale- based, or not directly searchable, as these tasks require iterative reasoning to progressively uncover the necessary evidence for answer generation QQia et al. 2025a].
Recent advances in test-time scaling techniques have significantly strengthened LLMs' reasoning abilities, enabling complex behaviors such as long tfain-of-though reasoning and dynamic tool use (Snell et al. 2024, Muennighoff et al. 2025). Reasoning- bbasod models such as OpenAl's ol and DeepSeek RI have demonstrated significant gains on challenging tasks, particularly areas like mathematical problem-solving and coding, where iterative self-reflection and refinement are essential for success [OpenAl 2024 DeepSeek-AV 2025). Inspired by these developments, natural extension
Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

I am attending ng NeurIPS, Do I need a visa?
Tofo Patch
<TThiiiNNuurIS 2023 be held tn San DLop. user 12 B.nn
Finoo Fartcc
Roword Final Arawer
Sub query
LLM
Where will NountPs be held this year?
What 15 the user's 's
What 15 the US v1.5 polycy?
notionalityy
Reword Step-ifise Information gain
Search
Search
ARSNOT
Web enonledge
User Knowleedge
Reward
Figure I: Illustration of InForage. Complex tasks often require multi-hop reasoning and iterative retrieval across dispersed knowledge sources. InForage integrates retrieval into each reasoning step and assigns structured rewards based the relevance and utility of each retrieved patch. This dynamic supervision encourages the model to progressively gather, evaluate, and integrate civennee-miroriin human-like information foraging.
for retrieaal-augment generation is to shift retrieval from static, pre-inference step dynamic, inference-time process. This transition allows LLMs to iteratively retrieve and adaptively integrate external knowledge during reasoning, aligning more closely with the evolving and multi-layered information needs inherent to complex information seeking tasks eL al. 2025
Jin et al., 2025a].
However, effectively supporting such dynamic search-enhanced reasoning for complex tasks presents two key challenges. First, due to the implicit and evolving nature of such tasks, it is difficult to retrieve all necessary evidence through single retrieval action. Each retrieval typically uncovers only a local information patch, which is a partial subset of the expected knowledge space, and requires iterative accumulation and integration via reasoning to form a complete answer [Qian et al. 2023b). Second, the value of an information patch is not intrinsic to the retrieved content itself but depends on its contribution to the overall reasoning process and final answer accuracy (Zhu et al. 2024] Zhou et al., 2024]. Therefore, each retrieval decision significantly influences not only the immediate reasoning step but also the ultimate correctness of the final response. To illustrate these challenges concretely, consider a simplified case shown in Figure [ "I am attending NeurIPS. Do I need visa?" Addressing this question requires navigating the vast information space by iteratively retrieving local information patches from distinct sources (e.g., external web pages and personal profiles). The system must first uncover the event's location before accurately resolving the user's actual visa requirement. These sequential dependencies, where each retrieval clarifies only part of the information need, highlight the importance of adaptive, iterative retrieval strategies capable of dynamically refining the reasoning trajectory.
While such information-seelee seeking tasks remain challenging for current retrieval- -ugmeente systems, humans often resolve them efficiently with just a few iterative online searches [Nakano et al. 2021 Shani et al. 2024]. This suggests that humans inherently possess adaptive strategies for navigating complex information spaces. Drawing inspiration from cognitive science, we turn to Information Foraging Theory (IFT) (Pirolli and Card, 1999]. which provides a formal explanation for how humans strategically seek information by balancing the expected value gained from an information patch against the cognitive and time costs involved in exploring it. Central to IFT is the concept of information scent, defined as the perceived relevance or utility of available informational cues. Analogous to how animals follow scent trails to valuable resources, humans use information scent to guide their search toward promising information sources. Translating this analogy into retrieval- augmented reasoning, the model's 's intermediate reasoning steps and generated subqueries can be viewed as dynamically evolving assessments of information scent. Stronger information scent thus corresponds to higher-quality reasoning paths and subqueries, which lead retrieval actions toward more relevant and information patches. Consequently, optimizing intermediate information gain throughout the reasoning trajectory becomes as critical as ensuring the correctness of the final answer.
Building on this perspective, we propose InForage, a reinforcement learning (RL) framework designed enhance the search-augmented reasoning capabilities of LLMs. Previous reasoning methods typically optimize models based solely on the correctness of final answers, overlooking the crucial role of intermediate retrieval steps. In contrast, InForage explicitly rewards effective information-seeklee behavior at each stage of the reasoming process, recognizing that meaningful retrieval actions incrementally shape both the reasoning trajectory and the accuracy of the final response. Inspired by IFT, we introduce three complementary reward mechanisms to incentivize comprehensive reasoning behaviors: (1) an Outcome Reward, which credits trajectories leading to correct final answers; (2) an Information Gain Reward, which rewards intermediate retrieval

---

steps that uncover valuable evidence; and (3) an Efficiency Penalty, which discourages unnecessarily prolonged reasoning, encouraging concise and cost-effective information foraging.
Most existing QA datasets provide only final question-answer pairs, lacking records of intermediate reasoning or retrieval steps. Moreover, they often feature shallow queries that can be resolved with one or two retrievals, limiting their utility for training models on complex, multi-step reasoning. To address this gap, we construct a dataset that captures fine-grained human information--eekknng trajectories through open-ended web browsing. Starting from a seed claim, annotators iteratively query search engines, select relevant documents, and extract rationale- subsequent queries. After gathering sufficient evidence, we use a strong LLM (e.g., GPT-4o) to generate QA pairs that require multi-hop reasoning and layered evidence integration. Each example involves at least three information hops or intersecting conditions to ensure true complexity. Crucially, the dataset records every step of the search and reasoning process, enabling reward supervision over final answer correctness, intermediate retrieval quality, and overall reasoning efficiency.
claims to formulate
We evaluate InForage on standard QA, multi-hop reasoning, and a self-constructed real-time web QA benchmark. Results show that InForage consistently outperforms baselines, validating the effectiveness of learning from richly supervised, search-augmented reasoning data.
In summary, the contributions of this paper are threefold: (1) We formalize cearch-enhange reason- ing through information foraging theory, modeling retrieval decisions dynamic optimization of information scent and patch value along the reasoning trajectory. (2) We propose InForage, a reinforcement learning framework that jointly optimizes outcome correctness, intermediate infor- mation gain, and reasoning efficiency, enabling LLMs to perform adaptive, multi-step information foraging. (3) We construct a human-guided search reasoning dataset that captures multi-step web browsing trajectories and rationale-dependent query formulation, providing the structured supervision necessary to train and evaluate InForage effectively.
2 Method
2.1
Preliminary
The information-ssekinng process using LLMs can be formulated as y O()), where q is the input query. Y' is the generated answer, and (()) denotes the LLM.Sinne the knowledge embedded in most LLMs is fixed after training and difficult to update, incorporating external information has become common strategy to enhance their performance on information-sseking tasks. Retrieval-augmentee generation (RAG) is a widely adopted approach that follows this paradigm. In RAG, given a query 4. the system first retrieves relevant knowledge K. from an external knowledge base D using retriever T(.), and then generates the final answer conditioned on both q and K.. The process can be defined as:
y = ==O(q K)), K. K=
[g 1D),
where I'(-) represents the retrieval function and D is the external knowledge corpus.
However, in RAG systems, performance is tightly upper-bounded by retrieval quality: if the retrieved knowledge IC sub-optimal, the generated answer ) is likely to be flawed. This limitation becomes more pronounced in complex knowledge discovery tasks, where information needs are implicit or multi-layered. In such cases, effective information gathering requires iterative reasoning and adaptive retrieval over multiple steps.
Recent advances in reasoning LLMs enable a more dynamic retrieval paradigm, shifting retrieval from static pre-inference step to adaptive, inference- time mechanism. Specifically, the generated output Y typically contains a reasoning segment and an answering segment, denoted as (Jhhink: Vanswer) E y. During reasoning, the model generates trajectory of intermediate reasoning steps (or subqueries), exploring potential paths toward solving the task. Retrieval is interleaved into this process as:
y = - (g Viain & (I))) K, = I(g:"h (D), q. E Vihink
(2)
where u.uh denotes the subqueries generated during reasoning, used to retrieve intermediate know K,, The operator & indicates that retrieved knowledge is dynamically interleaved into the evolving reasoning trajectory.
3

---

2.2 Information Foraging Perspective on Search-Enhanced Reasoning
Suppose the expected knowledge space required for solving a complex information-seelt seeking task consists of n documents, denoted as D = .di.....dd. d. Generating the correct answer necessitates recovering the complete set D in the reasoning process. At the t-th retrieval step, given the generated subquery th the model may retrieve only a partial information patch Kr S "D", rather than the full set. Consequently, an ideal optimization objective is to sequentially gather all necessary information patches in as few reasoning-retrieval steps as possible, enabling accurate answer generation with minimal search effort.
Inspired by Information Foraging Theory, we characterize the search-enhanced reasoning process as information foraging, where the evolving information scent is expressed through the model's intermediate reasoning steps and generated subqueries. Guided by this scent, the model iteratively em- ploys retrieval tools gather local information patches from the broader knowledge space. strong information scent reff--effeecte in coherent reasoning trajectories and effective subqueries facilitatss the retrieval of increasingly relevant documents. Crucially, while accurate final answers depend on the successful accumulation of relevant information, even in cases where the final prediction is incorrect, a reasoning trajectory that effectively gathers valuable knowledge patches remains meaningful and should be explicitly rewarded. Formally, this process can be expressed as the following objective:
E ((()answer) + a-c ((KKD')) ST. .8..
max ( Fezt
1=1
where S(Vanswer) denotes the evaluation metric of the final generated answer, C (UL,,
measures the coverage of the retrieved patches relative to the expected knowledge D', T is the total number of reasoning-retrieval steps, and a, 3 E (0, 1) are weighting factors controlling the trade-off between information completeness and trajectory efficiency.
2.3
The proposed method: InForage
Building on the information foraging perspective, we propose InForage, a search-enhanced reasoning method that enables LLMs to iteratively reason, retrieve, and integrate external knowledge for solving complex information-seeking tasks. The reasoning trajectory of InForage is structured into specialized stages and can be formalized as:
y <hhink> reasoning content </think> <search> subquery </search>
information scent
<info> retrieved information </info> <answer> final answer </answer>.
information patch
As illustrated above, the model's generation process y unfolds as a sequential composition of blocks: <think> block capturing intermediate reasoning contents, a <search> block emitting subqueries, corresponding retrieval results encapsulated within <info> blocks, and finally, a <answer> block producing the final response. Throughout this process, the evolving information scent expressed via reasoning contents and subqueries guides the model to retrieve information patches that incremen- tally reconstruct the expected knowledge space, ultimately supporting accurate answer generation.
Reward Design for InForage. Reinforcement learning plays a key role in training reasoning models by enabling them to self-explore diverse reasoning trajectories and rewarding those that prove effective. This aligns model behavior with the objective of adaptive information seeking and efficient decision-making, However, since these trajectories self-generate and do not always yield correct final answers, relying solely on outcome-based rewards leads to sparse supervision signals-making the training process harder to optimize and less sample-efficient [Chen al. 2025]. Following the optimization goal defined in Eq. 3 we decompose the overall reward into three complementary components: Outcome Reward, Information Gain Reward, and Efficiency Penalty. Together, these rewards guide the model to reason more strategically, forage information effectively, and minimize unnecessary retrieval steps.
(1) Outcome Reward: At the end of each rollout, we extract the predicted answer Vanswer and
evaluate it using task-specific metric S(Vanswer), such as Exact Match or Fl score. The Outcome

---

Reward directly reflects the final task success:
Routcome a(nanswer).
(5)
(2) Information Gain Reward: A strong information scent leads to the retrieval of more relevant documents, enhancing intermediate knowledge acquisition. To capture this behavior, we compute the cumulative coverage of the retrieved patches against the expected knowledge set DD Specifically, at each retrieval step t, we evaluate the partial coverage (U'-1 Kr. D") and define the Information Gain Reward the maximum coverage achieved during the reasoning trajectory:
(ÚKK..)) D'
R Rgain
max C T...T 7=1
(6)
(3) Efficiency Penalty: Since the minimal trajectory for search-enhanced reasoning involves at least two steps (one for searching and one for answering), penalize excessively long reasoning paths. The Efficiency Penalty defined as an exponential decay applied to the total reward:
Refficiency =
3max(0.T -2)
where
0 < B <<k 1.
Final Reward: The final reward assigned rollout aggregates the three components as:
R =
Reticiency (Routcome + a Rgain)
(8)
where a E (0, 1) balances the emphasis between final answer correctness and intermediate informa- tion gathering.
Optimization. We first train the foundation LLMs using supervised fine-tuning (SFT) to enable the model to perform iterative reasoning and retrieval. Specifically, we construct the SFT dataset by providing gathered human web browsing trajectories and corresponding QA pairs to a strong LLM,, which then generates step-wise reasoning and retrieval responses as the training target.
Subsequently, we optimize InForage using Proximal Policy Optimization (PPO) [Schulman et al. 2017], guided by the rule-based rewards defined previously. For each input prompt p, the model generates a trajectory y, interleaved with retrieved information patches K At each generation step f, calculate the reward F'r and estimate the advantage A, via Generalized Advantage Estimation (GAE) [Schulman et al. 2016). Let V'rt denote the prefix generated up to step t, and Kce represent the retrieved information patches available thus far. The PPO training objective is defined as:
To(2% 12'e,
(0) E, [min (rede, clip(r,, 1 - 1 +)).))
(9)
Jee (1i, 1 DetsKee))
where Ty denotes the current policy, Tegy is the sampling policy from the previous iteration, and is the elipping parameter. Following Jin et al.) (2025a], restrict gradient updates exclusively to model-generated tokens, excluding tokens originating from retrieved information patches IC by applying appropriate masking.
3 Experiments
3.1 Settings
Baselines: We compare InForage against both non-retrieval and retrieval-augmenter baselines. For non-retrieval methods, we consider: (1) Vanilla, prompting the LLM to directly generate answers; (2) SFT, fine-tuning the LLM on the same QA pairs used by InForage: (3) Reasoning, training the LLM with reinforcement learning on QA pairs following DeepSeek-AT [2025]. For retrieval- augmented baselines, we include: (1) Vanilla RAG, which retrieves top-k documents once and
prepends them to the prompt; (2) IRCoT [Trivedi et al., (202aa. alternating retrieval with chain-of- thought reasoning; (3) RQRAG (Cha et al. 2024), which refines initial queries through rewriting and decomposition to improve retrieval accuracy; (4) Self-RAG [Asai et al. 2023), which introduces self-reflection mechanism allowing the model to critique and revise its own outputs based on retrieved
evidence; (5) Search-o1
[Li et al. 2025 which enhances LLMs with an agentic retrieval module
5

---

and Reason-in -neemmmens component for structured document reasoning; (6) Search-R1 [Jin et al. 2025a]. which learns to generate multiple search queries during reasoning via reinforcement learning to optimize multi-turn retrieval interactions.
Datasets: We evaluate on the following datasets: Natural Questions [K (Kwiatkowski et al. 2019
TriviaQA [Joshi et al. 2017]. PopQA (Mallen et al. 2022], HotpoiQA (Yang et al. [2018] | 2WikiMul- tihopQA (Ho et al. 2020]. MuSiQue (Trivedi et al. 2022b), Bamboogle [Press al. 2023], and
self-constructed real-time web QA dataset. We report exact match (EM) as the primary metric.
3.2 Implementaion Details
We use Qwen-2.5 Instruct models (3B and 7B) as our foundation LLMs for InForage. For SFT, we first sample 4,000 question- pairs from self-constructed training datasets. To generate corresponding reasoning trajectories, we condition Qwen-2.5-72B on each QA pair along with its associated relevant claims. These model-generated trajectories are then used fine-tune the foundation models for two epochs using a learning rate of x 10-5. Following SFT, we perform RL with PPO over 300 steps, using learning rate of 10 and warm-up ratio of 0.5. The RL training corpus includes data from our self-constructed dataset, Natural Questions (NQ), and HotpotQA. For NQ and HotpotQA,
use Wikipedia the knowledge source; for the self-constructed dataset, we use the cached web pages gathered during its creation. Structured rewards (Eq. () with 0.2 and 0.95 are applied only to the self-constructed dataset, as the others lack intermediate traces.
During inference, we use the E5 encoder [Wang et al,, 2024 and the Wikipedia dump from FlashRAG [Jin al. [2225b the retrieval backend for open-domain and multi-hop QA tasks. For complex, real-time web-based tasks, we cache Google Search results and scrape the full content of retrieved pages. Retrieval over this corpus is performed using the BGE-M3 retriever, and set the maximum number of reasoning steps to 6.
Regarding baselines, we adopt reported results when available (e.g., Search-R1 [Jin et al. 2025al ) and reproduce results using official code and checkpoints when necessary. For RQRAG and Self-RAG, evaluate their official 7B checkpoints using FlashRAG. All other baselines use Qwen2.5-3B- Instruct as the foundation model. For consistency, all sirievall-base methods use top-k = 3 retrieved documents. All training and evaluation were conducted using 8 NVIDIA A800-80G GPUs. We provide all source codes, datasets and prompts in this repository
3.3 Training Dataset Construction
To effectively train InForage, we require structured supervision signals for intermediate information- gathering steps. However, most existing QA datasets provide only QA pairs without detailed records of the informationnsseeing process, making them unsuitable for this purpose. To address this gap, construct large-scale dataset designed explicitly to capture comprehensive human information- secking trajectories, inspired by human information foraging behaviors.
Open-Ended Information Browsing and Annotation: To ensure that each example genuinely requires multi-step reasoning and cannot be answered via simple direct search, we adopt an open- ended information browsing paradigm inspired by realistic human behavior. Annotators begin with a seed factoid claim and use the Google Search API to retrieve a search engine results page (SERP). They then iteratively select relevant web pages, extract new "bridge" claims, and expand the context around the original claim through a sirrivalland-extrratiin loop. This process is manually performed for 500 samples. To enforce reasoning complexity, we require that each example integrates at least three non-redundant claims or conditions that must be jointly considered to identify the correct nuswer-ensurine that the final query cannot be resolved by any single piece of information alone. Once a sufficient set of intermediate claims is collected, we use GPT-4o to synthesize a coherent dnestion-answer pair that reflects embedded multi-step reasoning. Human annotators further verify each generated example to ensure there is no information leakage or ambiguity, guaranteeing high- quality supervision signals that reflect realistic, layered information-seeking behavior. An overview of our annotation interface is provided in Figure 3 and Figure 4
Scaling via Automated Annotation: that maintaining consistency and logical coherence within the browsing trajectories critical for high-quality data. To enable scalability while preserving trajectory quality, distill effective
Analysis of the initial 500 manually annotated samples reveals

---

Table 1: Main experimental results. The best results highlighted in bold, and second- results are underlined. For RQRAG and Self-RAG, we use their official 7B checkpoints. All other baselines, including InForage, built Qwen-2.5-3B-Instruee All RAG-based methods apply top-k 3. InForage is trained on the training sets of datasets denoted with "
Method
NQ*
TQA
PopQA
HQA"
2Wiki
Mus
Bamb
Self"
Ave.
Non-RAG Methods
Vanilla
12.1
28.8
13.0
15.9
24.8
4.0
12.9
SFT
24.2
26.3
12.0
20.1
25.2
6.1
13.2
8.1
16.9
Reasoning
24.0
40.2
15.2
20.8
28.0
8.1
16.5
6.7
19.9
RAG
Methods
RAG
34.8
54.4
38.7
25.5
22.6
4.7
8.0
23.4
26.5
IRCoT
31.2
20.0
17.1
24.0
0.8
RQRAG
32.6
52.5
39.4
28.5
30.7
10.1
12.9
28.2
29.4
Self-RAG
36.4
38.2
23.2
15.7
11.3
3.9
5.6
24.0
19.8
Scarch-ol
23.8
47.2
26.2
22.1
21.8
5.4
32.0
36.7
26.9
Scarch-R RI-PPO
32.3
53.7
36.4
30.8
33.6
10.5
31.5
25.6
31.8
40.9
55.2
40.5
34.5
36.9
15.4
32.0
29.2
35.6
InForage (Ours)
59.7
45.2
42.8
17.2
36.0
44.1
prompting strategies from these manually annotated examples. We then leverage GPT-4o as an autonomous agent automate and scale up the annotation process, systematically generating 20,000 coherent reasoning trajectories, each culminating in complex question-answer pair. The final dataset comprises a structured split of 19,500 training examples and 500 evaluation examples. The 500 evaluation samples denoted as the Self dataset conissettcooppseeeeeeee real-time, open-ended web tasks that demand multi-hop reasoning. offering a challenging benchmark for search-enhanced reasoning.
Diverse Web Corpus for Knowledge Freshness:
To mitigate the risk of prior knowledge memoriza- tion within the language model, exclusively crawl web pages published between January 1, 2025, and March 31, 2025. The resulting corpus spans diverse domains-including science, technology, health, culture, and general knowledge- ensure comprehensive topical coverage. After rigorous filtering for quality and relevance, we retain approximately 80,000 high-quality web pages. We employ Qwen2. 5-72B, a strong open-source language model, systematically extract structured factoid claims from each page, resulting in a total of 172,000 validated claims used seed inputs and intermediate annotations.
Golden Evidence for Structured Supervision: Crucially, each annotated example includes explicit records of the golden web URLs that serve as the verified factual basis for the final question-answer pairs. This meticulous documentation facilitates precise evaluation of retrieval performance during training. Retrieval actions that successfully identify evidence from these golden URLs are rewarded accordingly, directly incentivizing effective intermediate retrieval behaviors alongside accurate final-answer generation and overall reasoning efficiency.
3.4 Main Experiment
In Table we present the main experimental results, from which have several key findings: (1) InForage consistently outperforms all baselines across both in-domain and out-of-domain datasets, demonstrating strong robustness and generalizability. This includes superior performance not only on standard QA benchmarks like NQ and TriviaQA but also on more challenging settings such our self- constructed web QA set, which require real-time, multi-step reasoning over open-ended knowledge sources. (2) Search-enhanced reasoning methods (c.g., Search-RI and InForage) show particular strength on multi-hop tasks such as 2Wiki and MuSiQue. Their ability to iteratively decompose queries, formulate subgoals, and retrieve contextually relevant information mirrors human groblem-solving behavior and leads consistently stronger results compared to static, one-shot retrieval pipelines. InForage further benefits from reward- guided reasoning, which encourages the generation of effective subqueries and efficient trajectories.
(3) While prior retrieval- aarreval-augmented methods (e.g., Self-RAG, RQRAG) often perform well on narrow tasks with clearly stated queries, their performance degrades under ambiguity or evolving information needs. contrast, InForage demonstrates greater resilience across task types, confirming hypothesis that adaptive, reasoning-
7

---

w/o SFT
w/o Self'
w/o EP
w/o IGR
w! GRPO
wi PPO
w Qwen-2.5-7B
60
50
40
30
20
TriviaQA
NQ
HolpoiQA
2Wiki
Bamboogle
Self
Ave.
Figure 2: Ablation study of InForage. w/o SFT and w/o Self denote the removal of the supervised fine-tuning stage and the exclusion of the self-constructed dataset, respectively. w/o EP and w/o IGR indicate the ablation of the efficiency penalty and information gain reward. w/ GRPO and w/ PPO compare the use of two different reinforcement learning algorithms. w/ Qwen-2.5-7B refers to replacing the foundation model with Qwen-2.5-7B.
Table 2: Case study using InForage. For clarity, we present only the key content extracted from each retrieved document within the <info> blocks.
Query: Which tennis player with more than ten major titles has excelled at recent top Florida tournament and achieved a significant ranking milestone?
Ground-truth target:
Novak Djokovic
RAG: Sofia Kenin Gemini 2.5 Pro (search): Novak Djokovic
<think> First, identify players who have won more than ten major (Grand Slam) titles. </think>
<search> Which tennis players have more than ten Grand Slam titles? </search>
<info> Novak Djokovic, Rafael Nadal, and have each won more than ten Grand Slam titles. </info> <think> Next, determine top-tier tennis tournament held in Florida that could serve basis for evaluating recent performance. </think>
<search> What is a major tennis tournament held Florida? </search>
<info> The Miami Open is a high-profile anmual tennis tournament held in Florida. </info> <think> Check which of the shortlisted players (Djokovic, Nadal ...) has recently performed well at the Miami Open. </think>
<search> Who among Djokovic, Nadal... has recently recorded wins at Miami Open? </search> <info> Novak Djokovic has recently won matches at the Miami Open. </info>
<think> Finally, confirm whether this player has achieved notable long-term ranking milestone that sets them apart in ATP history. </think>
<search> Has Novak Djokovic reached any major career ranking longevity milestones? </search>
<info> Novak Djokovic has surpassed 1000 weeks ranked within the top 100. </info>
<answer> Novak Djokovic </ a/aswer>> < InForage's Answer.
aware retrieval is crucial for complex tasks. Its principled integration of retrieval into the reasoning loop, guided by outcome, information gain, and efficiency signals, enables the model dynamically navigate complex knowledge spaces more effectively than static or manually designed strategies.
3.5 Discussion
Ablation Study To evaluate the contributions of InForage's key design components, we conduct comprehensive ablation studies, as summarized Figure _ Our findings are follows: (1) Training Settings: Leveraging our self-constructed dataset for both supervised fine-tuning (SFT) and reinforce- ment learning (RL) yields consistent performance improvements. This confirms that high- quighttulity, reasoning aaiione training data is essential for search-enbance reasoning methods particularly, SFT provides a strong initialization that benefits subsequent RL optimization. (2) Reward Design: Ablating the information gain reward (IGR) leads to consistent performance drop across datasets, validating its effectiveness in promoting meaningful intermediate retrievals. Removing the efficiency penalty (EP) also degrades performance, except on self-constructed dataset. This suggests that

---

complex, real-world information seekiin tasks may inherently require longer reasoning chains, mak- ing the balance between step efficiency and answer completeness task-dependent. (3) RL Algorithms: Comparing PPO and GRPO, we observe that PPO generally yields better results. While GRPO occasionally outperforms PPO on specific datasets, PPO's s use of learned reward signals (rather than rule-based estimates) appears better suited for evaluating nuanced reasoning quality in complex tasks. (4) Model Scaling: Replacing the 3B foundation model with Qwen2.5-7B leads to further gains across most benchmarks, demonstrating that InForage effectively scales with model capacity and can harness larger LLMs for improved performance.
Case Study To concretely illustrate the reasoning process of InForage, conduct case study on a test example from our celf-constructed dataset. As shown in Table 2 the input query asks about a tennis player who satisfies multiple overlapping conditions, each corresponding to several potential candidates. Only by combining all the constraints can the correct answer be uniquely identified. This type of embedded informmtioonseekinn task poses a significant challenge for traditional RAG methods, which rely on one- one-pass retrieval and often miss critical evidence. As a result, vanilla RAG fails to find the correct answer. In contrast, InForage accurately decomposes the layered intent of the query, iteratively retrieves relevant evidence, and successfully identifies the correct answer denoos--dmoosttraiing its strong search-enhanced reasoning capability. For comparison, Gemini 2.5 Pro, state-of-the-aa LLM with built-in search tools, also produces the correct answer. However, In InForage achieves this using a much smaller 3B model, highlighting its efficiency and effectiveness.
4 Related Work
Retrieval-Augmented LLMs:
Despite the rapid progress of L.MSs OpenAl 2023 Group 2025
their inherent limitations- such as hallucination and outdated knowledge remain obstacles to real-
world deployment (Gao et al. 2024 Zhao et al. 2024a]. Retrieval-Augmeente Generation (RAG), first proposed by Lewis et al. 2020b), addresses these challenges by equipping LLMs with external retrieval capabilities to provide contextually relevant, up-to-date information [Izaard and Grave Gao et al., 2024). This paradigm not only improves factual accuracy but also enhances temporal relevance. Subsequent research has focused on two fronts: improving retrieval quality to raise the generation ceiling [Qian et al., [202aa, Gao et al. [2024]. and optimizing the integration of retrieved content into the generative process [Jang et al. 2023 /Zhao et al. 2024b). However, traditional RAG methods typically assume well-detined queries and structured knowledge, limiting their scope to simple factoid QA (Nogueira and Cho 2020 Lewis al., 2020b). J. Recent work has emphasized the inadequacy of this static setup for real-world tasks, where information needs are often implicit, ambiguous, or evolving (Qian et al., 2024b). These settings demand adaptive, multi-step retrieval guided by iterative reasoning. To this end, retrieval-augmeentee reasoning has emerged, with recent innovations embracing more dynamic and structured approaches: GraphRAG (Edge et al. 2024] and HipPoRAG [Jimenez Gutierrez et al., 2024] enhance global awareness through graph- based representations, while agent-driven systems like ActiveRAG [Xu et al., 2024) integrate planning and evidence aggregation. Together, these advances highlight a growing need for RAG systems that move beyond static setrieval-towre reasoning-awan aware, goal-driven interaction with external knowledge to meet the demands of complex, open-ended tasks.
2021
Reasoning LLMs: Enhancing the reasoning capabilities of LLMs has become central focus in advancing their ability to tackle complex, real-world tasks. Progress has been made across the entire training pipeline: pretraining on code and mathematical data fosters structured thinking [Wei]
et al. 2022]; supervised fine-tuning on reasoning-rich datasets improves response fidelity; and alignment through reinforcement Icarning and preference modeling refines multistep reasoning behavior [Gulcehre et al. 2023 Kumar et al. 2024 Zhang et al., 2024]. In parallel, prompting strategies- such as th-uugh [Wei et al.] 2022). tree-of-thought (Yao et al., [024]] and ReAct (Yao et al. [0233 guuide models to explicitly decompose problems and explore multiple solution paths. Notably, reasoning-centrii models like OpenAl's ol and DeepSeek RI have achieved impressive performance challenging domains such mathematical problem-solving and coding, where iterative reflection and progressive refinement are key [OpenAl 2024 DeepSeek-Al 2025).
However, these models typically operate on static internal knowledge, limiting their adaptability in knowledge-sparse or dynamic contexts. To overcome this constraint, recent research has shifted toward search-augmented reasoning, which equips LLMs with the ability to issue subqueries and

---

iteratively gather external evidence during inference. For instance, Search-R1 I [Jin et al. 2025a introduces a reinforcement learning framework that teaches LLMs when and how to interact with search engines. Search-ol [L et al.) 2025 integrates retrieval directly into the ol reasoning loop and introduces a
module filter noisy content before reintegration. Extending further into open- open-world settings, DeepResearcher (Zheng et al. [0255 applies end-to-end RL to train agents that can navigate unstructured web environments, demonstrating emergent behaviors such as planning, evidence synthesis, and self-correction.
Building on this evolving landscape, we propose InForage, a framework that models search-enhanced reasoning as an information foraging process. Unlike prior methods that reward only final answers, effective subqueries, relevant retrievals,
InForage optimizes the full reasoning dnjiectry-encourrae and efficient solutions. By bridging cognitive theory and reinforcement learning, it offers a more adaptive and holistic approach to retrieval-augmentie reasoning.
Conclusion
This paper presents InForage, reinforcement learning framework that advances search-augmented reasoning in LLMs by drawing on principles from Information Foraging Theory. Rather than treating retrieval as static, pre-inference step, InForage integrates it dynamically into the reasoning process, rewarding not only correct final answers but also informative intermediate retrievals and concise solution paths. This structured optimization encourages models to emulate human-like information-seedt seekong behaviors by formulating subqueries, evaluating partial evidence, and refining reasoning trajectories. Empirical results across broad range of benchmarks show that In InForage consistently surpasses all baselines. Its iterative reasoning capabilities prove especially effective on complex queries requiring layered evidence integration, demonstrating robust generalization across task types. Moreover, InForage exhibits strong scalability with larger model sizes and comprehensive analyses validate the effectiveness of InForage's 's method design. Together, these findings highlight the importance of coupling reasoning with retrieval in a learning-driven, context-sensitive manner. InForage offers unified and principled solution that closes the gap between static retrieval pipelines and the dynamic, adaptive reasoning needed for real-world knowledge-intensiee applications.
Acknowledgement
This work was supported by National Natural Science Foundation of China No. 62502049.
References
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, al. Reureeeal-uugmented genera- tion for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33: 9459-9474, 2020a.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang. Junjie Zhang, Zican Dong, Yifan Du Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian Yun Nie, and Ji-Rong Wen. A survey of large language models, 2024a.
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Qianyu Guo, Meng Wang, and Haofen Wang. aetrieval-augmeent generation for large language models: A survey, 2024.
Hongjin Qian, Zheng Liu, Chao Gao, Yankai Wang. Defu Lian, and Zhicheng Dou. Hawkbench: Investigating resilience of rag methods on stratified information-seeking tasks, 2025a.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling I1m test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03311, 2024.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel J. Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. CoRR, abs/2501. 2025.
10

---

OpenAl. Openai ol system card. CoRR, abs/2412.16720) 2024.
DeepSeek-AL, Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. CoRR, abs/2501. 12948, 2025.
Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. Search-ol: Agentic search-enhanced large reasoning models. CoRR, abs/2501.05366 2025.
Bowen Jin, Hansi Zeng. Zhenrui Yue, Dong Wang, Hamed Zamani, and Jiawei Han. Search- rl: Training Ilms to reason and leverage search engines with reinforcement learning. CoRR, (bss/550309514 2025a.
Hongjin Qian, Zheng Liu, Peitian Zhang, Kelong Mao, Defu Lian, Zhicheng Dou, and Tiejun Huang. Memorag: Boosting long context processing with global memory-enhanced retrieval augmentation. In Proceedings of the ACM on Web Conference 2025, WWW '25, page 2366-2377 New York, NY, USA, 2025b. Association for Computing Machinery.
Yutao Zhu, Huaying Yuan, Shuting Wang. Jiongnan Liu, Wenhan Liu, Chenlong Deng. Haonan Chen, Zhicheng Dou, and Ji-Rong Wen. Large language models for information retrieval: A survey, 2024.
Yujia Zhou, Yan Liu, Xiaoxi Li, Jiajie Jin, Hongjin Qian, Zheng Liu, Chaozhuo Li, Zhicheng Dou, Tsung-Yi Yi Ho, and Philip S. Yu. Trustworthiness retrieval-augmmented generation systems: A survey. CoRR, abs/2409. 10102, 2024.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answeriig with human feedback. arXiv preprint 21X1v:211 2021.
Lior Shani, Aviv Rosenberg, Asaf Cassel, Oran Lang, Daniele Calandriello, Avital Zipori, Hila Noga, Orgad Keller, Bilal Piot, Idan Szpektor, et al. Multi-turn reinforcement learning from preference human feedback. arXiv preprint arXivv:405.1 14655, 2024.
Peter Pirolli and Stuart Card. Information foraging. Psychological review, 106(4):643, 1999.
Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng. Jiannan Guan, Peng Wang. Mengkang Hu, Yuhang Zhou, Te Gao, and Wanxiang Che. Towards reasoning era: A survey of long chain-of- thought for reasoning large language models. arXiv preprint arXiv:2503.0956: 2025.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, NWW11707006447, 2017.
John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. In Yoshua Bengio and Yann LeCun, editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Interleaving retrieval with tffin-of-thought reasoning for knowledge-intensive multi-step questions. arXiv preprint arXiv:2212. 10500, 2022a.
Chi-Min Chan, Chunpu Xu, Ruibin Yuan, Hongyin Luo, Wei Xue. Yike Guo, and Jie Fu. RQ-RAG: learning to refine queries for retrieval augmented generation. CoRR, abs/2404.00610,, 2024. doi: 1044550/ARXXV22244..0001/
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reftection. arXiv preprint arXiv:2310. 102211005511111 //511, 2023.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466, 2019.

---

Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv: metri07050..5// 2017.
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Hannaneh Hajishirzi, and Daniel Khashabi. When not to trust language models: Investigating effectiveness and limitations of parametric and non non-parametric memories. arXiv preprint 0rXiv:2212.10511. 2022.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint 18XXv:/809.09900
2018.
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing multi-hop qa dataset for comprehensive evaluation of reasoning steps. arXiv preprint arXiv:2011.01 1.01060, 2020.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539-554 2022b.
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models, 2023,
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly contrastive pre-training, 2024.
Jajje Jin, Yutao Zhu, Zhicheng Dou, Guanting Dong. Xinyu Yang, Chenghao Zhang. Tong Zhao, Zhao Yang, and Ji-Rong Wen. Flashrag: A modular toolkit for efficient eerieeel-augmeente generation research. In Companion Proceedings of the ACM on Web Conference 2025, pages 737-740, 2025b.
OpenAl. Gpt-4 technical report, 2023.
Qwen Group. Qwen2.5 technical report, 2025.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-Aummented Generation for mnowledge-intensin NLP tasks. In Advances in Neural Information Processing Systems, volume 33, pages 9459-9472 2020b.
Gautier Izacard and Édouard Grave. Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 874-880, 2021.
Hongjin Qian, Zheng Liu, Kelong Mao, Yujia Zhou, and Zhicheng Dou. Grounding language model with chunking free in-context retrieval. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 1298-1311. Association for Computational Linguistics, 2024a.
Zhengbao Jiang, Frank F. Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi- Yu, Yiming Yang, Jamie Callan, and Graham Neubig. Active retrieval augmented generation. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 7969-7992. Association for Computational Linguistics, 2023.
Qingfei Zhao, Ruobing Wang, Yukuo Cen, Daren Zha, Shicheng Tan, Yuxiao Dong, and Jie Tang. Longrag: dual-perspectivv retrieval- atrmeval-augmenneed generation paradigm for long-context question answering. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, pages 22600-22632. Association for Computational Linguistics, 2024b.
Rodrigo Nogueira and Kyunghyun Cho. 101.::190..04085, 2020.
Passage re-ranking with bert.
arXiv preprint
12

---

Hongjin Qian, Zheng Liu. Peitian Zhang, Kelong Mao, Yujia Zhou, Xu Chen, and Zhicheng Dou. Are long-Ilms a necessity for long-context tasks? arXiv preprint arXiv:2405.15316 2024b.
Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson. From local to global: A graph rag approach to query-focused summarization, 2024.
Bernal Jimenez Gutierrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, and Yu Su. Hipporag: Neurobio- logically inspired long-term memory for large language models. Advances in Neural Information Processing Systems, 37:59532-59569, 2024.
Zhipeng Xu, Zhenghao Liu, Yibin Liu, Chenyan Xiong, Yukun Yan, Shuo Wang, Shi Yu, Zhiyuan Liu, and Ge Yu. Activerag: Revealing the treasures of knowledge via active learning. CoRR, abs/2402.13550 13547, 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308, 089988 2023.
Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, al. Training language models to self-correct via reinforcement learning. arXiv preprint 2rXiv:2409.122 12917, 2024.
Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mets*: Llm self-training via process reward guided tree search. Advances in Neural Information Processing Systems, 37.44735-67777 2024.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023.
Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. Deepresearcher: Scaling deep research via reinforcement learning in real-worke environments, 2025.
13

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
In Section 3, including the main experiments and discussions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer:[Yes Yes]
Justification:
In Appendix, we have a Limitation section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
This paper does not contain theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer:[Yes
Justification:
We have disclosed necessary details for our result reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer:[ Yes]
Justification: In Appendix, we have a section to discuss the implementation details of our paper. We also provide source codes and training script in the supplement material. Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
//nips. //nips.cc/ppl wwwppbblccgguiicuceccccccoe.oopp
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: Yes]
Justification:
In Appendix, have implementaion details section to disclose these details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:[Yess Yes]
Justification:
We did t-test for the main experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: Yes]
Justification: We disclose the required computing resources the implementation details section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:[Yer [es]
Justification:
We follow the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:[ Yes]
Justification: discuss impact of this paper in Appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer:[NA]
Justification: the models in this paper are trained for specific search scenarios, which does not pose such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer:[Yes]
Justification:
We credited all used resources in this paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.paaersstt paperswi thcode. cow/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer[[Yes
Justification:
We introduced the details about our constructed training data the main
content.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer:[NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer:[NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
19

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer:[Yes]
Justification: We have described the usage of LLMs as core component of our method in the paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy btttps://neuriiee wwweerrppps.ccoooeeeeeeeeeee /t//oonfeeencee//222222 for what should or should be described.
20

---

A Prompts
Below we present the core prompts used this paper. The first prompt is designed for search- enhanced reasoning, guiding the model to perform step-by-step retrieval and inference. The second prompt is used to generate question-answer pairs from a batch of extracted factual claims. For completeness, additional prompts-such those used for claim extraction-are provided in the source code included in the supplementary materials.
Prompt for Search-Enhances Reasoning
You are an intelligent agent designed solve complex queries or tasks by retrieving external knowledge and reasoning step by step.
Please follow these instructions carefully:
1. For each piece of information you receive:
Think step by step and explain your reasoning inside <think> and </think> tags.
2. If you need more information to proceed:
Issue a search by writing your subquery inside <search> and </search> tags.
Retrieved results will appear between <evidence> and </evidence> tags.
You can conduct multiple searches as needed.
3. When you have collected enough information:
Provide your final answer using the <answer> and </answer> tags.
Do not include explanations reasoning in the answer block.
Keep your answer concise.
Now, solve the following task: Task: [question]
Prompt for Query-Answer Pair Generation
Based on the following evidence, generate complex multi-hop query that requires connecting multiple pieces of information
Create a challenging question that requires reasoning across multiple facts. The question should be specific enough that can only be answered by connecting several pieces of evidence together.
For example, given the evidence list:
(Example Evidence List]
The multi-hop query could be: (Example Query)
Requirement:
1. Ensure Coherence: Make sure the question flows logically from the combined information and is clear and unambiguous
2. Formulate the Question: Create question that cannot be answered by relying on just one
of the sentences but instead requires understanding and linking the information from all of the sources.
3. Ensure Multi-hops The question should require at least 3 logical steps to answer.
4. The answer should be specific and short. Now, based on the following evidence, generate multi-hop query that requires at least logical steps answer: (evidence_stt
Generate a multi-hop query that requires least logical steps answer. Output in the following format: ("query": "multi-hop query", "answer": "answer")
B Limitations and Broader Impact
Limitations. In this work, we introduce InForage, a reinforcement learning framework that en- hances search-augmented reasoning by aligning retrieval actions with evolving reasoning goals, inspired by Information Foraging Theory. While our results demonstrate strong performance across range of QA tasks, several limitations remain.
21

---

First, due to computational constraints, our experiments primarily focus on Qwen2.5-3B and 7B models. We do not explore larger foundation models or alternative model families (c.g. LLaMA, Mixtral), which could further enhance performance. Future work will expand evaluation across more model scales and architectures to validate generalizability.
Second, while InForage shares high-level goals with emerging search-enhanced reasoning models, many these works are still in progress or unpublished the time of submission. As such, we do not include all of them in our experimental comparisons, though we provide design-level discussions highlight conceptual differences.
Third, our ceff-constructed dataset focuses on QA tasks with short-form answers to ensure verifiabil- ity cyu-cl for sule-based reward assignment. While we believe the InForage framework is extensible to complex tasks such as long-form synthesis, report writing, and multi-document summarization, these applications not explored in this paper.
Broader Impact. The central contribution of this paper is the shift of retrieval from a static, pre-inference step dynamic, eessoning-integratn process. This mirrors how humans seek infor- mation-thinking searching, and composing in parallel-and offers more flexible and cognitively aligned paradigm for knowledge-intensive tasks. We believe this capability can generalize domains where adaptive reasoning is essential, including scientific writing, investigative journalism, legal analysis, and knowledge curation.
Moreover, while we focus on retrieval as the primary external tool, the proposed framework and dataset pipeline are tool-agnostic. Our methodology with other tools such as code compilers, search APIs, or structured databases. By offering a com- plete pipeline from dataset construction to multi-stage reward design and reinforcement learn- ing InForage lays the groundwork for building general-purpose purpose rool-augmented reasoning agents that are better aligned with real-world human workflows.
be extended to optimize interactions
22

---

Depty
tolcst wids
Selected Claims
Currently Selected Record
4ad Harissnn unkverity
Added Records
Tapic commos trwer
Target Pancee Harroon
Current Search Query
Select Search Query
Focedd ghen vor
Use "op Query
Use Torget Quiry
Grentercutten
Generat Queston
Ciror All vecoods
the Uniernty Oregat
Febredh
Use Lusaon Query
Current Search Results
Anmotat
Select Search Method
Cacbe Soarh
Google Search
Search Results
Result #1
Result #2
Result #3
University
University Athiecics Offical Athierica Webaate
Admrsssios Financial Apply UO
Acadimic Research Excelience
Quick "ickets rollow
Learn adriissoen
en Sox Modia Conanl
fr en ald ded Univeruu of
ardtecooo musi and sclene asstrcc art and disrete
Ualten Support Donate
designed support
metweraiie, henloeer inowrtion
right Orogan Nil...
Use Reaut
Use Rezsht
this Dorutt
Figure 3: Annotation page for construct the training dataset.
Claims Extracted from Web
Selected Claims
Added Records
Claim
Claim
Focee) up Aerppiaaen.
Topic: regular updatee
Toplc: highest-rated 2006 recruitin the West
Toplc commitment
Corne
kendre Honter cureetly the the West.
commisted to baiversilyyy Ougann
nquitaly and the latest commentents.
Toceet remnttennt
Torget
Targrt 2008 Supritte Football Connitonent thckee
Generate Quaitio
Osar All feeords
complex pairs
X
Jise th CCalre
Roirash
Cenerate
Make Question More Complee
Anactation Staisties
('ouery:: univeratity hiighss
Quaatore: thoteeeeee
thoupd SuptWost Trodee??
Oregae
Ouridion
Figure 4: Annotation page for construct the training dataset.
23


Reviews:
[{"review_id": "9cucYlIg7l", "reviewer": "Reviewer_XiLb", "summary": "The paper presents a new technique for training deep research-like systems. It goes beyond search-r1, using a mixture of outcome reward, information gain reward, and length penalty. My understanding is that the information gain reward is the most novel component, and it is quite intuitive why it would be helpful. It is not a pure RL approach. There is an initial SFT stage followed by PPO. The results across multiple relevant datasets show clear improvement over search-r1.", "questions": "How extensively did you attempt to get GRPO to improve over PPO? Does your comment about learned rewards imply that training was not stable? Was Pass@K sufficiently high to your liking? It is interesting to know whether we can expect models to automatically discover better trajectories, or we simply need to mimic the collected SFT data.\n\nCould information gain be approximated by a judge w/ limited or no information of gold documents?", "limitations": "n/a", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. This is one of the hottest topic areas, combining search and multi-step agentic rollouts. In particular, this adds an interesting reward in the form of information gain. There remains only limited research on RL + search + generation despite it's popularity.\n\n2. The result shows a clear improvement over the next best method.\n\n3. There is extensive ablation to help understand the importance of SFT, RL algo, rewards, etc.\n\nWeaknesses\n\n1. Perhaps would be more interesting to explore datasets with many gold retrieval labels. Although this seems alleviated by the Self dataset. The Self dataset could include more details, and it is not clear if it is as high quality as the other included benchmarks --- it is strange it is the one dataset where GRPO did not do as well given that the reward seems most aligned w/ the Self dataset.\n\n2. SFT seems to be particularly critical. In other words, collecting more of this trajectory data could be greater value than algorithmic developments in the short term.\n\n3. It could be interesting to evaluate deep research industry systems more directly, although would be understandably out of scope.", "was_revised": true, "final_justification": "I think the paper has many strengths and is significant given that deep research is highly popular yet very few technical details have been published on how to create a deep research system. The concerns were minor, yet the author response addresses them nicely. I did not see any major concerns from others that convince me to change my score.", "rebuttal": "We greatly appreciate reviewer **XiLb’s** recognition of the significance, clarity, and originality of our work. We are especially grateful for the positive comments regarding the introduction of information gain as a reward signal, the comprehensive ablation studies, and the clear improvements demonstrated over prior methods.\n\nYour recognition is greatly appreciated by our research team.\n\nBelow, we address your questions and suggestions in detail.\n\n------\n\n## **1. Exploring Datasets with Gold Retrieval Labels & the Quality of the SELF Dataset**\n\nWe fully agree that evaluating on datasets with many gold retrieval labels provides meaningful comparison. However, most widely-used QA benchmarks lack gold retrieval annotations, which motivated our construction of the **SELF** dataset.\n\nAs described in Section 3.2, the SELF dataset was built by first manually browsing the web to collect authentic human browsing trajectories and identify key decision points. To scale up data collection while maintaining quality, we leveraged strong LLMs as autonomous agents to plan and make retrieval decisions, based on these initial human examples. We then performed strict quality control, filtering out queries that could be correctly answered without retrieval, or that could not be correctly answered even with all relevant information.\n\n*We will add more details about the construction and quality assurance of the SELF dataset in the revised version.*\n\nRegarding the performance of GRPO on the SELF dataset, although one might expect GRPO to excel given the reward signal alignment, we observed that PPO slightly outperformed GRPO, especially on SELF.\n\nWe believe this is because our InForage model is first supervised-fine-tuned on constructed reasoning trajectories from SELF, giving PPO’s critic a well-informed starting point and leading to more accurate advantage estimation. In contrast, GRPO is less able to benefit from this supervised initialization, resulting in its slightly lower performance.\n\n------\n\n## **2. The Critical Role of SFT Data Collection**\n\nThank you for highlighting this point. We agree that collecting high-quality trajectory data for SFT is particularly valuable at the current stage. High-quality trajectories provide rich, step-by-step supervision signals that closely mirror the actual reasoning and retrieval processes required by the target tasks. This, in turn, makes it easier for the model to receive rewards during the RL training stage, thereby alleviating the sparse reward issue.\n\nNonetheless, we believe that both high-quality data and algorithmic advances are crucial and complementary.\n\nRich supervision signals from data unlock the full potential of advanced algorithms, while improved algorithms help us better utilize and generalize from the available data.\n\n*We will further emphasize this interplay in the revised version.*\n\n------\n\n## **3. Direct Evaluation on Deep Research Industry Systems**\n\nWe appreciate this suggestion.\n\nDirect evaluation on commercial deep research systems would indeed be valuable; however, current industrial systems are often closed-source, use non-aligned retrieval modules, or lack transparent implementation details, making direct comparison infeasible at this stage.\n\n------\n\n## **Responses to Specific Questions**\n\n### **1. How extensively did you attempt to get GRPO to improve over PPO?**\n\nAs noted, the SFT stage enables PPO’s critic to start from a well-informed value function, leading to more accurate advantage estimation and higher performance in our setting. We conducted extensive ablation, but GRPO was generally less able to benefit from this supervised initialization.\n\n### **2. Does your comment about learned rewards imply that training was not stable?**\n\nWe found that GRPO’s stability depends on the choice of group size and the reliability of verifiable reward signals. In our experiments, PPO’s critic structure contributed to more stable and effective training.\n\n### **3. Was Pass@K sufficiently high to your liking?**\n\nWe primarily report pass@1 accuracy, as aggregating multiple answers from different rollouts is non-trivial for QA tasks and may not reflect realistic usage.\n\nWe consider pass@1 a more practical and fair metric.\n\nAs shown in our results, InForage consistently outperforms baseline models.\n\n### **4. Can models automatically discover better trajectories, or do they mainly mimic SFT data?**\n\nThank you for raising this point. Our SFT data, synthesized from real web browsing records, provides high-quality supervision and is critical for model foundation. During RL post-training, the model is encouraged to autonomously explore and discover improved trajectories guided by reward signals. In practice, RL-based post-training further improves performance on top of SFT, suggesting the model can move beyond simply mimicking supervised data.\n\n### **5. Could information gain be approximated by a judge without access to gold documents?**\n\nInformation gain is a crucial signal, but many datasets lack intermediate labels. To address this, we consider two potential strategies:\n\n- **(1)** Using strong models with predefined workflows to simulate and recover valid intermediate steps, retaining those trajectories that ultimately lead to correct answers as additional training data;\n- **(2)** Employing a relevance model to estimate information gain via query-document relevance.\n\nHowever, the latter’s accuracy is highly dependent on the relevance model.\n\n*We will discuss these strategies and their limitations more thoroughly in the revised manuscript.*"}, {"review_id": "iRJTXs0bV3", "reviewer": "Reviewer_sR8g", "summary": "This paper introduces InForage, a reinforcement learning framework that enhances search-augmented reasoning in LLMs by drawing inspiration from Information Foraging Theory (IFT). Unlike traditional retrieval-augmented generation methods that perform static pre-inference retrieval, InForage dynamically integrates retrieval during the reasoning process. The method uses three complementary reward mechanisms: outcome reward, information gain reward, and efficiency penalty. The authors evaluate on multiple QA benchmarks, showing consistent improvements over baselines.", "questions": "1. Coverage function definition: How exactly is the coverage function C in Eq. 3 computed? What constitutes the \"expected knowledge set D*\" in practice, and how sensitive are results to this definition?\n\n2. Baseline comparisons: It might be helpful to compare InForage with FLARE (Forward-Looking Active REtrieval) and more sophisticated IRCoT variants not included? Also, many close-source LLMs have search API (eg. GPT-4o) and enables search with reasoning. It would be more comprehensive to include close-source LLM as well in the experiments as well.\n\n3. Scalability validation: Given that most experiments use 3B models, can you provide more comprehensive results on larger models (13B+, like LLaMa3) to validate that the approach scales effectively?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths:\n\nQuality: The work is technically sound with a well-designed three-component reward system that addresses both intermediate retrieval quality and final outcomes. The experimental methodology is thorough, including proper ablation studies and statistical testing. \n\nClarity: The paper is generally well-organized with clear motivation from Information Foraging Theory.\n\nSignificance: Addresses a key open problem—adaptive retrieval during reasoning—and delivers improvements with small models. Provides a pipeline to construct information-seeking QA dataset.\n\nOriginality: While iterative retrieval has been explored, the specific application of Information Foraging Theory provides a novel theoretical lens. The multi-faceted reward design combining outcome, information gain, and efficiency is a meaningful contribution that differentiates this work from prior approaches.\n\n### Weaknesses:\n\nQuality: Limited evaluation on larger models raises scalability questions. Missing comparisons with some relevant recent baselines (FLARE, advanced IRCoT variants). The efficiency penalty shows inconsistent benefits, suggesting the reward design may need refinement.\n\nClarity: Key technical details lack precision - the coverage function C in Eq. 3 is undefined, and the operationalization of \"information scent\" from theory to implementation is unclear.\n\nSignificance: Ablation study shows that performances heavily relies on self-constructed dataset which might be a scalability concern.\n\nOriginality:  Reward shaping (answer + coverage + length) resembles Search-R1-GRPO and REST-style self-training; novelty is incremental.", "was_revised": true, "final_justification": "During rebuttal, the author well addressed my concerns of the paper on several perspectives - better clarity on coverage function and reward design, additional experiments with different baselines and scale, and novel design tailored for search-augmented reasoning model. \nI'd love to consider this paper as well-motivated, clear narrative, and experimentally solid. I'd grant my final rate as 4: borderline accept.", "rebuttal": "We sincerely thank Reviewer **sR8g** for the thorough and thoughtful assessment of our work, as well as for highlighting the technical soundness, theoretical motivation, and practical contributions of InForage. We address each of your comments and questions in detail below.\n\n## 1. Baseline Comparisons\n\n**Reviewer:**  \n*It might be helpful to compare InForage with FLARE (Forward-Looking Active REtrieval). Also, many closed-source LLMs have search APIs (e.g., GPT-4o); it would be more comprehensive to include closed-source LLMs in the experiments. Scalability validation: Given that most experiments use 3B models, can you provide more comprehensive results on larger models (13B+, like LLaMa3) to validate that the approach scales effectively?*\n\n**Response:**  \n\n\n|          | Foundation Model | NQ       | TQA      | HotpotQA | 2wiki    | PopQA    |\n| -------- | ---------------- | -------- | -------- | -------- | -------- | -------- |\n| FLARE    | Qwen 2.5-3B      | 21.2     | 50.2     | 24.3     | 33.2     | 20.5     |\n| InForage | Qwen 2.5-3B      | 42.1     | **59.7** | 40.9     | 42.8     | 45.2     |\n| FLARE    | Qwen 2.5-7B      | 22.5     | 55.8     | 28.0     | 35.9     | 22.7     |\n| InForage | Qwen 2.5-7B      | **47.8** | 59.2     | **48.1** | **46.2** | **49.1** |\n\n(1). Per the reviewer’s suggestion, we have added experimental comparisons with FLARE on both Qwen 2.5-3B and 7B backbones. Across the evaluation datasets, InForage consistently outperforms FLARE.\n\n|                | HotpotQA | 2wiki    | PopQA    |\n| ------------  | -------- | -------- | -------- |\n| GPT-4o-mini      | 34.5     | 30.7     | 39.7     |\n| GPT-4o      | 45.8     | 50.5     | 52.1     |\n| InForage-3B    | 40.9     | 42.8     | 45.2     |\n| InForage-7B     | 48.1     | 46.2     | 49.1     |\n| InForage-14B     | **50.9** | **51.4** | **52.9** |\n\n(2). Per the reviewer’s suggestion, in addition to the 3B and 7B versions of InForage discussed in the paper, we also report results for InForage-14B. Furthermore, we evaluate against commercial closed-source LLMs, specifically OpenAI’s GPT-4o and GPT-4o-mini, both equipped with search tools via API. The results demonstrate that InForage achieves superior performance compared to these commercial APIs. A potential reason is that commercial APIs typically invoke the search tool in a single pass, which may not sufficiently accumulate information for multi-hop QA tasks.\n\nBesides, across the 3B, 7B, and 14B backbone models, InForage consistently improves its performance, verifying the scalability of our approach.\n\n---\n\n## 2. Efficiency Penalty and Reward Design\n\n**Reviewer:**  \n*The efficiency penalty shows inconsistent benefits, suggesting the reward design may need refinement.*\n\n**Response:**  \nThank you for your observation. As discussed in our paper, the efficiency penalty is designed to balance reasoning efficiency and depth. Its optimal value may depend on the nature of the task: tasks requiring fewer retrieval steps and more sensitive to efficiency benefit from a larger penalty, while tasks demanding deeper reasoning may benefit from a smaller penalty. Our ablation study shows that most evaluation datasets benefit from the efficiency penalty, except for the SELF dataset—which typically requires more reasoning steps, as it is intentionally designed to be as complex as possible. In practical scenarios, such highly complex tasks are uncommon, so the efficiency penalty remains effective for most real-world applications.\n\n\n---\n\n## 3. Coverage Function Definition, Implementation, and Sensitivity\n\n**Reviewer:**  \n*Key technical details lack precision – the coverage function C in Eq. 3 is undefined, and the operationalization of “information scent” from theory to implementation is unclear.*  \n*How exactly is the coverage function C in Eq. 3 computed? What constitutes the “expected knowledge set D*” in practice, and how sensitive are results to this definition?\n\n**Response:**  \nThank you for highlighting these important points. In our framework, inspired by Information Foraging Theory, “information scent” refers to the evolving trail of reasoning steps and subqueries that guide retrieval. The coverage function \\($\\mathcal{C}$\\) in Eq. 3 quantifies how well the union of all retrieved knowledge patches matches the expected knowledge set \\($\\mathcal{D}^*\\$), typically measured as the recall (or proportion) of gold evidence covered during the reasoning trajectory. Speficially, \n\n- **Coverage function computation:**  \n  As described in Section 2.3 (and Eq. 6), the coverage function \\( $C\\left( \\bigcup_{\\tau=1}^t \\mathcal{K}_\\tau, \\mathcal{D}^* \\right) $\\) measures the cumulative coverage of all retrieved knowledge patches up to step \\( $t$ \\) against the expected knowledge set \\( $\\mathcal{D}^*$ \\). At each retrieval step, we compute the overlap between all retrieved sets so far and the gold set, and the information gain reward is defined as the maximum coverage achieved throughout the reasoning process.\n- **Expected knowledge set \\($\\mathcal{D}^*$\\):**  \nIn practice, $\\mathcal{D}^*$ is constructed from annotated gold evidence, represented by the complete set of URLs, with each URL corresponding to a gold document deemed sufficient to support the final answer for each instance. This set serves as the authoritative reference for both reward computation and evaluation throughout training and testing.\n- **Sensitivity analysis:**  \n  Through ablation studies, we find that the information gain reward consistently benefits our method design, validating its effectiveness across various datasets. This demonstrates that our approach remains robust as long as \\($\\mathcal{D}^*$\\) accurately reflects the core factual basis for each instance.\n\nThis design rewards not only correct final answers but also reasoning trajectories that effectively accumulate relevant information, providing a principled and practical operationalization of information scent. We will further clarify these definitions and implementation details in the revised manuscript.\n\n---\n\n## 4. Scalability and Use of Self-Constructed Dataset\n\n**Reviewer:**  \n*Ablation study shows that performance heavily relies on self-constructed dataset, which might be a scalability concern.*\n\n**Response:**  \nWe appreciate this important concern. The self-constructed dataset was carefully designed to align with our training schema and task requirements. In practice, this data brought a notable performance improvement. Our dataset construction process began with manual annotation and was subsequently scaled up using strong LLMs. To promote transparency and reproducibility, we have included the annotation interface and data generation scripts in the supplementary material. We believe this approach can be efficiently scaled or adapted to other domains by leveraging these resources.\n\n---\n\n## 5. Incremental Novelty over Related Reward Shaping Approaches\n\n**Reviewer:**  \n*Reward shaping (answer + coverage + length) resembles Search-R1-GRPO and REST-style self-training; novelty is incremental.*\n\n**Response:**  \nThank you for raising this point. Incentivizing LLMs for search-enhanced reasoning is indeed an active area of research. Search-R1 is a representative recent work, which, like most similar approaches, applies only final answer correctness as the reward signal. At the time of our submission, there was limited systematic construction of reward signals tailored specifically for search-augmented reasoning models. Our core contribution lies in proposing a theoretically motivated learning framework, integrating carefully designed and complementary reward components, and constructing the necessary data to enable and evaluate this framework. We believe this solid step provides a practical foundation for further research in this direction.\n\n---\n\n\n\nOnce again, thank you for your constructive feedback and detailed review. We will incorporate clarifications and additional results as described above in the revised manuscript."}, {"review_id": "zmMDpe6qI6", "reviewer": "Reviewer_ipjm", "summary": "The paper introduces a search augmented reasoning approach in LLMs by using information foraging principle (e.g. information scent, information patches, cost and gain). Beyond correctness of the final answer, the proposed approach rewards effective information seeking behavior at each stage of the reasoning process as well. Effectiveness is measured as: 1- coverage of the retrieved information patches compared to the expected knowledge; 2- total number of reasoning/retrieval steps as a penalty.  Authors use a data set consisting of human web browsing trajectories and corresponding QA pairs for Supervised Fine Tuning the foundation LLM and use the reward rule to optimize its performance. \n\nThe paper reports experimental results measuring the performance of the proposed method against a set of alternatives (e.g. vanilla prompting, SFT, reasoning, vanilla RAG, and a few state of the art RAG based methods)  over multiple data sets including a data set designed by the authors which includes more nuanced QA pairs with coherent reasoning trajectories. Experimental results show superior performance for the proposed method over the alternatives in the studied domains. Authors conduct ablation studies to measure the contributions of various components of their proposed method. Results suggest that both information gain and efficiency penalty contribute positively to the overall performance. It is noted that for complex information seeking tasks that benefit from more reasoning steps, dropping the penalty term improves performance.", "questions": "- The paper does not provide details on how information gain reward. It is clear that information gain is measured as cumulative coverage of the retrieved patches against the expected knowledge set. It would be useful if the authors provided more explanation, perhaps even a concrete example, to further clarify this critical component in the reward function.\n\n- For SFT, authors mention that they construct the dataset by \"providing gathered human web browsing trajectories and corresponding QA pairs\". Can authors explain more what they mean by web browsing trajectories? If it means prompts used to search the web it makes sense, however, if it consists of web URLs I am not clear how they are being used in SFT.", "limitations": "Authors have addressed/acknowledged the following limitations in the appendix section of their work:\n\n- Computational costs lead to utilizing relatively smaller models, which may negatively impact performance.\n- Not being able to report and compare performance with more recent alternative approaches that are still in progress.\n- Not being able to evaluate the performance of the proposed method for tasks beyond simple QA with short answers and exact match evaluation.\n\nThe following are limitations that are not adequately discussed by authors:\n- Computational cost compared with other alternative methods as opposed to just a barrier to explore bigger models for empirical studies.\n- Lower debuggability due to using a more complex optimization scheme.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- The intuition behind using information foraging for RAG is solid and well justified. \n- The main components of the proposed method are clearly explained and overall the paper is easy to follow.\n- Experimental results cover a good set of alternative methods and data sets. The dataset synthesized by the authors offers additional value by providing Q&As with their corresponding reasoning trajectories and covering more complex tasks. Additionally, the authors provide a reasonable amount of rationale for the superior or lower performance observed in their main and ablation studies.\n\n**Weaknesses**\n\n- Authors could have covered and discussed further details about limitations of the proposed method and complexities of using it (refer to the limitation section).", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and constructive review. We appreciate your recognition of the strengths of our work and your insightful questions and suggestions. Below, we provide detailed responses to each of your comments and outline the revisions we will make to address your feedback.\n\n### 1.  Details and Example for Information Gain Reward\n\n**Reviewer:**  \n*It is clear that information gain is measured as cumulative coverage of the retrieved patches against the expected knowledge set. It would be useful if the authors provided more explanation, perhaps even a concrete example, to further clarify this critical component in the reward function.*\n\n**Response:**  \nThank you for highlighting the need for a more concrete explanation of the information gain reward. We are happy to clarify both the computation and provide an illustrative example.\n\nThe information gain reward measures how much of the expected knowledge set $\\mathcal{D}^*$ has been cumulatively retrieved by the agent up to each reasoning step. At each step \\($t$\\), we evaluate the coverage as:  \n\n$C \\left( \\bigcup_{\\tau=1}^t \\mathcal{K}_\\tau,\\, \\mathcal{D}^* \\right)$\n\nwhere \\($\\mathcal{K}_\\tau$\\) is the set of evidence (e.g., retrieved web pages) obtained at step \\($\\tau$\\). The information gain reward is the maximum coverage achieved across all steps:  \n\n$R_\\text{gain} = \\max_{t=1,…,T} C \\left( \\bigcup_{\\tau=1}^t \\mathcal{K}_\\tau,\\, \\mathcal{D}^* \\right)$\n\n**Example:**  \nSuppose the query is:  \n“Do I need a visa for this year's NeurIPS conference?”\n\nThe expected knowledge set ($\\mathcal{D}^*$) includes:  \n(1) The official NeurIPS conference website’s visa information page, and  \n(2) The official government immigration website for the host country (providing general visa policy and requirements).\n\n- **Step 1:** The agent retrieves the NeurIPS official website visa information page. Coverage is 0.5 (1 out of 2 gold sources found).\n- **Step 2:** The agent retrieves the host country's government immigration website detailing visa policies. Coverage becomes 1.0 (both gold sources found).\n\nAt each step, coverage is calculated as the proportion of gold sources retrieved so far, and the information gain reward is the maximum value achieved during the search process. This encourages the agent to efficiently collect all necessary evidence through stepwise search.\n\nWe will add this concrete example to the revised manuscript for clarity.\n\n---\n\n### 2. Clarification of Web Browsing Trajectories in SFT\n\n**Reviewer:**  \n*Can authors explain more what they mean by web browsing trajectories? If it means prompts used to search the web it makes sense, however, if it consists of web URLs I am not clear how they are being used in SFT.*\n\n**Response:**  \n\nThank you for your question. In Section 3.2, we define “web browsing trajectories” as the full step-by-step process by which a human or agent searches for and gathers information from the web to answer complex questions. Each trajectory includes the sequence of search queries, the URLs retrieved and visited, and the key facts or claims extracted from these web pages—all represented in structured JSON format. For example:\n\n```\n[\n  {\n    \"query\": \"NeurIPS 2025\",\n    \"id\": 0,\n    \"page_content\": \"...\",\n    \"url\": \"...\",\n    \"extracted_claim\": \"NeurIPS 2025 will be held in ...\"\n  },\n  ...\n]\n```\nNext, we use strong LLMs to convert these JSON-format web browsing trajectories into plain text following the agentic reasoning paradigm (defined in Eq. 4), where the model alternates between internal reasoning, issuing help requests, and integrating retrieved web evidence at each step.\nNote that **web URLs themselves are not directly used for SFT supervision.**\n\nDuring RL training, URLs are included as concrete evidence for intermediate reasoning steps, supporting the computation of the information gain reward and the supervision of retrieval behaviors.\n\n### 3. Limitation: Computational Cost and Debuggability\n\n**Reviewer:**  \n*Computational cost compared with other alternative methods as opposed to just a barrier to explore bigger models for empirical studies. Lower debuggability due to using a more complex optimization scheme.*\n\n**Response:**  \nThank you for pointing out these important limitations.\n\n- **Computational Cost:**  \n  Our framework introduces additional computational overhead compared to some traditional approaches, not only as a barrier to scaling up model size, but also as an intrinsic cost of the more complex optimization and reasoning procedures required.\n\n- **Debuggability:**  \n  IIncorporating tool calling into the reasoning process increases the complexity of debugging, as it can be challenging to pinpoint which specific step led to an error. However, compared to models that only perform end-to-end answer prediction, our approach provides richer intermediate records—such as the expected evidence set—which help illuminate the reasoning process. These intermediate signals not only allow us to better analyze the model’s behavior but also offer improved opportunities for debugging and error analysis compared to standard end-to-end approaches.\n\nThank you for raising these points. We will explicitly discuss these limitations and their practical implications in the revised manuscript."}, {"review_id": "gQLaYqhAlo", "reviewer": "Reviewer_LJkD", "summary": "The paper presents InForage, a reinforcement learning approach that enables LLMs to perform dynamic, multi-step information retrieval during reasoning, rather than relying on static pre-inference retrieval. The approach uses three types of rewards to train models: outcome rewards for correct final answers, information gain rewards for valuable intermediate retrieval steps, and efficiency penalties to discourage unnecessarily long reasoning chains. The authors construct a new training dataset that captures human information-seeking trajectories through iterative web browsing. The method outperforms baselines in a newly developed real-time web QA dataset.", "questions": "- The newly constructed training data and evaluation data would be useful for the community.  Do the authors plan to release them?\n- Table 2's case study is helpful. Could the authors provide a bit more qualitative examples?  Did the authors find any failure modes?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- The method is interesting and reasonable. The formulation of combining three types of rewards: Outcome Reward, Information Gain Reward, Efficiency Penalty seems effective.\n- The newly constructed training data and evaluation data seem to be high quality and would be useful for the community if released\n\nWeakness\n- No major weakness. Refer to the questions below.", "was_revised": true, "final_justification": "Issues resolved", "rebuttal": "Thank you for your positive feedback and thoughtful questions. We appreciate your recognition of the quality and usefulness of our method and dataset. We address each point below:\n\n- ---\n\n### 1. Data Release\n\n**Reviewer:**  \n  *The newly constructed training data and evaluation data would be useful for the community. Do the authors plan to release them?*\n\n**Response:**  \n  Yes, we will release both the training and evaluation data, and will continue to improve them for broader community benefit.\n\n- ---\n\n### 2. Qualitative Examples and Failure Modes\n\n**Reviewer:**  \n  *Table 2's case study is helpful. Could the authors provide a bit more qualitative examples? Did the authors find any failure modes?*\n\n**Response:**  \n  Thank you for your suggestion. We will add more qualitative examples to the revised version to illustrate both successful and failure cases in detail. A representative failuire question is:\n\n- > **Question:** If we assume all articles published by Nature in 2020 (articles only, not book reviews/columns, etc) relied on statistical significance to justify their findings and they on average came to a p-value of 0.04, how many papers would be incorrect as to their claims of statistical significance? Round the value up to the next integer.  **(correct answer: 41)**\n\nThe reasoning trajectory of Inforage-7B and Gemini2.5 pro shows here:\n\n| InForage-7B:                                                 |\n| ------------------------------------------------------------ |\n| \\<think\\> Okay, let me tackle this question step by step. The user wants to know how many Nature articles from 2020 that used a p-value of 0.04 might be incorrect in their claims of statistical significance.\\</think\\> |\n| \\<search\\> I need to recall what a p-value means.   \\</search\\> |\n| \\<info\\>Typically, a p-value below 0.05 is considered statistically significant \\</info\\> |\n| \\<think\\> Wait, the question doesn't give the total number of articles published by Nature in 2020. Oh, right! I need to find out how many articles Nature published in 2020. Because without that number, I can't compute the actual count.\\</think\\> |\n| \\<search\\> Nature Articles 2020   \\</search\\>                |\n| \\<info\\> the number of articles published by Nature 2020 is 1002\\</info\\> |\n| \\<think\\> Previous search results show that using a p-value threshold of 0.05 corresponds to a Type I error rate, meaning that 5% of the articles would be expected to be false positives. Given a total of 1002 articles, 5% of 1002 is 50.1, which we round up to 51.\\</think\\> |\n| \\<answer\\> 51 \\</answer\\>                                    |\n|                                                              |\n| **Gemini2.5 Pro:**                                           |\n| **Reasoning Process:** Defining the Publication Set -> Refining the Query ->Revising Search Terms .... |\n| I was unable to find the exact number of Nature research articles published in 2020 despite extensive searching. Based on a reliable Clarivate blog post reporting about 800 articles in 2019, I have estimated the 2020 figure to be approximately 831, and have clearly indicated this as an informed estimation in my work. |\n| **Calculate the number of incorrect claims**: By multiplying the total number of articles by the p-value, we can estimate the number of papers with statistically insignificant findings. 831*0.04=33.24, round to 34 |\n| **Answer**: 34                                |\n||\n\nBoth methods failed on this example, but for different reasons.\n\nFor InForage, the model ignored the specific instruction in the question to use a p-value of 0.04, and instead searched for the typical p-value threshold for significance testing (i.e., 0.05). As a result, although it correctly retrieved the number of Nature papers published, it applied the wrong statistical threshold, leading to an incorrect final answer.\n\nGemini 2.5-Pro, on the other hand, failed to retrieve the exact number of Nature research articles published in 2020 after several search attempts. It resorted to estimating this number based on secondary sources, and while it followed the correct calculation process, the final answer was still incorrect due to the inaccurate input.\n\nThis case highlights a specific failure mode of InForage: improper task planning. In this scenario, the model should have first identified the correct number of Nature articles and then directly performed the computation as instructed. Its additional, unnecessary planning—such as searching for a typical p-value—introduced extra bias and ultimately led to the wrong answer.\n\nWe will include this failure example in the revised manuscript to illustrate current limitations and inform future improvements."}]

Author rebuttal:

Meta-review: The paper proposes InForage, a reinforcement learning framework for search-enhanced reasoning grounded in Information Foraging Theory. It rewards intermediate retrieval quality (information gain/coverage) alongside final answer accuracy and an efficiency penalty, trained with human-guided web-browsing trajectories. Across general QA, multi-hop, and a new real-time web QA setting, InForage outperforms baselines.  

### Strengths of the paper: 
- The reviewers (XiLb) highlight the novelty of the information-gain reward and the paper’s timeliness for agentic, search-based RL; the proposed method made clear improvements over Search-R1 and it conducted extensive ablations dissecting SFT/RL/rewards. 
- The work is technically sound with a well-designed three-component reward, thorough methodology (ablations/statistics), clear IFT motivation, and meaningful significance. 
- The reviewers find the IFT grounding solid and well-justified, presentation clear, and experiments comprehensive across methods/datasets. 

### Weaknesses of the paper: 
- The reviewer sR8g requests larger-model scaling evidence (13B+) and additional baselines (e.g., FLARE, advanced IRCoT). The reviewer notes inconsistent benefits from the efficiency penalty and missing precision on key definitions (coverage C, “information scent”) and also worries about reliance on the self-constructed dataset. 
- The reviewer XiLb wants datasets with gold retrieval labels, observes SFT is particularly critical, and suggests direct comparisons to industry deep-research systems. 
- The reviewer ipjm asks for more detail on the information-gain reward (ideally a worked example) and a fuller discussion of limitations/complexities, including compute costs and debuggability. 

### Primary reasons for Accept (Spotlight)
The primary reasons for recommending Accept (Spotlight) are that the paper makes a principled and timely contribution by operationalizing Information Foraging Theory into a tractable reinforcement learning framework for search-enhanced reasoning, introducing a mid-trajectory information-gain reward that is both novel and well-justified. Empirically, the method demonstrates consistent and robust improvements across multiple QA benchmarks and model scales, including outperforming strong baselines such as FLARE and showing competitive results against closed-source deep-research APIs. The authors further strengthened the submission during rebuttal by adding larger-model results (14B), expanded baseline comparisons, and clearer formalizations of the information-gain and coverage metrics. 

### Summary of the discussion and rebuttal
The authors provided detailed and constructive responses to the reviewers’ concerns. For R-XiLb, they clarified the differences between PPO and GRPO, explained rollout settings, and showed that reinforcement learning leads to deeper and more effective search trajectories beyond SFT alone. For R-sR8g, the authors expanded their experiments by adding FLARE baselines at 3B/7B scales, reporting 14B model results, and clarifying the definitions of coverage and information scent, which addressed the reviewer’s main concerns and led to an upgraded score. For R-ipjm, they provided a worked example of the information-gain reward and clarified the nature of their browsing trajectories, while acknowledging limitations such as computational cost and dataset reliance. Overall, the rebuttal successfully addressed key issues raised by the reviewers, strengthening the paper’s clarity, empirical support, and theoretical framing.
Decision: Accept (spotlight)