WebSeer: Training Deeper Search Agents through Reinforcement Learning with Self-Reflection

## Abstract

Abstract Search agents have achieved significant advancements in enabling intelligent information retrieval and decision-making within interactive environments.
Although reinforcement learning has been employed to train agentic models capable of more dynamic interactive retrieval, existing methods are limited by shallow tool-use depth and the accumulation of errors over multiple iterative interactions.
In this paper, we present WebSeer, a more intelligent search agent trained via reinforcement learning enhanced with a self-reflection mechanism. Specifically, we construct a large dataset annotated with reflection patterns and design a two-stage training framework that unifies cold start and reinforcement learning within the self-reflection paradigm for real-world web-based environments, which enables the model to generate longer and more reflective tool-use trajectories.
Our approach substantially extends tool-use chains and improves answer accuracy. Using a single 14B model, we achieve state-of-the-art results on HotpotQA and SimpleQA, with accuracies of 72.3% and 90.0%, respectively, and demonstrate strong generalization to out-of-distribution datasets. The code is available at https://github.com/99hgz/WebSeer .

## 1 Introduction

Large language models (LLMs) have demonstrated remarkable performance across a wide range of natural language processing tasks, including question answering, summarization, and dialogue generation (Hendrycks et al., 2021; Rein et al., 2024).
However, relying solely on the parametric knowledge of language models poses fundamental limitations: it is static, often outdated, and prone to hallucinations (Sardana, 2025). To overcome these challenges, retrieval-augmented generation (RAG) (Lewis et al., 2020) approaches have been developed to enable models to access and retrieve external documents dynamically.

With the continuous advancement of model capabilities, agentic RAG (Trivedi et al., 2023; Li et al., 2025; Jin et al., 2025) has emerged as a powerful paradigm. This design empowers models to follow more complex reasoning trajectories. Unlike traditional RAG systems (Asai et al., 2023; Trivedi et al., 2023; Yu et al., 2024), agentic RAG can freely browse vast knowledge sources available on the internet and leverage tools such as code execution to extend their skills, enabling them to tackle a more diverse range of tasks.
While agentic RAG greatly extends the scope of tool use and demonstrates strong potential, existing approaches exhibits several notable limitations in practice. In particular, when faced with complex or open-domain tasks, current systems often struggle to maintain coherent reasoning chains and robust retrieval. Errors introduced at intermediate steps can easily accumulate, while the lack of effective coordination across components makes it difficult to achieve reliable end-to-end performance. Thus, despite its promise, agentic RAG still confronts a set of fundamental challenges that must be addressed.

Figure: Figure 1: Comparasion between different search agent: Search-r1(Jin et al., 2025) faces shallow search and error buildup, while WebSeer trained using our self-reflection paradigm significantly improve tool calls and maintain rigorous reasoning.
Refer to caption: x1.png

1) Insufficient Search Calls.
The most common issue when it comes to model invocation tools to solve problems is Insufficient API Calls (Kokane et al., 2025), which is also often seen in the RAG scenario, as shown in the Appendix [B](https://arxiv.org/html/2510.18798v1#A2). This may be due to models being biased toward synthesizing the currently available information into a plausible answer rather than actively seeking new or complementary knowledge. Existing work typically exhibits short tool-use chains (Jin et al., 2025; Song et al., 2025), suggesting that models may fall into suboptimal behaviors—prematurely producing an answer instead of verifying its correctness or exploring further evidence.

2) Lack of Spontaneous Self-Reflection Mechanisms.
Current search agents lack spontaneous reflection steps in RAG scenario: models neither actively cross-verify information nor autonomously rewrite queries or backtrack retrievals when uncertain. As a result, if initial retrieval is incomplete, the generation phase expands answers based on flawed or partial context, amplifying early errors.

3) Neglect of Real-World Web Scenarios.
Most existing work focuses on retrieval from local vector databases, with limited attention to more complex and open-ended web agent scenarios.

In this paper, we introduce Webseer, a novel search agent designed to tackle complex real-world multi-hop question answering tasks.
Different from prior approaches (Jin et al., 2025; Zheng et al., 2025), WebSeer explicitly encourages deeper exploration and integrates a build-in self-reflection mechanism, enabling the model to backtrack, reformulate queries, and iteratively refine its reasoning process.
As shown in Figure [1](https://arxiv.org/html/2510.18798v1#S1.F1), previous agentic RAG primarily rely on short tool-use chains and often terminate once a superficially plausible answer is formed. In contrast, WebSeer actively prolongs the search trajectory and incorporates reflection steps, and revises its queries when uncertainty is detected. This design enables Webseer to gather more comprehensive evidence, mitigate the accumulation of errors, and improve robustness in open-domian multi-hop reasoning. Specifically, we design a two-stage training framework that unifies cold start and reinforcement learning within the self-reflection paradigm. A central component of this framework is Self-Reflective Reinforcement Learning (SRRL), which leverages answer correctness signals during multi-turn interactions to more effectively encourage reflective behavior. To ground the model in realistic web scenarios, we equip it with three complementary tools: a web search API for external knowledge acquisition, a webpage reader for lightweight comprehension of web content, and a code executor for precise computation.

To support this framework, we construct a high-quality dataset of long-horizon reasoning trajectories through rejection sampling. These trajectories contain multiple rounds of answer refinement and substantially longer tool-use chains compared to conventional dialogue datasets. This training framework significantly increases the average length of tool invocation chains while maintaining rigorous reasoning quality. Compared to prior work (Zheng et al., 2025), all decisions and tool interactions are handled by a single model, eliminating the need for auxiliary agent controllers or stronger backbone models.

Overall, our contributions are three-fold:

- •
We propose the first unified two-stage training framework with a self-reflection paradigm, enabling a more intelligent search agent that improves search depth, breadth, and accuracy.
- •
We propose a novel SFT data synthesis method that encourages proactive tool invocation, resulting in significantly longer and more complex tool-use chains;
- •
Through extensive experiments, we demonstrate notable improvements in answer accuracy, achieving SOTA on HotpotQA with a 72.3% accuracy and on simpleQA with a 90.0% accuracy. Our proposed training framework shows explicit self-verification behaviors and strong generalization to OOD datasets.

## 2 WebSeer

We introduce WebSeer, a search agent for multi-hop question answering in real-world web settings. The model is trained with our proposed unified two-stage framework based on self-reflection as shown in Fig [2](https://arxiv.org/html/2510.18798v1#S2.F2). To obtain diverse high-quality data, we propose multi-turn rejection sampling for collecting positive trajectories used in supervised fine-tuning, enabling the model to learn reflective reasoning patterns. Building on this, we develop Self-Reflective Reinforcement Learning (SRRL), which incorporates feedback and preserves reflective context during RL, guided by effective reward design to achieve robust and optimal performance.

Figure: Figure 2: Overview of our two-stage training framework. In the first stage, we construct self-reflective, tool-augmented reasoning trajectories. In the second stage, we apply SRRL, allowing the model to iteratively refine and resubmit answers with F1-based feedback.
Refer to caption: x2.png

### 2.1 Task Formulation

For a given problem, our objective is to construct a reasoning chain augmented by tool invocations. Specifically, the reasoning chain consists of $N$ sequential steps, each comprising a model-generated output, a tool invocation, and the resulting observation. At each step, the model initially generates reasoning outputs based on the current context, then decides to invoke one or more external tools. The tool invocation parameters are extracted by the decoder, and subsequently, the invoked tools are executed, returning observations integrated into the reasoning context. The reasoning chain terminates if the model either abstains from further tool invocations or invokes a special submit-answer tool, signaling completion. Additionally, a predefined maximum step limit $T_{\text{max}}$ is enforced to prevent indefinite chaining, thus constraining the length of the reasoning trajectory.

### 2.2 Tool Design

In this work, we design three specialized tools to support external knowledge acquisition and reasoning execution. Additionally, we treat the Answer Submission step itself as a tool, ensuring that the model explicitly decides when to terminate reasoning and output the final answer.

Search Engine
This tool receives a set of keywords and performs a Google search. It returns the top page of search results in a structured format, including the title, URL, and snippet for each result.

Webpage Reader
Due to the prohibitive input length of raw HTML, we design the webpage reader to act as a lightweight, model-mediated summarization interface. Given a specific URL and a question, the system fetches the HTML content of the page and prompts the same language model to answer the question based on the page content. The tool then returns the model-generated answer.

Code Executor
This tool accepts a Python code snippet, executes the code in a controlled environment, and returns the standard output as the tool result.

### 2.3 Self-reflective Reasoning Chain Construction

Deepseek R1 (DeepSeek-AI et al., 2025) has highlighted the importance of cold-start in reinforcement learning training. Lee et al. (2025) provided several trajectories for training models to solve multi-hop QA problems. However, these trajectories only included cases where the correct answer was successfully found, without teaching the model how to handle situations in which an incorrect answer might arise. To address this limitation, we propose a multi-turn rejection sampling method to collect reasoning paths that incorporate reflective patterns. To overcome this limitation, we propose a multi-turn rejection sampling method that collects reasoning paths enriched with reflective patterns. To encourage trajectory diversity, we impose no restrictions on the form of reflection itself, retaining only those reasoning paths that ultimately converge to the correct solution.

Let $\mathcal{D}={(x_{i},y_{i}^{\ast})}_{i=1}^{N}$ denote a multi-hop QA dataset with ground-truth answers $y_{i}^{\ast}$. We consider two models: (i) a *reasoner* $G$, which, given an instance and its interaction history, generates a tool-augmented reasoning path until producing a final answer; and (ii) an independent *verifier* $V$, which assesses the factual correctness of a proposed answer by invoking the same tools and ultimately returning a judgment.
At reflection step $t\in\{1,\dots,n_{\max}\}$ for instance $x_{i}$, the reasoner receives the concatenated history
$H_{t-1}=\{P_{1},R_{1},\dots,P_{t-1},R_{t-1}\},$
and generates a tool-augmented path $P_{t}$ that culminates in an answer proposal
$\hat{y}_{i}^{(t)}\;\leftarrow\;G(x_{i},H_{t-1}).$
To evaluate this proposal, we query the verifier up to a budget of $K$, yielding a verification outcome $R_{t}\sim V(\,\cdot\mid x_{i},P_{t},\hat{y}_{i}^{(t)}\,)$.

Each $R_{t}$ consists of (i) a binary judgment $J_{t}\in\{\textsc{correct},\textsc{incorrect}\}$ regarding $\hat{y}_{i}^{(t)}$, and (ii) a tool-augmented path that may be appended to the full path.

Validity predicate.
We define a predicate
$\Psi\!\left(R_{t},\hat{y}_{i}^{(t)},y_{i}^{\ast}\right)\in\{0,1\}$
that evaluates whether the judgment in $R_{t}$ *matches factual correctness*, i.e.,

$$ $\Psi\!\left(R_{t},\hat{y}_{i}^{(t)},y_{i}^{\ast}\right)=\begin{cases}1,&\text{if }\bigl(J_{t}=\textsc{correct}\ \wedge\ \hat{y}_{i}^{(t)}=y_{i}^{\ast}\bigr)\ \text{ or }\bigl(J_{t}^{=}\textsc{incorrect}\ \wedge\ \hat{y}_{i}^{(t)}\neq y_{i}^{\ast}\bigr),\[4.0pt] 0,&\text{otherwise.}\end{cases}$ $$

So, if there exists
$\Psi=1$, we *accept* the verifier’s feedback and update the history via concatenation,
$H_{t}\;\leftarrow\;H_{t-1}\cup\{P_{t},R_{t}\}.$
Otherwise, we re-query the verifier to sample the next outcome

$$ $R_{t}^{(m)}\sim V(\,\cdot\mid x_{i},P_{t},\hat{y}_{i}^{(t)}\,),\quad m=1,2,\dots,K,$ $$

and evaluate $\Psi\!\left(R_{t}^{(m)},\hat{y}_{i}^{(t)},y_{i}^{\ast}\right)$.

If all $K$ attempts fail, then the instance $x_{i}$ is discarded, and we directly proceed to the next problem instance $x_{i+1}$.

Iteration and termination.
The above process iterates for $t=1,2,\dots$ with the updated history until one of the following conditions holds:

$$ $\textbf{(Success)}\quad\hat{y}_{i}^{(t)}=y_{i}^{\ast}\ \text{and}\ J_{t}=\textsc{correct},$ $$

in which case we halt and record the finalized, verified trajectory
$\mathcal{T}_{i}\;=\;\{P_{1},R_{1},P_{2},R_{2},\dots,P_{t},R_{t}\},$

$$ $\textbf{(Budget stop)}\quad t=n_{\max},$ $$

in which case the instance is not recorded as a successful trajectory.

Supervised fine-tuning.
Let $\{\mathcal{T}_{i}\}_{i=1}^{N}$ denote the all set of successful trajectories, where each trajectory $\mathcal{T}_{i}$ is represented as a token sequence
$\mathcal{T}_{i}=\{y^{(i)}_{1},y^{(i)}_{2},\dots,y^{(i)}_{T_{i}}\}.$
To stabilize subsequent reinforcement learning and mitigate degenerate exploration, we perform supervised fine-tuning (SFT) of the model parameters $\theta$ on the dataset $\{(x_{i},\mathcal{T}_{i})\}_{i=1}^{N}$. Following empirical findings on iterative search training (Zhang et al., 2025), we adopt a masked autoregressive negative log-likelihood (NLL) objective that excludes external observation tokens from the loss.

Let $\mathcal{O}\subset\mathcal{T}$ denote the subsequence of tokens in $\mathcal{T}$ corresponding to tool observations. The masked training objective is defined as

$$ $\mathcal{L}(x,\mathcal{T};\theta)\;=\;-\frac{\sum_{t=1}^{T}\mathbb{I}[\,y_{t}\notin\mathcal{O}\,]\cdot\log p_{\theta}\!\left(y_{t}\mid x,y_{<t}\right)}{\sum_{t=1}^{T}\mathbb{I}[\,y_{t}\notin\mathcal{O}\,]},$ $$

where $\mathbb{I}[\cdot]$ denotes the indicator function.

This masking restricts the loss to the agent’s own outputs—such as internal reasoning steps and tool-calling decisions—while excluding literal tool observations. In doing so, the objective encourages the model to faithfully reproduce the supervised reasoning process (e.g., when to retrieve and how to compose intermediate steps) while ignoring raw tool outputs, a practice shown to improve both performance and robustness.

### 2.4 Self-Reflective Reinforcement Learning (SRRL)

Unlike other previous training frameworks, our reinforcement learning framework unifies SFT and RL under the self-reflection mechanism, which we call Self-Reflective Reinforcement Learning (SRRL). Specifically, SRRL as allows the model to submit answers multiple times within a single dialogue turn. This design enables the model to iteratively refine its reasoning based on external feedback, leading to more stable and effective exploration.

Formally, given an input query $x$ and the ground-truth answer $y^{\ast}$, the LLM interacts with external tools to produce an evolving trajectory

$$ $\mathcal{T}=\{(a_{1},o_{1}),(a_{2},o_{2}),\dots,(a_{T},o_{T})\},$ $$

where at step $t$, $a_{t}$ denotes the agent’s action (e.g., a tool call or an answer_submit), and $o_{t}$ denotes the resulting observation.
In particular, when the action corresponds to the special tool answer_submit, the submitted answer $\hat{y}^{(t)}$ is compared with $y^{\ast}$ and return

$$ $r^{(t)}=\mathrm{F}_{1}\!\big(\hat{y}^{(t)},\,y^{\ast}\big)\in[0,1].$ $$

The scalar feedback $r^{(t)}$ is returned *as text* and appended to the dialogue context.
If $r^{(t)}$ is below a predefined threshold, the environment allows the model to continue reasoning, enabling the model to revise its reasoning and potentially submit an improved answer at a later step.

We employ the Group Relative Policy Optimization (GRPO) algorithm (Shao et al., 2024) to optimize the policy $\pi_{\theta}$ parameterized by the model. In addition, following DAPO (Yu et al., 2025), we incorporate the *clip-higher* mechanism to better accommodate asymmetric update ranges. Concretely, given a batch of queries $(q,a)\sim\mathcal{D}$ and $T$ sampled reasoning trajectories $\{o_{i}\}_{i=1}^{T}$ from the old policy $\pi_{\theta_{\text{old}}}$, the DAPO objective is defined as

$$ $\displaystyle\mathcal{J}_{\text{DAPO}}(\theta)=\mathbb{E}_{(q,a)\sim\mathcal{D},\,\{o_{i}\}\sim\pi_{\theta_{\text{old}}}(\cdot|q)}\Bigg[\frac{\sum_{i=1}^{G}\sum_{t=1}^{|o_{i}|}\min\!\Big(r_{i,t}(\theta)\hat{A}_{i,t},\,\text{clip}\!\big(r_{i,t}(\theta),1-\epsilon_{\text{low}},1+\epsilon_{\text{high}}\big)\hat{A}_{i,t}\Big)}{\sum_{i=1}^{G}|o_{i}|}\Bigg],$ (1) $$

where $r_{i,t}(\theta)$ denotes the importance sampling ratio between the new and old policies at timestep $t$, and $\hat{A}_{i,t}$ is the estimated advantage. The asymmetric clipping parameters $\epsilon_{\text{low}}$ and $\epsilon_{\text{high}}$ control the update range, preventing overfitting to noisy advantages.

### 2.5 Reward Design

Because in addition to the correctness of the answer, we also care about how many times the agent has tried, we define a trajectory-wise reward. Let a trajectory be denoted by
$\tau=\{(o_{t},a_{t},r_{t})\}_{t=1}^{N}$, where $o_{t}$ is the observation (context), $a_{t}$ is the model output
including potential tool invocations, and $r_{t}$ is the instantaneous reward. The total trajectory-wise reward
is then given by

$$ $R(\tau)=R_{\text{format}}(\tau)+R_{\text{correct}}(\tau).$ (2) $$

For format, let $\lvert y\rvert$ be the output length, $L_{\text{expect}}$ the safe-zone threshold, and $L_{\max}$ the hard limit. The reward is

$$ $R_{\text{format}}(\tau)=\begin{cases}0,&\lvert y\rvert\leq L_{\text{expect}},\[6.0pt] -\dfrac{\lvert y\rvert-L_{\text{expect}}}{L_{\max}-L_{\text{expect}}},&L_{\text{expect}}<\lvert y\rvert\leq L_{\max},\[12.0pt] -1,&\lvert y\rvert>L_{\max}.\end{cases}$ (3) $$

Thus, outputs in the safe zone incur no penalty, those in the transition region are linearly penalized, and overly long ones receive the maximum penalty.

For correctness, let $r\in[0,1]$ be a task-specific score (e.g., F1) and $T$ the number of submission attempts. To discourage resubmissions, we apply an exponential discount $\alpha\in(0,1]$:

$$ $R_{\text{correct}}(\tau)=r\cdot\alpha^{T}.$ (4) $$

## 3 Experiments

### 3.1 Experiment Setup

#### Datasets.

We evaluate our model on a diverse suite of open-domain QA benchmarks, spanning both in-domain and out-of-domain settings. Following Zheng et al. (2025), we adopt the same evaluation split: 512 examples sampled from the development sets of NQ (Kwiatkowski et al., 2019), TQ (Joshi et al., 2017), HotpotQA (Yang et al., 2018), 2WikiMultiHopQA (Ho et al., 2020), MuSiQue (Trivedi et al., 2022), PopQA (Mallen et al., 2023), FanoutQA (Zhu et al., 2024), FRAMES (El Asri et al., 2017), and SimpleQA (Wei et al., 2024), along with 125 examples from Bamboogle.

Because valid answers in open-domain QA often admit multiple surface forms, rule-based string-matching metrics can lead to inaccurate performance estimates. To address this, we adopt LLM-as-a-Judge, following the methodology and prompt template of Zheng et al. (2025). During evaluation, all models are restricted to submitting a single answer. The full evaluation prompt and implementation details are provided in Appendix [A](https://arxiv.org/html/2510.18798v1#A1).

#### Baselines.

We compare our approach against several strong baselines that represent different paradigms for reasoning and retrieval in open-domain QA:
(1) Closed-book (CoT): The model answers questions using only its internal parametric knowledge, without any external retrieval, following a chain-of-thought prompting strategy. We choose Qwen2.5-7B-Instruct (Qwen et al., 2025) as the base model.
(2) Local RAG: The model is allowed to access a local vector-based retrieval system, where the knowledge source consists of the English Wikipedia page dump dated March 1, 2022. We test Qwen2.5-7B-Instruct (Qwen et al., 2025), Search-r1 (Jin et al., 2025), and R1-Searcher (Song et al., 2025) in this setting.
(3) Web Agents: The model is equipped with web-based tool access, including search engine querying and webpage parsing through Markdown conversion. We test DeepResearcher (Zheng et al., 2025) in this setting.

#### Implementation Details

During inference, we use the Google Web Search API for real-time retrieval and the Jina API to bypass anti-crawling and extract clean, LLM-friendly text.

For training, we prioritize cost, stability, and consistency by restricting retrieval to Wikipedia via the Google Site Search API and fetching full pages through the official Wikipedia API. Training uses the verl framework (Sheng et al., 2025), sampling 12 prompts per step, each with 8 candidate trajectories and up to 30 interaction turns. This controlled setup ensures stable, noise-reduced signals while still exposing the model to realistic retrieval and comprehension tasks. We trained a total of 100 steps, spending 60 A800 GPU hours.

**Table 1: Main results on seven multi-hop question answering (MHQA) benchmarks. All the results labelled with ^† are taken from (Zheng et al., 2025).**
| Method | Inference | In Domain | Out of Domain |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Environment | NQ | TQ | Hotpot | 2Wiki | Avg | Musique | Bamb | PopQA | Avg |  |
| CoT^† | - | 32.0 | 48.2 | 27.9 | 27.3 | 33.9 | 7.4 | 21.6 | 15.0 | 14.7 |
| CoT+RAG^† | Local RAG | 59.6 | 75.8 | 43.8 | 24.8 | 51.0 | 10.0 | 27.2 | 48.8 | 28.7 |
| Search-o1^† | Web Search | 55.1 | 69.5 | 42.4 | 37.7 | 51.2 | 19.7 | 53.6 | 43.4 | 38.9 |
| 7B/8B Models |  |  |  |  |  |  |  |  |  |  |
| Qwen3-8B w/ Tools | Local RAG | 67.0 | 76.4 | 50.8 | 33.0 | 56.8 | 18.4 | 43.2 | 44.0 | 35.2 |
| Search-r1-base^† | Local RAG | 60.0 | 76.2 | 63.0 | 47.9 | 61.8 | 27.5 | 57.6 | 47.0 | 44.0 |
| Search-r1-instruct^† | Local RAG | 49.6 | 49.2 | 52.5 | 48.8 | 50.0 | 28.3 | 47.2 | 44.5 | 49.5 |
| R1-Searcher^† | Web Search | 52.3 | 79.1 | 53.1 | 65.8 | 62.6 | 25.6 | 65.6 | 43.4 | 44.9 |
| DeepResearcher^† | Web Search | 61.9 | 85.0 | 64.3 | 66.6 | 69.5 | 29.3 | 72.8 | 52.7 | 51.6 |
| 14B Models |  |  |  |  |  |  |  |  |  |  |
| Qwen2.5-14B w/ Tools | Local RAG | 72.1 | 83.8 | 62.9 | 70.9 | 72.4 | 29.7 | 72.0 | 46.1 | 49.3 |
| Qwen2.5-14B w/ Tools | Web Search | 72.5 | 87.9 | 67.9 | 80.3 | 77.2 | 26.6 | 73.6 | 54.7 | 51.6 |
| Qwen3-14B w/ Tools | Local RAG | 73.1 | 80.9 | 54.9 | 52.5 | 65.4 | 22.7 | 63.2 | 46.7 | 44.2 |
| Search-r1 | Local RAG | 66.9 | 82.6 | 69.8 | 57.0 | 69.1 | 36.9 | 64.8 | 56.3 | 52.7 |
| WebSeer | Local RAG | 81.9 | 86.7 | 70.9 | 76.0 | 78.9 | 35.0 | 81.6 | 60.6 | 59.1 |
| WebSeer | Web Search | 82.8 | 91.0 | 72.3 | 84.2 | 82.6 | 35.2 | 80.0 | 58.0 | 57.7 |

### 3.2 Main Results

Figure: Figure 3: Evaluation on three harder benchmarks. Qwen2.5-14B and WebSeer use web search engine, while Search-r1 relies on local RAG.

Table [1](https://arxiv.org/html/2510.18798v1#S3.T1) reports the performance of our method against baselines on seven multi-hop QA benchmarks. Our approach consistently achieves the best results, substantially outperforming both closed-book and retrieval-augmented baselines. On in-domain tasks, our model reaches an average accuracy of 82.4%, exceeding the previous state-of-the-art method Search-r1 by 12.5 points. The largest gains are observed on NQ and 2WikiMultiHopQA, with improvements of 15.9 and 27.2 points, respectively.

Figure: Figure 4: Accuracy and average tool call times for Qwen2.5 Models on HotpotQA and SimpleQA

Beyond in-domain evaluation, our method also demonstrates strong generalization on out-of-distribution (OOD) datasets, indicating that it does not merely overfit to the retrieval distribution encountered during training. Instead, it learns reasoning patterns and retrieval strategies that transfer effectively to unseen question types, domains, and web sources. In this regime, WebSeer benefits from local RAG: on Bamboogle, it achieves 81.6%, a substantial 12.8-point improvement over the prior best, while on PopQA it reaches 60.6%. These results highlight the effectiveness of our reinforcement learning framework and tool-augmented reasoning design in enabling robust cross-domain generalization.

We further evaluate on three challenging benchmarks: FanOutQA, Frames, and SimpleQA (Table [3](https://arxiv.org/html/2510.18798v1#S3.F3)). On FanOutQA, a fully OOD multi-document QA benchmark, our model attains a loose accuracy of 55.4, surpassing all baselines and nearly matching GPT-4o (55.8)(^1^11[https://fanoutqa.com/leaderboard/](https://fanoutqa.com/leaderboard/)). It also achieves 56.1% on Frames and 90.0% on SimpleQA, while the RL-only Search-r1 model performs poorly across these datasets. These results underscore the strong generalization of our approach: despite being trained under site-restricted search, it performs even better when deployed in the open web, demonstrating a robust and transferable retrieval–reasoning policy that adapts to diverse domains.

### 3.3 Quantitative Analysis

#### Model capacity matters for complex tool using.

We find that sufficient model capacity is essential for multi-step reasoning in search agents. As shown in Table [4](https://arxiv.org/html/2510.18798v1#S3.F4), SFT consistently increases tool usage across scales but its effect on accuracy is uneven: the 3B model drops 3.52 points on HotpotQA yet gains 8.06 on SimpleQA, while the 7B model degrades on both. Only the 14B model achieves consistent improvements in both tool usage and accuracy, underscoring the role of scale.

Applying RL after SFT reinforces this pattern: the 14B model improves steadily, while smaller models (3B, 7B) show little benefit and suffer from instability, including repetitive text and malformed JSON that causes failed tool calls. Although rewards may rise initially, behavior often collapses. Overall, sufficient scale is crucial for stable reasoning and reliable tool-augmented decision making.

#### Training progressively shapes tool-use behavior from underuse to strategic deployment.

Figure: Figure 5: Tool call distributions on HotpotQA across three training stages: Pre-SFT, Post-SFT, and Post-RL.
Refer to caption: x3.png

We examine tool invocation distributions across three development stages: pre-SFT, post-SFT, and post-RL. Using HotpotQA trajectories, we plot interaction counts per example (Figure [5](https://arxiv.org/html/2510.18798v1#S3.F5)), revealing how supervision and reinforcement learning shape reasoning depth and tool use. Before SFT, tool usage is limited, with most conversations involving around three calls—suggesting a conservative strategy arising from insufficient mastery of tool behaviors. After SFT, the distribution shifts markedly rightward, peaking at 10 calls and extending up to 50, indicating more active and flexible tool engagement, often in lengthy multi-step interactions. Following RL, the distribution sharpens between 5 and 8 calls, with very high and very low counts becoming rare.

Although we did not penalize underuse during RL, the model rarely produces trajectories with fewer than five calls. This implies that repeated tool use is implicitly reinforced, as it aids verification and validation. Overall, training progression shows a shift from underuse, to overuse, to strategic use. RL fine-tuning improves task performance and yields more stable, efficient behavior, encouraging sufficient—but not excessive—tool invocation without hard-coded constraints.

Figure: Figure 6: Impact of SFT data composition on tool usage and accuracy. We vary the ratio between single-pass correct trajectories and multi-refinement trajectories during SFT.
Refer to caption: x4.png

Data mixing ratio in SFT are also key to performance improvement.

We further examine the effect of data composition during the SFT stage. As shown in Figure [6](https://arxiv.org/html/2510.18798v1#S3.F6), the ratio between single-pass correct trajectories—where the model produces the correct answer in a single step—and multi-refinement trajectories—where multiple reasoning or retrieval steps are required—plays a pivotal role in shaping model behavior. Increasing the share of longer reasoning trajectories encourages more frequent tool usage, but does not necessarily translate into higher accuracy. Striking an appropriate balance between the two types of trajectories is therefore essential for effective SFT, and can even determine the success of subsequent RL fine-tuning.

### 3.4 Ablation Studies

Figure: Figure 7: (Top) Limiting the model to submit only one answer results in a decrease in the final model’s performance. (Bottom) Without complex data for SFT, the 14B model collapses with decreasing rewards, while the 7B model shows slight reward improvement but often generates garbled text before tool calls, potentially leading to failure.
Refer to caption: x5.png

In this section, we present ablation experiments to assess the contribution of key components in our framework. We evaluate two variants: (1) restricting the model to produce only a single answer during reinforcement learning, and (2) training without cold-start initialization. Results are summarized in Table [7](https://arxiv.org/html/2510.18798v1#S3.F7).

Our analysis yields two main observations:
(1) Each component is critical to the success of training—removing any of them consistently degrades performance.
(2) The cold-start strategy is especially important, as it substantially improves the model’s ability to develop effective tool-use behaviors. For more challenging tasks, we further find that high-quality SFT data is indispensable for stable training.

## 4 Related Work

LLM With Tools
A growing body of work focuses on tool-augmented LLMs which models designed to interact with external tools (e.g., APIs, databases) to perform complex tasks. ToolLLaMA (Qin et al., 2023) applies structured decision-tree search to decompose multi-step instructions into tool calls, demonstrating that explicit planning over API chains enhances compositional reasoning. The ToolPlanner (Wu et al., 2024) model further refines this dynamic, shifting from overly detailed instruction prompting to more realistic interaction schemas across multi-granularity instruction tasks. However, limitations remain in completing harder tasks. Tool-Star (Dong et al., 2025) generates tool usage trajectories as cold-start data through prompt-based sampling, then designs a multi-tool self-critique reinforcement learning framework with hierarchical reward design to enhance the LLM’s reward understanding in real-world scenarios and promote effective tool collaboration, representing the latest training paradigm.

Reasoning Agentic RAG
Early RAG approaches are primarily linear or branching structures (Chen et al., 2024; Gao et al., 2024a; b). They typically rely on manually crafted prompts or fixed execution workflows, which severely constrain the model’s autonomy and flexibility. Recent work has begun to incorporate reinforcement learning into RAG (Jin et al., 2025; Song et al., 2025). These methods adopt an outcome-driven RL framework that enables the model to explore how to invoke external retrieval systems during the reasoning process.
DeepResearcher (Zheng et al., 2025) extends this line of work to the web search setting, but their approach still depends on stronger models to act as agents for webpage navigation. (Shi et al., 2025) propose Pangu DeepDiver, which combines a carefully constructed dataset designed to foster information-seeking behavior in open-world internet environments with a specialized DeepDiver framework to enhance search capabilities.
However, despite these advances, the reasoning chains produced by current methods remain relatively shallow and are insufficient for solving more complex or open-ended tasks.

## 5 Conclusion

In this work, we introduced WebSeer, a novel agent training paradigm tailored for real-world web-based retrieval environments. By synthesizing multi-refinement reasoning trajectories through rejection sampling and incorporating self-reflective reinforcement learning (SRRL), WebSeer learns to perform deeper, more robust reasoning that mimics human information-seeking behavior. Through extensive experiments across a wide range of open-domain and out-of-domain question answering benchmarks, WebSeer consistently outperforms existing baselines achieving state-of-the-art performance on datasets such as HotpotQA, Bamboogle, and SimpleQA.
WebSeer lays a foundation for more general-purpose reasoning agents that can seamlessly interact with dynamic, heterogeneous web environments.

## References

- Asai et al. (2023)
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi.
Self-rag: Learning to retrieve, generate, and critique through self-reflection.
In *The Twelfth International Conference on Learning Representations*, 2023.
- Chen et al. (2024)
Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun.
Benchmarking large language models in retrieval-augmented generation.
*Proceedings of the AAAI Conference on Artificial Intelligence*, 38(16):17754–17762, Mar. 2024.
doi: 10.1609/aaai.v38i16.29728.
URL [https://ojs.aaai.org/index.php/AAAI/article/view/29728](https://ojs.aaai.org/index.php/AAAI/article/view/29728).
- DeepSeek-AI et al. (2025)
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong
Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu,
Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang.
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025.
URL [https://arxiv.org/abs/2501.12948](https://arxiv.org/abs/2501.12948).
- Dong et al. (2025)
Guanting Dong, Yifei Chen, Xiaoxi Li, Jiajie Jin, Hongjin Qian, Yutao Zhu, Hangyu Mao, Guorui Zhou, Zhicheng Dou, and Ji-Rong Wen.
Tool-star: Empowering llm-brained multi-tool reasoner via reinforcement learning, 2025.
URL [https://arxiv.org/abs/2505.16410](https://arxiv.org/abs/2505.16410).
- El Asri et al. (2017)
Layla El Asri, Hannes Schulz, Shikhar Sharma, Jeremie Zumer, Justin Harris, Emery Fine, Rahul Mehrotra, and Kaheer Suleman.
Frames: a corpus for adding memory to goal-oriented dialogue systems.
In Kristiina Jokinen, Manfred Stede, David DeVault, and Annie Louis (eds.), *Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue*, pp. 207–219, Saarbrücken, Germany, August 2017. Association for Computational Linguistics.
doi: 10.18653/v1/W17-5526.
URL [https://aclanthology.org/W17-5526/](https://aclanthology.org/W17-5526/).
- Gao et al. (2024a)
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang.
Retrieval-augmented generation for large language models: A survey, 2024a.
URL [https://arxiv.org/abs/2312.10997](https://arxiv.org/abs/2312.10997).
- Gao et al. (2024b)
Yunfan Gao, Yun Xiong, Meng Wang, and Haofen Wang.
Modular rag: Transforming rag systems into lego-like reconfigurable frameworks, 2024b.
URL [https://arxiv.org/abs/2407.21059](https://arxiv.org/abs/2407.21059).
- Hendrycks et al. (2021)
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.
Measuring massive multitask language understanding.
*Proceedings of the International Conference on Learning Representations (ICLR)*, 2021.
- Ho et al. (2020)
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa.
Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps, 2020.
URL [https://arxiv.org/abs/2011.01060](https://arxiv.org/abs/2011.01060).
- Jin et al. (2025)
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han.
Search-r1: Training llms to reason and leverage search engines with reinforcement learning, 2025.
URL [https://arxiv.org/abs/2503.09516](https://arxiv.org/abs/2503.09516).
- Joshi et al. (2017)
Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer.
Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension, 2017.
URL [https://arxiv.org/abs/1705.03551](https://arxiv.org/abs/1705.03551).
- Kokane et al. (2025)
Shirley Kokane, Ming Zhu, Tulika Awalgaonkar, Jianguo Zhang, Thai Hoang, Akshara Prabhakar, Zuxin Liu, Tian Lan, Liangwei Yang, Juntao Tan, Rithesh Murthy, Weiran Yao, Zhiwei Liu, Juan Carlos Niebles, Huan Wang, Shelby Heinecke, Caiming Xiong, and Silivo Savarese.
Toolscan: A benchmark for characterizing errors in tool-use llms, 2025.
URL [https://arxiv.org/abs/2411.13547](https://arxiv.org/abs/2411.13547).
- Kwiatkowski et al. (2019)
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov.
Natural questions: A benchmark for question answering research.
*Transactions of the Association for Computational Linguistics*, 7:452–466, 2019.
doi: 10.1162/tacl_a_00276.
URL [https://aclanthology.org/Q19-1026/](https://aclanthology.org/Q19-1026/).
- Lee et al. (2025)
Zhicheng Lee, Shulin Cao, Jinxin Liu, Jiajie Zhang, Weichuan Liu, Xiaoyin Che, Lei Hou, and Juanzi Li.
Rearag: Knowledge-guided reasoning enhances factuality of large reasoning models with iterative retrieval augmented generation, 2025.
URL [https://arxiv.org/abs/2503.21729](https://arxiv.org/abs/2503.21729).
- Lewis et al. (2020)
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al.
Retrieval-augmented generation for knowledge-intensive nlp tasks.
*Advances in neural information processing systems*, 33:9459–9474, 2020.
- Li et al. (2025)
Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou.
Search-o1: Agentic search-enhanced large reasoning models, 2025.
URL [https://arxiv.org/abs/2501.05366](https://arxiv.org/abs/2501.05366).
- Mallen et al. (2023)
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi.
When not to trust language models: Investigating effectiveness of parametric and non-parametric memories.
In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), *Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 9802–9822, Toronto, Canada, July 2023. Association for Computational Linguistics.
doi: 10.18653/v1/2023.acl-long.546.
URL [https://aclanthology.org/2023.acl-long.546/](https://aclanthology.org/2023.acl-long.546/).
- Qin et al. (2023)
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun.
Toolllm: Facilitating large language models to master 16000+ real-world apis, 2023.
URL [https://arxiv.org/abs/2307.16789](https://arxiv.org/abs/2307.16789).
- Qwen et al. (2025)
Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu.
Qwen2.5 technical report, 2025.
URL [https://arxiv.org/abs/2412.15115](https://arxiv.org/abs/2412.15115).
- Rein et al. (2024)
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman.
Gpqa: A graduate-level google-proof q&a benchmark.
In *First Conference on Language Modeling*, 2024.
- Sardana (2025)
Ashish Sardana.
Real-time evaluation models for rag: Who detects hallucinations best?
*arXiv preprint arXiv:2503.21157*, 2025.
- Shao et al. (2024)
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al.
Deepseekmath: Pushing the limits of mathematical reasoning in open language models.
*arXiv preprint arXiv:2402.03300*, 2024.
- Sheng et al. (2025)
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu.
Hybridflow: A flexible and efficient rlhf framework.
In *Proceedings of the Twentieth European Conference on Computer Systems*, pp. 1279–1297, 2025.
- Shi et al. (2025)
Wenxuan Shi, Haochen Tan, Chuqiao Kuang, Xiaoguang Li, Xiaozhe Ren, Chen Zhang, Hanting Chen, Yasheng Wang, Lifeng Shang, Fisher Yu, and Yunhe Wang.
Pangu deepdiver: Adaptive search intensity scaling via open-web reinforcement learning, 2025.
URL [https://arxiv.org/abs/2505.24332](https://arxiv.org/abs/2505.24332).
- Song et al. (2025)
Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen.
R1-searcher: Incentivizing the search capability in llms via reinforcement learning, 2025.
URL [https://arxiv.org/abs/2503.05592](https://arxiv.org/abs/2503.05592).
- Trivedi et al. (2022)
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal.
Musique: Multihop questions via single-hop question composition.
*Transactions of the Association for Computational Linguistics*, 10:539–554, 2022.
- Trivedi et al. (2023)
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal.
Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions, 2023.
URL [https://arxiv.org/abs/2212.10509](https://arxiv.org/abs/2212.10509).
- Wei et al. (2024)
Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus.
Measuring short-form factuality in large language models, 2024.
URL [https://arxiv.org/abs/2411.04368](https://arxiv.org/abs/2411.04368).
- Wu et al. (2024)
Qinzhuo Wu, Wei Liu, Jian Luan, and Bin Wang.
Toolplanner: A tool augmented llm for multi granularity instructions with path planning and feedback, 2024.
URL [https://arxiv.org/abs/2409.14826](https://arxiv.org/abs/2409.14826).
- Yang et al. (2018)
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning.
Hotpotqa: A dataset for diverse, explainable multi-hop question answering.
*arXiv preprint arXiv:1809.09600*, 2018.
- Yu et al. (2025)
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang.
Dapo: An open-source llm reinforcement learning system at scale, 2025.
URL [https://arxiv.org/abs/2503.14476](https://arxiv.org/abs/2503.14476).
- Yu et al. (2024)
Yue Yu, Wei Ping, Zihan Liu, Boxin Wang, Jiaxuan You, Chao Zhang, Mohammad Shoeybi, and Bryan Catanzaro.
Rankrag: Unifying context ranking with retrieval-augmented generation in llms.
*Advances in Neural Information Processing Systems*, 37:121156–121184, 2024.
- Zhang et al. (2025)
Dingchu Zhang, Yida Zhao, Jialong Wu, Baixuan Li, Wenbiao Yin, Liwen Zhang, Yong Jiang, Yufeng Li, Kewei Tu, Pengjun Xie, and Fei Huang.
Evolvesearch: An iterative self-evolving search agent, 2025.
URL [https://arxiv.org/abs/2505.22501](https://arxiv.org/abs/2505.22501).
- Zheng et al. (2025)
Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu.
Deepresearcher: Scaling deep research via reinforcement learning in real-world environments, 2025.
URL [https://arxiv.org/abs/2504.03160](https://arxiv.org/abs/2504.03160).
- Zhu et al. (2024)
Andrew Zhu, Alyssa Hwang, Liam Dugan, and Chris Callison-Burch.
FanOutQA: A multi-hop, multi-document question answering benchmark for large language models.
In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)*, pp. 18–37, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
doi: 10.18653/v1/2024.acl-short.2.
URL [https://aclanthology.org/2024.acl-short.2/](https://aclanthology.org/2024.acl-short.2/).

## Appendix A Appendix

## Appendix B Case study

Case study 1.1 is generated by the Qwen2.5-14B-instruct model and exhibits a brittle reasoning process; it over-relies on information parsed from search snippets, leading to an incorrect inference about one player’s batting hand, and prematurely abandons a sub-task after a single, poorly formulated query fails. This results in an incomplete and partially erroneous answer. In contrast, WebSeer employs a more robust, iterative strategy. It consistently pairs web searches with targeted page queries for verification, avoiding the errors from snippet parsing. Crucially, when faced with an initial search failure for a specific entity (player Jeff Austin), it demonstrates effective error recovery by refining its search query with disambiguating context (‘1998 MLB draft‘). This methodical approach of iterative refinement and verification, despite requiring more tool interactions, allows the second agent to successfully decompose the problem and provide a complete and correct solution.

Case study 2.1 involved more than a dozen tool calls: multiple search calls to locate the “Panther tank” article and repeated query_on_page checks to confirm it was officially the Panzerkampfwagen V Panther deployed on both Eastern and Western fronts from mid-1943, followed by additional query_on_page calls verifying that the Jagdpanther tank destroyer used its armor and suspension; by repeatedly cross-validating the tank’s name, deployment timeline, and derivative chassis, the process covered all key details and reinforced the conclusion with multiple lines of evidence, perfectly demonstrating how to leverage tools systematically and efficiently to support rigorous reasoning.

## Appendix C The usage of LLM

Portions of this paper were polished by the large language models (LLMs), which were used to improve the clarity, grammar, and presentation of the text. The models were not used to generate research ideas, conduct experiments, or analyze results; all conceptual contributions and empirical findings are the work of the authors. We carefully reviewed and edited all LLM-generated suggestions to ensure accuracy and alignment with the intended meaning.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: See weaknesses.
soundness: 3
strengths: - The paper studies a relevant problem: Tool calling (such as Web searches) is a very timely and challenging area of research, unveiling a lot of potential for LLMs. 
- The presented WebSteer approach is sound and, to the best of my knowledge, novel.
- The experiments ablate two key choices made: 1) self-reflection and 2) the cold-start initialization via SFT, both proved to be beneficial.
- Trained LLMs reach SOTA benchmark scores for their sizes.
summary: This paper presents WebSeer, a novel approach for training LLM search agents, i.e. agents able to perform iterative calls to Web search engines to answer a given query. The approach consists of two main stages: 1) supervised fine-tuning (SFT) on high quality multiturn search interactions,  and 2) a novel self-reflective reinforcement learning (SRRL) pipeline. The approach is used to finetuning Qwen 14B and achieves SOTA benchmark scores for it's size when given access to Web search.
weaknesses: - The analysis claims that the method's effectiveness is heavily dependent on model capacity. Only the 14B model showed consistent performance and stable behavior improvements, while smaller 3B and 7B models showed degraded performance after SFT and instability during RL. I find this quite puzzling and would like to know if the authors have further explanations on why this is the case. In particular, some qualitative example after SFT or during RL would further illustrate the cause. 

- The ablations focus primarily on the training methodology (SRRL, cold-start). It would be beneficial to see further ablations, e.g., on the impact of the reward design components ($R_{format}$ and $R_{correct}$).

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: Questions regarding the RL algorithm:
1. In SRRL, if a rollout achieves a low reward in an earlier step but a better reward in a later step (via self-reflection), how is the advantage calculated? Is it based only on the final reward, or does it consider every reward achieved throughout the trajectory?
2. Alternatively, is every answer submission considered a separate rollout? (e.g., separate trajectories sharing initial steps but with different ending steps/submission times).
3. Following 2., if not, does it mean the model will generate a trajectory containing multiple answer submissions? In that case, how do you determine when to end an trajecotry during inference?
4. Could you elaborate on how the model's behavior differs between SRRL and regular GRPO? More analysis or examples here would be helpful.

Questions regarding experiments:
1. What is the specific base model used for training (e.g., Qwen2.5-14B or Qwen3-14B)?
2. It appears the "Qwen3-14B w/ Web Search tools" baseline is missing from Table 1. Can you add the results?
3. Figure 5 is difficult to interpret. It might be clearer to present a direct numerical comparison between your model vs. the no-SFT baseline, perhaps in a format similar to Table 5.
soundness: 3
strengths: 1. The paper tackles the relevant and interesting problem of training tool-use agents. It presents a solid approach based on RL, with a notable mechanism for encouraging self-reflection (though some technical details remain unclear, as noted in the questions below).
2. The empirical performance is strong. The model outperforms baselines on most target datasets and also demonstrates good OOD generalization.
3. The analysis in Sec. 3.3 provides helpful insights on behaviors of tool-use agents trained with RL and empirical lessons on how to train them.
summary: This paper proposes WebSeer, a tool-use agent trained with reinforcement learning (RL) that incorporates a self-reflection mechanism. Specifically, the agent is equipped with searching, webpage reading, and code execution tools. The training process contains a supervised fine-tuning (SFT) stage followed by an RL stage. This RL stage mostly follows GRPO, but is designed to encourage self-reflection. The resulting 14B model achieves state-of-the-art results on several multi-hop question answering datasets and demonstrates strong generalization to out-of-domain datasets.
weaknesses: 1. Limited discussion of related work: There is a significant body of work on using RL for general tool use, not limited to search [1,2,3]. The paper would be stronger if it more comprehensively discussed and positioned itself relative to this broader line of related work.
2. Narrow experimental scope: The evaluation is focused on multi-hop question answering. While I think this is acceptable and does not undermine the paper's core contributions, showing the framework's effectiveness on other types of tasks would make the work even more impactful. If time and resources do not permit, it would be great if the authors can include some discussions.

[1] ReTool: Reinforcement Learning for Strategic Tool Use in LLMs
[2] Tool-Star: Empowering LLM-Brained Multi-Tool Reasoner via Reinforcement Learning
[3] Synthetic Data Generation & Multi-Step RL for Reasoning & Tool Use

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: - Restricted to open transfer
Did you consider building a training set that mixes a small portion of open-web episodes with wiki episodes, then run SFT and also SRRL on the mixed set? If yes, how did accuracy, chain length, and number of submissions change as the share of open-web episodes grew? If not, what behavior would you expect and why?

- SFT verifier details
What model do you use for the verifier? What is the prompt, decoding setup, retry budget, and timeout? What rule accepts or rejects a candidate answer, and does the verifier use the same tools and constraints as the agent?

- Verifier impact
How much does verifier stringency change the SFT distribution and later RL stability? If you lower or raise the verifier bar while keeping SFT size fixed, how do single-step share, tool-call patterns, and final accuracy move?
soundness: 4
strengths: - Clear and practical idea: learn long, reflective tool use in a clean Wiki-only setup, then run on the open web. The two-stage SFT → SRRL recipe is simple, and “submit answer” is treated as a tool, which makes stopping explicit.
- Strong results across many benchmarks and tough OOD sets, with consistent gains over strong web-agent baselines.
- Good behavior analysis and ablations that explain why it works: tool-call depth becomes more balanced, removing SRRL hurts, and the SFT data mix matters.
- Interfaces and evaluation are well specified, with concrete tool APIs and LLM-as-a-judge prompts, plus transparent training and compute details.
- Meaningful impact: the method makes web agents plan deeper and verify better, and the recipe is easy to adopt in other agent settings.
summary: The paper introduces WebSeer, a web search agent that learns to build deeper and more reliable tool chains for multi hop questions. It first collects long reflective trajectories by rejection sampling for supervised fine tuning, then runs a self reflective RL stage where the model can resubmit an answer, read feedback, and refine its plan. The agent uses a small toolset for search, page reading, and code execution, and it treats submitting the final answer as a tool. Training is site restricted on Wikipedia, while inference runs on the open web. A 14B model beats strong baselines on seven benchmarks and also does well on FanOutQA, FRAMES, SimpleQA, and Bamboogle. Taken together, the two stage recipe turns shallow tool use into steadier and longer chains and lets one model plan, act, verify, and decide when to stop. It also shows that learning in a clean, restricted setup can carry over to the open web with strong gains.
weaknesses: - The open-web transfer story is promising but the ablations are only partial. The paper already shows that limiting RL to a single submission hurts, analyzes tool-use distributions, and studies how the SFT data mix changes behavior. What’s missing are comparisons across training regimes (restricted vs mixed vs open), a sensitivity check for the page-reading/normalization stack, and systematic curves that relate accuracy to chain length under different tool budgets, search-depth caps, and submission-discount settings. A short discussion or small exploratory study on these fronts would make the restricted-to-open claim feel more complete.

- The SFT verifier is under-specified and hard to reason about. We don’t know the model family/size, the prompt template and decoding settings, or the retry budget. It’s unclear how it accepts or rejects a candidate (string match vs semantic check), how it handles aliases and partial answers, or whether it uses the same tools and constraints as the agent. We also don’t see rates: how often it disagrees and forces a re-check, what fraction of trajectories get kept, how long those trajectories are, and how much of the SFT set ends up single-step vs multi-step. Without these basics, reproducibility is limited and it’s difficult to tell how much the verifier shapes the data distribution and later RL stability.

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: How accurate is the independent verifier during trajectory construction? Did you measure verifier precision/recall or its effect on SFT data quality?

How sensitive is WebSeer’s performance to the number of verifier retries 
$K$ and the validity predicate threshold?

In SRRL, how did you balance the trade-off between early termination (short trajectories) and the benefit of deeper reflection?

Could the approach generalize beyond QA (e.g., web-based decision-making or scientific reasoning tasks)?

What is the relative wall-clock cost of SFT vs SRRL stages, and is RL training feasible for larger backbones (e.g., 70B models)?

Would a joint training of verifier and reasoner be possible or beneficial compared to separate optimization?
soundness: 3
strengths: Clear motivation and novelty: The idea of explicit self-reflection integrated into both SFT and RL phases is well justified and fills a gap between shallow tool use and deep verification-oriented reasoning.

Strong empirical performance: WebSeer consistently outperforms baselines on seven benchmarks, with notable OOD generalization (e.g., +15.9 on NQ, +27.2 on 2Wiki, +4.0 avg over Qwen2.5-14B).

Well-structured training pipeline: The paper carefully separates data construction, SFT masking (to avoid overfitting to observations), and SRRL fine-tuning, providing a replicable framework.

Insightful analysis: The ablation studies show how self-reflective trajectories and multi-submission feedback both contribute to deeper tool usage and higher final accuracy.
summary: This paper tackles the challenge of training open-domain, web-based question-answering agents capable of multi-hop reasoning and robust self-correction. Existing web agents often stop prematurely, fail to verify their reasoning, or overfit to closed-corpus retrieval. The proposed system, WebSeer, aims to build deeper and more reflective reasoning trajectories by combining:

- Self-Reflective Supervised Fine-Tuning (SFT): multi-turn rejection-sampling trajectories verified by an independent verifier model, teaching the agent to backtrack and self-correct.

- Self-Reflective Reinforcement Learning (SRRL): a reinforcement phase where the model can submit intermediate answers multiple times, receive F1-based feedback, and continue improving under a shaped reward.

The final agent interacts with real web tools (search, webpage reader, code executor) and is evaluated across seven multi-hop QA benchmarks, achieving state-of-the-art performance and strong generalization to out-of-distribution datasets.

Contributions:

Introduces a two-stage self-reflection training pipeline that operationalizes “reason-verify-refine” behaviors in web agents.

Proposes a self-reflective RL framework (SRRL) that allows iterative answer submission and feedback-guided reasoning.

Designs a trajectory-level reward that balances answer correctness, brevity, and depth of reasoning.

Demonstrates substantial improvements over strong baselines (Search-r1, Qwen2.5-14B, etc.) on both in-domain and open-web QA datasets.
weaknesses: Limited transparency of verifier reliability: The paper assumes the verifier’s correctness predicate $\psi$
 yields high-quality reflective trajectories, but quantitative verifier accuracy or error propagation analysis is missing.

Reward design justification: The choice of exponential discount on multiple submissions is somewhat ad-hoc; an analysis of sensitivity to discount parameters or exploration–exploitation balance would strengthen claims.

Computational cost: The reflective data collection (multi-turn verifier rejection sampling) seems expensive; runtime statistics or data efficiency analysis are lacking.

Ablation on real-web vs synthetic setting: While generalization results are good, it remains unclear whether the model trained with restricted search data directly scales to dynamic, noisy real-web environments.

Clarity and formalism: Some algorithmic details (e.g., GRPO vs DAPO implementation differences) could be more rigorously formalized.

---

confidence: 3
contribution: 3
initial_rating: 4
presentation: 2
questions: Please refer to weakness #1 and #2.
soundness: 3
strengths: The research problem—improving the depth and self-correction capabilities of search agents—is interesting and important, as current agentic systems may struggle with error accumulation and shallow reasoning.


The proposed two-stage training framework, which combines supervised fine-tuning on curated reflective trajectories with reinforcement learning, is a logical approach to the problem.


The reward mechanism in the RL stage, which provides explicit feedback on answer quality and allows for multiple submission attempts, is a reasonable design choice.
summary: This paper introduces a search agent designed for multi-step question-answering on the web. The agent's core innovation is a two-stage training framework: 1) using multi-turn rejection sampling, where a reasoner model's proposed answers are validated by a separate verifier, to generate a high-quality dataset of self-correcting reasoning paths. 2) using Self-Reflective Reinforcement Learning,  where the agent can submit multiple answers and receive textual feedback on its performance, allowing it to iteratively refine its strategy.
weaknesses: Overall, the presentation of the methodology could be further improved.

For example, the paper's methodology relies on an independent "verifier" model (V) to generate the SFT dataset. The description of this verifier is insufficient (only mentioned between line 188 and line 197). It is unclear how the verifier is implemented. If the verifier itself is unreliable or follows an incorrect tool-use path, its judgments would be flawed, which could introduce noise or incorrect patterns into the SFT dataset and undermine the entire training pipeline. The paper needs to detail the verifier's implementation.


In addition, the term "F1-based feedback" is not standard in reinforcement learning literature. While the paper explains it as a feedback signal, it should be more formally defined as part of the observation space or reward function to avoid ambiguity. (It is possible that the authors refer to F1 score in classification. But according to line 253, it appears that F1 is calculated based on one predicted answer and one ground-truth answer, while the F1 score in classification should be calculated based on a set of predictions and GT answers.)


(Minor issue) The paper requires careful proofreading. For instance, lines 182 and 183 contain nearly identical, repetitive sentences: "To address this limitation, we propose a multi-turn rejection sampling method to collect reasoning paths that incorporate reflective patterns. To overcome this limitation, we propose a multi-turn rejection sampling method that collects reasoning paths enriched with reflective patterns."

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: 1. The paper emphasizes that WebSeer benefits from a self-reflection mechanism. But in Case study 1.2, the agent's outputs show no reasoning or reflection step. It directly generate tool calls in each round. Could the author clarify how self-reflection is implemented?
soundness: 2
strengths: 1. Training an autonomous search agent capable of conducting deep, strategic, and reflective searches addresses an important and challenging research problem, and also has significant practical application value.

2. WebSeer extends to real-world search scenarios and incorporates more reasonable tools, such as Webpage Reader, enhancing its applicability.

3. The method achieves SOTA performance on HotPotQA and SimpleQA benchmarks.
summary: The paper proposes WebSeer, a search agent capable of performing multi-turn, real-world searches for QA tasks. The authors introduce a two-stage training pipeline that includes SFT and RL. After training, WebSeer demonstrates improvements in both the number of effective tool calls and accuracy.
weaknesses: 1. In the experiments (Table 1), WebSeer (a 14B model) is compared against baselines using smaller 7B/8B models. This raises uncertainty about whether the observed improvements result from the proposed method itself or simply from scaling up model size.

2. The SRRL method is a key contribution of the paper. However, Figure 5 (top) shows that SRRL only improves accuracy by 3.6% on HotPotQA and 2.9% on SimpleQA. Were the experiments conducted multiple times to assess variance? The relatively small gains make it unclear whether the improvement is statistically significant. 

3. A main feature of SRRL is that it allows the model to submit multiple answers within a single dialogue turn. However, there is no ablation study isolating the effect of this design choice. How would performance change under a conventional setting that allows only one submission during training?

4. At Line 241, the paper claims that its RL framework unifies SFT and RL under a self-reflection mechanism. However, Sec 2.4 does not describe how this unification is achieved. Based on the content, SRRL appears to directly apply GRPO for policy optimization.

5. (minor) Lines 182-183 contains repeated sentences.


Decision: Accept (Poster)