Unlocking Long-Horizon Agentic Search with Large-Scale End-to-End RL
Authors: Jiaxuan Gao
Wei Fu
Minyang Xie
Shusheng Xu
Chuyi He
Zhiyu Mei
Banghua Zhu
Yi Wu
Venue: ICLR 2026 Poster (2026)
Keywords: Agentic RL; Asynchronous RL; Search Agent
Primary area: foundation or frontier models, including LLMs
TL;DR: This paper introduces training expert-level search agents with large-scale RL. By using 128 turn limit and high-quality synthetic data, the agent learns complex long-horizon search strategies, reaching competitive results on frontier benchmarks.

Abstract: Recent advancements in LLM-based agents have demonstrated remarkable capabilities in handling knowledge-intensive tasks using external tools. One representative example is search agent. Existing open-source search agents heavily rely on advanced commercial LLMs: they either collect trajectories from the larger, stronger models for supervised fine-tuning or directly use them as specialized tools. In this work, we develop ASearcher, a single-model search agent purely trained by reinforcement learning (RL) without using any commercial APIs for data or tools. Based on an RL-trained QwQ-32B model, ASearcher is capable of conducting complex reasoning, such as uncertainty analysis and conflict verification, and achieve comparable performances to commercial search agents. There are two key techniques to unlock such long-horizon information-seeking abilities: first, we design a two-staged agentic process to synthesize high-quality QA pairs as the training data for RL; second, we conduct large-scale long-horizon RL, allowing the agent to take up to 128 actions per rollout for sufficient exploration. In particular, after RL training, ASearcher achieved scores of GAIA 58.1, xBench 51.1, and Frames 74.5 using only basic search tools. Furthermore, ASearcher also demonstrates strong zero-shot transferability: ASearcher can be further augmented with an additional summary tool, which is supported by DeepSeek-V3, and test-time scaling, which aggregates the answer from 16 parallel rollouts. With both zero-shot enhancements, the performances of ASearcher further rise to 71.8, 75.0, and 83.4, respectively, outperforming OpenAI DeepResearch and Kimi-Researcher, suggesting the great potential of RL scaling for agentic tasks. We release all the code and data at an anonymous link. The model will be released after the review process.

Published as a conference paper at ICLR 2026
UNLOCKING LONG-HORIZON AGENTIC SEARCH WITH LARGE-SCALE END-TO-END RL
Jiaxuan Gaol? Wei F Minyang Xie' Shusheng Xu²,
Chuyi He?, Zhiyu Mei?, Banghua Zhu?, Wu
1 III, Tsinghua University, Ant Group, AReal. Team 3 University of Washington
samjia20008gmal com, ixwuyi@gmaill com
ABSTRACT
Recent advancements in LLM-based agents have demonstrated remarkable capabil- ities in handling knowledge-intenn inteessiv tasks using external tools. One representative example is search agent. Existing open-source search agents heavily rely on ad- vanced commercial LLMs: they either collect trajectories from the larger, stronger models for supervised fine-tuning or directly use them as specialized tools. In this work, we develop ASearcher, a single-model search agent purely trained by reinforcement learning (RL) without using any commercial APIs for data or tools. Based on an RL-trained QwQ-32B model, ASearcher is capable of conducting com- plex reasoning, such as uncertainty analysis and conflict verification, and achieves comparable performances to commercial search agents. There are two key tech- niques to unlock such long-horizon information-seeking abilities: first, we design a two-staged agentic process to synthesize high-quality QA pairs as the training data for RL; second, take up to 128 actions per rollout for sufficient exploration. In particular, after RL training, ASearcher achieved scores of GAIA 58.1, xBench 51.1, and Frames 74.5 using only basic search tools. Furthermore, ASearcher also demonstrates strong zero-shot transferability: ASearcher can be further augmented with additional summary tool, which is supported by DeepSeek-V. V3, and test-time scaling, which aggregates the answer from 16 parallel rollouts. With both zero-shov shot enhancements, the performances of ASearcher further rise to 71.8, 75.0, and 83.4, respectively, outperforming OpenAl DeepResearch and Kimi-Researcher, suggesting the great potential of RI scaling for agentic tasks. We release all the code and data at anonymous link
conduct large-scale long- hong-horizon RL, allowing the agent
The model will released after the review process.
Pass@1
Stage MIN.
Stage 2
Stage
Stage
100
After RL
Before RI
Tool Calls Trai
100
Word Count/ TTaj
alternatively
however
80
After zhro Transfera TIS
71.8
75.0
74.5
AVG.
check
previous
40
mention
43.7
20
28.7
200
XBench DeepSearch
100
Training Step
400
Training Step
Figure I: (Left) End-to-end RL brings substantial improvements to a simple agent: Through RL training, our agent, ASearcher, obtains +15.0, +22.4, and +15.6 improvements on GAIA, xBench, and Frames, respectively. (Middle) During RL training, ASearcher learns to conduct long-horizon search, with tool calls progressing from average of only 1.67 initially over 20 tool calls in latter training stages. (Right) Count of keywords during the training process reveals emergence of complex search behaviors including reflective behaviors and referencing external information. Our detailed case study in Appendix B also shows that the agent learns expert-level search strategies.
Corresponding author

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Recent advances in LLM-based agents have demonstrated remarkable capabilities in solving complex.
showledge-intensive intensive problems by leveraging single or multiple external tools (Xi et al., 2025 Yao
al. [023] Wang et al. 2024a). Among the diverse capabilities, deep information retrieval using
tools stands out as a particularly critical aspect of advanced search agents (OpenAl, 2025 Google
Team 2025: Perplexity Team 2025). For a concrete example, seemingly simple question like "How many gold medals did China win the 2012 London Olympics?" actually requires intricate reasoning. At the time, China was credited with 38 golds, but a decade later, two doping disqualifications in women's 's race walking led to additional gold medal to China, raising the total to 39. This illustrates how search agents must reconcile historical records with noisy, sometimes conflicting, information from diverse sources and identify the underlying causes for the conflicts to deliver accurate answers.
To equip agents with this deep retrieval capability, recent open-source approaches frequently depend on commercial L.MMs, either generate expert data to serve specialized sub-modules within
complex multi-model framework (Li al. 2025a [Tao et al., 2025 Li al. 2022h) Team 2025a). For
instance, AFM (Li al. 2025b) collects supervised fine-tuning data from a multi-agent framework powered by multiple ad vanced commercial models such as Claude-Sonnet-4 (Anthropic) 2025) and Gemini-2.55
(Google Team 2025). On the other hand, Miro Thinker [Team] (202a)) employs distinct commercial LLMs and VLMs for specialized tasks such as audio transcription, visual question answering. and complex reasoning. This reliance on proprietary models raises fundamental question: can achieve the performance of commercial systems without dependence on commercial models?
In this work, we present ASearcher, a search agent trained solely by Reinforcement Learning (RL., and show that purely end-to-end RL can enable the emergence of advanced long- oong-horizon search strategies, despite that ASearcher is based on a single model using only search tools. Particularly, two techniques are the key to the advanced information-seeking abilities in ASearcher. First, we develop a scalable QA synthesis agent to generate a high-quality dataset of 25.6k challenging QA pairs that necessitate multi turtiittuuuun tool use. In training time, we employ a two-stage curriculum that progressively focuses on challenging tasks. Specifically, the agent is initially trained over questions spanning diverse difficulties, including those easy questions requiring only one After the agent learns preliminary search capabilities, we shift the training distribution to focus on long-horizon tasks that require a minimum of five tool calls. Second, we train the agent with large-scale long-horizon RL. with large turn limit of 128 per rollout trajectory. A large turn limit encourages the exploration and discovery of sophisticated, long-horizon strategies. To ensure high training efficiency, we employ fully asynchronous agentic RL training based on AReaL (Fu et al. 2025) that decouples trajectory collection from weight updates for training efficiency.
tool calls.
We use large reasoning model QwQ-32B (Team] 2025b) as the base model in our experiments. During RL training, our agent, ASearcher, learns to conduct significantly more complex searches, with an increasing average number of tool calls from only 1.67 calls at the beginning to more than 20 calls. A detailed case study and keyword analysis further reveal the emergence of complex search behaviors, such conducting uncertainty analysis and verification searches. Our finding on search agents is akin to DeepSeek-RR R1 (DeepSeek-AI et al. 2025), where the emergent reasoning capabilities can be fully incentivized by RL. We evaluate our agents on challenging benchmarks, including GAIA MMialon et al., 2023) xBBenhh-DeepSearch "Xbench-Team 2025). and Frames (Krishna et al.) 2024). With only single model and basic search tools, ASearcher achieves competitive scores of 51.7, and 74.5 (Avg@4) on GAIA, xBench, and Frames. respectively, demonstrating that strong performance is attainable with single-model design. Finally, ASearcher also demonstrates strong zero-shot transfer ability to external summary tools. By employing an external summary tool supported by DeepSeek-V V3 and applying test-time scaling techniques, where we aggregate the conclusions from 16 indepdent runs, the performance can be further enhanced, rising to 71.8, 75.0, and 83.4, respectively, and achieving results competitive with commercial systems.
2 RELATED WORKS
Search Agents. Some works have investigated agent workflows to leverage tools for solving complex tasks (Li et al. 2025c Zhao et al. (022).. Prompt- methods, though effective for rapid development, are fundamentally limited by the capacity of the underlying L.M.s. Some works
2

---

Published as a conference paper at ICLR 2026
attempt construct SFT trajectories for LLMs. For instance, Asai et al. (2023); Yu et al. (2024)
leverage LLMs to synthesize trajectories for SFT. Prior works investigate Reinforcement learning
(RL) methods to enhance the LLM-based agents. mostly focusing multi-hop QA benchmarks. Jin
et al. 2025); Song et al. 2025); Chen et al. :025):(Zeen [Zeng et al.
2025) perform RI. training with multi-hop QA data and observe increasing amount of tool calls. More recently, researchers focus
on deep rescarch tasks, by fine-tuning complex prompt-based agents through offline RL (Li et al. 2025d), SFT on diverse trajectories (Sun* al. 2025 Li et al. 2025a Team 2025a), adopting multi- framework (Li et al. 2025b), and constructing challenging QAs for RL training. Tao al. 2025). In this work, we focus on deep information retrieval, and show that RL alone can equip
singgeemodel agent with advanced long-horizon search capabilities.
Agentic Reinforcement Learning. Recent works have begun to improve the agentic capabilities of LLMs and LRMs through online Reinforcement Learning. Prior works have investigated agentic RL in various domains, such as search agents (Jin al. 2025
al. 2025a d Tao al. 2025) and coding (Luo et al. 2025a Wei et al. 2025). critical aspect of agentic RL is to activate and enhance the tool-calling capability of the models. Wei et al. (2025) and Luo al. (2025a) investigate training coding agents for resolving real-world engineering questions. More recently, Li et al. (2025a); Tao al. 2025);
al. (2025b); [Tam (2025a) investigate training deep search agents by first running Supervised Fine-TTunig to equip the agent with basic long-horizon search capability as cold-start and then running Reinforcement Learning to further enhance the agent. this work, we focus on search agents and show that simple agent can learn complex long-horizon search strategies through reinforcement learning.
ASearcher
In this work, we present ASearcher, which unlocks search intelligence in search agents through large-scale asynchronous RL training. In the subsequent sections, we present the agent design, the training data as well as data synthesis agent, and fully asynchronous reinforcement learning training.
3.1
AGENT DESIGN: ONE SINGLE MODEL FOR ALL
If Action is conswer>
User Query
Reasoning
Answer
<thitko ... </think>
Action <search>
Response
History
Analysis carabyiis> Available Info.
Search Engine
Entries
Analysis Action
Plane </analysis>
Action
Action <acoess>>
Browsse
Let Look
Webpage
Summarize
S 128 turns
V Single Agent
V No External Model API Call
Figure 2: ASearcher utilizes simple agent design with two basic tools including search and browsing tools, without relying on any external models. The agent capable of both reasoning and summarizing lengthy web contents.
We employ simple agent design in ASearcher, as illustrated in Fig.
Agent Input. In each turn, the search agent takes in the user query as well as the history of resolving the user query. The history contains previous tool responses including search results and webpage summaries, and also previous analysis and history actions.
Reasoning, Analysis, and Action. In each turn, given the user query and history context, the agent generates three components further conduct in-depth analysis and exploration,
3

---

Published as a conference paper at ICLR 2026
Synthetic QA & Facts
Extract Fact & Inject
Quality Verification
Search
Daniel Charbonell is
Engine
Caban outficlder playod for San Jose Jiants
Step Busic Quality Check Salvable w. supporting facts Clarity ofe questice
Synthetic QA
Injection
Browset
Inject
meny yeus Charbonell'ss
Q: How many yeus was the controct signed 2014 by the Cabas cunfeelder who plaved for Ser Juse Giarss
Francico Giants
Difficullt Measurement
sianed 2014!
A: Four years
Select Info. & Fuzz
Supporting Facts
Two years
Three years
Four years
Fuzz
2014
Duniel signed
San
Francisco
Qu How many yeus was
Step Uniqueness
Dariet Charbozell's coutroct with Sar Francisco Glants whes signeil early 20/
I8 yaa also correct?
Update question & facts verification successes
Figure 3: Data Synthesis Agent. Starting from a seed QA, the data synthesis agent iteratively modifies the question through two actions, Injection and Fuzz.
Reasoning: In this part, the agent conducts internal reasoning over the current situation. Since we instantiate the agent with large reasoning models such as QwQ-32B, generating reasoning process naturally supported by the underlying model. the reasoning process, the agent analyzes available information, evaluates the query resolving progress, reflects over previous results, determines the unresolved aspects, and deduces concrete plans for future turns. Note that the reasoning part is only used for guiding the generation of subsequent analysis and action. Since this reasoning part usually contain noisy and lengthy model- generated texts, the reasoning part is not included in future history ensure clean history.
Analysis:
The analysis part is summarization of the reasoning part, where the agent extracts the key conclusions derived from the reasoning process and also makes a plan for the subsequent step.
After thoroughly analyzing the current state with the reasoning and analysis parts, the agent finally determines the next-step action. The agent could either answer the question and terminate the execution process, or invoking external tools to obtain new information from external sources.
Action:
Tools. When the agent determines to invoke external tools, two basic tools are available: a search engine and a web browser. When the action is "<search>", the search engine takes a query as input and returns relevant snippets along with corresponding URLs. When the action is "<access>" the web browser accepts an URL and returns content of the webpage.
Webpage Summarization. Note that 32K tokens. Therefore, we split the webpage into several chunks, with a maximum character count of 10k per chunk. We employ the agent summarize each chunk into a compact summary.
webpages usually very long, easily exceed
3.2 TRAINING DATA
Our training data are from two primary sources, including samples filtered from open-source datasets and synthetic high-quality question-anse ansowrr (QA) pairs.
We begin with the training sets of HotpotQA Yang et al. 2018) and 2Wiki- MultiHopQA (Ho et al. 2020). We employ a masel-based filtering process. We first train a model on the full set of open-source data with RL fllowing Jin et al. (2025). and then generate 16 responses for each question using the trained model. Finally, we filter out questions that are too hard for the model or too easy for the model. Finally, from total of 304k QA pairs, we retain 16k challenging samples.
Open-source Data.
Data Synthesis Agent. We further develop a data synthesis agent supported by QwQ-32B to create high-quality question-answer pairs. As shown in Fig. 3 the data synthesis agent begins with seed
4

---

Published as a conference paper at ICLR 2026
question, and iteratively modifies the question to increase the complexity. To ensure the synthetic question is strictly aligned with reliable sources, a list of supporting facts facts obtained during the question synthesis process is continuously updated. At each step, the agent automatically selects between two key actions,
Action Injection aims to enrich the context the question by inserting facts related to the question. The agent first selects an entity in the question and then obtains one piece of related fact about the selected entity from external sources such as Wikipedia. Then new question is proposed by injecting the fact into the question.
Action 2: Fuzzing blurs certain details in the question to increase the uncertainty level of the question. For example, "Catskill Mountain Railroad" could be replaced with historic mountain railway".
To ensure that a synthetic question is of high quality and to precisely evaluate the difficulty, we incorporate rigorous quality verification phase for assessing synthetic questions. This verification phase includes three steps: basic quality check that assess the clarity and resolvability of the question, difficulty measurement by employing QwQ-32B to direct generate answers without tools, and answer uniqueness check by evaluating whether any of the mismatched answers generated during the Difficulty Measurement step could serve as alternative valid answers.
Through iterative injection and fuzzing, the data synthesis agent produces questions that involve complex information and high uncertainty, requiring extensive search and reasoning find the correct answer. After completing the question synthesis process, we filter out questions that the LRM can directly generate the correct answer without using tools.
Two-Stage Curriculum. During training time, we employ two-stage training data scheme. In the first stage, apply RL training on the full training set, which include QAs spanning different difficulties. This wide range of training distribution trains the agent to equip basic tool-calling and reasoning capabilities. In the second stage, to further activate the long-horte morizon search capability of the agent, we remove QAs that are solvable with less than 5 tool calls and use the rest data for the second training stage.
3.3
ASYNCHRONOUS AGENTIC RL TRAINING
3.3.1 EFFICIENCY CHALLENGE OF LONG-HORIZON AGENTIC RL
High Variance in Trajectory Execution Time. During RL training, we use large turn limit of 128. In practice, when using a large turn limit at training time, long trajectories intro- duce significant variance in execution time. We first analyze the number of tool calls during RL training our QwQ agent (Fig. [T) and observe that the longest trajectories can span dozens more tool calls than shorter ones. Second, we also report the total number of output tokens per trajectory during training in Fig. 4 As illustrated in the figure, the training process involves extremely long trajectories. The data distribution reveals that these lengthy trajectories constitute very small proportion of the samples. This disparity could lead to highly unpredictable per-trajectory runtime, complicating training efficiency.
Frequency
2k
15 2812
100k
200k
4G0k
Total Oulput Tokens
Figure Distribution of the out- put lengths of trajectories generated during RI. training (recorded from Step 290 to Step 310).
Efficiency Issues of Agentic RL Training. Both prolonged execution and high runtime variance degrade RL training ef- ficiency. We take one-step-off RL training system (Luo et al.) 2025b) as a representative example for batch generation RL systems. As shown in Fig. [5] though this system overlaps trajectory rollouts with model training, batch generation remains bottlenecked by the slowest trajectory (e.g., trajectory 7), causing GPU idle time and under-utilization.
5

---

A. The goal of the agent Published as a conference paper at ICLR 2026
Time Training Step N
Train Step
Train Step N-1
One-Step-Off
Idle Time
Traj
rollout
Traj
Traj TO
Traj
Blocked by LONG trajs?
Fully Asyne RL
Trnj
Traj
Traj 5
Truj
Traj
Fally
Traj
Traj 4
Tmij
Traj &
Traj 9
Traj 10
Traj 11
Try
idle time!
Faster Training
Train N+2
Time of Training
Figure 5: One-Step-off RL v.s. Fully Asynchronous RL. In batch generation systems, a batch should wait for the longest trajectory, leading significant GPU idle time. contrast, fully asynchronous RL achieves faster training than batch generation RL by fully decoupling training and trajectory generation, achieving near-full resource utilization for trajectory generation.
3.3.2 FULLY ASYNCHRONOUS RL TRAINING.
To ensure efficient agentic RL training, we adopt fully asynchronous training paradigm. Notably, approach incorporates asynchoriizatinn at the two distinet aspects.
Asynchronous Trajectory Rollouts. Trajectory rollouts are collected in parallel and do not directly interfere with each other. Each trajectory independently sends tool calling requests to corresponding servers and LLM generation requests to the LLM inference engine. Concurrent requests from different trajectories are automatically handled by the servers. Fully independent trajectory execution ensures a trajectory does not need to wait for other trajectories when generating LLM responses and waiting for tool calling responses, thereby improving training efficiency.
Decoupled Rollout and Training. Besides asynchronous rollout, trajectory rollouts and model updates are also fully decoupled. In Fig. we compare our fully asynchronous RL training with one-step-off RL training, which utilizes asynchronous rollout within batches. fully asynchronous RL training, long trajectories do not block generation and span multiple versions, significantly reducing GPU idle time and achieving near-full GPU utilization during generation. On the training side, a training step is launched as soon sufficient trajectories collected to form batch.
3.4 TRAINING DETAILS
MDP Formulation. We follow the formulation of Markov Decision Process (MDP). Formally, an MDP defined by the tuple (S, A, T, R). Here represents the state space, usually containing the history, search results, and retrieved webpages. A denotes the action space and an action includes tokens generated by the agent. Some tool calling could be extracted from the action through specific tags, e.g. <search> search query </search>. T(s'ls, a) is the transition function. At each timestep, the agent receives state 8, and generates an action as with policy
to maximize the return J(x)
E E R(s,, at) a x(s))
GRPO Training. We employ the GRPO Shao et al., 2024 algorithm train search agents. Specifically, for each input question G trajectories TI, T2. ... To generated where T, (sh, ag. 8f ... s..). To optimize the agent, we employ the following loss,
T lafl
Tolat at Tasn (1 186, ax,ez)
(GR = E. (T))( (1x)
Σ
min
Ass
Ei
la
iml
t= mml
TolLigl8t Cr,cs)
clip
c, + A.
(1)
maru (ak,s re; a(,z))

---

Published as a conference paper at ICLR 2026
where is hyperparameter, and A, is the advantage for the i-th trajectory, computed based on the relative rewards of all trajectories within each group.
Dynamic Filtering. To enhance training efficiency, implement dynamic filtering to exclude queries that lack meaningful training signals. Specifically, we remove queries where all responses yield identical rewards (resulting in zero advantages).
Reward Function. We adopt a sparse-reward setting where rewards are computed at trajectory completion. For reward function, we utilize LLM-as-Judgel the reward function and omit format rewards, as large reasoning models such as QwQ-32B could inherently maintain proper output formatting with high probability.
al. 2023 Wang et al., 2024b) as
EXPERIMENTS
4.1 EXPERIMENT SETUP
Benchmarks. We conduct evaluation on suite of challenging benchmarks, including Frames Kr
ishna et al., 2024), GAIA (Mialon et al. 2023). XBench-Deeppearch (Xbench- Team 2025). and
HLE(Li et al. (025d). Frames contains 824 challenging questions for evaluating the ability of the agent to synthesize accurate responses from multiple sources. GAIA contains real-world questions that demand multi-turn tool calls and step-by-step problem solving. xBBench-DeepSearch consists of 100 challenging Chinese questions constructed by human experts, evaluating the agent's 's in-depth planning and reasoning capabilities. HLE (Human's Last Exam) features expert-level difficulty across a wide range of disciplines, not only requiring the agent to search for related materials, but also understanding and solving domain-specific questions. For GAIA, we use the 103 examples from the text-only validation subset (Li et al. 2025c For HLE, we use the 500- size subset following Li et al.
(202dd).
Baselines.
We compare ASearcher against different sets of baselines:
Commercial Deep Research Agents. We make a comparison with OpenAl DeepRe-
search (OpenAl 2025) and Remi-Researchhr (MoonshotAl 2025).
General LLMs using Tools. We evaluates general LLMs equipped with external tools
including Qwen3-30B-A3B Qwen3-235B-A225 (Yang al.) 20225, OpenAl-o3 (OpenAI 2025), DeepSeek-RI (DeepSeek-Al et al. 2025), and Claude-4-Sonnet (Anthropic) 2025).
Open-source Search Agents. Finally, we make comparison with set of 32B-scale open-source search agents, including Scarch-ol(QwQ-3
(Li et al. (202cc). Search-R1- 32B (Jin et al.) 2025), WebThinker-QwQQ (Li et al. (025d) 2025d) Simpie QwQ (Sun et al. 2025) and WebDancer-32B (Wu et al. 2025), WehSailor-32B (Li et al. 2025a). and WehShaper-322
(Tao et al., 2025). We also include AFM-RI-332B ( et al. 2025b). that
adopts a multi-agent design, and Miro Thinner--2n-BDDDa DPO (Team 2025a). that utilizes more tools beyond search tools, including tools for image and audio processing.
Evaluation Metrics. We adopt LLMMass-Jdge (LasJ) the main metric for evaluating the perfor- mance. For LLM-as-Judge, an LLM Dowee2.5-72B-nnstruue is prompted to assess the correctness of outputs. For ASearcher, We report pass @ score by evaluating seeds. For baselines, report the scores reported in official reports if there are any.
Training Details of ASeaccher. We the turn limit as 128 and the batch size is set as 64 for ASearcher. We use Adam W optimizer with a learning rate of 2e-6. Our training framework is built up on AReal. (Fu et al. 2025). Training of ASearcher takes approximately 16k H800 GPU hours.
4.2 MAIN RESULTS
Benchmark Performance. Table Ishows experiment results challenging QA tasks that require advanced and search strategies. Our agent, ASearcher, achieves competitive Pass I scores on GAIA, Frames, and HIE,, outperforming a wide range of previous 32B-scale agents. Notably, ASearcher
7

---

Published as a conference paper at ICLR 2026
Table I: Pass@1 results of ASearcher and baselines. 'indicates results are obtained from official reports. For open-source search agents, use "No Commercial LLM" to indicate that the agent does not use commercial models as a component of agent design or for data collection, and "Non-Search Tools" to indicate the tools used by the agent besides basic search tools. ASearcher outperforms wide range of 32B-scale open-source agents, with single model and no extra tools. When integrating external models summary tool and applying test-time scaling approach, ASearcher able to achieve on-par performance with commercial systems.
Method
No Commercial LLM
Non-Scarch Tools
GALA
ABench DeepSearch
Frames
HLE
Commercial Deep Research Agents
Kimi Reseurcher
69.0
78.8
26.9"
OpenAl DeepResearch
62.0
26.6'
General M. using Took
OpenAT-o3
70.5
66.7
84.0"
20.2
Qwen3-80H-A33
56.41
Qwea3-239B-A225
45.61
46.0
20,0
DeepSeek-RI R1
55,0
82.01
24.8'
Claude Sonnet
68.31
64.5"
80.71
20.31
Open-source Search Apents
Scarc--o (QwQ))
Search-R1-32B R1-32B
28.6
19.5
44.1
WebThinken QwQ
48.5
32.8
57.7
15.8
Simple DS-QwQ
50.57
35.8
68.81
WehDenee-Quw
51.51
WebSailor-328
53.21
53.3
69.81
WehStuper-328
AFM-RL-32B
Code Sandbox
55.31
18.0
Visual Answering.
MicoThinke-52B-DDD
x
Audia Transcription, Linux Sandbox
56.0"
74,81
20.6
Ours (QQQ-22B)
ASnorcher
58.7
$1.1
74.5
21.5
Summary-Deesseekva
X
60.3
56.4
76.6
23.4
Test-time Search
even achieves par performance as MiroTinker-32B-DD that uses extra tools besides search tools. These results further highlight the superiority handling long-horizon and real-world tool use.
Zero-Shot Transfer with Summary Tool & Test-time Scaling, Although adopt single-model during training time, the agent design of ASearcher is generalizable. Specifically, the webpage summarization process could be supported by external models. We here DeepSeek-V. V3 as the webpage summarization model. From Table [ it clear that using a more powerful model for summarization improves the performance of ASearcher. We further investigate test-time scaling. Specifically, we run K = 16 independent runs for each problem in the test set and aggregate the conclusions from these independent runs with DeepSeck-V3. As shown in Table [ test-time scaling approach leads to competitive performance with the commercial agents, including Limi-Rescarcher, OpenAI DeepResearch, and OpenAI o3. Please refer Appendix. E for additional results and implementation details of Test-time Scaling.
4.3
EMERGENT BEHAVIORS
search
doc
Word Count Traj
alternatively
Word Count/ Traj
mention source earllier
Waw My Ma
wait
check
confirm
previous
however
3k
2x.
0.
50
150
200
250
300
50
100
150
200
250
300
350
400
Training Step
Training Step
Figure Left: Word count of reflective keywords during training time. Right: Word count of keywords indicating explicit reference of external information.
8

---

Published as a conference paper at ICLR 2026
Keyword Analysis. In Fig. 6 we plot the word count of different keywords during the training process. These keywords include words that indicate reflection behaviors of the agent, such as "alternatively", "however", and "wait", and also words indicating that the agent is referencing to external information, such as "doc" and "mention". From Fig. [ it is clear that the agent learns to reflect over previous actions and conclusions. Interestingly, we also of RL training where the training focuses on challenging tasks, the agent learns to refer to external information, evidenced by the increment in the word count of "doc" and "mention" after step 200.
that, in the second stage
Case Study In Fig. 7, we provide a detailed case study on an extremely challenging question from GAIA (Mialon et al. 2023). Specifically, we analyze Search-RI R1-32B (Tin et al. 2025) and ASearcher. The detailed trajectories are provided in Appendix B In this question, to identify the correct answer, the search agent should first find out the mentioned species according to condition "genus named for Copenhagen" identify the correct 2021 1 article based on the citation in the wikipedia page of the species, and then find out the papers of the two mentioned persons.
In Fig. 7 Search-Ri- R1-32B is unable to decompose the complex query into individual components, consequently only making ambiguous queries that involve too many unknown information. The agent also has severe hallucinations, producing conclusions that are not supported by the search results. Finally, it fails to resolve all unknown information. In contrast, ASearcher decomposes the complex query into precise queries. ASearcher could also summarize all related information from webpage and analyze all candidate answers. Finally, after the correct answer "Mice" is found, the agent spends further turns verifying previous conclusions before reporting the final answer.
An Example Question from GAIA
Q: What animals that were mentioned in:
both llias Lagkouvardos's and Olga Tapia's papers the alvei species of the genus named for Copenhagen outside the bibliographies and also the 2021 article cited on the alvei species' Wikipedia page about multicenter, randomized, double-blind study?
A: Mice
Rearch-R1-325
Lack of Verificution
Unable Decompose
Hallucinations
Complex Query
... searches
Q
Ilias Lagkouvardos alvei species of the genus named for Copenhagen
Ilias Lagkouvardos alvei species of the genus named for Copenhagen
AAshiguour scarch
Arsswer is Pigs
Amihigoou search
alivi species possibly Capranoouus
*End with 3 searcher without confirmation
Naisy contents
-Searcher-Web-eee QwQ (Ours)
Identify Relevant Source
Grounded Verification
Uincertaitt-AAwar Analysis
Wikipedia page of Hafnia alvei *Frification
Focused seurch
Wikipedia page of Hafnia alvei
Confirm the 2021 study relies on prior mouse studies.
the genus named for Copenhagen
Extract Key Info.
Edoutfy the article
Q
03ga Tapla Hafnia alvei mouse studies
Exttract Key Info.
2021 article a human clinical on "overweight humans"
species Halnia
"eriffcattin
Analyzing key animals: 1 related to Hafnia alvei 2.Piss: unrelated
Clear evidence fs found: ...
the 2021 article
*Vorification
3.Dairy animals: unrelated
Answer Mice
Figure 7: A case study complex query from GAIA. Search-R1-32B is unable to break down the complex question and has severe hallucinations. It is also worth noting that, since the turn limit is set as small value, c.g. 4, during training, the model only exhibits a short tool-use horizon. Our end-to-end RL agent, ASearcher. exhibits key behaviors featuring Search Intelligence: uncertainty- aware reasoning (list and examine candidate answers), precise extraction from noisy contents, and grounded verification.
4.4 ABLATION STUDY
We perform ablation studies over various key components of ASearcher. Specifically, we aim to answer the following questions,
9

---

Published as a conference paper at ICLR 2026
How does the training- taining-time time turn limit influence the final performance and learned behaviors?
Q1:
Q2: ties?
the two-stage curriculum necessary for activating the long-horizon search capabili-
Q3:
How does the training data of ASearcher compare with training data of baseline?
Experiment Setup. To carry out the ablation studies, we perform RL training starting from the Stage checkpoint of ASearcher. All ablation studies are trained for 200 steps to ensure the same number of training steps as the full training recipe of ASearcher. The ablation studies evaluated on GAIA (Mialon et al. 2023) and xBench-Dee Deaeh- (Xbench-Teer Team 2025)
Table Ablation Study on Training-time Turn Limit and Data Quality. The ablation studies reveal that both large turn limil and high-quality training data are the key to unlocking the long-horizon search capability of the model.
of Tool Calls at Training Time
GAIA
XBBech-Deeppearch
ASearcher (Full)
26.59
58.7
51.1
Ablating Training-time Turn Limit
ASearcher Turn Limit=10
3.48
49.2
39.3
Ablating Training Data
ASearcher w. Stage Data Only
5.40
51.6
43.0
ASearcher AFM Data
4.12
50.9
39.9
The results of ablation study is as shown in Table. 2 We highlight key conclusions derived from
ablation analysis.
(Q1) Training-time Turn Limit. large turn limit is crucial for enabling the model's long-horizon search capability. Specifically, training with large limit significantly surpasses training with a small turn limit (e.g.. Turn Limit = 10). With a large turn limit, the agent provided with a rich exploration space for complex search strategies.
(Q2) Two-Stage Curriculum. Compared with continued training using only Stage I data, the full training recipe incorporating two-stage curriculum helps the agent to learn stable long-horizon search by focusing challenging queries that necessitate least five tool calls. This shows the importance of the progressive curriculum for mastering complex tasks.
(Q3) Data Quality. We also compare our training data with that concurrent work, AFM (Li et al. 2025b), by applying asynchronous RL training with 128-turn limit using the AFM training data. Training with AFM data is unable to incentivize complex search capabilities and achieves sub-optimal benchmark performance, highlighting the high quality and complexity of our synthesis data.
CONCLUSION
In this work, we present ASearcher, investigating large-scale RL training for search agents. Our contribution includes a fully asynchronous agentic RL training system and a data synthesis agent for large-scale high-quality QA construction. Through large-scale RL training, ASearcher demonstrates emergent complex strategies using only a single model and basic search tools. We hope our work could benefit future work training advanced agents for broader range of applications.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Anthropic. Claude takes research to new places. research April 2025. Accessed: 2025-04-01.
httpp::/ww..attrropc.ccoom
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Self-reflective retrieval augmented generation. In NeurlPS 2023 workshop on instruction tuning and instruction following, 2023.
Mingyang Chen, Tianpeng Li, Haoze Sun, Yijie Zhou, Chenzheng Zhu, Haofen Wang, Jeff Z Pan, Wen Zhang, Huajun Chen, Fan Yang, et al. Learning to reason with search for ilms via reinforcement learning. arXiv preprint arXiv:2503. 19470, 2025.
DeepSeck-Al, Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng. Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangho Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang. Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo. Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang. Minghua Zhang, Minghui Tang. Meng Li, Miaojun Wang, Mingming Li. Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang. Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. Chen, R. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye. Tao Yun, Tian Pei. Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L.. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang. Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo. Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui L., Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xic, Zhengyan Zhang, Zhewen Hao. Zhicheng Ma. Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, 2025.
URL http:tps//aaxii..rrttteee http:///wwiv..///////11211111111
Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei, Jiashu Wang, Tongkai Yang. Binhang Yuan, and Yi Wu. Areal: A large-scale asynchronous reinforcement learning system for language reasoning, 2025. URL
abs/2505. 24298
Google Team. Introducing Gemini deep research, 2025. URL https:///eminng google/ Accessed: 2025-04-06.
Seeervlewwdeeepreeserro
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing multi-hop qa dataset for comprehensive evaluation of reasoning steps, 2020. URL http:tts////axx..orrgg
abs/ 201/2011120 01060
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training Imms to reason and leverage search engines with reinforcement learning, arXiv preprint arXiv:2503.095 095116 2025.
Satyapriya Krishna, Kalpesh Krishna, Anhad Mohananey, Steven Schwarez, Adam Stambler, Shyam Upadhyay, and Manaal Faruqui. Fact, fetch, and reason: A unified evaluation of retrieval- augmented generation, 2024. URL //trxi//xvvvvia wwwppxiwwwr.aa//oooeeoooooo /249 12941
11

---

Published as a conference paper at ICLR 2026
Kuan Li, Zhongwang Zhang, Huifeng Yin, Liwen Zhang, Litu Ou, Jialong Wu, Wenbiao Yin, Baixuan Li, Zhengwei Tao, Xinyu Wang, et al. Websailor: Navigating super-human reasoning for web agent. arXiv preprint arXiv:2507.025900 2025a.
Weizhen Li. Jianbo Lin, Zhuosong Jiang, Jingyi Cao, Xinpeng Liu, Jiayu Zhang, Zhenqiang Huang, Qianben Chen, Weichen Sun, Qiexiang Wang, Hongxuan Lu, Tianrui Qin, Chenghao Zhu, Yi Yao, Shuying Fan, Xiaowan Li, Tiannan Wang, Pai Liu, King Zhu, He Zhu, Dingfeng Shi, Piaohong Wang, Yeyi Guan, Xiangru Tang, Minghao Liu, Yuchen Eleanor Jiang, Jian Yang, Jiaheng Liu, Ge Zhang, and Wangchunshu Zhou. Chain-of-agents End-to-end agent foundation models via multi-agent distillation and agentic rl, 2025b. URL (RLtttts://arxxt wttp://..rrabb////////0/
13167
Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. Search-ol: Agentic search-enhanced large reasoning models. arXiv preprint arXiv:2501.053666, 2025c.
Xiaoxi Li, Jiajie Jin, Guanting Dong, Hongjin Qian, Yutao Zhu, Yongkang Wu, Ji- Ji-Rong Wen, and Zhicheng Dou. Webthinker: Empowering large reasoning models with deep research capability. arXiv preprint arXiv:2504.21776 2025d.
Yuxuan Liu, Tianchi Yang. Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, and Qi Zhang. Calibrating I1m- bmsddsed evaluator. arXiv preprint arXiv:2309. 2023.
Michael Luo, Naman Jain, Jaskirat Singh, Sijun Tan, Ameen Patel, Qingyang Wu, Alpay Ariyak, Colin Cai, Shang Zhu Tarun Venkat, Ben Athiwaratkun, Manan Roongta, Ce Zhang, Li Erran Li, Raluca Ada Popa, Koushik Sen, and Ion Stoica. Deepswe: Training a state-of-the-art coding agent from scratch by scaling rl. https: paatee//y/eettttttt-baoaaee www.nnotiio...omoo motiinnsiteee//// site/ DeepSWE- Trainii--innn------ aeanng--FFullln Fallauuullppppaaaee Open sarrnnd-orree---e- State- the Art- toooiing-ggeen by S-a-laaaaanng-nnn 2025a. Notion Blog.
Michael Luo. Sijun Tan, Roy Huang, Ameen Patel, Alpay Ariyak, Qingyang Wu, Xiaoxiang Shi, Rachel Xin, Colin Cai, Maurice Weber, Ce Zhang, Li Er- Li. Raluca Ada Popa. and Ion Stoica. Deepcoder: A fully open-source 14b coder at 03-mini level. hatpa://preetty-raiio-- www.:/iooo............. htt.bbo....oioonii..io site/ DeepCoder-A-FRe Ly-Oully-Op-----e Source- -aweee------ooor--at-aaaa- oatpl:/ooooo.eeaaeaaaaaaaaaoeaa0 mini -even----eell--e 2025b. Notion Blog.
Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: benchmark for general ai assistants. In The Twelfth International Conference Learning Representations, 2023.
MoonshotAL. Kimi-researcheer http:://///oooooooooooo..o.oooia 2025.
OpenAI. Introducing openai o3 and o4-mini. Sttrroducing----and--o0
https://operaai.com/nneet
2025.
OpenAl. Introducing deep research, 2025. URL https://ggeraa.comnnnee
introducing-ddee--reeee Accessed: 2025-04-06.
Perplexity Team. Introducing Perplexity deep reeprraarcc http:/roooccccccoo..peeeeeeeeeee
2025. URLAtp:://w..peeee research Accessed: 2025-04-06.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemat- ical reasoning in open language models. arXiv preprint 2303:0002..33300, 2024.
Huatong Song. Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji- Ri-Rong Wen. RI-searcher: Incentivizing the search capability in Imms via reinforcement learning. arXiv preprint 25Xivv2503.05592 2025.
12

---

Published as a conference paper at ICLR 2026
Shuang Sun", Huatong Song*, Yuhao Wang. Ruiyang Ren, Jinhao Jiang. Junjie Zhang, Lei Fang, Zhongyuan Wang, and Ji- Ji-Rong Wen Wayne Xin Zhao. Simpledeepsearcher: Deep information seeking via web cob-powered reasoning trajectory synthesis. 2025. URL https:/ htttps://gttubb com/ RUCA BoxA/Box//aa Simp @DeepSearcher
Zhengwei Tao, Jialong Wu, Wenbiao Yin, Junkai Zhang, Baixuan Li, Haiyang Shen, Kuan Li, Liwen Zhang, Xinyu Wang, Yong Jiang, Pengjun Xie. Fei Huang, and Jingren Zhou. Webshaper: Agentically data synthesizing via information -eeeanniinnnniinngggg formalization, 2025. URL https://
Trxiv.org/ abs 2200 15061
MiroMind AI Team. Mirothinker: An open-source agentic model series trained for deep re- search and complex, long-horizon problem solving. httts:://itthb..co wtw.:ttww...cooiiioiMioodaa... MiroThinker 2025a.
Qwen Team. Qwq 32b: Embracing the power of reinforcement learning, March 2025b. URL github wt/blog//bbaaaooeo 32bb
Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6): (166):88645,,. 2024a.
Minzheng Wang, Longze Chen, Cheng Fu, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, et al. Leave no document behind: Benchmarking long-contens lims with extended multi-doc qa. arXiv preprint arXiv:2406. /7419, 2024b.
Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida I. Wang. Swe-rl: Advancing Ilm reasoning via reinforcement learning on open software evolution, 2025. httLhhtps://artt org/bbs/
2502. 18449
Jialong Wu, Baixuan Li, Runnan Fang, Wenbiao Yin, Liwen Zhang, Zhengwei Tao, Dingchu Zhang, Zekun Xi, Gang Fu, Yong Jiang, et al. Webdancer: Towards autonomous information seeking agency. arXiv preprint arXiv:2505.22644 2025.
Xbench-Team. Xbench-deepsearch, 2025. URL
Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. The rise and potential of large language model based agents: A survey. Science China Information Sciences, 68(2):121 1211 2025.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghho Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang. Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su. Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. ht..htts::// /tt///aivvv... org. abs /505 09388
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809 09600, 2018
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023.
Tian Yu, Shaolei Zhang, and Yang Feng. Auto-rag: Autonomous retrieval- generation for large language models. arXiv preprint arXiv:2411.19444 2024.
Xinjie Zhao, Fan Gao, Xingyu Song, Yingjian Chen, Rui Yang, Yanran Fu, Yuyang Wang, Yusuke Iwasawa, Yutaka Matsuo, and Irene Li. Reagent Reversible multi-agent reasoning for knowledge- enhanced multi-hop qa. arXiv preprint arXiv:2503.0695), 2025.
13

---

Published as a conference paper at ICLR 2026
Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai. Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. Deepresearcher: Sealing deep research via reinforcement learning in worldworld environments. arXiv preprint ar arXhv:2504.03160, 2025.
A LLM USAGE
In the development process of this work, LLM were used in the writing and polishing of the manuscript. Specifically, we used an LLM to assist in refining the language, improving readability, and ensuring clarity various sections of the paper. The LLM helped with tasks such as sentence rephrasing, latex formatting, and grammar checking.
B FULL CASE STUDY
In this section, we provide detailed case study on an extremely challenging question from GAIA Mi alon et al. ) Specifically, we analyze Search-R1-32B (Jin et al. 2025) and Search-ol (QwQ et al. 2025c) in Fig. 8
Solution Path of the Sample Question. In Fig. our case study is carried out on a question requiring finding some specific animal given 2 conditions and 4 unknown variables. To identify the correct answer, the search agent should first find out the mentioned species U1 according to condition C1, identify the correct article U2 that satisfies condition C2, and then find out the papers listed in U3.1 and U3.2. Finally, the correct answer should be determined by cross referencing the article U2 and the papers U3.1& U3.2. To summarize, this example is challenging for several main reasons,
The question involves multiple unknown variables that could point to many different entities. For example, the 2021 article U2 could point to any article published in 2021 and could only be determined given the condition C2 and the alvei species U1.
High Uncertainty:
Requirement for Exact Information Extraction: To find the answer, the agent should list all animals mentioned on the webpages and making cross-document comparison. This would require the agent precisely extract key information from the vast, noisy web contents, instead of simply summarizing the webpages.
Misleading Answers: During the process of solving this task, there could be multiple misleading answers, such "pigs". The agent should rigorously confirm its conclusions by checking the intended answer in all related webpages and documents.
Existing Online RL Approaches Fail to Learn Complex Search Strategies. In Fig. 8 Search- R1-32B is not able to decompose the complex query into individual components, consequently only making redundant queries that involve too many unknown information. The agent also has severe hallucinations, producing conclusions that are not supported by the search results. Finally, it fails to resolve all unknown variables. This case study shows that existing online RL approaches only incentivize elementary search strategies. It is also worth noting that, since the turn limit is set small value, e.g. 4, during training, the model only exhibits a short tool-use horizon.
Prompt-based LLM Agents Could Fail Due to Insufficient Capability of the LLM. In Fig. 8 Search-ol (QwQ) can find the species name U1, as well as the 2021 article U2 and papers &3.1& U3.2 through large amount of tool calls. However, when trying to find the answer, Search-ol (QwQ) would easily miss key information. Consequently, the agent makes incorrect conclusions. Notably, even when the agent finds information that directly links to the correct answer, is still misguided by previous incorrect conclusions. Finally, the agent is unable to verify the correctness of previous conclusions. This case study reveals that, though an open-source model that is not explicitly trained on agentic tasks can perform extensive tool calls, it could not make expert-level reasoning based on the retrieved contents and history contexts.
A.earcher-
We also analyze the search strategy of our end-to-end RL agent, ASearcher- teb--wQ.As shown in Fig. ASearcher- Serrcher--ee--ee Web- QwQ decomposes the complex query into precise and focused queries. Unlike Search-ol (QwQ) that visits large amount of websites after each search
14

---

Published as a conference paper at ICLR 2026
An Example Question from GAIA
What animals that were mentioned in both us. Ilias Lagkouvardos's and usi Olga Tapia's papers the alvei species of the genus named Copenhagen outside the bibliographies were also present in the 02 2021 article cited on the ut alvei species' Wikipedia page about C2 a multicenter, randomized, double-blind study?
Q:
Mice
Question Structure
U3.1 I. L.'s paper
Which animal?
mentioned
U3.2 O. T.'s paper
Target
Condition
is in
about
Unknown
cite wiki
U1. alvei species
article
of
Ut
U2
UB
Unknown
Resolved
multicenter, random, double blind study
C1. genus named Copenhagen
Search Query
Browsing
LLM Gen
Search-R1-328
Search-o1 (QwQ)
ASearcher-Web
Q Ilaas Lagikuuardds alvel species of the genus named
Q
Ct the genus namedd for Copenhagen
Q
the genus named
Copenhagen
for Copenhagen
snarch
Fooased
"Addsaion, andkgaoar
Extrrtt Key Info,
Wollectnation
*Extract Key Infos
Hafma alve
L possibly Coprocuccus
Hafrna alvel
8F Wikipedie
Helnie alvei
In0 Olga Tapin's papers on the alvel species of ct the
Ilias Lagknuvardos Hafnii alvei
Eatrect Key lafi.
"ideniff the article
namedd for Copenhagen
1B3 2021 article is human clinical
"Edandden, ondigasor warch
Hollacination
Viceutff
M.Mic related to uI. Hafinia alvel 2 Pigg unrelated
Analyzing key animals:
Pigs
"The Takes of Artisanal and Industrial Giootyr Microbiona"
3 S.Dairy animals: unrelated
2021 article cited species' Wikipedie page multicenter, randomized, double-bling blind study
Key lnfis.
"Comprehenstve analysie
Q
llies Lagkouvardos
us Hafnia alvel animal studies
Olge Tapia's paper mention animals directly
Medandart, unfigenas sarch
Extract Key
Wellacinatinn
"Whong Conciasione
"denndf the paper
2023 article confirms pigs mentioned. Pige
US2
Olge Tapia & Hatnia alvel
Ilaas Lagkouvardos
''Hannia reduces jotake mass obose mice".
"Adcmiity paper Usz
"Explicir Roference
'Min Kny hylk
J32 Olga Tapla
usr Olga Tapia's paper doesn't
Hafnia alvel animal studios
mention animalls directly
"Howng Cancissior
find
"Idemtify paper
Hafnia Wikipedia page citations
Although no animal is found, voterinary perspective in iss Olga Tapia's ioa's paper related to Lagk''s mouse studies
Fall bo Movelty Kay Jafis
Croms-Dac Inferaace
cr "multicenter randomized
-Mice
double-biind study Hafnia
Wilopedia page of Hafnia alvei
alvei
Confirm the 2021 study relies
studieses.
2021 article *Fidd
"Conyfrmation
diinical and mentioned Mice
USa
Precis Search
Maybe Mice mitake
Key Info.
Clear evidence found
The arwwer no animals
"HHalini HA4507 Reduces
Maybe the gosstion Aaa
Modul Hyperpaggic..
"Eopfici Reference
the 2021 article
conclusioos
*Confimation
Goats
Mice
Figure 8: case study on a complex query from GAIA. Rearch-R1-320 is unable to break down the complex question and has severe hallucinations. Search-ol (QwQ) can identify the corrects articles through extensive tool calls, but casily misses key information and fails to verify wrong conclusions. Our end-to-end RL agent, ASearcher-Web- Web-QwQ, exhibits key behaviors featuring Search Intelligence: uncertainty-aware reasoning (list and examine candidate answers), precise extraction from noisy contents, cross dross-document inference, and rigorous confirmation.
15

---

Published as a conference paper at ICLR 2026
query, ASearcher- Seeerucchee-eb--weeeee focuses visiting the most relevant website. ASearcher-War Web- summarizes all related information from a website. Specifically, all candidate answers are listed and carefully analyzed by the agent. When trying to search for related facts in the papers the agent explicitly references the key information. When the search results do not directly point to the desired target, e.g. when searching with "Olga Tapia (U3.2) Hafnia alvei (U1) animal studies" to find the animals related to Olga Tapia's s paper, the agent does not get a clear information but is able to infer the correct answer by make connection with the other paper U3.1. After the correct answer "Mice" is found, the agent spends further turns on confirming previous conclusions before reporting the final answer. In summary, ASearcher successfully train a search agent that exhibits complex behaviors that feature Search Intelligence,
Uncertainty-awan reasoning: the agent exhaustively lists and examines all possibilities for uncertain entities
Price Key Information Extraction: the agent is able identify the key information from vast, noisy web contents.
Cross-document Inference: the agent is able to infer critical conclusions by making connections among multiple documents.
Rigorous Confirmation: the agent verifies the correctness of previous conclusions with additional tool calls.
C DATA SYNTHESIS AGENT
C.I DETAILS OF DATA SYNTHESIS
We develop a data synthesis agent create high-quality question-answer pairs. As shown in Fig. the data synthesis agent begins with a seed question, and iteratively modifies the question to increase the complexity. To ensure the synthetic question strictly aligned with reliable sources, a list of supporting facts obtained during the question synthesis process is kept and continuously updated for quality verification. At each step, given the current question and list of supporting facts, the agent automatically selects between two key actions,
Action 1: Injection aims to enrich the context the question by inserting facts related to the question. The agent first selects an entity in the question and then obtains one piece of related fact about the selected entity from external sources such as Wikipedia. Then a new question is proposed by injecting the fact into the question. This injection action increases complexity of the question.
Action 2: Fuzzing blurs certain details in the question to increase the uncertainty level of the question. For example, "Catskill Mountain Railroad" could be replaced with "a historic mountain railway". Through fuzzing the question multiple times, both the uncertainty level and difficulty of the question would gradually increase.
To ensure that a synthetic question is of high quality and to precisely evaluate the difficulty, we incorporate a rigorous quality verification phase for assessing synthetic questions,
Step 1. Basic Quality. We employ LLM to assess the basic quality of each question. This verification includes checking the clarity of the question and verifying whether the question-answer pair accurate based on the supporting facts. This quality control step ensures that each question guestion-answer answer pair is properly grounded in reliable sources.
Step 2. Difficulty Measurement. We employ a cutting-edge LRM (e.g., QwQ-32B) to generate multiple answers directly for the synthetic question, without using any external tool. This verification process also serves as a measure of question difficulty.
Step 3. Answer Uniqueness. The fuzzing action may loosen constraints excessively, compromising the uniqueness of the answer. To prevent ambiguity resulting from multiple correct answers, we evaluate whether any of the mismatched answers generated during the Difficulty Measurement step could serve alternative valid answers.
16

---

Published as a conference paper at ICLR 2026
Through iterative injection and fuzzing, the data synthesis agent produces questions that involve complex information and high uncertainty, requiring extensive search and reasoning to find the correct answer. After completing the question synthesis process, filter out questions that the LRM can directly generate the correct answer without relying on search tools. Since these questions can be answered solely based on the intrinsic knowledge of the model, they provide little value for enhancing search capabilities.
Starting with 14,107 seed questions, we perform an average of 6.3 injections and 3.2 fuzzes per question. From the synthetic pool, we select up to three high-quality variations per seed question. This curation process produces a final dataset of 25,624 entries, with the selected questions averaging 4.27 injections and 2.10 fuzzes each.
C.2 EXAMPLE OF SYNTHETIC QA
Table 3: Examples of the synthetic questions, where red indicates injected facts and cyan represents fuzzed content.
Round
Action
Question
Seed QA
When was Michael P. Hein born?
Round
Injection
When was the Eckerd College alumnus who served as the first County Executive of Ulster County, New York, and graduated with Bachelor of Arts in Business Administration born?
Round 2
Injection
When was the individual born who, County Executive of Ulster County, New York, permitted the Catskill Mountain Railroad continue operations between Kingston and Hurley during the 2016 United States House of Representatives elections and also held that position during the 2018 elections?
Round 3
Fuzzing
When was the individual born who, as County Executive of Ulster County, New York, permitted historic mountain railway to continue operations between Kingston and Hurley during the 2016 United States House of Representatives elections and also held that position during the 2018 elections?
Seed QA
Where is the Riggs Hamilton American Legion Post No. 20 located?
Round 1
Injection
Where is the American Legion Post in Russellville, Arkansas, built in 1934 and recognized notable example of WPA Rustic architecture and listed the National Register of Historie Places located?
Round 2
Fuzzing
Where is the American Legion Post in Russellville, Arkansas, built in the early 1930s and recognized as a notable example New Deal-era public works archi- tecture and listed on the National Register of Historic Places located?
Round
Fuzzing
Where the veterans' organization's 's building in Russellville, Arkansas, built in the early 1930s and recognized as a notable example of New Deal-era public works architecture and listed on the National Register of Historic Places located?
We provide two illustrative examples in Tab. Starting with simple question, the injection action replaces specific entities with related factual details. For instance, "Michael P. Hein" is expanded to "who served as the first County Executive of Ulster County, New York.."'. The fuzzing action introduces ambiguity by generalizing precise information, replacing the exact year "1934" with "the early 1930s" or substituting "Catskill Mountain R with "a historic mountain railway."
17

---

Published as a conference paper at ICLR 2026
0.6
fuzz Injection
0.5
Proportion
Proportion
Proportion
0.1
0.2
0.10
0.05
0.1
0.00
15
10
15
0.00
0.25
0.50
0.75
1.00
Number
Figure 9: Statistics from our data synthesis process. (Left) The distribution of the number of supporting facts. (Middle) The distribution of the number of fuzz actions and injection actions. (Right) The accuracy distribution of QwQ-32B in answering the generated questions without using any tools.
C.3 SYNTEHTIC QA STATISTICS
D PROMPTS USED FOR DATA SYNTHESIS, JUDGE, AND TEST-TIME SEARCH
Prompt for Synthesis Agent Action Selection
You autonomous agent for constructing general-domain QAs. Now given the current QA and relevant information. Choose one of the following action to make the question more challenging.
The current question-anse aanioorr--swrrrrrr pair:
{question) }
You can choose one action from the following types: {actions]
Prompt Description of SELECT (the First Step of Injection)
SELECT: select one entity from the relevant entity list. Once such an entity is selected, an external tool will improve the difficulty of the question by replacing information about this entity in the question with sub-questions that take this entity as the answer.
you choose SELECT, the output should be in json format: "'json
"action": "SELECT".
"target": url of the selected entity, note that you should only select the entity from the relevant entity list of the question and make sure the url exactly match the url the relevant entity list. "note": short description of the rationale behind the selection
18

---

Published as a conference paper at ICLR 2026
Prompt Description of FUZZ (the Fuzzing Operation)
FUZZ: fuzz places of information in the question to make the question more challenging. Note that if you choose FUZZ, should make sure the resulted question is still clear and has an unique answer as the original one. You should choose FUZZ only when you find certain pieces of information could clearly point to the correct answer without extensive research to find relevant information.
you choose FUZZ, the output should be in json format: "jjson
"action": "FUZZ",
"question": the modified question after the FUZZ operation
"note": short description why and how the FUZZ operation happens
Prompt of Combining Two Questions (the Second Step of Injection)
You are an autonomous agent for constructing general-domain QAs. Now given two questions, combine these questions into one. Specially, the answer to the second question is related some entity in the first question. To make the combined question challenging, you need to remove information about the answer of the second question and ensure the answer of the combined question remains the same as the answer of the first question. Please ensure that the combined question is clear, solvable, and has an unique answer.
The first Question: (questionA)
The second Question:
[questionB]
Relevant statements:
{statements)
The output should be in json format: "'json
"question": the proposed question,
"answer": the answer
"note": one short description of how the two questions are combined
19

---

Published as a conference paper at ICLR 2026
Prompt of Basic Quality Check
Check the validity of this question-answer pair given its relevant information.
The question is valid is and only if:
1. the question is not simple concatenation of two more questions
the provided answer is the only correct answer to the question
3. the question has unique answer the question can be solved based on the relevant statements
The question-answer pair and the relevant information: {question
You should reply "yes" "no" indicating the validity of the question-answe ansswer pair.
You should think step-by-step first before the final judgement in json format:
Analysis
// your analysis
Final Judgement "json
"judgement;; "yes" or "no"
Prompt of Checking Whether Answer Could be an Alternative Answer
Determine whether the predicted answer is also correct the question. Specially, the predicted answer different from the ground-truth answer, and you should check whether the predicted answer fits with all constraints in the question and is also correct the question.
Question: (question)
Ground-truth answer: (gt_answer)
Facts supporting the ground-truth answer:
"'xt
[statements]
...
Predicted answer: [pred_answer]
"json
judgement": "yes" or "no"
20

---

Published as a conference paper at ICLR 2026
Prompt of LLM-as asJJudge
You are an evaluation assistant. Please determine if the predicted answer is equivalent the labeled answer.
Question: {question]
Labeled Answer: (gt answer)
Predicted Answer: [pred_answer)
Did the model give an answer **'quivalent** the labeled answer? Please re- spond with "Correct" if they are equivalent, or "Incorrect" if they are not equivalent.
The output should in the following json format: "json
"rationale": your rationale for the judgement, as text, "judgemen" your judgement, can only be "Correct" "Incorrect",
E ADDITIONAL RESULTS
ApplyingA.Searcher ASearcher Training on Small Roa-Rnnson Model. We apply our training recipe on 2wen non-reasoning model with much smaller scale. The results show that recipe is also able to enhance the search capability of a small, non-reasoning model.
Table 4: Results on Qwen2.5--BBInnstunn
GAIA
xBench
Before RL
18.2
23.0
After RL
27.7 (+9.5)
27.8 (+4.8)
This demonstrates that,
Our pipeline works effectively on smaller models. Pure RL training also works for non- reasoning models.
The significant performance improvement not solely due to the choice of the base model, QwQ, but comes from the full training recipe.
21

---

Published as a conference paper at ICLR 2026
Prompt for Aggregating K Independently Generated Trajectories Test-time Search
Given question and K independently generated results, please determine the most reliable answer.
Question: {question |
Result 1: {result_1)
Result 2: [result_2] 2
Result K: fresult_Ke K]
The output should in the following json format: "'json
"rationale": your rationale text, "answer": the most reliable final answer,
Evaluating Baseline with Summary Tool & Test-time Scaling. We conduct a comparison with WebSailor-32B (Li et al. 2025a) when using DeepSeek-V. for summary and test-time search with K 16, under turn limit of 128. Since WebSailor does not produce comprchensive analysis at the final step, we directly feed all the trajectories into DeepSeek- V3 to determine the most reliable answer when performing Test-time Search. The results show that:
The baseline, WebSailor-32B. only conducts short-horizon search, and does not show an advanced capability of utilizing more tool calls for rigorous conclusion verification.
When employing test-tims augmentation approaches, ASearcher significantly outperforms WebSailor-32B. We hypothesize that this is because WebSailor does not produce a compact analysis, and, therefore, the most reliable answer could not be easily determined from set noisy trajectorics.
Method
GAIA
xBench
ASearcher (Ours) + Summary w. DeepSeek-VV V3 & Test-time Search (K 16)
58.7 7
51.1
71.8
75.0
WebSailor-32B
53.2
53.3
Summary w. DeepSeck-V3 & Test-time Search (K 16)
60.2
64.0
Table 5: WebSailor-32B with Test-Time Augmentations
Method
GAIA
xBench
ASearcher (Ours)
13.81
13.67
WebSailor-32B
4.24
5.17
Table 6: Number of Tool Calls ASearcher & WebSailor-32B
22


Reviews:
[{"review_id": "hopb0p8RlR", "reviewer": "Reviewer_1mWo", "summary": "This work proposed a strong training method where the performance gain comes mostly from an improved tool-use ability of an open-sourced model. ASearcher is able to process long trajectories and the work has developed efficient computing infra to leverage the computing resources. The performance of ASearcher is compared comprehensively on prevailing benchmark against a comprehensive set of baseline.", "questions": "NA", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Clear presentation and writing\n- Comprehensive baselines \n- Convincing results\n- Effective and efficient approach", "weaknesses": "I don't have too much complaints about this work. The novelty of this work is moderate (in terms of using standard GRPO with fully async training) while I acknowledge the experimental efforts. I would suggest to open-source everything including the data filtering and all the components in the training. I have one main concern tho:\n\nThere are some components in the introduced method such as Dynamic Filtering and data filtering etc, e.g.,\n```\n Finally, we filter out questions that are too hard for the modelor too easy for the model. Finally, from a total of 304k QA pairs, we retain 16k challenging samples\n```\nMaybe I missed, but I don't see ablation studies about them.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hU6Pk7nMbl", "reviewer": "Reviewer_HJhQ", "summary": "This paper presents ASearcher, a single-model search agent trained with large-scale reinforcement learning , demonstrating long-horizon reasoning and competitive performance.", "questions": "Please refer to the weaknesses section", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper is generally well-organized.\n2. The paper involves a substantial amount of engineering work, including large-scale RL training, data synthesis. \n3. Demonstrates impressive long-horizon reasoning performance.", "weaknesses": "1. While the paper emphasizes end-to-end RL for long-horizon search, similar ideas have already been explored by Search-R1, WebThinker, websailor, and Chain-of-Agents, which also use RL or multi-agent training to enhance tool use. The main difference here fully asynchronous RL is more of a systems optimization than a conceptual breakthrough in agent reasoning or policy learning. While technically well-executed, the contribution is rather straightforward and does not introduce substantial new insights\n\n2.  The reported 16k H800 GPU hours make the approach extremely resource-intensive and difficult to reproduce, especially for academic or smaller research groups, which greatly limits its practical contribution and accessibility to the broader community.\n\n3. The paper’s contribution, scope, and novelty are quite unclear. It reads more like a well-executed technical report or open-source project than a research paper with distinct conceptual advances. The work mainly combines and scales up ideas that are already known in the community. Moreover, the performance gains might largely reflect the inherent capability of QwQ-32B and the heavy use of synthetic QA generation and data filtering, rather than the RL algorithm itself. \n\n4. It is also unclear what training setup was used for the baselines—did they use the same data, or were they retrained? If the goal is to compare RL algorithms, the training data and environment should be kept identical; if the goal is to emphasize data synthesis or agentic setup, experiments with smaller models would help validate that claim. \n\n5. In addition, I did not see any ablation studies analyzing the contribution of each component (e.g., data quality, RL updates), which makes it difficult to attribute where the actual improvements come from. \n\n6. The entire pipeline feels more like a collection of separately optimized components rather than a truly “end-to-end” system", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8WItLDsilV", "reviewer": "Reviewer_yr7v", "summary": "In this paper, the authors propose an LLM-based search agent named ASearcher, which is built from an open-source model (QwQ-32B). It is trained purely using end-to-end Reinforcement Learning (RL). Unlike many existing approaches, this work avoids any reliance on commercial LLM APIs for generating supervised fine-tuning data or for serving as specialized sub-modules. The paper provides a novel data synthesis agent along with a large-scale, long-horizon RL training framework. The proposed ASearcher agent achieves state-of-the-art performance among 32B-scale open-source agents on benchmarks like GAIA, xBench, and Frames.", "questions": "Please check the issues and questions mentioned in the \"Weaknesses\" section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. In general, the paper is well written with good structure and is easy to follow.\n\n2. In this paper, the authors are trying to tackle an important and challenging task. Training agents that perform deep, long-horizon web search is timely and practically useful. The paper frames these challenges very well and clearly.\n\n3. The proposed method demonstrates that a single open-model (QwQ-32B) trained purely with RL, which has no commercial LLMs for data or submodules, can learn sophisticated multi-turn search strategies is a meaningful contribution to the RL-for-agents literature. The result challenges the current trend of heavily relying on multi-model/commercial LLM pipelines.\n\n4. The injection plus fuzzing loop, combined with verification checks, is a practical method to produce hard, tool-use-requiring QA pairs. The authors also provide useful statistics and examples in the appendix.", "weaknesses": "1. It would be better if more ablation studies could be provided. The paper presents two major contributions: the novel QA synthesis pipeline and the long-horizon asynchronous RL system. However, their relative importance is not disentangled.\n\n2. The data synthesis agent uses external sources to support facts and then rejects questions solvable without tools. But more detail is expected: what external sources were used (Wikipedia only?), how is answer uniqueness verified, and what mechanisms prevent the synthetic QA from leaking evaluation set contents or being too close to benchmark questions?\n\n3. The proposed system relies on a very large language model (QwQ-32B) and extensive GPU hours (16k H800 GPU-hours), which is far more expensive than simply using commercial APIs for data generation or inference.\n\n4. I wonder if the proposed method can perform well on the other common base models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r5idQo8GAi", "reviewer": "Reviewer_JUpp", "summary": "This paper introduces ASearch, a single-model search agent purely trained by RL without using commercial APIs for data or tools. Firstly, a data synthesis agent is used to apply question injection and fuzzing to open-source datasets, increasing the complexity of training data, followed by three steps of quality verification. Secondly, a two-stage curriculum RL training is employed. The paper further accelerates the training process by asynchronizing both trajectory rollout and training. Experiments show that ASearch can achieve better performance compared to many strong baselines by using only an open-source 32B model.", "questions": "1. Many prompts used in this paper have not been disclosed, such as the prompts for question injection and fuzzing, quality verification, LLM-as-judge, and others.\n\n2. How would it impact if all baselines were allowed a maximum of 128 rounds of tool calls?\n\n3. How would the effects be if other baselines used Summary or TTS? How is TTS specifically implemented - based on self-consistency or pass@K?\n\n4. The reasons behind this paper's ability to surpass baselines are not very clear. Why can using a 32B model to synthesize QA and training with zero-RL achieve better results than other baselines using stronger models to synthesize trajectories with a cold start + RL approach?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "ASearch can achieve excellent results without relying on API model-synthesized trajectories, and the training process is fully RL without cold-start. The proposed asynchronous rollout and training strategy can effectively enhance training efficiency, enabling the model to make more tool calls.", "weaknesses": "1. There are several crucial details in the method that are unclear. For example: 1) How is the uniqueness of answers determined during the quality verification process? How does the model know if an answer is unique without executing an action? 2) How are the answers for the synthesized new questions determined? 3) How is question injection and fuzzing implemented, and what prompts are used?\n\n2. The scalability of the method in this paper is questionable. Although the paper does not rely on commercial API models, QwQ-32B already possesses sufficient reasoning capabilities compared with most open-source models. If replaced with models at the 7B or 14B level, can the pipeline in this paper successfully boost the model's performance? Furthermore, the data synthesis in this paper relies on existing open-source datasets. The community is more concerned about how to construct a large number of high-quality and complex query-answer pairs without relying on open-source data.\n\n3. There are risks regarding the fairness of the baseline comparisons in Table 1. For example, do the various baselines use the same training data? Additionally, this paper allows a maximum of 128 tool calls, far more than other baselines. Does this setting itself bring significant performance gains?\n\n4. Furthermore, the significant performance improvement achieved through zero RL in this paper may be related to QwQ being a reasoning model. Would switching to a non-reasoning model imply the need for a cold start?\n\n5. Most of the analytical experiments in this paper are case studies and lack strong persuasiveness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)