Beyond Ten Turns: Unlocking Long-Horizon Agentic Search with Large-Scale
  Asynchronous RL
Authors: name: Jiaxuan Gao, name: Wei Fu, name: Minyang Xie, name: Shusheng Xu, name: Chuyi He, name: Zhiyu Mei, name: Banghua Zhu, name: Yi Wu

Recent advancements in LLM-based agents have demonstrated remarkable
capabilities in handling complex, knowledge-intensive tasks by integrating
external tools. Among diverse choices of tools, search tools play a pivotal
role in accessing vast external knowledge. However, open-source agents still
fall short of achieving expert-level Search Intelligence, the ability to
resolve ambiguous queries, generate precise searches, analyze results, and
conduct thorough exploration. Existing approaches fall short in scalability,
efficiency, and data quality. For example, small turn limits in existing online
RL methods, e.g. <=10, restrict complex strategy learning. This paper
introduces ASearcher, an open-source project for large-scale RL training of
search agents. Our key contributions include: (1) Scalable fully asynchronous
RL training that enables long-horizon search while maintaining high training
efficiency. (2) A prompt-based LLM agent that autonomously synthesizes
high-quality and challenging QAs, creating a large-scale QA dataset. Through RL
training, our prompt-based QwQ-32B agent achieves substantial improvements,
with 46.7% and 20.8% Avg@4 gains on xBench and GAIA, respectively. Notably, our
agent exhibits extreme long-horizon search, with tool calls exceeding 40 turns
and output tokens exceeding 150k during training time. With a simple agent
design and no external LLMs, ASearcher-Web-QwQ achieves Avg@4 scores of 42.1 on
xBench and 52.8 on GAIA, surpassing existing open-source 32B agents. We
open-source our models, training data, and codes in
https://github.com/inclusionAI/ASearcher.