SSRL: Self-Search Reinforcement Learning
Authors: name: Yuchen Fan; name: Kaiyan Zhang; name: Heng Zhou; name: Yuxin Zuo; name: Yanxu Chen; name: Yu Fu; name: Xinwei Long; name: Xuekai Zhu; name: Che Jiang; name: Yuchen Zhang; name: Li Kang; name: Gang Chen; name: Cheng Huang; name: Zhizhou He; name: Bingning Wang; name: Lei Bai; name: Ning Ding; name: Bowen Zhou
Published: 2025-08-14 17:46:01

Abstract: We investigate the potential of large language models (LLMs) to serve as
efficient simulators for agentic search tasks in reinforcement learning (RL),
thereby reducing dependence on costly interactions with external search
engines. To this end, we first quantify the intrinsic search capability of LLMs
via structured prompting and repeated sampling, which we term Self-Search. Our
results reveal that LLMs exhibit strong scaling behavior with respect to the
inference budget, achieving high pass@k on question-answering benchmarks,
including the challenging BrowseComp task. Building on these observations, we
introduce Self-Search RL (SSRL), which enhances LLMs' Self-Search capability
through format-based and rule-based rewards. SSRL enables models to iteratively
refine their knowledge utilization internally, without requiring access to
external tools. Empirical evaluations demonstrate that SSRL-trained policy
models provide a cost-effective and stable environment for search-driven RL
training, reducing reliance on external search engines and facilitating robust
sim-to-real transfer. We draw the following conclusions: 1) LLMs possess world
knowledge that can be effectively elicited to achieve high performance; 2) SSRL
demonstrates the potential of leveraging internal knowledge to reduce
hallucination; 3) SSRL-trained models integrate seamlessly with external search
engines without additional effort. Our findings highlight the potential of LLMs
to support more scalable RL agent training.