AceSearcher: Bootstrapping Reasoning and Search for LLMs via Reinforced
  Self-Play
Authors: name: Ran Xu; name: Yuchen Zhuang; name: Zihan Dong; name: Jonathan Wang; name: Yue Yu; name: Joyce C. Ho; name: Linjun Zhang; name: Haoyu Wang; name: Wenqi Shi; name: Carl Yang
Published: 2025-09-29 02:14:30

Abstract: Search-augmented LLMs often struggle with complex reasoning tasks due to
ineffective multi-hop retrieval and limited reasoning ability. We propose
AceSearcher, a cooperative self-play framework that trains a single large
language model (LLM) to alternate between two roles: a decomposer that breaks
down complex queries and a solver that integrates retrieved contexts for answer
generation. AceSearcher couples supervised fine-tuning on a diverse mixture of
search, reasoning, and decomposition tasks with reinforcement fine-tuning
optimized for final answer accuracy, eliminating the need for intermediate
annotations. Extensive experiments on three reasoning-intensive tasks across 10
datasets show that AceSearcher outperforms state-of-the-art baselines,
achieving an average exact match improvement of 7.6%. Remarkably, on
document-level finance reasoning tasks, AceSearcher-32B matches the performance
of the DeepSeek-V3 model using less than 5% of its parameters. Even at smaller
scales (1.5B and 8B), AceSearcher often surpasses existing search-augmented
LLMs with up to 9x more parameters, highlighting its exceptional efficiency and
effectiveness in tackling complex reasoning tasks. Our code will be published
at https://github.com/ritaranx/AceSearcher and
https://huggingface.co/AceSearcher.