AceSearcher: Bootstrapping Reasoning and Search for LLMs via Reinforced
  Self-Play
Authors: name: Ran Xu, name: Yuchen Zhuang, name: Zihan Dong, name: Jonathan Wang, name: Yue Yu, name: Joyce C. Ho, name: Linjun Zhang, name: Haoyu Wang, name: Wenqi Shi, name: Carl Yang

Search-augmented LLMs often struggle with complex reasoning tasks due to
ineffective multi-hop retrieval and limited reasoning ability. We propose
AceSearcher, a cooperative self-play framework that trains a single large
language model (LLM) to alternate between two roles: a decomposer that breaks
down complex queries and a solver that integrates retrieved contexts for answer
generation. AceSearcher couples supervised fine-tuning on a diverse mixture of
search, reasoning, and decomposition tasks with reinforcement fine-tuning
optimized for final answer accuracy, eliminating the need for intermediate
annotations. Extensive experiments on three reasoning-intensive tasks across 10
datasets show that AceSearcher outperforms state-of-the-art baselines,
achieving an average exact match improvement of 7.6%. Remarkably, on
document-level finance reasoning tasks, AceSearcher-32B matches the performance
of the DeepSeek-V3 model using less than 5% of its parameters. Even at smaller
scales (1.5B and 8B), AceSearcher often surpasses existing search-augmented
LLMs with up to 9x more parameters, highlighting its exceptional efficiency and
effectiveness in tackling complex reasoning tasks. Our code will be published
at https://github.com/ritaranx/AceSearcher and
https://huggingface.co/AceSearcher.