Search and Refine During Think: Autonomous Retrieval-Augmented Reasoning
  of LLMs
Authors: name: Yaorui Shi; name: Sihang Li; name: Chang Wu; name: Zhiyuan Liu; name: Junfeng Fang; name: Hengxing Cai; name: An Zhang; name: Xiang Wang
Published: 2025-05-16 14:11:29

Abstract: Large language models have demonstrated impressive reasoning capabilities but
are inherently limited by their knowledge reservoir. Retrieval-augmented
reasoning mitigates this limitation by allowing LLMs to query external
resources, but existing methods often retrieve irrelevant or noisy information,
hindering accurate reasoning. In this paper, we propose AutoRefine, a
reinforcement learning post-training framework that adopts a new
``search-and-refine-during-think'' paradigm. AutoRefine introduces explicit
knowledge refinement steps between successive search calls, enabling the model
to iteratively filter, distill, and organize evidence before generating an
answer. Furthermore, we incorporate tailored retrieval-specific rewards
alongside answer correctness rewards using group relative policy optimization.
Experiments on single-hop and multi-hop QA benchmarks demonstrate that
AutoRefine significantly outperforms existing approaches, particularly in
complex, multi-hop reasoning scenarios. Detailed analysis shows that AutoRefine
issues frequent, higher-quality searches and synthesizes evidence effectively.