R1-Searcher++: Incentivizing the Dynamic Knowledge Acquisition of LLMs
  via Reinforcement Learning
Authors: name: Huatong Song, name: Jinhao Jiang, name: Wenqing Tian, name: Zhipeng Chen, name: Yuhuan Wu, name: Jiahao Zhao, name: Yingqian Min, name: Wayne Xin Zhao, name: Lei Fang, name: Ji-Rong Wen

Large Language Models (LLMs) are powerful but prone to hallucinations due to
static knowledge. Retrieval-Augmented Generation (RAG) helps by injecting
external information, but current methods often are costly, generalize poorly,
or ignore the internal knowledge of the model. In this paper, we introduce
R1-Searcher++, a novel framework designed to train LLMs to adaptively leverage
both internal and external knowledge sources. R1-Searcher++ employs a two-stage
training strategy: an initial SFT Cold-start phase for preliminary format
learning, followed by RL for Dynamic Knowledge Acquisition. The RL stage uses
outcome-supervision to encourage exploration, incorporates a reward mechanism
for internal knowledge utilization, and integrates a memorization mechanism to
continuously assimilate retrieved information, thereby enriching the model's
internal knowledge. By leveraging internal knowledge and external search
engine, the model continuously improves its capabilities, enabling efficient
retrieval-augmented reasoning. Our experiments demonstrate that R1-Searcher++
outperforms previous RAG and reasoning methods and achieves efficient
retrieval. The code is available at
https://github.com/RUCAIBox/R1-Searcher-plus.