Agentic Reinforced Policy Optimization
Authors: name: Guanting Dong, name: Hangyu Mao, name: Kai Ma, name: Licheng Bao, name: Yifei Chen, name: Zhongyuan Wang, name: Zhongxia Chen, name: Jiazhen Du, name: Huiyang Wang, name: Fuzheng Zhang, name: Guorui Zhou, name: Yutao Zhu, name: Ji-Rong Wen, name: Zhicheng Dou

Large-scale reinforcement learning with verifiable rewards (RLVR) has
demonstrated its effectiveness in harnessing the potential of large language
models (LLMs) for single-turn reasoning tasks. In realistic reasoning
scenarios, LLMs can often utilize external tools to assist in task-solving
processes. However, current RL algorithms inadequately balance the models'
intrinsic long-horizon reasoning capabilities and their proficiency in
multi-turn tool interactions. To bridge this gap, we propose Agentic Reinforced
Policy Optimization (ARPO), a novel agentic RL algorithm tailored for training
multi-turn LLM-based agents. Through preliminary experiments, we observe that
LLMs tend to exhibit highly uncertain behavior, characterized by an increase in
the entropy distribution of generated tokens, immediately following
interactions with external tools. Motivated by this observation, ARPO
incorporates an entropy-based adaptive rollout mechanism, dynamically balancing
global trajectory sampling and step-level sampling, thereby promoting
exploration at steps with high uncertainty after tool usage. By integrating an
advantage attribution estimation, ARPO enables LLMs to internalize advantage
differences in stepwise tool-use interactions. Our experiments across 13
challenging benchmarks in computational reasoning, knowledge reasoning, and
deep search domains demonstrate ARPO's superiority over trajectory-level RL
algorithms. Remarkably, ARPO achieves improved performance using only half of
the tool-use budget required by existing methods, offering a scalable solution
for aligning LLM-based agents with real-time dynamic environments. Our code and
datasets are released at https://github.com/dongguanting/ARPO