PVPO: Pre-Estimated Value-Based Policy Optimization for Agentic
  Reasoning
Authors: name: Wenfeng Feng, name: Penghong Zhao, name: Guochao Jiang, name: Chuzhan Hao, name: Yuewei Zhang, name: Hao Wang
Organization: Apsara Stack MaaS

Critic-free reinforcement learning methods, particularly group policies, have
attracted considerable attention for their efficiency in complex tasks.
However, these methods rely heavily on multiple sampling and comparisons within
the policy to estimate advantage, which may cause the policy to fall into local
optimum and increase computational cost. To address these issues, we propose
PVPO, an efficient reinforcement learning method enhanced by an advantage
reference anchor and data pre-sampling. Specifically, we use the reference
model to rollout in advance and employ the calculated reward score as a
reference anchor. Our approach effectively corrects the cumulative bias
introduced by intra-group comparisons and significantly reduces reliance on the
number of rollouts during training. Meanwhile, the reference model can assess
sample difficulty during data pre-sampling, enabling effective selection of
high-gain data to improve training efficiency. Moreover, PVPO is orthogonal to
other advanced critic-free RL algorithms, making it compatible with and
complementary to these methods. Experiments conducted on nine datasets across
two domains demonstrate that PVPO achieves State-Of-The-Art (SOTA) performance.
Our approach not only demonstrates robust generalization across multiple tasks,
but also exhibits scalable performance across models of varying scales.