VAPO: Efficient and Reliable Reinforcement Learning for Advanced
  Reasoning Tasks
Authors: name: YuYue, name: Yufeng Yuan, name: Qiying Yu, name: Xiaochen Zuo, name: Ruofei Zhu, name: Wenyuan Xu, name: Jiaze Chen, name: Chengyi Wang, name: TianTian Fan, name: Zhengyin Du, name: Xiangpeng Wei, name: Gaohong Liu, name: Juncai Liu, name: Lingjun Liu, name: Haibin Lin, name: Zhiqi Lin, name: Bole Ma, name: Chi Zhang, name: Mofan Zhang, name: Wang Zhang, name: Hang Zhu, name: Ru Zhang, name: Xin Liu, name: Mingxuan Wang, name: Yonghui Wu, name: Lin Yan

We present VAPO, Value-based Augmented Proximal Policy Optimization framework
for reasoning models., a novel framework tailored for reasoning models within
the value-based paradigm. Benchmarked the AIME 2024 dataset, VAPO, built on the
Qwen 32B pre-trained model, attains a state-of-the-art score of
60.4. In direct comparison under identical experimental settings,
VAPO outperforms the previously reported results of DeepSeek-R1-Zero-Qwen-32B
and DAPO by more than 10 points. The training process of VAPO stands out for
its stability and efficiency. It reaches state-of-the-art performance within a
mere 5,000 steps. Moreover, across multiple independent runs, no training
crashes occur, underscoring its reliability. This research delves into long
chain-of-thought (long-CoT) reasoning using a value-based reinforcement
learning framework. We pinpoint three key challenges that plague value-based
methods: value model bias, the presence of heterogeneous sequence lengths, and
the sparsity of reward signals. Through systematic design, VAPO offers an
integrated solution that effectively alleviates these challenges, enabling
enhanced performance in long-CoT reasoning tasks.