Lean and Mean: Decoupled Value Policy Optimization with Global Value
  Guidance
Authors: name: Chenghua Huang; name: Lu Wang; name: Fangkai Yang; name: Pu Zhao; name: Zhixu Li; name: Qingwei Lin; name: Dongmei Zhang; name: Saravan Rajmohan; name: Qi Zhang
Published: 2025-02-24 08:11:33

Abstract: Proximal Policy Optimization (PPO)-based Reinforcement Learning from Human
Feedback (RLHF) is essential for aligning large language models (LLMs) with
human preferences. It requires joint training of an actor and critic with a
pretrained, fixed reward model for guidance. This approach increases
computational complexity and instability due to actor-critic interdependence.
Additionally, PPO lacks access to true environment rewards in LLM tasks,
limiting its adaptability. Under such conditions, pretraining a value model or
a reward model becomes equivalent, as both provide fixed supervisory signals
without new ground-truth feedback. To address these issues, we propose
Decoupled Value Policy Optimization (DVPO), a lean framework that
replaces traditional reward modeling with a pretrained global value model
(GVM). The GVM is conditioned on policy trajectories and predicts token-level
return-to-go estimates. By decoupling value model from policy training (via
frozen GVM-driven RL objectives), DVPO eliminates actor-critic interdependence,
reducing GPU memory usage by 40\% and training time by 35\% compared to
conventional RLHF. Experiments across benchmarks show DVPO outperforms
efficient RLHF methods (e.g., DPO) while matching state-of-the-art PPO in
performance.