Lean and Mean: Decoupled Value Policy Optimization with Global Value
  Guidance
Authors: name: Chenghua Huang, name: Lu Wang, name: Fangkai Yang, name: Pu Zhao, name: Zhixu Li, name: Qingwei Lin, name: Dongmei Zhang, name: Saravan Rajmohan, name: Qi Zhang

Proximal Policy Optimization (PPO)-based Reinforcement Learning from Human
Feedback (RLHF) is essential for aligning large language models (LLMs) with
human preferences. It requires joint training of an actor and critic with a
pretrained, fixed reward model for guidance. This approach increases
computational complexity and instability due to actor-critic interdependence.
Additionally, PPO lacks access to true environment rewards in LLM tasks,
limiting its adaptability. Under such conditions, pretraining a value model or
a reward model becomes equivalent, as both provide fixed supervisory signals
without new ground-truth feedback. To address these issues, we propose
Decoupled Value Policy Optimization (DVPO), a lean framework that
replaces traditional reward modeling with a pretrained global value model
(GVM). The GVM is conditioned on policy trajectories and predicts token-level
return-to-go estimates. By decoupling value model from policy training (via
frozen GVM-driven RL objectives), DVPO eliminates actor-critic interdependence,
reducing GPU memory usage by 40\% and training time by 35\% compared to
conventional RLHF. Experiments across benchmarks show DVPO outperforms
efficient RLHF methods (e.g., DPO) while matching state-of-the-art PPO in
performance.