Secrets of RLHF in Large Language Models Part I: PPO
Authors: name: Rui Zheng, name: Shihan Dou, name: Songyang Gao, name: Wei Shen, name: Binghai Wang, name: Yan Liu, name: Senjie Jin, name: Qin Liu, name: Limao Xiong, name: Lu Chen, name: Zhiheng Xi, name: Yuhao Zhou, name: Nuo Xu, name: Wenbin Lai, name: Minghao Zhu, name: Rongxiang Weng, name: Wensen Cheng, name: Cheng Chang, name: Zhangyue Yin, name: Yuan Hua, name: Haoran Huang, name: Tianxiang Sun, name: Hang Yan, name: Tao Gui, name: Qi Zhang, name: Xipeng Qiu, name: Xuanjing Huang

Large language models (LLMs) have formulated a blueprint for the advancement
of artificial general intelligence. Its primary objective is to function as a
human-centric (helpful, honest, and harmless) assistant. Alignment with humans
assumes paramount significance, and reinforcement learning with human feedback
(RLHF) emerges as the pivotal technological paradigm underpinning this pursuit.
Current technical routes usually include reward models to measure
human preferences, Proximal Policy Optimization (PPO) to optimize
policy model outputs, and process supervision to improve step-by-step
reasoning capabilities. However, due to the challenges of reward design,
environment interaction, and agent training, coupled with huge trial and error
cost of large language models, there is a significant barrier for AI
researchers to motivate the development of technical alignment and safe landing
of LLMs. The stable training of RLHF has still been a puzzle. In the first
report, we dissect the framework of RLHF, re-evaluate the inner workings of
PPO, and explore how the parts comprising PPO algorithms impact policy agent
training. We identify policy constraints being the key factor for the effective
implementation of the PPO algorithm. Therefore, we explore the PPO-max, an
advanced version of PPO algorithm, to efficiently improve the training
stability of the policy model. Based on our main results, we perform a
comprehensive analysis of RLHF abilities compared with SFT models and ChatGPT.
The absence of open-source implementations has posed significant challenges to
the investigation of LLMs alignment. Therefore, we are eager to release
technical reports, reward models and PPO codes