OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework
Authors: name: Jian Hu; name: Xibin Wu; name: Weixun Wang; name: Xianyu; name: Dehao Zhang; name: Yu Cao
Published: 2024-05-20 01:04:40

Abstract: Large Language Models (LLMs) fine-tuned via Reinforcement Learning from Human
Feedback (RLHF) and Reinforcement Learning with Verifiable Rewards (RLVR)
significantly improve the alignment of human-AI values and further raise the
upper bound of AI capabilities, particularly in reasoning-intensive,
long-context Chain-of-Thought (long-CoT) tasks. However, existing RLHF (or
RLVR) frameworks commonly face challenges such as inference bottlenecks and
complexity barriers, restricting their accessibility for newcomers. To bridge
this gap, we introduce OpenRLHF, a user-friendly, scalable, and easy-to-learn
open-source RLHF framework built upon Ray, vLLM, DeepSpeed, and HuggingFace
Transformers, featuring a simplified design, clear code structure, and
comprehensive documentation to facilitate entry for researchers and
practitioners. Experimental results show that OpenRLHF achieves superior
training efficiency with speedups ranging from 1.22x to 1.68x across different
model sizes compared to state-of-the-art frameworks, while requiring
significantly fewer lines of code for implementation. OpenRLHF is publicly
available at https://github.com/OpenRLHF/OpenRLHF, and has already been adopted
by leading institutions to accelerate RLHF research and learning.