Reward-Robust RLHF in LLMs
Authors: name: Yuzi Yan; name: Xingzhou Lou; name: Jialian Li; name: Yiping Zhang; name: Jian Xie; name: Chao Yu; name: Yu Wang; name: Dong Yan; name: Yuan Shen
Published: 2024-09-18 02:35:41

Abstract: As Large Language Models (LLMs) continue to progress toward more advanced
forms of intelligence, Reinforcement Learning from Human Feedback (RLHF) is
increasingly seen as a key pathway toward achieving Artificial General
Intelligence (AGI). However, the reliance on reward-model-based (RM-based)
alignment methods introduces significant challenges due to the inherent
instability and imperfections of Reward Models (RMs), which can lead to
critical issues such as reward hacking and misalignment with human intentions.
In this paper, we introduce a reward-robust RLHF framework aimed at addressing
these fundamental challenges, paving the way for more reliable and resilient
learning in LLMs. Our approach introduces a novel optimization objective that
carefully balances performance and robustness by incorporating Bayesian Reward
Model Ensembles (BRME) to model the uncertainty set of reward functions. This
allows the framework to integrate both nominal performance and minimum reward
signals, ensuring more stable learning even with imperfect reward models.
Empirical results demonstrate that our framework consistently outperforms
traditional RLHF across diverse benchmarks, showing improved accuracy and
long-term stability. We also provide a theoretical analysis, demonstrating that
reward-robust RLHF approaches the stability of constant reward settings, which
proves to be effective in a stochastic-case analysis. Together, these
contributions highlight the framework potential to enhance both the performance
and stability of LLM alignment with RLHF.