Scaling Laws for Reward Model Overoptimization in Direct Alignment
  Algorithms
Authors: name: Rafael Rafailov; name: Yaswanth Chittepu; name: Ryan Park; name: Harshit Sikchi; name: Joey Hejna; name: Bradley Knox; name: Chelsea Finn; name: Scott Niekum
Published: 2024-06-05 03:41:37

Abstract: Reinforcement Learning from Human Feedback (RLHF) has been crucial to the
recent success of Large Language Models (LLMs), however, it is often a complex
and brittle process. In the classical RLHF framework, a reward model is first
trained to represent human preferences, which is in turn used by an online
reinforcement learning (RL) algorithm to optimize the LLM. A prominent issue
with such methods is reward over-optimization or reward hacking,
where performance as measured by the learned proxy reward model increases, but
true quality plateaus or even deteriorates. Direct Alignment Algorithms (DDAs)
like Direct Preference Optimization have emerged as alternatives to the
classical RLHF pipeline by circumventing the reward modeling phase. However,
although DAAs do not use a separate proxy reward model, they still commonly
deteriorate from over-optimization. While the so-called reward hacking
phenomenon is not well-defined for DAAs, we still uncover similar trends: at
higher KL budgets, DAA algorithms exhibit similar degradation patterns to their
classic RLHF counterparts. In particular, we find that DAA methods deteriorate
not only across a wide range of KL budgets but also often before even a single
epoch of the dataset is completed. Through extensive empirical experimentation,
this work formulates and formalizes the reward over-optimization or hacking
problem for DAAs and explores its consequences across objectives, training
regimes, and model scales.