Understanding the performance gap between online and offline alignment
  algorithms
Authors: name: Yunhao Tang; name: Daniel Zhaohan Guo; name: Zeyu Zheng; name: Daniele Calandriello; name: Yuan Cao; name: Eugene Tarassov; name: Rémi Munos; name: Bernardo Ávila Pires; name: Michal Valko; name: Yong Cheng; name: Will Dabney
Published: 2024-05-14 09:12:30

Abstract: Reinforcement learning from human feedback (RLHF) is the canonical framework
for large language model alignment. However, rising popularity in offline
alignment algorithms challenge the need for on-policy sampling in RLHF. Within
the context of reward over-optimization, we start with an opening set of
experiments that demonstrate the clear advantage of online methods over offline
methods. This prompts us to investigate the causes to the performance
discrepancy through a series of carefully designed experimental ablations. We
show empirically that hypotheses such as offline data coverage and data quality
by itself cannot convincingly explain the performance difference. We also find
that while offline algorithms train policy to become good at pairwise
classification, it is worse at generations; in the meantime the policies
trained by online algorithms are good at generations while worse at pairwise
classification. This hints at a unique interplay between discriminative and
generative capabilities, which is greatly impacted by the sampling process.
Lastly, we observe that the performance discrepancy persists for both
contrastive and non-contrastive loss functions, and appears not to be addressed
by simply scaling up policy networks. Taken together, our study sheds light on
the pivotal role of on-policy sampling in AI alignment, and hints at certain
fundamental challenges of offline alignment algorithms.