RLAIF: Scaling Reinforcement Learning from Human Feedback with AI
  Feedback
Authors: name: Harrison Lee; name: Samrat Phatale; name: Hassan Mansoor; name: Kellie Lu; name: Thomas Mesnard; name: Colton Bishop; name: Victor Carbune; name: Abhinav Rastogi
Published: 2023-09-01 05:53:33

Abstract: Reinforcement learning from human feedback (RLHF) is effective at aligning
large language models (LLMs) to human preferences, but gathering high quality
human preference labels is a key bottleneck. We conduct a head-to-head
comparison of RLHF vs. RL from AI Feedback (RLAIF) - a technique where
preferences are labeled by an off-the-shelf LLM in lieu of humans, and we find
that they result in similar improvements. On the task of summarization, human
evaluators prefer generations from both RLAIF and RLHF over a baseline
supervised fine-tuned model in ~70% of cases. Furthermore, when asked to rate
RLAIF vs. RLHF summaries, humans prefer both at equal rates. These results
suggest that RLAIF can yield human-level performance, offering a potential
solution to the scalability limitations of RLHF.