RLCD: Reinforcement Learning from Contrast Distillation for Language
  Model Alignment
Authors: name: Kevin Yang; name: Dan Klein; name: Asli Celikyilmaz; name: Nanyun Peng; name: Yuandong Tian
Published: 2023-07-24 17:23:22

Abstract: We propose Reinforcement Learning from Contrast Distillation (RLCD), a method
for aligning language models to follow natural language principles without
using human feedback. RLCD trains a preference model using simulated preference
pairs that contain both a high-quality and low-quality example, generated using
contrasting positive and negative prompts. The preference model is then used to
improve a base unaligned language model via reinforcement learning.
Empirically, RLCD outperforms RLAIF (Bai et al., 2022b) and context
distillation (Huang et al., 2022) baselines across three diverse alignment
tasks--harmlessness, helpfulness, and story outline generation--and on both 7B
and 30B model scales for preference data simulation.