Offline Regularised Reinforcement Learning for Large Language Models
  Alignment
Authors: name: Pierre Harvey Richemond, name: Yunhao Tang, name: Daniel Guo, name: Daniele Calandriello, name: Mohammad Gheshlaghi Azar, name: Rafael Rafailov, name: Bernardo Avila Pires, name: Eugene Tarassov, name: Lucas Spangher, name: Will Ellsworth, name: Aliaksei Severyn, name: Jonathan Mallinson, name: Lior Shani, name: Gil Shamir, name: Rishabh Joshi, name: Tianqi Liu, name: Remi Munos, name: Bilal Piot

The dominant framework for alignment of large language models (LLM), whether
through reinforcement learning from human feedback or direct preference
optimisation, is to learn from preference data. This involves building datasets
where each element is a quadruplet composed of a prompt, two independent
responses (completions of the prompt) and a human preference between the two
independent responses, yielding a preferred and a dis-preferred response. Such
data is typically scarce and expensive to collect. On the other hand,
single-trajectory datasets where each element is a triplet composed of a
prompt, a response and a human feedback is naturally more abundant. The
canonical element of such datasets is for instance an LLM's response to a
user's prompt followed by a user's feedback such as a thumbs-up/down.
Consequently, in this work, we propose DRO, or Direct Reward
Optimisation, as a framework and associated algorithms that do not require
pairwise preferences. DRO uses a simple mean-squared objective that can be
implemented in various ways. We validate our findings empirically, using T5
encoder-decoder language models, and show DRO's performance over selected
baselines such as Kahneman-Tversky Optimization (KTO). Thus, we confirm that
DRO is a simple and empirically compelling method for single-trajectory policy
optimisation.