Please explain like I am a layperson: 


Source: https://arxiv.org/abs/2305.18290
Direct Preference Optimization (DPO)
A Simplified Explanation
João Lages
João Lages


In May 2023, this algorithm was introduced in Direct Preference Optimization: Your Language Model is Secretly a Reward Model.

DPO comes as a direct alternative to Reinforcement Learning from Human Feedback (RLHF) and has been gaining a lot of popularity since it does not require a reward model. It was recently used in Zephyr, the best 7B language model at the time of writing.

This method is applied to generative language models (LMs), such as GPT, Llama, Zephyr, and T5.

The objective of DPO is the same as RLHF: improve the alignment of language models to human preferences.

Preference Data
DPO is applied to preference data, which basically consists of a dataset of triplets (prompt, chosen answer, rejected answer).

In other words, for each prompt, there is a better response and a worse response.


Preference data collection in ChatGPT (Image by author)
This type of data is also collected for RLHF, with the objective to train a reward model, that is later used to train the model with reinforcement learning.

In DPO, there is no reinforcement learning, and the model is directly optimized in this preference data.

Also interesting to note, unlike RLHF, these responses do not have to be sampled from the language model that we are optimizing.

DPO Fine-tuning

Fine-tuning the main language model (LM) using DPO (Image by author)
At the beginning of the fine-tuning process, an exact copy is done for the language model (LM) that is being trained, and its trainable parameters are frozen.

For each datapoint, the chosen and rejected responses are scored by both the trained and the frozen language model. This score is the product of the probabilities associated with the desired response token for each step.

Since these generative language models use a causal decoder, we can calculate this score in a single forward pass, as seen in the image below.


How the language model (LM) scores a chosen/rejected response for a given prompt: for each generation step, the probability of generating the response token is selected, and these probabilities are multiplied at the end (Image by author)
With the chosen and rejected responses scored, we can calculate the ratio between the scores given by the trained language model, Rpolicy, and the ones given by the frozen language model, Rreference.

These ratios are then used to calculate the final loss that is used to modify the model weights in the gradient descent update:


DPO loss (Image by author)
where β is a hyperparameter (Zephyr used β=0.1) and σ is the sigmoid function.

Note: you may see this loss equation written in log-scale, which is the recommended way to implement it, for numerical stability.

Conclusions
Direct Preference Optimization is a stable, performant, and computationally lightweight algorithm.

Unlike its predecessor, RLHF, DPO eliminates the need for fitting a reward model, sampling from the language model during fine-tuning, or performing significant hyperparameter tuning.

In summary, DPO is not just a new algorithm; it’s a game-changer, simplifying and enhancing the way we build language models that truly understand and cater to human needs."