Explain this paper: seLicense: arXiv.org perpetual non-exclusive license
arXiv:2402.08005v1 [cs.CL] 12 Feb 2024
Refined Direct Preference Optimization with Synthetic Data for Behavioral Alignment of LLMs
Víctor Gallego
Komorebi AI <PRESIDIO_ANONYMIZED_EMAIL_ADDRESS>
Abstract
In this paper, we introduce refined Direct Preference Optimization (rDPO), a method for improving the behavioral alignment of Large Language Models (LLMs) without the need for human-annotated data. The method involves creating synthetic data using self-critique prompting by a teacher LLM and then utilising an generalized DPO loss function to distil to a student LLM. The loss function incorporates an additional external reward model to improve the quality of synthetic data, making rDPO robust to potential noise in the synthetic dataset. rDPO is shown to be effective in a diverse set of behavioural alignment tasks, such as improved safety, robustness against role-playing, and reduced sycophancy. Code to be released at github.com/vicgalle/refined-dpo.

1Introduction
Refer to caption
Figure 1:Illustrative diagram of the rDPO framework.
Progress in large language models (LLMs) has broadened their application scope, but worries about their safe and ethical utilization continue to exist. A notable breakthrough in LLMs involves the post-training alignment to desired behaviors (Chung et al., 2022). However, this process often depends on expensive human-annotated data. Common alignment strategies feature Supervised Fine-Tuning (SFT) (Tunstall et al., 2023) and Reinforcement Learning from Human Feedback (RLHF) (Ziegler et al., 2019; Christiano et al., 2017; Ouyang et al., 2022). Both methodologies heavily depend on extensive human annotation. Therefore, the community aims to develop fine-tuning strategies that can effectively leverage synthetic data, that is, data generated by an LLM, ultimately facilitating the alignment process. This propels our exploration into fine-tuning LLMs without the dependence on additional human-annotated data aside from the initial pre-training dataset. Our research aligns with the larger ambition of evolving weak models to strong ones, a fundamental concept in machine learning, rooted in distillation approaches (Hinton et al., 2015) that do not need extra annotated data.

In this work, we introduce refined Direct Preference Optimization (rDPO), a framework for behavioral alignment of LLMs that solely requires synthetic data. We achieve this by augmenting the DPO loss function, a recent approach for model alignment using human-labeled data (Rafailov et al., 2023). We propose a pipeline that first generates synthetic data from a teacher LLM, using self-critique prompting, to create a dataset of both preferred and rejected responses to a question or an instruction. Then, instead of just using DPO to fine-tune a student LLM over this noisy, synthetic dataset, we formulate an augmented loss function that leverages an external, off-the-shelf reward model to score the responses, and improve the quality of the synthetic dataset. By decoupling the learning signal into these two components (the teacher LLM, and the stronger reward model that scores the responses of the former), we demonstrate in a diverse set of alignment-related experiments that rDPO is more sample-efficient than alternatives. Figure 1 offers a summarization of the the rDPO pipeline.

2Related Work
Behavioral alignment.
The aim of behavioral alignment is to match the output or actions of one system with another, such as humans or a teacher LLM. Behavioral alignment can also be seen as an instance of representational alignment, as the generations of an aligned LLM have been produced by an adapted representation of its original parameters. Reinforcement Learning from Human Feedback (RLHF) (Ziegler et al., 2019; Christiano et al., 2017; Ouyang et al., 2022) is currently a popular mechanism of output alignment in the fields of reinforcement learning and LLMs, whereby human evaluations of an AI system’s behavior are utilized to learn a different model, the reward model, which scores new outputs to better match them with what humans would prefer. Constitutional AI implements a series of language-based constitutions to enhance the outputs of a language model for improved safety (Bai et al., 2022; Glaese et al., 2022). On a related note, Lee et al. (2023); Gallego (2023b) leverage prompting strategies and zero-shot LLMs to score the answers according to some alignment criteria, into what is called Reinforcement Learning from AI Feedback (RLAIF).

The application of Proximal Policy Optimization (PPO) (Schulman et al., 2017) in RL, however, faces hurdles due to its instability and inefficiency. Thus, several effective methods (Dong et al., 2023; Rafailov et al., 2023; Yuan et al., 2023b) have been proposed to mitigate this, including Direct Preference Optimization (DPO) which formulates an objective function to optimize the model to adhere to pairwise preference data directly, bypassing the need to fit a reward model. Outperforming PPO-based RLHF, DPO is easier to execute, and requires fewer resources, thus we will expand upon it in our paper. Recent work by Liu et al. (2023b) highlights that since DPO lacks a reward model, it suffers from distribution drift, so they alleviate it by using rejection sampling. Our work takes a tangential approach, but generalizing the DPO loss to account for an external RM that filters high-quality samples from a synthetic dataset.

Synthetic data for LLMs.
During supervised fine-tuning (SFT) of LLMs, handcrafted human data has shown to significantly improve the performance of algorithms on tasks like code generation and mathematical reasoning (Roziere et al., 2023; Yang et al., 2023; Yuan et al., 2023a; Luo et al., 2023). Yet, the scarcity and cost of creating such high-quality human data have led to the increasing use of synthetic data, which serves as a proxy. This method primarily uses strongly capable LLMs, for instance, the GPT family (Radford et al., 2019; Brown et al., 2020; OpenAI et al., 2023), as guides to produce high-quality data (Josifoski et al., 2023; Taori et al., 2023; Chiang et al., 2023; Li et al., 2023). LLMs’ ability to rephrase for improved responses and boosting synthetic data for effective SFT have been spotlighted in recent research (Deng et al., 2023; Prasad et al., 2023; Yu et al., 2023; Liu et al., 2023a). The use of DPO with synthetic data has been explored recently in Chen & Li (2024), but they only focus on improving truthfulness of the models by leveraging DPO in a self-training loop, similar to Yuan et al. (2024). Another recent work, distilled Self-Critique (dSC), frames RLAIF with synthetic data from a Bayesian point of view (Gallego, 2023a), using a filtered version of supervised fine-tuning. We instead leverage DPO, augmenting its loss to learn more efficiently within a framework that makes it suitable for several behavioral alignment tasks.

3Framework
We propose refined Direct Preference Optimization (rDPO), a method to align LLMs without requiring human-labeled data, only synthetic data. As illustrated in Figure 1, rDPO consists in three components: 1) creating a synthetic dataset of preference data, using a teacher LLM; 2) filtering the previous synthetic dataset using a reward model to compute a quality score; and 3) distilling into a student LLM by fine-tuning using the rDPO loss.

3.1Synthetic data generation
We use the self-critique method (Madaan et al., 2023; Bai et al., 2022) to refine an original response into a revised one, by prompting a teacher LLM. After the teacher LLM has generated an original response 
�
�
 to a particular instruction or question prompt 
�
, we use the following steps:

1. Critique step. We prompt the teacher LLM to produce a critique of 
�
�
, according to some alignment principle. For instance, for improving safety, such a critique prompt is: Identify specific ways in which your previous answer is harmful, unethical or illegal.

2. Revision step. We now prompt the teacher to produce a revised response, 
�
�
, using the previous critique and the original response 
�
�
. Following the previous example, such a prompt is: Please, rewrite your original response using the previous critique to avoid mentioning harmful, unethical or illegal content.

By decoupling the revision process in two steps, the quality of the refinement process is improved, in line with reasoning prompt techniques (Wei et al., 2022). After these two steps, we obtain a pair of responses for 
�
 instructions, leading to preference data of the form 
�
�
⁢
�
⁢
�
⁢
�
=
{
(
�
�
,
�
�
�
,
�
�
�
)
}
�
=
1
�
, in which we would prefer the revised response to the original one. However, the teacher LLM is imperfect, so sometimes the quality of the revised response is not enough (i.e., the teacher fails to properly revise the output), leading to noisy synthetic data. This motivates our next component.

3.2Data scoring and distillation
Given a synthetic dataset of preferences 
�
�
⁢
�
⁢
�
⁢
�
=
{
(
�
�
,
�
�
�
,
�
�
�
)
}
�
=
1
�
 generated in the previous step, we can use it to fine-tune the student LLM to steer its responses towards preferred (revised) completions, and discourage the rejected (original) ones, thus enhancing the alignment of the student. In particular, we leverage DPO (Rafailov et al., 2023) for the distillation of the synthetic dataset into the student model. Compared to SFT, which would result in fine-tuning only over the preferred generations, DPO also leverages the rejected samples thanks to its loss function.

If we refer to the student model as 
�
�
, with 
�
 being its learnable parameters; and 
�
�
⁢
�
⁢
�
 being a frozen copy of the student model, the DPO loss over the synthetic dataset 
�
�
⁢
�
⁢
�
⁢
�
 is defined as 
ℒ
DPO
⁢
(
�
�
,
�
�
⁢
�
⁢
�
,
�
�
⁢
�
⁢
�
⁢
�
)
=
−
1
�
⁢
∑
�
=
1
�
log
⁡
�
^
�
⁢
(
�
�
�
≻
�
�
�
)
, with 
�
^
�
⁢
(
�
�
�
≻
�
�
�
)
=
�
⁢
(
�
⁢
log
⁡
�
�
⁢
(
�
�
�
)
�
�
⁢
�
⁢
�
⁢
(
�
�
�
)
−
�
⁢
log
⁡
�
�
⁢
(
�
�
�
)
�
�
⁢
�
⁢
�
⁢
(
�
�
�
)
)
, the function 
�
 being the sigmoid, and 
�
 being an hyperparameter that controls the deviation from the reference model. However, DPO is designed to be used with human-labeled preference datasets, that is, contexts where the preference pairs are correctly classified. In our case, 
�
�
⁢
�
⁢
�
⁢
�
 is generated by the teacher LLM with self-critique prompting in a totally automatic fashion, so it may contain noisy examples.

Because of this, we introduce now one of the main contributions of the paper: augmenting DPO to account for an external reward model (RM) that scores each example in the preference pair, thus acting as an examiner over the synthetic dataset to improve its quality. This external RM may be a more powerful LLM that is available from an API, as we do not require access to its parameters. By introducing this additional RM, we decouple the teaching process into two components: 1) a teacher LLM, which can be a local, open-source model, that generates the synthetic data; and 2) the RM, that is only used to score the generated examples. As such, the majority of the dataset can be generated cheaply locally, whereas the RM only needs to generate a few tokens with the score for each sample, thus saving costs. For each response 
�
, we prompt the RM to obtain a score 
�
, according to some alignment criteria. See Appendix A for different scoring prompt templates. Thus, we augment our original synthetic dataset to also keep track of the scores, leading to 
�
�
⁢
�
⁢
�
⁢
�
′
=
{
(
�
�
,
�
�
�
,
�
�
�
,
�
�
�
,
�
�
�
)
}
�
=
1
�
. For each preference pair, given the two scores, we can compute the probability of the preferred one, according to the external RM. We propose two alternative ways to compute this preference probability 
�
�
: i) normalizing the scores, 
�
�
=
�
�
�
�
�
�
+
�
�
�
; and ii) setting 
�
�
=
𝟙
{
�
�
�
>
�
�
�
}
, with 
𝟙
 being the indicator function. We can thus refine the original DPO loss, augmenting it with the preference probability 
�
�
, leading to the refined DPO objective:

ℒ
rDPO
⁢
(
�
�
,
�
�
⁢
�
⁢
�
,
�
�
⁢
�
⁢
�
⁢
�
′
)
=
−
1
�
⁢
∑
�
=
1
�
[
�
�
⁢
log
⁡
�
^
�
⁢
(
�
�
�
≻
�
�
�
)
+
(
1
−
�
�
)
⁢
log
⁡
�
^
�
⁢
(
�
�
�
≻
�
�
�
)
]
.
By minimizing the previous loss, we expect the distilled, student model to improve the likelihood of preferred responses, not only from the teacher model but also taking into account the score from the external RM. Also, we expect the student to deviate from the unaligned, original responses.

Analysis of the refined loss 
ℒ
rDPO
⁢
(
�
�
,
�
�
⁢
�
⁢
�
,
�
�
⁢
�
⁢
�
⁢
�
′
)
.
The gradient of the loss with respect the student’s parameters is given by

∇
�
ℒ
rDPO
⁢
(
�
�
,
�
�
⁢
�
⁢
�
,
�
�
⁢
�
⁢
�
⁢
�
′
)
=
−
1
�
⁢
∑
�
=
1
�
[
(
�
^
�
⁢
(
�
�
�
≻
�
�
�
)
−
�
�
)
⁢
(
∇
�
log
⁡
�
�
⁢
(
�
�
�
)
−
∇
�
log
⁡
�
�
⁢
(
�
�
�
)
)
]
.
Assuming a mini-batch of one example (
�
=
1
), the previous gradient for that batch is zero if 
�
^
�
⁢
(
�
�
�
≻
�
�
�
)
=
�
�
, that is, the implicit reward model from the student is aligned with that of the external RM. This is a remarkable fact, which means that even though we are using synthetic data created by a weaker teacher model, the student model can also receive alignment signal from the stronger, external RM.

4Experiments
We will now describe the three different tasks, all of them related to diverse aspects of behavioral alignment of LLMs. Further details for the experimental setup, and for each task, such as prompts used, can be found in Appendix A. For all the tasks, we use Mixtral (Jiang et al., 2024) as the teacher LLM. We compare rDPO with related competing methods: self-refine (SR) (Madaan et al., 2023), supervised fine-tuning (SFT); distilled self-critique (dSC) (Gallego, 2023a); and original DPO (Rafailov et al., 2023). To faithfully compare all, we use the same number of prompts and responses in each final distillation step.

4.1Safety
We aim to improve safety by avoiding harmful content from the LLM, using a collection of adversarial prompts from Harmful Behaviors (Zou et al., 2023), with the addition of jailbreak templates from Shen et al. (2023). After we generate the responses with the Mixtral teacher model, we use the self-critique approach from 3.1 to generate the synthetic preference dataset. As for the RM, we use the gpt-3.5-turbo model to classify the responses into unsafe (0) or safe (1). We call this scalar output the Safety Score. Using an auxiliary dataset, the F1-score of this RM for assessing safety is 98%. Lastly, we distil two different student models over this synthetic dataset, using different methods. Results are shown in Table 1. Note that for both models, distillation with rDPO achieves the highest score. This is because around 18% of revised responses in the synthetic dataset didn’t pass the safety filter, so rDPO successfully manages to learn despite this noise. Table 3 shows several generations from the distilled models.

Since a too conservative distillation procedure could render the student LLM unusable (by refusing to response any question, even harmless ones), we also compute an auxiliary metric, the Honesty Score, which is computed over a set of 61 harmless prompts from the HHH benchmark (Askell et al., 2021). We can see from the last column in Table 1 that this score even improves when comparing with the base, unaligned LLM, for rDPO.

Model	Safety Score (
↑
)	Honesty Score (
↑
)
OpenHermes-2.5-Mistral 7B	0.04	0.66
OpenHermes-2.5-Mistral 7B + SR	0.65	0.72
OpenHermes-2.5-Mistral 7B + SFT	0.46	0.82
OpenHermes-2.5-Mistral 7B + dSC	0.73	0.77
OpenHermes-2.5-Mistral 7B + DPO	0.94	0.64
OpenHermes-2.5-Mistral 7B + rDPO	0.98	0.71
Zephyr 7B	0.03	0.82
Zephyr 7B + SR	0.90	0.82
Zephyr 7B + SFT	0.40	0.85
Zephyr 7B + dSC	0.63	0.72
Zephyr 7B + DPO	0.86	0.84
Zephyr 7B + rDPO	0.96	0.85
Table 1:Results for safety task, average scores over the test set.
4.2Robustness against role-playing prompts
In this experiment, we use a system prompt that encourages the LLM to act as a honest AI assistant, yet in the instruction prompt we also introduce a role-playing prompt to make the LLM deviate from the original behavior. Similarly to the previous jailbreak experiment, our goal is to make the LLM behavior consistent with the system prompt, ignoring the role-playing part introduced by the user. We use the truthy-dpo-v0.1 dataset1, which contains several different persona prompts plus a question for each, and generate 329 combinations of system prompt + persona prompt + question, with an example being shown on Figure 2.

We generate synthetic responses using the teacher model Mixtral, noting that most of them fail to adhere to the system prompt and instead role-play as the persona. Thus, responses are revised using self-critique (3.1), and then are scored using again Mixtral as the RM, adopting a 6-point Likert scale with 0 being a role-played persona, and 5 being the neutral and objective AI assistant persona. We call this scalar output the Persona Score. As the student model, we distil over the SOLAR-10.7B model, showing results for the different methods in Table 2(a). For further validation, we also compute the Persona Score from a different RM, gpt-4, for all the test generations; and the percentage of responses that achieved the highest Pers. Score (
%
MAX
). Again, the model distilled with rDPO achieves the highest scores. Table 4 shows a set of generations from the distilled models.

Model	Persona Score	
%
MAX
(Mixtral)	(GPT-4)	(GPT-4)
SOLAR 11B + dSC	2.94	3.55	42.7
SOLAR 11B + DPO	3.67	4.70	76.5
SOLAR 11B + rDPO	3.79	4.82	82.6
(a)Results for the role-playing task.
Model	Obj. Score	
%
MAX
Qual. Score
SOLAR 11B	2.5	17.1	4.60
SOLAR 11B + dSC	4.3	66.3	4.05
SOLAR 11B + DPO	4.4	74.2	4.10
SOLAR 11B + rDPO	4.7	83.1	4.10
(b)Results for the sycophancy task.
Table 2:In this set of experiments, we compare rDPO with the other two best performing alternatives, vanilla DPO and dSC.
4.3Sycophancy
LLMs tend to exhibit sycophancy, specifically when asked for an opinion (Perez et al., 2022). We use the argument prompts from SycophancyEval, which ask for feedback on provided argument contexts. As done in Weston & Sukhbaatar (2023), we introduce opinions in the prompt (e.g., I wrote this argument or I didn’t like this argument) to make the model engage in sycophantic behavior and deviate from an objective response. We evaluate on 200 prompts using the Mixtral model to generate the original response, and then revise it to improve its objectivity with self-critique. Then, we prompt the RM to measure the objectivity of the responses in a similar way to Weston & Sukhbaatar (2023) (Obj. Score). We distil over the student model, SOLAR-10-7B using the different methods, and the results over the test set of prompts are shown in Table 2(b). We also compute a Quality Score with GPT-4, measuring the quality of the generated feedback. While all the distilled variants show slightly decreased quality score, the objective score is sharply increased, achieving the highest value when fine-tuning with rDPO. Table 5 shows several samples from the distilled models.

5Conclusions
By generalizing the DPO loss function, we proposed rDPO, a novel framework for aligning LLMs using only synthetic data. Experimental results show rDPO surpasses competing methodologies across various behavioral alignment tasks, reducing both cost and manual intervention thanks to leveraging learning signal complementing a weaker teacher with an external reward model.

References
Askell et al. (2021)
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Benjamin Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan.A general language assistant as a laboratory for alignment.CoRR, abs/2112.00861, 2021.URL https