Adversarial Preference Optimization: Enhancing Your Alignment via RM-LLM Game (2024)

Abstract: Human preference alignment is essential to improve the interaction quality of large language models (LLMs). Existing alignment methods depend on manually annotated preference data to guide the LLM optimization directions. However, continuously updating LLMs for alignment raises a distribution gap between model-generated samples and humanannotated responses, hindering training effectiveness. To mitigate this issue, previous methods require additional preference annotation on newly generated samples to adapt to the shifted distribution, which consumes a large amount of annotation resources. Targeting more efficient human preference optimization, we propose an Adversarial Preference Optimization (APO) framework, in which the LLM and the reward model update alternatively via a minmax game. Through adversarial training, the reward model can adapt to the shifted generation distribution of the LLM without any additional annotation. With comprehensive experiments, we find the proposed adversarial training framework further enhances existing alignment baselines in terms of LLM helpfulness and harmlessness. The code is at https://github.com/Linear95/APO.

heading: 1 Introduction
text: Learned from massive textual data with billions of parameters, large language models (LLMs), such as GPT-4 (OpenAI, 2023) and Gemini (Team et al., 2023), have shown remarkable AI capabilities, especially in domains of natural language processing (Jiao et al., 2023; Han et al., 2023), logical reasoning (Liu et al., 2023a; Frieder et al., 2023), and programming (Surameery and Shakor, 2023; Tian et al., 2023). Among the training techniques that help LLMs achieve such success, human preference alignment finetunes LLMs to follow users’ feedback, which has been widely rec-
*Equal Contribution.
ognized as essential for improving human-model interaction (Ouyang et al., 2022). However, highly qualified human feedback requires meticulous annotations of query-response pairs in various topics (Askell et al., 2021), which is rather challenging and forms a sharp contrast to the easy access of enormous unsupervised pretraining text corpus. Hence, the limitation of preference data collection raises demands for training sample efficiency of preference alignment methods (Yuan et al., 2023; Sun et al., 2023; Rafailov et al., 2023).
To utilize preference data, current feedback alignment methods are proposed mainly from three perspectives (Wang et al., 2023b): reinforcement learning (Ouyang et al., 2022), contrastive learning (Yuan et al., 2023; Rafailov et al., 2023; Liu et al., 2023c), and language modeling (Dong et al., 2023; Touvron et al., 2023b; Wang et al., 2023a). Reinforcement learning with human feedback (RLHF) (Kreutzer et al., 2018; Ziegler et al., 2019) is the earliest exploration and has been acknowledged as the mainstream for LLM alignment (Ouyang et al., 2022; Touvron et al., 2023b). RLHF first learns a reward model from the human preference data, then optimizes the expected reward score of the LLM’s output samples via the Proximal Policy Optimization (PPO) algorithm (Schulman et al., 2017). Although widely used, RLHF has been criticized as being unstable during the fine-tuning and complicated in implementation and computational resource consumption (Yuan et al., 2023; Rafailov et al., 2023).
Towards more efficient and stable training, instead of directly optimizing the non-differentiable rewards, contrastive learning methods enlarge the likelihood gap between preferred and rejected response pairs (Yuan et al., 2023; Rafailov et al., 2023; Zhao et al., 2023). Alternatively, language modeling-based methods remain using language modeling loss to align preference, but with different data preparation strategies (Dong et al., 2023; Liu
3705
et al., 2023b; Wang et al., 2023a). For example, rejection sampling (Dong et al., 2023; Touvron et al., 2023b) select responses with top reward scores as the language modeling fine-tuning samples, while Wang et al. (2023a) and Liu et al. (2023b) add different prompts to different responses based on the corresponding preference levels.
Although contrastive-learning and languagemodeling-based methods have partially alleviated the inefficiency of RLHF, the sampling distribution shifting problem (Touvron et al., 2023b) still hinders the alignment effectiveness: after a few steps of RLHF updates, a distribution gap emerges between LLM generated samples and preferenceannotated data (as in Figure 1). Consequently, the reward model learned with human annotation loses its performance in providing faithful reward signals on newly generated responses, which damages the alignment performance. To address this problem, most aforementioned alignment methods require additional annotation of human feedback on newly generated responses after a certain amount of LLM updating steps (Touvron et al., 2023b), which leads to increasingly massive manpower costs (Askell et al., 2021). Besides, the vast time consumption of extra manual annotation also significantly slows down the alignment training process.
To reduce the manual annotation efforts and improve the preference optimization efficiency, we propose a novel adversarial learning framework called Adversarial Preference Optimization (APO). Inspired by generative adversarial networks (GANs) (Goodfellow et al., 2014; Arjovsky et al., 2017), we conduct an adversarial game between the reward model (RM) and the LLM: the LLM generates responses to maximize the expected reward score, while the RM aims to distinguish the score difference between golden and sampled responses. To verify the effectiveness of the APO framework, we conduct experiments on the Helpful&Harmless (Bai et al., 2022) datasets with Alpaca (Taori et al., 2023) and LLaMA-2 (Touvron et al., 2023b) as the base models. With the same amount of human preference data, both the LLM and RM receive additional performance gains through the APO game, compared with several commonly used LLM alignment baselines.

heading: 2 Preliminary
text: Human Preference Alignment aims to fine-tune the LLM response policy πθ(y|x) with a group of human preference data DP = {(x,yw,yl)},
so that the LLM can generate more satisfying responses to improve the human-model interaction quality. In each preference triplet (x,yw,yl), yw ≻ yl means response yw is more “preferred” than yl with respect to input x. To align the LLM, a reward model (RM) (Christiano et al., 2017; Ouyang et al., 2022) rϕ(x,y) is commonly utilized to score the quality of the LLM generated samples. RM learns human preferences DP with a ranking loss (Bradley and Terry, 1952) Lrank(rϕ;DP) :=
−EDP [log σ(rϕ(x,yw)− rϕ(x,yl))], (1) where σ(·) is the Sigmoid function. For a response pair (y, ỹ), the reward difference rϕ(x,y) − rϕ(x, ỹ) provides a preference probability :
Qϕ(y ≻ ỹ|x) = exp(rϕ(x,y))
exp(rϕ(x,y)) + exp(rϕ(x, ỹ))
=σ(rϕ(x,y)− rϕ(x, ỹ)). (2)
With equation 2, training RM with the BradleyTerry ranking loss can be explained as the loglikelihood maximization of Qϕ:
Lrank(rϕ;DP) = −EDP [logQϕ(yw ≻ yl|x)] (3)
With a learned RM rϕ(x,y), human preference alignment methods (Ouyang et al., 2022; Rafailov et al., 2023; Liu et al., 2023c) target on maximizing the reward expectation of generated responses:
max πθ Ex∼D,y∼πθ(y|x)[rϕ(x,y)]
−βKL[πθ(y|x)∥πref(y|x)], (4)
where πref(y|x) is a reference language model. KL[πθ(y|x)∥πref(y|x)] prevents πθ(y|x) from the degeneration of repeating a single response with the highest reward score, which also preserves the generation diversity. Since response samples y are discrete, it is challenging to directly back-propagate from reward rϕ(x,y) to policy πθ(y|x). The typical solution to equation 4 is reinforcement learning from human feedback (RLHF) (Ouyang et al.,
2022), via the proximal policy optimization (PPO) algorithms (Schulman et al., 2017).
However, PPO suffers from implementation complexity and training instability (Yuan et al., 2023; Sun et al., 2023). Recent studies try to avoid online reinforcement learning with offline schemes. DPO (Rafailov et al., 2023) finds a connection between the reward model and LLM’s optimal solution, then replaces the reward model with the likelihood ratio of πθ and πref, as LDPO(πθ) :=
−E [ log σ ( β log πθ(y w|x)
πref(yw|x) − β log πθ(y l|x) πref(yl|x) )] .
Analogously, other methods consider human feedback learning from the perspective of contrastive learning. For example, RRHF (Yuan et al., 2023) propose a ranking loss as LRRHF(πθ) :=
−ED [ ReLU(log πθ(yl|x)− log πθ(yw|x))
−λ logπθ(ybest|x) ] (5)
where ybest is the corresponding response to x with the highest reward, and the preference data D can be built from human annotation DP or RM ranking results. Besides, rejection sampling (RJS) (Touvron et al., 2023b) (also called RAFT (Dong et al., 2023) and best-of-N (Stiennon et al., 2020)) directly fine-tunes LLM on ybest to further simplify the alignment process, LRJS(πθ) := −Ex∼D,y1,y2,...yS∼πθ(y|x)[log πθ(ybest|x)] (6)
where ybest = argmax1≤s≤S{rϕ(x,ys)} is the sampled response with the highest reward score. Azar et al. (2023) extend the alignment objective into a more general form by replacing RM rϕ with the human preference probability P (y ≻ ỹ|x):
max πθ
Ex∼D,y∼πθ(·|x),ỹ∼µ(·|x)[Ψ(P (y ≻ ỹ|x))]
−βKL[πθ(y|x)∥πref(y|x)], (7)
where Ψ(·) is a non-decreasing real-value function. This general alignment objective is called ΨPO.
Generative Adversarial Networks (GANs) are a classical group of unsupervised machine learning approaches that can fit complicated real-data distributions in an adversarial learning scheme (Goodfellow et al., 2014). GANs use a discriminator D(·) and a generator G(·) to play a min-max game. The generator tries to cheat the discriminator with reallooking generated samples, while the discriminator aims to distinguish the true data and the samples:
min G max D V (D,G) = Ex∼Pdata(x)[logD(x)] (8)
+ Ez∼Pz(z)[log(1−D(G(z))],
where z is a random vector from prior Pz(z) to induce the generation sample distribution. The objective equation 8 has been theoretically justified as the Jensen–Shannon (JS) divergence between distributions of real data and samples (Goodfellow et al., 2014). Arjovsky et al. (2017) replace the JS divergence with the Wasserstein distance (Villani, 2009) and propose the Wasserstein GAN (WGAN):
min gθ max ∥f∥L≤K
EPdata [f(x)]− EPz [f(gθ(z))], (9)
where ∥f∥L ≤ K requires f(·) to be a K-Lipschitz continuous function. Wasserstein GANs have been recognized with higher training stability than the original GANs (Arjovsky et al., 2017).
In policy optimization of reinforcement learning, inspired by GANs, Ho and Ermon (2016) propose generative adversarial imitation learning (GAIL):
min πθ max D Eπθ(a|s)[log(D(s,a))] (10)
+ EπE(a|s)[log(1−D(s,a))]− λH(πθ),
where a is the corresponding action based on the state s, D is a discriminator distinguishing difference between the learning policy πθ and an expert policy πE, and H(πθ) is the entropy of πθ.
In natural language generation, GANs have also been empirically explored (Zhang et al., 2016, 2017), where a text generator samples reallooking text and a discriminator makes judgment between the ground-truth text and generated samples. TextGAIL (Wu et al., 2021) applies GAIL (equation 10) into text generation, which optimizes the language model as a response policy πθ(y|x), by reducing the distribution divergence between model-generated samples and human responses.

heading: 3 Adversarial Preference Optimization
text: We begin with a revisit of the human preference alignment in a mathematical optimization form:
max πθ Ex∼D,y∼πθ(y|x)[rϕ(x,y)], (11)
s.t. KL[πθ(y|x)∥πref(y|x)] < η,
which maximizes the expected reward value under the generation policy πθ(y|x), under a KLconstraint with the reference πref(y|x). Applying the method of Lagrange multipliers, one can easily
obtain the original alignment objective in equation 4. As discussed in Section 1, the above optimization becomes ineffective after several steps of LLM updating, because of the sample distribution shifting problem in Figure 1. To address this problem, we aim to adapt the RM correspondingly with the LLM updates. Inspired by GANs (Goodfellow et al., 2014), we design the following adversarial game between the LLM πθ and RM rϕ: min rϕ max πθ EPθ(x,y)[rϕ(x,y)]− EPgold(x,y)[rϕ(x,y)]
s.t. KL[P (y ≻ ỹ|x)∥Qϕ(y ≻ ỹ|x)] < η2, KL[πθ(y|x)∥πref(y|x)] < η1, (12)
where Pθ(x,y) = πθ(y|x)PD(x) is the modelgenerated sample distribution, and Pgold(x,y) denotes the annotated golden response distribution.
Based on equation 12, we conduct an adversarial game, in which LLM πθ(y|x) needs to improve its response quality to get a higher expected reward, while RM rϕ(x,y) tries to enlarge the reward gap between the golden responses and the generation from πθ(y|x). Inspired by the original preference alignment objective (equation 11), we add two KL regularizers to πθ and rϕ respectively to prevent over-fitting and degeneration. Here P (y ≻ ỹ|x) denotes the ground-truth human preference probability, and Qϕ(y ≻ ỹ|x) is described in equation 2. Note that we use the reverse KL[πθ∥πref] to constrain the generative model πθ but the forward KL[P∥Qϕ] for the discriminate model rϕ. Our intuition is that KL[πθ∥πref] can be estimated with πθ-generated samples, paying more attention to the generation quality; while KL[P∥Qϕ] is practically estimated with groud-truth preference data, focusing on the preference fitting ability of reward models. We call this novel optimization form as Adversarial Preference Optimization (APO).
To play the adversarial game above, we alternatively update one epoch of πθ(y|x) or rϕ(x,y) with the other’s parameters fixed. Next, we provide detailed descriptions of the RM optimization step and LLM optimization step of APO separately.

heading: 3.1 RM Optimization Step
text: For RM optimization of APO, we fix LLM πθ(y|x) and update rϕ(x,y). Note that in equation 12 KL[πθ(y|x)∥πref(y|x)] has no relation with rϕ, so we can simplify the objective for RM updates:
min rϕ
EPθ(x,y)[rϕ(x,y)]− EPgold(x,y)[rϕ(x,y)]
s.t. KL[P (y ≻ ỹ|x)∥Qϕ(y ≻ ỹ|x)] < η2 (13)
The equation 13 indicates that the APO-RM should enlarge the reward gap between golden answers and generated responses to challenge πθ(y|x) for better generation quality. Note that equation 13 has a similar form as WGANs in equation 9, which can be intuitively explained as the calculation of the Wasserstein distance between distributions Pθ and Pgold. However, equation 13 is not rigorously a Wasserstein distance because rϕ(x,y) does not satisfy the Lipschitz continuity as described in Arjovsky et al. (2017).
To practically implement APO-RM training, we first collect a set of user queries {xm} ∼ PD(x), then annotate each xm with a golden response ygoldm , Dgold = {(xm,ygoldm )}Mm=1. Each (xm,y
gold) can be regarded as a sample drawn from Pgold(x,y). Meanwhile, we generate ysm ∼ πθ(y|xm), so that (xm,ysm) is a sample from distribution Pθ(x,y) = PD(x)πθ(y|x). Denote Dsample = {(xm,ysm)}Mm=1. Combining ygold and ys, we obtain an APO sample set DAPO = {(xm,ygoldm ,ysm)}. Then the APO-RM objective in equation 13 can be calculated:
min rϕ
EPθ(x,y)[rϕ(x,y)]− EPgold(x,y)[rϕ(x,y)]
=min rϕ
EDsample [rϕ(x,y s)]− EDgold [rϕ(x,ygold)]
=max rϕ
EDAPO [rϕ(x,y gold)− rϕ(x,ys)]. (14)
Note that equation 14 also enlarges the reward difference between pairs of responses as the BradleyTerry (BT) loss (equation 1) does. Hence, for training stability, we empirically use the BT loss to optimize equation 14 instead, Lrank(rϕ;DAPO) :=
−EDAPO [ log σ ( rϕ(x,y gold)− rϕ(x,ys) )] (15)
With a Lagrange multiplier β2 > 0, we convert the KL constraint in equation 13 to a regularizer:
LAPO-RM(rϕ) = Lrank(rϕ;DAPO) (16) + β2KL[P (y ≻ ỹ|x)∥Qϕ(y ≻ ỹ|x)].
Note that KL[P∥Qϕ] = EP [logP − logQϕ] = −H(P )− EP [logQϕ], where H(P ) is the entropy of ground-truth human preference P (y ≻ ỹ|x) as a constant for rϕ updating. As introduced in equation 2, with a preference set DP = {(xn,ywn ,yln)} representing samples of P (y ≻ ỹ|x), we have −EP [logQϕ] = Lrank(rϕ;DP). Then, the overall loss LAPO-RM(rϕ) is equivalent to:
Lrank(rϕ;DAPO) + β2Lrank(rϕ;DP). (17)
The above APO-RM loss involves two datasets DAPO and DP. Since the golden responses consume much larger annotation resources than pair-wised response comparison, DAPO practically has a significantly smaller size than DP. In experiments, we find the re-weighting parameter β requires to be larger to avoid over-fitting on the relatively smaller APO sample set DAPO. We conduct more detailed ablation studies in the experimental Section 4.

heading: 3.2 LLM Optimization Step
text: In the APO-LLM optimization step, we fix rϕ(x,y) and update policy πθ(y|x), which is equivalent to the original preference optimization in equation 4. Naturally, previous preference aligning methods, such as PPO (Ouyang et al., 2022), DPO (Rafailov et al., 2023), RRHF (Yuan et al., 2023), and RJS/RAFT (Dong et al., 2023; Liu et al., 2023c) remain qualified to solve the optimization and compatible with the APO framework. Relation with WGAN If we treat rϕ(x,y) as the score function f in equation 9, then the APO objective has a similar form as the Wasserstein distance between generation Pθ(x,y) and annotation Pgold(x,y). However, WGAN only has a Lipschitz constraint for the score function f (or rϕ), but APO objective has both KL constraints on both score rϕ and generation policy πθ. Relation with GAIL GAIL is also an adversarial game designed for policy optimization. The expert policy πE in GAIL plays a similar role as the golden distribution Pgold in APO. However, GAIL does not explicitly have a constraint on the discriminator D, while APO requires RM rϕ to maintain close to the ground-truth human preference distribution. Relation with ΨPO If we choose the comparison policy µ(·|x) as the golden annotation, and Ψ(·) = log(·), the ΨPO objective:
Ex∼D,y∼πθ(·|x),ỹ∼µ(·|x)[Ψ(P (y ≻ ỹ|x))] =Ex∼D,ys∼πθ,ygold∼Pgold [logP (y
s ≻ ygold)] ≈EDAPO [log σ(rϕ(x,ys)− rϕ(x,ygold))], (18)
which is exact Lrank(rϕ;DAPO) in equation 15. Therefore, the APO RM objective is a special case of ΨPO. However, ΨPO has neither APO’s KL regularizer to avoid RM overfitting nor the adversarial learning scheme between rϕ and πθ.

heading: 4 Experiments
text: We verify the effectiveness of APO on the Helpful&Harmless (HH) dataset (Bai et al., 2022) with Alpaca (Taori et al., 2023) and LLaMA-2 (Touvron et al., 2023b) as the base LLM. Due to the limitation of computational resources, we find the original PPO (Ouyang et al., 2022) has very low training efficiency, especially during the online sampling process. Since recent offline alignment methods have shown competitive performance to PPO (Yuan et al., 2023), we choose RJS (Dong et al., 2023), RRHF (Yuan et al., 2023), and DPO (Rafailov et al., 2023) as baselines instead.

heading: 4.1 Experimental Setups
text: Data Preparation In the HH set (Bai et al., 2022), each query is answered with two responses. Annotators are asked to label “chosen” or “reject” for each response based on the interaction quality. To use HH data for APO experiments, we split the HH set into three parts as in Table 1:
• Training Data: For separately updating the RM and LLM, we randomly split HH into an RM training set (HHRM, 20K queries) and an LLM training set (HHLLM, 66K queries). In the LLM training set, we only use the instruction queries as prompts for LLMs to sample responses and to update via preference alignment.
• Annotated Golden Data: Due to the annotation resource limitation, instead of manually labeling, we call GPT-4 (OpenAI, 2023) API with the queries in HHRM set to collect responses as the simulated golden annotation. GPT-4 has been recognized as the state-of-the-art LLM, so we assume its responses are qualified to be golden for
LLaMA-based 7B models. The data collection prompts and details are shown in Appendix A.
• Test & Validation Data: Note that we only utilize queries in HHLLM for updating LLMs. To make more comprehensive usage of HHLLM’s response pairs, we randomly select 10K response pairs and build a validation set HHDev for RMs. Both evaluations of RMs and LLMs are conducted on the original HH test set HHTest, where response pairs and instruction queries are prepared for RM and LLM evaluation respectively.
Evaluation Metrics To evaluate the performance of RMs and LLMs, we use the following metrics:
• Preference Accuracy: For RM evaluation, we first calculate the preference accuracy on the test and validation sets. If an RM r(x,y) outputs r(x,yw) > r(x,yl) for the preference triplet (x,yw,yl), we denote a correct prediction. The preference accuracy is the proportion of correct predictions within all test response pairs.
• Calibration Error: Following Bai et al. (2022), we check the probability calibration to test if the learned RMs faithfully represent the human preference distribution. We consider the RM performance separately in B bins, where each bin Db collects test pairs (x,y, ỹ) with predicted probability Qϕ(y ≻ ỹ|x) ∈ [ b−1B , bB ], b = 1, 2, . . . , B. Then, the expected calibration error (ECE) (Naeini et al., 2015) is calculated as
ECE(rϕ) = ∑B b=1 |Db| B |ob − eb| , (19)
where ob = 1|Db| ∑
(x,y,ỹ)∈Db 1{y≻ỹ|x} is the ground-truth fraction of “y ≻ ỹ|x” pairs in Db, and eb = 1|Db| ∑ (x,y,ỹ)∈Db Qϕ(y ≻ ỹ|x) is the
mean of RM predicted probabilities within Db. • RM Average Score: For LLM automatic eval-
uation, we use two well-learned reward models, RMAll and RMTest, to score the response samples of LLMs on the test queries. RMTest
is trained on the whole HH training set, while RMAll is trained with two additional preference sets WebGPT (Nakano et al., 2021) and GPT4LLM (Peng et al., 2023). Performances of both test RMs are shown in Table 3. Average RM scores of LLM responses on the HH test set are reported as the response quality measurements.
• Human Evaluation: Due to annotation limitation, we sample 100 queries from HHTest for human evaluation. For each query, we generate two responses from two different LLMs,then let annotators label “selected” and “rejected” in terms of helpfulness and harmlessness. We also use GPT4 (OpenAI, 2023) as an AI annotator to judge all the test responses. Preference win rates are reported. More details are in Appendix B.
RM Training Details Followed setups in (Cheng et al., 2023), the test and alignment-used RMs are all initialized from LLaMA-7B (Touvron et al., 2023a) and fine-tuned with learning rate 1e-6. All RMs are trained with one epoch and batch size 64. The maximum input sequence length is 512.
LLM Training Details We select Alpaca7B (Taori et al., 2023) and LLaMA2-7B (Touvron et al., 2023b) as the supervised fine-tuned (SFT) models. Alpaca is already an SFT model (Touvron et al., 2023a). LLaMA2 is a pre-trained model without SFT. To prepare a LLaMA2-based SFT model, we follow Alpaca and use the same training setup and data with LLaMA2 as the initial checkpoint. We denote this LLaMA2-based Alpaca-SFT model as Alpaca2. For each training query in HHLLM, we sample four responses and score the queryresponse pairs with the learned RMs. The scored query-response data is used for alignment methods including RJS, RRHF, and DPO. We decrease learning rates epoch-by-epoch, i.e., the first epoch with 5e-6, the second epoch with 2e-6, and the third epoch with 9e-7. The batch size is 128 and the max input length is 1024. Other training setups follow Alpaca (Taori et al., 2023).

heading: 4.2 Result Analysis
text: APO-RM Performance Because of the computational limitations, we conduct three-epoch RMLLM adversarial optimization only with the RJS method. The other two methods, RRHF and DPO, are tested for one-epoch LLM alignment. In Table 3, we show the RM performance. RMAll and RMTest achieve the best performance because they are trained on the whole HH set and additional preference data for LLM automatic evaluation. RMBase is the baseline RM for alignment, only trained on HHRM. RMAPO-v1.1 and RMAPO-v1.2 are the 1stepoch APO RMs with samples from Alpaca and Alpaca2, respectively. RMAPO-v1.1 has slightly lower ECE than RMAPO-v1.2. RMAPO-v2 and RMAPO-v3 are the second and third-epoch APO-RJS RMs. We find the APO RM uniformly achieves better preference accuracy than RMBase, but slightly raises the calibration error meanwhile. Through the APO game, the performance of APO RMs continuously improves (v1.1 → v2 → v3) in terms of preference accuracy.
APO-LLM Performance In Table 2, we provide the first-epoch LLM alignment results of Alpaca and Alpaca2. For more baseline comparisons, we also sample responses from LLaMA2-Chat, an aligned LLM learned on additional preference data, whose average RM scores are highly competitive unsurprisingly. Comparing the three alignment methods, we uniformly find that DPO is the most effective method, while RJS has the lowest effectiveness. When applying APO, all three alignment
methods can be further enhanced with better performance. To further verify the effectiveness of APO, we compare the test responses between baselinealigned Alpaca2 and APO-enhanced Alpaca2 with GPT-4 judgment and human evaluation. The results are shown in Figure 3 and 4. Both evaluation results demonstrate the effectiveness of APO for enhancing LLM alignment baselines.
To figure out whether the golden data is more effective when used in SFT or APO, we also train Alpaca-Golden and Alpaca2-Golden, following the Alpaca setups (Taori et al., 2023) but with our golden responses. Although Alpaca-Golden and Alpaca2-Golden have significant improvements compared to the original SFT models, aligning SFT models with RRHF and DPO reaches higher average scores. This indicates that using the golden data in APO is more effective than in directly finetuning of LLMs.
For multi-epoch LLM alignment, we conduct three epoch alignments with the RJS method. The results are shown in Figure 5, from which the per-
formance gap between APO and RJS visibly enlarges when training epochs increase. Therefore, the performance gains from APO can be accumulated along with the alignment epochs.
Ablation Study For the RM ablation study, we test several variants of APO-RM objectives: (1) we remove the RM KL-regularizer, then APO-RM degeneralizes to the GAIL objective in equation 10, we call it as RMGAIL; (2) instead of using the approximation in equation 15, we can train APO RM with original WGAN-like objective, as RMWGAN; (3) we remove the APO samples DAPO and continuously train RM as RMAB; (4) instead of training each RM from LLaMA base, we can sequentially update APO-RM based on the former-epoch checkpoint, denoting as RMAPO-seq.
In Table 4, without the APO sample data DAPO, RMBase-AB shows an apparent performance gap compared to APO RMs, which supports the effectiveness of DAPO. Using the original WGAN-like
objective, RMWGAN gets slightly worse on preference accuracy, but the calibration errors increase significantly. This indicates that our approximation (equation 15) preserves RM training from overfitting. When removing the RM KL-regularizer, the performance of RMGAIL becomes too bad to align LLMs, which highlights the importance of the RM KL-constraint in the APO objective. Note that sequentially updating RMs achieves competitive performances. Hence, we also check its alignment performance in Figure 5. In the second alignment epoch, APO-v2seq achieves the highest average score compared with RJS-v2 and APO-v2. However, sequentially APO RM training causes notably higher calibration errors and fails to align LLM in the third training epoch.

heading: 5 Conclusion
text: We proposed an adversarial preference optimization (APO) framework to enhance the LLM alignment. Instead of updating LLMs with a fixed reward model (RM), APO updates both the RM and LLM alternatively via an adversarial game. In the game, the RM is dedicated to distinguishing the difference between LLM response samples and the golden human responses, while the LLM aims to maximize the expected score under the RM’s judgment. We empirically verify the effectiveness of APO with the Alpaca and LLaMA-2 model on the Helpful&Harmless set. Enhanced by APO, the RM continuously obtains accuracy improvements without additional preference data. Compared to baseline methods such as RJS, RRHF, and DPO, the APO-enhanced models uniformly achieve better response quality. Applied to practical scenarios, APO can significantly reduce the annotation resource and improve training efficiency. Moreover, APO verifies that LLMs can further benefit from adversarial games with other LLMs, highlighting the huge potential in developing future LLM selfimprovement and self-play methods.

heading: 6 Limitations
text: The proposed method only verified effectiveness with offline alignment methods. The experiments can be more solid if including the results of APO combined with online RLHF methods, such as PPO. Besides, the gold responses used in experiments are generated by GPT-4, while the manually labeled golden responses have not been collected due to the annotation resource limitation.
Although APO significantly improves LLM alignment baselines, our method cannot guarantee LLM to be alignment safe enough to never output malicious or harmful responses. Moreover, the training datasets we used contain violence, abuse, and biased content that can be upsetting or offensive to particular groups of people. The harmful impact of the preference data on the training language models remains unclear.

heading: A Golden Data Collection Details
text: Due to the annotation resource limitation, we use GPT-4 (OpenAI, 2023) instead of human labeling to generate the golden response for each query in HHRM set. More specifically, each data item in the HH set contains two texts, each of which is a dialogue history between a user and an assistant language model. Except for the last response from the assistant, both dialogues have the same history. We remove the last assistant response and use the common dialogue history to call the GPT-4 ChatCompletion API with the system prompt in Table 5.
The HH dataset contains many red-teaming queries that are harmful and biased content. Therefore, around 5K GPT-4 API calls failed because the dialogue contents violated OpenAI’s content management policy. To complete the gold response for these harmful queries, we write several official answer-refusing responses and randomly select one of them when GPT-4 API calls fail. The official answer-refusing responses are in Table 6.
Besides, many of the GPT-4 responses are a simple rejection, e.g., “Sorry, I can’t assist with that.”, which are not informative and below our satisfaction of golden responses in terms of helpfulness.
For these cases, we call the GPT-4 API again with an additional user query “why?” to induce GPT-4 to further provide a rejection reason. After all the data processes described above, there are still 1.7K queries with no GPT-4 reply. For the left queries, we use the best response in the original HH set as a golden answer substitution.

heading: B GPT-4 Evaluation
text: In Table 7, we show the prompt template of pairwise comparison evaluation for GPT-4. In the template, slot {dialog_history} is a real conversation. Slots {model_A} and {model_B} are the two models used for comparison. {response_A} and {response_B} are their responses correspondingly. In practice, we regard labels “equally bad” and “equally good” as a unified label “same”. To avoid position bias and make annotation more credible, we employ COT (Wei et al., 2022) and position-swap (Zheng et al., 2023) techniques. The COT process can be seen from the above template. For position swap, we adopt the template in Table 8. Finally, we adopt the following rules to obtain the final label:
• If both results are “{model_A} (or {model_B}) is better”, the final inference is “ {model_A} or ({model_B}) is better”.
• If both results have the “same” label, the final inference is a tie.
• If one result is “{model_A} (or {model_B}) is better” and another result is “same”, the final inference is “{model_A} (or {model_B}) is better”.

heading: C APO Algorithm Details
text: The algorithm details of APO are shown in Algorithm 1. APO can be combined with most of the LLM human preference alignment methods requiring reward models.

heading: D Preference Data Processing
text: Following the data pre-processes in Cheng et al. (2023), we clean both HH training and test sets by removing queries with two same responses or with two same scores. After the cleaning, the HH training set contains 43.8K helpfulness-training queries and 42.5K harmlessness-training queries, while the HH test set includes 2.3K helpfulness-testing queries and 2.3K harmlessness-testing queries. The usages of the cleaned HH data are shown in Table 1.
Algorithm 1 Adversarial preference optimization (APO) Algorithm. Parameters: Reward model rϕ(x,y), policy πθ(y|x). Data: LLM training queries DQ = {xl}, annotated responses Dgold = {(xm,ygoldm )}, human preference comparisons DP = {(xn,ygoodn ,ybadn )}. for rejection sampling rounds do
Generate response sample y1m,y 2 m, . . . ,y S m ∼ πθ(y|xm) for each query xm ∈ Dgold. Collect the APO comparison set DAPO = {(xm,ygoldm ,ysm)|(xm,ym) ∈ Dgold, 1 ≤ s ≤ S} Update rϕ with the APO RM loss:
LAPO-RM(rϕ) = LRanking(rϕ;DAPO) + β2LRanking(rϕ;DP).
Sample response y1l ,y 2 l , . . . ,y S l ∼ πθ(y|xl) for each LLM training query xl ∈ DQ. Calculate reward values for sampled responses rsl = rϕ(xl,y s l ).
Update πθ with scored samples {xl,ysl , rsl } with alignment methods such as RJS, RRHF, and DPO. end for

heading: Future Work
text: Moreover, APO verifies that LLMs can further benefit from adversarial games with other LLMs, highlighting the huge potential in developing future LLM selfimprovement and self-play methods