RainbowPO: A Unified Framework for Combining Improvements in Preference Optimization

## Abstract

Abstract Recently, numerous preference optimization algorithms have been introduced as extensions to the Direct Preference Optimization (DPO) family. While these methods have successfully aligned models with human preferences, there is a lack of understanding regarding the contributions of their additional components. Moreover, fair and consistent comparisons are scarce, making it difficult to discern which components genuinely enhance downstream performance.
In this work, we propose RainbowPO , a unified framework that demystifies the effectiveness of existing DPO methods by categorizing their key components into seven broad directions. We integrate these components into a single cohesive objective, enhancing the performance of each individual element. Through extensive experiments, we demonstrate that RainbowPO outperforms existing DPO variants. Additionally, we provide insights to guide researchers in developing new DPO methods and assist practitioners in their implementations.

## 1 Introduction

Reinforcement Learning with Human Feedback (RLHF) (Ouyang et al., ; Stiennon et al., ; Ziegler et al., ) has significantly contributed to the success of recently released Large Language Models (LLMs) such as InstructGPT (Ouyang et al., ), ChatGPT, and GPT4 (Achiam et al., ). However, RLHF is a complex and resource intensive process and requires training a reward model. An alternative to RLHF is Direct Preference Optimization (DPO) (Rafailov et al., ) that directly optimizes policies from pairwise preferences by minimizing a supervised learning loss objective, which is viewed as the maximum likelihood estimate for the reward model in RLHF. This approach allows DPO and similar other DPO variants to bypass the use of RL, resulting in faster speed of end-to-end training and better resource efficiency, while achieving comparable or superior performance to RLHF in downstream tasks such as summarization (Rafailov et al., ).

DPO and its success during training foundation models like LLama series (Dubey et al., ; Touvron et al., ), Mistral (Jiang et al., ), has garnered significant research attention in the LLM alignment space (Winata et al., ; Wang et al., ), leading to the development of various extensions. These include variants beyond pairwise ranking, such as KTO (Ethayarajh et al., ; Song et al., ), unified perspectives on loss parameterization, such as IPO (Azar et al., ) and GPO (Tang et al., ), distribution correction methods like RSO (Liu et al., ) and WPO (Zhou et al., ), and reference model-free alternatives, such as CPO (Xu et al., ), ORPO (Hong et al., ), and SimPO (Meng et al., ).
Each of these DPO variants claims to outperform the original DPO in downstream task evaluations by introducing specific components, or mathematically modifying the loss objective.
In the rest of the paper, we will refer to all DPO variants collectively as xPOs for simplicity.

Comparing these xPOs proposed in the literature is not always straightforward due to differences in the base model size and architecture,
the alignment datasets, the experimental setup as well as the evaluation metrics. Subsequently, it becomes difficult to assess the effectiveness and choose among different xPO methods given a problem. A brute force comparison across all existing methods is prohibitively expensive and inefficient.
Therefore, it is crucial that we study the performance characteristics of each proposed method in the literature by evaluating the xPOs’ performances under at least one convincing and representative setup.

Further, despite the success of the xPO family, a fundamental question remains unexplored:

$$ What are the components proposed in xPOs that actually improve the performance over DPO ? $$

Surprisingly, there is still a lack of comprehensive work studying the progress in the literature and summarizing the core practical components of these methods that lead to performance improvement of the DPO objective. To demystify the reasons for their effectiveness, we hypothesize that the main benefits of these methods stem from the combination of several mathematically orthogonal effective components. In this paper, we validate our hypothesis by decomposing the xPOs and identifying these orthogonal components upon DPO. We further assess their effectiveness through downstream task evaluations, ruling out the components that do not contribute to performance improvements.
Given these orthogonal identified beneficial components for preference optimization, a natural question arises:

$$ Can these individual components complement each other and be effectively combined? $$

Our question is largely motivated by the previous study Rainbow (Hessel et al., ) that explored improvements over Deep Q-Networks algorithm (DQN) (Mnih et al., ) in traditional Reinforcement Learning (RL). The summarization and comparsion in Hessel et al. () greatly enhances the understanding for improving DQN, and the resulting algorithm Rainbow, still serves as a benchmark (Raffin et al., ). However, such a study for RLHF is still underexplored. This shows a gap in the literature, that elicits an answer to the question of combining different xPO extensions evaluated in a comprehensive setting. To bridge this gap, we propose RainbowPO, a unified framework that integrates existing xPOs’ components, and deploys useful and essential components in a principled manner to achieve better performance. To conclude, our contributions in this paper are as follows:

- (1)
We conduct a comprehensive study on more than 10 offline representative variants of DPOs (xPOs) from a practical aspect by analyzing their loss functions for optimization. We conclude analyze several mathematically orthogonal directions along which these methods propose to optimize over the original DPO loss, analyze the usefulness of each method theoretically and empirically, and provide comparisons under the same representative setup.
- (2)
We identify and summarize seven broad components across all DPO extensions: length normalization, link function, margin / home advantage, reference policy, contextual scaling, rejection sampling optimization (RSO), and supervised fine-tuning (SFT) loss, and justify that four of them are effective through extensive hyper-parameters search, model training and evaluations. Additionally, we also propose a better way of formulating the reference policy by a linear mixing of the SFT policy and the margin, and demonstrate the advantage of this approach over using just reference policy (in DPO) or just margin (in SimPO).
- (3)
Finally, we propose RainbowPO(^1^11The trained \gradientHSBRainbowPO0,240,200240,240,200 will be released upon acceptance. The code will be released publicly.), a DPO variant that combines three essential and orthogonal components from existing xPOs. Combining other adjustment on training epochs and optimization hyper-parameters, we show that our algorithms perform the best among all open-sourced algorithms when tuning Llama3-8B-Instruct, as the best of our knowledge.
In the widely adopted benchmark Alpaca-Eval2, RainbowPO improves Llama3-8B-Instruct from 22.92% to 51.66% for Length Controlled Win Rate, with access to the offline preference dataset and no further online sampling.
We also perform an ablation study and show that all adopted elements in RainbowPO are indeed necessary to improve the performance over DPO and achieve the best result.

Related Work. Below we provide a (non-exhaustive) list of other relevant references to this work.

Compared to human feedback in original RLHF, existing works have improved the scalability by utilizing AI feedback (Bai et al., ; Lee et al., ). For such need of constructing better AI feedback, recent works also proposed various reward models for formulating better preference datasets, like PairRM (Jiang et al., ), ArmoRM (Wang et al., ), RRM (Liu et al., ), and RM benchmarks like Reward Bench (Lambert et al., ).

We also find works that target at understanding DPO methods related to our work. Liu et al. () studies the effect of reference policy in the preference optimization; Saeidi et al. () compare the performance of DPO, IPO, CPO, KTO for tuning Mistral 7B (Jiang et al., ) based models, and mainly studied the roles of SFT stage for alignment methods.

The rest of the paper is organized as follows. We provide backgrounds on RLHF and DPO in Section 2. In Section 3, we summarize the current directions in existing xPOs and the development of RainbowPO, followed by detailed experimental results in Section 4. Finally, we present our conclusion in Section 5.

## 2 Preliminaries and Motivation

In this section, we first briefly introduce RLHF and DPO as the foundation method, and then discuss on extensions of DPO (xPOs) to understand what are the components proposed in the literature.

RLHF starts with fine-tuning a pre-trained large language model by supervised learning on high-quality data for some downstream tasks of interest (e.g., dialogue, summarization, etc.), to acquire a model $\pi^{\mathrm{SFT}}$.
This step is referred to as the SFT phase. For instance, for training InstructGPT (Ouyang et al., ),
GPT-3 (Brown et al., ) is first fine-tuned on the given input prompt distribution. The second stage of RLHF is known as reward modeling, i.e., researchers collect preferences $\mathcal{D}=(x,y^{w},y^{l})$ on the generations of fine-tuned model $\pi^{\mathrm{SFT}}$, and learns a reward model $r^{*}(x,y)$ that could represent the quality or the rating of generation $y$ with respect to prompt $x$. The final step is policy optimization on $\pi_{\mathrm{SFT}}=\pi_{\mathrm{ref}}$, by maximizing a regularized reward to obtain the optimal policy model $\pi^{*}$ through reinforcement learning:

$$ $\displaystyle\max_{\theta}\mathbb{E}_{x\sim\mathcal{D}}\left[\mathbb{E}_{y\sim \pi_{\theta}(y\mid x)}\left[r^{*}(x,y)\right]-\beta\mathrm{KL}\left(\pi_{ \theta}(\cdot\mid x)\|\pi_{\mathrm{ref}}(\cdot\mid x)\right)\right]$ (1) $$

For ease of reference, we add a table of notations in Table 8 in Appendix C and more detailed description of RLHF in Appendix A.1.

### 2.1 Direct Preference Optimization (DPO)

One disadvantage of RLHF is that the RL step often requires substantial computational effort (e.g., to carry out PPO). The idea of DPO is to combine the reward model and RL in RLHF into a single objective, bypassing the computation in the RL step. Given the same preference pairs $\mathcal{D}=(x,y^{w},y^{l})$ utilized for reward modeling in RLHF, the DPO objective yields:

$$ $\min_{\theta}\mathcal{L}_{\mathrm{DPO}}\left(\pi_{\theta};\pi_{\mathrm{ref}} \right):=-\mathbb{E}_{\left(x,y_{w},y_{l}\right)\sim\mathcal{D}}\left[\log \sigma\left(\beta\log\frac{\pi_{\theta}\left(y_{w}\mid x\right)}{\pi_{\text{ ref }}\left(y_{w}\mid x\right)}-\beta\log\frac{\pi_{\theta}\left(y_{l}\mid x \right)}{\pi_{\text{ref }}\left(y_{l}\mid x\right)}\right)\right],$ (2) $$

where $\sigma(\cdot)$ is the sigmoid function and $\beta$ is a regularization parameter for tuning. DPO thus yields a supervised learning problem, and requires much less computation than the RL based RLHF. The objective in Equation 2 can be understood as maximizing the likelihood difference between the preference pair, making the model more likely to generate the preferred answers than unpreferred.

### 2.2 Motivation: Revisiting xPOs

Since DPO is proposed, there is huge interest in developing and improving DPO, leading to numerous xPOs. Different xPOs can be motivated by theoretical concerns like relaxing or extending preference distribution assumptions in IPO and Mallows DPO, human aware loss function in KTO, or from practical aspects like reference model-free alternatives, like CPO, ORPO and SimPO. We provide an non-exhaustive list in Table 7 in Appendix C for the ease of comparison.

Despite different motivations, xPOs all have a main loss objective that they optimize. We thus take the loss objectives as the first class citizen, and mathematically understand the parts that are commonly adopted or differ in xPOs. Before going into detailed categorization, we want to first argue that, in existing preference optimization literature, there lacks a work in revisiting and examining the DPO variants in their objectives mathematically and comprehensively. As a consequence, some papers may have implicitly proposed some designs for improvement and even didn’t highlight it.
As a motivating example, we revisit ORPO objective proposes to maximize an odd ratio difference (for an event A with probability $p$, the odds ratio is defined as $p/(1-p)$):

$$ $\mathcal{L}_{\mathrm{ORPO}}\left(\pi_{\theta}\right)=-\mathbb{E}\left[ \underbrace{\log p_{\theta}(y_{w}|x)}_{\mathcal{L}_{\text{ORPO-SFT}}} \underbrace{+\lambda\log\sigma\left(\log\frac{p_{\theta}(y_{w}|x)}{1-p_{\theta }(y_{w}|x)}-\log\frac{p_{\theta}(y_{l}|x)}{1-p_{\theta}(y_{l}|x)}\right)}_{ \lambda\cdot\mathcal{L}_{\text{ORPO-PO}}}\right],$ (3) $$

in which the expectation is for $\left(x,y_{w},y_{l}\right)\sim\mathcal{D}$, and $p_{\theta}(y|x)=\exp\left(\frac{1}{|y|}\log\pi_{\theta}(y|x)\right)$. Rewriting terms in $\mathcal{L}_{\text{PO}}$, we could derive a upper bound as (see proof in Appendix B.1):

$$ $\mathcal{L}_{\mathrm{ORPO-PO}}\leq-\mathbb{E}\log\sigma\left(\frac{1}{1-p_{ \theta}(y_{l}|x)}\underbrace{\left(\frac{1}{|y_{w}|}\log\pi_{\theta}(y_{w}|x)- \frac{1}{|y_{l}|}\log\pi_{\theta}(y_{l}|x)\right)}_{\Delta_{\theta}}\right):= \bar{\mathcal{L}}_{\text{PO}},$ (4) $$

if assuming $\Delta_{\theta}>0$ for all $x$. The upper bound $\bar{\mathcal{L}}_{\text{PO}}$ is sharp, in the sense that $\bar{\mathcal{L}}_{\text{PO}}-\mathcal{L}_{\text{PO}}=\mathcal{O}(\Delta_{
\theta})^{2}$; thus minimizing ORPO loss could be understood as CPO with length normalization and a contextual dependent $\beta$. Length Normalization is one of the key ideas adopted in SimPO–though proposed after ORPO. This thus calls for a comprehensive analysis of the core contributed elements in different xPOs so far, as many methods may already overlap in contributed directions without awareness, and bringing this out right away could possibly prevent repetitive work or efforts in the future.

Following similar analysis of different representative XPO methods for pairwise preferences, including DPO (Rafailov et al., ), IPO (Azar et al., ), CPO (Xu et al., ), GPO (Tang et al., ), RSO (Liu et al., ), ODPO (Amini et al., ), ORPO (Hong et al., ), Mallows-DPO (Chen et al., ), SimPO (Meng et al., ), we come up with seven broad categories, which is able to explain most popular DPO variants in the literature, as in Table 1, This provides a straightforward illustration of the main ideas and connections of existing methods. The meanings and details of the categories are elaborated in Section 3.

**Table 1: Mapping of xPOs with mathematically orthogonal components and validation results of their effectiveness by the downstream task evaluations.**
| Method | Length Norm. | Link Func. | Home Adv. | Ref. Policy | Contextual Scaling | RS | SFT Loss |
| --- | --- | --- | --- | --- | --- | --- | --- |
| DPO | $\times$ | logistic | $\times$ | SFT | $\times$ | $\times$ | $\times$ |
| SLiC-HF | $\times$ | hinge | $\times$ | SFT | $\times$ | $\times$ | $\checkmark$ |
| IPO | $\times$ | square | $\times$ | SFT | $\times$ | $\times$ | $\times$ |
| CPO | $\times$ | logistic | $\times$ | Free | $\times$ | $\times$ | $\checkmark$ |
| RSO | $\times$ | logistic / hinge | $\times$ | SFT | $\times$ | $\checkmark$ | $\times$ |
| ODPO | $\times$ | logistic | $\checkmark$ | SFT | $\times$ | $\times$ | $\times$ |
| ORPO | $\checkmark$ | logistic | $\times$ | Free | implicitly | $\times$ | $\checkmark$ |
| WPO | $\times$ | logistic | $\times$ | SFT | $\checkmark$ | $\times$ | $\times$ |
| Mallows-DPO | $\times$ | logistic | $\times$ | SFT | $\checkmark$ | $\times$ | $\times$ |
| SimPO | $\checkmark$ | logistic | $\checkmark$ | Free | $\times$ | $\times$ | $\times$ |
| \gradientHSBRainbowPO0,240,200240,240,200 | $\checkmark$ | logistic | $\times$ | mixing | $\checkmark$ | $\times$ | $\times$ |

## 3 RainbowPO : A Unified Framework

### 3.1 Component Descriptions

We first explain in detail about the components we categorized, after which we propose a generic framework, RainbowPO, to combine these components.

#### Length Normalization.

The literature has noticed a verbosity issue of DPO aligned models. To address this, one promising direction noticed in the literature is to incorporate explicit verbosity penalties, like in R-DPO (Park et al., ) and SimPO (Meng et al., ):

$$ $\displaystyle r^{\text{LR}}_{\theta}(x,y)=r_{\theta}(x,y)-\alpha|y|,\,\text{ and }\,r^{\text{LN}}_{\theta}(x,y)=\frac{1}{|y|}r_{\theta}(x,y),$ (5) $$

in which $r_{\theta}(x,y)=\log\frac{\pi_{\theta}(y|x)}{\pi_{\text{ref}}(y|x)}$ is the implicit reward model (Rafailov et al., ). From an optimization perspective, maximization with respect to $r^{\text{LN}}_{\theta}(x,y)$ is equivalent to $r^{\text{LR}}_{\theta}(x,y)$ with a specific $\alpha$ (might be prompt $x$ dependent). Why length normalization could help prevent the verbosity issues can be explained through examining the gradient of the loss respectively: $\nabla_{\theta}\mathcal{L}_{\mathrm{LN-DPO}}\left(\pi_{\theta};\pi_{\text{ref}
}\right)=$

$$ $-\beta\mathbb{E}\left[\sigma\left(r^{\text{LN}}_{\theta}\left(x,y_{l}\right)-r ^{\text{LN}}_{\theta}\left(x,y_{w}\right)\right)\left(\frac{1}{|y^{w}|}\nabla_ {\theta}\log\pi_{\theta}\left(y_{w}\mid x\right)-\frac{1}{|y^{l}|}\nabla_{ \theta}\log\pi_{\theta}\left(y_{l}\mid x\right)\right)\right],$ (6) $$

thus the gradient of length normalized DPO can be understood as taking a discount factor $\frac{1}{|y^{w}|}$ of the length for longer sequence. We also empirically justify the effectiveness of length normalization by comparing to the vanilla DPO trained models, and witness the consistent smaller average length, independent of the regularization constant $\beta$. See results of average length in Section 4.

#### Link Function.

SLiC-HF (Zhao et al., ) and GPO (Tang et al., ) both realized that the DPO objective could be understood as taking $f$ as $\log\sigma(\cdot)$ in:

$$ $\displaystyle\mathcal{L}_{\text{GPO}}=\mathbb{E}\left[f\left(\beta\log\frac{ \pi_{\theta}(y_{w}|x)}{\pi_{\text{ref}}(y_{w}|x)}-\beta\log\frac{\pi_{\theta}( y_{l}|x)}{\pi_{\text{ref}}(y_{l}|x)}\right)\right],$ (7) $$

thus unifying DPO, IPO, SLiC (without SFT loss). We did an exclusive parameter search for DPO and IPO separately, however, according to our experiments, DPO is empirically still better than IPO in validation metric as AlpacaEval for both evaluators (Llama3-70B and GPT4-1106 preview), despite the weaker assumption of preferences in IPO.

#### Home Advantage / Margin.

In SliC, IPO, ORPO, SimPO, there exists a term which also targets at encouraging the difference between the reward model difference. It is also referred in SimPO as the term of home advantage $\gamma$ (the terminology comes from an extension of the vanilla Bradley-Terry Model): $\operatorname{logit}(\operatorname{Prob}(i\text{ beats }j))=r_{i}-r_{j}-\gamma.$ Thus the likelihood could be written as:

$$ $\displaystyle p^{*}\left(y_{1}\succ y_{2}\mid x\right)$ $\displaystyle=\sigma\left(r^{*}\left(x,y_{1}\right)-r^{*}\left(x,y_{2}\right)- \gamma\right),$ (8) $$

which takes the losing prompt in a home advantage when $\gamma>0$. SimPO shows the effectiveness of this margin under the reference-free setup; however, when we adopt the margin for vanilla DPO (i.e. with the reference policy) with the optimal $\beta$, we do not witness an increase of the performance when adjusting the margin, either further adopting DPO with length normalization or not. In Figure 1(a), the performance steadily decreases when increasing the margin in DPO. This questions the true explanation about the effectiveness of the margin term in SimPO. We provide the answer as understanding margin as a reference policy.

#### Reference Policy.

DPO takes the SFT policy as the reference policy motivated by the standard RLHF pipeline. However, recently proposed methods like CPO, ORPO, and SimPO (Xu et al., ; Hong et al., ; Meng et al., ) all suggested a reference-free objective could yield the same or even better performance. CPO and ORPO further utilized an extra SFT loss to force regularization, while for SimPO, such regularization is not enforced. Given our prior examination that home advantage can hardly improve over DPO, we argue that the margin term in SimPO should be understood as a term for “reference policy” instead of the “home advantage”.

Concretely, we could hypothesize that there exists a “good policy” $\pi_{\gamma}$ such that, for each prompt and preference pairs in the dataset, the normalized log likelihood ratio of preferred response to non-preferred response is a positive constant, which we denote as $\pi_{\gamma}$. $\pi_{\gamma}$’s normalized implicit reward model is perfect at in-distribution pairwise classification and yields $\frac{\pi_{\gamma}\left(y_{w}\mid x\right)^{1/|y^{w}|}}{\pi_{\gamma}\left(y_{l
}\mid x\right)^{1/|y^{l}|}}=\exp(\gamma)$ for any $x$. If so, the loss of SimPO could be rewritten as the DPO with length normalization and a different reference policy:

$$ $\displaystyle\mathcal{L}_{\text{SimPO}}$ $\displaystyle=-\underset{\left(x,y_{w},y_{l}\right)\sim\mathcal{D}}{\mathbb{E} }\log\sigma\left(\frac{\beta}{|y_{w}|}\log\pi_{\theta}(y_{w}|x)-\frac{\beta}{| y_{l}|}\log\pi_{\theta}(y_{l}|x)-\beta\cdot\gamma\right),$ (9) $\displaystyle=-\underset{\left(x,y_{w},y_{l}\right)\sim\mathcal{D}^{*}}{ \mathbb{E}}\log\sigma\left(\frac{\beta}{|y^{w}|}\log\frac{\pi_{\theta}\left(y_ {w}\mid x\right)}{\pi_{\gamma}\left(y_{w}\mid x\right)}-\frac{\beta}{|y^{l}|} \log\frac{\pi_{\theta}\left(y_{l}\mid x\right)}{\pi_{\gamma}\left(y_{l}\mid x \right)}\right).$ $$

This transformation motivates us to further propose a new mechanism which we call as mixing reference policy. If taking $\pi_{\text{sft}}$ as the reference policy is too conservative (not strong enough), and taking $\pi_{\gamma}$ policy can help improve the performance but totally neglects the original SFT model implicit preference, can we benefit from a mixing of these two policies? The answer is YES. Consider a exponential mixing (or a linear combination of the loglikehood) with $\alpha\in[0,1]$, defined as:

$$ $\pi_{\alpha}(y\mid x)\propto\pi^{\alpha}_{\text{ref}}(y\mid x)\cdot\pi^{1- \alpha}_{\gamma}(y\mid x).$ (10) $$

If we use $\pi_{\alpha}$ as the reference policy, we can write $\mathcal{L}_{\text{LN-DPO}}(\pi_{\theta};\pi_{\alpha})$ as a pratical form:

$$ $-\underset{\left(x,y_{w},y_{l}\right)\sim\mathcal{D}}{\mathbb{E}}\log\sigma \left(\beta\log\frac{\pi_{\theta}\left(y_{w}\mid x\right)^{1/|y^{w}|}}{\pi_{ \theta}\left(y_{l}\mid x\right)^{1/|y^{l}|}}-\alpha\beta\log\frac{\pi_{\text{ ref }}\left(y_{w}\mid x\right)^{1/|y^{w}|}}{\pi_{\text{ref }}\left(y_{l}\mid x \right)^{1/|y^{l}|}}-(1-\alpha)\gamma\right).$ (11) $$

Notice that $\mathcal{L}_{\text{LN-DPO}}(\pi_{\theta};\pi_{\alpha})=\mathcal{L}_{\text{LN-
DPO}}(\pi_{\theta};\pi_{0})=\mathcal{L}_{\text{SimPO}}(\pi_{\theta})$, thus SimPO is an instance of mixing policy by taking $\alpha=0$; $\mathcal{L}_{\text{LN-DPO}}(\pi_{\theta};\pi_{1})=\mathcal{L}_{\text{LN-DPO}}(
\pi_{\theta})$, thus $\alpha=1$ corresponds to DPO applied with length normalization. Because of convexity of $-\log(\sigma(\cdot))$, the mixing policy objective can also be understood as a lower bound of the linear combination of the LN-DPO loss and SimPO loss:

$$ $\mathcal{\mathcal{L}_{\text{LN-DPO}}(\pi_{\theta};\pi^{\alpha})}\leq\alpha \cdot\mathcal{\mathcal{L}_{\text{LN-DPO}}(\pi_{\theta};\pi_{\text{ref}})}+(1- \alpha)\mathcal{\mathcal{L}_{\text{SimPO}}(\pi_{\theta};\pi_{\text{ref}})}.$ (12) $$

According to our experiment results, we indeed find that there exists $\alpha\in(0,1)$ that performs better than both sides, see Figure 1(b). Recent work (Liu et al., ) analyze the role of reference model, and argue that stronger reference model could benefit DPO; our finding is consistent, as we further explicitly design a choice of better reference model for better performance.

#### Rejection Sampling.

Since the proposal of DPO, there is controversy on the exact equivalence of DPO and RLHF. RSO (Liu et al., ), further pointed out that the data should be generated from the optimal policy if treating DPO objective as maximum likelihood estimation. Thus RSO adopts a statistical rejection sampling for sampling preference dataset generated by the optimal policy to mitigate this distribution difference in DPO.

Figure: Algorithm 1 RS^+ for preferences formulation.

To address the intrinstic different variance schedules of reward model for different prompts, and stablize the process for formulating the preference dataset, we also adopt a modified version of RSO by computing the percentile reward (or the ranking reward) in the whole generation set instead of utilizing the true reward, which we found that can stabilize the generation and yield better results when further applied with DPO, as in Algorithm 1. Similar to in RSO (Liu et al., ), we search the best temperature hyper-parameter for RSO through the downstream task performance as the validation metric, which we detail in Figure 1(c). We then use the empirically best performed temperature constant $\tau$ to formulate the prefer-
ence dataset as $\mathcal{D}_{\text{RS}}$.

#### Contextual Scaling.

Existing work also considered the contextual difference: some preference pairs might be of higher uncertainty or have more dispersion. In this paper, we adopt the idea of Mallows-DPO by introducing a contextual scaling factor $\phi(x)$ on the likelihood difference,
e.g., for DPO:

$$ $-\log\sigma\left(\phi(x)\left[\beta\log\frac{\pi_{\theta}(y_{w}|x)}{\pi_{\text {ref}}(y_{w}|x)}-\beta\log\frac{\pi_{\theta}(y_{l}|x)}{\pi_{\text{ref}}(y_{l}| x)}\right]\right),$ (13) $$

which is motivated by the Mallows ranking model and the form of
Mallows-DPO (Chen et al., ). In Mallows-DPO, $\phi(x)$ corresponds to a normalized predictive entropy of the preference pair $(x,y^{w},y^{l})$:

$$ $\phi(x)=-\log\left(\frac{\sum_{i=1}^{N-1}\left[H_{\pi_{\mathrm{ref}}}\left(Y_{ i+1}\mid Y_{i}=y_{i}^{w}\right)+H_{\pi_{\mathrm{ref}}}\left(Y_{i+1}\mid Y_{i}= y_{i}^{l}\right)\right]}{2\log n}\right).$ (14) $$

#### SFT Loss.

SFT loss is straightforward by adding extra SFT loss term on the winning answer, or a reference answer (for regularization, which appears for reference-free methods like CPO and ORPO); however, according to our experiments, we find that adding SFT loss could largely degrade the performance.

### 3.2 Unified Formulation: Rainbow

Combining the advances proposed above, we propose a following preference optimization objective, for which we refer our method as RainbowPO:

$$ $\mathcal{L}_{\textsc{RainbowPO}}\left(\pi_{\theta};\pi_{\mathrm{ref}}\right):= -\underset{\left(x,y_{w},y_{l}\right)\sim\mathcal{D}^{*}}{\mathbb{E}}f\left[ \phi(x)\left(\frac{\beta}{|y^{w}|^{\eta}}\log\frac{\pi_{\theta}\left(y_{w}\mid x \right)}{\pi_{\alpha}\left(y_{w}\mid x\right)}-\frac{\beta}{|y^{l}|^{\eta}} \log\frac{\pi_{\theta}\left(y_{l}\mid x\right)}{\pi_{\alpha}\left(y_{l}\mid x \right)}\right)\right],$ (15) $$

in which $\eta\in\{0,1\}$, $\pi_{\alpha}$ is as defined in Equation 10. The preference dataset $\mathcal{D}^{*}$ can be $\mathcal{D}_{\text{RS}}$, if we have access to the reward model’s true value, which means that the preference dataset is formulated by by rejection sampling from the original dataset’s prompts as mentioned in Algorithm 1. If the reward model is black-box oracle, namely we cannot access the true reward value, we will always utilize the usual formulation way of preference dataset $\mathcal{D}$, with the details in the Experiment Section.

Like all other XPO methods, to achieve a final satisfactory performance, RainbowPO can introduce an extensive amount of hyper-parameter search for the best performing $f$, $\alpha$, $\beta$, $\gamma$ and whether $\eta=1$. For efficient hyper-parameter search, we conducted a greedy search method with the help of our framework and decomposition of effective elements: we search for the best hyper-parameters for those that affects the performance in the most when we gradually add designs to the preference optimization methods. For example, when adding length normalization to the methods, we only search for the best hyper-parameter for the regularization parameter $\beta$, and will fix the learning rate and all the training args, which prevents the parameter searching space from exploding.

## 4 Experiments

To evaluate the performance of the xPOs algorithms, we conducted extensive experiments on training models with various xPOs configurations and compared their instruction-following capabilities.

#### Experimental Setup.

We choose Llama3-8B-Instruct(^2^22[https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct)) as our model base to fine tune, mainly because that aligning this widely adopted and flagship instruct model is of great interest to the whole community and meets the standard as a representative setup for alignment. It can also help mitigate the uncertainty from probably not perfectly supervised fine-tuned models.

For evaluation metric, we use widely adopted benchmark Alpaca Eval2, which is composed of 805 questions and evaluate the instruction following capability of the model. The win rate is by default annotated by GPT4 through LLM-as-a-judge, and the resulting win rate has a 68.5% consistency according to official AlpacaEval website. To cross validate the effectiveness of the model and mitigate possible bias of GPT4, we also adopt Llama3-70B instruct as the judge, which is reported to have a 67.5% win rate consistency to humans.

For formulating the preference dataset $\mathcal{D}$, we follow the standard RLHF pipeline by directly generating answers from the model (which is thus an on-policy dataset, but the algorithm is still offline) and get AI feedbacks as in SimPO (Meng et al., ): we generate 5 answers from Llama3-8B-Instruct for each prompt in UltraFeedback (Cui et al., ), rank them with scores evaluated by ArmoRM (Wang et al., ),
and choose the best/worst one as winning/losing answer to form the preference pairs. For training, we adopted the popular library Transformer Reinforcement Learning (TRL(^3^33[https://huggingface.co/docs/trl/index](https://huggingface.co/docs/trl/index))), which already implemented most aforementioned xPOs algorithms and make everything under the same backend and easy to reproduce. If not specified, we train the model with 3 training epochs, which typically yields better performance for each xPOs according to our replication.

### 4.1 Effectiveness of Different Components

#### Individual Components Results.

We first study the effectiveness of adding individual components. We use $+$ to denote that only the component(s) after is added to DPO baseline as in Table 2. From the results, we could notice that some components may not provide firm improvement over the baseline not matter being added individually or combined. For example, for home advantage term, we tune different values under the best performed $\beta$ for DPO, and also always witness a degradation in the performance, see Figure 1(a). For link function, we examine the square loss in IPO and also cannot see performance gain over the DPO baseline. Other components (LN, Mixing reference policy, CS) indeed help improve the metric even added individually. Compared to SimPO, using mixing reference policy yields also better results as in Figure 1(b). The average win rate gain is reported in the last column.

**Table 2: Model performance results on each component. (based on training-epochs = 3)**
| Models | AlpacaEval (GPT4) | AlpacaEval (Llama3-70B) |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | LC WR (%) | $\Delta$ (%) | WR (%) | $\Delta$ (%) | LC WR (%) | $\Delta$ (%) | WR (%) | $\Delta$ (%) | Avg. $\Delta$ (%) |
| Base model | 41.88 | - | 42.29 | - | 57.78 | - | 57.96 | - | - |
| + Length Norm. (LN) | 44.27 | 2.39 | 42.37 | 0.08 | 61.37 | 3.59 | 58.94 | 0.98 | + 1.76 |
| + Ref. Policy Mixing (Mix) | 40.18 | -1.7 | 41.25 | -1.04 | 60.67 | 2.89 | 57.95 | -0.01 | + 0.04 |
| + Contextual Scaling (CS) | 41.14 | -0.74 | 41.44 | -0.85 | 60.06 | 2.28 | 57.90 | -0.06 | + 0.16 |
| + Link Function (LF) | 39.53 | -2.35 | 39.07 | -3.22 | 58.13 | 0.35 | 56.34 | -1.62 | - 1.21 |
| + Home Advantage (HA) | 41.70 | -0.18 | 39.85 | -2.44 | 59.01 | 1.23 | 56.41 | -1.55 | - 0.74 |
| + Rejection Sampling (RSO) | 42.87 | 0.99 | 42.50 | 0.21 | 58.86 | 1.08 | 56.02 | -1.94 | + 0.09 |
| Base model + LN | 44.27 | - | 42.37 | - | 61.37 | - | 58.94 | - | - |
| + LN + Mix | 47.45 | 3.18 | 45.89 | 3.52 | 61.91 | 0.54 | 58.07 | -0.87 | + 1.59 |
| + LN + CS | 45.92 | 1.65 | 42.36 | -0.01 | 61.88 | 0.51 | 58.20 | -0.74 | + 0.35 |
| + LN + HA | 42.77 | -1.50 | 41.38 | -0.99 | 60.99 | -0.38 | 59.78 | 0.84 | - 0.51 |
| + LN + RS | 43.22 | -1.05 | 41.96 | -0.41 | 61.03 | -0.34 | 57.02 | -1.92 | - 0.93 |
| + LN + SFT Loss | 39.90 | -4.37 | 38.66 | -3.71 | 60.42 | -0.95 | 58.94 | 0.00 | - 2.26 |

Figure: (a) Effects of Home Advantage / Margin.
Refer to caption: x1.png

#### Components Combination Results.

Given the effectiveness of length normalization, we further test the combination of LN and other components. We do find that mixing policy could help improve the performance much more remarkablly when combined with LN-DPO than just DPO: it provides a 1.6% win rate extra gain compared to only 0.04% on DPO. However, we found that the RSO can improve DPO, but will yield worse performance when applied with length normalization. Thus, we do find that despite that these elements are apparently mathematically orthogonal, they are not empirically independent. Given the positive results and effectiveness of length normalization, mixing reference policy and contextual scaling, we propose \gradientHSBRainbowPO0,240,200240,240,200, as the combination of these three elements. We then examine the effectiveness of our method and each elements by gradually combining the elements one by one and greedy search of the best hyper-parameters. We finally achieve a 51.66% win rate for AlpacaEval2, surpassing the GPT4-1106 preview.

**Table 3: Evaluation of RainbowPO by adding new components consecutively.**
| Models | AlpacaEval (GPT4) | AlpacaEval (Llama3-70B) |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| \gradientHSBRainbowPO0,240,200240,240,200 | LC WR (%) | $\sigma$ | WR (%) | $\sigma$ | LC WR (%) | WR (%) | avg length ($\downarrow$) |
| Base model | 41.88 | 0.77 | 42.29 | 1.46 | 57.78 | 57.96 | 2,169 |
| $\oplus$ Length Norm. | 44.27 | 0.75 | 42.37 | 1.45 | 61.37 | 58.94 | 1,942 |
| $\oplus$ Ref. Policy Mixing | 47.45 | 0.70 | 45.89 | 1.49 | 61.91 | 58.07 | 1890 |
| $\oplus$ Warm-up Adjustment | 48.52 | 0.80 | 45.88 | 1.45 | 63.37 | 59.95 | 1,919 |
| $\oplus$ Contextual Scaling | 51.66 | 0.78 | 47.92 | 1.49 | 63.94 | 59.69 | 1,878 |

#### Ablations on RainbowPO.

We also conduct an ablation study of our proposed RainbowPO algorithm. All components of our proposed in our algorithm is useful, as in Table 4, for which we use $\oplus$ to denote that the methods are based on composition of the method on previous line and new elements in Table 4. We notice that adding length normalization is indeed important and of the most critical importance among the components for RainbowPO.

**Table 4: Ablation study of the newly proposed elements in RainbowPOwithout the use of a trained reward model.**
| Models | AlpacaEval (GPT4) | AlpacaEval (Llama3-70B) |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  | LC WR (%) | $\sigma$ | WR (%) | $\sigma$ | LC WR (%) | WR (%) | avg length ($\downarrow$) |
| \gradientHSBRainbowPO0,240,200240,240,200 | 51.66 | 0.78 | 47.92 | 1.49 | 63.94 | 59.69 | 1,878 |
| $-$ Ref. Policy Mixing | 50.52 | 0.78 | 47.49 | 1.46 | 64.64 | 60.43 | 1,886 |
| $-$ Contextual Scaling | 48.40 | 0.80 | 44.57 | 1.47 | 60.90 | 56.46 | 1,843 |
| $-$ Length Normalization | 45.68 | 0.78 | 42.43 | 1.47 | 57.43 | 58.01 | 2108 |

### 4.2 Comparison with Baseline Methods

Table 6 shows the comparison between \gradientHSBRainbowPO0,240,200240,240,200 with the baselines. For a fair comparsion, we first compare RainbowPO with the baselines in one training epoch, shown in Table 5, RainbowPO performs the best and also beats SimPO under GPT4 as a judge while achieving lower average length.

**Table 5: Methods comparison under one training epoch.**
| Models | AlpacaEval (GPT4) | AlpacaEval (Llama3-70B) |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | LC WR (%) | WR (%) | $\sigma$ | LC WR (%) | WR (%) | avg length ($\downarrow$) |
| DPO (Rafailov et al., ) | 37.95 | 37.36 | 1.42 | 55.46 | 54.03 | 1,989 |
| IPO (Azar et al., ) | 34.80 | 34.52 | 1.40 | 52.67 | 50.93 | 1,956 |
| KTO (Ethayarajh et al., ) | 35.61 | 33.19 | 1.38 | 55.94 | 51.74 | 1,876 |
| CPO (Xu et al., ) | 31.89 | 34.92 | 1.38 | 53.33 | 54.84 | 2,155 |
| ORPO (Hong et al., ) | 22.91 | 22.59 | 1.24 | 48.41 | 45.90 | 1,914 |
| SimPO (Meng et al., ) | 47.96 | 41.17 | 1.44 | 61.94 | 54.22 | 1,730 |
| \gradientHSBRainbowPO0,240,200240,240,200 (1 epoch) | 48.08 | 42.53 | 1.43 | 61.36 | 54.60 | 1,683 |

When we increased the training epoch to 3, interestingly, we also noticed that the same phenomenon as what (Meng et al., ) reported: SimPO rarely benefits from more epochs of training. However, RainbowPO and DPO both gets an increase in the winning rate after another two epochs of training, making the RainbowPO get a 51.66% win rate against GPT4 under GPT4 as a judge. This advantage not only benefits the final performance, but also might play larger impact when the alignment dataset is small or expensive to collect and will be beneficial to reuse, which is quite common in reality.

**Table 6: Methods comparison under three training epochs. ^∗Hyper-parameters are further adjusted for the best performance.**
| Models | AlpacaEval (GPT4) | AlpacaEval (Llama3-70B) |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | LC WR (%) | WR (%) | $\sigma$ | LC WR (%) | WR (%) | avg length ($\downarrow$) |
| DPO^∗ (Rafailov et al., ) | 43.65 | 43.94 | 1.46 | 60.13 | 58.20 | 2,284 |
| SimPO^∗ (Meng et al., ) | 48.40 | 44.57 | 1.47 | 60.90 | 56.46 | 1,843 |
| \gradientHSBRainbowPO0,240,200240,240,200 (3 epochs) | 51.66 | 47.92 | 1.49 | 63.94 | 59.69 | 1,878 |

### 4.3 Limitations and Future Work

#### Broader tasks.

In this paper, we focus our evaluation on models trained with LLama3-8B Instruct as the base model. Exploring other models of varying sizes, such as Gemma (Team et al., ) or Mistral (Jiang et al., ), could possibly enhance the generalizability of our findings. It will also be beneficial if we could repeat the pipelines and compare the algorithms’ performance on other LLM evaluation metrics, like arena-hard or MT-bench, though MT-bench is known to be less tinguishable for RLHF algorithms. Other directions include benchmarking the effectiveness of alignment algorithms on improving other capabilities of LLM other than instruction following, like reasoning Xiong et al. (). However, due to constraints in computing resources and time, we defer this investigation to future work. Nevertheless, we believe that our work provides a unified and comprehensive framework for helping to find the best preference optimization algorithms, and further pushing the boundary of offline RLHF for LLMs.

#### Ideas from other xPOs .

We were not able to explore other aspects of existing DPO variants in detail, and there might be still promising candidates in further improving the preference of RainbowPO. Some methods that propose to update the reference policy dynamically: sDPO (Kim et al., ), TR-DPO (Gorbatovski et al., ). Additionally, we also recognize the recent literature in pursuing online methods, such as online DPO (Guo et al., ) or iterative DPO (Yuan et al., ; Xiong et al., ), which provide valuable insights on possibly further improving the downstream task performance: we will pursue them in future research. Other extensions beyond RLHF include, Nash Learning from human feedback (Munos et al., ), and self-play preference optimization (Chen et al., ).

#### Demystifying observations.

We also made some interesting observations in the paper, which we fail to find proper mathematical explanations and may boost further research. For example, the RainbowPO objective could benefit much more than SimPO objective when increasing the training epochs, but the mathematical reasons for such phenomenons are still unknown. In addition, we found some mathematically orthogonal components are actually not empirically independent, for example, RSO can improve DPO, but can not be readily combined with other components like length normalization. It is also interesting to see the some combination of components reach effects “$1+1>2$”; it will be interesting to understand the deeper underlying reasons and could potentially lead to better algorithms.

## 5 Conclusion

In this paper, we propose RainbowPO, a comprehensive framework that demystifies and enhances existing DPO methods through the integration of key components into a unified objective. Our findings highlight the effectiveness of length normalization, reference policy mixing, and contextual scaling, while also highlighting the promise of warm-up adjustments. However, the selective application of rejection sampling and home advantage is not providing incremental improvements when paired with the other methods. By demonstrating that these enhancements can coexist within a single algorithm to achieve state-of-the-art performance, we pave the way for future research and practical applications. We aim for this work to serve as a foundation for refining DPO methodologies and to inspire further exploration of untested components for integrated agents.

## Acknowledgements

We thank Stephen Rawls, Supriyo Chakraborty and Kartik Balasubramaniam for helpful feedbacks and pointers to relevant works. We also thank Benjamin Therien, Ashwinee Panda, Zain Sarwar, Akshaj Kumar Veldanda and Andrei Mircea for helpful discussions.

Wenpin Tang and Hanyang Zhao are supported by NSF grant DMS-2206038, a start-up grant at Columbia University, and the Columbia Innovation Hub grant. The works of Hanyang Zhao and David D. Yao are part of a Columbia-CityU/HK collaborative project that is supported by InnotHK Initiative, The Government of the HKSAR and the AIFT Lab.

## Appendix A Background on RLHF and RL

### A.1 RLHF

RLHF (Ouyang et al., ; Stiennon et al., ; Ziegler et al., ).
On top of $\pi^{\mathrm{SFT}}$, RLHF is proposed to serve as the next step to conduct further fine-tuning to generate high-quality outputs as judged by humans.
Given a generative model $\pi$, the model $\pi$ is prompted with prompts $x$ to produce pairs of answers (or, “completions”),
$\left\{y_{1},y_{2}\right\}\sim\pi(y\mid x)$,
which are then presented to human labelers
who express preferences for one completion over the other.
Denote by $y_{w}\succ y_{l}\mid x$,
meaning that $y_{w}\in\left\{y_{1},y_{2}\right\}$ is preferred over $y_{l}\in\left\{y_{1},y_{2}\right\}$. The preferences are assumed to be generated by some latent reward model $r^{*}(x,y)$,
which we do not have access to.
Based on the collected preference data $\{x^{(i)},y_{w}^{(i)},y_{l}^{(i})\}_{i=1}^{N}$, RLHF consists of first learning a reward model $r(x,y)$,
followed by learning a policy $\pi_{r}(y\mid x)$
in which the prompt $x$ is the state, and the completion $y$ is the action.

(a) Reward Model.
To capture the underlying human preferences,
RLHF assumes the Bradley-Terry model (Bradley & Terry, ) that stipulates the pairwise preference distribution:

$$ $p^{*}\left(y_{1}\succ y_{2}\mid x\right):=\frac{\exp\left(r^{*}\left(x,y_{1} \right)\right)}{\exp\left(r^{*}\left(x,y_{1}\right)\right)+\exp\left(r^{*} \left(x,y_{2}\right)\right)}=\sigma\left(r^{*}\left(x,y_{1}\right)-r^{*}\left( x,y_{2}\right)\right),$ (16) $$

where $\sigma(\cdot)$ is the sigmoid function.
Given access to a static dataset of comparisons
$\mathcal{D}=\{x^{(i)},y_{w}^{(i)},y_{l}^{(i)}\}_{i=1,\ldots,N}$,
RLHF seeks to approximate the latent reward $r^{*}(x,y)$ by a family of functions $\{r_{\psi}(x,y)\}_{\psi}$,
and estimate the parameters by minimizing the (negative) log-likelihood loss
$\min_{\psi}\mathcal{L}\left(r_{\psi},\mathcal{D}\right):=-\mathbb{E}_{\left(x,
y_{w},y_{l}\right)\sim\mathcal{D}}\left[\log\sigma\left(r_{\psi}\left(x,y_{w}
\right)-r_{\psi}\left(x,y_{l}\right)\right)\right]$.
Denote by $r_{\psi_{*}}(x,y)$ the solution to this problem.

(b) RL. The learned reward function $r_{\psi_{*}}(x,y)$ is then used to provide feedback to the language model.
More precisely, the following KL-regularized RL problem is considered:

$$ $\displaystyle\max_{\pi}\mathbb{E}_{x\sim\mathcal{D}}\left[\mathbb{E}_{y\sim\pi (y\mid x)}\left[r_{\psi_{*}}(x,y)\right]-\beta\mathrm{KL}\left(\pi(\cdot\mid x )\|\pi_{\mathrm{ref}}(\cdot\mid x)\right)\right]$ (17) $$

where $\beta>0$ is a hyper-parameter controlling the deviation from the reference policy
$\pi_{\mathrm{ref}}=\pi^{\mathrm{SFT}}$.
The regularization is important as it prevents deviating too far from the SFT model that is trained to conform to the true preference,
while maintaining the generation diversity to avoid mode-collapsing to a single high-reward answer.
In view of equation 17, RLHF uses the reward function
$r(x,y)=r_{\psi}(x,y)-\beta\left(\log\pi(y\mid x)-\log\pi_{\text{ref }}(y\mid x
)\right)$,
and solves the RL problem by proximal policy optimization (PPO) (Schulman et al., ).

### A.2 DPO

One disadvantage of RLHF is that the RL step often requires substantial computational effort (e.g., to carry out PPO). The idea of DPO is to combine the reward model and RL in RLHF into a single objective, bypassing the computation in the RL step. The key realization is that given a reward function $r(x,y)$,
the RL problem in equation 17
has a closed-form solution
$\pi_{r}(y\mid x)=\frac{1}{Z(x)}\pi_{\text{ref }}(y\mid x)\exp\left(\frac{1}{
\beta}r(x,y)\right)$,
where $Z(x)=\sum_{y}\pi_{\text{ref }}(y\mid x)\exp\left(\frac{1}{\beta}r(x,y)\right)$.
Rewrite the above as
$r(x,y)=\beta\log\frac{\pi_{r}(y\mid x)}{\pi_{\text{ref }}(y\mid x)}+\beta\log Z
(x)$.
Through this change of variables, the latent reward $r^{*}(x,y)$ can be expressed in terms of the optimal
policy $\pi^{*}(y\mid x)$,
the reference policy $\pi_{\text{ref }}(y\mid x)$
and a constant $Z^{*}(x)$.
Substituting this $r^{*}$ expression into equation 16 yields:

$$ $p^{*}\left(y_{1}\succ y_{2}\mid x\right)=\sigma\left(\beta\log\frac{\pi^{*} \left(y_{1}\mid x\right)}{\pi_{\text{ref }}\left(y_{1}\mid x\right)}-\beta\log \frac{\pi^{*}\left(y_{2}\mid x\right)}{\pi_{\text{ref }}\left(y_{2}\mid x \right)}\right),$ (18) $$

where $Z^{*}(x)$ cancels out.
the preference distribution only depends on $\pi^{*}(y\mid x)$ and $\pi_{\text{ref }}(y\mid x)$.
The expression in equation 18 motivates the DPO objective:

$$ $\min_{\theta}\mathcal{L}_{\mathrm{DPO}}\left(\pi_{\theta};\pi_{\mathrm{ref}} \right):=-\mathbb{E}_{\left(x,y_{w},y_{l}\right)\sim\mathcal{D}}\left[\log \sigma\left(\beta\log\frac{\pi_{\theta}\left(y_{w}\mid x\right)}{\pi_{\text{ ref }}\left(y_{w}\mid x\right)}-\beta\log\frac{\pi_{\theta}\left(y_{l}\mid x \right)}{\pi_{\text{ref }}\left(y_{l}\mid x\right)}\right)\right],$ (19) $$

## Appendix B Proofs and Details

### B.1 Proof of ORPO upper bound in equation 4

Here we prove that the part of preference optimization in ORPO’s loss yields an upper bound which has instinct connection to SimPO loss, specifically the idea of length normalization.

###### Theorem 1 .

Assume that the normalized implicit reward model difference for preference pairs:

$$ $\Delta_{\theta}(x,y^{w},y^{l})=\frac{1}{|y_{w}|}\log\pi_{\theta}(y_{w}|x)- \frac{1}{|y_{l}|}\log\pi_{\theta}(y_{l}|x)\geq 0$ $$

almost surely. Then for the part of preference optimization
in ORPO loss, i.e.

$$ $\mathcal{L}_{\text{ORPO-PO}}\left(\pi_{\theta}\right)=-\mathbb{E}_{(x,y^{w},y^ {l})\sim\mathcal{D}}\left[\log\sigma\left(\log\frac{p_{\theta}(y_{w}|x)}{1-p_{ \theta}(y_{w}|x)}-\log\frac{p_{\theta}(y_{l}|x)}{1-p_{\theta}(y_{l}|x)}\right) \right],$ (20) $$

has an upper bound such that

$$ $\mathcal{L}_{\mathrm{ORPO-PO}}\leq-\mathbb{E}\log\sigma\left(\frac{1}{1-p_{ \theta}(y_{l}|x)}\left(\frac{1}{|y_{w}|}\log\pi_{\theta}(y_{w}|x)-\frac{1}{|y_ {l}|}\log\pi_{\theta}(y_{l}|x)\right)\right).$ (21) $$

###### Proof.

Since $-\log\sigma(\cdot)$ is a monotone decreasing function, when $1>x>y>0$, it suffices to prove that for any $x$,

$$ $\log\left(\frac{x}{1-x}\right)-\log\left(\frac{y}{1-y}\right)\geq\frac{1}{1-y} \log\left(\frac{x}{y}\right),$ (22) $$

in which $x=p_{\theta}(y_{w}|x)$, $y=p_{\theta}(y_{l}|x)$. The inequality is equivalent to:

$$ $f(x):=\log\left(\frac{x}{1-x}\right)-\frac{1}{1-y}\log\left(\frac{x}{y}\right) \geq\log\left(\frac{y}{1-y}\right).$ (23) $$

Taking gradient of $f(x)$ with respect to $x$, we have:

$$ $f^{\prime}(x)=\frac{1}{x}+\frac{1}{1-x}-\frac{1}{1-y}\cdot\frac{1}{x}=\frac{1} {x(1-x)}-\frac{1}{x(1-y)}\geq 0.$ $$

Moreover, we have $f(y)=\log(\frac{y}{1-y})$,
which yields the desired result.
∎

###### Theorem 1 .

###### Proof.

### B.2 Mathematical explanation of Different XPOs

Most xPOs’ categories are straightforward by directly checking their loss objective, see Table 7. Less explainable is ORPO, which we have proved in Theorem 1.

## Appendix C Miscellaneous

**Table 7: Various preference optimization DPO objectives. The table is inspired from Meng et al. () and Winata et al. ().**
| Method | Objective |
| --- | --- |
| DPO | $-\log\sigma\left(\beta_{\text{reg}}\log\frac{\pi_{\theta}(y_{w}|x)}{\pi_{\text<br>{ref}}(y_{w}|x)}-\beta_{\text{reg}}\log\frac{\pi_{\theta}(y_{l}|x)}{\pi_{\text<br>{ref}}(y_{l}|x)}\right)$ |
| IPO | $\left(\beta_{\text{reg}}\log\frac{\pi_{\theta}(y_{w}|x)}{\pi_{\text{ref}}(y_{w<br>}|x)}-\beta_{\text{reg}}\log\frac{\pi_{\theta}(y_{l}|x)}{\pi_{\text{ref}}(y_{l<br>}|x)}-\frac{1}{2}\right)^{2}$ |
| $f$-DPO | $-\log\sigma\left(\beta_{\text{reg}}f^{\prime}\left(\frac{\pi_{\bm{\theta}}<br>\left(y_{w}\mid x\right)}{\pi_{\mathrm{ref}}\left(y_{w}\mid x\right)}\right)-<br>\beta_{\text{reg}}f^{\prime}\left(\frac{\pi_{\bm{\theta}}\left(y_{l}\mid x<br>\right)}{\pi_{\mathrm{ref}}\left(y_{l}\mid x\right)}\right)\right)$ |
| KTO | $-\lambda_{w}\sigma\left(\beta_{\text{reg}}\log\frac{\pi_{\theta}(y_{w}|x)}{\pi<br>_{\text{ref}}(y_{w}|x)}-z_{\text{ref}}\right)-\lambda_{l}\sigma\left(z_{\text{<br>ref}}-\beta_{\text{reg}}\log\frac{\pi_{\theta}(y_{l}|x)}{\pi_{\text{ref}}(y_{l<br>}|x)}\right),\,$ |
|  | $\text{where}\,\,z_{\text{ref}}=\mathbb{E}_{(x,y)\sim\mathcal{D}}\left[\beta_{<br>\text{reg}}\text{KL}\left(\pi_{\theta}(y|x)||\pi_{\text{ref}}(y|x)\right)\right]$ |
| ODPO | $-\log\sigma\left(\beta_{\text{reg}}\log\frac{\pi_{\theta}(y_{w}|x)}{\pi_{\text<br>{ref}}(y_{w}|x)}-\beta_{\text{reg}}\log\frac{\pi_{\theta}(y_{l}|x)}{\pi_{\text<br>{ref}}(y_{l}|x)}-\Delta_{r}(x)\right)$ |
| Mallows-DPO | $-\log\sigma\left(\phi(x)\left[\beta_{\text{reg}}\log\frac{\pi_{\theta}(y_{w}|x<br>)}{\pi_{\text{ref}}(y_{w}|x)}-\beta_{\text{reg}}\log\frac{\pi_{\theta}(y_{l}|x<br>)}{\pi_{\text{ref}}(y_{l}|x)}\right]\right)$ |
| R-DPO | $-\log\sigma\left(\beta_{\text{reg}}\log\frac{\pi_{\theta}(y_{w}|x)}{\pi_{\text<br>{ref}}(y_{w}|x)}-\beta_{\text{reg}}\log\frac{\pi_{\theta}(y_{l}|x)}{\pi_{\text<br>{ref}}(y_{l}|x)}-\left(\alpha|y_{w}|-\alpha|y_{l}|\right)\right)$ |
| CPO | $-\log p_{\theta}(y_{w}|x)-\log\sigma\left(\beta_{\text{reg}}\log\pi_{\theta}(y<br>_{w}|x)-\beta_{\text{reg}}\log\pi_{\theta}(y_{l}|x)\right)$ |
| ORPO | $-\log p_{\theta}(y_{w}|x)-\lambda\log\sigma\left(\log\frac{p_{\theta}(y_{w}|x)<br>}{1-p_{\theta}(y_{w}|x)}-\log\frac{p_{\theta}(y_{l}|x)}{1-p_{\theta}(y_{l}|x)}<br>\right),\,$ |
|  | $\text{where}\,\,p_{\theta}(y|x)=\exp\left(\frac{1}{|y|}\log\pi_{\theta}(y|x)\right)$ |
| SimPO | $-\log\sigma\left(\frac{\beta_{\text{reg}}}{|y_{w}|}\log\pi_{\theta}(y_{w}|x)-<br>\frac{\beta_{\text{reg}}}{|y_{l}|}\log\pi_{\theta}(y_{l}|x)-\gamma\right)$ |

Table 8 lists the common notations used in this paper. The table serves as a quick reference guide for understanding the mathematical expressions and technical terms used throughout the paper.

**Table 8: Table of Terminology and Notation.**
| Name | Notation | Description |
| --- | --- | --- |
| Input Sequence | $x$ | Input sequence that is passed to the model. |
| Output Sequence | $y$ | Expected label or output of the model. |
| Dispreferred Response | $y_{l}$ | Negative samples for reward model training. |
| Preferred Response | $y_{w}$ | Positive samples for reward model training. |
| Optimal Policy Model | $\pi^{*}$ | Optimal policy model. |
| Policy Model | $\pi_{\theta}$ | Generative model that takes the input prompt and |
|  |  | returns a sequence of output or probability distribution. |
| Reference Policy Model | $\pi_{\text{ref}}$ | Generative model that is used as a reference to |
|  |  | ensure the policy model is not deviated significantly. |
| Preference Dataset | $\mathcal{D}_{\text{pref}}$ | Dataset with a set of preferred and dispreferred. |
|  |  | responses to train a reward model. |
| SFT Dataset | $\mathcal{D}_{\text{sft}}$ | Dataset with a set of input and label for supervised |
|  |  | fine-tuning. |
| Loss Function | $\mathcal{L}$ | Loss function. |
| Regularization Hyper-parameters | $\alpha,\beta$ | Regularization Hyper-parameters for preference tuning. |
| Reward | $r$ | Reward score. |
| Target Reward Margin | $\gamma$ | The margin separating the winning and losing responses. |

## Appendix D Experimental details

Here we report the best hyper-parameters we searched which corresponds to our final results. We include the modified dpo trainer and training scripts in the supplementary materials.

**Table 9: Hyper-parameters for results reported in Table 2.**
| Models | $\beta$ | $\alpha$ | $\gamma$ | $\tau$ | SFT $\lambda$ | lr | WR |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Base model | 0.01 | 1 | 0 | $\infty$ | 0 | 3e^-7 | 0.1 |
| + Length Norm. (LN) | 10 | 1 | 0 | $\infty$ | 0 | e^-6 | 0.1 |
| + Ref. Policy Mixing (Mix) | 0.01 | 0.25 | 0.1 | $\infty$ | 0 | 3e^-7 | 0.1 |
| + Contextual Scaling (CS) | 0.01 | 1 | 0 | $\infty$ | 0 | 3e^-7 | 0.1 |
| + Link Function (LF) | 0.001 | 1 | 0 | $\infty$ | 0 | 3e^-7 | 0.1 |
| + Home Advantage (HA) | 0.005 | 1 | 0.001 | $\infty$ | 0 | 3e^-7 | 0.1 |
| + Rejection Sampling (RSO) | 0.01 | 1 | 0 | 0.2 | 0 | 3e^-7 | 0.1 |
| Base model + LN | 10 | 1 | 0 | $\infty$ | 0 | e^-6 | 0.1 |
| + LN + Mix | 10 | 0.25 | 0.1 | $\infty$ | 0 | e^-6 | 0.1 |
| + LN + CS | 10 | 1 | 0 | $\infty$ | 0 | e^-6 | 0.1 |
| + LN + HA | 10 | 1 | 0.05 | $\infty$ | 0 | e^-6 | 0.1 |
| + LN + RS | 10 | 1 | 0 | 0.2 | 0 | e^-6 | 0.1 |
| + LN + SFT Loss | 10 | 1 | 0 | $\infty$ | 0.1 | e^-6 | 0.1 |

**Table 10: Hyper-parameters for results reported in Table 3.**
| Models | $\beta$ | $\alpha$ | $\gamma$ | lr | WR/WS |
| --- | --- | --- | --- | --- | --- |
| Base model | 0.01 | 1 | 0 | 3e^-7 | 0.1 |
| + Length Norm. (LN) | 10 | 1 | 0 | e^-6 | 0.1 |
| + Ref. Policy Mixing (Mix) | 10 | 0.25 | 0.1 | e^-6 | 0.1 |
| + Warm-up Adjustment | 10 | 0.25 | 0.1 | e^-6 | 150 |
| + Contextual Scaling (CS) | 10 | 0.25 | 0.1 | e^-6 | 150 |

**Table 11: Hyper-parameters for Table 6.**
| Models | $\beta$ | $\alpha$ | $\gamma$ | lr | WR/WS |
| --- | --- | --- | --- | --- | --- |
| DPO^∗ (Rafailov et al., ) | 0.01 | 1 | 0 | 3e^-7 | 150 |
| SimPO^∗ (Meng et al., ) | 10 | 0 | 0.1 | e^-6 | 150 |
| \gradientHSBRainbowPO0,240,200240,240,200^∗  (3 epochs) | 10 | 0.25 | 0.1 | e^-6 | 150 |

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: - Could you please elaborate on the evaluation? What was the base model against which the win rate was  evaluated in Section 4.
- Could you please elaborate on the use of link functions?
- Could you please elaborate on the mixing reference policy results in Section 4? What are the policies being mixed? How is the "good" policy \pi_\gamma obtained?
soundness: 3
strengths: - A careful analysis of different DPO methods, identifying their components and understanding the contribution of components of different DPO methods to performance is a worthwhile endeavor. The main technical contribution is a careful analysis and rewriting of the objective functions of recently proposed DPO methods and unifying them as a single objective function. 
- Another contribution is the proposed the mixed reference policy obtained by mixing a fine tuned policy (model) with a policy that maintains a certain margin or odds ratio of the likelihood of the preferred to dispreferred responses.
- I believe the main conceptual contribution is the development of common frameworks and pipelines that allow researchers and practitioners to evaluate different components and techniques for DPO. As such, the paper is well motivated and the findings of this paper and follow on works that attempt to perform such evaluations will be of significance as alignment of generative language models is an important problem and existing reinforcement learning based approaches are computationally expensive in comparison to DPO.
- The main finding that length normalization, policy mixing and contextual scaling contributed towards increasing performance are a potentially interesting takeaway that may be significant to practitioners.
summary: This paper aims to provide a unified framework for direct preference optimization (DPO) methods which are becoming a popular and successful method to train or finetune generative language models, and in particular, for better alignment with human preferences. In DPO, we are given a dataset of triples consisting of a prompt and a pair of potential responses, one of which is known to be preferred over the other. The high level goal is to fine tune a large language model in a manner that maximizes the difference in the likelihood of the preferred and dispreferred responses. The performance of a model is measured by the win rate of responses of the model being preferred over the responses of another model over a collection of inputs as predicted by an automatic evaluator which is typically another generative language model trained to predict which of two responses a human annotator would prefer.

The paper surveys, then carefully analyzes the objective functions of different algorithms in a growing family of DPO like models, identifies their components and then formulates an objective function for their proposed RainbowPO method which incorporates the components and techniques of previous DPO methods. The contribution of the different components of the objective function can be tuned smoothly using hyperparameters. The paper then goes to evaluate the effectiveness and contributions of each component to the performance of DPO methods.

After rebuttal:
I am increasing my score from 5 to 6. The authors have addressed my questions. Overall, I lean towards acceptance if there is room to accommodate it.
weaknesses: - The paper is overall poorly written with several missing explanations and gaps in the logic. While it is possible to guess what is meant, a reader unfamiliar with the area will likely find it very hard to follow.
- At a high level, the organization of the paper makes it hard to read as the text on the background and proposed RainbowPO model frequently references results in the later experimental section which distracts from understanding the components of the DPO objective functions.
- The technical contributions and novelty are limited beyond the creation of the unified objective function and evaluation pipeline.

Some examples of issues with the writing:
- line 153: ... there lacks a work .. please consider revising. This is just one minor example. There are several more throughout the paper and they are too numerous to list. Please consider performing a thorough reading of the paper.
- please consider elaborating on the verbosity issue and how it may be potentially a way to game the evaluator
- different places use y_l and y^l. Is this a typo? Similar issues exist elsewhere. Preferably please keep the usage of subscript and superscript consistent. For example, see in (10) the use of \pi_\alpha and \pi_{ref}^\alpha. Consider e.g. using \pi_\alpha, \pi_\alpha^{ref} and \pi*{ref} as appropriate instead
- abbreviation LC (length controlled) is not defined or explained anywhere making it hard to understand results in Section 4.1
- in general, it would be nice to have a more clear discussion of the evaluation framework and evaluation criteria. As it stands, the paper assumes too much background knowledge e.g. about AlpacaEval.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. For eq (1), how to obtain the policy $\pi_\gamma$ in the practical setting?
2. What is the purpose of raising eq (12)? If left side is a lower bound then whether minimizing the lower bound can imply anything?
soundness: 3
strengths: 1. This work is the first to provide a comprehensive analysis of the effectiveness of various DPO variants under the same setting, making it a valuable contribution to the community.

2. The study offers an in-depth examination of the components of DPO, supported by thorough experiments that validate the analysis, resulting in a solid and reliable evaluation.

3. By combining all effective components, this work successfully formulates a new variant of DPO and demonstrates its superior performance over existing algorithms, showcasing the potential for further advancements in the field.
summary: The paper extracts key components from various DPO variants and provides theoretical analysis for each component. It then establishes a unified formulation that encompasses most DPO variants. Empirically, the author conducts a series of ablation studies to analyze the effectiveness of each component in DPO variants and identifies the key components that contribute to metric gains. By combining these components, the author proposes a new variant of DPO called RainbowPO, which outperforms all other DPO variants.
weaknesses: 1. The evaluation benchmark is limited to AlpacaEval, which may not provide sufficient evidence to support the claims regarding the effectiveness of each component and the advantages of the proposed RainbowPO. A more comprehensive evaluation across multiple benchmarks would be necessary to further validate these findings.

2. The advantage of introducing key components in RainbowPO lacks systematic explanation and remains at a purely empirical level. It is unclear whether the same advantages can be transferred to other tasks, highlighting the need for further investigation into the generalizability of these findings.

3. The proposed method introduces significant additional complexity compared to baseline methods, which may impact its practicality and scalability. Further research is needed to determine whether the benefits of RainbowPO outweigh the increased complexity it introduces.

---

confidence: 2
contribution: 3
initial_rating: 5
presentation: 3
questions: Please answer the questions mentioned in ‘Weaknesses’.
soundness: 3
strengths: 1. The paper strived to provide a systematic and comprehensive study of existing preference optimization methods. The identification of the seven components leads to a structured framework for evaluating xPO methods, which is useful considering the large, and still growing number of xPO methods. 

2. The paper contributed clarity and insights on how different components of an xPO method affect the method’s performance. The experiment evaluation offers interesting results illustrating the value of adding components to an xPO model.
summary: The paper developed Rainbow PO as a unified framework for direct preference optimization. Motivated by the challenge in comparing various xPO methods in literature, the authors began the study by analyzing more than 10 existing xPO methods and identified seven mathematically orthogonal components for mapping these methods. Among these components, they justified four of them as effective. Based on these findings, they proposed RainbowPO to combine these components. One notable novelty is that RainbowPO adopts a mixing reference policy mechanism. The authors conducted experiments to illustrate RainbowPO's superior performance over previous DPO variants.
weaknesses: There lack transparency and explanation on the identification of the seven components. It would be helpful to include further details on the identification process. Relevant questions include: Are these components necessary and sufficient to categorize current and potential new xPO models? Have similar components been applied or mentioned in literature on xPO? Are there any potential interactions among components that would affect the performance of xPO?

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - DPO techniques have been reported to lack generalizability in out-of-distribution tasks [1]. To address this limitation, learning chain-of-thought (CoT) policies has been proposed in the literature. Can the method proposed in this paper be adapted to learn a CoT policy?

- Robust DPO [2] is another approach to make DPO robust to noisy preferences. Is there a reason why this variant is not considered in your work? 

- Have you considered approaches [3] which removes the process of learning a reward model by directly learning though supervised learning?

- Apart from Win ratio (WR), have you measured BERTScore?

1: Choi, Eugene, et al. "Robust Chain of Thoughts Preference Optimization." Seventeenth European Workshop on Reinforcement Learning.

2: Chowdhury, Sayak Ray, Anush Kini, and Nagarajan Natarajan. "Provably Robust DPO: Aligning Language Models with Noisy Feedback."
 Forty-first International Conference on Machine Learning.

3: Hejna, Joey, et al. "Contrastive Preference Learning: Learning from Human Feedback without Reinforcement Learning." The Twelfth International Conference on Learning Representations.
soundness: 3
strengths: **Originality:** This paper performs a meta-analysis on different variants of DPO.

**Quality**: The experimental evaluation comparing different components is the main strength of the paper.

**Clarity**: The text is written with clarity and well organized. It is easy to follow, even for someone like me, who is not up-to-date with the recent developments in LLM finetuning techniques. However, some of the terminologies introduced require explanations (see weakness).

**Significance**: Fine-tuning large language models (LLMs) for preference optimization has garnered increasing attention recently. A meta-study comparing different variants of preference optimization in LLMs would be highly valuable for both academics and practitioners. This work combines seven variants and promises to make the implementation publicly available, which would be a significant contribution to the community.
summary: Direct Preference Optimization (DPO) has emerged as an effective approach for fine-tuning large language models (LLMs) to align with human preferences. Various versions of DPO have been proposed in the literature; however, an extensive comparison between these different DPO variants is still lacking.
This paper aims to analyze various versions of DPO to identify components that enhance performance across different applications. To this end,  seven key components have been identified  across ten DPO variants. By combining these components, a new approach that achieves state-of-the-art performance has been proposed. This approach is conceptually similar to the Rainbow framework [1], which unified various enhancements of Deep Q-Networks (DQNs) from the literature.

This paper explains seven components, namely--Length Normalization, Link Function, Home Advantage, Mixing Reference Policy, Rejection Sampling, Contextual Scaling and SFT Loss. Then a new loss is formulated combining the seven components. Finally, the experimental evaluation reveals that Length Normalization, Contextual Scaling  and Rejection Sampling contribute to improvement over the baseline DPO.

The paper could be a significant contribution in the field of Fine-tuning large language models. 
However, to meet the publication standards of a top-tier conference, it requires an additional round of thorough editing.

[1] Hessel, Matteo, et al. "Rainbow: Combining improvements in deep reinforcement learning." Proceedings of the AAAI conference on artificial intelligence. Vol. 32. No. 1. 2018.
weaknesses: - Some acronyms, such as IPO and KTO, are never explained. Given that this paper is a meta-analysis, it’s important to define all acronyms for the reader’s convenience.

- Define LC WR in the caption or in a footnote in Table 4.

- It seems the term 'warm-up adjustment'  is never introduced before Table 3.

- The color coding in Table 4 is counterintuitive; using red to indicate better-performing models and green for worse-performing models is disorienting for the readers.

- The results in Table 4 are based on training for 3 epochs. Could these outcomes differ if the model were trained for additional epochs? Maybe you can  conduct an ablation study on the impact of number of epochs or plot training loss against number of epochs.

- Is there any explanation why rejection sampling does not perform well with length normalization?

I would be happy to raise my score, if the authors reply to my questions.

Meta-review: This work reviews several improvements over DPO and identifies "orthogonal" complementary components whose contribution to task alignment has not been thoroighly examined.  Their proposed training procedure, RainbowDPO, combines aspects of multiple approaches, which can be parameterized by multiple hyperparameters.  The authors investigate the effects of each of these mechanisms, finding that length normalization, reference policy mixing and contextual scaling then to produce the largest improvements in terms of automated eval (AlpacaEval2) metrics and length.

Reviewers found the paper well-written and commended the authors' effort at summarizing and evaluating recent DPO methods within the same framework. Reviewers found the insights and unified parameterization of DPO to be valuable.  After the rebuttal stage, all reviewers gave the paper a "marginally above acceptance threshold" rating and advocated for acceptance.

There were a few weaknesses pointed out by the reviewers which I hope can be addressed in a future revision.  In particular, Nycb calls out that the evaluation benchmark is limited to AlpacaEval2 scores for a fixed set of tasks, so it is not clear how the findings generalize across different tasks.  Evaluation across multiple tasks can help with understanding generalization here.

The authors' method for identifying important hyperparameters (and manual hyperparameter optimization) is ad hoc and could easily be evaluated more rigorously via sensitivity analysis (e.g., via sobol indicies or derivative-based methods), and optimal HPs can be found efficiently via Bayesian optimization (or multi-objective BO if targeting eval metrics and length).  Such optimization is easy to do with off-the-shelf tools.  It also appears that the authors only performed HPO for RainbowPO and not other methods; performing HPO for each individual method (of course, using a valid train/test/validation split) is crucial for making comparisons between methods.  The authors appear to only consider a limited number of training epochs (1 and 3), which to the AC feels fairly arbitrary.  Since there is heterogeneity between how the different methods/HPs, it might be beneficial to consider this as a tunable HP.

Decision: Accept (Poster)