RainbowPO: A Unified Framework for Combining Improvements in Preference Optimization
Authors: Hanyang Zhao
Genta Indra Winata
Anirban Das
Shi-Xiong Zhang
David Yao
Wenpin Tang
Sambit Sahu
Venue: ICLR 2025 Poster (2025)
Keywords: Alignment
Preference Optimization
Reinforcement Learning from Human Feedback
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We propose RainbowPO, a unified framework that demystifies the effectiveness of existing DPO methods by grouping them into seven broad categories and integrating them into a single cohesive objective.

Abstract: Recently, numerous preference optimization algorithms have been introduced as extensions to the Direct Preference Optimization (DPO) family. While these methods have successfully aligned models with human preferences, there is a lack of understanding regarding the contributions of their additional components. Moreover, fair and consistent comparisons are scarce, making it difficult to discern which components genuinely enhance downstream performance. In this work, we propose RainbowPO, a unified framework that demystifies the effectiveness of existing DPO methods by categorizing their key components into seven broad directions. We integrate these components into a single cohesive objective, enhancing the performance of each individual element. Through extensive experiments, we demonstrate that RainbowPO outperforms existing DPO variants. Additionally, we provide insights to guide researchers in developing new DPO methods and assist practitioners in their implementations.

Published as a conference paper at ICLR 2025
RAINBOW PO: A UNIFIED FRAMEWORK FOR COMBIN- ING IMPROVEMENTS IN PREFERENCE OPTIMIZATION
Hanyang Zhao'r, Genta Indra Winata 2, Anirban Das Shi-Xiong Zhang", David D. Yao', Wenpin Tang', Sambit Sahu²
CColumbia University Capital One
ABSTRACT
Recently, numerous preference optimization algorithms have been introduced extensions to the Direct Preference Optimization famiily While these meth- ods have successfully aligned models with human preferences, there is lack of understanding regarding the contributions of their additional components. More- over, fair and consistent comparisons are scarce, making it difficult to discern which components genuinely enhance downstream performance. In this work, we propose RAINBOWPO, a unified framework that demystifies the effectiveness of existing DPO methods by categorizing their key components into seven broad directions. We integrate these components into a single cohesive objective, en- hancing the performance of each individual element. Through extensive exper- iments, we demonstrate that RAINBOWPO outperforms existing DPO variants. Additionally, we provide insights guide researchers in developing methods and assist practitioneers in their implementations. Our code is available at attps: /tt:://gtttht com wwtpe////aaaaaaneeeeeeee Receeeeeeeeehhee wtteeeearre/aaaanree.....
DPO
INTRODUCTION
Reinforcement Learning with Human Feedback (RLHF) (Ouyang et al. 2022, Stiennon et al. 2020
Ziegler al., 2019) has significantly contributed the success of recently released Large Language Models (LLMs) such as InstructGPT (Ouyang et al., 2022 ChatGPT, and GPT4 (Achiam et al.) 2023 However, RLHF a complex and resource intensive process and requires training a reward model. An alternative to RLHF is Direct Preference Optimization (DPO) (Rafailov et al. 2023) that directly optimizes policies from pairwise preferences by minimizing a supervised learning loss objective, which is viewed the maximum likelihood estimate for the reward model in RLHF. This approach allows DPO and other DPO variants to bypass the use of RL, resulting in faster speed of end-to-end training and better resource efficiency, while achieving comparable superior performance to RLHF in downstream tasks such as summarization (Rafailov et al.) 2023).
DPO and its success during training foundation models like Llama series (Dubey al., 2024 Tou-
vron et al., [202)] Mistral (Diang et al.) 2023a), has garnered significant research attention in the LLM alignment space (Winata et al. [224]] Wang al. 2024b), leading the development of var- ious extensions. These include variants beyond pairwise ranking, such as Kahneman & Tversky Optimization (KTO, Ethayarajh al. (2023) and MallowsPO (Chen al. (024a), unified perspec- tives on loss parameterization, such as ldntitty Preference Optimization (IPO, Azar et al. (2024)) and Generalized Preference Optimization (GPO, Tang et al. (2024), distribution correction methods like Rejection Sampling Optimization (RSO, Liu et al. (2023)), and reference model-free alternatives, such as Contrastive Preference Optimization (CPO, Xu et al. (2024)) Odds Ratio Preference Op- timization (ORPO, Hong et al. (2024)), and Simple Preference Optimization (SimPO, Meng et al. (2024). Each of these DPO variants claims to outperform the original DPO in downstream task evaluations by introducing specific additional components, mathematically modifying the loss objective. In the rest of the paper, we will refer DPO variants collectively as xPOs for simplicity.
Comparing these xPOS proposed in the literature is not always straightforward due to differences in the base model size and architecture, the alignment datasets, the experimental setup as well as the
Equul Contribution. Contacts: h22684 ambii edd, genta wi nata@capi ta lone. com, ani rban. wass@capptallonee.ccomm

---

Published as a conference paper at ICLR 2025
evaluation metrics. Subsequently, it becomes difficult assess the effectiveness and choose among different XPO methods given a problem. A brute force comparison across all existing methods is prohibitively expensive and inefficient. Therefore, it is crucial that we study the performance characteristics of each proposed method in the literature by evaluating the xPOs' performances under at least one convincing and representative setup. Further, despite the success of the XPO family, a fundamental question remains unexplored:
What are the components proposed in XPOS that actually improve the performance over DPO?
Surprisingly, there is still a lack of comprehensive work studying the progress in the literature and summarizing the core practical components of these methods that lead to improvement over DPO. To demystify the reasons for their effectiveness, we hypothesize that the main benefits of these methods stem from the combination of several mathematically orthogonal effective components. In this paper, we validate our hypothesis by decomposing the xPOs and identifying these orthogonal components upon DPO. We further assess their effectiveness through downstream task evaluations, ruling out the components that do not contribute to performance improvements. Given these orthog- onal identified beneficial components for preference optimization, a natural question arises:
Can these individual components complement each other and be effectively combined?
Our question is largely motivated by the previous study RAINBOW (Hessel et al. 2018 that explored improvements over Deep Q-Networks algorithm (DQN) (Mnih et 2l.1|2015) traditional Reinforce- ment Learning (RL). The summarization and comparsion in Hessel et al. (2018) greatly enhances the understanding for improving DQN, and the resulting algorithm Rainbow, still serves as a benchmark (Raffin et al. 2021). However, such study for RLHF is still underexplored. This shows gap in the literature, that elicits an answer to the question of combining different XPO extensions evaluated in comprehensive setting. To bridge this gap, propose RAINBOWPO, unified framework that integrates existing xPOs' components, and deploys useful and essential components in a principled manner to achieve better performance. To conclude, contributions in this paper are as follows:
(1) We conduct comprchensive study on more than 10 offline representative variants of DPOs (XPOs) from a practical aspect by analyzing their loss functions for optimization. We conclude several mathematically orthogonal directions along which these methods propose to optimize over the original DPO loss, analyze the usefulness of each method theoretically and empirically, and provide comparisons under the same representative setup.
(2) We identify and summarize 7 broad components across all DPO extensions: length normal- ization, link function, margin / home advantage, reference policy, contextual scaling, rejection sampling optimization (RSO), and supervised fine-tuning (SFT) loss, and justify that four of them are effective through extensive hyper-pane paper-prraettess search, model training and evaluations. Additionally, we also propose better way of formulating the reference policy by mixing the SFT policy with margin (see details in the reference policy of Section 3. and demonstrate the advantage of this approach over using just SET policy (in DPO) or just margin (in SimPO).
(3) Finally, we propose RAINBOWPO, a DPO variant that combines three essential and orthog- onal components from existing XPOS. Combining other adjustment on optimization hyper- parameters, we show that our algorithms perform the best among all open-sourced algorithms when tuning Llama3-8B-Insteen
In the widely adopted LLM benchmark Alpaca-Eval2 D R RAINBOWPO improves Llama3-8B-IIntruus from 22.92% to 51.66% for Length Controlled Win Rate (L.C WR), with just access to reward model form the offline preference dataset and no further online sampling. We also perform an ablation study and show that all adopted elements in RAINBOWPO are indeed necessary to achieve the best result.
Related Work. Below provide (non-exhaustive) list of other relevant references this work.
Compared to human feedback in original RLHF, existing works have improved the scalability by utilizing Al feedback (Bai et al.) 2022 Lee et al.) 2023). For such need of constructing better Al feedback, recent works also proposed various reward models for formulating better preference
datasets, like PairRM (Jiang al. 2023b). ArmoRM (Wang al. 2024a), RRM (Liu et al. 2024a)
and RM benchmarks like Reward Bench (Lambert et al. 2024).
https: https::/ggithubb.com/ttttt http://ttttttltaaapppaaaaaaceeee 1paca_eva
2

---

Published as a conference paper at ICLR 2025
We also find works that target understanding DPO methods related to our work. Liu al. (2024b) studies the effect of reference policy in the preference optimization; Sacidi et al. (024)) compare the performance of DPO, IPO, CPO, KTO for tuning Mistral 7B OJang al. [203a based models, and mainly studied the roles SFT stage for alignment methods.
The rest of the paper is organized follows. We provide backgrounds RLHF and DPO in Section 2 In Section 3] we summarize the current directions in existing xPOs and the development of RAINBOWPO, followed by detailed experimental results in Section Finally, we present our conclusion in Section
2 PRELIMINARIES AND MOTIVATION
In this section, we first briefly introduce RLHF and DPO as the foundation method, and then discuss on extensions of DPO (XPOS) to understand what are the components proposed in the literature.
RLHF starts with fine-tuning a pre-trained large language model by supervised learning on high- quality data for some downstream tasks of interest (e.g., dialogue, summarization, etc.), to acquire a model x- SFT This step is referred to as the SFT phase. For instance, for training Instruct- GPT (Ouyang al. 2022). GPT-3 (Brown al. 2020) first fine-tuned on the given input prompt distribution. The second stage of RLHF is known as reward modeling, i.e., researchers collect pref- erences D (x. Nw: 31) on the generations of fine-tuned model π TSPT, and learns a reward model y) that could represent the quality or the rating of generation y with respect to prompt z. The final step is policy optimization on TTFT Tref by maximizing regularized reward obtain the optimal policy model x² through reinforcement learning:
max EE [Ey-=n(vlz)) [r" (z, y)] - 3KL (TO(- ()]|Frr(( [x))]]
(1)
0
in which 3 > O denotes the regularization constant. For ease of reference, we prvide more detailed description of RLHF in Appendix A.T and we also add table of notations in Table Appendix B
2.1 DIRECT PREFERENCE OPTIMIZATION (DPO)
One disadvantage of RLHF is that the RL step often requires substantial computational effort (e.g., to earry out PPO). The idea of DPO is to combine the reward model and RL in RLHF into a single ob- jective, bypassing the computation in the RL step. Given the same preference pairs 'D = (x,, Yar v1) utilized for reward modeling in RLHF, the DPO objective yields:
min LDPO (mD; Tret) == - -E (x.. log (glog To (yar |x)
ae (y xx)
)
33 log
(2)
Tel (yro ar)
Tref (yn x:)
where (-)) is the sigmoid function and 3 is regularization parameter similar to the one RLHF. DPO thus yields supervised learning problem, and requires much less computation than the RL based RLHF The objective in Equation2jaan be understood as maximizing the likelihood difference between the preference pairs under the policy, encouraging the model to more likely generate the preferred answers than non-preferred. We refer more details of DPO derivation in Appendix A.2
2.2 MOTIVATION: REVISITING xPOS
Since DPO is proposed, there is huge interest in developing and improving DPO, leading nu- merous xPOs. Different xPOS can be motivated by theoretical concerns like relaxing or extending preference distribution assumptions in IPO and MallowsPO. human aware loss function in KTO, or from practical aspects like reference model-free alternatives, like CPO, ORPO and SimPO. We provide an non-exhaustive list in Table g in Appendix for the ease of revisiting and comparison.
Despite their differing motivations, xPOS share primary objective to optimize. We thus take the loss objectives as the first class citizen, and mathematically understand the parts that are commonly adopted or differ in XPOs. Before going into detailed categorization, want to first argue that, in existing preference optimization literature, there is lack of comprehensive studies on revisiting and examining DPO variants in their mathmatical objectives. As consequence, some papers may have implicitly proposed some designs for improvement and even didn't highlight it. As a motivating example, we revisit ORPO objective, which proposes to maximize odd ratio difference (for an
3

---

Published as a conference paper at ICLR 2025
event A with probability p. the odds ratio is defined as p/(1 p)) between the winning and losing answers:
Po (Bw (2)
po (0(
LORPO (Te)
TE -Eloog Po (Wwr + A log log
log
1 Po (x)
1 Pg (
(To)
X-Cro(TT)
in which the expectation for (z, yuu s1) D. and pe (ylar) exp
1 log no (wlx)) Rewriting
terms in Cpo (TT),, could derive upper bound (see proof in Appendix D.I
1
1
LP (TT) I - log o - pe (B(l1)))
log To (Bul
log To (8(
= Lro (TO) (4)
180
if assuming Ag for all æ. The upper bound Lro sharp, as LPo Lro O thus minimizing ORPO loss could be understood as (namely B/:l and 1/ 1/1yr]. see more detailed explanation in Section B and contextual dependent ((x) = 1/(1 (uil x))) Length normalization is one of the key ideas adopted in SimPO:
free DPO with length normalization
log To To - ) (5)
LSmpO () = - -x
log
[Vnol log To
which evident in SimPO objective, though proposed after ORPO. This connection is however unaware by the literature. This thus calls for a comprehensive analysis of the contributed elements in different XPOs far, as many methods may overlap in contributed directions without awareness, and bringing this out right away could possibly prevent repetitive work or efforts in the future.
Following similar analysis of different representative XPO methods for pairwise preferences, in- 2023). IPO (Azar et al. (2024). CPO (Xu et al., 2024). GPO (Tang
cluding DPO Ratailov et al.
2023), ODPÓ (Amini et al. 2024), ORPO (Hong et al. 2024). Mal- lowsPO (Chen et al. 2024a). SimPO (Meng al. (024). we come up with seven broad categories,
al., 2024). RSO (Liu al.
which is able to explain most popular DPO variants in the literature, as in Table [ This provides
straightforward illustration of the main ideas and connections of existing methods. The meanings and details the categories are elaborated in Section
Method
Length Norm.
Link Func.
Home Adv.
Ref. Policy
Contextual Sealing
RS
SFT Loss
DPO
logistic
SFT SFT
SLiC-HF
hinge
x
IPO
X
square
SFT
X
X
X
CPO
X
logistic
X
Free
X
X
RSO
X
logistic hinge
SFT
X
4
X
ODPO
logistic
SFT
ORPO
logistic
Free
implicitly
WPO
logistic
SFT
X
MallowsPO
X
logistic
SFT
X
SimPO
logistic
Free
PO
logistic
mixing
X
Table 1: Mapping of XPOS with mathematically orthogonal components (see more details in Ap- and validation results of their effectiveness by the downstream task evaluations.
pendix
RAINBOWPO: A UNIFIED FRAMEWORK
3.1 COMPONENT DESCRIPTIONS
We first explain in detail about the components categorized, after which propose generic framework, which we name as RainbowPO, to combine these components.
Length Normalization.
The literature has noticed a verbosity issue of DPO aligned models, as the aligned model may generate answers significantly longer than both preferred and rejected answers (Park et al. 2024). This also could lead to an inflated win wate when evaluating model performance, L.MM-as-a-judge can be susceptible length bias: Wang et al. (023) has noticed that when

---

Published as a conference paper at ICLR 2025
evaluating 13B parameter models in head-to-head comparisons with the Davinci-003 model, win rates have a strong correlation (0.96) with the average number of unique tokens the model's response. To address this issue, promising direction noticed in the literature is to incorporate explicit length penalties, like in R-DPO (Park et al. 2024) and SimPO (Meng et al. 2024):
L. v) ro( y) alyl. and (2, ro
(z. v) ro( v),
in which (2( v) log [ is the implicit reward model (Rafailov et al. 2023
From an
[))))
optimization perspective, maximization with respect to rh (x: y) is equivalent to IR specific or (might be prompt dependent). Directly applied to DPO, the resulting objective yields:
(x, y) with a
B
TD (1ho
x))
B
To (3/1 ax)
LLN-DPO (TT; Teet) ==
E >
log a
log
log
(7)
(2,9ta M/
[Mwl
Tref (3hu
x))
Tref (1/1 x))
Why length normalization could help prevent the verbosity issues can be explained through exam- ining the gradient of the loss respectively: VoLIN- (Toi Teer)
1 |8
1 V log To (v x)) 1
) (8)
-I (ro (z, vi) L.N (xr, vw))
Ve log No (ywe x)
thus the gradient of length normalized DPO can be understood as taking a discount factor M of the length for longer sequence. We also empirically justify the effectiveness of length normalization by comparing to the vanilla DPO trained models, and witness the consistent smaller average length, independent of the regularization constant 3. See results of average length in Section
Link Function. SLiC-HF (Zhao et al. 2023 and GPO (Tang al.
2024) both realized that the DPO objective could be understood as taking f (we refer this as link function) as - log (-) in:
To - 3 Blog [er((Vllz)
To(u (z)
LOPo
E
log
Teed (M1la))
thus unifying DPO, IPO, SLiC (without SFT loss) as instances of taking f as logistic - log (-), square (x: 1/2)² or hinge function max(0, - x)) for some margin 5 > 0 (see more details in Ap- pendix C. separately. For identifying the best link function, we did an exclusive parameter search for DPO and IPO separately, however
found DPO, i.e. adopting - log a(-) as the link func- tion, empirically performs better than IPO evaluated by AlpacaEval2, despite the weaker theoretical assumption of preferences in IPO. Thus stick to the link function be -log (-) in this paper.
Home Advantage/ /Margin. In SliC, IPO, ORPO, SimPO, there exists term which also targets at encouraging the difference between the reward model difference. is also referred in SimPO as the term of home advantage 7 (the terminology comes from extension of the vanilla Bradley-Terry beats j)) = T; Tÿ 7 Thus the likelihood could be written as:
Model):
p" (y1 1/2 .r) (r* (Ir,1)) r" (a, 7)
(10)
which takes the losing prompt in a home advantage when 7 > 0. SimPO shows the effectiveness of this margin under the reference- free setup; however, when we adopt the margin for vanilla DPO (i.e. with the reference policy) with the optimal 3, we do not witness increase of the performance when adjusting the margin, either further adopting DPO with length normalization or not. In Figure la the performance steadily decreases when increasing the margin y in DPO, i.e. we optimize:
To (yo xx) Teef (yw x))
.. To (y x))
LDPO+ (TT: Tref, 7) = - (n DoD
log
[log
neet (y) xx)
(11)
This questions the true explanation about the effectiveness of the margin term in SimPO. We provide the answer as understanding margin as a reference policy right in the next paragraph.
Reference Policy. DPO takes the SFT policy the reference policy motivated by the standard
RLHF pipeline. However, recently proposed methods like CPO, ORPO, and SimPO (Xu et al.
2024
Hong al. 2024 Meng al. 2024 all suggested reference-free objective could yield
5

---

Published as a conference paper at ICLR 2025
the same even better performance. CPO and ORPO further utilized extra SFT loss force regularization, while for SimPO, such regularization is not enforced. Given our prior examination that home advantage Eq. should be understood as a term for "reference policy" instead of the "home advantage".
hardly improve over DPO, we argue that the margin term in SimPO loss
Concretely, we could hypothesize that there exists "good policy" Th such that, for each prompt and preference pairs in the dataset, the normalized log likelihood ratio of preferred response to non- preferred response is a positive constant, which we denote as Tyy We assume that TTy 's normalized implicit reward model is perfect at in-distribution pairwise classification and yields (x)(=/==) (=(()1))">
exp(7) for any If so, the loss of SimPO (defined in Equation 5 could be rewritten as:
(
T56 (Yw
x:) B3 To (y1 x:)
SsimP (Toin) (a.
E
log
log Try (yw
log
(12)
xx)
18
Thy (31 x))
This transformation motivates (reference) policy. If taking Tsht as the reference policy is too conservative (not strong enough), and taking T.y policy can help improve the performance but totally neglects the original SFT model implicit preference information, can we benefit from a mixing of these two policies? The answer is YES. Consider exponential mixing of likelihood of reference policy Th and Tyet (we use Ta,y to denote the relevance of the resulting reference policy to both a and 7) with a E [0, 1], defined as:
further propose new mechanism which we call as mixing
(a, 2) ( |z) T..
x x).
(13)
Then if we use Ta,7 as the reference policy in (7). we yield LIN Ta,7) with a practical form:
To (yhe x)) 11
Tief (yho x)) /lowl
E log
B log
To (y x)) 1/bnl o. log
nyet (yi x)) //mml (1 - a)>) (mm)
(14)
Notice that LIN-DPO (T;; Ta,1) = LIN-DRO (To (To: ) = Lsimro 7T) ) thus SimPO is an instance of mixing policy by taking a 0: LLN- (To; x1.)) LIN-DPO (T)) thus a corresponds to DPO applied with length normalization as in Equation 7 For finding good To,y, we first take a = 0 and tune the best y, which is similar to SimPO; we then afterwards tune a using obtained 7.
According to our experiment results, we indeed find that there exists a E (0. 1) that performs better than both sides (i.e. a = 0 or = 1), see Figure [b] Recent work (Liu al.) (002b) analyze the role of reference model, and argue that stronger reference model could benefit DPO; our finding is consistent, as we further explicitly design a choice of better reference model for better performance.
Rejection Sampling. Since the proposal of DPO, there is controversy on the exact equivalence of DPO and RLHF. RSO (Liu et al. 2023), further pointed out that the data should be generated from the optimal policy if treating DPO objective as maximum likelihood estimation. Thus RSO adopts a statistical rejection sampling for sampling preference dataset D generated by the optimal policy to mitigate this distribution difference in DPO.
To address the intrinstic different variance schedules of reward model for different prompts, and stablize the process for formu- lating the preference dataset, we also adopt modified version of RSO by computing the per- centile reward (or the ranking reward) in the whole generation set instead of utilizing the true reward, which we found that can stabilize the generation and yield better results when fur- ther applied with DPO, as in Algorithm[ [
Algorithm RS for preferences formulation. For each prompt æ, start with an empty set y + 0- Generate N > M answers ~ Tary x). for i i N as candidates.
Compute each yi's percentile Pi(x)) based on r(z, 31) over the whole N answers for prompt x. Initialize counting number j = 0.
while |>]| < M do
= + 1 and generate u U 1] if S exp((P; 1)/r) then Accept yi and add it to V.
Similar to in RSO (Liu et al. 2023), we search the best temperature hyper-parameter for RSO through the downstream task performance as the validation metric, which we detail in Figure Ic We then use the empirically best performed temperature constant 7 to formulate the prefer- ence dataset as DRS-
else
Reject y.. end if end while
Let yo may r(z, y)
Let 3 arg minyey ((x,

---

Published as a conference paper at ICLR 2025
Contextual Scaling. Existing work also considered the contextual difference: preference pairs might be of higher uncertainty or have more dispersion. In this paper, we adopt the idea of MallowsPO in Chen et al. (2024a) by introducing contextual scaling factor d(x) on the likelihood difference, which yields:
E log a (6(x) Blog To(Uw)))) ,9 log =o(sn(z)) 1)+ (15)
( (Tel)
Tre((Hwlz))
[
if adding this factor to DPO objective. The scaling factor is motivated by the Mallows ranking model which has a natural carrier of dispersion index. In MallowsPO, o(z) corresponds to a normalized predictive entropy of the preference pair (xe, yeu, D1) (yo,, and yı, denote the 7th component of yo and y
MN
11=1 [H (Yi+1 Y = yw,i) + H... (Y4+1 Y; = st:i)l
(xx) = log
(16)
2 log
SFT Loss. SFT loss is straightforward by adding extra SFT loss term the winning answer, or a reference answer (for regularization, which appears for reference-fre methods like CPO and ORPO). For example, adding SFT for DPO with preferred/winning answers, we yield:
LSFT (mo; A) = -E log a (3 (Blog To (yur
xx) x))
To (y) x:) Teet (y1 x)) + A log((To (Nw x)))
3 log
Tee (ywr
(17)
However, we find that adding SFT loss could largely degrade the performance, as in Section
3.2 UNIFIED FORMULATION: RAINBOW
Combining the advances proposed above, we propose following preference optimization objective, for which we refer our method as RAINBOWPO:
To (yw
(y1 x))
PRAINBOWP (TT;; Trat) (ar-gha BB
IE
((x))
log
log
|
Ta.y
(Vor
ar)
Wa.7
(31
2))
(18)
which 77 E {0, 1}, and Ta,y as defined in Equation 13 The preference dataset D* can be DRS. which means that the preference dataset is formulated by by rejection sampling from the original dataset's 's prompts as mentioned in Algorithm [ if we have access to a reward model's ' true value. If the reward model is black-box oracle, namely we cannot access the true reward value, will always utilize the usual formulation way of preference dataset D, detailed in the experiment section.
Hyper-parameeers. Like all other xPOs, achieve the best performance, RainbowPO can intro- duce an extensive amount of hyper-parameter search for the best performing f. a., B, T and whether r For efficient hyper-parameter search, we conducted a greedy search method with the help of our framework and decomposition of effective elements: we search for the best for those that affects the performance in the most when we gradually add designs to the preference optimization methods. For example, when adding length normalization to the methods, we only search for the best hyper-parameter for the regularization parameter 3, and will fix the learning rate and all the training args, which prevents the parameter searching space from exploding.
4 EXPERIMENTS
To evaluate the performance of the xPOS algorithms, we conducted extensive experiments on train- ing models with various xPOS configurations and compared their instruction-followinn capabilities.
Experimental Setup. We choose Jlama3-8B-IInsrrue as our model base to fine tune, mainly because that aligning this widely adopted and flagship instruct model is of great interest to the whole community and meets the standard as a representative setup for alignment. It can also help mitigate the uncertainty from probably not perfectly supervised fine-tuned models.
For evaluation metric, we use widely adopted benchmark AlpacaEval2, which is composed of 805 questions and evaluate the instruction following capability of the model. AlpacaEval2 evaluates
httts::/hugginggfaeee www.ce...ommeeeeo 11ma 1amaa /Meta ama 3- 88- Instrut

---

Published as a conference paper at ICLR 2025
models with the win rates (WR) of model generations against the reference/base answers generated by GPT4-turbo. The comparisons are by default annotated by a GPT4-tuzbs and the resulting official AlpacaEval2 website.
WR has a 68.5% consistency with human evaluation, according Length Controlled (I.C) Win Rate (WR) is a debiased version of the WR that control for the length of the outputs and increase the WR's correlation to Chat Arena while significantly decreasing the length gameability of the annotator. To cross validate the effectiveness of the model and mitigate possible bias of GPT4, we also adopt Llama3-70B instruct as the judge, which is reported to have 67.5% win rate consistency to humans (close to the performance of CPT4). We include more detailed background information of AlpacaEval2 Appendix E..
For formulating the preference dataset D, we follow the standard RLHF pipeline by directly generat- ing answers from the model (which thus an on-policy dataset, but the algorithm is still offline) and get Al feedbacks as in SimPO (Meng et al. 2024); we generate 5 answers from Llama3-8B-Instt for each prompt in UltraFeddbcck (Cui et al. 2023), rank them with scores evaluated by ArmoRM (Wang et al. 2024a), and choose the best/worst one as winning/losing answer to form the preference pairs. For training, we adopted the popular library Transformer Reinforcement Learning (TRL) which already implemented most aforementioned xPOs algorithms and make everything under the same backend and easy to reproduce. If not specified, we train the model with 3 training epochs, which typically yields better performance for each XPOS according our replication.
4.1 EFFECTIVENESS OF DIFFERENT COMPONENTS
Individual Components Results. We first study the effectiveness of adding individual compo- nents. We use + to denote that only the component(s) after is added to DPO baseline as in Table From the results, we could notice that some components may not provide firm improvement over the baseline, no matter being added individually or combined. For example, for home advantage, we tune different values under the best performed /3 for DPO, and also always witness degradation in performances, see Figure La For link function, we examine the square loss in IPO and did not see performance gain over the DPO baseline. Other components (LN, Mixing reference policy, CS) in- deed help improve the metric even added individually. Compared to SimPO, using mixing reference policy yields also better results as in Figure 1b
2
The average WR gain is reported in the last column.
Models
AlpacuEval2 (GPT4)
AlpacaEval2 (Ilama3-700)
LC WR (%)
(%)
WR (%)
A (%)
LC WR (%)
(5)
WR (%)
A (S)
Avg. A (8)
Base model
41.88
42,29
57.78
$7.96
+ Length Norm. (LN)
44.27
2.39
42.37
0.08
61.37
3.59
58.94
0.98
+ 1.76
Ref. Policy Mixing (Mix)
40.18
-1.7
41.25
-1.04
60.67
2.89
57.95
-0.01
+0.04
+ Contextual Scaling (CS)
41.14
-0.74
41.44
-0.85
60.06
2.28
57.90
-0.06
+ 0.16
Link Function (LF)
39.53
-2.35
39.07
-3.22
58.13
0.35
56.34
-1.62
-1.11
+ Home Advantage (HA)
41.70
-0.18
39.85
-2.44
59.01
1.23
56.41
-1.55
-0.74
Rejection Sampling (RSO)
42.87
0.99
42,50
0.21
58.86
1.08
56.02
-1.94
0.09
Base model + LN
44.27
42.37
61.37
58.94
I.N. + Mix
47.45
3.18
45.89
3.52
61.91
0.54
58.07
-0.87
1.59
+ L. CS
45.92
1.65
42.36
.0.01
61.88
0.51
58.20
.0.74
+0.35
+ LL HA
42.77
-1.50
41.38
-0.99
60.99
-0.38
59.78
0.84
-0.51
+ I.N + RS
43.22
-1.05
41.90
-0.41
61.03
-0.34
$7.02
-1.92
00.3
LL SFT Loss
39.90
-4.37
38.66
-3.71
60.42
-0.95
58.94
0.00
-2.66
Table 2: Model performance results on each component after model training for 3 epochs.
Components Combination Results. Given the effectiveness of length normalization, we further test the combination of LN and other components. We do find that mixing policy could help improve the performance much more remarkablly when combined with LN-DPO than just DPO: it provides a 1.6% win rate extra gain compared to only 0.04% on DPO. However, we found that the RSO can improve DPO, but will yield worse performance when applied with length normalization. Thus, do find that despite that these elements are apparently mathematically orthogonal, they are not em- pirically independent. Given the positive results and effectiveness of length normalization, mixing reference policy and contextual scaling, we propose RainbowPO, as the combination of these three elements. We then examine the effectiveness of our method and each elements by gradually com-
https://Imarena.ai
https::/hugginggfaee wtt.ac.......c/ceceeeeeee/ecc/ee
8

---

Published as a conference paper at ICLR 2025
Win
mixing corstant
Ter pemperature
50
425
nete (K6)
Vin (40)
Winrate (N4
60
LC etn rate (GPP4)
44
Win Wirrale (GP74)
siir [Llamed 1081
rate
20
Win rale (Jama3-7261
(SP74)
0.01
0.05
0.2
0.5
0.00
0.25
0.50
0.75
1.00
02
0.5
1.0
2.0
5.0
Tomp
(a) Effects of Home Advantage / (b) Effects of Reference Policy Margin.
(c) Performance Difference for dif- ferent temperature in RSO.
Mixing.
Figure Dynamics of changing home adv., reference policy mixing and different temp. RSO.
bining the elements one by and greedy search of the best hyper- syper-parameters. We finally achieve 51.66% win rate for AlpacaEval2, surpassing the GPT4-1106 preview (see details in Appendix
Models
AlpacaEval2 (GPT4)
AlpacaEval2 (Llama3-70BB) WR
RainbowPO
LC WR (%)
WR (%)
LC WR (%)
avg length
model
41.88
0.77
42.29
1.46
57.78
$7.96
2,169
Length Norm.
44,27
0.75
42.37
1.45
61.37
58.94
1,942
Ref. Policy Mixing
47.45
0.70
45.89
1.49
61.91
58.07
1890
Warm-up Adjustment
48.52
0.80
45.88
1.45
63.37
59.95
1,919
Contextual Scaling
51.66
0.78
47.92
1.49
63.94
39.69
1,878
Table 3: Evaluation of RAINBOWPO by adding new components consecutively.
Ablations on RainbowPO. We also conduct an ablation study of our proposed RAINBOWPO algorithm. All components of our proposed in our algorithm useful, as Table A for which we use @ to denote that the methods are based on composition of the method on previous line and new elements in Table We notice that adding length normalization is indeed important and the most critical importance among the components for RainbowPO. We also include ablations training epochs Appendix which showcases that epochs yield the sweet spot.
Models
Avpa2aEval2 (GPT4)
AlpacaEval2 (Llama3-70B) WR (%)
LC WR (%)
WR (%)
LCC WR (%)
avg length (L)
RainhowPO
51.66
0.78
47,92
1.49
63.94
59.69
1,878
Ref. Policy Mixing
50.52
0.78
47.49
1.46
64.64
60.43
1,886
Contextual Scaling
48.52
0.80
45.88
1.45
63.37
59.95
1,919
Length Normalization
45.68
0.78
42.43
1.47
57.43
58.01
2108
Table 4: Ablation study of the newly proposed elements in RAINBOWPO.
4.2 COMPARISON WITH BASELINE METHODS
Table 6 shows the comparison between RainbowPO with the baselines. For fair comparsion, we first compare RainbowPO with the baselines one training epoch, shown in Table RainbowPO performs the best, beating SimPO while achieving lower average length.
Models
AlpacaEval2 (GPT4)
AvpacaEval2 (1 (llama3-70B) WR (%)
LC WR (%)
WR (%)
LC WR (%)
avg length ())
DPO RRfailov 2023
37.95
37.36
1.42
55.46
54.03
1,989
IPO ( Azar 2024
34.80
34.52
1.40
52.67
50.93
1,956
KTO
Ethayarah
2023
35.61
33.19
1.38
55,94
51.74
1,876
CPO
Xu 2024
31.89
34.92
1.38
53.33
54.84
2,155
ORPO Hong 2024
22.91
22.59
1.24
48.41
45.90
1,914
SimPO MMeng
2024
47.96
41.17
1.44
61.94
54.22
1,730
RainbowPO (1 epoch)
48.08
42.53
1.43
61.36
54.60
1,683
Table 5: Methods comparison under one training epoch.

---

Published as a conference paper at ICLR 2025
When we increased the training epoch to 3, interestingly, we also noticed that the same phenomenon as what (Meng et al. 2024) reported: SimPO rarely benefits from more epochs of training. However, RainbowPO and DPO both gets increase in the winning rate after another two epochs of training, making the RainhowPO PO get a 51. 51.66% win rate against GPT4 under GPT4 as a judge. This advantage not only benefits the final performance, but also might play larger impact when the alignment dataset is small or expensive to collect and will be beneficial to reuse, which is quite common in reality.
Models
AlpacaEval2 (GPT4)
AlpacaEval2 (Llama3-70B) LC WR (%)
LC WR (%)
WR (%)
WR (%)
avg length (L)
DPO (Rafailov al. 2023
43.65
43,94
1.46
60.13
58.20
2,284
SimPO* Meng et al. 20224
48.40
44.57
1.47
60.90
56.46
1,843
RainbowPO (3 epochs)
51.66
47.92
1.49
63.94
59.69
1,878
Table 6: Methods comparison under three training epochs. "Hyper-parameters are further adjusted for the best performance.
4.3 LIMITATIONS AND FUTURE WORK
Broader tasks. In this paper, we focus our evaluation on models trained with LLama3-8B In- struct as the base model. Exploring other models of varying sizes, such as Gemma (Team et al.) 2024) or Mistral (Jiang et al. 2023a), could possibly enhance the generalizability of our findings. It will also be beneficial if we could repeat the pipelines and compare the algorithms" performance on other LLM evaluation metrics, like arena-hard or MT-bench, though MT-bench is known to be less tinguishable for RLHF algorithms. Other directions include benchmarking the effectiveness of alignment algorithms on improving other capabilities of LLM other than instruction following, like reasoning (Xiong al. 2024b) or COT (Choi al. 2024 However, due to constraints in com- puting resources and time, we defer this investigation to future work. Nevertheless, we believe that our work provides a unified and comprehensive framework for helping find the best preference optimization algorithms, and further pushing the boundary of offline RLHF for LLMs.
Ideas from other xPOS. We were not able to explore other aspects of existing DPO variants in detail, and there might be still promising candidates in further improving the preference of Rain- bowPO. Some methods that propose to update the reference policy dynamically: sDPO (Kim et al.
2024 TR-DPO (Gorbatovski et al. 2024); or learning from noisy preferences Chowdhury et al. Additionally, we also recognize the recent literature in pursuing online methods, such as online DPO (Guo al.) (2224) or iterative DPO (Yuan al. 2024 Xiong et al.] (022a)), which provide valuable insights on possibly further improving the downstream task performance: we will pursue them in future research. Other extensions beyond RLHF include, Nash Learning from human
(2024.
feedback (Munos et al. 2023), and self-play preference optimization (Chen et al., 2024b).
Demystifying observations. We also made some interesting observations in the paper, which we fail to find proper mathematical explanations and may boost further research. For example, the RainbowPO objective could benefit much more than SimPO when increasing the training epochs, but reasons for such phenomenons orthogonal components are actually not empirically independent, for example, RSO can improve DPO, but can not be readily combined with other components like length normalization. It is also interesting to see the some combination of components reach effects "1 + 1 > 2"; it will be inter- esting to understand the deeper underlying reasons and could potentially lead to better algorithms.
still unknown. In addition, found some mathematically
5 CONCLUSION
In this paper, we propose RAINBOWPO, a comprehensive framework that demystifies and enhances existing DPO methods through the integration of key components into a unified objective. Our find- ings highlight the effectiveness of length normalization, reference policy mixing, and contextual scaling. However, the selective application of rejection sampling and home advantage is not pro- viding incremental improvements either individually or when paired with the other methods. By demonstratin that these enhancements can coexist within a single algorithm to achieve tf-te-of-the- art performance for fine-tuning SOTA commercial LLM, pave the way for future research and practical applications. We aim for this work to serve as a foundation for refining DPO methodologies and to inspire further exploration of untested components for integrated agents.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We thank Stephen Rawls, Supriyo Chakraborty and Kartik Balasubramaniam for helpful feedbacks and pointers to relevant works. We also thank Benjamin Therien, Ashwinee Panda, Zain Sarwar, Akshaj Kumar Veldanda, and Andrei Mircea for helpful discussions.
Wenpin Tang and Hanyang Zhao are supported by NSF grant DMS-2206038, start-up grant at Columbia University, and the Columbia Innovation Hub grant. The works of Hanyang Zhao and David D. Yao are part of a Columbia-City Initiative, The Government of the HKSAR and the AIFT Lab.
collaborative project that is supported by InnoHK
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, and Shyamal Anadkat. GPT-4 technical report. arXiv:2303.08774, 2023.
Afra Amini, Tim Vieira, and Ryan Cotterell. Direct preference optimization with an offset. arXiv preprint 0rXiv:2402.10571, 2024.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. AISTATS, pp. 4447-4455, 2024.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, and Cameron McKinnon. Constitutional Al: Harm- lessness from AI feedback. (rxXi:2222.08077 2022.
Ralph Allan Bradley and Milton E Terry Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-344 1952.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, and Amanda Askell. Language models few-shot learners. In Neurips, volume 33, pp. 1877-1901, 2020.
Haoxian Chen, Hanyang Zhao, Henry Lam, David Yao, and Wenpin Tang. Mallowssdpo: Fine-tune your IIm with preference dispersions. arXiv preprint arXiv:2405. 2024a.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv:2401.01335, 2024b.
Eugene Choi, Arash Ahmadian, Olivier Pietquin, Matthicu Geist, and Mohammad Gheshlaghi Azar.
Robust chain of thoughts preference optimization. In Seventeenth European Workshop Rein- forcement Learning, 2024. URL htttpp:://operrevieee net wtt.:eeeeoormmm21111loolooa..
Sayak Ray Chowdhury, Anush Kini, and Nagarajan Natarajan. Provably robust dpo: Aligning language models with noisy feedback. arXiv preprint arXiv:2403.00409 2024.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ulrrafedbbcck: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01377, 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, and Angela Fan. The Ilama herd of models. arXiv preprini urXiv:2407.2 1783, 2024.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsumori B Hashimoto. Length-controlled al- pacaeval: A simple way debias automatic evaluators. arXiv preprint arXiv:2404.04475, 2024.
Kawin Ethayarajh, Winnie Xu, Dan Jurafsky, and Douwe Kiela. Human-ave aware loss functions (ha- los). Technical report, Contextual AI, 2023.
11

---

Published as a conference paper at ICLR 2025
Alexey Gorbatovski, Boris Shaposhnikov. Alexey Malakhov, Nikita Surnachev, Yaroslav Aksenov, Ian Maksimov. Nikita Balagansky, and Daniil Gavrilov. Learn your reference model for real good alignment. arXiv preprint arXiv:2404. 09966, 2024.
Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, and Bilal Piot. Direct language model alignment from online ai feedback. arXiv preprint arXiv:2402.04792 2024.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bital Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In AAAI, volume 32, 2018.
Jiwoo Hong, Noah Lee, and James Thore. Orpo: Monolithic preference optimization without reference model. arXiv preprint arXiv:2403.07691 2024.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, and Lucile Saulnier. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023a.
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. Lim-blender: Ensembling large language models with pairwise ranking and generative fusion. arXiv preprint arXiv:2306.0255 2023b.
Dahyun Kim, Yungi Kim, Wonho Song, Hyconwoo Kim, Yunsu Kim, Sanghoon Kim, and Chanjun Park. sdpo: Don't use your data all at once. arXiv preprint PrXiv:2403.192700 2024.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, and Yejin Choi. Rewardbench: Evaluating reward models for language modeling, arXiv preprint arXiv:2403. 13787, 2024.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi Rlaif: Scaling reinforcement learning from human feedback with Al feedback. arXiv:2309.0022 002677 2023.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instrwction-following models. /https://
_eaal] 2023.
Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. arXiv preprint arXiv:2309.0666 2023.
Tianqi Liu, Wei Xiong, Jie Ren, Lichang Chen, Junru Wu, Rishabh Joshi, Yang Gao, Jiaming Shen, Zhen Qin, and Tianhe Yu. Rrm: Robust reward model training mitigates reward hacking, arXiv preprint arXiv:2409. 13156, 2024a.
Yixin Liu, Pengfei Liu, and Arman Cohan. Understanding reference policies in direct preference optimization. arXiv preprint urXiv:2407.13700 2024b.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-free reward. arXiv preprint 2rXiv:2405.14744 2024.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, and Georg Ostrovski. Human-level control through deep reinforeement learning. nature. 518(7540):529-555 2015.
Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, and Andrea Michi. Nash learning from human feedback. arXiv:2312.00888, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, and Alex Ray. Training language models to follow instructions with human feedback. In Neurips, volume 35, pp. 27730-27744 2022.
12

---

Published as a conference paper at ICLR 2025
Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization. arXiv preprint arXiv:2403.19159, 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In Neurips, volume 36, 2023.
Antonin Rafinn, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dor- mann. Stable-baselines:: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268): 2021.
Amir Saeidi, Shivanshu Verma, and Chitta Baral. Insights into alignmefnt: Evaluating dpo and its variants across multiple tasks. arXiv preprint arXiv:2404.14723, 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv:1707.06344 2017.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. In Neurips, volume 33, pp. 3008-3021, 2020.
Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Rémi Munos, Mark Row- land, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires, and Bilal Piot. Generalized preference optimization: A unified approach to offline alignment. arXiv:2402.05744 2024.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, and Alexandre Ramé. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.001 18, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, and Shruti Bhosale. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Haoxiang Wang. Wei Xiong. Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-experts arXiv preprint arXiv:2406. 12845, 2024a.
Yizhong Wang, Hamish Ivison, Pradeep Dasigi. Jack Hessel, Tushar Khot, Khyathi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, al. How far camels go? exploring the state of instruction tuning on open resources. Advances in Neural Information Processing Systems, 36:74764-74786 2023.
Zhichao Wang, Bin Bi, Shiva Kumar Pentyala, Kiran Ramnath, Sougata Chaudhuri, Shubham Mehrotra, Xiang-Bi Bo Mao, and Sitaram Asur. A comprehensive survey of Ilm aligmment tech- niques: Rlhf, rlaif, ppo, dpo and more. arXiv preprint arXiv:2407.16216. 2024b.
Genta Indra Winata, Hanyang Zhao, Anirban Das, Wenpin Tang, David D Yao, Shi-Xiong Zhang, and Sambit Sahu. Preference tuning with human feedback on language, speech, and vision tasks: A survey. arXiv preprint arXiv:2409.1 1154., 2024.
Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang- Iterative preference learning from human feedback: Bridging theory and practice for rihf under kl-constraint. In Forty-first International Conference on Machine Learning, 2024a.
Wei Xiong, Chengshuai Shi, Jiaming Shen, Aviv Rosenberg, Zhen Qin, Daniele Calandriello, Misha Khalman, Rishabh Joshi, Bilal Piot, and Mohammad Salch. Building math agents with multi-turn iterative preference learning. arXiv preprint (tt.::24090023922. 2024b.
Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of LLM performance in machine translation. arXiv:2401.08417. 2024.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho. Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXiv:2401. 2024.
13

---

Published as a conference paper at ICLR 2025
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Salch, and Peter J Liu. Slic-hf: Sequence likelihood calibration with human feedback. wrXiv:2305.10425, 2023.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv:1909.08599, 2019.
14

---

Published as a conference paper at ICLR 2025
A BACKGROUND ON RLHF AND RL
A.1 RLHF
RLHF (Ouyang et al. 2022 Stiennon al., 2020 Ziegler et al. 2019). On top of T RLHF is proposed to serve as the next step to conduct further fine-tuning to generate high-quality outputs as judged by humans. Given a generative model x, the model π prompted with prompts x to produce pairs of answers (or, "completions") {v1, 1/2) Tyy x)), which are then presented to human la- belers who express preferences for one completion over the other. Denote by you > 3/t 3r, meaning that Vou E {y1 y22 is preferred over yt E {y1, 32 The preferences are assumed to be generated by some latent reward model r" (I. 3). which we do not have access to. Based on the collected prefer- (i) (8) )
ence data (zz(*), Yiu 3/ Fi=1 RLHF consists of first learning a reward model r(z:, y), followed by learning policy Ty (y/ 1) in which the prompt x is the state, and the completion is the action.
(a) Reward Model. To capture the underlying human preferences, RLHF assumes the Bradley-Tery
model (Bradley & Terry 1952) that stipulates the pairwise preference distribution:
exp (r" y1))
p" (yl > 1/2 x))
exp (r" (I,, 11)) + exp (r" (x., (2,, = a (r* (1) - r" (ar, (2), (19)
where al-) is the sigmoid function. Given access to a static dataset of comparisons D (x((), yar V1 (8)
li=l...,, RLHF seeks to approximate the latent reward (x, y) by family of func- tions (re y) ho. and estimate the parameters by minimizing the (negative) log-likelihood loss ming L (rw, D) == (.,.....)>> [log a (rw (I,, Ww) r (I, mn)]]. Denote by re_ (æ, y) the solu- tion to this problem.
(b) RL. The learned reward function rw, (z. y) is then used to provide feedback to the language model. More precisely, the following KL-regularized RL problem is considered:
max [Ey [rw. (x. y)1 - 3KL (IT(- 2>) [eee( =))]
(20)
where B >> is tlyper-paramesn controlling the deviation from the reference policy Neef SFT The regularization is important as prevents deviating too far from the SFT model that is trained to conform to the true preference, while maintaining the generation diversity to avoid mode- collapsing to single high-reward answer. In view of Equation 20 RLHF uses the reward function r(z, v) T (2r y) (log ( x)) log Tet (y) x)), and solves the RL problem by proximal policy optimization (PPO) (Schulman et al. 2017
A.2 DPO
One disadvantage of RLHF is that the RL step often requires substantial computational effort (e.g.. to carry out PPO). The idea of DPO is to combine the reward model and RL in RLHF into single objective, bypassing the computation in the RL step. The key realization that given a reward
function y), the RL problem in Equation 20 closed-form solution (y x)) 2(z) Tref (y)
a:) exp (1r(2, where Z(x) = Eg Teet
Trer (y x) exp ( (3r(z, v)). Rewrite the above as r(r. y)
Me(ulx) + 3 log ((x). Through this change of variables, the latent reward (xx, y) can be log Fee [)] expressed in terms of the optimal policy (y | z), the reference policy Tret (u | .)) and a constant z (x). Substituting this r* expression into Equation yields:
p² (y1 > U2 x) = a ( log
T (1) x) Log
n (y/2 x))
B log Tief (W2 xx)
(21)
Tref (y) x)
where (z) cancels out. the preference distribution only depends on T x) and Jeaf (y x)).
The expression in Equation 21 motivates the DPO objective:
To (yw xx)
To (31 x))
min LDPO (To: Tret) == (E. un DD log
BBlg
B log
Tief (yw x) ree (v) x))
(22)
15

---

Published as a conference paper at ICLR 2025
B MISCELLANEOUS
Table 7 lists the common notations used in this paper. The table serves as quick reference guide for understanding the mathematical expressions and technical terms used throughout the paper.
Name
Notation
Description
Input Sequence
Input sequence that is passed to the model. Expected label or output of the model.
Output Sequence
Dispreferred Response Preferred Response
y1
Negative samples for reward model training.
Positive samples for reward model training.
Optimal Policy Model
Optimal policy model.
Policy Model
Generative model that takes the input prompt and me
returns a sequence output or probability distribution. Generative model that used as a reference to ensure the policy model is not deviated significantly.
Reference Policy Model
Tel
Preference Dataset
Dataset with set of preferred and dispreferred responses. Dataset with a set of preferred and dispreferred responses sampled by Rejection Sampling.
Preference Dataset by RSO
DRSO
Loss Function
C
Loss function.
Regularization
B
Regularization Hyper- papee-parameter for preference tuning.
Mixing Hyper-parameter
Our proposed mixing coeflicient for better reference policy. Reward score.
Reward
Target Reward Margin
The margin separating the winning and losing responses.
Table 7: Table of Terminology and Notation.
16

---

Published as a conference paper at ICLR 2025
C MATHEMATICAL EXPLANATION OF DIFFERENT XPOS
Here first list popular XPOS variants in the literature in Meng al. (2024) and Winata al. 2024).
Method
Objective
DPO
-logo o ( log
Ne (V(W
-3 llg (v(l) West (1r1
Bee (Brn
IPO
(log log
To (Nw lxx) (ael (2x)
B log
(arel (2elz) Tire (uri
f-DPO
log a (3f'
No To(y 120) Trwat (thur (z)
38
To (z))
Towe (
KTO
-n
log
Rid Mon (2)
no (xx)
2ref
Aeor
2xef - -3log
(on (ar)
where Zref
IE ()) J- [3KL (Te (2/
ODPO
- -log
[3 log
no (Vel Thet (ee (x)
33 log
ne (we (x) (er((
- D (x)))
MallowsPO
log (Φ(x) [ log
To (ow (xc) Fest Feel
B log
To To Tree Pue
R-DPO
log
(3 log no (Vwl [x) (er (x) 13 log
mo (x)) Werr (ax
(alywl allee))
CPO
log pe (yhr (r) log (3 log TO ( (yw / /x) - B log To
ORPO
log pe (yu (r) X logg (log
po polvmlls (x)
log po (en (2)
I-pe 1-po(2112
where pa x.) exp ( [ log (o(())))
SimPO
logg
A log To (y
A lsol
log TO (2112)) -7)
Table 8: Various preference optimization DPO objectives. The table is inspired from Meng et al. 2024) and Winata et al. (2024).
Next we include the categorization of different methods, which appeared earlier in Table 9 and explain detailedly the reasons after.
Method
Length Norm.
Link Fune.
Home Adv.
Ref. Policy
Contextual Scaling
RS
SFT Loss
DPO
logistic
SFT
SLiC-IIF
hinge
SFT
IPO
square
SFT
X
X
CPO
X
logistic X
Free
X
X
RSO
X
logistic/ hinge
X
SFT
X
V
X
ODPO
X
logistic
X
X
X
ORPO
logistic
Free
implicitly
WPO
logistic
SFT
MallowsPO
logistic
SIT
SimPO
logistic
Free
PO
logistic
mixing
X
X
Table 9: Mapping of xPOS with mathematically orthogonal components and validation results of their effectiveness by the downstream task evaluations.
17

---

Published as a conference paper at ICLR 2025
C.1 DIRECT PREFERENCE OPTIMIZATION (DPO)
The loss of DPO (Rafailov et al. 2023 is:
We (yh
x)) x))
Te (yr
x))
LDPO (mos Tret) == TR (a (w:
log a log
B log
(23)
Tel
(yo
Tel
(y)
x3)
which can be seen that, as the baseline methods we investigate in this paper, length normalization adopted, link functiin log adopts the logistic function, home advantage can be seen as None
or O, reference policy is the SFT policy be aligned, contextual scaling, the dataset is the hybrid or offline dataset thus there is no rejection sampling. Also there is no SFT loss.
C.2 SEQUENCE LIKELIHOOD CALIBRATION FROM HUMAN FEEDBACK (SLIC-HF)
The loss of SliC-HF (Zao et al. 2023 is (Uret refers answer generated by Teet):
(TI; Tyot) = E (a.go.. max (0, 6 log Te (3/ (z) + log Te (x)) log Te (Mrer
(24)
rank calibration loss
SFT
which can be seen that, no length normalization is adopted, link function max 5 x) adopts the hinge function, home advantage can seen as None or O, reference policy is the SFT policy to be aligned (in the regularization term), contextual scaling, the dataset is the hybrid offline dataset thus there is no rejection sampling. There is an SFT loss, which acts as the role regularization.
C.3 IDENTITY PREFERENCE OPTIMIZATION (IPO)
The loss IPO Azar et al. 2024 is:
Te (y/w x) Tiet (yw xx)
B log me (y1 x))
1 2
LIPO (To: Tret) Eaganonn)>
(Blog
(25)
Tea (y x)
2
which can seen that, no length normalization adopted, link function (x 1/2)² adopts the square function, home advantage can be seen as None or O, reference policy is the SFT policy to be aligned, no contextual scaling, the dataset is the hybrid or offline dataset thus there is no rejection sampling. Also there is no SFT loss.
C.4
CPO
CPO (Xu et al. 2024) is motivated to improve the memory and speed efficiency of DPO by neglect- ing the reference policy, further accompanied by SFT loss term:
To ( (x: (Blog
CCPO (Te) = - -E (x (1) log (e (x) + log
(26)
Te (3 (x)
which can be seen that, no length normalization adopted, link function log a adopts the logistic function, home advantage can be seen as None or 0, reference policy is None (or free), no contextual scaling, the dataset is the hybrid or offline dataset thus there is no rejection sampling. Also there an SFT loss.
C.5 RSO
The loss of RSO (Rafailov et al. 2023) is:
xx)
Te (yn (x)
Lnso (mo; Reat)
we D
log
B log
3 log Trel (3)) x))
(27)
Wee
(yar
which can be seen that, it only differs from DPO in applying rejection sampling for formulating the preference dataset.
18

---

Published as a conference paper at ICLR 2025
C.6 ODPO
ODPO (Amini et al. 2024) proposed to add a margin to capture the significance of preference pairs; they model this margin, they call offset A, monotonically increasing function /()) of the difference between the scores associated with the responses:
(r, Vus 31) af (score (he) score (ar, 91)),
(28)
where a a hyper-paramest that controls the extent to which an offset should be enforced. The resulting objective becomes:
LODPO (To; Tref) ==
log [o(Uw(z)) To(M Dr (xx Mers ar)) ) (Tr (x)
logo
log
(29)
Tie (yel)))
ODPO only differs from DPO in applying a contextual dependent margin/home advantage.
C.7 ORPO
Opposed to maximizing the likelihood ratios of winning and losing answers in the preference pair in DPO, ORPO (Hong al. 2024) propose that odds ratio can be sensible choice.
CORPO (Ta) =
po (y (x)
log Po (y/ (x)
- -B(x.ou
log (o (x)
Alog
log
(30)
(D (x)
pe (30l
where pe(y)()) = exp [p] log (o(y)z)) ORPO is similar to CPO in the sense that is also refer- ence model free and combined with SFT loss; in addition, notably that ORPO also adopts form of length regularization by normalizing the likelihoods with respect to the length, as in the definition of (x)); finally, they compute odds ratio instead of the original likelihood ratio.
ORPO differs from DPO in being reference-mode free and also yields a SFT term for regularization. Its contextual scaling implicit, shown in Theorem for ORPO equivalent objective.
C.8 MALLOWSPO
Chen et al. (2024a) propose a contextual scaled objective derived from MLE under Mallows: com- pared DPO that puts equal weights on each prompt and preference pairs, the resulting MallowsPO adds contextual scaling factor ((x) that represents this dispersion of the preferences of answers to each prompt x:
MallowaPP (To; Tref) ==
To(Hwolz)) B log
T ]
E B(s.De log (((x) 3log
(31)
Teat (yellz)))))
Teet Teer (z)
To compute this dispersion, MallowsPO provided a direct approach by using normalized predictive entropy of preference pairs {v.", v/)i=......N with N = max(1y")) lv'))
M H (Y4+1 Y v:")) H (Y141 YY; = vf)] )
((x) = - log
(32)
log(n)
MallowsPO differs from DPO in adding contextual scaling factor; others are the same.
C.9 SIMPO
SimPO Meng et al. 2024) proposes simple yet effective objective that claimed to match or even outperform the performance of DPO:
3 [
3 Iyl
log no ( (100)2))) -7)] (33)
LsimP (TT) = - F(c..n..0))>
log
log Te (v)
19

---

Published as a conference paper at ICLR 2025
where y is introduced target reward margin to help separating the winning and losing responses. SimPO is similar to CPO in the sense of being reference model free; also adopted the length normalization for the likelihoods in ORPO; finally, it additionally introduced a constant margin to tuned that could help to further improve the performance by encouraging a larger difference between the normalized likelihoods.
SimPO differs from DPO in adopting length normalization, a margin/home advantage term and is also reference-model free.
D PROOFS AND DETAILS
D.1 PROOF OF ORPO UPPER BOUND EQUATION 4
Here we prove that the part of preference optimization in ORPO's 's loss yields an upper bound which has instinct connection to SimPO loss, specifically the idea of length normalization.
Theorem 1. Assume that the normalized implicit reward model difference for preference pairs:
1
1
Ao(a, Ywu y1)
log no (Yho (r)
log To (z) 2 0
lyvl
almost surely. Then for the part of preference optimization ORPO loss, i.e.
log po(ywl po (y
log (o(()(2)) po po (x)
CORPO-rO (TT) B (a. log
(34)
has an upper bound such that
CORPO-PO EElogo ())
log To (b
log (b(((()))
(35)
180
Proof. Since - log (-) is monotone decreasing function, when prove that for any x,
0. it suffices to
x log
y
1
log
2
log
(36)
1 1
which x (o 1), y = (o(()((2)) The inequality is equivalent to:
f(x) = log
1 - - 1
log
2 log
y
(37)
y/
1 -
Taking gradient of f(z) with respect to x, we have:
1
1
f() x α T
2
x01
2)
x - y)
Moreover, we have f(y) log( (y))), which yields the desired result.
20

---

Published as a conference paper at ICLR 2025
E EXPERIMENTAL DETAILS
E.1 ALPACAEVAL AND ALPACAEVAL2 INTRODUCTION
We briefly introduce AlpacaEval (Li al. 2023 and its evaluation pipeline and metric. This part is mainly modified and summarized from descriptions in AlpacaEval official Github repository
In practice, for either training LLM comparison between different LLMs, we need to evaluate an mnstruction-folloinng model (e.g.. ChatGPT). However, evaluation of such models typically requires human interactions, which time-consuming. expensive, and hard to replicate.
AlpacaEval. AlpacaEval is an LLM-based automatic evaluation that is fast, cheap, replicable, and validated against 20K human annotations. operates on fixed set of 805 instructions chosen be representative of user interactions on the Alpaca web demo [ and has been widely adopted for model development. Concretely, AlpacaEval adopts an automatic evaluator that has high agreement with humans (validated on 20K annotations). and evaluates model by measuring the fraction of times a powerful LLM (e.g., GPT-4) prefers the outputs from that model over outputs from a reference model. The evaluators of AlpacaEval enable caching and output randomization by default.
AlpacaEval2. AlpacaEval2 (Dubois et al. 2024), also called Length Controlled AlpacaEval, is length-debiasel version of AlpacaEval. One of the major issues of AlpacaEval is that increase the win-rate by increasing the length of outputs. The main idea Length Controlled (LC) WR is that for each model, AlpacaEvva22 will fit logistic regression the autoannotator given: (1) the instruction, (2) the model, and (3) the difference of length between the baseline and model output. Given such a logistic regression AlpacaEval2 can then try predict the counterfactual "what would the preference be if the model's 's output had the same length as the baseline" by setting the length difference A AlpacaFval2 then obtains the LCC win rates.
predict the preference of
By averaging over this length-controlled preference,
Length controlled win-rates increase the correlation between AlpacaEval's leaderboard and Chat Arena from 0.93 to 0.98 Spearman correlation, while significantly decreasing the length gameability of the annotator. We refer the more concrete details of the datasets, comparison of models or evalu- ators and leaderboards to AlpacaEval's ''s official website https: /ttp//gttttttt//// wtt.:/ccctttttttteeoe lab/ stanford alpacal
Our paper. In this paper, adopted two evaluators: GPT4 turbo and Llama3 70B to evaluate our models' generations against base/reference answers generated by GPT4 turbo. We adopt the default template for evaluators provided by AlpacaEval and adopt the following fixed generation config for each model: max new tokens 4096, temperature 0.7 and top p 0.1.
E.2 TRAINING DETAILS
Here we report the best hyper-pane saper-parrmeters we searched which corresponds to our final results. We include the modified dpo trainer and training scripts in the supplementary materials.
E.3 ABLATIONS ON TRAINING EPOCHS
We also conduct ablation studies on the training epochs, under the other best hyper-parameter: we found under the 3 training epochs. The training loss be found in Figure and the AlpacaEval results could be found Table [3] The results indeed show that 3 training epochs yield the sweet spot, further additional epochs could slightly degrade the performance.
21

---

Published as a conference paper at ICLR 2025
Models
SFT A
Ir
WR
Base model
0.01
1
00
0
3e 7
0.1
Length Norm. (LNN
10
0
0.1
+ Ref. Policy Mixing (Mix)
0.01
0.25
0.1
00
0
3e 7
0.1
Contextual Scaling (CS)
0.01
0
3e
0.1
+ Link Function (LF)
0.001
00
0
3-
0.1
+ Home Advantage (HA)
0.005
0.001
30-
0.1
+ Rejection Sampling (RSO)
0.01
0.2
0
3c
0.1
Basc model + LN
10
06
0.1
+ LN Mix
10
0.25
0.1
0
0.1
+ LN + CS
00
e-
0.1
+ LN HA
10
0.05
0.1
LN RS
10
0.2
0.1
+ LN + SFT Loss
10
00
0.1
c
0.1
Table 10: Hyper-parameters for results reported in Table
Models
3
a
Y
Ir
WR/WS
Base model
0.01
1
0
3e
0.1
+ Length Norm. (LN)
10
0
6
0.1
Ref. Policy Mixing (Mix)
10
0.25
0.1
e-6
0.1
+
Warm-up Adjustment
10
0.25
0.1
e
150
+ Contextual Scaling (CS)
10
0.25
0.1
e
150
Table 11: Elyper-paameee for results reported in Table
Models
@
Ir
WR/WS
DPO* (Rafailov al. 2023
0.01
3e-7
150
SimPO Meng et al. 2024)
10
0
0.1
e
150
RainbowPO (3 epochs)
10
0.25
0.1
e-
150
Table 12: Hyper-parameters for Table
Models
AlpacaEval2 (GPT4)
LC WR (%)
WR (%)
avg length (4)
RainbowPO
51.66
0.78
47.92
1.49
1,878
Training Epoch
46.36
0.77
39.32
1.43
1,717
2 Training Epochs
50.88
0.77
47.34
1.47
1,912
4 Training Epochs
50.61
0.74
47.02
1.49
1874
Table 13: Ablation of training epochs for RAINBOWPO (3 training epochs) using the same other hyper-parameters.
22

---

Published as a conference paper at ICLR 2025
train/loss
RainbowPO training loss
1
0.8
0.6
0.2
train/epoch
3
Figure 2: Training loss with respect to training epochs.
23


Reviews:
[{"review_id": "CaGfgSKCUj", "reviewer": "Reviewer_FkHK", "summary": "This paper aims to provide a unified framework for direct preference optimization (DPO) methods which are becoming a popular and successful method to train or finetune generative language models, and in particular, for better alignment with human preferences. In DPO, we are given a dataset of triples consisting of a prompt and a pair of potential responses, one of which is known to be preferred over the other. The high level goal is to fine tune a large language model in a manner that maximizes the difference in the likelihood of the preferred and dispreferred responses. The performance of a model is measured by the win rate of responses of the model being preferred over the responses of another model over a collection of inputs as predicted by an automatic evaluator which is typically another generative language model trained to predict which of two responses a human annotator would prefer.\n\nThe paper surveys, then carefully analyzes the objective functions of different algorithms in a growing family of DPO like models, identifies their components and then formulates an objective function for their proposed RainbowPO method which incorporates the components and techniques of previous DPO methods. The contribution of the different components of the objective function can be tuned smoothly using hyperparameters. The paper then goes to evaluate the effectiveness and contributions of each component to the performance of DPO methods.\n\nAfter rebuttal:\nI am increasing my score from 5 to 6. The authors have addressed my questions. Overall, I lean towards acceptance if there is room to accommodate it.", "questions": "- Could you please elaborate on the evaluation? What was the base model against which the win rate was  evaluated in Section 4.\n- Could you please elaborate on the use of link functions?\n- Could you please elaborate on the mixing reference policy results in Section 4? What are the policies being mixed? How is the \"good\" policy \\pi_\\gamma obtained?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- A careful analysis of different DPO methods, identifying their components and understanding the contribution of components of different DPO methods to performance is a worthwhile endeavor. The main technical contribution is a careful analysis and rewriting of the objective functions of recently proposed DPO methods and unifying them as a single objective function. \n- Another contribution is the proposed the mixed reference policy obtained by mixing a fine tuned policy (model) with a policy that maintains a certain margin or odds ratio of the likelihood of the preferred to dispreferred responses.\n- I believe the main conceptual contribution is the development of common frameworks and pipelines that allow researchers and practitioners to evaluate different components and techniques for DPO. As such, the paper is well motivated and the findings of this paper and follow on works that attempt to perform such evaluations will be of significance as alignment of generative language models is an important problem and existing reinforcement learning based approaches are computationally expensive in comparison to DPO.\n- The main finding that length normalization, policy mixing and contextual scaling contributed towards increasing performance are a potentially interesting takeaway that may be significant to practitioners.", "weaknesses": "- The paper is overall poorly written with several missing explanations and gaps in the logic. While it is possible to guess what is meant, a reader unfamiliar with the area will likely find it very hard to follow.\n- At a high level, the organization of the paper makes it hard to read as the text on the background and proposed RainbowPO model frequently references results in the later experimental section which distracts from understanding the components of the DPO objective functions.\n- The technical contributions and novelty are limited beyond the creation of the unified objective function and evaluation pipeline.\n\nSome examples of issues with the writing:\n- line 153: ... there lacks a work .. please consider revising. This is just one minor example. There are several more throughout the paper and they are too numerous to list. Please consider performing a thorough reading of the paper.\n- please consider elaborating on the verbosity issue and how it may be potentially a way to game the evaluator\n- different places use y_l and y^l. Is this a typo? Similar issues exist elsewhere. Preferably please keep the usage of subscript and superscript consistent. For example, see in (10) the use of \\pi_\\alpha and \\pi_{ref}^\\alpha. Consider e.g. using \\pi_\\alpha, \\pi_\\alpha^{ref} and \\pi*{ref} as appropriate instead\n- abbreviation LC (length controlled) is not defined or explained anywhere making it hard to understand results in Section 4.1\n- in general, it would be nice to have a more clear discussion of the evaluation framework and evaluation criteria. As it stands, the paper assumes too much background knowledge e.g. about AlpacaEval.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OB8GayidMu", "reviewer": "Reviewer_Nycb", "summary": "The paper extracts key components from various DPO variants and provides theoretical analysis for each component. It then establishes a unified formulation that encompasses most DPO variants. Empirically, the author conducts a series of ablation studies to analyze the effectiveness of each component in DPO variants and identifies the key components that contribute to metric gains. By combining these components, the author proposes a new variant of DPO called RainbowPO, which outperforms all other DPO variants.", "questions": "1. For eq (1), how to obtain the policy $\\pi_\\gamma$ in the practical setting?\n2. What is the purpose of raising eq (12)? If left side is a lower bound then whether minimizing the lower bound can imply anything?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work is the first to provide a comprehensive analysis of the effectiveness of various DPO variants under the same setting, making it a valuable contribution to the community.\n\n2. The study offers an in-depth examination of the components of DPO, supported by thorough experiments that validate the analysis, resulting in a solid and reliable evaluation.\n\n3. By combining all effective components, this work successfully formulates a new variant of DPO and demonstrates its superior performance over existing algorithms, showcasing the potential for further advancements in the field.", "weaknesses": "1. The evaluation benchmark is limited to AlpacaEval, which may not provide sufficient evidence to support the claims regarding the effectiveness of each component and the advantages of the proposed RainbowPO. A more comprehensive evaluation across multiple benchmarks would be necessary to further validate these findings.\n\n2. The advantage of introducing key components in RainbowPO lacks systematic explanation and remains at a purely empirical level. It is unclear whether the same advantages can be transferred to other tasks, highlighting the need for further investigation into the generalizability of these findings.\n\n3. The proposed method introduces significant additional complexity compared to baseline methods, which may impact its practicality and scalability. Further research is needed to determine whether the benefits of RainbowPO outweigh the increased complexity it introduces.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fhrkbdYzbs", "reviewer": "Reviewer_eopu", "summary": "The paper developed Rainbow PO as a unified framework for direct preference optimization. Motivated by the challenge in comparing various xPO methods in literature, the authors began the study by analyzing more than 10 existing xPO methods and identified seven mathematically orthogonal components for mapping these methods. Among these components, they justified four of them as effective. Based on these findings, they proposed RainbowPO to combine these components. One notable novelty is that RainbowPO adopts a mixing reference policy mechanism. The authors conducted experiments to illustrate RainbowPO's superior performance over previous DPO variants.", "questions": "Please answer the questions mentioned in ‘Weaknesses’.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper strived to provide a systematic and comprehensive study of existing preference optimization methods. The identification of the seven components leads to a structured framework for evaluating xPO methods, which is useful considering the large, and still growing number of xPO methods. \n\n2. The paper contributed clarity and insights on how different components of an xPO method affect the method’s performance. The experiment evaluation offers interesting results illustrating the value of adding components to an xPO model.", "weaknesses": "There lack transparency and explanation on the identification of the seven components. It would be helpful to include further details on the identification process. Relevant questions include: Are these components necessary and sufficient to categorize current and potential new xPO models? Have similar components been applied or mentioned in literature on xPO? Are there any potential interactions among components that would affect the performance of xPO?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qgblreGY91", "reviewer": "Reviewer_89tQ", "summary": "Direct Preference Optimization (DPO) has emerged as an effective approach for fine-tuning large language models (LLMs) to align with human preferences. Various versions of DPO have been proposed in the literature; however, an extensive comparison between these different DPO variants is still lacking.\nThis paper aims to analyze various versions of DPO to identify components that enhance performance across different applications. To this end,  seven key components have been identified  across ten DPO variants. By combining these components, a new approach that achieves state-of-the-art performance has been proposed. This approach is conceptually similar to the Rainbow framework [1], which unified various enhancements of Deep Q-Networks (DQNs) from the literature.\n\nThis paper explains seven components, namely--Length Normalization, Link Function, Home Advantage, Mixing Reference Policy, Rejection Sampling, Contextual Scaling and SFT Loss. Then a new loss is formulated combining the seven components. Finally, the experimental evaluation reveals that Length Normalization, Contextual Scaling  and Rejection Sampling contribute to improvement over the baseline DPO.\n\nThe paper could be a significant contribution in the field of Fine-tuning large language models. \nHowever, to meet the publication standards of a top-tier conference, it requires an additional round of thorough editing.\n\n[1] Hessel, Matteo, et al. \"Rainbow: Combining improvements in deep reinforcement learning.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 32. No. 1. 2018.", "questions": "- DPO techniques have been reported to lack generalizability in out-of-distribution tasks [1]. To address this limitation, learning chain-of-thought (CoT) policies has been proposed in the literature. Can the method proposed in this paper be adapted to learn a CoT policy?\n\n- Robust DPO [2] is another approach to make DPO robust to noisy preferences. Is there a reason why this variant is not considered in your work? \n\n- Have you considered approaches [3] which removes the process of learning a reward model by directly learning though supervised learning?\n\n- Apart from Win ratio (WR), have you measured BERTScore?\n\n1: Choi, Eugene, et al. \"Robust Chain of Thoughts Preference Optimization.\" Seventeenth European Workshop on Reinforcement Learning.\n\n2: Chowdhury, Sayak Ray, Anush Kini, and Nagarajan Natarajan. \"Provably Robust DPO: Aligning Language Models with Noisy Feedback.\"\n Forty-first International Conference on Machine Learning.\n\n3: Hejna, Joey, et al. \"Contrastive Preference Learning: Learning from Human Feedback without Reinforcement Learning.\" The Twelfth International Conference on Learning Representations.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Originality:** This paper performs a meta-analysis on different variants of DPO.\n\n**Quality**: The experimental evaluation comparing different components is the main strength of the paper.\n\n**Clarity**: The text is written with clarity and well organized. It is easy to follow, even for someone like me, who is not up-to-date with the recent developments in LLM finetuning techniques. However, some of the terminologies introduced require explanations (see weakness).\n\n**Significance**: Fine-tuning large language models (LLMs) for preference optimization has garnered increasing attention recently. A meta-study comparing different variants of preference optimization in LLMs would be highly valuable for both academics and practitioners. This work combines seven variants and promises to make the implementation publicly available, which would be a significant contribution to the community.", "weaknesses": "- Some acronyms, such as IPO and KTO, are never explained. Given that this paper is a meta-analysis, it’s important to define all acronyms for the reader’s convenience.\n\n- Define LC WR in the caption or in a footnote in Table 4.\n\n- It seems the term 'warm-up adjustment'  is never introduced before Table 3.\n\n- The color coding in Table 4 is counterintuitive; using red to indicate better-performing models and green for worse-performing models is disorienting for the readers.\n\n- The results in Table 4 are based on training for 3 epochs. Could these outcomes differ if the model were trained for additional epochs? Maybe you can  conduct an ablation study on the impact of number of epochs or plot training loss against number of epochs.\n\n- Is there any explanation why rejection sampling does not perform well with length normalization?\n\nI would be happy to raise my score, if the authors reply to my questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)