On a Connection Between Imitation Learning and RLHF
Authors: Teng Xiao
Yige Yuan
Mingxiao Li
Zhengyu Chen
Vasant G Honavar
Venue: ICLR 2025 Poster (2025)
Keywords: Alignment
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: This work studies the alignment of large language models with preference data from an imitation learning  perspective. We establish a close theoretical connection between reinforcement learning from human feedback RLHF and imitation learning (IL), revealing that RLHF implicitly performs imitation learning on the preference data distribution. Building on this connection, we propose DIL, a principled framework that directly optimizes the imitation learning objective. DIL provides a unified imitation learning perspective on alignment, encompassing existing alignment algorithms as special cases while naturally introducing new variants. By bridging IL and RLHF, DIL offers new insights into alignment with RLHF. Extensive experiments demonstrate that DIL outperforms existing methods on various challenging benchmarks.

Published as a conference paper at ICLR 2025
ON A CONNECTION BETWEEN IMITATION LEARNING AND RLHF
Teng Xiao Yige Yuan Mingxiao Lit, Zhengyu Chen Vasant G Honavar
Pennsylvania State University & University of Chinese Academy of Sciences
Tencent Al Lab Meituan Inc
tengxiao@psu. edu, vhonavar@pau. edu
ABSTRACT
This work studies the alignment of large language models with preference data from an imitation learning perspective. We establish a close theoretical connection be- tween reinforcement learning from human feedback (RLHF) and imitation learning (IL), revealing that RLHF implicitly performs imitation learning on the preference data distribution, Building on this connection, we propose DIL, a principled frame- work that directly optimizes the imitation learning objective. DIL provides unified imitation learning perspective on alignment, encompassing existing alignment al- gorithms as special cases while naturally introducing new variants. By bridging IL and RLHF, DIL offers new insights into alignment with RLHF. Extensive experiments demonstrate that DIL outperforms existing methods various challenging bench- marks. The code for DIL available at https: : htt//ggtttt github. cww/tengxiaol 1/DLL.
INTRODUCTION
Aligning large language models (LLMs) with human preferences is essential to ensure that the responses generated by LLMs align with human expectations (Bai et al., 2022; Ouyang et al., 2022; Stiennon et al., 2020). Recently, Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al., 2022; Christiano et al., 2017) has become a widely adopted framework for fine-tuning language models according to human preference data. This approach typically involves training reward model based on human feedback and subsequently employing reinforcement learning (RL) algorithms, such as PPO (Schulman et al., 2017), to optimize the model to maximize the reward signal.
RLHE has demonstrated impressive efficacy across a diverse range of tasks, from programming to creative writing. However, its dependence two-step reinforcement learning presents challenges, such as computational inefficiency and instability during training (Engstrom et al., 2020; Rafailov al., 2024). To mitigate these limitations, alternative one-step approaches, such as direct preference optimization (DPO) and its variants, have been proposed (Rafailov et al., 2024; Meng et al., 2024; Tajwar et al., 2024). These methods replace RLHF with supervised learning, eliminating the need for explicit reward modeling. Instead, they directly define an implicit reward based on the likelihood of preference data, resulting in significant gains in efficiency while preserving competitive performance.
While DPO theoretically aims to discover identical optimal policies as RLHF, it and its variants fundamentally adhere to the reward maximization objective and are determined by parametric models such as the Bradley-Terry (BT) model (Bradley & Terry, 1952), making them prone to overfitting (Yuan et al., 2024; Xiao et al., 2024b) and resulting in suboptimal alignment with preference data (Xu et al., 2024c; Wu et al., 2024). This raises fundamental and open research question: Can we understand and design effective preference optimization algorithms from a new perspective?
In this paper, we revisit RLHF from the perspective of imitation learning. In particular, show that RLHF is a special case of a general imitation learning problem expressed exclusively in terms of pairwise preferences. We theoretically demonstrate that alignment with RLHF closely resembles imitation learning and implicitly optimizes the same objective. We then leverage this insight to design DIL, general framework for effective alignment based density ratio reward estimation.
Our primary technical contributions are as follows: (i) We prove that RLHF for alignment is essentially an imitation learning problem and provide novel analysis that offers explicit guidance for alignemnt

---

Published as a conference paper at ICLR 2025
algorithm design. (ii) We propose DIL, simple and generalized imitation learning framework for aligmment. DIL unifies imitation learning on preference data and bridges the gap between density ratio estimation and preference alignment. (ii) Empirically, we validate the effectiveness of DIL on widely used benchmarks, demonstrating that it outperforms previous alignment methods.
2 RELATED WORK
Reinforcement Learning from Human Feedback. RLHF has emerged as an effective approach for aligning LLMs with human preferences (Christiano et al., 2017). It involves first training a reward model from human feedback through supervised learning, which is then used to optimize a policy via RL algoritmms, such as PPO (Schulman et al., 2017). RLHF has been successfully applied to wide range of tasks, including summarization (Stiennon et al., 2020), instruction following (Ouyang et al., 2022), safety improvement (Bai et al., 2022), and truthfulness enhancement (Tian et al., 2023).
Offiine Preference Optimization. Recent literature highlights the inherent complexity of RLHF, prompting the search for more efficient offline alternatives. A significant advancement in this area is DPO (Rafailov et al., 2024). Unlike RLHF, which first learns an explicit reward model and then fits the policy rewards, DPO bypasses this second approximation by directly learning policy from collected data, without the need for reward modeling. DPO implicitly optimizes the same objective as existing RLHF algorithms, but it is simpler to implement and more straightforward to train. Other offline alignment methods, such as IPO (Azar et al., 2024), KTO (Ethayarajh et al., 2024), and others (Zhao al., 2023; Xiao & Wang, 2021; Xu et al., 2024a; Meng al., 2024; Xiao et al., 2025), have also been proposed. In contrast, we rethink and design the alignment objective from novel offline imitation learning perspective. We show that RLHF can theoretically be viewed as imitation learning, which fits the chosen response distribution by minimizing the reverse KI. divergence.
Imitation Learning. Classical imitation learning (IL) methods often frame II. as inverse reinforce- ment learning (IRL) to better utilize expert demonstrations (Sammut et al., 1992; Abbeel & Ng, 2004). In the seminal work (Ho & Ermon, 2016), the authors introduce GAIL, which bypasses inner-loop reinforcement learning (RL) by establishing connection between IL and generative adversarial networks (GANs) (Goodfellow et al., 2020). GAIL and its successor, AIRL (Fu et al., 2018), have made significant strides. However, these online methods typically require substantial environmental interactions, limiting their deployment in cost-sensitive or seficty-snssiive domains. To address this issue, recent work offline II. (Garg et al., 2021) focuses on learning a reward function from offline datasets to understand and generalize the intentions underlying expert behavior. IQ-Learn (Garg al., 2021) simplifies AIRL's game-theoretic objective over policy and reward functions into an optimization over the soft Q-function, which implicitly represents both reward and policy. Recently, some works (Sun & van der Schaar, 2024; Wulfmeier et al., 2024; Xiao al., 2024a) have applied imitation learning to the alignment of large language models. Different from the above works, in this paper, we aim at building a close theoretical connection between RLHF and imitation learning. revealing that RLHF implicitly performs imitation learning on the chosen data distribution.
3 NOTATIONS AND PRELIMINARIES
Problem Setup. Let the text sequence x = [.1. .... denote the input prompt, and Yw [y1, .] and y, denote two responses, typically sampled from the same reference policy Ter(y x). The response pairs are then presented to human labelers (or an oracle) who express preferences for responses given the prompt, denoted as y > y: x, where yu and y: denote preferred and dispreferred responses, respectively. The preference distribution is typically expressed as:
and
p (y > yt x) = yw) - r (x.
(I)
where g is the sigmoid function (x) Ite-x based on the Gradley-Tery (BT) preference assump- tion (Bradley & Terry, 1952). Given preference dataset DD containing feedback (x, the goal of alignment is to learn an L.M policy T(y x) based on the preference data.
Reinforcement Learning from Human Feedback. Given the estimated reward function r(x,y), dictating the human preferences, RLHF fine-tunes policy ma by optimizing the following objective:
max E (y)x) R
yr( - 3D (o (y x) [Tee((y xx]
(2)
2

---

Published as a conference paper at ICLR 2025
where B >> 0 is an appropriate KL penalty coefficient. RLHF typically optimizes the above objective in Equation (2) using RL algorithms, such as PPO (Ouyang et al., 2022; Schulman et al., 2017).
Reward Modeling. One standard approach to reward modeling is to fit reward function T (x, y) with the BT preference model Equation (1). Specifically, the reward function rx(x, y) can be estimated by maximizing the log-likelihood preference feedback (x, Yw, Yr):
(Ø: D) = (.) [ log (ro(x, To (x. xi))
(3)
Supervised Fine-tuning (SFT). Given demonstration dataset, the objective of SFT minimizing the negative log-likelihood over the demonstration data as follows:
CSFT D) (xxx)))> [log (y xx]]
SFT is equivalent to behavior cloning (BC) (Pomerleau, 1988). a classical offline imitation learning method that minimizes the forward KL divergence between the learned policy and data policy:
min KL (Tdasaly
(
x)) -B log Te (y xx)
easy to see that the BC problem above shares the same optimal solutions as SFT in expectation.
Directed Preference Optimization. To simplify the optimization process of RLHF, DPO uses the log-like log-likelihood of the learning policy to implicitly represent the reward function:
ro y) = 3 [log ( x) loog ( x)] + , log Zo (x),
(6)
where Z(x) = Ey T x) exp(re (x. y) is the partition function. By incorporating this
reward into the BT model in Equation (1), DPO (Rafailov et al., 2024) objective enables the comparison of response pairs, facilitating the discrimination between preferred and dispreferred responses:
x)
T
LDPO (0; D) EIx.Y=.Y))D log ar (3 (3log log
-3 :3log
(7)
Tref (y
x)
Tref (y:
Energy-based Models. Energy-based based models (EBMs) (LeCun et al., 2006) define the distribution through an energy function. For y E IRP, its probability density can be expressed as follows:
po(y)
- (opp-Eo(y))/Zo(>>
(8)
= where Ey) RR + R is the energy function, mapping the data point to scalar, and Zo(y) Ey exp(-Eo(y)) the unknown normalization constant (Song & Kingma, 2021).
METHODOLOGY
4.1 RLHF IS A FORM OF IMITATION LEARNING
In this section, we connect RLHF to the imitation learning framework. We show that RLHF is a special case of imitation learning problem on the distribution chosen response with the reverse KL divergence. Specifically, we firstly define the following policy based on EBMs (Haarnoja et al.77)11)
T (y x) Treryy x)exp
where & denotes the parameter and (x) = Ey Trerly x) exp(ra((x, y)). To learn the parameter Φ. one can apply behavior cloning (Pomerleau, 1988). a classical and widely used imitation learning method, which frames the task as minimizing the KL divergence between the policy T and the expert policy Tehosen generating the chosen response y w. In other works, IL learns the parameter d such that the model distribution imitates the distribution chosen response in the preference dataset:
min KL (Tehoseen (y
x) x)] [ (y) (x).
(10)
Minimizing the above forward KL divergence with the chosen responses on preference data gives us: E Exx.,= - log The (y xxx)prr((x,,w))/22
min +
min E Bx +
(x, + log E, mirly x)exp (ro(x,))))
(1I)
3

---

Published as a conference paper at ICLR 2025
There are several options for sampling from the reference distribution ( ( x). A choice that simplifies the above expression and yields RLHF in practice is Ty x) = 20()' = yd) + 21 =
Ym). In this case, the sample-bass based approximation of the second term gives us:
min (x) (x, yw) + log (exp(r>(x, vw)) + exp(ra(xxxi)))
-E log (ro(x, To (x, (x))
(12)
One can note that the above imitation learning loss over energy energy-based based policy exactly the same as the reward loss based on BT assumption in Equation (3) in RLHF. By optimizing this loss function, we can directly obtain the optimal energy-based policy in Equation (9). Unfortunately, even we use the estimate r.. it is still expensive to estimate the partition function Z (x), making this representation difficult to use in practice and significantly higher inference cost (Rafailov et al., 2024). To address utilize the reverse knowledge distillation which distills the optimal policy in Equation (9) into a analytical policy by using the following reverse KL divergence, which allows the final policy Te require only single sample during the inference time:
this problem,
min KL ( (y x)) x))xxpr= (xterrxxxx))///////// 3)/8)/Z((x)).
(13)
where 3 is the temperature hyperparameter in distillation process. This gives the following objective function after removing multiplicative and additive constants:
(()) - -Exe [re (x,y) KT. (To(y xe]l x))
(14)
One can observe that this distillation objective exactly corresponds to the RL. objective in Equation (2).
In summary, we provide two key insights: (i) Reward learning in RLHF is equivalent to an imitation learning problem against the chosen responses, achieved by minimizing the forward KL divergence between Tchosen and Th based on the EBMs shown in Equation (12). (ii) The RL step in RLHF can be interpreted as a reverse knowledge distillation process, where the imitated policy TT. based on EBMs, is distilled into final analytical policy me by minimizing the reverse KL divergence in Equation (13), with the temperature determining the level of KL regularization. Formally, we have:
Proposition 4.1. Suppose the chosen response distribution p(y x), the EBM To x). and the model Th (y x). KL-regularized RLHF with B = 1 can be viewed as the following problem:
minKL[ Too || TT)) s.t. T arg min (T.Coohoeen TΦ),,
(15)
where n. Wehosen (y = | x) = oo x) is the equilibrium.
Thus, conducting imitation learning on the chosen response corresponds to solving a standard KI.-regularized RLHF problem. Additionally, we observe that the upper-level objective essentially optimizes a reverse KL (RKL) divergence, KL || Tchosen), given that π optimum achieved by the lower-level objective.
Wehosen, which is the
An interesting question is why SFT. which directly optimizes the forward KL (FKL) (chhose me in Equation (5). performs worse than RLHF for alignment. Theoretically, minimizing SET and RLHF should lead to the same optimal solution Tg. However, achieving this in practice requires full data coverage and infinite computations, conditions that are rarely met. Consequently, in practical settings, minimizing either KL divergence results in learned policies with distinct properties, discussed in (Murphy, 2012; Tajwar et al., 2024). Specifically, FKL KL(Tchosen Te) promotes mass-covering behavior, whereas RKL KL(T || Tehasen) encourages mode-secking behavior (Tajwar al., 2024; Nachum et al., 2016; Agarwal et al., 2019). Mass coverino encourages assigning equal probability to all responses in the dataset, leading to an overestimation of the long tail of the target distribution, while mode-seeking concentrates the probability mass on specific high-reward regions. Thus, alignment focuses on generating a certain subset of high-reward responses, which more effectively achieved by minimizing reverse KI., as theoretically shown by (Tajwar et al., 2024; Ji et al., 2024).
4.2 DIRECT IMITATION LEARNING
In the last section, we revisit RLHF from the perspective of imitation learning. Our analysis explicitly suggests that RLHF is essentially optimized to align closely with the distribution of the chosen
4

---

Published as a conference paper at ICLR 2025
Table 1:
Summary of the variants DIL with different h-functions for Bregman divergence: (D)
Eahheeen (x) [Ci (fo)] + E Frejectea [e (fa)] as a function of log ratio fo - ( x)/mioly x)).
1-Bregman Density Ratio Estimation
A-function
(fo)
(=1(f))
LSIF (Kanamori al. 2009)
1/2/2
-ef-
BCB (Hastic al., 2009)
M()
logr - + 1ylogr +1)
log(1 + +
log(1+
UKL (Nguyen al., 2010)
k(r) rlogr
-fo
responses. The sample-based approximation of EBMs in RLHF results in reward loss similar to the BT model, as shown in Equation (12). However, the BT assumption may not always hold true, discussed in (Azar et al., 2024; Munos et al., 2023; Sun & van der Schaar, 2024). Based on these insights, we propose a novel alignment method, DIL, without the BT assumption. We directly formulate the objective of imitation learning minimizing the reverse KI. divergence between mo and the unknown distribution of the chosen response Tehosen (Kostrikov et al., 2019; Fu et al., 2018):
min CDm. (0) KT.
(y x)] Fchosee (y xx) (y)(x))))
log (e xx) (y xx)]] (16)
where we minimize RKL divergence, rather than FKL divergence as in SFT, as shown Equation (5).
However, mode-seeking with reverse KI. divergence is generally challenging. Directly optimizing Equation (16) does not effectively leverage chosen preference data, particularly because the data policy Thhosen is unknown. In the RL literature, these challenges have been addressed through adversarial training (Ho & Ermon, 2016; Fu et al., 20. However, these methods require learning reward function using complex and unstable adversarial training. which is impractical for large models. In this paper, we propose a straightforward alternative that leverages preference data without learning reward function via adversarial training. We reformulate the DIL objective as follows:
cchosen (y x)
To x)
max E ( 0
log Trol (y x)
log
Trof (y x)
E (x)(y xx) log r(x, y) - KL ( x) (re x)),
(17)
where r(x, y) Rehoons can be viewed as an auxiliary reward function. Equations (16) and (17)
[x(((
are equivalent by adding and subtracting the same term of log (y x) in the expectation.
Interestingly, we find that even when only prefer- ence data is available, this objective takes form similar to the RLHF objective in Equation (2). The primary difference lies the reward being the estimated log density ratio, which is often not readily accessible in real-world applications. Optimizing this objective, which involves the density ratio y), J. not straightforward, In the next section, we demonstrate how to effi- ciently optimize it by effectively utilizing offline human preference data.
A(6
LSIF
BCE
UKL
Figure 1: The illustration of different losses (LSIF, BCE, and UKL), shown in Table 1.
4.3 DENSITY RATIO REWARD ESTIMATION
Before delving into the problem in Equation (17), we first describe how calculate the auxiliary reward function in terms of the density ratio. In the tabular setting, we can directly compute Tref(y x) and Rchosen (y x). However, in a high-dimensional language domain, estimating the densities separately and then calculating their ratio hardly works well due to eттоr accumulation. In this paper, we choose directly estimate the density ratio Tehosen (y x) Tret (y x) based on the Bregman divergence (Sugiyama et al., 2012). Suppose r" (x(xy) = Wchosen (y /rat (y x) is the target density ratio to be estimated with a parameterized discriminator rg- Then, we have:
min Dj (r" [[r = E, (y x) (r (x)r Φ
E, Tro (y xx) (x)((((((x)))) (r)((x,))) - ah (h (x)) -roxx,)))),, (18)
where B is the data-level Bregman divergence. For a twice continuously differentiable convex function i with bounded derivative Oh, this divergence quantifies the discrepancy between two
5

---

Published as a conference paper at ICLR 2025
density-ratios. Subtracting constant Ey (y xx)((r yx,,),, we obtain (up to constant):
Σy (r x) (s (x, (x) -0--hhun (T)((x)))) Σ, " x) (xx (19)
A few non-exhaustive examples of the Bregman divergence are Least-Squared Importance Fitting (LSIF) (Kanamori et al., 2009), Binary Cross Entropy (BCE) (Hastie et al., 2009), and the unbounded Kullback-Leti (UKL) (Nguyen et al., 2010). For example, LSIF defines hLSIF = (r - 1)2/2, which results in the following instance of Bregman divergence on the density ratio:
min DALSI (r" (1 E, Te (y (x) Tchosen (y x)))(x.y)
(20)
In this case, cumple-based approximation of Equation (20) leads to the following loss function:
[(Ø: Ex 1 (x, vi) - re (x, yw.)
(21)
Here, we use the set of rejected responses yt ~ Tref x) to approximate the expectations under Tref (y x). It is acceptable to the set of rejected responses yI from the preference dataset D to approximate the expectations. We can even make use of both chosen responses and rejected responses to approximate these expectations. However, since our goal is to decrease the likelihood of rejected responses, we choose to use the rejected responses to approximate the expectations, and we find it empirically works well. Intuitively, the first term pushes the model to decrease the density ratio of the rejected response, while the second term increases the density ratio of the chosen response. In addition, this direct estimation approach with Bregman divergence suggests divergence family for density ratio estimation as shown in Table I; see Appendix A for further discussion of other h functions such as BCE (Hastic et al., 2009) and UKL (Nguyen et al., 2010). We also empirically analyze the effect of using different objectives with h functions in Section 6.3.
4.4 OPTIMIZATION
So far, we have observed that the RI.-style objective in Equation (17), combined with density ratio estimation in Equation (21), can effectively leverage the preference dataset for imitation learning. However, this two-step process is complex and unstable, requiring first the fitting of reward model to estimate the density ratio and then fine-tuning the language model policy using the RL-style objective in Equation (17). To address these challenges, introduce simpler approach that directly optimizes the imitation learning objective, bypassing the need for RL training and density ratio estimation. The core idea lies in a specialized parameterization of the density ratio reward, which allows for the direct extraction of the optimal policy, eliminating the need for an RL loop (Rafailov et al., 2024). Notably, the optimal policy in Equation (17) has closed-form solution, shown by (Rafailov et al., 2024);
x (y x) = Z(x) Fref (y x) exp (log r (x.
(22)
where Z(x) - Z(x)=Ez Ey Tree (y/x)) exp (logr" (x, y))=)) Ey Tehosen (y x) 1, meaning that the optimal x" (y(((x) is forced to be self-normalized! This characteristic, determined by the reward definition in Equation (17). is super beneficial as it allows our imitation learning to theoretically generalize to broader class of loss functions beyond the pairwise BT preference model used in DPO. Taking the logarithm of both sides of Equation (22) and then with some algebra, we obtain the following:
π (y x) Trof (y x)
log
logr (x,
(33)
where r" (x(,y) y) is the density ratio estimated by Equation (21) the preference dataset. Since the optimal density ratio is now represented in terms of the optimal policy, rather than the discriminator model, can explicitly derive the following maximum likelihood objective for a parameterized policy over the preference dataset (Rafailov et al., 2024). Analogous to the approach used for density ratio estimation and utilizing a change of variables, we can formalize our DIL objective as follows:
(p D) E(x.n To(y x)
+ 1 To (y: xx )²).
(24)
(ye x) Trer x)

---

Published as a conference paper at ICLR 2025
where we directly fit an implicit density ratio in Equation (21) using alternative parameterization in Equation (23). Interestingly, there are no hyperparameters in our loss, yet achieves promising performance, demonstrated in our experiments. Since our procedure is equivalent to fitting a reparameterized density ratio estimation model, it theoretically conducts imitation learning by minimizing RKI. divergence against the unknown distribution of the chosen response. Table I shows a family of objectives that meet the definition of Bregman divergence.
4.5 DISCUSSION: DPO IS A SPECIAL CASE OF DIL
In this section, we show that DPD can be also viewed as a special case of our framework by using contrastive predictive coding (CPC) (also known InfoNCE) (Oord et al., 2018) for density ratio estimation. Given the prompt distribution p(x) and the conditional distribution of the chosen response Tchosen y x). sample x p(x). Tchosen (y x)), and yı (T x). CPC optimizes:
exp(f<(x /w)/
Lopo(d: D) - -E(x. log exp(fØ yw/ + exp(fe & (x v1)BB)
(25)
where fo : X' x + R is a parametric critic function. The optimal critic for this CPC with one negative sample satisfies the following (Zheng et al., 2024; Ma & Collins, 2018; Oord al., 2018):
Tchosen (y x) Toot (y x)c(x)
f (xx log
logr" (x, log c(x),
(26)
where c(x) is function (Oord al., 2018; Zheng et al., 2024), that depends on x but not y. Thus, CPC also estimates the density ratio reward in IL objective in Equation (17). Similar to Section 4.4, by using the closed-form optimal policy in Equation (22) and using a change of variables, we have:
DI. (0: D) = -E(x.y...i)ND log (B
T x)
Blog
Ty x) Tee (y: xx) }
(27)
Teet (Y
x)
which exactly the same objective as the well-known DPO. Thus, our framework enables us to reinterpret DPO. Specifically, we demonstrate that DPO also falls under the imitation learning objective in Equation (16) and essentially employs the CPC method for density ratio reward estimation.
EXPERIMENTS
Datasets. We evaluate DIL on widely used datasets: the UltraFeedback Binarized dataset (Cui et al., 2023; Tunstall et al., 2023), the Reddit TL:DR summarization dataset (Völske et al., 2017), and the Anthropic-HH dataset (Bai et al., 2022). The details of these datasets are provided in Appendix B.
Tasks and Evaluation. Following previous work (Rafailov et al., 2024; Tunstall et al., 2023), we evaluate methods fine-tuned on the UltraFeedback Binarized dataset across tasks on the Open LLM Leaderboard (Gao al., 2023). The Anthropic HH dataset is used for dialogue generation to produce helpful and harmless responses (Rafailov et al., 2024). For summarization, we use the Reddit TL;DR dataset. For these tasks, we use GPT-4 for zero-shot pairwise evaluation (see prompts in Appendix B.2). The task and evaluation details are also provided in Appendix B.2.
Models. For summarization and dialogue generation tasks, we use Pythia- arthia-2.. 2.2.2.. 8b (Biderman et al., 2023) as our base model, with the model after SFT serving as a reference model, following (Rafailov et al., 2024). For fine-tuni ane-tuning on the UltraFeedback Binarized dataset, we use Mistral-77 MasstalllB-Baas 7BaBase (Tun- stall et al., 2023) and Llama3-8b-SFT used in (Meng et al., 2024) as our base models.
Baselines and Implementation. We compare DIL with the following state-of-the-art baselines: DPO (Rafailov et al., 2024), f-DPO (Wang et al., 2024a), IPO (Azar et al., 2024), SLiC (Zhao et al., 2023), CPO (Xu et al., 2024b), and SimPO (Meng et al., 2024). We thoroughly tuned the hyperparameters for each baseline and reported the best performance. The details of the baselines and the hyperparameter search space can be found in Appendix B.3. The density ratio in Section 4.3 is estimated through optimization toward the Bregman divergence. A variety of functions meet the requirements of h, but in all experiments, we choose the widely used LSIF as the default objective. The effect of using different density ratio estimation objectives is empirically analyzed in Section 6.3.
7

---

Published as a conference paper at ICLR 2025
Table 2:
Evaluation results on various tasks from the Huggingface Open Leaderboard v1 and v2. The best and second best performance under each dataset are marked with boldface and underline.
Model ()) Benchmark
MMLU-PRO
BBH
MUSR
MATH
GSM8K
ARC
SFT
27.58
41.26
41.93
2.34
28.13
58.28
Mistral 7B- Buse
DPO (Rafailov et al., 2024)
26.73
43.27
43.65
1.36
21.76
61.26
SLiC (Zhao et al., 2023)
26.52
42.33
33.74
1.38
33.74
55.38
E-DPO (Wang
2024a)
25.96
42.39
37.82
1.27
23.18
62.01
IPO (Azar et al., 2024)
25.87
40,59
42.15
1.25
27.14
60.84
KTO (Ethayarajh ct al., 2024)
27.35
42.37
43.17
2.34
36.58
62.39
CPO (Xu et al., 2024b)
27.04
42.05
42.15
2.15
33.06
$7.00
SimPO (Meng al., 2024)
27.13
42.94
39.68
2.49
22.21
62.63
DIL w/ LSIF
27.44
43.59
44.05
2.95
32.19
63.31
SFT
31,00
46.16
41.27
3.70
46.32
60.15
LLama Buse
DPO (Rafailov al., 2024)
31.58
47.80
40.48
4.53
38.67
64.42
SLiC (Zhao et al., 2023)
31.11
46.53
40.55
3.92
48.82
61.43
f-DPO (Wang al., 2024a)
30.85
47.55
40.39
4.37
39.55
62.85
IPO (Azar al., 2024)
30.18
46.78
39.58
22.67
62.88
KTO (Ethayarajh et al., 2024)
31.16
47.92
40.24
4.13
38.99
63.
CPO (Xu et al., 2024b)
30.95
47.17
41.59
4.25
46.93
61.69
SimPO (Meng et al., 2024)
31.61
48.38
40.08
4.23
31.54
65.19
DIL w/ LSIF
32.22
48.78
42.75
4.68
48.98
65.37
Table 3: Win rates computed by GPT-4 against the SFT generated response and the chosen responses on the TL:DR summarization and Anthropic-HH datasets on Pythia-2.8b The best and second best performance under each dataset are marked with boldface and underline, respectively.
Dataset (-+)
TL,DR Summarization
Authropic-HH
Method (4) Metric (-)
vs SFT
vs Chosen
Average
vs SFT
vs Chosen
Average
DPO (Rafailov et al., 2024)
71.22
57.58
64.40
69.32
59.35
64.34
SLiC (Zhao et al., 2023)
68.61
55.72
62.17
65.52
57.71
61.62
f-DPO (Wang al., 2024a)
66. 19
51.37
58.78
60.21
52.38
$6.30
IPO (Azar al., 2024)
77.
56.51
64.34
63.19
55.12
$9.16
CPO (Xu al., 2024h)
73.13
58.89
66.01
72.30
63.39
67.86
SimPO (Meng et al., 2024)
69.71
54.38
62.05
67.85
57.51
62.68
DIL w/ LSIF
75.47
60.25
67.86
73.32
65.02
69.17
EXPERIMENTAL RESULTS
6.1 PERFORMANCE COMPARISON ON BENCHMARKS
In this section, shown in Ta-
ble 2, we compare the perfor- mance of DIL against other align- ment methods on Open LLM Leaderboard. Our results show that DIL exhibits remarkable ef- fectiveness in improving perfor- mance. Overall, DIL consis- tently outperforms state-of-the- art SimPO and DPO in various benchmarks. For instance, on LLama3, the improvements 7.5% over SimPO. Notably, we observe DPO and SimPO hurt the overall performance reasoning- heavy tasks such as GSM8K. This indicates that both SimPO and DPO might not be suitable to improve reasoning abilities, which is consistent with findings in concurrent works (Pal et al., 2024; Meng al., 2024). In contrast, DIL shows clear improvements in both the Mistral and I.I LLLama3
Table 4: Ablation study on h-function of Bregman divergence: We
observe that these variants of DIL can further bring improvements.
Model (1) Benchmark (-1)
MUSR
MATH
GSM8K
43.59
44.05
2.95
32.19
Mistral-7B Base
DIL
UKL
43.92
45.11
2.04
30.71
DIL BCE
45.13
43.92
2.79
33.13
DIL LSIF
48.78
42.75
4.68
48.98
LLuma3-8B
Base
DIL
UKL
49.71
43.01
4.98
50.95
DIL w/
BCE
48.96
47.35
5.06
49,36
notable on the Math benchmarks, with relative gains exceeding
most
8

---

Published as a conference paper at ICLR 2025
Chosen
Rejected
Margins
-0.50
1.2
LIF
Boellbood
-0.75
-1.0
1.0
UK
-1.00
0.8
SimPo
DEL
0.6
E270
DEL UKL
DTL: DCE
-2.0
DIC. BCB
0.4
-2.00
-1.75
SinPFO
DPP
-2.5
DPO
Sin20
0.2
Steps
Steps
Seeps
Figure 2: The training dynamics of DIL variants, DPO and SimPO on Mistral show that DIL exhibits the smallest decline in chosen likelihoods, while still increasing the likelihood margins between rejected and chosen responses, compared to SimPO and DPO. In contrast, SimPO and DPO progres- sively focuses on unlearning the chosen responses, leading to poor performance on reasoning tasks.
models. These findings underscore the effectiveness of DIL. These improvements can be attributed to avoiding the BT assumption and preventing the likelihood decrease of chosen responses.
6.2 PERFORMANCE COMPARISON WITH HUMAN PREFERENCES
We also explore learning from real human preferences, focusing summarization and dialogue generation tasks. Specifically, we utilize the Reddit TL:DR dataset for summarization and the Anthropic-HH HH dataset for dialogue generation. We employ Pythia-2.8B (Biderman al., 2023) the base model and fine-tuned it on the chosen completions to train a reference model, ensuring that the completions remained within the model's distribution. Table 3 presents the GPT-4 evaluation results, indicating that DIL outperforms baselines when compared to both the SFT and the chosen responses. Notably, DIL aligns better with human preferences than baselines, achieving win rate of least 60% against the chosen responses. This highlights the strong potential of DIL for aligning with human preferences. Furthermore, GPT-4 consistently favored DIL over both baselines and chosen responses, demonstrating improvements of DIL over baselines in both helpfulness and harmlessness.
6.3 FURTHER ANALYSTS
Generalization other objectives. As mentioned in Section 4.3, our approach to conducting imitation learning from preference data generalizes in a straightforward manner to other density ratio estimations, including UKL and BCE. Table shows the comparison on UltraFeedback. We can observe that different variants of the h-function can lead to general improvements across various benchmarks. Specifically, UKL performs best on BBH, achieving the highest scores on both the Mistral and LLama3 models. BCE achieves a significant improvement on MUSR, with notable 7.27% increase. These results indicate that appropriate variant can further enhance our performance.
Training Dynamics. We also analyze the likelihood patterns during the training process of DIL. Figure illustrates the likelihood trends of SimPO and DIL on UltraFeedback. We observe that the likelihood of rejected responses continues decrease, while the margin between chosen and rejected responses steadily increases. However, for DPO and SimPO, the likelihood of chosen responses drops below zero and continues decline. These results validate our motivation and demonstrate the effectiveness of DIL in preventing the likelihood of chosen responses from decreasing. This also explains why DIL generally enhances downstream task performance, particularly reasoning-hee heavy tasks such as math, as shown in Table 2.
7 CONCLUSION
We consider the problem of aligning large language models with preference data. We provide a novel perspective imitation learning for the alignment problem, and demonstrate RLHF/DPO essentially conduct imitation learning on the distribution of chosen response with reverse KL divergence. Building upon this connection, objective based on Bregman divergence and can more effectively preserve reasoning abilities than baselines while aligning with human preferences. Empirical result shows that DIL establishes superior performance on a comprehensive set of benchmarks and different families of language models. We hope that our work will inspire future research preference alignment with imitation learning.
propose DIL, which directly optimizes the imitation learning

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
The work of Vasant G Honavar and Teng Xiao was supported in part by grants from the National Science Foundation (2226025, 2225824), the National Center for Advancing Translational Sciences, and the National Institutes of Health (ULI TR002014).
REFERENCES
Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, pp. 2004.
Rishabh Agarwal, Chen Liang, Dale Schuurmans, and Mohammad Norouzi. Learning to generalize from sparse and underspecified rewards. In International conference on machine learning, pp.
130-140. PMLR, 2019.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. International Conference on Artificial Intelligence and Statistics, pp-
4447-4455. PMLR, 2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05863 2022.
Edward Beeching, Clémentine Fourrier, Nathan Habib. Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open I1m leaderboard (2023-2024). 2023.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430. PMLR, 2023.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, pp. 324 345, 1952.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: (rX3.0:800.05557 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 14168, 2021.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arX1v:2310.0137,, 2023.
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep policy gradients: A case study on ppo and trpo. In International Conference on Learning Representations, 2020.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024.
Clémentine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. Open 11m leaderboard v2. 2024.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse reinforce- ment learning. In International Conference Learning Representations, 2018.
10

---

Published as a conference paper at ICLR 2025
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac' h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. framework for few-shot language model evaluation, 12 2023.
Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. Iq-learn: Inverse soft-q learning for imitation. Advances in Neural Information Processing Systems, 34: 0028-4039, 2021.
Ian Goodfellow, Jean Pouget-Abadic, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, pp. 139-144, 2020.
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International conference on machine learning, pp. 1352-1361,
2017.
Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd Edition. Springer Series in Statisties. Springer, 2009.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardr. Measuring mathematical problem solving with the math dataset. arXiv, 2021.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
Haozhe Ji, Cheng Lu, Yilin Niu, Pei Ke, Hongning Wang, Jun Zhu Jie Tang, and Minlie Huang. Towards efficient and exact optimization of language model alignment. arXiv preprint arXiv:2402.00856 2024.
Takafumi Kanamori, Shohei Hido, and Masashi Sugiyama. A least-squares approach direct importance estimation. J. Mach. Learn. Res., pp. 1391-1445, 2009.
Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980. 2014.
Ilya Kostrikov, Ofir Nachum, and Jonathan Tompson. Imitation learning via off-policy distribution matching. In International Conference on Learning Representations, 2019.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, Fujic Huang, et al. A tutorial on energy-based learning. Predicting structured data, 2006.
Zhuang Ma and Michael Collins. Noise contrastive estimation and negative sampling for conditional models: Consistency and statistical efficiency. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3698-3707, 2018
Yu Meng, Mengzhou Xia, and Danqi Chen, Simpo: Simple preference optimization with reference- free reward. arXiv preprint 4rXiv:2405.14734 2024.
Rémi Munos, Michal Valko, Daniele Calandricllo, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Andrea Michi, al. Nash learning from human feedback. arXiv preprint arXiv:23/2.008880 2023.
Kevin P Murphy. Machine learning: probabilistic perspective. MIT press, 2012.
Ofir Nachum, Mohammad Norouzi, and Dale Schuurmans. Improving policy gradient by exploring under-appreciated rewards. arXiv preprint arXiv:1611.093211 2016.
XuanLong Nguyen, Martin J. Wainwright, and Michael I. Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Trans. Inf. Theory, 56(11)55447-5611 2010.
11

---

Published as a conference paper at ICLR 2025
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, pp. 27730- 27744, 2022.
Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with dpo-positive. arXiv preprint orXiv:2402.13228 2024.
Dean A Pomerleau. Alvinn:
An autonomous land vehicle in a neural network. Advances in neural information processing systems, 1, 1988.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
Claude Sammut, Scott Hurst. Dana Kedzier, and Donald Michie. Learning fly. In Machine Learning Proceedings 1992, pp. 385-393, 1992.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: (eXX::1707.063477 2017.
Yang Song and Diederik P Kingma. How to train your energy-based models. arXiv preprint arXiv:2101.03228, 2021.
Zayne Sprague, Xi Ye, Kaj Bostrom, Swarat Chaudhuri, and Greg Durrett. Musr: Testing the limits of thain-of-thought with multistep soft reasoning. arXiv preprint arXiv:2310. 16049, 2024.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedhack. Advances in Neural Information Processing Systems, 00:3008-002,, 2020.
Masashi Sugiyama, Taiji Suzuki, Shinichi Nakajima, Hisashi Kashima, Paul Von Bünau, and Motoaki Kawanabe. Direct importance estimation for covariate shift adaptation. Annals of the Institute of Statistical Mathematics, 60:699-746, 2008.
Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Density-ratio matching under the bregman divergence: a unified framework of density-ratio estimation. Annals of the Institute of Statistical Mathematics, 64:1009-1044, 2012.
Hao Sun and Mihaela van der Schaar. Inverse-rlinmment Inverse reinforcement learning from demonstrations for Ilm aligmment. arXiv preprint arXiv:2405.15624, 2024.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challenging big-bench tasks and whether chain-of- thought can solve them. arXiv, 2022.
Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Stefano Ermon, Chelsea Finn, and Aviral Kumar. Preference fine-tuning of lims should leverage suboptimal, on-policy data. arXiv preprint (tfi:2404.14367. 2024.
Katherine Tian, Eric Mitchell, Huaxiu Yao. Christopher D Manning, and Chelsea Finn. Fine-tuning language models for factuality. arXiv preprint arXiv:2311.0840) 2023.
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of Im alignment. arXiv preprint arXiv:2310.16944 2023.
Michael Völske, Martin Potthast, Shahbaz Syed, and Benno Stein. Tl: dr. Mining reddit to learn automatic summarization. In Proceedings of the Workshop on New Frontiers in Summarization, pp. 69-63, 2017.
12

---

Published as a conference paper at ICLR 2025
Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. Beyond reverse KL: generaliz- ing direct preference optimization with diverse divergence constraints. In The Twelfth International Conference Learning Representations, 2024a.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: more robust and challenging multi-task language understanding benchmark. arXiv, 2024b.
Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment. arXiv preprint arXiv:2405.0m 00675, 2024.
Markus Wulfmeier, Michael Bloesch, Nino Vieillard, Arun Ahuja, Jorg Bornschein, Sandy Huang, Artem Sokolov, Matt Barnes, Guillaume Desjardins, Alex Bewley, al. Imitating language via scalable inverse reinforcement learning. arXiv preprint arXiv:2409.01360 2024.
Teng Xiao and Donglin Wang. A general offline reinforcement learning framework for interactive recommendation. Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
Teng Xiao, Mingxiao Li, Yige Yuan, Huaisheng Zhu, Chao Cui, and Vasant G Honavar. How to leverage demonstration data in alignment for large language model? self-imitation learning perspective. arXiv preprint arXiv:2440..00000 10093, 2024a.
Teng Xiao, Yige Yuan, Huaisheng Zhu, Mingxiao Li, and Vasant G. Honavar. Cal-dpo: Calibrated direct preference optimization for language model alignment. Advances in Neural Information Processing Systems, 37, 2024b.
Teng Xiao, Yige Yuan, Zhengyu Chen, Mingxiao Li, Shangsong Liang, Zhaochun Ren, and Vasant G Honavar. Simper: minimalist approach to preference alignment without hyperparameters. arXiv preprint arXiv:2502.00888 2025.
Haoran Xu, Amr Sharaf, Yummo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of Ilm performance in machine translation. arXiv preprint (rXiv:2401.084/1 2024a.
Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of llm performance machine translation. In Forty-first International Conference on Machine Learning. 2024b.
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for I1m alignment? a comprehensive study. arXiv preprint arXiv:2404. 10719, 2024c.
Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, et al. Advancing Ilm reasoning generalists with preference trees. arXiv preprint Stt::22044020775 2024.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.. 10425, 2023.
Chongyi Zheng, Ruslan Salakhutdinov, and Benjamin Eysenbach. Contrastive difference predictive coding. In The Twelfth International Conference on Learning Representations, 2024.
13

---

Published as a conference paper at ICLR 2025
A DETAILS OF METHODS FOR DENSITY RATIO ESTIMATION
We overview examples of density ratio estimation methods under Bregman Divergence framework.
Least Squares Importance Fitting (LSIF). LSIF (Kanamori et al., 2009) minimizes the squared error between density ratio model and the true density ratio yM defined follows:
DiLsIr (r" ((r)) = Em..t ((T yx. - (x)
= E [(r" (x, (x, 2 2 (x) + e (x,((x y))
(28)
where the first term in the above equation is constant w.r.t Φb This empirical risk minimization equal minimizing the empirical BD defined in Equation (18) with h(r) (r 1)2/2.
KL Importance Estimation Procedure (KLIEP). KLIEP is derived from the unnormalized Kull- back-Leibles (UKL) divergence objective (Sugiyama et al., 2008; Nguyen et al., 2010), which uses h(r) r log(r) - r. Ignoring terms irrelevant to the optimization, we obtain (up to constant):
D DEKLIPF (r" Eer [r(x E Te
[og (r(x,)))
(29)
KLIEP is also known solving a Lagrangian of the constrained problem with further imposing constraint that the ratio model r(x, y) is non-negative and normalized as follows:
max E Eehwomn [log (r(x.y))
(30)
s.t. Emter [r(x = 1 and y) 2 0 for all (x,y).
(31)
Binary Cross Entropy. By using h(r) log(r) - (1 + (1 + r), we obtain the following Bregman Divergence called the Binary Cross Entropy (BCE) divergence:
DRECE (r" [[r)) = -Ex,et log
(1=x
(T+rix.v))
1
E aahcven
log
(_xxx))) y)
) 1 (+
r(x,
This Bregman divergence derived from a formulation of logistic regression (Sugiyama et al., 2012).
B EXPERIMENTAL DETAILS
B.1 THE DETAILS OF DATASETS
UltraFeedback Binarized (Cui et al., 2023; Tunstall et al., 2023): This dataset' consists of 64k prompts, each paired with four completions generated by various open-source and proprietary models. GPT4 assigns scores to these completions based on helpfulness, honesty, and other criteria. Binary preferences then constructed by selecting the completion with the highest score as the chosen response, while one of the remaining three completions is randomly selected as the rejected response.
Anthropic-HH (Bai al., 2022): The Anthropic Helpful and Harmless dialogue dataset² comprises 170k dialogues between humans and LLM assistants, designed for evaluating single-turn dialogue tasks. Each dialogue consists of human query and model responses rated for helpfulness and harmlessness. Following DPO (Rafailov et al., 2024), the chosen responses from this dataset were used during the supervised me-tuning (SFT) phase.
Reddit TL;DR Summarization (Völske et al., 2017): This dataset comprises Reddit forum posts curated for summarization tasks with associated preference labels. Following prior work (Stiennon et al., 2020), we use filtered version of this dataset to train the SFT policy and leverage its preference labels during the subsequent alignment phase.
http:://wgggnattt...eeeeeeess/er incrpe://huggingface.cc httppp//waagggggaeccccc/aaeeeeep
binarized
/ htt.:///gg....o/aaaaaaaaa/aa///a
feedback
14

---

Published as a conference paper at ICLR 2025
Table 5: The hyperparameter search space for the baselines.
Method
Method
DPO/--DP
B E [0.01, 0.1]
IPO
E [0.01, 0.5,
x= 1.0
À E [0.1, 1.0, 10.0]
CPO
SLiC
B E (0.01,0.05,4 0.1]
8 E (0.1, 2.0
= Au 1.0
E [2.0. 2.5]
KTO
SimPO
B E [0.01, 0.1]
E [0.3, 0.5, 1.0, 1.2, 1.4, 1.6
B.2 THE DETAILS OF TASKS AND EVALUATION
This section introduces the benchmark for model evaluation. The model fine-tuned on the UltraFeed- back Binarized dataset is evaluated following previous works (Rafailov al., 2024; Tunstall et al., 2023) using the HuggingFace Open LLM Leaderboard v1 and v2 (Beeching et al., 2023; Fourrier al., 2024). 'Tee evaluation includes MMUL--PR,, BBH, MUSR, MATH, GSM8K, and ARC. Models fine-tuned on the Anthropic-HH dataset follow the evaluation protocol outlined by (Rafailov al., 2024), leveraging GPT-4 for zero-shot pairwise evaluation.
MMUL-PRO (Wang et al., 2024b): Short hand for Massive Multitask Language Understanding Professional. This dataset builds on the MMLU by incorporating more complex multiple-choice questions and undergoing rigorous expert review to enhance quality, difficulity and reduce data biases.
BBH (Suzgun al., 2022): Short hand for Big Bench Hard. This benchmark includes 23 tasks selected from BigBench testing capabilities in arithmetic, comprehension, and general knowledge.
MUSR (Sprague al., 2024): Short hand for Multistep Soft Reasoning. This benchmark contains complex scenarios to assesses capacity to integrate information and reason across long contexts.
MATH (Hendrycks al., 2021): This collection comprises math problems for high-school competi- tions, consistently presented using LaTeX and Asymptote to ensure clear and precise formatting.
GSM8K (5-shot) (Cobbe et al., 2021): This benchmark consists of grade school math problems to test the model's capability to navigate and solve complex, multi-step mathematical challenges.
ARC (25-shot) (Clark et al., 2018): Short hand for A12 Reasoning Challenge. This science-focused benchmark includes questions from a grade-school curriculum to test factual and logical reasoning.
GPT-4 Evaluation (Rafailov al., 2024): The safety models trained on Anthropic HH is assessed using the Anthropic HH test set, with preferred responses serving as benchmarks. GPT-4's evaluations are aligned with human judgments, ensuring reliable safety assessments. The model version used for these evaluations is 00014, with specific prompts detailed in Table 6.
B.3 IMPLEMENTATION DETAILS
Training For the general hyperparameters, we closely followed the configurations used in SimPO. Specifically, during the SFT stage, applied a learning rate of 2 x 10 For both the SFT and preference optimization stages, we used a batch size of 128, a maximum sequence length of 2048, and implemented cosine learning
schedule with 10% warmup steps for single epoch, utilizing the Adam optimizer (Kingma, 2014). These settings were maintained consistently across all experiments to ensure uniformity and comparability. For method-specific hyperparameters, we also adhered to the search strategy outlined in SimPO. For each baseline method, which had its own unique set of hyperparameters, the search strategy is detailed in Table 5. Learning rates for each method were individually searched within the range of [3e - 5e 7, 6e - 7, le 6]. For our methods, we also normalize the likelihood of the response, calculated as the average log probability of all tokens in response using the policy model, to avoid the length bias as in SimPO. All training experiments
httpp////wuugggggeeeee/////////a haa.aceeeepppnniinnnneaaaeerrdrr hwtp:/oodddo/oooooeeeeeeeeeeeeee 11m aat_a__mm__eeeerreeeeee leaderboard http://pppppmmmmmeeeeeeeeddd/deo 11m, leaderboard
15

---

Published as a conference paper at ICLR 2025
described in this paper were conducted using either four NVIDIA A100 80GB GPUs with 128 batchsize, utilizing the codebase from the alignment-hande haad-hokk repository."
Evaluation For the evaluation on the Open LLM Leaderboard VI, use 5-shot setting for GSM8K, and 25-shot for ARC. For the evaluation on Open LLM Leaderboard V2, all tasks were performed by the default setting of the leaderboard.
For the following dialogue history chatbot, which response more helpful and harmless?
Dialogue history:
(dialogue history)
Response 1:
(Response 1}
Response (Response 2
FIRST provide one-sentence comparison of the two responses and explain which you feel is helpful and harmless. SECOND, indicate which response is more helpful and harmless. Your response should use the format:
line, atate only "1" "2"
Comparison: <one-sentence comparison and explanation>
More helpful: <"1" "2">
Table 6: GPT-4 prompt (Rafailov et al., 2024) for the safety evaluation on the Anthropic-Hl HH dataset.
C FUTURE WORK
This paper presents several exciting directions for future work. First, we aim to develop deeper theoretical understanding of which density-ratio estimation techniques are most effective for align- ment. Our current analysis is limited the offline setting and does not account for on-policy learning, where the policy interacts with the reward model during training. Exploring DIL in an on-policy learning scenario would be particularly interesting. Moreover, the relationship between DIL and DPO structurally resembles the connection between Bregman divergence and contrastive predictive coding, suggesting that further exploration of this link could be fruitful. While this paper primarily focuses on three density-ratio estimation loss functions, investigating DIL with additional loss functions would also be a compelling direction for future research.
handbook
16


Reviews:
[{"review_id": "JjkapYWinf", "reviewer": "Reviewer_JHUL", "summary": "The paper makes a connection between various approaches for RLHF of large language models and imitation learning. In particular the authors re-derive a well known connection between probabilistic inference and reinforcement learning which associates the reward function with the energy of a Boltzmann distribution (see e.g. [1] for a good review of all the related methods and derivations) for the special case of RLHF.\nFrom this perspective classical reward model learning can be derived as matching the energy to the generated responses with highest reward. Based on this the authors then derive a surrogate objective (DIL) that is closely related to DPO and other RLHF algorithms that exist, but which makes less assumptions on the form of the reward model. They show empirical evaluations on language modeling which match/gives slight improvement over DPO.\n\n[1] Levine, Sergey. \"Reinforcement learning and control as probabilistic inference: Tutorial and review.\" arXiv preprint arXiv:1805.00909 (2018).", "questions": "A discussion and answers regarding the weaknesses listed above would be appreciated. And if the authors can provide some more rationale and clean-up the derivations the score could be improved.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The connection between RLHF and imitation learning approaches is highly relevant to the community and the first part of the paper (background and initial derivation up to Eq.12-14) is well presented and leaves the reader with a condensed and improved understanding of how different existing algorithms relate (although perhaps more references to the literature could help, see weaknesses below).\n- Any improvement over DPO (which is perhaps the predominant algorithm at least for offline RLHF from fixed datasets) is relevant to the community.\n- The benchmarks used are open and relevant and at reasonable scale (i.e. 7B models)", "weaknesses": "- My first main gripe with the paper is that the idea that RLHF in it's entirety is performing imitation learning seems to stand on shaky foundations. A lot of leaps from one objective to another are required to arrive at this conclusion and a lot of the nuances in the differences between different objectives get lost along the way (that are already well discussed in the literature see e.g. Sergey Levine's review and also existing literature on offline RL as probabilistic inference). For example, the title says \"RLHF secretly performs imitation learning\" then up to Eq. 12 this thread is followed closely, and I find the connection that is made between reward model learning and the imitation learning perspective insightful, however directly after the authors make a leap to knowledge distillation / minimizing the reverse KL, which then attains the actual RL objective. This objective then is no longer directly related to learning from a dataset of reference or \"chosen\" examples (as would be the case in imitation learning) but instead can be understood as imitating an optimal policy (and not any policy that generated the dataset) on the state distribution induced by the currently learned policy (see also [3]). It thus really is RL (and not just imitation learning) and has to \"deal\" with all the problems RL comes with, i.e. exploration of the energy landscape of the optimal policy is required, premature convergence could be an issue etc.. The fact that the energy itself is given by a reward model that comes from matching chosen examples on a pre-collected dataset has no bearing on this. This is easy to see as depending on the temperature (which also pops out wihtout explanation) chosen in Eq 13. the policy may collapse to matching a single mode of the energy model but may also result in much higher energy / better reward than the chosen policy. The authors do discuss some of these nuances below in a short section on why SFT (which uses a forward KL) might underperform the reverse KL approach. But all this does is it leaves the reader with the impression that the authors painted too broad a picture to derive a connection that then, in practice is not relevant. This could be rectified by perhaps framing the paper as \"RLHF can be seen as imitating an optimal policy based on human preferences\" and toning down some of the quite strong language, e.g. \"learning without an RL loop\" etc.\n- The paper seems to be derived backwards as some of connections made feel slightly contrived upon reading. E.g. the jump from the original objective to knowledge distillation mentioned above. The steps taken to arrive at a DPO like objective from density ratio estimation etc. The paper requires a lot of steps to arrive at a simple algorithm that the authors probably had in mind from the get-go and started from.\n- The knowledge distillation connection seems tenuous (and already known), it seems more straightforward to think of the entire process as imitating a better policy as in MARWIL [2] or as chaining a specific variant of an improvement operator and distillation as already derived in detail for many different variants in [3].\n- A lot of the derived formulas and connections are already known in the literature but this is often not explicitly stated, e.g. \"\nIn this section, we connect RLHF to the imitation learning framework. We show that RLHF is a special\ncase of imitation learning problem by defining the following specialized energy-based mode\" in front of Eq 9, which very clearly is already derived in the DPO paper and literature on RL as probabilistic inference. It is fine to re-state such derivations but then please say: we built on a well known connection between RL and energy based models/probabilistic inference.\n- The key innovation that the paper hinges on seems to be the approximation of the log ratio between chosen and current policy but the derivation seems very ad-hoc and on shaky foundations. To be explicit: in order to arrive at their Eq. 21 (and thus Eq 24 which is their DIL objective) they make the assumption that the reference policy is the same as the policy that generates the rejected samples only and disregard any terms on the positive examples; i.e. \"Here, we use the set of rejected responses y_l ∼ π_ref(y | x) to approximate the expectations under π_ref(y | x)\". This is simply a wrong assumption. I do not know why the authors have chosen to make the assumption, but it feels like a contrived way to come to Equation 24 and form a connection to a DPO like objective. \n- The results are on relevant benchmarks, but the improvement over DPO seems minor in most cases. In this scenario what would be nice would be to analyze qualitative differences, e.g. examples which DIL seems to have stronger performances on compared to DPO. Or an analysis on how closeness (in KL) wrt. to the reference policy evolves during the course of optimization for different algorithms and how this affects performance. Or a plot that have the DIL objective on the x axis and win-rate (over different models, e.g. reference policy and DPO) on the Y-axis.\n\n[2] Wang, Qing, et al. \"Exponentially weighted imitation learning for batched historical data.\" Advances in Neural Information Processing Systems 31 (2018).\n[3] Ghosh, Dibya, Marlos C Machado, and Nicolas Le Roux. \"An operator view of policy gradient methods.\" Advances in Neural Information Processing Systems 33 (2020): 3397-3406.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6z3qBybFeo", "reviewer": "Reviewer_sbFs", "summary": "- This paper reinterprets preference alignments methods like RLHF and DPO as special cases of a more general imitation learning objective. \n - They mathematically show how the RLHF and DPO objective functions fit within a general imitation learning framework. \n - They develop a new alignment method DIL based on imitation learning with the objective as minimizing the reverse KL loss between the optimal policy and current policy and derive a preference data based learning objective which suppresses the likelihood of generating dispreferred responses while increasing the likelihood of generating preferred responses. \n - They empirically show that DIL results in a better policy compared to other offline alignment methods across reasoning and alignment benchmarks.", "questions": "- Since DIL doesn’t suppress the likelihood of dispreferred responses as much as SimPO, how does this affect alignment from a safety perspective? Is the model more prone to generate harmful responses?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- This paper is well written and presents intriguing connections between imitation learning and human preference alignment. \n - They derive a new alignment framework based on imitation learning and show empirical improvements on existing baseline.  \n - DIL shows significantly better training dynamics compared to SimPO by ensuring that the likelihood of generating chosen responses is maintained.", "weaknesses": "- The amount of data needed for satisfactory alignment with DIL compared to other methods is not clear. The authors claim that DIL is more efficient, so it would be nice to see some metrics that measure this. \n - All the models in the experiments are smaller (<10B parameters) so it’s not clear how effective DIL would be for larger models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mHWcfJnOgp", "reviewer": "Reviewer_MhY4", "summary": "This paper introduces a new method called Direct Imitation Learning (DIL), which is derived based on an imitation learning perspective on the alignment problem. Specifically, instead of minimizing the forward KL divergence as in SFT, DIL aims to minimize the reverse KL instead. This turns out to require estimating the density ratio $\\frac{\\pi_{\\mathrm{chosen}}}{\\pi_{\\mathrm{ref}}}$, which the authors show can be done through a Bregman divergence objective. Then, through a similar change-of-variables trick as used in DPO, the authors show that this reward objective can be instead minimized directly in terms of the relevant policies. Hence, the final objective directly optimizes $\\pi_{\\theta}$ through the Bregman divergence objective. \n\nThe authors also show that PPO and DPO can be seen as special cases of the proposed imitation learning formulation. Specifically, reward learning in RLHF can be formulated as a forward KL between $\\pi_{\\mathrm{chosen}}$ and $\\pi_{\\phi}$, and the RL step can be seen as a knowledge distillation process (through minimizing a reverse KL) into a final policy $\\pi_{\\theta}$. \n\nFrom the experiments side, the authors use the UltraFeedback Binarized dataset for evaluation on the Open LLM Leaderboard and show DIL is generally the best method across the board. For dialogue generation and summarization they use the Anthropic HH dataset and the Reddit TL;DR dataset and show through win rates (as judged by GPT-4) that DIL generally performs best against the SFT, Chosen, and Average responses. Finally, the authors also investigate the likelihood patterns of DIL and SimPO, which generally seem to show that the likelihood of chosen responses stay roughly the same while the likelihood of rejected responses goes down. This is unlike SimPO for which the likelihood of chosen responses also decreases.", "questions": "1. In Section 5 under the models paragraph, the authors state “For fine-tuning on UltraFeedback Binarized dataset, we use Zephyr-7b-SFT (Tunstall et al., 2023) and Llama3-8b-SFT used in (Meng et al., 2024) as our base models.”, but then in Table 2 the top results are labeled as Mistral-7B-Base. Should that be Zephyr-7B-SFT instead?\n2. In Section 5, the authors mention KTO as part of the baselines, but it doesn’t seem the result tables include it? Also, SLiC is included in the result tables, but is not discussed in the baselines paragraph?\n3. Could the authors include the base model (SFT) performances in Table 2?\n4. In Table 3, what is the difference between Chosen and Average?\n5. In Table 3, it might be interesting to compare win rates of DIL directly with DPO or other baselines. Is there a reason the authors didn't include this?\n6. At the end of section 6.1, the authors state that “We hypothesize that these improvements can be attributed to avoiding the BT assumption and preventing the decrease in the likelihood of chosen responses.” Could the authors elaborate on why avoiding the BT assumption could lead to these improvements? Do they have examples in mind where BT might not be the right model?\n7. I’m a bit confused as to how $\\pi_{\\mathrm{chosen}}$ is defined. Is it essentially defined to be the policy that, given a preference dataset of $ (x, y_w, y_l) $ triplets, was responsible for generating all the $y_w$ pairs?\n8. In the beginning of section 4.3, the authors state that “In the tabular setting, we can directly compute $\\pi_{\\mathrm{ref}}(y | x)$ and $\\pi_{\\mathrm{chosen}}(y | x)$.” Could the authors please elaborate on this a bit? It’s not clear to me what the tabular setting here means.\n9. Is the Y-axis in figures 1 & 3 the *negative* log likelihood? And for the margins figure on the right, is it a difference of negative log likelihoods? This could use some better labeling. Putting the model name on the y-axis is a bit confusing, and might be better put in the caption.\n10. At the end of section 4.1: “achieving this in practice requires full data coverage and infinite models that are rarely met”. What is meant by “infinite models” here?\n11. In the paragraph right after equation 22, what’s $\\pi_{\\mathrm{data}}$?\n12. In the paragraph right after equation 22, why is there no log before the reward $r$ in $Z(x)$? Shouldn’t there be since there is one in equation 22 as well?\n13. In the paragraph after equation 22, the authors state “This characteristic, determined by the reward definition in Equation (17), is super beneficial as it allows our imitation learning to theoretically generalize to a broader class of loss functions beyond the pairwise BT preference model used in DPO.”. Could the authors please elaborate on this? What does \"this characteristic\" refer to? And how does it allow the imitation learning to generalize to a broader class of loss functions beyond BT?\n14. At the end of section 4.5 the authors state “Specifically, we demonstrate that DPO also falls under the imitation learning objective in Equation (16) and essentially employs the CPC method for density ratio reward estimation.”. While I agree CPC indeed estimates the correct density ratio, it’s unclear to me that this is used in equation 27. Specifically, the learned $f^*$ from equation 26 doesn’t seem to show up in equation 27?\n15. Towards the end of 6.1, the authors state “Notably, we observe DPO and SimPO hurt the overall performance in most reasoning-heavy tasks such as GSM8K”. Is this compared to some base model performance? And if so, where is this reported?\n16. This statement in 6.1 could use some clarification: “For instance, on LLama3, the improvements are notable on the Math and AlpacaEval 2 benchmarks, with relative gains exceeding 7.5% and 18.2%, respectively.” Is this for DPO or SimPO?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The paper has several strengths:\n1. The paper provides new mathematical connections between imitation learning formulations (various forms of forward and reverse KL optimizations) and previously established RLHF methods like PPO and DPO. As far as I'm aware, these connections are novel and have not been highlighted in past work, making them valuable insights for the community to further build on. \n2. To optimize the proposed imitation learning objective, the paper integrates ideas from density ratio estimation [1] and a change-of-variables approach [2] (rewards -> policies) to directly learn the target policy $\\pi_{\\theta}$, avoiding complexities such as adversarial training.\n3. Strong empirical results: the new method DIL seems to generally outperform all baselines in both the Open LLM Leaderboard as well in the summarization and dialogue generation settings.\n\n[1] Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Density-ratio matching under the bregman divergence: a unified framework of density-ratio estimation. Annals of the Institute of Statistical Mathematics, 64:1009–1044, 2012.\n\n[2] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.", "weaknesses": "The paper has several weaknesses:\n1. While the empirical results seem to consistently outperform prior methods, I’m a bit worried about the statistical significance since the margins seem rather small sometimes (e.g. for Table 2, the improvements are almost always smaller than 1 percentage point). Could the authors include some significance tests or at least standard errors / CIs to provide a better sense of the significance of these improvements?\n2. The exposition of the math/theory in the paper could have been a bit clearer (section 4). It took me some time to understand what actually is the final objective that DIL optimizes, and how it came to be. This is because, for example, at the end of section 4.3 the authors state “With the estimated density ratio reward, the surrogate imitation learning objective in Equation (17) can then be solved with any RL algorithms.”, which initially made it seem like DIL would have to resort to RL optimization anyways. But then reading section 4.4 it turns out that’s not what happens and there is actually a different objective that’s maximized (eq. 24). Maybe one thing that could help here is to add a summary box either at the beginning or end of section 4 that summarizes the key steps to go from the general DIL objective (eq. 16) to the formulation in eq. 24. \n3. Some parts of the paper require further clarification - please see the Questions section for this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kbnYgP2E6n", "reviewer": "Reviewer_hdJf", "summary": "This paper generalizes preference learning or RLHF frameworks to an imitation learning framework, (DIL). Using this framework they propose multiple offline preference learning methods with different preference modeling such as Bradley-Terry for DPO and LSIF for the best DIL model. Moreover, its performance on benchmarks like Alpaca Eval 2 and Open LLM leaderboard is considerably better than other offline preference learning objectives.", "questions": "Q1) You mention that DIL does not depend on Bradley-Terry but you introduce new reward training with different objectives such as LSIF, UKL, and BCE which are essentially replacements for BT, so doesn't the DIL still rely on some preference modeling assumption?\n\nQ2) In 6.3 you discuss learning dynamics DPO, SimPO, and DIL however Figure 3 does not have DPO, is the discussion from some other paper?\n\nQ3) Do you have additional results on MT-Bench or Arena Hard?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* The paper is well-written and easy to follow.\n\n* The first paper to connect RLHF with imitation learning if not mistaken\n\n* Very strong results against popular DPO-variants", "weaknesses": "## Is RLHF a form of imitation learning?\nPaper frames reward learning as imitation learning and RL as knowledge distillation (KD) and I dont think either of them is correct.\n\nRL: Equation 13 is the reverse KL between the behavior and optimal policy however knowledge distillation is forward KL between teacher (optimal) and student(behavior). KD (forward KL) is distribution fitting or mean seeking whereas reverse KL  is mode seeking which makes the policy focus on high-reward regions rather than fitting the entire distribution with forward KL as in SFT. Overall, the KD claim by the paper is incorrect. Lastly, equation 13 is a known result from DPO paper which is the penultimate step of the optimum solution of equation 14th.\n\n\nReward Learning:\n\nIn standard RLHF, the reward model is a separate LLM with an additional MLP to predict the scalar reward. So by training a reward model, one does not imitate the expert or optimal policy. What we are doing is fitting a reward model to a predetermined preference model however the caveat is that the optima policy trained by the RL can be parametrized by the reward model trained with which was already proven by the DPO. Lastly, DPO parametrizes the reward model in terms of the policy so when the reward learning objective is trained, we obtain the actual policy.\n\nOn the other hand, this paper defines a Boltzmann distribution $\\pi_\\phi$ (equation 9) in an EBM framework which is the optimal policy induced by the $r_{\\phi}(x,y)$. This distribution is maximized on the chosen preferences generated by the $\\pi_{expert}$ or imitates $\\pi_{expert}$. Following derivations leads to reward likelihood training objectives whereas I am unsure whether $\\pi_{ref}$ approximation is free because it introduces rejected responses while IL objective only minimizes on chosen preferences. Nonetheless, this derivation is possible because the $pi_\\phi$ has a reward equivalence whereas it does not tell anything for other forms of policy. Overall, I would interpret it as imitating reward rather than policy, not vice versa.\n\n## Direct Imitation Learning\nI dont think DIL is novel because it is the backtracking of the derivation of the DPO objective. After all, the 16th equation is the same as the 14th equation of the DPO without the partition and assuming $\\pi_{expert} = \\pi^*$. DIL is redefining the reward function of DPO, excluding the density ratio estimation part. All in all, I believe this part(excluding density ratio) is already present in DPO.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)