Joint Reward and Policy Learning with Demonstrations and Human Feedback Improves Alignment
Authors: Chenliang Li
Siliang Zeng
Zeyi Liao
Jiaxiang Li
Dongyeop Kang
Alfredo Garcia
Mingyi Hong
Venue: ICLR 2025 Spotlight (2025)
Keywords: Alignment
Inverse Reinforcement Learning
Reinforment Learning from Human Feedback
Primary area: foundation or frontier models, including LLMs

Abstract: Aligning to human preferences and/or intentions is an important requirement for contemporary foundation models. To ensure alignment, popular approaches such as reinforcement learning with human feedback (RLHF) break down the task into three stages: (i) a model is computed with supervised fine-tuning (SFT) based upon large demonstrations data, (ii) a reward model (RM) is estimated based upon human feedback data, and (iii) reinforcement learning (RL) is used to further refine the SFT model by optimizing the estimated reward model.  Demonstrations and human feedback data reflect human user preferences in different ways. As a result, the reward model estimate obtained from only human feedback data is likely not as accurate as a reward model estimate obtained from both demonstration and human feedback data. A policy model that optimizes the reward model estimate obtained from both demonstration and human feedback data will likely exhibit better alignment performance. We introduce a tractable algorithm for finding the reward and policy models and provide a finite-time performance guarantee. Additionally, we demonstrate the efficiency of the proposed solution with extensive experiments including alignment problems in LLMs and robotic control problems in MuJoCo. We observe that the proposed solutions outperform the existing alignment algorithm by large margins, especially when the amounts of demonstration and preference data are unbalanced.

Published as a conference paper at ICLR 2025
LEARNING REWARD AND POLICY JOINTLY FROM DEMONSTRATION AND PREFERENCE IMPROVES ALIGNMENT
Chenliang Li', Siliang Zeng", Zeyi Liao², Jiaxiang Li2, Dongyeop Kang².
Alfredo Garcia', Mingyi Hong"
Texas A&M University, College Station, University of Minnesota, Twin Cities, The Ohio State University, Columbus
ABSTRACT
Aligning human preferences and/or intentions is an important requirement for contemporary foundation models. To ensure alignment, popular approaches such as reinforcement learning with human feedback (RLHF) break down the task into three stages: (i) a model is computed with supervised fine-tuning (SFT) based upon large demonstrations data, (ii) a reward model (RM) estimated based upon human feedback data, and (iii) reinforcement learning (RL) is used to further re- fine the SFT model by optimizing the estimated reward model. Demonstrations and human feedback data reflect human user preferences in different ways. As a result, the reward model estimate obtained from only human feedback data is likely not accurate tion and human feedback data. A policy model that optimizes the reward model estimate obtained from both demonstration and human feedback data will likely exhibit better alignment performance. We introduce a tractable algorithm for find- ing the reward and policy models and provide tinite-time performance guaran- tee. Additionally, we demonstrate the efficiency of the proposed solution with extensive experiments including alignment problems I.LMs and robotic con- trol problems in MuloCo. We observe that the proposed solutions outperform the existing alignment algorithm by large margins, especially when the amounts demonstration and preference data are unbalanced.
reward model estimate obtained from both demonstra-
INTRODUCTION
As ChatGPT has taken the world by storm, it is clear that systems will soon become ubiquitous in our lives. For instance, Large Language Models (LLMs) have been used to solve hard problems including video gaming (Berner et al., 2019 Mnih et al. 2013), autonomous control (Bellemare all 2020), and robotic manipulation (Kalashnikov et 2018 Kober & Peters 2008). In this context, the notion of alignment plays an increasingly important role the design and training of AI systems. Loosely speaking, alignment refers to the performance guarantee that the AI system will generate outcomes that are intended or preferred by the human user without undesirable side effects or behaviors such as deception (Park et al. 2023) or manipulation (Perez et al.) 2022). As human user intentions or preferences may vary under specific contexts, it is critical that the AI system adapts evolving user preferences and/or intentions (Leike et al. 2018).
The alignment problem is a learning problem with (at least) three types of input data: the demon- stration data (consists of prompts and gunan-eeeerated continuations), the preference data (con- sists of prompts and pairs of human-ranked responses), as well as prompts without any responses. Moreover, the process aligning an LLM model is typically undertaken in successive stages. For example, the well-known RLHF approach adopted by Ouyang et al. (2022) starts with a supervised fine-tuning model (SFT) followed by reward model (RM) estimation based upon
preference data. The process closes with final alignment stage in which reinforcement learning (RL) is used to optimize the estimated reward model. Similar strategies have been used in other related works such as Rafailov al. (2023): al. (2023): Zhu et al. (2023) Liu al. (2023) The

---

Published as a conference paper at ICLR 2025
approach to alignment based successive stages may facilitate computation, but it is at the expense of inefficeent exploitation of data. To illustrate, consider the three-stage RLHF approach proposed in Ouyang al. (2022), in the extreme case where the amount of high-quality preference data is quite limited, the reward model trained cannot adequately reflect the preferences of the human, which may lead to unsatisfactory performance in the RI. stage. Further, the reward model estimate ob- tained from only the preference data fails to exploit the information about human users' preferences that are implicit in demonstration data. It therefore reasonable expect that a policy model that is fine-tuned with the reward model estimate obtained from both demonstration and human feedback may exhibit better alignment performance.
An alternative to the successive approach to alignment consists of jointly training the reward and policy models by leveraging demonstration and preference data. contrast to the successive ap- proach adopted in most of the current alignment approaches, the joint approach to reward and policy learning makes use of all available data, hence mitigating the risk of optimizing an inaccurate reward model. However, joint approach to learning reward and policy models may improve alignment at the expense of potentially significant additional computational effort.
Contribution. We introduce an algorithm jointly learning reward and policy models named Align- ment with Integrated Human Feedback (AIHF) with a finite-time performance guarantee. This ap- proach leverages recent advances in Inverse Reinforcement Learning (IRL) (Arora & Doshi 2021 Zeng et al. 2022b), stochastic choice theory (Blavatskyy & &Pogrebna 2010) and bi-level optimiza- tion (Hong al. 2020 Ji et al.) 2021; Khanduri et al.) 2021). The proposed formulation integrates SFT, RM, and RL into single stage, so that reward modeling and policy optimization can filly leverage all the available human feedback data. More specifically, in the proposed algorithm, the policy is updated to improve alignment with the current reward model estimate and the reward model updated to improve the fit to demonstration and human feedback data. As result, upon convergence, the resulting reward and policy models are consistent in the sense that (i) the policy model optimal with respect to the reward model and (ii) the reward model maximizes the fit to both demonstration and human feedback data. Several existing alignment schemes, such as RLHF Ouyang et al., 2022) and DPO (Rafailov et al., 2023) and some of their extensions can be seen as particular instances of the proposed formulation. We provide ample empirical evidence that the proposed AIHF solution outperforms the existing alignment algorithms by large margins, especially when the data is unbalanced, where the quality and/or quantity of one data category is worse/smaller than that of the other.
PRELIMINARIES AND RELATED WORK
2.1 NOTATION
The Finite-Horizon MDP Model. A Markov decision process (MDP) is the tuple (S, A, P. P.P.rr wherein S denotes the state space, A denotes the action space, P: x A x x [0, 1] denotes the transition probabilities, p(-) is the initial state distribution, r : x A - IR denotes the reward function and 7 (0, 1) denotes the discount factor. For every 8t S, a randomized policy (-1se) a probability distribution in A LAI, the unit simplex in RR Al Define T == {s )z) as a
(finite horizon T') trajectory of state and action pairs. Let Mr C II (S x A) denote all feasible state/action sequence of length T.
MDP Model of LLM. The generation of text by language model can be seen as sampling from policies an MDP model. Specifically, each state 8; = (2,, y1 includes the prompt x and all response tokens produced up to that point V1:1-1- Each action a₂ Yt represents a token from the vocabulary. The transition kernel P is deterministic, i.c. given tokens 8t = (z. 31:--1) and ag yt. the environment will transition to 8t+1 = (x, y1=)). An LLM be seen policy T (-(s.) so that a response of length T > 0 to prompt x is obtained with probability: (ynnilliitty(yy:rrrorr (x) ==
II [J,, 1))
Human Feedback Data, Let := (yITT, .r) denote a finite text produced in response to prompt x.
For pair of sequences (Ti, Tw)) (which we assume of the same length T for ease of exposition) we write 7. Ta to indicate the sequence T., is preferred over the sequence 7) Following the Bradley- Terry-Luce (BTL) model (Bradley & Terry 1952), the distribution of preferences over pairs (Ti, Twr)
can be modeled as follows:
exp R( 0) P(r > nn)= exp R(T 0) + exp R(T; = (B((r::) - R(n;
(1)
2

---

Published as a conference paper at ICLR 2025
H
{
(2) RM
I
RM
Policy
Preference data
RL
Preference data
I
H
Demonstration data
SFT Model
(3) Policy
Demonstration
Alignzeet
Reinforcement Leamiing with Human
Aligmment with Integrated Human Feedback (A (AIHF) (Proposed)
Feedback (RLHF): (1)-(2)-(33
Figure 1: Comparison of the RLHF (left) with the proposed AIHF (right).
where is the sigmoid function and R(T;0) Eizi ('r(s;; and r(se, ae; 0) is reward model parametrized by R2d
2.2 THE RLHF PIPELINE
RLHF popular technique for fine-tuning Al systems to align with human preferences and val- ues. The RLHF approach proposed in SStennon et al. (2020); Ouyang et al. (2022) consists of the following three-stage: the supervised fine-tuning (SFT) stage, where the demonstration data used to fine-tune the model in a supervised manner; 2) the reward modeling (RM) stage, where the preference data is used train a reward model; 3) the reinforcement learning (RL) stage, where the SFT model is further fine-tuned by running RL using the trained reward model. Specifically, the RLHF pipeline can formally described as follows:
Supervised Fine-Tuning (SFT): Given a demonstration dataset D consisting of sequences of the form {(s,, h220 the goal is the find the policy #SFT(-1s)) that maximizes likelihood, i.c.:
TSFT arg max BTND log II T (m(alss)))* 120
Reward Modeling (RM): Based upon a dataset of preferences over pairs (7,, Twr) the estimation of reward learning problem can be formulated the following Tradley-Terry-Lann (BTL) model (Bradley Terry 1952) (with 3 > hyper--paree sapper-arammter)::
max RRM (0) = IE (F( log ( R(m; ((): Be
(3)
Reinforcement Learning (RI)): Let Op denote the solution problem The last stage in the RLHF development pipeline consists of solving the problem:
Σ
> [r 0p) 3DKL ( (T((s))]]ssrr(((s))
TRLHF arg max E
(4)
120
s(ales) where DKL (T (-(1):) |SFT (TST((1ss))) (1111) = (als)) log (apr(a(s.) is the Kullback-Leible (KI.) di-
vergence, #SFT is the supervised fine-tuning model. Due to the space limit, we put the rest of the literature review the Appendix A.
ALIGNMENT WITH INTEGRATED HUMAN FEEDBACK (AIHF)
As mentioned before, the reward model obtained in 3 fails to exploit the information about human users' preferences that are implicit demonstration data. As result, the fine-tuned model obtained with RLHF may exhibit unsatisfactory alignment performance (this phenomenon will be discussed more concretely in Sec. 3.4). Below we introduce a new approach jointly train reward and policy models by simultaneously leveraging demonstration and human feedback data.
3.1 FETA-
Towards developing an approach that can model the entire alignment process with a common parametrization for both policy and reward models, consider the following formulatoo,,,,,on, termed
3

---

Published as a conference paper at ICLR 2025
Alignment with Integrated Human Feedback (AIHF):
(AIHF)
max L( == w (1(e)) + L2((8:))) B
(5a)
s.t. RB := arg max La( R(:0))
(5b)
where 9 E Rd is a parameter; (1(0) is measure of fit of the parameterized policy Ty to demon- stration data and L2 ()(:;))) is measure of fit of the parameterized reward model R(:;0) to the preference data and [T O)) is a measure of performance of policy π with respect to reward model R): wi 2 versus preference data. Note that in the lower level policy optimization, the optimal policy corre- sponding the the reward model R(:;0) 0) is actually determined by the reward parameter 0, where we can denote the optimal policy under the the reward model R(:: f) as TRo := Therefore we simply put Ta since the optimal policy under a certain reward model R(:; () is actually determined by the reward parameters 9.
one balancing coefficient reflecting the relative size of demonstration
See Fig. [ for an illustration of AIHF. The AIHF (5 is a meta-problem that models the alignment problem. has two levels: an upper-level problem which the goal find policy and reward models that jointly maximize a measure of fit to demonstrations and preference datasets; and a lower- level problem which ensures that the policy model optimizes performance with respect to the reward model. Its components can be customized to yield specific alignment formulations and algorithms. Before diving into various custonizations,
us discuss the advantages of this formulation.
Generality. One can specialize the loss functions and problem parameters to yield number of existing alignment formulations. Such generality implies that algorithms developed for (5) are easily applicable to different special formulations it covers. For more details see Sec. 3.3
Joint optimization. The formulation jointly optimizes the reward and the policy. One benefit here is that it can strengthen the reward model through integrating both demonstrations and pairwise com- parisons. Compared with the standard RLHF pipeline, through integrating additional data source such demonstrations train the reward model, it can further boost the policy optimization sub- routine to achieve better alignment performance. See Sec. 4 for detailed discussion on how the reward parameter is updated by leveraging such demonstration, and see Tab. 4 for the experimental comparison between the reward model learned by RL.FF and by our AIHF (5)
Dataset Integration. Clearly, the reward learning process leverages all the available data, there- fore, can expect that high-quality reward model and its induced optimal policy can still be obtained even under unfavorable situations where the preference data is not sufficient. See Sec. for experimental evidences.
3.2 SPECIFICATION OF AIHF
In this section, we specify the formulation (5) Let begin with the choice of L1 It can be directly instantiated by using one objective similar to 2 which is the likelihood function over the collected expert demonstrations. Note that aim optimize the reward parameter 0 to align with human feedback in (5a). thus the objective of L, can be specialized as a maximum likelihood function over expert demonstrations as below:
(T(T) = ENND log II (TT((uls)))*] EYND
Σ of log Te (colse)
(6)
120
+2.0
Here To optimizes the measure of performance L2(T;; R(:; for reward model R(: 0) as:
Ra;T;)(;00)) R(: = ESo--TTT R(T; - 3 Σ D'T. (T((11))))))
120
where 70 is some initial policy and 9 > 0 is temperature parameter.
Next, we specify L2. To ensure internal model consistency, we identify the likelihood function for preference data so it is in accordance with the preferences implied by the reward model R[;; 0) used the definitions of L1 and L3. Thus, the optimal distribution μı over the set of Tollon sequence of state-action pairs is defined as follows:
pg == arg max Erwny [R(T;)) [Dl((1])
HEAT
4

---

Published as a conference paper at ICLR 2025
where Ar denotes the simplex Mr and de is prior distribution the trajectories. It can be shown the solution of the above problem is of the form:
1 (T (R(T; 0)/3) (o(T) ExEN, (T)) exp (R(T':0)/8))
With this result, can now obtain model for the likelihood that sequence т; is preferred over Tj. By the independence of irrelevant alternatives property (Fudenberg et al. 2015) the optimal choice μax when the set of feasible choices is reduced from Hy to just the the two-tuple {7, Tw
Pa(Tw) This the likelihood that sequence Tw is preferred over 7 is given by Po(TT > TI) == +Hoo))) motivates the choice of (0) as the following likelihood function:
(T(TT)
L2(R(:;0) 0)) = Br TD)E log mo(Tw) ) + (T()T))
1 (Tw) exp (R(Tw:
(E log (T())) exp (M((Tu;)) μ (m) exp (Him: 0)
With n equal to the uniform distribution on Hr, this model is equivalent the BTL model (3)
L2
BBL = (RM(() = -rr log (a(R(Tw:)) - A((::))))
(8)
3.3 SPECIAL CASES OF AIHF
Next, we discuss how formulation (5) can be specialized to some of the known alignment algorithms. Specialization to RLHF-Type Approach. First, we set the coefficient wi = 0 in (5). we obtain:
max (2 S.L. na == arg max La(T; R):
Noticed that now the upper- and lower-level problems are completely decomposable, since the upper-level problem solves for the reward parameterization 6, while the lower-level problem solves for the policy (for the given reward), yielding two separate problems, which are exactly the RM and the RL problems in the typical RLHF approach.
Specialization to DPO-Type Approach. Consider the relationship between formulation 5 with the DPO-type approaches. Let set the following objective function L, ÊSET and L2 ERM, and assume that T = 1 for the generation process. Relaxing the constraint (5b) which ensures the policy is optimal w.r.t. certain parameterized model, can obtain a DPO-type formulation:
max; L(T) wi Eyewen log log ((B))sB)
+B BiG log (B log sla,les) ("(a)))) Blog steilas)
(10)
The above formulation specializes to Liu et al. (024), which is slightly generalized version of DPO when both demonstration and preference data are used. Setting wi = reduces to the problem solved by DPO; see Rafailov et al. (2023 Eq. (2)).
Specialization to Self-Play Approach. Define E(-) as monotonic and convex loss function, con- (1) (R(=B:0) - R(T; 0)), and setting L, and L according to (8) and 7 respectively. Note that the choice of L1 means that given demonstration data, will find a policy which generates trajectories that match the rewards the demonstration data. Again using DPO type of reformulation, by substituting the reward expression obtained from the optimal policy 42 to L1 and selecting the (-) as E(-), then the AIHF problem in this case becomes:
sider setting 7. == wi
Blog (()
3 Blogg T(
max L(T) = E (mm(((s) log
T (a
(ā(sF))
n
T(ar(s)) 3log <(ai(s)) π (a
log
B log
(11)
T (aylo;))
Note that the first part of the above formulation is similar to what has been proposed in SPIN (Chen all 2024), which only utilizes the SFT data.
5

---

Published as a conference paper at ICLR 2025
3.4 WHY AIHF CAN OUTPERFORM TWO-STAGE ALIGNMENT APPROACHES?
To understand the difference between the proposed approach and the successive stages approach of the standard alignment pipeline, let us consider the a static setting with action set is A := .N. 771;72 ). reward function R(-)) A + R. and demonstration 'D and preference dataset P. In what follows, we compare the optimal solutions for policies obtained by different alignment ap-
proaches. Due to space limitation, all derivation in this section is relegated to Appendix A.3
Policy with Demonstration Data. It can be easily shown that when only the demonstration data D is available, the probability of generating i-th data equals to its empirical probability, i..., TSFT(T,)
# (T. P Assume that such policy is parameterized by an implicit reward function Rp, using the
following softmax choice model where тÜ A is selected with probability कद (R) Σز exp(
exp R. /8)
where R R(T:). Assuming reference value Rop(T1) R1, then the optimal rewards satisfies
(See Sec. A.3.1
# (Ti E ED [D] = (RD)
(T)(R((TT//))) exp(RD
i NY..
(12)
This implicit reward will be used shortly to characterize the RLHF policy. Policy with Preference Only Data. Next, it can be shown that when only the preference data P is available, the reward estimation problem is defined as:
xt (R) log x7 (R) + πz (R)
Rp arg max RMM (R) E ((
(13)
R
Again with fixed reference value (T1 R1, the solution is (see Sec. A.3.1
Ejstes (Po-ol
((P)
(14)
Σ Eiizi (Podle (1.5) (T* (Rp))
(1
where (Perjl = #(T: > Tj in P} and (Pizl == (Po-sl + iPiril and (-(
is the expected number of times an action other than T; or is selected when sampling actions from æ infinitely many times.
ExEAM(iy Tk)
RLHF Policy. Based on the above results, in Sec. A.3.2 we show that the RLHF approach has the optimal policy π RLLE (Tr) = ग. RRD + Rpp). That is, the RLHF policy can be seen as the softmax policy for the sum of reward estimators obtained from demonstrations and preferences separately. ATHF Policy, Finally, we also find that the AIHF policy is of the form:
#(r. in D} + Eisi (Pinsl
π
(T:) [D] + Ejmi Pio (1.)) (T" (RAIHF))
(15)
Discussion. Let us summarize our findings. First, RRLFF takes the form of softmax of the sum of two rewards, one learned from the SFT stage one learned from reward training stage provides some interesting insight to this popular approach. Second, ATHF is more robust than π RL.P To see this, suppose that [D] >> [P], i.e. there is more demonstration than preference data. In this case, the policy estimator in (55) will be largely defined by the demonstration data (which reasonable) whereas the RLHF policy can be noisy since it (soft) maximizes the sum of two reward estimators:
one that is more accurate (i.e. the one based on demonstrations, Rp)) and one that is less accurate (i.e. the one based on preferences Rip). A similar observation can be made when [D] & [P]: Sec. A.3.2 for more details. Third, π AAHF can have less variance as compared with n RLHF Indeed, AIIT takes a form of weighted average of the policies estimated separately with demonstration and preference data, as by using (12) and (14). we can re-write (15) as:
Eyec PPisle-can (v (kr)
IDI E,, Ie (in ( (- (RAIH))) x. (Ri) +
|P+ IPis w.s (== (RAIH))) #3 (Rp)
Such averaging entails reduced variance. We also include simple numerical examples in the Ap- pendix A.3.3 to further illustrate this point.

---

Published as a conference paper at ICLR 2025
4 PROPOSED ALGORITHM FOR AIHF TRAINING
We are now ready to design algorithms for the proposed AIHF formulation (5) To begin with, first note that (5) takes a hierarchical form, and it belongs to the class of problem named bi-level optimization, first developed in the 70s (Fiacco & McCormick 1990). and recently found many applications in machine learning (Wang et al. 2021; Liu et al. 2021[[202| bi-level problems are not easy optimize; more specifically, in (5). the upper-level problem (5a) is a function of both the lower-level optimal solution To and the true parameter 0. It follows that a (stochastic) first-order algorithm for [(0) involves some (potentially non-trivial) implicit gradient computation which often involves computing the Hessian matrix for the lower-level objective func- tion. Fortunately, as we will show shortly, with some special choices of L1, L2, La, one can design some simple and very efficient algorithms.
Generically speaking,
Before we go to details, we note that throughout this section, we assume that we are searching for a good policy Th and a reward estimate -;:-;)) to align with human feedback, where the policy me is an optimal solution w.r.t. the certain reward estimate r(s:;0) according to the policy optimization problem (5b). Due to such optimal policy constraint w.r.t. one explicit reward estimate, we design an algorithm to solve such a single-stage, bi-level problem which is different from DPO (Rafailov et all 2023) that simply optimizes the fixed loss function
10 directly.
On a high level, the proposed algorithm alternates between policy alignment step (which updates with a fixed reward r(;, :; 0)), and a reward alignment step (which updates 0 using a stochastic gradient, a function of the demonstration and preference data). Next, we study these steps in detail.
Policy Alignment Step. One can adopt the standard approaches such the well-known proximal policy optimization (PPO) (Schulman et al., 2017) algorithm to obtain an approximate optimal pol- icy which solves 77)- It is worth noting that, when considering T 1, our discussion leading to (42) indicates the optimal policy takes a much simpler form. In this case, it is possible to consider simpler method than running PPO to obtain the optimal policy. One alternative way is use baseline estimated reward value to perform variance reduction (Li et al. 2023), thus reducing the computational complexity.
is important to note that, the point of the above discussion is that these different choices for solving the policy alignment problem can be incorporated into our overall approach. Reward Alignment. In this step, we use stochastic gradient-type algorithm to optimize L(0). Towards this end, first, observe that
(I wn ( V (())
(16)
Clearly, regardless of the choice of L2, L2 is relatively easy to compute because the objective is directly related to f since L2) can be regarded as supervised learning loss and do not involve the optimal policy Tg. In particular, we have the following expressions:
V (T) I (rw Vo log [R(((T=;)) 0) R(7;0)))
(17a)
On the contrary, the computation of VL, (TT) is more involved, since L1 depends on 0 implicitly through the corresponding optimal policy ag. Fortunately, the following lemma indicates that this gradient has simple and intuitive form as well, and the proof can be found in Appendix A.4.2
Lemma 4.1 Suppose that L1 takes the form of the objective (6) for reward learning from demon- strations, and suppose that Lg takes the form ( with ci-) being the KL-divergence w.r.t. some initial policy T Then we have the following expression:
VoLi(TO) BBraaR (Vo[_,(() H()'
(18)
where To is the optimal policy given the reward model parameterized by 0, with the expression 40)
Intuitively, if the current policy my has not matched E yet. then the reward should be improved by going towards the direction suggested by the expert trajectories, while going away from those generated by the current policy. Similar to the BTL model, from the gradient expression (18) it is clear that the optimization is toward the direction of increasing the gap between the reward of the real samples (demonstrations) and the synthetic ones (model generated continuations).
In practice, a few approximations need to be made to obtain a stochastic gradient of L First, similarly, as before, the precise expectation cannot be obtained because the ground truth policy
7

---

Published as a conference paper at ICLR 2025
Algorithm 1: Aligument with Integrated Human Feedback (AIHF)
Input: Initialize reward parameter 6 and policy model n², the stepsize of reward update 1.tt P, D
denote the preference and the demonstration data, respectively.
for Iteration & 0, ...... do
Policy Alignment: Optimizing La by RL subroutine, e-g. PPO, obtain one improved policy k+1
Data Sample I: Sample an expert trajectory ~ D and agent trajectory from r' x *t41
Data Sample II: Sample preference pair (Tw > 7)) ~ P
Estimating Gradient: Calculate one gradient estimator g == wigf + 인 of
(0) = wh VoLi (0) VoL2(0)
Reward Alignment: gl++1 8t "19
end
T unknown. Denote an offline demonstration dataset as DE := (r h. then one can replace the expectations E,~rE by EE,DE. Second, in the second expectation in (18), the trajectories 7' are sampled from Tæ. the optimal policy for a fixed reward parameterization by This means that the policy alignment step has to identify the optimal policy To first, which, due to limitations such as computational constraints, and non-linear parameterization, is generally not possible. Instead, we propose sample from the current policy ***1 obtained from the previous policy optimization step, where index & represents the iteration counter. Following the approximation steps mentioned above, construct stochastic estimator 9k to approximate the exact gradient V ((0x) in (16) follows:
9k = wigh + 9 = (VoR(TE 02) VoR(TT 0x)) (1 (R(TT) 02) - ((Tk':0)))
x (VoR(TE W (2) VoR(TE 0k))
(19)
The above two steps are summarized Algorithm let us remark on the computational complexity of the proposed algorithm. Note that our algorithm is motivated by class of popular algorithms in bi-level optimization, where the upper-level and lower-level problems are updated alternatingly us- ing stochastic optimization (Hong et al. 2023). We conclude the section by theoretically inspecting the proposed algorithms.
Theorem 4.1 Suppose Assumptions 7 hold. Selecting stepsize a == K for the reward update
step 19) where on > 0 and iterations to be run by the algorithm. Then the following result holds:
(U, IJ are some fixed constants, and K is the total number of
K
E
log m6+1 - llog no, = [1]
(20a)
K
E
-0 - +O(K1)
(20b)
K
K=
where log T Tew bl == max.es, log nh+ (als)- Te, (als)] In particular, setting 1/2,, then both quantities in 20ah and (20b) converge with the rate (KK-1/2)
The above theorem shows that Alg. could converge to stationary point if we take a large loop number K. Note that details and proofs of the result above delegated to Appendix A.4
EXPERIMENTS
In this section, we provide numerical evaluations of the proposed method 5 (Alg. 1 and its vari-
ants (TO) and (11), and comparing them with state-of-ther of-the-art methods RLHF (Ouyng et al.) 2022), DPO (Rafailov et al. 2023), IPO (Calandriello et al. 2024) and SPIN (Chen et al. 2024). Our ex- periments demonstrate the advantages of the proposed methods in the following aspects: (1) Reward learning from demonstration and preference is the key to improving over standard RLHF, (2) Using demonstration in reward learning could increase model improvement efficiency (w.rt. the KL diver- gence violation) (3) AIHF could reduce the effect of distribution mismatch caused by the sequential alignment method, thus break the performance limits of the state-of-the-ar methods.
8

---

Published as a conference paper at ICLR 2025
Averoipe Score
Le
Awainng
64-
60
6 SEERT TUR MOTEREEE
DIrT wotkaand All lean Ed midee)
All Inon fre modE
AIE tear www.liitteeeee rode)
The Number Policy Optimization Step
The Number of Poliyy Optimication Step
The Number of Polcy Optimization Step
(a) Pythia- 160M
(b) Pythia- IB
(c) Pythia-2.8B
Figure 2: Experiment results of
policy models, with the reward model trained from Pythia-1.4DB We record the average scores (across three trials) of AIHF and RLHF the Authropic-Hae test dataset (See Tab. 5 in Appendix for more comparisons with other algorithms)
Models and datasets. In the first setting, we test Alg. I on Anthropic-HH (Bai et al. 2022 datase with (relatively small) Pythia (Biderman et al., 2023) model as policy models. Anthropic- Heropic--HHao is preference dataset collected from 52B LLMs that provide two continuations based helpfulness and harmlessness, and we pick 10k chosen/preferreed continuation data to form the demonstration dataset, while others serve preference dataset and RI. prompt dataset. For the HH dataset, We first fine-tune the language models (Pythia- Pyythia-100/1BB// 160M/1//2.8BBB through supervised fine-tuning over all chosen responses from the HH dataset for epoch, we call full-SFT model and use it as our base model. Moreover, we also SFT the language model using the selected top 10k chosen responses and name it demonstration-SV SFT model. For each policy model, we use the exact same model Pythia-1 1.4B as the reward model.
The other setting we test is on 7B models. We use Ultrafeedback as our preference dataset (61.1k preference data) and Ultrachat2001 as the demonstration dataset (208k demonstration data), with mistral-7b-sft-- [] (Jiang et al.) 2023) as our base model. We use the same mistral-7b-sst-beta model as the initialization of the reward model.
Evaluation. For the Authropic-Hle dataset, we present the reward evaluated by the PKU- Alignment/bcaver--bbWe/ -30-wwaard model al. 2024). In our 7B model experiments, we adopt the widely recognized HuggingFace Open LLM Leaderboard framework (Beeching et al. 2023). This evaluation suite measures LLM performance across six tasks: commonsense reasoning (Are (Clark)
şt all 2018), HellaSwag (Zellers et al.] 2019). Winogrande (Sakaguchi et al.] (021 ) multi-task
language understanding (MMLU (Hendrycks et 2020)), mimicking human falsehoods (Truth-
fulQA (Lin et al., (202T)), and math problem-sol ving (GSM8K (Cobbe et al. 2021 Additional implementation details can be found Appendix A.2
Results of small model (1B and 2.8B) experiments. We observe that the proposed AIHF per- forms effectively when initiated from both the demonstration-SFF model and the full-SFT model. As shown in Fig. 2 utilizing the same data, AIHF algorithm can eventually outperform RLHF irre- spective of the initial model. Furthermore, according to the numerical results as shown in Fig. 3(a] see that the proposed AIHF algorithm has smaller deviation from the base model compared with the RLHF algorithm. This benefit of the AIHF approach due to the fact that we incorporate the maximum likelihood IRL objective for both reward learning and policy learning. In this case, both reward model and policy model will be trained to align with the demonstrations, which are also used in the training process of the SFT stage. We also conducted study the demonstration/prefff ence data ratio in Fig. 3(b) and 3(c) We observe that AIHF consistently outperforms RLHF across different demoostraation/preferrnec data ratio. Furthermore, also record the performance of the two variants, namely AIHF-DPO (10) and Self-Play-AIHE (IT), also RLHF, DPO, IPO and SPIN in Tab. Our proposed AIHF still outperforms all these methods this experiment setting.
Results of large model (7B) experiments. We run AIHF, AIHF-DPO, Self-Play-AIin along with other methods on the 7B experiment setting. The results are presented in Fig. (the numbers are
Dataset available at ht http:///wwwiiccee//////////tttta
? Models available at ott.bbaaiif.ooo///ggoooo/eeeeeeo
Available ht
binari Zeal
'Available at http://////aaaaaa.oooa///////aaa
Available http/////a//ag.......//////////r

---

Published as a conference paper at ICLR 2025
Average Score
Alr
Arerage
sua for anvertaion Fomm
cary di reot
offie whth dimceiiaie
Dils Number Policy Optimization
The Number of Policy Optimization Step
The Number Policy Optimization Step
(a) KI. divergence to demonstration- (b) ATHF vs RLHF with 10k (c) AIHF vs RLHF with 10k SFT policy
demonstration, 5k preference
demonstration, 10k preference
Figure 3: Experiment results on Pythia- 1B policy models, where the reward model is trained from Pythia-1.4B models. We record the average scores of AIHF and RLHF on the Anthropic-HH test dataset, reporting the results across three different trials.
Mistral-77 Be-SFT
SPIN
IPO
Seft Play AIHF
Zopbya-78-9eta
DPO
AIHE-DPO
90
80
70
Scores
Arc Challenge
TruthfulgA
Winogrande
GSMBK
HellaSwag
MMLU
Average
Tasks
Figure 4: Performance comparison between AIHF-DPO, Self-Play AIHF training across the six benchmark datasets (See also Table 3 in the Appendix).
recorded in Tab. 3). where we can see that similar to the IB setting, AIHF is consistently outper- forming other methods. Additionally, both AIHF-DPO and Self-Play AIHF effectively outperform RLHF model (zephyr-7b-beta). The success of AIHF, as well as Self-Play AIHF and AIHF-DPO further suggests that joint learning from demonstration and preference is indeed beneficial for the alignment. We also conduct ablation study with different choice of wh in (TO), as shown in Tab. b the improvement of joint learning methods over baseline is robust. Furthermore, We also evaluate the reward models estimated using different methods (DPO, standard preference learning and AIHF) over the widely used RewardBench (Lambert al. 2024). The results, illustrated in Tab. 4 in Ap- pendix, show that the reward model trained through the AIHF can achieve significant improvement (especially on reasoning tasks) compared to both standard BTL reward model in (3) and implicit reward model in DPO.
Other Results. Due to the page limits, leave two additional experiments in the appendix: 1) movie review generation with positive sentiment on IMDb dataset (Maas et al.) [011]. 2) experiment on Robotics control tasks in MuJoCo (Toorov et al. 2012). For the result of MuJoCo Experiment A.2.1 we observe that even though Behavior Cloning (BC)SFT could provide a high-performing initialization, RLHF still fails to improve policy quality in the following RL stage. In the contract, AIHF can effectively integrate preferences and demonstrations, leading to more robust reward function and consequently, high-quality policy. For the IMDB result (Fig. ( We show that AIHF is able to alleviate the distribution mismatch between the generated trajectories by the policy, and the data that the learned reward model able to rank.
6 CONCLUSION
In this work, we study the alignment problem when diverse data sources from human feedback are available. Furthermore, we have developed an algorithmic framework that can integrate both expert demonstration and pairwise comparison data from human feedback to learn the reward functions for further guiding policy learning/model fine-tuning in the alignment pipeline. Through extensive evaluations robotic control tasks and large language model alignment tasks, demonstrate that our proposed method can outperform existing benchmarks on alignment tasks and is able to recover better reward model guide policy learning.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
M. Hong. S. Zeng and J. Li are supported partially by NSF under the grants EPCN-2311007, ECCS- 2426064 and CCF-2414372, also by Minnesota Supercomputing Institute. A. Garcia and C. Li are partially supported by 2011NF-22-1-0113 ECCS-224401 and CCF-2414373.
REFERENCES
Saurabh Arora and Prashant Doshi. survey of inverse reinforcement learning: Challenges, meth- ods and progress. Artificial Intelligence, 297:103500, 2021.
Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. A general theoretical paradigm to understand learning from human preferences. arXiv preprint arXiv:2310. 12036, 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.0553 2022.
Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Ra- jani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open Ilm leaderboard. https:
77
I Ilm leaderboa rd, 2023.
Mare G Bellemare, Salvatore Candido, Pablo Samuel Castro, Jun Gong, Marlos C Machado, Sub- hodeep Moitra, Sameera S Ponda, and Ziyu Wang. Autonomous navigation of stratospheric hal- loons using reinforcement learning. Nature, 588(7836):77-82, 2020.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, al. Dota 2 with large scale deep reinforcement learning. arXiv preprint /9/2.066//00 2019.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle OBrien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430. PMLR, 2023.
Erdem BIyIk, Dylan P Losey, Malayandi Palan, Nicholas C Landolfi, Gleb Shevchuk, and Dorsa Sadigh. Learning reward functions from diverse sources of human feedback: Optimally inte- grating demonstrations and preferences. The International Journal of Robotics Research, 41(1): 45-67, 2022.
Pavlo R Blavatskyy and Ganna Pogrebna. Models of stochastic choice and decision theories: Why both are important for analyzing decisions. Journal of Applied Econometrics, 25(6)9963-986 2010.
Michael Bloem and Nicholas Bambos. Infinite time horizon maximum causal entropy inverse re- inforcement learning. In 53rd IEEE conference decision and control, pp. 4911-4916. IEEE, 2014.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 3943/4):324-355, 1952.
Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond sub- optimal demonstrations via inverse reinforcement learning from observations. In International conference on machine learning, pp. 783-792. PMLR, 2019.
Daniele Calandriello, Daniel Guo, Remi Munos, Mark Rowland, Yunhao Tang, Bernardo Avila Pires, Pierre Harvey Richemond, Charline Le Lan, Michal Valko, Tianqi Liu, et al. Human alignment of large language models through online preference optimisation. arXiv preprint arXiv:2403.0886 08633, 2024.
11

---

Published as a conference paper at ICLR 2025
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self- fine- tine-nuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01335, 2024.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint 1803vv1803.05444 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2 14168, 2021.
Christian Daniel, Malte Viering, Jan Metz, Oliver Kroemer, and Jan Peters. Active reward learning. In Robotics: Science and systems, volume 98, 2014.
Anthony V Fiacco and Garth P McCormick. Nonlinear programming: sequential unconstrained minimization techniques. SIAM, 1990.
Johannes Fischer, Christoph Eyberg, Moritz Werling, and Martin Lauer. Sampling- inverse reinforcement learning algorithms with safety constraints. In 2021 IEEE/RSJ International Con- ference Intelligent Robots and Systems (IROS), pp. 791-798. IEFF, 2021.
Drew Fudenberg. Ryota lijima, and Tomasz Strzalecki. Stochastic choice and revealed perturbed utility. Econometricu, 83(6):2371-2409, 2015.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learming, pp. 1861-1870. PMLR, 2018.
Joey Hejna and Dorsa Sadigh. Inverse preference learning; Preference-based rl without a reward function. arXiv preprint arXiv:2305.15363 2023.
Joey Hejna and Dorsa Sadigh. Inverse preference learning: Preference-based rl without a reward function. Advances in Neural Information Processing Systems, 36, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mcasuring massive multitask language understanding. arXiv preprint arXiv:2009.000 03300, 2020.
Jiwoo Hong, Noah Lee, and James Thome. Orpo: Monolithic preference optimization without reference model. arXiv preprint arXiv:2403.07661, 2024.
Mingyi Hong, Hoi-To Wai, Zhaoran Wang, and Zhuoran Yang. A two-timescale framework for bilevel optimization: Complexity analysis and application to actor-critic. arXiv preprint arXiv:2007.05170. 2020.
Mingyi Hong, Hoi-To Wai, Zhaoran Wang, and Zhuoran Yang. A two-timescale stochastic algorithm framework for bilevel optimization: Complexity analysis and application to actor-critic. SIAM Journal on Optimization, 33(1):147-18)) 2023.
Borja Iharz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. Advances in neural information processing systems, 31, 2018.
Jiaming Ji, Mickel Liu, Josef Dai, Xuchai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of Ilm via human- -eeeannreeeerece dataset. Advances in Neural Information Processing Systems, 36, 2024.
Kaiyi Ji, Junjie Yang, and Yingbin Liang. Bilevel optimization: Convergence analysis and enhanced PMLR, 2021.
design. In International Conference Machine Learning, pp.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.0682. 2023.
12

---

Published as a conference paper at ICLR 2025
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishman, Vincent Vanhoucke, et al. Scalable deep reinforce- ment learning for vision-based robotic manipulation. In Conference Robot Learning, pp- 651-673. PMLR, 2018.
Prashant Khanduri, Siliang Zeng, Mingyi Hong. Hoi-To Wai, Zhaoran Wang, and Zhuoran Yang. A near-optimal algorithm for stochastic bilevel optimization via double-momentum. Advances in Neural Information Processing Systems, 34, 2021.
Jens Kober and Jan Peters. Policy search for motor primitives in robotics. Advances in neural information processing systems, 21, 2008.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Prin- ciples, pp. 61 1-1, 2023.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprini arXiv:2403. 13787, 2778,,00 2024.
Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: a research direction. arXiv preprint arXiv:1811.0777 2018.
Sergey Levine, Zoran Popovic, and Vladlen Koltun. Nonlinear inverse reinforcement learning with gaussian processes. Advances in neural information processing systems, 24, 2011.
Ziniu Li, Tian Xu, Yushun Zhang, Yang Yu, Ruoyu Sun, and Zhi-Quan Luo. Remax: A
simple, effective, and efficient method for aligning large language models. arXiv preprint WxXiv:2310.10500 2023.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv: 109.07998, 2021.
Risheng Liu, Jiaxin Gao, Jin Zhang, Deyu Meng, and Zhouchen Lin. Investigating bi-level optimiza- tion for learning and vision from a unified perspective: A survey and beyond. IEEE Transactions on Pallern Analysis and Machine Intelligence, 44(12):1005510007, 2021.
Risheng Liu, Pan Mu, Xiaoming Yuan, Shangzhi Zeng, and Jin Zhang. A general descent aggrega- tion framework for gradient-based bi-level optimization. IEEE Transactions on Patlern Analysis and Machine Intelligence, 45(1):38-57, 2022.
Shicheng Liu and Minghui Zhu. Distributed inverse constrained reinforcement learning for multi- agent systems. Advances in Neural Information Processing Systems, 35:33444-33453 2022.
Shicheng Liu and Minghui Zhu. Meta inverse constrained reinforcement learning: Convergence guarantee and generalization analysis. In The Twelfth International Conference on Learning Rep- resentations, 2023.
Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. arXiv preprint arXiv:2309.06657 2023.
Zhihan Liu, Miao Lu, Shenao Zhang, Boyi Liu, Hongyi Guo, Yingxiang Yang, Jose Blanchet, and Zhaoran Wang. Provably mitigating overoptimization in rlhf: Your sft loss is implicitly an adver- sarial regularizer. arXiv preprint urXiv:2405. 264226405.666,,, 2024.
Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguisties: Human language technologies, pp. 142-150, 2011.
13

---

Published as a conference paper at ICLR 2025
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforeement learning, nature, 5 518(7540):529--5555 2015.
2015.
Vivek Myers, Erdem Biyik, Nima Anari, and Dorsa Sadigh. Learning multimodal rewards from rankings. In Conference on Robot Learning, pp. 342-352. PMLR, 2022.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Malayandi Palan, Nicholas C Landolfi, Gleb Shevchuk, and Dorsa Sadigh. Learning reward func- tions by integrating human demonstrations and preferences. arXiv preprint arXiv:1906.000 2019.
Peter S Park, Simon Goldstein, Aidan O'Gara, Michael Chen, and Dan Hendrycks. Ai deception: A survey of examples, risks, and potential solutions. arXiv preprint arXiv:2308. (tt.:23....17552, 2023.
Ethan Perez, Sam Ringer, Kamilé Lukošiūté, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, et al. Discovering language model behaviors with model-written evaluations. arXiv preprint (rXi:22222.0925/ 2022.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. arXiv preprint arXiv:2305. 102000..82200 18290, 2023.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. SC20: International Conference for High Perfor- mance Computing, Networking. Storage and Analysis, pp. 1-16, 2020. doi: 10. 10.1109/SC41440 2020.00024.
Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 661-668. JMLR Workshop and Conference Proceedings, 2010.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications of the ACM, 64(9)999-106, 2021.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 0637777777477, 2017.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-383 3021, 2020.
Emanuel Todorov, Tom Frez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pp. 5026-5033. IEEE, 2012.
Masatoshi Uehara, Nathan Kallus, Jason D Lee, and Wen Sun. Refined calue-based offline rl under realizability and partial coverage. arXiv preprint arXiv:2302.02392 2023.
Runzhong Wang, Zhigang Hua, Gan Liu, Jiayi Zhang, Junchi Yan, Feng Qi, Shuang Yang, Jun Zhou, and Xiaokang Yang. A bi-level framework for learning to solve combinatorial optimization on graphs. Advances in Neural Information Processing Systems, 34:21453-2 2021.
Markus Wulfmeier, Peter Ondruska, and Ingmar Posner. Maximum entropy deep inverse reinforce- ment learning. arXiv preprint 150.::1507.0888 2015.
Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint, In Forty-first International Conference on Machine Learning, 2024.
14

---

Published as a conference paper at ICLR 2025
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for Ilm alignment? a comprehensive study. arXiv preprint arXiv:2404. 240.:29004.0079,, 2024.
Tengyu Xu, Zhe Wang, and Yingbin Liang. Improving sample complexity bounds for (natural) actor- critic algorithms. Advances in Neural Information Processing Systems, 33:4358-4330 2020.
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songlang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint NrXiv:2304.05302. 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? arXiv preprint arXiv:1905.07830 2019.
Siliang Zeng, Mingyi Hong, and Alfredo Garcia. Structural estimation of markov decision processes in high-dimensional state space with finite-time guarantees. arXiv preprint arXiv:2210.0/282, 2022a.
Siliang Zeng, Chenliang Li, Alfredo Garcia, and Mingyi Hong. Maximum-likelihonn inverse re- inforcement learning with finite-time guarantees. Advances in Neural Information Processing Systems, 2022b.
Zhengyuan Zhou, Michael Bloem, and Nicholas Bambos. Infinite time horizon maximum causal entropy inverse reinforcement learning. IEEE Transactions Automatic Control, 63(9): 2787-2802, 2017.
Banghua Zhu, Hiteshi Sharma, Felipe Vieira Frujeri, Shi Dong, Chenguang Zhu, Michael Jordan, and Jiantao Jiao. Fine-tuning language models with advantage-indum induced policy alignment. arXiv preprint 2rX.v:2306.02231, 2023.
Brian D) Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Carnegie Mellon University, 2010.
Brian D Ziebart, J Andrew Bagnell, and Anind K Dey. The principle of maximum causal entropy for estimating interacting processes. IEEE Transactions on Information Theory, 2013.
A APPENDIX
A.1 LITERATURE REVIEW
A.1.I REWARD LEARNING USING DEMONSTRATION DATA
In the RL literature, a line of work referred to as Inverse Reinforcement Learning (IRL) proposes jointly learn the reward and policy from expert demonstration data. Specifically, the target is to find the parameterized reward function r(s, a; () (resp. an optimal polity n (s, a)) that best explains (resp. mimies) an expert policy π given the demonstration data D. For example, the well-known maximum entropy IRL (MaxEnt-IRL) framework (Ziebart 2010: Ziebart et al.) 2013 Bloem & Bambos 2014 Zhou al. 2017) finds policy maximizing entropy subject the expected features that match the empirical averages in the experts observation dataset. However, this approach can only used to model linear rewards.
Subsequent works such as Levine et al. (2011) Wulfmeier et al. (2055) y Zeng et al. (2022b) further improve the MaxEnt-IRL. method so that nonlinear reward can be used. For example, ZZng et al. 2022a) proposed maximum likelihood IRL (ML-IRL) formulation based the Dynamic Discrete Choice (DDC) model, and a nonlinear reward function is utilized. Additionally, some works Liu & Zhu (2023 2022) extend IRL to multi-agent settings, broadening its applicability to scenarios involving multiple decision-makers.
It is also shown that when the reward function is linearly parameterized, then the MaxEnt is the Lagrangian dual of the ML-IRL problem (Zeng et al, 2022a). On the other hand, is worth men- tioning that best knowledge, almost all IRL-based methods only utilize demonstration
15

---

Published as a conference paper at ICLR 2025
data, which can be problematic because large amount of high-quality demonstration data is typi- cally hard to obtain. Further, it well-known that using demonstration only cannot extract precise human preference, especially in safety-related tasks where the boundaries between permissible and impermissible actions need to be precisely determined; see, e.g., .s..her et al. (202T) which shows that insufficient demonstration dataset could lead to high generalization
A.1.2 JOINT LEARNING FROM DEMONSTRATION AND PREFERENCE
Combining data from demonstrations and human feedback to achieve alignment has also been stud- ied in the robotics literature. In Ibarz et al. (2018), the authors first combine two approaches to learn from human feedback: expert demonstrations and trajectory preferences. The addition of demon- strations to learning from preferences typically results in substantial performance gains compared with using either demonstrations or preferences isolation. In Palan et al. (2019) and BByık et al. (022), the authors integrate diverse sources of human feedback including demonstrations and pair- wise comparisons a Bayesian approach to learn reward functions that are assumed to be linear in carefully selected features and evaluate their proposed method on robot learning platform. More- over, their proposed methods need to actively generate preference queries, which are expensive to collect in practical applications. In contrast, the approach proposed in this paper not Bayesian and does not include the requirement that the reward model is linear in pre-selected features.
A.1.3 OTHER APPROACHES TO ALIGNMENT
Other approaches to alignment include Direct Preference optimization (DPO) (Rafailov et al. 2023) and Inverse Preference Learning (IPL) (Hejna & Sadigh 2023) both remove the need for explicit reward modeling, and they directly extract the policy from preferences. This greatly reduced the training complexity, but has been observed that these algorithms can be unstable in the training process (Azar et al. 2023 Xu et al. 2024). There is also large number of works that aim to learn reward functions from rating (Daniel et al. 2014) or ranking (Yuan al., [023] Myers al.) 2022). Hong et al. (2024) proposed single-stage supervised learning algorithm ORPO that can perform supervised fine-tuning and preference alignment in one training session without maintaining. How- ever, all of these works highly rely high-quality human feedback, which is often more difficult and expensive to obtain.
A.2
EXPERIMENT SETUP AND ADDITIONAL RESULT
A.2.1 MUJoCo TASKS
In MuloCo, we consider several robotic control tasks with continuous action space. We evaluate the performance of our proposed algorithm on aligning robot behaviors with provided demonstrations and preference data. After the robot training stage, we leverage the ground-truth reward function from the environment to evaluate the performance.
Data. Following the similar data generation pipeline in Brown et al. (2019). generate the expert demonstrations and preference dataset as follows. We first train an expert agent by leveraging the ground-truth reward function and the popular Soft Actor-Critic (SAC) algorithm HHarnoja et al. (201)), which is developed to solve policy optimization problems with continuous action space. During the training process, we save the policy checkpoints and collect 30k samples from each checkpoint. To achieve precise control of dataset quality, we categorize the data collected into three different classes: low-, medium-, and high-quality datasets according the performance of the checkpoints. Then we combine the low- and medium-quality data as the preference dataset and use high-quality as demonstration data.
Results. We show that AIHF is able to integrate (insufficient amount of) demonstration data and (not-so-high-qualiig preference data to generate high-quality policy, and significantly outperforms the RLHF and IPI. Note that IPL. (Hejna & Sadigh) 2024) is actually the DPO-type algorithm applied to multi-horizon MDP, therefore it is an ideal baseline for the MuJoCo setting (since the underlying problem is a multi-horizon MDP). In Figure 5 we can see that AIHF outperforms both RLHF and IPL algorithms. We also observe that due to the limited number of demonstration data, even BC could provide high- perfoming initialization, RLHF still fails to improve policy quality in the following RL stage Ross & Bagnell (2010); Zeng et al. (20226). Moreover, since the preference data quality is only of lowto-mmedium quality, the RI. step based the learned reward
16

---

Published as a conference paper at ICLR 2025
1000
1000
100
The Number Policy Optimization Stopd*
Number of Policy Optimization Staps
The Number Policy Optimiza Stapa"
1.6 1.6
1.30
Figure 5: Top-Left: Hopper Environment: Top-Right: HalfCheetah Environment; Bottom: Walker2d Environment: AIHF (orange) vs RLHF (blue) vs IPL (purlle) (Hejna & Sadigh 2024); results are averaged over independent runs. We use 10k demonstrations and 20k preferences. The RLHF and IPL curve is initialized from a policy pre-trained by BC; the AIHF from a random policy. The performance is compared against the # of SAC steps performed (for AIHF each policy alignment performs 5k steps of SAC.)
model fails to significantly boost the fine-tuning performance. In contrast, clearly the proposed AIHF can effectively integrate preferences and demonstrations, leading more robust reward function and consequently, high-quality policy.
In SAC, both the policy network and Q network (64, 64) MLPs with ReLU activation function, and the step size is set to 3 10 we parameterize the reward function by a (64, 64) MLPs with Adam the optimizer, and the step size
ReLU activation function. For the reward network, is set to be * 10-1
The quality of the preference dataset and demonstration dataset listed in Tab. 1
Dataset
Non-prefer Data
Prefer Data
Demonstration Data
Task
Hopper-vi v2 HalfCheetah-v2
2345.20 + 329.93 93
3024.63 + 40.52
3559.61 + 73.12
7226.37 + 126.88
9434.42 + 1315.13
11635.42 + 236.51
Walker2d-v2
3952.60 + 451.55
5091.71 = 291.73
5453.41 +: 71.07
Table 1: The quality of preference and demonstration.
A.2.2 SENTIMENT-CONTROLE GENERATION
Dataset Generation: In the IMDb sentiment completion task, we generate the demonstrations and preference datasets using the following procedure. Initially, we train a Language Model by em- ploying the ground-truth reward function DISTILBERT-IMDE and the Proximal Policy Optimization (PPO) algorithm on 30% of the training dataset for IMDb. Throughout the training process, we save the policy checkpoint every 500 PPO steps. Subsequently, we select an additional 40% of the train- ing dataset and generate response for each prompt for each checkpoint. According to the evaluation score of each generation, we categorize the data collected into different classes: low-, medium-, and high-quality datasets, then we combine low-quality and medium-qualle as preference datasets, and use high-quality as demonstration datasets.
Training: After acquiring the preference and demonstration datasets, we train the proposed algo- rithm AIHF and baselines on the remaining 30% of prompts from the training dataset. We evaluate the performance of each algorithm using the test datascts for IMDb and HH, along with their corre- sponding ground truth reward functions. For the GPU resources, we use 8x A100 40G for all the experiments.
Policy Quality. We find that the proposed approach works well when either preference or demonstration data, or both, are limited. From the | we see that by using the same amount of data (10k preference, 10k demonstration). AIHF-based algorithms achieve faster convergence than their RL and DPO counterparts.
Results:
17

---

Published as a conference paper at ICLR 2025
Results: Distribution Mismatch. We show that AIHF is able alleviate the distribution mismatch between the generated trajectories by the policy, and the data that the learned reward model is able to rank. To evaluate the extend of such mismatch, we the following three steps: (1) use Ik preference, Ik demonstration to train policy and reward model for RLHF and AIHF (2) for a
given of prompts from test dataset, RLHF and AIHF perform generation; (3) use the trained reward models to rate the generation; (4) compare with the score generated by the ground truth reward WWWERR/DDSTILBERT-III produced by AIHF aligns closely with that of the ground truth reward, whereas that generated by RLHF exhibits a poor match. These results show that the reward model learned by AIHF is able to correctly evaluate the generation produced by the final policy.
Fig. illustrates that the reward score distribution
grundduuh
PRequenTy
homrlleee Score
No Normallodd Score
Figure 6: Comparison of the distribution of reward score generated by the trained reward models, and the ground truth reward model. RLHF vs ground truth (left); AIHF vs ground truth (right).
From Fig proposed algorithm AIHF could obtain higher rewards than baseline methods in the IMDb setting for almost all KL values. Although AIHF might get a low score from the ground truth reward model in the earlier step, AIHF would get higher reward with more iteration and optimization steps. This indicates that with the mix of demonstration data and preference data, we could prevent the policy from known issues of reward hacking, especially when the policy learned more human-aligned features beyond base models (high KL value). Moreover, AIHF is persistent in the number preference data, presenting that AIHF could still gain benefit from the limited preference data in more optimization steps as long as the demonstration data is high quality enough.
verpence
divegence
Figure 7: The frontier of expected reward vs KL the reference policy in IMDB dataset. fix the demonstration number to 3k Left: Using 1k preference; Middle: Using 2k preferences; Right: Using 3k preference
A.2.3 THE RESULT OF IB AND 2.8B EXPERIMENTS
Additional Results. We show here the distribution of the reward values of the continuation gener- ated by the reward models trained by AIHF and RLHE. In Fig. g we can observe the distribution of AIHF and RLHF have overlaps in low-quality continuation, however, AIHF can generate more high- quality continuations compared to RLHF, which shows that joint optimization can more effectively align the policy model with the demonstration distribution.
18

---

Published as a conference paper at ICLR 2025
AII
Normlizzd
Figure 8:
The Reward Distribution of Helpfulnes--controlledd Generation. Left: Result
160m model, Right: Results on 1B model, This figure reports the reward distribution of generation evaluated by PKU-Aliimmeen/baav=-bb3...... -.r--weard for AIHF and RLHE.
We record the performance of the two variants, namely AIHF-DPO (T0) and Self-Play-AIHE (II) also RLHF, DPO, IPO and SPIN in Tab. 5 Our proposed AIHF still outperforms all these methods in this experiment setting.
A.2.4 THE RESULT OF 7B EXPERIMENTS
We utilize DeepSpeed ZeRO-3 (Rajbhandari et al. 2020) to reduce the memory cost, and we VLLM (Kwon et al. 2023) for accelerate data generation/inference We use eight NVIDIA A A100- 40G do the training with per-device batch size of for 7B model. We train all models with 6 precision. We set the learning rate to he 5e-7 for the 7B model with the cosine learning rate scheduler, and the max sequence length is set to be 512.
Our device cannot conduct the implementation of the standard AIHF (5) using PPO. Therefore, we employed online DPO approach (Xiong et al. 2024) for the lower-level policy optimization: 1) at each policy optimization iteration, we first generate multiple continuations from the current pol- icy model over the given prompt dataset, 2) then we utilize our estimated explicit reward models to score each continuation, 3) over each prompt, choose the generated continuation with the highest reward score and the one with lowest reward score as a generated preference pair to run DPO algo- adopt the same
rithm to fine-tune the policy model. For Self-Play AIHF (II) strategy as Chen et al. (024), at each epoch, we generate samples with picked 50k data and generate continuation (-1ss) using the current model then optimize
implementation,
(II) with the sampled ā.
To evaluate the effectiveness of our approach, we assessed our fine-tuned models on the widely used HuggingFace Open LLM Leaderboard (Beeching et al. (023) in Table 2 We also list the metric and number of shots used for LLM evaluation on each dataset in Table 3 Moreover, in Tab. 4 we also utilize the Reward-Bende (Lambert al. 2024 evaluate the explicit reward models (estimated by standard preference learning in (3) and AIHF) and the implicit reward model (extracted from DPO).
The implicit reward is calculated by the policy model similar to what the DPO paper Rafailov al.
2023) proposed, i.e. r(a ({ == log( s(als), (( ), the first explicit reward is estimated by BTL reward model and second explicit reward is estimated by AIHF. The evaluation results show that AIHF can outperform the benchmark methods in terms of the performance for both policy models and reward models. We also conduct ablation study with different choice of wi in (1O), as shown in Tab. the improvement of joint learning methods over baseline is robust.
Dataset
Are Challenge
TruthfulQA MC2
Winogrande
GSM8K
HellaSwag
MMLU
Metric
strict-match
Num. of Shots
25
10
Table 2: A summarization of the benchmarks we use in this work. We list the metric and number of shots used for L.LM evaluation each dataset.
A.3 WHY AIHF CAN OUTPERFORM TWO- TWO-STAGE ALIGNMENT APPROACHES
In this section, we provide detailed analysis for Sec.
3.4
19

---

Published as a conference paper at ICLR 2025
Tasks
Are Challenge
TruthfulQA MC2
Winogrande
GSM8k
HellaSwug
MMLU
Avg
m1 serral-bb-ste -fttbeta
$4.69%
42.96%
77.27%:
39.88%
82.23%
59.72%
$9,464%
zephyr- bb-bet
59.64%
55.18%
77.82%
33.51%
84.19%
59.76%
61.68%
SPIN
58.45%
43.66%
78.30%
39.50%
83.59%
58.60%
60.35%
DPC
62.80%
53.17%
79.40%
39.20%
85. 85.13%
59.41% %
63.19%
IPO
58.02%
79.24%
42.91%
83.93%
60.07%
08%
AIHF-OPO
61.17%
60.03%
79.00%
39.80%
85.71%
60.02%
64.29%
Sel F-play ATEP
61.77%
58. 38.29%
78.53%
44.20%
85.53%
58.665
64.50%
63.90%
58.38%
79.24%
40.56%
86.23%
60.18%
64.75%
Table 3: Test performance of AIHF-DPO and Self-Play AIHF based on mistral-7b-sft-bett across
HuggingFace Open LLM Leaderboard datasets.
Reward Model
Chat
Chat Hard
Safety
Reasoning
Average
DPO Reward Model
37.43%
55.92%
64, 14%
47.33%
51.21%
BTL Reward Model
95.11 %
56.58% 55.37%
63.69% 63.98%
69.22%
71.15%
AIHF Reward Model
94.41%
76.75%
72.63%
Table 4: Evaluation of Reward Models in Reward-BBnnd
A.3.1 SFT AND RLHF POLICY
We revisit the RLHF pipeline in the context of simple softmax choice model where E A is selected with probability
(R) = exp(Ra/B)
(21)
Σ (xpp(R333)
For simplicity, we have defined R; = R(T;). Using this policy model, below let us analyze different policies according to different ways that the rewards are learned.
Supervised Fine- Fine-Tuning (SFT): Given a demonstration dataset D the goal is the find the policy TSFT that maximizes likelihood, i.e.:
N Σ 1=1
N
SFT = arg max
BROD [log कス (R)), s.t. क ( =1
(22)
Next let us identify the optimal reward and its corresponding policy in this case. First, let find the reward function. Write (2) in terms of reward optimization, we have:
N
cxp(Re/B) =: (R)
arg MaX Rd
E log Σ/
(23)
I.
The partial derivative of the objective function w.r.t. a component Ri, i [1, N] given by:
@L)R
Σ
BTAD
1
exp(Ra/3)
+ EE
exp(Ba/3)
0R:
E exp(pxp) R2/B)
3
Ej= exp(
E:lss
1) i
1
#(T
D)
Σ
E
exp( (Dp(R//3)
(24)
B
PI
Σ exp(R,/8)
Setting the above partial gradient to zero, obtain
# #
D}
cxp(RR((T)/3) N Σ (xpp(p(((T)///))
i [I:
(25)
D
where we used (T(T))ss to denote the optimal reward estimated from the set of SFT data D. Then according to the definition of the policy given earlier, we obtain
#(T, D)
|ミ (Rtp
Vie
20

---

Published as a conference paper at ICLR 2025
Self-Play-
Demonstaaions, preferences
0.463 0.003
0.048
0666
0.710
0.085
0,810
0.157
10% Demensirations 10s preferences
0.474 0.052
0.625 1 0.048
0.650
0.017
0.896
0.095
0.758 1. 0.028
0.574
0.096
1.190 0.009
Table 5: Evaluated reward scores of fine-tuned 1B Pythia models in HH dataset, evaluated by PKU- Aligmmen/bewver:-bv3300-eeeee w.r...
Are Challenge
TruthfulQA
Winogrande
GSMSK
HellaSwag
MMI.U
Avg
XIFF-DPO (w) 0.01)
61.86%
57.55%
79.08%
36.61%
85 85.58%
60.09%
63.46%
ATHP-DFC = 0.001;
63.25%
58.73%
79.16%
36.84%
85.59%
59.26/%
63. 63.80%
AIHE-DPO (ay 0.0001)
61.17%
60.03% %
79.00%
39.80%
85.71 %
60.02%
64.28%
Table Test performance of AIHF-DPO over different choices the balancing coefficient across HuggingFace Open LLM Leaderboard datasets.
Note that the optimal reward learned from the SFT data RD should satisfy the system (55 that there is unique solution the following system of equations:
It turns
# (T. E D} exp (xp(RD(T:)/B) D
iE (2.
(26)
exp ey=ee RoD (T)))/)
with (D(TT) Rí fixed reference value. A simple argument is provided below. The system 26 can be re-written as:
#(T #(n E ED) 1 (Ro(T.) Ri) i (2...., NJ
log
log
[D]
[
Hence, rewards are uniquely determined by demonstration data with fixed reference value Rp(1)) = Rg.
It is important to note that, despite the fact that in the SFT we may not directly learn reward function, the above analysis says that we are implicitly learning reward Rp, assuming that the policy is parameterized as (2I). This implicit reward will be used later to analyze the RLHF policy. Indeed, if solve the following SFT policy optimization problem directly
max Σ BRED log(me),
s.t.
Σ = 1,
(27)
RED
one can easily obtain that the optimal policy is given by o. # #T,ED) V Vie [1,...N] PD|
Reward Modeling with Preference Data: Now let us analyze the case where the reward only learned from preference dataset. With preference data P = {(T. > m,)}. the BTL model is:
P(r > T,) T;); (B(()) R(T,)) = +;( + + (R)
(
The reward estimation problem is defined as:
(i( log (R) 3 (R)
RP arg max L2(R)= = Σ B(
(28)
exp(Rr/3)
Σ
E log exp(Rr/3)) exp 13/3)
(29)
21

---

Published as a conference paper at ICLR 2025
Take the gradient w.r.t Ri, we obtain
1
(xp(Re/3)
((
RR:
3 3
exp(Rr/3) + exp( 12//3)
1
(pp(R//3)
E (ry+tt))
() + exp R3/3)
jij
1
P Pizsl
ا
(1
(xp(Re/5)
1 ト:
(P]
exp(Re/B) exp( R3/B) /B)
IP Paril
exp (9)
3
IP]
exp (3) + expl B2 R2/B)
1
IPozjl
IPa.l
exp (Re/)) exp(Re/ (3) + exp(R2//
3
|P|
|P|
1
T3
|Pr.l
(30)
B
|Ptll
(R( + rī. (R)
|P]
where in the above derivation we have defined (Paril. Setting the gradient to zero, we obtain
[Pi>jl # (T > Tj P} and |Pal == |Po- +
Pi-sl P
7 (Rp)
Pijl |P]
Σ J:
0.
(31)
(Rp) + xz (Rp)
We shall denote by Ri as a solution to the above system of equations. The first-order condition (3) can be written in implicit form as:
Ejsti IPinsl
E |Pirsl
Ejires (Piril
T. (Rp) Ejsei (Ro (Rp)
Pull
Σ Ejsti E-EEarres
Ensti (Pis] P- (0(0-(ts) (TT (Rpp)
(32)
where we have defined (+) == (1 -EEEEA mz) as the expected number of times an action other than T; or Ty selected when sampling actions from ı infinitely many times.
RLHF Policy. Based on the above two analysis, we are ready to analyze the RLHF policy. The
RLHF policy is defined as follows:
RLIF A
BT [Rp(Tt) - (T(IaS))
arg max REAM
where Rp is the estimator obtained from preference data, 7 ,FT is the SFT model trained with demonstration dataset D. and A is the probability simplex.
It can be easily shown that the solution -. RLLF is of the form:
T SST (T.) exp ((p(()))
TSPT((T.)
π ILHH
(Ti)
د SFT (7,) exp (p(()))))
exp 3 RD(T.) exp g/2 Rp((T)
(using (26))
N Σ== exp ( (T;) exp (pp()))))))
exp ( (PD(T)) + (T.():))
En exp ( (RD(TT) + (T))))
= = [No (Ro + Ptp)
(33)
22

---

Published as a conference paper at ICLR 2025
A.3.2 THE AIHF POLICY
The AIHF Policy. The proposed AIHF estimation problem is
RAIPF arg max (pppppp =
(34)
where (i( == EE o, (R) and L2(lt : [(r)<n)>>poggg
-,( [5) and 7. ( is
zi (n)
given by (2T).
Similarly as before, taking gradient w.r.t. Rú, and leverage 24 and (30), we obtain
deD+ R)
# in D}
(
D DI (R)DI +
Σ
|P--l
Σ jti
(Pal
(35)
m; +
JipZ
Setting the above condition to zero, we obtain that the AIHF reward should satisfy the following
#(T. in DJ +
Σ
|Payl= m. (RAIHF)
(ID)+
Σ
(RAIHF) + πz (RAIHP)) Ti
|Pa.l
83
j:
= m (RAIHE)
(ID) + Σ IPisle-(hs)) (FAIHE))
Or equivalently,
# TTe in D) Ejgs IPisil
TH RAIIF
(36)
[D] + Σ (Pij (-s))-().)) (6.1) (T" (RAIHF))
The system (36 has a unique solution ATHF with fixed reference value RATHF (T)) = R1.
Discussion. Now let us compare the RLHF policy 33 and AIHF policy (36 First, observe that RLHF policy is a function of the sum of two reward functions, one learned from the demonstration data and one learned from the preference data, while the AIHF policy takes a more complex form. To have better understanding about the two policies, let consider the extreme cases where the SFT and the preference datasets are unbalanced, where either [D] >> |P] or [P] (D].
Case [D] >> |P]. In this case, πं (RAIHF) xe #[r: D) क. (Ri), so the AIHF policy [D]
will ignore the preference dataset, while focusing on the SFT policy; however, for the RLHF policy, one needs first separately performs the reward estimation, but in this case Rp can be very noisy due lack of data; Then the RLHF policy will also be noisy since will be influenced by the noisy reward function Rp.
Case [P| > (D). In this case, let's further assume that Σ j:jzz (Pa-jl #(T in D}, V i, that is, for each trajectory тG the number of times it appears in the preference data is much larger than its appearance the demonstration data. In this case, it easy to see that x; (RAIHF) R π. (Rp), while the RLHF policy will still be noisy as in the first case.
A.3.3 NUMERICAL EXAMPLES
Example 1: With 3- and only two actions Ti and 72. Since (_(122) = = P-(2.1)(T) 1, 36) that:
follows from equations (26).
#(n in D) + ### #(22 x x i)
ATHF : (
[D]
+ [P]
D
PP
RD)
m (Rp).
D
D
+ P
Slightly abusing notations, let कं π: RR*)
where R is the ground- trouh reward. It follows that
=i---i) Var( (Rb)) = D Var( ( (Rp) (1-xi)
and
P
Var(T AIHP = π (1 .
min Var( min(Varr(T (Rp)), Var( (Rp)))
[ + P
23

---

Published as a conference paper at ICLR 2025
Hence, the AIHF policy estimate has less variance than either the policy estimated obtained from only demonstrations or preferences.
To further illustrate, suppose the ground truth is (m)))) = Rd (12) and in the dataset there are more
demonstrations than preferences, i.e. [D] > IP|:
#(n in DJ (T2 in D - 50, #( > 72 6,
# > TI in 4.
With the given data, T Tti (Rp) #(n in D
50
and the solution to (32) yields
100
exp (T(
6
ti Ti(ppp exp (m) exp (12)
(T)
10"
Hence, m1 AHH
100 To+
q (RD) + 10+100 qं (pp)
11o It follows from 33 that:
πं FT exp RT((T)
RLHI
"1
SFT exp (T) n SPT expfp(12)
"i
exp (T((TT)
6
exp Rp(T)) + exp (T2)
10
In this example, the RLHF policy estimator is the furthest away from ground-truth, because it does not correctly use the information provided by the demonstration data which in this case happens by chance to be correct SPT पं [t)
and in the dataset there As second example, again suppose the ground truth is R (TI) = Rt (T2) are more preferences than demonstrations, [P] [D]:
{1 in D) 6, {TT in D)- 4, { > in P} ( > in P} = 50. (37)
In this case, x #SFT .t (RD) and the solution 32 yields
cxp R
50
m (kpp)
cxp Rp(T)) + exp Rp((22
100"
follows from 33 that:
SFT exp RT((T)
RLHF
"1
SFT
exp RS (T) + no
exp (p(22)
SFT
6
π
SFT +
FT
10
Hence, T
56 In this example, the RLHF policy estimator is again farthest from ground-truth, because it does not correctly dismiss the information provided by the demonstration data which is less informative than preference.
10+ 100 Ti (R))
10
100 -ं (up)
100
110
Example 2: Let us use an illustrative example to show that RLHF method will result in signif- icant data under-utilization when the demonstration coverage limited. With 3 1, assume that there are 50 actions, i.c. A = 21, 50} and each with ground-truth reward defined -p) 20t
by (T(TT) = oVI="
where f = 0.5 and a = 2. Assume we can sample demonstra- tion and preference from the ground truth reward distribution: demonstrations are sampled from the multinomial distribution, while preferences are sampled from the BTL model.
In extreme scenario, let demonstrations only include actions through 45, i.e. D n (45, 46 ... 50] = Ø, while preferences have full coverage across all actions. In the subse- quent experiment, we initially sample 2000 demonstrations using the multinomial distribution र exp R
exp and obtain 200 preferences for each preference pair with P(i > j) exp texp X?Ç" We then calculate the RLHF and AIHF policies as in 33 and (36 to obtain the result depicted in Figure
Σ3 exp R
24

---

Published as a conference paper at ICLR 2025
AlFF RUNF
SFT
0.720
Ground Truth
Probability
0.726
0.724
0.722
0.220
0318
10
Action Index
Figure 9: The optimal policy of RLHF, SFT, AIHF, and Ground-truth distribution. The left region of the red dotted line is included in the demonstration, while the right region is uncovered. We report the results with 100 random repeats.
From the result shown in Figure 9] we demonstrate that both SFT and RLHF transfer the weight from uncovered action to covered actions when demonstration coverage is limited, as indicated by (TF 0, T; {45, 46 ...... .0}. Consequently, the weight of covered actions significantly higher than the ground truth. However, this issue does not occur when jointly optimizing the demon- stration and preference in the AIHF method.
A.4 PROOFS
A.4.1 USEFUL LEMMAS
Assumption (Ergodicity) For any policy n, assume the Markov chain with transition kernel P is irreducible and aperiodic under policy π. Then there exist constants K. >> 0 and p E (0, 1) such that
[[P((e E 1180 =) - -xx(-)lTVV S kpp', V120
sup SES
where || ||- -llvv is the total variation (TV) norm; is the stationary state distribution under æ.
Assumption assumes the Markov chain mixes at geometric rate. It is common assumption in the literature of RL, which holds for any time-homope soee-hooogeeeuuss Markov chain with finite-state space or any uniformly ergodic Markov chain with general-state space.
Assumption For any E S, A and any reward parameter 8, the the following holds:
|| az 0) - T.r,
(38a)
Var(s, a; 01) - V Ver 0z) I 1g 02
(38b)
where L, and Lg are positive constants.
we next provide the following Lipschitz properties:
Lemma A.1 Suppose Assumptions hold. For any reward parameter 0₁ and O2 the following
results hold:
12 50, (s,a) Qrost (s,a)) L, 0211.
vs S
(39a)
- S L. 021l
(39b)
where Q (---) denotes the soft Q-function under the reward function r(,;;0) and the policy ne- The positive constants Ly and L are defined Appendix A.4.3
A.4.2 PROOF OF LEMMA
Before we proceed to the proof of Lemma 4.1 we have the following remark.
25

---

Published as a conference paper at ICLR 2025
Remark: The KL-regularized MDP problem described by 5b) and 7 has closed-form solution:
x rx(als)
Ty (a|s)
(40)
Ea (ā exp((o(s.
where the corresponding value function Vo and the Q-function Qo are defined as below:
Vo(s) == Er-ry [((T0)) - 3 > r' DeL (1(1)))]](((((1)))) 80 = s)
(41a)
Qo(s,a) a) +yBB [Vo(s']
(41b)
Further, assuming that T 1, i.e., -= (so, ao). and considering the LLM alignment problem as a sequence-level training problem (this is a popular simplification in language models, see, e.g., Rafailov al. (2023)), the closed-form expression of Ty in (40) can be reduced to:
x (als) exp( r(s, a; 0))
no(als) EaEA (als) exp( a(s,,;
(42)
Here, under a reward parameter 8 and the corresponding optimal policy тo of (40). Moreover, under fixed reward parameter 0. have defined the optimal policy aa below:
To arg B,ANK Σ 7d (1(s,, 3DRL (=(-1s))1==((111)
According to Uehara et al. 2023), the optimal policy mo of has the closed form expression as below:
n (als) exp (o(saa0)
(a(s)))
3
VsES
(43)
EaeA T (ā]s) exp (ullsaa)))
Based on the closed form of ne. we can also obtain the closed form of Va as following:
(als) (o(s,a)
Vo(s)
3 log
Σ
(44)
8
BEAA
Then we can re-write the demonstration loss (0) as below:
(0) = EVEN
y' log mo(aelss)
π (arls)) exp (olssaa))
EfEnTE
7"t log
ErEA ( exp (olgena)
log (7 (a'(((st) exp @o(o),a))) log Σ
(ālst) exp @o(ot,a)
EmE
B
aeA
Qa(s),a)
20(5c, a)
EfEnTE
log n (ayls))
log
(āls.) exp
3 aed
Q0(se, à)
3 B
3 (log=((a.ls)) + Qu(s, ar) log
*((āse) exp
3
aEA
1
E, RouT
Blog " (a.lst) + Qo(se, Vo(ot)
(45)
26

---

Published as a conference paper at ICLR 2025
Then take gradient of (i( w.r.t the reward parameter 8, we have the following expression:
8
(0)
B.y ByRRggh
Voglog log x (ar(s:) + VoQu - -VoV((()))
VoVo(ss))
B
VoQo(st, - -VoVols)))-VVooos))
B,E
B
Br NNT
ac: ao -
B
1 Foom VoVo(so)
B. BNTR
(46)
In order to calculate the expression of VL, (0), we further derive the expression of V Vo)o
(Blog VoVe(so) Vo Blog
Qu(50. a)
Σ
x (also) exp
B
aEA t
(a) exp Qulso.a)
VoQo(s,a)
Eeea T (a|s0) exp
Q0(cp.0)
B
REA
1
VuQo(so,0) a)) (a
Ver(so: ao; + +Vooo(()))
E A TWT
Vor(st,
(47)
By plugging 47 into (46), obtain the following expression:
(O)
E. R
7 Vorr(st, ar:
R, Branno
(or((s)
(48)
3
A.4.3 PROOF OF LEMMA A.
To prove Lemma A.I we will prove the equality (39a) and the equality (3bb) respectively. The constants Lg and Le Lemma A.I has the expression:
L
2L CLg.1C |AI
2Lg 1-
Lg == 1-
La:= :=
+
1-7
A.4.4 PROOF OF INEQUALITY
39A
In this subsection, we prove the inequality (39a) in Lemma A.1
We show that Quaft has bounded gradient with respect to any reward parameter 6, then the inequality (39a) holds due to the mean value theorem. According to the soft Bellman equation, have shown the explicit expression of V (s, a) for any S and A. Using this
27

---

Published as a conference paper at ICLR 2025
expression, we have the following series of relations:
VaQ To (8, a)l =
B (10 Vor(so. a0: +V VoVo(s))
Σ
Ver(s 0)
(50, (8.00
120
(10)
xd Vor(se, ac: 0)
(5o, (80, ao) = (s,a)
t20
(bi))
x Bynno
Σ
7'th To (800:00 = -s,,a)
120
Lr 1 7
(49)
where (i) is from the equality (47 in the proof Lemma A.1 (ii) follows Jensen's 's inequality and (iii) follows the inequality (38a) in Assumption 2 To complete this proof, the mean value
theorem to show that
Qrop Ton a) - Qroy mog (s. $
max Vo (s, a)ll 1101 - 82ll $ Lg - 021 (50)
L. where the last inequality follows 49 and we denote by = I-7" Therefore, we have proved the Lipschitz continuous inequality in a3
A.4.5 PROOF INEQUALITY
39B
In this section, we prove the inequality (39b) in Lemma A.I
According to Lemma A.I the gradient VL, (0) is expressed as:
V (1) B
Σ (or
E
Σ
EvVor(ss
(51)
120
120
Using the above relation, we have
(0:) - VL (02)ll
(
E R
Σ
/Varr(s, 01)
Er
Σ -Vorr
$20
E
E
Σ
7" Vor(se, ari 02)
-
ETTweez
Σ
ar: 02)
120
120
E ET
Vor 01))))
"orrss
:=term
Braro
Σ (Ver(st,
> Ver(ss,
(52)
120
120
:=term B
where (i) follows the exact gradient expression in equation A and term B in (52).
5T
Then we separately analyze term
28

---

Published as a conference paper at ICLR 2025
For term A, it follows that
B Br~r Σ
Bra
Σ
E E
x Vor(s,,
(or(st,
(6))
E3 E
Σ
و'L 10
02ll
120
Lg 110 02ll
(53)
where (i) follows Jensen's 's inequality and (ii) is from (8bb in Assumption 2
For the term B, it holds that
E
(5) Er
Σ
7'
orr
120
B B
y Vor(se,
Var (ar
120
120
+ E B
7" Ver(ss (
-Brnsoo
Σ
y Ver aci (
120
120
(5t) 1
E B(aa) Vor(sn.
vor(s,,(;:1))
1 - ?
E
Vor(se Vor(s,,a1:0))
(181)
Ver(se,a;01)) d(s,a;Te.) d(s, az Tea))
+ Er
Σ k20
1gl101 E
021l
sES,aEd
2Lr
ld( No. d(., Te, TT, 1|r Lg
1 00 82ll
(54)
where (i) follows the triangle inequality, (ii) is from Jensen's inequality and the definition of the discounted state-action visitation measure d(s, a; T) == (1 - 7) (a|s) Σ 7'p= (st
8|80 ~ n);
(iii) is from (38h) in Assumption [ (iv) from 38a and the definition of the total variation norm.
Consider the L2 term:
(2 == (T. > [log (a (R(Twi:) 0) R(TG 0)))
where (x) sigmoid function defined by: (ar) = Ite-x. We have
Vol2(0) E B [[1 o [[(Tw; 0) R(T; 0))) R(Ti 0) VoR(T;))))
= -B(rrtt>>a"" [(Vo 0) V(R(T 0)) - a aaRo (R(Tw: 0) - R(T; (V R(Tw; V ((((T))))
Using the triangle inequality, obtain the following equation:
VV2(Twu n7, 7; (2 n7: 02)
E Errwwar (Vo VoR(T; (VoR(Tu: - Vo -V
=
(o(()))a R( ( VoR(T:
o (R(TTi; 02)
VoR(T:
(55)
:=term
29

---

Published as a conference paper at ICLR 2025
First we bound the term A of (55)
((( Σ
"VVro - (Σ
5 [Vor((" - 7 ''vrrss at: 02) )
term A =
120
120
Vor(s , ay - -1) +3 -) Xuw-
120
120
2Lg 10 021
(56)
1 7
Then we bounded term B of (55
term B
, 01) 7 (( 1)
()
-Voor- of:
120
120
Σ
r 02) x ('r( 02)
or
10 02) - (Vor af;
120
10 ,"r(s""00000 01)
~'r o(:
(r"""" - ~
120
120
r'((s, at 01) (s)
( a2; 02) - (orr
120
+
En'r( a.;
01) 01) ~'m (5)
Vor(s a 02) [
12
120
g'r(sp 02) , - 7/r(s), at:
"orr - ~
100
120
rr((s 01) / (r(((.d(:88)) di:b))
7"t asr (or(st
120
120
Vorrss w ai; 02) 7 Vor(s di 02)
||| |+|
Σ 7" Var(s -7:00 (or (vr(st,
120
120
V (or 30 ai; 02) 7 V V ( at 01) -a
7" Vor - Vor -
120
< 2Lg 10: 02 + 1
Lg ||0 021 = 3Lg
10 - 02ll
(57)
Plugging the inequalities
53
(54 to 52 it holds that
V ((22)l
< 2L,
+ 6Lg 110 - 021l
d(., Toa I|T
7
(0)
2L, Ca
6Lg g 110 - 821l
11gsott
+
1 1-7
(1)
2L, Cav [8)- AA
6Lg
-Gian nog 11x0 -- 110 02
7
(68)
(2 |
GT.g
110 - 0211-
(58)
1-7
Define the constant Le B.igh.Ca
5Lg T-y, we have the following inequality:
1-7
VL(0)) < Lo - 0211-
Therefore, we complete the proof of the inequality (39b) in Lemma A.I
30

---

Published as a conference paper at ICLR 2025
A.5 PROOF OF THEOREM 4.1
In this section, we prove 20a and (00b) respectively, to show the convergence of the lower-level problem and the upper-level problem.
A.5.1 PROOF OF (20A)
In this proof, we first show the convergence of the lower-level variable {TT Recall that we approximate the optimal policy no, by T each iteration k. We first analyze the approximation error between T0, and + as follows. For any S and A, we have the following relation:
log (TT+1 (a)s)) - log (To,, (a|s))
T: (a 8) exp (Qran (8,
.Tu x (als) exp (Q (s,
(
log
- log
Ea exp x0 (āls)a (ā]s) TT (s,
La 2 (ālss) exp (Qro 190z (s, (s,
(di)
uunft rag Mi (8, Va Fo (s, a)) + log Σ
n" (āls) exp (Ora (s, a))
log Σ R (āls) exp (Qron Tay (s, (s,ā))
(59)
where (i) follows (40): ; (ii) is by the triangle inequality. We further analyze the second term in (59
We first denote the operator log( exp(v)ll1) == log(I) EdEA exp(va)]lı) where the vector w, U E R|A) and v = [01, D2 ... v All. w = [wi, W2, ... W||| Then for any v', v" E RI we
have the following relation:
log (Ilw' exp(t')llı) log (Ilw" exp(x"))l1) ( (u' v", V log (Ilw exp(t))la) lu=o))
S ||o Ilo V. log (Ilw exp(x) (b) -=v ll
(1t) | 10o
(60)
where (i) follows the mean value theorem and ve a convex combination of o' and v"; (ii) follows the following equalities:
wy exp(1:)
[V log (xwxx))))))
V log ([/wex(ee)l))l==== We e RI
Σ LISaSIAl Wa cxp(va)'
Through plugging (00) into (59 holds that
log (T (als)) log (Te, (als))
2 me (s, a)
Qoait .5, (s, a)) max Q me (s, ā) aeA
soft to (+)a)
(61)
Taking the infinity norm over TR| [ the following result holds:
|| log M log Te 1l.0 S 21 Toy - Quan QTLL .To 11
(62)
where log Tkt1 log Te. lloc max log (a+i(als)) log Top (als) and ||Q .
Qsoft .To,
|1 max,eS, Qrof .e (s,a) Quott Fa, (s)
Based on the inequality 622 we analyze ||Que0 (Q
Quali
Illo to show the convergence of the Orag ma,
policy estimates. It leads to the following analysis:
-Qro .To. 11o0
= -11Qroft
-Qooft +
Qsoft rog 11
Osoft Coy
+ rop
1l
1.
11Q
- Qroof
1 all0 + ||Q
1022-1 11 + 1|Quan san
Qraft Qop_ 18
lloc
rog Jop
11Q A8
Qrozn
rog
T.g 10k 0k-1ll + | x|Q
Quoft
1l + 11Q
10aah .
Quon
-oe-, 1l,c + 221gg100
0k-1ll
(63)
31

---

Published as a conference paper at ICLR 2025
where (i) is from (39a] in Lemma A.1 (ii) follows (3aa). Based on 63 we further analyze the two
terms in 63 as below.
Recall we have the soft Bellman operator expressed as below:
Ta(Q)(s, a) = r(s, a; 0) + P. log <dd(d(s)) exp (Q(s', ('))
(64)
According to the soft Bellman operator, holds that
Qran -h-1 (s,a) rss, a; 0k) + p(ss. Vooo no m (s')]
log #h+ (a'ls')
= r 0x) + 7 (1lsa)))))) (100)
Qron Orgg (s', (s',
log ro (o((ass))
(
log + 1 (a' (s')
2 rss, az 0k) / a (1))))
+
Qrecim itte (5',
Q
log (o(a'
( r(s, a; 0k) pl
log Σ
x" (a') (s)) exp (Tra (5 (6,,
(ii))
To, (Qroft TT (8,
(65)
where (i) follows the policy improvement result(ii) follows the definition Th (als)
7 (als) exp (. (s.a))
(iii) follows the definition of the soft Bellman operator (64
Σa (a]e) exp (2th, (s.a))
For any S and A, it holds that
Qooft _8,, as)
Qray (s,
Qroy me, (s.a) To (Q (s, a)
(66)
where is due to the fact that mo, is the optimal policy under reward parameter 0&: (ii) from 65
Hence, it further leads to
(i)
soft rer
Softt 1
1lc
I
1
To (Q Ty 2l10
11
(it)
[T (Orog mon
Top (Qrol @ mlo0
(111) 711Quof
S
reL
Quai 1og
Iloe
(67)
where (i) is from (66): (ii) is from the fixed-point property in (3)) (iii) is from the contraction property 12 Therefore, we have the following result:
1 Qion -o
lloc
1
1 n0 llce + +2Lg -2gg100 10x
Qsoft ri
0k-1ll
(i)
I ? 10 Tk-1
Q
soft
softt
1l + +2Lgl10x - 0k- ,
(68)
where (i) is from
633; (ii) is from 67
To show the convergence of the soft Q-function based on (68), we further analyze the error between the reward parameters 0₂ and O--1. Recall in Alg I the updates in reward parameters (19
0: = 0g + ogjt-1
where we denote = {(s,, 0 h(0, 7) = Eiz0 o a,; 0) and 9k-1 is the stochastic
gradient estimator at iteration & - 1. Here, T denotes the trajectory sampled from the expert's dataset D at iteration A: and - denotes the trajectory sampled from the agent's policy m,
32

---

Published as a conference paper at ICLR 2025
time & - 1. T3 denote the trajectory sampled from the preference dataset. Then according to the inequality (38a) in Assumption 2 we could show that
1 || - [
TT-1) h(0 T T--1) 11 R-1) h(00-11, 1-11)
2Lr
2Lr
4hg
(69)
1 1-7
where the last equality follows the fact that we have defined the constant Ly further show that
Then we could
10 soft Orgia
1l
7
+112
rop 1 1l + 4Lg110x - 0x-1ll
Tirk T-
(12)
71 Quaat
re .- 11 + 4 4act,ll9n
(68)
<
?11Q
Quai
_ 1lo + Sa.i
(70)
where (i) is from 68); (ii) follows the reward update scheme; (iii) is from 669
Summing the inequality 770 from to k
K, holds that
K
K
Σ
1 T
Q Tug Tnee
80 S
1 Cog mm - -ran 1lc0 + 8aK L
(71)
k==
Rearranging the inequality 71 and divided (1 by on both sides, it holds that
Quont son 1 1lo0 S
7 K
11000
Qsoft rag
11
1| rerg RK
Q Pregi
11
+
80x
K
(72)
Dividing the constant - on both sides of (72). holds that
700
8L
11 Oore.m
sooft rop Ko
1l S
+ 1- a
K
mTk
K(1 - 7)
where we denote Co
11Quan
1
Quaft Too
Il
We could also write the inequality above
K
1Quen xx
Quoaf itoz
1|xc
K
k=0
7Co
Co
1Q
-raf -oo
Tog 1loc
8L2
+ a
K(1 7)
K
Co
8L2
1 1-7 a
K(1 -7)
Recall the stepsize is defined as a =
an K
where o O. Then we have the following result:
1
Qsont - 1lc0 =0(K-))+0L8 O(K-9)
(73)
K
k=
With the inequality 662 it follows that
2 K
K log Th+i log me, 1100
1
Qroll -ioll no,
10 +0(-)))((K-)))
Therefore, complete the proof of (00 in Theorem 4.1
33

---

Published as a conference paper at ICLR 2025
A.5.2 PROOF oF (20B
In this part, we prove the convergence reward parameters (0x F220-
We have the following result of the objective function L(0)):
(1) L(0x+1) 2 L(0%) + (VL(0)),0x++
Le
04)
10 - -2x12
2
L(Ok) + 9a)V Le07
11
-.(0)) + (V(V VI((O=)) + + Loa2
|1
(10) 2 L(OK) 9 = -8L 8L.L3a²
(74)
where (i) is from the Lipschitz smooth property in (3bb) of Lemma A. (ii) follows the update
scheme 199; (iii) is from constant bound in 669 Taking an expectation over the both sides of 774)
it holds that
E [[((+++1]]
[[L() + oE
(VL(Ok). VI((E)) +aB [L L2
[[((0=)) + oF
(VI(()), VI(Ox) (ol) aF |VT(0()))1
-8T, Tg
=E [L(6k) +o
VI((k); E
Σ
E
Voor(ss 0k))
200
Σ (1 ( 0z) / r(3// o,)))) a ;w) V Vor(sh of:
+ E(n
120
+oB
|1122)11
L.
(1)
2 [[(()>)]
4a E
Eynna Σ
E
Σ
En'Vor(s,
120
120
term
+ aE [V( - 8L.L3a0
(75)
(i) is due to the fact that ||V[()]] 4T. and Eigh, VoI2((+)10a) 0.
34

---

Published as a conference paper at ICLR 2025
Then we further analyze the term A below:
E
Σ
Ver
ErNTE++
Σ Vorrss 0x)
1
E
((e.a) [Vor(s,
E (a) Var(s, 0z)]
(1))
2
Vor(s, a; 0x)1 Billd(s - d max sES,aEA
1-y
(10)
2hr
$
[[]ld(;;,0,] dl::" (1==++)llll
1-7
(10)
π (a)s)) exp exp Qion mo (8,
x (als) exp Q me+1 (8,
$ 2L, C,E.
|
log
>a (ālss exp Qron t (5,
Ea (ā]s)) exp Qran te
(8, ④)
(9)
2 22, CaE
Qsoft ma + log Σ x (als) exp Qiam Fup (s, log Σ T (ā exp Qran ma+i (s.
(v))
(S) AAEE 11Qrog Toe
-Qratt Qoe .Th 20 + || rog
Quah Oreg
CL, S |A[E [1Q - To 11o0
(76)
(ii) is due distribution mismatch between (1 -1-77 ()](ss (iaals)EE E120 7(p"(s: s|so 1))
where (i) follows the definition d(s, a;
visitation measures; (iii) follows the inequal- ity (38a) in Assumption 2. the inequality (iv) follows Lemma A.2 and the fact that ma, (+18) x
x (-sss) exp (Quan
(er 10 (8,-)) T-11(-1s) x (-Is) exp (Qra (s,-)) and the constant Ly :=
(v) follows the (vi) follows the conversion between Frobenius norm and infinity norm
Through plugging the inequality (76) into 75 it leads to
[10++1)]
2 [[,() - 2
E
Brazen
7 Vor ori 0x)
120
120
of. [[VI(())111
1T
[[ - LaCa VISI
|A
soft 11
2ooft nx |1:0
+ alE
[|V
8LeL.a0
where (i) follows the inequality 76).
Rearranging the inequality above and denote C1 == 8CaLL ( -AA, it holds that
LE [[ < 8LoLaa² + aC Qnol n0,
Quail me |1co] + B[L(On+1) - L(Ow)]
Summing the inequality above from 0 K 1 and dividing both sides by aK, it holds that
C I - -Qran iTE 11 + [((k)) L(0))
E [[VL((W))]] 8L.LZ +
Ka
(77)
Note that the log-likelihood function L(Ok) is negative and L(0g) is a bounded constant. Then we could plug (7) into 77
it holds that
=(K->) 1= O(K-)t)
(78)
K K=0
which completes the proof for the inequality
20b
35

---

Published as a conference paper at ICLR 2025
A.6 AUXILIARY LEMMAS
Lemma A.2 et al. 2020 Lemma 3)) Consider the initialization distribution 17(-) and transition kernel P(-|s, a). Under vee) and P(-ls,a), a), denote doolts)) the state-action visitation distribution of MDP with the Boltzman policy parameterized by parameter w. Suppose Assumption 7 holds, for all policy parameter and w', we have
[(w((t))) dw (xv I Callw w''l
(79)
where Ca is positive constant.
Next, to facilitate analysis for KL-regularized MDPs, we introduce soft Bellman optimality oper- ator R R |A| [|S| follows:
log π (a'ls') log - (a' s')
((Q) a) r(s, a) PPBa))() max E (1)))(
(((ss,a')
(80)
m(-la)
In the following lemma, the properties of KL-regularized MDPs are characterized.
Lemma A.3 (The operator T defined in 80 satisfies the properties below:
TT has the following closed-form expression:
T(Q)(s,a) r(s, (7B.)))BBBPPPp... log Σ =ad(''))) exp (Q(s', (xp((())) (216.
(81)
T is a y-contraction in the foR norm, namely, for any Q1, Q2 E RIIII|AI it holds that
IT(Q1) > S Q1 Q2lloo
(82)
Under a given reward function r(-,-), the corresponding optimal soft Q-function Q a unique fixed point of the operator T, namely,
T(Q Quaf
(83)
We refine its analysis as below.
We first show that
Ea-x(1s) ((ss.a)
log (als) log T (a|s)
Σ T(als) log
T (als) (xs[ss))x((((sa)) (als)
( 2 log
Σ
π (als) exp (Q(s,
(84)
where (i) is from Jensen's inequality. Moreover, the equality between both sides of (i) holds when the policy ı has the expression (-(ss) & x (aaas)ex(((() Therefore, through applying the inequality (84) (80). obtains that
log Σ n (alss exp (Q(s', .d)) .
T(Q)(s,)) r(s, a) + (Bemp((aa))
(85)
which proves the equality 81).
We define ||Q: Q211 == max,eS, (Qi(s, (2((s,a) and c 11Q: Q2lloo. Then for any E S and A, follows that
log Σ क (als) exp (Qi
< log
Σ
n (als) exp (Qa(s, +6)
=log exp(e) Σ n (als) (Q2(s,
+ log
Σ n (als) (Q2(s.
36

---

Published as a conference paper at ICLR 2025
Similarly, it is easy to obtain that log (E, (als) exp (Qı(s,a)))
+
log EE, (als) exp (Qz(s, a))) Hence, it leads the contraction property that
[[T(Q)) < S 76 7 Q2 oo
(86)
which proves the contraction property (82). Moreover, have
log Σ x (a'ls') exp (Q (s', (e,,'')) ]
T(s, a(s,a) + (
Qioi (s, a)
(87)
where (i) follows the equality (85)
Based on the definition of the soft Q-function Q we have
log π (a'ls') 1 (a' + Q (s', a')]]-
Q (s, = r(s,a) + tB Bu' (16)
(88)
We prove the equality (ii) in (87 through combining 888 and the fact that the optimal soft policy has the closed form n² (-|s) x² (-1s") exp (Q (s. Suppose two different fixed points of the soft Bellman operator exist, then it contradicts with the contraction property in (86).
Hence, proved the uniqueness of the optimal soft Q-function Q Moreover, the optimal soft
Q-function Q is fixed point to the soft Bellman operator T in 83
37


Reviews:
[{"review_id": "amlsDWXtni", "reviewer": "Reviewer_7hXs", "summary": "This work introduces the Alignment with Integrated Human Feedback (AIHF) framework for learning a policy from both demonstrations and preferences. AIHF poses reward and policy learning as a single bi-level optimization problem where the outer loop maximizes optimizes the policy to fit the demonstrations and the reward to fit the preferences and the inner optimizes the policy with respect to the learned reward. The paper illustrates how AIHF connects to prior alignment algorithms and proposes a concrete instantiation of AIHF. Empirically, AIHF improves the alignment over standard RLHF.", "questions": "1. In Section 6, Why not compare all of AIHF, Self-Play-AIHF, and AIHF-DPO in Figures 2 and 3? Does the algorithm proposed in Section 4 empirically outperform AIHF-DPO and Self-Play-AIHF? \n\n1. Are the improvements of AIHF over the bsae Zephyr-Beta model significant? It appears the average improvement is only a couple of percent?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The single stage-learning of reward and policy from AIHF learns a more robust reward model that leverages both demonstrations and preferences compared to two stage approaches that first learn a reward function from only preference data.\n\n1. The paper demonstrates how the AIHF framework can be specialized to an RLHF, DPO, or self-play like approach. This shows that AIHF offers a more general alignment formulation.\n\n1. Section 3.4 theoretical and numerical evidence for why AIHF is superior to a two-stage alignment process like in standard RLHF.\n\n1. The paper provides performance guarantees for the proposed AIHF algorithm. \n\n1. AIHF outperforms RLHF and regular SFT on the Anthropic-HH dataset across several Pythia model sizes. \n\n1. AIHF with the DPO and Self-Play instantiation improves the performance of an RLHF model when trained with the Ultrafeedback-binary preference dataset and Ultrachat200k demonstration dataset.\n\n1. Results in the supplementary also show AIHF improves performance relative to RLHF for continuous control tasks.", "weaknesses": "1. The paper claims that AIHF outperforms existing alignment methods when the data is unbalanced (L78, L322), but this claim does not appear supported by the results in Section 5. The results in Figure 4 right show AIHF suffering as the preference and demonstrations become unbalanced. Contrary to the caption in Figure 4, these results also do not test if AIHF outperforms RLHF with different demonstration ratios since no RLHF result is displayed in Figure 4 right. This leaves it unclear if AIHF does have any benefit over existing alignment algorithms in unbalanced datasets.\n\n1. The evaluation of AIHF in Section 5 is hard to follow. Figure 2 evaluates the performance of the proposed AIHF algorithm from Section 4, but Figure 3 evaluates the DPO and Self-Play versions. This makes it difficult to evaluate the empirical significance of the AIHF algorithm proposed in Section 4.\n\n\n1. Insufficient empirical comparison to prior work. The paper does not evaluate the performance of the specialized variants of AIHF against the existing versions of the algorithms such as DPO and SPIN. These comparisons are crucial for evaluating the benefits of the AIHF framework. Additional comparisons to other existing such as IPO [1] would also strengthen the results.\n\n1. The paper does not clearly discuss the limitations of AIHF. \n\n1. The MuJoCo results in Appendix A.2.1 are missing important comparisons. Again, Figure 5 does not compare against existing alignment algorithms like DPO. Additional results comparing the quality and balance of this preference and demonstration data would also strengthen these results. This would confirm if RLHF is indeed suffering due to low-quality preference data as claimed. \n\nMinor:\n1. L476 should explicitly reference Figure 4. When first reading, it was unclear where this study was located.\n\n1. Figure 3 should provide exact numbers of the bars in the chart for more detailed empirical comparisons since many of the results are very close.\n\n[1] Calandriello, Daniele, et al. \"Human alignment of large language models through online preference optimisation.\" arXiv preprint arXiv:2403.08635 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kJF7m2kVq1", "reviewer": "Reviewer_mtpH", "summary": "The paper proposes a novel learning framework that provides a unified view for reinforcement learning, learning from demonstration, and preference learning. The key idea is to solve all three components simultaneously instead of following the staged approach of typical reinforcement learning with human feedback. The paper discusses many interesting insights, such as framing other algorithms as a special version of the proposed framework, insights into why the proposed method works better, and so on. The proposed work is validated mainly on LLM training problems, but also Mujoco simulated environments are also discussed in the appendix.", "questions": "I would appreciate it if the authors could resolve my questions above.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "* The paper presented a novel perspective for preference learning + reinforcement learning problems to approach them simultaneously rather than solving them as separate stages.\n* The paper proposed a practical learning algorithm and evaluated it on large-scale LLM data.\n* The paper provides numerous interesting insights.", "weaknesses": "* In my humble opinion, Section 3.4. WHY AIHF CAN OUTPERFORM TWO-STAGE ALIGNMENT APPROACHES can be improved. Overall, it discusses some mathematical reasons why AIHF works better than RLHF. However, I feel like it depends on several assumptions, such as |D| >> |P|. But is it always true? I always thought preference data was much cheaper than demonstration because it is only a yes/no binary question. \n* Also, eventually, RL will dominate, and it can achieve the desirable performance no matter what kinds of data are provided. \n* The section is not written concisely compared to its importance. I think it would be better if there was one paragraph that summarized general insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zevjJCtRwG", "reviewer": "Reviewer_Ewdq", "summary": "This paper addresses the limitations of current alignment methods, particularly highlighting that the reward model may not be sufficiently well-trained and that demonstrations contain additional information valuable for reward models. The authors then propose a joint learning framework for both reward and policy models to mitigate these issues, demonstrating that their approach can outperform existing RLHF solutions.", "questions": "1. The joint training is implemented using a shared parameter θ for both the policy and reward models, which is somewhat unclear. Would it be possible to decouple this into two independent parameters for each model, or are the two models intended to share parameters entirely? This distinction needs further clarification.\n\n2. Figure 2 presents a performance comparison of Pythia models with varying parameter sizes. As the number of parameters increases, the performance gap appears to narrow. Is there any further analysis or explanation provided for this observation?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper offers comprehensive theoretical proofs and thorough explanations.", "weaknesses": "1. Policy used during preference collection: It is unclear which policy is used during this phase. Based on my understanding, the LLM employs the SFT model to gather human feedback. Meanwhile, the proposed approach seems to assume that the preference dataset is pre-existing before training. In other words, please explicitly state which policy is used to generate samples for human feedback and clarify if a pre-existing preference dataset is assumed to be available or it will be generated during the training process.\n\n2. Unbalanced data claim: The claim that the proposed method performs better with unbalanced data is questionable. Typically, data preprocessing can effectively address this issue. Please provide more evidence supporting this claim, such as by comparing the method to baseline approaches that employ standard data preprocessing techniques for handling imbalanced datasets.\n\n3. Reward model improvement assumption: The framework assumes that incorporating demonstrations leads to a better-trained reward model. However, the paper lacks direct evidence showing that the reward model improves as a result.\n\n4. Effect of human feedback vs. demonstrations: In typical LLM training, the number of demonstrations is orders of magnitude larger than the human feedback dataset. The reviewer is concerned that, during joint training of the reward and policy models, the influence of human feedback might be diminished. Please discuss strategies for balancing the influence of demonstrations and human feedback during joint training, or provide experiments that illustrate the relative impact of each data source on the final model performance.\n\nThere are also a few minor issues with the paper that need attention:\n\n(1). The notation for human feedback data in line 099 is identical to the notation used for the trajectory in line 088, which may cause confusion.\n\n(2). Line 232 defines V_theta, but it does not seem to be utilized anywhere in the rest of the paper (except appendix).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YdqweoOID4", "reviewer": "Reviewer_1nbf", "summary": "This paper proposes a new framework that utilizes both demonstration data and preference data for better alignment. This idea is novel and interesting. Moreover, the paper also provides a general theoretical bi-level formulation that not only induces the proposed AIHF, but also can reduce to some major RLHF and IRL methods as special cases. The paper proposes an efficient single-loop algorithm to solve the bi-level optimization problem and theoretically guarantee the finite-time convergence of the proposed algorithm. Extensive empirical evaluations are provided to validate the effectiveness of the proposed method.", "questions": "1. In the introduction (lines 63-64), it is said that \"a joint approach to learning reward and policy models may improve alignment at\nthe expense of potentially significant additional computational effort\". At that time, I expected that this method would require additional demonstration data, compared to the standard two-stage RLHF method. However, Figure 1 shows that the demonstration data in AIHF is the demonstration data used for SFT in the standard RLHF, so that there is no additional demonstration data needed? Then I am not sure why the proposed method may potentially lead to additional computation. Of course, the computation is higher compared to RLHF using preference data only. However, RLHF also needs demonstration data to first compute SFT policy. If we compare \"AIHF\" and \"SFT+RLHF\", intuitively \"SFT+RLHF\" will be more computationally expensive because it uses the same amount data as AIHF and it solves two separate optimization problems. The counterpart of AIHF is not RLHF but RLHF+SFT, right? So that we need to compare AIHF and RLHF+SFT.\n\n2. In Section 3.4, it is shown that AIHF policy is somehow a weighted average of IRL policy from demonstrations and RLHF policy from preferences, therefore the AIHF policy reduces variance. I agree that this average can reduce variance. Suppose the demonstration needed for SFT is the same demonstration data in AIHF, the standard RLHF has a KL regularization $D_{KL}(\\pi||\\pi_{SFT})$ (which relates the learned policy $\\pi$ to the policy $\\pi_{SFT}$ learned from demonstration data). If we linearize the KL regularization, this may also lead to a (weight) average of $\\pi_{SFT}$ (demonstration) and the RLHF policy learned from preference?", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This paper proposes a novel and effective integration of IRL and RLHF to improvement alignment. I am actually very happy with this paper due to this novel integration and the associated theoretical framework. Moreover, I think that this paper opens a door for future research on the integration of IRL and RLHF for better alignment. The strengths of this paper include:\n1. Novel and interesting idea of the integration of IRL and RLHF for better alignment.\n2. A general bi-level formulation of this integration which can also reduce to some major IRL and RLHF methods as special cases.\n3. Excellent presentation where the authors explicitly deliver their ideas. More importantly, the authors provide insights to help readers better understand why the proposed framework can lead to better alignment. These insights in Section 3.4 are very helpful for readers to get an initial understanding of the advantages of the proposed framework.\n4. Solid theoretical guarantee for the proposed algorithm.\n\nIn general, I think that this paper can contribute to the RLHF and IRL community.", "weaknesses": "There is no obvious weakness of this paper. Please see questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)