Getting More Juice Out of the SFT Data: Reward Learning from Human Demonstration Improves SFT for LLM Alignment
Authors: Jiaxiang Li
Siliang Zeng
Hoi To Wai
Chenliang Li
Alfredo Garcia
Mingyi Hong
Venue: NeurIPS 2024 poster (2024)
Keywords: Large language models
Fine-tune
alignment
Reinforcement learning
Primary area: natural_language_processing
TL;DR: Reward Learning from Human Demonstration Improves SFT for LLM Alignment

Abstract: Aligning human preference and value is an important requirement for contemporary foundation models. State-of-the-art techniques such as Reinforcement Learning from Human Feedback (RLHF) often consist of two stages: 1) supervised fine-tuning (SFT), where the model is fine-tuned by learning from human demonstration data; 2) Preference learning, where preference data is used to learn a reward model, which is in turn used by a reinforcement learning (RL) step to fine-tune the model. Such reward model serves as a proxy to human preference, and it is critical to guide the RL step towards improving the model quality. In this work, we argue that the SFT stage significantly benefits from learning a reward model as well. Instead of using the human demonstration data directly via supervised learning, we propose to leverage an Inverse Reinforcement Learning (IRL) technique to {\it simultaneously} build an reward model and a policy model. This approach leads to new SFT algorithms that are not only efficient to implement, but are robust to the presence of low-quality supervised learning data. Moreover, we discover a connection between the proposed IRL based approach, and a recent line of works called Self-Play Fine-tune (SPIN, \cite{chen2024self}). Theoretically, we show that the proposed algorithms converge to the stationary solutions of the IRL problem. Empirically, we align 1B and 7B models using proposed methods and evaluate them on a reward benchmark model and the HuggingFace Open LLM Leaderboard. The proposed methods show significant performance improvement over existing SFT approaches. Our results indicate that it is beneficial to leverage reward learning throughout the entire alignment process. Our code is available at \url{https://github.com/JasonJiaxiangLi/Reward_learning_SFT}.

Getting More Juice Out of the SFT Data: Reward Learning from Human Demonstration Improves SFT for LLM Alignment
Jiaxiang Li University Minnesota Minneapolis, MN, USA 110037550umn, edu
Siliang Zeng University of Minnesota Minneapolis, MN, USA zeng01760umn. edu
Hoi-To Wai Chinese University of Hong Kong Hong Kong htwai@se. cuhk. edu. hk
Chenliang Li Texas A&M University College Station, TX, USA
Alfredo Gareia Texas A&M University College Station, TX, USA alfredo garcia@tamu.ed edu
Mingyi Hong
University of Minnesota Minneapolis, MN, USA mhong@umn.co edu
chenliangl @tamu. edu
Abstract
Aligning human preference and value is an important requirement for contemporary foundation models. State-of-th offtthe-rtt techniques such Reinforcement Learning from Human Feedback (RLHF) often consist of two stages: 1) supervised fine- mniig (SFT), where the model fine-tuned by learning from human demonstration data; 2) Preference learning, where preference data is used to learn reward model, which in turn used by a reinforcement learning (RL) step to fine-tune the model. Such reward model serves as proxy to human preference, and it is critical to guide the RL step towards improving the model quality. this work, we argue that the SFT stage significantly benefits from learning a reward model well. Instead of using the human demonstration data directly via supervised learning, propose leverage Inverse Reinforcement Learning (IRL) technique simultaneously build an reward model and policy model. This approach leads new SFT algorithms that are not only efficient to implement, but are robust to the presence of low-quality supervised learning data. Moreover, we discover connection between the proposed IRL. based approach, and recent line of works called Self-Play Fine-tune (SPIN, Chen et al. [2024])- Theoretically, we show that the proposed algorithms converge to the stationary solutions of the IRL problem. Empirically, we align IB and 7B models using proposed methods and evaluate them on reward benchmark model and the HuggingFace Open LLM Leaderboard. The proposed methods show significant performance improvement over existing SFT approaches. Our results indicate that beneficial to leverage reward learning throughout the entire alignment process. Our code is available at https:/ / github...... com/ www/Jasona Jason axi angLi/ wawgg//eewww..... learning_S
Introduction
Large Language Models (LL.Ms) have become the cornerstone of modern artificial intelligence applications. They are believed to lead the way towards artificial general intelligence [Bubeck et al., 2023], also have shown great capabilities towards specialized domains such math problem solving [Cobbe et al., 2021, Trinh et al., 2024, Wei et al., 2022, Lewkowycz et al., 2022], code generation [Chen al., 2021, Austin al., 2021, Li al., 2022], text generation [Anil et al., 2023, Touvron et al., 2023, Thoppilan et al., 2022], etc. Usually, researchers need to align the pre-trained LLMs
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

with certain exquisitely prepared human-labeled data to achieve desired performance over certain tasks, process which is thus known as alignment or fine-tuning The alignment datasets can be categorized into two classes: the demonstration data, with the input prompt and a human response; and the preference data, with the input prompt and two responses, where human labeler will pick a chosen one and a rejected one. With the alignment datasets, one could employ methods like supervised fine-tune (SFT, Ouyang et al. [2022]. Tunstall et al. [2023], Chung et al. [2024]) for aligning demonstration datasets, and reinforcement learning from human feedback (RLHF, Christiano et al. [2017], Ouyang et al. [2022]) and direct preference optimization (DPO, Rafailov et al. [2024]) for aligning preference datasets. More specifically, RLHF explicitly trains reward model and uses reinforcement learning (in particular, policy optimization) methods to obtain a fine-tuned version of the LLM; the other hand, DPO and many of its extensions simplifies the RLHF by training the LLM policy model directly, while implicitly learns the reward model via log of the ratio of likelihood between the learned model and reference model. In practice, both types of methods exhibit better performance over SFT on the demonstration datasets, and they are adopted by sfte-offthheart L.Ms,, for example ChatGPT benefited from RLHF (see Ouyang et al. [2022]), zephyr benefited from DPO (see Tunstall et al. [2023]).
It is interesting to observe that, when dealing with preference data, state-of-the-art methods usually build an (explicit or implicit) reward model to evaluate the quality of responses for a given prompt. On the contrary, typically case? One may argue that, for a given set of prompts, preference datasets contain explicit preference information which is not found in the demonstration datasets; since this kind of information is harder to extract, it motivates the use of complicated methods such as reward modeling. However, since human preferences are also implicit in the demonstration data, one can argue that training reward model that encodes human value distilled from these datasets may help to boost the alignment capability of the LLM. Indeed, in the RL literature, it is known that if the agents are given a set of demonstration data, then the so-called inverse RL methods (which learns the reward and policy simultaneously) can outperform the behavior cloning methods (which corresponds to supervised fine-tune in LLM alignments) by a large margin. In Markov decision process (MDP), it is likely that supervised learning methods which naively fit the demonstration data will suffer from the distribution shift problem - the fine-tuned policy from supervised learning can produce unsatisfactory generations in certain states which were unseen in the training dataset [Ross et al., 2011]. Through formulating the learning from demonstration problem in a MDP setting, typical inverse reinforcement learning methods [Ziebart et al., 2008, Ross et al., 201 1, Zeng et al., 2022] can alleviate such distribution shift issues. Witnessing the success in ChatGPT, where the aligmment of LLMs is modelled in the MDP setting due to the auto-regressive process, one would expect that the LLM alignment with demonstration datasets can be improved as well through deploying imitation learning / inverse reinforcement learning methods.
reward modeling is done for demonstration datasets. Why this is the
Inspired by the above observation, we pose the following question:
Does building a reward model using the demonstration data benefit the alignment process?
Contribution of this work. This paper answers the above question affirmatively. Specifically, by developing a framework based on certain IRL technique, we show that building a reward from demonstration datasets can significantly improve the quality of the resulting model, as compared to that obtained by standard reward- -fre SFT (1). Our main contributions are listed as below:
We develop a new reward-basce SFT approach, which takes the form of a bilevel optimization, where in the lower-level, LLM policy is learned via policy optimization for given reward, while in the upper-level, the reward model is optimized so to maximize the likelihood for observing the demonstration data.
Based on the above formulation, we propose two alignment algorithms, one learns the reward model explicitly, and the other implicitly. For the first algorithm, we show that the reward learned from only demonstration data already possesses strong capabilities in distinguishing between chosen and rejected responses; see Figure and our experiment for details. For the second algorithm, we made an interesting observation that implicitly learning a reward is equivalent to improving the model by comparing the demonstration data with the synthetic data generated by the past models. Somewhat surprisingly, the resulting algorithm is closely related to the self-play fine-tune (SPIN, Chen et al. (2024]) algorithm, recently proposed from a completely different viewpoint. is worth pointing out that unlike SPIN, our proposed algorithms have finite-time convergence guarantees.
2

---

We demonstrate the power of the proposed approach theoretically and numerically. We prove that our implicit reward learning algorithm converges to some stationary point of our proposed formulation. We show that the proposed algorithms outperform vanilla SFT in almost all cases we have tested, for example the model performance on HuggingFace Open L.LM Leaderboard increases from 59.47% % to 61.03% To our knowledge, this is the first work that formally demonstrate the power of reward learning when dealing with demonstration data for LLM alignment.
Inonstration
Policy (LLM)
SFT
Sehavior Cloning
0.30
Policy
Log. pww.alttltyyyyommmmmm gap
Reward
(LUM)
0.06
RFT
SFTJoggts_go
RFT. notis
IRFT Jogissgts_go0
0.02
Demonstration
Policy (LLM)
SelF generatiiiioo dradient
IRFT
-0.02
(non-grefferee data)
0.00
0.25
0.30
0.75
1.00
1.25
3.50
1.75
2.00
Training epachs
Figure 1: Left: Difference between SFT and the two proposed methods: RFT (Algorithm 1) and IRFT (Algorithm 2): Right: Log probability gap between the chosen/preferred continuation and the sencteed/non--referree continuations for different methods. All methods only consume the chosen/preferred data, but RFT and IRFT can effectively distinguish between chosen and rejected continuations; see Example in Sec. 3 for the detailed settings.
Notations. We use Tylaz) to denote the LLM output probability for continuation y with input prompt v, and we refer to as the policy. We use the notation It (y)2:; 0) if the model is directly parameterized by parameters 0. For the case when и is indirectly determined by parameter 6 we use notation To (z). We use D {xx y)} to denote the demonstration dataset and P {(x, thes 9:) for the preference dataset, where yu is preferred over yt. Since we assume that the demonstration continuations y are collected from human expert distribution, & P, ~ π (+12) when taking the expectations, where is the distribution of the input prompts when collecting the data. We similarly have the notation p: (y yhe) n (-(ar) for the preference dataset.
also denote (x, y) D as
2 Preliminaries
Consider a Large Language Model (LLM) parameterized by and denote the output probability by <[y|æ;; 9) where sequence of output continuation. Typical LLM is an auto-regressive model, meaning that predicts the output probability of the yj given all tokens in and vıj == [y,,..., Øy-1] (y<r is null), i.e.
..., ..., Fx is the sequence of input prompts and y 191, ... Vml is the
<y]:; 0)
II <u j=1
In this paper, do not focus the architecture design of LLMs. We will fix the LLM architecture and always denote it as probability model Tyyk:; 0). The following discussions review two common procedures for fine-tuning 0: (1) supervised fine tuning (SFT) over demonstration dataset, (2) reinforcement learning with human feedback (RLHF) over preference dataset that consists of two steps: LLM gregmment/iinettiinng based on reward model using policy optimization; and reward learning process to learn the optimal reward for the preference dataset.
SFT. Given a demonstration dataset D = {xx,)) the SFT optimizes the following problem:
max ŞRFT ()) := (m [log (y(x;00) 0
It is easy to see that the above problem shares the same optimal solutions as ming Exsp (T (+|x) ||T (-|æ; 0))] The latter shows that SFT aims imitating the demon- stration dataset via minimizing the KL divergence. It is worth noting that the SFT stage described
3

---

here is closely related to the imitation learning approach used in the RL literature for learning from demonstration [Osa et al., 2018], whose goal is to mimic the policy of an expert.
RLHE Suppose that we have reward model r(c y; Φ) (parameterized by & and to be defined later) for any given input and output pair (Ir v). the LLM can be fine tuned by the following RL problem:
max LRL (+) == -
[r Φ)] Ex=p|PRL( (-|2; 0) TTel (-1z))),
(2)
where Tref is fixed reference model. Note that the KL regularization term in (2) is not computable
given the sheer amount of possible output y (which could be corpus_siee _sizee max
sizzmmasssquuenee sequence length in most
language model tasks), therefore (2) is usually solved by standard policy optimization techniques such as REINFORCE [Ahmadian et al., 2024] or PPO [Schulman et al., 2017].
To find an appropriate reward model r(z, y: Φ), RLHF (see e.g., Christiano et al. [2017]) leverages a set of preference dataset P := {z, Yuu, st)]. where each data contains a pair of output you Ii, and Yo is preferred over yi by human labeler (denoted as yeu > yt). The Bradley-Tery model [Bradley and Terry, 1952] assumes that the probability of choosing yo over y/ is
exp(r()w;;a)
IP (yw > Yt ax) exp(r()wix)) + exp (r (Mii=)
(r(Hwir) r ())
One could formulate the following problem to find the reward model:
max (RM(()) := Ex-p, (ww)-=P((11)) log (a(((z, yue u) r(xx yn +))
(3)
+
It is widely observed in the literature that, models trained via episodically learning the policy (2) and learning the reward (3) typically outperforms those that are only trained using SFT [Ouyang et al., 2022]. The reward model guides the performance of the LLM and allows better generalization ability via the consistent input of the preference data from human labeler. Follow up works such as DPO proposes to incorporate reward learning implicitly by utilizing the structure of the optimal solution of the RL problem (2); for more details about the DPO, see Rafailov et al. [2024].
Discussion. At this point, let us take a step back and think about the above process. The LLM alignment problem takes human labeled demonstration and preference data to produce an aligned model. Clearly, both kinds of data encode information about how human would like the LLM output to be, but the processes of extracting such information is quite different (i.e., supervised learning vs RL). A series of questions naturally arises: Is supervised learning the best way to extract human inclination from the demonstration data? Can we also learn a reward model from the demonstration data to gauge human preference? Will policy model learned via such reward improve the supervised learning approach? In the next section, we will dive deep to carefully address these questions.
3 Reward Learning and Policy Fine Tuning from Demonstration Data
In this section, argue that reward learning from the demonstration dataset can benefit the LLM alignment problem. To do so, we develop a joint reward learning and policy fine tuning formulation and understand its capabilities in improving the LLM policy. The new formulation inspired to develop two reward learning paradigms: i) Explicit reward learning, where a (parameterized) reward model is learned together with the language model policy, and iř) Implicit reward learning, where the reward model is learned implicitly through directly optimizing the policy, avoiding learning two models simultaneously.
3.1 Joint Reward-learning and Policy Fine-tuning by Inverse RL
A challenge with learning only from the demonstration dataset is that the Bradley-Terry model (3) no longer be used due the lack of pairs of preference data. However, all is not lost as we recall that it is the value of the reward model that should be used in the fine-tuning process (2). Therefore, with only demonstration data DD, reasonable formulation is to combine the supervised learning problem (1) with the optimal policy generation problem (2), by requiring that the generated policy to 'match' with the demonstration dataset. With this intuition in mind, we consider the joint reward and policy learning problem via a maximum likelihood inverse reinforcement learning (ML-IRL) formulation [Ziebart et al., 2008, 2013, Zeng et al., 2022]:
max {(0) := Exxp.y (+1=)) [log To (y x)]]
0
(4)
s.t. no = arg max E Ey-n [r (2. y; 0) 33DLL
n

---

The above problem has bilevel structure which trains a reward model (z, V: 0). At the upper level, its objective is similar that of SFT (1), but is evaluated on the policy To induced by the reward model r (2r, y; 0); meanwhile, this policy is found in the lower level using the RI. objective (2).
There are several advantages of the bilevel formulation (4) over standard SFT (1). First, we notice formulating SFT as RL/ IRL problem can alleviate distribution shift and improve the generalization power [Ross al., 2011]. I]. In fact, we observe that (4) tends to give less extreme policy even when the demonstration dataset is extreme. The latter is observed in the following stylized example.
Example 1. Suppose we have only one state (input prompt) x and three actions (continuations) 91. U2. Va- Let the reference model Ted be a uniform distribution over all continuations, and the demonstration dataset is D {ys ). One could easily compute the optimal solution for (1) and (4) by first-order optimality conditions. From Table we can see that SFT (imitation learning) pushes all the likelihood toward the demonstration dataset, whereas ML-IRL (4) maintains non-zero weights for unseen data in the demonstration datasets. This is particular useful when we want to fine-tune from a
pre-trained model, which is presumed to be powerful and have useful information already.
Second, since the lower level problem in (4) encapsulates a generation process, it is antici- pated that the proposed method can better distin- guish between the preferred and non roe-preferred data than SFT, even it is only trained on the
Action
1/1
1/2
1/3
Tref D
0.33
0.33
0.33
{va)
demonstration dataset. The following numerical example highlights this point:
TSFT
0.0
0.0
1.0
2
2
R/8
Example 2. We compare the solution of SFT (I) and IRL (4) numerically, where the latter is solved using two algorithms RFT and IRFT (to be introduced shortly). We choose the prefer- ence based dataset Anthropic- HH and only keep the preferred continuation to form a demonstra- tion dataset D = {< } to implement SFT and IRL. We then compute the log probability
TIRI
24e
2+e
Table 1: A state-less counter-example with three actions where IRL-based fine-tune (4) shows regu- larization effect over SFT (I) to maintain weights over unseen data in the demonstration dataset D.
Here we assume r E [0, R).
gap (yg [z)) log( π (v][z)) between the preferred ye and non- non-preferred yn on the test dataset; see Figure 1 right side. We observe that although all three methods are not exposed the non- preferred data y/ during the training process, the IRL-based methods effectively distinguish the preferred continuation over the non-preferred one, while SFT assigns larger probability to the non- preferred continuation (see Section 5 for the details of the implementation).
Comparing to SFT (1), the bilevel problem (4) appears to be more complicated. In particular, solving standard bilevel optimization problem typically involves computation of Hessian matrices, which is too expensive for LLM related applications [Liu et al., 2023]. Fortunately, in our next result, we show that the bilevel problem can be significantly simplified (proof in Appendix B):
Lemma 3.1. Problem (4) is equivalent to the following minimax optimization problem:
(xx y: 0) r(z, v: 0) + DKLL (
max min E Epnp
(5)
3
The above reformulation is remarkable. First, minimax problem is much easier to solve as compared with bilevel problem, e.g., a simple alternating minimization can yield reasonably good solution; see Algorithm for such an algorithm, and Sec. 3.3 for its theoretical analysis. More importantly, shows that even only the demonstration data is available, the reward optimization problem takes similar form as what has been used in RLHF (3), where not one but two reward functions contrasted. The key difference here is that one reward is evaluated on the continuation y in D, the other is evaluated believe that such contrast is the key reason that enables the IRL based formulation to distinguish the preferred continuations over the prefferrd ones; see Example 2 and Figure 1.
which is the continuation generated from the current policy x(-[x). We
Now that we have turned the original bilevel problem (4) into a minimax optimization problem (5), we can naturally develop a gradient sesffent-deseen-asceen type algorithm for (5), which alternates between updating the policy according to the current reward, and updating the reward based on the current policy an algorithm that we call Reward-learninn Fine-tune (RFT), see Algorithm 1. Note that in
5

---

Algorithm 1: Reward-learning Fine-Tune (RFT)
Input: Initialize reward parameter 00(0-1.K 1.K = 0o) and policy model x², the stepsize reward update 7/2, and T, K the outer and inner iterations.
for t = 0, 1.......
do
Take 0._0 = 1.K
Data Sample: Sample state Ii,k p, expert response Ye I (- and agent response Ur. ~ x (x), for = 0, 1, ...,
for k 0, ,..,,, K do
Estimate Gradient: Calculate the stochastic gradient Ot.k w.r.t. A via
A = Vor Me 0(,,)) - 3 Ver(Tr D1,A: 01,k)
Reward Alignment: 0,,,+1 == 0e. + 1e9t.
end for
Policy Alignment: Update the optimal x (y((z) x exp(r(æ, y; k)) according to (9)
end for
the data sampling step, we sample the response from the current model for the next K iterations. If data size * epoch, the sampling process would be done for every iteration. In batch size.
we take K = I and 7 =
practice however we take relative small T and large K, because frequent on-line sampling is time consuming; see Section 4 for the implementation details.
3.2 Implicit Reward-learning Fine-tuning via Self-generation
So far we have seen that (4) (equivalently (5)) can efficiently utilize the demonstration dataset for better alignment. However, the computation cost for training two models (reward and policy) is significantly higher than the standard SFT. It turns out that (4) can be simplified into a supervised learning problem. Observe the following property (see Appendix B for proof):
Lemma 3.2. For the loss function in (4), we have:
To (ylax Teat (y|zr)
To (z)z)
Vol(O) -EENppy (11))
Ve Velog log
Vo log
(6)
Toof (1)(z)
The proof the above lemma uses the identity r(x, y: 0) = 3 log <
(p))((( + 3 log Zo(x) for some
constants Zø(r); see, e.g. Rafailov et al. [2024]. Again it is remarkable that, despite the fact that the IRL formulation only consumes the demonstration data D = x, y, the gradient of the IRL loss takes the form the difference of two gradients, one related to the demonstration data, the other related to the data generated by the current policy.
Lemma 3.2 leads to simple scheme for implicit reward- saaarddbasedddde supervised fine-tune (IRFT) for each training batch, it samples the response from the current model, and construct the gradient estimator (6) directly to update the parameters O. This results in Algorithm 2, which is an SGD type algorithm for (4). In Algorithm 2, we use double loop since generation at each step might again significantly take more time, similar to Algorithm 1. If K = 1 we get single loop algorithm where we generate for every training step on the input batch.
3.3 Convergence Theory
We conclude the section by theoretically inspecting the proposed algorithms. Note that details and proof's of convergence theorem are moved to Appendix B due to page limits. We observe:
Theorem 3.1. Under Assumption B. for Algorithm and 2 with ny = 0(1VTTK) we have
min
BlI[V(((..)))111 O
Theorem 3.1 indicates that the convergence dependency is (1/ ((1/VKT)) KT) (assuming T > K), which indicates that the algorithm could converge stationary point if we take both the inner loop and outer loop reasonably large. This is slightly contrary to the intuition since with larger inner loop number K, we having more biased estimators. This theorem shows that this biasedness actually wouldn's 't harm the final convergence, thus validate our practice of having relative large inner loop number K in practice (since generating at each training iteration is time-comeene tmme-oonsuming..

---

Algorithm 2: Implicit Reward-learning Fine-Tune (IRFT)
Input: Initialize model parameter 00(0-1,K 0o). the stepsize of reward update ne. and T, K the outer and inner iterations.
2: Output: 0
3: for = 0, - do
4: Take 0.,0 = 0,-1,K
5: Data Sample: Sample state It,A ~ p. an expert response vt 7 (- and agent response Dt.k To. (- (3),a) ), for k 0, U,,,.... K -
6: for k = 0, 1, ,..., K do
7:
Estimate Gradient: Calculate the stochastic estimator (e(O,,,) via (6), i.e.
(e(Or,,)) = Vo.,. log Te,, (nn.))
-V.,, log =o,, (z.))
8: Implicit Reward Alignment: Update 02, - Os, + 1/ ((,k)
9: end for
10:
end for
4 Discussions
Implementation details of RFT. As mentioned, training a reward model and a policy at the same time is costly. In our experiments, we discovered that the reward alignment step can be completely separated from the policy alignment step. In particular, we take T -==1 1 and K data size epoch batch
so that we train the reward over the entire dataset and then switch to the policy alignment. In our experiments, we indeed observe that only one round of above procedure can readily show superior performance over SFT and implicit reward-learning methods for pythia-1... 4b model.
Implementation details of IRFT. It is worth noticing that in (6), the policy π is not parameterized by O directly. our numerical experiment, we directly parameterize the LLM by 0, making (6) the gradient of an supervised optimization problem itself. Meanwhile, it is not straightforward to calculate the self- generatieen gradient (6) directly, thus we need to design a loss function for back- propaggttni in main-stream packages such as Py PyTorc and TensorFlow. In practice, at each training iteration we first sample y ~ π (-|[:; 0) and pass the following loss function
π (v] 9) log
T (ū 0)
h log
Troe (y)z)
Trat (ū
into the standard optimizers (such as SGD or Adam) for back pack-propagation. Here h is nonlinear function. We take = log a where a the logistic loss function (l) == log(1 + exp(-()) as in Rafailov et al. [2024], Chen et al. [2024] for its non-negativity, smoothness and exponentially decaying tail to avoid excessive growth in the absolute value of the log-likelihood.
Discussion on the computational costs. For Algorithm 1, we need to maintain a reward model and policy model (which is the LLM), and this is doubling the standard LLM fine-tuning. Thus the memory consumption and computation time of Algorithm 1 is similar to the standard RLHF process (RLHF = reward learning policy optimization); For Algorithm 2, we simply need to maintain the policy (LLM) model, and the memory consumption would be exactly the same as the standard SFT. whereas the computation time would involving generating for the entire training sample, which would be of similar level as the standard policy optimization process (same computational time as SPIN). Note that standard policy optimization process equivalent to the time of standard SFT and a generation process toward all training input prompts.
We summarize the memory consump- tion and the computational time of the proposed methods in Table assum- ing that the reward and policy mod- els are of same size. Here "Forward" means the memory required for stor- ing model in inference mode, and "Backward" is the memory required for storing a model training mode, including weights, activations and gradients; also "SFT" means the computational time standard
Method
Peak Memory
Computation Time
Algorithm
Borward+Backward
2SFT+Generation
Algorithm
Backward
SFT+Generation
Table 2: Table summarizing the computational costs of pro- posed methods.
7

---

SFT, and "Generation" means the time to generate continuations for each of the input training prompts. Therefore "2SFT+ is roughly the same time as standard RLHF.
Comparison to SPIN. We discuss here the connection between our proposed algorithms with the self-play fine-tune algorithm (SPIN in Chen et al. [2024]), which also maximizes the gap between two rewards. First, SPIN is motivated by certain two-player games, while in our case, we show that the difference of two rewards in (5) naturally comes from a single, reward learning agent; see (4).
Second, IRFT covers SPIN as a special case. In particular, if we take T = and K as the total number of training iterations, the IRFT algorithm is equivalent to SPIN. In practice, we tested on different choices of T and show that a reasonable generation frequency can results a strong model performance.
Finally, since SPIN does not involve explicit reward learning, its connection to RFT is relatively remote. It is worth noting that the relation between the proposed Algorithm 1 and Algorithm 2 is similar to that of RLHF to DPO. There has been intensive discussions regarding whether reward-based or reward-frres free algorithm gives better model performances, but this topic is beyond the scope of the current paper. We refer to Xu et al. [2024] for a comprehensive study.
5 Numerical experiments
In this section we study the proposed Algorithm and 2 numerically. Our experiment mainly show the advantages of the proposed methods in the following aspects: (1) Reward learning is key to improve over standard SFT, even if we do not have preference dataset; (2) The double loop design in both Algorithm and enable us to explore appropriate parameter settings that could break the performance limits of the state-of-the-ane methods, including SFT and SPIN.
5.1 Experiment Setup
Model and Datasets. Since reward-based methods be costly by training two models at the same time, we mainly test Algorithm 1 on pythia-1b reward model and pythia-1. .b policy model [Biderman et al., 2023]. We tested pythia on Anthropic-HH dataset (Bai et al., 2022]. Anthropic-HH is a preference dataset that provide two continuations based on helpfulness and harmlessness, and we only pick 10k chosen/preferree continuation data to form the demonstration dataset, which enable us to check the log likelihood of the non- non-preferred continuation without feeding the model with such data. At each iteration, we train our model for 2 epochs (seeing each data for two times).
Algorithm 2 is tested on two models: pythia-1 1ichia-1.440 .b 4b and zephyr- (opphyr-bb-stie -ere------ttffte full [Tunstall et al., 2023]. We tested Ultrachat200k dataset by HuggingFace, which is subset of the high quality demonstration UltraChat dataset[Ding et al., 2023] for text generation and dialogue. For Ultra- chat200k, we adopt the same strategy as Chen al. [2024] to pick up 50k data for training. At each iteration, we again train our model for 2 epochs.
Evaluation. For the Anthropic-HH dataset, we show the reward evaluated by the PKU-Aligmmne www.ent/beaeer-7bbbbr ......02e0 O-reward [Dai et al., 2024, Ji et al., 2023] model which is a tailored for evaluating human prefer-
popular 7b model fine-tuned from 1aaa-11ammaaaaaaaaaaa ences regarding helpfulness and harmlessness. We also record win rate of the two proposed methods over base model and SFT model. For the Ultrachat200k dataset, follow the widely used Hugging- Face Open LLM Leaderboard [Beeching et al., 2023]. This evaluation package assess an LLM based on six tasks: LLMS commonsense reasoning (Arc Clark et al. [2018], HellaSwag Zellers al. [2019], Winogrande Sakaguchi et al. [2021]), multi-task language understanding (MMLU Hendrycks et al. [2020]), human falsehood mimic (TruthfulQA Lin et al. [2021] IJ) and math problem solving (GSM8K, Cobbe et al. [2021]). See the appendix for more implementation details.
5.2 Results of RFT (Algorithm 1)
We present the result of Algorithm 1 over Anthropic-Hi dataset. We first fine-tuned pythia-1 1..4..44 1.bb using supervised fine-tune over the entire dataset (160k training data in total) using only the pre- ferred/chosen data for 10 epochs and pick up the checkpoint with the best testing accuracy as our base model. We then use PKU-Alignn reward model. We use this model to pick 10k data from Anthropic-HH dataset with the highest reward scores. Next, we fine-tune the base model using SFT and Algorithm 1. Figure shows the experiment results averaged reward and win rate, where record the average score (by
bbwettr--bbb-ee/eeee 0-reward model as our ground truth
8

---

PKU-Alig www.ggneeeeennnnnnnnnno 10bb-33.0-rwaal. O-reward) of the continuation generated for test datasets, also the win rate (ratio of samples where the reward of our model's generation is higher than the model compared) of the proposed Algorithm 1 over the full SFT base model and the top 10k SFT model. The figures show that the proposed algorithm improves over the SFT models terms of effectively improve the helpfulness and harmlessness of the model continuation.
1.5
Wenrete over full
Winraze over top 10k, SPT
1.0
0.5
0.6
0.0
-2.5
Average score of over tpp data
SFT on top 16k demonstrat on epaches
-1.0
SFT entrr dataset caret mined mode))
0.3
12500 15000 17500
policy learnng 15000 17500
(a) Average Score
(b) Win Rate
Figure 2: Algorithm 1 fine-tuning result of pythia-1 4b 4b over Anthropic- (with top 10k data picked by PKU-A1 PKU-Aligmmen www.lllmmgtbbaaeeeeeeeee wewt//br-errrrbreeeeeeee -0- 0 rewardd.. We record the average score of test dataset on the left figure and the win rate of Algorithm over the (full SFT) base model and the SFT model.
We remind the readers that the advantage of Algorithm over SFT in Figure 2 can be partially explained by Figure right side: despite the fact that SFT, Algorithm 1 and 2 are only observing chosen/preferred data, the latter still outperforms SFT since they discourage the likelihood of the synthetic non-preferred data, thus bringing better performance and robustness for the model.
5.3 Results of IRFT (Algorithm 2)
Different from the time consuming Algorithm 1, Algorithm 2 is more capable of handling large data and models. We first present the result for pythia- 1.aiia1.4bbb models over Ultrachat200k data. We remind the reader again that T tested on different choices of I'' and identify that T' = 5 to 8 gives the best performance in the Open LLM Leaderboard evaluations.
Algorithm 2 is equivalent SPIN [Chen al., 2024]'. We
The Open LLM Leaderboard result is presented in Table 3. We have the following main observations based on the results in Table 3:
1. SFT is not efficient in terms of boosting the pre-trained model performance on downstream tasks comparing methods which promote the decreasing of the likelihood of synthetic data, namely SPIN and IRFT;
SPIN and IRFT (Algorithm are both capable of further improving the performance of pythia model over downstream tasks, whereas IRFT shows better results due to more frequent generation comparing SPIN. IRFT with T outperforms both SFT and SPIN on most of the tasks as well as the average score;
3. More frequent generation might also result in more variances, therefore a reasonable (around 5) results in the best evaluation performance. Careful hyperparameter tuning might be needed for different models and datasets when applying our method, while we recommend using T = 5 as the default setting.
Apparently 1b model is not strong enough to handle hard tasks, e.g. GSM8k and all model perfor- mances
desirable. Now we present the result for zephyr-7b-sft-full] We remind the reader that this is a fully SFT-ed model and further SFT would only detriment the model performance (see Chen al. [2024]). The results are presented in Table where we can see that similar the lb case, both SPIN and IRFT could effectively improve the performance of SFT-ed model and the average performance of IRFT with T' = 5 stands out. The success of IRFT and SPIN further suggest that reward learning is indeed beneficial for aligning with demonstration data.
'IFTT with T = 1 and 2 epochs is equivalent to SPIN iteration 0, and T' = 2 with 2 epochs for each 1 is equivalent to SPIN iteration etc.

---

Table 3: Test performance of SPIN [Chen et al., 2024] and IRFT (Algorithm 2) based on pythia-1. 4b across HuggingFace Open LLM Leaderboard datasets. We keep training for 2 epochs after each generation process and K are calculated after this rule.
Tasks Metrics
K
Are_Are acc morm
TruthfuQQA
Winogrande scc
GSM8K exact_ munch
HellaSwag acc porm
MMLU
Average
scc
acc
Pythia-1.44b
1.44
53.55
25.63
37.27
SFT
sumppie
54.74
30,93
57.30
2.05
52.98
25.62
37.27
(SPIN)
54.00
57.70
1.36
53.76
25.34
37.35
IRFT (SPIN iter
52.85
32.04
57.38
1.74
53.57
25.49
37.18
IRFT
53.75
31.67
56,91
1.74
54.79
25.32
37.36
IRFT
53.75
31.92
57.85
2.43
54.77
25.44
37.69
IRFT
53.75
31.40
56.91
2.35
54.62
25.52
37.43
IRFT
56.34
31.54
58.41
1.59
54.54
25.69
37.57
Table 4: Test performance of SPIN [Chen et al., 2024] and IRFT (Algorithm 2) based xephyr-7b-sft-full across HuggingFace Open LLM Leaderboard datasets.
Tisks
T
K
Are
TruthtulQA
Winogrande acc
GSMSK
HellaSwag
MMLU
Average
2aphyt-7b-art-full
0
O
74.83
34.07
76.09
31.92
81.09
58.86
59.48
IRFT (SPIN)
75.08
36.57
76.01
33,59
82.81
57.83
60.32
IRFT (SPIN iter 2)
2
+
76.13
36.56
76.64
35.56
83.39
57.82
61.02
IRFT
5
75,82
39.99
77.19
31.24
82.07
$7.93
60.71
34.34
IRFT
8
75.23
36.67
75.85
31.84
80.89
58.60
59.85
IRFT
16
Tacleine
75.79
35.55
76.56
32.52
82.3
58.77
60.25
Conclusions and Limitations
In this paper proposed reward-leanne learning approaches for aligning LLMs with demonstration datasets. We show both theoretically and numerically the great potential of reward- -eard-i for alignment even without preference dataset. Our theory only indicate the convergence of the proposed algorithm to stationary point, and it is not clear what the policy converges to. The additional computation resources required for tuning models generate synthetic data in algorithms works include exploring reward-learingg for larger models and more complicated demonstration tasks, boosting the algorithm efficiency, and understanding how synthetic negative sample helps the LLMs to distinguish the preference dataset, etc.
not negligible. Future
Acknowledgments
M. Hong, S. Zeng and Li are supported partially by NSF under the grants EPCN-2311007, ECCS- 2426064 and SCFF-4114372 also by Minnesota Supercomputing Institute. A. Garcia and C. Li are partially supported by ECCS-2240789 and CCF-2414373.
References
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint 2402. 22..2220221441 14740, 2024.
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, al. Palm technical report. arXiv preprint arXiv:2305. 10403, 2023.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07733 2021.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.00 0006622, 2022.
10

---

Edward Beeching, Clémentine Fourrier, Nathan Habib. Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open Ilm leaderboard. https:
//huggingface. co/
/pppoppnn_____ 11m leaderboard, 2023.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397-2430. PMLR, 2023.
Michael Bloem and Nicholas Bambos. Infinite time horizon maximum causal entropy inverse reinforcement learning. In 53rd IEEE conference on decision and control, pages 4911-4916. IEEE, 2014.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-344 1952.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712. 2023.
Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. arXiv preprint arXiv:2312.099300 2023.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint (XXiv:2007.03374 2021.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.0133 2024.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling cittructionfinetuuee language models. Journal of Machine Learning Research, 25(70):1-53, 2024.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. arXiv preprint arXiv: arXiv: 180.:005703 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, al. Training verifiers solve math word problems. arXiv preprint 11..::2100..4....
2021.
Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe rihf: Safe reinforcement learning from human feedback. In The Twelfth International Conference on Learning Representations, 2024. URL https: pettts::/opennt http://ppppeeeeeeeeeeeeeeeee/eet net/ www.:/wwwwwwewtnn////eeee formm?dd= TyFrPOKYXw.
Tri Dao. Flashattention-2:: Faster attention with better parallelism and work partitioning. arXiv preprint XXXi:23077008611 2023.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2300.. /4233, 2023.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac"h, h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. framework for few-shot language model evaluation, 12 2023. URL
org/ recordd//1226663.
11

---

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009. 03300, 2020.
Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. Neural networks for machine learning lecture 6a overview of mini-batch gradient descent. Cited on, 14(8):2, 2012.
Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of LLM via human-preferrenee dataset. In Thirty-sevnnth Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL httph:t////// http////nnvvii.neeteeeeee gOQovXbFu3.
Sergey Levine, Zoran Popovic, and Vladlen Koltun. Nonlinear inverse reinforcement learning with gaussian processes. Advances in neural information processing systems, 24, 2011.
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857, 2022.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, al. Competition-leve code generation with alphacode. Science, 378(6624):1092 1097, 2022.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2 XXi:2200.00758, 2021.
Fei Liu et al. Learning to summarize from human feedback. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020.
Hong Liu, Zhiyuan Li, David Hall, Percy Liang, and Tengyu Ma. Sophia: A scalable stochastic second-order optimizer for language model pre-training arXiv preprint 14344 2023.
Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J Andrew Bagnell, Pieter Abbeel, Jan Peters, et al. An algorithmic perspective on imitation learning. Foundations and Trends@ in Robotics, 7(1-2): 1-179, 2018.
Long Ouyang. Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Dean A Pomerleau. Alvinn: An autonomous land vehicle in neural network. Advances in neural information processing systems, 1, 1988.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
Samyam Rajbhandari,, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1-16. IEEE, 2020.
Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannanch Hajishirzi, and Yejin Choi. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization. arXiv preprint arXiv:2210.01241 2022.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Proceedings. 2011.
12

---

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99--06. 2021.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.063447 2017.
Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Peter Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron Parisi, et al. Beyond human data: Scaling self-training for problem-solving with language models. arXiv preprint arXiv:2312.0653 2023.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng- fiee-TTee Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint 2xXiv:2001.08..39 2022.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09228 2023.
Tricu H Trinh, Yuhuai Wu, Quoc V Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations. Nature, 625(7995):476-488) 2024.
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of Im alignment. arXiv preprint arXiv:2310. 16944, 2023.
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, and Shengyi Huang. Trl: Transformer reinforcement learning. https://gtt thub. com/ huggingface ttreee/rr/.a....e 2020.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24835 2022.
Ronald J Williams. Simple statistical tradient-followigg algorithms for connectionist reinforcement
learning. Machine learning, 8:229-255 -266, 1992.
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for I1m alignment? a comprehensive study. arXiv preprint arXiv:2404.10719 2024.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXiv:2401. 10020, 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019.
Siliang Zeng, Chenliang Li, Alfredo Garcia, and Mingyi Hong. Maximum-likelibon inverse reinforcement learning with finite-time guarantees. Advances in Neural Information Processing Systems, 121101222100133 2022.
Brian Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pages 1433-1438. Chicago, II. USA, 2008
Brian D) Ziebart, J Andrew Bagnell, and Anind K Dey. The principle of maximum causal entropy for estimating interacting processes. IEEE Transactions on Information Theory, 1964),996--198, 2013.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford. Dario Amodei. Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arxiv: 1909 08593, 2019.
13

---

Appendix
A
Related works
Fine-tuning language models prevailing to improve LLMs performance on various instructional tasks, and has shown great success in enabling LLMs to generalize to efficiently respond out-of- sample instructions [Chung et al., 2024]. Despite many successful applications of SFT, people soon realized the great potential of reward learning and reinforcement learning based fine-tuning preference datasets for different tasks, including text-summarizing [Liu et al., 2020, Ziegler et al., 2019], story-telling [Ziegler et al., 2019], mnstruction-following [Ouyang al., 2022, Ramamurthy et al., 2022], etc. Equipped with the popular gradley-Terry model [Bradley and Terry, 1952], RLHF fine-tune language model using policy optimization methods, such as REINFORCE [Williams, 1992], proximal policy optimization (PPO, Schulman et al. [2017]) and lot more. On major obstacle for preference dataset fine-tuning the costly and time-consuming process of human labeling, and methods such as self-play fine-tune (SPIN, Chen et al. [2024]), synthetic data with binary feedback in self-training [Singh et al., 2023], weak-to-strong generalization [Burns al., 2023] and self-rewarding fine-tuning [Yuan et al., 2024] seek for improvement over SFT under weaker data supervisions comparing to preference datasets. In particular, SPIN generates synthetic samples for input prompts in the demonstration dataset and use them as the rejected data to for 'pseudo' preference data. As we will see, SPIN actually coincides with our implicit reward learning approach where we motivate the synthetic data in a more natural way.
In the reinforcement learning (RL) literature, inverse reinforcement learning (IRL) proposes to jointly learn the reward r which best explains an expert policy σ and the policy which in turn mimics this expert policy F from demonstration data. The most popular framework the maximum entropy IRL (MaxEnt-IRL. framework Ziebart et al. [2008], Levine et al. [201 Ziebart et al. [2013], Bloem and Bambos [2014], Zeng
[2022]. which seeks for policy maximizing the reward that matches the empirical averages in expert's is demonstrations data. MaxEnt-IRR IRL utilizes only the demonstration dataset for reward learning and already yields superior performance over plain behavior cloning [Pomerleau, 1988, Osa et al., 2018] approach on various RL tasks.
B Proofs for Section 3
restate and prove Lemma 3.1:
Lemma B.1. Problem (4) is equivalent the following minimax optimizatioon problem:
rxx, y/; 0) r(x, 5ū: + DKI. (1(())1)](
max min E (x
(8)
3
Proof. It is straightforward to see that the lower-level problem in (4) enjoys a closed-form solution:
()()) Tiee (yer(y)a))
( r v:
EgEA Tree (ū)z)) exp
( ( Ø; 0)
where A is the set of all possible responses. Plugging (9) into (4), obtain:
max Ban log (Tre(Ulz)))))) exp ( (1
log
Σ
Feet (ūlz) exp rr, ū: 0)
0
DEA
(10)
Utilizing the following identity:
Tref (ū]z)) exp (5r(e.000) 8: max By un) Trr, ã: 0)] DEL [[- [T(((
log
BEAA
obtain the following max-min problem (omitting some constant terms): r(x,, y: 0)
rxx T: 0) + DKL (1(()))]]m((((()))
max min E. (1))
(11)
0
11
3
14

---

The proof is completed.
Next, we restate and prove Lemma 3.2: Lemma B.2. For the loss function in (4), we have:
To (u(z)) Trat (y/ar)
Te t
vo( = (aa))
Ve log
V log
(12)
Trof (7)xc)
which refer the self-generation gradient, since at each iteration need generate one sample output ÿ from the current policy не and calculate the difference of the two rewards.
Proof. Omitting the constant terms not related to 0 in (10), we have
Tret (8lz)) (3r(0, (((o..::)) 8: )
max {()) Ex= fr(a, v1 log
Σ vEA
(13)
Calculating the derivative we get
VoL(O) (1 Vor Exup
Vo log
Σ UEA
Teal (ūlx)) exp (gr(x.
(xe(( exp (1)(2.2100)
1
Exmp
jEA For exp (tr(e, Vor(z, y:
3
Diee
MEA
(1 Vor
E Vor
3
E Be-ey-r (11_))____(111)))_))")))) Vor(
Ver(z,ū;0)
which implies that to minimize {()), one should always generate samples based on the current estimation of the policy To (z) and then update.
Now from (9) we get:
TT(y)=)
(xz y: B log (y|z) + 3 log (x)
(14)
where Zo (z) is the denominator of (9). In the view of (14), actually directly estimate:
Ve log (o(a))
To T
( = ((1))
Vo log
(15)
Teet(
Teel (ū]x))
The proof is completed.
Now we move to the proof for Section 3.3. We state the assumption needed for proving the final result:
Assumption B.1. For Algorithm and 2, we assume that
1. The policy distribution ne is uniformly lower and upper bounded, i.e.
Tiin I (= S xmax
where 0 < Tmin < Tmaxx for all
2. VTe is bounded, i.e. (V < for all z;
3. Vno is Lipschitz, i.e. [[ (u/x) - VTo, (v|x)) I L1 0211. all x and y;
where is defined in (9).
The above assumption can readily establish the assumption below, which is needed for our final convergence result.
15

---

Assumption B.2. For Algorithm 2, we assume that
1. f. is I-Lipschitz smooth w.r.I. 0, i.e.
((((1) Ve((22) - L||0] 02l
2. The stochastic estimator Ve is bounded, i.e.
|( G
These are all standard assumptions in nonconvex smooth stochastic optimization. We have the following lemma:
Lemma B.3. If Assumption B.1 holds, Assumption B.2 also holds with the following parameters:
Lo(3I
+ In)
2T.
G
2
Tmin
Tmin
Proof. We just show the value for L since G can be similarly computed. Since
(v(ar)))
To To (j)(z)) Tred (7/(r)
ve(0)
Vo lolo
Vo log
Tret
we have
Ve((2))
(u(x)
me, (77 ar)
E (erp) (1(x)
V log
V log
Tref (y/(z)
Tree []
To (3/ (z))
na, (57 (r)
Eu-paraa (11)
log
Vo log
Teef (ee((y)))) (x)
Tee (1)
Te (y(xr)
me. [z)
E Barpp.yrr (-1x)
V log
Vo log
3
Teef (y)l))
Trae (ū]
(16)
To (3/ (xr)
na, (37 (r)
(1)pp (-1x)
Vo log
Vo log
Tref (y) (
Tref (i
Te (y
Te (y (0)
+ 8
E aap (11) (11x
Vo log
V log
Tref (re((1/(z))
Nzet (0/x))
Te, (3)(z)
me, (37 (r)
P
V log
Vo log
Traf
For the first part, since
log ()((x))
we have
no. (y/x)
no. (5]xx)
log ne, (3)
Te, (ūlax
E Expp. (ll.) (-1a))
Va log Trat (y/xr)
Vo log
+ Va log
Teat (z)x))
Trof (3/ x))
Teet (il
V VT,, (y/2x)
Vmo, (y/(2)
VTO, (ī) (7)
V (17 (x)
E P ((1)))) (-1x))
n (z))x))
no (3)
no, (ū(x))
no (x)
Tee, (y)az)
VTe, (y/1²
VTT, ( x)) mo, (0)z)
VTe. (ū)x)
uo pom.p.y.ott B (112).000 (-1xx)
To, (y/ (r)
To. (3/(x))
B p.p.yo ((1)))/
no. (7]z)
|o. (t)lar) V Vee (y/ar) ne, (v/x) V (v|x)
E (112), (11x0)
+ (same term for g))
3
Te. (y ( ( (v)x)
Mmax L1 L
0 - 021
3
n 222
16

---

For the second term in the last line of (16), we have
1
To (1)(z))
T (7/
E (
Vo log
V log
B
Trof (y) (z)
Tref [0]2))
Tee (3/ (z)
0 (07
Paoparaa tr (110)
Vo log
Vo log
Tref (ylar)
rel (3)
Toa (ulxx)
Toa (ū xr)
E BompertE((aa)
Ve log
V log
(Te, (ū) To, Tos (ū(zz)
8
(2)(2)
Tref (ū]xx)
DEA
1 2Lo
E B.p
Σ gEA
To (5|z) - To, (ū1zz))
3 Tmin
2T.0
To, (0)(2)))
To (0)z)
2
B
Σ
no, (8/zz)
00 - 021
S Tmin
To (3)z)
8 T
MEA
Plugging these back to (16) we get
2
T L, + 2
- ve(02))l S [
110
02l
min
Now since we generate at the beginning of the inner loop, the estimator Ve(O0,,)) is not an unbiased estimator of Ve(Or.x) for any ls >> i.e.
1
TO. (Ye, [1e,k)
To. (Vr, [r,
ve(Oc.)) =
B Blaen ) m (116))))
Ve log
V log
Tref (3r (21,k)
Tef (2t (t,Nk
(17)
Te.. (yt, Tref (ye,(
To. (Ur, (t,,k)
#EVE(OL,)) = - (. - (1kw)))))
V log
V log
Teee [z,
(18)
We thus need to carefully analyze this biasedness so that the convergence can be boosted by large K, since large K will result very large bias.
Now we are ready to re-state and prove Theorem 3.1:
Theorem B.1. Suppose Assumption B.1 holds, then for Algorithm 1 and 2 with n = 0(1/VTK) have
oo Ao LG
i.a
min
+
VTK
T
where Do ----((0)
((On) and omit constant factors in Ô.
Proof. We prove directly for Algorithm 2 since the gradient estimator (12) and the estimator Jt.k Algorithm (we do solve the subproblem to its optimum) are both for the original bilevel problem (4).
From the Lipschitz gradient of we have
DLL 2
E(Or.u+1) 2 ((0..)) (VV((L))) Ve(B,..))
(E(()..)
i.c.
mel nellVe(B,,))11== (f(0),4t1) - e(B,,))) + + - v(((..). + m (
Taking expectation to 0,, and by Assumption B.2, we have
LG
n. V( (EE(0,, e(0r,,)) + n -VV(L...)-EEV...........)). v((O,..)
2
17

---

where the expectation is taken w.r.t the sample Ve.k to generate the estimator of current iteration. Sum up from k to k K we get
K-1
K V(O
11 LG K
mE [E(OL))) &
2
(19)
Since the expectation is taken only on the random sample at current iteration, and we know that the true gradient and the approximated gradient are (17) and (18), we have the following estimate:
[[Ve(0..)) - EV)(0,,))
To.,, (31,k)21,.))
To,, (Fe, (24,))
Ex,, k - (in ((1ma)))
log
Ve log
B
Teet (Vt (Fe,k)
Teef (ir, (e
Te., (Ve,
To., (31, (20,6)
a))
V log
Ve log
Tref (1t, (1eRk)
Tree (1/t, (it
/ Vo log To (Ue, (k,
To., (57 [1
Vo log
(TO, (1 (2),)) To, (0t dÿ
13
ne ( 1t,,k)
2L0
|10,,k 0r,oll
3 Tmin
= 1 2L we thus have:
Denote
K-1
K
(V((0..)) - -V ve(O..))
- -VV00.))
k=
k=
K-1
(e(0r.))) EVE(O.))
1
Σ
[[(((0,)))11
2
2
1' K-1
K-1
1 K-1
0 02.011+
[(((,.,)
Σ
Σ
((,..))
+
Σ
[((00)))
2
2
2
A=
k=0
K=U
Therefore
G2 K(K
-
K-1
(VE(0c)) EVC(O.) V((O.)) 5
[(((0.))1
2
2
Substituting back into (19) leads to
K-1
K(K- -1)
1 L.
(( < -(((O)))-))(((O))))) + c2
+
2
2
K=0
Summing up from f = to T gives
T-IK-1
K(K-1) 1
T-
L
VE(O,.)) s (k)0)
G2
2
2
1= 0=o
t=0
With a constant step size ?i- = 17 > 0, we have
K-1
B((Or-1,K-))
K -
LG²
E V
+ 2
+ 17
2TK
1TK
1= K=
Taking 17 (0////T)))) we get
1 K-
(E((O--KKI)-0(011100 - LG
Lz6²
+ T
TK K=
VTK
Hence as T + x0, the rate O(I/TKK).
18

---

C Implementation details of the numerical experiments
We follow the code as in SPIN [Chen et al., 2024], where we utilize DeepSpeed ZeRO-3 [Rajbhandari al., 2020] and FlashAttention-- [Dao, 2023] to reduce the memory cost. We use RMSProp [Hinton et al., 2012] optimizer with no weight decay. For lb models, we use two NVIDIA A100-40G to do the training with per device batch size of for Algorithm and per device batch size of 8 for Algorithm 2. For 7b models we use eight NVIDIA A A100-40G to do the training with per device batch size of 2. We train all models with 6 prevision. We set the peak learning rate to be 5e-7 for first two epochs and Ic-7 for the next two epochs. We fix 3 = 0.1 and consider the max sequence length to be 1024 for lb models and 2048 for 7b models. We use the same prompt template "### Instruction: prompt' n n### Response: as in Chen et al. [2024]. For the policy optimization step in Algorithm we use the PPO trainer in the TRL package [von Werra et al., 2020J. For the Hugging Open LLM Leaderboard evaluation, we use the Language Model Evaluation Harness library (v0.4.2) [Gao et al., 2023], and we also use the same number of few-shots as in Chen al. [2024].
Finally, in Table 5, we further provide the generation examples of our fine-tuned model in Table 4.
D Additional numerical results based on LoRA
During the reviewing and discussion periods, conducted extra experiments with LoRA to provide further evidences for this work.
We first provide the result of 7b experiments with LoRA in Table 6. In this setting we see a significant improvement over the pretrained model (zephyr-7b-sft-ful baseline and a 1% lift from SPIN. SFT in contrast can only achieve less than 1% lift from the base line. The reason here might due to the limited model size when using LoRA, contrastive training better helps the model distinguishing the referred and non-preferred continuations, yielding better performance over the standard SFT.
where we observe 2.3% lift from the
As a side note, we do not anticipate to significantly outperform SPIN since algorithmically our proposed IRFT method includes SPIN as a special case. Rather, one of our main objective is to provide theoretical foundation for the contrastive-type training algorithm, such as SPIN, which can all be studied under the bilevel inverse RL framework. The comparison with SPIN largely indicates that SPIN is still a RL-based fine-tuning method, suggesting an alternative interpretation that leads to provable convergence in lieu of the two-player game interpretation in Chen et al. [2024].
Reconciling our result with SPIN [Chen et al., 2024] Readers may realize that the result in Section 5.3 is different from SPIN [Chen et al., 2024]. We believe that this is due two reasons:
1. First, the baseline in our paper is different from SPIN paper: We believe a different baseline model is used in SPIN as evidenced in the Github discussions and the SPIN paper was released before the baseline is fully trained (Jan 2 vs Jan 10)², In particular, in Table 3 of the SPIN paper, the base model yields a "26.76" accuracy for GSMSk dataset, but we observe "31.92" which is significantly higher. We notice that newer version of both the model zephyr-7b-sft-fur
and the Im_ im_eval evaluation package* which both our paper and SPIN use for evaluation are used in our paper. We run test different versions of base model and eval codebase and obtain Table 7, where we indeed see that the new version of the base model has a significant lift in the performance on GSM8k comparing to the old version. We remark that SPIN paper observes the most significant increase of SPIN algorithm on GSM8k task (from 26.76 to 35.10) 10). Since we use the newest model in all our experiments, it leaves much less space for us to improve from.
2. Second, we should not compare the iter3's s 8.63% increase in Table 3 of SPIN with our paper's s 2.66% increase directly. When we say we take = 5 and epoch=2 as in Table 4, we essential split the data into 5 chunks and generate more frequently than SPIN, but still consume and generate for all the training data for 2 epochs in total (SPIN iterO also trained for 2 epochs). So what we need to compare is the first iteration of SPIN (which is SPIN
see for example the discussion in this link.
e See their model commit history.
See their codebase.
19

---

Table 5: Generation example of fine-tuned models in Table 4.
Prompt
Craft an intriguing opening paragraph for a fictional short story. The story should involve character who wakes up one morning to find that they can time travel.
Spphy-70-sft-full
It was just another ordinary morning for Emily, until she opened her eyes and found herself in a different time and place. She was lying on her bed, surrounded by strange decor and people dressed in clothes from a bygone era. Emily rubbed her eyes, thinking she must be dreaming. But as she got up, she realized that she could feel the weight of the day ahead of her, much like she always did. It was only then that she realized that she had the power to time travel.
IRFT T (SPIN iter 0)
As soon as her eyes fluttered open, something was different. The room looked the same, the sun streaming in through the same win- dow, but something felt off. It wasn't 't until she tried to sit up that she realized what it was her body felt lighter, as if she were floating on air. Confused, she looked around, searching for an explanation, and that's when she saw it - the clock on her bedside table was ticking backwards. Panic set in as she tried to sit up, but her body refused to obey her commands. And just like that, she was hurled through time, landing with a thud in a completely different era.
IRFT T = (SPIN iter I)
As soon as the alarm clock shrilled, something felt different to James, but he brushed it aside and reached out to turn it off, just like any other morning. Yet, as soon as he sat up, he felt a strange pull as if he was being dragged backwards, and his room began spin in a dizzying whirlwind of colors. In the blink of an eye, he found himself standing in different place altogether, and he knew that he had just woken up in a different time.
IRFT T = 5 (2 epochs)
opened my eyes groggily, ready to start another mundane day. But as rubbed my eyes and stretched my arms, something strange happened. I felt sudden jolt of energy course through my veins, and before knew it, the world around me had transformed. The room that was in no longer my own, and was standing in the middle of a bustling medieval town square. I blinked in disbelief, trying to make sense of what had just happened. Had gone mad, or was dreaming? But as I looked around me, realized with awe that I had somehow managed to time travel.
IRFT T (4 epochs)
It was just another regular morning for Emily, until as soon as she opened her eyes the room started to blur and spin. Panicking, she tried to stand up and grab the nightstand but her hand went right through it. Confused and terrified, she tried to scream, but as soon as the sound started to come out of her throat, she was engulfed by bright light that covered her body. When Emily opened her eyes again, she realized that she had traveled back in time, to a moment when she was 9 years old and standing at the foot of her parents' bed, ready to tell them the good news about acing her history test.
iterO in SPIN's original paper). Table 4 essentially indicates that, under our fair comparison setting, every iteration of our proposed algorithm outperforms every iteration of SPIN.
Last, we provide a simple experiment to show how accurate the reward learned by our model is. This experiment also addresses the generalization ability of the reward. Since we train our 7b model with a high-quality dataset (ultrachat) and we believe that the corresponding implicit reward r log( (og(TT/Tret) should already be pretty accurate terms of distinguish the good over the rejected continuations. Therefore we did simple test: we construct the implicit reward by
20

---

Table 6: Test performance of SPIN [Chen et al.. 2024] and IRFT (Algorithm 2) based on zephyr- sflyy-7b-sft-full across HuggingFace Open LLM Leaderboard datasets. this table we test with LoRA (r = 64).
Tasks
T
K
Are acc_norm
TruthiulQA
Winogrande
GSMSK
HellaSwag acc_norm
MMLU
Average
Metries
zophyr-bb-stt--fe 7b-.tt-ffll
0
D
74.83
34,07
76.09
31,92
81.09
58.86
59.48
75.20
IRFT (SPIN)
75.31
35.67
75.85
34.5
81,98
57.46
60.13
IRFT
74.92
37.96
76.95
35.25
82.48
57.66
60.87
Table 7: Test performance of different versions of the base model sephyr-bb-sft-teeee
Version
Im_oval version
Are_ Are_challenge
TruthfulQA mc2
Winogrande
GSMSK
HcllaSwag
MMLU
Average
Newest
v0.4.0
58.02
40,40
76.16
34.19
80
57.46
57.85
82.79
58.97
58.54
equation log(Te//ma Frer) where we compare different me (pretrained, SFT, SPIN and IRFT) on the ultrafeedback dataset (note that we did not do training on this dataset) which is a preference dataset. We believe that the accurate reward model preferred and rejected continuations, and we compute the ratio of r(preferred) >> r r(rejected) (which called win-rate in some literature) and obtain Table 8, where we can see that IRFT improves the implicit reward's distinguishability of chosen over rejected.
log( /rer) should be able to distinguish the
Table Win-rate of models trained by different methods. In this table we test with LoRA (r = 64).
Model SFT
SPIN (IRFT 1)
IRFT =
Win-rate
42.6%
42.8%
55.6%
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We show both theoretical and numerical results that support our claims in Section 3, 3.3 and 5.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
Justification: We made it very clear that we focus demonstration dataset. We also include limitation section the end.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
22

---

Answer: [Yes]
Justification:
Assumptions and proofs can be found in the appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide the details of the experiment in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
23

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Code is now released since it's accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
wttps.cc/ppblccc//eccccccpcppppp getic//guuee..ee CoodeSummii www.nnoolicciooeooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See the appendix for the implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We include the variance for all the plots in this work.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: See the appendix for the implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttp///eurrpp.ee/ee//ee
Answer: [Yes]
Justification: Yes
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: don't think there are direct societal impacts from this work.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper doesn't 't contain there high risk models or data.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The experiment part acknowledges all models and data we use.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
26

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This is work proposing training frameworks and not including any new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Not applicable
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "EMvz3DfGLj", "reviewer": "Reviewer_AMst", "summary": "This paper proposes two methods focusing on RLHF, namely RFT and IRFT. The takeaway message is that the SFT stage also significantly benefits from learning a reward model instead of using the human demonstration data directly via supervised learning.", "questions": "How can the reward model obtain the generalization power? Can the author say more about it, in my understanding, the main issue that affects the generalization is the scale of the demonstration dataset.", "limitations": "See weaknesses. I have to say I am not an expert in this field, therefore, I am willing to change my attitude if I find I am wrong.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is clear, illustrating the differences between similar works.\n2.  The motivation is clear, and the method is reasonable. Most importantly, it provides parts of theoretical analysis for the convergence.", "weaknesses": "1. The training procedure is complicated, meaning there are many issues in tunning. Is there any computation cost analysis for better understanding the limitations of this method?\n2. Though the author discusses SPIN, the truth is that equation 4.7 in SPIN is very similar to equation 6 in this paper. \n3. The experimental results are not strong. It did not surpass the SPIN by a large margin. Are there any other baselines that can be incorporated such as DPO?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> 1. The training procedure is complicated, meaning there are many issues in tunning. Is there any computation cost analysis for better understanding the limitations of this method?\n\n**Response**: We thank the reviewer for bringing this issue up. For Algorithm 1 in the paper, we need to maintain a reward model and a policy model (which is the LLM), and this is doubling the standard LLM fine-tuning. In short, the memory consumption and computation time of Algorithm 1 is similar to the standard RLHF process, where we also have the reward learning and policy optimization processes; For Algorithm 2, we simply need to maintain the policy (LLM) model, and the memory consumption would be exactly the **same** as the standard SFT, whereas the computation time would involving generating for the entire training sample, which would be of similar level as the standard RLHF process.\nWe summarize the memory consumption and the computational time of the proposed methods in the table below, *assuming that the reward and policy models are of same size*. Here *Forward* means the memory required for storing a model in inference mode, and *Backward* is the memory required for storing a model in training mode, including weights, activations and gradients; also *SFT* means the computational time as standard SFT, and *Generation* means the time to generate continuations for each of the input training prompts. *2SFT+Generation* is roughly the same time as standard RLHF.\n\n| Method | Peak Memory    | Computational Time |\n| ------- | -------- | ------- |\n| Algorithm 1 | Forward+Backward  | 2SFT+Generation |\n| Algorithm 2 | Backward | SFT+Generation |\n\n> 2. Though the author discusses SPIN, the truth is that equation 4.7 in SPIN is very similar to equation 6 in this paper.\n\n**Response**: As we stated from Line 251 page 7, our Algorithm 2 include the algorithm in SPIN as a special case and our inverse reinforcement learning framework naturally introduces a difference of log probabilities formulation. We believe that the claim \"the truth is that ... is very similar\" is not very accurate, but instead we would say that the algorithms **coincide under certain conditions**. There is still a key difference between (4.7) in SPIN and our equation 6: in our equation 6 we take the negative sample $\\tilde{y}$ generated by our current model $p_{\\mathbf{\\theta}}$, whereas in SPIN the negative sample $\\tilde{y}$ is sampled from the model from the based model of current training epoch $p_{\\mathbf{\\theta}_t}$.\n\n> 3. The experimental results are not strong. It did not surpass the SPIN by a large margin. Are there any other baselines that can be incorporated such as DPO?\n\n**Response**: We don't think it necessary to compare with DPO. As we made it clear in the abstract, our proposed framwork is a method for the supervised fine-tune stage of alignment, which means that we do not have access to a preference dataset but only a demonstration dataset. Direct preference optimization (DPO) is not designed for aligning demonstration dataset. In fact, our proposed method can be regarded as a DPO-type algorithm for the supervised fine-tune stage. So we cannot compare our method fairly with DPO, at least not when DPO utilizes preference data.\nHowever, in order to further show the strength of our algorithms, we tested our algorithm for 7b models with additional parameter-efficient fine-tuning (PEFT) settings and show that the proposed method could yield a **2.3% improvement** comparing to the baselines. The result tables are in our general rebuttal to all reviewers. We hope this could give more strong evidence on the superiority of the proposed methods.\n\n> 4. How can the reward model obtain the generalization power? Can the author say more about it, in my understanding, the main issue that affects the generalization is the scale of the demonstration dataset.\n\n**Response**: The power of this proposed method is that we can learn a reward model even without the preference dataset, and we completely agree with the reviewer on this point that the generalization power relies on the scale and quality of the demonstration dataset. For example, we train our 7b model with a high-quality dataset (**ultrachat**) and we believe that the corresponding implicit reward $r=\\log(\\frac{\\pi_{\\mathbf{\\theta}}}{\\pi_{\\text{ref}}})$ should already obtain certain generalization ability. Therefore we did a simple test below to show the generalization power of the reward model learned from the proposed algorithm: we construct the implicit reward by equation $r=\\log(\\frac{\\pi_{\\mathbf{\\theta}}}{\\pi_{\\text{ref}}})$ where we compare different $\\pi_{\\mathbf{\\theta}}$ (SFT, SPIN and IRFT) on the **ultrafeedback** dataset (note that we **did not do training on this dataset**) which is a preference dataset. We believe that the reward model $r=\\log(\\frac{\\pi_{\\mathbf{\\theta}}}{\\pi_{\\text{ref}}})$ with a better generalization ability should be able to distinguish the preferred and rejected continuations, and we compute the ratio of $r(\\text{preferred})>r(\\text{rejected})$ and obtain the follow table:\n| Model     | SFT | SPIN (IRFT with $T=1$) | IRFT ($T=5$) |\n| -------- | ------- | ------- | ------- |\n| Ratio $r(\\text{preferred})>r(\\text{rejected})$ | 42.6% | 42.8% | 55.6% |\n\nwhere we can see that IRFT **significantly inproves the implicit reward's distinguishability of chosen over rejected, indicating a superior generalization ability.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f22BmLXq03", "reviewer": "Reviewer_J5fv", "summary": "The authors propose using inverse reinforcement learning (IRL) on demonstration data in place of supervised learning, as is typically done for LLMs. The intuition is that human preferences are also encoded in demonstrations collected for SFT. Concretely, the authors propose a bilevel optimization approach with policy learning at the lower level and reward learning at the upper level. Two methods are proposed for alignment: with either implicit or explicit reward learning. The authors demonstrate improved performance when used to finetune smaller language models, compared to existing SFT approaches.", "questions": "See Weaknesses section for suggestions.", "limitations": "The paper is missing more thorough discussion on the method’s limitations and weaknesses (e.g. sensitivity to T, more complicated training process compared to SFT, etc.).", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The method seems (to the best of my knowledge) to be novel and interesting, tying together recent LLM literature and IRL methods. \n- The authors compare against relevant baselines (normal SFT, or SPIN) across a suite of benchmarks and two model sizes.\n- The overarching idea and question the paper strives to answer is of importance and relevance to the research community, as the proposed methods enable extracting more value out of SFT data.", "weaknesses": "- The experimental results seem to support the efficacy of the proposed algorithms, however the gains across various tasks with IRFT seem fairly marginal (it would help if Table 2 and 3 contained error bars, perhaps with policies finetuned with different seeds). As it stands, it seems like different benchmarks results are sensitive to different hyperparameters (choice of T), and the baselines outperform IRFT on 3/6 of the benchmarks in Table 3, suggesting that the additional complexity of the method does not necessarily lead to improved performance across all tasks. For example, the best average performance of IRFT is 61.03, whereas the best baseline achieves 61.02.\n- Furthermore, the results seem more mixed in Table 3 compared to Table 2, suggesting that it's not clear the method is still effective as model sizes increase.\n- It would help to present the SFT results in Table 3. Even though the authors note that further SFT could degrade performance, it would help to see how effectively using the demonstration data with IRFT or SPIN compares against naive SFT. \n- Minor note: line 157 seems incomplete, \"even when the demonstration policy is\"...(extreme)?\n- Minor note: the notation is a bit confusing, with theta referring to both the model parameters and the reward model parameters. For clarity, it would help if separate variables were used.\n- Minor note: missing reference to appendix in line 226.\n- Minor note: grammar error in line 320 -- \"SPIN and IRFT are both capable of further improv(ing) the performance..\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The experimental results seem to support the efficacy of the proposed algorithms, however the gains across various tasks with IRFT seem fairly marginal (it would help if Table 2 and 3 contained error bars, perhaps with policies finetuned with different seeds). As it stands, it seems like different benchmarks results are sensitive to different hyperparameters (choice of T), and the baselines outperform IRFT on 3/6 of the benchmarks in Table 3, suggesting that the additional complexity of the method does not necessarily lead to improved performance across all tasks. For example, the best average performance of IRFT is 61.03, whereas the best baseline achieves 61.02.\n\n**Response**: To further show the strength of our algorithms, we tested our algorithm for 7b models with parameter-efficient fine-tuning (PEFT) settings and show that the proposed method could yield a **2.3% improvement** comparing to the baselines. The result tables are in our general rebuttal to all reviewers. We hope this could give more strong evidence on the superiority of the proposed method.\n\n> Furthermore, the results seem more mixed in Table 3 compared to Table 2, suggesting that it's not clear the method is still effective as model sizes increase.\n\n**Response**: Again please see our new results included in the general rebuttal to all reviewers. First, we show that for the LoRA setting, our proposed method significantly outperforms SFT. Additionally, we also conduct a reward accuracy analysis in the general rebuttal where we show that the implicit reward learned by our method is more accurate than that of SFT and SPIN. We believe the extra evidence should further support that our proposed method yields significant improvements over SFT. \n\n> It would help to present the SFT results in Table 3. Even though the authors note that further SFT could degrade performance, it would help to see how effectively using the demonstration data with IRFT or SPIN compares against naive SFT.\n\n**Response**: We thank the reviewer for this suggestion. Our claim \"SFT could degrade performance\" is largely from [1] page 8 \"SFT on further epochs 2 and 3 fails to yield more than 1% improvement\". We also verified this during the rebuttal period. Due to the limit time during the rebuttal period, we were only able to conduct the experiment on LoRA and the result can be see in the general rebuttal to all reviewers, where we indeed see **less than 1% lift of SFT from the baseline**, verifying the claim made by [1].\nWe will rephrase \"SFT could degrade performance\" to \"SFT on further epochs 2 and 3 fails to yield more than 1% improvement\" in the revised paper.\n\n> Minor note: line 157 seems incomplete, \"even when the demonstration policy is\"...(extreme)?\n\n**Response**: We thank the reviewer for the reminder. We meant to say \"even when the demonstration policy is extreme\" and will modify it accordingly.\n\n> Minor note: the notation is a bit confusing, with theta referring to both the model parameters and the reward model parameters. For clarity, it would help if separate variables were used.\n\n**Response**: We thank the reviewer for the reminder. For RFT (Algorithm 1), the $\\theta$ is the parameter for reward since the policy is determined by tge reward; For IRFT (Algorithm 2) $\\theta$ is the parameter for the policy. We will modify the $\\theta$ in Algorithm 1 into $\\phi$ in our revised paper.\n\n> Minor note: missing reference to appendix in line 226.\n\n**Response**: We thank the reviewer for the reminder. We modfied it correspondingly.\n\n> Minor note: grammar error in line 320 -- \"SPIN and IRFT are both capable of further improv(ing) the performance..\"\n\n**Response**: We thank the reviewer for the reminder. We modfied it correspondingly.\n\n> The paper is missing more thorough discussion on the method’s limitations and weaknesses (e.g. sensitivity to T, more complicated training process compared to SFT, etc.).\n\n**Response**: We thank the reviewer for raising this issue up. We believe IRFT is robust to hyperparameters in general. We test all our experiment of IRFT, SPIN and SFT with the same learning rate, epochs and batchsize. We do have new hyperparameters $T$ and $K$. However when you fix $T$, total epochs and the training dataset, $K$ is automatically determined. Therefore the only parameter one needs to determine in extra is $T$. We found that in practice taking $T=5$ (or multiples of 5, meaning training for more epochs) would result in the best performance for both 1b and 7b models. In short, the only extra parameter we have is $T$ and we believe $T=5$ is readily a good candidate for most of the alignment tasks. \n\nPlease refer to our general rebuttal to all authors for the computational cost analysis.\n\n**References**:\n\n[1] Chen, Zixiang, et al. \"Self-play fine-tuning converts weak language models to strong language models.\" International Conference on Machine Learning (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pPnz1P7936", "reviewer": "Reviewer_RhUZ", "summary": "This paper proposes to study if IRL techniques can be used for aligning large language models. They propose two different algorithms: one that explicitly learns a reward model and one that implicitly learns the reward in the policy. These reward models are learned by contrasting expert generations and the policy generations.", "questions": "- How accurate is the learned reward model?\n- The experimental setting is not very clear. Is Algorithm 1 used on the top 10K data selected by the reward model? If so this is really problematic: shouldn’t you fine tune on the whole dataset (or a randomly downsampled subset)? Can you assume that you will have access to 10K samples with high reward scores? And wouldn't just be better to use the reward model used for data selection for RL?\n- How robust is IRFT to hyperparameter setting compared to SFT?\n- Is 1-2% increase on the OpenLLM benchmark meaningful?", "limitations": "Yes, the authors discuss limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper writing and motivation is clear.\n- The paper includes in depth theoretical analysis.\n- They evaluate the trained LLMs on a wide variety of benchmarks.", "weaknesses": "- The experiments are not very convincing. For the Open LLM leaderboard experiments, the gain is really small (around 1 or 2%). It seems likely that this gain is due to variance in the training process, especially since IRFT has ~1% variance in different hyperparameter settings.\n- There isn’t much discussion of the computational cost of the algorithms.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The experiments are not very convincing. For the Open LLM leaderboard experiments, the gain is really small (around 1 or 2%). It seems likely that this gain is due to variance in the training process, especially since IRFT has ~1% variance in different hyperparameter settings.\n\n**Response**: To further show the strength of our algorithms, we tested two extra experiments and inclide the results in the general rebuttal to all reviewers. First, we test our algorithm for 7b models with parameter-efficient fine-tuning (PEFT) settings and show that the proposed method could yield a **2.3% improvement** comparing to the baselines. Additionally, we also conduct a reward accuracy analysis in the general rebuttal where we show that the implicit reward learned by our method is more accurate than that of SFT and SPIN (please also see our answer to your third question). We hope this could give more strong evidence on the superiority of the proposed method.\n\n> There isn’t much discussion of the computational cost of the algorithms.\n\n**Response**: We thank the reviewer for bringing this issue up. We refer to the general rebuttal to all reviewers where we include a detailed computational cost analysis.\n\n> How accurate is the learned reward model?\n\n**Response**: We conduct a very simple experiment to show how accurate the reward learned by our model is. Since we train our 7b model with a high-quality dataset (**ultrachat**) and the corresponding implicit reward $r=\\log(\\frac{\\pi_{\\mathbf{\\theta}}}{\\pi_{\\text{ref}}})$ should already be pretty accurate in terms of distinguish the good over the rejected continuations. Therefore we construct the implicit reward by equation $r=\\log(\\frac{\\pi_{\\mathbf{\\theta}}}{\\pi_{\\text{ref}}})$ where we compare different $\\pi_{\\mathbf{\\theta}}$ (pretrained, SFT, SPIN and IRFT) on the **ultrafeedback** dataset (note that we **did not do training on this dataset**) which is a preference dataset. We believe that the accurate reward model $r=\\log(\\frac{\\pi_{\\mathbf{\\theta}}}{\\pi_{\\text{ref}}})$ should be able to distinguish the preferred and rejected continuations, and we compute the ratio of $r(\\text{preferred})>r(\\text{rejected})$ and obtain the follow table:\n| Model     | SFT | SPIN (IRFT with $T=1$) | IRFT ($T=5$) |\n| -------- | ------- | ------- | ------- |\n| Ratio $r(\\text{preferred})>r(\\text{rejected})$ | 42.6% | 42.8% | 55.6% |\n\nwhere we can see that IRFT **significantly improves the implicit reward's distinguishability of chosen over rejected**. We hope this addresses the reviewer's concern.\n\n> The experimental setting is not very clear. Is Algorithm 1 used on the top 10K data selected by the reward model? If so this is really problematic: shouldn’t you fine tune on the whole dataset (or a randomly downsampled subset)? Can you assume that you will have access to 10K samples with high reward scores? And wouldn't just be better to use the reward model used for data selection for RL?\n\n**Response**: We thank the reviewer for the question. For the experiment on Algorithm 1, we want to demonstrate that with **high quality** demonstration data, our proposed Algorithm 1 is more capable of learning an accurate reward model that is able to distinguish the good over the rejected continuations, by assigning higher scores to the good continuations. Note that we need to construct a \"high quality dataset\" at the first place. The experiment setting is that we use a well-established reward model (beaver-7b-v3.0-reward) to pick up good demonstration data (which are the top 10k data in terms of reward score), and **fairly** train the model with SFT and our method on these 10k data, then compute the rewards of the continuations generated by SFT trained model and our trained model.\n\nWe are not assuming we always have access to top 10k samples, instead, top 10k sample is used as a relatively simple way to obtain high quality data to verify our presumption that our method is more capable of getting better reward if we have good demonstration data.\n\nIn practice, the implication of this experiment is that, if you have good demonstration data (under certain score criteria) at hand and would like your language model to align with it, then our proposed method should be more capable of extract this underlying score criteria and better aligning with it. **We do not assume that we always have a well-established reward model on hand in practice**. \n\n> How robust is IRFT to hyperparameter setting compared to SFT?\n\n**Response**: We thank the reviewer for the question and we believe IRFT is robust to hyperparameters in general. We test all our experiment of IRFT, SPIN and SFT with the same learning rate, epochs and batchsize. We do have new hyperparameters $T$ and $K$. However when you fix $T$, total epochs and the training dataset, $K$ is automatically determined. Therefore the only parameter one needs to determine in extra is $T$. We found that in practice taking $T=5$ (or multiples of 5, meaning training for more epochs) would result in the best performance for both 1b and 7b models. In short, the only extra parameter we have is $T$ and we believe $T=5$ is readily a good candidate for most of the alignment tasks.\n\n> Is 1-2% increase on the OpenLLM benchmark meaningful? \n\n**Response**: Yes, we do believe that it is meaningful. The openLLM leaderboard benchmark is one of the most heavily-used (SPIN also primarily used this) influential benchmark in the community where it does not rely on large models such as GPT4 but only on multiple downstream tasks to test the performance of different LLMs. For models like Llama3-70B, they in general also only achieve only 1 to 2% increase over the previous state-of-the-art (if you check their hugginface webpage, they also tested on tasks such as MMLU, GSM8K and Winogrande). Therefore we believe we are making meaningful progress using the same tasks as SPIN on 7b models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OmQLVgotj5", "reviewer": "Reviewer_Tyd8", "summary": "This paper proposes an approach to aligning Large Language Models (LLMs) with human preferences by integrating Inverse Reinforcement Learning (IRL) into the supervised fine-tuning (SFT) stage. The proposed IRL-based method simultaneously builds a reward model and a policy model during the SFT stage, enhancing efficiency and robustness against low-quality data.", "questions": "1. It seems that RFT and IRFT perform equally. What's the takeaway from these two methods? Moreover, how to choose them wisely.\n2. Typo in L226, there's an error in referring to the Appendix.\n3. In L204-206 \"In 204 practice however we take a relatively small T and large K, because frequent on-line sampling is time 205 consuming\" How does this parameter choice affect the final performance, especially compared to large T and Large K?", "limitations": "The limitations are included in the conclusion section.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper introduces a novel IRL-based method for the SFT stage, providing a fresh perspective on improving LLM alignment.\n\n2. The paper provides a strong theoretical basis for the proposed algorithms, ensuring that they converge to stationary solutions of the IRL problem.\n\n3. The evaluation includes both theoretical analysis and empirical testing on large-scale models, ensuring the validity of the results.", "weaknesses": "1. The paper does not address the scalability of the proposed methods to even larger models or different types of LLMs beyond the ones tested.\n\n2. The evaluation results only present the empirical performance but lack experiments on computation costs introduced by this method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The paper does not address the scalability of the proposed methods to even larger models or different types of LLMs beyond the ones tested.\n\n**Response**: We thank the reviewer for the suggestion. Here we believe provide a theoretical analysis would be enough since we know exactly how much memory and time each of the proposed algorithm would need comparing to SFT. For Algorithm 1 in the paper, we need to maintain a reward model and a policy model (which is the LLM), and this is doubling the standard LLM fine-tuning. In short, the memory consumption and computation time of Algorithm 1 is similar to the standard RLHF process, where we also have the reward learning and policy optimization processes; For Algorithm 2, we simply need to maintain the policy (LLM) model, and the memory consumption would be exactly the **same** as the standard SFT, whereas the computation time would involving generating for the entire training sample, which would be of similar level as the standard RLHF process.\nWe summarize the memory consumption and the computational time of the proposed methods in the table below, *assuming that the reward and policy models are of same size*. Here *Forward* means the memory required for storing a model in inference mode, and *Backward* is the memory required for storing a model in training mode, including weights, activations and gradients; also *SFT* means the computational time as standard SFT, and *Generation* means the time to generate continuations for each of the input training prompts. *2SFT+Generation* is roughly the same time as standard RLHF.\n\n| Method | Peak Memory    | Computational Time |\n| ------- | -------- | ------- |\n| Algorithm 1 | Forward+Backward  | 2SFT+Generation |\n| Algorithm 2 | Backward | SFT+Generation |\n\n> The evaluation results only present the empirical performance but lack experiments on computation costs introduced by this method.\n\n**Response**: We thank the reviewer for the question. We refer to the answer to the previous question, where we do include the computational time analysis of the proposed methods. In short, Algorithm 1 is similar to RLHF, and Algortihm 2 is similar to a generation plus a DPO process.\n\n> It seems that RFT and IRFT perform equally. What's the takeaway from these two methods? Moreover, how to choose them wisely.\n\n**Response**: The difference between RFT and IRFT are similar to that of RLHF and DPO. RFT produces an explicit reward model for further generalization and IRFT is more memory and time efficient. We advocate using IRFT if you are not looking for an explicit reward model from the demonstration dataset (see our general response for the detail of the generalization ability of the reward model).\n\n> Typo in L226, there's an error in referring to the Appendix.\n\n**Response**: We thank the reviewer for the reminder. We modfied it correspondingly.\n\n> In L204-206 \"In 204 practice however we take a relatively small T and large K, because frequent on-line sampling is time 205 consuming\" How does this parameter choice affect the final performance, especially compared to large T and Large K?\n\n**Response**: We did our test on a large T and small K, i.e. we generate samples for every training batch and do the update. The performance is largely similar to the SFT performance (less than 1% lift from the baseline), as shown below (see our general rebuttal to all reviewers for the implementation details):\n| Task     | Arc | TruthfulQA | Winogrande | GSM8k | Hellaswag | MMLU | Average |\n| -------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- |\n| zephyr-7b-sft-full  | 74.83    | 34.07    | 76.09    | 31.92    | 81.09    | 58.86    | 59.48    |\n| SFT | 75.20    | 34.18    | 76.16    | 34.95    | 80.96    | 57.71    | 59.86    |\n| IRFT (T=1, SPIN)    | 75.31    | 35.67    | 75.85    | 34.5    | 81.98    | 57.46    | 60.13    |\n| IRFT (T=5)    | 74.92    | 37.96    | 76.95    | 35.25    | 82.48    | 57.66    | **60.87**    |\n| IRFT (T=#batch)    | 75.23    | 33.58    | 75.37    | 33.13    | 82.26    | 57.68    | 59.54    |\n\nwhere T=#batch is the case when $T$ is large and $K$ is small.\n\nIn short, we believe that generating too frequently would not only be time-consuming, but also detrimental to the model performance (since we take to much variance into account at each stochastic gradient step for generation). We would recommend a reasonable frequency in generation which leads to the optimal performance, such as what we observed when we generate 5 to 10 times for the entire training dataset.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for valuable comments. In this general rebuttal, we provide answers and results to questions that are raised by multiple reviewers. We wish our extra analysis and results could help you better understanding the contribution of the work, which we believe is significant to the community.

# Additional experiments to show the effectiveness of proposed algorithms

## LoRA experiments
We first sincerely apologize for an important typo in our initial draft. In Table 3, we claimed that we use LoRA for 7b models. This is not true since **we actually did a full fine-tune** with the proposed method. We did use LoRA to validate the method but did not collect the results. We now collect the correct results where we test our algorithm with LoRA (r=64) as follows:
| Task     | Arc | TruthfulQA | Winogrande | GSM8k | Hellaswag | MMLU | Average |
| -------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- |
| zephyr-7b-sft-full  | 74.83    | 34.07    | 76.09    | 31.92    | 81.09    | 58.86    | 59.48    |
| SFT | 75.20    | 34.18    | 76.16    | 34.95    | 80.96    | 57.71    | 59.86    |
| IRFT (T=1, SPIN)    | 75.31    | 35.67    | 75.85    | 34.5    | 81.98    | 57.46    | 60.13    |
| IRFT (T=5)    | 74.92    | 37.96    | 76.95    | 35.25    | 82.48    | 57.66    | **60.87**    |

We compared the standard SFT, SPIN (IRFT with T=1) and IRFT with T=5. In this setting we see a significant improvement over the pretrained model (zephyr-7b-sft-full), where we observe **2.3% lift from the baseline and a 1% lift from SPIN**. SFT in contrast can only achieve less than 1% lift from the base line. The reason here might be due to the limited model size when using LoRA, a constrastive training better helps the model distinguishing the referred and non-preferred continuations, yielding better performance over the standard SFT. 

As a side note, we **do not** anticipate to significantly outperform SPIN since algorithmically our proposed IRFT method includes SPIN as a **special case**. Rather, one of our main objective is to provide a theoretical foundation for the constrastive-type training algorithm, such as SPIN, which can all be studied under the bilevel inverse RL framework. The comparison with SPIN largely indicates that SPIN is still a RL-based fine-tuning method, suggesting an alternative interpretation that leads to provable convergence in lieu of the two-player game interpretation in [1].

## The accuracy/generalization ability of the learned reward model
We conduct a simple experiment to show how accurate the reward learned by our model is. This experiment also addresses the generalization ability of the reward. Since we train our 7b model with a high-quality dataset (**ultrachat**) and we believe that the corresponding implicit reward $r=\log(\frac{\pi_{\mathbf{\theta}}}{\pi_{\text{ref}}})$ should already be pretty accurate in terms of distinguish the good over the rejected continuations. Therefore we did a simple test: we construct the implicit reward by equation $r=\log(\frac{\pi_{\mathbf{\theta}}}{\pi_{\text{ref}}})$ where we compare different $\pi_{\mathbf{\theta}}$ (pretrained, SFT, SPIN and IRFT) on the **ultrafeedback** dataset (note that we **did not do training on this dataset**) which is a preference dataset. We believe that the accurate reward model $r=\log(\frac{\pi_{\mathbf{\theta}}}{\pi_{\text{ref}}})$ should be able to distinguish the preferred and rejected continuations, and we compute the ratio of $r(\text{preferred})>r(\text{rejected})$ and obtain the follow table:
| Model     | SFT | SPIN (IRFT with $T=1$) | IRFT ($T=5$) |
| -------- | ------- | ------- | ------- |
| Ratio $r(\text{preferred})>r(\text{rejected})$ | 42.6% | 42.8% | 55.6% |

where we can see that IRFT **significantly improves the implicit reward's distinguishability of chosen over rejected**. We hope this addresses the reviewer's concern.

# Computational cost analysis
For Algorithm 1 in the paper, we need to maintain a reward model and a policy model (which is the LLM), and this is doubling the standard LLM fine-tuning. In short, the memory consumption and computation time of Algorithm 1 is similar to the standard RLHF process (RLHF=reward learning + policy optimization); For Algorithm 2, we simply need to maintain the policy (LLM) model, and the memory consumption would be exactly the **same** as the standard SFT, whereas the computation time would involving generating for the entire training sample, which would be of similar level as the standard policy optimization process (same computational time as SPIN). Note that standard policy optimization process is equivalent to the time of standard SFT and a generation process toward all training input prompts.

We thus summarize the memory consumption and the computational time of the proposed methods in the table below, *assuming that the reward and policy models are of same size*. Here *Forward* means the memory required for storing a model in inference mode, and *Backward* is the memory required for storing a model in training mode, including weights, activations and gradients; also *SFT* means the computational time as standard SFT, and *Generation* means the time to generate continuations for each of the input training prompts. *2SFT+Generation* is roughly the same time as standard RLHF.

| Method | Peak Memory    | Computational Time |
| ------- | -------- | ------- |
| Algorithm 1 | Forward+Backward  | 2SFT+Generation |
| Algorithm 2 | Backward | SFT+Generation |

Reference:

[1] Chen, Zixiang, et al. "Self-play fine-tuning converts weak language models to strong language models." International Conference on Machine Learning (2024).

Meta-review: This work proposes a novel large language model (LLM) alignment approach using Reward Learning from Human Demonstration. All reviewers consistently recommended accepting this work. AC agrees that this work is interesting and deserves to be published on NeurIPS 2024. The reviewers did raise some valuable concerns that should be addressed in the final camera-ready version of the paper. The authors are encouraged to make the necessary changes in the final version.
Decision: Accept (poster)