SeRA: Self-Reviewing and Alignment of LLMs using Implicit Reward Margins
Authors: Jongwoo Ko
Saket Dingliwal
Bhavana Ganesh
Sailik Sengupta
Sravan Babu Bodapati
Aram Galstyan
Venue: ICLR 2025 Poster (2025)
Keywords: Preference Alignment
Large Language Models
Implicit Reward Margin
Sample Selection
Preference Bootstrapping
Primary area: foundation or frontier models, including LLMs

Abstract: Direct alignment algorithms (DAAs), such as direct preference optimization (DPO), have become popular alternatives to Reinforcement Learning from Human Feedback (RLHF) due to their simplicity, efficiency, and stability. However, the preferences used by DAAs are usually collected before alignment training begins and remain unchanged (off-policy). This design leads to two problems where the policy model (1) picks up on spurious correlations in the dataset (as opposed to only learning alignment to human preferences), and (2) overfits to feedback on off-policy trajectories that have less likelihood of being generated by the updated policy model. To address these issues, we introduce Self-Reviewing and Alignment (SeRA), a cost-efficient and effective method that can be readily combined with existing DAAs. SeRA comprises of two components: (1) sample selection using implicit reward margin to alleviate over-optimization on such undesired features, and (2) preference bootstrapping using implicit rewards to augment preference data with updated policy models in a cost-efficient manner. Extensive experiments, including on instruction-following tasks, demonstrate the effectiveness and generality of SeRA in training LLMs with diverse offline preference datasets and and DAAs.

Published as a conference paper at ICLR 2025
SERA: SELF-REVIEWING AND ALIGNMENT OF LLMS USING IMPLICIT REWARD MARGINS
Jongwoo Ko
Saket Dingliwal?
Bhavana Ganesh
Sailik Sengupta
Sravan Bodapati
Aram Galstyan
KAIST AI
28 mazon AGI
3 a WS AI Labs
ABSTRACT
Direct alignment algorithms (DAAs), such direct preference optimization (DPO), have become popular alternatives for Reinforcement Learning from Hu- man Feedback (RLHF) due their simplicity, efficiency, and stability. However, the preferences used in DAAs are usually collected before the alignment train- ing begins and remain unchanged (off-polle policy). This design leads to two problems where the policy model (1) picks up on spurious correlations in the dataset (as opposed to learning the intended alignment expressed in the human preference labels), and (2) overfits to feedback on off-policy trajectories that have less likeli- hood being generated by the updated policy model. To address these issues, we introduce Sclf- Reviewing and Alignment (SeRA), cost-efficient and effective method that can be readily combined with existing DAAS. SeRA comprises of components: (1) sample selection using implicit reward margins, which helps alleviate over-fitting to some undesired features, and (2) preference bootstrapping using implicit rewards augment preference data with updated policy models in a cost-efficient manner. Extensive experimentation, including some on instruction- following tasks, demonstrate the effectiveness and generality of SeRA in training LLMs on offline preference datasets with DAAS.
INTRODUCTION
Large Language models (LLMs; Achiam al. 2023; Team al. 2023; Jiang et al. 2024) have shown mastery on multitude of tasks in artificial intelligence (AI), ranging from creative writing (Wang al., 2024b) to code generation (Li al., 2023a), and mathematical reasoning (Ahn et al., 2024). With success, comes concerns related to their safety, reliability, and potential for misuse in sensitive domains like social manipulation, cyber-attacks, etc. To address some of these challenges, works have considered aligning LLMs to human values/preferences using approaches like Reinforcement Learning from Human Feedback (RLHF; Ouyang et al. 2022; Bai et al. 2022a).
Initial RLHF approaches, such Proximal Policy Optimization (PPO), necessitated the need for two separate models a policy model (the LLM model to be aligned) and a reward model (Schulman al., 2017; Ouyang al., 2022). Such approaches often result in challenges related stability and scalability (Rafailov et al., 2023; Zhao et al., 2023; Ilaaa Team, 2024). To mitigate these shortcom- ings, Direct Alignment Algorithms (DAAs), such direct preference optimization (DPO; Rafailov et al. 2023), sequence likelihood calibration with human feedback (SLiC-HF; Zhao et al. 2023), and identity policy optimization (IPO; Azar et al. 2024), have emerged as popular alternatives. In DAA, we directly update the policy model/LLM using (some closed-from solution of) the pairwise pref- erence data without the need for an explicit reward model, making the alignment process simpler, more efficient and stable compared to earlier methods (Rafailov et al., 2023).
However, DAAS leverage preference rating on off-policy trajectories collected prior to alignment tuning (at times, generated by different I.L.Mpolicy--mode (Zhao et al., 2023; Tunstall et al., 2023)), resulting in two challenges. First, the off-policy preference data, which still require labor- intensive annotations, often contain noisy features orthogonal to the true preference objective (e.g. longer responses are more preferred; Park et al. 2024). Thus, training on this data can teach the policy model to over-fit to such noise (Mitchell, 2023; Chowdhury al., 2024), thereby learn spuri- ous correlations (Park et al., 2024; Rafailov et al., 2023). While some works try to consider explicit
Work done research intern mazon. Correspondence to jengwoo ko@kai at.ac. .rr

---

Published as a conference paper at ICLR 2025
regularization to alleviate over-fitting to spurious features (Park et al., 2024), identifying all possible spurious features is often challenging and incomplete (Rafailov et al., 2024). Second, the preference feedback, being off- off-poliey, cannot aid the policy model to obtain feedback during alignment training. As the off-policy and the on-policy may not belong to the same distribu- tion, this indirect feedback can inhibit improvement of the policy model (Tajwar et al., 2024). The latter shortcoming is partially addressed by PPO-like methods, where the stand-in reward model can rate on-policy generations (Guo al., 2024).
its own generations
To inherit the best of both worlds, several works have explored using from Al-Feedback (RLAIF; Guo et al. 2024; Rosset et al. 2024; Lee et al. 2023: Bai et al. 2022b). These approaches aim to mimic the human's 's preference rating behavior using high-quality LLMs (via API access). Unfortunately, this approximate
are both inefficient and expensive. For example, Direct Nash optimization (DNO; Rosset et al. 2024) incurs a cost of $34,000 to obtain preference annotations using GPT-4 (Zheng et al., 2023). Thus, the motivations of bypassing training of a reward model lands up eventually incurring exorbitant costs.
Contributions. We propose to augment DAAs with novel strategy, Alignment (SeRA) ) that uses Implicit Reward Margin (IRM; defined as Equation 5) for off-policy sample selection and iteratively bootstraps preference data for alignment. Specifically, we show that:
and
IRM-based Off- Off-policy Sample Selection helps mitigate over-optimization of policy models to spurious correlations (such as considering response length to gauge preference). Its efficacy and cost-efffectiveness is consistent across various DAAs, datasets, problems, and model variants.
IRM-based Preference Data Bootstrapping mitigates DAAS from continuously updating policy models with off-policy data and proposes a decoding and rejection sampling approach to extract informative policy pairs (based on IRM) for continual alignment training. This improves the efficacy and efficiency of DAAs without the need for expensive (& external) reward models.
Better Performance and Versatility: We empirically showed that SeRA can be widely used across various DAAS (e.g.. DPO, IPO, SLIC-HF, SimPO) and on various L.MSs (e.g.. Tamyl. 1.IB, Pythia-2.8B, Mistral-7B) consistently outperforming SoTA baselines (Kim et al., 2024a; Pattnaik et al., 2024). Finally, we conduct an exploratory analysis and ablations to better under- stand SeRA.
2 BACKGROUND
In this section, we provide brief overview of related works ($2.1) and discuss some preliminar- ies ($2.2) necessary to understand our contribution. A more comprehensive discussion of related works is available in Appendix A.
2.1 RELATED WORK
Actor-critic RLHF frameworks (Christiano et al., 2017; Stiemnon et al., 2020; Bai et al., 2022a; Ouyang et al., 2022) seeks to align language models to human preferences, but is often unstable during training and memory-intensive (requiring the policy model and reward model to be on device simultaneously). To mitigate this, several algorithms, such as direct preference optimization (DPO; Rafailov et al. 2023) and sequence likelihood calibration (SLiC-HF; Zhao et al. 2023), learn the contrastive preference in the offline setting using a closed-form loss function without the need for an critic/reward model. Azar et al. (2024) argued that without regularization, policy overfit to deterministic preferences and introduced identity preference optimization (IPO) to directly optimize offline preference probabilities with regularization. Moreover, the development of offline preference datasets like UltraFeedback (Cui et al., 2023) and OpenOrca (I.ian et al., 2023) increases the practical accessibility of aligning LLMs.
easily
Despite their effectiveness, offline approaches can suffer from distribution mismatch between the initial data and updated policy model (Guo al., 2024; Ko et al., 2024). To address this, Guo et al. (2024) proposed online Al-feedback (OAIF) by using another LLM to annotate which of two deiine-sampled outputs from the current policy is preferred. Rosset al. (2024) suggested direct Nash optimization (DNO), which is trained on iteratively updated preference pairs from the recent policy model based on theoretical soundness. While effective, these methods incur high expenses for human labeling or proprietary L.LM API calls, making them difficult to implement practically.
Another cost-efficient alternative is using policy models as preference labelers or leveraging implicit preference superiority with high-quality human responses. Self-rewarding LMs (Yuan et al., 2024)
2

---

Published as a conference paper at ICLR 2025
Model Types
Model Sizes Pyeria)
Dircct Aignment Algorthmms
Training Datasets
66-
56-
Tulma 2ythe-2.85 Viatx-78
6.0
SUC G.0.x
Capybers
Figure I: Motivation of SeRA: : IRM-based selection showed higher win-rates (measured by Claude 3-as-a-Judge (Zheng al., 2023; Anthropic, 2024)) against the base SFT model across variety of settings- (a) Model types: TinyLlama (Zhang et al., 2024), Pythia-2.8B (Biderman et al., 2023), Phi-2 (Li et al., 2023c), and Mistral (Jiang al., 2023). (b) Model size: various model sizes of Pythia (410M, 1.OB, 2.8B, 6.9B). (c) DAAs: DPO (Rafailov et al., 2023), IPO (Azar et al., 2024), SLiC (Zhao al., 2023), and KTO-Pair (Ethayarajh al., 2024) and (d) training datasets.
studied the benefits of iteratively training on preferences derived from recent policy's s sampled out- puts. However, in their work, they used the LLM itself as the annotator based prompting (Zheng et al., 2023). which is only valid for LLMs capable of following the prompt properly. Self-play fine- tuning (SPIN; Chen et al. 2024b) and adversarial preference optimization (APO; Cheng et al. 2023) are both iterative LLM training techniques that are compatible with contrastive losses. However, they make the very limiting assumption that the high-quality predefined response is better than the policy generation, without considering any annotator feedback (Rosset et al., 2024).
2.2 PRELIMINARIES
Reinforcement Learning From Human Feedback. The goal of RLHF is to optimize the policy LLM Te such that it maximizes the expected value the reward function. A common approach to modeling the reward function is using Bradley-Teny (BT; Bradley & Terry 1952) model:
exp (r(x, ))
pyy > yr (x) exp (r(æ;, yv)) + exp (r(æ, 9:))
orr(x, yw) r(x, y:)),
(I)
where yu (and yr) denotes the preferred/winning (and losing) policy, p denotes the preference dis- tribution that approximates an unobserved latent reward r(x, y). and is the logistic function. To achieve this, RLHF first trains a reward model (x y). Then, RLHF updates та with an on- policy RL algorithm like PPO (Schulman al., 2017), iteratively optimizing the model provide responses more preferred by human. The most common objective is
CRLHI Eg-DD.y [x, 8) 30DL (|
(2)
which enforces a KL divergence (Kullback & Leibler, 1951) penalty with a reference distribution (x)), to prevent the LLM xy from straying too far from its initialization. The hyper-parameete balances between exploiting the reward function and deviating from (y)r).).
Direct Alignment Algorithms. RLHF is computationally expensive and unstable (Rafailov al., 2023; Llama Team, 2024). Thus, many algorithms (Rafailov et al., 2023; Zhao et al., 2023; Azar et al., 2024) have been proposed to overcome these challenges. A common idea is to analyti- cally derive the optimal policy and parameterize using the reward function from Equation 2. In DPO (Rafailov et al., 2023), the optimal policy x² under the BT model satisfies:
p" (y yi x=) (1 (1+ +1+0g exp Blog
TT" (vI(z) B Bog Tean [x)
π (3( (x))
Nyat y
where (B > x:) is underlying true preference, the probability that y is more preferred than yr. With human preference data expressed in terms of the optimal policy rather than the reward model, we can formulate a maximum likelihood objective for a parameterized policy zq-
To (y
(y)((x
Cppo (x, yu yi) = - lolog log σ Log
B log
(4)
Tea T (ww
Tee (y xx)
Current methods for LLM aligmment first collect dataset of pairwise preferences by obtaining two responses to an input prompt æ generated using an LLM. Then, human AI annotators rank these responses, yielding preferred response yo and a less preferred one y!-
3

---

Published as a conference paper at ICLR 2025
3 OUR METHOD:
AND ALIGNMENT
In contrast to recent works on DAAS (Rosset al., 2024; Guo al., 2024). we propose SeRA to improve alignment with DAA without the need for any external supervision. While self- highlights that a policy model can dual up as a reward model (Weng et al., 2022), we show that this self-supervision can be done implicitly for alignment using the notion of Implicit Reward Mar- gin (IRM). After defining IRM in this section, we describe (1) an IRM-based off-policy sample selection, and (2) an IRMM-based iterative preference data bootstrapping. Finally, describe our approach SeRA and how it can be seamlessly incorporated into DAAs.
3.1 IMPLICIT REWARD MARGIN (IRM)
While works have shown that alignment datasets may have ambiguous preferences (Yang et al., 2023; Chowdhury et al., 2024), even the use of unambiguous preference data can result in policy models unintentionally learning spurious correlations (e.g. on response length) (Park et al., 2024; Rafailov et al., 2024). Recent works consider selecting preference samples (æ,, ywu yi) where there exists large difference in preferences (i.e. You >> Yt) to help mitigate the noise in the learn ing (Yang et al., 2023; Rosset et al., 2024). We leverage this observation and introduce a Implicit Reward Margin (IRM) to quantify the difference between the preference over responses using the policy model TT itself. Formally, we define IRM as follows where Teet is the reference model:
m(x You y = B (r(æ, yvo) - r(x,, yi)) = log To (y log
To (3)
(5)
Uno az)
Tel (3e (3t ax:
which follows from Equation 3 and Equation 4. Unlike Rafailov et al. (2023), we omit the normal-
ization term for the reward r(æ, == -log
me [o(l) Tid (y/a)
for tractability.
3.2 IRM-BASED OFF-POLICY SAMPLE SELECTION
We note that a policy model ma can generate mis-calibrated rewards for individual trajectories (Pan- ickssery al., 2024), but empirically study if the reward margin between preferred (gw)) less-preferred trajectories (yn) can provide relevant signals to choose samples for improving align- ment. Finally, we propose to filter out training samples with lower IRMs. To motivate our choice, we compare against other sample selection methods in the Figure 1. -R-M- sample selection (in green) consistently leads to higher win rates against the SFT model, when compared alignment on the entire preference samples (in sky-bluc), or strong baseliense that using RLAIF with GPT-4 score (Zheng al. (2023); in orange) or the log probability of गet (Pattnaik al. (2024); in pink) for sample selection. These gains hold true across model types, model size, DAAS, and datasets. Moreover, methods that use GPT-4 to obtain reward signals for sampling (Rosset et al., 2024; Zheng et al., 2023) incur high-cost; in comparison, our sample selection strategy that leverages aa to com- pute IRM is inexpensive and more effective.
Mitigating over-optimization to spurious correlation. To understand why IRM selection bene- fits alignment, we first highlight that recent works have shown that DAAs prioritize features of the data based on their complexity and prevalence, often resulting in learning preference signals based on dimensions orthogonal the alignment objective, such the length of responses (Park et al., 2024; Rafailov et al., 2024). Thus, we empirically analyze if IRM's sample selection strategy can weed out such noisy examples, thereby preventing over-optimization. To this extent, we study var- ious correlations on TinyLlama on the UltraFeedback dataset for the four aforementioned sample selection strategies.
In Figure 2, we can observe that the rewards of the model trained on IRM-based sample selection (fourth column) has the highest R² scores with GPT-4 score, while also having the lowest R² score with the response lengths. This indicates that the policy model is highly aligned with the preferences of GPT-4 but does not learn the spurious feature of response length. Even though the rewards of the model without any sample selection (first column) also have the highest correlation with GPT-4 score, their correlation with response length is also high, a result consistent with previous works (Park et al., 2024; Rafailov et al., 2024) highlighting that DAAs indeed risks over-optimization where the model can learn spurious correlations. These results showcase that the proposed IRM selection strategy can mitigate over-optimization problems to response length, while other selection
4

---

Published as a conference paper at ICLR 2025
Select
Reward Margin Selection (Slope 05)
0.91 (R)
0.07)
0.22/8*= 0.00)
Reward
40
Srore
OPTE Soond
07T4 Score
OPTI4 Scord
0.02 r2e 0.23)
(Stopa DUDL II 0.18)
(Slope -2.02 (R 0.577
Chlope 0.01/ (3)
10-
10-
Roward
10-
30
-40
200
100
600
66c
200
620 Phcpprra Longth
460
900
200
400
600
EC0
200
400
623
800
Papporte Length
Prspuros Langet
Raageroo Longth
Figure 2: Comparison of correlations between the rewards of the trained policy model and features (e.g., GPT-4 score and response length) across different selection methods (i.e., no selection, GPT-4 score-based, log probability of the reference model, and IRM-based selection). [Row 1J Correla- tion between GPT-4 Score & implicit reward (i.e, r(x, )w.)) for Yw. [Row 21 Correlation between response length (i.e. ly and the implicit reward for chosen responses. The model with IRM selection (i.e. [Column 4)) shows the highest R² score for the first row, but the lowest R² score for the second row. This indicates that IRM-based selection strategy can effectively mitigate the over-optimization on response length (Park et al., 2024).
methods such GPT-4 (second column; Rosset et al. 2024)! and log probability of the reference model (third column: Pattnaik et al. 2024) don't 't mitigate or, at times, exacerbate the undesired bias.
Theoretical Intuition of Sample Selection. Here, we additionally provide mathematical sup- port for the empirical phenomena observed in Figure 2. Research has shown that DNNs first learn explicit features and then memorize minor ones (e.g., spurious features, noisy labels) dur- ing the later training stage (Zhang al., 2017; Liu et al., 2020; Nam et al., 2020; Liu et al., 2021). Inspired by this, we intend the policy LLM to first learn the human preference using con- As the estimated probability
fident preference pairs where true preferences (y > (1(x²)=
Po(w > y1(
[llg Ne(M= (x) 33 log Fur((>i [x) is modeled by IRM in Eqn. using the
sa(g) lac)
lz)
BT model, IRM selection removes samples with small margins that might lead to over-optimization on spurious features or noisy preferences.
Remark (Informal Statement for Theorem 1). Under DPO training. with probability at least
1-8. 8. the upper bound of the risk of f(z Ywu 31) r(x, yw) r(xx M1). denoted as R(f), over the underlying distribution P, where the sample set 5 = (x. Yw: y1) ~ P. can be defined as:
(VSO (
log(M|s1/0) SS|
Mog(Mis)/5) S
R() S R(J:S)++++ O (Brlleo)) - 01122) +0
where Misi is uniform covering number. This reflects the risk of classifying preferable responses is upper bounded by the empirical risk over observed samples, the difference between estimated and true preferences. (See Appendix B for the proof.)
Remark indicates that training on clear preference samples reduces the upper bound of the risk compared to training on ambiguous samples. The inherent problem of DPO is that is treats all pref- erence samples equally (i.e., p(y > z)(z) 1, V(z, yu 3t) E S) even when the true preferences are not, i.e. p" (yo yr ar) # p" (y'6 > v/l==". By minimizing the empirical risk on such datasets, the difference between e and p² increases, which can results in over-fitting to spurious features. However, by training LLMs confident samples, we can achieve both lower empirical risk and less overfitting (i.e., small differences between pa and p²). This theoretical interpretation supports the experimental observations shown in Figure 2. We further discuss each term in detail in Appendix B.
3.3 IRM-BASED PREFERENCE DATA BOOTSTRAPPING
As mentioned earlier, DAA'S use of off- policy preference annotation suffers from distributional mis- match between the sequences observed during training (which are from different LLMs ahead of
'This result is consistent with recent works indicating that even GPT-4 evaluation has length bias (Wu & Aji, 2023; Koo et al., 2023; Dubois al., 2024a). This problem might also affect off-the-shelf reward models based existing work reward hacking of reward models (Gao al., 2023), as shown in Table 3.
5

---

Published as a conference paper at ICLR 2025
Algorithm 1: Self- and Alignment
1: Input: Preference dataset D, SFT model mh, DAA loss function &
2: Hyper-parameters: # off-policy samples (k), # bootstrapped samples (k), Training epochs T
3: Output: Policy model TØr
4: /* Update policy model with offline dataset for t=1* Nol [+) meo (2) (
5: Obtain n0, arg max, 1 (I)
((B log NolU= i0) soo (Dula))
3 log
6: for i E ( { do
7: Initialize M < 0. 0 M.
8: for (xx(*) , Vi E
(4) (1)
9:
/* [$3.2] Compute IRM for Sample Selection */
10:
m( = (m((x()) You
I/ Margin based on ensemble reward (Eq. 7)
11:
M MU (m())
// Store IRM for each data-point
12:
/* [S3.3] IRM-based Preference Bootstrapping */
13:
VI...., ÿR -T._(110))
// Sample on-policy trajectories
Fea t r((x(1), mj)) Vj (I....,RR R}
I/ Get implicit reward for trajectories (Eq 7)
15:
D+ DU Yarg max, Pea Yarg min, Feass)
// Select trajectories to maximize IRM
16:
M-Mu (max, F2,ij miny Pr,1,)
// Store maximum IRM
17:
end for
18:
/ Select samples with the highest IRM */
19:
M.M t sort(M, revesed=True), sort(M, revesed=True)
Dj (1x(0, yo 1)))) E m(() MM: k))) U {(x10) 1,,0,,n, you yi EED mm Mi: R))
(i)
(()
(1)
20:
21:
/* Update policy model */
Obtain no, arg maxg E M)). [ log no, (2)
ne (gula)
3 Blog =
22:
no, (31 (z)
23:
end for
training) and those generated by the iteratively updated policy LLM (Arora et al., 2022; Ko al., 2024), leading to reduced efficacy of DAAs (Guo et al., 2024). To address this, we present prefer- ence data bootstrapping that considers decoding- -imee strategy to sample candidate pairs from the (updated) policy LLM, followed by rejection sampling of pairs with low IRM.
To build a preference pair, we sample the (2 2) distinct candidate responses yi Vn (1)x()) from a query (i) by using decoding-time sampling. We then compute the implicit reward for each response j(E {I,...,(R)) using the term (1) (() in Equation 5 and select the pair that maximizes the IRM.
Our method bears similarity to self-rewarding LMs (SRLM; Yuan et al. 2024) that reward responses by using LLM-as-a- audge (Zheng et al., 2023). Unfortunately, this approach works reliably only when one uses LLMs with sufficient mnstruction-folloving capability (see Table 5), which leads to significant costs and limited applicability for smaller LLMs. In contrast, approach is versatile across LLMs with a wide range of capacities and boasts strong empirical efficacy (see Table 6).
3.4 SERA: SELF- REVIEWING AND ALIGNMENT
We first describe how our two proposed components can be incorporated in an iterative manner with DPO and then extend it for other DAAS. Algorithm 1 provides overview of SeRA. As our method leverages the policy model from previous iterations both for sample selection ($3.2) and preference data bootstrapping ($3.3), we consider an iterative approach alignment training. Similar recent works (Kim et al., 2024a; Pattnaik et al., 2024), we also update the reference model nie to the latest updated policy model after every iteration. At iteration I, we obtain O, using the following update step (instead of Equation 4 used in DPO):
Ta (y)w() Blog To To(3l (c)
)
8, + arg max B(D.N..M~- log (Bog
(6)
To.., (y
To. (3) x
where D, is dataset obtained at the start of iteration t, by combining is samples from poolcy dataset using $3.2 and & bootstrapped samples using method $3.3 with the highest IRM. At L. 0.

---

Published as a conference paper at ICLR 2025
consider the reference model to be the SFT model (i.e. Tea TSFT) that we seek to align. Note that at t = 1, Dr consists only of the original offline samples, as no aligned model is available yet.
Ensemble of Reward Margin across Different Iteration. As SeRA depends leveraging the policy model being trained instead of external reward model supervision, risks introduction of undesired bias or reward hacking (Pan et al., 2024) that can manifest as part of the implicit rewards used to calculate IRM. To alleviate this, we apply ( yw: y1) == (1/3) (rr ywo) re y:)
rather than using Eqn. (5), where Va (t 3) is defined follows:
(a y) (1 - 7) log Te: (y)x)
To (y)x) y log T (y/xx log Th. (y) xx) To. To._ (y|x))
(7)
To (y|z)
To, ( xz) (
TH, a (
where ? is the ensemble coefficient. This is conducted both for preference pairs in the original datasets ($3.2) and generated samples ($3.3) to build D; for iteration t. For t < 3, we use rz y)
== (3/l)
log s0, [)]
Empirically, this approach showed consistent effectiveness ($5; Figure 5a).
Extension to Other DAAS. Instead of BCE loss function of IRM (i.e., m(x; Ywe y:)) in Eqn. (4),
SLiC-HF (Zhao et al., 2023) minimizes a hinge loss function of IRM:
LE Yw. vi) = max 0, B (10g
( (2))
log To (2c)
(8)
Teel y (Be (x)
Teet (1) (x)
where 1/8 acts as the margin of miss-classification (Guo et al., 2024). minimizes the square loss function of IRM:
IPO (Azar et al., 2024)
(10gs
To (7o (2)
Taa (3n (x)
1
LIPO (r. Yw y1)
log
Trerl Vh ax)
Tee Ter (3t (z)
2,3
Note that IPO and SLiC-HF, in contrast to DPO, doesn't 't assume a preference model like BT. Despite of such independence, the objective functions still contain the IRM term enabling to easily extend SeRA to these DAAs. Thus, we use SeRA with these DAAs for our upcoming experiments.
4
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Setup. Our setup resembles Tunstall et al. (2023) and Hong et al. (2024), where they consider three models- Tlaya (Zhang et al., 2024), Pythia-2.8D (Biderman et al., 2023), and Mistral- 7B (Jiang et al., 2023). Initially, these models are fine-tuned on UltraChat-200k (Tunstall et al., 2023) followed by an alignment with DAAS and preference pairs from UltraFeedback. We use the binary version of UltraFeedback (which contains two response pairs with corresponding ratings for given input query) (Tunstall al., 2023) for all DAAS except Curri-DPO (Pattnaik et al., 2024), where we use the original UltraFeedback (that contains four response pairs) (Cui et al., 2023). In addition, we will also showcase SeRA's 's prowess on other popular alignment datasets like HH- RLHF (Bai et al., 2022a) and TL:DR (Stiennon et al., 2020). A detailed description of the datasets and the implementation can be found in Appendix C. For all experiments, we set T 3, 7 0.3. If N represent the total number of preference pairs in the offline dataset, we choose our values of k. l s.t. k + k- = N, allowing us a fair comparison against other baseline in terms of sample size. Based on a validation set, we set k = 0.7N k- 0.3NN (labelled as SeRA in our tables). However, to showcase the full potential of our proposed method, we relax the sum-constraint and also report numbers for SeRA, where k = 0.7N k = 2N.
Baselines. We compare SeRA to several previous DAAS. First, we consider DPO (Rafailov et al., 2023) and its variants: Iterative DPO (Kim al., 2024a; Rosset al., 2024), Curri-DPO (Pattnaik et al., 2024). SPIN (Chen et al., 2024b), and ORPO (Hong et al., 2024). Then, we use SeRA with IPO (Azar et al., 2024) and SLiC-HF (Zhao al., 2023) and compare against the original method and its iterative version. For experiments on synthetic noisy preference datasets, we also include cDPO (Mitchell, 2023) and rDPO (Chowdhury al., 2024).
Evaluation. We consider four popular test benchmarks AAlpaca bench (Dubois et al., 2024b), Vi- cuna bench (Chiang et al., 2023), Evol Instruct (Xu et al., 2023) test set, and the UltraFeedback (Cui et al., 2023; Tunstall et al., 2023) test set. For metrics, we use Claude 3 (Anthropic, 2024) as a judge to evaluate the quality of the generated response and report win rate compared another model.
7

---

Published as a conference paper at ICLR 2025
Table 1: Comparison performance where TinyLlama-I.Ie Pythia-2.8B. and Mistral-7B are fine-tuned on UltraChat- and preference pairs are sampled from UltraFeedback dataset with DPO (Rafailov et al., 2023) or its variants (Pattnaik et al., 2024; Chen et al., 2024b; Hong et al., 2024). Single represents average from single answer grading prompt while SFT, G3.5, G4 indicate the pairwise comparison (Zheng et al., 2023)
hetween SFT model, text-davi nci -003, and 9pt4 turbo. The best and second best performances highlighted bold and underline. j indicate results from original models shared on HuggingFace.
Model
Size
Technique
Alpaca SFT
Eval
Vicuna Eval Single
Evel-Instruct Single
UltraFeed Single
Single
G3.5
G4
SFT
SFT
SFT
SFT
5.41
28.9
1.7
6,05
5.48
4.98
5.82
55.1
2.9
57.5
5.50
5.34
Tlay
1.1B
leeraivv DPO
6.08
64.5
36.1
3.2
7.09
77.5
6.03
62.4
5.49
64.4
Curri-DOO
6.16
40.7
2.8
77.5
6.00
5.63
SeRA-DPO DPO
6,58
74,2
48.7
5.1
7.55
85.8
6.40
72.3
5.92
73.1
best-SeRA.
6.47
70.0
49.1
6.1
7.31
82.5
6.18
70.0
6.06
79.5
SFT
5.34
28.1
2.2
6.19
5.73
5.09
DPO
3.67
53.0
31.6
2.9
6.30
50.0
6.02
58.7
5.38
57.5
Pythin
2.8B
Ieeative DPO
6.05
63.2
40.6
3.0
7.23
70.0
6.16
60.8
5.67
65.9
Curri DPO
6.06
62.7
39.6
2.7
7.03
62.5
6.39
68.1
5.67
72.0
SaRA. DPO
6.40
70.7
50.2
6.0
7.59
76.8
6.68
74.2
5.93
71.7
bast-SeRA DPO
6.59
77.4
52.2
7.7
7.53
83.8
6.54
79.1
6.00
76.6
SFT
7.46
75.9
5.4
8.08
7.73
6.76
DPO
7.37
71.4
74.9
17.5
8.39
83.8
7.15
61.7
6.42
65.6
Iterative DPO
8.35
89.0
93.2
19.7
8.74
88.2
8.49
87.8
7.56
81.8
Mistral
7B
CurriDPOO
93.1
13.8
92.5
80.8
SPIN
7.64
65.1
77.1
8.4
8.40
70.0
7.66
61.7
6.63
61.0
ORPO
8.29
84.
91.6
14.5
8.82
83.8
8.61
86.0
7.58
78.8
SeRA- DPO
8.38
86.9
93.4
19.5
8.83
88.8
8.61
86.0
7.67
$4.4
bast-SeRA DPO
8.56
92.7
94.8
27.3
8.94
91.3
8.68
90.1
7.90
90.6
Comparison of performance where FinyLlama-L.ID 1.IB and Pythia-2.8B fine-tuned UltraChat- 200k and preference pairs are sampled from UltraFeedback dataset with IPO (Azar et al., 2024) and SLiC- HF (Zhao et al., 2023). The best performances are highlighted bold.
Table 2:
Alpaca Eval SFT
Vieuna Eval Single
Evol-Instruct Single
UltraFeed Single SFT
Method
Size
Technique
Single
G3.5
G4
SFT
SFT
SFT
5.41
1.7
IPO
5.94
54.1
29.3
2.7
6.84
63.8
5.91
50.7
5.55
64.7
Iterative IPO
5.98
56.5
30.1
2.2
6.94
63.8
6.00
59.2
5.65
64.7
TinyLloma
1.1B
SeRA-IPO
69.3
45.6
4.8
7.31
81.3
6.12
69.2
5.76
70.5
SLiC
6.19
65.9
40.1
3.5
7.27
71.3
6.05
65.1
5.57
66.8
Iterative SLiC
6.41
70.6
47.8
4.8
7.23
78.8
6.06
65.1
5.73
69.1
SeRa-SLiC
6.50
70.8
47.2
5.4
7,58
76.3
6.06
67.9
5.88
72.8
SFT
5.34
28.1
2.2
6.19
5.73
5.09
IPO
6.15
67.1
41.8
4.5
7.32
80.0
6.27
66.5
5.65
75.3
Iterative IPO
75.2
5.0
6.68
6.02
75.5
Pythia
2.8B
SeRA IPO
6.71
78.5
53.6
6.1
7.74
83,8
6.83
72.7
6.14
78.7
SLiC
6.07
65.2
41.7
2.8
7.06
66.3
6.24
60.8
5.66
69.9
Iterative SLiC
6.38
72.3
47.1
5.6
7.54
83.8
6.48
73.4
5.84
70.0
SeRA-SLiC
6,57
74.3
52.9
6.2
7.62
85.0
6.47
71.1
6.03
74.3
The win-rate adjudication assigns a weight of 1 to wins, 0.5 to ties, and 0 to loss. We also use a single answer grading prompt provide 1-10 rating for the responses generated by the model. The evaluation prompts are taken from Zheng et al. (2023) and provided in Appendix C.3 for reference.
4.2 RESULTS
Combining with DPO. We assess the general cnatuuttionfollowinnn abilities of LLMs by com- paring the DPO-based preference alignment algorithms. Tab. depicts (1) the cestruction-follwiinn performance of both variants of SeRA, showing their superiority over their baselines across diverse L.Ms and evaluation metries, and (2) that increased preference bootstrapping is more effective for stronger LLMs (e.g., Mistral-7B), as seen the larger performance gap between variants. These results highlight the versatility of the proposed method across different experimental setups.
Extension to Various DAAS. To show that SeRA can improve various DAAS, we report instruction-followingg performance of SeRA integrating with IPO (Azar et al., 2024) and SLiC- HF (Zhao et al., 2023) in Table 2. Similar to Table 1, SeRA consistently improved the performance of both IPO and SL.C-HF. Compared to the DPO and SLiC-HF cases, the performance improvement on when SeRA is used with IPO is relatively smaller. We note that IPO has been shown to effectively alleviate over-optimization in comparison to other DAAs (Rafailov et al., 2024); hence the gains ob-
8

---

Published as a conference paper at ICLR 2025
tained by SeRA indicate other potential upsides. Indirectly, these results support argument that SeRA addresses the challenge of over-optimization. We additionally provided the effectiveness of SeRA on different types of implicit reward, SimPO (Meng et al., 2024), in Appendix D.3.
Robustness to Noisy Preferences. To il- lustrate the effectiveness of our proposed SeRA on noisy annotations, we experimented with synthetically noised preference datasets. Similar to prior works (Chowdhury et al., 2024; Wang et al., 2024a), we consider the standard noise model and randomly swap the chosen and rejected responses. Our experiments evaluate two setups- 20% and 40% noise in the Ultra- Feedback dataset. On TinyLlama trained on each noise split, we highlight the performance of various robust DAAs in Figure 3. In most of the evaluation benchmarks, SeRA showed su- periority over other baselines. Moreover, the performance gap between DPO, iterative DPO, and SeRA-DPO much larger than that on standard dataset. This indicate that our reward margin selec- tion effectively filters-out noisy samples with swapped preferences.
SPART
PMC
Seacth
Figure 3: Comparison of performance on 20% (left) and 40% (right) of noisy preference
Efficacy on Different Preference Datasets. To demonstrate the generality of SeRA, we also conducted experiments with TinyLlama on two other datasets: HH-RLHF (Ganguli et all,, 2022) and TL;DR (Stiennon et al., 2020). Since these two datasets do not contain explicit GPT-4 scores for each response, We utilized Mistral-7B-based reward model that achieves high performance in RewardBench (Lambert al., 2024) implement online feedback from stronger LLMs (OAIF; Guo et al. 2024), which requires significantly more computational resources compared ours. Further, we also showcase the importance of sample selection alone by bench- marking SeRA with k 0. Table 3 shows that the effectiveness of our approach is not limited to any particular preference dataset. It overcomes problems like spurious correlations that practically exists in every preference dataset. Further, can achieve this better performance at lower cost.
Table 3: Results HH-RLHF and TL:DR.
SeRA
SeRA
DPO IDPO OAIF
(K: = 0) (k- 0.3N)
HH-RLHF
56.7 63.0
64.6
65.4
66.1
TL.DR 59.7
59.6
59.5
61.0
62.7
5 ANALYSIS AND DISCUSSION
Component Analysis. In this section, we evaluate the efficacy of sample selection and preference bootstrapping in our proposed method. We plot the winning rate for dif- ferent ratios of l and l in Figure 4. The solid lines repre- sent results obtained by different iterations in SeRA with
UltraFeed Performance
72.5
Winning Rate (5)
70.0
65.0
varying ratios of l and k maintaining A: + k- = N, while the dotted lines represent results obtained by using only IRM selection where Ao 0.7.N, & 0. The results in- dicate that (1) IRM-based selection shows consistent im- using only offline samples (0% on x-axis), or only generated samples (100%), and (2) combining IRM-based sample selection and prefer- ence bootstrapping at moderate levels (iee. 30- 70%) leads to consistent improvement, highlighting the impor- tance of both components in SeRA. Without offline sam- ples, IRM and generated samples may lack valuable information for effective training, while relying solely offline data cause overfitting. Thus, the combination of selection and bootstrapping is crucial. Across iterations, this combination consistently outperforms solely using all offline sam- ples, only generated samples, IRM selection alone, confirming the importance and synergy of each component in SeRA.
Iter
Iter 2
62.5
her (k 0,K 0.7M)
Iter 3
60.0
Ierr 0E 0.7M)
0%
70%
provement in comparison
% Generated Samples (ANN R+ E
Figure 4: Pairwise comparison of mod- cls with varying percentage k and is while ensuring a fixed-sized training set (i.e. k; k + ke N).
Analysis on Ensemble across Different Iterations. We also investigate the efficacy of the en- semble across different iterations, reporting the performance and similarity of selected samples for

---

Published as a conference paper at ICLR 2025
Ubba Perforaane
Sibirity ford
Modila
20
33
CTT
0.00
Coub
55
2:00
300
0.5
0.5
0.00
LDO
0140
0.00
L00
0..00
24
1x0 0.0
702
0:00
1L.
20
15
19
6.
0.75
0.30
0.37
6.00
2L00
0.00
2444 0.76
2.40
640
0.05
79
0.00
1:00
0.00 0.02
0:74
0.30
0.64
0.0
47
Ui Los Hi0 H44
1:4 0.
Ln
100 184
(a) Final Performance
(b) Similarity
(a) LLMs
(b) DAAS
The performance and similarity of se- Figure 6: The similarity of selected sample sets lected sample sets across different values of the between different LLMs or DAAS with training ensemble coefficient y in TinyLlama-
Figure 5:
iterations.
the TinyLlama case, when combining IRM in the second and third iterations. Although we see high winning rate in Figure 5a, a moderate range of ensemble coefficients 7 (i.e. 0.1-0.5) leads to greater performance improvements compared to higher coefficients (i.e., 0.7-0.9). This suggests that the reward margin generated by the latter policy models play a more significant role. However, employing an ensemble of reward margins can effectively mitigate the deterioration caused by un- desirable model bias. Additionally, Figure 5b examines the similarity of selected samples across different y values. The results show that mild levels of 7 only slightly alter the selected samples, with similarity scores ranging from 0.88 0.94. This slight modification crucial as it ensures the retention of important samples while preventing undesired bias in the model.
Selection Behavior. We also checked the difference in the selected sample set with different train- ing configurations such DAAS, iterations, and LLMs. Figure 6a and Figure 6b depict the Jac- card similarity (Murphy, 1996) between two different selected sample sets from different LLMs and DAAS, respectively. All selected sets contain 70% of the samples from the original UltraFeedback dataset. For all axes, the first and second letters indicate the LLM (e.g.. L,, P. and M indicate Tinyl. lama, Pythia-2.8B, and Mistral-7B) and DAA (e.g.. D, I, and S indicate DPO, IPO, and SLiC-HF), and the third number indicates the iteration count. In general, we observe high similarity in the se- lected samples across LLMS (2 0.83) and across DAAS (2 0.74). Across iterations, we also observe a slight variance in the samples selected (0.903 on avg.).
More Preference Bootstrapping. To uncover the full potential of SeRA, we train with more self-generated samples. We fix k at O.7N. We compare the cases with
Ultrafeed Performance
Vinning Rate (36)
77.5
Iter
75.0
k ranging from 0 to 3N. The results in Figure 7 illus- trate that increasing the generated samples up to 2NV leads to performance improvement, highlighting the efficacy of preference bootstrapping. However, we observe a minor drop at 3.N, possibly pointing to undesirable model bias or Suplicaateless-divere sider advanced decoding approaches Huang et al. (2024) to address the latter in the future.
72.5
70.0
0.3 0.7
2.0
e Generation (x Total Dataset Size (N))
generated data. We hope to con-
Figure 7: Pairwise comparison of mod- els with fixed k = 0.7.N and varying k
6 CONCLUSION
In this work, we proposed SeRA that incorporates two components into Direct Alignment Algo- rithms (DAAs)- (1) an implicit reward margin-based sample selection of offline datasets mitigates over-optimization, supported by empirical evidence; and (2) a self-reviewed preference bootstrap- ping to enable direct feedback that considers the a reward ensemble over policy models across iterations (that helps to develop robust reward met- ric) to enable high-quality dataset construction; the ensemble. Extensive experiments on diverse training configurations, across various LLMs, DAAS, and datasets, demonstrated the superior perfor- mance of SeRA. This highlights that even without additional supervision from humans or stronger LLMs, we can squeeze existing offline preference datasets further using policy model's 's metrics (e.g. reward margins) to improve the policy model. We believe that our work demonstrates the possibility of self evolutvolttion of LLMs for achieving super-alignment (Burns et al., 2023) without supervision or with minimal supervision.
mismatch. In addition, we use
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanezyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self- generated mistakes. In The Tweifth International Conference on Learning Representations, 2024.
Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. Large language models for mathematical reasoning: Progresses and challenges. arXiv preprint arXiv:2402.00157 2024.
Sumycong Ahn, Sihyeon Kim, Jongwoo Ko, and Se- Young Yun. Fine tuning pre trained models for robustness under noisy labels. arXiv preprint arXiv:2310.1766., 2023.
Anthropic. The claude 3 model family: Opus, sonnet, haiku. http::://wwwwcc..ccc anthropic.. com/ be 9b0199001la0. wawbb79ac..ccm http:///aa....aaaaaaaaaaaaaabbbb
3. 3.pdf, 2024.
Kushal Arora, Layla El Asri, Hareesh Bahuleyan, and Jackic Chi Kit Cheung, Why exposure bias matters: An imitation learning perspective of error accumulation in language generation. arXiv preprint arXiv:2204.01177 2022.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learn- ing from human preferences. In International Conference on Artificial Intelligence and Statistics, pp. 4447-4455. PMLR, 2024.
Yuntao Bai. Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022a.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et Constitutional ai: Harm- lessness from ai feedback. arXiv preprint arXiv:22
2022b.
George Bennett. Probability inequalities for the sum of independent random variables. Journal of the American Statistical Association, 57(297):33-45, 1962.
Stella Biderman, Hailey Schoelkopf. Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430. PMLR, 2023.
Ralph Allan Bradley and Milton E Terry. Rank analysis incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-345, 1952.
Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbren- ner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-to-stre to-sto general- ization: Eliciting strong capabilities with weak supervision. arXiv preprini urXiv:2312.09390. 2023.
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexci Efros, and Lun-Yan Zhu. Dataset distillation by matching training trajectories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4750-4759 2022.
Chengliang Chai, Jiabin Liu, Nan Tang, Ju Fan, Dongjing Miao, Jiayi Wang. Yuyu Luo, and Guo- liang Li. Goodcore: Data-effective and data-efficient machine learning through coreset selection over incomplete data. Proceedings of the ACM on Management of Data, 1(2):1-27, 2023.
Changyu Chen, Zichen Liu, Chao Du, Tianyu Pang, Qian Liu, Arunesh Sinha, Pradeep Varakan- tham, and Min Lin. Bootstrapping language models with dpo implicit rewards. arXiv preprint 09662406.097600 2024a.
11

---

Published as a conference paper at ICLR 2025
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self- fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01335, 2024b.
Pengyu Cheng, Yifan Yang, Jian Li, Yong Dai, and Nan Du. Adversarial preference optimization. arXiv preprint arXiv:231 2023.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng. Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuma Imsys. org (accessed 14 April 2023), 2(3);6, 2023.
Sayak Ray Chowdhury, Anush Kini, and Nagarajan Natarajan. Provably robust dpo: Aligning language models with noisy feedback. arXiv preprint arXiv:2403.00409 2024.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing sys- Tems, 30, 2017.
Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. In The Twelfth International Conference on Learning Representations, 2024. URL httph:tps//// /openrevv ew. net/ / forum? d=do d=dcjtMYkpXx..
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback, 2023,
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversa- tions. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.). Proceedings the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 3029-3051, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 18653/v1/0233. wer...mmmlppmm.... main. 183. URL https: / awwaaaccanncloooooee org/ 2 2023. emnlp-ma i in. 183.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled al- pacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04477. 2024a.
Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36, 2024b.
Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alexander Nicholas D' A'mour Krishnamurthy Dj Dvijotham, Adam Fisch, Katherine A Heller, Stephen Robert Pfohl, Deepak Ramachandran, Peter Shaw, and Jonathan Berant. Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking. In First Conference on Language Modeling. 2024. URL https: openreview net/ forum? id=5ulGpUkKtG.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306 2024.
Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2009.078858. 2022.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference Machine Learning, pp. 10835-10866. PMLR, 2023.
Cean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Beraaro Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-21284. 2020.
12

---

Published as a conference paper at ICLR 2025
Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model alignment from online ai feedback. arXiv preprint WfXi:2442.00492. 2024.
Jiwoo Hong, Noah Lee, and James Thome. Reference- monolithic preference optimization with odds ratio. arXiv preprini arXiv:2403.07661, 2024.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference on Learning Representations, 2022. URL https::/openteee wptp//veeeeeeeeeeeeeeeeeeeeeeeee net forum? id=nZeVKeeFYE9.
James Y Huang, Sailik Sengupta, Daniele Bonadiman, Yi-an Lai, Arshit Gupta, Nikolaos Pappas, Saab Mansour, Katrin Kirchoff, and Dan Roth. Deal: Decoding- time alignment for large language models. arXiv preprint arXhv:2402.06144 2024.
Albert Q Jiang, Alexandre Sablayrolles. Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint urXiv:2310006825, 2023.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprini arXiv:2401.0400 2024.
Dahyun Kim, Yungi Kim, Wonho Song, Hyeonwoo Kim, Yunsu Kim, Sanghoon Kim, and Chanjun Park. sdpo: Don't use your data all at once. arXiv preprint htt::2403.1927000 2024a.
Dongyoung Kim, Kimin Lee, Jinwoo Shin, and Jachyung Kim. Aligning large language models with self geferated preference data, arXiv preprint arXiv:2406. 04412, 2024b.
Tachyeon Kim, Jongwoo Ko, JinHwan Choi, Young Yun, et al. Fine samples for learning with noisy labels. Advances in Neural Information Processing Systems, 34:24137-241499 2021.
Jongwoo Ko. Sumyeong Ahn, and Se- Yo-Young Yun. Efficient utilization of pre-trained model for learning with noisy labels. In ICLR 2023 Workshop on Pitfalls of limited data and computation for Trustworthy ML. 2023a.
Jongwoo Ko, Bongsoo Yi, and Se- YouYo Yun. A gift from label smoothing: robust training with adaptive label smoothing via auxiliary classifier under label noise. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 8325-8333 2023b.
Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se- Ye-Young Yun. Distillm: Towards streamlined distillation for large language models. arXiv preprint arXhv:2402.0388) 2024.
Ryan Koo, Minhwa Lee, Vipul Rabeja, Jong Inn Park, Zae Myung Kim. and Dongyeop Kang. Benchmarking cognitive biases in large language models as evaluators. arXiv preprint arXiv:2309. 170/2, 2023.
Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathe- matical statistics, 22(1):79-866 86, 1951.
Woosuk Kwon, Zhuohan Li. Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Prin- ciples, pp. 61 611-626, 2023.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.13787 2024.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprini arXiv:2309.00227. 2023.
13

---

Published as a conference paper at ICLR 2025
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-iman image pre- training for unified vision- lanonnla understanding and generation. In International conference on PMLR, 2022.
machine learning, pp.
Raymond Li, Loubna Ben allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia LI, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Joel Lamy-Poirier, Joao Monteiro, Nicolas Gontier, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Ben Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason T Stillenman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey. Zhihan Zhang, Urvashi Bhattacharyya, Wenhao Yu, Sasha Luccioni. Paulo Villegas, Fedor Zhdanov, Tony Lee, Nadav Timor, Jennifer Ding, Claire S Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro Von Werra, and Harm de Vries. Starcoder: may the source be with you! Transactions on Machine Learn- ing Research, 2023a. ISSN 2835-8856. URL / http://pppeeeeeeeeeeee net. fent/foruust moorumiin ?id= KoFOg4 Reproducibility Certification.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models. https:://// http//ggtuuubbbb com/ htt///ttsuuull1
alpaca_evall 2023b.
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lec. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.08 054663399956,..... 2023c.
Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". Openorca: An open dataset of gpt augmented flan reasoning traces. https: petps::/httppe //ttps:h//////////// huggingface. co/ Copppennccceeccpeeeee OpennOrca/eeeeeeeeeeee Pacca/Oppe /OpenOrra..aa. 2023.
Aiwei Liu, Haoping Bai, Zhiyun Lu, Xiang Kong, Simon Wang. Jiulong Shan, Meng Cao, and Lijie Wen. Direct large language model alignment through self-rewarding contrastive prompt distillation. arXiv preprint arXiv:2402.11907 2024a.
Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In International Conference on Machine Learning, pp. 6781-6792. PMLR, 2021.
Sheng Liu, Jonathan Niles- Narges Razavian, and Carlos Fernandez-Granda, Early-learning regularization prevents memorization of noisy labels. Advances in neurul information processing systems, 33:20331-20342, 2020.
Zhihan Liu, Miao Lu, Shenao Zhang, Boyi Liu, Hongyi Guo, Yingxiang Yang, Jose Blanchet, and Zhaoran Wang. Provably mitigating overoptimization in rlhf: Your sft loss is implicitly an adver- sarial regularizer. arXiv preprint arXiv:2405. /6436, 2024b.
Meta AI Llama Team. The llama 3 herd of models. Arxiv, 2024.
Andreas Maurer and Massimiliano Pontil. Empirical bernstein bounds and sample variance pe- nalization. In COLT, 2009, URL http:///bll. mimiitrrer.ce de cabb/con/cco
co1t2009. html 4MaurerP99
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. arXiv preprint arXiv:2405. 14734, 2024.
Aditya K Menon, Ankit Singh Rawat, Sashank Reddi, Seungyeon Kim, and Sanjiv Kumar. A statistical perspective on distillation. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 7632-7642. PMLR, 18-24 Jul 2021. htts::/: htttps://proceediiiiii ppw.eediig....... mlr. www.sppess//. v113/319/// wtwp///eeenna.n/////ee//e/a..aat html.
Baharan Mirzasoleiman, Kaidi Cao, and Jure Leskovec. Coresets for robust training of deep neural networks against noisy labels. Advances in Neural Information Processing Systems, 33:1 1465- 11477, 2020.
14

---

Published as a conference paper at ICLR 2025
Eric Mitchell. A note dpo with noisy preferences & relationship to ipo, 2023.
Allan H Murphy. The finley affair: A signal event in the history of forecast verification. Weather forecsstingg,, 11(1):3-20, 1996.
Junhyun Nam, Hyuntak Cha, Sungsoo Ahn, Jaeho Lee, and Jinwoo Shin. Learning from failure: De-biasing classifier from biased classifier. Advances in Neural Information Processing Systems, 13:20673-20684 2020.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Jane Pan, He He, Samuel R Bowman, and Shi Feng. Spontaneous reward hacking in iterative self- refinement. arXiv preprint Stt::2400.00544,, 2024.
Arjun Panickssery, Samuel R Bowman, and Shi Feng. Llm evaluators recognize and favor their own generations. arXiv preprint 1xXiv:2404.13076 2024.
Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization. arXiv preprint arXiv:2403. 19159, 2024.
Pulkit Pattnaik, Rishabh Maheshwary, Kelechi Ogueji, Vikas Yadav, and Sathwik Tejaswi Mad- husudhan. Enhancing alignment using curriculum learning & ranked preferences. Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Com- putational Linguistics: EMNLP 2024, pp. 12891-12907, Miami, Florida, USA, November 2024. 1
Association for Computational Linguistics. doi: 10. 00.18653/v1/2222 findings-emmll.754. URL
https: /taps::/aclaeee acl antho wogthhloyy......oooooooooooooo.m org. 2024. f f ndi maainngs-en emn 1 p 754.
David Pollard. Convergence of stochastic processes. Springer Science & Business Media, 2012.
Rafael Arehit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2023.
Rafael Rafailov, Yaswanth Chittepu, Ryan Park, Harshit Sikehi, Joey Hejna, Bradley Knox, Chelsea Finn, and Scott Niekum. Scaling laws for reward model overoptimization in direct alignment algorithms, 2024.
Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie. Direct nash optimization: Teaching language models to self-improve with general preferences. arXiv preprint arXivv2404.037755 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint orXiv:/707.06: 2017.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Ste- fano Ermon, Chelsea Finn, and Aviral Kumar. Preference fine-tuning of Ilms should leverage suboptimal, on-pol data, arXiv preprint arXiv:2404.1436 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 2023.
Robert Tibshirani and Bradley Efron. An introduction to the bootstrap. Monographs on statistics and applied probability, 57(1) 1-66, 1993.
Hugo Touvron, Louis Martin, Kevin Stone. Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint urXiv:2307.09288. 2023.
15

---

Published as a conference paper at ICLR 2025
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of Im alignment. arXiv preprint arXiv:2310.16944 2023.
Aad W Van der Vaart. Asymptotic statistics, volume Cambridge university press, 2000.
Binghai Wang, Rui Zheng. Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, et al. Secrets of rlhf in large language models part ii: Reward modeling. arXiv preprint arXiv:2401.0608/ 2024a.
Tiannan Wang, Jiamin Chen, Qingrui Jia, Shuai Wang, Ruoyu Fang, Huilin Wang, Zhaowei Gao, Chunzhao Xie, Chuou Xu, Jihong Dai, et al. Weaver: Foundation models for creative writing. arXiv preprint arXiv:2401.17266 2024b.
Yizhong Wang, Yeganch Kordi, Swaroop Mishra. Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannanch Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13484- 13508. Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/ 11/2023.acl-oog..
URL https:tppp... : / aaclaaaoooaoooooooooooooo wwg/ 2023 acl-long......... 754.
Yixuan Weng. Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao. Large language models are better reasoners with self-verification. arXiv preprint arXiv:2212.095661 2022.
Minghao Wu and Alham Fikri Aji. Style over substance: Evaluation biases for large language models. arXiv preprint vrXiv:2307.03025 2023.
Xiaobo Xia, Tongliang Liu, Bo Han, Mingming Gong, Jun Yu, Gang Niu, and Masashi Sugiyama. Sample selection with uncertainty of losses for learning with noisy labels. In International Confer- ence on Learning Representations, 2022. URL https:/ http////peeeeeeeeeeeeeeeeee net forum?in iid XENE4QUL4IM.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.122444 2023.
Kevin Yang, Dan Klein, Asli Celikyilmaz, Nanyun Peng, and Yuandong Tian. Rled: Reinforcement learning from contrastive distillation for Im alignment. In The Twelfth International Conference on Learning Representations, 2023.
Yongjin Yang, Sihyeon Kim, Hojung Jung, Sangmin Bae, SangMook Kim, Se- Young Yun, and Kimin Lee. Automated filtering of human feedback data for aligning text-to- -mate diffusion models. In The Thirteenth International Conference on Learning Representations, 2025. URL https: htttps://ppentee /tpp//eeeeeeeeeee/////ee/eeeeeee net forum? -
jVVNPHtV..
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho. Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXiv:2401.1002) 2024.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. International Conference on Learning Rep- resentations, 2017. URL https: / opwnreview.....eeoe/eeeee net www.mrrruidooooeeee forum?id-SSyydB3x:
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model, 2024.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Salch, and Peter J Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.10425, 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging jlme with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2023.
16

---

Published as a conference paper at ICLR 2025
Supplementary Material
SeRA: Self-Reviewing and Alignment of LLMs using Implicit Reward Margins
A ADDITIONAL RELATED WORKS
Here, further discuss additional related works regarding our proposed method, SeRA. Specifi- cally, we focus on sample selection approaches and bootstrapping in the context of previous deep learning research, rather than preference optimization of LLMs.
oner-optimization in LLM Alignment. Gao al. (2023) defined over-optimization the phe- nomenon where optimizing a model too aggressively against a static reward model eventually harms the true objective, aligning with Goodhart's 's law. Coste et al. (2024) and Eisenstein al. (2024) proposed using ensembles of multiple reward models to mitigate over-optimization in RLHF fine- tuning. Liu et al. (2024b) introduced regularized preference optimization by adding an SFT loss to prevent werroptimization, making it both simple to implement and effective in practice. Liu et al. (2024a) proposed a contrastive approach that evaluates preference data by comparing output prob- abilities between responses under two contrastive prompts, using self-rewarding contrastive prompt distillation to align LLMs without numan--nnottaan preference data. Recently, Rafailov et al. (2024) showed that over-optimization problems occur not only in RLHF fine-tuning but also in DAAs.
Sample Selection Approach. SeRA, primarily relies on selecting samples based on the internal reward policy models to minimize distribution shift and reduce performance deterioration when using offline preference datasets. While previous efforts have focused on improving training effi- ciency (Cazenavette et al., 2022; Chai et al., 2023) or effectiveness (Kim et al., 2021; Xia et al., 2022) by selecting core subset of the entire training data, and coreset selection (Mirzasoleiman et al., 2020) aims to enhance training efficiency by choosing a small subset that maintains the statis- tical properties of the original dataset, method takes different approach. Instead of just improv- ing training efficiency, we focus on eliminating preference pairs that cause performance degradation due distribution shift. Inspired by work on sample selection for robust training in the presence of noisy labels (Kim et al., 2021; Ko et al., 2023b;a; Ahn et al., 2023), our approach aims to select informative samples that clearly align with human preferences based on IRM. Similar to our work, Yang et al. (2025) tried to select sample with large reward margin by using proxy reward model in diffusion models.
Training with Self-Generated Response. Another main component of our proposed method is exploiting self-generated responses during the training procedure. Agarwal et al. (2024) proposed on-policy distillation, which uses guneettgeneraaee outputs as input for both student and teacher models to reduce distribution shift between training and inference. Our method relates to bootstrap techniques (Tibshirani & Efron, 1993) by having policy models produce preference pairs from given queries. Grill et al. (2020) introduced self-supervised learning framework with two types of net- online and target-wherr the online network predicts the same as the target network on different augmented views. Li et al. (2022) introduced BLIP to pre-train vision-language models by effectively utilizing noisy web data and bootstrapped captions.
Concurrent to work, Chen al. (2024a); Kim al. (2024b) suggested constructing preference samples by training policy LLMs without using external reward models. However, while these works did not explore other DPO variants, such as SLiC-HF and IPO, our work demonstrated their effectiveness on various tasks. Additionally, we provide the statistical grounding of our choice of IRM and empirically enhance the method by introducing an ensemble of different iterations to improve the effecttveness of our IRM-based technique.
B PROOF FOR THEORETICAL INTUITION
Here, we provide the derivation of our theoretical results described in Theorem that deepens the mathematical understanding on how reward margin based sample selection can prevent the over- optimization.
Consider sample defined are two outputs where one is preferred over the other yne > yt- We can now define a margin as f r(æ. yw) r(ar, 91) that encodes the difference is reward assigned the two output
(x, Bwe 31) where is given prompt/input, and (ywW))
17

---

Published as a conference paper at ICLR 2025
trajectories. We can now define risk function R(f) over this margin by leveraging loss function over f as follows:
R() = Eg [((f(s))) = E, Ip" (s)(s)(((((s)))
(10)
As the true preference distribution IP is unknown, we can approximate Equation 10 via the empirical risk over sample a N-sized set of samples S ~ dpd as,
N Σ (((ss)))
R(;S)= = N n=1
(11)
We also represent Equation 10 the Bayes-distilled risk over the sample set S ~ P.V as,
1 N R+(J:S)= = N R=1
Σ ((.))s))(((sss))
(12)
Now, both the standard empirical risk R(f:S) S) (in Equation 11) and Bayes-distilled risk R, (f: (in Equation 12) unbiased estimates the population risk R(.). First, show that the Bayes- distilled risk has lower variance, compared to its empirical risk counterpart.
Lemma B.1 (Menon et al. 2021). For any fixed predictor f - R.
VaNeN (R.(F:)) S Suap [A(::))
where V denotes variance, and equality holds and only is S, the loss values {(((s) are constant on the support p'
Proof. The detailed proof illustrated Lemma 1 of Menon et (2021).
Given Lemma B.1, note that the Bayes-distilled risk more effective estimator that the standard empirical risk due to its small variance. However, since the value of p" (I)) is unknown, we can ap- proximate p" (x:) with p(x). which is the estimated probability produced by the model. Additionally, we approximate Equation 12 using the distilled risk over a sample S ~ p.V as,
R(F
Σ ((s,)Te(f(ss) N
1
Now, we provide the mathematical intuition that IRM-based sample selection can effective by suggesting the upper bound of the distilled risk, computed by p(-). For this, we first definte two terms.
Definition 1 (Covering Number). Let (A, be metric space where distance measure. A set C is an e-cover of A vo E A, By E C such that d(x,, y) < e. The covering number (c, A, d) is defined as size of the smallest c-cover:
set of points and is
N(e,
min[[C] s.1.
an e-cover)
Definition 2 (Uniform Covering Number), For e > 0, a function class H. and an integer N, the uniform covering number. Ne. N, is defined
(o, H,
sup N(e, H(x,)))),1111 TEXN
H} and for A C RN the number N - the smallest cardinality Anl of set An C A such that A contained in the union of e-balls centered at points in Ag in the metric induced by ||- llooo
where H(x) (..
Here, also describe in Lemma B.2 and Lemma B.3 based above definition.
18

---

Published as a conference paper at ICLR 2025
Lemma B.2 (Theorem 6, Maurer & Pontil 2009). Let X be random variable with values in set with X with distribution P and let F be a class of hypotheses f: x - [0, 1]. Fix d E (0,1), N 16 and MN No N. H. 2N). Then, with probability least 1 - 8 over S ~ P.N. we have
V18V log(10.Mx/0)
15 15
P(f) - -N:
V
N
N
Proof. The detailed proof is illustrated in Maurer & Pontil (2009).
Lemma B.3 (Modification from Menon et al. 2021). Pick any bounded loss f. Fix a hypothesss class F of predictors f X + R2,, with induced class H [0, 1] of function h(s) = pp ( Suppose H" has uniform covering number No.. Then, for any 5 E (0, 1), with probability at least over S ~ P.
log(MN/0) log(MN/0) (V
R(S) S (c:)) +0
N
where My No H², 2N, and (f) is the empirical variance of the loss values ((
Proof. Note that the use of Big-O 0 (O) lets us drop the constants and consider 2 instead of 1 from Lemma B.2. Beyond this, this is a simple consequence the uniform convergence version of Bennet's inequality (Bennett, 1962).
N-1
B. PROOF OF MAIN THEOREM
Theorem (Formal Statement). Fix a hypothesis class F of predictors J : S + RR with induced class H. [0, 1 of functions h(s) = (f Suppose H has uniform covering number Noo. Then, for any d E (0, 1), with probability at least 1 - over S,
(VWWL)) log(MN/0) log(Mx/0)
R() < 5) + + (Exl((()) - -p (8) +
N
N
where : H. and V (f) is the empirical variance of the loss values.
Proof. Let R(()) E [) and A R(f:S) R(f). From Lemma. B.3, with probability
1 - 5, following holds:
RR) < R)J;5) + (
log( log(MN/0)
log(MN) /8)
(13)
N
N
where MN Noo N. H. 2N) and VN is the empirical variance of the loss values. Furthermore, the following holds
(R() R()I = b [) - R. (f:
S E [[l((s) (s)]l2 -=((s)la-1((((()))a [|((f()))ll))
(14)
where the last inequality is by the
inttuualtt..... For a constant C. it holds that
E [IlØ(s) p" (s) [I((f())) < E - [ (I((ss
(15)
C (1])(s) p" (s)]l2)
where the first line is by the equivalence of norms. From Eqn. (14) and Eqn. (15), we have R(f) < R(f) + C- [Ilp(s) p" (s)
(16)
By reordering terms to the right-hand side Equation 13 and then adding Equation and Equa-
tion 16, we have:
(1/V.- =0
log(MN/o) log(MN/8) N
R(F) < R(f;S) +
+ C.C. [Ilp(s) - p" (s)
N
19

---

Published as a conference paper at ICLR 2025
[System]
Please act an impartial judge and evaluate the quality of the responses provided by two Al assistants to the user question displayed below. You should choose the assistant that follows the user's instructions and answers the user's question better. Your evaluation should consider factors such as the helpfulness, relevance, accuracy. depth, creativity, and level of detail of their responses. Begin your evaluation by comparing the responses and provide short explanation. Avoid any position biases and ensure that the order which the responses were presented does not influence your decision. Do not allow the length of the responses to influence your evaluation. Do not favor certain names of the assistants. Be as objective as possible. After providing your explanation, output your final verdict by strictly following this format: "[[A]] if assistant A is hetter, "[[B]]" assistant is better, and "[[CI]" for tie.
[User Question] {question)
[The Start of Assistant A's Answer] {answer_a)
('The End of Assistant A's Answer]
[TThe Start of Assistant B's Answer]
(answer_h)
('Th End of Assistant B's Answer]
Figure The pairwise comparison prompt introduced in LLM Judd (Zheng et al., 2023).
Discussion on Upper Bound of Theorem 1. This statement suggests that the upper bound for the risk of the classification function f (i.e., R(f)) can be divided into three terms: empirical risk, the difference between true and estimated probabilities, and function of the uniform covering number. For simplicity, we assume that the function achieves an empirical risk of 0 for both training on selected samples and on the entire set of samples, and we focus the last two terms.
First, as the norm of the difference between the estimated probability p(-) and the true probability ("() grows larger; Our IRM-based sample selection method practically employs approximated p()) for binary cases (iee, O or 1) instead of using p(-) with continuous values. This approach intuitively still results in a smaller
compared to training all samples in the dataset,
only allocate the value 1 for sample s with p(s) ~ 1.
Additionally, the third term grows when either the uniform covering number or the empirical vari- ance of the loss values increases. Training on clearly distinct samples lowers the function complexity and naturally reduces the Lipschitz constant of the corresponding function. According to Van der Vaart (2000) and Pollard (2012), the uniform covering number increases as the Lipschitz constant grows. This implies that a function trained on selected samples will have smaller uniform covering number than function trained on the entire set of samples.
Clarification on the Bradley- Terry Preference Model. While BT models handle preferences stochastically by assigning multiple labels to a single sample, practical implementations for DPO (and other DAAS not based on BT models) are typically deterministic. Due to labeling costs and efficiency considerations, In practice, all annotations are treated as: p(y > y1 x) = for each sample (I, Yw Mt), meaning DPO treats all preference samples equally. Our theoretical results are valid under this practical setup, which aligns with previous empirical findings.
usually train on only few, or even single, preference pair per prompt.
C EXPERIMENTAL SETUP
Here, we elaborate the detailed experimental setup regarding the datasets used (SC.1), training de- tails ($C.2), and evaluation details (SC.3).
C.I DESCRIPTIONS FOR TRAINING DATASET AND EVALUATION BENCHMARK
We apply SeRA on preference datasets and instruction followiinnn datasets. We provide detailed de- scriptions of the datasets used.
20

---

Published as a conference paper at ICLR 2025
[System]
Please act as an impartial judge and evaluate the quality of the response provided by an Al assistant to the user question displayed below. Your evaluation should consider factors such as the helpfulness, relevance, accuracy, depth, creativity, and level of detail of the response. Begin your evaluation by providing short explanation. Be objective possible. After providing your explanation, please rate the response scale of to 10 by strictly following this format: "[[rating]]", for example: "Rating: [[5]]".
[Question] {question}
[Thee Start of Assistant's 's Answer] {answer]
[The End of Assistant's Answer]
Figure 9: The single answer grading prompt introduced in LLM-as-a-Judge (Zheng et al., 2023).
Ultra Chot (instruction-folloving Tunstall et al. 2023 2): This is a heavily filtered version of UltraChat (Ding et al., 2023), originally used to train Zephyr-7B-3 (Tunstall et al., 2023). It is obtained from the original version, which consists of 1.4M dialogues generated by ChatGPT and spans wide range of topics, by removing the dialogues that contain grammatical errors or where the assistant replies with phrases like "I do not have emotions" " or don't 't have opinions."
FltraFeedback (preference dataset; Cui et al. 2023; Tunstall et al. 202334> This is a large- scale, fine- ene-ggrined, and diverse preference dataset used for training powerful reward models and critic models. Cui et al. (2023) collected about 64k prompts from diverse resources, including UltraChat, ShareGPT, and Evol-Instruction (Xu al., 2023). They used these prompts to query multiple LLMs, generating four different responses for each prompt. The responses were anno- tated using GPT-4 to collect high-quality preferences based on instruction-followe truthfulness, honesty, and helpfulness.
We the binarized version (Tunstall et al., 2023), which created by selecting the highest overall score as the "chosen" response and randomly picking one of the remaining three as the "rejected" one, except for Curri-DPO (Pattnaik al., 2024), which was trained on the original version (Cui et al., 2023). The binarized version provides train and test splits of the prompt and response pairs; based on this, we train the models on the train split and evaluate the trained models on the test split.
HH-RLHF (preference datasets; Bai al. 2022a $): This dataset is about human preference garding helpfulness and harmlessness Bai et al. (2022a), and it was originally used to train pref- erence (or reward) models for subsequent RLHF training. Each example in the dataset contains pair of texts, one "chosen" and one "rejected""
TL:DR (preference datasets; Stiennon et al. 2020%): This is the dataset of human feedback that was released for reward modeling. In Stiennon et al. (2020), a reward model was trained using human feedback and then used train summarization model to align with human preferences. The summaries used for training the reward model in the paper come from the TL;DR dataset, with additional validation and test data coming from the TL:DR dataset, CNN articles, and DailyMail articles.
AlpacaEval inrrruction-oolowinne Dubois et al. 2024b ?): This dataset is slight modifications (or simplification) of the AlpacaFarm evaluation set. Dubois et al. (2024b) first merged the instruction and input fields into single instruction field. This affects 1/4 of the examples in the AlpacaFarm evaluation set, all of which are from the Self-Instruct (Wang et al., 2023). This dataset contains 805 challenging questions.
binarized
ottess/sgggggigeeeee///aeessssgp from feeckbe
httpa////aggingeeeee////eeeeeset eval
21

---

Published as a conference paper at ICLR 2025
Vicuna Evaluation (instruction-folloor following: Chiang al. 2023 "): We also use 80 challenging questions that were used for evaluating Vicuna, following Pattnaik et al. (2024).
Evol-Instruct Evaluation (instruction-folloiing Xu et al. 2023 9): Similar to Vicuna, Evol- Instruct (Xu al., 2023) contains 218 questions, spanning multiple topics generated using the Evol-Instruct procedure. Evol-Instruct explicitly provides the category, including math, coding, and reasoning, for each evaluation sample. Based on this, we report the sub-category performance for these categories in Table 11.
C.2 TRAINING DETAILS
Here, we describe the hyperparameters and implementation details for training with SeRA. Our hyperparameters are shown in Tab.4. For Mistral-7B, follow the experimental setup described in the official repository" of Tunstall et al. (2023), except for the rank for LoRA (Hu et al., 2022), changing it to 8. For other models, while matching the effective batch size with Mistral-7B by considering the batch size and gradient accumulation.
the maximum batch size that fits A A00 40GB GPUs,
Table Hyperparameter values used in SeRA SeRA experiments section 4 and section 5.
Hyperparameter
TinLLLMA-L.IB
Pythia-2.8B
Mistral-7B
Fine-tuning method
Full fine-tuning
LoRA (r 8)
Learning
3.0 10 6
5.0 10
DAAS Parameter (3)
0.2 (DPO) / 0.2 (SLiC-HF)/ 1.0 (IPO)
0.01 (DPO)
Batch Size
4
Gradient Accumulation
2
4
4
# Iterations
(1 epoch per iteration)
Selection Proportion (K)
0.7
For the DAAs parameter we search for the optimal values among 0.05, 0.2, 1.0 for TinyLLaMA- 1.IB and reuse it for Pythia-2.8B in all experimental setups. To generate the diverse candidate responses for preference bootstrapping, we sample the responses with a temperature of 0.7 and p of 0.95 for nucleus sampling in training procedure. For all experiments, we generate the 4 response for every single prompt.
C.3 EVALUATION
For evaluating the trained policy LLMs, we applied single NVIDIA A100 40GB GPU for sam- pling the responses from each model using a temperature of 1.0, a max-length limit of 512. For Claude 3 (Anthropic, 2024) evaluation,
the pairwise comparison prompt and the single answer grading prompt which are depicted in Fig. 8 and Fig. 9 with setting the temperature of 0.7. For all evaluation datasets, we utilize the pairwise comparison for comparing preference opti- mized LLMs to corresponding SFT model (SFT in Tab. 1 & Tab. 2) and 1-10 single answer grading. Additionally, for Alpaca evaluation dataset, we further utilize pairwise comparison for comparing preference optimized LLMss to pext-davinci--nn and gpt 4_turbo (G4 in Tab. & Tab. 2) responses. For pairwise comparison, we reported the weighted win rate score, which allocates 1 for a win and 0.5 for tie, following Zheng et al. (2023).
(G3.5 in Tab. & Tab. 2; Li et al. 2023b)
D ADDITIONAL RESULTS
D.1 COMPARISON WITH SELF-REWARDE LMS
Versatility Comparison. We provide the LLM-as-a-Judge prompt, which is introduced Yuan et al. (2024), in Fig. 10. As shown in Tab. 5, when LLMs do not have sufficient capacity follow the instructions, it is difficult to exploit the self-rewarding mechanism introduced Yuan et al. (2024). For example, 11n (Zhang al., 2024) returned meaningless results, which cannot be
wardll testset. testset.jaon! jsonl
22

---

Published as a conference paper at ICLR 2025
Review the user's question and the corresponding response using the additive S-point scoring system described below. Points accumulated based on the satisfaction of each criterion:
Add point if the response is relevant and provides some information related to the user's inquiry, even if it is incomplete or contains some irrelevant content.
Add another point if the response addresses substantial portion of the user's question, but does completely resolve the query provide direct answer.
Award a third point way, regardless of whether it seems to have been written by an AI Assistant or it has elements typically found in blogs or search results.
response answers the basic elements of the user's question a useful
Grant fourth point if the response is clearly written from Al Assistant's perspective, ad- dressing the user's s question directly and comprehensively, and is well-organized and helpful, even there slight room for improvement clarity, conciseness or focus.
Bestow fifth point for response that is impeccably tailored to the user's 's question by an Al Assistant, without extraneous information, reflecting expert knowledge, and demonstrating high-quality, engaging, and insightful answer.
User: <INTTRUCTION HERE
<response>>> INSTRUCTION HERE </response>>
After examining the user's instruction and the response:
-Briefly justify your total score, up to 100 words.
Conclude with the score using the format: "Score: <total points>"
Remember to assess from the Al Assistant perspective, utilizing web search knowledge as neces- sary. To evaluate the response in alignment with this additive scoring model, we'll systematically attribute points based on the outlined criteria.
The LLM-as-a- prompt introduced in Yuan et al. (2024) enables an LLM to act as a reward model and provide self-rewards for its own model generations.
Figure 10:
Table 5: Examples for evaluation results for Self-rewarding LMs (Yuan et al., 2024) prompting in Fig, 10 using TinyLlama-L.. I.IB and Mistral-7B.
Tiny- 1.1B
If the Al Ascistant's 's responsc meets all of the seoting criteria, but it's not obvious you how they ochieved this seare, thes you have high probubility of assigning the points to the right crierion. II, the other hand, the Al Assistant' responsc falls short of meeting all of the criteria, then the Al Asisdaat unlikely to he suinbble assistant for this talk.
Example x
User: Finsl Instrctione: Genenne resuunrr from the AT Assistent that moets the user's expretation. As the user completes submining and given response.
Example
response based the process, wring their questions hunesily with much detail possible.
questio, the system should
keep engaged
Mistral-8
Example
Teral
struttred answer, making solid response
Ovenil, reporee
Al Assicant's persective.
comprehenene unckrstanding mequsted
response comains relevant information requires additional benelii from more examples and details
Tocal Score: 10 (Explutation:
Specitically. snswer long-lerm effests spilk on saaions spesies and crosystoms anower showeases solid undersanding of the impacts of ail spills on constal habitts.)
Example
sheet-orm
Score: ( The respoee provides minimal relevant informstion and lacks depth explaining the impacts ail spills constal hosiitos. It fulls address the key arpects such as the specific consequences on marine life. the process of habinat recovery. and the effectiveness of different remediation inciniques. The snower needs significant imgrovement to demonstrate a thorough undentanding of tepic.
Example
properly used for rating the responses. On the other hand, Mistral-7B (Jiang et al., 2023) might return (i.e., Ex. and 2 vs. Ex. 3) different scales for the ratings, which might also be hard use as a robust rating. While Ex. showed a reasonable rating with the corresponding reason, in Ex. 2, the models returned a total score of 10, which is over the scale (scale of five), and for Ex. 3, the model returned an inconsistent response between the total score and the corresponding reason. Similarly, in the original work of SRLM, the authors conducted experiments on Llama2-70B (Touvron et al., 2023), which has an enormous number of parameters and model capacity. However, our IRM-based preference bootstrapping methods can be widely used regardless of the capacity of trained policy models, demonstrated by their effectiveness in experimental results.
23

---

Published as a conference paper at ICLR 2025
Table 7: Comparison of performance where TinyLlama-I.IB (Zhang al., 2024) is fine-tuned on UltraChat-200k (Ding et al., 2023) and preference pairs are sampled from Ultraeedbback dataset Tunstall al. (2023) with synthetically injected noisy with probability of 0.2.
Alpaca Eval Single
Vicuna Single
Eval SFT
Evol-Instruet
UltraFeed Single
Method
Size
Technique
SFT
Single
SFT
SFT
SFT
5.41
6.05
5.48
4.98
DPO
5.60
50.2
6.43
59.4
5.50
50.2
5.05
55.1
Iterative DPO
5.79
57.0
6.82
65.6
5.75
57.1
5.40
63.5
TinyL.ama
1.1B
cDPO
5.43
47.8
6.23
5,42
53.5
rDPO
6.37
7.06
6.07
63.3
5.73
70.6
IPO
6.37
67.6
7.29
80.0
6.05
68.1
5.56
69.0
SeRA -DPO
6.49
71.2
7.38
81.3
6.24
66.1
5.77
71.1
Table 8: Comparison of performance where TinyLLama-1.11 (Zhang et al., 2024) is fine-tuned UltraChat-200k (Ding al., 2023) and preference pairs sampled from UltraFeedback dataset (Tunstall et al., 2023) with synthetically injected noisy with probability of 0.4.
Alpaca Eval Single
Vicuna Eval Single
Evol-Instruct Single
UltraFeed Single
Method
Size
Technique
SFT
SFT
SFT
SFT
SFT
5.41
6.05
5.48
4.98
DPO
5.52
6.29
50.6
5.58
48.4
4.96
53.6
Iterative DPO
5.45
47.8
6.33
56.3
5.62
55.7
56.2
TinyLlama
1.1B
cDPO
5.42
44.9
6.34
46.3
5.66
47.7
5.06
$2.6
rDPO
5.67
52.4
6.76
68.1
5.92
53.0
5.25
61.0
IPO
5.95
57.9
7.01
78.8
5.90
59.6
5.44
62.2
SeRA -DPO
6.28
66.1
7.21
73.8
5.93
62.6
5.64
68.3
Performance Comparison. We also demonstrated the effectiveness of our self-reviewed preference bootstrap- ping introduced in Sec. 3.3 compared to SRLM (Yuan et al., 2024) on Mistral-7B. As shown in Tab. 6, our reward margin method achieves higher performance (+0.21) on the Vicuna eval and 12.7x greater efficiency evaluation time. This is because SRLM depends on prompting and response generation, which is auto-regressive and computationally inefficient, while ours does not. also confirmed the Jaccard similarity between preference pairs built based on RM (or SRLM) and Claude 3 evaluation upon self-generated responses. As shown in the first column, RM achieves higher similarity with Claude 3 than SRLM that also support that our method is more effective than SRLM.
Table 6: Comparison between ours (RM)
and SRLM (Yuan al., 2024).
Claude 3
Vicuna
Eval. Time
RM SRLM
74%
8.53
4.3h
62%
8.32
54.6h
D.2 FULL NUMERICAL RESULTS FOR FIG.3
We provide the detailed numerical values for Fig. 3 in Tab. 7 and Tab. Our proposed method demonstrated its effectiveness on both 20% and 40% noisy preferences. Specifically, for 40% noisy preference, SeRA achieved higher performance on all evaluation datasets (except for Vicuna Eval on pairwise comparison with SFT) by large margin.
D.3 ADDITIONAL RESULTS ON SIMPO
Recently, Meng et al. (2024) suggested the using the average log probability of a sequence as the implicit reward as follows:
b log T yei).
TSimPo(2; y)
log no (2 y)
(17)
This reward formulation better aligns with model generation and eliminates the need for a reference model, making it more compute and memory efficient. Additionally, they provide new types of
24

---

Published as a conference paper at ICLR 2025
Table 9: Comparison performance where TinyLlama-I.IB is fine-tuned on UltraChat-200k and preference pairs are sampled from UltraFedbback dataset with SimPO (Meng et al., 2024). The best performance highlighted bold.
Alpaca Eval SFT
Vieuna Eval Single
Evol-Instruct
UltraFeed Single
Method
Size
Technique
Single
G3.5
G4
SFT
Single
SFT
SFT
SFT
5.41
28.9
1.7
4,98
Timy
SimPO
6.00
59.4
31.1
2.9
6.92
67.2
5.95
57.3
5.52
64.6
lterative SimPO
6.11
62.3
35.4
3.7
7.14
73.8
6.02
65.7
5.53
66.8
SeRA-SimPO
73.9
4.6
76.5
5.86
71.6
wi Satectoon (Siop 91 (87 3.081
GPT4 Score Selection (Sope 1.M4AA m.077)
Probability Seaerionn (9rp= 022 b.00)
Rewand Margin Selection (Sope in 2:00 (al 0.0m
-40
Soore
ScoN
(Stoe 0.02)
1.77
Revorre Morgin
20
30-
0774 Scane Offerenc Medid
OTT-4 Scne Dffenence (Deeee
OFT-4 Scon Peerce (Chesen Faatedl
0PT4 Ofhrence (Chese Patcheo
(Slac 0.0118 0:131
(Sape 0.01/A 0.181
(Slop 0.02/A" 0057)
15:pc 0.01AA 0.88)
202
201
502
200
402
500
200
4b0
400
2.30
6:31
100
Neporto Lingth
Rasprrs Langth
Easpetee
Desporas Jang3
Figure 11: Extended Results for Figure 2. [Row 1] Correlation between GPT-4 Score & implicit (x,x Vw. using GPT-4 Score
reward (i.e, y(x,yw) yu. [Row 2] Correlation between margin m m(x, & IRM. [Row 2] Correlation between response length (i.e. ly ) and the implicit reward for chosen responses. The model with IRM selection (i.e. [Column 4)) shows the highest R² score for the first and second rows, but the lowest I²2 score for the third row. These consistent results indicate that the IRM-based selection strategy can effectively mitigate over-optimization on response length (Park et al., 2024).
objective function based on their implicit reward as follows:
CsimPo (x. yw Mi) log
| y
log (2, Bw)
B 1y
log me (xx 91)
(18)
Here, we additionally provide the effectiveness of SeRA on SimPO, showcase the versatility of proposed method that can be widely applied to diverse range of implicit reward functions. Same as our experimental setup, we train TinyLlama- I.B on UltraFeedback (Tunstall et al., 2023) dataset with objective function defined in Equation 18. As shown in Table 9, SeRA consistently demon- strates its versatility, even for different types of implicit reward of DAAs. We believe that our proposed method can be applicable to various DAAS, including those that will be developed in the future.
D.4 FULL RESULTS OF FIGURE 2
We also provide the additional correlation between the IRM and the GPT-4 score gap between cho- and rejected responses that share the same prompt, shown in Figure 11. Similar to the relation- ship between implicit reward and GPT-4 score, our IRM-based selection showed a higher It² score compared other baselines (e.g., GPT-4 selection and log probability selection of the reference model).
25

---

Published as a conference paper at ICLR 2025
Table 11: Comparison of performance on the math, coding, and reasoning categories of Evol- Instruct, where TinyLLama--I11B is fine-tuned on UltraChat-200k, and preference pairs are sampled from the Ulrraeeddbcck dataset using DPO Rafailov al. (2023). The best performance high- lighted in bold.
TinyLlama-
Pythia-2.80
Mistral-7B
Category
Math
Coding
Reasoning
Math
Coding
Reasoning
Math
Coding
Reasoning
SFT
4.31
5.22
4.97
4.61
5.73
5.24
7.65
7.34
6.48
DPO
4.25
5.15
3.92
5.78
5.50
3.85
4.47
5.89
6.71
Iterative DPO
4.74
6,05
4.37
5.09
5.28
5.09
7.94
7.71
8,98
Curry-DPO
4.48
6.17
4.51
4.30
5.61
7.32
8.01
7.93
6.64
SeRA-DPS
4.55
6.73
5.15
5.82
5.74
6.06
8.12
8.61
8.19
To further support claim, additionally evaluate R scores between GPT-4 scores and implicit reward, as well as response lengths and implicit rewards, using vaying selection ratios (i.e., from 0.6 to 1.0 in intervals of 0.1) to illustrate the relationship be- tween IRM and spurious correlations. From the results, we can conclude in three directions: (I) Even with fewer training itera- tions at a selection ratio of 0.7, the R²2 score (vs. GPT-4 score) remains comparable to higher ratios (i.e., 0.8 to 1.0). (2) The R score (vs. Length) consistently increases as the selection ratio and
Table 10: R² between implicit reward and GPT-4 score, as well as response length, across different selection ratios
R²
GPT-4
vs. Length
0.6
0.07
0.05
0.7
0.08
0.08
0.8
0.08
0.10
training iterations grow. (3) At selection ratio of 0.6, both R²
0.9
0.08
0.11
scores are lower than 0.7, indicating that the LLM has not yet
1.0
0.08
0.13
fully fit the preference dataset. These findings support that claim that LLMs first learn explicit features before gradually memorizing spurious correlations.
D.5 QUESTION TYPE BREAKDOWN FOR EVOL-INSTRUCT
Our evaluation benchmarks, including Evol-Instruct, Vicuna, and UltraFeedback, contain multiple categories of evaluation samples. We report detailed grading results for sub- cub-categories of code (both code generation and debugging), reasoning, and math problems for Evol-Instruct, as it explicitly categorizes the question type for each evaluation sample. As shown in Table 11, SeRA achieves the highest performance in most experiments, except for reasoning with Mistral-7B, where it ranks second. These results demonstrate that SeRA is consistently effective across diverse sub-categories.
D.6 DISCUSSION ON COMPUTATION BUDGET
The computation budgest introduced by SeRA is mainly divided into three parts: (1) sample selec- tion for the off- off-policy dataset, (2) generation for preference boostrapping, and (3) sample selection for the boostrapped dataset from the trained LLMs. The fine-tuning processs remains computation- ally equivalent to DAAs, including DPO, IPO, SLiC-HE. We report the computational time for each component of SeRA in Table 12.
Table 12: Computational times for each component of SeRA
Process
Selection (Off-policy) Generation
Selection (Bootstrapped)
Training
Time
0.56h
0.38h
0.24h
3.27h
The results were obtained using amachine with A100 (40GB) GPUs on TinyLlama-L. 1. with = 0.7N and = O.3NN For generation, we utilized vLLM (Kwon et al., 2023), an advanced LLM inference framework. Overall, SeRA requires only 36% additional computational budget while achieving a 40%-65% performance gain (see Table 1) compared off- off-policy training, with no extra labeling costs like on-policy training.
26

---

Published as a conference paper at ICLR 2025
D.7 EXPERIMENTAL RESULTS FOR FURTHER ITERATION
To provide a thorough understanding of how the number of iterations affects SeRA's performance, report experiments with additional iterations, specifically from 2 5, evaluated on the Ultra- Feedback benchmark for both single and SFT performance. The experiments were conducted on TinyLlama-I.1B with k = 0.7N and = 0.3.V.
Table 13: Performance of SeRA for further training iterations on UltraFeedback test split with single answer grading (Single) and pairwise comparison compared to SFT model (SFT).
Iteration
Iter. 2
Iter.
Iter. 4
Metrie
Single
SFT
Single
SFT
Single
SFT
Single
SFT
Performance
5.85
70.2
5.92
73.1
5.92
74.5
5.94
74.8
Table 13 suggests that additional iterations may consistently improve performance (as performances for single grading are almost same across iterations 3 to 5), though the benefits diminish as iterations increase. We believe that future work could explore more scalable methods that continue to enhance performance as the number of iterations increases.
27


Reviews:
[{"review_id": "QmfnOyKerG", "reviewer": "Reviewer_FEME", "summary": "This work provide a method SeRA for Direct Alignment Algorithms (DAA). This method mainly uses the DAA policy as a reward model and provide implicit reward margins (IRM) for pairs. This IRM can be used for data selection and can also score the on-policy data to get more data. Many experiments are done to support the performance of this method.", "questions": "See weakness part above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work is well-written and easy to read.\n\n2. The proposed method is simple and don't need further human or AI annotation.\n\n3. This method is suitable for different DAA methods, which makes it a quite general method.\n\n4. The experimental results show the effectiveness.", "weaknesses": "1. As far as I can understand, I think this method mainly uses the DAA model as a reward model to generate on-policy signals. Therefore, I am quite curious about the comparison between SeRA and a direct DAA which uses on-policy preference data. For this direct DAA, the on-policy data might come from GPT-4 or a reward model trained from previous data. \n\n2. I hope more intuition about that SeRA can mitigate over-optimization can be provided. Fig 2 shows that SeRA selected data have less dependency on response length but I am curious why this dependency can be removed.\n\n3. I wonder if there are some other designs for r_t in Equ.(7), Sec. 3.4?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wgtDDFzWeC", "reviewer": "Reviewer_SFx5", "summary": "This paper studied the drawbacks of direct alignment algorithms that overfit the dataset and optimize the model to focus on the spurious correlations. To address these limitations, the authors proposed Self-Reviewing and Alignment which selects preference samples with high implicit reward margins.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to follow.\n2. The experiments are extensive and well support the claims in the paper.", "weaknesses": "1. Since the DPO implicit reward can serve as a good approximation of the preference score, e.g., some of the top reward models in RewardBench [1] are from DPO implicit rewards, relabeling the preference data with DPO reparameterized reward is a natural way, and very much like the standard LLM-as-judge and RM-as-judge methods used to rank the preference pairs. In fact, similar approaches have been explored in [2,3].\n2. The authors may argue that their method uses the implicit reward gap as a data selection metric, instead of only using the implicit reward for relabeling. However, given that implicit rewards are a form of reward, it might be the case that the data selection process mitigates the issues of DPO such as unlearning the high-quality chosen responses when the reward gap is small [2,4], which makes the performance improve. If this is true, then we may use any reward models to filter out similar-quality pairs, or use all preference data with the reward-aware algorithms proposed in [2,4].\n3. The proposed method is computationally expensive since the implicit rewards need to be calculated for each data point, while the iterative DPO does not. Besides, given that the base models (e.g. the SFT models) outperform GPT 3.5 by a large margin and UltraFeedback uses GPT 3.5 for preference ranking, it might be the case that the reason that SeRA outperforms DPO is that the implicit reward is more accurate, even not removing any data. Therefore, I would like to see how iterative DPO performs when using DPO implicit reward to rank the pairs.\n4. Based on the above comments, the evidence that SeRA is better than DPO since it addresses the spurious correlations issue may be weak.\n5. It is also unclear what Theorem 1 indicates and how it relates to the filtering process of SeRA.\n\n[1] Lambert et al., RewardBench: Evaluating Reward Models for Language Modeling.\\\n[2] Zhang et al., Reward-Augmented Data Enhances Direct Preference Alignment of LLMs.\\\n[3] Zhong et al., DPO Meets PPO: Reinforced Token Optimization for RLHF.\\\n[4] Nvidia, Nemotron-4 340B Technical Report.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cnsXAamZ5C", "reviewer": "Reviewer_pPvp", "summary": "The paper presents a new algorithm called Self-Reviewing and Alignment (SeRA) for direct aliment algorithms, which can be adapted to different preference optimization methods such as DPO, SLiC, and IPO. The two components of SeRA are sample selection and preference bootstrapping. Instead of using human feedback, AI judgment, or an external reward model to label the preference data, SeRA uses an implicit reward model for the sample selection and data bootstrapping. The paper provides a theoretical guarantee for DPO. Also, the paper presents extensive empirical studies across different datasets, model architectures, and alignment frameworks to show the effectiveness of SeRA.", "questions": "* Is there an estimation of the uniform covering number for various hypothesis classes, such as linear functions and neural networks?\n\n* Can the theoretical results presented in the paper be extended to apply to other SAA methods?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper presents both theoretical and empirical validation, covering a variety of models, datasets, and SAA methods. Furthermore, the component analysis is thorough, evaluating most elements of the algorithm for a more nuanced understanding.\n*  The IRM method enables training SAA iteratively without relying on external reward signals. This innovation simplifies the algorithm’s implementation and reduces the computational resources required.\n* The paper is well-structured, making the concepts accessible and the methodology easy to follow.", "weaknesses": "* The theoretical upper bound on the risk of the DPO is only weakly connected to the proposed SeRA. Additionally, the theoretical analysis does not account for the impact of multiple iterations on the algorithm's performance.\n\n* The paper does not provide a thorough study into how the number of iterations affects SeRA's performance. While Figures 4 and 7 indicate that iter 3 consistently outperforms iter 2, it remains unclear whether further increasing the number of iterations would lead to additional performance improvements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QOgkYzrK81", "reviewer": "Reviewer_AqA6", "summary": "Direct alignment algorithms (DAAs) like direct preference optimization (DPO) face challenges with static off-policy data, leading to spurious correlations and overfitting. In this paper, the authors propose a novel method called SeRA that addresses the mentioned issue by using sample selection and preference bootstrapping to improve alignment. The authors use extensive experiments to show SeRA enhances DAA performance on the Ultrafeedback dataset.", "questions": "1. In Line 246, the authors claim that the inherent problem of DPO is that DPO treats all preference samples equally, even when the true preferences are not correct. In the BT preference model, the annotation is stochastic instead of deterministic, which means that this phenomenon should not be an issue from the theoretical perspective. Could the authors give a more detailed explanation?\n\n2. The proposed method requires sampling multiple times for each prompt in the dataset. I wonder if the baseline methods and the proposed method have the same computation budget.\n\n3. In this paper, the authors consider introducing a filtration on self-generated responses to get a preference pair for each prompt. However, I am curious about what would happen if the authors use all the self-generated responses (e.g. N responses) in pairs to create N(N-1)/2 preference pair. Such an idea is used in SPPO ([1]), hence a comparison between the proposed method in this paper and SPPO may be important.\n\n4. Since the authors use the Ultrafeedback dataset for training, I am curious about the meaning of the 'Ultrafeedback' benchmark. Is this the test split of the Ultrafeedback? \n\n\n[1] Wu, Yue, et al. \"Self-play preference optimization for language model alignment.\" arXiv preprint arXiv:2405.00675 (2024).", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper provides a clear motivation for the authors' proposed method from both theoretical and empirical perspectives. The experimental results show that their proposed method has a significant improvement on the alignment task given the Ultrafeedback dataset and the authors perform extensive empirical analysis to explain the mechanism behind the proposed method.", "weaknesses": "1. The paper lacks more detailed discussions on the literature analyzing the overoptimization issue for LLM alignment and RLHF with self-generated data. For example, the authors could discuss the literature [1]-[5] in this paper.\n\n\n2. Compared with other papers that study LLM alignments (e.g. [5]. [6]), this paper lacks empirical results on the academic benchmarks in coding, reasoning, and math. The authors should evaluate their proposed methods on these academic benchmarks.\n\n3. It is a little confusing that the authors leave Remark 1 in Line 254 without an in-depth explanation of the derived bound. The derived bound seems to be adapted from the common analysis of empirical risk minimization. If there are novel analytical techniques for the current problem (analyze the performance of the proposed method), the authors should highlight the novel part and show that the vanilla DPO fails to have such a bound. \n\n4. Minor issues or suggestions:\n(1) In the comment on Line 14 of Algorithm 1, you should better mention the 'implicit reward' instead of 'reward'.\n(2) More descriptions of benchmarks should be given. (even in the appendix)\n\n\n[1] Coste, Thomas, et al. \"Reward Model Ensembles Help Mitigate Overoptimization.\" The Twelfth International Conference on Learning Representations.\n\n[2] Liu, Zhihan, et al. \"Provably mitigating overoptimization in rlhf: Your sft loss is implicitly an adversarial regularizer.\" arXiv preprint arXiv:2405.16436 (2024).\n\n[3] Eisenstein, Jacob, et al. \"Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking.\" arXiv preprint arXiv:2312.09244 (2023).\n\n[4] Liu, Aiwei, et al. \"Direct large language model alignment through self-rewarding contrastive prompt distillation.\" arXiv preprint arXiv:2402.11907 (2024).\n\n[5] Wu, Yue, et al. \"Self-play preference optimization for language model alignment.\" arXiv preprint arXiv:2405.00675 (2024).\n\n[6] Chen, Zixiang, et al. \"Self-play fine-tuning converts weak language models to strong language models.\" arXiv preprint arXiv:2401.01335 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)