Self-Exploring Language Models: Active Preference Elicitation for Online Alignment
Authors: Shenao Zhang
Donghan Yu
Hiteshi Sharma
Han Zhong
Zhihan Liu
Ziyi Yang
Shuohang Wang
Hany Hassan Awadalla
Zhaoran Wang
Venue: Accepted by TMLR ()

Abstract: Preference optimization, particularly through Reinforcement Learning from Human Feedback (RLHF), has achieved significant success in aligning Large Language Models (LLMs) to adhere to human intentions. Unlike offline alignment with a fixed dataset, online feedback collection from humans or AI on model generations typically leads to more capable reward models and better-aligned LLMs through an iterative process. However, achieving a globally accurate reward model requires systematic exploration to generate diverse responses that span the vast space of natural language. Random sampling from standard reward-maximizing LLMs alone is insufficient to fulfill this requirement. To address this issue, we propose a bilevel objective optimistically biased towards potentially high-reward responses to actively explore out-of-distribution regions. By solving the inner-level problem with the reparameterized reward function, the resulting algorithm, named Self-Exploring Language Models (SELM), eliminates the need for a separate RM and iteratively updates the LLM with a straightforward objective. Compared to Direct Preference Optimization (DPO), the SELM objective reduces indiscriminate favor of unseen extrapolations and enhances exploration efficiency. Our experimental results demonstrate that when fine-tuned on Zephyr-7B-SFT and Llama-3-8B-Instruct models, SELM significantly boosts the performance on instruction-following benchmarks such as MT-Bench and AlpacaEval 2.0, as well as various standard academic benchmarks in different settings.

Published in Transactions on Machine Learning Research (03/2025)
Self-Exploring Language Models: Active Preference Elicitation for Online Alignment
Shenao Zhang', Donghan Yu², Hiteshi Sharma?, Han Zhong Zhihan Liu Ziyi Yang?, Shuohang Wang?, Hany Hassan², Zhaoran Wang!
"Northwestern University *Microsoft 8 "Peking University
Reviewed on OpenReview: https: // openreview. net/ forum? Fd=FoQBB43
Abstract
Preference optimization, particularly through Reinforcement Learning from Human Feedback (RLHF), has achieved significant success in aligning Large Language Models (LLMs) to adhere to human intentions. Unlike oflline alignment. with a fixed dataset, online feedback collection from humans or Al model generations typically leads to more capable reward models and better-aligned LLMs through an iterative process. However, achieving globally accurate reward model requires systematic exploration to generate diverse responses that span the vast space of natural language. Random sampling from standard reward-maximizing LLMs alone is insufficient to fulfill this requirement. To address this issue, propose bilevel objective optimistically biased towards potentially high-reward responses to actively explore out-of-distributioo regions. By solving the inner-level problem with the reparameterized reward function, the resulting algorithm, named Self-Eaploring Language Models (SELM), eliminates the need for separate RM and iteratively updates the LLM with straightforward objective. Compared to Direct Preference Optimization (DPO), the SELM objective reduces indiscriminate lavor of unseen extrapolations and enhances exploration efficiency. Our experimental results demonstrate that when fine-tuned on Zephyr-7B-SFT and Llama-3- 8B-Instruct models, SELM significantly boosts the performance on instructtinn-folloiing benchmarks such as MT--Bench and AlpacaEval 2.0, as well as various standard academic benchmarks different settings.
1 Introduction
Large Language Models (LLMs) have recently achieved significant success largely due to their ability follow instructions with human intent. As the de facto method for aligning LLMs,, Reinforcement Learning from Human Feedback (RLHF) works by maximizing the reward function, either a separate model (Ouyang et al., 2022; Bai et al., 2022; Gao et al., 2023) reparameterized by the LLM policy (Rafailov et al., 2024b;a; Azar et al., 2023; Zhao et al., 2023), which is learned from the prompt-response preference data labeled by humans. The key to the success of alignment is the response diversity within the preference data, which prevents reward models (RMs) from getting stuck in local optima, thereby producing more capable language models.
Offline alignment methods (Rafailov et al., 2024b; Tang et al., 2024) attempt to manually construct diverse responses for fixed prompts (Cui et. al., 2023; Ivison el. al., 2023; Zhu et al., 2023), which, unfortunately, struggles to span the nearly infinite space of natural language. On the other hand, online aligument follows an ilerative procedure: sampling responses from the LLM and receiving feedback to form new preference data for RM training (Ouyang et al., 2022; Guo et al., 2024). The former step helps explore out-of-distribution (OOD) regions through randomness in sampling. However, in standard online RLHF franeworks, maximizing

---

Published in Transactions on Machine Learning Research (03/2025)
the expected reward learned from the collected data is the only objective for the LLM, sampling from which often leads responses clustered around local optima. This passive exploration mechanism can suffer from overfitting and premature convergence, leaving the potentially high-reward regions unxxplored.
To address this issue, we propose an active exploration method for online alignment that elicits novel favorable responses. In its simplest form, an optimism term a maxy r(z, v) is added to the reward-fitting loss (e-g., the negative log-likelihood L1, on dataset D), resulting in bilevel optimization objective for the reward model r: max max or( y) - (r;
(1)
where a is a hyperparameter controlling the degree of optimism. The intuition illustrated in Figure 1. Specifically, minimizing the vanilla reward-fitting loss Lr is likely to give a locally accurate RM that overfits the observed data and gets stuck in local minima. Random sampling from this vanilla RM may take a long time to explore the OOD regions that contain the best response. By incorporating the optimism term, we obtain an RM that both fits the data well and has a large max, r(x, v). This ensures that the greedy respouse from it is either globally optimal when uncertainty in high-reward regions is eliminated, or potentially good in unexplored areas where r(x, Mu) can be arbitrarily huge due to the relaxed reward-fitting loss. Feedback from humans on these responses Vu can then reduce uncertainty and train a more accurate R.M.
DONwAD observed data
optimistic RM
vanilla RM
uncertainty
ground-truth
y'
response y
Figure 1: Intuition of our method. For a fixed prompt a reward model r(x, tries to fit the ground-truth reward r" (zz 3). The blue and green RMs are equally good when using standard reward-fitting loss Ci,, since the observed preference data (red stars) are fitted equally well. However, the green RM has a larger max, r(x, u) and thus a lower optimistically biased loss Cn, - a max,, r(x, y). Therefore, the response Yo. that maximizes the optimistic RM can be elicited and proceeded for human feedback to reduce the uncertainty.
In this paper, we formulate this idea within the context of online direct alignment, where the LLM is iteratively updated without a separate RMM. We first introduce two modifications to the bilevel RM objective in (1), namely adding KL constraints and using relative maximum reward. Then we derive a simple LLM training objective by applying the closed-form solution of the inner-level problem and reparameteriving the reward with the LLM policy. The resulting iterative algorithm is called Delf-Eaploring Language Models (SELM). We show that the policy gradient of SELM is biased towards more rewarding areas. Furthermore, by reducing the chance of generating responses that are assigned low implicit rewards, SELM mitigates the indiscriminate favoring of unseen extrapolations in DPO (Rafailov et al., 2024b;a) and enhances exploration efficiency.
In experiments, we implement SELM using Zephyr-7B-SFT (Tunstall et al., 2023b) and Clama-3--B-IInstreee (Meta, 2024) as base models. By fine-tuning solely on the UltraFeedback (Cui et al., 2023) datasct and using the small-sized PairRM (Jiang el al., 2023) for ilerative AI feedback, SELM boosts the performance of Zephyr-7B-SFT and Llama-3-8B-Instruct by a large margin on AlpacaEval 2.0 (Dubois et al., 2024) (+16.24% and +11.75% LC win rates) and MT-Bench (Zheng et al., 2024) (+2.31 and +0.32) in 3 iterations. SELM also demonstrates strong performance on standard academic benchmarks and achieves higher pairwise LC win rates against the strong iterative DPO baseline.
2

---

Published in Transactions on Machine Learning Research (03/2025)
2
Related Work
Data Synthesis for LLMs A key challenge for fine-tuming language models to align with users' intentions lies in the collection of demonstrations, including both the mnstructionfolloviin expert data and the preference data. Gathering such data from human labelers is expensive and sometimes suffers from variant quality (Ouyang et al., 2022; Köpf et al., 2024). To address this issue, synthetic data (Liu et al., 2024a) has been used for aligning LLMs One line of work focuses on generating plansible instruction prompts for unlabeled data by regarding the target output as instruction-folloiing responses (Li et al., 2023a; Wu et al., 2023; Josifoski et al., 2023; Taori et al., 2023; Li al., 2024a). Besides, high-quality data can also be distilled from strong models for fine-tuning weaker ones (Gunasekar et al., 2023; Abdin et al., 2024; Li et al., 2023b; Ding et al., 2023; Peng ct al., 2023). To construct synthetic datasets for offline RLHF, popular pipeline (Cui et al., 2023; Tunstall et al., 2023b; Wang et al., 2024b; Tvison et al., 2023; Zhu et al., 2023) involves selecting responses sampled from various LLMs on set of prompts in the hope to increase the diversity of the data that can span the whole language space. However, data manually collected in such a passive way does not consider what improves the model most through its training, leaving the potentially high-reward regions unexplored.
Iterative Online Preference Optimization. Compared to offline RLHF algorithms (Rafailov et al., 2024b; Zhao et al., 2023; Azar el al., 2023) that collect preference datasels ahead of training, online RLHF (Ouyang et al., 2022; Guo et al., 2024), especially the iterative/batche el al., 2023; Chen et al., 2022; Gulcehre et al., 2023; Hoang Tran, 2024; Xiong el al., 2023; Calandriello et al., 2024; Rosset et al., 2024) has the potential to gather better and better synthetic data as the model improves. As a special case, self-aligned models match their responses with desired behaviors, such as model-generated feedback (Yuan et al., 2024; Yuanzhe Pang et al., 2024; Sun et al., 2024; Wang et al., 2024a). Unfortunately, the above methods still passively explore by relying on the randomness during sampling and easily get stuck at local optima and overfit to the current data due to the vast space of natural language. A notable exception is Dwaracherla et al. (2024), which proposed to use ensembles of RMs to approximately measure the uncertainty for posterior-sampling active exploration. On the contrary, our method explores based on the optimistic bias and does not estimate the uncertainty explicitly, bypassing the need to fit multiple RMs.
online RLHF (Bai al., 2022; Xu
Active Exploration. In fact, active exploration has been widely studied beyond LLMs. Similar Dwaracherla et al. (2024), most existing sample-efficient RL algorithms first estimate the uncertainty of the environment using historical data and then either plan with optimism (Auer, 2002; Russo & Van Roy, 2013; Jin et al., 2020; Mehta et al., 2023; Das et al., 2024), or select the optimal action from a statistically plausibly set of values sampled from the posterior distribution (Strens, 2000; Osband et al., 2013; 2023; Zhang, 2022; Li et al., 2024c). The proposed self-cxploration objective can be categorized as an optimism-based exploration method. However, most previous works involve explicitly maintaining confidence sets that contain the ground truth with high probability, which poscs considcrable challenges for implementation especially in high-dimensional spaces such as natural language. In contrast, our method only optimizes a single objective that integrates reward estimation and planning to automatically balance exploration and exploitation.. Ensemble methods (Osband et al., 2024; Chua et al., 2018; Lu & Van Roy, 2017) can serve as approximations to estimate the uncertainty but are still computationally inefficient.
Concurrent Work. We highlight the concurrent work (to the first version of the current paper) of Xie et al. (2024); Cen et al. (2024); Liu el al. (2024c), among which Xie et al. (2024) establishes the first analysis of the sample complexity of a DPO algorithm in the online setting of RLHF (formulated as MDPs). All of them focus on incorporating an SFT loss or a similar term (as bonus or penalty) alongside the DPO loss as an optimistic or pessimistic adjustment in the online or offline setting, respectively. Xie et al. (2024); Cen et al. (2024) and the current paper focus on the former, while Liu et al. (2024c) focuses on the latter. In the second version of the current paper, we provide the sample complexity of SELM following the proof technique of Xie et al. (2024). Through a reduction technique from Xie et al. (2024), we show how to connect the sample complexity of SELM to that of existing RL algorithms (Zhong et al., 2022; Liu et al., 2024b), which are not tailored to RLHF but enjoy strong theoretical guarantees.
3

---

Published in Transactions on Machine Learning Research (03/2025)
3 Background
Large Language Models. A language model Α typically takes the prompt X' as imput and outputs the response y E Here, X' and are finite spaces of prompts and responses, respectively. Given the prompt & E X, a discrete probability distribution x(- a) E Ay is generated, where Ay is the set of discrete distributions over y. After pretraining and Supervised Fine-Tuning (SFT), preference alignment is employed to enhance the ability of the language model to follow instructions with human intentions.
Reinforcement Learning from Human Feedback (RLHF). Standard RLHF frameworks consist. of learning a reward model and then optimizing the LLM policy using the learned reward.
Specifically, point-wise reward r(r, v) x - R represents the Elo score (Elo & Sloan, 1978) of the response y given the prompt Then the preference distribution can be expressed by the Bradley-Terre Terryy model that distinguishes between the preferred response yo and the dispreferred response y given prompt x, denoted as yo > Yt x, using the logistic function a:
plun y1 [a) = En [1(h prefers yho over yr given x:]
exp(r(x, yw))
= = Uw) r(z, = exp(r(æ, vw)) exp(r(x,w)))
(2)
where denotes the human rater and the expectation over account for the randommess of the choices human raters we ask for their preference. When provided a static dataset of N comparisons D = {xi, Ywris w the parameterized reward model can be learned by minimizing the following negative log- lieelikeihood loss:
[r;(r: = -B log a (r(x, r(x, y.)].
In the following sections, mainly consider the above formula of reward models that is learned from preference pairs. Using the learned reward, the LLM policy ग E Aÿ is optimized with reinforcement learning (RL) to maximize the expected reward while maintaining a small deviation from some base reference policy Trets i.e. maximizing the following objective
[(T) EEw (r(,,) v)] ] - 3DKL (I (ret);
(4)
where 3 is hyperparameter and (KL.(T || weer) == En- [KL(T(- (x) || Tyo((: x))] is the expected Kullback- Leibler (KT.) divergence. An ideal nee is the policy that helps mitigate the distribution shift issue (Rafailov et al., 2024b; Guo et al., 2024) between the true preference distribution and the policy æ during the off-policy RL training. Since we only have access to the dataset D sampled from the unavailable true preference distribution, Toof can be obtained by fine-tuning the preferred responses in D or simply setting nee SFT and performing RLHF based on the SFT model.
Direct Alignment from Preference. With the motivation to get rid separate reward model, which is computationally costly to train, recent works (Rafailov et al., 2024b; Azar et al., 2023; Zhao et al., 2023; Tunstall et al., 2023b; Ethayarajh al., 2024) derived the preference loss function of the policy by changing variables. Among them, DPO (Rafailov al., 2024b) shows that when the BT model in (2) can perfectly fit. the preference, the RLHF objective in (4) is equivalent to:
[ D) = -E Me BD) log a Bog TBu xx) - B log ा. T(8 xx) Tor (Nw xx) Trer x))
)
4

---

Published in Transactions on Machine Learning Research (03/2025)
4 Self-Exploring Language Models
4.1 RM-Free Objective for Active Exploration
In this section, we present several modifications to the optimistically biased objective (1) motivated in the introduction. Then we derive an RM-free objective for the LLM policy and analyze how active exploration works by examining its gradient.
First, we consider the equivalence of (1): max, -Lir(r;)) deterministic when optimal. To account for the change of relative to the reference policy Trefs we introduce two modifications: (1) replacing the optimistic bias term max; Eyy=/r(<; v)] with max= By r(x, and (2) incorporating KL-divergence loss term between and The to minimize the deviation between ı and Rref- Switching to relative rewards creates stronger incentive for exploration, they become significant only when π discovers responses that outperform those from Tref- Intuitively, it diverges from (1) primarily scenarios where the reward of responses from x is high, but the relative reward is low. This signals that responses from Te also receive high rewards and further exploration in this region is unnecessary.
max, Ey~ y)}; where the inner is
Formally, for the reward r, the bilevel optimization problem with optimism is formulated as:
max -r; + a max EaND, r(x, y) - r(x, w/) - BDKL Rret)
55)
F(T:)
where D. = {Ti, Vw 1i is the associated dataset at iteration I and Lir is defined in (3). The nested optimization in (5) can be handled by first solving the inner optimization F(T;r) r) to obtain n, that is optimal under r. The solution is follows and defer all the derivations in this section to Appendix A.
1
( x)) argmax F(TTr) (x) Trotl x) exp(r(s,u)/3)
where the partition function (,(x) Ey (1)() exp(r(x, v). By substituting = nr into F(T;r), we rewrite the bilevel objective in (5) as single-level onc:
max (rrr aF(mir)
Following the implicit reward formulation in DPO, we reparameterize the reward function with 0 E O as Fo(2, v) 3llog mo æ) log Trer(y æ)), which is the optimal solution of (4) and can express all reward classes consistent with the BT model as proved in (Rafailov al., 2024b). With the above change of variable, we obtain the RM-free objective for direct preference alignment with optimism:
max CpPo Dx)
log To(y x)].
(6)
We now analyze how this new objective encourages active exploration. Specifically, we derive the gradient of with respect to 8 as
A (E
( (FF((I, fo( vw)) log x) Va log (9 x))
VoLoPo(TA:D))
-a (exp((xxe v) log Rely x)].
(7)
We note that the second line, corresponding the gradient of the optimism term, decreases the log- ligllikl of response generated by vg that has a high value of exp(-fo(r,.)/B) Therefore, the added optimism term biases the gradient toward parameter regions that can elicit responses with high implicit reward fa. consistent with intuition outlined in Figure 1.
5

---

Published in Transactions on Machine Learning Research (03/2025)
This also explains why Ex Te] is minimized in objective (6), which is equivalent to maximizing the KL divergence between Fye and ne, while the reverse KL in the policy optimization objective (4) is minimized. For the DPO gradient VoCD Dt), the degree of deviation of policy œe from Frel only affects the preference estimated with fg. In other words, affa (z, s1) - fa(x, yw)) )) is a scalar value and the policy deviation only determines the step size of the policy gradient, instead of its direction. On the other hand, our added exploration term directly controls the direction of the gradient toward potentially more rewarding areas while still fitting the preference data in Dr. As more feedback data is collected iteratively, deviating from the unbiasedly fitted model incurs a higher DPO loss, which ultimately dominates our objective at convergence. This mechanism ensures that the resulting LLM effectively balances between exploring novel responses and exploiting previously observed ones, leading to more accurate and aligned model.
4.2 Algorithm
With the optimistically biased objective derived above, the language model can actively generate OOD responses worth exploring. Human or AI foodback follows reduce the uncertainty in these regions. These two steps are executed iteratively to get a more and more aligned model.
In practice, we split the offline preference dataset into three portions with equal sizes, one for each iteration. Besides, we use AI rankers, such external RMs, to provide feedback the model-generated response and the original chosen, rejected responses. The complete pseudocode of our algorithm, named Ealf-Exploging Language Models (SELM), is outlined in Algorithm 1. In experiments, we set T to 3.
Algorithm Self-Exploring Language Models (SELM)
Input: Reference model Tref, preference dataset D, online iterations T', optimism coefficient a. for iteration 1.2.....T do
Set D, as the 1-th portion of D and generate y ~ Trer(- x)) for each prompt x in D.. 3: Rank {y, yw 3 and update D, to contain the best (chosen) and worst (rejected) responses. 4: Train the LLM me = argmax,,, { CDPO (T8; Dz) - a.EEeD,, [log Ta(y x)]}, let Tref Te,
end for
5 Analysis
5.1 Sell-Exxloration Reduces Indiscriminate Favor of Unseen Extrapolations
It has been observed recently (Rafailov et al., 2024a; Pal et al., 2024; Xu et al., 2024) that DPO decreases the likelihood of responses generated by the reference policy. It because for any prompt T, at convergence when + Tref, it holds that
Ey (Z = Eyywner [log (o | .:) - log Tre x)] KL (x) TT( (
x)) < O,
while at the beginning of training when my Reefs the above terms zero. Thus, the expected implicit reward Pg well as the likelihood of me will decrease on the reference model's responses. This indicates that DPO stimulates a biased distribution favoring unseen extrapolated responses. In the online iterative setting that we consider, the LLM policy generates responses and receives preference feedback alternately, where biasing toward OOD regions may sometimes help discover outstanding novel responses. However, DPO indiscriminalely favors unseen extrapolations and passively explores based purely on the randomness inherent in sampling from the LLM. As a consequence, the vast space of natural language makes it almost impossible to exhaustively explore all the possible responses and identify those that most effectively benefit. alignment.
Next, we demonstrate that SELM mitigates this issue by performing guided exploration. Specifically, consider the proposed self-exploration objective in (6) that is derived from (5), which, in addition to the standard

---

Published in Transactions on Machine Learning Research (03/2025)
DPO loss, also minimizes Ez.y [log Toly x)]. We now investigate how the probability distribution changes with this term incorporated.
Theorem 5.1. For any e θ the policy parameter space, let Fp(x, y) = (log ( x) - log Teer(y x)) be the reparameterized implicit reward. Denote xp reward under the KL constraint, i.e.,
min as the policy that minimizes the expected implicit
min (- 2) = argmin E ( [f v)] + 5I (T Tp)-
(8)
Then minimizing E, y Dog To (y(æ)) decrcases the likelihood of responses sampled from apth
min Bu [log (1 [x] min Buyoruu (-Iw) [log ( [)].
The proofs for theorems in this section can found in Appendix and C. The above theorem states that maximizing the divergence between Te and Tret is essentially reducing the probability of generating responses with low implicit rewards reparameterized by any policy parameter p during training. In other words, the LLM policy not only exploits the existing preference data but also learns to avoid generating the text that is assigned a low reward value. This process occurs in every iteration with updated reference models. Consequently, responses with high potential rewards are selectively preferred and many commonplace responses receive small probability mass, thus mitigating the indiscriminate favoring of unseen responses and improving the exploration efficiency. In the next section, we will formally prove that the self-exploration mechanism sample-effciint.
5.2 Self-Exploration is Provably Effile-Efficient
Following the proof technique of Xie al. (2024), we provide the sample efficiency of the self-exploration mechanism by establishing sublinear cumulative regret. Specifically, the cumulative regret R(T) up to T iterations is defined as the cumulative performance discrepancy between the learned policy me at iteration and the optimal policy n² (with respect to the regularized objective in (4)) over the run of the algorithm:
T R(T)= = (("") -
The key idea is reduction technique from Xie et al. (2024), which connects the sample complexity of SELM to that of existing RL algorithms (Zhong et al., 2022; Liu et al., 2024b). is worth noting that the theoretical version of the exploratno mechanism (Algorithm 2) a bit different from the practical one used in the numerical experiments and is closer to the proposed algorithm in Xie et al. (2024).
Assumption 5.2 (Realizable Policy Class with Regularity Condition). We assume access to finite policy class containing the optimal n². Moreover, we assume that for any æ E Il and prompt-response pair (2, (2,v),
Tl x)) Tre (y xx)
log
S Rmax
Assumption 5.2 stipulates that IT sufficiently comprehensive to include the optimal policy. Additionally, it imposes bounded condition on log(=/Trre); which has been identified as the implicit reward function for DPO (Rafailov et al., 2024b). It naturally extends to the infinite case by introducing concepts like the covering number. An example of an infinite policy class is the log- linerr policy class, whose complexity is approximately d (sce, c.g., Zanette et al. (2021) for a dotailed dcrivation).
Theorem 5.3. Under Assumption 5.2, let 7 = / /TdGGEC/ and 8 (0, 1). Then with probability at least 5, have
log( (I))), a = (/(eexpp exp(44mmxx))
R(T) $ VdRGEC Rmmak) T Plg M//8)
7

---

Published in Transactions on Machine Learning Research (03/2025)
where X omits absolute constants, and dPGEC is preference-based version of Generalized Eluder Coefficient (GEC; Zhong et al., 2022) defined in Appendix C.1 capturing the complexity of learning problem. For log-linear policy class II = {mo me (y x) x exp((0(z, v). 8)) with d-dimensional feature Φ, it holds that dpGEC - O(d).
The proof technique is from Xie al. (2024), which connects RLHF with RL and allows us to use the preference- based version of GEC (Zhong et al., 2022; Liu et al., 2024b) as the complexity measure to characterize the cumulative regret R(T). We restate the proof technique from Xie et al. (2024) for completeness. We emphasize that is not a novel contribution of the present work. Since the cumulative regret is sublinear in the number of iterations T, the above theorem indicates that the policy converges to the optimal x within sufficient iterations. Moreover, by the standard online-to-batch argument, Theorem 5.3 shows that SELM is capable of finding an e-optimal policy with a sample complexity of Ö(1/s²). This highlights the sample efficiency of SELM from the theoretical perspective.
6
Experiments
6.1
Experiment Setup
We adopt UltraFeedback (Cui et al., 2023) as our training dataset, which contains 61k preference pairs of single-turn conversations. For the external ranker during online alignment, we choose the small-sized PairRM (0.4B) (Jiang al., 2023). All experiments are conducted on 8xA100 GPUs.
Due to the absence of performant open-source online direct alignment codebases at the time of this study, we first implement an iterative version of DPO as the baseline, adhering to the same steps Algorithm 1 but training the LLM with the standard DPO objective. Then conduct grid search over hyperparameters, such the batch size, learning rate, and iteration number, identify the optimal settings for the iterative DPO baseline. We follow these best settings to train SELM. In addition, we apply iterative DPO and SELM on instruction fine-tuned models. Specifically, we consider two series of LLMs: Zephyr (Tunstall et al., 2023b) and Llama-3 (Meta, 2024), to demonstrate the robustness of SELM. Since the official Zephyr-7B-8 model is fine-tuned with DPO on the same UltraFeedback dataset, to avoid overoptimization, we choose Zephyr-7B-STT' as the base model and perform 3 itcrations of SELM after a single itcration of standard DPO training on the first portion of the training data (we refer to this model as Zephyr-7B-DPO) For Hama-3-8B-Instruce that is already fino-tuned with RLHF, directly apply itcrations of SELM training.
6.2 Experiment Results
We first report the performance of SELM and the baselines on the Enaruuctionfollowinnn chat benchmarks AlpacaEval 2.0 (Dubois et al., 2024) and MT-BBench (Zheng et al., 2024) in Table 1. We can observe that for AlpacaEval 2.0, SELM significantly boosts Zephyr-7B-SFT and Llama-3-8E InIttuuut achieving length-controlled (LC) win rate improvements of +16.24% and +11.75%, respectively. This enhancement results in models that are competitive with or even superior to much larger LLMs, such as Yi-34B-Chi Chat (Young et al., 2024) and Llama-3-70pB Instructt For the multi-turn MT-Bench, which exhibits higher variance, report the average scores of SELM and DPO baselines across runs. We observe that SELM improves the scores by +2.31 and +0.32, respectively. Furthermore, the proposed method self-explores and enhances the model monotonically, with consistent performance improvements in cach iteration. This validates the robustness of our algorithm. Compared to other iterative post-training algorithms, such as SPIN (Chen et al., 2024), DNO (Rossct et al., 2024), and SPPO (Wu et al., 2024), SELM gains more improvements both
hhttps///bugggigffcccoo//// http://ggggaaaaccc////////////// httmm//saaaa---a-tbbeee hhtp:s//tuuggingiiee.mm/mmettt --a
8

---

Published in Transactions on Machine Learning Research (03/2025)
benchmarks when using the weaker base model (Zephyr-7B-SFT), and achieves the best performance when using
the base model.
Alpaca Eval Eval 2.0
MT-
Model
LC Win Rate
Win Rate
Avg.
Avgerage
1st Turn
2nd Turn
7ephyr-7B-SFT
8.01
4.63
916
5.30
5.63
1.97
Zephyr-7B-DPO
15.41
14.44
1752
7.31
7.55
7.07
DPO Tter (Zephyr)
20.53
16.69
1598
7.53
7.81
7.25
DPO Iter 2 (Zephyr)
22.12
19.82
1717
7.55
7.85
7.24
DPO Iter 3 (Zephyr)
22.19 (114.18)
19.88
1717
7.46
(12.16)
7.85
7.06
SELM Iter (Zephyr)
20.52
17.23
1624
7.53
7.74
7.31
SELM Iter 2 (Zephyr)
21.84
18.78
1665
7.61
7.85
7.38
SELM Iter 3 (Zephyr)
24.25 (1165(1))
21.05
1694
7.61 (12.31)
7.74
7.49
Mama-3-SB-
22.92
22.57
1899
7.93
8.47
7.38
DPO Iter (Llama3-It)
30.89
31.60
1979
8.07
8.44
7.70
DPO Iter 2 (Llama3-It)
33.91
32.95
1939
7.99
8.39
7.60
DPO Iter 3 (T.ama3-Tt)
33. (110.25)
32.18
1930
8.18 (10.25)
8.60
7.77
SELM Iter (Llama3-1t)
31.09
30.90
1956
8.09
8.57
SELM Iter 2 (Llama3-It)
33.53
32.61
1919
8.18
8.69
7.66
SELM Iter 3 (Llama3-It)
34.67 (111.75)
34.78
1948
8.25 (10.32)
8.53
7.98
SPIN
7.23
6.54
1426
6.54
6.94
6.14
Orca-2.5-SFT
10.76
6.99
1174
6.88
7.72
6.02
DNO (0rca-2.5-SFT)
22.59
24.97
2228
7.48
7.62
7.35
Mistral-7B In--uuttutt--00
19.39
15.75
1565
7.51
7.78
7.25
SPPO (Mistral-it)
28.53
31.02
2163
7.59
7.84
7.34
Yi-34B-Chat
27.19
21.23
2123
7.90
Ilama-3-70B-Imtrruet
33.17
33.18
1919
9.01
9.21
8.80
GPT-4 Turbo (04/09)
55.02
46.12
1802
9.19
9.38
9.00
Table 1: Results AlpacaEval 2.0 and MT-BBech averaged with 3 runs. Names inside the brackets are the models that aligned based upon. The red arrows indicate the increment or decrement from the base model. Compared to iterative DPO and other online aligument baselines, SELM gains more improvements based on the weaker Zephyr-7B-SFT model and achieves superior performance that is competitive with much larger SOTA models when fine-tuned from Maima-3-8B-Instruue
Notably, the implemented itcrative DPO is obtained through comprchensive grid scarches of hyporparameters and practical designs (see Appendix D for details), making it a strong baseline comparable with SOTA online alignment algorithms fine-tuned from more advanced models, such SPIN, DNO, and SPPO. For example, DPO Iter 3 (Zephyr) achieves an MT-BBench score of 7.46, representing a 2.16 improvement over Zephyr-SFT (5.30) and coming close to DNO (7.48), which is fine-tuned from the stronger model Orca-2.5-SFT (6.88). Additionally, SPPO achieves an MT-BBench score of 7.59, a modest improvement of 0.08 over Mistral-it (7.51).
We also conduct pairwise comparisons between SELM, iterative DPO, and the base models to validate the effectiveness of our method. The results for AlpacaEval 2.0 shown in Figure 2. We observe that with the same number of training iterations and data, SELM consistently outperforms the iterative DPO counterpart. Additionally, when using 7B-SFT-7B-SFT as the base model, SELM outperforms iterative DPO even when the latter is trained with twice the data.
Beyond instruction-follovii benchmarks, also evaluate SELM and the baselines on several academic benchmarks, including GSMSK (Cobbe al., 2021), HellaSwag (Zellers et al., 2019), ARC challenge (Clark et al., 2018), TruthfulQA (Lin et al., 2021), FQQBBecc (Paech, 2023), and OpenBookQA (OBQA) (Mihaylov et al., 2018). To better reflect the capabilities of LLMs, we adopt various settings for these benchmarks, including vero-shot, few-shot, and few-shot Chain-of-Thougeee (CoT) settings. The accuracy results for these

---

Published in Transactions on Machine Learning Research (03/2025)
Zephyr-7B-DPO
3ama-3-8B-Insee
SELM Iter 3
50.00
52.85
55.93
53.64
53.88
56.54
66.02
SELM
50.00
51.79
51.96
52.18
52.69
53.41
61.39
SELM Iter 47.15
55.60
58.33
63.56
DPO Iter3
48.21
50.00
50.10
50.46
52.28
53.71
60.70
SELM Iter 1 44.07
44.40
50.00
52.32
49.65
53.91
64.43
SELM Iter
284..04
49.90
50.00
51.24
51.91
52.91
60.57
DPO Iter3.
46.36
46.74
47.68
50.00
49.78
52.25
61.58
DPO Iter2
47.82
49.54
48.76
50.00
51.30
53.82
60.52
46.12
DPO Iterl
47.31
47.72
49.75
48.70
50.00
50.20
59.62
DPO Iterl 43.46
41.67
46.09
47.75
48.31
50.00
60.24
SELM Iter 1
46.59
46.29
47.09
46.18
49.80
50.00
59.23
Zephyr-78-DPO
33.98
34.44
35.57
38.42 38.71
39.76
50.00
Llmm33IIt 38.61
39.30
39.43 39.48
40.38
40.77
50.00
SELM Iter
SELM iter
SELM Iter 1
DPO Iter
DPO Iter 2.
DPO Iter 1 Zephyr-7B-DPO
SELM iter 3
DPO Iter 3
SELM Iter
DPO Iter
DPO Iter 1
SELM Iter 1
Llama3-It
Figure 2: Pairwise comparison between SELM, iterative DPO, and base models. Scores represent the LC win rates of the row models against the column models. Models positioned in higher rows have higher LC win rates against the base model and thus better performance.
multiple-choice QA benchmarks are provided in Table Tt can be observed that both our method and the basclines can degrade after the RLHF phase on some benchmarks, which is known as the alignment tax (Askell et al., 2021; Noukhovitch et al., 2024; Li et al., 2024b). Nevertheless, our method is still able to improve the base models most of the benchmarks and offers the best overall performance.
We note that SELM is one of the instantiations of the proposed self-exploration objective in (1), with reparameterized reward functions and algorithm-specfic designs described in Section 4.2, such as the dataset partition and update rule. However, this objective is not restricted to the current implementation and also be directly applied to any other online alignment framework, with without separate reward model, regardless of differences in algorithm designs. Thus, the proposed method is orthogonal and can be integrated directly into the recent online RLHF workflows (Dong et al., 2024; Xiong et al., 2023; Hu et al., 2024) that incorporate additional delicate designs with carcfully curated datascts.
6.3 Ablation Study
We first provide ablation studies better understand the explorative optimism term. We begin by investigating the effect of the optimism coefficient a. In Figure 3 (Left), we plot the LC win rates of SELM when using Zephyr-7B-SPP
the base model for different in the AlpacaEval 2.0 benchmark. We find that setting small a, such as 0.0001, leads to very similar behaviors to the iterative DPO (a 0) baseline, while SELM with a large a may become overly optimistic and thus not very effective. These results meet our expectations, suggesting that proper values of a are essential for achieving the best trade-off between exploration and exploitation.
Next, study the difference in reward distributions with varied a and iterations. Specifically, for prompts from the 2k test set of UltraFeedback, we greedily sample from the LLM and generate rewards for the responses with PairRM. We then caleulate the fraction of data that lies in each partition of rewards. The results for different a values of SELM Iter 2 (Zephyr) in Figure 3 (Middle) indicates that increasing a results in distributions that concentrated in higher-reward regions.
10

---

Published in Transactions on Machine Learning Research (03/2025)
GSMSK
HellaSwag
ARC
TruthfulQA
EQ
OBQA
Models
Average
(8-s CoT)
(10-s)
(25-s)
(0-s)
(0-s)
(10-s)
Zephyr-7B-SFT
43.8
82.2
57.4
43.6
39.1
35.4
50.3
Zephyr-7B-DRD
47.2
84.5
61.9
45.5
65.2
38.0
57.0
DPO Iter (Zephyr)
45.5
85.2
62.1
52.4
68.4
39.0
DPO Iter (Zephyr)
44.9
85.4
62.0
53.1
69.3
39.4
59.0
DPO Iter 3 (Zephyr)
43.2
85.2
60.8
52.5
69.1
39.6
58.4
SELM Ieer (Zephyr)
46.3
84.8
62.9
52.9
68.8
39.6
59.2
SELM Iter (Zephyr)
46.2
85.4
62.1
53.1
69.3
39.6
59.3
SELM Iter 3 (Zephyr)
43.8
85.4
61.9
52.4
69.9
39.8
58.9
Baama-3-8B-Instiiee
76.7
78.6
60.8
51.7
61.8
38.0
61.3
DPO Iter 1 (Llama3-1t)
78.5
81.7
63.9
55.5
64.1
42.6
64.4
DPO Iter 2 (Llama3-It)
79.4
81.7
64.4
56.4
64.3
42.6
64.8
DPO Iter 3 (Llama3-it)
80.1
81.7
64.1
56.5
64.1
42.6
64.8
SELM Tter (Llama3-It)
78.7
81.7
61.5
55.4
64.1
42.4
64.5
SELM
(Llama3-It)
79.3
81.8
64.77
56.5
64.2
42.6
64.9
SELM Iter (Llama3-It)
80.1
81.8
64.3
56.5
64.2
42.8
65.0
SPIN
44.7
85.9
65.9
55.6
54.4
39.6
57.7
Mistral-7BB Beraal-B--nsstreee Instruct-v0.2
43.4
85.3
63.4
67.5
65.9
41.2
61.1
SPPO (Mistral-it)
42.4
85.6
65.4
70.7
56.5
40.0
60.1
Table 2: Performance comparison between SELM and the baselines on academic multi-choice QA benchmarks in standard zero-shot, few-shot, and CoT scttings. Here, n-s refers to n-shot. The red and blue texts represent the best and the second-best results.
Ablation of AlpacaEval 2.0
Reward Distribution Different
Reward Distribution Shift
0.19
24-
0.01
a= 0.01
SELM Rer
0.001
SELM Iter 2
23
0.0005
Fraction of data
0.17
0.0005
0.0001
d 0.0001
0.10
22
0.08
21-
0.13
0.02
0.00
0.00
Itaration
Reward from PairRM
PairRM
Figure Ablation on the optimism coefficient a and the change of the reward distribution. Left: The length-controlled rates of SELM with different a on AlpacaEval 2.0. Middle: Comparison of reward distribut ions at iteration 2 with different a. Right: SELM initially explores and then shifts to higher-reward regions as more training iterations are performed.
Additionally, Figure (Right) demonstrates that the reward distribution shifts to the right (higher) training iterations are performed. This shift corresponds an initial exploration phase, where the LLM generates uncertain responses of varying quality, followed by an exploitation phase as feedback incorporated and more training data is collected.
We also conduct ablation studies the implicit reward captured by the SELM and DPO models. Recall that for both SELM and DPO, the implicit reward takes the form of fz y) = (log Te x) log Fre((y) x)). We calculate the reward difference FSELM (x, y) fDro 3) for each prompt æ in the Ultraleeddback holdout test set. Here, we study the implicit reward of the good (chosen) and bad (rejected) responses, so y y == и/ We then sort the reward difference and plot the results for Zephyr-based models after iteration in Figure 4. The plot clearly shows that for both chosen and rejected responses, SELM produces higher implicit rewards compared to DPO, aligning with the proposed optimistically biased self-cxploration objective.
11

---

Published in Transactions on Machine Learning Research (03/2025)
Reward Difference on Chosen Yw
Reward Difference on Rejected yr
(fSELM - -Ppro) Yw)
(fSELM (oopo) yi)
0.6
0.4
0.4
0.2
0.0
0.0
-0.2
-0.2
500
1000
1500
2000
500
1000
1500
2000
Test data index
Test data index
Figure 4: Difference of implicit reward between SELM and DPO on the chosen and rejected responses. SELM assigns higher implicit reward than DPO for both responses.
Reward Difference
In Section 5, show that. SELM engages in more tive exploration by prioritizing high-reward responses compared to DPO, which indiscriminately favors un- seen extrapolations and explores passively. To validate this, we sample three responses from SELM and DPO Iter 2 (Zephyr) for each prompt and we calculate the subtraction of the mean implicit rewards. As illus- trated in Figure 5, SELM consistently achieves higher implicit rewards across most prompts, with the positive reward differences being notably larger in magnitude, supporting our claim regarding SELM's active explo- ration behavior. We refer the readers to Appendix E for more ablation studies, including the effect of prompts across iterations, preference pair construction, and optimism bias versus data selection.
2.0
E [i] E [f]
1.5
1.0
0.5
0.0
0.5
1.0
500
1000
1500
2000
Test data index
Figure 5: SELM actively explores by favoring high- reward responses.
7 Conclusion & Future Work
In this paper, we introduced an active preference elicitation method for the online alignment of large language models. By incorporating an optimism term into the reward-fitting objective, the proposed bilevel self- exploring objective effectively balances between exploiting observed data and exploring potentially high-reward regions. Unlike slandard online RLHF algorithms that passively explore the response space by sampling from the training LLM, whose sole objective maximizing the expected learned reward, our method actively seeks diverse and high-quality responses. This self-exploration mechanism helps mitigate the risk of premature convergence and overfitting when the reward model is only locally accurate. To optimize this bilevel objective, solve the inner-level problem and reparameterize the reward with the LLM policy, resulting in simple yet novel iterative alignment algorithm called Belf-Exploring Languoge Models (SELM). Compared to DPO, SELM is provably sample-efficient and improves the exploration efficiency by selectively favoring responses with high potential rewards rather than indiscriminately sampling unseen responses.
Our experiments, conducted with Zephyr-7B-SFT and of SELM with consistent improvements on AlpacaEval 2.0, MT-Bench, and academie benchmarks. These results underscore the ability of SELM enhance the alignment and capabilitics of LLMs by promoting more diverse and high-quality responses. Since the proposed technique is orthogonal to the adopted online RLHF workflow, it will be interesting to apply our method within more sophisticated alignment frameworks with advanced designs, which we would like to leave future work.
models, demonstrate the efficacy
12

---

Published in Transactions on Machine Learning Research (03/2025)
References
Yasin Abbasi- Yadkor,, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. Advances in neurol information processing systems, 24, 2011.
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et. al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint orXiv:244.112119 2024.
Alekh Agarwal, Sham Kakade, Akshay Krishnamurthy, and Wen Sun. Flambe: Structural complexity and representation learning of low rank mdps. Advances in neural information processing systems, 33: 2020.
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. general language assistant as a laboratory for alignment. Xiviv preprint arXic:2212.00880 2021.
Peter Auer. Using confidence bounds for exploitation-exxllraaion trade-offs. Journal of Machine Learning Research, 3(Nov):397-44) 4722, 2002.
Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Danicle Calandricllo, Michal Valko, and Rémi Munos. A general theoretical paradigm to understand learning from human preferences. arXiv preprint Xrx::2330. 000000.220000 12036, 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint 2XK::2224405862, 2022.
Daniele Calandriello, Daniel Guo, Remi Munos, Mark Rowland, Yunhao Tang, Bernardo Avila Pires, Pierre Harvey Richemond, Charline Le Lan, Michal Valko, Tianqi Liu, al. Human alignment of large language models through online preference optimisation. arXiv preprint arXiv:2403.0680 08635, 2024.
Shicong Cen, Jincheng Mei, Katayoon Goshvadi, Hanjun Dai, Tong Yang, Sherry Yang, Dale Schuurmans, Yuejie Chi, and Bo Dai. Value-incentivized preference optimization: unified approach to online and offline rlhf. arXiv preprint ar drXic:240511932 2024.
Xiaoyu Chen, Han Zhong, Zhuoran Yang, Zhaoran Wang, and Liwei Wang. Human-in-the-loop: Provably efficient preference-basee reinforcement learning with general function approximation, International Conference on Machine Learning, pp- 3773 -3933. PMLR, 2022.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint 0011:32401.01335, 2024.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. Advances in neural information processing systems, 31, 2018.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. arXie preprint arXiv:1803.054235 2018.
Karl Cobbe, Vineet. Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworck, Jacob Hilton, Reichhro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint or Xin:21 10. 14168, 2021.
13

---

Published in Transactions on Machine Learning Research (03/2025)
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01377, 2023.
Nirjhar Das, Souradip Chakraborty, Aldo Pacchiano, and Sayak Ray Chowdhury. Provably sample efficient rlhf via active preference optimization. arXiv preprint arXiv:2402.10500, 2024.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint ar Xiv:23305 14239, 2023.
Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf. arXiv e-prints, pp- arXiv-2405, 2024.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint S00::24440044400 2024.
Vikranth Dwaracherla, Scyed Mohammad Asghari, Botao Hao, and Benjamin Van Roy. Efficient exploration for llms. arXiv preprinl arXiv:2402.00396, 2024.
Arpad E Elo and Sam Sloan. The rating of chessplayers: Past and present. Ishi Press International, 1978.
Kawin Ethayarajh, Winnie Xu, Niklas MuennighofT, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International PMLR, 2023.
Conference on Machine Learnöng, pp.
Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. Reinforced self-training (rest) for language modeling. arXiv preprint orXiv:2308.008 08998, 2023.
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need. arviv preprint 2rx:: 11644, 2023.
Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model alignment from online ai feedback. arXiv preprint arXiv:2402. 04792, 2024.
Braden Hancock Hoang Tran, Chris Glaze. Shorkelmiistral-pairm--- 2024. URL https://huggingfac.
Jian Hu, Xibin Wu, Weixun Wang, Xianyu, Dehao Zhang, and Yu Cao. Openrlhf: An easy-to-use, scalable and pegf-performance rlhf framework, 2024.
Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A Smith, Iz Beltagy, et al. Camels in changing climate: Enhancing Im adaptation with tulu 2. Xixe preprint orXiv:2811.10702 2023.
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. Llm-blender: Ensembling large language models with pairwise ranking and generative fusion. arXiv preprint ar XrXiv:2306. 02561, 2023.
Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael Jordan. Provably efficient reinforcement Icarning with linear function approximation. In Conference on learning theory, pp. 2137-21 2143. PMLR, 2020.
14

---

Published in Transactions on Machine Learning Research (03/2025)
Martin Josifoski, Marija Sakota, Maxime Peyrard, and Robert West. Exploiting asymmetry for synthetic training data generation: Synthie and the case of information extraction. arXiv preprint arXiv:2303.04192 2023.
Dahyun Kim, Yungi Kim, Wonho Song, Hyeonwoo Kim, Yunsu Kim, Sanghoon Kim, and Chanjun Park. sdpo: Don't use your data all at once. arXiv preprint 2rXiv:2403.12270, 2024.
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Due Nguyen, Oliver Stanley, Richard Nagyfi, et al. Openassistant conversations denoortaiiddd large language model alignment. Advances in Newral Information Processing Systems, 36, 2024.
Jiaxiang Li, Siliang Zong, Hoi-To Wai, Chenliang Li, Alfredo Garcia, and Mingyi Hong. Getting more juice out of the sft data: Reward learning from human demonstration improves sft for 1lm alignment. arXiv preprint Xrx::2400. 1888, 2024a.
Shengzhi Li, Rongyu Lin, and Shichao Pei. Multi-modal preference alignment remedies regression of visual instruction tuning on language model. ar preprint 10..2440.1884 2024b.
Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Luke Zettlemoyer, Omer Levy, Jason Weston, and Mike Lewis. Self-alignment with instruction backtranslation. arXw preprint arXiv:2308.0620 2023a.
Yingru Li, Jiawei Xu, Lei Han, and Zhi-Quan Luo. Hyperagent: A simple, scalable, elficient and provable reinforcement learning framework for complex environments. arXiv preprint ST5T:22402.1222 2024c.
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.05. 05466, 2023b.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint Xiv::21109 07958, 2021.
Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang, Jimmeng Rao, Steven Zheng, Daiyi Peng, Diyi Yang, Denny Zhou, and Andrew M. Dai. Best practices and lessons learned on synthetic data for language models, 2024a.
Zhihan Liu, Miao Lu, Wei Xiong, Han Zhong, Hao Hu, Shenao Zhang, Sirui Zheng, Zhuoran Yang, and Zhaoran Wang. Maximize to explore: One objective function fusing estimation, planning, and exploration. Advances in Neural Information Processing Systems, 36, 2024b.
Zhihan Liu, Miao Lu, Shenao Zhang, Boyi Liu, Hongyi Guo, Yingxiang Yang, Jose Blanchet, and Zhaoran Wang. Provably mitigating overoptimization in rlhf: Your sft loss is implicitly an adversarial regularizer. arXiv preprint (rX6::2/05.16436 2024c.
Xiuyuan Lu and Benjamin Van Roy. Ensemble sampling, Advances in neural information processing systems, 30, 2017.
Viraj Mchta, Vikramject Das, Ojash Neopane, Yijia Dai, Ilija Bogunovic, Jeff Schneider, and Willie Neiswanger. Sample efficient reinforcement learning from human feedback via active exploration. 2023.
Meta. Introducing meta 1lama 3: The most capable openly available 1lm to date. 2024. URL https:
//ai www.a.a.cccommmm...... mootta.co//boo/mmmee-laaaa-3-3
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? new datasct for open book question answering. arXiv preprint 1rxxx:1809. 009.02789...........900..9. 2018.
Michael Noukhovitch, Samuel Lavoie, Florian Strub, and Aaron C Courville. Language model alignment with elastic reset. Advances in Neural Information Processing Systems, 36, 2024.
15

---

Published in Transactions on Machine Learning Research (03/2025)
Ian Osband, Daniel Russo, and Benjamin Van Roy. (more) efficient reinforcement learning via posterior sampling. Advances in Neural Information Processing Systems, 26, 2013.
lan Osband, Zheng Wen, Seyed Mohammad Asghari, Vikranth Dwaracherla, Morteza Ibrahimi, Xiuyuan Lu, and Benjamin Van Roy. Approximate thompson sampling via epistemic neural networks. In Uncertainty in Artificial Intelligence, pp. 1586 1595. PMLR, 2023.
Tan Osband, Zheng Wen, Seyed Mohammad Asghari, Vikranth Dwaracherla, Morteza Tbrahimi, Xiuyuan Lu, and Benjamin Van Roy. Epistemic neural networks. Advances in Neural Information Processing Systems, 36, 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730 27744, 2022.
Samuel J Paech. Eq-bench: An emotional intelligence benchmark for large language models. arXiv preprinl arXiv:2312.0624 2023.
Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with dpo-positive. arXiv preprint arxi:24422. 100e2.13.22 13228, 2024.
Baolin Pong, Chumyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arxiv preprint ar arxi:23004.030 03277, 2023.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantago--wightee regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiw:1910.000 00177, 2019.
Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From r to q²: Your language model is secretly a q-function. arXiv preprint arXiv:2444.12958 2024a.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024b.
Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie. Direct nash optimization: Teaching language models to self- reff-improve with general preferences. arXiv preprint SKK::2444.03114 2024.
Daniel Russo and Benjamin Van Roy. Eluder dimension and the sample complexity of optimistic exploration. Advances in Neural Information Processing Systems, 26, 2013.
Malcolm Strens. A bayesian framework for reinforcement learning. In ICML, volune 2000, pp. 943 950, 2000.
Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. Principle-driven self-aligument of language models from scratch with minimal human supervision. Advances in Neural Information Processing Systems, 36, 2024.
Yunhao Tang, Daniel Zhaohan Guo, Zeyu Zheng, Daniele Calandriello, Yuan Cao, Eugene Tarassov, Rémi Munos, Bernardo Ávila Pires, Michal Valko, Yong Cheng, et al. Understanding the performance gap between ouline and offline alignment algorithuns arXiv preprint ar (05 2024.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instructionnfollloinng Ilama model. https://github. tatau-lab/stanforz laainnfdddallpaca_aaaaaaaaaaaaae 2023.
16

---

Published in Transactions on Machine Learning Research (03/2025)
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Shengyi Huang, Kashif Rasul, Alexander M. Rush, and Thomas Wolf. The alignment handbook. https://githuu.. www.:/uwww...ooogggggggg//////// weignennt-handbooe 2023a.
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Youess Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of Im alignment. arXiv preprint arXiv:2310.16944, 2023b.
Xiyao Wang, Jiuhai Chen, Zhaoyang Wang, Yuhang Zhou, Yiyang Zhou, Huaxiu Yao, Tianyi Zhou, Tom Goldstein, Parminder Bhatia, Furong Huang, et al. Enhancing visual-language modality alignment in large vision language models via self-immprvement. arXiv preprint arXiv:2405.15977 2024a.
Yizhong Wang, Hamish Tvison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, al. How far camels go? exploring the state of instruction tuning on open resources. Advances Neural Information Processing Systems, 36, 2024b.
Yuanhao Wang, Qinghua Liu, and Chi Jin. Is rlhf more difficult than standard rI? arNiv preprint 0rXiv23366.14111 2023.
Shengguang Wu, Keming Lu, Benfeng Xu, Junyang Lin, Qi Su, and Chang Zhou. Self-evolved diverse data sampling for efficient instruction tuning. arXiv preprint arXiv:2311.08182 2023.
Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment. arXie preprint arXiv:2405.000 000755 2024.
Tengyang Xie, Dylan J Foster, Akshay Krishnamurthy, Corby Rosset, Ahmed Awadallah, and Alexander Rakhlin. Exploratory preference optimization: Harnessing implicit q f--apprxximation for sample-efficient rlhf. Xiv preprint orXiv:2405.1000 2024.
Wei Xiong, Hanze Dong, Chenlu Ye, Han Zhong, Nan Jiang, and Tong Zhang. Gibbs sampling from human feedback: A provable kl-constrained framework for rlhf. arXie preprint arXi::2322. 11456, 2023.
Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, and Jason Weston. Some things are more cringe than others: Preference optimization with the pairwise cringe loss. arXiv preprint arXic:2312.16682 2023.
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for Ilm alignment? comprehensive study. arXiv preprint arXiv:3404.10719 2024.
Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2433.044 2024.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston Self-rewarding language models. arXiv preprint arXiv:2401.10022 2024.
Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Tterative reasoning preference optimization. arXie e-prints, pp. arXiv-2404, 2024.
Andrea Zanette, Martin J Wainwright, and Emma Brunskill. Provable benefits of actor-critic methods for offline reinforcement learning. Advances in neural information processing systems, 34:13626-136400, 2021.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXim:1905.078900 2019.
Shenao Zhang. Conservative dual policy optimization for efficient mosel-based reinforcement learning. A Advances in neural information processing systems, 25463, 2022.
17

---

Published in Transactions on Machine Learning Research (03/2025)
Tong Zhang. From E-entropy to kl-entropy: Analysis of minimum information complexity density estimation.
The Annals of Statistics, pp. 2180 2210, 2006.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiw preprint arXiv:2305. 10425, 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
Han Zhong, Wei Xiong, Sirui Zheng, Liwei Wang, Zhaoran Wang, Zhuoran Yang, and Tong Zhang. Gec: A uni- fied framework for interactive decision making in mdp, pomdp, and beyond. arXiv preprint ar Xr1:22211.0096,, 2022.
Han Zhong, Guhao Feng, Wei Xiong, Ti Zhao, Di He, Jiang Bian, and Liwei Wang. Dpo meets ppo: Reinforced token optimization for rlhf. arXiv preprint ar 2rXiv:2444.1892, 2024.
Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao. Starling-7h: Improving Ilm helpfulness and harmlessness with rlaif, November 2023.
A Derivations in Section 4.1
We begin by deriving (6). The solution for the inner-level optimization problem of (5) as follows: [M(x,)) r(z, [() (TKI (
max F(t;r) E D D.wora((()) o (1m
BEwD, | log E Byrmnaal))) [exp(r(<x, [xxp(rr v)/ Ex-Dew [r(2,
When the reward r is reparameterized by Fo(a,,) = 3(log ( x)) - log Tee x)), we have that the first. term in (9) is O. The bilevel objective (5) then becomes max (r: D.) (r((s)))))
[r(x,//)].
By reparameterizing the reward with the LLM, obtain the desired results in (6).
Then we provide the derivation of (7). We primarily consider the gradient of the newly incorporated term D.gr (x) [log To x)]. Specifically, we have
(n) log No ( =)] ELND Σ Tref(y)
2) log No (y m:)
Euw D.ym
Trefly x)
Vo log To m))
To .r)
-BnDD.yn D.y
Exp(-Fa(l..a log VoggTo) (x)
For the derivation of the DPO gradient VoLDro(To; Di), we refer the readers to Rafailov et al. (2024b).
B Proof of Theorem 5.1
Proof of Theorem 5.1. The solution to the KL-constrriined reward minimization objective (8) is
Tp min (y) z) = (リ exp(-ff(z, 3)))(((2))33)2
where (x(x) = E, (V 2) exp(-f, 8)33) 1. Then we have ami (v x)) = TTor(1/ x)) i.c., the reference policy Fee achieves the lowest implicit reward reparameterized by any p-
18

---

Published in Transactions on Machine Learning Research (03/2025)
C Proof of Theorem 5.3
We use the reduction technique from Xie et al. (2024) to connect the sample complexity of SELM to that of existing RL algorithms (Zhong et al., 2022; Liu et al., 2024b). We restate the proof technique from Xie et al. (2024) for completeness. We emphasize that it is not. a novel contribution of the present work. It worth noting that the thcorctical version of the sclf-cxploration mechanism (Algorithm 2) is a bit different from the practical one used the numerical experiments and is closer to the proposed algorithm in Xie et al. (2024).
We present the following theoretical version of the proposed self-exploration algorithm. The key modification in Algorithm lies in its pragmatic strategy for constructing the chosen and rejected responses. Despite this adjustment, the core principles of leveraging the celf-exploration objective during online alignment remain the same.
Algorithm Self-Exploring Language Models (SELM; Theoretical Version)
Input: Reference model Tref preference dataset Do 2, prompt distribution v, online iterations T", optimism coelficient a, To Trof
for iteration 1.2,...,3 do Sample T2 ~ v, y/ ~ TT- w)), yi ~ Tt-1( x)) 3: Update the preference data D; = Di-1 1 U {( vi yř)} 4: Train the LLM = argmax, { -pro (T; D.) PawuByy (10 TT [)]}, let Tra = me.
end for
Definition C.1 (Preference-basce based GEC). For the function class II, we define the preference-based GEC (PGEC) as the smallest dGPEC as
म.' (y x)) Tenr(y) (y Tead [x)
(T x)
x (v/ [x) Teat (3'1x)
TelV' x)) Tean (y xx)
E Ea B (e)
log
log
log
log
drGEC
Σ> 1=1
T 1-1
B
log xt (y) xx)
log
n (y (x) Teet(y) (x)
log
It' (y' x))
+ log
x ( (x)
Tref (y (a))
Tree (3/ (x)
Fref (y)
xr)
4V drGECT.
The definition of PGEC is a preference-based based version of Generalized Eluder Coefficient (GEC) proposed by (Zhong et al., 2022). Intuitively, both PGEC and GEC establish a crucial connection between prediction error and in-sample estimation error, effectively transforming regret minimization into an online estimation problem. For comprehensive explanation and in-depth discussion, readers are directed to Zhong al. (2022). slight difference that the PGEC here defined with respect to the policy class, while the GEC Zhong et al. (2022) is defined with respect to the model or value class. These can be connected if we regard the implicit reward class log(TT/Trel) /ret) as the model or value class. As important example, if consider the log-linear function class 11 ( ( x)) x (x(p((((,))))) y), v), ()) we can show that dPCEC Ö(d) by the elliptical patential lemuna (Abbasi-Yadkon) et al., 2011; Zhong et al., 2022). Another remark is that here the PGEC is defined in the bandit formulation, and it can be naturally extended to the token-wise MDP formulation (Zhong et al., 2024; Rafailov et al., 2024a; Xie et al., 2024) and further connects to the eluder dimension in the context of preference- based MDPs (Chen al., 2022; Wang et al., 2023). Specifically, we regard the generation process of I.LMs as token-level MDPs where the generation of each token serves as one step, the learning objective is maximizing
[Tr) EXOETNNT r(7) :llog T[ [x)
neat (T 2))
19

---

Published in Transactions on Machine Learning Research (03/2025)
Here T is the full trajectory starting from We can similarly define the PGEC (Definition C.1) for token-wise MDPs by replacing the response y. in the bandit formulation with the trajectories T, in the token-wise MDP formulation. We have the following informal theorem:
Theorem C.2 (Regret for MDP Formulation (informal)). With proper parameter choice, holds with probability at least - 0 that
R(T)& I VPGGE exp(2V P. log( /6);
where Vax is a bounded coefficient for token-wise MDPs, similar to the one described in Assumption 5.2.
C.1 Proof of Theorem 5.3
Proof of Theorem 5.3. We first decompose the regret as
R(T)=
- J (m=)]
Te x) Tex (y/ xx)
Bannyrr r(2, v) Blog
Be (slor)
r(x, y) -Blog
Treet (y/ xx)
TT" (y | (2) Basuyr-aaall)) ((x,))) v) Bog
(. 22) Teee x))
E. Ea-o.y-na((l))
r(x, v) B log
Teoo (y x))
where the last line uses the fact that
T" x) Teef x))
r(x,1) 4) - 3 Blog
B log (x)).
(10)
which is independent of the response y. As the optimal solution to the KL-constrained reward maximization objective, the proof of (10) can be found in many previous works (Rafailov et al., 2024b; Peng et al., 2019), e.g. see Appendix A.1 in Rafailov et al. (2024b). Rearranging the above regret decomposition, we have
(y x)) Toef (y) xx)
E r(x,)) log 3e x)) neer (1) x))
R(T)= Ea-ugrrna r(z.v) -Blog
Blog Tely x)) "x xx)
B Pannyy
r(x, log Tely (z, + log
Telv' x))
E
Reat
B Bes Blog (e x)
" x)
log nt (yl
log Te(y x))
TT" (v/ (v'1x) (x)
re(y' x))
18
E ((r.aa/))eessarrt)
log
log
(11)
Teet(y (z)
neef (y) x))
Teet (v/ x))
Teef (y' xx)
20

---

Published in Transactions on Machine Learning Research (03/2025)
where the last line uses (10). By the definition of PGEC in Definition C.1, we have
log n² (y 2))
(y x) Tree (y [x) ret (y/ x))
log xt' (y/ [x) Feet (y' (x)
log Tely' x) neot (1/ x))
log
T
c-1
log n" (yl xx) Tye (x)
n" (y (r)
T"(y' (v/ x)) Txerly xx)
n "s x))
dPCEC
log
log
+ log
Teat (3/ xx)
Tree (y' xx)
1m1
+ 444V/rae drGECT
dPGEC
x x"(ylar 2x) Teer(y (y xx)
n <'
x <'(v/ (x)
x
<'('' 22)
72
+1
log
log
log
+ log
4n
Tyet [x)
(ro [x)
Trol(y" x))
t=1
+4V dPGReT.
(12)
where the last inequality follows from the fact that VTy S xz(44n) m for any 2. > 0.
By the updating rule of Te1 argmax (-Croo D.) Egwr Byro (v x)]}; we have
LDPO ( Dc-1) (.B-)))---BBneegoooonor
E (1) olo ग. (y x)]
2 : - a R log क. " x)),
which equivalents to that
Te(y x) Blog x² x x))
S 8
E Ex-wsr-saa((1))))
(ADro(T"; - LDro(T; Dz-1))
(13)
We upper bound the right-hand hand side of (13) via the following lemma.
Lemma C.3 (Concentration). For any E [T] and 0 < < 1, it holds with probability that
(DpO(x"; Di-1) (DPO(M;; Di-1)
2
log T TT' [z) Tref(y) (x)
log T xx)
T (2(((x) x (v' w)) Weef
log
+ log
exp(4
(F x)) Weef (3/ ())
+ (o)(()))/8).
Proof. The proof of this lemma follows the standard MLE analysis (Zhang, 2006) and its application for standard swaadd-based RL (Agarwal et al., 2020; Liu et al., 2024b). Recent works (Liu et al., 2024c; Xie et al., 2024; Cen et al., 2024) also applies this result for RLHF. For brevity, we omit the detailed proof here and direct readers to these related works for the proof.
Combining (11), (12), (13), and Lemma C.3, together with the parameter choice (x(()xx(44mmmm)) we obtain
R(T) TTdCEE + Bn- exp(AAmax) log 4y deCIEC
S VdraEc exp(2Rmax) T log (10
where the last line follows from the fact that the proof of Theorem 5.3.
log [[])). Therefore, we finish
J
Experiment Setup
In experiments, we use the Alignment Handbook (Tunstall et al., 2023a) framework as our codebase. We
find the best hyperparameter settings for the strong iterative DPO baseline by conducting grid search
21

---

Published in Transactions on Machine Learning Research (03/2025)
over the iteration number, batch size, learning rate, and label update rule. The results for the Zephyr-based models are shown in Figure Specifically, we find that using the same amount of data, updating the model too many iterations can lead to instability. So we set the iteration number to 3 for Llamaš-It- Zephyr-based models (excluding the first iteration of DPO training). Besides, we observe that choosing different batch sizes has large effect the models' performance and the optimal batch size heavily depends on the model architecture. In experiments, we set the batch size 256 and 128 for the Zephyr-based and Llama3-It It- based models, respectively. For the learning rate, we consider three design choices: cyclic learning rate with constant cycle amplitude, linearly decayed cycle amplitude, and decayed cycle amplitude at the last iteration. We find that decaying cycle amplitude performs better than constant amplitudes in general. Thus, for Zephyr-based models, set the learning to 5e for the first three iterations and le 7 for the last iteration. In each iteration, the warmmu ratio 0.1. For Llama3- It-na--tt-baedd models, we use a linearly decayed learning rate from 5e - to le - within 3 iterations with the same warmup ratio. We also test two update ways for the preference data. One is to rank Ywu Mr Veef and keep the best and worst responses in the updated dataset, which is the setting that described in the main paper. The other is to compare Vhe and Shef and replace the chosen or rejected response by Uref based on the comparison result. We find that the former design performs better than the latter. We also compared with stepwise DPO (Kim et al., 2024), which updates the reference model at each iteration but uses the original dataset instead of the updated one. This demonstrates that exploring and collecting new data necessary.
and
Ablation Iteration Number
Ablation on Batch Size
Ablation Learning Rate
7.7
7.7
7.7
7.8
7.6
7.6
MT-Sench score
7.5
BeBeechh score
7.5
Bench score
7.5
7.4
iters
7.3
iters
7.3
7.3
Decayed LR at last iter
7.2
iters
7.
Batch size 256
7.2
Linearly Decayed LR
iters
Batch size 128
Constant LR
7.1
Offiine DPO
Offiine DPO
Offline DPO
7.0
20
30
40
50
60
7.0
10
20
30
40
50
60
7.0
10
20
30
40
50
60
Training data
Training data (k)
Training data (k)
Ablation Dataset Update
Ablation Dataset Update
Rank <y., y.w
Iterative DPO
7.6
Rank Yrer)
7.6
Stepwise DPO
ST-Bench score
7.5
Offline DPO
7.5
Offiine DPO
MT-Bench
7.3
7.3
7.2
7.2
7.0
0
10
20
30
40
50
60
7.0
10
20
30
40
50
Training data
Training data (k)
Figure 6: Ablation of the iterative DPO bascline. We conduct grid scarch over the iteration number, batch size, learning rate, and designs of the dataset update rule.
For the proposed SELM method, we follow the above hyperparameter settings for a fair comparison. The optimism coefficient a is scarched over 0.005, 0.001, 0.0005, and 0.0001 and selected based on the average external reward on the holdout test set of Feedabeeaaac.. We set a = 0.001 for Zephyr-based SELM and a = 0.0001 for Slama3-It-basee SELM. For training SELM based on other models, we recommend setting 0.005 or 0.001 it shows minimal sensitivity to variations.
22

---

Published in Transactions on Machine Learning Research (03/2025)
E Additional Experiments
E.1 Ablation on Prompts Across Iterations
We conducted additional experiments on the data selection during online alignment. Specifically, for both the iterative DPO baseline and SELM, update the policy the same prompt dataset for multiple iterations.
The results are shown in Table
MT-Bench
Zephyr-7B-SHP
5.30
Iterative DPO (same)
7.21
SELM (same)
7.52
Iterative DPO (diff)
7.46
SELM (diff)
7.61
Table 3: MT-Bench average score when using the same set and different sets of prompts across iterations.
We find that the performance both iterative DPO and SELM lags behind their counterparts that use varying prompts in different iterations. However, SELM is still able to outperform iterative DPO by a considerable margin.
E.2 Ablation Preference Pair Construction
Besides, we also implemented SELM following Algorithm sample six responses from the reference policy and use them both for calculating the optimism term in the objective and being ranked as preference pairs. The average scores on the MT-Bench are shown in Table 4.
MT-Bench
Zephyr-7B-SFFT
5.30
Iterative DPO
7.46
SELM (Alg. 1)
7.61
SELM (Alg. 2)
7.55
Table 4: MT-Bench average score when using different stratogics of preference pair construction.
It can be observed that both implementations of SELM demonstrate similar performance and consistently outperform iterative DPO. However, Algorithm 1, which we used in our main experiments, is more cost-efficient as it requires fewer samples to be drawn from the reference policy.
E.3 Ablation Optimistic Bias Versus Data Selection
We conducted additional ablations on the performance gains of SELM. Specifically, we study whether the gains come from the optimistically biased objective or the novel data as result of exploration. We considered the following two settings: performing DPO the preference data generated by the first iteration of SELM, and using the SELM objective on the original preference dataset. The results are shown in Table 5.
We observe that the data generated by SELM demonstrates superior performance when applied both the DPO and SELM objectives, primarily due to its ability to encourage exploration. Preference optimization SELM-generated data consistently outperforms optimization on data from the DPO model, suggesting that SELM's performance gains driven more by enhanced exploration than by improvements in the
23

---

Published in Transactions on Machine Learning Research (03/2025)
MT-Bench
DPO Iter 1
7.53
DPO Iter 2
7.55
SELM Iter
7.53
SELM Iter 2
7.61
SELM Iter data + DPO objective
7.64
DPO Iter 1 data + SELM objective
7.51
Table 5: MT-Bench average score when using different objectives on data generated from different models.
optimization objective. In our main experiments, we use the SELM objective directly rather than applying DPO to SELM-generated data, as this approach avoids the need to train multiple models at each iteration.
Additionally, we measure the semantic similarity of the generated responses between SELM and the base model, as well between DPO and the base model, using Sentence-BERT" The results show that SELM achieves similarity score of 0.69, which is lower than the 0.77 similarity score observed for DPO, This indicates that SELM facilitates more effective exploration compared to DPO.
24


Reviews:
[{"review_id": "kx9t1CwCQO", "reviewer": "Reviewer_aVa1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nExploration in online preference alignment is a topic worth studying. This work makes a reasonable contribution to this topic.\n\nThe experimental results look good, which verify the proposed method.\n\nThe paper is clearly written and easy to follow.\n\n**Weaknesses**\n\nThe implementation and experiments seem not the same as the online setting studied.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper study the problem of exploration in online preference alignment of large language models (LLMs). The authors propose Self-Exploring Language Models (SELM), which minimizes the original DPO loss while reducing the probability of generated answers using the current policy (Eq. (4.2)) to achieve exploration.\n\nThe authors show that (Theorem 5.3) SELM can be guaranteed to achieve sublinear cumulative regret under Assumption 5.2.\n\nThe authors conducted experiments on (LLM-as-a-judge tasks such as AlpacaEval 2.0, and MT-Bench and several other standard tasks (Table 2), using Llama-3-8B-Instruct and Zephyr-7B-SFT as the base models. The results show that SELM gains further improvements over DPO.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Algorithm 1 contains \"for iteration t = 1, 2, ..., T\" which seems to indicate there are many iterations. The introduction also gives audience this impression, since exploration is a problem where one context is encountered for many times and the policy has multiple chance to explore and then eventually figure out a good answer.\n\nHowever, as also noted in the paper, the prompt dataset was split into three equal-size parts, and then SELM only went through the dataset once, and T = 3 (there are only 3 iterations in Algorithm 1). This doesn't seem to be an \"online alignment\", since in this case each context will only be encountered once, and after that the policy will never have another chance to explore another answer?\n\nMaybe a better way to align the experiments and theory study is to make the implementation online, such that the policy can go over and over the prompt dataset for multiple times, which will be the true version of online alignment.", "broader_impact_concerns": "I do not have concerns on the ethical implications of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qqCjI2mRO6", "reviewer": "Reviewer_3cNb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**\n- The motivation of the proposed objective is very clear and the authors explained the motivation with the illustrated example intuitively. \n- The whole derivation is quite clear and easy to follow.\n- The experiment conducted in the paper is clear to follow and understand. \n\n**Weakness**\n- The objective in 4.1 is different from that of 1.1, especially change from the original reward objective to the relative objective, which I am not sure if the intuition still exists. \n- For the additional term, in Algorithm 1, the authors directly sample another response $y$ from the reference policy. However, as we know both the negative or the positive pairs could be from the reference policy, so it is unclear what if we replace the new response with existing responses.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors propose a new bilevel optimization objective for alignment methods of LLMs. This is achieved by adding an optimism term to the reward-fitting loss. By doing similar derivation with DPO, the authors derived a RM-free objective which consistent of the orignial DPO objective together with an optimism term which is the negative log likelihood of the current policy on the pairs generated by the previous reference policy. This new objective does not bring much additional computational costs compared with naive DPO while it can allow the model to enjoy the benefits of active exploration. Empirical experiments show that the new derived objective can effectively improve the alignment performance compared with naive DPO. To summarize, the authors make the following contributions:\n\n- The authors propose a new bilevel optimization objective which allows the LLM to perform active exploration during the alignment phase.\n- The authors derive a simple loss for the proposed bilevel optimization problem which only requires to add an additional term that is easy to calculate compared with DPO.\n- The authors conducted empirical experiments to show that the proposed methods can effectively achieve better performance compared with DPO.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Please add explanation of the new objective in 4.1, and explain the different of 4.1 and 1.1 and why 4.1 can still allow LLMs to perform active exploration.\n- Please conduct experiments and analysis when the new response $y$ is from either the negative or the positive response of the reference policy. \n- Please explain how the active exploration here is different from previous standard exploration techniques such as UCB in context bandit in the online settings.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "DK23QIxRxZ", "reviewer": "Reviewer_PkQt", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n\n- It is well-written and the motivation for improving exploration in online alignment is clearly presented.\n- They provide a justification of why the exploration term helps based on the gradient analysis and based on a comparison with DPO.\n- The experimental results are extensive. The authors use their algorithm to fine-tune two state-of-the-art base models and compare the performance on two instruction-following and other academic benchmarks.\n- The authors include an ablation study on the optimism coefficient $\\alpha$ to provide further insights into its impact on performance. Additionally, they conduct an ablation on the implicit reward $\\hat{r}_{\\theta}$ of SELM and DPO to provide insights into the learning dynamics.\n\n### Weaknesses / Discussion Points:\n\n- It seems that the main baseline for comparison is the custom iterative DPO method. I would have expected a more comprehensive comparison with other state-of-the-art online alignment baselines. \n- Following the previous point, it would be helpful to include additional details explaining the baselines compared with their method, such as SPIN, DNO, and SPPO.\n- To clarify, doesn’t the additional second term in the gradient in Equation $4.3$ become zero for high $\\hat{r}_\\theta$ due to exp$(-\\hat{r}/\\beta)$? Or is this controlled by the value of $\\beta$? Have you observed the training to become more sensitive to the choice of $\\beta$ compared to DPO?\n- Could you explain the reasoning behind the selection of $3$ iterations for the online alignment in practice?\n\nMinor comments:\n- I recommend using global numbering for the equations.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors study the setting of iterative online alignment for Large Language Models. \n- They derive an objective to update the LLM policy that is reward-free (similar to DPO) and incorporates an additional optimism term to support the exploration in out-of-distribution regions with high implicit reward.\n- With some modifications to the proposed bilevel optimization objective, they design a practical algorithm called SELM, which can be used for online preference alignment of modern LLMs.\n- They provide an analysis of the gradient of their proposed objective and show how the exploration term in SELM addresses DPO’s known issue of decreasing the likelihood of responses generated by the reference policy.\n- In the experimental section, they demonstrate that SELM outperforms other online alignment techniques, including iterative DPO.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would like to ask the authors to address and clarify the weaknesses and open questions/points.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VPzqM6yagN", "reviewer": "Reviewer_z5vv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n1. The theoretical analysis provides formal guarantees about sample efficiency, connecting exploration in preference optimization to established RL theory.\n\n2. The method is practically implementable and integrates well with existing DPO framework, requiring no separate reward model.\n\n3. Comprehensive ablation studies examine how the optimism coefficient α affects reward distributions and model behavior.\n\nWeaknesses\n\n1. While theory emphasizes exploration through optimistic bias, the actual algorithm operates more like a ranking-based data selection strategy.\n\n2. Improvements over iterative DPO are marginal (e.g., +0.07 on MT-Bench for Llama-3-8B), and there's no evidence these gains come from better exploration rather than other factors.\n\n3. Insufficient validation of core claims. No analysis demonstrates that SELM's exploration is more effective than DPO's inherent exploration through policy sampling.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces SELM, an LLM alignment framework that adds an optimistic bias term to DPO's objective. The authors claim this bias enables more effective exploration beyond DPO's passive sampling, leading to better-aligned models. The method is theoretically analyzed and evaluated on instruction-following benchmarks with Zephyr-7B and Llama-3-8B models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Bridge the theory-practice gap by either modifying the method to better match the theoretical analysis or revising the theory to explain the ranking mechanism.\n\n2. Analyze generated responses qualitatively and quantitatively to demonstrate SELM enables more effective exploration than DPO.\n\n3. Conduct ablations isolating gains from optimistic bias versus data selection to validate the core mechanism.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: **Brief Summary:**

The paper proposes an LLM alignment procedure through preference feedback in the online setting when more data can be collected. The online setting brings the question of efficient exploration. Self-Exploring Language Model (SELM) is a Reward Model-free online alignment method equipped with an active exploration mechanism. The paper theoretically studies a variant of SELM and empirically studies its performance.

**Summary of reviewer recommendations:**

We have four recommendations: 1x Accept, 2x Leaning Accept, and 1x Leaning Reject.
The positive reviews mentioned that they were satisfied with the authors rebuttal and the revisions in the paper.
The negative review mentions that "I am not convinced by the response that 1) the motivation of change the objective which is different from before; it looks like the changing is mainly due to derivation 2) the authors did not conduct the experiment as I requested (use either of the old sample instead of sampling new ones.)".

As we did not have a consensus, I read the paper myself. 

I believe the clarification regarding the motivation of the change of objective is satisfying enough. I can also see that the authors added new experiments to address the negative reviewer's concerns (Appendix E.2). Whether this is exactly what the reviewer hoped for or not is not completely clear as the reviewer did not have a discussion with the authors.

In my own reading of the paper, I also agree that the paper has a novel contribution and a good execution overall. But I also found a few points that require further clarifications before the paper can be published. Several of them are minor.


- Algorithm 1 allows any online iterations T. As far as I understand, the experiments use T = 3. This should be noted much earlier in the paper, perhaps even in the last paragraph of Introduction when the experiments are summarized.

- The regret is defined w.r.t. the optimal policy $\pi^*$ (Section 5.2). Is this the optimal policy of regularized objective (Eq. 4) or w.r.t. $\beta = 0$? I have the same question for $\pi^*$ appearing in the proof of Theorem 5.3 in Appendix C.1.

- Theorem 5.3 has the size of policy class $\Pi$. Nowhere it is assumed or mentioned that the policy class is finite. I understand that this finiteness assumption simplified the result, but if it is required for the results to be non-vacuous, it should be explicitly mentioned too.
Later, as an example, a log-linear model is introduced. For that class, the size is infinite. A clarification is needed.

- In Appendix C – Proof of Theorem 5.3, it is mentioned that "We emphasize that it is not a novel contribution of the present work". What is not a novel contribution? The "proof technique" or Theorem 5.3 or the analysis of Algorithm 2?

- The difference with Xie et al. (2024) needs more discussions. For example, it is mentioned that "[T]he theoretical version of the self-exploration mechanism (Algorithm 2) is a bit different from the practical one used in the numerical experiments and is closer to the proposed algorithm in Xie et al. (2024)".
This makes me wonder what the similarities and differences with Xie et al.'s XPO algorithm are.
Please carefully expand on this, delineate similarities and differences of both the algorithms and their theoretical results.


- Why does equality (10) hold? This seems to be Lemma C.3 of Xie et al. (2024) or at least a variant of it. If it is, please cite it. If not, please prove it.

- It is not clear to me how Eq. (10) implies the equality beforehand. Please clarify it.

- What is $Z_r$ in Eq. (10)?

- What is the meaning of an informal theorem (Theorem C.2)? Has this been proved? If not, it is a conjecture, and should be stated as such.

- During the rebuttal phase, some additional experiments were added in Appendix E. As far as I can see, there is no pointer from the main text to the results of this appendix. Please help the reader appreciate your results by briefly describing them much earlier in the paper.

Minor:
- Notations such as x and y are not defined when used in Eq. (1). Likewise is y_u on page 2.
- In Algorithm 2, Line 3, it is not clear which of $y_t^1$ and $y_t^2$ is the accepted and rejected response.

- Typo: In the statement of Theorem C.2, we have "toekn-wise".
- Typo: On page 21, we have "handsise".


Overall, I recommend acceptance of the paper after these clarifications have been made.
Decision: Accept (poster)