Online Preference Alignment for Language Models via Count-based Exploration
Authors: Chenjia Bai
Yang Zhang
Shuang Qiu
Qiaosheng Zhang
Kang Xu
Xuelong Li
Venue: ICLR 2025 Spotlight (2025)
Keywords: Reinforcement Learning from Human Feedback
RLHF
Preference Alignment
Exploration
LLMs
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose count-based online preference optimization for LLM alignment that leverages coin-flip counting to encourage exploration in online RLHF.

Abstract: Reinforcement Learning from Human Feedback (RLHF) has shown great potential in fine-tuning Large Language Models (LLMs) to align with human preferences. Existing methods perform preference alignment from a fixed dataset, which can be limited in data coverage and the resulting reward model is hard to generalize in out-of-distribution responses. Thus, online RLHF is more desirable to empower the LLM to explore outside the support of the initial dataset by iteratively collecting the prompt-response pairs. In this paper, we study the fundamental problem in online RLHF, i.e., how to explore for LLM. We give a theoretical motivation in linear reward assumption to show that an optimistic reward with an upper confidence bound (UCB) term leads to a provably efficient RLHF policy. Then, we reformulate our objective to direct preference optimization with an exploration term, where the UCB-term can be converted to a count-based exploration bonus. We further propose a practical algorithm, named Count-based Online Preference Optimization (COPO), which leverages a simple coin-flip counting module to estimate the pseudo-count of a prompt-response pair in previously collected data. COPO encourages LLMs to balance exploration and preference optimization in an iterative manner, which enlarges the exploration space and the entire data coverage of iterative LLM policies. We conduct online RLHF experiments on Zephyr and Llama-3 models. The results on instruction-following and standard academic benchmarks show that COPO significantly increases performance.

Published as a conference paper at ICLR 2025
ONLINE PREFERENCE ALIGNMENT FOR LANGUAGE MODELS VIA COUNT-BASED EXPLORATION
Chenjia Bai'. Yang Zhang?1, Shwang Qiu Qiaosheng Zhang", Kang Xu², Xuelong Li Institute of Artificial Intelligence (TeleAI), China Telecom, TTiingu University, City University of Hong Kong. *Shanghai Al Laboratory, Tencent Al Lab
"Shenzhen Research Institute of Northwestem Polytechnical University maicj@chinee eaeenaaeeeeeee wec.eeeecommmmmoo cn, xuelong 1 wwe.eeeeeer.....
ABSTRACT
Reinforcement Learning from Human Feedback (RLHF) has shown great poten- tial in fine-tuning Large Language Models (LLMs) align with human prefer- ences. Existing methods perform preference alignment from a fixed dataset, which can be limited in data coverage, and the resulting reward model is hard to general- ize in out-of-distributie power the LLM to explore outside the support of the initial dataset by iteratively collecting the prompt-response pairs. In this paper, we study the fundamental problem in online RLHF, i.e. how to explore for L.MM. We give a theoretical moti- vation in lincar reward assumption to show that an optimistic reward with an upper confidence bound (UCB) term leads to a provably efficient RLHF policy. Then, we reformulate our objective to direct preference optimization with an exploration term, where the UCB-ttrm can be converted to a count- exploration bonus. We further propose practical algorithm, named Count-based Online Preference Optimization (COPO), which leverages simple coin-flip counting module to es- timate the pseudo-count of a prompt-response pair in previously collected data. COPO encourages LLMs to balance exploration and preference optimization in an iterative manner, which enlarges the exploration space and the entire data cov- erage of iterative LLM policies. We conduct online RLHF experiments on Zephyr and Llama-3 models. The results on instruction-olllwwvii and standard academic benchmarks show that COPO significantly increases performance.
responses. Thus, online RLHF is more desirable to em-
INTRODUCTION
Reinforcement Learning from Human Feedback (RLHF) is key tool to align the behaviors of Large Language Models (LLMs) with human values and intentions (Christiano et al. 2017 Bai et al.) 20226 Ouyang et al.] 2022). By fine-tuning the pre-trained LLMs using human-labeled preference data, RLHF achieves enhanced performance and robustness to ensure that they operate consistently with user expectations. Existing RLHF methods focus mainly on preference alignment in an offline dataset by estimating reward functions through the Bradlcy-Terry (BT) model (Bradley & Terry 1952) and performing policy gradients to update the LLM as a policy (Ahmadian et al. (0024) to maximize rewards. Other methods perform Direct Preference Optimization (DPO) (Rafailov al.) 2023 Meng et al.] (2024) that considers a problem of restricted reward maximization to define the preference loss in LLM policy. However, both types of methods perform offline preference align- ment in a fixed dataset, which can be limited in the coverage of the preference data. As discussed in theoretical work (Xiong et al. 2024), learning an optimal policy through offline RLHF requires preference dataset with uniform coverage over the entire prompt-responss space, which is impos- sible to satisfy for existing preference datasets. Thus, the offline learned explicit or implicit reward model cannot accurately estimate the reward of prompt-response pairs that are cut-of-distribution (OOD) concerning the dataset, limiting the capacities of aligned LLMs (Rafailov et al. 2024a).
To address this problem, recent work attempts to extend offline RLHF to an online RLHF process by (i) generating new responses using the current LLM policy, (ii) obtaining preference labels from
"ooresponning Author

---

Published as a conference paper at ICLR 2025
human Al feedback, and (iii) performing preference alignment to update the LLM. The above steps can be repeated for several iterations to enhance the capabilities of LLMs. We highlight that the central problem in an online RLHF process is how to explore the prompt-response space in each iteration. Considering an extreme case where the L.M is a deterministic policy without exploration ability, the preference data collected in new iteration will not increase the coverage of preference data, which means that the LLM policy cannot be improved via multiple iterations. Similar to the exploration problem in the standard online Reinforcement Learning (RL) problem (Kearns & Singh 2002 Houthooft et al. 2016), systematic exploration in online RLHF is also important to efficientlyy explore the large space of token sequences and to collect informative experiences that could benefit policy learning the most. Recently, several works have tried to address this problem by introducing an optimism term reward or value estimation (Dwaracherla et al. 2024 Cen et al., 2024), guide the policy towards potentially high-reward responses (Zhang et al. 2024), or actively explore out- of-distribution regions (Xie et al. 2024). However, they mostly rely on the likelihood derived by the LLM itself to encourage the policy to be different from the policies in previous iterations, which lacks theoretical guarantees to empower the LLM to explore systematically based on the confidence of the learned reward model for prompt-response pairs.
In this paper, we propose a novel algorithm, named Count-based Online Preference Optimization (COPO), for efficient exploration in online RLHE Specifically, COPO extends count-based explo- ration (Strehl & Littman 2008 [ellemare et al., (016) that is provably efficient in online RL to online RLHF for systematic exploration of LLM. We start by constructing an optimistic RLHF problem with an optimistic reward function in a confidence set of the reward. In the linear reward assumption, our result shows that the reward with an upper-confidence bound (UCB) bonus leads to a provably efficient RLHF policy with ((VT)) bound. Then, we covert the UCB term to a general pseudo-couse special case of the linear case and make the UCB term easy to estimate. Finally, we formulate the optimistic objective that is theoretically grounded under DPO reward parameterization in general cases, which results in an optimization objective that combines the DPO objective and count-based exploration, where we adopt a differentiable coin-flip counting network (Lobel et al. 2023) to esti- mate the pseudo-counts of prompt-respons pairs via simple supervised regression.
of prompt-respone pair under the tabular MDP settings, which serve as a
Our contributions can be summarized follows: (i) We propose COPO to encourage the LLMs to balance exploration and preference optimization in an iterative manner, which enlarges the explo- ration space and whole data coverage of the iterative LLM policies; (ii) We construct a lightweight pseudo-counting module with several fully-connected layers based on the LLM,, which is theoret- ically grounded in policy optimization of online RLHF; (iii) we conduct RLHF experiments of COPO and several strong online RLHF baselines on Zephyr and Llama-3 models. The results of mnstruction-folloiin and academic benchmarks show that COPO achieves better performance.
2 PRELIMINARIES
We present the standard RLHF pipeline, summarized from the standard LLM alignment workflow. Specifically, a language model takes a prompt denoted by x E X, and generates a response denoted y E V. Accordingly, space, and consider the language model as a policy æ which maps x to a distribution over y. The standard RLHF process typically comprises 3 stages built a reference LLM Teef (i) collecting preference data with the aid of a human labeler or scoring model, (ii) modeling the reward function from the preference data, and (iii) fine-tuning the LLM initialized from Tref via RL.
take X' as the state space of the contextual bandit and y the action
Barhowiinnuyangggggoo Reward Modeling from Preference Data. Following Ouyang et al. (002)) Rafailov al. (203);
Zhu et al. (2023), we assume that there exists ground-truth reward function r" "xx, y) X x and the preference induced by the reward function satisfies the BT model, as
exp(r"(x, w1))
P(y X y2(x) exp(r*(a, vn) + exp(r" (x, = a (r" (x. vn) - (x, U2)),
where /1 > Y/ means y1 is preferred over 1/2 and (2) := 101=1111xppppp/ ex(1+) is the sigmoid function. Hence, a preference pair can be denoted by a tuple (x, yw yt). where you and y denotes the preferred and dispreferred response amongst (y1, y2) respectively.
2

---

Published as a conference paper at ICLR 2025
Given preference dataset D {(z, yius 31)} sampled from P. estimate the reward function r via maximum likelihood estimation,
TMLE = arg max r
(r(z)(((zvv)))) r(x, v())
(1)
RL Fine-tuning. In this stage, fine-tune the language model with the feedback provided by the reward function r. In particular, the goal of the language model и is to maximize the reward while remaining close to the initial reference language model Tref. thereby formulating the KL-regularized optimization problem which maximizes the following objective,
J( r) (x [r(z, v)] 3Er~pl (
(2)
where DKL ( the KL penalty coefficient, and is the distribution prompt sampled from. As result, the RL fine-tuned LLM ,. w.r.t. a given reward function is computed via,
the Kullback-Leible (KL) divergence from т Tyef 3 is
arg max J (T, r) arg max (x- r(z, v) - log [b(2 (y)r
Due to the discrete nature language generation, this objective is not differentiable and is typically optimized with RL algorithms. The classic approaches (Ziegler et al. 2019: Ouyang al. 2022 Bai
et al. 2022b) construct the reward f(z, y) = rr(, v) (log π x
log Tref (y) and maximize
using Proximal Policy Optimization (PPO) (Schulman et 2017).
Direct Preference Optimization. closed-form solution
Alternatively, the KI-rrgularized objective in Eq. 2 admits a
(y(x)) Tree(y| (xp(r(z,y//3)
(rr 2)
where [(r, x) Ey Teet (v) (y) exp(r(rx v)/3) is the partition function. Eq. in turn reparame- terizes the reward function r as,
r(z,v)= B log T. (v|x) - log ret (vla:) + log x))
Motivated by this reparameterization, DPO (Rafailov al. (023) substitutes Eq. 4 into the reward
MLE (Eq. (I), and integrates reward modeling and RL fine-tuning into single policy optimization
objective. DPO bypasses the need for explicitly learning the reward and optimization objective is
To (V
To (z)
arg min CDpo D) arg min
Σ
logo Blog
B log
(5)
Tiee (To
nef (31 (3((z))
(2,80 MI)
COUNT-BASED ONLINE PREFERENCE OPTIMIZATION
3.1 THEORETICAL MOTIVATION
We use calligraphic letters for sets, e.g., S and A. Given a set S, we write |S] to represent the cardinality of S. For vectors and b, we use (a, b) = b denote their inner product. We write ||a|| = Va T Ea as a semi-norm of a when E is some positive semi-definite matrix. We focus on linear reward settings for our theoretical motivation, and we will provide practical algorithm in the next section. Formally, we make the following assumption on the parameterization of the reward.
Assumption 1 (Linear Reward). The reward lies in the family of linear functions re(a, (a(xy) y) 00, v)) = 0e o(x u) for some known and fixed 6(x, y) X x y + Id with max.,y |Ø(2,, v)llz 1. Let 8* be the true parameter for the ground-truth reward function. To ensure the identifiability of 0" we let t* E 1g, where
θB= = E (1,00, = 0, 111122 < B).
(6)
For LLM, 6 can be considered as the last hidden state of the sequence. In the online RLHF process with several iterations, given preference dataset D. = ((x(9) yo the reward model is estimated via maximum likelihood estimation,
(0) ] at iteration t,
ÓMLI E arg min CD. (0), where LD. (0) 8
Σ log (10. (0.)(z()) (( o( (0 (1)) (7)
3

---

Published as a conference paper at ICLR 2025
When the solution is not unique, take all of the that achieves the minimum. For clarity, define the expected value function Ja (Tr) with MLE estimated reward MMLL and (T) with ground truth reward 0* respectively, as
6 v) v) MMLE]] BBEmp [DiI. (T) Bm- (1
(8)
(z) (T) - (a,, y) T [B [D (x)
(9)
We start with introducing lemma on bounding the estimation error conditioned on the data Di. Lemma 1. (huu al. 2023) For any À 0, letting 7 1/(2 e BB) with probability at
least 1 - 8, we have
+ log( 72n
10MLE 0 0'Ils,+NN/S SCC
+ AB2,
(10)
where DD. = 2x (o(x!1)
(1) o(x!0) (1) ()(((())
31
, (
-(x(0) ,zx(0,,) 1))
refer readers to Zhu et al. 2023) for detailed proof. Considering confidence set of parameters
d d+ log()) + xB33
O(fMLE 2) {o 0B| [[ AlI.p, SXC
(11)
7
Lemma 1 shows that with probability at least 8, one has 0* E O(θMLE, A). We thus construct the optimistic expected value function (T( which takes the upper confidence bound (UCB) as the reward estimate,
(m; D.) (E@(Bmrr:A)
Exas (+(()))) Exxp.errx 10 (0(2,,)))
max
El|Eu>plo(e E Exppl (al, (x)))) OMLE- BE DKL (I (x(l=)) ([)())) +
original KL-regularizs RL tuning objective (x)
optimistic exploration term (UCB-term)
(12)
where E CV
d+1og(1/6) 22n
AB2. The derivation of Eq. (1) deferred to Appendix A.1 The
first term in Eq.
12
corresponds to the classic two-stage RLHF methods: (i) learning the reward
model ÔMLE via MLE under the assumption of BT preference model, and (ii) learning a policy to maximize the estimated reward with KL regularization given Ô The second term, which distin- guishes (T( from Js (TT, is equivalent to measurement of how well the current dataset covers the distribution of responses generated by the target policy т. Now we analyze the suboptimality gap of the optimal policy derived from optimizing the optimistic expected value function Jq(T; (T: De). For the output policy म = arg max, [T( D:), we have the following theoretical guarantee.
For any A > 0, 8 > 0, with probability at least - 8, the optimal policy & w.t the optimistic expected value function (M;; D.) satisfies
Theorem 2.
log (1/8) n
SubOpt(At) s 2C
+ AB2 |E #,(x))]ll
(((=))ll((n -
(13)
The proof is deferred to Appendix A.2 By Theorem we can bound the suboptimality gap of the output policy At arg max, [T; D:) for given iteration t. Further, well the policy, resulted from optimizing Ja D.) for T iterations in an online RLHF manner, asymptotically converges to the true optimal policy x². With the total regret after T iterations are ready to state our main theorem which gives a ((VT) rregett bound in the linear reward setting.
analyze how
defined Regret(T) Emn()3(m*) (+t)),
Theorem 3. Assume that for each iteration 1 < t $ T, one preference pair sample is collected and Under Assumption 7 we set X 4 and denote
added into the dataset in the last iteration
4

---

Published as a conference paper at ICLR 2025
::= lo+lloll++11))) (dT))((())) with probability at least 1 - 8, the total regret Regret(T) after T iterations
satisfies
d + log(1/6) + AB2. ddc,
Regret(T) VT C1
(14)
72
where C1 is an absolute constant.
Theorem 3 shows that adopting an online learning paradigm with the optimistic learning objective
(T(;; Dz) as the optimization objective achieves at most ((VT) regret, providing a theoretical guarantee for our algorithm implemented based on Eq. (12), where Ô hides logarithmic dependence on and 1/6. This analysis can be readily extended to the case where mini-batch samples of size are collected in every iteration, producing an improved regret bound scaled by 1/Vk. The proof is deferred to the Appendix A.3
3.2 CОРO ALGORITHM
In the following, we elaborate on how our COPO actually implements the optimization objective (T( D.) in Eq. 12 for LLM alignment. The first term in Eq. 12 involves the same pipeline as the classic RLHF methods: (i) modeling the reward ÍMLE from the preference data De, and (ii) fine- tuning the LLM with the estimated reward MILE via RL, which can be integrated into single direct DPO objective according to Rafailov et al. (2023). Thus, we replace the first term with the objective (T
S Dt). where the LLM be optimized parameterized by ş. Note that the replacement with DPO objective implies that we implicitly reparameterize the reward function as r(z, ()) = 3(log (y) x)) log (y z)). It loses the need design and use the feature mapping o(I, y) in the practical implementation while our discussion remains valid in the linear reward setting.
Then, we have the following lemma adapted from Bai al. (2022a) build the re- lationship between the count-based exploration and the second UCB term proportional to ||Em=p|d( [ (x))]]](( +X1)) +
2022a a tabular case where the states and actions are finite, Lee, |X| < oo Lemma 2. (Bai et al.
and [)|| ler T and Φ(2, v) ((r))) be the one-hot canonical basis in Rd. The UCB
term [B (0x,, x (a:) (a)))]((( (So, +AIY-1 satisfies
|B T (x)))]
Buoper ( (VN
(15)
where ND. (x. y) is the visit counts of state-action (x. y) in the dataset D
We refer to Bai et al. 2202a 2024) for detailed proof. Lemma 2 shows that under the tabular set-
ting, the UCB term takes a simple form as the count-based bonus in the classic with exploration
(Strehl & Littman 2008 Bellemare et al., 2016). Combining Lemma and Eq. (12) altogether, we
finally derive the optimization objective of COPO, described as
max Joopo ( D:) = - p Dr, 9) + a EE (ulz) /// (V + (16)
optimistic term of COPO
where a and À are hyperparameters, and ND, (æ, y; i a counting function with trainable parameter & discussed in the next section. To demonstrate how our COPO objective implements optimism and elicits active exploration, we analyze its gradient to provide more intuitive explanation.
What does the COPO update do? Denoting the reward function Pil(,, u) B(log Th (vlæ:) log Teer (y)r(y)))), the gradient of COPO objective in Eq. (16) with respect to is derived as follows,
V Vepeopo T Dz) = (w. the W.)D (f v1) - i (r. yho (V log T (Bwl>) - V. log T.e (ur(x))) exp(f (a, v) 3)
+ a Eer wo
/ND (z. v: 0) V log (vkc))
(17)
where (z, 3) = lo (2)2)) 8 log net (z) parameterized by DPO reward. We note that the first term remains the same the original gradient of the DPO loss function, i.e., - (=:
5

---

Published as a conference paper at ICLR 2025
Algorithm 1 Count-based Online Preference Optimization (COPO)
Require: Reference model Tef preference dataset D, online iterations T, optimism coefficient a.
1: for iteration / = 1, ..,..77 T, do
2: Set DD as the 1-th portion of D and generate y ~ Terl( (x) for each prompt x: in D.-.
3: Rank (y. Yw, 3/ with score model and obtain Dj that contains the best and worst responses.
4: Update the parameter via min (fo; Detn) for the coin-flipping network via Eq. (18)
5: Update the LLM policy via maxn, Jeopo (Tip (
Dr) defined in Eq. (66 and set Th + TTf
6: end for
while the second term, corresponding to the gradient of the optimistic term of COPO, controls the optimization direction of LLM according to both the rewards and the visitation counts. Specifi- cally, it tends to increase the log-likelihood of response y generated by Ty toward potentially more rewarding areas when its visit counts No, (ar, y) in the past is relatively low, rather than those sponses with high visit counts. Consequently, the count-based bonus encourages active exploration toward not only high-reward but also more uncertain regions with respect to the regions the LLM has already confirmed, i.e., optimism in the face of uncertainty (Agarwal et al. 2019 Lattimore & Szepesváni 2020 Qiu et al., 2022] Yang et al. (022). In each update, we maximize the objective in Eq. (16) to make the tunettned LLM achieve a trade-off that balances the reward-maxing maximizing and trade-off (Mannor
highly uncertain-response pursuing, i.e., the well-known axplorationeexplotaatiii & Tiissikiis 2004). In each iteration of COPO, the LLM policy will collect novel prompt-response pairs on these uncertain regions, and we use an off-the-shelf reward model to construct Dr. Ideally, with the infinite number of iterations, the datasets U,D; collected by the LLM policies can cover the entire prompt-respons space with count-based exploration, and the DPO objective aims to find the best policy in such a space with wide data coverage. We give algorithmic description in Alg. [
3.3 PSEUDO-COUNT VIA COIN FLIPPING NETWORK
While such a count-based exploration objective in Eq. (6) has theoretical guarantees, it suffers from the issue that visit counts are not directly useful large space where same states are rarely visited more than once (Bellemare et al. (2016). There is no doubt that it would be significantly amplified in the prompt-response space of LLMs that composed of extremely vast discrete token sequences.
Inspired by the count-based exploration in RL (Bellemare et al. 2016 Ostrovski et al. (017). we can substitute the empirical visit count ND, (2,, v) with a pseudo-cownt ND, (2r, y) through density mod- els. However, requires saaning-positivee properties and powerful neural density models, making them impractical in online RLHF settings. Motivated by the recent progress estimating pseudo- counts without restrictions on the type of function approximator or the procedure used to train it, we instead simply apply a Coin Flipping Network (CFN) (Lobel et al. 2023) that directly predicts the count-based exploration bonus by solving a simple regression problem.
The Mechanism underlying CFN. The key insight in the CFN is that state's visitation count can be derived from the sampling distribution of Rademacher trials (or coin flips) made every time a state is encountered. The CFN fo parameterized by is learned by solving arg ming E (p [C( (s;,,)))))) s]abe]]] where L is the mean-square error loss function and Den is a dataset of state-label pairs for learning the CFN. In our case, the state s is the feature vector of prompt-response pair (z, Considering the fair coin-flip distribution C over outcomes (-1, 1). if we flip the coin m times and average the results into Zm, the second moment of #m is related to the inverse count: M2(2m) = -[ = Σ, Pr(zm = i) * 12 = 1/m. Furthermore, by flipping coins each time, the variance of 22 can be reduced by a factor of 3. which implies a reliable way for estimating the inverse count (Lobel et al. 2023). To this end, generate d-dimensional random vector c; ~ 11 as a label spane for state s.. The learning objective is described as
min Jetn (so; Det.) (e [[(s,, s Johel)) arg min Σ (Daul fo
||c
fo(s:))1? (18)
where fst) is neural network that extracts the feature vectors of prompt-response pairs. In practice, we adopt s; as the last hidden state of the fixed LLM with the prompt-response pair as input, and fo(:) is set to a lightweight network with several fully-connected layers.

---

Published as a conference paper at ICLR 2025
The dataset Dan is constructed by using prompt from Dj and responses generated by the LLM policy in the previous iteration, where each occurrence of a state is paired with a different random vector. In case where there are instances of the same state 8, in Dam the optimal solution fa satisfies (s.) = 1 Σ C according to Lobel et al. 2023), then the reciprocal pseudo-count can be estimated by
2 = トニ
Cit
d
1
IE
E
(19)
m
j=1
Thus, by training fo on the objective shown in Eq. bonus given by [I]fo(>)l3/d>> vI/N(+).
(88
we can simply approximate the count-based
4 RELATED WORKS
RLHF and iterative online RLHE
The RLHF framework used for aligning L.LSs was first in- troduced in Christiano et al. 2017); Ziegler et al. (2019) and further developed in Instruct-GPT Ouyang al. 2022). LLaMA-2 2 (Touvron al. 2023) and etc. These works share a similar pipeline that is typically made up of two separate stages: estimating a reward model based on the BT model (Bradley & Terry 1952) and using PPO (Schulman al. 2017) to optimize the reward signals to-
gether with a KL regularization. Several efforts have been made to simplify the preference alignment
procedure and improve the performance of RLHF (Zhao et al. 2023: Rafailov et al. 2023 Munos
et al. 2023; Azar et al., 2024, Guo et al. 2024, Swamy et al., 2024 Tang et al.] [2024] Ethayarajh
ctt al. 2024b). According to whether preference data collected before training or by using the cur-
rent policy during training, we can roughly divide these methods into two categories: offline RLHF and (iterative) online RLHF. offline RLHF, line of work studies direct preference learning, in- cluding DPO (Rafailov et al.) (023) and its variants (Xu et al. 2024 Lee et al. 20244 Rafailov et al.) 2024b). These algorithms integrate reward modeling and RL-ttuning into a single policy objective and optimize it directly on the offline preference dataset. It is observed that DPO-based algorithms are more stable than PPO (Tunstall et al. 2023 Dubois et al. 2024) and have also been adopted in preference learning for other RL problems (Yuan al. 2024bb Yu
al. 2024).
On the other hand, (iterative) online RLHF means that we can collect extra responses by sampling responses from the LLM itself and querying preference feedback from humans Al. This strategy can help mitigate the OOD issue of the learned reward model (Gao et al. (023) and gradually push beyond the boundary of human capabilities. In online RLHF, online exploration is crucial to in- crease the coverage of preference data that determines policy improvement. There are several works proposing various techniques encourage exploration for online RLHE Dwaracherla et al. (2024) proposed using the posterior of reward models to approximately measure the uncertainty for active exploration. XPO (Xie et al. 2024 leveraged the property of the approximation of the regularized value function under the toven-level MDP formulation with general function approximation. Sim- ilar to ours, SELM /Zhang et al., 2024 and VPO (Cen et al. 2024) considered the optimism from perspective of learning reward model, but achieved it by incorporating the maximum of the KL- regularized value function over the target LLM max, (T) into the reward modeling. Instead, our COPO implements optimism based on the confidence set of the reward MLE, which is provably ef- ficient and equivalent to count-based exploration in special cases. Other online RLHF works YYuan) et al. 2024a Lee al. 2024b Singhal et al. 2024) study how to automatically annotate preference labels for generated response pairs, while we adopt an off-the-shelf reward model to rank responses.
Count-based exploration. In both bandit and online RL, promising strategy for exploration is to incorporate a bonus to encourage the agent to gather informative data (Hao et al. 2023). which can be calculated based on count (Strehl & Littman 2008 Bellemare et al. (0166), prediction error (Pathak et al. (017), or random network distillation (RND, Burda et al. (2088). In theoretical RL, count-basee exploration provably efficient in tabular and linear MDPs (Strehl & Littman 2008 Jin et al.) 2020). which motivates us to focus on count- based exploration in online RLHF. In deep RL with large state space, count-based exploration can be extended to function approximation by using density models to calculate pseudo-counts (Bellemare et al.) 2016 Bai et al. (021bla), a). However, with these density-basee -based pseudo-counts come many restrictions that are challenging to fulfill. Other methods (Tang et al., 2017 Rashid et al. 2020) instead heavily incorporated domain knowledge to climinate the usage of density models. In contrast, CFN (Lobel et al. (022)) takes raw states as input and yields visitation count when optimized for a supervised learning objective.
7

---

Published as a conference paper at ICLR 2025
5 EXPERIMENTS
5.1 EXPERIMENT SETUP
Dataset and Ranker For preference alignment of LLMs, we select UltraFeedback 60K Cui et al. 2023) that contains preference pairs of single-turn conversation the preference dataset D J. Yer, Dw.3n). For t-iteration of online preference alignment, we generate response y for each prompt in Dj with the updated LLM, where D, is the t-th portion of the whole dataset D. Then we adopt a small-sized PairRM (0.4B) model (Jiang et al. 2023) to rank (y, Yw 3t) and update D. to contain the best (chosen) and worst (rejected) responses according the reward model, denoted as D,. Finally, we use D, for preference alignment with DPO and count-based exploration. In the next iteration, we use the updated LLM to generate response and construct D accordingly. We note that the performance of our method can be further improved by using the state-of-the-art reward models in RewardBench (Lambert et al.) 2024), , while we adopt a small-scale PairRM model for proof-of-concept verification and fair comparison with baselines.
Implementation of CFN We calculate the pseudo-count of the prompt-response pair via CFN. We implement CFN small fully- fully-connected network that contains two hidden layers with 32 and 20 units, respectively. CFN takes the last hidden state of the prompt-response pair extracted by a backbone LLM as the state, representing <(x,, y) in our theoretical analysis. Then FCN uses the state vector to calculate the pseudo-counts via So (0(2, y))- In the training of CFN, the parameters of backbone LLM are kept fixed, thus we only require small amount of computation to update the parameter of CFN, which counts the prompt-response pairs. The CFN network is trained with Detn each iteration and used to encourage exploration in LLM update with DPO objectives.
Baselines. We adopt an online version of DPO (Rafailov et al. 2023) as a baseline, where DPO trained on DD that contains responses of the updated L.MM policy. We also adopt SELM (Zhang et al., 2024) as the state-of-the online RLHF algorithm, which performs exploration towards potentially high-reward responses without considering the confidence of LLM in these responses. We adopt the same hyper-paramedd settings of online DPO and SELM in Zhang et al. (0024), where the algorithms are trained under the best hyper- syper-parameter setting via a grid search. Both SELM and online DPO are finetuned based on the SET model. For a comprehensive evaluation, adopt Zephyr (Tunstall et al. 2023) and Llama-3 (Meta 2024) for RLHF alignment. Specifically, choose Zephyr-7B-SFFT with a single iteration of standard DPO training the first portion of the training data, which is the same as SELM. And we directly perform preference alignment for Llama- SB-Instruct that has been tuned through SFT. For both Zephyr-7B-SBP and Llama-3-8P --8-tIttttttt,, we conduct 3 iterations of DPO/SELM/COPO alignment of training for comparison.
5.2 EXPERIMENT RESULTS
We evaluate our method on instruction-folloiinng benchmark AlpacaEval 2.0 (Dubois et al. 2024) and MT-Bench (Zheng et al. 2023). AlpacaFival addresses the consistency of results by using standardized process for comparing model outputs to reference responses. The evaluation set, Al- pacaFarm, while diverse, is designed to test models across broad range of simple instructions, ensuring a consistent benchmark for model performance. According to the results of AlpacaEval 2.0 in Table I we find COPO increases the LC win rate of AlpacaEval 2.0 from 22.19 to 27.21 for Zephyr-7B, and increases the LC win rate from 37.17 to 35.54 for Llama3-SB-It, which is a significant improvement in instruction-following tasks. The result signifies that the count-based objective enhances the exploration ability of the LLM, which results in better coverage of the un- derlying prompt-response space. Thus, the LLM policy obtains datasets with better coverage on the optimal prompt-response pairs and benefits the policy optimization of LLMs. As we discussed in the theoretical part, the UCB-based exploration reduces the suboptimality gap in preference op- timization, and the empirical result verifies the theoretical results. Regarding COPO results with we find that the proposed iterative algorithm armed with a count-based explo- ration term can even outperform much larger LLMs, such as Yi-34B-Chat (Young al. 2024) and Llama-3-708 7-0------7ru-tue (Dubey et al. 2024) in LC win rate. The evaluation results in MT-Bench show similar performance improvement compared to online DPO and outperform SELM, where COPO adopts pseudo-count for weighting in policy update compared to SELM. The results in MT-Bench outperform Yi-34B-Chat while still inferior - Instruct..
Mama-3-8B-IInstrus
8

---

Published as a conference paper at ICLR 2025
AlpacaEval 2.0
MT-Bench 1st Turn
Model
LC Win Rate
Win Rate
Avg. len
Avgerage
2nd Turn
Zephyr-7B-SFT
8.01
4.63
5.30
5.63
4.97
Zephyt-7B-DPO
15.41
14.44
1752
7.31
7.55
7.07
DPO Iter (Zephyr)
20.53
16.69
1598
7.53
7.81
7.25
DPO Iter 2 (Zephyr)
22.12
19.82
1717
7.55
7.85
7.24
DPO Iter 3 (Zephyr)
22.19 (714.18)
19.88 (115.25)
1717
7.46 (12.16)
7.85
7.06
SELM Iter (Zephyr)
20.52
17.23
1624
7.53
7.74
7.31
SELM Iter 2 (Zephyr)
21.84
18.78
1665
7,61
7.85
7.38
SELM Iter 3 (Zephyr)
24.25 (116.24)
21.05 (116.42)
1694
7.61 (12.31)
7.74
7.49
COPO Iter (7ephyr)
26.43
21.61
1633
7.68
7.72
7.64
COPO Iter 2 (Zephyr)
27.21 (119.20)
22.61
1655
7.78
7.85
7.71
COPO Ite 3 (Zephyr)
26,91
23.60 (118.97)
1739
7.79 (12.49)
7.89
7.69
Llama-3 3-BB ---ssrructt
22.92
22.57
1899
7,93
8.47
7.38
DPO fter (I.lama3IIt)
30,89
31,60
1979
8.07
8.44
7.70
DPO Trer (I.lmm3-Itt
33.91
32.95
1939
7.99
8.39
7.60
DPO Tter (L.lam3-It)
33.17 (110))5)
32.18 (19.61)
1930
8.18 (10.25)
8.60
7.77
SELM Iter (Llama3-It)
31,09
30.90
1956
8.09
8.57
7.61
SELM Iter 2 (Ilama3-It)
33.53
32.61
1919
8.18
8.69
7.66
SELM Iter 3 (Llama3-Ij
34.67 (111.55)
34.78 (112.21)
1948
8.25 (10.32)
8.53
7.98
CОРO Iter 1 (Llama3-It)
33.68
33.15
1959
8.12
8,38
7.86
COPO Iter 2 (Llama3-Ik
34,30
33.31
1939
8.25
8.49
8.01
COPO Iter 3 (Llama3-I))
35.54 (112.62)
32.94 (110.37)
1930
8.32 (10.39)
8,53
8.11
SPIN
7.23
6.54
1426
6.54
6.94
6.14
Orca-2.5-SFT
10.76
6.99
1174
6,88
7.72
6,02
DNO (Orca-2.5-SFI)
22.59
24.97
2228
7.48
7.62
7.35
Mistral-7B-Int -B-srrtc---nt
19.39
15.75
1565
7.51
7.78
7.25
SPPO (Mistral-it)
28.53
31.02
2163
7.59
7.84
7,34
Yi-34B-Cha
27.19
21.23
2123
7.90
Mlmma-3770B-Instrn
33.17
33.18
1919
9.01
9.21
8.80
GPT-4 Turbo (04/09)
55.02
46.12
1802
9.19
9.38
9.00
Table Results on AlpacaEval 2.0 and MT-Bench. The red arrows indicate the increment from the SFT model (i.e., Kephyr-7B-DPO and Llama-3-8D Llama-3-8D-Instruct)) Compared to online DPO and online SELM baselines, our method achieves superior performance and competitive with larger models.
According the result in AlpacaEval, method generally increases performance after each it- eration in the online RLHF process. However, in Zephyr experiments, we find the average length of response increases significantly in the last iterations, resulting in decreased LC win rate com- pared to previous iterations. This problem can be caused by the proposed exploration term, which can encourage the LLM policy to generate longer sentences that can be novel compared previ- ous responses. Therefore, future direction is to combine our method with existing length control methods reduce such a bias (Meng et al. 2024 Singhal et al. 2023).
We evaluate our approach and the baseline models using established academic benchmarks from the LLM leaderboard (Gao et al. 2024). such GSM8K (Cobbe al. 202T). HellaSwag (Zellers al., 2019), ARC challenge (Clark et al. 2018). TruthfulQA (Lin al. 2021). EQ-Bench (Paech) 2023) and OpenBookQA (OBQA) Mihaylov et al.) 2018). Following the settings SELM (Zhang et al, 2024 employ various Chain of Thought (CoT) configurations, including zero-shot and few- shot scenarios. Table presents the results for these benchmarks. Furthermore, our method exhibits improved performance these academic benchmarks after incorporating preference alignment language tasks. Our technique achieves suitable balance between maximizing rewards and explor- ing the response space without compromising the reasoning accuracy in academic tasks. However, since the preference dataset focuses on how well the model follows instructions or completes tasks intended by humans, such an alignment process might not necessarily align well with the require- ments of some academic benchmarks (e.g., ARC), which often require abstract reasoning, complex inference, or extensive factual knowledge that may not be enhanced by instruction following.
5.3 ABLATION STUDY
We conduct an ablation study Llama model to show the effectiveness of the proposed exploration term in COPO. TableBishows the results of the evaluation on AlpacaEval with different factors a of the exploration terms. According to the results, choosing suitable is important balance explo-
ration and preference alignment. We visualize the count-based term (i.e., Ezyep, 1/ v) X)

---

Published as a conference paper at ICLR 2025
GSM8K (8-s CoT)
HellaSwag (10-s)
ARC
TruthfulQA
OBQA
Models
Average
(25-s)
(0-s)
(0-s)
(10-s)
Zephyr-7B-SPT
43.8
82.2
57.4
43.6
39.1
35.4
50.3
Zephyr-7B-DPO
47.2
84.5
61.9
45.5
65.2
38.0
57.0
DPO Iter (Zephyr)
45.5
85.2
62.1
52.4
68.4
39.0
58.8
DPO Iter (Zephyr)
44.9
85.4
62.0
53.1
69.3
39.4
59.0
DPO (Zephyr)
43.2
60.8
52.5
39.6
58.4
SELM Iter (Zephyr)
46.3
84.8
62,9
52,9
68.8
39.6
59.2
SELM Iter 2 (Zephyr)
46.2
85.4
62.1
53.1
69.3
39.6
59.3
SELM Iter 3 (Zephyr)
43.8
85,4
61.9
52.4
69.9
39.8
58.9
COPO Iter (Zephyr)
46.8
85.0
62.4
53.0
68.7
39.3
59.2
COPO Iter 2 (Zephyr)
46.7
85.3
62.5
53.3
69.1
39.8
59.5
COPO (Zephyr)
47.0
85.4
62.9
53.4
69.9
40.3
59.9
1ama-33-88-Instmmm
76.7
78.6
60.8
51.7
61,8
38.0
61.3
DPO Ier (Llama3-I()
78.5
81.7
63.9
55.5
64.1
42.6
64.4
DPO Iter (Llama3-It)
79.4
81.7
64.4
56.4
64.3
42.6
64.8
DPO 3 (Llama3-It)
80.1
64.1
56.5
64.1
42.6
64.8
SELM
(Llamaa-It)
78.7
81.7
64.5
55.4
64.1
42.4
SELM Iter 2 (Llama3-It)
79.3
81.8
64.7
56.5
64.2
42.6
64.9
SELM Iter 3 (Ilama3-It)
80.1
81.8
64.3
56.5
64.2
42.8
65.0
COPO Iter (Ilama3-It)
79.1
81.7
64.3
56.4
64.3
43.0
64.8
COPO Iter 2 (Llama3-Iu
79.3
81.8
64.6
56.4
64.4
43.2
65.0
COPO Iter (I.lama3-It)
80.2
81.8
64.7
56.5
64.4
43.6
65.2
SPIN
44.7
85.9
65.9
55.6
54.4
39.6
57.7
43.4
85.3
63.4
67.5
65.9
41.2
61.1
SPPO (Mistral-it)
42.4
85.6
65.4
70.7
56.5
40.0
60.1
Table 2: Performance comparison between COPO and the baselines on academic multi-choice QA benchmarks in standard zero-shot, few-shot, and CoT' settings. Here, refers n-shot. The red and blue texts represent the best and the second-best results.
Fig. [ where the steps contain iterations split by dashed line. After each iteration, we collect new responses the prompt set and reupdate the CFN network. The result shows that large encourages the LLM to collect novel responses in the first two itcrations. After preference align- ment, the optimism term in the last iteration decreases the policy has converged to local-optimal solution and the performance will not increase in the last iteration. For small a, the LLM policy gradually collects novel responses in each iteration, while the final performance limited by its exploration ability. A suitable a enables the policy to focus on exploration in early iterations and preference alignment in later iterations, resulting in better final performance.
Large a Medium Small
Factor
Iterl
Iter2
Iter3
0.01
32.89
33.18
33.76
0.10
33.68
34.30
35.54
0.50
33.70
34.61
34.82
Table 3: COPO results Al- pacaEval 2.0 with different explo- ration factor in 3 iterations.
158
Figure Exploration rewards in iterations with different a.
CONCLUSION
This paper presents COPO, novel algorithm for online RLHF of LLMs. COPO integrates count- based exploration with the RLHF framework. achieving a tight regret bound policy through the use of an UCB bonus. COPO can balance exploration and preference optimization via a lightweight pseudo-counting module, and obtains superior performance in AlphaEval 2.0, MT-Bench, and LLM leaderboard evaluations compared to other leading online RLHF algorithms. A future direction is to further enhance the exploration ability of our method by using a changing prompt set, avoiding the prompt-response space.
the restrictions on the initial dataset and resulting in better coverage Meanwhile, an automatic tuned exploration factor according to the status of the LLM policy and the collected data would be better to balance exploration and alignment in different iterations.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
For the theoretical part, we provide the detailed theoretical proof in Appendix A part, we give experiment setup in Section | The hyper-parameters and implementation details given Appendix B The code released at htthtts://
For the practical
com/ wwt.eaaiioieeeeeeeaaeaa cttm/aaicceennaa/OOF
ACKNOWLEDGMENTS
This work supported by National Key Research and Development Program of China (Grant No.2024YFE0210900) and National Natural Science Foundation of China (Grant 62.6240.
REFERENCES
Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems, 24, 2011.
Alekh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun. Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep, 32:96, 2019.
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Ahmer Üstün, and Sara Hooker. Back to basies: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXiv:2402. 14740, 2024.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learn- ing from human preferences. In International Conference on Artificial Intelligence and Statistics, pp. 4447-4455. PMLR, 2024.
Chenjia Bai, Lingxiao Wang. Lei Han, Animesh Garg, Jianye Hao, Peng Liu, and Zhaoran Wang. Dynamic bottleneck for robust self-supervised exploration. Advances in Neural Information Pro- cessing Systems, 34:17007-17000 17020, 2021a.
Chenjia Bai, Lingxiao Wang, Lei Han, Jianye Hao, Animesh Garg, Peng Liu, and Zhaoran Wang. Principled exploration via optimistic bootstrapping and backward induction. In International Conference on Machine Learning, pp. 577-587. PMLR, 2021b.
Chenjia Bai, Lingxiao Wang, Zhuoran Yang. Zhi-Hong Deng, Animesh Garg, Peng Liu, and Zhao- ran Wang. Pessimistic bootstrapping for uncertainty-driver offline reinforcement learning. In International Conference on Learning Representations, 2022a.
Chenjia Bai, Lingxiao Wang, Jianye Hao, Zhuoran Yang, Bin Zhao, Zhen Wang, and Xuelong Li. Pessimistic value iteration for multi-task data sharing offline reinforcement learning. Artificial Intelligence, 326: 104048, 2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint M.Ni:22244008800 2022b.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances neural information pro- cessing systems, 29, 2016.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv: /2894, 2018.
Shicong Cen. Jincheng Mei, Katayoon Goshvadi. Hanjun Dai, Tong Yang, Sherry Yang. Dale Schu- urmans, Yuejie Chi, and Bo Dai. Salue-incentivized preference optimization: A unified approach to online and offline rlhf arXiv preprint arXiv:2405. 19320, 2024.

---

Published as a conference paper at ICLR 2025
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing sys- tems, 30, 2017.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try the ai2 reasoning challenge. arXiv preprint arXiv:1803.00 03457, 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2 14 168, 2021.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01377 2023.
Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arXiv:2405.077 240000786,, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.2 2024.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled al- pacaeval: A simple way to debias automatic evaluators. arXiv preprint 2xXiv2240.0044755 2024.
Vikranth Dwaracherla, Seyed Mohammad Asghari, Botao Hao, and Benjamin Van Roy. Efficient exploration for 1lms. arXiv preprint arXiv:2402.0039% 2024.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. In International Conference on Machine Learning, 2024a.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment prospect theoretic optimization. arXiv preprint arXiv:2402.01306 2024b.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference Machine Learning, pp. 10835-10866. PMLR, 2023.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024.
Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model alignment from online ai feedback. arXiv preprint anXiv:2402.00 04792, 2024.
Jianye Hao, Tianpei Yang, Hongyao Tang, Chenjia Bai, Jinyi Liu, Zhaopeng Meng, Peng Liu, and Zhen Wang. Exploration in deep reinforcement learning: From single- aagll-agenttee to multiagent domain. IEEE Transactions on Neural Networks and Learning Systems, 2023.
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. VIME: variational information maximizing exploration. In Advances in Neural Information Processing Systems, 2016.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference on Learning Representations, 2022. URL https: https://openzeview /pwp///eeeeeeeeeeee//eee net forum?
1d=nZeVKeeFYF9
12

---

Published as a conference paper at ICLR 2025
Hamish Ivison, Yizhong Wang. Jiacheng Liu, Zeqiu Wu, Valentina Pyatkin, Nathan Lambert, Noah A Smith, Yejin Choi, and Hannaneh Hajishirzi. Unpacking dpo and ppo: Disentangling best practices for learning from preference feedback. In Advances neural information process- ing systems, 2024.
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. LIm-blender: Ensembling large language models with pairwise ranking and generative fusion. arXiv preprint arXiv:2306.02550 2023.
Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on learning theory, pp. 2137-2143. PMLR, 2020.
Michael J. Kearns and Satinder P. Singh. Near-optimal reinforcement learning in polynomial time.
Machine Learning, 49(2-3):209-232, 2002.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXi::2403.1378) 2024.
Tor Lattimore and Csaba Szepesvári. Bandit algorithms. Cambridge University Press, 2020.
Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg, Jonathan K Kummerfeld, and Rada Mi- halcea. A mechanistic understanding of alignment algorithms: case study on dpo and toxicity. In International Conference on Machine Learning, 2024a.
Sangkyu Lee, Sungdong Kim, Ashkan Yousefpour, Minjoon Seo, Kang Min Yoo, and Youngjae Yu. Aligning large language models by on-policy self-judgment. arXiv preprint arXiv:240..1125. 2024b.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07955 2021.
Sam Lobel, Akhil Bagaria, and George Konidaris. Flipping coins estimate pseudocounts for exploration in reinforcement learning. In International Conference on Machine Learning, pp. 22594 22613. PMLR, 2023.
Shie Mannor and John N Tsitsiklis. The sample complexity of exploration in the multi-armed bandit problem. Journal of Machine Learning Research, 5(Jun):623-6i) 2004.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-free reward. arXiv preprint 24374 2024.
AI Meta. Introducing meta laaa 3: The most capable openly available Ilm date. Meta AI, 2024.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint wwXiv:1809.02789 2018.
Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Andrea Michi, et al. Nash learning from human feedback. arXiv preprint arXiv:2312.00888 2023.
Georg Ostrovski, Marc G Bellemare, Aäron Oord, and Rémi Munos. Count-based exploration with neural density models. In International conference on machine learning, pp. 2721-2730, PMLL, 2017.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Samuel Paech. Eq-bench: An emotional intelligence benchmark for large language models. arXiv preprint wxXiv:23/2.06281 2023.
13

---

Published as a conference paper at ICLR 2025
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity- exploration by self-supervised prediction. In International conference on machine learning, pp. 2778- 2787. PMLR. 2017.
Shuang Qiu, Lingxiao Wang, Chenjia Bai, Zhuoran Yang, and Zhaoran Wang. Contrastive ucb: Provably efficient contrastive self-supervised learning in online reinforcement learning. In Inter- national Conference on Machine Learning, pp. 18168-18210. PMIR, 2022.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2023.
Rafael Rafailov, Yaswanth Chittepu, Ryan Park, Harshit Sikchi, Joey Hejna, Bradley Knox, Chelsea Finn, and Scott Niekum. Scaling laws for reward model overoptimization direct alignment algorithms. arXiv preprint (rXXix:2406.02900 2024a.
Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From to qt:: Your language model is secretly a q- f-fnuction. arXiv preprint arXiv:2404.12355. 2024b.
Tabish Rashid, Bei Peng, Wendelin Boehmer, and Shimon Whiteson. Optimistic exploration even with a pessimistic initialisation. In International Conference on Learning Representations, 2020.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigating length correlations in rlhf. arXiv preprint arXiv:2310.03716, 2023.
Prasann Singhal, Nathan Lambert. Scott Nickum, Tanya Goyal, and Greg Durrett. D2po: Discriminator-guider dpo with response evaluation models. arXiv preprint arXiv:2405.01511, 2024.
Alexander L Strehl and Michael Littman. An analysis of model- interval estimation for markov decision processes. Journal of Computer and System Sciences,
2008.
Gokul Swamy, Christoph Dann, Rahul Kidambi, Zhiwei Steven Wu, and Alekh Agarwal. A minimaximalist approach to reinforcement learning from human feedback. arXiv preprint arXiv:2401.0444 044400000666... 2024.
Haoran Tang. Rein Houthooft, Davis Foote, Adam Stooke, OpenAl Xi Chen, Yan Duan, John Schul- man, Filip DeTurck, and Pieter Abbeel. exploration: study of count-based exploration for deep reinforcement learning. Advances in neural information processing systems, 30, 2017.
Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Rémi Munos, Mark Row- land, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires, and Bilal Piot. Gen- eralized preference optimization: unified approach to offline alignment. arXiv preprint arXiv:2402.05749 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288. 2023.
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation Im alignment. arXiv preprint arXiv:2310.16944, 2023.
Tengyang Xie, Dylan J Foster, Akshay Krishnamurthy, Corby Rosset, Ahmed Awadallah, and Alexander Rakhlin. Exploratory preference optimization: Harnessing implicit q apapproxiitiionn for sample-efficient rlhf. arXiv preprint arXiv:2405.. 1046, 2024.
Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint, In Forty-first International Conference on Machine Learning, 2024.
14

---

Published as a conference paper at ICLR 2025
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for Ilm alignment? comprehensive study. In International Conference on Machine Learning, 2024.
Rui Yang. Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, and Lei Han. Rorl: Ro- bust offline reinforcement learning via conservative smoothing. Advances in neural information processing systems, 35:23851-238001 233666, 2022.
Alex Young, Bei Chen, Chao Li, Chengen Huang. Ge Zhang, Guanwei Zhang. Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.00 0466222 2024.
Xudong Yu, Chenjia Bai, Haoran He, Changhong Wang, and Xuelong Li. Regularized conditional diffusion model for multi-task preference alignment. arXiv preprint arXiv:2404. 04920, 2024.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho. Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXiv:2401.10020, 2024a.
Xinyi Yuan, Zhiwei Shang, Zifan Wang, Chenkai Wang, Zhao Shan, Zhenchao Qi, Meixin Zhu, Chenjia Bai, and Xuelong Li. Preference aligned diffusion planner for quadrupedal locomotion control. arXiv preprint arXiv:2410.13588 2024b.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- 2019.
chine really finish your sentence? arXiv preprint
Shenao Zhang, Donghan Yu, Hiteshi Sharma, Ziyi Yang, Shuohang Wang, Hany Hassan, and Zhao- ran Wang. Self-exploring language models: Active preference elicitation for online alignment. arXiv preprint a400v:2405. 19332 2024.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305. 10425, 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt- metheench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-46623, 2023.
Banghua Zhu, Michael Jordan, and Jiantao Jiao. Principled reinforcement learning with human feed- back from pairwise or k-wise comparisons. In International Conference on Machine Learning. pp. 43037-43066 PMLR, 2023.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford. Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv: /9tp.09990009990 2019.
15

---

Published as a conference paper at ICLR 2025
A THEORETICAL PROOF
A.1 DERIVATION OF EQUATION
I2
Proof. For any E 0(0MLE, X), by Cauchy-Schwartz inequality, the following holds,
[ (BzzplØ(2, : (x))]) (0 ML.F) - ||0 M.LLlEs, +A (()))
SC d + log(!)
AB2. 1.P.I|Eapplla ( (()))lE
n
Thus, we have
(Bz=ol(a,<< (c)))) < (Bn~olo(z, ( (I)))) T M + &. || n mx))))ll (()))]l((m,++2)) +X1)- (20)
A feasible way maximize Eg- o( (+))] over the confidence @ A) is choose the RHS of Eq. (20 as the maximum. Q.E.D.
A.2 PROOF OF THEOREM 2
Proof. Note that the optimistic expected value function at iteration is
D:)
max (x>pxra( (Ø(x, v))l (<>(,, 3 (e
(E
For ease of presentation, we define
(T) = -Ee-oor (z, v)1 - 3 Ezep
EExp 0 6lz, (x)) 3 D ( (T( ([
and omit the D. in Jo(T;; D.), i.c. setting JT(TT) == Jo(T; D.).
According to Lemmal we know that E O(θMLE, A) with probability at least 1 ò. Consequently, have
(s( (x), for any
(21)
Meanwhile, by à. arg max, JT(TT) (T) we also have
Ja((tt) 2 Jz(
(22)
Combining Eq. 21 and Eq. (22). have
jp((t) 2 {T(TT) 2 [+(
Substituting this into the definition of the suboptimality gap, we achieve
SubOpt(ftt) = Jz(TT) Jz(+t) - ( - J
Here we need to introduce a necessary notation of ò::
0, arg max {(Ez>plo(z, (z)))) 3 Ex DKL (ft( mer (+1=))
(E
With the extra notation, we can further have SubOpt(#t)) S (ft)t)) - (ft+t)
Ex-pl(0, TT)x it (x))
Exmp ÔMIFE) (l (x)r) + (0v. - (i #x(x))
(110, 0 (o + lÍÂNE 0* [1 [| (0(2,, #: (so,
+N
+ log(1/6)
2C
fr 1
+ XB2.
22n
where the second inequality uses Cauchy- Schwarz inequality, and the last inequality is obtained by Lemma
16

---

Published as a conference paper at ICLR 2025
A.3 PROOF OF THEOREM
S
We first present an important lemma and corollary for special case. Then, we combine the lemma and corollary prove Theorem 3
Lemma 3. (Abbasi- Yadkori et al. 2011). Let for be bounded sequence in R satisfying
suPe20 | I 1. Let Ao E paxd be positive definite matrix. For any 0, define A
Ag + + 0,0]- Then, if the smallest eigenvalue of Ao satisfies >min 2 1, we have
(dt((A))
ddt((A))
log
0
2og
(23)
det(AA)
det (Ao)
Proof. The following proof refers mostly to Jin al. 2020 all j 2 0, we have
Since Amin Amin (Ao) and | S 1 for
oJA 1 Amin (Ao)) |1, S 1, Vj 2 0.
For any x E [0, 1]. it holds that log + x) S x - 2log(1 (1 + x). Therefore, we have
log(1 + + <
of A 2 log(1 oJA7-100)
(24)
j=1
Elementary algebra gives
det(A,) = det(A + t10T)) = det(At-)) det(1 + A, 1.00 (o0oTTA,) A,
= det((A, -
Ar det(Ao) II + of 1,))
Hence, we have
log(1 + 6J A6A771110 ,))) = det,))) log (Ao)
(25)
Combining Eq. (24 and Eq. 25 we conclude the proof of Eq. 23
Corollary 1. Under the same setting of Lemma
3 if the smallest eigenvalue of No satisfies
Amin (2a) 2 4, for reference vectors {uz) IR. which are also bounded: ||5 S for any
we have
(et( (ep)
det(Ar)
log
< (1i+ U;) () + +3) s 2 2lo
(26)
det (Ao)
Proof. By triangle inequality, we have
(Ø) + 1;) AA7 + M3)) I (AminA |10; + +2,11² < [Amin (Ao)) 21110g + ||>> s
when AMi) 2 4 and ||||| < 1. Then consider the determinant of the updated matrix Aç;
det(A,) = dct((A,-1 + (0. + uz) +v)
(27)
det(AAr-)) 1 + vz)( 1. T,
Applying the matrix determinant lemma, which states that for any invertible matrix A and vectors and v, we have det(AA + wv ") = det(AA) + v A'AAA A-))). Then we can simplify the expression as follows:
det(A,) = det(Ar (6r + vi) T A (#r, + vz))
(28)
17

---

Published as a conference paper at ICLR 2025
By recursively applying this step, obtain:
dct(Ar) = dct(AA)
+ A + v2)).
(29)
_=1
Taking the logarithm of both sides of the equation, we utilize the property of logarithms that the logarithm of a product is the sum of the logarithms:
log (At) = log (Ao) + Σ log(1 12) +z))
(30)
We rewrite it as:
(de((A))
log
det((A)
=
log (1 + (0i +2² A,] + v2)), Σ
(31)
j=1
and using the property of the logarithm that log +2)
log x) for x E [0, 1], we have:
(Ø, + [ +1) 2 (1 (6; + +1) T + +2))
(32)
Applying this inequality to the sum, obtain:
Σ vo)++ mz) S Σ log (1+ (Ø, + vz) " + v2))
(33)
j=1
This completes the proof by showing that:
(et(AA)
det (Ar)
log
(0s - ( + +2) s 2
(34)
dct(AA)
dee((Ag)
With similar derivation to the one for Eq.
23 we can conclude the proof of Eq. (26).
Corollary i indeed a special variant of Lemma 3 with accounting for a case where Φ, is replaced with Φ5 + Fj. It is important in RLHF as we estimate the reward model with the BT model over the preference data, which assumes that the preference signal is induced by the reward difference between the preferred response and the dispreferred response (i.e., (r(2,, yur) r(x, yt))). Then, under the assumption of linear reward, it further corresponds to the difference in the feature space, i.e., (xx yw) (xx, yt)-
Now we are ready to prove the main theorem. We restate our main theorem follows.
Theorem (Restatement of Theorem 3. Assume that for each iteration 1 < t < T, one preference pair sample collected and added into the dataset in the last iteration 1. Under Assumption 7 if we set X = 4 and denote i := log + (4T)/(dX)), with probability at least 1 8, the total regret Regret(T)) after T iterations satisfies
d + log(1/5)
(T Ci
+ AB2, vil
72
where C1 an absolute constant.
Proof. For simplicity, assume that during T' iterations in the online RLHF, the dataset which initialized as an empty set Do = 0. is collected according to the following protocol for every t 1 I,..,,. T.
Sample one response pair {x, Voe yr} from क.-1, label the data with the external labeler, and form the dataset DD for current iteration i which satisfies D, D U {xr Dor v:
18

---

Published as a conference paper at ICLR 2025
2. Compute the output policy te Re arg max, [T( D.).
By Theorem
2 wc have
Regret(T) (T)
Σ
(3((a)) J3(teel
t=1
d + 1og(1/8))
2C
+ +BB. (|Expp006 (=)))((=)))
722
+ log(1/))
- 2C
+ AB2.
Σ
E (. (E,,
(35)
72
1=1
Since we have |10(,, v)1l < 1 for any (r, and X = 4, by Corollary
we have
dett(Eprt + X7) del(En, + XT)
(B (o (x)))) (ED. + +1) (Eo~olEel,,, (x)))) -
2
where ED. Ei y ((x!), (1 (d(x()) (1) ((x(1)1 11 (T) Note that the trace of Σp, + AI holds
Σ |o(x 10,00))))))))) ((x // S dd + 4t 1=1
tr(DD, + A1) dd
where the last inequality results from triangle inequality. Denoting the eigenvalues of ED. + Al by {Ar) by AM-GM inequality, we thus have
dd + 41
det + AT)
II A S 1=1
then log det ED. + AI) S log A + d
At the same time, we have log det (At) < log det(22n, + Xl), thereby leading to
4T
Σ
+An- log 1
dX
Introducing 1 = log(1 + dA ), by Cauchy-Sehward inequality, we further have
1/2
Bamplo(x (x)))) (E +ND S VT.
Σ E (z (x))l +X
< V 2dc.
(36)
Finally, combining Eq. 35 and Eq. 36), we achieve
d + log
Regret ()) S VT Ci
AB2 dic
where = log(1 + II and Cj is an absolute constant.
TX
B IMPLEMENTATION DETAILS
In optimizing the COPO objective for LLM preference alignment, adopt Low-Rank Adapta- tion (LoRA) (Hu et al. 2022) rather than parammerr training due to limitations in computing resources. In contrast, the results of baseline methods reported in experiments obtained by
19

---

Published as a conference paper at ICLR 2025
Table 4: Key implementations of the text generation experiments.
Basic Parameters
Pre-training
-lama-3-88 Instruct Zephyr-7b-SFT
Hardware
NVIDIA A100 40G
Datatype
bfloat16
Fine-tuning strategy
LoRA (Hu al. 2022]
LoRA target module
q-proj & k-proj & v-proj & o-proj & gate-proy & up-proj & down-proj
LoRA r
128
LoRA alpha
128
LoRA dropout
0.05
Optimizer
Adamw torch
Train epoch
Per device batch-size
2
Accelerator
Deepspeed Zero3
Learning rate
5e-7 (1st iter), 3e-7 (2nd iter), lc-7 (3rd iter) cosine
learning rate scheduler
Learning rate warmup ratio
0.1
Preference dataset
HugginngFaceei44otttaeedddaaa binarried Ilm blandeeeder//
Reward model
CFN (Ours)
Learning rate Exploration factor (a:) Network architecture Activation Train epoch A
1e44
0.1 (Llama) & 0.01 (Zephyr) 4096 (last hidden-dim for LLM) + 32 20
LeakyRel.U & Linear
1
0.01
full-parameter tuning, which verifies the effective of our method in low-computation request. The training of our method conducted on 4x 11000400 GPUs
The implementation of COPO build on Alignment Handbook implementation (https:// which provides the basic DPO al- For the im-
github. com/ htwp://nnnneeaaiiggeeeeeaaaiiiin
gorithm based the TRL Repo (ht // com hugg htwffac/trrrrrrr plementation of iterative DPO, we adopt the same hyper-parameters as in the SELM implementation hhtps: http//gttuu/bbt com. shenac- zhang/ SELM). Since COPO also adopt online DPO the backbone, we use the same parametrrs as maline-DPo except for the CFN network.
COPO mainly contains three stages. (1) Iterative data collection. In this stage, use vLLM as the inference server to perform fast sam- pling of the LLM policy in the previous iteration. In sampling, set the temperature 0.0 and the top-p as 1.0. After generating response for each prompt, we use Ilm-blender with PairRM reward model (https: httph/hugggngc...eeee/ co wwl.mmmeeedee/eeee rRM) rank all three re- sponses (two from the previous dataset and one from sampling). The final dataset contains the best and worst responses as chosen and reject responses, respectively. We also store the response of the LLM policy in the dataset. (2) CFN training, We perform CFN training in the prompt-response pair sampled in the dataset using the previous LLM policy. We construct a lightweight CFN network Val-
https: // gightuu.bbb....
and use it similar to a value head that adheres to an LLM via the ueHead' class. For inference, the CFN takes the last hidden state of the LLM as input and output the prediction of the coin flips. In regression, we adopt the same 'CoinFlipMakee" as in that of on- line RL (https: / qi github.... com/ a am. obe CF).. We use two independent trainers for CFN and online DPO. (3) RLHF training. In this stage, CFN network is keep fixed to provide pseudo- count estimation for the sampled prompt-response pairs, which is used in our exploration objective integrating with online DPO. We summarize the key hyperparameters in Table
20

---

Published as a conference paper at ICLR 2025
C ADDITIONAL EXPERIMENTS
C.I. ADVERSARIAL CASES
COPO focuses improving the exploration ability of the LLM. We conduct adversarial that further restricts the data coverage of the initial preference dataset. In this case, the exploratory capacity of LLM becomes particularly important. Specifically, we use subset of the UltraFeddback dataset with only 20% samples, then we train online DPO, SELM, and COPO for 3 iterations. The performance evaluated on AlpacaEval 2.0 is given in the following table. Based on the results, we find that COPO significantly outperforms other methods where the dataset is quite limited, while other methods have a significant performance drop due to the insufficient exploration ability.
Table 5: Comparison to methods with 20% preference data (LC Win Rate).
Llama-3-8B-It 22.92
DPO (iterl)
DPO (iter2)
DPO (iter3)
26.60
28.01
28. 16
SELM (iterl)
SELM (iter2)
SELM (iter3)
27.81
28.79
29.25
COPO (iterl)
COPO (iter2)
COPO (iter3)
28.32
30.14
31.80
C.2 COMPARISON TO D2PO
D2PO-related methods also lie on the paradigm of online RLHE, but they mainly focus on how to automatically annotate preference labels for newly generated response pairs of LLMs. Specifically, Self-Rewarding LM (Yuan et al.) 2024a) uses the LLM-as-a-Judge ability of LLMs to evaluate re- sponse pairs, Judge-augmented SFT (Lee et al. 2024b) trains pairwise judgment model to output preference label as well as rationale, and Discriminator Guuiddd DPO (Singhal et al. 2024) trains a discriminative evaluation model to generate annotation for synthetic responses. In contrast, COPO directly adopts an off-the-shelf reward model to rank responses generated by LLMs, which is a 0.4B PairRM of small size in our experiment.
We add D2PO (Singhal et al. 2024) as a baseline by removing the PairRM model and training a dis- criminator via the Bearl model. We also change the backbone in D2PO from Llama-2-7B to -Lama-3-8B-Instruce The results on AlpacaEval 2.0 are given as follows. The result shows that the self-trained discriminator achieves competitive performance compared the online DPO base- line, which uses an off-the-shelf reward model, which signifies the effectiveness of D2PO. However, using a small reward model can be more efficient in practice.
Table 6: Comparison D2PO with PairRM.
DPO w/
DPO w/ PairRM Iter2 22.12
DPO w/ PairRM Iter3
D2PO
D2PO
D2PO
Method
PairRM Iterl 20.53
Iterl
Iter2
Iter3
LC Win Rate
22.19
20.10
21.95
22.03
C.3 COUNT-BASED EXPLORATION FOR KTO
Exploration is essential for RLHF since the preference data usually limited in data coverage, which makes DPO develop biased distribution favoring unseen responses, directly impacting the quality of the learned policy. The proposed exploration objective measures the visitation count of the generated prompt-response pair via a coin-flipping network, which can be combined with various online RLHF algorithms.
In this section, we add a new preference optimization objective in addition to DPO for experiments, i.e., KTO (Lthayarajh al. 2024ap. KTO maximizes the utility generations, rather than just the likelihood of preferences. It works effectively with binary feedback, which is more abundant and easier to collect than the preference data that the DPO requires. We adopt the implementation of the KTO loss function from TRL. [ and use online iterations for KTO similar to online DPO. We find
http:://wwggnnngaaae.cco///ct/// trainer
21

---

Published as a conference paper at ICLR 2025
that the count-based objective in COPO can also be combined with the online KTO algorithms to further enhance its performance.
Table 7: The result comparison of online KTT and count-based KTO.
KTO
KTO
KTO
KTO+ COPO
KTO+COPO
KTO+COPO
(iterl)
(iter2)
(iter3)
(iterl)
(iter2)
(iter3)
L.C. Win Rate
33.19
35.40
35.90
35.32
36.84
37.10
C.4
ABLATION OF REWARD MODEL
Due to the fact that online RLHF methods require additional reward module to give preference labels compared to the offline method, we adopt a small-size reward model to ensure that the ad- ditional requirement is minimal. Specifically, We follow the SELM baseline use a small-sized PairRM (0.4B) model to rank responses generated by LLM and contain the best and worst responses according to the reward model. As mentioned in our paper, using powerful reward model can further improve performance. As a result, we add additional experiments that use powerful 8B re- ward model from RewardBench (Lambert et al. 2024), named
Dong et al. 2024) for comparison. The learned models evaluated by AlpacaEval 2.0 are given in the following Table. The result shows that both COPO and SELM have improved in performance, with COPO showing more significant improvement. The reason is that the data coverage of generated responses in COPO broader than SELM since we adopt count-based exploration in alignment. Then an accurate reward model is required for preference labeling in such a large data coverage.
Table 8: The ablation of more powerful reward model with ArmoRM-Llamu3-8B.
SELM (w/ PairRM-O.4BB)
SELM
COPO
COPO
(w/ AmMRMM-BB)
(w/ PairRM-O.44B
(w/ ArmoRM-8B)
LC Win Rate
34.67
39.21
35.54
41.78
C.5 DISCUSSION OF BEST-OF-N AND PPO
The Best-of-N policy a method for aligning samples from LLMs to human preferences. It involves drawing N samples, ranking them, and returning the best. For best-of-N, we consider two types of methods to integrate this strategy into the RLHF process. (1) Using best-of-N to improve the quality of preference data in offline RLHF. Specifically, we use prompts from the UltraFeedback dataset and regenerate the chosen and rejected response pairs (yw, 31) with the LLM. For each prompt x, generate N responses using the SFT model with a low sampling temperature (e.g., 0.8 in the experiment). We then use PairRM to score these responses, selecting the highest-scoring one as yo and the lowest-scoring one yt. We denote this method DPO-best. (2) Using best-of-N as an exploration method in online RLHF, In each iteration of online DPO, we denote the current LLM policy и and the best-of-N variant as πT2 In this way, the से policy increases the margins between Ti and provides more exploration. We denote this variant by online-DPo DPO--bstt
The comparison evaluated in AlpacaEval 2s given the following table, where we use n 5. We find that the best-of-N strategy significantly improves the performance of the offline DPO method, which improves the coverage of the offline dataset. Meanwhile, the best-of-N strategy also enhances online DPO also improves the performance of online DPO. We note that although best-of-N provides another efficient exploration strategy, is more computationally expensive and still underperforms our method, which signifies that the proposed count-based objective is efficient and stronger exploration approach. We do not apply the best-of-N strategy in evaluation since all methods follow the same evaluation pipeline in a standard benchmark, including AlpacaEval 2.0, MT-Bench, and LLM leaderboard.
As for PPO, (1) we note that reproducing the successful RLHF results with PPO is challenging as it requires extensive efforts and resources that the open-source communities usually cannot afford, which has been discussed in previous works (Ivison al. 2024). Specifically, the PPO algorithm requires loading multiple LLMs simultaneously. including the actor (policy), critic (value network), reward model, and reference model (for KI. estimation), which places significant pressure GPU
22

---

Published as a conference paper at ICLR 2025
Table 9: The combination of DPOVonline-DPO with Best-of-N
DPO
DPO--est
online DPO
online DPO best
ours
LC Win Rate
22.5
26.0
33.17
34.12
35.54
memory that exceeds resources in group. (2) Meanwhile, the performance of PPO usually heavily relies on the quality of the reward model. Due to the narrow distribution coverage of the preference dataset, reward misspecification often occurs, which makes the reward model assign high value to out-of-distribution (OOD) samples and has the potential to be exploited during the RI. process. In contrast, using better reward model (Lambert al. 2024) trained on larger dataset will significantly boost the performance, while the comparison to the DPO-based method becomes unfair since it leverages knowledge beyond the fixed preference dataset.
D DISCUSSION OF OTHER EXPLORATION METHODS
The use of count-based bonus 1/ I/VN (s) was originally proposed in tabular cases to encourage ex- ploration in RL, where the count (s) of each state E S can be calculated accurately since the total number of states are finite. However, for envirouments with high nigh-dimensional state space, we cannot obtain the exact count for states since the state space is infinite, thus pseudo-counting mech- anism is required to estimate the Ñ (s). In online RL exploration literature O, several methods have been proposed to estimate the pseudo-counts, including neural density model, hash code, random- network distillation (RND), and CFN. However, find only CFN is suitable for LLM estimate the pseudo count of prompt-response pairs for the following reasons.
(1) For the neural density models (Bellemare et al. 2016), we have ((s) = (cPG,, - 1)- where (x,.(x) is the prediction gain that measures density changes after using the specific state to update the density model. This calculation process is hard to implement in LLM because training a density model for prompt-response pairs is highly expensive (Ostrovski et al.) 2017), and is also difficult to measure density changes by using a single prompt-response pair to update the LLM. (2) For hash code (Tang et al. 2017). requires mapping prompt-response pairs
hash table, and the method assigns exploration bonuses based on the frequency of state visits. This approach is also hard to implement since it requires training an autoencoder (AE) that takes a prompt-response pair as inputs and outputs the hash code in the latent space, then the AE is required to reconstruct the prompt- response pair. Meanwhile, can suffer from hash collisions, especially for the response space of LLMs, potentially leading to inaccurate exploration objectives. (3) RND (Burda et al. 2018) is popular exploration method in RL, determined by the output difference between a randomly initial- ized network and trained network's prediction of the state. Despite its simplicity and empirical success, RND's 's exploration bonus is challenging to interpret, as it is based on an unnormalized dis- tance within a neural network's latent space. Additionally, there are rigorous theoretical results to prove the RND reward is exactly the pseudocount of the state.
Compared to previous works, CFN (Lobel al. 2023) represents a significant advance in explo- ration methods using the Rademacher distribution to estimate pseudo counts. This approach does not rely on density estimation but instead uses the averaging of samples from the Rademacher dis- tribution to derive state visitation counts. CFN's key advantages include its theoretical grounding, which allows it to provide exploration bonuses equivalent to count-based objectives, and simplic- ity and case of training. Unlike other methods, CFN does not impose restrictions on the function approximator or training procedure, offering flexibility in model architecture selection. In our work, we set the CFN as lightweight network with several fully connected layers on top of pre- pre-trained hidden layer. The empirical result in online RL also demonstrates CFN's effectiveness, particularly in complex environments, and its ability to generalize well to novel states.
23


Reviews:
[{"review_id": "t0QgWyjuYJ", "reviewer": "Reviewer_4QFJ", "summary": "This paper introduces the COPO algorithm for better exploration of online RLHF formulation. The authors provide a theoretical background of the proposed method and explain the practical implementation of it. To be specific, it assumes that the reward function could be expressed as a dot product between a parameter vector and a feature vector. Based on it, the model incorporates a USB term for exploration. Also, they employ a lightweight Config Flipping Network (CFN) to obtain pseudo-visit counts for the large state space of language generation. Experimental results show the efficacy of the proposed COPO algorithm. It performs better than the other exploration-aware online RLHF baseline, SELM, in several benchmarks. When they adjust the coefficient for the exploration term, it supports the importance of the exploration term in their experiments.", "questions": "* How do we measure/believe the CFN works properly? (before testing the model on the final benchmarks.)\n  * (Declaimer) I do not have any idea about the original CFN work. However, the model (CFN module) seems too small to represent the hidden state of the modern LLM.\n  * If we can improve the performance of the module, will the final alignment performances be also improved?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "It tackles the important problem innate in RLHF-based alignment learning, i.e., exploration, with formal theoretical motivation and practical algorithm.", "weaknesses": "* Their explanation and selection of baseline methods are not enough. Especially, I think other iterative/online methods should have been \nconsidered in addition to the SELM. There are many variants to implement online DPO. For example, [1], [2], and [3].\n\n* The improvements seem marginal despite the complexity of the proposed method. The performance gaps among the baselines could be changed with different random seeds, especially for the LLM-as-a-Judge tasks.\n\n* It would be good if more benefits of the exploration were suggested. For example, it might be more robust in some adversarial cases.\n\n[1] Self-Rewarding Language Models (Yuan et al., 2024)\n\n[2] Aligning Large Language Models by On-Policy Self-Judgment (Lee et al., 2024)\n\n[3] D2PO: Discriminator-Guided DPO with Response Evaluation Models (Singhal et al., 2024)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UQORMfAxlT", "reviewer": "Reviewer_j1HG", "summary": "This paper identifies a central problem in current online RLHF methods, that is, efficiently exploring the prompt-response space in each iteration. To address this, the paper designs a novel RLHF algorithm named Count-based Online Preference Optimization (COPO), which re-formulates the original DPO objective to balance space exploration and preference optimization. The authors adopt a simple coin-flip counting network approach to support COPO's pseudo-count, achieving promising experimental results.", "questions": "1. What would be the experimental results if COPO's main ideas were applied to other online RLHF algorithms?\n\n2. Are there other alternative or explorable counting algorithms?\n\n3. How would choosing a stronger Reward model (or Judge model) affect algorithm performance, and can this be demonstrated with specific experimental results?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Solid theoretical analysis. This work is built upon a solid theoretical foundation, starting from a relatively reasonable linear reward assumption and ultimately analyzing the regret upper bound of the optimization objective, providing a more credible motivation.\n\n2. The authors demonstrate good insights into the existing problems in online RLHF, specifically the focus on preference optimization while lacking effective space exploration. Around this issue, they propose an algorithm suitable for LLMs with vast token sequence spaces.\n\n3. The proposed COPO algorithm shows notable improvements after multiple iterations.", "weaknesses": "1. The proposed COPO algorithm is only implemented based on the DPO optimization objective, without verification of its adaptability to improve other online RLHF algorithms.\n\n2. The visit count item in the COPO algorithm relies solely on a pseudo-count mechanism implemented by a coin flipping network (CFN) with accompanying random labels, which seems too simplistic. More convincing visit count algorithms need to be explored.\n\n3. As mentioned by the authors in the experimental section, a better reward model could improve COPO's performance. However, this point wasn't explored in the experiments. In fact, the choice of different scoring RMs has a significant impact on the experiment and should be given more attention.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t1cpQGeN22", "reviewer": "Reviewer_pkVP", "summary": "This paper introduces an online RLHF approach via the Count-based Online Preference Optimization (COPO) algorithm, which integrates count-based exploration and direct preference optimization to improve performance. Experiments on Zephyr and Llama-3 models demonstrate COPO’s effectiveness in boosting instruction-following capabilities and general academic benchmarks.", "questions": "Please see the weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper rigorously establishes the theoretical underpinnings of COPO, providing a regret-bound analysis that demonstrates the efficiency of the UCB-based exploration strategy.", "weaknesses": "It seems there’s no notable improvement over SELM for the Llama-3-8B Instruct model in Table 1, especially when considering the potential variance from using GPT-4 as the evaluator. And can authors provide more baselines, for example comparison with Best-of-N and PPO?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)