Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective
Authors: Jiawei Huang
Bingcong Li
Christoph Dann
Niao He
Venue: ICML 2025 poster (2025)
Keywords: Reinforcement Learning from Human Feedback
Reward transfer
sample-efficient reinforcement learning
Primary area: theory->reinforcement_learning_and_planning
TL;DR: We study the reward transfer in online RLHF. We propose a theoretical transfer learning algorithm with provable benefits, and then develop an empirical version with improved scalability and experimental evaluations.

Abstract: Sample efficiency is critical for online Reinforcement Learning from Human Feedback (RLHF). While existing works investigate sample-efficient online exploration strategies, the potential of utilizing misspecified yet relevant reward models to accelerate learning remains underexplored. This paper studies how to transfer knowledge from those imperfect reward models in online RLHF. We start by identifying a novel property due to KL-regularization in the RLHF objective: \emph{a policy's coverability of the optimal policy is captured by its sub-optimality}. Building on this insight, we propose novel transfer learning principles and a theoretical algorithm---**T**ransfer **P**olicy **O**ptimization (**TPO**)---with provable benefits compared to standard online learning. Empirically, inspired by our theoretical findings, we develop a win-rate-based transfer policy selection strategy with improved computational efficiency. Moreover, our empirical transfer learning technique is modular and can be integrated with various policy optimization methods, such as DPO, IPO and XPO, to further enhance their performance. We validate the effectiveness of our method through experiments on summarization tasks.

Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective
Jiawei Huang' Bingeong Li
Christoph Dann Niao He
Abstract
preference labels, there has been significant attention in re- ducing the sample complexity the amount of data required for training- online RLHF through efficient exploration strategies (Cen et al., 2024; Wang et al., 2023; Xie et al., 2024; Zhang et al., 2024). However, largely overlooked op- portunity is additionally leverage existing reward models for annotation, which have already aligned partially with the human preferences. The growing number of available open- source and high-quality reward models trained on diverse tasks provide a rich pool of candidates for transfer learn- ing. Harnessing guidance embedded in such reward models holds great potential for improving sample efficiency.
Sample efficiency critical for online Reinforce- ment Learning from Human Feedback (RLHF). While existing works investigate sample-efficient online exploration strategies, the potential of uti- lizing misspecified yet relevant reward models to accelerate learning remains underexplored. This paper studies how to transfer knowledge from those imperfect reward models in online RLHE. We start by identifying novel property due to KL- regularization in the RLHF objective: a policy's coverability of the optimal policy is captured by its sub-optimality. Building on this insight, we pro- pose novel transfer learning principles and a theo- retical algorithm-Trunsfee Policy Optimization (TPO>withh provable benefits compared to stan- dard online learning. Empirically, inspired by our theoretical findings, we develop a win-rate-based transfer policy selection strategy with improved empiri- computational efficiency. Moreover, cal transfer learning technique is modular and can be integrated with various policy optimization methods, such as DPO, IPO and XPO, to further enhance their performance. We validate the effec- tiveness of our method through experiments on summarization tasks.
There are variety of practical scenarios where such source reward models be effectively utilized. Firstly, reward models trained relevant tasks often prove to be valuable in similar tasks. A notable example is cross-lingual reward transfer (Hong et al., 2024; Wu et al., 2024), where reward models in one language can provide effective guidance for tasks in another. Secondly, informative evaluation can also be obtained from well-trained LLMs, such as GPT, LLaMA and Gemini (Achiam al., 2023; Dubey et al., 2024; Team et al., 2024). For certain tasks, such models can provide evaluation closely aligned with human preferences; see e.g., Ji et al. (2023); Lee et al. (2023). Lastly, there are scenar- ios where rule-based or heuristic reward functions built upon experts knowledge and accumulated experience- are inexpensive to obtain and instructive in evaluating the LLM. Taking summarization tasks as an example, expert sum- maries available on datasets such XSum (Narayan et al., 2018) and TL-DR (Völske al., 2017). Similarity with those expert solutions can be measured through metrics such as ROUGE (Lin, 2004) and BERTSSoOee (Zhang et al., 2019), and be employed for scoring the LLM generations.
1. Introduction
Reinforcement Learning from Human Feedback (RLHF) has achieved remarkable success in fine-tuning Large- Language Models (LLMs) to align with human preferences (Bai et al., 2022: Christiano et al., 2017; Ouyang et al., 2022). Using datasets annotated with human preferences reflecting human intrinsic reward model, RLHF optimizes LLM policies with reinforcement learning (RL) techniques. Due to the high cost of collecting large amounts of human
Motivated by these considerations, this paper studies how to leverage imperfect reward models to learn a near-optimal policy with fewer human annotations. We consider the case where several source reward models are available, yet their quality, i.c., the similarity to human rewards, is unknown priori. Our contributions are summarized as follows.
'ETH Zurich GGoogle Research, Correspondence to: Jiawei Huang jjiawei.huang wet.ffff.t.cc..mm ethz.ch>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
In Sec. 3, we identify a distinctive property of RLHF aris- ing from its KI. regularization: for any prospective policy

---

Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective
candidates, its coverability for the optimal policy improves as its policy value increases. Enlightened by this, we pro- pose two new principles for transfer learning in the context of RLHF (illustrated in Fig. 1).
Reward Transfer in Online RLHF
Online Data pfhallal al
π
Offine leamiin with online data
Firstly, policy value serves as a crucial criterion to se- lect the transfer policy, since exploiting policies with high values does not conflict with exploration. Secondly, com- bining insights from offline RL theory, we prove that the policy distilled by offline learning techniques from the online data generated by any no-regret online algorithm, converges to the optimal one at a rate of Ö(T=*) after a finite time. Such bound improves existing sample complexity results and regret bounds in online RLHF by eliminating the dependencies on the size of the state and action space, or the complexity of the policy class (up to log-covering number). This suggests the offline policy computed with the data from the online learning process is a promising candidate to transfer from, which we term as "self-transfer learming".
Online Human Annotation
Transfer Learning
Wrrannder
argmmax policy value
Transfer Policy Candidates
Hosti
Ground-True Human Faedback
Imperfact Reward Models (Unknown Quality)
Figure I. The standard online RLHF pipeline only involves learn- ing from online human feedback (left). Our setting additionally leverages available imperfect reward models via transfer learning (right). Inspired by the structure induced by KL. regularization, we propose novel principles for transfer learning in online RLHF: (1) selecting transfer policy Vitastter with the highest policy value; (2) self-transfer sarning-involvingg as a candidate the policy mosd distilled from online collected data by offline learning techniques.
In Sec. 4, following above principles, we design a transfer learning algorithm named Transfer Policy Optimization (TPO; Alg. 1) with provable benefits. At the core of TPO is the self-transfer learning and an adaptive policy selection strategy that picks transfer policies based on estimated value gaps. In the early stage, the cumulative online regret of TPO can be significantly reduced, as long as one of the source rewards is of high quality. After finite time, the self- transfer mechanism ensures the regret of TPO grows only at a rate of Ö(VT), independent of the standard structural complexity measures. Compared with transfer learning in the pure-reward maximization RL, our result is novel in that it exploits the policy coverage property induced by the regularization term in RLHF.
Sample Complexity in RLHF Online RLHF emphasizes strategic exploration for sample-efficient learning in tabular and linear settings (Du et al., 2024; Novoseller et al., 2020; Pacchiano et al., 2021; Xu et al., 2020), as well as more general function approximation cases (Cen et al., 2024; Chen et al., 2022; Wang et al., 2023; Xie et al., 2024; Xiong et al., 2024; Ye et al., 2024; Zhang et al., 2024). Our work further improves sample efficiency by leveraging imperfect reward models that are readily available in variety of practical scenarios. As an alternative, offline RLHF (Huang et al., 2024; Liu et al., 2024; Zhan et al., 2023) focuses on exploiting pre-collected datasets without exploration. What lies in between online/offline RL is hybrid RL (Chang et al., 2024; Gao et al., 2024). These methods harness online feedback, while assuming the reference policy provides good coverage and only engaging in passive exploration.
To reduce computational overheads and improve scalablity to practical RLHF scenarios, in Sec. 5, we propose empirical version of TPO that selects transfer policy based on the win rates competing with the learning policy, rather than relying on the estimated policy values. On the one hand, win rates can be estimated much more efficiently than the policy values. On the other hand, as justified by theory, win rates help to identify lower bounds for the coverability of the optimal policy. Notably, our empirical TPO is general: its core transfer learning technique can be modularized and combined with various reward-model- free policy optimization methods, such as DPO (Rafailov et al., 2024), IPO (Azar et al., 2024), XPO (Xie et al., 2024), to boost their performance. The effectiveness of approach is demonstrated in fine-tuning T5 models summarization tasks in Sec. 6.
Transfer Learning in RL and RLHF Transfer learning in pure-reward maximization RL has been extensively investi- gated in previous literature (Taylor & Stone, 2009; Zhu et al., 2023), and theoretical guarantees have been established un- der various conditions (Golowich & Moitra, 2022; Huang & He, 2023; Huang et al., 2022; Mann Choe, 2013). Unlike previous works, this paper unveils new insights for trans- fer learning enabled by the KL regularization in RLHF In particular, it enables us to design policy-value-base trans- fer policy selection strategy, and identify a unique regime, i.e., "self-transfer learning", that can significantly improve sample efficiency. We defer more discussions to Sec. 3.2.
1.1. Related Work
We summarize the most related ones on sample complexity and transfer RL. here and defer the others to Appx. B.2.
Most works on transfer learning in RLHF focus on empirical
2

---

Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective
approaches. For example, (Hong et al., 2024; Wu et al., 2024) investigate the cross-lingual reward transfer. To our knowledge, our empirical algorithm (Alg. 3) is novel in that it studies active transfer policy selection, which is still underexplored in existing literature. We further distinguish our work from RLAIF (Ji et al., 2023; Lee et al., 2023) or reward model selection literature (Nguyen et al., 2024). Their goal is to align LLM policies with surrogate reward models, while we study how to leverage those surrogates to accelerate the alignment with ground-truth human rewards.
in practice, for detailed reasons discussed in Appx. B.I.
We use r" : x A - [0, R] to denote the unknown true reward function determining the human's preference P,-. For simplicity, we omit r* and use (T( as short note for JJ( r(). Following previous works (Xie et al., 2024; Zhang et al., 2024), we consider the function approximation setting with access to a policy candidates class II (II) < +o0) satisfying standard assumptions as follows:
Assumption A. The policy class II satisfies: (I) Realiz-
ability: The optimal policy n..- E IT. (II) Bounded Policy Ratio: For any π E II, a log sTaas] [auc(a[s) 8
2. Preliminary
In this section, we review the mathematical formulation of RLHF for LLMs and introduce our reward transfer setting.
(alss)) Note that max,,0 log (aer(als)
1 holds as long as r E 3
[0, R] (see Lem. B.1 in Appx. B.1). Thus, Assump. A-(II) is not actually an assumption, but can be interpreted as an additional filtering step by leveraging the boundary of r".
2.1. Mathematical Formulation of RLHF
We adopt the contextual bandits framework (Lattimore & Szepesvári, 2020; Ouyang et al., 2022), where we treat the prompt space as the state space S and the response space as the action space A. Without loss of generality, we assume that both |S] and |A| are finite. We denote p E (SS) as the prompt distribution. An LLM can be modeled by policy T S + (AA), where, given a prompt & E S, the responses are generated from the conditional distribution a ~ π (-Iss). Throughout this paper, we assume that all considered L.LM policy π have positive support over the entire state-action space, that is, min,,a (a]s)) > O. This is ensured in practice by the softmax layer in LLM architectures.
A Additional Notation We use the standard big-O notations and use (.) (e.g. Ô) to suppress logarithmic terms, including log-covering numbers. Given II satisfying Assump. A, CTI)(TI) denotes its convex hull, and R. denotes the reward class induced by 11 via Eq. (2), s.t., (1) Vr E R r E [0, Rt]: (2) Er R' - ग.... We defer to Appx. B.1 for detailed converting process. Besides, we denote [n] == {1, 2, ..,, n} and a A b min (a, b). We refer the reader to Appx. A for table of commonly used notation in this paper.
2.2. Reward Transfer Setting
Reward Model and Preference Model A reward model is a function r S x A + [o, Rl.. where R is a constant indicating the largest possible reward value. In the RLHF setting, the reward r is unobservable and we can only access its induced preference model, denoted by P.. Pr 8 a, a) denotes the probability that is preferable to ā given (y- 1) or not (y- 0), by reward model Following previous works, we consider the Bradley- Terry (BT) model (Bradley & Terry, 1952): (y = |18 a, ă) = (r(s,a)) r(s, d()), where (xx) = 1/ (1 + ") is the sigmoid function.
We assume there are W source reward models available, de- noted by { tivated in Sec. I, those reward models are accessible in many scenes. Given a source reward y", let N.... be the correspond- ing source policy and denote A (w)) := Ja( - J2(T;")) as its policy value gap. We define Amin == min A(w) to be the minimal gap for all source policies. Note that Amin 2 0 and Amin 0 implies r (rr") Twe[] We do not assume prior knowledge on { {A(un (w) (w)])e[[].
Jw=1 s.t. Vw, 8, a, r" (s, a) E [0, It.. As mo-
LLM Policy as Reward Model Eq. (2) implies that there is a way to convert a given LLM policy ' to a reward model. Concretely, by choosing an arbitrary distribution mu. we can interpret / log To[-1. as a reward function that π aligns with given mo as the reference policy (Rosset et al., 2024). There- fore, while we consistently use the term "reward transfer" throughout the paper for clarity, our framework is general to handle transfer learning from any LLM policy through the underlying reward function aligns with.
RLHF Learning Setting Given a reward model r, in the context of RLHF for L.LM fine-tuning, we are interested in optimizing the following the KL-regularized objective:
x(-l-)
J.. f- arg (TT)) x
with (T(TT)) = Esppara (r(8, a)] (KL((T (m))))))) (1)
where we use Trat denote the pretrained reference pol- icy and KL(TT] (err)) == Fs-p, [oog (= e) The above optimization problem yields a closed-form solution:
T(a|s)
2.3. Background on Policy Coverability
(als) Teer (als)e "mm
We first introduce the formal definition of the policy cov- erage coefficient, which measures how well a given policy distribution covers the other.
Here is a moderate constant and is critical for RLHF
3

---

Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective
Definition 2.2. Given any n, R, the coverage coefficient for के by π is defined by Cov := [.spp. [[
The key insight of Lem. 3.1 is that: for any prospective candidates of the optimal policy (i.e., conv(IT)') or any transfer candidates (i.e., {Tz=> } its coverability of m-. is controlled by its policy value gap. Intuitively, Covâle becomes extremely large or even unbounded if there is a significant distribution shift between π and म.. However, in the presence of regularization (8 > 0), we should only consider policies with bounded policy ratio relative Tze (see Assump. A-(II), and exclude those (near- deeer--detemmiiistie ones from our policy candidate class II, because none of them can be (near-Joptima) In other words, regularization leverages prior knowledge from ūeer and enables a free policy filtration step before learning begins, ensuring that the remaining policies exhibit a favorable structure (Lem. 3.
The concept of policy coverage originally emerged from offline RL (Chen& & Jiang, 2019; Yang et al., 2020; Zhan et al., 2022), where one aims to find good policy given fixed dataset. The coverage of the optimal policy by the data-generating policy naturally governs the size of the dataset required to find the optimal policy. Policy coverage was later extended to online RL, inducing novel complexity measures to characterize intrinsic statistical difficulty of the underlying MDP (Amortila ct al., 2024; Xie et al., 2022).
Compared to alternative complexity measures, policy cover- age is particularly suited for studying sample complexity in RLHF, since optimizing exploring directly on the policy (LLM) space is preferred for its computational tractability. We use coverage as an analytical tool for reward transfer, and our results are based on RPO (Liu et al., 2024) and XPO (Xie et al., 2024) for offline and online RLHF, respectively: Lemma 2.3 (Offline RLHF; Thm. 5.3 in (Liu et al., 2024); Informal). Given a dataset D generated by a policy n²,
To understand why such property is uniquely arising from regularization, consider bandit instance with single op- timal arm and multiple suboptimal arms yields rewards R and R 2s, respectively. In pure reward maximization RL (B = 0), the optimal policy Tp. is deterministic. A policy class II satisfying Assump. A may include several suboptimal deterministic policies. The coverage coefficient between any of them and T.p is infinity, while their subopti- mal gaps are 2. and can be arbitrarily small.
running RPO with any R. including r" and TI yields म, s.t.,
VI E II, Jg(x)) - JT( = Ö(E2R "ov=l= (D]-
Lemma 2.4 (Online RLHF; Thm. 3.1 in (Xie al., 2024); Informal). Running XPO with II satisfying As- sump. A for T steps yields a policy sequence {T') JE= s.t. EL Js(xt-) - J3(T') = Ö(R c2R. (ooos (II)))
3.2. New Insights for Transfer Learning in RLHF
In the online RLHF, the primary goal of exploration is to discover high-reward regions, i.e., the states and actions cov- ered by the optimal policy. Therefore, in our reward transfer setup, we propose to transfer from the policy with the best coverage T.... Inspired by Lem. 3.1, we identify two novel principles for transfer learning for RLHF objective, which we will further explore in later sections.
Lem. 2.3 states that it possible to compute offline policy competitive with any policy well-covered by the dataset. Lem. 2.4 suggests that the sample efficiency of online RLHF can be characterized by the Los coverability Co (II). Here, &o (II) by Xie et al. (2022) is a worst-case version of our Def. 2.2 as it takes the maximum over s, a and स. See Appx. D for the formal definition.
Principle 1: Select Transfer Policies with High Policy Value By Lem. 3.1, exploiting policy with high value for data collection could "help" exploration, because such a policy inherently provides good coverage for ग.... In other words, regularization reconciles the trade-off between explo- ration and exploitation. This insight allows us to use policy value as a criterion and transfer from the policy achieving the highest value among all candidates. This strategy also practical given that policy values can be estimated well.
3. The Blessing of Regularization: A Policy Coverage Perspective
Unlike classical pure-reward maximization RL, the RLHF objective in (1) incorporates regularization with respect to Tzet. We start by identifying distinctive properties associ- ated with such regularization in Sec. 3.1, and discuss their implications on transfer learning in RLHF in Sec. 3.2.
We emphasize that this principle is unique in the regular- ized setting. As exemplified by the bandit instance before, near-optimality does not imply good coverage for xy. in the absence of regularization. To avoid negative transfer in pure reward maximization setting, previous algorithms typically rely on additional assumptions about task similarity and employ sophisticated strategies to balance exploiting good source tasks with exploration (Golowich & Moitra, 2022;
3.1. Structural Property Induced by Regularization
Lemma 3.1. Under Assump. A and assume Td E [0, R) all we E [W], then, for any policy E (I) 1W
J=(TZ-) - -J)(T)
1o < 1 + (
8
Herre we consider the convex hull in order incorporate all possible uniform mixture policies induced by IT.
where k(x) := (x-1)²
z-1-logz Ozx).
4

---

Can RLHF be More Efficient with Imperfect Reward Modcls? A Policy Coverage Perspective
Huang & He, 2023), which can be challenging to general- ize beyond the tabular setting. In contrast, regularization enables us to filter transfer policies directly with their policy value, facilitating the applicablity beyond the tabular setup.
Outline of Main Algorithm Our main algorithm TPO is provided in Alg. 1, which leverages Alg. 2 T-aasfeee Policy Selection (TPS) as subroutine to select source policies to transfer from. TPO can be regarded as mixture of standard online learning and transfer learning, balanced through a a E (0, 1). Motivated by the implication of Thm. 3.2, TPO returns the detailed policy computed with all the data collected. For convenience, we divide the total number of iterations T into K = T/NN blocks, each containing N sub-iterations. In each block, we first run a iterations of an OnLine learning algorithm A1 goz. followed by (1 a).N iterations of transfer learning with policy selected by Alg. 2. Here, Aog can be any online algorithm with per-step no-regret guarantees, for example, XPO in Lem. 2.4. To save space, we defer to Appx. D the formal behavior assumption on Alg (Def. D.2) and concrete examples with verifications.
Principle 2: Transfer from the Policy Distilled from Online Data-the "Self-Transte Learning" We first introduce key result by combining Lem. 3.1 and offline RLHF result in Lem. 2.3.
Theorem 3.2. Under Assump. A. w.p. 1- ô, given an online dataset DD generated by policy series {x²) E conv(IT) running RPO with conv(II) and R. on 'D yields a distilled
policy TDat such that,
Jo(TTT-) - (T(Da))) I
(4)
0(02h (1
(1 x(+x(e #1)
Jz(TT-) -
BT
A Preview of Main Theorem Before diving into the de- tails, we first highlight the benefits of transfer learning by presenting an informal corollary regarding the regret bound of TPO, under concrete choices of A gou and a.
To understand the significance of Thm. 3.2, consider the case when {T²) Heery are produced by a no-regret online learning algorithm, such as XPO in Lem. 2.4. As a result, the term Σi= J=(((=)-J in Eq. (4) diminishes to O as T increases. This implies that the policy Das distilled from online data by offline learning techniques converges to nτ.- at rate of O(T- ), which does not depend on |8), or other complexity measures such as Co (IT) in Lem. 2.4. This result not only strictly improves the sample complexity bounds for existing online RLHF algorithms (Cen et al., 2024; Xie et al., 2024; Xiong et al., 2024; Zhang et al., 2024), but also reveals a fundamental difference from the pure reward maximization setting, where lower bounds depending on those structural complexity factors have been established (Auer et al., 2002; Dani et al., 2008). We defer detailed discussions to Appx. E.3.
ST
Corollary 4.1. Choosing XPO (Xie et al., 2024) as A Al and a = c E small and (VTT) regret after T is large enough.
#, TPO achieves Ô(W VT) regret when T is
Coro. 4.1 is implied by our main result in Thm. 4.4. We refer to Remark F.3 for a detailed quantification of "small" and "large enough". Compared with previous online RLHF results without transfer learning, in the early stage, TPO im- proves the structural complexity measure coefficients (e.g. Co in XPO) to the number of source tasks W, which is usually much smaller. Besides, it even gets rid of such coefficient term and achieves O(VT) regret over time.
Next, we take a closer look at the transfer policy selection steps in Alg. 2 in Sec. 4. and provide detailed analyses and discussion of TPO in Sec. 4.2.
More importantly, the faster rate of the convergence of Dal to गदद also indicates the potential of using Dat as a candi- date for policy transfer. We term this regime as welf-transfer learning", and refer Taat as the "self-transser policy". No- tably, Dast continuously improves and converges to TP. as the dataset grows, while the source policies { Ter= re- tain fixed non-zero value gaps due to the imperfections in reward models (x"") " W This reveals another benefit of self-transfer learning: it helps to avoid being restricted by suboptimal source reward models.
4.1. Details for Alg. 2: The Transfer Policy Selection
The design of Alg. 2 follows the two principles in Sec. 3.2, which are: (1) transfer the policy with the highest (esti- mated) policy value, because higher policy value implies better coverage for . (2) include the self-transfer policy as a candidate, because it progressively converges to n..- at faster rate than the best-known ones for online policies.
4. Provably Efficient Transfer Learning
We first clarify some notation. Given a dataset D := {(s a², ".,y", yy, )))) Lo(r) denotes the average nega- tive ligeliieddood (NLL) loss regarding the reward r:
In this section, we develop provably efficient transfer leam- ing algorithms based on the principles in Sec. 3.2.
See Cond. C.I for the definition of data generation process. Beeondd sample complexity, a regret bound improved to Ö(VT) for online RLHF can be established. We defer it to Coro, 4.5 after presenting our main results.
(r(r)=
1
Σ
-y' log o (r(s), a') - r(s' ,0))
D
(1 1--v v²) log a (r(s) (z)) - r(s', (5)
5

---

Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective
Algorithm I: Transfer Policy Optimization (TPO)
1 Input: Block size N; Number of blocks K = T//;; {r") sw=1i 11; (0, (0,1)
2 For all (k,n), n). D denotes all the data collected up to (k:, n), and D only includes those collected by All got. See detailed definitions in Appx. F.I.
3 for k = 1, 1..... K do
for n .... do
in S aN then khnn +- -Al (aT, 11, 6; Di else Tknn + II, (r")) >===i Dk.n)
Collect data (m kin, a . n) ~ P x x Rnn x Teet x (+).)))
7
end end
return mp. computed by RPO with D
9
We will use pp [r] := Ee-pp.rx [rss, a)] as short note. In Line 4 of Alg. 2, (ww; (w: == Eis I[x² ni,] denotes the number samples collected with Tf in the dataset, fol-
and Def. D.2, w.p. 1- 8, in each call Alg. 2:
Jo(T,,) -Jg(T - -6 Re
vC(I))
(Cov ^
lowing the convention that (.) = +00 if N = O. In Line 6, we leverage RPO (Liu et al., 2024) to compute the celf-transfer policy Dau and a reward function FDsn. To save space, defer the details of RPO Appx. C.
VID
I (iTDas D) S (T( - Jو (TTae))
(6)
Here π :- TD Eis DD T² denotes the mixture policy. In the LHS, the coefficient takes minimum over two factors Cov C.v=,=/=m0 1 and /C(TT)/a, resulting from two different ways to estimate the value gap of TDst. According to offline RLHF theory (see Lem. C.2), TDsa competitive with any π E conv(II) well-covered by the dataset distribution, or equivalently, (3( (J((T = Jg(m,-) J2(T)) + Ö(D) à Cov)) ). By choosing т = ,.,,, we obtain the first bound with the factor Co Next, considering π akN EiskJsaN xッJ, the uniform mixture of poli- cies generated by Algo. so far, leads to the second bound involving VC(II)/.. This also explains why we still in- volve normal online learning in TPO-to provide another safeguard for the quality of the transfer policy.
Next, we explain our value estimation strategy. Note that in RLHF setting, we cannot access r directly but only the pref- erence comparison samples following the BT model. Thus, we instead estimate the value gain relative to Jg(Frer).
Optimistic Estimation for - (F(e))) For poli- cies induced by imperfect source reward models, we adopt UCB-style optimistic policy evaluation to efficiently balance exploration and exploitation. Intuitively, by utilizing the MLE reward estimator ÍALE. the estimation error PNLL - r* under the distribution of ,,... is related to the number of samples from -.. occurring in the dataset. Therefore, we can quantify the value estimation error as follows.
Lemma 4.2 (Value Est Error for { )we(w)). Under As- sump. A and Def. D.2, w.p. 1 5, in each call of Alg. 2:
4.2. Main Theorem and Interpretation
Vie E [W],
) - (T( < (TTT::))
We establish the per-step regret bound for TPO below. Theorem 4.4 (Total Regret). Suppose Algar is a no- regret instance satisfying Def. D.2, whose regret grows as ÖRReR / CCTT)TT) for any intermediate step T and some policy class complexity measure C(TI). Then, w.p. 1 - 25, for any T/K- < t S T. ranning TPO yields a regret bound:
22n
<Ja(T -Ja(Trer) +0( VN(w;D)
Pessimistic Estimation for (T The main challenge in estimating the value of TDat is that, Daal is not fixed but changing and improving. The previous optimistic strategy is not applicable here, since the coverage of maa by the dataset is unclear, making difficult quantify the uncertainty in estimation via count-based bonus term. For- tunately, given that Daai is improving over time, it is more important when it surpasses all the other source policies. Therefore, it suffices to construct a tight lower bound for ( - J, (Js(Trer); see line 7. By leveraging fxL. and the optimality of (TD PDa) for the RPO loss, we can show:
(t) Σ J3(T,,)- J (T)(())))))))))))))))))))))))))) = Reg + Reg Ter TSt
Regal (t) -Ō(Re VaC(II)))
Reg (6) =ō
Σ
Amin A M(T),n(r)
(8)
Tst:aN (stt)a<n(T)SSNN
eRR
+ +22R (1-o)WWA
A(w)
Lemma 4.3 (Value Est Error for apa)). Under Assump. A
(w))
6

---

Can RLHF be More Efficient with Imperfect Reward Modcls? A Policy Coverage Perspective
Algorithm 2: Transfer Policy Selection (TPS)
1 Input: Source tasks {r") J w==i Policy class 11; T, 8; Dataset D := {[s([(())))))) a', ā', y H')DD:
2 // Optimistic estimation for (T( (Trer)
3 FELL < arg min, Lr(r).
4 Vw [W], (TTLe D) + E Ep. [ML=] pe [xLe]] - (T [Tee)) + 16e2R
N(w:D] log I|TT
5 // Pessimistic estimation for J/p (TDsa) Jp (TTer).
6 Daa, PDat + (PT) R" D, with 17 c. (1+ (1+ ) V [2]
log I|T 6
7 V(TDa: D)- <- E Ep [aa]
E Ferr [Faa] - Tree) ID((aa) (.D((PL))) 2
n|T log
[D]
6
8 return arg max heepe (I:;)) D). // Selecting Transfer Policy by Estimated Value
Here we denote k((T) := [ and n(T) T be the block index and inner iteration index for step T; ((()),(()) == Ö((Re2 Covii-lma Ii A yC(l)) V²),, where
T mod N to
much lower than the policy class complexity measure C(TI).
Therefore, Eq. (9) implies a significant improvement over the typical online learning regret bound without transfer.
mix Eisg Th((), is the mixture policy up to Ti (ww) and Amin denote value gaps as defined in Sec. 2.2.
Stage 2: M
W In this case, the second term in Eq. (8)
A min
is controlled by O(EwE(w) X(ي)) O W and we have the following regret bound:
Amii = 0 O(Vi),
We decompose the total regret into two parts depending on their origins. Regót (t) comes from the regret by running the online algorithm Al1 It is weighted by since we only allocate a- a-proportion of the samples for Algos. Reglet represents the regret from transfer policies. The first term in Eq. (8) reflects the benefits of utilizing transfer policy over online learning. Here Amin is contributed by source reward models { Jwe[w]. and the term (((),(((F) is due to the telf-transser policy" TDal. as we derived in Lem. 4.3. The second term in Eq. (8) results from the imperfection of source reward models: without prior knowledge on their quality, additional cost has to be paid during exploration.
(t) Regtr = =Ö(V CITY a2
A E.
(10)
At the first glance, the RHS is controlled by Ö( (((I))//// /a), which implies transfer learning at most suffer a factor of 1 1/a larger regrets than no transfer. However, in fact, the term
V Erss (Cov")-/=in yields a much tighter bound, which only grows as Ö(VE) after finite time, and is independent of C(II). To see this, by Lem. 3.1 and the concavity of
Ja(-), we have Cov Inch 1++Ō(((e ) RegEl + RRogeet ). 2t
Note that Eq. (9) and (10) already indicate a regret upper bound Reg (t) VTT where I is short note of coefficient depending a, W, (A(w))wE(W) and C(II), but not f. This implies Cov Irim converges to at the rate of O(1/V),, and Regi") = Ō(Vi) after finite time.
Next, we elaborate the benefits of transfer learning by taking a closer look at Regif= in Eq. (8). Note that the lower Reg (t) is, the faster म.. in TPO converges to n... When Amin 0, i.e. r* is realizable in (r")) (4) = Ö(VWE Ew)A(w)> ((w))) and the benefit of transfer learning is clear. Thus, in the following, only focus on the case Amin > O. We separately consider two scenarios, according to the relationship between / and Amin- For clarity, we will omit the constant terms R and e
(r"") [e[[W]. we have 1
Reg
Although the above provable benefits in Stage 2 result pri- marily from "self-transfer learning", high-quality source reward models also play an important role here. According to Eq. (9), small Amin can lead to small I and therefore, accelerate the convergence of Cov | towards 1.
A² This corresponds to the early learning Stage 1: t < W2
hyperparameter. Without prior knowledge, a simple choice is a = e with guarantee in Coro. 4. Under prior beliefs that high-quality source reward models are available, may prefer smaller a. Besides, due to the self-transfer learning, it is wise to gradually decay a to 0 as the iteration number grows.
Remarks on choice of a We treat
stage, when is relatively small. In this case, Thm. 4.4
implies the following regret bound:
Regin: = O(VI- Amint)) Õ(W Vt), (9)
which can be further improved to Ö(VWI) if
W
This suggests at the earlier stage, the benefits of transfer is contributed mostly by the source reward models (r") }" In general, we can expect the number of source tasks W
Improved regret bound for standard online RLHF as a implication Although our focus is transfer learning, the standard online RLHF setting can be recovered when no
7

---

Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective
Algorithm 3: Empirical TPO
1 Input: K, N and {r" bwe wij Initialize: Toe Trefi
2 For all (k,n) E [K] x [M]] and all w E [W], DK. = ( ahis, مام yyhd)}. Nk.() (-) = Ejen I[[ Add, and ": > moad == Nلंيo Eyen I[. = nhod
3 for k: = 2..,, K do
for n2 = ....N do
Ye [W], WR
+ -P (T7 > moa) + ev log //Nk." (TTT))
evlog
WR T + Pr- (To + TOL) = 0.5. // WR tet
can be treated as hyperparameter taking value other than 0.5.
7
n Thn +- arg (E 1 (( WR
Collect online data (shin, ahin, ah." ./h p Th.n x T x Prr (1)ss))
9
end
k+1
Algpo (m DI:;+1);
10
end
12 return T K+1
source tasks are present, i.e., W 0. As stated below, TPO achieves Ö(VT) regret over time by purely utilizing "self-transfer learning", thereby strictly improving existing results (Cen et al., 2024; Xie et al., 2024; Xiong et al., 2024; Zhang et al., 2024)*.
Lemma 5.1. Under BT-modef, for any π: Cov
V (1+21P,- (x > R)) log <>U,5
max
1+2
7 + V [Ja(m
23
Note that we may not identify the policy with the best cov- erage for स.. through the lower bound above. However, still provides useful guidance for practice: we filter out policies yielding high lower bound. In Lem. 5.1, for any fixed 7 and comparator म., the lower bound for Cov increases as Pr, (τ > R) decay to O, suggesting prioritizing transferring from policies with high win rates.
Corollary 4.5. When W = O, under the same condition of Thm. 4,4, TPO reduces to a standard online RLHF method with OVTT) regret after finite time.
5. From Theory to an Empirical Algorithm
The key question now is how to choose the comparator R. According to Lem. 5.1, ideally, the comparator should be close to n..,, that Jn( allowing the win rate term to dominate the lower bound. Since we do not know ,,- in advance, empirically, we can choose the learning policy as the comparator, which is optimized and progressively converges to n....
In terms of computational overheads, TPO requires solving multiple minimax optimization problems, which restricts its applicability to fine-tune LLMs in practice. To address this, adhering to the design principles of TPO, we introduce more computationally efficient alternative in Alg. 3.
Ja( becomes negligible,
Key Insight: Estimating Win Rates instead of Policy Values As discussed in Sec. 4, several optimization steps are designed to estimate policy values used for transfer policy selection, because they help to identify the policies' coverability for optimal policy (i.e. lov The key insight in our empirical algorithm design is to find a more accessible indicator infer Co This leads to the policy win rates, i.e., the probability that human prefer the generation by one policy over another. Formally, given two policies π, न. the win rate of ने over π is defined by: (T, > π) == E.ppp..0 a' rust TP,- (y = 1|8, lss,a, a')]
From Insights to Practice Next, we walk through em- pirical TPO Alg. 3 and explain how we integrate these insights into the algorithm design. Alg. 3 utilizes an itera- tive online learning framework, which repeatedly collects online data and optimizes the policy. We start by initializ- ing the online learning policy Thu with the reference pol- icy Trat. For computational efficiency, in each iteration k we avoid separately computing online exploration policies and self-transfer learning policies as done in TPO. Instead, we only compute one policy T (updated from कें. by Algeo Here Algpo is a placeholder for arbitrary Policy Optimization algorithm, and we do not restrict the concrete choice. Such design increases the modularity of our em-
Win rates between two policies can be unbiasedly estimated by querying human preferences with their generated re- sponses. Moreover, win rates be used to construct lower bound for I. stated in Lem. 5.1 below.
'Lem. 5.1 can be generalized beyond BT-model. Besides, it is possible to construct lower bound involving P.. (# T) instead. See Lem. G.3 and Remark G.4 Appx. G for more details.
While omitted in our result, the dependence on the log- covering number (e.g., log [T]) matches with those previous works.
8

---

Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective
pirical TPO, making it possible to combine with various policy optimization methods and enhance their performance. For example, Algro may be instantiated by DPO (Rafailov et al., 2024), resulting in a transfer learning framework built on iterative-DPO (Xiong et al., 2024; Yuan et al., 2024). Be- sides, one may consider other advanced (online) methods, such as XPO (Xie et al., 2024), IPO (Azar et al., 2024), etc.
Without Transfer
Purely Exploit ROUGE-Lsum
Purely Exploit
La-Lar
Iter 1
52.1 + 1.2
53.1 + 1.1
49.5 = 0.9
Iter 2
53.3 = 1.6
54.5 + 1.3
49.1 = 0.4
Iter
54.0 + 1.2
53.3 1.5
50.6 = 0.3
Table I. Win rates (%) of the policies trained by empirical TPO (Alg. 3) are competed with 3 baselines, presented across 3 columns. Baseline (I): without transfer, ie.. iterative- DPO. Baseline (II): purely utilizing ROUGE-LSum (the lowest-quality source task) in transfer learning. Baseline (III): purely utilizing T5-Large (the highest-qualitg source task) in transfer learning. Results are aver- aged with 3 random seeds and 95% confidence levels are reported.
As the core ingredients of our empirical TPO, during data collection, the algorithm selects the policy zhm E (Tp- W w== U (moL) J with the highest win rate when compet- ing against ToL. Intuitively, we encourage transfer learning if { WW includes high-quality candidates; otherwise, the algorithm conducts standard iterative policy optimiza- tion with Algro by default. This strategy also aligns with the heuristic principle: learn from an expert until surpassing it. Lastly, since the win rates are unknown in advance, the selection process is formulated as multi-armed bandit prob- lem. We employ a UCB subroutine (line 7) to balance the exploration and exploitation during the win rates estimation.
i.c., nh. = T. in Alg. 3 for the selected r".. Here the worst and best reward models are selected hased on the final policy value when aligning with the given reward model.
Table. 1 reports the win rates of the policies learned by Alg. 3 competing with the three baselines. As shown in Col- umn 1, comparing with normal online learning, our transfer strategy demonstrates clear advantages. Furthermore, Col- umn 2 and 3 suggest that, without prior knowledge of source tasks quality, our method avoids being misled by low-quality tasks and achieves competitive performance compared to exploiting the best reward candidate.
6. Experiments
In this section, we evaluate Alg. 3 on the summarization task using the XSum dataset (Narayan et al., 2018). We consider T5 series models (Raffel et al., 2020) and choose T5- (80M) as the base model for fine-tuning. Human reward r" is simulated by the reward model (Dong et al., 2024) distilled from Llama3-8B (Dubey et al., 2024). For the source rewards in transfer learning, we consider a collection of imperfect reward models and L.LM policies, including, (a) ROUGE-LLum score (Lin, 2004), (b) BERTScore (Zhang et al., 2019), (c) T5-base (250M), (d) T5-large (770M). As illustrated in Sec. 2.2, for LLM policies (c) and (d), we treat their log-llootee probabilliyy predictions on the given prompt 8 and response a as the reward scores.
Notably, as suggested by the additional results in Appx. I, T improves over time and P,, (TT... > T for any w E [W] continuously decreases. In iteration 3, our em- pirical TPO automatically switches back to online learning and avoids being restricted by the sub-optimality of source reward models. the end, it results in higher win rates than purely exploiting the best source reward model T5-Large over 3 iterations.
7. Conclusion
For Algpo we consider three instantiations: DPO (Rafailov et al., 2024), IPO (Azar et al., 2024) and XPO (Xie al., 2024). To save space, we present and interpret the results with DPO as the choice below and defer other experiment results and also the concrete setups to Appx. I.
This paper studies reward transfer in the context of online RLHF, We contribute TPO, a provable and efficient transfer learning algorithm that leverages the structure induced by the KI. regularizer. Based on that, we further develop UCB-based empirical alternative and evaluate its effective- ness through LLM experiments. Several promising direc- tions remain for future exploration. Firstly, an interesting avenue is to develop transfer learning strategies beyond RLHF setting, for example, the Nash Learning from Human Feedback setting. Secondly, while we focus on policy-level transfer, finer-grained prompt-wise knowledge transfer may be possible, which allows transfer from different poli- cies in different states. Thirdly, due to resource limitations, we leave the examination of our methods in fine-tuning much larger-scale language models to the future work.
Experiment Results and Discussion We run Alg. 3 for K = 3 iterations and compare its performance with three baselines: (I) vanilla ite without transfer learn- ing (i.e., setting Algro DPO and W = 0); (II) purely exploiting the worst source reward-ROUGE score; (III) purely exploiting the best source Soward-TS-large Con- cretely, baseline (I) removes the transfer learning compo- nent in Alg. 3 by assigning N = = u... for all n E [N]. For basclines (II) and (III), the worst (ROUGE-LSum) and best (T5-Large) reward models from the candidates (a)-(d) are selected, and pure transfer learning is then performed using the responses recommended by the chosen reward models,
9

---

Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective
Acknowledgement
Chen, J. and Jiang, N. Information-theorrtic considerations in batch reinforcement learning- In International Con- ference Machine Learning, pp. 1042-1051. PMLR, 2019.
The work is supported by ETH research grant and Swiss National Science Foundation (SNSF) Project Funding No. 200021-207343 and SNSF Starting Grant.
Chen, X., Zhong, H., Yang, Z., Wang, Z. and Wang, L. Human corman-in-thee in-the-loop: Provably efficient preference-basse reinforcement learning with general function approxima- tion. In International Conference on Machine Learning. pp. 3773-3793 PMLR, 2022.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information pro- cessing systems, 30, 2017.
Reproducibility Statement
The code of all the experiments and the running in- structions can be found in https:/ /gi thub. com/ jiaweihhuang/ RLHE _ewarddraasse.r
Dani, V., Hayes, T. P., and Kakade, S. M. Stochastic linear optimization under bandit feedback. In COLT, volume 2, pp. 3. 2008.
References
Dong, H., Xiong, W., Pang, B., Wang, H., Zhao, H., Zhou, Y., Jiang, N., Sahoo, D., Xiong, C., and Zhang, T. Rlhf workflow: From reward modeling to online rlhf, 2024. URL https://arxii. org/abs/2405.0786. 2024.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S.,
Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Du, S., Kakade, S., Lee, J., Lovett, S., Mahajan, G., Sun, W., and Wang, R. Bilinear classes: A structural framework for provable generalization in rl. In International Conference on Machine Learning, pp. 1826-2836. PMLR. 2021.
Amortila, P., Foster, D. J., and Krishnamurthy, A. Scal- able online exploration via coverability. arXiv preprint arXiv:2403.06577 2024.
Auer, P.. Cesa- Bianchi, N., Freund, Y., and Schapire, R. E. The nonstochastic multiarmed bandit problem. SIAM journal on computing, 32(1):48-77 2002.
Du, Y., Winnicki, A., Dalal, G., Mannor, S., and Srikant, R. Exploration policy optimization in rlhf: Theoret- ical insights on efficient data utilization. arXiv preprint arXiv:2402.1 2tf.::2002.03444 10342, 2024.
Azar, M. G., Guo, Z. D., Piot, B., Munos, R., Rowland, M., Valko, M., and Calandriello, D. A general theoretical paradigm understand learning from human preferences. In International Conference Artificial Intelligence and Statistics, pp. 4447-4455. PMLR, 2024.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXi::4407221783, 2024.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training helpful and harmless assistant with rein- forcement learning from human feedback. arXiv preprint arXiv:2204.05880 05862, 2022.
Foster, D. J., Kakade, M.. Qian, J., and Rakhlin, A. The statistical complexity of interactive decision mak- ing. arXiv preprint arXiv:21 /2. 13487, 2021.
Gao, L., Schulman, J., and Hilton, J. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835-10866. PMLR, 2023.
Bradley, R. A. and Terry, M. E. Rank analysis of incom- plete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
Gao, Z., Chang. J. D., Zhan, W., Oertell, O., Swamy, G., Brantley, K., Joachims, T., Bagnell, A., Lee, J. D., and Sun, W. Rebel: Reinforeement learning via regressing relative rewards. arXiv preprint arXiv:2404.16767 2024.
Cen. S., Mei, J., Goshvadi, K., Dai, H., Yang, T., Yang. S., Schuurmans, D., Chi, Y., and Dai, B. Value-incentivized preference optimization: A unified approach to online and offline rlhf. arXiv preprint arXiv:2405. 19320, 2024.
Geist, M., Scherrer, B., and Pietquin, O. A theory of regu- larized markov decision processes. In International Con- ference on Machine Learning. pp. 2160-2169. PMLR, 2019.
Chang, J. D., Zhan, W., Oertell, O., Brantley, K., Misra, D., Lee, J. D., and Sun, W. Dataset reset policy optimization for rlhf. arXiv preprint ar0: 200.049900 2024.
10

---

Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective
Gibbs, A. L. and Su, F. E. On choosing and bounding probability metrics. International statistical review, 70 (3):419-435, 2002.
low bellman rank are pac-leamable. In International Con- ference Machine Learning, pp. 1704-1713. PMLR, 2017.
Golowich, N. and Moitra, A. Can q-learning be improved with advice? In Conference on Learning Theory, pp. 4548-4619 PMLR, 2022.
Jin, C., Liu, Q. and Miryoosefi, S. Bellman eluder di- mension: New rich classes of rl problems, and sample- efficient algorithms. Advances in neural information 2021a.
processing systems,
Guo, S., Zhang, B., Liu, T., Liu, T., Khalman, M., Llinares, F., Rame, A., Mesnard, T., Zhao, Y., Piot, B., et al. Direct language model alignment from online ai feedback. arXiv preprint ar Ntl:2402.04792, 2024.
Jin, Y,, Yang, Z., and Wang. Z. Is pessimism provably efficient for offline ri? In International Conference on Machine Learning, pp. 5084-5096. PMLR, 2021b.
Hong, J., Lee, N., Martinez-Castaño, R., Rodríguez, C.,, and Thome,, J. Cross- looss-l transfer of reward models in multilingual alignment. arXiv preprint arXiv:2410. 1802,, 2024.
Lattimore, T. and Szepesvári, C. Bandit algorithms. Cam- bridge University Press, 2020.
Lee, H., Phatale, S., Mansoor, H., Lu, K. R., Mesnard, T, Ferret, J., Bishop, C., Hall, E., Carbune, V., and Rastogi, A. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. 2023.
Huang, A., Zhan, W., Xie, T., Lee, D., Sun, W., Krishna- murthy, A., and Foster, D. J. Correcting the mythos of kl-regularization: Direct alignment without overparame- terization via chi-squared preference optimization. arXiv e-prints, pp. arXiv-2407, 2024.
Lin, C.-Y. Rouge: package for automatic evaluation of summaries. Text summarization branches out, pp. 74-81, 2004.
Huang. J. and He, N. Robust knowledge transfer in tiered reinforcement learning. Advances in Neural Information Processing Systems, 36:52073-52085 2023.
Liu, Z., Lu, M.. Zhang, S., Liu, B., Guo, H., Yang, Y., Blanchet, J., and Wang, Z. Provably mitigating overopti- mization in rlhf: Your sft loss is implicitly an adversarial regularizer. arXiv preprint arXiv:2405. /6436, 2024.
Huang, J., Zhao, L., Qin, T., Chen, W., Jiang, N., and Liu, T.- Y. Tiered reinforcement learning: Pessimism in the face of uncertainty and constant regret. Advances in Neural Information Processing Systems, 35:679-690, 2022.
Mann, T. A. and Choe, Y. Directed exploration in reinforce- ment learning with transferred knowledge. In European Workshop on Reinforcement Learning, pp. 59-76. PMLR, 2013.
Jaques, N., Gu, S., Bahdanau, D., Hernández-Lobato., J. M.,
Tumnrr, R. E., and Eck, D. Sequence tutor: Conserva- tive fine-tuning of sequence generation models with ki- control. In International Conference on Machine Learn- ing. pp. 1645-1654, PMLR, 2017.
Munos, R., Valko, M., Calandriello, D., Azar, M. G., Row- land, M., Guo, Z. D., Tang, Y., Geist, M., Mesnard, T., Michi, A., et al. Nash learning from human feedback. arXiv preprint arXiv: 12.022282000066 2023.
Jaques, N., Ghandeharioun, A., Shen, H., Ferguson, C. Lapedriza, A., Jones, N., Gu, S., and Picard, R. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXiv: 1907.00007 2019.
Narayan, S., Cohen, S. B., and Lapata, M. Don't give me the details, just the summary! topic-aware convolu- tional neural networks for extreme summarization. ArXiv, abs/ has//888..0 0011180...8777 000.008755500 2018
Nguyen, D., Prasad, A., Stengel-Eskin, E., and Bansal, M. Laser: Learning to adaptively select reward models with multi-armed bandits. arXiv preprint arXiv:2410.01735. 2024.
Ji, J., Qiu, T., Chen, B., Zhang, B., Lou, H., Wang, K., Duan, Y., He, Z., Zhou, J., Zhang, Z.,, et al. Ai alignment: A comprehensive survey, arXiv preprint arXiv:2310. 19852, 2023.
Novoseller, E., Wei, Y., Sui, Y., Yue, Y., and Burdick, J. Dueling posterior sampling for preference- rein- forcement learning. In Conference on Uncertainty in Artificial Intelligence, pp. 1029-1038. PMLR, 2020.
Jiang N. and Huang, J. Minimax value interval for off- policy evaluation and policy optimization. Advances in Neural Information Processing Systems, 33:2747-2755 2020.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S.. Slama, K., Ray, A., et al. Training language models to follow instructions
Jiang, N., Krishnamurthy, A., Agarwal, A., Langford, J., and Schapire, R. Contextual decision processes with

---

Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective
with human feedback. Advances in neural information processing systems, 35:27730-27744 2022.
exploration. In International Conference on Machine Learning, pp. 34161-34221. PMLR, 2023.
Pacchiano, A., Saha, A., and Lee, J. Dueling rl: reinforce- ment learning with trajectory preferences. arXiv preprint arXiv:21 2021.
Uchara, M., Huang, J., and Jiang, N. Minimax weight and q-function learning for off-policy evaluation. In Interna- tional Conference on Machine Learning, pp. 9659-9668. PMLR, 2020.
Pang, R. Y., Yuan, W., Cho, K., He, H., Sukhbaatar, S., and Weston, J. Iterative reasoning preference optimization. arXiv preprint ar arXiv:2404. 19733, 2024.
Völske, M., Potthast, M.. Syed, S., and Stein, B. TL;DR: Mining Reddit to learn automatic summarization. In Proceedings of the Workshop on New Frontiers in Sum- marization, pp. 59-63, Copenhagen, Denmark, Septem- ber 2017. Association for Computational Linguistics. doi: 10. 10.18653\w1/W117455008 URL https: /www. aclweba www.w..gggg...togggooooooooo
Rafailov, R., Sharma, A., Mitchell, E., Manning, D., Er- mon, S., and Finn, C. Direct preference optimization: Your language model is secretly reward model. Ad- vances in Neural Information Processing Systems, 36, 2024.
Wang, Y., Liu, Q. and Jin, C. Is rlhf more difficult than standard rl? arXiv preprint arXiv:2306. 14111, 2023.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 (140):1-67, 2020.
Wu, Z., Balashankar, A., Kim, Y., Eisenstein, J., and Beirami, A. Reuse your rewards: Reward model trans- fer for zero-shot tross-lingual alignment. arXiv preprint arXiv:2404 2404.22880,, 2024.
Rosset, C., Cheng, C.-A., Mitra, A., Santacroce, M., Awadal- lah, A., and Xie, T. Direct nash optimization: Teaching language models to self-improve with general preferences. arXiv preprint a5427:2404.03715 2024.
Xie, T., Cheng, C.-A., Jiang, N., Mineiro, P., and Agarwal, A. Bellman-consistent pessimism for offline reinforce- ment learning. Advances in neural information process- ing systems, 34:6683 1116683-6694,, 2021.
Russo, D. and Van Roy, B. Eluder dimension and the sample complexity of optimistic exploration. Advances in Neural Information Processing Systems, 26, 2013.
Xie, T., Foster, D. J., Bai, Y., Jiang, N., and Kakade, S. M. The role of coverage in online reinforcement learning. arXiv preprint arXiv:2210.04155 2022.
Sason, I. and Verdú, S. f-divergence inequalities. IEEE Transactions on Information Theory, 62(11):5973-6006, 2016.
Xie, T., Foster, J. Krishnamurthy, A., Rosset, C., Awadal- lah, A., and Rakhlin, A. Exploratory preference optimiza- tion: Harnessing implicit q"-approximation for sample- efficient rlhf. arXiv preprint arXiv.2405.22046. 2024.
Shi, R., Zhou, R., and Du S. S. The crucial role of samplers in online direct preference optimization. arXiv preprint arXiv:2409. / 19605, 2024.
Xiong, W., Dong, H., Ye, C., Wang, Z., Zhong, H., Ji, H., Jiang, N., and Zhang, T. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. In Forty-first International Con- ference on Machine Learning, 2024.
Swamy, G., Dann, C.,, Kidambi, R., Wu, 7. S., and Agarwal, A. A minimaximalist approach to reinforcement learning from human feedback. arXiv preprint trXiv:2401.04033 2024.
Xu, J., Lee, A., Sukhbaatar, S., and Weston, J. Some things are more cringe than others: Preference opti- mization with the pairwise cringe loss. arXiv preprint arXiv:2312. 2023.
Taylor, M. E. and Stone, P. Transfer learning for reinforce- ment learning domains: A survey. Journal of Machine Learning Research, 10(7), 2009.
Team, G., Georgiev, P., Lei, V. I., Burnell, R., Bai, L.,, Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al. Gemini 1.5: Unlocking multimodal understand- ing across millions of tokens of context. arXiv preprint arXiv:2403.05530. 2024.
Xu, Y., Wang, R., Yang, L., Singh, A., and Dubrawski, A. Preference- reinforcement learning with finite-time guarantees. Advances in Neural Information Processing Systems, 04118784-18794 2020.
Yang, M., Nachum, O., Dai, B., Li, L., and Schuurmans, D. Off-policy evaluation via the regularized lagrangian. Advances in Neural Information Processing Systems, 33: 6551-6561, 2020.
Tiapkin, D., Belomestny, D., Calandriello, D., Moulines, E., Munos, R., Naumov, A., Perrault, P., Tang, Y., Valko, M., and Menard, P. Fast rates for maximum entropy
12

---

Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective
Ye, C., Xiong, W., Zhang, Y., Jiang, N., and Zhang. T. A the- oretical analysis of nash learning from human feedback under general kl-regularized preference. arXiv preprint arXiv:2402.073// 2024.
Yuan, W. Pang, R. Y., Cho, K., Sukhbaatar, S., Xu, J., and Weston, Self-rewarding language models. arXiv preprint arXiv:2401.10020 2024.
Yue, Y., Broder, J., Kleinherg, R., and Joachims, T. The k-armed dueling bandits problem. Journal of Computer and System Sciences, 7865):1588-1556 2012.
Zhan, W., Huang, B., Huang, A., Jiang, N., and Lee, J. Of- fline reinforcement learning with realizability and single- policy concentrability. In Conference on Learning Theory, pp. 2730-2775. PMLR, 2022.
Zhan, W., Uehara, M., Kallus, N., Lee, J. D., and Sun, W. Provable offline preference-based reinforcement learning. arXiv preprint arXiv:2305.1453 14816, 2023.
Zhang, S., Yu, D., Sharma, H., Zhong, H., Liu, Z., Yang, 7., Wang, S., Hassan, H., and Wang, Z. Self-exploring language models: Active preference elicitation for online alignment. arXiv preprint vXXi:22405.19332. 2024.
Zhang. T., Kishore, V., Wu, F., Weinberger, K. Q., and Artzi, Y. Bertscore: Evaluating text generation with bert. arXiv preprint Www::II9..0999.... 2019.
Zhao, H., Ye, C., Gu, Q., and Zhang, T. Sharp analysis for kl-regularized contextual bandits and rlhf. arXiv preprint arXiv:2411.0463 2024.
Zhu, Z., Lin, K., Jain, A. K., and Zhou, J. Transfer learning in deep reinforcement learning: A survey. IEEE Trans- actions on Pattern Analysis and Machine Intelligence, 2023.
Ziebart, B. D. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Carnegie Mel- lon University, 2010.
Ziebart, D., Maas, L., Bagnell, A., Dey, K., al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pp. 1433-1438. Chicago, IL, USA, 2008.
13

---

Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective
Outline of the Appendix
Appx. A: Frequently Used Notation.
Appx. B: Missing Details in the Main Text.
Appx. C: Offline Learning Results in Previous Literature.
Appx. D: Verification for Online Learning Oracle Example in Sec.
Appx. E: Proofs for Results in Section 3.
Appx. F: Proofs for the Main Algorithm and Results in Sec. 4.
Appx. G: Connection between Win Rate and Policy Coverage Coefficient.
Appx. H: Useful Lemmas.
Appx. Missing Experiment Details.
14

---

Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective
A. Frequently Used Notation
Notation
Description
S,A
State space and action space
P
Prompt distribution (initial state distribution)
Reward model
r*
Ground-truth reward model (reflecting human preferences)
(r PT > T) JEEW]
Preference under
Win rate of over iτ under r Imperfect source reward model
LLM policy
Tii
Uniform mixture policy & Eise X' of a policy sequence n] ,.,,,,"'" Sometimes, given dataset D = {(x', a t') with bit abuse of notation, we use =D to refer the mixture policy [2 Lisp T.
T'.
Covala
Coverage coefficient
II
The policy class
R.
The reward function class converted from II, see Appx. B.1
cIIv(II)
Convex hull of II
B
Regularization coefficient in RLHF objective
(-)-)
Regularized policy value (Eq. (1))
Aww)
Value gap for Ty i.e. J3( Jo JT( Minimal value gap min (ww)
Amin
a a
min(a, b}
I]]
(1.2....,1) m}
(0) 6(-)
Standard Big-O notations, (-) omits the log terms.
For completeness, we provide the definition of convex hull here.
Definition A.1 (Convex Hull). Given policy class II with finite cardinality (i.e. [[] < +ox), we denote conv(Il) as its convex hull, such that, Vn E [N*], VX A" 2 0 with E X = 1, and any π ...., T7" E: II, we have:
Σ X conv(ll).
Remark A.2. Note that in the contextual bandit setting, the state action density induced by policy and the policy distribution collapse with each other. Therefore, given a policy sequence π² r',... n², the uniform mixture policy = (-I-) Eis n' ('(-)) is directly valid policy mapping from S to (AA), which induces the state-action density mfix (-I-).
Besides, we will use OL and OFF as abbreviations of "online learning" and "offline learning", respectively.
B. Missing Details in the Main Text
B.I. Extended Preliminary
More Elaborations on the Necessity of Regularization The RLHF objective Eq. (1) typically involves a regularization term 3 * 0. This regularization is critical in practice for several reasons. Firstly, it prevents overfitting to human preferences, which can possibly be noisy and biased (Gao et al., 2023; Ouyang et al., 2022). Moreover, pure reward maximization prefers near-deterministic policies, potentially causing mode collapse. In contrast, regularization encourages the fine-tuned model to retain diversity from the reference policy (Jaques et al., 2017: 2019). Thirdly, reference policies are pretrained on significantly larger corpus than the post-training data, enabling them to encode more general general-purpose purpose knowledge. Regularization helps mitigate catastrophic forgetting, ensuring the model retains this broad knowledge base.
15

---

Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective
Formal Definition for R. Given policy class IT satisfying Assump. A, use R to denote the reward function class converted from II, such that (1) Vr RD r(:)) E [0, RI: (2) Fr R क m.. A possible construction satisfying this is given by
T(als) min 3 log
(Talls)
R = (1,) Clip [3 log
Frar (a'ls) π E 11).
(11)
T-ar (als)
d'EA
The rationale behind such a construction lies in that M,p provably differs from r* by at most of an action-independent constant under Assump. A. We prove this in the following.
For any E S, we denote == arg ming'e log mi (a'ls) According to Eq. (2) and the fact that r" E [0, RR, for any (a)ls)
E S and a, a' E A, we have:
n.. (als) min 3 log
-. (a'ls) Trot (a'ls)
S 3 log
(" -s"(s, S R,
Trat (a|s)
e'EAA
where the first inequality is because o, takes the minimal over A. Therefore, "|=;- (8, a) E [0, R] and r|;- (s,,)) r" rs(ssa)) which is intiendnddeee..nt.. In another word, M|=;- induces the same optimal policy nT...
Under the objective in Eq. (1), the realizability assumption in (Liu et al., 2024) can be relaxed and the reward model class Rl can be used in their RPO objective. Because any per-state action-independent shift on the reward space does not change the induced policy in Eq. (2). As a result, throughout this paper, we will not distinguish between r" and "|x
Remarks on Assumption A-(II)
Lemma B.1. (s, a) E [0, for all (s, E ES* A, we have:
Th (a|s)
Rmax
max sES,aEA
log
(r)r(a
5
3
Proof. By definition, for any 8,
[(.a)) /Z(s),
Ya A, (als) (aes(a 6
where Z(s): = :5)==A (alr() b Because r (s, E [0, RJ. obviously, 5 Z(s) Se Therefore,
T.. (als)
R r" rr(() a) log Z()) S max log 2(x), log (()) < 3
R
Va E A,
flog
11-
Toot (a(s)
B.2. Other Related Works
Other Related RLHF Literature Various approaches have been developed for reward-model-free online exploration. For example, DPO (Rafailov al., 2024) implicitly optimizes the same objective RLHF without explicit reward modeling. DPO is further extended to different settings, see e.g., online DPO (Guo et al., 2024), iterative DPO (Dong et al., 2024; Pang et al., 2024; Xu et al., 2023), etc.
Another direction is to go beyond Bradley- -erry reward model assumption. A particularly promising set of techniques formulates RLHF as a two-player zero-sum game (Yue et al., 2012), aiming to select policies preferred by the rater to others (Munos al., 2023; Rosset et al., 2024; Swamy al., 2024; Ye al., 2024). Investigating knowledge transfer within this framework is exciting direction for future work.
RL Theory in Pure-Reward Maximization Setting In the classical pure-reward maximization RL setting, sample efficiency is a central topic, with extensive research dedicated to strategic exploration and fundamental complexity measures for online learning (Du et al., 2021; Foster et al., 2021; Jiang et al., 2017; Jin et al., 202 Russo & Van Roy, 2013).
Besides the literature already mentioned in Sec. 2.3, there is a rich literature (Jiang & Huang, 2020; Jin et al., 2021b; Uchara al., 2020; Xie et al., 2021) investigating the role of policy coverage (or density ratio) in offline learning.
16

---

Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective
Regularized RL Sample complexity in regularized RL has also been studied in previous works (Geist al., 2019; Tiapkin al., 2023; Ziehart, 2010; Ziebart et al., 2008). Nonetheless, most of them focus tabular settings and do not consider the transfer learning.
C. Offline Learning Results in Previous Literature
In this section, we recall and adapt some results from (Liu et al., 2024), which are useful for proofs in other places.
RPO Optimization Objective For completeness, we provide the optimization objective of RPO. Given a policy class IT and reward function class R, the RPO objective solves mini-max optimization problem defined follows:
RIO R, D, n) arg max min (r(r) + T.B.sp. ant [r(s, a) r(s,a))) (x(
(12)
Tell rER.
where we choose uzer the base policy in (Liu al., 2024). In Alg. 2, set IT conv(II) and R R Condition C.1 (Sequential Data Generation). We say a dataset == {(st at a', .yy, (x (sspp generated sequentially, is generated following:
Vi S [D], n' Alg(-1((s), a² a. v. (.px))) Biss),
s² s'~ a' ~ (+1s), a ~ xree(t|s"),, ~ "
where Alg denotes an algorithm computing the next policy only with the interaction history.
Lemma C.2. [Adapted from Thm. 5.3 in (Liu er al., 2024)] Under Assump. A. given any 5 (0, 1), by rumning RPO
(Eq. (12)) with conv(IT), R", 5 and dataset 'D = {(s a², n² v xt x')ssp satisfying Cond. C.I, I, by choosing
(1 + B & /2/ 5
we have:
| log
Va (TI, Ja(T)) Ja(xDa)) S C 2R Cov
D
8
where we use mix =
1 D Eigipj r² as short note of the uniform mixture policy.
Proof. The main difference comparing with (Liu et al., 2024) is that we consider sequentially generated dataset while they study dataset generated by fixed dataset distribution. In the following, show how to extend their results setting.
Firstly, we check the assumptions. Note that we consider feed RPO (Liu et al., 2024) by the reward function class R. converted from a policy class II satisfying Assump. A, through Eq. (1I), Therefore, the optimal reward is also realizabile in R. and the basic assumptions required by RPO (Liu et al., 2024) are satisfied.
Next, we adapt the proofs in (Liu et al., 2024). Note that we can directly start with their Eq. (D.4), because their bounds Eq. (D.2) and Eq. (D.3) only involve optimality of the choice of Dest and realizability. We move the KL-regularization terms to the LHS and merge (TT) and J3 (Daa), and choose Nzef as the base policy in RPO. The adapted results to our notations would be:
VI cIIv JD(T) Jo(
< max E Barppora((e) (s, r(a))) ( - r(s,a)) rERn
+ (Cp(r)) Co(r))
Recall Cp is the (unnormalized) negative log-likelihood (NLL) loss, defined in Eq. (5), Since the dataset D is generated sequentially (Cond. C.I), we can apply the concentration results in Lem. H.2, which a variant Lemma D.1 in (Liu et al., 2024) for sequentially generated data:
w.p. - 8, VT E conv(ll), JT(T) - (D(DDa))
<T Seo
(s, (r-=(8,a) ( (8,
17

---

Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective
+ +1 (LD((r) (
< (1s)
11|
(1pb) 11
log
7
8
(1)))
- (s.))))))))))))))))_)))))))))))) (r-n (s,
2
|TT log
E Bespann (1s)) (1 []]] 7/D
1
8
where we denote re= == arg maXTER" Bemp, (a) (sr))(( r (s, - (r(s. a) - (s,
The rest of the proofs in (Liu et al., 2024) can be adapted here, and by choosing 1/ (1 + td) 24 inherit the following guarantee:
log
we can
v5 +1
II
w.p. 1 5, 5,
VI E cInv JT( - (TDa) S 4
(
(R" n. mee=) + +11
log
mis
Here C. (R. T Tret) is the coverage coefficient (adapted from Assump. 5.2 (Liu et al., 2024)) with T which can be upper bounded by:
r(s, (t
Es-p.av (sll (r, (r(s,
Cap (R T; T
S max rERu
Bevpann (110) (11)) (11a [1ll(r - (8, - (r(*, - r(8, (8
T (als)
Sv E Eenparr (-1+)
mix (als)
(AM-GM inequality; Holds for any and therefore including the one achieves the maximum)
Co
Therefore, we finish the proof. We simplify the upper bound by using Cov and e 2
D. Details for Online Learning Oracle Example in Sec. 4
Definition D.1 (Ioo Coverability; (Xie et al., 2022; 2024)). The Loo coverability is defined by:
inf sup sup (als)
Cov (II)
HEA(S)xA(A) aE u(als)
Definition D.2 (No-Regret Online Algorithm). Given any 8 e (0, 1), iteration number T, and a policy class II satisfying Assump. A, the online learning algorithm A1 You iteratively computes policy to collect samples and conducts no-regret learning. W.p. 6, it produces a sequence of online policies n| "..,,",, TT, such that, Vt E [T].
I 8
Es(Ttt)) - Ja(TT) s CoL R l(I))
where Cot, > 0 and co 2 1 are absolute constants, and C(LL) denotes some complexity measure for II.
Proposition D.3. [Example for Online Oracle in Def. D.2] The XPO algorithm in (Xie et al., 2024) can fulfill the requirements Def. D.2.
Proof. We start by generalizing Eq.(35) in the proof of Theorem 3.1 in (Xie al., 2024) to all E [T. Note that we consider bandit setting R in (Xie al., 2024) collapse with R. Suppose at the end of iteration i, XPO generated sequence of policies I1 .... T², we have:
Jo( - [T(
1=1
18

---

Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective
6lt
SECRLHF (II, i, 3, rer)
1 (0
+
R²
Blog x [a|s)]
t
2tyt
2 t
7
[3log
<'(lss) r" () - 3 log . (ā
+ (s, a) }
+ 1)E
2/
Trat Trat (a'18)
Trar (a)s)
where we choose ((1) in (Xie et al., 2024) be Ther and denote mix 1-1 Ej=1
Note that Lemma C.5 in (Xie et al., 2024) holds w.p. 1 for all & E [7]. Following their proofs till Eq.(43) in (Xie et al., 2024), we can show that for any [T]
Jo( - J3(T') & O(Re2 SECRLHF t, B, Tras)
I 8 )
log
t
Based on the arguments in (Xie et al., 2024), SECRLHE Rzee) can be controlled by co Cov log log ([[[t) for some absolute constant Co, > 0. Here Cov is the Loo coverability coefficient (Def. D.1) and plays the role. Therefore, we finish the verification.
E. Proofs for Results in Section 3
E.1. Proof for Lemma 3.1
We first introduce some useful results from (Sason & Verdú, 2016). Given two probability distribution P, Q E A(A), we use oo (PI to denote the Renyi divergence order a = +oc. We follow the definition of x -ddieergee in (Sason Verdú, 2016) as follows:
P(x) ((x))
(PI(() Egnpl
-
Lemma E.1 (Theorem 7 in (Sason & Verdú, 2016)). Given P,Q E A(A), such that P & Q and P(a).Q(o) > for all E A, we have:
KL(PIIQ) I (i (e(ee:.( KL(QIIP),
where K1 (0,1)U(1,++))) > (0, +00), H1 (t) = (t-1) (t-1))
t log 1lggt(1-t))
Lemma E.2 (Eq. 182; Theorem in (Sason & Verdú, 2016) for 2). Under the same condition as Lem, E.
Ki (e (PIIQ) KL(PIQQ
(P(12))
where (t) (l
[1-1)3
Lemma E.3. For any policy n.
Ja(T)-) J&(T) = =BEa-p KL (-)))|]=;= (-1s))
Proof. A shorter proof can be done by directly assigning in Lemma 3.1 of (Xie et al., 2024), and here provide another one without detouring through it.
Jz(m -3(
HE B.ppparr: flog TC. (als)
BE ant blog <aass) -o (a|s)
=E -Ea-pann, 1r (s, a)] Exppaan (s,
Tra
=BEarparat Hog -e (als)
Tror (a|s) 3E Eearpan [log मí. (als)
3E Hog fafs) (a(s)
3L 3B P 1log π a 3)
Trar (a|s)
Tror (als)
Trat (a|s)
19

---

Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective
πT T(ass) -.. (a|s)
=pBgrpp log
-
where the second equality holds because for any 8,
r't a*(s,a)= = 8 Ty. (als) {z + Z(s)
for some (s)) independent w.r.t. a. Lemma 3.1. Under Assump. and assume rd E [0, for all w [W], then, for any policy a E conv(II)
f w
Jwmi
Cov =1+ 11+Ki x(e 2) Ja( J3(
(3)
8
(x-1)² where k(x) = 2-1-1ogz
((r).
We prove a stronger result in Lem. E.4 below, where we consider the policy class including all the policy having bounded ratio with Tret.
log π (als) R 3
g = S max
S
s,a
TTer(a))) (a|s)
Lem. 3.1 then holds directly as a corollary by combining with Lem. H.1, Lem. B.1 and the fact that r E [0, R] for all w E [W]. Lemma E.4. For any policy π E II gg.
Co 51 1f =1+ 28 Ja( - JJ(T)
(13)
where k(z) (x-1)² 2-1-log2 Ozr).
Proof. Given any sg. consider fixed 0, and apply Lem. E. and Lem. E.2 with P (-1s) and
Q = x--ss). Since those two lemmas holds when P # Q we first check the case when m, (-1s) # x-1ss):
Eawn, (a/s) T.. (als) T(ass)
x (T (11))l=((1s))
(L(T
m2(()
k2(C) Ki (() (L(T((1s)]
(C 1)2
(L(T(11)1
- 1-1- log <
where use = e (11-(())) > 1 as short note.
define <(x) -- ln
(a- 1)²
Note that,
2(x
- 1)2(1 - x x-1)
xx(x)
I - log x
(x - 1 - 1ogx)2
T
2(x - 1) 2 2log x
(x - 1)(1 x 11)
T. 1
log x
1 log
1
T. -
2 20og2
T log T
T. log x
20

---

Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective
Now, we consider g(a) = x= 2 looga for (1, +00). Note that, g(1) 0 and
2
g (x)
x² x
Therefore, n' x((x:) 2 O, which implies x<x) is increasing for all > 1.
Under Assump. A,
( (15)]((((1)) log exp( (r-(s)
2h 3
S (aass)
which implies Ç S e Therefore,
TØ- (als) Sx(e #) KL( (-1s).
Ea (als).
T(ass)
Note that the above inequality also holds when (-11s) -.. (-1s). Therefore, combining with Lem. E.3, we have:
(a|s) T.- (als)
Co =E =BsppanTT
f x<ee 2n (LL (+s))
T(ass)
=1 k(e J2 ) (T(T)
E.2. Another Bound for Policy Coverage Coefficient
In the following, we provide another bound for the coverage coefficient between the optimal policies induced by different reward models. Although do not use this lemma in the proofs for other results in this paper, it indicates different upper bound, and possibly, it is tighter than the one in Lem. 3.1 in some cases.
Lemma E.5. Under Assump. A, given any bounded reward model r, and the associated optimal policy n.z (defined by Eq. (L)), the coverage coefficient between n. and nेद can be controlled by:
Cov" < min Eaup [a [exp( Ir (s, a) r(s. a) - 61 I11
DER
Proof. By definition, the state-wise coverage coefficient
T (als)
10 : Ea (11)
(a|s)
r*t (s, r(3,a))) 2,(s)
E Rawn (11)) (exp(
)]
(s)
Here we denote (s) = E. (als) exp( ( r(s, a)) and similar for Z,- (s). Therefore,
(x) Z,- (8)
(als) (xpe) a))
Σn ()) expl ( -
( (s, = Banny (exp( r(s, - ()) )}
(s)
3
We remark that one important fact we leverage in the second equality is that कテ. (als) for all A. Considering introducing arbitrary b E R, we should have:
1-. =Eawag (1)) exxp( 1" (s, a) r(s, a) + +b)] Eonzz (xp) r(s,a) a) -(ss, a) -b
3
I (s, r(8, a) + +./
<E² <R (11)) eepp
Given that is arbirtary, we can pick the best one:
B2- [exp( 1r (r*(s) a) r(s, a) bb }.
Im min B22 beR
3
21

---

Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective
E.3. Proof for Theorem 3.2
Theorem 3.2. Under Assump. A, w.p. 1, 6, given an online dataset DD generated by a policy series {T²)(=1 =t=1 conv(ll). running RPO with coI((II) and R D yields distilled policy Tpat- such that,
Jz( - JB(TDal) $
(4)
o ( (1 (1 K(e 2
( J
7
3T
We refer to Lem. C.2 for the detailed hyperparameter setups,
Proof. By Lem. C.2, w.p. 8,
Ja(mt.) JI(ADa)) Curre Cov lx
log
11
where =I' = 1 Eneryy x' is the uniform mixture policy, and the coverage coefficient can be upper bounded by:
Jo( -
Cov - 1i SI + x)
(Lem. 3.1)
B
T
JB( JI(
$ + k(e 28
3T
Here in the last step, we use the fact that KI. divergence is convex, and therefore,
Mxxx S +E= [[(r")(Trer),
which implies ( 2 Σ" JT(T)).
Implication for Online RLHF have the following corollary.
If we consider the policy sequence generated by a no-regret online learning algorithm, we
Corollary E.6. Under Assump. A, suppose x²,..., x² generated by a no-regret online learning algorithm
with Σ JT( JØ(T²) O(C(IIVVT) for some structural complexity measure C(II), as long as T ((8--C((I C(II) 2x? (e))) )). running RPO yields offtine policy s.t. J O(
The proof is straightforward by noting that Ei= Je(a(-))Ja(aa) ((C(TT)TTTT) which decays to 0 as T increases.
3T
Coro. E,6 is remarkable as it implies an ( sample complexity bound to learn an E-optimal policy for online RLHF (for smaller than threshold), which does not depend on the number of states and actions or other complexity measures. In contrast, previous online RLHF literature (Cen et al., 2024; Xie et al., 2024; Xiong et al., 2024; Zhang et al., 2024), for the uniform mixture policy T := ΣT n², the regret-to-Par tP- conversion implics a value gap ( Ja( =
Ö( c(1) which has additional factor C(T) regarding the complexity of the function class. This suggests strict improvement.
Moreover, this marks fundamental difference from the pure reward maximization setting, where lower bounds depending on those factors has been established (Auer et al., 2002; Dani et al., 2008).
Other Previous Works Reporting Faster Convergence Rate Several recent works also report faster convergence rate lower bounds for online pure reward maximization RL, by exploiting the structure induced
than the information-theoure by KL. regularization. (Shi et al., 2024) investigates the tabular softmax parametrization setting and establishes quadratic convergence results. In contrast, our result is more general, applying to arbitrary policy class.
The work of (Zhao et al., 2024) is more related ours. They consider general reward function classes and derive an PolyPD)) sample complexity bound, where D is coefficient related the coverage of the distribution Wreé-
"See Cond. C.1 for the definition data generation process.
22

---

Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective
While their dependence is better than ours, their definition of D is not always satisfactory. For example, in the worst case would have D 9 min,es p(z))- )- This indicates that their bound scales with the number of states, once noticing that
min n(a) is no smaller than |S|. In contrast, the largest coverage-related coefficient in our result is O(x² 2n )) = O an which remains small and is free of [S]. Therefore, our Coro. E.6 can outperform the bound in (Zhao et al., 2024) in many
)
scenarios.
More importantly, the primary focus of our work is on reward transfer, which is orthogonal to these studies.
F. Proofs for the Main Algorithm and Results in Sec. 4
F.1. Additional Algorithm Details
Missing Details for TPO (Alg. 1) For any given (k. n) E [K] x [N],
use
D
Urckori=k.jen (st), abd ano yhd, xid} to denote all the collected data up to step (k.,)) UickiSaN or iSiANN i ars and , denotes the data collected by Algou up to step (k, n).
D
F22 Some Useful Lemmas
Lemma F.1 (MLE Reward Estimation Error). In each call of Alg. 2 with policy class II satisfyyng Assump. and a dataset D generated by sequence of policies ne ,e,,..,, IPI, then, for any policy n, given any E (0,1), with probability at least
1 - 8, for all w E [W1, we have:
(Epaalr*] E Ep. (Ep.. [Fwall - E. [HES))
Cov
16E2R
log
where T
D EiSiD/ X² short note.
Proof. For any policy II, by applying Lem. H.3 with x and +- PALE. we have:
R 1pp [r-]
E P. (r*1) - Cep.-(Tsnas Eppt,er (FOLE))
< Baoppent ("( - (s,)a) (fxnr(s, ()) a))1
1
28V262R
Cov nD
E Exppa (1s)) [H (1|s, a, a. (-1s, a, a))]
D
By applying Lem. H.2, and the fact that FRLE, r" Rl for any 5 (0, w.p. 1 - 8, we have:
1
E
(P (+s, a)la (-Is, a))
D
2
< (f Lr(r") )+++
+
log
D
2
11
_
log
(Assump. A and PriE minimizes the negative log-likelihood/
D
6
Therefore, we finish the proof.
Lemma 4.2 (Value Est Error for f hwe(w)) Under Assump. A and Def. D.2, w.p. 1- - 6, in each call of Alg. 2:
Vw E [W]. Jo( Jg(TTer)) S V (TT D)
22R
{JO( -g((Tee)) Ō(
/ VNww; D)
23

---

Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective
Proof. Note that Cav Pai [D]
(( where recall that Nw;; D) Eig I - denotes the number of
occurrences of Tp in the dataset. By Lem. F.I, w.p. 1 8', for all w E [W]. and any (k: n) E [K] x [M] occurs in the call of Alg. such that n > aN:
( (Eaozi [r*] - Beporee (r7)) p:ppi [FiLe]] Epp (Fouz))
$1662R
W Nww Di.,)) log 8
Recall
I[TT
( := :Bpp [FLE] Ep,t [Pnee] (KL [IzEe) + 16c2R
log
N (ww;
(w; DR..)
By taking the union bound for all T iterations (choosing 8' = 6/TT), we finish the proof.
Lemma F.2 (Estimation Error for Self-Transfe Policy). For any k >> 1 and aN < S N, in each call of Alg. 2 in the iteration (k, n) of Alg. with a dataset D == {(s', a a", y., n Higho) satisfying Cond. C.I, then given any E (0, 1), w.p.
1 8:
(IDna D) <Ja(F (T(()
Ja(TTer) - Re.n. (coutiibe A vC(L)
[I[T
(iTna D) (TT(TT
log Ro
D
5
where we use Amix : D
Σ D
IPI as short note, and el some absolute constant.
Proof. Recall that
(Tiaa D) ==E
p. p.anal [Paul Ep. [Paal] ( (ree)
LD (Fia))
LD - bonus,
17
"
[[[ Here we use bonus == 2c . e2R, / [D) log a
as a short note of the bonus term. By definition,
(TTDaa: D)
E E. [r'] E.tteer [r*] (KLL Mrat) + LD((r") 1 Lo LDD (Frux) bonus
17 (Pessimistic estimation of fast in Eq. (12))
2
II
<Jo(FDaa) (T(() + D)
log
bonus
(Lem, H.2)
<Ja(FDa)) - Ja(TroF) + 2c. 22
log
bonus.
(14)
D
5
The last step because of our choice of (1 -2
24 log
For the lower bound, note that for any policy ı E conv(II), we have:
JO(TT) - Jo(TTre)) (TTDat D)
1ppxx("' ".prreee (KL(T||TZeE)
Ep.amal DDsul - E Epptrer |Daul BKL Tres) + LD (Foul))))
+ LD (fuL)) + bonus 17
1
< <(pp "Bptteel (r*] (TI(TT) min rERi (Bn.(r) - By. [r] (T(( + ID(r))
(Optimality of TDai in RPO:)
24

---

Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective
1 LD("") bonus
(fuE minimizes Lp)
(s, - (x, + (x, - LD(rr)) LD (rx:)) + bonus
S [1,"
" (We use TnpD to denote the reward achieves the above minimum)
2
log
11
8/2222
Cov
E pe
[ (PP (1)
1)
8
D
Pappor aranet
(1),,,,))) (-1,, bonus
(Lem. H.2 and Lem. H.3)
II
4R Cov
<
log
8
+ 64ne
D
bonus
(ar bz S o²
4b
I1TT
<4c2 22 Cov a
log
+ bonus.
(15)
D
n| where the last step is because of our choice of 1 c. (1 ( 24 log 8
Next, we evaluate some choice æ. We first consider π = म.. E conv(ll), the above result implies,
JD( Ja(TToe)) (TTDai D) S 4cz 22R Cov ii V D
log
IT bonus.
(16)
8
Secondly, we consider the mixture policy mix == ब(K- Zi=1 k-1 T divergence, (T) is concave in æ, by Jensen's inequality, we have:
cInv(II). Because of the convexity of KL
Jz( Jp ( (T) E Eenpaaanll (-16) (s, (s, 3KL ( ((
<Ja(TT))
1
Σ
(eepp (8, (KI((T)) [
alk: I)N
15JSaN
C(II)
|||
2C(II) akN
<Cor. RE
eo
S Cou Re
Re
log
(Cond. D.2)
a(k 1)N
Note that 12/ 2 a(k- N) which implies Cov Tii
a(k-1JA 1JN S
KN
Therefore, by Eq. (15),
1IN
e 28 2
11TT
Jo ( (T(o)))) (TTDa D) S 402
log
D
8
Combining the above two inequalities together, we have:
Ja(TT²)) - Jæ( - V (TDal; D) k-1 1) + Jo (x
<J - (T(
k-1
(o(Tee))= V (TDal: D)
2IIII)
I
2
Ro
log 0u
+ 40g
2R
log
+ bonus
akN
6
a
D
C(II)
II T 8
ScgR. 2R
log
+ bonus.
(17)
a D
Therefore, under our choice bonus 2c
log
Eq. (14), Eq. (16) and Eq. (17) imply,
D
(TTDal D) SJa(NDa))
J (TToF)
25

---

Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective
(Cov A v
(TDaa D) 2Ja(m (T(()) è' Re
log
V
[
6
Lemma 4.3 (Value Est Error for TDst). Under Assump. and Def. D.2, w.p. in each call of Alg. 2:
Ja( Jg( Ō R
vC(T))
(Cov lo
VPDI
a
(TDai D) S Jg( (r(T))
(6)
Proof. By applying Lem. F.2 with appropriate constants, and taking the union bound over all iterations, we can finish the proof.
F33. Proof for Thm. 4.4
Theorem 4.4 (Total Regret), Suppose go is no-regret instance satisfying Def. D.2, whose regret grows as ORR22R ( /C(LI)T) for any intermediate step T and some policy class complexity measure C(II). Then, w.p. 1 - 28, for any T/K K t < T, rumning TPO yields regret bound:
Σ JOTT ) = Regot + Reg TSA
(t)
Regor (1) = (Re (
(7)
Reg =
Σ
Ami A b(r),((r)
(8)
TStaaN
eAR
+ 28
v(1 (1-Aa)
Σ
A(w)
Here we denote k(T) = - and TT(T) := mod N to be the block index and inner iteration index for step t; e M()),((F) := Ö(Re Cov 1s A /C(TT)
/ where T = 1 *isn+*(0)nn *(0,,((() is the mixture policy up to т; A(w) and Amin
denote value gaps as defined Sec. 2.2.
Throught the proof, follow the convention that 1/0 +00.
Proof. Since we divide the total budget T to K batches with batch size N, we will use two indices K E [K] and Ñ e [N] to represent the current iteration number, i.e. the Ñ-th iteration in the K-th batch. We will divide the indices of previous iterations to two parts, depending on whether we conduct normal online learning (the first aN samples in each batch) or do transfer learning (the rest (1 a samples in each batch):
T R = [(k, n) < K,n S aN, or k k K ANN],
IIrf [(k, K, <n n or aN <nS KNN
IR I - [(k, n) K, N, or k K, n N}.
For the policies generated by online algorithm, under the condition Def. D.2, w.p. 8, for any K E [K], N E [N] we have:
Σ
Jn( Jo( z 2R V C(UI)I log
(18)
K,N
8
(Z R.N
Next, we focus on the performance of transfer policies. We first introduce a few notation for convenience.
26

---

Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective
Additional Notations We use Dal denote the offline policy computed by Alg. called by Alg. iteration (k, for some a:N < n S N. We denote ek. == Dai = > } to be the event that Alg. returns knn "Dit as the policy, and use E = (,,- = to denote the event that Alg. 2 pick and return TT. Besides, use - U [we - kn short note for the event that Alg. does not return the offline policy "Daii Recall the definition A(w) -
R.n
and Amin = [[W] (w). We will use w" to denote the index of the task achieves Amin (or any of the tasks if multiple maximizers exist). Given the dataset Di we use " k D
EijjDD... T be the uniform mixture policy from
mix
D
Then, we decompose the accumulative value gap depending whether Dall true not. We use [[E] as the indicator function, which takes value if happens and otherwise O. For any K [K], N [N], we have:
ckin
Σ
Ja(m - (T(
(k.n)EIte N
Σ
TE (Ja( Jz(nk())
Σ
(19)
(k.))EZIE K,N
(k K,N
Part-(1) Upper Bound the First Part in Eq. (19) We first bound the accumulative error when T[ 1. On the good
events in Lem. 4.2 and Lem. 4.3 (which holds w.p. 8), I[ = implics
V (DDa D&SR)
max ( Dk. WE[W]
max Jn(T,=)) -p((T Jz( (I(T=or) -min: wE[W]
and as implied by Lem. 4.3
Ja(m²²) Je( SAmins
Ja( - Jo(TDal) Re (cov VC(TI)
|||
log co
D
8
Combining all the results above, we conclude that
Jo( - (TDD <Amin A 02 RE 28 (Cov*tilatun A Ve(li)
log
DDim
5
-Amin A k.a
Here for simplicity, we use
kn = = Rn (CouTiulete" ^
C(IT)
[
log co
V
D
8
as short note, indexed by k, n. Therefore,
Σ
(J - -J((Thi))
Σ
T
( (Amin A A.)).
(20)
Its K.N
(k,n)eItsd
Part-(2) Upper Bound the Second Part in Eq. (19) Next, we bound the accumulative error when II & 1. Note that,
Σ
I
J=(a)."))) -
Σ
Σ
I| [(ww)
(k.))ezus
(R.))EIIEC K.N
K.
27

---

Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective
Here only focus those source tasks with A(w) 0, since transferring from Th with A(w) does not incur regret. We separate source tasks into
22 (w E [W]] 2mmin) ) and W>2Amin {wo E [W]]A(w) > 2A m. For w E W22dmm. on the same good events in Lem, 4.2 and Lem. 4.3, [[[ew"] 1 implies
min =]( Jz(TT J (x(e))))
2Ja(TT--) - (r(Trer) - kk (T, "AA""(" ; 1)
2J - Jo((Te)) - RR (Trm:: D
1
II WT
2JO(TT.)) - (D(T) - ( ) + Ja((Tet)) 32 e2.e.n. 2n
log
(w; Dk.n)
6
I|T
=Aww) 32 e2R
(w:; DE.)) log
In the following, we use
32 as short note, then the above implies
CeEAR
IIT
c A(w)²
Nww Dk.n)
(Ww - (A(w) Amin)
log
log
6
and therefore,
16c7e
Ve E W22Amm
Σ
I[ A(w) 1
log
8
(k.n) R.N
For w E W52dmun we introduce a new event En.k 2r 2 Amins we automatically have:
= f2 < A min Note that, when I| Amin = 0, i.e.
2i<AAmin
Yw E We2dmin I[Em"1A(u) s 21 Amia I 41[E (Amin A m.)).
(21)
On the other hand, on the good events of Lem. 4.2 and Lem. 4.3, when T[g n en.k
&2icAmml = 1, we must have:
Ja(m (T( - k. Si (mban Dk.m-1)
(Lem. 4.2 and Lem. 4.3)
< Dkin-l)
(w is chosen)
IWT
<J Jo( 32.
log
V
N(w; Dk,n)
which implies,
c7eAR
WW
e 4R
WIT
N(w; D&v))= 5
log
log
(Amin
A 2
min
Therefore,
8c7E4R
4R
Yw E Wg2dmmn
Σ (k.)) KN
I[g n cn.k
[A()) -
log
A
w
8
Combining with Eq. (21), we have:
8c7EAR
N
Ye Wezdmm
Σ
1180
44
Σ
1em (Amin mn)
log
A (w)
(ken)eZ K
RNN
By merging the analysis for w Wg2am... and W>22mm. we have:
16c7e
Vw E [W].
Σ
[(w)
<
log
4
Σ
(18n*JAnm A m.
(22)
A ww)
(k,))
(k,n)
R.
R.
28

---

Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective
Note that for those (ww) & 4cje2max
log
we automatically have
R.8
[I
Σ
I[ [(ww) S4c Eamos
log
I
5
(k,n)EZ N.N
R,N I[E log
WT
=Aq 22max emax
Σ (k.))EI R.N
4c e On the other hand, when A(w)
log
ITT the bound in Eq. (22) is tighter, since
16 AR
11
WT
log
I emmx 2max
Σ (K.N)E
New
log
A w))
8
5
Combining the above discussions,
Σ (k.
I|- - -z(
E
E
(k,n) K.N
WE(W
K.N
A(w)>0
min( 16cfe A(w)
II
IT
Σ
log
4c 2max
Σ
Iew
log
[W[
(Z
R
+4
Σ
1 Amin n.
(k, N.N
160f 4R Aww)
IITT
I1[T
< <minf
log
Σ
4e E 2max
Σ
I[W log
6
W][
[W]
(KI))EZEE KN
A(w)20
A(m)20
(min(a,b) + min (x. v) I minfa + + + y))
+4 Σ
1 Amin n.k
(k.,) Ez
K.N
160f 4n
WT
WT
< minf
log
4c 2
TIee K.N
log
A Aww)
WE[W]
A(w)>0
(Cauchy-Schwarz inequality and Σ wc(W]
[[
<
Tize
K
A(w]20
+4 Σ
1 Amin A m.
(23)
K,
Merge Everything Together
Combining Eq. (20) and Eq. (23), we have:
Σ
Jo( - (i))
(k,n)
R
Σ (km)
(Jo((())TTTT))) (o( +
Σ
II-
))())))))))))
(k.))
16cfAAR Aww)
WW
WT
Sminf Σ
log
4c
2max
WIZT log K.N
WE[W]
A(w]>0
29

---

Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective
Σ
Aii A p.
(k.))EZ R
Combining the value gap for the online parts, we have:
Σ
Ja(
(K
IaT
Co,,
Re2R
((1))T2 K,N
log
+ 4
Σ
A
Amin A m.k
16c7e uR w
[I[T
[1T
min
log
4o
2mas
I log K,N
we [W]
A(w)20
Note that for K > 1, denote & = (K- 1)N + N, we have:
S aKN S 20 (1- s 2(1 a)) R.N
Therefore.
Σ
Jz(TT-)) (
(K,
eAR
-ō(
Σ
Amin mik + RR v VaC(I)t + Sana ((1- a(-
A(w)
(k:))EIS R
wE(W] A(w)>U
where in the last step, we omit the constant and logarithmic terms. By replacing 'R + 1. E(K f Erso
k t- k(T) = [ and + n²() == T%N, we finish the proof.
Remark F.3. Under the choices of Algoz XPO (Xie al., 2024) and e $. according Eq. (9), TPO achieves W22 On the other hand, by Lem. 3.1,
Õ( VT) regret T
A
Reger + Regrrt) = + Ō(K(e #
C(I))
Cov Iiie
a V
Bt
Therefore, consider
((=²t ie em where é(.) hides at most poly-log of T. At least when T To. Eq. (10) implies
a²
(t) Reg = 1 + O(1) < C' for some constant C. Since a = e ", we also have Regar (T) = Ö(VT) for any T > 0. As a result, the total regret of TPO grows as Ö(VT) long T >>>>>>>> To.
a
G. Connection between Win Rate and Policy Coverage Coefficient
Lemma G.1. Given two probability vector u, v ((A) and reward function A -> TR, consider a preference model based on r, satisfying,
1 2'
P, = 1la,a') 1,(y=1la,a')
for any a, a' E satisfying r(a) 2 r(a'). Then,
Σ /u(()a(a)) s min 3>0
1 v (7 + 2V > u))
7
where (u 0) := E Pr = 1/a,a'))
30

---

Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective
Proof. We first of all sort the action space A Assered {a1, (2...., a A according reward function r, such that, for any Si<j [A, r(a:) I r(az). Besides, (.). to denote the cumulative distribution function regarding
S sis AA), F"((;)= = Σ u(a;);
==1
and Fd is defined similarly. Then we have:
u(a;)
u(aav(a))
uula))((a))
v F" ('((a))v(a)
(Introducting a parameter y > 0)
7+ Fu(a)
AEA
A
u(a;)
Σ + F*()))ua)
(Cauchy
(Cauchy-Schwarz Schwarz inequality)
F'((aa)
u(as) (a;)
V 21, u),
[AA 1==1 v(a;) = and
where in the last step, we use the fact that 7
| (u > u) -' 1|a a')l Σ v(a:) Σ (y laa; 2
Σ
v(a.) (".s).
j=1
For the first part, we can upper bound by the following:
A
u(a:)
LAI
u(as)
++ شز u(a,)
Σ 1
7 + F
(a;)
Σj=1 u(a,)
7+ + u(a,)
7 + - u(a,)
log
(1 (1- S log 4
7 + ンドー u(a))
1=1
1 1+7
< log
Therefore, EaEA /u(a)u(a) S v(+ (F > u)) log and finish the proof.
14
Since y is arbitrary, we can take the minimum over y > 0,
Lemma G.2. For any policy n, म.
(7+ + 2P, (x(11s) > #(+ls))) log + 7 7
TV(((1)) & min 720
V
1+?
- E.- TV ((-)
min 7>0
min
(n + 2P, ( #)) log
?
Proof.
1 - (V(((1))) <1 - - (T(((1)) Σ Vi(als)((als)
/ ( 2P, (T(-1s) + x(-1s)) log 1+7
< min 750
where in the last step we apply Lem. G.1 with (-|s) as v(-). #(-1s) as u(.), r" (s. as the reward function. Then, we finish the proof for the first inequality.
31

---

Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective
For the second inequality, by taking the expectation over p and the concavity of v function, we have:
720 / (1+ 2P, (T > ④) log +
By (T & min
y
By choosing = Pr. (I R)) (note that this choice ensures Y > 0 since ((--.) > 0), we finish the proof.
Lemma G.3. [The Complete Version of Lem. 5.1] Given any policy n, under the assumption that P.- 11s, for any o' E A satisfying r (s,a) 2 r" (s, a') we have;
1+7
-
Cova.-1
max 7>0,8
V 7 + 2F, +2, (1
TT) log
(24)
7
23
1+7
-J((T)
-1
Covt.-17
max 7300.t
VV 7 PP 3) log
+
(25)
23
where it is an arbitrary intermediate policy, (T x R) == Benparns (Pr = 118, and (* m.
1 - Pp (TT > F) = "o 1|5, 0, d')].
Proof. We have:
Ty- (als)
Eawn, (1a)]
- =x ( (-1))[[(((ss))
[(ass)
(T. (-18)1
(Theorem in (Gibbs & Su, 2002))
2 (1 1.
(Bretagnolle- inequality)
Now, we introduce an arbitrary intermediate policy म.
(V (TV((()))
TV(T( (1s))
(Reverse triangle inequality)
(TV( 1 (1
(Pinsker's inequality)
Applying Lem. G.2 with (T, < (n, .). we have:
E Bans (11)) Ty- (a|s)
T(als)
(V(((())) + (T)))
(7 + (f(1s)) > π ((1ss)) log 1
+
()) -,- (-;-ss)))
By taking the expectation over ~ p. and leveraging the convexity of 1/2 and the concavity of V functions, we have:
Egrppan (:le) - (a|s)
Taass)
E Eapply / (1 2P, (T((ss) > T(- log 1 + Ea / [KL(
(+s)
1
V + 2P i > T)) log 1t7 7 (
+ Ja(n;- J=J
29
Note that the above results hold for any 7 > 0 and any म. we finish the proof by taking the maximum over them.
The second inequality in Lem. G.3 can be proved similarly by applying Lem. G.2 with (m, 7) (T,, Ø). All the discussion the same.
Remark G.4. We provide some remarks about Lem. G.3.
32

---

Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective
Lem. 5.1 is direct corollary of Eq. (25).
Notably, Eq. (24) has a different implication compared with Eq. (25) that we follow the algorithm design in Sec. 5. More coneretely, Eq. (24) suggests we should also disregard those source policies that strongly dominate I when close to ,..-. This makes sense because those source policies may achieve high rewards or win rates by incurring high KL divergence with Tref, and therefore, they may not provide good coverage for Tr
However, in Alg. 3, we intentionally do not filter out but instead prioritize source policies with exceptionally high win rates. Because they likely provide good coverage for high-reward regions and can be advantageous for practical LLM training. Nonetheless, for completeness, we bring this eeeory-practice gap into attention.
H. Useful Lemmas
Lemma H.1 (Convex Hull Fulfills Assump. A). Given satisfying Assump. A, conv(II) also satisfies Assump. A.
Proof. The realizability condition is obviously. We verify Assump, Note that for any a E conv(IT), there exists and π = Ei= A' which implies,
X , A" 2 0 and n' .... x" 11, s.t. Ei= X'
Talss) Frof (a)s)
(als) Tro- (a|s)
R z 2 exp(-
<(ass) (a(s)
x (als) Trof (a)s) S exp( z).
V8, a
Therefore, (IIv fulfills Assump. A-(II,, which finishes the proof.
Lemma H.2 (MLE Guarantees; Adapated from Lemma C.6 in (Xie et al., 2024)). Consider a policy class 11 satisfying Assump. A, and recall the reward class Rd converted by Eq. (11). Given a dataset D = {(s', a',"",",,u,, ligi satisfying ",,'
Cond. C.I and any 8 E (0, 1), w.p.
1
2 [
[I
Vr E R
E (onpperr((())) (1 (1s,,, s Lp(r) - LD(r") + log
D
EIDI
The proof is almost identical to Lemma C.6 (Xie et al., 2024), except we replace Fx in their paper by Pr.. So we omit it here. Besides, note that our NLL loss normalized, while (Xie al., 2024) consider unnormalized version. This results in the additional [D] factor here.
Proof.
Lemma H.3 (From reward error to Hellinger Distance). Given any policy n, any reward function r with bounded value range | R. R), and another arbitrary ãे with positive support on S A, we have:
Ex-par (ss) (1 -r(.,)))rss.)) - - (r(s,a)-r(s, - -(s,,))
28 2R
COVE Esupantl a).a *) H (P a)IP, ( a))]
Proof. For any reward function r, we have:
[ [rp E Ep (r*))
[ppnIr] E Ep [ri)
E
r- (s, (s, (r(s,a)- r(s, r(s,
<4c Benp
(s, rs (r((,))) -r(s,))
(Lem. H.4 with C' -2R)
T (als)
2n
Σ
Vp(s) (āss
(a]s)
8,0,a
p(s) ( ((als) (r((s,a)))))) (ss0a) ((((s.a)-r(8.))) - (((,)))r(s,,))
(a(s)
24
Σ
p(s)TTor (a]s)
7(a|s)
3.a.ā
33

---

Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective
V Pee (M()))))(())))))))))))))) of a -(),
(Cauchy- Schharz inequality)
=422 V Cov V (1)) -rs,a) - - (r((s,))
r(s,a))
(26)
Note that,
(00-po0*(11)..00 (r)(s,)) - -r((.)))
<8E (0ppa Vo(r"(s, - (s, - r(s,
r(o, [)] ((x y)² S 1(x + vV vD)
(Bppp.a07(()))) HH (F, (-18, a, P- (1)))
(27)
By plugging into Eq. (26), we finish the proof.
Lemma H.4 (Sigmoid Function). Given T, y E C) some 0,
I - y S 4 4xxp(C))o(z)) o(y)).
Proof. Without loss of generality, we assume x y. Because a(-) monotonically increasing function and it is
continuous, we know there exists [y, 2] s.t.
a (xx - (v)
= 0' =
y
Because x, y/ E [- C,, we have o (-C) z (2) S oCC). Note that the axis of symmetry of function f(a) - a(1 a) 1 1/2 and -C) a(-O) 2. Therefore,
x yl o'(z) - o(v)l < (1 exp(C)) + exp(-O)) exxp)))))))(())) -((()) -o())
<2 + (xp(C))(())) ovv)l 4exp(C))o(()) - a(v)
I. Experiment Details and Additional Results L.. Details in Experiment Setup
Setup of y" Due to the high cost of collecting real human feedback, Llama3-8B (Dubey et al., 2024) to simulate the ground-truth human annotations. More concretely, adopt 1 (Dong et al., 2024) as the true reward model r", which is distilled from meta- MMetaMMeeeeeeeee ama- 3- 8B Instruct. This reward model can be queried with prompt-response pairs and returns reward scores for each of them.
preferences generated by
Best-of-N as an Approximation m... We recall that we consider source tasks for transfer learning, including, (a) ROUGE-Lsum score (Lin, 2004), (b) BERTScore (Zhang et al., 2019), (c) T5-base (250M) Thases (d) T5-large (770M) Taarge
To reduce computational complexity, instead of explicitly training the optimal policies { )ee[W associated with each source reward model, use Best-of-N (BoN) approach, a.k.a. rejection sampling, to approximate the responses generated by T. Specifically, we generate N² responses by the online learning policy m in Alg. 3, rank them according to the reward model, and select the top-ranked ones.
Furthermore, even for source LLM policies (3) and (4), we find that transferring from BoN-selected responses generated by S.. actually outperforms directly using the responses generated by T5-base/large. We hypothesize that is because the responses by T5- T5-base/large usually have quite low probability of being generated by the online learning policy, leading to
To distinguish with N used to denote the block size Alg. 3, we use to denote the size of BoN
34

---

Can RLHF be More Efficient with Imperfect Reward Modcls? A Policy Coverage Perspective
distribution shift that complicates learning. In contrast, BoN-selected responses maintain non-trivial probability of being sampled, without significant distributional mismatch.
Next, we elaborate the BoN process with more details. In our experiment, we choose N = 32. For source reward models between the generated response
(I) and (2), we generate N responses, and we compute the and the human-provided summary in XSum dataset as the reward value. For source policies (3) and (4), motivated by the closed-form solution in Eq. (2), given any prompt 8 and response a, we infer the log-probability of T5-base/large predicting a given as the reward score, i.e. log Tals)(a)s) or log (Targeaass) In another word, we interpret rS-base/large as the optimal policies fine-tuned from uniform distribution align with some reward models, which we treated as source rewards for transfer learning.
Training Details Our training setup is based on and adapted from (Xie et al., 2024; Xiong et al., 2024). We run for 3 iterations (K = 3), and in each iteration, we sample training dataset of size 10k (i.e., the block size N =10k). For each prompt, we collect 8 responses as follows. Firstly, we generate N + 4 responses by the online learning policy To.. The initial N responses are used for Best-of-N (BoN) selection, where we choose a source reward model via the UCB strategy in Alg. 3, and then pick the top4 from N responses with the highest source rewards. These 4 responses are merged with the remaining 4 responses and we get a total of 8 responses. After that, we query r" to label the reward for those responses, and record the ones with the highest and lowest rewards to serve as positive and negative samples for DPO training.
In contrast to the procedure presented in Alg. 3, we utilize 8 responses for each prompt. Therefore, the win rates are computed in relatively different way. Specially, we set ft = 1 if the response achieving the highest reward comes from the 4 responses selected by the BoN step, and yh. = 0 otherwise. The updates of the win rates estimation and the
computation of UCB bonus terms align with Alg. 3, except that we set WR TeL 0.55 instead of 0.5. This adjustment establishes higher threshold for enabling transfer learning, requiring source tasks to outperform the baseline policy Th. by a larger margin before being considered. We believe it enhances overall performance.
Regarding other hyperparameters during the training, the learning rate is 5e-5 with a cosine annealing schedule. Training is conducted on 4 H-100 GPUs with total batch size 64. We set the constant parts in the UCB bonus ev log 1 = 1.0 in practice, considering the value range of win rates is [0, 1].
Evaluation Details During the evaluation phase, we randomly sample 10k prompts from XSum test dataset without repetition. For each prompt, we generate one response for each of the policies being compared, and query their reward values from r" (i.e., the s ffaire ffairrc/ES rXC//cCcFfaaa Laii--LLLAA--RK-----.. frequency of that one policy generates a response with higher rewards than the other across the 10k prompts.
1 reward model). The win rate is then estimated as the
1.2. Additional Experiment Results
Results under Other Choices for Algro in Alg. 3 In the following, we report the results with two alternative instaniations of Algpo: by optimizing the XPO loss (Xie et al., 2024) or the IPO loss (Azar et al., 2024). All the training setups are the same as the experiments where Algeo is DPO, except that we choose a smaller learning rate le-5 for Algpo is IPO.
Remark I.1. Different from DPO and IPO, XPO is an online algorithm itself and in their original design, the pairs of online data are generated by online exploration policy and another fixed base policy, respectively. However, empirically, Xie et al. (2024) follow the iterative-DPO and utilize the same online learning policy to generate pairs of online data. This exactly aligns with the no transfer baseline we compete with-instantiatinn Algro with XPO in Alg. 3 and setting W = O, which we refer as XtP in this paper.
Investigation on Source Task Selection Fig. 2 provides further investigations on the source task selection process. For each iteration A = 1, 2, 3, we count the number of times that ,.. is occupied by different transfer policies { ]we(w] or the online learning policy #be (i.e. without transfer), and provide the results on the top sub-figure in Fig. 2. Besides, in the bottom sub-figure, we report the win rates P,,. (T > x(.) ) for all ( [we[w] U (m J. As illustrated, the UCB sub-routine efficiently explores and identify the source task with the highest win rates against the learning policy =
Notably, as the improvement of n over the three iterations, we can observe the transition from transfer learning by leveraging high-quality source tasks to standard online learning. In other words, our method can automatically switch back to online learning and avoid being restricted by source reward models.
35

---

Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective
Without Transfer
Purely Exploit ROUGE-Lsun
Purely Exploit T5-Large
Without Transfer
Purely Exploit ROUGE-Lsum
Purely Exploit
T5-Large
Iter
52.3 + 1.0
50.4 1.6
49.9 + 0.4
Iter
52.3 + 1.1
53.4
50.2 0.3
Iter
55.2 + 1.4
52.3 0.3
50.1 + 0.3
Iter
51.6 + 1.3
54.7 1.6
49.1 1.3
Iter
55.3 + 1.1
51.8 0.5
50.3 + 0.5
Iter 3
52.2 + 1.6
53.8 + 2.9
49.2 + 1.1
(a) IPO as Algro in Alg.
(b) XPO as Algro in Alg.
Table Similar to Table we report the win rates (%) of the policies trained by empirical TPO (Alg. 3) competed with baselines, presented across 3 columns. Baseline (I): without transfer, iee. iterative-IPO or iterative-XPO, Baseline (II): purely utilizing ROUGE- LSum (the lowest-quality source task) transfer learning. Baseline (IIf): purely utilizing T5-Large (the highest-quality source task) transfer lcarning. Results are averaged with random seeds and 95% confidence levels reported.
Allocation of Transfer Policies
Iterl
Iter2
Iter3
10k
Win Rates
Iterl
Iter2
Iter3
0.7
0.6
0.5
0.3
Figure Deeper investigation on the source reward models selection process. We report the allocation transfer budgets on cach source tasks averaged over trials (top figure) and the win rates P.- (- x(.) (bottom figure) for iterations A 1,2, Due space limit, we use abbreviation rather than the full name of source tasks. R. B. TB, TL and NT stand for ROUGE- BERTScore, T5-Base, T5-Large and No Transfer, respectively.
36


Reviews:
[{"review_id": "Rc3jGipX4M", "reviewer": "Reviewer_WBNK", "summary": "This paper studies RLHF under the contextual bandit setting with KL\nregularization. In usual bandit problems, there is a need to balance exploration\nand exploitation. However, they show that there is a \"blessing of\nregularization\" in which the a policy that has low policy value gap will also be\na good exploration policy. In particular, this means that due to the presence of\nregularization, the two goals of exploration and exploitation are aligned, and\ndo not require additional assumptions that are usually made about the problem to\navoid negative transfer. Using this idea, the authors design the Transfer Policy\nOptimization (TPO) algorithm for this problem, prove a offline policy value gap,\nprove a regret bound, propose an alternative algorithm that is more\ncomputationally efficient, and perform some empirical validation.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback and constructive suggestions! We address your specific comments in the following.\n\n## 1. Claims And Evidence\n\n### 1.1 About Claim in Thm. 3.2\n\nThank you for the suggestion. We will follow it and clarify our claim in our next revision.\n\n> ...why the second phase is also independent of $\\beta$?\n\nAs predicted by the offline learning theory, the offline policy converges to $\\pi^*_{r^*}$ at a rate of $\\tilde{O}(Cov^{\\pi^*_{r^*}|\\pi_{mix}^T} T^{-1/2})$, where $\\pi_{mix}^T := \\frac{1}{T}\\sum_{t=1}^T \\pi^t$ is the uniform mixture of the policies collecting data. Technically, in the second phase, $\\pi_{mix}^T$ is very close to $\\pi^*_{r^*}$, and as implied by Lem. 3.1, $Cov^{\\pi^*_{r^*}|\\pi_{mix}^T}$ is at constant level, which results in the $\\tilde{O}(T^{-1/2})$ rate. From this view, $\\beta$ only matters in how fast $Cov^{\\pi^*_{r^*}|\\pi_{mix}^T}$ reduces to constant as $T$ grows.\n\n\n### 1.2 About structure of Section 3 and 4\n\nThank you for pointing it out, and we will try our best to make it more understandable. The core insight behind our discussion in Sec. 3 and 4 (and also empirical TPO in Sec. 5) is that **\"select and transfer from the policy with the best coverage of $\\pi^{\\*}_{r^{\\*}}$\"**. It may help the reader to grasp the main algorithm if we highlight this statement earlier (e.g. Sec. 3.2).\n\n## 2. Methods And Evaluation Criteria & Experimental Designs Or Analyses\n\n> It is also not clear to me whether Table 1 is a 'good' empirical result…\n\nWin rate is a common metric in evaluating the LLM performance. A \"(50+$p$)%\" win rate against no-transfer baseline roughly means that, per 100 prompts, the LLM fine-tuned by empirical TPO is preferable on $50+p$ out of 100, which is $2p$ more than no-transfer baseline ($50 - p$ out of 100).\n\nBesides, the performance gap may be further enlarged by increasing training batch size and training epochs, incorporating better source reward models, adjusting test-time decoding temperature, etc. Notably, the quality of the inital policy also affects the magtitude of the performance gap---**the more near-optimal the initial policy is, the smaller the maximal possible performance gap can be**, although empirical TPO can still outperform the no-transfer baseline.\n\nTherefore, our primary goal in experiments is to demonstrate the potential of our transfer learning techniques in improving sample efficiency, especially, reward transfer remains an underexplored area in online RLHF.\n\n> ...is the point that only three iterations were needed?\n\nHere we follow the related literature [1,2], where running for three iterations is a common choice in algorithm evaluation (due to limited computational resources).\n\n> Would the trend in improvement continue with additional iterations?\n\nAs reported in Fig. 2 in Appx. I, in the 3rd iteration, empirical TPO already switches back to online learning without transfer, as the online policy can already outperform the best source policy in win rates. If we continue training for more iterations, the comparison with the no-transfer baseline effectively reduces to running the same online algorithm but with a better initial policy. Overall, we can expect the advantage of empirical TPO to persist, although the performance gap may decrease as both of them are converging to the optimal policy.\n\n## 3. Relation To Broader Scientific Literature\n\nTo our knowledge, most existing transfer learning literature, especially those theoretical ones, focuses on pure-reward maximization setup. We are the first to identify special structure (Lem. 3.1) induced by KL regularization, and propose novel transfer learning principles specialized in this setting.\n\n## 4. Other Strengths And Weaknesses\nThank you for your suggestions! We will take them into consideration in our next revision.\n\n[1] Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint\n\n[2] Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF", "claims_and_evidence": "1. While I understand the claim that Theorem 3.2 shows that the offline policy\n   converges at an asymptotic rate that does not depend on any structural\n   complexity measure (that you are treating everything except $T$ as a\n   constant), I think it may be clearer to say more explicitly that there are\n   two phases to the convergence rate. In the first phase, convergence rate\n   decreases at a rate of $O(e^{1/\\beta}\\mathcal{C}(\\Pi)(\\beta T)^{-1})$, while\n   in the second, it decreases at a rate of $O(T^{-1/2})$. In particular, if\n   this complexity measure $\\mathcal{C}(\\Pi)$ used in Corollary E.6 is infinite,\n   then the second phase that is independent of $\\mathcal{C}(\\Pi)$ never happens\n   (or, if it is allowed to grow with $T$).\n\n   Preserving $\\beta$ and $\\mathcal{C}(\\Pi)$ in the bound makes it clear that\n   they do matter in convergence, but only in the first phase. This is a pretty\n   interesting phenomenon, and maybe it would be nice to expand further on this\n   discussion. For example, do you have any intuition/justification for why the\n   second phase is also independent of $\\beta$? \n\n2. I thought Section 4 was a bit confusing to read. I think the last two\n   paragraphs of Section 3 are quite important to understand Section 4. But\n   somehow, it is not obvious until one understands the TPO algorithm why they\n   are important. Is there a way to write Sections 3 and 4 so that it is easier\n   to grasp what the algorithm? Part of the problem is that the way that Theorem\n   3.2 relates to the whole paper and TPO is also not initially clear. I like\n   the line near the end of Section 5 \"learn from an expert until surpassing\n   it\". Perhaps that idea could be more explicit earlier in the paper and could\n   be a guiding thought figure throughout the paper.", "methods_and_evaluation_criteria": "1. The experiments were a bit confusing to me. A lot of details were missing\n   from the main paper and are in the appendix. It is also not clear to me\n   whether Table 1 is a 'good' empirical result. These numbers do not really\n   seem to say that the proposed empirical method beats the baselines\n   (everything seem quite close to 50% win rate). Or, is the point that only\n   three iterations were needed? Would the trend in improvement continue with\n   additional iterations?", "theoretical_claims": "I did not have time to check proofs and for the most part only read the main\npaper.", "experimental_designs_or_analyses": "As I mentioned before, the experiments were somewhat confusing. It would be nice\nif there were more justification for why these experiments were performed.", "relation_to_broader_scientific_literature": "I found the high-level ideas of this paper quite interesting, especially the\nidea of self-transfer. I almost wish there more of a focus on this and transfer\nfor KL-regularized bandits, and less RLHF. In other words, are the same ideas be\npresent even in the very basic bandit setting? In terms of exposition, I think I\nwould've really appreciated learning the basic ideas in a much simpler setting.\nAlthough, I understand that it is also good to connect to the current RLHF/LLM\nresearch interests across the community.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "1. In Section 2, Additional Notation, you write that \"$\\mathcal{R}^\\Pi$ denotes\n   the reward class converted from $\\Pi$.\" I don’t believe \"converted\" is\n   standard terminology in RL. If what’s meant is the reward class induced by\n   $\\Pi$ via the one-to-one mapping in Eq. (2), it may be helpful to say that\n   directly to avoid confusion.\n\n2. I think overall the writing is not bad, but the overall structure could do\n   with a bit more thought. It was not trivial to understand the structure of\n   the paper and required very non-linear reading. Perhaps this is due to my\n   lack of familiarity with the area, however.", "other_comments_or_suggestions": "The review above is written by me. Below, I asked chatgpt to help summarize it. I think it captured my overall impression quite well:\n\n```[ChatGPT summary of my review]: This paper presents a theoretically motivated and timely contribution to RLHF by showing that KL regularization aligns exploration and exploitation, enabling safe and effective transfer from both external and self-generated policies. I found the core idea of \"self-transfer\" particularly compelling, and the theoretical results are thoughtfully developed. However, I believe the presentation could be improved—the connection between Sections 3 and 4 is not immediately clear, and the empirical results, while suggestive, are limited in scope and detail. Additionally, I think the claim of complexity-independent convergence could be more carefully nuanced to reflect the two-phase nature of the bound. Despite these issues, the paper makes a valuable contribution to understanding transfer in regularized bandit-style RLHF, and I recommend it for acceptance with revisions focused on clarity and empirical depth.```\n\nFinally, though I do believe I understood the paper, I don't work directly in RL or RLHF, so my confidence score is somewhat low because I am not very familiar about how this paper is situated within the field.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KtvhV9eG4k", "reviewer": "Reviewer_UGku", "summary": "The paper proposes a transfer learning algorithm that utilizes offline and online preference-based policy learning methods for RLHF. They provide a policy selection algorithm in each step where a new policy is selected based on a set of imperfect reward models and is used to further augment the training dataset. They provide theoretical evidence, a regret bound on the learned policy, for their proposed method and an emprical computationally efficient algorithm as a practical alternative for their method. \nThe motivation and flow of the paper is well-written and the principles explained about the concept of coverage and its use in the construction of the algorithm gives very clear picture of the whole idea of the paper.\nThey provide a regret bound that is independent from size of the state and action space and the complexity of the policy space.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback. It seems there may be some misunderstandings regarding our setting and several our key claims. To clarify, we start with a general remark, followed by detailed point-to-point responses. We hope our replies improve the clarity of our submission and help the reviewer evaluate our paper.\n## General Remark for Clarification\nAs stated in Sec. 1&2 and Fig. 1, **we target at improving sample efficiency in online RLHF by transferring from imperfect source RMs**, i.e. learning $\\pi^\\*_{r^\\*}$  from online human feedbacks associated with $r^\\*$, while leveraging auxiliary RMs $r^1, \\dots, r^W$.\n* Our theoretical method TPO (Alg 1&2) **is not offline but instead an online reward transfer algorithm**, and we just use an offline subroutine (RPO) to compute a transfer candidate (Lines 6–7 in Alg. 2). Besides, **we did not claim Thm. 3.2 is a contribution for offline literature**. Instead, we claim it improves the existing convergence rate of online RLHF methods, which motivates \"self-transfer learning\". **Our core theoretical contribution lies in analyzing the benefits of transfer learning in online RLHF**. As stated in Thm. 4.3 and Sec. 4.2, TPO improves the online RLHF results given good source RMs and self-transfer learning.\n* Empirical TPO (Alg. 3) is closely aligned with the theoretical insights behind TPO—**transfer from policy with better coverage for $\\pi^{\\*}\\_{r{\\*}}$ (i.e., low $Cov^{\\pi^{\\*}\\_{r^{\\*}}|.}$)**. TPO follows it by selecting policy with its value gap (upper bound of $Cov^{\\pi^*_{r^*}|.}$ by Lem. 3.1), while empirical TPO utilizes win rates (lower bound of $Cov^{\\pi^*_{r^*}|\\cdot}$ by Lem. 5.1, but more scalable).\n## 1. Claims and Evidence\n>Theorem 3.2...violates the authors' claim.\n\nIn Thm 3.2, we only omit $\\log|\\Pi|$—the log covering number. We apologize that our wording is not precise enough, and will revise to \"no dependence…up to log-covering number factors\". However, such log terms are standard even in supervised learning, and removing any policy coverage terms in previous online RLHF bounds is a significant improvement. \n\nWe also clarify that Thm. 3.2 is about convergence rate, not regret bound, and it is only part of the contributions.\n>The method...not offline…not online…\n\nWe apologize for the confusion. We study reward transfer in **online setting**, and the term \"offline\" appears in paper just because we use an offline method (RPO) to compute a policy. We will reword \"offline\" to \"distilled\" to avoid confusion.\n>the contribution...to…RPO and XPO is not clear\n\nWe reply to it together in point 2 below.\n## 2. Methods And Evaluation Criteria & Experimental Designs Or Analyses\n\nGiven that we study the online setting, offline methods (e.g. RPO) operate under different assumptions and objectives and are naturally not comparable.\n\nRegarding online RLHF methods, note that other existing methods (XPO, IPO, etc) cannot handle transfer settings, hence they are not directly comparable. Besides, the core technique in empirical TPO is the \"win rate-based source policy selection via UCB\", which adapts to the best source RMs and switches back to normal online learning if no benefits in transfer, *without prior knowledge on task quality*. Therefore, **we view other online methods (e.g. XPO, IPO) not as competing baselines, but rather as complementary approaches that can be enhanced with our transfer learning techniques**. For limited space, we refer to \"1.1 Comparison with other online algorithms\" in our response to Reviewer yCvD for additional discussion and experimental support.\n\nOur main goal in experiments is to show the effectiveness of empirical TPO, and our experiment results clearly demonstrate these. Besides, win rate is a standard metric in evaluating RLHF methods. We note that accuracy is not typically used as a standard metric for summarization tasks.\n## 3. Theoretical Claims\n>​​Removal of...in the offline learning literature is not a contribution…\n\nThere is a misinterpretation of our contribution. We study the online setting and our main contribution is to **improve sample efficiency in online RLHF by transfer learning**.\n>...bounded policy ratio is very restrictive…\n\nSuch an assumption is standard in online RLHF literature [1,2]. Besides, as stated in Footnote 1 (Page 3), it is not essentially an assumption, but an additional preprocessing step given a realizable policy class, because $r^* \\in [0, R]$ implies $||\\log \\pi^*_{r^*}/\\pi_{ref}||_\\infty \\leq R/\\beta$.\n \n## 4. Other Strengths And Weaknesses\n>...empirical algorithm, differs significantly from the main algorithm…the source selection algorithm is totally changed\n\nWe respectfully disagree with it. Both our TPO and empirical TPO share the same insight (see second bullet in general remark).\n\n[1] Exploratory preference optimization: Harnessing implicit q*-approximation for sample-efficient rlhf\n\n[2] Self-Exploring Language Models: Active Preference Elicitation for Online Alignment", "claims_and_evidence": "The main claims of the paper include,\n1. The design of an algorithm for RLHF (TPO) with proved regret bound.\n2. Proposition of the computationally efficient version of TPO\n3. The importance of policy value as a criterion for selecting the policy to generate training data.\n4. The proposition of a policy learned by offline data, that is proved to have a regret bound of $O(1/\\sqrt{T})$ with no dependence on the action and state spaces size or any complexity measures on the policy space.\n\nThe first three claims are well justified, but the fourth claim, which I assume is the main contribution of the paper, is not correctly justified. Theorem 3.2 is proposed to provide a regret bound on the offline policy and is claimed to be independent of **any complexity measure on policy class**. But actually, when looking at the complete term of the bound in the Appendix, it **depends** on $|\\Pi|$, which violates the authors' claim.\n\nAnother problem with this claim that affects the whole idea of the paper is that we use the term **Offline** when we don't have access to the environment anymore and have to teach the model with the prepared, fixed training dataset. This is one of the core applications of offline methods that work even when we can't interact with the environment anymore. Here, to train the offline model, we require a no-regret online method to generate the dataset, so we have to access the environment continuously for the value difference term in the bound to vanish, hence the so-called offline policy is no longer offline, it requires access to the environment to generate a continuously improving dataset to learn from.\n\nAltogether, the contribution of the paper compared to many algorithms in this field like RPO and XPO is not clear. The method requires access to the environment (so it's not offline) and it uses the same data sample many times during the training (so it's not online), and also the proved bound doesn't seem to have any advantage over RPO or XPO in terms of T, or other factors.", "methods_and_evaluation_criteria": "The theoretical evaluation is the conventional regret bound, which is standard and reasonable.\nThe experimental evaluation is not very convincing, as it is the win rate of the policies compared to each other. An absolute measure of performance, like accuracy in the preference dataset, could be a better metric, as it can be compared with any baseline or SOTA method.", "theoretical_claims": "As mentioned in the claims section, theorem 3.2 provides a regret bound on the offline policy that is claimed to be independent of policy class complexity, but it's not.\nThere is another serious problem with the theoretical claims, and it is the removal of the best policy coverage term for the offline policy. For offline learning, coverage is necessary because we don't have control over the offline dataset. If we can control the offline dataset, we can trivially generate customized datasets that don't have a high coverage issue. Removal of the coverage term in the offline learning literature is not a contribution, as the coverage term makes the bound instance dependent and tighter, based on the quality of the dataset and algorithms that don't have the coverage term, often have looser bound, because the bound should work on **any dataset**.\n\nAlso, the assumption of a bounded policy ratio is very restrictive and not realistic in most of the applications.", "experimental_designs_or_analyses": "Experiments are not complete. The complete algorithm that automatically selects the best source model is missed in the main paper results, only fixed source selection methods are reported and tested. Also the only compared baseline is iterative-DPO, and algorithms like RPO, XPO, IPO, and SimPO are missed as baseline.\nThe evaluation criterion is not justifiable, because it is only the win rate of different policies compared to each other.\nThe experiments section seems more like a small ablation study section rather than the main experiment section. Even the appendix doesn't cover the lack of experiments and insufficient experimental evidence. It should at least contain the absolute performance of the complete method on difference preference datasets and comparison with online and offline RLHF methods like RPO and XPO.", "relation_to_broader_scientific_literature": "The paper's main goal is to solve the RLHF task by generating a good training dataset and train a model on the generated dataset. It is linked to both offline and online policy learning methods that use human-annotated preference data. However the focus of the paper is not generating data from human feedback, but from a set of already available reward models.", "essential_references_not_discussed": "I didn't recognize any missed essential references.", "other_strengths_and_weaknesses": "The whole paper fails to convince its contribution to the big literature in RLHF, \nMoreover, the provided empirical algorithm, differs significantly from the main algorithm, in the very important part of the source selection. This will make the validity of the theoretical evidence questionable for the empirical algorithm. It is common to have an empirical algorithm that is different from the theoretically justified method because of some estimations required in practice, but in this work, the difference is not just because of an estimation, the source selection algorithm is totally changed.", "other_comments_or_suggestions": "I have no other comments.", "questions_for_authors": "I don't have any questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uACcadLqhx", "reviewer": "Reviewer_kNcQ", "summary": "This paper studies the provably benefits of transferring knowledge from imperfect reward models (RMs) in online reinforcement learning from human feedback (RLHF). First, this paper identifies an important property specific to KL-regularized RLHF: the coverability for the optimal policy can be upper bounded by the policy value gap. This implies in order to obtain a dataset with good coverage, it is sufficient to rollout the policy with high value. Guided by this principle, this paper proposes a “self-transfer learning” procedure, which first runs a online no-regret method to generate a dataset, followed by employing an offline method to output the final policy. This paper proves that this procedure enjoys a sub-optimality bound of $\\mathcal{T^{-1/2}}$, which improves previous results by removing the dependence of certain complexity measure in the dominating term.\n\nAgain guided by the above principle, this paper further proposes a transfer learning method TPO. TPO first runs certain iterations of online algorithm and then switches to a transfer policy selection (TPS) procedure, which selects the policy with the highest optimistic estimate of the value. This paper provides regret bound of TPO, demonstrating that 1). in the early stage, the regret of TPO is reduced by leveraging the imperfect RMs; 2). after finite iterations, the regret bound almost reduces to the one of self-transfer learning.\n\nFinally, this paper proposes an empirical TPO method, which selects the policy based on the wining rate. The effectiveness of TPO is validated on a simple summarization task.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback and constructive suggestions! We address your specific comments in the following.\n\n## 1. Claims And Evidence\n\n### 1.1 About Theorem 3.2\n\nAs correctly pointed out, the sub-optimality gap is $\\tilde{O}(T^{-1/2} + Cov(\\Pi) T^{-1})$, and that’s why we claim the suboptimality/regret does not depend on complexity measure **after finite time**. As long as $T \\geq \\Omega(Cov(\\Pi)^2)$, we have $\\tilde{O}(T^{-1/2} + Cov(\\Pi) T^{-1}) = \\tilde{O}(T^{-1/2})$, and $T$ becomes the only dominating term. We will make this point clear in our revision.\n\n### 1.2 A few rigorousness Issue\n\nThanks for pointing them out! We will revise our statements to improve their rigor as recommended.\n\n\n## 2. Theoretical Claims\nFirstly, notice that the second term of Eq.(8) takes the minimum over $\\sum_w 1/\\Delta(w)$ and $\\sqrt{Wt}$. When $\\Delta(w)$ is very small, $\\sqrt{Wt}$ avoids the upper bound being extremely large.\n\nSecondly, as in most regret bounds in online learning, our result characterizes the worst-case behavior. Analogous to multi-armed bandit (MAB) settings, for fixed $T$, in the worst case we may have $\\Delta(w) = \\sqrt{T/W}$, resulting in regret that matches our bound.\n\nMoreover, there is a simple refinement can be done: additionally taking the minimum over the current bound and $\\sum_w \\Delta(w) N(w,T) = O(\\sum_w \\Delta(w) T)$, where $N(w,T)$ denotes the number of times we transfer from model $\\pi^*_{r^w}$ in $T$ iterations. This may align more closely with the reviewer’s intuition. In fact, this basic regret bound was the starting point of deriving Theorem 4.3.\n\n## 3. Experimental Designs Or Analyses\n\nThanks for raising this valid point. Despite difference in design, both TPO and its empirical version are grounded in the same core theoretical insight—**transfer from policy with better coverage for $\\pi^{\\*}\\_{r^{\\*}}$ (i.e. low $Cov^{\\pi^{\\*}\\_{r^{\\*}}|.}$)**. The value estimation is preferable from a theoretical standpoint, but it is often computationally expensive in practice. To address this, our empirical TPO uses win rate as a proxy, because it is more scalable and characterizes the lower bound for $Cov^{\\pi^{\\*}\\_{r^{\\*}}|.}$ (Lem. 5.1). We view this as a reasonable trade-off between theoretical rigor and empirical scalability.\n\n\n## 4. Other Comments Or Suggestions\n> Could we leverage...while achieving an improved regret bound?\n\nThis is a very interesting question. We conjecture it is possible to design more elegant algorithms with better regret bound, and we believe it is a highly valuable direction. We hope our work can serve as an initial step and inspire future developments, and we leave this for future work.", "claims_and_evidence": "I feel main claims in this paper are well supported by clear evidence. I only have some minor questions.\n\n1. In the discussion below Theorem 3.2, the paper emphasizes that the suboptimality does not depend on |S|, |A| or other complexity measure. But if I understand correctly, the suboptimality is of $\\mathcal{O} (T^{-1/2} + Cov (|\\Pi|)T^{-1} )$, which still depends on the complexity measure.\n2. In the first paragraph in Section 2.2, the paper states that $\\Delta_{\\min}=0$ implies the realizability of the reward class. This is not very rigorous because a reward with a constant shift also ensures the same optimal policy. The same issues also happens in “there is a one-to-one correspondence XXX”.", "methods_and_evaluation_criteria": "The methodology of this paper builds on the policy coverage perspective commonly used in existing offline RLHF theory, along with a novel structure property induced by KL regularization. I find this methodological approach well-reasoned.\n\nBesides, the experiments are conducted on a standard summarization task, which looks solid to me.", "theoretical_claims": "The results appear to be mathematically sound, though I have a question about Theorem 4.3, which contains a seemingly counterintuitive element. Specifically, the second term in Eq.(8) contains a $\\sum_{w} 1/\\Delta (w)$ quantity that increases as the error of source RMs decreases. This seems to contradict the intuition that higher-quality RMs should lead to lower regret.", "experimental_designs_or_analyses": "The experimental designs are adequate. However, the empirical results have a weak connection to the theory. This is primarily because empirical TPO uses the winning rate to select policies. Unlike the policy value, the winning rate cannot provide an upper bound for the coverage coefficient—a crucial element in the theoretical framework established in this paper.", "relation_to_broader_scientific_literature": "This paper studies online RLHF for LLM alignment and may inspire advancements towards sample-efficient methods for training LLMs.", "essential_references_not_discussed": "To the best of my knowledge, this paper provides a sufficiently thorough discussion of all closely related works.", "other_strengths_and_weaknesses": "Please see my comments in the above part.", "other_comments_or_suggestions": "The proposed self-transfer learning process employs two existing RLHF methods, which may not be computationally efficient since it involves two separate policy optimization procedures. Could we leverage the structural property to design a new online RLHF method with a single policy optimization procedure, while achieving an improved regret bound?", "questions_for_authors": "Please see my questions in the above part.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VrIXrvQIUg", "reviewer": "Reviewer_yCvD", "summary": "This paper investigates reward transfer in the context of online active RLHF. Existing investigations (based on active preference collection using on-policy sampling) have regret bounds proportional to instance-dependent properties, such as the cardinality of the action space. This investigation assumes access to imperfect reward models, and makes a novel connection between the coverage of the optimal policy due to the policies induced by these reward models, and their sub-optimality gaps with respect to the optimal policy. Leveraging this insight, the authors propose a policy selection routine to speed up convergence in the early stages. This subroutine works by forming estimates of the policy value induced by the imperfect reward models, and prescribing the policy which has the largest value. In terms of performance, the proposed algorithm is shown to exhibit sublinear regret which *does not depend on structural complexity measures*.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback and insightful suggestions! We address your comments as follows.\n## 1. Methods And Evaluation Criteria & Experimental Designs Or Analyses\n### 1.1 Comparison with other online algorithms\nRegarding online RLHF methods, we first point out that other existing methods (XPO, IPO [1] etc) cannot handle transfer settings, hence they are not directly comparable. Besides, the core technique in empirical TPO is the \"win rate-based source policy selection via UCB\", which allow us to adapt to the best source RMs and switch back to normal online learning if no benefits in transfer, *without prior knowledge on task quality*. Therefore, **we view other online methods (e.g. XPO) not as competing baselines, but rather as complementary approaches that can be enhanced with our transfer learning techniques**.\n\nTo make it clear, in our next revision, we will replace “DPO” (line 11, Alg. 3) with $Alg_{PO}$, which serves as a placeholder for any **P**olicy **O**ptimization oracle (e.g. DPO, XPO, IPO etc). This change aligns with the usage of generic placeholder $Alg_{OL}$ in TPO (Alg. 1).\n\nTo support this point, similar to Table 1 in paper, below we report win rates comparisons when we instantiate $Alg_{PO}$ by optimizing XPO and IPO loss, respectively, keeping other settings the same as Sec. 6 (except using learning rate 1e-5 in IPO). These results demonstrate that our transfer learning techniques can be effectively combined with different policy learners, leading to consistent performance improvements. We believe this highlights the modularity and generality of our framework, and we thank the reviewer again for prompting this valuable point.\n* **Empirical TPO (Alg. 3) by replacing DPO (line 11) with XPO**\n||Without Transfer|Purely Exploit ROUGE|Purely Exploit T5-Large|\n|:-: |:-:|:-:|:-:|\n|Iter 1|52.3$\\pm$1.1|53.4$\\pm$0.8|50.2$\\pm$0.3|\n|Iter 2|51.6$\\pm$1.3|54.7$\\pm$1.6|49.1$\\pm$1.3|\n|Iter 3|52.2$\\pm$1.6|53.8$\\pm$2.9|49.2$\\pm$1.1|\n* **Empirical TPO (Alg. 3) by replacing DPO (line 11) with IPO**\n||Without Transfer|Purely Exploit ROUGE|Purely Exploit T5-Large|\n|:-: |:-:|:-:|:-:|\n|Iter 1|52.3$\\pm$1.0|50.4$\\pm$1.6|49.9$\\pm$0.4|\n|Iter 2|55.2$\\pm$1.4|52.3$\\pm$0.3|50.1$\\pm$0.3|\n|Iter 3|55.3$\\pm$1.1|51.8$\\pm$0.5|50.3$\\pm$0.5|\n### 1.2 Additional benchmarks\n\nWe consider summarization task for the following reasons:\n1. Summarization task is important and it is also widely used in RLHF [2, 3].\n2. Summarization task is quite suitable for our reward model transfer setup. There are various choices for additional reward models with different qualities, such as similarity scores (ROUGE, BERTScore) with human expert summaries, advanced LLMs, etc.\n\nWe believe it is an interesting direction to evaluate our algorithms with other LLMs and benchmarks (e.g. those in [4] as suggested). Given that our experiments already effectively demonstrate the advantage of our proposed approach and due to limited computational resources, we leave further evaluation to the future works.\n## 2. Questions For Authors\n### 2.1 Why not estimate policy value directly\n\nThe main reason is that under the Bradley-Terry assumption, the preference distribution $\\mathbb{P}_{r^*}(\\cdot|s,a,\\tilde{a}) = \\sigma(r^*(s,a)-r^*(s,\\tilde{a}))$ is invariant under the transformation that $r^*(s,a)\\rightarrow r^*(s,a)+b(s)$, where $b$ is an arbitrary state-dependent function. As a result, we can at best identify the true reward $r^*(s,a)$ up to a state-dependent shift, which can largely bias the policy evaluation and make it unreliable.\n\nIn contrast, introducing $J_\\beta(\\pi_{\\text{ref}})$ as the baseline can cancel out the bias term. This enables consistent estimation for the value difference $J_\\beta(\\pi^*_{r^w})-J_\\beta(\\pi_{\\text{ref}})$.\n### 2.2 Clarification on the second term in Eq. (8)\n\nFirstly, notice that the second term of Eq. (8) takes the minimum over $\\sum_w1/\\Delta(w)$ and $\\sqrt{Wt}$. When $\\Delta(w)$ is very small, $\\sqrt{Wt}$ avoids the upper bound being extremely large.\n\nSecondly, as in most regret bounds in online learning, our result characterizes the worst-case behavior. Analogous to multi-armed bandit (MAB) settings, for fixed $T$, in the worst case we may have $\\Delta(w)=\\sqrt{T/W}$, resulting in regret that matches our bound.\n\nMoreover, there is a simple refinement can be done: additionally taking the minimum over the current bound and $\\sum_w\\Delta(w)N(w,T)=O(\\sum_w\\Delta(w) T)$, where $N(w,T)$ denotes the number of times we transfer from model $r^w$ in $T$ iterations. This may align more closely with the reviewer’s intuition. In fact, this basic regret bound was actually the starting point of deriving Thm. 4.3.\n\n[1] A General Theoretical Paradigm to Understand Learning from Human Preferences\n\n[2] Conditional Language Policy: A General Framework for Steerable Multi-Objective Finetuning.\n\n[3] BOND: Aligning LLMs with Best-of-N Distillation\n\n[4] Reinforcement learning from human feedback with active queries", "claims_and_evidence": "The claims are adequately substantiated.", "methods_and_evaluation_criteria": "I am not entirely convinced by the choice of baselines for comparing the proposed algorithm in empirical evaluations. \n\n- Since the paper shows improved regret compared against on-policy algorithms, why not compare against one of them (e.g. XPO)?\n- Why only choose summarization task as a baseline? What about other baselines (e.g., the ones considered in [1])?\n\n[1] Ji, K., He, J., & Gu, Q. (2024). Reinforcement learning from human feedback with active queries. arXiv preprint arXiv:2402.09401.", "theoretical_claims": "I did not go over the appendices, but the claims in the paper look correct.", "experimental_designs_or_analyses": "The experiment section is fairly limited. I have discussed my concerns about the baselines under \"Methods And Evaluation Criteria\".", "relation_to_broader_scientific_literature": "The contributions of this paper are very relevant to the RLHF community. To the best of my knowledge, reward transfer has not been investigated in the RLHF context. Furthermore, this paper makes some important theoretical observations, for example, relating the coverage of the optimal policy with respect to any policy, with the sub-optimality gap of the policy. Furthermore, the proposed algorithms improve upon existing regret bounds by getting rid of instance-dependence, which is also an important contribution. These strengths contribute towards my decision to lean towards acceptance of this paper, even though the simulations can be improved.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "These have been discussed.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "I have the following questions for the authors:\n\n1. Since we do not have access to the optimal reward $r^\\star$, we may not obtain the value o the imperfect reward models with respect to the optimal policy. To circumvent this, the authors propose to estimate the value gap $J_{\\beta}(\\pi^\\star_{r_w}) - J_{\\beta}(\\pi_{\\rm ref})$. Instead, what if we use the an estimate of the value function $J_{\\beta}(\\pi^\\star_{r_w})$ for policy evaluation? More specifically, form an optimistic estimate of $\\widehat r$, compute its value under $\\pi^\\star_{r_w}$, and select the source model with the largest estimated value. Why do we require to introduce $\\pi_{\\rm ref}$?\n\n2. In (8), the second term is contributed by the imperfection of the source reward models. However, if the gaps are small (near perfect source models), I wonder why it hurts the regret. Imagine having $\\Delta_{\\min}$ non-zero, but vanishingly small. In that case, the second term in (8) explodes. Is this a limitation of the analysis?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}]

Author rebuttal:

Meta-review: The reviewers appreciated the paper revealing the insight that due to reguarization, small policy value gap implies policy covering the optimal \pi_{r^*}^*, which can be vieweed as a way to perform self-transfer: using previously-trained policy to collect data can yield good coverage. They also appreciate the algorithmic insight that in the presence of imperfect reward models, these models can be used until the learned policy surpasses it. They also pointed out some weaknesses of the paper: 

- the theoretical algorithm and the empirical one were somewhat disconnected, in the difference in the way the data collection policy is chosen

- comparison of the theroetical guarantee of TPO and baselines can be clarified more; the bound in Theorem 4.3 looks fairly complicated; does it provide guidance on how to choose \alpha? The paper had some discussing on choosing \alpha as a sequence, but it is not clear how to choose it apriori if the quality of the reward models were unknown ahead of time.  

Due to these considerations this paper is on the borderline.
Decision: Accept (poster)