Online Iterative Reinforcement Learning from Human Feedback with General Preference Model

Authors: chenlu  ye
wei  xiong
yuheng  zhang
hanze  dong
nan  jiang
tong  zhang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 15:22:40

Abstract: We investigate Reinforcement Learning from Human Feedback (RLHF) in the context of a general preference oracle. In particular, we do not assume the existence of a reward function and an oracle preference signal drawn from the Bradley-Terry model as most of the prior works do.  We consider a standard mathematical formulation, the reverse-KL regularized minimax game between two LLMs for RLHF under general preference oracle. The learning objective of this formulation is to find a policy so that it is consistently preferred by the KL-regularized preference oracle over any competing LLMs. We show that this framework is strictly more general than the reward-based one, and propose sample-efficient algorithms for both the offline learning from a pre-collected preference dataset and online learning where we can query the preference oracle along the way of training. Empirical studies verify the effectiveness of the proposed framework.

Main paper: 
 introduction: 
 Reinforcement Learning from Human Feedback (RLHF) has emerged as a pivotal technique in adapting machine learning to leverage relative feedback, especially in aligning Large Language Models (LLMs) with human values and preferences [14, 90] . Notable examples include ChatGPT [49] , Claude [2] , and Bard [29] . The primary goal of RLHF in the context of LLMs is to adjust the responses generated by LLMs so that they are more favorably received by human evaluators.Inspired by the standard LLM alignment workflow [50, 5, 60] , we characterize an LLM by a policy π, which takes a prompt x ∈ X and produces a response a ∈ A from the distribution π(•|x). In a typical LLM training pipeline [50, 60, 49] , the tuning process begins with a pretrained model, which is subsequently fine-tuned using specialized and instructional data to produce an initial model π 0 . The initial model π 0 is then aligned with a prompt set from some distribution x ∼ d 0 . The key component in RLHF is the General Preference Oracle, which is mathematically defined as follows. Definition 1 (General Preference Oracle). There exists a preference oracle P : X × A × A → [0, 1], and we can query it to receive the preference signal:y ∼ Ber P(a 1 ≻ a 2 |x, a 1 , a 2 )where y = 1 means a 1 is preferred to a 2 , and y = 0 means that a 2 is preferred.Instead of directly optimizing against the preference oracle P, the existing prevalent RLHF framework is reward-based [50, 60] , which consists of three steps: (1) preference data collection, (2) reward modeling, and (3) policy optimization. Specifically, the preference dataset D consists of multiple tuples of the form (x, a 1 , a 2 , y), whose collection process can be modeled as:EQUATIONwhere π 1 D and π 2 D are behavior policies and are typically set as π 0 [60, 43] or some powerful closedform LLMs [16] . The second step is reward modeling, which is the origin of the name "reward-based". This step can be viewed as a kind of inverse RL [89] , which models some difficult-to-specify goals (preferred by the human or AI evaluators) as a scalar reward signal. Specifically, the Bradley-Terry (BT) model [9] , a framework widely adopted in Ouyang et al. [50] , Bai et al. [4] , Touvron et al. [60] , Rafailov et al. [53] , Xiong et al. [72] , assumes that there exists a ground-truth reward function P * and the preference model satisfies:EQUATIONwhere σ(z) = 1/(1 + exp(-z)) is the sigmoid function. Then, the reward model is taken as the Maximum Likelihood Estimation (MLE) of the BT model on the preference dataset D [e.g., 51, 48, 50, 4, 60] and is used in subsequent policy optimization steps to provide a signal for algorithms like Proximal Policy Optimization [56] . Despite its successes, the existence of a reward function and the BT model are strong assumptions, which may not fully capture the complicated human preferences. In particular, the BT model assumes that human preference is transitive, which means that if we prefer A to B (P(A ≻ B|x, A, B) > 0.5) and we prefer B to C, then it automatically holds that we prefer A to C. This assumption, however, is contradicted by evidence of intransitivity in human decision-making [62, 45] . This limitation is particularly pronounced if we consider the population-level preferences, where the ultimate preference signal is aggregated across diverse human groups [45] . This may further be evidenced that in the practical RLHF, the accuracy of the learned BT model is around 70% [4, 60, 16] , suggesting the challenges in approximating the complicated human preference by BT model. While there are some recent efforts to bypass reward modeling [53, 85] , they are still fundamentally derived from the reward-based preference model and suffer from the aforementioned issues. In contrast, the general preference oracle defined in Definition 1 is strictly more general than the BT model and can capture a more complicated preference pattern from the definition itself. It allows an intransitive preference model and can further capture the preference feedback from AI [5] , with a notable example of GPT-4 [49] , which is widely used for model evaluations in practice and may more accurately reflect real user experience [60, 18, 53, 72] . Moreover, from a practical side, the preference model construction tends to be more efficient than the reward function in terms of ranking accuracy. This is evidenced by the fact that the preference model, pairRM with 0.4B parameters [34] , performs comparably to a LLaMA2-13B-based reward model across a diverse set of preference targets [16] . As a case study, we train a reward model based on the Bradley-Terry (BT) model and a preference model with the same starting checkpoint Gemma-2B-it [59] and preference dataset 8 , with results presented in Table 1 and the training details are deferred to Section 5. As we can see, the preference model achieves much higher test accuracy in the reasoning task while maintaining comparable results in other tasks. Meanwhile, the training set we use is rather limited in the reasoning data (math and coding), so the reasoning task can be viewed as an out-of-distribution task. In this sense, the preference model may also provide a better generalization compared to the reward model. The results also extend to another case study with LLaMA3-8B-instruct, where the preference model shows promising potential in the improvement of reasoning tasks. We refer interested readers to check Zhao et al. [85] , Liu et al. [43] for further examples with similar observations. The advantage in ranking accuracy is not only directly beneficial for the algorithms that depend on ranking information [18, 30] , but also improves the performance of algorithms derived from the reward-based framework (i.e., Bradley-Terry model), as evidenced by the results in the study of (iterative) DPO [72, 31] .Given all these considerations, our study focuses on exploring the theoretical properties of RLHF under the general preference oracle (Definition 1), with the goal of advancing practical algorithmic designs. We summarize our contributions as follows:• We make the first attempt to study the theoretical learnability of RLHF under general preference oracle with KL regularization, in the offline setting with a pre-collected preference dataset and the online setting where we can query human feedback along the way of training, which demonstrates the potential of reward-model-free learning under general preference; • We propose sample-efficient algorithms in both the offline setting and online setting and establish the finite-sample theoretical guarantees under standard coverage and exploration conditions; • We show that the theoretical insights can be used to guide practical algorithmic designs with a reasonable approximation of the computational oracle. 
problem formulation: 
 In this section, we formulate the RLHF with general preference learning. Suppose that there exists a preference function P * : X × A × A → R which represents the prefererence of one action a 1 over another a 2 given a prompt x: P * (x, a 1 , a 2 ) = P(a 1 ≻ a 2 |x, a 1 , a 2 ). In practical applications, we want to make the resulting LLM π close to π 0 [90, 50, 4, 53] . Therefore, we adopt the following KL-regularized objective:EQUATIONOne primary reason to consider the regularized target is that the constructed preference model is only locally accurate, i.e., it performs well when there is little distribution shift. For instance, if the preference model is fine-tuned on a preference dataset collected by the initial model π 0 , it improves the in-distribution generalization, but the resulting model often performs poorly out-of-distribution [10] . Meanwhile, even if we require human labelers to give feedback along the way, the choices of the labelers may not be representative enough or the labelers can make mistakes due to limited time, attention, or care [27] . Moreover, the KL divergence in the target ensures that the resulting policy is stochastic instead of deterministic (given a suitable initial checkpoint), thereby more accurately reflecting the dynamics of generative language models.We choose P * as the target mostly for historical reasons [22, 65] . A choice is the relative preference log(P * (x, a 1 , a 2 )/(1-P * (x, a 1 , a 2 ))), which is equal to R * (x, a 1 )-R * (x, a 2 ) when the BT model holds so that (3) becomes two decoupled regularized-reward maximization problems in this case and automatically reduces to the setting considered in the previous work Xiong et al. [72] . While we do not handle this target directly, the analysis techniques presented in this paper readily apply to it with slight modifications.Nash Equilibrium and Best Response. Without loss of generality, we restrict our attention to the policy class Π consisting of the policies with the same support as π 0 and denote the unique Nash equilibrium (known as the Minimax Winner [57, 38, 24] or the von Neumann Winner [22] ) as the solution of the following minimax problem as:(π 1define the notion of best response. For function J and policy π 1 , the best response to π 1 is defined as argmin π 2 ∈Π J(π 1 , π 2 ) and the value is denoted by J(π 1 , †) = min π 2 ∈Π J(π 1 , π 2 ). Similarly, for π 2 , we have J( †, π 2 ) = max π 1 ∈Π J(π 1 , π 2 ). In particular, since π 1 * and π 2 * are the Nash equilibrium, they are the best response to each other.Function Approximation. Suppose that we have access to a function class P ⊂ (X × A × A → R) (e.g. neural network), which provides us with a set of candidates to approximate the P * , and also the preference functions P ∈ P satisfies P (x, a 1 , a 2 ) = 1 -P (x, a 2 , a 1 ). We make the following assumptions on the class P. Assumption 1. Assume that P is finite and the capacity of the class is large enough so that P * ∈ P.The finite class assumption is for a clear presentation and the results readily generalize to an infinite class with a bounded covering number by the standard discretization technique. We define a theoretical computation oracle as follows and defer the practical implementations to the experiment section. Definition 2 (Nash Equilibrium Oracle). For a given preference function P ∈ P and a reference policy π 0 , we can compute the Nash Equilibrium policyEQUATIONwhich means that the max-player is consistently preferred by the KL-regularized preference in the face of any competing policy π ′ up to a relaxation of ϵ. To stress the non-symmetric structures of the two players, we refer to the max-player as the main agent, which aims to find her ϵ-approximate Nash policy, and refer to the min-player as the enhancer, which is designed to facilitate the main agent's learning. In particular, when η is large enough so that the KL is roughly omitted, then, we can further obtain that minπ 2 ∈Π E x∼d0 E a 1 ∼π 1 ,a 2 ∼π 2 P * (x, a 1 , a 2 ) ≥ 0.5 -ϵ.In this case, the obtained policy π1 is consistently preferred by the preference oracle P * against any competing policies. We mention in passing that the KL penalty coefficient η > 0 exhibits a trade-off between being preferred by the oracle P * and staying close to the initial model π 0 , and reflects the degree of our belief in the oracle P * . In practice, η is typically treated as a hyper-parameter and is adjusted by parameter search [32] .Compared to the previous literature formulating the preference learning as finding a Nash equilibrium, although we focus on optimizing the policy for the max-player, we can also have a duality gap guarantee because of the symmetry of the objective function: J(π 1 , π 2 ) = 1 -J(π 2 , π 1 ). To see this, we decompose the duality gap into the suboptimality for the max-player π1 and the min-player π2 :Algorithm 1 Pessimistic Equilibrium Learning from Human Feedback E x∼d 0 E a 1 ∼π 1 ,a 2 ∼π 2 P (x, a 1 , a 2 ) + η -1 ln π0(a 1 |x) π 1 (a 1 |x) -η -1 ln π0(a 2 |x) π 2 (a 2 |x) .(9) 5: Output: π1 . to be independently collected as in (1) . We measure the suboptimality of the learned policy π1 by the gap between the Nash value and the best response value:EQUATIONwhere the KL-regularized function J is defined in (3) . Similar to the reward-based framework [50] , one natural approach is a two-staged method: (1) Construct an empirical preference model (reward model in the literature) by maximizing the log-likelihood function:EQUATION(2) Solve the policy by plugging the learned preference model P into the Nash Equilibrium Oracle 2. However, this framework typically leads to severe reward over-optimization issue [26] , meaning that while the model is preferred by the learned P , it may not achieve good performance under the evaluation of P * . This is because, with finite D off drawn from some behavior policy, it is unlikely to provide an accurate estimation for all the prompt-response pairs. Therefore, imposing heavy optimization pressure toward P will push the model to exploit these unreliable estimations to chase for a high proxy metric, thus leading to a worse performance under the ground truth P * . 
learning with pessimism: 
 The recent advances in the offline RL theory have demonstrated that the principle of pessimism with a conservative estimation is statistically efficient for offline learning across a diverse set of scenarios [35, 54, 69, 79, 86, 17, 71, 84] . In this section, we connect the KL-reversed minimax game in (3) with offline RL by pessimism via version space 9 .We introduce our algorithm, Pessimistic Equilibrium Learning from Human Feedback (PELHF) in Algorithm 1. Given an offline dataset D off , we first obtain the maximum likelihood estimation (MLE) P by maximizing (7) . Rather than directly planning with this empirical P , we form a version space P that contains P * ∈ P with a high probability under a suitable choice of β, as we show in Lemma 1. For each policy π 1 , we take the minimum preference function over P and the best responded π 2 as its conservative value estimation:Ĵoff (π 1 ) = min π 2 ∈Π min P ∈ P E x∼d 0 E a 1 ∼π 1 ,a 2 ∼π 2 P (x, a 1 , a 2 ) + η -1 ln π0(a 1 |x) π 1 (a 1 |x) -η -1 ln π0(a 2 |x) π 2 (a 2 |x) .Then, we solve the minimax game concerning this conservative value estimator. With this pessimistic modification, the resulting algorithm enjoys the following theoretical guarantee. Theorem 1.[Proof] If Assumption 1 holds, and we set λ = log(|P|/δ) and β 2 = 2 log(|P|/δ), then, with probability at least 1 -δ, the output policy of Algorithm 1 satisfiesThis theorem shows that the suboptimality gap depends on how the target (π 1 * , π 2 ) is covered by the offline dataset, where π 2 is maximized over the policy set Π. This coverage coefficient resembles the unilateral coverage 10 for Markov games [17, 86] . Then, a natural question is whether a good coverage condition (C(π 1 * , π D , P) is small) is practical in the context of LLMs. Unfortunately, since the response is usually long in practice, the distribution shift between policies is also very large. We summarize some observations here. First, along the way of the RLHF training, the average density ratio π(a|x)/π 0 (a|x) > exp (25) as reported in Figure 13 of Bai et al. [4] . See similar results of rejection sampling fine-tuning [18] and DPO [53] . Second, for a case study, we use the Gemma-7B-it as the behavior policy to collect data for aligning Gemma-2B-it [59] with 15k prompt from [16] . Then, we calculate the average KL divergence between Gemma-7B-it and Gemma-2B-it as 456.4. This evidence indicates that the coverage coefficient probably explodes in realistic scenarios. Therefore, it is unlikely to expect that we can learn the optimal policy from a pre-collected dataset. This motivates us to consider the online setting, where we can further query the preference oracle during the training to enrich the dataset thus enhancing our models continuously. 4 Iterative RLHF with Online Exploration 
setup of iterative rlhf: 
 The major difference between the online and offline settings is that online algorithms can further query the preference oracle P * along the way of training. Since updating the LLMs is expensive, we consider the batch online setting for a sparse policy update. Specifically, for each batch t ∈ [T ], we first update the policy pair (π 1 t , π2 t ) based on the historical information collected so far. Then, we collect m tuples: we sample a random prompt by x t,i ∼ d 0 , collect two responses by (a 1 t,i , a 2 t,i ) ∼ (π 1 t , π2 t ), and query the preference signal y t,i ∼ Ber(P * (x t,i , a 1 t,i , a 2 t,i )). Here the batch size m is usually very large compared to the typically adopted mini-batch update. To distinguish this from the sequential online setting where we update policy after collecting a single preference pair, we refer to this learning paradigm as the iterative RLHF.The primary advantage of online learning is that we can strategically choose the behavior policies in each iteration to improve the coverage of the collected data, which is referred to as the exploration in the literature. To achieve this goal, we need to quantify the data uncertainty to guide the exploration direction. To this end, we present the notions of information ratio and eluder coefficient.Information Ratio and Eluder Coefficient. Distinct from the offline setting where we assume the coverage condition of a pre-collected dataset D off , online exploration makes it possible to upper bound the suboptimality by the complexity of the function space. We leverage the notion of the eluder coefficient, which limits the generalization from visited state-action distributions to unseen parts. Definition 3 (Information Ratio and Eluder Coefficient). At round t, given an estimation P ∈ P, we define the information ratio for any two policy π 1 , π 2 asΓt(λ, π 1 , π 2 ) = sup P ∈P |E x∼d 0 [P (x, π 1 , π 2 ) -P (x, π 1 , π 2 )]| λ + t-1 s=1 E xs∼d 0 ,a 1 s ∼π 1 s ,a 2 s ∼π 2 s (P (xs, a 1 s , a 2 s ) -P (xs, a 1 s , a 2 s )) 2 .Then, the eluder coefficient is given by d(P, λ, T) := sup π 1 1:T ,π 2 1:T T t=1 min(1, (Γ t (λ, π 1 t , π 2 t )) 2). The information ratio and eluder coefficient considered here have also been adopted in the literature [e.g., 64, 28, 70, 74, 1] . Essentially, the information ratio compares the out-of-sample error on the unseen data with the in-sample error measured on the historical data, and can be interpreted as the worst-case ratio between them (as we take supreme over all possible P ∈ P). Meanwhile, the eluder coefficient limits the extent to which we can be "surprised" by the new out-of-sample distributions, given the historical data collected so far. The uncertainty for the preference model aligns with the uncertainty for the BT model under boundedness conditions, which is illustrated in the following example. We defer the details to Appendix D.1.Example 1 (Uncertainty in Bradley-Terry model with linear reward). Suppose the reward function can be embedded into a d-dimensional vector space {r(x, a) = ⟨θ, ϕ(x, a)⟩ : θ ∈ R d , ∥θ∥ ≤ B, ∥ϕ(x, a)∥ ≤ 1}. Then, if we define the covariance matrix asΣ t = t-1 s=1 E x∼d0,a 1 ∼π 1 s ,a 2 ∼π 2 s (ϕ(x, a 1 ) -ϕ(x, a 2 )) ⊤ (ϕ(x, a 1 ) -ϕ(x, a 2 )) + λ(1 + e B ) 2 I, we have Γt(λ, π 1 , π 2 ) ≤ (1 + e B )∥ϕ(x, π 1 ) -ϕ(x, π 2 )∥ Σ -1 t .Algorithm 2 Optimistic Equilibrium Learning from Human Feedback with Enhancer 1: Input: Preference space P, policy class Π, parameter η, λ > 0. 2: for t=1,. . . ,T do 3:Exploitation with the main agent: compute the MLE Pt with ℓD 1:t-1 defined in (7) and compute Nash equilibrium by calling the Nash equilibrium oracle 2:EQUATIONExploration with the enhancer: compute enhancer to maximize the uncertainty:EQUATIONCollectEQUATIONi ) ; 6: end for 7: Output: the best policy in (π 1 1:T ) by a validation set.We refer interested readers to Du et al. [20] , Zhong et al. [87] , Xie et al. [70] for the extensive examples when d(P, λ, T ) can have a sub-linear dependency on T . We are now ready to present the algorithm for the online setting, as summarized in Algorithm 2. Specifically, for each iteration, the main agent exploits the information contained in the data collected so far by computing the MLE Pt and solving the minimax game with respect to it to get π1 t . The enhancer, however, aims to facilitate the main agent's learning by maximizing the uncertainty relative to the π1 t . Finally, we use the policy pair to collect m preference pairs and query oracle P * to get the preference signals. Notably, to facilitate the computation for the main agent, instead of adding optimism to the value function, we impose the exploration role on the enhancer. This choice turns out to be important when we move toward practical algorithms with reasonable approximations, as we detail in Section 5. We now present the main theoretical guarantee for Algorithm 2. Theorem 2. [Proof] Under Assumption 1, for any ϵ > 0, if we set the total iterations T = min{n ∈N + : n ≥ 2d(P, λ, n)}, batch size m = 18T log(2T |P|/δ)/ϵ 2 , β = 2T log(2T |P|/δ)/m,and λ = 2T log(2T |P|/δ)/m for Algorithm 2, then, with probability at least 1 -δ, there exists a t 0 ∈ [T ],J(π 1 * , π 2 * ) -J(π 1 t0 , †) ≤ ϵ.The theorem states that with suitable hyper-parameter choices, after T iterations (up to log factors), we can find an ϵ-approximate Nash policy π1 t0 for the max-player. Here T depends on the eluder coefficient that is intrinsic to the preference model and characterizes the complexity of the problem. Key Ideas. We present a brief discussion of the key analysis ideas. Similar to Lemma 1, the MLE P ensures a controllable in-sample error (with details in the Appendix D). Recalling that the uncertainty bonus is essentially the worst-case ratio between the out-of-sample error (our learning target) and the in-sample error, to finally bound the out-of-sample error, we need to explore each direction where we are uncertain about so that the average uncertainty bonus is sufficiently small. Since the main agent is greedy (takes the best guess we can obtain so far), the enhancer plays the exploration role by maximizing the uncertainty relative to the π1 t . Then, since the eluder dimension is finite:T t=1 min 1, (Γ t (λ, π1 t , π2 t )) 2 ≤ d(P, λ, T ), there exists at least a t 0 ∈ [T ] such that the value at t 0 is smaller or equal to the average value:min 1, (Γ t (λ, π1 t , π2 t )) 2 ≤ d(P, λ, T )/T ≤ 1/2.Hence, with a proper m, we can obtain the result of Theorem 2.In practice, searching for the most uncertain policy in the whole policy space can be challenging and the enhancer policy itself does not enjoy any theoretical guarantee. We may slightly modify Algorithm 2 by restricting the exploration step to the following subsetEQUATION)where β is the parameter defined in Theorem 2. This set is never empty because we can prove that both π1 t and argmin π ′ J(π 1 t , π ′ ) belong to Π t . Intuitively, maintaining a small KL divergence against π1 t corresponds to exploiting the historical information, and maximizing the uncertainty relative to π1 t leads to more information gain. The choice of Π t represents a refined trade-off between these two different goals, thus making π2 t also converge to π * . The details are deferred to Appendix D.2. 
practical implementation of preference model and iterative rlhf: 
 In this section, we discuss how to implement the theoretical Algorithm 2 for the online setting.Main agent approximates Nash equilibrium oracle via self-play IPO. Approximating the information-theoretical oracle 2 given a known preference model has been studied in Munos et al. [46] , Calandriello et al. [11] . The proposed algorithm, self-play IPO, can serve as a reasonable approximation of the oracle by optimizing the following loss function:EQUATIONwhere SG[π] means that although we generate data from policy π, but we do not compute the gradient for this data-generation process. Moreover, according to Proposition 4.1 of Calandriello et al. [11] , the minimizer of ( 13) is the unique Nash policy of the (10) .Enhancer explores via rejection sampling. According to (12) , the enhancer aims to find a policy that ( 1) is close to the main agent's policy π1 t ;(2) maximizes the uncertainty relative to the π1 t . However, since for the general neural network, the uncertainty estimator does not admit a closed form, in practice, we typically resort to heuristic methods. One popular way in the context of alignment is the rejection sampling [47, 18, 43, 31, 76] . Specifically, given a prompt x, we use π1 t to independently sample n responses, use a tournament-style procedure to get the best response (and reject all other responses), and take the best responses as π2 t . In other words, we take the policy induced by rejection sampling with π1 t and P * as the enhancer policy π2 t . In this way, the π2 t enlarges the margins between π1 t while maintaining a moderate KL divergence. For instance, in the special case of the BT model, if we rank the samples via the learned reward, the KL divergence is upper bounded by log n -(n -1)/n and is usually far better than this conservative estimation [6] .Preference model construction. We follow Zhao et al. [85] , Liu et al. [43] , Dong et al. [19] to utilize the fact that the LLM is the next token predictor for the preference modeling. Specifically, we have a preference pair (x, a 1 , a 2 , A), where A means that the first response is better, which is formatted asinstruction = [CONTEXT] {x} [RESPONSE A] {a 1 } [RESPONSE B] {a 2 }, and label = A.Then, we simply treat the preference modeling as an instruction-following task to fine-tune the model on these instruction-label pairs. In particular, to mitigate the position bias (the preference model may prefer the response that is given in the position of RESPONSE A), we randomly switch the order of the two responses in the data formatting process. During inference, we simply use the probability of decoding A as the P (x, a 1 , a 2 ). We mention in passing that it is also possible to include a rubric in the instruction template to guide the model's prediction and achieve better results [52] . We observe the benefits of the additional prompt engineering in early experiments but decide to use the current version because the main focus is to verify the effectiveness of general preference structure. This implementation is also referred to as the Generative RM in subsequent works. 
experiments: 
 Model, Dataset, and Evaluation. We adopt the widely used open-source model Zephyr-SFT-7B [61] as the starting checkpoint, which is based on the Mistral-7B-v0.1 11 and fine-tuned on 200K high-quality Ultra-chat data [16] . We use the Ultra-feedback [16] as our prompt set. We divide the prompt set into the train set (60K), validation set (1K), and test set (3K). We mainly use head-to-head comparisons to evaluate the resulting models. In particular, we consider two types of win rate: 1) the win rate measured by the ground-truth LLaMA3-8B-based preference model on the hand-out test set from UltraFeedback; 2) the win rate measured by the GPT-4 Preview (11/06) on an out-of-distribution prompt set AlpacaEval2 [21] . Specifically, for the first evaluation, we use the best DPO model as the reference model, and for the AlpacaEval2, the GPT-4 Preview (11/06) is used as a reference model, and as the judge at the same time.Method and Competitors. We consider the implementation of Algorithm 2 with self-play IPO and rejection sampling as discussed in Section 5. We iterate for three iterations in total and for each iteration, we retrain a preference model using all the historical data, and run self-play IPO from the initial checkpoint π 0 (i.e., Zephyr-7B-SFT). For simplicity, we refer to this algorithm as Online ELHF IPO. We use the offline DPO [53] , offline IPO [3] , and SFT model as the baseline. In particular, we do not further fine-tune the Zephyr-7B-SFT on the preferred responses of Ultra-Feedback because the quality of Ultra-Feedback is lower than that of Ultra-Chat, which is generated by Chat-GPT APIs. For DPO, we follow Xiong et al. [72] , Tunstall et al. [61] , Rafailov et al. [53] to set the KL coefficient as η = 0.1. For IPO, we search the hyper-parameter in {0.1, 0.5, 1.0} and report the results in Table 2 .Clearly, the model with η = 0.1 beats all other IPO models and the SFT model with large margins, so we set η = 0.1 for the offline IPO and the Online ELHF IPO algorithm in the subsequent studies.Simulation framework. For all the offline algorithms, we sample two responses per prompt of the train set and use the LLaMA3-8B-based preference model to give the preference signal. Then, we run offline DPO and IPO with the synthetic dataset. For the Online ELHF, we set n = 4 in the rejection sampling process and use a tournament-style ranking method (so that the complexity of rejection sampling is linear in n) to find the best response.We use the open-source project TRL 12 to implement IPO and DPO. In particular, we have implemented IPO with log-likelihood/perplexity (perplexity is averaged log-likelihood by sequence length), where the original authors of IPO suggest that log-likelihoodbased implementation is unstable (see the huggingface blog 13 for details). We also found that the IPO without average cannot normally converge and is of poor performance and take the perplexity implementation accordingly. For DPO, we implement the vanilla version as the baseline. We present the main result in Table 3 . It is clear that Online ELHF-IPO outperforms the baselines. 
related work: 
 This section focuses on the theoretical aspects. A general discussion is provided in Appendix B.1Theoretical Study of Reward-based RLHF. The theoretical study of policy optimization from preference feedback dated back to the dueling bandits [e.g., 78, 55, 7] . This was later extended to the online RL setting by Xu et al. [73] , Novoseller et al. [48] , Pacchiano et al. [51] , Chen et al. [12] , including tabular online RLHF with finite state, and general function approximation for capturing real problems with large state spaces. Zhan et al. [81] , Wu and Sun [68] further encompasses the development of reward-free learning type algorithms and sampling-based algorithms for online RLHF. Apart from the online setting, there is another line of works [88, 80, 40] studying the rewardbased RLHF in the offline setting, which learns from a pre-determined offline dataset with suitable coverage condition over the state-action space. However, they consider reward maximization and deviate from the practical applications (e.g., these frameworks admit a deterministic optimal policy).Recently, Xiong et al. [72] first formulated the RLHF as a reverse-KL regularized contextual bandit and provided finite-sample guarantees in offline, online, and hybrid settings. We remark that all these papers consider only the reward-based RLHF framework, thus differing from ours.Theoretical Study of RLHF under General Preference Oracle. Our work is related to Dudík et al. [22] and Wang et al. [65] . They investigate preference-based RLHF under a general preference model. The major difference is that we consider the reverse-KL regularized preference, aligning closely with recent LLM advancements [90, 50, 4, 53] , while previous work only considers the non-regularized one. Meanwhile, Dudík et al. [22] considers the problem of finite action, while our work and Wang et al. [65] consider the problem with large or even infinite state-action under function approximation.In terms of learning paradigm and algorithmic design, we consider both offline learning from a precollected dataset and batch online learning with a sparse policy update, while Dudík et al. [22] , Wang et al. [65] studies sequential online learning that updates policy in each step, which is not feasible in the context of LLMs. Moreover, we demonstrate that the proposed algorithms can be reasonably implemented in practice, but Dudík et al. [22] , Wang et al. [65] only focus on information-theoretical algorithms. To summarize, the framework in this work accurately reflects real-world alignment practices thus aligning more closely with the RLHF practice. Our work is closely related to the IPO [3] and Nash learning [46] , which also motivate new algorithmic design with a general preference oracle. We comment on the similarities and differences between our framework and theirs as follows.In terms of the problem setting, our work and Nash learning consider the minimax game under the reverse-KL regularized preference, while IPO can be interpreted to find the best response of the fixed reference policy, and may be considered as a special case of the game formulation. In terms of learning paradigm, both the IPO and Nash learning only consider learning toward a fixed and known preference oracle, and study the optimization property of the problem: how to compute the optimal policy under the given preference oracle. In contrast, we study the statistical property, where the preference model needed to be learned and our goal is to find the optimal policy under the underlying ground-truth preference model. In particular, the computational challenge is hidden in Definition 2 and Munos et al. [46] provides a reasonable approximation of the planning oracle. In this sense, our work and Munos et al. [46] are complementary to each other. Finally, the concurrent work Swamy et al. [58] studies the non-regularized general preference model in the sequential online setting and aims to find the Nash equilibrium in the context of continuous control tasks. In terms of the observation model, they assume access to the preference score P(a 1 ≻ a 2 |x, a 1 , a 2 ), while we only observe the preference signal y ∼ Ber(P(a 1 ≻ a 2 |x, a 1 , a 2 )). Moreover, they design online RLHF algorithms based on a reduction to the no-regret algorithm like Hedge [25] , whose techniques are fundamentally different from ours. 
conclusion: 
 In this paper, we study the RLHF under a general preference oracle that can capture the non-transitive preferences. Specifically, we formulate the problem as a KL-regularized minimax game between two LLMs, and propose statistically efficient algorithms in both the offline and online settings. The proposed algorithms, with a carefully crafted non-symmetric algorithmic structure, can be practically implemented with reasonable approximations of the information-theoretical computational oracles. We hope our findings can advance the understanding of preference signal modeling in RLHF and stimulate further research beyond the classic reward-based framework. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: Theory paper, no visible societal impact found in a short term.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Theory paper, no such risk.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: All datasets are used properly.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: All authors provided valuable contributions to this project, each bringing unique expertise and insights that were crucial for its success.• CY investigated the general preference problem, proved the theoretical results for both offline and online settings, and wrote the main part of the paper.• WX first proved the effectiveness of the general preference model, proposed the online iterative algorithm, contributed to the proof and paper writing, and contributed to the experiment.• YZ proved the properties of the general preference problem, made important contributions to the offline result and the proof, contributed to the paper writing.• HD designed the practical implementation under generalized preference model and conducted most experiments to show the effectiveness of the proposed algorithm.• NJ and TZ supported and advised the junior authors' works, provided computational resources and suggested experiments and writings. The inner product of two vectors z ⊤ 1 z2. ∥z∥ΣThe induced norm√ z ⊤ Σz. X , AThe state (prompt) space and the action (response) space. P, P *The preference oracle defined in Definition 1 and P * (x, a 1 , a 2 ) = P(a 1 ≻ a 2 |x, a 1 , a 2 ).The candidate set of preference model to approximate P * . y ∈ {0, 1}Preference signal. π, Π Policy and policy class.The KL-regularized target defined in (3). ηThe coefficient of KL penalty, defined in (3). ℓ DThe log-likelihood function defined in (7) . d0Distribution of state (prompt). σ(•) σ(z) = 1/(1 + exp(-z)) is the sigmoid function.Coverage term for version space-based Algorithm 1 defined in Theorem 1.Coverage term for uncertainty bonus based Algorithm 3 defined in Theorem 3. C ((π 1 , π 2 ), π D , P)Refined coverage term defined in Theorem 4.Γ(λ, π 1 , π 2 )Information ratio defined in Definition (3). Γ m t (λ, π 1 , π 2 ), Γ(x, π 1 , π 2 ) Uncertainty bonus defined in (11) and (19) .The eluder coefficient defined in Definition 3.A variant of O that omits logarithmic terms.RLHF. RLHF was first popularized in the deep RL literature by Christiano et al. [14] , which served to direct the attention of the RL community to the preference-based feedback, but may further date back to Bennett et al. [8] , Knox and Stone [36] in the context of machine learning. It has attracted significant attention recently, mainly due to its tremendous success in Chat-GPT [49] . The most popular and standard RLHF framework is outlined in Ouyang et al. [50] , Touvron et al. [60] and we have described the details in Section 1. In terms of reward optimization, PPO [56] is the most well-known algorithm in LLM alignment literature. However, tuning the PPO algorithm to the best performance requires extensive efforts and the result of Chat-GPT4 [49] has not been widely reproduced so far. This motivates another line of works of algorithms that are based on supervised learning. For instance, Dong et al. [18] , Yuan et al. [77] , Touvron et al. [60] , Gulcehre et al. [30] , Ji et al. [33] propose reward ranked finetuning, (also known as rejection sampling finetuning), which essentially learns from the best-of-n policy [47] to maximize the reward. The reward-ranked finetuning algorithm is a stable policy optimization algorithm with minimal hyper-parameter configuration and was applied to the RLHF of LLaMA2 [60] . However, it is also observed that the reward ranked finetuning algorithm leads to considerable forgetting in a wide range of tasks (also referred to as the alignment tax), as the algorithmic design only considers reward optimization [60, 42, 13] . One approach to mitigate this issue is to use the KL-regularized formulation, which is widely adopted in the deep RL approach (e.g. PPO) [90, 67, 50, 4, 37, 41] , and other supervised-learning-based algorithms [53, 63, 43, 3] , whose theoretical property is studied in Xiong et al. [72] . Among them, (offline) Direct Preference Optimization (DPO) [53] has emerged as an attractive alternative approach to PPO with notable stability and competitive performance. Xiong et al. [72] , Hoang Tran [31] , Yuan et al. [76] further extend the offline DPO to the iterative (online) variant, and the resulting models demonstrate impressive performance [31, 19] . However, all these algorithms are designed under the reward-based RLHF framework to maximize the underlying reward function (with appropriate regularization).Bradley-Terry model construction. We follow the previous works [50, 4] to initialize the reward model using an SFT model but replace the last layer with a linear head to predict a scalar score.The loss function of reward modeling is the negative log-likelihood so that minimizing the loss is equivalent to MLE:L RM (θ) = -E x,a w ,a l ∼D log σ r θ (x, a w ) -r θ (x, a l ) ,where a w is the preferred response over a l . We train the model for one epoch and use a batch size of 256, a learning rate of lr = 1e-5, and a cosine learning rate schedule with a warm-up ratio of 0.03.Ground-truth preference model for simulation. Ideally, the P * is supposed to be a group of human labelers or closed-source LLMs like Chat-GPT. Unfortunately, due to resource constraints, we cannot afford the cost of using these preference oracles. Instead, we follow Gao et al. [26] to use a strong preference model to serve as the P * in the simulation. Specifically, we adopt the LLaMA3-8B, and train the preference model on a diverse set of open-source preference datasets including HH-RLHF [4] , Stanford Human Preferences Dataset (SHP) [23] , Ultra-feedback [16] , HelpSteer [66] , distilabel-capybara 14 , distilabel-orca 15 , and UltraInteract 16 . Motivated by the Theorem 1 as well as the practical application [50] , we include more than 1 comparison pair when a prompt is with more than 2 responses for better coverage. To be specific,• for SHP, we only use the samples with score ratio > 2, and for each prompt, we take at most 5 comparison pairs;• for HelpSteer, we use all the possible pairs except for those with the same score where the score is averaged over helpfulness and correctness;• for UltraFeedback, we use all possible pairs except for those with the same score where the score is averaged over all attributes;• for UltraInteract, we take a subset of 150K pairs into the mixture.We have about 700K preference pairs in our training stage. We use the package axolotl 17 to perform supervised fine-tuning, with the detailed hyper-parameters given in Appendix B.2. The resulting preference models are evaluated by the reward bench [39] , with the results summarized in Table 1 .The preference model based on LLaMA3-8B-it achieves state-of-the-art test accuracy and can serve as a stable preference oracle for the simulation study.We present the hyper-parameters in Table 5 . All experiments are conducted on 8×A100-40G with Deepspeed ZeRO-3. Examples from Ultra-feedback. We provide several examples here:• Create a list of three mistakes to avoid when designing an AI assistant.• Pretend you're a next.js expert, write ad copy about a free trial on Vercel.• Can you describe the role of photography in shaping the art world?C Proofs for the Offline Setting C.1 Proof for Theorem 1 Lemma 1. Under Assumption 1, with probability at least 1 -δ, we haven i=1 ( P (x i , a 1 i , a 2 i ) -P * (x i , a 1 i , a 2 i )) 2 ≤ log(|P|/δ).Proof of Lemma 1. For any fixed function P ∈ P, we first upper bound its logarithmic moment generating function:EQUATION)We continue to lower-bound the Hellinger byEQUATION)where the inequality uses the fact that for any distribution p, q, H(p, q) ≥ TV(p, q) according to Theorem B.9 of Zhang [83] .Then, by invoking Lemma 6, we obtain for any P ∈ P, with probability at least 1 -δ,n i=1 log P (y i |x i , a 1 i , a 2 i ) P * (y i |x i , a 1 i , a 2 i ) ≤ log(|P|/δ) + log E exp n i=1 log P (y i |x i , a 1 i , a 2 i ) P * (y i |x i , a 1 i , a 2 i ) ≤ - n i=1 H P (y i |x i , a 1 i , a 2 i )∥P * (y i |x i , a 1 i , a 2 i ) 2 + log(|P|/δ) ≤ - n i=1 (P (x i , a 1 i , a 2 i ) -P * (x i , a 1 i , a 2 i )) 2 + log(|P|/δ),where the second inequality uses (14) , and the last inequality uses (15) . By taking P as P , since P is the MLE, we getn i=1 ( P (x i , a 1 i , a 2 i ) -P * (x i , a 1 i , a 2 i )) 2 ≤ n i=1 log P * (y i |x i , a 1 i , a 2 i ) P P (y i |x i , a 1 i , a 2 i ) + log(|P|/δ) ≤ log(|P|/δ).Proof of Theorem 1. Let(π 1 , π 2 ) = arg max π 1 ∈Π arg min π 2 ∈Π min P ∈ P E x∼d0 E a 1 ∼π 1 ,a 2 ∼π 2 P (x, a 1 , a 2 ) + η -1 log π 0 (a 1 |x) π 1 (a 1 |x) -η -1 log π 0 (a 2 |x) π 2 (a 2 |x) .and use the notationJ(π 1 , π 2 ) = min P ∈ P E x∼d0 E a 1 ∼π 1 ,a 2 ∼π 2 P (x, a 1 , a 2 ) + η -1 log π 0 (a 1 |x) π 1 (a 1 |x) -η -1 log π 0 (a 2 |x) π 2 (a 2 |x) .where we define Γ(π 1 , π 2 ) := supP ∈ P |E x∼d0 [P (x, π 1 , π 2 ) -P (x, π 1 , π 2 )]| λ + n i=1 (P (x i , a 1 i , a 2 i ) -P (x i , a 1 i , a 2 i )) 2.By the optimality of π1 , term q 3 ≤ 0. Since π 2 is the best response to π1 with respect to J, we have q 4 ≤ 0. From Lemma 1, we know that P ⋆ ∈ P, thus q 5 ≤ 0. Putting everything together, we obtain thatEQUATIONThen, by invoking Lemma 8 with a union bound over P ∈ P, with probability at least 1 -δ, we obtain that0.5nE x∼d0,a 1 ∼π 1 D ,a 2 ∼π 2 D (P (x, a 1 , a 2 ) -P (x, a 1 , a 2 )) 2 ≤ n i=1 (P (x i , a 1 i , a 2 i ) -P (x i , a 1 i , a 2 i )) 2 + log(|P|/δ),which implies that with probability at least 1 -δ,Γ(π 1 * , π 2 * ) = sup P ∈P |E x∼d0 [P (x, π 1 * , π 2 * ) -P (x, π 1 * , π 2 )]| λ + n i=1 (P (x i , a 1 i , a 2 i ) -P (x i , a 1 i , a 2 i )) 2 = 2C(π 1 * , π D , P) n .Hence, we complete the proof.In this subsection, we introduce another offline algorithm, Pessimistic Equilibrium Learning from Human Feedback (PELHF) with Uncertainty Bonus in Algorithm 3. Given an offline dataset D off , we first obtain the maximum likelihood estimation (MLE) by maximizing (7) . Then, we take the lower confidence bound (LCB) for the max-player as the preference estimations by subtracting a bonus function βΓ(•, •, •):EQUATION)Then, we obtain the policy π1 by solving the minimax problems with J. We now discuss how to construct the bonus function to ensure pessimism.Bonus Construction. The bonus function Γ : X × A × A → R + serves to control the point-wise confidence interval so that with high probability, P (x, a 1 , a 2 ) -βΓ(x, a 1 , a 2 ) ≤ P * (x, a 1 , a 2 ) ≤ P (x, a 1 , a 2 ) + βΓ(x, a 1 , a 2 ) holds for any (x, a 1 , a 2 ). To this end, we construct the bonus as the ratio between the out-of-sample error and the in-sample error on the preference dataset D off :EQUATION)where we also set β as an upper bound of the λ-regularized in-sample error. This uncertainty is also characterized by the relative preference function class and shares a similar spirit with the information ratio considered in Zhang [83] , Ye et al. [74, 75] , which depicts the uncertainty with respect to the value function class. Also see Definition 3 for a more detailed illustration.Algorithm 3 Pessimistic Equilibrium Learning from Human Feedback with Uncertainty Bonus1: Input: Dataset D off = {(xi, a 1 i , a 2 i , yi)} n i=1, preference space P, policy class Π, parameter η, β > 0. 2: Compute the MLE P with ℓD off defined in (7) and construct bonus as in (19) . 3: Compute the best policy under conservative estimationEQUATION)4: Output: π1 .Now, we are ready to present the suboptimality bound of π1 from Algorithm 3.Theorem 3. If we set λ = log(|P|/δ) and β 2 = 2 log(|P|/δ), then, with probability at least 1 -δ, the output policy of Algorithm 3 satisfiesJ(π * 1 , π * 2 ) -J(π 1 , †) ≤ 4β C(π 1 * , π D , P) n -E x∼d0 η -1 D KL (π 1 * (•|x)∥π 1 (•|x)) .whereC(π 1 * , π D , P) = max π 2 ∈Π E x∼d0 sup P ∈P (P (x, π 1 * , π 2 ) -P (x, π 1 * , π 2 )) 2 E x∼d0,a 1 ∼π 1 D ,a 2 ∼π 2 D (P (x, a 1 , a 2 ) -P (x, a 1 , a 2 )) 2 .Proof. Recall that our pessimistic value estimations areJ(x, π 1 , π 2 ) = E a 1 ∼π 1 ,a 2 ∼π 2 P (x, a 1 , a 2 ) -βΓ(x, a 1 , a 2 ) + η -1 log π 0 (a 1 |x) π 1 (a 1 |x) -η -1 log π 0 (a 2 |x) π 2 (a 2 |x) .For convenience, we also use the notationJ(x, π 1 , π 2 ) = E a 1 ∼π 1 ,a 2 ∼π 2 P * (x, a 1 , a 2 ) + η -1 log π 0 (a 1 |x) π 1 (a 1 |x) -η -1 log π 0 (a 2 |x) π 2 (a 2 |x) .We decompose the suboptimality gap of π1 at prompt x as follows:J(x, π 1 * , π 2 * ) -J(x, π1 , †) ≤ J(x, π1 , π 2 ) -J(x, π1 , †) p 1 + J(x, π 1 * , π 2 ) -J(x, π1 , π 2 ) p 2 + J(x, π 1 * , π 2 ) -J(x, π 1 * , π 2 ) p 3 + J(x, π 1 * , π 2 * ) -J(x, π 1 * , π 2 ) p 4 .We proceed based on assuming the following event holds:n i=1 ( P (x i , a 1 i , a 2 i ) -P * (x i , a 1 i , a 2 i )) 2 ≤ β 2 /2.For the term p 1 , we havep 1 =J(x, π1 , π 2 ) -min π 2 P * (x, π1 , π 2 ) -η -1 D KL (π 1 (•|x)∥π 0 (•|x)) + η -1 D KL (π 2 (•|x)∥π 0 (•|x)) =J(x, π1 , π 2 ) -min π 2 P * (x, π1 , π 2 ) -P (x, π1 , π 2 ) + P (x, π1 , π 2 ) -η -1 D KL (π 1 (•|x)∥π 0 (•|x)) + η -1 D KL (π 2 (•|x)∥π 0 ≤J(x, π1 , π 2 ) -min π 2 P (x, π1 , π 2 ) -βΓ(x, π1 , π 2 ) -η -1 D KL (π 1 (•|x)∥π 0 (•|x)) + η -1 D KL (π 2 (•|x)∥π 0 (•|x)) =0,where the inequality is becauseP * (x, π1 , π 2 ) -P (x, π1 , π 2 ) ≥ -λ + n i=1 (P * (x i , a 1 i , a 2 i ) -P (x i , a 1 i , a 2 i )) 2 • sup P ′ ∈P |E a 1 ∼π 1 ,a2∼π 2 [P ′ (x, a 1 , a 2 ) -P (x, a 1 , a 2 )]| λ + n i=1 (P ′ (x i , a 1 i , a 2 i ) -P (x i , a 1 i , a 2 i )) 2 ≥ -βΓ(x, π1 , π 2 ).Here the last step uses Lemma 1 to bound the in-sample error. For the term p 2 , we can write it asp 2 = P (x, π 1 * , π 2 ) -βΓ(x, π 1 * , π 2 ) -P (x, π1 , π 2 ) + βΓ(x, π1 , π 2 ) -η -1 D KL (π 1 * (•|x)∥π 0 (•|x)) + η -1 D KL (π 1 (•|x)∥π 0 (•|x)). We note that π1 = argmax π 1 J(x, π 1 , π 2 ) = argmax π 1 E a 1 ∼π 1 ,a 2 ∼ π 2 P (x, a 1 , a 2 ) -βΓ(x, a 1 , a 2 ) + η -1 log π 0 (a 1 |x) π 1 (a 1 |x) .Therefore, we can invoke Lemma 9 with π = π 1 * , π = π1 , and P (x, a) = P (x, a, π 2 ) -βΓ(x, a, π 2 ) to obtainP (x, π 1 0.5nE x∼d0,a 1 ∼π 1 D ,a 2 ∼π 2 D (P (x, a 1 , a 2 ) -P (x s , a 1 s , a 2 s )) 2 ≤ n i=1 (P (x i , a 1 i , a 2 i ) -P * (x i , a 1 i , a 2 i )) 2 + log(|P|/δ),which implies that probability at least 1 -δ,E x∼d0 Γ(x, π 1 * , π 2 ) =E x∼d0 sup P ∈P |P (x, π 1 * , π 2 ) -P (x, π 1 * , π 2 )| λ + n i=1 (P (x i , a 1 i , a 2 i ) -P (x i , a 1 i , a 2 i )) 2 ≤E x∼d0 sup P ∈P |P (x, π 1 * , π 2 ) -P (x, π 1 * , π 2 )| λ -log(|P|/δ) + 0.5nE x∼d0,a 1 ∼π 1 D ,a 2 ∼π 2 D (P (x, a 1 , a 2 ) -P (x s , a 1 , a ) ) 2 = 2 n E x∼d0 sup P ∈P |P (x, π 1 * , π 2 ) -P (x, π 1 * , π 2 )| E x∼d0,a 1 ∼π 1 D ,a 2 ∼π 2 D (P (x s , a 1 , a 2 ) -P (x, a 1 , a 2 )) 2 ≤ 2 C(π 1 * , π D , P) n ,where the second equality holds due to λ = log(|P|/δ). Therefore, we complete the proof.Comparison between Bonus and Version Space. Compared to the bound in Theorem 1, the bound in Theorem 3 enjoys an additional negative KL divergence term between π 1 * and π1 . Both Theorem 1 and Theorem 3 depend on a distribution-shift term between Nash policy π 1 * and the policy π D that the data is complied with. The difference is that Theorem 1 enjoys a sharper term C because of Jensen's inequality and the expectations are inside the sup operator. In terms of applicability, the version-space-based pessimism is preferred because it does not require a point-wise uncertainty estimator, thus applying to general cases. In contrast, point-wise pessimism, or more generally, optimism/pessimism via a biased target may be easier to heuristically approximate in practice, as shown in Coste et al. [15] , Xie et al. [69] , Zhang [82] , Liu et al. [44] .In this subsection, we show that with an improved analysis, Algorithm 1 enjoys a refined coverage condition, similar to the coverage notion in [84] . Theorem 4. If Assumption 1 holds, and we set λ = log(|P|/δ) and β 2 = 2 log(|P|/δ), then, with probability at least 1 -δ, the output policy of Algorithm 1 satisfiesJ(π * 1 , π * 2 ) -J(π 1 , †) ≤ min π 2 ∈Π 4β C ((π 1 * , π 2 ), π D , P) n + subopt π 1We can prove that Algorithm 3 also enjoys a similar bound and coverage condition. We now provide a breakdown of the terms in Theorem 4.Comparison to Unilateral Coverage. The unilateral coverage [17, 86] requires the dataset to cover all unilateral pairs (π 1 * , π 2 ) for any π 2 ∈ Π, making the bound in Theorem 1 depend on the coverage term of the worst pair. In contrast, the bound in Theorem 4 automatically adapts to the best π 2 , achieving the trade-off between the coverage term and the suboptimality term, thus offering a more flexible coverage condition.Proof of Theorem 2. We start with the error estimation. Similar to the proof of Lemma 1 but with an additional union bound over t ∈ [T ], we have with probability at least 1 -δ/2, for anyt ∈ [T ], 1 m m i=1 ( Pt (x t,i , a 1 t,i , a 2 t,i ) -P * (x t,i , a 1 t,i , a 2 t,i )) 2 ≤ log(2T |P|/δ) m ,which implies that we can set β 2 = T log(2T |P|/δ) m so that β Γ m t is a valid uncertainty bonus:EQUATIONWe proceed to prove that there exists at least one iteration, the out-of-sample error is close to the in-sample error. According to the Definition 3, we know that for any sequence {(π 1 t , π2 t )} T t=1 , it holds thatT t=1 min 1, (Γ t (λ, π1 t , π2 t )) 2 ≤ d(P, λ, T ).Since each term on the left-hand side is non-negative, we know that there exists at least a t 0 ∈ [T ] such that the value at t 0 is smaller or equal to the average value:min 1, (Γ t0 (λ, π1 t0 , π2 t0 )) 2 ≤ d(P, λ, T ) T ≤ 1 2 ,which further implies that (Γ t0 (λ, π1 t0 , π2 t0 )) 2 ≤ 1 2 . We use the notationπ 2 t = argmin π ′ J(π 1 t , π ′ ) and Ĵ(x, π 1 , π 2 ) = P (x, π 1 , π 2 ) -η -1 D KL (π 1 (a 1 |x)∥π 0 (a 1 |x)) + η -1 D KL (π 2 (a 1 |x)∥π 0 (a 1 |x)).For each t ∈ [T ], the suboptimality for the max-player can be written asJ(π * 1 , π * 2 ) -J(π 1 t , †) =E x∼d0 J(x, π * , π * ) -Ĵ(x, π1 t , π 2 t ) + Ĵ(x, π1 t , π 2 t ) -J(x, π1 t , π 2 t ) ≤E x∼d0 -P (x, π1 t , π 2 t ) + η -1 D KL (π 1 t (•|x)∥π 0 (•|x)) -η -1 D KL ( π 2 t (•|x)∥π 0 (•|x)) + β Γ m t (λ, π1 t , π 2 t ) ≤E x∼d0 -P (x, π1 t , π1 t ) + η -1 D KL (π 1 t (•|x), π 0 (•|x)) -η -1 D KL (π 1 t (•|x), π 0 ) -η -1 D KL (x, π 2 t , π1 t ) + β Γ m t (λ, π1 t , π2 t ) =β Γ m t (λ, π1 t , π2 t ) -η -1 E x∼d0 D KL ( π 2 t (•|x)∥π 1 t (•|x)),where the first inequality uses (21) and J * (x) = J * (x, π * , π * ) = 0, in the second inequality we use the definition of π2 t , and P (x, π, π) = 0 in the last equality. We proceed to connect the empirical bonus with the information ratio. Combining Lemma 8 with a union bound over (P, s) ∈ P × [T ], with probability at least 1 -δ/2, we know that 0.5E xs∼d0,a Γ m t (λ, π 1 , π 2 ) ≤ sup P ∈P |E x [P (x, π 1 , π 2 ) -P (x, π 1 , π 2 )]| λ -T log(2T |P|/δ) m + 1 2 s=1 E xs∼d0,a 1 s ∼π 1 s ,a 2 s ∼ϕ 2 s (P (x s , a 1 s , a 2 s ) -P (x s , a 1 s , a 2 s )) 2 ≤ sup P ∈P |E x [P (x, π 1 , π 2 ) -P (x, π 1 , π 2 )]| 1 2 λ + 1 2 t-1 s=1 E xs∼d0,a 1 s ∼π 1 s ,a 2 s ∼ϕ 2 s (P (x s , a 1 s , a 2 s ) -P (x s , a 1 s , a 2 s )) 2 ≤ sup P ∈P √ 2 • |E x [P (x, π 1 , π 2 ) -P (x, π 1 , π 2 )]| λ + t-1 s=1 E xs∼d0,a 1 s ∼π 1 s ,a 2 s ∼ϕ 2 s (P (x s , a 1 s , a 2 s ) -P (x s , a 1 s , a 2 s )) 2 ≤ √ 2 • Γ t (λ, π1 t , π2 t ).Here the second inequality is because λ = 2T log(2T |P|/δ) m . Putting all together, we prove that with probability at least 1 -δ,J(π * 1 , π * 2 ) -J(π 1 t0 , †) ≤ E x∼d0 3βΓ m t0 (π 1 t0 , π2 t0 ) -η -1 D KL (π 2 t (•|x)∥ π 2 t (•|x)) ≤ 3 √ 2βΓ t0 (λ, π1 t0 , π2 t0 ) -η -1 E x D KL (π 2 t (•|x)∥ π 2 t (•|x)) ≤ 3 2T log(2T |P|/δ) m -η -1 D KL (π 2 t (•|x)∥ π 2 t (•|x)).Setting m = 18T log(2T |P|/δ) ϵ 2 finishes the proof.Recall that in Example 1, we suppose that the reward function can be embedded into a d-dimensional vector space {r(x, a) = ⟨θ, ϕ(x, a)⟩ : θ ∈ R d , ∥θ∥ ≤ B, ∥ϕ(x, a)∥ ≤ 1}. Then, if we define the covariance matrix asΣ t = t-1 s=1 E x∼d0,a 1 ∼π 1 s ,a 2 ∼π 2 s (ϕ(x, a 1 ) -ϕ(x, a 2 )) ⊤ (ϕ(x, a 1 ) -ϕ(x, a 2 )) + λ(1 + e B ) 2 I.By invoking the Lagrange's Mean Value Theorem, we have for any two parameters θ 1 , θ 2 ,P θ1 (x, a 1 , a 2 ) -P θ2 (x, a 1 , a 2 ) = 1 1 + exp(θ ⊤ 1 (ϕ(x, a 2 ) -ϕ(x, a 1 ))) - 1 1 + exp(θ ⊤ 2 (ϕ(x, a 2 ) -ϕ(x, a 1 ))) ≤ (θ 1 -θ 2 ) ⊤ (ϕ(x, a 2 ) -ϕ(x, a 1 )) ,andP θ1 (x, a 1 , a 2 ) -P θ2 (x, a 1 , a 2 ) ≥ 1 1 + e B (θ 1 -θ 2 ) ⊤ (ϕ(x, a 2 ) -ϕ(x, a 1 )) .We use the short-hand notation ϕ(x, π) = E a∼π ϕ(x, a). Then the uncertainty can be bounded byΓ t (λ, π 1 , π 2 ) = sup θ |E x∼d0 [P θ (x, π 1 , π 2 ) -P θ (x, π 1 , π 2 )]| λ + t-1 s=1 E xs∼d0,a 1 s ∼π 1 s ,a 2 s ∼π 2 s (P θ (x s , a 1 s , a 2 s ) -P θ (x s , a 1 s , a 2 s )) 2 ≤ sup θ (θ -θ) ⊤ E x [ϕ(x, π 2 ) -ϕ(x, π 1 )] λ + t-1 s=1 E xs∼d0,a 1 s ∼π 1 s ,a 2 s ∼π 2 s ( 1 1+e B (θ -θ) ⊤ (ϕ(x, π 2 ) -ϕ(x, π 1 )) ) 2 ≤(1 + e B ) sup θ ∥θ -θ∥ Σt ∥ϕ(x, π 1 ) -ϕ(x, π 2 )∥ Σ -1 t (θ -θ) ⊤ Σ t (θ -θ) =(1 + e B )∥ϕ(x, π 1 ) -ϕ(x, π 2 )∥ Σ -1 t .This uncertainty bonus is consistent with that of the reward-based RLHF framework up to some multiplicative factor of regularization parameter [72] and the boundness parameter. Compute Nash equilibrium by calling the Oracle 2:Dπ1 t = argmax π 1 ∈Π min π 2 ∈Π E x∼d 0 ,a 1 ∼π 1 ,a 2 ∼π 2 Pt(x, a 1 , a 2 ) + η -1 log π0(a 1 |x) π 1 (a 1 |x) -η -1 log π0(a 2 |x) π 2 (a 2 |x) ,Exploration with the enhancer: construct bonus Γ m t (λ, π 1 , π 2 ) := sup P ∈P |E x∼d 0 [P (x, π 1 , π 2 ) -Pt(x, π 1 , π 2 )]| λ + 1 m t-Compute enhancer to maximize the uncertainty:EQUATIONi ) ; 9: end for 10: Output: the best policy in (π 1 1:T ) by a validation set.Lemma 2. Under Algorithm 4, given the policy of the main agent π1 t , we consider the version space with β 2 = log(2T |P|/δ)/m:EQUATIONπ1 ))}. Then, with probability at least 1 -δ, we know that argmin π ′ J(π 1 t , π ′ ) ∈ Π t for all t ∈ [T ].Proof. First, sinceπ1 t = argmax π E a∼π(•|x) (1 -Pt (x, π1 t , a)) -ηD KL (π(•|x)∥π 0 (•|x)) ,by using Lemma 9, we have for any policy π ∈ Π,EQUATIONAdditionally, by the definition thatπ 2 t = argmin π ′ J(π 1 t , π ′ ) = argmin π ′ E x P * (x, π1 t , π ′ ) + η -1 D KL (π ′ (•|x)∥π 0 (•|x)) ,we haveE x P * (x, π1 t , π 2 t )+η -1 D KL ( π 2 t (•|x)∥π 0 (•|x)) ≤ E x P * (x, π1 t , π1 t )+η -1 D KL (π 1 t (•|x)∥π 0 (•|x)) , which implies that 0 ≤E x P * (x, π1 t , π1 t ) -P * (x, π1 t , π 2 t ) + η -1 D KL (π 1 t (•|x)∥π 0 (•|x)) -η -1 D KL ( π 2 t (•|x)∥π 0 (•|x)) =E x P * (x, π1 t , π1 t ) -Pt (x, π1 t , π1 t ) -(P * (x, π1 t , π 2 t ) -Pt (x, π1 t , π 2 t )) + Pt (x, π1 t , π1 t ) -Pt (x, π1 t , π 2 t ) + η -1 D KL (π 1 t (•|x)∥π 0 (•|x)) -η -1 D KL ( π 2 t (•|x)∥π 0 (•|x)) ≤β( Γ m t (λ, π1 t , π1 t ) + Γ m t (λ, π1 t , π 2 t )) -η -1 E x D KL ( π 2 t (•|x)∥π 1 t (•|x)),where the last inequality uses ( 21) and ( 23) since πt 1 is the Nash equilibrium of Ĵt . Therefore, we conclude the proof. , we have with probability at least 1 -δ,J( †, π2 t0 ) -J(π * , π * ) ≤ O(ϵ 1/2 -η -1 D KL (π 2 t0 (•|x)∥ π 2 t0 (•|x))).Proof. Under the condition of Theorem 2, we haveJ(π 1 t , †) -J(π 2 t , †) ≤ min π ′ J(π 1 t , π ′ ) -min π ′ J(π 2 t -π1 t , π ′ ) -min π ′ J(π 1 t , π ′ ) ≤ min π ′ E x (π 1 t -π2 t )(a|x) • π ′ (a ′ |x) • J(x, a, a ′ )d(a, a ′ ) ≤(B + 1)E x ∥π 1 t (•|x) -π2 t (•|x)∥ 1 ≤(B + 1) D KL (π 1 t (•|x)∥π 2 t (•|x)) ≤(B + 1) ηβ( Γ m t (λ, π1 t , π2 t ) + Γ m t (λ, π1 t , π1 t )) ≤(B + 1) 2ηβ Γ m t (λ, π1 t , π2 t ),where the second last inequality invokes Lemma 2, and the last inequality holds due to π1 t ∈ Π t and π2 t = argmax π∈Πt Γ m t (λ, π1 t , π). Hence, at time t 0 in Theorem 2, we deduce that the suboptimality for the min-player isJ( †, π2 t0 ) -J(π * , π * ) =J( †, π2 t0 ) -J( †, π1 t0 ) + J( †, π1 t0 ) -J(π * , π * ) =J(π 1 t0 , †) -J(π 2 1 , †) + J(π * , π * ) -J(π 1 t0 , †) ≤(B + 1) 2ηβΓ m t0 (π 1 t0 , π2 t0 ) + 3 2T log(2T |P|/δ) m -η -1 D KL (π 2 t0 (•|x)∥ π 2 t0 (•|x)) ≤O B T log(2T |P|/δ) m 1/4 + T log(2T |P|/δ) m -η -1 D KL (π 2 t0 (•|x)∥ π 2 t0 (•|x)) Setting m = T B 4 log(2T |P|/δ) ϵ 2, we getJ( †, π2 t0 ) -J(π * , π * ) ≤ O(ϵ 1/2 -η -1 D KL (π 2 t0 (•|x)∥ π 2 t0 (•|x))).Lemma 6 (Martingale Exponential Inequalities). Consider a sequence of random functions ξ 1 (Z 1 ), • • • , ξ t (Z t ), . . . with respect to filtration {F t }. We have for any δ ∈ (0, 1) and λ > 0:EQUATIONt ) and Z t = (Z 1 , . . . , Z t ).Proof. See Theorem 13.2 of Zhang [83] for a detailed proof.Lemma 7 (Sion's minimax theorem). Let X be a compact convex subset of a linear topological space and Y a convex subset of a linear topological space. If f : X × Y → R satisfies• for any fixed x ∈ X, f (x, •) is upper semicontinuous and quasi-concave on Y ;• for any fixed y ∈ Y , f (•, y) is lower semicontinuous and quasi-convex on X, then we have min Lemma 8 (Multiplicative Chernoff Bounds). Assume that X ∈ [0, 1] with EX = µ. Then for all ϵ > 0,P Xn ≥ (1 + ϵ)µ ≤ exp -2nµϵ 2 2 + ϵ P Xn ≤ (1 -ϵ)µ ≤ exp -2nµϵ 2 2 .Moreover, for t > 0, we haveP Xn ≥ µ + 2µt n + t 3n ≤ exp(-t).Proof. Refer to the proof of Corollary 2.18 in Zhang [83] .Lemma 9 (Policy optimization error). For any two policies π, π ∈ Π such that support(π) = support(π 0 ) andπ(•|x) = argmax π 1 ∈Π E a∼π 1 (•|x) P (x, a) + η -1 log π 0 (a|x) π 1 (a|x) ,Suppose that the KL divergences between them are finite and well defined. Then, we haveE x∼d0 E π [ P (x, a)] -E π [ P (x, a)] + η -1 D KL (π(•|x)∥π 0 (•|x)) -η -1 D KL (π(•|x)∥π 0 (•|x)) = -η -1 E x∼d0 D KL (π(•|x)∥π(•|x)).Proof. See the proof in Lemma 2.4 of Xiong et al. [72] .• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).We remark that the mixture of the open-source preference dataset and hyper-parameters are mainly tuned for the BT model with > 2000 A100 hours, while the preference model adopts most of them directly. Therefore, we expect that the preference model maybe even better with a more refined hyper-parameter search.We introduce another algorithm achieving pessimism via uncertainty bonus construction, see Appendix C.2.In Appendix C.3, we show that with an improved analysis, Algorithm 1 enjoys a refined coverage condition, similar to the coverage notion in[84].https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1https://github.com/huggingface/trlhttps://huggingface.co/blog/pref-tuninghttps://huggingface.co/datasets/argilla/distilabel-capybara-dpo-7k-binarizedhttps://huggingface.co/datasets/argilla/distilabel-intel-orca-dpo-pairsopenbmb/UltraInteract_pairhttps://github.com/OpenAccess-AI-Collective/axolotl 


Decision: This work makes an interesting contribution to the growing literature on online RLHF by considering a minimax game between two LLMs for RLHF under general preference oracle. All the reviewers agreed that the theoretical results of the paper are sound, and novel. Regarding the experimental part, some reviewers raised concerns that comparing the online RLHF method with the existing offline RLHF method might not be fair. However, the authors have addressed this concern in the follow-up response. Overall, I think this paper makes a useful contribution to the active literature on RLHF. It would be great if the authors could add the details and discussions regarding the experiments in the next version of the paper.

Reviews: review: paper_summary: paper_summary: The authors develop a theoretical framework based on a reverse-KL regularized minimax game and introduce sample-efficient algorithms suitable for both offline and online learning scenarios. Empirical results validate the proposed method, demonstrating its superior performance compared to traditional reward-based models across various tasks.
strength_weakness: strength_weakness: The empirical validation, while promising, is limited in scope. More extensive experiments, including comparisons with a broader range of state-of-the-art RLHF methods, would strengthen the paper.
questions: questions: How can we maintain the consistency of the learning process in the absence of the transitive assumption? For instance, when preferences are conflictive.
limitations: limitations: NA.
paper_summary: paper_summary: The paper considers the problem of RLHF under a general preference model, going beyond the reward-based Bradley-Terry model. In particular, they cast the problem as a KL regularized minimax game between two LLMs, and show that their framework is strictly more general than reward-based RL. They propose algorithms for both offline and iterative online learning under this framework, with guarantees. Finally, they run experiments comparing their work to existing RLHF algorithms like DPO and IPO.
strength_weakness: strength_weakness: 1. The paper should also have tested using $\log(p/(1-p))$ as a target, since that corresponds to the DPO target and an empirical comparison is also made to DPO. Their current target corresponds to the IPO target, as the paper implicitly recognises.
2. The theoretical version of the algorithm is quite standard and while this alone would not be a weakness, the practical version of the algorithm seems quite far from the theoretical version. Most strikingly, we use rejection sampling for the enhancer. I mentioned the latter as a strength due to the novel thought needed for designing the practical algorithm, but it is also a weakness because the theoretical guarantees do not signal much about the practical version due to this.
3. This is my main qualm: The iterative method queries the learned preference model adaptively, so it has a clear adaptive advantage over purely offline methods like DPO. The paper should also compare to iterative methods that use reward-based preference models, such as the one in [1].
4. There is also growing work in generalizing from standard reward based preference models to accommodate partial observability, which form an important special case of a fully general preference model. Works such as [2,3] are worth discussing and comparing to in related work.
5. This is not really a weakness, but some minor typos need to be fixed, such as a dangling [Proof] right after "Theorem 1" and "Theorem 2". Table 3's caption says offline DPO, but it should say offline IPO.

Refs:

1. RLHF Workflow: From Reward Modeling to Online RLHF. Dong et al, 2024.

2. When Your AIs Deceive You: Challenges of Partial Observability in Reinforcement Learning from Human Feedback. Lang et al, 2024.

3. A Theoretical Framework for Partially Observed Reward-States in RLHF. Kausik et al, 2024.
questions: questions: 1. Are their other options that the authors had while choosing a practical enhancer? I'm curious why they ended up choosing rejection sampling.
2. While the importance sampling ratio seems large in practice for the offline method, I'm wondering if there is a way to estimate the coverage coefficient too. This is a low priority question, but I am wondering if you have considered computing this to have a more compelling story.
limitations: limitations: Yes
paper_summary: paper_summary: This paper introduces a two-play game approach to the RLHF problems. Both offline and online algorithms are studied, including a combination of pessimism in offline learning and exploration/exploitation in the online setting. The authors conducted experiments to verify the effectiveness of their algorithm. 
The paper is mostly well-written and clear, yet the experimental verification is not strong enough to support the paper's claims.
strength_weakness: strength_weakness: Mainly on novelty and soundness of empirical study.
questions: questions: Could the authors elaborate more on the experiment details? From the current write-up, it is relatively hard for readers to capture how the training process is in either online or offline settings. Specifically, 
1) how the reward models are trained? In Appendix A.2 the authors mentioned the bradley-terry RM and the oracle RM, is the oracle RM trained through next token prediction and with max length = 1 such that the probability of A is the score?
2) with those reward models, what is the exact policy optimization method used in the experiments (this is especially unclear for the offline setting). 
3) Could the authors explain what are the motivations for the special choices in Section 5. Are there other alternatives?

The empirical results are impressive, however, I have some concerns regarding the fairness in comparisons:
1) the authors choose to compare mainly against offline methods, this seems not supportive enough for the proposed algorithm: could the authors also compare to other online RLHF algorithms? Only through this set of experiments will we be able to draw conclusions on the effectiveness of the proposed method --- rather than the superiority of online approaches over offline.
2) I reckon the implementation and reproduction of existing algorithms is not easy --- even only changing the dataset. Nonetheless, I would be keen to see some experiments regarding on how much effort would be needed in getting a well-performing algorithm. In other words, would it be easier to perform hyper-param sweeping for the proposed method than IPO / DPO (and their online alternatives)?

Ablation studies are required to draw conclusions of each element in the authors' empirical designs. For instance, the effectiveness of the pessimism objective, the necessity of each agent in the optimization problem (i.e., the min-player and its exploration.)

Regarding the novelty, could the author contrast the following literature (most of those are cited), and distinguish their contribution? I use [ - ] to denote their main message that is related to this paper:

[online RLHF is better than offline] https://arxiv.org/abs/2402.04792
[Nash learning from direct preference] https://arxiv.org/abs/2312.00886
[general preference optimization] https://arxiv.org/pdf/2402.05749
[self-play in RLHF] https://arxiv.org/abs/2401.01335
limitations: limitations: Please refer to the question section. My current score is mainly based on the soundness of the experiments.
paper_summary: paper_summary: This paper explores RLHF with a general preference model. Specifically, the authors formulate a learning objective aimed at identifying a policy consistently preferred by the KL-regularized preference oracle. Furthermore, the authors propose a sample-efficient training algorithm applicable to both online and offline learning scenarios. The empirical studies conducted validate the effectiveness of the proposed framework.
strength_weakness: strength_weakness: 1.	It would be beneficial if the authors could include examples of the prompts used for training the preference model, as well as responses generated by different methods. This addition would offer readers a clearer understanding of the practical application and comparative effectiveness of the proposed approach.

2.	The reproducibility of the proposed method is not fully clear. Providing a detailed description of the entire framework or algorithm would greatly assist in understanding and replicating the research findings.
questions: questions: 1.	Is Assumption 1 considered mild in real-world applications?

2.	How does the preference model outperform the Bradley-Terry (BT) model, given that their training objectives in Equations 2 and 7 appear to be similar?

3.	Why does the proposed method outperform the baseline in the offline setting despite suffering from a severe overoptimization problem?
limitations: limitations: Yes

Comments: title: Raising score
comment: Thank you for your reply and for raising the score! We are happy to see that our response has addressed your concerns. Thanks again for your constructive feedback.
The authors have addressed my concerns on an already excellently written paper. I am raising my score to a 7. I believe that this paper fully deserves to be accepted.
Thank you for your reply and increasing your score! We are happy to see that our response addressed your concerns. Thanks again for your valuable feedback.
Thank you for your response. My concerns have been addressed. Accordingly, I've increased my score.
Thank you for your reply and for increasing your rating!  We will integrate these discussions into the next revision.
I appreciate the authors' detailed responses. I would encourage the authors to include those discussions in the revision of their paper.

I've increased my score accordingly.