Title: Is RLHF More Difficult than Standard RL? A Theoretical Perspective
Venue: NeurIPS 2023
Accepted: yes
Keywords: reinforcement learning theory

reinforcement learning from human feedback

preference-based reinforcement learning
Abstract: Reinforcement learning from Human Feedback (RLHF) learns from preference signals, while standard Reinforcement Learning (RL) directly learns from reward signals. Preferences arguably contain less information than rewards, which makes preference-based RL seemingly more difficult. This paper theoretically proves that, for a wide range of preference models, we can solve preference-based RL directly using existing algorithms and techniques for reward-based RL, with small or no extra costs. Specifically, (1) for preferences that are drawn from reward-based probabilistic models, we reduce the problem to robust reward-based RL that can tolerate small errors in rewards; (2) for general arbitrary preferences where the objective is to find the von Neumann winner, we reduce the problem to multiagent reward-based RL which finds Nash equilibria for factored Markov games under a restricted set of policies. The latter case can be further reduce to adversarial MDP when preferences only depend on the final state. We instantiate all reward-based RL subroutines by concrete provable algorithms, and apply our theory to a large class of models including tabular MDPs and MDPs with generic function approximation. We further provide guarantees when K-wise comparisons are available.

Is RLHF More Difficult than Standard RL? A Theoretical Perspective

Abstract

Reinforcement learning from Human Feedback (RLHF) learns from preference signals, while standard Reinforcement Learning (RL) directly learns from reward signals. Preferences arguably contain less information than rewards, which makes preference-based RL seemingly more difficult. This paper theoretically proves that, for a wide range of preference models, we can solve preference-based RL directly using existing algorithms and techniques for reward-based RL, with small or no extra costs. Specifically, (1) for preferences that are drawn from reward-based probabilistic models, we reduce the problem to robust reward-based RL that can tolerate small errors in rewards; (2) for general arbitrary preferences where the objective is to find the von Neumann winner, we reduce the problem to multiagent reward-based RL which finds Nash equilibria for factored Markov games under a restricted set of policies. The latter case can be further reduced to adversarial MDP when preferences only depend on the final state. We instantiate all reward-based RL subroutines by concrete provable algorithms, and apply our theory to a large class of models including tabular MDPs and MDPs with generic function approximation. We further provide guarantees when K-wise comparisons are available.

1 Introduction

Reinforcement learning (RL) is a control-theoretic problem in which agents take a sequence of actions, receive reward feedback from the environment, and aim to find good policies that maximize the cumulative rewards. The reward labels can be objective measures of success (winning in a game of Go [Silver et al., 2017]) or more often hand-designed measures of progress (gaining gold in DOTA2 [Berner et al., 2019]). The empirical success of RL in various domains [Mnih et al., 2013, Vinyals et al., 2019, Todorov et al., 2012] crucially relies on the availability and quality of reward signals. However, this also presents a limitation for applying standard reinforcement learning when designing a good reward function is difficult.

An important approach that addresses this challenge is reinforcement learning from Human feedback (RLHF), where RL agents learn from preference feedback provided by humans. Preference feedback is arguably more intuitive to human users, more aligned with human values and easier to solicit in applications such as recommendation systems and image generation [Pereira et al., 2019, Lee et al., 2023]. Empirically, RLHF is a key ingredient empowering the successes in tasks ranging from robotics [Jain et al., 2013] to large language models [Ouyang et al., 2022].

A simple observation about preference feedback is that preferences can always be reconstructed from reward signals. In other words, preference feedback contains arguably less information than scalar rewards, which may render the RLHF problem more challenging. A natural question to ask is:

Is preference-based RL more difficult than reward-based RL?

Existing research on preference-based RL [Chen et al., 2022, Pacchiano et al., 2021, Novoseller et al., 2020, Xu et al., 2020, Zhu et al., 2023] has established efficient guarantees for learning a near-optimal policy from preference feedback. These works typically develop specialized algorithms and analysis in a white-box fashion, instead of building on existing techniques in standard RL. This leaves it open whether it is necessary to develop new theoretical foundation for preference-based RL in parallel to standard reward-based RL.

This work presents a comprehensive set of results on provably efficient RLHF under a wide range of preference models. We develop new simple reduction-based approaches that solve preference-based RL by reducing it to existing frameworks in reward-based RL, with little to no additional cost:

For utility-based preferences-those drawn from a reward-based probabilistic model (see Section 2.1), we prove a reduction from preference-based RL to reward-based RL with robustness guarantees via a new Preference-to-Reward Interface (P2R, Algorithm 1). Our approach incurs no sample complexity overhead and the human query complexity 1 does not scale with the sample complexity of the RL algorithm. We instantiate our framework for comparisons based on (1) immediate reward of the current state-action pair or (2) cumulative reward of the trajectory, and apply existing reward-based RL algorithms to directly find near-optimal policies for RLHF in a large class of models including tabular MDPs, linear MDPs, and MDPs with low Bellman-Eluder dimension, etc. We further provide complexity guarantees when K-wise comparisons are available.

For general (arbitrary) preferences, we consider the objective of the von Neumann winner [see, e.g., Dudík et al., 2015, Kreweras, 1965], a solution concept that always exists and extends the Condorcet winner. We reduce this problem to multiagent reward-based RL which finds Nash equilibria for a special class of factored two-player Markov games under a restricted set of policies. When preferences only depend on the final state, we prove that such factored Markov games can be solved by both players running Adversarial Markov Decision Processes (AMDP) algorithms independently. For preferences that depend on entire trajectory, we develop an adapted version of optimistic Maximum Likelihood Estimation (OMLE) algorithm [Liu et al., 2022a], which handles this factored Markov games under general function approximation.

Notably, our algorithmic solutions are either reductions to standard reward-based RL problems or adaptations of existing algorithms (OMLE). This suggests that technically, preference feedback is not difficult to address given the existing knowledge of RL with reward feedback. Nevertheless, our impossibility results for utility-based preference (Lemma 2, 3) and reduction for general preferences also highlight several important conceptual differences between RLHF and standard RL.

1.1 Related work

Dueling bandits. Dueling bandits [Yue et al., 2012, Bengs et al., 2021] can be seen as a special case of preference based RL with H = 1 . Many assumptions later applied to preference based RL, such as an underlying utility model with a link function [Yue and Joachims, 2009], the Plackett-Luce model [Saha and Gopalan, 2019], and the Condorcet winner [Zoghi et al., 2014] can be traced back to literature on dueling bandits. A reduction from preference feedback to reward feedback for bandits is proposed by Ailon et al. [2014]. The concept of the von Neumann winner, which we employ for general preferences, has been considered in Dudík et al. [2015] for contextual dueling bandits.

RL from human feedback. Using human preferences in RL has been studied for at least a decade [Jain et al., 2013, Busa-Fekete et al., 2014] and is later incorporated with Deep RL [Christiano et al., 2017]. It has found empirical success in robotics [Jain et al., 2013, Abramson et al., 2022, Ding et al., 2023], game playing [Ibarz et al., 2018] and fine-tuning large language models [Ziegler et al., 2019, Ouyang et al., 2022, Bai et al., 2022]. RL with other forms of human feedback, such as demonstrations and scalar ratings, has also been considered in previous research [Finn et al., 2016, Warnell et al., 2018, Arakawa et al., 2018] but falls beyond the scope of this paper.

Theory of Preference-based RL. For utility-based preferences, Novoseller et al. [2020], Pacchiano et al. [2021] and Zhan et al. [2023b] consider tabular or linear MDPs, and assume that the per-

trajectory reward is linear in trajectory features. Xu et al. [2020] also considers the tabular setting. However, instead of assuming an explicit link function, several structural properties of the preference is assumed which guarantee a Condorcet winner and ensure small regret of a black-box dueling bandit algorithm. Finally, recent work of Zhu et al. [2023], Zhan et al. [2023a] consider utility-based preferences in the offline setting, assuming the algorithm is provided with a pre-collected human preference (and transition) dataset with good coverage. Compared to the above works, this paper considers the online setting and derives results for utility-based preferences with general function approximation, which are significantly more general than those for tabular or linear MDPs.

For general preferences, Chen et al. [2022] also develops sample-efficient algorithms for finding the von Neumann winner 2 . Their algorithm is computationally inefficient in general even when restricted to the tabular setting. Compared to this result, our AMDP-based reduction algorithm (Section 4.2) is computationally efficient in the tabular or linear setting when the comparison only depends on the final states. For general trajectory-based comparison, our results apply to a richer class of RL problems including POMDPs.

Finally, we remark that all prior results develop specialized algorithms and analysis for preferencebased RL in a white-box fashion. In contrast, we develop reduction-based algorithms which can directly utilize state-of-the-art results in reward-based RL for preference-based RL. This reduction approach enables the significant generality of the results in this paper compared to prior works.

2 Preliminaries

We consider reinforcement learning in episodic MDPs, specified by a tuple ( H, S , A , P ) . Here S is the state space, A is the action space, and H is the length of each episode. P is the transition probability function; for each h ∈ [ H ] and s, a ∈ S × A , P h ( ·| s, a ) specifies the distribution of the next state. A trajectory τ ∈ ( S × A ) H is a sequence of interactions ( s 1 , a 1 , · · · , s H , a H ) with the MDP. A Markov policy π = { π h : S → ∆ A } h ∈ [ H ] specifies an action distribution based on the current state, while a general policy π = { π h : ( S × A ) h -1 ×S → ∆ A } h ∈ [ H ] can choose a random action based on the whole history up to timestep h .

In RLHF, an algorithm interacts with a reward-less MDP environment and may query comparison oracle (human evaluators) for preference information. We consider two types of preferences: utilitybased ones and general ones.

2.1 Utility-based preferences

For utility-based comparison, we assume there exists an underlying reward function r ⋆ : ( S×A ) H → [0 , 1] . Given a reward function, the value of a policy can be defined as E π [ ∑ H h =1 r ⋆ ( s h , a h )] , i.e. the expected cumulative reward obtained by executing the policy. An optimal policy is one that maximizes E π [ ∑ H h =1 r ⋆ ( s h , a h )] . We say that π is ϵ -optimal if E π [ ∑ H h =1 r ⋆ ( s h , a h )] ≥ max π ′ E π ′ [ ∑ H h =1 r ⋆ ( s h , a h )] -ϵ . We also consider a setting where the utility is only available for a whole trajectory. In this case, we assume that there is an underlying trajectory reward function r ⋆ : ( S × A ) H → [0 , H ] , which assigns a scalar utility to each trajectory. In this case the value of a policy can be defined similarly as E τ ∼ π [ r ( τ )] .

In preference based RL, the reward is assumed to be unobservable, but is respected by the comparison oracle which models human evaluators.

Definition 1 (Comparison oracle) . A comparison oracle takes in two trajectories τ 1 , τ 2 and returns

where σ ( · ) is a link function, and r ⋆ ( · ) is the underlying reward function.

Here Ber ( p ) denotes a Bernoulli distribution with mean p . The comparison outcome o = 1 indicates τ 1 ≻ τ 2 , and vice versa. We additionally require that the inputs τ 1 , τ 2 to the comparison oracle are feasible in the sense that they should be actual trajectories generated by the algorithm and cannot be synthesized artificially. This is motivated by the potential difficulty of asking human evaluators to compare out-of-distribution samples (e.g. random pixels).

In this work, we also consider ( s, a ) preferences , where Pr[( s 1 , a 1 ) ≻ ( s 2 , a 2 )] = σ ( r ( s 1 , a 1 ) -r ( s 2 , a 2 )) . For notational simplicity, we will use τ to denote a state-action pair ( s, a ) (which can be thought as an incomplete trajectory) so that ( s, a ) preferences can be seen as a special case of the comparison oracle (Definition 1). See more details in Remark 1.

2.2 General preferences

For general preferences, we assume that for every trajectory pair τ, τ ′ , the probability that a human evaluator prefers s over s ′ is

A general preference may not be realizable by a utility model, so we cannot define the optimal policy in the usual sense. Instead, we follow Dudík et al. [2015] and consider an alternative solution concept, the von Neumann winner (see Definition 5).

2.3 Function approximation

We first introduce the concept of eluder dimension, which has been widely used in RL to measure the difficulty of function approximation.

Definition 2 (eluder dimension) . For any function class F ⊆ ( X → R ) , its Eluder dimension dim E ( F , ϵ ) is defined as the length of the longest sequence { x 1 , x 2 , . . . , x n } ⊆ X such that there exists ϵ ′ ≥ ϵ so that for all i ∈ [ n ] , x i is ϵ ′ -independent of its prefix sequence { x 1 , . . . , x i -1 } , in the sense that there exists some f i , g i ∈ F such that

Intuitively, eluder dimension measures the number of worst-case mistakes one has to make in order to identify an unknown function from the class F . It is often used as a sufficient condition to prove sample efficiency guarantees for optimism-based algorithms.

As in many works on function approximation, we assume knowledge a class of reward functions R and realizability.

Assumption 1 (Realizability) . r ⋆ ∈ R .

We further use R := { r + c | c ∈ [ -H, 0] , r ∈ R} to denote the reward function class augmented with a bias term. The inclusion of an additional bias is due to the assumption that preference feedback is based on reward differences , so we could only learn r ⋆ up to a constant. We note that for the d -dimension linear reward class R linear , dim E ( R linear ) ≤ ˜ O ( d ) , and that for a general reward class R , dim E ( R , ϵ ) ≤ O ( H dim E ( R , ϵ/ 2) 1 . 5 /ϵ ) . The details can be found in Appendix B.

3 Utility-based RLHF

Given access to comparison oracles instead of scalar rewards, a natural idea is to convert preferences back to scalar reward signals, so that standard RL algorithms can be trained on top of them. In this section, we introduce an efficient reduction from RLHF to standard RL through a preference-toreward interface. On a high level, the interface provides approximate reward labels for standard RL training, and only queries the comparison oracle when uncertainty is large. The reduction incurs small sample complexity overhead, and the number of queries to human evaluators does not scale with the sample complexity of the RL algorithm using it. Moreover, it solicits feedback from the comparison oracle in a limited number of batches, simplifying the training schedule.

3.1 A preference-to-reward interface

The interaction protocol of an RL algorithm with the Preference-to-Reward (P2R) Interface is shown in Fig. 1. P2R maintains a confidence set of rewards B r . When the RL algorithm wishes to learn the reward label of a trajectory τ , P2R checks whether B r approximately agrees on the reward of τ . If so, it can return a reward label with no queries to the comparison oracle; if not, it will query the comparison oracle on τ and a fixed trajectory τ 0 , and update the confidence set of reward functions. The details of P2R are presented in Algorithm 1.

Algorithm 1 Preference-to-Reward (P2R) Interface

1: B r ←R , D ← {} , D hist ←{}

2: Execute the random policy to collect τ 0

3: Upon query of trajectory τ :

4: if (ˆ r, τ ) ∈ D hist then

5: Return ˆ r

6: if max r,r ′ ∈B r ( r ( τ ) -r ( τ 0 )) -( r ′ ( τ ) -r ′ ( τ 0 )) < 2 ϵ 0 then

7: ˆ r ← r ( τ ) -r ( τ 0 ) for an arbitrary r ∈ B r

8: D hist ←D hist ∪ (ˆ r, τ )

9: else

10: Query comparison oracle m times on τ and τ 0 ; compute average comparison result ¯ o

11: ˆ r ← argmin x ∈ [ -H,H ] | σ ( x ) -¯ o | , D ← D ∪ (ˆ r, τ ) , D hist ←D hist ∪ (ˆ r, τ )

12: Update B r

13: Return ˆ r

(𝑠, 𝑎)

𝜏

𝑠′

𝜏, 𝜏 0

𝑜

Ƹ 𝑟

Preference-to-Reward

Interface

Reward-less

MDP

Comparison

Oracle

Optional

RL

Algorithm

The performance of P2R depends on the RL algorithm running on top of it. In particular, we define sample complexity of a standard reward-based RL algorithm A as follows.

Definition 3. An RL algorithm A is g ( ϵ ) -robust and has sample complexity C ( ϵ, δ ) if it can output an ϵ -optimal policy using C ( ϵ, δ ) samples with probability at least 1 -δ , even if the reward of each trajectory τ is perturbed by ε ( τ ) with ∥ ε ( τ ) ∥ ∞ ≤ g ( ϵ ) .

We would like to note that the requirement of being g ( ϵ ) -robust is typically not restrictive. In fact, any tabular RL algorithm with sample complexity would be O ( ϵ/H ) -robust with the same sample complexity, while many algorithms with linear function approximation are O ( ϵ/ poly( d, H )) -robust [Jin et al., 2020b, Zanette et al., 2020] (again with the same sample complexity). We can further show that one can effortlessly convert any standard RL algorithms with sample complexity C into an O (1 / C ) -robust one by using the procedure described in Lemma B.3.

Remark 1 (Trajectory vs. ( s, a ) preferences) . So far, we presented the comparison oracle (Definition 1) and the P2R (Algorithm 1) using trajectory rewards. This would require the RL algorithm using P2R to be one that learns from once-per-trajectory scalar reward. To be compatible with standard RL algorithms, we can formally set τ as an 'incomplete trajectory' ( s, a ) in both Definition 1 and Algorithm 1. This would not change the theoretical results regarding the P2R reduction.

Theoretical analysis of P2R We assume that σ ( · ) is known and satisfies the following regularity assumption.

Assumption 2 is common in bandits literature [Li et al., 2017] and is satisfied by the popular BradleyTerry model, where σ is the logistic function. It is further motivated by Lemma 2 and Lemma 3: when σ is unknown or has no gradient lower bounds, the optimal policy can be impossible to identify.

Lemma 2. If σ is unknown, even if there are only two possible candidates, the optimal policy is indeterminate.

Lemma 3. If σ ′ ( · ) is not lower bounded, for instance when σ ( x ) = 1 2 (1 + sign ( x )) , the optimal policy is indeterminate.

P2R enjoys the following theoretical guarantee when we choose ϵ 0 = g ( ϵ ) / 2 , β = ϵ 2 0 4 , d R = dim E ( R , ϵ 0 ) and m = Θ ( d R ln( d R /δ ) ϵ 2 0 α 2 ) .

Theorem 4. Suppose that Assumption 1 and 2 hold, and that A is an g ( ϵ ) -robust RL algorithm with sample complexity C ( ϵ, δ ) . By running A with the interface in Algorithm 1, we can learn an ϵ -optimal policy using C ( ϵ, δ ) samples and ˜ O ( d 2 R α 2 g ( ϵ ) 2 ) queries to the comparison oracle with probability 1 -2 δ .

The full proof of Theorem 4 is deferred to Appendix B.3. The key idea of the analysis is to use the fact that r ⋆ ∈ B r and the condition in Line 6 to show that the returned reward labels ˆ r are approximate accurate, and to use properties of eluder dimension to bound the number of samples for which the comparison oracle is called.

Theorem 4 shows that P2R is a provably efficient reduction: any standard RL algorithm A combined with P2R induces a provably efficient algorithm that learns an approximately optimal policy from preference feedback. The number of required interactions with the MDP environment is identical to that of the standard RL algorithm, and the query complexity only scales with the complexity of learning the reward function.

Comparison with other approaches A more straightforward reduction would be extensively querying the comparison oracle for every sample generated by the RL algorithm. While this direct reduction would not suffer from increased sample complexity, it encounters two other drawbacks: (1) the oracle complexity, or the total workload for human evaluators, increases proportionally with the sample complexity of the RL algorithm, which can be prohibitive; (2) the RL training would need to pause and wait for human feedback at every iteration, creating substantial scheduling overhead.

Another method to construct reward feedback is to learn a full reward function directly before running the RL algorithm, as in Ouyang et al. [2022]. However, without pre-existing high-quality offline datasets, collecting the samples for reward learning would require solving an exploration problem at least as hard as RL itself [Jin et al., 2020a], resulting in significant sample complexity overhead. In P2R, the exploration problem is solved by the RL algorithm using it.

Compared to the two alternative approaches, our reduction achieves the best of both worlds by avoiding sample complexity overhead with a query complexity that does not scale with the sample complexity.

3.2 Instantiations of P2R

When combined with existing sample complexity results, Theorem 4 directly implies concrete sample and query complexity bounds for preference-based RL in many settings, with no statistical and small computational overhead.

( s, a ) preferences We first consider the comparison that is based on the immediate reward of the current state-action pair. Here we give tabular MDPs and MDPs with low Bellman-Eluder dimension [Jin et al., 2021] as two examples.

Example 1 (Tabular MDPs) . Our first example is tabular MDPs whose state and action spaces are finite and small. In this case d R = ˜ O ( |S||A| ) . The UCBVI-BF algorithm, proposed by Azar et al. [2017], is a model-based tabular RL algorithm which uses upper-confidence bound value iteration with Bernstein-Freedman bonuses. UCBVI-BF has sample complexity C ( ϵ, δ ) = O ( H 3 |S||A| /ϵ 2 ) and is O ( ϵ/H ) robust due to Lemma B.4.

Proposition 5. Algorithm 1 with UCBVI-BF learns an ϵ -optimal policy of a tabular MDP from preference feedback using ˜ O ( H 3 |S||A| ϵ 2 ) episodes of interaction with the environment and ˜ O ( H 2 |S| 2 |A| 2 α 2 ϵ 2 ) queries to the comparison oracle. The algorithm is computationally efficient.

Example 2 (Low Bellman-eluder dimension) . Bellman-eluder dimension [Jin et al., 2021] is a complexity measure for function approximation RL with a Q-value function class F . It can be

shown that a large class of RL problems, including tabular MDPs, linear MDPs, reactive POMDPs and low Bellman rank problems, all have small Bellman-eluder dimension. Furthermore, Jin et al. [2021] designed an algorithm, named GOLF, which (i) first constructs a confidence set for the optimal Q-functions by including all the candidates with small temporal difference loss, (ii) then optimistically picks Q-estimate from the confidence set and executes its greedy policy, and (iii) repeats (i) and (ii) using the newly collected data. Assuming that F satisfies realizability and completeness property, GOLF is g ( ϵ ) = Θ ( ϵ √ d BE H 2 ) -robust with sample complexity C ( ϵ, δ ) = ˜ O ( d BE H 4 ln |F| ϵ 2 ) where d BE is the Bellman-eluder dimension of the problem. By applying Theorem 4, we immediately have the following result.

Proposition 6. Algorithm 1 with GOLF [Jin et al., 2021] learns an ϵ -optimal policy of RL problems with Bellman-Eluder dimension d BE in ˜ O ( d BE H 4 ln |F| ϵ 2 ) episodes of interaction with the environment and ˜ O ( d BE d 2 R H 2 α 2 ϵ 2 ) queries to the comparison oracle.

Trajectory-based preferences When the reward function is trajectory-based, we can instantiate P2R with the OMLE algorithm [Liu et al., 2022a] to solve any model-based RLHF of low generalized eluder dimension. In brief, OMLE is an optimism-based algorithm that maintains a model confidence set. This set comprises model candidates that demonstrate high log-likelihood on previously collected data. And in each iteration, the algorithm chooses a model estimate optimistically and executes its greedy policy to collect new data.

Example 3 (Low generalized eluder dimension) . Generalized eluder dimension [Liu et al., 2022a] is a complexity measure for function approximation RL with a transition function class P . In Appendix D.1, we show that a simple adaptation of OMLE is g ( ϵ ) = Θ( ϵ √ d R ) -robust with sample complexity C ( ϵ, δ ) = ˜ O ( H 2 d P | Π exp | 2 ln |P| ϵ 2 + Hd R | Π exp | ϵ ) , where d P denotes the generalized eluder dimension of P , | Π exp | is a parameter in the OMLE algorithm, and d R = dim E ( R , ϵ ) . Plugging it back into Theorem 4, we obtain the following result.

Proposition 7. Algorithm 1 with OMLE learns an ϵ -optimal policy of RL problems with low generalized eluder dimension in ˜ O ( H 2 d P | Π exp | 2 ln |P| ϵ 2 + Hd R | Π exp | ϵ ) episodes of interaction with the environment and O ( d R d 2 R 2 2 ) queries to the comparison oracle.

˜ α ϵ

Liu et al. [2022a] prove that a wide range of model-based reinforcement learning problems have a low generalized eluder dimension d P and only require a mild | Π exp | to run the OMLE algorithm. Examples of such problems include tabular MDPs, factored MDPs, observable POMDPs, and decodable POMDPs. For a formal definition of generalized eluder dimension and more details on the aforementioned bound and examples, we refer interested readers to Appendix D.1 or Liu et al. [2022a]. Finally, we remark that it is possible to apply the P2R framework in other settings with different complexity measures, such as DEC [Foster et al., 2021] or GEC [Zhong et al., 2022], by making some minor modifications to the corresponding algorithms to ensure robustness.

3.3 Extension to K -wise comparison

In this subsection, we briefly discuss how our results can be extended to K -wise comparison assuming the following Plackett-Luce (PL) model [Luce, 1959, Plackett, 1975] of K item preferences.

Definition 4 (Plackett-Luce model) . The oracle takes in K trajectories τ 1 , . . . , τ K and outputs a permutation ϕ : [ K ] → [ K ] with probability P ( ϕ ) = ∏ K k =1 exp( η · r ( τ ϕ -1 ( k ) )) ∑ K t = k exp( η · r ( τ ϕ -1 ( t ) )) .

Note that when K = 2 , the above PL model reduces to a pair-wise trajectory-type comparison oracle (Definition 1) with σ ( x ) = exp( ηx ) / (exp( ηx ) + 1) which satisfies Assumption 2 with α = Θ( η exp( -ηH )) . The PL model satisfies the following useful property, which is a corollary of its internal consistency [Hunter, 2004].

Property 8 (Hunter [2004, p396]) . For any disjoint sets { i 1 , j 1 } , . . . , { i k , j k } ⊆ [ K ] , the following pair-wise comparisons are mutually independent : 1 ( ϕ ( i 1 ) > ϕ ( j 1 )) , . . . , 1 ( ϕ ( i k ) > ϕ ( j k )) where ϕ is a permutation sampled from PL ( τ 1 , . . . , τ K ) . Moreover, 1 ( ϕ ( i m ) < ϕ ( j m )) ∼ Ber ( σ ( r ( τ i m ) -r ( τ j m ))) , where σ ( x ) = exp( ηx ) / (exp( ηx ) + 1) .

This property enables 'batch querying' the preferences on ⌊ K/ 2 ⌋ pairs of ( τ 1 , τ 2 ) in parallel, which returns ⌊ K/ 2 ⌋ independent pairwise comparisons outcomes. This would enable us to reduce the number of queries by a factor of Ω( K ) for small K in both Algorithm 1 and 3.

Theorem 9 (P2R with K -wise comparison) . Suppose A is an g ( ϵ ) -robust RL algorithm with sample complexity C ( ϵ, δ ) , and assume the same conditions and the same choice of parameters as in Theorem 4. By running A with the interface in Algorithm 1, we can learn an ϵ -optimal policy using C ( ϵ, δ ) samples and ˜ O ( d 2 R α 2 g ( ϵ ) 2 min { K,m } ) queries to the K -wise comparison oracle with probability 1 -2 δ .

Theorem 9 is a direct consequence of Theorem 4: If K ≥ 2 m , we can obtain m independent comparisons between two trajectories by a single query to the K -wise comparison oracle and therefore reduce the overall query complexity in Theorem 4 by a factor of m ; otherwise, we can get m independent comparisons by making O ( m/K ) queries to the K -wise comparison oracle, which reduces the overall query complexity by a factor of K .

However, the above parallelization benefits of using K -wise comparison might be an artifact of the PL model: it seems improbable that the same human evaluator would independently rank ⌊ K/ 2 ⌋ copies of item A and item B . It remains an interesting problem to develop K -wise comparison models more suitable to RLHF.

4 RLHF From General Preferences

The utility-based approach imposes strong assumptions on human preferences. Not only is the matrix M [ τ, τ ′ ] in (1) assumed to be exactly realizable by σ ( r ( τ ) -r ( τ ′ )) , but also σ is assumed to be known and have a gradient lower bound. Moreover, the utility-based approach assumes that transitivity : if Pr[ τ 1 ≻ τ 2 ] ≥ 0 . 5 , Pr[ τ 2 ≻ τ 3 ] ≥ 0 . 5 , then Pr[ τ 1 ≻ τ 3 ] ≥ 0 . 5 . However, experiments have shown that human preferences can be intransitive [Tversky, 1969]. These limitations of the utility-based approach motivates us to consider general preferences.

A general preference may not be realizable by a utility model, so we cannot define the optimal policy in the usual sense. Instead, we follow Dudík et al. [2015] and consider an alternative solution concept, the von Neumann winner .

Definition 5. π ⋆ is the von Neumann winner policy if ( π ⋆ , π ⋆ ) is a symmetric Nash equilibrium of the constant-sum game: max π min π ′ E τ ∼ π,τ ′ ∼ π ′ M [ τ, τ ′ ] .

The duality gap of the game is defined as

We say that π is an ϵ -approximate von Neumann winner if the duality gap of ( π, π ) is at most ϵ . The von Neumann winner has been studied under the name of maximal lotteries in the context of social choice theory [Kreweras, 1965, Fishburn, 1984]. The von Neumann winner is a natural generalization of the optimal policy concept for non-utility based preferences. It is known that

Intuitively, the von Neumann winner π ⋆ is a randomized policy that 'beats' any other policy π ′ in the sense that E τ ∼ π ⋆ ,τ ′ ∼ π ′ M [ τ, τ ′ ] ≥ 1 / 2 ;

If the utility-based preference model holds and the transitions are deterministic, the von Neumann winner is the optimal policy;

The von Neumann winner is the only solution concept that satisfies population-consistency and composition-consistency in social choice theory [Brandl et al., 2016].

Finding the von Neumann winner seems prima facie quite different from standard RL tasks. However, in this section we will show how finding the von Neumann winner can be reduced to finding the restricted Nash equilibrium in a type of Markov games. For preferences based on the final state, we can further reduce the problem to RL in adversarial MDP.

4.1 A reduction to Markov games

Factorized and independent Markov games (FI-MG). Consider a two-player zero-sum Markov games with state space S = S (1) ×S (2) , action space A (1) and A (2) for each player respectively,

transition kernel { P h } h ∈ [ H ] and reward function r . We say the Markov game is factorized and independent if the transition kernel is factorized:

The above definition implies that the Markov game can be partitioned into two MDPs, where the transition dynamics are controlled separately by each player, and are completely independent of each other. The only source of correlation between the two MDPs arises from the reward function, which is permitted to depend on the joint trajectory from both MDPs. Building on the above factorization structure, we define partial trajectory τ i,h := ( s ( i ) 1 , a ( i ) 1 , . . . , s ( i ) h ) that consists of states of the i -th MDP factor and actions of the i -th player. Furthermore, we define a restricted policy class Π i that contains all policies that map a partial trajectory to a distribution in ∆ A i , i.e.,

And the goal is to learn a restricted Nash equilibrium ( µ ⋆ , ν ⋆ ) ∈ Π 1 × Π 2 such that

Finding von Neumann winner via learning restricted Nash. Weclaim that finding an approximate von Neumann winner can be reduced to learning an approximate restricted Nash equilibrium in a FI-MG. The reduction is straightforward: we simply create a Markov game that consists of two independent copies of the original MDP and control the dynamics in the i -th copy by the i -th player's actions. Such construction is clearly factorized and independent. Moreover, the restricted policy class Π i is equivalent to the universal policy class in the original MDP. We further define the reward function as r ( τ, τ ′ ) = M [ τ, τ ′ ] where M is the general preference function. By definition 5, we immediately obtain the following equivalence relation.

Proposition 10. If ( µ ⋆ , ν ⋆ ) is a restricted Nash equilibrium of the above FI-MG, then both µ ⋆ and ν ⋆ are von Neumann winner in the original problem.

The problem we are faced with now is how to learn restricted Nash equilibria in FI-MG. In the following sections, we present two approaches that leverage existing RL algorithms to solve this problem: (i) when the preference function depends solely on the final states of the two input trajectories, each player can independently execute an adversarial MDP algorithm; (ii) for general preference functions, a straightforward adaptation of the OMLE algorithm is sufficient under certain eluder-type conditions.

4.2 Learning from final-state-based preferences via adversarial MDPs

In this section, we consider a special case where the preference depends solely on the final states of the two input trajectories, i.e., M ( τ, τ ′ ) = M ( s H , s ′ H ) . Given the previous equivalence relation between von Neumann winner and restricted Nash in FI-MG, one natural idea is to apply no-regret learning algorithms, as it is well-known that running two copies of no-regret online learning algorithms against each other can be used to compute Nash equilibria in zero-sum normal-form games. Since this paper focuses on sequential decision making, we need no-regret learning algorithms for adversarial MDPs, which we define below.

Adversarial MDPs. In the adversarial MDP problem, the algorithm interacts with a series of MDPs with the same unknown transition but adversarially chosen rewards for each episode. Formally, there exists an unknown groundtruth transition function P = { P h } H h =1 . At the beginning of the k -th episode, the algorithm chooses a policy π k and then the adversary picks a reward function r k = { r k h } H h =1 . After that, the algorithm observes a trajectory τ k = ( s k 1 , a k 1 , y k 1 , . . . , s k H , a k H , y k H ) sampled from executing policy π k in the MDP parameterized by P and r k , where E [ y k h | s k h , a k h ] = r k h ( s k h , a k h ) . We define the regret of an adversarial MDP algorithm A to be the gap between the algorithm's expected payoff and the best payoff achievable by the best fixed Markov policy:

Now we explain how to learn a von Neumann winner via running adversarial MDP algorithms. We simply create two copies of the original MDP and instantiate two adversarial MDP algorithms A 1 and

A 2 to control each of them separately. To execute A 1 and A 2 , we need to provide reward feedback to them in each k -th episode. Denote by s k, (1) H and s k, (2) H the final states A 1 and A 2 observe in the k -th episode. We will feed y k ∼ Ber ( M ( s k, (1) H , s k, (2) H )) into A 1 and 1 -y k into A 2 as their reward at step H -1 , respectively. And all other steps have zero reward feedback.The formal pseudocode is provided in Algorithm 4 (Appendix E). The following theorem states that as long as the invoked adversarial MDP algorithm has sublinear regret, the above scheme learns an approximate von Neumann winner in a sample-efficient manner.

Theorem 11. Suppose Regret K ( A ) ≤ βK 1 -c with probability at least 1 -δ for some c ∈ (0 , 1) . Then Algorithm 4 with K = (4 β/ϵ ) 1 /c outputs an ϵ -approximate von Neumann winner with probability at least 1 -2 δ .

In order to demonstrate the applicability of Theorem 11, we offer two examples where sublinear regret can be achieved in adversarial MDPs via computationally efficient algorithms. The first one is adversarial tabular MDPs where the number of states and actions are finite, i.e., |S| , |A| ≤ + ∞ .

Example 4 (adversarial tabular MDPs) . Jin et al. [2019] proposed an algorithm A with Regret K ( A ) ≤ ˜ O ( √ |S| 2 |A| H 3 K ) . Plugging it into Theorem 11 leads to K = ˜ O ( |S| 2 |A| H 3 /ϵ 2 ) sample complexity and query complexity for learning ϵ -approximate von Neumann winner.

The second example is adversarial linear MDPs where the number of states and actions can be infinitely large while the transition and reward functions admit special linear structure. See Sherman et al. [2023] for the precise formulation of the adversarial linear MDP problem.

Example 5 (adversarial linear MDPs) . Sherman et al. [2023] proposed an algorithm A with Regret K ( A ) ≤ ˜ O ( dH 2 K 6 / 7 ) for online learning in adversarial linear MDPs. 3 Combining it with Theorem 11 leads to K = ˜ O ( d 7 H 14 /ϵ 7 ) sample complexity and query complexity for learning ϵ -approximate restricted Nash equilibria.

4.3 Learning from trajectory-based preferences via OMLE_equilibrium

In this section, we consider the more general case where the preference M [ τ, τ ′ ] is allowed to depend arbitrarily on the two input trajectories. Similar to the utility-based setting, we assume that the learner is provided with a preference class M⊆ (( S × A ) H × ( S × A ) H → [0 , 1]) and transition function class P a priori, which contains the groundtruth preference and transition we are interacting with. Previously, we have established the reduction from learning the von Neumann winner to learning restricted Nash in FI-MG. In addition, learning restricted Nash in FI-MG is in fact a special case of learning Nash equilibrium in partially observable Markov games (POMGs). As a result, we can directly adapt the existing OMLE algorithm for learning Nash in POMGs [Liu et al., 2022b] to our setting, with only minor modifications required to learn the von Neumann winner. We defer the algorithmic details for this approach (Algorithm 5) to Appendix F, and present only the theoretical guarantee here.

Theorem 12. Suppose Assumption 3 holds. There exist absolute constant c 1 and c 2 such that for any ( T, δ ) ∈ N × (0 , 1] if we choose β M = c 1 ln( |M| T/δ ) and β P = c 1 ln( |M| T/δ ) in Algorithm 5, then with probability at least 1 -δ , the duality gap of the output policy of Algorithm 5 is at most

where d M = dim E ( M , 1 /T )

It has been proven that a wide range of RL problems admit a regret formulation of ξ = ˜ O ( √ d P β | Π exp | T ) with mild d P and | Π exp | [Liu et al., 2022a]. These problems include, but are not limited to, tabular MDPs, factored MDPs, linear kernel MDPs, observable POMDPs, and decodable POMDPs. For more details, please refer to Appendix D.1 or Liu et al. [2022a]. For problems that satisfy ξ = ˜ O ( √ d P β P | Π exp | T ) , Theorem D.1 implies a sample complexity of

The sample complexity for specific tractable problems can be derived by plugging their precise formulation of ξ (provided in Appendix D.1) into the above bound.

References

Josh Abramson, Arun Ahuja, Federico Carnevale, Petko Georgiev, Alex Goldin, Alden Hung, Jessica Landon, Jirka Lhotka, Timothy Lillicrap, Alistair Muldal, et al. Improving multimodal interactive agents with reinforcement learning from human feedback. arXiv preprint arXiv:2211.11602 , 2022.

Nir Ailon, Zohar Karnin, and Thorsten Joachims. Reducing dueling bandits to cardinal bandits. In International Conference on Machine Learning , pages 856-864. PMLR, 2014.

Riku Arakawa, Sosuke Kobayashi, Yuya Unno, Yuta Tsuboi, and Shin-ichi Maeda. Dqntamer: Human-in-the-loop reinforcement learning with intractable feedback. arXiv preprint arXiv:1810.11748 , 2018.

Mohammad Gheshlaghi Azar, Ian Osband, and Rémi Munos. Minimax regret bounds for reinforcement learning. In International Conference on Machine Learning , pages 263-272. PMLR, 2017.

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 , 2022.

Viktor Bengs, Róbert Busa-Fekete, Adil El Mesaoudi-Paul, and Eyke Hüllermeier. Preference-based online learning with dueling bandits: A survey. The Journal of Machine Learning Research , 22(1): 278-385, 2021.

Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław D˛ ebiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680 , 2019.

Florian Brandl, Felix Brandt, and Hans Georg Seedig. Consistent probabilistic social choice. Econometrica , 84(5):1839-1880, 2016.

Róbert Busa-Fekete, Balázs Szörényi, Paul Weng, Weiwei Cheng, and Eyke Hüllermeier. Preferencebased reinforcement learning: evolutionary direct policy search using a preference-based racing algorithm. Machine learning , 97:327-351, 2014.

Xiaoyu Chen, Han Zhong, Zhuoran Yang, Zhaoran Wang, and Liwei Wang. Human-in-the-loop: Provably efficient preference-based reinforcement learning with general function approximation. In International Conference on Machine Learning , pages 3773-3793. PMLR, 2022.

Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems , 30, 2017.

Zihan Ding, Yuanpei Chen, Allen Z Ren, Shixiang Shane Gu, Hao Dong, and Chi Jin. Learning a universal human prior for dexterous manipulation from human preference. arXiv preprint arXiv:2304.04602 , 2023.

Miroslav Dudík, Katja Hofmann, Robert E Schapire, Aleksandrs Slivkins, and Masrour Zoghi. Contextual dueling bandits. In Conference on Learning Theory , pages 563-587. PMLR, 2015.

Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International conference on machine learning , pages 49-58. PMLR, 2016.

Peter C Fishburn. Probabilistic social choice based on simple voting comparisons. The Review of Economic Studies , 51(4):683-692, 1984.

Dylan J Foster, Sham M Kakade, Jian Qian, and Alexander Rakhlin. The statistical complexity of interactive decision making. arXiv preprint arXiv:2112.13487 , 2021.

David R Hunter. Mm algorithms for generalized bradley-terry models. The annals of statistics , 32(1): 384-406, 2004.

Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. Advances in neural information processing systems , 31, 2018.

Ashesh Jain, Brian Wojcik, Thorsten Joachims, and Ashutosh Saxena. Learning trajectory preferences for manipulators via iterative improvement. Advances in neural information processing systems , 26, 2013.

Chi Jin, Tiancheng Jin, Haipeng Luo, Suvrit Sra, and Tiancheng Yu. Learning adversarial mdps with bandit feedback and unknown transition. arXiv preprint arXiv:1912.01192 , 2019.

Chi Jin, Akshay Krishnamurthy, Max Simchowitz, and Tiancheng Yu. Reward-free exploration for reinforcement learning. In International Conference on Machine Learning , pages 4870-4879. PMLR, 2020a.

Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory , pages 2137-2143. PMLR, 2020b.

Chi Jin, Qinghua Liu, and Sobhan Miryoosefi. Bellman eluder dimension: New rich classes of rl problems, and sample-efficient algorithms. Advances in neural information processing systems , 34:13406-13418, 2021.

Germain Kreweras. Aggregation of preference orderings. In Mathematics and Social Sciences I: Proceedings of the seminars of Menthon-Saint-Bernard, France (1-27 July 1960) and of Gösing, Austria (3-27 July 1962) , pages 73-79, 1965.

Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint arXiv:2302.12192 , 2023.

Lihong Li, Yu Lu, and Dengyong Zhou. Provably optimal algorithms for generalized linear contextual bandits. In International Conference on Machine Learning , pages 2071-2080. PMLR, 2017.

Qinghua Liu, Praneeth Netrapalli, Csaba Szepesvari, and Chi Jin. Optimistic mle-a generic model-based algorithm for partially observable sequential decision making. arXiv preprint arXiv:2209.14997 , 2022a.

Qinghua Liu, Csaba Szepesvári, and Chi Jin. Sample-efficient reinforcement learning of partially observable markov games. arXiv preprint arXiv:2206.01315 , 2022b.

Robert D Luce. Individual choice behavior: A theoretical analysis . John Willey and sons, 1959.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 , 2013.

Ellen Novoseller, Yibing Wei, Yanan Sui, Yisong Yue, and Joel Burdick. Dueling posterior sampling for preference-based reinforcement learning. In Conference on Uncertainty in Artificial Intelligence , pages 1029-1038. PMLR, 2020.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 35: 27730-27744, 2022.

Aldo Pacchiano, Aadirupa Saha, and Jonathan Lee. Dueling rl: reinforcement learning with trajectory preferences. arXiv preprint arXiv:2111.04850 , 2021.

Bruno L Pereira, Alberto Ueda, Gustavo Penha, Rodrygo LT Santos, and Nivio Ziviani. Online learning to rank for sequential music recommendation. In Proceedings of the 13th ACM Conference on Recommender Systems , pages 237-245, 2019.

Robin L Plackett. The analysis of permutations. Journal of the Royal Statistical Society Series C: Applied Statistics , 24(2):193-202, 1975.

Daniel Russo and Benjamin Van Roy. Eluder dimension and the sample complexity of optimistic exploration. Advances in Neural Information Processing Systems , 26, 2013.

Aadirupa Saha and Aditya Gopalan. Pac battling bandits in the plackett-luce model. In Algorithmic Learning Theory , pages 700-737. PMLR, 2019.

Uri Sherman, Tomer Koren, and Yishay Mansour. Improved regret for efficient online reinforcement learning with linear function approximation. arXiv preprint arXiv:2301.13087 , 2023.

David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature , 550(7676):354-359, 2017.

Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 5026-5033. IEEE, 2012. doi: 10.1109/IROS.2012.6386109.

Amos Tversky. Intransitivity of preferences. Psychological review , 76(1):31, 1969.

Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature , 575(7782):350-354, 2019.

Garrett Warnell, Nicholas Waytowich, Vernon Lawhern, and Peter Stone. Deep tamer: Interactive agent shaping in high-dimensional state spaces. In Proceedings of the AAAI conference on artificial intelligence , volume 32, 2018.

Yichong Xu, Ruosong Wang, Lin Yang, Aarti Singh, and Artur Dubrawski. Preference-based reinforcement learning with finite-time guarantees. Advances in Neural Information Processing Systems , 33:18784-18794, 2020.

Yisong Yue and Thorsten Joachims. Interactively optimizing information retrieval systems as a dueling bandits problem. In Proceedings of the 26th Annual International Conference on Machine Learning , pages 1201-1208, 2009.

Yisong Yue, Josef Broder, Robert Kleinberg, and Thorsten Joachims. The k-armed dueling bandits problem. Journal of Computer and System Sciences , 78(5):1538-1556, 2012.

Andrea Zanette, Alessandro Lazaric, Mykel Kochenderfer, and Emma Brunskill. Learning near optimal policies with low inherent bellman error. In International Conference on Machine Learning , pages 10978-10989. PMLR, 2020.

Wenhao Zhan, Masatoshi Uehara, Nathan Kallus, Jason D Lee, and Wen Sun. Provable offline reinforcement learning with human feedback. arXiv preprint arXiv:2305.14816 , 2023a.

Wenhao Zhan, Masatoshi Uehara, Wen Sun, and Jason D Lee. How to query human feedback efficiently in rl? arXiv preprint arXiv:2305.18505 , 2023b.

Han Zhong, Wei Xiong, Sirui Zheng, Liwei Wang, Zhaoran Wang, Zhuoran Yang, and Tong Zhang. Aposterior sampling framework for interactive decision making. arXiv preprint arXiv:2211.01962 , 2022.

Banghua Zhu, Jiantao Jiao, and Michael I Jordan. Principled reinforcement learning with human feedback from pairwise or k -wise comparisons. arXiv preprint arXiv:2301.11270 , 2023.

Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593 , 2019.

Masrour Zoghi, Shimon Whiteson, Remi Munos, and Maarten Rijke. Relative upper confidence bound for the k-armed dueling bandit problem. In International conference on machine learning , pages 10-18. PMLR, 2014.

A Proofs of Impossibility Results

Proof of Lemma 2. Consider two link functions σ 1 ( x ) := 1 1+exp( -x ) and

where α 1 = 1 , α 2 = -0 . 484 , α 3 = 0 . 3 Consider an MDP with H = 2 and initial state s 0 . Suppose that there are three terminal states s 1 , s 2 , s 3 , where we observe that the trajectory preferences only depend on the terminal state in the following way:

This can be explained by both σ 1 with r (1) = { s 0 : 0 , s 1 : 0 . 847 , s 2 : 0 , s 3 : -1 . 386 } , and σ 2 with r (2) = { s 0 : 0 , s 1 : 0 . 2 , s 2 : 0 , s 3 : -0 . 3 } .

Suppose that state s 0 has two actions a and b , which leads to distributions { 0 . 39 : s 1 , 0 . 61 : s 3 } and { 1 : s 2 } respectively. Under r (1) and r (2) , the optimal action would be a and b respectively. Therefore without knowledge of the link function, it is impossible to identify the optimal policy even with perfect knowledge of the transition and comparison probabilities.

Proof of Lemma 3. Similarly, consider an MDP with H = 2 and initial state s 0 . Suppose that there are three terminal states s 1 , s 2 , s 3 , where we observe that

Meanwhile the state s 0 has two actions a and b , leading to distributions { 0 . 5 : s 1 , 0 . 5 : s 3 } and { 1 : s 2 } respectively. In this case, both r (1) = { s 0 : 0 , s 1 : 0 . 5 , s 2 : 0 , s 3 : -1 } and r (2) = { s 0 : 0 , s 1 : 1 , s 2 : 0 , s 3 : -0 . 5 } fits the comparison results perfectly. However under r (1) , the optimal action is b while under r (2) , the optimal action is a .

B Proofs for P2R

B.1 Properties of eluder dimension

We begin by proving two properties of the eluder dimension of R .

Lemma B.1. Consider R linear := { θ ⊤ x } , where x ∈ X ⊆ R d , ∥ θ ∥ 2 ≤ γ . Then there exists absolute constant C such that

Therefore R linear can be seen as a ( d +1) -dimensional linear function class with parameter norm ∥ [ θ, c ] ∥ 2 ≤ γ + H . The statement is then a direct corollary of Russo and Van Roy [2013, Proposition 6].

Lemma B.2. For a general function class R with domain X and range [0 , H ] , for ϵ < 1 ,

Proof. Suppose that there exists a sequence

where r i ∈ R , c i ∈ [ -H, 0] , x i ∈ X and y i ∈ R , such that for all i ∈ [ m ]

Proof. Note that

By definition, dim E ( R , ϵ ) is the largest m so that such a sequence exists. By the pigeon-hole principle, there exists a subset I ⊆ [ m ] of size at least k = m ⌈ H/ϵ 0 ⌉ and ¯ c ∈ [0 , H -ϵ 0 ] such that ∀ i ∈ I , c i ∈ [¯ c, ¯ c + ϵ 0 ] . Denote the subsequence indexed by I as { r ′ i , c ′ i , x ′ i , y ′ i } i ∈ [ k ] . Define ˜ y i := y ′ i -¯ c . Now consider the sequence { r ′ i , x ′ i , ˜ y i } i ∈ [ k ] . By definition ∀ i ∈ [ k ]

It follows that

We can choose ϵ 0 := ( Hϵ 2 16 m ) 1 / 3 so that ϵ 0 ≤ ϵ/ (4 √ k ) . Then we can guarantee

By Jin et al. [2021, Proposition 43],

B.2 Properties related to robustness

Lemma B.3 (Robustness to perturbation) . Any RL algorithm A with sample complexity C ( ϵ, δ ) can be converted to an algorithm A ′ that is 1 C ( ϵ,δ ) -robust with sample complexity C ( ϵ, δ/ 3) .

Proof. Consider the following modification of A : instead of using reward r directly, we project r to +2 H and -2 H unbiasedly; that is, the algorithm receives the binarized rewards

By the definition of sample complexity, when using samples of b ( r ⋆ ) , A outputs a policy π 0 for r ⋆ that is ϵ -optimal with probability 1 -δ in K := C ( ϵ, δ ) episodes. Denote the trajectories generated by running A on b ( r ⋆ ) by τ 1 , · · · , τ K . Now suppose that for τ k , the reward label is perturbed from b ( r ⋆ ( τ k )) to b ( r ′ k ) with | r ′ k -r ⋆ ( τ k ) | ≤ ϵ ′ := ( C ( ϵ, δ )) -1 ; denote the output policy of A by π ′ . It can be shown that

Therefore the total density ratio

It follows that the density ratio between π and ˆ π is also bounded by e . Therefore the probability that ˆ π is not ϵ -optimal is at most 3 δ . Rescaling δ proves the lemma.

Lemma B.4. Any tabular RL algorithm A with sample complexity C ( ϵ, δ ) is ϵ/ (4 H ) robust with sample complexity C ( ϵ/ 2 , δ ) .

In other words, which gives

Proof. Suppose that A is run on perturbed rewards, where rewards for trajectory τ is changed by ε ( τ ) . By definition, using C ( ϵ/ 2 , δ ) samples and with probability 1 -δ , it outputs an ϵ/ 2 -optimal policy ˆ π with respect to the perturbed reward function r + ε , where ∥ ε ∥ ∞ ≤ ϵ/ 4 H . Denote the value function of policy π with respect to reward r by V π,r . Further denote the optimal value function with respect to r by π ⋆ . It holds that for any policy π ,

In other words ˆ π is indeed an ϵ -optimal policy with respect to the unperturbed rewards r .

B.3 Proof of Theorem 4

Lemma B.5. With m = Θ ( ln(1 /δ ′ ) α 2 ϵ ′ 2 ) , for each τ such that the comparison oracle is queried, with probability 1 -δ ′ ,

Proof. Suppose that the comparison oracle is queried for τ and the average outcome is ¯ o . By Hoeffding bound, with probability 1 -δ ′ ,

Since ˆ r ( τ ) = argmin x ∈ [ -H,H ] | σ ( x ) -¯ o | ,

It follows that

Lemma B.6. Set m = Θ( d ln( d/δ ) ϵ 2 0 α 2 ) and β = ϵ 2 0 4 . With probability 1 -δ , the number of samples on which the comparison oracle is queried is at most dim E ( R , ϵ 0 ) .

When the comparison oracle is queried, max r,r ′ ∈B r ([ r ( τ ) -r ( τ 0 )] -[ r ′ ( τ ) -r ′ ( τ 0 )]) > 2 ϵ 0 which means that either | ˜ r ( τ ) -˜ r ⋆ ( τ ) | > ϵ 0 or | ˜ r ′ ( τ ) -˜ r ⋆ ( τ ) | > ϵ 0 . Suppose that there are K trajectories which require querying comparison oracle. Suppose that the dataset is composed of

and ˜ r 1 , . . . , ˜ r K ∈ R are the functions that satisfy | ˜ r k ( τ k ) -˜ r ⋆ ( τ k ) | > ϵ 0 . We now verify that ( ˜ r 1 , τ 1 , ˜ r 2 , τ 2 , · · · , ˜ r K , τ K ) is an eluder sequence (with respect to function class R ).

The confidence set condition implies

Therefore

With probability 1 -δ , ∀ k ≤ K ∧ 2 d , | ˆ r k -˜ r ⋆ ( τ k ) | ≤ ϵ 0 4 √ d (by Lemma B.5). Then for any i ≤ k

as long as K ≤ 2 d . In other words, with probability 1 -δ , ( ˜ r 1 , τ 1 , ˜ r 2 , τ 2 , · · · , ˜ r K ∧ 2 d , τ K ∧ 2 d ) is an eluder sequence, which by Definition 2 cannot have length more than d := dim E ( R , ϵ 0 ) . It follows that K ≤ dim E ( R , ϵ 0 ) .

Lemma B.7. With probability 1 -δ , r ⋆ ∈ B r throughout the execution of Algorithm 1.

Proof. By Lemma B.5 and Lemma B.6, with probability 1 -δ , at every step

Lemma B.8. With probability 1 -δ , for each τ in Line 3 of Algorithm 1, the returned reward ˆ r satisfies

Proof. We already know that this is true for τ such that the comparison oracle is queried. However, if it is not queried, then

Since r ⋆ ∈ B r (by Lemma B.7), this immediately implies | ˆ r -( r ⋆ ( τ ) -r ⋆ ( τ 0 )) | ≤ 2 ϵ 0 .

By Lemma B.8 (rescaling δ ), with probability 1 -δ , the reward returned by the reward interface is g ( ϵ ) -close to ˜ r ⋆ := r ⋆ -r ⋆ ( τ 0 ) throughout the execution of the algorithm. By the definition of sample complexity, with probability 1 -δ , the policy returned by A is ϵ -optimal for ˜ r ⋆ , which implies that it is also ϵ -optimal for r ⋆ . The number of samples (episodes) is bounded by C ( ϵ, δ ) . Finally by Lemma B.6, the number of queries to the comparison oracle is at most

C P-OMLE: Improved query complexity via white-box modification

While the P2R interface provides a clean and effortless recipe for modifying standard RL algorithms to work with preference feedback, it incurs a large query cost to the comparison oracle, e.g., cubic dependence on d R in Example 3. We believe that this disadvantage is caused by the black-box nature of P2R, and that better query complexities can be achieved by modifying standard RL algorithms in a white-box manner and specialized analysis. In this section, we take OMLE [Liu et al., 2022a] as an example and introduce a standalone adaptation to trajectory preference feedback with improved query complexity. We expect other optimism-based algorithms (e.g., UCBVI-BF and GOLF) can be modified in a similar white-box manner to achieve better query complexity.

The details of the Preference-based OMLE algorithm (P-OMLE) is provided in Algorithm 2. Compared to OMLE with reward feedback (Algorithm 3), the only difference is that now the reward confidence set is computed using preference feedback directly using log-likelihood. Denote by V π r,p the expected cumulative reward the leaner will receive if she follows policy π in model ( p, r ) . In the t -th iteration, the algorithm follows the following steps:

Optimistic planning: Find the policy-model pair ( π, r, p ) that maximizes the value function V π r,p ;

Data collection: Construct an exploration policy set Π exp ( π t ) 4 and collect trajectories by running all policies in Π exp ( π t ) ;

Confidence set update: Update the confidence set using the updated log-likelihood:

Algorithm 2 Preference-based OMLE (P-OMLE)

1

We have the follwoing guarantee for Algorithm 2.

Theorem C.1. Suppose Assumption, 1, 2 and 3 hold. There exists an absolute constant c 1 , c 2 > 0 such that for any ( T, δ ) ∈ N × (0 , 1] if we choose β R = c 1 ln( |R| T/δ ) and β P = c 1 ln( |M| T/δ ) in Algorithm 2, then with probability at least 1 -δ , we have that for all t ∈ [ T ] ,

where d R = dim E ( R , 1 / √ T ) .

For problems that satisfy ξ = ˜ O ( √ d P β P | Π exp | T ) , Theorem C.1 implies both the sample complexity and the query complexity for learning an ϵ -optimal policy is upper bounded by:

Compared to the result derived through the P2R interface (Example 3), the sample complexity here is basically the same while the query complexity has improved dependence on d R (from cubic to linear). Nonetheless, the query complexity here additionally depends on the complexity of learning the transition model, which is not the case in Example 3.

C.1 Proofs for P-OMLE

The proof of Theorem C.1 largely follows from that of Theorem D.1. We first introduce several useful notations. Denote by B t M , B t P the preference, transition confidence set in the t -th iteration, which satisfy B t = B t P ×B t M . Denote the groundtruth transition and preference by p ⋆ and M ⋆ . Denote the trajectories generated when running Algorithm 2 by τ 1 , · · · , τ T .

Similar to Lemma D.3, we have that the confidence set satisfies the following properties.

Lemma C.2. There exists absolute constant c 2 such that under the same condition as Theorem C.1, we have that with probability at least 1 -δ : for all t ∈ [ T ]

( r ⋆ , p ⋆ ) ∈ B t ,

Proof. For the first two statements, see the proof of Theorem 3.2 in Liu et al. [2022a]. For the third bullet point, see Liu et al. [2022a, Proposition B.2], which implies that

Proof of Theorem C.1. By using the first relation in Lemma C.2 and the definition of ( π t , r t , p t ) ,

We can control the first term by the second inequality in Lemma lem:omle-pref

For the second term, by Azuma-Hoeffding inequality and by combining the second inequality in Lemma C.2 with the regret guarantee for eluder dimension,

D OMLEwith Perturbed Reward

D.1 Algorithm details and theoretical guarantees

In this section, we modify the optimistic MLE (OMLE) algorithm [Liu et al., 2022a] to deal with unknown reward functions. The adapted algorithm can then be used with Algorithm 1 as described in Example 3. OMLE is a model-based algorithm that requires a model class P in addition to the reward class R . On a high level, OMLE maintains a joint confidence set B in R×P . In the t -th iteration, the algorithm follows the following steps:

Optimistic planning: Find the policy-model pair ( π, r, p ) that maximizes the value function V π r,p which is the expected cumulative reward the leaner will receive if she follows policy π in a model with transition p and reward r ;

Data collection: Construct an exploration policy set Π exp ( π t ) 5 and collect trajectories by running all policies in Π exp ( π t ) ;

Confidence set update: Update the confidence set using the updated log-likelihood.

The main modification we make is the confidence set of r , since the original OMLE algorithm assumes that r is known. The pseudocode of our adapted OMLE algorithm is provided in Algorithm 3.

Algorithm 3 Optimistic MLE with ϵ ′ -Perturbed Reward Feedback

1: B 1 ←R×P

2: Execute an arbitrary policy to collect trajectory τ 0

3: for t = 1 , . . . , T do

4: Compute ( π t , r t , p t ) = arg max π, ( r,p ) ∈B t V π r,p

5: Execute π t to collect a trajectory τ , receive reward ˆ r , add ( τ, ˆ r ) into D rwd

6: for each π ∈ Π exp ( π t ) do

7: Execute π to collect a trajectory τ , add ( π, τ ) into D trans

8: Update

In Line 8, the log-likelihood function is defined as

where P π p ( τ ) denotes the probability of observing trajectory τ in a model with transition function p under policy π . Other algorithmic parameters T and β P are chosen in the statement of Theorem D.1.

To obtain sample efficiency guarantees for OMLE, the following assumption is needed.

Assumption 3 (Generalized eluder-type condition) . There exists a real number d P ∈ R + and a function ζ such that: for any ( T, ∆) ∈ N × R + , transitions { p t } t ∈ [ T ] and policies { π t } t ∈ [ T ] , we have

Here P π p is the distribution of trajectories under model p and policy π , while | Π exp | := max π | Π exp ( π ) | is the largest possible number of exploration policies.

Assumption 3 shares a similar intuition with the pigeon-hole principle and the elliptical potential lemma, which play important roles in the analysis of tabular MDP and linear bandits respectively. In particular, the ξ function measures the worst-case growth rate of the cumulative error and is the central quantity characterizing the hardness of the problem. Liu et al. [2022a] prove that a wide range of RL problems satisfy Assumption 3 with moderate d P , | Π exp | and ξ = ˜ O ( √ d P ∆ | Π exp | K ) , including tabular MDPs, factored MDPs, observable POMDPs and decodable POMDPs (see Liu et al. [2022a] for more details).

Theorem D.1. Suppose Assumption 1 and 3 hold. There exists absolute constant c 1 , c 2 > 0 such that for any ( T, δ ) ∈ N × (0 , 1] , if we choose β P = c 1 ln( |P|| Π exp | T/δ ) in Algorithm 3, then with probability at least 1 -δ , we have that for all t ∈ [ T ] ,

For problems that satisfy ξ = ˜ O ( √ d P β P | Π exp | T ) , Theorem D.1 implies a sample complexity of

for learning an O ( ϵ ) -optimal policy, if we have ω = ϵ and ϵ ′ = ϵ/ √ d R . The sample complexity for specific tractable problems can be found in Appendix D.2.

D.2 Examples satisfying generalized eluder-type condition

In this section, we present several canonical examples that satisfy the generalized eluder-type condition with ξ = ˜ O ( √ d P β P | Π exp | T ) . More examples can be found in Liu et al. [2022a].

Example 6 (Finite-precision Factored MDPs) . In factored MDPs, each state s consists of m factors denoted by ( s [1] , · · · , s [ m ]) ∈ X m . The transition structure is also factored as

where pa i ⊆ [ m ] is the parent set of i . The reward function is similarly factored:

Define B := ∑ m i =1 |X| pa i . Factored MDPs satisfy (with | Π exp | = 1 and d P = m 2 |A| 2 B 2 poly( H ) )

Moreover ln |P| ≤ mbAB , ln |R| ≤ mb |X| , where b is the number of bits needed to specify each entry of P ( s h +1 || s h , a h ) or r h ( s ) 6 . Therefore Theorem D.1 implies a sample complexity of

To proceed, we define partially observable Markov decision process (POMDPs).

Definition 6. In a POMDP, states are hidden from the learner and only observations emitted by states can be observed. Formally, at each step h ∈ [ H ] , the learner observes o h ∼ O h ( · | s h ) where s h is the current state and O h ( · | s h ) ∈ ∆ A is the observation distribution conditioning on the current state being s h . Then the learner takes action a h and the environment transitions to s h +1 ∼ P h ( · | s h , a h ) .

Liu et al. [2022a] prove that the following subclasses of POMDPs satisfy Assumption 3 with moderate d P and | Π exp | .

Example 7 ( α -observable POMDPs) . We say a POMDP is α -observable if for every µ, µ ′ ∈ ∆ S ,

Intuitively, the above relation implies that different state distributions will induce different observation distributions, and the parameter α measures the amount of information preserved after mapping states to observations. It is proved that α -observable POMDPs satisfy Condition 3 with Π exp ( π ) = { π } and d P = poly( S, A, α -1 , H ) [Liu et al., 2022a].

For simplicity of notations, let u ( h ) = max { 1 , h -m +1 } .

Example 8 ( m -step decodable POMDPs) . We say a POMDP is m -step decodable if there exists a set of decoder functions { ϕ h } h ∈ [ H ] such that for every h ∈ [ H ] , s h = ϕ h (( o, a ) u ( h ): h -1 , o h ) . In other words, the current state can be uniquely identified from the most recent m -step observations and actions. It is proved that α -observable POMDPs satisfy Condition 3 with | Π exp | = A m and d P = poly( L, A m , H ) where L = max h rank ( P h ) denotes the rank of the transition matrices { P h } h ∈ [ H ] ⊆ R SA × S [Liu et al., 2022a].

D.3 Proof of Theorem D.1

We first define some useful notations. Denote by D t rwd , D t trans the reward, transition dataset at the end of the t -th iteration. We further denote

By the definition of B t , we have that B t = B t R ×B t P . Denote by ( τ t , ˆ r t ) the trajectory-reward pair added into D rwd in the t -th iteration of Algorithm 3.

By the definition of B t R and the fact that all reward feedback is at most ϵ ′ -corrupted, we directly have that the confidence set B t R always contain the groundtruth reward function.

Moreover, according to Liu et al. [2022a], the transition confidence set B t P always satisfies the following properties.

Lemma D.3 (Liu et al. [2022a]) . There exists absolute constant c 2 such that under Assumption 3 and the same choice of β P as in Theorem D.1, we have that with probability at least 1 -δ :

p ⋆ ∈ B t P , for all t ∈ [ T ] ,

The first relation states that the transition confidence set contains the groundtruth transition model with high probability. And the second one states that if we use an arbitrary model p ∈ B t P to predict the transition dynamics under policy π t , then the cumulative prediction error over T iterations is upper bounded by function ξ .

Proof of Theorem D.1. In the following proof, we will assume the two relations in Lemma D.3 hold.

We have that

where the first inequality uses the definition of ( π t , r t , p t ) and the relation ( r t , p t ) ∈ B t , the third one holds with probability at least 1 -δ by Azuma-Hoeffding inequality, the fourth one uses the second relation in Lemma D.3, and the last one invokes the standard regret guarantee for eluder dimension (e.g., Russo and Van Roy [2013]) where d R = dim E ( R , ϵ ′ / 2) .

E Additional details and proofs for Section 4.2

E.1 Algorithm details

In Algorithm 4, we describe how to learn an approximate von Neumann winner via running two copies of an arbitrary adversarial MDP algorithm. Specifically, we maintain two algorithm instances A (1) and A (2) . In the k -th iteration, we first sample two trajectories ( s (1) 1: H , a (1) 1: H ) and ( s (2) 1: H , a (2) 1: H )

without reward by executing π (1) k and π (2) k , the two output policies of A (1) and A (2) , respectively. Then we input ( s (1) H , s (2) H ) into the comparison oracle and get a binary feedback y . After that, we augment ( s (1) 1: H -1 , a (1) 1: H -1 ) with zero reward at the first H -2 steps and reward y at step H -1 , which we feed into A (1) . Similarly we create feedback for A (2) by using ( s (2) 1: H -1 , a (2) 1: H -1 ) and reward 1 -y . The final output policy ¯ π (1) is a uniform mixture of all the policies A (1) has produced during K iterations.

Algorithm 4 Learning von Neumann Winner via Adversarial MDP Algorithms

Initialize two algorithm instances A (1) , A (2) for adversarial MDPs with horizon length H -1 for k = 1 , · · · , K do

Query comparison oracle y ∼ Ber( M ( s (1) H , s (2) H ))

Output average policy mixture ¯ π (1) where ¯ π (1) := Unif ( { π (1) k } k ∈ [ K ] )

Converting the output policy to a Markov policy. Note that the output policy ¯ π is a general non-Markov policy. However, we can convert it to a Markov policy in a sample-efficient manner for tabular MDPs through a simple procedure: execute ¯ π for N episodes, then compute the empirical policy 7

where J h ( s, a ) and J h ( s ) denote the visitation counters at state-action pair ( s, a ) and state s at step h , respectively. The following lemma claims that the resulted Markov policy ˆ π is also an approximate restricted Nash equilibrium.

Lemma E.1. If ¯ π is an ϵ -approximate von Neumann winner, then ˆ π is a 2 ϵ -approximate von Neumann winner with probability at least 1 -δ , provided that N = ˜ Ω ( SA ϵ 2 ) .

E.2 Proof of Theorem 11

Proof. Define U ( π, π ′ ) := E s H ∼ π,s ′ H ∼ π ′ M [ s H , s ′ H ] . The AMDP regret of A (1) gives

where the second inequality uses the fact that there always exists a Markov best response because only the state distribution at the final step matters in the definition of U .

Similarly the regret of A (2) gives

Summing the two inequalities gives

In other words

By the symmetry of preference function, we further have

E.3 Details for adversarial linear MDPs.

To apply the regret guarantees from existing works on adversarial linear MDP [e.g., Sherman et al., 2023], we need to show the constructed reward signal in Algorithm 4 is linearly realizable. Since the reward signal is zero for the first H -2 steps, we only need to consider step H -1 . Recall the definition of linear MDPs requires that there exist feature mappings ϕ and ψ such that P h ( s ′ | s, a ) = ⟨ ϕ h ( s, a ) , ψ ( s ′ ) ⟩ . By the bilinear structure of transition, the conditional expectation of reward at step H -1 in the k -th iteration can be written as

Therefore, the reward function constructed for A (1) is a linear in the feature mapping ϕ . Similarly, we can show the reward function constructed for A (2) is also linear.

E.4 Proof of Lemma E.1

Proof. Let ι = c ln( SAHN/δ ) where c is a large absolute constant. By standard concentration, with probability at least 1 -δ , we have that for all s , if P ¯ π ( s h = s ) ≥ ι/N , then J h ( s ) ≥ N P ¯ π ( s h = s ) / 2 . With slight abuse of notation, we define ¯ π h ( · | s h ) = ∑ a 1: h -1 ,s 1: h -1 ¯ π h ( · | a 1: h -1 , s 1: h ) . Therefore, we have

Given a Markov policy π 1 and general policy π 2 , we define

It follows that for any policy π ,

Therefore,

F Additional details and proofs for Section 4.3

F.1 Algorithm details

To state the algorithm in a more compact way, we first introduce several notations. We denote the expected winning times of policy π against policy π ′ in a RLHF instance with transition p and preference M by

Furthermore, we denote the best-response value against policy π as

Algorithm 5 Learning von Neumann winner via Optimistic MLE

1: B 1 ←P×M

2: execute an arbitrary policy to collect trajectory τ 0

3: for t = 1 , . . . , T do

4: compute optimistic von Neumann winner ( π t , M t , p t ) = arg max π, ( p,M ) ∈B t V π, † p,M

5: compute optimistic best-response ( π t , M t , p t ) = arg min π ′ , ( p,M ) ∈B t V π t ,π p,M

6: sample τ t ∼ π t and τ t ∼ π t

7: invoke comparison oracle on ( τ t , τ t ) to get y t , add ( τ t , τ t , y t ) into D pref

8: for each π ∈ (Π exp ( π t ) ⋃ Π exp ( π t )) do

9: execute π to collect a trajectory τ , add ( π, τ ) into D trans

10: update

We provide the pseudocode of learning von Neumann winner via optimistic MLE in Algorithm 5. In each iteration t ∈ [ T ] , the algorithm performs the following three key steps:

Optimistic planning: Compute the most optimistic von Neumann winner π t by picking the most optimistic model-preference candidate ( M,p ) in the current confidence set B t . Then compute the most optimistic best-response to π t , denoted as π t .

Data collection: Sample two trajectories τ t and τ t from π t and π t . Then input them into the comparison oracle to get feedback y t , which is added into the preference dataset D pref . And similar to the standard OMLE, we also execute policies from the exploration policy set that is constructed by using π t and π t , and add the collected data into the transition dataset D trans .

Confidence set update: Update the confidence set using the updated log-likelihood, which is the same as Algorithm 3 except that we replace the utility-basede preference therein by general preference

′

and the minimax value as

F.2 Proof of Theorem 12

We first introduce several useful notations. Denote by B t M , B t P the preference, transition confidence set in the t -th iteration, which satisfy B t = B t P × B t M . Denote the groundtruth transition and preference by p ⋆ and M ⋆ . To prove Theorem 12, it suffices to bound

Similar to the proof of Theorem D.1, we first state several key properties of the MLE confidence set B t , which are trivial extensions of the confidence set properties in Liu et al. [2022a].

Lemma F.1 (Liu et al. [2022a]) . Under the same condition as Theorem 12, we have that with probability at least 1 -δ : for all t ∈ [ T ]

The first relation states that confidence set B t contains the groundtruth transition-preference model with high probability. The second one resembles the second relation in Lemma D.3. The third one states that any preference model M in confidence set B t M can well predict the preference over the previously collected trajectory pairs.

By using the first relation in Lemma F.1 and the definition of π t and π t , we have

By the second relation in Lemma F.1 and Definition 3, we have

where c is an absolute constant.

Combining the third relation in Lemma F.1 with the regret bound of eluder dimension (e.g., Lemma 2 in Russo and Van Roy [2013]), we have

Putting all pieces together, we have

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider the RLHF setting. First, in the case where there (a) exists a ground-truth utility function and (b) feedback is positive with higher probability when there is a larger difference in rewards, they derive an algorithm to iteratively winnow down a ball of reward functions. When such structure cannot be assumed, they derive a reduction to adversarial MDPS for solving for the von Neumann winner.\n\nSTRENGTHS:\n(+) The paper is clearly written and the algorithms are intuitive and reasonable.\n\n(+) I haven't seen someone consider the von Neumann winner before in the RLHF space.\n\nWEAKNESSES:\n(-) I think there's some weird formatting on the top of pg. 8 as the line numbers stop. Could you fix this? Also, I think you're missing a superscript $(i)$ in the definition of a partial trajectory. Also, please add some space between line 343 and the equation below.\n\n(-) There are no experiments whatsoever.\n\nQUESTIONS:\n1) Could you add in https://arxiv.org/pdf/2305.18505.pdf and https://arxiv.org/pdf/2305.14816.pdf to your related work section and discuss how your approach differs from theirs?\n\n2) Do you get any sort of agnostic guarantee for Alg. 1 in the non-realizable case?\n\n3) Are \"oracle complexity\" (line 169) and \"query complexity\" (footnote 2) the same thing? If so, could you pick a term and stick to it?\n\n4) Imagine if instead of receiving stochastic feedback, you received a deterministic (+/-) label from the oracle. While this is a simpler setting in some sense, I'm not sure if you'd still be able to compare things to a fixed initial trajectory. Do you have any thoughts on how you could modify Alg. 1 to handle this setting?\n\n5) Could you clarify how you could run Alg. 1 with k-wise comparison feedback? I found section 3.3 of the paper unfortunately vague.\n\n6) Footnote 3's reasoning is a bit artificial. Could you instead motivate the setting you consider by noting that for empirical RLHF, people usually treat it as a \"bandit\" problem and only give feedback at the ending?\n\n7) Could you run something like NPG instead as a no-regret algorithm for the adversarial MDPs your reduction requires solving? If so, that might be an illustrative point to add in given it's a bit closer to the PPO-style algorithms people use in practical RLHF. Also, given everything is fully observed, is there an algorithm that gets a tighter rate than OMLE? It sort of feels like you're using a heavier hammer than necessary.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nRefined writing: The objective of this paper is to establish a theoretical foundation for reinforcement learning based on human feedback preferences. The authors conduct an analysis on two aspects: (1) utility-based preferences in tabular MDPs, linear MDPs, and MDPs with low Bellman-Ruler dimension, and (2) general preferences considering the assumption of the calculation of von Neumann winners can be reduced to finding NEs of games with independent transition dynamics.\n\nSTRENGTHS:\nRefined writing: The current limitations in the theoretical understanding of reinforcement learning from human feedback have prompted the need for addressing crucial questions, particularly in light of the significant success achieved by RLHF in applications such as ChatGPT. This paper effectively tackles these important questions, contributing to a deeper understanding of the subject matter.\n\nWEAKNESSES:\nThe research topic of this paper is indeed interesting, and the reviewer initially had high expectations. However, there is a significant discrepancy between the paper's title and the actual content presented in the theoretical analyses.\n\nThe first part of analyses primarily focus on the utility-based preferences, which is applicable to simple linear MDPs or MDPs with lower Bellman-Ruler dimensions. It is challenging to extend the conclusion that \"human query feedback does not scale with the sample complexity of the reinforcement learning algorithm\" to more powerful function approximators commonly used in complex learning settings. Additionally, it raises the question of whether human feedback is even necessary for learning in these simple MDPs. The reviewer thinks that high-cost factors like human feedback are typically introduced in sophisticated tasks such as large language models and autonomous driving.\n\nCertain sections of the paper, particularly the discussion on general preferences, are challenging to follow. There is a lack of exploration regarding how the theoretical analyses and computation of the von Neumann winner can be translated into an algorithm for relating preferences to rewards. This omission makes it difficult to assess the practicality and applicability of the proposed method, as it is based on the assumption of factorizable underlying Markov games. In realistic scenarios, independent transition dynamics are not commonly observed. This raises concerns about the completeness of this section in the paper.\n\nIf the results of the paper are limited to simple MDPs, the authors may consider changing the title to reflect this specificity.\n\nBased on the reviewer's assessment, the current state of the paper does not warrant immediate publication. The authors are encouraged to reorganize their work and submit a more comprehensive version to top machine learning venues in the future.\n\nQUESTIONS:\nPlease find the questions in the weakness section.\n\nLIMITATIONS:\nThis paper didn't discuss limitations of their analyses in a separate section, but it clearly states their assumptions when introducing the theoretical findings.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study the problem of learning in preference-based RL and investigate the question of whether preference based RL is any harder that reward based RL. They show that for preferences that are based on an underlying reward function, preference based RL is no harder than reward based RL for most of the theoretical RL settings including tabular MDPs, linear MDPs, MDPs with finite eluder dimension. They propose an efficient algorithm that can solve this problem by reducing it to the reward based RL and solving the reward version instead. Further, they demonstrate that this reduction does not incur any additional sample complexity, requiring humans to provide preference on only a small portion of trajectories collected by the algorithm.\n\nFor general preference function, they propose a framework of factored Multi-Agent MDP to find the “von Neumann winner” solution concept. They further provide algorithms based on Adversarial MDP and Optimistic MLE to solve the problem in the setting where preference are entirely based on last state and on complete trajectory respectively.\n\nSTRENGTHS:\n- The paper is well written and except minor notation issues it is a good read.\n    \n- The authors provide algorithms to solve the preference based RL problem with a theoretical guarantee on sample and computation complexity in different RL settings including finite state, linear and bounded eluder dimension settings.\n    \n- The authors also study this problem in general preference feedback setting and propose the solution concept of “von Neumann winner”. They further propose algorithms when preference depend  on the final state or entirety of the two compared trajectories.\n\nWEAKNESSES:\n- The paper assumes that humans can provide preferential feedback at trajectory level which is not always easy.\n- The concept of Eluder dimension and following results are not very intuitive.\n- The paper lacks any experimental results but the theoretical contribution is strong enough.\n\nQUESTIONS:\n1. In line 116-117 : Why is underlying reward function not state-action Markovian? The corresponding value notation is not correct.\n2. Can you compare two states as in line 139 or is it two (state,action) pair?\n3. Why the input trajectories needs to be feasible with respect to underlying environment? Where is this assumption used?\n\nLIMITATIONS:\n- The paper does not address the setting when users can only provide preference at state and not trajectory level.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-09"}

comments: SUMMARY:
The authors attempt to show the conditions under which RLHF is theoretically identical to standard RL where a reward function is specified as a part of the environment. The algorithm P2R Interface is given as way to learn from preference feedback such that all requirements are met for RLHF to be identical to standard RL. The reduction of RLHF to standard RL is discussed in the context of utility-baed preferences and general preferences that do not meet the requirements of a utility function.

STRENGTHS:
- The paper includes an extensive related works section.

WEAKNESSES:
- The paper is difficult for me to follow and understand. It is very full of jargon for which no explanation is provided.
- It is not well motivated why the "reduction" the authors propose to make is necessary and the benefits it carries.
- A discussion section to wrap up what has been shown in the paper would be helpful.
- Figure 1 is missing.

QUESTIONS:
- In the utility-based preferences scenario the authors state that the trajectories given as input to the comparison oracle must generated by the policy. How does this work in the case of Ouyang et al. [2022]'s approach where the reward model is learned in advance and therefore are not generated by the policy?

LIMITATIONS:
- The paper is non-trivial for someone not an expert on the specific topic of the paper to follow. It would be great for the paper to be more easily understandable by those with a background in preference-based RL, so that they may incorporate the learnings into their work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-24