SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yuanhao Wang, Qinghua Liu, Chi Jin
{yuanhao,qinghual,chij}@princeton.edu
Reinforcement learning from Human Feedback (RLHF) learns from preference signals, while standard Reinforcement Learning (RL) directly learns from reward signals. Preferences arguably contain less information than rewards, which makes preference-based RL seemingly more difficult. This paper theoretically proves that, for a wide range of preference models, we can solve preference-based RL directly using existing algorithms and techniques for reward-based RL, with small or no extra costs. Specifically, (1) for preferences that are drawn from reward-based probabilistic models, we reduce the problem to robust reward-based RL that can tolerate small errors in rewards; (2) for general arbitrary preferences where the objective is to find the von Neumann winner, we reduce the problem to multiagent reward-based RL which finds Nash equilibria for factored Markov games under a restricted set of policies. The latter case can be further reduced to adversarial MDP when preferences only depend on the final state. We instantiate all reward-based RL subroutines by concrete provable algorithms, and apply our theory to a large class of models including tabular MDPs and MDPs with generic function approximation. We further provide guarantees when K-wise comparisons are available.
Reinforcement learning (RL) is a control-theoretic problem in which agents take a sequence of actions, receive reward feedback from the environment, and aim to find good policies that maximize the cumulative rewards. The reward labels can be objective measures of success (winning in a game of Go [Silver et al., 2017]) or more often hand-designed measures of progress (gaining gold in DOTA2 [Berner et al., 2019]). The empirical success of RL in various domains [Mnih et al., 2013, Vinyals et al., 2019, Todorov et al., 2012] crucially relies on the availability and quality of reward signals. However, this also presents a limitation for applying standard reinforcement learning when designing a good reward function is difficult.
An important approach that addresses this challenge is reinforcement learning from Human feedback (RLHF), where RL agents learn from preference feedback provided by humans. Preference feedback is arguably more intuitive to human users, more aligned with human values and easier to solicit in applications such as recommendation systems and image generation [Pereira et al., 2019, Lee et al., 2023]. Empirically, RLHF is a key ingredient empowering the successes in tasks ranging from robotics [Jain et al., 2013] to large language models [Ouyang et al., 2022].
A simple observation about preference feedback is that preferences can always be reconstructed from reward signals. In other words, preference feedback contains arguably less information than scalar rewards, which may render the RLHF problem more challenging. A natural question to ask is:
Is preference-based RL more difficult than reward-based RL?
Existing research on preference-based RL [Chen et al., 2022, Pacchiano et al., 2021, Novoseller et al., 2020, Xu et al., 2020, Zhu et al., 2023] has established efficient guarantees for learning a near-optimal policy from preference feedback. These works typically develop specialized algorithms and analysis in a white-box fashion, instead of building on existing techniques in standard RL. This leaves it open whether it is necessary to develop new theoretical foundation for preference-based RL in parallel to standard reward-based RL.
This work presents a comprehensive set of results on provably efficient RLHF under a wide range of preference models. We develop new simple reduction-based approaches that solve preference-based RL by reducing it to existing frameworks in reward-based RL, with little to no additional cost:
- For utility-based preferences—those drawn from a reward-based probabilistic model (see Section 2.1), we prove a reduction from preference-based RL to reward-based RL with robustness guarantees via a new Preference-to-Reward Interface (P2R, Algorithm 1). Our approach incurs no sample complexity overhead and the human query complexity  ${}^{1}$  does not scale with the sample complexity of the RL algorithm. We instantiate our framework for comparisons based on (1) immediate reward of the current state-action pair or (2) cumulative reward of the trajectory, and apply existing reward-based RL algorithms to directly find near-optimal policies for RLHF in a large class of models including tabular MDPs, linear MDPs, and MDPs with low Bellman-Eluder dimension, etc. We further provide complexity guarantees when K-wise comparisons are available.
- For general (arbitrary) preferences, we consider the objective of the von Neumann winner [see, e.g., Dudík et al., 2015, Kreweras, 1965], a solution concept that always exists and extends the Condorcet winner. We reduce this problem to multiagent reward-based RL which finds Nash equilibria for a special class of factored two-player Markov games under a restricted set of policies. When preferences only depend on the final state, we prove that such factored Markov games can be solved by both players running Adversarial Markov Decision Processes (AMDP) algorithms independently. For preferences that depend on entire trajectory, we develop an adapted version of optimistic Maximum Likelihood Estimation (OMLE) algorithm [Liu et al., 2022a], which handles this factored Markov games under general function approximation.
Notably, our algorithmic solutions are either reductions to standard reward-based RL problems or adaptations of existing algorithms (OMLE). This suggests that technically, preference feedback is not difficult to address given the existing knowledge of RL with reward feedback. Nevertheless, our impossibility results for utility-based preference (Lemma 2, 3) and reduction for general preferences also highlight several important conceptual differences between RLHF and standard RL.
Dueling bandits. Dueling bandits [Yue et al., 2012, Bengs et al., 2021] can be seen as a special case of preference based RL with  $H = 1$ . Many assumptions later applied to preference based RL, such as an underlying utility model with a link function [Yue and Joachims, 2009], the Plackett-Luce model [Saha and Gopalan, 2019], and the Condorcet winner [Zoghi et al., 2014] can be traced back to literature onueling bandits. A reduction from preference feedback to reward feedback for bandits is proposed by Ailon et al. [2014]. The concept of the von Neumann winner, which we employ for general preferences, has been considered in Dudík et al. [2015] for contextualueling bandits.
RL from human feedback. Using human preferences in RL has been studied for at least a decade [Jain et al., 2013, Busa-Fekete et al., 2014] and is later incorporated with Deep RL [Christiano et al., 2017]. It has found empirical success in robotics [Jain et al., 2013, Abramson et al., 2022, Ding et al., 2023], game playing [Ibarz et al., 2018] and fine-tuning large language models [Ziegler et al., 2019, Ouyang et al., 2022, Bai et al., 2022]. RL with other forms of human feedback, such as demonstrations and scalar ratings, has also been considered in previous research [Finn et al., 2016, Warnell et al., 2018, Arakawa et al., 2018] but falls beyond the scope of this paper.
Theory of Preference-based RL. For utility-based preferences, Novoseller et al. [2020], Pacchiano et al. [2021] and Zhan et al. [2023b] consider tabular or linear MDPs, and assume that the per
Throughout this paper, by sample complexity we mean the number of interactions with the MDP, while the query complexity refers the total number of calls of human evaluators.
trajectory reward is linear in trajectory features. Xu et al. [2020] also considers the tabular setting. However, instead of assuming an explicit link function, several structural properties of the preference is assumed which guarantee a Condorcet winner and ensure small regret of a black-box dueling bandit algorithm. Finally, recent work of Zhu et al. [2023], Zhan et al. [2023a] consider utility-based preferences in the offline setting, assuming the algorithm is provided with a pre-collected human preference (and transition) dataset with good coverage. Compared to the above works, this paper considers the online setting and derives results for utility-based preferences with general function approximation, which are significantly more general than those for tabular or linear MDPs.
For general preferences, Chen et al. [2022] also develops sample-efficient algorithms for finding the von Neumann winner  ${}^{2}$  . Their algorithm is computationally inefficient in general even when restricted to the tabular setting. Compared to this result, our AMDP-based reduction algorithm (Section 4.2) is computationally efficient in the tabular or linear setting when the comparison only depends on the final states. For general trajectory-based comparison, our results apply to a richer class of RL problems including POMDPs.
Finally, we remark that all prior results develop specialized algorithms and analysis for preference-based RL in a white-box fashion. In contrast, we develop reduction-based algorithms which can directly utilize state-of-the-art results in reward-based RL for preference-based RL. This reduction approach enables the significant generality of the results in this paper compared to prior works.
We consider reinforcement learning in episodic MDPs, specified by a tuple  $(H, S, A, \mathbb{P})$ . Here  $S$  is the state space,  $A$  is the action space, and  $H$  is the length of each episode.  $\mathbb{P}$  is the transition probability function; for each  $h \in [H]$  and  $s, a \in S \times A$ ,  $\mathbb{P}_h(\cdot | s, a)$  specifies the distribution of the next state. A trajectory  $\tau \in (S \times A)^H$  is a sequence of interactions  $(s_1, a_1, \dots, s_H, a_H)$  with the MDP. A Markov policy  $\pi = \{\pi_h : S \to \Delta_A\}_{h \in [H]}$  specifies an action distribution based on the current state, while a general policy  $\pi = \{\pi_h : (S \times A)^{h-1} \times S \to \Delta_A\}_{h \in [H]}$  can choose a random action based on the whole history up to timestep  $h$ .
In RLHF, an algorithm interacts with a reward-less MDP environment and may query comparison oracle (human evaluators) for preference information. We consider two types of preferences: utility-based ones and general ones.
2.1 Utility-based preferences
For utility-based comparison, we assume there exists an underlying reward function  $r^{\star}:(\mathcal{S}\times \mathcal{A})^{H}\to [0,1]$ . Given a reward function, the value of a policy can be defined as  $\mathbb{E}_{\pi}[\sum_{h = 1}^{H}r^{\star}(s_h,a_h)]$ , i.e. the expected cumulative reward obtained by executing the policy. An optimal policy is one that maximizes  $\mathbb{E}_{\pi}[\sum_{h = 1}^{H}r^{\star}(s_h,a_h)]$ . We say that  $\pi$  is  $\epsilon$ -optimal if  $\mathbb{E}_{\pi}[\sum_{h = 1}^{H}r^{\star}(s_h,a_h)]\geq \max_{\pi '}\mathbb{E}_{\pi '}[\sum_{h = 1}^{H}r^{\star}(s_h,a_h)] - \epsilon$ . We also consider a setting where the utility is only available for a whole trajectory. In this case, we assume that there is an underlying trajectory reward function  $r^{\star}:(\mathcal{S}\times \mathcal{A})^{H}\to [0,H]$ , which assigns a scalar utility to each trajectory. In this case the value of a policy can be defined similarly as  $\mathbb{E}_{\tau \sim \pi}[r(\tau)]$ .
In preference based RL, the reward is assumed to be unobservable, but is respected by the comparison oracle which models human evaluators.
Definition 1 (Comparison oracle). A comparison oracle takes in two trajectories  $\tau_{1}$ ,  $\tau_{2}$  and returns
$$
o \sim \operatorname {B e r} \left(\sigma \left(r ^ {\star} \left(\tau_ {1}\right) - r ^ {\star} \left(\tau_ {2}\right)\right)\right),
$$
where  $\sigma (\cdot)$  is a link function, and  $r^{\star}(\cdot)$  is the underlying reward function.
Here  $\operatorname{Ber}(p)$  denotes a Bernoulli distribution with mean  $p$ . The comparison outcome  $o = 1$  indicates  $\tau_{1} \succ \tau_{2}$ , and vice versa. We additionally require that the inputs  $\tau_{1}, \tau_{2}$  to the comparison oracle are feasible in the sense that they should be actual trajectories generated by the algorithm and cannot be synthesized artificially. This is motivated by the potential difficulty of asking human evaluators to compare out-of-distribution samples (e.g. random pixels).
2While their paper makes Assumption 3.1 in Chen et al. [2022] that seemingly assumes a Condorcet winner, this assumption actually always holds as  $\pi^{\star}$  can be a general randomized policy, so their solution concept coincides with the von Neumann winner.
In this work, we also consider  $(s,a)$  preferences, where  $\operatorname*{Pr}[(s_1,a_1)\succ (s_2,a_2)] = \sigma (r(s_1,a_1) - r(s_2,a_2))$ . For notational simplicity, we will use  $\tau$  to denote a state-action pair  $(s,a)$  (which can be thought as an incomplete trajectory) so that  $(s,a)$  preferences can be seen as a special case of the comparison oracle (Definition 1). See more details in Remark 1.
2.2 General preferences
For general preferences, we assume that for every trajectory pair  $\tau, \tau'$ , the probability that a human evaluator prefers  $s$  over  $s'$  is
$$
M [ \tau , \tau^ {\prime} ] = \Pr [ \tau \succ \tau^ {\prime} ]. \tag {1}
$$
A general preference may not be realizable by a utility model, so we cannot define the optimal policy in the usual sense. Instead, we follow Dudík et al. [2015] and consider an alternative solution concept, the von Neumann winner (see Definition 5).
2.3 Function approximation
We first introduce the concept of eluder dimension, which has been widely used in RL to measure the difficulty of function approximation.
Definition 2 (eluder dimension). For any function class  $\mathcal{F} \subseteq (\mathcal{X} \to \mathbb{R})$ , its Eluder dimension  $\dim_{\mathrm{E}}(\mathcal{F}, \epsilon)$  is defined as the length of the longest sequence  $\{x_1, x_2, \ldots, x_n\} \subseteq \mathcal{X}$  such that there exists  $\epsilon' \geq \epsilon$  so that for all  $i \in [n]$ ,  $x_i$  is  $\epsilon'$ -independent of its prefix sequence  $\{x_1, \ldots, x_{i-1}\}$ , in the sense that there exists some  $f_i, g_i \in \mathcal{F}$  such that
$$
\sqrt {\sum_ {j = 1} ^ {i - 1} \left((f _ {i} - g _ {i}) (x _ {j})\right) ^ {2}} \leq \epsilon^ {\prime} \mathrm {a n d} | (f _ {i} - g _ {i}) (x _ {i}) | \geq \epsilon^ {\prime}.
$$
Intuitively, eluder dimension measures the number of worst-case mistakes one has to make in order to identify an unknown function from the class  $\mathcal{F}$ . It is often used as a sufficient condition to prove sample efficiency guarantees for optimism-based algorithms.
As in many works on function approximation, we assume knowledge a class of reward functions  $\mathcal{R}$  and realizability.
Assumption 1 (Realizability).  $r^{\star} \in \mathcal{R}$
We further use  $\overline{\mathcal{R}} := \{r + c | c \in [-H, 0], r \in \mathcal{R}\}$  to denote the reward function class augmented with a bias term. The inclusion of an additional bias is due to the assumption that preference feedback is based on reward differences, so we could only learn  $r^{\star}$  up to a constant. We note that for the  $d$ -dimension linear reward class  $\mathcal{R}_{\mathrm{linear}}$ ,  $\dim_{\mathrm{E}}(\overline{\mathcal{R}}_{\mathrm{linear}}) \leq \widetilde{O}(d)$ , and that for a general reward class  $\mathcal{R}$ ,  $\dim_{\mathrm{E}}(\overline{\mathcal{R}}, \epsilon) \leq \mathcal{O}(H \dim_{\mathrm{E}}(\mathcal{R}, \epsilon / 2)^{1.5} / \epsilon)$ . The details can be found in Appendix B.
3 Utility-based RLHF
Given access to comparison oracles instead of scalar rewards, a natural idea is to convert preferences back to scalar reward signals, so that standard RL algorithms can be trained on top of them. In this section, we introduce an efficient reduction from RLHF to standard RL through a preference-to-reward interface. On a high level, the interface provides approximate reward labels for standard RL training, and only queries the comparison oracle when uncertainty is large. The reduction incurs small sample complexity overhead, and the number of queries to human evaluators does not scale with the sample complexity of the RL algorithm using it. Moreover, it solicits feedback from the comparison oracle in a limited number of batches, simplifying the training schedule.
3.1 A preference-to-reward interface
The interaction protocol of an RL algorithm with the Preference-to-Reward (P2R) Interface is shown in Fig. 1. P2R maintains a confidence set of rewards  $B_r$ . When the RL algorithm wishes to learn the reward label of a trajectory  $\tau$ , P2R checks whether  $B_r$  approximately agrees on the reward of  $\tau$ . If so, it can return a reward label with no queries to the comparison oracle; if not, it will query the comparison oracle on  $\tau$  and a fixed trajectory  $\tau_0$ , and update the confidence set of reward functions. The details of P2R are presented in Algorithm 1.
Algorithm 1 Preference-to-Reward (P2R) Interface
1:  $\mathcal{B}_r\gets \mathcal{R},\mathcal{D}\gets \{\} ,\mathcal{D}_{\mathrm{hist}}\gets \{\}$
2: Execute the random policy to collect  $\tau_0$
3: Upon query of trajectory  $\tau$ :
4: if  $(\hat{r},\tau)\in \mathcal{D}_{\mathrm{hist}}$  then
5: Return  $\hat{r}$
6: if  $\max_{r,r' \in \mathcal{B}_r} (r(\tau) - r(\tau_0)) - (r'(\tau) - r'(\tau_0)) < 2\epsilon_0$  then
7:  $\hat{r} \gets r(\tau) - r(\tau_0)$  for an arbitrary  $r \in \mathcal{B}_r$
8:  $\mathcal{D}_{\mathrm{hist}}\gets \mathcal{D}_{\mathrm{hist}}\cup (\hat{r},\tau)$
9: else
10: Query comparison oracle  $m$  times on  $\tau$  and  $\tau_0$ ; compute average comparison result  $\bar{\sigma}$
11:  $\hat{r} \gets \operatorname*{argmin}_{x \in [-H, H]} |\sigma(x) - \bar{o}|$ ,  $\mathcal{D} \gets \mathcal{D} \cup (\hat{r}, \tau)$ ,  $\mathcal{D}_{\mathrm{hist}} \gets \mathcal{D}_{\mathrm{hist}} \cup (\hat{r}, \tau)$
12: Update  $\mathcal{B}_r$
$$
\mathcal {B} _ {r} \gets \left\{r \in \mathcal {B} _ {r}: \sum_ {(\hat {r}, \tau) \in \mathcal {D}} \left(r (\tau) - r (\tau_ {0}) - \hat {r}\right) ^ {2} \leq \beta \right\}
$$
13: Return  $\hat{r}$
Figure 1: Interaction protocol with the reward learning interface.
The performance of P2R depends on the RL algorithm running on top of it. In particular, we define sample complexity of a standard reward-based RL algorithm  $\mathcal{A}$  as follows.
Definition 3. An RL algorithm  $\mathcal{A}$  is  $g(\epsilon)$ -robust and has sample complexity  $\mathcal{C}(\epsilon, \delta)$  if it can output an  $\epsilon$ -optimal policy using  $\mathcal{C}(\epsilon, \delta)$  samples with probability at least  $1 - \delta$ , even if the reward of each trajectory  $\tau$  is perturbed by  $\varepsilon(\tau)$  with  $\| \varepsilon(\tau) \|_{\infty} \leq g(\epsilon)$ .
We would like to note that the requirement of being  $g(\epsilon)$ -robust is typically not restrictive. In fact, any tabular RL algorithm with sample complexity would be  $O(\epsilon / H)$ -robust with the same sample complexity, while many algorithms with linear function approximation are  $O(\epsilon / \mathrm{poly}(d, H))$ -robust [Jin et al., 2020b, Zanette et al., 2020] (again with the same sample complexity). We can further show that one can effortlessly convert any standard RL algorithms with sample complexity  $\mathcal{C}$  into an  $O(1 / \mathcal{C})$ -robust one by using the procedure described in Lemma B.3.
Remark 1 (Trajectory vs.  $(s, a)$  preferences). So far, we presented the comparison oracle (Definition 1) and the P2R (Algorithm 1) using trajectory rewards. This would require the RL algorithm using P2R to be one that learns from once-per-trajectory scalar reward. To be compatible with standard RL algorithms, we can formally set  $\tau$  as an "incomplete trajectory"  $(s, a)$  in both Definition 1 and Algorithm 1. This would not change the theoretical results regarding the P2R reduction.
Theoretical analysis of P2R We assume that  $\sigma (\cdot)$  is known and satisfies the following regularity assumption.
Assumption 2.  $\sigma (0) = \frac{1}{2}$ ; for  $x\in [-H,H]$ ,  $\sigma '(x)\geq \alpha >0$
Assumption 2 is common in bandits literature [Li et al., 2017] and is satisfied by the popular Bradley-Terry model, where  $\sigma$  is the logistic function. It is further motivated by Lemma 2 and Lemma 3: when  $\sigma$  is unknown or has no gradient lower bounds, the optimal policy can be impossible to identify.
Lemma 2. If  $\sigma$  is unknown, even if there are only two possible candidates, the optimal policy is indeterminate.
Lemma 3. If  $\sigma'(\cdot)$  is not lower bounded, for instance when  $\sigma(x) = \frac{1}{2}(1 + \mathrm{sign}(x))$ , the optimal policy is indeterminate.
P2R enjoys the following theoretical guarantee when we choose  $\epsilon_0 = g(\epsilon) / 2$ ,  $\beta = \frac{\epsilon_0^2}{4}$ ,  $d_{\overline{\mathcal{R}}} = \dim_{\mathrm{E}}(\overline{\mathcal{R}},\epsilon_0)$  and  $m = \Theta \left(\frac{d_{\overline{\mathcal{R}}}\ln(d_{\overline{\mathcal{R}}}/\delta)}{\epsilon_0^2\alpha^2}\right)$ .
Theorem 4. Suppose that Assumption 1 and 2 hold, and that  $\mathcal{A}$  is an  $g(\epsilon)$ -robust RL algorithm with sample complexity  $\mathcal{C}(\epsilon, \delta)$ . By running  $\mathcal{A}$  with the interface in Algorithm 1, we can learn an  $\epsilon$ -optimal policy using  $\mathcal{C}(\epsilon, \delta)$  samples and  $\widetilde{\mathcal{O}}\left(\frac{d^2}{\alpha^2 g(\epsilon)^2}\right)$  queries to the comparison oracle with probability  $1 - 2\delta$ .
The full proof of Theorem 4 is deferred to Appendix B.3. The key idea of the analysis is to use the fact that  $r^{\star} \in \mathcal{B}_r$  and the condition in Line 6 to show that the returned reward labels  $\hat{r}$  are approximate accurate, and to use properties of eluder dimension to bound the number of samples for which the comparison oracle is called.
Theorem 4 shows that P2R is a provably efficient reduction: any standard RL algorithm  $\mathcal{A}$  combined with P2R induces a provably efficient algorithm that learns an approximately optimal policy from preference feedback. The number of required interactions with the MDP environment is identical to that of the standard RL algorithm, and the query complexity only scales with the complexity of learning the reward function.
Comparison with other approaches A more straightforward reduction would be extensively querying the comparison oracle for every sample generated by the RL algorithm. While this direct reduction would not suffer from increased sample complexity, it encounters two other drawbacks: (1) the oracle complexity, or the total workload for human evaluators, increases proportionally with the sample complexity of the RL algorithm, which can be prohibitive; (2) the RL training would need to pause and wait for human feedback at every iteration, creating substantial scheduling overhead.
Another method to construct reward feedback is to learn a full reward function directly before running the RL algorithm, as in Ouyang et al. [2022]. However, without pre-existing high-quality offline datasets, collecting the samples for reward learning would require solving an exploration problem at least as hard as RL itself [Jin et al., 2020a], resulting in significant sample complexity overhead. In P2R, the exploration problem is solved by the RL algorithm using it.
Compared to the two alternative approaches, our reduction achieves the best of both worlds by avoiding sample complexity overhead with a query complexity that does not scale with the sample complexity.
3.2 Instantiations of P2R
When combined with existing sample complexity results, Theorem 4 directly implies concrete sample and query complexity bounds for preference-based RL in many settings, with no statistical and small computational overhead.
$(s,a)$  preferences We first consider the comparison that is based on the immediate reward of the current state-action pair. Here we give tabular MDPs and MDPs with low Bellman-Eluder dimension [Jin et al., 2021] as two examples.