SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ayush Sekhari* MIT
sekhari@mit.edu
Wen Sun
ws455@cornell.edu
Karthik Sridharan
ks999@cornell.edu
Runzhe Wu
rw646@cornell.edu
We consider the problem of contextual bandits and imitation learning, where the learner lacks direct knowledge of the executed action's reward. Instead, the learner can actively request the expert at each round to compare two actions and receive noisy preference feedback. The learner's objective is two-fold: to minimize regret associated with the executed actions, while simultaneously, minimizing the number of comparison queries made to the expert. In this paper, we assume that the learner has access to a function class that can represent the expert's preference model under appropriate link functions and present an algorithm that leverages an online regression oracle with respect to this function class. For the contextual bandit setting, our algorithm achieves a regret bound that combines the best of both worlds, scaling as  $O(\min \{\sqrt{T}, d / \Delta\})$ , where  $T$  represents the number of interactions,  $d$  represents the eluder dimension of the function class, and  $\Delta$  represents the minimum preference of the optimal action over any suboptimal action under all contexts. Our algorithm does not require the knowledge of  $\Delta$ , and the obtained regret bound is comparable to what can be achieved in the standard contextual bandits setting where the learner observes reward signals at each round. Additionally, our algorithm makes only  $O(\min \{T, d^2 / \Delta^2\})$  queries to the expert. We then extend our algorithm to the imitation learning setting, where the agent engages with an unknown environment in episodes of length  $H$ , and provide similar guarantees regarding regret and query complexity. Interestingly, with preference-based feedback, our imitation learning algorithm can learn a policy outperforming a sub-optimal expert, matching the result from interactive imitation learning algorithms [Ross and Bagnell, 2014] that require access to the expert's actions and also reward signals.
Human feedback for training machine learning models has been widely used in scenarios including robotics [Ross et al., 2011, 2013, Jain et al., 2015, Laskey et al., 2016, Christiano et al., 2017] and natural language processing [Stiennon et al., 2020, Ouyang et al., 2022]. By integrating human feedback into the training process, these prior works provide techniques to align machine-learning models with human intention and enable high-quality human-machine interaction (e.g., ChatGPT).
*Authors are listed in alphabetical order of their last names.
Existing methods generally leverage two types of human feedback. The first is the action from human experts, which is the dominant feedback mode in the literature of imitation learning and learning from demonstrations [Abbeel and Ng, 2004, Ziebart et al., 2008, Daumé et al., 2009, Ross et al., 2011, Ross and Bagnell, 2014, Sun et al., 2017, Osa et al., 2018, Li et al., 2023]. The second type of feedback is preference-based feedback, which involves comparing pairs of actions. In this approach, the expert provides feedback by indicating their preference between two options selected by the learner. While both types of feedback have their applications, our focus in this work is on preference-based feedback, which is particularly suitable for scenarios where it is challenging for human experts to recommend the exact optimal action while making pairwise comparisons is much easier.
Learning via preference-based feedback has been extensively studied, particularly in the field of dueling bandits [Yue and Joachims, 2011, Yue et al., 2012, Zoghi et al., 2014, Ailon et al., 2014, Komiyama et al., 2015, Wu and Liu, 2016, Saha and Gaillard, 2021, Bengs et al., 2021, Saha and Gaillard, 2022] and contextual dueling bandits [Dudík et al., 2015, Saha, 2021, Saha and Krishnamurthy, 2022, Bengs et al., 2022, Wu et al., 2023]. Different from the standard bandit setting, the learner proposes two actions in dueling bandits and only gets noisy preference feedback from the human expert. Follow-up works extend the preference-based learning model from the one-step bandit setting to the multi-step decision-making (e.g., imitation learning and reinforcement learning) setting [Chu and Ghahramani, 2005, Sadigh et al., 2017, Christiano et al., 2017, Lee et al., 2021b, Chen et al., 2022, Saha et al., 2023]. These studies mainly focus on learning a high-quality policy from human feedback, without concerning the question of active query in order to minimize the query complexity.
However, query complexity is an important metric to optimize when learning from human feedback, as human feedback is expensive to collect [Lightman et al., 2023]. For instance, InstructGPT [Ouyang et al., 2022] is trained only on around 30K pieces of human feedback, which is significantly fewer than the internet-scale dataset used for pre-training the base model GPT3, indicating the challenge of scaling up the size of human feedback datasets. In other areas, such as robotics, learning from human feedback is also not easy, and prior studies (e.g., Cohn et al. [2011], Zhang et al. [2022], Myers et al. [2023]) have explored this issue from various perspectives. Ross et al. [2013], Laskey et al. [2016] pointed out that querying human feedback in the learning loop is challenging, and extensively querying for feedback puts too much burden on the human experts.
In this work, we design principled algorithms that learn from preference-based feedback while at the same time minimizing query complexity under the settings of contextual bandits [Auer et al., 2002] and imitation learning [Ross et al., 2011]. Our main contributions can be summarized as follows.
- In the contextual dueling bandits setting, the stochastic preference feedback is generated based on some preference matrix [Saha and Krishnamurthy, 2022]. We propose an algorithm (named AURORA – in short of Active preference  $qUeRy$  for contextual bAndits) that achieves a best-of both-worlds regret bound (i.e., achieving the minimum of the worst-case regret and an instance dependent regret), while at the same providing an instance-dependent query complexity bound. For benign instances with small eluder dimension and large gap, our regret and query complexity bounds both scale with  $\ln(T)$  where  $T$  is the total number of interactions in contextual bandits.
- In imitation learning, the stochastic preference feedback is generated based on the reward-to-go of the expert's policy (e.g., the expert prefers actions that lead to higher reward-to-go). We propose an algorithm named AURORAE, in short of Active preference  $qUeRyfOR imitAton lLearning$ , which instantiates  $H$  instances of AURORA, one per each time step for the finite horizon Markov Decision Process (MDP), where  $H$  is the horizon. By leveraging preference-based feedback, we show that, interestingly, our algorithm can learn to outperform the expert when the expert is suboptimal. Such a result is beyond the scope of the classic imitation learning algorithm DAGGER and previously can only be achieved by algorithms like AGGREVATE(D) [Ross and Bagnell, 2014, Sun et al., 2017, Cheng and Boots, 2018] and LOLS [Chang et al., 2015] which require direct access to expert's actions and reward signal – a much stronger feedback mode than ours.
To the best of our knowledge, for both contextual bandit and imitation learning with preference-based feedback, our algorithms are the first to achieve best-of-both-worlds regret bounds via active querying.
Selective Sampling. Numerous studies have been conducted on selective sampling across various settings [Cesa-Bianchi et al., 2005, Dekel et al., 2012, Agarwal, 2013, Hanneke and Yang, 2015,
2021, Zhu and Nowak, 2022], with the work of Sekhari et al. [2023] being closest to ours. Sekhari et al. [2023] presented a suite of provably efficient algorithms that are applicable to settings including contextual bandits and imitation learning. The primary distinction between our setting and the prior works lies in the feedback modality—we assume preference-based feedback, whereas they assume direct label feedback or reward signals.
Contextual bandits with preference feedback. Dudík et al. [2015] is the first to consider contextual dueling bandits, and one of their algorithms achieves the optimal regret rate. Saha and Krishnamurthy [2022] studied contextual dueling bandits using a value function class and proposed an algorithm based on a reduction to online regression, which also achieves an optimal worst-case regret bound. In this paper, we mainly follow the setting of the latter and make notable improvements in two aspects: (1) in addition to the  $O(\sqrt{AT})$  optimal regret rate where  $A$  is the number of actions and  $T$  is the number of interaction rounds, we established an instance-dependent regret upper bound that can be significantly smaller when the bandit exhibits a favorable structure; (2) our algorithm has an instance-dependent upper bound on the number of queries, and thus when the underlying instance is well behaved (has small eluder dimension and large gap), we will make significantly fewer queries.
Another related work is Saha and Gaillard [2022] which achieves the best-of-both-worlds regret for non-contextual dueling bandits. Our setting is more general due to the context and general function approximation, which enables us to leverage function classes beyond linear and tabular cases.
RL with preference feedback. RL with preference feedback has been widely employed in recent advancements in AI [Ouyang et al., 2022, OpenAI, 2023]. According to Wirth et al. [2017], there are generally three types of preference feedback: action preferences [Firnkranz et al., 2012], state preferences [Wirth and Fornkranz, 2014], and trajectory preferences [Busa-Fekete et al., 2014, Novoseller et al., 2020, Xu et al., 2020, Lee et al., 2021a, Chen et al., 2022, Saha et al., 2023, Pacchiano et al., 2021, Biyik and Sadigh, 2018, Taranovic et al., 2022, Sadigh et al., 2017]. We focus on the action preference with the goal of achieving tight regret bounds and query complexities.
The concurrent work from Zhan et al. [2023] investigates the experimental design in both the trajectories-based and action-based preference settings, for which they decouple the process of collecting trajectories from querying for human feedback. Their action-based setting is the same as ours, but they mainly focus on linear parameterization, while our approach is a reduction to online regression and can leverage general function approximation beyond linear function classes.
Imitation learning. In imitation learning, two common feedback modalities are typically considered: expert demonstration and preference. The former involves directly acquiring expert actions [Ross et al., 2011, Ross and Bagnell, 2014, Sun et al., 2017, Chang et al., 2015, Sekhari et al., 2023], while the latter focuses on obtaining preferences between selected options [Chu and Ghahramani, 2005, Lee et al., 2021b, Zhu et al., 2023]. Brown et al. [2019, 2020] leveraged both demonstrations and preference-based information and empirically showed that their algorithm can learn to outperform experts. Our imitation learning setting belongs to the second category, and we established bounds on the regret and the query complexity for our algorithm. We show that our algorithm can learn a policy that can provably outperform the expert (when it is suboptimal for the underlying environment).
2.1 Contextual Bandits with Preference-Based Feedback
In this section, we introduce the contextual dueling bandits setting. For notation, we denote  $[N]$  as the integer set  $\{1,\dots ,N\}$  and denote  $\Delta (S)$  as the set of distributions over a set  $S$
We assume a context set  $\mathcal{X}$  and an action space  $\mathcal{A} = [A]$ . At each round  $t \in [T]$ , a context  $x_{t}$  is drawn adversarially, and the learner's task is to pick a pair of actions  $(a_{t}, b_{t}) \in \mathcal{A} \times \mathcal{A}$  and then decide whether to make a query to the expert. If making a query, a noisy feedback  $y_{t} \in \{-1, 1\}$  is revealed to the learner regarding whether  $a_{t}$  or  $b_{t}$  is better. We assume that the expert relies on a preference function  $f^{\star}: \mathcal{X} \times \mathcal{A} \times \mathcal{A} \to [-1, 1]$  to samples its preference feedback  $y_{t}$ :
$$
\Pr \left(a _ {t} \text {i s p r e f e r e r t b o} _ {t} \mid x _ {t}\right) := \Pr \left(y _ {t} = 1 \mid x _ {t}, a _ {t}, b _ {t}\right) = \phi \left(f ^ {\star} \left(x _ {t}, a _ {t}, b _ {t}\right)\right)
$$
where  $\phi(d): [-1, 1] \to [0, 1]$  is the link function, which satisfies  $\phi(d) + \phi(-d) = 1$  for any  $d$ . If the learner does not make a query, it will not receive any feedback for the selected actions  $a_t$  and  $b_t$ . Let  $Z_t \in \{0, 1\}$  indicate whether the learner makes a query at round  $t$ .
We assume that the learner has access to a function class  $\mathcal{F} \subseteq \mathcal{X} \times \mathcal{A} \times \mathcal{A} \to [-1, 1]$  that realizes  $f^{\star}$ . Furthermore, we assume that  $f^{\star}$ , as well as the functions in  $\mathcal{F}$ , is transitive and anti-symmetric.
Assumption 1. We assume  $f^{\star} \in \mathcal{F}$  and any functions  $f \in \mathcal{F}$  satisfies the following two properties: (1) transitivity: for any  $x \in \mathcal{X}$  and  $a, b, c \in \mathcal{A}$ , if  $f(x, a, b) > 0$  and  $f(x, b, c) > 0$ , then we must have  $f(x, a, c) > 0$ ; (2) anti-symmetry:  $f(x, a, b) = -f(x, b, a)$  for any  $x \in \mathcal{X}$  and any  $a, b \in \mathcal{A}$ .
We provide an example below for which Assumption 1 is satisfied.
Example 1. Assume there exists a function  $r^{\star}:\mathcal{X}\times \mathcal{A}\to [0,1]$  such that  $f^{\star}(x,a,b) = r^{\star}(x,a) - r^{\star}(x,b)$  for any  $x\in \mathcal{X}$  and  $a,b\in \mathcal{A}$ . Typically, such a function  $r^\star$  represents the "reward function" of the contextual bandit. In such a scenario, we can first parameterize a reward class  $\mathcal{R}\subseteq \mathcal{X}\times \mathcal{A}\rightarrow [0,1]$  and define  $\mathcal{F} = \{f:f(x,a,b) = r(x,a) - r(x,b),r\in \mathcal{R}\}$ . Moreover, it is common to have  $\phi (d)\coloneqq 1 / (1 + \exp (-d))$  in this setting, which recovers the Bradley-Terry-Luce (BTL) model [Bradley and Terry, 1952] — a commonly used model in practice for learning reward models [Christiano et al., 2017].
Assumption 1 ensures the existence of an optimal arm, as stated below.
Lemma 1. Under Assumption 1, for any function  $f \in \mathcal{F}$  and any context  $x \in \mathcal{X}$ , there exists an arm  $a \in \mathcal{A}$  such that  $f(x, a, b) \geq 0$  for any arm  $b \in \mathcal{A}$ . We denote this best arm by  $\pi_f(x) \coloneqq a$ .<sup>2</sup>
The learner's goal is to minimize the regret while also minimizing the number of queries, defined as:
$$
\operatorname {R e g r e t} _ {T} ^ {\mathrm {C B}} := \sum_ {t = 1} ^ {T} \left(f ^ {\star} \left(x _ {t}, \pi_ {f ^ {\star}} \left(x _ {t}\right), a _ {t}\right) + f ^ {\star} \left(x _ {t}, \pi_ {f ^ {\star}} \left(x _ {t}\right), b _ {t}\right)\right), \quad \operatorname {Q u e r i e s} _ {T} ^ {\mathrm {C B}} := \sum_ {t = 1} ^ {T} Z _ {t}.
$$
It is worth noting that when  $f^{\star}$  is the difference in rewards (as in Example 1), the regret defined above reduces to the standard regret of a contextual bandit. We also remark that our feedback model generalizes that of Saha and Krishnamurthy [2022] in that we assume an additional link function  $\phi$ , while they assume the feedback is sampled from  $\operatorname*{Pr}(y = 1 \mid x, a, b) = (P_t[a_t, b_t] + 1)/2$  where  $P_t$  is a preference matrix. Their loss function is captured in our setting (Example 2). However, Saha and Krishnamurthy [2022] do not assume transitivity.
2.2 Imitation Learning with Preference-Based Feedback
In our imitation learning setup, we consider that the learner operates in a finite-horizon Markov decision process (MDP), which is a tuple  $M(\mathcal{X}, \mathcal{A}, r, P, H)$  where  $\mathcal{X}$  is the state space,  $\mathcal{A}$  is the action space,  $P$  is the transition kernel,  $r: \mathcal{X} \times \mathcal{A} \to [0,1]$  is the reward function, and  $H$  is the length of each episode. The interaction between the learner and the environment proceeds as follows: at each episode  $t \in [T]$ , the learner receives an initial state  $x_{t,0}$  which could be chosen adversarially. Then, the learner interacts with the environment for  $H$  steps. At each step  $h$ , the learner first decides whether to make a query. If making a query, the learner needs to select a pair of actions  $(a_{t,h}, b_{t,h}) \in \mathcal{A} \times \mathcal{A}$  upon which a feedback  $y_{t,h} \in \{-1,1\}$  is revealed to the learner regarding which action is preferred from the expert's perspective. Here the feedback is sampled according to
$\operatorname{Pr}(a_{t,h}$  is preferred to  $b_{t,h} \mid x_{t,h}, h) \coloneqq \operatorname{Pr}(y_{t,h} = 1 \mid x_{t,h}, a_{t,h}, b_{t,h}, h) = \phi \big(f_h^\star(x, a_{t,h}, b_{t,h})\big)$ .
Irrespective of whether the learner made a query, it then picks a single action from  $a_{t,h}, b_{t,h}$  and transit to the next step (our algorithm will just pick an action uniformly at random from  $a_{t,h}, b_{t,h}$ ). After  $H$  steps, the next episode starts. Let  $Z_{t,h} \in \{0,1\}$  indicate whether the learner decided to query at step  $h$  in episode  $t$ . We assume that the function class  $\mathcal{F}$  is a product of  $H$  classes, i.e.,  $\mathcal{F} = \mathcal{F}_0 \times \dots \mathcal{F}_{H-1}$  where, for each  $h$ , we use  $\mathcal{F}_h = \{f : \mathcal{X} \times \mathcal{A} \times \mathcal{A} \to [-1,1]\}$  to model  $f_h^{\star}$  and assume that  $\mathcal{F}_h$  satisfies Assumption 1.
A policy is a mapping  $\pi : \mathcal{X} \to \Delta(\mathcal{A})$ . For a policy  $\pi$ , the state value function for a state  $x$  at step  $h$  is defined as  $V_h^\pi(x) \coloneqq \mathbb{E}[\sum_{i=h}^{H-1} r_i | x_h = x]$  and the state-action value function for a state-action pair  $(x, a)$  is  $Q_h^\pi(x, a) \coloneqq \mathbb{E}[\sum_{i=h}^{H-1} r_i | x_h = x, a_h = a]$ , where the expectations are taken w.r.t. the trajectories sampled by  $\pi$  in the underlying MDP.
In the imitation learning setting, we assume that the expert (who gives the preference-based feedback) is equipped with a markovian policy  $\pi_{e}$ , and that the preference of the expert is dependent on the
2When the best arms is not unique, the ties are broken arbitrarily but consistently.
reward-to-go under  $\pi_e$  (i.e. on a state  $x$ , actions with higher values of  $Q^{\pi_e}(s,a)$  will be preferred by the expert). Formalizing this intuition, we assume that  $f_h^\star$  is defined such that as  $f_h^\star (x,a,b)\coloneqq Q_h^{\pi_e}(x,a) - Q_h^{\pi_e}(x,b)$ . The goal of the learner is still to minimize the regret and number of queries:
$$
\operatorname {R e g r e t} _ {T} ^ {\mathrm {I L}} := \sum_ {t = 1} ^ {T} \left(V _ {0} ^ {\pi_ {e}} \left(x _ {t, 0}\right) - V _ {0} ^ {\pi_ {t}} \left(x _ {t, 0}\right)\right), \quad \operatorname {Q u e r i e s} _ {T} ^ {\mathrm {I L}} := \sum_ {t = 1} ^ {T} \sum_ {h = 0} ^ {H - 1} Z _ {t, h}.
$$
Here  $\pi_t$  is the strategy the learner uses to select actions at episode  $t$ .
2.3 Link Function and Online Regression Oracle
Following the standard practice in the literature [Agarwal, 2013], we assume  $\phi$  is the derivative of some  $\alpha$ -strongly convex function (Definition 3)  $\Phi : [-1,1] \to \mathbb{R}$  and define the associated loss function as  $\ell_{\phi}(d,y) = \Phi(d) - d(y + 1)/2$ . Additionally, in line with prior works [Foster et al., 2021, Foster and Rakhlin, 2020, Simchi-Levi and Xu, 2022, Foster et al., 2018a, Sekhari et al., 2023], our algorithm utilizes an online regression oracle, which is assumed to have a sublinear regret guarantee w.r.t.  $\mathcal{F}$  on arbitrary data sequences.
Assumption 2. We assume the learner has access to an online regression oracle pertaining to the loss  $\ell_{\phi}$  such that for any sequence  $\{(x_1, a_1, b_1, y_1), \ldots, (x_T, a_T, b_T, y_T)\}$  where the label  $y_t$  is generated by  $y_t \sim \phi(f^*(x_t, a_t, b_t))$ , we have
$$
\sum_ {t = 1} ^ {T} \ell_ {\phi} \left(f _ {t} \left(x _ {t}, a _ {t}, b _ {t}\right), y _ {t}\right) - \inf  _ {f \in \mathcal {F}} \ell_ {\phi} \left(f \left(x _ {t}, a _ {t}, b _ {t}\right), y _ {t}\right) \leq \Upsilon (\mathcal {F}, T)
$$
for some  $\Upsilon(\mathcal{F}, T)$  that grows sublinearly with respect to  $T$ . For notational simplicity, whenever clear from the context, we define  $\Upsilon := \Upsilon(\mathcal{F}, T)$ .
Here  $\Upsilon$  represents the regret upper bound and is typically of logarithmic order in  $T$  or the cardinality of the function class  $\mathcal{F}$  in many cases (here we drop the dependence on  $T$  in notation for simplicity). We provide a few examples below.
Example 2 (Squared loss). If we consider  $\Phi(d) = d^2/4 + d/2 + 1/4$ , which is  $1/4$ -strongly convex, then we obtain  $\phi(d) = (d+1)/2$  and  $\ell_{\phi}(d,y) = (d-y)^2/4$ , thereby recovering the squared loss, which has been widely studied in prior works. For example, Rakhlin and Sridharan [2014] characterized the minimax rates for online square loss regression in terms of the offset sequential Rademacher complexity, resulting in favorable bounds for the regret. Specifically, we have  $\Upsilon = O(\log |\mathcal{F}|)$  assuming the function class  $\mathcal{F}$  is finite, and  $\Upsilon = O(d\log(T))$  assuming  $\mathcal{F}$  is a  $d$ -dimensional linear class. We also kindly refer the readers to Krishnamurthy et al. [2017], Foster et al. [2018a] for efficient implementations.
Example 3 (Logistic loss). When  $\Phi(d) = \log(1 + \exp(d))$  which is strongly convex at  $[-1,1]$ , we have  $\phi(d) = 1 / (1 + \exp(-d))$  and  $\ell_{\phi}(d,y) = \log(1 + \exp(-yd))$ . Thus, we recover the logistic regression loss, which allows us to use online logistic regression and achieve  $\Upsilon = O(\log|\mathcal{F}|)$  assuming finite  $\mathcal{F}$ . There have been numerous endeavors in minimizing the log loss, such as Foster et al. [2018b] and Cesa-Bianchi and Lugosi [2006, Chapter 9].
3 Contextual Bandits with Preference-Based Active Queries
We first present the algorithm, named AURORA, for contextual dueling bandits, as shown in Algorithm 1. At each round  $t \in [T]$ , the learner first constructs a version space  $\mathcal{F}_t$  containing all functions close to past predictors on the observed data. Here, the threshold  $\beta$  set to  $4\Upsilon/\alpha + (16 + 24\alpha)\log(4\delta^{-1}\log(T)) / \alpha^2$  ensures that  $f^\star \in \mathcal{F}_t$  for any  $t \in [T]$  with probability at least  $1 - \delta$  (Lemma 9). Thus,  $\mathcal{A}_t$  is non-empty for all  $t \in [T]$  and correspondingly Line 17 is well defined. The learner then forms a candidate arm set  $\mathcal{A}_t$  consisting of greedy arms induced by all functions in the version space. When  $|\mathcal{A}_t| = 1$ , the only arm in the set is the optimal arm since  $f^\star \in \mathcal{F}_t$ , and thus no query is needed ( $Z_t = 0$ ). However, when  $|\mathcal{A}_t| > 1$ , any arm in  $\mathcal{A}_t$  could potentially be the optimal arm, and thus the learner needs to make a comparison query to obtain more information.
3The online regression oracle updates as follows: in each iteration, after seeing  $x_{t},a_{t},b_{t}$  , it proposes a decision  $f_{t}$  , then  $y_{t}$  is revealed and the online regression oracle incurs loss  $\ell_{\phi}(f_t(x_t,a_t,b_t),y_t)$
Algorithm 1 Active preference qUeRy fOR contextual bAndits (AURORA)
Require: Function class  $\mathcal{F}$ , confidence parameter  $\beta = \frac{4\Upsilon}{\alpha} + \frac{16 + 24\alpha}{\alpha^2} \log \left(4\delta^{-1} \log (T)\right)$ .
1: Online regression oracle produces  $f_{1}$ .
2: for  $t = 1,2,\ldots ,T$  do
3: Learner receives context  $x_{t}$ , computes the version space