SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathcal {F} _ {t} \leftarrow \left\{f \in \mathcal {F}: \sum_ {s = 1} ^ {t - 1} Z _ {s} \left(f \left(x _ {s}, a _ {s}, b _ {s}\right) - f _ {s} \left(x _ {s}, a _ {s}, b _ {s}\right)\right) ^ {2} \leq \beta \right\}.
$$
and the candidate arm set  $\mathcal{A}_t\gets \{\pi_f(x_t):\forall f\in \mathcal{F}_t\}$
4: Learner decides whether to query  $Z_{t} \gets 11\{|\mathcal{A}_{t}| > 1\}$ .
5: if  $Z_{t} = 1$  then
6: Set  $w_{t}\gets \sup_{a,b\in \mathcal{A}_{t}}\sup_{f,f^{\prime}\in \mathcal{F}_{t}}f(x_{t},a,b) - f^{\prime}(x_{t},a,b)$
7: Set  $\lambda_{t}\gets 1\{\sum_{s = 1}^{t - 1}Z_{s}w_{s}\geq \sqrt{AT / \beta}\} .$
8: if  $\lambda_t = 0$  then
9:  $p_t \gets \operatorname{Uniform}(\mathcal{A}_t)$ .
0: else
1:  $\gamma_{t}\gets \sqrt{AT / \beta}.$
2: Let  $p_t$  be a solution of  $\max_{a \in \mathcal{A}_t} \sum_b f_t(x_t, a, b) p_t(b) + \frac{2}{\gamma_t p_t(a)} \leq \frac{5A}{\gamma_t}$ .
3: end if
4: Learner samples  $a_{t}, b_{t} \sim p_{t}$  independently and receives the feedback  $y_{t}$ .
5: Learner feeds  $((x_{t},a_{t},b_{t}),y_{t})$  to the online regression oracle which returns  $f_{t + 1}$
6: else
7: Learner sets  $a_{t}$  and  $b_{t}$  to be the only action in  $\mathcal{A}_t$ , and plays them.
8:  $f_{t + 1}\gets f_t$
9: end if
20: end for
Next, we explain the learner's strategy for making queries. Firstly, the learner computes  $w_{t}$ , which is the "width" of the version space. Specifically,  $w_{t}$  overestimates the instantaneous regret for playing any arm in  $\mathcal{A}_t$  (Lemma 8). Then, the learner defines  $\lambda_{t}$  that indicates if the estimated cumulative regret  $\sum_{s=1}^{t-1} Z_{s} w_{s}$  has exceeded  $\sqrt{AT / \beta}$ . Note that  $Z_{t}$  is multiplied to  $w_{t}$  since no regret is incurred when  $Z_{t} = 0$ . The learner then chooses the actions (to be queried) depending on the values of  $\lambda_{t}$ :
- If  $\lambda_{t} = 0$ , the cumulative reward has not yet exceeded  $\sqrt{AT / \beta} = O(\sqrt{T})$ , so the learner will explore as much as possible by uniform sampling from  $\mathcal{A}_t$ .
- If  $\lambda_{t} = 1$ , the regret may have reached  $O(\sqrt{T})$ , and therefore the learner uses a technique similar to inverse gap weighting (IGW), as inspired by Saha and Krishnamurthy [2022], to achieve a better balance between exploration and exploitation. Specifically, the learner solves the convex program in Line 12, which is feasible and whose solution  $p_t$  satisfies (Lemma 11)
$$
\underset {a \sim p _ {t}} {\mathbb {E}} \left[ f ^ {\star} \left(x _ {t}, \pi_ {f ^ {\star}} (x), a\right) \right] = O \left(\gamma_ {t} \underset {a, b \sim p _ {t}} {\mathbb {E}} \left[ \left(f _ {t} \left(x _ {t}, a, b\right) - f ^ {\star} \left(x _ {t}, a, b\right)\right) ^ {2} \right] + \frac {A}{\gamma_ {t}}\right). \tag {1}
$$
As a result of the above relation, we can convert the instantaneous regret to the point-wise error between the predictor  $f_{t}$  and the truth  $f^{\star}$  plus an additive  $A / \gamma_{t}$ . This allows us to bound the cumulative point-wise error by the regret of the online regression oracle. In the special case where there exists a "reward function"  $r: \mathcal{X} \times \mathcal{A} \to [0,1]$  for each  $f \in \mathcal{F}$  such that  $f(x,a,b) = r(x,a) - r(x,b)$  (Example 1), the solution  $p_{t}$  can be directly written as
$$
p _ {t} (a) = \left\{ \begin{array}{l l} \frac {1}{A + \gamma_ {t} \big (r _ {t} (x _ {t} , \pi_ {f _ {t}} (x _ {t})) - r _ {t} (x _ {t} , a) \big)} & a \neq \pi_ {f _ {t}} (x _ {t}) \\ 1 - \sum_ {a ^ {\prime} \neq \pi_ {f _ {t}} (x _ {t})} p _ {t} (a ^ {\prime}) & a = \pi_ {f _ {t}} (x _ {t}) \end{array} \right.,
$$
where  $r_t$  is the reward function associated with  $f_t$ , i.e.,  $f_t(x, a, b) = r_t(x, a) - r_t(x, b)$ . This is the standard IGW exploration strategy [Foster and Rakhlin, 2020] and leads to the same guarantee as (1) (see Lemma 12).
It is convex as it can be written as  $|\mathcal{A}_t|$  convex constraints:  $\sum_{b}f_{t}(x_{t},a,b)p_{t}(b) + \frac{2}{\gamma_{t}p_{t}(a)}\leq \frac{5A}{\gamma_{t}},\forall a\in \mathcal{A}_{t}$
We discuss the computational tractability of Algorithm 1 in Appendix A. In short, it is computationally tractable for some structured function classes (e.g. linear and tabular function classes). For general function classes, it is also efficient given a regression oracle.
3.1 Theoretical Analysis
Towards the statistical guarantees of Algorithm 1, we employ two quantities to characterize a contextual bandit instance: the uniform gap and the eluder dimension, which are introduced below.
Assumption 3 (Uniform gap). We assume the optimal arm  $\pi_{f^{\star}}(x)$  induced by  $f^{\star}$  under any context  $x\in \mathcal{X}$  is unique. Further, we assume a uniform gap  $\Delta \coloneqq \inf_x\inf_{a\neq \pi_{f^{\star}}(x)}f^{\star}(x,\pi_{f^{\star}}(x),a) > 0$
We note that the existence of a uniform gap is a standard assumption in the literature of contextual bandits [Dani et al., 2008, Abbasi-Yadkori et al., 2011, Audibert et al., 2010, Garivier et al., 2019, Foster and Rakhlin, 2020, Foster et al., 2021]. Next, we introduce the eluder dimension [Russo and Van Roy, 2013] and begin by defining “ $\epsilon$ -dependence”.
Definition 1 ( $\epsilon$ -dependence). Let  $\mathcal{G} \subseteq \mathcal{X} \to \mathbb{R}$  be any function class. We say an element  $x \in \mathcal{X}$  is  $\epsilon$ -dependent on  $\{x_1, x_2, \ldots, x_n\} \subseteq \mathcal{X}$  with respect to  $\mathcal{G}$  if any pair of functions  $g, g' \in \mathcal{G}$  satisfying  $\sum_{i=1}^{n} (g(x_i) - g'(x_i))^2 \leq \epsilon^2$  also satisfies  $g(x) - g'(x) \leq \epsilon$ . Otherwise, we say  $x$  is  $\epsilon$ -independent of  $\{x_1, x_2, \ldots, x_n\}$ .
Definition 2 (Eluder dimension). The  $\epsilon$ -eluder dimension of a function class  $\mathcal{G} \subseteq \mathcal{X} \to \mathbb{R}$ , denoted by  $\dim_E(\mathcal{G}, \epsilon)$ , is the length  $d$  of the longest sequence of elements in  $\mathcal{X}$  satisfying that there exists some  $\epsilon' \geq \epsilon$  such that every element in the sequence is  $\epsilon'$ -independent of its predecessors.
Eluder dimension is a complexity measure for function classes and has been used in the literature of bandits and RL extensively [Chen et al., 2022, Osband and Van Roy, 2014, Wang et al., 2020, Foster et al., 2021, Wen and Van Roy, 2013, Jain et al., 2015, Ayoub et al., 2020, Ishfaq et al., 2021, Huang et al., 2022]. Examples where the eluder dimension is small include linear functions, generalized linear models, and functions in Reproducing Kernel Hilbert Space (RKHS).
Given these quantities, we are ready to state our main results. The proofs are provided in Appendix C.
Theorem 1. Under Assumptions 1 to 3, Algorithm 1 guarantees the following upper bounds of the regret and the number of queries:
$$
\operatorname {R e g r e t} _ {T} ^ {\mathrm {C B}} = \widetilde {O} \left(\min  \left\{\sqrt {A T \beta}, \frac {A ^ {2} \beta^ {2} \dim_ {E} (\mathcal {F} , \Delta)}{\Delta} \right\}\right),
$$
$$
\mathrm {Q u e r i e s} _ {T} ^ {\mathrm {C B}} = \widetilde {O} \left(\min \left\{T, \frac {A ^ {3} \beta^ {3} \mathrm {d i m} _ {E} ^ {2} (\mathcal {F} , \Delta)}{\Delta^ {2}} \right\}\right)
$$
with probability at least  $1 - \delta$ . We recall that  $\beta = O(\alpha^{-1}\Upsilon + \alpha^{-2}\log(\delta^{-1}\log(T)))$ , and  $\alpha$  represents the coefficient of strong convexity of  $\Phi$ . Logarithmic terms are hidden in the upper bounds for brevity.
For commonly used loss function (Examples 2 and 3),  $\beta$  only scales logarithmically with the size (or effective size) of  $\mathcal{F}$  and is thus mild. For instance, for a finite class  $\mathcal{F}$ ,  $\beta$  depends on  $\log |\mathcal{F}|$ . Furthermore, when  $\mathcal{F}$  is infinite, we can replace  $|\mathcal{F}|$  by the covering number of  $\mathcal{F}$  following the standard techniques: for  $d$ -dimensional linear function class,  $\beta$  will have a dependence of  $O(d)$  (effective complexity of  $\mathcal{F}$ ), and for the tabular class,  $\beta$  will have a dependence of  $O(SA^2)$ . In other words, the rate of  $\beta$  is acceptable as long as the complexity of the function class  $\mathcal{F}$  is acceptable.
Best-of-both-worlds guarantee. We observe that both the regret bound and the query complexity bound consist of two components: the worst-case bound and the instance-dependent bound. The worst-case bound provides a guarantee under all circumstances, while the instance-dependent one may significantly improve the upper bound when the underlying problem is well-behaved (i.e., has a small eluder dimension and a large gap).
Lower bounds. To see whether these upper bounds are tight, we provide a lower bound which follows from a reduction from regular multi-armed bandits to contextual dueling bandits.
Theorem 2 (Lower bounds). The following two claims hold: (1) For any algorithm, there exists an instance that leads to  $\mathrm{Regret}_T^{\mathrm{CB}} = \Omega (\sqrt{AT})$ ; (2) For any algorithm achieving a worse-case expected regret upper bound in the form of  $\mathbb{E}[\mathrm{Regret}_T^{\mathrm{CB}}] = O(\sqrt{AT})$ , there exists an instance with gap  $\Delta = \sqrt{A / T}$  that results in  $\mathbb{E}[\mathrm{Regret}_T^{\mathrm{CB}}] = \Omega (A / \Delta)$  and  $\mathbb{E}[\mathrm{Queries}_T^{\mathrm{CB}}] = \Omega (A / \Delta^2) = \Omega (T)$ .
By relating these lower bounds to Theorem 1, we conclude that our algorithm achieves a tight dependence on the gap  $\Delta$  and the number of rounds  $T$  up to logarithmic factors. Furthermore, as an additional contribution, we establish an alternative lower bound in Section C.4.1 by conditioning on the limit of regret, rather than the worst-case regret as assumed in Theorem 2.
Intuition of proofs. We next provide intuition for why our algorithm has the aforementioned theoretical guarantees. First, we observe that from the definition of  $\lambda_{t}$ , the left term inside the indicator is non-decreasing, which allows us to divide rounds into two phases. In the first phase,  $\lambda_{t}$  is always 0, and then at some point, it changes to 1 and remains 1 for the rest rounds. After realizing this, we first explain the intuition of the worst-case regret. In the first phase, as  $w_{t}$  is an overestimate of the instantaneous regret (see Lemma 8), the accumulated regret in this phase cannot exceed  $O(\sqrt{T})$ . In the second phase, we adapt the analysis of IGW to this scenario to obtain an  $O(\sqrt{T})$  upper bound. A similar technique has been used in Saha and Krishnamurthy [2022], Foster et al. [2021]. As the regret in both phases is at most  $O(\sqrt{T})$ , the total regret cannot exceed  $O(\sqrt{T})$ . Next, we explain the intuition of instance-dependent regret. Due to the existence of a uniform gap  $\Delta$ , we can first prove that as long as  $|\mathcal{A}_t| > 1$ , we must have  $w_{t} \geq \Delta$  (see Lemma 7). This means that for all rounds that may incur regret, the corresponding width is at least  $\Delta$ . However, this cannot happen too many times as this frequency is bounded by the eluder dimension, which leads to an instance-dependent regret upper bound. Leveraging a similar technique, we can also obtain an upper bound on the number of queries.
Comparison to regret bounds of dueling bandits. As established by prior works [Yue et al., 2012, Saha and Gaillard, 2022], for dueling bandits, the minimax regret rate is  $\tilde{\Theta} (\sqrt{AT})$  and the instance-dependent rate is  $\tilde{\Theta}\left(A / \Delta\right)$ . If we reduce our result (Theorem 1) into the dueling bandits setting, we will get
$$
\mathrm {R e g r e t} _ {T} = \tilde {O} \left(\min \left\{\sqrt {A T}, \frac {A ^ {2} \dim_ {E} (\mathcal {F} , \frac {\Delta}{2 A ^ {2}})}{\Delta} \right\}\right) = \tilde {O} \left(\min \left\{\sqrt {A T}, \frac {A ^ {3}}{\Delta} \right\}\right)
$$
where the second equality holds since the eluder dimension is upper bounded by  $A$  for dueling bandits. We observe that the worst-case regret rate is the same, but there is a gap of  $A^2$  in the instance-dependent bound. The improvement of this gap is an interesting future direction.
Comparison to MINMAXDB [Saha and Krishnamurthy, 2022]. In this prior work, the authors assume that  $\operatorname{Pr}(y = 1|x,a,b) = (f^{\star}(x,a,b) + 1) / 2$ , which is a specification of our feedback model (Example 2). While our worst-case regret bound matches their regret bound, our paper improves upon their results by having an additional instance-dependent regret bound that depends on the eluder dimension and gap. Furthermore, we also provide bounds on the query complexity which could be small for benign instances while MINMAXDB simply queries on every round.
Comparison to ADACB [Foster et al., 2021]. Our method shares some similarities with Foster et al. [2021], especially in terms of theoretical results, but differs in two aspects: (1) they assume regular contextual bandits where the learner observes the reward directly, while we assume preference feedback, and (2) they assume a stochastic setting where contexts are drawn i.i.d., but we assume that the context is adversarially chosen. While these two settings may not be directly comparable, it should be noted that [Foster et al., 2021] do not aim to minimize query complexity.
Results without the uniform gap assumption. We highlight that Theorem 1 can naturally extend to scenarios where a uniform gap does not exist (i.e., when Assumption 3 is not satisfied) without any modifications to the algorithm. The result is stated below, which is analogous to Theorem 1.
Theorem 3. Under Assumptions 1 and 2, Algorithm 1 guarantees the following upper bounds of the regret and the number of queries:
$$
\mathrm {R e g r e t} _ {T} ^ {\mathrm {C B}} = \widetilde {O} \left(\min  \left\{\sqrt {A T \beta}, \min  _ {\epsilon > 0} \left\{T _ {\epsilon} \beta + \frac {A ^ {2} \beta^ {2} \dim_ {E} (\mathcal {F} , \epsilon)}{\epsilon} \right\} \right\}\right),
$$
$$
\mathrm {Q u e r i e s} _ {T} ^ {\mathrm {C B}} = \widetilde {O} \left(\min  \left\{T, \min  _ {\epsilon > 0} \left\{T _ {\epsilon} ^ {2} \beta / A + \frac {A ^ {3} \beta^ {3} \dim_ {E} ^ {2} (\mathcal {F} , \epsilon)}{\epsilon^ {2}} \right\} \right\}\right)
$$
with probability at least  $1 - \delta$ . Here we define the gap of context  $x$  as  $\operatorname{Gap}(x) \coloneqq \min_{a \neq \pi_{f^*}(x)} f^*(x, \pi_{f^*}(x), a)$  and the number of rounds where contexts have small gap as  $T_\epsilon \coloneqq \sum_{t=1}^{T} \mathbb{1}\{\operatorname{Gap}(x_t) \leq \epsilon\}$ . We also recall that  $\beta = O(\alpha^{-1}\Upsilon + \alpha^{-2}\log(\delta^{-1}\log(T)))$ , and  $\alpha$  denotes the coefficient of strong convexity of  $\Phi$ .
Algorithm 2 Active preference qUeRy fOR imitAtion Iearning (AURORAE)
Require: Function class  $\mathcal{F}_0,\mathcal{F}_1,\dots ,\mathcal{F}_{H - 1}$  , confidence parameter  $\beta$    
1: Learner creates  $H$  instances of Algorithm 1: AURORAh(Fh,β) for  $h = 0,1,\ldots ,H - 1$    
2: for  $t = 1,2,\ldots ,T$  do   
3: Learner receive initial state  $x_{t,0}$    
4: for  $h = 0,1,\ldots ,H - 1$  do   
5: Learner feeds  $x_{t,h}$  to AURORAh(Fh,β), and receives back  $a_{t,h},b_{t,h},Z_{t,h}$    
6: if  $Z_{t,h} = 1$  then   
7: Learner receives feedback  $y_{t,h}$    
8: Learner feeds  $((x_{t,h},a_{t,h},b_{t,h}),y_{t,h})$  to AURORAh(Fh,β) to update its online regres- sion oracle and local variables.   
9: end if   
10: Learner executes  $a\sim$  Uniform(  $\{a_{t,h},b_{t,h}\}$  ) and transits to  $x_{t,h + 1}$    
11: end for   
12: end for
Compared to Theorem 1, the above result has an extra gap-dependent term,  $T_{\epsilon}$ , measuring how many times the context falls into a small-gap region. We highlight that  $T_{\epsilon}$  is small under certain conditions such as the Tsybakov noise condition [Tsybakov, 2004]. It is also worth mentioning that our algorithm is agnostic to  $\epsilon$ , thus allowing us to take the minimum over all  $\epsilon > 0$ .
Comparison to SAGE-BANDIT [Sekhari et al., 2023]. Theorem 3 is similar to Theorem 4 in Sekhari et al. [2023], which studies active queries in contextual bandits with standard reward signal. Although our result looks slightly worse in terms of the factor  $A$ , we believe that this inferiority is reasonable since our approach requires two actions to form a query, thus analytically expanding the action space to  $\mathcal{A}^2$ . Whether this dependency can be improved remains a question for future investigation.
4 Imitation Learning with Preference-Based Active Queries
In this section, we introduce our second algorithm, which is presented in Algorithm 2 for imitation learning. In essence, the learner treats the MDP as a concatenation of  $H$  contextual bandits and runs an instance of AURORA (Algorithm 1) for each time step. Specifically, the learner first creates  $H$  instances of AURORA, denoted by  $\mathrm{AURORAH}_h$  (for  $h = 0,\dots ,H - 1$ ). Here,  $\mathrm{AURORAH}_h$  should be thought of as an interactive program that takes the context  $x$  as input and outputs  $a,b$  and  $Z$ . At each episode  $t$ , and each step  $h$  therein, the learner first feeds the current state  $x_{t,h}$  to  $\mathrm{AURORAH}_h$  as the context; then,  $\mathrm{AURORAH}_h$  decides whether to query (i.e.  $Z_{t,h}$ ) and returns the actions  $a_{t,h}$  and  $b_{t,h}$ . If it decides to make a query, the learner will ask for the feedback  $y_{t,h}$  on the proposed actions  $a_{t,h}, b_{t,h}$ , and provide the information  $((x_{t,h}, a_{t,h}, b_{t,h}), y_{t,h})$  back to  $\mathrm{AURORAH}_h$  to update its online regression oracle (and other local variables). We recall that the noisy binary feedback  $y_{t,h}$  is sampled as  $y_{t,h} \sim \phi(Q_h^{\pi_e}(x_{t,h}, a_{t,h}) - Q_h^{\pi_e}(x_{t,h}, b_{t,h}))$ , and also emphasize that the learner neither has access to  $a \sim \pi_e(x_{t,h})$  like in DAGGER [Ross et al., 2011] nor reward-to-go like in AGGREVATE(D) [Ross and Bagnell, 2014, Sun et al., 2017]. Finally, the learner chooses one of the two actions uniformly at random, executes it in the underlying MDP, and transits to the next state  $x_{t,h + 1}$  in the episode. The above process is then repeated with  $\mathrm{AURORAH}_{h + 1}$  till the episode ends. We name this algorithm AURORAE, the plural form of AURORA, which signifies that the algorithm is essentially a stack of multiple AURORA instances.
4.1 Theoretical Analysis
As Algorithm 2 is essentially a stack of Algorithm 1, we can inherit many of the theoretical guarantees from the previous section. To state the results, we first extend Assumption 3 into imitation learning.
Assumption 4 (Uniform Gap). Let  $f_h^\star$  be defined such that for any  $x \in \mathcal{X}$ ,  $a, b \in \mathcal{A}^2$ ,  $f_h^\star(x, a, b) = Q_h^{\pi_e}(x, a) - Q_h^{\pi_e}(x, b)$ . For all  $h$ , we assume the optimal action for  $f_h^\star$  under any state  $x \in \mathcal{X}$  is unique. Further, we assume a uniform gap  $\Delta := \inf_h \inf_x \inf_{a \neq \pi_{f_h^\star}(x)} f_h^\star(x, \pi_{f_h^\star}(x), a) > 0$ .
This assumption essentially says that  $Q_h^{\pi_e}$  has a gap in actions. We remark that, just as Assumption 3 is a common condition in the bandit literature, Assumption 4 is also common in MDPs [Du et al.,
2019, Foster et al., 2021, Simchowitz and Jamieson, 2019, Jin and Luo, 2020, Lykouris et al., 2021, He et al., 2021]. The theoretical guarantee for Algorithm 2 is presented in Theorem 4. We note a technical difference between this result and Theorem 1: although we treat the MDP as a concatenation of  $H$  contextual bandits, the instantaneous regret of imitation learning is defined as the performance gap between the combined policy  $\pi_t$  derived from the  $H$  instances as a cohesive unit and the expert policy. This necessitates the use of performance difference lemma (Lemma 5) to get a unified result.
Theorem 4. Under Assumptions 1, 2 and 4, Algorithm 2 guarantees the following upper bounds of the regret and the number of queries:
$$
\operatorname {R e g r e t} _ {T} ^ {\mathrm {I L}} \leq \widetilde {O} \left(H \cdot \min  \left\{\sqrt {A T \beta}, \frac {A ^ {2} \beta^ {2} \dim_ {E} (\mathcal {F} , \Delta)}{\Delta} \right\}\right) - \operatorname {A d v} _ {T},
$$
$$
\mathrm {Q u e r i e s} _ {T} ^ {\mathrm {I L}} \leq \widetilde {O} \left(H \cdot \min \left\{T, \frac {A ^ {3} \beta^ {3} \mathrm {d i m} _ {E} ^ {2} (\mathcal {F} , \Delta)}{\Delta^ {2}} \right\}\right)
$$
with probability at least  $1 - \delta$ . Here  $\mathrm{Adv}_T \coloneqq \sum_{t=1}^T \sum_{h=0}^{H-1} \mathbb{E}_{x_{t,h}} \sim d_{x_{t,0},h}^{\pi_t}[\max_a A_h^{\pi_e}(x_{t,h},a)]$  is non-negative, and  $d_{x_{t,0},h}^{\pi_t}(x)$  denotes the probability of  $\pi_t$  reaching the state  $x$  at time step  $h$  starting from initial state  $x_{t,0}$ . In the above,  $\beta = O(\alpha^{-1}\Upsilon + \alpha^{-2}\log(H\delta^{-1}\log(T)))$  and  $\alpha$  denotes the coefficient of strong convexity of  $\Phi$ .
Compared to Theorem 1, the main terms of the upper bounds for imitation learning are precisely the bounds in Theorem 1 multiplied by  $H$ . In the proof presented in Appendix C.6, we use the performance difference lemma to reduce the regret of imitation learning to the sum of the regret of  $H$  contextualueling bandits, which explains this additional factor of  $H$ .
Another interesting point is that the main term of the regret upper bound is subtracted by a non-negative term  $\mathrm{Adv}_T$ , which measures the degree to which we can outperform the expert policy. In other words, our algorithm not only competes with the expert policy but can also surpass it to some extent. To see this, let us consider the average regret, which is defined as  $\mathrm{AveRegret}_T^{\mathrm{IL}} := \mathrm{Regret}_T^{\mathrm{IL}} / T = \sum_{t=1}^{T} (V_0^{\pi_e}(x_{t,0}) - V_0^{\pi_t}(x_{t,0})) / T$ . Then, Theorem 4 implies that  $\mathrm{AveRegret}_T^{\mathrm{IL}} = O(H\sqrt{A\beta / T}) - \mathrm{Adv}_T / T$  where we have simplified it by ignoring the instance-dependent upper bound and logarithmic factors for clarity. Now, consider a case where  $\max_a A_h^{\pi_e}(x, a) > \alpha_0$  for some constant  $\alpha_0 > 0$  for all  $x$  and  $h$ . This can happen when the expert policy is suboptimal for every state. Consequently, we have  $\mathrm{Adv}_T > \alpha_0 HT$ . In this case, the average regret is further bounded by  $\mathrm{AveRegret}_T^{\mathrm{IL}} = O(H\sqrt{A\beta / T}) - \alpha_0 H$ . When  $T \to \infty$ , we have  $\mathrm{AveRegret}_T^{\mathrm{IL}} \to -\alpha_0 H < 0$ . This means that the best (or average) learned policy will eventually outperform the expert policy. This guarantee is stronger than that of DAGGER [Ross et al., 2011] in that DAGGER cannot ensure the learned policy is better than the expert policy regardless of how suboptimal the expert may be. While this may look surprising at first glance since we are operating under a somewhat weaker query mode than that of DAGGER, we note that by querying experts for comparisons on pairs of actions with feedback sampling as  $y \sim \phi(Q^{\pi_e}(x, a) - Q^{\pi_e}(x, b))$ , it is possible to identify the action that maximizes  $Q^{\pi_e}(x, a)$  (even if we cannot identify the value  $Q^{\pi_e}(x, a)$ ). Finally, we remark that our worst-case regret bound is similar to that of Ross and Bagnell [2014], Sun et al. [2017], which can also outperform a suboptimal expert but require access to both expert's actions and reward signals — a much stronger query model than ours.
We presented interactive decision-making algorithms that learn from preference-based feedback while minimizing query complexity. Our algorithms for contextual bandits and imitation learning share worst-case regret bounds similar to the bounds of the state-of-the-art algorithms in standard settings while maintaining instance-dependent regret bounds and query complexity bounds. Notably, our imitation learning algorithm can outperform suboptimal experts, matching the result of Ross and Bagnell [2014], Sun et al. [2017], which operates under much stronger feedback.