SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ayush Sekhari $^{1*}$
Runzhe  $\mathbf{W}\mathbf{u}^{2\dagger}$
$^{1}$ MIT,  $^{2}$ Cornell University
We consider the problem of Imitation Learning (IL) by actively querying noisy expert for feedback. While imitation learning has been empirically successful, much of prior work assumes access to noiseless expert feedback which is not practical in many applications. In fact, when one only has access to noisy expert feedback, algorithms that rely on purely offline data (non-interactive IL) can be shown to need a prohibitively large number of samples to be successful. In contrast, in this work, we provide an interactive algorithm for IL that uses selective sampling to actively query the noisy expert for feedback. Our contributions are twofold: First, we provide a new selective sampling algorithm that works with general function classes and multiple actions, and obtains the best-known bounds for the regret and the number of queries. Next, we extend this analysis to the problem of IL with noisy expert feedback and provide a new IL algorithm that makes limited queries. Our algorithm for selective sampling leverages function approximation, and relies on an online regression oracle w.r.t. the given model class to predict actions, and to decide whether to query the expert for its label. On the theoretical side, the regret bound of our algorithm is upper bounded by the regret of the online regression oracle, while the query complexity additionally depends on the eluder dimension of the model class. We complement this with a lower bound that demonstrates that our results are tight. We extend our selective sampling algorithm for IL with general function approximation and provide bounds on both the regret and the number of queries made to the noisy expert. A key novelty here is that our regret and query complexity bounds only depend on the number of times the optimal policy (and not the noisy expert, or the learner) go to states that have a small margin.
From the classic supervised learning setting to the more complex problems like interactive Imitation Learning (IL) [Ross et al., 2011], high-quality labels or supervision is often expensive and hard to obtain. Thus, one wishes to develop algorithms that do not require a label for every data sample presented during the learning process. Active learning or selective sampling is a learning paradigm that is designed to reduce query complexity by only querying for labels at selected data points, and has been extensively studied in both theory and practice [Agarwal, 2013, Dekel et al., 2012, Hanneke and Yang, 2021, Zhu and Nowak, 2022, Cesa-Bianchi et al., 2005, Hanneke and Yang, 2015].
In this work, we study selective sampling and its application to interactive Imitation Learning [Ross et al., 2011]. Our goal is to design algorithms that can leverage general function approximation and online regression oracles to achieve small regret on predicting the correct labels, and at the
$^{\dagger}$ Emails: sekhari@mit.edu, {ks999, ws455, rw646}@cornell.edu
same time minimize the number of expert queries made (query complexity). Towards this goal, we first study selective sampling which is an online active learning framework, and provide regret and query complexity bounds for general function classes (used to model the experts). Our key results in selective sampling are obtained by developing a connection between the regret of the online regression oracles and the regret of predicting the correct labels. Additionally, we bound the query complexity using the eluder dimension [Russo and Van Roy, 2013] of the underlying function class used to model the expert. We complement our results with a lower bound indicating that a dependence on an eluder dimension like complexity measure is unavoidable in the query complexity in the worst case. In particular, we provide lower bounds in terms of the star number of the function class—a quantity closely related to the eluder dimension. Our new selective sampling algorithm, called SAGE, can operate under fairly general modeling assumptions, loss functions, and allows for multiple labels (i.e., multi-class classification).
We then extend our selective sampling algorithm to the interactive IL framework proposed by Ross et al. [2011] to reduce the query complexity. While the DAgger algorithm proposed by Ross et al. [2011] has been extensively used in various robotics applications (e.g., Ross et al. [2013], Pan et al. [2018]), it often requires a large number of expert queries. There have been some efforts on reducing the expert query complexity by leveraging ideas from active learning (e.g., Laskey et al. [2016], Brantley et al. [2020]), however, these prior attempts do not have theoretical guarantees on bounding expert's query complexity. In this work, we provide the first provably correct algorithm for interactive IL with general function classes, called RAVIOLI, which not only achieves strong regret bounds in terms of maximizing the underlying reward functions, but also enjoys a small query complexity. Furthermore, we note that RAVIOLI operates under significantly weaker assumptions as compared to the prior works, like Ross et al. [2011], on interactive IL. In particular, we only assume access to a noisy expert, as compared to the prior works that assume that the expert is noiseless. In fact, for the noisy setting, we show that one can not even hope to learn from purely offline expert demonstrations unless one has exponentially in horizon  $H$  many samples. Such a strong separation does not hold in the noiseless setting.
Our bounds depend on the margin of the noisy expert, which intuitively quantifies the confidence level of the expert. In particular, the margin is large for states where the expert is very confident in terms of providing the correct labels, while on the other hand, the margin is small on the states where the expert is less confident and subsequently provides more noisy labels as feedback. Such kind of margin condition was missing in prior works, like Ross et al. [2011], which assumes that the expert can provide confident labels everywhere. Additionally, we note that our margin assumption is quite mild as we only assume that the expert has a large margin under the states that could be visited by the noiseless expert (however, the states visited by the learner, or by following the noisy expert, may not have a small margin).
We then extend our results to the multiple expert setting where the learner has access to  $M$  many experts/teachers who may have different expertise at different parts of the state space. In particular, there is no expert who can singlehandedly perform well on the underlying environment, but an aggregation of their policies can lead to good performance. Such an assumption holds in various applications and has been recently explored in continuous control tasks like robotics and discrete tasks like chess and Minigrid [Beliaev et al., 2022]. Similar to the single expert setting, we model the expertise of the experts in multiple expert setting using the concept of margins. Different experts have different margin functions, capturing the fact that experts may have different expertise at different parts of the state space. Prior work from Cheng et al. [2020] also considers multiple experts in IL and provides meaningful regret bounds, however, their assumption on the experts is much stronger than us: they assume that for any state, there at least exists one expert who can achieve high reward-to-go if the expert took over the control starting from this state till the end of the episode. Furthermore, Cheng et al. [2020] considers the setting where one can also query for the reward signals, whereas we do not require access to any reward signals.
2 Contributions and Overview of Results
2.1 Selective Sampling
Online selective sampling models the interaction between a learner and an adversary over  $T$  rounds. At the beginning of each round of the interaction, the adversary presents a context  $x_{t}$  to the learner. After receiving the context, the learner makes a prediction  $\hat{y}_t \in [K]$ , where  $K$  denotes the number of
actions. Then, the learner needs to make a choice of whether or not to query an expert who is assumed to have some knowledge about the true label for all the presented contexts. The experts knowledge about the true label is modeled via the ground truth modeling function  $f^{\star}$ , which is assumed to belong to a given function class  $\mathcal{F}$  but is unknown to the learner. If the learner decides to query for the label, then the expert will return a noisy label  $y_{t}$  sampled using  $f^{\star}$ . If the learner does not query, then the learner does not receive any feedback in this round. The learner makes an update based on the latest information it has, and moves on to the next round of the interaction. The goal of the learner is to compete with the expert policy  $\pi^{\star}$ , that is defined using the experts model  $f^{\star}$ . In the selective sampling setting, we are concerned with two things: the total regret of the learner w.r.t. the policy  $\pi^{\star}$ , and the number of expert queries that the learner makes. Our key contributions are as follows:
- We provide a new selective sampling algorithm (Algorithm 1) that relies on an online regression oracle w.r.t.  $\mathcal{F}$  (where  $\mathcal{F}$  is the given model class) to make predictions and to decide whether to query for labels. Our algorithm can handle multiple actions, adversarial contexts, arbitrary model class  $\mathcal{F}$ , and fairly general modeling assumptions (that we discuss in more detail in Section 3), and enjoys the following regret bound and query complexity:
$$
\operatorname {R e g} _ {T} = \widetilde {\mathcal {O}} \left(\inf  _ {\varepsilon} \left\{\varepsilon T _ {\varepsilon} + \frac {\operatorname {R e g} (\mathcal {F} ; T)}{\varepsilon} \right\}\right) \quad \text {a n d} \quad N _ {T} = \widetilde {\mathcal {O}} \left(\inf  _ {\varepsilon} \left\{T _ {\varepsilon} + \frac {\operatorname {R e g} (\mathcal {F} ; T) \cdot \mathfrak {E} (\mathcal {F} , \varepsilon ; f ^ {\star})}{\varepsilon^ {2}} \right\}\right). \tag {1}
$$
where  $\operatorname{Reg}(\mathcal{F};T)$  denotes the regret bound for the online regression oracle on  $\mathcal{F}$ ,  $\mathfrak{E}(\mathcal{F},\varepsilon ;f^{\star})$  denotes the eluder dimension of  $\mathcal{F}$ , and  $T_{\varepsilon}$  denotes the number of rounds at which the margin of the experts predictions is smaller than  $\varepsilon$  (the exact notion of margin is defined in Section 3).
- We show via a lower bound that, without additional assumptions, the dependence on the eluder dimension in the query complexity bound (1) is unavoidable if we desire a regret bound of the form (1), even when  $T_{\varepsilon} = 0$ . The details are located in Section 3.2.
- For the stochastic setting, where the context  $\{x_{t}\}_{t\leq T}$  are sampled i.i.d. from a fixed unknown distribution, we provide an alternate algorithm (Algorithm 3) that enjoys the same regret bound as (1) but whose query complexity scales with the disagreement coefficient of  $\mathcal{F}$  instead of the eluder dimension (Theorem 2). Since the disagreement coefficient is always smaller than the eluder dimension, Theorem 2 yields an improvement in the query complexity.
2.2 Imitation Learning
We then move to the more challenging Imitation Learning (IL) setting, where the learner operates in an episodic finite horizon Markov Decision Process (MDP), and can query a noisy expert for feedback (i.e. the expert action) on the states that it visits. The interaction proceeds in  $T$  episodes of length  $H$  each. In episode  $t$ , at each time step  $h \in [H]$  and on the state  $x_{t,h}$ , the learner chooses an action  $\widehat{y}_{t,h}$  and transitions to state  $x_{t,h+1}$ . However, the learner does not receive any reward signal. Instead, the learner can actively choose to query an expert who has some knowledge about the correct action to be taken on  $x_{t,h}$ , and gives back noisy feedback  $y_{t,h}$  about this action. Similar to the selective sampling setting, the experts know about the true label is modeled via the ground truth modeling function  $f_h^{\star}$ , which is assumed to belong to a given function class  $\mathcal{F}_h$  but is unknown to the learner. The goal of the learner is to compete with the optimal policy  $\pi^{\star}$  of the (noiseless) expert. Our key contributions in IL are:
- In Section 4, we first demonstrate an exponential separation in terms of task horizon  $H$  in the sample complexity, for learning via offline expert demonstration only vs interactive querying of experts, when the feedback from the expert is noisy.
- We then provide a general IL algorithm (in Algorithm 2) that relies on online regression oracles w.r.t.  $\{\mathcal{F}_h\}_{h \leq H}$  to predict actions, and to decide whether to query for labels. Similar to the selective sampling setting, the regret bound for our algorithm scales with the regret of the online regression oracles, and the query complexity bound has an additional dependence on the eluder dimension. Furthermore, our algorithm can handle multiple actions, adversarially changing dynamics, arbitrary model class  $\mathcal{F}$ , and fairly general modeling assumptions.
- A key difference from our results in selective sampling is that the term  $T_{\varepsilon}$  that appears in our regret and query complexity bounds in IL denote the number of time steps in which the expert
policy  $\pi^{\star}$  has a small margin (instead of the number of time steps when the learner's policy has a small margin). In fact, the learner and the expert trajectories could be completely different from each other, and we only pay in the margin term if the expert trajectory at that time step would have a low margin. See Section 4 for the exact definition of margin.
- In Section 4.1, we provide extensions to our algorithm when the learner can query  $M$  experts at each round. In particular, we do not assume that any of the experts is singlehandedly optimal for the entire state space, but that there exist aggregation functions of these experts' predictions that perform well in practice, and with which we compete.
3 Selective Sampling
In the problem of selective sampling, on every round  $t$ , nature (or an adversary) produces a context  $x_{t}$ . The learner then receives this context and predicts a label  $\widehat{y}_{t} \in [K]$  for that context. The learner also computes a query condition  $Z_{t} \in \{0,1\}$  for that context. If  $Z_{t} = 1$ , the learner requests for label  $y_{t} \in [K]$  corresponding to the  $x_{t}$ , and if not, the learner receives no feedback on the label for that round. Let  $\mathcal{F}$  be a model class such that each model  $f \in \mathcal{F}$  maps contexts  $x$  to scores  $f(x) \in \mathbb{R}^{K}$ . In this work we assume that while contexts can be chosen arbitrarily, the label  $y_{t}$  corresponding to a context  $x_{t}$  is drawn from a distribution over labels specified by the score  $f^{\star}(x_{t})$  where  $f^{\star} \in \mathcal{F}$  is a fixed model unknown to the learner. We assume that a link function  $\phi : \mathbb{R}^{K} \mapsto \Delta(K)$  maps scores to distributions and assume that the noisy label  $y_{t}$  is sampled as
$$
y _ {t} \sim \phi \left(f ^ {\star} \left(x _ {t}\right)\right). \tag {2}
$$
In this work, we assume that the link function  $\phi(v) = \nabla \Phi(v)$  for some  $\Phi: \mathbb{R}^K \mapsto \mathbb{R}$  (see Agarwal [2013] for more details) which satisfies the following assumption.
Assumption 1. The function  $\Phi$  is  $\lambda$ -strongly-convex and  $\gamma$ -smooth, i.e. for all  $u, u' \in \mathbb{R}^K$ ,
$$
\frac {\lambda}{2} \left\| u ^ {\prime} - u \right\| _ {2} ^ {2} \leq \Phi (u ^ {\prime}) - \Phi (u) - \left\langle \nabla \Phi (u), u ^ {\prime} - u \right\rangle \leq \frac {\gamma}{2} \left\| u ^ {\prime} - u \right\| _ {2} ^ {2}.
$$
Our main contribution in this section is a selective sampling algorithm that uses online non-parametric regression w.r.t. the model class  $\mathcal{F}$  as a black box. Specifically, define the loss function corresponding to the link function  $\phi$  as  $\ell_{\phi}(v,y) = \Phi (v) - v[y]$  where  $v\in \mathbb{R}^{K}$  and  $y\in [K]$ . We assume that the learner has access to an online regression oracle for the loss  $\ell_{\phi}$  (which is a convex loss) w.r.t. the class  $\mathcal{F}$ , that for any sequence  $\{(x_1,y_1),\ldots ,(x_T,y_T)\}$  guarantees the regret bound
$$
\sum_ {s = 1} ^ {T} \ell_ {\phi} \left(f _ {s} \left(x _ {s}\right), y _ {s}\right) - \inf  _ {f \in \mathcal {F}} \sum_ {s = 1} ^ {T} \ell_ {\phi} \left(f \left(x _ {s}\right), y _ {s}\right) \leq \operatorname {R e g} ^ {\ell_ {\phi}} (\mathcal {F}; T). \tag {3}
$$
When  $\phi$  is identity (under which the models in  $\mathcal{F}$  directly map to distributions over the labels), then  $\ell_{\phi}$  denotes the standard square loss, and we need a bound on the regret w.r.t. the square loss, denoted by  $\mathrm{Reg}^{\mathrm{sq}}(\mathcal{F};T)$ . When  $\phi$  is the Boltzman distribution mapping (given by  $\Phi$  being the softmax function) then  $\ell_{\phi}$  is the logistic loss, and we need an online logistic regression oracle for  $\mathcal{F}$ . Minimax rates for the regret bound in (3) are well known:
- Square-loss regression: Rakhlin and Sridharan [2014] characterized the minimax rates for online square loss regression in terms of the offset sequential Rademacher complexity of  $\mathcal{F}$ , which for example, leads to regret bound  $\mathrm{Reg}^{\mathrm{sq}}(\mathcal{F};T) = O(\log |\mathcal{F}|)$  for finite function classes  $\mathcal{F}$ , and  $\mathrm{Reg}^{\mathrm{sq}}(\mathcal{F};T) = O(d\log (T))$  when  $\mathcal{F}$  is a  $d$ -dimensional linear class. More examples can be found in Rakhlin and Sridharan [2014, Section 4]. We refer the readers to Krishnamurthy et al. [2017], Foster et al. [2018a] for efficient implementations.
- Logistic-loss regression: When  $\mathcal{F}$  is finite, we have the regret bound  $\operatorname{Reg}(\mathcal{F};T) \leq O(\log |\mathcal{F}|)$  [Cesa-Bianchi and Lugosi, 2006, Chapter 9]. For learning linear predictors, there exists efficient improper learner with regret bound  $\operatorname{Reg}(\mathcal{F};T) \leq O(d\log |T|)$  [Foster et al., 2018b]. More examples can be found in Foster et al. [2018b, Section 7] and Rakhlin and Sridharan [2015].
When one deals with complex model classes  $\mathcal{F}$  such that the labeling concept class corresponding to  $\mathcal{F}$  could possibly have infinite VC dimension (like it is typically the case), then one needs to naturally
rely on a margin-based analysis [Tsybakov, 2004, Shalev-Shwartz and Ben-David, 2014, Dekel et al., 2012]. For  $p \in \mathbb{R}^K$ , we use the following well-known notion of margin for multiclass settings:
$$
\operatorname {M a r g i n} (p) = \phi (p) \left[ k ^ {\star} \right] - \max  _ {k ^ {\prime} \neq k ^ {\star}} \phi (p) \left[ k ^ {\prime} \right] \quad \text {w h e r e} \quad k ^ {\star} \in \underset {k} {\operatorname {a r g m a x}} \phi (p) [ k ], \tag {4}
$$
A key quantity that appears in our results is the number of  $x_{t}$ 's that fall within an  $\varepsilon$  margin region,
$$
T _ {\varepsilon} = \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\operatorname {M a r g i n} \left(f ^ {\star} \left(x _ {t}\right)\right) \leq \varepsilon \right\}.
$$
$T_{\varepsilon}$  denotes the number of times where even the Bayes optimal classifier is confused about the correct label on  $x_{t}$ , and has confidence less than  $\varepsilon$ . The algorithm relies on an online regression oracle mentioned above to produce the predictor  $f_{t}$  at every round. The predicted label  $\widehat{y}_{t} = \text{SelectAction}(f_{t}(x_{t})) = \arg \max_{k} \phi(f_{t}(x_{t}))[k]$  is picked based on the score  $f_{t}(x_{t})$  (where  $\widehat{y}_{t}$  is the label with the largest score). The learner updates the regression oracle on only those rounds in which it makes a query. Our main algorithm for selective sampling is provided in Algorithm 1.
Algorithm 1 Selective SAmplinG with Expert Feedback (SAGE)
Input: Parameters  $\delta, \gamma, \lambda, T$ , function class  $\mathcal{F}$ , and online regression oracle Oracle w.r.t  $\ell_{\phi}$ .
1: Set  $\Psi_{\delta}^{\ell_{\phi}}(\mathcal{F},T) = \frac{4}{\lambda}\mathrm{Reg}^{\ell_{\phi}}(\mathcal{F};T) + \frac{112}{\lambda^2}\log (4\log^2 (T) / \delta)$ , Compute  $f_{1}\gets \operatorname {O r a c l e}_{1}(\varnothing)$ .
2: for  $t = 1$  to  $T$  do
3: Nature chooses  $x_{t}$ .
4: Learner plays the action  $\widehat{y}_t = \text{SelectAction}(f_t(x_t))$ .
5: Learner computes
$$
\Delta_ {t} \left(x _ {t}\right) := \max  _ {f \in \mathcal {F}} \| f (x _ {t}) - f _ {t} (x _ {t}) \| \quad \text {s . t .} \quad \sum_ {s = 1} ^ {t - 1} Z _ {s} \| f (x _ {s}) - f _ {s} (x _ {s}) \| ^ {2} \leq \Psi_ {\delta} ^ {\ell_ {\phi}} (\mathcal {F}, T). \tag {5}
$$
6: Learner decides whether to query:  $Z_{t} = \mathbf{1}\{\mathrm{Margin}(f_{t}(x_{t}))\leq 2\gamma \Delta_{t}(x_{t})\}$
7: if  $Z_{t} = 1$  then
8: Learner queries the label  $y_{t}$  on  $x_{t}$ .
9:  $f_{t + 1}\gets \mathrm{O r a c l e}_{t}(\{x_{t},y_{t}\})$
10: else
11:  $f_{t + 1}\gets f_t$
Our goal in this work is twofold: Firstly, we would like Algorithm 2 to have a low regret w.r.t. the optimal model  $f^{\star}$ , defined as
$$
\operatorname {R e g} _ {T} = \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\widehat {y} _ {t} \neq y _ {t} \right\} - \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\text {S e l e c t A c t i o n} \left(f ^ {\star} \left(x _ {t}\right)\right) \neq y _ {t} \right\}
$$
Simultaneously, we also aim to make as few label queries  $N_{T} = \sum_{t=1}^{T} Z_{t}$  as possible. Before delving into our results, we first recall the following variant of eluder-dimension [Russo and Van Roy, 2013, Foster et al., 2020, Zhu and Nowak, 2022].
Definition 1 (Scale-sensitive eluder dimension (normed version)). Fix any  $f^{\star} \in \mathcal{F}$ , and define  $\widetilde{\mathfrak{E}}(\mathcal{F},\beta;f^{\star})$  to be the length of the longest sequence of contexts  $x_{1},x_{2},\ldots x_{m}$  such that for all  $i$ , there exists  $f_{i} \in \mathcal{F}$  such that
$$
\left\| f _ {i} \left(x _ {i}\right) - f ^ {\star} \left(x _ {i}\right) \right\| > \beta , \quad a n d \quad \sum_ {j <   i} \left\| f _ {i} \left(x _ {j}\right) - f ^ {\star} \left(x _ {j}\right) \right\| ^ {2} \leq \beta^ {2}.
$$
The value function eluder dimension is defined as  $\mathfrak{E}(\mathcal{F},\beta^{\prime};f^{\star}) = \sup_{\beta \geq \beta^{\prime}}\widetilde{\mathfrak{E}} (\mathcal{F},\beta ;f^{\star})$
Bounds on the eluder dimension for various function classes are well known, e.g. when  $\mathcal{F}$  is finite,  $\mathfrak{E}(\mathcal{F},\beta^{\prime};f^{\star})\leq |\mathcal{F}| - 1$ , and when  $\mathcal{F}$  is the set of  $d$ -dimensional functions with bounded norm, then  $\mathfrak{E}(\mathcal{F},\beta^{\prime};f^{\star}) = O(d)$ . We refer the reader to Russo and Van Roy [2013], Mou et al. [2020], Li et al. [2022] for more examples. The following theorem is our main result for selective sampling:
3 Throughout the paper, we assume that the ties in argmax or argmin are broken arbitrarily, but consistently.
4 Unless explicitly specified, the action set is given by  $\mathcal{A} = [K] = \{1,\dots ,K\}$  where  $K\geq 2$
Theorem 1. Let  $\delta \in (0,1)$ . Under the modeling assumptions above (in (2), (3) and (4)), with probability at least  $1 - \delta$ , Algorithm 1 obtains the regret bound