Title: Selective Sampling and Imitation Learning via Online Regression
Venue: NeurIPS 2023
Accepted: yes
Keywords: Selective Sampling

Imitation Learning

Learning from Expert Feedback

Theory

General purpose algorithms
Abstract: We consider the problem of Imitation Learning (IL) by actively querying noisy expert for feedback. While imitation learning has been empirically successful, much of prior work assumes access to noiseless expert feedback which is not practical in many applications. In fact, when one only has access to noisy expert feedback, algorithms that rely on purely offline data (non-interactive IL) can be shown to need a prohibitively large number of samples to be successful. In contrast, in this work, we provide an interactive algorithm for IL that uses selective sampling to actively query the noisy expert for feedback. Our contributions are twofold: First,  we provide a new selective sampling algorithm that works with general function classes and multiple actions, and obtains the best-known bounds for the regret and the number of queries. Next, we extend this analysis to the problem of IL with noisy expert feedback and provide a new IL algorithm that  makes limited queries. 

 Our algorithm for selective sampling leverages function approximation, and relies on an online regression oracle w.r.t.~the given model class to predict actions, and to decide whether to query the expert for its label. On the theoretical side, the regret bound of our algorithm is upper bounded by the regret of the online regression oracle, while the query complexity additionally depends on the eluder dimension of the model class. We complement this with a  lower bound that demonstrates that our results are tight. We extend our selective sampling algorithm for IL with general function approximation and provide bounds on both the regret and the number of queries made to the noisy expert. A key novelty here is that our regret and query complexity bounds only depend on the number of times the optimal policy (and not the noisy expert, or the learner) go to states that have a small margin.

Selective Sampling and Imitation Learning via Online Regression

Abstract

We consider the problem of Imitation Learning (IL) by actively querying noisy expert for feedback. While imitation learning has been empirically successful, much of prior work assumes access to noiseless expert feedback which is not practical in many applications. In fact, when one only has access to noisy expert feedback, algorithms that rely on purely offline data (non-interactive IL) can be shown to need a prohibitively large number of samples to be successful. In contrast, in this work, we provide an interactive algorithm for IL that uses selective sampling to actively query the noisy expert for feedback. Our contributions are twofold: First, we provide a new selective sampling algorithm that works with general function classes and multiple actions, and obtains the best-known bounds for the regret and the number of queries. Next, we extend this analysis to the problem of IL with noisy expert feedback and provide a new IL algorithm that makes limited queries. Our algorithm for selective sampling leverages function approximation, and relies on an online regression oracle w.r.t. the given model class to predict actions, and to decide whether to query the expert for its label. On the theoretical side, the regret bound of our algorithm is upper bounded by the regret of the online regression oracle, while the query complexity additionally depends on the eluder dimension of the model class. We complement this with a lower bound that demonstrates that our results are tight. We extend our selective sampling algorithm for IL with general function approximation and provide bounds on both the regret and the number of queries made to the noisy expert. A key novelty here is that our regret and query complexity bounds only depend on the number of times the optimal policy (and not the noisy expert, or the learner) go to states that have a small margin.

1 Introduction

From the classic supervised learning setting to the more complex problems like interactive Imitation Learning (IL) [Ross et al., 2011], high-quality labels or supervision is often expensive and hard to obtain. Thus, one wishes to develop algorithms that do not require a label for every data sample presented during the learning process. Active learning or selective sampling is a learning paradigm that is designed to reduce query complexity by only querying for labels at selected data points, and has been extensively studied in both theory and practice [Agarwal, 2013, Dekel et al., 2012, Hanneke and Yang, 2021, Zhu and Nowak, 2022, Cesa-Bianchi et al., 2005, Hanneke and Yang, 2015].

In this work, we study selective sampling and its application to interactive Imitation Learning [Ross et al., 2011]. Our goal is to design algorithms that can leverage general function approximation and online regression oracles to achieve small regret on predicting the correct labels, and at the

same time minimize the number of expert queries made (query complexity). Towards this goal, we first study selective sampling which is an online active learning framework, and provide regret and query complexity bounds for general function classes (used to model the experts). Our key results in selective sampling are obtained by developing a connection between the regret of the online regression oracles and the regret of predicting the correct labels. Additionally, we bound the query complexity using the eluder dimension [Russo and Van Roy, 2013] of the underlying function class used to model the expert. We complement our results with a lower bound indicating that a dependence on an eluder dimension like complexity measure is unavoidable in the query complexity in the worst case. In particular, we provide lower bounds in terms of the star number of the function class-a quantity closely related to the eluder dimension. Our new selective sampling algorithm, called SAGE , can operate under fairly general modeling assumptions, loss functions, and allows for multiple labels (i.e., multi-class classification).

We then extend our selective sampling algorithm to the interactive IL framework proposed by Ross et al. [2011] to reduce the query complexity. While the DAgger algorithm proposed by Ross et al. [2011] has been extensively used in various robotics applications (e.g., Ross et al. [2013], Pan et al. [2018]), it often requires a large number of expert queries. There have been some efforts on reducing the expert query complexity by leveraging ideas from active learning (e.g., Laskey et al. [2016], Brantley et al. [2020]), however, these prior attempts do not have theoretical guarantees on bounding expert's query complexity. In this work, we provide the first provably correct algorithm for interactive IL with general function classes, called RAVIOLI , which not only achieves strong regret bounds in terms of maximizing the underlying reward functions, but also enjoys a small query complexity. Furthermore, we note that RAVIOLI operates under significantly weaker assumptions as compared to the prior works, like Ross et al. [2011], on interactive IL. In particular, we only assume access to a noisy expert, as compared to the prior works that assume that the expert is noiseless. In fact, for the noisy setting, we show that one can not even hope to learn from purely offline expert demonstrations unless one has exponentially in horizon H many samples. Such a strong separation does not hold in the noiseless setting.

Our bounds depend on the margin of the noisy expert, which intuitively quantifies the confidence level of the expert. In particular, the margin is large for states where the expert is very confident in terms of providing the correct labels, while on the other hand, the margin is small on the states where the expert is less confident and subsequently provides more noisy labels as feedback. Such kind of margin condition was missing in prior works, like Ross et al. [2011], which assumes that the expert can provide confident labels everywhere. Additionally, we note that our margin assumption is quite mild as we only assume that the expert has a large margin under the states that could be visited by the noiseless expert (however, the states visited by the learner, or by following the noisy expert, may not have a small margin).

We then extend our results to the multiple expert setting where the learner has access to M many experts/teachers who may have different expertise at different parts of the state space. In particular, there is no expert who can singlehandedly perform well on the underlying environment, but an aggregation of their policies can lead to good performance. Such an assumption holds in various applications and has been recently explored in continuous control tasks like robotics and discrete tasks like chess and Minigrid [Beliaev et al., 2022]. Similar to the single expert setting, we model the expertise of the experts in multiple expert setting using the concept of margins. Different experts have different margin functions, capturing the fact that experts may have different expertise at different parts of the state space. Prior work from Cheng et al. [2020] also considers multiple experts in IL and provides meaningful regret bounds, however, their assumption on the experts is much stronger than us: they assume that for any state, there at least exists one expert who can achieve high reward-to-go if the expert took over the control starting from this state till the end of the episode. Furthermore, Cheng et al. [2020] considers the setting where one can also query for the reward signals, whereas we do not require access to any reward signals.

2 Contributions and Overview of Results

2.1 Selective Sampling

Online selective sampling models the interaction between a learner and an adversary over T rounds. At the beginning of each round of the interaction, the adversary presents a context x t to the learner. After receiving the context, the learner makes a prediction ˆ y t ∈ [ K ] , where K denotes the number of

actions. Then, the learner needs to make a choice of whether or not to query an expert who is assumed to have some knowledge about the true label for all the presented contexts. The experts knowledge about the true label is modeled via the ground truth modeling function f ⋆ , which is assumed to belong to a given function class F but is unknown to the learner. If the learner decides to query for the label, then the expert will return a noisy label y t sampled using f ⋆ . If the learner does not query, then the learner does not receive any feedback in this round. The learner makes an update based on the latest information it has, and moves on to the next round of the interaction. The goal of the learner is to compete with the expert policy π ⋆ , that is defined using the experts model f ⋆ . In the selective sampling setting, we are concerned with two things: the total regret of the learner w.r.t. the policy π ⋆ , and the number of expert queries that the learner makes. Our key contributions are as follows:

We provide a new selective sampling algorithm (Algorithm 1) that relies on an online regression oracle w.r.t. F (where F is the given model class) to make predictions and to decide whether to query for labels. Our algorithm can handle multiple actions, adversarial contexts, arbitrary model class F , and fairly general modeling assumptions (that we discuss in more detail in Section 3), and enjoys the following regret bound and query complexity:

where Reg (F ; T ) denotes the regret bound for the online regression oracle on F , E (F , ε ; f ⋆ ) denotes the eluder dimension of F , and T ε denotes the number of rounds at which the margin of the experts predictions is smaller than ε (the exact notion of margin is defined in Section 3).

We show via a lower bound that, without additional assumptions, the dependence on the eluder dimension in the query complexity bound (1) is unavoidable if we desire a regret bound of the form (1), even when T ε = 0 . The details are located in Section 3.2.

For the stochastic setting, where the context { x t } t ≤ T are sampled i.i.d. from a fixed unknown distribution, we provide an alternate algorithm (Algorithm 3) that enjoys the same regret bound as (1) but whose query complexity scales with the disagreement coefficient of F instead of the eluder dimension (Theorem 2). Since the disagreement coefficient is always smaller than the eluder dimension, Theorem 2 yields an improvement in the query complexity.

2.2 Imitation Learning

We then move to the more challenging Imitation Learning (IL) setting, where the learner operates in an episodic finite horizon Markov Decision Process (MDP), and can query a noisy expert for feedback (i.e. the expert action) on the states that it visits. The interaction proceeds in T episodes of length H each. In episode t , at each time step h ∈ [ H ] and on the state x t,h , the learner chooses an action ̂ y t,h and transitions to state x t,h + 1 . However, the learner does not receive any reward signal. Instead, the learner can actively choose to query an expert who has some knowledge about the correct action to be taken on x t,h , and gives back noisy feedback y t,h about this action. Similar to the selective sampling setting, the experts knowledge about the true label is modeled via the ground truth modeling function f ⋆ h , which is assumed to belong to a given function class F h but is unknown to the learner. The goal of the learner is to compete with the optimal policy π ⋆ of the (noiseless) expert. Our key contributions in IL are:

In Section 4, we first demonstrate an exponential separation in terms of task horizon H in the sample complexity, for learning via offline expert demonstration only vs interactive querying of experts, when the feedback from the expert is noisy.

We then provide a general IL algorithm (in Algorithm 2) that relies on online regression oracles w.r.t. {F h } h ≤ H to predict actions, and to decide whether to query for labels. Similar to the selective sampling setting, the regret bound for our algorithm scales with the regret of the online regression oracles, and the query complexity bound has an additional dependence on the eluder dimension. Furthermore, our algorithm can handle multiple actions, adversarially changing dynamics, arbitrary model class F , and fairly general modeling assumptions.

A key difference from our results in selective sampling is that the term T ε that appears in our regret and query complexity bounds in IL denote the number of time steps in which the expert

policy π ⋆ has a small margin (instead of the number of time steps when the learner's policy has a small margin). In fact, the learner and the expert trajectories could be completely different from each other, and we only pay in the margin term if the expert trajectory at that time step would have a low margin. See Section 4 for the exact definition of margin.

In Section 4.1, we provide extensions to our algorithm when the learner can query M experts at each round. In particular, we do not assume that any of the experts is singlehandedly optimal for the entire state space, but that there exist aggregation functions of these experts' predictions that perform well in practice, and with which we compete.

3 Selective Sampling

In the problem of selective sampling, on every round t , nature (or an adversary) produces a context x t . The learner then receives this context and predicts a label ̂ y t ∈ [ K ] for that context. The learner also computes a query condition Z t ∈ { 0 , 1 } for that context. If Z t = 1 , the learner requests for label y t ∈ [ K ] corresponding to the x t , and if not, the learner receives no feedback on the label for that round. Let F be a model class such that each model f ∈ F maps contexts x to scores f ( x ) ∈ R K . In this work we assume that while contexts can be chosen arbitrarily, the label y t corresponding to a context x t is drawn from a distribution over labels specified by the score f ⋆ ( x t ) where f ⋆ ∈ F is a fixed model unknown to the learner. We assume that a link function ϕ ∶ R K ↦ ∆ ( K ) maps scores to distributions and assume that the noisy label y t is sampled as

In this work, we assume that the link function ϕ ( v ) = ∇ Φ ( v ) for some Φ ∶ R K ↦ R (see Agarwal [2013] for more details) which satisfies the following assumption.

Assumption 1. The function Φ is λ -strongly-convex and γ -smooth, i.e. for all u, u ′ ∈ R K ,

Our main contribution in this section is a selective sampling algorithm that uses online non-parametric regression w.r.t. the model class F as a black box. Specifically, define the loss function corresponding to the link function ϕ as ℓ ϕ ( v, y ) = Φ ( v ) -v [ y ] where v ∈ R K and y ∈ [ K ] . We assume that the learner has access to an online regression oracle for the loss ℓ ϕ (which is a convex loss) w.r.t. the class F , that for any sequence {( x 1 , y 1 ) , . . . , ( x T , y T )} guarantees the regret bound

When ϕ is identity (under which the models in F directly map to distributions over the labels), then ℓ ϕ denotes the standard square loss, and we need a bound on the regret w.r.t. the square loss, denoted by Reg sq (F ; T ) . When ϕ is the Boltzman distribution mapping (given by Φ being the softmax function) then ℓ ϕ is the logistic loss, and we need an online logistic regression oracle for F . Minimax rates for the regret bound in (3) are well known:

Square-loss regression: Rakhlin and Sridharan [2014] characterized the minimax rates for online square loss regression in terms of the offset sequential Rademacher complexity of F , which for example, leads to regret bound Reg sq (F ; T ) = O ( log ∣F∣) for finite function classes F , and Reg sq (F ; T ) = O ( d log ( T )) when F is a d -dimensional linear class. More examples can be found in Rakhlin and Sridharan [2014, Section 4]. We refer the readers to Krishnamurthy et al. [2017], Foster et al. [2018a] for efficient implementations.

Logistic-loss regression: When F is finite, we have the regret bound Reg (F ; T ) ≤ O ( log ∣F∣) [Cesa-Bianchi and Lugosi, 2006, Chapter 9]. For learning linear predictors, there exists efficient improper learner with regret bound Reg (F ; T ) ≤ O ( d log ∣ T ∣) [Foster et al., 2018b]. More examples can be found in Foster et al. [2018b, Section 7] and Rakhlin and Sridharan [2015].

When one deals with complex model classes F such that the labeling concept class corresponding to F could possibly have infinite VC dimension (like it is typically the case), then one needs to naturally

rely on a margin-based analysis [Tsybakov, 2004, Shalev-Shwartz and Ben-David, 2014, Dekel et al., 2012]. For p ∈ R K , we use the following well-known notion of margin for multiclass settings 3 :

A key quantity that appears in our results is the number of x t 's that fall within an ε margin region,

T ε denotes the number of times where even the Bayes optimal classifier is confused about the correct label on x t , and has confidence less than ε . The algorithm relies on an online regression oracle mentioned above to produce the predictor f t at every round. The predicted label ̂ y t = SelectAction ( f t ( x t )) = argmax k ϕ ( f t ( x t ))[ k ] is picked based on the score f t ( x t ) (where ̂ y t is the label with the largest score). The learner updates the regression oracle on only those rounds in which it makes a query. Our main algorithm for selective sampling is provided in Algorithm 1. 4

Algorithm 1 S elective S A mplin G with E xpert Feedback ( SAGE )

Input: Parameters δ, γ, λ, T , function class F , and online regression oracle Oracle w.r.t ℓ ϕ .

1: Set Ψ ℓ ϕ δ (F , T ) = 4 λ Reg ℓ ϕ (F ; T ) + 112 λ 2 log ( 4log 2 ( T )/ δ ) , Compute f 1 ← Oracle 1 (∅) .

2: for t = 1 to T do

3: Nature chooses x t .

4: Learner plays the action ̂ y t = SelectAction ( f t ( x t )) .

5: Learner computes

6: Learner decides whether to query: Z t = 1 { Margin ( f t ( x t )) ≤ 2 γ ∆ t ( x t )} .

7: if Z t = 1 then

8: Learner queries the label y t on x t .

9: f t + 1 ← Oracle t ({ x t , y t }) .

10: else

11: f t + 1 ← f t .

Our goal in this work is twofold: Firstly, we would like Algorithm 2 to have a low regret w.r.t. the optimal model f ⋆ , defined as

Simultaneously, we also aim to make as few label queries N T = ∑ T t = 1 Z t as possible. Before delving into our results, we first recall the following variant of eluder-dimension [Russo and Van Roy, 2013, Foster et al., 2020, Zhu and Nowak, 2022].

Definition 1 (Scale-sensitive eluder dimension (normed version)) . Fix any f ⋆ ∈ F , and define ̃ E (F , β ; f ⋆ ) to be the length of the longest sequence of contexts x 1 , x 2 , . . . x m such that for all i , there exists f i ∈ F such that

The value function eluder dimension is defined as E (F , β ′ ; f ⋆ ) = sup β ≥ β ′ ̃ E (F , β ; f ⋆ ) .

Bounds on the eluder dimension for various function classes are well known, e.g. when F is finite, E (F , β ′ ; f ⋆ ) ≤ ∣F∣ 1 , and when F is the set of d -dimensional function with bounded norm, then E (F , β ′ ; f ⋆ ) = O ( d ) . We refer the reader to Russo and Van Roy [2013], Mou et al. [2020], Li et al. [2022] for more examples. The following theorem is our main result for selective sampling:

Theorem 1. Let δ ∈ ( 0 , 1 ) . Under the modeling assumptions above (in (2) , (3) and (4) ), with probability at least 1 -δ , Algorithm 1 obtains the regret bound

A few points are in order:

It must be noted that for most settings we consider, as an example if model class F is finite, one typically has that Reg (F ; T ) ≤ log ∣F∣ . Thus, in the case where one has a hard margin condition i.e. T ε 0 = 0 for some ε 0 > 0 , we get Reg T ≤ O ( log ∣F∣ ε 0 ) and N T ≤ O ( E (F ,ε ; f ⋆ ) log ∣F∣ ε 2 0 ) .

Our regret bound does not depend on the eluder dimension. However, the query complexity bound has a dependence on eluder dimension. Thus, for function classes for which the eluder dimension is large, the regret bound is still optimal while the number of label queries may be large.

3.1 Selective Sampling in the Stochastic Setting

So far we assumed that the contexts { x t } t ≥ 0 could be chosen in a possibly adversarial fashion, and thus our bound on the number of label queries scales with the eluder dimension. However, it turns out that if the contexts are drawn i.i.d. from some (unknown) distribution µ , then one can improve the query complexity to scale with the value function disagreement coefficient of F (defined below) which is always smaller than the eluder dimension (Lemma 6).

Definition 2 (Scale sensitive disagreement coefficient (normed version), Foster et al. [2020]) . Let F ⊆ {X ↦ R K } . For any f ⋆ ∈ F , and β 0 , ε 0 > 0 , the value function disagreement coefficient θ val (F , ε 0 , β 0 ; f ⋆ ) is defined as

The key idea that gives us the above improvement, of replacing the eluder dimension by disagreement coefficient in the query complexity bound, is to use epoching for the query condition, while still using an online regression oracle to make predictions. The exact algorithm is given in Appendix E.4.

Theorem 2. Let δ ∈ ( 0 , 1 ) , and consider the modeling assumptions in (2) , (3) and (4) . Furthermore, suppose that x t is sampled i.i.d. from µ , where µ is a fixed distribution. Then, with probability at least 1 -δ , Algorithm 3 obtains the bounds 5

We note that Algorithm 3 automatically adapts to Tsybakov noise condition with respect to µ . Corollary 1 (Tsybakov noise condition, Tsybakov [2004]) . Suppose there exists constants c, ρ ≥ 0 s.t. Pr x ∼ µ ( Margin ( f ⋆ ( x )) ≤ ε ) ≤ cε ρ for all ε ∈ ( 0 , 1 ) , and consider the same modeling assumptions as in Theorem 2. Then, with probability at least 1 -δ , Algorithm 3 obtains the bound

where the ̃ O(⋅) notation hides poly-logarithmic factors of γ, λ, c, ρ and log ( T / δ ) .

A detailed comparison of our results with the relevant prior works is given in Appendix E.

3.2 Lower Bounds (Binary Action Case)

We supplement the above upper bound with a lower bound in terms of the star number of F (defined below). The star number is bounded from above by the eluder dimension which appears in our upper bounds (Lemma 6). While star number may not be lower bounded by eluder dimension in general, for many commonly considered classes, star number is of the same order as the eluder dimension [Foster et al., 2020]. For the sake of a clean presentation, we restrict our lower bound to the binary actions case, although one can easily extend the lower bound to the multiple actions case.

Definition 3 (scale-sensitive star number) . For any ζ ∈ ( 0 , 1 ) and β ∈ ( 0 , ζ / 2 ) , define s val (F , ζ, β ) as the largest m such that there exists target function f ⋆ ∈ F and sequence x 1 , . . . , x m ∈ X s.t. ∀ i ∈ [ m ] , ∣ f ⋆ ( x i )∣ > ζ , ∃ f i ∈ F s.t.,

The below theorem provides a lower bound on number of queries, in terms of star number for any algorithm that guarantees a non-trivial regret bound.

Theorem 3. Given a function class F and some desired margin ζ > 0 , define β ∈ ( 0 , ζ / 2 ) be the largest number such that β 2 ≤ min { ζ 2 / s val (F , ζ, β ) , ζ 2 / 16 } . Then, for any algorithm that guarantees regret bound of E [ Reg T ] ≤ 64 ζT s val (F ,ζ,β ) on all instances with margin ζ / 2 , there exists a distribution µ over X and a target function f ⋆ ∈ F with margin 6 ζ such that the number of queries N T made by the algorithm on that instance in T rounds of interaction satisfy

The above lower bound demonstrates that for any algorithm that has a sublinear regret guarantee, a dependence on an additional complexity measure like the star number (or the eluder dimension) is unavoidable in the number of queries in the worst case. This suggests that our upper bound cannot be further improved beyond the discrepancy between the star number and eluder dimension. The following corrolary illustrates the above lower bound.

Corollary 2. There exists a class F with ∣F∣ = √ T , and s val (F , ζ, β ) = O ( √ T ) for any β = O ( 1 ) and ζ = O ( 1 ) , such that any algorithm that makes less than √ T number of label queries, will have a regret of at least E [ Reg T ] ≥ √ T on some instance with margin ζ .

4 Imitation Learning ( H > 1 ) with Selective Queries to an Expert

The problem of Imitation Learning (IL) consists of learning policies in MDPs when one has access to an expert (aka the teacher) that can make suggestions on which actions to take at a given state. IL has enjoyed tremendous empirical success, and various different interaction models have been considered. In the simplest IL setting, studied under the umbrella of offline RL [Levine et al., 2020] or Behavior Cloning [Ross and Bagnell, 2010, Torabi et al., 2018], the learner is given an offline dataset of trajectories (state and action pairs) from an expert and aims to output a well-performing policy. Here, the learner is not allowed any interaction with the expert, and can only rely on the provided dataset of expert demonstrations for learning. A much stronger IL setting is the one where the learner can interact with the expert, and rely on its feedback on states that it reaches by executing its own policies.

In their seminal work, Ross et al. [2011] proposed a framework for interactive imitation learning via reduction to online learning and classification tasks. This has been extensively studied in the IL literature (e.g., Ross and Bagnell [2014], Sun et al. [2017], Cheng and Boots [2018]). The algorithm DAgger from [Ross et al., 2011] has enjoyed great empirical success. On the theoretical side, however, performance guarantees for DAgger only hold under the assumption that, when queried, the expert makes action suggestions from a very good policy π ⋆ that we would like to compete with. However, in practice, human demonstrators are far from being optimal and suggestions from experts should be modeled as noisy suggestions that only correlate with π ⋆ . It turns out that IL where one only has

access to noisy expert suggestions is drastically different from the noiseless setting. For instance, in the sequel, we show that there can be an exponential separation in terms of the dependence on horizon H in the sample complexity of learning purely from offline demonstration vs learning with online interactions.

Formally, we consider interactive IL in an episodic finite horizon Markov Decision Process (MDP), where the learner can query a noisy expert for feedback (i.e., action) on the states that it visits. The game proceeds in T episodes. In each episode t , the nature picks the initial state x t, 1 for h = 1 ; then for every time step h ∈ [ H ] , the learner proposes an action ˆ y t,h ∈ [ K ] given the current state x t,h ; then the system proceeds by selecting the next state x t ; h + 1 ← T t,h ( x t,h , ˆ y t,h ) , where T t,h ∶ X × Y ↦ X denotes the deterministic dynamics at timestep h of round t and is unknown to the learner. The learner then decides whether to query the expert for feedback. If the learner queries, it receives a recommended action from the expert, and otherwise the learner does not receive any additional information. The game moves on to the next time step h + 1 , and moves to the next episode t + 1 when it reaches to time step H in the current episode. We now describe the expert model. With f ⋆ h being the underlying score function at time step h , the expert feedback is sampled from a distribution ϕ ( f ⋆ h ( x )) ∈ ∆ ( K ) , with ϕ ∶ R K ↦ R K being some link function (e.g., ϕ ( p )[ i ] ∝ exp ( p [ i ]) ). The goal of the leaner is to perform as well as the Bayes optimal policy 7 defined as π ⋆ h ( x ) ∶= argmax a ∈[ K ] ϕ ( f ⋆ h ( x )) . In particular, the learner aims to find a sequence of policies { π t } t ≤ T that have a small cumulative regret defined w.r.t. some (unknown) reward function under possibly adversarial (and unknown) transition dynamics { T t,h } h ≤ H,t ≤ T . At the same time, the learner wants to minimize the number of queries made to the expert. Formally, we consider counterfactual regret defined as

where x t,h are the states reached by the learner corresponding to the chosen actions and the dynamics, and x π ⋆ t,h denotes the states that would have been generated if we executed π ⋆ from the beginning of the episode under the same dynamics. The query complexity N T is the total number of queries to the expert across all H steps in T episodes.

Given the selective sampling results we provided in the earlier section, one may be tempted to apply them to the imitation learning problem. However, there is a caveat. A key to the reduction in Ross et al. [2011] is to apply Performance Difference Lemma (PDL) to reduce the problem of IL to online classification under the sequence of state distributions induced by the policies played by the learning algorithm. Hence, if one blindly applied this reduction, then in the margin term, one would need to account for the states that the learner visits (which could be arbitrary). Thus, for DAgger to have meaningful bounds, we would require a large margin over the entire state space. This is too much to ask for in practical applications. Consider the example of learning autonomous driving from a human driver as the expert. It is reasonable to believe that human drivers can confidently provide the right actions when they are driving themselves or are faced with situations they are more familiar with. However, assuming that the human driver is going to be confident in an unfamiliar situation (e.g., an emergency situation that is not often encountered by the human driver), is a strong assumption. Towards that end, we make a significantly weaker, and much more realistic, margin assumption that the expert has a large margin only on the state distribution induced by π ⋆ , and not on the state distribution of the learner or the noisy expert. 8 In particular, we define T ε,h to denote the total number of episodes where the comparator policy π ⋆ visits a state with low margin at time step h , i.e., T ε,h = ∑ T t = 1 1 { Margin ( f ⋆ h ( x π ⋆ t,h )) ≤ ε } .

We now proceed to our main results in this section. Learning from a noisy expert is indeed very challenging. In fact, learning from noisy expert feedback may even be statistically intractable in the non-interactive IL setting, where the learner is only limited to accessing offline noisy expert demonstrations for learning, e.g. in offline RL, Behavior Cloning, etc. The following lower bound formalizes this. In fact, the same lower bound also shows that AggreVaTe [Ross and Bagnell, 2014] style algorithms would not succeed under noisy expert feedback, AggreVaTe relies on roll-outs obtained by running the (noisy) expert suggestions.

Proposition 1 (Lower bound for learning from non-interactive noisy demonstrations) . There exists an MDP, for every h ≤ H , a function class F h with ∣F h ∣ ≤ 2 H , a noisy expert whose optimal policy π ⋆ ( x ) = argmax a ( f ⋆ h ( x )[ a ]) for some f ⋆ h ∈ F h with T ε,h = 0 for any ε ≤ 1 / 4 , such than any non-interactive algorithm needs Ω ( 2 H ) many noisy expert trajectory demonstrations to learn, with probability at least 3 / 4 , a policy ̂ π that is 1 / 8 -suboptimal w.r.t. π ⋆ .

Proposition 1 implies that in order to learn with a reasonable sample complexity (that is polynomial in H ), a learner must be able to interactively query the expert. In Algorithm 2, we provide an interactive imitation learning algorithm (with selective querying) that can learn from noisy expert feedback. The regret bound and query complexity bounds for Algorithm 2 are:

Algorithm 2 Inte RA cti V e I mitati O n L earning V I a Active Expert Querying ( RAVIOLI )

Input: Params δ, γ, λ, T , function classes {F h } h ≤ H , online regression oracle Oracle h w.r.t. ℓ ϕ for h ∈ [ H ] .

1: Set Ψ ℓ ϕ δ (F h , T ) = 4 λ Reg ℓ ϕ (F h ; T ) + 112 λ 2 log ( 4 H log 2 ( T )/ δ ) .

2: Compute f 1 ,h = Oracle 1 ,h (∅) for h ∈ [ H ] .

3: for t = 1 to T do

4: Nature chooses the state x t, 1 .

5: for h = 1 to H do

6: Learner plays ̂ y t,h = SelectAction ( f t,h ( x t,h ))

7: Learner transitions to the next state in this round x t,h + 1 ← T t,h ( x t,h , ̂ y t,h ) .

8: Learner computes

9: Learner decides whether to query: Z t,h = 1 { Margin ( f t,h ( x t,h )) ≤ 2 γ ∆ t,h } .

10:

if

Z

t,h

=

1

then

11: Learner queries the label y t,h for x t,h .

13: else

Theorem 4. Let δ ∈ ( 0 , 1 ) . Under the modeling assumptions above, with probability at least 1 -δ , Algorithm 2 obtains:

Since the above bound holds for any sequence of dynamics { T h,t } h ≤ H,t ≤ T , the result of Theorem 4 also holds for the stochastic IL setting where the transition dynamic is stochastic but fixed during the interaction. In particular, setting T h,t ∼ T h sampled i.i.d. from a fixed stochastic dynamics { T h } h ≤ H recovers a similar bound for the stochastic setting.

4.1 Learning from Multiple Experts

In Dekel et al. [2012], the problem of selective sampling from multiple experts is considered with the main motivation being that we can consider each expert as being confident (and correct) in certain states or scenarios, and we would like to learn from their joint feedback. The goal there is to perform not only as well as the best of them individually but even as well as the best combination of them. Consider the example of learning to drive from human demonstrations, we might have one human demonstrator who is an expert in highway driving, another human who is an expert in city driving, and the third one in off-road conditions. Each expert is confident in their own terrain, but we would like to learn a policy that can perform well in all terrains.

The formal model is similar to the single-expert case, but we now have M experts. For every time step h ≤ H , the m -th expert has an underlying ground truth model f ⋆ ,m h ∈ F m h that it uses to produce

its label, i.e. for a given state x h it draws its label as y m h ∼ ϕ ( f ⋆ ,m h ( x h )) , where ϕ is the link function. On rounds in which the learner queries for the experts feedback, it gets back a label from each of the M experts, i.e. { y 1 h , . . . , y M h } . While on every query the learner gets a different label from each expert, its objective is to perform as well as a comparator policy that is defined w.r.t. some ground truth aggregation function that we define next.

The aggregation function A ∶ ∆ ([ K ]) M ↦ ∆ ([ K ]) , known to the learner, combines the recommendation of the M experts to obtain a ground truth label for the corresponding state. In particular, on a given state x h , the label y h is samples as:

Given the aggregation function A and the above label generation process, the policy π ⋆ that we wish to compete with in our regret bound is simply the Bayes optimal predictor given by

where SelectAction ∶ ∆ ( K ) ↦ [ K ] is given by SelectAction ( p ) = argmax k ∈[ K ] p [ K ] . Some illustrative examples of aggregation functions are given in Appendix F.5. Our main Theorem 5 below bounds the number of label queries to the experts, and regret with respect to this π ⋆ , and is obtained using the imitation learning algorithm given in Algorithm 4 in Appendix F.5.

Our bounds depend on a margin term T ε,h , that captures the number of rounds in which the Bayes optimal predictor π ⋆ can flip its label if our estimates of the M experts are off by at most ε (in ℓ ∞ norm). Similar to the single expert case, we only pay in the margin term for time steps in which the counterfactual trajectory w.r.t. the policy π ⋆ has a small-margin. We note that while the trajectories taken by the learner or the noisy experts may go through states that have a large-margin, the margin term T ε,h that appears in our bounds only accounts for time steps when the comparator policy π ⋆ (the optimal aggregation of expert recommendations) would go to a small-margin region, which could be much smaller. For the ease of notation, we defer the exact definition of margin, and the term T ε,h to Appendix F.5, and state the main result below:

Theorem 5. Let δ ∈ ( 0 , 1 ) . Under the modeling assumptions above for the multiple experts setting, with probability at least 1 -δ , the imitation learning Algorithm 4 (given in the appendix) obtains:

In Appendix A, we evaluate our IL algorithm on the Cartpole environment, with single and multiple experts. We found that our algorithm can match the performance of passive querying algorithms while making a significantly lesser number of expert queries. Finally, note that setting H = 1 in the above result, recovers an algorithm, and a similar result for selective sampling with multiple experts.

Conclusion

In this paper, or goal is to develop algorithms for online IL with active queries with small regret and query complexity bounds. Towards that end, we started by considering the selective sampling setting (IL with H = 1 ), and provided a selective sampling algorithm that can work with general function classes F and modeling assumptions, and relies on access to an online regression oracle w.r.t. F to make its predictions (Section 3). The provided regret and query complexity bounds depend on the margin of the expert model. We then extended our selective sampling algorithm to interactive IL (Section 4). For IL, we showed that the margin term that appears in the regret and the query complexity depends on the margin of the expert on counterfactual trajectories that would have been observed on following the expert policy (that we wish to compare to), instead of the trajectories that the learner observes. Thus, if the expert always chooses actions that leads to states where it is confident (i.e. has less margin), the margin term will be smaller. We also considered extensions to learning with multiple experts.

References

Alekh Agarwal. Selective sampling algorithms for cost-sensitive multiclass prediction. In International Conference on Machine Learning , pages 1220-1228. PMLR, 2013.

Andrew G Barto, Richard S Sutton, and Charles W Anderson. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE transactions on systems, man, and cybernetics , (5): 834-846, 1983.

Mark Beliaev, Andy Shih, Stefano Ermon, Dorsa Sadigh, and Ramtin Pedarsani. Imitation learning by estimating expertise of demonstrators. In International Conference on Machine Learning , pages 1732-1748. PMLR, 2022.

Kiante Brantley, Wen Sun, and Mikael Henaff. Disagreement-regularized imitation learning. In International Conference on Learning Representations , 2019.

Kianté Brantley, Amr Sharaf, and Hal Daumé III. Active imitation learning with noisy guidance. arXiv preprint arXiv:2005.12801 , 2020.

Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540 , 2016.

Zhangjie Cao, Zihan Wang, and Dorsa Sadigh. Learning from imperfect demonstrations via adversarial confidence transfer. In 2022 International Conference on Robotics and Automation (ICRA) , pages 441-447. IEEE, 2022.

Nicolo Cesa-Bianchi and Gábor Lugosi. Prediction, learning, and games . Cambridge university press, 2006.

Nicolo Cesa-Bianchi, Gábor Lugosi, and Gilles Stoltz. Minimizing regret with label efficient prediction. IEEE Transactions on Information Theory , 51(6):2152-2162, 2005.

Kai-Wei Chang, Akshay Krishnamurthy, Alekh Agarwal, Hal Daumé III, and John Langford. Learning to search better than your teacher. In International Conference on Machine Learning , pages 20582066. PMLR, 2015.

Ching-An Cheng and Byron Boots. Convergence of value aggregation for imitation learning. In International Conference on Artificial Intelligence and Statistics , pages 1801-1809. PMLR, 2018.

Ching-An Cheng, Andrey Kolobov, and Alekh Agarwal. Policy improvement via imitation of multiple oracles. Advances in Neural Information Processing Systems , 33:5587-5598, 2020.

Ofer Dekel, Claudio Gentile, and Karthik Sridharan. Selective sampling and active learning from single and multiple experts. The Journal of Machine Learning Research , 13(1):2655-2697, 2012.

Maximilian Du, Suraj Nair, Dorsa Sadigh, and Chelsea Finn. Behavior retrieval: Few-shot imitation learning by querying unlabeled datasets. arXiv preprint arXiv:2304.08742 , 2023.

Dylan Foster, Alekh Agarwal, Miroslav Dudík, Haipeng Luo, and Robert Schapire. Practical contextual bandits with regression oracles. In International Conference on Machine Learning , pages 1539-1548. PMLR, 2018a.

Dylan J Foster, Satyen Kale, Haipeng Luo, Mehryar Mohri, and Karthik Sridharan. Logistic regression: The importance of being improper. In Conference On Learning Theory , pages 167-208. PMLR, 2018b.

Dylan J Foster, Alexander Rakhlin, David Simchi-Levi, and Yunzong Xu. Instance-dependent complexity of contextual bandits and reinforcement learning: A disagreement-based perspective. arXiv preprint arXiv:2010.03104 , 2020.

Steve Hanneke and Liu Yang. Minimax analysis of active learning. J. Mach. Learn. Res. , 16(1): 3487-3602, 2015.

Steve Hanneke and Liu Yang. Toward a general theory of online selective sampling: Trading off mistakes and queries. In International Conference on Artificial Intelligence and Statistics , pages 3997-4005. PMLR, 2021.

Yilun Hao, Ruinan Wang, Zhangjie Cao, Zihan Wang, Yuchen Cui, and Dorsa Sadigh. Masked imitation learning: Discovering environment-invariant modalities in multimodal demonstrations. arXiv preprint arXiv:2209.07682 , 2022.

Joey Hejna and Dorsa Sadigh. Inverse preference learning: Preference-based rl without a reward function. arXiv preprint arXiv:2305.15363 , 2023.

Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In In Proc. 19th International Conference on Machine Learning . Citeseer, 2002.

Akshay Krishnamurthy, Alekh Agarwal, Tzu-Kuo Huang, Hal Daumé III, and John Langford. Active learning for cost-sensitive classification. In International Conference on Machine Learning , pages 1915-1924. PMLR, 2017.

Michael Laskey, Sam Staszak, Wesley Yu-Shu Hsieh, Jeffrey Mahler, Florian T Pokorny, Anca D Dragan, and Ken Goldberg. Shiv: Reducing supervisor burden in dagger using support vectors for efficient learning from demonstrations in high dimensional state spaces. In 2016 IEEE International Conference on Robotics and Automation (ICRA) , pages 462-469. IEEE, 2016.

Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 , 2020.

Gene Li, Pritish Kamath, Dylan J Foster, and Nati Srebro. Understanding the eluder dimension. Advances in Neural Information Processing Systems , 35:23737-23750, 2022.

Shahar Mendelson. Rademacher averages and phase transitions in glivenko-cantelli classes. IEEE transactions on Information Theory , 48(1):251-263, 2002.

Wenlong Mou, Zheng Wen, and Xi Chen. On the sample complexity of reinforcement learning with policy space generalization. arXiv preprint arXiv:2008.07353 , 2020.

Khanh Nguyen and Hal Daumé III. Active imitation learning from multiple non-deterministic teachers: Formulation, challenges, and algorithms. arXiv preprint arXiv:2006.07777 , 2020.

Ian Osband and Benjamin Van Roy. Model-based reinforcement learning and the eluder dimension. Advances in Neural Information Processing Systems , 27, 2014.

Yunpeng Pan, Ching-An Cheng, Kamil Saigol, Keuntak Lee, Xinyan Yan, Evangelos Theodorou, and Byron Boots. Agile autonomous driving using end-to-end deep imitation learning. In Robotics: science and systems , 2018.

Alexander Rakhlin and Karthik Sridharan. Online non-parametric regression. In Conference on Learning Theory , pages 1232-1264. PMLR, 2014.

Alexander Rakhlin and Karthik Sridharan. Online nonparametric regression with general loss functions. arXiv preprint arXiv:1501.06598 , 2015.

Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Proceedings of the thirteenth international conference on artificial intelligence and statistics , pages 661-668. JMLR Workshop and Conference Proceedings, 2010.

Stephane Ross and J Andrew Bagnell. Reinforcement and imitation learning via interactive no-regret learning. arXiv preprint arXiv:1406.5979 , 2014.

Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics , pages 627-635. JMLR Workshop and Conference Proceedings, 2011.

Stéphane Ross, Narek Melik-Barkhudarov, Kumar Shaurya Shankar, Andreas Wendel, Debadeepta Dey, J Andrew Bagnell, and Martial Hebert. Learning monocular reactive uav control in cluttered natural environments. In 2013 IEEE international conference on robotics and automation , pages 1765-1772. IEEE, 2013.

Daniel Russo and Benjamin Van Roy. eluder dimension and the sample complexity of optimistic exploration. Advances in Neural Information Processing Systems , 26, 2013.

Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithms . Cambridge university press, 2014.

Nathan Srebro, Karthik Sridharan, and Ambuj Tewari. Smoothness, low noise and fast rates. Advances in neural information processing systems , 23, 2010.

Wen Sun, Arun Venkatraman, Geoffrey J Gordon, Byron Boots, and J Andrew Bagnell. Deeply aggrevated: Differentiable imitation learning for sequential prediction. In International conference on machine learning , pages 3309-3318. PMLR, 2017.

Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. arXiv preprint arXiv:1805.01954 , 2018.

Alexander B Tsybakov. Optimal aggregation of classifiers in statistical learning. The Annals of Statistics , 32(1):135-166, 2004.

Yinglun Zhu and Robert Nowak. Efficient active learning with abstention. arXiv preprint arXiv:2204.00043 , 2022.

Contents of Appendix

A Experiments

We conduct experiments to verify our theory. To this end, we first introduce the simulator, Cart Pole [Barto et al., 1983, Brockman et al., 2016], and then explain the implementation of our algorithm and the baselines. Finally, we present the results.

Cart Pole. Cart Pole is a classical control problem, in which a pole is attached by an un-actuated joint to a cart. The goal is to balance the pole by applying force to the cart either towards the left or towards the right (so binary action). The episode is terminated once either the pole is out of balance or the cart deviates too far from the origin. A reward of 1 is obtained in each time step (however, the algorithm does not get any reward signal). The observations are four-dimensional, with the values representing the cart's position, velocity, the pole's angle, and angular velocity. The action is binary, indicating the force is either to the left or to the right.

Expert policies generation. We first generate an optimal policy π ⋆ (that attains the maximum possible reward of 500) by policy gradient. We notice that when running the optimal policy π ⋆ , the absolute value of the cart's position only lies in [ 0 , 2 ] . Hence, to generate M experts, we first divide this interval into M sub-intervals [ a 0 , a 1 ] , [ a 1 , a 2 ] , . . . , [ a n -1 , a M ] ( a 0 = 0 and a M = 2 ) by geometric progression. For the i -th expert, it plays the same action as π ⋆ when the absolute value of the cart's position is in the interval [ a i -1 , a i ] and plays uniformly at random outside of this interval. We find that using such generation, each expert individually cannot achieve a good performance (when M > 1 ), while a proper combination of them can still be as strong as π ⋆ . We conduct experiment for M = 1 , 2 , 3 , and 5 , respectively. Given this design of expert generation, when the cart is in the sub-interval [ a i -1 , a i ] , the only expert with non-zero margin is exactly the i -th expert.

Implementation. The algorithm is similar to Algorithm 4 but with some modification for practical purpose. First, we use a neural network (single hidden layer neural network,with 4 neurons in the hidden layer) as our function class {F m h } h ≤ H,m ≤ M . Second, we specify SelectAction to pick the action of the most confident expert, i.e.,

Since we are considering binary action, we assume f i t,h ( x ) ∈ [-1 , 1 ] , and the action space is {-1 , 1 } . Third, to compute ∆ m t,h efficiently, we apply the Lagrange multiplier to (60) to arrive at the following equivalent problem:

Then we treat the Lagrange multiplier α as a constant, which converts the problem into the following:

The study of varying α is shown in Figure 1. We found that small values (e.g., α = 1 ) mostly lead to poor performance, while the results are fairly similar for large values. In our key experiments, we choose α = 50 when the number of experts is 1, 2 or 3, and choose 200 for 5-expert experiments. We note that since computing (9) for each time step involves repetitively fitting neural networks, which is time-consuming, we do a warm start at each round. In particular, we set the initial weights for the neural network of each round to be the weights of the trained network from the previous round. We also implemented early stopping that stops the iteration if the loss does not significantly decrease for multiple consecutive iterations. The online regression oracle Oracle is instantiated as applying gradient descent for certain steps on the mean squared loss over all data collected so far, using warm start for speedup as well.

We first conduct experiments on a single expert setting. In Figure 2 we plot the curves of return and number of queries with respect to iterations for our method, and compare to DAgger (which passively makes queries at every time step; Ross and Bagnell [2014]). We note that while our algorithm does

0

200

400

600

800

Number of Queries

0

200

400

Return

1 expert

0

250

500

750

1000

Number of Queries

0

200

400

2 experts

=1

=10

=50

=200

0

500

1000

1500

Number of Queries

0

200

400

3 experts

0

1000 2000 3000 4000

Number of Queries

0

200

400

5 experts

not converge to the optimal value as fast as DAgger, the number of queries made by our algorithm is significantly fewer, which means that our method is indeed balancing the speed of learning and the number of queries.

0

25

50

75

Iteration

0

200

400

Return

0

25

50

75

Iteration

0

2000

4000

6000

8000

Number of Queries

Ours

Passive

In additional to DAgger, we also compare to the following baselines:

Passive learning. By passive learning, we mean running our algorithms with Z t,h = 1 , i.e., making queries whenever possible. Based on different styles of expert feedback, we divide the passive learning baselines into two: noisy experts and noiseless experts . For the former we get the noisy label y m t,h for x t,h (generated by y m t,h ∼ ϕ ( f ⋆ ,m h ( x t,h )) ), and for the latter we directly get the action of the optimal policy (i.e. the action π ⋆ h ( x t,h ) ). Intuitively, noiseless feedback is more helpful than the noisy one.

MAMBA. We compare our algorithm with (a slight variant of) MAMBA [Cheng et al., 2020]. At each time step, it creates copies of the environment and run each expert policy on these copies, and then it selects the action of the expert policy with the highest return. For simplicity, we refer to this algorithm as MAMBA. Note that MAMBA assumes that one has access to the underlying reward function. Thus this baseline is using significantly more information than our approach.

Best expert. We also compared our algorithm with the best expert policy.

The main results are shown in Figure 3. We first noticed that our algorithm outperforms passive learning with noisy experts in all settings. Moreover, we beat the noiseless version when there is only one expert. Intuitively, getting feedback from noiseless experts is a very strong assumption and it is not surprising to see that the performance is improved with this stronger feedback. Note that our algorithm is only getting noisy labels as feedback. We also note that, despite the fact that MAMBA achieves better results than the best expert policy (in terms of the value function), it is still worse than our algorithm. Indeed, MAMBA does not even learn a policy that can solve the task when M ≥ 2 . This is because by our construction of experts, there is no single expert that is capable of solving the task alone. Note that MAMBA performs well in the one expert case because in that case, the (single) expert can reliably solve the control task.

0

250

500

750

1000

Number of Queries

0

200

400

Return

1 expert

0

250

500

750

1000

Number of Queries

0

200

400

2 experts

Ours

Passive (noisy experts)

Passive (noiseless experts)

Best expert

MAMBA

0

250

500

750

1000

Number of Queries

0

200

400

3 experts

0

500

1000 1500 2000

Number of Queries

0

200

400

5 experts

B Further Discussion on Related Works

Selective Sampling. There is a large bank of both theoretical and empirical work for active learning and selective sampling. Perhaps the work closest to ours is the work of Zhu and Nowak [2022]. In this paper, the authors consider binary classification problem and provide bounds on number of queries and bound on excess risk in the active learning framework. Their algorithm also relies on regression oracle. However, there are many key differences: Firstly, their guarantees for regret for selective sampling problem (see for instance Theorem 10 on page 28 of Zhu and Nowak [2022]) has a dependence on disagreement coefficient in the regret bound as well as number of queries. On the contrary, as we show in our work, one only needs to pay for eluder dimension or disagreement coefficient in query complexity and not in regret bound. Furthermore, we supplement our result with lower bound showing that unless one has label complexity that depends on star number (and hence can be also related to worst case disagreement coefficient), one can not get a small enough regret bound. So the separation between regret bound (that is independent of eluder dimension/star number/disagreement coefficient) and query complexity (that depends on those quantities) is real. Secondly, the results in [Zhu and Nowak, 2022] dont automatically adapt to the margin region and in general there is no way to estimate the parameters of Tsybakov's noise condition. Finally, their regret bounds depend on pseudo dimension and are thus generally suboptimal for complex F .

Imitation Learning. IL has enjoyed tremendous research from both theoretical and empirical perspective in the last decade; notable references include Ross et al. [2011], Ross and Bagnell [2014], Sun et al. [2017], Chang et al. [2015], Brantley et al. [2019, 2020], Nguyen and Daumé III [2020]. Ross et al. [2011] initiated research on using online regression oracles to model the expert feedback, and provided regret bounds for IL. The key differences between our work and prior theoretical works on IL are as follows: Firstly, we consider active querying, and provide query complexity bounds for our algorithms. Secondly, and more importantly, we consider interactive IL with noisy expert feedback whereas prior works was restricted to exact expert feedback. Finally, our regret and query complexity bounds scale with the number of times when the comparator policy (induced by the expert) goes to the states where expert has a small margin (instead of the number of times when the learner goes to such states). In many cases, the margin error term corresponding to the comparator policy could be much smaller. On the empirical side, there is a long line of research that provided algorithms and empirical heuristics for making IL sample efficient by modeling the experts in both single expert and multiple expert settings [Beliaev et al., 2022, Cao et al., 2022, Hejna and Sadigh, 2023, Du et al., 2023, Hao et al., 2022]; however most of these algorithm do not come with any rigorous guarantees.

C Discussion on Computationally Efficiency

Given the theoretical nature of this paper, our focus is to understand the statistically efficient (query complexity), and to develop algorithms, for selective sampling and imitation learning with general model classes F , given access to an online regression oracle w.r.t. F . For a general function class, our algorithm is computationally inefficient. However, in many cases, our algorithm (or its slight modification) can also be implemented efficiently. We describe some scenarios below:

1D-Linear models : When f ∶ X ↦ R is linear, the optimization objective in (5) in Algorithm 1 (or (6) in Algorithm 2) can be efficiently by instead solving the objectives

and

and then picking the maximum absolute value. Both of these new objectives are linear functions, with convex constraints, and thus can be solved efficiently using a standard solver (e.g. CvXOPT).

Differentiable parameterizations : When class F could be parameterized in a differentiable way, e.g. using a neural network, we can simply add the constraints as a penalty (with the appropriate multiplicative scale parameter) to convert (5) into an unconstrained optimization problem w.r.t the parameters θ (of the differentiable parameterization) and then solve it using SGD algorithm. While this is only a heuristic, it works well in practice and is precisely what we do for our experiments in A with 2 layer neural networks.

Efficient Implementation of (5) via calls to a Regression Oracles w.r.t. F . Below we discuss some scenarios and minor modifications of our algorithms under which the computation of ∆ t (as in (5) in Algorithm 1 or (6) in Algorithm 2) can be performed efficiently via calls to a regression oracle w.r.t. F . Suppose that F is closed under convexification. We consider two scenarios:

Binary Actions Setting : In this case, we can simply choose F to be a class of 1D functions, since we can model the expert using a function f ⋆ ∶ X ↦ R ∈ F where for any x , f ⋆ ( x ) denotes the probability of choosing the first action. The probability of choosing the second action would then be 1 -f ⋆ ( x ) . In this case, (5) in Algorithm 1 simply reduces to

The above can be implemented efficiently using the techniques from Foster et al. [2018a]. In particular, let r max denote the maximum value that f ( x ) can take. We can solve the above objective using the BINSEARCH procedure in Foster et al. [2018a] where we perform a binary search over a weight parameter w , by solving for each w the optimization problems:

and

both of which can be efficiently implemented using square loss regression oracles to F . We refer to Foster et al. [2018a] for more details.

Multiple Actions Setting: Suppose that F is a product class of {F k } for different actions k ∈ [ K ] . We can get an oracle-efficient algorithm for a slight modification of (5), at a price of an extra multiplicative K factor in the query complexity bound. Consider the ∆ t ( x t ) given by max f ∈F ∥ f ( x t )-f t ( x t )∥ ∞ s.t. ∑ t -1 s = 1 Z s ∥ f ( x s )-f s ( x s )∥ 2 ≤ Ψ , which is equal to max k ∈[ K ] max f ∈F ∣ f k ( x t )-f t,k ( x t )∣ s.t. ∑ t -1 s = 1 Z s ∥ f ( x s )-f s ( x s )∥ 2 ≤ Ψ , which can again be implemented via calls to a square loss regression oracle w.r.t. F by using BINSEARCH procedure in Foster et al. [2018a] (similar to what we did for the Binary actions case above).

D Useful Tools and Notation

Additional notation. Throughout the paper, we assume that the ties are broken arbitrarily but consistently. Vector-valued variables are denoted with small alphabets like u, v , etc, and matrixvalued variables are denoted with capital alphabets like F,G , etc. For any two distributions D 1 and D 2 , we define KL ( D 1 ∥ D 2 ) to denote the KL divergence between D 1 and D 2 . Furthermore, kl ( b 1 ∥ b 2 ) denotes the KL divergence between Bernoulli ( b 1 ) and Bernoulli ( b 2 ) . Finally, we assume that ∥ f ( x )∥ ≤ B ≤ 1 for any f ∈ F and x ∈ X .

The following lemma is used throughout the appendix, and its proof is trivial.

Lemma 1. Let E 1 and E 2 be any two events such that E 1 /Leftrightline⇒ E 2 then 1 {E 1 } ≤ 1 {E 2 } .

D.1 Basic Probabilistic Tools

Lemma 2 (Theorem 1 in Srebro et al. [2010]) . Let T > 0 , and let F = {X × Y} be an arbitrary function class, and ℓ be an γ -smooth and non-negative loss such that ∣ ℓ ( f ( x ) , y )∣ ≤ B for all x ∈ X , y ∈ Y , f ∈ F . For any δ > 0 , we have with probability at least 1 -δ over a random sample of size T , for any f ∈ F ,

where c 1 < 10 5 is a numeric constant, and Rad T (F) denotes the Rademacher complexity of the class F .

The precise value of the numeric constant c 1 in the above can be derived from Srebro et al. [2010] and Mendelson [2002]. Note that for finite function classes, we have Rad T (F) = O( √ log (∣F∣) / T ) and thus the second term above is bounded by ̃ O( log (∣F∣)) . In general, we have that T Rad 2 T (F) = ̃ O( Reg sq (F ; T )) [Rakhlin and Sridharan, 2014], and thus the second term is always dominated by the other terms in our regret and query complexity bounds.

The following inequalities are well-known; we use the version stated in Zhu and Nowak [2022].

Lemma 3 (Freedman's inequality) . Let { X t } t ≤ T be a real-valued martingale different sequence adapted to the filtration F t , and let E t [⋅] ∶= E [⋅ ∣ F t -1 ] . If ∣ X t ∣ ≤ B almost surely, then for any η ∈ ( 0 , 1 / B ) , the following holds with probability at least 1 -δ :

Lemma 4. Let { X t } t ≤ T be a sequence of positive valued random variables adapted to the filtration F t , and and let E t [⋅] ∶= E [⋅ ∣ F t -1 ] . If X t ≤ B almost surely, then with probability at least 1 -δ ,

and

D.2 Online Learning

Lemma 5. Suppose that the labels are generated according to the (2) where the link function satisfies Assumption 1. Additionally, assume that the regression oracle satisfies the guarantee (3) . Then, for any δ ≤ 1 / e and T ≥ 3 , with probability at least 1 -δ , we have for all t ≤ T ,

where B is defined such that sup x f ( x ) ≤ B .

Proof. Using Agarwal [2013, Lemma 2] along with an Union bound implies that for all t ≤ T ,

Plugging in the regret bound (3) in the above, we get that

D.3 Eluder Dimension, Disagreement Coefficient, and Star Number

For the sake of completeness, we recall the scalar versions of scale-sensitive eluder dimension, and disagreement coefficient introduced in Russo and Van Roy [2013], Foster et al. [2020], which is defined for a class F ⊆ {X ↦ R } of scalar valued functions.

Definition 4 (Scale-sensitive eluder dimension (scalar version), Russo and Van Roy [2013], Foster et al. [2020]) . Let F ⊆ {X ↦ R } . Fix any f ⋆ ∈ F , and define E ′ (F , β ; f ⋆ ) to be the length of the longest sequence of contexts x 1 , x 2 , . . . x m such that for all i , there exists f i ∈ F such that

We define the scale-sensitive eluder dimension as E (F , β 0 ; f ⋆ ) ∶= sup β 0 ≥ β E ′ (F , β ; f ⋆ ) .

We next provide some examples of function classes with bounded eluder dimension. The examples ( a ) - ( c ) first appeared in Russo and Van Roy [2013], and ( d ) first appeared in Osband and Van Roy [2014].

( a ) For any function class F ∶ {X ↦ R } and f ⋆ ∈ F , E (F , β 0 ; f ⋆ ) ≤ O (∣X∣) .

( b ) For the class F of linear functions on a known feature map ϕ i.e. , F = { f ∣ f ( x ) = ⟨ θ f , ϕ ( x )⟩ , θ f ∈ R d , ∥ θ f ∥ ≤ 1 } , we have E (F , β 0 ; f ⋆ ) ≤ O ( d log ( 1 / ε )) .

( c ) For the class F of generalized linear functions on a known feature map ϕ i.e. , F = { f ∣ f ( x ) = g (⟨ θ f , ϕ ( x )⟩) , θ f ∈ R d , ∥ θ f ∥ ≤ 1 } where g is an increasing continuously differentiable function, we have E (F , β 0 ; f ⋆ ) ≤ O ( dr 2 log ( L / ε )) where r = sup θ,x g ′ (⟨ θ,x ⟩) / inf θ,x g ′ (⟨ θ,x ⟩) and L = sup θ,x g ′ (⟨ θ, ϕ ( x )⟩) .

( d ) For the class F of quadratic functions on a known feature map ϕ i.e. F = { f ∣ f ( x ) = ϕ ( x ) ⊺ Σ f ϕ ( x ) , Σ f ∈ R d × d , ∥ Σ f ∥ F ≤ 1 } .

We next recall the definition of scale-sensitive disaggrement coefficient which appears in our bounds for the case of stochastic contexts.

Definition 5 (Scale-sensitive disagreement coefficient (scalar version), Foster et al. [2020]) . Let F ⊆ {X ↦ R } . For any f ⋆ ∈ F , and γ 0 , ε 0 > 0 , the value function disagreement coefficient θ val (F , ε 0 , γ 0 ; f ⋆ ) is defined as

As we will show in Lemma 6 below, the scale-sensitive disagreement coefficient of F is always bounded by the eluder dimension of F upto a constant factor on the dependence on ε 0 and γ 0 . However, the disagreement coefficient can be significantly smaller than the eluder dimension because it can leverage additional distributional structure. We refer the reader to Foster et al. [2020] for bounds on the eluder dimension, and the disagreement coefficient for various function classes. In the following, we extend the above definitions to vector-valued functions to account for the vector-valued function classes that we consider in this work.

Definition 6 (Scale-sensitive eluder dimension (normed version)) . Let F ⊆ {X ↦ R K } . Fix any f ⋆ ∈ F , and define ̃ E (F , β ; f ⋆ ) to be the length of the longest sequence of contexts x 1 , x 2 , . . . x m such that for all i , there exists f i ∈ F such that

We define the scale-sensitive eluder dimension as E (F , β ′ ; f ⋆ ) = sup β ≥ β ′ ̃ E (F , β ; f ⋆ ) .

We note that the normed eluder dimension can be lower bounded in terms of the eluder dimension of scalar-valued function class obtained by projecting the output of the functions in F along different coordinates. Let F j = { P j f ∣ P j f ( x ) = f ( x )[ j ] , f ∈ F} , then clearly, for any f ⋆ ∈ F , E (F , β ; f ⋆ ) ≥ sup j ∈[ d ] E ( P j F , β ; P j f ⋆ ) . Furthermore, we always have that E (F , β ; f ⋆ ) ≤ κ ∑ d j = 1 E ( P j F , β ; P j f ⋆ ) , where κ hides poly ( d ) factors. We next define the normed version of disagreement coefficient for vector-valued functions.

Definition 7 (Scale sensitive disagreement coefficient (normed version), Foster et al. [2020]) . Let F ⊆ {X ↦ R K } . For any f ⋆ ∈ F , and β 0 , ε 0 > 0 , the value function disagreement coefficient θ val (F , ε 0 , β 0 ; f ⋆ ) is defined as

We additionally also define the following bivariate version of eluder dimension for vector-valued functions.

Definition 8 (Scale-sensitive eluder dimension (bivariate version)) . Let F ⊆ {X ↦ R K } . Fix any f ⋆ ∈ F , and define ˇ E ′ (F , β ; f ⋆ ) to be the length of the longest sequence of contexts and actions ( x 1 , y 1 ) , ( x 2 , y 2 ) . . . ( x m , y m ) such that for all i , there exists f i ∈ F such that

We define the scale sensitive eluder dimension (mixed version) as ˇ E (F , β ; f ⋆ ) ∶= sup β ≥ β 0 ˇ E ′ (F , β 0 ; f ⋆ ) .

We next define the strong variant of scale-sensitive star number.

Definition 9 (scale-sensitive star number (strong version), Foster et al. [2020]) . Let F ⊆ {X ↦ R K } . For any f ⋆ ∈ F and β > 0 , let ˇ s val (F , β ) denote the length of the longest sequence of contexts { x 1 , . . . , x m } such that for all i , there exists f i ∈ F such that

We define the scale-sensitive star number as ˇ s val (F , β ) ∶= sup β > β 0 ˇ s val (F , β 0 ) .

The next result provides a relation between the star number, disagreement coefficient and the eluder dimension.

Lemma 6 (Foster et al. [2020]) . Suppose F ⊆ {X ↦ R K } is a uniform Glivenko-Cantelli class. For any f ⋆ ∈ F and γ, ε > 0 , we have s val (F , β ; f ⋆ ) ≤ E (F , β ; f ⋆ ) , θ val (F , ε, γ ; f ⋆ ) ≤ 4 ( s val (F , γ ; f ⋆ )) 2 and θ val (F , ε, γ ; f ⋆ ) ≤ 4 E (F , γ ; f ⋆ ) .

The following two technical lemmas are useful in bounding the total number of queries made by our selective sampling and imitation learning algorithms. We first provide a technical result which bounds the number of times we can find a function f ′ in a refinement F t of F , such that f ′ is sufficiently far away from f ⋆ ∈ F . This result is a variant of Russo and Van Roy [2013, Lemma 3], and first appears in Foster et al. [2020, Lemma E.4].

Lemma 7. Let { x t , y t , Z t } T t = 1 be sequence of tuples, where x t ∈ X and Z t ∈ { 0 , 1 } . Fix any f ⋆ ∈ F , and define the set F t = { f ∈ F ∣ ∑ t -1 s = 1 Z s ( f ( x s )[ y s ] -f ⋆ ( x s )[ y s ]) 2 ≤ β 2 } . Then, for any ζ > 0 ,

Proof. We first note that we can always remove a tuple {( x t , y t ) , Z t } whenever Z t = 0 without any effect on the conclusion. Hence, we can assume Z t = 1 for all t ∈ [ T ] without loss of generality. Then the rest of the proof essentialy follows from Foster et al. [2020, Lemma E.4]. For completeness, we state the full proof here.

For simplicity of presentation, we say ( x t , y t ) is ζ -independent of ( x 1 , y 1 ) , . . . , ( x t -1 , y t -1 ) if there exists f ∈ F such that ∣ f ( x t )[ y t ] -f ⋆ ( x t )[ y t ]∣ ≥ ζ and ∑ t -1 s = 1 ( f ( x s )[ y s ] -f ⋆ ( x s )[ y s ]) 2 ≤ ζ 2 . Otherwise, we say x is ζ -dependent. The proof consists of the following two claims.

First, we claim that for any t ∈ [ T ] , if there exists f ∈ F t such that ∣ f ( x t )[ y t ] -f ⋆ ( x t )[ y t ]∣ ≥ ζ , then x t is ζ -dependent on at most β 2 / ζ 2 disjoint sequences of ( x 1 , y 1 ) , . . . , ( x t -1 , y t -1 ) . To show this, let's say x t is ζ -dependent on a particular subsequence ( x i 1 , y i 1 ) , . . . , ( x i k , y i k ) while ∣ f ( x )[ y ] -f ⋆ ( x )[ y ]∣ ≥ ζ . Then it must holds that

If there are M such disjoint subsequence, then we can add them up and obtain the following:

By the construction of F t , the left-hand side above is at most β 2 . Hence we conclude that β 2 ≥ Mζ 2 , which implies M ≤ β 2 / ζ 2 .

Second, we claim that for any k and any sequence ( x 1 , y 1 ) , . . . , ( x k , y k ) , there exists j ≤ k such that x j is ζ -dependent on at least N ∶= ⌊ k / ˇ E (F , ζ ; f ⋆ )⌋ disjoint subsequences of ( x 1 , y 1 ) , . . . , ( x j -1 , y j -1 ) . This can be proved by construction. Let B 1 , . . . , B N be N subsequences of ( x 1 , y 1 ) , . . . , ( x k , y k ) and are initialized with B i = {( x i , y i )} . Then we repeat the following process for j = N + 1 , N + 2 . . . , k .

We first check if x j is ζ -dependent on B i for all i ∈ [ N ] . If so, we are done.

Otherwise, pick an arbitrary i ∈ [ N ] for which x j is ζ -independent of B i and append ( x j , y j ) to B i , i.e., B i ← B i ∪{( x j , y j )} .

If we don't reach any j while running the above process for which the first statement above is satisfied, we should end up with ∑ N i = 1 ∣ B i ∣ = k ≥ N ⋅ ˇ E (F , ζ ; f ⋆ ) . We note that by construction ∣ B i ∣ ≤ ˇ E (F , ζ ; f ⋆ ) and thus ∣ B i ∣ = ˇ E (F , ζ ; f ⋆ ) for all i ∈ [ N ] , which implies x k must be ζ -dependent on all B i .

Finally, let x i 1 , . . . , x i k be the subsequence where, for all s ∈ [ k ] , there exists f ∈ F i s such that ∣ f ( x i s )[ y i s ] -f ⋆ ( x i s )[ y i s ]∣ ≥ ζ . By our first claim we know each element of this subsequence is ζ -dependent on at most β 2 / ζ 2 disjoint subsequences. By the second claim, we know that there exists an element that is ζ -dependent on at least ⌊ k / ˇ E (F , ζ ; f ⋆ )⌋ disjoint subsequences. So we must have ⌊ k / ˇ E (F , ζ ; f ⋆ )⌋ ≤ β 2 / ζ 2 . Hence, k ≤ ( β 2 / ζ 2 + 1 ) ⋅ ˇ E (F , ζ ; f ⋆ ) .

The following is an extension of Lemma 7 that holds for the normed version of eluder dimension given in Definition 1. The proof is essentially the same so we skip it for conciseness.

Lemma 8. Let { x t , Z t } T t = 1 be sequence of tuples, where x t ∈ X and Z t ∈ { 0 , 1 } . Fix any f ⋆ ∈ F , and define the set F t = { f ∈ F ∣ ∑ t -1 s = 1 Z s ∥ f ( x s ) -f ⋆ ( x s )∥ 2 ≤ β 2 } . Then, for any ζ > 0 ,

E Selective Sampling: Learning from Single Expert

E.1 Comparison to Related Works

Selective Sampling. There is a large bank of both theoretical and empirical work for active learning and selective sampling. Perhaps the work closest to ours is the work of Zhu and Nowak [2022]. In this paper, the authors consider binary classification problem and provide bounds on number of queries and bound on excess risk in the active learning framework. Their algorithm also relies on regression oracle. However, there are many key differences: Firstly, their guarantees for regret for selective sampling problem (see for instance Theorem 10 on page 28 of Zhu and Nowak [2022]) has a dependence on disagreement coefficient in the regret bound as well as number of queries. On the contrary, as we show in our work, one only needs to pay for eluder dimension or disagreement coefficient in query complexity and not in regret bound. Furthermore, we supplement our result with lower bound showing that unless one has label complexity that depends on star number (and hence can be also related to worst case disagreement coefficient), one can not get a small enough regret bound. So the separation between regret bound (that is independent of eluder dimension/star number/disagreement coefficient) and query complexity (that depends on those quantities) is real. Secondly, the results in [Zhu and Nowak, 2022] dont automatically adapt to the margin region and in general there is no way to estimate the parameters of Tsybakov's noise condition. Finally, their regret bounds depend on pseudo dimension and are thus generally suboptimal for complex F .

E.2 Proof Sketch for Selective Sampling and Binary Labels

Let A = { 1 , 2 } , and the link function ϕ ( z ) = z corresponding to square-loss ℓ ϕ = ( v -y ) 2 / 2 ; here λ = γ = 1 .

Let ¯ F ⊆ {X ↦ [1 , 1 ]} be a function class, and ¯ f ⋆ ∈ F . We assume that for any context x , the label y is drawn according to the distribution Pr ( y t = 2 ) = 1 + ¯ f ⋆ ( x ) / 2 . Using ¯ F , we can define the score function class F = { f ¯ f ∣ ¯ f ∈ ¯ F} where f ( x ) = 1 2 ( 1 -¯ f ( x ) , 1 + ¯ f ( x )) ⊺ ∈ [ 0 , 1 ] 2 , and additionally define f ⋆ = f ¯ f ⋆ . Clearly, the Bayes optimal predictor that chooses the action with the largest score is given by SelectAction ( f ⋆ ( x )) = 1 + sign ( ¯ f ⋆ ( x )) . Furthermore, Margin ( f ⋆ ( x )) ∶= ∣ Pr ( y = 2 ∣ x ) -Pr ( y = 1 ∣ x )∣ = ∣ ¯ f ⋆ ( x )∣ which implies that T ε = ∑ T t = 1 1 {∣ ¯ f ⋆ ( x t )∣ ≤ ε } . Finally, the oracle in (3) reduces to a square-loss online regression oracle, which implies that with probability at least 1 -δ , for all t ≤ T ,

The above implies that ¯ f ⋆ satisfies the constraints in (5) with the right choice of constants, λ , and γ , and thus ∣ ¯ f t ( x t ) -¯ f ⋆ ( x t )∣ ≤ ∆ t ( x t ) (see Lemma 10 for proof). However, since the query condition in Algorithm 1 is Z t = 1 {∣ ¯ f t ( x t )∣ ≤ ∆ t ( x t )} , we have that if Z t = 0 , then ∣ ¯ f t ( x t )∣ > ∆ t ( x t ) which implies that sign ( ¯ f ⋆ ( x t )) = sign ( ¯ f t ( x t )) . Thus,

Regret bound. Using the fact that y t ∼ 1 + Ber ( 1 + ¯ f ⋆ ( x t ) / 2 ) , ̂ y t = SelectAction ( f t ( x t )) = 1 + sign ( ¯ f t ( x t )) , we have

The right hand side above can be split and upper bound via the following three terms:

where the first term is T ε , and the last term is zero due to (11). The term T A denotes the regret for the rounds in which the learner queries for the label, and the margin for ¯ f ⋆ ( x t ) is larger than ε . We note that

where the inequality holds because ∣ ¯ f ⋆ ( x t )-¯ f t ( x t )∣ ≥ ∣ ¯ f ⋆ ( x t )∣ since they have opposite signs. Using the fact that 1 { a ≥ b } ≤ a / b for all a, b ≥ 0 , and the bound in (10), we get

Gathering all the terms, we get

Query complexity. Plugging in the query rule, and splitting as in the regret bound, we get

T C denotes the rounds in which we make a query, ∆ t ( x t ) ≤ ε / 3 , and the margin for ¯ f ⋆ ( x t ) is larger than ε . Since ∣ ¯ f t ( x t ) -¯ f ⋆ ( x t )∣ ≤ ∆ t ( x t ) (as shown above), we have

Thus,

T D is bounded by the number of rounds for which we make a query and ∆ t ( x t ) ≥ ε / 3 . Using the properties of eluder dimension, we get that

Gathering all the terms, we conclude

In Appendix E.3, we provide the complete proof and show how to generalize it for multiple actions, link function ϕ and corresponding regression oracles w.r.t. ℓ ϕ .

E.3 Proof of Theorem 1

Before delving into the proof, we recall the relevant notation. In Algorithm 1,

The label y t ∼ ϕ ( f ⋆ ( x t )) , where ϕ denotes the link-function given in (2).

The function SelectAction ( f t ( x t )) ∶= argmax k ϕ ( f t ( x t ))[ k ] .

For any vector v ∈ R K , the margin is given by the gap between the value at the largest and the second largest coordinate, i.e.

where k ⋆ ∈ argmax k ∈[ K ] ϕ ( v )[ k ] .

We also define T ε = ∑ T t = 1 1 { Margin ( f ⋆ ( x t )) ≤ ε } to denote the number of samples within T rounds of interaction for which the margin w.r.t. f ⋆ is smaller than ε .

We define the function Gap ∶ R K ×[ K ] ↦ R + as

to denote the gap between the largest and the k -th coordinate of v .

E.3.1 Supporting Technical Results

Lemma 9. For any u , and k ′ ≠ argmax k ϕ ( u )[ k ] ,

Proof. Let k ⋆ = argmax k ϕ ( u )[ k ] . By definition,

The following technical result establishes a certain favorable property for the function f ⋆ , whose proof follows from the regret bound of the online oracle used in Algorithm 1.

Lemma 10. With probability at least 1 -δ , the function f ⋆ ∈ F satisfies the following for all t ≤ T :

Proof. The desired result follows from an application of Lemma 5, where we note that we do not query oracle when Z s = 0 , and thus do not count the time steps for which Z s = 0 .

Throughout the proof, we condition on the 1 -δ probability event that Lemma 10 holds. The next technical lemma allows us to bound the number of times when we query for the label and ∆ t ( x t ) ≥ ζ in terms of the eluder dimension (normed version) of the function class F . Note that Lemma 11 holds even if the sequence { x t } t ≤ T could be adversarially generated.

Lemma 11. Let f ⋆ satisfy Lemma 10, and let ∆ t ( x t ) be defined in (5) in Algorithm 1. Then, for any ζ > 0 , with probability at least 1 -δ ,

where E denotes the eluder dimension is given in Definition 1.

Proof. Let f ⋆ t denote the maximizer of (5) at round t on point x t . Thus,

However, recall that Lemma 10 implies that, with probability at least 1 -δ , the function f ⋆ satisfies the bound

Using (13), (14) and Triangle inequality, we get that

Next, note that, an application of Triangle inequality implies that ∥ f ⋆ t ( x t ) -f t ( x t )∥ ≤ ∥ f ⋆ t ( x t ) -f ⋆ ( x t )∥ + ∥ f ⋆ ( x t ) -f t ( x t )∥ . Thus,

where in the last line we used Lemma 10 to bound the second term. In the following, we show how to bound the first term. Recall that for any t ≤ T , the function f ⋆ t satisfies (15). Thus, we wish to bound

for all t ≤ T . An application of Lemma 8 in the above implies that

where in the last line, we used the fact that Ψ ℓ ϕ δ (F ,T ) / ζ 2 ≥ 1 , for our parameter setting.

Plugging in the bound (17) in (16), and using the fact that E (F , ζ / 2 ; f ⋆ ) ≥ 1 , we get that

The next two technical lemma's relate the margin to the gap between functions, and are useful in the analysis for regret / total number of queries.

Lemma 12. Suppose the functions π 1 and π 2 are defined such that π i ( x ) = argmax k ∈[ K ] ϕ ( f i ( x ))[ k ] . Then, for any x for which π 1 ( x ) ≠ π 2 ( x ) , we have

where γ -denotes the Lipschitz parameter of the link function ϕ .

Proof. First note ϕ f 2 x π 2 x ϕ f 2 x π 1 x by the definition of π 2 . Thus,

Using the fact that ϕ is γ -Lipschitz, we immediately get that

Lemma 13. For any two function f 1 , f 2 ∈ F , and x ∈ X ,

Proof. For the ease of notation, define

where ties are broken arbitrarily but consistently. Similarly, we define

Thus, we have that and

Finally, also note that for any coordinate k ,

We now proceed with the proof. Plugging in the form in (19), we get that

where the first inequality uses the fact that k 2 is the maximizer coordinate of ϕ ( f 2 ( x )) and the last inequality uses (20). In the following, we bound the second term in the right hand side above under the following three cases:

Case 1: k ′ 2 ≠ k 1 : Since k ′ 2 ≠ k 1 , we note that replacing k ′ 1 by k ′ 2 in the second term will only increase the value (see the definition in (18)). Thus,

where the last line uses (20).

Case 2a: k ′ 2 = k 1 , k 2 = k ′ 1 : Using definition of k 2 in (18), we note that

where the last line uses (20).

Case 2b: k ′ 2 = k 1 , k 2 ≠ k ′ 1 : Using the fact that k ′ 2 = k 1 and that k 2 ≠ k ′ 2 , we get that k 2 ≠ k 1 . Thus using the definition of k ′ 1 along with the fact that k 2 ≠ k 1 , we get that

where the second last line uses definition of k 2 and the last line uses (20).

Combining all the above bounds together implies that

The final statement follows since ϕ is γ

E.3.2 Regret Bound

For the ease of notation, for the rest of the proof in this section we define the function π ⋆ such that

Additionally, we recall that for any time t , ̂ y t = SelectAction ( f t ( x t )) = argmax k ϕ ( f t ( x ))[ k ] . Starting from the definition of the regret, we have

where the second last line uses the probabilistic model from which labels are generated, and the last inequality plugs in the definition of Gap from (45). Let ε > 0 be a free parameter. We can decompose the above regret bound further as:

Using the fact that y t ( x t ) = argmax k ∈[ K ] ϕ ( f t ( x ))[ k ] along with the definition of Gap and Lemma 12, we get that

-Lipschitz.

where the second inequality holds because Gap ( f ⋆ ( x t ) , ̂ y t ) ≤ ε implies that Margin ( f ⋆ ( x t )) ≤ ε whenever ̂ y t ≠ π ⋆ ( x t ) . In the last line above, we plugged in the definition of T ε , and defined T A and T B as the second term and the last term respectively (upto constants). We bound them separately below:

Bound on T A : We note that

where the second line follows from Lemma 12 and because ̂ y t ≠ π ⋆ ( x t ) . Using the fact that 1 { a ≥ b } ≤ a / b for all a, b ≥ 0 , we get that

Bound on T B : Fix any t ≤ T , and note that Lemma 10 implies that ∑ t s = 1 ∥ f ⋆ ( x t )-f t ( x t )∥ 2 ≤ Ψ ℓ ϕ δ (F , T ) . Thus f ⋆ satisfies the constraint in the definition of ∆ t in (5) and we must have that

Plugging in the definition of Z t , we note that

where the second inequality is due Lemma 12. However, note that the term inside the indicator contradicts (23) (which always holds). Thus,

Combining the bounds (22) and (24), we get that

where the last inequality is due to Lemma 10.

Since ε is a free parameter above, the final bound follows by choosing the best parameter ε , and by plugging in the form of Ψ ℓ ϕ δ (F , T ) .

E.3.3 Total Number of Queries

We use the notation N T to denote the total number of expert queries made by the learner within T rounds of interactions. Let ε > 0 be a free parameter. Using the definition of Z t , we have that

where in the last line we use the definition of T ε , and defined T D and T E respectively. We bound them separately below:

Bound on T D . Recall (23) which implies that f ⋆ satisfies the bound ∥ f t ( x t ) -f ⋆ ( x t )∥ ≤ ∆ t ( x t ) . Thus, using Lemma 13, we get that

. The above implies that

where the last line follows from the fact that all the conditions inside the indictor can not hold simultaneously.

Bound on T E . We note that

where the last line follows from setting ζ = ε / 4 γ in Lemma 11.

Gathering the bounds above, we get that

Since ε is a free parameter above, the final bound follows by choosing the best parameter ε , and by plugging in the form of Ψ ℓ ϕ δ (F , T ) .

E.4 Proof of Theorem 2

Before delving into the proof, we recall the relevant notation. In Algorithm 3,

Algorithm 3 Selective Sampling with Expert Feedback for Stochastic Contexts

Input: Parameters δ, γ, λ, T , function class F , and online regression oracle Oracle w.r.t ℓ ϕ .

1: Set Ψ ℓ ϕ δ (F , T ) = 4 λ Reg ℓ ϕ (F ; T ) + 112 λ 2 log ( 4log 2 ( T )/ δ ) , Compute f 1 ← Oracle 1 (∅) .

2: Set E = ⌈ log ( T )⌉ and τ e = 2 e -1 for e ≤ E .

3: for e = 1 , . . . , E -1 do

4: Learner constructs the feasible set of optimal functions F e as

5: for t ← τ e to τ e + 1 -1 do

6: Nature samples x t from an (unknown) distribution µ .

7: Learner computes

8: Learner decides whether to query: Z t = 1 { Margin ( g t ( x t )) ≤ 2 γ ∆ e ( x t )} .

9: if Z t = 1 then

10:

Learner plays the action

̂

y

t

=

SelectAction

11: Learner queries the label y t on x t .

12: f t + 1 ← Oracle t ({ x t , y t }) .

13: else

14: Learner plays the action y SelectAction g x .

15: f t + 1 f t .

̂ t = ( t ( t )) ←

The label y t ∼ ϕ ( f ⋆ ( x t )) , where ϕ denotes the link-function given in (2).

The function SelectAction ( f t ( x t )) ∶= argmax k ϕ ( f t ( x t ))[ k ] .

For any vector v ∈ R K , the margin is given by the gap between the value at the largest and the second largest coordinate, i.e.

where k ⋆ ∈ argmax k ∈[ K ] ϕ ( v )[ k ] .

We also define T ε = ∑ T t = 1 1 { Margin ( f ⋆ ( x t )) ≤ ε } to denote the number of samples within T rounds of interaction for which the margin w.r.t. f ⋆ is smaller than ε .

We define the function Gap ∶ R K ×[ K ] ↦ R + as

to denote the gap between the largest and the k -th coordinate of v . Recall that Lemma 9 holds.

Additionally, we define the function Z e to denote the query condition

The definition in (29) suggests that for all t ∈ [ τ e , τ e + 1 ) , Z t = Z e ( x t ) , for all e ≤ E -1 .

Intuition for epoching. We next provide intuition on why epoching in needed in Algorithm 3 to get the improved query complexity bound. From the proof sketch in Section E.2 , the term ∑ T t = 1 Z t 1 { ∆ t ( x t ) ≥ ε } appearing in the query complexity bound is handled using the eluder dimension of F . When x t is sampled i.i.d. we wish to bound this using disagreement-coefficient instead. However, note that in Algorithm 1 the query condition Z t depends on the samples { x s } s < t drawn in all previous time steps and the corresponding query conditions { Z s } s < t . This introduces a bias, and thus the terms Z t 1 { ∆ t ( x t ) ≥ ε } are no longer independent to each other. Thus, we can not directly used distributional properties like the disagreement coefficient to bound the query complexity. Algorithm 3 fixes this issue by defining epochs of doubling length such that the query condition in

(

f

t

(

x

t

))

.

epoch e only depends on the samples presented to the learner at time steps before this epoch (i.e. in time steps 1 ≤ t ≤ τ e -1 . Thus, the terms Z t 1 { ∆ t ( x t ) ≥ ε } for τ e ≤ t < τ e + 1 are i.i.d. allowing us to get bounds in terms of distributional properties like the disagreement coefficient of F .

However, note that whenever we query in Algorithm 3, we still choose the labels according to the estimate from the online regression oracle and thus the regret bound remains unchanged.

E.4.1 Supporting Technical Results

The following lemma establishes useful technical properties of the function f ⋆ and the sets F e .

Lemma 14. Suppose Algorithm 3 is run on the sequence { x t } t ≤ T drawn i.i.d. from the unknown distribution µ . Then, with probability at least 1 -δ , each of the following holds:

( a ) For all t ≤ T , the function f ⋆ ∈ F satisfies

where Ψ ℓ ϕ δ (F , T ) = 4 λ Reg ℓ ϕ (F ; T ) + 112 λ 2 log ( 4log 2 ( T )/ δ ) .

Thus, f ⋆ ∈ F e for all e ≤ E -1 , and ∥ f ⋆ ( x t ) -g t ( x t )∥ ≤ ∆ e ( x t ) for all τ e ≤ t ≤ τ e + 1 -1 .

( b ) For any function f ∈ F e , we have

where ̂ Ψ ℓ ϕ δ (F ; T ) ∶= 2Ψ ℓ ϕ δ (F , T )+ 4 c 2 ( log 4 ( T ) sup τ ≤ T ( τ Rad 2 τ (F)) + 2log ( T ) log ( E / δ )) .

( c ) For any e ≤ E , and any function f ∈ F e , we have

where the sub-distributions ¯ µ ¯ e ( x ) ∶= Z ¯ e ( x ) µ ( x ) and ¯ ν e ∶= 1 τ e -τ 1 ∑ e -1 ¯ e = 1 ( τ ¯ e + 1 -τ ¯ e ) ¯ µ ¯ e .

( d ) For any ¯ e < e , the corresponding sets F e and F ¯ e satisfy the relation F e ⊆ F ¯ e .

( e ) For any ¯ e ≤ e , we have ¯ µ e ≼ ¯ µ ¯ e .

Proof. We prove each part separately below:

( a ) An application of Lemma 5, where we note that we do not query oracle when Z s = 0 , and thus do not count the time steps for which Z s = 0 , implies that

for all t ≤ T with probability at least 1 -δ . Using the above for t = τ e + 1 -1 implies that f ⋆ ∈ F e for all e ≤ E -1 . Since, we also have that g t ∈ F e (by construction) for all τ e ≤ t ≤ τ e + 1 -1 , plugging in the definition of ∆ e ( x t ) , we immediately get that ∥ f ⋆ ( x t ) -g t ( x t )∥ ≤ ∆ e ( x t ) .

( b ) Fix any epoch number ¯ e ≤ E -1 , and consider the time steps τ ¯ e ≤ t < τ ¯ e + 1 . Define the loss function

where Z ¯ e denotes the query conditions at epoch ¯ e (defined in (29)), and recall that Z ¯ e does not depend on any samples that are drawn at epoch ¯ e (by definition). Furthermore, note that

ℓ ¯ e is 2 -smooth w.r.t. f and satisfies ℓ ¯ e ( f ( x ) , f ⋆ ( x )) ≤ 4 for all f, f ⋆ ∈ F and x . Thus using Lemma 2, we get that for any f ∈ F e , with probability at least 1 -δ / E ,

where in the last line we used the fact that F e ⊆ F . Summing the above for all ¯ e ≤ e -1 , we get that for any f ∈ F e ,

where the last line follows by using the definition of F e , and that E ≤ ⌈ log ( T )⌉ , and that F e ⊆ F .

( c ) Starting from part-( b ) , we first note that

Additionally, also note that

where the last two lines use the fact that the query condition Z ¯ e does not depend on the samples from rounds t = τ ¯ e to τ ¯ e + 1 -1 . Plugging in the definition of ¯ µ ¯ e in the above, we get that

where in the second line, we used the fact that the sub-distribution ¯ ν e ∶= 1 τ e -τ 1 ∑ e -1 ¯ e = 1 ( τ ¯ e + 1 -τ ¯ e ) ¯ µ ¯ e .

Combining (30) and (31), we get that

( d ) The argument follows from the definition of the set F e as any function f ∈ F e that satisfies

also satisfies the constraint

for any ¯ e ≤ e , since the left hand side consists of lesser number of terms and all terms are non-negative. Thus, F e ⊆ F ¯ e .

( e ) Recall that for any e ≤ E -1 , the sub-probability measure ¯ µ e ( x ) ∶= Z e ( x ) µ ( x ) where Z e ( x ) = 1 { min g ∈F e ∥ g ( x )∥ ≤ ∆ e ( x )} , and ∆ e ( x ) = max f ′ ,f ∈F e ∥ f ( x ) -f ′ ( x )∥ . First note that for any ¯ e ≤ e ,

where the inequality above holds because F e ⊆ F ¯ e due to part-(d) above. Furthermore,

again because F e ⊆ F ¯ e . Thus,

Lemma15. Let ε 0 , γ 0 ≥ 0 , and f ⋆ ∈ F . Then, for any sub distribution ¯ µ such that E x ∼ ¯ µ [ 1 { x ∈ X}] > 0 , ε ≥ ε 0 and γ ≥ γ 0 ,

Proof. The key idea in the proof is to go from sub distributions to distributions, and then invoking the definition of θ from Definition 2. Define κ = E x ∼ ¯ µ [ 1 { x ∈ X}] . Since 0 < κ ≤ 1 , we can define a probability measure µ such that µ ( x ) = ¯ µ ( x ) / κ . Thus, for any ε ≥ ε 0 and γ ≥ γ 0 ,

where in the second last line, we defined ¯ γ = γ / √ k , and the last line used the fact that both ε ≥ ε 0 and ¯ γ ≥ γ 0 . The final statement follows by noting the fact that µ is a distribution and the definition of the disagreement coefficient θ val ( ; ) from Definition 2.

The following technical result will be useful in bounding the query complexity for Algorithm 3.

Lemma 16. For any t ≤ T , let e ( t ) denotes the epoch number such that τ e ( t ) ≤ t < τ e ( t )+ 1 . Let f ⋆ ∈ F satisfy Lemma 14, and let ∆ e ( t ) ( x t ) be defined in Algorithm 3. Then, for any ζ > 0 , with probability at least 1 -δ ,

Proof. Recall the definition of the query rule Z e given in (29), and note that the function Z e is independent of the samples { x t } τ e + 1 -1 t = τ e chosen by the nature for time steps at epoch e . Additionally, also recall that at every time step, x t is sampled independently from the distribution µ . Thus, using the query condition Z e , we can define the sub-probability measure

such that ¯ µ e ( x ) = µ ( x ) whenever Z e ( x ) = 1 and is 0 otherwise. Furthermore, for any e ∈ E -1 , we define the sub-probability measure ¯ ν e as

Wenowmove to the main proof. First fix any epoch e ≤ E -1 , and consider any round t ∈ [ τ e , τ e + 1 -1 ] . Using the definition of ∆ e ( x t ) and definition of Z e from (29) in the above, we get that

where the second line follows because f ⋆ ∈ F e , and because sup f,f ′ ∈F e ∥ f ( x t ) -f ′ ( x t )∥ ≤ 2sup f ∈F e ∥ f ( x t ) -f ⋆ ( x t )∥ due to Triangle inequality. Plugging in the definition of ¯ µ e from (32) in the above we get that

for all ¯ e ≤ e , where the last inequality follows from Lemma 14-( e ) . Since the above holds for all ¯ e ≤ e , we immediately get that

where the sub-probability measure ¯ ν ¯ e is defined in (33). Additionally, recall that Lemma 14-( b ) implies that with probability at least 1 -δ any f ∈ F e satisfies

Conditioning on the above event, and plugging it in (35), we get that

where the last inequality uses Lemma 15.

Summing up the bound in (37) for each term t = 1 to T , we get that

where the second inequality uses the fact that τ e + 1 = 2 τ e and that τ 1 = 1 , the third inequality holds due to monotonicity of θ val (F , ζ 2 , ⋅ ; f ⋆ ) and the last line simply plugs in the value of E = log ( T ) .

Using Lemma 4 with the above bound for the sequence of random variable X t = Z t 1 { ∆ e ( x t ) > ζ } , we get that with probability at least 1 -δ ,

T ∑ t = 1 Z t 1 { ∆ e ( x t ) > ζ } ≤ 3 2 T ∑ t = 1 E x t [ Z t 1 { ∆ e ( x t ) > ζ }] + 4log ( 2 / δ ) ≤ 12log ( T ) ⋅ ̂ Ψ ℓ ϕ δ (F ; T ) ζ 2 ⋅ θ val ⎛ ⎝ F , ζ 2 , ̂ Ψ ℓ ϕ δ (F ; T ) T ; f ⋆ ⎞ ⎠ + 4log ( 2 / δ ) . The final result follows by taking a union bound of the above and the event in (36).

E.4.2 Regret Bound

For the ease of notation, through the proofs in this section we define the operators π ⋆ as

Furthermore, recall that ̂ y t denotes the action chosen by the learner at round t of interaction. Starting from the definition of the regret, we get that

where the last inequality plugs in the definition of Gap from (45). Let ε > 0 be a free parameter. We can decompose the above regret bound further as:

Using the fact that y t ( x t ) = argmax k ∈[ K ] ϕ ( f t ( x ))[ k ] in the above along with the definition of Gap and Lemma 12, we get that

where the second inequality holds because Gap ( f ⋆ ( x t ) , ̂ y t ) ≤ ε implies that Margin ( f ⋆ ( x t )) ≤ ε whenever ̂ y t ≠ π ⋆ ( x t ) . In the last line we plugged in the definition of T ε , and defined T A and T B as the second term and the last term respectively. We bound term separately below:

Bound on T A : Note that whenever Z t = 1 , we choose ̂ y t = argmax k ϕ ( f t ( x t ))[ k ] . Thus,

where the second line follows from Lemma 12 and because ̂ y t ≠ π ⋆ ( x t ) . Using the fact that 1 { a ≥ b } ≤ a / b for all a, b ≥ 0 , we get that

Bound on T B : Fix any t ≤ T , and let e be such that τ e ≤ t < τ e + 1 . Next, note that from Lemma 14, we have

Plugging in the definition of Z t , we note that

where the second inequality is due Lemma 12 and by noting that ̂ y t ≠ π ⋆ ( x t ) and that when Z t = 0 , we choose ̂ y t = argmax k ϕ ( g t ( x t ))[ k ] . However, note that the term inside the indicator contradicts (39) (which always holds). Thus,

Combining the bounds (38) and (40), we get that:

where the last inequality is due to Lemma 14.

Since ε is a free parameter above, the final bound follows by choosing the best parameter ε , and by plugging in the form of Ψ ℓ ϕ δ (F , T ) .

E.4.3 Total Number of Queries

Let N T denote the total number of expert queries made by the learner within T rounds of interactions. For the ease of notation, define ∆ t ( x t ) = ∆ e ( t ) ( x t ) where e ( t ) denotes the epoch number for which τ e ( t ) ≤ t < τ e ( t )+ 1 . Additionally, let ε > 0 be a free parameter. Thus,

where in the last line we used the definition of T ε and defined T D and T E respectively, which we bound separately below.

Bound on T D . From Lemma 14 recall that ∥ f ⋆ ( x t ) -g t ( x t )∥ ≤ ∆ e ( x t ) . Thus, for any x t for which ∥ g t ( x t )∥ ≤ ∆ e ( x t ) , Lemma 13 implies that Margin ( f ⋆ ( x t )) ≤ 2 γ ∥ f t ( x t ) -f ⋆ ( x t )∥ + Margin ( g t ( x t )) ≤ 2 γ ∆ t ( x t ) + Margin ( f t ( x t )) .

The above implies that

where the last line follows from the fact that all the conditions inside the indictor can not hold simultaneously for any ε > 0 .

Bound on T E . We note that

Using Lemma 16 with ζ = ε / 4 γ to bound the term on the right hand side above, we get that with probability at least 1 -2 δ ,

Gathering the bounds above, we get that

Since ε is a free parameter above, the final bound follows by choosing the best parameter ε , and by plugging in the form of ̂ Ψ ℓ ϕ δ (F ; T ) .

E.5 Proof of Corollary 1

Note that the Tsybakov noise condition implies that there exists constants c, ρ ≥ 0 such that for all ε ∈ ( 0 , 1 ) :

Thus, using Lemma 4, we get that

Using the above in the bound for Theorem 2, we get that for any ε > 0 ,

Setting ε = ( γ 2 λCT Reg ℓ ϕ (F ; T )) 1 ρ + 2 in the above implies that

Similarly, we can bound the query complexity bound for any ε > 0 as:

Setting ε = ( γ 2 λT ⋅ Reg ℓ ϕ (F ; T ) ⋅ θ val (F , ε / 8 γ , Reg ℓ ϕ (F ; T ) / T ; f ⋆ )) 1 ρ + 2 in the above implies that

E.6 Proofs for Lower Bounds in Section 3.2

The proof of Theorem 3 below follows along the lines of the proof of the lower bound in Theorem 28 of Foster et al. [2020] with minor changes.

Proof of Theorem 3. Let β ≤ ζ / 2 be the largest number such that β 2 ≤ min { ζ 2 / s val (F , ζ, β ) , ζ 2 / 16 } . Given the function class F , assume that m = s val (F , ζ, β ) > 0 (the lower bound is obvious when the star number is 0 ). Let the target function f ⋆ , the data sequence x 1 , . . . , x m , and the function f 1 , . . . , f m ∈ F be the witnesses for the fact that the star number is m (see Definition 3). First, for any i ∈ [ m ] , we have that ∣ f ⋆ ( x i )∣ > ζ by definition of the star number. Next note that, for each f i , we have that, for any j ≠ i ,

On the other hand, from our definition of star number we have that,

Hence we are guaranteed that each f i has a margin of at least ζ / 2 on x 1 , . . . , x m . Now consider the distribution µ over the context to be the uniform distribution over { x 1 , . . . , x m } . Also let P i ( y = 1 ∣ x ) = 1 + f i ( x ) / 2 be the conditional probability of label given context x . Let D i denote the joint distribution over X × {± 1 } given by drawing x 's from µ and labels from P i . Additionally, let D 0 be given by drawing x 's from µ and y conditioned on x as P ( y = 1 ∣ x ) = 1 + f ⋆ ( x ) / 2 . Finally, let ν = 0 with probability 1 / 2 and ν ∼ Uniform ([ m ]) with probability 1 / 2 . Note that by our premise,

where the value of the constant c will be set later. Furthermore, let p t ∶ X ↦ ∆ (Y) denote the distribution over the label chosen by the given algorithm at round t given the history ( x 1 , y 1 , . . . , x t -1 , y t -1 ) . We note that for any i ∈ [ m ] ,

where in the first inequality we used that f i has a margin of at least ζ / 2 for any x ∈ { x 1 , . . . , x m } as shown above, and the last inequality simply follows from the fact that

and via an application of the Jensen's inequality. Using the property that for any distribution with margin at least ζ / 2 , the algorithm satisfies E D i [ Reg T ] ≤ cζT / m , the above implies that

Hence, setting c = 64 and using Markov's inequality, we have that

which implies that

Further note that ∥ sign ( f j ) -sign ( f i )∥ L 1 ( µ ) > 4 m and thus, condition on the fact that ν is not equal to 0 , we can can identify ν correctly with probability at least 1 -1 / 16 . Thus, considering the reference measure D 0 and using Fano's inequality, we must have that

Now we are left with bounding KL ( D 0 ∥ D i ) . To this end, we first make a simple observation that the distribution on the x t 's is the same under D 0 and D i . Hence on rounds t where Z t = 0 we do not query for the labels in these rounds, and thus we do not glean any new information to distinguish D i from D 0 . In other words, we only need to consider rounds when Z t = 1 . Hence we have:

Assuming ζ > 1 / 4 and using the bound on KL between Bernoulli variables we get,

where N i = ∣{ t ∶ Z t = 1 , x t = x i } , and we used item-( 3 ) in the definition of star number for the ζ 2 term and item-( 1 ) for the β 2 term. Hence we have that,

Since β 2 ≤ ζ 2 / m , we have that

Hence we conclude that,

which yields the desired lower bound.

Proof of Corollary 2. Consider the function class F = { f 0 , f 1 , . . . , f √ T } where f 0 ( x i ) = 1 / 2 + ζ for every x 1 , . . . , x √ T , and where f i ( x i ) = 1 / 2 -ζ and f i ( x j ) = 1 / 2 + ζ for any j ≠ i . Note that selecting f ⋆ = f 0 and f 1 , . . . , f m on x 1 , . . . , x √ T , we can show that the star number s val (F , 1 / 8 , 1 / 2 ) = O ( √ T ) (the disagreement coefficient of F is also O ( √ T ) ). Thus, using the converse of Theorem 3 we get that if the number of queries is smaller than √ T , then there must exist some data distribution under which the regret bound of the algorithm is larger that √ T .

F Imitation Learning: Learning from Single Expert

F.1 Imitation Learning Tools

We first recall useful additional notation. Recall that a policy π is a mapping from the states X to actions A . For any h ≤ H , and random variable Z ( x h , a h ) , we use the notation E π [ Z ( x h , a h )] to denote the expectation w.r.t. trajectories { x 1 , a 1 . . . , x H , a H } sampled using the policy π .

The following lemma is the standard performance difference lemma, well known in the imitation learning and reinforcement learning literature.

Lemma17 (Performance Difference Lemma; Kakade and Langford [2002], Ross and Bagnell [2014]) . For any MDP M , and any two arbitrary stationary policies π and ̃ π , we have

where A π is the advantage function of the policy π in MDP M , i.e., A π h ( x, a ) = Q π h ( x, a ) -V π h ( x ) .

F.2 Proof of Proposition 1

MDP construction. The underlying MDP is a binary tree of depth H . In particular, we construct the deterministic MDP M = (X , A , P, r, x 1 ) where state space X = ∪ H h = 1 X h with X h = { x h,i } 2 h -1 i = 1 (we assume that x 1 = x 1 , 1 ), action space A = { 0 , 1 } , reward r is such that r ( x, a ) = Bern ( 1 2 + 1 4 1 { x = x ⋆ }) for some special state x ⋆ ∈ X H . The transition dynamics P is deterministic and defines a binary tree over X , i.e. for any h and x h,i , P ( x ′ ∣ x h,i , a ) = 1 if x ′ = x h + 1 , 2 i -1 and a = 0 , or x ′ = x h + 1 , 2 i + 1 and a = 1 , else P ( x ′ ∣ x, a ) = 0 .

We next define the expert policy π ⋆ , expert model f ⋆ and the class F . First, for any path τ = ( x 1 , a 1 , . . . , x H , a H ) from the root state x 1 to a terminal state x H at the layer H , define the policy π τ as

In particular, π τ is defined such that for any state on the path τ , we choose the corresponding action in τ , and for any state outside of τ , we choose an arbitrary (deterministic) action. Let T denote the set of all 2 H many paths from the root note x 1 to a leaf node x H ∈ X H . We define the class Π = { π τ ∣ τ ∈ T } , and F = { f τ ∣ τ ∈ T } , where for any τ , we define f τ ∶ X ↦ R 2 as

Next, let τ ⋆ = ( x 1 , a ′ 1 , x ′ 2 , . . . , x ′ H -1 , a ′ H -1 , x ⋆ ) be the path from the root x 1 to the special state x ⋆ ∈ X H on the underlying binary tree. We finally define π ⋆ = π τ ⋆ and f ⋆ = f τ ⋆ .

Lower bound. Given the MDP construction, the class F , f ⋆ and π ⋆ above, we now proceed to the desired lower bound for non-interactive imitation learning. First, note that π ⋆ ( x ) = argmax a ( f ⋆ ( x )[ a ]) for any x ∈ X . Furthermore, Margin ( f ⋆ (( x )) = 1 2 ∣ f ( x )[ 0 ] -f ( x )[ 1 ]∣ = 1 4 for all x ∈ X . Thus, for any ε ≤ 1 4 , T ε,h = 0 .

Next, for any policy π , note that V π = 1 2 + 1 4 1 { π = π ⋆ } . Thus, π ⋆ is the unique 1 / 8 -suboptimal policy. Additionally, consider a noisy expert that draws its label according to (2) with the link function ϕ ( z ) = z , i.e. on the state x , the expert draws its label from a ∼ f ⋆ ( x ) . Now, suppose that the learner is given a dataset D of m many trajectories drawn this noisy expert. There are two scenarios: either D does not contain τ ∗ , or D contains the trajectory τ ∗ .

In the first case, the learner is restricted to finding π ⋆ by eliminating all other π ≠ π ⋆ using the observations D . Since, ∣ Π ∣ = 2 H and each policy in the class is associated with a different path on the tree, we must have that m = O ( 2 H ) .

In the second case, we need τ ⋆ ∈ D . However, note that probability of observing the trajectory τ ⋆ when following the actions proposed by the noisy expert is Pr ( τ ⋆ ∣ a h ∼ f ⋆ ( x h )) = ( 3 / 4 ) H . Thus, in order to observe τ ⋆ with probability at least 3 / 4 in the dataset D , we need m = O (( 4 / 3 ) H ) .

In both the scenarios above, we need to collect exponentially many samples.

F.3 Proof of Theorem 4

Before delving into the proof, we recall the relevant notation. In Algorithm 2, for any h ≤ H ,

The label y t,h ∼ ϕ ( f ⋆ ( x t,h )) , , where ϕ denotes the link-function given in (2).

The function SelectAction ( f t,h ( x t,h )) = argmax k ϕ ( f t,h ( x t,h ))[ k ] .

For any vector v ∈ R K , the margin is given by the gap between the value at the largest and the second largest coordinate (under the link function ϕ ), i.e.

where k ⋆ ∈ argmax k ∈[ K ] ϕ ( v )[ k ] .

We define T ε = ∑ T t = 1 ∑ H h = 1 1 { Margin ( f ⋆ h ( x t,h )) ≤ ε } to denote the number of samples within T rounds of interaction for which the margin w.r.t. f ⋆ h is smaller than ε .

The trajectory at round t is generated using the dynamics { T t,h } h ≤ H to determine the states that the learner observes, starting from the state x 1 .

At round t , the learner collects data using the policy π t such that at time h , and state x , the action π t ( x ) = SelectAction ( f t,h ( x t,h )) .

For any policy π , let τ π t denote the (counterfactual) trajectory that one would obtain by running π on the deterministic dynamics { T t,h } h ≤ H with the start state x t, 1 , i.e.

For a trajectory τ = { x 1 , a 1 , . . . , x H , a H } , we define the total return

Additionally, for any policy π and dynamics { T h } h ≤ H , we define the trajectory obtained by running the policy π as

We define the function Gap ∶ R K ×[ K ] ↦ R + as

to denote the gap between the largest and the k -th coordinate of v , and note that Margin ( v ) ≤ Gap ( v, k ) for all k ≠ k ⋆ (due to Lemma 9).

F.3.1 Supporting Technical Results

We first define a useful technical lemma which allows us to bound the gap between the total returns for policies π 1 and π 2 , under the dynamics { T h } h ≤ H . Recall that for a policy π , we define the trajectory τ π under { T h } h ≤ H and the start state x 1 as the trajectory { x π 1 , π ( x π 1 ) , . . . , x π H , π ( x π H )} where x π 1 = x 1 , and x π h + 1 ← T h ( x π h , π ( x π h )) .

Lemma 18. Let { T h } h ≤ H be a deterministic dynamics, and let x 1 be the start state. Let π 1 and π 2 be any two deterministic policies, and let τ π 1 = { x π 1 1 , π 1 ( x π 1 1 ) , x π 1 2 , . . . } and τ π 2 = { x π 2 1 , π 1 ( x π 2 1 ) , x π 2 2 , . . . } be two trajectories drawn using π 1 and π 2 on { T h } h ≤ H with start state x 1 . Then, for any set X ⊆ X , the total trajectory rewards satisfy

Proof. Let h ≤ H denote the first timestep at which the policies π 1 and π 2 choose different actions under { T h } h ≤ H . Since the trajectories τ π 1 = { x π 1 1 , π 1 ( x π 1 1 ) , x π 1 2 , . . . } and τ π 2 = { x π 2 1 , π 1 ( x π 2 1 ) , x π 2 2 , . . . } are obtained by evolving through (the deterministic dynamics) { T h } h ≤ H using policies π 1 and π 2 respectively, and with the same state state x 1 , we have that

Starting from the definition of the cumulative reward R (⋅) , we have that

where the last line uses the fact that the trajectories (and thus the rewards) τ π 1 and τ π 2 are identical for the first h -1 states and actions (see (46)). Since (46) also implies that x π 1 h = x π 2 h , for the ease of notation we define x h = x π 1 h = x π 2 h . Using the fact that ∣ r ( x, a )∣ ≤ 1 and that π 1 ( x h ) ≠ π 2 ( x h ) (by definition of h ), we can bound the above as

where the equality in second last line plugs in the fact that x h = x π 1 h = x π 2 h , and the last inequality is a straightforward upper bound.

We will also be using Lemma 9 and Lemma 13 from Appendix E.3 for bounding the Margin in the regret bound proofs. Finally, we note the following properties of the function f ⋆ h .

Lemma 19. With probability at least 1 -δ , the function f ⋆ h satisfies for any h ≤ H and t ≤ T ,

Proof. ( a ) We first note that we do not query oracle when Z s,h = 0 , and thus we can ignore the time steps for which Z s,h = 0 . Hence, for each h ∈ [ H ] , applying Lemma 5 along with the fact that sup x,f ∈F h ∣ f ( x )∣ ≤ 1 yields

for all t ≤ T . Then, we take the union bound for all h ∈ [ H ] , which completes the proof.

( b ) The second part follows from using the observation in part-(a) that f ⋆ h satisfies the constraint in the definition of ∆ t,h given in (6), and thus ∥ f ⋆ h ( x t,h ) -f t,h ( x t,h )∥ ≤ ∆ t,h ( x t,h ) .

and

The next technical lemma bounds the number of times when ∆ t,h ( x t,h ) ≥ ζ and we query the expert. Note that Lemma 20 holds even if the sequence { x t,h } t ≤ T was adversarially generated.

Lemma20. Let f ⋆ satisfy Lemma 19, and let ∆ t,h ( x t ) be defined in (6) . Suppose we run Algorithm 2 on data sequence {{ x t,h } h ≤ H } t ≤ T , and let Z t,h be as defined in line 9. Then, for any ζ > 0 , with probability at least 1 -Hδ , for any h ≤ H ,

where E denotes the eluder dimension is given in Definition 1.

Proof. The proof is identical to the proof of Lemma 11 by replacing all ∣⋅∣ with ∥⋅∥ , and substitute the corresponding bounds for f ⋆ h via Lemma 19 (instead of using Lemma 10). We skip the proof for conciseness.

F.3.2 Regret Bound

Recall that the trajectory at round t is generated using the dynamics { T t,h } h ≤ H . Define the policy π t and π ⋆ such that for any h ≤ H and x ∈ X h ,

Furthermore, for any policy π , let τ π t denote the trajectory that one would obtain by running π on the deterministic dynamics { T t,h } h ≤ H with the start state x t, 1 , i.e.

where x π t, 1 = x t, 1 and x π t,h + 1 = T t,h ( x π t,h , π ( x π t,h )) . Note that Algorithm 2 collects trajectories using the policy π t at round t . Thus, we have that

where x t,h denotes the state at time step h in round t of Algorithm 2. Finally, let ε > 0 be a free parameter. We now have all the notation to proceed to the proof on our regret bound.

Step 1: Bounding the difference in return at round t . Fix any t ≤ T , and let τ π t t and τ π ⋆ t denote the trajectories that would have been sampled using the policies π t and the policy π ⋆ at round t . Furthermore, define the set X ε as

Using Lemma 18 for the policies π t and π ⋆ , and the set X ε defined above, we get that 9

where the second line is obtained by plugging in (49) and the last line simply defines T A and T B to be the second and the third terms in the previous line without the 2 H multiplicative factor. We bound T A and T B separately below.

Bound on term T A . Using the definition of X ε from (50), we note that

where in the second last line we used the definition of Margin ( f ⋆ h ( x t,h )) along with the fact that π ⋆ ( x t,h ) ≠ π t ( x t,h ) . Using the relation in Lemma 12 for the term inside the indicator, we can further bound the above as

where in the second inequality we used: 1 { a ≥ b } ≤ a 2 / b 2 for any a, b ≥ 0 .

Bound on term T B . Before delving into the proof, first note that Lemma 19-( b ) implies that

Next, note that

where in the last line we just plugged in the query condition under which Z t,h = 0 . However note that the above two conditions inside the indicator imply that

where the second line uses the definition of Margin (⋅) and the fact that π t ( x t,h ) ≠ π ⋆ ( x t,h ) , the last line is due to Lemma 12 . Thus,

but the conditions inside the indicator in the above contradicts (53) (which holds with probability 1 -δ ). Thus, with probability at least 1 -δ ,

Plugging in the bounds on T A and T B in (52), we get that with probability at least 1 -δ ,

Step 2: Bound on total regret. Using the bound in (55) for each round t , we get that

where in the last line we use the definition of T ε,h , and plug in the bound in Lemma 19. Recall that T ε,h denotes the number of times when the comparator policy π ⋆ enters the region X ε of states of small expert margin. Using the form of Ψ ℓ ϕ δ (F h , T ) and ignoring log factors and constants, we get

Since ε is a free parameter above, the final bound follows by taking inf over all feasible ε .

F.3.3 Total Number of Queries

Let N T denote the total number of expert queries made by the learner within T rounds of interaction (with H steps per round). For t ≤ T , let h t denote the first timestep at which Z t,h t = 1 at round t . Additionally, let ε > 0 be a free parameter. Thus, we have that

where T C , T D and T E are the first, second and the third term respectively in the previous line. We bound them separately below:

For the second term, note that

where in second inequality above, we used the fact that Z t,h = 0 (and thus ¯ Z t,h = 1 ) for all h ≤ h t , by the definition of h t . However note that the right hand side in the last inequality is equivalent to the term T B defined above (where sum is now till h t instead of H ). Thus, using the bound in (54) in the above, we immediately get that

For the first term in (58), using the condition that π ⋆ ( x t,h ) = π t ( x t,h ) for all h ≤ h t , we get that x t,h = x π ⋆ t,h and thus

Gathering the two terms above, and plugging in the definition of T ε,h , we get that

Bound on T D . Using the definition of the set X ε and Z t,h , we note that

Recall that Lemma 19 implies that with probability at least 1 -δ ,

using which with Lemma 13 implies that

Using the above bound with the conditions in (59) implies that

where the last equality holds because the two conditions in the indicator in the previous line can never occur simultaneously.

Bound on T E . Note that

An application of Lemma 20 in the above for each h ≤ H implies that

Gathering the bound above, we get that

Plugging in the form of Ψ ℓ ϕ δ (F h , T ) and ignoring log factors and constants, we get that

Notice that ε is a free parameter above so the final bound follows by taking inf over all feasible ε . s

F.4 Proof for the Stochastic Setting

Algorithm 2 considers arbitrary deterministic dynamics {{ T t,h } h ≤ H } t ≤ T . When the underlying dynamics T is stochastic, we can simply simulate Algorithm 2, where we set { T t,h } h ≤ H = T (⋅ ; ι t ) where ι t is drawn i.i.d. for every t ≤ T . In the following, we provide regret and query complexity bounds for stochastic dynamics.

Regret bound. Note that Theorem 4 bounds the difference of cumulative rewards of trajectories drawn using the policies π t and π ⋆ on the adversarially chosen deterministic dynamics { T t,h } H h = 1 respectively. In particular, we bound

On the other hand, when the dynamics is stochastic, we aim to bound the gap between the expected values V π ⋆ -V π t obtained under the stochastic dynamics T . We obtain this bound by pushing all the stochasticity into the choice of random seed ι . Fix any t ≤ T , and consider the deterministic dynamics ( T t, 1 , . . . , T t,H ) obtained by setting the random seed to be ι t in the stochastic dynamics T , i.e. ( T t, 1 , . . . , T t,H ) ∶= T ( ; ι t ) . Thus, for any policy π

In the following, we will bound the difference in the value function V π -V π t , by appealing to the regret bound in the proof of Theorem 4 using appropriate concentration inequalities. First, recall that in Algorithm 2, the dynamics { T t,h } h ≤ H is chosen before the round t , and that the policy π t only depends on the interaction till round t -1 . Thus,

where the last holds due to Lemma 18 and the set X ε is defined in (50). An application of Lemma 4 in the above implies that with probability at least 1 -δ ,

The rest of the proof is identical to the proof of Theorem 4 from (51) onwards. They query complexity can be similarly computed.

We next provide the proofs for learning from multiple experts.

F.5 Proof of Theorem 5

Algorithm 4 Inte RA cti V e I mitati O n L earning V I a Active Queries to M Experts ( RAVIOLI -M )

Input: Parameters δ, γ, λ, T , function classes {F m h } h ≤ H,m ≤ M , online oracles { Oracle m h } h ≤ H,m ≤ M w.r.t. ℓ ϕ .

2: Compute f m 1 ,h = Oracle 1 ,h (∅) for each h ∈ [ H ] and m ∈ [ M ] .

3: for t = 1 to T do

4: Nature chooses the state x t, 1 .

5: for h = 1 to H do

7: Learner plays ̂ y t,h = SelectAction ( F t,h ( x t,h )) .

9: For each m ∈ [ M ] , learner computes

and defines

⃗

∆

t,h

(

x

t,h

) = [

∆

1

t,h

(

x

t,h

)

, . . . ,

∆

M

t,h

(

x

t,h

)]

.

13:

m

for its label

15:

else

We first discuss the setup and the relevant notation. We are in the imitation learning setup introduced in Appendix F.3. In particular, the learner interacts with M experts in T episodes/rounds, each consisting of H timesteps. For any h ≤ H , each of the M experts have a ground truth model f ⋆ ,m h respectively. Given the context x t,h for time step h in round t , the learner plays the actions ̂ y t,h ∈ [ K ] , and can additionally choose to query the experts (by setting Q t,h = 1 ) to receive noisy feedback y m t,h ∼ ϕ ( f ⋆ ,m h ( x t )) from each expert m ∈ [ M ] . After playing the chosen action ̂ y t,h , the learner then transitions to state x t,h + 1 ← T t,h ( x t,h , ̂ y t,h ) where { T t,h } h ∈[ H ] is a sequence of deterministic dynamics (unknown to the learner).

In Algorithm 4, for any round t ≤ T and h ≤ H :

The aggregation function A ∶ R K × M ↦ R K , known to the learner, maps the predictions of the estimated experts to distributions over actions. Some illustrative examples are given below to illustrate the generality of our setup:

( a ) Random aggregation: Given a state x h , the aggregation rule chooses an expert uniformly at random and returns the label y h sampled from its model. In particular,

( b ) Majority label : A is deterministic. Given a state x t , the aggregation rule chooses the label y h ∈ [ K ] which is the top preference for the majority of the experts. In particular,

y

m

t,h for

x

t,h

.

( c ) Majority-of-confident-experts: This aggregation rule is also deterministic, and was first introduced in Dekel et al. [2012]. Given a state x t , the aggregation rule chooses the label y h ∈ [ K ] which is the top preference for the majority of the ρ -confident experts on x h i.e. the experts whose margin on x h is larger than ρ . In particular,

where Margin ( f ⋆ ,m h ( x h ) > ρ ) = max k 1 ( ϕ ( f ⋆ ,m h ( x h ))[ k 1 ] -( max k 2 ≠ k 1 ϕ ( f ⋆ ,m h ( x h ))[ k 2 ])) . This aggregation rule is useful when there may be many experts that give equal weights to the top and the second-to-top coordinates w.r.t. their respective models, and hence can not be confidently accounted for in the majority rule. Furthermore, instead of choosing the majority label, similar to Dekel et al. [2012], one can also return the label sampled according to a uniform distribution over ρ -confident experts.

The function SelectAction ∶ R K × M ↦[ K ] chooses the action to play at round t , and is defined as:

where F t,h ( x t,h )) = [ f 1 t,h ( x t,h )) , . . . , f M t,h ( x t,h ))] , and ϕ denotes the link-function given in (2).

Our goal in Algorithm 4 is to compete with the policy π ⋆ defined such that for any x ∈ X h ,

Given the context x t,h and the function F t,h ∶ X ↦ R K × M , the learner decided whether to query via Z t,h = Query ( F t,h ( x t,h ) , ⃗ ∆ t,h ( x t,h )) where we define the function Query ∶ R K × M × R M as

At round t , the learner interactions with transition dynamics { T t,h } h ≤ H and collects data. Without loss of generality, we assume that the learner always starts from the state x t, 1 . We next recall the interaction at round t :

The learner collects data using the policy π t , defined such that

for any h ≤ H , and state x ∈ X h .

For any policy π , we use the notation τ π t to denote the (counterfactual) trajectory that would have been generated by running π on the deterministic dynamics { T t,h } h ≤ H with the start state x t, 1 , i.e.

where x π t, 1 = x t, 1 and x π + = T t,h ( x π , π ( x π )) .

For any trajectory τ = { x 1 , a 1 , . . . , x H , a H } , we define the total return

The goal of the learner is to minimize its regret which is given by

Finally, our bounds depend on the notation of margin defined w.r.t. the function Query defined above. In particular, for a sequence of contexts {{ T t,h } h ≤ H } t ≤ T , we define T ε,h as

F.5.1 Supporting Technical Results

In the above we count for the number of time steps when the counterfactual states { x π ⋆ t,h } t ≤ T , reached under the given dynamics if we had executed π ⋆ , are within the ε -margin region. Note that even though the observed states {{ x t,h } h ≤ H } t ≤ T are such that ∑ T t = 1 1 { Query ( F ⋆ h ( x t,h ) , ε ⃗ ✶ ) = 1 } is large, we would not pay for this in our margin term T ε,h .

Lemma 21. With probability at least 1 -δ , for any m ≤ M , and t ≤ T and h ≤ H , the function f ⋆ ,m satisfies

where Ψ ℓ ϕ δ (F m h , T ) = 4 λ Reg ℓ ϕ (F m h ; T ) + 112 λ 2 log ( 4 MH log 2 ( T )/ δ ) Proof.

( a ) We first note that we do not query oracle when Z s,h = 0 , and thus we can ignore the time steps for which Z s,h = 0 . Hence, for each h ∈ [ H ] and m ∈ [ M ] , applying Lemma 5 yields

for all t ≤ T . Then, we take the union bound for all h ∈ [ H ] and m ∈ [ M ] , which completes the proof.

( b ) The second part follows from using part-(a) along with the definition in (60).

The next lemma bound the number of times when ∆ m t,h ( x t,h ) ≥ ζ , and we query. Note that Lemma 22 holds even if the sequence { x t,h } t ≤ T was adversarially generated.

Lemma 22. Let f ⋆ ,m satisfy Lemma 21 , and let ∆ m t,h ( x t,h ) be defined in Algorithm 4. Suppose Algorithm 4 is run on the data sequence { x t,h } t ≤ 1 , and let Z t,h be defined in line 10. Then, for any ζ > 0 , with probability at least 1 -Mδ , for any m ∈ [ M ] , and h ≤ H ,

where E denotes the eluder dimension given in Definition 1.

Proof. The proof is identical to the proof of Lemma 11 where we handle each m ∈ [ M ] and h ∈ [ H ] separately, and substitute the corresponding bounds for f ⋆ ,m h via Lemma 21 (instead of using Lemma 10). We skip the proof for conciseness.

F.5.2 Regret Bound

Suppose the trajectories at round t are generated using the deterministic dynamics { T t, 1 , . . . , T t,H } = T (⋅ ; ι t ) where ι t denotes the random seed that captures all of the stochasticity at round t 10 .

Recall that the policies π ⋆ and π t such that for any h ≤ H and x ∈ X h , π ⋆ ( x ) = SelectAction ( F ⋆ h ( x )) , and, π t ( x ) = SelectAction ( F t,h ( x )) . Note that Algorithm 4 collects trajectories using the policy π t at round t . Thus, we have

where x t,h denotes the state at time step h in round t of Algorithm 4. Finally, let ε > 0 be a free parameter. We start with the bound on the regret at time t .

Step 1: Bounding the difference in cumulative return at round t . Fix any t ≤ T , and let τ π t t and τ π ⋆ t denote the trajectories that would have been sampled using the policies π t and the policy π ⋆ at round t . Furthermore, define the set X ε as

Using Lemma 18 for the policies π t and π ⋆ , and the set X ε defined above, we get that

where the second line is obtained by using the relation (68) in the second line. The last line simply defines T A and T B to be the second and the third term in the previous line, respectively, without the 2 H multiplicative factor. We bound these two terms separately below:

Bound on T A . Using the definition of X ε from (69), we note that

where the last line follows from the fact that the definition of Query and the fact that π t ( x t,h ) ≠ π ⋆ ( x t,h ) implies that there exists some m ∈ [ M ] for which ∥ f m t,h ( x t,h ) -f ⋆ ,m h ( x t,h )∥ > ε . The above implies that

Bound on T B . First note that Lemma 21 implies that with probability at least 1 -δ , for all m ≤ M and h ≤ H ,

Next, note that

where in the last line follows from plugging in the query condition under which Z t,h = 0 . However note that for any h ≤ H for which Query ( F t,h ( x t,h ) , ⃗ ∆ t,h ( x t,h )) = 0 , by the definition of Query and the fact that π t ( x t,h ) ≠ π ⋆ ( x t,h ) , there must exist some m ∈ [ M ] such that

However, the above contradicts (71), and thus with probability at least 1 -δ ,

Plugging the above bounds on T A and T B in (70), we get that

Step 2: Aggregating over all time steps. Using the bound in (74) for each round t , we get that

Using the fact that 1 { a ≥ b } ≤ a 2 / b 2 for any a, b ≥ 0 , and the definition of T ε,h in the above, we get that

where the last line follows from using the bound in Lemma 21.

Plugging in the form of Ψ ℓ ϕ δ (F m h , T ) and ignoring log factors and constants, we get that

Notice that ε is a free parameter above so the final bound follows by taking inf over all feasible ε .

F.5.3 Total Number of Queries

Fix any t ≤ T , and let h t denote the first time step at round t for which Z t,h t = 1 , if such a time-step exists (and is set to be H + 1 otherwise). We first observe that for all h ≤ h t , we have π ⋆ ( x t,h ) = π t ( x t,h ) . To see this, note that

where in second inequality above, we used the fact that Z t,h = 0 (and thus ¯ Z t,h = 1 ) for all h < h t , by the definition of h t . Observe that the right hand side in the last inequality above is equivalent to the term (72) in the bound on T B above (where sum is now till h t instead of H ). Thus, using the bound in (73), we get that and thus

Next, let ε > 0 be a free parameter, and note that plugging in the definition of h t , we get that the total number of samples is bounded as:

where T C , T D and T E are the first, second and the third term respectively in the previous line. We bound them separately below.

Bound on T C . Fix any t ≤ T . Using the relation in (76), note that π ⋆ ( x t,h ) = π t ( x t,h ) for all h < h t . Thus, the corresponding trajectories would be identical till time step h t , which implies that x t,h t = x π ⋆ t,h t . Using this property in the T C , we get that

where the last line plugs in the definition of T ε,h .

Bound on T D . First note that

In the following, we will show that all the conditions in the above indicator can not hold simultaneously. First note that since Query ( F t,h t ( x t,h t ) , ⃗ ∆ t,h t ( x t,h t )) = 1 , there exists an ̃ F such that

and

On the other hand, recall that Lemma 21 implies that

Since, sup m ∆ m t,h t ( x t,h t ) ≤ ε / 4 , an application of Triangle inequality along with the bounds (78) and (79) imply that

But the above contradicts the fact that Query ( F ⋆ ( x t,h t ) , ε ⃗ ✶ ) = 0 since both ̃ F and F t satisfy the norm constraints in the definition of Query , but we can not simultaneously have that

SelectAction ( F ⋆ ( x t,h t ))) = SelectAction ( F t,h t ( x t,h t )) = SelectAction ( ̃ F ( x t,h t )) , due to (77). Thus, we must have that

Bound on T E . We note that

where the last line simply upper bound the term for h t by the corresponding terms for all h ≤ H .

Using Lemma 22 to bound the term in the right hand side for each m ∈ [ M ] and h ≤ H , we get that

Gathering the bound above, we get that

Plugging in the form of Ψ ℓ ϕ δ (F m h , T ) and ignoring log factors and constants, we get that

Notice that ε is a free parameter above so the final bound follows by taking inf over all feasible ε .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors make several contributions. Firstly, they propose a novel selective sampling algorithm (Algorithm 1) that utilizes a regression oracle and provide regret bounds and query complexity analysis. Secondly, they present a lower bound that demonstrates the unavoidable dependency of query complexity on the eluder dimension without additional assumptions. Thirdly, they introduce Algorithm 2, which is designed for to iid sample distributions and employs the disagreement coefficient as a substitute for the eluder dimension. For imitation learning, the authors prove an exponential gap in sample complexity between stochastic and deterministic expert demonstrations in the context of offline imitation learning. Lastly, they propose imitation learning algorithms that utilize regression oracles with selective sampling for both single and multiple stochastic expert settings.\n\nSTRENGTHS:\n1.This work provides valuable insights into the field of neural network-based imitation learning with selective sampling, laying the foundations for future advancements in practical and efficient algorithms.\n\n2.The authors' contributions encompass both selective sampling and imitation learning, showcasing the potential for designing imitation learning algorithms that offer provable efficiency. Their research opens up exciting possibilities for further advancements in the field.\n\nWEAKNESSES:\n1.While this work presents self-contained proofs, it lacks a direct comparison of the imitation learning regret and query complexity of the proposed algorithm with Dagger and other existing algorithms. This omission makes it challenging to assess the efficiency and performance of the proposed approach accurately.\n\n2.The experiment conducted on Cartpole may not provide sufficient insight and understanding of the algorithm's performance. Furthermore, the use of sophisticated selective sampling techniques may introduce significant computational overhead when applied to more complex experimental environments such as hopper and walker. This computational burden hinders the evaluation of the algorithm and makes it difficult to further explore its effectiveness in practical scenarios.\n\n3.The usage of a deterministic Markov Decision Process (MDP) with a fixed starting state, which may not align with standard imitation learning problems. This restriction limits the applicability of the guarantees provided in this work to more general and diverse scenarios.\n\nQUESTIONS:\nQuestions:\n1.Lemma 18 does not appear to me a variant of the performance difference lemma, where in the second inequality of the equation after line 1075 the value on the right is already $2H$, the upper bound of the performance difference. Please explain if this implies the other results that applies lemma 18 are already greater than $2H$.\n\n2.In line 281, for AggreVate, I believe it takes $O(\\log(H))$ rounds for the probability of executing expert policy to be lower than $1/H^2$, where by the union bound we have the probability of executing the learner's policy for whole $H$ steps greater than $(H-1)/H$. If mixing with the noisy expert is the only issue, I don't agree that AggreVaTe style algorithm should sample $\\Omega(2^H)$ samples if I am allowed to mix the proposed algorithm 2 with noisy expert policy as AggreVaTe-style in the roll out. \n\n3.I am confused why in line 266 we need a large margin over state space for Dagger instead of the recoverability assumption. By the performance difference lemma, we only need the expert's advantage function to be bounded by some value smaller than $H$ for Dagger to achieve better regret guarantee than behavior cloning. Please feel free to let me know if you need any reference.\n\n4.I am curious why the number of queries for Dagger is not a straight line, which may due to sampling trajectories instead. Please provide more experimental details including model and training details for more transparent presentation.\n\n5.Does the analysis in this work generalizable to non-deterministic learner policies?\n\n6.Is proposition 1 extendable to stochastic transition dynamic setting? I am afraid the stochastic transition dynamics will make any roll out policy hard to reach the designed leaf node.\n\n7.Is line 5 in algorithm 1 easy to compute when the policy class is exponentially large?\n\n8.Querying all $M$ experts and assigning $H \\times M$ learner policies is quite computationally demanding, any method to reduce to learning only one policy under some assumption of realizability?\n\n\nMinor suggestions:\n\n1.In line 1006 it would be better to define $r(x,a) = 0$ for none leaf nodes.\n\n2.In line 1022 use $ \\forall \\pi \\in \\Pi$ to avoid confusion.\n\n3.In line 257, could you please clarify the definition of your regret, which is missing the expectation over transition dynamics. I assume you are in the deterministic transition setting. The state that would have been generated is a bit vague, please point out the deterministic MDP setting with unique starting state directly.\n\n4.In line 256 'the learner proposes an action' is a bit vague, I suppose you are suggesting a deterministic learner's policy in stead of a stochastic random policy like a noisy expert. Could you please make it more clear by mentioning 'deterministic'?\n\n5.In line 491, though using the last round policy as warm start may result in similar performance as training from random initialization. It would be beneficial to conduct one extra experiment comparing the result of using warm start and naively start from random initialization for each round, which would empirically justify this speedup.\n\n6.In the Figure 2, Dagger takes around 13 iterations to converge, where it would be helpful to mark the exact number of annotations for Dagger to achieve near expert performance as a dotted line on the right of figure 2.\n\nLIMITATIONS:\n1.The lack of a direct theoretical comparison between the proposed algorithm's imitation learning regret and query complexity with Dagger and other preexisting algorithms hinders the interpretation of its efficiency. Comparative evaluations against existing algorithms would provide a clearer understanding of the algorithm's performance and effectiveness.\n\n2.The experiment conducted solely on the Cartpole environment may not offer sufficient insights into the algorithm's behavior and capabilities. Additionally, the computational overhead associated with the sophisticated selective sampling technique might pose challenges when applying the algorithm to more complex environments like hopper and walker. This limitation makes it difficult to thoroughly evaluate the algorithm's performance in a broader range of scenarios.\n\n3.The use of a deterministic Markov Decision Process (MDP) with a fixed starting state deviates from the standard settings in imitation learning problems. This restriction limits the guarantee and applicability of the proposed algorithm, as it may not generalize well to more diverse and realistic scenarios encountered in practice.\n\n4.While Proposition 1 is supported by theoretical proof, it would be beneficial to include empirical observations to demonstrate its importance and practical relevance. Additionally, conducting experiments that involve behavior cloning would provide further insights and comparisons, enhancing the experimental evaluation and supporting the theoretical claims.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposed a selective sampling scheme on an online regression oracle over a function class $\\mathcal{F}$ and time horizon $T$ for multiclass setting, which has the regret guarantee $Reg^{\\ell_{\\phi} } (\\mathcal{F}, T)$, and we assume the label $y_t$ is sampled through a link function $\\phi$. The algorithm actively acquire a label when the margin (the difference among top two action scores) are small, which resembles sampling by disagreement in active learning research.\n\nAssuming the problem has a margin greater than some $\\epsilon$ at all round and adversarially chosen context $x_t$, the sampling scheme achieve regret $\\tilde{O}( Reg^{\\ell_{\\phi} } (\\mathcal{F}, T) / \\epsilon) $, and the labelling complexity as $\\tilde{O} ( Reg^{\\ell_{\\phi} } (\\mathcal{F}, T) / \\epsilon^2 )$ ( also have dependence on eluder dimension of $\\mathcal{F}$). In the case that the context $x_t$ is stochastically chosen, the algorithm achieves the same regret but fewer labels. The result can also be extended to imitation learning, where the regret bound and query number scales with episode number $H$ and teacher number $M$.\n\nSTRENGTHS:\nThe paper is well-written and proposed a simple sampling rule based on the margin on the fly. The appendix was well managed and easy to follow. \n\nThe regret bound captures the difficulty of the problem and the regret of the online regression oracle, which is very general. The paper also shows lower bound on the tightness of the label complexity. \n\nThe proposed sampling rule was derived for both adversarial and stochastically chosen context, showing the later is improved in terms of labelling complexity. The sampling rule, and analysis can also be extended to imitation learning. Experimental results shows the trade off between convergence speed and labelling complexity.\n\nWEAKNESSES:\nSome possible extensions:\n\n1. Since the experiment shows $\\alpha$ in equation (7) was experimentally determined due to the complexity of finding the exact Lagrangian multiplier, whether theoretical bound can include the scenario: if $\\Delta_t(x_t)$ can not be solved exactly. Instead we can only use an approximation denote as $\\tilde{\\Delta}_t(x) $ such that $| \\tilde{\\Delta}_t(x) - \\Delta(x_t) | \\le \\delta $ for some threshold $\\delta$. How does this error $\\delta$ affects regret and sampling number.\n\n2. For imitation learning multiple teacher, it looks like the regret and query number for Multiple teacher $M > 1$ (theorem 5) has a worse guarantee in terms of $M$ comparing to single teacher in theorem 4, due to $\\sum_{m=1}^{M} $. This is counter intuitive in the sense that more resources was used. Whether the algorithm can be extended so we have regret and query number with $\\min_{m \\in [M] }$ instead of $\\sum_{m=1}^{M}$. An example to illustrate current concern: in the single teacher setting, we use ${Oracle}_1$ has regret $R_1$. For the multiple teacher setting, we use ${Oracle}_1, {Oracle}_2, \\cdots, {Oracle}_M$, which all have regret $R_1$, then multiple teacher setting is picking up $M R_1$ and single teacher is only $R_1$. Although they are using the same quality of teacher oracles.\n\nQUESTIONS:\nsee above\n\nLIMITATIONS:\nThe paper might have been limited by the submission page limit, which ends without an explicit discussion session. The paper already summarized its finding and significances. Currently limitation / improvement is addressed in the weakness session, a broader and domain specific limitation, future direction should be included if more space is permitted.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies selective sampling and imitation learning. In this setting a series of states arrives online, and a learner has the ability to choose to query expert labels/policy in order to correctly label the states/perform as well as an expert policy. The goal is to minimize the regret (i.e. the drop in performance relative to what the expert would achieve on the particular sequence) while also making as few queries to the expert labels/policy.\n\nThe authors design algorithms under the additional assumption that there is black-box access to an online regression oracle for the class of functions $\\mathcal{F}$ under consideration. The algorithms achieve regret bounds that depend linearly on the regret achieved by the online regression oracle, plus the number of times that the expert-provided label function has a small-margin in the online sequence. These regret bounds are essentially best-possible considering the performance of the regression oracle. For the number of expert queries, the bounds for the algorithms grow proportional the product of the regret bounds of the oracle and the eluder dimension. The authors further show that the dependence on Eluder dimension is necessary.\n\nSTRENGTHS:\nThe paper provides a clear general method to transform an online-regression oracle for a given function class $\\mathcal{F}$ into an algorithm for active learning using the class $\\mathcal{F}$. This method essentially maintains the regret of the original regression oracle (which has access to all the labels), while making as few oracle queries to the expert labels/policy as possible (without making further assumptions).\n\nWEAKNESSES:\n1. Both algorithms in the paper rely on solving an optimization problem (Equation (5) for Algorithm 1 and Equation (6) for Algorithm 2). It is not immediately clear how to use the online regression oracle to solve these problems. If they cannot be solved via the oracle, it seems likely that they are generally intractable for many function classes $\\mathcal{F}$ of interest. This seems like an important issue for both algorithms, and significantly limits the utility of the method if an additional oracle is required to solve the problems in (5) and (6).\n\n2. A minor point: the notation on line 9 of Algorithm 1 was initially confusing to me. Presumably the regression oracle is called with all previously queried labels, but the notation actually used was never introduced and so it is unclear what was actually meant.\n\nQUESTIONS:\n1. I don't immediately see why the optimization problem in Algorithm 1 Equation (5) is tractable, even given an online regression oracle for the function class $\\mathcal{F}$. Am I missing something or is another assumption needed here (e.g. that we have a second oracle to solve the optimization problem in (5))? Alleviating my concerns about this issue would cause me to increase my score.\n\n2. Can you clarify exactly which sequence is sent to the oracle in line 9 of algorithm 1?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
The paper studies algorithms for selective sampling with applications to imitation learning.
In the former setting the authors propose an algorithm build around an online regression oracle and derive bounds on the overall regret and number of queries.
These are provided for both a general (potentially adversarial) setting as well as a potentially more favourable stochastic setting. A key component is the dependence on the eluder dimension of the regressor family and the authors show that a dependency on this is unavoidable by also providing matching lower bounds.

In the imitation learning setting the notion of an expert confidence *margin* is introduced. The authors argue that in an offline setting requirements for expert trajectories can grow exponentially w.r.t. to the length of the considered time horizon and consequently an online learning setup is required to achieve reasonable regret. Based on the algorithm from the selective sampling setting a matching formulation for imitation learning with corresponding regret derivations is presented. Finally, the notion of expert uncertainty encoded in margin scores allows for an extension to a multi-expert setting.

STRENGTHS:
The paper is well structured and clearly elucidates the short-comings of prior work it addresses. The extensions proposed to overcome them are clearly motivated and the proposed algorithms appear sound.

WEAKNESSES:
Despite being motivated by making the algorithms more fit for practical applications, there is a rather limited experimental evaluation of the proposed algorithms. It would be interesting to compare their empirical performance to existing baselines such as DAgger.

QUESTIONS:
-

LIMITATIONS:
-
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-26