SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathrm {R e g} _ {T} = \widetilde {\mathcal {O}} \bigg (\inf _ {\varepsilon} \bigg \{\varepsilon T _ {\varepsilon} + \frac {\gamma^ {2}}{\lambda \varepsilon} \mathrm {R e g} ^ {\ell_ {\phi}} (\mathcal {F}; T) + \frac {\gamma^ {2}}{\lambda^ {2} \varepsilon} \log (1 / \delta) \bigg \} \bigg), \qquad a n d, \\ N _ {T} = \widetilde {\mathcal {O}} \bigg (\inf  _ {\varepsilon} \Bigg \{T _ {\varepsilon} + \frac {\gamma^ {2}}{\lambda \varepsilon^ {2}} \cdot \operatorname {R e g} ^ {\ell_ {\phi}} (\mathcal {F}; T) \cdot \mathfrak {E} \big (\mathcal {F}, \varepsilon / 4 \gamma ; f ^ {\star} \big) + \frac {\gamma^ {2}}{\lambda^ {2} \varepsilon^ {2}} \log (1 / \delta) \Bigg \} \bigg). \\ \end{array}
$$
A few points are in order:
- It must be noted that for most settings we consider, as an example if model class  $\mathcal{F}$  is finite, one typically has that  $\operatorname{Reg}(\mathcal{F};T)\leq \log |\mathcal{F}|$ . Thus, in the case where one has a hard margin condition i.e.  $T_{\varepsilon_0} = 0$  for some  $\varepsilon_0 > 0$ , we get  $\operatorname{Reg}_T\leq O\left(\frac{\log|\mathcal{F}|}{\varepsilon_0}\right)$  and  $N_{T}\leq O\left(\frac{\mathfrak{E}(\mathcal{F},\varepsilon;f^{*})\log|\mathcal{F}|}{\varepsilon_{0}^{2}}\right)$ .
- Our regret bound does not depend on the eluder dimension. However, the query complexity bound has a dependence on eluder dimension. Thus, for function classes for which the eluder dimension is large, the regret bound is still optimal while the number of label queries may be large.
3.1 Selective Sampling in the Stochastic Setting
So far we assumed that the contexts  $\{x_{t}\}_{t\geq 0}$  could be chosen in a possibly adversarial fashion, and thus our bound on the number of label queries scales with the eluder dimension. However, it turns out that if the contexts are drawn i.i.d. from some (unknown) distribution  $\mu$ , then one can improve the query complexity to scale with the value function disagreement coefficient of  $\mathcal{F}$  (defined below) which is always smaller than the eluder dimension (Lemma 6).
Definition 2 (Scale sensitive disagreement coefficient (normed version), Foster et al. [2020]). Let  $\mathcal{F} \subseteq \{\mathcal{X} \mapsto \mathbb{R}^K\}$ . For any  $f^{\star} \in \mathcal{F}$ , and  $\beta_0, \varepsilon_0 > 0$ , the value function disagreement coefficient  $\theta^{\mathrm{val}}(\mathcal{F}, \varepsilon_0, \beta_0; f^{\star})$  is defined as
$$
\sup  _ {\mu} \sup  _ {\beta > \beta_ {0}, \varepsilon > \varepsilon_ {0}} \left\{\frac {\varepsilon^ {2}}{\beta^ {2}} \cdot \Pr_ {x \sim \mu} (\exists f \in \mathcal {F} | \| f (x) - f ^ {\star} (x) \| > \varepsilon , \| f - f ^ {\star} \| _ {\mu} \leq \beta) \right\} \vee 1
$$
where  $\| f\|_{\mu} = \sqrt{\mathbb{E}_{x\sim\mu}\left[\|f(x)\|^2\right]}$
The key idea that gives us the above improvement, of replacing the eluder dimension by disagreement coefficient in the query complexity bound, is to use epoching for the query condition, while still using an online regression oracle to make predictions. The exact algorithm is given in Appendix E.4.
Theorem 2. Let  $\delta \in (0,1)$ , and consider the modeling assumptions in (2), (3) and (4). Furthermore, suppose that  $x_{t}$  is sampled i.i.d. from  $\mu$ , where  $\mu$  is a fixed distribution. Then, with probability at least  $1 - \delta$ , Algorithm 3 obtains the bounds
$$
\begin{array}{l} \operatorname {R e g} _ {T} = \widetilde {\mathcal {O}} \left(\inf  _ {\varepsilon} \left\{\varepsilon T _ {\varepsilon} + \frac {\gamma^ {2}}{\lambda \varepsilon} \operatorname {R e g} ^ {\ell_ {\phi}} (\mathcal {F}; T) \right\}\right), \quad a n d, \\ N _ {T} = \widetilde {\mathcal {O}} \bigg (\inf  _ {\varepsilon} \Bigg \{T _ {\varepsilon} + \frac {\gamma^ {2}}{\lambda \varepsilon^ {2}} \cdot \operatorname {R e g} ^ {\ell_ {\phi}} (\mathcal {F}; T) \cdot \theta^ {\mathrm {v a l}} \big (\mathcal {F}, \varepsilon / 8 _ {\gamma}, \operatorname {R e g} ^ {\ell_ {\phi}} (\mathcal {F}; T) / T; f ^ {\star} \big) \Bigg \} \bigg). \\ \end{array}
$$
We note that Algorithm 3 automatically adapts to Tsybakov noise condition with respect to  $\mu$ .
Corollary 1 (Tsybakov noise condition, Tsybakov [2004]). Suppose there exists constants  $c, \rho \geq 0$  s.t.  $\operatorname{Pr}_{x \sim \mu}(\text{Margin}(f^{\star}(x)) \leq \varepsilon) \leq c\varepsilon^{\rho}$  for all  $\varepsilon \in (0,1)$ , and consider the same modeling assumptions as in Theorem 2. Then, with probability at least  $1 - \delta$ , Algorithm 3 obtains the bound
$$
\begin{array}{l} \operatorname {R e g} _ {T} = \widetilde {\mathcal {O}} \bigg (\left(\operatorname {R e g} ^ {\ell_ {\phi}} \left(\mathcal {F}; T\right)\right) ^ {\frac {\rho + 1}{\rho + 2}} \cdot (T) ^ {\frac {1}{\rho + 2}} \bigg), \qquad a n d, \\ N _ {T} = \widetilde {\mathcal {O}} \Big (\big (\mathrm {R e g} ^ {\ell_ {\phi}} (\mathcal {F}; T) \cdot \theta^ {\mathrm {v a l}} \big (\mathcal {F}, \varepsilon / 8 \gamma , \mathrm {R e g} ^ {\ell_ {\phi}} (\mathcal {F}; T) / T; f ^ {\star}) \big) \big) ^ {\frac {\rho}{\rho + 2}} \cdot T ^ {\frac {2}{\rho + 2}} \Big). \\ \end{array}
$$
where the  $\widetilde{\mathcal{O}} (\cdot)$  notation hides poly-logarithmic factors of  $\gamma ,\lambda ,c,\rho$  and  $\log (T / \delta)$
A detailed comparison of our results with the relevant prior works is given in Appendix E.
In the rest of the paper, the notation  $\widetilde{\mathcal{O}}$  hides additive  $\log (1 / \delta)$ -factors which, for constant  $\delta$  and in all the results, are asymptotically dominated by the other terms presented in the displayed bounds.
3.2 Lower Bounds (Binary Action Case)
We supplement the above upper bound with a lower bound in terms of the star number of  $\mathcal{F}$  (defined below). The star number is bounded from above by the eluder dimension which appears in our upper bounds (Lemma 6). While star number may not be lower bounded by eluder dimension in general, for many commonly considered classes, star number is of the same order as the eluder dimension [Foster et al., 2020]. For the sake of a clean presentation, we restrict our lower bound to the binary actions case, although one can easily extend the lower bound to the multiple actions case.
Definition 3 (scale-sensitive star number). For any  $\zeta \in (0,1)$  and  $\beta \in (0,\zeta /2)$ , define  $\mathfrak{s}^{\mathrm{val}}(\mathcal{F},\zeta ,\beta)$  as the largest  $m$  such that there exists target function  $f^{\star}\in \mathcal{F}$  and sequence  $x_{1},\ldots ,x_{m}\in \mathcal{X}$  s.t.  $\forall i\in [m],|f^{\star}(x_i)| > \zeta ,\exists f_i\in \mathcal{F}$  s.t.,
(1)  $\sum_{j\neq i}(f_i(x_j) - f^{\star}(x_j))^2 <  \beta^2$
(2)  $|f_{i}(x_{i})| > \zeta / 2$  and  $f_{i}(x_{i})f^{\star}(x_{i}) < 0$
(3)  $|f_{i}(x_{i}) - f^{\star}(x_{i})|\leq 2\zeta$
The below theorem provides a lower bound on number of queries, in terms of star number for any algorithm that guarantees a non-trivial regret bound.
Theorem 3. Given a function class  $\mathcal{F}$  and some desired margin  $\zeta >0$ , define  $\beta \in (0,\zeta /2)$  be the largest number such that  $\beta^2\leq \min \{\zeta^2 /\mathfrak{s}^{\mathrm{val}}(\mathcal{F},\zeta ,\beta),\zeta^2 /16\}$ . Then, for any algorithm that guarantees regret bound of  $\mathbb{E}[\mathrm{Reg}_T]\leq 64\frac{\zeta T}{\mathfrak{s}^{\mathrm{val}}(\mathcal{F},\zeta,\beta)}$  on all instances with margin  $\zeta /2$ , there exists a distribution  $\mu$  over  $\mathcal{X}$  and a target function  $f^{\star}\in \mathcal{F}$  with margin $^6$ $\zeta$  such that the number of queries  $N_{T}$  made by the algorithm on that instance in  $T$  rounds of interaction satisfy
$$
\mathbb {E} \left[ N _ {T} \right] = \Omega \left(\frac {\mathfrak {s} ^ {\text {v a l}} (\mathcal {F} , \zeta , \beta)}{4 0 \zeta^ {2}}\right).
$$
The above lower bound demonstrates that for any algorithm that has a sublinear regret guarantee, a dependence on an additional complexity measure like the star number (or the eluder dimension) is unavoidable in the number of queries in the worst case. This suggests that our upper bound cannot be further improved beyond the discrepancy between the star number and eluder dimension. The following corollary illustrates the above lower bound.
Corollary 2. There exists a class  $\mathcal{F}$  with  $|\mathcal{F}| = \sqrt{T}$ , and  $\mathfrak{s}^{\mathrm{val}}(\mathcal{F},\zeta,\beta) = O(\sqrt{T})$  for any  $\beta = O(1)$  and  $\zeta = O(1)$ , such that any algorithm that makes less than  $\sqrt{T}$  number of label queries, will have a regret of at least  $\mathbb{E}[\mathrm{Reg}_T] \geq \sqrt{T}$  on some instance with margin  $\zeta$ .
4 Imitation Learning  $(H > 1)$  with Selective Queries to an Expert
The problem of Imitation Learning (IL) consists of learning policies in MDPs when one has access to an expert (aka the teacher) that can make suggestions on which actions to take at a given state. IL has enjoyed tremendous empirical success, and various different interaction models have been considered. In the simplest IL setting, studied under the umbrella of offline RL [Levine et al., 2020] or Behavior Cloning [Ross and Bagnell, 2010, Torabi et al., 2018], the learner is given an offline dataset of trajectories (state and action pairs) from an expert and aims to output a well-performing policy. Here, the learner is not allowed any interaction with the expert, and can only rely on the provided dataset of expert demonstrations for learning. A much stronger IL setting is the one where the learner can interact with the expert, and rely on its feedback on states that it reaches by executing its own policies.
In their seminal work, Ross et al. [2011] proposed a framework for interactive imitation learning via reduction to online learning and classification tasks. This has been extensively studied in the IL literature (e.g., Ross and Bagnell [2014], Sun et al. [2017], Cheng and Boots [2018]). The algorithm DAgger from [Ross et al., 2011] has enjoyed great empirical success. On the theoretical side, however, performance guarantees for DAgger only hold under the assumption that, when queried, the expert makes action suggestions from a very good policy  $\pi^{\star}$  that we would like to compete with. However, in practice, human demonstrators are far from being optimal and suggestions from experts should be modeled as noisy suggestions that only correlate with  $\pi^{\star}$ . It turns out that IL where one only has
6When  $\mathcal{A} = \{1,2\}$  , recall that  $\operatorname {Margin}(f(x)) = |\operatorname *{Pr}(y = 2\mid f(x)) - \operatorname *{Pr}(y = 1\mid f(x))| = |f(x)|.$
access to noisy expert suggestions is drastically different from the noiseless setting. For instance, in the sequel, we show that there can be an exponential separation in terms of the dependence on horizon  $H$  in the sample complexity of learning purely from offline demonstration vs learning with online interactions.
Formally, we consider interactive IL in an episodic finite horizon Markov Decision Process (MDP), where the learner can query a noisy expert for feedback (i.e., action) on the states that it visits. The game proceeds in  $T$  episodes. In each episode  $t$ , the nature picks the initial state  $x_{t,1}$  for  $h = 1$ ; then for every time step  $h \in [H]$ , the learner proposes an action  $\hat{y}_{t,h} \in [K]$  given the current state  $x_{t,h}$ ; then the system proceeds by selecting the next state  $x_{t; h + 1} \gets \mathbb{T}_{t,h}(x_{t,h}, \hat{y}_{t,h})$ , where  $\mathbb{T}_{t,h}: \mathcal{X} \times \mathcal{Y} \mapsto \mathcal{X}$  denotes the deterministic dynamics at timestep  $h$  of round  $t$  and is unknown to the learner. The learner then decides whether to query the expert for feedback. If the learner queries, it receives a recommended action from the expert, and otherwise the learner does not receive any additional information. The game moves on to the next time step  $h + 1$ , and moves to the next episode  $t + 1$  when it reaches to time step  $H$  in the current episode. We now describe the expert model. With  $f_h^{\star}$  being the underlying score function at time step  $h$ , the expert feedback is sampled from a distribution  $\phi(f_h^{\star}(x)) \in \Delta(K)$ , with  $\phi: \mathbb{R}^K \mapsto \mathbb{R}^K$  being some link function (e.g.,  $\phi(p)[i] \propto \exp(p[i])$ ). The goal of the leaner is to perform as well as the Bayes optimal policy defined as  $\pi_h^{\star}(x) := \operatorname{argmax}_{a \in [K]} \phi(f_h^{\star}(x))$ . In particular, the leaner aims to find a sequence of policies  $\{\pi_t\}_{t \leq T}$  that have a small cumulative regret defined w.r.t. some (unknown) reward function under possibly adversarial (and unknown) transition dynamics  $\{\mathbb{T}_{t,h}\}_{h \leq H, t \leq T}$ . At the same time, the leaner wants to minimize the number of queries made to the expert. Formally, we consider counterfactual regret defined as
$$
\operatorname {R e g} _ {T} = \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} r \left(x _ {t, h} ^ {\pi^ {\star}}, \pi_ {h} ^ {\star} \left(x _ {t, h} ^ {\pi^ {\star}}\right)\right) - \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {T} r \left(x _ {t, h}, \hat {y} _ {t, h}\right)
$$
where  $x_{t,h}$  are the states reached by the learner corresponding to the chosen actions and the dynamics, and  $x_{t,h}^{\pi^{\star}}$  denotes the states that would have been generated if we executed  $\pi^{\star}$  from the beginning of the episode under the same dynamics. The query complexity  $N_T$  is the total number of queries to the expert across all  $H$  steps in  $T$  episodes.
Given the selective sampling results we provided in the earlier section, one may be tempted to apply them to the imitation learning problem. However, there is a caveat. A key to the reduction in Ross et al. [2011] is to apply Performance Difference Lemma (PDL) to reduce the problem of IL to online classification under the sequence of state distributions induced by the policies played by the learning algorithm. Hence, if one blindly applied this reduction, then in the margin term, one would need to account for the states that the learner visits (which could be arbitrary). Thus, for DAgger to have meaningful bounds, we would require a large margin over the entire state space. This is too much to ask for in practical applications. Consider the example of learning autonomous driving from a human driver as the expert. It is reasonable to believe that human drivers can confidently provide the right actions when they are driving themselves or are faced with situations they are more familiar with. However, assuming that the human driver is going to be confident in an unfamiliar situation (e.g., an emergency situation that is not often encountered by the human driver), is a strong assumption. Towards that end, we make a significantly weaker, and much more realistic, margin assumption that the expert has a large margin only on the state distribution induced by  $\pi^{\star}$ , and not on the state distribution of the learner or the noisy expert. In particular, we define  $T_{\varepsilon,h}$  to denote the total number of episodes where the comparator policy  $\pi^{\star}$  visits a state with low margin at time step  $h$ , i.e.,  $T_{\varepsilon,h} = \sum_{t=1}^{T} \mathbf{1}\{Margin(f_h^*(x_{t,h}^{\pi^{\star}})) \leq \varepsilon\}$ .
We now proceed to our main results in this section. Learning from a noisy expert is indeed very challenging. In fact, learning from noisy expert feedback may even be statistically intractable in the non-interactive IL setting, where the learner is only limited to accessing offline noisy expert demonstrations for learning, e.g. in offline RL, Behavior Cloning, etc. The following lower bound formalizes this. In fact, the same lower bound also shows that AggreVaTe [Ross and Bagnell, 2014] style algorithms would not succeed under noisy expert feedback, AggreVaTe relies on roll-outs obtained by running the (noisy) expert suggestions.
Note that the comparator policy  $\pi^*$  reflects the experts models, and may not be the optimal policy for the underlying MDP.
The precise definition of the Margin for IL is given in the appendix.
Proposition 1 (Lower bound for learning from non-interactive noisy demonstrations). There exists an MDP, for every  $h \leq H$ , a function class  $\mathcal{F}_h$  with  $|\mathcal{F}_h| \leq 2^H$ , a noisy expert whose optimal policy  $\pi^\star(x) = \operatorname{argmax}_a (f_h^\star(x)[a])$  for some  $f_h^\star \in \mathcal{F}_h$  with  $T_{\varepsilon,h} = 0$  for any  $\varepsilon \leq 1/4$ , such that any non-interactive algorithm needs  $\Omega(2^H)$  many noisy expert trajectory demonstrations to learn, with probability at least  $3/4$ , a policy  $\widehat{\pi}$  that is  $1/8$ -suboptimal w.r.t.  $\pi^\star$ .
Proposition 1 implies that in order to learn with a reasonable sample complexity (that is polynomial in  $H$ ), a learner must be able to interactively query the expert. In Algorithm 2, we provide an interactive imitation learning algorithm (with selective querying) that can learn from noisy expert feedback. The regret bound and query complexity bounds for Algorithm 2 are:
Algorithm 2 InteRActiVe ImitatiOn Learning VIa Active Expert Querying (RAVIOLI)
Input: Params  $\delta ,\gamma ,\lambda ,T$  function classes  $\{\mathcal{F}_h\}_{h\leq H}$  , online regression oracle Oracle  $h$  w.r.t.  $\ell_{\phi}$  for  $h\in [H]$    
1: Set  $\Psi_{\delta}^{\ell_{\phi}}(\mathcal{F}_h,T) = \frac{4}{\lambda}\mathrm{Reg}^{\ell_{\phi}}(\mathcal{F}_h;T) + \frac{112}{\lambda^2}\log (4H\log^2 (T) / \delta)$    
2: Compute  $f_{1,h} = \mathrm{Oracle}_{1,h}(\emptyset)$  for  $h\in [H]$    
3: for  $t = 1$  to  $T$  do   
4: Nature chooses the state  $x_{t,1}$    
5: for  $h = 1$  to  $H$  do   
6: Learner plays  $\widehat{y}_{t,h} = \mathrm{SelectAction}(f_{t,h}(x_{t,h}))$    
7: Learner transitions to the next state in this round  $x_{t,h + 1}\gets \mathbb{T}_{t,h}(x_{t,h},\widehat{y}_{t,h})$    
8: Learner computes   
 $\Delta_{t,h}\coloneqq \max_{f\in \mathcal{F}_h}\| f(x_{t,h}) - f_{t,h}(x_{t,h})\|$  s.t.  $\sum_{s = 1}^{t - 1}Z_{s,h}\| f(x_{s,h}) - f_{s,h}(x_{s,h})\| ^2\leq \Psi_\delta^{\ell_\phi}(\mathcal{F}_h,T).$  (6)   
9: Learner decides whether to query:  $Z_{t,h} = 1$  {Margin  $(f_{t,h}(x_{t,h}))\leq 2\gamma \Delta_{t,h}$ }   
10: if  $Z_{t,h} = 1$  then   
11: Learner queries the label  $y_{t,h}$  for  $x_{t,h}$    
12:  $f_{t + 1,h}\gets \mathrm{Oracle}_{t + 1,h}(\{x_{t,h},y_{t,h}\})$    
13: else   
14:  $f_{t + 1,h}\gets f_{t,h}$
Theorem 4. Let  $\delta \in (0,1)$ . Under the modeling assumptions above, with probability at least  $1 - \delta$ , Algorithm 2 obtains:
$$
\operatorname {R e g} _ {T} = \widetilde {\mathcal {O}} \left(\inf  _ {\varepsilon} \left\{H \sum_ {h = 1} ^ {H} T _ {\varepsilon , h} + \frac {H \gamma^ {2}}{\lambda \varepsilon^ {2}} \sum_ {h = 1} ^ {H} \operatorname {R e g} ^ {\ell_ {\phi}} \left(\mathcal {F} _ {h}; T\right) \right\}\right), \quad a n d,
$$
$$
N _ {T} = \widetilde {\mathcal {O}} \left(\inf  _ {\varepsilon} \left\{H \sum_ {h = 1} ^ {H} T _ {\varepsilon , h} + \frac {H \gamma^ {2}}{\lambda \varepsilon^ {2}} \sum_ {h = 1} ^ {H} \operatorname {R e g} ^ {\ell_ {\phi}} \left(\mathcal {F} _ {h}; T\right) \cdot \mathfrak {E} \left(\mathcal {F} _ {h}, ^ {\varepsilon} / 8 \gamma ; f _ {h} ^ {*}\right) \right\}\right).
$$
Since the above bound holds for any sequence of dynamics  $\{\mathbb{T}_{h,t}\}_{h\leq H,t\leq T}$ , the result of Theorem 4 also holds for the stochastic IL setting where the transition dynamic is stochastic but fixed during the interaction. In particular, setting  $\mathbb{T}_{h,t}\sim \mathcal{T}_h$  sampled i.i.d. from a fixed stochastic dynamics  $\{\mathcal{T}_h\}_{h\leq H}$  recovers a similar bound for the stochastic setting.
4.1 Learning from Multiple Experts
In Dekel et al. [2012], the problem of selective sampling from multiple experts is considered with the main motivation being that we can consider each expert as being confident (and correct) in certain states or scenarios, and we would like to learn from their joint feedback. The goal there is to perform not only as well as the best of them individually but even as well as the best combination of them. Consider the example of learning to drive from human demonstrations, we might have one human demonstrator who is an expert in highway driving, another human who is an expert in city driving, and the third one in off-road conditions. Each expert is confident in their own terrain, but we would like to learn a policy that can perform well in all terrains.
The formal model is similar to the single-expert case, but we now have  $M$  experts. For every time step  $h \leq H$ , the  $m$ -th expert has an underlying ground truth model  $f_h^{\star,m} \in \mathcal{F}_h^m$  that it uses to produce
its label, i.e. for a given state  $x_{h}$  it draws its label as  $y_{h}^{m} \sim \phi(f_{h}^{\star,m}(x_{h}))$ , where  $\phi$  is the link function. On rounds in which the learner queries for the experts feedback, it gets back a label from each of the  $M$  experts, i.e.  $\{y_{h}^{1},\ldots ,y_{h}^{M}\}$ . While on every query the learner gets a different label from each expert, its objective is to perform as well as a comparator policy that is defined w.r.t. some ground truth aggregation function that we define next.
The aggregation function  $\mathcal{A}:\Delta ([K])^{M}\mapsto \Delta ([K])$ , known to the learner, combines the recommendation of the  $M$  experts to obtain a ground truth label for the corresponding state. In particular, on a given state  $x_{h}$ , the label  $y_{h}$  is samples as:
$$
y _ {h} \sim \mathscr {A} \left(\phi \left(f _ {h} ^ {\star , 1} \left(x _ {h}\right)\right), \dots , \phi \left(f _ {h} ^ {\star , M} \left(x _ {h}\right)\right)\right). \tag {7}
$$
Given the aggregation function  $\mathcal{A}$  and the above label generation process, the policy  $\pi^{\star}$  that we wish to compete with in our regret bound is simply the Bayes optimal predictor given by
$$
\pi^ {\star} \left(x _ {h}\right) = \operatorname {S e l e c t A c t i o n} \left(\mathscr {A} \left(\phi \left(f _ {h} ^ {\star , 1} \left(x _ {h}\right)\right), \dots , \phi \left(f _ {h} ^ {\star , M} \left(x _ {h}\right)\right)\right)\right), \tag {8}
$$
where SelectAction:  $\Delta(K) \mapsto [K]$  is given by SelectAction  $(p) = \operatorname{argmax}_{k \in [K]} p[K]$ . Some illustrative examples of aggregation functions are given in Appendix F.5. Our main Theorem 5 below bounds the number of label queries to the experts, and regret with respect to this  $\pi^{\star}$ , and is obtained using the imitation learning algorithm given in Algorithm 4 in Appendix F.5.
Our bounds depend on a margin term  $T_{\varepsilon,h}$ , that captures the number of rounds in which the Bayes optimal predictor  $\pi^{\star}$  can flip its label if our estimates of the  $M$  experts are off by at most  $\varepsilon$  (in  $\ell_{\infty}$  norm). Similar to the single expert case, we only pay in the margin term for time steps in which the counterfactual trajectory w.r.t. the policy  $\pi^{\star}$  has a small-margin. We note that while the trajectories taken by the learner or the noisy experts may go through states that have a large-margin, the margin term  $T_{\varepsilon,h}$  that appears in our bounds only accounts for time steps when the comparator policy  $\pi^{\star}$  (the optimal aggregation of expert recommendations) would go to a small-margin region, which could be much smaller. For the ease of notation, we defer the exact definition of margin, and the term  $T_{\varepsilon,h}$  to Appendix F.5, and state the main result below:
Theorem 5. Let  $\delta \in (0,1)$ . Under the modeling assumptions above for the multiple experts setting, with probability at least  $1 - \delta$ , the imitation learning Algorithm 4 (given in the appendix) obtains:
$$
\operatorname {R e g} _ {T} = \widetilde {\mathcal {O}} \left(\inf  _ {\varepsilon} \left\{H \sum_ {h = 1} ^ {H} T _ {\varepsilon , h} + \frac {H}{\lambda \varepsilon^ {2}} \sum_ {m = 1} ^ {M} \sum_ {h = 1} ^ {H} \operatorname {R e g} ^ {\ell_ {\phi}} \left(\mathcal {F} _ {h} ^ {m}; T\right) \right\}\right), \quad a n d,
$$