SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} R \left(\tau_ {t} ^ {\pi^ {*}}\right) - R \left(\tau_ {t} ^ {\pi_ {t}}\right) \leq 2 H \sum_ {h = 1} ^ {H} \mathbf {1} \left\{x _ {t, h} ^ {\pi^ {*}} \in \mathrm {X} _ {\varepsilon} \right\} + 2 H \sum_ {h = 1} ^ {H} \mathbf {1} \left\{\pi_ {t} \left(x _ {t, h} ^ {\pi_ {t}}\right) \neq \pi^ {*} \left(x _ {t, h} ^ {\pi_ {t}}\right), x _ {t, h} ^ {\pi_ {t}} \notin \mathrm {X} _ {\varepsilon} \right\} \tag {51} \\ = 2 H \sum_ {h = 1} ^ {H} \mathbf {1} \left\{x _ {t, h} ^ {\pi^ {\star}} \in \mathrm {X} _ {\varepsilon} \right\} + 2 H \sum_ {h = 1} ^ {H} \mathbf {1} \left\{\pi_ {t} \left(x _ {t, h}\right) \neq \pi^ {\star} \left(x _ {t, h}\right), x _ {t, h} \notin \mathrm {X} _ {\varepsilon} \right\} \\ = 2 H \sum_ {h = 1} ^ {H} \mathbf {1} \left\{x _ {t, h} ^ {\pi^ {\star}} \in \mathrm {X} _ {\varepsilon} \right\} + 2 H \sum_ {h = 1} ^ {H} Z _ {t, h} \mathbf {1} \left\{\pi_ {t} \left(x _ {t, h}\right) \neq \pi^ {\star} \left(x _ {t, h}\right), x _ {t, h} \notin \mathrm {X} _ {\varepsilon} \right\} \\ + 2 H \sum_ {h = 1} ^ {H} \bar {Z} _ {t, h} \mathbf {1} \left\{\pi_ {t} \left(x _ {t, h}\right) \neq \pi^ {\star} \left(x _ {t, h}\right), x _ {t, h} \notin \mathrm {X} _ {\varepsilon} \right\} \\ \leq 2 H \sum_ {h = 1} ^ {H} \mathbf {1} \left\{x _ {t, h} ^ {\pi^ {\star}} \in \mathrm {X} _ {\varepsilon} \right\} + 2 H \sum_ {h = 1} ^ {H} Z _ {t, h} \mathbf {1} \left\{\pi_ {t} \left(x _ {t, h}\right) \neq \pi^ {\star} \left(x _ {t, h}\right), x _ {t, h} \notin \mathrm {X} _ {\varepsilon} \right\} \\ \end{array}
$$
The key advantage of using Lemma 18 is that the first term  $\sum_{h=1}^{H} \mathbf{1}\{x_{t,h}^{\pi^*} \in \mathbf{X}_{\varepsilon}\}$  accounts for the number of steps at which a counterfactual trajectory sampled using  $\pi^*$  goes to the state space with margin less than  $\varepsilon$ . Thus, we only pay for the number of times when the comparator policy  $\pi^*$  would go to states with  $\varepsilon$ -margin (instead of when  $\pi_t$  does to such states).
$$
\begin{array}{l} + 2 H \sum_ {h = 1} ^ {H} \bar {Z} _ {t, h} \mathbf {1} \left\{\pi_ {t} \left(x _ {t, h}\right) \neq \pi^ {\star} \left(x _ {t, h}\right) \right\} \\ = 2 H \sum_ {h = 1} ^ {H} \mathbf {1} \left\{x _ {t, h} ^ {\pi^ {\star}} \in \mathrm {X} _ {\varepsilon} \right\} + 2 H \mathrm {T} _ {A} + 2 H \mathrm {T} _ {B}, \tag {52} \\ \end{array}
$$
where the second line is obtained by plugging in (49) and the last line simply defines  $\mathrm{T}_A$  and  $\mathrm{T}_B$  to be the second and the third terms in the previous line without the  $2H$  multiplicative factor.
We bound  $\mathrm{T}_A$  and  $\mathrm{T}_B$  separately below.
- Bound on term  $\mathrm{T}_A$ . Using the definition of  $\mathbf{X}_{\varepsilon}$  from (50), we note that
$$
\begin{array}{l} \mathrm {T} _ {A} = \sum_ {h = 1} ^ {H} Z _ {t, h} \mathbf {1} \left\{\pi_ {t} \left(x _ {t, h}\right) \neq \pi^ {\star} \left(x _ {t, h}\right), x _ {t, h} \notin \mathrm {X} _ {\varepsilon} \right\} \\ = \sum_ {h = 1} ^ {H} Z _ {t, h} \mathbf {1} \left\{\pi_ {t} \left(x _ {t, h}\right) \neq \pi^ {\star} \left(x _ {t, h}\right), \operatorname {M a r g i n} \left(f _ {h} ^ {\star} \left(x _ {t, h}\right)\right) > \varepsilon \right\} \\ \leq \sum_ {h = 1} ^ {H} Z _ {t, h} \mathbf {1} \left\{\pi^ {\star} \left(x _ {t, h}\right) \neq \pi_ {t} \left(x _ {t, h}\right), \phi \left(f _ {h} ^ {\star} \left(x _ {t, h}\right)\right) \left[ \pi^ {\star} \left(x _ {t, h}\right) \right] - \phi \left(f _ {h} ^ {\star} \left(x _ {t, h}\right)\right) \left[ \pi_ {t} \left(x _ {t, h}\right) \right] \geq \varepsilon \right\} \\ \leq \sum_ {h = 1} ^ {H} Z _ {t, h} \mathbf {1} \left\{\phi \left(f _ {h} ^ {\star} \left(x _ {t, h}\right)\right) \left[ \pi^ {\star} \left(x _ {t, h}\right) \right] - \phi \left(f _ {h} ^ {\star} \left(x _ {t, h}\right)\right) \left[ \pi_ {t} \left(x _ {t, h}\right) \right] \geq \varepsilon \right\}, \\ \end{array}
$$
where in the second last line we used the definition of  $\mathrm{Margin}(f_h^\star (x_{t,h}))$  along with the fact that  $\pi^{\star}(x_{t,h})\neq \pi_t(x_{t,h})$ . Using the relation in Lemma 12 for the term inside the indicator, we can further bound the above as
$$
\begin{array}{l} \mathrm {T} _ {A} \leq \sum_ {h = 1} ^ {H} Z _ {t, h} \mathbf {1} \left\{2 \gamma \left\| f _ {h} ^ {\star} \left(x _ {t, h}\right) - f _ {t, h} \left(x _ {t, h}\right) \right\| \geq \varepsilon \right\} \\ \leq \frac {4 \gamma^ {2}}{\varepsilon^ {2}} \sum_ {h = 1} ^ {H} Z _ {t, h} \| f _ {h} ^ {\star} (x _ {t, h}) - f _ {t, h} (x _ {t, h}) \| ^ {2}, \\ \end{array}
$$
where in the second inequality we used:  $\mathbf{1}\{a\geq b\} \leq a^2 /b^2$  for any  $a,b\geq 0$
- Bound on term  $\mathrm{T}_B$ . Before delving into the proof, first note that Lemma 19-(b) implies that
$$
\left\| f _ {h} ^ {*} \left(x _ {t, h}\right) - f _ {t, h} \left(x _ {t, h}\right) \right\| \leq \Delta_ {t, h} \left(x _ {t, h}\right). \tag {53}
$$
$$
\begin{array}{l} \mathrm {T} _ {B} = \sum_ {h = 1} ^ {H} \bar {Z} _ {t, h} \mathbf {1} \left\{\pi_ {t} \left(x _ {t, h}\right) \neq \pi^ {\star} \left(x _ {t, h}\right) \right\} \\ = \sum_ {h = 1} ^ {H} \mathbf {1} \left\{\operatorname {M a r g i n} \left(f _ {t, h} \left(x _ {t, h}\right)\right) > 2 \gamma \Delta_ {t, h} \left(x _ {t, h}\right), \pi_ {t} \left(x _ {t, h}\right) \neq \pi^ {\star} \left(x _ {t, h}\right) \right\}, \\ \end{array}
$$
where in the last line we just plugged in the query condition under which  $Z_{t,h} = 0$ . However note that the above two conditions inside the indicator imply that
$$
\begin{array}{l} 2 \gamma \Delta_ {t, h} \left(x _ {t, h}\right) <   \operatorname {M a r g i n} \left(f _ {t, h} \left(x _ {t, h}\right)\right) \\ \leq \phi \left(f _ {t, h} \left(x _ {t, h}\right)\right) \left[ \pi_ {t} \left(x _ {t, h}\right) \right] - \phi \left(f _ {t, h} \left(x _ {t, h}\right)\right) \left[ \pi^ {\star} \left(x _ {t, h}\right) \right] \\ \leq 2 \gamma \| f _ {t, h} (x _ {t, h}) - f _ {h} ^ {*} (x _ {t, h}) \|, \\ \end{array}
$$
where the second line uses the definition of  $\mathrm{Margin}(\cdot)$  and the fact that  $\pi_t(x_{t,h})\neq \pi^{\star}(x_{t,h})$  the last line is due to Lemma 12. Thus,
$$
\mathrm {T} _ {B} \leq \sum_ {h = 1} ^ {H} \mathbf {1} \left\{\left\| f _ {t, h} \left(x _ {t, h}\right) - f _ {h} ^ {\star} \left(x _ {t, h}\right) \right\| > \Delta_ {t, h} \left(x _ {t, h}\right) \right\},
$$
but the conditions inside the indicator in the above contradicts (53) (which holds with probability  $1 - \delta$ ). Thus, with probability at least  $1 - \delta$
$$
\mathrm {T} _ {B} = 0. \tag {54}
$$
Plugging in the bounds on  $\mathrm{T}_A$  and  $\mathrm{T}_B$  in (52), we get that with probability at least  $1 - \delta$
$$
R \left(\tau_ {t} ^ {\pi^ {*}}\right) - R \left(\tau_ {t} ^ {\pi_ {t}}\right) \leq 2 H \sum_ {h = 1} ^ {H} \mathbf {1} \left\{x _ {t, h} ^ {\pi^ {*}} \in \mathrm {X} _ {\varepsilon} \right\} + \frac {8 H \gamma^ {2}}{\varepsilon^ {2}} \sum_ {h = 1} ^ {H} Z _ {t, h} \| f _ {h} ^ {*} \left(x _ {t, h}\right) - f _ {t, h} \left(x _ {t, h}\right) \| ^ {2}. \tag {55}
$$
Step 2: Bound on total regret. Using the bound in (55) for each round  $t$ , we get that
$$
\begin{array}{l} \operatorname {R e g} _ {T} = \sum_ {t = 1} ^ {T} \left(R \left(\tau_ {t} ^ {\pi^ {*}}\right) - R \left(\tau_ {t} ^ {\pi_ {t}}\right)\right) \\ \leq 2 H \sum_ {h = 1} ^ {H} \sum_ {t = 1} ^ {T} \mathbf {1} \left\{x _ {t, h} ^ {\pi^ {\star}} \in \mathrm {X} _ {\varepsilon} \right\} + \frac {8 H \gamma^ {2}}{\varepsilon^ {2}} \sum_ {h = 1} ^ {H} \sum_ {t = 1} ^ {T} Z _ {t, h} \left\| f _ {h} ^ {\star} \left(x _ {t, h}\right) - f _ {t, h} \left(x _ {t, h}\right) \right\| ^ {2} \\ \leq 2 H \sum_ {h = 1} ^ {H} T _ {\varepsilon , h} + \frac {8 H \gamma^ {2}}{\varepsilon^ {2}} \sum_ {h = 1} ^ {H} \Psi_ {\delta} ^ {\ell_ {\phi}} \left(\mathcal {F} _ {h}, T\right), \tag {56} \\ \end{array}
$$
where in the last line we use the definition of  $T_{\varepsilon,h}$ , and plug in the bound in Lemma 19. Recall that  $T_{\varepsilon,h}$  denotes the number of times when the comparator policy  $\pi^{\star}$  enters the region  $\mathbf{X}_{\varepsilon}$  of states of small expert margin. Using the form of  $\Psi_{\delta}^{\ell_{\phi}}(\mathcal{F}_h,T)$  and ignoring log factors and constants, we get
$$
\operatorname {R e g} _ {T} = \widetilde {\mathcal {O}} \left(H \sum_ {h = 1} ^ {H} T _ {\varepsilon , h} + \frac {H \gamma^ {2}}{\lambda \varepsilon^ {2}} \sum_ {h = 1} ^ {H} \operatorname {R e g} ^ {\ell_ {\phi}} \left(\mathcal {F} _ {h}; T\right) + \frac {H \gamma^ {2}}{\lambda^ {2} \varepsilon^ {2}} \log (1 / \delta)\right).
$$
Since  $\varepsilon$  is a free parameter above, the final bound follows by taking inf over all feasible  $\varepsilon$ .
F.3.3 Total Number of Queries
Let  $N_{T}$  denote the total number of expert queries made by the learner within  $T$  rounds of interaction (with  $H$  steps per round). For  $t \leq T$ , let  $h_t$  denote the first timestep at which  $Z_{t,h_t} = 1$  at round  $t$ . Additionally, let  $\varepsilon > 0$  be a free parameter. Thus, we have that
$$
\begin{array}{l} N _ {T} = \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} Z _ {t, h} \tag {57} \\ \leq H \sum_ {t = 1} ^ {T} Z _ {t, h _ {t}} \\ = H \sum_ {t = 1} ^ {T} Z _ {t, h _ {t}} \mathbf {1} \left\{x _ {t, h _ {t}} \in \mathrm {X} _ {\varepsilon} \right\} + H \sum_ {t = 1} ^ {T} Z _ {t, h _ {t}} \mathbf {1} \left\{x _ {t, h _ {t}} \notin \mathrm {X} _ {\varepsilon} \right\} \\ \leq H \sum_ {t = 1} ^ {T} Z _ {t, h _ {t}} \mathbf {1} \left\{x _ {t, h _ {t}} \in \mathrm {X} _ {\varepsilon} \right\} + H \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} Z _ {t, h} \mathbf {1} \left\{x _ {t, h} \notin \mathrm {X} _ {\varepsilon} \right\} \\ = H \sum_ {t = 1} ^ {T} Z _ {t, h _ {t}} \mathbf {1} \left\{x _ {t, h _ {t}} \in \mathbb {X} _ {\varepsilon} \right\} + H \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} Z _ {t, h} \mathbf {1} \left\{x _ {t, h} \notin \mathbb {X} _ {\varepsilon}, \Delta_ {t, h} \left(x _ {t, h}\right) \leq \frac {\varepsilon}{4 \gamma} \right\} \\ + H \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} Z _ {t, h} \mathbf {1} \left\{x _ {t, h} \notin \mathrm {X} _ {\varepsilon}, \Delta_ {t, h} \left(x _ {t, h}\right) > \frac {\varepsilon}{4 \gamma} \right\} \\ = \mathrm {T} _ {C} + H \mathrm {T} _ {D} + H \mathrm {T} _ {E}, \\ \end{array}
$$
where  $\mathrm{T}_C$ ,  $\mathrm{T}_D$  and  $\mathrm{T}_E$  are the first, second and the third term respectively in the previous line. We bound them separately below:
- Bound on  $\mathbb{T}_C$ . Fix any  $t \leq T$ , and note that
$$
\begin{array}{l} \left(\mathrm {T} _ {C}\right) _ {t} = H Z _ {t, h _ {t}} \mathbf {1} \left\{x _ {t, h _ {t}} \in \mathrm {X} _ {\varepsilon} \right\} \\ = H Z _ {t, h _ {t}} \mathbf {1} \left\{x _ {t, h _ {t}} \in \mathbf {X} _ {\varepsilon} \right\} \mathbf {1} \left\{\forall h <   h _ {t}: \pi^ {\star} \left(x _ {t, h}\right) = \pi_ {t} \left(x _ {t, h}\right) \right\} \\ + H Z _ {t, h _ {t}} \mathbf {1} \left\{x _ {t, h _ {t}} \in \mathrm {X} _ {\varepsilon} \right\} \mathbf {1} \left\{\exists h <   h _ {t}: \pi^ {*} \left(x _ {t, h}\right) \neq \pi_ {t} \left(x _ {t, h}\right) \right\}. \tag {58} \\ \end{array}
$$
$$
Z _ {t, h _ {t}} \mathbf {1} \left\{x _ {t, h _ {t}} \in \mathbf {X} _ {\varepsilon} \right\} \mathbf {1} \left\{\exists h <   h _ {t}: \pi^ {\star} \left(x _ {t, h}\right) \neq \pi_ {t} \left(x _ {t, h}\right) \right\} \leq \sum_ {h = 1} ^ {h _ {t}} Z _ {t, h _ {t}} \mathbf {1} \left\{\pi^ {\star} \left(x _ {t, h}\right) \neq \pi_ {t} \left(x _ {t, h}\right) \right\}
$$
$$
\begin{array}{l} \leq \sum_ {h = 1} ^ {h _ {t}} Z _ {t, h _ {t}} \bar {Z} _ {t, h} \mathbf {1} \left\{\pi^ {\star} \left(x _ {t, h}\right) \neq \pi_ {t} \left(x _ {t, h}\right) \right\} \\ \leq \sum_ {h = 1} ^ {h _ {t}} \bar {Z} _ {t, h} \mathbf {1} \left\{\pi^ {\star} \left(x _ {t, h}\right) \neq \pi_ {t} \left(x _ {t, h}\right) \right\} \\ \end{array}
$$
where in second inequality above, we used the fact that  $Z_{t,h} = 0$  (and thus  $\bar{Z}_{t,h} = 1$ ) for all  $h \leq h_t$ , by the definition of  $h_t$ . However note that the right hand side in the last inequality is equivalent to the term  $\mathbb{T}_B$  defined above (where sum is now till  $h_t$  instead of  $H$ ). Thus, using the bound in (54) in the above, we immediately get that
$$
Z _ {t, h _ {t}} \mathbf {1} \left\{x _ {t, h _ {t}} \in \mathbb {X} _ {\varepsilon} \right\} \mathbf {1} \left\{\exists h <   h _ {t}: \pi^ {\star} \left(x _ {t, h}\right) \neq \pi_ {t} \left(x _ {t, h}\right) \right\} = 0.
$$
For the first term in (58), using the condition that  $\pi^{\star}(x_{t,h}) = \pi_t(x_{t,h})$  for all  $h\leq h_t$ , we get that  $x_{t,h} = x_{t,h}^{\pi^*}$  and thus
$$
\begin{array}{l} H Z _ {t, h _ {t}} \mathbf {1} \left\{x _ {t, h _ {t}} \in \mathrm {X} _ {\varepsilon} \right\} \mathbf {1} \left\{\forall h \leq h _ {t}: \pi^ {\star} \left(x _ {t, h}\right) = \pi_ {t} \left(x _ {t, h}\right) \right\} \leq H Z _ {t, h _ {t}} \mathbf {1} \left\{x _ {t, h _ {t}} ^ {\pi^ {\star}} \in \mathrm {X} _ {\varepsilon} \right\} \\ \leq H \mathbf {1} \{x _ {t, h _ {t}} ^ {\pi^ {*}} \in \mathbb {X} _ {\varepsilon} \} \\ \leq H \sum_ {h = 1} ^ {H} \mathbf {1} \left\{x _ {t, h} ^ {\pi^ {*}} \in \mathrm {X} _ {\varepsilon} \right\}. \\ \end{array}
$$
Gathering the two terms above, and plugging in the definition of  $T_{\varepsilon,h}$ , we get that
$$
\mathrm {T} _ {C} \leq H \sum_ {h = 1} ^ {H} \sum_ {t = 1} ^ {T} \mathbf {1} \left\{x _ {t, h} ^ {\pi^ {\star}} \in \mathrm {X} _ {\varepsilon} \right\} = H \sum_ {h = 1} ^ {H} T _ {\varepsilon , h}.
$$
- Bound on  $\mathbf{T}_D$ . Using the definition of the set  $\mathbf{X}_{\varepsilon}$  and  $Z_{t,h}$ , we note that
$$
\begin{array}{l} \left(\mathrm {T} _ {D}\right) _ {t} = \sum_ {h = 1} ^ {H} Z _ {t, h} \mathbf {1} \left\{x _ {t, h} \notin \mathrm {X} _ {\varepsilon}, \Delta_ {t, h} \left(x _ {t, h}\right) \leq \frac {\varepsilon}{4 \gamma} \right\} \\ = \sum_ {h = 1} ^ {H} \mathbf {1} \left\{\operatorname {M a r g i n} \left(f _ {t, h} \left(x _ {t, h}\right)\right) \leq 2 \gamma \Delta_ {t, h} \left(x _ {t, h}\right), \operatorname {M a r g i n} \left(f _ {h} ^ {\star} \left(x _ {t, h}\right)\right) > \varepsilon , \Delta_ {t, h} \left(x _ {t, h}\right) \leq \frac {\varepsilon}{4 \gamma} \right\} \tag {59} \\ \end{array}
$$
Recall that Lemma 19 implies that with probability at least  $1 - \delta$
$$
\left\| f _ {h} ^ {\star} \left(x _ {t, h}\right) - f _ {t, h} \left(x _ {t, h}\right) \right\| \leq \Delta_ {t, h} \left(x _ {t, h}\right),
$$
using which with Lemma 13 implies that
$$
\begin{array}{l} \operatorname {M a r g i n} \left(f _ {h} ^ {\star} \left(x _ {t, h}\right)\right) \leq \operatorname {M a r g i n} \left(f _ {t, h} \left(x _ {t, h}\right)\right) + 2 \gamma \left\| f _ {h} ^ {\star} \left(x _ {t, h}\right) - f _ {t, h} \left(x _ {t, h}\right) \right\| \\ \leq \operatorname {M a r g i n} \left(f _ {t, h} \left(x _ {t, h}\right)\right) + 2 \gamma \Delta_ {t, h} \left(x _ {t, h}\right). \\ \end{array}
$$
Using the above bound with the conditions in (59) implies that
$$
\begin{array}{l} \left(\mathrm {T} _ {D}\right) _ {t} = \sum_ {h = 1} ^ {H} \mathbf {1} \left\{\operatorname {M a r g i n} \left(f _ {h} ^ {\star} \left(x _ {t, h}\right)\right) \leq 4 \gamma \Delta_ {t, h} \left(x _ {t, h}\right), \operatorname {M a r g i n} \left(f _ {h} ^ {\star} \left(x _ {t, h}\right)\right) > \varepsilon , \Delta_ {t, h} \left(x _ {t, h}\right) \leq \frac {\varepsilon}{4 \gamma} \right\} \\ = \sum_ {h = 1} ^ {H} \mathbf {1} \left\{\operatorname {M a r g i n} \left(f _ {h} ^ {\star} \left(x _ {t, h}\right)\right) \leq \varepsilon , \operatorname {M a r g i n} \left(f _ {h} ^ {\star} \left(x _ {t, h}\right)\right) > \varepsilon \right\} \\ = 0, \\ \end{array}
$$
where the last equality holds because the two conditions in the indicator in the previous line can never occur simultaneously.
- Bound on  $\mathrm{T}_E$ . Note that
$$
\mathbf {T} _ {E} = \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} Z _ {t, h} \mathbf {1} \left\{x _ {t, h} \notin \mathrm {X} _ {\varepsilon}, \Delta_ {t, h} \left(x _ {t, h}\right) > \frac {\varepsilon}{4 \gamma} \right\}
$$
$$
\leq \sum_ {h = 1} ^ {H} \sum_ {t = 1} ^ {T} Z _ {t, h} \mathbf {1} \left\{\Delta_ {t, h} \left(x _ {t, h}\right) > \frac {\varepsilon}{4 \gamma} \right\}.
$$
An application of Lemma 20 in the above for each  $h \leq H$  implies that
$$
\mathbf {T} _ {E} \leq \sum_ {h = 1} ^ {H} \frac {3 2 0 \gamma^ {2} \Psi_ {\delta} ^ {\ell_ {\phi}} (\mathcal {F} _ {h} , T)}{\varepsilon^ {2}} \cdot \mathfrak {E} (\mathcal {F} _ {h}, \frac {\varepsilon}{8 \gamma}; f _ {h} ^ {\star}).
$$
Gathering the bound above, we get that
$$
N _ {T} \leq H \sum_ {h = 1} ^ {H} T _ {\varepsilon , h} + \frac {3 2 0 H \gamma^ {2}}{\varepsilon^ {2}} \sum_ {h = 1} ^ {H} \Psi_ {\delta} ^ {\ell_ {\phi}} (\mathcal {F} _ {h}, T) \cdot \mathfrak {E} (\mathcal {F} _ {h}, \frac {\varepsilon}{8 \gamma}; f _ {h} ^ {\star}).
$$
Plugging in the form of  $\Psi_{\delta}^{\ell_{\phi}}(\mathcal{F}_h,T)$  and ignoring log factors and constants, we get that
$$
N _ {T} \leq \widetilde {\mathcal {O}} \bigg (H \sum_ {h = 1} ^ {H} T _ {\varepsilon , h} + \frac {H \gamma^ {2}}{\lambda \varepsilon^ {2}} \sum_ {h = 1} ^ {H} \mathrm {R e g} ^ {\ell_ {\phi}} (\mathcal {F} _ {h}; T) \cdot \mathfrak {E} (\mathcal {F} _ {h}, ^ {\varepsilon} / 8 \gamma ; f _ {h} ^ {\star}) + \frac {H \gamma^ {2}}{\lambda^ {2} \varepsilon^ {2}} \log (1 / \delta) \bigg).
$$
Notice that  $\varepsilon$  is a free parameter above so the final bound follows by taking inf over all feasible  $\varepsilon$ . s
F.4 Proof for the Stochastic Setting
Algorithm 2 considers arbitrary deterministic dynamics  $\{\{\mathbb{T}_{t,h}\}_{h\leq H}\}_{t\leq T}$ . When the underlying dynamics  $\mathcal{T}$  is stochastic, we can simply simulate Algorithm 2, where we set  $\{\mathbb{T}_{t,h}\}_{h\leq H} = \mathcal{T}(\cdot ;\iota_t)$  where  $\iota_{t}$  is drawn i.i.d. for every  $t\leq T$ . In the following, we provide regret and query complexity bounds for stochastic dynamics.
Regret bound. Note that Theorem 4 bounds the difference of cumulative rewards of trajectories drawn using the policies  $\pi_t$  and  $\pi^{\star}$  on the adversarially chosen deterministic dynamics  $\{\mathbb{T}_{t,h}\}_{h = 1}^{H}$  respectively. In particular, we bound
$$
\mathrm {R e g} _ {T} = \sum_ {t = 1} ^ {T} \Big (R (\tau_ {t} ^ {\pi^ {*}}) - R (\tau_ {t} ^ {\pi_ {t}}) \Big).
$$
On the other hand, when the dynamics is stochastic, we aim to bound the gap between the expected values  $V^{\pi^{\star}} - V^{\pi_{t}}$  obtained under the stochastic dynamics  $\mathcal{T}$ . We obtain this bound by pushing all the stochasticity into the choice of random seed  $\iota$ . Fix any  $t \leq T$ , and consider the deterministic dynamics  $(\mathbb{T}_{t,1},\ldots ,\mathbb{T}_{t,H})$  obtained by setting the random seed to be  $\iota_t$  in the stochastic dynamics  $\mathcal{T}$ , i.e.  $(\mathbb{T}_{t,1},\dots,\mathbb{T}_{t,H}) := \mathcal{T}(\cdot ;\iota_t)$ . Thus, for any policy  $\pi$
$$
V ^ {\pi} = \mathbb {E} _ {\iota_ {t}} \left[ R \big (\tau_ {t} ^ {\pi} \big) \mid \left(\mathbb {T} _ {t, 1}, \dots , \mathbb {T} _ {t, H}\right) = \mathcal {T} \big (\mathbf {\Omega}; \iota_ {t} \big) \right].
$$
In the following, we will bound the difference in the value function  $V^{\pi} - V^{\pi_t}$ , by appealing to the regret bound in the proof of Theorem 4 using appropriate concentration inequalities. First, recall that in Algorithm 2, the dynamics  $\{\mathbb{T}_{t,h}\}_{h\leq H}$  is chosen before the round  $t$ , and that the policy  $\pi_t$  only depends on the interaction till round  $t - 1$ . Thus,
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} V ^ {\pi} - V ^ {\pi_ {t}} = \sum_ {t = 1} ^ {T} \mathbb {E} _ {\iota_ {t}} \left[ R \left(\tau_ {t} ^ {\pi}\right) - R \left(\tau_ {t} ^ {\pi^ {*}}\right) \right] \\ \leq \sum_ {t = 1} ^ {T} \mathbb {E} _ {\iota_ {t}} \left[ 2 H \sum_ {h = 1} ^ {H} \mathbf {1} \left\{x _ {t, h} ^ {\pi^ {\star}} \in \mathrm {X} _ {\varepsilon} \right\} + 2 H \sum_ {h = 1} ^ {H} \mathbf {1} \left\{\pi_ {t} \left(x _ {t, h} ^ {\pi_ {t}}\right) \neq \pi^ {\star} \left(x _ {t, h} ^ {\pi_ {t}}\right), x _ {t, h} ^ {\pi_ {t}} \notin \mathrm {X} _ {\varepsilon} \right\} \right], \\ \end{array}
$$
where the last holds due to Lemma 18 and the set  $\mathbf{X}_{\varepsilon}$  is defined in (50). An application of Lemma 4 in the above implies that with probability at least  $1 - \delta$ ,
$$
\sum_ {t = 1} ^ {T} V ^ {\pi} - V ^ {\pi_ {t}} \leq 4 H \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \mathbf {1} \{x _ {t, h} ^ {\pi^ {\star}} \in \mathbf {X} _ {\varepsilon} \} + 4 H \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \mathbf {1} \{\pi_ {t} (x _ {t, h} ^ {\pi_ {t}}) \neq \pi^ {\star} (x _ {t, h} ^ {\pi_ {t}}), x _ {t, h} ^ {\pi_ {t}} \notin \mathbf {X} _ {\varepsilon} \} + 3 2 H ^ {2} \log (2 / \delta).
$$
The rest of the proof is identical to the proof of Theorem 4 from (51) onwards. They query complexity can be similarly computed.
We next provide the proofs for learning from multiple experts.
F.5 Proof of Theorem 5
Algorithm 4 InterACTiVe ImitatiOn Learning VIa Active Queries to  $M$  Experts (RAVIOLI-M)
Input: Parameters  $\delta, \gamma, \lambda, T$ , function classes  $\{\mathcal{F}_h^m\}_{h \leq H, m \leq M}$ , online oracles  $\{\mathrm{Oracle}_h^m\}_{h \leq H, m \leq M}$  w.r.t.  $\ell_{\phi}$ .  
1: Set  $\Psi_{\delta}^{\ell_{\phi}}(\mathcal{F}_h^m, T) = \frac{4}{\lambda} \mathrm{Reg}^{\ell_{\phi}}(\mathcal{F}_h^m; T) + \frac{112}{\lambda^2} \log(4MH \log^2(T)/\delta)$ .  
2: Compute  $f_{1,h}^{m} = \mathrm{Oracle}_{1,h}(\emptyset)$  for each  $h \in [H]$  and  $m \in [M]$ .  
3: for  $t = 1$  to  $T$  do  
4: Nature chooses the state  $x_{t,1}$ .  
5: for  $h = 1$  to  $H$  do  
6: Define  $F_{t,h}^{m}(x) := [f_{t,h}^{1}(x), \ldots, f_{t,h}^{M}(x)]$ .  
7: Learner plays  $\widehat{y}_{t,h} = \text{SelectAction}(F_{t,h}(x_{t,h}))$ .  
8: Learner transitions to the next state in this round  $x_{t,h+1} \gets \mathbb{T}_{t,h}(x_{t,h}, \widehat{y}_{t,h})$ .  
9: For each  $m \in [M]$ , learner computes  $\Delta_{t,h}^{m}(x_{t,h}) := \max_{f \in \mathcal{F}_{h}^{m}} \|f(x_{t,h}) - f_{t,h}^{m}(x_{t,h})\|$  s.t.  $\sum_{s=1}^{t-1} Z_{s,h} \|f(x_{s,h}) - f_{s,h}^{m}(x_{s,h})\|^2 \leq \Psi_{\delta}^{\ell_{\phi}}(\mathcal{F}_{h}^{m}, T)$ . (60) and defines  $\vec{\Delta}_{t,h}(x_{t,h}) = [\Delta_{t,h}^{1}(x_{t,h}), \ldots, \Delta_{t,h}^{M}(x_{t,h})]$ .  
10: Learner decides whether to query:  $Z_{t,h} = \text{Query}(F_{t,h}(x_{t,h}), \vec{\Delta}_{t,h}(x_{t,h}))$   
11: if  $Z_{t,h} = 1$  then  
12: for  $m = 1$  to  $M$  do  
13: Learner queries expert  $m$  for its label  $y_{t,h}^{m}$  for  $x_{t,h}$ .  
14:  $f_{t+1,h}^{m} \gets \text{Oracle}_{t+1,h}^{m}(\{x_{t,h}, y_{t,h}\})$   
15: else  
16:  $f_{t+1,h}^{m} \gets f_{t,h}^{m}$  for each  $m \in [M]$ .
We first discuss the setup and the relevant notation. We are in the imitation learning setup introduced in Appendix F.3. In particular, the learner interacts with  $M$  experts in  $T$  episodes/rounds, each consisting of  $H$  timesteps. For any  $h \leq H$ , each of the  $M$  experts have a ground truth model  $f_{h}^{\star,m}$  respectively. Given the context  $x_{t,h}$  for time step  $h$  in round  $t$ , the learner plays the actions  $\widehat{y}_{t,h} \in [K]$ , and can additionally choose to query the experts (by setting  $Q_{t,h} = 1$ ) to receive noisy feedback  $y_{t,h}^{m} \sim \phi(f_{h}^{\star,m}(x_{t}))$  from each expert  $m \in [M]$ . After playing the chosen action  $\widehat{y}_{t,h}$ , the learner then transitions to state  $x_{t,h+1} \gets \mathbb{T}_{t,h}(x_{t,h}, \widehat{y}_{t,h})$  where  $\{\mathbb{T}_{t,h}\}_{h \in [H]}$  is a sequence of deterministic dynamics (unknown to the learner).
In Algorithm 4, for any round  $t \leq T$  and  $h \leq H$ :
- The aggregation function  $\mathcal{A}:\mathbb{R}^{K\times M}\mapsto \mathbb{R}^K$ , known to the learner, maps the predictions of the estimated experts to distributions over actions. Some illustrative examples are given below to illustrate the generality of our setup:
(a) Random aggregation: Given a state  $x_{h}$ , the aggregation rule chooses an expert uniformly at random and returns the label  $y_{h}$  sampled from its model. In particular,