SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where the value of the constant  $c$  will be set later. Furthermore, let  $p_t: \mathcal{X} \mapsto \Delta(\mathcal{Y})$  denote the distribution over the label chosen by the given algorithm at round  $t$  given the history  $(x_1, y_1, \ldots, x_{t-1}, y_{t-1})$ . We note that for any  $i \in [m]$ ,
$$
\begin{array}{l} \mathbb {E} _ {D _ {i}} \left[ \operatorname {R e g} _ {T} \right] \geq \mathbb {E} _ {D _ {i}} \left[ \sum_ {t = 1} ^ {T} \frac {\zeta}{2} \mathbb {E} _ {x \sim \mu} \mathbb {E} _ {\hat {y} \sim p _ {t} (x)} \left[ \mathbf {1} \{f _ {i} (x) \hat {y} <   0 \} \right] \right] \\ = \frac {T \zeta}{2} \mathbb {E} _ {D _ {i}} \left[ \frac {1}{T} \sum_ {t = 1} ^ {T} \mathbb {E} _ {x \sim \mu} \mathbb {E} _ {\hat {y} \sim p _ {t} (x)} \mathbf {1} \left\{\operatorname {s i g n} \left(f _ {i} (x)\right) \neq \hat {y} \right\} \right] \\ \geq \frac {T \zeta}{4} \mathbb {E} _ {D _ {i}} \left\| \frac {1}{T} \sum_ {t = 1} ^ {T} p _ {t} - \operatorname {s i g n} (f _ {i}) \right\| _ {L _ {1} (\mu)} \\ \end{array}
$$
where in the first inequality we used that  $f_{i}$  has a margin of at least  $\zeta / 2$  for any  $x \in \{x^{1}, \ldots, x^{m}\}$  as shown above, and the last inequality simply follows from the fact that
$$
\left| \mathbb {E} _ {\hat {y} \sim p _ {t} (x)} \mathbf {1} \{\operatorname {s i g n} \left(f _ {i} (x)\right) \neq \hat {y} \} \right| = \left| \mathbb {E} _ {\hat {y} \sim p _ {t} (x)} \left(\mathbf {1} \{\operatorname {s i g n} \left(f _ {i} (x)\right) = 1 \} - \mathbf {1} \{\widehat {y} = 0 \}\right) \right| = \frac {1}{2} \left| p _ {t} (x) - f _ {i} (x) \right|,
$$
and via an application of the Jensen's inequality. Using the property that for any distribution with margin at least  $\zeta /2$ , the algorithm satisfies  $\mathbb{E}_{D_i}[\mathrm{Reg}_T]\leq c\zeta T / m$ , the above implies that
$$
\mathbb {E} _ {D _ {i}} \left\| \frac {1}{T} \sum_ {t = 1} ^ {T} p _ {t} - \operatorname {s i g n} (f _ {i}) \right\| _ {L _ {1} (\mu)} \leq \frac {8 c}{m},
$$
$$
\frac {1}{m} \sum_ {i = 1} ^ {m} \mathbb {E} _ {D _ {i}} \left\| \frac {1}{T} \sum_ {t = 1} ^ {T} p _ {t} - \operatorname {s i g n} \left(f _ {i}\right) \right\| _ {L _ {1} (\mu)} \leq \frac {8 c}{m}. \tag {42}
$$
Hence, setting  $c = 64$  and using Markov's inequality, we have that
$$
\frac {1}{m} \sum_ {i = 1} ^ {m} \Pr_ {D _ {i}} \left(\left\| \frac {1}{T} \sum_ {t = 1} ^ {T} p _ {t} - \operatorname {s i g n} (f _ {i}) \right\| _ {L _ {1} (\mu)} > \frac {2}{m}\right) \leq \frac {1}{1 6}.
$$
Further note that  $\| \mathrm{sign}(f_j) - \mathrm{sign}(f_i)\|_{L_1(\mu)} > \frac{4}{m}$  and thus, condition on the fact that  $\nu$  is not equal to 0, we can identify  $\nu$  correctly with probability at least  $1 - 1/16$ . Thus, considering the reference measure  $D_0$  and using Fano's inequality, we must have that
$$
\log (2) \leq \frac {1}{m} \sum_ {i = 1} ^ {m} \mathrm {K L} \left(D _ {0} \| D _ {i}\right)
$$
Now we are left with bounding  $\mathrm{KL}(D_0\| D_i)$ . To this end, we first make a simple observation that the distribution on the  $x_{t}$ 's is the same under  $D_0$  and  $D_{i}$ . Hence on rounds  $t$  where  $Z_{t} = 0$  we do not query for the labels in these rounds, and thus we do not glean any new information to distinguish  $D_{i}$  from  $D_0$ . In other words, we only need to consider rounds when  $Z_{t} = 1$ . Hence we have:
$$
\operatorname {K L} \left(D _ {0} \| D _ {i}\right) = \mathbb {E} _ {D _ {0}} \left[ \sum_ {t = 1} ^ {T} Z _ {t} \cdot \operatorname {k l} \left(P _ {0} \left(y _ {t} = 1 | x _ {t}\right) \| P _ {i} \left(y _ {t} = 1 | x _ {t}\right)\right) \right]
$$
Assuming  $\zeta > 1/4$  and using the bound on KL between Bernoulli variables we get,
$$
\operatorname {K L} \left(D _ {0} \| D _ {i}\right) \leq \mathbb {E} _ {D _ {0}} \left[ 3 2 N _ {i} \zeta^ {2} + 8 \left(\max  _ {j \neq i} N _ {j}\right) \beta^ {2} \right]
$$
where  $N_{i} = |\{t:Z_{t} = 1,x_{t} = x^{i}\}$ , and we used item-(3) in the definition of star number for the  $\zeta^2$  term and item-(1) for the  $\beta^2$  term. Hence we have that,
$$
\begin{array}{l} \log (2) \leq \frac {1}{m} \sum_ {i = 1} ^ {m} \mathbb {E} _ {D _ {0}} \left[ 3 2 N _ {i} \zeta^ {2} + 8 \left(\max  _ {j \neq i} N _ {j}\right) \beta^ {2} \right] \\ \leq \frac {3 2}{m} \mathbb {E} _ {D _ {0}} \left[ \sum_ {i = 1} ^ {m} N _ {i} \right] \zeta^ {2} + 8 \mathbb {E} _ {D _ {0}} \left[ \max _ {j} N _ {j} \right] \beta^ {2} \\ \leq \frac {3 2}{m} \mathbb {E} _ {D _ {0}} [ N _ {T} ] \zeta^ {2} + 8 \mathbb {E} _ {D _ {0}} [ N _ {T} ] \beta^ {2} \\ \end{array}
$$
Since  $\beta^2\leq \zeta^2 /m$  , we have that
$$
\log (2) \leq \frac {4 0}{m} \mathbb {E} _ {D _ {0}} \left[ N _ {T} \right] \zeta^ {2}
$$
Hence we conclude that,
$$
\mathbb {E} _ {D _ {0}} \left[ N _ {T} \right] \geq \frac {\log (2) m}{4 0 \zeta^ {2}}
$$
which yields the desired lower bound.
Proof of Corollary 2. Consider the function class  $\mathcal{F} = \{f_0, f_1, \ldots, f_{\sqrt{T}}\}$  where  $f_0(x_i) = 1/2 + \zeta$  for every  $x_1, \ldots, x_{\sqrt{T}}$ , and where  $f_i(x_i) = 1/2 - \zeta$  and  $f_i(x_j) = 1/2 + \zeta$  for any  $j \neq i$ . Note that selecting  $f^{\star} = f_0$  and  $f_1, \ldots, f_m$  on  $x_1, \ldots, x_{\sqrt{T}}$ , we can show that the star number  $\mathfrak{s}^{\mathrm{val}}(\mathcal{F}, 1/8, 1/2) = O(\sqrt{T})$  (the disagreement coefficient of  $\mathcal{F}$  is also  $O(\sqrt{T})$ ). Thus, using the converse of Theorem 3 we get that if the number of queries is smaller than  $\sqrt{T}$ , then there must exist some data distribution under which the regret bound of the algorithm is larger than  $\sqrt{T}$ .
F Imitation Learning: Learning from Single Expert
F.1 Imitation Learning Tools
We first recall useful additional notation. Recall that a policy  $\pi$  is a mapping from the states  $\mathcal{X}$  to actions  $\mathcal{A}$ . For any  $h \leq H$ , and random variable  $Z(x_{h},a_{h})$ , we use the notation  $\mathbb{E}_{\pi}[Z(x_h,a_h)]$  to denote the expectation w.r.t. trajectories  $\{x_{1},a_{1}\ldots ,x_{H},a_{H}\}$  sampled using the policy  $\pi$ .
The following lemma is the standard performance difference lemma, well known in the imitation learning and reinforcement learning literature.
Lemma 17 (Performance Difference Lemma; Kakade and Langford [2002], Ross and Bagnell [2014]). For any MDP  $M$ , and any two arbitrary stationary policies  $\pi$  and  $\widetilde{\pi}$ , we have
$$
V ^ {\pi} - V ^ {\widetilde {\pi}} = \sum_ {h = 1} ^ {H} \mathbb {E} _ {x _ {h}, a _ {h} \sim d _ {h} ^ {\widetilde {\pi}}} \left[ - A _ {h} ^ {\pi} \left(x _ {h}, a _ {h}\right) \right],
$$
where  $A^{\pi}$  is the advantage function of the policy  $\pi$  in MDP  $M$ , i.e.,  $A_h^\pi (x,a) = Q_h^\pi (x,a) - V_h^\pi (x)$ .
F.2 Proof of Proposition 1
MDP construction. The underlying MDP is a binary tree of depth  $H$ . In particular, we construct the deterministic MDP  $M = (\mathcal{X},\mathcal{A},P,r,x_1)$  where state space  $\mathcal{X} = \cup_{h=1}^{H}\mathcal{X}_h$  with  $\mathcal{X}_h = \{x_{h,i}\}_{i=1}^{2^{h-1}}$  (we assume that  $x_1 = x_{1,1}$ ), action space  $\mathcal{A} = \{0,1\}$ , reward  $r$  is such that  $r(x,a) = \operatorname{Bern}\left(\frac{1}{2} + \frac{1}{4}\mathbf{1}\{x = x^{\star}\}\right)$  for some special state  $x^{\star} \in \mathcal{X}_H$ . The transition dynamics  $P$  is deterministic and defines a binary tree over  $\mathcal{X}$ , i.e. for any  $h$  and  $x_{h,i}$ ,  $P(x' | x_{h,i},a) = 1$  if  $x' = x_{h+1,2i-1}$  and  $a = 0$ , or  $x' = x_{h+1,2i+1}$  and  $a = 1$ , else  $P(x' | x,a) = 0$ .
We next define the expert policy  $\pi^{\star}$ , expert model  $f^{\star}$  and the class  $\mathcal{F}$ . First, for any path  $\tau = (x_{1},a_{1},\ldots ,x_{H},a_{H})$  from the root state  $x_{1}$  to a terminal state  $x_{H}$  at the layer  $H$ , define the policy  $\pi_{\tau}$  as
$$
\pi_ {\tau} (x _ {h}) = \left\{ \begin{array}{l l} a _ {h} & \text {i f} \quad (x _ {h}, a _ {h}) \in \tau \\ \bar {a} _ {h} \gets \text {U n i f o r m} (\{0, 1 \}) & \text {o t h e r w i s e} \end{array} \right..
$$
In particular,  $\pi_{\tau}$  is defined such that for any state on the path  $\tau$ , we choose the corresponding action in  $\tau$ , and for any state outside of  $\tau$ , we choose an arbitrary (deterministic) action. Let  $\mathcal{T}$  denote the set of all  $2^{H}$  many paths from the root note  $x_{1}$  to a leaf node  $x_{H} \in \mathcal{X}_{H}$ . We define the class  $\Pi = \{\pi_{\tau} \mid \tau \in \mathcal{T}\}$ , and  $\mathcal{F} = \{f_{\tau} \mid \tau \in \mathcal{T}\}$ , where for any  $\tau$ , we define  $f_{\tau}: \mathcal{X} \mapsto \mathbb{R}^2$  as
$$
f _ {\tau} (x) = \left\{ \begin{array}{l l} \left(3 / 4,   1 / 4\right) & \text {i f} \qquad \pi_ {\tau} (x) = 0 \\ \left(1 / 4,   3 / 4\right) & \text {i f} \qquad \pi_ {\tau} (x) = 1 \end{array} \right..
$$
Next, let  $\tau^{\star} = (x_{1},a_{1}^{\prime},x_{2}^{\prime},\ldots ,x_{H - 1}^{\prime},a_{H - 1}^{\prime},x^{\star})$  be the path from the root  $x_{1}$  to the special state  $x^{\star}\in \mathcal{X}_{H}$  on the underlying binary tree. We finally define  $\pi^{\star} = \pi_{\tau^{\star}}$  and  $f^{\star} = f_{\tau^{\star}}$
Lower bound. Given the MDP construction, the class  $\mathcal{F}$ ,  $f^{\star}$  and  $\pi^{\star}$  above, we now proceed to the desired lower bound for non-interactive imitation learning. First, note that  $\pi^{\star}(x) = \operatorname{argmax}_{a}(f^{\star}(x)[a])$  for any  $x \in \mathcal{X}$ . Furthermore,  $\operatorname{Margin}(f^{\star}((x)) = \frac{1}{2}|f(x)[0] - f(x)[1]| = \frac{1}{4}$  for all  $x \in \mathcal{X}$ . Thus, for any  $\varepsilon \leq \frac{1}{4}$ ,  $T_{\varepsilon,h} = 0$ .
Next, for any policy  $\pi$ , note that  $V^{\pi} = \frac{1}{2} + \frac{1}{4}\mathbf{1}\{\pi = \pi^{\star}\}$ . Thus,  $\pi^{\star}$  is the unique  $1/8$ -suboptimal policy. Additionally, consider a noisy expert that draws its label according to (2) with the link function  $\phi(z) = z$ , i.e., on the state  $x$ , the expert draws its label from  $a \sim f^{\star}(x)$ . Now, suppose that the learner is given a dataset  $\mathcal{D}$  of  $m$  many trajectories drawn this noisy expert. There are two scenarios: either  $\mathcal{D}$  does not contain  $\tau^{\star}$ , or  $\mathcal{D}$  contains the trajectory  $\tau^{\star}$ .
- In the first case, the learner is restricted to finding  $\pi^{\star}$  by eliminating all other  $\pi \neq \pi^{\star}$  using the observations  $\mathcal{D}$ . Since,  $|\Pi| = 2^{H}$  and each policy in the class is associated with a different path on the tree, we must have that  $m = O(2^{H})$ .
- In the second case, we need  $\tau^{\star} \in \mathcal{D}$ . However, note that probability of observing the trajectory  $\tau^{\star}$  when following the actions proposed by the noisy expert is  $\operatorname*{Pr}(\tau^{\star} \mid a_h \sim f^{\star}(x_h)) = (3/4)^H$ . Thus, in order to observe  $\tau^{\star}$  with probability at least  $3/4$  in the dataset  $\mathcal{D}$ , we need  $m = O((4/3)^H)$ .
In both the scenarios above, we need to collect exponentially many samples.
F.3 Proof of Theorem 4
Before delving into the proof, we recall the relevant notation. In Algorithm 2, for any  $h \leq H$ ,
- The label  $y_{t,h} \sim \phi(f^*(x_{t,h}))$ , where  $\phi$  denotes the link-function given in (2).
- The function SelectAction  $(f_{t,h}(x_{t,h})) = \operatorname{argmax}_k \phi(f_{t,h}(x_{t,h}))[k]$ .
- For any vector  $v \in \mathbb{R}^K$ , the margin is given by the gap between the value at the largest and the second largest coordinate (under the link function  $\phi$ ), i.e.
- We define  $T_{\varepsilon} = \sum_{t=1}^{T} \sum_{h=1}^{H} \mathbf{1}\{\mathrm{Margin}(f_h^\star(x_{t,h})) \leq \varepsilon\}$  to denote the number of samples within  $T$  rounds of interaction for which the margin w.r.t.  $f_h^\star$  is smaller than  $\varepsilon$ .
- The trajectory at round  $t$  is generated using the dynamics  $\{\mathbb{T}_{t,h}\}_{h\leq H}$  to determine the states that the learner observes, starting from the state  $x_{1}$ .
- At round  $t$ , the learner collects data using the policy  $\pi_t$  such that at time  $h$ , and state  $x$ , the action  $\pi_t(x) = \text{SelectAction}(f_{t,h}(x_{t,h}))$ .
- For any policy  $\pi$ , let  $\tau_t^\pi$  denote the (counterfactual) trajectory that one would obtain by running  $\pi$  on the deterministic dynamics  $\{\mathbb{T}_{t,h}\}_{h\leq H}$  with the start state  $x_{t,1}$ , i.e.
$$
\tau_ {t} ^ {\pi} = \left\{x _ {t, 1} ^ {\pi}, \pi \left(x _ {t, 1} ^ {\pi}\right), \dots , x _ {t, H} ^ {\pi}, \pi \left(x _ {t, H} ^ {\pi}\right) \right\} \tag {43}
$$
where  $x_{t,1}^{\pi} = x_{t,1}$  and  $x_{t,h + 1}^{\pi} = \mathbb{T}_{t,h}(x_{t,h}^{\pi},\pi (x_{t,h}^{\pi}))$
- For a trajectory  $\tau = \{x_1, a_1, \ldots, x_H, a_H\}$ , we define the total return
$$
R (\tau) = \sum_ {h = 1} ^ {H} r \left(x _ {h}, a _ {h}\right). \tag {44}
$$
- Additionally, for any policy  $\pi$  and dynamics  $\{\mathbb{T}_h\}_{h\leq H}$ , we define the trajectory obtained by running the policy  $\pi$  as
$$
\tau^ {\pi} = \left\{x _ {1} ^ {\pi}, \pi_ {1} \left(x _ {1} ^ {\pi}\right), x _ {2} ^ {\pi}, \dots \right\}.
$$
- We define the function  $\operatorname{Gap}:\mathbb{R}^K\times [K]\mapsto \mathbb{R}^+$  as
$$
\operatorname {G a p} (v, k) = \max  _ {k ^ {\prime}} \phi (v) \left[ k ^ {\prime} \right] - \phi (v) [ k ], \tag {45}
$$
to denote the gap between the largest and the  $k$ -th coordinate of  $v$ , and note that  $\operatorname{Margin}(v) \leq \operatorname{Gap}(v, k)$  for all  $k \neq k^{\star}$  (due to Lemma 9).
F.3.1 Supporting Technical Results
We first define a useful technical lemma which allows us to bound the gap between the total returns for policies  $\pi_1$  and  $\pi_2$ , under the dynamics  $\{\mathbb{T}_h\}_{h\leq H}$ . Recall that for a policy  $\pi$ , we define the trajectory  $\tau^{\pi}$  under  $\{\mathbb{T}_h\}_{h\leq H}$  and the start state  $x_{1}$  as the trajectory  $\{x_1^\pi, \pi(x_1^\pi), \ldots, x_H^\pi, \pi(x_H^\pi)\}$  where  $x_{1}^{\pi} = x_{1}$ , and  $x_{h+1}^{\pi} \gets \mathbb{T}_h(x_h^\pi, \pi(x_h^\pi))$ .
Lemma 18. Let  $\{\mathbb{T}_h\}_{h\leq H}$  be a deterministic dynamics, and let  $x_{1}$  be the start state. Let  $\pi_1$  and  $\pi_2$  be any two deterministic policies, and let  $\tau^{\pi_1} = \{x_1^{\pi_1},\pi_1(x_1^{\pi_1}),x_2^{\pi_1},\ldots \}$  and  $\tau^{\pi_2} = \{x_1^{\pi_2},\pi_1(x_1^{\pi_2}),x_2^{\pi_2},\ldots \}$  be two trajectories drawn using  $\pi_1$  and  $\pi_2$  on  $\{\mathbb{T}_h\}_{h\leq H}$  with start state  $x_{1}$ . Then, for any set  $\mathbf{x}\subseteq \mathcal{X}$ , the total trajectory rewards satisfy
$$
R \left(\tau^ {\pi_ {1}}\right) - R \left(\tau^ {\pi_ {2}}\right) \leq 2 H \sum_ {h = 1} ^ {H} \mathbf {1} \left\{x _ {h} ^ {\pi_ {1}} \in \mathrm {X} \right\} + 2 H \sum_ {h = 1} ^ {H} \mathbf {1} \left\{\pi_ {2} \left(x _ {h} ^ {\pi_ {2}}\right) \neq \pi_ {1} \left(x _ {h} ^ {\pi_ {2}}\right), x _ {h} ^ {\pi_ {2}} \notin \mathrm {X} \right\}.
$$
Proof. Let  $\mathfrak{h} \leq H$  denote the first timestep at which the policies  $\pi_1$  and  $\pi_2$  choose different actions under  $\{\mathbb{T}_h\}_{h \leq H}$ . Since the trajectories  $\tau^{\pi_1} = \{x_1^{\pi_1}, \pi_1(x_1^{\pi_1}), x_2^{\pi_1}, \ldots\}$  and  $\tau^{\pi_2} = \{x_1^{\pi_2}, \pi_1(x_1^{\pi_2}), x_2^{\pi_2}, \ldots\}$  are obtained by evolving through (the deterministic dynamics)  $\{\mathbb{T}_h\}_{h \leq H}$  using policies  $\pi_1$  and  $\pi_2$  respectively, and with the same state state  $x_1$ , we have that
$$
x _ {h} ^ {\pi_ {1}} = x _ {h} ^ {\pi_ {2}} \quad \text {f o r a l l} h \leq \mathfrak {h},
$$
$$
\pi_ {1} \left(x _ {h} ^ {\pi_ {1}}\right) = \pi_ {2} \left(x _ {h} ^ {\pi_ {2}}\right) \quad \text {f o r a l l} h \leq \mathfrak {h} - 1. \tag {46}
$$
Starting from the definition of the cumulative reward  $R(\cdot)$ , we have that
$$
\begin{array}{l} R \left(\tau^ {\pi_ {1}}\right) - R \left(\tau^ {\pi_ {2}}\right) = \sum_ {h = 1} ^ {H} \left(r \left(x _ {h} ^ {\pi_ {1}}, \pi_ {1} \left(x _ {h} ^ {\pi_ {1}}\right)\right) - r \left(x _ {h} ^ {\pi_ {2}}, \pi_ {2} \left(x _ {h} ^ {\pi_ {2}}\right)\right)\right) \\ = \sum_ {h = 1} ^ {\mathfrak {h} - 1} \left(r \left(x _ {h} ^ {\pi_ {1}}, \pi_ {1} \left(x _ {h} ^ {\pi_ {1}}\right)\right) - r \left(x _ {h} ^ {\pi_ {2}}, \pi_ {2} \left(x _ {h} ^ {\pi_ {2}}\right)\right)\right) + \sum_ {h = \mathfrak {h}} ^ {H} \left(r \left(x _ {h} ^ {\pi_ {1}}, \pi_ {1} \left(x _ {h} ^ {\pi_ {1}}\right)\right) - r \left(x _ {h} ^ {\pi_ {2}}, \pi_ {2} \left(x _ {h} ^ {\pi_ {2}}\right)\right)\right) \\ = \sum_ {h = \mathfrak {h}} ^ {H} \left(r \left(x _ {h} ^ {\pi_ {1}}, \pi_ {1} \left(x _ {h} ^ {\pi_ {1}}\right)\right) - r \left(x _ {h} ^ {\pi_ {2}}, \pi_ {2} \left(x _ {h} ^ {\pi_ {2}}\right)\right)\right), \\ \end{array}
$$
where the last line uses the fact that the trajectories (and thus the rewards)  $\tau^{\pi_1}$  and  $\tau^{\pi_2}$  are identical for the first  $\mathfrak{h} - 1$  states and actions (see (46)). Since (46) also implies that  $x_{\mathfrak{h}}^{\pi_1} = x_{\mathfrak{h}}^{\pi_2}$ , for the ease of notation we define  $x_{\mathfrak{h}} = x_{\mathfrak{h}}^{\pi_1} = x_{\mathfrak{h}}^{\pi_2}$ . Using the fact that  $|r(x, a)| \leq 1$  and that  $\pi_1(x_{\mathfrak{h}}) \neq \pi_2(x_{\mathfrak{h}})$  (by definition of  $\mathfrak{h}$ ), we can bound the above as
$$
\begin{array}{l} R \left(\tau^ {\pi_ {1}}\right) - R \left(\tau^ {\pi_ {2}}\right) \leq 2 (H - \mathfrak {h} + 1) \mathbf {1} \left\{\pi_ {1} \left(x _ {\mathfrak {h}}\right) \neq \pi_ {2} \left(x _ {\mathfrak {h}}\right) \right\} \\ \leq 2 H \mathbf {1} \left\{\pi_ {1} \left(x _ {\mathfrak {h}}\right) \neq \pi_ {2} \left(x _ {\mathfrak {h}}\right) \right\} \\ = 2 H \mathbf {1} \left\{\pi_ {1} \left(x _ {\mathfrak {h}}\right) \neq \pi_ {2} \left(x _ {\mathfrak {h}}\right), x _ {\mathfrak {h}} \in \mathrm {X} \right\} + 2 H \mathbf {1} \left\{\pi_ {1} \left(x _ {\mathfrak {h}}\right) \neq \pi_ {2} \left(x _ {\mathfrak {h}}\right), x _ {\mathfrak {h}} \notin \mathrm {X} \right\} \\ \leq 2 H \mathbf {1} \left\{x _ {\mathfrak {h}} \in \mathrm {X} \right\} + 2 H \mathbf {1} \left\{\pi_ {1} \left(x _ {\mathfrak {h}}\right) \neq \pi_ {2} \left(x _ {\mathfrak {h}}\right), x _ {\mathfrak {h}} \notin \mathrm {X} \right\} \\ = 2 H 1 \left\{x _ {\mathfrak {h}} ^ {\pi_ {1}} \in \mathbf {X} \right\} + 2 H 1 \left\{\pi_ {2} \left(x _ {\mathfrak {h}} ^ {\pi_ {2}}\right) \neq \pi_ {1} \left(x _ {\mathfrak {h}} ^ {\pi_ {2}}\right), x _ {\mathfrak {h}} ^ {\pi_ {2}} \notin \mathbf {X} \right\} \\ \leq 2 H \sum_ {h = 1} ^ {H} \mathbf {1} \left\{x _ {h} ^ {\pi_ {1}} \in \mathrm {X} \right\} + 2 H \sum_ {h = 1} ^ {H} \mathbf {1} \left\{\pi_ {2} \left(x _ {h} ^ {\pi_ {2}}\right) \neq \pi_ {1} \left(x _ {h} ^ {\pi_ {2}}\right), x _ {h} ^ {\pi_ {2}} \notin \mathrm {X} \right\}, \\ \end{array}
$$
where the equality in second last line plugs in the fact that  $x_{\mathfrak{h}} = x_{\mathfrak{h}}^{\pi_1} = x_{\mathfrak{h}}^{\pi_2}$ , and the last inequality is a straightforward upper bound.
We will also be using Lemma 9 and Lemma 13 from Appendix E.3 for bounding the Margin in the regret bound proofs. Finally, we note the following properties of the function  $f_h^\star$ .
Lemma 19. With probability at least  $1 - \delta$ , the function  $f_h^\star$  satisfies for any  $h \leq H$  and  $t \leq T$ ,
(a)  $\sum_{s=1}^{t-1} Z_{s,h} \| f_h^\star(x_{s,h}) - f_{s,h}(x_{s,h}) \|^2 \leq \Psi_\delta^{\ell_\phi}(\mathcal{F}_h, T)$ ,
(b)  $\| f_h^\star (x_{t,h}) - f_{t,h}(x_{t,h})\| \leq \Delta_{t,h}(x_{t,h}),$
where  $\Psi_{\delta}^{\ell_{\phi}}(\mathcal{F}_h,T) = \frac{4}{\lambda}\mathrm{Reg}^{\ell_{\phi}}(\mathcal{F}_h;T) + \frac{112}{\lambda^2}\log (4H\log^2 (T) / \delta).$
Proof. (a) We first note that we do not query oracle when  $Z_{s,h} = 0$ , and thus we can ignore the time steps for which  $Z_{s,h} = 0$ . Hence, for each  $h \in [H]$ , applying Lemma 5 along with the fact that  $\sup_{x,f \in \mathcal{F}_h} |f(x)| \leq 1$  yields
$$
\sum_ {s = 1} ^ {t - 1} Z _ {s, h} \left\| f _ {h} ^ {\star} \left(x _ {s, h}\right) - f _ {s, h} \left(x _ {s, h}\right) \right\| ^ {2} \leq \frac {4}{\lambda} \operatorname {R e g} ^ {\ell_ {\phi}} \left(\mathcal {F} _ {h}; T\right) + \frac {1 1 2}{\lambda^ {2}} \log \left(4 \log^ {2} (T) / \delta\right)
$$
for all  $t \leq T$ . Then, we take the union bound for all  $h \in [H]$ , which completes the proof.
(b) The second part follows from using the observation in part-(a) that  $f_h^\star$  satisfies the constraint in the definition of  $\Delta_{t,h}$  given in (6), and thus  $\| f_h^\star(x_{t,h}) - f_{t,h}(x_{t,h}) \| \leq \Delta_{t,h}(x_{t,h})$ .
The next technical lemma bounds the number of times when  $\Delta_{t,h}(x_{t,h})\geq \zeta$  and we query the expert. Note that Lemma 20 holds even if the sequence  $\{x_{t,h}\}_{t\leq T}$  was adversarially generated.
Lemma 20. Let  $f^{\star}$  satisfy Lemma 19, and let  $\Delta_{t,h}(x_t)$  be defined in (6). Suppose we run Algorithm 2 on data sequence  $\{\{x_{t,h}\}_{h\leq H}\}_{t\leq T}$ , and let  $Z_{t,h}$  be as defined in line 9. Then, for any  $\zeta > 0$ , with probability at least  $1 - H\delta$ , for any  $h\leq H$
$$
\sum_ {t = 1} ^ {T} Z _ {t, h} \mathbf {1} \{\Delta_ {t, h} (x _ {t, h}) \geq \zeta \} \leq \frac {2 0 \Psi_ {\delta} ^ {\ell_ {\phi}} (\mathcal {F} _ {h} , T)}{\zeta^ {2}} \cdot \mathfrak {E} (\mathcal {F} _ {h}, \frac {\zeta}{2}; f _ {h} ^ {\star}),
$$
Proof. The proof is identical to the proof of Lemma 11 by replacing all  $|\cdot|$  with  $\|\cdot\|$ , and substitute the corresponding bounds for  $f_h^\star$  via Lemma 19 (instead of using Lemma 10). We skip the proof for conciseness.
F.3.2 Regret Bound
Recall that the trajectory at round  $t$  is generated using the dynamics  $\{\mathbb{T}_{t,h}\}_{h\leq H}$ . Define the policy  $\pi_t$  and  $\pi^*$  such that for any  $h\leq H$  and  $x\in \mathcal{X}_h$
$$
\pi_ {t} \left(x _ {h}\right) = \operatorname {S e l e c t A c t i o n} \left(f _ {t, h} \left(x _ {h}\right)\right), \quad \text {a n d}, \quad \pi^ {\star} \left(x _ {h}\right) = \operatorname {S e l e c t A c t i o n} \left(f _ {h} ^ {\star} \left(x _ {h}\right)\right). \tag {47}
$$
Furthermore, for any policy  $\pi$ , let  $\tau_t^\pi$  denote the trajectory that one would obtain by running  $\pi$  on the deterministic dynamics  $\{\mathbb{T}_{t,h}\}_{h\leq H}$  with the start state  $x_{t,1}$ , i.e.
$$
\tau_ {t} ^ {\pi} = \left\{x _ {t, 1} ^ {\pi}, \pi \left(x _ {t, 1} ^ {\pi}\right), \dots , x _ {t, H} ^ {\pi}, \pi \left(x _ {t, H} ^ {\pi}\right) \right\} \tag {48}
$$
where  $x_{t,1}^{\pi} = x_{t,1}$  and  $x_{t,h+1}^{\pi} = \mathbb{T}_{t,h}(x_{t,h}^{\pi}, \pi(x_{t,h}^{\pi}))$ . Note that Algorithm 2 collects trajectories using the policy  $\pi_t$  at round  $t$ . Thus, we have that
$$
x _ {t, h} ^ {\pi_ {t}} = x _ {t, h}, \tag {49}
$$
where  $x_{t,h}$  denotes the state at time step  $h$  in round  $t$  of Algorithm 2. Finally, let  $\varepsilon > 0$  be a free parameter. We now have all the notation to proceed to the proof on our regret bound.
Step 1: Bounding the difference in return at round  $t$ . Fix any  $t \leq T$ , and let  $\tau_t^{\pi_t}$  and  $\tau_t^{\pi^\star}$  denote the trajectories that would have been sampled using the policies  $\pi_t$  and the policy  $\pi^\star$  at round  $t$ . Furthermore, define the set  $X_{\varepsilon}$  as
$$
\mathrm {X} _ {\varepsilon} := \bigcup_ {h = 1} ^ {H} \left\{x \in \mathcal {X} _ {h} \mid \operatorname {M a r g i n} \left(f _ {h} ^ {\star} (x)\right) \leq \varepsilon \right\} \tag {50}
$$
Using Lemma 18 for the policies  $\pi_t$  and  $\pi^{\star}$ , and the set  $\mathbf{X}_{\varepsilon}$  defined above, we get that