SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
y _ {h} \sim \phi \left(f _ {h} ^ {\star , \widetilde {m}} \left(x _ {h}\right)\right), \quad \text {w h e r e} \quad \widetilde {m} \sim \operatorname {U n i f o r m} ([ M ]).
$$
Here, the distribution  $\mathcal{A}\big(\phi (f_h^{\star ,1}(x_h)),\ldots ,\phi (f_h^{\star ,M}(x_h))\big) = \frac{1}{M}\sum_{m = 1}^{M}\phi (f^{\star ,m}(x_h)).$
(b) Majority label:  $\mathcal{A}$  is deterministic. Given a state  $x_{t}$ , the aggregation rule chooses the label  $y_{h} \in [K]$  which is the top preference for the majority of the experts. In particular,
$$
y _ {h} = \mathcal {A} \big (\phi \big (f _ {h} ^ {\star , 1} (x _ {h}) \big), \ldots , \phi \big (f _ {h} ^ {\star , M} (x _ {h}) \big) \big) = \operatorname * {a r g m a x} _ {k \in [ K ]} \sum_ {m = 1} ^ {M} \mathbf {1} \{k = \operatorname * {a r g m a x} _ {\widetilde {k} \in [ K ]} \phi \big (f _ {h} ^ {\star , m} (x _ {h}) [ \widetilde {k} ] \big) \}.
$$
(c) Majority-of-confident-experts: This aggregation rule is also deterministic, and was first introduced in Dekel et al. [2012]. Given a state  $x_{t}$ , the aggregation rule chooses the label  $y_{h} \in [K]$  which is the top preference for the majority of the  $\rho$ -confident experts on  $x_{h}$  i.e. the experts whose margin on  $x_{h}$  is larger than  $\rho$ . In particular,
$$
\begin{array}{l} y _ {h} = \mathcal {A} \left(\phi \left(f _ {h} ^ {\star , 1} \left(x _ {h}\right)\right), \dots , \phi \left(f _ {h} ^ {\star , M} \left(x _ {h}\right)\right)\right) \\ = \operatorname * {a r g m a x} _ {k \in [ K ]} \sum_ {m = 1} ^ {M} \mathbf {1} \left\{k = \operatorname * {a r g m a x} _ {\widetilde {k} \in [ K ]} \phi \big (f _ {h} ^ {\star , m} \big (x _ {h} \big) [ \widetilde {k} ] \big) \text {a n d M a r g i n} \big (\phi \big (f _ {h} ^ {\star , m} \big (x _ {h} \big) \big) > \rho \big) \right\}, \\ \end{array}
$$
where  $\operatorname{Margin}(f_h^{\star,m}(x_h) > \rho) = \max_{k_1} (\phi(f_h^{\star,m}(x_h))[k_1] - \left(\max_{k_2 \neq k_1} \phi(f_h^{\star,m}(x_h))[k_2]\right))$ . This aggregation rule is useful when there may be many experts that give equal weights to the top and the second-to-top coordinates w.r.t. their respective models, and hence can not be confidently accounted for in the majority rule. Furthermore, instead of choosing the majority label, similar to Dekel et al. [2012], one can also return the label sampled according to a uniform distribution over  $\rho$ -confident experts.
- The function SelectAction:  $\mathbb{R}^{K\times M}\mapsto [K]$  chooses the action to play at round  $t$ , and is defined as:
$$
\operatorname {S e l e c t A c t i o n} \left(F _ {t, h} \left(x _ {t, h}\right)\right) = \underset {k} {\operatorname {a r g m a x}} \mathscr {A} \left(\phi \left(F _ {t, h} \left(x _ {t, h}\right)\right)\right) [ k ], \tag {61}
$$
where  $F_{t,h}(x_{t,h}) = [f_{t,h}^{1}(x_{t,h}),\dots ,f_{t,h}^{M}(x_{t,h}))]$ , and  $\phi$  denotes the link-function given in (2).
- Our goal in Algorithm 4 is to compete with the policy  $\pi^{\star}$  defined such that for any  $x\in \mathcal{X}_h$
$$
\pi^ {*} (x) = \operatorname {S e l e c t A c t i o n} \left(F _ {h} ^ {*} (x)\right) \tag {62}
$$
where  $F_{h}^{\star}(x) = [f_{h}^{\star,1}(x), \ldots, f_{h}^{\star,1}(x)] \in \mathbb{R}^{K \times M}$ .
- Given the context  $x_{t,h}$  and the function  $F_{t,h}: \mathcal{X} \mapsto \mathbb{R}^{K \times M}$ , the learner decided whether to query via  $Z_{t,h} = \text{Query}(F_{t,h}(x_{t,h}), \vec{\Delta}_{t,h}(x_{t,h}))$  where we define the function  $\text{Query}: \mathbb{R}^{K \times M} \times \mathbb{R}^M$  as
$$
\operatorname {Q u e r y} (U; \vec {\varepsilon}) := \sup  _ {V \in \mathbb {R} ^ {K \times M}} \mathbf {1} \left\{\text {S e l e c t A c t i o n} (U) \neq \text {S e l e c t A c t i o n} (V) \right\}
$$
$$
\text {s . t .} \| U [ \therefore , m ] - V [ \therefore , m ] \| _ {2} \leq \bar {\varepsilon} [ m ] \quad \forall m \leq M. \tag {63}
$$
At round  $t$ , the learner interactions with transition dynamics  $\{\mathbb{T}_{t,h}\}_{h\leq H}$  and collects data. Without loss of generality, we assume that the learner always starts from the state  $x_{t,1}$ . We next recall the interaction at round  $t$ :
- The learner collects data using the policy  $\pi_t$ , defined such that
$$
\pi_ {t} (x) = \operatorname {S e l e c t A c t i o n} \left(f _ {t, h} \left(x _ {t, h}\right)\right).
$$
for any  $h \leq H$ , and state  $x \in \mathcal{X}_h$ .
- For any policy  $\pi$ , we use the notation  $\tau_t^\pi$  to denote the (counterfactual) trajectory that would have been generated by running  $\pi$  on the deterministic dynamics  $\{\mathbb{T}_{t,h}\}_{h\leq H}$  with the start state  $x_{t,1}$ , i.e.
$$
\tau_ {t} ^ {\pi} = \left\{x _ {t, 1} ^ {\pi}, \pi \left(x _ {t, 1} ^ {\pi}\right), \dots , x _ {t, H} ^ {\pi}, \pi \left(x _ {t, H} ^ {\pi}\right) \right\}, \tag {64}
$$
- For any trajectory  $\tau = \{x_1, a_1, \dots, x_H, a_H\}$ , we define the total return
$$
R (\tau) = \sum_ {h = 1} ^ {H} r \left(x _ {h}, a _ {h}\right). \tag {65}
$$
The goal of the learner is to minimize its regret which is given by
$$
\operatorname {R e g} _ {T} = \sum_ {t = 1} ^ {T} R \left(\tau_ {t} ^ {\pi^ {*}}\right) - \sum_ {t = 1} ^ {T} R \left(\tau_ {t} ^ {\pi_ {t}}\right). \tag {66}
$$
Finally, our bounds depend on the notation of margin defined w.r.t. the function Query defined above. In particular, for a sequence of contexts  $\{\{T_{t,h}\}_{h\leq H}\}_{t\leq T}$ , we define  $T_{\varepsilon,h}$  as
$$
T _ {\varepsilon , h} = \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\text {Q u e r y} \left(F _ {h} ^ {\star} \left(x _ {t, h} ^ {\pi^ {*}}\right), \varepsilon \vec {\mathbb {I}}\right) = 1 \right\}. \tag {67}
$$
In the above we count for the number of time steps when the counterfactual states  $\{x_{t,h}^{\pi^*}\}_{t\leq T}$ , reached under the given dynamics if we had executed  $\pi^*$ , are within the  $\varepsilon$ -margin region. Note that even though the observed states  $\{\{x_{t,h}\}_{h\leq H}\}_{t\leq T}$  are such that  $\sum_{t = 1}^{T}\mathbf{1}\{\mathsf{Query}(F_h^\star (x_{t,h}),\varepsilon \vec{\mathbb{I}}) = 1\}$  is large, we would not pay for this in our margin term  $T_{\varepsilon ,h}$ .
F.5.1 Supporting Technical Results
Lemma 21. With probability at least  $1 - \delta$ , for any  $m \leq M$ , and  $t \leq T$  and  $h \leq H$ , the function  $f^{\star, m}$  satisfies
(a)  $\sum_{s=1}^{t-1} Z_{s,h} \| f_h^{\star,m}(x_{s,h}) - f_{s,h}^m(x_{s,h}) \|^2 \leq \Psi_{\delta}^{\ell_\phi}(\mathcal{F}_h^m, T)$ ,
(b)  $\| f_h^{\star ,m}(x_{t,h}) - f_{t,h}^m (x_{t,h})\| \leq \Delta_{t,h}^m (x_{t,h}),$
where  $\Psi_{\delta}^{\ell_{\phi}}(\mathcal{F}_h^m,T) = \frac{4}{\lambda}\mathrm{Reg}^{\ell_{\phi}}(\mathcal{F}_h^m;T) + \frac{112}{\lambda^2}\log (4MH\log^2 (T) / \delta)$ .
(a) We first note that we do not query oracle when  $Z_{s,h} = 0$ , and thus we can ignore the time steps for which  $Z_{s,h} = 0$ . Hence, for each  $h \in [H]$  and  $m \in [M]$ , applying Lemma 5 yields
$$
\sum_ {s = 1} ^ {t - 1} Z _ {s, h} \left\| f _ {h} ^ {\star , m} \left(x _ {s, h}\right) - f _ {s, h} \left(x _ {s, h}\right) \right\| ^ {2} \leq \frac {4}{\lambda} \operatorname {R e g} ^ {\ell_ {\phi}} \left(\mathcal {F} _ {h} ^ {m}; T\right) + \frac {1 1 2}{\lambda^ {2}} \log \left(4 \log^ {2} (T) / \delta\right)
$$
for all  $t \leq T$ . Then, we take the union bound for all  $h \in [H]$  and  $m \in [M]$ , which completes the proof.
(b) The second part follows from using part-(a) along with the definition in (60).
The next lemma bound the number of times when  $\Delta_{t,h}^{m}(x_{t,h})\geq \zeta$ , and we query. Note that Lemma 22 holds even if the sequence  $\{x_{t,h}\}_{t\leq T}$  was adversarially generated.
Lemma 22. Let  $f^{\star, m}$  satisfy Lemma 21, and let  $\Delta_{t,h}^{m}(x_{t,h})$  be defined in Algorithm 4. Suppose Algorithm 4 is run on the data sequence  $\{x_{t,h}\}_{t \leq 1}$ , and let  $Z_{t,h}$  be defined in line 10. Then, for any  $\zeta > 0$ , with probability at least  $1 - M\delta$ , for any  $m \in [M]$ , and  $h \leq H$ ,
$$
\sum_ {t = 1} ^ {T} Z _ {t, h} \mathbf {1} \bigl \{\Delta_ {t, h} ^ {m} (x _ {t}) \geq \zeta \bigr \} \leq \frac {2 0 \Psi_ {\delta} ^ {\ell_ {\phi}} (\mathcal {F} _ {h} ^ {m} , T)}{\zeta^ {2}} \cdot \mathfrak {E} (\mathcal {F} _ {h} ^ {m}, \zeta / 2; f _ {h} ^ {\star , m}),
$$
where  $\mathfrak{E}$  denotes the eluder dimension given in Definition 1.
Proof. The proof is identical to the proof of Lemma 11 where we handle each  $m \in [M]$  and  $h \in [H]$  separately, and substitute the corresponding bounds for  $f_h^{\star,m}$  via Lemma 21 (instead of using Lemma 10). We skip the proof for conciseness.
F.5.2 Regret Bound
Suppose the trajectories at round  $t$  are generated using the deterministic dynamics  $\{\mathbb{T}_{t,1},\dots ,\mathbb{T}_{t,H}\} = \mathcal{T}(\cdot ;\iota_t)$  where  $\iota_{t}$  denotes the random seed that captures all of the stochasticity at round  $t^{10}$ .
Recall that the policies  $\pi^{\star}$  and  $\pi_t$  such that for any  $h \leq H$  and  $x \in \mathcal{X}_h$ ,  $\pi^{\star}(x) = \text{SelectAction}(F_h^*(x))$ , and  $\pi_t(x) = \text{SelectAction}(F_{t,h}(x))$ . Note that Algorithm 4 collects trajectories using the policy  $\pi_t$  at round  $t$ . Thus, we have
$$
x _ {t, h} ^ {\pi_ {t}} = x _ {t, h}, \tag {68}
$$
where  $x_{t,h}$  denotes the state at time step  $h$  in round  $t$  of Algorithm 4. Finally, let  $\varepsilon > 0$  be a free parameter. We start with the bound on the regret at time  $t$ .
Step 1: Bounding the difference in cumulative return at round  $t$ . Fix any  $t \leq T$ , and let  $\tau_t^{\pi_t}$  and  $\tau_t^{\pi^*}$  denote the trajectories that would have been sampled using the policies  $\pi_t$  and the policy  $\pi^*$  at round  $t$ . Furthermore, define the set  $X_{\varepsilon}$  as
$$
\mathrm {X} _ {\varepsilon} := \bigcup_ {h = 1} ^ {H} \left\{x \in \mathcal {X} _ {h} \mid \operatorname {Q u e r y} \left(F _ {h} ^ {\star} (x), \varepsilon \vec {\Pi}\right) = 1 \right\} \tag {69}
$$
$$
\begin{array}{l} R \left(\tau_ {t} ^ {\pi^ {*}}\right) - R \left(\tau_ {t} ^ {\pi_ {t}}\right) \leq 2 H \sum_ {h = 1} ^ {H} \mathbf {1} \left\{x _ {t, h} ^ {\pi^ {*}} \in \mathrm {X} _ {\varepsilon} \right\} + 2 H \sum_ {h = 1} ^ {H} \mathbf {1} \left\{\pi_ {t} \left(x _ {t, h} ^ {\pi_ {t}}\right) \neq \pi^ {*} \left(x _ {t, h} ^ {\pi_ {t}}\right), x _ {t, h} ^ {\pi_ {t}} \notin \mathrm {X} _ {\varepsilon} \right\} \\ = 2 H \sum_ {h = 1} ^ {H} \mathbf {1} \left\{x _ {t, h} ^ {\pi^ {\star}} \in \mathrm {X} _ {\varepsilon} \right\} + 2 H \sum_ {h = 1} ^ {H} \mathbf {1} \left\{\pi_ {t} \left(x _ {t, h}\right) \neq \pi^ {\star} \left(x _ {t, h}\right), x _ {t, h} \notin \mathrm {X} _ {\varepsilon} \right\} \\ = 2 H \sum_ {h = 1} ^ {H} \mathbf {1} \left\{x _ {t, h} ^ {\pi^ {\star}} \in \mathrm {X} _ {\varepsilon} \right\} + 2 H \sum_ {h = 1} ^ {H} Z _ {t, h} \mathbf {1} \left\{\pi_ {t} \left(x _ {t, h}\right) \neq \pi^ {\star} \left(x _ {t, h}\right), x _ {t, h} \notin \mathrm {X} _ {\varepsilon} \right\} \\ + 2 H \sum_ {h = 1} ^ {H} \bar {Z} _ {t, h} \mathbf {1} \left\{\pi_ {t} \left(x _ {t, h}\right) \neq \pi^ {\star} \left(x _ {t, h}\right), x _ {t, h} \notin \mathrm {X} _ {\varepsilon} \right\} \\ = 2 H \sum_ {h = 1} ^ {H} \mathbf {1} \left\{x _ {t, h} ^ {\pi^ {\star}} \in \mathrm {X} _ {\varepsilon} \right\} + 2 H \mathrm {T} _ {A} + 2 H \mathrm {T} _ {B}, \tag {70} \\ \end{array}
$$
where the second line is obtained by using the relation (68) in the second line. The last line simply defines  $\mathrm{T}_A$  and  $\mathrm{T}_B$  to be the second and the third term in the previous line, respectively, without the  $2H$  multiplicative factor. We bound these two terms separately below:
- Bound on  $\mathrm{T}_A$ . Using the definition of  $\mathbf{X}_{\varepsilon}$  from (69), we note that
$$
\begin{array}{l} \mathrm {T} _ {A} = \sum_ {h = 1} ^ {H} Z _ {t, h} \mathbf {1} \left\{\pi_ {t} \left(x _ {t, h}\right) \neq \pi^ {\star} \left(x _ {t, h}\right), x _ {t, h} \notin \mathrm {X} _ {\varepsilon} \right\} \\ = \sum_ {h = 1} ^ {H} Z _ {t, h} \mathbf {1} \left\{\pi_ {t} \left(x _ {t, h}\right) \neq \pi^ {\star} \left(x _ {t, h}\right), \mathbf {Q u e r y} \left(F _ {h} ^ {\star} \left(x _ {t, h}\right), \varepsilon \vec {\mathbb {I}}\right) = 0 \right\} \\ = \sum_ {h = 1} ^ {H} Z _ {t, h} \mathbf {1} \left\{\exists m \in [ m ]: \| f _ {t, h} ^ {m} \left(x _ {t, h}\right) - f _ {h} ^ {\star , m} \left(x _ {t, h}\right) \| > \varepsilon \right\}, \\ \end{array}
$$
where the last line follows from the fact that the definition of  $\mathbf{Query}$  and the fact that  $\pi_t(x_{t,h})\neq \pi^{\star}(x_{t,h})$  implies that there exists some  $m\in [M]$  for which  $\| f_{t,h}^{m}(x_{t,h}) - f_h^{\star ,m}(x_{t,h})\| >\varepsilon$ . The above implies that
$$
\mathrm {T} _ {A} \leq \sum_ {m = 1} ^ {M} \sum_ {h = 1} ^ {H} Z _ {t, h} \mathbf {1} \left\{\left\| f _ {t, h} ^ {m} \left(x _ {t, h}\right) - f _ {h} ^ {\star , m} \left(x _ {t, h}\right) \right\| > \varepsilon \right\}.
$$
<sup>10</sup>We use random seed  $\iota_t$  to capture all the stochasticity in the choice of  $\{\mathbb{T}_{t,h}\}_{h\leq H,t\leq T}$ . However, all our proofs extend to IL learning with an arbitrary, and possibly adversarial, choice of  $\{\mathbb{T}_{t,h}\}_{h\leq H,t\leq T}$
- Bound on  $\mathrm{T}_B$ . First note that Lemma 21 implies that with probability at least  $1 - \delta$ , for all  $m \leq M$  and  $h \leq H$ ,
$$
\left\| f _ {h} ^ {\star , m} \left(x _ {t, h}\right) - f _ {t, h} ^ {m} \left(x _ {t, h}\right) \right\| \leq \Delta_ {t, h} ^ {m} \left(x _ {t, h}\right). \tag {71}
$$
$$
\begin{array}{l} \mathrm {T} _ {B} \leq \sum_ {h = 1} ^ {H} \bar {Z} _ {t, h} \mathbf {1} \left\{\pi_ {t} \left(x _ {t, h}\right) \neq \pi^ {\star} \left(x _ {t, h}\right) \right\} \tag {72} \\ = \sum_ {h = 1} ^ {H} \mathbf {1} \left\{\text {Q u e r y} \left(F _ {t, h} \left(x _ {t, h}\right), \vec {\Delta} _ {t, h} \left(x _ {t, h}\right)\right) = 0, \pi_ {t} \left(x _ {t, h}\right) \neq \pi^ {\star} \left(x _ {t, h}\right) \right\}, \\ \end{array}
$$
where in the last line follows from plugging in the query condition under which  $Z_{t,h} = 0$ . However note that for any  $h \leq H$  for which  $\mathbf{Query}(F_{t,h}(x_{t,h}), \vec{\Delta}_{t,h}(x_{t,h})) = 0$ , by the definition of Query and the fact that  $\pi_t(x_{t,h}) \neq \pi^\star (x_{t,h})$ , there must exist some  $m \in [M]$  such that
$$
\left\| f _ {h} ^ {x, m} \left(x _ {t, h}\right) - f _ {t, h} ^ {m} \left(x _ {t, h}\right) \right\| > \Delta_ {t, h} ^ {m} \left(x _ {t, h}\right).
$$
However, the above contradicts (71), and thus with probability at least  $1 - \delta$
$$
\mathrm {T} _ {B} = 0. \tag {73}
$$
Plugging the above bounds on  $\mathrm{T}_A$  and  $\mathrm{T}_B$  in (70), we get that
$$
R \left(\tau_ {t} ^ {\pi^ {*}}\right) - R \left(\tau_ {t} ^ {\pi_ {t}}\right) \leq 2 H \sum_ {h = 1} ^ {H} \mathbf {1} \left\{x _ {t, h} ^ {\pi^ {*}} \in \mathrm {X} _ {\varepsilon} \right\} + 2 H \sum_ {m = 1} ^ {M} \sum_ {h = 1} ^ {H} Z _ {t, h} \mathbf {1} \left\{\left\| f _ {t, h} ^ {m} \left(x _ {t, h}\right) - f _ {h} ^ {\star , m} \left(x _ {t, h}\right) \right\| > \varepsilon \right\}. \tag {74}
$$
Step 2: Aggregating over all time steps. Using the bound in (74) for each round  $t$ , we get that
$$
\begin{array}{l} \operatorname {R e g} _ {T} = \sum_ {t = 1} ^ {T} \left(R \left(\tau_ {t} ^ {\pi^ {*}}\right) - R \left(\tau_ {t} ^ {\pi_ {t}}\right)\right) \\ \leq 2 H \sum_ {h = 1} ^ {H} \sum_ {t = 1} ^ {T} \mathbf {1} \left\{x _ {t, h} ^ {\pi^ {\star}} \in \mathrm {X} _ {\varepsilon} \right\} + 2 H \sum_ {t = 1} ^ {T} \sum_ {m = 1} ^ {M} \sum_ {h = 1} ^ {H} Z _ {t, h} \mathbf {1} \left\{\left\| f _ {t, h} ^ {m} \left(x _ {t, h}\right) - f _ {h} ^ {\star , m} \left(x _ {t, h}\right) \right\| > \varepsilon \right\}. \\ \end{array}
$$
Using the fact that  $\mathbf{1}\{a\geq b\} \leq a^2 /b^2$  for any  $a,b\geq 0$ , and the definition of  $T_{\varepsilon ,h}$  in the above, we get that
$$
\begin{array}{l} \mathrm {R e g} _ {T} \leq 2 H \sum_ {h = 1} ^ {H} T _ {\varepsilon , h} + 2 H \sum_ {t = 1} ^ {T} \sum_ {m = 1} ^ {M} \sum_ {h = 1} ^ {H} Z _ {t, h} \frac {\| f _ {t , h} ^ {m} (x _ {t , h}) - f _ {h} ^ {\star , m} (x _ {t , h}) \| ^ {2}}{\varepsilon^ {2}} \\ \leq 2 H \sum_ {h = 1} ^ {H} T _ {\varepsilon , h} + \frac {2 H}{\varepsilon^ {2}} \sum_ {m = 1} ^ {M} \sum_ {h = 1} ^ {H} \Psi_ {\delta} ^ {\ell_ {\phi}} \left(\mathcal {F} _ {h} ^ {m}, T\right). \tag {75} \\ \end{array}
$$
where the last line follows from using the bound in Lemma 21.
Plugging in the form of  $\Psi_{\delta}^{\ell_{\phi}}(\mathcal{F}_h^m,T)$  and ignoring log factors and constants, we get that
$$
\operatorname {R e g} _ {T} \lesssim H \sum_ {h = 1} ^ {H} T _ {\varepsilon , h} + \frac {H}{\lambda \varepsilon^ {2}} \sum_ {m = 1} ^ {M} \sum_ {h = 1} ^ {H} \operatorname {R e g} ^ {\ell_ {\phi}} (\mathcal {F} _ {h} ^ {m}; T) + \frac {M H ^ {2}}{\lambda^ {2} \varepsilon^ {2}} \log (1 / \delta).
$$
Notice that  $\varepsilon$  is a free parameter above so the final bound follows by taking inf over all feasible  $\varepsilon$ .
F.5.3 Total Number of Queries
Fix any  $t \leq T$ , and let  $h_t$  denote the first time step at round  $t$  for which  $Z_{t,h_t} = 1$ , if such a time-step exists (and is set to be  $H + 1$  otherwise). We first observe that for all  $h \leq h_t$ , we have  $\pi^{\star}(x_{t,h}) = \pi_t(x_{t,h})$ . To see this, note that
$$
Z _ {t, h _ {t}} \mathbf {1} \left\{\exists h <   h _ {t}: \pi^ {\star} \left(x _ {t, h}\right) \neq \pi_ {t} \left(x _ {t, h}\right) \right\} \leq \sum_ {h = 1} ^ {h _ {t} - 1} Z _ {t, h _ {t}} \mathbf {1} \left\{\pi^ {\star} \left(x _ {t, h}\right) \neq \pi_ {t} \left(x _ {t, h}\right) \right\}
$$
$$
\begin{array}{l} \leq \sum_ {h = 1} ^ {h _ {t} - 1} Z _ {t, h _ {t}} \bar {Z} _ {t, h} \mathbf {1} \left\{\pi^ {\star} \left(x _ {t, h}\right) \neq \pi_ {t} \left(x _ {t, h}\right) \right\} \\ \leq \sum_ {h = 1} ^ {h _ {t} - 1} \bar {Z} _ {t, h} \mathbf {1} \left\{\pi^ {\star} \left(x _ {t, h}\right) \neq \pi_ {t} \left(x _ {t, h}\right) \right\} \\ \end{array}
$$
where in second inequality above, we used the fact that  $Z_{t,h} = 0$  (and thus  $\bar{Z}_{t,h} = 1$ ) for all  $h < h_t$ , by the definition of  $h_t$ . Observe that the right hand side in the last inequality above is equivalent to the term (72) in the bound on  $\mathbf{T}_B$  above (where sum is now till  $h_t$  instead of  $H$ ). Thus, using the bound in (73), we get that
$$
Z _ {t, h _ {t}} \mathbf {1} \left\{\exists h <   h _ {t}: \pi^ {\star} \left(x _ {t, h}\right) \neq \pi_ {t} \left(x _ {t, h}\right) \right\} = 0,
$$
$$
\pi^ {*} \left(x _ {t, h}\right) = \pi_ {t} \left(x _ {t, h}\right) \quad \text {f o r a l l} h \leq h _ {t}. \tag {76}
$$
Next, let  $\varepsilon > 0$  be a free parameter, and note that plugging in the definition of  $h_t$ , we get that the total number of samples is bounded as:
$$
\begin{array}{l} N _ {T} = \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} Z _ {t, h} \\ \leq H \sum_ {t = 1} ^ {T} Z _ {t, h _ {t}} \\ = H \sum_ {t = 1} ^ {T} Z _ {t, h _ {t}} \mathbf {1} \left\{x _ {t, h _ {t}} \in \mathbb {X} _ {\varepsilon} \right\} + H \sum_ {t = 1} ^ {T} Z _ {t, h _ {t}} \mathbf {1} \left\{x _ {t, h _ {t}} \notin \mathbb {X} _ {\varepsilon} \right\} \\ = H \sum_ {t = 1} ^ {T} Z _ {t, h _ {t}} \mathbf {1} \left\{x _ {t, h _ {t}} \in \mathrm {X} _ {\varepsilon} \right\} + H \sum_ {t = 1} ^ {T} Z _ {t, h _ {t}} \mathbf {1} \left\{x _ {t, h _ {t}} \notin \mathrm {X} _ {\varepsilon}, \| \Delta_ {t, h _ {t}} ^ {m} \left(x _ {t, h _ {t}}\right) \| _ {\infty} \leq \frac {\varepsilon}{4} \right\} \\ + H \sum_ {t = 1} ^ {T} Z _ {t, h _ {t}} \mathbf {1} \left\{x _ {t, h _ {t}} \notin \mathrm {X} _ {\varepsilon}, \| \Delta_ {t, h _ {t}} ^ {m} \left(x _ {t, h _ {t}}\right) \| _ {\infty} > \frac {\varepsilon}{4} \right\} \\ = \mathrm {T} _ {C} + \mathrm {T} _ {D} + \mathrm {T} _ {E}, \\ \end{array}
$$
where  $\mathrm{T}_C$ ,  $\mathrm{T}_D$  and  $\mathrm{T}_E$  are the first, second and the third term respectively in the previous line. We bound them separately below.
- Bound on  $\mathbb{T}_C$ . Fix any  $t \leq T$ . Using the relation in (76), note that  $\pi^{\star}(x_{t,h}) = \pi_t(x_{t,h})$  for all  $h < h_t$ . Thus, the corresponding trajectories would be identical till time step  $h_t$ , which implies that  $x_{t,h_t} = x_{t,h_t}^{\pi^{\star}}$ . Using this property in the  $\mathbb{T}_C$ , we get that
$$
\begin{array}{l} \left(\mathsf {T} _ {C}\right) _ {t} = H \sum_ {t = 1} ^ {T} Z _ {t, h _ {t}} \mathbf {1} \left\{x _ {t, h _ {t}} \in \mathbb {X} _ {\varepsilon} \right\} \\ = H \sum_ {t = 1} ^ {T} Z _ {t, h _ {t}} \mathbf {1} \left\{x _ {t, h _ {t}} ^ {\pi^ {*}} \in \mathrm {X} _ {\varepsilon} \right\} \\ \leq H \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} Z _ {t, h} \mathbf {1} \left\{x _ {t, h} ^ {\pi^ {*}} \in \mathrm {X} _ {\varepsilon} \right\} \\ = H \sum_ {h = 1} ^ {H} T _ {\varepsilon , h}, \\ \end{array}
$$
where the last line plugs in the definition of  $T_{\varepsilon,h}$ .
- Bound on  $\mathrm{T}_D$ . First note that
$$
\left(\mathsf {T} _ {D}\right) _ {t} = H \mathbf {1} \{\text {Q u e r y} \big (F _ {t, h _ {t}} \left(x _ {t, h _ {t}}\right), \vec {\Delta} _ {t, h _ {t}} \left(x _ {t, h _ {t}}\right) \big) = 1, \text {Q u e r y} \big (F ^ {\star} \left(x _ {t, h _ {t}}\right), \varepsilon \vec {\mathbb {I}} \big) = 0, \sup  _ {m \in [ M ]} \Delta_ {t} ^ {m} \left(x _ {t, h _ {t}}\right) \leq \varepsilon / 4 \}.
$$
In the following, we will show that all the conditions in the above indicator can not hold simultaneously. First note that since  $\mathsf{Query}\big(F_{t,h_t}(x_{t,h_t}),\vec{\Delta}_{t,h_t}(x_{t,h_t})\big) = 1$ , there exists an  $\widetilde{F}$  such that
$$
\operatorname {S e l e c t A c t i o n} \left(\widetilde {F} \left(x _ {t, h _ {t}}\right)\right) \neq \operatorname {S e l e c t A c t i o n} \left(F _ {t, h _ {t}} \left(x _ {t, h _ {t}}\right)\right) \tag {77}
$$
$$
\forall m \in [ M ]: \quad \| \widetilde {F} \left(x _ {t, h _ {t}}\right) [:, m ] - F _ {t, h _ {t}} \left(x _ {t, h _ {t}}\right) [:, m ] \| \leq \Delta_ {t, h _ {t}} ^ {m} \left(x _ {t, h _ {t}}\right). \tag {78}
$$
On the other hand, recall that Lemma 21 implies that
$$
\forall m \in [ M ]: \quad \| F ^ {\star} \left(x _ {t, h _ {t}}\right) [:, m ] - F _ {t, h _ {t}} \left(x _ {t, h _ {t}}\right) [:, m ] \| \leq \Delta_ {t, h _ {t}} ^ {m} \left(x _ {t, h _ {t}}\right). \tag {79}
$$
Since,  $\sup_{m}\Delta_{t,h_t}^m (x_{t,h_t})\leq \varepsilon /4$  , an application of Triangle inequality along with the bounds (78) and (79) imply that
$$
\forall m \in [ M ]: \quad \| F ^ {\star} \left(x _ {t, h _ {t}}\right) [:, m ] - \widetilde {F} \left(x _ {t, h _ {t}}\right) [:, m ] \| \leq 2 \Delta_ {t, h _ {t}} ^ {m} \left(x _ {t, h _ {t}}\right) <   \varepsilon . \tag {80}
$$
But the above contradicts the fact that  $\mathsf{Query}(F^{\star}(x_{t,h_t}),\varepsilon \vec{\mathbb{1}}) = 0$  since both  $\widetilde{F}$  and  $F_{t}$  satisfy the norm constraints in the definition of  $\mathsf{Query}$ , but we can not simultaneously have that
SelectAction  $\left(F^{\star}\left(x_{t,h_t}\right)\right) =$  SelectAction  $\left(F_{t,h_t}\left(x_{t,h_t}\right)\right) =$  SelectAction  $\left(\widetilde{F}\left(x_{t,h_t}\right)\right)$
due to (77). Thus, we must have that
$$
\left(\mathrm {T} _ {D}\right) _ {t} = 0.
$$
$$
\begin{array}{l} \mathsf {T} _ {E} \leq H \sum_ {t = 1} ^ {T} Z _ {t, h _ {t}} \mathbf {1} \{\| \vec {\Delta} _ {t, h _ {t}} (x _ {t, h _ {t}}) \| _ {\infty} > \varepsilon / 4 \} \\ = H \sum_ {t = 1} ^ {T} Z _ {t, h _ {t}} {\bf 1} \{\exists m \in [ M ]: \Delta_ {t, h _ {t}} ^ {m} (x _ {t, h _ {t}}) > \varepsilon / 4 \} \\ \leq H \sum_ {m = 1} ^ {M} \sum_ {t = 1} ^ {T} Z _ {t, h _ {t}} {\bf 1} \{\Delta_ {t, h _ {t}} ^ {m} (x _ {t, h _ {t}}) > \varepsilon / 4 \} \\ \leq H \sum_ {h = 1} ^ {H} \sum_ {m = 1} ^ {M} \sum_ {t = 1} ^ {T} Z _ {t, h} {\bf 1} \{\Delta_ {t, h} ^ {m} (x _ {t, h}) > \varepsilon / 4 \}, \\ \end{array}
$$
where the last line simply upper bound the term for  $h_t$  by the corresponding terms for all  $h \leq H$ .
Using Lemma 22 to bound the term in the right hand side for each  $m \in [M]$  and  $h \leq H$ , we get that