SlowGuess/NeurIPS_2023_OCR huggingface dataset:
such that  $\bar{\mu}_e(x) = \mu (x)$  whenever  $Z_{e}(x) = 1$  and is 0 otherwise. Furthermore, for any  $e\in E - 1$ , we define the sub-probability measure  $\bar{\nu}_{e}$  as
$$
\bar {\nu} _ {e} = \frac {1}{\tau_ {e} - \tau_ {1}} \sum_ {\bar {e} = 1} ^ {e - 1} \left(\tau_ {\bar {e} + 1} - \tau_ {\bar {e}}\right) \bar {\mu} _ {\bar {e}}. \tag {33}
$$
We now move to the main proof. First fix any epoch  $e \leq E - 1$ , and consider any round  $t \in [\tau_e, \tau_{e + 1} - 1]$ . Using the definition of  $\Delta_e(x_t)$  and definition of  $Z^e$  from (29) in the above, we get that
$$
\begin{array}{l} \mathbb {E} _ {x _ {t} \sim \mu} \left[ Z _ {t} \mathbf {1} \left\{\Delta_ {e} (x _ {t}) > \zeta \right\} \right] = \mathbb {E} _ {x \sim \mu} \left[ Z ^ {e} (x _ {t}) \mathbf {1} \left\{\sup  _ {f, f ^ {\prime} \in \mathcal {F} _ {e}} \| f (x _ {t}) - f ^ {\prime} (x _ {t}) \| > \zeta \right\} \right] \\ \leq \mathbb {E} _ {x \sim \mu} \left[ Z ^ {e} (x _ {t}) \mathbf {1} \left\{\sup  _ {f \in \mathcal {F} _ {e}} \| f (x _ {t}) - f ^ {\star} (x _ {t}) \| > \frac {\zeta}{2} \right\} \right] \\ \end{array}
$$
where the second line follows because  $f^{\star} \in \mathcal{F}_e$ , and because  $\sup_{f, f' \in \mathcal{F}_e} \| f(x_t) - f'(x_t) \| \leq 2 \sup_{f \in \mathcal{F}_e} \| f(x_t) - f^*(x_t) \|$  due to Triangle inequality. Plugging in the definition of  $\bar{\mu}_e$  from (32) in the above we get that
$$
\begin{array}{l} \mathbb {E} _ {x _ {t} \sim \mu} \left[ Z _ {t} \mathbf {1} \left\{\Delta_ {e} (x _ {t}) > \zeta \right\} \right] \leq \mathbb {E} _ {x \sim \bar {\mu} _ {e}} \left[ \mathbf {1} \left\{\sup  _ {f \in \mathcal {F} _ {e}} \| f (x _ {t}) - f ^ {\star} (x _ {t}) \| > \frac {\zeta}{2} \right\} \right] \\ \leq \mathbb {E} _ {x \sim \bar {\mu} _ {\bar {\varepsilon}}} \left[ \mathbf {1} \left\{\sup  _ {f \in \mathcal {F} _ {e}} \| f (x _ {t}) - f ^ {\star} (x _ {t}) \| > \frac {\zeta}{2} \right\} \right]. \tag {34} \\ \end{array}
$$
for all  $\bar{e} \leq e$ , where the last inequality follows from Lemma 14-(e). Since the above holds for all  $\bar{e} \leq e$ , we immediately get that
$$
\begin{array}{l} \mathbb {E} _ {x _ {t} \sim \mu} \left[ Z _ {t} \mathbf {1} \left\{\Delta_ {e} (x _ {t}) > \zeta \right\} \right] \leq \mathbb {E} _ {x \sim \bar {\nu} _ {\bar {\epsilon}}} \left[ \mathbf {1} \left\{\sup  _ {f \in \mathcal {F} _ {e}} \| f (x _ {t}) - f ^ {\star} (x _ {t}) \| > \frac {\zeta}{2} \right\} \right] \\ = \mathbb {E} _ {x \sim \bar {\nu} _ {e}} \left[ \mathbf {1} \left\{\exists f \in \mathcal {F} _ {e}: \| f (x) - f ^ {\star} (x) \| > \frac {\zeta}{2} \right\} \right], \tag {35} \\ \end{array}
$$
where the sub-probability measure  $\bar{\nu}_{\bar{e}}$  is defined in (33). Additionally, recall that Lemma 14-(b) implies that with probability at least  $1 - \delta$  any  $f \in \mathcal{F}_e$  satisfies
$$
\left\| f \left(x _ {s}\right) - f ^ {\star} \left(x _ {s}\right) \right\| _ {\bar {\nu} _ {e}} \leq \sqrt {\frac {\widehat {\Psi} _ {\delta} ^ {\ell_ {\phi}} (\mathcal {F} ; T)}{\tau_ {e} - 1}}. \tag {36}
$$
Conditioning on the above event, and plugging it in (35), we get that
$$
\begin{array}{l} \mathbb {E} _ {x _ {t} \sim \mu} [ Z _ {t} \mathbf {1} \{\Delta_ {e} (x _ {t}) > \zeta \} ] \\ \leq \mathbb {E} _ {x \sim \bar {\nu} _ {\bar {e}}} \left[ \mathbf {1} \left\{\exists f \in \mathcal {F} _ {e}: \| f (x) - f ^ {\star} (x) \| > \frac {\zeta}{2}, \| f (x _ {s}) - f ^ {\star} (x _ {s}) \| _ {\bar {\nu} _ {e}} \leq \sqrt {\frac {\widehat {\Psi} _ {\delta} ^ {\ell_ {\phi}} (\mathcal {F} ; T)}{\tau_ {e} - 1}} \right\} \right] \\ \end{array}
$$
$$
\leq 4 \cdot \frac {\widehat {\Psi} _ {\delta} ^ {\ell_ {\phi}} (\mathcal {F} ; T)}{(\tau_ {e} - 1) \zeta^ {2}} \cdot \theta^ {\text {v a l}} \left(\mathcal {F}, \frac {\zeta}{2}, \frac {\widehat {\Psi} _ {\delta} ^ {\ell_ {\phi}} (\mathcal {F} ; T)}{\tau_ {e} - 1}; f ^ {\star}\right), \tag {37}
$$
where the last inequality uses Lemma 15.
Summing up the bound in (37) for each term  $t = 1$  to  $T$ , we get that
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \mathbb {E} _ {x _ {t}} \left[ Z _ {t} \mathbf {1} \left\{\Delta_ {e} (x _ {t}) > \zeta \right\} \right] = \sum_ {e = 1} ^ {E - 1} \sum_ {t = \tau_ {e}} ^ {\tau_ {e + 1} - 1} \mathbb {E} _ {x _ {t}} \left[ Z _ {t} \mathbf {1} \left\{\Delta_ {e} (x _ {t}) > \zeta \right\} \right] \\ \leq 4 \sum_ {e = 1} ^ {E - 1} \left(\tau_ {e + 1} - \tau_ {e}\right) \frac {\widehat {\Psi} _ {\delta} ^ {\ell_ {\phi}} (\mathcal {F} ; T)}{(\tau_ {e} - 1) \zeta^ {2}} \cdot \theta^ {\mathrm {v a l}} \left(\mathcal {F}, \frac {\zeta}{2}, \frac {\widehat {\Psi} _ {\delta} ^ {\ell_ {\phi}} (\mathcal {F} ; T)}{\tau_ {e} - 1}; f ^ {\star}\right) \\ \leq 8 \sum_ {e = 1} ^ {E - 1} \frac {\widehat {\Psi} _ {\delta} ^ {\ell_ {\phi}} (\mathcal {F} ; T)}{\zeta^ {2}} \cdot \theta^ {\text {v a l}} \left(\mathcal {F}, \frac {\zeta}{2}, \frac {\widehat {\Psi} _ {\delta} ^ {\ell_ {\phi}} (\mathcal {F} ; T)}{\tau_ {e} - 1}; f ^ {\star}\right) \\ \leq 8 \sum_ {e = 1} ^ {E - 1} \frac {\widehat {\Psi} _ {\delta} ^ {\ell_ {\phi}} (\mathcal {F} ; T)}{\zeta^ {2}} \cdot \theta^ {\text {v a l}} \left(\mathcal {F}, \frac {\zeta}{2}, \frac {\widehat {\Psi} _ {\delta} ^ {\ell_ {\phi}} (\mathcal {F} ; T)}{T}; f ^ {\star}\right) \\ \leq 8 \log (T) \cdot \frac {\widehat {\Psi} _ {\delta} ^ {\ell_ {\phi}} (\mathcal {F} ; T)}{\zeta^ {2}} \cdot \theta^ {\mathrm {v a l}} \left(\mathcal {F}, \frac {\zeta}{2}, \frac {\widehat {\Psi} _ {\delta} ^ {\ell_ {\phi}} (\mathcal {F} ; T)}{T}; f ^ {\star}\right) \\ \end{array}
$$
where the second inequality uses the fact that  $\tau_{e + 1} = 2\tau_{e}$  and that  $\tau_{1} = 1$ , the third inequality holds due to monotonicity of  $\theta^{\mathrm{val}}\big(\mathcal{F},\frac{\zeta}{2},\ddots ;f^{\star}\big)$  and the last line simply plugs in the value of  $E = \log (T)$ .
Using Lemma 4 with the above bound for the sequence of random variable  $X_{t} = Z_{t}\mathbf{1}\{\Delta_{e}(x_{t}) > \zeta \}$ , we get that with probability at least  $1 - \delta$ ,
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} Z _ {t} \mathbf {1} \{\Delta_ {e} (x _ {t}) > \zeta \} \leq \frac {3}{2} \sum_ {t = 1} ^ {T} \mathbb {E} _ {x _ {t}} [ Z _ {t} \mathbf {1} \{\Delta_ {e} (x _ {t}) > \zeta \} ] + 4 \log (2 / \delta) \\ \leq 1 2 \log (T) \cdot \frac {\widehat {\Psi} _ {\delta} ^ {\ell_ {\phi}} (\mathcal {F} ; T)}{\zeta^ {2}} \cdot \theta^ {\mathrm {v a l}} \bigg (\mathcal {F}, \frac {\zeta}{2}, \frac {\widehat {\Psi} _ {\delta} ^ {\ell_ {\phi}} (\mathcal {F} ; T)}{T}; f ^ {\star} \bigg) + 4 \log (2 / \delta). \\ \end{array}
$$
The final result follows by taking a union bound of the above and the event in (36).
E.4.2 Regret Bound
For the ease of notation, through the proofs in this section we define the operators  $\pi^{\star}$  as
$$
\pi^{\star}(x) = \operatorname *{argmax}_{k}\phi (f^{\star}(x))[k].
$$
Furthermore, recall that  $\widehat{y}_t$  denotes the action chosen by the learner at round  $t$  of interaction. Starting from the definition of the regret, we get that
$$
\begin{array}{l} \operatorname {R e g} _ {T} = \sum_ {t = 1} ^ {T} \Pr \left(\widehat {y} _ {t} \neq y _ {t}\right) - \Pr \left(\pi^ {\star} \left(x _ {t}\right) \neq y _ {t}\right) \\ = \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\widehat {y _ {t}} \neq \pi^ {\star} (x _ {t}) \right\} \cdot \left| \Pr \left(y _ {t} = \pi^ {\star} (x _ {t})\right) - \Pr \left(y _ {t} = \widehat {y _ {t}}\right) \right| \\ = \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\widehat {y} _ {t} \neq \pi^ {\star} \left(x _ {t}\right) \right\} \cdot | \phi \left(f ^ {\star} \left(x _ {t}\right)\right) \left[ \pi^ {\star} \left(x _ {t}\right) \right] - \phi \left(f ^ {\star} \left(x _ {t}\right)\right) [ \widehat {y} _ {t} ] | \\ \leq \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\widehat {y} _ {t} \neq \pi^ {\star} \left(x _ {t}\right) \right\} \cdot \operatorname {G a p} \left(f ^ {\star} \left(x _ {t}\right), \widehat {y} _ {t}\right), \\ \end{array}
$$
where the last inequality plugs in the definition of Gap from (45). Let  $\varepsilon > 0$  be a free parameter. We can decompose the above regret bound further as:
$$
\operatorname {R e g} _ {T} \leq \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\widehat {y} _ {t} \neq \pi^ {\star} \left(x _ {t}\right), \operatorname {G a p} \left(f ^ {\star} \left(x _ {t}\right), \widehat {y} _ {t}\right) \leq \varepsilon \right\} \cdot \operatorname {G a p} \left(f ^ {\star} \left(x _ {t}\right), \widehat {y} _ {t}\right)
$$
$$
+ \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\widehat {y} _ {t} \neq \pi^ {\star} \left(x _ {t}\right), \operatorname {G a p} \left(f ^ {\star} \left(x _ {t}\right), \widehat {y} _ {t}\right) > \varepsilon \right\} \cdot \operatorname {G a p} \left(f ^ {\star} \left(x _ {t}\right), \widehat {y} _ {t}\right)
$$
Using the fact that  $y_{t}(x_{t}) = \operatorname{argmax}_{k\in [K]}\phi (f_{t}(x))[k]$  in the above along with the definition of Gap and Lemma 12, we get that
$$
\begin{array}{l} \operatorname {R e g} _ {T} \leq \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\widehat {y} _ {t} \neq \pi^ {\star} \left(x _ {t}\right), \operatorname {G a p} \left(f ^ {\star} \left(x _ {t}\right), \widehat {y} _ {t}\right) \leq \varepsilon \right\} \cdot \varepsilon \\ + 2 \gamma \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\widehat {y} _ {t} \neq \pi^ {\star} \left(x _ {t}\right), \operatorname {G a p} \left(f ^ {\star} \left(x _ {t}\right), \widehat {y} _ {t}\right) > \varepsilon \right\} \cdot \| f ^ {\star} (x _ {t}) - f _ {t} (x _ {t}) \| \\ \leq \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\operatorname {M a r g i n} \left(f ^ {\star} \left(x _ {t}\right)\right) \leq \varepsilon \right\} \cdot \varepsilon \\ + 2 \gamma \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\widehat {y} _ {t} \neq \pi^ {\star} \left(x _ {t}\right), \operatorname {G a p} \left(f ^ {\star} \left(x _ {t}\right), \widehat {y} _ {t}\right) > \varepsilon \right\} \cdot \| f ^ {\star} (x _ {t}) - f _ {t} (x _ {t}) \| \\ \leq \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\operatorname {M a r g i n} \left(f ^ {\star} \left(x _ {t}\right)\right) \leq \varepsilon \right\} \cdot \varepsilon \\ + 2 \gamma \sum_ {t = 1} ^ {T} Z _ {t} \mathbf {1} \left\{\widehat {y} _ {t} \neq \pi^ {\star} \left(x _ {t}\right), \operatorname {G a p} \left(f ^ {\star} \left(x _ {t}\right), \widehat {y} _ {t}\right) > \varepsilon \right\} \cdot \| f ^ {\star} \left(x _ {t}\right) - f _ {t} \left(x _ {t}\right) \| \\ + 2 \gamma \sum_ {t = 1} ^ {T} \bar {Z} _ {t} \mathbf {1} \left\{\widehat {y} _ {t} \neq \pi^ {\star} \left(x _ {t}\right) \right\} \cdot \| f ^ {\star} \left(x _ {t}\right) - f _ {t} \left(x _ {t}\right) \| \\ = T _ {\varepsilon} \cdot \varepsilon + 2 \gamma \cdot T _ {A} + 2 \gamma \cdot T _ {B} \cdot \| f ^ {\star} (x _ {t}) - f _ {t} (x _ {t}) \|, \\ \end{array}
$$
where the second inequality holds because  $\mathrm{Gap}(f^{\star}(x_t),\widehat{y}_t)\leq \varepsilon$  implies that  $\mathrm{Margin}(f^{\star}(x_t))\leq \varepsilon$  whenever  $\widehat{y}_t\neq \pi^{\star}(x_t)$ . In the last line we plugged in the definition of  $T_{\varepsilon}$ , and defined  $\mathrm{T}_A$  and  $\mathrm{T}_B$  as the second term and the last term respectively. We bound term separately below:
- Bound on  $\mathrm{T}_A$ : Note that whenever  $Z_{t} = 1$ , we choose  $\widehat{y_t} = \arg \max_k\phi (f_t(x_t))[k]$ . Thus,
$$
\begin{array}{l} \mathrm {T} _ {A} = \sum_ {t = 1} ^ {T} Z _ {t} \mathbf {1} \left\{\widehat {y} _ {t} \neq \pi^ {\star} (x _ {t}), \operatorname {G a p} \left(f ^ {\star} (x _ {t}), \widehat {y} _ {t}\right) > \varepsilon \right\} \cdot \| f ^ {\star} (x _ {t}) - f _ {t} (x _ {t}) \| \\ \leq \sum_ {t = 1} ^ {T} Z _ {t} \mathbf {1} \left\{\left\| f ^ {\star} \left(x _ {t}\right) - f _ {t} \left(x _ {t}\right) \right\| > \varepsilon / 2 \gamma \right\} \cdot \left\| f ^ {\star} \left(x _ {t}\right) - f _ {t} \left(x _ {t}\right) \right\| \\ \end{array}
$$
where the second line follows from Lemma 12 and because  $\widehat{y_t} \neq \pi^\star(x_t)$ . Using the fact that  $\mathbf{1}\{a \geq b\} \leq a / b$  for all  $a, b \geq 0$ , we get that
$$
\mathrm {T} _ {A} \leq 4 \gamma \sum_ {t = 1} ^ {T} Z _ {t} \frac {\left\| f ^ {\star} \left(x _ {t}\right) - f _ {t} \left(x _ {t}\right) \right\| ^ {2}}{\varepsilon}. \tag {38}
$$
- Bound on  $\mathrm{T}_B$ : Fix any  $t \leq T$ , and let  $e$  be such that  $\tau_e \leq t < \tau_{e+1}$ . Next, note that from Lemma 14, we have
$$
\left\| f ^ {\star} \left(x _ {t}\right) - g _ {t} \left(x _ {t}\right) \right\| \leq \Delta_ {e} \left(x _ {t}\right). \tag {39}
$$
$$
\begin{array}{l} \mathrm {T} _ {B} = \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\operatorname {M a r g i n} \left(g _ {t} \left(x _ {t}\right)\right) > 2 \gamma \Delta_ {e} \left(x _ {t}\right), \widehat {y} _ {t} \neq \pi^ {\star} \left(x _ {t}\right) \right\} \\ \leq \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\left\| g _ {t} \left(x _ {t}\right) - f ^ {\star} \left(x _ {t}\right) \right\| > \Delta_ {e} \left(x _ {t}\right), \operatorname {M a r g i n} \left(f ^ {\star} \left(x _ {t}\right)\right) > \varepsilon \right\}, \\ \end{array}
$$
where the second inequality is due Lemma 12 and by noting that  $\widehat{y}_t \neq \pi^\star(x_t)$  and that when  $Z_t = 0$ , we choose  $\widehat{y}_t = \operatorname{argmax}_k \phi(g_t(x_t))[k]$ . However, note that the term inside the indicator contradicts (39) (which always holds). Thus,
$$
\mathrm {T} _ {B} = 0. \tag {40}
$$
Combining the bounds (38) and (40), we get that:
$$
\begin{array}{l} \operatorname {R e g} _ {T} \leq \varepsilon T _ {\varepsilon} + 8 \gamma^ {2} \sum_ {t = 1} ^ {T} Z _ {t} \frac {\left\| f _ {t} \left(x _ {t}\right) - f ^ {\star} \left(x _ {t}\right) \right\| ^ {2}}{\varepsilon} \\ \leq \varepsilon T _ {\varepsilon} + \frac {8 \gamma^ {2}}{\varepsilon} \Psi_ {\delta} ^ {\ell_ {\phi}} (\mathcal {F}, T), \\ \end{array}
$$
where the last inequality is due to Lemma 14.
E.4.3 Total Number of Queries
Let  $N_{T}$  denote the total number of expert queries made by the learner within  $T$  rounds of interactions. For the ease of notation, define  $\Delta_t(x_t) = \Delta_{e(t)}(x_t)$  where  $e(t)$  denotes the epoch number for which  $\tau_{e(t)} \leq t < \tau_{e(t) + 1}$ . Additionally, let  $\varepsilon > 0$  be a free parameter. Thus,
$$
\begin{array}{l} N _ {T} = \sum_ {t = 1} ^ {T} Z _ {t} \\ = \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\operatorname {M a r g i n} \left(g _ {t} \left(x _ {t}\right)\right) \leq 2 \gamma \Delta_ {t} \left(x _ {t}\right) \right\} \\ = \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\operatorname {M a r g i n} \left(g _ {t} \left(x _ {t}\right)\right) \leq 2 \gamma \Delta_ {t} \left(x _ {t}\right), \operatorname {M a r g i n} \left(f ^ {\star} \left(x _ {t}\right)\right) \leq \varepsilon \right\} \\ + \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\operatorname {M a r g i n} \left(g _ {t} \left(x _ {t}\right)\right) \leq 2 \gamma \Delta_ {t} \left(x _ {t}\right), \operatorname {M a r g i n} \left(f ^ {\star} \left(x _ {t}\right)\right) > \varepsilon \right\} \\ \leq \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\operatorname {M a r g i n} \left(f ^ {\star} \left(x _ {t}\right)\right) \leq \varepsilon \right\} \\ + \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\operatorname {M a r g i n} \left(g _ {t} \left(x _ {t}\right)\right) \leq 2 \gamma \Delta_ {t} \left(x _ {t}\right), \operatorname {M a r g i n} \left(f ^ {\star} \left(x _ {t}\right)\right) > \varepsilon , \Delta_ {t} \left(x _ {t}\right) \leq^ {\varepsilon} / 4 \gamma \right\} \\ + \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\operatorname {M a r g i n} \left(g _ {t} \left(x _ {t}\right)\right) \leq 2 \gamma \Delta_ {t} \left(x _ {t}\right), \operatorname {M a r g i n} \left(f ^ {\star} \left(x _ {t}\right)\right) > \varepsilon , \Delta_ {t} \left(x _ {t}\right) > \varepsilon / 4 \gamma \right\} \\ = T _ {\varepsilon} + \mathrm {T} _ {D} + \mathrm {T} _ {E}, \\ \end{array}
$$
where in the last line we used the definition of  $T_{\varepsilon}$  and defined  $\mathrm{T}_D$  and  $\mathrm{T}_E$  respectively, which we bound separately below.
- Bound on  $\mathrm{T}_D$ . From Lemma 14 recall that  $\| f^{\star}(x_t) - g_t(x_t)\| \leq \Delta_e(x_t)$ . Thus, for any  $x_{t}$  for which  $\| g_t(x_t)\| \leq \Delta_e(x_t)$ , Lemma 13 implies that
$$
\operatorname {M a r g i n} \left(f ^ {\star} \left(x _ {t}\right)\right) \leq 2 \gamma \left\| f _ {t} \left(x _ {t}\right) - f ^ {\star} \left(x _ {t}\right) \right\| + \operatorname {M a r g i n} \left(g _ {t} \left(x _ {t}\right)\right) \leq 2 \gamma \Delta_ {t} \left(x _ {t}\right) + \operatorname {M a r g i n} \left(f _ {t} \left(x _ {t}\right)\right).
$$
$$
\begin{array}{l} \mathrm {T} _ {D} = \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\operatorname {M a r g i n} \left(g _ {t} \left(x _ {t}\right)\right) \leq 2 \gamma \Delta_ {t} \left(x _ {t}\right), \operatorname {M a r g i n} \left(f ^ {\star} \left(x _ {t}\right)\right) > \varepsilon , \Delta_ {t} \left(x _ {t}\right) \leq^ {\varepsilon} / 4 \gamma \right\} \\ \leq \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\operatorname {M a r g i n} \left(f ^ {\star} \left(x _ {t}\right)\right) \leq 4 \gamma \Delta_ {t} \left(x _ {t}\right), \operatorname {M a r g i n} \left(f ^ {\star} \left(x _ {t}\right)\right) > \varepsilon , \Delta_ {t} \left(x _ {t}\right) \leq^ {\varepsilon / 4 \gamma} \right\} \\ \leq 0, \\ \end{array}
$$
where the last line follows from the fact that all the conditions inside the indictor can not hold simultaneously for any  $\varepsilon > 0$ .
- Bound on  $\mathrm{T}_{E}$ . We note that
$$
\mathrm {T} _ {E} = \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\operatorname {M a r g i n} \left(g _ {t} \left(x _ {t}\right)\right) \leq 2 \gamma \Delta_ {t} \left(x _ {t}\right), \operatorname {M a r g i n} \left(f ^ {\star} \left(x _ {t}\right)\right) > \varepsilon , \Delta_ {t} \left(x _ {t}\right) > \varepsilon / 4 \gamma \right\}
$$
$$
\begin{array}{l} \leq \sum_ {t = 1} ^ {T} Z _ {t} \mathbf {1} \left\{\Delta_ {t} (x _ {t}) \geq \varepsilon / 4 \gamma \right\} \\ \leq \sum_ {t = 1} ^ {T} Z _ {t} \mathbf {1} \bigl \{\Delta_ {e (t)} (x _ {t}) \geq^ {\varepsilon} / 4 \gamma \bigr \}. \\ \end{array}
$$
Using Lemma 16 with  $\zeta = \varepsilon /4\gamma$  to bound the term on the right hand side above, we get that with probability at least  $1 - 2\delta$
$$
\mathrm {T} _ {E} \leq O \bigg (\log (T) \gamma^ {2} \cdot \frac {\widehat {\Psi} _ {\delta} ^ {\ell_ {\phi}} (\mathcal {F} ; T)}{\varepsilon^ {2}} \cdot \theta^ {\mathrm {v a l}} \bigg (\mathcal {F}, \frac {\varepsilon}{8 \gamma}, \frac {\widehat {\Psi} _ {\delta} ^ {\ell_ {\phi}} (\mathcal {F} ; T)}{T}; f ^ {\star} \bigg) + \log (2 / \delta) \bigg).
$$
$$
N _ {T} \leq T _ {\varepsilon} + O \bigg (\log (T) \gamma^ {2} \cdot \frac {\widehat {\Psi} _ {\delta} ^ {\ell_ {\phi}} (\mathcal {F} ; T)}{\varepsilon^ {2}} \cdot \theta^ {\mathrm {v a l}} \bigg (\mathcal {F}, \frac {\varepsilon}{8 \gamma}, \frac {\widehat {\Psi} _ {\delta} ^ {\ell_ {\phi}} (\mathcal {F} ; T)}{T}; f ^ {\star} \bigg) + \log (2 / \delta) \bigg).
$$
Since  $\varepsilon$  is a free parameter above, the final bound follows by choosing the best parameter  $\varepsilon$ , and by plugging in the form of  $\widehat{\Psi}_{\delta}^{\ell_{\phi}}(\mathcal{F};T)$ .
E.5 Proof of Corollary 1
Note that the Tsybakov noise condition implies that there exists constants  $c, \rho \geq 0$  such that for all  $\varepsilon \in (0,1)$ :
$$
\Pr_ {x \sim \mu} (\operatorname {M a r g i n} (f ^ {\star} (x _ {t})) \leq \varepsilon) \leq c \varepsilon^ {\rho}.
$$
Thus, using Lemma 4, we get that
$$
\begin{array}{l} T _ {\varepsilon} = \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\operatorname {M a r g i n} \left(f ^ {\star} \left(x _ {t}\right)\right) \leq \varepsilon \right\} \\ \leq \frac {3 T}{2} \Pr_ {x \sim \mu} \left(\operatorname {M a r g i n} \left(f ^ {\star} (x)\right) \leq \varepsilon\right) + 4 \log (2 / \delta) \\ \leq 2 c T \varepsilon^ {\rho} + 4 \log (2 / \delta). \\ \end{array}
$$
Using the above in the bound for Theorem 2, we get that for any  $\varepsilon >0$
$$
\mathrm {R e g} _ {T} \lesssim c T \varepsilon^ {\rho + 1} + \frac {\gamma^ {2}}{\lambda \varepsilon} \mathrm {R e g} ^ {\ell_ {\phi}} (\mathcal {F}; T) + \log (1 / \delta),
$$
Setting  $\varepsilon = \left(\frac{\gamma^2}{\lambda CT}\mathrm{Reg}^{\ell_{\phi}}(\mathcal{F};T)\right)^{\frac{1}{\rho + 2}}$  in the above implies that
$$
\operatorname {R e g} _ {T} \lesssim \left(\frac {\gamma^ {2}}{\lambda} c ^ {\frac {1}{\rho + 1}}\right) ^ {\frac {\rho + 1}{\rho + 2}} \left(\operatorname {R e g} ^ {\ell_ {\phi}} (\mathcal {F}; T)\right) ^ {\frac {\rho + 1}{\rho + 2}} \cdot (T) ^ {\frac {1}{\rho + 2}} + \log (1 / \delta).
$$
Similarly, we can bound the query complexity bound for any  $\varepsilon > 0$  as:
$$
N _ {T} \lesssim T \varepsilon^ {\rho} + \frac {\gamma^ {2}}{\lambda \varepsilon^ {2}} \cdot \operatorname {R e g} ^ {\ell_ {\phi}} (\mathcal {F}; T) \cdot \theta^ {\text {v a l}} \left(\mathcal {F}, \varepsilon / 8 _ {\gamma}, \operatorname {R e g} ^ {\ell_ {\phi}} (\mathcal {F}; T) / T; f ^ {\star}\right) + \log (1 / \delta).
$$
Setting  $\varepsilon = \left(\frac{\gamma^2}{\lambda T} \cdot \operatorname{Reg}^{\ell_{\phi}}(\mathcal{F};T) \cdot \theta^{\mathrm{val}}\left(\mathcal{F},\varepsilon /8\gamma, \operatorname{Reg}^{\ell_{\phi}}(\mathcal{F};T) / T;f^{\star}\right)\right)^{\frac{1}{\rho + 2}}$  in the above implies that
$$
N _ {T} \leq \left(\frac {\gamma^ {2}}{\lambda} \cdot \mathrm {R e g} ^ {\ell_ {\phi}} (\mathcal {F}; T) \cdot \theta^ {\mathrm {v a l}} \big (\mathcal {F}, \varepsilon / 8 _ {\gamma}, \mathrm {R e g} ^ {\ell_ {\phi}} (\mathcal {F}; T) / _ {T}; f ^ {\star} \big)\right) ^ {\frac {\rho}{\rho + 2}} \cdot T ^ {\frac {2}{\rho + 2}}.
$$
E.6 Proofs for Lower Bounds in Section 3.2
The proof of Theorem 3 below follows along the lines of the proof of the lower bound in Theorem 28 of Foster et al. [2020] with minor changes.
Proof of Theorem 3. Let  $\beta \leq \zeta /2$  be the largest number such that  $\beta^2\leq \min \{\zeta^2 /\mathfrak{s}^{\mathrm{val}}(\mathcal{F},\zeta ,\beta),\zeta^2 /16\}$ . Given the function class  $\mathcal{F}$ , assume that  $m = \mathfrak{s}^{\mathrm{val}}(\mathcal{F},\zeta ,\beta) > 0$  (the lower bound is obvious when the star number is 0). Let the target function  $f^{\star}$ , the data sequence  $x^{1},\ldots ,x^{m}$ , and the function  $f_{1},\dots,f_{m}\in \mathcal{F}$  be the witnesses for the fact that the star number is  $m$  (see Definition 3). First, for any  $i\in [m]$ , we have that  $|f^{\star}(x^{i})| > \zeta$  by definition of the star number. Next note that, for each  $f_{i}$  we have that, for any  $j\neq i$ ,
$$
\left| f _ {i} \left(x ^ {j}\right) \right| \geq \left| f ^ {\star} \left(x ^ {j}\right) \right| - \left| f _ {i} \left(x ^ {j}\right) - f ^ {\star} \left(x ^ {j}\right) \right| \geq \zeta - \beta \geq \zeta / 2
$$
On the other hand, from our definition of star number we have that,
$$
\left| f _ {i} \left(x ^ {i}\right) \right| \geq \zeta / 2.
$$
Hence we are guaranteed that each  $f_{i}$  has a margin of at least  $\zeta / 2$  on  $x^{1}, \ldots, x^{m}$ . Now consider the distribution  $\mu$  over the context to be the uniform distribution over  $\{x^{1}, \ldots, x^{m}\}$ . Also let  $P^{i}(y = 1 \mid x) = 1 + f_{i}(x) / 2$  be the conditional probability of label given context  $x$ . Let  $D_{i}$  denote the joint distribution over  $\mathcal{X} \times \{\pm 1\}$  given by drawing  $x$ 's from  $\mu$  and labels from  $P^{i}$ . Additionally, let  $D_{0}$  be given by drawing  $x$ 's from  $\mu$  and  $y$  conditioned on  $x$  as  $P(y = 1 \mid x) = 1 + f^{\star}(x) / 2$ . Finally, let  $\nu = 0$  with probability  $1 / 2$  and  $\nu \sim \mathrm{Uniform}([m])$  with probability  $1 / 2$ . Note that by our premise,
$$
\frac {1}{2} \mathbb {E} _ {D _ {0}} \left[ \operatorname {R e g} _ {T} \right] + \frac {1}{2 m} \sum_ {i = 1} ^ {m} \mathbb {E} _ {D _ {i}} \left[ \operatorname {R e g} _ {T} \right] = \mathbb {E} _ {\nu} \left[ \mathbb {E} _ {D _ {\nu}} \left[ \operatorname {R e g} _ {T} \right] \right] \leq c \frac {\zeta T}{m}, \tag {41}
$$