SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathrm {T} _ {B} = \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\operatorname {M a r g i n} \left(f _ {t} \left(x _ {t}\right)\right) > 2 \gamma \Delta_ {t} \left(x _ {t}\right), \widehat {y} _ {t} \neq \pi^ {\star} \left(x _ {t}\right) \right\} \\ \leq \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\left\| f _ {t} \left(x _ {t}\right) - f ^ {\star} \left(x _ {t}\right) \right\| > \Delta_ {t} \left(x _ {t}\right) \right\}, \\ \end{array}
$$
where the second inequality is due Lemma 12. However, note that the term inside the indicator contradicts (23) (which always holds). Thus,
$$
\mathrm {T} _ {B} = 0. \tag {24}
$$
Combining the bounds (22) and (24), we get that
$$
\begin{array}{l} \mathrm {R e g} _ {T} \leq \varepsilon T _ {\varepsilon} + 8 \gamma^ {2} \sum_ {t = 1} ^ {T} Z _ {t} \frac {\| f _ {t} (x _ {t}) - f ^ {\star} (x _ {t}) \| ^ {2}}{\varepsilon} \\ \leq \varepsilon T _ {\varepsilon} + \frac {8 \gamma^ {2}}{\varepsilon} \Psi_ {\delta} ^ {\ell_ {\phi}} (\mathcal {F}, T), \\ \end{array}
$$
where the last inequality is due to Lemma 10.
Since  $\varepsilon$  is a free parameter above, the final bound follows by choosing the best parameter  $\varepsilon$ , and by plugging in the form of  $\Psi_{\delta}^{\ell_{\phi}}(\mathcal{F},T)$ .
E.3.3 Total Number of Queries
We use the notation  $N_{T}$  to denote the total number of expert queries made by the learner within  $T$  rounds of interactions. Let  $\varepsilon > 0$  be a free parameter. Using the definition of  $Z_{t}$ , we have that
$$
N _ {T} = \sum_ {t = 1} ^ {T} Z _ {t}
$$
$$
\begin{array}{l} = \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\operatorname {M a r g i n} \left(f _ {t} \left(x _ {t}\right)\right) \leq 2 \gamma \Delta_ {t} \left(x _ {t}\right) \right\} \\ = \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\operatorname {M a r g i n} \left(f _ {t} \left(x _ {t}\right)\right) \leq 2 \gamma \Delta_ {t} \left(x _ {t}\right), \operatorname {M a r g i n} \left(f ^ {\star} \left(x _ {t}\right)\right) \leq \varepsilon \right\} \\ + \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\operatorname {M a r g i n} \left(f _ {t} \left(x _ {t}\right)\right) \leq 2 \gamma \Delta_ {t} \left(x _ {t}\right), \operatorname {M a r g i n} \left(f ^ {\star} \left(x _ {t}\right)\right) > \varepsilon \right\} \\ \leq \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\operatorname {M a r g i n} \left(f ^ {\star} \left(x _ {t}\right)\right) \leq \varepsilon \right\} \\ + \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\operatorname {M a r g i n} \left(f _ {t} \left(x _ {t}\right)\right) \leq 2 \gamma \Delta_ {t} \left(x _ {t}\right), \operatorname {M a r g i n} \left(f ^ {\star} \left(x _ {t}\right)\right) > \varepsilon , \Delta_ {t} \left(x _ {t}\right) \leq \varepsilon / 4 \gamma \right\} \\ + \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\operatorname {M a r g i n} \left(f _ {t} \left(x _ {t}\right)\right) \leq 2 \gamma \Delta_ {t} \left(x _ {t}\right), \operatorname {M a r g i n} \left(f ^ {\star} \left(x _ {t}\right)\right) > \varepsilon , \Delta_ {t} \left(x _ {t}\right) > \varepsilon / 4 \gamma \right\} \tag {25} \\ = T _ {\varepsilon} + \mathrm {T} _ {D} + \mathrm {T} _ {E}, \\ \end{array}
$$
where in the last line we use the definition of  $T_{\varepsilon}$ , and defined  $\mathrm{T}_D$  and  $\mathrm{T}_E$  respectively. We bound them separately below:
- Bound on  $\mathrm{T}_D$ . Recall (23) which implies that  $f^{\star}$  satisfies the bound  $\| f_t(x_t) - f^{\star}(x_t)\| \leq \Delta_t(x_t)$ . Thus, using Lemma 13, we get that
$$
\operatorname {M a r g i n} \left(f ^ {\star} \left(x _ {t}\right)\right) \leq 2 \gamma \left\| f _ {t} \left(x _ {t}\right) - f ^ {\star} \left(x _ {t}\right) \right\| + t \operatorname {M a r g i n} \left(f _ {t} \left(x _ {t}\right)\right) \leq 2 \gamma \Delta_ {t} \left(x _ {t}\right) + \operatorname {M a r g i n} \left(f _ {t} \left(x _ {t}\right)\right).
$$
The above implies that
$$
\begin{array}{l} \mathrm {T} _ {D} = \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\operatorname {M a r g i n} \left(f _ {t} \left(x _ {t}\right)\right) \leq 2 \gamma \Delta_ {t} \left(x _ {t}\right), \operatorname {M a r g i n} \left(f ^ {\star} \left(x _ {t}\right)\right) > \varepsilon , \Delta_ {t} \left(x _ {t}\right) \leq \varepsilon / 4 \gamma \right\} \\ \leq \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\operatorname {M a r g i n} \left(f ^ {\star} \left(x _ {t}\right)\right) \leq 4 \gamma \Delta_ {t} \left(x _ {t}\right), \operatorname {M a r g i n} \left(f ^ {\star} \left(x _ {t}\right)\right) > \varepsilon , \Delta_ {t} \left(x _ {t}\right) \leq \varepsilon / 4 \gamma \right\} \\ \leq 0, \\ \end{array}
$$
where the last line follows from the fact that all the conditions inside the indictor can not hold simultaneously.
- Bound on  $\mathrm{T}_E$ . We note that
$$
\begin{array}{l} \mathrm {T} _ {E} = \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\operatorname {M a r g i n} \left(f _ {t} \left(x _ {t}\right)\right) \leq 2 \gamma \Delta_ {t} \left(x _ {t}\right), \operatorname {M a r g i n} \left(f ^ {\star} \left(x _ {t}\right)\right) > \varepsilon , \Delta_ {t} \left(x _ {t}\right) > \varepsilon / 4 \gamma \right\} \\ \leq \sum_ {t = 1} ^ {T} Z _ {t} \mathbf {1} \left\{\Delta_ {t} \left(x _ {t}\right) \geq \varepsilon / 4 \gamma \right\} \\ \leq \frac {3 2 0 \Psi_ {\delta} ^ {\ell_ {\phi}} (\mathcal {F} , T)}{\varepsilon^ {2}} \cdot \mathfrak {E} \big (\mathcal {F}, _ {\varepsilon / 4 \gamma ;} f ^ {\star} \big). \\ \end{array}
$$
where the last line follows from setting  $\zeta = \varepsilon /4\gamma$  in Lemma 11.
Gathering the bounds above, we get that
$$
N _ {T} \leq T _ {\varepsilon} + \frac {6 4 0 \gamma^ {2} \Psi_ {\delta} ^ {\ell_ {\phi}} (\mathcal {F} , T)}{\varepsilon^ {2}} \cdot \mathfrak {E} (\mathcal {F}, \varepsilon / 4 \gamma ; f ^ {\star}).
$$
Before delving into the proof, we recall the relevant notation. In Algorithm 3,
Algorithm 3 Selective Sampling with Expert Feedback for Stochastic Contexts
Input: Parameters  $\delta, \gamma, \lambda, T$ , function class  $\mathcal{F}$ , and online regression oracle Oracle w.r.t  $\ell_{\phi}$ .  
1: Set  $\Psi_{\delta}^{\ell_{\phi}}(\mathcal{F}, T) = \frac{4}{\lambda} \operatorname{Reg}^{\ell_{\phi}}(\mathcal{F}; T) + \frac{112}{\lambda^2} \log(4 \log^2(T)/\delta)$ , Compute  $f_1 \gets \text{Oracle}_1(\emptyset)$ .  
2: Set  $E = [\log(T)]$  and  $\tau_e = 2^{e-1}$  for  $e \leq E$ .  
3: for  $e = 1, \ldots, E-1$  do  
4: Learner constructs the feasible set of optimal functions  $\mathcal{F}_e$  as  
[ \mathcal{F}_e = \left\{ f \in \mathcal{F} | \sum_{s=1}^{\tau_e-1} Z_s \| f(x_s) - f_s(x_s) \|^2 \leq \Psi_{\delta}^{\ell_{\phi}}(\mathcal{F}, T) \right\}. ]  
5: for  $t \gets \tau_e$  to  $\tau_{e+1} - 1$  do  
6: Nature samples  $x_t$  from an (unknown) distribution  $\mu$ .  
7: Learner computes  
[ g_t \in \operatorname*{argmin}_{g \in \mathcal{F}_e} \text{Margin}(g(x_t)), \quad \text{and}, \quad \Delta_e(x_t) := \max_{f, f' \in \mathcal{F}_e} \| f(x_t) - f'(x_t) \|. ]  
8: Learner decides whether to query:  $Z_t = 1$  {Margin  $(g_t(x_t)) \leq 2\gamma \Delta_e(x_t)$ }.  
9: if  $Z_t = 1$  then  
10: Learner plays the action  $\widehat{y}_t = \text{SelectAction}(f_t(x_t))$ .  
11: Learner queries the label  $y_t$  on  $x_t$ .  
12:  $f_{t+1} \gets \text{Oracle}_t(\{x_t, y_t\})$ .  
13: else  
14: Learner plays the action  $\widehat{y}_t = \text{SelectAction}(g_t(x_t))$ .  
15:  $f_{t+1} \gets f_t$ .
- The label  $y_{t} \sim \phi(f^{\star}(x_{t}))$ , where  $\phi$  denotes the link-function given in (2).
- The function SelectAction  $(f_{t}(x_{t}))\coloneqq \operatorname{argmax}_{k}\phi (f_{t}(x_{t}))[k]$
- For any vector  $v \in \mathbb{R}^K$ , the margin is given by the gap between the value at the largest and the second largest coordinate, i.e.
$$
\operatorname {M a r g i n} (v) = \phi (v) [ k ^ {\star} ] - \max  _ {k \neq k ^ {\star}} \phi (v) [ k ],
$$
- We also define  $T_{\varepsilon} = \sum_{t=1}^{T} \mathbf{1}\{\text{Margin}(f^{\star}(x_t)) \leq \varepsilon\}$  to denote the number of samples within  $T$  rounds of interaction for which the margin w.r.t.  $f^{\star}$  is smaller than  $\varepsilon$ .
- We define the function  $\operatorname{Gap}:\mathbb{R}^K\times [K]\mapsto \mathbb{R}^+$  as
$$
\operatorname {G a p} (v, k) = \max  _ {k ^ {\prime}} \phi (v) \left[ k ^ {\prime} \right] - \phi (v) [ k ], \tag {28}
$$
to denote the gap between the largest and the  $k$ -th coordinate of  $v$ . Recall that Lemma 9 holds.
- Additionally, we define the function  $Z^{e}$  to denote the query condition
$$
Z ^ {e} (x) = \mathbf {1} \left\{\inf  _ {g \in \mathcal {F} _ {e}} \operatorname {M a r g i n} (g (x)) \leq 2 \gamma \sup  _ {f, f ^ {\prime} \in \mathcal {F} _ {e}} \| f (x) - f ^ {\prime} (x) \| \right\}. \tag {29}
$$
The definition in (29) suggests that for all  $t \in [\tau_e, \tau_{e+1})$ ,  $Z_t = Z^e(x_t)$ , for all  $e \leq E-1$ .
Intuition for epoching. We next provide intuition on why epoching in needed in Algorithm 3 to get the improved query complexity bound. From the proof sketch in Section E.2, the term  $\sum_{t=1}^{T} Z_t \mathbf{1}\{\Delta_t(x_t) \geq \varepsilon\}$  appearing in the query complexity bound is handled using the eluder dimension of  $\mathcal{F}$ . When  $x_t$  is sampled i.i.d. we wish to bound this using disagreement-coefficient instead. However, note that in Algorithm 1 the query condition  $Z_t$  depends on the samples  $\{x_s\}_{s < t}$  drawn in all previous time steps and the corresponding query conditions  $\{Z_s\}_{s < t}$ . This introduces a bias, and thus the terms  $Z_t \mathbf{1}\{\Delta_t(x_t) \geq \varepsilon\}$  are no longer independent to each other. Thus, we can not directly use distributional properties like the disagreement coefficient to bound the query complexity. Algorithm 3 fixes this issue by defining epochs of doubling length such that the query condition in
epoch  $e$  only depends on the samples presented to the learner at time steps before this epoch (i.e. in time steps  $1 \leq t \leq \tau_{e} - 1$ ). Thus, the terms  $Z_{t}\mathbf{1}\{\Delta_{t}(x_{t}) \geq \varepsilon\}$  for  $\tau_{e} \leq t < \tau_{e + 1}$  are i.i.d. allowing us to get bounds in terms of distributional properties like the disagreement coefficient of  $\mathcal{F}$ .
However, note that whenever we query in Algorithm 3, we still choose the labels according to the estimate from the online regression oracle and thus the regret bound remains unchanged.
E.4.1 Supporting Technical Results
The following lemma establishes useful technical properties of the function  $f^{\star}$  and the sets  $\mathcal{F}_e$ .
Lemma 14. Suppose Algorithm 3 is run on the sequence  $\{x_{t}\}_{t\leq T}$  drawn i.i.d. from the unknown distribution  $\mu$ . Then, with probability at least  $1 - \delta$ , each of the following holds:
(a) For all  $t \leq T$ , the function  $f^{\star} \in \mathcal{F}$  satisfies
where  $\Psi_{\delta}^{\ell_{\phi}}(\mathcal{F},T) = \frac{4}{\lambda}\mathrm{Reg}^{\ell_{\phi}}(\mathcal{F};T) + \frac{112}{\lambda^2}\log (4\log^2 (T) / \delta).$
Thus,  $f^{\star} \in \mathcal{F}_{e}$  for all  $e \leq E - 1$ , and  $\| f^{\star}(x_{t}) - g_{t}(x_{t}) \| \leq \Delta_{e}(x_{t})$  for all  $\tau_{e} \leq t \leq \tau_{e + 1} - 1$ .
(b) For any function  $f \in \mathcal{F}_e$ , we have
$$
\mathbb {E} \left[ \sum_ {s = 1} ^ {\tau_ {e} - 1} Z _ {s} \| f (x _ {s}) - f ^ {\star} (x _ {s}) \| ^ {2} \right] \leq \widehat {\Psi} _ {\delta} ^ {\ell_ {\phi}} (\mathcal {F}; T),
$$
where  $\widehat{\Psi}_{\delta}^{\ell_{\phi}}(\mathcal{F};T)\coloneqq 2\Psi_{\delta}^{\ell_{\phi}}(\mathcal{F},T) + 4c_2\bigl (\log^4 (T)\sup_{\tau \leq T}\bigl (\tau \mathrm{Rad}_\tau^2 (\mathcal{F})\bigr) + 2\log (T)\log (E / \delta)\bigr).$
(c) For any  $e \leq E$ , and any function  $f \in \mathcal{F}_e$ , we have
$$
\left\| f \left(x _ {s}\right) - f ^ {\star} \left(x _ {s}\right) \right\| _ {\bar {\nu} _ {e}} \leq \sqrt {\frac {\widehat {\Psi} _ {\delta} ^ {\ell_ {\phi}} (\mathcal {F} ; T)}{\tau_ {e} - 1}}
$$
where the sub-distributions  $\bar{\mu}_{\bar{e}}(x) \coloneqq Z^{\bar{e}}(x)\mu (x)$  and  $\bar{\nu}_e \coloneqq \frac{1}{\tau_e - \tau_1}\sum_{\bar{e} = 1}^{e - 1}(\tau_{\bar{e} +1} - \tau_{\bar{e}})\bar{\mu}_{\bar{e}}$
(d) For any  $\bar{e} <  e$  , the corresponding sets  $\mathcal{F}_e$  and  $\mathcal{F}_{\bar{e}}$  satisfy the relation  $\mathcal{F}_e\subseteq \mathcal{F}_{\bar{e}}$
(e) For any  $\bar{e} \leq e$ , we have  $\bar{\mu}_e \preccurlyeq \bar{\mu}_{\bar{e}}$ .
Proof. We prove each part separately below:
(a) An application of Lemma 5, where we note that we do not query oracle when  $Z_{s} = 0$ , and thus do not count the time steps for which  $Z_{s} = 0$ , implies that
$$
\sum_ {s = 1} ^ {t} Z _ {s} \left\| f ^ {\star} (x _ {s}) - f _ {s} (x _ {s}) \right\| ^ {2} \leq \frac {4}{\lambda} \mathrm {R e g} ^ {\ell_ {\phi}} (\mathcal {F}; T) + \frac {1 1 2}{\lambda^ {2}} \log (4 \log^ {2} (T) / \delta) =: \Psi_ {\delta} ^ {\ell_ {\phi}} (\mathcal {F}, T)
$$
for all  $t \leq T$  with probability at least  $1 - \delta$ . Using the above for  $t = \tau_{e+1} - 1$  implies that  $f^{\star} \in \mathcal{F}_e$  for all  $e \leq E - 1$ . Since, we also have that  $g_t \in \mathcal{F}_e$  (by construction) for all  $\tau_e \leq t \leq \tau_{e+1} - 1$ , plugging in the definition of  $\Delta_e(x_t)$ , we immediately get that  $\| f^{\star}(x_t) - g_t(x_t) \| \leq \Delta_e(x_t)$ .
(b) Fix any epoch number  $\bar{e} \leq E - 1$ , and consider the time steps  $\tau_{\bar{e}} \leq t < \tau_{\bar{e} + 1}$ . Define the loss function
$$
\ell_ {\bar {e}} (f (x), f ^ {\star} (x)) = Z ^ {\bar {e}} (x) \| f (x) - f ^ {\star} (x) \| ^ {2}
$$
where  $Z^{\bar{e}}$  denotes the query conditions at epoch  $\bar{e}$  (defined in (29)), and recall that  $Z^{\bar{e}}$  does not depend on any samples that are drawn at epoch  $\bar{e}$  (by definition). Furthermore, note that
$\ell_{\bar{e}}$  is 2-smooth w.r.t.  $f$  and satisfies  $\ell_{\bar{e}}(f(x), f^{\star}(x)) \leq 4$  for all  $f, f^{\star} \in \mathcal{F}$  and  $x$ . Thus using Lemma 2, we get that for any  $f \in \mathcal{F}_e$ , with probability at least  $1 - \delta / E$ ,
$$
\begin{array}{l} \mathbb {E} \bigg [ \sum_ {s = \tau_ {\bar {e}}} ^ {\tau_ {\bar {e} + 1} - 1} Z ^ {\bar {e}} (x _ {s}) \| f (x _ {s}) - f ^ {\star} (x _ {s}) \| ^ {2} \bigg ] \\ \leq 2 \sum_ {s = \tau_ {\bar {e}}} ^ {\tau_ {\bar {e} + 1} - 1} Z ^ {\bar {e}} \left(x _ {s}\right) \| f \left(x _ {s}\right) - f ^ {\star} \left(x _ {s}\right) \| ^ {2} + c _ {2} \left(2 \tau_ {e} \log^ {3} \left(\tau_ {e}\right) \operatorname {R a d} _ {\tau_ {e}} ^ {2} \left(\mathcal {F} _ {e}\right) + 4 \log \left(E / \delta\right)\right) \\ \leq 2 \sum_ {s = \tau_ {\bar {e}}} ^ {\tau_ {\bar {e} + 1} - 1} Z ^ {\bar {e}} (x _ {s}) \| f (x _ {s}) - f ^ {\star} (x _ {s}) \| ^ {2} + 2 c _ {2} \left(\log^ {3} (T) \sup  _ {\tau \leq T} \big (\tau \operatorname {R a d} _ {\tau} ^ {2} (\mathcal {F}) \big) + 2 \log (E / \delta)\right), \\ \end{array}
$$
where in the last line we used the fact that  $\mathcal{F}_e\subseteq \mathcal{F}$ . Summing the above for all  $\bar{e}\leq e - 1$ , we get that for any  $f\in \mathcal{F}_e$
$$
\begin{array}{l} \mathbb {E} \bigg [ \sum_ {t = 1} ^ {\tau_ {e} - 1} Z ^ {e} (x _ {s}) \| f (x _ {s}) - f ^ {\star} (x _ {s}) \| ^ {2} \bigg ] \\ \leq 2 \sum_ {t = 1} ^ {\tau_ {e} - 1} Z ^ {e} \left(x _ {s}\right) \left\| f (x _ {s}) - f ^ {\star} (x _ {s}) \right\| ^ {2} + 4 c _ {2} E \left(\log^ {3} (T) \sup  _ {\tau \leq T} \left(\tau \operatorname {R a d} _ {\tau} ^ {2} (\mathcal {F})\right) + 2 \log (E / \delta)\right) \\ \leq 2 \Psi_ {\delta} ^ {\ell_ {\phi}} (\mathcal {F}, T) + 4 c _ {2} \left(\log^ {4} (T) \sup  _ {\tau \leq T} \bigl (\tau \mathrm {R a d} _ {\tau} ^ {2} (\mathcal {F}) \bigr) + 2 \log (T) \log (E / \delta)\right), \\ \end{array}
$$
where the last line follows by using the definition of  $\mathcal{F}_e$ , and that  $E \leq \lceil \log (T) \rceil$ , and that  $\mathcal{F}_e \subseteq \mathcal{F}$ .
(c) Starting from part-(b), we first note that
$$
\mathbb {E} \left[ \sum_ {s = 1} ^ {\tau_ {e} - 1} Z _ {s} \| f (x _ {s}) - f ^ {\star} (x _ {s}) \| ^ {2} \right] \leq \widehat {\Psi} _ {\delta} ^ {\ell_ {\phi}} (\mathcal {F}; T). \tag {30}
$$
Additionally, also note that
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {s = 1} ^ {\tau_ {e} - 1} Z _ {s} \left\| f (x _ {s}) - f ^ {\star} (x _ {s}) \right\| ^ {2} \right] = \mathbb {E} \left[ \sum_ {\bar {e} = 1} ^ {e - 1} \sum_ {s = \tau_ {\bar {e}}} ^ {\tau_ {\bar {e} + 1} - 1} Z _ {s} \left\| f (x _ {s}) - f ^ {\star} (x _ {s}) \right\| ^ {2} \right] \\ = \mathbb {E} \left[ \sum_ {\bar {e} = 1} ^ {e - 1} \sum_ {s = \tau_ {\bar {e}}} ^ {\tau_ {\bar {e} + 1} - 1} Z ^ {\bar {e}} (x _ {s}) \| f (x _ {s}) - f ^ {\star} (x _ {s}) \| ^ {2} \right] \\ = \sum_ {\bar {e} = 1} ^ {e - 1} \sum_ {s = \tau_ {\bar {e}}} ^ {\tau_ {\bar {e} + 1} - 1} \mathbb {E} _ {x _ {s} \sim \mu} \left[ Z ^ {\bar {e}} (x _ {s}) \| f (x _ {s}) - f ^ {\star} (x _ {s}) \| ^ {2} \right] \\ = \sum_ {\bar {e} = 1} ^ {e - 1} \left(\tau_ {\bar {e} + 1} - \tau_ {\bar {e}}\right) \mathbb {E} _ {x \sim \mu} \left[ Z ^ {\bar {e}} (x) \| f (x) - f ^ {\star} (x) \| ^ {2} \right], \\ \end{array}
$$
where the last two lines use the fact that the query condition  $Z^{\overline{e}}$  does not depend on the samples from rounds  $t = \tau_{\overline{e}}$  to  $\tau_{\overline{e} + 1} - 1$ . Plugging in the definition of  $\bar{\mu}_{\overline{e}}$  in the above, we get that
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {s = 1} ^ {\tau_ {e} - 1} Z _ {s} \left\| f (x _ {s}) - f ^ {\star} (x _ {s}) \right\| ^ {2} \right] = \sum_ {\bar {e} = 1} ^ {e - 1} \left(\tau_ {\bar {e} + 1} - \tau_ {\bar {e}}\right) \cdot \mathbb {E} _ {x \sim \bar {\mu} _ {\bar {e}}} \left[ \left\| f (x _ {s}) - f ^ {\star} (x _ {s}) \right\| ^ {2} \right] \\ = \left(\tau_ {e} - \tau_ {1}\right) \cdot \mathbb {E} _ {x \sim \bar {\nu} _ {e}} \left[ \left\| f (x _ {s}) - f ^ {\star} (x _ {s}) \right\| ^ {2} \right] \\ = \left(\tau_ {e} - \tau_ {1}\right) \cdot \| f (x _ {s}) - f ^ {\star} (x _ {s}) \| _ {\bar {\nu} _ {e}} ^ {2}, \tag {31} \\ \end{array}
$$
where in the second line, we used the fact that the sub-distribution  $\bar{\nu}_{e} := \frac{1}{\tau_{e} - \tau_{1}} \sum_{\bar{e} = 1}^{e - 1} \left( \tau_{\bar{e} + 1} - \tau_{\bar{e}} \right) \bar{\mu}_{\bar{e}}$ .
Combining (30) and (31), we get that
$$
\left\| f (x _ {s}) - f ^ {\star} (x _ {s}) \right\| _ {\bar {\nu} _ {e}} \leq \sqrt {\frac {\widehat {\Psi} _ {\delta} ^ {\ell_ {\phi}} (\mathcal {F} ; T)}{\tau_ {e} - 1}}
$$
(d) The argument follows from the definition of the set  $\mathcal{F}_e$  as any function  $f\in \mathcal{F}_e$  that satisfies
$$
\sum_ {s = 1} ^ {\tau_ {e} - 1} Z _ {s} \left\| f (x _ {s}) - f _ {s} (x _ {s}) \right\| ^ {2} \leq \Psi_ {\delta} ^ {\ell_ {\phi}} (\mathcal {F}, T),
$$
also satisfies the constraint
$$
\sum_ {s = 1} ^ {\tau_ {\bar {e}} - 1} Z _ {s} \left\| f (x _ {s}) - f _ {s} (x _ {s}) \right\| ^ {2} \leq \Psi_ {\delta} ^ {\ell_ {\phi}} (\mathcal {F}, T),
$$
for any  $\bar{e} \leq e$ , since the left hand side consists of lesser number of terms and all terms are non-negative. Thus,  $\mathcal{F}_e \subseteq \mathcal{F}_{\bar{e}}$ .
(e) Recall that for any  $e \leq E - 1$ , the sub-probability measure  $\bar{\mu}_e(x) \coloneqq Z^e(x)\mu(x)$  where  $Z^e(x) = \mathbf{1}\{\min_{g \in \mathcal{F}_e} \|g(x)\| \leq \Delta_e(x)\}$ , and  $\Delta_e(x) = \max_{f', f \in \mathcal{F}_e} \|f(x) - f'(x)\|$ . First note that for any  $\bar{e} \leq e$ ,
$$
\Delta_ {e} (x) = \max  _ {f ^ {\prime}, f \in \mathcal {F} _ {e}} \| f (x) - f ^ {\prime} (x) \| \leq \max  _ {f ^ {\prime}, f \in \mathcal {F} _ {\bar {e}}} \| f (x) - f ^ {\prime} (x) \| = \Delta_ {\bar {e}} (x),
$$
where the inequality above holds because  $\mathcal{F}_e\subseteq \mathcal{F}_{\bar{e}}$  due to part-(d) above. Furthermore,
$$
\min  _ {g \in \mathcal {F} _ {e}} \| g (x) \| \geq \min  _ {g \in \mathcal {F} _ {\bar {e}}} \| g (x) \|,
$$
again because  $\mathcal{F}_e\subseteq \mathcal{F}_{\bar{e}}$  . Thus,
$$
Z ^ {e} (x) = \mathbf {1} \left\{\min  _ {g \in \mathcal {F} _ {e}} \| g (x) \| \leq \Delta_ {e} (x) \right\} \leq \mathbf {1} \left\{\min  _ {g \in \mathcal {F} _ {\bar {e}}} \| g (x) \| \leq \Delta_ {\bar {e}} (x) \right\} \leq Z ^ {\bar {e}} (x).
$$
The above implies that  $\bar{\mu}_e\leq \bar{\mu}_{\bar{e}}$
Lemma 15. Let  $\varepsilon_0, \gamma_0 \geq 0$ , and  $f^{\star} \in \mathcal{F}$ . Then, for any sub distribution  $\bar{\mu}$  such that  $\mathbb{E}_{x \sim \bar{\mu}}[\mathbf{1}\{x \in \mathcal{X}\}] > 0$ ,  $\varepsilon \geq \varepsilon_0$  and  $\gamma \geq \gamma_0$ ,
$$
\frac {\varepsilon^ {2}}{\gamma^ {2}} \Pr_ {x \sim \bar {\mu}} \big (\exists f \in \mathcal {F}: \| f (x) - f ^ {\star} (x) \| > \varepsilon , \| f (x _ {s}) - f ^ {\star} (x _ {s}) \| _ {\bar {\mu}} \leq \gamma \big) \leq \theta^ {\mathrm {v a l}} (\mathcal {F}, \varepsilon_ {0}, \gamma_ {0}; f ^ {\star}).
$$
Proof. The key idea in the proof is to go from sub distributions to distributions, and then invoking the definition of  $\theta$  from Definition 2. Define  $\kappa = \mathbb{E}_{x\sim \bar{\mu}}[\mathbf{1}\{x\in \mathcal{X}\} ]$ . Since  $0 < \kappa \leq 1$ , we can define a probability measure  $\mu$  such that  $\mu (x) = \bar{\mu} (x) / \kappa$ . Thus, for any  $\varepsilon \geq \varepsilon_0$  and  $\gamma \geq \gamma_0$ ,
$$
\begin{array}{l} \frac {\varepsilon^ {2}}{\gamma^ {2}} \Pr_ {x \sim \bar {\mu}} \left(\exists f \in \mathcal {F}: \| f (x) - f ^ {\star} (x) \| > \varepsilon , \| f (x _ {s}) - f ^ {\star} (x _ {s}) \| _ {\bar {\mu}} \leq \gamma\right) \\ = \frac {\varepsilon^ {2}}{\gamma^ {2} / k} \Pr_ {x \sim \mu} \left(\exists f \in \mathcal {F}: \| f (x) - f ^ {\star} (x) \| > \varepsilon , \| f (x _ {s}) - f ^ {\star} (x _ {s}) \| _ {\bar {\mu}} \leq \gamma\right) \\ \leq \frac {\varepsilon^ {2}}{\gamma^ {2} / k} \Pr_ {x \sim \mu} \left(\exists f \in \mathcal {F}: \| f (x) - f ^ {\star} (x) \| > \varepsilon , \| f (x _ {s}) - f ^ {\star} (x _ {s}) \| _ {\mu} \leq \gamma / \sqrt {k}\right) \\ = \frac {\varepsilon^ {2}}{\bar {\gamma} ^ {2}} \Pr_ {x \sim \mu} \left(\exists f \in \mathcal {F}: \| f (x) - f ^ {\star} (x) \| > \varepsilon , \| f (x _ {s}) - f ^ {\star} (x _ {s}) \| _ {\mu} \leq \bar {\gamma}\right) \\ \leq \sup  _ {\varepsilon \geq \varepsilon_ {0}, \bar {\gamma} \geq \gamma_ {0}} \frac {\varepsilon^ {2}}{\bar {\gamma} ^ {2}} \Pr_ {x \sim \mu} \left(\exists f \in \mathcal {F}: \| f (x) - f ^ {\star} (x) \| > \varepsilon , \| f (x _ {s}) - f ^ {\star} (x _ {s}) \| _ {\mu} \leq \bar {\gamma}\right), \\ \end{array}
$$
where in the second last line, we defined  $\bar{\gamma} = \gamma / \sqrt{k}$ , and the last line used the fact that both  $\varepsilon \geq \varepsilon_0$  and  $\bar{\gamma} \geq \gamma_0$ . The final statement follows by noting the fact that  $\mu$  is a distribution and the definition of the disagreement coefficient  $\theta^{\mathrm{val}}(;)$  from Definition 2.
The following technical result will be useful in bounding the query complexity for Algorithm 3.
Lemma 16. For any  $t \leq T$ , let  $e(t)$  denotes the epoch number such that  $\tau_{e(t)} \leq t < \tau_{e(t)+1}$ . Let  $f^{\star} \in \mathcal{F}$  satisfy Lemma 14, and let  $\Delta_{e(t)}(x_t)$  be defined in Algorithm 3. Then, for any  $\zeta > 0$ , with probability at least  $1 - \delta$ ,
$$
\sum_ {t = 1} ^ {T} Z _ {t} \mathbf {1} \big \{\Delta_ {e (t)} (x _ {t}) \geq \zeta \big \} \leq 1 2 \log (T) \cdot \frac {\widehat {\Psi} _ {\delta} ^ {\ell_ {\phi}} (\mathcal {F} ; T)}{\zeta^ {2}} \cdot \theta^ {\mathrm {v a l}} \bigg (\mathcal {F}, \frac {\zeta}{2}, \frac {\widehat {\Psi} _ {\delta} ^ {\ell_ {\phi}} (\mathcal {F} ; T)}{T}; f ^ {\star} \bigg) + 4 \log (2 / \delta).
$$
Proof. Recall the definition of the query rule  $Z^{e}$  given in (29), and note that the function  $Z^{e}$  is independent of the samples  $\{x_{t}\}_{t = \tau_{e}}^{\tau_{e + 1} - 1}$  chosen by the nature for time steps at epoch  $e$ . Additionally, also recall that at every time step,  $x_{t}$  is sampled independently from the distribution  $\mu$ . Thus, using the query condition  $Z^{e}$ , we can define the sub-probability measure
$$
\bar {\mu} _ {e} := \mu (x) Z ^ {e} (x), \tag {32}
$$