SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lemma 2 (Theorem 1 in Srebro et al. [2010]). Let  $T > 0$ , and let  $\mathcal{F} = \{\mathcal{X} \times \mathcal{Y}\}$  be an arbitrary function class, and  $\ell$  be an  $\gamma$ -smooth and non-negative loss such that  $|\ell(f(x), y)| \leq B$  for all  $x \in \mathcal{X}, y \in \mathcal{Y}, f \in \mathcal{F}$ . For any  $\delta > 0$ , we have with probability at least  $1 - \delta$  over a random sample of size  $T$ , for any  $f \in \mathcal{F}$ ,
$$
T \mathbb {E} _ {(x, y) \sim \mu} [ \ell (f (x), y) ] \leq 2 \sum_ {t = 1} ^ {T} \ell (f (x _ {t}), y _ {t}) + c _ {1} \left(H T \log^ {3} (T) \operatorname {R a d} _ {T} ^ {2} (\mathcal {F}) + B \log (1 / \delta)\right)
$$
where  $c_{1} < 10^{5}$  is a numeric constant, and  $\mathsf{Rad}_T(\mathcal{F})$  denotes the Rademacher complexity of the class  $\mathcal{F}$ .
The precise value of the numeric constant  $c_{1}$  in the above can be derived from Srebro et al. [2010] and Mendelson [2002]. Note that for finite function classes, we have  $\mathrm{Rad}_T(\mathcal{F}) = \mathcal{O}\Big(\sqrt{\log(|\mathcal{F}|)/T}\Big)$  and thus the second term above is bounded by  $\widetilde{\mathcal{O}}(\log(|\mathcal{F}|))$ . In general, we have that  $T\mathrm{Rad}_T^2(\mathcal{F}) = \widetilde{\mathcal{O}}(\mathrm{Reg}^{\mathrm{sq}}(\mathcal{F};T))$  [Rakhlin and Sridharan, 2014], and thus the second term is always dominated by the other terms in our regret and query complexity bounds.
The following inequalities are well-known; we use the version stated in Zhu and Nowak [2022].
Lemma 3 (Freedman's inequality). Let  $\{X_{t}\}_{t\leq T}$  be a real-valued martingale different sequence adapted to the filtration  $\mathfrak{F}_t$ , and let  $\mathbb{E}_t[\cdot ]\coloneqq \mathbb{E}[\cdot |\mathfrak{F}_{t - 1}]$ . If  $|X_{t}|\leq B$  almost surely, then for any  $\eta \in (0,1 / B)$ , the following holds with probability at least  $1 - \delta$ :
$$
\sum_ {t = 1} ^ {T} X _ {t} \leq \eta \sum_ {t = 1} ^ {T} \mathbb {E} _ {t} \left[ X _ {t} ^ {2} \right] + \frac {B \log (1 / \delta)}{\eta}.
$$
Lemma 4. Let  $\{X_{t}\}_{t\leq T}$  be a sequence of positive valued random variables adapted to the filtration  $\mathfrak{F}_t$ , and let  $\mathbb{E}_t[\cdot ]\coloneqq \mathbb{E}[\cdot \mid \mathfrak{F}_{t - 1}]$ . If  $X_{t}\leq B$  almost surely, then with probability at least  $1 - \delta$
$$
\sum_ {t = 1} ^ {T} X _ {t} \leq \frac {3}{2} \sum_ {t = 1} ^ {T} \mathbb {E} _ {t} [ X _ {t} ] + 4 B \log (2 / \delta),
$$
$$
\sum_ {t = 1} ^ {T} \mathbb {E} _ {t} \left[ X _ {t} \right] \leq 2 \sum_ {t = 1} ^ {T} X _ {t} + 8 B \log (2 / \delta).
$$
Lemma 5. Suppose that the labels are generated according to the (2) where the link function satisfies Assumption 1. Additionally, assume that the regression oracle satisfies the guarantee (3). Then, for any  $\delta \leq 1 / e$  and  $T\geq 3$ , with probability at least  $1 - \delta$ , we have for all  $t\leq T$
$$
\sum_ {s = 1} ^ {t} \| f _ {s} (x _ {s}) - f ^ {\star} (x _ {s}) \| ^ {2} \leq \Psi_ {\delta} ^ {\ell_ {\phi}} (\mathcal {F}, T) := \frac {4}{\lambda} \mathrm {R e g} ^ {\ell_ {\phi}} (\mathcal {F}; T) + \frac {1 1 2}{\lambda^ {2}} \log (4 \log^ {2} (T) / \delta),
$$
where  $B$  is defined such that  $\sup_x f(x) \leq B$ .
Proof. Using Agarwal [2013, Lemma 2] along with an Union bound implies that for all  $t \leq T$ ,
$$
\sum_ {s = 1} ^ {t} \left\| f _ {s} \left(x _ {s}\right) - f ^ {\star} \left(x _ {s}\right) \right\| ^ {2} \leq \frac {4}{\lambda} \sum_ {s = 1} ^ {t} \left(\ell_ {\phi} \left(f _ {s} \left(x _ {s}\right), y _ {s}\right) - \ell_ {\phi} \left(f ^ {\star} \left(x _ {s}\right), y _ {s}\right)\right) + \frac {1 1 2}{\lambda^ {2}} \log \left(4 \log^ {2} (T) / \delta\right).
$$
Plugging in the regret bound (3) in the above, we get that
$$
\sum_ {s = 1} ^ {t} \left\| f _ {s} \left(x _ {s}\right) - f ^ {\star} \left(x _ {s}\right) \right\| ^ {2} \leq \sum_ {s = 1} ^ {T} \left\| f _ {s} \left(x _ {s}\right) - f ^ {\star} \left(x _ {s}\right) \right\| ^ {2} \leq \frac {4}{\lambda} \operatorname {R e g} ^ {\ell_ {\phi}} (\mathcal {F}; T) + \frac {1 1 2}{\lambda^ {2}} \log (4 \log^ {2} (T) / \delta).
$$
D.3 Eluder Dimension, Disagreement Coefficient, and Star Number
For the sake of completeness, we recall the scalar versions of scale-sensitive eluder dimension, and disagreement coefficient introduced in Russo and Van Roy [2013], Foster et al. [2020], which is defined for a class  $\mathcal{F} \subseteq \{\mathcal{X} \mapsto \mathbb{R}\}$  of scalar valued functions.
Definition 4 (Scale-sensitive eluder dimension (scalar version), Russo and Van Roy [2013], Foster et al. [2020]). Let  $\mathcal{F} \subseteq \{\mathcal{X} \mapsto \mathbb{R}\}$ . Fix any  $f^{\star} \in \mathcal{F}$ , and define  $\mathfrak{E}'(\mathcal{F},\beta;f^{\star})$  to be the length of the longest sequence of contexts  $x_{1}, x_{2}, \ldots, x_{m}$  such that for all  $i$ , there exists  $f_{i} \in \mathcal{F}$  such that
$$
\left| f _ {i} \left(x _ {i}\right) - f ^ {\star} \left(x _ {i}\right) \right| > \beta , \quad a n d \quad \sum_ {j <   i} \left(f _ {i} \left(x _ {j}\right) - f ^ {\star} \left(x _ {j}\right)\right) ^ {2} \leq \beta^ {2}.
$$
We define the scale-sensitive eluder dimension as  $\mathfrak{E}(\mathcal{F},\beta_0;f^{\star})\coloneqq \sup_{\beta_0\geq \beta}\mathfrak{E}'(\mathcal{F},\beta ;f^{\star})$
We next provide some examples of function classes with bounded eluder dimension. The examples  $(a) - (c)$  first appeared in Russo and Van Roy [2013], and  $(d)$  first appeared in Osband and Van Roy [2014].
(a) For any function class  $\mathcal{F}:\{\mathcal{X}\mapsto \mathbb{R}\}$  and  $f^{\star}\in \mathcal{F},\mathfrak{E}(\mathcal{F},\beta_{0};f^{\star})\leq O(|\mathcal{X}|).$
(b) For the class  $\mathcal{F}$  of linear functions on a known feature map  $\phi$  i.e.,  $\mathcal{F} = \{f \mid f(x) = \langle \theta_f, \phi(x) \rangle, \theta_f \in \mathbb{R}^d, \| \theta_f \| \leq 1\}$ , we have  $\mathfrak{E}(\mathcal{F}, \beta_0; f^\star) \leq O(d \log(1 / \varepsilon))$ .
(c) For the class  $\mathcal{F}$  of generalized linear functions on a known feature map  $\phi$  i.e.,  $\mathcal{F} = \{f \mid f(x) = g(\langle \theta_f, \phi(x) \rangle), \theta_f \in \mathbb{R}^d, \| \theta_f \| \leq 1\}$  where  $g$  is an increasing continuously differentiable function, we have  $\mathfrak{E}(\mathcal{F}, \beta_0; f^\star) \leq O(dr^2 \log (L / \varepsilon))$  where  $r = \sup_{\theta, x} g'(\langle \theta, x \rangle) / \inf_{\theta, x} g'(\langle \theta, x \rangle)$  and  $L = \sup_{\theta, x} g'(\langle \theta, \phi(x) \rangle)$ .
(d) For the class  $\mathcal{F}$  of quadratic functions on a known feature map  $\phi$  i.e.  $\mathcal{F} = \{f\mid f(x) = \phi (x)^{\top}\Sigma_{f}\phi (x),\Sigma_{f}\in \mathbb{R}^{d\times d},\| \Sigma_{f}\|_{F}\leq 1\}$ .
We next recall the definition of scale-sensitive disaggrement coefficient which appears in our bounds for the case of stochastic contexts.
Definition 5 (Scale-sensitive disagreement coefficient (scalar version), Foster et al. [2020]). Let  $\mathcal{F} \subseteq \{\mathcal{X} \mapsto \mathbb{R}\}$ . For any  $f^{\star} \in \mathcal{F}$ , and  $\gamma_0, \varepsilon_0 > 0$ , the value function disagreement coefficient  $\theta^{\mathrm{val}}(\mathcal{F}, \varepsilon_0, \gamma_0; f^{\star})$  is defined as
$$
\sup  _ {\mu} \sup  _ {\gamma > \gamma_ {0}, \varepsilon > \varepsilon_ {0}} \left\{\frac {\varepsilon^ {2}}{\gamma^ {2}} \cdot \Pr_ {x \sim \mu} (\exists f \in \mathcal {F} | | f (x) - f ^ {\star} (x) | > \varepsilon , \| f - f ^ {\star} \| _ {\mu} \leq \gamma) \right\} \vee 1
$$
where  $\| f\| = \sqrt{\mathbb{E}_{x\sim\mu}[f^2(x)]}$
As we will show in Lemma 6 below, the scale-sensitive disagreement coefficient of  $\mathcal{F}$  is always bounded by the eluder dimension of  $\mathcal{F}$  up to a constant factor on the dependence on  $\varepsilon_0$  and  $\gamma_0$ . However, the disagreement coefficient can be significantly smaller than the eluder dimension because it can leverage additional distributional structure. We refer the reader to Foster et al. [2020] for bounds on the eluder dimension, and the disagreement coefficient for various function classes. In the following, we extend the above definitions to vector-valued functions to account for the vector-valued function classes that we consider in this work.
Definition 6 (Scale-sensitive eluder dimension (normed version)). Let  $\mathcal{F} \subseteq \{\mathcal{X} \mapsto \mathbb{R}^K\}$ . Fix any  $f^{\star} \in \mathcal{F}$ , and define  $\widetilde{\mathfrak{E}}(\mathcal{F}, \beta; f^{\star})$  to be the length of the longest sequence of contexts  $x_1, x_2, \ldots, x_m$  such that for all  $i$ , there exists  $f_i \in \mathcal{F}$  such that
We define the scale-sensitive eluder dimension as  $\mathfrak{E}(\mathcal{F},\beta^{\prime};f^{\star}) = \sup_{\beta \geq \beta^{\prime}}\widetilde{\mathfrak{E}} (\mathcal{F},\beta ;f^{\star})$
We note that the normed eluder dimension can be lower bounded in terms of the eluder dimension of scalar-valued function class obtained by projecting the output of the functions in  $\mathcal{F}$  along different coordinates. Let  $\mathcal{F}_j = \{P_jf\mid P_jf(x) = f(x)[j], f\in \mathcal{F}\}$ , then clearly, for any  $f^{\star}\in \mathcal{F}$ ,  $\mathfrak{E}(\mathcal{F},\beta ;f^{\star})\geq \sup_{j\in [d]}\mathfrak{E}(P_j\mathcal{F},\beta ;P_jf^{\star})$ . Furthermore, we always have that  $\mathfrak{E}(\mathcal{F},\beta ;f^{\star})\leq \kappa \sum_{j = 1}^{d}\mathfrak{E}(P_j\mathcal{F},\beta ;P_jf^{\star})$ , where  $\kappa$  hides poly(d) factors. We next define the normed version of disagreement coefficient for vector-valued functions.
Definition 7 (Scale sensitive disagreement coefficient (normed version), Foster et al. [2020]). Let  $\mathcal{F} \subseteq \{\mathcal{X} \mapsto \mathbb{R}^K\}$ . For any  $f^{\star} \in \mathcal{F}$ , and  $\beta_0, \varepsilon_0 > 0$ , the value function disagreement coefficient  $\theta^{\mathrm{val}}(\mathcal{F}, \varepsilon_0, \beta_0; f^{\star})$  is defined as
$$
\sup  _ {\mu} \sup  _ {\beta > \beta_ {0}, \varepsilon > \varepsilon_ {0}} \left\{\frac {\varepsilon^ {2}}{\beta^ {2}} \cdot \Pr_ {x \sim \mu} \left(\exists f \in \mathcal {F} \mid \| f (x) - f ^ {\star} (x) \| > \varepsilon , \| f - f ^ {\star} \| _ {\mu} \leq \beta\right) \right\} \vee 1
$$
where  $\| f\|_{\mu} = \sqrt{\mathbb{E}_{x\sim\mu}\left[\|f(x)\|^2\right]}$ .
We additionally also define the following bivariate version of eluder dimension for vector-valued functions.
Definition 8 (Scale-sensitive eluder dimension (bivariate version)). Let  $\mathcal{F} \subseteq \{\mathcal{X} \mapsto \mathbb{R}^K\}$ . Fix any  $f^\star \in \mathcal{F}$ , and define  $\check{\mathfrak{E}}'(\mathcal{F}, \beta; f^\star)$  to be the length of the longest sequence of contexts and actions  $(x_1, y_1), (x_2, y_2) \ldots (x_m, y_m)$  such that for all  $i$ , there exists  $f_i \in \mathcal{F}$  such that
$$
\left| f _ {i} \left(x _ {i}\right) \left[ y _ {i} \right] - f ^ {\star} \left(x _ {i}\right) \left[ y _ {i} \right] \right| > \beta , \quad a n d \quad \sum_ {j <   i} \left(f _ {i} \left(x _ {j}\right) \left[ y _ {j} \right] - f ^ {\star} \left(x _ {j}\right) \left[ y _ {j} \right]\right) ^ {2} \leq \beta^ {2}.
$$
We define the scale sensitive eluder dimension (mixed version) as  $\mathfrak{E}(\mathcal{F},\beta ;f^{\star})\coloneqq$ $\sup_{\beta \geq \beta_0}\mathring{\mathfrak{E}}'(\mathcal{F},\beta_0;f^\star)$
We next define the strong variant of scale-sensitive star number.
Definition 9 (scale-sensitive star number (strong version), Foster et al. [2020]). Let  $\mathcal{F} \subseteq \{\mathcal{X} \mapsto \mathbb{R}^K\}$ . For any  $f^\star \in \mathcal{F}$  and  $\beta > 0$ , let  $\underline{\mathfrak{s}}^{\mathrm{val}}(\mathcal{F},\beta)$  denote the length of the longest sequence of contexts  $\{x_1,\ldots ,x_m\}$  such that for all  $i$ , there exists  $f_{i} \in \mathcal{F}$  such that
$$
\left\| f _ {i} \left(x _ {i}\right) - f ^ {\star} \left(x _ {i}\right) \right\| > \beta , \quad a n d \quad \sum_ {j \neq i} \left\| f _ {i} \left(x _ {j}\right) - f ^ {\star} \left(x _ {j}\right) \right\| ^ {2} \leq \beta^ {2}.
$$
We define the scale-sensitive star number as  $\check{\mathfrak{s}}^{\mathrm{val}}(\mathcal{F},\beta)\coloneqq \sup_{\beta >\beta_0}\check{\mathfrak{s}}^{\mathrm{val}}(\mathcal{F},\beta_0)$ .
The next result provides a relation between the star number, disagreement coefficient and the eluder dimension.
Lemma 6 (Foster et al. [2020]). Suppose  $\mathcal{F} \subseteq \{\mathcal{X} \mapsto \mathbb{R}^K\}$  is a uniform Glivenko-Cantelli class. For any  $f^\star \in \mathcal{F}$  and  $\gamma, \varepsilon > 0$ , we have  $\mathfrak{s}^{\mathrm{val}}(\mathcal{F}, \beta; f^\star) \leq \mathfrak{E}(\mathcal{F}, \beta; f^\star)$ ,  $\theta^{\mathrm{val}}(\mathcal{F}, \varepsilon, \gamma; f^\star) \leq 4(\mathfrak{s}^{\mathrm{val}}(\mathcal{F}, \gamma; f^\star))^2$  and  $\theta^{\mathrm{val}}(\mathcal{F}, \varepsilon, \gamma; f^\star) \leq 4\mathfrak{E}(\mathcal{F}, \gamma; f^\star)$ .
The following two technical lemmas are useful in bounding the total number of queries made by our selective sampling and imitation learning algorithms. We first provide a technical result which bounds the number of times we can find a function  $f'$  in a refinement  $\mathcal{F}_t$  of  $\mathcal{F}$ , such that  $f'$  is sufficiently far away from  $f^{\star} \in \mathcal{F}$ . This result is a variant of Russo and Van Roy [2013, Lemma 3], and first appears in Foster et al. [2020, Lemma E.4].
Lemma 7. Let  $\{x_{t},y_{t},Z_{t}\}_{t = 1}^{T}$  be sequence of tuples, where  $x_{t}\in \mathcal{X}$  and  $Z_{t}\in \{0,1\}$ . Fix any  $f^{\star}\in \mathcal{F}$  and define the set  $\mathcal{F}_t = \{f\in \mathcal{F}\mid \sum_{s = 1}^{t - 1}Z_s(f(x_s)[y_s] - f^\star (x_s)[y_s])^2\leq \beta^2\}$ . Then, for any  $\zeta >0$
$$
\sum_ {t = 1} ^ {T} Z _ {t} \mathbf {1} \left\{\exists f ^ {\prime} \in \mathcal {F} _ {t}: \left(f ^ {\prime} \left(x _ {t}\right) [ y _ {t} ] - f ^ {\star} \left(x _ {t}\right) [ y _ {t} ]\right) \geq \zeta \right\} \leq \left(\frac {\beta^ {2}}{\zeta^ {2}} + 1\right) \check {\mathfrak {E}} (\mathcal {F}, \zeta ; f ^ {\star}).
$$
Proof. We first note that we can always remove a tuple  $\{(x_t, y_t), Z_t\}$  whenever  $Z_t = 0$  without any effect on the conclusion. Hence, we can assume  $Z_t = 1$  for all  $t \in [T]$  without loss of generality. Then the rest of the proof essentially follows from Foster et al. [2020, Lemma E.4]. For completeness, we state the full proof here.
For simplicity of presentation, we say  $(x_{t},y_{t})$  is  $\zeta$ -independent of  $(x_{1},y_{1}),\ldots ,(x_{t - 1},y_{t - 1})$  if there exists  $f\in \mathcal{F}$  such that  $|f(x_{t})[y_{t}] - f^{\star}(x_{t})[y_{t}]\mid \geq \zeta$  and  $\sum_{s = 1}^{t - 1}(f(x_s)[y_s] - f^{\star}(x_s)[y_s])^2\leq \zeta^2$ . Otherwise, we say  $x$  is  $\zeta$ -dependent. The proof consists of the following two claims.
First, we claim that for any  $t \in [T]$ , if there exists  $f \in \mathcal{F}_t$  such that  $|f(x_t)[y_t] - f^\star(x_t)[y_t]| \geq \zeta$ , then  $x_t$  is  $\zeta$ -dependent on at most  $\beta^2 / \zeta^2$  disjoint sequences of  $(x_1, y_1), \ldots, (x_{t-1}, y_{t-1})$ . To show this, let's say  $x_t$  is  $\zeta$ -dependent on a particular subsequence  $(x_{i_1}, y_{i_1}), \ldots, (x_{i_k}, y_{i_k})$  while  $|f(x)[y] - f^\star(x)[y]| \geq \zeta$ . Then it must hold that
$$
\sum_ {j = 1} ^ {k} \left(f (x _ {i _ {j}}) [ y _ {i _ {j}} ] - f ^ {\star} (x _ {i _ {j}}) [ y _ {i _ {j}} ]\right) ^ {2} \geq \zeta^ {2}.
$$
If there are  $M$  such disjoint subsequence, then we can add them up and obtain the following:
$$
\sum_ {s = 1} ^ {t - 1} \Big (f (x _ {s}) [ y _ {s} ] - f ^ {\star} (x _ {s}) [ y _ {s} ] \Big) ^ {2} \geq M \zeta^ {2}.
$$
By the construction of  $\mathcal{F}_t$ , the left-hand side above is at most  $\beta^2$ . Hence we conclude that  $\beta^2 \geq M\zeta^2$  which implies  $M \leq \beta^2 / \zeta^2$ .
Second, we claim that for any  $k$  and any sequence  $(x_{1},y_{1}),\ldots ,(x_{k},y_{k})$ , there exists  $j\leq k$  such that  $x_{j}$  is  $\zeta$ -dependent on at least  $N\coloneqq \lfloor k / \mathfrak{E}(\mathcal{F},\zeta ;f^{\star})\rfloor$  disjoint subsequences of  $(x_{1},y_{1}),\ldots ,(x_{j - 1},y_{j - 1})$ . This can be proved by construction. Let  $B_{1},\dots ,B_{N}$  be  $N$  subsequences of  $(x_{1},y_{1}),\ldots ,(x_{k},y_{k})$  and are initialized with  $B_{i} = \{(x_{i},y_{i})\}$ . Then we repeat the following process for  $j = N + 1,N + 2\dots ,k$ .
- We first check if  $x_{j}$  is  $\zeta$ -dependent on  $B_{i}$  for all  $i \in [N]$ . If so, we are done.
- Otherwise, pick an arbitrary  $i \in [N]$  for which  $x_{j}$  is  $\zeta$ -independent of  $B_{i}$  and append  $(x_{j},y_{j})$  to  $B_{i}$ , i.e.,  $B_{i} \gets B_{i} \cup \{(x_{j},y_{j})\}$ .
If we don't reach any  $j$  while running the above process for which the first statement above is satisfied, we should end up with  $\sum_{i=1}^{N}|B_i| = k \geq N \cdot \check{\mathfrak{E}}(\mathcal{F},\zeta;f^{\star})$ . We note that by construction  $|B_i| \leq \check{\mathfrak{E}}(\mathcal{F},\zeta;f^{\star})$  and thus  $|B_i| = \check{\mathfrak{E}}(\mathcal{F},\zeta;f^{\star})$  for all  $i \in [N]$ , which implies  $x_k$  must be  $\zeta$ -dependent on all  $B_i$ .
Finally, let  $x_{i_1}, \ldots, x_{i_k}$  be the subsequence where, for all  $s \in [k]$ , there exists  $f \in \mathcal{F}_{i_s}$  such that  $|f(x_{i_s})[y_{i_s}] - f^{\star}(x_{i_s})[y_{i_s}]| \geq \zeta$ . By our first claim we know each element of this subsequence is  $\zeta$ -dependent on at most  $\beta^2 / \zeta^2$  disjoint subsequences. By the second claim, we know that there exists an element that is  $\zeta$ -dependent on at least  $\lfloor k / \check{\mathfrak{E}}(\mathcal{F}, \zeta; f^{\star}) \rfloor$  disjoint subsequences. So we must have  $\lfloor k / \check{\mathfrak{E}}(\mathcal{F}, \zeta; f^{\star}) \rfloor \leq \beta^2 / \zeta^2$ . Hence,  $k \leq (\beta^2 / \zeta^2 + 1) \cdot \check{\mathfrak{E}}(\mathcal{F}, \zeta; f^{\star})$ .
The following is an extension of Lemma 7 that holds for the normed version of eluder dimension given in Definition 1. The proof is essentially the same so we skip it for conciseness.
Lemma 8. Let  $\{x_{t},Z_{t}\}_{t = 1}^{T}$  be sequence of tuples, where  $x_{t}\in \mathcal{X}$  and  $Z_{t}\in \{0,1\}$ . Fix any  $f^{\star}\in \mathcal{F}$  and define the set  $\mathcal{F}_t = \{f\in \mathcal{F}\mid \sum_{s = 1}^{t - 1}Z_s\| f(x_s) - f^{\star}(x_s)\| ^2\leq \beta^2\}$ . Then, for any  $\zeta >0$
$$
\sum_ {t = 1} ^ {T} Z _ {t} \mathbf {1} \left\{\exists f ^ {\prime} \in \mathcal {F} _ {t}: \| f ^ {\prime} (x _ {t}) - f ^ {\star} (x _ {t}) \| \geq \zeta \right\} \leq \left(\frac {\beta^ {2}}{\zeta^ {2}} + 1\right) \mathfrak {E} (\mathcal {F}, \zeta ; f ^ {\star}).
$$
E Selective Sampling: Learning from Single Expert
E.1 Comparison to Related Works
E.2 Proof Sketch for Selective Sampling and Binary Labels
Let  $\mathcal{A} = \{1,2\}$ , and the link function  $\phi (z) = z$  corresponding to square-loss  $\ell_{\phi} = (v - y)^2 /2$ ; here  $\lambda = \gamma = 1$ .
Let  $\bar{\mathcal{F}}\subseteq \{\mathcal{X}\mapsto [-1,1]\}$  be a function class, and  $\bar{f}^{\star}\in \mathcal{F}$ . We assume that for any context  $x$ , the label  $y$  is drawn according to the distribution  $\operatorname *{Pr}(y_t = 2) = 1 + \bar{f}^\star (x) / 2$ . Using  $\bar{\mathcal{F}}$ , we can define the score function class  $\mathcal{F} = \{f_{\bar{f}}\mid \bar{f}\in \bar{\mathcal{F}}\}$  where  $f(x) = \frac{1}{2} (1 - \bar{f} (x),1 + \bar{f} (x))^{\top}\in [0,1]^{2}$ , and additionally define  $f^{\star} = f_{\bar{f}^{\star}}$ . Clearly, the Bayes optimal predictor that chooses the action with the largest score is given by SelectAction  $(f^{\star}(x)) = 1 + \mathrm{sign}(\bar{f}^{\star}(x))$ . Furthermore,  $\mathrm{Margin}(f^{\star}(x))\coloneqq |\mathrm{Pr}(y = 2|x) - \mathrm{Pr}(y = 1|x)| = |\bar{f}^{\star}(x)|$  which implies that  $T_{\varepsilon} = \sum_{t = 1}^{T}\mathbf{1}\{|f^{\star}(x_{t})|\leq \varepsilon \}$ . Finally, the oracle in (3) reduces to a square-loss online regression oracle, which implies that with probability at least  $1 - \delta$ , for all  $t\le T$ ,
$$
\sum_ {s = 1} ^ {t} Z _ {s} \left(\bar {f} _ {s} \left(x _ {s}\right) - \bar {f} ^ {\star} \left(x _ {s}\right)\right) ^ {2} \lesssim \sum_ {s = 1} ^ {t} Z _ {s} \left(\bar {f} _ {s} \left(x _ {s}\right) - y _ {s}\right) ^ {2} - \sum_ {s = 1} ^ {t} Z _ {s} \left(f ^ {\star} \left(x _ {s}\right) - y _ {s}\right) ^ {2} \lesssim \operatorname {R e g} ^ {\mathrm {s q}} (\bar {\mathcal {F}}; T) + \log (T / \delta), \tag {10}
$$
The above implies that  $\bar{f}^{\star}$  satisfies the constraints in (5) with the right choice of constants,  $\lambda$ , and  $\gamma$ , and thus  $|\bar{f}_t(x_t) - \bar{f}^\star (x_t)|\leq \Delta_t(x_t)$  (see Lemma 10 for proof). However, since the query condition in Algorithm 1 is  $Z_{t} = \mathbf{1}\{|f_{t}(x_{t})|\leq \Delta_{t}(x_{t})\}$ , we have that if  $Z_{t} = 0$ , then  $|\bar{f}_t(x_t)| > \Delta_t(x_t)$  which implies that  $\mathrm{sign}(\bar{f}^{\star}(x_t)) = \mathrm{sign}(\bar{f}_t(x_t))$ . Thus,
$$
\sum_ {s = 1} ^ {t} \bar {Z} _ {s} \mathbf {1} \left\{\operatorname {s i g n} \left(\bar {f} ^ {\star} \left(x _ {t}\right)\right) \neq \operatorname {s i g n} \left(\bar {f} _ {t} \left(x _ {t}\right)\right) \right\} = 0. \tag {11}
$$
Regret bound. Using the fact that  $y_{t} \sim 1 + \mathrm{Ber}(1 + \bar{f}^{*}(x_{t}) / 2)$ ,  $\widehat{y}_{t} = \text{SelectAction}(f_{t}(x_{t})) = 1 + \mathrm{sign}(\bar{f}_{t}(x_{t}))$ , we have
$$
\begin{array}{l} \operatorname {R e g} _ {T} = \sum_ {t = 1} ^ {T} \Pr \left(\widehat {y} _ {t} \neq y _ {t}\right) - \Pr \left(\text {S e l e c t A c t i o n} \left(f ^ {\star} \left(x _ {t}\right)\right) \neq y _ {t}\right) \\ \leq \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\operatorname {s i g n} \left(\bar {f} _ {t} \left(x _ {t}\right)\right) \neq \operatorname {s i g n} \left(\bar {f} ^ {\star} \left(x _ {t}\right)\right) \right\} \cdot | 2 \Pr \left(y _ {t} = 1\right) - 1 | \\ = \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\operatorname {s i g n} \left(\bar {f} _ {t} \left(x _ {t}\right)\right) \neq \operatorname {s i g n} \left(\bar {f} ^ {\star} \left(x _ {t}\right)\right) \right\} \cdot | \bar {f} ^ {\star} \left(x _ {t}\right) | \\ \end{array}
$$
The right hand side above can be split and upper bound via the following three terms:
$$
\operatorname {R e g} _ {T} \leq \varepsilon \sum_ {t = 1} ^ {T} \mathbf {1} \left\{\left| \bar {f} ^ {\star} \left(x _ {t}\right) \right| \leq \varepsilon \right\} + \sum_ {t = 1} ^ {T} Z _ {t} \mathbf {1} \left\{\operatorname {s i g n} \left(\bar {f} _ {t} \left(x _ {t}\right)\right) \neq \operatorname {s i g n} \left(\bar {f} ^ {\star} \left(x _ {t}\right)\right), \left| \bar {f} ^ {\star} \left(x _ {t}\right) \right| > \varepsilon \right\} \cdot | \bar {f} ^ {\star} \left(x _ {t}\right) |
$$
$$
\begin{array}{l} + \sum_ {t = 1} ^ {T} \bar {Z} _ {t} \mathbf {1} \left\{\operatorname {s i g n} \left(\bar {f} _ {t} \left(x _ {t}\right)\right) \neq \operatorname {s i g n} \left(\bar {f} ^ {\star} \left(x _ {t}\right)\right) \right\} \cdot | \bar {f} ^ {\star} \left(x _ {t}\right). \\ = \varepsilon T _ {\varepsilon} + \underbrace {\sum_ {t = 1} ^ {T} Z _ {t} \mathbf {1} \left\{\operatorname {s i g n} \left(\bar {f} _ {t} \left(x _ {t}\right)\right) \neq \operatorname {s i g n} \left(\bar {f} ^ {\star} \left(x _ {t}\right)\right) , | \bar {f} ^ {\star} \left(x _ {t}\right) | > \varepsilon \right\} \cdot | \bar {f} ^ {\star} \left(x _ {t}\right) |} _ {:= \mathrm {T} _ {A}}, \\ \end{array}
$$
where the first term is  $T_{\varepsilon}$ , and the last term is zero due to (11). The term  $\mathrm{T}_A$  denotes the regret for the rounds in which the learner queries for the label, and the margin for  $\tilde{f}^{\star}(x_t)$  is larger than  $\varepsilon$ . We note that