SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbb {E} \big [ K _ {i} ^ {2} \mid \{z _ {\ell}, \widehat {\rho} _ {\ell} \} _ {\ell = 1} ^ {i} \big ] \leq \beta^ {2} \mathbb {E} \Big [ \big (\widehat {\rho} _ {i} - \rho (z _ {i} \mid z ^ {*}) \big) ^ {2} \mid \{z _ {\ell}, \widehat {\rho} _ {\ell} \} _ {\ell = 1} ^ {i} \Big ] = \beta^ {2} \big (\widehat {\rho} _ {i} - \rho (z _ {i} \mid z ^ {*}) \big) ^ {2}.
$$
Then, by plugging this into Freedman's inequality (Lemma 16) with  $\eta \coloneqq \frac{1}{4}\min (1 / \beta^2,1 / \beta \beta ')$  and  $\delta^{\prime}\coloneqq \delta /T$ , we get that for any fixed  $t\in [T]$ , with probability at least  $1 - \delta$
$$
\sum_ {i = 1} ^ {t} \xi_ {i} \left(\widehat {\rho_ {i}} - \rho \left(z _ {i} \mid z ^ {*}\right)\right) \leq \frac {1}{4} \sum_ {i = 1} ^ {t} \left(\widehat {\rho_ {i}} - \rho \left(z _ {i} \mid z ^ {*}\right)\right) ^ {2} + 4 \beta \max  \left\{\beta , \beta^ {\prime} \right\} \ln \left(\frac {T}{\delta}\right). \tag {16}
$$
Plugging Eq. (16) back into Eq. (15) and rearranging terms yields
$$
\sum_ {i = 1} ^ {t} \left(\widehat {\rho} _ {i} - \rho \left(z _ {i} \mid z ^ {*}\right)\right) ^ {2} \leq 2 \operatorname {R e g r e t} _ {\mathcal {R} e g} (t) + 1 6 \beta \max  \left\{\beta , \beta^ {\prime} \right\} \ln \left(\frac {T}{\delta}\right) \tag {17}
$$
with probability at least  $1 - \delta$  for all  $t\in [T]$ . Thus we conclude that with probability at least  $1 - \delta$ , for all  $t\in [T]$
$$
z ^ {*} \in \left\{z \in \mathcal {Z} _ {\alpha}: \sum_ {i = 1} ^ {t} \left(\rho \left(z _ {i} \mid z\right) - \widehat {\rho_ {i}}\right) ^ {2} \leq C _ {T, \delta} \right\}.
$$
By the triangle inequality,
$$
\sqrt {\sum_ {i = 1} ^ {t} (\rho (z _ {i} \mid z ^ {*}) - \rho (z _ {i} \mid z _ {t + 1})) ^ {2}} \leq \sqrt {\sum_ {i = 1} ^ {t} (\rho (z _ {i} \mid z ^ {*}) - \widehat {\rho} _ {i}) ^ {2}} + \sqrt {\sum_ {i = 1} ^ {t} (\widehat {\rho} _ {i} - \rho (z _ {i} \mid z _ {t + 1})) ^ {2}}.
$$
Since by definition  $z_{t + 1} = \operatorname{argmin}_{z\in \mathcal{Z}_{\alpha}}\sum_{i = 1}^{t}\left(\rho (z_{i}\mid z) - \widehat{\rho_{i}}\right)^{2}$ , we have
$$
\sum_ {i = 1} ^ {t} \left(\rho (z _ {i} \mid z _ {t + 1}) - \widehat {\rho} _ {i}\right) ^ {2} \leq \sum_ {i = 1} ^ {t} \left(\rho (z _ {i} \mid z ^ {*}) - \widehat {\rho} _ {i}\right) ^ {2}.
$$
Substituting back into the triangle inequality above,
$$
\sqrt {\sum_ {i = 1} ^ {t} \left(\rho \left(z _ {i} \mid z ^ {*}\right) - \rho \left(z _ {i} \mid z _ {t + 1}\right)\right) ^ {2}} \leq 2 \sqrt {\sum_ {i = 1} ^ {t} \left(\rho \left(z _ {i} \mid z ^ {*}\right) - \widehat {\rho_ {i}}\right) ^ {2}},
$$
$$
\sum_ {i = 1} ^ {t} \left(\rho \left(z _ {i} \mid z ^ {*}\right) - \rho \left(z _ {i} \mid z _ {t + 1}\right)\right) ^ {2} \leq 4 \sum_ {i = 1} ^ {t} \left(\rho \left(z _ {i} \mid z ^ {*}\right) - \widehat {\rho} _ {i}\right) ^ {2}.
$$
Plugging Eq. (17) on the right-hand side yields
$$
\sum_ {i = 1} ^ {t} \left(\rho \left(z _ {i} \mid z ^ {*}\right) - \rho \left(z _ {i} \mid z _ {t + 1}\right)\right) ^ {2} \leq 8 \operatorname {R e g r e t} _ {\mathcal {R} e g} (t) + 6 4 \beta \max  \left\{\beta , \beta^ {\prime} \right\} \ln \left(\frac {T}{\delta}\right).
$$
The result follows by using the monotonicity of  $\mathrm{Regret}_{\mathcal{R}eg}(t)$ .
A.4 Proof of Lemma 5
The proof uses Turán's Theorem [26], a standard result from extremal graph theory that bounds the number of edges of a graph that does not contain a clique of a given size:
Theorem 21 (Turán's Theorem). Let  $G = (V, E)$  be an undirected graph without self-loops and whose largest clique is of size at most  $d$ . Then
$$
| E | \leq \left(1 - \frac {1}{d}\right) \frac {| V | ^ {2}}{2}.
$$
We now turn to the proof of Lemma 5. First note that if  $\epsilon \geq 1 + \alpha$  then no query is  $\epsilon$ -bad, because  $\alpha - \epsilon \leq -1 \leq \rho(z \mid z^{*})$  for every  $z \in \mathcal{Z}$ , and therefore the lemma holds. In the remainder of the proof, we assume that  $0 < \epsilon < 1 + \alpha$ .
Consider the queries  $z_{1},\ldots ,z_{T}$  and their corresponding values relative to  $z^{*}$ , denoted as  $v_{t} = \rho (z_{t}|z^{*})$  for  $t\in [T]$ . A query  $z_{t}$  is  $\epsilon$ -bad if its corresponding value  $v_{t}$  is in the interval  $I = [-1,\alpha -\epsilon)$ . The proof proceeds by partitioning the interval  $I$  into subintervals and separately bounding the number of values  $v_{t}$  in each subinterval.
To define these subintervals, let  $q = 1 + \frac{1}{\sqrt{d}}$ , and consider the sequence of suboptimality gaps  $\epsilon_{i} = q^{i - 1}\epsilon$  for  $i = 1,\dots ,n + 1$ , where
$$
n = \left\lceil \log_ {q} \left(\frac {1 + \alpha}{\epsilon}\right)\right\rceil .
$$
The gaps  $\epsilon_{i}$  form an increasing sequence  $\epsilon, q\epsilon, q^{2}\epsilon, \ldots$  such that the last element satisfies
$$
\epsilon_ {n + 1} = q ^ {n} \epsilon \geq \left(\frac {1 + \alpha}{\epsilon}\right) \epsilon = 1 + \alpha .
$$
Using these gaps we define intervals  $I_{i} = [\alpha - \epsilon_{i + 1}, \alpha - \epsilon_{i})$  for  $i = 1, \dots, n$ . Since  $\epsilon_{n + 1} \geq 1 + \alpha$ , the union  $I_{1} \cup \dots \cup I_{n} = [\alpha - \epsilon_{n + 1}, \alpha - \epsilon)$  covers the interval  $I$ . We bound the number of values  $v_{t}$  in each interval  $I_{i}$ .
Let  $S_{i}$  be the set of query indices with values in  $I_{i}$ , that is  $S_{i} = \{t \in [T] : \rho(z_{t} \mid z^{*}) \in I_{i}\}$ , let  $m_{i} = |S_{i}|$ , and assume that  $m_{i} \geq 2$  (the case  $m_{i} \leq 1$  will be dealt with later). Furthermore, let  $c_{i} = \alpha - (\epsilon_{i+1} + \epsilon_{i}) / 2$  be the midpoint of the interval  $I_{i}$ . Since the width of the interval  $I_{i}$  is  $\epsilon_{i+1} - \epsilon_{i} = (q-1)\epsilon_{i} = \epsilon_{i} / \sqrt{d}$ , we obtain
$$
\left| \rho \left(z _ {t} \mid z ^ {*}\right) - c _ {i} \right| \leq \frac {\epsilon_ {i}}{2 \sqrt {d}} \tag {18}
$$
for all  $t\in S_i$
Let  $d_{i} = \mathsf{d}_{\rho}(\mathcal{Z},\alpha ,\epsilon_{i})$  be the (non-monotonic) dissimilarity dimension with respect to the suboptimality gap  $\epsilon_{i}$ . Since  $\alpha \leq \alpha^{*}$  and  $\epsilon_{i}\geq \epsilon$ , we have  $1\leq d_{i}\leq d$ . We construct an upper bound on  $m_{i}$  exploiting the fact that  $d_{i}$  is the dissimilarity dimension with respect to  $\epsilon_{i}$ .
In the rest of the proof we refer to a pair of queries with indices  $s, t \in S_i$  such that  $s < t$  as dissimilar if
$$
\left| \rho (z _ {s} \mid z _ {t}) - c _ {i} \right| \leq \frac {\epsilon_ {i}}{\sqrt {d _ {i}}}.
$$
This is exactly the property appearing in the definition of the dissimilarity dimension with respect to  $\epsilon_{i}$ , and so there cannot be more than  $d_{i}$  queries such that every pair is dissimilar (note that all the queries  $z_{t}$  satisfy  $\rho(z_{t} \mid z_{t}) \geq \alpha$  thanks to  $\alpha$ -large self-evaluations).
The derivation of the bound on  $m_i$  proceeds in several steps. First, we identify pairs of dissimilar queries and construct a graph where each edge corresponds to a dissimilar pair. Second, we use Turán's Theorem (Theorem 21) to upper bound the number of such pairs, using the fact that the graph cannot contain a clique of size greater than  $d_i$ . Finally, using the bound on the number of dissimilar pairs, we bound  $m_i$ .
To start, let  $t_1 < t_2 < \ldots < t_{m_i}$  be the query indices included in  $S_i$ , and let  $k \in \{2, \ldots, m_i\}$ . Consider a uniform distribution over  $\ell \in [k - 1]$ . Then by Markov's inequality and the decaying estimation error property, we obtain
$$
\begin{array}{l} \frac {1}{k - 1} \sum_ {\ell = 1} ^ {k - 1} \mathbf {1} \left[ \left(\rho \left(z _ {t _ {\ell}} \mid z _ {t _ {k}}\right) - \rho \left(z _ {t _ {\ell}} \mid z ^ {*}\right)\right) ^ {2} \geq \frac {\epsilon_ {i} ^ {2}}{4 d} \right] \leq \left[ \frac {1}{k - 1} \sum_ {\ell = 1} ^ {k - 1} \left(\rho \left(z _ {t _ {\ell}} \mid z _ {t _ {k}}\right) - \rho \left(z _ {t _ {\ell}} \mid z ^ {*}\right)\right) ^ {2} \right] \cdot \frac {4 d}{\epsilon_ {i} ^ {2}} \\ \leq \left[ \frac {1}{k - 1} \sum_ {s = 1} ^ {t _ {k} - 1} \left(\rho \left(z _ {s} \mid z _ {t _ {k}}\right) - \rho \left(z _ {s} \mid z ^ {*}\right)\right) ^ {2} \right] \cdot \frac {4 d}{\epsilon_ {i} ^ {2}} \\ \leq \frac {C _ {T , \delta}}{k - 1} \cdot \frac {4 d}{\epsilon_ {i} ^ {2}}. \\ \end{array}
$$
Multiplying by  $k - 1$ , we therefore obtain
$$
\left| \left\{s \in S _ {i}: s <   t _ {k} \mathrm {a n d} \left| \rho (z _ {s} \mid z _ {t _ {k}}) - \rho (z _ {s} \mid z ^ {*}) \right| \geq \frac {\epsilon_ {i}}{2 \sqrt {d}} \right\} \right| \leq \frac {4 d C _ {T , \delta}}{\epsilon_ {i} ^ {2}},
$$
and summing across all  $k \in \{2, \ldots, m_i\}$  then yields
$$
\left| \left\{s, t \in S _ {i}: s <   t \text {a n d} \left| \rho \left(z _ {s} \mid z _ {t}\right) - \rho \left(z _ {s} \mid z ^ {*}\right) \right| \geq \frac {\epsilon_ {i}}{2 \sqrt {d}} \right\} \right| \leq m _ {i} \frac {4 d C _ {T , \delta}}{\epsilon_ {i} ^ {2}}. \tag {19}
$$
We next construct an undirected graph without self-loops,  $G_{i} = (V_{i},E_{i})$ . The vertex set of the graph is  $V_{i} = S_{i}$ . The edge set is defined to be
$$
E _ {i} = \left\{\{t, s \} \subseteq S _ {i}: s <   t \text {a n d} \left| \rho \left(z _ {s} \mid z _ {t}\right) - \rho \left(z _ {s} \mid z ^ {*}\right) \right| <   \frac {\epsilon_ {i}}{2 \sqrt {d}} \right\}.
$$
By comparing with Eq. (19), we obtain
$$
\left| E _ {i} \right| \geq \frac {m _ {i} \left(m _ {i} - 1\right)}{2} - m _ {i} \frac {4 d C _ {T , \delta}}{\epsilon_ {i} ^ {2}}. \tag {20}
$$
Note that any pair of vertices  $s < t$  connected by an edge corresponds to a dissimilar pair of queries:
$$
\begin{array}{l} \left| \rho \left(z _ {s} \mid z _ {t}\right) - c _ {i} \right| \leq \left| \rho \left(z _ {s} \mid z _ {t}\right) - \rho \left(z _ {s} \mid z ^ {*}\right) \right| + \left| \rho \left(z _ {s} \mid z ^ {*}\right) - c _ {i} \right| \\ <   \frac {\epsilon_ {i}}{2 \sqrt {d}} + \frac {\epsilon_ {i}}{2 \sqrt {d}} \\ \leq \frac {\epsilon_ {i}}{\sqrt {d _ {i}}}, \\ \end{array}
$$
where the first inequality is the triangular inequality, the second inequality follows by combining the definition of  $E_{i}$  and Eq. (18), and the final one is from the fact that  $d_{i}\leq d$ . From the definition of the dissimilarity coefficient, the largest clique in  $G_{i}$  is of size at most  $d_{i}$ . Using Turán's Theorem, we thus must have
$$
| E _ {i} | \leq \left(1 - \frac {1}{d _ {i}}\right) \cdot \frac {m _ {i} ^ {2}}{2} \leq \left(1 - \frac {1}{d}\right) \cdot \frac {m _ {i} ^ {2}}{2}.
$$
Combining with the lower bound on  $|E_i|$  from Eq. (20), we obtain
$$
\frac {m _ {i} (m _ {i} - 1)}{2} - m _ {i} \frac {4 d C _ {T , \delta}}{\epsilon_ {i} ^ {2}} \leq \left(1 - \frac {1}{d}\right) \cdot \frac {m _ {i} ^ {2}}{2}.
$$
Dividing by  $m_i$ , multiplying by  $2d$ , and rearranging then yields
$$
m _ {i} \leq 2 d \left(\frac {1}{2} + \frac {4 d C _ {T , \delta}}{\epsilon_ {i} ^ {2}}\right) = d + \frac {8 d ^ {2} C _ {T , \delta}}{\epsilon_ {i} ^ {2}}.
$$
We have originally assumed that  $m_i \geq 2$ , but the bound that we have just derived also holds when  $m_i \leq 1$  (because  $d \geq 1$ ).
To complete the proof it suffices to sum up the upper bounds on  $m_{i}$  across  $i = 1,\dots ,n$  ..
$$
\begin{array}{l} \sum_ {i = 1} ^ {n} m _ {i} = \sum_ {i = 1} ^ {n} \left[ d + \frac {8 d ^ {2} C _ {T , \delta}}{\epsilon^ {2}} \cdot \left(1 / q ^ {2}\right) ^ {i - 1} \right] \\ \leq n d + \frac {8 d ^ {2} C _ {T , \delta}}{\epsilon^ {2}} \cdot \frac {1}{1 - \left(1 / q ^ {2}\right)}. \tag {21} \\ \end{array}
$$
To bound  $n$ , we use the fact that  $\alpha \leq 1$ , the inequality  $\ln (1 + x) \geq \frac{x}{1 + x}$  (which holds for  $x \geq 0$ ) and the fact that  $d \geq 1$ :
$$
\begin{array}{l} n \leq 1 + \log_ {q} (2 / \epsilon) \\ = 1 + \frac {\ln (2 / \epsilon)}{\ln \left(1 + \frac {1}{\sqrt {d}}\right)} \leq 1 + [ \ln (2 / \epsilon) ] \cdot \frac {1 + \frac {1}{\sqrt {d}}}{\frac {1}{\sqrt {d}}} = 1 + (\sqrt {d} + 1) \ln (2 / \epsilon) \\ \leq 2 \ln 2 + 2 \sqrt {d} \ln (2 / \epsilon) \leq 2 \sqrt {d} \ln (4 / \epsilon). \\ \end{array}
$$
$$
1 - \frac {1}{q ^ {2}} = 1 - \frac {1}{1 + \frac {2}{\sqrt {d}} + \frac {1}{d}} \geq 1 - \frac {1}{1 + \frac {2}{\sqrt {d}}} = \frac {\frac {2}{\sqrt {d}}}{1 + \frac {2}{\sqrt {d}}} = \frac {2}{\sqrt {d} + 2} \geq \frac {2}{3 \sqrt {d}}.
$$
Plugging these back in Eq. (21) yields
$$
\sum_ {i = 1} ^ {n} m _ {i} \leq 2 d ^ {1. 5} \ln (4 / \epsilon) + \frac {1 2 d ^ {2 . 5} C _ {T , \delta}}{\epsilon^ {2}},
$$
completing the proof of the main claim of the lemma.
The second claim holds vacuously when  $T = 0$ , so assume that  $T \geq 1$ . If  $C_{T,\delta} \geq \ln(2T)$ , and using the fact that  $(\ln x) \leq x$  and  $2 \leq 3 \ln 2$ , we can write
$$
2 d ^ {1. 5} \ln (4 / \epsilon) = d ^ {1. 5} \ln (1 6 / \epsilon^ {2}) \leq \frac {1 6 d ^ {1 . 5}}{\epsilon^ {2}} \leq \frac {2 4 (\ln 2) d ^ {1 . 5}}{\epsilon^ {2}} \leq \frac {2 4 d ^ {2 . 5}}{\epsilon^ {2}} \cdot \ln (2 T) \leq \frac {2 4 d ^ {2 . 5} C _ {T , \delta}}{\epsilon^ {2}},
$$
which yields the first part of the second claim. The second part is immediate by plugging in  $C_{T,\delta} = 0$  in the main claim.
A.5 Useful lemmas
In this subsection we prove two lemmas that will be needed for the proofs of the main results (Theorems 6 and 7) in Appendices A.6 and A.7. They both rely on a standard technique of bounding a sum by a definite integral:
Proposition 22. Let  $f: \mathbb{R} \to \mathbb{R}$  be a non-increasing function and  $T \geq 1$ . Then
$$
\sum_ {t = 1} ^ {T} f (t) \leq f (1) + \int_ {1} ^ {T} f (t) \mathrm {d} t.
$$
Proof. The proof is immediate by noting that  $f(t) \leq \int_{t - 1}^{t} f(t) \, \mathrm{d}t$ .
In the lemmas below we write  $\mathbb{R}_+$  to denote  $[0, +\infty)$ .
Lemma 23. Let  $q_{1},\ldots ,q_{T}$  be a sequence in  $\mathbb{R}_+$ , and let  $\kappa :\mathbb{R}_{+}\to \mathbb{R}_{+}$  be a non-increasing function such that for all  $\epsilon >0$
$$
\sum_ {t = 1} ^ {T} \mathbf {1} (q _ {t} \geq \epsilon) \leq \frac {\kappa (\epsilon)}{\epsilon^ {2}}.
$$
Then, for any  $\tau \geq 0$
$$
\sum_ {t = 1} ^ {T} q _ {t} \leq T \tau + 2 \sqrt {\kappa (\tau) T}.
$$
Proof. First, since we are only concerned with bounding the sum  $\sum_{t} q_{t}$ , we assume without loss of generality that the sequence is in descending order, i.e.,  $q_{1} \geq \dots \geq q_{T}$ . Then, for any  $\tau \geq 0$ ,
$$
\sum_ {t = 1} ^ {T} q _ {t} = \sum_ {t = 1} ^ {T} q _ {t} \mathbf {1} \left(q _ {t} \leq \tau\right) + \sum_ {t = 1} ^ {T} q _ {t} \mathbf {1} \left(q _ {t} > \tau\right) \leq T \tau + \sum_ {t = 1} ^ {T} q _ {t} \mathbf {1} \left(q _ {t} > \tau\right). \tag {22}
$$
Consider any  $k$  such that  $q_{k} > \tau$ . Since the sequence  $q_{1}, \ldots, q_{T}$  is non-increasing, we have
$$
k \leq \sum_ {t = 1} ^ {T} \mathbf {1} (q _ {t} \geq q _ {k}) \leq \frac {\kappa (q _ {k})}{q _ {k} ^ {2}} \leq \frac {\kappa (\tau)}{q _ {k} ^ {2}},
$$
where the last inequality follows by the monotonicity of  $\kappa$ . This in turn implies that  $q_{k} \leq \sqrt{\frac{\kappa(\tau)}{k}}$ . Therefore,
$$
\sum_ {t = 1} ^ {T} q _ {t} \mathbf {1} \left(q _ {t} > \tau\right) \leq \sum_ {t = 1} ^ {T} \sqrt {\frac {\kappa (\tau)}{t}}. \tag {23}
$$
By Proposition 22,
$$
\sum_ {t = 1} ^ {T} \frac {1}{\sqrt {t}} \leq 1 + 2 \sqrt {T} - 2 \sqrt {1} <   2 \sqrt {T}. \tag {24}
$$
Combining Eqs. (22), (23) and (24), we get
Lemma 24. Let  $a > 0$ , let  $q_{1},\ldots ,q_{T}$  be a sequence of reals in  $[0,a]$ , and let  $\kappa :\mathbb{R}_{+}\to \mathbb{R}_{+}$  be a non-increasing function such that for all  $\epsilon \in (0,a]$
$$
\sum_ {t = 1} ^ {T} \mathbf {1} (q _ {t} \geq \epsilon) \leq \kappa (\epsilon) \ln \left(\frac {a}{\epsilon}\right).
$$
$$
\sum_ {t = 1} ^ {T} q _ {t} \leq T \tau + a [ 1 + \kappa (\tau) ] \exp \left(- \frac {1}{\kappa (\tau)}\right).
$$
Proof. We follow a similar proof strategy as in Lemma 23 and start by bounding the sum  $\sum_{t} q_{t}$ . We assume without loss of generality that the sequence is in descending order, i.e.,  $q_{1} \geq \dots \geq q_{T}$ . Then, for any  $\tau \geq 0$ ,
$$
\sum_ {t = 1} ^ {T} q _ {t} = \sum_ {t = 1} ^ {T} q _ {t} \mathbf {1} \left(q _ {t} \leq \tau\right) + \sum_ {t = 1} ^ {T} q _ {t} \mathbf {1} \left(q _ {t} > \tau\right) \leq T \tau + \sum_ {t = 1} ^ {T} q _ {t} \mathbf {1} \left(q _ {t} > \tau\right). \tag {25}
$$
Consider any  $k$  such that  $q_{k} > \tau$ . Then
$$
k \leq \sum_ {t = 1} ^ {T} \mathbf {1} \left(q _ {t} \geq q _ {k}\right) \leq \kappa \left(q _ {k}\right) \ln \left(\frac {a}{q _ {k}}\right) \leq \kappa (\tau) \ln \left(\frac {a}{q _ {k}}\right),
$$
where the last inequality follows by the monotonicity of  $\kappa$  and the fact that  $\ln (a / q_k) \geq 0$ . This in turn implies that  $q_k \leq a \exp \left(-\frac{k}{\kappa(\tau)}\right)$ . Therefore,
$$
\sum_ {t = 1} ^ {T} q _ {t} \mathbf {1} \left(q _ {t} > \tau\right) \leq \sum_ {t = 1} ^ {T} a \exp \left(- \frac {t}{\kappa (\tau)}\right). \tag {26}
$$
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \exp \left(- \frac {t}{\kappa (\tau)}\right) \leq \exp \left(- \frac {1}{\kappa (\tau)}\right) - \kappa (\tau) \left(\exp \left(- \frac {T}{\kappa (\tau)}\right) - \exp \left(- \frac {1}{\kappa (\tau)}\right)\right) \\ \leq \left[ 1 + \kappa (\tau) \right] \exp \left(- \frac {1}{\kappa (\tau)}\right). \tag {27} \\ \end{array}
$$
Combining Eqs. (25), (26) and (27), we get
$$
\sum_ {t = 1} ^ {T} q _ {t} \leq T \tau + a [ 1 + \kappa (\tau) ] \exp \left(- \frac {1}{\kappa (\tau)}\right),
$$
A.6 Proof of Theorem 6
Throughout the proof we use the shorthand  $d_{\epsilon} = \overline{\mathsf{d}}_{\rho}(\mathcal{Z},\alpha ,\epsilon)$ , so  $d = d_{1 / T}$ . The proof proceeds by applying Lemmas 23 and 24 to the bounds on the number of bad queries from Lemma 5. Specifically, let  $q_{t} = [\alpha -\rho (z_{t}|z^{*})]_{+}$  denote the suboptimality of each query  $z_{t}$  made by the algorithm. Then, for any  $\epsilon >0$ , the number of  $\epsilon$ -bad queries can be written as  $\sum_{t = 1}^{T}\mathbf{1}(q_t\geq \epsilon)$ .
First consider the case  $C_{T,\delta} \geq \ln(2T)$ . By Lemma 5, with probability at least  $1 - \delta$ , the number of  $\epsilon$ -bad queries is at most  $36d_{\epsilon}^{2.5}C_{T,\delta} / \epsilon^2$ . Setting  $\kappa(\epsilon) = 36d_{\epsilon}^{2.5}C_{T,\delta}$ , we apply Lemma 23, with  $\tau = 1 / T$ , to obtain that with probability at least  $1 - \delta$ ,
$$
\operatorname {R e g r e t} (T, \alpha) \leq \sum_ {t = 1} ^ {T} q _ {t} \leq 1 + 1 2 d ^ {1. 2 5} \sqrt {C _ {T , \delta} T}.
$$
If  $C_{T,\delta} = 0$ , then by Lemma 5, the number of  $\epsilon$ -bad queries is at most  $2d_{\epsilon}^{1.5}\ln (4 / \epsilon)$ . Setting  $a = 4$  and  $\kappa (\epsilon) = 2d_{\epsilon}^{1.5}$ , we apply Lemma 24, with  $\tau = 1 / T$ , to obtain
$$
\operatorname {R e g r e t} (T, \alpha) \leq \sum_ {t = 1} ^ {T} q _ {t} \leq 1 + 4 (1 + 2 d ^ {1. 5}) \exp \left(- \frac {1}{2 d ^ {1 . 5}}\right) \leq 1 + 1 2 d ^ {1. 5},
$$
A.7 Proof of Theorem 7
First, we consider the deterministic setting. By Lemma 5, at most  $2d^{1.5}\ln (4 / \epsilon)$  of queries issued by Alg are  $\epsilon$ -bad. Setting  $T > 2d^{1.5}\ln (4 / \epsilon)$  implies that at least one query is not  $\epsilon$ -bad. Thus, returning  $\hat{z}$  for which the observed reward is the largest guarantees that  $\rho (\hat{z} |z^{*})\geq \alpha -\epsilon$ , as needed.
Next, we prove the result for the case  $C_{T,\delta} \geq \ln(2T)$ . By Lemma 5, with probability at least  $1 - \delta / 2$ , there are at most  $\frac{16}{9\epsilon^2} \cdot 36d^{2.5}(C_{T,\delta / 2})$  queries that are  $3\epsilon / 4$ -bad. Setting  $T \geq 64d^{2.5}(C_{T,\delta / 2}) / \epsilon^2$ , implies that at least half of the queries are not  $3\epsilon / 4$ -bad. In the remainder of the proof, we only consider the high-probability event in which this is the case.
For  $n_1 = \lceil \log_2(4 / \delta) \rceil$  the probability that all  $n_1$  samples are  $3\epsilon / 4$ -bad is at most  $(1 / 2)^{n_1} \leq \delta / 4$ .
For  $n_2 = \lceil 128\ln (8n_1 / \delta) / \epsilon^2\rceil$  , by applying Hoeffding's inequality and union bound over each of the  $n_1$  rounds we get that with probability at most  $\delta /4$  there is some index  $\ell \leq n_{1}$  for which  $|\bar{r}_{t_{\ell}} - \rho (z_{t_{\ell}}|z^{*})| > \epsilon /8.$
Overall, with probability at least  $1 - \delta$  we get that there is at least one index  $j$  of the  $n_1$  sampled indices that is not  $3\epsilon /4$ -bad, and that  $|\bar{r}_{t_{\ell}} - \rho (z_{t_{\ell}}|z^{*})|\leq \epsilon /8$  for all  $\ell = 1,\ldots ,n_{1}$ . Therefore,
$$
\bar {r} _ {t _ {j}} \geq \rho \left(z _ {t _ {j}} \mid z ^ {*}\right) - \epsilon / 8 \geq \alpha - 3 \epsilon / 4 - \epsilon / 8 = \alpha - 7 \epsilon / 8.
$$
For all indices  $k$  that are  $\epsilon$ -bad we have
$$
\bar {r} _ {t _ {k}} \leq \rho \left(z _ {t _ {k}} \mid z ^ {*}\right) + \epsilon / 8 <   \alpha - \epsilon + \epsilon / 8 = \alpha - 7 \epsilon / 8.
$$
Thus, for all of the  $\epsilon$ -bad queries we have  $\bar{r}_{t_k} < \bar{r}_{t_j}$ , and so Algorithm 2 will not return any of the  $\epsilon$ -bad queries, because it is choosing the index with maximum value of  $\bar{r}_{t_\ell}$ . In other words, the returned query  $z_{t_{\hat{\ell}}}$  satisfies
$$
\rho (z _ {t _ {\ell}} \mid z ^ {*}) \geq \alpha - \epsilon .
$$
B Missing proofs of Section 4
First we discuss the connection between our SQ setting and the SQ model of Kearns [19]. We focus on two aspects in which they appear to differ and explain why these models are equivalent.
Correlational vs general statistical queries. The restriction of the SQ model in which the oracle may only output the approximate correlation between a query and the target function, termed correlational statistical query (CSQ), was studied by Bshouty and Feldman [7]. The CSQ oracle can be viewed as providing something akin to a negative distance between the query and the target. This is equivalent to the learning by distances framework of Ben-David et al. [5], who defined their model independently of Kearns [19]. Bshouty and Feldman [7] showed that an arbitrary statistical query can be answered by asking two SQs that are independent of the target and two CSQs. That is, in the distribution-dependent learning model (i.e., when the learner has access to the distribution over  $\mathcal{X}$ ), correlational queries can simulate general queries.
Adversarial vs statistical noise. The setting we consider in this work assumes stochastic query responses, similar to several previous works [13, 31, 4]. On the other hand, the original SQ model [19] assumed that the query oracle can respond with an adversarial (rather than statistical) noise, up to a pre-specified tolerance parameter  $\tau >0$ . The previous works have shown that the two noise models are equivalent [13, 31, 4]
B.1 Proof of Proposition 8
We first prove the first inequality of Eq. (3). Let  $\epsilon > 0$  and let  $d = d_{\mathrm{SQ}}(\epsilon)$ . Then there exists a sequence  $h_1, \ldots, h_d \in \mathcal{H}$  satisfying both conditions of Definition 4. Let  $d'$  be equal to the leftmost expression of Eq. (3). We aim to show  $d_\rho(\epsilon) \geq d'$ . Note that  $d' \leq d$ .
Let  $c$  be the midpoint between  $c_{min} = \min_{i < j}\langle h_i,h_j\rangle$  and  $c_{max} = \max_{i < j}\langle h_i,h_j\rangle$ . Then  $c\leq 1 - \epsilon$ . Moreover, for all  $i\neq j$
$$
| \langle h _ {i}, h _ {j} \rangle - c | \leq \frac {1}{2} | c _ {m a x} - c _ {m i n} | \leq \frac {1}{2 d} \leq \frac {\epsilon}{\sqrt {d ^ {\prime}}}
$$
where the last inequality follows from our choice of  $d'$  (which ensures  $d' \leq 4(d\epsilon)^2$ ). Thus,  $h_1, \ldots, h_{d'}$ , the first  $d'$  elements of the original sequence of hypotheses, satisfy Definition 1, proving the claim.