SlowGuess/NeurIPS_2023_OCR huggingface dataset:
It then remains to note that via our choice of exploration parameter and a union bound,
$$
2 \sum_ {t \in \mathbb {N}} \sum_ {k \in [ K ] \cup \{0 \}} \exp (- \beta (t, \delta)) \leq \delta ,
$$
and the result follows.
The following Lemma shows that, on rounds in which a point is removed from the active set, our estimate  $\widehat{p}$  remains within a constant factor of the true  $p$ .
Proposition A.2. On event  $\mathcal{E}$  we have that, for all rounds  $t$  such that a point is removed from the active set,  $2p/3 \leq \hat{p}_t \leq 4p/3$ .
Proof. Let  $t$  be a round in which a point is removed from the active set. On event  $\mathcal{E}$ , for all  $i \in [K]$
$$
\left| \mu_ {i} - \hat {\mu} _ {i} ^ {t} \right| \leq \Delta_ {(t)},
$$
we thus have  $|p - \hat{p}| \leq \Delta_{(t)}$  which, in combination with the fact  $\Delta_{(t)} \leq \hat{p} / 4$ , implies  $\Delta_{(t)} \leq (\hat{p} + \Delta_{(t)}) / 3$  and thus  $\Delta_{(t)} \leq p / 3$ .
For the purposes of the proof we split Theorem 3.2 into two parts. The first part is covered by the following lemma, which states that active-rank is  $\mathrm{PAC}(\varepsilon, \delta)$ .
Lemma A.3. For  $\varepsilon, \delta > 0$ ,  $1 / K \geq \varepsilon p$  on all problems  $\nu \in \mathcal{B}$ , on execution of active-rank, with output  $\hat{s}$ , we have that,
with probability greater than  $1 - \delta$ .
Proof. For the remainder of this proof we will work under the following event,
$$
\mathcal {E} = \bigcap_ {t \in \mathbb {N}} \bigcap_ {k \in [ K ]} \left\{\mu_ {k} \in [ \operatorname {L C B} (t, i), \operatorname {U C B} (t, i) ] \right\}.
$$
Proposition A.4. Let  $i \in [K]$  be removed from  $S_{t}$  at time  $t$ , such that  $\{j: |\mu_i - \mu_j| \leq \Delta_i\} \subset S_t$ . For all  $j: |\mu_i - \mu_j| \leq \Delta_j$ , we have that,  $\forall k \in S_t: |\mu_k - \mu_j| \geq \Delta_j$ ,
$$
\operatorname {s i g n} \left(\widehat {\mu_ {j}} ^ {\tau} - \widehat {\mu_ {k}} ^ {\tau}\right) = \operatorname {s i g n} \left(\mu_ {j} - \mu_ {k}\right).
$$
Proof. Firstly note that, for all  $s \geq t$ ,  $\Delta_{(s)} \leq \Delta_{(t)}$ . Thus for all  $j \in S_t$ ,  $|\widehat{\mu}_j^\tau - \mu_j| \leq \Delta_{(t)}$  and therefore for any point  $j: 4\Delta_{(t)} \leq \Delta_j$ , we have that, for all  $k \in S_t: |\mu_j - \mu_k| \leq \Delta_j$ ,
$$
\operatorname {s i g n} \left(\widehat {\mu} _ {j} ^ {\tau} - \widehat {\mu} _ {k} ^ {\tau}\right) = \operatorname {s i g n} \left(\mu_ {j} - \mu_ {k}\right).
$$
Thus to prove our result we must show that, for all  $j: |\mu_i - \mu_j| \leq \Delta_j$ ,  $\Delta_j \leq 4\Delta_{(t)}$ . Now, if  $i$  is removed from  $S_t$  at time  $t$ , we have that,
$$
\Delta_ {(t)} \leq \frac {1}{2 4} \left(\frac {K \varepsilon \widehat {p} _ {t}}{| U _ {i , t} (6 \Delta_ {(t)}) |} \wedge 1\right) \left(1 - \widehat {\mu} _ {i} ^ {t}\right),
$$
thus,
$$
\Delta_ {(t)} \leq \frac {1}{2 4} \left(1 - \widehat {\mu} _ {i} ^ {t}\right), \tag {8}
$$
and therefore via Proposition A.2,
$$
\left| U _ {t, i} \left(6 \Delta_ {(t)}\right) \right| \leq \frac {K \varepsilon \widehat {p} _ {t} \left(1 - \widehat {\mu} _ {i} ^ {t}\right)}{2 4 \Delta_ {(t)}} \leq \frac {K \varepsilon p \left(1 - \widehat {\mu} _ {i} ^ {t}\right)}{1 2 \Delta_ {(t)}} \leq \frac {K \varepsilon p \left(1 - \mu_ {i}\right)}{6 \Delta_ {(t)}},
$$
and we have that  $6\Delta_{(t)} \leq \Delta_i$ . Now let  $j: |\mu_j - \mu_i| \leq 4\Delta_{(t)}$ , we then have,  $|\widehat{\mu}_j - \widehat{\mu}_i| \leq 2\Delta_{(t)}$  and thus  $U_{j,t}(4\Delta_{(t)}) \subset U_{i,t}(6\Delta_{(t)})$  and furthermore  $\Delta_{(t)} \leq \frac{1}{12} (1 - \widehat{\mu}_j^t)$  therefore, via proposition A.2,
$$
\left| U _ {t, j} \left(4 \Delta_ {(t)}\right) \right| \leq \frac {K \varepsilon \widehat {p} _ {t} \left(1 - \widehat {\mu} _ {i} ^ {t}\right)}{2 4 \Delta_ {(t)}} \leq \frac {K \varepsilon p \left(1 - \widehat {\mu} _ {i} ^ {t}\right)}{1 2 \Delta_ {(t)}} \leq \frac {K \varepsilon p \left(1 - \mu_ {j}\right)}{6 \Delta_ {(t)}},
$$
and as a result  $\Delta_j \leq 4\Delta_{(t)}$ . Now take  $j: |\mu_j - \mu_i| \geq 4\Delta_{(t)}$ . If  $\Delta_j \leq 4\Delta_{(t)}$  we are done. Thus assume  $\Delta_j \geq 4\Delta_{(t)}$ . In this case, we have that  $j: |\mu_i - \mu_j| \geq \Delta_j$ . Thus we have shown that, for a point  $j \in S_t$ , if  $|\mu_i - \mu_j| \leq \Delta_j$ , then  $\Delta_{(t)} \leq \Delta_j / 4$  and the result follows.
Proposition A.5. At time  $t$ , let  $i \in S_t$ , we have that,  $\forall j \in [K] \setminus S_t: |\mu_i - \mu_j| \geq \Delta_i$ ,
$$
\operatorname {s i g n} \left(\widehat {\mu} _ {j} ^ {\tau} - \widehat {\mu} _ {i} ^ {\tau}\right) = \operatorname {s i g n} \left(\mu_ {j} - \mu_ {i}\right).
$$
and furthermore,  $\forall j:\{k:|\mu_j - \mu_k|\} \not\subset S_t,\forall k\in [K]:|\mu_k - \mu_j|\leq \Delta_j,$
Proof. The proof will follow by induction. Assume at time  $t$  that  $\forall i \in S_{t-1}$  we have that,  $\forall j \in [K] \setminus S_{t-1}: |\mu_i - \mu_j| \geq \Delta_i$ ,
$$
\operatorname {s i g n} \left(\widehat {\mu} _ {j} ^ {\tau} - \widehat {\mu} _ {i} ^ {\tau}\right) = \operatorname {s i g n} \left(\mu_ {j} - \mu_ {i}\right),
$$
and that,  $\forall j\in [K]:\{k:|\mu_j - \mu_k|\} \not\subset S_{t - 1},\forall k:|\mu_k - \mu_j|\leq \Delta_j,$
Now let  $\tilde{i} \in S_t$ , and  $j \in [K] \setminus S_t: |\mu_{\tilde{i}} - \mu_j| \geq \Delta_{\tilde{i}}$ , we must show that,
$$
\operatorname {s i g n} \left(\widehat {\mu} _ {j} ^ {\tau} - \widehat {\mu} _ {\tilde {i}} ^ {\tau}\right) = \operatorname {s i g n} \left(\mu_ {j} - \mu_ {\tilde {i}}\right),
$$
and that,  $\forall j\in [K]:\{k:|\mu_j - \mu_k|\} \not\subset S_t,\forall i:|\mu_k - \mu_j|\leq \Delta_j$
$$
\operatorname {s i g n} \left(\widehat {\mu} _ {j} ^ {\tau} - \widehat {\mu} _ {k} ^ {\tau}\right) = \operatorname {s i g n} \left(\mu_ {j} - \mu_ {i}\right).
$$
For the first statement, assume  $j \in S_{t-1}$  and that  $\{k : |\mu_j - \mu_k|\} \subset S_{t-1}$ , as otherwise we are done, via the inductive assumption. The point  $j$  is then removed from the active set at the end of round  $t+1$  and the statement thus follows via Proposition A.4.
For the second statement, for  $j \in S_{t-1}$  assume  $\{k : |\mu_j - \mu_k| \} \subset S_{t-1}$ , as otherwise we are done via the inductive assumption. Therefore, there must exist such an  $k$ , removed from the active set at the end of round  $t-1$ , such that  $|\mu_k - \mu_j| \leq \Delta_j$ . The result then follows via Proposition A.4.
For a set  $C\subset [0,1]$  , define,
$$
\kappa (C) := \frac {1}{\lambda (C)} \int_ {C} \eta (x), d x.
$$
Let  $\alpha \in [0,1]$ , define the subset  $Z_{\alpha} \subset [0,1]$  such that,  $\mathbb{P}(X \in Z_{\alpha}|Y = -1) = \alpha$ , that is,
$$
\frac {\lambda (Z _ {\alpha}) (1 - \kappa (Z _ {\alpha}))}{1 - p} = \alpha ,
$$
and such that, for some  $i_{\alpha}\in [K]$
$$
\forall j: \mu_ {j} > \mu_ {i _ {\alpha}}, P _ {j} \subset Z _ {\alpha}, \quad \forall j: \mu_ {j} <   \mu_ {i _ {\alpha}}, P _ {j} \cap Z _ {\alpha} = \emptyset . \tag {9}
$$
We then have  $\mathrm{ROC}^* (\alpha) = \mathbb{P}(X\in Z_\alpha |Y = +1) = \frac{\lambda(Z_\alpha)(\kappa(Z_\alpha))}{p}$ . The choice of  $Z_{\alpha}$  is not necessarily unique, and as  $\eta$  me be constant across multiple sections of the grid  $i_{\alpha}$  is also not necessarily unique, in this case we take arbitrary  $Z_{\alpha}$ ,  $i_{\alpha}$ . Now define the subset  $\hat{Z}_{\alpha}\in [0,1]$  such that,
$$
\frac {\lambda (\hat {Z} _ {\alpha}) (1 - \kappa (\hat {Z} _ {\alpha}))}{1 - p} = \alpha ,
$$
$$
\forall x \in \hat {Z} _ {\alpha}, y \notin \hat {Z} _ {\alpha}, s _ {\hat {P}} (x) \geq s _ {\hat {P}} (y),
$$
so  $\mathrm{ROC}(s,\alpha) = \frac{\lambda(\hat{Z}_{\alpha})(\kappa(\hat{Z}_{\alpha}))}{p}$ . Again  $\hat{Z}_{\alpha}$  is not necessarily unique, in which case we choose arbitrarily. Via Propositions A.4 and A.5, we have that,  $\forall j\in [K]:|\mu_j - \mu_{i_\alpha}|\leq \Delta_{i_\alpha}$ ,
$$
\operatorname {s i g n} \left(\widehat {\mu} _ {j} ^ {\tau} - \widehat {\mu} _ {i _ {\alpha}} ^ {\tau}\right) = \operatorname {s i g n} \left(\mu_ {j} - \mu_ {i _ {\alpha}}\right). \tag {10}
$$
$$
Z _ {\alpha} ^ {\prime} = \left\{x \in Z _ {\alpha}: | \eta (x) - \mu_ {i _ {\alpha}} | \leq \Delta_ {i _ {\alpha}} \right\}, \quad \hat {Z} _ {\alpha} ^ {\prime} = \left\{x \in \hat {Z} _ {\alpha}: | \eta (x) - \mu_ {i _ {\alpha}} | \leq \Delta_ {i _ {\alpha}} \right\}.
$$
Via Equation 10, we have that,
$$
\operatorname {R O C} (\alpha , \eta) - \operatorname {R O C} (\alpha , s _ {\hat {\mathcal {P}}}) = \frac {\lambda \left(Z _ {\alpha} ^ {\prime}\right) \kappa \left(Z _ {\alpha} ^ {\prime}\right)}{p} - \frac {\lambda \left(\hat {Z} _ {\alpha} ^ {\prime}\right) \kappa \left(\hat {Z} _ {\alpha} ^ {\prime}\right)}{p}.
$$
Before finalising the proof we must lower bound  $\lambda (\hat{Z}_{\alpha}^{\prime})$  and  $\kappa (\hat{Z}_{\alpha}^{\prime})$ . We first lower bound  $\kappa (\hat{Z}_{\alpha}^{\prime})$ .
$$
\kappa \left(\hat {Z} _ {\alpha} ^ {\prime}\right) \geq \mu_ {i _ {\alpha}} - \Delta_ {i _ {\alpha}}, \quad \kappa \left(Z _ {\alpha} ^ {\prime}\right), \leq \mu_ {i _ {\alpha}} + \Delta_ {i _ {\alpha}}. \tag {11}
$$
We will now lower bound  $\lambda (\hat{Z}_{\alpha}^{\prime})$
$$
\frac {\lambda \left(\hat {Z} _ {\alpha} ^ {\prime}\right)}{\lambda \left(Z _ {\alpha} ^ {\prime}\right)} = \frac {1 - \kappa \left(Z _ {\alpha} ^ {\prime}\right)}{1 - \kappa \left(\hat {Z} _ {\alpha} ^ {\prime}\right)} \leq \frac {1 - \mu_ {i _ {\alpha}} + \Delta_ {i _ {\alpha}}}{1 - \mu_ {i _ {\alpha}} - \Delta_ {i _ {\alpha}}} \tag {12}
$$
Via combinations of Equations (11) and (12), we have,
$$
\begin{array}{l} \frac {\lambda \left(Z _ {\alpha} ^ {\prime}\right) \kappa \left(Z _ {\alpha} ^ {\prime}\right)}{p} - \frac {\lambda \left(\hat {Z} _ {\alpha} ^ {\prime}\right) \kappa \left(\hat {Z} _ {\alpha} ^ {\prime}\right)}{p} \leq \frac {1}{p} \left(\lambda \left(Z _ {\alpha} ^ {\prime}\right) \left(\mu_ {i _ {\alpha}} + \Delta_ {i _ {\alpha}}\right) - \lambda \left(\hat {Z} _ {\alpha} ^ {\prime}\right) \left(\mu_ {i _ {\alpha}} - \Delta_ {i _ {\alpha}}\right)\right) (13) \\ \leq \frac {1}{p} \left(\lambda \left(Z _ {\alpha} ^ {\prime}\right) \left(\mu_ {i _ {\alpha}} + \Delta_ {i _ {\alpha}}\right) - \lambda \left(Z _ {\alpha} ^ {\prime}\right) \frac {\left(\mu_ {i _ {\alpha}} - \Delta_ {i _ {\alpha}}\right) \left(1 - \mu_ {i _ {\alpha}} + \Delta_ {i _ {\alpha}}\right)}{1 - \mu_ {i _ {\alpha}} - \Delta_ {i _ {\alpha}}}\right) (14) \\ \leq \frac {2 \lambda \left(Z _ {\alpha} ^ {\prime}\right) \Delta_ {i _ {\alpha}}}{p \left(1 - \mu_ {i _ {\alpha}} - \Delta_ {i _ {\alpha}}\right)} \leq \frac {2 \lambda \left(Z _ {\alpha} ^ {\prime}\right) \Delta_ {i _ {\alpha}}}{p \left(1 - \mu_ {i _ {\alpha}}\right)} (15) \\ \end{array}
$$
It remains to remark that,  $\Delta_{i_{\alpha}} \leq \frac{\varepsilon p}{\lambda(Z_{\alpha}')}$ , by definition, and thus,
$$
\operatorname {R O C} (\alpha , \eta) - \operatorname {R O C} \left(\alpha , s _ {\hat {\mathcal {P}}}\right) \leq \varepsilon .
$$
As we chose  $\alpha$  w.l.o.g the proof then follows.
A.1 Proof of stopping time for active-rank
We will now prove the second part of Theorem 3.2, the upper bound on the expected sampling time of active-rank.
Lemma A.6. For  $\varepsilon, \delta > 0, \gamma > 480 / \log(K), 1 / K \geq p\varepsilon$ , with  $\beta(t, \delta) = c_{\gamma} \log(t^2 K^2 / \delta)$  where  $c_{\gamma}$  is a constant depending only on  $\gamma$ , on all problems  $\nu \in \mathcal{B}$  such that  $\forall i \in [K], \Delta_i \leq 1 - \mu_i$ , we have that, for  $\gamma > 1$ , the expected stopping time of active-rank is upper bounded by the following,
$$
c _ {\gamma} ^ {\prime} \sum_ {i \in [ K ]} H _ {i} ^ {(2)} \log \left(c _ {\gamma} ^ {\prime \prime} H _ {i} ^ {(2)} K ^ {2} / \delta\right),
$$
where  $c_{\gamma}^{\prime}, c_{\gamma}^{\prime \prime}$  are constants depending only on  $\gamma$ .
Proof. We upper bound  $\mathbb{E}[\tau]$  as follows. We denote the number of times a section of the grid  $[G_i, G_{i+1})$  has been sampled by the learner, up to and including time  $t$  as,  $N_i(t)$ . Let  $\tau_i = N_i(\tau)$ .
$$
\begin{array}{l} \mathbb {E} \left[ \tau_ {i} \right] \leq \sum_ {t = 1} ^ {\infty} \mathbb {P} \left(\tau_ {i} \geq t\right) (16) \\ \leq \sum_ {t = 1} ^ {\infty} \mathbb {P} (i \notin \mathcal {Q} _ {t}). (17) \\ \end{array}
$$
Define the event,
$$
\begin{array}{l} \xi_ {i, t} := \{\forall j \in S _ {t}, (\mathrm {U C B} (t, j) \leq \mu_ {j} + \Delta_ {i} / 9 6) \cup \{\forall j \in S _ {t}, \mathrm {L C B} (t, j) \geq \mu_ {j} - \Delta_ {i} / 9 6) \} \\ \cup \left\{p - \Delta_ {(t)} \leq \widehat {p} _ {t} \leq p + \Delta_ {(t)} \right\}. \\ \end{array}
$$
Proposition A.7. For  $i \in [K]$  such that  $\Delta_i \leq 2p$ ,  $t > 0$ , we have that, on  $\xi_{i,t}$ ,  $i \in \mathcal{Q}_t$ .
Proof. First note that, under event  $\xi_{i,t}$ ,  $\Delta_{(t)} \leq \Delta_i / 96$  and thus for a  $j \in U_{i,t}(6\Delta_{(t)})$ , we have,
$$
\left| \mu_ {i} - \mu_ {j} \right| \leq \Delta_ {i} / 9 6 + \Delta_ {i} / 9 6 + 6 \Delta_ {i} / 9 6 \leq \Delta_ {i},
$$
$$
\left| U _ {i, t} \left(6 \Delta_ {(t)}\right) \right| \leq \left| \left\{j: \left| \mu_ {i} - \mu_ {j} \right| \leq \Delta_ {i} \right\} \right| \tag {18}
$$
and furthermore,  $\Delta_{(t)}\leq p / 4$  which implies  $\widehat{p}\leq 2p$  . Now, via definition of  $\Delta_i$
$$
\Delta_ {i} \leq \left(\frac {K p \varepsilon}{| \{j : | \mu_ {i} - \mu_ {j} | \leq \Delta_ {i} \} |} \wedge 1\right) (1 - \mu_ {i}), \tag {19}
$$
In the case,  $\frac{Kp\varepsilon}{|\{j:|\mu_i - \mu_j|\leq\Delta_i\}|}\geq 1,$
$$
\Delta_ {(t)} \leq \frac {\left(1 - \widehat {\mu} _ {i}\right) + \Delta_ {(t)}}{9 6} \leq \left(1 - \widehat {\mu} _ {i}\right) / 4 8
$$
and we have that
$$
\frac {K p \varepsilon}{| \{j : | \mu_ {i} - \mu_ {j} | \geq \Delta_ {i} \} |} \leq (1 + \Delta_ {(t)}) \frac {K \widehat {p} \varepsilon}{| U _ {i , t} (6 \Delta_ {(t)}) |} \leq \frac {3 K \widehat {p} \varepsilon}{2 | U _ {i , t} (6 \Delta_ {(t)}) |}
$$
$$
\frac {K \widehat {p} \varepsilon}{| U _ {i , t} (6 \Delta_ {(t)}) |} \geq \frac {2}{3}
$$
finally leading to,
$$
\Delta_ {(t)} \leq \frac {K \varepsilon \hat {p} _ {t} (1 - \hat {\mu} _ {i} ^ {t})}{2 4 | U _ {i , t} (6 \Delta_ {(t)}) |}.
$$
Then let us assume  $\frac{Kp\varepsilon}{|\{j:|\mu_i - \mu_j|\leq\Delta_i\}|}\leq 1$  . In this case,
$$
\Delta_ {(t)} \leq \Delta_ {i} / 9 6 \leq \frac {K \varepsilon p (1 - \mu_ {i})}{9 6 | U _ {i , t} (6 \Delta_ {(t)}) |} \leq \frac {K \varepsilon \widehat {p} ((1 - \widehat {\mu} _ {i} ^ {t}) + \Delta_ {(t)})}{4 8 | U _ {i , t} (6 \Delta_ {(t)}) |} \leq \frac {K \varepsilon \widehat {p} (1 - \widehat {\mu} _ {i} ^ {t})}{2 4 | U _ {i , t} (6 \Delta_ {(t)}) |},
$$
where the third inequality comes from the fact that  $\Delta_{(t)}\leq p / 4$  which implies  $\widehat{p}\leq 2p$
In what follows, assume  $i$  is such that  $\Delta_i \leq 2p$ . Via combination Proposition A.7 and Equation (17), we have that,
$$
\mathbb {E} [ \tau_ {i} ] \leq \sum_ {t = 1} ^ {\infty} \mathbb {P} \left(\xi_ {i, t} ^ {c}\right).
$$
We will now upper bound  $\sum_{t=1}^{\infty} \mathbb{P}(\xi_{i,t})$ . For  $\gamma > 1$  let,
$$
T _ {0} ^ {i} = \min  \left(t: \frac {\beta (t , \delta)}{t} \leq \min  _ {i \in [ K ]} \left(\frac {\mathrm {k l} (\mu_ {i} , \mu_ {i} + \Delta_ {i} / 9 6)}{\log (K) \gamma} \wedge \frac {\mathrm {k l} (\mu_ {i} , \mu_ {i} - \Delta_ {i} / 9 6)}{\log (K) \gamma}\right)\right).
$$
We have that, for all  $t > T_0^i$ ,  $j \in S_t$
$$
\mathbb {P} (\mathrm {U C B} (t, j) \geq \mu_ {j} + \Delta_ {i} / 9 6) \leq \mathbb {P} \left(\mathrm {k l} \left(\hat {\mu} _ {j} ^ {t}, \mu_ {j} + \Delta_ {i} / 9 6\right) \leq \frac {\mathrm {k l} \left(\mu_ {j} , \mu_ {j} + \Delta_ {i} / 9 6\right)}{\log (K) \gamma}\right). \tag {20}
$$
Let,
$$
r (\gamma) = \left\{x \in (\mu_ {j}, \mu_ {j} + \Delta_ {i} / 9 6): \mathrm {k l} (x, \mu_ {j} + \Delta_ {i} / 9 6) = \mathrm {k l} (\mu_ {j}, \mu_ {j} + \Delta_ {i} / 9 6) / (\log (K) \gamma) \right\}.
$$
Consider the function  $\phi(x) = \mathrm{kl}(\mu_j + x, \mu_j + \Delta_i / 96)$ , on the interval  $[0, \Delta_i / 96]$ . Via the properties of the KL divergence,  $\phi$  is convex and  $\phi(\Delta_i / 96) = 0$ . As a result,  $\phi(x) \leq (1 - x) \mathrm{kl}(\mu_j, \mu_j + \Delta_i / 96) / (2\Delta_i)$ , which implies,
$$
r (\gamma) \geq \mu_ {j} + \Delta_ {i} \left(\frac {1}{9 6} - \frac {2}{\log (K) \gamma}\right) \geq \mu_ {j} + \frac {\Delta_ {i}}{1 9 2},
$$
for  $\log (K)\gamma >384$  .We now have, for all  $t > T_0^i$
$$
\begin{array}{l} \mathbb {P} \left(\mathrm {k l} \left(\hat {\mu} _ {j} ^ {t}, \mu_ {j} + \Delta_ {i} / 9 6\right) \leq \frac {\mathrm {k l} \left(\mu_ {j} , \mu_ {j} + \Delta_ {i} / 9 6\right)}{\log (K) \gamma}\right) = \mathbb {P} \left(\mathrm {k l} \left(\hat {\mu} _ {j} ^ {t}, \mu_ {j} + \Delta_ {i} / 9 6\right) \leq \mathrm {k l} (r (\gamma), \mu_ {j} + \Delta_ {i} / 9 6)\right) \\ = \mathbb {P} \left(\hat {\mu} _ {j} ^ {t} \geq r (\gamma)\right) \\ \leq \exp (- t \operatorname {k l} (\mu_ {j}, r (\gamma))) \\ \leq \exp \left(- \frac {\log (K) \gamma \operatorname {k l} (\mu_ {j} , r (\gamma))}{\operatorname {k l} (\mu_ {j} , \mu_ {j} + \Delta_ {i} / 9 6)} \beta (t, \delta)\right) \\ \leq \exp \left(- \frac {\gamma \operatorname {k l} \left(\mu_ {j} , \mu_ {j} + \frac {\Delta_ {i}}{1 9 2}\right)}{\operatorname {k l} \left(\mu_ {j} , \mu_ {j} + \Delta_ {i} / 9 6\right)} \beta (t, \delta)\right) \\ \leq \exp (- \log (K) \beta (t, \delta) c _ {\gamma}) \\ \end{array}
$$
where  $c_{\gamma}$  is a constant depending only on  $\gamma$ . Thus, via Equation (20), for  $t \geq T_0^i$
$$
\mathbb {P} (\mathrm {U C B} (t, j) \geq \mu_ {j} + \Delta_ {i} / 2) \leq \exp (- \log (K) \beta (t, \delta) c _ {\gamma}),
$$
via similar reasoning we have also that,
$$
\mathbb {P} (\mathrm {L C B} (t, j) \leq \mu_ {j} - \Delta_ {i} / 2) \leq \exp (- \log (K) \beta (t, \delta) c _ {\gamma}),
$$
and furthermore,  $\mathbb{P}(\widehat{p_t}\notin [p - \Delta_{(t)},p + \Delta_{(t)}])\leq \exp (-\beta (t,\delta))$  , and now via a union bound,
$$
\sum_ {t = T _ {0} ^ {i} + 1} ^ {\infty} \mathbb {P} \left(\xi_ {t, i} ^ {c}\right) \leq c \exp \left(- c _ {\gamma}\right). \tag {21}
$$
where  $c > 0$  is an absolute constant. Thus we have shown that for a point  $i:\Delta_i\leq 2p$
$$
\mathbb {E} [ \tau_ {i} ] \leq T _ {0} ^ {i} + c K \exp (- c _ {\gamma}).
$$
It is then straight forward to note that for any point  $i:\Delta_i\geq 2p$
$$
\mathbb {E} \left[ \tau_ {i} \right] \leq T _ {0} ^ {i ^ {\prime}} + c K \exp \left(- c _ {\gamma}\right),
$$
where  $i'$  is any point  $i'$  such that  $\Delta_{i'} \leq 2p$ . It remains to upper bound  $T_0^i$ . Via definition of the exploration parameter, we have that,
$$
T _ {0} ^ {i} \leq H _ {i} ^ {(2)} \log \left(\tilde {c} _ {\gamma} H _ {i} ^ {(2)} K / \delta\right),
$$
where  $\tilde{c}_{\gamma} > 0$  is an absolute constant depending only on  $\gamma$ . Before continuing, we demonstrate the following Lemma-
Lemma A.8. On all problems  $\nu \in \mathcal{B}$ , we have that,
$$
\left| \left\{i: \Delta_ {i} <   2 p \right\} \right| \geq K / 4.
$$
Proof. Firstly assume that,
$$
\left| \left\{i: \Delta_ {i} \geq \mu_ {i} \right\} \right| \geq K / 4
$$
and define  $i^{*} = \max \{i:\Delta_{i}\geq \mu \}$ . We then have that,
$$
\forall j: \Delta_ {j} \geq \mu_ {j}, | \mu_ {i ^ {*}} - \mu_ {j} | \leq \Delta_ {i ^ {*}},
$$
and thus,
$$
\Delta_ {i ^ {*}} \leq \frac {K \varepsilon p}{| \{j : | \mu_ {i ^ {*}} - \mu_ {j} | \leq \Delta_ {i ^ {*}} \} |} \leq \varepsilon p / 4 \leq p.
$$
Thus we have  $|\{i:\Delta_i < p\} |\geq K / 4$  . Now assume,
$$
\left| \left\{i: \Delta_ {i} \geq \mu_ {i} \right\} \right| \leq K / 4.
$$
As we must have  $|\{i : \mu_i > 2p\}| \leq K/2$ , we thus have,  $|\{i : \Delta_i < 2p\}| \geq K/4$
With Lemma A.8, we can then upper bound  $\mathbb{E}[\tau ]$  as follows,
$$
\mathbb {E} [ \tau ] \leq 4 \sum_ {i: \Delta_ {i} \leq 2 p} \mathbb {E} [ \tau_ {i} ] \leq 4 \sum_ {i \in [ K ]} H _ {i} ^ {(2)} \log (\tilde {c} _ {\gamma} H _ {i} ^ {(2)} K / \delta) + c _ {\gamma} ^ {\prime} K \exp (- c _ {\gamma}),
$$
B Proof of Theorem 3.3
Proof. The proof will follow by application of a Fano type inequality on a well chosen set of problems.
Step 1: Constructing our well chosen set of problems We define a set of grid points  $U_0, U_1, \ldots$  recursively as follows,  $U_0 = \arg \min(\Delta_i)$ , for  $m \geq 0$  we then define
$$
U _ {m + 1} = \arg \min  \left\{\Delta_ {i}: \forall j \leq m, \left| \mu_ {U _ {j}} - \mu_ {i} \right| \geq 3 \Delta_ {i} + 3 \Delta_ {U _ {j}} \right\}.
$$
Let  $M$  be the largest integer for which  $U_{M}$  exists. Note that the sequence  $(\Delta_{U_m})_{m > M}$  is monotonically increasing and furthermore, for all  $i\in [K]$ ,
$$
\left| \left\{m \in [ M ]: \left| 3 \mu_ {U _ {m}} - 3 \mu_ {i} \right| \geq \Delta_ {U _ {m}} + \Delta_ {i} \right\} \right| \leq 2. \tag {22}
$$
We then define the corresponding set of groups,  $D_0, D_1, \ldots, D_M$  as follows. For  $i \in [K]$  let  $m, n \in [M]$  be such that,  $\mu_{U_m} \leq \mu_i \leq \mu_{U_n}$  then set  $i^+ = m \vee n$  and  $i^- = m \wedge n$ . If  $|\mu_i - \mu_{i^*}| \leq \Delta_i + \Delta_{i^+}$  then  $i \in D_{i^+}$ , otherwise  $i \in D_{i^-}$ .
Proposition B.1. For all  $m\in [M]$  we have that,  $\forall i\in D_m,\Delta_{U_m}\leq \Delta_i$
Proof. W.l.o.g assume  $\mu_{i} \geq \mu_{U_{m}}$ . Let  $j$  be such that  $|\mu_{U_{j}} - \mu_{i}| \leq 3\Delta_{i} + 3\Delta_{U_{j}}$ . Via Equation (22) we have that  $\mu_{U_{m}} \leq \mu_{i} \leq \mu_{U_{j}}$  and by definition of  $D_{m}, D_{j}, m > j$ . Thus  $\# j < m: |\mu_{i} - \mu_{U_{j}}| \leq 3\Delta_{U_{j}} + 3\Delta_{i}$  and therefore, if  $\Delta_{i} < \Delta_{U_{m}}$ , then
$$
\arg \min  \left\{\Delta_ {i}: \forall j \leq m - 1, \left| \mu_ {U _ {j}} - \mu_ {i} \right| \geq 3 \Delta_ {m} + 3 \Delta_ {j} \right\} \neq U _ {m}
$$
which is a contradiction via the definition of  $U_{m}$ .
For  $m\in [M]$  , set  $W_{m} = \{i:|\mu_{i} - \mu_{U_{m}}|\leq 3\Delta_{U_{m}}\}$
Proposition B.2. For all  $m\in [M]$
$$
\left| \left\{i \in D _ {m}: \left| \mu_ {i} - \mu_ {U _ {m}} \right| \geq 3 \Delta_ {U _ {m}} \right\} \right| \leq 3 6 | W _ {m} |
$$
Proof. Firstly, by definition we have that,
$$
\Delta_ {U _ {m}} \geq \frac {K \varepsilon p \left(1 - \mu_ {U _ {m}}\right)}{\left| W _ {m} \right|}. \tag {23}
$$
Now, let  $i^{*} = \arg \max_{i\in D_{m}\setminus W_{m}}(\Delta_{i})$ . As we have that  $\forall i\in D_m, |\mu_i - \mu_{U_m}| \leq 3\Delta_i + 3\Delta_{U_m}$ , the following holds,
$$
\forall i \in D _ {m} \backslash W _ {m}, | \mu_ {i} - \mu_ {i ^ {*}} | \leq 3 \Delta_ {i ^ {*}} + 3 \Delta_ {U _ {m}}.
$$
Now consider the set of adjacent grid points  $L \subset D_m \setminus W_m$  such that  $\lambda\left(\bigcup_{i \in L}[G_i, G_{i+1}]\right) \leq \Delta_{i^*}$ , for  $j \in L$ ,
$$
\begin{array}{l} \Delta_ {j} \leq \frac {K \varepsilon p \left(1 - \mu_ {j}\right)}{| L |} (24) \\ \leq \frac {K \varepsilon p \left(\left(1 - \mu_ {U _ {m}}\right) + 3 \Delta_ {j} + 3 \Delta_ {U _ {m}}\right)}{| L |} (25) \\ \leq \frac {K \varepsilon p \left(1 - \mu_ {U _ {m}}\right)}{| L |} + \frac {6 \Delta_ {j}}{| L |} (26) \\ \end{array}
$$
Where the final inequality comes from the fact we assume  $K\varepsilon p < 1$ . Thus if we assume  $|L| \geq 12|W_m|$ , we have that,  $\Delta_j \leq \frac{K\varepsilon p(1 - \mu_{U_m})}{2|C_k|}$ , a contradiction via proposition B.1 and Equation (23)
The proof of the following proposition follows via the same argument as in the proof of Proposition B.2.
Proposition B.3. For all  $m \in [M]$ ,
$$
\left| W _ {m} \right| \leq 3 \left| \left\{i \in W _ {m}: \left| \mu_ {i} - \mu_ {U _ {m}} \right| \leq \Delta_ {U _ {m}} \right\} \right|
$$
We are now ready to construct our set of problems. Consider a family of problems  $\nu^Q$  indexed by  $Q\in \{0,1\} ^K$ , where the target function  $\eta_{Q}$  corresponding to  $\nu^{Q}$  is defined as follows. Set the coefficients,
$$
(\beta_ {m} ^ {1}, \beta_ {m} ^ {2}) _ {m \in [ M ]}
$$
and for  $m\in [M]$ , set  $\beta_1^m = \mu_{U_m} + 4 / 3\Delta_{U_m},\beta_2 = \mu_{U_m} - 4 / 3\Delta_{U_m}$ , also define the sets