SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\forall \varepsilon > 0, \quad i _ {\mu} (\varepsilon) := \left\{ \begin{array}{l l} i & \text {i f} \varepsilon \in \left[ \Delta_ {i}, \Delta_ {i + 1}\right) \\ C _ {\mu} & \text {i f} \varepsilon \geq \Delta_ {C _ {\mu}} \end{array} \right. \quad \text {w i t h} \quad i \in [ 1, C _ {\mu} - 1 ]. \tag {21}
$$
It is direct to see that  $i_{\mu}(\varepsilon) = \max \{i \in [C_{\mu}], \mathcal{C}_{\mu}(i) \subset \mathcal{I}_{\varepsilon}(\mu)\}$  for all  $\varepsilon > 0$ . For all  $i \in [C_{\mu} - 1]$ , let us define  $H_{i}(\mu, \varepsilon_{0}) = \min_{j \in [i]} \max \{\bar{H}_{i,j}(\mu, \varepsilon_{0}), \tilde{H}_{i,j}(\mu, \varepsilon_{0})\}$  with  $\bar{H}_{i,j}(\mu, \varepsilon_{0})$  and  $\tilde{H}_{i,j}(\mu, \varepsilon_{0})$  as in Theorem 6. Let  $\varepsilon_{1} \geq 0$  and  $\varepsilon \in [0, \varepsilon_{1}]$ . Then, we have by direct manipulations that
$$
\inf _ {\varepsilon \in [ 0, \varepsilon_ {1} ]} \max  \{\bar {H} _ {\varepsilon , \varepsilon_ {1}} (\mu , \varepsilon_ {0}), \tilde {H} _ {\varepsilon , \varepsilon_ {1}} (\mu , \varepsilon_ {0}) \} = H _ {i _ {\mu} (\varepsilon_ {1})} (\mu , \varepsilon_ {0}),
$$
since  $\tilde{H}_{\varepsilon, \varepsilon_1}(\mu, \varepsilon_0) = \bar{H}_{i_\mu(\varepsilon_1), i_\mu(\varepsilon)}(\mu, \varepsilon_0)$  and  $\tilde{H}_{\varepsilon, \varepsilon_1}(\mu, \varepsilon_0) = \tilde{H}_{i_\mu(\varepsilon_1), i_\mu(\varepsilon)}(\mu, \varepsilon_0)$ . This concludes the first part of Theorem 6, i.e.
$$
\mathbb {P} _ {\nu} (\hat {i} _ {n} \notin \mathcal {I} _ {\varepsilon_ {1}} (\mu)) \leq \mathbb {1} (\varepsilon_ {1} <   \Delta_ {\max}) \frac {K (K + 1)}{2} e ^ {2} (2 + \log n) ^ {2} p \left(\frac {n - 5 K ^ {2} / 2}{8 H _ {i _ {\mu} (\varepsilon_ {1})} (\mu , \varepsilon_ {0})}\right).
$$
Simple regret Since  $\mu_{\star} - \mu_{i_n}$  is a positive random variable, we obtain that
$$
\mathbb {E} _ {\nu} \left[ \mu_ {\star} - \mu_ {i _ {n}} \right] = \int_ {\varepsilon_ {1} = 0} ^ {+ \infty} \mathbb {P} _ {\nu} \left(\mu_ {\star} - \mu_ {i _ {n}} > \varepsilon_ {1}\right) d \varepsilon_ {1} = \int_ {\varepsilon_ {1} = 0} ^ {+ \infty} \mathbb {P} _ {\nu} \left(i _ {n} \notin \mathcal {I} _ {\varepsilon_ {1}} (\mu)\right) d \varepsilon_ {1}.
$$
As a function of  $\varepsilon$ , our upper bound on  $\mathbb{P}_{\nu}(\hat{\iota}_n \notin \mathcal{I}_{\varepsilon_1}(\mu))$  is piecewise constant on  $[\Delta_i, \Delta_{i+1})$  for all  $i \in [C_\mu - 1]$  and has null value for  $\varepsilon_1 \geq \Delta_{\max}$ . Our upper is informative, i.e. smaller than 1, for all  $n \geq D_\mu$  where
$$
\begin{array}{l} D _ {\mu} = 8 H _ {1} (\mu , \varepsilon_ {0}) h _ {2} \left(8 H _ {1} (\mu , \varepsilon_ {0}), 5 K ^ {2} / 2, 2 + \log (K (K + 1) / 2)\right) + 5 K ^ {2} / 2 \\ > \sup  \left\{x \mid x <   8 H _ {1} (\mu , \varepsilon_ {0}) \overline {{W}} _ {- 1} (2 \log (2 + \log x) + 2 + \log (K (K + 1) / 2)) + 5 K ^ {2} / 2 \right\}, \\ \end{array}
$$
where the inequality is obtained by using Lemma 51, and  $h_2$  is defined therein. Therefore, we obtain that, for all  $n \geq D_{\mu}$ ,
$$
\mathbb {E} _ {\nu} [ \mu_ {\star} - \mu_ {i _ {n}} ] \leq \frac {K (K + 1)}{2} e ^ {2} (2 + \log n) ^ {2} \sum_ {i \in [ C _ {\mu} - 1 ]} (\Delta_ {i + 1} - \Delta_ {i}) p \left(\frac {n - 5 K ^ {2} / 2}{8 H _ {i} (\mu , \varepsilon_ {0})}\right),
$$
which concludes the second part of Theorem 6.
F.2 Unverifiable sample complexity
The  $(\varepsilon, \delta)$ -unverifiable sample complexity is defined as the expectation of the smallest stopping time  $\tilde{\tau}$  satisfying  $\mathbb{P}(\forall t \geq \tilde{\tau}, \hat{i}_n \in \mathcal{I}_{\varepsilon}(\mu)) \geq 1 - \delta$ . Theorem 7 gives an upper bound on the unverifiable expected sample complexity of the EB-TC $_{\varepsilon_0}$  algorithm with fixed proportions  $\beta = 1/2$ .
Theorem 7. Let  $\varepsilon_0 > 0$  and  $\delta \in (0,1)$ . The  $EB-TC_{\varepsilon_0}$  algorithm with fixed proportions  $\beta = 1/2$  satisfies that, for all  $\nu \in \mathcal{D}^K$  with mean  $\mu$ , for all  $\varepsilon \geq 0$ ,
$$
\mathbb {P} _ {\nu} \left(\forall n > U _ {i _ {\mu} (\varepsilon), \delta} (\mu , \varepsilon_ {0}), \hat {i} _ {n} \in \mathcal {I} _ {\varepsilon} (\mu)\right) \geq 1 - \delta .
$$
The times  $U_{i_{\mu}(\varepsilon),\delta}(\mu ,\varepsilon_0)$  are the  $(\varepsilon ,\delta)$ -unverifiable sample complexity of  $EB-TC_{\varepsilon_0}$  defined as
$$
\forall i \in [ C _ {\mu} - 1 ], \quad U _ {i, \delta} (\mu , \varepsilon_ {0}) = h _ {2} \left(\log (1 / \delta), 8 H _ {i} (\mu , \varepsilon_ {0}), 8 H _ {i} (\mu , \varepsilon_ {0}) \log \left(K (K + 1) / 2\right) + 5 K ^ {2} / 2\right), \tag {22}
$$
with  $(H_{i}(\mu ,\varepsilon_{0}))_{i\in [C_{\mu} - 1]}$  defined in Theorem 6. The function  $h_2(\log (1 / \delta),A,B) =$ $2A\overline{W}_{-1}(\frac{1}{2}\log (1 / \delta) + \frac{B}{2A} +\log (2A))$  satisfies that  $h_2(\log (1 / \delta),A,B) = _{\delta \to 0}A\log (1 / \delta) +$ $\mathcal{O}(\log \log (1 / \delta))$
Proof. In Appendix F.1, we consider the concentration event  $\tilde{\mathcal{E}}_{n,\delta}$  that involved tighter concentration results with thresholds  $\tilde{f}_1(n,\delta)$  and  $\tilde{f}_2(n,\delta)$ . Let  $n > K$  and  $\delta \in (0,1)$ . It is direct to see that the same argument holds for the concentration events  $\mathcal{E}_{n,\delta} = \mathcal{E}_{n,\delta}^1 \cap \mathcal{E}_{n,\delta}^2$  with  $(\mathcal{E}_{n,\delta}^1)_{n > K}$  and  $(\mathcal{E}_{n,\delta}^2)_{n > K}$  as in (24) and (25) for  $s = 0$ , i.e.
$$
\mathcal {E} _ {n, \delta} ^ {1} = \left\{\forall k \in [ K ], \forall t \leq n, | \mu_ {t, k} - \mu_ {k} | <   \sqrt {\frac {2 f _ {1} (n , \delta)}{N _ {t , k}}} \right\},
$$
$$
\mathcal {E} _ {n, \delta} ^ {2} = \left\{\forall (i, k) \in [ K ] ^ {2} \text {s . t .} i \neq k, \forall t \leq n, \frac {\left| (\mu_ {t , i} - \mu_ {t , k}) - (\mu_ {i} - \mu_ {k}) \right|}{\sqrt {1 / N _ {t , i} + 1 / N _ {t , k}}} <   \sqrt {2 f _ {2} (n , \delta)} \right\},
$$
where  $f_{1}(n,\delta) = \log (1 / \delta) + \log n + \log \left(K(K + 1) / 2\right)$  and  $f_{2}(n,\delta) = \log (1 / \delta) + 2\log n + \log \left(K(K + 1) / 2\right)$ , which satisfy  $f_{1}(n,\delta)\leq f_{2}(n,\delta)$ . Using Lemmas 40 and 41, we obtain that  $\mathbb{P}_{\nu}(\mathcal{E}_{n,\delta}^{\mathbf{C}})\leq \delta$ .
In the following, we use the notation on the gaps introduced in Appendix F.1, the complexities  $(H_{i}(\mu ,\varepsilon_{0}))_{i\in [C_{\mu} - 1]}$  defined in Theorem 6 and the function  $i_{\mu}(\varepsilon)$  defined in (21). To prove Theorem 6, we obtain as an intermediary step that: for all  $n > E_{\varepsilon_1,\varepsilon_0,\mu}(\delta)$ ,  $\{\hat{i}_n\notin \mathcal{I}_{\varepsilon_1}(\mu)\} \subseteq \mathcal{E}_{n,\delta}^{\complement}$ , where
$$
\begin{array}{l} E _ {\varepsilon_ {1}, \varepsilon_ {0}, \mu} (\delta) := \sup  \left\{n \mid n \leq 8 \inf  _ {\varepsilon \in [ 0, \varepsilon_ {1} ]} \max  \{\bar {H} _ {\varepsilon , \varepsilon_ {1}} (\mu , \varepsilon_ {0}), \tilde {H} _ {\varepsilon , \varepsilon_ {1}} (\mu , \varepsilon_ {0}) \} f _ {2} (n, \delta) + 5 K ^ {2} / 2 \right\} \\ = \sup  \left\{n \mid n \leq 8 H _ {i _ {\mu} (\varepsilon_ {1})} f _ {2} (n, \delta) + 5 K ^ {2} / 2 \right\}, \\ \end{array}
$$
where the equality holds by using the rewriting of the complexities done in the end of Appendix F.1. Let  $A = 8H_{i_{\mu}(\varepsilon_1)}$  and  $B = 8H_{i_{\mu}(\varepsilon_1)}\log (K(K + 1) / 2) + 5K^2 /2$ . Using a proof similar to Lemma 51, applying Lemma 49 yields that
$$
\begin{array}{l} n > E _ {\varepsilon_ {1}, \varepsilon_ {0}, \mu} (\delta) \iff n > 2 A \log n + A \log (1 / \delta) + B \\ \Longleftrightarrow \frac {n}{2 A} - \log \left(\frac {n}{2 A}\right) > \frac {1}{2} \log (1 / \delta) + \frac {B}{2 A} + \log (2 A) \\ \Longleftrightarrow \quad n > 2 A \overline {{W}} _ {- 1} \left(\frac {1}{2} \log (1 / \delta) + \frac {B}{2 A} + \log (2 A)\right), \\ \end{array}
$$
Let us define  $h_2(\log(1/\delta), A, B) = 2A\overline{W}_{-1}\left(\frac{1}{2}\log(1/\delta) + \frac{B}{2A} + \log(2A)\right)$  and, for all  $i \in [C_\mu - 1]$ , we define
$$
U _ {i, \delta} (\mu , \varepsilon_ {0}) = h _ {2} \left(\log (1 / \delta), 8 H _ {i} (\mu , \varepsilon_ {0}), 8 H _ {i} (\mu , \varepsilon_ {0}) \log \left(K (K + 1) / 2\right) + 5 K ^ {2} / 2\right).
$$
Therefore, we have shown that, for all  $\varepsilon_1\geq 0$
$$
\{\exists n > U _ {i _ {\mu} (\varepsilon_ {1}), \delta} (\mu , \varepsilon_ {0}), \hat {i} _ {n} \notin \mathcal {I} _ {\varepsilon_ {1}} (\mu) \} \subseteq \mathcal {E} _ {n, \delta} ^ {\mathbf {C}}
$$
Using that  $\mathbb{P}_{\nu}(\mathcal{E}_{n,\delta}^{\complement})\leq \delta$  , we can conclude that, for all  $\varepsilon_{1}\geq 0$
$$
\mathbb {P} _ {\nu} \left(\forall n > U _ {i _ {\mu} (\varepsilon_ {1}), \delta} (\mu , \varepsilon_ {0}), \hat {i} _ {n} \in \mathcal {I} _ {\varepsilon_ {1}} (\mu)\right) \geq 1 - \delta .
$$
F.3 Cumulative regret of induced policy
Given a stream of recommendations  $(\hat{\iota}_n)_{n > K}$ , an external agent aiming at minimizing its cumulative regret can simply pull the arm  $\hat{\iota}_n$  at time  $n$ . Since the EB-TCa algorithm has anytime guarantees on the simple regret of this recommendation, the regret of the induced policy is constant (Corollary 2).
Corollary 2. Let  $\varepsilon_0 > 0$ . Let  $(\hat{i}_n)_{n > K}$  be the recommendations of the  $EB-TC_{\varepsilon_0}$  algorithm with fixed proportions  $\beta = 1/2$ . An agent pulling  $\hat{i}_n$  at time  $n$  has constant induced regret, i.e.
$$
\forall \nu \in \mathcal {D} ^ {K}, \forall T > K, \sum_ {n = K + 1} ^ {T} \mathbb {E} _ {\nu} [ \mu_ {\star} - \mu_ {\hat {i} _ {n}} ] = \mathcal {O} \left(\frac {K ^ {3} \Delta_ {\max}}{\min \{\Delta_ {\min} , \varepsilon_ {0} \} ^ {2}} \left(\log \frac {K}{\min \{\Delta_ {\min} , \varepsilon_ {0} \} ^ {2}}\right) ^ {2}\right).
$$
Proof. Using that  $H_{1}(\mu, \varepsilon_{0}) = \max_{i \in [C_{\mu} - 1]} H_{i}(\mu, \varepsilon_{0})$ , a direct upper bound based on Corollary 1 yields that, for all  $n \geq D_{\mu}$ ,
$$
\mathbb {E} _ {\nu} [ \mu_ {\star} - \mu_ {\hat {i} _ {n}} ] \leq \frac {K (K + 1)}{2} e ^ {2} (2 + \log n) ^ {2} \Delta_ {\max} \frac {n - 5 K ^ {2} / 2}{8 H _ {1} (\mu , \varepsilon_ {0})} \exp \left(- \frac {n - 5 K ^ {2} / 2}{8 H _ {1} (\mu , \varepsilon_ {0})}\right).
$$
Let  $c > 0$ . Let  $f(c, x) = (c + \log(x))^2 x \exp(-x)$  for all  $x \geq 1$ . Let  $A > 0$ ,  $B > 0$ . By integration, for all  $C \geq A + B$ , we obtain
$$
\sum_ {n = C} ^ {T} f (c, (n - B) / A) \leq A \int_ {1} ^ {+ \infty} f (c, x) \mathrm {d} x = A \left(c ^ {2} + 2 c C _ {1} + C _ {2}\right).
$$
where  $C_{\alpha} = \int_{1}^{+\infty}\log (x)^{\alpha}\exp (-x)\mathrm{d}x < + \infty$  for all  $\alpha \geq 0$ . For  $n\geq 5K^2$ , we have  $n\leq 2(n - 5K^2 /2)$ . Recall that  $D_{\mu} = 8H_{1}(\mu ,\varepsilon_{0})h_{2}\left(8H_{1}(\mu ,\varepsilon_{0}),5K^{2} / 2,2 + \log \left(K(K + 1) / 2\right)\right) + 5K^{2} / 2$  with  $h_2$  defined in Lemma 51. For all  $n < 5K^2 /2 + D_\mu$ , we have trivially that  $\mathbb{E}_{\nu}[\mu_{\star} - \mu_{\hat{i}_n}]\leq \Delta_{\max}$ . Therefore, we obtain
$$
\begin{array}{l} \sum_ {n = K + 1} ^ {T} \mathbb {E} _ {\nu} [ \mu_ {\star} - \mu_ {\hat {i} _ {n}} ] \leq (5 K ^ {2} / 2 + D _ {\mu}) \Delta_ {\max } + \frac {K (K + 1)}{2} e ^ {2} \Delta_ {\max } \\ \sum_ {n = 5 K ^ {2} + 8 H _ {1} (\mu , \varepsilon_ {0})} ^ {T} f \left(2 + \log \left(1 6 H _ {1} (\mu , \varepsilon_ {0})\right), \frac {n - 5 K ^ {2} / 2}{8 H _ {1} (\mu , \varepsilon_ {0})}\right) \\ \leq (5 K ^ {2} / 2 + D _ {\mu}) \Delta_ {\max} + \frac {K (K + 1)}{2} e ^ {2} \Delta_ {\max} \\ 8 H _ {1} (\mu , \varepsilon_ {0}) \left(\left(2 + \log \left(1 6 H _ {1} (\mu , \varepsilon_ {0})\right)\right) ^ {2} + 2 C _ {1} \left(2 + \log \left(1 6 H _ {1} (\mu , \varepsilon_ {0})\right)\right) + C _ {2}\right). \\ \end{array}
$$
Studying the function  $h_2(x, y, z)$ , it is direct to see that  $h_2(x, y, z) =_{x \to +\infty} o(\log (x)^{\alpha})$  for all  $\alpha > 0$ , hence  $\Delta_{\max} D_\mu$  is dominated by the other term. Therefore, by studying the dominating term when  $\min \{ \Delta_{\min}, \varepsilon_0 \} \to 0$ , our upper bound scales as
$$
\mathcal {O} \left(K ^ {2} \Delta_ {\max} H _ {1} (\mu , \varepsilon_ {0}) (\log H _ {1} (\mu , \varepsilon_ {0})) ^ {2}\right) = \mathcal {O} \left(\frac {K ^ {3} \Delta_ {\max}}{\min \{\Delta_ {\min} , \varepsilon_ {0} \} ^ {2}} \left(\log \frac {K}{\min \{\Delta_ {\min} , \varepsilon_ {0} \} ^ {2}}\right) ^ {2}\right),
$$
where we used that  $H_{1}(\mu ,\varepsilon_{0}) = K(2\Delta_{\min}^{-1} + 3\varepsilon_{0}^{-1})^{2} = \mathcal{O}\left(\frac{K}{\min\{\Delta_{\min},\varepsilon_{0}\}^{2}}\right)$ .
The idea of decoupling exploration and exploitation when minimizing the regret in the multi-armed bandits literature was introduced by [2]. At each round, the agent is allowed to choose one arm to explore and one arm to exploit at every round. While the agent suffers the loss from the exploited arm, it observes the one of the explored arm without cost. In the stochastic regime for instances having a unique best arm, the Decoupled-Tsallis-INF algorithm [33] achieves a constant regret  $\mathcal{O}(K / \Delta_{\mathrm{min}})$ . Therefore, in this setting, our constant regret in Corollary 2 does have the best achievable scaling in  $K$  and  $\Delta_{\mathrm{min}}$ .
Beyond synchronized policy When the external agent plays  $\hat{\iota}_n$  at time  $n$ , it is said to be synchronized with the recommendation rule. In all generality, one could pull arm  $\hat{\iota}_{\mathcal{L}(n)}$  at time  $n$ , where  $\mathcal{L}:\mathbb{N}\to \mathbb{N}$  is a non-decreasing function, and obtain similar guarantees as Corollary 2. The agent can be slower than the recommendation rule (e.g. offline hyper-parameter optimization) hence  $\mathcal{L}$  can be rapidly increasing, e.g.  $\mathcal{L}(n) = Bn + K$  where  $B > 1$ . The recommendation rule can be slower than the agent (e.g. pulling the same arm multiple times) hence  $\mathcal{L}$  can be piecewise constant with small steps, e.g.  $\mathcal{L}(n) = \lceil n / B\rceil +K$ . The communication between the agent and the recommendation rule can happen infrequently (e.g. paying to access the recommendation) hence  $\mathcal{L}$  can be piecewise constant with large steps, e.g.  $\mathcal{L}(n) = (\lceil n / B\rceil)^{\alpha} + K$  where  $\alpha >1$ .
G Concentration results
The proof of Lemma 3 is given in Appendix G.1. Appendix G.2 gathers concentration results to control the empirical means and gaps.
G.1 Proof of Lemma 3
Proving that a GLR stopping rule ensures the algorithm to be  $(\varepsilon_1, \delta)$ -PAC is done by leveraging concentration results. In particular, we build upon Theorem 9 of [27] which is restated below. While Theorem 9 was only stated for Gaussian distributions, it is direct to notice that the result also holds for sub-Gaussian distributions with variance proxy  $\sigma^2 = 1$  (as the authors mentioned).
Lemma 38 (Theorem 9 of [27]). Consider a sub-Gaussian bandit  $\nu$  with means  $\mu \in \mathbb{R}^K$ . Let  $S \subseteq [K]$  and  $x > 0$ .
$$
\mathbb {P} _ {\nu} \left[ \exists n \in \mathbb {N}, \sum_ {k \in S} \frac {N _ {n , k}}{2} \left(\mu_ {n, k} - \mu_ {k}\right) ^ {2} > \sum_ {k \in S} 2 \log \left(4 + \log \left(N _ {n, k}\right)\right) + | S | \mathcal {C} _ {G} \left(\frac {x}{| S |}\right) \right] \leq e ^ {- x}
$$
where  $\mathcal{C}_G$  is defined in [27] by  $\mathcal{C}_G(x) = \min_{\lambda \in ]1 / 2,1]}\frac{g_G(\lambda) + x}{\lambda}$  and
$$
g _ {G} (\lambda) = 2 \lambda - 2 \lambda \log (4 \lambda) + \log \zeta (2 \lambda) - \frac {1}{2} \log (1 - \lambda), \tag {23}
$$
where  $\zeta$  is the Riemann  $\zeta$  function and  $\mathcal{C}_G(x) \approx x + \log (x)$ .
Since  $\hat{i}_n = i^{\star}(\mu_n)$ , standard manipulations yield that for all  $i \neq \hat{i}_n$
$$
\begin{array}{l} \frac {\left(\mu_ {n , \hat {i} _ {n}} - \mu_ {n , i} + \varepsilon_ {1}\right) ^ {2}}{1 / N _ {n , \hat {i} _ {n}} + 1 / N _ {n , i}} = \inf  _ {u \in \mathbb {R}} \left(N _ {n, \hat {i} _ {n}} \left(\mu_ {n, \hat {i} _ {n}} - u\right) ^ {2} + N _ {n, i} \left(\mu_ {n, i} - u - \varepsilon_ {1}\right) ^ {2}\right) \\ = \inf _ {y \geq x + \varepsilon_ {1}} \left(N _ {n, i _ {n}} (\mu_ {n, i _ {n}} - x) ^ {2} + N _ {n, i} (\mu_ {n, i} - y) ^ {2}\right). \\ \end{array}
$$
Let  $i^{\star} = i^{\star}(\mu)$ . Using the stopping rule (3) and the above manipulations, we obtain
$$
\begin{array}{l} \mathbb {P} _ {\nu} \left(\tau_ {\varepsilon_ {1}, \delta} <   + \infty , \hat {i} _ {\tau_ {\varepsilon_ {1}, \delta}} \notin \mathcal {I} _ {\varepsilon_ {1}} (\mu)\right) \\ \leq \mathbb {P} _ {\nu} (\exists n \in \mathbb {N}, \exists i \notin \mathcal {I} _ {\varepsilon_ {1}} (\mu), i = i ^ {\star} (\mu_ {n}), \\ \min  _ {k \neq i} \inf  _ {y \geq x + \varepsilon_ {1}} \left(N _ {n, \hat {i} _ {n}} (\mu_ {n, \hat {i} _ {n}} - x) ^ {2} + N _ {n, i} (\mu_ {n, i} - y) ^ {2}\right) \geq 2 c (n - 1, \delta) \\ \leq \mathbb {P} _ {\nu} (\exists n \in \mathbb {N}, \exists i \notin \mathcal {I} _ {\varepsilon_ {1}} (\mu), i = i ^ {\star} (\mu_ {n}), \\ \left. \frac {N _ {n , i}}{2} \left(\mu_ {n, i} - \mu_ {i}\right) ^ {2} + \frac {N _ {n , i ^ {*}}}{2} \left(\mu_ {n, i ^ {*}} - \mu_ {i ^ {*}}\right) ^ {2} \geq c (n - 1, \delta)\right) \\ \leq \sum_ {i \notin \mathcal {I} _ {\varepsilon_ {1}} (\mu)} \mathbb {P} _ {\nu} \left(\exists n \in \mathbb {N}, \frac {N _ {n , i}}{2} (\mu_ {n, i} - \mu_ {i}) ^ {2} + \frac {N _ {n , i ^ {\star}}}{2} (\mu_ {n, i ^ {\star}} - \mu_ {i ^ {\star}}) ^ {2} \geq c (n - 1, \delta)\right), \\ \end{array}
$$
where the second inequality is obtained with  $(k,x,y) = (i^{\star},\mu_i,\mu_{i^{\star}})$  since  $i\notin \mathcal{I}_{\varepsilon_1}(\mu)$ , and the third by union bound. By concavity of  $x\mapsto \log (4 + \log (x))$  and  $N_{n,i^{\star}} + N_{n,i}\leq \sum_{k\in [K]}N_{n,k} = n - 1$ , we obtain
$$
\forall i \notin \mathcal {I} _ {\varepsilon_ {1}} (\mu), \quad \log (4 + \log N _ {n, i ^ {*}}) + \log (4 + \log N _ {n, i}) \leq 2 \log (4 + \log ((n - 1) / 2))
$$
Combining the above with Lemma 38 for all  $i \notin \mathcal{I}_{\varepsilon_1}(\mu)$  and using that  $|\mathcal{I}_{\varepsilon_1}(\mu)| \leq K - 1$ , we obtain
$$
\mathbb {P} _ {\nu} \left(\tau_ {\delta} <   + \infty , \hat {\imath} _ {\tau_ {\delta}} \notin \mathcal {I} _ {\varepsilon_ {1}} (\mu)\right) \leq \sum_ {i \notin \mathcal {I} _ {\varepsilon_ {1}} (\mu)} \frac {\delta}{K - 1} \leq \delta .
$$
G.2 Sequence of concentration events
Lemma 39 is a standard concentration result for sub-Gaussian distribution, hence we omit the proof.
Lemma 39. Let  $X$  be an observation from a sub-Gaussian distribution with mean 0 and variance proxy  $\sigma^2$ . Then, for all  $\delta \in (0,1]$ ,
$$
\mathbb {P} _ {X} \left(| X | \geq \sigma \sqrt {2 \log (1 / \delta)}\right) \leq \delta .
$$
Lemma 40 gives a sequence of concentration events under which the empirical means are close to their true values.
Lemma 40. Let  $\delta \in (0,1]$  and  $s\geq 0$  . For all  $n > K$  , let  $f_{1}(x,\delta) = \log (1 / \delta) + (1 + s)\log x$  and
$$
\mathcal {E} _ {n, \delta} ^ {1} := \left\{\forall k \in [ K ], \forall t \leq n, | \mu_ {t, k} - \mu_ {k} | <   \sqrt {\frac {2 f _ {1} (n , \delta)}{N _ {t , k}}} \right\}. \tag {24}
$$
Then, for all  $n > K$ ,  $\mathbb{P}_{\nu}((\mathcal{E}_{n,\delta}^{1})^{\complement}) \leq \frac{K\delta}{n^{s}}$ .
Proof. Let  $(X_{s})_{s\in [n]}$  be i.i.d. observations from one sub-Gaussian distribution with mean 0 and variance proxy  $\sigma^2 = 1$ . Then,  $\frac{1}{m}\sum_{i = 1}^{m}X_{i}$  is sub-Gaussian with mean 0 and variance proxy  $\sigma^2 = 1 / m$ . By union bound over  $[K]$  and over  $m\in [n]$ , we obtain
$$
\begin{array}{l} \mathbb {P} _ {\nu} \left(\exists k \in [ K ], \exists t \leq n, | \mu_ {t, k} - \mu_ {k} | \geq \sqrt {\frac {2 f _ {1} (n , \delta)}{N _ {t , k}}}\right) \\ \leq \sum_ {k \in [ K ]} \sum_ {m \in [ n ]} \mathbb {P} \left(\left| \frac {1}{m} \sum_ {s \in [ m ]} X _ {s} \right| \geq \sqrt {\frac {2 f _ {1} (n , \delta)}{m}}\right) \\ \leq \delta \sum_ {k \in [ K ]} \sum_ {m \in [ n ]} n ^ {- (1 + s)} = K \delta n ^ {- s}, \\ \end{array}
$$
where we used that  $\mu_{t,k} - \mu_k = \frac{1}{N_{t,k}}\sum_{s=1}^{t}\mathbb{1}(I_s = k)X_{s,k}$  and concentration results for sub-Gaussian observations (Lemma 39).
Lemma 41 gives a sequence of concentration events under which the empirical gaps are close to their true values.
Lemma 41. Let  $\delta \in (0,1]$  and  $s\geq 0$  . For all  $n > K$  , let  $f_{2}(x,\delta) = \log (1 / \delta) + (2 + s)\log (x)$  and
$$
\mathcal {E} _ {n, \delta} ^ {2} := \left\{\forall (i, k) \in [ K ] ^ {2} \text {s . t .} i \neq k, \forall t \leq n, \frac {\left| \left(\mu_ {t , i} - \mu_ {t , k}\right) - \left(\mu_ {i} - \mu_ {k}\right) \right|}{\sqrt {1 / N _ {t , i} + 1 / N _ {t , k}}} <   \sqrt {2 f _ {2} (n , \delta)} \right\}. \tag {25}
$$
Then, for all  $n > K$ ,  $\mathbb{P}_{\nu}((\mathcal{E}_{n,\delta}^2)^\complement)\leq \frac{K(K - 1)\delta}{2n^s}$ .
Proof. Let  $(X_{s})_{s\in [n]}$  and  $(Y_{s})_{s\in [n]}$  be two streams of i.i.d. observations from two sub-Gaussian distributions with mean 0 and variance proxy  $\sigma^2 = 1$ . Then,  $\frac{1}{m_1}\sum_{i = 1}^{m_1}X_i - \frac{1}{m_2}\sum_{i = 1}^{m_2}Y_i$  is sub-Gaussian with mean 0 and variance proxy  $\sigma^2 = \frac{1}{m_1} +\frac{1}{m_2}$ . By union bound, we obtain
$$
\begin{array}{l} \mathbb {P} _ {\nu} \left(\exists (i, k) \in [ K ] ^ {2} \text {s . t .} i \neq k, \exists t \leq n, \left| \frac {(\mu_ {t , i} - \mu_ {t , k}) - (\mu_ {i} - \mu_ {k})}{\sqrt {1 / N _ {t , i} + 1 / N _ {t , k}}} \right| \leq \sqrt {2 f _ {2} (n , \delta)}\right) \\ \leq \sum_ {(i, k) \in [ K ] ^ {2}, i \neq k} \sum_ {(m _ {1}, m _ {2}) \in [ n ] ^ {2}} \mathbb {P} \left(\left| \frac {1}{m _ {1}} \sum_ {i = 1} ^ {m _ {1}} X _ {i} - \frac {1}{m _ {2}} \sum_ {i = 1} ^ {m _ {2}} Y _ {i} \right| \leq - \sqrt {2 f _ {2} (n , \delta) (1 / m _ {1} + 1 / m _ {2})}\right) \\ \leq \delta \sum_ {(i, k) \in [ K ] ^ {2}, i \neq k} \sum_ {(m _ {1}, m _ {2}) \in [ n ] ^ {2}} n ^ {- (2 + s)} = \frac {K (K - 1)}{2} \delta n ^ {- s}, \\ \end{array}
$$
where we used that  $(\mu_{t,i^{\star}} - \mu_{i^{\star}}) - (\mu_{t,k} - \mu_{k}) = \frac{1}{N_{t,i^{\star}}}\sum_{s=1}^{t}\mathbb{1}(I_{s} = i^{\star})X_{s,i^{\star}} - \frac{1}{N_{t,k}}\sum_{s=1}^{t}\mathbb{1}(I_{s} = k)X_{s,k}$  and concentration results for sub-Gaussian observations (Lemma 39).
Tighter concentration Lemma 42 provides concentration results on the empirical means, which are tighter than the one obtained in Lemma 40.
Lemma 42. Let  $\delta \in (0,1]$  and  $s\geq 0$  .Let  $\overline{W}_{-1}$  defined in Lemma 49. For all  $n > K$  , let
$$
\tilde {f} _ {1} (n, \delta) = \frac {1}{2} \bar {W} _ {- 1} \left(2 \log (1 / \delta) + 2 s \log n + 2 \log (2 + \log n) + 2\right), \tag {26}
$$
$$
\tilde {\mathcal {E}} _ {n, \delta} ^ {1} = \left\{\forall k \in [ K ], \forall t \leq n, | \mu_ {t, k} - \mu_ {k} | <   \sqrt {\frac {2 \tilde {f} _ {1} (n , \delta)}{N _ {t , k}}} \right\}. \tag {27}
$$
Then, for all  $n > K$ ,  $\mathbb{P}_{\nu}((\tilde{\mathcal{E}}_{n,\delta}^{1})^{\mathbb{C}}) \leq \frac{K\delta}{n^{s}}$ .
Proof. Let  $(X_{s})_{s\in [n]}$  be i.i.d. observations from one sub-Gaussian distribution with mean 0 and variance proxy  $\sigma^2 = 1$ . Let  $S_{t} = \sum_{s\in [t]}X_{s}$ . To derived concentration result, we use peeling.
Let  $\eta > 0$ ,  $\gamma > 0$  and  $D = \lceil \frac{\log(n)}{\log(1 + \eta)} \rceil$ . For all  $i \in [D]$ , let  $N_i = (1 + \eta)^{i-1}$ . For all  $i \in [D]$ , we define the family of priors  $f_{N_i, \gamma}(x) = \sqrt{\frac{\gamma N_i}{2\pi}} \exp \left(-\frac{x^2 \gamma N_i}{2}\right)$  with weights  $w_i = \frac{1}{D}$  and process
$$
\overline {{M}} (t) = \sum_ {i \in [ D ]} w _ {i} \int f _ {N _ {i}, \gamma} (x) \exp \left(x S _ {t} - \frac {1}{2} x ^ {2} t\right) d x,
$$