SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lemma 6 Let  $W_{1}, W_{2}, \dots, W_{T}$  be identically and independently distributed random variables with common expectation  $\mu = \mathbb{E}[W_1]$  and sampled from a distribution  $\nu$  that belongs to a canonical exponential family  $\mathcal{P}$ . For any  $s \leqslant T$ , let  $S_{s} = \sum_{i=1}^{s} W_{i}$  and  $\bar{\mu}_{s} = \frac{S_{s}}{s}$  denote the sum and the empirical mean of the first  $s$  samples. Let  $T_{1} \leqslant T_{2} \leqslant T$  be two real numbers in  $\mathbb{R}^{+}$ . For any  $l > 2$ , the following holds:
$$
\mathbb {P} \left(\exists s \in \left[ T _ {1}, T _ {2} \right], \bar {\mu} _ {s} \leqslant \mu , s \mathrm {K L} (\bar {\mu} _ {s}, \mu) \geqslant l\right) \leqslant \min  \left\{\frac {T _ {2} - T _ {1} + 1}{\exp (l)}, \frac {e l (\log T _ {2} - \log T _ {1}) + e}{\exp (l)} \right\}, \tag {6a}
$$
$$
\mathbb {P} \left(\exists s \in \left[ T _ {1}, T _ {2} \right], \bar {\mu} _ {s} \geqslant \mu , s \mathrm {K L} (\bar {\mu} _ {s}, \mu) \geqslant l\right) \leqslant \min  \left\{\frac {T _ {2} - T _ {1} + 1}{\exp (l)}, \frac {e l (\log T _ {2} - \log T _ {1}) + e}{\exp (l)} \right\}. \tag {6b}
$$
Lemma 7 Let  $W_{1}, W_{2}, \dots, W_{T}$  be identically and independently distributed random variables with common expectation  $\mu = \mathbb{E}[W_1]$  and sampled from a distribution  $\nu$  that belongs to a canonical
exponential family  $\mathcal{P}$ . For any  $s \leqslant T$ , let  $S_s = \sum_{i=1}^{s} W_i$  and  $\bar{\mu}_s = \frac{S_s}{s}$  denote the sum and the empirical mean of the first  $s$  samples. Let  $\mathsf{UCB}_s = \arg \max_{\mu \geqslant \bar{r}_s} \{ s \mathsf{KL}(\bar{\mu}_s, \mu) \leqslant l \}$  be the upper confidence bound for empirical mean. For any  $l > 2$ , any  $T_1 \leqslant T$ , any  $\mu' > \mu$ , and any  $\varepsilon > 0$ , the following holds:
$$
\sum_ {s = 1} ^ {T _ {1}} \mathbb {P} \left(\cup C B _ {s} \geqslant \mu^ {\prime}\right) \leqslant \frac {(1 + \varepsilon) l}{K L (\mu , \mu^ {\prime})} + \frac {\beta_ {2} (\varepsilon)}{T ^ {\beta_ {1} (\varepsilon)}}, \tag {7}
$$
where  $\beta_{1}(\varepsilon) = \mathcal{O}(\varepsilon^{2})$  and  $\beta_{2}(\varepsilon) = \mathcal{O}(\varepsilon^{-2})$  are constants.
C.2 Proof of Regret Optimality for KL-EOCP
In this section, we provide a complete proof of Theorem 4. We prove the following Theorem which characterizes the finite-time performance of Algorithm 3, and Theorem 4 can be derived directly.
Theorem 7 If  $l = \log (T) + 4\sqrt{2\log(T)}$  and  $T \geqslant \max \{16, A, 8lKL_{\mathrm{lb}}^{-2}\}$ , the regret of KL-EOCP algorithm in Algorithm 3 can be upper bounded as:
$$
\operatorname {R e g} _ {\boldsymbol {\mu}} ^ {\mathrm {K L - E O C P}} (T) \leqslant \sum_ {a: \Delta_ {a} > 0} \left(\frac {\Delta_ {a} \log T}{\operatorname {K L} \left(\mu_ {a} , \mu_ {1}\right)} + \frac {1 0 \Delta_ {a} \log^ {\frac {3}{4}} T}{\operatorname {K L} \left(\mu_ {a} , \mu_ {1}\right)}\right) + o (1).
$$
Remark: The asymptotic result is clear from Theorem.7 by letting  $T$  increases to infinity, i.e.,
$$
\limsup_{T\to \infty}\frac{\operatorname{Reg}_{\boldsymbol{\mu}}^{\mathrm{KL - EOCP}}(T)}{\log T}\leqslant \sum_{a:\Delta_{a} > 0}\frac{\Delta_{a}}{\operatorname{KL}(\mu_{a},\mu_{1})}.
$$
Proof. Without loss of generality, let arm 1 be the unique best arm. From the regret decomposition lemma [38, Lemma 4.5], we can decompose the regret of Algorithm 3 to the number of pulls for each sub-optimal arm as follows:
Then, the key to bound the total regret is to bound the number of pulls for each sub-optimal arms. Since our KL-EOCP algorithm has a clear separation of exploration and exploitation phases, so for any sub-optimal action  $a$ , we can bound its pulls in different phases as follows:
$$
\mathbb {E} \big [ N _ {a} (T) \big ] = \underbrace {\mathbb {E} \left[ N _ {a} (T _ {\mathrm {c}}) \right]} _ {I _ {1}} + (T - T _ {\mathrm {c}}) \underbrace {\mathbb {P} (\hat {a} = a)} _ {I _ {2}},
$$
where recall that  $T_{\mathrm{c}}$  is the end of exploration phase and  $\hat{a}$  is the arm that the algorithm commits to during exploitation phase.
Bounding  $I_{1}$ : We first bound  $I_{1}$ , which is similar to the proof of bounding the regret for the traditional KL-UCB algorithm [12]. We have:
$$
I _ {1} = \mathbb {E} \left[ \sum_ {t = 1} ^ {T _ {\mathrm {c}}} \mathbb {1} _ {A _ {t} = a} \right] = 1 + \underbrace {\mathbb {E} \left[ \sum_ {t = A + 1} ^ {T _ {\mathrm {c}}} \mathbb {1} _ {A _ {t} = a} \mathbb {1} _ {\cup \mathcal {C B} _ {t - 1} (a) \geqslant \mu_ {1}} \right]} _ {A _ {1}} + \underbrace {\mathbb {E} \left[ \sum_ {t = A + 1} ^ {T _ {\mathrm {c}}} \mathbb {1} _ {A _ {t} = a} \mathbb {1} _ {\cup \mathcal {C B} _ {t - 1} (a) <   \mu_ {1}} \right]} _ {A _ {2}},
$$
Event  $\{\mathsf{UCB}_{t - 1}(1) < \mu_1\}$  means that at time step  $t$ , the mean estimation  $\bar{r}_{t - 1}(1)$  of the optimal arm from previous pulls is lower than  $\mu_{1} - b_{t - 1}(1)$ , which incurs a large deviation, so we can bound this event with a union over all time steps:
$$
\left\{\cup \mathcal {C B} _ {t - 1} (1) <   \mu_ {1} \right\} \subset \left\{\exists t ^ {\prime} \in [ A, T _ {\mathrm {c}}), \cup \mathcal {C B} _ {t ^ {\prime}} (1) <   \mu_ {1} \right\},
$$
$$
\begin{array}{l} \sum_ {t = A + 1} ^ {T _ {\mathrm {c}}} \mathbb {P} \left(\mathsf {U C B} _ {t - 1} (1) <   \mu_ {1}\right) \leqslant \sum_ {t = A + 1} ^ {T _ {\mathrm {c}}} \mathbb {P} \left(\exists t ^ {\prime} \in [ A, T _ {\mathrm {c}}), \mathsf {U C B} _ {t ^ {\prime}} (1) <   \mu_ {1}\right) \\ = (T _ {\mathrm {c}} - A) \mathbb {P} \left(\exists t ^ {\prime} \in \left[ A, T _ {\mathrm {c}}\right), \mathsf {U C B} _ {t ^ {\prime}} (1) <   \mu_ {1}\right). \\ \end{array}
$$
For two time steps  $T_{\mathrm{c}} < t_{2}$ , if there is no pulls for arm 1 between them, then the term  $\mathsf{UCB}_{t'}(1)$  will remain the same through  $t' \in [T_{\mathrm{c}}, t_{2}]$ . So instead of counting on the time steps, we can count the number of pulls for arm 1 instead. We have:
$$
\left\{\exists t ^ {\prime} \in \left[ A, T _ {\mathrm {c}}\right), \cup \mathsf {C B} _ {t ^ {\prime}} (1) <   \mu_ {1} \right\} \subset \left\{\exists s \in \left[ 1, N _ {T _ {\mathrm {c}}} (a)\right), \cup \mathsf {C B} _ {s, 1} <   \mu_ {1} \right\}.
$$
Notice that  $N_{T_{\mathrm{c}}}(a) \leqslant T_{\mathrm{c}} - A + 1$ , so we can bound the probability as follows:
$$
\begin{array}{l} \mathbb {P} \left(\exists t ^ {\prime} \in [ A, T _ {\mathrm {c}}), \cup \mathrm {C B} _ {t ^ {\prime}} (1) <   \mu_ {1}\right) \leqslant \mathbb {P} (\exists s \in [ 1, T _ {\mathrm {c}} - A + 1), \cup \mathrm {C B} _ {s, 1} <   \mu_ {1}) \\ = \mathbb {P} \left(\exists s \in [ 1, T _ {\mathrm {c}} - A + 1), \max  _ {\mu \geqslant \bar {r} _ {s, 1}} \left\{\mathsf {K L} (\bar {r} _ {s, 1}, \mu) \leqslant \frac {l}{s} \right\} <   \mu_ {1}\right). \\ \end{array}
$$
Notice that for any  $s$  and under event  $\{\mathsf{UCB}_{s,1} < \mu_1\}$ , we have  $\mu_1 \geqslant \mathsf{UCB}_{s,1} \geqslant \bar{r}_{s,1}$ . By definition of  $\mathsf{UCB}_{s,1}$ , we also have  $\mathsf{KL}(\bar{r}_{s,1}, \mathsf{UCB}_{s,1}) = \frac{l}{s}$ . Therefore, we come to the conclusion that  $\mathsf{KL}(\bar{r}_{s,1}, \mu) \geqslant \frac{l}{s}$ . using Eq. (6a) of Lemma 6, we have:
$$
\begin{array}{l} \mathbb {P} \left(\exists s \in [ 1, T _ {\mathrm {c}} - A + 1), \max  _ {\mu \geqslant \bar {r} _ {s, 1}} \left\{\mathsf {K L} (\bar {r} _ {s, 1}, \mu) \leqslant \frac {l}{s} \right\} <   \mu_ {1}\right) \\ \leqslant \mathbb {P} (\exists s \in [ 1, T _ {\mathrm {c}} - A + 1), s \mathrm {K L} (\bar {r} _ {s, 1}, \mu) \geqslant l) \\ \leqslant \frac {T _ {\mathrm {c}} - A}{\exp (l)}. \\ \end{array}
$$
Using the fact that  $\exp (l)\geqslant T\log^4 T$  when  $T\geqslant 16$  and putting the above bound back to the bound of  $A_{2}$ , we have:
$$
A _ {2} \leqslant \frac {(T _ {\mathrm {c}} - A) ^ {2}}{T \log^ {4} T} = \frac {\left(\frac {8 A (\log T + 4 \sqrt {2 \log T})}{\mathrm {K L} _ {\mathrm {l b}} ^ {2}}\right) ^ {2}}{T \log^ {2} T} = o \left(\frac {1}{T}\right),
$$
$$
A _ {1} = \mathbb {E} \left[ \sum_ {t = A + 1} ^ {T _ {\mathrm {c}}} \mathbb {1} _ {A _ {t} = a} \mathbb {1} _ {\cup C B _ {t - 1} (a) \geqslant \mu_ {1}} \right] = \mathbb {E} \left[ \sum_ {t = A + 1} ^ {T _ {\mathrm {c}}} \mathbb {1} _ {A _ {t} = a} \mathbb {1} _ {\cup C B _ {N _ {t - 1} (a), a} \geqslant \mu_ {1}} \right].
$$
Notice that the number of pulls  $N_{t-1}(a)$  will only increase by 1 every time there is a new pull, i.e., when  $A_t = a$ . Otherwise, the term inside summation is 0. So instead of counting on the time step  $t$ , we can count over the number of pulls over arm  $a$  as follows:
$$
\mathbb {E} \left[ \sum_ {t = A + 1} ^ {T _ {\mathrm {c}}} \mathbb {1} _ {A _ {t} = a} \mathbb {1} _ {\cup \mathcal {C B} _ {N _ {t - 1} (a), a} \geqslant \mu_ {1}} \right] = \mathbb {E} \left[ \sum_ {s = 1} ^ {N _ {T _ {\mathrm {c}}} (a)} \mathbb {1} _ {\cup \mathcal {C B} _ {s, a} \geqslant \mu_ {1}} \right] \leqslant \sum_ {s = 1} ^ {T _ {\mathrm {c}}} \mathbb {P} \left(\cup \mathcal {C B} _ {s, a} \geqslant \mu_ {1}\right),
$$
where the last inequality is due to  $N_{T_{\mathrm{c}}}(\alpha) \leqslant T_{\mathrm{c}}$ . Using Lemma 7, we can bound the sum of probabilities. Specifically, for any  $\varepsilon > 0$ , there exists two constants  $\beta_1(\varepsilon) = \mathcal{O}(\varepsilon^2)$  and  $\beta_2(\varepsilon) = \mathcal{O}(\varepsilon^{-2})$  such that:
$$
\sum_ {s = 1} ^ {T _ {\mathrm {c}}} \mathbb {P} \left(\mathsf {U C B} _ {s, a} \geqslant \mu_ {1}\right) \leqslant \frac {(1 + \varepsilon) l}{\mathsf {K L} \left(\mu_ {a} , \mu_ {1}\right)} + \frac {\beta_ {2} (\varepsilon)}{T ^ {\beta_ {1} (\varepsilon)}}.
$$
Let  $\epsilon = \log^{-\frac{1}{4}}T$ , and when  $T\geqslant 16$ , we have  $\log T\geqslant \sqrt{\log T}$  and  $4\sqrt{2\log T}\leqslant 5\log^{\frac{3}{4}}T$ , so we have:
$$
\sum_ {s = 1} ^ {T _ {\mathrm {c}}} \mathbb {P} \left(\mathsf {U C B} _ {s, a} \geqslant \mu_ {1}\right) \leqslant \frac {\log T}{\mathsf {K L} \left(\mu_ {a} , \mu_ {1}\right)} + \frac {1 0 \log^ {\frac {3}{4}} T}{\mathsf {K L} \left(\mu_ {a} , \mu_ {1}\right)} + o (1).
$$
Therefore, combining the bounds on  $A_{1}$  and  $A_{2}$ , we can bound  $I_{1}$  as follows:
$$
I _ {1} \leqslant \frac {\log T}{\mathsf {K L} (\mu_ {a} , \mu_ {1})} + \frac {1 0 \log^ {\frac {3}{4}} T}{\mathsf {K L} (\mu_ {a} , \mu_ {1})} + o (1).
$$
Bounding  $I_{2}$ : After the exploration phase, recall that we select the arm with largest LCB to commit to. The idea of proof is very similar to the regret bound in [2] and our proof of Theorem 5. We can first bound the probability of selecting a sub-optimal arm  $a$  as:
$$
\begin{array}{l} \mathbb {P} (\hat {a} = a) \leqslant \mathbb {P} \left(\mathsf {L C B} _ {T _ {\mathrm {c}}} (a) \geqslant \mathsf {L C B} _ {T _ {\mathrm {c}}} (1)\right) \\ = \mathbb {P} \left(\mathsf {L C B} _ {T _ {\mathrm {c}}} (a) \geqslant \mathsf {L C B} _ {T _ {\mathrm {c}}} (1), N _ {T _ {\mathrm {c}}} (1) > \frac {4 l}{\mathsf {K L} _ {\mathrm {l b}}}\right) \\ + \mathbb {P} \left(\mathsf {L C B} _ {T _ {\mathrm {c}}} (a) \geqslant \mathsf {L C B} _ {T _ {\mathrm {c}}} (1), N _ {T _ {\mathrm {c}}} (1) \leqslant \frac {4 l}{\mathsf {K L} _ {\mathrm {l b}}}\right) \\ \leqslant \underbrace {\mathbb {P} \left(\mathrm {L C B} _ {T _ {\mathrm {c}}} (a) \geqslant \mathrm {L C B} _ {T _ {\mathrm {c}}} (1) , N _ {T _ {\mathrm {c}}} (1) > \frac {4 l}{\mathrm {K L} _ {\mathrm {l b}}}\right)} _ {B _ {1}} + \underbrace {\mathbb {P} \left(N _ {T _ {\mathrm {c}}} (1) \leqslant \frac {4 l}{\mathrm {K L} _ {\mathrm {l b}}}\right)} _ {B _ {2}}. \\ \end{array}
$$
$$
\mathbb {P} \left(N _ {T _ {\mathrm {c}}} (1) \leqslant \frac {4 l}{\mathrm {K L} _ {\mathrm {l b}}}\right) = \mathbb {P} \left(\sum_ {a: \Delta_ {a} > 0} N _ {T _ {\mathrm {c}}} (a) > T _ {\mathrm {c}} - \frac {4 l}{\mathrm {K L} _ {\mathrm {l b}}}\right).
$$
Recall that  $T_{\mathrm{c}} = \frac{4Al}{KL_{\mathrm{lb}}} + A$ , so we have:
$$
\begin{array}{l} \mathbb {P} \left(N _ {T _ {\mathrm {c}}} (1) \leqslant \frac {4 l}{\mathsf {K L} _ {\mathrm {l b}}}\right) = \mathbb {P} \left(\sum_ {a: \Delta_ {a} > 0} N _ {T _ {\mathrm {c}}} (a) > \frac {4 (A - 1) l}{\mathsf {K L} _ {\mathrm {l b}}} + A\right) \\ \leqslant \mathbb {P} \left(\exists a > 1, N _ {T _ {\mathrm {c}}} (a) > \frac {4 l}{\mathsf {K L} _ {\mathrm {l b}}} + 1\right). \\ \end{array}
$$
$$
\mathbb {P} \left(\exists a > 1, N _ {T _ {\mathrm {c}}} (a) > \frac {4 l}{\mathsf {K L} _ {\mathrm {l b}}} + 1\right) \leqslant \sum_ {a: \Delta_ {a} > 0} \mathbb {P} \left(N _ {T _ {\mathrm {c}}} (a) > \frac {4 l}{\mathsf {K L} _ {\mathrm {l b}}} + 1\right).
$$
$$
\begin{array}{l} \mathbb {P} \left(N _ {T _ {\mathrm {c}}} (a) > \frac {4 l}{K L _ {\mathrm {l b}}} + 1\right) \\ \leqslant \mathbb {P} \left(\exists t \in [ A + 1, T _ {\mathrm {c}} ], N _ {t - 1} (a) = \left\lceil \frac {4 l}{\mathsf {K L} _ {\mathrm {l b}}} \right\rceil , A _ {t} = a\right) \\ \leqslant \mathbb {P} \left(\exists t \in [ A + 1, T _ {\mathrm {c}} ], N _ {t - 1} (a) = \left\lceil \frac {4 l}{\mathsf {K L} _ {\mathrm {l b}}} \right\rceil , \mathsf {U C B} _ {t - 1} (1) \leqslant \mathsf {U C B} _ {t - 1} (a)\right) \\ \leqslant \underbrace {\mathbb {P} \left(\exists t \in [ A + 1 , T _ {\mathrm {c}} ] , N _ {t - 1} (a) = \left\lceil \frac {4 l}{\mathsf {K L} _ {\mathrm {l b}}} \right\rceil , \mu_ {1} \leqslant \mathsf {U C B} _ {t - 1} (a)\right)} _ {B _ {3}} \\ + \underbrace {\mathbb {P} \left(\exists t \in \left[ A + 1 , T _ {c} \right] , \mathsf {U C B} _ {t - 1} (1) \leqslant \mu_ {1}\right)} _ {B _ {4}}. \\ \end{array}
$$
$$
\begin{array}{l} B _ {4} \leqslant \mathbb {P} (\exists s \in [ 1, T _ {\mathrm {c}} - A + 1), \cup \mathrm {C B} _ {s, 1} \leqslant \mu_ {1}) \\ \leqslant \mathbb {P} (\exists s \in [ 1, T _ {\mathrm {c}} - A + 1), \bar {r} _ {s, 1} \leqslant \mu_ {1}, s \mathrm {K L} (\bar {r} _ {s, 1}, \mu_ {1}) \geqslant l) \\ \leqslant \frac {T _ {\mathrm {c}} - A}{\exp l}, \\ \end{array}
$$
where the second inequality is because under event  $\{\mathsf{UCB}_{s,1}\leqslant \mu_1\}$  and according to the definition of  $\mathsf{UCB}_{s,1}$ , we have  $\bar{r}_{s,1}\leqslant \mathsf{UCB}_{s,1}\leqslant \mu_1$ , and since  $s\mathsf{KL}(\bar{r}_{s,1},\mathsf{UCB}_{s,1}) = l$  with  $\mu_{1} > \mathsf{UCB}_{s,1}$ , we have  $s\mathsf{KL}(\bar{r}_{s,1},\mu_1)\geqslant l$ . The last inequality uses Lemma 6. Similar to the procedure of bounding  $I_{1}$ , notice that  $\exp (l)\geqslant T\log^4 T$ , we have:
$$
B _ {4} \leqslant \frac {4 A (\log T + 4 \sqrt {2 \log T})}{\mathsf {K L} _ {\mathrm {l b}}} \frac {1}{T \log^ {4} T} \leqslant \frac {2 0 A}{\mathsf {K L} _ {\mathrm {l b}} T \log^ {3} T}.
$$
On the other hand, let  $\gamma = \left\lceil \frac{4l}{KL_{1b}} \right\rceil$ , and we also change the count from time step  $t$  to the number of pulls for arm  $a$ . Then,  $B_{3}$  can be expressed as follows:
$$
B _ {3} = \mathbb {P} \left(\exists t \in [ A + 1, T _ {\mathrm {c}} - 1 ], N _ {t - 1} (a) = \gamma , \mu_ {1} \leqslant \bigcup \mathrm {C B} _ {N _ {t - 1} (a), a}\right) = \mathbb {P} \left(\mu_ {1} \leqslant \bigcup \mathrm {C B} _ {\gamma , a}\right).
$$
For any pair of means  $x, y \in [0,1]$ , define  $\mathsf{KL}^{+}(x,y) = \mathsf{KL}(x,y)\mathbb{1}_{x < y}$ . Then, we have:
$$
\mathbb {P} \left(\mu_ {1} \leqslant \cup \mathrm {C B} _ {\gamma , a}\right) \leqslant \mathbb {P} \left(\mathrm {K L} ^ {+} (\bar {r} _ {\gamma , a}, \mu_ {1}) \leqslant \frac {l}{\gamma}\right).
$$
Notice that  $\gamma = \left\lceil \frac{4l}{\mathrm{KL}_{\mathrm{lb}}}\right\rceil \geqslant \frac{4l}{\mathrm{KL}(\mu_a,\mu_1)}$ , we can also bound  $B_{3}$  as follows:
$$
B _ {3} \leqslant \mathbb {P} \left(\mathrm {K L} ^ {+} (\bar {r} _ {\gamma , a}, \mu_ {1}) \leqslant \frac {\mathrm {K L} (\mu_ {a} , \mu_ {1})}{4}\right).
$$
Recall that  $\mu_{a}^{\prime}\in (\mu_{a},\mu_{1})$  such that  $\mathsf{KL}(\mu_a^\prime ,\mu_1) = \frac{\mathsf{KL}(\mu_a,\mu_1)}{4}$ , so we have  $\bar{r}_{\gamma ,a}\geqslant \mu_a^\prime$  under the event that  $\left\{\mathsf{KL}^{+}(\bar{r}_{\gamma ,a},\mu_1)\leqslant \frac{\mathsf{KL}(\mu_a,\mu_1)}{4}\right\}$ , and thus  $\mathsf{KL}(\bar{r}_{\gamma ,a},\mu_a)\geqslant \mathsf{KL}(\mu_a^\prime ,\mu_a)$ . By Lemma 5, we have:
$$
\mathbb {P} \left(\mathsf {K L} ^ {+} (\bar {r} _ {\gamma , a}, \mu_ {1}) \leqslant \frac {\mathsf {K L} (\mu_ {a} , \mu_ {1})}{4}\right) \leqslant \mathbb {P} \left(\mathsf {K L} (\bar {r} _ {\gamma , a}, \mu_ {a}) \geqslant \mathsf {K L} (\mu_ {a} ^ {\prime}, \mu_ {a})\right) \leqslant \exp (- \gamma \mathsf {K L} (\mu_ {a} ^ {\prime}, \mu_ {a})).
$$
Notice that  $\gamma \mathsf{KL}(r(\gamma),\mu_a)\geqslant \frac{4l\mathsf{KL}(\mu_a',\mu_a)}{\mathsf{KL}_{\mathrm{lb}}}\geqslant l$  by the definition of  $\mathsf{KL}_{\mathrm{min}}$ , we then have:
$$
B _ {3} \leqslant \exp (- l) \leqslant \frac {1}{T \log^ {4} T}.
$$
The last inequality is due to the fact that  $\exp (l)\geqslant T\log^4 T$ . So combine  $B_{3}$  and  $B_{4}$  together, we can bound  $B_{2}$  as follows:
$$
B _ {2} \leqslant \sum_ {a: \Delta_ {a} > 0} (B _ {3} + B _ {4}) \leqslant A \left(\frac {2 0 A}{\mathsf {K L} _ {\mathrm {l b}} T \log^ {3} T} + \frac {1}{T \log^ {4} T}\right) = o \left(\frac {1}{T}\right).
$$
$$
\begin{array}{l} B _ {1} = \mathbb {P} \left(\mathrm {L C B} _ {T _ {\mathrm {c}}} (a) \geqslant \mathrm {L C B} _ {T _ {\mathrm {c}}} (1), N _ {T _ {\mathrm {c}}} (1) > \frac {4 l}{\mathrm {K L} _ {\mathrm {l b}}}\right) \\ \leqslant \underbrace {\mathbb {P} \left(\mu_ {a} \geqslant \operatorname {L C B} _ {T _ {c}} (1) , N _ {T _ {c}} (1) > \frac {4 l}{\operatorname {K L} _ {\mathrm {l b}}}\right)} _ {B _ {5}} + \underbrace {\mathbb {P} \left(\mu_ {a} \leqslant \operatorname {L C B} _ {T _ {c}} (a)\right)} _ {B _ {6}}. \\ \end{array}
$$
$$
\begin{array}{l} B _ {6} \leqslant \mathbb {P} (\exists s \in [ 1, T _ {\mathrm {c}} - A + 1 ], \mathrm {L C B} _ {s, a} \geqslant \mu_ {a}) \\ = \mathbb {P} \left(\exists s \in \left[ 1, T _ {\mathrm {c}} - A + 1 \right], \bar {r} _ {s, a} \geqslant \mu_ {a}, s \mathsf {K L} \left(\bar {r} _ {s, a}, \mu_ {a}\right) \geqslant l\right) \\ \leqslant \frac {T _ {\mathrm {c}} - A + 1}{\exp (l)}, \\ \end{array}
$$
where in the last inequality, we can apply Lemma (6). Since  $\log T \geqslant \sqrt{2\log T}$  and  $\exp(l) \geqslant \frac{1}{T\log^4 T}$  when  $T \geqslant 16$ , we have:
$$
B _ {6} \leqslant \frac {4 A \left(\log T + 4 \sqrt {2 \log T}\right)}{\mathsf {K L} _ {\mathrm {l b}} T \log^ {4} T} + \frac {1}{T \log^ {4} T} = o \left(\frac {1}{T}\right).
$$
For any pair of means  $x, y \in [0,1]$ , define  $\mathsf{KL}^{-}(x,y) = \mathsf{KL}(x,y)\mathbb{1}_{x > y}$ . Similarly, term  $B_{5}$  can be expressed as:
$$
\begin{array}{l} B _ {5} \leqslant \mathbb {P} \left(\exists s > \frac {4 l}{\mathrm {K L} _ {\mathrm {l b}}}, \mu_ {a} \geqslant \mathrm {L C B} _ {s, 1}\right) \\ \leqslant \mathbb {P} \left(\exists s > \frac {4 l}{K L _ {\mathrm {l b}}}, s K L ^ {-} \left(\bar {r} _ {s, 1}, \mu_ {a}\right) \leqslant l\right) \\ \leqslant \mathbb {P} \left(\exists s > \frac {4 l}{\mathsf {K L} _ {\mathrm {l b}}}, \mathsf {K L} ^ {-} (\bar {r} _ {s, 1}, \mu_ {a}) \leqslant \frac {\mathsf {K L} (\mu_ {a} , \mu_ {1})}{4}\right). \\ \end{array}
$$
Recall that  $\mu_{a}^{\prime}\in (\mu_{a},\mu_{1})$  satisfy  $\mathsf{KL}(\mu_a',\mu_a) = \frac{\mathsf{KL}(\mu_a,\mu_1)}{4}$ , so under event  $\{\mathsf{KL}^{-}(\bar{r}_{s,1},\mu_a)\leqslant \frac{\mathsf{KL}(\mu_a,\mu_1)}{4}\}$  we can conclude  $\bar{r}_{s,1}\leqslant \mu_a^\prime$ , which means  $\mathsf{KL}(\bar{r}_{s,1},\mu_1)\geqslant \mathsf{KL}(\mu_a^\prime ,\mu_1)$ , so we have:
$$
\begin{array}{l} \mathbb {P} \left(\exists s > \frac {4 l}{\mathrm {K L} _ {\mathrm {l b}}}, \mathrm {K L} ^ {-} (\bar {r} _ {s, 1}, \mu_ {a}) \leqslant \frac {\mathrm {K L} (\mu_ {a} , \mu_ {1})}{4}\right) \\ \leqslant \mathbb {P} \left(\exists s > \frac {4 l}{\mathsf {K L} _ {\mathrm {l b}}}, \bar {r} _ {s, 1} \leqslant \mu_ {1}, \mathsf {K L} (\bar {r} _ {s, 1}, \mu_ {1}) \geqslant \mathsf {K L} \left(\mu_ {a} ^ {\prime}, \mu_ {1}\right)\right) \\ \leqslant \mathbb {P} \left(\exists s > \frac {4 l}{K L _ {\mathrm {l b}}}, \bar {r} _ {s, 1} \leqslant \mu_ {1}, s K L (\bar {r} _ {s, 1}, \mu_ {1}) \geqslant \frac {4 l K L (\mu_ {a} ^ {\prime} , \mu_ {1})}{K L _ {\mathrm {l b}}}\right) \\ \leqslant \frac {e l \log T + e}{\exp \left(\frac {4 l \mathrm {K L} \left(\mu_ {a} ^ {\prime} , \mu_ {1}\right)}{\mathrm {K L} _ {\mathrm {l b}}}\right)}, \\ \end{array}
$$
where the last inequality comes from Lemma 6. By the definition of  $\mathsf{KL}_{\mathrm{min}}$ , we know that  $\frac{4l\mathsf{KL}(\mu_a',\mu_1)}{\mathsf{KL}_{\mathrm{lb}}}\geqslant l$ , so we have:
$$
B _ {5} \leqslant \frac {e l \log T + e}{\exp (l)} \leqslant \frac {1 0 e \log^ {2} T}{T \log^ {4} T} = o \left(\frac {1}{T}\right).
$$
Therefore, collecting the bounds for  $B_{5}$  and  $B_{6}$  and  $B_{2}$ , we have a bound for  $I_{2}$  as follows:
$$
I _ {2} \leqslant B _ {1} + B _ {2} \leqslant B _ {5} + B _ {6} + B _ {2} = o \left(\frac {1}{T}\right).
$$
Finally, putting the bounds on  $I_{1}$  and  $I_{2}$  together, we have:
$$
\mathbb {E} \left[ N _ {a} (T) \right] \leqslant I _ {1} + T I _ {2} \leqslant \frac {\log T}{\operatorname {K L} \left(\mu_ {a} , \mu_ {1}\right)} + \frac {1 0 \log^ {\frac {3}{4}} T}{\operatorname {K L} \left(\mu_ {a} , \mu_ {1}\right)} + o (1).
$$
$$
\operatorname {R e g} _ {\boldsymbol {\mu}} ^ {\mathrm {K L - E O C P}} (T) \leqslant \sum_ {a: \Delta_ {a} > 0} \left(\frac {\Delta_ {a} \log T}{\operatorname {K L} (\mu_ {a} , \mu_ {1})} + \frac {1 0 \Delta_ {a} \log^ {\frac {3}{4}} T}{\operatorname {K L} (\mu_ {a} , \mu_ {1})}\right) + o (1).
$$
D Proof of Concentration Inequalities
In this section, we provide the proof of the concentration inequalities presented in previous sections.
D.1 Proof of Lemma. 3
The proof of Lemma. 3 relies on the maximal inequalities in Lemma. 1 and Lemma. 2, and the famous Hoeffding's inequality which we state here for the sake of completeness.
Lemma 8 (Hoeffding's Inequality) Let  $(W_{i})_{i = 1}^{T}$  be i.i.d.  $\sigma$ -sub-Gaussian random variables with  $\mathbb{E}[W_1] = 0$ , we have:
$$
\mathbb {P} \left(\frac {\sum_ {i = 1} ^ {T} W _ {i}}{T} \geqslant \delta\right) \leqslant \exp \left(- \frac {T \delta^ {2}}{2 \sigma^ {2}}\right).
$$
Eq. (4a) of Lemma 3: We now prove Eq. (4a) of Lemma 3. We will first base our proof on the anytime concentration on union bound, which is tight only when  $T_{2} - T_{1}$  is relatively small. This procedure will result in the first term in the minimum at the RHS. To prove a stronger result when  $T_{2} - T_{1}$  is relatively large, we resort to the technique of "peeling device" which divide the time horizon into exponential grids, where we will perform maximal inequality inside each grid and a union bound over the grids. This will result in the second term in the minimum at the RHS. We first perform union bound on  $s$  as follows:
$$
\begin{array}{l} \mathbb {P} \left(\exists s \in (T _ {1}, T _ {2} ], \frac {\sum_ {i = 1} ^ {s} W _ {i}}{s} + \sqrt {\frac {2 l}{s}} \leqslant 0\right) = \mathbb {P} \left(\exists s \in (T _ {1}, T _ {2} ], \sum_ {i = 1} ^ {s} W _ {i} + \sqrt {2 l s} \leqslant 0\right) \\ \leqslant \sum_ {j = T _ {1} + 1} ^ {T _ {2}} \mathbb {P} \left(s = j, \sum_ {i = 1} ^ {s} W _ {i} + \sqrt {2 l s} \leqslant 0\right). \\ \end{array}
$$
Then, we can bound each probability with Hoeffding's concentration inequality as follows:
$$
\mathbb {P} \left(s = j, \sum_ {i = 1} ^ {s} W _ {i} + \sqrt {2 l s} \leqslant 0\right) = \mathbb {P} \left(\sum_ {i = 1} ^ {j} W _ {i} + \sqrt {2 j l} \leqslant 0\right) \leqslant \exp \left(- \frac {\left(\sqrt {2 j l}\right) ^ {2}}{2 j}\right) = \frac {1}{\exp (l)}.
$$
Therefore, summing up the probabilities, we will have:
$$
\mathbb {P} \left(\exists s \in \left(T _ {1}, T _ {2} \right], \frac {\sum_ {i = 1} ^ {s} W _ {i}}{s} + \sqrt {\frac {2 l}{s}} \leqslant 0\right) \leqslant \sum_ {j = T _ {1} + 1} ^ {T _ {2}} \frac {1}{\exp (l)} = \frac {T _ {2} - T _ {1}}{\exp (l)}. \tag {8}
$$