SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Next, we apply the peeling method to prove the second inequality. Take  $\beta > 1$  to be a constant. Let  $M = \left\lfloor \log_{\beta} \frac{T_2}{T_1} \right\rfloor$ , we apply peeling method on  $s$  and divide the time horizon over exponential grids  $[T_1, T_1\beta], [T_1\beta, T_1\beta^2], \dots, [T_1\beta^M, T_2]$  as follows:
$$
\begin{array}{l} \mathbb {P} \left(\exists s \in (T _ {1}, T _ {2} ], \frac {\sum_ {i = 1} ^ {s} W _ {i}}{s} + \sqrt {\frac {2 l}{s}} \leqslant 0\right) = \mathbb {P} \left(\exists s \in (T _ {1}, T _ {2} ], \sum_ {i = 1} ^ {s} W _ {i} + \sqrt {2 l s} \leqslant 0\right) \\ \leqslant \sum_ {j = 0} ^ {M} \mathbb {P} \left(\exists s \in \left[ T _ {1} \beta^ {j}, T _ {1} \beta^ {j + 1} \right], \sum_ {i = 1} ^ {s} W _ {i} + \sqrt {2 l s} \leqslant 0\right). \\ \end{array}
$$
Since at each grid,  $s \geqslant T_1\beta^j$ , so we can upper bound each probability as:
$$
\mathbb {P} \left(\exists s \in [ T _ {1} \beta^ {j}, T _ {1} \beta^ {j + 1} ], \sum_ {i = 1} ^ {s} W _ {i} + \sqrt {2 l s} \leqslant 0\right) \leqslant \mathbb {P} \left(\exists s \in [ T _ {1} \beta^ {j}, T _ {1} \beta^ {j + 1} ], \sum_ {i = 1} ^ {s} W _ {i} + \sqrt {2 T _ {1} \beta^ {j} l} \leqslant 0\right).
$$
Let  $\beta = \frac{l}{l - 1}$ , then according to the anytime concentration inequality from Lemma 1, we have:
$$
\mathbb {P} \left(\exists s \in [ T _ {1} \beta^ {j}, T _ {1} \beta^ {j + 1} ], \sum_ {i = 1} ^ {s} W _ {i} + \sqrt {2 T _ {1} \beta^ {j} l} \leqslant 0\right) \leqslant \exp \left(- \frac {2 T _ {1} \beta^ {j} l}{2 T _ {1} \beta^ {j + 1}}\right) = \exp \left(- \frac {l}{\beta}\right) = \frac {e}{\exp (l)}.
$$
Then, summing up the probabilities, we can bound the total probability as follows:
$$
\mathbb {P} \left(\exists s \in (T _ {1}, T _ {2} ], \frac {\sum_ {i = 1} ^ {s} W _ {i}}{s} + \sqrt {\frac {2 l}{s}} \leqslant 0\right) \leqslant \frac {e (M + 1)}{\exp (l)} \leqslant \frac {e (\log T _ {2} - \log T _ {1})}{\log \beta \exp (l)} + \frac {e}{\exp l},
$$
where the last inequality is due to the definition of  $M = \left\lfloor \log_{\beta} \frac{T_2}{T_1} \right\rfloor$ . Notice that when  $l \geqslant 2$ ,  $\log \beta = \log \left( \frac{l}{l - 1} \right) \geqslant \frac{1}{l}$ , so we can further upper bound the probability as:
$$
\mathbb {P} \left(\exists s \in \left(T _ {1}, T _ {2} \right], \frac {\sum_ {i = 1} ^ {s} W _ {i}}{s} + \sqrt {\frac {2 l}{s}} \leqslant 0\right) \leqslant \frac {e l (\log T _ {2} - \log T _ {1})}{\exp (l)} + \frac {e}{\exp l}. \tag {9}
$$
Finally, combining the two bounds Eq. (8) and Eq. (9) together, we can prove the first result of Lemma. 3 as follows:
$$
\mathbb {P} \left(\exists s \in (T _ {1}, T _ {2} ], \frac {\sum_ {i = 1} ^ {s} W _ {i}}{s} + \sqrt {\frac {2 l}{s}} \leqslant 0\right) \leqslant \min  \left\{\frac {T _ {2} - T _ {1}}{\exp (l)}, \frac {e l (\log T _ {2} - \log T _ {1})}{\exp (l)} + \frac {e}{\exp l} \right\}.
$$
Eq. (4b) of Lemma 3: Next, we prove Eq. (4b) of Lemma 3. Our result is only based on performing union bound, but one can also modify the proof of Eq. (4a) with peeling trick to prove Eq. (4b). However, the result from peeling trick is no better than simply performing union bound, at least not order-wise better. So for simplicity, we apply union bound to the probability as follows:
$$
\begin{array}{l} \mathbb {P} \left(\exists s \in [ T _ {1}, T _ {2} ], \frac {\sum_ {i = 1} ^ {s} W _ {i}}{s} + \sqrt {\frac {2 l}{s}} + \delta \leqslant 0\right) = \mathbb {P} \left(\exists s \in [ T _ {1}, T _ {2} ], \sum_ {i = 1} ^ {s} W _ {i} + \sqrt {2 l s} + \delta s \leqslant 0\right) \\ \leqslant \sum_ {j = T _ {1}} ^ {T _ {2}} \mathbb {P} \left(s = j, \sum_ {i = 1} ^ {s} W _ {i} + \sqrt {2 l s} + \delta s \leqslant 0\right) \\ = \sum_ {j = T _ {1}} ^ {T _ {2}} \mathbb {P} \left(\sum_ {i = 1} ^ {j} W _ {i} + \sqrt {2 j l} + \delta j \leqslant 0\right). \\ \end{array}
$$
Then, we can apply Hoeffding's inequality to upper bound each probability as follows:
$$
\begin{array}{l} \mathbb {P} \left(\sum_ {i = 1} ^ {j} W _ {i} + \sqrt {2 j l} + \delta j \leqslant 0\right) \leqslant \exp \left(- \frac {\left(\sqrt {2 j l} + \delta j\right) ^ {2}}{2 j}\right) \\ = \exp \left(- \frac {2 j l + 2 \sqrt {2 j l} \delta j + \delta^ {2} j ^ {2}}{2 j}\right) \\ \leqslant \frac {\exp \left(- \frac {\delta^ {2}}{2} j\right)}{\exp \left(l + \delta \sqrt {2 T _ {1} l}\right)}, \\ \end{array}
$$
where the last inequality is due to  $j \geqslant T_{1}$ . So summing up all the probabilities, we can bound the anytime concentration as:
$$
\begin{array}{l} \mathbb {P} \left(\exists s \in [ T _ {1}, T _ {2} ], \frac {\sum_ {i = 1} ^ {s} W _ {i}}{s} + \sqrt {\frac {2 l}{s}} + \delta \leqslant 0\right) \leqslant \sum_ {j = T _ {1}} ^ {T _ {2}} \frac {\exp \left(- \frac {\delta^ {2}}{2} j\right)}{\exp \left(l + \delta \sqrt {2 T _ {1} l}\right)} \\ \leqslant \frac {1}{\exp \left(l + \delta \sqrt {2 T _ {1} l}\right)} \frac {\exp \left(- \frac {\delta^ {2} T _ {1}}{2}\right)}{1 - \exp \left(- \frac {\delta^ {2}}{2}\right)} \\ \leqslant \frac {4}{\delta^ {2} \exp \left(\left(\sqrt {l} + \delta \sqrt {\frac {T _ {1}}{2}}\right) ^ {2}\right)}. \\ \end{array}
$$
where the last step is due to  $1 - e^{-\frac{\delta^2}{2}} \geqslant \frac{\delta^2}{2}$  when  $\delta \in [0, \sqrt{3}]$ . Then we finish the proof of Eq. (4b) of Lemma 3.
Eq. (4c) of Lemma 3: Finally, we prove Eq. (4a) of Lemma 3 which bounds the summation of probability for deviation events over the time horizon. Since  $l \geqslant \frac{T_1\delta^2}{2}$ , we define  $\gamma = \frac{2l}{\delta^2} \geqslant T_1$ . Then, we can bound the probabilities when  $s \leqslant \gamma$  by 1 as follows:
$$
\sum_ {s = T _ {1} + 1} ^ {T _ {2}} \mathbb {P} \left(\frac {\sum_ {i = 1} ^ {s} W _ {i}}{s} + \sqrt {\frac {2 l}{s}} \geqslant \delta\right) \leqslant \gamma - T _ {1} + \sum_ {s = [ \gamma ]} ^ {T _ {2}} \mathbb {P} \left(\frac {\sum_ {i = 1} ^ {s} W _ {i}}{s} + \sqrt {\frac {2 l}{s}} \geqslant \delta\right).
$$
When  $s \geqslant \lceil \gamma \rceil$ , we have:  $\sqrt{\frac{2l}{s}} = \delta \sqrt{\frac{\gamma}{s}}$ . So, for each probability inside the summation, we have:
$$
\mathbb {P} \left(\frac {\sum_ {i = 1} ^ {s} W _ {i}}{s} + \sqrt {\frac {2 l}{s}} \geqslant \delta\right) = \mathbb {P} \left(\frac {\sum_ {i = 1} ^ {s} W _ {i}}{s} \geqslant \delta - \sqrt {\frac {2 l}{s}}\right) \leqslant \mathbb {P} \left(\frac {\sum_ {i = 1} ^ {s} W _ {i}}{s} \geqslant \delta \left(1 - \sqrt {\frac {\gamma}{s}}\right)\right).
$$
Then, we can bound the probability with Hoeffding's inequality as follows:
$$
\mathbb {P} \left(\frac {\sum_ {i = 1} ^ {s} W _ {i}}{s} \geqslant \delta \left(1 - \sqrt {\frac {\gamma}{s}}\right)\right) \leqslant \exp \left(- \frac {s \delta^ {2}}{2} \left(1 - \sqrt {\frac {\gamma}{s}}\right) ^ {2}\right) = \exp \left(- \frac {\delta^ {2}}{2} \left(\sqrt {s} - \sqrt {\gamma}\right) ^ {2}\right).
$$
Notice that the upper bound function  $\exp \left(-\frac{\delta^2}{2} (\sqrt{s} -\sqrt{\gamma})^2\right)$  on the RHS is uni-modal when  $s\geqslant \gamma$  If a function  $f(s)$  is uni-modal, then we can bound the summation  $\sum_{s = \gamma}^{\infty}$  with the sum of max  $s$ $f(s)$  and integral  $\int_{\gamma}^{\infty}f(s)ds$  . Therefore, putting the upper bound back to the summation, we have:
$$
\begin{array}{l} \sum_ {s = \lceil \gamma \rceil} ^ {T _ {2}} \mathbb {P} \left(\frac {\sum_ {i = 1} ^ {s} W _ {i}}{s} + \sqrt {\frac {2 l}{s}} \geqslant \delta\right) \leqslant \sum_ {s = \lceil \gamma \rceil} ^ {T _ {2}} \exp \left(- \frac {\delta^ {2}}{2} \left(\sqrt {s} - \sqrt {\gamma}\right) ^ {2}\right) \\ \leqslant \sum_ {s = \lceil \gamma \rceil} ^ {\infty} \exp \left(- \frac {\delta^ {2}}{2} \left(\sqrt {s} - \sqrt {\gamma}\right) ^ {2}\right) \\ \leqslant 1 + \int_ {\gamma} ^ {\infty} \exp \left(- \frac {\delta^ {2}}{2} \left(\sqrt {s} - \sqrt {\gamma}\right) ^ {2}\right) d s \\ = 1 + \frac {2}{\delta^ {2}} + \frac {\sqrt {2 \pi \gamma}}{\delta}. \\ \end{array}
$$
So the whole term can be bounded by:
$$
\sum_ {s = T _ {1} + 1} ^ {T _ {2}} \mathbb {P} \left(\frac {\sum_ {i = 1} ^ {s} W _ {i}}{s} + \sqrt {\frac {2 l}{s}} \geqslant \delta\right) \leqslant \gamma - T _ {1} + 1 + \frac {2}{\delta^ {2}} + \frac {\sqrt {2 \pi \gamma}}{\delta} = \frac {2 l + \sqrt {4 \pi l} + 2}{\delta^ {2}} + 1 - T _ {1},
$$
which completes the proof of Lemma 3.
D.2 Proof of Lemma. 5
We only prove the inequality when  $\bar{\mu}_s\leqslant \mu$ . The other inequality can be proved exactly the same way. For every  $\lambda \in \mathbb{R}$ , let  $\phi_{\mu}(\lambda) = \log \mathbb{E}\bigl [\exp (\lambda X_1)\bigr ]$  which is well-defined and finite by assumption. Let  $W_0^{\lambda} = 1$  and for  $s\geqslant 1$ , we define  $W_{t}^{\lambda} = \exp \left(\lambda S_{s} - s\phi_{\mu}(\lambda)\right)$ . We show that  $\left(W_s^\lambda\right)_{s\geqslant 1}$  is a martingale with respect to the  $\sigma$ -field  $\mathcal{F}_s = \sigma (W_1,\dots ,W_s)$ . In fact,
$$
\begin{array}{l} \mathbb {E} \left[ W _ {s + 1} ^ {\lambda} | \mathcal {F} _ {s} \right] = \mathbb {E} \left[ \exp \left(\lambda S _ {s} + \lambda W _ {s + 1} - s \phi_ {\mu} (\lambda) - \phi_ {\mu} (\lambda)\right) | \mathcal {F} _ {s} \right] \\ = \exp (\lambda S _ {s} - s \phi_ {\mu} (\lambda)) \mathbb {E} [ \exp (\lambda W _ {s + 1} - \phi_ {\mu} (\lambda)) | \mathcal {F} _ {s} ] \\ = W _ {s} ^ {\lambda} \frac {\mathbb {E} \left[ \exp \left(\lambda W _ {s + 1}\right) \right]}{\mathbb {E} \exp \left[ \lambda X _ {1} \right]} \\ = W _ {s} ^ {\lambda}, \\ \end{array}
$$
where the second equality is because  $S_{s}$  is measurable w.r.t.  $\mathcal{F}_s$ , and the third equality uses the fact that  $W_{s + 1}$  is independent w.r.t.  $\mathcal{F}_s$ . The last equality is due to the i.i.d. nature of  $W_{1}$  and  $W_{s + 1}$ . Let  $x\in [0,\mu ]$  be such that  $\mathsf{KL}(x,\mu) = \delta /s$ , and let  $\lambda (x) = \theta^x -\theta^\mu$ . It is worth noting that since  $\theta^{\mu}$  is a monotonically non-decreasing function since its inverse function  $\mu = b^{\prime}(\theta)$  is monotonically non-decreasing. So we have  $\lambda (x)\leqslant 0$  since  $x\leqslant \mu$ . Observe that:
$$
\bar {\mu} _ {s} \leqslant \mu , \quad \mathsf {K L} \left(\bar {\mu} _ {s}, \mu\right) \geqslant \frac {\delta}{s}, \quad \text {a n d ,} \quad x \leqslant \mu , \quad \mathsf {K L} \left(x, \mu\right) = \frac {\delta}{s}.
$$
Then it holds that  $x \geqslant \bar{\mu}_s$ . Notice that for natural parameter exponential family,  $\phi_{\mu}(\lambda) = b(\lambda + \theta^{\mu}) - b(\theta^{\mu})$ . Hence on the event  $\{\bar{\mu}_s < \mu\} \cap \{sKL(\bar{\mu}_s, \mu) \geqslant \delta\}$ , we have:
$$
\lambda (x) \bar {\mu} _ {s} - \phi_ {\mu} (\lambda (x)) \geqslant \lambda (x) x - \phi_ {\mu} (\lambda (x)) = x (\theta^ {x} - \theta^ {\mu}) - b (\theta^ {x}) + b (\theta^ {\mu}) = \mathsf {K L} (x, \mu) = \frac {\delta}{s},
$$
where the first inequality is because  $\lambda (x) < 0$ , and the second last equality uses the expression of KL divergence for natural exponential families. Putting everything together, we have:
$$
\begin{array}{l} \mathbb {P} \left(\bar {\mu} _ {s} \leqslant \mu , s \mathrm {K L} \left(\bar {\mu} _ {s}, \mu\right) \geqslant \delta\right) \leqslant \mathbb {P} \left(\lambda (x) \bar {\mu} _ {s} - \phi_ {\mu} (\lambda (x)) \geqslant \frac {\delta}{s}\right) \\ = \mathbb {P} \left(W _ {s} ^ {\lambda (x)} \geqslant \delta\right) \\ \leqslant \mathbb {E} \left[ W _ {s} ^ {\lambda (x)} \right] \exp (- \delta), \\ \end{array}
$$
where the last inequality uses Markov inequality. Since  $W_{s}^{\lambda (x)}$  is a martingale, so we have:
$$
\mathbb {P} \left(\bar {\mu} _ {s} \leqslant \mu , s \mathsf {K L} (\bar {\mu} _ {s}, \mu) \geqslant \delta\right) \leqslant \mathbb {E} \big [ W _ {0} ^ {\lambda (x)} \big ] \exp (- \delta) = \exp (- \delta).
$$
D.3 Proof of Lemma 6
We only prove the inequality when  $\bar{\mu}_s\leqslant \mu$ . The other inequality can be proved exactly the same way. The proof of Lemma/ 6 partly resembles the Proof of Lemma. 5. However, to show an any-time concentration bound, we need either the union bound or a peeling trick. We first apply union bound as follows:
$$
\begin{array}{l} \mathbb {P} \left(\exists s \in \left(T _ {1}, T _ {2} \right], \bar {\mu} _ {s} \leqslant \mu , s \mathrm {K L} (\bar {\mu} _ {s}, \mu) \geqslant l\right) \leqslant \sum_ {s = T _ {1} + 1} ^ {T _ {2}} \mathbb {P} \left(\bar {\mu} _ {s} \leqslant \mu , s \mathrm {K L} (\bar {\mu} _ {s}, \mu) \geqslant l\right) \\ \leqslant \left(T _ {2} - T _ {1}\right) \exp (- l), \\ \end{array}
$$
which obtains the first term in the RHS of Lemma 6. Next, we apply the peeling trick. For every  $\lambda \in \mathbb{R}$ , let  $\phi_{\mu}(\lambda) = \log \mathbb{E}[\exp (\lambda X_1)]$  which is well-defined and finite by assumption. Let  $W_0^\lambda = 1$  and for  $s\geqslant 1$ , we define  $W_{t}^{\lambda} = \exp \left(\lambda S_{s} - s\phi_{\mu}(\lambda)\right)$ . Recall that  $\left(W_s^\lambda\right)_{s\geqslant 1}$  is a martingale with respect to the  $\sigma$ -field  $\mathcal{F}_s = \sigma (W_1,\dots ,W_s)$ . Take  $\beta >1$  to be a constant. Let  $M = \lfloor \log_{\beta}\frac{T_2}{T_1}\rfloor$ , we apply peeling method on  $s$  and divide the time horizon over exponential grids  $[T_1,T_1\beta ],[T_1\beta ,T_1\beta^2 ],\dots ,[T_1\beta^M,T_2]$  as follows:
$$
\mathbb {P} \left(\exists s \in \left(T _ {1}, T _ {2} \right], \bar {\mu} _ {s} \leqslant \mu , s \mathsf {K L} \left(\bar {\mu} _ {s}, \mu\right) \geqslant l\right) \leqslant \sum_ {i = 0} ^ {M} \mathbb {P} \left(\exists s \in \left[ T _ {1} \beta^ {i}, T _ {1} \beta^ {i + 1} \right], \bar {\mu} _ {s} \leqslant \mu , s \mathsf {K L} \left(\bar {\mu} _ {s}, \mu\right) \geqslant l\right).
$$
Let  $s_i = T_1\beta^i$  and let  $x \leqslant \mu$  such that  $s\mathsf{KL}(x,\mu) = l$ . Let  $\lambda(x) = \theta^x - \theta^\mu < 0$ . Then, we have  $\mathsf{KL}(x,\mu) = \lambda(x)x - \phi_\mu(\lambda(x))$ . Consider  $z$  such that  $z_i < \mu$  and  $\mathsf{KL}(z_i,\mu) = \frac{l}{s_i}$ , so we have when  $s \in [s_i, s_{i+1}]$
$$
\mathsf {K L} (\bar {\mu} _ {s}, \mu) \geqslant \frac {l}{s} \geqslant \frac {l}{s _ {i + 1}} = \mathsf {K L} (z _ {i + 1}, \mu).
$$
So we can conclude that  $\bar{\mu}_s\lesssim z_{i + 1}$  Also, we have:
$$
\mathsf {K L} (z _ {i + 1}, \mu) = \frac {l}{s _ {i + 1}} = \frac {1}{\beta} \frac {l}{s _ {i}} \geqslant \frac {1}{\beta} \frac {l}{s}.
$$
$$
\lambda (z _ {i + 1}) \bar {\mu} _ {s} - \phi_ {\mu} (\lambda (z _ {i + 1})) \geqslant \lambda (z _ {i + 1}) z _ {i + 1} - \phi_ {\mu} (\lambda (z _ {i + 1})) = \mathsf {K L} (z _ {i + 1}, \mu) \geqslant \frac {l}{\beta s}.
$$
So we can bound each probability as:
$$
\begin{array}{l} \mathbb {P} \left(\exists s \in \left[ z _ {i}, z _ {i + 1} \right], \bar {\mu} _ {s} \leqslant \mu , s \mathrm {K L} (\bar {\mu} _ {s}, \mu) \geqslant \delta\right) \leqslant \mathbb {P} \left(\lambda \left(z _ {i + 1}\right) \bar {\mu} _ {s} - \phi_ {\mu} (\lambda \left(z _ {i + 1}\right)) \geqslant \frac {l}{\beta s}\right) \\ = \mathbb {P} \left(W _ {s} ^ {\lambda (z _ {i + 1})} \geqslant \exp \left(\frac {l}{\beta}\right)\right) \\ \leqslant \mathbb {E} \left[ W _ {s} ^ {\lambda (z _ {i + 1})} \right] \exp \left(- \frac {l}{\beta}\right), \\ \end{array}
$$
where the last inequality uses Markov inequality. Since  $W_{s}^{\lambda (z_{i + 1})}$  is a martingale, so we have:
$$
\mathbb {P} \left(\exists s \in \left[ z _ {i}, z _ {i + 1} \right], \bar {\mu} _ {s} \leqslant \mu , s \mathrm {K L} \left(\bar {\mu} _ {s}, \mu\right) \geqslant \delta\right) \leqslant \mathbb {E} \left[ W _ {0} ^ {\lambda \left(z _ {i + 1}\right)} \right] \exp \left(- \frac {l}{\beta}\right) = \frac {e}{\exp (l)},
$$
where in the last step, we choose  $\beta = \frac{l}{l - 1}$ . Then, summing up the probabilities, we can bound the total probability as follows:
$$
\mathbb {P} \left(\exists s \in (T _ {1}, T _ {2} ], \bar {\mu} _ {s} \leqslant \mu , s \mathsf {K L} (\bar {\mu} _ {s}, \mu) \geqslant l\right) \leqslant \sum_ {j = 0} ^ {M} \frac {e}{\exp (l)} = \frac {e (M + 1)}{\exp (l)} \leqslant \frac {e (\log T _ {2} - \log T _ {1})}{\log \beta \exp (l)} + \frac {e}{\exp l},
$$
$$
\mathbb {P} \left(\exists s \in \left(T _ {1}, T _ {2} \right], \bar {\mu} _ {s} \leqslant \mu , s \mathrm {K L} \left(\bar {\mu} _ {s}, \mu\right) \geqslant l\right) \leqslant \frac {e l \left(\log T _ {2} - \log T _ {1}\right)}{\exp (l)} + \frac {e}{\exp l}. \tag {10}
$$
D.4 Proof of Lemma. 7
Our proof is based on the analysis of Theorem 2 of [12]. For any pair of means  $x, y \in [0,1]$ , define  $\mathsf{KL}^{+}(x,y) = \mathsf{KL}(x,y)\mathbb{1}_{x < y}$ . Then for a fixed  $s$ , under event  $\{\mathsf{UCB}_s \geqslant \mu'\}$  we have either  $\mu' < \bar{\mu}_s$  or  $\mu' > \bar{\mu}_s$  but  $s\mathsf{KL}(\bar{\mu}_s, \mu') \leqslant l$ . So in general we can conclude that  $s\mathsf{KL}^{+}(\bar{\mu}_s, \mu') \leqslant l$ . Then we can bound the sum of probabilities as follows:
$$
\sum_ {s = 1} ^ {T _ {1}} \mathbb {P} \left(\mathrm {U C B} _ {s} \geqslant \mu^ {\prime}\right) \leqslant \sum_ {s = 1} ^ {T _ {1}} \mathbb {P} \left(s \mathrm {K L} ^ {+} \left(\bar {\mu} _ {s}, \mu^ {\prime}\right) \leqslant l\right).
$$
Define  $\gamma = \frac{(1 + \varepsilon)l}{\mathrm{KL}^{+}(\mu,\mu^{\prime})} = \frac{(1 + \varepsilon)l}{\mathrm{KL}(\mu,\mu^{\prime})}$ , then if  $T_{1} > \gamma$ , we can bound the first  $\gamma$  terms in the summation with 1. If otherwise  $T_{1} \leqslant \gamma$ , the whole summation is bounded by  $\gamma$ . So without loss of generality, assume  $\gamma < T_{1}$  and let  $\varepsilon > 0$  be a constant, we have:
$$
\begin{array}{l} \sum_ {s = 1} ^ {T _ {1}} \mathbb {P} \left(s \mathrm {K L} ^ {+} \left(\bar {\mu} _ {s}, \mu^ {\prime}\right) \leqslant l\right) \leqslant \gamma + \sum_ {s = \lceil \gamma \rceil} ^ {T _ {1}} \mathbb {P} \left(s \mathrm {K L} ^ {+} \left(\bar {\mu} _ {s}, \mu^ {\prime}\right) \leqslant l\right) \\ \leqslant \gamma + \sum_ {s = \lceil \gamma \rceil} ^ {T _ {1}} \mathbb {P} \left(\gamma K L ^ {+} \left(\bar {\mu} _ {s}, \mu^ {\prime}\right) \leqslant l\right) \\ = \gamma + \sum_ {s = \lceil \gamma \rceil} ^ {T _ {1}} \mathbb {P} \left(\mathsf {K L} ^ {+} (\bar {\mu} _ {s}, \mu^ {\prime}) \leqslant \frac {\mathsf {K L} (\mu , \mu^ {\prime})}{1 + \varepsilon}\right), \\ \end{array}
$$
where the second inequality is due to  $\mathsf{KL}^{+}(\bar{\mu}_{s},\mu^{\prime}) > 0$ . For any  $s$ , let  $r(\varepsilon)\in (\mu ,\mu^{\prime})$  such that  $\mathsf{KL}(r(\varepsilon),\mu^{\prime}) = \frac{\mathsf{KL}(\mu,\mu^{\prime})}{1 + \varepsilon}$ . If  $\mathsf{KL}^{+}(\bar{\mu}_{s},\mu^{\prime})\leqslant \frac{\mathsf{KL}(\mu,\mu^{\prime})}{1 + \varepsilon}$ , we have  $\bar{\mu}_s\geqslant r(\varepsilon)$ . Hence,
$$
\mathbb {P} \left(\mathsf {K L} ^ {+} (\bar {\mu} _ {s}, \mu^ {\prime}) \leqslant \frac {\mathsf {K L} (\mu , \mu^ {\prime})}{1 + \varepsilon}\right) \leqslant \mathbb {P} \left(\bar {\mu} _ {s} \geqslant \mu , \mathsf {K L} (\bar {\mu} _ {s}, \mu) \geqslant \mathsf {K L} (r (\varepsilon), \mu)\right) \leqslant \exp (- s \mathsf {K L} (r (\varepsilon), \mu)),
$$
where the last inequality uses Lemma 5. So we can bound the sum of probabilities as follows:
$$
\sum_ {s = 1} ^ {T _ {1}} \mathbb {P} \left(s \mathrm {K L} ^ {+} (\bar {\mu} _ {s}, \mu^ {\prime}) \leqslant l\right) \leqslant \gamma + \sum_ {s = \lceil \gamma \rceil} ^ {\infty} \exp (- s \mathrm {K L} (r (\varepsilon), \mu)) \leqslant \gamma + \frac {\exp (- \gamma \mathrm {K L} (r (\varepsilon) , \mu))}{1 - \exp (- \mathrm {K L} (r (\varepsilon) , \mu))}.
$$
Notice that  $\exp (-\gamma \mathsf{KL}(r(\varepsilon),\mu)) = \exp \left(-l\frac{(1 + \varepsilon)\mathsf{KL}(r(\varepsilon),\mu)}{\mathsf{KL}(\mu,\mu')}\right) \leqslant T^{-\beta_1(\varepsilon)}$ , where  $\beta_{1}(\varepsilon) = \frac{(1 + \varepsilon)\mathsf{KL}(r(\varepsilon),\mu)}{\mathsf{KL}(\mu,\mu')}$ . Let  $\beta_{2}(\varepsilon) = \frac{1}{1 - \exp(-\mathsf{KL}(r(\varepsilon),\mu))}$ . It is easy to check that  $r(\varepsilon) = \mu + \mathcal{O}(\varepsilon)$ , so we have  $\beta_{1}(\varepsilon) = \mathcal{O}(\varepsilon^{2})$  and  $\beta_{2}(\varepsilon) = \mathcal{O}(\varepsilon^{-2})$ . So we have:
$$
\sum_ {s = 1} ^ {T _ {1}} \mathbb {P} \left(\mathrm {U C B} _ {s} \geqslant \mu^ {\prime}\right) \leqslant \sum_ {s = 1} ^ {T _ {1}} \mathbb {P} \left(s \mathrm {K L} ^ {+} \left(\bar {\mu} _ {s}, \mu^ {\prime}\right) \leqslant l\right) \leqslant \frac {(1 + \varepsilon) l}{\mathrm {K L} \left(\mu , \mu^ {\prime}\right)} + \frac {\beta_ {2} (\varepsilon)}{T ^ {\beta_ {1} (\varepsilon)}}.
$$
Remark: Let  $\varepsilon = l^{-\frac{1}{4}}$ , we have:
$$
\sum_ {s = 1} ^ {T _ {1}} \mathbb {P} \left(\mathrm {U C B} _ {s} \geqslant \mu^ {\prime}\right) \leqslant \frac {(1 + \varepsilon) l}{\mathrm {K L} (\mu , \mu^ {\prime})} + \frac {\beta_ {2} (\varepsilon)}{T ^ {\beta_ {1} (\varepsilon)}} \leqslant \frac {l + l ^ {\frac {3}{4}}}{\mathrm {K L} (\mu , \mu^ {\prime})} + \mathcal {O} \left(\frac {\sqrt {l}}{\exp \left(\sqrt {l}\right)}\right).
$$
(a) Gaussian Bandits
Figure 2: Regret performance in bandit models with 4 arms.
(b) Bernoulli Bandits
E Numerical Experiments Beyond 2-Armed Bandits
In this section, we present numerical experiments based on bandits with more than two arms. We consider a four armed bandit model with expected rewards [0.7, 0.2, 0.2, 0.2]. We study the cumulative regret performance of all our proposed algorithms, i.e., EOCP in Algorithm 1, EOCP-UG in Algorithm 2, and KL-EOCP in Algorithm 3, with both Gaussian and Bernoulli rewards. Since some baseline algorithms we chose in Section 6 (e.g., DETC and BAI-ETC) does not directly generalize to the multi-arm model, we replace them with DETC-K [23] and TAS-ETC respectively, where the TAS-ETC algorithm first uses the famous Track and Stop [36] algorithm to identify the best arm and commits to it for the rest of horizon. Additional to all other baselines we used in Sec. 6, we also compare our algorithms to the Action-Elimination algorithm [42], which provably has both  $\mathcal{O}(\log T)$  regret and  $\mathcal{O}(\log T)$  commitment time. We choose the reward of all sub-optimal arms to be the same value so that BAI based algorithms can utilize a closed form expression to solve the optimal arm pull fraction  $w^{*}$  and does not need to solve a non-convex minimax problem. We choose  $\Delta_{\mathrm{lb}}$  to be 0.5 for EOCP and KL-EOCP, and the results are shown in Fig. 2 and averaged over  $10^{4}$  iterations.
In both Gaussian and Bernoulli bandits, we observe very similar trends compared to Fig. 1 in two-armed models. The TAS-ETC algorithm and DETC-K algorithm incurs high regret over the total time horizon due to their aggressive exploration at the beginning of the trials, similar to the performance of BAI-ETC and DETC in 2-armed models. The Action Elimination algorithm incurs even larger regret because of its almost uniform sampling rule at the beginning of the trials. It selects sub-optimal actions even more often than BAI based algorithms and could not identify the optimal action as quickly. On the other hand, our proposed algorithms take a more delicate exploration strategy and result in lower regret, which indicates that the EOCP algorithm and its variants generalize well beyond two-armed bandit models. The over-exploration phenomenon is also witnessed in this experiment the UCB algorithm and the KL-UCB algorithm continues to explore sub-optimal actions even after the EOCP algorithms have identified the best action. The regret of UCB algorithms continues to increase during the whole time horizon, which shows that over-exploration is a common and fundamental issue in optimistic algorithms. Preventing over-exploration will significantly boost the performance of algorithms not only in bandits but also in other online decision-making problems.