SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {P} \left(\mathrm {L C B} _ {T _ {\mathrm {c}}} (a) \geqslant \mathrm {L C B} _ {T _ {\mathrm {c}}} (1), \tilde {a} = 1\right) \\ \leqslant \mathbb {P} \left(\bar {r} _ {T _ {\mathrm {c}}} (a) - \sqrt {\frac {2 l}{N _ {T _ {\mathrm {c}}} (a)}} \geqslant \bar {r} _ {T _ {\mathrm {c}}} (1) - \sqrt {\frac {2 l}{N _ {T _ {\mathrm {c}}} (1)}}, N _ {T _ {\mathrm {c}}} (1) \geqslant l N _ {T _ {\mathrm {c}}} (a)\right). \\ \end{array}
$$
Similarly, we separate the two random variables as follows:
$$
\begin{array}{l} \mathbb {P} \left(\mathrm {L C B} _ {T _ {\mathrm {c}}} (a) \geqslant \mathrm {L C B} _ {T _ {\mathrm {c}}} (1), \tilde {a} = 1\right) \\ \leqslant \mathbb {P} \left(\bar {r} _ {T _ {\mathrm {c}}} (a) - \sqrt {\frac {2 l}{N _ {T _ {\mathrm {c}}} (a)}} \geqslant \mu_ {1} - 2 \sqrt {\frac {2 l}{N _ {T _ {\mathrm {c}}} (1)}}, N _ {T _ {\mathrm {c}}} (1) \geqslant l N _ {T _ {\mathrm {c}}} (a)\right) \\ + \mathbb {P} \left(\bar {r} _ {T _ {\mathrm {c}}} (1) \leqslant \mu_ {1} - \sqrt {\frac {2 l}{N _ {T _ {\mathrm {c}}} (1)}}\right) \\ \leqslant \underbrace {\mathbb {P} \left(\left(\mu_ {a} - \bar {r} _ {T _ {\mathrm {c}}} (a)\right) + \left(1 - \frac {2}{\sqrt {l}}\right) \sqrt {\frac {2 l}{N _ {T _ {\mathrm {c}}} (a)}} + \Delta_ {a} \leqslant 0\right)} _ {B _ {3}} + \underbrace {\mathbb {P} \left(\bar {r} _ {T _ {\mathrm {c}}} (1) \leqslant \mu_ {1} - \sqrt {\frac {2 l}{N _ {T _ {\mathrm {c}}} (1)}}\right)} _ {B _ {4}}. \\ \end{array}
$$
For  $B_{4}$ , notice that  $T_{\mathrm{c}}$  is a random stopping time, so we bound the probability over all time step when the number of pulls is larger than  $l$  as follows:
$$
B _ {4} \leqslant \mathbb {P} \left(\exists s \in [ l, T ], \bar {r} _ {s, 1} - \mu_ {1} + \sqrt {\frac {2 l}{s}} \leqslant 0\right) = \mathbb {P} \left(\exists s \in [ l, T ], \frac {\sum_ {i = 1} ^ {s} \left(W _ {1 , i} - \mu_ {1}\right)}{s} + \sqrt {\frac {2 l}{s}} \leqslant 0\right).
$$
Since  $W_{1,i} - \mu_1$  is 1-subgaussian, we can use Eq. (4a) from the concentration Lemma. 3 to bound  $B_4$  as follows:
$$
B _ {4} \leqslant \mathbb {P} \left(\exists s \in [ l, T ], \frac {\sum_ {i = 1} ^ {s} (W _ {1 , i} - \mu_ {1})}{s} + \sqrt {\frac {2 l}{s}} \leqslant 0\right) \leqslant \frac {e l \log T + e}{\exp l} = o \left(\frac {1}{T}\right),
$$
where the last inequality is due to  $\sqrt{2\log T} \leqslant \log T$  when  $T \geqslant 9$  and  $\exp(l) \geqslant T\log^4 T$ . On the other hand, bounding  $B_3$  is similar to the proof of probability upper bound regarding  $\tilde{a}$ . recall that  $\alpha = \left(1 - \frac{2}{\sqrt{l}}\right)$ , and notice that  $T_c$  is a random variable, so we use a union over all possible arm pulls
to bound the event on time step  $T_{\mathrm{c}}$  as follows:
$$
\begin{array}{l} B _ {3} \leqslant \mathbb {P} \left(\exists s \leqslant T, (\mu_ {a} - \bar {r} _ {s, a}) + \sqrt {\frac {2 \alpha^ {2} l}{s}} + \Delta_ {a} \leqslant 0\right) \\ \leqslant \mathbb {P} \left(\exists s \leqslant T, \frac {\sum_ {i = 1} ^ {s} \left(\mu_ {a} - W _ {a , i}\right)}{s} + \sqrt {\frac {2 \alpha^ {2} l}{s}} + \Delta_ {a} \leqslant 0\right) \\ \leqslant \frac {4}{\Delta_ {a} ^ {2} \exp (\alpha^ {2} l)}, \\ \end{array}
$$
where the last inequality uses the second concentration inequality Eq. (4b) from Lemma 3 and the fact that  $\mu_{a} - W_{a,i}$  is 1-subgaussian. Then when  $T$  is large enough, we have:
$$
B _ {3} \leqslant \frac {4}{\Delta_ {a} ^ {2} T}.
$$
Combining  $B_{3}$  and  $B_{4}$ , we can bound the probability that we select the wrong arm as:
$$
\mathbb {P} \left(\mathsf {L C B} _ {T _ {\mathrm {c}}} (a) \geqslant \mathsf {L C B} _ {T _ {\mathrm {c}}} (1), \tilde {a} = 1\right) \leqslant B _ {3} + B _ {4} \leqslant \frac {4}{\Delta_ {a} ^ {2} T} + o \left(\frac {1}{T}\right) \leqslant \mathcal {O} \left(\frac {1}{T}\right).
$$
Therefore, we finally bound  $I_{2}$  as follows:
$$
I _ {2} = \mathbb {P} (\hat {a} = a) \leqslant \mathbb {P} (\hat {a} = a, \tilde {a} = 1) + \mathbb {P} (\tilde {a} \neq 1) = \mathcal {O} \left(\frac {1}{T}\right).
$$
Here, the bound on  $I_{2}$  shows that the confidence level of our algorithm is  $\mathcal{O}(T^{-1})$  indicated in Corollary 2. Combining  $I_{1}$  and  $I_{2}$ , we can bound  $\mathbb{E}[N_a(T)]$  in finite time as follows:
$$
\mathbb {E} \left[ N _ {a} (T) \right] \leqslant I _ {1} + T I _ {2} \leqslant \frac {2 \log T}{\Delta_ {a} ^ {2}} + \frac {(8 + \sqrt {2 0 \pi}) \sqrt {\log T}}{\Delta_ {a} ^ {2}} + \mathcal {O} (1).
$$
So by the regret decomposition lemma, we can bound the regret performance as:
$$
\operatorname{Reg}_{\boldsymbol{\mu}}^{\text{EOCP-UG}}(T)\leqslant \sum_{a:\Delta_{a} > 0}\left(\frac{2\log T}{\Delta_{a}} +\frac{(8 + \sqrt{20\pi})\sqrt{\log T}}{\Delta_{a}}\right) + \mathcal{O}(1).
$$
B.4 Proof of Lemma. 4 and Corollary. 2
Consider a specific sub-optimal arm  $a$ . We first show that it can only be pulled  $\mathcal{O}(\log T)$  with high probability during the exploration phase due to the dynamic of our exploration strategy, i.e., the UCB exploration strategy. To be specific, we intend to show the following inequality:
$$
\mathbb {P} \left(\exists t \leqslant T _ {\mathrm {c}}, N _ {t} (a) \geqslant \frac {8 l}{\Delta_ {a} ^ {2}} + 1\right) \leqslant \frac {1 0 e}{T \log^ {2} T}.
$$
Since during the exploration phase, we select the arm with the largest UCB to explore, if there exists a time  $t$  which the number of pulls for action  $a$  is larger than  $\frac{8l}{\Delta_a^2} + 1$ , which means that there exists a time  $t' \in [A, t)$ , where at time  $t' + 1$  the number of pulls for previous rounds  $N_{t'}(a)$  is exactly  $\frac{8l}{\Delta_a^2}$  and arm  $a$  has the largest UCB. Therefore, we can bound the probability as:
$$
\mathbb {P} \left(\exists t \leqslant T _ {c}, N _ {t} (a) \geqslant \frac {8 l}{\Delta_ {a} ^ {2}} + 1\right) \leqslant \mathbb {P} \left(\exists t ^ {\prime} \leqslant t \leqslant T, N _ {t ^ {\prime}} (a) = \frac {8 l}{\Delta_ {a} ^ {2}}, a = \underset {a ^ {\prime}} {\arg \max } \mathsf {U C B} _ {t ^ {\prime}} (a ^ {\prime})\right).
$$
Therefore, the UCB of arm  $a$  should be larger than the UCB of the optimal arm 1, i.e.,  $\mathsf{UCB}_{t'}(a) \geqslant \mathsf{UCB}_{t'}(1)$ . So we can further derive an upper bound as follows:
$$
\begin{array}{l} \mathbb {P} \left(\exists t ^ {\prime} \leqslant t \leqslant T, N _ {t ^ {\prime}} (a) = \left[ \frac {8 l}{\Delta_ {a} ^ {2}} \right], a = \underset {a ^ {\prime}} {\arg \max } \cup \mathrm {C B} _ {t ^ {\prime}} \left(a ^ {\prime}\right)\right) \\ \leqslant \mathbb {P} \left(\exists t ^ {\prime} \leqslant T, N _ {t ^ {\prime}} (a) = \left[ \frac {8 l}{\Delta_ {a} ^ {2}} \right], \cup C B _ {t ^ {\prime}} (a) \geqslant U C B _ {t ^ {\prime}} (1)\right) \\ = \mathbb {P} \left(\exists t ^ {\prime} \leqslant T, N _ {t ^ {\prime}} (a) = \left\lceil \frac {8 l}{\Delta_ {a} ^ {2}} \right\rceil , \bar {r} _ {t ^ {\prime}} (a) + \sqrt {\frac {2 l}{N _ {t ^ {\prime}} (a)}} \geqslant \bar {r} _ {t ^ {\prime}} (1) + \sqrt {\frac {2 l}{N _ {t ^ {\prime}} (1)}}\right) \\ \leqslant \mathbb {P} \left(\exists t ^ {\prime} \leqslant T, \bar {r} _ {\left[ \frac {8 l}{\Delta_ {a} ^ {2}} \right], a} + \frac {\Delta_ {a}}{2} \geqslant \bar {r} _ {t ^ {\prime}} (1) + \sqrt {\frac {2 l}{N _ {t ^ {\prime}} (1)}}\right). \\ \end{array}
$$
Then, we can separate the two empirical means  $\bar{r}\left[\frac{8l}{\Delta_a^2}\right], a$  and  $\bar{r}_{t'}(1)$  with a union bound as follows:
$$
\begin{array}{l} \mathbb {P} \left(\exists t ^ {\prime} \leqslant T, \bar {r} _ {\left[ \frac {\mathrm {s} l}{\Delta_ {a} ^ {2}} \right], a} + \frac {\Delta_ {a}}{2} \geqslant \bar {r} _ {t ^ {\prime}} (1) + \sqrt {\frac {2 l}{N _ {t ^ {\prime}} (1)}}\right) \\ \leqslant \underbrace {\mathbb {P} \left(\exists t ^ {\prime} \leqslant T , \bar {r} _ {t ^ {\prime}} (1) + \sqrt {\frac {2 l}{N _ {t ^ {\prime}} (1)}} \leqslant \mu_ {a} + \Delta_ {a}\right)} _ {A _ {1}} + \underbrace {\mathbb {P} \left(\bar {r} _ {\left[ \frac {8 l}{\Delta_ {a} ^ {2}} \right] , a} \geqslant \mu_ {a} + \frac {\Delta_ {a}}{2}\right)} _ {A _ {2}}. \\ \end{array}
$$
The term  $A_{2}$  can be easily bounded through Hoeffding's inequality as:
$$
A _ {2} = \mathbb {P} \left(\frac {\sum_ {i = 1} ^ {\left\lceil \frac {8 l}{\Delta_ {a} ^ {2}} \right\rceil} \left(W _ {a , i} - \mu_ {a}\right)}{\left\lceil \frac {8 l}{\Delta_ {a} ^ {2}} \right\rceil} \geqslant \frac {\Delta_ {a}}{2}\right) \leqslant \exp \left(- \left\lceil \frac {8 l}{\Delta_ {a} ^ {2}} \right\rceil \frac {\Delta_ {a} ^ {2}}{8}\right) \leqslant \frac {1}{\exp (l)},
$$
where the first inequality is due to the fact that  $W_{a,i} - \mu_a$  is 1-subgaussian. For term  $A_1$ , we notice that the empirical estimation  $\bar{r}_{t'}(1)$  and the counter  $N_{t'}(1)$  will remain the same if there is no pull for arm 1, so we can switch the count of time step  $t'$  to the count of number of pulls for  $N_t'(1)$ . To be specific,
$$
A _ {1} \leqslant \mathbb {P} \left(\exists s \leqslant T, \bar {r} _ {s, 1} + \sqrt {\frac {2 l}{s}} \leqslant \mu_ {a} + \Delta_ {a}\right) = \mathbb {P} \left(\exists s \leqslant T, \frac {\sum_ {i = 1} ^ {s} \left(W _ {1 , i} - \mu_ {1}\right)}{s} + \sqrt {\frac {2 l}{s}} \leqslant 0\right).
$$
Since  $W_{1,i} - \mu_1$  is 1-subgaussian, we can use the concentration result Eq. (4a) of Lemma 3 to bound  $A_1$  as follows:
$$
\mathbb {P} \left(\exists s \leqslant T, \frac {\sum_ {i = 1} ^ {s} \left(W _ {1 , i} - \mu_ {1}\right)}{s} + \sqrt {\frac {2 l}{s}} \leqslant 0\right) \leqslant \frac {e l \log T + e}{\exp (l)}.
$$
Therefore, we have can bound the probability of overpull for sub-optimal arm  $a$  as follows:
$$
\mathbb {P} \left(\exists t \leqslant T _ {\mathrm {c}}, N _ {t} (a) \geqslant \frac {8 l}{\Delta_ {a} ^ {2}} + 1\right) \leqslant A _ {1} + A _ {2} \leqslant \frac {e (\log T + 4 \sqrt {2 \log T}) \log T + 2 e}{\exp (l)} \leqslant \frac {1 0 e}{T \log^ {2} T},
$$
where the third inequality is due to  $\log T \geqslant \sqrt{2\log T}$  when  $T \geqslant 9$  and the last inequality is due to  $\exp(l) \geqslant T\log^4 T$ . Recall that in Algorithm 2, the exploration phase will stop if there exists an arm  $\tilde{a}$  whose pulls  $N_{T_c}(\tilde{a})$  at the stopping time have exceeds  $l$  times of all other arms, i.e.,
$$
l \max  _ {a \neq \tilde {a}} N _ {T _ {\mathrm {c}}} (a) + 2 \geqslant N _ {T _ {\mathrm {c}}} (\tilde {a}) > l \max  _ {a \neq \tilde {a}} N _ {T _ {\mathrm {c}}} (a) + 1.
$$
So if  $T_{\mathrm{c}}$  is larger than  $\sum_{a:\Delta_a > 0} \frac{8(l + 1)^2}{\Delta_a^2} + A(l + 2)$ , it means that there exists at least one sub-optimal arm  $a$  whose pulls  $N_{T_{\mathrm{c}}}(\alpha)$  is larger than  $\frac{8l}{\Delta_a^2} + 1$ . So we have:
$$
\begin{array}{l} \mathbb {P} \left(T _ {\mathrm {c}} \geqslant \sum_ {a: \Delta_ {a} > 0} \frac {8 (l + 1) ^ {2}}{\Delta_ {a} ^ {2}} + A (l + 2)\right) \leqslant \mathbb {P} \left(\exists a: \Delta_ {a} > 0, N _ {T _ {\mathrm {c}}} (a) \geqslant \frac {8 l}{\Delta_ {a} ^ {2}} + 1\right) \\ \leqslant \sum_ {a: \Delta_ {a} > 0} \mathbb {P} \left(\exists t \leqslant T _ {\mathrm {c}}, N _ {t} (a) \geqslant \frac {8 l}{\Delta_ {a} ^ {2}} + 1\right) \\ \leqslant \frac {1 0 e A}{T \log^ {2} T}, \\ \end{array}
$$
where the second inequality, we use union bound over  $a$ . Then Corollary 2 can be easily proved with:
$$
\begin{array}{l} \mathsf {S C C} _ {\mu} ^ {\mathsf {E O C P - U G}} (T) = \mathbb {E} [ T _ {\mathrm {c}} ] \\ = \mathbb {E} \left[ T _ {\mathrm {c}} \mathbb {1} _ {T _ {\mathrm {c}} \geqslant \sum_ {a: \Delta_ {a} > 0} \frac {8 (l + 1) ^ {2}}{\Delta_ {a} ^ {2}} + A (l + 2)} \right] + \mathbb {E} \left[ T _ {\mathrm {c}} \mathbb {1} _ {T _ {\mathrm {c}} <   \sum_ {a: \Delta_ {a} > 0} \frac {8 (l + 1) ^ {2}}{\Delta_ {a} ^ {2}} + A (l + 2)} \right] \\ \leqslant T \mathbb {P} \left(T _ {\mathrm {c}} \geqslant \sum_ {a: \Delta_ {a} > 0} \frac {8 (l + 1) ^ {2}}{\Delta_ {a} ^ {2}} + A (l + 2)\right) + \sum_ {a: \Delta_ {a} > 0} \frac {8 (l + 1) ^ {2}}{\Delta_ {a} ^ {2}} + A (l + 2) \\ \leqslant \sum_ {a: \Delta_ {a} > 0} \frac {8 \log^ {2} T + 8 0 \log^ {\frac {3}{2}} T + 2 0 0 \log T}{\Delta_ {a} ^ {2}} + 6 A \log T + \frac {1 0 e A}{\log^ {2} T}. \\ \end{array}
$$
Therefore, taking  $T$  to infinity, we have:
$$
\limsup_{T\to \infty}\frac{\mathsf{SCC}^{\mathsf{EOCP - UG}}_{\boldsymbol{\mu}}(T)}{\log^{2}T} = \sum_{a:\Delta_{a} > 0}\frac{8}{\Delta_{a}^{2}}.
$$
B.5 Proof of Theorem. 3
We first prove the fundamental limits of sample complexity until commitment in the pre-determined stopping time setting. Suppose the bandit problem (instance) is as follows: the reward expectation of action 1 is  $\mu_{1}$  and the reward expectation of action 2 is  $\mu_{2}$ . We assume  $\mu_{1} > \mu_{2}$  and  $\Delta = \mu_{1} - \mu_{2}$ . Consider another bandit instance with reward expectations  $\lambda_{1}$  and  $\lambda_{2}$  for the two actions respectively, and  $\lambda_{1} + \Delta \leqslant \lambda_{2}$ . By the "transportation" lemma [32, Lemma. 1] when  $T \geqslant 10$ , we have for any stopping time  $\tau$  that:
$$
\mathbb {E} _ {\boldsymbol {\mu}} \left[ N _ {1} (\tau) \right] \mathrm {K L} \left(\mu_ {1}, \lambda_ {1}\right) + \mathbb {E} _ {\boldsymbol {\mu}} \left[ N _ {2} (\tau) \right] \mathrm {K L} \left(\mu_ {2}, \lambda_ {2}\right) \geqslant \log \left(\frac {T}{2 . 4}\right).
$$
Since under Gaussian bandit, the KL divergence is simply the squared norm, we have:
$$
\mathbb {E} _ {\boldsymbol {\mu}} \left[ N _ {1} (\tau) \right] \frac {\left(\mu_ {1} - \lambda_ {1}\right) ^ {2}}{2} + \mathbb {E} _ {\boldsymbol {\mu}} \left[ N _ {2} (\tau) \right] \frac {\left(\mu_ {2} - \lambda_ {2}\right) ^ {2}}{2} \geqslant \log \left(\frac {T}{2 . 4}\right).
$$
Since the inequality holds for any  $\lambda_{1}$  and  $\lambda_{2}$  such that that  $\lambda_1 + \Delta \leqslant \lambda_2$ , we can minimize the LHS to obtain tighter bounds. So minimizing the LHS over  $\lambda_{1}$  and  $\lambda_{2}$  gives:
$$
\lambda_ {1} = \frac {\mu_ {1} \mathbb {E} _ {\boldsymbol {\mu}} [ N _ {1} (\tau) ] + (\mu_ {2} - \Delta) \mathbb {E} _ {\boldsymbol {\mu}} [ N _ {2} (\tau) ]}{\mathbb {E} _ {\boldsymbol {\mu}} [ N _ {1} (\tau) ] + \mathbb {E} _ {\boldsymbol {\mu}} [ N _ {2} (\tau) ]}, \quad \lambda_ {2} = \frac {(\mu_ {1} + \Delta) \mathbb {E} _ {\boldsymbol {\mu}} [ N _ {1} (\tau) ] + \mu_ {2} \mathbb {E} _ {\boldsymbol {\mu}} [ N _ {2} (\tau) ]}{\mathbb {E} _ {\boldsymbol {\mu}} [ N _ {1} (\tau) ] + \mathbb {E} _ {\boldsymbol {\mu}} [ N _ {2} (\tau) ]}.
$$
So plug the optimization result into the "transportation" lemma, we have:
$$
\frac {2 \Delta^ {2} \mathbb {E} _ {\boldsymbol {\mu}} [ N _ {2} (\tau) ] ^ {2}}{(\mathbb {E} _ {\boldsymbol {\mu}} [ N _ {1} (\tau) ] + \mathbb {E} _ {\boldsymbol {\mu}} [ N _ {2} (\tau) ]) ^ {2}} + \frac {2 \Delta^ {2} \mathbb {E} _ {\boldsymbol {\mu}} [ N _ {1} (\tau) ] ^ {2}}{(\mathbb {E} _ {\boldsymbol {\mu}} [ N _ {1} (\tau) ] + \mathbb {E} _ {\boldsymbol {\mu}} [ N _ {2} (\tau) ]) ^ {2}} \geqslant \log \left(\frac {T}{2 . 4}\right).
$$
Therefore, rearranging the terms in the inequality, we can derive a lower bound on  $\mathbb{E}_{\mu}[N_1(\tau)]$  as follows:
$$
\mathbb {E} _ {\boldsymbol {\mu}} \left[ N _ {1} (\tau) \right] \geqslant \frac {2 \log \left(\frac {T}{2 . 4}\right) \mathbb {E} _ {\boldsymbol {\mu}} \left[ N _ {2} (\tau) \right]}{2 \Delta^ {2} \mathbb {E} _ {\boldsymbol {\mu}} \left[ N _ {2} (\tau) \right] - 2 \log \left(\frac {T}{2 . 4}\right)}.
$$
Let  $\tau = T_{\mathrm{c}}$  be the fixed length of exploration. Thus, we can derive the following lower bound on the expectation of stopping time as follows:
$$
\mathbb {E} _ {\boldsymbol {\mu}} [ T _ {\mathrm {c}} ] = \mathbb {E} _ {\boldsymbol {\mu}} [ N _ {1} (T _ {\mathrm {c}}) ] + \mathbb {E} _ {\boldsymbol {\mu}} [ N _ {2} (T _ {\mathrm {c}}) ] \geqslant \frac {2 \Delta^ {2} (\mathbb {E} _ {\boldsymbol {\mu}} [ N _ {2} (T _ {\mathrm {c}}) ]) ^ {2}}{2 \Delta^ {2} \mathbb {E} _ {\boldsymbol {\mu}} [ N _ {2} (T _ {\mathrm {c}}) ] - 2 \log \left(\frac {T}{2 . 4}\right)}.
$$
According to our assumption, the algorithm has  $c$ -logarithm regret violation with  $\mathcal{O}(T^{-1})$  confidence, which means the number of pulls for the sub-optimal action is upper and lower bounded when  $T$  is large enough. So we have:
$$
\left| \mathbb {E} _ {\boldsymbol {\mu}} \left[ N _ {2} (T) \right] - \frac {2 \log (T)}{\Delta^ {2}} \right| = \mathcal {O} (\log^ {c} T).
$$
Recall that  $\hat{a}$  is the action the algorithm chooses to commit to and action 1 is the optimal action in instance  $\pmb{\mu}$ , so we have  $\mathbb{E}\left[N_2(T)\right] \leqslant \mathbb{E}_{\pmb{\mu}}\left[N_2(T_{\mathrm{c}})\right] + T\mathbb{P}_{\pmb{\mu}}(\hat{a} \neq 1)$ . Since the algorithm has  $\mathcal{O}(T^{-1})$  confidence, so we have  $\mathbb{P}_{\pmb{\mu}}(\hat{a} \neq 1) = \mathcal{O}(T^{-1})$  and  $\mathbb{E}_{\pmb{\mu}}\left[N_2(T)\right] \leqslant \mathbb{E}_{\pmb{\mu}}\left[N_2(T_{\mathrm{c}})\right] + \mathcal{O}(1)$ . So we can lower bound the numerator as follows:
$$
\begin{array}{l} 2 \Delta^ {2} \left(\mathbb {E} _ {\boldsymbol {\mu}} \left[ N _ {2} \left(T _ {\mathrm {c}}\right) \right]\right) ^ {2} \geqslant 2 \Delta^ {2} \left(\mathbb {E} _ {\boldsymbol {\mu}} \left[ N _ {2} (T) \right] - \mathcal {O} (1)\right) ^ {2} \\ \geqslant 2 \Delta^ {2} \left(\frac {4}{\Delta^ {4}} \log^ {2} T - \mathcal {O} (\log^ {1 + c} T)\right) \\ = \frac {8}{\Delta^ {2}} \log^ {2} T - \mathcal {O} (\log^ {1 + c} T). \\ \end{array}
$$
Notice that  $\mathbb{E}_{\mu}[N_2(T_{\mathrm{c}})] \leqslant \mathbb{E}_{\mu}[N_2(T)]$ . For the denominator, we can derive an upper bound similarly as follows:
$$
\begin{array}{l} 2 \Delta^ {2} \mathbb {E} _ {\boldsymbol {\mu}} \left[ N _ {2} \left(T _ {\mathrm {c}}\right) \right] - 2 \log \left(\frac {T}{2 . 4}\right) \leqslant 2 \Delta^ {2} \mathbb {E} _ {\boldsymbol {\mu}} \left[ N _ {2} (T) \right] - 2 \log \left(\frac {T}{2 . 4}\right) \\ \leqslant 2 \Delta^ {2} \frac {2}{\Delta^ {2}} \log T + \mathcal {O} (\log^ {c} T) - 2 \log \left(\frac {T}{2 . 4}\right) \\ \leqslant 2 \log T + \mathcal {O} (\log^ {c} T), \\ \end{array}
$$
where the second inequality uses the fact that the algorithm has  $c$ -logarithm regret violation. So combining both bounds in the numerator and the denominator, we have:
$$
\mathbb {E} _ {\boldsymbol {\mu}} [ T _ {\mathrm {c}} ] \geqslant \frac {\frac {8}{\Delta^ {2}} \log^ {2} T - \mathcal {O} (\log^ {1 + c} T)}{2 \log T + \mathcal {O} (\log^ {c} T)} = \Omega \left(\frac {\log T}{\Delta^ {2}}\right),
$$
which concludes the proof for the pre-determined stopping time setting. In the adaptive stopping time setting, we can use the same procedure to prove the lower bound of sample complexity until commitment, so we also create another bandit instance with reward expectations  $\lambda_{1}$  and  $\lambda_{2}$ . Unlike in the pre-determined stopping time setting, now  $\lambda_{1}$  and  $\lambda_{2}$  satisfies  $\lambda_{1} \leqslant \lambda_{2}$ . We also use the "transportation" lemma and optimize over  $\lambda_{1}$  and  $\lambda_{2}$  to get a lower bound for  $\mathbb{E}[T_{\mathrm{c}}]$  as follows:
$$
\mathbb {E} _ {\boldsymbol {\mu}} \left[ T _ {\mathrm {c}} \right] = \mathbb {E} _ {\boldsymbol {\mu}} \left[ N _ {1} \left(T _ {\mathrm {c}}\right) \right] + \mathbb {E} _ {\boldsymbol {\mu}} \left[ N _ {2} \left(T _ {\mathrm {c}}\right) \right] \geqslant \frac {\Delta^ {2} \left(\mathbb {E} _ {\boldsymbol {\mu}} \left[ N _ {2} \left(T _ {\mathrm {c}}\right) \right]\right) ^ {2}}{\Delta^ {2} \mathbb {E} _ {\boldsymbol {\mu}} \left[ N _ {2} \left(T _ {\mathrm {c}}\right) \right] - 2 \log \left(\frac {T}{2 . 4}\right)}.
$$
Similar to the proof of pre-determined stopping time setting, we utilize the fact that  $\mathbb{E}_{\mu}\left[N_2(T_{\mathrm{c}})\right] \leqslant \mathbb{E}_{\mu}\left[N_2(T)\right] \leqslant \mathbb{E}_{\mu}\left[N_2(T_{\mathrm{c}})\right] + \mathcal{O}(1)$  and the algorithm has  $c$ -logarithm regret violation to bound the numerator and denominators separately. For the numerator, we have:
$$
\begin{array}{l} \Delta^ {2} \left(\mathbb {E} _ {\boldsymbol {\mu}} \left[ N _ {2} \left(T _ {\mathrm {c}}\right) \right]\right) ^ {2} \geqslant \Delta^ {2} \left(\mathbb {E} _ {\boldsymbol {\mu}} \left[ N _ {2} (T) \right] - \mathcal {O} (1)\right) ^ {2} \\ \geqslant \Delta^ {2} \left(\frac {4}{\Delta^ {4}} \log^ {2} T - \mathcal {O} (\log^ {1 + c} T)\right) \\ = \frac {4}{\Delta^ {2}} \log^ {2} T - \mathcal {O} (\log^ {1 + c} T). \\ \end{array}
$$
For the denominator, we use the fact that the algorithm has  $c$ -logarithm regret violation:
$$
\begin{array}{l} \Delta^ {2} \mathbb {E} _ {\boldsymbol {\mu}} \left[ N _ {2} \left(T _ {\mathrm {c}}\right) \right] - 2 \log \left(\frac {T}{2 . 4}\right) \leqslant \Delta^ {2} \frac {2}{\Delta^ {2}} \log T + \mathcal {O} \left(\log^ {c} T\right) - 2 \log \left(\frac {T}{2 . 4}\right) \\ \leqslant \mathcal {O} (\log^ {c} T). \\ \end{array}
$$
So combining both bounds in the numerator and the denominator, we have:
$$
\mathbb {E} _ {\boldsymbol {\mu}} [ T _ {\mathrm {c}} ] \geqslant \frac {\frac {4}{\Delta^ {2}} \log^ {2} T - \mathcal {O} (\log^ {1 + c} T)}{\mathcal {O} (\log^ {c} T)} = \Omega \left(\frac {\log^ {2 - c} T}{\Delta^ {2}}\right).
$$
C Proofs of Main Results for KL-EOCP
Now we are ready to prove the regret and sample complexity until commitment results for our KL-EOCP Algorithms, i.e., Algorithm 3. Without loss of generality, suppose action 1 is the unique optimal action. Recall that for any time index  $t$ ,  $\mathsf{UCB}_t(a)$  and  $\mathsf{LCB}_t(a)$  denotes be KL upper and lower confidence bound of action  $a$  as:
$$
\mathsf{UCB}_{t}(a) = \operatorname *{arg  max}_{\mu \geqslant \bar{r}_{t}(a)}\{N_{t}(a)\mathsf{KL}(\bar{r}_{t}(a),\mu)\leqslant l\} ,\quad \mathsf{LCB}_{t}(a) = \operatorname *{arg  min}_{\mu \leqslant \bar{r}_{t}(a)}\{N_{t}(a)\mathsf{KL}(\bar{r}_{t}(a),\mu)\leqslant l\} .
$$
Throughout the proof section, we let  $W_{a,i}$  be the  $i$ -th sample from pulling arm  $a$  the  $i$ -th time. Recall that  $T_{\mathrm{c}}$  is the length of exploration phase. Let  $\bar{r}_{a,s}$  be the empirical mean of arm  $a$  after it has been pulled  $s$  times. Moreover, we let:
$$
\mathsf{UCB}_{s,a} = \operatorname *{arg  max}_{\mu \geqslant \bar{r}_{s,a}}\left\{s\mathsf{KL}(\bar{r}_{s,a},\mu)\leqslant l\right\} ,\quad \mathsf{LCB}_{s,a} = \operatorname *{arg  min}_{\mu \leqslant \bar{r}_{s,a}}\left\{s\mathsf{KL}(\bar{r}_{s,a},\mu)\leqslant l\right\} .
$$
In order to incorporate this new definition of UCB and LCB, we need new sets of concentration inequalities, which will be summarized in the next subsection. The proof will be delayed to Sec. D.
C.1 Concentration Inequalities for Natural Exponential Families
Since we no longer assume the distributions  $\left[\nu_{1},\dots ,\nu_{A}\right]$  are subgaussian any more, the concentration inequalities from Sec. B.1, especially Lemma. 3, no longer holds. Therefore, we need a set of new concentrations specialized in the natural exponential family regime. To be specific, we want to bound the probability that the KL divergence of empirical estimations and the true expectation is very large. First, we provide a concentration lemma which is analogous to the Hoeffding's inequality widely used in the subgaussian scenario as follows:
Lemma 5 Let  $W_{1}, W_{2}, \dots, W_{T}$  be identically and independently distributed random variables with common expectation  $\mu = \mathbb{E}[W_1]$  and sampled from a distribution  $\nu$  that belongs to a canonical exponential family  $\mathcal{P}$ . For any  $s \leqslant T$ , let  $S_{s} = \sum_{i=1}^{s} W_{i}$  and  $\bar{\mu}_{s} = \frac{S_{s}}{s}$  denote the sum and the empirical mean of the first  $s$  samples. For any  $\delta > 0$ , we have:
$$
\mathbb {P} \left(\bar {\mu} _ {s} \leqslant \mu , s \mathrm {K L} \left(\bar {\mu} _ {s}, \mu\right) \geqslant \delta\right) \leqslant \exp (- \delta), \tag {5a}
$$
$$
\mathbb {P} \left(\bar {\mu} _ {s} \geqslant \mu , s \mathrm {K L} (\bar {\mu} _ {s}, \mu) \geqslant \delta\right) \leqslant \exp (- \delta). \tag {5b}
$$
Based on Lemma. 5 we propose the following lemmas characterizing the any-time concentration property of random variables.