SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The best arm identification algorithms consist of three components: an action sampling rule deciding which action to choose at each round, a stopping rule deciding a time  $\tau$  to stop collecting new samples, and a decision rule which outputs a best action candidate  $\hat{a}$ . We call an algorithm  $\delta$ -PAC if the probability of outputting a sub-optimal action is less than  $\delta$ , i.e.,  $\mathbb{P}_{\mu}(\hat{a} \neq 1) \leqslant \delta$ . Here,  $\delta$  is also called the confidence level. The performance of best arm identification algorithms is measured by both the sample complexity until identifying the optimal action and the confidence level. The theoretical limits for best arm identification algorithms are also well studied in [36, 32], which shows that any  $\delta$ -PAC algorithm would incur at least  $\mathcal{O}(\log \delta^{-1})$  sample complexity asymptotically. The constant in front of the logarithmic term depends on the bandit instance, i.e., the reward expectation and distribution of every action. Based on the this lower bound, the authors of [36] proposed the Track and Stop (TAS) algorithm which proved to be asymptotic optimal if the distribution associated with each action is from a single parameter natural exponential family. The TAS algorithm estimates the expectation of each action and at the same time calculates the optimal proportion of pulls for each action so that the optimal action is identifiable from the sub-optimal ones. Then, it designs a feedback control dynamic to track this proportion, similar to the Max-weight dynamic in queueing systems [37]. The stopping criterion of TAS is based on a generalized likelihood test. When the optimal action is identifiable from sub-optimal ones, the optimal action candidate  $\hat{a}$  is chosen to be the action which has the largest empirical reward mean.
A.3 Explore-Then-Commit Algorithms
A natural bridge between regret minimization and best arm identification problems, which also takes into account action commitment, is the explore-then-commit algorithms [38, 22, 23, 39, 40, 41]. Researchers have long been hoping that such algorithms will achieve the best of both worlds: maintaining low regret and identifying the optimal action quickly. In these algorithms, a clear separation of exploration phase and exploitation phase exists. At each round, the agent will only make action selection decisions based on the samples collected from the exploration phase, and commitment to a single action during the exploitation phase is required. No samples from the exploitation phase can be utilized although the agent may determine the length of both phases. It is clear that explore-then-commit algorithms can be easily designed from best arm identification algorithms, i.e., one would first run the best arm identification algorithm in the exploration phase, and then commit to the action  $\hat{a}$  found by the algorithm. However, it is shown in [22] that these type of BAI-ETC algorithms is essentially regret sub-optimal. To be specific, the regret lower bound of such algorithms is asymptotically twice as large as the upper bound of optimal regret minimization algorithms such as UCB and Thompson Sampling even with careful tuning. A recent work [23] provides a double explore-then-commit algorithm called DETC which is shown to be asymptotically regret optimal. But the algorithm itself is very complex and requires multiple stages of exploration and exploitation. This trait makes its sample complexity to identify the optimal action large, i.e.,  $\Omega (\log^2 T)$ . Moreover, in empirical studies, the DETC algorithm incurs very large regret which is no match for the vanilla UCB algorithm. Another work [24] proposes an exploration algorithm with inflated UCB, which can be naturally adapted to an ETC algorithm. Even though the sample complexity is order optimal, i.e.,  $\mathcal{O}(\log T)$ , the regret performance is essentially sub-optimal due to the inflation of UCB bonus.
A.4 Offline Stochastic Bandits and Reinforcement Learning
Our action identification policy is inspired by the principle of pessimism widely adopted in offline stochastic bandit problems [26, 27, 25] and reinforcement learning problems [28, 30, 29]. In offline bandit problems, the agent is not allowed to interact with the environment at will. Instead, it is provided with a training dataset which contains action and reward pairs collected from the same bandit problem. Based on this dataset, the agent is asked to choose the optimal action. It is shown in [25] that greedily selecting the action with largest empirical mean would fail to produce the optimal action in some bandit problems. This failure results from the randomness of samples in the dataset and the imbalanced number of samples for each action. Instead, choosing the action with largest lower confidence bound to combat the imbalanced uncertainty between estimations of different actions leads to better performance as justified in [27, 25]. Similar to the UCB algorithm, the penalty term between the empirical mean and the lower confidence bound for each action decreases as the number of pulls increases, so the action which has not been tried frequently will suffer large penalty. In this way, the agent will avoid selecting an action which has large uncertainty to enhance stability. Our proposed algorithm incorporates the idea of pessimism in action identification.
B Proofs of Main Results for EOCP and EOCP-UG
In this section, we provide the proofs of main results presented in Section 4. Throughout the proof section, we let  $W_{a,i}$  be the  $i$ -th sample from pulling arm  $a$  the  $i$ -th time. Let  $\bar{r}_{a,s}$  be the empirical mean of arm  $a$  after it has been pulled  $s$  times. Before proving the theorems, we provide several lemmas which gather specific large deviation results useful in our analyses. Then we prove the regret performance based on the concentration lemmas.
B.1 Concentration Inequalities
We first present two concentration lemmas which characterize the sum and mean of empirical realizations of independent sub-Gaussian random variables as follows:
Lemma 1 (Theorem 9.2 in [38]) Let  $W_{1}, W_{2}, \dots, W_{T}$  be a sequence of independent  $\sigma$ -subgaussian random variables with  $\mathbb{E}[W_1] = 0$ . Then, for any  $\delta > 0$ , we have:
$$
\mathbb {P} \left(\exists s \leqslant T, \sum_ {i = 1} ^ {s} W _ {i} \geqslant \delta\right) \leqslant \exp \left(- \frac {\delta^ {2}}{2 T \sigma^ {2}}\right). \tag {2}
$$
Lemma 2 (Lemma C.3 in [23]) Let  $T_{1} \leqslant T_{2} \leqslant T$  be two real numbers in  $\mathbb{R}^{+}$ . Let  $W_{1}, W_{2}, \dots, W_{T}$  be a sequence of identically and independently distributed random variables according to a  $\sigma$ -subgaussian distribution with  $\mathbb{E}[W_1] = 0$ . Then, for any  $\delta > 0$ , we have:
$$
\mathbb {P} \left(\exists T _ {1} \leqslant s \leqslant T _ {2}, \frac {\sum_ {i = 1} ^ {s} W _ {i}}{s} \geqslant \delta\right) \leqslant \exp \left(- \frac {T _ {1} \delta^ {2}}{2 \sigma^ {2}}\right) \tag {3}
$$
The proofs of aforementioned concentration lemmas can be found in the references respectively. Now we present the concentration results for our design of confidence bonuses as follows:
Lemma 3 Let  $W_{1}, W_{2}, \dots, W_{T}$  be identically and independently distributed 1-sub-Gaussian random variables with  $\mathbb{E}[W_1] = 0$ . Let  $T_{1} \leqslant T_{2} \leqslant T$ , then the following holds:
(a) if  $l \geqslant 2$ ,  $\mathbb{P}\left(\exists s \in (T_1, T_2], \frac{\sum_{i=1}^{s} W_i}{s} + \sqrt{\frac{2l}{s}} \leqslant 0\right) \leqslant \frac{\min\{T_2 - T_1, el(\log T_2 - \log T_1) + e\}}{\exp(l)}$ ; (4a)
(b) if  $\delta \in (0, \sqrt{3}]$ ,  $\mathbb{P}\left(\exists s \in [T_1, T_2], \frac{\sum_{i=1}^{s} W_i}{s} + \sqrt{\frac{2l}{s}} + \delta \leqslant 0\right) \leqslant \frac{4}{\delta^2 \exp\left(\left(\sqrt{l} + \delta \sqrt{\frac{T_1}{2}}\right)^2\right)}$ ; (4b)
(c) if  $l \geqslant \frac{T_1\delta^2}{2}$ ,  $\sum_{s = T_1 + 1}^{T_2} \mathbb{P}\left(\frac{\sum_{i=1}^{s} W_i}{s} + \sqrt{\frac{2l}{s}} \geqslant \delta\right) \leqslant \frac{2l + \sqrt{4\pi l} + 2}{\delta^2} + 1 - T_1$ . (4c)
The proof of the Lemma. 3 will be delayed to Sec. D.
B.2 Proof of Regret Optimality for EOCP with Pre-Determined Stopping Time
We prove the following Theorem which characterizes the finite-time performance of Algorithm. 1. Theorem. 1 can be directly derived.
Theorem 5 If  $l = \log (T) + 4\sqrt{2\log(T)}$  and  $T \geqslant \max \{16, A, 16l\Delta_{\mathrm{lb}}^{-2}\}$ , the expected regret of the EOCP algorithm in Algorithm. 1 with pre-determined stopping time is upper bounded by:
$$
\operatorname {R e g} _ {\mu} ^ {\mathrm {E O C P}} (T) \leqslant \sum_ {a: \Delta_ {a} > 0} \left(\frac {2 \log T}{\Delta_ {a}} + \frac {(8 + \sqrt {2 0 \pi}) \sqrt {\log T}}{\Delta_ {a}} + \frac {2}{\Delta_ {a}} + \Delta_ {a}\right) + o (1).
$$
Remark: The asymptotic regret upper bound is clear from Theorem.5 by letting  $T$  increases to infinity, i.e.,
$$
\begin{array}{l} \lim  _ {T \rightarrow \infty} \frac {\operatorname {R e g} _ {\mu} ^ {\mathrm {E O C P}} (T)}{\log T} \leqslant \lim  _ {T \rightarrow \infty} \sum_ {a: \Delta_ {a} > 0} \left(\frac {2}{\Delta_ {a}} + \frac {(8 + \sqrt {2 0 \pi})}{\Delta_ {a} \sqrt {\log T}} + \frac {2}{\Delta_ {a} \log T} + \frac {\Delta_ {a}}{\log T}\right) + o (1) \\ = \sum_ {a: \Delta_ {a} > 0} \frac {2}{\Delta_ {a}}. \\ \end{array}
$$
Proof. Without loss of generality, assume action 1 is the unique optimal action. From the regret decomposition lemma [38, Lemma 4.5], we can decompose the regret of Algorithm 1 to the number of pulls for each sub-optimal arm as follows:
$$
\operatorname {R e g} _ {\mu} ^ {\mathrm {E O C P}} (T) = \sum_ {a: \Delta_ {a} > 0} \Delta_ {a} \mathbb {E} \left[ N _ {T} (a) \right].
$$
Then, the key to bound the total regret is to bound the number of pulls for each sub-optimal arms. Since our EOCP algorithm has a clear separation of exploration and exploitation phases, so for any sub-optimal action  $a$ , we can bound its pulls in different phases as follows:
$$
\mathbb {E} \big [ N _ {a} (T) \big ] \leqslant \underbrace {\mathbb {E} \left[ N _ {a} (T _ {\mathrm {c}}) \right]} _ {I _ {1}} + (T - T _ {\mathrm {c}}) \underbrace {\mathbb {P} (\hat {a} = a)} _ {I _ {2}},
$$
where  $T_{\mathrm{c}}$  is the end of exploration phase and  $\hat{a}$  is the action that the algorithm commits to. Notice that the bound on  $I_{2}$  gives the confidence level result provided in Corollary 1.
Bounding  $I_{1}$ : We first bound  $I_{1}$ , which is similar to the proof of bounding the regret for the traditional UCB algorithm [12, 10, 22]. We have:
$$
I _ {1} = \mathbb {E} \left[ \sum_ {t = 1} ^ {T _ {\mathrm {c}}} \mathbb {1} _ {A _ {t} = a} \right] = 1 + \underbrace {\mathbb {E} \left[ \sum_ {t = A + 1} ^ {T _ {\mathrm {c}}} \mathbb {1} _ {A _ {t} = a} \mathbb {1} _ {\mathsf {U C B} _ {t - 1} (a) \geqslant \mu_ {1}} \right]} _ {A _ {1}} + \underbrace {\mathbb {E} \left[ \sum_ {t = A + 1} ^ {T _ {\mathrm {c}}} \mathbb {1} _ {A _ {t} = a} \mathbb {1} _ {\mathsf {U C B} _ {t - 1} (a) <   \mu_ {1}} \right]} _ {A _ {2}}.
$$
Notice that  $A_{t} = a$  indicates  $\mathsf{UCB}_{t - 1}(1) < \mathsf{UCB}_{t - 1}(a)$  due to the dynamic of UCB exploration, we can bound the last term as:
$$
A _ {2} \leqslant \sum_ {t = A + 1} ^ {T _ {\mathrm {c}}} \mathbb {E} \left[ \mathbb {1} _ {\mathsf {U C B} _ {t - 1} (1) <   \mu_ {1}} \right] = \sum_ {t = A + 1} ^ {T _ {\mathrm {c}}} \mathbb {P} \left(\mathsf {U C B} _ {t - 1} (1) <   \mu_ {1}\right).
$$
Event  $\{\mathsf{UCB}_{t - 1}(1) < \mu_1\}$  means that at round  $t$ , the mean estimation  $\bar{r}_{t - 1}(1)$  of the optimal arm from previous pulls is lower than  $\mu_{1} - b_{t - 1}(1)$ , which incurs a large deviation, so we can bound this event with a union over all rounds in the exploration phase:
$$
\left\{\mathsf {U C B} _ {t - 1} (1) <   \mu_ {1} \right\} \subset \left\{\exists t ^ {\prime} \in \left[ A, T _ {\mathrm {c}}\right), \mathsf {U C B} _ {t ^ {\prime}} (1) <   \mu_ {1} \right\},
$$
which doesn't depend on round number  $t$  any more, so we have:
$$
\begin{array}{l} \sum_ {t = A + 1} ^ {T _ {\mathrm {c}}} \mathbb {P} \left(\mathsf {U C B} _ {t - 1} (1) <   \mu_ {1}\right) \leqslant \sum_ {t = A + 1} ^ {T _ {\mathrm {c}}} \mathbb {P} \left(\exists t ^ {\prime} \in \left[ A, T _ {\mathrm {c}}\right), \mathsf {U C B} _ {t ^ {\prime}} (1) <   \mu_ {1}\right) \\ = \left(T _ {\mathrm {c}} - A\right) \mathbb {P} \left(\exists t ^ {\prime} \in \left[ A, T _ {\mathrm {c}}\right), \cup \mathrm {C B} _ {t ^ {\prime}} (1) <   \mu_ {1}\right) \\ = \left(T _ {\mathrm {c}} - A\right) \mathbb {P} \left(\exists t ^ {\prime} \in [ A, T _ {\mathrm {c}}), \bar {r} _ {t ^ {\prime}} (1) + \sqrt {\frac {2 l}{N _ {t ^ {\prime}} (1)}} <   \mu_ {1}\right). \\ \end{array}
$$
It is worth noting that if there is no pulls for arm 1 between a time interval, its empirical mean and number of pulls will remain the same. So we have:
$$
\left\{\exists t ^ {\prime} \in [ A, T _ {\mathrm {c}}), \bar {r} _ {t ^ {\prime}} (1) + \sqrt {\frac {2 l}{N _ {t ^ {\prime}} (1)}} <   \mu_ {1} \right\} \subset \left\{\exists s \in [ 1, N _ {T _ {\mathrm {c}}} (1)), \bar {r} _ {s, 1} + \sqrt {\frac {2 l}{s}} <   \mu_ {1} \right\}.
$$
Notice that  $N_{T_{\mathrm{c}}}(1) \leqslant T_{\mathrm{c}} - A + 1$ , so we can bound the probability as follows:
$$
\begin{array}{l} \mathbb {P} \left(\exists t ^ {\prime} \in [ A, T _ {\mathrm {c}}), \bar {r} _ {t ^ {\prime}} (1) + \sqrt {\frac {2 l}{N _ {t ^ {\prime}} (1)}} <   \mu_ {1}\right) \\ \leqslant \mathbb {P} \left(\exists s \in [ 1, T _ {\mathrm {c}} - A + 1), \bar {r} _ {s, 1} - \mu_ {1} + \sqrt {\frac {2 l}{s}} <   0\right) \\ \leqslant \mathbb {P} \left(\exists s \in [ 1, T _ {\mathrm {c}} - A + 1), \frac {\sum_ {i = 1} ^ {s} \left(W _ {1 , i} - \mu_ {1}\right)}{i} + \sqrt {\frac {2 l}{s}} <   0\right) \\ \leqslant \frac {T _ {\mathrm {c}} - A + 1}{\exp (l)}, \\ \end{array}
$$
where the last inequality uses Eq. (4a) of Lemma 3 and the fact that  $W_{1,i} - \mu_1$  is 1-sub-Gaussian with zero mean. Notice that  $\sqrt{\log T} \geqslant \log \log T$  when  $T$  is large and we have:
$$
\exp (l) = \exp \left(\log T + 4 \sqrt {2 \log T}\right) \geqslant \exp (\log T + 4 \log \log T) = \frac {1}{T \log^ {4} T}.
$$
So putting the above bound back to the bound of  $A_{2}$ , we have:
$$
A _ {2} \leqslant \frac {\left(T _ {\mathrm {c}} - A + 1\right) ^ {2}}{T \log^ {2} T} = \frac {\left(\frac {8 A \left(\log T + 4 \sqrt {2 \log T}\right)}{\Delta_ {\mathrm {l b}} ^ {2}} + 1\right) ^ {2}}{T \log^ {4} T} = o \left(\frac {1}{T}\right),
$$
where the last inequality is due to the fact that when  $T \geqslant 3$ ,  $\sqrt{\log T} \leqslant \log T$ . Next, we attempt to bound the middle term  $A_{1}$  as follows:
$$
A _ {1} = \mathbb {E} \left[ \sum_ {t = A + 1} ^ {T _ {\mathrm {c}}} \mathbb {1} _ {A _ {t} = a} \mathbb {1} _ {\cup \mathsf {C B} _ {t - 1} (a) \geqslant \mu_ {1}} \right] = \mathbb {E} \left[ \sum_ {t = A + 1} ^ {T _ {\mathrm {c}}} \mathbb {1} _ {A _ {t} = a} \mathbb {1} _ {\bar {r} _ {t - 1} (a) + \sqrt {\frac {2 l}{N _ {t - 1} (a)}} \geqslant \mu_ {1}} \right].
$$
Notice that the number of pulls  $N_{t}(a)$  will only increase by 1 every time there is new pull, i.e., when  $A_{t} = a$ . Otherwise, the term inside summation is 0. So instead of counting on the time step  $t$ , we can count over the number of pulls over arm  $a$  as follows:
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {t = A + 1} ^ {T _ {\mathrm {c}}} \mathbb {1} _ {A _ {t} = a} \mathbb {1} _ {\bar {r} _ {t - 1} (a) + \sqrt {\frac {2 l}{N _ {t - 1} (a)}} \geqslant \mu_ {1}} \right] = \mathbb {E} \left[ \sum_ {s = 1} ^ {N _ {T _ {\mathrm {c}}} (a)} \mathbb {1} _ {\bar {r} _ {s, a} + \sqrt {\frac {2 l}{s}} \geqslant \mu_ {1}} \right] \\ \leqslant \sum_ {s = 1} ^ {T _ {\mathrm {c}}} \mathbb {P} \left(\bar {r} _ {s, a} - \mu_ {a} + \sqrt {\frac {2 l}{s}} \geqslant \Delta_ {a}\right), \\ \end{array}
$$
where the last inequality is due to  $N_{T_{\mathrm{c}}}(a) \leqslant T_{\mathrm{c}}$ . Then, by Eq. (4c) of Lemma 3, we have:
$$
\begin{array}{l} \sum_ {s = 1} ^ {T _ {\mathrm {c}}} \mathbb {P} \left(\bar {r} _ {s, a} - \mu_ {a} + \sqrt {\frac {2 l}{s}} \geqslant \Delta_ {a}\right) = \sum_ {s = 1} ^ {T _ {\mathrm {c}}} \mathbb {P} \left(\frac {\sum_ {i = 1} ^ {s} \left(W _ {a , i} - \mu_ {a}\right)}{i} + \sqrt {\frac {2 l}{s}} \geqslant \Delta_ {a}\right) \\ \leqslant \frac {2 (\log T + 4 \sqrt {2 \log T}) + \sqrt {4 \pi (\log T + 4 \sqrt {2 \log T})} + 2}{\Delta_ {a} ^ {2}} + 1 \\ \leqslant \frac {2 \log T}{\Delta_ {a} ^ {2}} + \frac {(8 + \sqrt {2 0 \pi}) \sqrt {\log T}}{\Delta_ {a} ^ {2}} + \frac {2}{\Delta_ {a} ^ {2}} + 1, \\ \end{array}
$$
where the first inequality is due to the fact that  $W_{a,i} - \mu_a$  is 1-sub-Gaussian with zero mean, and the last inequality is due to  $\sqrt{2\log T} \leqslant \log T$  when  $T \geqslant 9$ . Therefore, combining the bounds on  $A_1$  and  $A_2$ , we can bound  $I_1$  as follows:
$$
I _ {1} \leqslant \frac {2 \log T}{\Delta_ {a} ^ {2}} + \frac {(8 + \sqrt {2 0 \pi}) \sqrt {\log T}}{\Delta_ {a} ^ {2}} + \frac {2}{\Delta_ {a} ^ {2}} + 1 + o (1).
$$
Bounding  $I_{2}$ : The bound on  $I_{2}$  gives the confidence level result provided in Corollary 1. After the exploration phase, recall that we select the arm with the largest lower confidence bound to commit to. The idea of bounding  $I_{2}$  is very similar to the regret bound in [2] where since we use UCB to explore in the exploration phase, the optimal arm should be pulled very often such that its bonus becomes very small when exploration phase ends. Then selecting the arm with largest LCB will ensure we select the optimal arm with high probability. This is because the LCB of the optimal arm is larger than the true means of any sub-optimal arms, and the true means of sub-optimal arms are larger than their respective LCB, both with high probability. We can first bound the probability of selecting a sub-optimal arm  $a$  as:
$$
\begin{array}{l} \mathbb {P} (\hat {a} = a) \leqslant \mathbb {P} \left(\mathrm {L C B} _ {T _ {\mathrm {c}}} (a) \geqslant \mathrm {L C B} _ {T _ {\mathrm {c}}} (1)\right) \\ \leqslant \underbrace {\mathbb {P} \left(\mathrm {L C B} _ {T _ {\mathrm {c}}} (a) \geqslant \mathrm {L C B} _ {T _ {\mathrm {c}}} (1) , N _ {T _ {\mathrm {c}}} (1) > \frac {8 l}{\Delta_ {\mathrm {l b}} ^ {2}}\right)} _ {B _ {1}} + \underbrace {\mathbb {P} \left(N _ {T _ {\mathrm {c}}} (1) \leqslant \frac {8 l}{\Delta_ {\mathrm {l b}} ^ {2}}\right)} _ {B _ {2}}. \\ \end{array}
$$
We first bound the term  $B_{2}$  which states that during the first exploration phase with UCB exploration, the optimal arm is under-pulled, which means that one of the sub-optimal arms have been pulled with larger number of times. Therefore, we have:
$$
\mathbb {P} \left(N _ {T _ {\mathrm {c}}} (1) \leqslant \frac {8 l}{\Delta_ {\mathrm {l b}} ^ {2}}\right) = \mathbb {P} \left(\sum_ {a: \Delta_ {a} > 0} N _ {T _ {\mathrm {c}}} (a) > T _ {\mathrm {c}} - \frac {8 l}{\Delta_ {\mathrm {l b}} ^ {2}}\right).
$$
Recall that  $T_{\mathrm{c}} = \frac{8Al}{\Delta_{\mathrm{lb}}^{2}} + A$ , so we have:
$$
\begin{array}{l} \mathbb {P} \left(N _ {T _ {\mathrm {c}}} (1) \leqslant \frac {8 l}{\Delta_ {\mathrm {l b}} ^ {2}}\right) = \mathbb {P} \left(\sum_ {a: \Delta_ {a} > 0} N _ {T _ {\mathrm {c}}} (a) > \frac {8 (A - 1) l}{\Delta_ {\mathrm {l b}} ^ {2}} + A\right) \\ \leqslant \mathbb {P} \left(\exists a > 1, N _ {T _ {\mathrm {c}}} (a) > \frac {8 l}{\Delta_ {\mathrm {l b}} ^ {2}} + 1\right). \\ \end{array}
$$
Then by union bound, we have:
$$
\mathbb {P} \left(\exists a > 1, N _ {T _ {\mathrm {c}}} (a) > \frac {8 l}{\Delta_ {\mathrm {l b}} ^ {2}} + 1\right) \leqslant \sum_ {a: \Delta_ {a} > 0} \mathbb {P} \left(N _ {T _ {\mathrm {c}}} (a) > \frac {8 l}{\Delta_ {\mathrm {l b}} ^ {2}} + 1\right).
$$
Consider a fixed sub-optimal arm  $a$ , then for each probability inside the summation, we have:
$$
\begin{array}{l} \mathbb {P} \left(N _ {T _ {\mathrm {c}}} (a) > \frac {8 l}{\Delta_ {\mathrm {l b}} ^ {2}} + 1\right) \\ \leqslant \mathbb {P} \left(\exists t \in [ A + 1, T _ {\mathrm {c}} ], N _ {t - 1} (a) = \left\lceil \frac {8 l}{\Delta_ {\mathrm {l b}} ^ {2}} \right\rceil , A _ {t} = 2\right) \\ \leqslant \mathbb {P} \left(\exists t \in [ A + 1, T _ {\mathrm {c}} ], N _ {t - 1} (a) = \left\lceil \frac {8 l}{\Delta_ {\mathrm {l b}} ^ {2}} \right\rceil , \cup \mathrm {C B} _ {t - 1} (1) \leqslant \cup \mathrm {C B} _ {t - 1} (a)\right) \\ \leqslant \underbrace {\mathbb {P} \left(\exists t \in [ A + 1 , T _ {\mathrm {c}} ] , N _ {t - 1} (a) = \left\lceil \frac {8 l}{\Delta_ {\mathrm {l b}} ^ {2}} \right\rceil , \mu_ {1} \leqslant \mathsf {U C B} _ {t - 1} (a)\right)} _ {B _ {3}} \\ + \underbrace {\mathbb {P} \left(\exists t \in [ A + 1 , T _ {c} ] , \mathsf {U C B} _ {t - 1} (1) \leqslant \mu_ {1}\right)} _ {B _ {4}}. \\ \end{array}
$$
Notice that  $B_{4}$  can be bounded from concentration lemma. First, we switch the count from time step  $t$  to the number of pulls for arm 1 since the empirical estimation  $\bar{r}_{t-1}(1)$  and count for pulls  $N_{t-1}(1)$  won't change unless there is a new pull. Then, we will apply Eq. (4a) to the probability as follows:
$$
\begin{array}{l} B _ {4} = \mathbb {P} \left(\exists t \in [ A + 1, T _ {\mathrm {c}} ], \bar {r} _ {t - 1} (1) - \mu_ {1} + \sqrt {\frac {2 l}{N _ {t - 1} (1)}} \leqslant 0\right) \\ \leqslant \mathbb {P} \left(\exists s \in [ 1, T _ {\mathrm {c}} - A + 1), \bar {r} _ {s, 1} - \mu_ {1} + \sqrt {\frac {2 l}{s}} \leqslant 0\right) \\ \leqslant \mathbb {P} \left(\exists s \in [ 1, T _ {\mathrm {c}} - A + 1), \frac {\sum_ {i = 1} ^ {s} \left(W _ {1 , i} - \mu_ {1}\right)}{s} + \sqrt {\frac {2 l}{s}} \leqslant 0\right) \\ \leqslant \frac {T _ {\mathrm {c}} - A}{\exp l}, \\ \end{array}
$$
where the last inequality is due to the fact that  $W_{1,i} - \mu_1$  is 1-sub-Gaussian with zero mean. Similar to the procedure of bounding  $I_1$ , notice that  $\exp (l)\geqslant T\log^4 T$ , we have:
$$
B _ {4} \leqslant \frac {8 A (\log T + 4 \sqrt {2 \log T})}{\Delta_ {\mathrm {l b}} ^ {2}} \frac {1}{T \log^ {4} T} = o \left(\frac {1}{T}\right).
$$
On the other hand, let  $\gamma = \left\lceil \frac{8l}{\Delta_{\mathrm{lb}}^2} \right\rceil$ , and we also change the count from time step  $t$  to the number of pulls for arm  $a$ . Then,  $B_3$  can be expressed as follows:
$$
\begin{array}{l} B _ {3} = \mathbb {P} \left(\exists t \in [ A + 1, T _ {\mathrm {c}} - 1 ], N _ {t - 1} (a) = \gamma , \mu_ {1} \leqslant \bar {r} _ {t - 1} (a) + \sqrt {\frac {2 l}{N _ {t - 1} (a)}}\right) \\ = \mathbb {P} \left(\bar {r} _ {\gamma , a} + \sqrt {\frac {2 l}{\gamma}} > \mu_ {1}\right). \\ \end{array}
$$
Notice that  $\gamma \geqslant \frac{8l}{\Delta_{1b}^2} \geqslant \frac{8l}{\Delta_a^2}$ , so we have  $\sqrt{\frac{2l}{\gamma}} \leqslant \sqrt{\frac{2l\Delta_a^2}{8l}} = \frac{\Delta_a}{2}$ . Then  $B_3$  can be bounded as follows:
$$
B _ {3} \leqslant \mathbb {P} \left(\bar {r} _ {\gamma , a} + \frac {\Delta_ {a}}{2} > \mu_ {1}\right) = \mathbb {P} \left(\bar {r} _ {\gamma , a} - \mu_ {a} > \frac {\Delta_ {a}}{2}\right) \leqslant \exp \left(- \frac {\gamma \Delta_ {a} ^ {2}}{8}\right) \leqslant \frac {1}{\exp (l)} \leqslant \frac {1}{T \log^ {4} T},
$$
where the first inequality uses Hoeffding's inequality and the second inequality uses the lower bound on  $\gamma$  mentioned above. The last inequality is due to the fact that  $\exp (l)\geqslant T\log^4 T$ . So combine  $B_{3}$  and  $B_{4}$  together, we can bound  $B_{2}$  as follows:
$$
B _ {2} \leqslant \sum_ {a: \Delta_ {a} > 0} (B _ {3} + B _ {4}) = o \left(\frac {1}{T}\right).
$$
Next, we attempt to bound  $B_{1}$ . Notice that  $B_{1}$  can also be bounded as follows: