SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} B _ {1} = \mathbb {P} \left(\mathrm {L C B} _ {T _ {\mathrm {c}}} (a) \geqslant \mathrm {L C B} _ {T _ {\mathrm {c}}} (1), N _ {T _ {\mathrm {c}}} (1) > \frac {8 l}{\Delta_ {\mathrm {l b}} ^ {2}}\right) \\ \leqslant \underbrace {\mathbb {P} \left(\mu_ {a} \geqslant \operatorname {L C B} _ {T _ {\mathrm {c}}} (1) , N _ {T _ {\mathrm {c}}} (1) > \frac {8 l}{\Delta_ {\mathrm {l b}} ^ {2}}\right)} _ {B _ {5}} + \underbrace {\mathbb {P} \left(\mu_ {a} \leqslant \operatorname {L C B} _ {T _ {\mathrm {c}}} (a)\right)} _ {B _ {6}}. \\ \end{array}
$$
The term  $B_{6}$  can be expressed by counting the number of pulls of arm  $a$  as follows:
$$
\begin{array}{l} B _ {6} = \mathbb {P} \left(\mu_ {a} \leqslant \bar {r} _ {T _ {\mathrm {c}}} (a) - \sqrt {\frac {2 l}{N _ {T _ {\mathrm {c}}} (a)}}\right) \\ \leqslant \mathbb {P} \left(\exists s \in [ 1, T _ {\mathrm {c}} - A + 1 ], \mu_ {a} - \bar {r} _ {s, a} + \sqrt {\frac {2 l}{s}} \leqslant 0\right) \\ = \mathbb {P} \left(\exists s \in [ 1, T _ {\mathrm {c}} - A + 1 ], \frac {\sum_ {i = 1} ^ {s} (\mu_ {a} - W _ {a , i})}{s} + \sqrt {\frac {2 l}{s}} \leqslant 0\right). \\ \end{array}
$$
Notice that  $(\mu_{a} - W_{a,i})$  is 1-sub-Gaussian with zero mean, so we can apply concentration lemma Eq. (4a) from Lemma. (3) as follows:
$$
B _ {6} \leqslant \frac {T _ {\mathrm {c}} - A + 1}{\exp (l)} = \frac {8 A (\log T + 4 \sqrt {2 \log T})}{\Delta_ {\mathrm {l b}} ^ {2} \exp (l)} + \frac {1}{\exp (l)} = o \left(\frac {1}{T}\right),
$$
where the last inequality is due to  $\log T\geqslant \sqrt{2\log T}$  and  $\exp (l)\geqslant T\log^4 T$  when  $T\geqslant 9$ . Similarly, term  $B_{5}$  can be expressed as:
$$
B _ {5} = \mathbb {P} \left(\mu_ {a} \geqslant \bar {r} _ {T _ {\mathrm {c}}} (1) - \sqrt {\frac {2 l}{N _ {T _ {\mathrm {c}}} (1)}}, N _ {T _ {\mathrm {c}}} (1) > \frac {8 l}{\Delta_ {\mathrm {l b}} ^ {2}}\right).
$$
First notice that when  $N_{T_{\mathrm{c}}} (1) > \frac{8l}{\Delta_{\mathrm{lb}}^{2}}$ , we have the bonus term  $b_{T_{\mathrm{c}}} (1) = \sqrt{\frac{2l}{N_{T_{\mathrm{c}}} (1)}} \leqslant \frac{\Delta_{\mathrm{lb}}}{2} \leqslant \frac{\Delta_{a}}{2}$ . Then we have:
$$
B _ {5} \leqslant \mathbb {P} \left(\mu_ {a} \geqslant \bar {r} _ {T _ {\mathrm {c}}} (1) - \frac {\Delta_ {a}}{2}, N _ {T _ {\mathrm {c}}} (1) > \frac {8 l}{\Delta_ {\mathrm {l b}} ^ {2}}\right) = \mathbb {P} \left(\mu_ {1} - \bar {r} _ {T _ {\mathrm {c}}} (1) \geqslant \frac {\Delta_ {a}}{2}, N _ {T _ {\mathrm {c}}} (1) > \frac {8 l}{\Delta_ {\mathrm {l b}} ^ {2}}\right).
$$
Then it is equivalent to count over the number of pulls for arm 1:
$$
\begin{array}{l} B _ {5} \leqslant \mathbb {P} \left(\exists s \in \left[ \frac {8 l}{\Delta_ {\mathrm {l b}} ^ {2}}, T \right], \mu_ {1} - \bar {r} _ {s, 1} \geqslant \frac {\Delta_ {a}}{2}\right) \\ = \mathbb {P} \left(\exists s \in \left[ \frac {8 l}{\Delta_ {\mathrm {l b}} ^ {2}}, T \right], \frac {\sum_ {i = 1} ^ {s} \left(\mu_ {1} - W _ {1 , i}\right)}{s} \geqslant \frac {\Delta_ {a}}{2}\right). \\ \end{array}
$$
Then, by the maximal concentration Eq. (3) from Lemma 2, we can bound  $B_{5}$  as follows:
$$
\mathbb {P} \left(\exists s \in \left[ \frac {8 l}{\Delta_ {\mathrm {l b}} ^ {2}}, T \right], \frac {\sum_ {i = 1} ^ {s} \left(\mu_ {1} - W _ {1 , i}\right)}{s} \geqslant \frac {\Delta_ {a}}{2}\right) \leqslant \exp \left(- \frac {\frac {8 l}{\Delta_ {\mathrm {l b}} ^ {2}} \left(\frac {\Delta_ {a}}{2}\right) ^ {2}}{2}\right) = \frac {1}{\exp (l)} \leqslant \frac {1}{T \log^ {4} T}.
$$
Therefore, collecting the bounds for  $B_{5}$  and  $B_{6}$ , we have a bound for  $B_{1}$  as follows:
$$
B _ {1} \leqslant B _ {5} + B _ {6} = o \left(\frac {1}{T}\right).
$$
And therefore from the bounds of  $B_{1}$  and  $B_{2}$ , we can bound term  $I_{2}$  as follows:
$$
I _ {2} \leqslant B _ {1} + B _ {2} = o \left(\frac {1}{T}\right).
$$
This means the confidence of selecting the wrong action to commit to is  $o(T^{-1})$  as indicated in Corollary 1. Finally, putting the bounds on  $I_{1}$  and  $I_{2}$  together, we have:
$$
\begin{array}{l} \mathbb {E} \left[ N _ {a} (T) \right] \leqslant I _ {1} + T I _ {2} \\ \leqslant \frac {2 \log T}{\Delta_ {a} ^ {2}} + \frac {(8 + \sqrt {2 0 \pi}) \sqrt {\log T}}{\Delta_ {a} ^ {2}} + \frac {2}{\Delta_ {a} ^ {2}} + 1 + o (1). \\ \end{array}
$$
Therefore, by the regret decomposition lemma, we can bound the total regret as follows:
$$
\begin{array}{l} \operatorname {R e g} _ {\boldsymbol {\mu}} (T) \leqslant \sum_ {a: \Delta_ {a} > 0} \left(\frac {2 \log T}{\Delta_ {a}} + \frac {(8 + \sqrt {2 0 \pi}) \sqrt {\log T}}{\Delta_ {a}} + \frac {2}{\Delta_ {a}} + \Delta_ {a} + o (1)\right) \\ = \sum_ {a: \Delta_ {a} > 0} \left(\frac {2 \log T}{\Delta_ {a}} + \frac {(8 + \sqrt {2 0 \pi}) \sqrt {\log T}}{\Delta_ {a}} + \frac {2}{\Delta_ {a}} + \Delta_ {a}\right) + o (1). \\ \end{array}
$$
B.3 Proof of Regret Optimality for EOCP-UG with Adaptive Stopping Time
We prove the following theorem which characterizes the finite-time performance of Algorithm. 2. Theorem. 2 can be derived directly.
Theorem 6 If  $l = \log (T) + 4\sqrt{2\log(T)}$  and  $T \geqslant \max \{16, A, 16l\Delta_{\min}^{-2}\}$ , the regret of EOCP-UG algorithm in Algorithm 2 with adaptive stopping time can be upper bounded as:
$$
\operatorname {R e g} _ {\mu} ^ {\text {E O C P - U G}} (T) \leqslant \sum_ {a: \Delta_ {a} > 0} \left(\frac {2 \log T}{\Delta_ {a}} + \frac {(8 + \sqrt {2 0 \pi}) \sqrt {\log T}}{\Delta_ {a}}\right) + \mathcal {O} (1).
$$
Remark: The asymptotic regret upper bound is clear from Theorem.6 by letting  $T$  increases to infinity, i.e.,
$$
\limsup_{T\to \infty}\frac{\operatorname{Reg}_{\mu}^{\text{EOCP-UG}}(T)}{\log T}\leqslant \limsup_{T\to \infty}\sum_{a:\Delta_{a} > 0}\left(\frac{2}{\Delta_{a}} +\frac{(8 + \sqrt{20\pi})}{\Delta_{a}\sqrt{\log T}}\right) + \mathcal{O}(\log^{-1}(T)) = \sum_{a:\Delta_{a} > 0}\frac{2}{\Delta_{a}}.
$$
Proof. Without loss of generality, let action 1 be the unique optimal action. The first step for proving the regret performance is regret decomposition lemma [38, Lemma 4.5]. We also decompose the regret of Algorithm 2 into the number of pulls for each sub-optimal arm as follows:
$$
\operatorname {R e g} _ {\boldsymbol {\mu}} (T) = \sum_ {a: \Delta_ {a} > 0} \Delta_ {a} \mathbb {E} \left[ N _ {T} (a) \right].
$$
Then, for a specific sub-optimal arm  $a$ , we bound the number of pulls. Since our EOCP-UG algorithm has a clear separation of exploration and exploitation phases, we can bound the pulls in the two phases respectively. However, the unique characteristic of unknown gap scenario is we don't have a fixed end time of exploration phase. Recall that  $T_{\mathrm{c}} \leqslant T$  is the stopping time that the exploration phase ends and  $\hat{a}$  is the arm we choose for commitment, so we can decompose the number of pulls into two phases as:
$$
\mathbb {E} \big [ N _ {a} (T) \big ] = \mathbb {E} \big [ N _ {a} (T _ {\mathrm {c}}) \big ] + \mathbb {E} \big [ (T - T _ {\mathrm {c}}) \mathbb {1} _ {\hat {a} = a} \big ] \leqslant \underbrace {\mathbb {E} \big [ N _ {a} (T _ {\mathrm {c}}) \big ]} _ {I _ {1}} + T \underbrace {\mathbb {P} (\hat {a} = a)} _ {I _ {2}}.
$$
Notice that upper bound of  $I_{2}$  gives the confidence level result in Corollary 2. We then bound the two terms  $I_{1}$  and  $I_{2}$  separately. In order to simplify the proof, we first prove a lemma which characterizes a high probability upper bound for the stopping time  $T_{\mathrm{c}}$ . This lemma also proves the sample complexity to commitment result in Corollary 2. The proof of Lemma 4 will be delayed.
Lemma 4 If  $l = \log (T) + 4\sqrt{2\log(T)}$  and  $T \geqslant \max \{16, A, 16l\Delta_{\min}^{-2}\}$ , our stopping time  $T_{c}$  for exploration of Algorithm. 2 is upper bounded with high probability:
$$
\mathbb {P} \left(T _ {\mathrm {c}} \geqslant \sum_ {a: \Delta_ {a} > 0} \frac {8 (l + 1) ^ {2}}{\Delta_ {a} ^ {2}} + A (l + 2)\right) \leqslant \frac {1 0 e A}{T \log^ {2} T}.
$$
Bounding  $I_{1}$ : With the help of Lemma 4, we have a high probability upper bound for the exploration phase. We let  $T_{\mathrm{c}}^{\mathrm{u}} = \sum_{a:\Delta_{a} > 0}\frac{8(l + 1)^{2}}{\Delta_{a}^{2}} +A(l + 2)$  to be the high probability upper bound of our stopping time to end the exploration phase. Notice that  $N_{a}(T_{\mathrm{c}})\leqslant T$ , we have:
$$
\begin{array}{l} I _ {1} = \mathbb {E} \left[ N _ {a} \left(T _ {\mathrm {c}}\right) \mathbb {1} _ {T _ {\mathrm {c}} \geqslant T _ {\mathrm {c}} ^ {\mathrm {u}}} \right] + \mathbb {E} \left[ N _ {a} \left(T _ {\mathrm {c}}\right) \mathbb {1} _ {T _ {\mathrm {c}} \leqslant T _ {\mathrm {c}} ^ {\mathrm {u}}} \right] \leqslant T \mathbb {P} \left(T _ {\mathrm {c}} \geqslant T _ {\mathrm {c}} ^ {\mathrm {u}}\right) + \mathbb {E} \left[ N _ {a} \left(T _ {\mathrm {c}}\right) \mathbb {1} _ {T _ {\mathrm {c}} \leqslant T _ {\mathrm {c}} ^ {\mathrm {u}}} \right] \\ \leqslant \underbrace {\mathbb {E} \left[ N _ {a} \left(T _ {\mathrm {c}}\right) \mathbb {1} _ {T _ {\mathrm {c}} \leqslant T _ {\mathrm {c}} ^ {\mathrm {u}}} \right]} _ {I _ {3}} + \frac {1 0 e A}{\log^ {2} T}. \\ \end{array}
$$
Then, bounding  $I_{3}$  is similar to bounding  $I_{1}$  for Theorem 5. We decompose  $I_{3}$  as follows:
$$
\begin{array}{l} I _ {3} = \mathbb {E} \left[ \mathbb {1} _ {T _ {\mathrm {c}} \leqslant T _ {\mathrm {c}} ^ {\mathrm {u}}} \sum_ {t = 1} ^ {T _ {\mathrm {c}}} \mathbb {1} _ {A _ {t} = a} \right] \\ = 1 + \underbrace {\mathbb {E} \left[ \mathbb {1} _ {T _ {\mathrm {c}} \leqslant T _ {\mathrm {c}} ^ {\mathrm {u}}} \sum_ {t = A + 1} ^ {T _ {\mathrm {c}}} \mathbb {1} _ {A _ {t} = a} \mathbb {1} _ {\cup \mathcal {C B} _ {t - 1} (a) \geqslant \mu_ {1}} \right]} _ {A _ {1}} + \underbrace {\mathbb {E} \left[ \mathbb {1} _ {T _ {\mathrm {c}} \leqslant T _ {\mathrm {c}} ^ {\mathrm {u}}} \sum_ {t = A + 1} ^ {T _ {\mathrm {c}}} \mathbb {1} _ {A _ {t} = a} \mathbb {1} _ {\cup \mathcal {C B} _ {{t - 1}} (a) <   \mu_ {1}} \right]} _ {A _ {2}}, \\ \end{array}
$$
In order to bound  $A_{1}$  and  $A_{2}$ , we assume there is a virtual process that after the end time  $T_{\mathrm{c}}$  of exploration phase, it continues to select the arm with largest UCB and receive the corresponding reward until time  $T_{\mathrm{c}}^{\mathrm{u}}$ . This is only a virtual process used in our proof, while in reality our algorithm will stop exploration after stopping time  $T_{\mathrm{c}}$ . We will use  $\mathbb{E}'$  and  $\mathbb{P}'$  to denote the expectation and probability over this virtual process. Then, we can bound  $A_{2}$  as follows:
$$
\begin{array}{l} A _ {2} \leqslant \mathbb {E} \left[ \mathbb {1} _ {T _ {\mathrm {c}} \leqslant T _ {\mathrm {c}} ^ {\mathrm {u}}} \sum_ {t = A + 1} ^ {T _ {\mathrm {c}}} \mathbb {1} _ {\cup C B _ {t - 1} (1) <   \mu_ {1}} \right] \leqslant \mathbb {E} ^ {\prime} \left[ \sum_ {t = A + 1} ^ {T _ {\mathrm {c}} ^ {\mathrm {u}}} \mathbb {1} _ {\cup C B _ {t - 1} (1) <   \mu_ {1}} \right] \\ = \sum_ {t = A + 1} ^ {T _ {\mathrm {c}} ^ {\mathrm {u}}} \mathbb {P} ^ {\prime} \left(\mathrm {U C B} _ {t - 1} (1) <   \mu_ {1}\right). \\ \end{array}
$$
where the first inequality is because  $A_{t} = a$  indicates  $\mathsf{UCB}_{t - 1}(1) < \mathsf{UCB}_{t - 1}(a)$  due to the dynamic of UCB exploration. The second inequality is because  $\mathbb{E}$  and  $\mathbb{E}'$  are totally the same for the first  $T_{\mathrm{c}}$  time steps. This step also allows us to bound  $A_{2}$  over the events on a different probability measure  $\mathbb{E}'$  and  $\mathbb{P}'$ . Event  $\{\mathsf{UCB}_{t - 1}(1) < \mu_1\}$  means that at time step  $t$ , the mean estimation  $\bar{r}_{t - 1}(1)$  of the optimal arm from previous pulls is lower than  $\mu_{1} - b_{t - 1}(1)$ , which incurs a large deviation, so we can bound this event with a union over all time steps:
$$
\left\{\cup \mathrm {C B} _ {t - 1} (1) <   \mu_ {1} \right\} \subset \left\{\exists t ^ {\prime} \in [ A, T _ {\mathrm {c}} ^ {\mathrm {u}}), \cup \mathrm {C B} _ {t ^ {\prime}} (1) <   \mu_ {1} \right\},
$$
which doesn't depend on time step  $t$  any more, so we have:
$$
\begin{array}{l} \sum_ {t = A + 1} ^ {T _ {\mathrm {c}} ^ {\mathrm {u}}} \mathbb {P} ^ {\prime} \left(\mathrm {U C B} _ {t - 1} (1) <   \mu_ {1}\right) \leqslant \sum_ {t = A + 1} ^ {T _ {\mathrm {c}} ^ {\mathrm {u}}} \mathbb {P} ^ {\prime} \left(\exists t ^ {\prime} \in \left[ A, T _ {\mathrm {c}} ^ {\mathrm {u}}\right), \mathrm {U C B} _ {t ^ {\prime}} (1) <   \mu_ {1}\right) \\ = \left(T _ {\mathrm {c}} ^ {\mathrm {u}} - A\right) \mathbb {P} ^ {\prime} \left(\exists t ^ {\prime} \in \left[ A, T _ {\mathrm {c}} ^ {\mathrm {u}}\right), \cup \mathsf {C B} _ {t ^ {\prime}} (1) <   \mu_ {1}\right) \\ = \left(T _ {\mathrm {c}} ^ {\mathrm {u}} - A\right) \mathbb {P} ^ {\prime} \left(\exists t ^ {\prime} \in \left[ A, T _ {\mathrm {c}} ^ {\mathrm {u}}\right), \bar {r} _ {t ^ {\prime}} (1) + \sqrt {\frac {2 l}{N _ {t ^ {\prime}} (1)}} <   \mu_ {1}\right). \\ \end{array}
$$
If there is no pulls for arm 1 in a time interval, the empirical mean and number of pulls will remain the same. So instead of counting on the time steps, we can count the number of pulls for arm 1:
$$
\left\{\exists t ^ {\prime} \in \left[ A, T _ {c} ^ {u}\right), \bar {r} _ {t ^ {\prime}} (1) + \sqrt {\frac {2 l}{N _ {t ^ {\prime}} (1)}} <   \mu_ {1} \right\} \subset \left\{\exists s \in \left[ 1, N _ {T _ {c} ^ {u}} (a)\right), \bar {r} _ {s, 1} + \sqrt {\frac {2 l}{s}} <   \mu_ {1} \right\}.
$$
Notice that  $N_{T_{\mathrm{c}}^{\mathrm{u}}}(\alpha) \leqslant T_{\mathrm{c}}^{\mathrm{u}} - A + 1$ , so we can bound the probability as follows:
$$
\begin{array}{l} \mathbb {P} ^ {\prime} \left(\exists t ^ {\prime} \in \left[ A, T _ {\mathrm {c}} ^ {\mathrm {u}}\right), \bar {r} _ {t ^ {\prime}} (1) + \sqrt {\frac {2 l}{N _ {t ^ {\prime}} (1)}} <   \mu_ {1}\right) \\ \leqslant \mathbb {P} ^ {\prime} \left(\exists s \in [ 1, T _ {c} ^ {u} - A + 1), \bar {r} _ {s, 1} - \mu_ {1} + \sqrt {\frac {2 l}{s}} <   0\right) \\ \leqslant \mathbb {P} ^ {\prime} \left(\exists s \in [ 1, T _ {\mathrm {c}} ^ {\mathrm {u}} - A + 1), \frac {\sum_ {i = 1} ^ {s} \left(W _ {1 , i} - \mu_ {1}\right)}{i} + \sqrt {\frac {2 l}{s}} <   0\right) \\ \leqslant \frac {T _ {\mathrm {c}} ^ {\mathrm {u}} - A}{\exp (l)}, \\ \end{array}
$$
where the last inequality uses Eq. (4a) of Lemma 3 and the fact that  $W_{1,i} - \mu_1$  is 1-sub-Gaussian with zero mean. Notice that  $\exp(l) \geqslant T \log^4 T$ , so putting the above bound back to the bound of  $A_2$ , we have:
$$
A _ {2} \leqslant \frac {\left(\sum_ {a : \Delta_ {a} > 0} \frac {8 l ^ {2}}{\Delta_ {a} ^ {2}} + A l - A\right) ^ {2}}{T \log^ {4} T} \leqslant \frac {\left(\sum_ {a : \Delta_ {a} > 0} \frac {2 0 0 \log^ {2} T}{\Delta_ {a} ^ {2}} + 5 A \log T\right) ^ {2}}{T \log^ {4} T} = \mathcal {O} (T ^ {- 1}).
$$
where the first inequality is due to the fact that when  $T \geqslant 3$ ,  $\sqrt{\log T} \leqslant \log T$ . Next, we attempt to bound  $A_{1}$  as follows:
$$
A _ {1} = \mathbb {E} \left[ \mathbb {1} _ {T _ {\mathrm {c}} \leqslant T _ {\mathrm {c}} ^ {\mathrm {u}}} \sum_ {t = A + 1} ^ {T _ {\mathrm {c}} ^ {\mathrm {u}}} \mathbb {1} _ {A _ {t} = a} \mathbb {1} _ {\mathsf {U C B} _ {t - 1} (a) \geqslant \mu_ {1}} \right] \leqslant \mathbb {E} ^ {\prime} \left[ \sum_ {t = A + 1} ^ {T _ {\mathrm {c}} ^ {\mathrm {u}}} \mathbb {1} _ {A _ {t} = a} \mathbb {1} _ {\bar {r} _ {t - 1} (a) + \sqrt {\frac {2 l}{N _ {t - 1} (a)}} \geqslant \mu_ {1}} \right],
$$
where the inequality is also due to the fact that the virtual and real processes are identical before time  $T_{c}$ . Notice that the number of pulls  $N_{t}(a)$  will only increase by 1 every time there is new pull, i.e., when  $A_{t} = a$ . Otherwise, the term inside summation is 0. So instead of counting on the time step  $t$ , we can count over the number of pulls over arm  $a$  as follows:
$$
\begin{array}{l} \mathbb {E} ^ {\prime} \left[ \sum_ {t = A + 1} ^ {T _ {\mathrm {c}} ^ {\mathrm {u}}} \mathbb {1} _ {A _ {t} = a} \mathbb {1} _ {\bar {r} _ {t - 1} (a) + \sqrt {\frac {2 l}{N _ {t - 1} (a)}} \geqslant \mu_ {1}} \right] = \mathbb {E} ^ {\prime} \left[ \sum_ {s = 1} ^ {N _ {T _ {\mathrm {c}} ^ {\mathrm {u}}} (a)} \mathbb {1} _ {\bar {r} _ {s, a} + \sqrt {\frac {2 l}{s}} \geqslant \mu_ {1}} \right] \\ \leqslant \sum_ {s = 1} ^ {T _ {\mathrm {c}} ^ {\mathrm {u}}} \mathbb {P} ^ {\prime} \left(\bar {r} _ {s, a} - \mu_ {a} + \sqrt {\frac {2 l}{s}} \geqslant \Delta_ {a}\right), \\ \end{array}
$$
where the last inequality is due to  $N_{T_{\mathrm{c}}^{\mathrm{u}}}(\alpha) \leqslant T_{\mathrm{c}}^{\mathrm{u}}$ . Then, by Eq. (4c) of Lemma 3, we have:
$$
\begin{array}{l} \sum_ {s = 1} ^ {T _ {\mathrm {c}} ^ {\mathrm {u}}} \mathbb {P} ^ {\prime} \left(\bar {r} _ {s, a} - \mu_ {a} + \sqrt {\frac {2 l}{s}} \geqslant \Delta_ {a}\right) = \sum_ {s = 1} ^ {T _ {\mathrm {c}} ^ {\mathrm {u}}} \mathbb {P} ^ {\prime} \left(\frac {\sum_ {i = 1} ^ {s} \left(W _ {a , i} - \mu_ {a}\right)}{i} + \sqrt {\frac {2 l}{s}} \geqslant \Delta_ {a}\right) \\ \leqslant \frac {2 (\log T + 4 \sqrt {2 \log T}) + \sqrt {4 \pi (\log T + 4 \sqrt {2 \log T})} + 2}{\Delta_ {a} ^ {2}} + 1 \\ \leqslant \frac {2 \log T}{\Delta_ {a} ^ {2}} + \frac {(8 + \sqrt {2 0 \pi}) \sqrt {\log T}}{\Delta_ {a} ^ {2}} + \frac {2}{\Delta_ {a} ^ {2}} + 1, \\ \end{array}
$$
where the first inequality is due to the fact that  $W_{a,i} - \mu_a$  is 1-sub-Gaussian with zero mean, and the last inequality is due to  $\sqrt{2\log T} \leqslant \log T$  when  $T \geqslant 9$ . Therefore, combining the bounds on  $A_1$  and  $A_2$ , we can bound  $I_3$  as follows:
$$
I _ {3} \leqslant \frac {2 \log T}{\Delta_ {a} ^ {2}} + \frac {(8 + \sqrt {2 0 \pi}) \sqrt {\log T}}{\Delta_ {a} ^ {2}} + \frac {2}{\Delta_ {a} ^ {2}} + 1 + \mathcal {O} (T ^ {- 1}).
$$
Therefore, a similar bound can be established on  $I_{1}$  as follows:
$$
I _ {1} \leqslant \frac {2 \log T}{\Delta_ {a} ^ {2}} + \frac {(8 + \sqrt {2 0 \pi}) \sqrt {\log T}}{\Delta_ {a} ^ {2}} + \frac {2}{\Delta_ {a} ^ {2}} + 1 + \frac {1 0 e A}{\log^ {2} T} + \mathcal {O} (T ^ {- 1}).
$$
Bounding  $I_{2}$ : Recall that our stopping criterion is when there exists an arm  $\tilde{a}$  whose number of pulls is significantly larger than other arms, i.e.,  $N_{T_{\mathrm{c}}}(\tilde{a}) \geqslant l \max_{a \neq \tilde{a}} N_{T_{\mathrm{c}}}(\tilde{a})$ . Therefore, its bonus  $b_{T_{\mathrm{c}}}(\tilde{a})$  should be very small compared to other arms. Also recall that we select the arm  $\hat{a}$  with highest LCB to commit to, so the proof follows two steps. First we will show that with high probability the arm  $\tilde{a}$  with most number of pulls is the best arm. Then we will show that under this circumstance, the maximum LCB arm is also the best arm. Therefore, consider an arbitrary sub-optimal arm  $a$ :
$$
\mathbb {P} (\tilde {a} = a) \leqslant \mathbb {P} \left(N _ {T _ {\mathrm {c}}} (a) \geqslant \lceil l N _ {T _ {\mathrm {c}}} (1) \rceil + 1\right).
$$
Notice that if  $N_{T_{\mathrm{c}}}(\alpha) \geqslant \lceil lN_{T_{\mathrm{c}}}(\alpha)\rceil +1$  at the end time  $T_{\mathrm{c}}$ , we must have pulled arm  $a$  at time step  $T_{\mathrm{c}}$ . This also means that the number of pulls  $N_{T_{\mathrm{c}} - 1}(\alpha)$  for arm  $a$  after time step  $T_{\mathrm{c}} - 1$  is exactly  $\lceil lN_{T_{\mathrm{c}}}(\alpha)\rceil$  and  $N_{T_{\mathrm{c}} - 1}(1) = N_{T_{\mathrm{c}}}(\alpha)$ . It also means that arm  $a$  has the largest UCB. So we have:
$$
\begin{array}{l} \mathbb {P} \left(N _ {T _ {\mathrm {c}}} (a) \geqslant \lceil N _ {T _ {\mathrm {c}}} (1) \rceil + 1\right) \\ \leqslant \mathbb {P} \left(N _ {T _ {\mathrm {c}} - 1} (a) = \left[ l N _ {T _ {\mathrm {c}} - 1} (1) \right], \cup \mathsf {C B} _ {T _ {\mathrm {c}} - 1} (a) \geqslant \cup \mathsf {C B} _ {T _ {\mathrm {c}} - 1} (1)\right) \\ \leqslant \mathbb {P} \left(\exists t \leqslant T _ {\mathrm {c}}, N _ {t} (a) = \lceil l N _ {t} (1) \rceil , \bar {r} _ {t} (a) + \sqrt {\frac {2 l}{N _ {t} (a)}} \geqslant \bar {r} _ {t} (1) + \sqrt {\frac {2 l}{N _ {t} (1)}}\right). \\ \end{array}
$$
Notice that we can separate the two random quantities with union bound as follows:
$$
\begin{array}{l} \mathbb {P} \left(N _ {T _ {\mathrm {c}}} (a) \geqslant \lceil N _ {T _ {\mathrm {c}}} (1) \rceil + 1\right) \\ \leqslant \underbrace {\mathbb {P} \left(\exists t \leqslant T _ {\mathrm {c}} , \bar {r} _ {t} (a) \geqslant \mu_ {a} + \sqrt {\frac {2 l}{N _ {t} (a)}}\right)} _ {B _ {1}} \\ + \underbrace {\mathbb {P} \left(\exists t \leqslant T _ {\mathrm {c}} , N _ {t} (a) = \lceil l N _ {t} (1) \rceil , \mu_ {a} + 2 \sqrt {\frac {2 l}{N _ {t} (a)}} \geqslant \bar {r} _ {t} (1) + \sqrt {\frac {2 l}{N _ {t} (1)}}\right)} _ {B _ {2}}. \\ \end{array}
$$
We first bound  $B_{1}$ . Notice that we can switch from counting of time step  $t$  to count the number of pulls for arm  $a$ . It is clear that  $N_{t}(a) \leqslant T_{\mathrm{c}} \leqslant T$  when  $t \leqslant T_{\mathrm{c}}$ , so we have:
$$
\begin{array}{l} B _ {1} \leqslant \mathbb {P} \left(\exists s \leqslant T, \bar {r} _ {s, a} \geqslant \mu_ {a} + \sqrt {\frac {2 l}{s}}\right) \\ = \mathbb {P} \left(\exists s \leqslant T, \frac {\sum_ {i = 1} ^ {s} \left(\mu_ {a} - W _ {a , i}\right)}{s} + \sqrt {\frac {2 l}{s}} \leqslant 0\right) \\ \leqslant \frac {e l \log T + e}{\exp l}, \\ \end{array}
$$
where the last inequality uses Eq. (4a) of Lemma 3. Also notice that  $\exp(l) \geqslant T \log^4 T$ , so we have:
$$
B _ {1} \leqslant \frac {1 0 e \log^ {2} T + e}{T \log^ {4} T} = o \left(\frac {1}{T}\right).
$$
Next, we bound the term  $B_{2}$ , we can rearrange the terms inside  $B_{2}$  as follows:
$$
B _ {2} = \mathbb {P} \left(\exists t \leqslant T _ {\mathrm {c}}, N _ {t} (a) = \lceil l N _ {t} (1) \rceil , \bar {r} _ {t} (1) - \mu_ {1} + \left(1 - \frac {2}{\sqrt {l}}\right) \sqrt {\frac {2 l}{N _ {t} (1)}} + \Delta_ {a} \leqslant 0\right).
$$
Denote  $\alpha = 1 - \frac{2}{\sqrt{t}}$ , and switch the count from time step  $t$  to the number of pulls for arm 1, we can bound  $B_{2}$  as follows:
$$
\begin{array}{l} B _ {2} \leqslant \mathbb {P} \left(\exists s \leqslant T, \bar {r} _ {s, 1} - \mu_ {1} + \sqrt {\frac {2 \alpha^ {2} l}{s}} + \Delta_ {a} \leqslant 0\right) \\ = \mathbb {P} \left(\exists s \leqslant T, \frac {\sum_ {i = 1} ^ {s} \left(W _ {1 , i} - \mu_ {1}\right)}{s} + \sqrt {\frac {2 \alpha^ {2} l}{s}} + \Delta_ {a} \leqslant 0\right) \\ \leqslant \frac {4}{\Delta_ {a} ^ {2} \exp (\alpha^ {2} l)}. \\ \end{array}
$$
When  $T$  is large enough, we have  $\alpha^2 l = l - 4\sqrt{l} + 4 \geqslant \log T + 4$ , so we have:
$$
B _ {2} \leqslant \frac {4}{\Delta_ {a} ^ {2} T}.
$$
Therefore, combining  $B_{1}$  and  $B_{2}$  together, we can bound the probability as follows:
$$
\mathbb {P} \left(\tilde {a} = a\right) \leqslant \mathbb {P} \left(N _ {T _ {\mathrm {c}}} (a) \geqslant \left\lceil l N _ {T _ {\mathrm {c}}} (1) \right\rceil + 1\right) \leqslant B _ {1} + B _ {2} \leqslant \mathcal {O} \left(\frac {1}{T}\right).
$$
Recall that arm 1 is the optimal arm. Therefore, by a union bound, we can characterize the probability that of event  $\{\tilde{a} \neq 1\}$  as follows:
$$
\mathbb {P} \left(\tilde {a} \neq 1\right) \leqslant \sum_ {a: \Delta_ {a} > 0} \mathbb {P} \left(\tilde {a} = a\right) \leqslant \mathcal {O} \left(\frac {1}{T}\right).
$$
Next, we investigate the probability of choosing the wrong arm  $\hat{a}$  to commit to. Recall that for the arm which we commit to, we choose the one with the largest LCB. Consider any sub-optimal arm  $a$  if we wrongly choose the arm  $\hat{a} = a$ , it means its LCB should be larger than the LCB of the optimal arm, which with high probability has the largest number of pulls. So, we have:
$$
\mathbb {P} \left(\hat {a} = a\right) \leqslant \mathbb {P} \left(\hat {a} = a, \tilde {a} = 1\right) + \mathbb {P} \left(\tilde {a} \neq 1\right) \leqslant \mathbb {P} \left(\mathrm {L C B} _ {T _ {\mathrm {c}}} (a) \geqslant \mathrm {L C B} _ {T _ {\mathrm {c}}} (1), \tilde {a} = 1\right) + \mathcal {O} \left(\frac {1}{T}\right).
$$
When  $\tilde{a} = 1$ , arm 1 has  $l$  times more pulls than arm  $a$  when the exploration phase stops, so we have: