SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
R (n) \leq 8 \sigma^ {2} \log (1 / \delta) \log n \sum_ {a \in \mathcal {A}} \int_ {\theta_ {- a}} h _ {a} \left(\theta_ {a} ^ {*} - \xi_ {a} \left(\theta_ {a} ^ {*}\right)\right) h _ {- a} \left(\theta_ {- a}\right) \mathrm {d} \theta_ {- a} + C,
$$
where  $C = 8\sigma^2 K\log (1 / \delta)\sqrt{\log n} +2(\sqrt{2\log(1 / \delta)} +2K)\sigma Kn\delta +\sum_{a\in \mathcal{A}}\mathbb{E}\left[\Delta_a\right] + 1.$
The proof is in Appendix A.4. For  $\delta = 1 / n$ , the bound is  $O(c_h\log^2 n)$  and similar to Corollary 2. The main difference is in the additional factor  $\frac{\sigma^2}{2\sigma_0^2\log n}$  in Corollary 2, which decreases the bound. This means that the regret of BayesUCB improves as  $\sigma_0$  decreases while that of UCB1 may not change much. In fact, the regret bound of BayesUCB is  $O(1)$  as  $\sigma_0 \to 0$  (Appendix B) while that of UCB1 remains logarithmic. This is expected because BayesUCB has more information about the random instance  $\theta$  as  $\sigma_0$  decreases, while the frequentist algorithm is oblivious to the prior.
4.3 BayesUCB in Bernoulli Bandit
Theorem 1 and Corollary 2 can be straightforwardly extended to Bernoulli bandits because
$$
\mathbb {P} \left(\left| \theta_ {a} - \hat {\theta} _ {t, a} \right| \geq C _ {t, a} \mid H _ {t}\right) \leq 2 \delta
$$
holds for any action  $a$  and history  $H_{t}$  (Section 3.2). We state the extension below and prove it in Appendix A.5.
Theorem 5. For any  $\varepsilon >0$  and  $\delta \in (0,1)$ , the  $n$ -round Bayes regret of BayesUCB in a  $K$ -armed Bernoulli bandit is bounded as
$$
R (n) \leq \mathbb {E} \left[ \sum_ {a \neq A _ {*}} \frac {2 \log (1 / \delta)}{\Delta_ {a} ^ {\varepsilon}} - (\alpha_ {a} + \beta_ {a} + 1) \Delta_ {a} ^ {\varepsilon} \right] + C,
$$
where  $C = \varepsilon n + 2Kn\delta$  is a low-order term.
Moreover, let  $\lambda = \min_{a\in \mathcal{A}}\alpha_{a} + \beta_{a} + 1$  and  $\lambda \leq 2\log (1 / \delta) n^2\log \log n$ . Then
$$
R (n) \leq \left[ 2 \log (1 / \delta) \log n - \frac {\lambda}{2 \log n} \right] \sum_ {a \in \mathcal {A}} \int_ {\theta_ {- a}} h _ {a} \left(\theta_ {a} ^ {*} - \xi_ {a} \left(\theta_ {a} ^ {*}\right)\right) h _ {- a} \left(\theta_ {- a}\right) \mathrm {d} \theta_ {- a} + C,
$$
where  $C = 2K\log (1 / \delta)\sqrt{\log n} +2Kn\delta +1$  is a low-order term.
4.4 BayesUCB in Linear Bandit
Now we present a gap-dependent Bayes regret bound for BayesUCB in a linear bandit with a finite number of actions. The bound depends on a random minimum gap. To control the gap, we clip it as  $\Delta_{\mathrm{min}}^{\varepsilon} = \max \left\{\Delta_{\mathrm{min}}, \varepsilon \right\}$ .
Theorem 6. Suppose that  $\| \theta \| _2\leq L_*$  holds with probability at least  $1 - \delta_{*}$ . Let  $\| a\| _2\leq L$  for any action  $a\in \mathcal{A}$ . Then for any  $\varepsilon >0$  and  $\delta \in (0,1)$ , the  $n$ -round Bayes regret of linear BayesUCB is bounded as
$$
R (n) \leq 8 \mathbb {E} \left[ \frac {1}{\Delta_ {\min } ^ {\varepsilon}} \right] \frac {\sigma_ {0 , \max } ^ {2} d}{\log \left(1 + \frac {\sigma_ {0 , \max } ^ {2}}{\sigma^ {2}}\right)} \log \left(1 + \frac {\sigma_ {0 , \max } ^ {2} n}{\sigma^ {2} d}\right) \log (1 / \delta) + \varepsilon n + 4 L L _ {*} K n \delta
$$
with probability at least  $1 - \delta_{*}$ , where  $\sigma_{0,\max} = \sqrt{\lambda_1(\Sigma_0)L}$ .
The proof is in Appendix A.6. For  $\varepsilon = 1 / n$  and  $\delta = 1 / n$ , the bound is  $O(dc_{\Delta}\log^{2}n)$ , where  $c_{\Delta}$  is a constant depending on the gaps of random bandit instances. The bound is remarkably similar to the frequentist  $O(d\Delta_{\min}^{-1}\log^{2}n)$  bound in Theorem 5 of Abbasi-Yadkori et al. [2011], where  $\Delta_{\min}$  is the minimum gap. There are two differences. First, we integrate  $\Delta_{\min}^{-1}$  over the prior. Second, our bound decreases as the prior becomes more informative,  $\sigma_{0,\max}\rightarrow 0$ .
In a Gaussian bandit, the bound becomes  $O(K\mathbb{E}\left[1 / \Delta_{\mathrm{min}}^{\varepsilon}\right]\log^{2}n)$ . Therefore, it is comparable to Theorem 1 up to an additional logarithmic factor in  $n$ . This is due to a more general proof technique, which allows for dependencies between the mean rewards of actions. We also note that Theorem 6 does not assume that the prior is factored, unlike Theorem 1.
5 Comparison to Prior Works
This section is organized as follows. In Section 5.1, we show that the bound in Theorem 5 matches an existing lower bound of Lai [1987] asymptotically. In Section 5.2, we compare our logarithmic bounds to prior  $\tilde{O} (\sqrt{n})$  bounds. Finally, in Section 5.3, we outline the key steps in our analyses and how they differ from prior works.
5.1 Matching Lower Bound
In frequentist bandit analyses, it is standard to compare asymptotic lower bounds to finite-time upper bounds because finite-time logarithmic lower bounds do not exist [Lattimore and Szepesvari, 2019]. We follow the same approach when arguing that our finite-time upper bounds are order optimal.
The results in Lai [1987] are for single-parameter exponential-family reward distributions, which excludes Gaussian rewards. Therefore, we argue about the tightness of Theorem 5 only. Specifically, we take the second bound in Theorem 5, set  $\delta = 1 / n$ , and let  $n \to \infty$ . In this case,  $\frac{\lambda}{2\log n} \to 0$  and  $\xi_{a}(\cdot) \to 0$ , and the bound matches up to constant factors the lower bound in Lai [1987] (claim (ii) in their Theorem 3), which is
$$
\Omega \left(\log^ {2} n \sum_ {a \in \mathcal {A}} \int_ {\theta_ {- a}} h _ {a} \left(\theta_ {a} ^ {*}\right) h _ {- a} \left(\theta_ {- a}\right) \mathrm {d} \theta_ {- a}\right). \tag {2}
$$
Lai [1987] also matched this lower bound with an asymptotic upper bound for a frequentist policy.
Our finite-time upper bounds also reveal an interesting difference from the asymptotic lower bound in (2), which may deserve more future attention. More specifically, the regret bound of BayesUCB (Corollary 2) improves with prior information while that of UCB1 (Theorem 4) does not. We observe these improvements empirically as well (Section 6 and Appendix D). However, both bounds are the same asymptotically. This is because the benefit of knowing the prior vanishes in asymptotic analyses, since  $\frac{\sigma^2}{2\sigma_0^2\log n} \to 0$  in Corollary 2 as  $n \to \infty$ . This motivates the need for finite-time logarithmic Bayes regret lower bounds, which do not exist.
5.2 Prior Bayes Regret Upper Bounds
Theorem 1 and Corollary 2 are major improvements upon existing  $\tilde{O}(\sqrt{n})$  bounds. For instance, take a prior-dependent bound in Lemma 4 of Kveton et al. [2021], which holds for both BayesUCB and Thompson sampling due to a well-known equivalence of their analyses [Russo and Van Roy, 2014, Hong et al., 2020]. For  $\delta = 1/n$ , their leading term becomes
$$
4 \sqrt {2 \sigma^ {2} K \log n} \left(\sqrt {n + \sigma^ {2} \sigma_ {0} ^ {- 2} K} - \sqrt {\sigma^ {2} \sigma_ {0} ^ {- 2} K}\right). \tag {3}
$$
Similarly to Theorem 1 and Corollary 2, (3) decreases as the prior concentrates and becomes more informative,  $\sigma_0\rightarrow 0$ . However, the bound is  $\tilde{O} (\sqrt{n})$ . Moreover, it does not depend on prior means  $\mu_0$  or the gaps of random bandit instances. Therefore, it cannot capture low-regret regimes due to large random gaps  $\Delta_{a}^{\varepsilon}$  in Theorem 1 or a small complexity term in Corollary 2. We demonstrate it empirically in Section 6.
When the random gaps  $\Delta_{a}^{\varepsilon}$  in Theorem 1 are small or the complexity term in Corollary 2 is large, our bounds can be worse than  $\tilde{O} (\sqrt{n})$  bounds. This is analogous to the relation of the gap-dependent and gap-free frequentist bounds [Lattimore and Szepesvari, 2019]. Specifically, a gap-dependent bound of UCB1 in a  $K$ -armed bandit with 1-sub-Gaussian rewards (Theorem 7.1) is  $O(K\Delta_{\min}^{-1}\log n)$ , where  $\Delta_{\min}$  is the minimum gap. A corresponding gap-free bound (Theorem 7.2) is  $O(\sqrt{Kn\log n})$ . The latter is smaller when the gap is small,  $\Delta_{\min} = o(\sqrt{(K\log n) / n})$ . To get the best bound, the minimum of the two should be taken, and the same is true in our Bayesian setting.
No prior-dependent Bayes regret lower bound exists in linear bandits. Thus we treat  $\mathbb{E}\left[1 / \Delta_{\min}^{\varepsilon}\right]$  in Theorem 6 as the complexity term and do not further bound it as in Corollary 2. To compare our bound fairly to existing  $\tilde{O} (\sqrt{n})$  bounds, we derive an  $\tilde{O} (\sqrt{n})$  bound in Appendix C, by a relatively minor change in the proof of Theorem 6. A similar bound can be obtained by adapting the proofs of Lu and Van Roy [2019] and Hong et al. [2022] to a linear bandit with a finite number of actions. The leading term of the bound is
$$
2 \sqrt {\frac {2 \sigma_ {0 , \max } ^ {2} d n}{\log \left(1 + \frac {\sigma_ {0 , \max } ^ {2}}{\sigma^ {2}}\right)} \log \left(1 + \frac {\sigma_ {0 , \max } ^ {2} n}{\sigma^ {2} d}\right) \log (1 / \delta)}. \tag {4}
$$
Similarly to Theorem 6, (4) decreases as the prior becomes more informative,  $\sigma_{0,\max} \to 0$ . However, the bound is  $\tilde{O}(\sqrt{n})$  and does not depend on the gaps of random bandit instances. Hence it cannot capture low-regret regimes due to a large random minimum gap  $\Delta_{\min}$  in Theorem 6. We validate it empirically in Section 6.
5.3 Technical Novelty
All modern Bayesian analyses follow Russo and Van Roy [2014], who derived the first finite-time  $\tilde{O} (\sqrt{n})$  Bayes regret bounds for BayesUCB and Thompson sampling. The key idea in their analyses is that conditioned on history, the optimal and taken actions are identically distributed, and that the upper confidence bounds are deterministic functions of the history. This is where the randomness of instances in Bayesian bandits is used. Using this, the regret at round  $t$  is bounded by the confidence interval width of the taken action, and the usual  $\tilde{O} (\sqrt{n})$  bounds can be obtained by summing up the confidence interval widths over  $n$  rounds.
The main difference in our work is that we first bound the regret in a fixed bandit instance, similarly to frequentist analyses. The bound involves  $\Delta_{a}^{-1}$  and is derived using biased Bayesian confidence intervals. The rest of our analysis is Bayesian in two parts: we prove that the confidence intervals fail with a low probability and bound random  $\Delta_{a}^{-1}$ , following a similar technique to Lai [1987]. The resulting logarithmic Bayes regret bounds cannot be derived using the techniques of Russo and Van Roy [2014], as these become loose when the confidence interval widths are introduced.
Asymptotic logarithmic Bayes regret bounds were derived in Lai [1987]. From this analysis, we use only the technique for bounding  $\Delta_a^{-1}$  when proving Corollary 2 and Theorem 5. The central part of our proof is a finite-time per-instance bound on the number of times that a suboptimal action is taken. This quantity is bounded based on the assumption that the action with the highest UCB is taken. A
Figure 1: Gaussian bandit as (a) the prior width  $\sigma_0$  and (b) the prior gap  $\Delta_0$  change.
comparable argument in Theorem 2 of Lai [1987] is asymptotic and on average over random bandit instances.
We experiment with UCB algorithms in two environments: Gaussian bandits (Section 3.1) and linear bandits with Gaussian rewards (Section 3.3). In both experiments, the horizon is  $n = 1000$  rounds. All results are averaged over 10 000 random runs. Shaded regions in the plots are standard errors of the estimates. They are generally small because the number of runs is high.
6.1 Gaussian Bandit
The first problem is a  $K$ -armed bandit with  $K = 10$  actions (Section 3.1). The prior width is  $\sigma_0 = 1$ . The prior mean is  $\mu_0$ , where  $\mu_{0,1} = \Delta_0$  and  $\mu_{0,a} = 0$  for  $a > 1$ . We set  $\Delta_0 = 1$  and call it the prior gap. We vary  $\sigma_0$  and  $\Delta_0$  in our experiments, and observe how the regret and its upper bounds change as the problem hardness varies (Sections 4.1 and 5.2).
We plot five trends: (a) Bayes regret of BayesUCB. (b) Bayes regret of UCB1 (Section 4.2), which is a comparable frequentist algorithm to BayesUCB. (c) A leading term of the BayesUCB regret bound in Theorem 1, where  $\varepsilon = 1/n$  and  $\delta = 1/n$ . (d) A leading term of the UCB1 regret bound: This is the same as (c) with  $\sigma_0 = \infty$ . (e) An existing  $\tilde{O}(\sqrt{n})$  regret bound in (3).
Our results are reported in Figure 1. We observe three major trends. First, the regret of BayesUCB decreases as the problem becomes easier, either  $\sigma_0\rightarrow 0$  or  $\Delta_0\to \infty$ . It is also lower than that of UCB1, which does not leverage the prior. Second, the regret bound of BayesUCB is tighter than that of UCB1, due to capturing the benefit of the prior. Finally, the logarithmic regret bounds are tighter than the  $\tilde{O} (\sqrt{n})$  bound. In addition, the  $\tilde{O} (\sqrt{n})$  bound depends on the prior only through  $\sigma_0$  and thus remains constant as the prior gap  $\Delta_0$  changes.
In Appendix D, we compare BayesUCB to UCB1 more comprehensively for various  $K$ ,  $\sigma$ ,  $\Delta_0$ , and  $\sigma_0$ . In all experiments, BayesUCB has a lower regret than UCB1. This also happens when the noise is not Gaussian, which a testament to the robustness of Bayesian methods to model misspecification.
6.2 Linear Bandit with Gaussian Rewards
The second problem is a linear bandit with  $K = 30$  actions in  $d = 10$  dimensions (Section 3.3). The prior covariance is  $\Sigma_0 = \sigma_0^2 I_d$ . The prior mean is  $\theta_0$ , where  $\theta_{0,1} = \Delta_0$  and  $\theta_{0,i} = -1$  for  $i > 1$ . As in Section 6.1, we set  $\Delta_0 = 1$  and call it the prior gap. The action set  $\mathcal{A}$  is generated as follows. The first  $d$  actions are the canonical basis in  $\mathbb{R}^d$ . The remaining  $K - d$  actions are sampled uniformly at random from the positive orthant and scaled to unit length. This ensures that the first action has the highest mean reward, of  $\Delta_0$ , under the prior mean  $\theta_0$ . We vary  $\sigma_0$  and  $\Delta_0$ , and observe how the regret and its upper bounds change as the problem hardness varies (Sections 4.4 and 5.2). We plot three trends: (a) Bayes regret of BayesUCB. (b) A leading term of the BayesUCB regret bound in Theorem 6, where  $\sigma_{0,\max} = \sigma_0$ ,  $\varepsilon = 1 / n$ , and  $\delta = 1 / n$ . (c) An existing  $\tilde{O}(\sqrt{n})$  regret bound in (4).
Figure 2: Linear bandit as (a) the prior width  $\sigma_0$  and (b) the prior gap  $\Delta_0$  change.
Our results are reported in Figure 2 and we observe three major trends. First, the regret of BayesUCB decreases as the problem becomes easier, either  $\sigma_0\rightarrow 0$  or  $\Delta_0\to \infty$ . Second, the regret bound of BayesUCB decreases as the problem becomes easier. Finally, our logarithmic regret bound can also be tighter than the  $\tilde{O} (\sqrt{n})$  bound. In particular, the  $\tilde{O} (\sqrt{n})$  bound depends on the prior only through  $\sigma_0$ , and thus remains constant as the prior gap  $\Delta_0$  changes. We discuss when our bounds could be looser than  $\tilde{O} (\sqrt{n})$  bounds in Section 5.2.
Finite-time logarithmic frequentist regret bounds are the standard way of analyzing  $K$ -armed bandits [Auer et al., 2002, Garivier and Cappe, 2011, Agrawal and Goyal, 2012]. In our work, we prove the first comparable finite-time bounds, logarithmic in  $n$ , in the Bayesian setting. This is a major step in theory and a significant improvement upon prior  $\tilde{O}(\sqrt{n})$  Bayes regret bounds that have become standard in the literature. Comparing to frequentist regret bounds, our bounds capture the value of prior information given to the learner. Our proof technique is general and we also apply it to linear bandits.
This work can be extended in many directions. First, our analyses only need closed-form posteriors, which are available for other reward distributions than Gaussian and Bernoulli. Second, our linear bandit analysis (Section 4.4) seems preliminary when compared to our multi-armed bandit analyses. As an example, the complexity term  $\mathbb{E}\left[1 / \Delta_{\mathrm{min}}^{\varepsilon}\right]$  in Theorem 6 could be bounded as in Corollary 2 and Theorem 5. We do not do this because the main reason for deriving the  $O(c_h\log^2 n)$  bound in Theorem 5, an upper bound on the corresponding  $O(c_{\Delta}\log n)$  bound, is that it matches the lower bound in (2). No such instance-dependent lower bound exists in Bayesian linear bandits. Third, we believe that our approach can be extended to information-theory bounds [Russo and Van Roy, 2016] and discuss it in Appendix E. Fourth, although we only analyze BayesUCB, we believe that similar guarantees can be obtained for Thompson sampling. Finally, we would like to extend our results to reinforcement learning, for instance by building on the work of Lu and Van Roy [2019].
There have been recent attempts in theory [Wagenmaker and Foster, 2023] to design general adaptive algorithms with finite-time instance-dependent bounds based on optimal allocations. The promise of these methods is a higher statistical efficiency than exploring by optimism, which we adopt in this work. One of their shortcomings is that they are not guaranteed to be computationally efficient, as discussed in Section 2.2 of Wagenmaker and Foster [2023]. This work is also frequentist.
Yasin Abbasi-Yadkori, David Pal, and Csaba Szepesvari. Improved algorithms for linear stochastic bandits. In Advances in Neural Information Processing Systems 24, pages 2312-2320, 2011.
Marc Abeille and Alessandro Lazaric. Linear Thompson sampling revisited. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, 2017.
Tor Lattimore and Csaba Szepesvari. Bandit Algorithms. Cambridge University Press, 2019.
Lihong Li, Wei Chu, John Langford, and Robert Schapire. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th International Conference on World Wide Web, 2010.
Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: A novel bandit-based approach to hyperparameter optimization. Journal of Machine Learning Research, 18(185):1-52, 2018.
Shuai Li, Alexandros Karatzoglou, and Claudio Gentile. Collaborative filtering bandits. In Proceedings of the 39th Annual International ACM SIGIR Conference, 2016.
Xiuyuan Lu and Benjamin Van Roy. Information-theoretic confidence bounds for reinforcement learning. In Advances in Neural Information Processing Systems 32, 2019.
Olivier Marchal and Julyan Arbel. On the sub-Gaussianity of the beta and Dirichlet distributions. Electronic Communications in Probability, 22:1-14, 2017.
Daniel Russo and Benjamin Van Roy. Learning to optimize via posterior sampling. Mathematics of Operations Research, 39(4):1221-1243, 2014.
Daniel Russo and Benjamin Van Roy. An information-theoretic analysis of Thompson sampling. Journal of Machine Learning Research, 17(68):1-30, 2016.
Daniel Russo, Benjamin Van Roy, Abbas Kazerouni, Ian Osband, and Zheng Wen. A tutorial on Thompson sampling. Foundations and Trends in Machine Learning, 11(1):1-96, 2018.
Max Simchowitz, Christopher Tosh, Akshay Krishnamurthy, Daniel Hsu, Thodoris Lykouris, Miro Dudik, and Robert Schapire. Bayesian decision-making under misspecified priors with applications to meta-learning. In Advances in Neural Information Processing Systems 34, 2021.
William R. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3-4):285-294, 1933.
John Tsitsiklis. A short proof of the gittins index theorem. Neural Computation, 4(1):194-199, 1994.
Andrew Wagenmaker and Dylan Foster. Instance-optimality in interactive decision making: Toward a non-asymptotic theory. In Proceedings of the 36th Annual Conference on Learning Theory, 2023.
Zhi Wang, Chicheng Zhang, Manish Kumar Singh, Laurel Riek, and Kamalika Chaudhuri. Multitask bandit learning through heterogeneous feedback aggregation. In Proceedings of the 24th International Conference on Artificial Intelligence and Statistics, 2021.
Xiaoxue Zhao, Weinan Zhang, and Jun Wang. Interactive collaborative filtering. In Proceedings of the 22nd ACM International Conference on Information and Knowledge Management, pages 1411-1420, 2013.
We present a general approach for deriving finite-time logarithmic Bayes regret bounds. We start with  $K$ -armed bandits and then extend it to linear bandits.
Let  $E_{t} = \left\{\forall a\in \mathcal{A}:|\theta_{a} - \hat{\theta}_{t,a}|\leq C_{t,a}\right\}$  be the event that all confidence intervals at round  $t$  hold. Fix  $\varepsilon >0$ . We start with decomposing the  $n$ -round regret as
$$
\begin{array}{l} \sum_ {t = 1} ^ {n} \mathbb {E} \left[ \Delta_ {A _ {t}} \right] \leq \sum_ {t = 1} ^ {n} \mathbb {E} \left[ \Delta_ {A _ {t}} \mathbb {1} \left\{\Delta_ {A _ {t}} \geq \varepsilon , E _ {t} \right\} \right] + \sum_ {t = 1} ^ {n} \mathbb {E} \left[ \Delta_ {A _ {t}} \mathbb {1} \left\{\Delta_ {A _ {t}} <   \varepsilon \right\} \right] + \tag {5} \\ \sum_ {t = 1} ^ {n} \mathbb {E} \left[ \Delta_ {A _ {t}} \mathbb {1} \left\{\bar {E} _ {t} \right\} \right]. \\ \end{array}
$$
We bound the first term using the design of BayesUCB and its closed-form posteriors.
Case 1: Event  $E_{t}$  occurs and the gap is large,  $\Delta_{A_t} \geq \varepsilon$ . Since  $E_{t}$  occurs and the action with the highest UCB is taken, the regret at round  $t$  can be bounded as
$$
\Delta_ {A _ {t}} = \theta_ {A _ {*}} - \theta_ {A _ {t}} \leq \theta_ {A _ {*}} - U _ {t, A _ {*}} + U _ {t, A _ {t}} - \theta_ {A _ {t}} \leq U _ {t, A _ {t}} - \theta_ {A _ {t}} \leq 2 C _ {t, A _ {t}}.
$$
In the second inequality, we use that  $\theta_{A_{*}}\leq U_{t,A_{*}}$  on event  $E_{t}$ . This implies that on event  $E_{t}$ , action  $a$  can be taken only if
$$
\Delta_ {a} \leq 2 C _ {t, a} = 2 \sqrt {2 \hat {\sigma} _ {t , a} ^ {2} \log (1 / \delta)} = 2 \sqrt {\frac {2 \log (1 / \delta)}{\sigma_ {0} ^ {- 2} + \sigma^ {- 2} N _ {t , a}}},
$$
which can be rearranged as
$$
N _ {t, a} \leq \frac {8 \sigma^ {2} \log (1 / \delta)}{\Delta_ {a} ^ {2}} - \sigma^ {2} \sigma_ {0} ^ {- 2}.
$$
Therefore, the number of times that action  $a$  is taken in  $n$  rounds while all confidence intervals hold,  $N_{a} = \sum_{t=1}^{n} \mathbb{1}\{A_{t} = a, E_{t}\}$ , is bounded as
$$
N _ {a} \leq \frac {8 \sigma^ {2} \log (1 / \delta)}{\Delta_ {a} ^ {2}} - \sigma^ {2} \sigma_ {0} ^ {- 2}. \tag {6}
$$
Now we apply this inequality to bound the first term in (5) as
$$
\sum_ {t = 1} ^ {n} \mathbb {E} \left[ \Delta_ {A _ {t}} \mathbb {1} \left\{\Delta_ {A _ {t}} \geq \varepsilon , E _ {t} \right\} \right] \leq \mathbb {E} \left[ \sum_ {a \neq A _ {*}} \left(\frac {8 \sigma^ {2} \log (1 / \delta)}{\Delta_ {a}} - \sigma^ {2} \sigma_ {0} ^ {- 2} \Delta_ {a}\right) \mathbb {1} \left\{\Delta_ {a} \geq \varepsilon \right\} \right]. \tag {7}
$$
Case 2: The gap is small,  $\Delta_{A_t} < \varepsilon$ . Then naively  $\sum_{t=1}^{n} \mathbb{E}\left[\Delta_{A_t} \mathbb{1}\{\Delta_{A_t} < \varepsilon\}\right] < \varepsilon n$ .
Case 3: Event  $E_{t}$  does not occur. The last term in (5) can be bounded as