SlowGuess/NeurIPS_2023_OCR huggingface dataset:
While, to the best of our knowledge, Bipartite Ranking has not yet been considered under active learning, there are several related settings. Firstly, it is important to note, that as we assume the posterior  $\eta$  is piecewise constant on the grid of size  $K$ , we can view our problems as a  $K$  armed bandit. In the case where  $K = 2$  the bipartite ranking problem becomes akin to best arm identification (BAI) for the two armed bandit, also known as A/B-Testing. In BAI for the two armed bandit, the learner sequentially draws samples from two distributions  $\nu_{A}, \nu_{B}$  with respective means  $\mu_{A}, \mu_{B}$ . Their objective is then carry out the hypothesis test,  $\mathcal{H}_0 := (\mu_A \leq \mu_B)$ ,  $\mathcal{H}_1 := (\mu_a > \mu_B)$  in as few samples as possible. A/B Testing is considered in an active, fix confidence regime, in Kaufmann and Kalyanakrishnan (2013), wherein they prove a lower bound on the expected sampling time of any  $\mathrm{PAC}(\varepsilon, \delta)$  as of the order  $\frac{\log(1 / \delta)}{\mathrm{kl}(\mu_A, \mu_B)}$ . Note that, in the case where  $K = 2$  active-rank matches said lower bound up to logarithmic terms. In the fixed confidence regime, the BAI problem has also been generalised for larger  $K > 2$  - see Garivier and Kaufmann (2016), Jamieson and Talwalkar (2016) along with the TopM problem, where the learner must output the  $M$  best arms - see Kaufmann et al. (2014), Kalyanakrishnan et al. (2012), however, for  $K > 2$  both BAI and TopM problems are no longer comparable to our setting.
Aside from BAI and the TopM problem, there are several other settings in active learning that, while not directly comparable to our own, are worth mentioning. The first is active clustering. Several works have considered clustering in an online framework, see Choromanska and Monteleoni (2012), Liberty et al. (2016), Cohen-Addad et al. (2021) and Khaleghi et al. (2012). In the above works new observations from certain arms become available to the learner at each time step, however the learner does not actively choose which arms to pull and therefore the flavour of the above literature is very different to our setting. Much closer is the work of Yang et al. (2022) in which the authors consider a active clustering problem, represented as  $K$  armed  $d$  dimensional bandit, where the arms are split into  $M$  clusters. They work in a  $\mathrm{PAC}(\delta)$  setup where their goal is to recover the entire clustering of the arms, with probability greater than  $1 - \delta$  in as few samples as possible. Comparing to our setting, if one is to view a section of the grid on which  $\eta$  is constant as a single cluster, by retrieving the clustering of the arms one can then easily do ranking. Their results differ to our own in several key ways though. Firstly their algorithm takes the number of clusters  $M$  as a parameter, this highlights the main difference between their setting and our own. In the Bipartite ranking problem, assuming  $\varepsilon$  is not very small, one does not have to recover exactly all the clusters to ensure regret under the  $d_{\infty}$  norm is less than  $\varepsilon$ . Therefore we do not need to know the number of clusters and our algorithm must be able to exploit larger  $\varepsilon$  to achieve smaller stopping times. The second key difference is that the results of Yang et al. (2022) are hold only in the asymptotics, that is as  $\delta \rightarrow 0$ . Their algorithm employs a forced exploration phase, which ensures each arm is pulled at least a sub linear rate. Essentially, this means that in such an asymptotic setting, the means of the arms are known to the learner, which naturally drastically changes the nature of their results. Extension to bounds for fixed  $\delta > 0$  would be none trivial, noted as potential future work in Yang et al. (2022), and essential if one were to compare to our confidence setting.
Also of note is active multi class classification. In Krishnamurthy et al. (2017) they consider a cost sensitive classification problem, where the learner receives input examples  $x \in \mathcal{X}$  and cost vectors  $c \in \mathbb{R}^K$ , where  $c(y)$  is the cost of predicting label  $y$  on  $x$ . For each input example received the learner is able to query a subset of labels. The objective is to then train a classifier with minimal expected loss in as few queries as possible. The results of Krishnamurthy et al. (2017) cannot be directly compared to our own, as in our setting there is no such thing as the cost of a classifier at a given point  $x \in \mathcal{X}$ , as the cost miss ranking a section of [0, 1] is dependent on our ranking of the entire feature space.
3 Our results
3.1 The active-rank Algorithm
Our algorithm active-rank maintains an active set of grid points across several rounds. At the beginning of each round active-rank draws a sample, uniformly, from all points of the grid in the active set and at the end of each round, eliminates points from the active set based on a specific criterion. We track the empirical mean of all samples drawn from the  $i$ th point of the grid,  $[G_i, G_{i+1})$  up to round  $t$  as  $\hat{\mu}_i^t$ . At the beginning of each round  $t$ , for each grid point  $i \in [K]$ , we will maintain an upper and lower confidence bound, on  $\mu_i$ , which we term the UCB and LCB index respectively. At time  $t$ , for each grid point  $i \in [K]$  and exploration parameter  $\beta(t, \delta): \mathbb{N} \times [0,1] \to \mathbb{R}_+$ , remaining in the active set, we then define the LCB index,
$$
\operatorname {L C B} (t, i) := \min  \left\{q \in \left[ 0, \hat {\mu} _ {i} ^ {t} \right]: \mathrm {k l} \left(\hat {\mu} _ {i} ^ {t}, q\right) \leq \frac {\beta (t , \delta)}{t} \right\}, \tag {3}
$$
and the UCB index,
$$
\operatorname {U C B} (t, i) := \max  \left\{q \in \left[ \hat {\mu} _ {i} ^ {t}, 1 \right]: \operatorname {k l} \left(\hat {\mu} _ {i} ^ {t}, q\right) \leq \frac {\beta (t , \delta)}{t} \right\}. \tag {4}
$$
Let  $S_{t}$  denote the active set at the beginning of round  $t$ , via careful choice of exploration parameter the following Lemma holds, the proof of which can be found in Section A of the supplementary material.
Lemma 3.1. We have that, the event
$$
\mathcal {E} = \bigcap_ {t \in \mathbb {N}} \bigcap_ {i \in [ S _ {t} ]} \left\{\mu_ {k} \in \left[ \operatorname {L C B} (t, i), \operatorname {U C B} (t, i) \right] \right\},
$$
occurs with probability greater than  $1 - \delta$ .
For  $i \in [K]$ , time  $t$  and  $z \in \mathbb{R}$  define,  $U_{i,t}(z) := \{j \in S_t : j \neq i, |\widehat{\mu}_i^t - \widehat{\mu}_j^t| \leq z\}$ . Following Lemma 2.2, our intuition would be to then remove a point  $i$  from the active set if  $\left|U_{i,t}(\mathrm{UCB}(i,t) - \mathrm{LCB}(i,t))\right| \leq \frac{cK\varepsilon p(1 - \widehat{\mu}_{i,t})}{|\mathrm{UCB}(i,t) - \mathrm{LCB}(i,t)|}$ , for some well chosen constant  $c > 0$ . However, due to the technical difficulty of the proof, we make the following concession. For  $t > 0$ , let  $S_t$  be the list  $S$  at time  $t$ . At time  $t$  let  $\Delta_{(t)} = \max_{i \in S_t} (\mathrm{UCB}(t,i) - \mathrm{LCB}(t,i))$ . Furthermore, at time  $t$  define the set of grid points,
$$
\mathcal {Q} _ {t} := \left\{i \in [ K ]: \Delta_ {(t)} \leq \frac {1}{2 4} \left(\frac {K \varepsilon \widehat {p} _ {t}}{\left| U _ {i , t} (6 \Delta_ {(t)}) \right|} \wedge 1\right) \left(1 - \widehat {\mu} _ {i} ^ {t}\right) \right\}. \tag {5}
$$
If a point exists in  $\mathcal{Q}_t$ , we remove it from the active set. Note that active-rank does not take the average of the posterior  $p$  as a parameter. Instead we show it is possible to use an estimate  $\widehat{p}_t$  which updates round by round.
Elimination algorithms such as active-rank have seen wide usage in the literature for BAI, see In Paulson (1964),Mannor and Tsitsiklis (2004), Even-Dar et al. (2002) and Even-Dar et al. (2006). However, closer to our work is the Racing algorithm Kaufmann and Kalyanakrishnan (2013), designed for the TopM problem, where, as in our approach, the confidence bounds used are based on the  $\mathrm{kl}$  divergence as opposed to Hoeffdings. Their elimination criterion, however, differs considerably to our own. For simplicity let us consider the Top1 problem, that is BAI - the following arguments can be extended in the case of TopM. The racing algorithm of Kaufmann and Kalyanakrishnan (2013) eliminates an arm  $i\in [K]$  from the active set, at time  $t$ , when, the positive gap the lower confidence bound around the highest empirical mean and the upper confidence bound at point  $i$  is greater than  $\varepsilon$ . However, due to the global nature of the ranking problem, in our setting, the decision to remove a point from the active set is not made based on the distance to another single point. We rather consider a condition on the local smoothness of the posterior around the point  $i$ . An additional difficulty that arises here is that the local smoothness around a point can potentially depend upon points no longer in the active set and once a point is no longer in the active set, we essentially have no control of the width of its confidence interval.
3.2 Theoretical bounds
Algorithm 1 active-rank
Initialise:  $S = [K],t = 1$    
repeat for  $i\in S$  do Sample a point drawn uniformly from  $[G_i,G_{i + 1})$  end for Let  $p_t$  be a point drawn uniformly from [0,1] and update  $\hat{p}_t = ((t - 1)\hat{p}_{t - 1} + p_t) / t$  for  $i\in S$  do if  $i\in \mathcal{Q}_t,\Delta_{(t)}\leq \hat{p}_t / 4$  then  $S = S\setminus \{i\}$  end if end for  $t = t + 1$    
until  $S = \emptyset$    
Let  $\hat{\sigma}_t\in \mathfrak{G}_K$  be the permutation sorting the list  $(\widehat{\mu}_i^t)_{i\in [K]}$  into ascending order.   
Output:  $\hat{s} = \sum_{i\in K}i\cdot \mathbb{I}\{x\in [G_{\hat{\sigma}(i)},G_{\hat{\sigma}(i + 1)})\}$
Proving active-rank is  $\mathrm{PAC}(\varepsilon, \delta)$  and upper bounding the expected sampling time Theorem 3.2 demonstrates that our algorithm active-rank is  $\mathrm{PAC}(\varepsilon, \delta)$  and provides a problem dependent upper bound on it's expected sampling time. The proof can be found in Section A of the supplementary material. Theorem 3.2 makes no assumption on the posterior  $\eta$ , aside from it being piecewise constant on the grid of size  $K$ , i.e. Assumption 2.1. For  $i \in [K]$ , set  $H_i^{(2)} = \max_{j \in [K]} \left( \frac{1}{\mathrm{kl}(\mu_j, \mu_j + \Delta_i / 8)} \vee \frac{1}{\mathrm{kl}(\mu_j, \mu_j - \Delta_i / 8)} \right)$ .
Theorem 3.2. For  $\varepsilon, \delta > 0, \gamma > 480 / \log(K)$ , with  $\beta(t, \delta) = c_{\gamma} \log(t^2 K^2 / \delta)$  where  $c_{\gamma}$  is a constant depending only on  $\gamma$ , on all problems  $\nu \in \mathcal{B}$ , on execution of active-rank, with output  $\hat{s}$ , we have that,
$$
d _ {\infty} (\hat {s}, \eta) \leq \varepsilon ,
$$
with probability greater than  $1 - \delta$ . Furthermore, the expected stopping time of active-rank is upper bounded by the following,
$$
c _ {\gamma} ^ {\prime} \sum_ {i \in [ K ]} H _ {i} ^ {(2)} \log \left(c _ {\gamma} ^ {\prime \prime} H _ {i} ^ {(2)} K / \delta\right),
$$
where  $c_{\gamma}^{\prime}, c_{\gamma}^{\prime \prime}$ , are constants depending only on  $\gamma$ .
Lower bound Theorem 3.3 provides a problem dependent lower bound on the expected sampling time of any  $\mathrm{PAC}(\varepsilon, \delta)$  policy. The proof of Theorem 3.3 can be found in Section B of the supplementary material.
Theorem 3.3. Let  $\varepsilon \in [0,1/4)$ ,  $0 < \delta < 1 - \exp(-1/8)$  and  $\nu \in \mathcal{B}$  such that  $K\varepsilon p < 1/8$ . For any  $PAC(\varepsilon, \delta)$  policy  $\pi$ , there exists a problem  $\bar{\nu} \in \mathcal{B}$  such that, for all  $i \in [K]$ ,  $\bar{\Delta}_i \geq \Delta_i / 2$  where  $\bar{\Delta}_i$  is the gap of the  $i$ th grid point on problem  $\bar{\nu}$ , where the expected stopping time of policy  $\pi$  on problem  $\bar{\nu}$  is bounded as follows,
$$
\mathbb {E} _ {\bar {\nu}, \pi} \left[ \tau_ {\pi} ^ {\bar {\nu}} \right] \geq c ^ {\prime} \sum_ {i \in [ K ]} \bar {H} _ {i} ^ {(1)},
$$
where  $c' > 0$  is an absolute constant and  $\bar{H}_i^{(1)}$  is the complexity of point  $i$  on problem  $\bar{\nu}$ . Furthermore we have that,
$$
\sum_ {i \in [ K ]} \bar {H} _ {i} ^ {(1)} \geq c \sum_ {i \in [ K ]} H _ {i} ^ {(1)},
$$
where  $c > 0$  is an absolute constant.
The proof of Theorem 3.3 follows from a novel application of a Fano type inequality on a well chosen set of problems.
Gap between upper and lower bound There are essentially two components in the gap between the bounds of Theorems, 3.3 and 3.2. The first is the additional logarithmic dependency upon  $K$  present in our upper bound. Despite being logarithmic this dependence is potentially significant, as in practical situations, the size of grid needed, for the assumption that the posterior  $\eta$  is piecewise constant, may be very large. The second component, in the gap between upper and lower bounds is the difference in the  $H_{i}^{(1)}$  and  $H_{i}^{(2)}$  terms. The reason  $H_{i}^{(2)}$  appears in Theorem 3.2 is that, the decision to remove a point  $i\in [K]$  from our active set is made based on the minimum width of confidence interval across the entire grid,  $\Delta_{(t)}$  as opposed to the local width at  $i$ ,  $\mathrm{UCB}(i,t) - \mathrm{LCB}(i,t)$ , see Equation (5). As we are dealing with Bernoulli distributions and kl divergence based confidence bounds, for a fixed number of samples, points close to zero or one will have tighter confidence bounds
and thus may be sampled more than is necessary. If one were to assume that the posterior  $\eta$  exists solely in the interval  $[\gamma, 1 - \gamma]$  for some  $\gamma > 0$ , then for all  $i \in [K]$ ,  $H_i^{(2)}$  and  $H_i^{(1)}$  will be with a constant factor of each other, with that constant depending on  $\gamma$ .
In authors opinion, both the logarithmic dependency on  $K$  and usage of  $\Delta_{(t)}$  may be removed. However, this would require several non trivial modifications to the proof of Theorem 3.2, see the discussion in Section A of the supplementary material for details. As, to the best of our knowledge, this is the first work to consider the bipartite ranking problem in an active learning setting, we present Theorem 3.2 as it stands and leave the aforementioned improvements for future works.
In this section we discuss practical cases based on synthetic data. For all experiments  $\delta$  is fixed at 0.01 and the constants used are smaller than their theoretical counterparts, which are typically overestimated, furthermore  $\widehat{p}$  is calculated with all previous samples. As represented in Figure 3, each cell  $i$  is assigned a level value  $\mu_{i}$  so that  $\eta$  follows the Assumption 2.1. Without loss of generality, we assume that  $\eta$  can be described as an increasing family  $(\mu_i)_{i\in [K]}$ . Our study scenarios are then as follows:
- Scenario 1:  $(\mu_i)_{i\in [K]} = (0,0.28,0.3,0.38)$  and  $K = 16$
- Scenario 2:  $(\mu_i)_{i\in [K]} = (0.8((i - 1) / K)^4 +0.1)_{i\in [K]}$  and  $K = 64$
- Scenario 3:  $(\mu_i)_{i \in [K]}$  is sub-sampled (with replacement) of  $((i - 1)/K)^4)_{i \in [100]}$  and  $K = 64$
- Scenario 4:  $\mu_{i} = 0.8((i - 1) / K) + 0.1\forall i\in [K]\backslash \{7,8\}$  with  $\mu_7 = 0.8(6 / K) + 0.3$ $\mu_{8} = 0.8(7 / K) - 0.1$  and  $K = 16$
The objective of these scenarios is to evaluate the capacity of the algorithm on different cases. As shown in Figure 3, scenarios 1 and 3 will have variable jumps and cell sizes.
Competing algorithms To our knowledge there is no algorithm dealing with the active learning for bipartite ranking problem, we thus compare to the following naive approaches. Passive rank: each new point  $a_{t}$  is drawn uniformly on [0, 1]. Naive rank: each new point  $a_{t}$  is sampled in  $P_{i_{t}}$  s.t.  $i_{t} := \arg \max_{i \in [K]} \mathrm{UCB}(t, i) - \mathrm{LCB}(t, i)$ , this algorithm reduces the bias in an undifferentiated way without considering the problem as global (requiring peer-to-peer comparison). Active classification: for this algorithm we consider binary classification with threshold 0.5. The set of active cells  $S$ , as defined in Algorithm 1 is then  $S_{t} = \{i \in [K]; 0.5 \in [\hat{\mu}_{i}^{t} - \mathrm{LCB}(t, i); \hat{\mu}_{i}^{t} + \mathrm{UCB}(t, i)]\} \cup \left\{i; \operatorname*{argmin}_{i \in [K]} (|0.5 - \hat{\mu}_{i}^{t}|)\right\}$ . As the competing algorithms do not output a stopping time, for a single  $\eta$ , algorithm active-rank is run across several values of  $\varepsilon$ , following a geometric sequence of common ratio 0.99 and initial value 1. The values of  $\varepsilon$  then plotted against the respective stopping times of active-rank.
Figure 3: Different scenarios chosen for the experiments
Interpretation of results On scenario 1, the simplest case, and to an extent scenario 2, active-rank and passive suffer near identical regret for larger sample sizes. On all other scenarios active-rank outperforms all competitors. The fact that active-rank does not reach extremely small values of regret suggests that analysis for much higher sample size may be interesting, however this creates issues in computation time, the same goes for larger  $K$ . Also, the uniform
Figure 4: Regret estimated by Monte Carlo for 100 realizations of each algorithm, corresponding to the scenarios of the figure 3 respectively.
approach still performs relatively well, and appears difficult to fool. Some more work may be needed to find a setting in which uniform sampling suffers considerably.
To the best of our knowledge we have developed the first rigorous framework for the active bipartite ranking problem and our algorithm, active-rank, is the first to tackle said problem. Our upper bound on performance of active-rank matches our lower bound up to logarithmic terms, in the case where the posterior  $\eta$  is not very close to 0 or 1 at any point. As well as theoretical guarantees we have demonstrated good practical performance of active-rank, on synthetic data, in various settings. We conclude with some perspectives for future research.
An obvious path for future research, is to replace the Assumption 2.1 with a smoothness assumption on the posterior  $\eta$ , e.g. a Hölder condition. The setting would then be equivalent to a continuous armed bandit as opposed to a finite armed bandit. Assuming the learner has knowledge of the Hölder coefficient, a standard approach in continuous armed bandits is to first discretise and then apply classic techniques from finite armed bandits, carefully choosing the discretisation level to balance the discretisation error and classical regret. It is of our opinion that such an approach would not be sufficient in our case. We conjecture that to achieve optimal or near optimal performance the learner must vary the level of discretisation across the feature space, based on the flatness of the posterior function  $\eta$  and placement on the ROC curve. Also, under such a Hölder condition, the extension to higher dimensions would no longer be immediate.
Agarwal, S., Graepel, T., Herbrich, R., Har-Peled, S., and Roth, D. (2005). Generalization bounds for the area under the ROC curve. J. Mach. Learn. Res., 6:393-425.
Choromanska, A. and Monteleoni, C. (2012). Online clustering with experts. In Artificial Intelligence and Statistics, pages 227-235. PMLR.
Clémenton, S. and Robbiano, S. (2011). Minimax learning rates for bipartite ranking and plug-in rules. In Proceedings of ICML, number 1.
Clémenton, S. and Vayatis, N. (2008). Overlaying classifiers: a practical approach for optimal scoring. Constructive Approximation, ...
Clémençon, S. and Vayatis, N. (2009). Tree-based ranking methods. IEEE Transactions on Information Theory, 55(9):4316-4336.
Clémençon, S., Lugosi, G., and Vayatis, N. (2008). Ranking and Empirical Minimization of U-Statistics. The Annals of Statistics, 36(2):844-874.
Clemencon, S. and Vayatis, N. (2009). On partitioning rules for bipartite ranking. In van Dyk, D. and Welling, M., editors, Proceedings of the Twelfth International Conference on Artificial Intelligence and Statistics, volume 5 of Proceedings of Machine Learning Research, pages 97-104, Hilton Clearwater Beach Resort, Clearwater Beach, Florida USA. PMLR.
Cohen-Addad, V., Guedj, B., Kanade, V., and Rom, G. (2021). Online k-means clustering. In International Conference on Artificial Intelligence and Statistics, pages 1126-1134. PMLR.
Even-Dar, E., Mannor, S., and Mansour, Y. (2002). Pac bounds for multi-armed bandit and markov decision processes. In International Conference on Computational Learning Theory, pages 255-270. Springer.
Even-Dar, E., Mannor, S., and Mansour, Y. (2006). Action elimination and stopping conditions for the multi-armed bandit and reinforcement learning problems. Journal of machine learning research, 7(Jun):1079-1105.
Garivier, A. and Kaufmann, E. (2016). Optimal best arm identification with fixed confidence. In Conference on Learning Theory, pages 998-1027. PMLR.
Jamieson, K. and Talwalkar, A. (2016). Non-stochastic best arm identification and hyperparameter optimization. In Artificial intelligence and statistics, pages 240-248. PMLR.
Kalyanakrishnan, S., Tewari, A., Auer, P., and Stone, P. (2012). Pac subset selection in stochastic multi-armed bandits. In ICML, volume 12, pages 655-662.
Kaufmann, E., Cappé, O., and Garivier, A. (2014). On the complexity of a/b testing. In Conference on Learning Theory, pages 461-481. PMLR.
Kaufmann, E. and Kalyanakrishnan, S. (2013). Information complexity in bandit subset selection. In Conference on Learning Theory, pages 228-251. PMLR.
Khaleghi, A., Ryabko, D., Mary, J., and Preux, P. (2012). Online clustering of processes. In Artificial Intelligence and Statistics, pages 601-609. PMLR.
Krishnamurthy, A., Agarwal, A., Huang, T., III, H. D., and Langford, J. (2017). Active learning for cost-sensitive classification. CoRR, abs/1703.01014.
Liberty, E., Sriharsha, R., and Sviridenko, M. (2016). An algorithm for online k-means clustering. In 2016 Proceedings of the eighteenth workshop on algorithm engineering and experiments (ALENEX), pages 81-89. SIAM.
Mannor, S. and Tsitsiklis, J. N. (2004). The sample complexity of exploration in the multi-armed bandit problem. Journal of Machine Learning Research, 5(Jun):623-648.
Menon, A. and Williamsson, R. (2016). Bipartite ranking: A risk theoretic perspective. Journal of Machine Learning Research, 7:1-102.
Paulson, E. (1964). A sequential procedure for selecting the population with the largest mean from  $k$  normal populations. The Annals of Mathematical Statistics, 35(1):174-180.
Yang, J., Zhong, Z., and Tan, V. Y. (2022). Optimal clustering with bandit feedback. arXiv preprint arXiv:2202.04294.
A Proof of theorem 3.2
Before the proof of Theorem 3.2 we prove some initial lemmas, the first of which is Lemma 3.1, which lower bounds the probability of our favourable event  $\mathcal{E}$  by  $1 - \delta$ . We prove a slightly extended version of Lemma 3.1. At time  $t$ , define the LCB index of  $\widehat{p}$  as,
$$
\operatorname {L C B} (t, 0) := \min  \left\{q \in [ 0, \widehat {p} _ {t} ]: \operatorname {k l} (\widehat {p} _ {t}, q) \leq \frac {\beta (t , \delta)}{t} \right\}, \tag {6}
$$
$$
\operatorname {U C B} (t, 0) := \max  \left\{q \in [ \widehat {p} _ {t}, 1 ]: \operatorname {k l} (\widehat {p} _ {t}, q) \leq \frac {\beta (t , \delta)}{t} \right\}. \tag {7}
$$
Note that as  $\widehat{p_t} \in [\min(\mu_i^t), \max(\mu_i^t)]$ , we have that,
$$
\mathrm {U C B} (0, t) - \mathrm {L C B} (0, t) \leq \Delta_ {(t)}.
$$
The extended version of Lemma 3.1 is then as follows,
Lemma A.1. We have that, the event
$$
\mathcal {E} = \bigcap_ {t \in \mathbb {N}} \bigcap_ {i \in [ S _ {t} ] \cup \{0 \}} \left\{\mu_ {k} \in [ \mathrm {L C B} (t, i), \mathrm {U C B} (t, i) ] \right\},
$$
Proof. Via Chernoff's inequality, for  $i \in [K] \cup \{0\}$ , at time  $t$  we have that,
$$
\mathbb {P} (\operatorname {L C B} (t, k) \geq \mu_ {k}) \leq \exp (- \beta (t, \delta)),
$$
$$
\mathbb {P} (\mathrm {U C B} (t, k) \leq \mu_ {k}) \leq \exp (- \beta (t, \delta)).
$$