SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Marc Jourdan
marc.jourdan@inria.fr
Rémy Degenne  
remy.degenne@inria.fr
<sup>1</sup> Univ. Lille, CNRS, Inria, Centrale Lille, UMR 9189-CRISTAL, F-59000 Lille, France
A Top Two sampling rule for bandit identification is a method which selects the next arm to sample from among two candidate arms, a leader and a challenger. Due to their simplicity and good empirical performance, they have received increased attention in recent years. However, for fixed-confidence best arm identification, theoretical guarantees for Top Two methods have only been obtained in the asymptotic regime, when the error level vanishes. In this paper, we derive the first non-asymptotic upper bound on the expected sample complexity of a Top Two algorithm, which holds for any error level. Our analysis highlights sufficient properties for a regret minimization algorithm to be used as leader. These properties are satisfied by the UCB algorithm, and our proposed UCB-based Top Two algorithm simultaneously enjoys non-asymptotic guarantees and competitive empirical performance.
Faced with a collection of items ("arms") with unknown probability distributions, a question that arises in many applications is to find the distribution with the largest mean, which is referred to as the best arm. Different approaches have been considered depending on the data collection process. Sequential hypothesis testing [10, 37] encompasses situations where there is no control on the collected samples. Experimental design [7, 35] aims at choosing the data collection scheme a priori. In the multi-armed bandit [4, 20] and the ranking and selection [18] literature, an algorithm chooses sequentially the distribution from which it will collect an additional sample based on past data.
In order to have theoretical guarantees for this identification problem, one should adopt a statistical model on the underlying distributions. While parametric models are reasonable for applications such as A/B testing [28], they are unrealistic in other fields such as agriculture [21]. Despite the restricted scope of its applications, studying the identification task for Gaussian distributions is a natural first step. Hopefully the insights gained will then be generalized to wider classes of distributions.
In the fixed confidence identification problem, an algorithm aims at identifying the best arm with an error of at most  $\delta \in (0,1)$  while using as few samples as possible. Since each sample has a cost, those algorithms should provide an upper bound on the expected number of samples used by the algorithm before stopping. For those guarantees to be useful in practice, they should hold for any  $\delta$ , which is referred to as the non-asymptotic (or moderate) regime. In contrast, the asymptotic regime considers vanishing error level, i.e.  $\delta \to 0$ . For Gaussian distributions, Top Two algorithms [36, 39]
have only been studied in the asymptotic regime. We show the first non-asymptotic guaranty for a Top Two algorithm holding for any instance with a unique best arm.
1.1 Setting and related work
A Gaussian bandit problem is described by  $K$  arms whose probability distributions belong to the set  $\mathcal{D}$  of Gaussian distributions with known variance  $\sigma^2$ . By rescaling, we assume  $\sigma_i^2 = 1$  for all  $i \in [K]$ . Since an element of  $\mathcal{D}$  is uniquely characterized by its mean, the vector  $\mu \in \mathbb{R}^K$  refers to a  $K$ -arms Gaussian bandit. Let  $\triangle_K \subseteq \mathbb{R}^K$  be the  $(K-1)$ -dimensional simplex.
A best arm identification (BAI) algorithm aims at identifying an arm with highest mean parameter, i.e. an arm belonging to the set  $i^{\star}(\mu) = \arg \max_{i\in [K]}\mu_{i}$ . At each time  $n\in \mathbb{N}$ , the algorithm (1) chooses an arm  $I_{n}$  based on previous observations, (2) observes a sample  $X_{n,I_n}\sim \mathcal{N}(\mu_{I_n},1)$ , and (3) decides whether it should stop and return an arm  $\hat{i}_n$  or continue sampling. We consider the fixed confidence identification setting, in which the probability of error of an algorithm is required to be less than a given  $\delta \in (0,1)$  on all instances  $\mu$ . The sample complexity of an algorithm corresponds to its stopping time  $\tau_{\delta}$ , which counts the number of rounds before termination. An algorithm is said to be  $\delta$ -correct on  $\mathcal{D}^K$  if  $\mathbb{P}_{\mu}(\tau_{\delta} < +\infty, \hat{i}_{\tau_{\delta}} \notin i^{\star}(\mu)) \leq \delta$  for all  $\mu \in \mathcal{D}^K$  [14]. We aim at designing  $\delta$ -correct algorithms minimizing  $\mathbb{E}[\tau_{\delta}]$ .
As done in all the literature on fixed-confidence BAI, we assume that there is a unique best arm and we denote it by  $i^{\star}(\mu)$  or  $i^{\star}$  when  $\mu$  is clear from the context. To ensure  $\delta$ -correctness on  $\mathcal{D}^{K}$ , an algorithm has to be able to distinguish the unknown  $\mu$  from any instance having a different best arm, hence it needs to estimate the gaps between arms. Lemma 1.1 gives a lower bound on the expected sample complexity which is known to be tight in the asymptotic regime, i.e. when  $\delta$  goes to zero.
Lemma 1.1 ([17]). An algorithm which is  $\delta$ -correct on all problems in  $\mathcal{D}^K$  satisfies that for all  $\mu \in \mathbb{R}^K$ ,  $\mathbb{E}_{\mu}[\tau_{\delta}] \geq T^{\star}(\mu) \log(1/(2.4\delta))$  where  $T^{\star}(\mu) = \min_{\beta \in (0,1)} T_{\beta}^{\star}(\mu)$  and, for all  $\beta \in (0,1)$ ,
$$
T _ {\beta} ^ {\star} (\mu) ^ {- 1} := \max  _ {w \in \triangle_ {K}: w _ {i ^ {\star} (\mu)} = \beta} \min  _ {i \neq i ^ {\star} (\mu)} \frac {\left(\mu_ {i ^ {\star} (\mu)} - \mu_ {i}\right) ^ {2}}{2 \left(1 / \beta + 1 / w _ {i}\right)}.
$$
When considering the sub-class of algorithms allocating a fraction  $\beta$  of their sample to the best arm, we obtain a lower bound as in Lemma 1.1 with  $T_{\beta}^{\star}(\mu)$  instead of  $T^{\star}(\mu)$ . An algorithm is said to be asymptotically optimal (resp.  $\beta$ -optimal) if its sample complexity matches that lower bound asymptotically, that is if  $\lim \sup_{\delta \to 0} \mathbb{E}_{\mu}[\tau_{\delta}] / \log(1/\delta) \leq T^{\star}(\mu)$  (resp.  $T_{\beta}^{\star}(\mu)$ ). [38] showed the worst-case inequality  $T_{1/2}^{\star}(\mu) \leq 2T^{\star}(\mu)$  for any single-parameter exponential families. Therefore, the expected sample complexity of an asymptotically  $\beta$ -optimal algorithm with  $\beta = 1/2$  is at worst twice higher than that of any asymptotically optimal algorithm. Leveraging the symmetry of Gaussian distributions, a tighter worst-case inequality can be derived (Lemma C.6). The allocations  $w^{\star}(\mu)$  and  $w_{\beta}^{\star}(\mu)$  realizing  $T^{\star}(\mu)$  and  $T_{\beta}^{\star}(\mu)$  are known to be unique, and satisfy  $\min_{i \in [K]} \min \{w^{\star}(\mu)_i, w_{\beta}^{\star}(\mu)_i\} > 0$ . [6] showed that  $2 \leq w^{\star}(\mu)_{i^{\star}}^{-1} \leq \sqrt{K - 1} + 1$  for Gaussian distributions (Lemma C.4).
Related work The first BAI algorithms were introduced and studied under the assumption that the observation have bounded support, with a known upper bound [15, 24, 16, 19]. The sample complexity bounds proved for these algorithms scale as the sum of squared inverse gap, i.e.  $H(\mu) \coloneqq 2\Delta_{\mathrm{min}}^{-2} + \sum_{i \neq i^{\star}} 2(\mu_{i^{\star}} - \mu_i)^{-2}$  where  $\Delta_{\mathrm{min}} \coloneqq \min_{i \neq i^{\star}} (\mu_{i^{\star}} - \mu_i)$ , which satisfies  $H(\mu) \leq T^{\star}(\mu) \leq 2H(\mu)$  [17]. Following their work, a rich literature designed asymptotically optimal algorithms in the fixed-confidence setting for parametric distributions, such as single-parameter exponential families, and non-parametric distributions such as bounded ones. Those algorithms build on two main ideas. The Tracking approach computes at each round the optimal allocation for the empirical estimator, and then tracks it [17]. To achieve lower computational cost, Game-based algorithms [12] view  $T^{\star}(\mu)^{-1}$  as a min-max game between the learner and the nature, and design saddle-point algorithms to solve it sequentially.
Top Two algorithms arose as an identification strategy based on the Thompson Sampling algorithm for regret minimization [41]: [38] introduced Top Two Probability Sampling (TTPS) and Top Two Thompson Sampling (TTTS). Adopting a Bayesian viewpoint, Russo studied the convergence rate of the posterior probability that  $i^{\star}$  is not the best arm, under some conditions on the prior. For Gaussian bandits, other Bayesian Top Two algorithms with frequentist components have been shown to be
asymptotically  $\beta$ -optimal: Top Two Expected Improvement (TTEI, [36]) and Top Two Transportation Cost (T3C, [39]). [21] introduces fully frequentist Top Two algorithms. Their analysis proves asymptotic  $\beta$ -optimality for several Top Two algorithms and distribution classes, beyond Gaussian. [34] provides guarantees for single-parameter exponential families, at the price of adding forced exploration. [44] proposes an algorithm to tackle the top- $k$  identification problem and introduces information-directed selection (IDS) to choose  $\beta$  in an adaptive manner, which differs from the one proposed in [33]. In addition to their success in the fixed-confidence setting, Top Two algorithms have also been studied for fixed-budget problems [2], in which guarantees on the error probability should be given after  $T$  samples. While existing Top Two sampling rules differ by how they choose the leader and the challenger, they all sample the leader with probability  $\beta$ . By design, Top Two algorithms with a fixed  $\beta$  can reach  $\beta$ -optimality at best, and cannot be optimal on all instances  $\mu$ .
Shortcomings of the asymptotic regime While the literature provides a detailed understanding of the asymptotic regime, many interesting questions are unanswered in the non-asymptotic regime. Recent works [9, 40, 32, 31] have shown that the sample complexity is affected by strong moderate confidence terms (independent of  $\delta$ ). The analysis of [21] applies to their  $\beta$ -EB-TC algorithm whose empirical stopping times is order of magnitude larger than its competitors for  $\delta = 0.01$ . Since the proof of asymptotic  $\beta$ -optimality hides design flaws, non-asymptotic guarantees should be derived to understand which Top Two algorithms will perform well in practice for any reasonable choice of  $\delta$ .
Our main contribution is to propose the first non-asymptotic analysis of Top Two algorithms. We identify sufficient properties of the leader (seen as a regret-minimization algorithm) for it to hold. This solves two open problems: obtaining an upper bound which (1) is non-asymptotic (Theorem 2.4 holds for any  $\delta$ ) and (2) holds for all instances having a unique best arm (i.e. sub-optimal arms can have the same mean, which was not allowed in the analyzes of existing Top Two algorithms). As a consequence, we propose the TTUCB (Top Two UCB) algorithm which builds on the UCB algorithm.
By using tracking instead of sampling to choose between the leader and the challenger, TTUCB is the first Top Two algorithm which is asymptotically  $\beta$ -optimal (Theorem 2.3) and has non-asymptotic guarantees (Theorem 2.4). Our experiments reveal that TTUCB performs on par with existing Top Two algorithms, which are only proven to be asymptotically  $\beta$ -optimal, even for large sets of arms. Numerically, we show that considering adaptive proportions compared to a fixed  $\beta = 1/2$  yields a significant speed-up on hard instances, and to a moderate improvement on random instances.
2 UCB-based Top Two algorithm
We propose a fully deterministic Top Two algorithm based on UCB [5], named TTUCB and detailed in Algorithm 1. We prove a non-asymptotic upper bound on the expected sample complexity holding for any instance having a unique best arm.
Stopping and recommendation rules The  $\sigma$ -algebra  $\mathcal{F}_n := \sigma(\{I_t, X_{t,I_t}\}_{t \in [n-1]})$  encompasses all the information available to the agent before time  $n$ . Let  $N_{n,i} := \sum_{t \in [n-1]} \mathbb{1}(I_t = i)$  be the number of pulls of arm  $i$  before time  $n$ , and its empirical mean by  $\mu_{n,i} := \frac{1}{N_{n,i}} \sum_{t \in [n-1]} X_{t,I_t} \mathbb{1}(I_t = i)$ .
The algorithm stops as soon as the generalized likelihood ratio exceeds a threshold  $c(n - 1, \delta)$ , i.e.
$$
\min  _ {i \neq \hat {i} _ {n}} \frac {\mu_ {n , \hat {i} _ {n}} - \mu_ {n , i}}{\sqrt {1 / N _ {n , \hat {i} _ {n}} + 1 / N _ {n , i}}} \geq \sqrt {2 c (n - 1 , \delta)}, \tag {1}
$$
where we recommend  $\hat{u}_n = \arg \max_{i\in [K]}\mu_{n,i}$  at time  $n$ . Lemma 2.1 provides an explicit threshold ensuring  $\delta$ -correctness, which relies on concentration inequalities derived in [27].
Lemma 2.1. Let  $\mathcal{C}_G$  defined in (16) s.t.  $\mathcal{C}_G(x)\approx x + \log (x)$ . Given any sampling rule, taking
$$
c (n, \delta) = 2 \mathcal {C} _ {G} (\log ((K - 1) / \delta) / 2) + 4 \log (4 + \log (n / 2)) \tag {2}
$$
in the stopping rule (1) ensures  $\delta$ -correct for Gaussian distributions.
Algorithm 1 TTUCB
Input:  $(\beta, \delta) \in (0,1)^2$ , threshold  $c: \mathbb{N} \times (0,1) \to \mathbb{R}^+$  and function  $g: \mathbb{N} \to \mathbb{R}^+$ .
Pull once each arm  $i\in [K]$
for  $n > K$  do
Set  $\hat{\iota}_n = \arg \max_{i\in [K]}\mu_{n,i}$
If  $\min_{i\neq \hat{\imath}_n}\frac{\mu_{n,i_n} - \mu_{n,i}}{\sqrt{1 / N_{n,i_n} + 1 / N_{n,i}}}\geq \sqrt{2c(n - 1,\delta)}$  then return  $\hat{\imath}_n$ , else;
Set  $B_{n}^{\mathrm{UCB}} = \arg \max_{i\in [K]}\left\{\mu_{n,i} + \sqrt{\frac{g(n)}{N_{n,i}}}\right\}$  and  $C_n^{\mathrm{TC}} = \arg \min_{i\neq B_n^{\mathrm{UCB}}}\frac{(\mu_{n,B_n^{\mathrm{UCB}}} - \mu_{n,i})_+}{\sqrt{1 / N_{n,B_n^{\mathrm{UCB}}} + 1 / N_{n,i}}};$
Observe  $X_{n,I_n}$  by pulling  $I_{n} = B_{n}^{\mathrm{UCB}}$  if  $N_{n,B_n^{\mathrm{UCB}}}^{B_n^{\mathrm{UCB}}}\leq \beta L_{n + 1,B_n^{\mathrm{UCB}}}$ , else  $I_{n} = C_{n}^{\mathrm{TC}}$ ;
Sampling rule We initialize by sampling each arms once. At time  $n > K$ , a Top Two sampling rule defines a leader  $B_{n} \in [K]$  and a challenger  $C_n \neq B_n$ , and chooses  $I_{n} = B_{n}$  or  $I_{n} = C_{n}$  based on a fixed allocation  $\beta$ . In prior work this choice was done at random, which means that the leader was sampled with probability  $\beta$ . We replace randomization by tracking, and show similar theoretical and numerical results (see Figure 4 in Appendix G.2). For fixed  $\beta$ , we recommend to use  $\beta = 1 / 2$  without prior knowledge on the unknown mean parameters (see Section 3.4 for adaptive proportions). This recommendation is supported theoretically by the fact that  $w^{\star}(\mu)_{i^{\star}} \leq 1 / 2$  (Lemma C.4) and that  $T_{1 / 2}^{\star}(\mu) / T^{\star}(\mu)$  is significantly smaller than 2 for most instances (Lemma C.6 and Figure 2).
Let  $L_{n,i} \coloneqq \sum_{t \in [n-1]} \mathbb{1}(B_t = i)$  be the number of time arms  $i$  was the leader, and  $N_{n,j}^i \coloneqq \sum_{t \in [n-1]} \mathbb{1}((B_t, I_t) = (i, j))$  be the number of pulls of arms  $j$  at rounds in which  $i$  was the leader. We use  $K$  independent tracking procedures. A tracking procedure is a deterministic method to convert a sequence of allocations over arms into a sequence of arms, which ensures that the empirical proportions are close to the averaged allocation over arms. For each leader, we track the allocation  $(\beta, 1 - \beta)$  between the leader and the challenger. Formally, we set  $I_n = B_n$  if  $N_{n,B_n}^{B_n} \leq \beta L_{n+1,B_n}$ , else  $I_n = C_n$ . Using Theorem 6 in [13] for each tracking procedure yields Lemma 2.2.
Lemma 2.2. For all  $n > K$  and all  $i \in [K]$ , we have  $-1/2 \leq N_{n,i}^i - \beta L_{n,i} \leq 1$ .
Using tracking over randomization is motivated by practical and theoretical reasons. First, in some specific applications, the practitioner might be only willing to use a deterministic algorithm. Second, in the analysis, it is easier to control deterministic counts since it removes the need for martingales arguments to bound the deviations of the samples. Therefore, tracking simplifies the non-asymptotic analysis. Third, Lemma 2.2 shows that the speed of convergence is at least  $\mathcal{O}(1 / n)$  for tracking, while we would obtain a speed of  $O(1 / \sqrt{n})$  for randomization.
At time  $n$ , the UCB leader is defined as
$$
B _ {n} ^ {\mathrm {U C B}} = \underset {i \in [ K ]} {\arg \max } \left\{\mu_ {n, i} + \sqrt {g (n) / N _ {n , i}} \right\}, \tag {3}
$$
where  $\sqrt{g(n) / N_{n,i}}$  is a bonus coping for uncertainty. Let  $\alpha > 1$  and  $s > 1$  be two concentration parameters. The choice of  $g(n)$  should ensure that we have an upper confidence bound on  $\mu_i$  holding with high probability: with probability  $1 - Kn^{-s}$ , for all  $t \in [n^{1/\alpha}, n]$  and all arms  $i \in [K]$ ,  $\mu_i \in [\mu_{t,i} \pm \sqrt{g(t) / N_{t,i}}]$ . For Gaussian observations, a function  $g$  which is sufficient for the purpose of our proof can be obtained by a union bound over time, giving  $g_u(n) = 2\alpha (1 + s) \log n$ . We can improve on  $g_u$  with mixtures of martingales, yielding  $g_m(n) = \overline{W}_{-1}(2s\alpha \log (n) + 2\log (2 + \alpha \log n) + 2)$  with  $\overline{W}_{-1}(x) = -W_{-1}(-e^{-x})$  for all  $x \geq 1$ , where  $W_{-1}$  is the negative branch of the Lambert  $W$  function, and  $\overline{W}_{-1}(x) \approx x + \log (x)$ . A UCB leader with  $g_0(n) = 0$  recovers the Empirical Best (EB) leader [21]. Choosing  $g$  is central for empirical performance and non-asymptotic guarantees, but not for asymptotic ones. The lowest  $g$  will yield better empirical performance since larger  $g$  means more conservative confidence bounds. In our experiments where  $\alpha = s = 1.2$ , we will consider  $g_m$  since  $g_m(n) \leq g_u(n)$  for  $n \geq 50$ .
Given a leader  $B_{n}$ , the TC challenger is defined as
$$
C _ {n} ^ {\mathrm {T C}} = \underset {i \neq B _ {n}} {\arg \min } \frac {\left(\mu_ {n , B _ {n}} - \mu_ {n , i}\right) _ {+}}{\sqrt {1 / N _ {n , B _ {n}} + 1 / N _ {n , i}}} ， \tag {4}
$$
where  $x_{+} = \max \{x,0\}$ . [39] introduced the TC challenger as a computationally efficient approximation of the challenger in TTTS [38], which uses re-sampling till an unlikely event occurs. Both T3C and TTTS use the TS leader which takes the best arm of a vector of realization drawn from a sampler, e.g.  $\theta_{i}\sim \mathcal{N}(\mu_{n,i},1 / N_{n,i})$  for Gaussian distributions with unit variance.
Computational cost Computing the stopping rule (1) and the UCB leader (3) can be done in  $\mathcal{O}(K)$ . At time  $n$  where  $B_{n}$  coincides with  $\hat{\imath}_n$ , computing the TC challenger (4) is done as a by-product of the computation of the stopping rule, without additional cost. When  $B_{n} \neq \hat{\imath}_{n}$ , we draw at random an arm with larger empirical mean. The per-round computational and memory cost of TTUCB is  $\mathcal{O}(K)$ .
2.1 Sample complexity upper bound
Leveraging the unified analysis of Top Two algorithms proposed by [21], we obtain the asymptotic  $\beta$ -optimality of TTUCB (Theorem 2.3). After showing the required properties for the UCB leader, we proved that tracking Top Two algorithms have similar properties as their sampling-based counterparts.
Theorem 2.3. Let  $(\delta, \beta) \in (0,1)^2$ ,  $s > 1$  and  $\alpha > 1$ . Using the threshold (2) in (1) and  $g_u$  (or  $g_m$ ) in (3), the TTUCB algorithm is  $\delta$ -correct and asymptotically  $\beta$ -optimal for all  $\mu \in \mathbb{R}^K$  such that  $\min_{i \neq j} |\mu_i - \mu_j| > 0$ , i.e., it satisfies  $\lim_{\delta \to 0} \sup_{\tau \to 0} \mathbb{E}_\mu[\tau_\delta] / \log(1/\delta) \leq T_\beta^\star(\mu)$ .
Theorem 2.3 and guarantees for other Top Two algorithms hold only for arms having distinct means. Moreover, an asymptotic result provides no guarantees on the performance in moderate regime of  $\delta$ . We address those two limitations.
Non-asymptotic upper bound Theorem 2.4 gives an upper bound on the expected sample complexity holding for any  $\delta$  and any instance having a unique best arm. It is a direct corollary of a more general result holding for any  $\beta \in (0,1)$ ,  $s > 1$  and  $\alpha > 1$  (Theorem D.4).
Theorem 2.4. Let  $\delta \in (0,1)$ . Using the threshold (2) in (1) and  $g_{u}$  in (3) with  $s = \alpha = 1.2$ , the TTUCB algorithm with  $\beta = 1/2$  satisfies that, for all  $\mu \in \mathbb{R}^{K}$  such that  $|i^{\star}(\mu)| = 1$ ,
$$
\mathbb {E} _ {\mu} \left[ \tau_ {\delta} \right] \leq \inf  _ {w _ {0} \in [ 0, (K - 1) ^ {- 1} ]} \max  \left\{T _ {0} (\delta , w _ {0}), C _ {\mu} ^ {1. 2}, C _ {0} (w _ {0}) ^ {6}, (2 / \varepsilon) ^ {1. 2} \right\} + 1 2 K,
$$
where  $C_{\mu} = h_{1}$ $(26H(\mu))$ $C_0(w_0) = 2 / (\varepsilon a_\mu (w_0)) + 1$  with  $\varepsilon \in (0,1]$
$$
T _ {0} (\delta , w _ {0}) = \sup  \left\{n \mid n - 1 \leq 2 T _ {1 / 2} ^ {\star} (\mu) (1 + \varepsilon) ^ {2} (1 - w _ {0}) ^ {- d _ {\mu} (w _ {0})} (\sqrt {c (n - 1 , \delta)} + \sqrt {4 \log n}) ^ {2} \right\},
$$
with  $a_{\mu}(w_0) = (1 - w_0)^{d_\mu(w_0)} \max \{\min_{i \neq i^\star(\mu)} w_{1/2}^\star(\mu)_i, w_0/2\}$  and  $d_{\mu}(w_0) = |\{i \neq i^\star(\mu) | w_{1/2}^\star(\mu)_i < w_0/2\}|$ . The function  $h_1(x) := x\overline{W}_{-1} \left( \log(x) + \frac{2+2K}{x} \right)$  is positive, increasing for  $x \geq 2+2K$ , and satisfies  $h_1(x) \approx x (\log x + \log \log x)$ .
The TTUCB sampling rule using  $g_{m}$  in (3) satisfies a similar upper bound (Corollary D.5). Since Theorem 2.4 holds for any instance having a unique best arm, we corroborate the intuition that assuming  $\min_{i\neq j}|\mu_i - \mu_j| > 0$  is an artifact of the existing proof to obtain asymptotic  $\beta$ -optimality.
The upper bound on  $\mathbb{E}_{\mu}[\tau_{\delta}]$  involves several terms. The  $\delta$ -dependent term is  $T_0(\delta)$ . In the asymptotic regime, we can show that  $\lim \sup_{\delta \to 0} T_0(\delta) / \log(1 / \delta) \leq 2T_{1/2}^{\star}(\mu)$  by taking  $w_0 = 0$  and letting  $\varepsilon$  go to zero. While there is (sub-optimal) factor 2 in  $T_0(\delta)$ , Theorem 2.3 shows that TTUCB is asymptotically  $1/2$ -optimal. This factor is a price we paid to obtain more explicit non-asymptotic terms, and removing it would require more sophisticated arguments in order to control the convergence of the empirical proportions  $N_n / (n - 1)$  towards  $w_{1/2}^{\star}(\mu)$ .
In the regime where  $H(\mu) \to +\infty$ , the upper bound is dominated by the  $\delta$ -independent term  $C_{\mu}^{1,2}$  (when  $\alpha = 1.2$ ) with satisfies  $C_{\mu} = \mathcal{O}(H(\mu)\log H(\mu))$ . Compared to the best known upper and lower bounds in this regime (see discussion below), our non-asymptotic term has a sub-optimal scaling in  $\mathcal{O}((H(\mu)\log H(\mu))^{\alpha})$  with  $\alpha > 1$ . While taking  $\alpha \approx 1$  would mitigate this sub-optimality, it would yield a larger dependency in  $C_0(w_0)^{\alpha /(\alpha -1)}$ . Empirically, Figures 1(b) and 5 (Appendix G.2) hints that the empirical performance of TTUCB has a better scaling with  $H(\mu)$  than  $H(\mu)^{\alpha}$ .
For instances such that  $\min_{i\neq i^{\star}}w_{1 / 2}^{\star}(\mu)_i$  is arbitrarily small, taking  $w_0 = 0$  yields an arbitrarily large  $C_0(0)$ . By clipping with  $w_0 / 2$ , we circumvent this pitfall and ensure that  $C_0(w_0) = \mathcal{O}(K / \varepsilon)$ .