SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Qining Zhang  
University of Michigan, Ann Arbor  
qiningz@umich.edu
Lei Ying  
University of Michigan, Ann Arbor  
leiying@umich.edu
This paper considers a stochastic Multi-Armed Bandit (MAB) problem with dual objectives: (i) quick identification and commitment to the optimal arm, and (ii) reward maximization throughout a sequence of  $T$  consecutive rounds. Though each objective has been individually well-studied, i.e., best arm identification for (i) and regret minimization for (ii), the simultaneous realization of both objectives remains an open problem, despite its practical importance. This paper introduces Regret Optimal Best Arm Identification (ROBAI) which aims to achieve these dual objectives. To solve ROBAI with both pre-determined stopping time and adaptive stopping time requirements, we present an algorithm called EOCP and its variants respectively, which not only achieve asymptotic optimal regret in both Gaussian and general bandits, but also commit to the optimal arm in  $\mathcal{O}(\log T)$  rounds with predetermined stopping time and  $\mathcal{O}(\log^2 T)$  rounds with adaptive stopping time. We further characterize lower bounds on the commitment time (equivalent to the sample complexity) of ROBAI, showing that EOCP and its variants are sample optimal with pre-determined stopping time, and almost sample optimal with adaptive stopping time. Numerical results confirm our theoretical analysis and reveal an interesting "over-exploration" phenomenon carried by classic UCB algorithms, such that EOCP has smaller regret even though it stops exploration much earlier than UCB, i.e.,  $\mathcal{O}(\log T)$  versus  $\mathcal{O}(T)$ , which suggests over-exploration is unnecessary and potentially harmful to system performance.
The stochastic Multi-Armed Bandit (MAB) problem [1], which models a wide range of applications including online recommendations [2, 3, 4], job assignments [5, 6, 7], clinical trials [8, 9], and etc, is a sequential decision-making process between an agent and an environment which consists of a number of arms (actions). In this paper, we will use "arm" and "action" interchangeably. Most existing studies, say [10, 11, 12, 13, 14, 15], formulate a regret minimization problem to study the bandit model, where the agent aims to maximize the cumulative reward through interacting with the environment for a number of consecutive rounds. The well-known UCB algorithm [10] and its variants [12] are among the most popular bandit algorithms for regret minimization, which exhibit outstanding performances both theoretically and empirically in bandit models. Moreover, state-of-the-art reinforcement learning algorithms, e.g., [16, 17], are also motivated by the essence of UCB which sets up confidence intervals to encourage exploration. However, the UCB algorithms do not commit to a single arm. Instead, they continue to switch among all arms based on reward signals received. Since one of the ultimate goals of the MAB model is to learn the optimal arm, it would be ideal if the algorithm would commit to an arm quickly without sacrificing the regret. In fact and in practice, a number of applications such as occupational decisions [18], medicine release and pandemic control [19, 20], and long-term investments [21], require or prefer quick commitment to an action instead of continuous exploration. This motivated us to consider an MAB problem with dual
Algorithm	Regret	Setting	Optimality	Commitment	Confidence
UCB [22]	2+o(1)/Δ log T	Gaussian	Yes	T	N/A
KL-UCB [12]	Δ+o(1)/KL(μ2,μ1) log T	General	Yes	T	N/A
TS [15]	Δ+o(1)/KL(μ2,μ1) log T	General	Yes	T	N/A
BAI-ETC [22]	4+o(1)/Δ log T	Gaussian	No	O(log T)	O(T-1)
DETC [23]	2+o(1)/Δ log T	Gaussian	Yes	Ω(log2T)	O(T-1)
UCBα [24]	2α2+o(1)/Δ log T	Gaussian	No	O(log T)	O(T-1)
EOCP (Ours)	2+o(1)/Δ log T	Gaussian	Yes	O(log T)	O(T-1)
EOCP-UG (Ours)	2+o(1)/Δ log T	Gaussian	Yes	O(log2T)	O(T-1)
KL-EOCP (Ours)	Δ+o(1)/KL(μ2,μ1) log T	General	Yes	O(log T)	O(T-1)
LB(pre-determined)	2+o(1)/Δ log T	Gaussian	O(logcT)	O(log T)	O(T-1)
LB(adaptive)	2+o(1)/Δ log T	Gaussian	O(logcT)	O(log2-cT)	O(T-1)
Table 1: Caparison under 2-armed bandits where  $\Delta = |\mu_1 - \mu_2|$  is the expected reward difference,  $\mathrm{KL}(\mu_2,\mu_1)$  is the Kullback-Leibler divergence between reward distributions, and  $\alpha >1$ . EOCP and KL-EOCP use a pre-determined stopping time and require the knowledge of  $\Delta$ , while EOCP-UG uses an adaptive stopping time. Both LBs represent the commitment time lower bound under Gaussian bandits for regret optimal algorithms with  $\mathcal{O}(\log^c T)$  finite-time regret violation and  $\mathcal{O}(T^{-1})$  confidence.
objectives: (i) quick identification and commitment to the optimal arm, and (ii) minimization of the cumulative regret throughout a sequence of rounds.
Regret Optimal Best Arm Identification: The lack of commitment in traditional regret minimization formulation motivates us to propose a new viewpoint towards online decision-making called Regret Optimal Best Arm Identification (ROBAI), which intends to manage two goals at the same time: minimizing regret while committing to an arm quickly. Specifically, it is ideal for the agent to quickly commit to the optimal arm which has the highest expected reward while minimizing the exploration regret. To solve ROBAI, we need to answer three fundamental questions: (1) how should the learner explore arms while maintaining a low-regret performance (exploration strategy)? (2) when should the learner stop exploration and commit to an arm (stopping criterion)? and (3) which arm to commit to when the exploration ends (action selection strategy)? All three components need to be designed together to make the algorithm most efficient. The fundamental question this paper addresses is: Can we design an efficient algorithm that is both regret optimal and identifies the optimal arm quickly, and what are the fundamental limits of such algorithms?
Connection to Best Arm Identification: One approach people may take to solve ROBAI is the Best Arm Identification (BAI) algorithm, which studies how to learn the optimal arm with the minimum number of samples (rounds) and then commit to the selected arm. However, since BAI focuses purely on sample complexity (or commitment time), the algorithms for BAI explore sub-optimal arms too aggressively and too often, leading to large regret. As shown in [22], the regret is asymptotically at least twice as large as the regret under the classic UCB algorithm. Modifications shown in [23] may lead to better regret performance, but they also make the algorithm too complicated to find the optimal arm quickly. Moreover, these algorithms adapted from BAI often exhibit poor empirical regret performances as shown in our numerical experiments of Fig. 1.
Our Contributions: We propose an algorithm called EOCP, which stands for Explore Optimistically then Commit Pessimistically, to solve the ROBAI problem. It first uses an optimistic modified version of the classic UCB algorithm to explore arms with a slightly larger exploration function, and then it commits to the optimal arm candidate according to a pessimistic LCB algorithm when the exploration ends, by selecting the arm that has the largest lower confidence bound. The exploration and action identification strategies are respectively motivated by the inflated bonus trick [24] and the principle of pessimism from the literature of offline bandits [25, 26, 27] and offline reinforcement learning [28, 29, 30]. Our greatest contributions include designing new stopping rules with both predetermined stopping time (vanilla EOCP) and adaptive stopping time (the EOCP-UG variant), which probably balance the trade-off between regret minimization and action identification. We theoretically
show that both algorithms are asymptotically regret optimal in Gaussian bandit models. Moreover, EOCP and EOCP-UG algorithms commit to the optimal arm in  $\mathcal{O}(\log T)$  and  $\mathcal{O}(\log^2 T)$  number of rounds respectively, both with  $\mathcal{O}(T^{-1})$  confidence. We further characterize the fundamental commitment time (sample complexity until commitment) limits of action identification for regret optimal algorithms, which shows that  $\mathcal{O}(\log T)$  number of samples is always required with predetermined stopping time, and  $\mathcal{O}(\log^{2 - c}T)$  number of samples is required with adaptive stopping time if the finite-time regret of the algorithm does not exceed its asymptotic regret rate by  $\mathcal{O}(\log^c T)$ . This shows that EOCP is sample optimal and EOCP-UG is nearly sample optimal. We also propose an improved algorithm called KL-EOCP to achieve regret optimality in general bandits beyond Gaussian rewards. To the best of our knowledge, KL-EOCP is the first algorithm that not only achieves asymptotic regret optimality in general bandit models but also commits to the optimal arm in  $\mathcal{O}(\log T)$  rounds, which also matches the commitment time lower bound. The more detailed comparison between existing algorithms and our proposed algorithms with lower bounds is summarized in Tab. 1. Numerical experiments confirm the superiority of our proposed algorithm and show an interesting "over-exploration" phenomenon carried by UCB algorithms. As shown in Fig. 1, our EOCP algorithm reduces more than  $20\%$  of regret compared to the vanilla UCB algorithm by finding and committing to the optimal arm early.
Stochastic Multi-armed Bandits: A stochastic multi-armed bandit problem is an online decision-making process between an agent and an environment for a number of  $T$  consecutive rounds. At each round  $t \in \{1, 2, \dots, T\}$ , the agent can choose an action  $A_t$  among a set of actions (arms) with cardinality  $A$ , denoted by  $\mathcal{A} = \{1, 2, \dots, A\}$ , to interact with the environment. Each action  $a$  is associated with a probability distribution  $\nu_a$  and we denote the set of distributions as  $\pmb{\nu} = \{\nu_1, \dots, \nu_A\}$  with respective expectations  $\pmb{\mu} = \{\mu_1, \dots, \mu_A\}$  which is unknown to the agent a priori. The expectations are assumed to be bounded so without loss of generality, we have  $\mu_a \in [0, 1]$  for any action  $a$ . After the agent chooses an action, say action  $A_t$  at round  $t$ , it will observe an independent reward  $r_t$  which is sampled from the distribution  $\nu_{A_t}$  associated with the action  $A_t$  that it chooses. We define the optimal action  $a^*$  to be the action which has the highest expected reward, i.e.,  $\mu_{a^*} = \arg \max_{a \in \mathcal{A}} \mu_a$ , and for simplicity, we assume it is unique. Let  $\Delta_a = |\mu_a^* - \mu_a| \in (0, 1]$  to be the expected reward gap between the optimal action and a sub-optimal action  $a$ , and we use  $\Delta_{\min} = \min_{a : \Delta_a > 0} |\mu_a^* - \mu_a|$  to denote the minimum reward gap among all sub-optimal actions.
Regret: The goal of the agent is to maximize the expected cumulative reward from the total  $T$  rounds of interactions with the environment, i.e., to maximize  $\mathbb{E}[r_1 + r_2 + \dots + r_T]$ , where the expectation is taken over all randomness. The performance of any bandit algorithm Alg chosen by the agent is usually measured by the cumulative Regret up to round  $T$  defined as follows:
$$
\operatorname {R e g} _ {\boldsymbol {\mu}} ^ {\mathrm {A l g}} (T) = T \mu_ {a *} - \mathbb {E} _ {\boldsymbol {\mu}} \left[ \sum_ {t = 1} ^ {T} r _ {t} \right],
$$
where the subscript  $\mu$  denotes the bandit instance represented by the reward expectations. Maximizing reward is equivalent to minimizing the cumulative regret. For most of the algorithms to achieve this goal, the agent will make action-choosing decisions based on two statistics maintained and updated at each round for every action: the empirical mean  $\bar{r}_t(a)$  and the number of pulls  $N_{t}(a)$  in previous rounds. They are defined as:
$$
N _ {t} (a) = \sum_ {k = 1} ^ {t} \mathbb {1} _ {A _ {t} = a}, \quad \bar {r} _ {t} (a) = \frac {1}{N _ {t} (a)} \sum_ {k = 1} ^ {t} r _ {t} \mathbb {1} _ {A _ {t} = a}.
$$
The theoretical regret limit of any algorithm Alg is studied and characterized in [13], which shows:
$$
\lim  _ {T \rightarrow \infty} \frac {\operatorname {R e g} _ {\boldsymbol {\mu}} ^ {\mathrm {A l g}} (T)}{\log T} \geqslant \sum_ {a: \Delta_ {a} > 0} \frac {\Delta_ {a}}{\operatorname {K L} \left(\nu_ {a} , \nu_ {a ^ {*}}\right)}, \tag {1}
$$
where  $\mathsf{KL}(\cdot ,\cdot)$  denotes the Kullback-Leibler divergence between two distributions. We call an algorithm  $\mathsf{Alg}$  regret optimal (asymptotically) if the asymptotic regret performance of  $\mathsf{Alg}$  achieves this lower bound. Therefore, whether an algorithm is asymptotically regret optimal will depend on
the distributions  $\pmb{\nu}$  of the rewards. Specifically, for Gaussian bandits, the KL divergence between distributions  $\mathcal{N}(\mu_a,1)$  and  $\mathcal{N}(\mu_{a'}',1)$  is simply  $(\mu_{a} - \mu_{a'})^{2} / 2$ . So the asymptotic regret rate lower bound in the RHS for Gaussian bandits would be  $2\sum_{a:\Delta_a > 0}\Delta_a^{-1}$ .
**Commitment:** In ROBAI, commitment to a single action  $\hat{a}$  (ideally, the optimal action) is required. After a stopping time  $T_{\mathrm{c}}$ , the agent will not be allowed to switch actions and will commit to the same action until the end. We consider two categories of commitment: the pre-determined stopping-time setting and the adaptive stopping time setting. The pre-determined stopping time requires  $T_{\mathrm{c}}$  to be pre-specified before the first round of interaction, while the adaptive stopping criterion requires  $T_{\mathrm{c}}$  to be a stopping time measurable to the natural filtration. How quickly the agent commits is measured by the Sample Complexity until Commitment (also called commitment time) which is the expected number of exploration rounds, i.e.,  $\mathsf{SCC}_{\mu}^{\mathsf{Alg}}(T) = \mathbb{E}_{\mu}[T_{\mathrm{c}}]$ . The accuracy of identifying the optimal action is measured by the confidence, which is the probability that the agent commits to a sub-optimal action, i.e.,  $\mathbb{P}_{\mu}(\hat{a} \neq a^{*})$ . An ideal algorithm should minimize the commitment time while maintaining confidence lower than a pre-specified threshold.
ROBAI Problem Formulation: We use  $\Pi_{\mathrm{RO}}$  to denote the class of regret optimal algorithms which commit to the optimal action with a confidence lower than  $\mathcal{O}(T^{-1})$ , i.e.,
$$
\Pi_ {\mathrm {R O}} = \left\{\mathsf {A l g} \Bigg | \lim  _ {T \to \infty} \frac {\operatorname {R e g} _ {\boldsymbol {\mu}} ^ {\mathsf {A l g}} (T)}{\log T} \leqslant \sum_ {a: \Delta_ {a} > 0} \frac {\Delta_ {a}}{\operatorname {K L} (\nu_ {a} , \nu_ {a *})} \text {a n d} \mathbb {P} _ {\boldsymbol {\mu}} (\hat {a} \neq a ^ {*}) = \mathcal {O} \left(T ^ {- 1}\right) \right\}.
$$
ROBAI aims to design a regret optimal algorithm  $\mathsf{Alg} \in \Pi_{\mathsf{RO}}$  to minimize the commitment time:
$$
\min  \mathsf {S C C} _ {\boldsymbol {\mu}} ^ {\mathrm {A l g}} (T) = \mathbb {E} _ {\boldsymbol {\mu}} \left[ T _ {\mathrm {c}} \right], \quad \text {s . t .}, \quad \mathsf {A l g} \in \Pi_ {\mathrm {R O}}.
$$
3 Low-Complexity Algorithms
In this section, we propose a low-complexity algorithm called EOCP with a pre-determined stopping time to solve ROBAI. Then, we propose a variant called EOCP-UG with adaptive stopping time.
3.1 The Pre-determined Stopping Time Setting
The pre-determined stopping time setting is motivated by real-world applications such as A/B tests in medical experiments with operational or budget limits [31], where the number of testers is usually predefined and determined before the experiment starts. Therefore, we require the agent to pre-specified the stopping time  $T_{\mathrm{c}}$  before the first round. We also assume the algorithm knows a strictly positive lower bound  $\Delta_{\mathrm{lb}}$  on the minimum reward gap  $\Delta_{\mathrm{min}}$  between the optimal action and sub-optimal actions. We propose our EOCP algorithm in Algorithm 1.
Algorithm 1 EOCP with Pre-determined Stopping Time
Require: Exploration function  $l$ ; Lower bound  $\Delta_{\mathrm{lb}}$  on minimum reward gap.  
1: Let  $T_{\mathrm{c}} = \frac{8Al}{\Delta_{\mathrm{lb}}^{2}} + A$  be the pre-determined stopping time.  
2: Initialize by pulling each arm  $a$  once.  
3: for  $t = A + 1 : T_{\mathrm{c}}$  do  
4: Set uncertainty bonus  $b_{t-1}(a) = \sqrt{\frac{2l}{N_{t-1}(a)}}$ , and  $\mathsf{UCB}_{t-1}(a) = \bar{r}_{t-1}(a) + b_{t-1}(a)$ .  
5: Take action  $A_t = \arg \max_a \mathsf{UCB}_{t-1}(a)$ . // UCB Exploration  
6: end for  
7: Set bonus  $b_{T_{\mathrm{c}}} (a) = \sqrt{\frac{2l}{N_{T_{\mathrm{c}}} (a)}}$ , and  $\mathsf{LCB}_{T_{\mathrm{c}}} (a) = \bar{r}_{T_{\mathrm{c}}} (a) - b_{T_{\mathrm{c}}} (a)$ .  
8: For  $t \in [T_{\mathrm{c}} + 1, T]$ , commit to action  $\hat{a} = \arg \max_a \mathsf{LCB}_{T_{\mathrm{c}}} (a)$ . // LCB Commitment
In EOCP, the agent will spend the first  $A$  rounds exploring each arm once as a start. Our choice of exploration strategy is a modified version of the classic UCB algorithm where the agent will choose the action with the largest upper confidence bound in terms of empirical reward. The exploration function  $l$  controls the intensity of exploration to achieve the optimal trade-off between reducing uncertainty for action identification and minimizing regret. After the pre-determined stopping time  $T_{c}$ , the agent will commit to an action that has the largest lower confidence bound of empirical
reward. This LCB commitment strategy is inspired by the principle of pessimism from the literature of offline learning [25, 26, 27, 28, 29, 30], where the empirical reward of each action is penalized by the amount of uncertainty to combat the imbalanced data coverage of actions in the offline dataset. It is also shown that the pessimistic principle works well when the data coverage is concentrated on the optimal action, i.e., the optimal action has the largest number of pulls. This trait of the LCB algorithm matches the trait of UCB exploration, in which the optimal action will be chosen much more often than sub-optimal actions in exploration. So by designing such a proper  $T_{\mathrm{c}}$ , we will be able to achieve the best of both worlds: a low-regret exploration of the UCB algorithm, and a fast best arm identification through the choice of LCB algorithm.
3.2 The Adaptive Stopping Time Setting
In this setting, we do not assume the algorithm is provided a priori with additional information on the lower bound  $\Delta_{\mathrm{lb}}$  on the minimum reward gap, so there is no hope of designing a pre-determined stopping time. Instead, we design our stopping criterion based on the samples collected from the explorations, which leads to the fact that  $T_{\mathrm{c}}$  is a stopping time measurable to the natural filtration. We propose our EOCP-UG algorithm in Algorithm 2 corresponding to an unknown gap.
Algorithm 2 EOCP-UG with Adaptive Stopping Time
Require: Exploration function  $l$    
1: Initialize by pulling each arm  $a$  once.   
2: while  $\max_{a}\min_{a^{\prime}}N_{t - 1}(a) - lN_{t - 1}(a^{\prime})\leqslant 1$  do   
3: Set uncertainty bonus  $b_{t - 1}(a) = \sqrt{\frac{2l}{N_{t - 1}(a)}},\mathsf{UCB}_{t - 1}(a) = \bar{r}_{t - 1}(a) + b_{t - 1}(a).$    
4: Take action  $A_{t} = \arg \max_{a}\mathsf{UCB}_{t - 1}(a)$  //UCB Exploration   
5: end while   
6: Let  $T_{\mathrm{c}}\gets t - 1$  ,bonus  $b_{T_{\mathrm{c}}}(a) = \sqrt{\frac{2l}{N_{T_{\mathrm{c}}} (a)}}$  , and  $\mathsf{LCB}_{T_{\mathrm{c}}}(a) = \bar{r}_{T_{\mathrm{c}}}(a) - b_{T_{\mathrm{c}}}(a)$    
7: For  $t\in [T_{\mathrm{c}} + 1,T]$  , commit to the action  $\hat{a} = \arg \max_a\mathsf{LCB}_{T_c}(a)$  // LCB Commitment
In EOCP-UG, we use the same UCB exploration and LCB best action identification strategies as in the pre-determined stopping time setting. The only difference compared to EOCP comes from the new stopping rule based on the number of pulls  $N_{t}(a)$  for each action. Specifically, the exploration ends if there is an imbalanced fraction of  $N_{t}(a)$  among all actions, that is, one action has  $l$  times more pulls than all other actions in previous rounds. Here,  $l$  is the exploration function and we will select  $l$  to be slightly larger than  $\log T$  in later sections. The intuition of such a stopping criterion comes from the characteristics of UCB exploration, i.e., as round  $t$  increases, the algorithm will slowly adapt to choosing the optimal action more often. When the fraction between the number of pulls for the optimal action and any sub-optimal action is large enough, the optimal action will be identifiable. Note that in action identification, we can simply choose the action that has the largest number of pulls  $N_{T_{\mathrm{c}}}(a)$  when we stop, and obtain exactly the same performance guarantees. However, to keep it consistent with the pre-determined stopping time setting, we use the LCB commitment.
In this section, we assume the distributions  $\{\nu_{1},\dots ,\nu_{A}\}$  come from a Gaussian family, where  $\nu_{a}$  associated with action  $a$  follows a Gaussian distribution with mean  $\mu_{a}$  and unit variance, i.e.,  $\nu_{a}\sim \mathcal{N}(\mu_{a},1)$ . The results in this section can be easily generalized to sub-Gaussian distributions. The key idea towards the optimal trade-off between controlling regret and identifying the optimal action is inspired by the inflated bonus trick from [24], where we will choose the exploration function  $l$  to be slightly larger than the log  $T$  used in vanilla UCB algorithms to encourage more exploration of sub-optimal arms, i.e., we will select  $l = \log T + \mathcal{O}(\sqrt{\log T})$ .
4.1 Regret Optimality for Gaussian Bandits with Pre-Determined Stopping Time
In Theorem 1, we present the theoretical regret performance guarantee of the EOCP algorithm:
Theorem 1 Let  $l = \log (T) + \mathcal{O}(\sqrt{\log T})$  and when  $T$  is large enough, the expected regret of the EOCP algorithm in Algorithm 1 with pre-determined stopping time can be upper-bounded by:
$$
\operatorname {R e g} _ {\boldsymbol {\mu}} ^ {\mathrm {E O C P}} (T) \leqslant \sum_ {a: \Delta_ {a} > 0} \frac {2 \log T}{\Delta_ {a}} + \mathcal {O} \left(\frac {\sqrt {\log T}}{\Delta_ {\min}}\right).
$$
A direct asymptotic bound can be obtained from Theorem 1, i.e.,
$$
\limsup_{T\to \infty}\frac{\operatorname{Reg}_{\mu}^{\text{EOCP}}(T)}{\log T}\leqslant \sum_{a:\Delta_{a} > 0}\frac{2}{\Delta_{a}}.
$$
Comparing it to Eq. (1), it is clear that EOCP is asymptotically regret optimal in Gaussian bandits. The commitment time and confidence guarantees can be extracted from the setup of Algorithm. 1 itself and the proof of Theorem. 1. Recall that in Algorithm. 1, we pre-determined the length of exploration  $T_{\mathrm{c}}$  to be  $\mathcal{O}(\Delta_{\mathrm{lb}}^{-2}l)$ . The following corollary characterizes these parts of theoretical performance:
Corollary 1 Let  $l = \log (T) + \mathcal{O}(\sqrt{\log T})$ , the expected commitment time for EOCP in Algorithm 1 is given by
$$
\mathsf {S C C} _ {\mu} ^ {\mathsf {E O C P}} (T) = \mathcal {O} (\Delta_ {\mathrm {l b}} ^ {- 2} \log T),
$$
and the confidence level is  $\mathcal{O}(T^{-1})$
The complete proofs of Theorem 1 and Corollary 1 are provided in the supplementary material. In order to upper bound the cumulative regret of  $T$  rounds, we divide the total regret into the regret accumulated in exploration and the regret accumulated in commitment.
Bounding Regret from Exploration: To bound the regret accumulated in exploration and since we use a variant of UCB exploration, we follow the standard procedure of proofs for UCB algorithms, e.g., proof of Theorem 8 from. [22]. Then, this procedure results in an order  $\mathcal{O}(l)$  dominating regret term, which is  $\mathcal{O}(\log T)$  by the choice of our exploration function. Through carefully applying any-time concentration inequalities, we are able to show that the constant in front of this dominating regret term is exactly the constant we obtained in Theorem. 1.