Title: Fast and Regret Optimal Best Arm Identification: Fundamental Limits and Low-Complexity Algorithms
Venue: NeurIPS 2023
Accepted: yes
Keywords: stochastic multi-armed bandits

regret optimal best arm identification

commitment
Abstract: This paper considers a stochastic Multi-Armed Bandit (MAB) problem with dual objectives: (i) quick identification and commitment to the optimal arm, and (ii) reward maximization throughout a sequence of $T$ consecutive rounds. Though each objective has been individually well-studied, i.e., best arm identification for (i) and regret minimization for (ii), the simultaneous realization of both objectives remains an open problem, despite its practical importance. This paper introduces \emph{Regret Optimal Best Arm Identification} (ROBAI) which aims to achieve these dual objectives. To solve ROBAI with both pre-determined stopping time and adaptive stopping time requirements, we present an algorithm called EOCP and its variants respectively, which not only achieve asymptotic optimal regret in both Gaussian and general bandits, but also commit to the optimal arm in $\mathcal{O}(\log T)$ rounds with pre-determined stopping time and $\mathcal{O}(\log^2 T)$ rounds with adaptive stopping time. We further characterize lower bounds on the commitment time (equivalent to the sample complexity) of ROBAI, showing that EOCP and its variants are sample optimal with pre-determined stopping time, and almost sample optimal with adaptive stopping time. Numerical results confirm our theoretical analysis and reveal an interesting ``over-exploration'' phenomenon carried by classic UCB algorithms, such that EOCP has smaller regret even though it stops exploration much earlier than UCB, i.e., $\mathcal{O}(\log T)$ versus $\mathcal{O}(T)$, which suggests over-exploration is unnecessary and potentially harmful to system performance.

Fast and Regret Optimal Best Arm Identification: Fundamental Limits and Low-Complexity Algorithms

Abstract

This paper considers a stochastic Multi-Armed Bandit (MAB) problem with dual objectives: (i) quick identification and commitment to the optimal arm, and (ii) reward maximization throughout a sequence of T consecutive rounds. Though each objective has been individually well-studied, i.e., best arm identification for (i) and regret minimization for (ii), the simultaneous realization of both objectives remains an open problem, despite its practical importance. This paper introduces Regret Optimal Best Arm Identification (ROBAI) which aims to achieve these dual objectives. To solve ROBAI with both pre-determined stopping time and adaptive stopping time requirements, we present an algorithm called EOCP and its variants respectively, which not only achieve asymptotic optimal regret in both Gaussian and general bandits, but also commit to the optimal arm in O p log T q rounds with predetermined stopping time and O p log 2 T q rounds with adaptive stopping time. We further characterize lower bounds on the commitment time (equivalent to the sample complexity) of ROBAI, showing that EOCP and its variants are sample optimal with pre-determined stopping time, and almost sample optimal with adaptive stopping time. Numerical results confirm our theoretical analysis and reveal an interesting 'over-exploration' phenomenon carried by classic UCB algorithms, such that EOCP has smaller regret even though it stops exploration much earlier than UCB, i.e., O p log T q versus O p T q , which suggests over-exploration is unnecessary and potentially harmful to system performance.

1 Introduction

The stochastic Multi-Armed Bandit (MAB) problem [1], which models a wide range of applications including online recommendations [2, 3, 4], job assignments [5, 6, 7], clinical trials [8, 9], and etc, is a sequential decision-making process between an agent and an environment which consists of a number of arms (actions). In this paper, we will use 'arm' and 'action' interchangeably. Most existing studies, say [10, 11, 12, 13, 14, 15], formulate a regret minimization problem to study the bandit model, where the agent aims to maximize the cumulative reward through interacting with the environment for a number of consecutive rounds. The well-known UCB algorithm [10] and its variants [12] are among the most popular bandit algorithms for regret minimization, which exhibit outstanding performances both theoretically and empirically in bandit models. Moreover, state-of-the-art reinforcement learning algorithms, e.g., [16, 17], are also motivated by the essence of UCB which sets up confidence intervals to encourage exploration. However, the UCB algorithms do not commit to a single arm. Instead, they continue to switch among all arms based on reward signals received. Since one of the ultimate goals of the MAB model is to learn the optimal arm, it would be ideal if the algorithm would commit to an arm quickly without sacrificing the regret. In fact and in practice, a number of applications such as occupational decisions [18], medicine release and pandemic control [19, 20], and long-term investments [21], require or prefer quick commitment to an action instead of continuous exploration. This motivated us to consider an MAB problem with dual

objectives: (i) quick identification and commitment to the optimal arm, and (ii) minimization of the cumulative regret throughout a sequence of rounds.

Regret Optimal Best Arm Identification: The lack of commitment in traditional regret minimization formulation motivates us to propose a new viewpoint towards online decision-making called Regret Optimal Best Arm Identification (ROBAI), which intends to manage two goals at the same time: minimizing regret while committing to an arm quickly. Specifically, it is ideal for the agent to quickly commit to the optimal arm which has the highest expected reward while minimizing the exploration regret. To solve ROBAI, we need to answer three fundamental questions: (1) how should the learner explore arms while maintaining a low-regret performance ( exploration strategy )? (2) when should the learner stop exploration and commit to an arm ( stopping criterion )? and (3) which arm to commit to when the exploration ends ( action selection strategy )? All three components need to be designed together to make the algorithm most efficient. The fundamental question this paper addresses is: Can we design an efficient algorithm that is both regret optimal and identifies the optimal arm quickly, and what are the fundamental limits of such algorithms?

Connection to Best Arm Identification: One approach people may take to solve ROBAI is the Best Arm Identification (BAI) algorithm, which studies how to learn the optimal arm with the minimum number of samples (rounds) and then commit to the selected arm. However, since BAI focuses purely on sample complexity (or commitment time), the algorithms for BAI explore sub-optimal arms too aggressively and too often, leading to large regret. As shown in [22], the regret is asymptotically at least twice as large as the regret under the classic UCB algorithm. Modifications shown in [23] may lead to better regret performance, but they also make the algorithm too complicated to find the optimal arm quickly. Moreover, these algorithms adapted from BAI often exhibit poor empirical regret performances as shown in our numerical experiments of Fig. 1.

Our Contributions: We propose an algorithm called EOCP, which stands for Explore Optimistically then Commit Pessimistically , to solve the ROBAI problem. It first uses an optimistic modified version of the classic UCB algorithm to explore arms with a slightly larger exploration function, and then it commits to the optimal arm candidate according to a pessimistic LCB algorithm when the exploration ends, by selecting the arm that has the largest lower confidence bound. The exploration and action identification strategies are respectively motivated by the inflated bonus trick [24] and the principle of pessimism from the literature of offline bandits [25, 26, 27] and offline reinforcement learning [28, 29, 30]. Our greatest contributions include designing new stopping rules with both predetermined stopping time (vanilla EOCP) and adaptive stopping time (the EOCP-UG variant), which probably balance the trade-off between regret minimization and action identification. We theoretically

show that both algorithms are asymptotically regret optimal in Gaussian bandit models. Moreover, EOCP and EOCP-UG algorithms commit to the optimal arm in O p log T q and O p log 2 T q number of rounds respectively, both with O p T ´ 1 q confidence. We further characterize the fundamental commitment time (sample complexity until commitment) limits of action identification for regret optimal algorithms, which shows that O p log T q number of samples is always required with predetermined stopping time, and O p log 2 ´ c T q number of samples is required with adaptive stopping time if the finite-time regret of the algorithm does not exceed its asymptotic regret rate by O p log c T q . This shows that EOCP is sample optimal and EOCP-UG is nearly sample optimal. We also propose an improved algorithm called KL-EOCP to achieve regret optimality in general bandits beyond Gaussian rewards. To the best of our knowledge, KL-EOCP is the first algorithm that not only achieves asymptotic regret optimality in general bandit models but also commits to the optimal arm in O p log T q rounds, which also matches the commitment time lower bound. The more detailed comparison between existing algorithms and our proposed algorithms with lower bounds is summarized in Tab. 1. Numerical experiments confirm the superiority of our proposed algorithm and show an interesting 'over-exploration' phenomenon carried by UCB algorithms. As shown in Fig. 1, our EOCP algorithm reduces more than 20% of regret compared to the vanilla UCB algorithm by finding and committing to the optimal arm early.

2 Preliminaries

Stochastic Multi-armed Bandits: A stochastic multi-armed bandit problem is an online decisionmaking process between an agent and an environment for a number of T consecutive rounds. At each round t P t 1 , 2 , ¨ ¨ ¨ , T u , the agent can choose an action A t among a set of actions (arms) with cardinality A , denoted by A ' t 1 , 2 , ¨ ¨ ¨ , A u , to interact with the environment. Each action a is associated with a probability distribution ν a and we denote the set of distributions as ν ' t ν 1 , ¨ ¨ ¨ , ν A u with respective expectations µ ' t µ 1 , ¨ ¨ ¨ , µ A u which is unknown to the agent a priori. The expectations are assumed to be bounded so without loss of generality, we have µ a P r 0 , 1 s for any action a . After the agent chooses an action, say action A t at round t , it will observe an independent reward r t which is sampled from the distribution ν A t associated with the action A t that it chooses. We define the optimal action a ˚ to be the action which has the highest expected reward, i.e., µ a ˚ ' arg max a P A µ a , and for simplicity, we assume it is unique. Let ∆ a ' | µ a ˚ ´ µ a | P p 0 , 1 s to be the expected reward gap between the optimal action and a sub-optimal action a , and we use ∆ min ' min a :∆ a ą 0 | µ a ˚ ´ µ a | to denote the minimum reward gap among all sub-optimal actions.

Regret: The goal of the agent is to maximize the expected cumulative reward from the total T rounds of interactions with the environment, i.e., to maximize E r r 1 ` r 2 `¨¨¨ ` r T s , where the expectation is taken over all randomness. The performance of any bandit algorithm Alg chosen by the agent is usually measured by the cumulative Regret up to round T defined as follows:

where the subscript µ denotes the bandit instance represented by the reward expectations. Maximizing reward is equivalent to minimizing the cumulative regret. For most of the algorithms to achieve this goal, the agent will make action-choosing decisions based on two statistics maintained and updated at each round for every action: the empirical mean ¯ r t p a q and the number of pulls N t p a q in previous rounds. They are defined as:

The theoretical regret limit of any algorithm Alg is studied and characterized in [13], which shows:

where KL p¨ , ¨q denotes the Kullback-Leibler divergence between two distributions. We call an algorithm Alg regret optimal (asymptotically) if the asymptotic regret performance of Alg achieves this lower bound. Therefore, whether an algorithm is asymptotically regret optimal will depend on

the distributions ν of the rewards. Specifically, for Gaussian bandits, the KL divergence between distributions N p µ a , 1 q and N p µ a 1 , 1 q is simply p µ a ´ µ a 1 q 2 { 2 . So the asymptotic regret rate lower bound in the RHS for Gaussian bandits would be 2 ř a :∆ a ą 0 ∆ ´ 1 a .

Commitment: In ROBAI, commitment to a single action ˆ a (ideally, the optimal action) is required. After a stopping time T c , the agent will not be allowed to switch actions and will commit to the same action until the end. We consider two categories of commitment: the pre-determined stopping-time setting and the adaptive stopping time setting. The pre-determined stopping time requires T c to be pre-specified before the first round of interaction, while the adaptive stopping criterion requires T c to be a stopping time measurable to the natural filtration. How quickly the agent commits is measured by the Sample Complexity until Commitment (also called commitment time) which is the expected number of exploration rounds, i.e., SCC Alg µ p T q ' E µ r T c s . The accuracy of identifying the optimal action is measured by the confidence, which is the probability that the agent commits to a sub-optimal action, i.e., P µ p ˆ a ‰ a ˚ q . An ideal algorithm should minimize the commitment time while maintaining confidence lower than a pre-specified threshold.

ROBAI Problem Formulation: We use Π RO to denote the class of regret optimal algorithms which commit to the optimal action with a confidence lower than O p T ´ 1 q , i.e.,

ROBAI aims to design a regret optimal algorithm Alg P Π RO to minimize the commitment time:

3 Low-Complexity Algorithms

In this section, we propose a low-complexity algorithm called EOCP with a pre-determined stopping time to solve ROBAI. Then, we propose a variant called EOCP-UG with adaptive stopping time.

3.1 The Pre-determined Stopping Time Setting

The pre-determined stopping time setting is motivated by real-world applications such as A/B tests in medical experiments with operational or budget limits [31], where the number of testers is usually predesigned and determined before the experiment starts. Therefore, we require the agent to pre-specify the stopping time T c before the first round. We also assume the algorithm knows a strictly positive lower bound ∆ lb on the minimum reward gap ∆ min between the optimal action and sub-optimal actions. We propose our EOCP algorithm in Algorithm. 1.

Algorithm 1 EOCP with Pre-determined Stopping Time

Require: Exploration function l ; Lower bound ∆ lb on minimum reward gap.

1: Let T c ' 8 Al ∆ 2 ` A be the pre-determined stopping time.

lb 2: Initialize by pulling each arm a once.

3: for t ' A ` 1 : T c do

b

4: Set uncertainty bonus b t ´ 1 p a q ' 2 l N t ´ 1 p a q , and UCB t ´ 1 p a q ' ¯ r t ´ 1 p a q ` b t ´ 1 p a q .

5: Take action A t ' arg max a UCB t ´ 1 p a q . // UCB Exploration

6: end for

7: Set bonus b T c p a q ' b 2 l N T c p a q , and LCB T c p a q ' ¯ r T c p a q ´ b T c p a q .

8: For t P r T c ` 1 , T s , commit to action ˆ a ' arg max a LCB T c p a q . // LCB Commitment

In EOCP, the agent will spend the first A rounds exploring each arm once as a start. Our choice of exploration strategy is a modified version of the classic UCB algorithm where the agent will choose the action with the largest upper confidence bound in terms of empirical reward. The exploration function l controls the intensity of exploration to achieve the optimal trade-off between reducing uncertainty for action identification and minimizing regret. After the pre-determined stopping time T c , the agent will commit to an action that has the largest lower confidence bound of empirical

reward. This LCB commitment strategy is inspired by the principle of pessimism from the literature of offline learning [25, 26, 27, 28, 29, 30], where the empirical reward of each action is penalized by the amount of uncertainty to combat the imbalanced data coverage of actions in the offline dataset. It is also shown that the pessimistic principle works well when the data coverage is concentrated on the optimal action, i.e., the optimal action has the largest number of pulls. This trait of the LCB algorithm matches the trait of UCB exploration, in which the optimal action will be chosen much more often than sub-optimal actions in exploration. So by designing such a proper T c , we will be able to achieve the best of both worlds: a low-regret exploration of the UCB algorithm, and a fast best arm identification through the choice of LCB algorithm.

3.2 The Adaptive Stopping Time Setting

In this setting, we do not assume the algorithm is provided a priori with additional information on the lower bound ∆ lb on the minimum reward gap, so there is no hope of designing a pre-determined stopping time. Instead, we design our stopping criterion based on the samples collected from the explorations, which leads to the fact that T c is a stopping time measurable to the natural filtration. We propose our EOCP-UG algorithm in Algorithm 2 corresponding to an unknown gap.

Algorithm 2 EOCP-UG with Adaptive Stopping Time

Require: Exploration function l .

1: Initialize by pulling each arm a once.

2: while max a min a 1 N t ´ 1 p a q ´ lN t ´ 1 p a 1 q ď 1 do

b

3: Set uncertainty bonus b t ´ 1 p a q ' 2 l N t ´ 1 p a q , UCB t ´ 1 p a q ' ¯ r t ´ 1 p a q ` b t ´ 1 p a q .

4: Take action A t ' arg max a UCB t ´ 1 p a q . // UCB Exploration

5: end while

6: Let T c Ð t ´ 1 , bonus b T c p a q ' b 2 l N T c p a q , and LCB T c p a q ' ¯ r T c p a q ´ b T c p a q .

7: For t P r T c ` 1 , T s , commit to the action ˆ a ' arg max a LCB T c p a q . // LCB Commitment

In EOCP-UG, we use the same UCB exploration and LCB best action identification strategies as in the pre-determined stopping time setting. The only difference compared to EOCP comes from the new stopping rule based on the number of pulls N t p a q for each action. Specifically, the exploration ends if there is an imbalanced fraction of N t p a q among all actions, that is, one action has l times more pulls than all other actions in previous rounds. Here, l is the exploration function and we will select l to be slightly larger than log T in later sections. The intuition of such a stopping criterion comes from the characteristics of UCB exploration, i.e., as round t increases, the algorithm will slowly adapt to choosing the optimal action more often. When the fraction between the number of pulls for the optimal action and any sub-optimal action is large enough, the optimal action will be identifiable. Note that in action identification, we can simply choose the action that has the largest number of pulls N T c p a q when we stop, and obtain exactly the same performance guarantees. However, to keep it consistent with the pre-determined stopping time setting, we use the LCB commitment.

4 Main Results

In this section, we assume the distributions t ν 1 , ¨ ¨ ¨ , ν A u come from a Gaussian family, where ν a associated with action a follows a Gaussian distribution with mean µ a and unit variance, i.e., ν a ' N p µ a , 1 q . The results in this section can be easily generalized to sub-Gaussian distributions. The key idea towards the optimal trade-off between controlling regret and identifying the optimal action is inspired by the inflated bonus trick from [24], where we will choose the exploration function l to be slightly larger than the log T used in vanilla UCB algorithms to encourage more exploration of sub-optimal arms, i.e., we will select l ' log T ` O p ? log T q .

4.1 Regret Optimality for Gaussian Bandits with Pre-Determined Stopping Time

In Theorem. 1, we present the theoretical regret performance guarantee of the EOCP algorithm:

Theorem 1 Let l ' log p T q ` O p ? log T q and when T is large enough, the expected regret of the EOCP algorithm in Algorithm. 1 with pre-determined stopping time can be upper-bounded by:

A direct asymptotic bound can be obtained from Theorem. 1, i.e.,

Comparing it to Eq. (1), it is clear that EOCP is asymptotically regret optimal in Gaussian bandits. The commitment time and confidence guarantees can be extracted from the setup of Algorithm. 1 itself and the proof of Theorem. 1. Recall that in Algorithm. 1, we pre-determined the length of exploration T c to be O p ∆ ´ 2 lb l q . The following corollary characterizes these parts of theoretical performance:

Corollary 1 Let l ' log p T q ` O p ? log T q , the expected commitment time for EOCP in Algorithm. 1 is given by

and the confidence level is O p T ´ 1 q .

The complete proofs of Theorem. 1 and Corollary. 1 are provided in the supplementary material. In order to upper bound the cumulative regret of T rounds, we divide the total regret into the regret accumulated in exploration and the regret accumulated in commitment.

Bounding Regret from Exploration: To bound the regret accumulated in exploration and since we use a variant of UCB exploration, we follow the standard procedure of proofs for UCB algorithms, e.g., proof of Theorem 8 from. [22]. Then, this procedure results in an order O p l q dominating regret term, which is O p log T q by the choice of our exploration function. Through carefully applying any-time concentration inequalities, we are able to show that the constant in front of this dominating regret term is exactly the constant we obtained in Theorem. 1.

Bounding Regret from Committing to the Wrong Action: As for the regret accumulated from commitment, the key is to prove the O p T ´ 1 q confidence level upper bound presented in Corollary. 1. We follow a procedure similar to the proof of Theorem 1 in [2] to utilize the adaptivity of UCB exploration and the pessimistic LCB commitment. We first show that with high probability, the number of pulls N t p a q for any sub-optimal actions in the exploration phase is upper-bounded. This is because after a certain number of pulls, the uncertainty bonus b t ´ 1 p a q for any sub-optimal action will be so small that the upper confidence bound UCB t ´ 1 p a q cannot be larger than µ a ˚ , thus less than the upper confidence bound of the optimal action. Therefore, sub-optimal actions will not be chosen in future rounds. After our carefully designed T c , we make sure that the optimal action has the largest number of pulls N T c p a q among all actions, thus its bonus is so small so that its lower confidence bound LCB T c p a ˚ q is larger than the expectations µ a of any sub-optimal action a , and thus larger than the lower confidence bound of other actions. So with high probability, we will commit to the optimal action. Then, the O p T ´ 1 q confidence level will provide us with a constant regret in commitment, and the overall dominating regret comes from exploration.

Combining both bounds, we are able to show the regret performance upper bound in Theorem. 1.

4.2 Regret Optimality for Gaussian Bandits with Adaptive Stopping Time

We present the regret performance of EOCP-UG in Theorem. 2. Compared to Theorem. 1, EOCP-UG has exactly the same regret performance guarantees as EOCP even without the knowledge of a lower bound ∆ lb on the minimum reward gap. This implies that EOCP-UG adapts to the reward gap.

Theorem 2 Let l ' log p T q ` O p ? log T q and when T is large enough, the expected regret of the EOCP-UG algorithm in Algorithm. 2 with adaptive stopping time is upper-bounded by:

We can also directly derive an asymptotic regret bound which shows that EOCP-UG is regret optimal in Gaussian bandits:

With adaptive stopping, the sample complexity until the commitment is not a pre-determined value. However, we can still extract similar guarantees along with the confidence level from the proof of Theorem. 2. We present these results in the following corollary:

Corollary 2 Let l ' log p T q ` O p ? log T q , the sample complexity until commitment for EOCP -UG algorithm in Algorithm. 2 is upper-bounded by:

and the confidence level is upper bounded by O p T ´ 1 q .

Proof Roadmap: The complete proofs of Theorem. 2 and Corollary 2 are provided in the supplementary material. Compared to the proof of Theorem. 1, the major difference lies in bounding the regret of commitment. Similarly, we are required to bound the probability of committing to a sub-optimal action. We first show that when the agent stops exploration according to Line 2 of Algorithm. 2, the action that has the maximum number of pulls is the optimal action with high probability. Then, we show that under this event we will commit to the optimal action if we use LCB commitment.

Magic Choice of Exploration Function: Bounding the regret in both exploration and commitment requires a delicate analysis with any time concentration inequalities. Our choice of exploration function l plays an important role which manages the trade-off between low-regret exploration and high-probability optimal action commitment. If the exploration function is too large, i.e., if l ' 2 log T , the regret in exploration will not be optimal. If the exploration function is too small, i.e., if l ' log T , the probability of committing to the wrong action can not be bounded by O p T ´ 1 q . Our magic choice of exploration function l achieves the best of both worlds.

Loss of SCC from Unknown Gap: Even though we have the same regret performance, the guarantee for commitment time is O p log 2 T q which is worse than O p log T q in the pre-determined setting. So, is this order fundamental with adaptive stopping time? In the next section, we provide the answer by investigating the theoretical limits of commitment time for asymptotic regret optimal algorithms.

4.3 Fundamental Limits of Sample Complexity until Commitment

In order to answer the question regarding the fundamental sample complexity until commitment for regret optimal algorithms with both pre-determined and adaptive stopping times, we consider a simplified Gaussian bandit model where there are only 2 arms. The reward gap between the two actions is ∆ , so ∆ min ' ∆ . From Eq. (1), it is clear that the optimal regret of any algorithm is asymptotically 2∆ ´ 1 log p T q , so the set of all regret optimal algorithms is characterized by:

Furthermore, for each regret optimal algorithm, we say it has c -logarithm regret violation if there exists a constant c P p 0 , 1 q (choose the minimum c if there exists multiple) such that the following inequality is satisfied when T is large enough:

ˇ

ˇ

If an algorithm has c -logarithm regret violation, the largest additional lower order term in its finitetime regret bound should be O p log c T q . On the other hand, it also characterizes the convergence rate of the regret to its asymptote. For example, the vanilla UCB algorithm has at most 1 { 2 -logarithm regret violation, and our EOCP and EOCP-UG algorithms both have at most 1 { 2 -logarithm regret violation [22, Theorem. 8]. We then provide the following theorem to characterize the fundamental commitment time limits for algorithms in Π RO :

Theorem 3 (Information-Theoretic Limits of SCC ) Consider a 2 -armed Gaussian bandits, for any asymptotically regret optimal algorithm Alg which has c -logarithm regret violation, in order to guarantee O p T ´ 1 q confidence level, the sample complexity until commitment with pre-determined stopping time is lower bounded by

and with adaptive stopping time, the sample complexity until commitment is lower-bounded by:

Proof Roadmap: The proof is provided in the supplementary material. The proof idea relies on the well-known 'transportation' lemma [32, Lemma. 1] originally derived to prove the theoretical limits of best arm identification algorithms. This lemma characterizes the expected number of pulls for each action by hypothesis testing between the original bandit problem and another bandit instance with a different optimal action. Then by finding a proper bandit instance and combining the lemma with regret optimal algorithms, we will be able to prove Theorem. 3 for both settings.

Sample Optimality: It is shown by Corollary. 1 and Theorem. 3 together that our proposed EOCP algorithm achieves the optimal commitment time with O p log T q with pre-determined stopping time if the reward gap ∆ is known a priori. However, with adaptive stopping time, our EOCP-UG algorithm has 1 { 2 -logarithm regret violation and O p log 2 T q commitment time which is larger than the O p log 1 . 5 T q lower bound indicated by Theorem. 3. Even though EOCP-UG is not shown to be exactly sample optimal, we conjure that the performance gap comes from our analysis techniques which makes one of the bounds (maybe both) not tight. Whether EOCP-UG is indeed sample optimal, and if not, how to design regret optimal algorithms with O p log 1 . 5 T q commitment time remains open.

5 Regret Optimality for General Bandits

Even though EOCP is applicable in sub-Gaussian bandits, it is not regret optimal any more beyond Gaussian bandits. This can be seen by comparing Theorem. 1 and the fundamental regret limit (1) with Pinsker's inequality, and it is clear that the lower bound is smaller than our upper bound asymptotically beyond Gaussian bandits. To close this gap, we propose an improved algorithm called KL-EOCP which is provably regret optimal in general bandits.

Natural Exponential Family: We assume the reward distributions of each action belong to a natural exponential family, i.e., P ' tp ν θ q θ P Θ : dν θ { dξ ' exp p θx ´ b p θ qq h p x qu , where Θ Ă R is the set of all parameters θ such that the expectation µ is positive and bounded, i.e., µ P r 0 , 1 s . ξ is some reference measure on R and b : Θ Ñ R is a convex twice differentiable function. This distribution ν θ can also be parameterized by its expectation µ ' b 1 p θ q , the derivative of b p¨q , and for every µ we denote by ν µ the unique distribution in P with expectation µ and by θ µ its corresponding parameter. Gaussian distribution with unit variance is an example of this family. Moreover, the Kullback-Leibler divergence from ν θ 1 to ν θ 2 (with a little abuse of notation) can be expressed as [12]:

The set of exponential family bandit models ν ' p ν θ 1 , ¨ ¨ ¨ , ν θ A q can be characterized by the expectations of the actions µ ' p µ 1 , ¨ ¨ ¨ , µ A q . We assume for all λ P R , and θ P Θ the moment generating function M ν θ p λ q ' E ν θ r exp p λW qs for the distribution ν θ is well-defined and is finite.

Algorithm: Analog to ∆ lb in Algorithm. 1, the KL-EOCP algorithm requires the knowledge of a strictly positive lower bound KL lb on the 'minimum KL divergence', denoted as KL min , which captures the minimum reward distribution gap (distance) between the optimal action and any suboptimal action. Considering the asymmetricity of the KL divergence, we define KL min as follows:

where µ 1 a P p µ a , µ a ˚ q such that 4 KL p µ 1 a , µ a ˚ q ' KL p µ a , µ a ˚ q . The term 4 KL p µ 1 a , µ a q reflects the skew of KL divergence when the two distributions are switched. A simple lower bound to KL min can be easily computed given a lower bound ∆ lb on the minimum reward gap ∆ min with the expression of the exponential family, while our KL-EOCP algorithm can operate with any lower bound KL lb .

The KL-EOCP algorithm is summarized in Algorithm. 3. It designs the UCB and LCB bonuses based on the KL divergence of the reward distributions for all actions. In general, these designs would lead to smaller confidence intervals with the same O p T ´ 1 q concentration guarantees as first adopted in the KL-UCB algorithm proposed in [12]. If the lower bound information KL lb is not known a priori, we can combine Algorithm. 3 with the adaptive stopping time of EOCP-UG to deal with this setting.

Algorithm 3 KL-EOCP with Pre-Determined Stopping Time

Require: Exploration function l ; Lower bound KL lb on the minimum KL divergence.

1: Let T c ' 4 Al KL 2 ` A be the length of the exploration phase.

lb 2: Initialize by pulling each arm a once.

3: for t ' A ` 1 : T c do

4: Set upper confidence bound:

5: Take action A t ' arg max a UCB t ´ 1 p a q . // UCB Exploration

6: end for

7: Set lower confidence bound:

8: For t P r T c ` 1 , T s , commit to action ˆ a ' arg max a LCB T c p a q . // LCB Commitment

Regret Optimality in General Bandits: The theoretical regret performance of the KL-EOCP is summarized in the following Theorem. To the best of our knowledge, it is the first result achieving asymptotic regret optimality in general bandit problems with commitment.

Theorem 4 Let l ' log p T q ` O p ? log T q , when T is large enough and the reward distributions ν of each action belong to the same natural exponential family, the expected regret of KL-EOCP in Algorithm. 3 is upper-bounded by:

Therefore, it is clear that we can derive an asymptotic upper bound as follows:

which exactly matches the regret limit in Eq. (1). The complete proof of Theorem. 4 is provided in the supplementary materials. Even though the proof roadmap is similar to the proof of Theorem. 1, the major difference comes from the use of a tighter concentration lemma modified from [12, Theorem. 11] which captures the low probability event when the KL divergence of the empirical mean is far away from its expectation. The sample complexity until commitment and confidence level guarantees can also be extracted from the proof. Specifically, SCC KL -EOCP µ p T q ' O p KL ´ 1 lb log T q and the confidence level is upper bounded by O p T ´ 1 q .

6 Numerical Experiments

In this section, we study the empirical performance of our proposed EOCP algorithm with variants compared to existing algorithms in the literature, including BAI-ETC [22], UCB [10], KL-UCB [12], and DETC [23] in both Gaussian and Bernoulli bandit settings. In this section, we only test the algorithms on a two-armed bandit problem because some baselines are only applicable in two-armed settings. The performance of our algorithms in bandit models with multiple arms is demonstrated in Appendix. E. In the Gaussian setting, we test all the algorithms with distributions N p µ i , 1 q for arm

10 0

10 1

10 2

10 3

10 4

10 5

10 6

Number of Rounds

0

10

20

30

40

50

60

70

80

Regret

EOCP

EOCP-UG

UCB

BAI-ETC

DETC

(a) Gaussian Bandits

10 0

10 1

10 2

10 3

10 4

10 5

Number of Rounds

0

10

20

30

40

50

60

Regret

EOCP

EOCP-UG

KL-EOCP

UCB

KL-UCB

BAI-ETC

DETC

i ' 1 , 2 with a total of 10 6 rounds, and in the Bernoulli bandit setting, we test the algorithms with distribution Ber p µ i q for arm i ' 1 , 2 with a total of 10 5 rounds. We set µ 1 ' 0 . 7 and µ 2 ' 0 . 2 , so the gap between the arms is ∆ ' 0 . 5 . The results are averaged over 10 5 iterations and in Fig. 1.

In the Gaussian bandit setting, it shows that both BAI-ETC and DETC algorithms exhibit unsatisfactorily high regret. On the contrary, our proposed algorithms EOCP and EOCP-UG have lower final regret even compared to the popular UCB algorithm, surprisingly. Even though our algorithms use a larger exploration function than the vanilla UCB algorithm, and accumulate regret more quickly in exploration, i.e., approximately the first 1000 rounds, it allows us to commit to the optimal action. Our algorithms almost find the optimal action in all simulated traces which gives rise to the very slowly-increasing behavior of regret in the commitment phase. This phenomenon coincides with our theoretical analysis which shows that the regret in commitment is O p 1 q . However, the UCB algorithm continues to explore sub-optimal actions, so its regret continues to grow when the EOCP algorithm has already committed to the optimal action. Numerically, the EOCP algorithm reduces 20% of the final regret compared to UCB through early commitment, and we conjure that preventing the 'over-exploration' phenomenon of the UCB algorithm is behind the reason for such empirical regret reduction. Comparing the commitment time of EOCP-UG and EOCP, we can see that both algorithms stop exploration at approximately 1000 rounds. This means that not knowing the gap information won't harm the empirical sample complexity until commitment too much, which in turn may imply that our theoretical analysis of commitment time upper bound in Corollary. 2 is not tight enough. The same trend can be witnessed in the results of the Bernoulli bandits. However, in Bernoulli bandits, KL-UCB and KL-EOCP algorithms have a much better regret performance than other algorithms, which shows that knowledge of the reward distribution family improves the performance significantly.

7 Conclusion

We studied ROBAI which intends to both minimize regret and commit to the optimal action. We proposed EOCP with variants, which combine UCB exploration and LCB commitment with novel stopping criteria in both pre-determined and adaptive settings. We showed that both EOCP and EOCPUG are regret asymptotic optimal in Gaussian bandits with O p log T q and O p log 2 T q commitment time respectively, almost matching the theoretical limits we derived. For general bandits, we proposed KL-EOCP which is provably regret optimal. Numerical experiments confirmed the superiority of our algorithms and revealed the 'over-exploration' phenomenon of UCB algorithms.

Acknowledgments and Disclosure of Funding

The work of Qining Zhang and Lei Ying is supported in part by NSF under grants 2112471, 2134081, 2207548, and 222897.

References

William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika , 25(3-4):285-294, 1933.

Zixian Yang, Xin Liu, and Lei Ying. Exploration. exploitation, and engagement in multi-armed bandits with abandonment. In 2022 58th Annual Allerton Conference on Communication, Control, and Computing (Allerton) , pages 1-2, 2022.

Pushmeet Kohli, Mahyar Salek, and Greg Stoddard. A fast bandit algorithm for recommendation to users with heterogenous tastes. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 27, pages 1135-1141, 2013.

Chunqiu Zeng, Qing Wang, Shekoofeh Mokhtari, and Tao Li. Online context-aware recommendation with time varying multi-armed bandit. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining , pages 2025-2034, 2016.

Xin Liu, Bin Li, Pengyi Shi, and Lei Ying. Pond: Pessimistic-optimistic online dispatching. arXiv preprint arXiv:2010.09995 , 2020.

Jiatai Huang, Leana Golubchik, and Longbo Huang. Queue scheduling with adversarial bandit learning. arXiv preprint arXiv:2303.01745 , 2023.

Zixian Yang, R Srikant, and Lei Ying. Maxweight with discounted ucb: A provably stable scheduling policy for nonstationary multi-server systems with unknown statistics. arXiv preprint arXiv:2209.01126 , 2022.

Sofía S Villar, Jack Bowden, and James Wason. Multi-armed bandit models for the optimal design of clinical trials: benefits and challenges. Statistical science: a review journal of the Institute of Mathematical Statistics , 30(2):199, 2015.

Maryam Aziz, Emilie Kaufmann, and Marie-Karelle Riviere. On multi-armed bandit designs for dose-finding clinical trials. The Journal of Machine Learning Research , 22(1):686-723, 2021.

Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning , 47:235-256, 2002.

Shipra Agrawal and Navin Goyal. Analysis of thompson sampling for the multi-armed bandit problem. In Conference on learning theory , pages 39-1. JMLR Workshop and Conference Proceedings, 2012.

Aurélien Garivier and Olivier Cappé. The kl-ucb algorithm for bounded stochastic bandits and beyond. In Proceedings of the 24th annual conference on learning theory , pages 359-376. JMLR Workshop and Conference Proceedings, 2011.

Tze Leung Lai, Herbert Robbins, et al. Asymptotically efficient adaptive allocation rules. Advances in applied mathematics , 6(1):4-22, 1985.

Michael N Katehakis and Herbert Robbins. Sequential choice from several populations. Proceedings of the National Academy of Sciences , 92(19):8584-8585, 1995.

Emilie Kaufmann, Nathaniel Korda, and Rémi Munos. Thompson sampling: An asymptotically optimal finite-time analysis. In Algorithmic Learning Theory: 23rd International Conference, ALT 2012, Lyon, France, October 29-31, 2012. Proceedings 23 , pages 199-213. Springer, 2012.

Mohammad Gheshlaghi Azar, Ian Osband, and Rémi Munos. Minimax regret bounds for reinforcement learning. In International Conference on Machine Learning , pages 263-272. PMLR, 2017.

Chi Jin, Zeyuan Allen-Zhu, Sebastien Bubeck, and Michael I Jordan. Is q-learning provably efficient? Advances in neural information processing systems , 31, 2018.

Susan D Phillips, Nicholas J Pazienza, and Donna J Walsh. Decision making styles and progress in occupational decision making. Journal of Vocational Behavior , 25(1):96-105, 1984.

TJJOPS Higuchi. Mechanism of sustained-action medication. theoretical analysis of rate of release of solid drugs dispersed in solid matrices. Journal of pharmaceutical sciences , 52(12):1145-1149, 1963.

Qining Zhang, Honghao Wei, Weina Wang, and Lei Ying. On low-complexity quickest intervention of mutated diffusion processes through local approximation. In Proceedings of the Twenty-Third International Symposium on Theory, Algorithmic Foundations, and Protocol Design for Mobile Networks and Mobile Computing , pages 141-150, 2022.

Jeremy J Siegel. Stocks for the long run: The definitive guide to financial market returns & long-term investment strategies . McGraw-Hill Education, 2021.

Aurélien Garivier, Tor Lattimore, and Emilie Kaufmann. On explore-then-commit strategies. Advances in Neural Information Processing Systems , 29, 2016.

Tianyuan Jin, Pan Xu, Xiaokui Xiao, and Quanquan Gu. Double explore-then-commit: Asymptotic optimality and beyond. In Conference on Learning Theory , pages 2584-2633. PMLR, 2021.

Rémy Degenne, Thomas Nedelec, Clément Calauzènes, and Vianney Perchet. Bridging the gap between regret minimization and best arm identification, with application to a/b tests. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 1988-1996. PMLR, 2019.

Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. Bridging offline reinforcement learning and imitation learning: A tale of pessimism. Advances in Neural Information Processing Systems , 34:11702-11716, 2021.

Gen Li, Cong Ma, and Nathan Srebro. Pessimism for offline linear contextual bandits using lp confidence sets. arXiv preprint arXiv:2205.10671 , 2022.

Chenjun Xiao, Yifan Wu, Jincheng Mei, Bo Dai, Tor Lattimore, Lihong Li, Csaba Szepesvari, and Dale Schuurmans. On the optimality of batch policy optimization algorithms. In International Conference on Machine Learning , pages 11362-11371. PMLR, 2021.

Gen Li, Laixi Shi, Yuxin Chen, Yuejie Chi, and Yuting Wei. Settling the sample complexity of model-based offline reinforcement learning. arXiv preprint arXiv:2204.05275 , 2022.

Laixi Shi, Gen Li, Yuting Wei, Yuxin Chen, and Yuejie Chi. Pessimistic q-learning for offline reinforcement learning: Towards optimal sample complexity. arXiv preprint arXiv:2202.13890 , 2022.

Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline rl? In International Conference on Machine Learning , pages 5084-5096. PMLR, 2021.

Dan Siroker and Pete Koomen. A/B testing: The most powerful way to turn clicks into customers . John Wiley & Sons, 2015.

Emilie Kaufmann, Olivier Cappé, and Aurélien Garivier. On the complexity of best arm identification in multi-armed bandit models. Journal of Machine Learning Research , 17:1-42, 2016.

Emilie Kaufmann, Olivier Cappé, and Aurélien Garivier. On bayesian upper confidence bounds for bandit problems. In Artificial intelligence and statistics , pages 592-600. PMLR, 2012.

Daniel J Russo, Benjamin Van Roy, Abbas Kazerouni, Ian Osband, Zheng Wen, et al. A tutorial on thompson sampling. Foundations and Trends® in Machine Learning , 11(1):1-96, 2018.

Daniel Russo and Benjamin Van Roy. An information-theoretic analysis of thompson sampling. The Journal of Machine Learning Research , 17(1):2442-2471, 2016.

Aurélien Garivier and Emilie Kaufmann. Optimal best arm identification with fixed confidence. In Conference on Learning Theory , pages 998-1027. PMLR, 2016.

Rayadurgam Srikant and Lei Ying. Communication networks: an optimization, control, and stochastic networks perspective . Cambridge University Press, 2013.

Tor Lattimore and Csaba Szepesvári. Bandit algorithms . Cambridge University Press, 2020.

Guanyu Nie, Mridul Agarwal, Abhishek Kumar Umrawal, Vaneet Aggarwal, and Christopher John Quinn. An explore-then-commit algorithm for submodular maximization under full-bandit feedback. In Uncertainty in Artificial Intelligence , pages 1541-1551. PMLR, 2022.

Ali Yekkehkhany, Ebrahim Arian, Mohammad Hajiesmaili, and Rakesh Nagi. Risk-averse explore-then-commit algorithms for finite-time bandits. In 2019 IEEE 58th Conference on Decision and Control (CDC) , pages 8441-8446. IEEE, 2019.

Ali Yekkehkhany, Ebrahim Arian, Rakesh Nagi, and Ilan Shomorony. A cost-based analysis for risk-averse explore-then-commit finite-time bandits. IISE Transactions , 53(10):1094-1108, 2021.

Eyal Even-Dar, Shie Mannor, Yishay Mansour, and Sridhar Mahadevan. Action elimination and stopping conditions for the multi-armed bandit and reinforcement learning problems. Journal of machine learning research , 7(6), 2006.

A Related Works

In this section, we provide a more detailed review of related works. We first review two classic problem formulations of the multi-armed bandit model: the regret minimization problem and the best arm identification problem. Then we review previous works on explore-then-commit algorithms which takes commitment into account. Finally, we review the offline stochastic bandit literature which motivated our choice of pessimistic principle in action identification.

A.1 Regret Minimization

The theoretical limits of regret minimization have been revealed by [13, 14], which shows that the expected regret of any algorithm is lower bounded when horizon T approaches infinity:

where KL p¨ , ¨q denotes the Kullback-Leibler divergence between two distributions. Based on the asymptotic lower bound, we say an algorithm is asymptotically regret optimal if its regret performance achieves the regret lower bound asymptotically. Two sets of algorithms prevail in the regret minimization literature. One is the family of UCB algorithms [10, 12, 33], which reflects the principal of optimism in action selection to encourage exploration. To be specific, the UCB algorithms will select the action which has the largest upper confidence bound of reward estimation. This upper confidence bound represents the highest possible expected reward given the samples collected from previous rounds under a high probability event. Usually the additional bonus from the empirical mean to the upper confidence bound for a specific action decreases as the number of pulls increases. Therefore, actions which have not been tried frequently in previous rounds will have larger bonus. This trait encourages exploration. By designing the bonus carefully, one can find the optimal trade-off between exploration and exploitation. The other set of prevailing algorithms is the family of Thompson Sampling Algorithms [1, 34, 11, 15, 35]. The TS algorithm assumes each action is associated with a posterior distribution given the reward feedback from previous rounds. Then the agent will collect one virtual sample from each posterior distribution and choose the action which has the largest virtual sample. As the number of pulls grows, the posterior distribution will be more and more concentrated around the expectation so the virtual sample will also be closer to its expectation. On the other hand, the remaining randomness encourages exploration of other under-explored actions. However, both algorithms do not commit to a single action because their action selection policies have to be re-evaluated at each round based on new observations.

A.2 Best Arm Identification

The best arm identification algorithms consist of three components: an action sampling rule deciding which action to choose at each round, a stopping rule deciding a time τ to stop collecting new samples, and a decision rule which outputs a best action candidate ˆ a . We call an algorithm δ -PAC if the probability of outputting a sub-optimal action is less than δ , i.e., P µ p ˆ a ‰ 1 q ď δ . Here, δ is also called the confidence level. The performance of best arm identification algorithms is measured by both the sample complexity until identifying the optimal action and the confidence level. The theoretical limits for best arm identification algorithms are also well studied in [36, 32], which shows that any δ -PAC algorithm would incur at least O p log δ ´ 1 q sample complexity asymptotically. The constant in front of the logarithmic term depends on the bandit instance, i.e., the reward expectation and distribution of every action. Based on the this lower bound, the authors of [36] proposed the Track and Stop (TAS) algorithm which proved to be asymptotic optimal if the distribution associated with each action is from a single parameter natural exponential family. The TAS algorithm estimates the expectation of each action and at the same time calculates the optimal proportion of pulls for each action so that the optimal action is identifiable from the sub-optimal ones. Then, it designs a feedback control dynamic to track this proportion, similar to the Max-weight dynamic in queueing systems [37]. The stopping criterion of TAS is based on a generalized likelihood test. When the optimal action is identifiable from sub-optimal ones, the optimal action candidate ˆ a is chosen to be the action which has the largest empirical reward mean.

A.3 Explore-Then-Commit Algorithms

A natural bridge between regret minimization and best arm identification problems, which also takes into account action commitment, is the explore-then-commit algorithms [38, 22, 23, 39, 40, 41].Researchers have long been hoping that such algorithms will achieve the best of both worlds: maintaining low regret and identifying the optimal action quickly. In these algorithms, a clear separation of exploration phase and exploitation phase exists. At each round, the agent will only make action selection decisions based on the samples collected from the exploration phase, and commitment to a single action during the exploitation phase is required. No samples from the exploitation phase can be utilized although the agent may determine the length of both phases. It is clear that explore-then-commit algorithms can be easily designed from best arm identification algorithms, i.e., one would first run the best arm identification algorithm in the exploration phase, and then commit to the action ˆ a found by the algorithm. However, it is shown in [22] that these type of BAI-ETC algorithms is essentially regret sub-optimal. To be specific, the regret lower bound of such algorithms is asymptotically twice as large as the upper bound of optimal regret minimization algorithms such as UCB and Thompson Sampling even with careful tuning. A recent work [23] provides a double explore-then-commit algorithm called DETC which is shown to be asymptotically regret optimal. But the algorithm itself is very complex and requires multiple stages of exploration and exploitation. This trait makes its sample complexity to identify the optimal action large, i.e., Ω p log 2 T q . Moreover, in empirical studies, the DETC algorithm incurs very large regret which is no match for the vanilla UCB algorithm. Another work [24] proposes an exploration algorithm with inflated UCB, which can be naturally adapted to an ETC algorithm. Even though the sample complexity is order optimal, i.e., O p log T q , the regret performance is essentially sub-optimal due to the inflation of UCB bonus.

A.4 Offline Stochastic Bandits and Reinforcement Learning

Our action identification policy is inspired by the principle of pessimism widely adopted in offline stochastic bandit problems [26, 27, 25] and reinforcement learning problems [28, 30, 29]. In offline bandit problems, the agent is not allowed to interact with the environment at will. Instead, it is provided with a training dataset which contains action and reward pairs collected from the same bandit problem. Based on this dataset, the agent is asked to choose the optimal action. It is shown in [25] that greedily selecting the action with largest empirical mean would fail to produce the optimal action in some bandit problems. This failure results from the randomness of samples in the dataset and the imbalanced number of samples for each action. Instead, choosing the action with largest lower confidence bound to combat the imbalanced uncertainty between estimations of different actions leads to better performance as justified in [27, 25]. Similar to the UCB algorithm, the penalty term between the empirical mean and the lower confidence bound for each action decreases as the number of pulls increases, so the action which has not been tried frequently will suffer large penalty. In this way, the agent will avoid selecting an action which has large uncertainty to enhance stability. Our proposed algorithm incorporates the idea of pessimism in action identification.

B Proofs of Main Results for EOCP and EOCP-UG

In this section, we provide the proofs of main results presented in Section. 4. Throughout the proof section, we let W a,i be the i -th sample from pulling arm a the i -th time. Let ¯ r a,s be the empirical mean of arm a after it has been pulled s times. Before proving the theorems, we provide several lemmas which gathers specific large deviation results useful in our analyses. Then we prove the regret performance based on the concentration lemmas.

B.1 Concentration Inequalities

We first present two concentration lemmas which characterize the sum and mean of empirical realizations of independent sub-Gaussian random variables as follows:

Lemma 1 (Theorem 9.2 in [38]) Let W 1 , W 2 , ¨ ¨ ¨ , W T be a sequence of independent σ -subgaussian random variables with E r W 1 s ' 0 . Then, for any δ ą 0 , we have:

Lemma 2 (Lemma C.3 in [23]) Let T 1 ď T 2 ď T be two real numbers in R ` . Let W 1 , W 2 , ¨ ¨ ¨ , W T be a sequence of identically and independently distributed random variable according to a σ -subgaussian distribution with E r W 1 s ' 0 . Then, for any δ ą 0 , we have:

The proofs of aforementioned concentration lemmas can be found in the references respectively. Now we present the concentration results for our design of confidence bonuses as follows:

Lemma 3 Let W 1 , W 2 , ¨ ¨ ¨ , W T be identically and independently distributed 1-sub-Gaussian random variables with E r W 1 s ' 0 . Let T 1 ď T 2 ď T , then the following holds:

(4a)

The proof of the Lemma. 3 will be delayed to Sec. D.

B.2 Proof of Regret Optimality for EOCP with Pre-Determined Stopping Time

We prove the following Theorem which characterizes the finite-time performance of Algorithm. 1. Theorem. 1 can be directly derived.

Theorem 5 If l ' log p T q ` 4 a 2 log p T q and T ě max t 16 , A, 16 l ∆ ´ 2 lb u , the expected regret of the EOCP algorithm in Algorithm. 1 with pre-determined stopping time is upper bounded by:

Remark: The asymptotic regret upper bound is clear from Theorem.5 by letting T increases to infinity, i.e.,

Proof. Without loss of generality, assume action 1 is the unique optimal action. From the regret decomposition lemma [38, Lemma 4.5], we can decompose the regret of Algorithm. 1 to the number of pulls for each sub-optimal arm as follows:

Then, the key to bound the total regret is to bound the number of pulls for each sub-optimal arms. Since our EOCP algorithm has a clear separation of exploration and exploitation phases, so for any sub-optimal action a , we can bound its pulls in different phases as follows:

where T c is the end of exploration phase and ˆ a is the action that the algorithm commits to. Notice that the bound on I 2 gives the confidence level result provided in Corollary. 1.

Bounding I 1 : Wefirst bound I 1 , which is similar to the proof of bounding the regret for the traditional UCB algorithm [12, 10, 22]. We have:

Notice that A t ' a indicates UCB t ´ 1 p 1 q ă UCB t ´ 1 p a q due to the dynamic of UCB exploration, we can bound the last term as:

Event t UCB t ´ 1 p 1 q ă µ 1 u means that at round t , the mean estimation ¯ r t ´ 1 p 1 q of the optimal arm from previous pulls is lower than µ 1 ´ b t ´ 1 p 1 q , which incurs a large deviation, so we can bound this event with a union over all rounds in the exploration phase:

which doesn't depend on round number t any more, so we have:

It is worth-noting that if there is no pulls for arm 1 between a time interval, its empirical mean and number of pulls will remain the same. So we have:

Notice that N T c p 1 q ď T c ´ A ` 1 , so we can bound the probability as follows:

where the last inequality uses Eq. (4a) of Lemma 3 and the fact that W 1 ,i ´ µ 1 is 1 -sub-Gaussian with zero mean. Notice that ? log T ě log log T when T is large and we have:

So putting the above bound back to the bound of A 2 , we have:

where the last inequality is due to the fact that when T ě 3 , ? log T ď log T . Next, we attempt to bound the middle term A 1 as follows:

Notice that the number of pulls N t p a q will only increase by 1 every time there is new pull, i.e., when A t ' a . Otherwise, the term inside summation is 0 . So instead of counting on the time step t , we can count over the number of pulls over arm a as follows:

where the last inequality is due to N T c p a q ď T c . Then, by Eq. (4c) of Lemma. 3, we have:

where the first inequality is due to the fact that W a,i ´ µ a is 1 -sub-Gaussian with zero mean, and the last inequality is due to ? 2 log T ď log T when T ě 9 . Therefore, combining the bounds on A 1 and A 2 , we can bound I 1 as follows:

Bounding I 2 : The bound on I 2 gives the confidence level result provided in Corollary. 1. After the exploration phase, recall that we select the arm with the largest lower confidence bound to commit to. The idea of bounding I 2 is very similar to the regret bound in [2] where since we use UCB to explore in the exploration phase, the optimal arm should be pulled very often such that its bonus becomes very small when exploration phase ends. Then selecting the arm with largest LCB will ensure we select the optimal arm with high probability. This is because the LCB of the optimal arm is larger than the true means of any sub-optimal arms, and the true means of sub-optimal arms are larger than their respective LCB, both with high probability. We can first bound the probability of selecting a sub-optimal arm a as:

We first bound the term B 2 which states that during the first exploration phase with UCB exploration, the optimal arm is under-pulled, which means that one of the sub-optimal arms have been pulled with larger number of times. Therefore, we have:

Recall that T c ' 8 Al ∆ 2 lb ` A , so we have:

Then by union bound, we have:

Consider a fixed sub-optimal arm a , then for each probability inside the summation, we have:

Notice that B 4 can be bounded from concentration lemma. First, we switch the count from time step t to the number of pulls for arm 1 since the empirical estimation ¯ r t ´ 1 p 1 q and count for pulls N t ´ 1 p 1 q won't change unless there is a new pull. Then, we will apply Eq. (4a) to the probability as follows:

where the last inequality is due to the fact that W 1 ,i ´ µ 1 is 1 -sub-Gaussian with zero mean. Similar to the procedure of bounding I 1 , notice that exp p l q ě T log 4 T , we have:

Q

U

On the other hand, let γ ' 8 l ∆ 2 lb , and we also change the count from time step t to the number of pulls for arm a . Then, B 3 can be expressed as follows:

Notice that γ ě 8 l ∆ 2 lb ě 8 l ∆ 2 a , so we have b 2 l γ ď b 2 l ∆ 2 a 8 l ' ∆ a 2 . Then B 3 can be bounded as follows:

where the first inequality uses Hoeffding's inequality and the second inequality uses the lower bound on γ mentioned above. The last inequality is due to the fact that exp p l q ě T log 4 T . So combine B 3 and B 4 together, we can bound B 2 as follows:

Next, we attempt to bound B 1 . Notice that B 1 can also be bounded as follows:

The term B 6 can be expressed by counting the number of pulls of arm a as follows:

Notice that p µ a ´ W a,i q is 1 -sub-Gaussian with zero mean, so we can apply concentration lemma Eq. (4a) from Lemma. (3) as follows:

where the last inequality is due to log T ě ? 2 log T and exp p l q ě T log 4 T when T ě 9 . Similarly, term B 5 can be expressed as:

First notice that when N T c p 1 q ą 8 l ∆ 2 lb , we have the bonus term b T c p 1 q ' b 2 l N T c p 1 q ď ∆ lb 2 ď ∆ a 2 . Then we have:

Then it is equivalent to count over the number of pulls for arm 1 :

Then, by the maximal concentration Eq. (3) from Lemma. 2, we can bound B 5 as follows:

¨

˛

Therefore, collecting the bounds for B 5 and B 6 , we have a bound for B 1 as follows:

And therefore from the bounds of B 1 and B 2 , we can bound term I 2 as follows:

This means the the confidence of selecting the wrong action to commit to is o p T ´ 1 q as indicated in Corollary. 1. Finally, putting the bounds on I 1 and I 2 together, we have:

Therefore, by the regret decomposition lemma, we can bound the total regret as follows:

B.3 Proof of Regret Optimality for EOCP-UG with Adaptive Stopping Time

We prove the following theorem which characterizes the finite-time performance of Algorithm. 2. Theorem. 2 can be derived directly.

Theorem 6 If l ' log p T q ` 4 a 2 log p T q and T ě max t 16 , A, 16 l ∆ ´ 2 min u , the regret of EOCP-UG algorithm in Algorithm. 2 with adaptive stopping time can be upper bounded as:

Remark: The asymptotic regret upper bound is clear from Theorem.6 by letting T increases to infinity, i.e.,

Proof. Without loss of generality, let action 1 be the unique optimal action. The first step for proving the regret performance is regret decomposition lemma [38, Lemma 4.5]. We also decompose the regret of Algorithm. 2 into the number of pulls for each sub-optimal arm as follows:

Then, for a specific sub-optimal arm a , we bound the number of pulls. Since our EOCP -UG algorithm has a clear separation of exploration and exploitation phases, we can bound the pulls in the two phases respectively. However, the unique characteristic of unknown gap scenario is we don't have a fixed end time of exploration phase. Recall that T c ď T is the stopping time that the exploration phase ends and ˆ a is the arm we choose for commitment, so we can decompose the number of pulls into two phases as:

Notice that upper bound of I 2 gives the confidence level result in Corollary. 2. We then bound the two terms I 1 and I 2 separately. In order to simplify the proof, we first prove a lemma which characterizes a high probability upper bound for the stopping time T c . This lemma also proves the sample complexity to commitment result in Corollary. 2. The proof of Lemma. 4 will be delayed.

Lemma 4 If l ' log p T q ` 4 a 2 log p T q and T ě max t 16 , A, 16 l ∆ ´ 2 min u , our stopping time T c for exploration of Algorithm. 2 is upper bounded with high probability:

Bounding I 1 : With the help of Lemma. 4, we have a high probability upper bound for the exploration phase. We let T u c ' ř a :∆ a ą 0 8 p l ` 1 q 2 ∆ 2 a ` A p l ` 2 q to be the high probability upper bound of our stopping time to end the exploration phase. Notice that N a p T c q ď T , we have:

Then, bounding I 3 is similar to bounding I 1 for Theorem. 5. We decompose I 3 as follows:

In order to bound A 1 and A 2 , we assume there is a virtual process that after the end time T c of exploration phase, it continues to select the arm with largest UCB and receive the corresponding reward until time T u c . This is only a virtual process used in our proof, while in reality our algorithm will stop exploration after stopping time T c . We will use E 1 and P 1 to denote the expectation and probability over this virtual process. Then, we can bound A 2 as follows:

where the first inequality is because A t ' a indicates UCB t ´ 1 p 1 q ă UCB t ´ 1 p a q due to the dynamic of UCB exploration. The second inequality is because E and E 1 are totally the same for the first T c time steps. This step also allows us to bound A 2 over the events on a different probability measure E 1 and P 1 . Event t UCB t ´ 1 p 1 q ă µ 1 u means that at time step t , the mean estimation ¯ r t ´ 1 p 1 q of the optimal arm from previous pulls is lower than µ 1 ´ b t ´ 1 p 1 q , which incurs a large deviation, so we can bound this event with a union over all time steps:

which doesn't depend on time step t any more, so we have:

If there is no pulls for arm 1 in a time interval, the empirical mean and number of pulls will remain the same. So instead of counting on the time steps, we can count the number of pulls for arm 1 :

Notice that N T u c p a q ď T u c ´ A ` 1 , so we can bound the probability as follows:

where the last inequality uses Eq. (4a) of Lemma 3 and the fact that W 1 ,i ´ µ 1 is 1 -sub-Gaussian with zero mean. Notice that exp p l q ě T log 4 T , so putting the above bound back to the bound of A 2 , we have:

where the first inequality is due to the fact that when T ě 3 , ? log T ď log T . Next, we attempt to bound A 1 as follows:

where the inequality is also due to the fact that the virtual and real processes are identical before time T c . Notice that the number of pulls N t p a q will only increase by 1 every time there is new pull, i.e., when A t ' a . Otherwise, the term inside summation is 0 . So instead of counting on the time step t , we can count over the number of pulls over arm a as follows:

where the last inequality is due to N T u c p a q ď T u c . Then, by Eq. (4c) of Lemma. 3, we have:

where the first inequality is due to the fact that W a,i ´ µ a is 1 -sub-Gaussian with zero mean, and the last inequality is due to ? 2 log T ď log T when T ě 9 . Therefore, combining the bounds on A 1 and A 2 , we can bound I 3 as follows:

Therefore, a similar bound can be established on I 1 as follows:

Bounding I 2 : Recall that our stopping criterion is when there exists an arm ˜ a whose number of pulls is significantly larger than other arms, i.e., N T c p ˜ a q ě l max a ‰ ˜ a N T c p a q . Therefore, its bonus b T c p ˜ a q should be very small compared to other arms. Also recall that we select the arm ˆ a with highest LCB to commit to, so the proof follows two steps. First we will show that with high probability the arm ˜ a with most number of pulls is the best arm. Then we will show that under this circumstance, the maximum LCB arm is also the best arm. Therefore, consider an arbitrary sub-optimal arm a :

Notice that if N T c p a q ě r lN T c p 1 q s ` 1 at the end time T c , we must have pulled arm a at time step T c . This also means that the number of pulls N T c ´ 1 p a q for arm a after time step T c ´ 1 is exactly r lN T c p 1 q s and and N T c ´ 1 p 1 q ' N T c p 1 q . It also means that arm a has the largest UCB. So we have:

Notice that we can separate the two random quantities with union bound as follows:

We first bound B 1 . Notice that we can switch from counting of time step t to count the number of pulls for arm a . It is clear that N t p a q ď T c ď T when t ď T c , so we have:

where the last inequality uses Eq. (4a) of Lemma. 3. Also notice that exp p l q ě T log 4 T , so we have:

Next, we bound the term B 2 , we can rearrange the terms inside B 2 as follows:

Denote α ' 1 ´ 2 ? l , and switch the count from time step t to the number of pulls for arm 1 , we can bound B 2 as follows:

When T is large enough, we have α 2 l ' l ´ 4 ? l ` 4 ě log T ` 4 , so we have:

Therefore, combining B 1 and B 2 together, we can bound the probability as follows:

Recall that arm 1 is the optimal arm. Therefore, by a union bound, we can characterize the probability that of event t ˜ a ‰ 1 u as follows:

Next, we investigate the probability of choosing the wrong arm ˆ a to commit to. Recall that for the arm which we commit to, we choose the one with the largest LCB. Consider any sub-optimal arm a , if we wrongly choose the arm ˆ a ' a , it means its LCB should be larger than the LCB of the optimal arm, which with high probability has the largest number of pulls. So, we have:

When ˜ a ' 1 , arm 1 has l times more pulls than arm a when the exploration phase stops, so we have:

Similarly, we separate the two random variables as follows:

For B 4 , notice that T c is a random stopping time, so we bound the probability over all time step when the number of pulls is larger than l as follows:

Since W 1 ,i ´ µ 1 is 1 -subgaussian, we can use Eq. (4a) from the concentration Lemma. 3 to bound B 4 as follows:

where the last inequality is due to ? 2 log T ď log T when T ě 9 and exp p l q ě T log 4 T . On the other hand, bounding B 3 is similar to the proof of probability upper bound regarding ˜ a . recall that α ' ´ 1 ´ 2 ? l ¯ , and notice that T c is a random variable, so we use a union over all possible arm pulls

to bound the event on time step T c as follows:

where the last inequality uses the second concentration inequality Eq. (4b) from Lemma. 3 and the fact that µ a ´ W a,i is 1 -subgaussian. Then when T is large enough, we have:

Combining B 3 and B 4 , we can bound the probability that we select the wrong arm as:

Therefore, we finally bound I 2 as follows:

Here, the bound on I 2 shows that the confidence level of our algorithm is O p T ´ 1 q indicated in Corollary. 2. Combining I 1 and I 2 , we can bound E r N a p T qs in finite time as follows:

So by the regret decomposition lemma, we can bound the regret performance as:

B.4 Proof of Lemma. 4 and Corollary. 2

Consider a specific sub-optimal arm a . We first show that it can only be pulled O p log T q with high probability during the exploration phase due to the dynamic of our exploration strategy, i.e., the UCB exploration strategy. To be specific, we intend to show the following inequality:

Since during the exploration phase, we select the arm with the largest UCB to explore, if there exists a time t which the number of pulls for action a is larger than 8 l ∆ 2 a ` 1 , which means that there exists a time t 1 P r A,t q , where at time t 1 ` 1 the number of pulls for previous rounds N t 1 p a q is exactly 8 l ∆ 2 a and arm a has the largest UCB. Therefore, we can bound the probability as:

Therefore, the UCB of arm a should be larger than the UCB of the optimal arm 1 , i.e., UCB t 1 p a q ě UCB t 1 p 1 q . So we can further derive an upper bound as follows:

Then, we can separate the two empirical means ¯ r R 8 l ∆2 a V ,a and ¯ r t 1 p 1 q with a union bound as follows:

d

The term A 2 can be easily bounded through Hoeffding's inequality as:

where the first inequlity is due to the fact that W a,i ´ µ a is 1 -subgaussian. For term A 1 , we notice that the empirical estimation ¯ r t 1 p 1 q and the counter N t 1 p 1 q will remain the same if there is no pull for arm 1 , so we can switch the count of time step t 1 to the count of number of pulls for N 1 t p 1 q . To be specific,

Since W 1 ,i ´ µ 1 is 1 -subgaussian, we can use the concentration result Eq. (4a) of Lemma. 3 to bound A 1 as follows:

Therefore, we have can bound the probability of overpull for sub-optimal arm a as follows:

where the third inequality is due to log T ě ? 2 log T when T ě 9 and the last inequality is due to exp p l q ě T log 4 T . Recall that in Algorithm. 2, the exploration phase will stop if there exists an arm ˜ a whose pulls N T c p ˜ a q at the stopping time have exceeds l times of all other arms, i.e.,

So if T c is larger than ř a :∆ a ą 0 8 p l ` 1 q 2 ∆ 2 a ` A p l ` 2 q , it means that there exists at least one sub-optimal arm a whose pulls N T c p a q is larger than 8 l ∆ 2 a ` 1 . So we have:

where the second inequality, we use union bound over a . Then Corollary. 2 can be easily proved with:

Therefore, taking T to infinity, we have:

B.5 Proof of Theorem. 3

We first prove the fundamental limits of sample complexity until commitment in the pre-determined stopping time setting. Suppose the bandit problem (instance) is as follows: the reward expectation of action 1 is µ 1 and the reward expectation of action 2 is µ 2 . We assume µ 1 ą µ 2 and ∆ ' µ 1 ´ µ 2 . Consider another bandit instance with reward expectations λ 1 and λ 2 for the two actions respectively, and λ 1 ` ∆ ď λ 2 . By the 'transportation' lemma [32, Lemma. 1] when T ě 10 , we have for any stopping time τ that:

Since under Gaussian bandit, the KL divergence is simply the squared norm, we have:

Since the inequality holds for any λ 1 and λ 2 such that that λ 1 ` ∆ ď λ 2 , we can minimize the LHS to obtain tighter bounds. So minimizing the LHS over λ 1 and λ 2 gives:

So plug the optimization result into the 'transportation' lemma, we have:

Therefore, rearranging the terms in the inequality, we can derive a lower bound on E µ r N 1 p τ qs as follows:

Let τ ' T c be the fixed length of exploration. Thus, we can derive the following lower bound on the expectation of stopping time as follows:

According to our assumption, the algorithm has c -logarithm regret violation with O p T ´ 1 q confidence, which means the the number of pulls for the sub-optimal action is upper and lower bounded when T is large enough. So we have:

Recall that ˆ a is the action the algorithm chooses to commit to and action 1 is the optimal action in instance µ , so we have E r N 2 p T qs ď E µ r N 2 p T c qs ` T P µ p ˆ a ‰ 1 q . Since the algorithm has O p T ´ 1 q confidence, so we have P µ p ˆ a ‰ 1 q ' O p T ´ 1 q and E µ r N 2 p T qs ď E µ r N 2 p T c qs ` O p 1 q . So we can lower bound the numerator as follows:

Notice that E µ r N 2 p T c qs ď E µ r N 2 p T qs .For the denominator, we can derive an upper bound similarly as follows:

where the second inequality uses the fact that the algorithm has c -logarithm regret violation. So combining both bounds in the numerator and the denominator, we have:

which concludes the proof for the pre-determined stopping time setting. In the adaptive stopping time setting, we can use the same procedure to prove the lower bound of sample complexity until commitment, so we also create another bandit instance with reward expectations λ 1 and λ 2 . Unlike in the pre-determined stopping time setting, now λ 1 and λ 2 satisfies λ 1 ď λ 2 . We also use the 'transportation' lemma and optimize over λ 1 and λ 2 to get a lower bound for E r T c s as follows:

Similar to the proof of pre-determined stopping time setting, we utilize the fact that E µ r N 2 p T c qs ď E µ r N 2 p T qs ď E µ r N 2 p T c qs ` O p 1 q and the algorithm has c -logarithm regret violation to bound the numerator and denominators separately. For the numerator, we have:

For the denominator, we use the fact that the algorithm has c -logarithm regret violation:

So combining both bounds in the numerator and the denominator, we have:

C Proofs of Main Results for KL-EOCP

Now we are ready to prove the regret and sample complexity until commitment results for our KL-EOCP Algorithms, i.e., Algorithm. 3. Without loss of generality, suppose action 1 is the unique optimal action. Recall that for any time index t , UCB t p a q and LCB t p a q denotes be KL upper and lower confidence bound of action a as:

Throughout the proof section, we let W a,i be the i -th sample from pulling arm a the i -th time. Recall that T c is the length of exploration phase. Let ¯ r a,s be the empirical mean of arm a after it has been pulled s times. Moreover, we let:

In order to incorporate this new definition of UCB and LCB , we need new sets of concentration inequalities, which will be summarized in the next subsection. The proof will be delayed to Sec. D.

C.1 Concentration Inequalities for Natural Exponential Families

Since we no longer assume the distributions r ν 1 , ¨ ¨ ¨ , ν A s are subgaussian any more, the concentration inequalities from Sec. B.1, especially Lemma. 3, no longer holds. Therefore, we need a set of new concentrations specialized in the natural exponential family regime. To be specific, we want to bound the probability that the KL divergence of empirical estimations and the true expectation is very large. First, we provide a concentration lemma which is analogous to the Hoeffding's inequality widely used in the subgaussian scenario as follows:

Lemma 5 Let W 1 , W 2 , ¨ ¨ ¨ , W T be identically and independently distributed random variables with common expectation µ ' E r W 1 s and sampled from a distribution ν that belongs to a canonical exponential family P . For any s ď T , Let S s ' ř s i ' 1 W i and ¯ µ s ' S s s denote the sum and the empirical mean of the first s samples. For any δ ą 0 , we have:

Based on Lemma. 5 we propose the following lemmas characterizing the any-time concentration property of random variables.

Lemma 6 Let W 1 , W 2 , ¨ ¨ ¨ , W T be identically and independently distributed random variables with common expectation µ ' E r W 1 s and sampled from a distribution ν that belongs to a canonical exponential family P . For any s ď T , Let S s ' ř s i ' 1 W i and ¯ µ s ' S s s denote the sum and the empirical mean of the first s samples. Let T 1 ď T 2 ď T be two real numbers in R ` . For any l ą 2 , the following holds:

Lemma 7 Let W 1 , W 2 , ¨ ¨ ¨ , W T be identically and independently distributed random variables with common expectation µ ' E r W 1 s and sampled from a distribution ν that belongs to a canonical

exponential family P . For any s ď T , Let S s ' ř s i ' 1 W i and ¯ µ s ' S s s denote the sum and the empirical mean of the first s samples. Let UCB s ' arg max µ ě ¯ r s t s KL p ¯ µ s , µ q ď l u be the upper confidence bound for empirical mean. For any l ą 2 , any T 1 ď T , any µ 1 ą µ , and any ε ą 0 , the following holds:

where β 1 p ε q ' O p ε 2 q and β 2 p ε q ' O p ε ´ 2 q are constants.

C.2 Proof of Regret Optimality for KL-EOCP

In this section, we provide a complete proof of Theorem. 4. We prove the following Theorem which characterizes the finite-time performance of Algorithm. 3, and Theorem. 4 can be derived directly.

Theorem 7 If l ' log p T q ` 4 a 2 log p T q and T ě max t 16 , A, 8 l KL ´ 2 lb u , the regret of KL-EOCP algorithm in Algorithm. 3 can be upper bounded as:

Remark: The asymptotic result is clear from Theorem.7 by letting T increases to infinity, i.e.,

Proof. Without loss of generality, let arm 1 be the unique best arm. From the regret decomposition lemma [38, Lemma 4.5], we can decompose the regret of Algorithm. 3 to the number of pulls for each sub-optimal arm as follows:

Then, the key to bound the total regret is to bound the number of pulls for each sub-optimal arms. Since our KL-EOCP algorithm has a clear separation of exploration and exploitation phases, so for any sub-optimal action a , we can bound its pulls in different phases as follows:

where recall that T c is the end of exploration phase and ˆ a is the arm that the algorithm commits to during exploitation phase.

Bounding I 1 : Wefirst bound I 1 , which is similar to the proof of bounding the regret for the traditional KL-UCB algorithm [12]. We have:

Notice that A t ' a indicates UCB t ´ 1 p 1 q ă UCB t ´ 1 p a q due to the dynamic of UCB exploration, we can bound the last term as:

Event t UCB t ´ 1 p 1 q ă µ 1 u means that at time step t , the mean estimation ¯ r t ´ 1 p 1 q of the optimal arm from previous pulls is lower than µ 1 ´ b t ´ 1 p 1 q , which incurs a large deviation, so we can bound this event with a union over all time steps:

which doesn't depend on time step t any more, so we have:

For two time steps T c ă t 2 , if there is no pulls for arm 1 between them, then the term UCB t 1 p 1 q will remain the same through t 1 P r T c , t 2 s . So instead of counting on the time steps, we can count the number of pulls for arm 1 instead. We have:

Notice that N T c p a q ď T c ´ A ` 1 , so we can bound the probability as follows:

Notice that for any s and under event t UCB s, 1 ă µ 1 u , we have µ 1 ě UCB s, 1 ě ¯ r s, 1 . By definition of UCB s, 1 , we also have KL p ¯ r s, 1 , UCB s, 1 q ' l s . Therefore, we come to the conclusion that KL p ¯ r s, 1 , µ q ě l s . using Eq. (6a) of Lemma 6, we have:

Using the fact that exp p l q ě T log 4 T when T ě 16 and putting the above bound back to the bound of A 2 , we have:

where the last inequality is due to the fact that when T ě 3 , ? log T ď log T . Next, we attempt to bound the middle term A 1 as follows:

Notice that the number of pulls N t ´ 1 p a q will only increase by 1 every time there is new pull, i.e., when A t ' a . Otherwise, the term inside summation is 0 . So instead of counting on the time step t , we can count over the number of pulls over arm a as follows:

where the last inequality is due to N T c p a q ď T c . Using Lemma. 7, we can bound the sum of probabilities. Specifically, for any ε ą 0 , there exists two constants β 1 p ε q ' O p ε 2 q and β 2 p ε q ' O p ε ´ 2 q such that:

Let ϵ ' log ´ 1 4 T , and when T ě 16 , we have log T ě ? log T and 4 ? 2 log T ď 5 log 3 4 T , so we have:

Therefore, combining the bounds on A 1 and A 2 , we can bound I 1 as follows:

Bounding I 2 : After the exploration phase, recall that we select the arm with largest LCB to commit to. The idea of proof is very similar to the regret bound in [2] and our proof of Theorem. 5. We can first bound the probability of selecting a sub-optimal arm a as:

We first bound the term B 2 which states that during the first exploration phase with UCB exploration, the optimal arm is under-pulled, which means that one of the sub-optimal arms have been pulled with larger number of times. Therefore, we have:

Recall that T c ' 4 Al KL lb ` A , so we have:

Then by union bound, we have:

Consider a fixed sub-optimal arm a , then for each probability inside the summation, we have:

Notice that B 4 can be bounded from concentration lemma. First, we switch the count from time step t to the number of pulls for arm 1 since the empirical estimation ¯ r t ´ 1 p 1 q and count for pulls N t ´ 1 p 1 q won't change unless there is a new pull. Then, we will apply Eq. (4a) to the probability as follows:

where the second inequality is because under event t UCB s, 1 ď µ 1 u and according to the definition of UCB s, 1 , we have ¯ r s, 1 ď UCB s, 1 ď µ 1 , and since s KL p ¯ r s, 1 , UCB s, 1 q ' l with µ 1 ą UCB s, 1 , we have s KL p ¯ r s, 1 , µ 1 q ě l . The last inequality uses Lemma. 6. Similar to the procedure of bounding I 1 , notice that exp p l q ě T log 4 T , we have:

On the other hand, let γ ' Q 4 l KL lb U , and we also change the count from time step t to the number of pulls for arm a . Then, B 3 can be expressed as follows:

For any pair of means x, y P r 0 , 1 s , define KL ` p x, y q ' KL p x, y q x ă y . Then, we have:

Notice that γ ' Q 4 l KL lb U ě 4 l KL p µ a ,µ 1 q , we can also bound B 3 as follows:

Recall that µ 1 a P p µ a , µ 1 q such that KL p µ 1 a , µ 1 q ' KL p µ a ,µ 1 q 4 , so we have ¯ r γ,a ě µ 1 a under the event that ! KL ` p ¯ r γ,a , µ 1 q ď KL p µ a ,µ 1 q 4 ) , and thus KL p ¯ r γ,a , µ a q ě KL p µ 1 a , µ a q . By Lemma. 5, we have:

Notice that γ KL p r p γ q , µ a q ě 4 l KL p µ 1 a ,µ a q KL lb ě l by the definition of KL min , we then have:

The last inequality is due to the fact that exp p l q ě T log 4 T . So combine B 3 and B 4 together, we can bound B 2 as follows:

Next, we attempt to bound B 1 . Notice that B 1 can also be bounded as follows:

The term B 6 can be expressed by counting the number of pulls of arm a as follows:

where in the last inequality, we can apply Lemma. (6). Since log T ě ? 2 log T and exp p l q ě 1 T log 4 T when T ě 16 , we have:

For any pair of means x, y P r 0 , 1 s , define KL ´ p x, y q ' KL p x, y q ✶ x ą y . Similarly, term B 5 can be expressed as:

Recall that µ 1 a P p µ a , µ 1 q satisfy KL p µ 1 a , µ a q ' KL p µ a ,µ 1 q 4 , so under event t KL ´ p ¯ r s, 1 , µ a q ď KL p µ a ,µ 1 q 4 u we can conclude ¯ r s, 1 ď µ 1 a , which means KL p ¯ r s, 1 , µ 1 q ě KL p µ 1 a , µ 1 q , so we have:

where the last inequality comes from Lemma. 6. By the definition of KL min , we know that 4 l KL p µ 1 a ,µ 1 q KL lb ě l , so we have:

Therefore, collecting the bounds for B 5 and B 6 and B 2 , we have a bound for I 2 as follows:

Finally, putting the bounds on I 1 and I 2 together, we have:

Therefore, by the regret decomposition lemma, we can bound the total regret as follows:

D Proof of Concentration Inequalities

In this section, we provide the proof of the concentration inequalities presented in previous sections.

D.1 Proof of Lemma. 3

The proof of Lemma. 3 relies on the maximal inequalities in Lemma. 1 and Lemma. 2, and the famous Hoeffding's inequality which we state here for the sake of completeness.

Lemma 8 (Hoeffding's Inequality) Let p W i q T i ' 1 be i.i.d. σ -sub-Gaussian random variables with E r W 1 s ' 0 , we have:

Eq. (4a) of Lemma. 3: We now prove Eq. (4a) of Lemma. 3. We will first base our proof on the anytime concentration on union bound, which is tight only when T 2 ´ T 1 is relatively small. This procedure will result in the first term in the minimum at the RHS. To prove a stronger result when T 2 ´ T 1 is relatively large, we resort to the technique of "peeling device" which divide the time horizon into exponential grids, where we will perform maximal inequality inside each grid and a union bound over the grids. This will result in the second term in the minimum at the RHS. We first perform union bound on s as follows:

Then, we can bound each probability with Hoeffding's concentration inequality as follows:

Therefore, summing up the probabilities, we will have:

Next, we apply the peeling method to prove the second inequality. Take β ą 1 to be a constant. Let M ' t log β T 2 T 1 u , we apply peeling method on s and divide the time horizon over exponential grids r T 1 , T 1 β s , r T 1 β, T 1 β 2 s , ¨ ¨ ¨ , r T 1 β M , T 2 s as follows:

Since at each grid, s ě T 1 β j , so we can upper bound each probability as:

Let β ' l l ´ 1 , then according to the anytime concentration inequality from Lemma. 1, we have:

Then, summing up the probabilities, we can bound the total probability as follows:

where the last inequality is due to the definition of M ' t log β T 2 T 1 u . Notice that when l ě 2 , log β ' log ´ l l ´ 1 ¯ ě 1 l , so we can further upper bound the probability as:

Finally, combining the two bounds Eq. (8) and Eq. (9) together, we can prove the first result of Lemma. 3 as follows:

Eq. (4b) of Lemma. 3: Next, we prove Eq. (4b) of Lemma. 3. Our result is only based on performing union bound, but one can also modify the proof of Eq. (4a) with peeling trick to prove Eq. (4b). However, the result from peeling trick is no better than simply performing union bound, at least not order-wise better. So for simplicity, we apply union bound to the probability as follows:

Then, we can apply Hoeffding's inequality to upper bound each probability as follows:

where the last inequality is due to j ě T 1 . So summing up all the probabilities, we can bound the anytime concentration as:

where the last step is due to 1 ´ e ´ δ 2 2 ě δ 2 2 when δ P r 0 , ? 3 s . Then we finish the proof of Eq. (4b) of Lemma. 3.

Eq. (4c) of Lemma. 3: Finally, we prove Eq. (4a) of Lemma. 3 which bounds the summation of probability for deviation events over the time horizon. Since l ě T 1 δ 2 2 , we define γ ' 2 l δ 2 ě T 1 . Then, we can bound the probabilities when s ď γ by 1 as follows:

When s ě r γ s , we have: b 2 l s ' δ a γ s . So, for each probability inside the summation, we have:

Then, we can bound the probability with Hoeffding's inequality as follows:

Notice that the upper bound function exp p´ δ 2 2 p ? s ´ ? γ q 2 q on the RHS is uni-modal when s ě γ . If a function f p s q is uni-modal, then we can bound the summation ř 8 s ' γ with the sum of max s f p s q and integral ş 8 γ f p s q ds . Therefore, putting the upper bound back to the summation, we have:

So the whole term can be bounded by:

which completes the proof of Lemma. 3.

D.2 Proof of Lemma. 5

We only prove the inequality when ¯ µ s ď µ . The other inequality can be proved exactly the same way. For every λ P R , let ϕ µ p λ q ' log E r exp p λX 1 qs which is well-defined and finite by assumption. Let W λ 0 ' 1 and for s ě 1 , we define W λ t ' exp p λS s ´ sϕ µ p λ qq . We show that ` W λ s ˘ s ě 1 is a martingale with respect to the σ -field F s ' σ p W 1 , ¨ ¨ ¨ , W s q . In fact,

where the second equality is because S s is measurable w.r.t. F s , and the third equality uses the fact that W s ` 1 is independent w.r.t. F s . Th last equality is due to the i.i.d. nature of W 1 and W s ` 1 . Let x P r 0 , µ s be such that KL p x, µ q ' δ { s , and let λ p x q ' θ x ´ θ µ . It is worth-noting that since θ µ is a monotonically non-decreasing function since its inverse function µ ' b 1 p θ q is monotonically non-decreasing. So we have λ p x q ď 0 since x ď µ . Observe that:

Then it holds that x ě ¯ µ s . Notice that for natural parameter exponential family, ϕ µ p λ q ' b p λ ` θ µ q ´ b p θ µ q . Hence on the event t ¯ µ s ă µ u X t s KL p ¯ µ s , µ q ě δ u , we have:

where the first inequality is because λ p x q ă 0 , and the second last equality uses the expression of KL divergence for natural exponential families. Putting everything together, we have:

where the last inequality uses Markov inequality. Since W λ p x q s is a martingale, so we have:

D.3 Proof of Lemma 6

We only prove the inequality when ¯ µ s ď µ . The other inequality can be proved exactly the same way. The proof of Lemma/ 6 partly resembles the Proof of Lemma. 5. However, to show an any-time concentration bound, we need either the union bound or a peeling trick. We first apply union bound as follows:

which obtains the first term in the RHS of Lemma. 6. Next, we apply the peeling trick. For every λ P R , let ϕ µ p λ q ' log E r exp p λX 1 qs which is well-defined and finite by assumption. Let W λ 0 ' 1 and for s ě 1 , we define W λ t ' exp p λS s ´ sϕ µ p λ qq . Recall that ` W λ s ˘ s ě 1 is a martingale with respect to the σ -field F s ' σ p W 1 , ¨ ¨ ¨ , W s q . Take β ą 1 to be a constant. Let M ' t log β T 2 T 1 u , we apply peeling method on s and divide the time horizon over exponential grids r T 1 , T 1 β s , r T 1 β, T 1 β 2 s , ¨ ¨ ¨ , r T 1 β M , T 2 s as follows:

Let s i ' T 1 β i and let x ď µ such that s KL p x, µ q ' l . let λ p x q ' θ x ´ θ µ < 0. Then, we have KL p x, µ q ' λ p x q x ´ ϕ µ p λ p x qq . Consider z such that z i ă µ and KL p z i , µ q ' l s i , so we have when s P r s i , s i ` 1 s

So we can conclude that ¯ µ s ď z i ` 1 Also, we have:

Therefore, we have:

So we can bound each probability as:

where the last inequality uses Markov inequality. Since W λ p z i ` 1 q s is a martingale, so we have:

where in the last step, we choose β ' l l ´ 1 . Then, summing up the probabilities, we can bound the total probability as follows:

where the last inequality is due to the definition of M ' t log β T 2 T 1 u . Notice that when l ě 2 , log β ' log ´ l l ´ 1 ¯ ě 1 l , so we can further upper bound the probability as:

D.4 Proof of Lemma. 7

Our proof is based on the analysis of Theorem. 2 of [12]. For any pair of means x, y P r 0 , 1 s , define KL ` p x, y q ' KL p x, y q ✶ x ă y . Then for a fixed s , under event t UCB s ě µ 1 u we have either µ 1 ă ¯ µ s , or µ 1 ą ¯ µ s but s KL p ¯ µ s , µ 1 q ď l . So in general we can conclude that s KL ` p ¯ µ s , µ 1 q ď l . Then we can bound the sum of probabilities as follows:

Define γ ' p 1 ` ε q l KL ` p µ,µ 1 q ' p 1 ` ε q l KL p µ,µ 1 q , then if T 1 ą γ , we can bound the first γ terms in the summation with 1 . If otherwise T 1 ď γ , the whole summation is bounded by γ . So without loss with generality, assume γ ă T 1 and let ε ą 0 be a constant, we have:

where the second inequality is due to KL ` p ¯ µ s , µ 1 q ą 0 . For any s , let r p ε q P p µ, µ 1 q such that KL p r p ε q , µ 1 q ' KL p µ,µ 1 q 1 ` ε . if KL ` p ¯ µ s , µ 1 q ď KL p µ,µ 1 q 1 ` ε , we have ¯ µ s ě r p ε q . Hence,

where the last inequality uses Lemma. 5. So we can bound the sum of probabilities as follows:

Notice that exp p´ γ KL p r p ε q , µ qq ' exp ´ ´ l p 1 ` ε q KL p r p ε q ,µ q KL p µ,µ 1 q ¯ ď T ´ β 1 p ε q , where β 1 p ε q ' p 1 ` ε q KL p r p ε q ,µ q KL p µ,µ 1 q . Let β 2 p ε q ' 1 1 ´ exp p´ KL p r p ε q ,µ qq . It is easy to check that r p ε q ' µ ` O p ε q , so we have β 1 p ε q ' O p ε 2 q and β 2 p ε q ' O p ε ´ 2 q . So we have:

Remark: Let ε ' l ´ 1 4 , we have:

10 0

10 1

10 2

10 3

10 4

10 5

Number of Rounds

0

50

100

150

200

250

300

Regret

EOCP

EOCP-UG

UCB

TAS-ETC

Action-Elimination

DETC-K

(a) Gaussian Bandits

10 0

10 1

10 2

10 3

10 4

10 5

Number of Rounds

0

20

40

60

80

100

120

140

160

180

200

220

Regret

EOCP

EOCP-UG

UCB

TAS-ETC

Action-Elimination

DETC-K

KL-UCB

KL-EOCP

E Numerical Experiments Beyond 2-Armed Bandits

In this section, we present numerical experiments based on bandits with more than two arms. We consider a four armed bandit model with expected rewards r 0 . 7 , 0 . 2 , 0 . 2 , 0 . 2 s . We study the cumulative regret performance of all our proposed algorithms, i.e., EOCP in Algorithm. 1, EOCP-UG in Algorithm. 2, and KL-EOCP in Algorithm. 3, with both Gaussian and Bernoulli rewards. Since some baseline algorithms we chose in Section. 6 (e.g., DETC and BAI-ETC) does not directly generalize to the multi-arm model, we replace them with DETC-K [23] and TAS-ETC respectively, where the TAS-ETC algorithm first uses the famous Track and Stop [36] algorithm to identify the best arm and commits to it for the rest of horizon. Additional to all other baselines we used in Sec. 6, we also compare our algorithms to the Action-Elimination algorithm [42], which provably has both O p log T q regret and O p log T q commitment time. We choose the reward of all sub-optimal arms to be the same value so that BAI based algorithms can utilize a closed form expression to solve the optimal arm pull fraction w ˚ and does not need to solve a non-convex minimax problem. We choose ∆ lb to be 0 . 5 for EOCP and KL-EOCP, and the results are shown in Fig. 2 and averaged over 10 4 iterations.

In both Gaussian and Bernoulli bandits, we observe very similar trends compared to Fig. 1 in twoarmed models. the TAS-ETC algorithm and DETC-K algorithm incurs high regret over the total time horizon due to their aggressive exploration at the beginning of the trials, similar to the performance of BAI-ETC and DETC in 2-armed models. The Action Elimination algorithm incurs even larger regret because of its almost uniform sampling rule at the beginning of the trials. It selects sub-optimal actions even more often than BAI based algorithms and could not identify the optimal action as quickly. On the other hand, our proposed algorithms take a more delicate exploration strategy and result in lower regret, which indicates that the EOCP algorithm and its variants generalize well beyond two-armed bandit models. The over-exploration phenomenon is also witnessed in this experiment the UCB algorithm and the KL-UCB algorithm continues to explore sub-optimal actions even after the EOCP algorithms have identified the best action. The regret of UCB algorithms continues to increase during the whole time horizon, which shows that over-exploration is a common and fundamental issue in optimistic algorithms. Preventing over-exploration will significantly boost the performance of algorithms not only in bandits but also in other online decision-making problems.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work focuses on simultaneously achieving regret minimization and best arm identification in multi-armed bandits, which is called Regret-Optimal Best Arm Identification (ROBAI). That is, the goal is to identify with high probability and as fast as possible / within a certain time frame the optimal arm, and play until round T. In practice, authors propose three algorithmic contributions, which have guarantees on the cumulative regret, the time before commitment to exploiting a single arm (called stopping time in the paper) and the error on the identification of the optimal arm. The first two algorithms are asymptotically optimal regretwise for Gaussian bandits, and respectively tackle the pre-determined stopping time case and the adaptive stopping time case. They consist in exploring arms using the UCB principle, and committing to the arm which maximizes the LCB. The last one is a variant of the previous algorithms which achieves asymptotical regret optimality for subgaussian distributions, by leveraging KL confidence intervals.\n\nSTRENGTHS:\n- Originality: This paper tackles an interesting problem which has been seldom considered. Although the algorithms reuse well-known principles (UCB / LCB), their analysis allow to prove very good results (regret optimality, interesting bound for adaptive “stopping” time). Table 1 is clear and shows that the work is well-grounded in prior literature.\n- Quality: The results seem technically sound, although I did not check the appendix in detail.\n- Clarity: The submission and the proof sketches are clearly written.\n- Significance: This paper provides and substantiates an interesting insight on the behavior of UCB algorithms with respect to over-exploration\n\nWEAKNESSES:\n- Significance: I am not 100% convinced about the real-life applications of this framework. If the “stopping” time is fixed because of budget limits (predetermined setting), I don’t see what ROBAI brings more than classical regret minimization (i.e., why playing pessimistically at the end of the sampling phase is better), especially since it is not feasible in practice, as the related algorithm requires the knowledge of Delta_min to be optimal.\n\nQUESTIONS:\n-  What does ROBAI bring more than classical regret minimization (i.e., why playing pessimistically at the end of the sampling phase is better) in the predetermined stopping time setting?\n\nLIMITATIONS:\nThis paper deals with theoretical work, and does not raise significant concerns about negative impacts.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the classical multi-armed bandits problem with the goal to design algorithms achieving tight regret bound and tight sample complexity to identify the best arm simultaneously. To this end, three algorithms are proposed based on upper confidence exploration and lower confidence commitment for three settings: Gaussian bandits with known and unknown suboptimality gap $\\Delta_{\\min}$, general bandits when suboptimality gap (in terms of KL divergence) is known. It is proved that all the three algorithms enjoy tight regret bound up to the constant in front of the dominating term ($\\log T$), and they also have polylog sample complexity scaled linearly with $1 / \\Delta_{\\min}^2$. There is also a lower bound on the sample complexity given the regret to have a tight constant in the $\\log T$. The lower bound states that the rate of sample complexity is tight when $\\Delta_{\\min}$ is given, and can also be tight when $\\Delta_{\\min}$ is unknown as long as the order of remaining terms in the regret is smaller than $\\log T$.\n\nSTRENGTHS:\n1.The paper addresses the classical settings of regret minimization and best arm identification problems in bandit theory. The investigation of their combination opens up new avenues for fundamental methods in this field. The paper stands out for its clean and clear presentation, making it accessible to readers and facilitating understanding of the proposed techniques.\n\n2.The paper introduces several novel techniques that enable the simultaneous bounding of regret and sample complexity. The incorporation of lower confidence bound (LCB) commitment following upper confidence bound (UCB) exploration is a particularly interesting departure from the traditional approach of UCB commitment. This novel technique offers fresh insights and potential improvements in bandit algorithms. Additionally, the stopping criterion employed in the unknown gap setting is highly appreciated in theory by leveraging the deep connections between the number of pulls across different arms and the total pulls of suboptimal arms.\n\n3.The analysis of lower bounds in the paper sheds light on the fundamental limits and trade-offs involved in identifying the best arm. These findings provide valuable insights into the design of algorithms, emphasizing the need to balance regret and sample complexity. The paper's techniques offer the potential for achieving a near-perfect balance between these two factors in practical applications of multi-armed bandits.\n\nWEAKNESSES:\n1. The unified achievement of both optimal regret and optimal sample complexity is an important problem in theory and application. However, I am not fully convinced of the motivation of this paper to study the constant-level tight regret bound but rate-level tight sample complexity. It somehow looks like a beam search of different settings in multi-armed bandits that are not fully addressed, since algorithms with both rate-optimal regret and rate-optimal sample complexity have been extensively studied. The vanilla multi-armed bandits is a fundamental model in theory, and the algorithms of multi-armed bandits provide prominent insights in many other problems. However, the UCB and LCB algorithms, which are conceptual in nature, require significant domain-dependent modifications to be applicable in real-world scenarios. This lack of naturalness in the setting detracts from the paper, despite the appreciation for the introduction of new techniques.\n\n2. Two of the three proposed algorithms require the minimal gap to be known, which I believe is not very possible in application (except for the synthetic task). There are two reasons for it. Consider some tasks that aim to identify the user's favorite items: the expected reward is 1 if the item is liked by the user, and 0 otherwise, with some noise to model random attributes of the user. In this case, the gap is known and large, which greatly simplifies the problem. In general, it takes a very short period of time to find the favorite items even in the presence of noises, so it is totally acceptable that the regret is rate-optimal instead of constant-optimal. When the gap is not determined manually and small, it is not likely one can obtain the exact value of the gap. People often guess this gap by some binary search routines. However, the Algorithm 2 suffers a $\\log^2 T$ sample complexity, where $\\log T$ is the dominating term. This means the overall sample complexity of this algorithm is even not rate-optimal, though the regret is constant-optimal. It would be better in this case to use other methods such as elimination to keep both regret and sample complexity rate-optimal.\n\n3. The experimental results presented in the paper have limitations in their scope. Firstly, comparing and plotting cumulative rewards instead of regret would provide a more convincing demonstration, as the sum of rewards and regret is a linear function rather than a constant. Additionally, the experiments should be conducted in more diverse environments, including scenarios with a higher number of arms. The current results do not suggest a significant improvement over the vanilla UCB algorithm, raising questions about the practical advantage of the proposed EOCP and its variants (see the questions below).\n\nQUESTIONS:\n1. Why you only consider the setting where the regret should be constant-optimal while the sample complexity can be rate-optimal. From the perspective of theory, is it possible to obtain both constant-optimal algorithm or constant-optimal sample complexity and rate-optimal regret?\n\n2. How is the identification accuracy of the vanilla UCB algorithm at the step it gets the same regret of EOCP? Is this accuracy comparable to the commitment accuracy of EOCP?\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work studies how to design an algorithm with an asymptotically optimal regret rate such that it will also commit to the best arm with high probability after a stopping time (e.g., O(logT)). The paper proposes two algorithms in the Gaussian bandits setting: one with pre-determined stopping time (EOCP), which requires the knowledge of the minimum reward gap, and another with adaptive stopping time (EOCP-UG). The authors prove both algorithms are asymptotically optimal and will commit to the best arm with confidence O(1/T) after O(log(T)) and O(log^2(T)) respectively. The paper further shows the corresponding lower bounds for commitment times (expected stopping time) and finds EOCP is sample optimal and EOCP-UG is nearly sample optimal. In addition, the authors extend the EOCP to KL-EOCP for general bandits, whose commitment time also matches the lower bound.\n\nSTRENGTHS:\n1) In general, the studied question is interesting and the paper is well-written and easy to follow.\n\n2) The proposed algorithms are novel which are based on well-designed stopping rules.\n\n3) The theoretical results and the proofs are clear. The numerical results also show the advantages of the proposed algorithms compared to the benchmark algorithms, including UCB and BAI algorithms in the literature.\n\nWEAKNESSES:\n1) The problem setting is the fixed-confidence BAI setting (note for the setting of EOCP, the stopping time is pre-determined by the algorithm instead of by the environment). It would be great if the authors can give some discussions on the fixed budget setting (the stopping time is given), like the challenges of extending the proposed algorithms. \n\n2) In the general bandits setting, the authors only extend EOCP to KL-EOCP, which needs the KL_min of arms. The authors demonstrate if KL_min is not known, an extension of EOCP-UG can deal with this setting.  But there is no pseudo-code or provable results on it. More details or discussions would be helpful.\n\nMinor comments:\n\nIn Table 1, it might be helpful to add another column ‘Setting’, since ‘Optimality’ is a bit confusing for explaining ‘Gaussian’ or ’General’.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nNa.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper delves into the study of the 'Explore Then Commit' (ETC) policy, where the algorithm is divided into two stages: exploration and commitment. During exploration, the algorithm is permitted to switch actions, while the commitment phase restricts the algorithm to pulling only the commit arm. The primary objective of the algorithm is to reduce the expected commit time, denoted as $T_c$, and ensure that at time $T_c$, $\\hat a \\neq a^*$ holds true with a probability of $O(1/T)$. \n\nThree variations of the ETC policy are introduced in this paper: EOCP, EOCP-UG, and KL-EOCP.\n\nFor EOCP, it's structured for a known gap setting, achieving regret in the order of $2\\log T/\\Delta_i$. The authors assert that this outcome is asymptotically optimal. However, this claim appears to be incorrect. As per reference [On Explore-then-Commit Strategy], the asymptotic optimal regret for a known gap should be $\\log T/(2\\Delta_i)$. Furthermore, the authors have not discussed this setting's reference in a comprehensible manner. For instance, DETC with a known gap achieves exact asymptotic optimality.\n\nFor EOCP-UG, the authors show that it achieves asymptotic optimality with a commit time of $\\log^2 T$.\n\nAs for KL-EOCP, it necessitates knowledge of certain parameters for the design of the pre-determined stopping time.\n\nIn summary, the claim that the EOCP and KL-EOCP algorithms are asymptotically optimal appears to be overstated. The upper bound for EOCP-UG and its associated lower bound is the same as DETC. As for the lower bound, the assumption seems excessively strong. The authors did not clearly indicate whether algorithms must adhere to the equation given in Line 241.\n\nPrimarily, this paper targets two-armed bandit problems, focusing on asymptotic regret. However, I am intrigued by the finite-time bound. I would appreciate an algorithm that is not only asymptotically optimal but also demonstrates a robust finite-time bound. In addition, the paper only considers a fixed horizon T setting. What about the case with an unknown $T$? From my understanding, DECT and its variant [Almost Optimal Anytime Algorithm for Batched Multi-Armed Bandits], are also suitable for an unknown T setting. Furthermore, the necessity of the assumption in Line 141 is not clearly articulated and requires clarification.\n\nRegarding experimental results, the performance of ETC and DECT presented in this paper is inconsistent with previous papers. This discrepancy might be due to parameter adjustments in these algorithms. I would appreciate seeing empirical results with parameter tuning for both ETC and DETC.\n\n----\n\n# At the end of discussion phase\n\nThe authors fail to address my primary concerns. Below, I elaborate on these issues:\n\n1. Line 174 of the manuscript asserts that in the pre-determined setting (where the gap $\\Delta$ is known), $\\frac{2\\log T}{\\Delta}$ is asymptotically optimal. This claim is misleading. Algorithm 4 in reference [21] shows that in the pre-determined setting with $T_c = \\log^2 T$, there exists an algorithm with a regret of $\\frac{\\log T}{2\\Delta}$. Moreover, the known lower bound (Theorem 6 in [20]) for known gap setting is $\\frac{\\log T}{2\\Delta}$. Therefore, stating $ \\frac{2\\log T}{\\Delta}$ as asymptotically optimal is an overstatement. \n\nIn their response, the authors attempt to equate the lower bound of the pre-determined setting with the lower bound of the known gap setting in the ETC strategy [20]. This comparison is problematic and has the potential to mislead other readers. Specifically, Table 1 in the authors' response erroneously claims that $4\\log T/\\Delta$ is the lower bound for the known gap setting, whereas this is actually the lower bound proven for the **unknown gap** setting in Theorem 4 of [20]. Additionally, they assert that $2\\log T/\\Delta$ is the lower bound for the known gap and pre-determined settings, which contradicts Algorithm 4 in [21] that shows regret $\\frac{\\log T}{2\\Delta}$. \n\n\n2. As I pointed out in my initial review, DECT [21] (Algorithm 5 and 2) can be directly applied to the unknown gap setting. The results for DECT in an unknown gap setting are identical to those presented in this paper.\n\nGiven the paper's significant overstatements and the paper's limited contributions, I am revising my score from 4 to 3.\n\nSTRENGTHS:\nIn this paper, the authors propose an algorithm called Explore Optimistically then Commit Pessimistically (EOCP) to solve the Regret Optimal Best Arm Identification (ROBAI) problem. It first uses an optimistic modified UCB algorithm to explore actions with a slightly larger exploration function , and then commits to actions according to a pessimistic LCB algorithm when the exploration ends. The main contributions include designing new stopping rules with both pre-determined stopping time (vanilla EOCP) and adaptive stopping time (the EOCP-UG variant), which provably balance the trade-off between regret minimization and optimal action identification\n\nWEAKNESSES:\nPlease refer to the Summary.\n\nQUESTIONS:\nPlease refer to the Summary.\n\nLIMITATIONS:\nThe main consideration of mine is that the proposed algorithm considers asymptotic bounds other than finite $T$ (probabily unknown). This limits the contribution of the paper heavily.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-08"}