Title: Optimal Batched Best Arm Identification
Venue: NeurIPS 2024
Accepted: yes
Keywords: Best arm identification

Batched bandits
Abstract: We study the batched best arm identification (BBAI) problem, where the learner's goal is to identify the best arm while switching the policy as less as possible. In particular, we aim to find the best arm with probability $1-\delta$ for some small constant $\delta>0$ while minimizing both the sample complexity (total number of arm pulls) and the batch complexity (total number of batches). We propose the three-batch best arm identification (Tri-BBAI) algorithm, which is the first batched algorithm that achieves the optimal sample complexity in the asymptotic setting (i.e., $\delta\rightarrow 0$) and runs in $3$ batches in expectation. Based on Tri-BBAI, we further propose the almost optimal batched best arm identification (Opt-BBAI) algorithm, which is the first algorithm that achieves the near-optimal sample and batch complexity in the non-asymptotic setting (i.e., $1/\delta$ is finite), while enjoying the same batch and sample complexity as Tri-BBAI when $\delta$ tends to zero. Moreover, in the non-asymptotic setting, the complexity of previous batch algorithms is usually conditioned on the event that the best arm is returned (with a probability of at least $1-\delta$), which is potentially unbounded in cases where a sub-optimal arm is returned. In contrast, the complexity of Opt-BBAI does not rely on such an event. This is achieved through a novel procedure that we design for checking whether the best arm is eliminated, which is of independent interest.

Optimal Batched Best Arm Identification

Abstract

We study the batched best arm identification (BBAI) problem, where the learner's goal is to identify the best arm while switching the policy as less as possible. In particular, we aim to find the best arm with probability 1 -δ for some small constant δ > 0 while minimizing both the sample complexity (total number of arm pulls) and the batch complexity (total number of batches). We propose the three-batch best arm identification (Tri-BBAI) algorithm, which is the first batched algorithm that achieves the optimal sample complexity in the asymptotic setting (i.e., δ → 0 ) and runs in 3 batches in expectation. Based on Tri-BBAI, we further propose the almost optimal batched best arm identification (Opt-BBAI) algorithm, which is the first algorithm that achieves the near-optimal sample and batch complexity in the non-asymptotic setting (i.e., δ is finite), while enjoying the same batch and sample complexity as Tri-BBAI when δ tends to zero. Moreover, in the non-asymptotic setting, the complexity of previous batch algorithms is usually conditioned on the event that the best arm is returned (with a probability of at least 1 -δ ), which is potentially unbounded in cases where a sub-optimal arm is returned. In contrast, the complexity of Opt-BBAI does not rely on such an event. This is achieved through a novel procedure that we design for checking whether the best arm is eliminated, which is of independent interest.

1 Introduction

Multi-armed bandit (MAB) is a fundamental model in various online decision-making problems, including medical trials [40], online advertisement [5], and crowdsourcing [43]. These problems typically involve a bandit with multiple arms, where each arm follows an unknown distribution with a mean value. At each time step, the learner selects an arm, and receives a reward sample drawn from the chosen arm's distribution. Best arm identification (BAI) aims to identify the arm with the highest mean reward, which can be approached with a fixed budget or a fixed confidence level [14, 4, 19].

In this paper, we assume that there is a unique best arm and study BAI with fixed confidence. Specifically, we consider a set [ n ] = { 1 , 2 , . . . , n } of n arms, where each arm i is associated with a reward distribution having a mean value µ i . Without loss of generality, for a bandit instance denoted by µ = { µ 1 , µ 2 , . . . , µ n } , we assume that µ 1 > µ 2 ≥ · · · ≥ µ n . At each time step t , the learner selects an arm and observes a sample drawn independently from the chosen arm's distribution. In the fixed confidence setting, the learner aims to correctly identify the best arm (arm 1 in our context) with a probability of at least 1 -δ , where δ > 0 is a pre-specified confidence parameter. Meanwhile, the learner seeks to minimize the total number of arm pulls, also known as the sample complexity.

̸

Denote by a ∗ ( λ ) = arg max i λ i the best arm for an arbitrary bandit instance λ . Let Alt( µ ) = { λ : a ∗ ( λ ) = 1 } be a set of models that have a different best arm from the model µ , and P k = { w ∈ R n + : ∑ n i =1 w i = 1 } be the probability simplex. Garivier and Kaufmann [17] showed that for bandits with reward distributions that are continuously parameterized by their means, the number N δ of pulls

by any algorithm that returns the best arm with a probability of at least 1 -δ is bounded by

where T ∗ ( µ ) is defined according to

and d ( µ i , λ i ) is the Kullback-Leibler (KL) divergence of two arms' distributions with means µ i and λ i , respectively. We say that an algorithm achieves the asymptotically optimal sample complexity if it satisfies

The well-known Track-and-Stop algorithm [17] solves the BAI problem with asymptotically optimal sample complexity. However, it is a fully sequential algorithm, which is hard to be implemented in parallel. The learner in such an algorithm receives immediate feedback for each arm pull, and adjusts the strategy for the next arm selection based on the previous observations. Unfortunately, this sequential approach may not be feasible in many real-world applications. For instance, in medical trials, there is typically a waiting time before the efficacy of drugs becomes observable, making it impossible to conduct all tests sequentially. Instead, the learner needs to group the drugs into batches and test them in parallel. Similarly, in crowdsourcing, the goal is to identify the most reliable worker for a specific task by testing candidates with a sequence of questions. Again, there is often a waiting time for workers to finish all the questions, necessitating the grouping of workers into batches and conducting parallel tests. In such scenarios, the results of pulling arms are only available at the end of each batch [22, 1, 39, 37, 15, 24, 25, 38].

Motivated by these applications, we study the problem of batched best arm identification (BBAI). In BBAI, we are allowed to pull multiple arms in a single batch, but the results of these pulls are revealed only after the completion of the batch. The objective is to output the best arm with a high probability of at least 1 -δ , while minimizing both the sample complexity (total number of pulls) and the batch complexity (total number of batches). This leads to the following natural question:

Can we solve the BBAI problem with an asymptotically optimal sample complexity and only using a constant number of batches?

Furthermore, the aforementioned results hold only in the limit as the confidence parameter δ approaches zero, which may provide limited practical guidance since we often specify a fixed confidence level parameter δ > 0 . To address this, some algorithms [31, 21] have been proposed to solve the BAI problem with finite confidence. Specifically, for some universal constant C , these algorithms satisfy that with probability 1 -δ ,

where ∆ i = µ 1 -µ i . In addition, Jamieson et al. [21] demonstrated that for two-armed bandits, the term log log ∆ -1 2 ∆ 2 2 is optimal as ∆ 2 → 0 , where ∆ 2 is assumed to be the gap between the best arm and the second best arm. In the context of the batched setting, Jin et al. [22] proposed an algorithm that achieves the sample complexity in (1.4) within O (log ∗ ( n ) · log(1 / ∆ 2 )) batches, where log ∗ ( n ) is the iterated logarithm function 1 . Furthermore, Tao et al. [39] proved that for certain bandit instances, any algorithm that achieves the sample complexity bound shown in (1.4) requires at least Ω ( log ∆ -1 2 log log ∆ -1 2 ) batches. It should be noted that the batched lower bound proposed by Tao et al. [39] assumes δ as a constant, making it inapplicable in the asymptotic setting. Therefore, an additional question that arises is:

Can we achieve the optimal sample complexity in (1.3) and (1.4) adaptively, taking into account the specified confidence level parameter δ , while minimizing the number of batches?

In this work, we provide positive answers to both of the aforementioned questions. Specifically, the main contributions of our paper can be summarized as follows:

We propose Tri-BBAI (Algorithm 1) that returns the best arm with a probability of at least 1 -δ by pulling all arms for a total number of at most T ∗ ( µ ) log(1 /δ ) times when δ → 0 . Tri-BBAI employs three batches in expectation when δ → 0 . Therefore, Tri-BBAI achieves the optimal sample within constant batches. As a comparison, Track-and-Stop [17] also achieves the optimal sample complexity but requires solving the right-hand side of (1.2) after each arm pull, resulting in a batch complexity of the same order as the sample complexity, which is a significant computational overhead in practice.

Built upon Tri-BBAI, we further propose Opt-BBAI (Algorithm 2) that runs in O (log(1 / ∆ 2 )) expected batches and pulls at most O ( ∑ i> 1 ∆ -2 i log( n log ∆ -1 i ) ) expected number of arms for finite confidence δ . It is also important to note that Opt-BBAI achieves the same sample and expected batch complexity as Tri-BBAI asymptotically when δ → 0 . Moreover, for the finite confidence case, this sample complexity matches (1.4) within a log( · ) factor and matches the optimal batched complexity within a log log( · ) factor. To the best of our knowledge, Opt-BBAI is the first batched bandit algorithm that can achieve the optimal asymptotic sample complexity and the near-optimal non-asymptotic 2 sample complexity adaptively based on the assumption on δ .

Notably, in the non-asymptotic setting, the complexity of earlier batch algorithms [22, 32, 39, 24] typically depends on the event of returning the best arm, which occurs with a probability of at least 1 -δ . However, this complexity could potentially become unbounded if a sub-optimal arm is returned instead. Unlike these algorithms, the complexity of Opt-BBAI is not contingent on such an event. This is made by employing an innovative procedure to verify if the best arm is eliminated, a method that holds its independent interest. To the best of our knowledge, Opt-BBAI is the first algorithm that achieves the optimal asymptotic sample complexity while providing the optimal non-asymptotic sample complexity within logarithm factors.

We also conduct numerical experiments 3 to compare our proposed algorithms with the optimal sequential algorithm Track-and-Stop [17], and the batched algorithm Top-k δ -Elimination [22] on various problem instances. The results indicate that our algorithm significantly outperforms the Track and Stop method in terms of batch complexity, while its sample complexity is not much worse than that of Track and Stop. Additionally, our algorithm demonstrates a notable improvement in sample complexity compared to [22], while exhibiting similar batch complexity.

For ease of reference, we compare our results with existing work on batched bandits in Table 1.

2 Related Work

The BAI problem with fixed confidence is first studied for [0 , 1] bounded rewards by Even-Dar et al. [14]. The sample complexity of their algorithm scales with the sum of the squared inverse gap, i.e., H ( µ ) = ∑ i> 1 1 / ∆ 2 i . Garivier and Kaufmann [17] showed that H ( µ ) < T ∗ ( µ ) ≤ 2 H ( µ ) with T ∗ ( µ ) defined in (1.2) and proposed the Track-and-Stop algorithm, which is the first one in the literature proved to be asymptotically optimal. Later, Degenne et al. [12] viewed T ∗ ( µ ) as a min-max game and provided an efficient algorithm to solve it. Jourdan et al. [27] studied the asymptotically optimal sample complexity for any reward distributions with bounded support. Degenne et al. [13] studied pure exploration in linear bandits. Their proposed algorithm proved to be both asymptotically optimal and empirically efficient.

There are also many studies [30, 10, 8, 21, 31] that focus on providing non-asymptotic optimal sample complexity. The best non-asymptotic sample complexity was achieved by Chen et al. [10], which replaces the term log log ∆ -1 i in (1.4) with a much smaller term. Furthermore, when we allow a loss ϵ of the quality of the returned arm, the problem is known as ( ϵ, δ ) -PAC BAI, for which various algorithms [29, 14, 6, 9] are proposed, achieving the worst-case optimal sample complexity.

There are a few attempts to achieve both the asymptotic and non-asymptotic optimal sample complexity. Degenne et al. [12] provided a non-asymptotic sample complexity ˜ O ( nT ∗ ( µ ) 2 ) , which could be nT ∗ ( µ ) larger than the optimal sample complexity. Recently, Jourdan and Degenne [26] managed to achieve a sample complexity that is β -asymptotically optimal (with w 1 fixed at 1 /β in (1.2)), rendering it asymptotically optimal up to a factor of 1 /β . Meanwhile, they also reached a non-asymptotic sample complexity of O ( ( H ( µ ) · log H ( µ )) α ) 4 for some α > 1 , where H ( µ ) = ∑ i> 1 1 / ∆ 2 i . Wang et al. [42] explored both asymptotic and non-asymptotic sample complexities. Their algorithm achieves asymptotic optimality and shows a non-asymptotic sample complexity of O ( nH ( µ ) 4 /w 2 min ) . However, this non-asymptotic sample complexity is nH ( µ ) 3 /w 2 min away from being optimal. Jourdan et al. [28] studied ( ϵ, δ ) -PAC BAI, proposing an asymptotically optimal algorithm and providing non-asymptotic sample complexity. When ϵ = 0 , it aligns with our setting, our non-asymptotic sample complexity is better scaled. Specifically, Jourdan et al. [28] offered a non-asymptotic sample complexity scale of n/ ∆ 2 2 log(1 / ∆ 2 ) , whereas ours is more instance-sensitive, as our sample complexity is related to all gaps, not just ∆ 2 . Additionally, Jourdan et al. [28] considered a practical scenario where the algorithm can return a result at any time while still ensuring a good guarantee on the returned arm. For ease of reference, we summarize the sample complexity of different algorithms in Table 2. As shown in Table 2, our algorithm is the only one that achieves both the asymptotic optimality and non-asymptotic optimality within logarithm factors.

Another line of research [31, 4, 7] investigated BAI with a fixed budget, where the objective is to determine the best arm with the highest probability within T pulls. Audibert et al. [4] and Karnin et al. [31] offered finite-time bounds for this problem, while Carpentier and Locatelli [7] presented a tight lower bound, demonstrating that such finite-time bounds[4, 31] are optimal for certain bandit

instances. However, the asymptotic optimality for this problem remains unknown. Interested readers are referred to recent advancements[11, 35] in the asymptotic results of BAI with a fixed budget.

In addition, some recent works [1, 22, 39, 32] also focused on batched BAI in non-asymptotic setting. Agarwal et al. [1] studied the batched BAI problem under the assumption that ∆ 2 is known. They proposed an algorithm that has the worst-case optimal sample complexity of O ( n ∆ -2 2 ) and runs in log ∗ ( n ) batches. Later, Jin et al. [22] provided the algorithms that achieves the sample complexity given in (1.4) within ˜ O (log(1 / ∆ 2 )) batches, where ˜ O hides the log log( · ) factors. Tao et al. [39] studied the BAI problem in the general collaborative setting and showed that no algorithm can achieve (1.4) with o ((log ∆ -1 2 ) / log log ∆ -1 2 ) batches. Karpov et al. [32] further proposed an algorithm which has the sample complexity ∑ i> 1 log( n log ∆ -1 ) ∆ 2 i and the batch complexity O (log(1 / ∆ 2 )) . We note that the lower bound of batch complexity given by Tao et al. [39] can only be applied to a constant δ . In other words, the lower bound of complexity for the asymptotic setting remains unknown. Agrawal et al. [2] studied the optimal batch size for keeping the asymptotic optimality. They showed an algorithm with batch size m = o (log(1 /δ )) achieves the asymptotic optimality. The batch complexity of the algorithm is O ( T ∗ ( µ ) log(1 /δ ) /m ) . Wang et al. [42] provided an algorithm that uses O ( √ log δ -1 ) batches and retains asymptotic optimality for linear bandits. However, for the asymptotic setting, such batch size is still too large as it grows to infinity as δ decreases to 0.

3 Achieving Asymptotic Optimality with at Most Three Batches

3.1 Reward Distribution

We assume the reward distributions belong to a known one-parameter exponential family that is commonly considered in the literature [17, 16, 36]. In particular, the measure ν θ of such probability distributions with respect the model parameter θ satisfies d ν θ d ρ ( x ) = exp( xθ -b ( θ )) , for some measure ρ and b ( θ ) = log( ∫ e xθ d ρ ( x )) . For one-parameter exponential distributions, it is known that b ′ ( θ ) = E [ ν θ ] and the mapping b ′ ( θ ) ↦→ θ is one-to-one. Moreover, given any two mean values µ and µ ′ , we define d ( µ, µ ′ ) to be the Kullback-Leibler divergence between two distributions with mean values µ and µ ′ .

3.2 The Proposed Three-Batch Algorithm

Algorithm 1 shows the pseudo-code of our proposed Tri-BBAI algorithm. In particular, Tri-BBAI has four stages. In what follows, we elaborate on the details of each stage.

̸

Stage I: Initial exploration. In this stage, we pull each arm for L 1 times. Denote by i ∗ ( t ) the arm with the largest empirical mean at time t (i.e., after we pull all arms for a total number of t times), i.e., i ∗ ( t ) = max i ∈ [ n ] ˆ µ i ( t ) . Let τ 0 = nL 1 . Fix t = τ q -1 ≥ τ 0 , we let b q i = ˆ µ i ( t ) + ϵ for i = i ∗ ( t ) and b q i = ˆ µ i ( t ) -ϵ for i = i ∗ ( t ) . Let w ∗ ( µ ) = arg max w ∈P k inf λ ∈ Alt( µ ) ( ∑ n i =1 w i · d ( µ i , λ i )) . Then, for the aforementioned b q = { b q 1 , b q 2 , . . . , b q n } , we calculate w ∗ ( b q ) according to Lemma A.1 and T ∗ ( b q ) according to Lemma A.2. We note that arm 1 is assumed to be the arm with the highest mean in these two lemmas. However, in the context of b q , the index of i ∗ ( t ) might be different. To align with the standard practice, we can rearrange the indices of the arms in b q so that i ∗ ( t ) corresponds to index 1.

Purpose. To achieve the asymptotic optimality, we attempt to pull each arm i for around w ∗ i ( µ ) T ∗ ( µ ) times. We can show that with a high probability, w ∗ i ( b q ) T ∗ ( b q ) is close to w ∗ i ( µ ) T ∗ ( µ ) , which implies that pulling arm i for a number of times proportional to w ∗ i ( b q ) T ∗ ( b q ) is likely to ensure the asymptotic optimal sample complexity.

Stage II: Exploration using w ∗ ( b q ) and T ∗ ( b q ) . Stage II operates in batches with the maximum number of batches determined by log(1 /δ ) . At batch q , each arm i is pulled max p : p ∈ N ,p ∈ [1 ,q ] T q i times in total. Here

where the definition of w ∗ i ( b q ) and T ∗ ( b q ) could be found in Stage I. We then evaluate the stage switching condition, | w ∗ i ( b q ) -w ∗ i ( b q -1 ) | ≤ 1 / √ n for all i ∈ [ n ] . If this condition is met, we go to the next stage; otherwise, we proceed to the next batch within Stage II.

Purpose. Pulling arm i proportional to w ∗ i ( b q ) T ∗ ( b q ) provides statistical evidence for the reward distributions without sacrificing sample complexity compared to the optimality per our above discussion. Meanwhile, we also set a threshold L 2 to avoid over-exploration due to sampling errors from Stage I.

The rationale for running Stage II in multiple batches is based on empirical considerations. In experiments, δ is always finite. Consequently, the error of arm i , | ˆ µ i ( t ) -µ i | , remains constant since the number of pulls of arms is limited. Given that the stopping rule in Stage III is highly dependent on the error of arm i and the number of pulls T i := max p : p ∈ N ,p ≥ 1 T p i , there is a constant probability that the stopping rule may not be met, leading to significant sample costs in Stage IV. Adding the condition in Line 15 ensures that w ∗ ( b q ) converges and that the sample size T q i , as defined by w ∗ ( b q ) and T ∗ ( b q ) , closely approximates αw ∗ ( µ ) T ∗ ( µ ) log δ -1 . This alignment significantly increases the probability that the stopping rule will be satisfied in experiments.

Moreover, such modification doesn't hurt any theoretical results. To explain, in our analysis for Tri-BBAI, we demonstrate that as δ approaches 0 , w ∗ ( b q ) will be very close to w ∗ ( µ ) and the probability that Line 15 is not satisfied could be bounded by 1 / log 2 δ -1 , which means with high probability Stage II costs 2 batches. Besides, q ≤ log(1 /δ ) , which implies that even if Line 15 is not satisfied, the number of batches required for Stage II is at most log(1 /δ ) . Therefore, the expected number of batches required for Stage II is 2 as δ approaches 0.

Stage III: Statistical test with Chernoff's stopping rule. Denote by N i ( t ) the number of pulls of arm i at time t . For each pair of arms i and j , define their weighted empirical mean as

where ˆ µ i ( t ) and ˆ µ j ( t ) are the empirical means of arms i and j at time t . For ˆ µ i ( t ) ≥ ˆ µ j ( t ) , define

We test whether Chernoff's stopping condition is met (Line 21). If so, we return the arm with the largest empirical mean, i.e., i ∗ ( τ ) , where τ is the total number of pulls examined after Stage II.

Purpose. The intuition of using Chernoff's stopping rule for the statistical test is two-fold. Firstly, if Chernoff's stopping condition is met, with a probability of at least 1 -δ/ 2 , the returned arm i ∗ ( τ ) in Line 22 is the optimal arm (see Lemma B.1). Secondly, when δ is sufficiently small, with high probability, Chernoff's stopping condition holds (see Lemma B.4). As a consequence, our algorithm identifies the best arm successfully with a high probability of meeting the requirement.

Stage IV: Re-exploration. If the previous Chernoff's stopping condition is not met, we pull each arm until the total number of pulls of each arm eqauls L 3 taking into account the pulls in the previous stages (Line 24). Finally, the arm with the largest empirical mean is returned (Line 26).

Purpose. If Chernoff's stopping condition is not met, i ∗ ( τ ) might not be the optimal arm. In addition, when each arm is pulled for L 3 times, we are sufficiently confident that i ∗ ( t ) is the best arm. Since the probability of Stage IV happening is very small, its impact on the sample complexity is negligible.

3.3 Theoretical Guarantees of Tri-BBAI

In the following, we present the theoretical results for Algorithm 1.

Theorem 3.1 (Asymptotic Sample Complexity) . Given any δ > 0 , let ϵ = 1 log log( δ -1 ) , L 1 =

√ log δ -1 , L 2 = log δ -1 log log δ -1 n , and L 3 = (log δ -1 ) 2 . Meanwhile, for any given α ∈ (1 , e/ 2] , define function β ( t, δ ) as β ( t, δ ) = log(log(1 /δ ) t α /δ ) . 5 Then, for any bandit instance µ , Algorithm 1 satisfies

By letting α in Theorem 3.1 approach 1 (e.g., α = 1 + 1 / log δ -1 ), we obtain the asymptotic optimal sample complexity.

̸

Theorem 3.2 (Correctness) . Let ϵ , L 1 , L 2 , L 3 , α , and β ( t, δ ) be the same as in Theorem 3.1. Then, for sufficiently small δ > 0 , Algorithm 1 satisfies P µ ( i ∗ ( N δ ) = 1) ≤ δ .

Theorem 3.3 (Asumptotic Batch Complexity) . Let ϵ , L 1 , L 2 , L 3 , α , and β ( t, δ ) be the same as in Theorem 3.1. For sufficiently small δ > 0 , Algorithm 1 runs within 3 + o (1) batches in expectation. Besides, Algorithm 1 runs within 3 batches with probability 1 -1 / log(1 /δ 2 ) .

To the best of our knowledge, all previous works in the BAI literature [18, 12, 2, 42] that achieve the asymptotic optimal sample complexity require unbounded batches as δ → 0 . In contrast, Tri-BBAI achieves the asymptotic optimal sample complexity and runs within 3 batches in expectation, which is a significant improvement in the batched bandit setting where switching to new policies is expensive.

Remark 3.4. Apart from best arm identification, regret minimization is another popular task in bandits, where the aim is to maximize the total reward in T rounds. Jin et al. [25] proposed an algorithm that achieves a constant batch complexity for regret minimization and showed that their algorithm is optimal when T goes to infinity. In regret minimization, the cost of pulling the optimal arm is 0, indicating that the allocation w i (i.e., the proportion of pulling the optimal arm) is close to 1 . In the BAI problem, the main hardness is to find the allocation w i for each arm since even pulling arm 1 will increase the sample complexity of the algorithm. Therefore, the strategy proposed by Jin et al. [25] cannot be applied to the BAI problem.

4 Best of Both Worlds: Achieving Asymptotic and Non-asymptotic Optimalities

The Tri-BBAI algorithm is shown to enjoy the optimal sample complexity with only three batches in the asymptotic setting. However, in practice, we are limited to a finite number of samples and thus δ cannot go to zero, which is a critical concern in real-world applications. Consequently, obtaining the optimal sample and batch complexity in a non-asymptotic setting becomes the ultimate objective of a practical bandit strategy in BBAI. In this section, we introduce Opt-BBAI, which can attain the optimal sample and batch complexity in asymptotic settings and near-optimal sample and batch complexity in non-asymptotic settings.

We assume a bounded reward distribution within [0 , 1] , which aligns with the same setting in the literature [31, 21]. Again, we consider that the reward distribution belongs to a one-parameter exponential family. By refining Stage IV of Algorithm 1, we can achieve asymptotic optimality and near non-asymptotic optimality adaptively based on the assumption on δ in various settings.

The pseudo-code for the algorithm is provided in Algorithm 2. The main modification from Algorithm 1 occurs in Stage IV. Intuitively, if the algorithm cannot return at Stage III, then the value of log δ -1 may be comparable to other problem parameters, such as 1 / ∆ 2 . Therefore, we aim to achieve the best possible sample and batch complexity for the non-asymptotic scenario. Stage IV operates in rounds, progressively eliminating sub-optimal arms until a result is obtained. Each round consists of two components: Successive Elimination and Checking for Best Arm Elimination .

Successive Elimination. In the r -th round, we maintain a set S r , a potential set for the best arm. Each arm in S r is then pulled d r = 32 /ϵ 2 r log(2 /δ r ) times. At Line 11, all possible sub-optimal arms are eliminated. This first component of Stage IV borrows its idea from successive elimination [14].

Purpose. After d r number of pulls, arms are likely to be concentrated on their true means within a distance of ϵ r / 4 with a high probability. Hence, with high probability, the best arm is never eliminated at every round of Line 11, and the final remaining arm is the best arm.

Issue of Best Arm Elimination. Due to successive elimination, there is a small probability ( ≤ δ r ) that the best arm will be eliminated. The following example illustrates that, conditioning on this small-probability event, the sample and batch complexity of the algorithm could become infinite.

Example 4.1. Consider a bandit instance where µ 1 > µ 2 = µ 3 > µ 4 ≥ · · · ≥ µ n . If the best arm is eliminated at a certain round and never pulled again, the algorithm tasked with distinguishing between the 2nd and 3rd best arms will likely never terminate due to their equal means, leading to unbounded sample and batch complexity.

To address this issue, we introduce a Check for Best Arm Elimination component into Stage IV.

Checking for Best Arm Elimination. In the r -th round, we represent the total sample complexity used up to the r -th round as B r . We employ γ j as an upper bound for the probability that the best arm is eliminated in S j . If Line 16 is true ( B r γ j · 2 ℓ j > B j ), we adjust γ j to γ 2 j and pull each arm in S j for 32 /ϵ 2 j log(2 /γ j ) times (Line 18), subsequently updating their empirical mean (Line 19). Finally, we return a random arm in S r if the condition at Line 21 holds.

Purpose. If Line 16 is satisfied, it indicates that the sample costs, based on the event of the best arm being eliminated in the r -th round, exceed B j / 2 ℓ j . In this case, we increase the number of samples for arms in S j and re-evaluate if their empirical mean is lower than that of the current best arm ∗ . This ensures that the expected total sample costs, assuming the best arm is eliminated at S j , are bounded by ∑ ∞ ℓ j =1 B j / 2 ℓ j ≤ B j . If Line 21 holds, we randomly return an arm from S r . Since this only happens with a small probability, we still guarantee that Algorithm 2 will return the best arm with a probability of at least 1 -δ .

In what follows, we provide the theoretical results for Algorithm 2.

Theorem 4.2. Let ϵ , L 1 , L 2 , L 3 , α , and β ( t, δ ) be the same as in Theorem 3.1. For finite δ ∈ (0 , 1) , Algorithm 2 identifies the optimal arm with probability at least 1 -δ and there exists some universal constant C such that E [ N δ ] ≤ C ( ∑ i> 1 ∆ -2 i log ( n · log ∆ -1 i )) , and the algorithm runs in C log(1 / ∆ 2 ) expected batches.

When δ is allowed to go to zero, we also have the following result.

Theorem 4.3. Let ϵ , L 1 , L 2 , L 3 , α , and β ( t, δ ) be the same as in Theorem 3.1. Algorithm 2 identifies the optimal with probability at least 1 -δ and its sample complexity satisfies lim sup δ → 0 E µ [ N δ ] / log(1 /δ ) ≤ αT ∗ ( µ ) , and the expected batch complexity of Algorithm 2 converges to 3 when δ approaches 0.

The results in Theorems 4.2 and 4.3 state that Algorithm 2 achieves both the asymptotic optimal sample complexity and a constant batch complexity. Moreover, it also demonstrates near-optimal performance in both non-asymptotic sample complexity and batch complexity. Notably, this is the first algorithm that successfully attains the optimal or near-optimal sample and batch complexity, adaptively, in asymptotic and non-asymptotic settings.

Remark 4.4. Jin et al. [22] achieved near-optimal sample and batch complexity in a non-asymptotic setting. However, their results are contingent on the event that the algorithm can find the best arm (with probability 1 -δ ). Consequently, with a probability of δ there is no guarantee for its batch and sample complexity to be bounded. As demonstrated in Example 4.1, the batch and sample complexity in [22] could even be infinite.

In contrast, the batch and sample complexity introduced in Theorem 4.2 is near-optimal and does not rely on any specific event due to the procedure ' checking for best arm elimination ' we proposed. Our technique could be of independent interest and could be further applied to existing eliminationbased BAI algorithms [14, 21, 22, 31] to ensure that the sample and batch complexity is independent of the low-probability event that the best arm is eliminated.

5 Conclusion, Limitations, and Future Work

In this paper, we studied the BAI problem in the batched bandit setting. We proposed a novel algorithm, Tri-BBAI, which only needs 3 batches in expectation to find the best arm with probability 1 -δ and achieves the asymptotic optimal sample complexity. We further proposed Opt-BBAI and theoretically showed that Opt-BBAI has a near-optimal non-asymptotic sample and batch complexity while still maintaining the asymptotic optimality as Tri-BBAI does.

In our experiments, although Tri-BBAI utilizes a limited number of batches, its sample complexity does not match that of Garivier and Kaufmann [17]. Designing a batched algorithm with sample complexity comparable to Garivier and Kaufmann [17], while maintaining a constant number of batches, presents an intriguing challenge.

As for future work, an interesting direction is to investigate whether our 'checking for best arm elimination' could be beneficial to other elimination-based algorithms. Additionally, some research [3, 23] implied a strong correlation between batch complexity in batched bandit, and the memory complexity and pass complexity in streaming bandit. Thus, it could be valuable to assess if our techniques could enhance the results in the field of streaming bandit.

References

Arpit Agarwal, Shivani Agarwal, Sepehr Assadi, and Sanjeev Khanna. Learning with limited rounds of adaptivity: Coin tossing, multi-armed bandits, and ranking from pairwise comparisons. In Proc. COLT , pages 39-75, 2017. (pp. 2 and 5.)

Shubhada Agrawal, Sandeep Juneja, and Peter Glynn. Optimal δ -Correct Best-Arm Selection for Heavy-Tailed Distributions. In Algorithmic Learning Theory , pages 61-110. PMLR, 2020. (pp. 3, 5, and 7.)

Sepehr Assadi and Chen Wang. Exploration with limited memory: streaming algorithms for coin tossing, noisy comparisons, and multi-armed bandits. In Proceedings of the 52nd Annual ACM SIGACT Symposium on theory of computing , pages 1237-1250, 2020. (p. 10.)

Jean-Yves Audibert, Sébastien Bubeck, and Rémi Munos. Best arm identification in multi-armed bandits. In COLT , pages 41-53, 2010. (pp. 1 and 4.)

Dimitris Bertsimas and Adam J Mersereau. A learning approach for interactive marketing to a customer segment. Operations Research , 55(6):1120-1135, 2007. (p. 1.)

Wei Cao, Jian Li, Yufei Tao, and Zhize Li. On top-k selection in multi-armed bandits and hidden bipartite graphs. In Proc. NeurIPS , pages 1036-1044, 2015. (p. 4.)

Alexandra Carpentier and Andrea Locatelli. Tight (lower) bounds for the fixed budget best arm identification bandit problem. In Conference on Learning Theory , pages 590-604. PMLR, 2016. (p. 4.)

Lijie Chen and Jian Li. On the optimal sample complexity for best arm identification. arXiv preprint arXiv:1511.03774 , 2015. (p. 4.)

Lijie Chen, Anupam Gupta, and Jian Li. Pure exploration of multi-armed bandit under matroid constraints. In Proc. COLT , pages 647-669, 2016. (p. 4.)

Lijie Chen, Jian Li, and Mingda Qiao. Towards Instance Optimal Bounds for Best Arm Identification. In Proc. COLT , pages 535-592, 2017. (p. 4.)

Rémy Degenne. On the Existence of a Complexity in Fixed Budget Bandit Identification. arXiv preprint arXiv:2303.09468 , 2023. (p. 5.)

Rémy Degenne, Wouter M Koolen, and Pierre Ménard. Non-asymptotic pure exploration by solving games. Advances in Neural Information Processing Systems , 32, 2019. (pp. 4 and 7.)

Rémy Degenne, Pierre Ménard, Xuedong Shang, and Michal Valko. Gamification of pure exploration for linear bandits. In International Conference on Machine Learning , pages 24322442. PMLR, 2020. (p. 4.)

Eyal Even-Dar, Shie Mannor, and Yishay Mansour. PAC bounds for multi-armed bandit and Markov decision processes. In Proc. COLT , pages 255-270, 2002. (pp. 1, 4, 8, and 9.)

Zijun Gao, Yanjun Han, Zhimei Ren, and Zhengqing Zhou. Batched Multi-armed Bandits Problem. In Proc. NeurIPS , pages 501-511, 2019. (p. 2.)

Aurélien Garivier and Olivier Cappé. The KL-UCB algorithm for bounded stochastic bandits and beyond. In Proc. COLT , pages 359-376, 2011. (p. 5.)

Aurélien Garivier and Emilie Kaufmann. Optimal best arm identification with fixed confidence. In Proc. COLT , pages 998-1027, 2016. (pp. 1, 2, 3, 4, 5, 7, 10, 13, 25, and 26.)

Aurélien Garivier, Tor Lattimore, and Emilie Kaufmann. On explore-then-commit strategies. In Proc. NeurIPS , pages 784-792, 2016. (p. 7.)

Aditya Grover, Todor Markov, Peter Attia, Norman Jin, Nicolas Perkins, Bryan Cheong, Michael Chen, Zi Yang, Stephen Harris, William Chueh, et al. Best arm identification in multi-armed bandits with delayed feedback. In Proc. AISTATS , pages 833-842, 2018. (p. 1.)

Peter Harremoës. Bounds on tail probabilities in exponential families. arXiv preprint arXiv:1601.05179 , 2016. (p. 18.)

Kevin Jamieson, Matthew Malloy, Robert Nowak, and Sébastien Bubeck. lil'ucb: An optimal exploration algorithm for multi-armed bandits. In Proc. COLT , pages 423-439, 2014. (pp. 2, 4, 8, and 9.)

Tianyuan Jin, Shi Jieming, Xiaokui Xiao, and Enhong Chen. Efficient Pure Exploration in Adaptive Round model. In Proc. NeurIPS , pages 6605-6614, 2019. (pp. 2, 3, 5, 9, and 25.)

Tianyuan Jin, Keke Huang, Jing Tang, and Xiaokui Xiao. Optimal streaming algorithms for multi-armed bandits. In International Conference on Machine Learning , pages 5045-5054. PMLR, 2021. (pp. 10, 25, and 26.)

Tianyuan Jin, Jing Tang, Pan Xu, Keke Huang, Xiaokui Xiao, and Quanquan Gu. Almost optimal anytime algorithm for batched multi-armed bandits. In International Conference on Machine Learning , pages 5065-5073. PMLR, 2021. (pp. 2, 3, and 15.)

Tianyuan Jin, Pan Xu, Xiaokui Xiao, and Quanquan Gu. Double explore-then-commit: Asymptotic optimality and beyond. In Conference on Learning Theory , pages 2584-2633. PMLR, 2021. (pp. 2 and 7.)

Marc Jourdan and Rémy Degenne. Non-Asymptotic Analysis of a UCB-based Top Two Algorithm. arXiv preprint arXiv:2210.05431 , 2022. (pp. 3 and 4.)

Marc Jourdan, Rémy Degenne, Dorian Baudry, Rianne de Heide, and Emilie Kaufmann. Top two algorithms revisited. arXiv preprint arXiv:2206.05979 , 2022. (p. 4.)

Marc Jourdan, Rémy Degenne, and Emilie Kaufmann. An varepsilon -Best-Arm Identification Algorithm for Fixed-Confidence and Beyond. Advances in Neural Information Processing Systems , 36:16578-16649, 2023. (p. 4.)

Shivaram Kalyanakrishnan and Peter Stone. Efficient selection of multiple bandit arms: theory and practice. In Proc. ICML , pages 511-518, 2010. (p. 4.)

Shivaram Kalyanakrishnan, Ambuj Tewari, Peter Auer, and Peter Stone. PAC Subset Selection in Stochastic Multi-armed Bandits. In Proc. ICML , pages 655-662, 2012. (p. 4.)

Zohar Karnin, Tomer Koren, and Oren Somekh. Almost optimal exploration in multi-armed bandits. In Proc. ICML , pages 1238-1246, 2013. (pp. 2, 3, 4, 8, and 9.)

Nikolai Karpov, Qin Zhang, and Yuan Zhou. Collaborative top distribution identifications with limited interaction. In 2020 IEEE 61st Annual Symposium on Foundations of Computer Science (FOCS) , pages 160-171. IEEE, 2020. (pp. 3, 5, 25, and 26.)

Julian Katz-Samuels, Lalit Jain, Kevin G Jamieson, et al. An empirical process approach to the union bound: Practical algorithms for combinatorial and linear bandits. Advances in Neural Information Processing Systems , 33:10371-10382, 2020. (p. 4.)

Emilie Kaufmann and Wouter M Koolen. Mixture martingales revisited with applications to sequential tests and confidence intervals. The Journal of Machine Learning Research , 22(1): 11140-11183, 2021. (p. 7.)

Junpei Komiyama, Taira Tsuchiya, and Junya Honda. Minimax optimal algorithms for fixedbudget best arm identification. Advances in Neural Information Processing Systems , 35: 10393-10404, 2022. (p. 5.)

Pierre Ménard and Aurélien Garivier. A minimax and asymptotically optimal algorithm for stochastic bandits. In Proc. ALT , pages 223-237, 2017. (pp. 5, 15, and 18.)

Vianney Perchet, Philippe Rigollet, Sylvain Chassang, and Erik Snowberg. Batched bandit problems. The Annals of Statistics , 44(2):660-681, 2016. (p. 2.)

Xuanfei Ren, Tianyuan Jin, and Pan Xu. Optimal Batched Linear Bandits. In Forty-first International Conference on Machine Learning , volume 235, pages 42391-42416. PMLR, 2024. (p. 2.)

Chao Tao, Qin Zhang, and Yuan Zhou. Collaborative learning with limited interaction: Tight bounds for distributed exploration in multi-armed bandits. In 2019 IEEE 60th Annual Symposium on Foundations of Computer Science (FOCS) , pages 126-146. IEEE, 2019. (pp. 2, 3, and 5.)

William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika , 25(3/4):285-294, 1933. (p. 1.)

Roman Vershynin. High-dimensional probability: An introduction with applications in data science , volume 47. Cambridge university press, 2018. (p. 20.)

Po-An Wang, Ruo-Chun Tzeng, and Alexandre Proutiere. Fast pure exploration via frank-wolfe. Advances in Neural Information Processing Systems , 34:5810-5821, 2021. (pp. 3, 4, 5, and 7.)

Yuan Zhou, Xi Chen, and Jian Li. Optimal PAC multiple arm identification with applications to crowdsourcing. In Proc. ICML , pages 217-225, 2014. (p. 1.)

Notations.

Denote by ˆ µ s i the empirical mean of arm i after its s -th pull and by ˆ µ i ( t ) the empirical mean of arm i at time t (i.e., after a total number of t pulls are examined for all arms). We use ∆ i = µ 1 -µ i for the gap between the optimal arm and the i -th arm. Throughout the paper, we use the following asymptotic notation. Specifically we use f ( δ ) ≲ g ( δ ) to denote that there exists some δ 0 , such that for any δ < δ 0 , f ( δ ) ≤ g ( δ ) . Similarly, we use f ( δ ) ≳ g ( δ ) to denote that there exists some δ 0 , such that for any δ < δ 0 , f ( δ ) ≥ g ( δ ) .

A Computing w ∗ ( µ ) and T ∗ ( µ )

In this section, we introduce two useful lemmas that help us to calculate w ∗ ( b ) and T ∗ ( b ) in Algorithm 1 and Algorithm 2, which are also used in [17]. For ease of presentation, we first define some useful notations. For every c ∈ [0 , 1] , we define

For any arm i ∈ [ n ] \ { 1 } , we define

Lemma A.1 (Garivier and Kaufmann 17, Lemma 3) . For every w ∈ P n ,

̸

̸

̸

Lemma A.2 (Garivier and Kaufmann 17, Theorem 5) . For i ∈ [ n ] , let

where y ∗ is the unique solution of the equation F µ ( y ) = 1 , and where

in a continuous, increasing function on [0 , d ( µ 1 , µ 2 )) such that F µ ( y ) →∞ when y → d ( µ 1 , µ 2 ) .

B Proof of Theorems in Section 3

B.1 Proof of Theorem 3.2

Proof of Theorem 3.2. The proof of Theorem 3.2 requires the following two Lemmas, which guarantees the error returned by Line 22 and Line 26 of Algorithm 1 respectively.

Lemma B.1. [17, Proposition 12] Let µ be the exponential bandit model. Let δ ∈ (0 , 1) . For sufficiently small δ > 0 , the Chernoff's stopping rule Line 21 of Algorithm 1 with the threshold

ensures that P µ ( i ∗ ( N δ ) = 1) ≤ δ/ 2 .

̸

Recall ˆ µ s i is the empirical mean of arm i after its s -th pull. Let E 1 = {∀ s ≥ L 3 and i ∈ [ n ] : ˆ µ s i ∈ [ µ i -ϵ, µ i + ϵ ] } . The following lemma shows that P ( E c 1 ) ≤ δ/ 2 , where E c denotes the complement event of E .

Besides, and

Lemma B.2. For sufficiently small δ > 0 6 , we have

If E 1 happens, we have that at Line 26 of Algorithm 1, ˆ µ 1 ( t ) ≥ µ 1 -ϵ ≳ µ i + ϵ ≥ ˆ µ i ( t ) for all i > 1 , which means i ∗ ( t ) = 1 . Combining Lemma B.1 and Lemma B.2, Theorem 3.1 follows immediately.

B.2 Proof of Theorem 3.1

Proof of Theorem 3.1. Recall in Algorithm 1, b q 1 = ˆ µ i ∗ ( t ) ( t ) -ϵ and b q i = ˆ µ i ( t ) + ϵ for all i ∈ [ n ] \ { i ∗ ( t ) } , where t = τ q . Let E 0 be the intersection of the following four events:

Here (B.1a) and (B.1b) ensure that the estimation of w ∗ ( b q ) and T ∗ ( b q ) is close to w ∗ ( µ ) and T ∗ ( µ ) , and (B.1c) and (B.1d) ensure that arm 1 has the largest average reward at time τ q .

Lemma B.3. For sufficiently small δ > 0 , we have

Lemma B.4. If E 0 holds, then for sufficiently small δ > 0 , we have

From Lemma B.4, if δ is sufficiently small and E 0 holds, then Algorithm 1 will return at Line 22. Besides, we note that the total number of pulls of any arm is no more than L 3 = (log( δ -1 )) 2 times. Therefore,

We further have

This completes the proof.

B.3 Proof of Theorem 3.3

Proof of Theorem 3.3. Stage I costs one batch. For the Stage II, note that P ( E c 0 ) ≤ 1 log( δ -1 ) 2 and if E 0 occurs, then | w ∗ i ( b 2 ) -w ∗ i ( b 1 ) | ≤ 1 / √ n for all i ∈ [ n ] (from the second statement of Lemma B.4), which means Stage II costs 2 batches. Moreover, from Lemma B.4, if δ is sufficiently small and E 0 holds, then Algorithm 1 will return at Line 22. Otherwise, the algorithm goes to Stage IV and costs one batch. Therefore, for sufficiently small δ , the expected number of batches

Finally, if E 0 is true, the algorithm costs 3 batches. Therefore, with probability 1 -1 / log(1 /δ 2 ) , Algorithm 1 costs 3 batches.

B.4 Proof of Supporting Lemmas

The proof of Lemma B.3 requires the following useful inequalities.

Lemma B.5 (Maximal Inequality) . Let N and M be two positive integers, let γ > 0 , and ˆ µ n be the empirical mean of n random variables i.i.d. according to the arm's distribution with mean µ . Then, for x ≤ µ ,

where V 0 is the maximum variance of arm's distribution with mean µ ∈ [ x, µ ] and V 1 is the maximum variance of arm's distribution with mean µ ∈ [ µ, x ] .

Note that Lemma B.5 is an improved version of Lemma 4 of [36], where we use a much smaller variance upper bound to tighten the inequalities for the case x ≤ µ and the case x ≥ µ respectively.

Lemma B.6. [24, Proposition 1] For ϵ > 0 and µ ≤ µ ′ -ϵ ,

Now, we are ready to prove Lemma B.3.

Proof of Lemma B.3. Let V be the maximum variance of reward distribution with mean µ ∈ [ µ n , µ 1 ] . From Lemma B.5, we have that for s ≥ L 1 ,

where the second inequality is from Lemma B.5, and the last inequality is due to that for sufficiently small δ > 0 , L 1 = √ log δ -1 ≥ 8 V/ϵ 2 log( n (log δ -1 ) 2 ) . Similarly, for i ∈ [ n ] \ { 1 } ,

Define events:

and

Assume that both events A 1 and A 2 hold. Then, we have

̸

where in the last inequality we assumed ϵ ≤ min i =1 ∆ i . This further implies i ∗ ( τ q ) = 1 for any q ≥ 0 and thus b q -1 1 = ˆ µ 1 ( τ q ) -ϵ . Therefore, we have for any q ≥ 1

and for every x ≥ µ ,

which means E 0 defined in (B.1) occurs. Therefore, we have

where the last inequality is due to (B.6) and (B.7).

Proof of Lemma B.4. Assume E 0 is true. Recall that ∆ 2 = min i ∈ [ n ] \{ 1 } µ 1 -µ i . For sufficiently small δ > 0 such that ϵ < ∆ 2 / 8 , we have b q 1 ∈ [ µ 1 -∆ 2 / 4 , µ 1 ] and b q i ∈ [ µ i , µ i + ∆ 2 / 4] . Let L ( b ) = { λ ∈ S : λ 1 ∈ [ µ 1 -∆ 2 / 4 , µ 1 ] and for any i ∈ [ n ] \{ 1 } , λ i ∈ [ µ i , µ i +∆ 2 / 4] } . Therefore

Since max b ′ ∈L ( b ) αw ∗ i ( b ′ ) T ∗ ( b ′ ) < ∞ is independent of δ , we have

which implies T q i = min { αw ∗ i ( b q ) T ∗ ( b q ) log δ -1 , L 2 } = αw ∗ i ( b q ) T ∗ ( b q ) log δ -1 . Besides, as w ∗ ( b q ) and T ∗ ( b q ) are continuous on b q , we have that as b q → µ , w ∗ ( b q ) → w ∗ ( µ ) and T ∗ ( b q ) → T ∗ ( µ ) . Note that | b q i -µ i | ≤ 2 ϵ and ϵ approaches 0 as δ approaches 0 . Recall τ is the stopping time of Stage II and T i is the number of pulls of arm i at time τ . Therefore, for sufficiently small δ ,

Moreover, for sufficient smaller δ and ∀ i ∈ [ n ] , | w ∗ i ( b q ) -w ∗ i ( µ ) | = o (1) and thus

which means the condition in Line 15 is satisfied and the Algorithm goes to Stage III.

Assume τ = τ q ′ . We have

where the last inequality is due to T i ≥ T q ′ i = αw ∗ i ( b q ′ ) T ∗ ( b q ′ ) log(1 /δ ) . In what follows, we will show

The following minimization problem is a convex optimization problem

which is solved when we have λ 1 w = λ iw = ˆ µ 1 i ( τ ) . The proof of this lemma is based on the assumption that E 0 occurs. Note that E 0 occurs, then b q ′ 1 = ˆ µ 1 ( τ q ′ -1 ) -ϵ ≤ µ 1 -ϵ/ 2 ≤ ˆ µ 1 ( τ ) and b q ′ i = ˆ µ 1 ( τ q ′ -1 ) + ϵ > ˆ µ i ( τ ) for all i ∈ [ n ] \ { 1 } . Therefore, ˆ µ 1 ( τ ) ≥ b q ′ 1 ≳ b q ′ i ≥ ˆ µ i ( τ ) . Let

Then, λ w ∗ ( b ) ( b q ′ 1 , b q ′ i ) is the solution to minimizing w ∗ 1 ( b q ′ ) · d ( b q ′ 1 , x ) + w ∗ i ( b q ′ ) · d ( b q ′ i , x ) for x ∈ ( b q ′ i , b q ′ 1 ) .

Case 1: if ˆ µ 1 i ( τ ) ∈ ( b q ′ 1 , ˆ µ 1 ( τ )) , then

where the first and second inequalities are due to (B.5) and the last inequality is due to the fact that w ∗ 1 ( b q ′ ) · d ( b q ′ 1 , x ) + w ∗ i ( b q ′ ) · d ( b q ′ i , x ) achieves it minimum at x = λ w ∗ ( b q ′ ) ( b q ′ 1 , b q ′ i ) .

Case 2: if ˆ µ 1 i ( τ ) ∈ ( b q ′ i , b q ′ 1 ) , we have

where the first inequality is due to (B.5) and the last inequality is due to the fact that for x ∈ ( b q ′ i , b q ′ 1 ) , w ∗ 1 ( b q ′ ) · d ( b q ′ 1 , x ) + w ∗ i ( b q ′ ) · d ( b q ′ i , x ) achieves its minimum at x = λ w ∗ ( b q ′ ) ( b 1 . . . . . . q ′ , b q ′ i ) .

Case 3: if ˆ µ 1 i ( τ ) ∈ (ˆ µ i ( τ ) , b q ′ i ) , similar to (B.9) and (B.10), we have

Combine all three cases, we obtain

Note that ˆ µ 1 i ( τ ) = λ w ∗ ( b q ′ ) (ˆ µ 1 ( τ ) , ˆ µ i ( τ )) , we have

where the first inequality is due to (B.12) and the last inequality is due to (A.1). Note that conditioned on event E 0 , for sufficiently small δ > 0 , we have 1 = i ∗ ( τ ) . Therefore, for any i ∈ [ n ] \ { 1 } ,

where the first equality is from (3.3), the second equality is from (B.8), and the last inequality is from (B.13). Finally, for α ≥ 1 + 6 log log δ -1 / log δ -1 , we obtain

which completes the proof.

B.5 Proof of Maximum Inequality

Proof of Lemma B.5. Recall that the result of Lemma 4 of Ménard and Garivier [36] is: for ˆ µ n ≤ µ , -Nγ

By Lemma 1 of Harremoës [20], we have

where V ( y ) is the variance of the distribution with mean y . As a simple consequence of (B.15) and (B.14), we have

For the case when ˆ µ n ≥ µ , we can directly follow the idea of Lemma 4 of Ménard and Garivier [36]. For the case ˆ µ n ≥ µ , we aim to show

We let M ( λ ) be the log-moment generating function of ν θ . Recall that d ν θ d ρ ( x ) = exp( xθ -b ( θ )) . We use the following properties of one exponential family.

Let E [ ν θ ] = µ . Let λ = θ 1 -θ , z = b ′ ( θ 1 ) > µ , and γ = d ( z, µ ) , we have

Since d ( z, µ ) is monotone increasing for z > µ , we obtain λ > 0 . If event {∃ N ≤ n ≤ M,d (ˆ µ n , µ ) ≥ γ } and ˆ µ n ≥ µ , one have

By Doob's maximal inequality for the exponential martingale exp( λn ˆ µ n -nM ( λ )) ,

Combining above inequality and (B.15), we obtain for ˆ µ n ≥ µ

This completes the proof.

C Proof of Theorems in Section 4

In this proof, we define 'one round" as a single iteration of the While Loop of Algorithm 2.

Proof of Theorem 4.2. Proof of Correctness. We first show that the best arm is not eliminated by Line 11 for any round r . For any l , let E l be defined as follows.

Lemma C.1. Line 11 of Algorithm 2 satisfies

To streamline our presentation, we define ℓ js = s , γ js = ( δ j ) 2 s , and ˆ p js i as the s th updates of parameters ℓ j , γ j , and ˆ p j i , respectively, within the second For Loop . For any fixed j , we define U j ( s ) to be the following set of rounds, where ℓ j remains to be the same value s :

Then the condition of Line 21 could be represented as

The following lemma shows that with high probability, Algorithm 2 will not return at Line 22.

Lemma C.2. Line 21 of Algorithm 2 satisfies

Note that Algorithm 2 will return arm 1 at Line 24 if 1: Arm 1 is always maintained in S r , and 2: Algorithm 2 never return at Line 22.

According to Lemma C.1, there is a probability of at least 1 -δ/ 8 that Arm 1 will never be eliminated at Line 11. Given that Arm 1 is never eliminated at Line 11, Lemma C.2 suggests that there is a probability of at least 1 -3 δ/ 16 that we will never loop back at Line 22. Hence, with a probability of 1 -δ/ 2 , Stage IV will identify the optimal arm. By incorporating the results of Lemma B.1, we find that with a probability exceeding 1 -δ/ 2 -δ/ 2 , which is greater than 1 -δ , Algorithm 2 will return the optimal arm.

Proof of Sample Complexity. Let N ′ be the sample complexity of Stage IV. Let E = ∩ r> 1 E r . The following lemma shows the sample complexity conditioned on event E .

Lemma C.3. The expected sample complexity N ′ conditioned on event E has the order

Now, we consider the expected sample complexity if the best arm is eliminated at some round r . We prove the following lemma.

Lemma C.4. The sum of expected sample complexity N ′ at each loop has the order

The number of pulls of all arms in the first three stages is no more than max { n √ log δ -1 , L 2 } . By combining the above results, Lemma C.3 and Lemma C.4 together, we can obtain the following total sample complexity

Since in the non-asymptotic setting, δ is finite. Then

Proof of Batch Complexity. Just as we discussed in the proof of sample complexity, we first demonstrate the batch complexity conditioned on the event of E .

Lemma C.5. Conditional on event E , Algorithm 2 conducts O (log(1 / ∆ -1 2 )) batches in expectation.

Let B ′ be the number of batches we used in Stage IV. We consider the expected batch complexity if the best arm is eliminated at some round r . We prove the following lemma.

Lemma C.6. The sum of expected batch complexity B ′ at each loop has the order

By combining Lemma C.5, Lemma C.6, the fact that the first three stages use O (log(1 /δ )) batches, and δ is a constant in non-asymptotic setting, the batch complexity of Algorithm 2 is O (log(1 / ∆ -1 2 )) .

Proof of Theorem 4.3. Let E 0 be the event defined in (B.1) in the proof of Theorem 3.1. From Lemma B.4, if δ is sufficiently small and E 0 holds, then Algorithm 2 will return at Stage III. Note that the expected number of pulls in Stage IV is ≲ (log δ -1 ) 2 . Hence, the expected number of pulls of all arms is no more than 2(log δ -1 ) 2 times. Therefore, for any ϵ ′ > 0

Therefore, where ˆ µ = 1 /k ∑ k t =1 X t .

D.1 Proof of Lemma C.1

From Lemma D.1,

Applying union bound, we have

Moreover, there exists some universal constant C , such that the expected number of batches used in Stage IV is ≲ C log(1 / ∆ 2 ) · 1 {E c 0 } ≲ o (1) , where the last inequality is due to Lemma B.3. Form Theorem 3.3, the batch complexity of the first three Stage is 3 + o (1) . Therefore, the total expected batch complexity is 3 + o (1) .

D Proof of Supporting Lemmas

The proof of the supporting lemmas requires the following concentration inequalities.

Lemma D.1. [41, Theorem 2.2.6] Let X 1 , . . . , X k ∈ [0 , 1] be independent bounded random variables with mean µ . Then for any ϵ > 0

The aforementioned inequality suggests that, with a probability of at least 1 -δ/ 8 , the condition | ˆ p r i -µ i | ≤ ϵ r 8 holds true for any i ∈ [ n ] and r ≥ 1 . Then, for any r > 1 ,

Applying union bound, we have

Besides, from (D.3), we obtain

If E 3 and E 4 truly hold, we have that (1): for any j and arm i ∈ S j ,

where the second inequality is due to Line 11 of Algorithm 2; (2): for any fixed j , s , and any r ∈ U j ( s ) , where U j ( s ) is defined in (C.2) and represents the set of all rounds in which parameter ℓ j is updated for exactly s times,

Therefore, if E 3 and E 4 truly hold, the event ˆ p js i < ˆ p r ∗ -ϵ j 2 consistently holds for all j , s , i ∈ S j , and r ∈ U j ( s ) . It's noteworthy that, according to (D.4), P ( E 4 ) ≥ 1 -δ/ 16 , and from (D.5), P ( E 3 ) ≥ 1 -δ/ 8 . Therefore,

Therefore,

D.2 Proof of Lemma C.2

From Lemma D.1,

Define events and

D.3 Proof of Lemma C.3

We first focus on bounding the number of pulls of arm i within the first For Loop . Define

From Lemma D.1 and the union bound, for r ≥ r ( i ) ,

Let E r i be the event | ˆ p r i -µ i | ≤ ϵ r ( i ) / 8 and | ˆ p r 1 -µ 1 | ≤ ϵ r ( i ) / 8 truly hold at r -th round. Conditioned on E r i and E ,

which mean arm i will be eliminated. Therefore, the total sample cost of arm i in the first For Loop of Algorithm 2 is

Consequently, if we define H to be the total sample complexity in the first For Loop of Algorithm 2. Then we have

Now, we bound the sample complexity within the second For Loop . We let L j be the total number of pulls of arms in S j \ S j +1 within the second For Loop . We let

As per Line 16, n j represents the total count of arm re-pulls in the set S j \ S j +1 . Consequently, we can establish the following bound for L j .

where the first inequality is because γ j ( s +1) = γ 2 js and we pull the arms in S j /S j +1 for total

times at Line 18 and the last inequality is because for n j > 1 ,

where the first inequality is due to the definition of n j . Therefore,

Now, we conclude that conditioned on event E , the total expected sample complexity is

D.4 Proof of Lemma C.4

From Lemma D.1,

Applying union bound, we have

The aforementioned inequality suggests that, with a probability of at least 1 -( δ r ) 3 , the condition | ˆ p r i -µ i | ≤ ϵ r / 4 consistently holds true for any i ∈ S r . Then,

For any fixed j , recall the definition of U j ( s ) in (C.2). Assume E r , ( E r +1 ) c truly hold and denote U r ( s ) = { r 1 s , r 2 s , · · · , r s ′ s } as the rounds where ℓ r = s . We note that at r l s -th round, each arm within the first For Loop of Algorithm 2 is pulled at least

times. Besides, each arm within the second For Loop is pulled at least

times. Then, similar to (D.7), for r l s -th round, we have

Then, we apply a union bound over all rounds in U r ( s ) , we obtain

Moreover, if E r , ( E r +1 ) c happens, the best arm is eliminated at r -th round. Then for any rounds in U r ( s ) , we have pulled arm 1 for at least

times in the second For Loop . Recall that ˆ p js i is the s th updates of parameter ˆ p j i within the second For Loop. From Lemma D.1, we have that

Therefore, if E r , ( E r +1 ) c happens, with probability

Therefore, it holds that

and thus for all l ∈ [ s ′ ]

Finally,

D.5 Proof of Lemma C.5

We note that each round within the While Loop costs one batch. Let

Let E r be the event

which means Algorithm 2 returns at Line 22. Before we continue, we will first show that the number of pulls in the second For Loop is lower than the first For Loop . Assume the algorithm stops at r ′ -th round. Let s ′ j = max { s : B r ′ γ js · 2 s > B j } . The number of pulls for S j \ S j +1 at second For Loop is at most

Therefore, the total number of pulls within the second For Loop is at most

which means the number of for the second For Loop is lower than the first For Loop . Finally, we obtain

In the first inequality, we used the fact that if the algorithm stops in some rounds in U r ( s ) , the total number of pulls of all arms is at most B r / ( γ js · 2 s ) + B r / ( γ js · 2 s ) , whcih comes from the first and second For Loop respectively. Moreover, the factor (( δ r ) 3 ) 2 s -1 is because from (D.9), if E r , ( E r +1 ) c holds, then Algorithm 2 returns in some rounds in U r ( s ) (at Line 22) with probability at least 1 -(( δ r ) 3 ) 2 s -1 . The last equality is because from Lemma C.3, if E r , ( E r +1 ) c holds, then

Conditioned on E r and E ,

which means all sub-optimal arms have been eliminated and the algorithm returns. From Lemma D.1 and the union bound, for r ≥ r (2) ,

Therefore, the total batch cost is

This completes the proof.

D.6 Proof of Lemma C.6

The proof of this lemma is similar to that of Lemma C.4. we have the following results. 1: From (D.8), we have

2: From (D.9), we have if E r and ( E r +1 ) c truly hold, then with fixed ℓ s , Algorithm 2 returns at Line 22 with probability

We first compute the size of U r ( s ) . From Algorithm 2, we know that the arm kept in the set S r will be pulled 4 times larger compared to ( r -1) -th round. Besides, we update ℓ r to s +1 , if the number of pulls exceeds B r / (( δ r ) 2 s · 2 s ) (Line 16 of Algorithm 2). It is easy to see after log 4 n rounds, the number of pulls of any arm exceeds B r and then after ln 1 ( δ r ) 2 s · 2 s rounds, the number of pulls of single arm exceeds B r / (( δ r ) 2 s · 2 s ) . Therefore, the size of U r ( s ) is at most ln 1 ( δ r ) 2 s · 2 s +ln n . We let U be the total number of rounds used. Then, we obtain

We have shown in Lemma C.5, ∑ r> 1 r P ( E r , ( E r +1 ) c ) = O (log(1 / ∆ 2 )) . Therefore, the total number of rounds is

which completes the proof.

E Experiments

In this section, we compare our algorithms Tri-BBAI and Opt-BBAI with Track-and-Stop [17], Top-k δ -Elimination [22], ID-BAI [23] and CollabTopM [32] under bandit instances with Bernoulli rewards. All the experiments are repeated in 1000 trials. We perform all computations in Python on R9 5900HX for all our experiments. The implementation of this work can be found at https://github.com/panxulab/Optimal-Batched-Best-Arm-Identification

Data generation. For all experiments in this section, we set the number of arms n = 10 , where each arm has Bernoulli reward distribution with mean µ i for i ∈ [10] . More specifically, the mean rewards are generated by the following two cases.

Uniform: The best arm has µ 1 = 0 . 5 , and the mean rewards of the rest of the arms follow uniform distribution over [0 . 2 , 0 . 4] , i.e., µ i is uniformly generated from [0 . 2 , 0 . 4] for i ∈ [ n ] \ { 1 } .

Normal: The best arm has µ 1 = 0 . 6 , and the mean rewards of the rest of the arms are first generated from normal distribution N (0 . 2 , 0 . 2) and then projected to the interval [0 , 0 . 4] .

Implementation details. The hyperparameters of all methods are chosen as follows.

Track-and-Stop [17] is a fully sequential algorithm and thus the only parameter that needs to be set is the β ( t ) function in the Chernoff's stopping condition (similar to Stage III of Algorithm 1). Note that the theoretical value of β ( t ) in Track-and-Stop [17] is of the same order as presented in our Theorem 3.1. However, they found that a smaller value works better in practice. Therefore, we follow their experiments to set β = log ((log( t ) + 1) /δ ) .

Top-k δ -Elimination is a batched algorithm that eliminates the arms in batches. It has two parameters ϵ and δ . In our experiments, we fix ϵ = 0 . 1 .

ID-BAI [23] is designed to identify the best arm among a set of bandit arms with optimal instancedependent sample complexity. We use the same algorithm setting of the original paper in our experiments.

CollabTopM [32] is the algorithm to identify the Topm arms within a multi-agent setting. We set the m as 1 and the agents K as 1.

For Tri-BBAI and Opt-BBAI, we set α = 1 . 001 7 , and ϵ = 0 . 01 . We use the same β ( t ) function for Chernoff's stopping condition as in Track-and-Stop. Moreover, for the lengths of the batches, we set L 1 , L 2 and L 3 to be the value calculated by Theorem 3.1.

Results. We present a comprehensive comparison on the sample complexities and batch complexities of our algorithms and baseline algorithms in Tables 3 and 4. Notably, our algorithms Tri-BBAI and Opt-BBAI, also including Top-K δ Elimination and CollabTopM, require significantly fewer batches than Track-and-Stop. Furthermore, the sample complexity of Tri-BBAI and Opt-BBAI is significantly lower than that of Top-K δ Elimination, ID-BAI and CollabTopM. Additionally, the sample complexity of Tri-BBAI and Opt-BBAI is comparable to Track and Stop when δ is large, and it is at most 3.6 times greater than Track and Stop when δ is very small. Additionally, we also provide the runtime comparison in Tables 3 and 4. Our algorithms have a significantly reduced runtime compared to Track-and-Stop, achieving nearly 1000 × speedup.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims presented in the abstract and introduction accurately represent the contributions and scope of the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See the discussion in Section 5.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide the full set of assumptions and complete proofs.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer:[Yes]

Justification: We include all information needed to reproduce the main experimental results in Appendix E.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer:[Yes]

Justification: We have uploaded my source and will make the full code public if this work gets accepted.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have included all the test details in Appendix E.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report the error range in Tables 3 and 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We state all information on the computer resources in Appendix E.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conforms, in every respect, to the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA] .

Justification: There is no societal impact of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA] .

Justification: The paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Our paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA] .

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the Batched Best Arm Identification (BBAI) problem in multi-armed bandits. The goal is to design an efficient algorithm that correctly finds the arm with the highest mean with probability $\\ge 1 - \\delta$, while minimizing: (1) the sample complexity, defined as the total number of arm pulls; (2) the \"batch complexity\", defined as the number of rounds in which the algorithm requests arm pulls. (Formally, the algorithm specifies the number of times it pulls each arm in the $r$-th round after observing all the outcomes in the previous $r-1$ rounds.)\n\nThe main results of this work are:\n- An algorithm (termed Tri-BBAI) that achieves an asymptotically (i.e., as $\\delta \\to 0^{+}$) optimal sample complexity in three batches in expectation.\n- Another algorithm (termed Opt-BBAI) that achieves: (1) near-optimal sample and batch complexities in the non-asymptotic setting; (2) the same guarantees as Tri-BBAI in the asymptotic setting.\n\nSTRENGTHS:\n- This work studies a fairly natural problem. While the setup is not new, I liked that the authors explored certain perspectives that are different from prior work, namely: (1) asymptotic optimality of sample complexity; (2) focusing on the expected number of rounds, rather than treating it as a hard constraint.\n- The results are pretty strong: The algorithms achieve asymptotically optimal sample complexities within a (small) constant number of rounds (in expectation).\n- The presentation is clear in general. I found the main paper fairly easy to follow.\n\nWEAKNESSES:\nI think a major weakness of the work is that its main result is fairly intuitive, and arguably, its proof does not give new insights to the design of pure exploration algorithms. Here is why: For simplicity, suppose that we only have two arms with means $1/2+\\epsilon$ and $1/2-\\epsilon$, and the parameter $\\epsilon > 0$ is *unknown*.\n\nIn this case, the \"natural\" algorithm is to make geometrically decreasing guesses on $\\epsilon$: $\\epsilon_1, \\epsilon_2, \\ldots$, where each $\\epsilon_k = 2^{-k}$. At guess $\\epsilon_k$, we would pull each arm $(1/\\epsilon_k)^2$ times to get an $O(\\epsilon_k)$-approximation of the means. If the means are clearly separated, we get the answer; otherwise, we continue with smaller guesses. Clearly, this approach requires many rounds ($\\Omega(\\log(1/\\epsilon))$ rounds) of adaptivity.\n\nOn this instance, very roughly speaking, the authors' approach is to start with the guess, say, $\\epsilon_k = 1/\\log^{1/3}(1/\\delta)$. For each fixed $\\epsilon$, when $\\delta$ is small enough, the guess would be smaller than the actual $\\epsilon$, and the algorithm wins the game in $O(1)$ rounds. Also, this first round only takes $(1/\\epsilon_k)^2 = \\log^{2/3}(1/\\delta) \\ll \\log(1/\\delta)$ samples, so this will not affect the asymptotic behavior as $\\delta \\to 0^{+}$.\n\nIn general, the Tri-BBAI algorithm uses a round-robin strategy as an inefficient exploration round. The length of this round is chosen as a function of $\\delta$, so that the asymptotic behavior is not affected. Therefore, for each fixed instance, there exists some $\\delta_0 > 0$ such that whenever $\\delta < \\delta_0$, this inefficient exploration succeeds with a good probability, and may guide the algorithm to sample in an asymptotically optimal way in the rest. As a result, this asymptotic optimality might only hold for extremely small values of $\\delta$: From Line 492, the analysis needs $1/\\log\\log(1/\\delta) = \\epsilon \\le \\Delta_2$ to go through; in other words, $1/\\delta$ needs to be **doubly exponential** in $1/\\Delta_2$.\n\nAdmittedly, this weakness was addressed by the other algorithm Opt-BBAI, which achieves a sample complexity bound for finite $\\delta$ as well. However, it should be noted that the complexity contains a $\\sum_{i=2}^{n}(1/\\Delta_i)^2\\log n$ term, which could be higher than the optimal sample complexity by a $\\log n$ factor. In contrast, the state-of-the-art bounds for (non-batched) BAI (e.g., [Karnin-Koren-Somekh, ICML'13][Jamieson-Malloy-Nowak-Bubeck, COLT'14][Chen-Li, arXiv'15][Chen-Li-Qiao, COLT'17]) are tight up to a doubly-logarithmic factor.\n\nDespite the weakness mentioned above, I think this submission presents some solid work and a nice observation (Lines 72-73), namely, the need for many rounds of adaptivity only arises when $\\delta$ is small (or moderate), and goes away when we focus on the asymptotic regime. Therefore, I lean towards accepting the paper.\n\nQUESTIONS:\nI don't have specific questions for the authors.\n\nLIMITATIONS:\nLimitations have been adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-01"}

comments: SUMMARY:
The paper "Optimal Batched Best Arm Identification" introduces Tri-BBAI and Opt-BBAI algorithms to identify the best arm in multi-armed bandit settings. Tri-BBAI achieves optimal sample complexity with only three batches on average as the confidence parameter $\delta$ approaches zero. Opt-BBAI extends Tri-BBAI for finite-confidence settings, providing near-optimal sample and batch complexities for finite $\delta$.

STRENGTHS:
The paper is generally well-written. The theoretical analysis looks solid.

WEAKNESSES:
First, I don't think the regime when $\delta$ approaches $0$ is interesting. In a typical scenario, we want to set the confidence relatively small, and it doesn't make a lot of sense to consider the algorithm's performance on extremely small values.

The paper doesn't fairly compare previous results. It mentions the work "Collaborative top distribution identifications with limited interaction," and work "Optimal streaming algorithms for multi-armed bandits", but it doesn't include it in the table, which can lead to the impression that Opt-BBAI is the first algorithm to achieve $O(\log \frac{1}{\Delta_{2}})$ batch complexity. However, many algorithms have already achieved similar results. Moreover, these algorithms are not included in experiments. Significant work should be done to implement a fair comparison and properly acknowledge other works.

QUESTIONS:
N/A

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-07

comments: SUMMARY:
This paper considers the problem of BAI in the fixed confidence setting, in the context of finding the best arm in as few batches as possible. That is, instead of observing the reward of each arm pulled in turn, the learner makes multiple pulls in a single batch and only observes all rewards after the completion of said batch. The first result of the paper is to show that it is possible to achieve asymptotically optimal sample complexity, with only a constant number of batches. Specifically the authors propose the Tri-BBAI algorithm, which is shown to have asymptotically optimal sample complexity, PAC guarantees and requiring at most 3 batches in expectation, Theorems 3.1, 3.2, 3.3 respectively.

The authors then describe a second algorithm Opt-BBAI which has near optimal finite confidence guarantees as well as being asymptotically optimal.

STRENGTHS:
Best arm identification under fixed confidence is well studied problem and running algorithms in batches has clear practical relevance. Thus, achieving both asymptotically optimal sample and constant batch complexity is a nice result.

WEAKNESSES:
When considering finite confidence guarantees the authors could also compare with the recent work "An ε-Best-Arm Identification Algorithm for Fixed-Confidence and Beyond", Jourdan, Degenne, Kaufmann.

QUESTIONS:
Do the authors have an idea as to why the Tri-BBAI has poor experimental performace against Track and Stop?

LIMITATIONS:
No concern.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents two novel algorithms for the batched best arm identification (BBAI) problem. The first is the Tri-BBAI algorithm, which employs three batches with the expectation of achieving the asymptotic optimal sample complexity. Based on Tri-BBAI, the authors conceived the Opt-BBAI algorithm, which achieves the near-optimal sample and batch complexity when δ is finite and it does not tend to 0. Opt-BBAI enjoys the same sample and batch complexity of Tri-BBAI for the asymptotic setting\n\nSTRENGTHS:\n- Tri-BBAI is the first batched algorithm to achieve optimal sample complexity in an asymptotic setting\n- Opt-BBAI is the first batched algorithm to achieve near-optimal sample and batch complexity in non-asymptotic setting\n- Both algorithms are supported by valid theoretical analysis\n- Batched solutions may bring benefits to real-world scenarios that cannot rely on sequential methods, like Track-and-Stop algorithms\n\nWEAKNESSES:\n- The paper dedicates a significant portion to the introduction and related works, only beginning to describe methodologies and algorithms on page 5. I suggest reducing the introductory section to allow more space for discussing comparisons with existing algorithms (Table 2) and for elaborating on the experimental results.\n- The paragraph on the notation is partially useful, since it presents some notations that are never utilized in the main paper, but only in the appendix and may be employed to describe other unclear symbols.\n- I found some statements a bit misleading. For example, at line 251 the period ”In this section, we introduce Opt-BBAI, which can attain the optimal sample and batch complexity in both asymptotic and non-asymptotic settings adaptively [...]” is in contrast with lines 255-256 where, always regarding Opt-BBAI, is correctly said that ”we can achieve asymptotic optimality and near non-asymptotic optimality adaptively[...].\n- The claim at line 714 ”the sample complexity of Tri-BBAI and Opt-BBAI [...] is at most 2.6 times greater than Track and Stop when δ is very small” is not true, since in the second experiment for δ = 1 × 10−10 the ratio is evidently larger.\n\nQUESTIONS:\n- Reward distributions are assumed to belong to a single one-parameter exponential family, a common choice in literature. Are there any limitations in choosing different reward distributions that could affect the effectiveness of the presented approach?\n- I have some doubts about the necessity of presenting both Tri-BBAI and Opt-BBAI in the paper. If Opt-BBAI achieves the same performances of Tri-BBAI in the asymptotic settings, why is it necessary to describe Tri-BBAI as well, considering that Opt-BBAI also addresses non-asymptotic settings? Could you explain and give some intuition about the need to\npresent Tri-BBAI?\n- The choice of exactly three batches in the Tri-BBAI algorithm is not entirely clear. Additionally, I would like to ask the authors to provide some intuition regarding the optimal batch complexity of 2 for both Tri-BBAI and Opt-BBAI (shown in Table 1) compared to the structure of Tri-BBAI algorithm which employs at most 3 batches. What is the intuition behind these results? Why was a 2-batch algorithm not considered from the beginning?\n- Experiments have been carried on crafted environments. Since authors argued about the benefits that their work would bring in real-world settings, I would expect to find an experiment on a such real-world scenario.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}