Title: Covariance-adaptive best arm identification
Venue: NeurIPS 2023
Accepted: yes
Keywords: Multi-armed bandits

Best-arm identification

Adaptive identification
Abstract: We consider the problem of best arm identification in the multi-armed bandit model, under fixed confidence. Given a confidence input $\delta$, the goal is to identify the arm with the highest mean reward with a probability of at least $1 - \delta$, while minimizing the number of arm pulls. While the literature provides solutions to this problem under the assumption of independent arms distributions, we propose a more flexible scenario where arms can be dependent and rewards can be sampled simultaneously. This framework allows the learner to estimate the covariance among the arms distributions, enabling a more efficient identification of the best arm. The relaxed setting we propose is relevant in various applications, such as clinical trials, where similarities between patients or drugs suggest underlying correlations in the outcomes. We introduce new algorithms that adapt to the unknown covariance of the arms and demonstrate through theoretical guarantees that substantial improvement can be achieved over the standard setting. Additionally, we provide new lower bounds for the relaxed setting and present numerical simulations that support their theoretical findings.

Covariance adaptive best arm identification

Abstract

We consider the problem of best arm identification in the multi-armed bandit model, under fixed confidence. Given a confidence input δ , the goal is to identify the arm with the highest mean reward with a probability of at least 1 -δ , while minimizing the number of arm pulls. While the literature provides solutions to this problem under the assumption of independent arms distributions, we propose a more flexible scenario where arms can be dependent and rewards can be sampled simultaneously. This framework allows the learner to estimate the covariance among the arms distributions, enabling a more efficient identification of the best arm. The relaxed setting we propose is relevant in various applications, such as clinical trials, where similarities between patients or drugs suggest underlying correlations in the outcomes. We introduce new algorithms that adapt to the unknown covariance of the arms and demonstrate through theoretical guarantees that substantial improvement can be achieved over the standard setting. Additionally, we provide new lower bounds for the relaxed setting and present numerical simulations that support their theoretical findings.

1 Introduction and setting

Best arm identification (BAI) is a sequential learning and decision problem that refers to finding the arm with the largest mean (average reward) among a finite number of arms in a stochastic multi-armed bandit (MAB) setting. An MAB model ν is a set of K distributions in R : ν 1 , . . . , ν K , with means µ 1 , . . . , µ K . An "arm" is identified with the corresponding distribution index. The observation consists in sequential draws ("queries" or "arm pulls") from these distributions, and each such outcome is a "reward". The learner's goal is to identify the optimal arm i ∗ := Arg Max i ∈ J K K µ i , efficiently. There are two main variants of BAI problems: The fixed budget setting [1, 8], where given a fixed number of queries T , the learner allocates queries to candidates arms and provides a

guess for the optimal arm. The theoretical guarantee in this case takes the form of an upper bound on the probability p T of selecting a sub-optimal arm. The second variant is the fixed confidence setting [14, 20], where a confidence parameter δ ∈ (0 , 1) is given as an input to the learner, and the objective is to output an arm ψ ∈ J K K , such that P ( ψ = i ∗ ) ≥ 1 -δ , using the least number of arm pulls . The complexity in this case corresponds to the total number of queries made before the algorithm stops and gives a guess for the best arm that is valid with probability at least 1 -δ according to a specified stopping rule. In this paper we specifically focus on the fixed confidence setting.

The problem of best-arm identification with fixed confidence was extensively studied and is well understood in the literature [16, 19, 20, 14]. However, in these previous works, the problem was considered under the assumption that all observed rewards are independent. More precisely, in each round t , a fresh sample (reward vector), independent of the past, ( X 1 ,t , . . . , X K,t ) is secretly drawn from ν by the environment, and the learner is only allowed to choose one arm A t out of K (and observe its reward X A t ,t ). We relax this setting by allowing simultaneous queries. Specifically, we consider the MAB model ν as a joint probability distribution of the K arms, and in each round t the learner chooses a subset C t ⊂ J K K and observes the rewards ( X i,t ) i ∈ C t (see the Game Protocol 1). The high-level idea of our work is that allowing multiple queries per round opens up opportunities to estimate and leverage the underlying structure of the arms distribution, which would otherwise remain inaccessible with one-point feedback. This includes estimating the covariance between rewards at the same time point. It is important to note that our proposed algorithms do not require any prior knowledge of the covariance between arms.

Throughout this paper, we consider two cases: bounded rewards (in Section 4) and Gaussian rewards (in Section 5), with the following assumptions:

Assumption 1 . Suppose that:

IID assumption with respect to t : ( X t ) t ≥ 1 = ( X 1 ,t , . . . , X K,t ) t ≥ 1 are independent and identically distributed variables following ν .

There is only one optimal arm: ∣ ∣ ∣ Arg Max i ∈ J K K µ i ∣ ∣ ∣ = 1 .

Assumption 2 -B . Bounded rewards: The support of ν is in [0 , 1] K .

Assumption 2 -G . Gaussian rewards: ν is a multivariate normal distribution.

A round corresponds to an iteration in Protocol 1. Denote by i ∗ ∈ J K K the optimal arm. The learner algorithm consists of: a sequence of queried subsets ( C t ) t of J K K , such that subset C t at round t is chosen based on past observations, a halting condition to stop sampling (i.e. a stopping time written τ ) and an arm ψ to output after stopping the sampling procedure. The theoretical guarantees take the form of a high probability upper-bound on the total number N of queries made through the game:

2 Motivation and main contributions

The query complexity of best arm identification with independent rewards, when arms distributions are σ -sub-Gaussian is characterized by the following quantity:

Some δ -PAC algorithms [16] guarantee a total number of queries satisfying τ = ˜ O ( H ( ν ) log(1 /δ )) (where ˜ O ( . ) hides a logarithmic factor in the problem parameters).

Recall that the number of queries required for comparing the means of two variables, namely X 1 ∼ N ( µ 1 , 1) and X 2 ∼ N ( µ 2 , 1) , within a tolerance error δ , is approximately O ( log(1 /δ ) / ( µ 1 -µ 2 ) 2 ) . An alternative way to understand the sample complexity stated in equation (2) is to view the task of identifying the optimal arm i ∗ as synonymous with determining that the remaining arms J K K \ { i ∗ } are suboptimal. Consequently, the cost of identifying the best arm corresponds to the sum of the comparison costs between each suboptimal arm and the optimal one.

In practical settings, the arms distributions are not independent. This lack of independence can arise in various scenarios, such as clinical trials, where the effects of drugs on patients with similar traits or comparing drugs with similar components may exhibit underlying correlations. These correlations provide additional information that can potentially expedite the decision-making process [18].

In such cases, Protocol 1 allows the player to estimate the means and the covariances of arms. This additional information naturally raises the following question:

Can we accelerate best arm identification by leveraging the (unknown) covariance between the arms?

We give two arguments for a positive answer to the last question: When allowed simultaneous queries (more than one arm per round as in Protocol 1), the learner can adapt to the covariance between variables. To illustrate, consider the following toy example for 2 arms comparison: X 1 ∼ N ( µ 1 , 1) and X 2 = X 1 + Y where Y ∼ ϵ N (1 , 1) for some ϵ > 0 . BAI algorithms in one query per round framework require O ((1 + ϵ ) 2 log(1 /δ ) /ϵ 2 ) queries, which gives O (log(1 /δ ) /ϵ 2 ) for small ϵ . In contrast, when two queries per round are possible, the learner can perform the following test H 0 : ' E [ X 1 -X 2 ] > 0' against H 1 : ' E [ X 1 -X 2 ] ≤ 0' . Therefore using standard test algorithms adaptive to unknown variances, such as Student's t -test, leads to a number of queries of the order O (Var( X 1 -X 2 ) log(1 /δ ) /ϵ 2 ) = O (log(1 /δ )) . Hence a substantial improvement in the sample complexity can be achieved when leveraging the covariance. We can also go one step further to even reduce the sample complexity in some settings. Indeed, we can establish the sub-optimality of an arm i by comparing it with another sub-optimal arm j faster than when comparing it to the optimal arm i ∗ . To illustrate consider the following toy example with K = 3 : let X 1 ∼ N ( µ 1 , 1) , X 2 = X 1 + Y where Y ∼ ϵ N (1 , 1) and X 3 ∼ N ( µ 1 +2 ϵ, 1) , with X 1 , Y and X 3 independent. X 3 is clearly the optimal arm. Eliminating X 1 with a comparison with X 3 requires O (log(1 /δ ) /ϵ 2 ) queries, while comparing X 1 with the sub-optimal arm X 2 requires O (Var( X 1 -X 2 ) log(1 /δ ) /ϵ 2 ) = O (log(1 /δ )) .

The aforementioned arguments can be adapted to accommodate bounded variables, such as when comparing correlated Bernoulli variables. These arguments suggest that by utilizing multiple queries per round (as shown in Protocol 1), we can expedite the identification of the best arm compared to the standard one-query-per-setting approach. Specifically, for variables bounded by 1 , our algorithm ensures that the cost of eliminating a sub-optimal arm i ∈ J K K is given by:

The additional term 1 / ( µ j -µ i ) arises due to the sub-exponential tail behavior of the sum of bounded variables. To compare the quantity in (3) with its counterpart in the independent case, it is important to note that the minimum is taken over a set that includes the best arm, and that Var( X i ∗ -X i ) ≤ 2(Var( X i ∗ ) + Var( X i )) . Consequently, when the variables are bounded, the quantity (3) is no larger than a numerical constant times H ( ν ) (its independent case counterpart in 2), with potentially a significant improvements if there is positive correlation between an arm j ∈ J K K with a higher mean and arm i (so that Var( X j -X i ) is small). As a result, our algorithm for bounded variables has a sample complexity, up to a logarithmic factor, that corresponds to the sum of quantities (3) over all sub-optimal arms.

We expand our analysis to encompass the scenario where arms follow a Gaussian distribution. In this context, our procedure ensures that the cost of eliminating a sub-optimal arm i ∈ J K K is given by:

Similar to the setting with bounded variables, the aforementioned quantity is always smaller than its counterpart for independent arms when all variables X j have a unit variance.

In Section 6, we present two lower bound results for the sample complexity of best arm identification in our multiple query setting, specifically when the arms are correlated. Notably, these lower bounds pinpoint how the optimal sample complexity may decrease with the covariance between arms, making them the first of their kind in the context of best arm identification, to the best of our knowledge. The presented lower bounds are sharp, up to a logarithmic factor, in the case where all arms are positively correlated with the optimal arm. However, it remains an open question to determine a sharper lower bound applicable to a more general class of distributions with arbitrary covariance matrices.

In Section B of the appendix, we introduce a new algorithm that differs from the previous ones by performing comparisons between the candidate arm and a convex combination of the remaining arms, rather than pairwise comparisons. We provide theoretical guarantees for the resulting algorithm.

Lastly, we conduct numerical experiments using synthetic data to assess the practical relevance of our approach.

3 Related work

Best arm identification: BAI in the fixed confidence setting was studied by [11], [22], and [12], where the objective is to find ϵ -optimal arms under the PAC ('probably approximately correct") model. A summary of various optimal bounds for this problem is presented in [8, 20]. Prior works on fixed confidence BAI [25] and [17] developed strategies adaptive to the unknown variances of the arms. In contrast, our proposed algorithms demonstrate adaptability to all entries of the covariance matrix of the arms. In particular, we establish that in the worst-case scenario, where the arms are independent, our guarantees align with the guarantees provided by these previous approaches.

Covariance in the Multi-Armed Bandits model: Recently, the concept of leveraging arm dependencies in the multi-armed bandit (MAB) model for best-arm identification was explored in [15]. However, their framework heavily relies on prior knowledge, specifically upper bounds on the conditional expectation of rewards from unobserved arms given the chosen arm's reward. In a similar vein, a game protocol that allows simultaneous queries was examined in [21]. However, their objective differs from ours as their focus is on identifying the most correlated arms, whereas our primary goal is to identify the arm with the highest mean reward.

The extension of the standard multi-armed bandit setting to multiple-point bandit feedback was also considered in the stochastic combinatorial semi-bandit problem (2, 9, 10 and 13). At each round t ≥ 1 , the learner pulls m out of K arms and receives the sum of the pulled arms rewards. The objective is to minimize the cumulative regret with respect to the best choice of arms subset. [27] proposed an algorithm that adapts to the covariance of the covariates within the same arm. While this line of research shares the intuition of exploiting the covariance structure with our paper, there are essential differences between the two settings. In the combinatorial semi-bandit problem, the learner receives the sum of rewards from all selected arms in each round and aims to minimize cumulative regret, necessitating careful exploration during the game. In contrast, our approach does not impose any constraint on the number of queried arms per round, and our focus is purely on exploration.

Simultaneous queries of multiple arms was also considered in the context of graph-based bandit problems [7]. However, in these studies, it is assumed that the distributions of arms are independent.

Model selection racing: Racing algorithms for model selection refers to the problem of selecting the best model out of a finite set efficiently. The main idea consists of early elimination of poorly performing models and concentrating the selection effort on good models. This idea was seemingly first exploited in [23] through Hoeffding Racing. It consists of sequentially constructing a confidence interval for the generalization error of each (non-eliminated) model. Once two intervals become disjoint, the corresponding sub-optimal model is discarded. Later [25] presented an adaptive stopping algorithm using confidence regions derived with empirical Bernstein concentration inequality (3, 24). The resulting algorithm is adaptive to the unknown marginal variances of the models. Similarly, [4] presented a procedure centered around sequential hypothesis testing to make decisions between two possibilities. In their setup, they assume independent samples and use Bernstein concentration inequality tailored for bounded variables to adapt to the variances of the two variables under consideration.

While the idea of exploiting the possible dependence between models was shown [6, 26] to empirically outperform methods based on individual performance monitoring, there is an apparent lack of theoretical guarantees. This work aims to develop a control on the number of sufficient queries for

reliable best arm identification, while being adaptive to the unknown correlation of the candidate arms.

4 Algorithm and main theorem for bounded variables

In this section, we focus on the scenario where the arms are bounded by 1 . Let us establish the notation we will be using throughout. For each arm i ∈ J K K associated with the reward variable X i , we denote by µ i the mean of X i . Additionally, for any pair of arms i, j ∈ J K K , we denote V ij as the variance of the difference between X i and X j , i.e., V ij := Var( X i -X j ) . We remind the reader that the quantities V ij are unknown to the learner.

Algorithm 2 is developed based on the ideas introduced in Section 2, which involves conducting sequential tests between every pair ( i, j ) ∈ J K K × J K K of arms. The key element to adapt to the covariance between arms is the utilization of the empirical Bernstein's inequality [24] for the sequence of differences ( X i,t -X j,t ) t for i, j ∈ J K K . To that end, we introduce the following quantity:

where, ˆ µ i,t represents the empirical mean of the samples obtained from arm i up to round t , and ̂ V ij,t denotes the empirical variance associated with the difference variable ( X i -X j ) up to round t . The term α ( t, δ ) is defined as α ( t, δ ) := √ log(1 /δ t ) / ( t -1) , and δ t is given by δ/ (2 K 2 t ( t +1)) .

By leveraging the empirical Bernstein's inequality (restated in Theorem K.1), we can establish that if ˆ ∆ ij ( t, δ ) > 0 at any time t , then with a probability of at least 1 -δ , we have µ i < µ j . This observation indicates that arm i is sub-optimal. Furthermore, when µ j > µ i for i, j ∈ J K K , a sufficient sample sizes ensuring that the quantity ˆ ∆ ij ( t, δ ) is positive is proportional to:

Moreover, we show in Lemma E.5 in the appendix that the last quantity is necessary, up to a smaller numerical constant factor, in order to have ˆ ∆ ij ( t, δ ) > 0 .

Algorithm 2 follows a successive elimination approach based on the tests ˆ ∆ ij ( t, δ ) > 0 . Our objective is to ensure that any sub-optimal arm i is queried at most min j Λ ij times, where the minimum is taken over arms with means larger than µ i . However, this approach poses a challenge: the arm j ∗ that achieves the minimum of Λ ij may be eliminated early in the process, and the algorithm is then constrained to compare arm i with other arms costing larger complexity Λ ij . To address this limitation, it is useful to continue querying arms even after deciding their sub-optimality through the pairwise tests.

In Algorithm 2, we introduce two sets at round t : S t , which contains arms that are candidates to be optimal, initialized as S 1 = J K K , and C t , which represents the set of arms queried at round t . Naturally, C t contains S t and also includes arms that were freshly eliminated from S t , as we hope that these arms will help in further eliminating candidate arms from S t more quickly.

An important consideration is how long the algorithm should continue sampling an arm that has been eliminated from S t . In Theorem E.2, we prove that when arm j is eliminated at round t , it is sufficient to keep sampling it up to round 82 t (the constant 82 is discussed in Remark 1). The rationale behind this number of additional queries is explained in the sketch of the proof of Theorem 4.1. It suggests that if arm j fails to eliminate another arm after round 82 t , then the arm that eliminated j from S t can ensure faster eliminations for the remaining arms in S t .

Theorem 4.1. Suppose Assumption 1 and 2 -B holds. Consider Algorithm 2, with input δ ∈ (0 , 1) . We have with probability at least 1 -δ : the algorithm identifies the best arm and the total number of queries N satisfies:

where Λ = max i ∈ J K K min j ∈ J K K : µ j >µ i { V ij ( µ i -µ j ) 2 + 1 µ j -µ i } and c is a numerical constant.

Algorithm 2 Pairwise-BAI

Moreover, if we omit line 12 from Algorithm 2, that is we do not query non-candidate arms, we have with probability at least 1 -δ : the algorithm identifies the best arm and the total number of queries N satisfies:

where Λ is defined above and c is a numerical constant.

̸

Summary of the proof for bound (5) : Let i denote a sub-optimal arm and Υ i := Arg Min j Λ ij . First, we show that at round t , if i ∈ S t then necessarily we have Υ i ∩ C t = ∅ . We proceed by a contradiction argument: assume Υ i ∩ C t = ∅ , let j denote the element of Υ i with the largest mean. Let k denote the arm that eliminated j from S at a round s < t . Lemma E.3 shows that, since ˆ ∆ jk ( s, δ ) > 0 , we necessarily have log(1 /δ )Λ jk / 4 ≲ s . Moreover, j was kept up to round 82 s and in this last round we had ˆ ∆ ij (82 s, δ ) ≤ 0 , which gives by Lemma E.3: 82 s ≲ (25 / 2) log(1 /δ 82 s )Λ ij . Combining the two bounds on s , gives: Λ jk < Λ ij . Finally, we use the ultra-metric property satisfied by the quantities Λ uv (Lemma E.6), stating that: Λ ik ≤ max { Λ ij , Λ jk } . Combining the last inequality with the latter, we get Λ ik ≤ Λ ij , which means that k ∈ Υ i . The contradiction arises from the fact that j is the element of the largest mean in Υ i and arm k eliminated j (hence µ k > µ j ). We conclude that necessarily Υ i ∩ C t = ∅ , therefore by Lemma E.3, at a round of the order of min j Λ ij , we will necessarily have ˆ ∆ ij ( t, δ ) > 0 for some j ∈ C t .

Remark 1. From a practical standpoint, to keep sampling an arm eliminated at t for additional 81 t rounds is a conservative approach. The stated value of 81 is determined by specifics of the proof, but we believe that it can be optimized to a smaller constant based on the insights gained from numerical simulations. Furthermore, even if we omit the oversampling step, as presented in Theorem 4.1, the algorithm is still guaranteed to identify the best arm with probability 1 -δ . Only the query complexity guarantee is weaker, but the algorithm may still lead to effective arm elimination, although with potentially slightly slower convergence.

Remark 2. The idea of successive elimination based on evaluating the differences between variables was previously introduced in [28], in the context of model selection aggregation. Their analysis allows having a bound slightly looser than bound (6) (with the distances between variables instead of variances). On the other hand, Theorem 4.1 in our paper provides a sharper bound in (5) .

First, it is important to note that bound (5) is always sharper (up to a numerical constant factor) than bound (6) because the optimal arm is included in the set over which the minimum in bound (5) is taken. On the other hand, bound (5) can be smaller than bound (6) by a factor of 1 /K . This situation can arise in scenarios where the K -1 sub-optimal arms, which have close means, are highly correlated, while the optimal arm is independent of the rest. In such a situation, the terms in

̸

the sum in (5) corresponding to the correlated sub-optimal arms are relatively small, except for the arm (denoted as i ) with the largest mean among the K -1 correlated suboptimal rewards. That last remaining arm will be eliminated by the optimal arm, incurring a potentially large cost, but for that arm only. On the other hand, each term in the second bound (6) would be of the order of the last cost.

̸

To provide perspective on our guarantees compared to those developed in the independent one query per round setting, it is worth noting that the standard guarantees in that setting provide a sample complexity corresponding to ∑ i = i ∗ log(1 /δ ) / ( µ i ∗ -µ i ) 2 . Since variables are bounded by 1 , their variances are bounded by 1 . Therefore, in the worst case, we recover the previous guarantees with a numerical constant factor of 2 .

A refined adaptive algorithm presented in [25] also utilizes a successive elimination approach using confidence intervals for the arm means based on the empirical Bernstein's inequality. However, unlike our algorithm, they use the concentration inequality to evaluate each arm's mean independently of the other arms. Their approach allows for adaptability to individual arm variances, and is particularly beneficial are small, i.e., Var( X i ) ≪ 1 . The sample complexity of their algorithm is of the order of:

̸

where Γ := max i = i ∗ (Var( X i ) + Var( X i ∗ ) / ( µ i ∗ -µ i ∗ ) 2 . Neglecting numerical constant factors, the last bound is larger than both our bounds (5) and (6). This is because the variance of the differences can be bounded as follows: V ii ∗ ≤ 2(Var( X i ) + Var( X i ∗ )) .

̸

5 Algorithms and main theorem for Gaussian distributions

In this section, we address the scenario where arms are assumed to follow a Gaussian distribution. We consider a setting where the learner has no prior knowledge about the arms' distribution parameters, and we continue using the notation introduced in the previous section. The main difference between this case and the bounded variables setting, other than the form of the sample complexity obtained, is the behavior of the algorithm when variances of differences between variables tend to 0 , displayed by the second bound in Theorem 5.1.

Our algorithm relies on the empirical Bernstein inequality, which was originally designed in the literature for bounded variables [3, 24]. We have extended this inequality to accommodate Gaussian variables by leveraging existing Gaussian concentration results. Note that extending such inequalities more generally for sub-Gaussian variables is a non-trivial task. One possible direction is to suppose that arms follow a sub-Gaussian distribution and satisfy a Bernstein moment assumption (such extensions were pointed by works on bounded variables e.g., ? ). Given the last class of distributions, we can build on the standard Bernstein inequality with known variance, then plug in an estimate of the empirical variance leveraging the concentration of quadratic forms (see 5). However, it remains uncertain whether an extension for sub-Gaussian variables (without additional assumptions) is practically feasible.

We extend the previous algorithm to the Gaussian case by performing sequential tests between pairs of arms ( i, j ) ∈ J K K . We establish a confidence bound for the difference variables ( X i -X j ) (Lemma D.1 in the appendix) and introduce the following quantity:

where the f is defined by: f ( x ) = exp(2 x +1) if x ≥ 1 / 3 , and f ( x ) = 1 / (1 -2 x ) otherwise.

Using the empirical confidence bounds on the differences between arms, we apply the same procedure as in the case of bounded variables. We make one modification to Algorithm 2: we use the quantities ˆ ∆ ′ ij ( t, δ ) instead of ˆ ∆ ij ( t, δ ) .

By following the same analysis as in the bounded setting, we establish that the sample complexity required for the comparison tests between arms i and j is characterized by the quantity V ij ( µ i -µ j ) 2 ∨ 1 . The following theorem provides guarantees on the algorithm presented above:

Theorem 5.1. Suppose Assumptions 1 and 2 -G holds. Consider the algorithm described above, with input δ ∈ (0 , 1) . We have with probability at least 1 -δ : the algorithm identifies the best arm

and the total number of queries N satisfies:

where Λ = max i ∈ J K K min j ∈ J K K : µ j >µ i { V ij ( µ i -µ j ) 2 ∨ 1 } and c is a numerical constant.

Moreover, if we omit line 12 from Algorithm 2, that is we do not query non-candidate arms, we have with probability at least 1 -δ : the algorithm identifies the best arm and the total number of queries N satisfies:

̸

where Λ = max i = i ∗ V ii ∗ / ( µ i ∗ -µ i ) 2 is defined above and c is a numerical constant.

Remarks. On the sample complexity cost of being adaptive to the variance: If the sample size is larger than log(1 /δ ) , the cost of plugging in the empirical variance estimate into the Chernoff's concentration inequality is only a multiplicative constant slightly larger than one (nearly 1 + 2 √ log(1 /δ ) /n ). However, in the case of a small sample regime ( n < log(1 /δ ) ), the cost is a multiplicative factor of exp( √ log(1 /δ ) /n +1 / 2) due to the nature of the left tail of the chi-squared distribution (see Sections D and K of the appendix for detailed calculations). For most natural regimes, the number of queries made for each arm is larger than log(1 /δ ) , hence the last described effect does not arise. However, in some specific regimes (such as the case of very small variances of the arms) an optimal algorithm should query less than log(1 /δ ) samples, which necessitates introducing the exponential multiplicative term above into the concentration upper bound. This translates into a different form of guarantee presented in Theorem 5.1, inequality (9) . It is important to note the cost in this regime cannot be avoided as highlighted by our lower bound presented in Theorem 6.2.

̸

Theorem 5.1 above shows that Algorithm 2 is applied to Gaussian distribution guarantees that each sub-optimal arm i is eliminated after roughly min j V 2 ij / ( µ i -µ j ) 2 queries. Bounds (8) and (9) derived from our algorithm are smaller than the standard complexity bound in the independent case, which is given by ∑ i = i ∗ σ 2 / ( µ i -µ i ∗ ) 2 when all arms have variances smaller than σ 2 . It is important to note that unlike most existing procedures in the literature that achieve the standard complexity bound, our algorithm does not require knowledge of the parameter σ 2 .

̸

About the upper bound (9) : This bound can be further bounded by log( δ -1 ) ∑ i = i ∗ V ii ∗ ( µ i -µ i ∗ ) 2 ∨ 1 . The form presented in Theorem 5.1 is particularly sharp when the variances V ii ∗ tend to zero, resulting in a constant upper bound. Recently, in the independent setting where variances are unknown, [17] analyzed the comparison of two arms i and j . They derived a complexity of the order 1 / log ( 1 + ( µ i -µ j ) 2 ( σ 2 i + σ 2 j ) ) , where σ 2 i and σ 2 j are the variances of arms i and j , respectively. This result is reflected in our bound (9), where instead of the sum of variances, our bound considers the variance of the difference, leading to a sharper bound. In Section 6, we provide a lower bound that nearly matches (9).

6 Lower bounds

In this section, we present lower bounds for the problem of best arm identification with multiple queries per round, following Protocol 1. We provide lower bounds for both the bounded distributions setting (Theorem 6.1) and the Gaussian distributions setting (Theorem 6.2). It is important to note that our lower bounds are derived considering a class of correlated arm distributions. Therefore, the results obtained in the standard one-query-per-round setting [20] do not hold in our setting.

Our first lower bound is derived for the case where arms follow a Bernoulli distribution. Let µ = ( µ i ) i ∈ J K K be a sequence of means in [1 / 4 , 3 / 4] , denote by i ∗ the index of the largest mean. Consider a sequence of positive numbers V = ( V ii ∗ ) i ∈ J K K . We define B K ( µ , V ) as the set of Bernoulli arm distributions such that: (i) E [ X i ] = µ i ; (ii) Var( X i -X i ∗ ) ≤ V ii ∗ for all i ∈ J K K .

For Bernoulli distributions, the variances and means are linked. Specifically, Lemma K.8 demonstrates that for any pair of Bernoulli random variables ( B 1 , B 2 ) with means ( b 1 , b 2 ) , the following inequality holds: ( b 1 -b 2 ) -( b 1 -b 2 ) 2 ≤ Var( B 1 -B 2 ) ≤ min { 2 -( b 1 + b 2 ); b 1 + b 2 } -( b 1 -b 2 ) 2 .

We assume that the sequences V and µ satisfy the aforementioned condition for ( X i , X i ∗ ) i ∈ J K K , as otherwise the class B K ( µ , V ) would be an empty set. Theorem 6.1 provides a lower bound on the sample complexity required for best arm identification in the worst-case scenario over the class B K ( µ , V ) .

Theorem 6.1. Let K ≥ 2 and δ ∈ (0 , 1) . For any δ -sound algorithm, we have

where N is the total number of queries.

The presented lower bound takes into account the correlation between arms by incorporating the quantities V ii ∗ as upper bounds for the variances of the differences between arm i and the optimal arm i ∗ . This lower bound indicates that for class B K ( µ , V ) , Algorithm 2 is nearly optimal.

Next, we provide a lower bound in the Gaussian case. Let µ = ( µ i ) i ∈ J K K be a sequence of means, where i ∗ denotes the index of the largest mean. We also consider a sequence of positive numbers V = ( V ii ∗ ) i ∈ J K K . We define G K ( µ , V ) as the set of Gaussian arm distributions satisfying the following conditions: (i) E [ X i ] = µ i , (ii) Var( X i -X i ∗ ) = V ii ∗ , and (iii) Var( X i ) ≥ 1 for all i ∈ J K K . Theorem 6.2 provides a lower bound on the sample complexity required for best arm identification in the worst-case scenario over the class G K ( µ , V ) .

Theorem 6.2. Let K ≥ 2 and δ ∈ (0 , 1) . For any δ -sound algorithm, we have

where N is the total number of queries.

Theorem 6.2 demonstrates that our algorithm achieves near-optimal performance over G K ( µ , V ) , up to a logarithmic factor.

7 Numerical simulations

We consider the Gaussian rewards scenario. We compare our algorithm Pairwise-BAI (Algorithm 2) to 3 benchmark algorithms: Hoeffding race [23], adapted to the Gaussian setting (consisting of successive elimination based on Chernoff's bounds) and LUCB [19], which is an instantiation of the upper confidence bound (UCB) method. We assume that the last two algorithms have a prior knowledge on the variances of the arms. The third benchmark algorithm consists of using a successive elimination approach using the empirical estimates of the variances. We evaluated two variations of our algorithm. The first one, Pairwise-BAI+, implemented Algorithm 2 for Gaussian variables. In this instance, we modified line 12 by continuing to sample sub-optimal arms that were eliminated at round t until round 2 t instead of 82 t . We stress that both variants guarantee a δ -sound decision on the optimal arm (see Theorem 5.1). The second instance involved removing the last instruction, meaning we directly stopped querying sub-optimal arms. Figure 1 displays the average sample complexities for each considered algorithm. As expected, the larger the correlation between arms, the better Pairwise-BAI performs.

The second experiment aims to demonstrate that Algorithm 2 adapts to the covariance between suboptimal arms, as indicated by bound (8) in Theorem 4.1,rather than solely adapting to the correlation with the optimal arm as shown by (9). In this experiment, we consider that the arms are organized into clusters, where each pair of arms within the same cluster exhibits a high correlation (close to 1), while arms from different clusters are independent. According to bound (8) (see also discussion after Theorem 4.1) we expect to observe a gain of up to a factor corresponding to the number of arms per cluster compared to algorithms that do not consider covariance. (Since the total number of arms is kept fixed, the number of arms per cluster scales as the inverse of the number of clusters.)

Figure 1 illustrates the results, showing that the performance of Pairwise-BAI improves as the number of clusters decreases (indicating a larger number of correlated arms). This suggests that increasing

Number of queries

80000

60000

40000

20000

0

p=0

SamplecomplexityforBAl

Hoeffding race

Varianceadaptive

60000

LUCB

Pariwise-BAI+

50000

Pariwise-BAI

Number of queries

40000

30000

20000

10000

8

4

2

Number of clusters

p=0.5

p = 0.7

SamplecomplexityforBAl

Hoeffding race

LUCB

Pariwise-BAI+

Pariwise-BA1

6'0=d

̸

the number of correlated sub-optimal arms, which are independent from the optimal arm, still leads to significant performance improvement. These findings support the idea that Pairwise-BAI and Pairwise-BAI+ exhibit behavior that aligns more closely with bound (8) in Theorem 4.1, rather than bound (9).

In both experiments, we observe that Pairwise-BAI+ performs worse compared to Pairwise-BAI, indicating that, empirically, in the given scenarios, continuing to sample sub-optimal arms does not contribute to improved performance. While this modification provides better theoretical guarantees, it may not lead to empirical performance improvements in general scenarios.

8 Conclusion and future directions

This work gives rise to several open questions. Firstly, the presented lower bounds take into account partially the covariance between arms. It would be interesting to explore the development of a more precise lower bound that can adapt to any covariance matrix of the arms. Additionally, in terms of the upper bound guarantees, our focus has been on pairwise comparisons, along with an algorithm that compares candidate arms with convex combinations of the remaining arms (Section B). An interesting direction for further research would involve extending this analysis to an intermediate setting, involving comparisons with sparse combinations.

Acknowledgements: This work is supported by ANR-21-CE23-0035 (ASCAI) and ANR-19CHIA-0021-01 (BISCOTTE). This work was conducted while EM Saad was in INRAE Montpellier.

References

Jean-Yves Audibert and Sébastien Bubeck. Best arm identification in multi-armed bandits. In Conference on Learning Theory , 2010.

Jean-Yves Audibert, Sébastien Bubeck, and Gábor Lugosi. Regret in online combinatorial optimization. Mathematics of Operations Research , 39(1):31-45, 2014.

Jean-Yves Audibert, Rémi Munos, and Csaba Szepesvári. Tuning bandit algorithms in stochastic environments. In International conference on algorithmic learning theory , pages 150-165. Springer, 2007.

Akshay Balsubramani and Aaditya Ramdas. Sequential nonparametric testing with the law of the iterated logarithm. In Proceedings of the Thirty-Second Conference on Uncertainty in Artificial Intelligence , pages 42-51, 2016.

Pierre C Bellec. Concentration of quadratic forms under a Bernstein moment assumption. arXiv preprint arXiv:1901.08736 , 2019.

Mauro Birattari, Zhi Yuan, Prasanna Balaprakash, and Thomas Stützle. F-race and iterated F-race: An overview. Experimental methods for the analysis of optimization algorithms , pages 311-336, 2010.

Stephane Caron, Branislav Kveton, Marc Lelarge, and S Bhagat. Leveraging side observations in stochastic bandits. In UAI , 2012.

Alexandra Carpentier and Andrea Locatelli. Tight (lower) bounds for the fixed budget best arm identification bandit problem. In Conference on Learning Theory , pages 590-604. PMLR, 2016.

Nicolo Cesa-Bianchi and Gábor Lugosi. Combinatorial bandits. Journal of Computer and System Sciences , 78(5):1404-1422, 2012.

Wei Chen, Yajun Wang, and Yang Yuan. Combinatorial multi-armed bandit: General framework and applications. In International conference on machine learning , pages 151-159. PMLR, 2013.

Eyal Even-Dar, Shie Mannor, and Yishay Mansour. PAC bounds for multi-armed bandit and Markov decision processes. In International Conference on Computational Learning Theory , pages 255-270. Springer, 2002.

Eyal Even-Dar, Shie Mannor, Yishay Mansour, and Sridhar Mahadevan. Action elimination and stopping conditions for the multi-armed bandit and reinforcement learning problems. Journal of machine learning research , 7(6), 2006.

Yi Gai, Bhaskar Krishnamachari, and Rahul Jain. Combinatorial network optimization with unknown variables: Multi-armed bandits with linear rewards and individual observations. IEEE/ACM Transactions on Networking , 20(5):1466-1478, 2012.

Aurélien Garivier and Emilie Kaufmann. Optimal best arm identification with fixed confidence. In Conference on Learning Theory , pages 998-1027. PMLR, 2016.

Samarth Gupta, Gauri Joshi, and Osman Ya˘ gan. Best-arm identification in correlated multiarmed bandits. IEEE Journal on Selected Areas in Information Theory , 2(2):549-563, 2021.

Kevin Jamieson, Matthew Malloy, Robert Nowak, and Sébastien Bubeck. Lil'ucb: An optimal exploration algorithm for multi-armed bandits. In Conference on Learning Theory , pages 423-439. PMLR, 2014.

Marc Jourdan, Rémy Degenne, and Emilie Kaufmann. Dealing with unknown variances in best-arm identification. In International Conference on Algorithmic Learning Theory , pages 776-849. PMLR, 2023.

Brennan C Kahan and Tim P Morris. Improper analysis of trials randomised using stratified blocks or minimisation. Statistics in medicine , 31(4):328-340, 2012.

Shivaram Kalyanakrishnan, Ambuj Tewari, Peter Auer, and Peter Stone. PAC subset selection in stochastic multi-armed bandits. In ICML , volume 12, pages 655-662, 2012.

Emilie Kaufmann, Olivier Cappé, and Aurélien Garivier. On the complexity of best-arm identification in multi-armed bandit models. The Journal of Machine Learning Research , 17(1):1-42, 2016.

Che-Yu Liu and Sébastien Bubeck. Most correlated arms identification. In Conference on Learning Theory , pages 623-637. PMLR, 2014.

Shie Mannor and John N Tsitsiklis. The sample complexity of exploration in the multi-armed bandit problem. Journal of Machine Learning Research , 5(Jun):623-648, 2004.

Oded Maron and Andrew Moore. Hoeffding races: Accelerating model selection search for classification and function approximation. Advances in neural information processing systems , 6, 1993.

Andreas Maurer and Massimiliano Pontil. Empirical Bernstein bounds and sample-variance penalization. In COLT 2009 - The 22nd Conference on Learning Theory, Montreal, Quebec, Canada, June 18-21, 2009 , 2009.

Volodymyr Mnih, Csaba Szepesvári, and Jean-Yves Audibert. Empirical Bernstein stopping. In Proceedings of the 25th international conference on Machine learning , pages 672-679, 2008.

Andrew W Moore and Mary S Lee. Efficient algorithms for minimizing cross validation error. In Machine Learning Proceedings 1994 , pages 190-198. Elsevier, 1994.

Pierre Perrault, Michal Valko, and Vianney Perchet. Covariance-adapting algorithm for semibandits with application to sparse outcomes. In Conference on Learning Theory , pages 31523184. PMLR, 2020.

El Mehdi Saad and Gilles Blanchard. Fast rates for prediction with limited expert advice. Advances in Neural Information Processing Systems , 34, 2021.

Alexandre B Tsybakov. Optimal rates of aggregation. In Learning theory and kernel machines , pages 303-313. Springer, 2003.

Nicolas Verzelen. Minimax risks for sparse regressions: Ultra-high dimensional phenomenons. Electronic Journal of Statistics , 6:38 - 90, 2012.

Martin J Wainwright. High-dimensional statistics: A non-asymptotic viewpoint , volume 48. Cambridge University Press, 2019.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work focuses on best arm identification, that is, determining the arm with highest average return within a small number of samples (sample complexity). The twist is that authors propose a framework where sets of arms (of arbitrary size) can be queried at the same time, and the set of corresponding rewards is observed. This setting goes beyond classical frameworks, as observations in a set of simultaneously queried arms are dependent. Moreover, the arm pairwise covariances are unknown to the bandit algorithm. Authors propose an algorithm tackling this problem, which nearly matches associated lower bounds, and empirically evaluate it.\n\nSTRENGTHS:\n- Originality: Authors introduce a novel best arm identification setting which might have interesting real-life applications. Differences with prior works are thoroughly discussed and the discussion covers a large set of related settings. \n- Quality: The results in this paper seem technically sound, although I did not check the proofs in detail. The experimental study is convincing and supports theoretical claims. The code is available, which helps reproducibility and potential future follow-up.\n- Significance: The setting tackled in this paper introduces new technical challenges (in particular to derive lower bounds), and the technical tools used in proofs can be of independent interest.\n\nWEAKNESSES:\n- Clarity: I appreciated the many examples provided in order to illustrate intuitions behind the papers, although I believe the section from Line 59 to Line 123 is a bit messy and hard to follow. Adding more structure to this section and clearly highlighting the different points covered in that paragraph would really help.\n- Significance: The number of rounds after which an arm can be safely eliminated from the set of “queriable” arms seems really large and not very practical. However, this weakness has been identified in this paper and properly discussed (notably by providing a version of the upper bound when elimination happens when an arm is not more considered a candidate).\n\nQUESTIONS:\n- What would be the main hurdles in extending the lower bound theorems to any one-dimensional exponential family?\n\nLIMITATIONS:\nAuthors have adequately addressed the issue related to the number of rounds after which an arm is completely eliminated from future queries. This paper deals with very theoretical guarantees, and, as such, do not present significant direct negative societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors investigate the stochastic Best Arm Identification (BAI) problem when the arms have an unknown correlation structure.\nIn contrast, the majority of BAI literature focuses on independent arms with unknown variances.\nAuthors propose algorithms for BAI in two settings (bounded random variables and gaussian random variables) and analyze the sample complexity of these algorithms.\nIn order to leverage the correlation structure, multiple arms are allowed to be sampled at each round.\nThey show that improved sample complexity can be achieved by estimating the correlation structure; moreover, the sample complexity is within a constant factor of the sample complexity for independent arms (even when independence is unknown in advance).\nSimulations are run to demonstrate the method on a synthetic dataset.\n\nSTRENGTHS:\nThe main strength of the paper is an investigation into the stochastic BAI problem where independence between the arms is not assumed.\nAuthors develop algorithms where the covariance structure is estimated and used to help quickly eliminate suboptimal arms.\nOf course, multiple arms must be sampled simultaneously in order to estimate the covariance between them.\nOn one hand, the analysis (and simulations) show that nice speed-ups in sample complexity can be obtained when the covariance structure is taken into account.\nOn the other hand, the analysis seems to indicate that the asymptotic rate of the sample complexity does not increase even when the arms are independent (but this dependence is unknown).\nThis has exciting implications for a variety of real-world settings where arms may be expected to be correlated.\n\nWEAKNESSES:\nThere are two main weaknesses of the paper.\n\nThe first main weakness is that the paper seems to not discuss (and sometimes ignore?) what I would consider to be a very important consideration: the \"overhead cost\" of estimating the covariance.\nIt seems intuitive that a method which places no assumptions on the covariances -- and therefore must estimate them -- must incur some overhead cost when compared to a setting where there are structured assumptions on the covariance (e.g. the independent setting).\nThis overhead cost may not be very large, but I am pretty certain that it should exist.\nA couple of times throughout the paper (Lines 103, 247) authors claim that the sample complexity bounds they obtain are smaller than the independent case; however, it seems that they are smaller than a constant factor (say, 2) of the sample complexity of the independent case.\nIn terms of the asymptotic rate of the sample complexity, this doesn't matter; however, in terms of finite sample, practical performance, this does seem to matter and it points out exactly this \"overhead cost\".\nOverall, I think that the paper would be stronger if authors more appropriately discussed their results in this context, i.e. the existence of the overhead cost of estimating covariance relative to assuming it, but with large gains when dependencies help.\n\nThe second weakness of the paper is that the exact nature of the guarantees on the sample complexity are not so clear.\nFrom what I understand, these algorithms need to contain the best arm with probability $1 - \\delta$ for some total number of arm queries $N$ and this stopping time is a random variable.\nThe goal of the algorithm designer is that, under appropriate conditions, $N$ is small in some sense, as a random variable.\nSo what does it mean then in Theorems 4.1 and 5.1 that authors write $N \\leq XYZ$? Does this happen with probability 1 (that seems unlikely)?\nDoes this mean that $N$ is order $XYZ$ in probability? \nOr is $N$ not a random variable?\nPerhaps there is some convention in the BAI literature so that this is very clear; but to someone in adjcent fields, this could use clarification.\nI'm hopeful that this can be clarified and the fix is relatively easy.\n\nFurther comments on clarity and discussion of results are given below.\n\n### Minor Comments\n\n1. (Line 58): What is the $\\pi$ subscript in $N$ represent and why does it never appear again in the paper? Moreover, the last sentence in this paragraph seems to be a sentence fragment that is missing a part.\n2. (Line 64): Could you describe more precisely what \"comparing the means\" is referring to here? What is the precise statistical problem i.e. estimation, hypothesis testing etc ?\n3. (Line 71): A motivating example of clinical trials is discussed, \"whre the effects of drugs on patients with similar traits or comparing drugs with similar components may exhibit underlying correlations\". However, the correlation between counterfactual outcomes (arms) can't be simultaneosuly observed in a causal inference setting -- only one outcome may be observed for each patient, often known as the \"fundamental problem of causal inference\". I recommend rephrasing this motivation so that arms cannot be interpreted to be counterfactual (and thus not simulatenously observable) outcomes, or selecting a different motivating example.\n4. (line 103): authors write that \"quantity (3) is always smaller (up to a numerical constant) than its independent case counterpart\". I think this is phrased in an unfortunately misleading way. I presume that the authors mean \"quantity (3) is no larger than a constant times the independent case counterpart\". Unfortunately, writing the phrase \"always smaller\" gives the wrong impression that quantity (3) is at most quantity (2), which does not seem to be true. Great care should be given to statements like this, so they are not misrepresenting what authors have shown.\n5. (Line 140, 157) citations appear in parathensis when other citations appear in brackets. A consistent stlye should be used throughout the paper.\n6. (Line 142) \"The objective is to maximize the cumulative regret...\" is this a typo, i.e. \"maximize\" should be \"minimize\"?\n7. (Line 172): The estimated quantities $V_{i,j,t}$ and $\\mu_{i,t}$ should be defined (markdown doesn't like the hat notation, sorry for dropping it). In particular, these quantities are perhaps ill-defined without reference to the algorithm: how should we interpret the sample covariance up to iteration $t$ if arm $i$ was sampled at some round $s \\leq t$ but $j$ was not?\n8. (Line 7 of Algorithm 2): \"Jointly query all the experts in $C$\". The terminology \"experts\" is not defined, so I'd recommend not using it here.\n9. (Line 247): Authors write \"the sample complexity is on the order $X$ which is larger than both our bounds (5) and (6)\". If I understand correctly, it seems that the correct statement is that \"(5) and (6) are less than 2 $X$\". It seems very important for authors to correctly characterize their contributions so readers do not misinterpret them.\n10. (Line 275): the reference (20) is perhaps misplaced.\n11. (Line 305): \"$\\delta$-sound algorithm\" is used but not defined. From context clues, I think I understand, but authors should clarify it for the readers.\n12. (Line 309): Authors claim that \"Algorithm 2 is nearly optimal\" but this is up to a $\\log(K)$ and $\\log(\\Lambda)$ factors. I understand $K$, but I think that $\\log(\\Lambda)$ deserves some discussion. How large should we expect this to be?\n13. (Line 319): \"Theorem 6.2 demonstrates that our algorithm achieves near-optimal performance...up to a logarithmic factor\". Same as above.\n14. (Line 329): Authors write that \"We stress that both variants guarantee a $\\delta$-sound decision on the optimal arm\" but it seems that reducing $82 t$ to $2 t$ actually doesn't necessarily satisfy $\\delta$-sound? If I am correct in this understanding, then the claim should be modified appropriately.\n15. (Line 356): Authors write that there is \"an algorithm that compares candidate arms with convex combinations of the remaining arms\" but this is not discussed in the main body nor in the simulations. This seems it requires more discussion to better contextualize it within the current work or be removed all together.\n16. (Line 347): Authors write \"In both experiments, we observe that Pairwise-BAI+ performs worse compared to Pairwise-BAI, indicating that, empirically, in the given scenarios, continuing to sample sub-optimal arms does not contribute to improved performance\". There is either a typo here or I am a bit confused. I thought that Pairwise-BAI+ was where the number of extra rounds was reduced, so if it performs worse wouldn't that indicate that we need those extra rounds? I think there is a typo, but I want to bring this to the attention of the authors to confirm or deny.\n\nQUESTIONS:\nI think this paper makes some interesting and strong contributions.\nUnfortunately, I found some of the results and their discussions to be sufficiently confusing to a broader audience that I have set my score to \"borderline reject\".\nHowever, if authors can answer the following questions to clarify these points (especially the first two), then I am more than happy to raise my score.\n\n1. Can you please respond to my comments on the confusion regarding what $N \\leq XYZ$ means if $N$ is a random variable? I'm hopeful that you can clarify this for me.\n2. Can you please respond to my comments on the overhead cost of estimating the covariance structure and whether my assessment that the sample complexity bounds are within a factor of two of the independent case is correct?\n3. Can Assumption 3 be relaxed to subgaussian? If the answer is no, then why not?\n4. Can you comment more on the \"algorithm that compares candidate arms with convex combinations of the remaining arms\"? Why does it not appear in simulations and does it offer any additional benefits over the proposed algorithms?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
This paper considers the problem of best arm identification with covariance in the fixed confidence setting, where arms can be dependent and rewards can be sampled simultaneously. The authors design algorithms that adapt to the unknown covariance of arms and prove that substantial improvement can be achieved over the standard setting. The authors also provide lower bounds and experimental results that support their theoretical findings.

STRENGTHS:
1.	The considered problem is interesting and well-motivated.
2.	The theoretical analysis looks sound. Both upper and lower bounds are provided.

WEAKNESSES:
1.	What is the additional novelty and contribution of Theorem 5.1 compared to Theorem 4.1. It seems that there is no significant difference between the algorithm design and analysis for the bounded reward setting and that for the Gaussian reward setting. Why not unify them, e.g.,  present this work in the unified sub-Gaussian setting? Please correct me if my understanding is wrong.
2.	Could you compare Theorems 4.1 and 5.1with the results of existing covariance-adaptive bandit works?

QUESTIONS:
Please see the weaknesses above.

LIMITATIONS:
Please see the weaknesses above.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the question of identifying $\\epsilon$-optimal arms given a confidence input $\\delta$, or in other words, under the PAC model. Instead of pulling only one arm and observing the rewards, the authors leverage the underlying structure of arm distributions by allowing multiple queries per round (Protocol 1). Compared to related works, this paper looses the assumption of independent arms distributions, Protocol 1 can estimate the means and covariances of arms and accelerate the best arm identification by utilizing the extra information. They propose two algorithms, one for scenario where the arms are bounded, and one for arms follow a Gaussian distribution. The lower bounds for both algorithms are also provided.\n\nSTRENGTHS:\nThe idea of exploring the underlying structure of arms by allowing simultaneous queries is attractive. It not only solves the assumption of independent arms distributions, which is not realistic in various scenarios; but also accelerates best arm identification by leveraging the shared information between arms. Based on multiple queries protocol, the authors provide two main theorems for bounded variables and Gaussian distributions along with the corresponded lower bounds. They also manage to notice some failure mode of the algorithm, like shown in line 183, and give some solution. In addition, the paper is well-structured and easy to follow.\n\nWEAKNESSES:\nThe authors emphasize their results are adaptive to unknown covariance, but based on the algorithms and simulations, it can only solve the type of same covariance between all arms (correct me if I were wrong), which reduces the practical use of this paper. In addition, as stated in the related work part, the topic of best arm identification under PAC-Learning framework is well-studied, so as the stochastic combinatorial semi-bandit problem. The motivation and main contributions of this papers are a little bit unclear. As for the simulation part, It would be helpful to provide some explanation for the three chosen benchmark algorithms to add on reliability. I was expecting to see the comparison of proposed algorithm and old algorithms who assume independent arms on scenarios where the assumption fails.\n\nQUESTIONS:\n1. Under the PAC framework, except for the confidence input $\\delta$, the subtle difference $\\epsilon$ is also important and related to the sample complexity, but I didn't find the involvement of this parameter in algorithms (like Theorem 4.1, line198). \n2. If the assumption of arm independence holds, can we still use the proposed algorithms? Would there be a trade-off between complexity and generalization?\n3. Could you provide a concrete example of the failure case mentioned in line 183? And how probably will this kind of failure happen?\n\nLIMITATIONS:\nThis paper trying to utilize the covariances between arms to accelerate best arm identification, what if we step further, using the causal relationship instead of association? I know some research have been done in this area (like Lattimore, 2016) It would be interesting to consider this as a future direction. The proposed algorithms, although allow the dependent arms distributions, still have many constraints, like limited type of variables, only suitable for same covariance, pairwise comparisons, which reduce the practicality of this paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis study addresses the problem of best arm identification (BAI) in the context of dependent arms. Unlike conventional settings, efficiency can be enhanced by exploiting the inherent correlation structure. This setting holds broad applications, including in clinical trials. Specifically, the authors concentrate on bandit scenarios with bounded and Gaussian rewards. The validity of their approach is substantiated through simulation studies.\n\nSTRENGTHS:\nThe authors introduce a novel setting for Best Arm Identification (BAI) with fixed confidence. By accommodating correlation, we can devise more efficient strategies for identifying the optimal arm. This setting is intriguing and holds significant practical utility.\n\nWEAKNESSES:\nI believe several claims are not sufficiently substantiated. I have detailed these claims in the subsequent 'Questions' section. If the authors fail to adequately address my queries, these unsupported claims would constitute a weakness in the study.\n\nQUESTIONS:\n- In line 81, what does $\\epsilon\\mathcal{N}(1, 1)$ denote?\n- In line 83, the authors suggest that since two queries are possible, the learner can perform a $T$-test. However, even in standard BAI, a setting based on such a hypothesis test, as suggested by the authors, exists. For instance, refer to Balsubramani & Ramdas (2016). \n- Therefore, considering a hypothesis test in itself is feasible even in the standard setting. I believe that the key here is the variance reduction effect facilitated by two queries.\n- In lines 84-86, the authors present an example of two-armed Gaussian bandits. They argue that the lower bound in standard BAI is $O((1+\\epsilon)^2\\log (1/\\delta) / \\epsilon^2)$. This seems to be a lower bound for the one-armed bandit problem. As highlighted by Kaufmann et al. (2016), the lower bound for two-armed Gaussian bandits is given as $O(Var(X_1 - X_2)) \\log (1/\\delta) / \\epsilon^2)$ by assuming there are two arms, $Y_1 \\sim N(\\epsilon, Var(X_1 - X_2))$ and $Y_2 \\sim N(0, 0)$. As I stated earlier, the essence of this example is not two queries, but treating two arms as one and comparing this combined arm to an arm that returns zero.\n- For the related work section, I suggest considering the literature on BAI on graphs as it also deals with similar problems. Although the authors cite a few studies in this field, could these studies be related to yours?\n- In line 125, the authors state that \"BAI in the fixed confidence setting was studied by [8], [19], and [9], where the objective is to find $\\epsilon$-optimal arms under the PAC model.\" Fixed confidence BAI is not confined to $\\epsilon$-best arm identification. For instance, [17] considers a different objective.\n- Subsequently, the authors claim that \"A summary of various optimal bounds for this problem is presented in [5, 17].\" Here, [5] considers lower bounds in BAI with a fixed budget, which is a different setting. [17] addresses the fixed-confidence BAI problem, but the objective is not $\\epsilon$-optimal arms.\n- In addition to the related work raised by the authors, it seems that Kato and Ariu (2021) have explored BAI with dependent arms for more specific cases. For example, they consider two-armed Gaussian bandits where rewards are correlated via contextual information. How is their study relevant to yours?\n\nBalsubramani and Ramdas, Sequential nonparametric testing with the law of the iterated logarithm, UAI 2016.   \nKato and Ariu, The Role of Contextual Information in Best Arm Identification, 2021.\n\nLIMITATIONS:\nNone.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}