Title: Closing the Computational-Statistical Gap in Best Arm Identification for Combinatorial Semi-bandits
Venue: NeurIPS 2023
Accepted: yes
Keywords: best-arm identification; combinatorial semi-bandit; no-regret learning;
Abstract: We study the best arm identification problem in combinatorial semi-bandits in the fixed confidence setting. We present Perturbed Frank-Wolfe Sampling (P-FWS), an algorithm that (i) runs in polynomial time, (ii) achieves the instance-specific minimal sample complexity in the high confidence regime, and (iii) enjoys polynomial sample complexity guarantees in the moderate confidence regime. To our best knowledge, existing algorithms cannot achieve (ii) and (iii) simultaneously in vanilla bandits. With P-FWS, we close the computational-statistical gap in best arm identification in combinatorial semi-bandits. The design of P-FWS starts from the optimization problem that defines the information-theoretical and instance-specific sample complexity lower bound. P-FWS solves this problem in an online manner using, in each round, a single iteration of the Frank-Wolfe algorithm. Structural properties of the problem are leveraged to make the P-FWS successive updates computationally efficient. In turn, P-FWS only relies on a simple linear maximization oracle.

Closing the Computational-Statistical Gap in Best Arm Identification for Combinatorial Semi-bandits

Abstract

We study the best arm identification problem in combinatorial semi-bandits in the fixed confidence setting. We present Perturbed Frank-Wolfe Sampling ( P-FWS ), an algorithm that (i) runs in polynomial time, (ii) achieves the instance-specific minimal sample complexity in the high confidence regime, and (iii) enjoys polynomial sample complexity guarantees in the moderate confidence regime. To the best of our knowledge, even for the vanilla bandit problems, no algorithm was able to achieve (ii) and (iii) simultaneously. With P-FWS , we close the computational-statistical gap in best arm identification in combinatorial semi-bandits. The design of P-FWS starts from the optimization problem that defines the information-theoretical and instance-specific sample complexity lower bound. P-FWS solves this problem in an online manner using, in each round, a single iteration of the Frank-Wolfe algorithm. Structural properties of the problem are leveraged to make the P-FWS successive updates computationally efficient. In turn, P-FWS only relies on a simple linear maximization oracle.

1 Introduction

An efficient method to design statistically optimal algorithms solving active learning tasks (e.g., regret minimization or pure exploration in bandits and reinforcement learning) consists in the following two-step procedure. The first step amounts to deriving, through change-of-measure arguments, tight information-theoretical fundamental limits satisfied by a wide class of learning algorithms. These limits are often expressed as the solution of an optimization problem, referred in this paper to as the lower-bound problem . Interestingly, this solution specifies the instance-specific optimal exploration process: it characterizes the limiting behavior of the adaptive sampling rule any statistically optimal algorithm should implement. In the second step, the learning algorithm is designed so that its exploration process approaches the solution of the lower-bound problem. This design yields statistically optimal algorithms, but typically requires to repeatedly solve the lower-bound problem. This method has worked remarkably well for simple learning tasks such as regret minimization or best-arm identification with fixed confidence in classical stochastic bandits [Lai87, GC11, GK16], but also in bandits whose arm-to-average reward function satisfies simple structural properties (e.g., Lipschitz, unimodal) [MCP14, WTP21].

The method also provides a natural way of studying the computational-statistical gap [KLLM22] for active learning tasks. Indeed, if solving the lower-bound problem in polynomial time is possible, one

may hope to devise learning algorithms that are both statistically optimal and computationally efficient. As of now, however, the computational complexity of the lower-bound problem remains largely unexplored, except for simple learning tasks. For example, in the case of best policy identification in tabular Markov Decision Processes, the lower-bound problem is non-convex [AMP21] and its complexity and approximability are unclear.

In this paper, we leverage the aforementioned two-step procedure to assess the computationalstatistical gap for the best arm identification in combinatorial semi-bandits in the fixed confidence setting. We establish that, essentially, this gap does not exist (a result that was conjectured in [JMKK21]). Specifically, we present an algorithm that enjoys the following three properties: (i) it runs in polynomial time, (ii) its sample complexity matches the fundamental limits asymptotically in the high confidence regime, and (iii) its sample complexity is at most polynomial in the moderate confidence regime. Next, after formally introducing combinatorial semi-bandits, we describe our contributions and techniques in detail.

Best arm identification in combinatorial semi-bandits. In combinatorial semi-bandits [CBL12, CTMSP + 15], the learner sequentially selects an action from a combinatorial set X ⊂ { 0 , 1 } K . When in round t , the action x ( t ) = ( x 1 ( t ) , . . . , x K ( t )) ∈ X is chosen, the environment samples a K -dimensional vector y ( t ) whose distribution is assumed to be Gaussian N ( µ , I ) . The learner then receives the detailed reward vector x ( t ) ⊙ y ( t ) where ⊙ denotes the element-wise product (in other words, the learner observes the individual reward y k ( t ) of the arm k if and only if this arm is selected in round t , i.e., x k ( t ) = 1 ). The parameter µ characterizing the average rewards of the various arms is initially unknown. The goal of a learner is to identify the best action i ⋆ ( µ ) = argmax x ∈X ⟨ x , µ ⟩ with a given level of confidence 1 -δ , for some δ > 0 while minimizing the expected number of rounds needed. We assume that the best action is unique and denote by Λ = { µ ∈ R K : | i ⋆ ( µ ) | = 1 } the set of parameters satisfying this assumption. The learner strategy is defined by three components: (i) a sampling rule dictating the sequence of the selected actions; (ii) a stopping time τ defining the last round where the learner interacts with the environment; (iii) a decision rule specifying the action ˆ ı ∈ X believed to be optimal based on the data gathered until τ .

The sample complexity lower-bound problem. Consider the set of δ -PAC algorithms such that for any µ ∈ Λ , the best action is identified correctly with probability at least 1 -δ . We wish to find a δ -PAC algorithm with minimal expected sample complexity E µ [ τ ] . To this aim, using classical changeof-measure arguments [GK16], we may derive a lower bound of the expected sample complexity satisfied by any δ -PAC algorithm. This lower bound is given by 1 E µ [ τ ] ≥ T ⋆ ( µ ) kl ( δ, 1 -δ ) . The characteristic time T ⋆ ( µ ) is defined as the value of the following problem

̸

where 2 Σ = { ∑ x ∈X w x x : w ∈ Σ |X| } , kl ( a, b ) is the KL-divergence between two Bernoulli distributions with respective means a and b , and Alt ( µ ) = { λ ∈ Λ : i ⋆ ( λ ) = i ⋆ ( µ ) } is the set of confusing parameters. As it turns out (see Lemma 1), T ⋆ ( µ ) is at most quadratic in K , and hence the sample complexity lower bound is polynomial. (1) is a concave program over Σ [WTP21], and a point ω ⋆ in its solution set corresponds to an optimal allocation of action draws: an algorithm sampling actions according to ω ⋆ and equipped with an appropriate stopping rule would yield a sample complexity matching the lower bound. In this paper, we provide computationally efficient algorithms to solve (1) and show how these can be used to devise a δ -PAC best action identification algorithm with minimal sample complexity and running in polynomial time. We only assume that we have access to a computationally efficient Oracle, referred to as the LM (Linear Maximization) Oracle, identifying the best action should µ be known (but for any possible µ ). This assumption, made in all previous work on combinatorial semi-bandits (see e.g. [JMKK21, PBVP20]), is crucial as indeed, if there is no computationally efficient algorithm solving the offline problem argmax x ∈X ⟨ x , µ ⟩ with known µ , there is no hope to solve its online version with unknown µ in a computationally efficient manner. The assumption holds for a large array of combinatorial sets of actions [S + 03], including m -sets, matchings, (source-destination)-paths, spanning trees, matroids (refer to [CCG21b] for a thorough discussion).

The Most-Confusing-Parameter ( MCP ) algorithm. The difficulty of solving (1) lies in the inner optimization problem, i.e., in evaluating the objective function:

̸

where f x ( ω , µ ) = inf λ ∈C x ⟨ ω , ( µ -λ ) 2 2 ⟩ and C x = { λ ∈ R K : ⟨ λ , i ⋆ ( µ ) -x ⟩ < 0 } . Evaluating F µ ( ω ) is required to solve (1), but also in the design of an efficient stopping rule. Our first contribution is MCP (Most-Confusing-Parameter), a polynomial time algorithm able to approximate F µ ( ω ) for any given µ and ω . The algorithm's name refers to the fact that by computing F µ ( ω ) , we implicitly identify the most confusing parameter λ ⋆ ∈ arg inf λ ∈ Alt ( µ ) ⟨ ω , ( µ -λ ) 2 2 ⟩ . The design of MCP leverages a Lagrangian relaxation of the optimization problem defining f x ( ω , µ ) and exploits the fact that the Lagrange dual function linearly depends on x . In turn, this linearity allows us to make use of the LM Oracle. From these observations, we show that computing F µ ( ω ) boils down to solving a two-player game, for which one of the players can simply update her strategy using the LM Oracle. We prove the following informally stated theorem quantifying the performance of the MCP algorithm (see Theorem 3 for a more precise statement).

Theorem 1. For any ( ω , µ ) , the MCP algorithm with precision ϵ and certainty parameter θ returns ˆ F and ˆ x satisfying P µ [ F µ ( ω ) ≤ ˆ F ≤ (1 + ϵ ) F µ ( ω )] ≥ 1 -θ and ˆ F = f ˆ x ( ω , µ ) . The number of calls to the LM Oracle is, almost surely, at most polynomial in K , ϵ -1 , and ln θ -1 .

The Perturbed Frank-Wolfe Sampling ( P-FWS ) algorithm. The MCP algorithm allows us to solve the lower-bound problem (1) for any given µ . The latter is initially unknown, but could be estimated. A Track-and-Stop algorithm [GK16] solving (1) with this plug-in estimator in each round would yield asymptotically minimal sample complexity. It would however be computationally expensive. To circumvent this difficulty, as in [WTP21], our algorithm, P-FWS , performs a single iteration of the Frank-Wolfe algorithm for the program (1) instantiated with an estimator of µ . To apply the FrankWolfe algorithm, P-FWS uses stochastic smoothing techniques to approximate the non-differentiable objective function F µ by a smooth function. To estimate the gradient of the latter, P-FWS leverages both the LM Oracle and the MCP algorithm (more specifically its second output ˆ x ). Finally, P-FWS stopping rule takes the form of a classical Generalized Likelihood Ratio Test (GLRT) where the estimated objective function is compared to a time-dependent threshold. Hence the stopping rule also requires the MCP algorithm. We analyze the sample and computational complexities of P-FWS . Our main results are summarized in the following theorem (refer to Theorem 4 for details).

Theorem 2. For any δ ∈ (0 , 1) , P-FWS is δ -PAC, and for any ( ϵ, ˜ ϵ ) ∈ (0 , 1) small enough, its sample complexity satisfies:

̸

where H ( x ) = ln( x ) + lnln( x ) , c > 0 is a universal constant, and Ψ( ϵ, ˜ ϵ ) is polynomial in ϵ -1 , ˜ ϵ -1 , K , ∥ µ ∥ ∞ , and △ -1 min , where △ min = min x = i ⋆ ( µ ) ⟨ i ⋆ ( µ ) -x , µ ⟩ . Under P-FWS , the number of LM Oracle calls per round is at most polynomial in ln δ -1 and K . The total expected number of these calls is also polynomial.

To the best of our knowledge, P-FWS is the first polynomial time best action identification algorithm with minimal sample complexity in the high confidence regime (when δ tends to 0). Its sample complexity is also polynomial in K in the moderate confidence regime.

2 Preliminaries

We start by introducing some notation. We use bold lowercase letters (e.g., x ) for vectors, and bold uppercase letter (e.g., A ) for matrices. ⊙ (resp. ⊕ ) denotes the element-wise product (resp. sum over Z 2 ). For x ∈ R K , i ∈ N , x i = ( x i k ) k ∈ [ K ] is the i -th element-wise power of x . D = max x ∈X ∥ x ∥ 1 denotes the maximum number of arms part of an action. For any µ ∈ Λ , we define the sub-optimality gap of x ∈ X as △ x ( µ ) = ⟨ i ⋆ ( µ ) -x , µ ⟩ , and the minimal gap as △ min ( µ ) = min x = i ⋆ ( µ ) △ x ( µ ) . P µ (resp. E µ ) denotes the probability measure (resp. expectation) when the arm rewards are parametrized by µ . Whenever it is clear from the context, we will drop µ for simplicity, e.g. i ⋆ = i ⋆ ( µ ) , △ x = △ x ( µ ) , and △ min = min x = i ⋆ △ x . Refer to Appendix A for an exhaustive table of notation.

̸

̸

2.1 The lower-bound problem

Classical change-of-measure arguments lead to the asymptotic sample complexity lower bound E µ [ τ ] ≥ T ⋆ ( µ ) kl ( δ, 1 -δ ) where the characteristic time is defined in (1). To have a chance to develop a computationally efficient best action identification algorithm, we need that the sample complexity lower bound grows at most polynomially in K . This is indeed the case as stated in the following lemma, whose proof is provided in Appendix K.

We will use first-order methods to solve the lower-bound problem, and to this aim, we will need to evaluate the gradient w.r.t. ω of f x ( ω , µ ) . We can apply the envelop theorem [WTP21] to show that for ( ω , µ ) ∈ Σ + × Λ ,

where Σ + = Σ ∩ R K > 0 , λ ⋆ ω , µ ( x ) = argmin λ ∈ cl ( C x ) ⟨ ω , ( µ -λ ) 2 2 ⟩ and cl ( C x ) is the closure of C x (refer to Lemma 19 in Appendix G.2).

2.2 The Linear Maximization Oracle

As mentioned earlier, we assume that we have access to a computationally efficient Oracle, referred to as the LM (Linear Maximization) Oracle, identifying the best action if µ is known. More precisely, as in existing works in combinatorial semi-bandits [KWA + 14, PPV19, PBVP20], we make the following assumption.

Assumption 1. (i) There exists a polynomial-time algorithm identifying i ⋆ ( v ) for any v ∈ R K ; (ii) X is inclusion-wise maximal, i.e., there is no x , x ′ ∈ X s.t. x < x ′ ; (iii) for each k ∈ [ K ] , there exists x ∈ X such that x k = 1 ; (iv) |X| ≥ 2 .

Assumption 1 holds for combinatorial sets including m -sets, spanning forests, bipartite matching, s -t paths. For completeness, we verify the assumption for these action sets in Appendix J. In the design of our MCP algorithm, we will actually need to solve for some v ∈ R K the linear maximization problem max ⟨ x , v ⟩ over X \ { i ⋆ ( µ ) } ; in other words, we will probably need to identify the second best action. Fortunately, this can be done in a computationally efficient manner under Assumption 1. The following lemma formalizes this observation. Its proof, presented in Appendix J, is inspired by Lawler's m -best algorithm [Law72].

̸

Lemma 2. Let v ∈ R K and x ∈ X . Under Assumption 1, there exists an algorithm that solves max x ′ ∈X : x ′ = x ⟨ v , x ′ ⟩ by only making at most D queries to the LM Oracle.

3 Solving the lower bound problem: the MCP algorithm

Solving the lower bound problem first requires to evaluate its objective function F µ ( ω ) . A naive approach, enumerating f x ( ω , µ ) for all x ∈ X \ { i ⋆ } , would be computationally infeasible. In this section, we present and analyze MCP , an algorithm that approximates F µ ( ω ) by calling the LM Oracle a number of times growing at most polynomially in K .

3.1 Lagrangian relaxation

̸

The first step towards the design of MCP consists in considering the Lagrangian relaxation of the optimization problem defining f x ( ω , µ ) = inf λ ∈C x ⟨ ω , ( µ -λ ) 2 2 ⟩ (see e.g., [BV04, Vis21]). For any ( ω , µ ) ∈ Σ + × Λ and x = i ⋆ , the Lagrangian L ω , µ and Lagrange dual function g ω , µ of this problem are defined as, ∀ α ≥ 0 ,

respectively. The following proposition, proved in Appendix C.1, provides useful properties of g ω , µ :

Proposition 1. Let ( ω , µ ) ∈ Σ + × Λ and x ∈ X \ { i ⋆ ( µ ) } . (a) The Lagrange dual function is linear in x . More precisely, g ω , µ ( x , α ) = c ω , µ ( α ) + ⟨ ℓ ω , µ ( α ) , x ⟩

From Proposition 1 (c), strong duality holds for the program defining f x ( ω , µ ) , and we conclude:

̸

F µ ( ω ) can hence be seen as the value in a two-player game. The aforementioned properties of the Lagrange dual function will help to compute this value.

3.2 Solving the two-player game with no regret

There is a rich and growing literature on solving zero-sum games using no-regret algorithms, see for example [RS13, ALLW18, DFG21, ZODS21]. Our game has the particularity that the x -player has a discrete combinatorial action set whereas the α -player has a convex action set. Importantly, for this game, we wish not only to estimate its value F µ ( ω ) but also an equilibrium action x e such that F µ ( ω ) = max α ≥ 0 g ω , µ ( x e , α ) . Indeed, an estimate of x e will be needed when implementing the Frank-Wolfe algorithm and more specifically when estimating the gradient of F µ ( ω ) . To return such an estimate, one could think of leveraging results from the recent literature on last-iterate convergence, see e.g. [DP19, GPDO20, LNP + 21, WLZL21, APFS22, AAS + 23]. However, most of these results concern saddle-point problems only, and are not applicable in our setting. Here, we adopt a much simpler solution, and take advantage of the properties of the Lagrange dual function g ω , µ ( x , α ) to design an iterative procedure directly leading to estimates of ( F µ ( ω ) , x e ) . In this procedure, the two players successively update their actions until a stopping criterion is met, say up to the N -th iterations. The procedure generates a sequence { ( x ( n ) , α ( n ) ) } 1 ≤ n ≤ N , and from this sequence, estimates ( ˆ F, ˆ x ) of ( F µ ( ω ) , x e ) . The details of the resulting MCP algorithm are presented in Algorithm 1.

x -player. We use a variant of the Follow-the-Perturbed-Leader (FTPL) algorithm [Han57, KV05]. The x -player updates her action as follows:

̸

̸

where Z n is a random vector, exponentially distributed and with unit mean ( { Z n } n ≥ 1 are i.i.d.). Compared to the standard FTPL algorithm, we vary learning rate η n over time to get anytime guarantees (as we do not know a priori when the iterative procedure will stop). This kind of timevarying learning rate was also used in [Neu15] with a similar motivation. Note that thanks to the linearity of g ω , µ and Lemma 2, the x -player update can be computed using at most D calls to the LM Oracle.

α -player and MCP outputs. From Proposition 1, f x ( ω , µ ) = max α ≥ 0 g ω , µ ( x , α ) . This suggests that the α -player can just implement a best-response strategy: after the x -player action x ( n ) is selected, the α -player chooses α ( n ) = α ⋆ x ( n ) = △ x ( n ) ( µ ) ⟨ x ( n ) ⊕ i ⋆ ( µ ) , ω -1 ⟩ . This choice ensures that f x ( n ) ( ω , µ ) = g ω , µ ( x ( n ) , α ( n ) ) , and suggests natural outputs for MCP : should it stops after N iterations, it can return ˆ F = min n ∈ [ N ] g ω , µ ( x ( n ) , α ( n ) ) and ˆ x ∈ argmin n ∈ [ N ] g ω , µ ( x ( n ) , α ( n ) ) .

Stopping criterion. The design of the MCP stopping criterion relies on the convergence analysis and regret from the x -player perspective of the above iterative procedure, which we present in the next subsection. This convergence will be controlled by ℓ ω , µ ( α ⋆ x ) and its upper bound L ω , µ derived in Proposition 1. Introducing c θ = L ω , µ (4 √ K (ln K +1)+ √ ln( θ -1 ) / 2) , the MCP stopping criterion is: √ n > c θ (1 + ϵ ) / ( ϵ ˆ F ) . Since √ n strictly increases with n and since ˆ F ≥ F µ ( ω ) , this criterion ensures that the algorithm terminates in a finite number of iterates. Moreover, as shown in the next subsection, it also ensures that ˆ F returned by MCP is an (1 + ϵ ) -approximation of F µ ( ω ) with probability at least 1 -θ .

Algorithm 1: ( ϵ, θ ) -MCP ( ω , µ )

initialization: n = 1 , ˆ F = ∞ , c θ = L ω , µ ( 4 √ K (ln K +1) + √ ln( θ -1 ) / 2 ) ; √

θ

n

or

n

= 1)

(

n >

(

while

Sample

n

Z

x

(

n

)

α

if

n

∼

and

1

exp(1)

K

argmin

µ

(

=

i

̸

⋆

x

argmax

)

n

(

g

(

,

µ

ω

α

x

(

n

n

, α

)

+1

;

end return ( ˆ F, ˆ x ) ;

3.3 Performance analysis of the MCP algorithm

We start the analysis by quantifying the regret from the x -player perspective of MCP before its stops. The following lemma is proved in Appendix C.3.

Lemma 3. Let N ∈ N . Under ( ϵ, θ ) -MCP ( ω , µ ) ,

̸

Observe that on the one hand,

̸

always holds. On the other hand, if x e ∈ argmin x = i ⋆ max α ≥ 0 g ω , µ ( x , α ) , then we have:

̸

We conclude that for N such that √ N ≥ c θ (1+ ϵ ) ϵ ˆ F , Lemma 3 together with the inequalities (4) and (5) imply that ˆ F -F µ ( ω ) ≤ c θ √ N ≤ ϵ ˆ F 1+ ϵ holds with probability at least 1 -θ . Hence P [ ˆ F ≤ (1 + ϵ ) F µ ( ω ) ] ≥ 1 -θ . From this observation, we essentially deduce the following theorem, whose complete proof is given in Appendix C.2.

Theorem 3. Let ϵ, θ ∈ (0 , 1) . Under Assumption 1, for any ( ω , µ ) ∈ Σ + × Λ , the ( ϵ, θ ) -MCP ( ω , µ ) algorithm outputs ( ˆ F, ˆ x ) satisfying P [ F µ ( ω ) ≤ ˆ F ≤ (1 + ϵ ) F µ ( ω ) ] ≥ 1 -θ and ˆ F = max α ≥ 0 g ω , µ (ˆ x , α ) . Moreover, the number of LM Oracle calls the algorithm does is almost surely at most ⌈ c 2 θ (1+ ϵ ) 2 ϵ 2 F µ ( ω ) 2 ⌉ = O ( ∥ µ ∥ 4 ∞ ∥ ω -1 ∥ 2 ∞ K 3 D 5 ln K ln θ -1 ϵ 2 F µ ( ω ) 2 ) .

4 The Perturbed Frank-Wolfe Sampling ( P-FWS ) algorithm

̸

To identify an optimal sampling strategy, rather than solving the lower-bound problem in each round as a Track-and-Stop algorithm would [GK16], we devise P-FWS , an algorithm that performs a single iteration of the Frank-Wolfe algorithm for the lower-bound problem instantiated with an estimator of µ . This requires us to first smooth the objective function F µ ( ω ) = min x = i ⋆ f x ( ω , µ ) (the latter is not differentiable at points ω where the min is achieved for several sub-optimal actions x ). To this aim, we cannot leverage the same technique as in [WTP21], where r -subdifferential subspaces are built from gradients of f x ( ω , µ ) . These subspaces could indeed be generated by a number of vectors (here gradients) exponentially growing with K . Instead, to cope with the combinatorial

(

0

n

)

)

and set

c

η

n

-

1

m

( ∑

)

g

=1

(

,

µ

ω

<

ˆ

F

n

g

ω

)

, α

x

n

(

then

)

( ˆ

(1 +

=

,

√

µ

(

x

ϵ

)

K

/

(

ϵ

(ln

, α

(

m

)

F

ˆ

))

do

K

+1)

) +

⟨

Z

µ

/

(4

nL

2

ω

,

)

;

,

)

n

n

x

/η

⟩

(ties broken arbitrarily)

(uniqueness ensured by Proposition 1 (c))

F,

ˆ

x

)

(

g

ω

,

µ

(

x

(

n

)

;

, α

(

n

)

n

)

)

;

)

,

x

(

;

decision sets, P-FWS applies more standard stochastic smoothing techniques as described in the next subsection. All the ingredients of P-FWS are gathered in §4.2. By design, the algorithm just leverages the MCP algorithm as a subroutine, and hence only requires the LM Oracle. In §4.3, we analyze the performance of P-FWS .

4.1 Smoothing the objective function F µ

Here, we present and analyze a standard stochastic technique to smooth a function Φ . In P-FWS , this technique will be applied to the objective function Φ = F µ . Let Φ : R K > 0 ↦→ R be a concave and ℓ -Lipschitz function. Assume that the set of points where Φ is not differentiable is of Lebesguemeasure zero. To smooth Φ , we can take its average value in a neighborhood of the point considered, see e.g. [FKM05]. Formally, we define the stochastic smoothed approximate of Φ as:

where B 2 = { v ∈ R K : ∥ v ∥ 2 ≤ 1 } and η ∈ (0 , min k ∈ [ K ] ω k ) . The following proposition lists several properties of this smoothed function, and gathers together some of the results from [DBW12], see Appendix H for more details.

Proposition 2. For any ω ∈ Σ + and η ∈ (0 , min k ∈ [ K ] ω k ) , ¯ Φ η satisfies: (i) Φ( ω ) -ηℓ ≤ ¯ Φ η ( ω ) ≤ Φ( ω ) ; (ii) ∇ ¯ Φ µ ,η ( ω ) = E Z ∼ Uniform ( B 2 ) [ ∇ Φ µ ( ω + η Z )] ; (iii) ¯ Φ η is ℓK η -smooth; (iv) if η > η ′ > 0 , then ¯ Φ η ′ ( ω ) ≥ ¯ Φ η ( ω ) .

Note that with (i), we may control the approximation error between ¯ Φ η and Φ by η . (ii) and (iii) ensure the differentiability and smoothness of ¯ Φ η respectively. (iii) is equivalent to ¯ Φ η ( ω ′ ) ≤ ¯ Φ η ( ω )+ 〈 ∇ ¯ Φ η ( ω ) , ω ′ -ω 〉 + ℓK 2 η ∥ ω -ω ′ ∥ 2 2 , ∀ ω , ω ′ ∈ Σ + . Finally, (iv) stems from the concavity of Φ , and implies that the value Φ η ( ω ) monotonously increases while η decreases, and it is upper bounded by Φ( ω ) thanks to (i). The above results hold for Φ = F µ . Indeed, first it is clear that the definition (2) of F µ can be extended to R K ; then, it can be shown that F µ is Lipschitz-continuous and almost-everywhere differentiable - refer to Appendices I and H for formal proofs.

4.2 The algorithm

Before presenting P-FWS , we need to introduce the following notation. For t ≥ 1 , k ∈ [ K ] , we define N k ( t ) = ∑ t s =1 ✶ { x k ( s ) = 1 } , ˆ ω k ( t ) = N k ( t ) /t , and ˆ µ k ( t ) = ∑ t s =1 y k ( s ) ✶ { x k ( s ) = 1 } /N k ( t ) when N k ( t ) > 0 .

Sampling rule. The design of the sampling rule is driven by the following objectives: (i) the empirical allocation should converge to the solution of the lower-bound problem (1), and (ii) the number of calls to the LM Oracle should be controlled. To meet the first objective, we need in the Frank-Wolfe updates to plug an accurate estimator of µ in. The accuracy of our estimator will be guaranteed by alternating between forced exploration and FW update sampling phases. Now for the second objective, we also use forced exploration phases when in a Frank-Wolfe update, the required number of calls to the LM Oracle predicted by the upper bound presented in Theorem 3 is too large. In view of Lemma 1 and Theorem 3, this happens in round t if ∥ ˆ µ ( t -1) ∥ ∞ or △ min (ˆ µ ( t -1)) -1 is too large. Next, we describe the forced exploration and Frank-Wolfe update phases in detail.

Forced exploration. Initially, P-FWS applies the LM Oracle to compute the forced exploration set X 0 = { i ⋆ ( e k ) : k ∈ [ K ] } , where e k is the K -dimensional vector whose k -th component is equal to one and zero elsewhere. P-FWS then selects each action in X 0 once. Note that Assumption 1 (iii) ensures that the k -th component of i ⋆ ( e k ) is equal to one. In turn, this ensures that X 0 is a [ K ] -covering set, and that playing actions from X 0 is enough to estimate µ . P-FWS starts an exploration phase at rounds t such that √ t/ |X 0 | is an integer or such that the maximum of △ min (ˆ µ ( t -1)) -1 and ∥ ˆ µ ( t -1) ∥ ∞ is larger than √ t -1 . Whenever this happens, P-FWS pulls each x ∈ X 0 once.

Frank-Wolfe updates. When in round t , the algorithm is not in a forced exploration phase, it implements an iteration of the Frank-Wolfe algorithm applied to maximize the smoothed function ¯ F ˆ µ ( t -1) ,η t (ˆ ω ( t -1)) = E Z ∼ Uniform ( B 2 ) [ F ˆ µ ( t -1) (ˆ ω ( t -1) + η t Z ) ] . The sequence of parameters { η t } t ≥ 1 is chosen to ensure that η t chosen in (0 , min k ˆ ω k ( t )) , and hence ˆ ω ( t -1) + η t Z ∈ R K > 0 . Also note that in a round t where the algorithm is not in a forced exploration phase,

by definition △ min (ˆ µ ( t -1)) > 0 . This implies that ˆ µ ( t -1) ∈ Λ and that F ˆ µ ( t -1) and ¯ F ˆ µ ( t -1) ,η t (ˆ ω ( t -1)) are well-defined. Now an ideal FW update would consist in playing an action i ⋆ ( ∇ ¯ F ˆ µ ( t -1) ,η t (ˆ ω ( t -1))) = argmax x ∈X 〈 ∇ ¯ F ˆ µ ( t -1) ,η t (ˆ ω ( t -1)) , x 〉 , see e.g. [Jag13]. Unfortunately, we do not have access to ∇ ¯ F ˆ µ ( t -1) ,η t (ˆ ω ( t -1)) . But the latter can be approximated, as suggested in Proposition 2 (ii), by ∇ ˜ F ˆ µ ( t -1) ,η t ,n t (ˆ ω ( t -1)) = 1 n t ∑ n t m =1 ∇ f ˆ x m (ˆ ω ( t -1) + η t Z m , ˆ µ ( t -1)) , where Z 1 , · · · , Z n t i.i.d. ∼ Uniform ( B 2 ) , ˆ x m is the action return by ( ρ t , θ t ) -MCP (ˆ ω ( t -1) + η t Z m , ˆ µ ( t -1)) . P-FWS uses this approximation and the LM Oracle to select the action: x ( t ) ∈ i ⋆ ( ∇ ˜ F ˆ µ ( t -1) ,η t ,n t (ˆ ω ( t -1)) ) . The choices of the parameters η t , n t , ρ t and θ t do matter. η t impacts the sample complexity and should converge to 0 as t →∞ so that ¯ F µ ,η t ( ω ) → F µ ( ω ) at any point ω ∈ Σ + (this is a consequence of Proposition 2 (i)(iv)). η t should not decay too fast however as it would alter the smoothness of ¯ F µ ,η t . We will show that η t should actually decay as 1 / √ t . ( n t , ρ t , θ t ) impact the trade-off between the sample complexity and the computational complexity of the algorithm. We let n t → ∞ and ( ρ t , θ t ) → 0 as t → ∞ so that 〈 ∇ ˜ F µ ,η t ,n t ( ω ) -∇ ¯ F µ ,η t ( ω ) , x 〉 → 0 for any ( ω , x ) ∈ Σ + ×X .

Stopping and decision rule. As often in best arm identification algorithms, the P-FWS stopping rule takes the form of a GLRT:

where ϵ t ∈ R > 0 , ˆ F t is returned by the ( ϵ t , δ/t 2 ) -MCP (ˆ ω ( t ) , ˆ µ ( t ) ) algorithm. The function β satisfies

̸

Examples of function β satisfying the above conditions can be found in [GK16, JP20, KK21]. The condition (8) will ensure the δ -correctness of P-FWS , whereas (9) will control its sample complexity. Finally, the action returned by P-FWS is simply defined as ˆ ı = i ⋆ (ˆ µ ( τ )) . The complete pseudo-code of P-FWS is presented in Algorithm 2. 3

4.3 Non-asymptotic performance analysis of P-FWS

The following theorem provides an upper bound of the sample complexity of P-FWS valid for any confidence level δ , as well as the computational complexity of the algorithm.

Theorem 4. Let µ ∈ Λ and δ ∈ (0 , 1) . If P-FWS is parametrized using

̸

then (i) the algorithm finishes in finite time almost surely and P µ [ˆ ı = i ⋆ ( µ )] ≤ δ ; (ii) its sample complexity satisfies P µ [ lim sup δ → 0 τ ln δ -1 ≤ T ⋆ ( µ ) ] = 1 and for any ϵ, ˜ ϵ ∈ (0 , 1) with ϵ < min { 1 , 2 D 2 △ 2 min K , D 2 ∥ µ ∥ 2 ∞ 3 } ,

where H ( x ) = ln x +lnln x +1 and Ψ( ϵ, ˜ ϵ ) (refer to (34) for a detailed expression) is polynomial in ϵ -1 , ˜ ϵ -1 , K , ∥ µ ∥ ∞ , and △ min ( µ ) -1 ; (iii) the expected number of LM Oracle calls is upper bounded by a polynomial in ln δ -1 , K , ∥ µ ∥ ∞ , and △ min ( µ ) -1 .

The above theorem establishes the statistical asymptotic optimality of P-FWS since it implies that lim sup δ → 0 E µ [ τ ] / ln(1 /δ ) ≤ (1 + ˜ ϵ ) 2 / ( T ⋆ ( µ ) -1 -6 ϵ ) . This upper bound matches the sample complexity lower bound (1) when ϵ → 0 and ˜ ϵ → 0 .

Proof sketch. The complete proof of Theorem 4 is presented in Appendix D.

̸

(i) Correctness. To establish the δ -correctness of the algorithm, we introduce the event G under which ˆ F t , computed by ( ϵ t , δ/t 2 ) -MCP (ˆ ω ( t ) , ˆ µ ( t )) , is an (1 + ϵ t ) -approximation of F ˆ µ ( t ) (ˆ ω ( t )) in each round t ≥ 4 |X 0 | +1 . From Theorem 3, we deduce that P µ [ G c ] ≤ ∑ ∞ t =4 |X 0 | +1 δ/t 2 ≤ δ/ 4 |X 0 | . In view of (8), this implies that P µ [ˆ ı = i ⋆ ( µ )] ≤ δ .

(ii) Non-asymptotic sample complexity upper bound.

Step 1. (Concentration and certainty equivalence) We first define two good events, E (1) t and E (2) t . E (2) t corresponds to the event where ˆ µ ( t ) is close to µ , and its occurrence probability can be controlled using the forced exploration rounds and concentration inequalities. Under E (1) t , the selected action x ( t ) is close to the ideal FW update. Again using concentration results and the performance guarantees of MCP given in Theorem 3, we can control the occurrence probability of E (2) t . Overall, we show that ∑ ∞ t =1 P µ [( E (1) t ∩ E (2) t ) c ] < ∞ . To this aim, we derive several important continuity results presented in Appendix G. These results essentially allow us to study the convergence of the smoothed FW updates as if the certainty equivalence principle held, i.e., as if ˆ µ ( t ) = µ .

Step 2. (Convergence of the smoothed FW updates) We study the convergence assuming that ( E (1) t ∩ E (2) t ) holds. We first show that ¯ F µ ,η t is ℓ -Lipschitz and smooth for ℓ = 2 D 2 ∥ µ ∥ 2 ∞ , see Appendices H and I. Then, in Appendix E, we establish that the dynamics of ϕ t = max ω ∈ Σ F µ ( ω ) -F µ (ˆ ω ( t )) satisfy tϕ t ≤ ( t -1) ϕ t -1 + ℓ ( η t -1 + K 2 2 tη t ) . Observe that, as mentioned earlier, 1 / √ t is indeed the optimal scaling choice for η t . We deduce that after a certain finite number T 1 of rounds, ϕ t is sufficiently small and max {△ min (ˆ µ ( t )) -1 , ∥ ˆ µ ( t ) ∥ ∞ } ≤ √ t .

∑ ∞ t = T 1 +1 P µ [ ( E (1) t ∩ E (2) t ) c ] , and show that the second term in the r.h.s. in this inequality is equivalent to T ⋆ ( µ ) ln(1 /δ ) as δ → 0 using the property of the function β defining the stopping threshold and similar arguments as those used in [GK16, WTP21].

(iii) Expected number of LM Oracle calls. The MCP algorithm is called to compute ˆ F t and to perform the FW update only in rounds t such that max {△ min (ˆ µ ( t )) -1 , ∥ ˆ µ ( t ) ∥ ∞ } ≤ √ t . Thus, from Theorem 3 and Lemma 1, the number of LM Oracle calls per-round is a polynomial in t and K . As the E µ [ τ ] is polynomial (in ln δ -1 , K , ∥ µ ∥ ∞ and △ -1 min ), the expected number of LM Oracle calls is also polynomial in the same variables.

5 Related Work

We provide an exhaustive survey of the related literature in Appendix B. To summarize, to the best of our knowledge, CombGame [JMKK21] is the state-of-the-art algorithm for BAI in combinatorial semi-bandits in the high confidence regimes. A complete comparison to P-FWS is presented in Appendix B. CombGame was initially introduced in [DKM19] for classical bandit problems. There, the lower-bound problem is casted as a two-player game and the authors propose to use no-regret algorithms for each player to solve it. [JMKK21] adapts the algorithm for combinatorial semibandits, and provides a non-asymptotic sample complexity upper bound matching (1) asymptotically. However, the resulting algorithm requires to call an oracle solving the Most-Confusing-Parameter problem as our MCP algorithm. The authors of [JMKK21] conjectured the existence of such a computationally efficient oracle, and we establish this result here.

6 Conclusion

In this paper, we have presented P-FWS , the first computationally efficient and statistically optimal algorithm for the best arm identification problem in combinatorial semi-bandits. For this problem, we have studied the computational-statistical trade-off through the analysis of the optimization problem leading to instance-specific sample complexity lower bounds. This approach can be extended to study the computational-statistical gap in other learning tasks. Of particular interest are problems with an underlying structure (e.g. linear bandits [DMSV20, JP20], or RL in linear / low rank MDPs [AKKS20]). Most results on these problems are concerned with statistical efficiency, and ignore computational issues.

Acknowledgments and Disclosure of Funding

We thank Aristides Gionis and the anonymous reviewers for their valuable feedback. The research is funded by ERC Advanced Grant REBOUND (834862), the Wallenberg AI, Autonomous Systems and Software Program (WASP), and Digital Futures.

References

[AAS + 23] Kenshi Abe, Kaito Ariu, Mitsuki Sakamoto, Kentaro Toyoshima, and Atsushi Iwasaki. Last-iterate convergence with full-and noisy-information feedback in two-player zero-sum games. In Proc. of AISTATS , 2023.

[AKKS20] Alekh Agarwal, Sham Kakade, Akshay Krishnamurthy, and Wen Sun. Flambe: Structural complexity and representation learning of low rank mdps. In Proc. of NeurIPS , 2020.

[ALLW18] Jacob Abernethy, Kevin A Lai, Kfir Y Levy, and Jun-Kun Wang. Faster rates for convex-concave games. In Proc. of COLT , 2018.

[AMP21] Aymen Al Marjani and Alexandre Proutiere. Adaptive sampling for best policy identification in markov decision processes. In Proc. of ICML , 2021.

[APFS22] Ioannis Anagnostides, Ioannis Panageas, Gabriele Farina, and Tuomas Sandholm. On last-iterate convergence beyond zero-sum games. In Proc. of ICML , 2022.

[BBGS11] André Berger, Vincenzo Bonifaci, Fabrizio Grandoni, and Guido Schäfer. Budgeted matching and budgeted matroid intersection via the gasoline puzzle. Mathematical Programming , 2011.

[BGK22] Antoine Barrier, Aurélien Garivier, and Tomáš Kocák. A non-asymptotic approach to best-arm identification for gaussian bandits. In Proc. of AISTATS , 2022.

[BLM13] Stéphane Boucheron, Gábor Lugosi, and Pascal Massart. Concentration Inequalities: A Nonasymptotic Theory of Independence . Oxford University Press, 2013.

[BV04] Stephen Boyd and Lieven Vandenberghe. Convex optimization . Cambridge university press, 2004.

[CBL06] Nicolo Cesa-Bianchi and Gábor Lugosi. Prediction, learning, and games . Cambridge university press, 2006.

[CBL12] Nicolo Cesa-Bianchi and Gábor Lugosi. Combinatorial bandits. Journal of Computer and System Sciences , 2012.

[CCG21a] Thibaut Cuvelier, Richard Combes, and Eric Gourdin. Asymptotically optimal strategies for combinatorial semi-bandits in polynomial time. In Proc. of ALT , 2021.

[CCG21b] Thibaut Cuvelier, Richard Combes, and Eric Gourdin. Statistically efficient, polynomial-time algorithms for combinatorial semi-bandits. Proc. of SIGMETRICS , 2021.

[CGL16] Lijie Chen, Anupam Gupta, and Jian Li. Pure exploration of multi-armed bandit under matroid constraints. In Proc. of COLT , 2016.

[CGL + 17] Lijie Chen, Anupam Gupta, Jian Li, Mingda Qiao, and Ruosong Wang. Nearly optimal sampling algorithms for combinatorial pure exploration. In Proc. of COLT , 2017.

[CLK + 14] Shouyuan Chen, Tian Lin, Irwin King, Michael R Lyu, and Wei Chen. Combinatorial pure exploration of multi-armed bandits. In Proc. of NeurIPS , 2014.

[CMP17] Richard Combes, Stefan Magureanu, and Alexandre Proutiere. Minimal exploration in structured stochastic bandits. In Proc. of NeurIPS , 2017.

[CTMSP + 15] Richard Combes, Mohammad Sadegh Talebi Mazraeh Shahi, Alexandre Proutiere, et al. Combinatorial bandits revisited. In Proc. of NeurIPS , 2015.

[DBW12] John C Duchi, Peter L Bartlett, and Martin J Wainwright. Randomized smoothing for stochastic optimization. SIAM Journal on Optimization , 2012.

[DFG21] Constantinos Daskalakis, Maxwell Fishelson, and Noah Golowich. Near-optimal no-regret learning in general games. In Proc. of NeurIPS , 2021.

[DKC21] Yihan Du, Yuko Kuroki, and Wei Chen. Combinatorial pure exploration with fullbandit or partial linear feedback. In Proc. of AAAI , 2021.

[DKM19] Rémy Degenne, Wouter M Koolen, and Pierre Ménard. Non-asymptotic pure exploration by solving games. In Proc. of NeurIPS , 2019.

[DMSV20] Rémy Degenne, Pierre Ménard, Xuedong Shang, and Michal Valko. Gamification of pure exploration for linear bandits. In Proc. of ICML , 2020.

[DP19] Constantinos Daskalakis and Ioannis Panageas. Last-iterate convergence: Zero-sum games and constrained min-max optimization. Proc. of ITCS , 2019.

[FKM05] Abraham D Flaxman, Adam Tauman Kalai, and H Brendan McMahan. Online convex optimization in the bandit setting: gradient descent without a gradient. In Proc. of SODA , 2005.

[FKV14] Eugene A Feinberg, Pavlo O Kasyanov, and Mark Voorneveld. Berge's maximum theorem for noncompact image sets. Journal of Mathematical Analysis and Applications , 2014.

[GC11] A. Garivier and O. Cappé. The KL-UCB algorithm for bounded stochastic bandits and beyond. In Proc. of COLT , 2011.

[GH16] Dan Garber and Elad Hazan. A linearly convergent variant of the conditional gradient algorithm under strong convexity, with applications to online and stochastic optimization. SIAM Journal on Optimization , 2016.

[GK16] Aurélien Garivier and Emilie Kaufmann. Optimal best arm identification with fixed confidence. In Proc. of COLT , 2016.

[GPDO20] Noah Golowich, Sarath Pattathil, Constantinos Daskalakis, and Asuman Ozdaglar. Last iterate is slower than averaged iterate in smooth convex-concave saddle point problems. In Proc. of COLT , 2020.

[H + 16] Elad Hazan et al. Introduction to online convex optimization. Foundations and Trends® in Optimization , 2016.

[Han57] James Hannan. Approximation to bayes risk in repeated play. Contributions to the Theory of Games , 1957.

[HK12] Elad Hazan and Satyen Kale. Projection-free online learning. In Proc. of ICML , 2012.

[Jag13] Martin Jaggi. Revisiting frank-wolfe: Projection-free sparse convex optimization. In Proc. of ICML , 2013.

[JMKK21] Marc Jourdan, Mojmír Mutn` y, Johannes Kirschner, and Andreas Krause. Efficient pure exploration for combinatorial bandits with semi-bandit feedback. In Proc. of ALT , 2021.

[JP20] Yassir Jedra and Alexandre Proutiere. Optimal best-arm identification in linear bandits. In Proc. of NeurIPS , 2020.

[KCG16] Emilie Kaufmann, Olivier Cappé, and Aurélien Garivier. On the complexity of best-arm identification in multi-armed bandit models. JMLR , 2016.

[KK21] Emilie Kaufmann and Wouter M Koolen. Mixture martingales revisited with applications to sequential tests and confidence intervals. JMLR , 2021.

[KLLM22] Daniel Kane, Sihan Liu, Shachar Lovett, and Gaurav Mahajan. Computationalstatistical gap in reinforcement learning. In Proc. of COLT , 2022.

[KSJJ + 20] Julian Katz-Samuels, Lalit Jain, Kevin G Jamieson, et al. An empirical process approach to the union bound: Practical algorithms for combinatorial and linear bandits. In Proc. of NeurIPS , 2020.

[KTAS12] Shivaram Kalyanakrishnan, Ambuj Tewari, Peter Auer, and Peter Stone. Pac subset selection in stochastic multi-armed bandits. In Proc. of ICML , 2012.

[KV05] Adam Kalai and Santosh Vempala. Efficient algorithms for online decision problems. Journal of Computer and System Sciences , 2005.

[KWA + 14] Branislav Kveton, Zheng Wen, Azin Ashkan, Hoda Eydgahi, and Brian Eriksson. Matroid bandits: fast combinatorial optimization with learning. In Proc. of UAI , 2014.

[Lai87] Tze Leung Lai. Adaptive treatment allocation and the multi-armed bandit problem. The annals of statistics , 1987.

[Law72] Eugene L Lawler. A procedure for computing the k best solutions to discrete optimization problems and its application to the shortest path problem. Management science , 1972.

[LNP + 21] Qi Lei, Sai Ganesh Nagarajan, Ioannis Panageas, et al. Last iterate convergence in no-regret learning: constrained min-max optimization for convex-concave landscapes. In Proc. of AISTATS , 2021.

[MCP14] Stefan Magureanu, Richard Combes, and Alexandre Proutiere. Lipschitz bandits: Regret lower bounds and optimal algorithms. In Proc. of COLT , 2014.

[Neu15] Gergely Neu. First-order regret bounds for combinatorial semi-bandits. In Proc. of COLT , 2015.

[Oka73] Masashi Okamoto. Distinctness of the eigenvalues of a quadratic form in a multivariate sample. The Annals of Statistics , 1973.

[PBVP20] Pierre Perrault, Etienne Boursier, Michal Valko, and Vianney Perchet. Statistical efficiency of thompson sampling for combinatorial semi-bandits. In Proc. of NeurIPS , 2020.

[Per22] Pierre Perrault. When combinatorial thompson sampling meets approximation regret. In Proc. of NeurIPS , 2022.

[PPV19] Pierre Perrault, Vianney Perchet, and Michal Valko. Exploiting structure of uncertainty for efficient matroid semi-bandits. In Proc. of ICML , 2019.

[RG96] Ram Ravi and Michel X Goemans. The constrained minimum spanning tree problem. In Scandinavian Workshop on Algorithm Theory . Springer, 1996.

[RS13] Sasha Rakhlin and Karthik Sridharan. Optimization, learning, and games with predictable sequences. In Proc. of NeurIPS , 2013.

[S + 03] Alexander Schrijver et al. Combinatorial optimization: polyhedra and efficiency , volume 24. Springer, 2003.

[SN20] Arun Suggala and Praneeth Netrapalli. Follow the perturbed leader: Optimism and fast parallel algorithms for smooth minimax games. In Proc. of NeurIPS , 2020.

[Vis21] Nisheeth K. Vishnoi. Algorithms for Convex Optimization . Cambridge University Press, 2021.

[WLZL21] Chen-Yu Wei, Chung-Wei Lee, Mengxiao Zhang, and Haipeng Luo. Linear last-iterate convergence in constrained saddle-point optimization. In Prof. of ICLR , 2021.

[WTP21] Po-An Wang, Ruo-Chun Tzeng, and Alexandre Proutiere. Fast pure exploration via frank-wolfe. In Proc. of NeurIPS , 2021.

[ZODS21] Tom Zahavy, Brendan O'Donoghue, Guillaume Desjardins, and Satinder Singh. Reward is enough for convex mdps. In Proc. of NeurIPS , 2021.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents Perturbed Frank-Wolfe Sampling (P-FWS), an algorithm for the best arm identification problem in combinatorial semi-bandits in the fixed confidence setting. The algorithm achieves instance-specific minimal sample complexity in the high confidence regime and polynomial sample complexity guarantees in the moderate confidence regime. The authors show that P-FWS closes the computational-statistical gap in best arm identification in combinatorial semi-bandits. They describe the design of P-FWS, which starts from the optimization problem defining the information-theoretical and instance-specific sample complexity lower bound. P-FWS solves this problem in an online manner using the Frank-Wolfe algorithm with computationally efficient successive updates.\n\nSTRENGTHS:\nP-FWS is a new algorithm that addresses the best arm identification problem in combinatorial semi-bandits and achieves optimal sample complexity in both high and moderate confidence regimes. This makes it a significant contribution to the field. The paper highlights the closure of the computational-statistical gap in best arm identification in combinatorial semi-bandits. This is an important result, as it demonstrates the algorithm's efficiency in terms of both computational complexity and statistical performance.\n\nWEAKNESSES:\nThe paper primarily focuses on theoretical analysis and does not include empirical experiments or evaluations on real-world datasets. While the theoretical guarantees are valuable, empirical results could provide further insights into the algorithm's practical performance and generalizability.\n\nQUESTIONS:\nHow do you think your work could be extended beyond the LM Oracle ? For instance, in https://arxiv.org/abs/2302.11182, there are several combinatorial problems that do not fit your setting. I think it should be good to add this reference (and perhaps other) and mention (as future work or limitation) that more general combinatorial problems exist, and that your setting is restricted.\n\nLIMITATIONS:\nThe paper assumes the availability of a computationally efficient Linear Maximization (LM) Oracle, which can identify the best action given knowledge of the parameter µ. While this assumption may hold for some combinatorial sets of actions, it may not be applicable in all scenarios.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript studies the asymptotically optimal sample complexity for the best arm identification in stochastic combinatorial semi-bandits, under the fixed confidence setting. The main contribution is the introduction of a computationally efficient algorithm which achieves the asymptotically optimal sample complexity in the high-confidence regime, confirming the conjecture of [Jourdan et al. 2021] on the nonexistence of a statistical-computational gap. The proposed algorithm relies on the change-of-measure lower bound (Eqn. (1)) in [Garivier and Kaufmann, 2016], with two main ingredients: \n\n1. To solve the inner minimization problem, using the Lagrangian form the authors formulate it as a two-player zero-sum game. The minimizing player (choosing a super arm) uses a FTPL algorithm which is oracle-efficient based on a linear maximization (LM) oracle; the maximizing player (choosing the Lagrangian parameter) plays the best response. The standard result of online learning shows that the average plays converge to the minimax solution. Significance: medium. \n\n2. To solve the outer maximization problem, the authors use a perturbed Frank-Wolfe sampling algorithm. First, the objective function is smoothed by a Gaussian kernel to overcome the issue of multiple gradients. Second, the learner computes the estimated gradient for the smoothed objective and plays the best response to the estimated gradient. A stopping rule, as well as forced exploration rules, is also applied in the algorithm. Significance: medium.\n\nSTRENGTHS:\nThis manuscript proved the nonexistence of the statistical-computational gap in stochastic combinatorial semi-bandits and resolved a conjecture of [Jourdan et al. 2021]. The components in the algorithm design, i.e. the two-player games and the perturbed Frank-Wolfe sampling, are both interesting.\n\nWEAKNESSES:\nOverall I like this paper and lean towards acceptance. However, I do have some concerns on the significance of the results: \n\n1. The problem in study, i.e. a computationally efficient algorithm for best arm identification in stochastic combinatorial semi-bandits under the fixed confidence setting, is very specific and a bit narrow in scope. The proposed algorithm is very tailored to the current setting, and the results are asymptotic. The motivation of this manuscript seems to be mainly technical curiosity, where the proposed algorithm is too complicated and very unlikely to be used by practitioners. \n\n2. The nonexistence of statistical-computational gap is not very surprising, which slightly lowers the significance of the result. Also, the attained sample complexity is asymptotically optimal only in the high confidence regime and also involves a parameter polynomial in 1/Delta_min. \n\n3. The two-player zero-sum game approach for an oracle-efficient algorithm is nowadays pretty standard, especially when one player is playing FTPL and the other is using best response. So the significance of the first component is medium. \n\n4. The Frank-Wolfe sampling approach, albeit different from [Wang et al. 2021], is also pretty natural. The same also applies to the smoothing technique. The complicated notations and many seemingly arbitrary rules in the algorithm (more on the writing below) make it considerably harder to appreciate the main idea, and also degrade the significance. \n\nIn addition, I would like to point out that the writing quality of this manuscript is pretty poor. Just list several examples: \n\n1. The LM oracle is not even formally defined in the main text. \n\n2. Algorithm 2 is poorly explained. Why forced exploration when \\sqrt(t/|X_0|) \\in \\bN? Why does the best response to the estimated gradient correspond to a good decision? The definition of \\nabla F_{\\mu, \\eta, n} is deferred to a very later place. What is the intuition behind the stopping rule? What is the beta function in the stopping rule? Many components require explanations. \n\n3. Small places: \"would\" in Line 220, \"in\" in Line 257.\n\nQUESTIONS:\nThe remainder term in Theorem 2 depends on Delta_min, which is pretty unfortunate, for this quantity could be small in combinatorial bandits. I am wondering if one can argue that this term is always negligible compared with the main term? More specifically, when Delta_min is small, then the optimal sample complexity T^* would also be large; I'm wondering if T^* could always dominate the poly(1/Delta_min) term in the remainder.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper is about best arm identification with fixed confidence, in combinatorial semi-bandits. The state of the art in that setting is that we have algorithms that have optimal asymptotic sample complexity, but their computational complexity is very large. The authors provide a new algorithm which remediates this problem. The algorithm is asymptotically optimal and has polynomial computational complexity (in the number of arms and parameters of the problem like the minimal gap).\nThe main contribution is a method to compute efficiently the closest alternative to an instance of the combinatorial semi-bandit problem, which is a sub-routine is several BAI algorithms. That method is incorporated into a BAI algorithm based on Frank-Wolfe.\n\nSTRENGTHS:\nThe algorithm presented is the first to have polynomial computational complexity in combinatorial BAI. This is a significant achievement towards having practical algorithms for that important problem. The sample complexity is also asymptotically optimal.\n\nThe new method MCP is the result of an interesting reformulation of the problem and a thorough analysis of its properties. Its integration in a Frank-Wolfe algorithm also uses innovative methods compared to the existing FW algorithm for BAI.\n\nThe algorithms and the results are clearly explained.\n\nWEAKNESSES:\nThe dependence in K, D and other constants is indeed polynomial, but with very large exponents. Some bounds depend on K^15. Since this work presents the first algorithm with polynomial complexity, this is only a mild weakness.\n\nThe main weakness is that there is no experimental evaluation.\n- Is the method still very inefficient in practice? Can it actually run on a computer?\n- Some components of the algorithm seem to be included to make the analysis work, like the comparison of \\sqrt{t} with the norm of the empirical mean vector. A natural question is whether they are necessary in practice, or if they are artifacts of the analysis. This could have been studied empirically.\n- The paper does not provide only a computationally efficient subroutine for a known algorithm, but also proposes a new BAI algorithm based on Frank-Wolfe (another FW based method already exists, but it is not exactly the same). The sample complexity of the new method should then also be evaluated. Perhaps an innocuous looking modification made it very bad in practice, although the asymptotic guarantees are good?\n\nQUESTIONS:\nCan you provide an empirical analysis of the proposed method and compare it to other algorithms? Both for sample and computational complexity.\n\nLIMITATIONS:\nThe limitations of the paper are adequately discussed. No concern about societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a computationally efficient algorithm, P-FWS (Perturbed Frank-Wolfe Sampling), designed for best arm identification in Combinatorial semi-bandits. This algorithm operates in polynomial time and offers minimal sample complexity guarantees or polynomial sample complexity, depending on the problem regime. P-FWS is an online estimation algorithm that aims to achieve the information-theoretic lower bound at each stage. It leverages the Frank-Wolfe optimization routine as its core component, relying on linear maximization at each step. Overall, the paper presents an innovative and efficient approach to address the best arm identification problem in Combinatorial semi-bandits.\n\nSTRENGTHS:\nOne strength of the paper lies in its exploration of a novel and important problem within the bandit literature. The utilization of Perturbed Frank-Wolfe as the central algorithm for estimating the lower bound is a unique and valuable contribution that has the potential to complement and enhance existing works in the field. The paper showcases strong theoretical rigor by providing detailed proofs for all claims, ensuring the reliability and robustness of the presented findings. Overall, the paper's innovative problem formulation and theoretical foundation make it a valuable addition to the broader bandit literature.\n\nWEAKNESSES:\nOne weakness of the paper is its challenging readability and navigation. The excessive use of definitions, constants, and interruptions in the flow of the paper makes it difficult to understand and follow. This overload of information can overwhelm the reader and hinder comprehension.\n\nFurthermore, the practical motivation and real-world use cases of the proposed algorithms are discussed in a limited scope. It would be beneficial to expand on the potential applications and provide more examples of how such algorithms can be applied in real-world scenarios.\n\nAdditionally, the absence of simulations in the main paper is notable. Although the paper highlights the computational efficiency of the proposed algorithms, there is a lack of performance comparison with other non-efficient methods. Including such comparisons would provide valuable insights into the comparative advantages and disadvantages of the proposed approach.\n\nOverall, addressing these weaknesses would enhance the clarity, applicability, and overall impact of the paper.\n\nQUESTIONS:\nWould this paper have any significance or relevance to the part of the bandit literature that focuses on tracking the lower bound and sampling based on such lower bound estimations, which are often computationally infeasible?\n\nLIMITATIONS:\nNo potential negative societal impact discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}