Fast Pure Exploration via Frank-Wolfe

Abstract: We study the problem of active pure exploration with fixed confidence in generic stochastic bandit environments. The goal of the learner is to answer a query about the environment with a given level of certainty while minimizing her sampling budget. For this problem, instance-specific lower bounds on the expected sample complexity reveal the optimal proportions of arm draws an Oracle algorithm would apply. These proportions solve an optimization problem whose tractability strongly depends on the structural properties of the environment, but may be instrumental in the design of efficient learning algorithms. We devise Frank-Wolfe-based Sampling (FWS), a simple algorithm whose sample complexity matches the lower bounds for a wide class of pure exploration problems. The algorithm is computationally efficient as, to learn and track the optimal proportion of arm draws, it relies on a single iteration of Frank-Wolfe algorithm applied to the lower-bound optimization problem. We apply FWS to various pure exploration tasks, including best arm identification in unstructured, thresholded, linear, and Lipschitz bandits. Despite its simplicity, FWS is competitive compared to state-of-art algorithms.


Reviews and author responses:

This paper studies pure exploration in generic stochastic bandit problems under fixed confidence budgets. At the core of this approach is relying on a single iteration of frank-wolf (FW) to solve a non-smooth problem. The single iteration of FW reduces the computational burden, but is challenged by the non-smoothness. The authors successfully cope with the challenge and the resultant algorithm looks theoretically sound. In addition, the numerical results are promising in certain cases, though have a similar performance with Bernoulli and Gaussian rewards.  (+) Overall, this paper is technically sound. FW is a very good choice for such a problem since the constraint sets are in general polytopes. 

(-) This paper can benefit from adding more intuition, and some figures to explain the assumptions and related concepts. It is not easy from first read to gather all necessary info to understand the proposed algorithm. 

Suggestions and Questions:

i) Is there any intuition behind Assumption 3? And is it easy to verify in practice?

ii) What is the run time compared with e.g., [13] and [29]?

Minor: In line 339, it looks like there is a typo for the name of an algorithm from [9]. This paper mainly focuses on the theoretical side. The societal impact may depend on specific applications.

 Thanks for the detailed response. My concerns are addressed, and have increased my score. 

The paper analyses Frank-Wolfe for pure exploration problems on structured bandits.  To address smoothness issues that arise from a direct application of FW to the lower-bound optimization problem, the authors compute the best-response as the solution to a saddle point problem that includes the convex hull over gradients of approximate minimizers. The main result is that the resulting algorithm is PAC and asymptotically optimal on the sample complexity. Empirically, FW outperforms state-of-the-art methods on the presented benchmarks. The authors further characterize lower order terms of the sample complexity.  Originality: The authors adequately discuss related work. FW has been applied in the context of best-arm identification before, but the novelty is to show convergence and address the non-smoothness of the lower bound. The resulting algorithm remains relatively simple, however still requires to solve a (linear) saddle point problem on each step.

Quality & Clarity:
The paper is generally well-written and the important ideas are presented in the main paper. There is also a detailed appendix that includes several example applications. I have not checked the proofs in detail. It would be great to mention how the r-subdifferential space is used in the proof to show convergence of FW (ie summarizing Appendix L). Please also mention the choice of r_t in the main paper (l 653). Is the algorithm sensitive to the choice? What is the reason behind your choice? Does the algorithm empirically converge for r_t = 0?

Significance:
This paper makes an important technical contribution to the best-arm / pure-exploration literature; moreover the algorithm is relatively simple and outperforms other methods empirically. 

Minor: 
95: 'optimism in the face of uncertainty' is more common
114: 'the' objective function

[edit]
I'd like to thank the authors for their detailed responses and clarifications. The main limitation of the algorithm is that it still requires to solve a linear saddle point problem, which is more expensive than standard Frank-Wolfe. The compute complexity depends on how many near-optimal actions are in the r-subdifferential. My intuition is that at the equilibrium of (1), there can be a lot of actions in this set.

 Thank you for your insightful comments. We will address them and revise our paper accordingly. Please find below answers to each of your comments. Reference numbers correspond to those provided in the supplementary material (long version of the paper). 

Main comments:

1. Complexity of BC. In each round, BC solves a small LP with m variables where m is the dimension of the r-subdifferential space. We do not have guarantees on m but experiments suggest that it remains small – see Appendix D Fig. 1 where 40% of the time m=1 for unstructured bandits, and Appendix E Fig. 3 where m is almost always equal to 1. For Lipschitz bandits, we re-run our experiments, and in about 70% of rounds m=1; we will add these results in the paper.\
\
In comparison, in the case of Best Arm Identification (BAI) in unstructured bandits, the TaS algorithm [21] and LMA [39] have a complexity equal to $O(K(\log (\frac{1}{\epsilon})^2))$ (where epsilon is the precision of the binary search algorithm used to compute the update) and $O(K)$ per round, respectively. For the gamification approach [13], the time complexity mainly depends on the two regret minimization algorithms used as subroutines. The paper [39] provides a precise account of the complexity of various baseline algorithms (see Appendix B in [39]). From the above discussion, it is difficult to compare the complexity of the various algorithms analytically. We will provide results from experiments: BC has a better run time than TaS but worse than LMA.\
\
In the case of BAI in structured bandits, it is still difficult to compare the running time of various algorithms analytically.  However, BC is the only algorithm with strong theoretical and empirical performance guarantees and good running time. In linear bandits, all the presented algorithms are fast. LT-H [26] is the fastest but has a much higher sample complexity than BC; BC is faster than LinGame and LinGame-C [14]. In Lipschitz bandits, LMA [39] is faster than BC and BC is faster than TaS. But, as also noticed in [32], to make LMA work, we need to tune its learning rate as a function of the unknown parameter $\mu$ (which is impossible in practice). When using the learning rate proposed in [39], LMA is close to uniform sampling and has high sample complexity. In addition, we are not sure about its theoretical performance guarantees for Lipschitz bandits.

2. Name of the algorithm. We agree with you that the algorithm is now far from the initial idea of the Best Challenger algorithm proposed in [21]. We will follow your recommendation and call our algorithm NS-FW algorithm (Non-Smooth FW algorithm).
 
3. Avoiding the forced exploration phase. When the curvature remains bounded (e.g., in most threshold bandit problems), forced exploration can be avoided and upper confidence bounds as [5] can be used instead. When the curvature may become infinite, we need to exploit the specific structure of the problem to be able to remove the forced exploration phase. For example, for BAI in linear bandits, we only need to explore each dimension as done in [26].
 
Specific comment: 

Thanks for all the suggestions. We will modify the paper accordingly.

L168: Thanks for the references. We will add them.

L224: Yes, for all w.

L240: Then it is defined as 0.

L263: It should be corrected by adding “or $\mu \notin \Lambda$”.

L298: See above. 

L344: In linear bandits, there might be several optimal allocations. The latter form a convex set, see [26]. In such a case, we only need to implement an allocation close to this set.

L354: The upper bound is exponential in K, a dependency that comes from the very conservative use of concentration inequalities. For linear bandits, we replace the concentration inequality by that used and proved in [26].

L1012 and 990: We need Assumption 1, $\Lambda$ is an open set, to ensure that $\pi$ (or $\hat{\mu}(t-1)$) $\in \Lambda$ when $\xi(\epsilon)$ is small.

L998: We can fix this using another concentration inequality.

L1145: Infinity norm.

L1170: The equation comes from L1165; the inequality follows from L1164 and the fact that $z\in \Sigma$.

L1195: No, it holds as the solution $z(t)$ at the t-th round is computed based on the $\hat{\mu}(t-1)$.

L1218: We will correct it by including the event ${\hat{\mu}(t-1)\in \Lambda}$ into our good event.

L964: It can be any constant in (0,1).

L1252, L1004, L1174, L1218, L1245, L959, and L961: Thanks for these corrections, we will include them.


 Thank you for your insightful comments. We will address them and revise our paper accordingly. Please find answers to each of your comments below. Reference numbers correspond to those provided in the main document (short version of the paper).

(i) About Assumption 3. As described in the proof sketch of Theorem 1 (Section 4.3), we need to ensure that the functions are continuous in $\mu$. To conduct a non-asymptotic analysis of the sample complexity (i.e., Theorem 2), we need even stronger continuity results (at least Lipschitz continuity). The technical Assumption 3 allows us to upper bound the gradients of the functions $F_\mu$ and of that defined in (11) (as it turns out, these gradients involve the gradient of the function d, hence Assumption 3). Usually, Assumption 3 is easy to verify, and all the problems discussed in Appendix D, E, F, G satisfy this assumption.

(ii) About the run time of BC. In each round, BC solves a small LP with m variables where m is the dimension of the r-subdifferential space. We do not have guarantees on m but experiments suggest that it remains small – see Appendix D Fig. 1 where 40% of the time m=1 for unstructured bandits, and Appendix E Fig. 3 where m is almost always equal to 1. For Lipschitz bandits, we re-run our experiments and in about 70% of rounds m=1; we will add these results in the paper.

In comparison, in the case of Best Arm Identification (BAI) in unstructured bandits, the TaS algorithm [13] and LMA [29] have a complexity equal to $O(K\log(\frac{1}{\epsilon})^2))$ (where epsilon is the precision of the binary search algorithm used to compute the update) and $O(K)$ per round, respectively. For the gamification approach [8], the time complexity mainly depends on the two regret minimization algorithms used as subroutines. The paper [29] provides a precise account of the complexity of various baseline algorithms (see Appendix B in [29]). From the above discussion, it is difficult to compare the complexity of the various algorithms analytically. We will provide the results from experiments: BC has a better run time than TaS but worse than LMA.

In the case of BAI in structured bandits, it is still difficult to compare the running time of various algorithms analytically.  However, BC is the only algorithm with strong theoretical and empirical performance guarantees and good running time. In linear bandits, all the presented algorithms are fast. LT-H [18] is the fastest but has a much higher sample complexity than BC; BC is faster than LinGame and LinGame-C [9]. In Lipschitz bandits, LMA [29] is faster than BC and BC is faster than TaS. But, as also noticed in [23], to make LMA work, we need to tune its learning rate as a function of the unknown parameter $\mu$ (which is impossible in practice). When using the learning rate proposed in [29], LMA is close to uniform sampling and has high sample complexity. In addition, we are not sure about its theoretical performance guarantees for Lipschitz bandits. 


 Thank you for your insightful comments. We will address them and revise our paper accordingly. Please find answers to each of your comments below. Reference numbers correspond to those provided in the supplementary material (long version of the paper).

1. Choice of $r_t$. The performance of BC depends on the rate at which $r_t$ decays. Usually, the faster it decays, the smaller the sample complexity is. For BAI in unstructured bandits, when the decay rate is $t^{-0.5}$, BC requires about 200 more samples than when the rate is $t^{-0.9}$ (see Table 2). Note that if $r_t=0$, then according to (10), $H_{F_{\hat{\mu}(t-1)}}$ is empty. Now, if $r_t$ is equal to a very small constant so that $H_{F_{\hat{\mu}(t-1)}}$ only contains a single vector, then the behavior of BC is almost identical as that of the original BC algorithm proposed in [21]. In this case, BC fails to converge to the optimal allocation in some cases (this sequence violates (i) in Theorem 1). We choose ${r_t}$ as ${t^{-0.9}}$ because it works well for all the problems implemented in our paper.

2. Implementation complexity of BC. As you mention, the complexity of BC per round depends on the number m of near-optimal actions in the r-subdifferential space. For BAI in unstructured bandits, this number is m=1 40% of the time (see Figure 1 in Appendix D), and higher values of m occur less frequently (we will add numerical experiments showing the histogram for m). For BAI in linear bandits, surprisingly, m=1 most of the rounds (see Figure 3 in Appendix E). For Lipschitz bandits, m=1 for more than 70% of the rounds – we will add these results to the paper.


The authors consider the active exploration problem in the fixed confidence setting for stochastic bandits. They propose a variant of the Best Challenger (BC) algorithm by [21] where the sampling rule is based on a variant of the Frank-Wolfe (FW) algorithm for non-smooth functions. They proved that this algorithm is asymptotically optimal with respect to the lower bound by [21]. They compare empirically their algorithm to various baselines such that Track-and-Stop by [21], AdaHedge versus Best-Response by [13], Lazy Mirror Ascent by [29] and Optimistic Track and Stop by [13] for the following unstructured active exploration problem: threshold bandits, best arm identification, top-m bandits and dueling bandits. They also explain how to adapt the proposed BC algorithm to two structured settings namely linear bandits and Lipschitz bandits for best arm identification. Finally, they conduct experiments in these two additional settings.  The paper is well-written. The proofs seem globally correct (see specific comments for some details that are not completely clear, I did not check the appendix K and N). I think the proposed work is a valuable contribution to the fixed-confidence BAI topic. In particular, I find the proposed "fix" of the BC algorithm very interesting.

Main comments:
-Can you elaborate on the time complexity of your algorithm and compare it to the other baselines?
In particular, is it much faster to solve the linear program in (11) than directly find the maximum of F_{\mu} which can be done efficiently also (see [21]).
-I find the name of the algorithm BC and claiming that it is based on the FW algorithm a bit miss-leading. Indeed, at each step the proposed algorithm does not pull an arm between the best arm and a certain best challenger in fact z(t) could put mass on all the arms depending on the r-sub differential. This is far enough from the original BC algorithm to justify naming it differently. Similarly, the optimizer behind the sampling is not the FW algorithm by [12] even if it is close in the spirit. Maybe using the names, e.g.,  non-smooth FW and FW algorithm would make the introduction and the abstract clearer.
-Do you think it will possible to avoid the forced exploration? Indeed this seems a crucial point in the analysis here to bound the curvature cf assumption 2. In particular, this could be useful for the linear setting with many arms because with the force exploration you have to pull each time all the K arms.

Specific comments:
-L168: You can also mention Carpentier et al. (2011). Upper-confidence-bound algorithms for active learning in multi-armed bandits and Tarbouriech and Lazaric (2019) Active Exploration in Markov Decision Processes, for another pure exploration problem in bandit and reinforcement learning where FW is used.

-L216: maybe you should define at some point the FW algorithm at least in the appendix and highlight that you want to use it to find the optimal weights that are a minimizer of F.

-L224: Is it for all w?

-L237: maybe you should describe in detail the learning procedure before.

-L240: and if N_k(t) = 0?

-L263, Algorithm 1, while condition loop: a priori F_\mu is only defined for \mu in \Delta

-L298: Can you detail the overall time complexity of the proposed best challenger algorithm since it's a bit unclear because of the linear program to solve.

-L344: I do not understand this sentence especially given paragraph 5.2.

-L354: Can you elaborate on the dependence on the number of arms K of the bound? And does this theorem apply to linear bandits?

-L1012: you need to precise that \mu \in \Delta so that F_\mu and H_{F_\mu} are well defined.

-L990: how do we know that \pi = \hat{\mu}^{t-1} belongs to \Delta in order to apply Theorem 3?

-L1252: z_k(s) instead of z_k(t)

-L1004, Lemma 5: why there is a s\geq 1?

-L998, (44): it seems you upper bound \xi(\epsilon) by 1 maybe you should let \xi(\epsilon) = \min{1,\ldots}.

-L1145: with respect to which norm?

-L1170: |z-x|_{\infty} and there is no |z-x|_{\infty}^2 below no? 

-L1174: min_n \phi_n

-L1195: It should be \Delta_{t+1} and \Delta_{t} in (49) no?

-L1218: The equality is rather with x instead of w?

-L1221: again it seems that the case where \hat{\mu}(t-1) \notin \Delta is not considered.

-L1239: you should not rather control the norm of x(t) - x(Kq^2) here?

-L1245: an inequality is missing here.

-L959: w(t)

-L961: can you detail the inclusion? Is it for T\geq T(\delta)?

-L964: what is the definition of \tilde{\epsilon}? It is a theoretical paper without direct societal impact.

Meta-review: The paper makes a definitive contribution to the active exploration MAB literature. The core idea is not completely new but the modifications and variations proposed in the paper are meaningful advances relative to state of the art best arm identification algorithms. The paper is well written. The reviewers are all in favor of accepting, the dialogue with the authors was informative and honest with several issues still left open but for well justified reasons (e.g., computational complexity considerations). I would make this a clear accept but not a top paper because of the nature of contribution being relatively technical relative to antecedent work and some of the documented computational limitations. Thanks to the reviewers and authors for well done work and the fruitful exchange.  

Decision: accept