Combinatorial Pure Exploration with Bottleneck Reward Function

1 Introduction . The Multi-Armed Bandit ( MAB ) problem [ 25 , 30 , 4 , 2 ] is a classic model to solve the explorationexploitation trade-off in online decision making . Pure exploration [ 3 , 21 , 7 , 26 ] is an important variant of the MAB problem , which aims to identify the best arm under a given confidence or a given sample budget . There are various works studying pure exploration , such as top-k arm identification [ 17 , 21 , 7 , 24 ] , top-k arm under matriod constraints [ 9 ] and multi-bandit best arm identification [ 18 , 7 ] . The Combinatorial Pure Exploration ( CPE ) framework , firstly proposed by Chen et al . [ 11 ] , encompasses a rich class of pure exploration problems [ 3 , 21 , 9 ] . In CPE , there are a set of base arms , each associated with an unknown reward distribution . A subset of base arms is called a super arm , which follows a certain combinatorial structure . At each timestep , a learner plays a base arm and observes a random reward sampled from its distribution , with the objective to identify the optimal super arm 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . with the maximum expected reward . While Chen et al . [ 11 ] provide this general CPE framework , their algorithms and analytical techniques only work under the linear reward function and can not be applied to other nonlinear reward cases.1 However , in many real-world scenarios , the expected reward function is not necessarily linear . One of the common and important cases is the bottleneck reward function , i.e. , the expected reward of a super arm is the minimum expected reward of the base arms contained in it . For example , in communication networks [ 5 ] , the transmission speed of a path is usually determined by the link with the lowest rate , and a learner samples the links in order to find the optimal transmission path which maximizes its bottleneck link rate . In traffic scheduling [ 31 ] , a scheduling system collects the information of road segments in order to plan an efficient route which optimizes its most congested ( bottleneck ) road segment . In neural architecture search [ 32 ] , the overall efficiency of a network architecture is usually constrained by its worst module , and an agent samples the available modules with the objective to identify the best network architecture in combinatorial search space . In this paper , we study the Combinatorial Pure Exploration with the Bottleneck reward function ( CPE-B ) which aims to identify the optimal super arm with the maximum bottleneck value by querying the base arm rewards , where the bottleneck value of a super arm is defined as the minimum expected reward of its containing base arms . We consider two popular settings in pure exploration , i.e , fixed-confidence ( FC ) , where given confidence parameter δ , the learner aims to identify the optimal super arm with probability 1− δ and minimize the number of used samples ( sample complexity ) , and fixed-budget ( FB ) , where the learner needs to use a given sample budget to find the optimal super arm and minimize the error probability . Challenges of CPE-B . Compared to prior CPE works [ 11 , 10 , 20 ] , our CPE-B aims at utilizing the bottleneck property to save samples and achieve the statistical optimality . It faces with two unique challenges , i.e. , how to ( i ) achieve the tight base-arm-gap dependent sample complexity and ( ii ) avoid the dependence on unnecessary base arms in the results , while running in polynomial time . We use a simple example in Figure 1 to illustrate our challenges . In Figure 1 , there are six edges ( base arms ) and three s-t paths ( super arms ) , and the base arm reward w ( ei ) , base arm gap ∆ei , ej and super arm gap ∆M∗ , Msub are as shown in the figure . In order to identify the optimal path , all we need is to pull e1 , e2 , e4 to determine that e1 is worse than e2 and e4 , and e3 , e5 , e6 are useless for revealing the sub-optimality of M1 and M2 . In this case , the optimal sample complexity should be O ( ( 2∆2e2 , e1 + 1∆2e4 , e1 ) ln δ−1 ) , which depends on the tight base arm gaps and only includes the critical base arms ( e1 , e2 , e4 ) . However , if one naively adapts existing CPE algorithms [ 11 , 12 , 16 ] to work with bottleneck reward function , an inferior sample complexity of O ( ∑ ei , i∈ [ 6 ] 1 ∆2M∗ , Msub ln δ−1 ) is incurred , which depends on the loose super arm gaps and contains a summation over all base arms ( including the unnecessary e3 , e5 , e6 ) . Hence , our challenge falls on how to achieve such efficient sampling in an online environment , where we do not know which are critical base arms e1 , e2 , e4 but want to gather just enough information to identify the optimal super arm . We remark that , none of existing CPE studies can be applied to solve the unique challenges of CPE-B , and thus we develop brand-new techniques to handle them and attain the optimal results ( up to a logarithmic factor ) . Contributions . For CPE-B in the FC setting , ( i ) we first develop a novel algorithm BLUCB , which employs a bottleneck-adaptive sample strategy and achieves the tight base-arm-gap dependent sample complexity . ( ii ) We further propose an improved algorithm BLUCB-Parallel in high confidence regime , which adopts an efficient “ bottleneck-searching ” offline procedure and a novel “ check-nearbottleneck ” stopping condition . The sample complexity of BLUCB-Parallel drops the dependence on unnecessary base arms and achieves the optimality ( within a logarithmic factor ) under small enough δ . ( iii ) A matching sample complexity lower bound for the FC setting is also provided , which demonstrates the optimality of our algorithms . For the FB setting , ( iv ) we propose a novel algorithm BSAR with a special acceptance scheme for the bottleneck identification task . BSAR achieves the state-of-the-art error probability and is the first to run efficiently on fixed-budget path instances , 1The algorithmic designs and analytical tools ( e.g. , symmetric difference and exchange set ) in [ 11 ] all rely on the linear property and can not be applied to nonlinear reward cases , e.g , the bottleneck reward problem . compared to existing CPE algorithms . All our proposed algorithms run in polynomial time.2 The experimental results demonstrate that our algorithms significantly outperform the baselines . Due to space limit , we defer all the proofs to the supplementary material . 1.1 Related Work . In the following we briefly review the related work in the CPE literature . Chen et al . [ 11 ] firstly propose the CPE model and only consider the linear reward function ( CPE-L ) , and their results for CPE-L are further improved by [ 19 , 10 ] . Huang et al . [ 20 ] investigate the continuous and separable reward functions ( CPE-CS ) , but their algorithm only runs efficiently on simple cardinality constraint instances . All these works consider directly sampling base arms and getting their feedback . There are also several CPE studies which consider other forms of sampling and feedback . Chen et al . [ 12 ] propose the CPE for dueling bandit setting , where at each timestep the learner pulls a duel between two base arms and observes their comparison outcome . Kuroki et al . [ 23 ] study an online densest subgraph problem , where the decision is a subgraph and the feedback is the reward sum of the edges in the chosen subgraph ( i.e. , full-bandit feedback ) . Du et al . [ 16 ] investigate CPE with the full-bandit or partial linear feedback . All of the above studies consider the pure exploration setting , while in combinatorial bandits there are other works [ 13 , 15 , 14 ] studying the regret minimization setting ( CMAB ) . In CMAB , the learner plays a super arm and observes the rewards from all base arms contained in it , with goal of minimizing the regret , which is significantly different from our setting . Note that none of the above studies covers our CPE-B problem or can be adapted to solve the unique challenges of CPE-B , and thus CPE-B demands a new investigation . 2 Problem Formulation . In this section , we give the formal formulation of CPE-B . In this problem , a learner is given n base arms numbered by 1 , 2 , . . . , n. Each base arm e ∈ [ n ] is associated with an unknown reward distribution with the mean of w ( e ) and an R-sub-Gaussian tail , which is a standard assumption in bandits [ 1 , 11 , 26 , 29 ] . Let w = ( w ( 1 ) , . . . , w ( n ) ) > be the expected reward vector of base arms . The learner is also given a decision class M ⊆ 2 [ n ] , which is a collection of super arms ( subsets of base arms ) and generated from a certain combinatorial structure , such as s-t paths , maximum cardinality matchings , and spanning trees . For each super arm M ∈ M , we define its expected reward ( also called bottleneck value ) as MinW ( M , w ) = mine∈M w ( e ) ,3 i.e. , the minimum expected reward of its constituent base arms , which is so called bottleneck reward function . Let M∗ = argmaxM∈M MinW ( M , w ) be the optimal super arm with the maximum bottleneck value , and OPT = MinW ( M∗ , w ) be the optimal value . Following the pure exploration literature [ 17 , 11 , 10 , 12 ] , we assume that M∗ is unique , and this assumption can be removed in our extension to the PAC learning setting ( see the supplementary material ) . At each timestep , the learner plays ( or samples ) a base arm pt ∈ [ n ] and observes a random reward sampled from its reward distribution , where the sample is independent among different timestep t. The learner ’ s objective is to identify the optimal super arm M∗ fromM . For this identification task , we study two common metrics in pure exploration [ 21 , 7 , 26 , 10 ] , i.e. , fixed-confidence ( FC ) and fixed-budget ( FB ) settings . In the FC setting , given a confidence parameter δ ∈ ( 0 , 1 ) , the learner needs to identify M∗ with probability at least 1− δ and minimize the sample complexity , i.e. , the number of samples used . In the FB setting , the learner is given a fixed sample budget T , and needs to identify M∗ within T samples and minimize the error probability , i.e. , the probability of returning a wrong answer . 3 Algorithms for the Fixed-Confidence Setting . In this section , we first propose a simple algorithm BLUCB for the FC setting , which adopts a novel bottleneck-adaptive sample strategy to obtain the tight base-arm-gap dependent sample complexity . 2Here “ polynomial time ” refers to polynomial time in the number of base arms n ( which is equal to the number of edges E in our considered instances such as s-t paths , matchings and spanning trees ) . 3In general , the second input of function MinW can be any vector : for any M ∈ M and v ∈ Rn , MinW ( M , v ) = mine∈M v ( e ) . Algorithm 1 BLUCB , algorithm for CPE-B in the FC setting 1 : Input : M , δ ∈ ( 0 , 1 ) and MaxOracle . 2 : Initialize : play each e ∈ [ n ] once , and update empirical means ŵn+1 and Tn+1 3 : for t = n+ 1 , n+ 2 , . . . do 4 : radt ( e ) ← √ 2 ln ( 4nt 3 δ ) /Tt ( e ) , ∀e ∈ [ n ] 5 : wt ( e ) ← ŵt ( e ) − radt ( e ) , ∀e ∈ [ n ] 6 : w̄t ( e ) ← ŵt ( e ) + radt ( e ) , ∀e ∈ [ n ] 7 : Mt ← MaxOracle ( M , wt ) 8 : M̃t ← MaxOracle ( M\ S ( Mt ) , w̄t ) 9 : if MinW ( Mt , wt ) ≥ MinW ( M̃t , w̄t ) then 10 : return Mt 11 : end if 12 : ct ← argmine∈Mt wt ( e ) 13 : dt ← argmine∈M̃t wt ( e ) 14 : pt ← argmaxe∈ { ct , dt } radt ( e ) 15 : Play pt , and observe the reward 16 : Update empirical means ŵt+1 ( pt ) 17 : Update the number of samples Tt+1 ( pt ) 18 : end for We further develop an improvement BLUCB-Parallel in high confidence regime , whose sample complexity drops the dependence on unnecessary base arms for small enough δ . Both algorithms achieve the optimal sample complexity for a family of instances ( within a logarithmic factor ) . 3.1 Algorithm BLUCB with Base-arm-gap Dependent Results Algorithm 1 illustrates the proposed algorithm BLUCB for CPE-B in the FC setting . Here S ( Mt ) denotes the set of all supersets of super arm Mt ( Line 8 ) . Since the bottleneck reward function is monotonically decreasing , for any M ′ ∈ S ( Mt ) , we have MinW ( M ′ , w ) ≤ MinW ( Mt , w ) . Hence , to verify the optimality of Mt , we only need to compare Mt against super arms inM\S ( Mt ) , and this property will also be used in the later algorithm BLUCB-Parallel . BLUCB is allowed to access an efficient bottleneck maximization oracle MaxOracle ( F , v ) , which returns an optimal super arm from F with respect to v , i.e. , MaxOracle ( F , v ) ∈ argmaxM∈F MinW ( M , v ) . For F =M ( Line 7 ) , such an efficient oracle exists for many decision classes , such as the bottleneck shortest path [ 27 ] , bottleneck bipartite matching [ 28 ] and minimum bottleneck spanning tree [ 8 ] algorithms . For F =M\ S ( Mt ) ( Line 8 ) , we can also efficiently find the best super arm ( excluding the supersets of Mt ) by repeatedly removing each base arm in Mt and calling the basic maximization oracle , and then selecting the one with the maximum bottleneck value . We describe the procedure of BLUCB as follows : at each timestep t , we calculate the lower and upper confidence bounds of base arm rewards , denoted by wt and w̄t , respectively . Then , we call MaxOracle to find the super arm Mt with the maximum pessimistic bottleneck value fromM using wt ( Line 7 ) , and the super arm M̃t with the maximum optimistic bottleneck value fromM\ S ( Mt ) using w̄t ( Line 8 ) . Mt and M̃t are two critical super arms that determine when the algorithm should stop or not . If the pessimistic bottleneck value of Mt is higher than the optimistic bottleneck value of M̃t ( Line 9 ) , we can determine that Mt has the higher bottleneck value than any other super arm with high confidence , and then the algorithm can stop and output Mt . Otherwise , we select two base arms ct and dt with the minimum lower reward confidence bounds in Mt and M̃t respectively , and play the one with the larger confidence radius ( Lines 12-14 ) . Bottleneck-adaptive sample strategy . The “ select-minimum ” sample strategy in Lines 12-14 comes from an insight for the bottleneck problem : to determine that Mt has a higher bottleneck value than M̃t , it suffices to find a base arm from M̃t which is worse than any base arm ( the bottleneck base arm ) in Mt . To achieve this , base arms ct and dt , which have the most potential to be the bottlenecks of Mt and M̃t , are the most necessary ones to be sampled . This bottleneck-adaptive sample strategy is crucial for BLUCB to achieve the tight base-arm-gap dependent sample complexity . In contrast , the sample strategy of prior CPE algorithms [ 11 , 12 , 16 ] treats all base arms in critical super arms ( Mt and M̃t ) equally and does a uniform choice . If one naively adapts those algorithms with the current reward function MinW ( M , w ) , a loose super-arm-gap dependent sample complexity is incurred . To formally state the sample complexity of BLUCB , we introduce some notation and gap definition . Let N = { e | e /∈ M∗ , w ( e ) < OPT } and Ñ = { e | e /∈ M∗ , w ( e ) ≥ OPT } , which stand for the necessary and unnecessary base arms contained in the sub-optimal super arms , respectively . We define the reward gap for the FC setting as Algorithm 2 BLUCB-Parallel , an improved algorithm for the FC setting under small δ 1 : Input : δ ∈ ( 0 , 0.01 ) and sub-algorithm BLUCB-Verify . 2 : For k = 0 , 1 , . . . , let BLUCB-Verifyk be the sub-algorithm BLUCB-Verify with δk = δ 2k+1 3 : for t = 1 , 2 , . . . do 4 : for each k = 0 , 1 , . . . such that t mod 2k = 0 do 5 : Start or resume BLUCB-Verifyk with one sample , and then suspend BLUCB-Verifyk 6 : if BLUCB-Verifyk returns an answer Mout , then return Mout 7 : end for 8 : end for Algorithm 3 BLUCB-Verify , sub-algorithm of BLUCB-Parallel 1 : Input : M , δV ∈ ( 0 , 0.01 ) and MaxOracle . 2 : κ← 0.01 3 : M̂∗ , B̂sub←BLUCB-Explore ( M , κ , MaxOracle ) 4 : Initialize : play each e ∈ [ n ] once , and update empirical means ŵn+1 and Tn+1 5 : for t = n+ 1 , n+ 2 , . . . do 6 : radt ( e ) ←R √ 2 ln ( 4nt 3 δV ) /Tt ( e ) , ∀e∈ [ n ] 7 : wt ( e ) ← ŵt ( e ) − radt ( e ) , ∀e ∈ [ n ] 8 : w̄t ( e ) ← ŵt ( e ) + radt ( e ) , ∀e ∈ [ n ] 9 : M̃t = MaxOracle ( M\ S ( M̂∗ ) , w̄t ) 10 : if MinW ( M̂∗ , wt ) ≥ MinW ( M̃t , w̄t ) then 11 : return M̂∗ 12 : end if 13 : ct ← argmine∈M̂∗ wt ( e ) 14 : Ft ← { e ∈ B̂sub : w̄t ( e ) > wt ( ct ) } 15 : pt ← argmaxe∈Ft∪ { ct } radt ( e ) 16 : Play pt , and observe the reward 17 : Update empirical means ŵt+1 ( pt ) 18 : Update the number of samples Tt+1 ( pt ) 19 : end for Definition 1 ( Fixed-confidence Gap ) . ∆Ce = w ( e ) −maxM 6=M∗ MinW ( M , w ) , if e ∈M∗ , ( a ) w ( e ) −maxM∈M : e∈M MinW ( M , w ) , if e ∈ Ñ , ( b ) OPT−maxM∈M : e∈M MinW ( M , w ) , if e ∈ N. Now we present the sample complexity upper bound of BLUCB . Theorem 1 ( Fixed-confidence Upper Bound ) . With probability at least 1 − δ , algorithm BLUCB ( Algorithm 1 ) for CPE-B in the FC setting returns the optimal super arm with sample complexity O ∑ e∈ [ n ] R2 ( ∆Ce ) 2 ln ∑ e∈ [ n ] R2n ( ∆Ce ) 2δ . Base-arm-gap dependent sample complexity . Owing to the bottleneck-adaptive sample strategy , the reward gap ∆Ce ( Definition 1 ( a ) ( b ) ) is just defined as the difference between some critical bottleneck value and w ( e ) itself , instead of the bottleneck gap between two super arms , and thus our result depends on the tight base-arm-level ( instead of super-arm-level ) gaps . For example , in Figure 1 , BLUCB only spends Õ ( ( 2∆2e2 , e1 + ∑ i=3,4,5,6 1 ∆2ei , e1 ) ln δ−1 ) samples , while a naive adaptation of prior CPE algorithms [ 11 , 12 , 16 ] with the bottleneck reward function will cause a loose super-armgap dependent result Õ ( ∑ ei , i∈ [ 6 ] 1 ∆2M∗ , Msub ln δ−1 ) . Regarding the optimality , Theorem 1 matches the lower bound ( presented in Section 4 ) for some family of instances ( up to a logarithmic factor ) . However , in general cases there still exists a gap on those needless base arms Ñ ( e3 , e5 , e6 in Figure 1 ) , which are not contained in the lower bound . Next , we show how to bridge this gap .

Summary: The authors are interested in the combinatorial pure exploration problem, in which one aims at identifying the best "super arm" (a subset of "base arms") among a decision class that can arise from a certain combinatorial structure. In this paper, they focus on the combinatorial pure exploration problem in the specific case where the reward of a super arm is the minimum of the rewards of the base arms it contains. This very specific function -- called the bottleneck reward function -- raises very specific challenges when solving the combinatorial pure exploration problem.  The authors explain that none of the existing work on pure exploration or on multi armed bandits give good guarantees in the specific case of the bottleneck function, and propose tailor-made algorithms that exploit the specificities of the said function to reach optimal guarantees, both in the fixed-confidence setting and in the fixed-budget setting.