Combinatorial Pure Exploration with Bottleneck Reward Function

1 Introduction . The Multi-Armed Bandit ( MAB ) problem [ 25 , 30 , 4 , 2 ] is a classic model to solve the explorationexploitation trade-off in online decision making . Pure exploration [ 3 , 21 , 7 , 26 ] is an important variant of the MAB problem , which aims to identify the best arm under a given confidence or a given sample budget . There are various works studying pure exploration , such as top-k arm identification [ 17 , 21 , 7 , 24 ] , top-k arm under matriod constraints [ 9 ] and multi-bandit best arm identification [ 18 , 7 ] . The Combinatorial Pure Exploration ( CPE ) framework , firstly proposed by Chen et al . [ 11 ] , encompasses a rich class of pure exploration problems [ 3 , 21 , 9 ] . In CPE , there are a set of base arms , each associated with an unknown reward distribution . A subset of base arms is called a super arm , which follows a certain combinatorial structure . At each timestep , a learner plays a base arm and observes a random reward sampled from its distribution , with the objective to identify the optimal super arm 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . with the maximum expected reward . While Chen et al . [ 11 ] provide this general CPE framework , their algorithms and analytical techniques only work under the linear reward function and can not be applied to other nonlinear reward cases.1 However , in many real-world scenarios , the expected reward function is not necessarily linear . One of the common and important cases is the bottleneck reward function , i.e. , the expected reward of a super arm is the minimum expected reward of the base arms contained in it . For example , in communication networks [ 5 ] , the transmission speed of a path is usually determined by the link with the lowest rate , and a learner samples the links in order to find the optimal transmission path which maximizes its bottleneck link rate . In traffic scheduling [ 31 ] , a scheduling system collects the information of road segments in order to plan an efficient route which optimizes its most congested ( bottleneck ) road segment . In neural architecture search [ 32 ] , the overall efficiency of a network architecture is usually constrained by its worst module , and an agent samples the available modules with the objective to identify the best network architecture in combinatorial search space . In this paper , we study the Combinatorial Pure Exploration with the Bottleneck reward function ( CPE-B ) which aims to identify the optimal super arm with the maximum bottleneck value by querying the base arm rewards , where the bottleneck value of a super arm is defined as the minimum expected reward of its containing base arms . We consider two popular settings in pure exploration , i.e , fixed-confidence ( FC ) , where given confidence parameter δ , the learner aims to identify the optimal super arm with probability 1− δ and minimize the number of used samples ( sample complexity ) , and fixed-budget ( FB ) , where the learner needs to use a given sample budget to find the optimal super arm and minimize the error probability . Challenges of CPE-B . Compared to prior CPE works [ 11 , 10 , 20 ] , our CPE-B aims at utilizing the bottleneck property to save samples and achieve the statistical optimality . It faces with two unique challenges , i.e. , how to ( i ) achieve the tight base-arm-gap dependent sample complexity and ( ii ) avoid the dependence on unnecessary base arms in the results , while running in polynomial time . We use a simple example in Figure 1 to illustrate our challenges . In Figure 1 , there are six edges ( base arms ) and three s-t paths ( super arms ) , and the base arm reward w ( ei ) , base arm gap ∆ei , ej and super arm gap ∆M∗ , Msub are as shown in the figure . In order to identify the optimal path , all we need is to pull e1 , e2 , e4 to determine that e1 is worse than e2 and e4 , and e3 , e5 , e6 are useless for revealing the sub-optimality of M1 and M2 . In this case , the optimal sample complexity should be O ( ( 2∆2e2 , e1 + 1∆2e4 , e1 ) ln δ−1 ) , which depends on the tight base arm gaps and only includes the critical base arms ( e1 , e2 , e4 ) . However , if one naively adapts existing CPE algorithms [ 11 , 12 , 16 ] to work with bottleneck reward function , an inferior sample complexity of O ( ∑ ei , i∈ [ 6 ] 1 ∆2M∗ , Msub ln δ−1 ) is incurred , which depends on the loose super arm gaps and contains a summation over all base arms ( including the unnecessary e3 , e5 , e6 ) . Hence , our challenge falls on how to achieve such efficient sampling in an online environment , where we do not know which are critical base arms e1 , e2 , e4 but want to gather just enough information to identify the optimal super arm . We remark that , none of existing CPE studies can be applied to solve the unique challenges of CPE-B , and thus we develop brand-new techniques to handle them and attain the optimal results ( up to a logarithmic factor ) . Contributions . For CPE-B in the FC setting , ( i ) we first develop a novel algorithm BLUCB , which employs a bottleneck-adaptive sample strategy and achieves the tight base-arm-gap dependent sample complexity . ( ii ) We further propose an improved algorithm BLUCB-Parallel in high confidence regime , which adopts an efficient “ bottleneck-searching ” offline procedure and a novel “ check-nearbottleneck ” stopping condition . The sample complexity of BLUCB-Parallel drops the dependence on unnecessary base arms and achieves the optimality ( within a logarithmic factor ) under small enough δ . ( iii ) A matching sample complexity lower bound for the FC setting is also provided , which demonstrates the optimality of our algorithms . For the FB setting , ( iv ) we propose a novel algorithm BSAR with a special acceptance scheme for the bottleneck identification task . BSAR achieves the state-of-the-art error probability and is the first to run efficiently on fixed-budget path instances , 1The algorithmic designs and analytical tools ( e.g. , symmetric difference and exchange set ) in [ 11 ] all rely on the linear property and can not be applied to nonlinear reward cases , e.g , the bottleneck reward problem . compared to existing CPE algorithms . All our proposed algorithms run in polynomial time.2 The experimental results demonstrate that our algorithms significantly outperform the baselines . Due to space limit , we defer all the proofs to the supplementary material . 1.1 Related Work . In the following we briefly review the related work in the CPE literature . Chen et al . [ 11 ] firstly propose the CPE model and only consider the linear reward function ( CPE-L ) , and their results for CPE-L are further improved by [ 19 , 10 ] . Huang et al . [ 20 ] investigate the continuous and separable reward functions ( CPE-CS ) , but their algorithm only runs efficiently on simple cardinality constraint instances . All these works consider directly sampling base arms and getting their feedback . There are also several CPE studies which consider other forms of sampling and feedback . Chen et al . [ 12 ] propose the CPE for dueling bandit setting , where at each timestep the learner pulls a duel between two base arms and observes their comparison outcome . Kuroki et al . [ 23 ] study an online densest subgraph problem , where the decision is a subgraph and the feedback is the reward sum of the edges in the chosen subgraph ( i.e. , full-bandit feedback ) . Du et al . [ 16 ] investigate CPE with the full-bandit or partial linear feedback . All of the above studies consider the pure exploration setting , while in combinatorial bandits there are other works [ 13 , 15 , 14 ] studying the regret minimization setting ( CMAB ) . In CMAB , the learner plays a super arm and observes the rewards from all base arms contained in it , with goal of minimizing the regret , which is significantly different from our setting . Note that none of the above studies covers our CPE-B problem or can be adapted to solve the unique challenges of CPE-B , and thus CPE-B demands a new investigation . 2 Problem Formulation . In this section , we give the formal formulation of CPE-B . In this problem , a learner is given n base arms numbered by 1 , 2 , . . . , n. Each base arm e ∈ [ n ] is associated with an unknown reward distribution with the mean of w ( e ) and an R-sub-Gaussian tail , which is a standard assumption in bandits [ 1 , 11 , 26 , 29 ] . Let w = ( w ( 1 ) , . . . , w ( n ) ) > be the expected reward vector of base arms . The learner is also given a decision class M ⊆ 2 [ n ] , which is a collection of super arms ( subsets of base arms ) and generated from a certain combinatorial structure , such as s-t paths , maximum cardinality matchings , and spanning trees . For each super arm M ∈ M , we define its expected reward ( also called bottleneck value ) as MinW ( M , w ) = mine∈M w ( e ) ,3 i.e. , the minimum expected reward of its constituent base arms , which is so called bottleneck reward function . Let M∗ = argmaxM∈M MinW ( M , w ) be the optimal super arm with the maximum bottleneck value , and OPT = MinW ( M∗ , w ) be the optimal value . Following the pure exploration literature [ 17 , 11 , 10 , 12 ] , we assume that M∗ is unique , and this assumption can be removed in our extension to the PAC learning setting ( see the supplementary material ) . At each timestep , the learner plays ( or samples ) a base arm pt ∈ [ n ] and observes a random reward sampled from its reward distribution , where the sample is independent among different timestep t. The learner ’ s objective is to identify the optimal super arm M∗ fromM . For this identification task , we study two common metrics in pure exploration [ 21 , 7 , 26 , 10 ] , i.e. , fixed-confidence ( FC ) and fixed-budget ( FB ) settings . In the FC setting , given a confidence parameter δ ∈ ( 0 , 1 ) , the learner needs to identify M∗ with probability at least 1− δ and minimize the sample complexity , i.e. , the number of samples used . In the FB setting , the learner is given a fixed sample budget T , and needs to identify M∗ within T samples and minimize the error probability , i.e. , the probability of returning a wrong answer . 3 Algorithms for the Fixed-Confidence Setting . In this section , we first propose a simple algorithm BLUCB for the FC setting , which adopts a novel bottleneck-adaptive sample strategy to obtain the tight base-arm-gap dependent sample complexity . 2Here “ polynomial time ” refers to polynomial time in the number of base arms n ( which is equal to the number of edges E in our considered instances such as s-t paths , matchings and spanning trees ) . 3In general , the second input of function MinW can be any vector : for any M ∈ M and v ∈ Rn , MinW ( M , v ) = mine∈M v ( e ) . Algorithm 1 BLUCB , algorithm for CPE-B in the FC setting 1 : Input : M , δ ∈ ( 0 , 1 ) and MaxOracle . 2 : Initialize : play each e ∈ [ n ] once , and update empirical means ŵn+1 and Tn+1 3 : for t = n+ 1 , n+ 2 , . . . do 4 : radt ( e ) ← √ 2 ln ( 4nt 3 δ ) /Tt ( e ) , ∀e ∈ [ n ] 5 : wt ( e ) ← ŵt ( e ) − radt ( e ) , ∀e ∈ [ n ] 6 : w̄t ( e ) ← ŵt ( e ) + radt ( e ) , ∀e ∈ [ n ] 7 : Mt ← MaxOracle ( M , wt ) 8 : M̃t ← MaxOracle ( M\ S ( Mt ) , w̄t ) 9 : if MinW ( Mt , wt ) ≥ MinW ( M̃t , w̄t ) then 10 : return Mt 11 : end if 12 : ct ← argmine∈Mt wt ( e ) 13 : dt ← argmine∈M̃t wt ( e ) 14 : pt ← argmaxe∈ { ct , dt } radt ( e ) 15 : Play pt , and observe the reward 16 : Update empirical means ŵt+1 ( pt ) 17 : Update the number of samples Tt+1 ( pt ) 18 : end for We further develop an improvement BLUCB-Parallel in high confidence regime , whose sample complexity drops the dependence on unnecessary base arms for small enough δ . Both algorithms achieve the optimal sample complexity for a family of instances ( within a logarithmic factor ) . 3.1 Algorithm BLUCB with Base-arm-gap Dependent Results Algorithm 1 illustrates the proposed algorithm BLUCB for CPE-B in the FC setting . Here S ( Mt ) denotes the set of all supersets of super arm Mt ( Line 8 ) . Since the bottleneck reward function is monotonically decreasing , for any M ′ ∈ S ( Mt ) , we have MinW ( M ′ , w ) ≤ MinW ( Mt , w ) . Hence , to verify the optimality of Mt , we only need to compare Mt against super arms inM\S ( Mt ) , and this property will also be used in the later algorithm BLUCB-Parallel . BLUCB is allowed to access an efficient bottleneck maximization oracle MaxOracle ( F , v ) , which returns an optimal super arm from F with respect to v , i.e. , MaxOracle ( F , v ) ∈ argmaxM∈F MinW ( M , v ) . For F =M ( Line 7 ) , such an efficient oracle exists for many decision classes , such as the bottleneck shortest path [ 27 ] , bottleneck bipartite matching [ 28 ] and minimum bottleneck spanning tree [ 8 ] algorithms . For F =M\ S ( Mt ) ( Line 8 ) , we can also efficiently find the best super arm ( excluding the supersets of Mt ) by repeatedly removing each base arm in Mt and calling the basic maximization oracle , and then selecting the one with the maximum bottleneck value . We describe the procedure of BLUCB as follows : at each timestep t , we calculate the lower and upper confidence bounds of base arm rewards , denoted by wt and w̄t , respectively . Then , we call MaxOracle to find the super arm Mt with the maximum pessimistic bottleneck value fromM using wt ( Line 7 ) , and the super arm M̃t with the maximum optimistic bottleneck value fromM\ S ( Mt ) using w̄t ( Line 8 ) . Mt and M̃t are two critical super arms that determine when the algorithm should stop or not . If the pessimistic bottleneck value of Mt is higher than the optimistic bottleneck value of M̃t ( Line 9 ) , we can determine that Mt has the higher bottleneck value than any other super arm with high confidence , and then the algorithm can stop and output Mt . Otherwise , we select two base arms ct and dt with the minimum lower reward confidence bounds in Mt and M̃t respectively , and play the one with the larger confidence radius ( Lines 12-14 ) . Bottleneck-adaptive sample strategy . The “ select-minimum ” sample strategy in Lines 12-14 comes from an insight for the bottleneck problem : to determine that Mt has a higher bottleneck value than M̃t , it suffices to find a base arm from M̃t which is worse than any base arm ( the bottleneck base arm ) in Mt . To achieve this , base arms ct and dt , which have the most potential to be the bottlenecks of Mt and M̃t , are the most necessary ones to be sampled . This bottleneck-adaptive sample strategy is crucial for BLUCB to achieve the tight base-arm-gap dependent sample complexity . In contrast , the sample strategy of prior CPE algorithms [ 11 , 12 , 16 ] treats all base arms in critical super arms ( Mt and M̃t ) equally and does a uniform choice . If one naively adapts those algorithms with the current reward function MinW ( M , w ) , a loose super-arm-gap dependent sample complexity is incurred . To formally state the sample complexity of BLUCB , we introduce some notation and gap definition . Let N = { e | e /∈ M∗ , w ( e ) < OPT } and Ñ = { e | e /∈ M∗ , w ( e ) ≥ OPT } , which stand for the necessary and unnecessary base arms contained in the sub-optimal super arms , respectively . We define the reward gap for the FC setting as Algorithm 2 BLUCB-Parallel , an improved algorithm for the FC setting under small δ 1 : Input : δ ∈ ( 0 , 0.01 ) and sub-algorithm BLUCB-Verify . 2 : For k = 0 , 1 , . . . , let BLUCB-Verifyk be the sub-algorithm BLUCB-Verify with δk = δ 2k+1 3 : for t = 1 , 2 , . . . do 4 : for each k = 0 , 1 , . . . such that t mod 2k = 0 do 5 : Start or resume BLUCB-Verifyk with one sample , and then suspend BLUCB-Verifyk 6 : if BLUCB-Verifyk returns an answer Mout , then return Mout 7 : end for 8 : end for Algorithm 3 BLUCB-Verify , sub-algorithm of BLUCB-Parallel 1 : Input : M , δV ∈ ( 0 , 0.01 ) and MaxOracle . 2 : κ← 0.01 3 : M̂∗ , B̂sub←BLUCB-Explore ( M , κ , MaxOracle ) 4 : Initialize : play each e ∈ [ n ] once , and update empirical means ŵn+1 and Tn+1 5 : for t = n+ 1 , n+ 2 , . . . do 6 : radt ( e ) ←R √ 2 ln ( 4nt 3 δV ) /Tt ( e ) , ∀e∈ [ n ] 7 : wt ( e ) ← ŵt ( e ) − radt ( e ) , ∀e ∈ [ n ] 8 : w̄t ( e ) ← ŵt ( e ) + radt ( e ) , ∀e ∈ [ n ] 9 : M̃t = MaxOracle ( M\ S ( M̂∗ ) , w̄t ) 10 : if MinW ( M̂∗ , wt ) ≥ MinW ( M̃t , w̄t ) then 11 : return M̂∗ 12 : end if 13 : ct ← argmine∈M̂∗ wt ( e ) 14 : Ft ← { e ∈ B̂sub : w̄t ( e ) > wt ( ct ) } 15 : pt ← argmaxe∈Ft∪ { ct } radt ( e ) 16 : Play pt , and observe the reward 17 : Update empirical means ŵt+1 ( pt ) 18 : Update the number of samples Tt+1 ( pt ) 19 : end for Definition 1 ( Fixed-confidence Gap ) . ∆Ce = w ( e ) −maxM 6=M∗ MinW ( M , w ) , if e ∈M∗ , ( a ) w ( e ) −maxM∈M : e∈M MinW ( M , w ) , if e ∈ Ñ , ( b ) OPT−maxM∈M : e∈M MinW ( M , w ) , if e ∈ N. Now we present the sample complexity upper bound of BLUCB . Theorem 1 ( Fixed-confidence Upper Bound ) . With probability at least 1 − δ , algorithm BLUCB ( Algorithm 1 ) for CPE-B in the FC setting returns the optimal super arm with sample complexity O ∑ e∈ [ n ] R2 ( ∆Ce ) 2 ln ∑ e∈ [ n ] R2n ( ∆Ce ) 2δ . Base-arm-gap dependent sample complexity . Owing to the bottleneck-adaptive sample strategy , the reward gap ∆Ce ( Definition 1 ( a ) ( b ) ) is just defined as the difference between some critical bottleneck value and w ( e ) itself , instead of the bottleneck gap between two super arms , and thus our result depends on the tight base-arm-level ( instead of super-arm-level ) gaps . For example , in Figure 1 , BLUCB only spends Õ ( ( 2∆2e2 , e1 + ∑ i=3,4,5,6 1 ∆2ei , e1 ) ln δ−1 ) samples , while a naive adaptation of prior CPE algorithms [ 11 , 12 , 16 ] with the bottleneck reward function will cause a loose super-armgap dependent result Õ ( ∑ ei , i∈ [ 6 ] 1 ∆2M∗ , Msub ln δ−1 ) . Regarding the optimality , Theorem 1 matches the lower bound ( presented in Section 4 ) for some family of instances ( up to a logarithmic factor ) . However , in general cases there still exists a gap on those needless base arms Ñ ( e3 , e5 , e6 in Figure 1 ) , which are not contained in the lower bound . Next , we show how to bridge this gap .

Summary: The paper studies the extension of a combinatorial bandits problem studied by S. Chen et al, NeurIPS 2014. The extension here is to consider  (expected) reward as the minimum reward of the elements of a set rather than a linear combination as studied in Chen et al.   We are given a set of super-arms (which may overlap). The reward function associated with each super-arm is the minimum expected value of the reward associated with each of the base-arms in that super-arm. The problem is to find the best super-arm, the one with the maximal expected reward.   The paper provides algorithms, and upper and lower bounds on sample complexity for a given confidence bound, and error probability bound for a given budget.    The paper (including the supplemental material) is densely written with sparse discussion of the results. A table of notations would considerably help the reader.   Reported simulation results are limited.   Some of the motivating examples need better articulation (e.g. bottleneck path problem).