Thompson Sampling for (Combinatorial) Pure Exploration

1 INTRODUCTION . Pure exploration is an important task in online learning , and it tries to find out the target arm as fast as possible . In pure exploration of classic multi-armed bandit ( MAB ) ( Audibert et al. , 2010 ) , there are totally m arms , and each arm i is associated with a probability distribution Di with mean µi . Once arm i is pulled , it returns an observation ri , which is drawn independently from distributionDi by the environment . At each time step t , the learning policy π either chooses an arm i ( t ) to pull , or chooses to output an arm a ( t ) . The goal of the learning policy is to pull arms properly , such that with an error probability at most δ , its output arm a ( t ) is the optimal arm ( i.e. , a ( t ) = arg maxi∈ [ m ] µi ) with complexity ( the total number of observations ) as low as possible . Pure exploration is widely adopted in real applications . For example , in the selling procedure of cosmetics , there is always a testing phase before the commercialization phase ( Audibert et al. , 2010 ) . The goal of the testing phase is to help to maximize the cumulative reward collected in the commercialization phase . Therefore , instead of regret minimization ( Berry & Fristedt , 1985 ; Auer et al. , 2002 ) , the testing phase only needs to do exploration ( e.g. , to investigate which product is the most popular one ) , and wants to find out the target with both correctness guarantee and low cost . In real world , sometimes the system focuses on the best action under a specific combinatorial structure , instead of the best single arm ( Chen et al. , 2014 ) . For example , a network routing system needs to search for the path with minimum delay between the source and the destination . Since there can be an exponential number of paths , the cost of exploring them separately is unacceptable . Therefore , people choose to find out the best path by exploring the single edges , and this is a pure exploration problem instance in combinatorial multi-armed bandits ( CMAB ) . In this setting , we still pull single arms ( base arms ) at each time step , and there is a super arm set I ⊆ 2 [ m ] . The expected reward of a super arm S ∈ I is ∑ i∈S µi , i.e. , the sum of the expected rewards of its contained base arms . And the goal of the player is to find out the optimal super arm with an error probability at most δ . Most of the existing solutions for pure exploration follow the UCB approach ( Audibert et al. , 2010 ; Kalyanakrishnan et al. , 2012 ; Chen et al. , 2014 ; Kaufmann & Kalyanakrishnan , 2013 ) . They com- pute the confidence bounds for all the arms , and claim that one arm is optimal only if its lower confidence bound is larger than the upper confidence bounds of all the other arms . Though this approach is asymptotically optimal in pure exploration of classic MAB problems ( Kalyanakrishnan et al. , 2012 ; Kaufmann & Kalyanakrishnan , 2013 ) , it faces some challenges in the CMAB case ( Chen et al. , 2014 ) . In the UCB approach , for a super arm S , the algorithm usually uses the sum of upper confidence bounds of all its contained base arms as its upper confidence bound . This means that the gap between the empirical mean of super arm S and the used upper confidence bound of S is about Õ ( ∑ i∈S √ 1/Ni ) ( here Ni is the number of observations on base arm i ) . However , since the observations of different base arms are independent , the standard deviation of the empirical mean of super arm S is Õ ( √∑ i∈S 1/Ni ) , which is much smaller . This means that the used upper confidence bound is much larger than the real one . Combes et al . ( 2015 ) deal with this problem by computing the upper confidence bounds for all the super arms independently , which leads to an exponential time cost and can be hard to implement . In fact , existing efficient UCB-based solutions either suffer from a higher complexity bound ( Chen et al. , 2014 ; Gabillon et al. , 2016 ) , or need further assumptions on the combinatorial structure to achieve an optimal complexity bound ( Chen et al. , 2017 ) . Then a natural solution to deal with this challenge is to use random sample of arm i ( with mean to be its empirical mean and standard deviation to be Õ ( √ 1/Ni ) ) instead of its upper confidence bound to judge whether a super arm is optimal . If we let the random samples of different base arms be independent , then the gap between the empirical mean of super arm S and the sum of random samples within S is Õ ( √∑ i∈S 1/Ni ) , which has the same order as the gap between the empirical mean of super arm S and its real mean . Therefore , using independent random samples can behave better than using confidence bounds , and this is the key idea of Thompson Sampling ( TS ) ( Thompson , 1933 ; Kaufmann et al. , 2012 ; Agrawal & Goyal , 2013 ) . In fact , many prior works show that TS-based algorithms have smaller cumulative regret than UCB-based algorithms in regret minimization of CMAB model ( Wang & Chen , 2018 ; Perrault et al. , 2020 ) . However , existing researches only study using TS approaches under the Bayesian setting of pure exploration ( Li et al. , 2021 ) , and there still lack results of adapting TS to the frequentist setting . In this paper , we attempt to fill up this gap , and study using TS in pure exploration under the frequentist setting for both MAB and CMAB instances by adapting the idea of TS in the Bayesian setting ( Li et al. , 2021 ) . We emphasize that it is non-trivial to design ( and analyze ) a TS-based algorithm under the frequentist setting . The first challenge is that there is a lot more uncertainty in the random samples than the confidence bounds . In UCB-based algorithms , the upper confidence bound of the optimal arm is larger than its expected reward with high probability . Thus , the arm with the largest upper confidence bound is either an insufficiently learned sub-optimal arm ( i.e. , the number of its observations is not enough to make sure that it is a sub-optimal arm ) or the optimal arm , which means that the number of pulls on sufficiently learned sub-optimal arms is limited . However , for TS-based algorithms , there is always a constant probability that the random sample of the optimal arm is smaller than its expected reward . In this case , the arm with the largest random sample may be a sufficiently learned sub-optimal arm . Therefore , if we still choose to pull the arm with the largest random sample ( similar as UCB-based policies that pull the arm with the largest upper confidence bound ) , then the mechanism of the TS-based policy should be designed carefully to make sure that we can still obtain an upper bound for the number of pulls on sufficiently learned sub-optimal arms . Another challenge is that TS is an algorithm that follows the Bayesian framework and loses many good properties in the frequentist setting . In the Bayesian setting , at each time step t , the parameters of the game follow a posterior distribution Pt , and the random samples are drawn from Pt independently as well . Therefore , using the random samples to output the optimal arm can explicitly ensure the correctness of the TS-based algorithm . However , in the frequentist setting , the parameters of the game are fixed but unknown , and they have no such correlations with the random samples . Because of this , if we still use the random samples to output the optimal arm , then the distributions of random samples in the TS-based algorithm need to be chosen carefully to make sure that it still has correctness guarantee in the frequentist setting . Besides , the analysis of TS-based algorithms in pure exploration is also very different with regret minimization . In regret minimization , at each time step , we only need to draw one sample for each base arm . However , in pure exploration , one set of samples is not enough at each time step , since the algorithm needs to i ) check whether there is an arm that is the optimal one with high probability ; ii ) look for an arm that needs exploration , and none of these two goals can be achieved by only one set of samples . Therefore , we must draw several sets of samples to make decisions , and this may fail existing analysis in regret minimization . For example , the analysis in ( Agrawal & Goyal , 2013 ) obtains a lower bound for the probability that the optimal arm has the highest sample in one sample set . However , when we use several sets of samples to make decisions , we can notice the optimal arm only if it has the highest sample in most of these sample sets , which is much harder than it has the highest sample in only one sample set when its observations are bad ( e.g. , its empirical mean equals to its real expected reward minus the confidence radius ) . In this paper , we solve the above challenges , and design a TS-based verification framework TSVerify for ( combinatorial ) pure exploration under the frequentist setting . TS-Verify takes a target ( super ) arm as input , and aims to verify that this target ( super ) arm is optimal with error constraint δ . At each time step t , TS-Verify first draws independent random samples θki ( t ) for all the ( base ) arms i ∈ [ m ] and k = 1 , 2 , · · · , M ( i.e. , totallyM independent samples for each arm ) . Then it tries to find out the M best ( super ) arms under sample sets θk ( t ) = [ θk1 ( t ) , · · · , θkm ( t ) ] for k = 1 , 2 , · · · , M . If in most of these sample sets , the target ( super ) arm is the best one , then the algorithm will output that the target ( super ) arm is indeed optimal . Otherwise , the algorithm will choose to pull a ( base ) arm from the exploration set , which contains the target ( super ) arm and all the ( super ) arms that have been optimal in at least one sample set at this time step . In the rest of this paper , we mainly focus on the behaviour ( i.e. , correctness guarantee and complexity upper bound ) of this verification algorithm , which is almost the same as a complete pure exploration algorithm after we adapt the explore-then-verify framework ( Karnin , 2016 ; Chen et al. , 2017 ) . In the general CMAB setting , we show that TS-Verify is near optimal and behaves better than existing UCB-based algorithms ( Chen et al. , 2014 ) . The optimal algorithm ( Chen et al. , 2017 ) requires the combinatorial structure to satisfy some specific properties , which is less general than our results . As for the MAB case , we show that TS-Verify is asymptotically optimal , i.e. , it has a comparable complexity as existing optimal algorithms ( Kalyanakrishnan et al. , 2012 ; Kaufmann & Kalyanakrishnan , 2013 ) when δ → 0 . All these results indicate that out algorithm framework is efficient and general in dealing with pure exploration problems . To the best of our knowledge , this is the first result of using TS-based algorithms in pure exploration under the frequentist setting .

Summary: This paper designs the first TS-based algorithm to solve the pure exploration problem for both multi-armed bandit (MAB) and combinatorial MAB (CMAB) problems. It mainly focuses on the verification problem where a target arm is given and the algorithm needs to determine whether it is the optimal one.   Benefiting from independent samples of the TS algorithm, the TS-based verification framework achieves lower complexity than the UCB-based one.