Thompson Sampling for (Combinatorial) Pure Exploration

1 INTRODUCTION . Pure exploration is an important task in online learning , and it tries to find out the target arm as fast as possible . In pure exploration of classic multi-armed bandit ( MAB ) ( Audibert et al. , 2010 ) , there are totally m arms , and each arm i is associated with a probability distribution Di with mean µi . Once arm i is pulled , it returns an observation ri , which is drawn independently from distributionDi by the environment . At each time step t , the learning policy π either chooses an arm i ( t ) to pull , or chooses to output an arm a ( t ) . The goal of the learning policy is to pull arms properly , such that with an error probability at most δ , its output arm a ( t ) is the optimal arm ( i.e. , a ( t ) = arg maxi∈ [ m ] µi ) with complexity ( the total number of observations ) as low as possible . Pure exploration is widely adopted in real applications . For example , in the selling procedure of cosmetics , there is always a testing phase before the commercialization phase ( Audibert et al. , 2010 ) . The goal of the testing phase is to help to maximize the cumulative reward collected in the commercialization phase . Therefore , instead of regret minimization ( Berry & Fristedt , 1985 ; Auer et al. , 2002 ) , the testing phase only needs to do exploration ( e.g. , to investigate which product is the most popular one ) , and wants to find out the target with both correctness guarantee and low cost . In real world , sometimes the system focuses on the best action under a specific combinatorial structure , instead of the best single arm ( Chen et al. , 2014 ) . For example , a network routing system needs to search for the path with minimum delay between the source and the destination . Since there can be an exponential number of paths , the cost of exploring them separately is unacceptable . Therefore , people choose to find out the best path by exploring the single edges , and this is a pure exploration problem instance in combinatorial multi-armed bandits ( CMAB ) . In this setting , we still pull single arms ( base arms ) at each time step , and there is a super arm set I ⊆ 2 [ m ] . The expected reward of a super arm S ∈ I is ∑ i∈S µi , i.e. , the sum of the expected rewards of its contained base arms . And the goal of the player is to find out the optimal super arm with an error probability at most δ . Most of the existing solutions for pure exploration follow the UCB approach ( Audibert et al. , 2010 ; Kalyanakrishnan et al. , 2012 ; Chen et al. , 2014 ; Kaufmann & Kalyanakrishnan , 2013 ) . They com- pute the confidence bounds for all the arms , and claim that one arm is optimal only if its lower confidence bound is larger than the upper confidence bounds of all the other arms . Though this approach is asymptotically optimal in pure exploration of classic MAB problems ( Kalyanakrishnan et al. , 2012 ; Kaufmann & Kalyanakrishnan , 2013 ) , it faces some challenges in the CMAB case ( Chen et al. , 2014 ) . In the UCB approach , for a super arm S , the algorithm usually uses the sum of upper confidence bounds of all its contained base arms as its upper confidence bound . This means that the gap between the empirical mean of super arm S and the used upper confidence bound of S is about Õ ( ∑ i∈S √ 1/Ni ) ( here Ni is the number of observations on base arm i ) . However , since the observations of different base arms are independent , the standard deviation of the empirical mean of super arm S is Õ ( √∑ i∈S 1/Ni ) , which is much smaller . This means that the used upper confidence bound is much larger than the real one . Combes et al . ( 2015 ) deal with this problem by computing the upper confidence bounds for all the super arms independently , which leads to an exponential time cost and can be hard to implement . In fact , existing efficient UCB-based solutions either suffer from a higher complexity bound ( Chen et al. , 2014 ; Gabillon et al. , 2016 ) , or need further assumptions on the combinatorial structure to achieve an optimal complexity bound ( Chen et al. , 2017 ) . Then a natural solution to deal with this challenge is to use random sample of arm i ( with mean to be its empirical mean and standard deviation to be Õ ( √ 1/Ni ) ) instead of its upper confidence bound to judge whether a super arm is optimal . If we let the random samples of different base arms be independent , then the gap between the empirical mean of super arm S and the sum of random samples within S is Õ ( √∑ i∈S 1/Ni ) , which has the same order as the gap between the empirical mean of super arm S and its real mean . Therefore , using independent random samples can behave better than using confidence bounds , and this is the key idea of Thompson Sampling ( TS ) ( Thompson , 1933 ; Kaufmann et al. , 2012 ; Agrawal & Goyal , 2013 ) . In fact , many prior works show that TS-based algorithms have smaller cumulative regret than UCB-based algorithms in regret minimization of CMAB model ( Wang & Chen , 2018 ; Perrault et al. , 2020 ) . However , existing researches only study using TS approaches under the Bayesian setting of pure exploration ( Li et al. , 2021 ) , and there still lack results of adapting TS to the frequentist setting . In this paper , we attempt to fill up this gap , and study using TS in pure exploration under the frequentist setting for both MAB and CMAB instances by adapting the idea of TS in the Bayesian setting ( Li et al. , 2021 ) . We emphasize that it is non-trivial to design ( and analyze ) a TS-based algorithm under the frequentist setting . The first challenge is that there is a lot more uncertainty in the random samples than the confidence bounds . In UCB-based algorithms , the upper confidence bound of the optimal arm is larger than its expected reward with high probability . Thus , the arm with the largest upper confidence bound is either an insufficiently learned sub-optimal arm ( i.e. , the number of its observations is not enough to make sure that it is a sub-optimal arm ) or the optimal arm , which means that the number of pulls on sufficiently learned sub-optimal arms is limited . However , for TS-based algorithms , there is always a constant probability that the random sample of the optimal arm is smaller than its expected reward . In this case , the arm with the largest random sample may be a sufficiently learned sub-optimal arm . Therefore , if we still choose to pull the arm with the largest random sample ( similar as UCB-based policies that pull the arm with the largest upper confidence bound ) , then the mechanism of the TS-based policy should be designed carefully to make sure that we can still obtain an upper bound for the number of pulls on sufficiently learned sub-optimal arms . Another challenge is that TS is an algorithm that follows the Bayesian framework and loses many good properties in the frequentist setting . In the Bayesian setting , at each time step t , the parameters of the game follow a posterior distribution Pt , and the random samples are drawn from Pt independently as well . Therefore , using the random samples to output the optimal arm can explicitly ensure the correctness of the TS-based algorithm . However , in the frequentist setting , the parameters of the game are fixed but unknown , and they have no such correlations with the random samples . Because of this , if we still use the random samples to output the optimal arm , then the distributions of random samples in the TS-based algorithm need to be chosen carefully to make sure that it still has correctness guarantee in the frequentist setting . Besides , the analysis of TS-based algorithms in pure exploration is also very different with regret minimization . In regret minimization , at each time step , we only need to draw one sample for each base arm . However , in pure exploration , one set of samples is not enough at each time step , since the algorithm needs to i ) check whether there is an arm that is the optimal one with high probability ; ii ) look for an arm that needs exploration , and none of these two goals can be achieved by only one set of samples . Therefore , we must draw several sets of samples to make decisions , and this may fail existing analysis in regret minimization . For example , the analysis in ( Agrawal & Goyal , 2013 ) obtains a lower bound for the probability that the optimal arm has the highest sample in one sample set . However , when we use several sets of samples to make decisions , we can notice the optimal arm only if it has the highest sample in most of these sample sets , which is much harder than it has the highest sample in only one sample set when its observations are bad ( e.g. , its empirical mean equals to its real expected reward minus the confidence radius ) . In this paper , we solve the above challenges , and design a TS-based verification framework TSVerify for ( combinatorial ) pure exploration under the frequentist setting . TS-Verify takes a target ( super ) arm as input , and aims to verify that this target ( super ) arm is optimal with error constraint δ . At each time step t , TS-Verify first draws independent random samples θki ( t ) for all the ( base ) arms i ∈ [ m ] and k = 1 , 2 , · · · , M ( i.e. , totallyM independent samples for each arm ) . Then it tries to find out the M best ( super ) arms under sample sets θk ( t ) = [ θk1 ( t ) , · · · , θkm ( t ) ] for k = 1 , 2 , · · · , M . If in most of these sample sets , the target ( super ) arm is the best one , then the algorithm will output that the target ( super ) arm is indeed optimal . Otherwise , the algorithm will choose to pull a ( base ) arm from the exploration set , which contains the target ( super ) arm and all the ( super ) arms that have been optimal in at least one sample set at this time step . In the rest of this paper , we mainly focus on the behaviour ( i.e. , correctness guarantee and complexity upper bound ) of this verification algorithm , which is almost the same as a complete pure exploration algorithm after we adapt the explore-then-verify framework ( Karnin , 2016 ; Chen et al. , 2017 ) . In the general CMAB setting , we show that TS-Verify is near optimal and behaves better than existing UCB-based algorithms ( Chen et al. , 2014 ) . The optimal algorithm ( Chen et al. , 2017 ) requires the combinatorial structure to satisfy some specific properties , which is less general than our results . As for the MAB case , we show that TS-Verify is asymptotically optimal , i.e. , it has a comparable complexity as existing optimal algorithms ( Kalyanakrishnan et al. , 2012 ; Kaufmann & Kalyanakrishnan , 2013 ) when δ → 0 . All these results indicate that out algorithm framework is efficient and general in dealing with pure exploration problems . To the best of our knowledge , this is the first result of using TS-based algorithms in pure exploration under the frequentist setting .

Summary: Pure exploration plays an important role in online learning. Existing work mainly focuses on the UCB faces some challenges when looking for the best arm set under some specific combinatorial structures, so authors explore the idea of Thompson Sampling (TS) that uses independent random samples instead of the upper confidence bounds to make decisions, and design the first TS-based algorithm framework TS-Verify. In TS-Verify, the sum of independent random samples within arm set S will not exceed the exact upper confidence bound of S with high probability. As for pure exploration of classic multi-armed bandit, they show that TS-Verify achieves an asymptotically optimal complexity upper bound.  The TS-based verification framework TS- Verify takes a target (super) arm as input, and aims to verify that this target (super) arm is optimal with error constraint . At each time step, TS-Verify first draws independent random samples  for all the (base) arms. Then it tries to find out the M best (super) arms under sample sets.  If in most of these sample sets, the target (super) arm is the best one, then the algorithm will output that the target (super) arm is indeed optimal. Otherwise, the algorithm will choose to pull a (base) arm from the exploration set, which contains the target (super) arm and all the (super) arms that have been optimal in at least one sample set at this time step.