The Hardness Analysis of Thompson Sampling for Combinatorial Semi-bandits with Greedy Oracle

1 Introduction . Stochastic multi-armed bandit ( MAB ) problem [ 20 , 5 , 3 ] is a classical online learning framework . It has been extensively studied in the literature and has a wide range of applications [ 13 , 12 , 20 ] . The problem is modeled by a T -round game between the learning agent and the environment . The environment contains an arm set and each arm is associated with a reward distribution . At each round t , the agent first selects an arm , while the environment generates a random reward for each arm from its reward distribution . The agent then obtains the reward of the selected arm . The objective of the agent is to accumulate as many expected rewards over T rounds , or equivalent to minimizing the cumulative expected regret , which is defined as the cumulative difference between the expected reward of the optimal arm and the selected arms over T rounds . To achieve this long-horizon goal , the learning agent has to face the dilemma of exploration and exploitation in each round . The former aims to try arms that have not been observed enough times to get a potentially higher reward , the latter focuses on the arm with the best observed performance so far to maintain a high profit . How to balance the tradeoff between exploration and exploitation is the main focus of the MAB algorithms . One of the most popular bandit algorithms is the upper confidence bound ( UCB ) algorithm [ 5 ] . The algorithm aims to construct confidence sets for unknown expected rewards and selects arms according to their highest upper confidence bounds . The UCB-type algorithms have been widely studied and provided theoretical guarantees with regret upper bound of order O ( log T/ ) , where is the ⇤Corresponding author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . minimum gap between the expected reward of the optimal arm and any suboptimal arms . Thompson sampling ( TS ) [ 3 , 26 ] is another popular method to solve MAB problems . It is a randomized algorithm based on the Bayesian idea , which maintains an iteratively updated posterior distribution for each arm and chooses arms according to their probabilities of being the best one . The TS algorithm was introduced in the 1930s [ 28 ] , but its theoretical analysis is open until recent years [ 15 , 2 ] . It was proven that the regret upper bound of the TS algorithm is of the same order of O ( log T/ ) in MAB problems [ 3 ] . Benefited from the advantages of easier implementation and better empirical performance compared to UCB , the TS-type algorithms attract more attentions in recent years . Despite its importance , the MAB framework may fail to model many real applications since the agent ’ s action is usually not a single arm but a combination of several arms . This motivates the study on the combinatorial MAB ( CMAB ) problem [ 11 , 9 , 17 , 31 , 7 , 29 , 30 , 24 ] . In the CMAB framework , the agent selects a combination of base arms as an action to play in each round . All outcomes of these selected arms are then revealed to the agent , which is called semi-bandit feedback and is widely studied in the literature [ 7 , 29 , 32 , 30 , 14 , 24 ] . Such CMAB framework can cover many real application scenarios including probabilistic maximum coverage ( PMC ) [ 6 ] , online influence maximization ( OIM ) [ 29 ] , multiple-play MAB ( MP-MAB ) [ 17 ] and minimum spanning tree ( MST ) . TS-type algorithms have recently attracted a lot of interest in CMAB problems [ 17 , 30 , 14 , 24 ] . All of these works need an exact oracle to provide the optimal solution with sampled parameters as input in each round . However , such oracles are usually not feasible since many offline combinatorial optimization problems , such as the offline problem of PMC and OIM [ 16 ] , are NP-hard and only approximation oracles are available . With an example [ 30 ] illustrating the non-convergent regret of TS with an artificial approximation oracle designed for a specific problem instance , whether TS can work well in CMAB problems with common approximation oracles is still an open problem . One of the most common oracles for offline combinatorial optimization problems with a theoretical guarantee is the greedy algorithm . It sequentially finds the current optimal arm , or the optimal collection of multiple arms according to the structural correlations , to maximize the current total expected reward . When the termination condition is met , it will return the set of all arms found in previous steps as the solution . The termination condition is usually formulated by a constraint on the number of steps . For example , in the PMC , OIM , and MP-MAB problems , such a process is limited to continue K steps . The greedy algorithm can provide approximate solutions for offline problems of PMC [ 8 ] and OIM [ 16 ] , and exact optimal solutions for offline problems of MP-MAB [ 17 ] and MST [ 18 ] . In general , as long as the expected reward in a problem satisfies the monotonicity and submodularity on the action set , the greedy algorithm serves as an offline oracle to provide an approximate solution [ 23 ] . In this paper , we first formulate the CMAB problems with greedy oracle , which is general enough and covers PMC , MP-MAB , and many other problems . In this framework , the objective of the learning agent is to minimize the cumulative greedy regret , defined as the cumulative difference between the expected reward of the selected action and that of the greedy ’ s solution in the real environment . Focusing on a specific PMC problem instance , we derive the hardness analysis of the TS algorithm with the greedy oracle to solve CMAB problems . Due to the mismatch between the estimation gaps that need to be eliminated by exploration and the actual regret the algorithm needs to pay for each such exploration , the TS algorithm with greedy oracle in this CMAB problem can not achieve as good theoretical performance as previous MAB algorithms . A problem-dependent regret lower bound of order ⌦ ( log T/ 2 ) is provided to illustrate such hardness , where T is the time horizon and is some reward gap . By carefully exploiting the property of the greedy oracle , we also provide an almost matching problem-dependent regret upper bound , which is tight on that PMC problem instance and also recovers the main order of TS when solving MAB problems [ 3 ] . These results are the first theoretical results of TS with approximation oracle to solve CMAB problems , which show that the linear regret example in [ 30 ] does not hold for every approximation oracle . 2 Related Work . CMAB problems have been widely studied in the literature [ 11 , 6 , 7 , 29 , 30 , 24 , 32 , 21 , 14 ] . Here we mainly focus on the most relevant works . Gai et al . [ 11 ] first study the CMAB problem with linear reward , where the reward of an action is linear in the reward of base arms included in it . They introduce a UCB-type algorithm to solve such problems and allow approximation algorithms to serve as offline oracles . Later , Chen et al . [ 6 , 7 , 29 ] generalize this framework by considering a larger class of rewards and the case with probabilistically triggered arms ( CMAB-T ) . This framework only assumes the expected reward satisfies the monotonicity and Lipschitz condition on the mean vector of base arms . The combinatorial UCB ( CUCB ) algorithm is proposed to solve such general CMAB problems , which works with any offline oracle with approximation guarantees . When only approximation oracles are available , the goal of the algorithm is relaxed to minimize the cumulative approximation regret , which is defined as the difference between the expected reward of the selected action and that of the scaled optimal solution . The CUCB algorithm achieves the regret upper bound of order O ( log T/ min ) [ 29 ] , where min is the minimum reward gap from the scaled optimal solution over all suboptimal actions . Compared with UCB-type algorithms which need to compute upper confidence bounds for unknown means of base arms [ 6 , 7 , 29 ] , TS-type algorithms do not require the reward function to satisfy the monotonicity on the mean vector of base arms . Benefited from this and other advantages of easier implementation and better practical performances , TS-type algorithms have recently attracted a lot of interest in CMAB problems . Komiyama et al . [ 17 ] consider using the TS algorithm to solve the MP-MAB problem , where the agent needs to select K from m arms to maximize the sum of rewards over these selected K arms . They provide an optimal regret upper bound of order O ( log T/ K , K+1 ) for the TS algorithm to solve this problem , where K , K+1 is the reward gap between the K-th and ( K + 1 ) -th optimal arm . Later , Wang and Chen [ 30 ] consider using TS to solve more general CMAB problems where only the Lipschitz condition is assumed to be satisfied . Their regret upper bound of order O ( log T/ min ) matches the main order of the CUCB [ 29 ] in the same setting . The coefficient of this upper bound was recently improved by Perrault et al . [ 24 ] , who study the same CMAB setting . Huyuk et al . [ 14 ] extend the analysis of [ 30 ] and consider using the TS algorithm to solve the problem of CMAB-T . However , the current regret upper bound is O ( 1/p⇤ ) worse than CUCB [ 29 ] , where p⇤ is the minimum triggering probability . All of the above TS-based works need an exact oracle to provide the optimal solution with sampled parameters in each round . However , the exact oracles are usually not feasible as many combinatorial optimization problems , such as the offline problem of OIM and PMC , are NP-hard [ 16 ] . Wang and Chen [ 30 ] have constructed a problem instance and designed an approximation offline oracle for this problem instance . The analysis has shown that the TS algorithm suffers the linear regret of order O ( T ) when working with such an approximation oracle . However , this oracle is uncommon and artificial , thus can not represent the performance of TS when working with common approximation oracles . It is still a significant open problem that whether TS can perform well with approximation oracles . The greedy algorithm is one of the most important methods with approximation guarantees to solve combinatorial optimization problems . When the mean vector is known beforehand , we call the problem of finding the action with the best expected reward as offline problem . Using the greedy algorithm to solve offline combinatorial problems has been studied for decades , including the problem of shortest spanning subtree [ 18 ] , shortest connection networks [ 25 ] , set coverage [ 8 ] , influence maximization [ 16 ] , and general submodular optimization [ 23 ] . There is also a line of studies considering using greedy to solve online problems [ 4 , 19 , 10 , 27 , 22 ] , some of them require the exact reward function forms as prior knowledge . Among these works the most related to ours is [ 22 ] , both aiming to solve a general class of online problems . Lin et al . [ 22 ] consider using the online greedy strategy to make decisions based on UCB-type estimators . The algorithm sequentially selects a unit to maximize the current expected reward until no feasible unit can be selected in each round . In their setting , a unit conditioned on a set of previously selected units is regarded as an arm and the marginal reward of selecting this unit is the expected reward of this arm . Since the number of combinations of units is usually exponentially large , there is an exponential number of arms to explore , making the algorithm pay exponential memory cost in practical applications . Based on this framework , the algorithm needs to observe the marginal reward after the decision of each step to update the estimate on the arm . However , such observation may be not available as many combinatorial problems treat the action composed of several units as a whole and select them together . Compared to this work , our framework only needs polynomial memory cost and does not require the observation of the marginal reward . In this paper , we study the problem of CMAB with the common ( approximation ) greedy oracle and hope to answer the question of whether the TS algorithm can work well in this setting .

Summary: This paper studies a setting where Thompson Sampling with greedy approximation oracles is applied to combinatorial semi-bandit. The main contribution is to theoretically reveal how the suboptimality of greedy oracles can affect Thompson sampling in the worst case. Specifically, the greedy regret is shown to be lower bounded by $\Omega(\log T / \Delta^2)$. The authors also prove an almost matching upper bound.   --- After rebuttal ---  I appreciate the authors' kind response. Having read all the responses and review comments, I've found that the result is more significant than I thought. Therefore, I increased my score to 6.