The Hardness Analysis of Thompson Sampling for Combinatorial Semi-bandits with Greedy Oracle

1 Introduction . Stochastic multi-armed bandit ( MAB ) problem [ 20 , 5 , 3 ] is a classical online learning framework . It has been extensively studied in the literature and has a wide range of applications [ 13 , 12 , 20 ] . The problem is modeled by a T -round game between the learning agent and the environment . The environment contains an arm set and each arm is associated with a reward distribution . At each round t , the agent first selects an arm , while the environment generates a random reward for each arm from its reward distribution . The agent then obtains the reward of the selected arm . The objective of the agent is to accumulate as many expected rewards over T rounds , or equivalent to minimizing the cumulative expected regret , which is defined as the cumulative difference between the expected reward of the optimal arm and the selected arms over T rounds . To achieve this long-horizon goal , the learning agent has to face the dilemma of exploration and exploitation in each round . The former aims to try arms that have not been observed enough times to get a potentially higher reward , the latter focuses on the arm with the best observed performance so far to maintain a high profit . How to balance the tradeoff between exploration and exploitation is the main focus of the MAB algorithms . One of the most popular bandit algorithms is the upper confidence bound ( UCB ) algorithm [ 5 ] . The algorithm aims to construct confidence sets for unknown expected rewards and selects arms according to their highest upper confidence bounds . The UCB-type algorithms have been widely studied and provided theoretical guarantees with regret upper bound of order O ( log T/ ) , where is the ⇤Corresponding author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . minimum gap between the expected reward of the optimal arm and any suboptimal arms . Thompson sampling ( TS ) [ 3 , 26 ] is another popular method to solve MAB problems . It is a randomized algorithm based on the Bayesian idea , which maintains an iteratively updated posterior distribution for each arm and chooses arms according to their probabilities of being the best one . The TS algorithm was introduced in the 1930s [ 28 ] , but its theoretical analysis is open until recent years [ 15 , 2 ] . It was proven that the regret upper bound of the TS algorithm is of the same order of O ( log T/ ) in MAB problems [ 3 ] . Benefited from the advantages of easier implementation and better empirical performance compared to UCB , the TS-type algorithms attract more attentions in recent years . Despite its importance , the MAB framework may fail to model many real applications since the agent ’ s action is usually not a single arm but a combination of several arms . This motivates the study on the combinatorial MAB ( CMAB ) problem [ 11 , 9 , 17 , 31 , 7 , 29 , 30 , 24 ] . In the CMAB framework , the agent selects a combination of base arms as an action to play in each round . All outcomes of these selected arms are then revealed to the agent , which is called semi-bandit feedback and is widely studied in the literature [ 7 , 29 , 32 , 30 , 14 , 24 ] . Such CMAB framework can cover many real application scenarios including probabilistic maximum coverage ( PMC ) [ 6 ] , online influence maximization ( OIM ) [ 29 ] , multiple-play MAB ( MP-MAB ) [ 17 ] and minimum spanning tree ( MST ) . TS-type algorithms have recently attracted a lot of interest in CMAB problems [ 17 , 30 , 14 , 24 ] . All of these works need an exact oracle to provide the optimal solution with sampled parameters as input in each round . However , such oracles are usually not feasible since many offline combinatorial optimization problems , such as the offline problem of PMC and OIM [ 16 ] , are NP-hard and only approximation oracles are available . With an example [ 30 ] illustrating the non-convergent regret of TS with an artificial approximation oracle designed for a specific problem instance , whether TS can work well in CMAB problems with common approximation oracles is still an open problem . One of the most common oracles for offline combinatorial optimization problems with a theoretical guarantee is the greedy algorithm . It sequentially finds the current optimal arm , or the optimal collection of multiple arms according to the structural correlations , to maximize the current total expected reward . When the termination condition is met , it will return the set of all arms found in previous steps as the solution . The termination condition is usually formulated by a constraint on the number of steps . For example , in the PMC , OIM , and MP-MAB problems , such a process is limited to continue K steps . The greedy algorithm can provide approximate solutions for offline problems of PMC [ 8 ] and OIM [ 16 ] , and exact optimal solutions for offline problems of MP-MAB [ 17 ] and MST [ 18 ] . In general , as long as the expected reward in a problem satisfies the monotonicity and submodularity on the action set , the greedy algorithm serves as an offline oracle to provide an approximate solution [ 23 ] . In this paper , we first formulate the CMAB problems with greedy oracle , which is general enough and covers PMC , MP-MAB , and many other problems . In this framework , the objective of the learning agent is to minimize the cumulative greedy regret , defined as the cumulative difference between the expected reward of the selected action and that of the greedy ’ s solution in the real environment . Focusing on a specific PMC problem instance , we derive the hardness analysis of the TS algorithm with the greedy oracle to solve CMAB problems . Due to the mismatch between the estimation gaps that need to be eliminated by exploration and the actual regret the algorithm needs to pay for each such exploration , the TS algorithm with greedy oracle in this CMAB problem can not achieve as good theoretical performance as previous MAB algorithms . A problem-dependent regret lower bound of order ⌦ ( log T/ 2 ) is provided to illustrate such hardness , where T is the time horizon and is some reward gap . By carefully exploiting the property of the greedy oracle , we also provide an almost matching problem-dependent regret upper bound , which is tight on that PMC problem instance and also recovers the main order of TS when solving MAB problems [ 3 ] . These results are the first theoretical results of TS with approximation oracle to solve CMAB problems , which show that the linear regret example in [ 30 ] does not hold for every approximation oracle . 2 Related Work . CMAB problems have been widely studied in the literature [ 11 , 6 , 7 , 29 , 30 , 24 , 32 , 21 , 14 ] . Here we mainly focus on the most relevant works . Gai et al . [ 11 ] first study the CMAB problem with linear reward , where the reward of an action is linear in the reward of base arms included in it . They introduce a UCB-type algorithm to solve such problems and allow approximation algorithms to serve as offline oracles . Later , Chen et al . [ 6 , 7 , 29 ] generalize this framework by considering a larger class of rewards and the case with probabilistically triggered arms ( CMAB-T ) . This framework only assumes the expected reward satisfies the monotonicity and Lipschitz condition on the mean vector of base arms . The combinatorial UCB ( CUCB ) algorithm is proposed to solve such general CMAB problems , which works with any offline oracle with approximation guarantees . When only approximation oracles are available , the goal of the algorithm is relaxed to minimize the cumulative approximation regret , which is defined as the difference between the expected reward of the selected action and that of the scaled optimal solution . The CUCB algorithm achieves the regret upper bound of order O ( log T/ min ) [ 29 ] , where min is the minimum reward gap from the scaled optimal solution over all suboptimal actions . Compared with UCB-type algorithms which need to compute upper confidence bounds for unknown means of base arms [ 6 , 7 , 29 ] , TS-type algorithms do not require the reward function to satisfy the monotonicity on the mean vector of base arms . Benefited from this and other advantages of easier implementation and better practical performances , TS-type algorithms have recently attracted a lot of interest in CMAB problems . Komiyama et al . [ 17 ] consider using the TS algorithm to solve the MP-MAB problem , where the agent needs to select K from m arms to maximize the sum of rewards over these selected K arms . They provide an optimal regret upper bound of order O ( log T/ K , K+1 ) for the TS algorithm to solve this problem , where K , K+1 is the reward gap between the K-th and ( K + 1 ) -th optimal arm . Later , Wang and Chen [ 30 ] consider using TS to solve more general CMAB problems where only the Lipschitz condition is assumed to be satisfied . Their regret upper bound of order O ( log T/ min ) matches the main order of the CUCB [ 29 ] in the same setting . The coefficient of this upper bound was recently improved by Perrault et al . [ 24 ] , who study the same CMAB setting . Huyuk et al . [ 14 ] extend the analysis of [ 30 ] and consider using the TS algorithm to solve the problem of CMAB-T . However , the current regret upper bound is O ( 1/p⇤ ) worse than CUCB [ 29 ] , where p⇤ is the minimum triggering probability . All of the above TS-based works need an exact oracle to provide the optimal solution with sampled parameters in each round . However , the exact oracles are usually not feasible as many combinatorial optimization problems , such as the offline problem of OIM and PMC , are NP-hard [ 16 ] . Wang and Chen [ 30 ] have constructed a problem instance and designed an approximation offline oracle for this problem instance . The analysis has shown that the TS algorithm suffers the linear regret of order O ( T ) when working with such an approximation oracle . However , this oracle is uncommon and artificial , thus can not represent the performance of TS when working with common approximation oracles . It is still a significant open problem that whether TS can perform well with approximation oracles . The greedy algorithm is one of the most important methods with approximation guarantees to solve combinatorial optimization problems . When the mean vector is known beforehand , we call the problem of finding the action with the best expected reward as offline problem . Using the greedy algorithm to solve offline combinatorial problems has been studied for decades , including the problem of shortest spanning subtree [ 18 ] , shortest connection networks [ 25 ] , set coverage [ 8 ] , influence maximization [ 16 ] , and general submodular optimization [ 23 ] . There is also a line of studies considering using greedy to solve online problems [ 4 , 19 , 10 , 27 , 22 ] , some of them require the exact reward function forms as prior knowledge . Among these works the most related to ours is [ 22 ] , both aiming to solve a general class of online problems . Lin et al . [ 22 ] consider using the online greedy strategy to make decisions based on UCB-type estimators . The algorithm sequentially selects a unit to maximize the current expected reward until no feasible unit can be selected in each round . In their setting , a unit conditioned on a set of previously selected units is regarded as an arm and the marginal reward of selecting this unit is the expected reward of this arm . Since the number of combinations of units is usually exponentially large , there is an exponential number of arms to explore , making the algorithm pay exponential memory cost in practical applications . Based on this framework , the algorithm needs to observe the marginal reward after the decision of each step to update the estimate on the arm . However , such observation may be not available as many combinatorial problems treat the action composed of several units as a whole and select them together . Compared to this work , our framework only needs polynomial memory cost and does not require the observation of the marginal reward . In this paper , we study the problem of CMAB with the common ( approximation ) greedy oracle and hope to answer the question of whether the TS algorithm can work well in this setting .

Summary: The paper studies the combinatorial multi-armed bandit (CMAB) problem in a setting where exactly solving the corresponding offline optimization problem is not feasible (due to its computational hardness) and the learning agent only has access to a greedy oracle which approximately solves the offline optimization problem. In this framework, the paper analyzes the behavior of a natural variant of the Thompson sampling algorithm called Combinatorial Thompson sampling (CTS). The paper first gives an algorithm-dependent and problem-dependent lower bound showing that CTS faces a fundamental limit, then provides an almost matching upper bound.