On the Suboptimality  of Thompson Sampling in High Dimensions

1 Introduction . We consider the problem of combinatorial bandits with semi-bandit feedback . At time t = 1 , , ... , T a learner selects a decision x ( t ) ∈ X where X ⊂ { 0 , 1 } d is the set of available decisions . The environment then draws a random vector Z ( t ) ∈ Rd . The learner then observes Y ( t ) = x ( t ) Z ( t ) , where denotes the Hadamard ( elementwise ) product . This setting is called semi bandit feedback . We assume that ( Z ( t ) ) t≥1 are i.i.d. , and that Z1 ( t ) , ... , Zd ( t ) are independent and distributed as Zi ( t ) ∼ Bernoulli ( θi ) for all t , i . Then the learner receives a reward f ( x ( t ) , Z ( t ) ) where f is a known function . The goal is to minimize the regret : R ( T , θ ) = T max x∈X { Ef ( x , Z ( t ) ) } − T∑ t=1 Ef ( x ( t ) , Z ( t ) ) . Initially θ is unknown to the learner and minimizing regret involves exploring suboptimal decisions just enough in order to identify the optimal decision . For any decision x ∈ X , define the reward gap ∆x = max x∈X { Ef ( x , Z ( t ) ) } − Ef ( x , Z ( t ) ) , which is the amount of regret incurred by choosing x instead of an optimal decision x ? ∈ arg max x∈X { Ef ( x , Z ( t ) ) } , and ∆min = minx∈X : ∆x > 0 ∆x the minimal gap . We define m , maxx∈X ∑d i=1 |xi| the size of the maximal decision . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . For this problem , an algorithm which has attracted a lot of interest is Thompson Sampling ( TS ) , which at time t selects the decision maximizing x 7→ f ( x , V ( t ) ) where V ( t ) is a random variable distributed as the posterior distribution of θ knowing the information available at time t , which is Y ( 1 ) , ... , Y ( t− 1 ) . The prior distribution of θ can be chosen in various ways , the most natural being a non-informative distribution such as the uniform distribution . TS is usually computationally simple to implement , for instance when f is linear , since it involves maximizing f over X . Also , for some problem instances it tends to perform well numerically . A particular case of interest is linear combinatorial semi-bandits where f ( x , θ ) = θ > x so that the reward is a linear function of the decision . Our contribution . We show that the regret of TS in general does not scale polynomially in the ambient dimension d. ( i ) We provide several examples , both for linear and non-linear combinatorial bandits , where the regret of TS does not scale polynomially in the dimension d ( in fact in some cases it may scale even faster than exponentially in the dimension ) . In some cases , we show that one must wait for an amount of time greater than Ω ( dd ) for TS to perform at least as well as random choice where one simply chooses x ( t ) uniformly distributed in X at every round . Therefore , in high dimensions , in some instances , TS in general can perform strictly worse than random choice for all practically relevant time horizons . ( ii ) We show that the minimax regret of TS scales at least as Ω ( T 1− 1 d ) so that it is not minimax optimal , as there exists algorithms such as CUCB and ESCB with minimax regret O ( poly ( d ) √ T ( lnT ) ) . In fact , in high dimensions , the minimax regret of TS is almost linear . ( iii ) We further show that adding forced exploration as an initialization step to TS does not correct the minimax problem , so that this is not an artifact due to initialization . ( iv ) Using numerical experiments , we show that indeed , for reasonable time horizons , TS performs very poorly in high dimensions in some instances . We believe that our results highlight two general characteristics of TS . First , TS tends to be much more greedy than optimistic algorithms such as ESCB and CUCB . This greedy behavior explains why the regret of TS is , in some instances , much smaller than that of optimistic algorithms . In fact it is sometimes so greedy that it misses the optimal decision . Second , TS tends to be by nature a ” risky ” algorithm so that its regret exhibits very large fluctuations across runs . In some cases it finds the optimal arm very quickly and with little to no regret , while in other cases it simply misses the optimal decision and performs worse than random choice . Related work . Combinatorial bandits are a generalization of classical bandits studied in [ 15 ] . Several asymptotically optimal algorithms are known for classical bandits , including the algorithm of [ 16 ] , KL-UCB [ 4 ] , DMED [ 12 ] and TS [ 20 , 13 ] . Other algorithms include the celebrated UCB1 [ 2 ] . A large number of algorithms for combinatorial semi-bandits have been proposed , many of which naturally extend algorithms for classical bandits to the combinatorial setting . CUCB [ 5 , 14 ] is a natural extension of UCB1 to the combinatorial setting . ESCB [ 7 , 9 ] is an improvement of CUCB which leverages the independence of rewards between items . AESCB [ 8 ] is an approximate version of ESCB with roughly the same performance guarantees and reduced computational complexity . TS for combinatorial bandits was considered in [ 10 , 21 , 18 ] . Also , combinatorial semi bandits are a particular case of structured bandits , for which there exists asymptotically optimal algorithms such as OSSB [ 6 ] . Table 1 presents the best known regret upper bounds for CUCB , ESCB and TS . For completeness , we also recall the complete regret upper bound for ESCB as Theorem 9 ( Appendix B ) . We provide two types of bounds for TS : problem dependent bounds ( sometimes called gap-dependent bounds ) and minimax bounds ( or gap-free bounds ) . The former involves T , d , m and ∆min , while the latter hold for any value of ∆min . An important observation is that all of the known regret upper bounds for TS [ 10 ] , [ 21 ] , [ 18 ] feature at least one term that does not scale polynomially with the dimension . In particular , the paper [ 18 ] shows that there exists a universal constant C ≥ 0 such that the regret of TS is upper bounded by R ( T ) ≤ C [ d ( lnm ) 2 ln ( |X |T ) /∆min + dm3/∆2min +m ( ( m2 + 1 ) /∆min ) 2+4m ) ] . This is a general bound for all combinatorial sets of interest . The bound has a super exponential term ( that does not depend on T ) : m ( ( m2 + 1 ) /∆min ) 2+4m ) . Regret bounds for CUCB and ESCB do not feature this exponential dependency in the dimension . However , since TS tends to perform very well in all of the numerical experiments presented in the literature , a natural intuition would be that all known upper bounds are simply not sharp , and that the true regret of TS does not really grow exponentially with the dimension . We show in this work that this intuition is incorrect , and that the regret of TS really does scale ( at least ) exponentially in the dimension i.e . it suffers from the ” curse of dimensionality ” . This directly implies that , in high dimensions , and for some combinatorial sets , CUCB and ESCB perform much better than TS . In order to alleviate this problem , it would be natural to attempt to modify the prior distribution used by TS , since it is known to have a strong influence on its performance [ 3 , 17 ] . Similarly , in a related problem , [ 1 ] suggests to use correlated Thompson samples , and [ 11 ] studies the influence of the prior on numerical performance . We believe that this is an interesting open problem . 2 Model . 2.1 Problem Dependent Regret and Minimax Regret . In order to evaluate the performance of an algorithm over the set of instances θ ∈ [ 0 , 1 ] d , there are two main figures of merit that we study in this paper . The first is the problem-dependent regret which is R ( T , θ ) when θ ∈ [ 0 , 1 ] d is fixed . The second is the minimax regret which is the worse case over θ for T fixed : maxθ∈ [ 0,1 ] d R ( T , θ ) . 2.2 TS . The basic TS algorithm works as follows . For i = 1 , ... , d , define Ai ( t ) = ∑t−1 s=1 Zi ( s ) xi ( s ) and Bi ( t ) = ∑t−1 s=1 ( 1 − Zi ( s ) ) xi ( s ) which represent the number of successes and failures observed when getting a sample to estimate θi . We define Ni ( t ) = Ai ( t ) +Bi ( t ) = ∑t−1 s=1 xi ( s ) the number of samples available at time t to estimate θi , and θ̂i ( t ) = Ai ( t ) max ( Ni ( t ) , 1 ) , the corresponding estimate of θi which is simply the empirical mean . The TS algorithm selects decision x ( t ) ∈ arg max x∈X f ( x , V ( t ) ) where Vi ( t ) ∼ Beta ( Ai ( t ) + 1 , Bi ( t ) + 1 ) , and V1 ( t ) , ... , Vd ( t ) are independent . Vector V ( t ) is called the Thompson sample at time t. TS is based on a Bayesian argument , V ( t ) is drawn according to the posterior distribution of θ knowing the information available at time t , where the prior distribution for θ is uniform over [ 0 , 1 ] d. Choosing decision x ( t ) as done above should ensure that one explores just enough to find the optimal decision . In the linear case , the decision can be computed by linear maximization over X : x ( t ) ∈ arg max x∈X { V ( t ) > x } . This explains the practical appeal of TS , since whenever linear maximization over X can be implemented efficiently , the algorithm has low computational complexity . 2.3 TS with Forced Exploration . A natural extension of TS is to add ` forced exploration rounds , where ` is a fixed number , in order to avoid some artifacts that could possibly occur due to the prior distribution . The algorithm operates as follows . At time 1 ≤ t ≤ ` , one selects x ( t ) ∈ X such that xi ( t ) ( t ) = 1 with i ( t ) ∈ min i=1 , ... , d Ni ( t ) . Otherwise for t ≥ ` + 1 one selects x ( t ) ∈ arg max x∈X f ( x , V ( t ) ) , with V ( t ) the Thompson sample defined above . Namely , one first performs a forced exploration during ` rounds then apply TS . This guarantees that Ni ( t ) ≥ b ` /dc samples are available to estimate θi for all i = 1 , ... , d , then one subsequently applies TS . We call this variant TS with ` forced exploration rounds .

Summary: This paper focusses on Thompson Sampling for stochastic combinatorial semibandits. The main contributions of the paper are to formalise conditions under which Thompson Sampling does not behave optimally for these problems. Through per-instance and minimax analyses, the authors show that when Thompson Sampling is deployed for Bernoulli combinatorial semibandits, and is initialised with a uniform (Beta(1,1)) prior there exist particular action sets and underlying parameters of the reward distribution such that the regret grows sub-optimally. In particular the regret can be shown to be lower bounded by a function that is exponential in the number of arms, or linear in the time horizon for small enough (but potentially still large) time horizons. An empirical investigation ratifies these findings and demonstrates that this issue is not encountered by optimistic algorithms such as ESCB and CUCB.