On the Suboptimality  of Thompson Sampling in High Dimensions

1 Introduction . We consider the problem of combinatorial bandits with semi-bandit feedback . At time t = 1 , , ... , T a learner selects a decision x ( t ) ∈ X where X ⊂ { 0 , 1 } d is the set of available decisions . The environment then draws a random vector Z ( t ) ∈ Rd . The learner then observes Y ( t ) = x ( t ) Z ( t ) , where denotes the Hadamard ( elementwise ) product . This setting is called semi bandit feedback . We assume that ( Z ( t ) ) t≥1 are i.i.d. , and that Z1 ( t ) , ... , Zd ( t ) are independent and distributed as Zi ( t ) ∼ Bernoulli ( θi ) for all t , i . Then the learner receives a reward f ( x ( t ) , Z ( t ) ) where f is a known function . The goal is to minimize the regret : R ( T , θ ) = T max x∈X { Ef ( x , Z ( t ) ) } − T∑ t=1 Ef ( x ( t ) , Z ( t ) ) . Initially θ is unknown to the learner and minimizing regret involves exploring suboptimal decisions just enough in order to identify the optimal decision . For any decision x ∈ X , define the reward gap ∆x = max x∈X { Ef ( x , Z ( t ) ) } − Ef ( x , Z ( t ) ) , which is the amount of regret incurred by choosing x instead of an optimal decision x ? ∈ arg max x∈X { Ef ( x , Z ( t ) ) } , and ∆min = minx∈X : ∆x > 0 ∆x the minimal gap . We define m , maxx∈X ∑d i=1 |xi| the size of the maximal decision . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . For this problem , an algorithm which has attracted a lot of interest is Thompson Sampling ( TS ) , which at time t selects the decision maximizing x 7→ f ( x , V ( t ) ) where V ( t ) is a random variable distributed as the posterior distribution of θ knowing the information available at time t , which is Y ( 1 ) , ... , Y ( t− 1 ) . The prior distribution of θ can be chosen in various ways , the most natural being a non-informative distribution such as the uniform distribution . TS is usually computationally simple to implement , for instance when f is linear , since it involves maximizing f over X . Also , for some problem instances it tends to perform well numerically . A particular case of interest is linear combinatorial semi-bandits where f ( x , θ ) = θ > x so that the reward is a linear function of the decision . Our contribution . We show that the regret of TS in general does not scale polynomially in the ambient dimension d. ( i ) We provide several examples , both for linear and non-linear combinatorial bandits , where the regret of TS does not scale polynomially in the dimension d ( in fact in some cases it may scale even faster than exponentially in the dimension ) . In some cases , we show that one must wait for an amount of time greater than Ω ( dd ) for TS to perform at least as well as random choice where one simply chooses x ( t ) uniformly distributed in X at every round . Therefore , in high dimensions , in some instances , TS in general can perform strictly worse than random choice for all practically relevant time horizons . ( ii ) We show that the minimax regret of TS scales at least as Ω ( T 1− 1 d ) so that it is not minimax optimal , as there exists algorithms such as CUCB and ESCB with minimax regret O ( poly ( d ) √ T ( lnT ) ) . In fact , in high dimensions , the minimax regret of TS is almost linear . ( iii ) We further show that adding forced exploration as an initialization step to TS does not correct the minimax problem , so that this is not an artifact due to initialization . ( iv ) Using numerical experiments , we show that indeed , for reasonable time horizons , TS performs very poorly in high dimensions in some instances . We believe that our results highlight two general characteristics of TS . First , TS tends to be much more greedy than optimistic algorithms such as ESCB and CUCB . This greedy behavior explains why the regret of TS is , in some instances , much smaller than that of optimistic algorithms . In fact it is sometimes so greedy that it misses the optimal decision . Second , TS tends to be by nature a ” risky ” algorithm so that its regret exhibits very large fluctuations across runs . In some cases it finds the optimal arm very quickly and with little to no regret , while in other cases it simply misses the optimal decision and performs worse than random choice . Related work . Combinatorial bandits are a generalization of classical bandits studied in [ 15 ] . Several asymptotically optimal algorithms are known for classical bandits , including the algorithm of [ 16 ] , KL-UCB [ 4 ] , DMED [ 12 ] and TS [ 20 , 13 ] . Other algorithms include the celebrated UCB1 [ 2 ] . A large number of algorithms for combinatorial semi-bandits have been proposed , many of which naturally extend algorithms for classical bandits to the combinatorial setting . CUCB [ 5 , 14 ] is a natural extension of UCB1 to the combinatorial setting . ESCB [ 7 , 9 ] is an improvement of CUCB which leverages the independence of rewards between items . AESCB [ 8 ] is an approximate version of ESCB with roughly the same performance guarantees and reduced computational complexity . TS for combinatorial bandits was considered in [ 10 , 21 , 18 ] . Also , combinatorial semi bandits are a particular case of structured bandits , for which there exists asymptotically optimal algorithms such as OSSB [ 6 ] . Table 1 presents the best known regret upper bounds for CUCB , ESCB and TS . For completeness , we also recall the complete regret upper bound for ESCB as Theorem 9 ( Appendix B ) . We provide two types of bounds for TS : problem dependent bounds ( sometimes called gap-dependent bounds ) and minimax bounds ( or gap-free bounds ) . The former involves T , d , m and ∆min , while the latter hold for any value of ∆min . An important observation is that all of the known regret upper bounds for TS [ 10 ] , [ 21 ] , [ 18 ] feature at least one term that does not scale polynomially with the dimension . In particular , the paper [ 18 ] shows that there exists a universal constant C ≥ 0 such that the regret of TS is upper bounded by R ( T ) ≤ C [ d ( lnm ) 2 ln ( |X |T ) /∆min + dm3/∆2min +m ( ( m2 + 1 ) /∆min ) 2+4m ) ] . This is a general bound for all combinatorial sets of interest . The bound has a super exponential term ( that does not depend on T ) : m ( ( m2 + 1 ) /∆min ) 2+4m ) . Regret bounds for CUCB and ESCB do not feature this exponential dependency in the dimension . However , since TS tends to perform very well in all of the numerical experiments presented in the literature , a natural intuition would be that all known upper bounds are simply not sharp , and that the true regret of TS does not really grow exponentially with the dimension . We show in this work that this intuition is incorrect , and that the regret of TS really does scale ( at least ) exponentially in the dimension i.e . it suffers from the ” curse of dimensionality ” . This directly implies that , in high dimensions , and for some combinatorial sets , CUCB and ESCB perform much better than TS . In order to alleviate this problem , it would be natural to attempt to modify the prior distribution used by TS , since it is known to have a strong influence on its performance [ 3 , 17 ] . Similarly , in a related problem , [ 1 ] suggests to use correlated Thompson samples , and [ 11 ] studies the influence of the prior on numerical performance . We believe that this is an interesting open problem . 2 Model . 2.1 Problem Dependent Regret and Minimax Regret . In order to evaluate the performance of an algorithm over the set of instances θ ∈ [ 0 , 1 ] d , there are two main figures of merit that we study in this paper . The first is the problem-dependent regret which is R ( T , θ ) when θ ∈ [ 0 , 1 ] d is fixed . The second is the minimax regret which is the worse case over θ for T fixed : maxθ∈ [ 0,1 ] d R ( T , θ ) . 2.2 TS . The basic TS algorithm works as follows . For i = 1 , ... , d , define Ai ( t ) = ∑t−1 s=1 Zi ( s ) xi ( s ) and Bi ( t ) = ∑t−1 s=1 ( 1 − Zi ( s ) ) xi ( s ) which represent the number of successes and failures observed when getting a sample to estimate θi . We define Ni ( t ) = Ai ( t ) +Bi ( t ) = ∑t−1 s=1 xi ( s ) the number of samples available at time t to estimate θi , and θ̂i ( t ) = Ai ( t ) max ( Ni ( t ) , 1 ) , the corresponding estimate of θi which is simply the empirical mean . The TS algorithm selects decision x ( t ) ∈ arg max x∈X f ( x , V ( t ) ) where Vi ( t ) ∼ Beta ( Ai ( t ) + 1 , Bi ( t ) + 1 ) , and V1 ( t ) , ... , Vd ( t ) are independent . Vector V ( t ) is called the Thompson sample at time t. TS is based on a Bayesian argument , V ( t ) is drawn according to the posterior distribution of θ knowing the information available at time t , where the prior distribution for θ is uniform over [ 0 , 1 ] d. Choosing decision x ( t ) as done above should ensure that one explores just enough to find the optimal decision . In the linear case , the decision can be computed by linear maximization over X : x ( t ) ∈ arg max x∈X { V ( t ) > x } . This explains the practical appeal of TS , since whenever linear maximization over X can be implemented efficiently , the algorithm has low computational complexity . 2.3 TS with Forced Exploration . A natural extension of TS is to add ` forced exploration rounds , where ` is a fixed number , in order to avoid some artifacts that could possibly occur due to the prior distribution . The algorithm operates as follows . At time 1 ≤ t ≤ ` , one selects x ( t ) ∈ X such that xi ( t ) ( t ) = 1 with i ( t ) ∈ min i=1 , ... , d Ni ( t ) . Otherwise for t ≥ ` + 1 one selects x ( t ) ∈ arg max x∈X f ( x , V ( t ) ) , with V ( t ) the Thompson sample defined above . Namely , one first performs a forced exploration during ` rounds then apply TS . This guarantees that Ni ( t ) ≥ b ` /dc samples are available to estimate θi for all i = 1 , ... , d , then one subsequently applies TS . We call this variant TS with ` forced exploration rounds .

Summary: This paper shows that the Thompson Sampling (TS) is suboptimal for combinatorial semi-bandits. In particular, they give instances where the regret scales exponentially in the ambient dimension and shows that its minimax regret is nearly linear. They demonstrate this phenomenon theoretically and empirically.