Title: Thompson Sampling For Combinatorial Bandits: Polynomial Regret and Mismatched Sampling Paradox
Venue: NeurIPS 2024
Accepted: yes
Keywords: Combinatorial bandits

Thomspon Sampling
Abstract: We consider Thompson Sampling (TS) for linear combinatorial semi-bandits and subgaussian rewards. We propose the first known TS whose finite-time regret does not scale exponentially with the dimension of the problem. We further show the mismatched sampling paradox: A learner who knows the rewards distributions and samples from the correct posterior distribution can perform exponentially worse than a learner who does not know the rewards and simply samples from a well-chosen Gaussian posterior. The code used to generate the experiments is available at https://github.com/RaymZhang/CTS-Mismatched-Paradox

Thompson Sampling For Combinatorial Bandits: Polynomial Regret and Mismatched Sampling Paradox

Abstract

We consider Thompson Sampling (TS) for linear combinatorial semi-bandits and subgaussian rewards. We propose the first known TS whose finite-time regret does not scale exponentially with the dimension of the problem. We further show the mismatched sampling paradox: A learner who knows the rewards distributions and samples from the correct posterior distribution can perform exponentially worse than a learner who does not know the rewards and simply samples from a well-chosen Gaussian posterior. The code used to generate the experiments is available at https://github.com/RaymZhang/CTS-Mismatched-Paradox

1 Introduction and Setting

We consider the linear combinatorial bandit problem with semi-bandit feedback: At time t P r T s : GLYPH<16> t 1 , ..., T u a learner selects an action A p t q P A where the set of available actions A GLYPH<128> t 0 , 1 u d is a known combinatorial set, i.e., a set of binary vectors. Then the environment draws a random vector X p t q P R d , and the learner then observes Y p t q GLYPH<16> A p t q d X p t q , where d denotes the Hadamard (element-wise) product, and the learner obtains a reward of A p t q J X p t q . We assume that the vectors p X p t qq t ¥ 1 are drawn i.i.d. from some distribution with expectation E p X p t qq GLYPH<16> µ GLYPH<14> and that the entries of X p t q are independent. The vector µ GLYPH<14> lies in some Θ GLYPH<128> R d and is initially unknown to the learner. The learner wants to minimize the regret:

Where A GLYPH<14> P arg max A P A t A J µ GLYPH<14> u is the optimal action and ∆ A : GLYPH<16> A GLYPH<14>J µ GLYPH<14> GLYPH<1> A J µ GLYPH<14> is the reward gap between action A and optimal action A GLYPH<14> . The regret is the expected difference between the sum of rewards obtained by an oracle who knows µ GLYPH<14> and always selects the optimal decision and that obtained by the learner. We assume that the optimal decision is unique. To state regret bounds, we use the following notation. We denote by ∆ min : GLYPH<16> min A P A :∆ A ¡ 0 ∆ A the minimal reward gap and ∆ max : GLYPH<16> max A P A ∆ A the maximal reward gap, ∆ p t q : GLYPH<16> ∆ A p t q the reward gap of the action selected at time t , m : GLYPH<16> max A P A } A } 1 the maximal size of an action.

The regret depends on the distributions of the random vector X p t q , which generates the rewards, and we will assume throughout the paper that X p t q is σ 2 -subgaussian so that for all λ P R d :

This assumption holds in many scenarios of interest, for instance, when X p t q P r a, b s d with σ 2 GLYPH<16> p b GLYPH<1> a q 2 { 4 , or when X p t q is normally distributed with a covariance matrix smaller than σ 2 I d . We assume that σ is known, or at least upper-bounded.

One of the candidate algorithms for this problem is Thompson Sampling (TS), which is based on Bayesian inference. We consider a prior distribution π 0 over Θ , a likelihood function glyph[lscript] and π t the posterior distribution of µ GLYPH<14> at time t knowing the observations and the actions up to time t :

The TS algorithm with prior π 0 and likelihood glyph[lscript] consists in sampling θ p t q from distribution π t , and select action A p t q P arg max a P A A J θ p t q . The random vector θ p t q , called a Thompson Sample, acts as a proxy for the unknown µ GLYPH<14> and guides the exploration.

We will show how TS behaves with either Gaussian likelihood glyph[lscript] p x, µ q GLYPH<16> p 2 π q GLYPH<1> 1 { 2 e GLYPH<1>p x GLYPH<1> µ q 2 { 2 or Bernoulli likelihood glyph[lscript] p x, µ q GLYPH<16> 1 t x GLYPH<16> 1 u µ GLYPH<0> p 1 GLYPH<1> µ q 1 t x GLYPH<16> 0 u in the next sections. If the likelihood glyph[lscript] p X i p s q , µ i q is selected as the distribution of X i p s q knowing µ i then we say that TS is natural. If the likelihood glyph[lscript] p X i p s q , µ i q is different from the distribution of X i p s q knowing µ i , we will say that TS is mismatched. An example of mismatched TS would be to select glyph[lscript] as the Gaussian likelihood, although the actual distribution of X p t q knowing µ GLYPH<14> is, say, Bernoulli or some other bounded distribution. Ultimately, the likelihood function is a choice left up to the learner to control how TS explores the suboptimal actions. Perhaps counterintuitively, mismatched TS can outperform natural TS, as we will demonstrate. The prior π 0 can be chosen in various ways, for instance, Jeffrey's non-informative prior. It can even be chosen as an improper prior, where ‡ Θ π 0 p µ q dµ GLYPH<16> GLYPH<0>8 , as long as the integral in the definition of π t is well-defined.

TS is usually computationally simple to implement as it requires a linear maximization over the action space A at each step, which we assume can be done in polynomial time in m and d . This fact explains the practical appeal of TS since whenever linear maximization over A can be implemented efficiently; the algorithm has low computational complexity. Also, for some problem instances, it tends to perform well numerically.

2 Related Work and Contribution

Combinatorial bandits are a generalization of classical bandits studied in [17]. Several asymptotically optimal algorithms are known for classical bandits, including the algorithm of [18], KL-UCB [5], DMED [13] and TS [14, 22]. Other algorithms include the celebrated UCB1 [4]. Numerous algorithms for combinatorial semi-bandits have been proposed, many of which naturally extend algorithms for classical bandits to the combinatorial setting. CUCB [6, 16] is a natural extension of UCB1 to the combinatorial setting. ESCB [8, 10] is an improvement of CUCB, which leverages the independence of rewards between items. AESCB [9] is an approximate version of ESCB with roughly the same performance guarantees and reduced computational complexity. TS for combinatorial semi bandits was considered in [11, 15, 20, 21, 23] while TS for linear bandit was studied in [1, 3]. Also, combinatorial semi-bandits are a particular case of structured bandits, for which there exist asymptotically optimal algorithms such as OSSB [7]. It was also shown in [15] that TS is asymptotically and finite time optimal for matroid-like action sets. The Bayesian regret of TS has been extensively studied, e.g., [21].

Three types of regret bounds exist in the literature: R p T, µ GLYPH<14> q is the problem-dependent regret, i.e., the regret of the learner on the particular problem instance defined by µ GLYPH<14> . The minimax regret is max µ P Θ R p T, µ GLYPH<14> q and the Bayesian regret E µ GLYPH<14> r R p T, µ GLYPH<14> qs where µ GLYPH<14> is drawn according to some prior distribution. We will state regret bounds as a function of the parameters T , the time horizon; d , the problem dimension; m , the maximal size of an action; and ∆ min , the minimal gap.

GLYPH<1>

GLYPH<9>

The paper [16] showed that the problem-dependent regret of CUCB is upper bounded by O dm ln T ∆ min and its minimax regret is upper bounded by O p ? σ 2 dmT ln T GLYPH<0> dm q . [10] showed that the problem-

dependent regret of ESCB is upper bounded by O GLYPH<1> σ 2 d p ln m q 2 ln T ∆ min GLYPH<0> dm 3 ∆ 2 min GLYPH<9> and its minimax regret is upper bounded by O p a d p ln m q 2 T ln T GLYPH<0> dm q . [20] showed that the problem-dependent regret of TS is upper bounded by

While this bound is almost optimal in the asymptotic regime where T GLYPH<209> 8 , it is exponentially suboptimal in the finite time regime since the last term of this expression scales exponentially with m . Unless one assumes a particular type of action set as in [15], all the known generic regret upper bound for TS in the literature [11], [23], [20] feature an exponential dependency on m . [24] further showed that the problem-dependent regret of TS for some simple combinatorial set can be lower bounded by an expression scaling exponentially in m ,

They further showed that Thompson Sampling is not minimax optimal for some combinatorial bandit problems. Therefore, the exponential term is not an artefact of the analysis of [20]. It is also noted that the regret upper bounds for ESCB and CUCB do not feature this exponential dependency in m , suggesting that those algorithms are better in the finite time regime than the versions of TS analyzed so far in the literature. This is unfortunate because TS usually has very low computational complexity, and having an algorithm with both low computational complexity and low regret in the finite time regime would be highly desirable.

Our contribution : (i) We propose a new variant of TS with a regret upper bounded by:

where P is a polynomial in m,d, 1 ∆ min , ∆ max . This polynomial term is a clear improvement over the bound of [20] in the finite time, high dimensional regime where T is relatively small and m is large. Indeed, the last term in this bound P p m,d, 1 { ∆ min , ∆ max , σ q will be much smaller than the last term in the bound of [20] m GLYPH<0> σ p m 2 GLYPH<0> 1 q{ ∆ min GLYPH<8> 2 GLYPH<0> 4 m which is exponential in m . To design our variant, we add a slight exploration boost to TS, which vanishes as T GLYPH<209>8 but significantly impacts the algorithm behaviour when T is moderate and m is large. Also note that the improvement in the ln m term comes from a direct application of a result in [19].

(ii) We design new proof strategies to derive this upper bound, which are based on carefully bounding the sample path behaviour of TS. We believe those strategies are an essential contribution to the analysis of TS and enable us to show that with high probability, TS will sample the optimal action at least Ω p t α q times with α ¡ 0 . This number serves to control the transient behaviour of TS.

(iii) As a by-product, we show the mismatched sampling paradox of TS: in some cases, mismatched TS performs exponentially better than natural TS. For instance, in a problem where X p t q has Bernoulli distribution, a learner using a uniform (improper) prior and a Gaussian likelihood can perform exponentially better than a learner using a Beta prior (which includes Jeffreys' prior) and the Bernoulli likelihood. In essence, trying to exploit the learner's statistical knowledge about the model ends up harming them.

(iv) We confirm our theoretical predictions using numerical experiments, which clearly show that our variant of TS outperforms by several orders of magnitude the Beta-based versions studied in the literature whose regret scales exponentially in the ambient dimension.

3 Algorithms

In this section, we present three TS algorithms: B-CTS (Beta-Combinatorial Thompson Sampling), which is TS with a beta prior and a Bernoulli likelihood, G-CTS (Gaussian-Combinatorial Thompson Sampling) which is TS with a uniform (improper) prior and a Gaussian likelihood, and finally BG-CTS (Boosted Gaussian-Combinatorial Thompson Sampling), an algorithm we propose by introducing a carefully chosen exploration boost in G-CTS.

3.1 Notation

We use the following notation to state algorithms. We define the statistics N p t q : GLYPH<16> GLYPH<176> s Pr t GLYPH<1> 1 s A p s q , the vector containing the number of times each item has been selected, M A p t q : GLYPH<16> GLYPH<176> s Pr t GLYPH<1> 1 s 1 t A p s q GLYPH<16> A u the number of times action A has been selected until t , V p t q : GLYPH<16> D GLYPH<1> 1 N p t q the diagonal matrix whose diagonal elements are p 1 { N 1 p t q , ..., 1 { N d p t qq , p µ p t q : GLYPH<16> V p t q GLYPH<176> s Pr t GLYPH<1> 1 s X i p s q d A i p s q the empirical average estimator for µ GLYPH<14> . We denote by H p t q GLYPH<16> p A p s q , A p s q d X p s qq s Pr t GLYPH<1> 1 s the history which contains all the information collected by the learner up to time t , and which includes both the observations and the selected actions. For two vectors α, β in p R GLYPH<0> q d we denote by Beta p α, β q GLYPH<16> ´ d i GLYPH<16> 1 Beta p α i , β i q the distribution of a vector with independent entries, and where the i -th entry is Beta p α i , β i q distributed.

3.2 B-CTS

B-CTS (see 1 in the algorithm format) considers the prior π 0 GLYPH<16> Beta p α p 0 q , β p 0 qq where α p 0 q , β p 0 q , are two vectors in R d chosen by the learner and the Bernoulli likelihood glyph[lscript] p x, µ q GLYPH<16> 1 t x GLYPH<16> 1 u µ GLYPH<0> p 1 GLYPH<1> µ q 1 t x GLYPH<16> 0 u . If α p 0 q GLYPH<16> β p 0 q GLYPH<16> p 1 , ..., 1 q , then the prior π 0 is uniform over r 0 , 1 s d . If α p 0 q GLYPH<16> β p 0 q GLYPH<16> p 1 { 2 , ..., 1 { 2 q , then the prior is Jeffreys' non-informative prior, which is proportional to the square root of the determinant of the Fisher information matrix. In B-CTS, the posterior distribution π t is also a Beta distribution so that the Beta-CTS selects the action:

where vectors α p t q , β p t q are defined as:

3.3 G-CTS

G-CTS considers the improper prior π 0 which is constant and equal to 1 { σ on all R d and Gaussian likelihood glyph[lscript] p x, µ q GLYPH<16> p 2 πσ 2 q GLYPH<1> 1 { 2 e GLYPH<1>p X GLYPH<1> µ q 2 {p 2 σ 2 q , where σ 2 is the variance. Of course, since π 0 is improper, for π t to be well-defined, we require that enough samples have been collected so that N p t q ¥ 1 . This is easily achieved by selecting d actions A 1 , ..., A d that cover A in the sense that GLYPH<176> i Pr d s A i ¥ 1 , and initializing the algorithm by sampling each of them once. In G-CTS, the posterior distribution π t is also a Gaussian distribution so that the G-CTS selects the action:

3.4 BG-CTS

BG-CTS (see 2 in the algorithm format) is a modification of G-CTS that we propose and that selects the action

and λ P R GLYPH<0> is an input parameter of the algorithm. BG-CTS behaves like G-CTS with a timevarying boost in its exploration denoted by g p t q . This boost asymptotically behaves like a constant lim t GLYPH<209>8 g p t q GLYPH<16> 1 GLYPH<0> λ . This boost ensures a much better finite-time behaviour, especially in the moderate T , large m regime, to avoid the exponentially large regret that can occur in TS. The form of g p t q is not arbitrary and is derived from the self-normalized concentration inequalities that control the large deviations of vector p µ p t q . To make our analysis clearer, we will assume that there exists an exogenous process p Z p t qq t ¥ 1 of i.i.d. N p 0 , I d q vectors that serves as the random generator number for the Thompson samples with θ p t q GLYPH<16> p µ p t q GLYPH<0> σ a 2 g p t q V 1 2 p t q Z p t q .

This decomposition is useful for separating the algorithm's randomness from the bandit environment's randomness. We notice that for all s glyph[greaterorequalslant] t , Z p s q and the history H p t q are independent. Furthermore, we call Z p s q the random part of the Thompson sample, A J θ p t q the Thompson sample of action A

4 Main Result

We now state Theorem 1, our main result.

Theorem 1. For λ GLYPH<16> 1 , and σ 2 subgaussian rewards, the regret of BG-CTS is upper bounded by:

with C,C 1 universal constants and P a polynomial in m,d, ∆ , ∆ max , σ

Theorem 1 states that the regret of BG-CTS is upper bounded by an expression with both the correct behaviour when T is large i.e., both this bound and that of [20] give the same upper bound on lim sup T GLYPH<209>8 R p T,µ GLYPH<14> q ln T , but also a polynomial dependency in m,d, 1 ∆ min , ∆ max , σ . This result predicts that BG-CTS performs much better than other TS variants in the regime where the time horizon T is moderate and the decision size m is large.

A consequence of Theorem 1 combined with prior known results of [24] this is the mismatched sampling paradox for TS: a learner attempting to leverage his knowledge about the statistical model by using natural TS can perform exponentially worse than a learner willingly ignoring this knowledge and using mismatched TS by using an algorithm such as BG-CTS. Consider the example of [24] which features two disjoint actions of size m GLYPH<16> d { 2 written p 1 , ..., 1 , 0 , ..., 0 q and p 0 , ..., 0 , 1 , ..., 1 q and Bernoulli rewards. Suppose the learner attempts to leverage that she knows the rewards are Bernoulli and that the parameter space is r 0 , 1 s d . She will employ a uniform prior over r 0 , 1 s d and the Bernoulli likelihood. This means using B-CTS and getting a regret that scales exponentially with d as shown in [24]. Using B-CTS with Jeffrey's prior does not help either. On the other hand, if the learner pretends she does not know the parameter space nor the rewards distribution and uses B-CTS, she gets a regret scaling only polynomially in d . Furthermore, Bernoulli rewards are σ 2 subgaussian with σ 2 glyph[lessorequalslant] 1 { 4 as stated above, so our regret upper bound for BG-CTS applies to this example.

At first glance, it seems outright absurd to use a prior whose support is the whole of R d instead of the actual parameter space r 0 , 1 s d , and using a Gaussian likelihood, which is continuous when the rewards are binary, but this paradoxically gives exponentially better performance. This paradox leads us to believe one should be careful when using posterior sampling for regret minimization. While this is natural for Bayesian inference, things seem to be much more complex when solving bandit problems, which feature both inference and control/exploration.

5 Regret Analysis

In this section, we describe how to prove our main result. Due to space constraints, some proof elements are relegated to the appendix. In particular, to make this proof self-contained, we reproduce (without their proofs) the results from previous work that we use for our analysis. A reader can try to follow the proof with the help of the diagram in figure 2.

A fundamental idea of our analysis is to consider the event A t where both events occur :

When A t occurs, we say that we observe a clean run up to time t . A clean run up to time t implies that the Thompson sample of any action A at any time s P r t s cannot exceed the sum of its expected value and a bonus proportional to A J V 1 2 p t q A , which can be interpreted as the confidence bonus used in the CUCB algorithm. A clean run also implies that there exist many instants at which the Thompson sample of the optimal action is at least as large as its expected reward A GLYPH<14>J µ GLYPH<14> .

5.1 Probability of observing a clean run

We first now show that most runs are clean, i.e., clean runs occur with high probability. Proposition 2 states that the probability of a non-clean run up to time t is much smaller than 1 { t , and therefore non-clean runs cause little regret.

Proposition 2. For all t ¥ C 5 , we have P p A t q ¥ 1 GLYPH<1> 4 dt GLYPH<1> 2 GLYPH<1> t GLYPH<1> 1 p ln t q GLYPH<1> 2 GLYPH<1> e GLYPH<1> C 4 t α with C 4 GLYPH<16> C 2 { 8 and C 5 GLYPH<16> 23 .

Proof : The proof is relatively technical, and involves decomposing A t according to the fluctuations of θ p t q and p µ p t q . We decompose the Thompson sample of the optimal action as follows:

with

which represent the deviation between the empirical mean and the expected reward, and the deviation of the Thompson sample from the expected value of the Thompson sample.

We introduce the following deviation events

Each of those events can be interpreted as follows. B t means that the empirical mean of some item deviates from its expected value at least once, C t means that the randomization in the Thompson sample is abnormally large at least once, D t implies that the empirical mean of the optimal action deviates from its expected value at least once, and E t means that there exist too few instants at which S GLYPH<14> p t q is reasonably large.

Assume that none of B t , C t , D t , E t occur. For all A P A and all s P r t s :

since if B t does not occur:

and if C t does not occur and because g p t q GLYPH<160> 2 p 2 m GLYPH<0> 1 q GLYPH<160> 6 m see lemma f. 7:

Furthermore if D t and E t do not occur, there exists at least C 2 t α instants such that S GLYPH<14> p s q ¥ σ a 2 f p t q and U GLYPH<14> p s q ¥ GLYPH<1> σ a 2 f p t q , which implies A GLYPH<14>J θ p s q ¥ A GLYPH<14>J µ GLYPH<14> . This means that:

Therefore A t occurs, and we have a clean run. Hence P p A t q ¥ 1 GLYPH<1> P p B t q GLYPH<1> P p C t q GLYPH<1> P p D t q GLYPH<1> P p E t q . We now upper bound the probability of each event separately.

5.1.1 Probability of B t

Using a union bound P p B t q / GLYPH<176> i Pr d s P GLYPH<1> max s Pr t s a V i p s qp µ GLYPH<14> i p s q GLYPH<1> p µ p s qq ¥ σ ? 8 ln t GLYPH<9> / 2 dt GLYPH<1> 2 . We used the concentration inequality first derived by [16] in their analysis of CUCB and recalled in lemma 7.

5.1.2 Probability of C t

Using a union bound and a Chernoff bound for the Gaussian distribution (lemma 10), wheres Q is the tail function of the standard Gaussian distribution :

5.1.3 Probability of D t

We have for t glyph[greaterorequalslant] 2 , P p D t q / t GLYPH<1> 1 p ln t q GLYPH<1> 2 from the concentration inequality derived by [10] in their analysis of ESCB and recalled in lemma 5.

5.1.4 Probability of E t

In order to control the probability of E t , consider the following counting process:

We wish to show that, with high probability, W p s q ¥ C 2 t α . One may readily check that W p s q is a sum of binary variables and that its conditional expected increment verifies:

since, conditional to H p s q , S GLYPH<14> p s q has a gaussian distribution with mean 0 and variance 2 g p s q . Let us lower bound of the sum of p . By considering t ¥ C 5 we have

using the fact that p is increasing in s , and the study of f p t q g p t { 2 q done in lemma f. 1 , and lemma 10 on the asymptotic behaviour of the Q function.

We can now conclude by applying a multiplicative Azuma-Hoeffding style bound to W p s q presented in lemma 6 in the appendix. With C 4 GLYPH<16> C 2 { 8 we have :

5.1.5 Putting everything together

Adding up the four previous bounds, for all t ¥ C 5 , P p A t q ¥ 1 GLYPH<1> 4 dt GLYPH<1> 2 GLYPH<1> t GLYPH<1> 1 p ln t q GLYPH<1> 2 GLYPH<1> e GLYPH<1> C 4 t α .

5.2 Thompson sample for the optimal action on clean run

We have already established that clean runs occur with high probability, and now we concentrate on how the algorithm behaves on those runs. Proposition 3 further shows that the optimal action will be selected numerous times when a clean run occurs. In turn, the Thompson sample of the optimal action will be arbitrarily close to its expected reward. This argument is the cornerstone of our analysis (that we believe to be missing in the previous analysis of [20]) and will allow us to control the transient behaviour of the algorithm.

Proposition 3. For t ¥ P 1 p m,d, 1 ∆ min , σ q , if A t occurs, then we must have M A GLYPH<14> p t q ¥ C 6 t α and A GLYPH<14>J θ p t q ¥ A GLYPH<14>J µ GLYPH<14> GLYPH<1> h p t q . With P 1 a polynomial in m,d, 1 ∆ min , σ , C 6 GLYPH<16> C 4 { 2 , and h p t q GLYPH<16> C 1 σm b m ln t C 6 t α . It is noted that lim t GLYPH<209>8 h p t q GLYPH<16> 0

Proof: Let us consider a clean run. We can count the number of times the optimal action was not chosen and the variance term of the action is greater than ∆ min :

And since we have a clean run: |t s P r t s : A GLYPH<14>J θ p s q ¥ A GLYPH<14>J µ GLYPH<14> u| ¥ C 2 t α . At those times, if the variance term of the action played is less than ∆ min , then it means that the optimal action has been played due to the first condition of A t . So we get :

With C 6 GLYPH<16> C 2 { 2 for t ¥ P 1 p m,d, 1 ∆ min , σ q : GLYPH<16> GLYPH<0> 1 α GLYPH<8> 1 GLYPH<0> 2 α GLYPH<0> 1 GLYPH<1> 1 e GLYPH<8> GLYPH<1> 1 { α GLYPH<1> C 2 1 C 2 GLYPH<9> 1 GLYPH<0> 1 { α GLYPH<1> dσ 2 m 3 ∆ 2 min GLYPH<9> 1 GLYPH<0> 1 { using lemma f. 8. Recall that under A :

Since M A GLYPH<14> p t q ¥ C 6 t α has been selected at least C 6 t α times up to time t , we have A GLYPH<14>J V 1 { 2 A GLYPH<14> / m ? C 6 t α so we get the announced result :

5.3 Regret upper bound

We can now analyze the regret. Let us define some more events at time t :

"

*

The event Z t means a suboptimal play. F t implies that the empirical mean of one of the items in the action selected at time t deviates from its expectation. G t means that the Thompson sample of the decision played is far from its true value, and finally, H t is for when the Thompson sample from the arm played is far from its empirical mean. The complete event system decomposed as follows.

5.3.1 Regret due to s A t

Using proposition 2 we have that P p s A t q / 4 dt GLYPH<1> 2 GLYPH<0> t GLYPH<1> 1 p ln t q GLYPH<1> 2 GLYPH<0> e GLYPH<1> C 4 t α . Then we can use the fact that GLYPH<176> t P N GLYPH<14> 1 t 2 GLYPH<16> π 2 6 , GLYPH<176> t P N GLYPH<14> 1 t p ln t q 2 GLYPH<160> 4 . And furthermore, with lemma f. 9 we have that GLYPH<176> t P N GLYPH<14> e GLYPH<1> C 4 t α GLYPH<160> C GLYPH<1> 1 { α 4 α Γ p 1 α q . Therefore, the regret caused by s A t is upper bounded by:

5.3.2 Regret due to s G t X A t

We use proposition 3 and we get that for t ¥ P 1 p m,d, 1 { ∆ min , σ q , A GLYPH<14>J θ p t q ¡ A GLYPH<14>J µ GLYPH<14> GLYPH<1> h p t q . Combining with event s G t , we know that when h p t q GLYPH<160> ∆ min { 4 , the only action that can be played is the optimal one. We recall the formula of h p t q GLYPH<16> C 1 σm b m ln t C 6 t α And using lemma f. 8, this happens for t ¡ P 2 p m, 1 ∆ min , σ q : GLYPH<16> GLYPH<0> 1 α GLYPH<8> 1 GLYPH<0> 2 { α GLYPH<0> 1 GLYPH<1> 1 e GLYPH<8> GLYPH<1> 1 { α GLYPH<1> 16 C 2 1 C 6 GLYPH<9> 1 GLYPH<0> 1 { α GLYPH<1> σ 2 m 3 ∆ 2 min GLYPH<9> 1 GLYPH<0> 1 { α So the regret caused by this term is upper bounded by :

5.3.3 Regret due to F t

This result comes from lemma 2 from [6] and is reproduced here in lemma 8. By setting glyph[epsilon1] GLYPH<16> ∆ min 4 , we have

5.3.4 Regret due to H

t

We show in lemma 9 in the appendix that P p H t q GLYPH<160> 1 t 2 and GLYPH<176> t Pr T s E r ∆ p t q 1 t H t us GLYPH<160> ∆ max π 2 6 .

5.3.5 Regret due to s F t X G t X s H t

We use lemma 4 in this appendix, and we get with C GLYPH<16> 768 , C 1 GLYPH<16> 2304 ln 2 that :

5.3.6 Putting everything together

Finally, we can put everything together and obtain the regret upper bound found in 1 with the following polynomial constant term :

The degree of this polynomial depends on 1 GLYPH<0> 1 { α GLYPH<160> 10 with α GLYPH<16> 0 . 131 . So the degrees of the polynomial in m,d, 1 { ∆ min , σ, ∆ max are respectively 30 , 10 , 20 , 20 , 1 .

6 Numerical experiments

In this section we perform numerical experiments with Beta CTS, BG-CTS and ESCB on a case where there are only two actions A GLYPH<16> A 1 , A 2 ( of size m GLYPH<16> d { 2 with A 1 GLYPH<16> p 1 , ..., 1 , 0 , ..., 0 q and A 2 GLYPH<16> p 0 , ..., 0 , 1 , ..., 1 q . This action set exhibited exponential regret in [24]. We set µ GLYPH<14> GLYPH<16> p 0 . 7 , ..., 0 . 7 , 0 . 9 , ..., 0 . 9 q with a Bernoulli distribution. The algorithm Beta CTS Uniform prior is initialized with the uniform distribution, while the Beta CTS Jeffreys is initialized with the Jeffreys prior on r 0 , 1 s , which puts more weight around the extremities of r 0 , 1 s , increasing exploration.

300000

B-CTS-Jeffreys

250000

B-CTS-Uniform

ESCB

BG-CTS

umulative

150000

100000

50000

2500

5000

7500

10000

12500

15000

17500

20000

t

106

B-CTS-Jeffreys

10

B-CTS-Uniform

ESCB

Final mean regret

BG-CTS

104

103

102

10

0

10

20

30

40

50

60

Number of arms

In the first experiment, we set a time horizon of T GLYPH<16> 2 GLYPH<2> 10 4 , and each decision has m GLYPH<16> 50 items. We run the experiment 100 times and plot the average regret over time and two empirical standard deviations in Figure 1a. The regret is nearly linear for the Beta-based Thompson samplings, whereas the subgaussian Thompson sampling and ESCB showcase regret of magnitude much lower. We set a time horizon in the second experiment T GLYPH<16> 1 GLYPH<2> 10 4 . For each decision size m P t 5 , 25 , 45 , 65 u , we run the experiments 150 times, and we plot the final regret as a function of m in Figure 1b. In the Beta-based Thompson samplings, the final regret and its variance rapidly increase with m . In comparison, BG-CTS and ESCB do not seem to be affected.

7 Conclusion

We proposed a Boosted variance Gaussian Thompson Sampling for linear combinatorial bandits (BG-CTS) and proved using novel strategies that its regret is bounded polynomially. This variant of TS far outperforms the classical TS by several orders of magnitude on a 2 decisions Bernoulli reward example.

References

Marc Abeille and Alessandro Lazaric. 'Linear thompson sampling revisited'. In: Proc. of AISTATS . 2017.

Shipra Agrawal and Navin Goyal. 'Analysis of Thompson Sampling for the Multi-armed Bandit Problem'. In: Proceedings of the 25th Annual Conference on Learning Theory . 2012.

Shipra Agrawal and Navin Goyal. 'Thompson sampling for contextual bandits with linear payoffs'. In: Proc. of ICML . PMLR. 2013.

P. Auer, N. Cesa-Bianchi, and P. Fischer. 'Finite time analysis of the multiarmed bandit problem'. In: Machine Learning 47.2-3 (2002), pp. 235-256.

O. Cappé et al. 'Kullback-Leibler Upper Confidence Bounds for Optimal Sequential Allocation'. In: Annals of Statistics 41.3 (June 2013), pp. 516-541.

Wei Chen, Yajun Wang, and Yang Yuan. 'Combinatorial Multi-Armed Bandit: General Framework and Applications'. In: Proc. of ICML . 2013.

Richard Combes, Stefan Magureanu, and Alexandre Proutiere. 'Minimal Exploration in Structured Stochastic Bandits'. In: Proc. of NeurIPS . 2017.

Richard Combes et al. 'Combinatorial Bandits Revisited'. In: Proc. of NeurIPS . 2015.

Thibaut Cuvelier, Richard Combes, and Eric Gourdin. 'Statistically Efficient, PolynomialTime Algorithms for Combinatorial Semi-Bandits'. In: Proc. ACM Meas. Anal. Comput. Syst. 5.1 (2021), 09:1-09:31.

Remy Degenne and Vianney Perchet. 'Combinatorial semi-bandit with known covariance'. In: Proc. of NeurIPS . 2016.

Aditya Gopalan, Shie Mannor, and Yishay Mansour. 'Thompson Sampling for Complex Online Problems'. In: Proc. of ICML . 2014.

Robert D. Gordon. 'Values of Mills' Ratio of Area to Bounding Ordinate and of the Normal Probability Integral for Large Values of the Argument'. In: The Annals of Mathematical Statistics (1941).

Junya Honda and Akimichi Takemura. 'An Asymptotically Optimal Bandit Algorithm for Bounded Support Models.' In: Proc. of COLT . Jan. 2010, pp. 67-79.

Emilie Kaufmann, Nathaniel Korda, and Rémi Munos. 'Thompson Sampling: An Asymptotically Optimal Finite-Time Analysis'. In: Proc. of ALT . 2012.

Junpei Komiyama, Junya Honda, and Hiroshi Nakagawa. 'Optimal regret analysis of thompson sampling in stochastic multi-armed bandit problem with multiple plays'. In: Proc. of ICML . PMLR. 2015.

Branislav Kveton et al. 'Tight Regret Bounds for Stochastic Combinatorial Semi-Bandits'. In: Proc. of AISTATS . 2015.

T.L. Lai and H. Robbins. 'Asymptotically efficient adaptive allocation rules'. In: Advances in Applied Mathematics 6.1 (1985), pp. 4-2.

Tze Leung Lai. 'Adaptive Treatment Allocation and the Multi-Armed Bandit Problem'. In: Ann. Statist. 15.3 (Sept. 1987), pp. 1091-1114.

Pierre Perrault. 'When Combinatorial Thompson Sampling meets Approximation Regret'. In: Proc. of NeurIPS . 2023.

Pierre Perrault et al. 'Statistical Efficiency of Thompson Sampling for Combinatorial SemiBandits'. In: Proc. of NeurIPS . 2020.

Daniel Russo and Benjamin Van Roy. 'An information-theoretic analysis of thompson sampling'. In: The Journal of Machine Learning Research 17.1 (2016), pp. 2442-2471.

William R. Thompson. 'On the likelihood that one unknown probability exceeds another in view of the evidence of two samples'. In: Biometrika 25.3-4 (1933), pp. 285-294.

Siwei Wang and Wei Chen. 'Thompson Sampling for Combinatorial Semi-Bandits'. In: Proc. of ICML . 2018.

Raymond Zhang and Richard Combes. 'On the Suboptimality of Thompson Sampling in High Dimensions'. In: Proc. of NeurIPS . 2021.

A Glossary

List notation related to problem setting

List notations related to the algorithm and its analysis

Notation Description

P

P

2

min

{

Polynomial term in that represents the waiting time so that the average

m,

1

∆

, σ

mean of the best action is close to its true mean

Polynomial term in m,d,

annalysis of [10]

List of events

List of the constants used in the paper

Notation Description

C

C

8

C

C

C

C

C

C

C

α

?

1

2

3

4

5

6

1

1

2

3

4

5

GLYPH<16>

GLYPH<16>

GLYPH<16>

GLYPH<16>

GLYPH<16>

6

GLYPH<16>

GLYPH<16>

1

GLYPH<16>

GLYPH<16>

2

GLYPH<0>

4

{

13

1

?

C

72

?

1

GLYPH<0>

C

.

2

3

{

2

238

8

2

{

23

C

4

{

768

constant in front of the log term in the regret bound

2304 ln

3

constant in front of loglog term in the regret bound

{

4

GLYPH<1> p

p

1

2

{

q

2

qp

C

3

q

2

GLYPH<19>

0

.

131

2

C

C

C

C

C

C

C

C

α

1

{

∆

min

, σ

replacing the exponential term in the previous

3

?

2

π

Page

List

8

9

Page

List

5

5

5

5

5

7

9

9

5

B Algorithm

Here are the algorithm for Beta Combinatorial Thompson Sampling (Beta CTS) and Boosted Gaussian Combinatorial Thompson Sampling (BG-CTS) that we use in the paper.

Algorithm 2: Boosted Gaussian Combinatorial Thompson Sampling (BG-CTS) (with improper prior)

Input: λ ¡ 0 , σ ¡ 0

1 Initialization : @ i P r d s , N i p 0 q GLYPH<16> 0 , p µ i p 0 q GLYPH<16> 0 Uniform improper distribution on R d , select decisions until min i Pr d s N i p t q ¡ 0 . Update @ i P r d s , N i p 0 q , µ i p 0 q accordingly. Generate @ t P r T s , @ i P r d s , Z i p t q GLYPH<18> N p 0 , 1 q i.i.d.

2 for t GLYPH<16> 1 , ..., T do

3 Compute θ i p t q GLYPH<16> p µ i p t GLYPH<1> 1 q GLYPH<0> σ b 2 g p t q N i p t GLYPH<1> 1 q Z i p t q .

4 Compute A p t q GLYPH<16> arg max A P A t A T θ p t qu .

5 The environment draws @ i P r d s , X i p t q .

6 Observe X p t q d A p t q , Receive reward A p t q T X p t q .

7 Update @ i P A p t q , N i p t q GLYPH<16> N i p t GLYPH<1> 1 q GLYPH<0> 1 , p µ i p t q GLYPH<16> N i p t qGLYPH<1> 1 N i p t q p µ i p t GLYPH<1> 1 q GLYPH<0> X i p t q N i p t q .

8 end

C Proofs of main results

Here in 2 is the diagram of the regret decomposition on the complete event system in green. In red is the novel part of the proof that we introduce. It replaces the step 4 of the proof in [20] wich was inspired by [23] who we think addapted ideas from [2] and [14]. We, in some sense, rediscovered those ideas for the case of combinatorial bandits by controling the numbre of times the optimal action is played

And therefore (This is lemma 4 from [19]):

And

, o²d² ln(2)n(1 +e)

Section 5.3.3

Regret

心

(Lemma 4 Perrault 2022)

Section 5.3.5

Regret

min

Section 5.3.4

%

16

Regret

(Lemma 9)

max max {Pi (m, d, 1/△min, 0), P2 (m, 1/△min, o)}

(Proposition3and lemma8)

Section 5.3.2

Regret

(Proposition 2)

With probability

( usooi)

4dt-2+t-lm(t)+ e-Gda

Yes?Withprobability

(ogiodo)

Proposition 2

Regret

(Lemma 9)

Section 5.3.1

Proof. G t implies that we have

Then s F t respectively s H t imply that GLYPH<176> i P A p t q p p µ i p t q GLYPH<1> µ GLYPH<14> i q GLYPH<160> ∆ t 4 respectively GLYPH<176> i P A p t q p θ i p t q GLYPH<1> p µ i p t qq GLYPH<160>

σ b 8 ˜ f p t q A J p t q V p t q A p t q . Therefore, we have that :

Then because ˜ f is increasing, we can use lemma 4 from [19] with β i,T GLYPH<16> 64 σ 2 ˜ f p T q . And we have that @ i, p i GLYPH<16> 1 because we are not working on triggering bandits. ( β i,T and p i are from their notation, We can also notice that because each arm is played at least once in our algorithm we do not have the first term in d ∆ max that is counted elsewhere) We have that :

To make the formula more readable we use that for m ¡ 2 we have that 5 ln m ¡ 3 so that :

By definition of ˜ f , we have that C GLYPH<16> 768 . And assuming m glyph[greaterorequalslant] 1 , d glyph[greaterorequalslant] 3 we have that p m GLYPH<0> 2 qp 1 GLYPH<0> d ln 2 q GLYPH<160> 3 dm ln 2 so C 1 can be taken as 2304 ln 2 and the last constant is 1152 ln p 2 q .

D Concentration results

Lemma 5. For t glyph[greaterorequalslant] 2 , let λ ¡ 0 , let δ t ¡ 0 . Let f p ˜ δ t q : GLYPH<16> ln p 1 δ t q GLYPH<0> m ln ln t GLYPH<0> m 2 ln GLYPH<0> 1 GLYPH<0> e λ GLYPH<8> . Then the event D t GLYPH<16> t max s Pr t s U GLYPH<14> p s q ¥ σ a 2 p 1 GLYPH<0> λ q f p δ t qu happen with probability P p D t q GLYPH<160> δ t

Proof. See lemma 3 of [10]. Moreover, notice that in their lemma 6, they use a union bound on all the possible values of p N i q i P A GLYPH<14> P r t s m to get their result. (This is symbolized by their set D a )

Lemma 6 (Multiplicative Azuma Chernoff) . Let p W t q t P N GLYPH<14> be a sum of random variable : @ t P N GLYPH<14> , W t GLYPH<16> t GLYPH<176> s GLYPH<16> 1 X s Where p X t q t P N GLYPH<14> verify that there exist p p t q t P N Ps 0 , 1 r such that :

And p H p t qq t P N GLYPH<14> is a filtration where @ t P N GLYPH<14> , X t is H p t q measurable. We note m t GLYPH<16> t GLYPH<176> t GLYPH<16> 1 p t 1 .

Let t P N GLYPH<14> , @ δ ¡ 0 we have that :

Proof. For the first inequality , let t P N GLYPH<14> , λ, x P R GLYPH<0> , we have by Markov inequality:

Let s P r t s , by definition of X s and using that @ x P R , exp p x q glyph[greaterorequalslant] 1 GLYPH<0> x we have that :

Then, by the tower property of the expectation and induction:

Combining with the first equation, we have :

This is true for all x, λ P R GLYPH<0> so by setting λ GLYPH<16> ln p 1 GLYPH<0> δ q and x GLYPH<16> p 1 GLYPH<0> δ q m t we have :

Because @ δ ¡ 0 , δ GLYPH<1> p 1 GLYPH<0> δ q ln p 1 GLYPH<0> δ q GLYPH<160> 2 GLYPH<0> δ .

For the second inequality , let λ P R GLYPH<0> :

Let s P r t s , by definition of X s we have that :

and @ 1 ¡ δ glyph[greaterorequalslant] 0

Then, by the tower property of the expectation and induction:

Combining with the first equation, we have :

This is true for all x, λ P R GLYPH<0> so by setting λ GLYPH<16> GLYPH<1> ln p 1 GLYPH<1> δ q and x GLYPH<16> p 1 GLYPH<1> δ q m t we have :

GLYPH<1>

Lemma 7. Let t P r T s , let us consider the event :

We have that P p B p t qq GLYPH<160> d t 2 . This result can be in part found in [16] in their proof of lemma 1.

Proof. We control it with the deviation of individual arms.

We have by a double union bound, and because the rewards are σ 2 subgaussian using Hoeffding:

We have that B p t q GLYPH<128> GLYPH<148> i P d M i p t q . So by union bound P p B p t qq GLYPH<160> 2 d t 2 .

Lemma 8. Let i P r d s , t P r T s , let us consider the event :

We have :

And :

This is a result from [6] lemma 2 adapted to the σ 2 subgaussian case.

GLYPH<9>

Proof. We use a union bound and Hoeffding's inequality for σ 2 subgaussian random variable. Let i P r d s ,

Then decomposing F p t q with a union bound of the i P A p t q , p µ i p t q GLYPH<1> µ GLYPH<14> i ¡ ∆ min 2 m GLYPH<1> glyph[epsilon1] m ( and summing over i P r d s we get the second result.

Lemma 9. Let t P r T s we have :

Which implies that P p H p t q| H p t qq GLYPH<160> 1 t 2 .

Defining :

we have that g p t q ln p| A | t q GLYPH<160> ˜ f p t q and thus:

Finally:

Proof. Let c ¡ 0 by union bound we have :

By setting c GLYPH<16> a 4 ln p| A | t q ¡ 1 we have | A | exp GLYPH<1> GLYPH<1> c 2 2 GLYPH<9> GLYPH<16> 1 t 2 hence the first result.

Then we have that : ln p| A |q glyph[lessorequalslant] d ln p 2 q so that :

Hence, the second result. And by summing over t :

We recall simple tail bounds for Gaussian random variables. (see for instance [12])

Lemma 10. Consider Z GLYPH<18> N p 0 , 1 q , then we have

It is also noted that by symmetry P p| Z | ¥ x q GLYPH<16> 2 Q p x q .

furthermore for all x ¥ 0 :

As well as:

E Functions Study

Lemma f. 1. Let f be the following function :

We have

Proof.

Using f. 2 we have that :

Using f. 3 we have that for t ¡ 23 , @ m glyph[greaterorequalslant] 1

But thanks to f. 4, the right-hand side is decreasing in t , so :

Lemma f. 2. Let a glyph[greaterorequalslant] b glyph[greaterorequalslant] 0 and let g : R GLYPH<0> GLYPH<209> R GLYPH<0> be the following function :

is decreasing in m for t ¡ 23 .

Which means that for t ¡ 23 , for @ m glyph[greaterorequalslant] 1 :

Then @ t glyph[greaterorequalslant] 0 , g p t q GLYPH<160> g p 0 q GLYPH<160> a b

Proof. Let t P R GLYPH<0> , we have :

So g is decreasing in t .

Proof. Let l p m q GLYPH<16> a GLYPH<0> mb c GLYPH<0> md , its derivative is :

So l 1 p m q ¡ 0 i.i.f. bc GLYPH<1> ad ¡ 0 . Furthermore, ln ln t ln p t 2 q GLYPH<1> ln ln p t 2 q ln t GLYPH<160> 0 for t ¡ 23 So @ t ¡ 23 , f p t, m q is decreasing in m and @ t ¡ 23 , f p t, m q GLYPH<160> f p t, 1 q .

is decreasing.

Proof. We differentiate, and we obtain

We expend and simplify :

We have :

and and

Therefore @ t ¡ 3 , f 1 p t q GLYPH<160> 0 and the function f is decreasing.

Lemma f. 5. Let c ¡ 1 be a positive constant and let f be the following function :

Then for t ¡ 2 e we have

Proof. For t ¡ 2 e we have that 1 GLYPH<160> c 2 ln p t { 2 q so :

For t ¡ 2 e we have that 1 a ln p t { 2 q GLYPH<16> exp GLYPH<0> GLYPH<1> 1 2 ln ln p t { 2 q GLYPH<8> ¡ exp GLYPH<0> GLYPH<1> 1 4 ln p t { 2 q GLYPH<8> . Therefore :

Lemma f. 6.

Lemma f. 7. Let f, g the following function :

For t ¡ e and t ¡ 1 GLYPH<0> e λ we have ln ln t ln t GLYPH<160> 1 2 thus and

Proof. For t ¡ e, ln t ¡ ln ln t and for t ¡ 1 GLYPH<0> e λ , ln t ¡ ln p 1 GLYPH<0> e λ q thus the result

Lemma f. 8. Let α, c P R GLYPH<14> GLYPH<0> , two strictly positive constants such that c α ¡ 1 and define the function f p t q : GLYPH<16> t α GLYPH<1> c ln t . We have that @ t ¡ GLYPH<1> c α ln c α 1 GLYPH<1> 1 e GLYPH<9> 1 α , f p t q ¡ 0 . Furthermore, we have : GLYPH<1> c α ln c α 1 GLYPH<1> 1 e GLYPH<9> 1 α GLYPH<160> GLYPH<0> 1 α GLYPH<8> 1 GLYPH<0> 2 α GLYPH<0> 1 GLYPH<1> 1 e GLYPH<8> GLYPH<1> 1 α c 1 GLYPH<0> 1 α

Proof. Let's study the function G p T q GLYPH<16> T GLYPH<1> c ln GLYPH<1> T 1 α GLYPH<9> GLYPH<16> T GLYPH<1> c α ln p T q . For T glyph[greaterorequalslant] c α e by using the concavity of the logarithm and differentiating T GLYPH<222>GLYPH<209> c α ln p T q at the point c α e we have that :

Then for T glyph[greaterorequalslant] c α e

So for T ¡ c α ln p c α q 1 GLYPH<1> 1 e , T GLYPH<1> c α ln p T q ¡ 0 . Which means that by a change of variable that for t α ¡ c α ln p c α q 1 GLYPH<1> 1 e GLYPH<240>æ t ¡ p c α ln c α 1 GLYPH<1> 1 e q 1 α we have that f p t q ¡ 0 . Hence, the first result.

We have using that ln p x q GLYPH<160> x :

Finally, combined with the rest, we get the second result.

Lemma f. 9. Let c ¡ 0 , α ¡ 0 the series GLYPH<176> J t GLYPH<16> 1 exp pGLYPH<1> ct α q converges when T GLYPH<209> GLYPH<0>8 and :

Proof. Because the function t GLYPH<222>GLYPH<209> exp pGLYPH<1> ct α q is decreasing, by an integral test for convergence, we have that :

A primitive of the function : t GLYPH<222>GLYPH<209> exp pGLYPH<1> ct α q is T GLYPH<222>GLYPH<209> c GLYPH<1> 1 α α GLYPH<0>8 ‡ cT α t 1 α GLYPH<1> 1 e GLYPH<1> t dt . So :

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We introduced the setting and the assumption needed so that our main result is applicable. We tried not to overstate our results and be as close as possible of our theoretical results.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: As a theoretical paper, we only claim that our result is true under some assumptions. We discussed the computational complexity of our method under the assumption that a certain linear combinatorial problem can be solved efficiently.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We clearly stated the assumptions needed for our main result to hold. We provided complete proof of our results in the appendix with its lemmas. And we refer to previous works by citing them when needed.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: While this is not an experimental-focused paper, we provided all the details needed to reproduce the plots found section 6: The parameters used, the number of arms, and the number of runs.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The paper generates synthetic data of the bandit model considered. The code is available at https://github.com/RaymZhang/CTS-Mismatched-Paradox

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so No is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The parameters of the algorithm and the environment are provided.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We plot the confidence intervals of the regret in the simulation section, with 2 standard deviations error intervals.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [No]

Justification: Because of lack of space. Here they are: the experiments were run on a single core of i9 (i9-12900H) laptop computer with 32GB of RAM with python 3. The experiments took less than 1 hour to run.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We don't think that our research violates the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA] .

Justification: The work is for the moment very theoretical. It could be somehow used in some optimization algorithms but are far from real life applications.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA] .

Justification: This is theoretical work and no data or model was released.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer:[NA]

Justification: Apart from python commonly available libraries, no asset was used.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA] .

Justification: No new assets were introduced.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA] .

Justification: Synthetic data was used.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA] .

Justification: No human data used.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores CTS for the linear combinatorial semi-bandit problem with subgaussian rewards. It introduces a novel TS algorithm that avoids exponential regret scaling with problem dimensionality. Theoretical bounds and experiments are provided.\n\nSTRENGTHS:\nThe paper addresses a significant limitation in existing CTS algorithms by avoiding exponential regret scaling with problem dimensionality, which is a notable contribution to the field. It establishes new regret bounds that improve upon previous results in terms of finite-time performance. This is further shown experimentally. \n\nThe paper is well-organized and clearly presents its ideas.\n\nWEAKNESSES:\nThe paper places significant emphasis on the \"mismatched sampling paradox,\" showcasing situations where a divergence between assumed and actual reward distributions can unexpectedly improve performance. While intriguing, this phenomenon isn't entirely novel, as it's known that Thompson Sampling (TS) can adapt to various posterior distributions. The paper's focus on this paradox, including its prominence in the title, may overshadow what could be considered the core contribution: a little boost on the exploration of CTS to better control the regret constant term.\n\nThe paper does not sufficiently clarify how the exploration boost is integrated into the analysis. The proof strategy is not well synthesized.\n\n[minor] Your main term scale with log(m)^2, but it is now possible to get a log(m) term instead. For example, use Lemma 4 in https://arxiv.org/pdf/2302.11182. I think this could be good to at least cite this paper mentioning that this could be done. line 203-204, I think the event D_t should contains the norm infty.\n\nQUESTIONS:\nCan you explain better what makes your analysis work exactly ? What are the proof techniques that you used, and how this differs from [18] ? I put a good score for this paper as I think it worth it. But I can consider putting a lower score if a not satisfying answer is given to the questions.\n\nLIMITATIONS:\nThe analysis only work for linear reward. Can you discuss why this is the case and how would it be possible to extend it the more general cases?", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses Thompson Sampling for stochastic combinatorial bandits with sub-Gaussian rewards. In this area, previous work has identified the interesting phenomenon that some versions of Thompson sampling incur a per-instance regret which is exponential in the (maximum) number of arms pulled per round, m. This is not experienced by UCB-style algorithms, although those tend to have a worse computational complexity.\n\nThis paper furthers the discussion around the performance of TS in combinatorial bandits by identifying a variant which incurs only polynomial dependence on m. This is achieved by using a distribution with inflated variance to draw Thompson samples, rather than the 'true' posterior.\n\nThe paper contains non-trivial theoretical work to derive this bound, and contains an experiment demonstrating the superiority of the new TS approach over 'natural' variants and its comparable performance to ESCB, a state-of-the-art UCB-based algorithm.\n\nSTRENGTHS:\nI find the research questions considered in this paper very interesting, and think that the multi-armed bandits community at NeurIPS will value this work. Thompson Sampling is a very popular algorithm for all kinds of bandit problems, and the fact that its most natural extension to combinatorial bandits has poor performance is an important issue. The theoretical results in this paper meaningfully contribute to the understanding of where 'natural' Thompson sampling is necessary and where it is only 'some notion of adding randomness' that is needed.\n\nThe theoretical work is based on state-of-the-art tools, does include some novel steps and is likely to be of its own interest. It does not appear to be a routine translation of existing tools.\n\nThe paper is mostly well-written and clear, and for someone acquainted with combinatorial bandits it is quite easy to follow.\n\nWEAKNESSES:\nThe main weakness of the paper is that more could be done to articulate exactly where the results improve over existing bounds. The exponential and polynomial dependence on m is, as the paper states, in the constant order (wrt T) term. The main comparison between the results of the present paper and [18] is in terms of order results, and some constants are potentially large. It leaves the reader uncertain as to where in the space of T, m and d, the bound of this paper improves upon the bound of [18]. Adding some clarity around this would improve the paper, and help to establish the extent of its contribution.\n\nThere are some further points where I feel clarification could be made around theoretical and experimental results. I ask questions about these in the section below, and they are also central to me assessing the extent of the contribution.\n\nThere are some parts where the explanations are probably a bit too brief to be accessible to those without expert knowledge of the field - e.g. lines 52-64 discussing related literature use undefined technical terminology and discuss papers very quickly.\n\nQUESTIONS:\n1. For which values of m and T specifically does the bound of this paper improve upon the bound in [18]? I appreciate this is problem dependent, but if this could at least be answered for the example considered in the experiments, that would be beneficial.\n\n2. The situation in the experiments seems as though it could also be tackled by a 2-armed bandit policy that receives full-bandit feedback on [0,d/2] and essentially ignores the combinatorial structure. If those rewards were scaled to [0,1] would the combinatorial algorithms be outperformed by algorithms for 2-armed bandits?\n\n3. When introducing B-CTS and G-CTS could you make clear if these are the variants to which the results of [18] and [22] apply?\n\n4. The conclusion that using a mismatched variance term outperforms the natural variance is an interesting one, and seems to mirror some findings in the bandit literature that randomisation strategies that are not TS-like or based on a posterior distribution (explicitly) can perform well in bandit problems (e.g. Perturbed History Exploration and bootstrapping based approaches, Kveton et al. (2019) and subsequent work). Can you comment on whether this polynomial dependence relies on using something close to the posterior, or whether other randomisation strategies would seem to be sensible candidates?\n\n5. Can you add to section 5 some clarity on what steps are novel and what are inspired by previous theoretical work? Some of the construction of a clear run seems to bear resemblance to (now) classic proofs for Thompson sampling (Agrawal and Goyal (2012), Kaufmann et al (2012), etc.) and the idea of using sample paths may have roots in other work too? (I'm not certain on that)\n\n\n\nAgrawal and Goyal (2012) http://proceedings.mlr.press/v23/agrawal12/agrawal12.pdf\nKaufmann et al (2012) https://arxiv.org/pdf/1205.4217\nKveton, Szepesvari, et al. (2019) https://arxiv.org/abs/1902.10089\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-03"}

comments: SUMMARY:
The authors present a new Thompson Sampling algorithm for linear combinatorial stochastic semi-bandits. The algorithm provably achieves a better finite-time regret than previous works, and specifically without an exponential dependency on the dimension of the problem.  The authors also present a "paradox" that shows using posterior knowledge is not always beneficial.

STRENGTHS:
* The paper is presented clearly and the theory is sound.
* The experiments suggest the algorithm is useful in practice.
* The presented paradox is original and interesting

WEAKNESSES:
* The main result improves the known regret only for the term that does not depend on the time horizon, which is usually less interesting.
* Code is not provided for the numerical experiments.

QUESTIONS:
* In line 26, do you mean $X(t)$ is uniformly distributed?
* Can you provide summary of the algorithms as figures? It is hard to follow in the text what is the exact algorithm
* Can you provide any intuitive explanation for the presented "paradox"?

LIMITATIONS:
The authors address the limitations of the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-07

comments: SUMMARY:
This paper proposes a modified version of posterior sampling that achieves optimal asymptotic regret bound, providing an algorithm through the methodology of Thompson Sampling that achieves such a bound.

STRENGTHS:
This is a technical paper, and the message is clear and intriguing. This paper validates the methodology of Thompson Sampling in the context of combinatorial bandits in terms of achieving polynomial dependence on the number of dimensions. The technique is novel, to my knowledge.

WEAKNESSES:
I am exactly doing this line of research, so I am not able to follow all the proof details. The paper is notation-heavy; for instance, there are nine events defined, eight of which are some sort of deviations. It is hard to parse and keep track of for a person outside the exact line of research.  I appreciate the authors' effort to explain, but I do believe in the main paper, the authors should write a more sketchy proof to highlight the key technique to prove novelty than writing a semi-rigorous proof: for instance, out of the eight deviation events, which are standard and bounded by conventional results? which are novel contributions? Of all the techniques involved, which is the key to this new result? In all, I feel it would be of great help if a clear logic line for the proof is demonstrated for people outside this line of research. At the current state, it is rather hard for me to evaluate the technical contribution.

QUESTIONS:
What is the guarantee for G-CTS?

LIMITATIONS:
adequately addressed
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-10