Title: Ensemble sampling for linear bandits: small ensembles suffice
Venue: NeurIPS 2024
Accepted: yes
Keywords: linear

bandits

ensemble sampling
Abstract: We provide the first useful and rigorous analysis of ensemble sampling for the stochastic linear bandit setting. In particular, we show that, under standard assumptions, for a $d$-dimensional stochastic linear bandit with an interaction horizon $T$, ensemble sampling with an ensemble of size of order $\smash{d \log T}$ incurs regret at most of the order $\smash{(d \log T)^{5/2} \sqrt{T}}$. Ours is the first result in any structured setting not to require the size of the ensemble to scale linearly with $T$---which defeats the purpose of ensemble sampling---while obtaining  near $\smash{\sqrt{T}}$ order regret. Our result is also the first to allow for infinite action sets.

Ensemble sampling for linear bandits: small ensembles suffice

Abstract

We provide the first useful and rigorous analysis of ensemble sampling for the stochastic linear bandit setting. In particular, we show that, under standard assumptions, for a d -dimensional stochastic linear bandit with an interaction horizon T , ensemble sampling with an ensemble of size of order d log T incurs regret at most of the order ( d log T ) 5 / 2 √ T . Ours is the first result in any structured setting not to require the size of the ensemble to scale linearly with T -which defeats the purpose of ensemble sampling-while obtaining near √ T order regret. Our result is also the first to allow for infinite action sets.

1 Introduction

Ensemble sampling (Lu and Van Roy, 2017) is a family of randomised algorithms for balancing exploration and exploitation within sequential decision-making tasks. These algorithms maintain an ensemble of perturbed models of the value of the available actions and, in each step, select an action that has the highest value according to a model chosen uniformly at random from the ensemble. The ensemble is then incrementally updated the new observations.

Ensemble sampling was introduced as an alternative to Thompson sampling (Thompson, 1933) that is tractable whenever incremental model updates are cheap (Osband et al., 2016; Lu and Van Roy, 2017). It is thus particularly popular in deep reinforcement learning, where the models (neural networks) are trained via gradient descent in an incremental fashion. Therein, ensemble sampling features directly in algorithms such as bootstrapped DQN and ensemble+ (Osband et al., 2016, 2018), as part of other methods (Dimakopoulou and Van Roy, 2018; Curi et al., 2020), and as the motivation for methods, including random network distillation (Burda et al., 2018) and hypermodels for exploration (Dwaracherla et al., 2020). Ensemble sampling has also been applied to online recommendation systems (Lu et al., 2018; Hao et al., 2020; Zhu and Van Roy, 2021), the behavioural sciences (Eckles and Kaptein, 2019) and to marketing (Yang et al., 2020).

Despite its practicality and simple nature, ensemble sampling has thus far resisted analysis. Indeed, Qin et al. (2022), who showed a bound of order √ T on its Bayesian regret under the impractical condition that the size of the ensemble scales at least linearly with the horizon T , summarise the state-of-the-art in this regard as follows:

'A lot of work has attempted to analyze ensemble sampling, but none of them has been successful. '

Readers familiar with the literature know that the analysis of randomised exploration methods, such as Thompson sampling, or perturbed history exploration (Kveton et al., 2020; Janz et al., 2024), relies on showing that at each step there is a constant probability of choosing a model that overestimates the true value (Agrawal and Goyal, 2013; Abeille and Lazaric, 2017). For these methods, the analysis

is relatively simple, because the algorithms can be described as first fitting a model to the past data, and then using a fresh source of randomness to perturb the model, which is then used to derive the action to be used. Ensemble sampling does not fit this pattern: here, the distribution of models given the past is controlled only implicitly, making the analysis challenging.

Our contribution is a guarantee that (a symmetrised version of) ensemble sampling, given an ensemble size logarithmic in T and linear in the number of features d , incurs regret no worse than order ( d log T ) 5 / 2 √ T . This is the first successful analysis of ensemble sampling in the stochastic linear bandit setting, or indeed, any structured setting (see Remarks 6 and 7 for a discussion of this claim). Our result is based on (a slight extension of) the now-standard framework of Abeille and Lazaric (2017), and as such it ought to be possible to extend it to the usual settings: generalised linear bandits (Filippi et al., 2010), kernelised bandits (Srinivas et al., 2010), deep learning (via the neural tangent kernel, per Jacot et al., 2018) and reinforcement learning (following the work of Zanette et al., 2020).

2 Linear ensemble sampling

We now outline our problem setting, a version of the linear ensemble sampling algorithm and our main result, and discuss these in the context of prior literature. We encourage readers less familiar with the motivation around ensemble sampling to consult Lu and Van Roy (2017) or Osband et al. (2019); our focus will be on analysis. We will use the following notation:

Sets and real numbers We write N + = { 1 , 2 , . . . } , N = N + ∪ { 0 } and [ n ] = { 1 , 2 , . . . , n } . We use the shorthand ( a t ) t for a sequence indexed by t in N or N + when this index set can be deduced from the context. For a, b ∈ R , that is, for real numbers a and b , a ∧ b denotes their minimum and a ∨ b their maximum.

Vectors and matrices We identify vectors and linear operators with matrices. We write ∥ · ∥ 2 for the 2-norm of a vector, and ∥ · ∥ for the operator norm of a matrix corresponding to the largest singular value, which we denote by s 1 ( · ) ; we write s 2 ( · ) , s 3 ( · ) , . . . for the remaining singular values, ordered descendingly. For vectors u, v of matching dimension, ⟨ u, v ⟩ = u T v denotes their usual inner product. We write B d 2 for the 2-ball in R d , S d -1 2 = ∂ B d 2 for its surface, the ( d -1) -sphere, and H u for the closed half-space { v ∈ R d : ⟨ u, v ⟩ ≥ 1 } .

Probability We work on a probability space with probability measure P , and denote the corresponding expectation operator by E . We write σ ( · ) for the σ -algebra generated by the argument. For an event A , we write 1 [ A ] for the indicator function of A .

Uniform distribution We write U ( B ) for the uniform distribution over a set B (when well-defined), and U ( B ) ⊗ m for its m -fold outer product; that is, ( U 1 , . . . , U m ) ∼ U ( B ) ⊗ m are m i.i.d. random variables with common law U ( B ) .

2.1 Problem setting: stochastic linear bandits

We consider the standard stochastic linear bandit setting. At each step t ∈ N + , a learner selects an action X t from an action set X , a compact subset of B d 2 , and receives a random reward Y t ∈ R that obeys the following assumption:

Assumption 1. At each time step t ∈ N + , the agent receives a reward of the form Y t = ⟨ X t , θ ⋆ ⟩ + Z t with θ ⋆ ∈ B d 2 a fixed instance parameter and Z t a random variable satisfying

almost surely, where F t -1 = σ ( X 1 , Y 1 , . . . , X t -1 , Y t -1 ) and A t is the σ -algebra generated by the random variables used by the learner to select its actions up to and including time t .

The learner is given a horizon T ∈ N + and its performance is evaluated by the (pseudo-)regret, R ( T ) , incurred for the first T steps, defined as

The smaller the regret, the better the learner. Under our assumptions, up to logarithmic factors, the regret of the best learners is of order d √ T (Chapters 19-24, Lattimore and Szepesvári, 2020). Our

main result will show that if a learner follows the upcoming ensemble sampling algorithm to select the actions X 1 , . . . , X T , the regret it incurs will satisfy a similar high probability bound, with a slightly worse dependence on the dimension d .

2.2 Algorithm: linear ensemble sampling

Linear ensemble sampling, listed as Algorithm 1, proceeds as follows. At the outset, we fix an ensemble size m ∈ N + , a regularisation parameter λ > 0 and a sequence of perturbation scale parameters r 0 , r 1 , r 2 , . . . > 0 . Then, before the start of each round t ∈ N + , linear ensemble sampling computes m +1 d -dimensional vectors. The first of these is the usual ridge regression estimate of θ ⋆ ,

The remaining m parameter vectors, which shall serve as perturbations, are of the form

where S j 0 ∈ R d is a random initialisation , and is taken to be uniform on λ √ d S d -1 2 , and U j 1 , U j 2 , . . . are random targets , uniform on the interval [ -1 , 1] . All of these random variables are independent of the past at the time they are sampled and across the m -many replications. The algorithm then selects a random index J t ∈ [ m ] and sign ξ t ∈ {± 1 } , both uniformly distributed over their respective ranges, computes the perturbed parameter

with ties dealt with in a measurable way. All in all, the vectors

serve as an ensemble of 2 m -many perturbed estimates of θ ⋆ , and in each round the algorithm acts optimally with respect to one of these, selected at uniformly at random.

Our linear ensemble sampling algorithm deviates in two ways from that of Lu and Van Roy (2017):

The random sequence of targets used to fit the ensembles in our algorithm consists of uniform random variables rather than the Gaussians used in the previous literature. This choice simplifies the proofs, but our results hold (up to constant factors) for any suitable subgaussian targets, including Gaussian-see Remark 14 for a sketch of the argument.

We symmetrise our perturbations using Rademacher random variables, which does not feature in previous formulations of the ensemble sampling algorithm. This helps to create a more uniform distribution of the perturbations ( ˜ θ j t ) t around zero with minimal computational overhead, and is important for our proof technique.

In Appendix A, we provide reformulation of our linear ensemble sampling algorithm that is more in line with the style of presentation of the algorithm given by Lu and Van Roy (2017).

2.3 Regret bound for linear ensemble sampling

Our advertised result is captured in the following theorem. To state the theorem we need the sequence

Here and in related quantities, the superscripted δ expresses the dependence on a δ ∈ (0 , 1] .

and selects an action

Algorithm 1 Linear ensemble sampling

Theorem 1. Fix δ ∈ (0 , 1] and take r t = 7 β δ t for all t ∈ N , λ ≥ 5 and m ≥ 400 log( NT/δ ) for N = (134 √ 1 + T/λ ) d . Then there exists a universal constant C > 0 such that, with probability at least 1 -δ , the regret incurred by a learner following linear ensemble sampling with these parameters in our stochastic linear bandit setting (formalised in Assumption 1) is bounded as

Remark 1. If δ ≥ 1 /T α for some α > 0 and we take λ = 5 and m to be as small as possible given the constraint of Theorem 1, then

for some constants C α , C ′ α > 0 that depend on α only and where the bound on the regret holds with probability 1 -δ . That is, for polynomially sized confidence parameters, the ensemble size scales linearly with d log( T ) , while the regret scales with d 5 / 2 √ T up to logarithmic-inT factors. The latter scaling is slightly worse than that obtained for Thompson sampling (cf. Theorem 17), where the regret scales with d 3 / 2 √ T , again, up to logarithmic-inT factors.

Remark 2. Theorem 1 does not recover the expected behaviour for large ensemble sizes; that is, as m →∞ . On one hand, this is not an issue: we are interested in the practical regime where the ensemble size is small. On the other, it suggests that better analysis might be possible. In Remark 12, we discuss a potential looseness in our analysis, which, if addressed, would result in the removal of a factor of m from the bound, thus bringing the regret in line with that of Thompson sampling. The technique that would be required to do so could then also be used to change the remaining √ m dependence to an order √ d log T/δ dependence, thus recovering a bound with the right dependence on m . However, as discussed in Remark 12, such improvements, if possible, may be hard to attain.

Remark 3. Our ensemble sampling algorithm requires the ensemble size m to be fixed in advance. As m depends on the horizon T , the method only provides guarantees for a fixed, finite horizon T , and our regret bound has a direct dependence on T . The doubling trick would be a simple, but somewhat unsatisfactory way of removing this dependence. An alternative is to grow the ensemble online. However, a naive implementation of growing the ensemble size would require storing all past observations and computation per time step also growing with time, which is counter to the idea of a fast incremental method.

Remark 4. In light of Theorem 1, linear ensemble sampling might be seen as a less effective and less computationally efficient version of linear Thompson sampling. This is correct: in the linear setting, where Thompson sampling may be implemented in closed form, ensemble sampling has no clear advantages. With that said, variants of ensemble sampling and related bootstrapping methods are popular in more complex settings where closed forms are not available (say, deep reinforcement learning, per Osband et al., 2016, 2018), and the linear setting provides an important testing ground for the soundness of these algorithms.

Remark 5. Instead of using a randomly sampled sign and index ( ξ t , J t ) , we could use the upperconfidence-bound strategy of selecting the actions using a maximum over the ensemble, that is

It is immediate from the proof of Theorem 1 that this would yield a regret bound scaling with d 3 / 2 . This raises the question of whether this max-over-ensemble approach is actually superior, or whether it's just that its much simpler analysis more readily yields a good bound.

2.4 Comparison to related results

We now discuss the results of Lu and Van Roy (2017), Qin et al. (2022), and Lee and Oh (2024), showing that our result is the first to begin justifying the practical effectiveness of ensemble sampling.

Remark 6. The work of Lu and Van Roy (2017) makes strong claims on the frequentist regret of linear ensemble sampling. However, as pointed out by Qin et al. (2022), and confirmed by Lu and Van Roy (2017) in their updated arxiv manuscript, the analysis of Lu and Van Roy (2017) is flawed.

Remark 7. The only correct result on the regret of linear ensemble sampling is by Qin et al. (2022), which gives that for a d -dimensional linear bandit with an action set X of cardinality K , the Bayesian regret incurred-that is, average regret for θ ⋆ ∼ N (0 , I d ) -is bounded as

for some universal constant C > 0 . Observe that this bound needs an ensemble size linear in T for Bayesian regret that scales as √ T (up to constant and polylogarithmic factors), which largely defeats the purpose of ensemble sampling. Furthermore, the ensemble size m needs to scale linearly with K to get a log K overall dependence on K . Therefore, to tackle a bandit with X = B d 2 using the bound of Qin et al. (2022) and discretisation, since order 2 d -1 -many actions would be needed to guarantee a small discretisation error, the ensemble size m would need to scale exponentially in d .

Remark 8. Building on our result, Lee and Oh (2024) have shown that for a K -armed linear bandit, an ensemble of size on the order of K log T suffices to ensure a bound on R ( T ) on the order of d 3 / 2 √ T , ignoring logarithmic-inT factors. This gives a regret bound tighter by a factor of d relative to ours, but at the cost of replacing the dimension d by the number of arms K in the ensemble size. Recalling that, K can be exponential in d (per Remark 12), their result may require rather large ensemble sizes. Lee and Oh (2024) also contribute some rather curious remarks about our proofs.

3 Analysis of linear ensemble sampling

Our analysis of ensemble sampling will be based on a master theorem , presented in Section 3.1 and proven in Appendix B. The master theorem provides a method for obtaining regret bound for Thompson-sampling-like randomised algorithms. Thereafter, in Section 3.2, we apply the said master theorem to linear ensemble sampling, using some intermediate results proven in Sections 3.3 and 3.4, and some routine calculations that have been deferred to the appendix. Additionally, in Appendix D, we validate our master theorem by demonstrating that it recovers a previous result for a more classical Thompson-sampling-type algorithm.

3.1 Master Theorem: a regret bound for optimistic randomised algorithms

Our analysis of ensemble sampling will rely on the revered principle of optimism . To make this precise, consider a fixed instance parameter θ ⋆ ∈ R d . Writing J ( θ ) = max x ∈X ⟨ x, θ ⟩ , we call

the set of parameters optimistic for θ ⋆ . We now present a 'master theorem' that bounds the regret of any algorithm that chooses actions based on a randomly chosen parameter in terms of the probabilities that, given the past, the algorithm samples a parameter that falls in the intersection of ellipsoidal confidence sets and the optimistic region Θ OPT . This result generalises a similar theorem stated for Thompson sampling by Abeille and Lazaric (2017), extending it to allow for finitely supported perturbations (critical for the proof of Theorem 1) and for a finer control over the way dependencies between time-steps are handled.

Theorem 2 (Master regret bound) . Fix T ∈ N + ∪ { + ∞} , δ ∈ (0 , 1] , λ ≥ 1 , and let ( V t , ˆ θ t ) t be defined per equation (1) . Let Assumption 1 hold, recalling the filtrations ( F t ) t and ( A t ) t defined therein, and let ( A ′ t ) t be any filtration satisfying

Let ( b t ) t ∈ N be a sequence of σ ( F t ∪ A ′ t ) t -adapted nonnegative random variables, and define the ellipsoids

Let ( θ t ) t be a ( σ ( F t ∪ A t )) t -adapted R d -valued sequence and suppose that

Suppose further that there exist events E T and E ⋆ T satisfying

Then, writing

we have that on a subset of E T ∩ E ⋆ T of probability at least 1 -3 δ , for all τ ∈ [ T ] ,

We defer the proof of Theorem 2 to Appendix B.

Remark 9. To apply the theorem, we need to show that our algorithm generates ( θ t ) t such that the probability of each θ t landing in Θ OPT ∩ Θ t -1 , conditional on σ ( F t -1 ∪ A ′ t -1 ) , is bounded away from zero for all t ∈ [ T ] . We have two degrees of freedom in our analysis:

We can choose ( b t ) t , the widths of the ellipsoids (Θ t ) t . Larger widths may make it easier to bound ( p t ) t away from zero, but at a linear cost in the regret bound.

We can choose A ′ t -1 , the 'point' between observing Y t -1 and selecting θ t with respect to which we consider the aforementioned conditional probabilities defined.

Remark 10. The introduction of A ′ t -1 serves to model the case where θ t is sampled from a distribution P t that itself depends on X 1 , Y 1 , . . . , X t -1 , Y t -1 in a random manner. The σ -algebra A ′ t -1 is then such that P t is σ ( F t -1 ∪ A ′ t -1 ) -measurable.

The upcoming two lemmas will be helpful our applications of the Master Theorem; both are stated in terms of the random functions ψ t : R d → R d given by

The first is the classic concentration result of Abbasi-Yadkori et al. (2011), and the second, Proposition 5 of Abeille and Lazaric (2017).

Lemma 3. Under Assumption 1, for any δ ∈ (0 , 1] , P ( ∀ t ∈ N , θ ⋆ ∈ ψ δ t ( B d 2 )) ≥ 1 -δ .

Lemma 4. If Assumption 1 holds and θ ⋆ ∈ ψ δ t ( B d 2 ) , then for any measure Q over R d and a > 0 ,

where we recall that H u denotes the closed halfspace { v ∈ R d : ⟨ v, u ⟩ ≥ 1 } .

We provide a short, clean proof of Lemma 4 in Appendix C.

3.2 Proof of Theorem 1: regret bound for linear ensemble sampling

Let Γ 0 , Γ 1 , . . . be the sequence of random matrices in R d × m with the j th column given by

Recall that S j t = S j 0 + ∑ t i =1 U j i X i and observe that each ( S j ) t is a vector-valued random walk with increments given by ( U t X t ) t , where ( X t ) t is a serially correlated vector-valued sequence and ( U j t ) t is a sequence of independent U ([ -1 , 1]) variables, and (Γ j t ) t is a normalised version of ( S j t ) t . Consider the extremal singular values of Γ t ,

The lower of these may be interpreted as capturing how well the columns Γ 1 t , . . . Γ t m cover all directions u ∈ S d -1 2 , and the upper, their maximal deviations from zero. The following theorem, proven in Section 3.3, shows that, for a sufficiently large m , with high probability, for all t ∈ [ T ] , all the singular values of Γ t -1 are on the order of √ m .

Theorem 5. For λ ≥ 5 and m ≥ 400 log(3 + T ) ∨ 10 d , the event

Proof of Theorem 1. We will use the master theorem, Theorem 2. We let the filtrations ( A t ) t and ( A ′ t ) t needed in this theorem be defined recursively via

With this, the conditions concerning these filtrations hold.

We let b t = aβ δ t for a = 10 √ m , for each t ∈ N , which is σ ( F t ∪ A ′ t ) t -adapted, as required. For the two required events, we take the event E T from Theorem 5 and the event E ⋆ T = ∩ t ∈ N { θ ⋆ ∈ ψ t ( B d 2 ) } . To see that these satisfy the requirements given in equation (2), observe the following:

Conditions on E T By our assumptions on m and λ , the conditions for Theorem 5 are satisfied. Hence, P ( E T ) ≥ 1 -δ . We now show that on E T , for all t ∈ [ T ] , θ t ∈ Θ t -1 . Note this is equivalent to the statement that on E T , for all t ∈ [ T ] , ∥ ξ t Γ J t t -1 ∥ 2 ≤ b t -1 /r t -1 = 10 √ m/ 7 . And indeed, on E T ,

Conditions on E ⋆ T Since a ≥ 1 , ψ t ( B d 2 ) ⊂ ψ t ( a B d 2 ) = Θ t , and thus, by Lemma 3, θ ⋆ ∈ Θ t for all t ∈ N jointly with probability at least 1 -δ .

It remains to lower-bound the sequence ( p t ) t . For this, define P ′ t -1 ( · ) = P ( · | σ ( F t -1 ∪ A ′ t -1 )) . Fixing t ∈ N + , writing Q ( A ) = P ′ t -1 ( θ t ∈ A ) , we have that on E ⋆ T ,

where the last equality used the definitions of P ′ t -1 , F t -1 and A ′ t -1 .

We now show that on E T , regardless of the choice of u , for at least one ( s, j ) ∈ {± 1 } × [ m ] , 7 s Γ j t -1 ∈ H u ∩ a B d 2 . Assume thus that E T holds. First observe that for any ( s, j ) , by equation (3), 7 s Γ j t -1 ∈ a B d 2 . Hence, it remains to show that for any u , for some ( s, j ) , 7 s Γ j t -1 ∈ H u ; this holds because for any u ,

Hence, on E T ∩ E ⋆ T , p t -1 ≥ 1 / (2 m ) and thus b t -1 /p t -1 ≤ 20 m 3 / 2 β δ t -1 . Inserting this bound into the regret bound of Theorem 2 establishes the result.

Remark 11 (On symmetrisation) . If the algorithm were run without symmetrisation, following the steps of the proof we see that we need to show that on E T , regardless the choice of u , for at least one of j ∈ [ m ] , 7Γ j t -1 ∈ H u . In the presence of symmetrisation, the equivalent statement is that regardless of the choice of u , for at least one 'particle' j , we have either 7Γ j t -1 ∈ H u or 7Γ j t -1 ∈ H -u . Through equation (4) , the latter reduces to studying quadratic forms, which lead to convenient algebra.

Remark 12 (Can we improve our bound?) . For any u ∈ S d -1 2 , we lower bound the maximum max j ⟨ 7Γ j t -1 , u ⟩ 2 by the average 1 m ∑ m j =1 ⟨ 7Γ j t -1 , u ⟩ 2 , which we show exceeds 1, thus establishing the existence of at least one element ± 7Γ j t -1 in H u (out of 2 m ). This gives a 1 / (2 m ) lower bound for p t -1 . However, lower bounding a maximum by an average might be wasteful. If, instead, we managed to lower bound the median (or any other fixed-proportion order statistic) of ⟨ 7Γ 1 t -1 , u ⟩ 2 , . . . , ⟨ 7Γ t m -1 , u ⟩ 2 , we would be able to conclude that p t is lower bounded by a constant (since a constant proportion of 'particles' are then contained in H ± u ), and thus conclude that linear ensemble sampling performs similarly to Thompson sampling (see Appendix D for an analysis of the latter). However, order statistics are considerably more technical to work with than averages, especially in a non-i.d.d. setting, and we do not currently know if such a result is attainable (for order statistics, see Gordon et al. (2012) and Litvak and Tikhomirov (2018) and the references within).

Remark 13 (Upper versus lower bound) . When it comes to singular values, the difficulty is usually in the lower bounds. This is so for our problem as well. Consider our use of the upper bound in the middle inequality of equation (3) : all we really needed there is that ∥ Γ j t -1 ∥ 2 is upper bounded for all j ∈ [ m ] and t ∈ [ T ] . But, for any fixed j ∈ [ m ] , we can apply the standard method-of-mixtures of bound of Abbasi-Yadkori et al. (2011) to obtain that ∥ Γ j t -1 ∥ 2 = O ( √ d log T ) for all t ∈ [ T ] . An application of the union bound over the j ∈ [ m ] then recovers what we need, up to logarithmic terms. We note that the √ d factor is strictly necessary, per Lattimore (2023).

3.3 Setting up to prove Theorem 5: bound on singular values

Theorem 5 for t = 0 follows by standard methods (see Appendix E for proof):

Lemma 6. Whenever m ≥ 10 d ,

̸

To extend the result to t > 0 , we will consider the processes ( R j t ( u )) t and ( R t ( u )) t defined for u ∈ R d , u = 0 by

̸

Note that for v = V 1 / 2 t u = 0 one has R j t ( u ) = ⟨ v, Γ j t ⟩ 2 / ∥ v ∥ 2 . Since V t is positive-definite (and hence a bijection when viewed as a linear map) we observe the following relations:

Claim 7. For all t ≥ 0 , j ∈ [ m ] ,

̸

̸

̸

and likewise inf u =0 R t ( u ) = s 2 d (Γ t ) /m .

In the upcoming section (Section 3.4) we establish the following bound on R t ( u ) for a fixed u ∈ S d -1 2 . Lemma 8. Fix u ∈ S d -1 2 and λ ≥ 5 . Suppose that m ≥ 400 log(3 + 2 T ) . Then, there exists an event E ′ T with P ( E ′ T ) ≥ 1 -Te -m 400 such that on E ′ T ∩ { 1 2 ≤ R 0 ( u ) ≤ 3 2 } ,

Our proof of Theorem 5 employs the above pointwise bound together with a covering argument over S d -1 2 . For that, we need the following Lipschitzness result, proven in Appendix F (by simple algebra), and a standard bound on the size of ϵ -nets of S d -1 2 (Lemma 4.10 in Pisier, 1999).

Lemma 9. On S d -1 2 , R t is L -Lipschitz with L ≤ 4 ∥ Γ t ∥ 2 ∥ V 1 / 2 t ∥ / ( m √ λ ) .

Lemma 10. For all ϵ in (0 , 1] , there exists an ϵ -net N of S d -1 2 with |N| ≤ ( 1 + 2 ϵ ) d .

Proof of Theorem 5. Let N ϵ be a minimal ϵ -net of S d -1 2 and define the event

̸

̸

We will now confirm that, for a suitable choice of ϵ , E ϵ is a subset of the event in Theorem 5, and that P ( E ϵ ) ≥ 1 -NTe -m 400 , which establishes the theorem.

Let u ∈ S d -1 2 and v ∈ N ϵ be such that ∥ u -v ∥ 2 ≤ ϵ . From Lemma 9, and choosing ϵ = √ λ/ (132 √ T + λ ) , and noting that ∥ V 1 / 2 t ∥ ≤ √ T + λ , we get

Then on E ϵ , for our choice of ϵ ,

̸

Solving for ∥ Γ t ∥ 2 , we have that ∥ Γ t ∥ 2 = s 2 1 (Γ t ) ≤ 165 96 m . The same argument also gives that, on E ϵ ,

Loosening these bounds slightly, we have that on E ϵ , √ m/ 7 ≤ s d (Γ t ) ≤ s 1 (Γ t ) ≤ 10 √ m/ 7 .

The probability that E ϵ occurs is at least the probability that the event of Lemma 6 occurs and that the event of Lemma 8 occurs for each u ∈ N ϵ , noting that the former ensures 1 2 ≤ R 0 ( u ) ≤ 3 2 for all u ∈ N ϵ . Taking a union bound over these events, we have that

We conclude by noting that, by Lemma 10, for our choice of ϵ , N ≥ |N ϵ | +1 .

3.4 Proof of Lemma 8

Since we now consider a fixed u ∈ S d -1 2 , we will write R j t := R j t ( u ) and R t := R t ( u ) . Let ( A ′′ t ) t be the filtration given by A ′′ t = σ ( A t ∪ σ ( ξ t +1 , J t +1 )) for each t ∈ N , and let E ′′ t denote the σ ( F t ∪ A ′′ t ∪ σ ( X t +1 )) -conditional expectation, which will be used to integrate out the random targets U 1 t +1 , . . . , U t m +1 .

With that, we define

to be the drift and the noise of the process ( R t ) t ∈ N , respectively. Also let

be the strength of the drift. These are related by the following two results:

Claim 11. D t = ( 2 3 -R t ) Q t for all t ∈ N .

Lemma 12. For any T ∈ N + and m ≥ 400 log(3 + T ) , there exists an event E with P ( E ) ≥ 1 -Te -m 400 , such that on E ∩ { R 0 ≤ 2 } , for all 0 ≤ τ ≤ t < T ,

The lemma is proven in Appendix G, and the claim in Appendix H. The constant 2 3 above is just the almost sure value of E ′′ t [( U j t +1 ) 2 ] ; see also Remark 14 for a discussion of the significance of the ( U j t +1 ) 2 terms and how we bound these in the proof of said lemma.

Proof of Lemma 8. Fix 0 ≤ τ ≤ t < T . We can decompose R t +1 as

which unrolled back to τ and combined with Claim 11 gives us that

Observe from the above that the process R 0 , R 1 , . . . drifts towards 2 3 , with strength proportional the level of deviation, scaled by Q i . We will now show that on the event of Lemma 12, whenever R t moves sufficiently far away from 2 3 , the drift will overwhelm the effect of the noises ( W t ) t . Assume henceforth that the aforementioned event holds.

Lower bound. We consider the excursions of ( R t ) t where it goes and stays below 1 / 2 . Let 0 ≤ τ < s ≤ T be endpoints of such a maximal excursion, in the sense that R τ ≥ 1 / 2 , R τ +1 , . . . , R s < 1 / 2 and if s +1 ≤ T then R s +1 ≥ 1 / 2 . Our goal is to show that for any τ ≤ t < s , R t +1 ≥ 9 / 100 , which suffices to prove the lower bound. Fix t ∈ [ τ, s ) . From equation (5) and since the event from Lemma 12 holds, defining α = 1 / 10 ,

Upper bound. The upper bound follows near-verbatim, taking τ with R τ ≤ 3 2 < R τ +1 .

Remark 14. The proof of Lemma 8 was where we use that the targets ( U j t ) are uniform-or, in particular, that they are bounded random variables-for each W t +1 features ( U j t ) 2 terms, and might otherwise be only sub-exponential. Of course, in that case, we would simply use a truncation argument: pick some truncation level a > 0 , set W ′ t +1 = W t +1 ∧ a for each t ∈ N + and work with the process given by the recursion R ′ t +1 = W ′ t +1 + D t + R ′ t . Then, R t ≥ R ′ t for all t ∈ N , and the truncated noises ( W ′ t +1 ) t are once again subgaussian, so our approach to lower bounding R t would also work for R ′ t . We could then establish what we needed from the upper bound (that is, a bound for the quantity in equation (3) ) as discussed in Remark 13, which does not require bounded targets.

4 Discussion

We showed that linear ensemble sampling can work with relatively small ensembles, providing the first useful theoretical grounding for the method (see Remarks 6 and 7 for comparisons). We do, however, believe our result to be loose (see Remarks 2 and 12 for discussion); resolving this would make for an important step forward. Moreover, our algorithm uses a certain symmetrisation not used within the work of Lu and Van Roy (2017) (see Section 2.2 and Remark 11). While this symmetrisation comes with no particular downsides, we would nonetheless be curious to see whether there is a clean way of making the analysis go through without it. A natural further question is whether the idea of adding noise to the rewards is the right approach to the explore-vs-exploit dilemma. On this, first, in Janz et al. (2024) we showed that beyond the linear setting, it is losses rather than rewards that should be perturbed-with the two approaches being equivalent in the linear-Gaussian setting. Second, the very recent work of Cassel et al. (2024) shows that in the multi-armed bandit setting, a simple bootstrap-based method, which takes a max over the ensemble (as in Remark 5), yields instance dependent bounds. This raises the question of the trade-offs, if any, between randomising over the ensemble and taking a maximum, and between bootstrapping and using perturbations.

References

Abbasi-Yadkori, D. Pál, and C. Szepesvári. Improved algorithms for linear stochastic bandits. Advances in Neural Information Processing Systems , 2011. Cited on pages 6, 8.

Abeille and A. Lazaric. Linear Thompson sampling revisited. Electronic Journal of Statistics , 11(2):5165-5197, 2017. Cited on pages 1, 2, 5, 6, 15, 16.

Agrawal and N. Goyal. Thompson sampling for contextual bandits with linear payoffs. In International Conference on Machine Learning , 2013. Cited on pages 1, 15.

Boucheron, G. Lugosi, and P. Massart. Concentration Inequalities: A Nonasymptotic Theory of Independence . Oxford University Press, 2013. Cited on pages 13, 17.

Burda, H. Edwards, A. Storkey, and O. Klimov. Exploration by random network distillation. arXiv preprint arXiv:1810.12894 , 2018. Cited on page 1.

Cassel, O. Levy, and Y. Mansour. Batch Ensemble for Variance Dependent Regret in Stochastic Bandits. arXiv preprint arXiv:2409.08570 , 2024. Cited on page 10.

Curi, F. Berkenkamp, and A. Krause. Efficient model-based reinforcement learning through optimistic policy search and planning. Advances in Neural Information Processing Systems , 2020. Cited on page 1.

H. de la Pena, M. J. Klass, and T. Leung Lai. Self-normalized processes: exponential inequalities, moment bounds and iterated logarithm laws. The Annals of Probability , 2004. Cited on page 18.

Dimakopoulou and B. Van Roy. Coordinated exploration in concurrent reinforcement learning. In International Conference on Machine Learning , 2018. Cited on page 1.

Dwaracherla, X. Lu, M. Ibrahimi, I. Osband, Z. Wen, and B. Van Roy. Hypermodels for exploration. arXiv preprint arXiv:2006.07464 , 2020. Cited on page 1.

Eckles and M. Kaptein. Bootstrap Thompson sampling and sequential decision problems in the behavioral sciences. Sage Open , 9(2):2158244019851675, 2019. Cited on page 1.

W. Fang. Symmetric multivariate and related distributions . Chapman and Hall/CRC, 1990. Cited on page 16.

Filippi, O. Cappe, A. Garivier, and C. Szepesvári. Parametric bandits: The generalized linear case. In Advances in Neural Information Processing Systems , 2010. Cited on page 2.

Gordon, A. E. Litvak, C. Schütt, and E. Werner. Uniform estimates for order statistics and Orlicz functions. Positivity , 16(1):1-28, 2012. Cited on page 8.

Hao, J. Zhou, Z. Wen, and W. W. Sun. Low-rank tensor bandits. arXiv preprint arXiv:2007.15788 , 2020. Cited on page 1.

Jacot, F. Gabriel, and C. Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in Neural Information Processing Systems , 2018. Cited on page 2.

Janz, S. Liu, A. Ayoub, and C. Szepesvári. Exploration via linearly perturbed loss minimisation. In International Conference on Artificial Intelligence and Statistics , 2024. Cited on pages 1, 10.

Kveton, C. Szepesvári, M. Ghavamzadeh, and C. Boutilier. Perturbed-History Exploration in Stochastic Linear Bandits. In Proceedings of The 35th Uncertainty in Artificial Intelligence Conference , volume 115 of Proceedings of Machine Learning Research , pages 530-540. PMLR, 2020. Cited on page 1.

Lattimore. A lower bound for linear and kernel regression with adaptive covariates. In The Thirty Sixth Annual Conference on Learning Theory , 2023. Cited on page 8.

Lattimore and C. Szepesvári. Bandit algorithms . Cambridge University Press, 2020. Cited on pages 2, 13, 18.

Lee and M.-h. Oh. Improved Regret of Linear Ensemble Sampling. In Advances in Neural Information Processing Systems , 2024. Cited on page 5.

E. Litvak and K. Tikhomirov. Order statistics of vectors with dependent coordinates, and the Karhunen-Loève basis. The Annals of Applied Probability , 28(4):2083-2104, 2018. Cited on page 8.

Lu and B. Van Roy. Ensemble sampling. Advances in Neural Information Processing Systems , 2017. Cited on pages 1-3, 5, 10, 13.

Lu, Z. Wen, and B. Kveton. Efficient online recommendation via low-rank ensemble sampling. In Proceedings of the 12th ACM Conference on Recommender Systems , 2018. Cited on page 1.

Osband, J. Aslanides, and A. Cassirer. Randomized prior functions for deep reinforcement learning. Advances in Neural Information Processing Systems , 2018. Cited on pages 1, 4.

Osband, C. Blundell, A. Pritzel, and B. Van Roy. Deep exploration via bootstrapped DQN. Advances in Neural Information Processing Systems , 2016. Cited on pages 1, 4.

Osband, B. Van Roy, D. J. Russo, Z. Wen, et al. Deep Exploration via Randomized Value Functions. Journal of Machine Learning Research , 20(124):1-62, 2019. Cited on page 2.

Pisier. The volume of convex bodies and Banach space geometry , volume 94. Cambridge University Press, 1999. Cited on page 8.

Qin, Z. Wen, X. Lu, and B. Van Roy. An analysis of ensemble sampling. In Advances in Neural Information Processing Systems , 2022. Cited on pages 1, 5.

Skorski. Bernstein-type bounds for beta distribution. Modern Stochastics: Theory and Applications , 10(2):211-228, 2023. Cited on page 16.

Srinivas, A. Krause, S. M. Kakade, and M. Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. In 2010. Cited on page 2.

R. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika , 25(3-4):285-294, 1933. Cited on page 1.

Vershynin. High-dimensional probability: an introduction with applications in data science , volume 47. Cambridge University Press, 2018. Cited on page 16.

Yang, D. Eckles, P. Dhillon, and S. Aral. Targeting for long-term outcomes. arXiv preprint arXiv:2010.15835 , 2020. Cited on page 1.

Zanette, D. Brandfonbrener, E. Brunskill, M. Pirotta, and A. Lazaric. Frequentist regret bounds for randomized least-squares value iteration. In International Conference on Artificial Intelligence and Statistics , 2020. Cited on page 2.

Zhu and B. Van Roy. Deep Exploration for Recommendation Systems. arXiv preprint arXiv:2109.12509 , 2021. Cited on page 1.

A A reformulation of Algorithm 1 in the style of Lu and Van Roy (2017)

Below in Algorithm 2, we list a reformulation of Algorithm 1 written using updates in the style of Lu and Van Roy (2017), with one difference: the algorithm here accepts only a single time-stationary perturbation scale r > 0 (this is unavoidable with the Lu and Van Roy (2017) style of updates). Taking r = 7 ˜ β δ T , where ˜ β δ T is the deterministic upper bound on β δ 0 , . . . , β δ T given by

allows for the same guarantee as given in Theorem 1, but with ˜ β δ T in place of β δ τ -1 .

To confirm that Algorithm 2 is equivalent to Algorithm 1 if the sequence r 0 , r 1 , . . . is set to the common value r , it suffices to compare the list of parameters w 1 t -1 , . . . , w 2 t m -1 used here with the list

at each step of the algorithm. We leave this as a simple algebraic exercise for the reader.

B Proof of Theorem 2: master regret bound

We need the following concentration inequality, a simple consequence of Exercise 20.8 in Lattimore and Szepesvári (2020) 2 and Hoeffding's lemma (Lemma 2.2, Boucheron et al., 2013), and the elliptical potential lemma (Lemma 19.4 in Lattimore and Szepesvári, 2020).

Lemma 13. Fix 0 < δ ≤ 1 . Let ( ξ t ) t ∈ N + be a real-valued martingale difference sequence satisfying | ξ t | ≤ c almost surely for each t ∈ N + and some c > 0 . Then,

Lemma 14 (Elliptical potential lemma) . Let ( x t ) t ∈ N + be a sequence of vectors in B d 2 , let V 0 = λI for some λ ≥ 1 and V t = V 0 + ∑ t i =1 x i x T i for each t ∈ N + . Then, for all τ ∈ N + ,

Recall that J ( θ ) = max x ∈X ⟨ x, θ ⟩ is the support function of X , and observe the following: Claim 15. For any t ∈ N + , X t is a subgradient of J at θ t .

+ d

which is the inequality that defines a subgradient.

Proof of Theorem 2. Throughout this proof, we work on the intersection E T ∩ E ⋆ T , and therefore in particular use that for any t ∈ [ T ] , θ t , θ ⋆ ∈ Θ t . We let γ t = 2 b t , and observe that this is the V t -weighted Euclidean norm width of Θ t , in the sense that

We will also use the shorthands F ′ t = σ ( F t ∪ A ′ t ) , P ′ t = P ( · | F ′ t ) and E ′ t = E [ · | F ′ t ] .

The proof is based on decomposing the regret into two parts, which we then control separately:

Fix t ∈ [ τ ] and consider the second term, J ( θ t ) -⟨ X t , θ ⋆ ⟩ . We have that

where the first inequality is by Cauchy-Schwartz, the second uses that θ t , θ ⋆ ∈ Θ t -1 and then equation (7).

Now consider the first term, J ( θ ⋆ ) -J ( θ t ) . Let θ -be a minimiser J over Θ t -1 (which is welldefined, since J is continuous and Θ t -1 closed) and let θ + be an element of Θ OPT ∩ Θ t -1 (which is non-empty, since it contains at least θ ⋆ ). Then, since θ ⋆ , θ t ∈ Θ t -1 , we have the bound

It follows that for any probability measure Q over Θ OPT ∩ Θ t -1 ,

Let Θ OPT t -1 = Θ OPT ∩ Θ t -1 , and take Q = Q t -1 in the integral above defined by

which yields the bound

where for p t -1 = 0 we take the upper bound to be positive infinity. Observing that X t is a subgradient of J at θ t (Claim 15 and equation (6)) and applying Cauchy-Schwartz, we have that

Moreover, since θ -∈ Θ t -1 , observing that norms are non-negative, that Θ OPT t -1 ⊂ Θ t -1 and using equation (7),

and therefore, since γ t -1 is F ′ t -1 -measurable (by assumption), we have the bound

Chaining the above inequalities and writing these in terms of ∆ t = E ′ t -1 [ ∥ X t ∥ V -1 t -1 ] -∥ X t ∥ V -1 t -1 , we have the bound

Combining equations (9) and (10) with the regret decomposition in equation (8),

for any τ ∈ [ T ] . For the first sum in that upper bound, by Cauchy-Schwartz and the elliptical potential lemma (Lemma 14, which can be applied because by assumption λ ≥ 1 ), for any τ ∈ N + ,

To deal with the second sum, observe that since for all t ∈ N + , V t -1 ⪰ λI and X t ∈ B d 2 ,

Moreover, observe that (∆ t ) t is an ( F ′ t ) t -adapted martingale difference sequence. We thus apply Lemma 13 with c = 2 / √ λ , to obtain the deviation probability bound

The bounds on the two sums, equation (12) and equation (13), when inserted into equation (11) and combined with a union bound, yield the claimed result.

C Proof of Lemma 4: optimism for elliptical confidence sets

Lemma 16. Let F : R d → R be a convex function and let u be its maximizer over the unit ball. Then, for any v ∈ H u . = { v ∈ R d : ⟨ v, u ⟩ ≥ 1 } , we have F ( v ) ≥ F ( u ) .

Proof. For any v ∈ R d with ⟨ v, u ⟩ > 1 , the ray from v to u enters the interior of the unit ball. Hence, for any such v , there exists a z ∈ B d 2 and α ∈ (0 , 1) such that u = αz +(1 -α ) v . By convexity and maximality,

Hence, F ( u ) ≤ F ( v ) . Since any finite convex function on an open set is continuous, the result holds for any v ∈ H u .

Proof of Lemma 4. Write F = J ◦ ψ δ t ; since J is convex and ψ δ t is affine, F is convex. Let u + be the maximiser of F over B d 2 . Since F is a convex function and B d 2 is a convex set, u + ∈ ∂ B d 2 = S d -1 2 . By assumption, θ ⋆ ∈ ψ δ t ( B d 2 ) , and so J ( θ ⋆ ) ≤ F ( u + ) . By Lemma 16, F ( u + ) ≤ F ( u ′ ) for any u ′ ∈ H u + . Thus, ψ δ t ( H u + ) ⊂ Θ OPT , and so

Therefore, for any measure Q on R d ,

D Analysis of Thompson sampling via our master theorem

Algorithm 3 is a version of Thompson sampling we call confident linear Thompson sampling . It is extremely simple: at each step t ∈ [ T ] , it picks an action X t that is optimal according to an estimate θ t sampled uniformly on ψ δ t -1 ( √ d B d 2 ) = Θ t -1 , a √ d -inflation of the ridge regression confidence set. It differs from usual linear Thompson sampling of Agrawal and Goyal (2013) through the use of a uniform sampling distribution.

Remark 15. Our use of the uniform distribution to generate perturbed parameters is purely for the sake of a clean exposition, and for easy comparison with our linear ensemble sampling algorithm. Observe that the usual analysis for the Gaussian (or subgaussian) case begins by restricting to a high-probability event where every θ t lands within some inflated version of the corresponding Θ t -1 (as in Agrawal and Goyal, 2013; Abeille and Lazaric, 2017).

Algorithm 3 Confident linear Thompson sampling for t ∈ N + do

Sample

U

√

t 2 t t -1 t Compute some X t ∈ arg max x ∈X ⟨ x, θ t ⟩ , play action X t and receive reward Y t

∼ U

(

d

B

d

)

and compute

θ

=

ψ

δ

(

U

)

Theorem 17. Let Assumption 1 hold. Fix δ ∈ (0 , 1] and let λ ≥ 1 . There exist some universal constant C > 0 such that with probability 1 -δ , a learner using Algorithm 3 incurs regret satisfying

The above result recovers the same regret bound for confident linear Thompson sampling as that given by the analysis of Abeille and Lazaric (2017). The proof is as follows.

Proof of Theorem 17. Take T = ∞ . Fix b t = √ d for all t ∈ N . As each θ t is a uniform random variable on Θ t -1 for all t ∈ N , and so event E T holds almost surely. Moreover, as for ensemble sampling, we take E ⋆ T to be the event from the standard concentration result for ridge regression, Lemma 3, observing as before that, ψ δ t -1 ( B d 2 ) ⊂ ψ δ t -1 ( √ d B d 2 ) = ψ δ t -1 ( b t B d 2 ) .

We pick A ′ t -1 = A t -1 for all t ∈ N + . Now, to lower bound each p t -1 , note that on E ⋆ T , by Lemma 4 applied with Q ( A ) = P ( θ t ∈ A | σ ( F t -1 ∪ A ′ t -1 )) =: P ′ t -1 ( θ t ∈ A ) ,

And, since ψ δ t -1 is a bijection and U t is uniform on √ d B d 2 , and thus rotationally invariant,

As established in Appendix A of Abeille and Lazaric (2017), U ( H 1 ∩ √ d B d 2 ) ≥ 1 / (16 √ 3 π ) , independently of d . This means that, on E ⋆ T , b t -1 p t -1 ≤ 16 √ 3 πd for all t ∈ N + ; plugging this into the regret bound of Theorem 2 yields the claim.

E Proof of Lemma 6: singular values at initialisation

Lemma 6 follows by taking y = 81 m 2500 and ϵ = 1 20 in Theorem 18.

Theorem 18. Let U ∈ R m × d , m ≥ d , be a random matrix with independent rows U 1 , . . . , U m distributed uniformly on √ d S d -1 2 . Then, for all y > 0 , ϵ ∈ (0 , 1 / 2) , and with c ϵ = 1 / (1 -2 ϵ ) ,

The proof of Theorem 18 follows the approach of Chapter 4 of Vershynin (2018), but makes the constants explicit. For these constants, we will need the following claim:

Claim 19. Fix x ∈ S d -1 2 , let X ∼ U ( S d -1 2 ) and X 2 x = ⟨ X,x ⟩ 2 . Then, E X 2 x = 1 /d , and for some ν, c > 0 satisfying ν ≤ 2 /d 2 and c ≤ 4 /d and all 0 < s < 1 /c ,

Proof. It is known that X 2 x ∼ Beta( 1 2 , d -1 2 ) (see, for example, Theorem 1.5 and the discussion thereafter in Fang, 1990), the expectation of which is 1 /d . We thus need only look up moment generating function bounds for beta random variables. Skorski (2023) derives such in their proof of their Theorem 1, and our result follows by substituting in the parameters of our beta distribution, and bounding the resulting ν and c crudely.

We will need the next two results from Vershynin (2018), appearing as Lemma 4.1.5 and Exercise 4.4.3 respectively.

Lemma 20 (Appropximate isometry) . For any matrix A ∈ R m × d and ϵ ≥ 0 ,

Lemma 21 (Quadratic form on a net) . For a symmetric matrix A ∈ R d × d and an ϵ -net N of S d -1 2 with ϵ ∈ (0 , 1 / 2) ,

Proof of Theorem 18. Fix x ∈ S d -1 2 , and consider Z 2 x = 1 m ∥ Ux ∥ 2 2 = d m ∑ m j =1 ⟨ U j / √ d, x ⟩ 2 . Observe that each U j / √ d ∼ U ( S d -1 2 ) ; thus, by Claim 19 and since U 1 , . . . , U m are independent, adopting the notation of the claim, we have that, for all 0 < sd/m < 1 /c ,

Thus, | Z 2 x -1 | is what Boucheron et al. (2013) would term sub-gamma with parameters ( d 2 ν/m, cd/m ) on both tails. Applying the maximal-form of the there-stated Bernstein-type bound for sub-gamma random variables, a union bound over the two tails, and the bounds ν ≤ 2 /d 2 and c ≤ 4 /d from Claim 19, we have that, for all y > 0 ,

Now let N ϵ , ϵ ∈ (0 , 1 / 2) , be an ϵ -net of S d -1 2 . By Lemma 21,

Also, by our bound on nets from Lemma 10, |N ϵ | ≤ (1 + 2 /ϵ ) d . Thus, taking a union bound over x ∈ N ϵ , we have that for any y > 0 and ϵ ∈ (0 , 1 / 2) , the probability that

We conclude the proof by observing that sup x ∈ S d -1 2 | Z 2 x -1 | = ∥ U T U/m -I ∥ and applying Lemma 20.

F Proof of Lemma 9: Lipschitzness result

̸

Proof of Lemma 9. Fix u = 0 and let z = V 1 / 2 t u . Then,

Now note that for all non-negative a, b, A, B with b ≥ a ,

Let u, v ∈ S d -1 2 , z = V 1 / 2 t u , w = V 1 / 2 t v . Let ϵ = ∥ u -v ∥ 2 . Let A = ∥ Γ T t z ∥ 2 , B = ∥ Γ T t w ∥ 2 , a = ∥ z ∥ 2 , b = ∥ w ∥ 2 . Assume without loss of generality that b ≥ a . Since v ∈ S d -1 2 and b ≥ √ λ ,

and likewise

Putting things together, we have

G Proof of Lemma 12: concentration result

This proof will require the following definition of conditional subgaussianity, and the standard de la Peña-type concentration result for sequences of such random variables, stated thereafter.

Definition 22. Let A,B be random variables and F a σ -algebra. We say that A is F -conditionally B -subgaussian when B is non-negative and F -measurable, and for all s ∈ R ,

Lemma 23. Let ( A i , B i , H i ) i be such that each A i is H i -conditionally B i -subgaussian. Then, for any x, y > 0 ,

Lemma 23 is an immediate consequence of Theorem 2.1 in de la Pena et al. (2004), in particular comparing our Definition 22 with their condition (1.4). For more background, see Lattimore and Szepesvári (2020), Lemma 20.2, and the surrounding discussion.

We will also need the following three claims, verified in Appendix H. Therein,

Claim 24. Each W t +1 is σ ( F t ∪ A ′′ t ) -conditionally σ t / √ m -subgaussian.

Claim 25. For any 0 ≤ t ≤ τ < T , 1 + ∑ t i = τ σ 2 i ≤ 3 + ∑ t i = τ Q i R i .

Claim 26. For any 0 ≤ t ≤ τ < T , on the event { R 0 ≤ 2 } , 1 + ∑ t i = τ σ 2 i ≤ (3 + 2 T ) 2 .

Proof of Lemma 12. For any τ < T , by Claim 24 and Lemma 23 applied with y = 1 /m and any x > 0 , with probability 1 -exp {-x 2 } , for all t ∈ { τ, . . . , T -1 } ,

We take x = 2log(3 + 2 T ) , which by Claim 26 exceeds log( ∑ t i = τ σ 2 t +1) on { R 0 ≤ 2 } . Therefore, on the intersection of { R 0 ≤ 2 } and the event implicitly defined above,

By our assumption on m , √ 2 x/m ≤ 1 / 10 . Also, and using a trivial bound and Claim 25,

which combined with the display above yields the form of the claimed bound.

To conclude the proof, we take a union bound over τ ∈ { 0 , . . . , T -1 } and note that, by our assumption on m and choice of x , x/ 2 ≤ m/ 400 .

H Proofs of Claims 11 and 24 to 26

Let F ′′ t = σ ( F t ∪A ′′ t ∪ σ ( X t +1 )) , where we recall that A ′′ t = σ ( A t ∪ σ ( ξ t +1 , J t +1 , X t +1 )) for each t ∈ N , that E ′′ t denotes the F ′′ t -conditional expectation, and that the purpose of conditioning on F ′′ t will be to integrate out the random targets U 1 t +1 , . . . , U t m +1 .

Proof of Claim 11. Fix u ∈ S d -1 2 and note that

Observe that X t +1 and S j t = S j 0 + ∑ s ≤ t U j s X s are F ′′ t -measurable and that U j t +1 is independent of F ′′ t . The latter of these gives E ′′ t U j t +1 = 0 and E ′′ t ( U j t +1 ) 2 = 2 3 . With that, we have that

The statement follows by averaging over j ∈ { 1 , . . . , m } .

Proof of Claim 24. Subtracting the first expression on the right-hand side of equation (15) from equation (14) and averaging over j ∈ { 1 , . . . , m } , we see that

where H j t = ⟨ u, X t +1 ⟩⟨ u, S j t ⟩ / ∥ u ∥ 2 V t +1 . Note that Q t and H t are F ′′ t measurable and that U 1 t +1 , . . . , U t m +1 are independent of F ′′ t and one another, and that their absolute values are bounded by 1 . Thus, examining the two terms in the sum we see that:

Q t m ∑ m j =1 (( U j t +1 ) 2 -2 3 ) is F ′′ t -conditionally Q t √ m -subgaussian.

1 m ∑ m j =1 U j t +1 H j t is F ′′ t -conditionally H t √ 2 m -subgaussian, where

The result follows by recalling that the sum of an a -subgaussian random variable and a b -subgaussian random variable is √ 2( a 2 + b 2 ) -subgaussian.

The proof of the final two claims will require the following simple lemma.

Lemma 27. Let b 1 , b 2 , . . . be a sequence of real numbers in [0 , 1] . Then, for any λ > 0 and n ∈ N + ,

Proof. Let B 0 = 0 and for j ∈ [ n ] let B j = B j -1 + b j . The function f ( x ) = 1 λ + x is decreasing on [0 , ∞ ) . Hence, ∫ B j B j -1 f ( x ) dx ≥ b j f ( B j ) . Summing these up,

Evaluating the left-hand side and noting that B n ≤ n holds because b i ≤ 1 gives the first result. For the second sum, we use a similar argument with g ( x ) = 1 / ( λ + x ) 2 :

Proof of Claim 25. Noting that since ∥ u ∥ 2 = 1 and λ ≥ 5 , by Lemma 27,

and

Using these, we have

Proof of Claim 26. Since ( a + b ) 2 ≤ 2 a 2 +2 b 2 and by symmetry,

By definition, R i = 1 m ∑ m j =1 R j i , and by assumption R 0 ≤ 2 and i ≤ T -1 , so R i ≤ 4+2 i ≤ 2+2 T . And so,

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/pu blic/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents theoretical results for an exploration algorithm using ensemble sampling in the stochastic linear bandit setting. The proposed algorithm creates $2m = O(d \\log T)$ ensemble models, selects one model uniformly at random, and then chooses the greedy action based on the parameters of that model.\nThe proposed algorithm achieves a regret bound of $\\tilde{O}(d^{5/2} \\sqrt{T})$, and it does not depend on the size of the action set.\n\nSTRENGTHS:\n- Unlike previous ensemble sampling algorithms (Lu & Van Roy, 2017; Qin et al., 2022), the proposed algorithm presents a frequentist regret analysis with a reasonable ensemble size. The authors also explain well the differences (e.g., symmetrization) compared to existing randomized algorithms like Thompson sampling and perturbed history exploration.\n- The master theorem (Theorem 1) for analyzing the regret bound of the randomized algorithm using the probability that the selected parameter by the algorithm is more optimistic than the true parameter is also intriguing. Proof seems correct, even though I didn’t check very carefully.\n\nWEAKNESSES:\n- As noted in Remark 4, although this paper provides a meaningful theoretical result for ensemble sampling, compared to randomized algorithms like Thompson sampling and perturbed history exploration, the proposed algorithm is neither computationally efficient nor statistically efficient. Ensemble sampling is generally effective in complex settings such as DeepRL. However, the regret bound presented in this paper does not seem sufficient to prove the validity of the ensemble algorithm even in the linear setting.\n- Thompson sampling algorithms generally have looser regret bounds than UCB algorithms but are known to perform better empirically. It would be beneficial if there were experimental results showing that ensemble sampling algorithms could empirically outperform existing randomized algorithms.\n\nQUESTIONS:\n1. When I first saw the ensemble sampling setting, it reminded me of the LSVI-PHE algorithm for linear MDPs with perturbed history exploration (Ishfaq et al., 2021). Ishfaq et al. (2021) address algorithms in the MDP setting, but with a horizon length of 1 ($H=1$), it can align with the linear bandit setting. Ishfaq et al. (2021) estimate perturbed parameters using Gaussian noise on observed rewards from previous episodes, repeating this process $M$ times. Actions are chosen based on the most optimistic of these estimated parameters. If we consider each perturbed estimator as an ensemble model, we have an ensemble size of $M$. However, the algorithm in Ishfaq et al. guarantees a high probability regret bound of $\\tilde{O}(d^{3/2} \\sqrt{T})$ (when $H=1$), with a theoretical sampling size of $M=O(d)$.\nThe only difference from the proposed ensemble sampling in this paper is that this paper selects the ensemble model to play uniformly at random. If the most optimistic model were chosen as in Ishfaq et al., could a better regret bound be guaranteed? Additionally, are there any other differences between the algorithm in Ishfaq et al. and the one proposed in this paper?\n\n* * *\nIshfaq, Haque, et al. \"Randomized exploration in reinforcement learning with general value function approximation.\" International Conference on Machine Learning. PMLR, 2021.\n\nLIMITATIONS:\nThe authors have addressed the limitations in Section 4.\nThe content discussed in this paper appears to have little to no negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a regret analysis of ensemble sampling within the stochastic linear bandit framework. It demonstrates that an ensemble size scaling logarithmically with time and linearly with the number of features suffices, marking a theoretical advancement. The paper shows that under standard assumptions for a \\(d\\)-dimensional stochastic linear bandit with an interaction horizon \\(T\\), the regret is of the order \\((d \\log T)^{5/2} \\sqrt{T}\\). Although the regret bound is not as tight as the existing bounds for the linear bandits, this is the first analysis of ensemble sampling in a structured setting that avoids the need for the ensemble size to scale linearly with \\(\\sqrt{T}\\), which previously compromised the purpose of ensemble sampling, while still achieving near-optimal order regret.\n\nSTRENGTHS:\n- The paper makes a decent theoretical contribution by reducing the required scaling of ensemble size, which enhances the applicability and efficiency of ensemble sampling in linear bandit problems. By showing the better scaling of ensemble size, the paper provides a pathway for more efficient implementation of the ensemble sampling algorithms.\n- To my knowledge, this is the first correct frequentist regret analysis of linear ensemble sampling.\n\nWEAKNESSES:\n- One drawback is that, as the authors acknowledge, the regret bound of \\(O(d^{5/2} \\sqrt{T})\\) is relatively loose compared to existing linear bandit results. For example, TS-based algorithms like LinTS typically achieve \\(O(d^{3/2} \\sqrt{T})\\) regret, making the proposed bound clearly suboptimal.\n- The dependence on \\(m\\) in the regret bound indicates that as the ensemble size increases, the algorithm's regret performance worsens. This superlinear dependence, with regret being \\(O(m^{3/2})\\), raises concerns. Given this result, it is unclear why one would opt for this particular way (as proposed in the paper) of linear ensemble sampling.\n\n**Minor comment (but for clarity)**\n- I recommend that the authors explicitly write the linear \\(d\\)-dependence on the ensemble size \\(m\\) instead of shoving \\(d\\)-dependence into \\(N\\) in the theorem statement of Theorem 1 if possible, as I see that authors are transparent about $d \\log T$ dependence on $m$ in Remark 1.\n- I am not sure whether the expression \"**slightly** worse than that obtained for Thompson sampling\" (in Line 120) is adequate given that such a gap (extra $d$) can be seen as **significantly** by among many bandit researchers. I understand the authors' intention, but I suggest removing such an expression of \"slightly.\"\n\nQUESTIONS:\n- Regarding the comments in Remark 3, under unknown $T$, can't you derive some regret bound with the doubling epoch trick though? Or, do you argue that you cannot get any bound (sublinear in T) even with the doubling trick?\n\nLIMITATIONS:\nThere is no separate \"Limitations\" section. But the authors discuss the weakness of the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

comments: SUMMARY:
The authors study an upper confidence bound (UCB) type of ensemble sampling method specific to the stochastic linear bandits. Based on previous work, it improves the analysis by introducing Rademacher variables for symmetrising the ensemble. The authors are thus able to obtain $\sqrt{T}$ dependency in the regret with an ensemble size scaling with $d$ (dimension) and $\log T$ instead of $T$ (horizon).

STRENGTHS:
- Removing the linear dependency of $m$ (ensemble size) on $T$ (horizon) for this ensemble sampling method is significant.
- The master theorem in Section 3 introduces intermediate filtrations $A'$ that may be of independent interest.

WEAKNESSES:
- The point of introducing the ensemble sampling in [Lu and Van Roy 2017] seems to be approximating Thompson sampling for complex models where computation is intractable. Meanwhile, the regret of linear bandits is known and achievable by LinUCB. Although the analysis improves for using smaller ensemble size in linear bandits, it deviates from the difficult setup and hence undermines the contribution of this paper.
- The number and omnipresence of remarks in the main text obscures the core idea of this paper. For example, Remark 3 does not add much to its section (the upper bound) and is under-explained.

QUESTIONS:
- I may have missed this - can the authors point out the significance of the regularization parameter λ? It seems from the theorems (2 and 5) λ only needs to be larger than 5 (i.e. an absolute constant)?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

comments: SUMMARY:
This paper studies the ensemble sampling in the stochastic linear bandit setting, and shows that the ensemble sampling can achieve $\mathcal{O}( (d\log T)^{5/2}  \sqrt{T})$ regret with an ensemble size of order $\mathcal{O}(d log T)$. The authors claim this is the first meaningful result of ensemble sampling as it does not require sub-linear ensemble size.

STRENGTHS:
1. Though I have skimmed the proof of several lemmas, the analysis part seems to be rigorous and mathematically correct. The notations are clear defined before their references. 
2. The result which shows that ensemble learning is able to deal with linear bandit problem is quite interesting. This result would be applied in other similar settings.

WEAKNESSES:
1. It would be better to emphasis the differences between Algorithm 1 and LinUCB and explain how these differences work. The beta is quite similar. 
2. Though the authors refer the audience to the previous works for the motivation of studying ensemble sampling, it's still beneficial to include the related works on linear bandits and briefly discuss the specific advantages of using ensemble sampling.

QUESTIONS:
The questions are raised in the weakness section. I am willing to re-evaluate the scores if these questions are properly answered.

LIMITATIONS:
This paper is pure theoretical and does not have any limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12