Title: Improved Algorithms for Stochastic Linear Bandits Using Tail Bounds for Martingale Mixtures
Venue: NeurIPS 2023
Accepted: yes
Keywords: Linear bandits

confidence sequences

martingales

convex optimization

cumulative regret

regret analysis
Abstract: We present improved algorithms with worst-case regret guarantees for the stochastic linear bandit problem. The widely used "optimism in the face of uncertainty" principle reduces a stochastic bandit problem to the construction of a confidence sequence for the unknown reward function. The performance of the resulting bandit algorithm depends on the size of the confidence sequence, with smaller confidence sets yielding better empirical performance and stronger regret guarantees. In this work, we use a novel tail bound for adaptive martingale mixtures to construct confidence sequences which are suitable for stochastic bandits. These confidence sequences allow for efficient action selection via convex programming. We prove that a linear bandit algorithm based on our confidence sequences is guaranteed to achieve competitive worst-case regret. We show that our confidence sequences are tighter than competitors, both empirically and theoretically. Finally, we demonstrate that our tighter confidence sequences give improved performance in several hyperparameter tuning tasks.

Improved Algorithms for Stochastic Linear Bandits Using Tail Bounds for Martingale Mixtures

Abstract

We present improved algorithms with worst-case regret guarantees for the stochastic linear bandit problem. The widely used 'optimism in the face of uncertainty' principle reduces a stochastic bandit problem to the construction of a confidence sequence for the unknown reward function. The performance of the resulting bandit algorithm depends on the size of the confidence sequence, with smaller confidence sets yielding better empirical performance and stronger regret guarantees. In this work, we use a novel tail bound for adaptive martingale mixtures to construct confidence sequences which are suitable for stochastic bandits. These confidence sequences allow for efficient action selection via convex programming. We prove that a linear bandit algorithm based on our confidence sequences is guaranteed to achieve competitive worst-case regret. We show that our confidence sequences are tighter than competitors, both empirically and theoretically. Finally, we demonstrate that our tighter confidence sequences give improved performance in several hyperparameter tuning tasks.

1 Introduction

The stochastic linear bandit problem is a sequential decision-making problem where, in each round t , a learner chooses an action a t and then receives a stochastic reward r t for its choice of action. The expected value of each reward is a linear function ϕ ( a t ) ⊤ θ ∗ of a known feature vector ϕ ( a t ) associated with the corresponding action, while θ ∗ is unknown. The linear bandit problem has attracted a great deal of attention because it is expressive enough to be a faithful model of many real-world decision-making problems, such as news recommendation (Li et al., 2010) and dynamic pricing (Cohen et al., 2020), yet it is simple enough to make theoretical analysis tractable.

A popular way to design algorithms for linear bandits is to follow the principle of optimism in the face of uncertainty. This principle states that we should choose actions as if the expected reward function is as nice as plausibly possible. For linear bandits, the principle can be instantiated with a confidence sequence Θ 0 , Θ 1 , . . . for the parameter vector θ ∗ ∈ Θ of the expected reward function. A confidence sequence is a sequence of subsets of the full parameter space Θ , which is built iteratively as data becomes available and is constructed such that with high probability over the observed data, θ ∗ is contained in each confidence set Θ t . One can then run an upper confidence bound (UCB) algorithm, which at round t chooses an action a t +1 by maximising the UCB max θ ∈ Θ t { ϕ ( a ) ⊤ θ } with respect to a .

The popularity of UCB algorithms stems from the fact that they come with worst-case regret guarantees and often perform well in practice. However, the performance of a UCB algorithm is intimately

4

2

0

2

4

x

6

4

2

0

2

4

6

8

y

CMM-UCB

4

2

0

2

4

x

6

4

2

0

2

4

6

8

4

2

0

2

4

x

6

4

2

0

2

4

6

8

OFUL

tied to the size of the confidence sets it uses. The smaller the subsets in the confidence sequence, the better the regret bound and, perhaps more importantly, the better the algorithm performs in practice.

In this work, we develop a general-purpose tail bound for martingale mixtures, which can be used to construct confidence sequences. When we specialise our general results to the linear bandit problem, the maximisation problem to compute the UCB is a convex program. We maximise the UCB over actions via gradient-based methods, and investigate two procedures for computing the UCB along with its gradient: (a) Convex Martingale Mixture UCB (CMM-UCB): We employ a convex solver for the UCB maximisation and calculate its gradients via differentiable convex optimisation (Agrawal et al., 2019); (b) Analytic Martingale Mixture UCB (AMM-UCB): Weexploit weak Lagrangian duality to obtain an analytic upper bound on the UCB, whose gradient can be computed in closed-form or via standard automatic differentiation procedures.

Fig. 1 highlights a key observation: both of our UCBs are tighter than those used in the state-of-the-art OFUL algorithm (Abbasi-Yadkori et al., 2011) for stochastic linear bandits. We verify this claim empirically in Sec. 8 and prove it in App. C.2. We evaluate CMM-UCB, AMM-UCB, OFUL and various other linear bandit algorithms in several hyperparameter tuning problems (Sec. 8). We find that our tighter UCBs result linear bandit algorithms with better performance.

2 Related Work

Algorithms with regret guarantees have been developed for several variants of the stochastic linear bandit problem. Dani et al. (2008), Abbasi-Yadkori et al. (2009) and Rusmevichientong & Tsitsiklis (2010) proposed algorithms for a linear bandit problem where the action set is a fixed, possibly infinite subset of a finite-dimensional vector space. Auer (2002) and Chu et al. (2011) proposed algorithms for linear bandit problems where the action set has finite cardinality, but may change over time. Abbasi-Yadkori et al. (2011) proposed the OFUL algorithm for linear bandit problems with a changing and possibly infinite action set, which is essentially the problem that we investigate. We consider stochastic linear bandit problems where the reward function is a composition of a possibly non-linear feature map and a linear function. This can be seen as a restricted version of the stochastic kernelised bandit problem, where the kernel feature map is finite-dimensional. Srinivas et al. (2010); Valko et al. (2013); Chowdhury & Gopalan (2017); Camilleri et al. (2021); Salgia et al. (2021) and Li &Scarlett (2022) proposed algorithms with regret guarantees for various kernelised bandit problems.

In the bandit literature, confidence sets and confidence bounds constructed from online (e.g. non-i.i.d.) observation points for unknown linear functions have been proposed by Dani et al. (2008); Rusmevichientong & Tsitsiklis (2010) and Abbasi-Yadkori et al. (2011). Online confidence sets/bounds for unknown functions in separable Hilbert spaces and reproducing kernel Hilbert spaces (RKHSs) have been proposed by Srinivas et al. (2010); Abbasi-Yadkori (2012); Kirschner & Krause (2018) and Durand et al. (2018). Russo & Van Roy (2013) derived online confidence sets for unknown functions belonging to arbitrary function classes.

We use the term 'mixture of martingales', or martingale mixture, to refer to a martingale of the form E v ∼ P [ M t ( v )] , where ( M t ( v ) | t ∈ N ) is a collection of martingales indexed by the variable v ∈ V . Martingale mixtures can be traced back to (Darling & Robbins, 1968; Robbins, 1970), and have been used to construct confidence sequences since at least the work of Lai (1976). Proofs of tail bounds for martingale mixtures typically use the method of mixtures, which was first used by Robbins & Siegmund (1970) and was later popularised by de la Peña et al. (2004, 2009). Methods for martingale mixtures have seen renewed interest in the sequential testing literature (Howard et al., 2020; Kaufmann & Koolen, 2021). Examples of confidence sequences for bandits that use martingale mixtures include the works of Abbasi-Yadkori et al. (2011); Abbasi-Yadkori (2012); Kirschner & Krause (2018); Durand et al. (2018); Neiswanger & Ramdas (2021). Unlike in these examples, we construct confidence sequences based on adaptive martingale mixtures ( E v ∼ P t [ M t ( v )] | t ∈ N ) , where the mixture distribution P t can be refined as more data is acquired at each time t .

3 Problem Statement and Background

We consider a problem in which a learner plays a game over a sequence of T rounds, where T may not be known in advance. In each round t , the learner observes an action set A t and must choose an action a t ∈ A t . The learner then receives a reward r t = ϕ ( a t ) ⊤ θ ∗ + ϵ t . The feature map ϕ : A → R d is a known function that maps actions to d -dimensional feature vectors, where A = ⋃ t A t . θ ∗ ∈ R d is an unknown parameter with Euclidean norm bounded by some known B > 0 , i.e. ∥ θ ∗ ∥ 2 ≤ B . ϵ 1 , ϵ 2 , . . . , ϵ T are conditionally zero-mean σ -sub-Gaussian noise variables. These assumptions on θ ∗ and ϵ 1 , ϵ 2 , . . . , ϵ T are standard in the linear bandit literature, see e.g. (Abbasi-Yadkori et al., 2011). While our regret analysis applies to any action sets, our algorithms focus on the case where the action sets A t are continuous subsets of R d A .

The goal of the learner is to choose a sequence of actions that maximises the total expected reward, which is equal to ∑ T t =1 ϕ ( a t ) ⊤ θ ∗ after T rounds. We use cumulative regret, which is the difference between the total expected reward of the learner and the optimal strategy, to evaluate the learner. For a single round, we define the regret as ∆( a t ) = ϕ ( a ∗ t ) ⊤ θ ∗ -ϕ ( a t ) ⊤ θ ∗ , where a ∗ t = argmax a ∈A t { ϕ ( a ) ⊤ θ ∗ } . After T rounds, the cumulative regret is ∆ 1: T = ∑ T t =1 ∆( a t ) .

Confidence Sequences. For any level δ ∈ (0 , 1] , a (1 -δ ) -confidence sequence for the parameter vector θ ∗ is a sequence Θ 1 , Θ 2 , . . . of subsets of R d , such that each Θ t can be calculated using the data a 1 , r 1 , . . . , a t , r t and the sequence satisfies

A confidence sequence Θ 1 , Θ 2 , . . . is thus a sequence of data-dependent confidence sets such that with high probability over the random actions and rewards, θ ∗ ∈ Θ t holds for all t ≥ 1 simultaneously. We remark that the confidence sets Θ t in this paper are random closed sets in the sense of Def. 1.1.1 of (Molchanov, 2005), which implies that the event θ ∈ Θ t is actually measurable for all θ ∈ R d .

4 UCB Algorithms for Linear Bandits

We describe here how to transform confidence sets for θ ∗ into a UCB algorithm for the linear bandit problem. Such algorithms have appeared under various names, such as LinRel (Auer, 2002), LinUCB (Li et al., 2010) and OFUL (Abbasi-Yadkori et al., 2011). We refer to this meta algorithm as LinUCB, and give its pseudo-code in Algorithm 1. When run with our confidence sets, we call this algorithm CMM-UCB resp. AMM-UCB (see Sec. 6).

In each round t , the first step is to construct a confidence set Θ t from the previous observations { ( a k , r k ) } t k =1 . If θ ∗ ∈ Θ t with high probability, then for any action a ,

is an upper confidence bound (UCB) on ϕ ( a ) ⊤ θ ∗ . Taking min θ ∈ Θ t in (1) yields the lower confidence bound LCB Θ t ( a ) . Once a confidence set Θ t has been constructed and the next action set A t +1 has been observed, the LinUCB algorithm chooses the action

which maximises the UCB. The remaining challenge lies in the construction of the confidence sets.

5 Confidence Sequences from Martingale Mixtures

In this section, we develop a general-purpose tail bound for adaptive martingale mixtures. We then specialise our general result to the stochastic linear bandit setting, described in Sec. 3, and construct confidence sequences for the parameter θ ∗ .

5.1 General-Purpose Tail Bound for Adaptive Martingale Mixtures

We consider a general setting in which we are given a filtration ( H t | t ∈ N ) , a sequence of adapted random functions ( Z t : R → R | t ∈ N ) , and a sequence of predictable random variables ( λ t | t ∈ N ) . For f t ∈ R , we define the conditional cumulant generating function ψ t ( f t , λ t ) as

where the expectation E is over Z t ( f t ) and λ t (although λ t is non-random when conditioned on H t -1 ). We use the shorthand f t := ( f 1 , f 2 , . . . , f t ) and λ t := ( λ 1 , λ 2 , . . . , λ t ) . Let

M t ( f t , λ t ) is a slight generalisation of the martingale used in App. B.1 of (Russo & Van Roy, 2013). One can show that for any sequence ( f t | t ∈ N ) , ( M t ( f t , λ t ) | t ∈ N ) is a martingale and E [ M t ( f t , λ t )] = 1 (App. A.1). We will now construct an adaptive martingale mixture.

We call a data-dependent sequence of probability distributions ( P t | t ∈ N ) an adaptive sequence of mixture distributions if: (a) P t is a distribution over f t ∈ R t ; (b) P t is H t -1 -measurable; (c) the distributions are consistent in the sense that their marginals coincide, i.e. ∫ P t ( f t ) df t = P t -1 ( f t -1 ) for all t . These conditions on the sequence of distributions ensure that the martingale mixture ( E f t ∼ P t [ M t ( f t , λ t )] | t ∈ N ) is in fact a martingale. In App. A.1, we verify this and show that E [ E f t ∼ P t [ M t ( f t , λ t )]] = 1 . From here, we can use Ville's inequality for non-negative supermartingales (Ville, 1939) to obtain our general-purpose tail bound.

Theorem 5.1 (Tail Bound for Adaptive Martingale Mixtures) . For any δ ∈ (0 , 1) , any sequence of predictable random variables ( λ t | t ∈ N ) , and any adaptive sequence of mixture distributions ( P t | t ∈ N ) , the following holds with probability at least 1 -δ :

We provide a proof of this result in App. A.2. Note that if each ψ k ( f k , λ k ) in (3) is replaced by an upper bound on ψ k ( f k , λ k ) , the statement of the theorem still holds.

Thm. 5.1 is closely related to the general-purpose anytime PAC-Bayes bound in Thm. 3.1 of (Chugg et al., 2023). The main difference is that our inequality holds for adaptive sequences of mixture distributions or priors. PAC-Bayes bounds with somewhat similar adaptive sequences of mixture distributions/priors have recently been proposed by Haddouche & Guedj (2022, 2023).

5.2 Confidence Sequences for Stochastic Linear Bandits

We now specialise Thm. 5.1 to the stochastic linear bandit setting. For the filtration ( H t | t ∈ N ) , we set H t to be the σ -algebra generated by ( a 1 , r 1 , . . . , a t , r t , a t +1 ) . For reasons that will become clear, we choose Z t ( f t ) = ( f t -ϕ ( a t ) ⊤ θ ∗ ) ϵ t . Since Z ( f t ) is linear in the noise variable ϵ t , ψ t ( f t , λ t ) can be upper bounded using the sub-Gaussian property of ϵ t . We have

With this upper bound on ψ t ( f t , λ t ) , Thm. 5.1 implies that, with probability at least 1 -δ

Since Z f ( f t ) is linear in f t , this integral has a closed-form solution whenever the mixture distribution is a Gaussian P t = N ( µ t , T t ) . Although there is a closed-form solution for any predictable sequence ( λ t | t ∈ N ) (see App. B.1), we choose λ t ≡ 1 /σ 2 , which yields a relatively simple convex quadratic constraint for θ ∗ . Collecting the feature vectors in Φ t := [ ϕ ( a 1 ) , . . . , ϕ ( a t )] ⊤ ∈ R t × d and writing the reward vector r t := [ r 1 , . . . , r t ] ⊤ , we arrive at (see App. B.2)

This inequality has an attractive interpretation. At each step t of the bandit process, the (unknown) ground-truth reward vector Φ ⊤ t θ ∗ lies within a sphere around the observed reward vector r t , with squared radius equal to the RHS of (5). One can think of the mean vector µ t as a prediction of the reward vector r t , given the previous data a 1 , r 1 , . . . , a t -1 , r t -1 , a t . The covariance matrix T t can be thought of as the uncertainty associated with the prediction µ t . If µ t is a good prediction of r t , then the quadratic 'prediction error' term in (5) will be close to 0, and we can afford to choose T t close to zero to minimise the log determinant term. In this situation, (5) can give a much tighter constraint than the naive bound ∼ tσ 2 , especially when σ is a pessimistic upper bound on the true sub-Gaussian parameter. This naive bound follows from the observation that ∥ Φ t θ ∗ -r t ∥ 2 = ∥ ϵ t ∥ 2 , where ϵ t = ( ϵ 1 , . . . , ϵ t ) . Combining the constraint in (5) with our assumption ∥ θ ∗ ∥ 2 ≤ B yields our confidence sequence for θ ∗ .

Corollary 5.2 (Martingale Mixture Confidence Sequence) . For any adaptive sequence of mixture distributions P t = N ( µ t , T t ) , it holds with probability at least 1 -δ that for all t ≥ 1 simultaneously, θ ∗ lies in the set

where we define R 2 MM ,t as the right-hand-side of Eq. (5).

The boundaries of the constraints in (6) are both ellipsoids in R d , which means that each Θ t is the intersection of (the interiors of) two ellipsoids.

6 Martingale Mixture UCB Algorithms

In this section, we describe our CMM-UCB and AMM-UCB algorithms, which are two different implementations of LinUCB (Algorithm 1) with our confidence sequence from Corollary 5.2.

6.1 UCB Computation and Optimisation

To run the LinUCB action selection rule with our confidence sequence, we need to be able to maximise UCB Θ t ( a ) with respect to a . The value of the UCB at the action a is the solution of

This is a convex optimisation problem, which can be efficiently solved via convex programming. If the action sets have finite cardinality, UCB Θ t ( a ) can be maximised by solving (7) for each a ∈ A t and then comparing the solutions. If the action sets are continuous subsets of R d A , then exact maximisation of UCB Θ t ( a ) is (in general) infeasible. For example, when the feature map ϕ is linear in a , UCB Θ t ( · ) is the maximum over a set of linear functions, which is a convex function of a (see Eq. (3.7) in Sec. 3.2.3 of Boyd & Vandenberghe (2004)). Since maximisation of a convex function is (in general) NP-hard, exact maximisation of UCB Θ t ( a ) is also (in general) NP-hard. For this reason, when the action sets are continuous subsets of R d A (and ϕ is differentiable), we approximately maximise UCB Θ t ( a ) via gradient-based local search.

6.2 Convex Martingale Mixture UCB

Our Convex Martingale Mixture UCB (CMM-UCB) algorithm is based on computing (7) using numerical convex (conic) solvers from the CVXPY library (Diamond & Boyd, 2016; Agrawal et al., 2018). Note that (7) is already stated in a conic form, which is favourable for conic solvers (Boyd & Vandenberghe, 2004). Solving (7) numerically gives the tightest UCBs that can be obtained from our confidence sequence. To compute the gradient of UCB Θ t ( a ) with respect to the action a , we use recently developed methods for differentiating conic programs at their optimum (Agrawal et al., 2019), which are implemented in the cvxpylayers library.

6.3 Analytic Martingale Mixture UCB

Our Analytic Martingale Mixture UCB (AMM-UCB) algorithm uses an analytic upper bound on the solution of (7). The resulting analytic confidence bounds are looser than the numerical confidence bounds used by CMM-UCB, but are cheaper to evaluate and maximise. Theorem 6.1 states our upper bound on the solution of (7).

Theorem 6.1 (Analytic UCB) . For all α > 0 , we have

✶ In App. C.1, we derive this analytic UCB by partial optimisation of the Lagrangian dual function. Using strong duality, one can show that the analytic UCB minimised with respect to α is equal to UCB Θ t ( a ) . Due to the closed-form expression of the analytic UCB in (8), its gradient with respect to a can be computed using standard automatic differentiation packages.

6.4 Choosing the Mixture Distributions

Both of our algorithms require us to choose mixture distributions P t = N ( µ t , T t ) . The mixture distributions play a role similar to the priors used in the PAC-Bayes (Shawe-Taylor & Williamson, 1997; McAllester, 1998; Guedj, 2019; Alquier, 2021) and luckiness (Grünwald, 2007, 2023) frameworks. Our confidence sequences and regret bounds are valid for any sequence of adaptive mixture distributions, but (as seen in Eq. (5)) if better/worse mixture distributions are chosen, then our confidence sequences will get smaller/bigger and our regret guarantees will get tighter/looser.

Here, we describe some sensible choices for the mixture distributions. In order for a sequence of Gaussian mixture distributions ( N ( µ t , T t ) | t ∈ N ) to be a sequence of adaptive mixture distributions (as defined in Section 5.1), we require: (a) µ t and T t can only depend on a 1 , . . . , a t and r 1 , . . . , r t -1 ; (b) the first t -1 elements of µ t must be equal to µ t -1 ; (c) the upper left t -1 × t -1 block of T t must be T t -1 ; (d) T t must be positive (semi-)definite. These conditions are all satisfied if we use a mean vector µ t and covariance matrix T t of the form

where m : A → R is a mean function and k : A×A→ R is a positive-definite kernel function. For the linear bandit problem, it is natural to use a linear mean function m ( a ) = ϕ ( a ) ⊤ θ 0 and a linear kernel function k ( a, a ′ ) = ϕ ( a ) ⊤ Σ 0 ϕ ( a ′ ) (where Σ 0 is symmetric and positive-definite), since the resulting mixture distribution assigns non-zero probability only to those vectors of function values f t that could have been generated by a linear reward function. By direct computation, the Gaussian mixture distribution with this m and k , and with µ t and T t as in (9), is P t = N (Φ t θ 0 , Φ t Σ 0 Φ ⊤ t ) . When θ 0 = 0 and Σ 0 = ✶ , we recover what we call the standard mixture distributions P t = N ( 0 , Φ t Φ ⊤ t ) .

Choosing T t = Φ t Σ 0 Φ ⊤ t has the additional benefit that it allows for cheaper computation of the radius R MM ,t . In App. F, we show that one (only) has to compute the inverse and determinant of a d × d matrix instead of the inverse and determinant of the t × t matrix ✶ + T t /σ 2 . Finally, we remark that the requirement that ( N ( µ t , T t ) | t ∈ N ) is an adaptive sequence of mixture distributions allows for 'more adaptive' choices of µ t and T t . In App E.2, we describe and investigate a method for updating µ t and T t at each round t based on previously observed actions and rewards.

7 Regret Bounds

In this section, we establish cumulative regret bounds for our CMM-UCB and AMM-UCB algorithms. First, we state a data-dependent regret bound that illustrates how the radius of the analytic UCB from Sec. 6.3 influences the regret of both algorithms. Then, we prove a data-independent regret bound which illustrates the worst-case growth rate of the cumulative regret, with explicit dependence on the feature vector dimension d and the number of rounds T . We begin by stating the assumptions (which are standard) under which our regret bounds hold.

Assumption 7.1 (Sub-Gaussian noise) . Let H t denote (the σ -algebra generated by) ( a 1 , r 1 , . . . , a t , r t , a t +1 ) . Each noise variable ϵ t is conditionally zero-mean and σ -sub-Gaussian, which means

Assumption 7.2 (Bounded parameter vector) . For some B > 0 , ∥ θ ∗ ∥ 2 ≤ B .

Assumption 7.3 (Bounded feature vectors) . For some L > 0 , ∥ ϕ ( a ) ∥ 2 ≤ L for all a ∈ A .

Assumption 7.4 (Bounded expected reward) . For some C > 0 , ϕ ( a ) ⊤ θ ∗ ∈ [ -C, C ] for all a ∈ A .

We remark that to run our algorithms and evaluate the data-dependent regret bound in Thm. 7.5, we only need to know (upper bounds on) the sub-Gaussian parameter σ and the norm bound B . Assumption 7.2 and Assumption 7.3 together imply that Assumption 7.4 must hold with C ≤ LB . We nevertheless state it as a separate assumption because this leaves open the possibility that a better (than LB ) value for C is known.

7.1 Data-Dependent Regret Bounds

Several authors (Dani et al., 2008; Abbasi-Yadkori et al., 2011; Russo & Van Roy, 2013) have shown that the cumulative regret of a UCB algorithm can be upper bounded by the sum of the widths of the confidence sets or confidence bounds that it uses. The width of a confidence set Θ t at the action a is the difference between the corresponding UCB and the LCB at a (i.e., max θ ∈ Θ t { ϕ ( a ) ⊤ θ } -min θ ∈ Θ t { ϕ ( a ) ⊤ θ } ). In App. D.1, we show that if a 1 , a 2 , . . . , a T are the actions selected by our CMM-UCB algorithm, then

This gives a data-dependent cumulative regret bound for CMM-UCB. AMM-UCB has a similar data-dependent cumulative regret bound. In App. D.1, we show that if a 1 , a 2 , . . . , a T are the actions selected by our AMM-UCB algorithm, then

where AUCB Θ t ( a ) is the right-hand-side of (8) and ALCB Θ t ( a ) is the equivalent analytic LCB. Since, the analytic UCB/LCB is an upper/lower bound on the numerical UCB/LCB, the bound in Equation (11) also holds for the actions selected by CMM-UCB. By substituting in the expressions for the analytic UCB/LCBs, we obtain the following data-dependent cumulative regret bound for CMM-UCB and AMM-UCB.

Theorem 7.5. Suppose that assumptions 7.1-7.2 hold. For any adaptive sequence of mixture distributions P t = N ( µ t , T t ) , any δ ∈ (0 , 1) , any α > 0 and all T ≥ 1 , with probability at least 1 -δ , the cumulative regret of both CMM-UCB and AMM-UCB is bounded by

A proof is given in App. D.1. This regret bound tells us that if we choose an adaptive sequence of mixture distributions P t = N ( µ t , T t ) , such that the radii R AMM ,t are small, then we can expect to have small cumulative regret.

7.2 Data-Independent Regret Bounds

We now state a data-independent cumulative regret bound for the special case when the adaptive sequence of mixture distributions is P t = N ( 0 , c Φ t Φ ⊤ t ) , and α = σ 2 /c . These mixture distributions are scaled versions of the standard mixture distributions described in Sec. 6.4.

Theorem 7.6. Suppose that assumptions 7.1-7.4 hold. If for any c > 0 , the sequence of mixture distributions is P t = N ( 0 , c Φ t Φ ⊤ t ) , then for all T ≥ 1 , with probability at least 1 -δ , the cumulative regret of both CMM-UCB and AMM-UCB (with α = σ 2 /c ) is bounded by

Proof sketch. Choosing P t = N ( 0 , c Φ t Φ ⊤ t ) and α = σ 2 /c means that the two quadratic terms in R 2 AMM ,t cancel out. We then find a data-independent upper bound for the log det term. Following

Abbasi-Yadkori et al. (2011), the sum of the norms √ ϕ ( a t ) ⊤ (Φ ⊤ t -1 Φ t -1 + α ✶ ) -1 ϕ ( a t ) is upper bounded using an elliptical potential lemma. The result is the data-independent regret bound in the statement of the theorem.

In App. D.2, we give a proof of this special case. In addition, we also treat a more general case when the sequence of mixture distributions is P t = N (Φ t θ 0 , σ 2 0 Φ t Φ ⊤ t ) and α is any positive number. Using Eq. (5) with µ t = Φ t θ 0 and T t = σ 2 0 Φ t Φ ⊤ t , one can interpret θ 0 as a guess for θ ∗ and σ 0 as a guess for the distance between the reward vector r t and the prediction Φ t θ 0 .

Focusing on the dependence on d and T , this regret bound (and the more general one in App. D.2) is at most O ( d √ T ln( T )) , which matches OFUL and is minimax optimal up to the ln( T ) factor. If (upper bounds on) σ 2 , B , L and C are known, then we can evaluate this cumulative regret bound before running the algorithm.

8 Experiments

In all our experiments, we set δ = 0 . 01 . When using our analytic UCBs (Thm. 6.1), we always choose α = σ 2 . Unless stated otherwise, we use the standard mixture distributions P t = N ( 0 , Φ t Φ ⊤ t ) .

0

200

400

600

800

1000

10 0

10 1

Width (log scale)

Scaling With

T

CMM-UCB

OFUL

0

20

40

60

80

100

0

2

4

6

8

Scaling With

d

T

d

8.1 Upper and Lower Confidence Bounds

Compared Methods. We evaluate the following upper/lower confidence bounds: (a) CMM-UCB: our numerical UCBs/LCBs from Sect. 6.2; (b) AMM-UCB: our analytic UCBs/LCBs from Thm. 6.1; (c) OFUL: the UCBs/LCBs used by the OFUL algorithm (Abbasi-Yadkori et al., 2011); (d) Bayes: a Bayesian credible interval constructed from the Bayesian posterior for linear regression with a Gaussian prior and likelihood (see App. E.1 for details).

Experimental Setup. We conduct experiments on randomly generated linear functions of the form f ( x ) = ϕ ( x ) ⊤ θ ∗ , with inputs x ∈ R d X and θ ∗ ∈ R d , the latter drawn from a standard Gaussian distribution and if necessary scaled down to ∥ θ ∗ ∥ 2 ≤ 10 =: B . For the feature map ϕ : R d X → R d , we use Random Fourier Features (cf. Algorithm 1 of (Rahimi & Recht, 2007)). We investigate the properties of upper and lower confidence bounds constructed from random data sets { ( x t , y t ) } T t =1 , where y t = ϕ ( x t ) ⊤ θ ∗ + ϵ t , ϵ t ∼ N (0 , σ 2 ) and σ = 0 . 1 .

UCB/LCB Tightness. Fig. 1 shows the data { ( x t , y t ) } T t =1 and the UCBs/LCBs of CMM-UCB, AMM-UCB and OFUL for a randomly generated linear function with d X = 1 and d = 20 . In this example, the confidence bounds of CMM-UCB are slightly tighter than those of AMMUCB, which are considerably tighter than those of OFUL. Next, we investigate the tightness of the confidence bounds for functions with higher dimensional inputs ( d X = 10 ), a range of data set sizes ( T ∈ { 1 , 2 , 5 , 10 , 20 , 50 , 100 , 200 , 500 , 1000 } ) and a range feature vector dimensions ( d ∈ { 1 , 2 , 5 , 10 , 20 , 50 , 100 } ). For each T and d , we sample a random feature map ϕ and weight vector θ of appropriate size. Then, we sample random training data { ( x t , y t ) } T t =1 and random test points { x ′ t } 100 t =1 , where x k and x ′ t are drawn uniformly from the d X -dimensional unit hypercube. Finally, we use the training data to construct confidence bounds with each method and calculate the average width (UCB minus LCB) at the test points. Fig. 2 shows the average width of the CMM-UCB, AMM-UCB and OFUL confidence bounds with: d = 10 and varying T (left), and T = 100 and varying d (right). We observe the same pattern at every d and T : CMM-UCB produces the tightest confidence bounds followed by AMM-UCB and then OFUL.

Effect of the Mixture Distributions. Fig. 4 in App. E.1 shows the confidence bounds of CMMUCB and a Bayesian credible interval for different choices of the mixture distributions/prior P t . For a well-specified prior, either uninformative or informative, the Bayesian credible interval is slightly tighter than the confidence bounds of CMM-UCB. For a misspecified prior, the confidence bounds of CMM-UCB become looser whereas the Bayesian credible interval becomes wrong (not containing the ground-truth function). Here, misspecification refers solely to Bayesian prior misspecification. In Figs. 5 and 6 in App. E.2 we show that adaptive choices of the mixture distributions P t can lead to smaller confidence bounds and smaller cumulative regret in linear bandit problems.

8.2 Linear Bandits

Compared Methods. We compare: (a) CMM-UCB: cf. Sec. 6.2; (b) AMM-UCB: cf. Sec. 6.3; (c) OFUL: the OFUL algorithm (Abbasi-Yadkori et al., 2011), with regularisation parameter λ = α = σ 2 ; (d) IDS: the frequentist Information Directed Sampling (IDS) algorithm (Kirschner & Krause, 2018), specifically the DIDS-F version; (e) Freq-TS: Thompson Sampling with posterior covariance inflation (Agrawal & Goyal, 2013), which we call Frequentist Thompson Sampling.

100

200

300

400

500

t

0.5

0.6

0.7

0.8

0.9

Test Accuracy

Raisin

CMM-UCB

OFUL

IDS

Freq-TS

100

200

300

400

500

t

0.45

0.50

0.55

0.60

0.65

0.70

0.75

0.80

Maternal

100

200

300

400

500

t

0.60

0.65

0.70

0.75

0.80

0.85

0.90

0.95

1.00

Banknotes

Experimental Setup. We investigate whether our tighter upper confidence bounds translate to better UCB algorithms. We use each linear bandit algorithm to optimise the hyperparameters of a kernel Support Vector Machine (SVM) for three classification data sets from the UCI Machine Learning Repository (Dua & Graff, 2017): Raisin (Cinar et al., 2020), Maternal (Ahmed et al., 2020), and Banknotes. The expected reward function f ∗ ( a ) is the average test set accuracy of a kernel SVM trained using an ARD RBF kernel with hyperparameters a = ( C, γ ) , with C the regularisation and γ the length-scales. The observed reward r t is the validation set accuracy at a t . The feature map ϕ is a neural network layer with 20 outputs and random weights. We choose σ = 0 . 05 for the sub-Gaussian parameter and B = 10 , i.e. we assume that f ∗ ( a ) ≈ ϕ ( a ) ⊤ θ ∗ for some ∥ θ ∗ ∥ 2 ≤ 10 .

Results. Fig. 3 compares the average test accuracy (expected reward) obtained by each bandit algorithm for each data set and at each round t = 1 , . . . , 500 . Our CMM-UCB and AMM-UCB methods outperform all other methods. From the reward curves of CMM-UCB, AMM-UCB and OFUL, we observe that CMM-UCB outperforms AMM-UCB, which outperforms OFUL. Therefore, we can conclude that our tighter confidence bounds (compared to OFUL's) lead to UCB algorithms with improved performance.

9 Conclusion

In this paper, we developed a novel tail bound for adaptive martingale mixtures and showed that it can be used to construct tighter confidence sequences for linear bandits. We proved that our CMM-UCB and AMM-UCB algorithms match the worst-case cumulative regret of OFUL, and we found that our tighter confidence sequences allowed CMM-UCB and AMM-UCB to achieve greater average and maximum reward in several hyperparameter tuning problems.

A limitation of our algorithms is that they assume a linear expected reward function, which may not always be a realistic assumption for real-world bandit problems. Our general-purpose tail bound in Thm. 5.1 already allows one to derive confidence sequences for non-linear reward functions by simply choosing Z t ( f t ) = ( f t -f ∗ ( a t )) ϵ t , where f ∗ is the non-linear reward function. In the case where f ∗ lies in a reproducing kernel Hilbert space, the corresponding UCB is still the solution of a convex program. The main challenge in this setting is that the regret bound must now depend on quantities like the effective dimension or the maximum information gain of the kernel, since the dimension d of the feature vectors is d = ∞ for interesting kernels.

We believe that further investigation into the degree to which adaptive mixture distributions can lead to improved performance and regret bounds (see App. E.2) is another exciting topic for future work.

References

Abbasi-Yadkori, Y. Online learning for linearly parametrized control problems . PhD thesis, University of Alberta, 2012.

Abbasi-Yadkori, Y., Antos, A., and Szepesvári, C. Forced-exploration based algorithms for playing in stochastic linear bandits. In COLT Workshop on On-line Learning with Limited Feedback , volume 92, pp. 236, 2009.

Abbasi-Yadkori, Y., Pál, D., and Szepesvári, C. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems , 24, 2011.

Agrawal, A., Verschueren, R., Diamond, S., and Boyd, S. A rewriting system for convex optimization problems. Journal of Control and Decision , 5(1):42-60, 2018.

Agrawal, A., Amos, B., Barratt, S., Boyd, S., Diamond, S., and Kolter, J. Z. Differentiable convex optimization layers. Advances in neural information processing systems , 32, 2019.

Agrawal, S. and Goyal, N. Thompson sampling for contextual bandits with linear payoffs. In International conference on machine learning , pp. 127-135. PMLR, 2013.

Ahmed, M., Kashem, M. A., Rahman, M., and Khatun, S. Review and analysis of risk factor of maternal health in remote area using the internet of things (IoT). In InECCE2019: Proceedings of the 5th International Conference on Electrical, Control & Computer Engineering, Kuantan, Pahang, Malaysia, 29th July 2019 , pp. 357-365. Springer, 2020.

Alquier, P. User-friendly introduction to PAC-Bayes bounds, 2021. URL https://arxiv.org/ abs/2110.11216 .

Auer, P. Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research , 3(Nov):397-422, 2002.

Boyd, S. and Vandenberghe, L. Convex optimization . Cambridge university press, 2004.

Camilleri, R., Jamieson, K., and Katz-Samuels, J. High-dimensional experimental design and kernel bandits. In International Conference on Machine Learning , pp. 1227-1237. PMLR, 2021.

Chowdhury, S. R. and Gopalan, A. On kernelized multi-armed bandits. In International Conference on Machine Learning , pp. 844-853. PMLR, 2017.

Chu, W., Li, L., Reyzin, L., and Schapire, R. Contextual bandits with linear payoff functions. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , pp. 208-214. JMLR Workshop and Conference Proceedings, 2011.

Chugg, B., Wang, H., and Ramdas, A. A unified recipe for deriving (time-uniform) PAC-Bayes bounds. arXiv preprint arXiv:2302.03421 , 2023.

Cinar, I., Koklu, M., and Tasdemir, S. Classification of raisin grains using machine vision and artificial intelligence methods. Gazi Mühendislik Bilimleri Dergisi , 6(3):200-209, 2020.

Cohen, M. C., Lobel, I., and Paes Leme, R. Feature-based dynamic pricing. Management Science , 66(11):4921-4943, 2020.

Dani, V., Hayes, T. P., and Kakade, S. M. Stochastic linear optimization under bandit feedback. In COLT , pp. 355-366, 2008.

Darling, D. and Robbins, H. Some further remarks on inequalities for sample sums. Proceedings of the National Academy of Sciences , 60(4):1175-1182, 1968.

de la Peña, V. H., Klass, M. J., and Leung Lai, T. Self-normalized processes: exponential inequalities, moment bounds and iterated logarithm laws. Annals of Probability , 32:1902-1933, 2004.

de la Peña, V. H., Lai, T. L., and Shao, Q.-M. Self-normalized processes: Limit theory and Statistical Applications . Springer, 2009.

Diamond, S. and Boyd, S. CVXPY: A Python-embedded modeling language for convex optimization. Journal of Machine Learning Research , 17(83):1-5, 2016.

Donsker, M. D. and Varadhan, S. S. Asymptotic evaluation of certain Markov process expectations for large time-iii. Communications on pure and applied Mathematics , 29(4):389-461, 1976.

Dua, D. and Graff, C. UCI machine learning repository, 2017. URL http://archive.ics.uci. edu/ml .

Durand, A., Maillard, O.-A., and Pineau, J. Streaming kernel regression with provably adaptive mean, variance, and regularization. The Journal of Machine Learning Research , 19(1):650-683, 2018.

Grünwald, P. D. The minimum description length principle . MIT press, 2007.

Grünwald, P. D. The e-posterior. Philosophical Transactions of the Royal Society A , 381(2247): 20220146, 2023.

Guedj, B. A primer on PAC-Bayesian learning. In Proceedings of the second congress of the French Mathematical Society , volume 33, 2019. URL https://arxiv.org/abs/1901.05353 .

Haddouche, M. and Guedj, B. Online PAC-Bayes learning. In Advances in Neural Information Processing Systems , volume 35, pp. 25725-25738. Curran Associates, Inc., 2022.

Haddouche, M. and Guedj, B. PAC-Bayes generalisation bounds for heavy-tailed losses through supermartingales. Transactions on Machine Learning Research [TMLR] , 2023. ISSN 2835-8856. doi: 10.48550/ARXIV.2210.00928. URL https://openreview.net/forum?id=qxrwt6F3sf .

Howard, S. R., Ramdas, A., McAuliffe, J., and Sekhon, J. Time-uniform Chernoff bounds via nonnegative supermartingales. Probability Surveys , 17, 2020.

Kaufmann, E. and Koolen, W. M. Mixture martingales revisited with applications to sequential tests and confidence intervals. The Journal of Machine Learning Research , 22(1):11140-11183, 2021.

Kirschner, J. and Krause, A. Information directed sampling and bandits with heteroscedastic noise. In Conference On Learning Theory , pp. 358-384. PMLR, 2018.

Lai, T. L. On confidence sequences. The Annals of Statistics , pp. 265-280, 1976.

Li, L., Chu, W., Langford, J., and Schapire, R. E. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th international conference on World wide web , pp. 661-670, 2010.

Li, Z. and Scarlett, J. Gaussian process bandit optimization with few batches. In International Conference on Artificial Intelligence and Statistics , pp. 92-107. PMLR, 2022.

McAllester, D. A. Some PAC-Bayesian theorems. In Proceedings of the eleventh annual conference on Computational learning theory , pp. 230-234, 1998.

Molchanov, I. S. Theory of random sets , volume 19. Springer, 2005.

Neiswanger, W. and Ramdas, A. Uncertainty quantification using martingales for misspecified Gaussian processes. In Algorithmic Learning Theory , pp. 963-982. PMLR, 2021.

Petersen, K. B., Pedersen, M. S., et al. The matrix cookbook. Technical University of Denmark , 7 (15), 2008.

Rahimi, A. and Recht, B. Random features for large-scale kernel machines. Advances in neural information processing systems , 20, 2007.

Robbins, H. Statistical methods related to the law of the iterated logarithm. The Annals of Mathematical Statistics , 41(5):1397-1409, 1970.

Robbins, H. and Siegmund, D. Boundary crossing probabilities for the Wiener process and sample sums. The Annals of Mathematical Statistics , pp. 1410-1429, 1970.

Rusmevichientong, P. and Tsitsiklis, J. N. Linearly parameterized bandits. Mathematics of Operations Research , 35(2):395-411, 2010.

Russo, D. and Van Roy, B. Eluder dimension and the sample complexity of optimistic exploration. Advances in Neural Information Processing Systems , 26, 2013.

Salgia, S., Vakili, S., and Zhao, Q. A domain-shrinking based Bayesian optimization algorithm with order-optimal regret performance. Advances in Neural Information Processing Systems , 34: 28836-28847, 2021.

Shawe-Taylor, J. and Williamson, R. C. A PAC analysis of a Bayesian estimator. In Proceedings of the tenth annual conference on Computational learning theory , pp. 2-9, 1997.

Sherman, J. and Morrison, W. J. Adjustment of an inverse matrix corresponding to a change in one element of a given matrix. The Annals of Mathematical Statistics , 21(1):124-127, 1950.

Srinivas, N., Krause, A., Kakade, S., and Seeger, M. Gaussian process optimization in the bandit setting: No regret and experimental design. In Proc. International Conference on Machine Learning (ICML) , 2010.

Valko, M., Korda, N., Munos, R., Flaounas, I., and Cristianini, N. Finite-Time Analysis of Kernelised Contextual Bandits. In Uncertainty in Artificial Intelligence , 2013.

Ville, J. Etude critique de la notion de collectif. Bull. Amer. Math. Soc , 45(11):824, 1939.

A Proof of the General-Purpose Tail Bound for Adaptive Martingale Mixtures

A.1 Verifying Martingale Properties

First, we recall the definition of ( M t ( f t , λ t ) | t ∈ N ) in Eq. (3). We are given a filtration ( H t | t ∈ N ) , a sequence of adapted random functions ( Z t : R → R | t ∈ N ) , and a sequence of predictable random variables ( λ t | t ∈ N ) .

A filtration is an increasing sequence of σ -algebras H 0 ⊆ H 1 ⊆ H 2 · · · . Each σ -algebra H t represents the information available at time t . ( Z t : R → R | t ∈ N ) being a sequence of adapted (to the filtration ( H t | t ∈ N ) ) functions means that, when conditioned on H t , Z t is no longer random. ( λ t | t ∈ N ) being a sequence of predictable random variables means that, when conditioned on H t -1 , λ t is no longer random.

For a sequence of real numbers ( f t : t ∈ N ) , we define

where ψ t ( f t , λ t ) is the conditional cumulant generating function

Lemma A.1. For any sequence of real numbers ( f t | t ∈ N ) , ( M t ( f t , λ t ) | t ∈ N ) is a martingale and E [ M t ( f t , λ t )] = 1 for all t ∈ N .

Proof. For t = 1 , we have

Using the tower rule of conditional expectation, we also have

Now, we verify the martingale property. For any t ≥ 2 , we have

Using the tower rule again, we have for any t ≥ 2

Therefore, we have

Lemma A.2. For any adaptive sequence of mixture distributions ( P t | t ∈ N ) , ( E f t ∼ P t [ M t ( f t , λ t )] | t ∈ N ) is a martingale and E [ E f t ∼ P t [ M t ( f t , λ t )]] = 1 for all t ∈ N .

Proof. For any t ≥ 1 , since M t ( f t , λ t ) is non-negative and P t is H t -1 -measurable, Tonelli's theorem implies

The requirement that the distributions P 1 , P 2 , . . . have coinciding marginals, i.e. ∫ P t ( f t )d f t = P t -1 ( f t -1 ) , means that for all t ≥ 2

Using these two results, and the fact that ( M t ( f t , λ t ) | t ∈ N ) is a martingale for any sequence ( f t | t ∈ N ) , we now verify that the martingale mixture ( E f t ∼ P t [ M t ( f t , λ t )] | t ∈ N ) is a martingale with expected value 1. For t = 1 , we have

Using the tower rule as before, this also means that E [ E f 1 ∼ P 1 [ M 1 ( f 1 , λ 1 )] ] = 1 . For any t ≥ 2 , we have

Using the tower rule one more time, we have for any t ≥ 2

Therefore, we have

A.2 Proof of Theorem 5.1

To prove Thm. 5.1, we use Ville's inequality for non-negative supermartingales (Ville, 1939), which can be thought of as a time-uniform version of Markov's inequality. Instead of the martingale property E [ M t |H t -1 ] = M t -1 , a supermartingale satisfies E [ M t |H t -1 ] ≤ M t -1 , which means that any martingale is also a supermartingale. Therefore, Ville's inequality for non-negative supermartingales also holds for the non-negative martingale in Lemma A.2.

Lemma A.3 (Ville's inequality for non-negative supermartingales (Ville, 1939)) . Let ( M t | t ∈ N ) be a non-negative supermartingale with respect to the filtration ( H t | t ∈ N ) , which satisfies M 0 = 1 . For any δ ∈ (0 , 1] , it holds with probability at least 1 -δ :

Proof of Thm. 5.1. We choose an arbitrary δ ∈ (0 , 1] . From Lemma A.2, for any adaptive sequence of mixture distributions ( P t | t ∈ N ) , ( E f t ∼ P t [ M t ( f t , λ t )] | t ∈ N ) is a martingale and E [ E f t ∼ P t [ M t ( f t , λ t )]] = 1 . In addition, ( E f t ∼ P t [ M t ( f t , λ t )] | t ∈ N ) is clearly non-negative. Therefore, using Lemma A.3, with probability at least 1 -δ

Taking the logarithm of both sides yields the statement of Thm. 5.1.

B Closed-Form Gaussian Integration

Here, we calculate the integral in the inequality (see beginning of Sec. 5.2):

First, we rearrange the integrand into a more convenient form. For every k , using r k = ϕ ( a k ) ⊤ θ ∗ + ϵ k , we have

Therefore, we have that

Equation (12) can now be re-written as

In the special case where λ t ≡ 1 /σ 2 , we have λ k -σ 2 λ 2 k = 0 , which means that 1 2 ( λ k -σ 2 λ 2 k )( f k -ϕ ( a k ) ⊤ θ ∗ ) 2 disappears. In addition, ( λ k / 2)( ϕ ( a k ) ⊤ θ ∗ -r k ) 2 does not depend on f k , so it can be moved outside the integral.

B.1 General λ t

Let Λ t be the t × t diagonal matrix with diagonal elements λ 1 , λ 2 , . . . , λ t . Starting from (13), taking the logarithm of both sides, rearranging terms and then writing everything in matrix notation, we arrive at

The expected value inside the logarithm can be re-written as

We will calculate the integral by 'completing the square', i.e. rewriting the exponent in the form -1 2 ( f t -b ) ⊤ A ( f t -b ) + c , to recover the integral of a Gaussian density function. For a symmetric matrix A , we have

We also have

We now equate coefficients to find A , b and c . We find that A is

Note that A is symmetric. We find that b is

Finally, we find that c is

Now, we can rewrite and calculate the integral in (15) as

Substituting this into (14), we obtain the constraint

✶ Note that θ ∗ appears on both the left-hand-side and right-hand-side of this inequality. However, when Λ t -σ 2 Λ 2 t is the zero matrix (e.g. when λ t ≡ 1 /σ 2 ), all the θ ∗ -dependent terms on the right-hand-side disappear.

B.2 The Special Case λ t ≡ 1 /σ 2

Starting from (13), choosing λ t ≡ 1 /σ 2 , taking the logarithm of both sides and then rearranging terms, we arrive at

For any t × t covariance matrix T , let Z ( T ) denote the normalising constant of a Gaussian distribution with covariance T , so

For any t -dimensional vectors x and µ , and any t × t covariance matrix T , let p ( x | µ , T ) denote the density function of a Gaussian distribution with mean µ and covariance T , evaluated at x . This means that

We will use the product of Gaussians trick from (Petersen et al., 2008) (Section 8.1.8, Equation 371), which states

where

We have that

C Computing Upper Confidence Bounds

First, we state and prove some useful lemmas.

(Φ t θ -r t ) ⊤ (Φ t θ -r t )+ α θ ⊤ θ -R 2 MM ,t -αB 2 = ( θ -̂ θ α,t ) ⊤ ( Φ ⊤ t Φ t + α ✶ ) ( θ -̂ θ α,t ) -R 2 AMM ,t , where R 2 MM ,t is the squared radius quantity from Cor. 5.2 and

Proof. For any symmetric matrix A , we have

We also have

We can now find A , b and c by equating coefficients. We find that which is a symmetric matrix. We have

Finally, we have

Therefore, we have shown that

Lemma C.2. For any symmetric, positive-definite matrix A ∈ R d × d , any vectors a , b ∈ R d , any R > 0 , and any η < 0 ,

Proof. Let

The gradient and Hessian of f are

Since A is positive-definite and η < 0 , ∂ 2 ∂ θ 2 f ( θ ) is negative-definite for all θ ∈ R d . Therefore, any solution θ ∗ of ∂ ∂ θ f ( θ ) = 0 must be a maximiser of f ( θ ) . There is a unique solution, which is

Lemma C.3. For any symmetric, positive-definite matrix A ∈ R d × d , any vectors a , b ∈ R d , and any R > 0 ,

Proof. Let

The maximum is

The first and second derivatives of g are

Since A is positive-definite, d 2 d η 2 g ( η ) is positive for all η < 0 . Therefore, any negative solution η ∗ of d d η g ( η ) = 0 must be a minimiser of g ( η ) . There is a unique (negative) solution, which is

The minimum is

C.1 Analytic UCBs

Here, we prove Theorem 6.1, which states that for all α > 0 :

✶ Θ t is the confidence set at time t in our confidence sequence from Cor. 5.2 and R MM ,t is the radius from Cor. 5.2 and Eq. (5). As well as proving this statement, we will also show that if Θ t has an interior point, then when the right-hand-side of (18) is optimised with respect to α > 0 , the inequality in (18) becomes an equality, i.e.

Proof of Thm. 6.1. We use weak Lagrangian duality to prove the upper bound and strong Lagrangian duality to prove the second part. The convex optimisation problem max θ ∈ Θ t { ϕ ( a ) ⊤ θ } can be stated as

Rewriting both constraints in the form f ( θ ) ≤ 0 , we can see that the Lagrangian for this problem is

η 1 and η 2 are called the Lagrange multipliers. The Lagrange dual function (or just dual function) is

By weak duality, for any η 1 , η 2 ≤ 0 , the dual function is an upper bound on the solution of the primal problem in (20), i.e. for any η 1 , η 2 ≤ 0

Alternatively, (21) can be verified by starting from the inequality ϕ ( a ) ⊤ θ ≤ L ( θ , η 1 , η 2 ) for all θ ∈ Θ t , η 1 ≤ 0 , and η 2 ≤ 0 . The challenge is to set the Lagrange multipliers such that the dual function has a closed-form expression while being as close as possible to its minimum value min η 1 ,η 2 ≤ 0 { g ( η 1 , η 2 ) } . We will now show that for any α > 0 , min η ≤ 0 { g ( η, αη ) } has a closedform solution, which is the right-hand-side of (18). The Lagrangian, evaluated with the Lagrange multipliers η and αη , is

Using Lemma C.1, the Lagrangian can be rewritten as

Using Lemma C.3, we have

✶ This concludes the proof of Theorem 6.1. To prove (19), we use strong duality. Clearly min α> 0 min η ≤ 0 { g ( η, αη ) } = min η 1 ,η 2 ≤ 0 { g ( η 1 , η 2 ) } , so if we optimise the upper bound in (18) with respect to α , then we will recover the minimum of the dual function. If strong duality holds, then the minimum of the dual function is equal to max θ ∈ Θ t { ϕ ( a ) ⊤ θ } . Since max θ ∈ Θ t { ϕ ( a ) ⊤ θ } is a convex optimisation problem, we can use Slater's condition to obtain a sufficient condition for strong duality to hold. In particular, if Θ t has an interior point, then strong duality holds, which means

One can follow the same steps, with a few minor modifications, to prove a similar statement for lower confidence bounds. For all α > 0

If Θ t has an interior point, then

C.2 OFUL vs AMM-UCB (and CMM-UCB)

We will now show that for any value of the parameter α , we can choose a sequence of Gaussian mixture distributions, such that the confidence bounds of AMM-UCB (and therefore also CMM-UCB) are always tighter than the confidence bounds of OFUL (Abbasi-Yadkori et al., 2011).

To do this, we will use the following lemma.

Lemma C.4. For any γ > 0 , v ∈ R t and M ∈ R t × d , we have

Proof. We start with the identity

By post-multiplying both sides with ( M ⊤ M + γ ✶ ) -1 and pre-multiplying both sides with ( MM ⊤ + γ ✶ ) -1 , we obtain

Now, using (22), we have

With v = r t and M = Φ t , we obtain

We will also use the fact that, due to the Weinstein-Aronszajn identity, for any γ > 0

det( γ Φ ⊤ t Φ t + ✶ ) = det( γ Φ t Φ ⊤ t + ✶ ) . (23) For any α > 0 (in (Abbasi-Yadkori et al., 2011), what we call α is called λ ), the OFUL UCB states that

For any α > 0 and any δ ∈ (0 , 1] , this statement holds with probability at least 1 -δ for all t ≥ 0 and all a ∈ A . By comparison, our AMM-UCB holds uniformly over all t ≥ 0 , all a ∈ A and all α > 0 (i.e. we could optimise the AMM-UCB with respect to α in a data-dependent manner, which would yield our CMM-UCB).

Notice that for any history a 1 , r 1 , a 2 , r 2 , . . . and any α > 0 , the OFUL UCB is the same as our AMM-UCB, except that our AMM radius quantity R AMM ,t is replaced with R OFUL ,t . The same is true for the LCBs of OFUL and AMM-UCB (with the same R OFUL ,t ), so we will only focus on the UCBs. We will now show that for any history a 1 , r 1 , a 2 , r 2 , . . . and any α > 0 , we can chose a sequence of Gaussian mixture distributions such that R AMM ,t ≤ R OFUL ,t . This means that the UCBs of our CMM-UCB and AMM-UCB algorithms are never worse than the OFUL UCB.

Without loss of generality, suppose we choose α = σ 2 /c , for some c > 0 . With this choice, the OFUL radius is

For any α > 0 and a Gaussian mixture distribution P t = N ( µ t , T t ) , the squared AMM-UCB radius is

For AMM-UCB, we will use α = σ 2 /c and the scaled standard mixture distributions P t = N ( 0 , c Φ t Φ ⊤ t ) for each t . With these choices, and using Lemma C.4 and (23), the squared AMM-UCB radius is

Using the basic inequality √ a + b ≤ √ a + √ b for a, b ≥ 0 , we have

Therefore, the confidence bounds of AMM-UCB, with α = σ 2 /c and P t = N ( 0 , c Φ t Φ ⊤ t ) , are never looser than the confidence bounds of OFUL with an arbitrary α = σ 2 /c . Since ln ( det ( c σ 2 Φ ⊤ t Φ t + ✶ )) +2ln ( 1 δ ) and B 2 /c are strictly positive, there is actually a strict inequality. This means that the AMM-UCB (and CMM-UCB) confidence bounds are always strictly tighter than the OFUL confidence bounds.

Note that P t = N ( 0 , c Φ t Φ ⊤ t ) is not necessarily the best choice for the mixture distribution. With a better choice of the mixture distribution, e.g. a mixture distribution that is chosen using some prior knowledge about the expected reward function and/or refined using previously observed rewards, R AMM ,t will be smaller and the gap between AMM-UCB and OFUL will be greater.

D Cumulative Regret Bounds

In this section, we prove the cumulative regret bounds stated in Section 7. We prove the datadependent regret bound in Thm. 7.5. We also prove the data-independent regret bound in Thm. 7.6 and another data-independent regret bound, which holds for more general choices of the mixture distributions and the α parameter.

For convenience, we use some more compact notation in this section. For a symmetric positive semi-definite matrix A and vector x , let

Before presenting the proof of the main results, we state some useful lemmas.

Lemma D.1 (Donsker-Varadhan Change of Measure (Donsker & Varadhan, 1976)) . For any set X , any measurable function h : X → R and any probability distribution P ∈ P ( X ) (i.e. any distribution on X ), such that E x ∼ P [ e h ( x ) ] < ∞ , we have

By rearranging (25), we have

Lemma D.2 (Determinant-Trace Inequality (Abbasi-Yadkori et al., 2011)) . If assumption 7.3 holds (i.e. ∥ ϕ ( a ) ∥ 2 ≤ L ), then for any γ > 0

The Determinant-Trace Inequality in Lemma 10 of (Abbasi-Yadkori et al., 2011) is stated in the form

Lemma D.3. For any σ > 0 and any σ 0 > 0 , define Σ t = ( 1 σ 2 Φ ⊤ t Φ t + 1 σ 2 0 ✶ ) -1 . We have

Since det ( γ Φ ⊤ t Φ t + ✶ ) = det ( Φ ⊤ t Φ t +(1 /γ ) ✶ ) / det ((1 /γ ) ✶ ) , the statement in (27) follows from (28).

Proof. Since Σ t is positive-definite, its trace is positive. Now

Lemma D.4. For any σ > 0 and any σ 0 > 0 , the matrix Σ t = ( 1 σ 2 Φ ⊤ t Φ t + 1 σ 2 0 ✶ ) -1 satisfies

Proof. Let { γ i } d i =1 denote the eigenvalues of Φ ⊤ t Φ t . Since Φ ⊤ t Φ t is positive semi-definite, its eigenvalues are real and non-negative. From the definition of eigenvalues, one can verify that the eigenvalues of Σ t are { σ 2 γ i + σ 2 /σ 2 0 } d i =1 . Using this, we have

Lemma D.5. Let ϵ t denote the vector containing the first t noise variables (so r t = Φ t θ ∗ + ϵ t ). For any α > 0 , we have

Proof. Using r t = Φ t θ ∗ + ϵ t , we have

and

Therefore and

Theorem D.6 (Self-Normalised Bound for Vector-Valued Martingales (Theorem 1 of (Abbasi-Yadkori et al., 2011))) . Let ( H t | t ≥ 0) be a filtration. Let ( ϵ t | t ≥ 1) be a real-valued stochastic process such that ϵ t is H t -measurable and ϵ t is conditionally σ -sub-Gaussian for some σ > 0 . Let ( ϕ ( a t ) | t ≥ 1) be an R d -valued stochastic process such that ϕ ( a t ) is H t -1 -measurable. For any δ ∈ (0 , 1] and any α > 0 , with probability at least 1 -δ

Lemma D.7. For any symmetric positive semi-definite matrix A with largest eigenvalue γ max , we have

Proof. Let { γ i } d i =1 and { v i } d i =1 be the eigenvalues and eigenvectors of A . Since { v i } d i =1 form a basis, there are constants { c i } d i =1 such that x = ∑ d i =1 c i v i . We have

Proof. Since ln(1 + x ) / ln(2) is monotonically increasing in x , we only need to prove that x ≤ ln(1 + x ) / ln(2) for all x ∈ [0 , 1] . For any positive constant a , the function a ln(1 + x ) is concave on the domain [0 , 1] . Therefore, if x ≤ a ln(1 + x ) at the end points x = 0 and x = 1 , then x ≤ a ln(1 + x ) for every x ∈ [0 , 1] . At x = 0 , we have a ln(1 + x ) = 0 for any a , which means we can choose the smallest a such that 1 ≤ a ln(1 + 1) . By rearranging this inequality, we obtain a ≥ 1 / ln(2) .

D.1 Data-Dependent Regret Bound

First, we show that the cumulative regret of both of our algorithms can be upper bounded by the sum of the widths of the UCB/LCBs that they use. Let

In words, UCB Θ t ( a ) and LCB Θ t ( a ) are the upper and lower confidence bounds used by CMM-UCB (evaluated at a ). Similarly, let

AUCB Θ t ( a ) = ϕ ( a ) ⊤ ̂ θ α,t + R AMM ,t ∥ ϕ ( a ) ∥ (Φ ⊤ t Φ t + α ✶ ) -1 , ALCB Θ t ( a ) = ϕ ( a ) ⊤ ̂ θ α,t -R AMM ,t ∥ ϕ ( a ) ∥ (Φ ⊤ t Φ t + α ✶ ) -1 . AUCB Θ t ( a ) and ALCB Θ t ( a ) are the analytic upper and lower confidence bounds used by AMMUCB. Lemma D.9 shows that the cumulative regret of CMM-UCB and AMM-UCB can be upper bounded by the sum of the widths (UCB minus LCB) of the confidence bounds that they use.

Lemma D.9. Suppose the actions a 1 , a 2 , . . . are selected by the CMM-UCB algorithm. For any adaptive sequence of mixture distributions P t = N ( µ t , T t ) and any δ ∈ (0 , 1] , with probability at least 1 -δ

Suppose the actions a 1 , a 2 , . . . are selected by the AMM-UCB algorithm. For any adaptive sequence of mixture distributions P t = N ( µ t , T t ) and any δ ∈ (0 , 1] , with probability at least 1 -δ

Proof. Using Cor. 5.2 (i.e. the fact that Θ 1 , Θ 2 , . . . is a confidence sequence), for any adaptive sequence of mixture distributions P t = N ( µ t , T t ) and any δ ∈ (0 , 1] , with probability at least 1 -δ

Using Thm. 6.1, this implies

Let a 1 , a 2 , . . . be the actions selected by CMM-UCB, i.e. a t = argmax a ∈A t { UCB Θ t -1 ( a ) } . Then, with probability at least 1 -δ , we have

Now, let a 1 , a 2 , . . . be the actions selected by AMM-UCB, i.e. a t = argmax a ∈A t { AUCB Θ t -1 ( a ) } . Then, with probability at least 1 -δ , we have

Since ∀ α > 0 , a ∈ A and t ≥ 1 , AUCB Θ t -1 ( a ) ≥ UCB Θ t -1 ( a ) and ALCB Θ t -1 ( a ) ≤ LCB Θ t -1 ( a ) , (29) implies that (30) also holds when a 1 , a 2 , . . . are the actions selected by CMMUCB.

Proof of Theorem 7.5. We start by using Lemma D.9. Suppose a 1 , a 2 , . . . are the actions selected by CMM-UCB or AMM-UCB. For any adaptive sequence of mixture distributions P t = N ( µ t , T t ) and any δ ∈ (0 , 1] , with probability at least 1 -δ

Using the definitions of AUCB Θ t -1 ( a t ) and ALCB Θ t -1 ( a t ) , we have

D.2 Data-Independent Regret Bound

To establish data-independent regret bounds, we first prove data-independent upper bounds on the radius R AMM ,t and the norms ∥ ϕ ( a t ) ∥ (Φ ⊤ t -1 Φ t -1 + α ✶ ) -1 . Then, we take the data-dependent regret bound in Lemma D.9 and substitute in these bounds on the radius and the norms.

D.2.1 Bounding the Radius

Lemma D.10. If, for any c > 0 , the sequence of mixture distributions is P t = N ( 0 , c Φ t Φ ⊤ t ) and α = σ 2 /c , then

Proof. In Equation (24), we already saw that for this choice of α and the mixture distributions, we have

✶ To obtain a data-independent upper bound on the radius, all that remains is to upper bound ln ( det ( c σ 2 Φ ⊤ t Φ t + ✶ )) by a data-independent quantity. Using Lemma D.2, we have

Therefore

Lemma D.11. If, for any θ 0 ∈ R d and any σ 0 > 0 , the sequence of mixture distributions is P t = N (Φ t θ 0 , σ 2 0 Φ t Φ ⊤ t ) , then for any δ ∈ (0 , 1] and any α > 0 , with probability at least 1 -δ , for all t ≥ 1

Proof. In App. B.2 (see Equation 16), we saw that the squared radius R 2 MM ,t can be written as

Using the substitution Φ t θ = f t , (32) is equivalent to

✶ Using the Donsker-Varadhan change of measure inequality (specifically (26)), the first term on the right-hand-side of (33) is equal to

The bound on R 2 MM ,t in (34) becomes

This means that

If we evaluate this at any specific distribution Q , we obtain an upper bound on the infimum over Q . We choose Q = N ( θ ∗ , Σ t ) , where Σ t = ( 1 σ 2 Φ ⊤ t Φ t + 1 σ 2 0 ✶ ) -1 . Combining everything so far, we have

Finally, using Lemma D.5, then Theorem D.6 and Lemma D.7, and then Lemma D.2, for any δ ∈ (0 , 1] and any α > 0 , with probability at least 1 -δ , for all t ≥ 0 simultaneously

Substituting this into (35), we have

D.2.2 Bounding the Sum of Norms

We use the following upper bound on the sum of the squared norms.

Lemma D.12 (Lemma 11 of (Abbasi-Yadkori et al., 2011)) . For any α > 0 , we have

In Lemma 11 of (Abbasi-Yadkori et al., 2011), 1 / ln(2) ≈ 1 . 44 is replaced with 2. We achieve an improved constant by using Lemma D.8 instead of the looser bound min(1 , x ) ≤ 2 ln(1 + x ) , for x ≥ 0 .

D.2.3 Regret Bounds

We are now ready to prove our data-independent regret bounds.

Proof of Theorem 7.6. Following the same steps as in the proof of Lemma D.9, we can also obtain the following data-dependent bound on the per-round regret for actions selected by CMM-UCB or AMM-UCB. For the mixture distributions P t = N ( 0 t , c Φ t Φ ⊤ t ) , α = σ 2 /c and any δ ∈ (0 , 1] , with probability at least 1 -δ

✶ From Assumption 7.4 ( ϕ ( a ) ⊤ θ ∗ ∈ [ -C, C ] ), we have another bound on the per-round regret

The combination of (36) and (37) yields

Starting with the Cauchy-Schwarz inequality, we have

We will now use the upper bound on R 2 AMM ,t -1 from Lemma D.10. Let U 2 AMM ,t -1 denote this upper bound (i.e. the right-hand-side of (31)). We have

Finally, using the bound on the sum of norms in Lemma D.12, we have

Now, we state and prove a cumulative regret bound that holds for more general choices of the mixture distributions and the parameter α .

Theorem D.13. Suppose that assumptions 7.1-7.4 hold. If, for any θ 0 ∈ R d and any σ 0 > 0 , the sequence of mixture distributions is P t = N (Φ t θ 0 , σ 2 0 Φ t Φ ⊤ t ) , then for any δ ∈ (0 , 1 / 2] and any α > 0 , with probability at least 1 -2 δ , for all T ≥ 1 simultaneously, the cumulative regret of CMM-UCB and AMM-UCB is bounded by

where

Proof. Following the proof of Theorem 7.6, we obtain (with high probability)

This time, we use the bound on the radius from Lemma D.11. Let U AMM ,T -1 denote this bound on the radius (i.e. the square root of the right-hand-side of (39)). Also, note that this bound on the radius holds with probability at 1 -δ . Since U AMM ,T -1 is monotonically increasing with T , we have

Finally, we use Lemma D.12 to obtain

We used two inequalities that each hold with probability at least 1 -δ . By a union bound argument, the cumulative regret bound holds with probability at least 1 -2 δ .

E Additional Experiments

In this section, we present the results of some additional experiments in which we investigate the effect of the mixture distributions (or priors) on our upper and lower confidence bounds.

E.1 The Effect of The Mixture Distributions

We investigate how our CMM-UCB upper and lower confidence bounds behave when we provide an uninformative but well-specified mixture distribution/prior, an informative and well-specified mixture distribution/prior, and a misspecified mixture distribution/prior. Here misspecification refers to prior misspecification in a Bayesian sense. For reference, we compare the behaviour of our upper and lower confidence bounds with the upper and lower limits of a Bayesian credible interval.

For a fair comparison with CMM-UCB, we attempt to construct a Bayesian credible interval that holds with high probability for all rounds t ≥ 0 . However, whilst the CMM-UCB confidence set holds with high probability over the random draw of the data a 1 , r 1 , a 2 , r 2 , . . . , the Bayesian credible interval holds with high probability over the random draw of θ ∗ from a prior (for fixed data a 1 , r 1 , a 2 , r 2 , . . . ).

For the Bayesian credible interval, we use a Gaussian prior and assume θ ∗ ∼ N ( µ 0 , Σ 0 ) . We assume a Gaussian likelihood function, i.e. rewards are of the form r t = ϕ ( a t ) ⊤ θ + ϵ t , where ϵ t ∼ N (0 , σ 2 ) . The Bayesian posterior for θ ∗ is another Gaussian N ( µ t , Σ t ) , where

Using Bayes' rule, at any round t , we have θ ∗ ∼ N ( µ t , Σ t ) . Therefore

where χ 2 ( d ) is a chi-squared distribution with d degrees of freedom. Let Q d ( · ) be the quantile function of the chi-squared distribution with d degrees of freedom. With probability at least 1 -δ t (over the random draw of θ ∗ from N ( µ t , Σ t ) )

Using a union bound argument, if δ t = 6 δ ( t +1) 2 π 2 , then (40) holds with probability at least 1 -δ t for all t ≥ 0 . Therefore, if θ ∗ ∼ N ( µ 0 , Σ 0 ) , then with high probability the following credible sets contain θ ∗ for all t ≥ 0 simultaneously:

The upper limit of the credible interval for this credible set (and the one we use in Figure 4) is

We compute confidence bounds/credible intervals for a randomly generated linear function of the form f ( x ) = ϕ ( x ) ⊤ θ ∗ , with inputs x ∈ R and θ ∗ ∈ R 20 , the latter drawn from a standard Gaussian distribution and if necessary scaled down to ∥ θ ∗ ∥ 2 ≤ 10 =: B . For the feature map ϕ , we use Random Fourier Features. We generate random data { ( x k , y k ) } t k =1 , where y k = ϕ ( x k ) ⊤ θ ∗ + η k , η k ∼ N (0 , σ 2 ) and σ = 0 . 1 (so the Gaussian likelihood is well-specified).

4

2

0

2

4

7.5

5.0

2.5

0.0

2.5

5.0

CMM-UCB

4

2

0

2

4

7.5

5.0

2.5

0.0

2.5

5.0

Bayes

4

2

0

2

4

7.5

5.0

2.5

0.0

2.5

5.0

4

2

0

2

4

7.5

5.0

2.5

0.0

2.5

5.0

4

2

0

2

4

7.5

5.0

2.5

0.0

2.5

5.0

4

2

0

2

4

7.5

5.0

2.5

0.0

2.5

5.0

Figure 4 shows the CMM-UCB upper and lower confidence bounds (left) and the Bayesian credible intervals (right) with different choices of the prior. We use roughly equivalent priors for both methods. If the Bayesian credible interval uses the prior θ ∗ ∼ N ( µ 0 , Σ 0 ) , then CMM-UCB uses the induced distribution over the function values Φ t θ ∗ , i.e. f t ∼ N (Φ t µ 0 , Φ t Σ 0 Φ ⊤ t ) .

In the top and middle rows of Figure 4, where the prior is well-specified, we observe that the CMMUCB upper and lower confidence bounds are slightly looser than the Bayesian credible intervals. In the bottom row of Figure 4, when the prior is misspecified, the CMM-UCB interval gets looser whereas the Bayesian credible interval becomes wrong. In summary, the Bayesian credible interval appears to be slightly tighter when the prior is well-specified and at least somewhat informative, but CMM-UCB is robust to 'misspecified' mixture distributions.

E.2 Benefits of Adaptive Mixture Distributions

In this section, we investigate a method for refining µ t and T t based on previously observed actions and rewards. Recall that µ t and T t must be chosen such that: (a) µ t and T t can only depend on a 1 , . . . , a t and r 1 , . . . , r t -1 ; (b) the first t -1 elements of µ t must be equal to µ t -1 ; (c) the upper left t -1 × t -1 block of T t must be T t -1 ; (d) T t must be positive (semi-)definite.

As in Sec. 6.4, m and k are any fixed mean and kernel functions. Each new row and column of T t is set using an adaptive kernel function k t -1 . For β > 0 , define

✶ where k t ( a ) = [ k ( a, a 1 ) , . . . , k ( a, a t )] ⊤ and K t is the kernel matrix whose ( i, j ) th element is k ( a i , a j ) . For T t , we choose

The i th column and i th row of this matrix depend only on only a 1 , r 1 , . . . , a i . Our motivation for this kernel function is: (a) generalising the usual Bayesian Gaussian process (GP) posterior covariance, one can show that if the kernel function k is positive definite, then T t is positive semi-definite; (b) k t is the Bayesian GP posterior covariance function (with a Gaussian likelihood with variance β ). Each new element of µ t is set by evaluating an adaptive mean function m t -1 at the latest action a t . Define

For µ t , we choose

The i th element, m i -1 ( a i ) , depends on only a 1 , r 1 , . . . , a i , so this is a valid choice for µ t . Note that m t is the Bayesian GP posterior mean function (again with a Gaussian likelihood with variance β ).

We now compare the standard 'non-adaptive' mixture distributions (with µ t and T t as in (9)) and an adaptive sequence of Gaussian mixture distributions (with µ t as in (42) and T t as in (41)). With both sequences of mixture distributions, we use m ( a ) = 0 and k ( a, a ′ ) = ϕ ( a ) ⊤ ϕ ( a ′ ) . For the adaptive sequence of mixture distributions, we set β = 4 σ 2 .

4

2

0

2

4

6

4

2

0

2

4

Non-Adaptive

Adaptive

Figure 5 shows upper and lower confidence bounds for a randomly generated linear function f ( x ) = ϕ ( x ) ⊤ θ ∗ , where x ∈ R , θ ∗ ∈ R 20 , and ϕ is a random Fourier feature map. In this example, the adaptive sequence of mixture distributions leads to tighter upper and lower confidence bounds.

100

200

300

400

500

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Radius

Radius

Adaptive AMM-UCB

100

200

300

400

500

0.50

0.55

0.60

0.65

0.70

0.75

0.80

0.85

Test Accuracy

Test Accuracy / Expected Reward

T

T

Figure 6 shows the radius R AMM ,t (left) and test accuracy (right) for AMM-UCB in the SVM hyperparameter tuning problem with the Raisin data set (described in Sec. 8.2). We observe that AMM-UCB with the adaptive mixture distributions achieves slightly higher test accuracy.

In Sec. D.2.1, we saw that for a standard sequence of mixture distributions, R AMM ,T could be upper bounded by a data-independent quantity of order O ( √ d ln( T )) . In Figure 6, R AMM ,T does appear to grow roughly logarithmically with T when the mixture distributions are the standard choice. However, R AMM ,T appears to be bounded by a constant when the mixture distributions are adaptive. If, when using adaptive mixture distributions, we could prove a data-independent bound on R AMM ,T of order O ( √ d ) , then we would be able to improve our data-independent cumulative regret bounds to O ( d √ T ln( T )) (rather than O ( d √ T ln( T )) ). This would be within a √ ln( T ) factor of the lower bound Ω( d √ T ) .

F Efficient Radius Computation

If we compute the squared radius R 2 MM ,t using the expression in (5), then we have to compute the inverse and determinant of the t × t matrix ✶ + T t /σ 2 . We will now show that for any mixture distribution of the form P t = N ( µ t , Φ t Σ 0 Φ ⊤ t ) , where Σ 0 is symmetric and positive-definite, we can re-write the expression for R 2 MM ,t such that we instead need to compute the inverse and determinant of a d × d matrix. When P t = N ( µ t , Φ t Σ 0 Φ ⊤ t ) , the squared radius R 2 MM ,t is equal to

By using the Weinstein-Aronszajn identity, and then doing some algebra, we have

Using Lemma C.4 with γ = σ 2 , v = µ t -r t and M = Φ t Σ 1 / 2 0 , we have

The resulting expression for R 2 MM ,t is rather cumbersome, but the upshot is that we now (only) need to compute the inverse and determinant of the d × d matrix Φ ⊤ t Φ t + σ 2 Σ -1 0 . Since, we compute R 2 MM ,t at each round t , we can update the inverse of Φ ⊤ t Φ t + σ 2 Σ -1 0 incrementally using the Sherman-Morrison formula (Sherman & Morrison, 1950). The determinant of Φ ⊤ t Φ t + σ 2 Σ -1 0 can be updated incrementally using the relation

which can be found in Eq. (6) in Lemma 11 of (Abbasi-Yadkori et al., 2011). Additionally, recall that (see Eq. (24)) when P t = N ( 0 , c Φ t Φ ⊤ t ) and α = σ 2 /c for any c > 0 , the radius R AMM ,t of our analytic confidence bounds simplifies to

Note that the inverse (Φ ⊤ t Φ t +( σ 2 /c ) ✶ ) -1 still appears in the expression for AUCB Θ t ( a ) and in the incremental determinant update rule.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the stochastic linear bandits as introduced in Abbasi-Yadkori et al., 2011. Here, at each time step $t$, an action set $\\mathcal{A}_t$ is given. The learner then selects an action $a_t \\in \\mathcal{A}_t$, which maps to a feature vector $\\phi(a_t)$, and subsequently receives a reward $\\phi(a_t)^{\\mathsf{T}}\\pmb{\\theta}^*+\\epsilon_t$. The objective is to maximize the cumulative rewards over a designated time horizon $T$.\n\nThe main contributions of this paper can be summarized as:\n\n1. The authors propose a general approach based on the notion of Martingale Mixtures to create confidence sets. These are subsequently utilized in the meta LinUCB algorithm to derive the bandit algorithms. It is further demonstrated that such algorithms can be efficiently computed via convex optimization.\n\n2. The paper shows that such bandit algorithm derived from a suitable selection of mixture distributions $P_t$ attains the same worst-case regret as found in Abbasi-Yadkori et al., 2011.\n\n3. Evidence is presented to show that the algorithm derived in this paper outperforms the approach proposed in Abbasi-Yadkori et al., 2011 when applied to several real-world datasets.\n\nSTRENGTHS:\nWhile I'm not thoroughly acquainted with the most recent literature on stochastic linear bandits, this paper appears to present compelling results. The idea of using mixture martingales to derive linear bandit algorithms is innovative and provides a general methodology for generating new algorithms. The authors supplement their theoretical contributions with empirical evidence showing that their algorithm outperforms those presented in previous literature, further strengthening their findings.\n\nWEAKNESSES:\nThe downside of the paper lies in its inability to improve upon the theoretical worst-case bound as shown in Abbasi-Yadkori et al., 2011. The strength of this paper would be significantly enhanced if the authors could demonstrate that employing the methodology from their current work could theoretically offer improved bounds compared to previous results (or new results in novel settings).\n\nTypos:\n- Line 216,  $\\sum_{t=1}^T$ is missing\n- Line 696,  AUCB appeared twice\n\nQUESTIONS:\nI'm curious, how would the present work compare to the general approach as in \"The Statistical Complexity of Interactive Decision Making\" by D. J. Foster, S. M. Kakade, J. Qian and A. Rakhlin.?\n\nLIMITATIONS:\nNo issue with negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nA very well written paper introducing a slightly different way of constructing confidence sets for $\\theta^\\star$ in the adaptive regression setting. The two main differences are that the paper bounds the norm of the observation noises $\\epsilon_t$ directly, rather than the projection of the noises that features in the standard bound. This is done by showing that the method of mixtures can be used with a suitably adapted sequence of mixing measures, and choosing that mixture appropriately.\n\nSTRENGTHS:\nPaper is _very_ well written. Method introduced is neat and the perspective taken in deriving it will be of interest to other researchers in adaptive regression/design.\n\nWEAKNESSES:\nI don't believe the paper has any significant weaknesses. One could argue that it has, perhaps, limited scope: but bandits and adaptive regression/design are very popular topics nowadays, and this is an interesting read for many people interested in those.\n\nOne issue I feel strongly about, but that can be easily addressed: you advertise that your confidence intervals are robust to misspecification (abstract, line 12). I was very disappointed to see, when I got to section 8.1 and particularly lines 282-289, that you mean misspecification to a prior in a Bayesian setting. The general setting of your work is frequentist, and in the frequentist setting, misspecification has a well understood meaning which, of course, does not coincide with that which you show. It is unclear that your method is any better in this respect than the standard concentration inequality used in OFUL (Yasin's original work); and indeed, that's a generous interpretation: it is generally accepted that bounds derived under frequentist assumptions work well in a misspecified Bayesian setting. Please remove this claim from your abstract.\n\nAnother issue is that your plots are unreadable when printed in grayscale (all lines look the same). I'm reviewing a grayscale printed version of this paper, so cannot asses empirical performance from plots. Fortunately for you, I also happen to care little for empirical results.\n\nI have some minor feedback, solely for the purpose of improving the manuscript:\n\n-High level: please reiterate in the introduction, e.g. on line 44, that your UCBs are tighter in an empirical sense; that you have not (to my understanding) shown them to be tighter in a theoretical sense. Your results are neat: by risking the perception that you might be overclaiming/misleading you'd be doing yourself a disservice. \n\n-High level: your method is highly related to the rather excellent paper \n\n-Line 65: the result you cite Chowdhury & Gopalan 2017 for is implied directly by theorem 4.1 in Yasin Abbasi-Yadkori's PhD thesis (2012); that the result of Chowdhury & Gopalan was novel is an error in the literature that ought not be propagated.\n\n-Lines 97-98, you state that $\\mathcal{H}_t$-measurability equates to 'can be calculated using the data available just after reward $r_t$ is revealed'. From the rest of the paper, I know that you know that this isn't true. It may seem like a nice simplifying explanation, but its misleading, and often inexperienced authors make a mess of things because they take that to be the definition. Indeed, in your setting, the event $\\{ \\theta^\\star \\in \\Theta_t\\}$ is $\\mathcal{H}_t$ measurable (assuming $\\Theta_t$ is a closed set, see next comment), since $\\theta^\\star$ is a constant and $\\Theta_t$ is a measurable random set. But $\\theta^\\star$ is explicitly unknown at the end of the $t$th iteration, and so the indicator of $\\theta^\\star \\in \\Theta_t$  _cannot_ be computed with the information available at that point. Please remove that statement, and if you feel the reader might need a primer on measurability, please include a reference to any standard measure/probability textbook.\n\n-Lines 95 to 100: you define what are effectively random sets with a certain property. One has to be careful around the definitions of random sets to ensure they behave in a way that one would expect. For example, we'd usually like that for a random set $A'$ subset of $A$, for any $a \\in A$, the event $\\{a \\in A'\\}$ is measurable. Your sets are closed and you work on a Polish space, so this is true; but I would point out that this is so (and indeed, you might run into trouble if the confidence sets were open). See Molchanov, Ilya: Theory of Random Sets. Springer London. 2017, 2nd edition. Proposition 1.1.2; that should be all you need.\n\n-Eq (5), I would point out that this is just the 2-norm of $\\epsilon_t$; this makes it much clearer, for example, where your naive bound of line 158 comes from.\n\n-On your assumptions 7.1-7.4: It seems to me that 7.2+7.3 together imply a bound of the form asked for in 7.4; is there a good reason you have a separate assumption 7.4? \n\nPS: I have not read the appendix. I am confident from the sketches in the main text that the result claimed goes through.\n\nQUESTIONS:\nI included some minor questions in the weaknesses section. I have no major questions for the authors.\n\nLIMITATIONS:\nSome aspects of the writing could be thought to overclaim, specifically when it comes to robustness to misspecification. I feel strongly that this ought to be addressed. But this is also easily fixable; I hope the authors do so.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the problem of stochastic linear contextual bandits, and proposes an improvement of the classic LinUCB / OFUL algorithmic template via a more sophisticated construction of the confidence sets for the hidden reward vector. The improvement comes from replacing the confidence ellipsoid used since the classic work of Abbasi-Yadkori et al. (based on the method of mixtures) with a tighter confidence set based on what the authors call \"adaptive martingale mixtures\". The authors eventually derive a confidence ellipsoid resembling the ones used by Russo and Van Roy, which enjoys the useful property of having a potentially data-dependent radius, and can also be turned into a confidence sequence very easily via an application of Ville's inequality. Using two different methods (an exact convex solver and an approximation of the optimal width), the authors then turn these confidence sets for theta into confidence bounds for the rewards of each action, and use the resulting bounds in a UCB scheme. The algorithms are then shown to outperform standard LinUCB in some simple experiments.\n\nSTRENGTHS:\nThe paper is superbly written and presents a very interesting technique, improving one of the main building blocks of UCB algorithms that have been used for over a decade without any substantial changes. The proposed techniques make use of techniques that have recently gained popularity for mean estimation and proving PAC-Bayesian generalization bounds. The derivations up until the end of Section 5 are very neat and satisfying, and I believe that everyone interested in confidence ellipsoids and linear bandits should find them interesting.\n\nWEAKNESSES:\nThe concrete approaches proposed in Section 6 are also nice but leave something to be desired. Perhaps it is my fault, but I have missed how one should choose the predictions mu_t and T_t. From what I understand, both approaches in Sections 6.1 and 6.2 should give valid results irrespective of the choice of these parameters, but I still wonder how the choice will impact the quality of the guarantees. The authors only suggest that choosing mu and T that \"are good predictors of the (stochastic) reward r\" will yield good results, but do not elaborate further. For instance, is setting mu_t as the least squares estimator and T_t as the Grammian a good idea? It feels somewhat unsatisfying to introduce all the possibilities for adaptivity and then simply set a constant lambda_t and go with the standard choice of mu and T... Also, the confidence set proposed in Section 6.2 doesn't seem all that different from the standard tail bound popularized by Abbasi-Yadkori et al. Accordingly, the regret bounds of Theorem 7.5 and 7.6 also take the same for as previous bounds, which is not unexpected given how the theorems are stated for rather generic choices of mu and T. I can see that the new bounds *could* be tighter, but at the moment the theory does not reflect this, which is somewhat disappointing.\n\nOne additional thing that I would like to comment on is the computational complexity of the resulting method. I understand that the UCB's obtained from the method are always convex in the feature representation of the actions (given how they are a maximum of linear functions). Thus, calculating the action with maximal UCB is a convex maximization problem, which is NP-hard in general. This is generally true for all UCB-like methods I can think of, so perhaps it is a bit odd to avoid discussing this question altogether in the paper and suggest that the gradient-based optimization scheme for finding the optimistic actions is a theoretically well-justified idea. (It is not, but I understand that it often still works in practice.) I think it would be nice to add a comment on this in order to not mislead more casual readers who may not be familiar with this computational difficulty.\n\nDespite all my criticism above, I am happy to support acceptance of this paper to the NeurIPS program. I am looking forward to future literature addressing the current limitations of the approach proposed in this otherwise very nice paper.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the stochastic linear bandits problem and proposes an improved algorithm with sub-linear regret guarantees. The improvement is achieved using a novel tail bound for adaptive martingale mixtures to construct tighter upper confidence bounds, which leads to a smaller regret than existing algorithms for linear bandits. The authors also verify the performance of the proposed algorithm via experiments on hyperparameter tuning tasks.\n\nSTRENGTHS:\n#### **The following are the strengths of the paper:**\n1. The performance of any upper confidence bound (UCB) based bandit algorithm depends on the tightness of the confidence bounds. This paper proposes a novel way (using the tail bound for adaptive martingale mixtures) to improve the upper confidence bound, leading to an improved UCB-based bandit algorithm with a smaller regret.\n\n2. The authors propose two novel methods for computing the confidence bounds: Convex Martingale Mixture UCB (CMM-UCB) and Analytic Martingale Mixture UCB (AMM-UCB). CMM-UCB uses a convex solver for the UCB maximization (gradient differentiable convex optimization), whereas AMM-UCB uses a weak Lagrangian duality to obtain an analytic UCB (gradient can be computed in closed-form or via standard automatic differentiation procedures). \n\n3. When the mixture distribution is a Gaussian distribution, the authors showed the sub-linear data dependent/ independent cumulative regret for CMM_UCB and AMM-UCB. The author also empirically validated the performance gain of the proposed methods over existing linear bandit algorithms.\n\nWEAKNESSES:\n#### **The following are the weaknesses of the paper:**\n1. Assumption of mixture distribution having Gaussian distributions: The regret bounds stated in the paper hold only when the mixture distribution is Gaussian. It is unclear from the paper how practical this assumption is and what the consequences are (especially in analysis) if this assumption does not hold. \n\n2. Linearity assumption: Assuming a linear relationship between the reward and action's features (in high dimensional space using a known feature map) restricts the applications of proposed methods. Even though authors claim their tail bounds can be used to derive confidence sequences for non-linear reward functions, it is unclear what are the challenges of extending their work to non-linear reward functions (or kernelized bandits).\n\n\n3. Unexplained notations: Many notations are not properly defined in the paper. For examples: \\\ni. Line 145: What is $Z_t(f_t)$ connection with existing regret analysis (e.g., OFUL)? \\\nii. Line 149: How are the parameters ($\\boldsymbol{\\mu}_t, \\boldsymbol{T}_t$) of Gaussian distribution computed? \\\niii. In Theorem 7.5: what is the regret upper bound in terms of $T$? \\\niv. Line 234: How to set the value of $c$? \\\nv: Line 248: What is $\\sigma_0$ and how to set its value?\n\nQUESTIONS:\nPlease address the weakness raised in ***Weaknesses**. \n\nMinor comment:\n1. Line  303: FTS: Freq-TS\n\nI can change my score based on the authors' responses.\n\nLIMITATIONS:\nI have raised a few limitations of the paper in my response to the ***Weaknesses**. Since the paper is a theoretical contribution to linear bandits literature, I do not find any potential negative societal impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-27"}