Title: On the Sublinear Regret of GP-UCB
Venue: NeurIPS 2023
Accepted: yes
Keywords: Kernel Bandits

Online Learning

Self-Normalized Concentration

Online Regression
Abstract: In the kernelized bandit problem, a learner aims to sequentially compute the optimum of a function lying in a reproducing kernel Hilbert space given only noisy evaluations at sequentially chosen points. In particular, the learner aims to minimize regret, which is a measure of the suboptimality of the choices made.
Arguably the most popular algorithm is the Gaussian Process Upper Confidence Bound (GP-UCB) algorithm, which involves acting based on a simple linear estimator of the unknown function.
Despite its popularity, existing analyses of GP-UCB give a suboptimal regret rate, which fails to be sublinear for many commonly used kernels such as the Matern kernel. This has led to a longstanding open question: are existing regret analyses for GP-UCB tight, or can bounds be improved by using more sophisticated analytical techniques?
In this work, we resolve this open question and show that GP-UCB enjoys nearly optimal regret. In particular, our results yield sublinear regret rates for the Matern kernel, improving over the state-of-the-art analyses and partially resolving a COLT open problem posed by Vakili et al. Our improvements rely on a key technical contribution --- regularizing kernel ridge estimators in proportion to the smoothness of the underlying kernel $k$. Applying this key idea together with a largely overlooked concentration result in separable Hilbert spaces (for which we provide an independent, simplified derivation), we are able to provide a tighter analysis of the GP-UCB algorithm.

On the Sublinear Regret of GP-UCB

Abstract

In the kernelized bandit problem, a learner aims to sequentially compute the optimum of a function lying in a reproducing kernel Hilbert space given only noisy evaluations at sequentially chosen points. In particular, the learner aims to minimize regret, which is a measure of the suboptimality of the choices made. Arguably the most popular algorithm is the Gaussian Process Upper Confidence Bound (GP-UCB) algorithm, which involves acting based on a simple linear estimator of the unknown function. Despite its popularity, existing analyses of GPUCBgive a suboptimal regret rate, which fails to be sublinear for many commonly used kernels such as the Mat´ ern kernel. This has led to a longstanding open question: are existing regret analyses for GP-UCB tight, or can bounds be improved by using more sophisticated analytical techniques? In this work, we resolve this open question and show that GP-UCB enjoys nearly optimal regret. In particular, our results yield sublinear regret rates for the Mat´ ern kernel, improving over the state-of-the-art analyses and partially resolving a COLT open problem posed by Vakili et al. Our improvements rely on a key technical contribution - regularizing kernel ridge estimators in proportion to the smoothness of the underlying kernel k . Applying this key idea together with a largely overlooked concentration result in separable Hilbert spaces (for which we provide an independent, simplified derivation), we are able to provide a tighter analysis of the GP-UCB algorithm.

1 Introduction

An essential problem in areas such as econometrics [12, 13], medicine [22, 23], optimal control [4, 3], and advertising [21] is to optimize an unknown function given bandit feedback , in which algorithms only get to observe the outcomes for the chosen actions. Due to the bandit feedback, there is a fundamental tradeoff between exploiting what has been observed about the local behavior of the function and exploring to learn more about the function's global behavior. There has been a long line of work on bandit learning that investigates this tradeoff across different settings, including multi-armed bandits [29, 19, 37], linear bandits [2, 30], and kernelized bandits [5, 26, 32].

In this work, we focus on the kernelized bandit framework, which can be viewed as an extension of the well-studied linear bandit setting to an infinite-dimensional reproducing kernel Hilbert space (or RKHS) ( H, 〈 · , · 〉 H ) . In this problem, there is some unknown function f ∗ : X → R of bounded norm in H , where X ⊂ R d is a bounded set. In each round t ∈ [ T ] , the learner uses previous observations to select an action X t ∈ X , and then observes feedback Y t := f ∗ ( X t ) + /epsilon1 t , where /epsilon1 t is a zero-mean noise variable. The learner aims to minimize (with high probability) the regret at time

T , which is defined as where x ∗ := arg max x ∈ X f ∗ ( x ) . The goal is to develop simple, efficient algorithms for the kernelized bandit problem that minimize regret R T . We make the following standard assumption. We also make assumptions on the underlying kernel k , which we discuss in Section 2.

Assumption 1. We assume that (a) there is some constant D > 0 known to the learner such that ‖ f ∗ ‖ H ≤ D and (b) for every t ≥ 1 , /epsilon1 t is σ -subGaussian conditioned on σ ( Y 1: t -1 , X 1: t ) .

Arguably the simplest algorithm for the kernelized bandit problem is GP-UCB (Gaussian process upper confidence bound) [31, 5]. GP-UCB works by maintaining a kernel ridge regression estimator of the unknown function f ∗ alongside a confidence ellipsoid, optimistically selecting in each round the action that provides the maximal payoff over all feasible functions. Not only is GP-UCB efficiently computable thanks to the kernel trick, but it also offers strong empirical guarantees [5]. The only seeming deficit of GP-UCB is its regret guarantee, as existing analyses only show that, with high probability, R T = ˜ O ( γ T √ T ) , where γ T is a kernel-dependent measure of complexity known as the maximum information gain [31, 6]. In contrast, more complicated, less computationally efficient algorithms such as SupKernelUCB [34, 25] have been shown to obtain regret bounds of ˜ O ( √ γ T T ) , improving over the analysis of GP-UCB by a multiplicative factor of √ γ T . This gap is stark as the bound ˜ O ( γ T √ T ) fails, in general, to be sub-linear for the practically relevant Mat´ ern kernel, whereas O ( √ γ T T ) is sublinear for any kernel experiencing polynomial eigendecay [32].

˜ This discrepancy has prompted the development of many variants of GP-UCB that, while less computationally efficient, offer better regret guarantees in some situations [17, 27, 28]. (See a detailed discussion of these algorithms along with other related work in Appendix A.) However, the following question remains an open problem in online learning [33]: are existing analyses of vanilla GP-UCB tight, or can an improved analysis show GP-UCB enjoys sublinear regret?

1.1 Contributions

In this work, we show that GP-UCB obtains almost optimal, sublinear regret for any kernel experiencing polynomial eigendecay. This, in particular, implies that GP-UCB obtains sublinear regret for the commonly used Mat´ ern family of kernels. We provide a brief roadmap of our paper below.

In Section 3, we provide background into self-normalized concentration in Hilbert spaces. In particular, in Theorem 1, we provide an independent, simplified derivation of a bound due to Abbasi-Yadkori [1], which concerns to self-normalized concentration of certain process in separable Hilbert spaces. This bound has been largely overlooked in the kernel bandit literature, so we draw attention to it in hopes it can be leveraged in solving further kernel-based learning problems. As opposed to the existing bound of Chowdhury and Gopalan [5], which involves employing a complicated 'double mixture' argument, the bound we present follows directly from applying the well-studied finite-dimensional method of mixtures alongside a simple truncation argument [7-9, 2]. These bounds are clean and show simple dependence on the regularization parameter.

In Section 4, we use leverage the self-normalized concentration detailed in Theorem 1 to provide an improved regret analysis for GP-UCB. By carefully choosing regularization parameters based on the smoothness of the underlying kernel, we demonstrate that GPUCBenjoys sublinear regret of ˜ O ( T 3+ β 2+2 β ) for any kernel experiencing ( C, β ) -polynomial eigendecay. As a special case of this result, we obtain regret bounds of ˜ O ( T ν +2 d 2 ν +2 d ) for the commonly used Mat´ ern kernel with smoothness ν in dimension d . Our new analysis improves over existing state-of-the-art analysis for GP-UCB, which fails to guarantee sublinear regret in general for the Mat´ ern kernel family [5], and thus partially resolves an open problem posed by [33] on the suboptimality of GP-UCB.

In sum, our results show that GP-UCB, the go-to algorithm for the kernelized bandit problem, is nearly optimal, coming close to the algorithm-independent lower bounds of Scarlett et al. [25]. Our

work thus can be seen as providing theoretical justification for the strong empirical performance of GP-UCB [31]. Perhaps the most important message of our work is the importance of careful regularization in online learning problems. While many existing bandit works treat the regularization parameter as a small, kernel-independent constant, we are able to obtain significant improvements by carefully selecting the regularization parameter. We hope our work will encourage others to pay close attention to the selection of regularization parameters in future works.

2 Background and Problem Statement

Notation. We briefly touch on basic definitions and notational conveniences that will be used throughout our work. If a 1 , . . . , a t ∈ R , we let a 1: t := ( a 1 , . . . , a t ) /latticetop . Let ( H, 〈 · , · 〉 H ) be a reproducing kernel Hilbert space associated with a kernel k : X × X → R . We refer to the identity operator on H as id H . This is distinct from the identity mapping on R d , which we will refer to as I d . For elements f, g ∈ H , we define their outer product as fg /latticetop := f 〈 g, · 〉 H and inner product as f /latticetop g := 〈 f, g 〉 H . For any t ≥ 1 and sequence of points x 1 , . . . , x t ∈ X (which will typically be understood from context), let Φ t := ( k ( · , x 1 ) , . . . , k ( · , x t )) /latticetop . We can respectively define the Gram matrix K t : R t → R t and covariance operator V t : H → H as K t := ( k ( x i , x j )) i,j ∈ [ t ] = Φ t Φ /latticetop t and V t := ∑ t s =1 k ( · , x s ) k ( · , x s ) /latticetop = Φ /latticetop t Φ t . These two operators essentially encode the same information about the observed data points, the former being easier to work with when actually performing computations (by use of the well known kernel trick) and latter being easier to algebraically manipulate.

Suppose A : H → H is a Hermitian operator of finite rank; enumerate its non-zero eigenvalues as λ 1 ( A ) , . . . , λ k ( A ) . We can define the Fredholm determinant of I + A as det( I + A ) := ∏ k m =1 (1 + λ i ( A )) [20]. For any t ≥ 1 , ρ > 0 , and x 1 , . . . , x t ∈ X , one can check via a straightforward computation that det( I t + ρ -1 K t ) = det(id H + ρ -1 V t ) , where K t and V t are the Gram matrix and covariance operator defined above. We, again, will use these two quantities interchangeably in the sequel, but will typically prefer the latter in our proofs.

If ( H, 〈 · , · 〉 H ) is a (now general) separable Hilbert space and ( ϕ n ) n ≥ 1 is an orthonormal basis for H , for any N ≥ 1 we can define the orthogonal projection operator π N : H → span { ϕ 1 , . . . , ϕ N } ⊂ H by π N f := ∑ N n =1 〈 f, ϕ n 〉 H ϕ n . We can correspondingly the define the projection onto the remaining basis functions to be the map π ⊥ N : H → span { ϕ 1 , . . . , ϕ N } ⊥ given by π ⊥ N f := f -π N f . Lastly, if A : H → H is a symmetric, bounded linear operator, we let λ max ( A ) denote the maximal eigenvalue of A , when such a value exists. In particular, λ max ( A ) will exist whenever A has a finite rank, as will typically be the case considered in this paper.

Basics on RKHSs. Let X ⊂ R d be some domain. A kernel is a positive semidefinite map k : X × X → R that is square-integrable, i.e. ∫ X ∫ X | k ( x, y ) | 2 dxdy < ∞ . Any kernel k has an associated reproducing kernel Hilbert space or RKHS ( H, 〈 · , · 〉 H ) containing the closed span of all partial kernel evaluations k ( · , x ) , x ∈ X . In particular, the inner product 〈 · , · 〉 H on H satisfies the reproducing relationship f ( x ) = 〈 f, k ( · , x ) 〉 H for all x ∈ X .

Akernel k can be associated with a corresponding Hilbert-Schmidt operator , which is the Hermitian operator T k : L 2 ( X ) → L 2 ( X ) given by ( T k f )( x ) := ∫ X f ( y ) k ( x, y ) dy for any x ∈ X . In short, T k can be thought of as 'smoothing out' or 'mollifying' a function f according to the similarity metric induced by k . T k plays a key role in kernelized learning through Mercer's Theorem , which gives an explicit representation for H in terms of the eigenvalues and eigenfunctions of T k .

Fact 1 ( Mercer's Theorem ) . Let ( H, 〈 · , · 〉 H ) be the RKHS associated with kernel k , and let ( µ n ) n ≥ 1 and ( φ n ) n ≥ 1 be the sequence of non-increasing eigenvalues and corresponding eigenfunctions for T k . Let ( ϕ n ) n ≥ 1 be the sequence of rescaled functions ϕ n := √ µ n φ n . Then,

and ( ϕ n ) n ≥ 1 forms an orthonormal basis for ( H, 〈 · , · 〉 H ) .

We make the following assumption throughout the remainder of our work, which is standard and comes from Vakili et al. [32].

Assumption 2 ( Assumption on kernel k ) . The kernel k : X × X → R satisfies (a) | k ( x, y ) | ≤ L for all x, y ∈ X , for some constant L > 0 and (b) | φ n ( x ) | ≤ B for all x ∈ X , for some B > 0 .

'Complexity' of RKHS's. By the eigendecay of a kernel k , we really mean the rate of decay of the sequence of eigenvalues ( µ n ) n ≥ 1 . In the literature, there are two common paradigms for studying the eigendecay of k : ( C 1 , C 2 , β ) -exponential eigendecay, under which ∀ n ≥ 1 , µ n ≤ C 1 exp( -C 2 n β ) , and ( C, β ) -polynomial eigendecay, under which ∀ n ≥ 1 , µ n ≤ Cn -β . For kernels experiencing exponential eigendecay, of which the squared exponential is the most important example, GP-UCB is known to be optimal up to poly-logarithmic factors. However, for kernels experiencing polynomial eigendecay, of which the Mat´ ern family is a common example, existing analyses of GP-UCB fail to yield sublinear regret. It is this latter case we focus on in this work.

Given the above representation in Fact 1, it is clear that the eigendecay of the kernel k governs the 'complexity' or 'size' of the RKHS H . We make this notion of complexity precise by discussing maximum information gain , a sequential, kernel-dependent quantity governing concentration and hardness of learning in RKHS's [6, 31, 32].

Let t ≥ 1 and ρ > 0 be arbitrary. The maximum information gain at time t with regularization ρ is the scalar γ t ( ρ ) given by

Our presentation of maximum information gain differs from some previous works in that we encode the regularization parameter ρ into our notation. This inclusion is key for our results, as we obtain improvements by carefully selecting ρ . Vakili et al. [32] bound the rate of growth of γ t ( ρ ) in terms of the rate of eigendecay of the kernel k . We leverage the following fact in our main results.

Fact 2 ( Corollary 1 in Vakili et al. [32] ) . Suppose that kernel k satisfies Assumption 2 and experiences ( C, β ) -polynomial eigendecay. Then, for any t ≥ 1 , we have

We last define the practically relevant Mat´ ern kernel and discuss its eigendecay.

Definition/Fact 3. The Mat´ ern kernel with bandwidth σ > 0 and smoothness ν > 1 / 2 is given by

where Γ is the gamma function and B ν is the modified Bessel function of the second kind. It is known that there is some constant C > 0 that may depend on σ but not on d or ν such that k ν , σ experiences ( C, 2 ν + d d ) -eigendecay [24, 32].

Basics on martingale concentration: A filtration ( F t ) t ≥ 0 is a sequence of σ -algebras satisfying F t ⊂ F t +1 for all t ≥ 1 . If ( M t ) t ≥ 0 is a H -valued process, we say ( M t ) t ≥ 0 is a martingale with respect to ( F t ) t ≥ 0 if (a) ( M t ) t ≥ 0 is ( F t ) t ≥ 0 -adapted, and (b) E ( M t | F t -1 ) = M t -1 for all t ≥ 1 . An R -valued process is called a supermartingale if the equality in (b) is replaced with ' ≤ ', i.e. supermartingales tend to decrease. Martingales are useful in many statistical applications due to their strong concentration of measure properties [15, 36]. The follow fact can be leveraged to provide time-uniform bounds on the growth of any non-negative supermartingale.

Fact 4 ( Ville's Inequality ) . Let ( M t ) t ≥ 0 be a non-negative supermartingale with respect to some filtration. Suppose E M 0 = 1 . Then, for any δ ∈ (0 , 1) , we have

See Howard et al. [14] for a self-contained proof of Ville's inequality, and many applications.

If F is a σ -algebra, and /epsilon1 is an R -valued random variable, we say /epsilon1 is σ -subGaussian conditioned on F if, for any λ ∈ R , we have log E ( e λ/epsilon1 | F ) ≤ λ 2 σ 2 2 ; in particular this condition implies

that /epsilon1 is mean zero. With this, we state the following result on self-normalized processes. To our understanding, the following result was first presented in some form as Example 4.2 of de la Pe˜ na et al. [8] (in the setting of continuous local martingales), and can be derived leveraging the argument of Theorem 1 in de la Pe˜ na et al. [9]. The exact form below was established (in the setting of discrete-time processes) in Theorem 1 of Abbasi-Yadkori et al. [2], which is commonly leveraged to construct confidence ellipsoids in the linear bandit setting.

Fact 5 ( Example 4.2 from [8], Theorem 1 from [2] ) . Let ( F t ) t ≥ 0 be a filtration, let ( X t ) t ≥ 1 be an ( F t ) t ≥ 0 -predictable sequence in R d , and let ( /epsilon1 t ) t ≥ 1 be a real-valued ( F t ) t ≥ 1 -adapted sequence such that conditional on F t -1 , /epsilon1 t is mean zero and σ -subGaussian. Then, for any ρ > 0 , the process ( M t ) t ≥ 0 given by is a non-negative supermartingale with respect to ( F t ) t ≥ 0 , where S t := ∑ t s =1 /epsilon1 s X s and V t := ∑ t s =1 X s X /latticetop s . Consequently, by Fact 4, for any confidence δ ∈ (0 , 1) , the following holds: with probability at least 1 -δ , simultaneously for all t ≥ 1 , we have

Note the simple dependence on the regularization parameter ρ > 0 in the above bound. While the regularization parameter ρ doesn't prove important in regret analysis for linear bandits (where ρ is treated as constant), the choice for ρ will be critical in our setting. In the following section, we will discuss how Fact 5 can be extended to the setting of separable Hilbert spaces essentially verbatim (an observation first noticed by Abbasi-Yadkori [1]).

3 A Remark on Self-Normalized Concentration in Hillbert Spaces

We begin by discussing a key, self-normalized concentration inequality for martingales. We use this bound in the sequel to construct simpler, more flexible confidence ellipsoids than currently exist for GP-UCB. The bound we present (in Theorem 1 below) is, more or less, equivalent to Corollary 3.5 in the thesis of Abbasi-Yadkori [1]. Our result is mildly more general in the sense that it directly argues that a target mixture process is a nonnegative supermartingale. The result in Abbasi-Yadkori [1] is more general in the sense it allows the regularization (or shift) matrix to be non-diagonal. Either concentration result is sufficient for the regret bounds obtained in the sequel.

The aforementioned corollary in [1], quite surprisingly, has not been referenced in central works on the kernelized bandit problem, namely Chowdhury and Gopalan [5] and Vakili et al. [32, 33]. In fact, strictly weaker versions of the conclusion have been independently rediscovered in the context of kernel regression [10]. We emphasize that this result of Abbasi-Yadkori [1] (and the surrounding technical conclusions) are very general and may allow for further improvements in problems related to kernelized learning.

We now present Theorem 1, providing a brief sketch and a full proof in Appendix B. We believe our proof, which directly shows a target process is a nonnegative supermartingale, is of independent interest when compared to that of Abbasi-Yadkori [1] due to its simplicity. In particular, our proof follows from first principles, avoiding advanced topological notions of convergence (e.g. in the weak operator topology) and existence of certain Gaussian measures on separable Hilbert spaces, which were heavily utilized in the proof of Corollary 3.5 in Abbasi-Yadkori [1].

Theorem 1 ( Self-normalized concentration in Hilbert spaces ) . Let ( F t ) t ≥ 0 be a filtration, ( f t ) t ≥ 1 be an ( F t ) t ≥ 0 -predictable sequence in a separable Hilbert space 1 H such that ‖ f t ‖ H < ∞ a.s. for all t ≥ 0 , and ( /epsilon1 t ) t ≥ 1 be an ( F t ) t ≥ 1 -adapted sequence in R such that conditioned on F t -1 , /epsilon1 t is mean zero and σ -subGaussian. Defining S t := ∑ t s =1 /epsilon1 s f s and V t := ∑ t s =1 f s f /latticetop s , we have that for any ρ > 0 , the process ( M t ) t ≥ 0 defined by

is a nonnegative supermartingale with respect to ( F t ) t ≥ 0 . Consequently, by Fact 4, for any δ ∈ (0 , 1) , with probability at least 1 -δ , simultaneously for all t ≥ 1 , we have

We can summarize our independent proof in two simple steps. First, following from Fact 5, the bound in Theorem 1 holds when we project S t and V t onto a finite number N of coordinates, defining a 'truncated' nonnegative supermartingale M ( N ) t . Secondly, we can make a limiting arugment, showing M ( N ) t is 'essentially' M t for large values of N .

Proof Sketch for Theorem 1. Let ( ϕ n ) n ≥ 1 be an orthonormal basis for H , and, for any N ≥ 1 , let π N denote the projection operator onto H N := span { ϕ 1 , . . . , ϕ N } . Note that the projected process ( π N S t ) t ≥ 1 is an H -valued martingale with respect to ( F t ) t ≥ 0 . Further, note that the projected variance process ( π N V t π /latticetop N ) t ≥ 0 satisfies

Since, for any N ≥ 1 , H N is a finite-dimensional Hilbert space, it follows from Lemma 1 that the process ( M ( N ) t ) t ≥ 0 given by is a nonnegative supermartingale with respect to ( F t ) t ≥ 0 . One can check that, for any t ≥ 0 , M ( N ) t - - - - → N →∞ M t . Thus, Fatou's Lemma implies

which proves the first part of the claim. The second part of the claim follows from applying Fact 4 to the defined nonnegative supermartingale and rearranging. See Appendix B for details. /squaresolid

The following corollary specializes Theorem 1 (and thus Corollary 3.5 of Abbasi-Yadkori [1]) to the case where H is a RKHS and f t = k ( · , X t ) , for all t ≥ 1 . In this special case, we can reframe the above theorem in terms familiar Gram matrix K t , assuming the quantity is invertible. While we prefer the simplicity and elegance of working directly in the RKHS H in the sequel, the follow corollary allows us to present Theorem 1 in a way that is computationally tractable.

Corollary 1. Let us assume the same setup as Theorem 1, and additionally assume that (a) ( H, 〈 · , · 〉 H ) is a RKHS associated with some kernel k , and (b) there is some X -valued ( F t ) t ≥ 0 -predictable process ( X t ) t ≥ 1 such that ( f t ) t ≥ 1 = ( k ( · , X t )) t ≥ 1 . Then, for any ρ > 0 and δ ∈ (0 , 1) , we have that, with probability at least 1 -δ , simultaneously for all t ≥ 0 ,

If, in addition, the Gram matrix K t = ( k ( X i , X j )) i,j ∈ [ t ] is invertible, we have the equality

We prove Corollary 1 in Appendix B. With this reframing of Theorem 1, we compare the concentration results of Theorem 1 (and thus Abbasi-Yadkori [1]) to the following, commonly leveraged result from Chowdhury and Gopalan [5].

Fact 6 ( Theorem 1 from Chowdhury and Gopalan [5] ) . Assume the same setup as Fact 5. Let η > 0 be arbitrary, and let K t := ( k ( X i , X j )) i,j ∈ [ t ] be the Gram matrix corresponding to observations made by time t ≥ 1 . Then, with probability at least 1 -δ , simultaneously for all t ≥ 1 , we have

To make comparison with this bound clear, we parameterize the bounds in the above fact in terms of η > 0 instead of ρ > 0 to emphasize the following difference: both sides of the bound presented in Theorem 1 shrink as ρ is increased, whereas both sides of the bound in Fact 6 increase as η grows. Thus, increasing ρ in Theorem 1 should be seen as decreasing η in the bound of Chowdhury and Gopalan [5]. The bounds in Corollary 1 and Fact 6 coincide when ρ = 1 and η ↓ 0 (per Lemma 1 in Chowdhury and Gopalan [5]), but are otherwise not equivalent for other choices of ρ and η .

We believe Theorem 1 and Corollary 3.5 of Abbasi-Yadkori [1] to be signficantly more usable than the result of Chowdhury and Gopalan [5] for several reasons. First, the aforementioned bounds directly extend the method of mixtures (in particular, Fact 5) to potentially infinite-dimensional Hilbert spaces. This similarity in form allows us to leverage existing analysis of Abbasi-Yadkori et al. [2] to prove our regret bounds, with only slight modifications. This is in contrast to the more cumbersome regret analysis that leverages Fact 6, which is not only more difficult to follow, but also obtains inferior, sometimes super-linear regret guarantees.

Second, we note that Theorem 1 provides a bound that has a simple dependence on ρ > 0 . In more detail, directly as a byproduct of the simplified bounds, Theorem 2 offers a regret bound that can readily be tuned in terms of ρ . Due to their use of a 'double mixture' technique in proving Fact 6, Chowdhury and Gopalan [5] essentially wind up with a nested, doubly-regularized matrix (( K t + η I t ) -1 + I t ) -1 / 2 with which they normalize the residuals /epsilon1 1: t . In particular, this more complicated normalization make it difficult to understand how varying η impacts regret guarantees, which we find to be essential for proving improved regret guarantees.

We note that the central bound discussed in this section does not provide an improvement in dependence on maximum information gain in the sense hypothesized by Vakili et al. [33]. In particular, the authors hypothesized the possibility of shaving a √ γ T multiplicative factor off of self-normalized concentration inequalities in RKHS's. This was shown in a recent work (see Lattimore [18]) to be impossible in general. Instead, Theorem 1 and Corollary 3.5 of Abbasi-Yadkori [1] give one access to a family of bounds parameterized by the regularization parameter ρ > 0 . As will be seen in the sequel, by optimizing over this parameter, one can obtain significant improvements in regret.

4 An Improved Regret Analysis of GP-UCB

In this section, we provide the second of our main contributions, which is an improved regret analysis for the GP-UCB algorithm. We provide a description of GP-UCB in Algorithm 1. While we state the algorithm directly in terms of quantities in the RKHS H , these quantities can be readily converted to those involving Gram matrices or Gaussian processes for those who prefer that perspective [5, 38].

As seen in Section 3, by carefully extending the 'method of mixtures' technique (originally by Robbins) of Abbasi-Yadkori et al. [2], Abbasi-Yadkori [1] and de la Pe˜ na et al. [7, 8] to Hilbert spaces, we can construct self-normalized concentration inequalities that have simple dependence on the regularization parameter ρ . These simplified bounds, in conjunction with information about the eigendecay of the kernel k [32], can be combined to carefully choose ρ to obtain improved regret. We now present our main result.

Theorem 2. Let T > 0 be a fixed time horizon, ρ > 0 a regularization parameter, and assume Assumptions 2 and 1 hold. Let δ ∈ (0 , 1) , and for t ≥ 1 define

Algorithm 1 Gaussian Process Upper Confidence Bound (GP-UCB)

Input: Regularization parameter ρ > 0 , norm bound D , confidence bounds ( U t ) t ≥ 1 , and time horizon T .

Then, with probability at least 1 -δ , the regret of Algorithm 1 run with parameters ρ , ( U t ) t ≥ 1 , D satisfies where in the big-Oh notation above we treat δ , D, σ , B , and L as being held constant. If the kernel k experiences ( C, β ) -polynomial eigendecay for some C > 0 and β > 1 , taking ρ = O ( T 1 1+ β ) yields R T = ˜ O ( T 3+ β 2+2 β ) 2 , which is always sub-linear in T .

While we present the above bound with a fixed time-horizon, it can be made anytime by carefully applying a standard doubling argument (see Lattimore and Szepesv´ ari [19], for instance). We specialize the above theorem to the case of the Mat´ ern kernel in the following corollary.

Corollary 2. Definition 3 states that the Mat´ ern kernel with smoothness ν > 1 / 2 in dimension d experiences ( C, 2 ν + d d ) -eigendecay, for some constnat C > 0 . Thus, GP-UCB obtains a regret rate of R T = ˜ O ( T ν +2 d 2 ν +2 d ) .

We note that our regret analysis is the first to show that GP-UCB attains sublinear regret for general kernels experiencing polynomial eigendecay. Of particular import is that Corollary 2 of Theorem 2 yields the first analysis of GP-UCB that implies sublinear regret for the Mat´ ern kernel under general settings of ambient dimension d and smoothness ν . A recent result by Janz [16], using a uniform lengthscale argument, demonstrates that GP-UCB obtains sublinear regret for the specific case of the Mat´ ern family when the parameter ν and dimension d satisfy a uniform boundedness condition independent of scale. Our results are (a) more general, holding for any kernel exhibiting polynomial eigendecay, (b) don't require checking uniform boundedness independent of scale condition, and (c) follow from a simple regularization based argument. In particular, the arguments of Janz [16] require advanced functional analytic and Fourier analytic machinery.

Wenote that our analysis does not obtain optimal regret, as the theoretically interesting but computationally cumbersome SupKernelUCB algorithm [25, 34] obtains a slightly improved regret bound of ˜ O ( T β +1 2 β ) for ( C, β ) -polynomial eigendecay and ˜ O ( T ν + d 2 ν + d ) for the Mat´ ern kernel with smoothness ν in dimension d . Due to the aforementioned result of Lattimore [18], which shows that improved dependence on maximum information gain cannot be generally obtained in Hilbert space concentration, we believe further improvements on regret analysis for GP-UCB may not possible.

To wrap up this section, we provide a proof sketch for Theorem 2. The entire proof, along with full statements and proofs of the technical lemmas, can be found in Appendix C.

Proof Sketch for Theorem 2 . Letting, for any t ∈ [ T ] , the 'instantaneous regret' be defined as r t := f ∗ ( x ∗ ) -f ∗ ( X t ) , a standard argument yields that, with probability at least 1 -δ , simultaneously for all t ∈ [ T ] ,

A further standard argument using Cauchy-Schwarz and an elliptical potential argument yields

which proves the first part of the claim. If, additionally, k experiences ( C, β ) -polynomial eigendecay, we know that γ T ( ρ ) = ˜ O ( ( T ρ ) 1 / β ) by Fact 2. Setting ρ := O ( T 1 1+ β ) thus yields proving the second part of the claim.

5 Conclusion

In this work, we present an improved analysis for the GP-UCB algorithm in the kernelized bandit problem. We provide the first analysis showing that GP-UCB obtains sublinear regret when the underlying kernel k experiences polynomial eigendecay, which in particular implies sublinear regret rates for the practically relevant Mat´ ern kernel. In particular, we show GP-UCB obtains regret ˜ O ( T 3+ β 2+2 β ) when k experiences ( C, β ) -polynomial eigendecay, and regret ˜ O ( T ν +2 d 2 ν +2 d ) for the Mat´ ern kernel with smoothness ν in dimension d .

Our contributions are twofold. First, we show the importance of finding the 'right' concentration inequality for tackling problems in online learning - in this case the correct bound being a selfnormalized inequality originally due to Abbasi-Yadkori [1]. We provide an independent proof of a result equivalent to Corollary 3.5 of Abbasi-Yadkori [1] in Theorem 1, and hope that our simplified, truncation-based analysis will make the result more accessible to researchers working on problems in kernelized learning. Second, we demonstrate the importance of regularization in the kernelized bandit problem. In particular, since the smoothness of the kernel k governs the hardness of learning, by regularizing in proportion to the rate of eigendecay of k , one can obtain significantly improved regret bounds.

A shortcoming of our work is that, despite obtaining the first generally sublinear regret bounds for GP-UCB, our rates are not optimal. In particular, there are discretization-based algorithms, such as SupKernelUCB [34], which obtain slightly better regret bounds of ˜ O ( T 1+ β 2 β ) for ( C, β ) -polynomial eigendecay. We hypothesize that the vanilla GP-UCB algorithm, which involves constructing confidence ellipsoids directly in the RKHS H , cannot obtain this rate.

The common line of reasoning [33] is that because the Lin-UCB (the equivalent algorithm in R d ) obtains the optimal regret rate of ˜ O ( d √ T ) in the linear bandit problem setting, then GP-UCB should attain optimal regret as well. In the linear bandit setting, there is no subtlety between estimating the optimal action and unknown slope vector, as these are one and the same. In the kernel bandit setting, estimating the function and optimal action are not equivalent tasks. In particular, the former serves in essence as a nuisance parameter in estimating the latter: tight estimation of unknown function under the Hilbert space norm implies tight estimation of the optimal action, but not the other way around. Existing optimal algorithms are successful because they discretize the input domain, which has finite metric dimension [26], and make no attempts to estimate the unknown function in RKHS norm. Since compact sets in RKHS's do not, in general, have finite metric dimension [35], this makes estimation of the unknown function a strictly more difficult task. In fact, recent work by Lattimore [18] demonstrate that self-normalized concentration in RKHS's, in general, cannot exhibit improved dependence on maximum information gain. This further supports our hypothesis on the further unimprovability of the regret analysis of GP-UCB past the improvements made in this paper.

/squaresolid

References

Yasin Abbasi-Yadkori. Online learning for linearly parametrized control problems. 2013.

Yasin Abbasi-Yadkori, D´ avid P´ al, and Csaba Szepesv´ ari. Improved algorithms for linear stochastic bandits. Advances in Neural Information Processing Systems , 24, 2011.

Rajeev Agrawal. The continuum-armed bandit problem. SIAM Journal on Control and Optimization , 33(6):1926-1951, 1995.

Andrew G Barto. Reinforcement learning control. Current Opinion in Neurobiology , 4(6): 888-893, 1994.

Sayak Ray Chowdhury and Aditya Gopalan. On kernelized multi-armed bandits. In International Conference on Machine Learning , pages 844-853. PMLR, 2017.

Thomas M Cover and Joy A Thomas. Information theory and statistics. Elements of Information Theory , 1(1):279-335, 1991.

Victor de la Pe˜ na, Michael J Klass, and Tze Leung Lai. Self-normalized processes: Exponential inequalities, moment bounds and iterated logarithm laws. The Annals of Probability , 32, 07 2004. doi: 10.1214/009117904000000397.

Victor de la Pe˜ na, Michael J Klass, and Tze Leung Lai. Pseudo-maximization and selfnormalized processes. Probability Surveys Vol , 4:172-192, 09 2007. doi: 10.1214/07-PS119.

Victor H de la Pe˜ na, Michael J Klass, and Tze Leung Lai. Theory and applications of multivariate self-normalized processes. Stochastic Processes and their Applications , 119(12):42104227, 2009.

Audrey Durand, Odalric-Ambrym Maillard, and Joelle Pineau. Streaming kernel regression with provably adaptive mean, variance, and regularization. The Journal of Machine Learning Research , 19(1):650-683, 2018.

Rick Durrett. Probability: theory and examples , volume 49. Cambridge university press, 2019.

Vivek Farias, Ciamac Moallemi, Tianyi Peng, and Andrew Zheng. Synthetically controlled bandits. arXiv preprint arXiv:2202.07079 , 2022.

Matthew Hoffman, Eric Brochu, and Nando De Freitas. Portfolio allocation for Bayesian optimization. In UAI , pages 327-336, 2011.

Steven R Howard, Aaditya Ramdas, Jon McAuliffe, and Jasjeet Sekhon. Time-uniform Chernoff bounds via nonnegative supermartingales. Probability Surveys , 17:257-317, 2020.

Steven R Howard, Aaditya Ramdas, Jon McAuliffe, and Jasjeet Sekhon. Time-uniform, nonparametric, nonasymptotic confidence sequences. The Annals of Statistics , 49(2), 2021.

David Janz. Sequential decision making with feature-linear models . PhD thesis, 2022.

David Janz, David Burt, and Javier Gonz´ alez. Bandit optimisation of functions in the mat´ ern kernel RKHS. In International Conference on Artificial Intelligence and Statistics , pages 2486-2495. PMLR, 2020.

Tor Lattimore. A lower bound for linear and kernel regression with adaptive covariates. In The Thirty Sixth Annual Conference on Learning Theory , pages 2095-2113. PMLR, 2023.

Tor Lattimore and Csaba Szepesv´ ari. Bandit Algorithms . Cambridge University Press, 2020.

Peter D Lax. Functional Analysis , volume 55. John Wiley & Sons, 2002.

Lihong Li, Wei Chu, John Langford, and Robert E Schapire. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th International Conference on World Wide web , pages 661-670, 2010.

Yishay Mansour, Aleksandrs Slivkins, and Vasilis Syrgkanis. Bayesian incentive-compatible bandit exploration. Operations Research , 68(4):1132-1161, 2020.

Aditya Mate, Jackson Killian, Haifeng Xu, Andrew Perrault, and Milind Tambe. Collapsing bandits and their application to public health intervention. Advances in Neural Information Processing Systems , 33:15639-15650, 2020.

Gabriele Santin and Robert Schaback. Approximation of eigenfunctions in kernel-based spaces. Advances in Computational Mathematics , 42(4):973-993, 2016.

Jonathan Scarlett, Ilija Bogunovic, and Volkan Cevher. Lower bounds on regret for noisy Gaussian process bandit optimization. In Conference on Learning Theory , pages 1723-1742. PMLR, 2017.

Shubhanshu Shekhar and Tara Javidi. Gaussian process bandits with adaptive discretization. Electronic Journal of Statistics , 12(2):3829 - 3874, 2018.

Shubhanshu Shekhar and Tara Javidi. Multi-scale zero-order optimization of smooth functions in an RKHS. 2022 IEEE International Symposium on Information Theory (ISIT) , pages 288293, 2020.

Shubhanshu Shekhar and Tara Javidi. Instance dependent regret analysis of kernelized bandits. In International Conference on Machine Learning , pages 19747-19772. PMLR, 2022.

Aleksandrs Slivkins et al. Introduction to multi-armed bandits. Foundations and Trends® in Machine Learning , 12(1-2):1-286, 2019.

Marta Soare, Alessandro Lazaric, and R´ emi Munos. Best-arm identification in linear bandits. Advances in Neural Information Processing Systems , 27, 2014.

Niranjan Srinivas, Andreas Krause, Sham M Kakade, and Matthias Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. International Conference on Machine Learning , 2009.

Sattar Vakili, Kia Khezeli, and Victor Picheny. On information gain and regret bounds in Gaussian process bandits. In International Conference on Artificial Intelligence and Statistics , pages 82-90. PMLR, 2021.

Sattar Vakili, Jonathan Scarlett, and Tara Javidi. Open problem: Tight online confidence intervals for RKHS elements. In Conference on Learning Theory , pages 4647-4652. PMLR, 2021.

Michal Valko, Nathaniel Korda, R´ emi Munos, Ilias Flaounas, and Nelo Cristianini. Finite-time analysis of kernelised contextual bandits. Proceedings of the 29th Conference on Uncertainty in Artificial Intelligence , 2013.

Martin J Wainwright. High-dimensional Statistics: A Non-asymptotic Viewpoint , volume 48. Cambridge university press, 2019.

Ian Waudby-Smith and Aaditya Ramdas. Estimating means of bounded random variables by betting. Journal of the Royal Statistical Society, Series B , 2023.

Peter Whittle. Multi-armed bandits and the Gittins index. Journal of the Royal Statistical Society: Series B (Methodological) , 42(2):143-149, 1980.

Christopher KI Williams and Carl Edward Rasmussen. Gaussian processes for machine learning , volume 2. MIT press Cambridge, MA, 2006.

Peer reviews:
comments: SUMMARY:
The paper proposes a new analysis for the popular GP-UCB algorithm. Under this new approach, it is shown that GP-UCB achieves a sub-linear regret (though not optimal), partially resolving the question of whether GP-UCB can achieve optimal regret.

STRENGTHS:
The paper proposes a new self-normalized martingale type inequality for infinite dimensional Hilbert spaces, which might be of independent interest to the community. The sub-linear convergence of GP-UCB shown by the paper is indeed of interest to better understand the observed performance of popular algorithm.

WEAKNESSES:
I can see that there is clearly certain improvement over the existing result. However, my concern is whether the result warrants a full publication. From what I understand, the important trick is to analyse in the Hilbert space representation, which allows for a tighter analysis and a cleaner dependence on the regularization parameter.

I don't think that the approach taken in Chowdhury and Gopalan is incompatible with the philosophy presented in this paper, although I do agree that reaching this conclusion (specifically dependence on the regularization parameter) might not be as easy.

So overall, I feel there is certainly _some_ novel contribution here, especially with regards to understanding the behaviour of GP-UCB. However, I don't think there is sufficient contribution for a full paper at NeurIPS. I feel the result is more appropriate for a shorter paper or a letter to a journal. I would encourage the authors to submit to such a venue.

QUESTIONS:
No particular question.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-21

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis is a very well written paper with nice, clean results on self-normalised concentration for seperable RKHS. The paper claims two contributions:\n1. A new self-normalised concentration result for seperable RKHSs.\n2. Improving the regret of GP-UCB by tuning the regularisation parameter.\n\nSTRENGTHS:\nThe write-up is of excellent quality. The bounds derived are very neat and strong. The trick used to get better regret is neat and easy to use.\n\nWEAKNESSES:\nThe result claimed in point 1 is actually well known, and has been for over a decade. See Theorem 4.1 in [1]. Indeed, the authors of the submitted manuscript state: \"To the best of our knowledge, Fact 6 is the only existing result on self-normalised, time-uniform concentration in RKHS's [sic]\", where Fact 6 is from [2], and Fact 6 is implied by (and strictly weaker than) Theorem 4.1 of [1]. This is unfortunate, and completely not the authors fault ([2] should not have been accepted in the first place). However, this isn't an issue that can be resolved by adding a citation to [1] in a camera ready version: even the paper's title is, after all, \"Improved Self-Normalized Concentration in Hilbert Spaces\". For this reason, I recommend that the paper is rejected.\n\nThe result claimed in point 2 relies on trading off the effective dimension term versus the regularisation term. This is a neat idea and very much ought to be published, by itself, in a paper that doesn't claim 1 as a contribution. Point 2 is very similar to Theorem 31 and 32 of [3]. There, the effective dimension is likewise traded off, but with RKHS norm (which the regularisation multiplies). Theorem 31 uses this trick to achieve a weaker result (the resulting implied regret is not sublinear for all $\\nu, d$ for the Mat\\'ern kernel); under an additional assumption, which has only been established for a particular choice of $d, \\nu$ (uniform boundedness of kernel eigenfunctions). Theorem 32 would give the same result for the Mat\\'ern kernel as in the submitted manuscript. There, the trade-off is done by changing lengthscale---doing it via regularisation, as done in the submission, is a much better idea.\n\nA rewrite of this paper that makes clear what is previous work and what is the authors contribution would warrant a strong accept.\n\n[1] Abbasi-Yadkori, Yasin. Online learning for linearly parametrized control problems. Diss. 2013.\n\n[2] Chowdhury, Sayak Ray, and Aditya Gopalan. \"On kernelized multi-armed bandits.\" International Conference on Machine Learning. PMLR, 2017.\n\n[3] Janz, David. Sequential decision making with feature-linear models. Diss. 2022.\n\nQUESTIONS:\nNone\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 1, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-30"}

comments: SUMMARY:
This paper addresses the kernelized bandit problem and focuses on improving the regret bounds of the Gaussian Process Upper Confidence Bound (GP-UCB) algorithm. The authors introduce novel techniques to achieve nearly optimal regret rates for GP-UCB. These improvements surpass previous state-of-the-art analyses and partially resolve an open problem posed by Vakili et al.

STRENGTHS:
The paper suggest a new analysis the leads to improved regret bounds for the GP-UCB algorithm. 

The technique used in the paper is based on a novel concentration inequality (Theorem 1) that may be of independent interest.
In addition, they demonstrate that the consideration of regularization based on kernel smoothness contribute to the nearly optimal regret rates as well.

WEAKNESSES:
The main weakness is perhaps clarity of presentation that renders the paper less accessible to a non-expert in the field. 

For example, the authors refer to the Matern kernel as a commonly used kernel, yet to someone not as familiar with the specific literature it is harder to interpret the result. Perhaps consider providing a bit more background on it. (e.g., it was not mentioned in [4]) 
Why is it commonly used and what are trade-offs with other kernels? 

Also consider including a table comparing the bounds with previous analysis, similar to Table 1 in [27], that would be helpful as well.

QUESTIONS:
see above

LIMITATIONS:
none
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper re-investigates the upper bound analysis of the popular GP-UCB algorithm (in its vanilla version), focusing on the cumulative regret for Matern kernel (and more generally, kernels with polynomial decay).  The study reveals that the previous non-sub-linear regret bounds can be surpassed.\n\nSTRENGTHS:\nFrom the overall achieved result of this paper, I would say the authors have made a great contribution to this community, addressing an existing major concern that GP-UCB on Matern kernel fails to achieve sub-linear cumulative regret.\n\nThis finding would have a significant impact on tons of exisintg upper bounds based on IGP-UCB (Chowdhury & Gopalan, 2017), and its certain variants. The behind insights may also inspire future improvements, to see if this bound can be finally optimal that counteracts the lower bound in (Scarlett et.al, 2017).\n\nWEAKNESSES:\nHowever, I have some reservations about their technical details, particularly regarding their derivation of Theorem 1. Here are several reasons:\n* The technical contribution of this paper appears to be non-significant, as it essentially extends a well-known result from linear bandits (Abbasi-Yadkori et.at, 2011), by fusing several exsiting results/facts.\n* The extension starts with assuming a finite-dimensional RKHS (with dimension N) and then somehow lets $N\\to\\infty$.\n* Though the truncation technique is widely used in analyzing infinite-dimensional spaces, it usually involves rigorous analysis of the residual term, which is not seeing to be well-treated in this paper.\n* In Appendix, line 461, the statement \"there exists some...\" seems to compress a lot of information.  While I understand the existence of such $N_t$ should be due to eigenvalue decays, such existence should be proven or more thoroughly discussed.\n* It is confusing to write  $||A||_H \\overrightarrow{dim(H)\\to\\infty} ||A||_H$, as the two RKHS spaces are essentially not the same (finite v.s. infinite), where they are both being denoted by the same $H$.\n* Overall, I was expected to see a more formal proof by properly handling the residual error term, but what I see is more like \"spamming\" $N\\to\\infty$.\n\nOn the other hand, the presentation is not appealing due to:\n* Section 1 lacks intuitions about the tools/ideas to be used, which is essential for a technical paper.\n* Section 2 is too long, and could be significantly reduced either by moving some less-important content to Appendix, or just providing informal results.\n* Some terminologies, like \"double mixture,\" are not explained, leaving readers without clear understanding.\n\nMiscellaneous minor issues:\n* The templates seem to be from NeurIPS 2020, not 2023\n* line 182: \"We can then making...\"\n* Appendix line 471: \"so so\"\n* What is $\\rho \\downarrow 0$?\n* \"Holder\" should be Hölder\n* No formal definition of RKHS norm?\n\nQUESTIONS:\nSince the self-normalized concentration can also be used to show bounds for GP-TS, as was done by (Chowdhury & Gopalan, 2017), does the improved Theorem 1 result in similar improved results for GP-TS?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the open question of Vakili et al, on the possibility of obtaining rate-optimal regret bounds for kernelized bandits under the RKHS assumption. Authors present a slightly different RKHS martingale bound, which allows them to obtain a regret of the form $\\rho\\sqrt{\\gamma_TT}+ \\gamma_T\\sqrt{T}$.  By carefully choosing the regularization parameter $\\rho$ corresponding to the smoothness of the kernel, they are able to improve the rate of regret of the well known GP-UCB. This is particularly important in the case of rough Matern kernels, where the classic bounds become vacuous, while this paper still shows sublinearity.\n\nSTRENGTHS:\n1. The paper gives non-vacuous regret bounds for GP-UCB under RKHS assumption, despite many of the classical prior works (e.g. Srinivas  et al or Chowdhury & Gopalan). \n\n2. I think the key strength is the nice and simple idea of tuning the regularization parameter according to the smoothness of the kernel. This choice makes a lot of sense and I am surprised that we have not been doing so all along.\n\n3. The proof technique (truncation trick) for obtaining the $\\mathcal H$-valued bound is also quite intuitive and interesting. I think the approach say in Lemma 2 may be used in other applications, to easily go from finite-dimensional analysis to kernels.\n\n4. The paper is very well written, its clear and has a great flow. It gives good education!\n\nWEAKNESSES:\n1.This paper does not resolve the open question of Vakili et al. As far as I know, the key question within the field is to whether we can get a $\\sqrt{T\\gamma_T}$ bound on the regret in the RKHS setting, as we do in the GP setting.\nEffectively, this paper improves the dependency of $\\gamma_T$ on $T$, so that while we still have a $\\gamma_T\\sqrt{T}$ bound, the gap with the lower bound [Scarlet et al 2017] and the GP upper bound is reduced.\n\nThe authors conjecture in the conclusion that fully closing this gap may not be possible in the Conclusion section. I believe that is correct. In fact, the paper \"A Lower Bound for Linear and Kernel Regression with Adaptive Covariates\" [Lattimore, COLT 2023] addresses the same open problem, and shows that such confidence sets cannot be improved by any estimator. \n\n2.I should also point out Chowdhurry & Gopalan is not the only result for bounding $\\mathcal H$-valued martingales. As far as I know, this was first introduced in Theorem 3.11 in the PhD Thesis of Yasin Abbasi Yadkori in 2013, titled \"Online Learning for Linearly Parametrized Control Problems\" published by University of Alberta. Could you please compare the Theorem 1 in the paper with Theorem 3.11 of Abbasi-Yadkori? Due to difference in notation I can rigorously compare the bounds. How do the proof techniques differ?\n\nQUESTIONS:\nIn Chowdhurry and Gopalan's paper, what would happen if we set $\\eta$ also as a function of $\\eta$?\nCould we in the end obtain the same rate for regret? The bounding process (rhs of the inequality) in Theorem 1, is similar to that of Lemma 1 in Chowdhurry & Gopalan, while I can see that the martingale is slightly different.\nIs this slight difference actually stopping them from getting the same rates? Or could it be that by tuning $\\eta$ different than them (the simply set it to $1/t$) a similar rate as Cor 2 may be obtained?\n\nLIMITATIONS:\nLimitations are adequately addressed. The paper \"A Lower Bound for Linear and Kernel Regression with Adaptive\nCovariates\" Lattimore COLT 2023 should be mentioned.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-27"}

comments: SUMMARY:
This paper investigates the performance of the Gaussian Process Upper Confidence Bound (GP-UCB) algorithm in kernelized bandit problems, focusing on the minimization of regret. The authors present a novel, self-normalized concentration inequality and a technique to regularize in proportion to the kernel's smoothness, which significantly simplifies the analysis of GP-UCB. The study reveals that GP-UCB, when properly regularized, provides nearly optimal, sublinear regret for kernels experiencing polynomial eigendecay, including the Matern kernel. The results validate the empirical performance of GP-UCB and emphasize the importance of thoughtful regularization in online learning problems.

STRENGTHS:
NA

WEAKNESSES:
NA

QUESTIONS:
NA

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-08-01