Optimal Order Simple Regret for Gaussian Process Bandits

√ γN/N ) bound on the simple regret performance of a pure exploration algorithm that is significantly tighter than the existing bounds . We show that this bound is order optimal up to logarithmic factors for the cases where a lower bound on regret is known . To establish these results , we prove novel and sharp confidence intervals for GP models applicable to RKHS elements which may be of broader interest . 1 Introduction . Sequential optimization has evolved into one of the fastest developing areas of machine learning [ 1 ] . We consider sequential optimization of an unknown objective function from noisy and expensive to evaluate zeroth-order1 observations . That is a ubiquitous problem in academic research and industrial production . Examples of applications include exploration in reinforcement learning , recommendation systems , medical analysis tools and speech recognizers [ 4 ] . A notable application in the field of machine learning is automated hyper-parameter tuning . Prevalent methods such as grid search can be prohibitively expensive [ 5 , 6 ] . Sequential optimization methods , on the other hand , are shown to efficiently find good hyper-parameters by an adaptive exploration of the hyper-parameter space [ 7 ] . Our sequential optimization setting is as follows . Consider an objective function f defined over a domain X ⊂ Rd , where d ∈ N is the dimension of the input . A learning algorithm is allowed to perform an adaptive exploration to sequentially observe the potentially corrupted values of the objective function { f ( xn ) + n } Nn=1 , where n are random noises . At the end of N exploration trials , the learning algorithm returns a candidate maximizer x̂∗N ∈ X of f . Let x∗ ∈ argmaxx∈X f ( x ) be a true optimal solution . We may measure the performance of the learning algorithm in terms of simple regret ; that is , the difference between the performance under the true optimal solution , f ( x∗ ) , and that under the learnt value , f ( x̂∗N ) . Our formulation falls under the general framework of continuum armed bandit that signifies receiving feedback only for the selected observation point xn at each time n [ 8 , 9 , 10 , 11 ] . Bandit problems have been extensively studied under numerous settings and various performance measures including simple 1Zeroth-order feedback signifies observations from f in contrast to first-order feedback which refers to observations from gradient of f as e.g . in stochastic gradient descent [ see , e.g. , 2 , 3 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . regret [ see , e.g. , 10 , 12 , 13 ] , cumulative regret [ see , e.g. , 14 , 15 , 16 ] , and best arm identification [ see , e.g. , 17 , 18 ] . The choice of performance measure strongly depends on the application . Simple regret is suitable for situations with a preliminary exploration phase ( for instance hyper-parameter tuning ) in which costs are not measured in terms of rewards but rather in terms of resources expended [ 10 ] . Due to infinite cardinality of the domain , approaching f ( x∗ ) is feasible only when appropriate regularity assumptions on f and noise are satisfied . Following a growing literature [ 19 , 20 , 21 , 22 ] , we focus on a variation of the problem where f is assumed to belong to a reproducing kernel Hilbert space ( RKHS ) that is a very general assumption . Almost all continuous functions can be approximated with the RKHS elements of practically relevant kernels such as Matérn family of kernels [ 19 ] . We consider two classes of noise : sub-Gaussian and light-tailed . Our regularity assumption on f allows us to utilize Gaussian processes ( GPs ) which provide powerful Bayesian ( surrogate ) models for f [ 23 ] . Sequential optimization based on GP models is often referred to as Bayesian optimization in the literature [ 4 , 24 , 25 ] . We build on prediction and uncertainty estimates provided by GP models to study an efficient adaptive exploration algorithm referred to as Maximum Variance Reduction ( MVR ) . Under simple regret measure , MVR embodies the simple principle of exploring the points with the highest variance first . Intuitively , the variance in the GP model is considered as a measure of uncertainty about the unknown objective function and the exploration steps are designed to maximally reduce the uncertainty . At the end of exploration trials , MVR returns a candidate maximizer based on the prediction provided by the learnt GP model . With its simple structure , MVR is amenable to a tight analysis that significantly improves the best known bounds on simple regret . To this end , we derive novel and sharp confidence intervals for GP models applicable to RKHS elements . In addition , we provide numerical experiments on the simple regret performance of MVR comparing it to GP-UCB [ 19 , 20 ] , GP-PI [ 26 ] and GP-EI [ 26 ] . 1.1 Main Results . We first derive novel confidence intervals for GP models applicable to RKHS elements ( Theorems 1 and 2 ) . As part of our analysis , we formulate the posterior variance of a GP model as the sum of two terms : the maximum prediction error from noise-free observations , and the effect of noise ( Proposition 1 ) . This interpretation elicits new connections between GP regression and kernel ridge regression [ 27 ] . These results are of interest on their own . We then build on the confidence intervals for GP models to provide a tight analysis of the simple regret of the MVR algorithm ( Theorem 3 ) . In particular , we prove a high probability Õ ( √ γN N ) 2 simple regret , where γN is the maximal information gain ( see § 2.4 ) . In comparison to Õ ( γN√N ) bounds on simple regret [ see , e.g. , 19 , 20 , 28 ] , we show an O ( √γN ) improvement . It is noteworthy that our bound guarantees convergence to the optimum value of f , while previous Õ ( γN√ N ) bounds do not , since although γN grows sublinearly with N , it can grow faster than √ N . We then specialize our results for the particular cases of practically relevant Matérn and Squared Exponential ( SE ) kernels . We show that our regret bounds match the lower bounds and close the gap reported in [ 28 , 29 ] , who showed that an average simple regret of requires N = Ω ( 1 2 ( log ( 1 ) ) d 2 ) exploration trials in the case of SE kernel . For the Matérn-ν kernel ( where ν is the smoothness parameter , see § 2.1 ) they gave the analogous bound of N = Ω ( ( 1 ) 2+ dν ) . They also reported a significant gap between these lower bounds and the upper bounds achieved by GP-UCB algorithm . In Corollary 1 , we show that our analysis of MVR closes this gap in the performance and establishes upper bounds matching the lower bounds up to logarithmic factors . In contrast to the existing results which mainly focus on Gaussian and sub-Gaussian distributions for noise , we extend our analysis to the more general class of light-tailed distributions , thus broadening the applicability of the results . This extension increases both the confidence interval width and the simple regret by only a multiplicative logarithmic factor . These results apply to e.g . the privacy preserving setting where often a light-tailed noise is employed [ 30 , 31 , 32 ] . 2The notations O and Õ are used to denote the mathematical order and the mathematical order up to logarithmic factors , respectively . 1.2 Literature Review . The celebrated work of Srinivas et al . [ 19 ] pioneered the analysis of GP bandits by proving an Õ ( γN √ N ) upper bound on the cumulative regret of GP-UCB , an optimistic optimization algorithm sequentially selecting xn which maximize an upper confidence bound index over the search space . That implies an Õ ( γN√ N ) simple regret [ 28 ] . Their analysis relied on deriving confidence intervals for GP models applicable to RKHS elements . They also considered a fully Bayesian setting where f is assumed to be a sample from a GP and noise is assumed to be Gaussian . [ 20 ] built on feature space representation of GP models and self-normalized martingale inequalities , first developed in [ 33 ] for linear bandits , to improve the confidence intervals of [ 19 ] by a multiplicative log ( N ) factor . That led to an improvement in the regret bounds by the same multiplicative log ( N ) factor . A discussion on the comparison between these results and the confidence intervals derived in this paper is provided in § 3.3 . A technical comparison with some recent advances in regret bounds requires introducing new notations and is deferred to § 4.4 . The performance of Bayesian optimization algorithms has been extensively studied under numerous settings including contextual information [ 34 ] , high dimensional spaces [ 35 , 36 ] , safety constraints [ 37 , 38 ] , parallelization [ 39 ] , meta-learning [ 40 ] , multi-fidelity evaluations [ 41 ] , ordinal models [ 42 ] , corruption tolerance [ 43 , 29 ] , and neural tangent kernels [ 44 , 45 ] . [ 46 ] introduced an adaptive discretization of the search space improving the computational complexity of a GP-UCB based algorithm . Sparse approximation of GP posteriors are shown to preserve the regret orders while improving the computational complexity of Bayesian optimization algorithms [ 36 , 47 , 48 ] . Under the RKHS setting with noisy observations , GP-TS [ 20 ] and GP-EI [ 49 , 50 ] are also shown to achieve the same regret guarantees as GP-UCB ( up to logarithmic factors ) . All these works report Õ ( γN√ N ) regret bounds . The regret bounds are also reported under other often simpler settings such as noise-free observations [ 51 , 52 , n = 0 , ∀n ] or a Bayesian regret that is averaged over a known prior on f [ 39 , 53 , 54 , 55 , 56 , 57 , 58 , 59 ] , rather than for a fixed and unknown f as in our setting . Other lines of work on continuum armed bandit exist , relying on other regularity assumptions such as Lipschitz continuity [ 9 , 11 , 12 , 60 ] , convexity [ 61 ] and unimodality [ 62 ] , to name a few . A notable example is [ 11 ] who showed that hierarchical algorithms based on tree search yieldO ( N d+1 d+2 ) cumulative regret . We do not compare with these results due to the inherent difference in the regularity assumptions . 1.3 Organization . In § 2 , the problem formulation , the regularity assumptions , and the preliminaries on RKHS and GP models are presented . The novel confidence intervals for GP models are proven in § 3 . MVR algorithm and its analysis are given in § 4 . The experiments are presented in § 5 . We conclude with a discussion in § 6 . 2 Problem Formulation and Preliminaries . Consider an objective function f : X → R , where X ⊆ Rd is a convex and compact domain . Consider an optimal point x∗ ∈ argmaxx∈X f ( x ) . A learning algorithm A sequentially selects observation points { xn ∈ X } n∈N and observes the corresponding noise disturbed objective values { yn = f ( xn ) + n } n∈N , where n is the observation noise . We use the notationsHn = { Xn , Yn } , Xn = [ x1 , x2 , ... , xn ] > , Yn = [ y1 , y2 , ... , yn ] > , xn ∈ X , yn ∈ R , for all n ≥ 1 . In a simple regret setting , the learning algorithm determines a sequence of mappings { Sn } n≥1 where each mapping Sn : Hn → X predicts a candidate maximizer x̂∗n . For algorithm A , the simple regret under a budget of N tries is defined as rAN = f ( x ∗ ) − f ( x̂∗N ) . ( 1 ) The budget N may be unknown a priori . Notationwise , we use Fn = [ f ( x1 ) , f ( x2 ) , . . . , f ( xn ) ] > and En = [ 1 , 2 , . . . , n ] > to denote the noise free part of the observations and the noise history , respectively , similar to Xn and Yn .

Summary: The paper derives improved upper bounds for the best-arm identification problem in GP bandits. Specifically, they improve the previously best-known upper bound on simple regret from O(\gamma_N/\sqrt{N}) to O(\sqrt{\gamma_N/N}), where \gamma_N is the maximal information gain and N is the number of function evaluations.   Removing the extra \sqrt{\gamma_N} factor is important as \gamma_N can grow faster than \sqrt{N}. In particular, for the special cases of Matern and SE kernels, the authors show that the new bound (unlike old ones) goes to 0 as N-->\infty and is only logarithmically far from previously established lower bounds.   The improvement in regret analysis boils down to a new confidence ellipsoid bound that is available for both subgaussian and light-tailed noise distributions. Unlike previous works, that focused on cumulative regret, the new confidence ellipsoid bound requires that query points x_1,...,x_n are independent of noise terms e_1,...,e_n for each n. This additional restriction gives improved bounds and is not restrictive for best-arm identification using a maximum variance reduction algorithm.