Optimal Order Simple Regret for Gaussian Process Bandits

√ γN/N ) bound on the simple regret performance of a pure exploration algorithm that is significantly tighter than the existing bounds . We show that this bound is order optimal up to logarithmic factors for the cases where a lower bound on regret is known . To establish these results , we prove novel and sharp confidence intervals for GP models applicable to RKHS elements which may be of broader interest . 1 Introduction . Sequential optimization has evolved into one of the fastest developing areas of machine learning [ 1 ] . We consider sequential optimization of an unknown objective function from noisy and expensive to evaluate zeroth-order1 observations . That is a ubiquitous problem in academic research and industrial production . Examples of applications include exploration in reinforcement learning , recommendation systems , medical analysis tools and speech recognizers [ 4 ] . A notable application in the field of machine learning is automated hyper-parameter tuning . Prevalent methods such as grid search can be prohibitively expensive [ 5 , 6 ] . Sequential optimization methods , on the other hand , are shown to efficiently find good hyper-parameters by an adaptive exploration of the hyper-parameter space [ 7 ] . Our sequential optimization setting is as follows . Consider an objective function f defined over a domain X ⊂ Rd , where d ∈ N is the dimension of the input . A learning algorithm is allowed to perform an adaptive exploration to sequentially observe the potentially corrupted values of the objective function { f ( xn ) + n } Nn=1 , where n are random noises . At the end of N exploration trials , the learning algorithm returns a candidate maximizer x̂∗N ∈ X of f . Let x∗ ∈ argmaxx∈X f ( x ) be a true optimal solution . We may measure the performance of the learning algorithm in terms of simple regret ; that is , the difference between the performance under the true optimal solution , f ( x∗ ) , and that under the learnt value , f ( x̂∗N ) . Our formulation falls under the general framework of continuum armed bandit that signifies receiving feedback only for the selected observation point xn at each time n [ 8 , 9 , 10 , 11 ] . Bandit problems have been extensively studied under numerous settings and various performance measures including simple 1Zeroth-order feedback signifies observations from f in contrast to first-order feedback which refers to observations from gradient of f as e.g . in stochastic gradient descent [ see , e.g. , 2 , 3 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . regret [ see , e.g. , 10 , 12 , 13 ] , cumulative regret [ see , e.g. , 14 , 15 , 16 ] , and best arm identification [ see , e.g. , 17 , 18 ] . The choice of performance measure strongly depends on the application . Simple regret is suitable for situations with a preliminary exploration phase ( for instance hyper-parameter tuning ) in which costs are not measured in terms of rewards but rather in terms of resources expended [ 10 ] . Due to infinite cardinality of the domain , approaching f ( x∗ ) is feasible only when appropriate regularity assumptions on f and noise are satisfied . Following a growing literature [ 19 , 20 , 21 , 22 ] , we focus on a variation of the problem where f is assumed to belong to a reproducing kernel Hilbert space ( RKHS ) that is a very general assumption . Almost all continuous functions can be approximated with the RKHS elements of practically relevant kernels such as Matérn family of kernels [ 19 ] . We consider two classes of noise : sub-Gaussian and light-tailed . Our regularity assumption on f allows us to utilize Gaussian processes ( GPs ) which provide powerful Bayesian ( surrogate ) models for f [ 23 ] . Sequential optimization based on GP models is often referred to as Bayesian optimization in the literature [ 4 , 24 , 25 ] . We build on prediction and uncertainty estimates provided by GP models to study an efficient adaptive exploration algorithm referred to as Maximum Variance Reduction ( MVR ) . Under simple regret measure , MVR embodies the simple principle of exploring the points with the highest variance first . Intuitively , the variance in the GP model is considered as a measure of uncertainty about the unknown objective function and the exploration steps are designed to maximally reduce the uncertainty . At the end of exploration trials , MVR returns a candidate maximizer based on the prediction provided by the learnt GP model . With its simple structure , MVR is amenable to a tight analysis that significantly improves the best known bounds on simple regret . To this end , we derive novel and sharp confidence intervals for GP models applicable to RKHS elements . In addition , we provide numerical experiments on the simple regret performance of MVR comparing it to GP-UCB [ 19 , 20 ] , GP-PI [ 26 ] and GP-EI [ 26 ] . 1.1 Main Results . We first derive novel confidence intervals for GP models applicable to RKHS elements ( Theorems 1 and 2 ) . As part of our analysis , we formulate the posterior variance of a GP model as the sum of two terms : the maximum prediction error from noise-free observations , and the effect of noise ( Proposition 1 ) . This interpretation elicits new connections between GP regression and kernel ridge regression [ 27 ] . These results are of interest on their own . We then build on the confidence intervals for GP models to provide a tight analysis of the simple regret of the MVR algorithm ( Theorem 3 ) . In particular , we prove a high probability Õ ( √ γN N ) 2 simple regret , where γN is the maximal information gain ( see § 2.4 ) . In comparison to Õ ( γN√N ) bounds on simple regret [ see , e.g. , 19 , 20 , 28 ] , we show an O ( √γN ) improvement . It is noteworthy that our bound guarantees convergence to the optimum value of f , while previous Õ ( γN√ N ) bounds do not , since although γN grows sublinearly with N , it can grow faster than √ N . We then specialize our results for the particular cases of practically relevant Matérn and Squared Exponential ( SE ) kernels . We show that our regret bounds match the lower bounds and close the gap reported in [ 28 , 29 ] , who showed that an average simple regret of requires N = Ω ( 1 2 ( log ( 1 ) ) d 2 ) exploration trials in the case of SE kernel . For the Matérn-ν kernel ( where ν is the smoothness parameter , see § 2.1 ) they gave the analogous bound of N = Ω ( ( 1 ) 2+ dν ) . They also reported a significant gap between these lower bounds and the upper bounds achieved by GP-UCB algorithm . In Corollary 1 , we show that our analysis of MVR closes this gap in the performance and establishes upper bounds matching the lower bounds up to logarithmic factors . In contrast to the existing results which mainly focus on Gaussian and sub-Gaussian distributions for noise , we extend our analysis to the more general class of light-tailed distributions , thus broadening the applicability of the results . This extension increases both the confidence interval width and the simple regret by only a multiplicative logarithmic factor . These results apply to e.g . the privacy preserving setting where often a light-tailed noise is employed [ 30 , 31 , 32 ] . 2The notations O and Õ are used to denote the mathematical order and the mathematical order up to logarithmic factors , respectively . 1.2 Literature Review . The celebrated work of Srinivas et al . [ 19 ] pioneered the analysis of GP bandits by proving an Õ ( γN √ N ) upper bound on the cumulative regret of GP-UCB , an optimistic optimization algorithm sequentially selecting xn which maximize an upper confidence bound index over the search space . That implies an Õ ( γN√ N ) simple regret [ 28 ] . Their analysis relied on deriving confidence intervals for GP models applicable to RKHS elements . They also considered a fully Bayesian setting where f is assumed to be a sample from a GP and noise is assumed to be Gaussian . [ 20 ] built on feature space representation of GP models and self-normalized martingale inequalities , first developed in [ 33 ] for linear bandits , to improve the confidence intervals of [ 19 ] by a multiplicative log ( N ) factor . That led to an improvement in the regret bounds by the same multiplicative log ( N ) factor . A discussion on the comparison between these results and the confidence intervals derived in this paper is provided in § 3.3 . A technical comparison with some recent advances in regret bounds requires introducing new notations and is deferred to § 4.4 . The performance of Bayesian optimization algorithms has been extensively studied under numerous settings including contextual information [ 34 ] , high dimensional spaces [ 35 , 36 ] , safety constraints [ 37 , 38 ] , parallelization [ 39 ] , meta-learning [ 40 ] , multi-fidelity evaluations [ 41 ] , ordinal models [ 42 ] , corruption tolerance [ 43 , 29 ] , and neural tangent kernels [ 44 , 45 ] . [ 46 ] introduced an adaptive discretization of the search space improving the computational complexity of a GP-UCB based algorithm . Sparse approximation of GP posteriors are shown to preserve the regret orders while improving the computational complexity of Bayesian optimization algorithms [ 36 , 47 , 48 ] . Under the RKHS setting with noisy observations , GP-TS [ 20 ] and GP-EI [ 49 , 50 ] are also shown to achieve the same regret guarantees as GP-UCB ( up to logarithmic factors ) . All these works report Õ ( γN√ N ) regret bounds . The regret bounds are also reported under other often simpler settings such as noise-free observations [ 51 , 52 , n = 0 , ∀n ] or a Bayesian regret that is averaged over a known prior on f [ 39 , 53 , 54 , 55 , 56 , 57 , 58 , 59 ] , rather than for a fixed and unknown f as in our setting . Other lines of work on continuum armed bandit exist , relying on other regularity assumptions such as Lipschitz continuity [ 9 , 11 , 12 , 60 ] , convexity [ 61 ] and unimodality [ 62 ] , to name a few . A notable example is [ 11 ] who showed that hierarchical algorithms based on tree search yieldO ( N d+1 d+2 ) cumulative regret . We do not compare with these results due to the inherent difference in the regularity assumptions . 1.3 Organization . In § 2 , the problem formulation , the regularity assumptions , and the preliminaries on RKHS and GP models are presented . The novel confidence intervals for GP models are proven in § 3 . MVR algorithm and its analysis are given in § 4 . The experiments are presented in § 5 . We conclude with a discussion in § 6 . 2 Problem Formulation and Preliminaries . Consider an objective function f : X → R , where X ⊆ Rd is a convex and compact domain . Consider an optimal point x∗ ∈ argmaxx∈X f ( x ) . A learning algorithm A sequentially selects observation points { xn ∈ X } n∈N and observes the corresponding noise disturbed objective values { yn = f ( xn ) + n } n∈N , where n is the observation noise . We use the notationsHn = { Xn , Yn } , Xn = [ x1 , x2 , ... , xn ] > , Yn = [ y1 , y2 , ... , yn ] > , xn ∈ X , yn ∈ R , for all n ≥ 1 . In a simple regret setting , the learning algorithm determines a sequence of mappings { Sn } n≥1 where each mapping Sn : Hn → X predicts a candidate maximizer x̂∗n . For algorithm A , the simple regret under a budget of N tries is defined as rAN = f ( x ∗ ) − f ( x̂∗N ) . ( 1 ) The budget N may be unknown a priori . Notationwise , we use Fn = [ f ( x1 ) , f ( x2 ) , . . . , f ( xn ) ] > and En = [ 1 , 2 , . . . , n ] > to denote the noise free part of the observations and the noise history , respectively , similar to Xn and Yn .

Summary: This paper considers the Gaussian process optimization problem where the objective function f lives in a reproducing kernel Hilbert space (RKHS). So far there is a significant gap between the lower and upper bound on the simple regret performance. The authors propose a full exploration algorithm, called Maximum Variance Reduction (MVR) which achieves a simple regret bound $\mathcal O^*(\frac{\sqrt{\gamma_T}}{T})$, where {\gamma_T} is the maximum information gain. This bound significantly improves the existing simple regret bounds. Further, on specific Square Exponential and Mat\’ern kernels, their bound matches the lower bound up to logarithmic factors. Further, the authors extend their results to the more general class of light-tail distributions, thus broadening the applicability of the results. Finally, they validate their theoretical results on several synthetic functions and show that their algorithm is better than other algorithms such as GP-PI, GP-EI, IGP-UCB.