Optimal Order Simple Regret for Gaussian Process Bandits

√ γN/N ) bound on the simple regret performance of a pure exploration algorithm that is significantly tighter than the existing bounds . We show that this bound is order optimal up to logarithmic factors for the cases where a lower bound on regret is known . To establish these results , we prove novel and sharp confidence intervals for GP models applicable to RKHS elements which may be of broader interest . 1 Introduction . Sequential optimization has evolved into one of the fastest developing areas of machine learning [ 1 ] . We consider sequential optimization of an unknown objective function from noisy and expensive to evaluate zeroth-order1 observations . That is a ubiquitous problem in academic research and industrial production . Examples of applications include exploration in reinforcement learning , recommendation systems , medical analysis tools and speech recognizers [ 4 ] . A notable application in the field of machine learning is automated hyper-parameter tuning . Prevalent methods such as grid search can be prohibitively expensive [ 5 , 6 ] . Sequential optimization methods , on the other hand , are shown to efficiently find good hyper-parameters by an adaptive exploration of the hyper-parameter space [ 7 ] . Our sequential optimization setting is as follows . Consider an objective function f defined over a domain X ⊂ Rd , where d ∈ N is the dimension of the input . A learning algorithm is allowed to perform an adaptive exploration to sequentially observe the potentially corrupted values of the objective function { f ( xn ) + n } Nn=1 , where n are random noises . At the end of N exploration trials , the learning algorithm returns a candidate maximizer x̂∗N ∈ X of f . Let x∗ ∈ argmaxx∈X f ( x ) be a true optimal solution . We may measure the performance of the learning algorithm in terms of simple regret ; that is , the difference between the performance under the true optimal solution , f ( x∗ ) , and that under the learnt value , f ( x̂∗N ) . Our formulation falls under the general framework of continuum armed bandit that signifies receiving feedback only for the selected observation point xn at each time n [ 8 , 9 , 10 , 11 ] . Bandit problems have been extensively studied under numerous settings and various performance measures including simple 1Zeroth-order feedback signifies observations from f in contrast to first-order feedback which refers to observations from gradient of f as e.g . in stochastic gradient descent [ see , e.g. , 2 , 3 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . regret [ see , e.g. , 10 , 12 , 13 ] , cumulative regret [ see , e.g. , 14 , 15 , 16 ] , and best arm identification [ see , e.g. , 17 , 18 ] . The choice of performance measure strongly depends on the application . Simple regret is suitable for situations with a preliminary exploration phase ( for instance hyper-parameter tuning ) in which costs are not measured in terms of rewards but rather in terms of resources expended [ 10 ] . Due to infinite cardinality of the domain , approaching f ( x∗ ) is feasible only when appropriate regularity assumptions on f and noise are satisfied . Following a growing literature [ 19 , 20 , 21 , 22 ] , we focus on a variation of the problem where f is assumed to belong to a reproducing kernel Hilbert space ( RKHS ) that is a very general assumption . Almost all continuous functions can be approximated with the RKHS elements of practically relevant kernels such as Matérn family of kernels [ 19 ] . We consider two classes of noise : sub-Gaussian and light-tailed . Our regularity assumption on f allows us to utilize Gaussian processes ( GPs ) which provide powerful Bayesian ( surrogate ) models for f [ 23 ] . Sequential optimization based on GP models is often referred to as Bayesian optimization in the literature [ 4 , 24 , 25 ] . We build on prediction and uncertainty estimates provided by GP models to study an efficient adaptive exploration algorithm referred to as Maximum Variance Reduction ( MVR ) . Under simple regret measure , MVR embodies the simple principle of exploring the points with the highest variance first . Intuitively , the variance in the GP model is considered as a measure of uncertainty about the unknown objective function and the exploration steps are designed to maximally reduce the uncertainty . At the end of exploration trials , MVR returns a candidate maximizer based on the prediction provided by the learnt GP model . With its simple structure , MVR is amenable to a tight analysis that significantly improves the best known bounds on simple regret . To this end , we derive novel and sharp confidence intervals for GP models applicable to RKHS elements . In addition , we provide numerical experiments on the simple regret performance of MVR comparing it to GP-UCB [ 19 , 20 ] , GP-PI [ 26 ] and GP-EI [ 26 ] . 1.1 Main Results . We first derive novel confidence intervals for GP models applicable to RKHS elements ( Theorems 1 and 2 ) . As part of our analysis , we formulate the posterior variance of a GP model as the sum of two terms : the maximum prediction error from noise-free observations , and the effect of noise ( Proposition 1 ) . This interpretation elicits new connections between GP regression and kernel ridge regression [ 27 ] . These results are of interest on their own . We then build on the confidence intervals for GP models to provide a tight analysis of the simple regret of the MVR algorithm ( Theorem 3 ) . In particular , we prove a high probability Õ ( √ γN N ) 2 simple regret , where γN is the maximal information gain ( see § 2.4 ) . In comparison to Õ ( γN√N ) bounds on simple regret [ see , e.g. , 19 , 20 , 28 ] , we show an O ( √γN ) improvement . It is noteworthy that our bound guarantees convergence to the optimum value of f , while previous Õ ( γN√ N ) bounds do not , since although γN grows sublinearly with N , it can grow faster than √ N . We then specialize our results for the particular cases of practically relevant Matérn and Squared Exponential ( SE ) kernels . We show that our regret bounds match the lower bounds and close the gap reported in [ 28 , 29 ] , who showed that an average simple regret of requires N = Ω ( 1 2 ( log ( 1 ) ) d 2 ) exploration trials in the case of SE kernel . For the Matérn-ν kernel ( where ν is the smoothness parameter , see § 2.1 ) they gave the analogous bound of N = Ω ( ( 1 ) 2+ dν ) . They also reported a significant gap between these lower bounds and the upper bounds achieved by GP-UCB algorithm . In Corollary 1 , we show that our analysis of MVR closes this gap in the performance and establishes upper bounds matching the lower bounds up to logarithmic factors . In contrast to the existing results which mainly focus on Gaussian and sub-Gaussian distributions for noise , we extend our analysis to the more general class of light-tailed distributions , thus broadening the applicability of the results . This extension increases both the confidence interval width and the simple regret by only a multiplicative logarithmic factor . These results apply to e.g . the privacy preserving setting where often a light-tailed noise is employed [ 30 , 31 , 32 ] . 2The notations O and Õ are used to denote the mathematical order and the mathematical order up to logarithmic factors , respectively . 1.2 Literature Review . The celebrated work of Srinivas et al . [ 19 ] pioneered the analysis of GP bandits by proving an Õ ( γN √ N ) upper bound on the cumulative regret of GP-UCB , an optimistic optimization algorithm sequentially selecting xn which maximize an upper confidence bound index over the search space . That implies an Õ ( γN√ N ) simple regret [ 28 ] . Their analysis relied on deriving confidence intervals for GP models applicable to RKHS elements . They also considered a fully Bayesian setting where f is assumed to be a sample from a GP and noise is assumed to be Gaussian . [ 20 ] built on feature space representation of GP models and self-normalized martingale inequalities , first developed in [ 33 ] for linear bandits , to improve the confidence intervals of [ 19 ] by a multiplicative log ( N ) factor . That led to an improvement in the regret bounds by the same multiplicative log ( N ) factor . A discussion on the comparison between these results and the confidence intervals derived in this paper is provided in § 3.3 . A technical comparison with some recent advances in regret bounds requires introducing new notations and is deferred to § 4.4 . The performance of Bayesian optimization algorithms has been extensively studied under numerous settings including contextual information [ 34 ] , high dimensional spaces [ 35 , 36 ] , safety constraints [ 37 , 38 ] , parallelization [ 39 ] , meta-learning [ 40 ] , multi-fidelity evaluations [ 41 ] , ordinal models [ 42 ] , corruption tolerance [ 43 , 29 ] , and neural tangent kernels [ 44 , 45 ] . [ 46 ] introduced an adaptive discretization of the search space improving the computational complexity of a GP-UCB based algorithm . Sparse approximation of GP posteriors are shown to preserve the regret orders while improving the computational complexity of Bayesian optimization algorithms [ 36 , 47 , 48 ] . Under the RKHS setting with noisy observations , GP-TS [ 20 ] and GP-EI [ 49 , 50 ] are also shown to achieve the same regret guarantees as GP-UCB ( up to logarithmic factors ) . All these works report Õ ( γN√ N ) regret bounds . The regret bounds are also reported under other often simpler settings such as noise-free observations [ 51 , 52 , n = 0 , ∀n ] or a Bayesian regret that is averaged over a known prior on f [ 39 , 53 , 54 , 55 , 56 , 57 , 58 , 59 ] , rather than for a fixed and unknown f as in our setting . Other lines of work on continuum armed bandit exist , relying on other regularity assumptions such as Lipschitz continuity [ 9 , 11 , 12 , 60 ] , convexity [ 61 ] and unimodality [ 62 ] , to name a few . A notable example is [ 11 ] who showed that hierarchical algorithms based on tree search yieldO ( N d+1 d+2 ) cumulative regret . We do not compare with these results due to the inherent difference in the regularity assumptions . 1.3 Organization . In § 2 , the problem formulation , the regularity assumptions , and the preliminaries on RKHS and GP models are presented . The novel confidence intervals for GP models are proven in § 3 . MVR algorithm and its analysis are given in § 4 . The experiments are presented in § 5 . We conclude with a discussion in § 6 . 2 Problem Formulation and Preliminaries . Consider an objective function f : X → R , where X ⊆ Rd is a convex and compact domain . Consider an optimal point x∗ ∈ argmaxx∈X f ( x ) . A learning algorithm A sequentially selects observation points { xn ∈ X } n∈N and observes the corresponding noise disturbed objective values { yn = f ( xn ) + n } n∈N , where n is the observation noise . We use the notationsHn = { Xn , Yn } , Xn = [ x1 , x2 , ... , xn ] > , Yn = [ y1 , y2 , ... , yn ] > , xn ∈ X , yn ∈ R , for all n ≥ 1 . In a simple regret setting , the learning algorithm determines a sequence of mappings { Sn } n≥1 where each mapping Sn : Hn → X predicts a candidate maximizer x̂∗n . For algorithm A , the simple regret under a budget of N tries is defined as rAN = f ( x ∗ ) − f ( x̂∗N ) . ( 1 ) The budget N may be unknown a priori . Notationwise , we use Fn = [ f ( x1 ) , f ( x2 ) , . . . , f ( xn ) ] > and En = [ 1 , 2 , . . . , n ] > to denote the noise free part of the observations and the noise history , respectively , similar to Xn and Yn .

Summary: The paper introduces a new approach for simple regret minimization in Bayesian optimization. This is a useful testbed as an intermediate setting harder than pure black-box optimization due to the presence of noise, but avoiding some of the worst case pitfalls of a cumulative regret analysis.  Note that a GP with noisy observation will use regularization in its posterior, and that standard analysis is to split the prediction error in a part depending on the regularized posterior as if it was trained on the noiseless feedback and an error term as if it was trained only on the noise. From this template the paper derives its two main novel approaches: - Prop. 1, a new equality that connects the maximum (worst-case) prediction error of a regularized posterior trained on (unavailable) noiseless observation to the posterior variance of the regularized GP plus a regularization term depending on the regularization (and therefore on the noise level)  - Thm. 1, where the main difference is that using Prop. 1 the authors construct a point-by-point concentration inequality rather than the more commonly martingale concentrations used in the literature that apply to the whole input space. The upside is that this allows for confidence intervals whose radius does not depend on the information gain $\gamma_N$, unlike most methods. The downside is that it requires the noise to be uncorrelated with the observations, which in optimization means that the candidates for evaluation are chosen without looking at the feedback at all. The authors also provide a variant (Thm. 2) for light-tailed noise.  With these two tools the regret analysis is pretty straightforward, relying on an ideal discretization of the input space to extend the for-any concentration to a for-all bound, and then union bound the success probability. The final tool required is a unsupervised strategy to select candidates, and the authors resort to greedy information gain maximization, playing a sort of optimism role in the inner optimization loop. Finally all the candidates are evaluated and the best candidate found is returned.  Compared to other simple regret minimization approaches, this shaves a $\sqrt{\gamma_N}$ factor off, taking it closer to its min-max lower bound.   Experimentally, the proposed approach does not seem to outperform existing methods, slightly weakening the argument that the tighter confidence interval improves exploration/exploitation trade-offs in practice.