Regret Bounds for Gaussian-Process Optimization in Large Domains

1 Introduction . In practice , nonconvex , gradient-free optimization problems arise frequently , for instance when tuning the parameters of an algorithm or optimizing the controller of a physical system ( see e.g . Shahriari et al . [ 2016 ] ) for more concrete examples ) . Different versions of this problem have been addressed e.g . in the multi-armed-bandit and the Bayesian-optimization literature ( which we review below ) . However , the situation where the number of admissible function evaluations is too small to identify the global optimum has so far received little attention , despite arising frequently in practice . For instance , when optimizing the hyper-parameters of a machine-learning method or the controllerparameters of a physical system , it is typically impossible to explore the domain exhaustively . We take first steps towards a better understanding of this setting through a theoretical analysis based on the adaptive-submodularity framework by Golovin and Krause [ 2011 ] . It is not the purpose of the present paper to propose a novel algorithm with better performance ( we modify the EI and UCB strategies merely to facilitate the proof ) , but rather to gain an understanding of how GP-optimization performs in the aforementioned setting , as a function of the number of evaluations and the domain size . As is done typically , we model the uncertainty about the underlying function as a Gaussian Process ( GP ) . The question is how close we can get to the global optimum with T function evaluations , where T is small relative to the domain of the function ( i.e. , it is impossible to identify the global optimum with high confidence with only T evaluations ) . As a performance measure , we use the Bayesian simple regret , i.e. , the expected difference between the optimal function value and the value attained by the algorithm . For the discrete case with domain size N , we derive a problemindependent regret bound ( i.e. , it only depends on T , N and is worst-case in terms of the GP prior ) for two optimization algorithms that are closely related to the well-known expected improvement ( EI , Jones et al . [ 1998 ] ) and the upper confidence bound ( UCB , Auer et al . [ 2002 ] , Srinivas et al . ∗manuel.wuthrich @ pm.me 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . [ 2010 ] ) methods . In contrast to related work , our bounds are non-vacuous even when we can only explore a small fraction of the function . We extend this result to continuous domains and show that the resulting bound scales better with the size of the function domain ( equivalently , the Lipschitz constant of the covariance function ) than related work ( Grünewälder et al . [ 2010 ] ) . 1.1 Related Work . The multi-armed bandit literature is closest to the present paper . In the multi-armed bandit problem , the agent faces a row of N slot machines ( also called one-armed bandits , hence the name ) . The agent can decide at each of T rounds which lever ( arm ) to pull , with the objective of maximizing the payoff . This problem was originally proposed in Robbins [ 1952 ] to study the trade-off between exploration and exploitation in a generic and principled way . Since then , this problem has been studied extensively and important theoretical results have been established . Each of the N arms has an associated reward distribution , with a fixed but unknown mean Fn . When pulling arm n at round t , we obtain a payoff Yt : = Fn + En , t where the noise is zero mean E [ En , t ] = 0 , independent across time steps and arms , and identically distributed for a given arm across time steps . Let us denote the arm pulled at time t by At . Performance is typically analyzed in terms of the regret , which is defined as the difference between the mean payoff of the optimal arm and the one pulled at time t Rt : = max n∈ [ N ] Fn − FAt . ( 1 ) Here , we are interested in the setting where En , t is small or even zero and we are allowed to pull only few arms T < N . We classify related work according to the information about the mean Fn and the noise En , t that is available to the agent . 1.1.1 No Prior Information . Traditionally , most authors considered the case where only very basic information about the payoff Fn + En , t is available to the agent , e.g . that its distribution is a member of a given family of distributions or that it takes values in an interval , typically [ 0 , 1 ] . In a seminal paper , Lai and Robbins [ 1985 ] showed that the cumulative regret ∑ t∈ [ T ] Rt grows at least logarithmically in T and proposed a policy that achieves this rate asymptotically . Later , Auer et al . [ 2002 ] proposed an upper confidence bound ( UCB ) strategy and proved that it achieves logarithmic cumulative regret in T uniformly in time , not just asymptotically . Since then , many related results have been obtained for UCB and other strategies , such as Thompson sampling Agrawal and Goyal [ 2011 ] , Kaufmann et al . [ 2012 ] . A number of similar results have also been obtained for the objective of best arm identification Madani et al . [ 2004 ] , Bubeck et al . [ 2009 ] , Audibert and Bubeck [ 2010 ] , where we do not care about the cumulative regret , but only about the lowest regret attained . However , all of these bounds are nontrivial only when the number of plays is larger than the number of arms T > N . This is not surprising , since no algorithm can be guaranteed to perform well with a low number of plays with such limited information . To see this , consider an example where Fi = 1 and Fn = 0 ∀n 6= i . Since the agent has no prior information about which is the right arm i , the best it can do is to randomly try out one after the other . Hence it is clear that to obtain meaningful bounds for T < N we need more prior knowledge about the reward distributions of the arms . 1.1.2 Structural Prior Information . In Dani et al . [ 2008 ] the authors consider the problem where the mean rewards at each bandit are a linear function of an unknown , K-dimensional vector . However , similarly to the work above , the difficulty addressed in this paper is mainly the noise , i.e . the fact that the same arm does not always yield the same reward . For En , t = 0 , the O ( K ) cumulative regret bounds derived in this paper are trivial , since with O ( K ) evaluations we can simply identify the linear function . 1.1.3 Gaussian Prior . More closely related to our work , a number of authors have considered Gaussian priors . Bull [ 2011 ] for instance provides asymptotic convergence rates for the EI strategy with Gaussian-process priors on the unknown function . Srinivas et al . [ 2010 ] propose a UCB algorithm for the Gaussian-process setting . The authors give finite-time bounds on the cumulative regret . However , these bounds are intended for a different setting than the one we consider here : 1 ) They allow for noisy observations and 2 ) they are only meaningful if we are allowed to make a sufficient number of evaluations to attain low uncertainty over the entire GP . Please see Appendix B for more details on the second point . Russo and Van Roy [ 2014 ] use a similar analysis to derive bounds for Thompson sampling , which are therefore subject to similar limitations . In contrast , the bounds in Russo and Van Roy [ 2016 ] do not depend on the entropy of the entire GP , but rather on the entropy of the optimal action . However , our goal here is to derive bounds that are meaningful even when the optimal action can not be found , i.e . its entropy remains large . De Freitas et al . [ 2012 ] complement the work in [ Srinivas et al. , 2010 ] by providing regret bounds for the setting where the function can be observed without noise . However , these are asymptotic and therefore not applicable to the setting we have in mind . Similarly to the present paper , Grünewälder et al . [ 2010 ] analyze the Bayesian simple regret of GP optimization . They provide a lower and an upper bound on the regret of the optimal policy for GPs on continuous domains with covariance functions that satisfy a continuity assumption . Here , we build on this work and derive a bound with an improved dependence on the Lipschitz constant of the covariance function , i.e. , our bound scales better with decreasing length-scales ( and , equivalently , larger domains ) . Unlike [ Grünewälder et al. , 2010 ] , we also consider GPs on finite domains without any restrictions on the covariance . 1.1.4 Adaptive Submodularity . The adaptive-submodularity framework of Golovin and Krause [ 2011 ] is in principle well suited for the kind of analysis we are interested in . However , we will show that the problem at hand is not adaptively submodular , but our proof is inspired by that framework . 2 Problem Definition . 2.1 The Finite Case . In this section we specify the type of bandit problem we are interested in more formally . The goal is to learn about a function with domain A = [ N ] ( we will use [ N ] to denote the set { 1 , ... , N } ) and co-domain R. We represent the function as a sequence F = ( Fn ) n∈ [ N ] . Our prior belief about the function F is assumed to be Gaussian F ∼N ( µ , Σ ) . ( 2 ) At each of the T iterations , we pick an action ( arm ) At from [ N ] at which we evaluate the function . After each action , an observation Yt ∈ R is returned to the agent Yt : = FAt . ( 3 ) Note that here we restrict ourselves to the case where the function can be evaluated without noise . For convenience , we introduce some additional random variables , based on which the optimization algorithm will pick where to evaluate the function next . We denote the posterior mean and covariance at time t by Mt : = E [ F |A : t , Y : t ] ( 4 ) Ct : = COV [ F |A : t , Y : t ] . ( 5 ) In addition , we will need the maximum and minimum observations up to time t Ŷt : = max k∈ [ t ] Yk ∀t ∈ { 1 , .. , T } ( 6 ) Y̌t : = min k∈ [ t ] Yk ∀t ∈ { 1 , .. , T } . ( 7 ) Furthermore , we will make statements about the difference between the smallest and the largest observed value ˆ̌Yt : = Ŷt − Y̌t ∀t ∈ { 1 , .. , T } . ( 8 ) Finally , for notational convenience we define Ŷ , Y̌ , ˆ̌Y : = ŶT , Y̌T , ˆ̌YT . ( 9 ) Analogously , let us define the function minimum F̌ : = minn∈ [ N ] Fn , maximum F̂ : = maxn∈ [ N ] Fn and difference ˆ̌F : = F̂ − F̌ .

Summary: This paper studies the problem of Gaussian process optimization when the number of arms (the domain) is larger than the horizon of the problem. In the finite arm case, it is assumed that the function is drawn from a zero-mean Gaussian prior and the learner is allowed to make T noiseless queries. Two algorithms based on expected improvement and UCB are proposed to solve this by achieving a simple regret that is some fraction of the optimal value. These are shown to be tight. It is shown that similar results can be extended to the continuous domain, but instead depending on a large Lipschitz constant.