A Domain-Shrinking based Bayesian Optimization Algorithm with Order-Optimal Regret Performance

1 Introduction . Consider a black-box optimization problem with an unknown objective function f : X → R , where X ⊂ Rd is a convex and compact set . The learner can access the function only through a noisy oracle , which , when queried with a point x ∈ X , returns a noisy function value at that point . The learning objective is to approach the maximizer x∗ of the function through a sequence of query points { xt } Tt=1 chosen sequentially in time . The learning efficiency is measured by cumulative regret given by R ( T ) = T∑ t=1 [ f ( x∗ ) − f ( xt ) ] . ( 1 ) This cumulative regret measure dictates the online nature of the problem : every query point during the learning process carries loss , not just the end point xT after learning concludes . The classical exploration-exploitation tradeoff in online learning hence ensues . 1.1 Gaussian Process Models . The above problem is ill-posed unless certain structure of the unknown objective function f is assumed to make learning x∗ feasible . One such structural assumption is the convexity of f , which leads to the class of stochastic convex optimization problems . Another class of black-box optimization problems that is gaining interest in recent years is kernel-based learning where f is assumed to live in a Reproducing Kernel Hilbert Space ( RKHS ) associated with a positive-definite kernel . An effective approach to kernel-based black-box optimization is Bayesian optimization that adopts a fictitious prior on the unknown function f . In other words , while f is deterministic , it is viewed internally by 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . the learning algorithm as a realization of a random process over X . A natural choice is the Gaussian process ( GP ) with a Gaussian prior due to the conjugate property that significantly simplifies the analytical form of the posterior distribution at each newly obtained observation . In a celebrated work , Srinivas et al . [ 1 ] proposed the GP-UCB algorithm that constructs a proxy of f using the upper confidence bound ( UCB ) concept first introduced in the classical multi-armed bandit problem [ 2 , 3 ] . Specifically , at each time instant t , a UCB of f is constructed using the closed-form posterior mean and standard deviation of the GP model of f . The algorithm then sets the next query point to be the maximizer of the UCB . Several variations of GP-UCB , tailored for different settings ( see Sec 1.3 ) , have since been developed . The GP-UCB family of algorithms generally enjoy good empirical performance in terms of regret . The analytical guarantees of their regret performance , however , leave considerable gaps to the existing lower bound [ 4 ] . More significantly , the state-of-the-art regret bound of GP-UCB does not guarantee a sublinear order in T for certain kernels , hence a lack of guaranteed convergence to f ( x∗ ) [ 4 , 5 ] . Another difficulty with the GP-UCB family of algorithms is their computational complexity , which can be prohibitive as the dimension d and/or the horizon length T grows . The computational complexity has two main sources : ( i ) the inversion of the covariance matrix in updating the posterior GP distribution , which has an O ( t3 ) complexity with t samples ; ( ii ) the maximization of the UCB proxy over the entire domain X at each time instant . In particular , due to the multi-modality of the UCB score , its maximization is often carried out using a grid search with an increasingly finer discretization of the entire domain . Specifically , due to analytical requirements , the discretization is typically assumed to grow in the order of O ( t2d ) [ 1 , 6 ] , resulting in an overall computational complexity of O ( T 2d+3 ) . Several studies exist that tackle the first source of high complexity of GP-UCB , using sparse matrix approximation techniques to reduce the complexity in the inversion of the covariance matrix ( see , e.g. , [ 7 , 8 ] ) . The second source , which is the dominating factor , has not been effectively addressed . 1.2 Main results . The goal of this work is to develop a GP-based Bayesian optimization algorithm with a regret guarantee that closes the gap to the lower bound . Furthermore , we tackle the second source of the complexity to ensure both learning efficiency and computational efficiency . Referred to as GP-ThreDS ( Thresholded Domain Shrinking ) , the proposed algorithm is rooted in the methodology of domain shrinking : it continuously prunes sub-performing regions of the domain X and zooms into increasingly smaller high-performing regions of X as time goes . The purpose of the domain shrinking is twofold . First , it ensures high learning efficiency by focusing queries on regions of X with function values approaching f ( x∗ ) . Second , it achieves computational efficiency by avoiding a global maximization of the proxy function over the entire domain X . Our specific approach to domain shrinking is built upon a sequence of localized searches on a growing binary tree that forms successively refined partitions of X . Starting from the root of the tree that represents the entire domain , the search progresses down the tree by adaptively pruning nodes that do not contain the maximizer with high probability , consequently zooming into increasingly smaller high-performing regions of X as the search deepens . Another progressive thread in this sequence of localized searches is the criterion for pruning the tree . Each localized search aims to identify nodes at a certain depth of the tree that contain points with function values exceeding a given threshold . The threshold is updated iteratively to approach the maximum function value f ( x∗ ) . More succinctly , the proposed algorithm is a sequence of localized searches in the domain of the function guided by an iterative search in the range of the function . The above domain shrinking approach via localized search is the primary contributing factor to improved performance in terms of both regret guarantee and computational complexity . In particular , the rate of domain shrinking is controlled to ensure not only the concentration of query points in highperforming regions , but also a constant-sized discretization at all times when estimating the function values . This constant-sized discretization allows a tighter regret analysis and results in a regret upper bound for GP-ThreDS that matches with the lower bound ( up to a poly-logarithmic factor ) . We show that the regret of GP-ThreDS is O ( √ TγT ) ( up to a poly-logarithmic factor ) , where γT denotes the maximum information gain after T steps and is representative of the effective dimension of the problem [ 9 , 10 ] . In the case of Matérn and Squared Exponential ( SE ) kernels where the lower bounds on regret are known , on substituting the improved bounds on γT from [ 11 ] , our results match the lower bounds and close the gap reported in [ 4 , 12 ] . In comparison , the state-of-the-art analysis of GP-UCB yields an O ( γT √ T ) regret bound [ e.g. , see , 6 , Theorem 3 ] . The O ( √ γT ) gap between the regret guarantees of GP-UCB and the proposed GP-ThreDS is significant : it can grow polynomially in T ( e.g . in the case of Matérn kernel ) . Computation-wise , the constant-sized discretization contrasts sharply with the growing ( at rateO ( t2d ) with time t ) discretization required by the GP-UCB family of algorithms . Another factor contributing to the reduced complexity is the relaxed search criterion that aims to determine only the existence of threshold-exceeding points , in contrast to finding a global maximizer as in the GP-UCB family of algorithms . As a result , GP-ThreDS reduces the computational complexity from O ( T 2d+3 ) as required by GP-UCB family of algorithms to O ( T 4 ) . 1.3 Related Work . There is a vast body of literature on numerical and theoretical analysis of Bayesian optimization algorithms . With our focus on a computationally efficient algorithm with a provable regret guarantee , the most relevant results to ours are [ 1 ] and [ 6 ] discussed above . [ 6 ] also proved the same O ( γT √ T ) regret holds for GP-TS , a Bayesian optimization algorithm based on Thompson sampling principle . Augmenting GP models with local polynomial estimators , [ 13 ] introduced LP-GP-UCB and established improved regret bounds for it under special cases [ see , 13 , Sec . 3.2 ] . However , for other cases , the regret guarantees for LP-GP-UCB remain in the same order as GP-UCB . More recently , [ 5 ] introduced π-GP-UCB , specific to Matérn family of kernels , that constructs a cover for the search space , as many hypercubes , and fits an independent GP to each cover element . This algorithm was proven to achieve sublinear regret across all parameters of the Matérn family . Almost all other algorithms in the GP-UCB family have a regret guarantee of O ( γT √ T ) , which is O ( √ γT ) greater than the lower bound and can grow polynomially in T . Two exceptions to this are the SupKernelUCB and the RIPS algorithms proposed in [ 10 ] and [ 14 ] which achieve a regret of O ( √ TγT ) for discrete action spaces.While this may be extendable to continuous spaces via a discretization argument as recently pointed out in [ 5 , 12 ] , the required discretization needs to grow polynomially in T , making it computationally expensive . Moreover , it has been noted that SupKernelUCB performs poorly in practice [ 5 , 8 , 12 ] . GP-ThreDS , on the other hand , is a computationally efficient algorithm that achieves tight regret bounds with good empirical performance ( see Sec . 5 ) . A comparison with other related works including the ones in different settings such as noise-free observations and random f are deferred to the supplementary . 2 Problem Statement . 2.1 Problem Formulation . We consider the problem of optimizing a fixed and unknown function f : X → R , where X ⊂ Rd is a convex and compact domain . A sequential optimization algorithm chooses a point xt ∈ X at each time instant t = 1 , 2 , . . . , and observes yt = f ( xt ) + t , where the noise sequence { t } ∞t=1 is assumed to be i.i.d . over t and R-sub-Gaussian for a fixed constant R ≥ 0 , i.e. , E [ eζ t ] ≤ exp ( ζ2R2/2 ) for all ζ ∈ R and t ∈ N. We assume a regularity condition on the objective function f that is commonly adopted under kernelized learning models . Specifically , we assume that f lives in a Reproducing Kernel Hilbert Space ( RKHS ) 1 associated with a positive definite kernel k : X × X → R. The RKHS norm of f is assumed to be bounded by a known constant B , that is , ‖f‖k ≤ B . We further assume that f is α-Hölder continuous , that is , |f ( x ) − f ( x′ ) | ≤ L‖x − x′‖α for all x , x′ ∈ X for some α ∈ ( 0 , 1 ] and L > 0 . This is a mild assumption as this is a direct consequence of RKHS assumption for commonly used kernels as shown in [ 13 ] . We also assume the knowledge of an interval [ a , b ] , such 1The RKHS , denoted by Hk , is a Hilbert space associated with a positive definite kernel k ( · , · ) and is fully specified by the kernel and vice versa . It is endowed with an inner product 〈·〉k that obeys the reproducing property , i.e. , g ( x ) = 〈g , k ( x , · ) 〉k for all g ∈ Hk . The inner product also induces a norm ‖g‖k = 〈g , g〉k . This norm is a measure of the smoothness of the function f with respect to the kernel k and is finite if and only if f ∈ Hk . that f ( x∗ ) ∈ [ a , b ] . This is also a mild assumption as domain-specific knowledge often provides us with bounds . For example , a common application of black-box optimization is hyperparameter tuning in deep learning models . The unknown function represents the accuracy of the model for a given set of hyperparameters . Since f represents the accuracy of the model , we have f ( x∗ ) ∈ [ 0 , 1 ] . For simplicity of notation , we assume X = [ 0 , 1 ] d and f ( x∗ ) ∈ [ 0 , 1 ] . It is straightforward to relax these assumptions to general compact domains and arbitrary bounded ranges [ a , b ] . Our objective is a computationally efficient algorithm with a guarantee on regret performance as defined in ( 1 ) . We provide high probability regret bounds that hold with probability at least 1− δ0 for any given δ0 ∈ ( 0 , 1 ) , a stronger performance guarantee than bounds on expected regret .

Summary: This paper proposes a GP based Bayesian optimization algorithm for regret minimization. This paper uses a domain-shrinking approach where the domain is explored as a tree. Using smoothness assumptions on the true function, it is possible to get upper and lower bounds on the function value in an entire region and this allows the algorithm to decide whether to subdivide a region to explore further or remove it entirely. A key idea and contribution of this work is that the number of leaves in the tree at any one time is a constant. Hence, the total computational complexity of this algorithm is reduced greatly compared to the standard complexity incurred by the famed GP-UCB algorithm. Additionally, the algorithm achieves order optimal performance, closing a gap of \sqrt(\gamma_t) observed in previous work. The paper closes with simple numerical experiments.