A Domain-Shrinking based Bayesian Optimization Algorithm with Order-Optimal Regret Performance

1 Introduction . Consider a black-box optimization problem with an unknown objective function f : X → R , where X ⊂ Rd is a convex and compact set . The learner can access the function only through a noisy oracle , which , when queried with a point x ∈ X , returns a noisy function value at that point . The learning objective is to approach the maximizer x∗ of the function through a sequence of query points { xt } Tt=1 chosen sequentially in time . The learning efficiency is measured by cumulative regret given by R ( T ) = T∑ t=1 [ f ( x∗ ) − f ( xt ) ] . ( 1 ) This cumulative regret measure dictates the online nature of the problem : every query point during the learning process carries loss , not just the end point xT after learning concludes . The classical exploration-exploitation tradeoff in online learning hence ensues . 1.1 Gaussian Process Models . The above problem is ill-posed unless certain structure of the unknown objective function f is assumed to make learning x∗ feasible . One such structural assumption is the convexity of f , which leads to the class of stochastic convex optimization problems . Another class of black-box optimization problems that is gaining interest in recent years is kernel-based learning where f is assumed to live in a Reproducing Kernel Hilbert Space ( RKHS ) associated with a positive-definite kernel . An effective approach to kernel-based black-box optimization is Bayesian optimization that adopts a fictitious prior on the unknown function f . In other words , while f is deterministic , it is viewed internally by 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . the learning algorithm as a realization of a random process over X . A natural choice is the Gaussian process ( GP ) with a Gaussian prior due to the conjugate property that significantly simplifies the analytical form of the posterior distribution at each newly obtained observation . In a celebrated work , Srinivas et al . [ 1 ] proposed the GP-UCB algorithm that constructs a proxy of f using the upper confidence bound ( UCB ) concept first introduced in the classical multi-armed bandit problem [ 2 , 3 ] . Specifically , at each time instant t , a UCB of f is constructed using the closed-form posterior mean and standard deviation of the GP model of f . The algorithm then sets the next query point to be the maximizer of the UCB . Several variations of GP-UCB , tailored for different settings ( see Sec 1.3 ) , have since been developed . The GP-UCB family of algorithms generally enjoy good empirical performance in terms of regret . The analytical guarantees of their regret performance , however , leave considerable gaps to the existing lower bound [ 4 ] . More significantly , the state-of-the-art regret bound of GP-UCB does not guarantee a sublinear order in T for certain kernels , hence a lack of guaranteed convergence to f ( x∗ ) [ 4 , 5 ] . Another difficulty with the GP-UCB family of algorithms is their computational complexity , which can be prohibitive as the dimension d and/or the horizon length T grows . The computational complexity has two main sources : ( i ) the inversion of the covariance matrix in updating the posterior GP distribution , which has an O ( t3 ) complexity with t samples ; ( ii ) the maximization of the UCB proxy over the entire domain X at each time instant . In particular , due to the multi-modality of the UCB score , its maximization is often carried out using a grid search with an increasingly finer discretization of the entire domain . Specifically , due to analytical requirements , the discretization is typically assumed to grow in the order of O ( t2d ) [ 1 , 6 ] , resulting in an overall computational complexity of O ( T 2d+3 ) . Several studies exist that tackle the first source of high complexity of GP-UCB , using sparse matrix approximation techniques to reduce the complexity in the inversion of the covariance matrix ( see , e.g. , [ 7 , 8 ] ) . The second source , which is the dominating factor , has not been effectively addressed . 1.2 Main results . The goal of this work is to develop a GP-based Bayesian optimization algorithm with a regret guarantee that closes the gap to the lower bound . Furthermore , we tackle the second source of the complexity to ensure both learning efficiency and computational efficiency . Referred to as GP-ThreDS ( Thresholded Domain Shrinking ) , the proposed algorithm is rooted in the methodology of domain shrinking : it continuously prunes sub-performing regions of the domain X and zooms into increasingly smaller high-performing regions of X as time goes . The purpose of the domain shrinking is twofold . First , it ensures high learning efficiency by focusing queries on regions of X with function values approaching f ( x∗ ) . Second , it achieves computational efficiency by avoiding a global maximization of the proxy function over the entire domain X . Our specific approach to domain shrinking is built upon a sequence of localized searches on a growing binary tree that forms successively refined partitions of X . Starting from the root of the tree that represents the entire domain , the search progresses down the tree by adaptively pruning nodes that do not contain the maximizer with high probability , consequently zooming into increasingly smaller high-performing regions of X as the search deepens . Another progressive thread in this sequence of localized searches is the criterion for pruning the tree . Each localized search aims to identify nodes at a certain depth of the tree that contain points with function values exceeding a given threshold . The threshold is updated iteratively to approach the maximum function value f ( x∗ ) . More succinctly , the proposed algorithm is a sequence of localized searches in the domain of the function guided by an iterative search in the range of the function . The above domain shrinking approach via localized search is the primary contributing factor to improved performance in terms of both regret guarantee and computational complexity . In particular , the rate of domain shrinking is controlled to ensure not only the concentration of query points in highperforming regions , but also a constant-sized discretization at all times when estimating the function values . This constant-sized discretization allows a tighter regret analysis and results in a regret upper bound for GP-ThreDS that matches with the lower bound ( up to a poly-logarithmic factor ) . We show that the regret of GP-ThreDS is O ( √ TγT ) ( up to a poly-logarithmic factor ) , where γT denotes the maximum information gain after T steps and is representative of the effective dimension of the problem [ 9 , 10 ] . In the case of Matérn and Squared Exponential ( SE ) kernels where the lower bounds on regret are known , on substituting the improved bounds on γT from [ 11 ] , our results match the lower bounds and close the gap reported in [ 4 , 12 ] . In comparison , the state-of-the-art analysis of GP-UCB yields an O ( γT √ T ) regret bound [ e.g. , see , 6 , Theorem 3 ] . The O ( √ γT ) gap between the regret guarantees of GP-UCB and the proposed GP-ThreDS is significant : it can grow polynomially in T ( e.g . in the case of Matérn kernel ) . Computation-wise , the constant-sized discretization contrasts sharply with the growing ( at rateO ( t2d ) with time t ) discretization required by the GP-UCB family of algorithms . Another factor contributing to the reduced complexity is the relaxed search criterion that aims to determine only the existence of threshold-exceeding points , in contrast to finding a global maximizer as in the GP-UCB family of algorithms . As a result , GP-ThreDS reduces the computational complexity from O ( T 2d+3 ) as required by GP-UCB family of algorithms to O ( T 4 ) . 1.3 Related Work . There is a vast body of literature on numerical and theoretical analysis of Bayesian optimization algorithms . With our focus on a computationally efficient algorithm with a provable regret guarantee , the most relevant results to ours are [ 1 ] and [ 6 ] discussed above . [ 6 ] also proved the same O ( γT √ T ) regret holds for GP-TS , a Bayesian optimization algorithm based on Thompson sampling principle . Augmenting GP models with local polynomial estimators , [ 13 ] introduced LP-GP-UCB and established improved regret bounds for it under special cases [ see , 13 , Sec . 3.2 ] . However , for other cases , the regret guarantees for LP-GP-UCB remain in the same order as GP-UCB . More recently , [ 5 ] introduced π-GP-UCB , specific to Matérn family of kernels , that constructs a cover for the search space , as many hypercubes , and fits an independent GP to each cover element . This algorithm was proven to achieve sublinear regret across all parameters of the Matérn family . Almost all other algorithms in the GP-UCB family have a regret guarantee of O ( γT √ T ) , which is O ( √ γT ) greater than the lower bound and can grow polynomially in T . Two exceptions to this are the SupKernelUCB and the RIPS algorithms proposed in [ 10 ] and [ 14 ] which achieve a regret of O ( √ TγT ) for discrete action spaces.While this may be extendable to continuous spaces via a discretization argument as recently pointed out in [ 5 , 12 ] , the required discretization needs to grow polynomially in T , making it computationally expensive . Moreover , it has been noted that SupKernelUCB performs poorly in practice [ 5 , 8 , 12 ] . GP-ThreDS , on the other hand , is a computationally efficient algorithm that achieves tight regret bounds with good empirical performance ( see Sec . 5 ) . A comparison with other related works including the ones in different settings such as noise-free observations and random f are deferred to the supplementary . 2 Problem Statement . 2.1 Problem Formulation . We consider the problem of optimizing a fixed and unknown function f : X → R , where X ⊂ Rd is a convex and compact domain . A sequential optimization algorithm chooses a point xt ∈ X at each time instant t = 1 , 2 , . . . , and observes yt = f ( xt ) + t , where the noise sequence { t } ∞t=1 is assumed to be i.i.d . over t and R-sub-Gaussian for a fixed constant R ≥ 0 , i.e. , E [ eζ t ] ≤ exp ( ζ2R2/2 ) for all ζ ∈ R and t ∈ N. We assume a regularity condition on the objective function f that is commonly adopted under kernelized learning models . Specifically , we assume that f lives in a Reproducing Kernel Hilbert Space ( RKHS ) 1 associated with a positive definite kernel k : X × X → R. The RKHS norm of f is assumed to be bounded by a known constant B , that is , ‖f‖k ≤ B . We further assume that f is α-Hölder continuous , that is , |f ( x ) − f ( x′ ) | ≤ L‖x − x′‖α for all x , x′ ∈ X for some α ∈ ( 0 , 1 ] and L > 0 . This is a mild assumption as this is a direct consequence of RKHS assumption for commonly used kernels as shown in [ 13 ] . We also assume the knowledge of an interval [ a , b ] , such 1The RKHS , denoted by Hk , is a Hilbert space associated with a positive definite kernel k ( · , · ) and is fully specified by the kernel and vice versa . It is endowed with an inner product 〈·〉k that obeys the reproducing property , i.e. , g ( x ) = 〈g , k ( x , · ) 〉k for all g ∈ Hk . The inner product also induces a norm ‖g‖k = 〈g , g〉k . This norm is a measure of the smoothness of the function f with respect to the kernel k and is finite if and only if f ∈ Hk . that f ( x∗ ) ∈ [ a , b ] . This is also a mild assumption as domain-specific knowledge often provides us with bounds . For example , a common application of black-box optimization is hyperparameter tuning in deep learning models . The unknown function represents the accuracy of the model for a given set of hyperparameters . Since f represents the accuracy of the model , we have f ( x∗ ) ∈ [ 0 , 1 ] . For simplicity of notation , we assume X = [ 0 , 1 ] d and f ( x∗ ) ∈ [ 0 , 1 ] . It is straightforward to relax these assumptions to general compact domains and arbitrary bounded ranges [ a , b ] . Our objective is a computationally efficient algorithm with a guarantee on regret performance as defined in ( 1 ) . We provide high probability regret bounds that hold with probability at least 1− δ0 for any given δ0 ∈ ( 0 , 1 ) , a stronger performance guarantee than bounds on expected regret .

Summary: The paper proposes a Guassian process-based Bayesian optimization algorithm for solving the black-box optimization problems with order-optimal regret guarantee. The main contribution of the paper is in domain-shrinking strategy implemented through tree-based region pruning and focusing on high-performing regions to offer high learning and computational efficiency.   The paper presents solid theoretical proofs of performance analysis, as well as empirical results on benchmark problems.