Instance-Dependent Bounds for Zeroth-order Lipschitz Optimization with Error Certificates

d. This result , which was only ( and partially ) known in dimension d = 1 , solves an open problem dating back to 1991 . In terms of techniques , our upper bound relies on a packing bound by Bouttier et al . ( 2020 ) for the Piyavskii-Shubert algorithm that we link to the above integral . We also show that a certified version of the computationally tractable DOO algorithm matches these packing and integral bounds . Our instance-dependent lower bound differs from traditional worst-case lower bounds in the Lipschitz setting and relies on a local worst-case analysis that could likely prove useful for other learning tasks . 1 Introduction . The problem of optimizing a black-box function f with as few evaluations of f as possible arises in many scientific and industrial fields such as computer experiments ( Jones et al. , 1998 ; Richet et al. , 2013 ) or automatic selection of hyperparameters in machine learning ( Bergstra et al. , 2011 ) . For safety-critical applications , e.g. , in aircraft or nuclear engineering , using sample-efficient methods is not enough . Certifying the accuracy of the output of the optimization method can be a crucial additional requirement ( Vanaret et al. , 2013 ) . As a concrete example , Azzimonti et al . ( 2021 ) describe a black-box function in nuclear engineering whose output is a k-effective multiplication factor , for which a higher value corresponds to a higher nuclear hazard . Certifying the optimization error is a way to certify the worst-case k-effective factor , which may be required by safety authorities . In this paper , we formally study the problem of finding and certifying an ε-approximate maximizer of a Lipschitz function f of d variables and characterize the optimal number of evaluations of any such function f to achieve this goal . We start by formally defining the setting . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . 1.1 Setting : Zeroth-order Lipschitz Optimization with Error Certificates . Let f : X → R be a function on a compact non-empty subset X of Rd and x ? ∈ X a maximizer . Lipschitz assumption . We assume that f is Lipschitz with respect to a norm ‖·‖ , that is , there exists L ≥ 0 such that ∣∣f ( x ) − f ( y ) ∣∣ ≤ L ‖x− y‖ for all x , y ∈ X . Furthermore , we assume such a Lipschitz bound L to be known . Even though the smallest Lipschitz constant Lip ( f ) : = min { L′ ≥ 0 : f is L′-Lipschitz } is well defined mathematically , it is rarely known exactly in practical black-box problems . As a theoretical curiosity , we will briefly discuss the case L = Lip ( f ) ( i.e. , when the best Lipschitz constant of the unknown black-box function f is known exactly ) in Section 4 , but for most of our results , we will make the following more realistic assumption . Assumption 1 . For some known Lipschitz constant L , the function f : X → R belongs to FL : = { g : X → R | g is Lipschitz and Lip ( g ) < L } . ( 1 ) The Lipschitzness of f implies the weaker property that f ( x ? ) − f ( x ) ≤ L ‖x ? − x‖ for all x ∈ X , sometimes referred to as Lipschitzness around a maximizer x ? ∈ X . Although this is not the focus of our work , we will mention when our results hold under this weaker assumption . Online learning protocol . We study the case in which f is black-box , i.e. , except for the a priori knowledge of L , we can only access f by sequentially querying its values at a sequence x1 , x2 , . . . ∈ X of points of our choice . At every round n ≥ 1 , the query point xn can be chosen as a deterministic function of the values f ( x1 ) , . . . , f ( xn−1 ) observed so far . At the end of round n , using all the values f ( x1 ) , . . . , f ( xn ) , the learner outputs two quantities : • a recommendation x ? n ∈ X , with the goal of minimizing the optimization error ( a.k.a . simple regret ) : max ( f ) − f ( x ? n ) ; • an error certificate ξn ≥ 0 , with the constraint to correctly upper bound the optimization error for any L-Lipschitz function f : X → R , i.e. , so that max ( f ) − f ( x ? n ) ≤ ξn . We call certified algorithm any algorithm for choosing such a sequence ( xn , x ? n , ξn ) n≥1 . Our goal is to quantify the smallest number of evaluations of f that certified algorithms need in order to find and certify an approximate maximizer of f at accuracy ε . This objective motivates the following definition . For any accuracy ε > 0 , we define the sample complexity ( that could also be called query complexity ) of a certified algorithm A for an L-Lipschitz function f as σ ( A , f , ε ) : = inf { n ≥ 1 : ξn ≤ ε } ∈ { 1 , 2 , . . . } ∪ { +∞ } . ( 2 ) This corresponds to the first time when we can stop the algorithm while being sure to have an ε-optimal recommendation x ? n . 1.2 Main Contributions and Outline of the Paper . The main result of this paper is a tight characterization ( up to a log factor ) of the optimal sample complexity of certified algorithms in any dimension d ≥ 1 , solving a three-decade old open problem raised by Hansen et al . ( 1991 ) . More precisely , we prove the following instance-dependent upper and lower bounds , which we later state formally in Theorem 3 of Section 4 ( see also discussions therein , as well as Propositions 2 and 3 for the limit case L = Lip ( f ) ) . Theorem ( Informal statement ) . Under a mild geometric assumption on X , there exists a computationally tractable algorithm A ( e.g. , c.DOO , Algorithm 1 ) such that , for some constants Cd , cd > 0 ( depending exponentially on the dimension d ) , any Lipschitz function f ∈ FL ( see ( 1 ) ) and any accuracy ε , σ ( A , f , ε ) ≤ Cd ∫ X dx ( f ( x ? ) − f ( x ) + ε ) d , ( 3 ) while any certified algorithm A′ must satisfy , for all f ∈ FL , and c ≈ cd ( 1− Lip ( f ) /L ) d/ log ( 1/ε ) , c ∫ X dx ( f ( x ? ) − f ( x ) + ε ) d ≤ σ ( A′ , f , ε ) . ( 4 ) In particular , this result extends to any dimension d ≥ 1 the upper bound proportional to∫ 1 0 dx/ ( f ( x ? ) − f ( x ) + ε ) that Hansen et al . ( 1991 ) derived in dimension d = 1 using arguments specific to the geometry of the real line . Detailed contributions and outline of the paper . We make the following contributions . • As a warmup , we show in Section 2 how to add error certificates to the DOO algorithm ( wellknown in the more classical zeroth-order Lipschitz optimization setting without error certificates , see Perevozchikov 1990 ; Munos 2011 ) . We then upper bound its sample complexity by the quantity SC ( f , ε ) defined in ( 5 ) below . This bound matches a recent bound derived by Bouttier et al . ( 2020 ) for a computationally much more expensive algorithm . In passing , we also slightly improve the packing arguments that Munos ( 2011 ) used in the non-certified setting . • In Section 3 we show that , under a mild geometric assumption on X , the complexity measure SC ( f , ε ) is actually proportional to the integral ∫ X dx/ ( f ( x ? ) − f ( x ) + ε ) d , which implies ( 3 ) above . This extends the bound of Hansen et al . ( 1991 ) ( d = 1 ) to any dimension d. • Finally , in Section 4 , we prove the instance-dependent lower bound ( 4 ) , which differs from traditional worst-case lower bounds in the Lipschitz setting . Our proof relies on a local worstcase analysis that could likely prove useful for other learning tasks . Some of the proofs are deferred to the Supplementary Material , where we also recall useful results on packing and covering numbers ( Section A ) , as well as provide a slightly improved sample complexity bound on the DOO algorithm in the more classical non-certified setting ( Section E ) . 1.3 Related Works . We detail below some connections with the global optimization and the bandit optimization literatures . Zeroth-order Lipschitz optimization with error certificates . The problem of optimizing a function with error certificates has been studied in different settings over the past decades . For instance , in convex optimization , an example of error certificate is given by the duality gap between primal and dual feasible points ( see , e.g. , Boyd and Vandenberghe 2004 ) . In our setting , namely , global zeroth-order Lipschitz optimization with error certificates , most of the attention seems to have been on the very natural ( yet computationally expensive ) algorithm introduced by Piyavskii ( 1972 ) and Shubert ( 1972 ) .1 In dimension d = 1 , Hansen et al . ( 1991 ) proved that its sample complexity σ ( PS , f , ε ) for L-Lipschitz functions f : [ 0 , 1 ] → R is at most proportional to the integral ∫ 1 0 ( f ( x ? ) − f ( x ) + ε ) −1 dx , and left the question of extending the results to arbitrary dimensions open , stating that the task of “ Extending the results of this paper to the multivariate case appears to be difficult ” . Recently , writing Xε : = { x ∈ X : max ( f ) − f ( x ) ≤ ε } for the set of ε-optimal points , X ( a , b ] : = { x ∈ X : a < max ( f ) − f ( x ) ≤ b } for the set of points in between a and b optimal , and N ( E , r ) for the packing number of a set E at scale r ( see Section 1.4 ) , Bouttier et al . ( 2020 , Theorem 2 ) proved a bound valid in any dimension d ≥ 1 roughly of this form : SC ( f , ε ) : = N ( Xε , ε L ) + mε∑ k=1 N ( X ( εk , εk−1 ] , εk L ) , ( 5 ) where the number of terms in the sum is mε : = ⌈ log2 ( ε0/ε ) ⌉ ( with ε0 : = L supx , y∈X ‖x− y‖ ) and the associated scales are given by εmε : = ε and εk : = ε02 −k for all k ∈ { 0 , 1 , . . . , mε − 1 } . The equivalence we prove in Section 3 between SC ( f , ε ) and ∫ X dx/ ( f ( x ? ) − f ( x ) + ε ) d solves in particular the question left open by Hansen et al . ( 1991 ) . The upper bound we prove for the certified DOO algorithm in Section 2 also indicates that the bound SC ( f , ε ) and the equivalent integral bound can be achieved with a computationally much more tractable algorithm.2 Indeed , the Piyavskii-Shubert algorithm requires at every step n to solve an inner global Lipschitz optimization problem close to the computation of a Voronoi diagram ( see discussion in Bouttier et al . 2020 , Section 1.1 ) , hence a running time believed to grow as nΩ ( d ) after n function evaluations . On the contrary , as detailed in Remark 1 ( Section 2 ) , the running time of our certified version of the DOO algorithm is only of the order of n log ( n ) after n of evaluations of f . 1For the interested reader who is unfamiliar with this classic algorithm , we added some details in Section D.3 of the Supplementary Material . 2Tractability refers to running time ( number of elementary operations ) and not number of evaluations of f . Connections with the bandit optimization literature : upper bounds . Our work is also strongly connected to the bandit optimization literature , in which multiple authors studied the global Lipschitz optimization problem with zeroth-order ( or bandit ) feedback , either with perfect ( deterministic ) or noisy ( stochastic ) observations . In the deterministic setting considered here , these papers show that though the number ( L/ε ) d of evaluations associated to a naive grid search is optimal for worst-case Lipschitz functions ( e.g. , Thm 1.1.2 by Nesterov 2003 ) , sequential algorithms can approximately optimize more benign functions with a much smaller number of evaluations . Examples of algorithms with such guarantees in the deterministic setting are the branch-and-bound algorithm by Perevozchikov ( 1990 ) , the DOO algorithm by Munos ( 2011 ) or the LIPO algorithm by Malherbe and Vayatis ( 2017 ) . Examples of algorithms in the stochastic setting are the HOO algorithm by Bubeck et al . ( 2011 ) or the ( generic yet computationally challenging ) Zooming algorithm by Kleinberg et al . ( 2008 , 2019 ) . More examples and references can be found in the textbooks by Munos ( 2014 ) and Slivkins ( 2019 ) . Note however that , except for the work of Bouttier et al . ( 2020 ) mentioned earlier , these bandit optimization papers did not address the problem of certifying the accuracy of the recommendations x ? n . Indeed , all bounds are related to a more classical notion of sample complexity , namely , the minimum number of queries made by an algorithm A before outputting an ε-optimal recommendation : ζ ( A , f , ε ) : = inf { n ≥ 1 : max ( f ) − f ( x ? n ) ≤ ε } ∈ { 1 , 2 , . . . } ∪ { +∞ } . ( 6 ) Though ζ ( A , f , ε ) is always upper bounded by σ ( A , f , ε ) defined in ( 2 ) , these two quantities can differ significantly , as shown by the simple example of constant functions f , for which ζ ( A , f , ε ) = 1 but σ ( A , f , ε ) ≈ ( L/ε ) d since the only way to certify that the output is ε-optimal is essentially to perform a grid-search with step-size roughly ε/L , so as to be sure there is no hidden bump of height more than ε . At a high level , the more “ constant ” a function is , the easier it is to recommend an ε-optimal point , but the harder it is to certify that such recommendation is actually a good recommendation . See the Supplementary Material ( Section E ) for a comparison of bounds . Despite this important difference , the bandit viewpoint ( using packing numbers instead of more specific one-dimensional arguments ) is key to obtain our multi-dimensional integral characterization . Comparison with existing lower bounds . Several lower bounds were derived in the bandit Lipschitz optimization setting without error certificates . When rewritten in terms of the accuracy ε and translated into our deterministic setting , the lower bounds of Horn ( 2006 ) ( when d ? = d/2 ) and of Bubeck et al . ( 2011 ) ( for any d ? ) are of the form infA supf∈Gd ? ζ ( A , f , ε ) & ( 1/ε ) d ? , where Gd ? is the subset of L-Lipschitz functions with near-optimality dimension at most d ? . These are worst-case ( minimax ) lower bounds . On the contrary , our instance-dependent lower bound ( 4 ) quantifies the minimum number of evaluations to certify an ε-optimal point for each function f ∈ FL . Note that here the certified setting enables to obtain meaningful instance-dependent lower bounds , whereas their non-certified counterparts would be trivial ( equal to one ) . Our proof relies on a local worst-case analysis in the same spirit as for distribution-dependent lower bounds in stochastic multi-armed bandits ( see , e.g. , Theorem 16.2 in Lattimore and Szepesvári 2020 ) , yet for continuous instead of finite action sets . We believe this lower bound technique should prove useful for other learning tasks .

Summary: The authors introduced the Certified DOO algorithm, which is a zero-order method, to find the maximal point of a Lipschitz function on a compact feasible set. The authors also provided the convergence rate of the Certified DOO algorithm and an instance-dependent lower bound for all certified algorithms. And the lower bound matches the upper bound op to a logarithm term.