Instance-Dependent Bounds for Zeroth-order Lipschitz Optimization with Error Certificates

d. This result , which was only ( and partially ) known in dimension d = 1 , solves an open problem dating back to 1991 . In terms of techniques , our upper bound relies on a packing bound by Bouttier et al . ( 2020 ) for the Piyavskii-Shubert algorithm that we link to the above integral . We also show that a certified version of the computationally tractable DOO algorithm matches these packing and integral bounds . Our instance-dependent lower bound differs from traditional worst-case lower bounds in the Lipschitz setting and relies on a local worst-case analysis that could likely prove useful for other learning tasks . 1 Introduction . The problem of optimizing a black-box function f with as few evaluations of f as possible arises in many scientific and industrial fields such as computer experiments ( Jones et al. , 1998 ; Richet et al. , 2013 ) or automatic selection of hyperparameters in machine learning ( Bergstra et al. , 2011 ) . For safety-critical applications , e.g. , in aircraft or nuclear engineering , using sample-efficient methods is not enough . Certifying the accuracy of the output of the optimization method can be a crucial additional requirement ( Vanaret et al. , 2013 ) . As a concrete example , Azzimonti et al . ( 2021 ) describe a black-box function in nuclear engineering whose output is a k-effective multiplication factor , for which a higher value corresponds to a higher nuclear hazard . Certifying the optimization error is a way to certify the worst-case k-effective factor , which may be required by safety authorities . In this paper , we formally study the problem of finding and certifying an ε-approximate maximizer of a Lipschitz function f of d variables and characterize the optimal number of evaluations of any such function f to achieve this goal . We start by formally defining the setting . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . 1.1 Setting : Zeroth-order Lipschitz Optimization with Error Certificates . Let f : X → R be a function on a compact non-empty subset X of Rd and x ? ∈ X a maximizer . Lipschitz assumption . We assume that f is Lipschitz with respect to a norm ‖·‖ , that is , there exists L ≥ 0 such that ∣∣f ( x ) − f ( y ) ∣∣ ≤ L ‖x− y‖ for all x , y ∈ X . Furthermore , we assume such a Lipschitz bound L to be known . Even though the smallest Lipschitz constant Lip ( f ) : = min { L′ ≥ 0 : f is L′-Lipschitz } is well defined mathematically , it is rarely known exactly in practical black-box problems . As a theoretical curiosity , we will briefly discuss the case L = Lip ( f ) ( i.e. , when the best Lipschitz constant of the unknown black-box function f is known exactly ) in Section 4 , but for most of our results , we will make the following more realistic assumption . Assumption 1 . For some known Lipschitz constant L , the function f : X → R belongs to FL : = { g : X → R | g is Lipschitz and Lip ( g ) < L } . ( 1 ) The Lipschitzness of f implies the weaker property that f ( x ? ) − f ( x ) ≤ L ‖x ? − x‖ for all x ∈ X , sometimes referred to as Lipschitzness around a maximizer x ? ∈ X . Although this is not the focus of our work , we will mention when our results hold under this weaker assumption . Online learning protocol . We study the case in which f is black-box , i.e. , except for the a priori knowledge of L , we can only access f by sequentially querying its values at a sequence x1 , x2 , . . . ∈ X of points of our choice . At every round n ≥ 1 , the query point xn can be chosen as a deterministic function of the values f ( x1 ) , . . . , f ( xn−1 ) observed so far . At the end of round n , using all the values f ( x1 ) , . . . , f ( xn ) , the learner outputs two quantities : • a recommendation x ? n ∈ X , with the goal of minimizing the optimization error ( a.k.a . simple regret ) : max ( f ) − f ( x ? n ) ; • an error certificate ξn ≥ 0 , with the constraint to correctly upper bound the optimization error for any L-Lipschitz function f : X → R , i.e. , so that max ( f ) − f ( x ? n ) ≤ ξn . We call certified algorithm any algorithm for choosing such a sequence ( xn , x ? n , ξn ) n≥1 . Our goal is to quantify the smallest number of evaluations of f that certified algorithms need in order to find and certify an approximate maximizer of f at accuracy ε . This objective motivates the following definition . For any accuracy ε > 0 , we define the sample complexity ( that could also be called query complexity ) of a certified algorithm A for an L-Lipschitz function f as σ ( A , f , ε ) : = inf { n ≥ 1 : ξn ≤ ε } ∈ { 1 , 2 , . . . } ∪ { +∞ } . ( 2 ) This corresponds to the first time when we can stop the algorithm while being sure to have an ε-optimal recommendation x ? n . 1.2 Main Contributions and Outline of the Paper . The main result of this paper is a tight characterization ( up to a log factor ) of the optimal sample complexity of certified algorithms in any dimension d ≥ 1 , solving a three-decade old open problem raised by Hansen et al . ( 1991 ) . More precisely , we prove the following instance-dependent upper and lower bounds , which we later state formally in Theorem 3 of Section 4 ( see also discussions therein , as well as Propositions 2 and 3 for the limit case L = Lip ( f ) ) . Theorem ( Informal statement ) . Under a mild geometric assumption on X , there exists a computationally tractable algorithm A ( e.g. , c.DOO , Algorithm 1 ) such that , for some constants Cd , cd > 0 ( depending exponentially on the dimension d ) , any Lipschitz function f ∈ FL ( see ( 1 ) ) and any accuracy ε , σ ( A , f , ε ) ≤ Cd ∫ X dx ( f ( x ? ) − f ( x ) + ε ) d , ( 3 ) while any certified algorithm A′ must satisfy , for all f ∈ FL , and c ≈ cd ( 1− Lip ( f ) /L ) d/ log ( 1/ε ) , c ∫ X dx ( f ( x ? ) − f ( x ) + ε ) d ≤ σ ( A′ , f , ε ) . ( 4 ) In particular , this result extends to any dimension d ≥ 1 the upper bound proportional to∫ 1 0 dx/ ( f ( x ? ) − f ( x ) + ε ) that Hansen et al . ( 1991 ) derived in dimension d = 1 using arguments specific to the geometry of the real line . Detailed contributions and outline of the paper . We make the following contributions . • As a warmup , we show in Section 2 how to add error certificates to the DOO algorithm ( wellknown in the more classical zeroth-order Lipschitz optimization setting without error certificates , see Perevozchikov 1990 ; Munos 2011 ) . We then upper bound its sample complexity by the quantity SC ( f , ε ) defined in ( 5 ) below . This bound matches a recent bound derived by Bouttier et al . ( 2020 ) for a computationally much more expensive algorithm . In passing , we also slightly improve the packing arguments that Munos ( 2011 ) used in the non-certified setting . • In Section 3 we show that , under a mild geometric assumption on X , the complexity measure SC ( f , ε ) is actually proportional to the integral ∫ X dx/ ( f ( x ? ) − f ( x ) + ε ) d , which implies ( 3 ) above . This extends the bound of Hansen et al . ( 1991 ) ( d = 1 ) to any dimension d. • Finally , in Section 4 , we prove the instance-dependent lower bound ( 4 ) , which differs from traditional worst-case lower bounds in the Lipschitz setting . Our proof relies on a local worstcase analysis that could likely prove useful for other learning tasks . Some of the proofs are deferred to the Supplementary Material , where we also recall useful results on packing and covering numbers ( Section A ) , as well as provide a slightly improved sample complexity bound on the DOO algorithm in the more classical non-certified setting ( Section E ) . 1.3 Related Works . We detail below some connections with the global optimization and the bandit optimization literatures . Zeroth-order Lipschitz optimization with error certificates . The problem of optimizing a function with error certificates has been studied in different settings over the past decades . For instance , in convex optimization , an example of error certificate is given by the duality gap between primal and dual feasible points ( see , e.g. , Boyd and Vandenberghe 2004 ) . In our setting , namely , global zeroth-order Lipschitz optimization with error certificates , most of the attention seems to have been on the very natural ( yet computationally expensive ) algorithm introduced by Piyavskii ( 1972 ) and Shubert ( 1972 ) .1 In dimension d = 1 , Hansen et al . ( 1991 ) proved that its sample complexity σ ( PS , f , ε ) for L-Lipschitz functions f : [ 0 , 1 ] → R is at most proportional to the integral ∫ 1 0 ( f ( x ? ) − f ( x ) + ε ) −1 dx , and left the question of extending the results to arbitrary dimensions open , stating that the task of “ Extending the results of this paper to the multivariate case appears to be difficult ” . Recently , writing Xε : = { x ∈ X : max ( f ) − f ( x ) ≤ ε } for the set of ε-optimal points , X ( a , b ] : = { x ∈ X : a < max ( f ) − f ( x ) ≤ b } for the set of points in between a and b optimal , and N ( E , r ) for the packing number of a set E at scale r ( see Section 1.4 ) , Bouttier et al . ( 2020 , Theorem 2 ) proved a bound valid in any dimension d ≥ 1 roughly of this form : SC ( f , ε ) : = N ( Xε , ε L ) + mε∑ k=1 N ( X ( εk , εk−1 ] , εk L ) , ( 5 ) where the number of terms in the sum is mε : = ⌈ log2 ( ε0/ε ) ⌉ ( with ε0 : = L supx , y∈X ‖x− y‖ ) and the associated scales are given by εmε : = ε and εk : = ε02 −k for all k ∈ { 0 , 1 , . . . , mε − 1 } . The equivalence we prove in Section 3 between SC ( f , ε ) and ∫ X dx/ ( f ( x ? ) − f ( x ) + ε ) d solves in particular the question left open by Hansen et al . ( 1991 ) . The upper bound we prove for the certified DOO algorithm in Section 2 also indicates that the bound SC ( f , ε ) and the equivalent integral bound can be achieved with a computationally much more tractable algorithm.2 Indeed , the Piyavskii-Shubert algorithm requires at every step n to solve an inner global Lipschitz optimization problem close to the computation of a Voronoi diagram ( see discussion in Bouttier et al . 2020 , Section 1.1 ) , hence a running time believed to grow as nΩ ( d ) after n function evaluations . On the contrary , as detailed in Remark 1 ( Section 2 ) , the running time of our certified version of the DOO algorithm is only of the order of n log ( n ) after n of evaluations of f . 1For the interested reader who is unfamiliar with this classic algorithm , we added some details in Section D.3 of the Supplementary Material . 2Tractability refers to running time ( number of elementary operations ) and not number of evaluations of f . Connections with the bandit optimization literature : upper bounds . Our work is also strongly connected to the bandit optimization literature , in which multiple authors studied the global Lipschitz optimization problem with zeroth-order ( or bandit ) feedback , either with perfect ( deterministic ) or noisy ( stochastic ) observations . In the deterministic setting considered here , these papers show that though the number ( L/ε ) d of evaluations associated to a naive grid search is optimal for worst-case Lipschitz functions ( e.g. , Thm 1.1.2 by Nesterov 2003 ) , sequential algorithms can approximately optimize more benign functions with a much smaller number of evaluations . Examples of algorithms with such guarantees in the deterministic setting are the branch-and-bound algorithm by Perevozchikov ( 1990 ) , the DOO algorithm by Munos ( 2011 ) or the LIPO algorithm by Malherbe and Vayatis ( 2017 ) . Examples of algorithms in the stochastic setting are the HOO algorithm by Bubeck et al . ( 2011 ) or the ( generic yet computationally challenging ) Zooming algorithm by Kleinberg et al . ( 2008 , 2019 ) . More examples and references can be found in the textbooks by Munos ( 2014 ) and Slivkins ( 2019 ) . Note however that , except for the work of Bouttier et al . ( 2020 ) mentioned earlier , these bandit optimization papers did not address the problem of certifying the accuracy of the recommendations x ? n . Indeed , all bounds are related to a more classical notion of sample complexity , namely , the minimum number of queries made by an algorithm A before outputting an ε-optimal recommendation : ζ ( A , f , ε ) : = inf { n ≥ 1 : max ( f ) − f ( x ? n ) ≤ ε } ∈ { 1 , 2 , . . . } ∪ { +∞ } . ( 6 ) Though ζ ( A , f , ε ) is always upper bounded by σ ( A , f , ε ) defined in ( 2 ) , these two quantities can differ significantly , as shown by the simple example of constant functions f , for which ζ ( A , f , ε ) = 1 but σ ( A , f , ε ) ≈ ( L/ε ) d since the only way to certify that the output is ε-optimal is essentially to perform a grid-search with step-size roughly ε/L , so as to be sure there is no hidden bump of height more than ε . At a high level , the more “ constant ” a function is , the easier it is to recommend an ε-optimal point , but the harder it is to certify that such recommendation is actually a good recommendation . See the Supplementary Material ( Section E ) for a comparison of bounds . Despite this important difference , the bandit viewpoint ( using packing numbers instead of more specific one-dimensional arguments ) is key to obtain our multi-dimensional integral characterization . Comparison with existing lower bounds . Several lower bounds were derived in the bandit Lipschitz optimization setting without error certificates . When rewritten in terms of the accuracy ε and translated into our deterministic setting , the lower bounds of Horn ( 2006 ) ( when d ? = d/2 ) and of Bubeck et al . ( 2011 ) ( for any d ? ) are of the form infA supf∈Gd ? ζ ( A , f , ε ) & ( 1/ε ) d ? , where Gd ? is the subset of L-Lipschitz functions with near-optimality dimension at most d ? . These are worst-case ( minimax ) lower bounds . On the contrary , our instance-dependent lower bound ( 4 ) quantifies the minimum number of evaluations to certify an ε-optimal point for each function f ∈ FL . Note that here the certified setting enables to obtain meaningful instance-dependent lower bounds , whereas their non-certified counterparts would be trivial ( equal to one ) . Our proof relies on a local worst-case analysis in the same spirit as for distribution-dependent lower bounds in stochastic multi-armed bandits ( see , e.g. , Theorem 16.2 in Lattimore and Szepesvári 2020 ) , yet for continuous instead of finite action sets . We believe this lower bound technique should prove useful for other learning tasks .

Summary: The paper studies the complexity of certifiable zeroth-order optimization of Lipschitz functions. Formally, given oracle access to evaluations of a $L$-Lipschitz function $f$ over a known compact domain $\mathcal{X}$ and an error parameter, $\epsilon > 0$, the goal is to find a point $\hat{x}$ satisfying $f (\hat{x}) \geq \max_x f (x) - \epsilon$ making as few oracle queries as possible. The paper characterizes the optimal oracle complexity of the problem in an instance dependent sense where the number of queries made to the oracle depend on the behavior of the particular function. Concretely, they show that the number of queries that necessarily must be made to the oracle is $\int_{\mathcal{X}} d \mathbf{x} / (\max(f) - f(\mathbf{x})  + \epsilon)^d$ up to constant factors depending (exponentially) on the dimension.