On the Convergence of Prior-Guided Zeroth-Order Optimization Algorithms

1 Introduction . Zeroth-order ( ZO ) optimization [ 18 ] provides powerful tools to deal with challenging tasks , such as query-based black-box adversarial attacks [ 7 , 13 ] , reinforcement learning [ 22 , 17 , 9 ] , meta-learning [ 1 ] , and hyperparameter tuning [ 23 ] , where the access to gradient information is either not available or too costly . ZO methods only assume an oracle access to the function value at any given point , instead of gradients as in first-order methods . The primary goal is to find a solution with as few queries to the function value oracle as possible . Recently , various ZO methods have been proposed in two main categories . One type is to obtain a gradient estimator and plug in some gradient-based methods . [ 21 ] analyzes the convergence of such methods with a random gradient-free ( RGF ) estimator obtained via finite difference along a random direction . The other type is directed search , where the update only depends on comparison between function values at different points . Such methods are robust against monotone transform of the objective function [ 24 , 10 , 3 ] . However , as they do not directly utilize function values , their query complexity is often higher than that of finite difference methods . Therefore , we focus on the first type of methods in this paper . Other methods exist such as CMA-ES [ 11 ] which is potentially better on objective functions with a rugged landscape , but lacks a general convergence guarantee . ZO methods are usually less efficient than first-order algorithms , as they typically take O ( d ) queries to reach a given precision , where d is the input dimension ( see Table 1 in [ 10 ] ) . Specifically , for the methods in [ 21 ] , the oracle query count is O ( d ) times larger than that of their corresponding schemes using gradients . This inefficiency stems from random search along uniformly distributed directions . To improve , various attempts have been made to augment random search with ( extra ) ∗J.Z . is the Corresponding Author . G.W . is now with School of Software , Shandong University . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . prior information . For instance , [ 14 , 19 ] use a time-dependent prior ( i.e. , the gradient estimated in the last iteration ) , while [ 16 , 8 , 4 ] use surrogate gradients obtained from other sources.2 Among these methods , [ 14 , 16 , 8 , 19 ] propose objective functions to describe the quality of a gradient estimator for justification or optimizing its hyperparameters ; for example , [ 19 ] uses a subspace estimator that maximizes its squared cosine similarity with the true gradient as the objective , and finds a better descent direction than both the prior direction and the randomly sampled direction . However , all these methods treat gradient estimation and the optimization algorithm separately , and it remains unclear whether are these gradient estimators and the corresponding prior-exploiting optimization methods theoretically sound ? and what role does the prior play in accelerating convergence ? In this paper , we attempt to answer these questions by establishing a formal connection between convergence rates and the quality of gradient estimates . Further , we develop a more efficient ZO algorithm with prior inspired by a theoretical analysis . First , we present a greedy descent framework of ZO methods and provide its convergence rate under smooth convex optimization , which is positively related to the squared cosine similarity between the gradient estimate and true gradient . As shown by [ 19 ] and our results , given some finite-difference queries , the optimal estimator maximizing the squared cosine similarity is the projection of true gradient on the subspace spanned by those queried directions . In the case with prior information , a natural such estimator is in the same form as the one in [ 19 ] . We call it PRGF estimator and analyze its convergence rate.3 Our results show that no matter what the prior is , the convergence rate of PRGF is at least the same as that of the RGF baseline [ 21 , 15 ] , and it could be significantly improved if given a useful prior.4 Such results shed light on exploring prior information in ZO optimization to accelerate convergence . Then , as a concrete example , we apply the analysis to History-PRGF [ 19 ] , which uses historical information ( i.e. , gradient estimate in the last iteration ) as the prior . [ 19 ] presents an analysis on linear functions , yet still lacking a convergence rate . We analyze on general L-smooth functions , and find that when the learning rate is smaller than the optimal value 1/L in smooth optimization , the expected squared cosine similarity could converge to a larger value , which compensates for the slowdown of convergence brought by the inappropriate choice of learning rate . We also show that History-PRGF admits a convergence rate independent of learning rate as long as it is in a fairly wide range . Finally , to further accelerate greedy descent methods , we present Prior-Guided ARS ( PARS ) , a new variant of Accelerated Random Search ( ARS ) [ 21 ] to utilize prior information . Technically , PARS is a non-trivial extension of ARS , as directly replacing the gradient estimator in ARS by the PRGF estimator would lose the convergence analysis . Thus , we present necessary extensions to ARS , and show that PARS has a convergence rate no worse than that of ARS and admits potential acceleration given a good prior . In particular , when the prior is chosen as the historical information , the resulting History-PARS is robust to learning rate in experiments . To our knowledge , History-PARS is the first ARS-based method that is empirically robust while retaining the convergence rate as ARS . Our experiments on numerical benchmarks and adversarial attacks confirm the theoretical results . 2 Setups . Assumptions on the problem class We consider unconstrained optimization , where the objective function f : Rd → R is convex and L-smooth for L ≥ 0 . Optionally , we require f to be τ -strongly convex for τ > 0 . We leave definitions of these concepts to Appendix A.1 . Directional derivative oracle In ZO optimization , we follow the finite difference approach , which makes more use of the queried function values than direct search and provides better gradient approximations than alternatives [ 2 ] . In particular , we consider the forward difference method : gµ ( v ; x ) : = f ( x+ µv ) − f ( x ) µ ≈ ∇f ( x ) > v , ( 1 ) 2Some work [ 7 , 14 , 25 ] restricts the random search to a more effective subspace reflecting the prior knowledge . But , this eliminates the possibility of convergence to the optimal solution . 3Note that the estimator is different from the P-RGF estimator in [ 8 ] . 4In this paper , RGF and PRGF could refer to either a gradient estimator or the greedy descent algorithm with the corresponding estimator , depending on the context . Algorithm 1 Greedy descent framework Input : L-smooth convex function f ; initialization x0 ; upper bound L̂ ( L̂ ≥ L ) ; iteration number T . Output : xT as the approximate minimizer of f . 1 : for t = 0 to T − 1 do 2 : Let vt be a random vector s.t . ‖vt‖ = 1 ; 3 : xt+1 ← xt − 1L̂gt , where gt ← ∇f ( xt ) > vt · vt ; 4 : end for 5 : return xT . where v is a vector with unit ` 2 norm ‖v‖ = 1 and µ is a small positive step . As long as the objective function is smooth , the error between the finite difference and the directional derivative could be uniformly bounded , as shown in the following proposition ( see Appendix A.2 for its proof ) . Proposition 1 . If f is L-smooth , then for any ( x , v ) with ‖v‖ = 1 , |gµ ( v ; x ) −∇f ( x ) > v| ≤ 12Lµ . Thus , in smooth optimization , the error brought by finite differences to the convergence bound can be analyzed in a principled way , and its impact tends to zero as µ→ 0 . We also choose µ as small as possible in practice . Hence , in the following analysis we directly assume the directional derivative oracle : suppose that we can obtain∇f ( x ) > v for any ( x , v ) in which ‖v‖ = 1 with one query . 3 Greedy descent framework and PRGF algorithm . We now introduce a greedy descent framework in ZO optimization which can be implemented with various gradient estimators . We first provide a general analysis , followed by a concrete example . 3.1 The greedy descent framework and general analysis . In first-order smooth convex optimization , a sufficient single-step decrease of the objective can guarantee convergence ( see Chapter 3.2 in [ 5 ] ) . Inspired by this fact , we design the update in an iteration to greedily seek for maximum decrease . Suppose we are currently at x , and want to update along the direction v. Without loss of generality , assume ‖v‖ = 1 and∇f ( x ) > v > 0 . To choose a suitable step size η that minimizes f ( x− ηv ) , we note that f ( x− ηv ) ≤ f ( x ) − η∇f ( x ) > v + 1 2 Lη2 : = F ( η ) ( 2 ) by smoothness of f . For the r.h.s , we have F ( η ) = f ( x ) − ( ∇f ( x ) > v ) 2 2L when η = ∇f ( x ) > v L , which minimizes F ( η ) . Thus , choosing such η could lead to a largest guaranteed decrease of f ( x−ηv ) from f ( x ) . In practice , the value of L is often unknown , but we can verify that as long as 0 < η ≤ ∇f ( x ) > v L , then f ( x− ηv ) ≤ F ( η ) < f ( x ) , i.e. , we can guarantee decrease of the objective ( regardless of the direction of v if∇f ( x ) > v > 0 ) . Based on the above discussion , we further allow v to be random and present the greedy descent framework in Algorithm 1 . Remark 1 . If vt ∼ U ( Sd−1 ) , i.e . vt is uniformly sampled from Sd−1 ( the unit sphere in Rd ) , then Algorithm 1 is similar to the simple random search in [ 21 ] except that vt is sampled from a Gaussian distribution there . Remark 2 . In general , vt could depend on the history ( i.e. , the randomness before sampling vt ) . For example , vt can be biased towards a vector pt that corresponds to prior information , and pt depends on the history since it depends on xt or the optimization trajectory . Theoretically speaking , vt is sampled from the conditional probability distribution Pr ( ·|Ft−1 ) where Ft−1 is a sub σ-algebra modelling the historical information . Ft−1 is important in our theoretical analysis since it tells how to perform conditional expectation given the history . We always require that Ft−1 includes all the randomness before iteration t to ensure that Lemma 1 ( and thus Theorems 1 and 2 ) and Theorem 5 hold . For further theoretical analysis of various implementations of the framework , Ft−1 remains to be specified by possibly also including some randomness in iteration t ( see e.g . Example 2 as an implementation of Algorithm 1 ) .5 5In mathematical language , if Ft−1 includes ( and only includes ) the randomness brought by random vectors { x1 , x2 , . . . , xn } , then Ft−1 is the σ-algebra generated by { x1 , x2 , . . . , xn } : Ft−1 is the smallest σ-algebra s.t . xi is Ft−1-measurable for all 1 ≤ i ≤ n. By Remark 2 , we introduce Et [ · ] : = E [ ·|Ft−1 ] to denote the conditional expectation given the history . In Algorithm 1 , under a suitable choice of Ft−1 , Et [ · ] roughly means only taking expectation w.r.t . vt. We let v : = v/‖v‖ denote the ` 2 normalization of vector v. We let x∗ denote one of the minimizers of f ( we assume such minimizer exists ) , and δt : = f ( xt ) − f ( x∗ ) . Thanks to the descent property in Algorithm 1 , we have the following lemma on single step progress . Lemma 1 ( Proof in Appendix B.1 ) . Let Ct : = ( ∇f ( xt ) > vt ) 2 and L′ : = L 1− ( 1−L L̂ ) 2 , then in Algorithm 1 , we have Et [ δt+1 ] ≤ δt − Et [ Ct ] 2L′ ‖∇f ( xt ) ‖2 . ( 3 ) We note that L′ = L̂/ ( 2−L/L̂ ) , so L̂2 ≤ L ′ ≤ L̂ and L′ ≥ L. To obtain a bound on E [ δT ] , one of the classical proofs ( see e.g. , Theorem 1 in [ 20 ] ) requires us to take expectation on both sides of Eq . ( 3 ) . Allowing the distribution of vt to be dependent on the history leads to additional technical difficulty : in the r.h.s of Eq . ( 3 ) , Et [ Ct ] becomes a random variable that is not independent of ‖∇f ( xt ) ‖2 . Thus , we can not simplify the term E [ Et [ Ct ] ‖∇f ( xt ) ‖2 ] if we take expectation.6 By using other techniques , we obtain the following main theorems on convergence rate . Theorem 1 ( Algorithm 1 , smooth and convex ; proof in Appendix B.1 ) . Let R : = maxx : f ( x ) ≤f ( x0 ) ‖x− x∗‖ and suppose R < ∞ . Then , in Algorithm 1 , we have E [ δT ] ≤ 2L′R2 ∑T−1 t=0 E [ 1 Et [ Ct ] ] T ( T + 1 ) . ( 4 ) Theorem 2 ( Algorithm 1 , smooth and strongly convex ; proof in Appendix B.1 ) . If f is also τ -strongly convex , then we have E δT exp ( − τL′ ∑T−1 t=0 Et [ Ct ] ) ≤ δ0 . ( 5 ) Remark 3 . In our results , the convergence rate depends on Et [ Ct ] in a more complicated way than only depending on E [ Ct ] = E [ Et [ Ct ] ] . For concrete cases , one may need to study the concentration properties of Et [ Ct ] besides its expectation , as in the proofs of Theorems 3 and 4 when we analyze History-PRGF , a special implementation in the greedy descent framework . In the strongly convex case , currently we can not directly obtain a final bound of E [ δT ] . However , the form of Eq . ( 5 ) is still useful since it roughly tells us that δT converges as the denominator exp ( − τL′ ∑T−1 t=0 Et [ Ct ] ) . For History-PRGF , we will turn this intuition into a rigorous theorem ( Theorem 4 ) since in that case we can prove that the denominator has a nice concentration property . Remark 4 . If we have a lower bound of Et [ Ct ] , e.g . Et [ Ct ] ≥ a > 0 , then we directly obtain that E [ δT ] ≤ 2L ′R2 a ( T+1 ) for Theorem 1 , and E [ δT ] ≤ δ0 exp ( − τ L′ aT ) for Theorem 2 . These could recover the ideal convergence rate for RGF estimator and the worst-case convergence rate for PRGF estimator , as explained in Examples 1 and 2 . From Theorems 1 and 2 , we see that a larger value of Ct would lead to a better bound . To find a good choice of vt in Algorithm 1 , it becomes natural to discuss the following problem . Suppose in an iteration in Algorithm 1 , we query the directional derivative oracle at xt along q directions { ui } qi=1 ( maybe randomly chosen ) and obtain the values of { ∇f ( xt ) > ui } qi=1 . We could use this information to construct a vector vt. What is the vt that maximizes Ct s.t . ‖vt‖ = 1 ? To answer this question , we give the following proposition based on Proposition 1 in [ 19 ] and additional justification . Proposition 2 ( Optimality of subspace estimator ; proof in Appendix B.2 ) . In one iteration of Algorithm 1 , if we have queried { ∇f ( xt ) > ui } qi=1 , then the optimal vt maximizing Ct s.t . ‖vt‖ = 1 should be in the following form : vt = ∇f ( xt ) A , where A : = span { u1 , u2 , . . . , uq } and ∇f ( xt ) A denotes the projection of∇f ( xt ) onto A . 6It is also the difficulty we faced in our very preliminary attempts of the theoretical analysis when f is not convex , and we leave its solution or workaround in the non-convex case as future work . Note that in Line 3 of Algorithm 1 , we have gt = ∇f ( xt ) > ∇f ( xt ) A · ∇f ( xt ) A = ∇f ( xt ) A . Therefore , the gradient estimator gt is equivalent to the projection of the gradient to the subspace A , which justifies its name of subspace estimator . Next we discuss some special cases of the subspace estimator . We leave detailed derivation in following examples to Appendix B.3 . Example 1 ( RGF ) . ui ∼ U ( Sd−1 ) for i = 1 , 2 , . . . , q . Without loss of generality , we assume they are orthonormal ( e.g. , via Gram-Schmidt orthogonalization ) .7 The corresponding estimator gt = ∑q i=1∇f ( xt ) > ui · ui ( vt = gt ) . When q = 1 , the estimator is similar to the random gradientfree oracle in [ 21 ] . With q ≥ 1 , it is essentially the same as the stochastic subspace estimator with columns from Haar-distributed random orthogonal matrix [ 15 ] and similar to the orthogonal ES estimator [ 9 ] . In theoretical analysis , we let Ft−1 only include the randomness before iteration t , and then we can prove that Et [ Ct ] = qd . By Theorems 1 and 2 , the convergence rate is E [ δT ] ≤ 2L′R2 dq T+1 for smooth convex case , and E [ δT ] ≤ δ0 exp ( − τL′ q dT ) for smooth and strongly convex case . The bound is the same as that in [ 15 ] . Since the query complexity in each iteration is proportional to q , the bound for total query complexity is indeed independent of q . Example 2 ( PRGF ) . With slight notation abuse , we assume the subspace in Proposition 2 is spanned by { p1 , . . . , pk , u1 , . . . , uq } , so each iteration takes q + k queries . Let p1 , · · · , pk be k non-zero vectors corresponding to the prior message ( e.g . the historical update , or the gradient of a surrogate model ) , and ui ∼ U ( Sd−1 ) for i = 1 , 2 , . . . , q . We note that intuitively we can not construct a better subspace since the only extra information we know is the k priors . In our analysis , we assume k = 1 for convenience , and we change the original notation p1 to pt to explicitly show the dependence of pt on the history . We note that pt could also depend on extra randomness in iteration t ( see e.g . the specification of pt in Appendix D.1.1 ) . For convenience of theoretical analysis , we require that pt is determined before sampling { u1 , u2 , . . . , uq } , and let Ft−1 also include the extra randomness of pt in iteration t ( not including the randomness of { u1 , u2 , . . . , uq } ) besides the randomness before iteration t. Then pt is always Ft−1-measurable , i.e . determined by the history . Without loss of generality , we assume { pt , u1 , . . . , uq } are orthonormal ( e.g. , via Gram-Schmidt orthogonalization ) . The corresponding estimator gt = ∇f ( xt ) > pt · pt + ∑q i=1∇f ( xt ) > ui · ui ( vt = gt ) , which is similar to the estimator in [ 19 ] . By [ 19 ] ( the expected drift of X2t in its Theorem 1 ) , we have Lemma 2 ( Proof in Appendix B.3.4 ) . In Algorithm 1 with PRGF estimator , Et [ Ct ] = Dt + qd−1 ( 1− Dt ) where Dt : = ( ∇f ( xt ) > pt ) 2 . Hence Et [ Ct ] ≥ qd holds . By Remark 4 , PRGF admits a guaranteed convergence rate of RGF and is potentially better given a good prior ( if Dt is large ) , but it costs an additional query per iteration . This shows soundness of the PRGF algorithm . For further theoretical analysis , we need to bound Dt . This could be done when using the historical prior introduced in Section 3.2 ( see Lemma 3 ) . Bounding Dt is usually challenging when a general prior is adopted , but if the prior is an approximate gradient ( such case appears in [ 16 ] ) , it may be possible . We leave related investigation as future work .

Summary: This paper studies the performance of zeroth order optimization algorithms where gradient estimation uses prior information. They provide a bound on the convergence analysis for such prior guided algorithms in terms of expected alignment of the gradient estimates to the true gradient. They use this framework to study historical priors and show that such a prior may counter the effects of using a sub-optimal learning rate. They also provide a new accelerated random search algorithm that incorporates prior information.