On the Convergence of Prior-Guided Zeroth-Order Optimization Algorithms

1 Introduction . Zeroth-order ( ZO ) optimization [ 18 ] provides powerful tools to deal with challenging tasks , such as query-based black-box adversarial attacks [ 7 , 13 ] , reinforcement learning [ 22 , 17 , 9 ] , meta-learning [ 1 ] , and hyperparameter tuning [ 23 ] , where the access to gradient information is either not available or too costly . ZO methods only assume an oracle access to the function value at any given point , instead of gradients as in first-order methods . The primary goal is to find a solution with as few queries to the function value oracle as possible . Recently , various ZO methods have been proposed in two main categories . One type is to obtain a gradient estimator and plug in some gradient-based methods . [ 21 ] analyzes the convergence of such methods with a random gradient-free ( RGF ) estimator obtained via finite difference along a random direction . The other type is directed search , where the update only depends on comparison between function values at different points . Such methods are robust against monotone transform of the objective function [ 24 , 10 , 3 ] . However , as they do not directly utilize function values , their query complexity is often higher than that of finite difference methods . Therefore , we focus on the first type of methods in this paper . Other methods exist such as CMA-ES [ 11 ] which is potentially better on objective functions with a rugged landscape , but lacks a general convergence guarantee . ZO methods are usually less efficient than first-order algorithms , as they typically take O ( d ) queries to reach a given precision , where d is the input dimension ( see Table 1 in [ 10 ] ) . Specifically , for the methods in [ 21 ] , the oracle query count is O ( d ) times larger than that of their corresponding schemes using gradients . This inefficiency stems from random search along uniformly distributed directions . To improve , various attempts have been made to augment random search with ( extra ) ∗J.Z . is the Corresponding Author . G.W . is now with School of Software , Shandong University . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . prior information . For instance , [ 14 , 19 ] use a time-dependent prior ( i.e. , the gradient estimated in the last iteration ) , while [ 16 , 8 , 4 ] use surrogate gradients obtained from other sources.2 Among these methods , [ 14 , 16 , 8 , 19 ] propose objective functions to describe the quality of a gradient estimator for justification or optimizing its hyperparameters ; for example , [ 19 ] uses a subspace estimator that maximizes its squared cosine similarity with the true gradient as the objective , and finds a better descent direction than both the prior direction and the randomly sampled direction . However , all these methods treat gradient estimation and the optimization algorithm separately , and it remains unclear whether are these gradient estimators and the corresponding prior-exploiting optimization methods theoretically sound ? and what role does the prior play in accelerating convergence ? In this paper , we attempt to answer these questions by establishing a formal connection between convergence rates and the quality of gradient estimates . Further , we develop a more efficient ZO algorithm with prior inspired by a theoretical analysis . First , we present a greedy descent framework of ZO methods and provide its convergence rate under smooth convex optimization , which is positively related to the squared cosine similarity between the gradient estimate and true gradient . As shown by [ 19 ] and our results , given some finite-difference queries , the optimal estimator maximizing the squared cosine similarity is the projection of true gradient on the subspace spanned by those queried directions . In the case with prior information , a natural such estimator is in the same form as the one in [ 19 ] . We call it PRGF estimator and analyze its convergence rate.3 Our results show that no matter what the prior is , the convergence rate of PRGF is at least the same as that of the RGF baseline [ 21 , 15 ] , and it could be significantly improved if given a useful prior.4 Such results shed light on exploring prior information in ZO optimization to accelerate convergence . Then , as a concrete example , we apply the analysis to History-PRGF [ 19 ] , which uses historical information ( i.e. , gradient estimate in the last iteration ) as the prior . [ 19 ] presents an analysis on linear functions , yet still lacking a convergence rate . We analyze on general L-smooth functions , and find that when the learning rate is smaller than the optimal value 1/L in smooth optimization , the expected squared cosine similarity could converge to a larger value , which compensates for the slowdown of convergence brought by the inappropriate choice of learning rate . We also show that History-PRGF admits a convergence rate independent of learning rate as long as it is in a fairly wide range . Finally , to further accelerate greedy descent methods , we present Prior-Guided ARS ( PARS ) , a new variant of Accelerated Random Search ( ARS ) [ 21 ] to utilize prior information . Technically , PARS is a non-trivial extension of ARS , as directly replacing the gradient estimator in ARS by the PRGF estimator would lose the convergence analysis . Thus , we present necessary extensions to ARS , and show that PARS has a convergence rate no worse than that of ARS and admits potential acceleration given a good prior . In particular , when the prior is chosen as the historical information , the resulting History-PARS is robust to learning rate in experiments . To our knowledge , History-PARS is the first ARS-based method that is empirically robust while retaining the convergence rate as ARS . Our experiments on numerical benchmarks and adversarial attacks confirm the theoretical results . 2 Setups . Assumptions on the problem class We consider unconstrained optimization , where the objective function f : Rd → R is convex and L-smooth for L ≥ 0 . Optionally , we require f to be τ -strongly convex for τ > 0 . We leave definitions of these concepts to Appendix A.1 . Directional derivative oracle In ZO optimization , we follow the finite difference approach , which makes more use of the queried function values than direct search and provides better gradient approximations than alternatives [ 2 ] . In particular , we consider the forward difference method : gµ ( v ; x ) : = f ( x+ µv ) − f ( x ) µ ≈ ∇f ( x ) > v , ( 1 ) 2Some work [ 7 , 14 , 25 ] restricts the random search to a more effective subspace reflecting the prior knowledge . But , this eliminates the possibility of convergence to the optimal solution . 3Note that the estimator is different from the P-RGF estimator in [ 8 ] . 4In this paper , RGF and PRGF could refer to either a gradient estimator or the greedy descent algorithm with the corresponding estimator , depending on the context . Algorithm 1 Greedy descent framework Input : L-smooth convex function f ; initialization x0 ; upper bound L̂ ( L̂ ≥ L ) ; iteration number T . Output : xT as the approximate minimizer of f . 1 : for t = 0 to T − 1 do 2 : Let vt be a random vector s.t . ‖vt‖ = 1 ; 3 : xt+1 ← xt − 1L̂gt , where gt ← ∇f ( xt ) > vt · vt ; 4 : end for 5 : return xT . where v is a vector with unit ` 2 norm ‖v‖ = 1 and µ is a small positive step . As long as the objective function is smooth , the error between the finite difference and the directional derivative could be uniformly bounded , as shown in the following proposition ( see Appendix A.2 for its proof ) . Proposition 1 . If f is L-smooth , then for any ( x , v ) with ‖v‖ = 1 , |gµ ( v ; x ) −∇f ( x ) > v| ≤ 12Lµ . Thus , in smooth optimization , the error brought by finite differences to the convergence bound can be analyzed in a principled way , and its impact tends to zero as µ→ 0 . We also choose µ as small as possible in practice . Hence , in the following analysis we directly assume the directional derivative oracle : suppose that we can obtain∇f ( x ) > v for any ( x , v ) in which ‖v‖ = 1 with one query . 3 Greedy descent framework and PRGF algorithm . We now introduce a greedy descent framework in ZO optimization which can be implemented with various gradient estimators . We first provide a general analysis , followed by a concrete example . 3.1 The greedy descent framework and general analysis . In first-order smooth convex optimization , a sufficient single-step decrease of the objective can guarantee convergence ( see Chapter 3.2 in [ 5 ] ) . Inspired by this fact , we design the update in an iteration to greedily seek for maximum decrease . Suppose we are currently at x , and want to update along the direction v. Without loss of generality , assume ‖v‖ = 1 and∇f ( x ) > v > 0 . To choose a suitable step size η that minimizes f ( x− ηv ) , we note that f ( x− ηv ) ≤ f ( x ) − η∇f ( x ) > v + 1 2 Lη2 : = F ( η ) ( 2 ) by smoothness of f . For the r.h.s , we have F ( η ) = f ( x ) − ( ∇f ( x ) > v ) 2 2L when η = ∇f ( x ) > v L , which minimizes F ( η ) . Thus , choosing such η could lead to a largest guaranteed decrease of f ( x−ηv ) from f ( x ) . In practice , the value of L is often unknown , but we can verify that as long as 0 < η ≤ ∇f ( x ) > v L , then f ( x− ηv ) ≤ F ( η ) < f ( x ) , i.e. , we can guarantee decrease of the objective ( regardless of the direction of v if∇f ( x ) > v > 0 ) . Based on the above discussion , we further allow v to be random and present the greedy descent framework in Algorithm 1 . Remark 1 . If vt ∼ U ( Sd−1 ) , i.e . vt is uniformly sampled from Sd−1 ( the unit sphere in Rd ) , then Algorithm 1 is similar to the simple random search in [ 21 ] except that vt is sampled from a Gaussian distribution there . Remark 2 . In general , vt could depend on the history ( i.e. , the randomness before sampling vt ) . For example , vt can be biased towards a vector pt that corresponds to prior information , and pt depends on the history since it depends on xt or the optimization trajectory . Theoretically speaking , vt is sampled from the conditional probability distribution Pr ( ·|Ft−1 ) where Ft−1 is a sub σ-algebra modelling the historical information . Ft−1 is important in our theoretical analysis since it tells how to perform conditional expectation given the history . We always require that Ft−1 includes all the randomness before iteration t to ensure that Lemma 1 ( and thus Theorems 1 and 2 ) and Theorem 5 hold . For further theoretical analysis of various implementations of the framework , Ft−1 remains to be specified by possibly also including some randomness in iteration t ( see e.g . Example 2 as an implementation of Algorithm 1 ) .5 5In mathematical language , if Ft−1 includes ( and only includes ) the randomness brought by random vectors { x1 , x2 , . . . , xn } , then Ft−1 is the σ-algebra generated by { x1 , x2 , . . . , xn } : Ft−1 is the smallest σ-algebra s.t . xi is Ft−1-measurable for all 1 ≤ i ≤ n. By Remark 2 , we introduce Et [ · ] : = E [ ·|Ft−1 ] to denote the conditional expectation given the history . In Algorithm 1 , under a suitable choice of Ft−1 , Et [ · ] roughly means only taking expectation w.r.t . vt. We let v : = v/‖v‖ denote the ` 2 normalization of vector v. We let x∗ denote one of the minimizers of f ( we assume such minimizer exists ) , and δt : = f ( xt ) − f ( x∗ ) . Thanks to the descent property in Algorithm 1 , we have the following lemma on single step progress . Lemma 1 ( Proof in Appendix B.1 ) . Let Ct : = ( ∇f ( xt ) > vt ) 2 and L′ : = L 1− ( 1−L L̂ ) 2 , then in Algorithm 1 , we have Et [ δt+1 ] ≤ δt − Et [ Ct ] 2L′ ‖∇f ( xt ) ‖2 . ( 3 ) We note that L′ = L̂/ ( 2−L/L̂ ) , so L̂2 ≤ L ′ ≤ L̂ and L′ ≥ L. To obtain a bound on E [ δT ] , one of the classical proofs ( see e.g. , Theorem 1 in [ 20 ] ) requires us to take expectation on both sides of Eq . ( 3 ) . Allowing the distribution of vt to be dependent on the history leads to additional technical difficulty : in the r.h.s of Eq . ( 3 ) , Et [ Ct ] becomes a random variable that is not independent of ‖∇f ( xt ) ‖2 . Thus , we can not simplify the term E [ Et [ Ct ] ‖∇f ( xt ) ‖2 ] if we take expectation.6 By using other techniques , we obtain the following main theorems on convergence rate . Theorem 1 ( Algorithm 1 , smooth and convex ; proof in Appendix B.1 ) . Let R : = maxx : f ( x ) ≤f ( x0 ) ‖x− x∗‖ and suppose R < ∞ . Then , in Algorithm 1 , we have E [ δT ] ≤ 2L′R2 ∑T−1 t=0 E [ 1 Et [ Ct ] ] T ( T + 1 ) . ( 4 ) Theorem 2 ( Algorithm 1 , smooth and strongly convex ; proof in Appendix B.1 ) . If f is also τ -strongly convex , then we have E δT exp ( − τL′ ∑T−1 t=0 Et [ Ct ] ) ≤ δ0 . ( 5 ) Remark 3 . In our results , the convergence rate depends on Et [ Ct ] in a more complicated way than only depending on E [ Ct ] = E [ Et [ Ct ] ] . For concrete cases , one may need to study the concentration properties of Et [ Ct ] besides its expectation , as in the proofs of Theorems 3 and 4 when we analyze History-PRGF , a special implementation in the greedy descent framework . In the strongly convex case , currently we can not directly obtain a final bound of E [ δT ] . However , the form of Eq . ( 5 ) is still useful since it roughly tells us that δT converges as the denominator exp ( − τL′ ∑T−1 t=0 Et [ Ct ] ) . For History-PRGF , we will turn this intuition into a rigorous theorem ( Theorem 4 ) since in that case we can prove that the denominator has a nice concentration property . Remark 4 . If we have a lower bound of Et [ Ct ] , e.g . Et [ Ct ] ≥ a > 0 , then we directly obtain that E [ δT ] ≤ 2L ′R2 a ( T+1 ) for Theorem 1 , and E [ δT ] ≤ δ0 exp ( − τ L′ aT ) for Theorem 2 . These could recover the ideal convergence rate for RGF estimator and the worst-case convergence rate for PRGF estimator , as explained in Examples 1 and 2 . From Theorems 1 and 2 , we see that a larger value of Ct would lead to a better bound . To find a good choice of vt in Algorithm 1 , it becomes natural to discuss the following problem . Suppose in an iteration in Algorithm 1 , we query the directional derivative oracle at xt along q directions { ui } qi=1 ( maybe randomly chosen ) and obtain the values of { ∇f ( xt ) > ui } qi=1 . We could use this information to construct a vector vt. What is the vt that maximizes Ct s.t . ‖vt‖ = 1 ? To answer this question , we give the following proposition based on Proposition 1 in [ 19 ] and additional justification . Proposition 2 ( Optimality of subspace estimator ; proof in Appendix B.2 ) . In one iteration of Algorithm 1 , if we have queried { ∇f ( xt ) > ui } qi=1 , then the optimal vt maximizing Ct s.t . ‖vt‖ = 1 should be in the following form : vt = ∇f ( xt ) A , where A : = span { u1 , u2 , . . . , uq } and ∇f ( xt ) A denotes the projection of∇f ( xt ) onto A . 6It is also the difficulty we faced in our very preliminary attempts of the theoretical analysis when f is not convex , and we leave its solution or workaround in the non-convex case as future work . Note that in Line 3 of Algorithm 1 , we have gt = ∇f ( xt ) > ∇f ( xt ) A · ∇f ( xt ) A = ∇f ( xt ) A . Therefore , the gradient estimator gt is equivalent to the projection of the gradient to the subspace A , which justifies its name of subspace estimator . Next we discuss some special cases of the subspace estimator . We leave detailed derivation in following examples to Appendix B.3 . Example 1 ( RGF ) . ui ∼ U ( Sd−1 ) for i = 1 , 2 , . . . , q . Without loss of generality , we assume they are orthonormal ( e.g. , via Gram-Schmidt orthogonalization ) .7 The corresponding estimator gt = ∑q i=1∇f ( xt ) > ui · ui ( vt = gt ) . When q = 1 , the estimator is similar to the random gradientfree oracle in [ 21 ] . With q ≥ 1 , it is essentially the same as the stochastic subspace estimator with columns from Haar-distributed random orthogonal matrix [ 15 ] and similar to the orthogonal ES estimator [ 9 ] . In theoretical analysis , we let Ft−1 only include the randomness before iteration t , and then we can prove that Et [ Ct ] = qd . By Theorems 1 and 2 , the convergence rate is E [ δT ] ≤ 2L′R2 dq T+1 for smooth convex case , and E [ δT ] ≤ δ0 exp ( − τL′ q dT ) for smooth and strongly convex case . The bound is the same as that in [ 15 ] . Since the query complexity in each iteration is proportional to q , the bound for total query complexity is indeed independent of q . Example 2 ( PRGF ) . With slight notation abuse , we assume the subspace in Proposition 2 is spanned by { p1 , . . . , pk , u1 , . . . , uq } , so each iteration takes q + k queries . Let p1 , · · · , pk be k non-zero vectors corresponding to the prior message ( e.g . the historical update , or the gradient of a surrogate model ) , and ui ∼ U ( Sd−1 ) for i = 1 , 2 , . . . , q . We note that intuitively we can not construct a better subspace since the only extra information we know is the k priors . In our analysis , we assume k = 1 for convenience , and we change the original notation p1 to pt to explicitly show the dependence of pt on the history . We note that pt could also depend on extra randomness in iteration t ( see e.g . the specification of pt in Appendix D.1.1 ) . For convenience of theoretical analysis , we require that pt is determined before sampling { u1 , u2 , . . . , uq } , and let Ft−1 also include the extra randomness of pt in iteration t ( not including the randomness of { u1 , u2 , . . . , uq } ) besides the randomness before iteration t. Then pt is always Ft−1-measurable , i.e . determined by the history . Without loss of generality , we assume { pt , u1 , . . . , uq } are orthonormal ( e.g. , via Gram-Schmidt orthogonalization ) . The corresponding estimator gt = ∇f ( xt ) > pt · pt + ∑q i=1∇f ( xt ) > ui · ui ( vt = gt ) , which is similar to the estimator in [ 19 ] . By [ 19 ] ( the expected drift of X2t in its Theorem 1 ) , we have Lemma 2 ( Proof in Appendix B.3.4 ) . In Algorithm 1 with PRGF estimator , Et [ Ct ] = Dt + qd−1 ( 1− Dt ) where Dt : = ( ∇f ( xt ) > pt ) 2 . Hence Et [ Ct ] ≥ qd holds . By Remark 4 , PRGF admits a guaranteed convergence rate of RGF and is potentially better given a good prior ( if Dt is large ) , but it costs an additional query per iteration . This shows soundness of the PRGF algorithm . For further theoretical analysis , we need to bound Dt . This could be done when using the historical prior introduced in Section 3.2 ( see Lemma 3 ) . Bounding Dt is usually challenging when a general prior is adopted , but if the prior is an approximate gradient ( such case appears in [ 16 ] ) , it may be possible . We leave related investigation as future work .

Summary: The paper provides a convergence analysis of a class of prior-guided zeroth-order (ZO) optimization algorithms and gives a prior-guided variant of the accelerated random search (ARS) algorithm.  Contribution: 1. Provides a complete convergence on some prior-guided zeroth-order optimization algorithms. 2. Provides a prior-guided variant of the ARS algorithm.