On the Convergence of Prior-Guided Zeroth-Order Optimization Algorithms
Authors: Shuyu Cheng
Guoqiang Wu
Jun Zhu
Venue: NeurIPS 2021 Poster (2021)
Keywords: zeroth-order optimization
gradient-free optimization
evolutionary strategies
convex optimization
randomized methods

Abstract: Zeroth-order (ZO) optimization is widely used to handle challenging tasks, such as query-based black-box adversarial attacks and reinforcement learning. Various attempts have been made to integrate prior information into the gradient estimation procedure based on finite differences, with promising empirical results. However, their convergence properties are not well understood. This paper makes an attempt to fill up this gap by analyzing the convergence of prior-guided ZO algorithms under a greedy descent framework with various gradient estimators. We provide a convergence guarantee for the prior-guided random gradient-free (PRGF) algorithms. Moreover, to further accelerate over greedy descent methods, we present a new accelerated random search (ARS) algorithm that incorporates prior information, together with a convergence analysis. Finally, our theoretical results are confirmed by experiments on several numerical benchmarks as well as adversarial attacks.

On the Convergence of Prior-Guided Zeroth-Order Optimization Algorithms
Shuyu Cheng, Guoqiang Wu, Jun Zhu"
Dept. of Comp. Sci. and Tech., BNRist Center, State Key Lab for Intell. Tech. & Sys., Institute for AI, Tsinghua-Bond Bosch Joint Center for ML, Tsinghua University. Beijing, 100084, China
Pazhou Lab, Guangzhou, 510330, China
chengay188nal tastinggaaaa.tttttaaaaaa.a.a edu.cn, guogtangausosgmal1.. .com dcszj@tainghua. edu.cn
Abstract
Zeroth-order (ZO) optimization is widely used to handle challenging tasks, such as query-based black-box adversarial attacks and reinforcement learning. Various attempts have been made to integrate prior information into the gradient estimation procedure based on finite differences, with promising empirical results. However, their convergence properties are well understood. This paper makes an attempt to fill up this gap by analyzing the convergence of prior-guided ZO algorithms under a greedy descent framework with various gradient estimators. We provide a convergence guarantee for the prior- prior-guided random gradient- free (PRGF) algo- rithms. Moreover, further accelerate over greedy descent methods, we present a new accelerated random search (ARS) algorithm that incorporates prior infor- mation, together with a convergence analysis. Finally, our theoretical results are confirmed by experiments on several numerical benchmarks as well as adversarial attacks. Our code available at htttts:///gtt //tpt/gggttttbtt cww/ ccwmmesy50011//g-222222
Introduction
Zeroth-order (ZO) optimization [88 provides powerful tools deal with challenging tasks, such as query-based black-box adversarial attacks 7 [3]. reinforcement learning [22] I7 9 meta-learning
[1]. and hyperparameter tuning (23 where the access to gradient information either not available or too costly. ZO methods only assume an oracle access to the function value at any given point, instead of gradients in first-order methods. The primary goal is find solution with as few queries the function value oracle as possible. Recently, various ZO methods have been proposed in two main categories. One type obtain a gradient estimator and plug some gradient- based methods. [2] analyzes the convergence of such methods with random gradient-free free (RGF) estimator obtained via finite difference along random direction. The other type directed search, where the update only depends on comparison between function values at different points. Such methods are robust against monotone transform of the objective function 24 10 3]. However, as they do directly utilize function values, their query complexity is often higher than that of finite difference methods. Therefore, we focus on the first type of methods this paper. Other methods exist such as CMA-ES | which is potentially better on objective functions with rugged landscape, but lacks general convergence guarantee.
ZO methods are usually less efficient than first-order algorithms. as they typically take O(d) queries reach a given precision, where d is the input dimension (see Table (1O).. Specifically, for
the methods in (21). the oracle query count is ((d) times larger than that of their corresponding schemes using gradients. This inefficiency stems from random search along uniformly distributed directions. To improve, various attempts have been made to augment random search with (extra)
'J.Z. is the Corresponding Author. W. is now with School of Software, Shandong University.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

19 use time- deneeddenddent prior (i.e., the gradient estimated in the 8 use surrogate gradients obtained from other sources Among these g I9 propose objective functions to describe the quality of a gradient estimator for justification or optimizing its hyperparameters; for example, [19] uses a subspace estimator that maximizes its squared cosine similarity with the true gradient as the objective, and finds better descent direction than both the prior direction and the randomly sampled direction. However, all these methods treat gradient estimation and the optimization algorithm separately, and it remains unclear whether are these gradient estimators and the corresponding prior-exploiting optimization methods theoretically sound? and what role does the prior play in accelerating convergence?
prior information. For instance,
14
last iteration), while T6
methods, [1 16
In this paper. we attempt to answer these questions by establishing a formal connection between convergence rates and the quality of gradient estimates. Further, we develop a more efficient ZO algorithm with prior inspired by a theoretical analysis. First, we present a greedy descent framework of 70 methods and provide its convergence rate under smooth convex optimization, which is positively related to the squared cosine similarity between the gradient estimate and true gradient. As shown by [1] and our results, given some finite-difference queries, the optimal estimator maximizing the squared cosine similarity is the projection of true gradient on the subspace spanned by those queried directions. In the case with prior information, natural such estimator is in the same form as the one in [9]. We call it PRGF estimator and analyze its convergence rate 3] Our results show that no matter what the prior is, the convergence rate of PRGF is at least the same that of the RGF baseline [5], and it could be significantly improved given a useful prior Such results shed light on exploring prior information in ZO optimization to accelerate convergence.
Then, as a concrete example, we apply the analysis to History-PRGr 19] which uses historical information (i.e., gradient estimate in the last iteration) as the prior. 19 presents an analysis on linear functions, yet still lacking a convergence rate. We analyze on general L-smooth functions, and find that when the learning rate is smaller than the optimal value I/L in smooth optimization, the expected squared cosine similarity could converge to a larger value, which compensates for the slowdown of convergence brought by the inappropriate choice of learning rate. We also show that History-PRGE admits a convergence rate independent of learning rate as long as it is in a fairly wide range.
Finally, to further accelerate greedy descent methods, we present Prior-Guide ARS (PARS), a new variant of Accelerated Random Search (ARS) (21 to utilize prior information. Technically, PARS is a non-trivial extension of ARS, as directly replacing the gradient estimator in ARS by the PRGF estimator would lose the convergence analysis. Thus, we present necessary extensions to ARS, and show that PARS has convergence rate no worse than that of ARS and admits potential acceleration given a good prior. In particular, when the prior is chosen as the historical information, the resulting Eestory-PARS is robust to learning rate in experiments. To our knowledge, History-PARS is the first ARS-based method that is empirically robust while retaining the convergence rate as ARS. Our experiments numerical benchmarks and adversarial attacks confirm the theoretical results.
2 Setups
Assumptions on the problem class We consider unconstrained optimization, where the objective function f Edd - IR is convex and L-smooth for L > 0. Optionally, require f to be r-strongly convex for r > 0. We leave definitions of these concepts to Appendix A.1
Directional derivative oracle In ZO optimization, we follow the finite difference approach, which makes more use of the queried function values than direct search and provides better gradient approximations than alternatives (2) In particular, we consider the forward difference method:
f(x + nv) f(x) VV(
gu (x
1
restricts the random search to a more effective subspace reflecting the prior knowledge. But, this eliminates the possibility of convergence to the optimal solution.
"Some work 74 25
3 Note that the estimator is different from the P-RGF estimator (8]
'In this paper, RGF and PRGF could refer to either a gradient estimator or the greedy descent algorithm with the corresponding estimator, depending the context.

---

Algorithm Greedy descent framework
Input: L-smooth convex function f: initialization zg; upper bound å (á 2 L); iteration number T. Output: 2T as the approximate minimizer of f-
1: for t- 0 to T do
2: Let U2 be random vector s.t. || 1; T, where Dr. & V (((z)) T V23
3: Tt+1
4: end for
5: return 2T.
where is vector with unit & norm ||a|| 1 and y is small positive step. As long as the objective function is smooth, the error between the finite difference and the directional derivative could be uniformly bounded, as shown in the following proposition (see Appendix AA. for its proof).
Proposition If f is L-smooth, then for any (z, v) with ||a||| = 1, 19, (v; v.(() T v| < 1 Lp.
Thus, in smooth optimization, the error brought by finite differences to the convergence bound can be analyzed in principled way, and its impact tends to zero as μ - 0. We also choose μ small possi- ble in practice. Hence, in the following analysis we directly assume the directional derivative oracle: T v for any (x, v) in which |||||| with one query.
suppose that can obtain (x)
3 Greedy descent framework and PRGF algorithm
We now introduce a greedy descent framework in ZO optimization which can be implemented with various gradient estimators. We first provide general analysis, followed by concrete example.
3.1 The greedy descent framework and general analysis
In first-order smooth convex optimization, a sufficient single-step decrease of the objective can guarantee convergence (see Chapter 3.2 in (3]). Inspired by this fact, we design the update in an iteration to greedily seek for maximum decrease. Suppose we are currently at æ, and want to update along the direction v. Without loss of generality, assume ||u|| and (f(z) v > 0. To choose suitable step size 17 that minimizes f(z 17)), we note that
f(x - (w) S f(x) (xx) + 2
=np²==F(mm
(2)
by smoothness of f. For the r.h.s, we have F(1) = f(x) ) [V)()) when Yf(z) which 2L
minimizes F(n). Thus, choosing such 1) could lead a largest guaranteed decrease of f(x n/e) from
f(x). In practice, the value of I. is often unknown, but we verify that long as 0 < イ < vf(x) then f(x - ryo) & F(n) < [xx) i.e., we can guarantee decrease of the objective (regardless of the direction of v (z) v > 0). Based on the above discussion, we further allow v to be random and present the greedy descent framework in Algorithm []
If U ~ (s==t))), i.e. vt uniformly sampled from gd-1 (the unit sphere in Rd),, then Algorithm is similar to the simple random search in (21) except that vt is sampled from a Gaussian distribution there.
Remark 1.
Remark 2. In general, in could depend on the history (i.e. the randomness before sampling ve). For example, Vt can be biased towards vector Pt that corresponds to prior information, and pt depends on the history since it depends on or the optimization trajectory.
Theoretically speaking, Vt is sampled from the conditional probability distribution Pr(-IF--1) where Ft-1 is a sub o-algebra modelling the historical information. F.-1 is important in our theoretical analysis since it tells how perform conditional expectation given the history.
We always require that Fi-1 includes all the randomness before iteration to ensure that Lemmal ! (and thus Theorems 7 and 2 and Theorem S hold. For further theoretical analysis of various implementations of the framework, F.-1 remains to be specified by possibly also including some randomness in iteration (see e.g. Example 2 as an implementation of Algorithm [
In mathematical language, F includes (and only includes) the randomness brought by random vectors (or, Pa,..., ... }. then F. is the alagebrra generated by {I1, Z2...., In J: Ft-1 1 is the smallest o-algebra s.t. 2; is Fi-1-measurabll measurable for all <
3

---

By Remark we introduce : to denote the conditional expectation given the history. In Algorithm| under suitable choice of F 1. E.[:] roughly means only taking expectation w.r.t 2t- We let e "illl denote the /2 normalization of vector We let x² denote one of the minimizers of (we assume such minimizer exists), and 6, == f(zz - f(x"). Thanks to the descent property in we have the following lemma on single step progress.
Algorithm 1
Let C2 == (f(x))) v.) and L' == T-(1- p)2 then in
Lemma 1 (Proof in Appendix B.1
Algorithm L
we have
Cr 2T'
Ec S
(3)
We note that I' = 1/22-/11 so 2 S 1' L and L' 2 L.
To obtain bound on E. one of the classical proofs (see e.g., Theorem 1 in 20 requires us to take expectation on both sides of Eq. ( Allowing the distribution of o, be dependent on the history leads to additional technical difficulty: in the r.h.s of Eq. (3). IE, [C.] becomes random variable that not independent of ||V [xx)]] Thus, we cannot simplify the term [ V expectation O By using other techniques, we obtain the following main theorems convergence rate.
[I2] if we take
Theorem 1 (Algorithm [
smooth and convex; proof in Appendix B.I.
Let R ==
((0 | x=1 and suppose R < Co. Then, in Algorithm have
RT E E
C.C.
E E
(4)
T(T +1)
Theorem 2 (Algorithm] smooth and strongly convex; proof in Appendix B.I. If is also T-strongly convex, then we have
ōr Σl E. (Ca))
E
S 60.
(5)
exp
Remark 3. In our results, the convergence rate depends on [c[ in more complicated way than only depending on [E[Ca] E [ For concrete cases, one may need to study the concentration
properties of [e[Ca]] besides its expectation, as in the proofs of Theorems 3 and when we analyze History-PRCC a special implementation in the greedy descent framework.
In the strongly convex case, currently we cannot directly obtain a final bound of E[8r]. However, the form of Eq. (5) is still useful since it roughly tells us that ôT converges as the denominator exp T. ET- E.C.. For History- PRGF, we will turn this intuition into a rigorous theorem (Theorem 4 since that case we can prove that the denominator has a nice concentration property. Remark 4. If we have lower bound of E,[C,], e.g. E, [,[C,]
0, then we directly obtain that [[ a(T+1] for Theorem! 1 and E[ör] S 80 exp(-FaTT) for Theorem 2 These could recover the ideal convergence rate for RGF estimator and the worst-case convergence rate for PRGF estimator, as explained in Examples and
R1
From Theorems 1 and 2 we see that a larger value of C. would lead to a better bound. To find a good choice of U/ in Algorithm it becomes natural to discuss the following problem. Suppose in an iteration in Algorithm we query the directional derivative oracle at Fg along q directions (u, ) (maybe randomly chosen) and obtain the values (V (x,) "u 1 We could use this information construct vector ve. What is the v, that maximizes C. s.t. [|ve]] 1? To answer this question, give the following proposition based on Proposition in [19 and additional justification.
Proposition (Optimality of subspace estimator; proof in Appendix B.2). In one iteration of Algorithm I if we have queried {Vr(r,) "u,9 then the optimal vn maximizing C2 s.t. || =
should be in the following form: et- f((1)) where A == span (n,, .2..... Ugg and V V (x:)A denotes the projection of V fxx) onto A.
'It is also the difficulty we faced in our very preliminary attempts of the theoretical analysis when f is not convex, and we leave its solution workaround in the non-convex case future work.

---

Note that in Line 3 of Algorithm we have 9: = vf(x,)
Vf(z)) (f(zz)
Vf(z))
Therefore. the gradient estimator 9: is equivalent to the projection of the gradient the subspace A. which justifies its name of subspace estimator. Next we discuss some special cases of the subspace estimator. We leave detailed derivation following examples to Appendix [B.3]
Example 1 (RGF), u, ~ U($ for i 1.2..... Without loss of generality, we assume they are orthonormal (e.g., via Gram-Schmidr orthogonalization]] | The corresponding estimator St = =' (x+))) "4:" U4 (e/- = 9t). When q 1, the estimator similar the random gradient- free oracle in 121y. With q 2 1, it is essentially the same as the stochastic subspace estimator with columns from Haar-distributed random orthogonal matrix [15] and similar to the orthogonal ES estimator (9. In theoretical analysis, we let F: only include the randomness before iteration t, and 2L' R2 then can prove that E, [Cr] 3. By Theorems I and 2 the convergence rate is E[ S T+1
for smooth convex case, and E E[ S 60 exp(-E TT) for smooth and strongly convex case. The bound is the same as that in [15]. Since the query complexity in each iteration is proportional to g, the bound for total query complexity is indeed independent of q.
Example 2 (PRGF). With slight notation abuse, we assume the subspace in Proposition is spanned by .pI,....PPh Pk. Mi...., Way. so each iteration takes q + k queries. Let pı,- Ps be k non-zero vectors corresponding to the prior message (e.g. the historical update, or the gradient of a surrogate ................................
model), and u, ~ (Sd- I) for i= = 1.2..... 9. We note that intuitively we cannot construct a better subspace since the only extra information we know is the priors. In our analysis, we assume k = 1 for convenience, and we change the original notation P1 to Pt to explicitly show the dependence of Pt on the history. We note that Pt could also depend on extra randomness in iteration f. (see e.g. the specification of in Appendix D.I.Iy. For convenience of theoretical analysis, we require that pt determined before sampling (11, 141,U2,-- U2...., Wyg, and let Ft-1 also include the extra randomness of Pt. in iteration t (not including the randomness of {ui, (02,...,2q)) besides the randomness before iteration t. Then pt is always Fi-1 7r-1-measurable i.e. determined by the history. Without loss of generality, assume {pe, U1, ....... ... 3gg are orthonormal (e.g., via Gram-Schmide orthogonalization) The corresponding estimator 9t VI(xx)) Pt. Pt + +E V((x)) u; uu (vr = Ti), which is similar to the estimator in 199. By (99 (the expected drift of X2 in its Theorem 1), we have
Lemma 2 (Proof in Appendix B.3.4). In Algorithm 7 with PRGF estimator, E, [Cr] D. + 2-
D₂) where De = ((())) Pt)
Hence E. [c.C á holds. By Remark PRGF admits a guaranteed convergence rate of RGF and is potentially better given a good prior (if D, is large). but it costs an additional query per iteration. This shows soundness of the PRGF algorithm. For furthrr theoretical analysis, we need to bound Di- This could be done when using the historical prior introduced in Section 3.2 ( (see Lemma Bounding Dj is usually challenging when general prior is adopted, but if the prior is approximate gradient (such case appears in (16)), it may be possible. We leave related investigation as future work.
3.2 Analysis on the PRGF algorithm with the historical prior
We apply the above analysis to a concrete example of the History-PRGF estimator [9], which considers the historical prior in the PRGF estimator. In this case, Lemma [1. Theorem and Theorem [ will manifest themselves by clearly stating the convergence rate which is robust to the learning rate.
Specifically, History-PRGF considers the historical prior follows; we let the prior be the direction of the previous gradient estimate &
we choose Pt 9t- i.e.,
This is equivalent to letting (z()) f(z)) (T)z)Teze T 1/-1
Pt Ut-1. Thus, in Algorithm In this form we require {wt-1, uj...., [g} ) to be orthonormal, so we first determine Ut-1, and then
+1 Vf(x,) (T Tu,- uy.
Thhe computational complexity of Gram-Schmidt orthogonalization over q vectors in Rd is Og²d).. There- fore, with a moderate value of q (e.g- E [10, 20] in our experiments), its cost is usually much smaller than that brought by O(g) function evaluations used to approximate the directional derivatives. We note that when using a numerical computing framework. for orthogonalization one could also adopt an efficient implementation. by calling the QR decomposition procedure such as torch. linalg. qr in Py PoTch.
*One can also utilize multiple historical priors (e.g. the last & updates with k > 1, as proposed and experimented in (19), but here we only analyze the l case.
5

---

sample f2,1=11 in A1, the (d- 1) diimennsional subspace of R perpendicular to 0t-1, and then do Srha--Shhmitt orthonormalization on {u, 1
To study the convergence rate, we first study evolution of C. under a general T-smooth function. This extends the analysis on linear functions (corresponding Algorithm [ intuitively, the change of the gradient should be smaller when the objective function is very smooth (L is small) or the learning rate is small (L is large). Since we care about the cosine similarity between the gradient and the prior, we prove the following lemma:
I. = 0) in [19]. Under the framework of
Lemma 3 (Proof in Appendix B.4. In History-PPRGP (p: = 1),, we have 2 (1-1111 C--1-
When É = L, Le., using the optimal learning rate, Lemma does not provide useful bound, since an optimal learning rate in smooth optimization could find an approximate minimizer along the update direction, so the update direction may be useless in next iteration. In this case, the historical prior does not provide acceleration. Hence, Lemma 3 explains the empirical findings in [19] that past directions be less useful when the learning rate is larger. However, in practice conservative learning rate, for the following reasons: 1) We usually do not know I., and the cost of tuning learning rate could be large; 2) Even if L is known, it only provides global bound, so fixed learning rate could be too conservative in the smoother local regions. In scenarios where L is too conservative (L L), History- PRGF could bring more acceleration over RGF.
often use
By Lemma we can assume D; (1 (1-24 2/E)²² C,-1 obtain lower bound of quantities about C.- Meanwhile, since D, means quality of the prior, the construction in Example 2 tells us relationship between C and Di- Then we have full knowledge of the evolution of C., and thus [,[., In
Appendix B.4 we discuss about evolution of [[C]] and show that [[[]] + 0 if g <
S.
Therefore, by Lemmal) 1
assuming E. [Ce] concentrates well around E[G; and hence C. PRGF could recover the single step progress with optimal learning rate (L L). since Eq. depends on [.(cll which is constant w.r.t. L' now. While the above discussion informal, based on Theorems [ and - we prove following theorems which show that convergence rate of History-PRGGr robust to choice of learning rate.
1' dt-
3 only
Theorem 3 (History-PRGF, smooth and convex; proof in Appendix(B.3 In the setting of Theorem [ (I-) denotes the ceiling function), we have
assuming d 1 2 S & S 1 and T >
2T R2
32
Elōr| S
+
(6)
T
+1
Sketch of the proof. The idea of the proof of Theorem is to show that for the random variable
[,[C,], its standard deviation EE,
[C.]] is small relative to its expectation E[E, [Ca]] = EC2,].
By Chebyshev's ' inequality, we can bound E actual proof we replace C2 that appears above with lower bound
Theorem with [[E[C.:]] [[C]]] In the
[Cc]
Theorem 4 (History-PRGE, smooth and strongly convex; proof in Appendix B.5.3p. Under the same conditions as in Theorem then assuming d 2 4, / S < 1, 2 < 0.24, and T 2 5 have
[ < 2 exp (-0.1 (T) 60.
This result seems somewhat surprising since Theorem does not directly give bound of [[6r]]
Sketch of the proof. The goal to show that the denominator in the l.h.s of Eq. (5) Theorem
exp(-7 i D E, [Ct],, concentrates very well. Indeed, the probability that exp( I Σ E, (C.))
is larger than exp(-0.19 (.T) is very small so that its influence can be bounded by another exp(-0.19 2 7 leading the coefficient in Eq. (7.. In our actual analysis we replace C. that appears above with lower bound F,.
Remark 5. As stated in Example 2 using RGF with the optimal learning rate, we have Eiōr| <
21 R²
T+1 for smooth and convex case, and E(ör] < exp (- (-) 60 for smooth and strongly convex

---

case. Therefore, History-PRGF with a suboptimal learning rate 1 under the condition 3't I could reach similar convergence rate to RGF with optimal learning rate (up to constant factors), which indicates that History-PRGG is more robust to learning rate than RGF.
Remark 6. We note that the constants the bounds are loose and have a large potential to be improved in future work, and empirically the convergence rate of History-PRGP often not worse than RGF using the optimal learning rate (see Fig. 2
As sidenote, discuss how to q in History-PRGE, The iteration complexity given by Theorems 3 and44is proportional to if we ignore the constants such as 22 + in Eq. (6) by Remark Meanwhile, we recall that each iteration of PRGF requires q + queries to the directional derivative oracle, so the total query complexity is roughly proportional to 2+11 Hence, a very small (e.g. q = 1) is
q
9
to enable robustness of History-PRGF to a wider range of the choice of Tearning rate, should not be too large. In summary. it is desirable set 9 to moderate value.
suboptimal. On the other hand, Theorems and 4 require
Note that if we adopt line search in Algorithm then one can adapt the learning rate in a huge range and reach the convergence guarantee with the optimal learning rate under weak assumptions. Nevertheless, it is still an intriguing fact that History-PRGP could perform similarly to methods adapting the learning rate, while its mechanism is very different. Meanwhile, History-PRoo is easier to be implemented and parallelized compared with methods like line search, since its implementation the same as that of the RGF baseline except that it records and uses the historical prior.
Extension of ARS framework and PARS algorithm
To further accelerate greedy descent methods, we extend our analysis to a new variant of Accelerated Random Search (ARS) 21 by incorporating prior information, under the smooth and convex setting
By delving into the proof in (11, we present our extension to ARS in Algorithm 2] state its conver- gence guarantee in Theorem and explain its design in the proof sketch in Appendix C.I
Algorithm 2 Extended accelerated random search framework
Input: L-smooth convex function fi initialization z0; L; iteration number T:ço >> O. Output: the approximate minimizer of
I: mo z0; for to T
R: ((((V)o))TT where yr. Vs and (2())) are defined following steps: 3: Find 0 > 0 such that S (-Eello2(w)11))
4:
Step 1:
(1 a1) + ooma, where is positive root of the equation of = 0. o1)
7641 (1 - au)
5:
Step 2: Let oe be a random vector s.t. |0:]| = 10:l=1:::]00 (1:) + V(())) T ve:
6: Step 3: Let (34) be unbiased estimator of ( i.e., B. [oo())) = vf());
7: 2141 + lt [ 9) (H:). m441 f- my ag 92());
8t
8: end for
9:
return TT-
Theorem 5 (Proof in Appendix C.I. for more explanation), then we have
In Algorithm 2 if0, is F-1- (see Appendix C.1
E (U(oT)- ((x))) (1
27
T0
< f(50) [x**++ Yu 1200 x1
(8)
"The procedure of ARS requires knowledge of the strong convexity parameter 7 (7 can be 0), but for clarity we only discuss the case = 0 here (i.e., we do not consider strong convexity), and leave the strongly convex case Appendix C.6
"eeys in this extension are: 1) We require [xg)( = v (yr). Thus, g2 g2(M:) could not be the PRGF estimator as it is biased towards the prior; 2) To accelraaee convergence. we need to find an appropriate 9, since it appears in Eq. [8). If we set O: to the value in ARS baseline, then potential acceleration is guaranteed.
7

---

Remark 7. we let 91 (3t) be the RGF estimator in Example S and let (2 = 4/4 91 (31e). we can E. (V
show that Elg2(1))] = V f and Os could be chosen as 4 since L.E, 92(v=]1 Le
L Then
roughly, the convergence rate x so the total query complexity is independent of When g = 1, ARS baseline is recovered. For convenience we call the algorithm ARS regardless of the value of 9- Remark 8. If we have uniform constant lower bound 0 > 0 such that Vt, O2 2 0, then we have
((((r))-((()))) (1 you VTTTT) (S(ZN) f(x=) ) + 70 I1:00 x x²11²) 2
(9)
Next we present Prior-Guided ARS (PARS) by specifying the choice of 9i (yt) and 92(2)) in Al- gorithm 2 when prior information P. Fd is available. Since we want to maximize the value
of f2. regarding 91 (31) we want to maximize E, (V)( Tx)) =x)²]. By Proposition and Exam-
ple 2 it is natural to let 9i (yr) be the PRGF estimator for V (Vtt). Then by Lemma
we have
E, (Vf( T v [Vff())) 21 D:)). where D, == ff(()) Pt The remain-
ing problem is to construct (v)).. an unbiased estimator of f (g,(9(()))
Vf())), and make The [1l92 (1x)) [1²) as small as possible. We leave the construction of (2())) ) in Appendix C.2
Finally, we calculate the following expression which appears in Line 3 Algorithm to complete
the description of PARS:
E. (VI(M)) )=
D. - (1-1D D.)
(10)
L (v)))1
i. (.. + - (1-)))
Since D; 2 0, the right-hand side is larger than v'/La" (by Remark 7 this value corresponds to the value 0g in ARS), so by Remark PARS is guaranteed convergence rate of ARS.
In implementation of PARS, we note that there remain two problems to solve. The first is that D, is
not accessible through one oracle query, since D, = (we) pt)² = (V P1/I)(()))) and ( )I requires estimation. Fortunately, the queries used to construct 91 (1x) and g2(V)) can also be used to estimate D,. With a moderate value of q. we can prove that considering the error brought by estimation of D,, a modified version of PARS is guaranteed to converge with high probability. We leave related discussion to Appendix C.3
The second is that Line of Algorithm 2 has subtlety that
E. [V va)"]
Yr depends on Oc, we cannot directly determine optimal O, satisfying 0₂ S L.E, (Mx)))
Theoretically, we can guess a conservative estimate of 0g and verify this inequality, but in practice adopt more aggressive strategy to find approximate solution of 6.. We leave the actual implementation, named PARS-Impl, in Appendix C.4
In PARS, if we adopt the historical prior as in Section i.e., letting Pt be the previous PRGF gradient estimator 91 (1. then arrive at novel algorithm named HfistoryPAARe Here, note that unlike the case in History- History-PRGE, it is more difficult to derive the evolution of 0, theoretically, so currently cannot prove theorems corresponding to Theorem 3 However, History-PARe can be guaranteed the convergence rate of ARS, which is desirable since we adopt line search in ARS to reach robustness against learning rate (e.g. in 24 ) currently there is no convergence guarantee. We present the actual implementation Distory-PARS-Imp in Appendix C.5land and empirically verify that History-PARS-Imy is robust to learning rate in Section 5
5
Experiments
5.1 Numerical benchmarks
We first experiment several closed-form test functions to support theoretical claims. We leave more details of experimental settings to Appendix D.I
First, we present experimental results when a general useful prior is provided. The prior- siiorr-gideed methods include PRGF, PARS (refers to PARS-Impl) and PARS-Naive (simply replacing the RGF estimator in ARS with the PRGF estimator). We adopt the setting in Section 4.1 of [16) in which the
8

---

PRGE
1927
-3
20.00
160
100
200
330
6.0
300
(a) fi
(b) f2
(c) fa
Figure 1: Experimental results using biased gradient as the prior (best viewed in color).
prior is a biased version of the true gradient. Our test functions are as follows: 1) fi is the "worst-case smooth convex function" used to construct the lower bound complexity of first-order optimization, as in 21: 2) f2 is a simple smooth and strongly convex function with a worst-case initialization: f2(2)
Σ (i. (2)()))), where x!"1 d. _ 0 for 2: and 3) f3 is the Rosenbrock function Ss in 12 which is a well-known non-convex function used to test the performance of optimization problems. For f1 and f2. we set i to ground truth value L; for f3. we search for best performance for each algorithm. We d = 256 for all test functions and set q such that each iteration of these algorithms costs 11 queries I to the directional derivative oracle 12 We plot the experimental results Fig. D where the horizontal axis represents the number of iterations divided by [4/11, and the vertical axis represents log [[oorrent) f(="), Methods without using the prior [(x))_(<x)) information are shown with dashed lines. We also plot the 95% confidence interval in the colored region. The results show that for these functions (which have ill-conditioned Hessians), ARS- basee methods perform better than the methods based on greedy descent. Importantly, the utilization of the prior could significantly accelerate convergence for both greedy descent and ARS. We note that the performance of our proposed PARS algorithm is better than PARS- -aive which naively replaces the gradient estimator the original ARS with the PRGF estimator, demonstrating the value of our algorithm design with convergence analysis.
Next, we verify the properties of History- PRGF and History-PARS, i.c., the iistorical-prior-eee guided algorithms. In this part we set d = 500. We first verify that they are robust against learning rate on fi and fe. and plot the results in Fig. (b) In the legend, for example, 'RGF' means RGF using the optimal learning rate (i. L), and 'RGF-0.02" means that the learning rate is set to 0.02 times of the optimal one (?. = 50L). We note that for PRGF and PARS, = 10, so 9 = 0.02. From Fig. (a)),,),,, we see that: 1) when using the optimal learning rate, the performance of prior-guided algorithms is not worse than that of its corresponding baseline; and 2) the performance of prior-guided algorithms under the sub-optimal learning rate such that / S 4 < 1 is least comparable to that of its corresponding baseline with optimal learning rate. However, for baseline algorithms (RGF and ARS), the convergence rate significantly degrades a smaller learning rate is set. In summary, we verify our claims that History-PRGE and History-PARR -PARS are robust to learning rate if 3 S S 1. Moreover, show that they can provide acceleration over baselines with optimal learning rate functions with varying local smoothness. We design a new test function as follows:
fa(x) = 1 21 r r>1
r rs1
T f2(x), where x = 5v x0 = for 2. (11)
We note that fa in regions far away from the origin is more smooth than in the region near the origin, and the global smoothness parameter is determined by the worst-case situation (the region near the origin). Therefore, baseline methods using an optimal learning rate could also manifest sub-optimal performance. Fig. ( shows the results. We can see that when utilizing the historical prior, the algorithm could show behaviors of adapting to the local smoothness, thus accelerating convergence when the learning rate is locally too conservative.
11 That is, for prior-guided algorithms we set g = 10, and for other algorithms (RGF and ARS) we set g = 11. 12ehe directional derivative is approximated by finite differences. In PARS, additional queries the directional derivative oracle per iteration are required to find 0, (see Appendix C.4
In Fig. 2 the setting of baseb methods are different from that in Fig. as explained in Appendix D.I
which leads to many differences of the ARS curves between Fig. 1 and Fig

---

AB
-022
-03
-01
-160
-04
AGP
-55
-03
70/7
00-
A25
-26
PARS-11
(a) f1
(b)
(c) f4
Figure 2: Experimental results using the historical prior.
5.2 Black-box adversarial attacks
In this section, we perform ZO optimization on real-world problems. We conduct score-based black- box targeted adversarial attacks on 500 images from MNIST and leave more details of experimental settings to Appendix D.2 In view of optimization, this corresponds to performing constrained maximization over [ Too respectively, where fi denotes the loss function to maximize in attacks. For each image we record the number of queries of f, used in optimization until the attack succeeds (when using the C&W loss function [6) this means fi > 0). For each optimization method, we report the median query number over these images (smaller is better) in Table The subscript of the method name indicates the learning rate 1/i. For all methods we set to 20. Since [8] has shown that using PRGF estimator with cansfer-based prior significantly outperforms using RGF estimator in adversarial attacks, for prior prior-guidee guided algorithms here we only include the historical prior case.
Table 1: Attack results on MNIST.
METHOD
MEDIAN QUERY
METHOD
MEDIAN QUERY
RGF
777
ARSp.2
735
RGF
1596
ARSp.1
1386
History-PRGF 0.2
484
History-PARS, 0.2
484
History-PRGi 0.1
572
History-PARS,
550
History-PRGE 0.05
704
History-PARS, 0.05
726
We found that in this task, be because 1) the numbers of iterations until success of attacks are too small to show the advantage of ARS; 2) currently ARS is not guaranteed to converge faster than RGF under non-convex problems. leave more evaluation of ARS- methods in adversarial attacks and further improvement of their performance as future work. Experimental results show that History-PRCC is more robust to learning rate than RGF. However, small learning rate could still lead to its deteriorated performance due to non-smoothness the objective function. The same statement holds for ARS- ARS-based algorithms.
methods perform comparably RGF-based ones. This could
Conclusion and discussion
In this paper, present convergence analysis existing prior-guided ZO optimization algorithms including PRGF and History-PRGE We further propose a novel mrior-guided ARS algorithm with convergence guarantee. Experimental results confirm our theoretical analysis.
Our limitations lie in: 1) we adopt a directional derivative oracle our analysis, so the error on the convergence bound brought by siite-difference approximation has not been clearly stated; and 2) our implementation PARS practice requires an approximate solution of 0., and accuracy and influence of this approximation is not well studied yet. We leave these as future work. Other future work includes extension of the theoretical analysis to non-convex cases, and more empirical studies in various application tasks.
10

---

Acknowledgements
This work was supported by the National Key Research and Development Program of China (No. 2020AAA0104304), NSFC Projects (Nos. 61620106010, 62061 136001, 1662113600,, 62076147, U19B2034, U19A2081, U181 1461), Beijing NSF Project (No. JQ19016), Beijing Academy of Huawei Joint Research Program, grant from Tsinghua
Artificial Intelligence (BAAD, Tsinghua- -singhua-Huawei Institute for Guo Qiang, Tiangngg Institute for Intelligent Computing, and the NVIDIA NVAIL Program with GPU/DGX Acceleration.
References
[1] Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning learn by gradient descent by gradient descent. arXiv preprint arXiv: /606.04474 2016.
[2] Albert Berahas, Liyuan Cao, Krzysztof Choromanski, and Katya Scheinberg. theoretical and empirical comparison of gradient approximations derivative-free optimization. Foundations of Computational Mathematics, pages 1-54, 2021.
[3] El Houcine Bergou, Eduard Gorbunov, and Peter Richtarik. Stochastic three points method for uncon- strained smooth minimization. SIAM Journal on Optimization, 30(4):2726-2749, 2020.
[4] Thomas Brunner, Frederik Dichl, Michael Truong Le, and Alois Knoll. Guessing smart: Biased sampling for efficient black-box adversarial attacks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4958 4968.996.220 2019.
[5] Sébastien Bubeck. Convex optimization: Algorithms and complexity. arXiv preprint arXiv:1405.4988 2014.
[6] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp), pages 39-57. IEEE, 2017.
[7] Pin- Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho- Jui Hsieh. Zoo: Zeroth order optimization based black-bor attacks to deep neural networks without training substitute models. In Proceedings of the 10th ACM workshop artificial intelligence and security, pages 16-26, 2017.
[8] Shuyu Cheng, Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu. Improving black-box adversarial attacks with a transfer-based prior. arXiv preprint arXiv:1966.066 06919, 2019.
[9] Krzysztof Choromanski, Mark Rowland, Vikas Sindhwani, Richard Turner, and Adrian Weller, Structured evolution with compact architectures for scalable policy optimization. In International Conference on Machine Learning, pages 970-978. PMLR, 2018
[10] Daniel Golovin, John Karro, Greg Kochanski, Chansoo Lee, Xingyou Song, and Qiuyi Zhang. Gradientless descent: High-dimensional zeroth-order optimization. arXiv preprint 191vv1991..66 2019.
[11] Nikolaus Hansen and Andreas Ostermeier. Adapting arbitrary normal mutation distributions in evolution strategies: The covariance matrix adaptation. In Proceedings of IEEE international conference on evolutionary computation, pages 312-317. IEEE, 1996.
[12] Nikolaus Hansen, Steffen Finck, Raymond Ros, and Anne Auger. Real-parameter black-box optimization benchmarking 2009: Noiseless functions definitions. PhD thesis, INRIA, 2009.
[13] Andrew Tlyas, Logan Engstrom, Anish Athalye, and Jessy Lin. Black-hox adversarial attacks with limited queries and information. In International Conference on Machine Learning, pages 2137-2146. PMLR, 2018.
[14] Andrew Ilyas, Logan Engstrom, and Aleksander Madry. Prior convictions: Black-box adversarial attacks with bandits and priors. arXiv preprint
2018
[15] David Kozak. Stephen Becker, Alireza Doostan, and Luis Tenorio. stochastic subspace approach to gradient-free optimization in high dimensions. Computational Optimization and Applications, 79(2): 339-368. 2021.
[16] Niru Maheswaranathan, Luke Metz, George Tucker, Dami Choi, and Jascha Sohl- -ick-Dicsttein Guided evolutionary strategies: Augmenting random search with surrogate gradients. In International Conference Machine Learning, pages 4264 4273, PMLR, 2019.

---

[17] Horia Mania. Aurelia Guy, and Benjamin Recht. Simple random search of static linear policies is competitive for reinforcement learning. In Proceedings the 32nd International Conference on Neural Information Procexsing Systems, pages 1805-1814, 2018.
[18] Matyas. Random optimization. Automation and Remote control, 26(2):246 253, 1965.
[19] Florian Meier, Asier Mujika, Marcelo Matheus Gauy, and Angelika Steger. Improving gradient estimation in evolutionary strategies with past descent directions. arXiv preprint arXiv:1910.05268, 2019.
[20] Yu Nesterov. Efficiency of coordinate descent methods on huge-scale optimization problems. SIAM Journal on Optimization, 22(2):341-36,, 2012.
[21] Yurii Nesterov and Vladimir Spokoiny. Random gradient-free minimization of convex functions. Founda- tions Computational Mathematics, 17(2):527-566, 2017.
[22] Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and llya Sutskever. Evolution strategies scalable alternative reinforcement learning. arXiv preprint urXiv: 770.03364, 2017.
[23] Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. arXiv preprint arXiv: /206. 2944, 2012.
[24] Sebastian U Stich, Christian L Muller, and Bernd Gartner. Optimization of convex functions with random pursuit. SIAM Journal on Optimizationn, 2342):184-1109, 2013.
[25] Chun-Chen Tu, Paishun Ting, Pin-Yu Yu Chen, Sijia Liu, Huan Zhang, Jinfeng Yi, Cho-Jui Hsich, and Shin- Ming Cheng. Autozoom: Autoencoder-basee zeroth order optimization method for attacking black-box neural networks. Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 742-749, 2019.
12


Reviews:
[{"review_id": "n3pXVPgmIOu", "reviewer": "Reviewer_xyD2", "summary": "This work analyzes the convergence rates of several existing prior guided zero order optimization methods. The zero order methods studied use statistical methods to provide an estimate of the function gradient using only function evaluations. This works studies the trade offs between gradient estimation accuracy and function evaluations as well as the effect of learning rate selection for the various estimates. This work also proposes a prior guided version of accelerated random search and studies its convergence. All algorithms studied are then numerically compared on benchmark functions as well as black box adversarial attacks. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors here focus on the effects of history dependent gradient estimates on zero order optimization.  The setting studied here is challenging because gradient estimates themselves are not statistically independent. This requires significant modifications to the standard convergence arguments and to the derived bounds that the authors clearly explain.  To the best of my knowledge, the technical arguments used here are novel.\n\nThe authors also try to explain the empirical success of history based gradient estimators. The authors prove that convergence can be accelerated by using previous gradient information when a suboptimal learning rate is used, which is in agreement with past empirical findings. I found this insight very helpful and relevant given that learning rate selection remains more art than science even for first order methods. \n\nFinally, the analysis of the prior guided version of Accelerated Random Search method requires  even more work as additional quantities beyond directional derivatives need to be estimated. The experiments showcase that the prior guided algorithms can be more efficient than their simpler counterparts. I believe that this work both explains the success of existing works but can also have standalone theoretical and practical impact.\n\nI acknowledge the authors' responses to all reviewers. I believe the additions proposed by the authors will make the submission stronger so I continue to be in favor of accepting this submission.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and negative social impact are adequately discussed.\n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "WDGaY5zfp0", "reviewer": "Reviewer_gEjB", "summary": "The paper provides a convergence analysis of a class of prior-guided zeroth-order (ZO) optimization algorithms and gives a prior-guided variant of the accelerated random search (ARS) algorithm. \nContribution:\n1. Provides a complete convergence on some prior-guided zeroth-order optimization algorithms.\n2. Provides a prior-guided variant of the ARS algorithm.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The presentation of the convergence analysis is very nice, very well-written, and easy to follow. It is shown through the analysis that a good prior is one with a larger correlation with the gradient which is intuitively true.\n2. The prior-guided ARS is a good addition to the class of prior-guided ZO algorithms. In addition, the new algorithm performs well on some convex functions.\n\nWeaknesses:\n1. It seems the analysis is just writing down traditional analysis of ZO algorithms with some quantities waiting to be instantiated to specific forms for different algorithms. I do not really see any unexpected new insights distilled from the analysis.\n2. The proposed prior-guided ARS seems does not to perform well on black-bok attacks which is a key application to ZO algorithms in machine learning. This limited the impact of the paper.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "KUtOzHMj8EQ", "reviewer": "Reviewer_c85q", "summary": "This paper studies the performance of zeroth order optimization algorithms where gradient estimation uses prior information. They provide a bound on the convergence analysis for such prior guided algorithms in terms of expected alignment of the gradient estimates to the true gradient. They use this framework to study historical priors and show that such a prior may counter the effects of using a sub-optimal learning rate. They also provide a new accelerated random search algorithm that incorporates prior information. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper studies an important and interesting problem. Their insights into the performance of the historical prior are useful. The paper is clearly written and well organized. \nMy concerns with this paper are as follows:\n- the difference between the bounds in the convex and strongly convex case are not well highlighted or discussed. Moreover, zeroth order algorithms are not typically applied to only convex functions so some discussion on possible extensions or the difficulty of should be included in this work. \n- while the bounds are used to study historical priors, it would benefit to work to either study other examples of priors or to demonstrate why their bounds in terms of the expected correlation between the prior and the true gradient are useful for other priors. Can we expect that such bounds are easy to compute for other priors? ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper does address the limitation of using the directional gradient oracle. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "HGxAADjTpG5", "reviewer": "Reviewer_1ukB", "summary": "This work studies the problem of zeroth-order optimization in a deterministic setting, where the exact function values at each point can be observed but the gradient of the function is not accessible. The main approach for solving such problems is first to estimate the gradient of the function and then apply a gradient method by plugging in this estimator instead of the gradient. The main goal of this work is to provide convergence guarantee for prior guided zeroth order algorithms.\n\nAssumptions on the objective function: The objective function is (strongly) convex and smooth.\n\nApproach: Since the paper assumes a deterministic setting, at each step by two function evaluations and choosing a small enough perturbation term, roughly they can observe the gradient of the function towards any arbitrary direction (page 3, equation 1). Also, they assume that at each step they can observe $q$ random directions of the gradient, which are given priorly (or they estimate them by $2q$ additional function evaluations). Then, they take advantage of these $q$ given directions and $g_{I}$s to construct $g_t$, where $g_t$ is the gradient estimator at step $t$, and $I<t$. \n\nContributions: Their results in Theorems 3 and 4 outline the convergence guarantees and robustness to the learning rate.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Before the authors' responses, some parts were not clear to me. They provide useful justification and clear the misunderstanding. I edited the review and raised my score. \n\n1. Theoretical claims are well supported. \n2. The technical aspects are well written. Also, in the appendix, they tried to cover the most general cases, which was quite nice.\n3. I encourage the authors to discuss the computational cost of Gram-Schmidt procedure diagonalization.\n4. Their results provide intuition about the practical success of History based algorithms and outline the robustness of History-PRGF against the choices of learning rate. \n5. The authors also provide convergence analysis for an accelerated greedy method, which is outlined in Theorem 5. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors briefly discuss the limitations of their work. I also pointed out some, in the Main Review section.\n\nSince the paper is quite technical, I except no direct social impact at this point.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: While there were some initial concerns from the reviewers, these were all clearly resolved following the author rebuttal, with the reviewers in question acknowledging the depth and clarity of the responses.  The reviewers are now in agreement that this paper is a valuable addition to the literature on zero-order optimization, and is suitable for publication at NeurIPS.  I do not have any specific reviewer points to highlight here, but I do ask that the authors carefully consider all of the reviewer comments when forming the camera-ready version.
Decision: Accept (Poster)