Black-Box Generalization: Stability of Zeroth-Order Learning
Authors: Konstantinos Nikolakakis
Farzin Haddadpour
Dionysios Kalogerias
Amin Karbasi
Venue: NeurIPS 2022 Accept (2022)
Keywords: Generalization Error
Zeroth-Order Optimization
Black-Box Learning

Abstract: We provide the first generalization error analysis for black-box learning through derivative-free optimization. Under the assumption of a Lipschitz and smooth unknown loss, we consider the Zeroth-order Stochastic Search (ZoSS) algorithm, that updates a $d$-dimensional model by replacing stochastic gradient directions with stochastic differences of $K+1$ perturbed loss evaluations per dataset (example) query. For both unbounded and bounded possibly nonconvex losses, we present the first generalization bounds for the ZoSS algorithm. These bounds coincide with those for SGD, and they are independent of $d$, $K$ and the batch size $m$, under appropriate choices of a slightly decreased learning rate. For bounded nonconvex losses and a batch size $m=1$, we additionally show that both generalization error and learning rate are independent of $d$ and $K$, and remain essentially the same as for the SGD, even for two function evaluations. Our results extensively extend and consistently recover established results for SGD in prior work, on both generalization bounds and corresponding learning rates. If additionally $m=n$, where $n$ is the dataset size, we recover generalization guarantees for full-batch GD as well. 

Black-Box Generalization: Stability of Zeroth-Order Learning
Konstantinos E. Nikolakakis
Farzin Haddadpour
Yale University
Yale University
konstantinos. www.ssiillllkeeecooo wak.kkllakkis@@ale...oo. edu
farzin. heaiaanhhdddppuu@ylle..coo
Dionysios S. Kalogerias Yale University
Amin Karbasi
Yale University & Google Research amin. karbasi@yale.... edu
Abstract
We provide the first generalization error analysis for black-box learning through derivative-free optimization. Under the assumption of Lipschitz and smooth unknown loss, we consider the Zeroth-order Stochastic Search (ZoSS) algorithm, that updates dddiimensional model by replacing stochastic gradient directions with stochastic differences of K + perturbed loss evaluations per dataset (example) query. For both unbounded and bounded possibly nonconvex losses, we present the first generalization bounds for the 7oSS algorithm. These bounds coincide with those for SGD, and they are independent d, K and the batch size m, under appropriate choices of slightly decreased learning rate. For bounded nonconvex losses and batch size m 1, we additionally show that both generalization error and learning rate are independent of d and K, and remain essentially the same as for the SGD, even for two function evaluations. Our results extensively extend and consistently recover established results for SGD in prior work, on both generalization bounds and corresponding learning rates. additionally m = n, where is the dataset size, recover generalization guarantees for full-batch GD as well.
Introduction
Learning methods often rely on empirical risk minimization objectives that highly depend on limited training data-set. Known gradient-based approaches such SGD train and generalize effectively reasonable time [1]. In contrast, emerging applications such as convex bandits [2- 4], black-box learning [5], federated learning [6], reinforcement learning [7, 8]. learning linear quadratic regulators [9, 10], and hyper-parameter tuning [11] stand in need of gradient-free learning algorithms [1]-14] due to unknown loss/model impossible gradient evaluation.
Given two or more function evaluations, zeroth-order algorithms (see, e.g., [14, 15)) aim estimate the true gradient for evaluating and updating model parameters (say, of dimension d). In particular, Zeroth-order Stochastic Search (ZoSS) [13, Corollary 2), [16, Algorithm 1] uses + function evaluations (K > 1), while deterministic zeroth-order approaches [5, Section 3.3] require at least K 2 + queries. The optimization error the ZoSS algorithm is optimal as shown in prior work for convex problems [13], and suffers at most a factor /d/K in the convergence rate as compared with SGD. In addition to the optimization error, the importance of generalization error raises the question of how well zeroth-order algorithms generalize to unseen examples. this paper, show that the generalization choice of slightly decreased learning rate. Assuming a Lipschitz and smooth loss function, we
of ZoSS essentially coincides with that of SGD, under the
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

establish generalization guarantees for ZoSS, by extending stability- based analysis for SGD [1] to the gradient- free setting. In particular, we rely on the celebrated result that uniform algorithmic stability implies generalization 11817,18]
Early works [17, 19-22] first introduced the notion of stability, and the connection between (uniform) stability and generalization. Recently, alternative notions of stability and generalization gain attention such as locally elastic stability [23]. VC-dimension/llatess measures [24], distributional stability [25- 27]. information theoretic bounds [16, 28-33] mainly based on assuming a sub-Gaussian loss, as well as connections between differential privacy and generalization 13-377..
In close relation to our paper, Hardt et al. [1] first showed uniform stability final-iterate bounds for vanilla SGD. More recent works develop alternative generalization error bounds based on high probability guarantees [38-41] and data-dependent variants [42], or under different assumptions than those of prior works such as strongly quasi-convex [43], non-smooth convex [44-4]], and pairwise losses [48,49). In the nonconvex case, [50] provide bounds that involve on-average variance of the stochastic gradients. Generalization performance of other algorithmic variants lately gain further attention, including SGD with early momentum [51], randomized coordinate descent [52], look-ahead approaches [53], noise injection methods [54], and stochastic gradient Langevin dynamics [55-62].
Recently, stability and generalization of full-bath GD has also been studied; see, e.g., [63-67]. In particular, Charles and Papailiopoulos. [64] showed instability of GD for nonconvex losses. Still, such instability does not imply a lower bound on the generalization error of GD (in expectation). In fact, Hoffer et al. [63] showed empirically that the generalization of GD is not affected by the batch-size, and for large enough number of iterations GD generalizes comparably to SGD. Our analysis agree with the empirical results of Hoffer et al. [63], as we show that (for smooth losses) the generalization of ZoSS (and thus of SGD) is independent of the batch size.
Notation. We denote the training data-set S of size n as {2 where 2; are i.i.d. observations of a random variable Z with unknown distribution D. The parameters of the model are vectors of dimension d, denoted by W' E E2d and We is the output at time of a (randomized) algorithm As. The (combined) loss function f(,; 2) I2d R- is uniformly Lipschitz and smooth for all 2 Z. We denote the Lipschitz constant as L and the smoothness parameter by 3. The number of function (i.e., loss) evaluations (required at each iteration of the ZoSS algorithm) represented by K 1 E N. We denote by A the smoothed approximation of the loss gradient, associated with parameter J.. The parameter K A /(3d 1)/K K + prominently appears in our results. We denote the gradient of the loss function with respect to model parameters W, by Vf(W,) = f 2) www= We denote
the mini batch at by Jr. and m & |Je]-
1.1 Contributions
Under the assumption of Lipschitz and smooth loss functions, we provide generalization guarantees for black-box learning, extending the analysis of prior work by Hardt et al. [1] to the gradient free setting. In particular, we establish uniform stability and generalization error bounds for the final iterate of the ZoSS algorithm; see Table for a summary of the results. In more detail, the contributions of this work are follows:
For unbounded and bounded losses, we show generalization error bounds identical to SGD, with slightly decreased learning rate. Specifically, the generalization error bounds independent of the dimension d, the number of evaluations K and the batch-size m. Further, a large enough
number of evaluations (K) provide fast generalization even in the high dimensional regime.
For bounded nonconvex losses and single (example) query updates (m = 1), we show that both the ZoSS generalization error and learning rate are independent of d and K, similar that of SGD [1. Theorem 3.8). This property guarantees efficient generalization even with two function evaluations.
In the full information regime (i.e., when the number of function evaluations K grow to oo), the ZoSS generalization bounds also provide guarantees for SGD by recovering the results in prior work [1]. Further, we derive novel SGD bounds for unbounded nonconvex losses, well as mini-batch SGD for any batch size. Our results subsume generalization guarantees for full-batch ZoSS and GD algorithms.

---

Generalization Error Bounds: ZoSS SGD
Algorithm
Bound
NC
UB
MB
ZoSS (this work)
1+ (C 1
((2 (22 c)C c (eT)
X
a S C/(tTk)
SGD, a S C/L
1+ (CB) (2C1.2) टÓगर (eT)
CB CPFT
x
Hardt et al. [1]
3e(1 + (CB) (1 + (2 c)C
ZoSS (this work) 04 S C/1
2n (independent of both d and K)
X
ZoSS (this work)
log 1+ C2
+ c)CL²
x
d
T8y / 1ad-11/R
SGD, Or C/T Hardt et al. [1]
2C
X
70SS (this work) 04 $ C/(TTR
(2 + (e
d
nz
(2 + c)C22
ZoSS (this work)
/
04 S ing(I+CC)
TBr
(proper choice of previous bound)
ZoSS (this work)
(2 (e min(C + 8-1, (C'oog()))
a: S ((((k)))
Table A list of the generalization error bounds developed herein for ZoSS (Eq. 6) in comparison with SGD, with μ cLi* /n3(3 + d)3/2, for > O. the table, "NC" and "UB" stand for "nonconvex" and "unbounded", respectively. "MB" corresponds to the mini-batch algorithm and for any batch size, Also, a₁ denotes the stepsize of ZoSS/SGD, and T the total number of iterations.
2 Problem Statement
Given data A {= i.id samples =, from an unknown distribution D, our goal to find parameters w of a learning model such that w E arg min, R(w), where R(w) & Ez=plf(wu Z)]. Since the distribution D is not known, we consider the empirical risk
Rs(w) A
Σ f(n, 24), 1=1
and the corresponding empirical risk minimization (ERM) problem to find w E arg min (s(w). For (randomized) algorithm As with input S and output W A(S), the excess risk Cexcess is bounded by the sum of the generalization error 6gen and the optimization enror copt:
Cexcess Es.A [R()] R(w") (R,A(R(W) Rs(W)) + (Es.A (Rs.)( R(w")
(2)
Egen
Capt
To analyze and control Egen. prove uniform stability bounds which imply generalization [1. Theorem 2.2]. Specifically, if for i.i.d. sequences S² E Z that differ one entry, we have sup, (fA(S) z) - -f f(A(S") =)] Eaab. for some Estb > 0, then Egen Easb Because the loss L-Lipschitz, Glab may then be chosen as Lsup AAl)A(S) - A(S")]
Our primary goal in this work is to develop uniform stability bounds for gradient-free algorithm As of the form W:++1 = wy a, A/,... where A/ only depends on loss function evaluations. To achieve this without introducing unnecessary assumptions, we consider novel algorithmic stability error decomposition approach. fact, the stability error introduced time by As breaks down into the stability error of SGD and approximation error due to missing gradient information. Let G.(+)

---

and (- be the following SGD update rules
Gr(w) w - at a (w, zi,,, (((w) aa
(3)
under inputs S, 5" respectively, and let i, independently by the random selection rule of the algorithm, for all T. Similarly the notation G-) and (.) to denote the iteration mappings of As, i.c.
.2...,;11 n be a random index chosen uniformly and
(.w - a,ASw.r. w. (w(()) - a, w.ri,"
(4)
Then, as we also discuss later (Lemmaa 1), the iterate stability error (ww) (ww') of As, for any w' E Rd and for all at è - T, may be decomposed
(ww)) ((()) x Gi(w) (w'))) + ff 2,,) - Af + [V 24, - Df st (5)
where EGBstab denotes the gradient-based stability error (associated with SGD), and Eest denotes the gradient approximation error. We now proceed by formally introducing ZoSS.
3 Zeroth-Order Stochastic Search (ZoSS)
As gradient-free alternative of the classical SGD algorithm, we consider the ZoSS scheme, with iterates generated according the following (single-example update) rule
K f(W, pUg 21.) S(We, 2nd) Uk. U LL μ R* (6)
W4+1 W at K
p
where a₁ 2 is the corresponding learning rate (for the mini-batch update rule we refer the reader to Section 5). At every iteration 7oSS generates K i.i.d. standard normal random vectors U h = 1..... K, and obtains + 1 loss evaluations on perturbed model inputs. Then ZoSS evaluates smoothed approximation of the gradient for some μ O. In light of the discussion in Section 2, we define the ZoSS smoothed gradient step at time t as
f(w + HUt, 21,) f(w, 24,
As
KK w.2y
AAKK wiyk
1
3.1 ZoSS Stability Error Decomposition
To show stability bounds for 7oSS, we decompose its eтror into two parts through the stability enror decomposition discussed Section Under the ZoSS update rule, Eq. (5) holds by considering the directions Af w,ss, and Af w.,, according to ZoSS smoothed approximations (7). Then for any
(((w") (w") w, w' IRd, the iterate stability error G.(w) ( based error EGBstab and approximation error Eest.
ZoSS at f, breaks down into the gradient
The error term CGBstab expresses the stability error of the gradient based mappings [1. Lemma 2.4] and inherits properties related to the SGD update rule. The error 6ett captures the approximation error of the ZoSS smoothed approximation and depends K and μ. The consistency of the smoothed approximation with respect to SGD follows from lim 20,00 A
K, w. A Vf(w, 2) for all I and E Z. Further, the stability error is also consistent since lim y to (ceal 0. Later on, the ZoSS eтror decomposition in Eq. (5) together with a variance reduction lemma (Lemma 10), to derive exact expressions on the iterate stability error G,(w) G (w") for fixed K and μ > (see Lemma 1). Although in this paper derive stability bounds and bounds on the Cgens the excess risk Cexcess depends on both errors (gen and Cupt- In the following section, we briefly discuss known results on the Cope of zeroth-order methods, including convex and nonconvex losses.
3.2 Optimization Error in Zeroth-Order Stochastic Approximation
Convergence rates of the ZoSS optimization error and related zeroth-order variants have been extensively studied in prior works; see e.g., [14, 68]. For the convex loss setting, when K +

---

function evaluations are available and no other information regarding the loss is given, the ZoSS algorithm achieves optimal rates with respect to the optimization eπror Copt. Specifically, under the assumption of closed and convex loss, Duchi et al. [13] provided a lower bound for the minimax convergence rate and showed that Copt = ( 22(vd/Kg K), for any algorithm that approximates the gradient given K + 1 evaluations. In the nonconvex setting Ghadimi et al. [69,7 70] established sample complexity guarantees for the zeroth-order approach to reach an approximate stationary point.
4
Main Results
For our analysis, we introduce the same assumptions on the loss function (Lipschitz and smooth) as appears in prior work [1]. Additionally, we exploit the ex-xannsivve and o-bounded properties of the SGD mappings G-(-) and (-) in Eq. (3). The mappings Gr(-) and G-(() are introduced for analysis purposes due to the stability error decomposition given in Eq. (5) and no further assumptions or properties are required for the zeroth- ooderrrrderrreee update rules Ge(-) and (.) given in Eq. (4). The --expansivityy of G.(-) holds for and a; S 2/3 [1, Lemma 3.6]. Note that G-(-) is always g-bounded (ar Loe) [1, Lemma 3.3.].
+ Bog if the loss is nonconvex, and 17 = 1 if the loss is convex
4.1 Stability Analysis
We derive generalization error bounds through uniform stability. To study the stability of ZoSS, we apply a variance reduction lemma that we provide in Appendix A. Exploiting the variance reduction lemma, we show growth recursion lemma for the iterates of the ZoSS.
Lemma 1 (ZoSS Growth Recursion) Consider the sequences of updates {G.]E= and (&:)
Let Wp w/ be the starting point, Wt+1 = (we) and w'++1 = (((() for any E TT)...,T). Then for any we, wh E Idd and 2 0 the following recursion holds
{ (1 In+ar auy 3d-1 | w'll + uBa, (3 + d)3/2 ||w, ww'll + +2,,L1 + 13a; + dj3/2
(000)=(()) if,,() C)(.).
[[]] ((((),( (1((il
The growth recursion of 7oSS characterizes the stability error that it is introduced by the 7oSS update and according to the outcome of the randomized selection rule at each iteration. Lemma 1 extends growth recursion results for SGD in prior work [1, Lemma 2.5] to the setting of the ZoSS algorithm. If K + oo and μ + 0 (while the rest of the parameters are fixed), then 1xx + 1, and the statement recovers that of the SGD [1, Lemma 2.5].
Proof of Lemma 1. Let and S' be two samples of size n differing in only a single example, and let Go(),, G,(O) be the update rules of the ZoSS for each of the sequences S, S' respectively. First under the event & A (G+(-)= = (.(+)} (see Eq. (4)), by applying the Taylor expansion there exist vectors W and Wi with 7th coordinates in the intervals (we(2, w(()) pU!!) U (w(²) nU(() w(()) and k.t) (w) + pUli (w, (G) wy ( =u((t)
h. (())), respectively, such we find that for any Wn, w/ Rd
it is true that
Ge(wx) (((()) G.(w.)) GG((w()
wt - w/ at
f 24,) v vf(wi, zi.) US)U
(8)
K
Ut + at K
0t
" 2 J [w. 25, )1 wo= Ub}
Σ
"UIV
2 [w, 21 2.) Ub Uh
K
K
2 w
= W/: az Vilee zi,) (w/ - 0a
G(we)
G((w)==G(w()
[L. Definition 2.3]: An update rule G(-) is expaasive [G(w) - G(w"ll || S w/ww w'll for all
w, w/ E Rd. [f]]w- G(w)]l S a then is bounded.
5

---

K
K
U [w, zi, Nw- Ut) Us + at Σ
V2 U [(w, 2i, Mm. wi, U2) Us
K
2 k w
K K
2.)) - V 25,) U - (Vs(ur, 2i,) v Vi(u/y 21,)))
A=1
We find (9) by adding and subtracting at V (Ww,, 24,) and ar Vf(wl zi, in Eq. (8). Recall that U independent for all k < K, t S T and that the mappings G(+) and G'(+) defined in Eq. (9), are n-expansive. The last display and the triangle inequality give
E|| ((((;((w)) (e())ll
20 K
K
uB
3d
< (G(w)) G(ww)ll+
E
[
tarl
B VlllVf(wy f(wr, 24) - V Vf(ww, II (10)
2
K
1=1
K
S 5n/lure
will
20 K
4
E
[|
3d 1
3|10, w/ll
(11)
2
3d -
1
11 + Vo
3
||wr - w'll + pBar + d)3/2
(12)
where (10) follows from (9) and Lemma 10, and for (11) we applied the n-expansivve property of G(-) (see [1. Lemma 2.4 and Lemma 3.6) and the 3 8-smoothness of the loss function.? Finally (12) holds since the random vectors U ~ NN, Ia) are identically distributed for all & E (1. K} and E < (3 + 2)3/2. Eq. (12) gives the first part of the recursion.
Similar to (9), under the event Ef (Gx()) # ())-), we find
(.( - (w's)
-w 04 Vf Zix)
(w/ o Vf +())
G(wss)
('(w))
K
Q1
" U (w, Ub Ut
0g
U f(w, 24 (ww Ut) Uh
K
2
k=1
K
Vf(w, Vf( U
(V 2.,) V((w( ((16)
(13)
By using the last display, triangle inequality, Lemma 10 and 3-smoothness, we find (w,) (((w(l
20
3d
< ([((w) G'(w))ll
Σ
E
+ ag
E V ffw, 24))
- 21
K
2
K
20
K
uB
3d
< min(n, 136, + 20, +
|
2Lav
(14)
K
2
K
k=1
< 5. + 20, 11Ba (3 d/3/2
(15)
where (14) follows from the triangle inequality and L L-piischitz condition, while the upper bound on [[G(w,) (w()))l comes from [1. Lemma 2.4]. Finally, (15) holds since 17 > 1 for both convex and nonconvex losses, o, = La, and E (3 d)3/2 for all & {1...., K}. This shows the second part of recursion.
For sake of brevity, let I be an adapted stopping time that corresponds to the first iteration index that the single distinct instance of the two data-sets S, S' is sampled by ZoSS. For any to (0, 10. n) we define the event Est, A (I > ta) = (5t, = 0}. The next result provides the stability bound.
For all Z and W d it is true that 2 (w. 2) w s B.

---

Lemma 2 (ZoSS Stability Nonconvex Loss) Assume that the loss function f(.,2) ) is L-Lipschitz and B-smooth for all E Z. Consider the ZoSS algorithm (6) with final mnal-iterate estimates WT and Wi,, corresponding to the data-sets S,S', respectively (that differ in exactly one entry). Then the discrepancy Ör
A | Will, under the event Eor,' satisfies the inequality
2L ( rd + pB + dd)//2
T
at II (1+ (1- j=t+1
T
16 Edro.
1
Σ 1
(16)
The corresponding bound of Lemma 2 for convex losses is slightly tighter than the bound in (16). Since the two bounds differ only by a constant, the consequent results of Lemma 2 are essentially identical for convex losses as well, We provide the equivalent version of Lemma 2 for convex losses in Appendix B.
Proof of Lemma 2. Consider the events & A (G((+) = G)}}} and & & (G(() * G)( (see Eq. (4)). Recall that P(E,) 1/n and P(E)) 1/n for all T. For any to 2 0, direct application of Lemma gives
16 [Es..)
Ee, Ed.,, (P(E))
((EE) Es,
1 n
1 -
Elbc+1lEr, Eo + n
Elōet1 Eg Edta
3d-11
3d -1 ( 1- - - a + n
$
17 + 0&
K
K
2a4h L
I'd + p.Bay + d)3/2.
(17)
With Rg 2 (n + on3(TT K - 1) + (1 1 3 - 1))/n) solving the recursion in (17) gives
T
T
E6 r + 13(3 + d)3/2
Σ
a1
II
R
(18)
1=
j=
We consider the last inequality for nonconvex loss functions with functions with 7 = to derive Lemma and Lemma 11 respectively (Appendix B).
1 + Ba, and convex loss
4.2 Generalization Error Bounds
For the first generalization error bound, we evaluate the right part of the inequality (16) for decreasing step size and bounded nonconvex loss. Then the Lipschitz condition provides a uniform stability condition for the loss and yields the next theorem.
Theorem (Nonconvex Bounded Loss Decreasing Stepsize) Assume that the loss f(-,2) [0, 1] is L-Lipschitz and ß-smooth for all E Z. Consider the ZoSS update rule (6) with T the total number of iterates. at S C/tl' K for some (fixed) C > and for all t S T, and fixed
S cl k /n3(3 + d/3/2 for some c > 0. Then the generalization error of ZOSS is bounded by
légen| (2+c) o (ET')
ca caiI
eCC
(2 (2+c)
0541
max 1, 1,1+ (CB) BC
(19)
n
eT
((2 (1+ (1+ C)
C8 (eT) CPTI.
(20)
Inequality (19), tighter version of (20), provides meaningful bound in marginal cases, i.c., el
lim E [If f(W;, (2 + c)C22 max log (2 ( ) .1 010
(21)
By neglecting the negative term (19) we find (20), that the 7oSS equivalent of SGD [1, Theorem 3.8]. When K + xo and 0, then 1dd + 1, and the incqualities (19), (20) reduce to
7

---

generalization bound for SGD. Inequality (20) matches that of [1, Theorem 3.8), and (19) provides a tighter generalization bound for SGD as well. We show Theorem 3 in Appendix A.
Next, we provide a bound on the generalization error for nonconvex losses that comes directly from Theorem 3. In contrast Theorem 3, the next result provides learning rate and a generalization error bounds, both of which are independent of the dimension and the number of function evaluations.
Corollary 4 Assume that the loss function f(:,2) [0, 1] is L-Lipschitz and 3-smooth for all 2 Z. Consider the ZoSS update rule (6) with μ S cLi' K/ ///g/((a. (1 + d)s/22) T the total number of iterates, and a S C/t.for some (fixdd) C > O and all t I T. Then the generalization error of ZoSS is bounded by
Egenl s (BC) (1+ (2+ 3
(22)
2n
As consequence, even in the high dimensional regime d + oo, two function evaluations (i.e., K = 1) are sufficient for the ZoSS to achieve Égen (O(T/n), with the learning rate being no smaller than that of SGD. We continue by providing the proof of Theorem 3. For the proof of Corollary see Appendix A.3.
In light of Theorem 3 and Corollary 4, we observe that the over-fitting phenomenon occurs in the gradient-free approach similarly to gradient- based algorithms. For general nonconvex (and convex) losses under standard step-size choices, the generalization error increases with respect T. Further, the effect of 3 affects both the stability (similarly to SGD in prior work) of the algorithm and the error approximation of the ZoSS. If 3 is large, then the expected approximation error (due to limited function evaluations) is also large [15] and the dependence on smoothness is unavoidable in black- box learning. In our results, this is expressed through the Growth Recursion of ZoSS (Lemma 2). that involves both the stability and approximation eπтоr per iteration. However, a smaller step-size (a, = 1/2t,31 k) mitigates the effect of on the bound. We refer the reader to Appendix E for a unified analysis of the excess risk, that captures the over-fitting and under-fitting trade-off.
Additionally, the number of iterations T is considered to be fixed and known (as in prior works including on average and high probability results on generalization). This is reasonable and quite standard because given the theoretical results, we know beforehand the appropriate choices of T that provide good trade-off between generalization and optimization. A classical setting is that of a fixed step-size at 1/T with T vn, which provides the well known generalization error bound for SGD with order ((1/ nn), as appears in very recent and timely prior works [32, Section 3.1], [45].
In the unbounded loss case, we apply Lemma 2 by setting to O (recall that to is free parameter, while the algorithm depends on the random variable Z). The next result provides a generalization error bound for the ZoSS algorithm with constant step size. In the first case of the theorem, also consider the convex loss as a representative result, as we show the same bound holds for an appropriate choice of greater learning rate than the learning rate of the nonconvex case. The convex case for the rest of the results of this work can be similarly derived.
Theorem 5 (Unbounded Loss Constant Step Size) Assume that the loss f(,, z) is L- L-Lipschitz, B- smooth for all E Z. Consider the ZoSS update rule (6) with μ S cLT* / (n + dj3/2) ) for some e <>0 O. Let T be the total number of iterates and for any t < T,
f(.2) is convex for all 2 E Z and a I minnlog (1+ -
1),2/3). or
if f( 2) is nonconvex and a < log (1 + /T3/TTLL for C > 0 then
(2 12+c C)L2 Iggeul <
(23)
n
if f())) is nonconvex and a I C/TT k. for some C > 0, then
L2 (2 c) (e -
Iegenl
(24)
nB
For the proof of Theorem 5 see Appendix A.4. In the following, we present the generalization error of ZoSS for an unbounded loss with a decreasing step size. Recall that the results for unbounded nonconvex loss also hold for the case of convex loss with similar bounds on the generalization error and learning rate (see the first case of Theorem 5).
8

---

Theorem 6 (Unbounded Loss Decreasing Step Size) Assume that the loss f(:,2) is L-Lipschirz.
3-smooth for all 2 Z. Consider ZoSS with update rule (6), T the total number of iterates, 0i S C/ for all I S T and for some C' > O, and Jt S k/(n((3 + d)3/2) ) for some > 0.
Then the generalization error of ZoSS is bounded by
(2 + ()T CB
Eggnn|
min (C+B-,Clog(ee
(25)
71
For the proof of Theorem see Appendix A.5. Note that the constant C' is free and controls the learning rate. Furthermore, it quantifies the trade-off between the speed of training and the generalization of the algorithm. In the next section, we consider the ZoSS algorithm with a mini- batch of size m for which we provide generalization error bounds. These results hold under the assumption of unbounded loss and for any batch size m including the case m
5 Generalization of Mini-Batch ZoSS
For the mini-batch version of ZoSS, at each iteration t, the randomized selection rule (uniformly) samples a batch Jr of size m and evaluates the new direction of the update by averaging the smoothed approximation A If pK. wzz
over the samples 2 E Je as
K f(w + pU 2J.,) - f(w,2Je,)
A w
A
KKq.UU"
w.Je
mK
1
Uk
(26)
=1
where k.s NO, Ia) are i...d (standard normal), and μ Rt The update rule of the mini- Wa.J. for all t S T, and we define Go, (w) - w - a, S W. (Gp((n) - w - af for Jr C S and Jú C respectively. Due to space limitation, we refer the reader to Appendix for the detailed stability analysis of ZoSS with mini-batch. Specifically, we prove growth recursion lemma for the mini-batch ZoSS updates (see Appendix C.1 for proof).
batch ZoSS is W4+1 = W, a,Af
w.J;
Lemma 7 (Mini-Batch ZoSS Growth Recursion) Consider the sequences of updates {Õs, JT=1 and {G', Jt=1 {G= and μ S cLT k/ (n3333 d)3/2). Let wa w/ be the starting point, W++1 = G2, (w)) /
w/g E RRd and 2 0 the following and w = G3 (w() for any E (1....,T) T}. Then for any we, recursion holds
(1 k dz cLos e
(-),()=0k) G2 if ff + G)
B[I[G, - G',,(' (w() S
(1 m-1 I 8, + 2 rd +
eLorr
n
Although the iterate stability error (at time t) in the growth recursion depends on the batch size m under the event (G,, (-) z 8% (-),, the stability bound on the final iterates is independent of m, and coincides with the single example updates (m 1, Lemma 2). Herein, we provide an informal statement of the result.
Lemma (Mini-Batch ZoSS Stability Nonconvex Loss) Consider the mini-batch ZoSS with any w batch size m S n, and iterates W+11 W, - a AKK Wie.. W Wi - aI p for all t < T,
with respect to the sequences S, Then the stability error dy sutisfies the inequality of Lemma 2.
We refer the reader to Appendix Section C.I, Theorem 14 for the formal statement of the result, Through the Lipschitz condition of the loss and Lemma 8, we show that the mini-batch ZoSS enjoys the same generalization error bounds as the case of single-query ZoSS (m = 1). As consequence, the batch size does not affect the generalization error.
Theorem (Mini-batch ZoSS Generalization Error) Let the loss function f(.,2) 2) be L-Lipschitz
and Smoohh (possibly nonconvex, possibly unbounded) for all 2 E Z. Then the bounds of Theorem
5 and Theorem 6 hold for the mini-batch ZoSS with iterate W W. ar A,A We.Je for all t < and any batch size m < n.
gK.z
2 As in the single-ques query (m = 1) ZoSS. under the assumption of convex loss, the stability error of mini-batch ZoSS satisfies the inequality (46). Appendix B, Lemma 11.

---

By letting K + oo and c - 0, the generalization error bounds of mini-batch ZoSS reduce to those of mini-batch SGD, extending results of the single-query (m 1) SGD that appeared in prior work [1]. Additionally, once K + 00, - 0 and m = n we obtain generalization guarantees for full-batch GD. For the sake of clarity and completeness we provide dedicated stability and generalization analysis of full-batch GD in Appendix D, Corollary 15.
6 Discussion: Black-box Adversarial Attack Design and Future Work
A standard, well-cited example of ZoSS application is adversarial learning as considered in [5], when the gradient is not known for the adversary (for additional applications for instance feder- ated/reeiffrreemett learning, linear quadratic regulators; see also Section for additional references). Notice that the algorithm in [5] is restrictive in the high dimensional regime since it requires 2d function evaluations per iteration. In contrast, ZoSS can be considered with any K evaluations (the trf is between accuracy and resource allocation, which is also controlled through K). If K d + evaluations are available we recover guarantees for the deterministic zeroth-order approaches (similar to [5]).
functions
Retrieving large number of function evaluations often is not possible in practice. When limited amount of function evaluations is available, the adversary obtains the solution (optimal attack) with an optimization error that scales by a factor of v /d/ K, and the generalization error of the attack is of the order vT/ /n under appropriate choices of the step-size, the smoothing parameter μ and K. Fine tuning of the these parameters might be useful in practice, but in general K should be chosen as large as possible.
contrast, μ should be small and satisfy the inequality μ S cLIv% /n, - d)3/2 (Theorem 6). For instance, in practice is often chosen between 10 and 10 (or even lower) and the ZoSS algorithm remains (numerically) stable.
For neural networks with smooth activation functions [71-73], the ZoSS algorithm does not require the smoothness parameter be necessarily known, however is large then the guarantees of the estimated model would be pessimistic. To ensure that the learning procedure is successful, the adversary can approximate 3 (since the loss not known) by estimating the (largest eigenvalue of the) Hessian through the available function evaluations [74, Section 4.1].
Although the non-smooth (convex) loss setting lies out of the scope of this work, it is expected to inherit properties and rates of the SGD for non-smooth losses (at least for sufficiently small smoothing parameter μ)). In fact, [45, page 3, Table 1] developed upper and lower bounds for the SGD in the non-smooth case, and they showed that standard step-size choices provide vacuous stability bound. Due to these inherent issues of non-smooth (and often convex only cases), the generalization error analysis of ZoSS for non-smooth losses remains open. Finally, cooommttion-theorede generalization error bounds of ZoSS can potentially provide further insight into the problem, due to the noisy updates of the algorithm, and consist part of future work.
7 Conclusion
In this paper, we characterized the generalization ability of black-box learning models. Specifically, we considered the Zeroth-order Stochastic Search (ZoSS) algorithm, which evaluates smoothed approximations of the unknown gradient of the loss by only relying on K loss evaluations. Under the assumptions of a Lipschitz and smooth (unknown) loss, we showed that the ZoSS algorithm achieves the same generalization error bounds as that of SGD, while the learning rate is slightly decreased compared to that of SGD. The efficient generalization ability of ZoSS, together with strong optimality results related to the optimization error by Duchi al. [13], makes a robust and powerful algorithm for a variety of black-box box learning applications and problems.
References
[1] Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochas- tic gradient descent. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1225-1234, New York, New York, USA, 20-22 Jun 2016. PMLR. URL: https: / http:///pccccece...///e/ mlr www.rrppees/vvee pwwss//4s//eeeeee hww.t16.d11htt
10

---

[2] Alekh Agarwal, Dean P Foster, Daniel J Hsu, Sham M Kakade, and Alexander Rakhlin. Stochastic convex optimization with bandit feedback. In J. Shawe-Taylor, R. Zemel, Bartlett, F. Pereira, and K. Q. Weinberger, editors, Advances in Neural Information Processing Syx- tems, volume 24. Curran Associates, Inc., 2011. URL:
paper/2011 / /1t.///0000000611666333333333330 PateeeeeeeeeeescSSaoaeeeeeeeeeee .df.
[3] Ohad Shamir. An optimal algorithm for bandit and zero-order convex optimization with two- point feedback. The Journal of Machine Learning Research, 18(1):1703-1713, 2017. URL:
https://www.jjp www.:jml...roo org/ wwwpg/appes//vvee wttperrs/oolume///111/ wwwllmm1//16-62//11111
[4] Arya Akhavan, Massimiliano Pontil, and Alexandre Tsybakov. Exploiting higher or- der smoothness in derivative-free optimization and continuous bandits. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neu- ral Information Processing Systems, volume 33, pages 9017-9027. Curran Asso- ciates, Inc., 2020. URL: https: http://oocoeiin.meerrre.ae httppps..c//ppp/////////// 1ttpbbbbbba11aaacc1dd11100001111
[5] Pin- Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh. Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models. In Proceedings of the 10th ACM workshop on artificial intelligence and security, pages 15-26,
2017. URL: https: http//dllccccomo// org/ htt///oi///// /oo//o....1./// 100.1145312200 21118877.2.1444...
[6] Zan Li and Li Chen. Communication-efico
decentralized zeroth-order method on heteroge- neous data. In 2021 13th International Conference on Wireless Communications and Signal Processing (WCSP), pages 1-6, 2021, doi: 1tt:11001111SP55555
2021.9611111111111 3620.
[7] Anirudh Vemula, Wen Sun, and J. Bagnell. Contrasting exploration in parameter and action space: A zeroth-order optimization perspective. In Kamalika Chaudhuri and Masashi Sugiyama, editors, Proceedings of the Ewenty-Second International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pages 2935.2935. PMI.R, 16-18 Apr 2019. URL: https: http://prreeeeen....r.peeeeeeeee www.g.s.....g.
v89/ www.walla.aaaae..e.eeeaaem
[8] Harshat Kumar, Dionysios S. Kalogerias, George J. Pappas, and Alejandro Ribeiro. Actor- only deterministic policy gradient via zeroth-order gradient oracles in action space. In 202 / IEÉE International Symposium on Information Theory (ISIT), pages 1676-1681, 2021. doi:
[9] Dhruv Malik, Ashwin Pananjady, Kush Bhatia, Koulik Khamaru, Peter Bartlett, and Martin Wainwright. Derivative-free methods for policy optimization: Guarantees for linear quadratic systems. In Kamalika Chaudhuri and Masashi Sugiyama, editors, Proceedings of the Twenty- Second International Conference on Artificial Intelligence and Statistics, volume 89 of Pro- ceedings of Machine Learning Research, pages https: https:///rroooe http:///pooeeees..n/i/ mlr www.:mess.....et.eee.mm.m...eeee wwwtttrrrssseeeeaaeeeeee https////mmmmm1m..amaaaaa
PMLR, 16-18 Apr 2019. URL:
[10] Hesameddin Mohammadi, Mahdi Soltanolkotabi, and Mihailo R. Jovanović. On the linear convergence of random search for discrete-time LQQ. IEEE Control Systems Letters, 5(3:9889- 994, 2021. doi: 10. 1119/1199991 20020.3006266
[1I] Jeremy Rapin and Olivier Teytaud. Nevergrad A gradient-free optimization platform. https:
2018.
[12] Luis Miguel Rios and Nikolaos V Sahinidis. Derivative-free optimization: review of algorithms
and comparison of software implementations. Journal of Global Optimization, 1643)11247- 2013. URL: URL. http://nnnee.oeeartieeeee//////e
s10898- 012-9911-1.
[13] John C. Duchi, Michael I. Jordan, Martin J. Wainwright, and Andre Wibisono. Optimal rates for zero-order convex optimization: The power of two function evaluations. IEEE Transactions on Information Theory, 61(55:2788-228) 2806. 2015. doi: 10.
[14] Yurii Nesterov and Vladimir Spokoiny. Random gradient-free minimization of convex func- tions. Foundations of Computational Mathematies, 17(2):527-566 2017. URL: https:
//1ink.springer. wwtpprinngr.cooooooeoo
1ttt:///1..100//10000000 015 0110000000-119999-----19
[15] Krishnakumar Balasubramanian and Saeed Ghadimi. Zeroth-order nonconvex stochastic opti- mization: Handling constraints, high dimensionality, and saddle points. Foundations of Com- putational Mathematics, pages 1-42, 2021. URL: httpp::///ii..sppiine.... wttpg/err..oo//ttiir.eee
10.1007/s 1000000/s12080000 0111112200000000-811101 021-0099----999-8...88
11

---

[16] Mahdi Haghifam, Gintare Karolina Dziugaite, Shay Moran, and Dan Roy. To- wards a unified information-theoretic framework for generalization. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 26370-26381. Curran As- sociates, Inc., 2021. URL: stttps:/pproceednngg mewriss. wtt.icissscccccccc ca/ pecpppprr/222//E1e
[17] Olivier Bousquet and André Elisseeff. Stability and generalization. The Journal of Ma- URL:
chine Learning Research, 2:499-526, 2002.
org/ www.llrrrrpppprrs////eee
bousquet022.htmm.
[18] Shai Shalev-Shwartz, Ohad Shamir, Nathan Srebro, and Karthik Sridharan. Learnability, stability and uniform convergence. The Journal of Machine Learning Research, 11:2635-26700 2010. URL: https://www.jnlr. http:/it..r.pror//////eaeeeeeeee shwartz10a.po
papers/ saaerssvolumee hwte/ooowllllleeeeeeeeeeeeeeeeee wwwllmeee/saaaeeeeeeeeeeeee wtwsaaaaeeeesaaatelllooo saaatteeeaaaaaaaeeeeeaaaaa
[19] Luc Devroye and al. Distribution-free performance bounds for potential function rules,
1979. URL: https: citeseerx.sst.su.. wwt/vidwdvvveeeeeeeeeeeeeeeeeeee wwwwwoc//uuaary/aaa////// /uwmaaay/doi11........ 11..:.1111....................1. 1...0.....18..777..
[20] Michael Kearns and Dana Ron. Algorithmic stability and sanity-check bounds for leave-
one-out vrlss-valiiation. Neural Computation, 11(6):1427-1453, 1999. do1:10.1162/
089976699300016004
[21] Sayan Mukherjee, Partha Niyogi, Tomaso Poggio, and Ryan Rifkin. Learning theory: stability is sufficient for generalization and necessary and sufficient for consistency of empirical risk minimization. Advances in Computational Muthematics, 25(1):161-193, 2006. URL: https:
//1ink.ppringer woom/articleee 0ttp:/00011000000040000000003330
[22] Olivier Bousquet and André Elissefff. Algorithmic stability and generalization performance. In T. Leen, T. Dietterich, and V. Tresp, editors, Advances in Neural Information Processing Systems, volume 13. MIT Press, 2001. URL: /ttptts:/ppooeeeeeteeeee ing.indinnnnnnn..e www.rrceeednnnnnnnii.........m paper/ 2000/f1le/49ad23te aat.aaadd....eaaaaaaeeee0000ee..
påf.
[23] Zhun Deng. Hangfeng He, and Weijie Su. Toward better generalization bounds with locally elastic stability. In Marina Mcila and Tong Zhang, editors, Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages -500. PMLR, 18-24 Jul 2021. deng21b.html.
URL: https: http://poooeeeeeeeeeeeee mlr press/v139/
[24] Yiding Jiang*, Behnam Neyshabur*, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic generalization measures and where to find them. In International Conference on Learning Representations, 2020. URL: htttt::://peeerviieee www.nnet/forummme../ formm www.orrmddssss SJg JBFvH.
[25] Raef Bassily, Kobbi Nissim, Adam Smith, Thomas Steinke, Uri Stemmer, and Jonathan Ullman. Algorithmic stability for adaptive data analysis. In Proceedings of the Forty-Eighth Annual ACM Symposium on Theory of Computing, STOC '16, page 1046-1059, New York, NY, USA, 2016. Association for Computing Machinery. doi : 1111111//2897511 2897556...
[26] Jonathan Ullman, Adam Smith, Kobbi Nissim, Uri Stemmer, and Thomas Steinke. The limits of post-selection generalization. In S. Bengio, H. Wallach, H. Larochelle. K. Grauman, N. Cesa- Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018 URL: https: http://wweeennngeeeee.e..e.eeeee htt.//baeeeeee000000066662621133
[27] Christopher Jung, Katrina Ligett, Seth Neel, Aaron Roth, Saeed
and Moshe Shenfeld. New Analysis of Differential Privacy's Generalization Guarantees (Invited Paper), page 9, Association for Computing Machinery, New York, NY, USA, 2021. URL: https: //doi.org/ /0///11.110 11.1101.44 /111/11444433 33655358....
[28] Thomas Steinke and Lydia Zakynthinou. Reasoning About Generalization via Conditional Mu- tual Information. In Jacob Abemnttyy and Shivani Agarwal, editors, Proceedings of Thirty Third Conference on Learning Theory, volume 125 of Proceedings of Machine Learning Research, pages 3437-3452. PMI.R, 09-12 Jul 2020. URL: https: http::s:////coeednnneeeee mlr wtt.rr.../ssv//.// steinke20a.htmm
[29] Ibrahim Alabdulmohsin. An Information-Theorre Route from Generalization in Expectation to Generalization in Probability. In Aarti Singh and Jerry Zhu, editors, Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, volume 54 of Proceedings
12

---

of Machine Learning Research, pages 92-100. PMLR, 20-22 Apr 2017. URL: https:// http://m....ep/////////oooooaaoa
[30] Aolin Xu and Maxim Raginsky. Information-thhoortic analysis of generalization capabil- ity of learning algorithms. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Sys- tems, volume 30. Curran Associates, Inc., 2017. URL: https: http://prrreeii..eeuurrrrra paper/2017/filee / 7688e446111 P pdf.
[31] Yuheng Bu, Shaofeng Zou, and Venugopal V. Veeravalli. Tightening mutual information based bounds on generalization error. In 20/9 IEEE International Symposium on Information Theory (ISIT), pages 587-591, 2019. doi: 10.1100.11911 20099/ISII.20000 AS1T.2019.8849590.
[32] Gergely Neu, Gintare Karolina Dziugaite, Mahdi Haghifam, and Daniel M. Roy. Information- theoretic generalization bounds for stochastic gradient descent. In Mikhail Belkin and Samory Kpotufe, editors, Proceedings of Thirty Fourth Conference Learning Theory, volume 134 of Proceedings of Machine Learning Research, pages 3526-3545. PMLR, 15-19 Aug 2021. URL: https://proceeddi: www.iing.mm...... wtt.g//wwr..r.es///ee
[33] Hrayr Harutyunyan, Maxim Raginsky, Greg Ver Steeg, and Aram Galstyan. Information- theoretic generalization bounds for black-box learning algorithms. In M. Ranzato, A. Beygelzimer. Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 24670-24682. Curran As- sociates, Inc., 2021. URL: https: attpps://proceddingg. www.iinssnnuuis....m ctwpippscccppppeeeeee pttppppppe//////////// (aa00a29661.61337122253300000000
[34] Fengxiang He, Shaopeng Fu, Bohan Wang, and Dacheng Tao. Robustness, privacy, and generalization of adversarial training. arXiv preprint arXiv:2012. 13573, 2020. URL.: https:
/ //axxiv.rrgaas//////o// /otpbb//2222227777
[35] Fengxiang He, Bohan Wang, and Dacheng Tao. Tighter generalization bounds for iterative differentially private learning algorithms. In Cassio de Campos and Marloes H. Maathuis, editors, Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence, volume 161 of Proceedings of Machine Learning Research, pages 802-812. PMLR, 27-30 Jul
2021. URL: https: htttps:/pproceednng wwwpeeiiinnnmmm.....mmmmmemmm.em wwwpppsseee//eeeeeeaaaer
[36] Zhenhuan Yang, Yunwen Lei, Siwei Lyu, and Yiming Ying. Stability and differential pri- vacy of stochastic gradient descent for pairwise learning with non-smooth loss. In Arindam Banerjee and Kenji Fukumizu, editors, Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pages 2026- 2034. PMLR, 13-15 Apr 2021. URL: https: / /ttp///ecccee...cc/oo .wr. wtt..//esss///// yang21c.html.
[37] Puyu Wang, Zhenhuan Yang. Yunwen Lei, Yiming Ying, and Hai Zhang. Differentially private empirical risk minimization for auc maximization. Neurocomputing, 461:419-43 43799943332200 2021.
[38] Vitaly Feldman and Jan Vondrak. Generalization bounds for uniformly stable algorithms. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Gar- netr, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL: Pateeedebbbbdd-PPae...oee pdf.
[39] Vitaly Feldman and Jan Vondrak. High probability generalization bounds for uniformly stable algorithms with nearly optimal rate. In Alina Beygelzimer and Daniel Hsu, editors, Proceedings of the Thirty-Second Conference on Learning Theory, volume 99 of Proceed- ings of Machine Learning Research, pages 1270-1279. PMLR, 25-28 Jun 2019. URL: attps://proceedings.. mml..ng...m.r...... /tt.eeeiiiinn.....prrsse///////e wwwpe//eeemaannnmeemeeee htm.
[40] Liam Madden, Emiliano Dall' Anese, and Stephen Becker. uniform stability bounds for nonconvex stochastic gradient descent. arXiv e-prints, pages
High probability convergence and
arXiv-2006, 2020. URL: https: /tt.:tps///axxx..orrr./o/ wttp://www..r.ab//0000000550
[41] Yegor Klochkov and Nikita Zhivotovskiy. Stability and deviation optimal risk bounds with convergence rate of (//n.. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 5065- 5076. Curran Associates, Inc., 2021. URL: https: httpttpsss//.oooednngs...oo
13

---

[42] Iija Kuzborskij and Christoph Lampert. Data-dependent stability of stochastic gradient de- scent. In Jennifer Dy and Andreas Krause. editors. Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2815-2824. PMLR, 10-15 Jul 2018. URL: https://proceedinng mlr wallrppres///e /v8/
kuzborski j18.html..
[43] Robert Mansel Gower, Nicolas Loizou, Xun Qian, Alibek Sailanbayev, Egor Shulgin, and Peter Richtárik. SGD:
General analysis and improved rates. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 5200-5209. PMLR, 09-15 Jun 2019. URL: https: http::///ppoceeding....om mlr. wttp./pprees/s//seii.aee/ss/
[44] Vitaly Feldman. Generalization of erm in stochastic convex optimization: The dimen- sion strikes back. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Gar- nett, editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL: attps::/prooeeddn..nnerrs httpar/ppppppppppp/////////////e 0ww.:344555990557777771111122110
[45] Raef Bassily, Vitaly Feldman, Cristóbal Guzmán, and Kunal Talwar. Stability of stochastic gradient descent on nonsmooth convex losses. In H. Larochelle, M. Ranzato, R. Hadsell, M. E. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1-39-. Curran Associates, Inc., 2020. URL: htttp:::/ppocceeeiiiiiee wtwp://eeddinnnnunis........ pdf.
paper/20200 file/ / /tt//22222222eee 2eeee//cecceee271222222
[46] Yunwen Lei and Yiming Ying. Fine-graines analysis of stability and generalization for stochastic gradient descent. In Hal Daumé III and Aarti Singh, editors. Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 5809-5819. PMLR, 13-18 Jul 2020. URL: https:
/ //proceedings.it w.r.........p........ prwss/ 11111999ee12.hhhaaee
[47] Yunwen Lei, Ting Hu, and Ke Tang. Generalization performance of multi-pass stochastic gradient descent with convex loss functions. The Journal of Machine Learning Research, 22(25): 41, 2021. URL: /ttp:// www.toorgg.//p..eeeeee httpp/ppppss///// wetpss//222/99//e 716.html.
org/
[48] Yunwen Lei, Mingrui Liu, and Yiming Ying. Generalization guarantee of SGD for pairwise learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 21216-21228 Curran Associates, Inc., 2021. URL: https::/prooeeediggs... http:///gnnnniiiis.cccp///////1o Paper....
[49] Yunwen Lei, Antoine Ledent, and Marius Kloft. Sharper generalization bounds for pair- wise learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 21236-21246. Cur- ran Associates, Inc., 2020. URL: http://///occcoe.ooooo. httpurippp.copppe/2/20000/0/ httpp/pppr///// /220:/200///ille 1a117388ee..ccb667331bb3331311/1
Paper..df.
[50] Yi Zhou, Yingbin Liang, and Huishuai Zhang. Understanding generalization error of SGD in nonconvex optimization. Machine Learning, pages 1-31, 2021. URL: https:
/
[51] Ali Ramezani Ramezani-Kebrya Kebrya, Ashish Khisti, and Ben Liang. On the generalization of stochastic gradient descent with momentum. arXiv preprint arXiv:2 / 1353,, 2021.
[52] Puyu Wang, Liang Wu, and Yunwen Lei. Stability and generalization for randomized coor- dinate descent. arXiv preprint arXiv:2108.07414, 2021. 2108.07414.
URL: bertps://arxiv.ort
[53] Pan Zhou, Hanshu Yan, Xiaotong Yuan, Jiashi Feng, and Shuicheng Yan. Towards understanding why lookahead generalizes better than sgd and beyond. In M. Ran- zato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Ad- vances in Neural Information Processing Systems, volume 34, pages 27290-27304. Cur- ran Associates, Inc., 2021. URL: https: httpttpppp//ppoooccnnnn..oo www.inns...uiis.ccco wttprrppp..cppppp////////// 35500277828874455bb55..00d..
[54] Yue Xing, Qifan Song, and Guang Cheng. On the algorithmic stability of adversarial training.
In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors,
14

---

Advances in Neural Information Processing Systems, volume 34, pages 26523-26535. Cur- ran Associates, Inc., 2021. URL:
www.ipe.cr.pppeeeeeeee wttppc/cpppppp///////e.em .df.
df
[55] Ankit Pensia, Varun Jog, and Po-Ling Loh. Generalization error bounds for noisy, iterative algorithms. In 2018 IEEE International Symposium on Information Theory (ISIT), pages 546-550, 2018, doi 1ad::011109/IIII.... 111.:/11...111117777777
[56] Wenlong Mou, Liwei Wang, Xiyu Zhai, and Kai Zheng. Generalization bounds of SGLD for non-convex learning: Two theoretical viewpoints. In Sébastien Bubeck, Vianney Perchet, and Philippe Rigollet, editors, Proceedings of the 31st Conference On Learning Theory, volume 75 of Proceedings of Machine Learning Research, pages 605-638, PMLR, 06-09 Jul 2018. URL: htt////mmm.aaa.....mm
https://proceediin mmr..ng.....r...mm. wwtpp/ppsssspeppeeepe pttp://ppppoooooaaaoo..eeeeeea/a
[57] Jian Li, Xuanyuan Luo, and Mingda Qiao. On generalization error bounds of noisy gradient methods for non-convex learning. In International Conference on Learning Representations, 2020. URL: httpps:/openreeieett
[58] Jeffrey Negrea, Mahdi Haghifam, Gintare Karolina Dziugaite, Ashish Khisti, and Daniel M. Roy. Information-theerrtic generalization bounds for SGLD via data-dependent estimates. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buc, E. Fox, and R. Gar- nett, editors. Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL: https: http:::://proceediggg. wwwprrreeeniinnn...r.r.........p httpppscccppppp//119919 0aaa11777e6377033700444422202020 Paper.....................
[59] Yikai Zhang, Wenjia Zhang, Sammy Bald, Vamsi Pritham Pingali, Chao Chen, and Mayank Goswami. Stability of SGD: Tightness analysis and improved bounds. In The 38th Conference on Uncertainty in Artificial Intelligence, 2022. URL: https: http://ppprrreee.teeotorttott
S1 2m12008j5lq.
[60] Tyler Farghly and Patrick Rebeschini. Time-independent generalization bounds for SGLD in non-convex settings. In M. Ranzato, A. Beygelzimer, Dauphin, P.S. Liang, and Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 19836- 19846. Curran Associates, Inc., 2021. URL: 2021/file/ /
wwwp:r/wwweendn..norrr.......... .df.
[61] Bohan Wang, Huishuai Zhang, Jieyu Zhang, Qi Meng, Wei Chen, and Tie-Yan Liu. Op- timizing information--theoreiee generalization bound via anisotropic noise of SGLD. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 26080-26090. Cur- ran Associates, Inc., 2021. URL: https://prroceedinga wtt.ggg.nuuriiscccoooom httppp/ccc/pppp///////111
[62] Hao Wang, Yizhe Huang, Rui Gao, and Flavio Calmon. Analyzing the general- ization capability of SGLD using properties of Gaussian channels. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 14222-24233 Curran As- sociates, Inc., 2021. URL: attps:// /tt7:6/f6aaaaaac000000000..2.22.
[63] Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: clos- ing the generalization gap in large batch training of neural networks. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett, editors. Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL: https: http:::s/ppoccedingg www.rr/ceeiinnnnuuuiis....mm A5e0ff60beeb08456fc777 www.ccfeeeeee2aa.e Papee...
[64] Zachary Charles and Dimitris Papailiopoulos. Stability and generalization of learning algo- rithms that converge to global optima. In Jennifer Dy and Andreas Krause, editors, Pro- ceedings of the 35th International Conference on Machine Learning, volume 80 of Pro- ceedings of Machine Learning Research, pages 745-754. PMLR, 10-15 Jul 2018 URL: https: httpp:://poeediiggll..peeeeee httpa///////caaeeeeeeeeeeae chwrreaarraaeee htm...mm..
[65] Idan Amir, Yair Carmon, Tomer Koren, and Roi Livni. Never go full batch (in stochastic convex optimization). In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman
15

---

Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 25033- 25043. Curran Associates, Inc., 2021. URL: atgp wttp:iis...rrr....c..eaeeaap
pdf.
[66] Idan Amir, Tomer Koren, and Roi Livni. SGD generalizes better than GD (and regularization doesn't help). In Mikhail Belkin and Samory Kpotufe, editors, Proceedings of Thirty Fourth Conference on Learning Theory, volume 134 of Proceedings of Machine Learning Research, pages 63-92. PMLR, 15-19 Aug 2021. URL:
mlr press/v134)
amir21a.html.
[67] Dominic Richards and IIja Kuzborskij. Stability &amp; generalisation of gradient de- scent for shallow neural networks without the neural tangent kernel. In M. Ran- zato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Ad- vances in Neural Information Processing Systems, volume 34, pages 8609-8621, Curran Associates, Inc., 2021. URL: https: http://///oooeengs.eeeeeee www.reediing.euurs..c/////eooeee /tteerrppp.cpppper///////// htt.://ccbcceeecee66226666666660
Paper .dff.
[68] Yurii Nesterov. Random gradient-free minimization of convex functions. core discussion papers 2011001, université catholique de louvain. Center for Operations Research and Econometrics (CORE), 2011. URL: https / eco.papers wwwapprssrepeccoom repeccoog. paper/ corlouvco/ 201/2000100001 1001.htm.
[69] Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. Found Comput Math, 23:35-76, 2022.
URL: https: httpp::/dddo.orrg.
10.1007/s10208- 020-0-0-22--999----
[70] Saeed Ghadimi, Guanghui Lan, and Hongchao Zhang. Mini-bated stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming, 155(1- 2):267-305, 2016. URL: https: pettps://innee /tti/ini.sspprr. wwillksppnnnggee wwwpinnr..ooo////e.e.e.. http//onnee////ee /ptttt//dff//// / 10. 1007/s10100 014-0846-1122
[71] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint www.oorgg////...oo/ooo 08415.
1rX1::1666.084115 2016. URL:
[72] Prajit Ramachandran, Barret Zoph, and Quoe V Le. Searching for activation functions. arXiv preprint 1rXXiv1710.059// 2017. URL: https: /ttppi/////xxxvorrr/.// abs 1710.0000 10.05991.
[73] Hao Zheng, Zhanlei Yang. Wenju Liu, Jizhong Liang, and Yanpeng Li. Improving deep neural networks using softplus units. In 2015 International Joint Conference on Neural Networks (IJCNN), pages 1-4, 2015. doi: 10.1109..99 10119999ICCC.... 2.JCN.20015... 2...22011...2..... 7280459.
[74] Krishnakumar Balasubramanian and Saeed Ghadimi. Zeroth-order nonconvex stochastic opti-
mization: Handling constraints, high dimensionality, and saddle points. Foundations of Compu- tational Mathematics, 22(1):35-76, 2022. URL: https: http://oo......11/////////////// 11/1110./1/11180005888
[75] Moritz Hardt, Benjamin Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. arXiv preprint arXiv:1509.0122 2015.
URL: https://
arxiv.org/ar abs/1509.00 00200.0
[76] Robert M Corless, Gaston H Gonnet, David EG Hare, David Jeffrey, and Donald E Knuth. On the Lambert W function. Advances in Computational mathematics, 5(1):329-359) 1996. URL:
Acknowledgments and Disclosure of Funding
We would like to thank the four anonymous reviewers for providing valuable comments and sugges- tions, which have improved the presentation of the results and the overall quality of our paper.
Amin Karbasi acknowledges funding in direct support of this work from NSF (ISS-1845032) ONR (N00014- 19-1-2406), and the AI Institute for Learning-Enaie Enabled Optimization at Scale (TILOS).
16

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See abstract and introduction.
(b) Did you describe the limitations of your work? [Yes] See Section 4 for discussions on the assumptions adopted, which are standard and common the related literature.
(c) Did you discuss any potential negative societal impacts of your work? [N/A] This paper presents theoretical contributions of generic nature and broad applicability.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full assumptions of all theoretical results? [Yes] See Table as well as all statement of our results.
(b) Did you include complete proofs of all theoretical results? [Yes] All results include detailed proofs, either the paper or in the supplement.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [N/A]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [N/A]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [N/A]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you't 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
17


Reviews:
[{"review_id": "hbx9pC3bYDm", "reviewer": "Reviewer_HbvR", "summary": "This paper provides generalization bounds for the ZoSS algorithm, based on the analysis of algorithmic stability. The bounds match the previous work of HRS16 on SGD. The main technical ingredient is a recursion lemma which controls how fast the expected distance between two ZoSS instances grow, when they differ on only one example. This result is then extended to a few different scenarios.", "questions": "What's 'high-probability analysis' (line 43)? Is it a typo?\n\nIs it possible to achieve similar results for one-point zeroth-order algorithms? For example, the 'gradient descent without a gradient' method of FKM04.", "limitations": "The authors addressed some limitations ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The method is similar to the analysis of SGD (HRS16) in spirit, and the achieved bound is not very surprising. Still, it's good to see similar generalization bounds can be obtained for zeroth-order optimization (though it shares the same drawbacks of HRS16 as well, such as the linear dependence on $T$).\n\nThe recursion lemma is interesting.\n\nThe writing is good and the paper is easy-to-follow in general.\n\nThe title seems a bit unclear and misleading to me. I suggest change it to be more specific to avoid confusion (like 'generalization bounds for zeroth-order optimization via algorithmic stability'?). The analysis also applies only to ZoSS algorithms and doesn't seem to be able to cover all zeroth-order algorithms.\n\nTo conclude, the results of this paper are solid and meaningful in that it provides the first generalization analysis for zeroth-order optimization. But it's felt that there isn't enough technical depth (incremental over HRS16) and the paper is a bit over-selling.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Tx2uc6aqw0I", "reviewer": "Reviewer_tUYw", "summary": "The paper establishes generalization error bounds of the gradient-free analogue of the SGD---zero-order stochastic search (ZoSS)---in which the gradient of the loss function at a point is approximated by $K + 1$ values ​​of the function in the vicinity of this point. The cases of nonconvex and bounded, convex and unbounded, nonconvex and unbounded loss functions are considered. However, in all cases, the authors require that the loss function be Lipschitz and smooth.", "questions": "How one shall choose the values of $K$ and $\\mu$ in practice?", "limitations": "I think that the requirement of Lipschitz and smoothness is the main limitation of this work.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The bounds are a natural generalization of the bounds by Hardt et al. (2016) to the case of ZoSS. It is noteworthy that the order of the convergence rate of the ZoSS is the same as that of the SGD.\n\n- The proofs of the theorems are quite accessible: the reader only needs a college-level knowledge of Calculus, Linear Algebra, and Probability.\n\n**Weaknesses**\n\n- The requirement that the loss function be smooth calls into question the motivation for this work. After all, if we have smoothness and we know the loss function, then we can use the SGD. If we don't know the loss function, then how can we check its smoothness?\n\n- The work lacks empirical validation of the theory on real data. It would also be great if the authors show a specific natural example where the ZoSS (under the paper's assumptions) is applicable and the SGD is not.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "OC939APveyd", "reviewer": "Reviewer_Eioh", "summary": "The paper analyses the uniform stability of zero-order optimisation algorithms. Using the Gaussian randomisation for the gradient estimator that was put forward and extensively analysed by Nesterov and Spokoiniy, the authors establish the stability result in the spirit of Hardt et al [1] (who were only dealing with the pure SGD). This work focuses only on the generalisation bounds and ignores the optimisation error, referring to the work of Duchi et al [13].", "questions": "I addressed my main concerns in the previous part. I will update this part after the discussion phase and the authors' responses.", "limitations": "The authors addressed the limitations of their work. I will update this part after the discussion phase and the authors' responses.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall the paper is relatively well written and easy to follow. My main concern is in the relevance and the novelty of the analysis. Nesterov and Spokoiniy [14] (see Theorem 1) show that the gradient estimator deployed by the authors of this submission is an *unbiased estimator* of a smoothed version of $f(\\cdot, z)$. As long as the original function is Lipschitz (which is the case here), the smoothed version is a good approximation of the original one and the quality of this approximation is controlled by the parameter $\\mu$. Thus, the algorithm analyzed by the authors of this submission is an SGD performed on a slightly different objective, which has identical regularity properties. Consequently, using the result of Hardt et al [1] in the context of SGD + the triangle inequality coupled with Theorem 1 from Nesterov and Spokoiniy [14], we immediately get the stability result for the zero-order variant considered in this submission (and it allows us to pick $\\mu$ properly). Thus, I wonder if the authors could actually compact all their analysis into a couple of lines outsourcing most of the analysis from [1] and [14]?\n\nWhat is actually pretty interesting is that if the original functions are NOT $\\beta$-smooth, then the smoothed function has a Lipschitz gradient with the constant that depends on $\\mu$. Hence, an interesting direction could be the exploration of the case of non-$\\beta$-smooth loss functions. Unfortunately, this case is not considered by the authors.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Hz96ltvueg-", "reviewer": "Reviewer_HnfQ", "summary": "\nIn this paper, the authors provide generalization error bounds for derivative-free updates under $L$-Lipschitz and $\\beta$-smooth assumption of the loss fucntion.  The main technique extends  (Hardt et al.  2016) by bounding an additional gradient approximation error term ($\\epsilon_{est}$).", "questions": "\n\n Could the authors explain more about the proof of Lemma 2 to clarify my above concern? if $P(\\mathcal{E}_t) = \\frac{1}{n}$ instead of $P(\\mathcal{E}_t) = 1- \\frac{1}{n}$, how does this influence the generalization bounds?\n  ", "limitations": "NA", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nPros.\n1.  The generalization bounds for zeroth-order gradient update are new, and it may be potential for black-box learning tasks.\n\n2.  The bounds are independent of the dimension $d$ and the number of function evaluations per step $K$.\n\nCons.\n\n1.  In the proof of Lemma 2 (Line 194), it seems that $P(\\mathcal{E}_t) = \\frac{1}{n}$ instead of $P(\\mathcal{E}_t) = 1- \\frac{1}{n}$.  \n    If so, how does this influence the following proof and main theorem?\n\n2. The $\\beta$-smooth assumption of the loss function is a little bit strong. It requires the second-order derivative to be bounded.  It narrows down the potential loss functions because many practical loss functions may not have second-order derivatives, which makes the bounds less appealing.\n\n\n===============================\n\nThanks for the authors' detailed response.  My main concern has been clarified. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept