SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(iii) Let  $\varepsilon \in (0,1)$ , let  $K \geq 4(f_h(x_0) - \min f)cL_0^3 d\sqrt{d}\varepsilon^{-2} / (\ell h)$  and choose  $\alpha = \sqrt{\frac{(f_h(x_0) - \min f)\ell h}{KcL_0^3d\sqrt{d}}}$ . Then we have that  $\eta_K^{(h)} \leq \varepsilon$ . Thus, the number of function evaluations required to get a precision  $\eta_k^h \leq \varepsilon$  is of the order  $\mathcal{O}(d\sqrt{d} h^{-1}\varepsilon^{-2})$ .
Relying on the results in [34], we show that this is related to a precise notion of approximate stationarity. To do so, we need to introduce a definition of subdifferential which is suitable to this setting. As shown in [34] the Clarke subdifferential is not the right notion, and the approximate Goldstein subdifferential should be used instead.
Definition 1 (Goldstein subdifferential and stationary point). Under Assumption 1, let  $x \in \mathbb{R}^d$  and  $h > 0$ . The  $h$ -Goldstein subdifferential of  $f$  at  $x$  is  $\partial_h f(x) \coloneqq \mathrm{conv}(\cup_{y \in \mathbb{B}_h^d(x)} \partial f(y))$  where  $\partial f$  is the Clarke subdifferential [13] and  $\mathbb{B}_h^d(x)$  is the ball centered in  $x$  with radius  $h$ . For  $\varepsilon \in (0,1)$ , a point  $x \in \mathbb{R}^d$  is a  $(h,\varepsilon)$ -Goldstein stationary point for the function  $f$  if  $\min \{\|g\| | g \in \partial_h f(x)\} \leq \varepsilon$ .
Corollary 3. Under the same assumptions of Theorem 2, fix  $K \in \mathbb{N}$  and let  $I$  be a random variable taking values in  $\{0, \dots, K - 1\}$  such that, for all  $i$ ,  $\mathbb{P}[I = i] = \alpha_i / A_{K - 1}$ . Let also  $S_k$  be defined as in Theorem 2. Then
$$
\mathbb {E} _ {I} \left[ \min  \{\| \eta \| ^ {2}: \eta \in \partial f _ {h} (x _ {I}) \} \right] \leq S _ {K} / A _ {K}.
$$
In the setting of Corollary 2 (iii), we have also that  $\mathbb{E}_I\left[\min \{\|\eta\|^2 : \eta \in \partial f_h(x_I)\}\right] \leq \varepsilon$ .
Discussion. In Theorem 2, we fix the smoothing of the target, i.e. we consider  $h_k$  constant, and we analyze the non-smooth non-convex setting providing a rate on the expected norm of the smoothed gradient. The resulting bound is composed of two parts. The first part is very natural, and due to the functional value at the initialization. The second part is the approximation error. Recall that Assumption 1 holds, and therefore  $f_h \leq f + L_0h$  due to Proposition 1. This suggests taking  $h$  as small as possible in order to reduce the gap between  $f_h$  and  $f$ . However, taking  $h$  too small would make the approximation error very big. In our analysis, we consider the case with  $h$  constant. Moreover, as for the convex case, the speed of the rate depends on  $A_k$  and so we would like to take the stepsize as large as possible. But to control the approximation error, we need to assume  $\alpha_k^2 \in \ell^1$ . In Corollary 2, we consider two choices of stepsize. The first choice satisfies the property of  $\alpha_k^2 \in \ell^1$ , while the second one analyzes the case of constant step-size. Comparing our rate to the one in [39] we see that we obtain a better dependence on the dimension in the complexity, both in terms of iterations and function evaluations. Our results match the one of [34, Theorem 3.2] in terms of rate and in terms of function evaluations. We get a better dependence on the dimension in the number of iterations. Note again that, despite the complexity in terms of the number of function evaluations being the same, the possibility of parallelization for the function evaluations yields a better result for our method. As for the convex setting, we have a tighter upper-bound on the variance of the estimator of the smoothed gradient with respect to the dimension - see [34, Lemma D.1]. Goldstein stationarity has been used to assess the approximate stationarity for first-order methods as well, see [15]. The latter work shows that a cutting plane algorithm achieves a rate of  $\mathcal{O}(d\varepsilon^{-3})$  for Lipschitz functions.
3.3 Smooth Convex setting
We consider now the smooth setting, i.e. we assume that the target function satisfies the following hypothesis.
Assumption 3 ( $L_{1}$ -Smooth). The function  $f$  is  $L_{1}$ -smooth; i.e. the function  $f$  is differentiable and, for some  $L_{1} > 0$ ,
$$
(\forall x, y \in \mathbb {R} ^ {d}) \quad \| \nabla f (x) - \nabla f (y) \| \leq L _ {1} \| x - y \|.
$$
This is the standard assumption for analyzing first-order methods and has been used in many other works in the literature for zeroth-order algorithms - see e.g. [39, 17]. As shown in previous works, if  $f$  satisfies Assumption 3 then also  $f_{h}$  satisfies it - see Proposition 1. We will consider also the following assumptions on the stepsize and the smoothing in order to guarantee convergence.
Assumption 4 (Smooth zeroth-order convergence conditions). The stepsize sequence  $(\alpha_{k})_{k\in \mathbb{N}}$  and the smoothing sequence  $(h_k)_{k\in \mathbb{N}}$  satisfy the following conditions:
$$
\alpha_ {k} \notin \ell^ {1} \quad a n d \quad \alpha_ {k} h _ {k} \in \ell^ {1}.
$$
Moreover,  $\alpha_{k}\leq \bar{\alpha} <  \ell /dL_{1}$  for every  $k\in \mathbb{N}$
Note that this is a weaker version of Assumption 2. Next, we state the main theorem for convex smooth functions.
Theorem 3 (Smooth convex). Under Assumptions 3 and 4, let  $(x_{k})_{k\in \mathbb{N}}$  be a sequence generated by Algorithm 1 and  $x^{*}\in \arg \min f$ . For every  $k\in \mathbb{N}$ , set  $A_{k} = \sum_{i = 0}^{k}\alpha_{i}$  and  $\bar{x}_k = \sum_{i = 0}^k\alpha_ix_i / A_k$ . Then, for every  $k\in \mathbb{N}$ ,
$$
\mathbb {E} [ f (\bar {x} _ {k}) - \min  f ] \leq \frac {D _ {k}}{A _ {k}} \quad w i t h \quad D _ {k} := \frac {\ell \Delta + d \bar {\alpha}}{2 \ell \Delta} \Big (S _ {k} + \sum_ {i = 0} ^ {k} \rho_ {i} \Big (\sqrt {S _ {i}} + \sum_ {j = 0} ^ {i} \rho_ {j} \Big) \Big),
$$
where  $S_{k}\coloneqq \| x_{0} - x^{*}\|^{2} + \sum_{i = 0}^{k}\frac{L_{1}^{2}d^{2}}{2\ell}\alpha_{i}^{2}h_{i}^{2},$ $\rho_{k}\coloneqq L_{1}d\alpha_{k}h_{k}$  , and  $\Delta \coloneqq \left(\frac{1}{L_1} -\frac{d}{\ell}\bar{\alpha}\right)$
Corollary 4. Under the same Assumptions of Theorem 3, the following hold.
(i) If for every  $k \in \mathbb{N}$  we set  $\alpha_{k} = \alpha > 0$  and  $h_{k} = h(k + 1)^{-\theta}$  for  $h > 0$  and  $\theta > 1$ , then
$$
\mathbb {E} [ f (\bar {x} _ {k}) - \min  f ] \leq \frac {C}{\alpha k},
$$
where  $C$  is a constant provided in the proof. Moreover, if  $\alpha < \ell/(2dL_1)$ ,  $\lim_{k\to \infty}f(x_k) = \min f$  a.s. and there exists a random variable  $\hat{x}$  taking values in  $\arg \min f$  s.t.  $x_{k}\rightarrow \hat{x}$  a.s.
(ii) If for every  $k \in \mathbb{N}$  we set  $\alpha_{k} = \alpha > 0$  and  $0 < h_{k} \leq h$ , then
$$
\mathbb {E} [ f (\bar {x} _ {k}) - \min f ] \leq \frac {C _ {1}}{k} + C _ {2} \alpha h + C _ {3} \alpha^ {2} h ^ {2} \sqrt {k} + C _ {4} \alpha^ {2} h ^ {2} k,
$$
where  $C_1, C_2, C_3$  and  $C_4$  are non-negative constants.
Discussion. As in the previous cases, the bound in Theorem 3 is composed by two terms: the error due to the initialization and the one due to the approximation. An important difference with the results in the non-smooth setting is that every term in the approximation error is decreasing with respect to the smoothing parameter  $h_k$ . This allows obtaining convergence also with the constant step-size scheme, taking  $h_k \in \ell^1$ . In Corollary 4 (i), we recover the result of [32, Theorem 5.4] with a specific choice of parameters  $\alpha, h$  (up to constants). The complexity depends on the choice of  $\alpha$ . Note that by Assumption 4,  $\alpha < \ell / (L_1 d)$  thus the dependence on the dimension in the rate will be at least  $d / \ell$ . In particular, taking  $\alpha = \ell / (2 d L_1)$ , we obtain the optimal complexity of  $\mathcal{O}(d \varepsilon^{-1})$  in terms of function evaluations. This result has a better dependence on the dimension than [39]. In Corollary 4 (ii), the dependence on the dimension in the complexity depends on the choice of  $\alpha$  and  $h$ . Moreover, the rate obtained is equal (up to constants) to the rate obtained in [32] in the same setting, i.e.  $\mathcal{O}(1 / k)$  (in which we hide the dependence on  $d$  and  $\ell$ ). As for [32], for the first setting we can prove the almost sure convergence of the iterates.
3.4 Smooth Non-Convex setting
To analyze the smooth non-convex setting, we introduce the following notation:
$$
\left(\forall k \in \mathbb {R} ^ {d}\right) \quad A _ {k} := \sum_ {i = 0} ^ {k} \alpha_ {i}, \quad \eta_ {k} := \left(\sum_ {i = 0} ^ {k} \alpha_ {i} \mathbb {E} [ \| \nabla f (x _ {i}) \| ^ {2} ]\right) / A _ {k}.
$$
Note that, in comparison with the quantity defined in Section 3.2, here  $\eta_{k}$  is related to the exact objective function  $f$  and not to its smoothed version  $f_{h}$ . Next, we state the main result for smooth non-convex functions.
Theorem 4 (Smooth non-convex). Suppose that Assumption 3 holds and assume that, for every  $k \in \mathbb{N}$ ,  $\alpha_k \leq \bar{\alpha} < \ell / (2dL_1)$ . Let  $(x_k)_{k \in \mathbb{N}}$  be a sequence generated by Algorithm 1. Then
$$
\eta_ {k} \leq \frac {1}{\Delta A _ {k}} \left(f (x _ {0}) - \min  f + \frac {L _ {1} ^ {2} d ^ {2}}{8} \sum_ {i = 0} ^ {k} \alpha_ {i} h _ {i} ^ {2} + \frac {L _ {1} ^ {3} d ^ {2}}{4 \ell} \sum_ {i = 0} ^ {k} \alpha_ {i} ^ {2} h _ {i} ^ {2}\right), \quad \Delta := \left(\frac {1}{2} - \frac {L _ {1} d}{\ell} \bar {\alpha}\right).
$$
Corollary 5. Under the assumptions of Theorem 4, the following hold.
(i) If  $\alpha_{k} = \alpha \leq \bar{\alpha}$  and  $h_k = h k^{-\theta}$  with  $h > 0$  and  $\theta >1$  , then
$$
\eta_ {k} \leq \left[ \frac {f (x _ {0}) - \min f}{\Delta \alpha} + \frac {C _ {1} d ^ {2} h ^ {2}}{\Delta} + \frac {C _ {2} \alpha h ^ {2} d ^ {2}}{\Delta \ell} \right] \cdot \frac {1}{k},
$$
where  $C_1$  and  $C_2$  are constants provided in the proof.
(ii) If  $\alpha_{k} = \alpha \leq \bar{\alpha}$  and  $h_k = h > 0$ , then
$$
\eta_ {k} \leq \frac {f (x _ {0}) - \min f}{\Delta \alpha k} + \frac {C _ {1} d ^ {2} h ^ {2}}{\Delta} + \frac {C _ {2} \alpha h ^ {2} d ^ {2}}{\Delta \ell},
$$
Discussion. As for the convex case, every term in the approximation error depends on the smoothing parameter  $h_k$ . In Corollary 5 (i), we take constant step-size and  $h_k \in \ell^1$ . With this choice of parameters, we get a rate of  $\mathcal{O}(1 / k)$  which matches with the result obtained by [39]. The dependence on the dimension depends on the choice of  $\alpha$  and  $h$ . Note that  $\alpha < \ell / (2dL_1)$ , thus taking  $h = \mathcal{O}(1 / d)$  in the rate we get a dependence on the dimension of  $d / \ell$ . Taking for instance  $\alpha = \ell / (3dL_1)$  and  $h = \mathcal{O}(1 / d)$ , we get a complexity of  $\mathcal{O}(d\varepsilon^{-1})$  in terms of function evaluations.
4 Numerical Results
In this section, we provide some numerical experiments to assess the performances of our algorithm. We consider two target functions: a convex smooth one and a convex non-smooth one. Details on target functions and parameters of the algorithms are reported in Appendix C. To report our findings, we run the experiments 10 times and provide the mean and standard deviation of the results.
How to choose the number of directions? In these experiments, we set a fixed budget of 4000 function evaluations and we consider  $d = 50$ . We investigate how the performance of Algorithm 1 changes as the value of  $\ell$  increases. In Figure 1, we observe the mean sequence  $f(x_{k}) - f(x^{*})$  after each function evaluation. If  $\ell > 1$ , then the target function values are repeated  $2\ell$  times, since we need to perform  $2\ell$  function evaluations to do one iteration. For a sufficiently large budget, increasing the number of directions  $\ell$  leads to better results compared to using a single direction in both smooth and non-smooth settings.
Comparison with finite-difference methods. Now, we compare Algorithm 1 with other finite-difference methods. More precisely, we consider finite differences with single (and multiple) Gaussian (and spherical) directions. The budget of function evaluations is 1000 and the ambient dimension is  $d = 10$ . For multiple direction methods, we fix the number of directions  $\ell = d$ . Further experiments are provided in Appendix F.
In Figure 2, we plot the sequence  $f(x_{k}) - f(x^{*})$  with respect to the number of function evaluations. While in terms of rates and complexity the different algorithms are the same, Algorithm 1 shows better performances than random directions approaches, and we believe this is due to the use of structured (i.e. orthogonal) directions. Indeed, orthogonal directions yield a better approximation of first-order information with respect to other methods. The practical advantages of structured directions were already observed in [32, 42, 5, 12] and these experiments confirm that the good practical behavior holds even in the nonsmooth setting.
We introduced and analyzed O-ZD a zeroth-order algorithm for non-smooth zeroth-order optimization. We analyzed the algorithm and derived rates for non-smooth and smooth functions. This work opens different research directions. An interesting one would be the introduction of a learning procedure for the orthogonal directions. Such an approach could have significant practical applications.
This project has been supported by the TraDE-OPT project, which received funding from the European Union's Horizon 2020 research and innovation program under the Marie Skłodowska-Curie grant agreement No 861137. L. R. and M. R. acknowledge the financial support of the European Research Council (grant SLING 819789), the AFOSR projects FA9550-18-1-7009 (European Office of Aerospace Research and Development), the EU H2020-MSCA-RISE project NoMADS - DLV-777826, and the Center for Brains, Minds and Machines (CBMM), funded by NSF STC award CCF-1231216. S. V. and L. R. acknowledge the support of the AFOSR project FA8655-22-1-7034. The research by S. V. and C. M. has been supported by the MIUR Excellence Department Project awarded to Dipartimento di Matematica, Università di Genova, CUP D33C23001110001. S. V. and C. M. are members of the Gruppo Nazionale per l'Analisi Matematica, la Probabilità e le loro Applicazioni (GNAMPA) of the Istituto Nazionale di Alta Matematica (INdAM). This work represents only the view of the authors. The European Commission and the other organizations are not responsible for any use that may be made of the information it contains.
[1] Hans Wilhelm Alt. Linear Functional Analysis: An Application-Oriented Introduction. Springer London, London, 2016.
[2] T. W. Anderson, I. Olkin, and L. G. Underhill. Generation of random orthogonal matrices. SIAM Journal on Scientific and Statistical Computing, 8(4):625-629, 1987.
[3] J.B. Baillon and G. Haddad. Quelques propriétés des opérateurs angle-bornés etn-cycliquement monotones. *Israel Journal of Mathematics*, 26(2):137-150, Jun 1977.
[4] A. Barvinok. Approximating orthogonal matrices by permutation matrices. Pure and applied mathematics quarterly, 2, 11 2005.
[5] A. S. Berahas, L. Cao, K. Choromanski, and K. Scheinberg. A theoretical and empirical comparison of gradient approximations in derivative-free optimization. Foundations of Computational Mathematics, 22(2):507-560, Apr 2022.
[6] D. P. Bertsekas. Stochastic optimization problems with nondifferentiable cost functionals. Journal of Optimization Theory and Applications, 12(2):218-231, Aug 1973.
[7] Å. Björck. Numerics of gram-schmidt orthogonalization. Linear Algebra and its Applications, 197-198:297-316, 1994.
[8] C. Boutsidis and A. Gittens. Improved matrix algorithms via the subsampled randomized hadamard transform. SIAM Journal on Matrix Analysis and Applications, 34(3):1301-1340, 2013.
[44] C. Rusu and L. Rosasco. Fast approximation of orthogonal matrices and application to pca. Signal Processing, 194:108451, 2022.
[45] T. Salimans, J. Ho, X. Chen, S. Sidor, and I. Sutskever. Evolution strategies as a scalable alternative to reinforcement learning, 2017.
[46] O. Shamir. An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. The Journal of Machine Learning Research, 18(1):1703-1713, 2017.
[47] C. Totzeck. Trends in Consensus-Based Optimization, pages 201–226. Springer International Publishing, Cham, 2022.
[48] T. Trogdon. On spectral and numerical properties of random butterfly matrices. Applied Mathematics Letters, 95:48-58, 2019.
[49] F. Yousefian, A. Nedic, and U. V. Shanbhag. On stochastic gradient and subgradient methods with adaptive steplength sequences. Automatica, 48(1):56-67, 2012.
A Auxiliary Results
In this appendix, we state and collect lemmas and propositions required to prove the main results.
Notation. In the following sections, we denote with  $\mathcal{F}_k$  the filtration  $\sigma(G_1, \dots, G_{k-1})$ . Moreover, to simplify the notation, we define  $g_k$  as the gradient surrogate in eq.(2) at time-step  $k$  i.e.  $g_k \coloneqq g_{(G_k, h_k)}(x_k)$  and  $g(\cdot) \coloneqq g_{(G,h)}(\cdot)$  for an arbitrary  $G \in O(d)$  and  $h > 0$ . We denote the normalized Haar measure [37] by  $\mu$ . We define the unit ball  $\mathbb{B}^d$  and the unit sphere  $\mathbb{S}^{d-1}$  as follows
$$
\mathbb {B} ^ {d} := \left\{v \in \mathbb {R} ^ {d} \mid \| v \| \leq 1 \right\} \quad \text {a n d} \quad \mathbb {S} ^ {d - 1} := \left\{v \in \mathbb {R} ^ {d} \mid \| v \| = 1 \right\}.
$$
We denote by  $\sigma$  and  $\sigma_N$  the spherical measure and the normalized spherical measure on  $\mathbb{S}^{d - 1}$ , respectively. Moreover, we denote with  $I_{d,\ell} \in \mathbb{R}^{d \times \ell}$  the (truncated) identity matrix.
Lemma 2. Let  $\beta (\mathbb{S}^{d - 1})$  be the surface area of  $\mathbb{S}^{d - 1}$  and let  $I\in \mathbb{R}^{d\times d}$  be the identity matrix. Then,
$$
\int_ {\mathbb {S} ^ {d - 1}} v v ^ {\intercal} d \sigma (v) = \frac {\beta (\mathbb {S} ^ {d - 1})}{d} I.
$$
Proof. This result is proved in [21, Lemma 7.3, point (b)].
Lemma 3. Let  $\phi :\mathbb{R}^d\to \mathbb{R}$  be a  $L$  -Lipschitz function . If  $u$  is uniformly distributed on  $\mathbb{S}^{d - 1}$  , then
$$
\left. \left(\mathbb {E} [ \phi (u) - \mathbb {E} [ \phi (u) ] ]\right)\right) ^ {2} \leq c \frac {L ^ {2}}{d},
$$
for some numerical constant  $c > 0$ .
Proof. The proof follows the same line as [46, Lemma 9].
A.1 Smoothing Lemma & Properties
In this appendix, we provide the proof of the Smoothing Lemma (i.e. Lemma 1).
Proof of Smoothing Lemma. By eq. (2),
$$
\mathbb {E} _ {G} \left[ g _ {(G, h)} (x) \right] = \frac {d}{\ell} \sum_ {i = 1} ^ {\ell} \int_ {O (d)} \frac {f (x + h G e _ {i}) - f (x - h G e _ {i})}{2 h} G e _ {i} d \mu (G).
$$
By [37, Theorem 3.7],
$$
\mathbb {E} _ {G} \left[ g _ {(G, h)} (x) \right] = \frac {d}{2 \ell h} \sum_ {i = 1} ^ {\ell} \int_ {\mathbb {S} ^ {d - 1}} \left(f \left(x + h v ^ {(i)}\right) - f \left(x - h v ^ {(i)}\right)\right) v ^ {(i)} d \sigma_ {N} \left(v ^ {(i)}\right).
$$
Since  $v^{(i)}$  is uniformly distributed on the sphere, which is symmetric with respect to the origin, we have
$$
\mathbb {E} _ {G} \left[ g _ {(G, h)} (x) \right] = \frac {d}{\ell h} \sum_ {i = 1} ^ {\ell} \int_ {\mathbb {S} ^ {d - 1}} f (x + h v ^ {(i)}) v ^ {(i)} d \sigma_ {N} (v ^ {(i)}).
$$
As a consequence of Stokes' Theorem (details in [18, Lemma 1] and [1, Theorem A8.8]), we get
$$
\mathbb {E} \left[ g _ {(G, h)} (x) \right] = \frac {1}{\ell} \sum_ {i = 1} ^ {\ell} \nabla f _ {h} (x) \quad \text {w i t h} \quad f _ {h} (x) := \frac {1}{\operatorname {v o l} (\mathbb {B} ^ {d})} \int_ {\mathbb {B} ^ {d}} f (x + h u) d u.
$$
Rearranging terms, we get the claim.
Proposition 1 (Smoothing properties). Let  $f_{h}$  be the smooth approximation of  $f$  defined in eq. (4). Then the following hold:
If  $f$  is convex then  $f_{h}$  is convex and, for every  $x\in \mathbb{R}^d$
$$
f (x) \leq f _ {h} (x).
$$
If  $f$  is  $L_0$ -Lipschitz continuous - i.e.  $\forall x, y \in \mathbb{R}^d$ ,  $|f(x) - f(y)| \leq L_0 \| x - y \|$ , then  $f_h$  is  $L_0$ -Lipschitz continuous, differentiable and for every  $x, y \in \mathbb{R}^d$
$$
\| \nabla f _ {h} (x) - \nabla f _ {h} (y) \| \leq \frac {L _ {0} \sqrt {d}}{h} \| x - y \| a n d f _ {h} (x) \leq f (x) + L _ {0} h.
$$
If  $f$  is  $L_1$ -smooth - i.e.  $f$  is differentiable and  $\forall x, y \in \mathbb{R}^d$ ,  $\| \nabla f(x) - \nabla f(y) \| \leq L_1 \| x - y \|$  then  $f_h$  is  $L_1$ -smooth and for every  $x \in \mathbb{R}^d$ ,
$$
\| \nabla f _ {h} (x) - \nabla f (x) \| \leq \frac {h d L _ {1}}{2} \quad a n d \quad f _ {h} (x) \leq f (x) + \frac {L _ {1}}{2} h ^ {2}.
$$
Proof. These are standard results proposed and proved in different works - see for example [16, Lemma 8],[21, Proposition 7.5],[34, Proposition 2.2],[49].  $\square$
Lemma 4 (Approximation Error). Let  $g(\cdot)$  be the surrogate defined in eq. (2) for arbitrary  $h > 0$  and  $G \in O(d)$ . Then the following hold:
(i) If  $f$  is  $L_0$ -Lipschitz (see Assumption 1), then, for every  $x \in \mathbb{R}^d$ ,
$$
\mathbb {E} _ {G} [ \| g (x) \| ^ {2} ] \leq 2 c \frac {d L _ {0} ^ {2}}{\ell},
$$
where  $c$  is a numerical constant.
(ii) If  $f$  is  $L_{1}$ -smooth (see Assumption 3), then, for every  $x \in \mathbb{R}^{d}$ ,
$$
E _ {G} \left[ \| g (x) \| ^ {2} \right] \leq \frac {2 d}{\ell} \| \nabla f (x) \| ^ {2} + \frac {L _ {1} ^ {2} d ^ {2}}{2 \ell} h ^ {2}.
$$
Proof. Note that, since directions are orthogonal, we have
$$
\mathbb {E} _ {G} [ \| g (x) \| ^ {2} ] = \frac {d ^ {2}}{4 \ell^ {2} h ^ {2}} \sum_ {i = 1} ^ {\ell} \mathbb {E} _ {G} [ (f (x + h G e _ {i}) - f (x - h G e _ {i})) ^ {2} \| G e _ {i} \| ^ {2} ].
$$
$$
\mathbb {E} _ {G} [ \| g (x) \| ^ {2} ] = \frac {d ^ {2}}{4 \ell^ {2} h ^ {2}} \sum_ {i = 1} ^ {\ell} \mathbb {E} _ {v _ {i}} [ (f (x + h v ^ {(i)}) - f (x - h v ^ {(i)})) ^ {2} \| v ^ {(i)} \| ^ {2} ], \tag {5}
$$
where each  $v^{(i)}$  is uniformly distributed on  $\mathbb{S}^{d - 1}$ .
(i): Set  $\gamma = \mathbb{E}_{v^{(i)}}[f(x + hv^{(i)})]$  for every  $i$  (this expectation does not depend on  $i$ ). Then
$$
\begin{array}{l} \mathbb {E} _ {G} [ \| g (x) \| ^ {2} ] = \frac {d ^ {2}}{4 \ell^ {2} h ^ {2}} \sum_ {i = 1} ^ {\ell} \mathbb {E} _ {v ^ {(i)}} [ (f (x + h v ^ {(i)}) - f (x - h v ^ {(i)}) + \gamma - \gamma) ^ {2} \| v ^ {(i)} \| ^ {2} ] \\ = \frac {d ^ {2}}{4 \ell^ {2} h ^ {2}} \sum_ {i = 1} ^ {\ell} \mathbb {E} _ {v ^ {(i)}} [ ((f (x + h v ^ {(i)}) - \gamma) - (f (x - h v ^ {(i)}) - \gamma)) ^ {2} \| v ^ {(i)} \| ^ {2} ] \\ \leq \frac {d ^ {2}}{2 \ell^ {2} h ^ {2}} \sum_ {i = 1} ^ {\ell} \mathbb {E} _ {v ^ {(i)}} \left[ \left(\left(f (x + h v ^ {(i)}) - \gamma\right) ^ {2} + \left(f (x - h v ^ {(i)}) - \gamma\right) ^ {2}\right) \| v ^ {(i)} \| ^ {2} \right] \\ = \frac {d ^ {2}}{2 \ell^ {2} h ^ {2}} \sum_ {i = 1} ^ {\ell} \left[ \mathbb {E} _ {v ^ {(i)}} [ (f (x + h v ^ {(i)}) - \gamma) ^ {2} \| v ^ {(i)} \| ^ {2} ] \right. \\ + \mathbb {E} _ {v ^ {(i)}} \big [ (f (x - h v ^ {(i)}) - \gamma) ^ {2} \| v ^ {(i)} \| ^ {2} \big ]. \\ \end{array}
$$
Since  $v^{(i)}$  is uniformly distributed on  $\mathbb{S}^{d - 1}$ , it satisfies  $\| v^{(i)}\| ^2 = 1$  and by symmetry we have
$$
\mathbb {E} _ {G} [ \| g (x) \| ^ {2} ] \leq \frac {d ^ {2}}{\ell^ {2} h ^ {2}} \sum_ {i = 1} ^ {\ell} \mathbb {E} _ {v ^ {(i)}} [ (f (x + h v ^ {(i)}) - \gamma) ^ {2} ].
$$
The definition of  $\gamma$  yields
$$
\begin{array}{l} \mathbb {E} _ {G} [ \| g (x) \| ^ {2} ] \leq \frac {d ^ {2}}{\ell^ {2} h ^ {2}} \sum_ {i = 1} ^ {\ell} \mathbb {E} _ {v ^ {(i)}} [ ((f (x + h v ^ {(i)}) - \gamma) ^ {2} ] \\ = \frac {d ^ {2}}{\ell^ {2} h ^ {2}} \sum_ {i = 1} ^ {\ell} \mathbb {E} _ {v ^ {(i)}} [ (f (x + h v ^ {(i)}) - \mathbb {E} _ {v ^ {(i)}} [ f (x + h v ^ {(i)} ]) ^ {2} ]. \\ \end{array}
$$
The claim follows by Lemma 3 and the fact that  $f(x + hv^{(i)})$  is  $hL_0$ -Lipschitz continuous w.r.t to  $v^{(i)}$ .
(ii): Equation (5) yields