SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Marco Rando*
Cesare Molinari†
Silvia Villa†
Lorenzo Rosasco*
Finite-difference methods are a class of algorithms designed to solve black-box optimization problems by approximating a gradient of the target function on a set of directions. In black-box optimization, the non-smooth setting is particularly relevant since, in practice, differentiability and smoothness assumptions cannot be verified. To cope with nonsmoothness, several authors use a smooth approximation of the target function and show that finite difference methods approximate its gradient. Recently, it has been proved that imposing a structure in the directions allows improving performance. However, only the smooth setting was considered. To close this gap, we introduce and analyze O-ZD, the first structured finite-difference algorithm for non-smooth black-box optimization. Our method exploits a smooth approximation of the target function and we prove that it approximates its gradient on a subset of random orthogonal directions. We analyze the convergence of O-ZD under different assumptions. For non-smooth convex functions, we obtain the optimal complexity. In the non-smooth non-convex setting, we characterize the number of iterations needed to bound the expected norm of the smoothed gradient. For smooth functions, our analysis recovers existing results for structured zeroth-order methods for the convex case and extends them to the non-convex setting. We conclude with numerical simulations where assumptions are satisfied, observing that our algorithm has very good practical performances.
Black-box optimization problems are a class of problems for which only values of the target function are available and no first-order information is provided. Typically, these problems arise when the evaluation of the objective function is based on a simulation and no analytical form of the gradient is accessible or its explicit calculation is too expensive [14, 45, 36, 35].
To face these problems, different methods that do not require first-order information have been proposed - see for instance [39, 47, 19, 26, 20, 30, 33] and references therein. These techniques are called derivative-free methods and a wide class of these is the class of finite-difference algorithms [31, 39, 17]. These iterative procedures mimic first-order optimization strategies by replacing the gradient of the objective function with an approximation built through finite differences in random directions.
Two types of finite-difference methods can be identified: unstructured and structured ones, depending on the way in which the random directions are generated. In the former, directions are sampled i.i.d. from some distribution [39, 17, 10, 46] while in the latter, directions have to satisfy some structural constraints, e.g. orthogonality [32, 42]. Several authors [32, 42, 5, 12] theoretically and empirically
$^{*}$ MaLGa-DIBRIS, University of Genova, IT (marco.rando@edu.unige.it, lorenzo.rosasco@unige.it).
†MaLGa - DIMA, University of Genova, Italy (molinari@dima.unige.it, silvia.villa@unige.it).
$^{\ddagger}$  Istituto Italiano di Tecnologia, Genova, Italy and CBMM - MIT, Cambridge, MA, USA
observed that imposing orthogonality among the directions provides better performance than using unstructured directions. Intuitively, imposing orthogonality allows us to avoid cases in which the gradient approximation is built using similar or redundant directions.
Notably, methods based on structured finite differences have been analyzed only for smooth and convex (or specific non-convex) target functions. This represents a strong limitation, since smoothness and convexity are in practice hardly verifiable, due to the nature of black-box optimization problems. The aim of this paper is the analysis of structured finite difference algorithms dropping smoothness and convexity assumptions.
For unstructured finite-difference methods, a common way to face nonsmoothness consists in introducing a smooth approximation of the target function, also known as "smoothing" [6, 16] and using it as a surrogate of the target. Although the gradient of the smoothing is not computable, different authors showed that for certain types of smoothing the unstructured finite-difference approximation provides an unbiased estimation of such a gradient - see, for example, [39, 46, 18, 22, 21, 24]. This key observation allows to prove that unstructured finite-difference methods approximate a solution in different nonsmooth settings [17, 39, 46, 18, 22, 34].
For structured finite-difference methods, the analysis in the nonsmooth setting is not available. A key step which is missing is the proof of the fact that the surrogate of the gradient built with structured directions is an estimation of the gradient of a suitable smoothing.
In this paper, we close the gap, and introduce O-ZD, a structured finite-difference algorithm in the non-smooth setting. The algorithm builds an approximation of the gradient of a smoothed target using a set of  $\ell \leq d$  orthogonal directions. We analyze the procedure proving that our finite-difference surrogate is an unbiased estimation of the gradient of a smoothing. We provide convergence rates for non-smooth convex functions with optimal dependence on the dimension [17] and rates for the non-smooth non-convex setting (for which lower bounds are not known [34]). Moreover, for non-smooth convex functions we provide the first proof of convergence of the iterates for structured finite differences in this setting. For smooth convex functions, we recover the standard results for structured zeroth-order methods [32, 42]. We conclude with numerical illustrations. To the best of our knowledge, this is the first work on nonsmooth finite-difference method with structured directions.
The paper is organized as follows. In Section 2, we introduce the problem and the algorithm. In Section 3, we state and discuss the main results. In Section 4 we provide some experiments and in 5 some final remarks.
2 Problem Setting & Algorithm
Given a function  $f: \mathbb{R}^d \to \mathbb{R}$  and assuming that  $f$  has at least a minimizer in  $\mathbb{R}^d$ , we consider the problem to find
$$
x ^ {*} \in \underset {x \in \mathbb {R} ^ {d}} {\arg \min } f (x). \tag {1}
$$
In particular, we consider the non-smooth setting where  $f$  might be non-differentiable. To solve problem (1), we propose a zeroth-order algorithm, namely an iterative procedure that uses only function values. At every iteration  $k \in \mathbb{N}$ , a first-order information of  $f$  is approximated with finite-differences using a set of  $\ell \leq d$  random orthogonal directions  $(p_k^{(i)})_{i=1}^{\ell}$ . Such orthogonal directions are represented as rotations (and reflections) of the first  $\ell$  vectors of the canonical basis  $(e_i)_{i=1}^{\ell}$ . We set  $p_k^{(i)} = G_k e_i$ , where  $G_k$  belongs to the orthogonal group defined as
$$
O (d) := \{G \in \mathbb {R} ^ {d \times d} | \det  G \neq 0 \wedge G ^ {- 1} = G ^ {\intercal} \}.
$$
Methods for generating orthogonal matrices are discussed in Appendix D. Given  $G \in O(d)$ ,  $0 < \ell \leq d$  and  $h > 0$ , we consider the following central finite-difference surrogate of the gradient information
$$
g _ {(G, h)} (x) = \frac {d}{\ell} \sum_ {i = 1} ^ {\ell} \frac {f (x + h G e _ {i}) - f (x - h G e _ {i})}{2 h} G e _ {i}. \tag {2}
$$
Then, we introduce the following algorithm.
Algorithm 1 O-ZD: Orthogonal Zeroth-order Descent
Input:  $x_0\in \mathbb{R}^d$ $(\alpha_{k})_{k\in \mathbb{N}}\subset \mathbb{R}_{+}$ $(h_k)_{k\in \mathbb{N}}\subset \mathbb{R}_+$ $\ell \in \mathbb{N}$  s.t.  $1\leq \ell \leq d$
for  $k = 1,\dots$  do
sample  $G_{k}$  i.i.d. from  $O(d)$
$$
x _ {k + 1} = x _ {k} - \alpha_ {k} g _ {\left(G _ {k}, h _ {k}\right)} \left(x _ {k}\right)
$$
Starting from an initial guess  $x_0 \in \mathbb{R}^d$ , at every iteration  $k \in \mathbb{N}$ , the algorithm samples an orthogonal matrix  $G_k$  i.i.d. from the orthogonal group  $O(d)$  and computes a surrogate  $g_{(G_k, h_k)}$  of the gradient at the current iterate  $x_k$ . Then, it computes  $x_{k+1}$  by moving in the opposite direction of  $g_{(G_k, h_k)}(x_k)$ . This approach belongs to the class of structured finite-difference methods, where a bunch of orthogonal directions is used to approximate a gradient of the target function [32, 42]. Such algorithms have been proposed and analyzed only for smooth functions. To cope with this limitation, and extend the analysis to the nonsmooth setting, we exploit a smoothing technique. For a fixed a probability measure  $\rho$  on  $\mathbb{R}^d$  and a positive parameter  $h > 0$  called smoothing parameter, we define the following smooth surrogate of  $f$
$$
f _ {h, \rho} (x) := \int f (x + h u) d \rho (u). \tag {3}
$$
As shown in [6],  $f_{h,\rho}$  is differentiable even when  $f$  is not. In the literature, different authors have used this strategy to face non-smooth zeroth-order optimization with random finite-difference methods [18, 39, 49, 22] fixing specific smoothing distribution, but no one applied and analyze it for structured methods.
In this work,  $\rho$  is the uniform distribution over the  $\ell_2$  unit ball  $\mathbb{B}^d$ , defining the smooth surrogate
$$
f _ {h} (x) = \frac {1}{\operatorname {v o l} (\mathbb {B} ^ {d})} \int_ {\mathbb {B} ^ {d}} f (x + h u) d u, \tag {4}
$$
where  $\mathrm{vol}(\mathbb{B}^d)$  denotes the volume of  $\mathbb{B}^d$ . One of our main contributions is the following Lemma which shows that the surrogate proposed in (2) is an unbiased estimator of the gradient of the smoothing in (4).
Lemma 1 (Smoothing Lemma). Given a probability space  $(\Omega, \mathcal{F}, \mathbb{P})$ , let  $G: \Omega \to O(d)$  be a random variable where  $O(d)$  is the orthogonal group endowed with the Borel  $\sigma$ -algebra. Assume that the probability distribution of  $G$  is the (normalized) Haar measure. Let  $h > 0$  and let  $g$  be the gradient surrogate defined in eq. (2). Then,
$$
(\forall x \in \mathbb {R} ^ {d}) \quad \mathbb {E} _ {G} [ g _ {(G, h)} (x) ] = \nabla f _ {h} (x),
$$
where  $f_{h}$  is the smooth approximation of the target function  $f$  defined in eq. (4).
The proof of Lemma 1 is provided in Appendix A.1.
Remark 1. Note that Lemma 1 holds also using  $g_{(G,h)}^F$  or  $g_{(G,h)}^S$  defined as
$$
g _ {(G, h)} ^ {F} (x) := \frac {d}{\ell} \sum_ {i = 1} ^ {\ell} \frac {f (x + h G e _ {i}) - f (x)}{h} G e _ {i} \quad a n d \quad g _ {(G, h)} ^ {S} (x) := \frac {d}{\ell} \sum_ {i = 1} ^ {\ell} \frac {f (x + h G e _ {i})}{h} G e _ {i},
$$
since  $\mathbb{E}_G[g_{(G,h)}(x)] = \mathbb{E}_G[g_{(G,h)}^F (x)] = \mathbb{E}_G[g_{(G,h)}^S (x)]$ . Despite these two estimators being computationally cheaper than the proposed one, we use central finite differences since they allow us to derive a better bound for  $\mathbb{E}_G[\| g_{(G,h)}(x)\| ^2 ]$  as observed in [46] for the case  $\ell = 1$ .
Thanks to Lemma 1, we can interpret each step of Algorithm 1 as a Stochastic Gradient Descent (SGD) on the smoothed function  $f_{h_k}$ . But the analysis of the proposed algorithm does not follow from the SGD one for two reasons. First, the smoothing parameter  $h_k$  changes along the iterations; second (and more importantly), the set of minimizers of  $f_h$  and  $f$  are different in general. However, we will take advantage of the fact that  $f_h$  can be seen as an approximation of  $f$ . The relationship between  $f$  and its approximation  $f_h$  depends on the properties of  $f$  - see Proposition 1 in Appendix A.
Our main contributions are the theoretical and numerical analysis of Algorithm 1 under different choices of the free parameters  $\alpha_{k}, h_{k}$ , namely the stepsize and the smoothing parameter. To the best of our knowledge, Algorithm 1 is the first structured zeroth-order method for non-smooth optimization.
2.1 Related Work
In practice, the advantage of the use of structured directions in finite-difference methods has been observed in several applications [12] and motivated their study. In [5], the authors theoretically and empirically observed that structured directions provide a better approximation of the gradient with respect to unstructured (Gaussian and spherical) ones. Next, we review the most related works, showing the differences with our algorithm.
Unstructured Finite-differences. Most of existing works focused on the theoretical analysis of methods using a single direction to approximate the gradient - see e.g. [39, 46, 22, 45, 18, 24, 34, 17, 10]. The main results existing so far analyze the convergence of the function values in expectation. They provide convergence rates in terms of the number of function evaluations and explicitly characterize the dependence on the dimension of the ambient space.
Smooth setting: In [39, 17], a finite difference algorithm with a single direction is analyzed. Rates on the expected function values are shown. In [39] the dependence on the dimension in the rates is not optimal. In [17], both single and multiple direction cases are analyzed and lower bounds are derived. However, only the convex setting is considered. In [24], both convex and non-convex settings are analyzed. They obtain optimal dependence on dimension in the complexity for convex functions. However, only the single-direction case is considered and only the smooth setting is analyzed. Comparing the result with our rates, Algorithm 1 achieves the same dependence on the dimension in the complexity taking  $\ell$  as a fraction of  $d$ . Note that, by parallelizing the computation of the function evaluations, we can get a better result.
Non-smooth setting: In [39, 17] the non-smooth setting is also analyzed. However, only the single direction case has been considered and both algorithms do not achieve the lower bound. More precisely, for convex functions, a complexity of  $\mathcal{O}(d^2\varepsilon^{-2})$  is achieved by [39] and  $\mathcal{O}(d\log (d)\varepsilon^{-2})$  by [17] while our algorithm gets the optimal dependence. Moreover, note that in [17] the strategy adopted (also called "double smoothing") requires tuning one more sequence of parameters, which is a challenging problem in practice, and only the convex setting is considered. In [39], also the non-convex setting is analyzed by bounding the expected norm of the smoothed gradient. However, they obtain a complexity of  $\mathcal{O}(d^3\varepsilon^{-2}h^{-1})$  while our algorithm gets a better dependence on the dimension. In [46], the optimal complexity is obtained for convex functions. However, the author does not analyze the non-convex setting. Moreover, note that, despite the complexity in terms of function evaluations being the same, our algorithm gets a better complexity in terms of the number of iterations since it uses multiple directions (and this is an advantage if we can parallelize the function evaluations). Furthermore, note that the method proposed in [46] can be seen as the special case of Algorithm 1 with  $\ell = 1$ . In [34] the single direction case is analyzed only in the non-convex setting. The dependence on the dimension of the complexity in the number of function evaluations achieved matches our result in this setting (again, in the number of iterations our method obtains a better dependence).
Structured Finite-difference. In [32, 42], authors analyze structured finite differences in both deterministic and stochastic settings. However, only the smooth convex setting is considered. In [12] orthogonal matrices are used to build directions but no analysis is provided. In [25], finite-difference with coordinate directions is analyzed. At every iteration,  $d + 1$  function evaluations are performed to compute the estimator and only the smooth setting is considered.
In this section, we analyze Algorithm 1 considering both non-smooth and smooth problems. We present the rates obtained by our method for convex and non-convex settings and compare them with those obtained by state-of-the-art methods. Proofs are provided in Appendix B. In the following, we call complexity in the number of iterations / function evaluations, respectively, the number of iterations / function evaluations required to achieve an accuracy  $\varepsilon \in (0,1)$ .
3.1 Non-smooth Convex Setting
In this section, we provide the main results for non-smooth convex functions. In particular, we will assume that the target function is convex and satisfy the following hypothesis.
Assumption 1 ( $L_0$ -Lipschitz continuous). The function  $f$  is  $L_0$ -Lipschitz continuous; i.e., for some  $L_0 > 0$ ,
$$
(\forall x, y \in \mathbb {R} ^ {d}) \quad | f (x) - f (y) | \leq L _ {0} \| x - y \|.
$$
Note that this assumption implies that also  $f_{h}$  is  $L_{0}$ -Lipschitz continuous - see Proposition 1. Moreover, to analyze the algorithm, we will consider the following parameter setting.
Assumption 2 (Zeroth-order non-smooth convergence conditions). The step-size sequence  $\alpha_{k}$  and the sequence of smoothing parameters  $h_k$  satisfy the following conditions:
$$
\alpha_ {k} \notin \ell^ {1}, \quad \alpha_ {k} ^ {2} \in \ell^ {1} \quad a n d \quad \alpha_ {k} h _ {k} \in \ell^ {1}.
$$
The assumption above is required to guarantee convergence to a solution. In particular, the first two conditions are common for subgradient method and stochastic gradient descent, while the third condition was already used in structured zeroth-order methods [32, 42] and links the decay of the smoothing parameter with the stepsize's one. An example of  $\alpha_{k}, h_{k}$  that satisfy Assumption 2 is  $\alpha_{k} = k^{-\theta}$  and  $h_k = k^{-\rho}$  with  $\theta \in (1/2, 1)$  and  $\rho$  s.t.  $\theta + \rho > 1$ .
We state now the main theorem for non-smooth convex functions.
Theorem 1 (Non-smooth convex). Under Assumption 1, assume that  $f$  is convex and let  $(x_{k})_{k\in \mathbb{N}}$  be a sequence generated by Algorithm 1. For every  $k\in \mathbb{N}$ , denote  $A_{k} = \sum_{i = 0}^{k}\alpha_{i}$  and set  $\bar{x}_k\coloneqq \sum_{i = 0}^{k}\alpha_{i}x_{i} / A_{k}$ . Then
$$
\mathbb {E} \left[ f \left(\bar {x} _ {k}\right) - \min  f \right] \leq S _ {k} / A _ {k} \quad \text {w i t h} \quad S _ {k} := \frac {\| x _ {0} - x ^ {*} \| ^ {2}}{2} + c \frac {d L _ {0} ^ {2}}{\ell} \sum_ {i = 0} ^ {k} \alpha_ {i} ^ {2} + L _ {0} \sum_ {i = 0} ^ {k} \alpha_ {i} h _ {i},
$$
where  $c > 0$  is a constant independent of the dimension and  $x^{*}$  is any solution in  $\arg \min f$ . Moreover, under Assumption 2, we have
$$
\lim  _ {k \rightarrow + \infty} f (x _ {k}) = \min  f \quad a. s,
$$
and that there exists a random variable  $x^{*}$  taking values in  $\arg \min f$  s.t.  $x_{k} \to x^{*}$  a.s.
In the next corollary, we derive explicit rates for specific choices of the parameters.
Corollary 1. Under the assumptions of Theorem 1, let  $x^{*} \in \arg \min f$ . Then, the following hold:
(i) Let  $\theta \in (1/2, 1)$  and  $\rho \in \mathbb{R}$  such that  $\theta + \rho > 1$ . For every  $k \in \mathbb{N}$ , let  $\alpha_k = \alpha(k + 1)^{-\theta}$  and  $h_k = h(k + 1)^{-\rho}$  with  $\alpha > 0$  and  $h > 0$ . Then
$$
\mathbb {E} [ f (\bar {x} _ {k}) - \min  f ] \leq \frac {C}{\alpha k ^ {1 - \theta}} + o \left(\frac {1}{k ^ {1 - \theta}}\right),
$$
for some constant  $C$  provided in the proof.
(ii) For every  $k \in \mathbb{N}$ , let  $\alpha_{k} = \alpha$  and  $h_k = h$  with  $\alpha, h > 0$ . Then
$$
\mathbb {E} [ f (\bar {x} _ {k}) - \min  f ] \leq \frac {\| x _ {0} - x ^ {*} \| ^ {2}}{2 \alpha k} + \frac {c d L _ {0} ^ {2}}{\ell} \alpha + L _ {0} h,
$$
where  $c$  is a constant independent of the dimension.
(iii) Fix an accuracy  $\varepsilon \in (0,1)$  and let  $K\geq 8(cL_0^2\| x_0 - x^*\| ^2)(d / \ell)\varepsilon^{-2}$ . Set  $\alpha_{k} = \sqrt{\frac{\ell}{d}}\frac{\|x_{0} - x^{*}\|}{\sqrt{2cK}L_{0}}$  and  $h_k = h\leq \varepsilon /(2L_0)$  for every  $k\leq K$ . Then
$$
\mathbb {E} \left[ f (\bar {x} _ {K}) - \min  f \right] \leq \varepsilon
$$
and the complexity in terms of number of iterations is  $\mathcal{O}((d / \ell)\varepsilon^{-2})$
Discussion. The bound in Theorem 1 depends on the initialization and on an additional quantity that can be interpreted as an approximation error. The latter is composed of two parts. The first one is generated by the approximation of the gradient of the smoothed function; while the second, involving  $h_k$ , is generated by the smoothing. Since the rate depends on  $1 / \sum_{i=0}^{k} \alpha_i$ , we would like to choose the stepsize as large as possible. However, to get convergence, we need to make the approximation errors
vanish sufficiently fast. To guarantee this, as we can observe from Theorem 1, we need to impose some conditions on the stepsize  $\alpha_{k}$  and on the smoothing parameter  $h_k$  (i.e. Assumption 2), that will slow down the decay of the first term. In Corollary 1, we provide two choices of parameters: the first one satisfies Assumption 2 and ensures convergence; the second one corresponds to constant stepsize and smoothing parameter. For the first choice, we recover the rate of the subgradient method in terms of  $k$ . In particular, for  $\theta$  approaching  $1/2$ , the convergence rate is arbitrarily close to  $O(k^{-1/2})$  and is similar to the one derived in [17, Theorem 2]. The dependence on the dimension depends on the choice of the constant  $\alpha$ . The optimal dependence is obtained with the choice  $\alpha = \sqrt{\ell / d}$ . Indeed, in that case, the complexity in the number of iterations is of the order  $\mathcal{O}((d / \ell)\varepsilon^{-2})$ , which is better than the one achieved by [17, Theorem 2] and [39]. Note that also [46, Corollary 1] and [22, Theorem 2.4] obtain a worse complexity in terms of the number of iterations (since they use a single direction), but the same complexity in the number of function evaluations. Clearly, since multiple directions are used, a single iteration of O-ZD will be more expensive than one iteration of [46, 22]. However, our algorithm is more efficient if the  $\ell$  function evaluations required at each iteration can be parallelized. On the more theoretical side, we observe that the advantage of multiple orthogonal directions is in the tighter bounds for the variance of the estimator, namely for  $\mathbb{E}[||g_{(G_k,h_k)}(x_k)||^2]$  - see [46, Lemma 5] and Lemma 4.
3.2 Non-smooth Non-convex Setting
To analyze the non-convex setting, following [39], we provide a bound on the averaged expected square norm of the gradient of the smoothed target. In particular, we use the following notation:
$$
\eta_ {k} ^ {(h)} := \left(\sum_ {i = 0} ^ {k} \alpha_ {i} \mathbb {E} [ \| \nabla f _ {h} (x _ {i}) \| ^ {2} ]\right) / A _ {k}, \quad \text {w h e r e} A _ {k} := \sum_ {i = 0} ^ {k} \alpha_ {i}.
$$
Next, we state the main theorem for the non-convex non-smooth setting.
Theorem 2 (Non-smooth non-convex). Under Assumption 1, let  $(x_{k})_{k\in \mathbb{N}}$  be a sequence generated by Algorithm 1 with, for every  $k\in \mathbb{N}$ ,  $h_k = h$  for some  $h > 0$ . Then
$$
\eta_ {k} ^ {(h)} \leq S _ {k} / A _ {k} \quad w i t h \quad S _ {k} := f _ {h} (x _ {0}) - \min  f + c \frac {L _ {0} ^ {3} d \sqrt {d}}{\ell} \sum_ {i = 0} ^ {k} \frac {\alpha_ {i} ^ {2}}{h}.
$$
In the next corollary, we derive the rates for specific choices of the parameters.
Corollary 2. Under the assumptions of Theorem 2, the following statements hold.
(i) If  $\alpha_{k} = \alpha (k + 1)^{-\theta}$  with  $\alpha >0$  and  $\theta \in (1 / 2,1)$ , then
$$
\eta_ {k} ^ {(h)} \leq C \frac {f _ {h} (x _ {0}) - \operatorname* {m i n} f}{\alpha k ^ {1 - \theta}} + o \left(\frac {1}{k ^ {1 - \theta}}\right),
$$
where  $C$  is a constant independent of the dimension.
(ii) If  $\alpha_{k} = \alpha$  with  $\alpha > 0$  for every  $k \in \mathbb{N}$ , then
$$
\eta_ {k} ^ {(h)} \leq \frac {f _ {h} \left(x _ {0}\right) - \operatorname* {m i n} f}{\alpha k} + \frac {c L _ {0} ^ {3} d \sqrt {d}}{\ell h} \alpha ,
$$