Title: An Optimal Structured Zeroth-order Algorithm for Non-smooth Optimization
Venue: NeurIPS 2023
Accepted: yes
Keywords: nonsmooth optimization;zeroth order optimization;nonsmooth zeroth-order
Abstract: Finite-difference methods are a class of algorithms designed to solve black-box optimization problems by approximating a gradient of the target function on a set of directions. In black-box optimization, the non-smooth setting is particularly relevant since, in practice, differentiability and smoothness assumptions cannot be verified. To cope with nonsmoothness, several authors use a smooth approximation of the target function and show that finite difference methods approximate its gradient. Recently, it has been proved that imposing a structure in the directions allows improving performance. However, only the smooth setting was considered. To close this gap, we introduce and analyze O-ZD, the first structured finite-difference algorithm for non-smooth black-box optimization. Our method exploits a smooth approximation of the target function and we prove that it approximates its gradient on a subset of random {\em orthogonal} directions. We analyze the convergence of O-ZD under different assumptions.  
For non-smooth convex functions, we obtain the optimal complexity. In the non-smooth non-convex setting, we characterize the number of iterations needed to bound the expected norm of the smoothed gradient. For smooth functions, our analysis recovers existing results for structured zeroth-order methods for the convex case and extends them to the non-convex setting. We conclude with numerical simulations where assumptions are satisfied, observing that our algorithm has very good practical performances.

An Optimal Structured Zeroth-order Algorithm for Non-smooth Optimization

Abstract

Finite-difference methods are a class of algorithms designed to solve black-box optimization problems by approximating a gradient of the target function on a set of directions. In black-box optimization, the non-smooth setting is particularly relevant since, in practice, differentiability and smoothness assumptions cannot be verified. To cope with nonsmoothness, several authors use a smooth approximation of the target function and show that finite difference methods approximate its gradient. Recently, it has been proved that imposing a structure in the directions allows improving performance. However, only the smooth setting was considered. To close this gap, we introduce and analyze O-ZD, the first structured finitedifference algorithm for non-smooth black-box optimization. Our method exploits a smooth approximation of the target function and we prove that it approximates its gradient on a subset of random orthogonal directions. We analyze the convergence of O-ZD under different assumptions. For non-smooth convex functions, we obtain the optimal complexity. In the non-smooth non-convex setting, we characterize the number of iterations needed to bound the expected norm of the smoothed gradient. For smooth functions, our analysis recovers existing results for structured zeroth-order methods for the convex case and extends them to the non-convex setting. We conclude with numerical simulations where assumptions are satisfied, observing that our algorithm has very good practical performances.

1 Introduction

Black-box optimization problems are a class of problems for which only values of the target function are available and no first-order information is provided. Typically, these problems arise when the evaluation of the objective function is based on a simulation and no analytical form of the gradient is accessible or its explicit calculation is too expensive [14, 45, 36, 35].

To face these problems, different methods that do not require first-order information have been proposed - see for instance [39, 47, 19, 26, 20, 30, 33] and references therein. These techniques are called derivative-free methods and a wide class of these is the class of finite-difference algorithms [31, 39, 17]. These iterative procedures mimic first-order optimization strategies by replacing the gradient of the objective function with an approximation built through finite differences in random directions.

Two types of finite-difference methods can be identified: unstructured and structured ones, depending on the way in which the random directions are generated. In the former, directions are sampled i.i.d. from some distribution [39, 17, 10, 46] while in the latter, directions have to satisfy some structural constraints, e.g. orthogonality [32, 42]. Several authors [32, 42, 5, 12] theoretically and empirically

observed that imposing orthogonality among the directions provides better performance than using unstructured directions. Intuitively, imposing orthogonality allows us to avoid cases in which the gradient approximation is built using similar or redundant directions.

Notably, methods based on structured finite differences have been analyzed only for smooth and convex (or specific non-convex) target functions. This represents a strong limitation, since smoothness and convexity are in practice hardly verifiable, due to the nature of black-box optimization problems. The aim of this paper is the analysis of structured finite difference algorithms dropping smoothness and convexity assumptions.

For unstructured finite-difference methods, a common way to face nonsmoothness consists in introducing a smooth approximation of the target function, also known as "smoothing" [6, 16] and using it as a surrogate of the target. Although the gradient of the smoothing is not computable, different authors showed that for certain types of smoothing the unstructured finite-difference approximation provides an unbiased estimation of such a gradient - see, for example, [39, 46, 18, 22, 21, 24]. This key observation allows to prove that unstructured finite-difference methods approximate a solution in different nonsmooth settings [17, 39, 46, 18, 22, 34].

For structured finite-difference methods, the analysis in the nonsmooth setting is not available. A key step which is missing is the proof of the fact that the surrogate of the gradient built with structured directions is an estimation of the gradient of a suitable smoothing.

In this paper, we close the gap, and introduce O-ZD, a structured finite-difference algorithm in the non-smooth setting. The algorithm builds an approximation of the gradient of a smoothed target using a set of ℓ ≤ d orthogonal directions. We analyze the procedure proving that our finite-difference surrogate is an unbiased estimation of the gradient of a smoothing. We provide convergence rates for non-smooth convex functions with optimal dependence on the dimension [17] and rates for the non-smooth non-convex setting (for which lower bounds are not known [34]). Moreover, for non-smooth convex functions we provide the first proof of convergence of the iterates for structured finite differences in this setting. For smooth convex functions, we recover the standard results for structured zeroth-order methods [32, 42]. We conclude with numerical illustrations. To the best of our knowledge, this is the first work on nonsmooth finite-difference method with structured directions.

The paper is organized as follows. In Section 2, we introduce the problem and the algorithm. In Section 3, we state and discuss the main results. In Section 4 we provide some experiments and in 5 some final remarks.

2 Problem Setting & Algorithm

Given a function f : R d → R and assuming that f has at least a minimizer in R d , we consider the problem to find

In particular, we consider the non-smooth setting where f might be non-differentiable. To solve problem (1), we propose a zeroth-order algorithm, namely an iterative procedure that uses only function values. At every iteration k ∈ N , a first-order information of f is approximated with finite-differences using a set of ℓ ≤ d random orthogonal directions ( p ( i ) k ) ℓ i =1 . Such orthogonal directions are represented as rotations (and reflections) of the first ℓ vectors of the canonical basis ( e i ) ℓ i =1 . We set p ( i ) k = G k e i , where G k belongs to the orthogonal group defined as

̸

Methods for generating orthogonal matrices are discussed in Appendix D. Given G ∈ O ( d ) , 0 < ℓ ≤ d and h > 0 , we consider the following central finite-difference surrogate of the gradient information

Then, we introduce the following algorithm.

Starting from an initial guess x 0 ∈ R d , at every iteration k ∈ N , the algorithm samples an orthogonal matrix G k i.i.d. from the orthogonal group O ( d ) and computes a surrogate g ( G k ,h k ) of the gradient at the current iterate x k . Then, it computes x k +1 by moving in the opposite direction of g ( G k ,h k ) ( x k ) . This approach belongs to the class of structured finite-difference methods, where a bunch of orthogonal directions is used to approximate a gradient of the target function [32, 42]. Such algorithms have been proposed and analyzed only for smooth functions. To cope with this limitation, and extend the analysis to the nonsmooth setting, we exploit a smoothing technique. For a fixed a probability measure ρ on R d and a positive parameter h > 0 called smoothing parameter , we define the following smooth surrogate of f

As shown in [6], f h,ρ is differentiable even when f is not. In the literature, different authors have used this strategy to face non-smooth zeroth-order optimization with random finite-difference methods [18, 39, 49, 22] fixing specific smoothing distribution, but no one applied and analyze it for structured methods.

In this work, ρ is the uniform distribution over the ℓ 2 unit ball B d , defining the smooth surrogate

where vol ( B d ) denotes the volume of B d . One of our main contributions is the following Lemma which shows that the surrogate proposed in (2) is an unbiased estimator of the gradient of the smoothing in (4).

Lemma 1 (Smoothing Lemma) . Given a probability space (Ω , F , P ) , let G : Ω → O ( d ) be a random variable where O ( d ) is the orthogonal group endowed with the Borel σ -algebra. Assume that the probability distribution of G is the (normalized) Haar measure. Let h > 0 and let g be the gradient surrogate defined in eq. (2) . Then,

where f h is the smooth approximation of the target function f defined in eq. (4) .

The proof of Lemma 1 is provided in Appendix A.1.

Remark 1. Note that Lemma 1 holds also using g F ( G,h ) or g S ( G,h ) defined as

since E G [ g ( G,h ) ( x )] = E G [ g F ( G,h ) ( x )] = E G [ g S ( G,h ) ( x )] . Despite these two estimators being computationally cheaper than the proposed one, we use central finite differences since they allow us to derive a better bound for E G [ ∥ g ( G,h ) ( x ) ∥ 2 ] as observed in [46] for the case ℓ = 1 .

Thanks to Lemma 1, we can interpret each step of Algorithm 1 as a Stochastic Gradient Descent (SGD) on the smoothed function f h k . But the analysis of the proposed algorithm does not follow from the SGD one for two reasons. First, the smoothing parameter h k changes along the iterations; second (and more importantly), the set of minimizers of f h and f are different in general. However, we will take advantage of the fact that f h can be seen as an approximation of f . The relationship between f and its approximation f h depends on the properties of f - see Proposition 1 in Appendix A.

Our main contributions are the theoretical and numerical analysis of Algorithm 1 under different choices of the free parameters α k , h k , namely the stepsize and the smoothing parameter. To the best of our knowledge, Algorithm 1 is the first structured zeroth-order method for non-smooth optimization.

2.1 Related Work

In practice, the advantage of the use of structured directions in finite-difference methods has been observed in several applications [12] and motivated their study. In [5], the authors theoretically and empirically observed that structured directions provide a better approximation of the gradient with respect to unstructured (Gaussian and spherical) ones. Next, we review the most related works, showing the differences with our algorithm.

Unstructured Finite-differences. Most of existing works focused on the theoretical analysis of methods using a single direction to approximate the gradient - see e.g. [39, 46, 22, 45, 18, 24, 34, 17, 10]. The main results existing so far analyze the convergence of the function values in expectation. They provide convergence rates in terms of the number of function evaluations and explicitly characterize the dependence on the dimension of the ambient space.

Smooth setting: In [39, 17], a finite difference algorithm with a single direction is analyzed. Rates on the expected function values are shown. In [39] the dependence on the dimension in the rates is not optimal. In [17], both single and multiple direction cases are analyzed and lower bounds are derived. However, only the convex setting is considered. In [24], both convex and non-convex settings are analyzed. They obtain optimal dependence on dimension in the complexity for convex functions. However, only the single-direction case is considered and only the smooth setting is analyzed. Comparing the result with our rates, Algorithm 1 achieves the same dependence on the dimension in the complexity taking ℓ as a fraction of d . Note that, by parallelizing the computation of the function evaluations, we can get a better result.

Non-smooth setting: In [39, 17] the non-smooth setting is also analyzed. However, only the single direction case has been considered and both algorithms do not achieve the lower bound. More precisely, for convex functions, a complexity of O ( d 2 ε -2 ) is achieved by [39] and O ( d log( d ) ε -2 ) by [17] while our algorithm gets the optimal dependence. Moreover, note that in [17] the strategy adopted (also called "double smoothing") requires tuning one more sequence of parameters, which is a challenging problem in practice, and only the convex setting is considered. In [39], also the non-convex setting is analyzed by bounding the expected norm of the smoothed gradient. However, they obtain a complexity of O ( d 3 ε -2 h -1 ) while our algorithm gets a better dependence on the dimension. In [46], the optimal complexity is obtained for convex functions. However, the author does not analyze the non-convex setting. Moreover, note that, despite the complexity in terms of function evaluations being the same, our algorithm gets a better complexity in terms of the number of iterations since it uses multiple directions (and this is an advantage if we can parallelize the function evaluations). Furthermore, note that the method proposed in [46] can be seen as the special case of Algorithm 1 with ℓ = 1 . In [34] the single direction case is analyzed only in the non-convex setting. The dependence on the dimension of the complexity in the number of function evaluations achieved matches our result in this setting (again, in the number of iterations our method obtains a better dependence).

Structured Finite-difference. In [32, 42], authors analyze structured finite differences in both deterministic and stochastic settings. However, only the smooth convex setting is considered. In [12] orthogonal matrices are used to build directions but no analysis is provided. In [25], finite-difference with coordinate directions is analyzed. At every iteration, d +1 function evaluations are performed to compute the estimator and only the smooth setting is considered.

3 Main Results

In this section, we analyze Algorithm 1 considering both non-smooth and smooth problems. We present the rates obtained by our method for convex and non-convex settings and compare them with those obtained by state-of-the-art methods. Proofs are provided in Appendix B. In the following, we call complexity in the number of iterations / function evaluations , respectively, the number of iterations / function evaluations required to achieve an accuracy ε ∈ (0 , 1) .

3.1 Non-smooth Convex Setting

In this section, we provide the main results for non-smooth convex functions. In particular, we will assume that the target function is convex and satisfy the following hypothesis.

Assumption 1 ( L 0 -Lipschitz continuous) . The function f is L 0 -Lipschitz continuous; i.e., for some L 0 > 0 ,

Note that this assumption implies that also f h is L 0 -Lipschitz continuous - see Proposition 1. Moreover, to analyze the algorithm, we will consider the following parameter setting.

Assumption 2 (Zeroth-order non-smooth convergence conditions) . The step-size sequence α k and the sequence of smoothing parameters h k satisfy the following conditions:

The assumption above is required to guarantee convergence to a solution. In particular, the first two conditions are common for subgradient method and stochastic gradient descent, while the third condition was already used in structured zeroth-order methods [32, 42] and links the decay of the smoothing parameter with the stepsize's one. An example of α k , h k that satisfy Assumption 2 is α k = k -θ and h k = k -ρ with θ ∈ (1 / 2 , 1) and ρ s.t. θ + ρ > 1 .

We state now the main theorem for non-smooth convex functions.

Theorem 1 (Non-smooth convex) . Under Assumption 1, assume that f is convex and let ( x k ) k ∈ N be a sequence generated by Algorithm 1. For every k ∈ N , denote A k = ∑ k i =0 α i and set ¯ x k := ∑ k i =0 α i x i /A k . Then

where c > 0 is a constant independent of the dimension and x ∗ is any solution in arg min f . Moreover, under Assumption 2, we have

and that there exists a random variable x ∗ taking values in arg min f s.t. x k → x ∗ a.s.

In the next corollary, we derive explicit rates for specific choices of the parameters.

Corollary 1. Under the assumptions of Theorem 1, let x ∗ ∈ arg min f . Then, the following hold:

(i) Let θ ∈ (1 / 2 , 1) and ρ ∈ R such that θ + ρ > 1 . For every k ∈ N , let α k = α ( k +1) -θ and h k = h ( k +1) -ρ with α > 0 and h > 0 . Then

for some constant C provided in the proof.

(ii) For every k ∈ N , let α k = α and h k = h with α, h > 0 . Then

where c is a constant independent of the dimension.

(iii) Fix an accuracy ε ∈ (0 , 1) and let K ≥ 8( cL 2 0 ∥ x 0 -x ∗ ∥ 2 )( d/ℓ ) ε -2 . Set α k = √ ℓ d ∥ x 0 -x ∗ ∥ √ 2 cKL 0 , and h = h ≤ ε/ (2 L ) for every k ≤ K . Then

and the complexity in terms of number of iterations is O (( d/ℓ ) ε -2 ) .

Discussion. The bound in Theorem 1 depends on the initialization and on an additional quantity that can be interpreted as an approximation error. The latter is composed of two parts. The first one is generated by the approximation of the gradient of the smoothed function; while the second, involving h k , is generated by the smoothing. Since the rate depends on 1 / ∑ k i =0 α i , we would like to choose the stepsize as large as possible. However, to get convergence, we need to make the approximation errors

vanish sufficiently fast. To guarantee this, as we can observe from Theorem 1, we need to impose some conditions on the stepsize α k and on the smoothing parameter h k (i.e. Assumption 2), that will slow down the decay of the first term. In Corollary 1, we provide two choices of parameters: the first one satisfies Assumption 2 and ensures convergence; the second one corresponds to constant stepsize and smoothing parameter. For the first choice, we recover the rate of the subgradient method in terms of k . In particular, for θ approaching 1 / 2 , the convergence rate is arbitrarily close to O ( k -1 / 2 ) and is similar to the one derived in [17, Theorem 2]. The dependence on the dimension depends on the choice of the constant α . The optimal dependence is obtained with the choice α = √ ℓ/d . Indeed, in that case, the complexity in the number of iterations is of the order O (( d/ℓ ) ε -2 ) , which is better than the one achieved by [17, Theorem 2] and [39]. Note that also [46, Corollary 1] and [22, Theorem 2.4] obtain a worse complexity in terms of the number of iterations (since they use a single direction), but the same complexity in the number of function evaluations. Clearly, since multiple directions are used, a single iteration of O-ZD will be more expensive than one iteration of [46, 22]. However, our algorithm is more efficient if the ℓ function evaluations required at each iteration can be parallelized. On the more theoretical side, we observe that the advantage of multiple orthogonal directions is in the tighter bounds for the variance of the estimator, namely for E [ ∥ g ( G k ,h k ) ( x k ) ∥ 2 ] - see [46, Lemma 5] and Lemma 4.

3.2 Non-smooth Non-convex Setting

To analyze the non-convex setting, following [39], we provide a bound on the averaged expected square norm of the gradient of the smoothed target. In particular, we use the following notation:

Next, we state the main theorem for the non-convex non-smooth setting.

Theorem 2 (Non-smooth non-convex) . Under Assumption 1, let ( x k ) k ∈ N be a sequence generated by Algorithm 1 with, for every k ∈ N , h k = h for some h > 0 . Then

In the next corollary, we derive the rates for specific choices of the parameters.

Corollary 2. Under the assumptions of Theorem 2, the following statements hold.

(i) If α k = α ( k +1) -θ with α > 0 and θ ∈ (1 / 2 , 1) , then

where C is a constant independent of the dimension.

(ii) If α k = α with α > 0 for every k ∈ N , then

where c is a constant independent of the dimension.

(iii) Let ε ∈ (0 , 1) , let K ≥ 4( f h ( x 0 ) -min f ) cL 3 0 d √ dε -2 / ( ℓh ) and choose α = √ ( f h ( x 0 ) -min f ) ℓh KcL 3 0 d √ d . Then we have that η ( h ) K ≤ ε . Thus, the number of function evaluations required to get a precision η h k ≤ ε is of the order O ( d √ dh -1 ε -2 ) .

Relying on the results in [34], we show that this is related to a precise notion of approximate stationarity. To do so, we need to introduce a definition of subdifferential which is suitable to this setting. As shown in [34] the Clarke subdifferential is not the right notion, and the approximate Goldstein subdifferential should be used instead.

Definition 1 (Goldstein subdifferential and stationary point) . Under Assumption 1, let x ∈ R d and h > 0 . The h -Goldstein subdifferential of f at x is ∂ h f ( x ) := conv ( ∪ y ∈ B d h ( x ) ∂f ( y )) where ∂f is the Clarke subdifferential [13] and B d h ( x ) is the ball centered in x with radius h . For ε ∈ (0 , 1) , a point x ∈ R d is a ( h, ε ) -Goldstein stationary point for the function f if min {∥ g ∥ | g ∈ ∂ h f ( x ) } ≤ ε .

Corollary 3. Under the same assumptions of Theorem 2, fix K ∈ N and let I be a random variable taking values in { 0 , . . . , K -1 } such that, for all i , P [ I = i ] = α i /A K -1 . Let also S k be defined as in Theorem 2. Then

In the setting of Corollary 2 (iii), we have also that E I [ min {∥ η ∥ 2 : η ∈ ∂f h ( x I ) } ] ≤ ε .

Discussion. In Theorem 2, we fix the smoothing of the target, i.e. we consider h k constant, and we analyze the non-smooth non-convex setting providing a rate on the expected norm of the smoothed gradient. The resulting bound is composed of two parts. The first part is very natural, and due to the functional value at the initialization. The second part is the approximation error. Recall that Assumption 1 holds, and therefore f h ≤ f + L 0 h due to Proposition 1. This suggests taking h as small as possible in order to reduce the gap between f h and f . However, taking h too small would make the approximation error very big. In our analysis, we consider the case with h constant. Moreover, as for the convex case, the speed of the rate depends on A k and so we would like to take the stepsize as large as possible. But to control the approximation error, we need to assume α 2 k ∈ ℓ 1 . In Corollary 2, we consider two choices of stepsize. The first choice satisfies the property of α 2 k ∈ ℓ 1 , while the second one analyzes the case of constant step-size. Comparing our rate to the one in [39] we see that we obtain a better dependence on the dimension in the complexity, both in terms of iterations and function evaluations. Our results match the one of [34, Theorem 3.2] in terms of rate and in terms of function evaluations. We get a better dependence on the dimension in the number of iterations. Note again that, despite the complexity in terms of the number of function evaluations being the same, the possibility of parallelization for the function evaluations yields a better result for our method. As for the convex setting, we have a tighter upper-bound on the variance of the estimator of the smoothed gradient with respect to the dimension - see [34, Lemma D.1]. Goldstein stationarity has been used to assess the approximate stationarity for first-order methods as well, see [15]. The latter work shows that a cutting plane algorithm achieves a rate of O ( dε -3 ) for Lipschitz functions.

3.3 Smooth Convex setting

We consider now the smooth setting, i.e. we assume that the target function satisfies the following hypothesis.

Assumption 3 ( L 1 -Smooth) . The function f is L 1 -smooth; i.e. the function f is differentiable and, for some L 1 > 0 ,

This is the standard assumption for analyzing first-order methods and has been used in many other works in the literature for zeroth-order algorithms - see e.g. [39, 17]. As shown in previous works, if f satisfies Assumption 3 then also f h satisfies it - see Proposition 1. We will consider also the following assumptions on the stepsize and the smoothing in order to guarantee convergence.

Assumption 4 (Smooth zeroth-order convergence conditions) . The stepsize sequence ( α k ) k ∈ N and the smoothing sequence ( h k ) k ∈ N satisfy the following conditions:

Moreover, α k ≤ ¯ α < ℓ/dL 1 for every k ∈ N .

Note that this is a weaker version of Assumption 2. Next, we state the main theorem for convex smooth functions.

Theorem 3 (Smooth convex) . Under Assumptions 3 and 4, let ( x ) N be a sequence generated by k k ∈ Algorithm 1 and x ∗ ∈ arg min f . For every k ∈ N , set A k = ∑ k i =0 α i and ¯ x k = k ∑ i =0 α i x i /A k . Then, for every k ∈ N ,

Corollary 4. Under the same Assumptions of Theorem 3, the following hold.

-θ

where C is a constant provided in the proof. Moreover, if α < ℓ/ (2 dL 1 ) , lim k →∞ f ( x k ) = min f a.s. and there exists a random variable ˆ x taking values in arg min f s.t. x k → ˆ x a.s.

(ii) If for every k ∈ N we set α k = α > 0 and 0 < h k ≤ h , then

where C 1 , C 2 , C 3 and C 4 are non-negative constants.

Discussion. As in the previous cases, the bound in Theorem 3 is composed by two terms: the error due to the initialization and the one due to the approximation. An important difference with the results in the non-smooth setting is that every term in the approximation error is decreasing with respect to the smoothing parameter h k . This allows obtaining convergence also with the constant step-size scheme, taking h k ∈ ℓ 1 . In Corollary 4 (i), we recover the result of [32, Theorem 5.4] with a specific choice of parameters α, h (up to constants). The complexity depends on the choice of α . Note that by Assumption 4, α < ℓ/ ( L 1 d ) thus the dependence on the dimension in the rate will be at least d/ℓ . In particular, taking α = ℓ/ (2 dL 1 ) , we obtain the optimal complexity of O ( dε -1 ) in terms of function evaluations. This result has a better dependence on the dimension than [39]. In Corollary 4 (ii), the dependence on the dimension in the complexity depends on the choice of α and h . Moreover, the rate obtained is equal (up to constants) to the rate obtained in [32] in the same setting, i.e. O (1 /k ) (in which we hide the dependence on d and ℓ ). As for [32], for the first setting we can prove the almost sure convergence of the iterates.

3.4 Smooth Non-Convex setting

To analyze the smooth non-convex setting, we introduce the following notation:

Note that, in comparison with the quantity defined in Section 3.2, here η k is related to the exact objective function f and not to its smoothed version f h . Next, we state the main result for smooth non-convex functions.

Theorem 4 (Smooth non-convex) . Suppose that Assumption 3 holds and assume that, for every k ∈ N , α k ≤ ¯ α < ℓ/ (2 dL 1 ) . Let ( x k ) k ∈ N be a sequence generated by Algorithm 1. Then

Corollary 5. Under the assumptions of Theorem 4, the following hold.

(i) If α k = α ≤ ¯ α and h k = hk -θ with h > 0 and θ > 1 , then

where C 1 and C 2 are constants provided in the proof.

(ii) If α k = α ≤ ¯ α and h k = h > 0 , then

where C 1 and C 2 are constants provided in the proof.

Discussion. As for the convex case, every term in the approximation error depends on the smoothing parameter h k . In Corollary 5 (i), we take constant step-size and h k ∈ ℓ 1 . With this choice of parameters, we get a rate of O (1 /k ) which matches with the result obtained by [39]. The dependence on the dimension depends on the choice of α and h . Note that α < ℓ/ (2 dL 1 ) , thus taking h = O (1 /d ) , in the rate we get a dependence on the dimension of d/ℓ . Taking for instance α = ℓ/ (3 dL 1 ) and h = O (1 /d ) , we get a complexity of O ( dε -1 ) in terms of function evaluations.

4 Numerical Results

In this section, we provide some numerical experiments to assess the performances of our algorithm. We consider two target functions: a convex smooth one and a convex non-smooth one. Details on target functions and parameters of the algorithms are reported in Appendix C. To report our findings, we run the experiments 10 times and provide the mean and standard deviation of the results.

How to choose the number of directions? In these experiments, we set a fixed budget of 4000 function evaluations and we consider d = 50 . We investigate how the performance of Algorithm 1 changes as the value of ℓ increases. In Figure 1, we observe the mean sequence f ( x k ) -f ( x ∗ ) after each function evaluation. If ℓ > 1 , then the target function values are repeated 2 ℓ times, since we need to perform 2 ℓ function evaluations to do one iteration. For a sufficiently large budget, increasing the number of directions ℓ leads to better results compared to using a single direction in both smooth and non-smooth settings.

0

500

1000

1500

2000

2500

3000

3500

4000

function evaluations

10 1

10 2

10 3

f

(

xk

) -

f

(

x

*

)

Smooth Convex function

ℓ

=1

ℓ

=5

ℓ

=10

ℓ

=20

ℓ

=50

0

500

1000

1500

2000

2500

3000

3500

4000

function evaluations

10 3

7×10 2

8×10 2

9×10 2

f

(

xk

) -

f

(

x

*

)

NonSmooth Convex function

Comparison with finite-difference methods. Now, we compare Algorithm 1 with other finitedifference methods. More precisely, we consider finite differences with single (and multiple) Gaussian (and spherical) directions. The budget of function evaluations is 1000 and the ambient dimension is d = 10 . For multiple direction methods, we fix the number of directions ℓ = d . Further experiments are provided in Appendix F.

0

200

400

600

800

1000

function evaluations

10 -2

10 -1

10 0

10 1

f

(

xk

) -

f

(

x

*

)

Smooth Convex Target

Single Gaussian

Single Spherical

Multi Gaussian

Multi Spherical

Ours

0

200

400

600

800

1000

function evaluations

10 0

10 1

f

(

xk

) -

f

(

x

*

)

NonSmooth Convex Target

In Figure 2, we plot the sequence f ( x k ) -f ( x ∗ ) with respect to the number of function evaluations. While in terms of rates and complexity the different algorithms are the same, Algorithm 1 shows better performances than random directions approaches, and we believe this is due to the use of structured (i.e. orthogonal) directions. Indeed, orthogonal directions yield a better approximation of first-order information with respect to other methods. The practical advantages of structured directions were already observed in [32, 42, 5, 12] and these experiments confirm that the good practical behavior holds even in the nonsmooth setting.

5 Conclusion

Weintroduced and analyzed O-ZD a zeroth-order algorithm for non-smooth zeroth-order optimization. We analyzed the algorithm and derived rates for non-smooth and smooth functions. This work opens different research directions. An interesting one would be the introduction of a learning procedure for the orthogonal directions. Such an approach could have significant practical applications.

Acknowledgments and Disclosure of Funding

This project has been supported by the TraDE-OPT project, which received funding from the European Union's Horizon 2020 research and innovation program under the Marie Skłodowska-Curie grant agreement No 861137. L. R. and M. R. acknowledge the financial support of the European Research Council (grant SLING 819789), the AFOSR projects FA9550-18-1-7009 (European Office of Aerospace Research and Development), the EU H2020-MSCA-RISE project NoMADS - DLV777826, and the Center for Brains, Minds and Machines (CBMM), funded by NSF STC award CCF-1231216. S. V. and L. R. acknowledge the support of the AFOSR project FA8655-22-1-7034. The research by S. V. and C. M. has been supported by the MIUR Excellence Department Project awarded to Dipartimento di Matematica, Università di Genova, CUP D33C23001110001. S. V. and C. M. are members of the Gruppo Nazionale per l'Analisi Matematica, la Probabilità e le loro Applicazioni (GNAMPA) of the Istituto Nazionale di Alta Matematica (INdAM). This work represents only the view of the authors. The European Commission and the other organizations are not responsible for any use that may be made of the information it contains.

References

Hans Wilhelm Alt. Linear Functional Analysis: An Application-Oriented Introduction . Springer London, London, 2016.

T. W. Anderson, I. Olkin, and L. G. Underhill. Generation of random orthogonal matrices. SIAM Journal on Scientific and Statistical Computing , 8(4):625-629, 1987.

J.B. Baillon and G. Haddad. Quelques propriétés des opérateurs angle-bornés etn-cycliquement monotones. Israel Journal of Mathematics , 26(2):137-150, Jun 1977.

A. Barvinok. Approximating orthogonal matrices by permutation matrices. Pure and applied mathematics quarterly , 2, 11 2005.

A. S. Berahas, L. Cao, K. Choromanski, and K. Scheinberg. A theoretical and empirical comparison of gradient approximations in derivative-free optimization. Foundations of Computational Mathematics , 22(2):507-560, Apr 2022.

D. P. Bertsekas. Stochastic optimization problems with nondifferentiable cost functionals. Journal of Optimization Theory and Applications , 12(2):218-231, Aug 1973.

Å. Björck. Numerics of gram-schmidt orthogonalization. Linear Algebra and its Applications , 197-198:297-316, 1994.

C. Boutsidis and A. Gittens. Improved matrix algorithms via the subsampled randomized hadamard transform. SIAM Journal on Matrix Analysis and Applications , 34(3):1301-1340, 2013.

Lars Buitinck, Gilles Louppe, Mathieu Blondel, Fabian Pedregosa, Andreas Mueller, Olivier Grisel, Vlad Niculae, Peter Prettenhofer, Alexandre Gramfort, Jaques Grobler, Robert Layton, Jake VanderPlas, Arnaud Joly, Brian Holt, and Gaël Varoquaux. API design for machine learning software: experiences from the scikit-learn project. In ECML PKDD Workshop: Languages for Data Mining and Machine Learning , pages 108-122, 2013.

R. Chen and S. Wild. Randomized derivative-free optimization of noisy convex functions. arXiv preprint arXiv:1507.03332 , 2015.

K. Choromanski, M. Rowland, W. Chen, and A. Weller. Unifying orthogonal monte carlo methods. In International Conference on Machine Learning , pages 1203-1212. PMLR, 2019.

K. Choromanski, M. Rowland, V. Sindhwani, R. Turner, and A. Weller. Structured evolution with compact architectures for scalable policy optimization. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 970-978. PMLR, 10-15 Jul 2018.

F. H. Clarke. Optimization and Nonsmooth Analysis . Society for Industrial and Applied Mathematics, 1990.

A. R. Conn, K. Scheinberg, and L. N. Vicente. Introduction to Derivative-Free Optimization . Society for Industrial and Applied Mathematics, 2009.

D. Davis, D. Drusvyatskiy, Y. T. Lee, S. Padmanabhan, and G. Ye. A gradient sampling method with complexity guarantees for lipschitz functions in high and low dimensions. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 6692-6703. Curran Associates, Inc., 2022.

J. C. Duchi, P. L. Bartlett, L. Peter, and M. J. Wainwright. Randomized smoothing for stochastic optimization. SIAM Journal on Optimization , 22(2):674-701, 2012.

J. C. Duchi, M. I. Jordan, M. J. Wainwright, and A. Wibisono. Optimal rates for zero-order convex optimization: The power of two function evaluations. IEEE Transactions on Information Theory , 61(5):2788-2806, 2015.

A. Flaxman, A. Tauman Kalai, and B. McMahan. Online convex optimization in the bandit setting: Gradient descent without a gradient. In SODA '05 Proceedings of the sixteenth annual ACM-SIAM symposium on Discrete algorithms , pages 385-394, January 2005.

M. Fornasier, T. Klock, and K. Riedl. Consensus-based optimization methods converge globally in mean-field law. arXiv 2103.15130 , 2021.

P. I. Frazier. Bayesian Optimization , chapter 11, pages 255-278. INFORMS, 2018.

X. Gao, B. Jiang, and S. Zhang. On the information-adaptive variants of the admm: An iteration complexity perspective. Journal of Scientific Computing , 76(1):327-363, Jul 2018.

A. Gasnikov, A. Novitskii, V. Novitskii, F. Abdukhakimov, D. Kamzolov, A. Beznosikov, M. Takac, P. Dvurechensky, and B. Gu. The power of first-order smooth optimization for black-box non-smooth problems. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 7241-7265. PMLR, 17-23 Jul 2022.

A. Genz. Methods for generating random orthogonal matrices. In Monte-Carlo and QuasiMonte Carlo Methods 1998: Proceedings of a Conference held at the Claremont Graduate University, Claremont, California, USA, June 22-26, 1998 , pages 199-213. Springer, 2000.

S. Ghadimi and G. Lan. Stochastic first- and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization , 23(4):2341-2368, 2013.

G. N. Grapiglia. Worst-case evaluation complexity of a derivative-free quadratic regularization method. Optimization Letters , Feb 2023.

N. Hansen. The CMA Evolution Strategy: A Comparing Review , pages 75-102. Springer Berlin Heidelberg, Berlin, Heidelberg, 2006.

C. R. Harris, K. J. Millman, S. J. van der Walt, R. Gommers, P. Virtanen, D. Cournapeau, E. Wieser, J. Taylor, S. Berg, N. J. Smith, R. Kern, M. Picus, S. Hoyer, M. H. van Kerkwijk, M. Brett, A. Haldane, J. Fernández del Río, M. Wiebe, P. Peterson, P. Gérard-Marchant, K. Sheppard, T. Reddy, W. Weckesser, H. Abbasi, C. Gohlke, and T. E. Oliphant. Array programming with NumPy. Nature , 585(7825):357-362, September 2020.

A. Hedayat and W. D. Wallis. Hadamard matrices and their applications. The Annals of Statistics , 6(6):1184-1238, 1978.

J. D. Hunter. Matplotlib: A 2d graphics environment. Computing in Science & Engineering , 9(3):90-95, 2007.

N. K. Jain, U. Nangia, and J. Jain. A review of particle swarm optimization. Journal of The Institution of Engineers (India): Series B , 99(4):407-411, Aug 2018.

J. Kiefer and J. Wolfowitz. Stochastic estimation of the maximum of a regression function. Annals of Mathematical Statistics , 23:462-466, 1952.

D. Kozak, C. Molinari, L. Rosasco, L. Tenorio, and S. Villa. Zeroth order optimization with orthogonal random directions, 2021.

R. M. Lewis, V. Torczon, and M. W. Trosset. Direct search methods: then and now. Journal of Computational and Applied Mathematics , 124(1):191-207, 2000. Numerical Analysis 2000. Vol. IV: Optimization and Nonlinear Equations.

T. Lin, Z. Zheng, and M. Jordan. Gradient-free methods for deterministic and stochastic nonsmooth nonconvex optimization. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 26160-26175. Curran Associates, Inc., 2022.

S. Liu, P. Y. Chen, B. Kailkhura, G. Zhang, A. O. Hero III, and P. K. Varshney. A primer on zeroth-order optimization in signal processing and machine learning: Principals, recent advances, and applications. IEEE Signal Processing Magazine , 37(5):43-54, 2020.

H. Mania, A. Guy, and B. Recht. Simple random search of static linear policies is competitive for reinforcement learning. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

P. Mattila. Geometry of Sets and Measures in Euclidean Spaces: Fractals and Rectifiability . Cambridge Studies in Advanced Mathematics. Cambridge University Press, 1995.

F. Mezzadri. How to generate random matrices from the classical compact groups. arXiv preprint math-ph/0609050 , 2006.

Y. Nesterov and V. Spokoiny. Random gradient-free minimization of convex functions. Foundations of Computational Mathematics , 17:527-566, 2017.

Z. Opial. Weak convergence of the sequence of successive approximations for nonexpansive mappings. Bulletin of the American Mathematical Society , 73(4):591 - 597, 1967.

B. T. Polyak. Introduction to optimization. Optimization Software Inc., Publications Division, New York , 1:32, 1987.

M. Rando, C. Molinari, S. Villa, and L. Rosasco. Stochastic zeroth order descent with structured directions, 2022.

H. Robbins and D. Siegmund. A convergence theorem for non negative almost supermartingales and some applications. In J. S. Rustagi, editor, Optimizing Methods in Statistics , pages 233-257. Academic Press, 1971.

C. Rusu and L. Rosasco. Fast approximation of orthogonal matrices and application to pca. Signal Processing , 194:108451, 2022.

T. Salimans, J. Ho, X. Chen, S. Sidor, and I. Sutskever. Evolution strategies as a scalable alternative to reinforcement learning, 2017.

O. Shamir. An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. The Journal of Machine Learning Research , 18(1):1703-1713, 2017.

C. Totzeck. Trends in Consensus-Based Optimization , pages 201-226. Springer International Publishing, Cham, 2022.

T. Trogdon. On spectral and numerical properties of random butterfly matrices. Applied Mathematics Letters , 95:48-58, 2019.

F. Yousefian, A. Nedi´ c, and U. V . Shanbhag. On stochastic gradient and subgradient methods with adaptive steplength sequences. Automatica , 48(1):56-67, 2012.

A Auxiliary Results

In this appendix, we state and collect lemmas and propositions required to prove the main results.

Notation. In the following sections, we denote with F k the filtration σ ( G 1 , · · · , G k -1 ) . Moreover, to simplify the notation, we define g k as the gradient surrogate in eq.(2) at time-step k i.e. g k := g ( G k ,h k ) ( x k ) and g ( · ) := g ( G,h ) ( · ) for an arbitrary G ∈ O ( d ) and h > 0 . We denote the normalized Haar measure [37] by µ . We define the unit ball B d and the unit sphere S d -1 as follow

We denote by σ and σ N the spherical measure and the normalized spherical measure on S d -1 , respectively. Moreover, we denote with I d,ℓ ∈ R d × ℓ the (truncated) identity matrix.

Lemma 2. Let β ( S d -1 ) be the surface area of S d -1 and let I ∈ R d × d be the identity matrix. Then,

Proof. This result is proved in [21, Lemma 7.3, point (b)].

Lemma 3. Let ϕ : R d → R be a L -Lipschitz function . If u is uniformly distributed on S d -1 , then

for some numerical constant c > 0 .

Proof. The proof follows the same line as [46, Lemma 9].

A.1 Smoothing Lemma & Properties

In this appendix, we provide the proof of the Smoothing Lemma (i.e. Lemma 1).

Proof of Smoothing Lemma. By eq. (2),

By [37, Theorem 3.7],

Since v ( i ) is uniformly distributed on the sphere, which is symmetric with respect to the origin, we have

As a consequence of Stokes' Theorem (details in [18, Lemma 1] and [1, Theorem A8.8]), we get

Rearranging terms, we get the claim.

Proposition 1 (Smoothing properties) . Let f h be the smooth approximation of f defined in eq. (4) . Then the following hold:

If f is convex then f h is convex and, for every x ∈ R d ,

If f is L 0 -Lipschitz continuous - i.e. ∀ x, y ∈ R d , | f ( x ) -f ( y ) | ≤ L 0 ∥ x -y ∥ , then f h is L 0 -Lipschitz continuous, differentiable and for every x, y ∈ R d

If f is L 1 -smooth - i.e. f is differentiable and ∀ x, y ∈ R d , ∥∇ f ( x ) -∇ f ( y ) ∥ ≤ L 1 ∥ x -y ∥ then f h is L 1 -smooth and for every x ∈ R d ,

Proof. These are standard results proposed and proved in different works - see for example [16, Lemma 8],[21, Proposition 7.5],[34, Proposition 2.2],[49].

Lemma 4 (Approximation Error) . Let g ( · ) be the surrogate defined in eq. (2) for arbitrary h > 0 and G ∈ O ( d ) . Then the following hold:

(i) If f is L 0 -Lipschitz (see Assumption 1), then, for every x ∈ R d ,

where c is a numerical constant.

(ii) If f is L 1 -smooth (see Assumption 3), then, for every x ∈ R d ,

Proof. Note that, since directions are orthogonal, we have

By [37, Theorem 3.7],

where each v ( i ) is uniformly distributed on S d -1 .

( i ) : Set γ = E v ( i ) [ f ( x + hv ( i ) )] for every i (this expectation does not depend on i ). Then

Since v ( i ) is uniformly distributed on S d -1 , it satisfies ∥ v ( i ) ∥ 2 = 1 and by symmetry we have

The definition of γ yields

The claim follows by Lemma 3 and the fact that f ( x + hv ( i ) ) is hL 0 -Lipschitz continuous w.r.t to v ( i ) . ( ii ) : Equation (5) yields

where the last equation follows by symmetry. Adding and subtracting 〈 ∇ f ( x ) , hv ( i ) 〉 we derive

Denote by β ( S d -1 ) the surface area of S d -1 . The Descent Lemma [41] implies

By Lemma 2, we get the claim. Indeed,

A.2 Auxiliary results and proofs for the nonsmooth setting, convex, and nonconvex.

In this subsection, for every k , we will denote by F k the σ -algebra σ ( G 0 , . . . , G k -1 ) .

Lemma 5. Let f : R d → R be a lower semi-continuous function and denote with S = arg min f and f ∗ = min f . Then,

̸

Proof. Since (B) holds, we have that exists ( x k j ) j ∈ N subsequence of ( x k ) k ∈ N such that f ( x k j ) → f ∗ . Since S = ∅ and (A) we have that

Thus, the sequence ( x k ) k ∈ N is bounded and, therefore, also ( x k j ) j ∈ N is bounded. Taking a convergent subsequence ( x k j n ) n ∈ N of ( x k j ) j ∈ N , we have that exists x ∞ s.t.

Since f is assumed to be lower semi-continuous, we have that

Thus, we have that x ∞ ∈ S which implies, by (A), that

Hence, since x k j n is a subsequence of x k ,

and, therefore, x k → x ∞ ∈ S .

Lemma 6 (Convergence: convex non-smooth) . Assume that f is convex and L 0 Lipschitz continuous. Let ( x k ) k ∈ N be the sequence generated by Algorithm 1 and let x ∗ ∈ arg min f . Then, for every k ∈ N , the following inequality holds:

where c is some non-negative constant independent from the dimension. Moreover, if the stepsizes satisfy Assumption 2, we have

and there exists a random variable ˆ x taking values in in arg min f such that x k → ˆ x a.s.

Since f h k is convex by Proposition 1 and E [ g k |F k ] = ∇ f h k ( x k ) (see Lemma 1), we have

Thus, taking the conditional expectation with respect to F k , by Lemma 4, we get,

Then, by Proposition 1,

Next suppose that Assumption 2 holds. Rearranging the terms,

with C k ∈ ℓ 1 and α k h k ∈ ℓ 1 . Therefore, Robbins-Siegmund Theorem [43] implies that ( ∥ x k -x ∗ ∥ ) k ∈ N is a.s. convergent, α k ( f ( x k ) -f ( x ∗ )) ∈ ℓ 1 a.s. and thus, since α k ̸∈ ℓ 1 ,

We derive from [32, Lemma 9.9] and Lemma 5 that there exists a random variable ˆ x taking values in arg min f such that x k → ˆ x a.s. Finally, continuity of f yields that lim k f ( x k ) = f ( x ∗ ) a.s.

In the next Lemma, to derive bounds on function values, we study the sequence ( f h k ( x k +1 ) -f h k ( x k )) k ∈ N . It is the difference between the smoothed function at iteration k evaluated at x k and at x k +1 . It corresponds to the function value decrease between the iterations k +1 and k if h k is constant.

Lemma 7 (Function Value decrease: nonconvex non-smooth setting) . Under Assumption 1, let ( x k ) k ∈ N be the sequence generated by Algorithm 1. Then,

where c is a numerical constant.

Proof. By Lemma 1, we have that f h k is L 0 √ d/h k -smooth. Thus, by the Descent Lemma [41],

Taking the conditional expectation with respect to F k ,

The claim follows from Lemma 4.

A.3 Auxiliary results for smooth setting.

Lemma 8 (Function value decrease: convex smooth setting) . Under Assumption 3 , let ( x k ) k ∈ N be the sequence generated by Algorithm 1. Then the following holds:

Proof. By the Descent Lemma [41] and Algorithm 1,

Taking the conditional expectation and by Lemma 4,

Adding and subtracting ∇ f ( x k ) ,

By Cauchy-Schwarz inequality and Proposition 1,

By Young's inequality,

This concludes the proof.

Lemma 9 (Convergence in smooth setting) . Let ( x k ) k ∈ N be the sequence generated by Algorithm 1 and let x ∗ ∈ arg min x ∈ R d f ( x ) . Then, under Assumption 3, the following inequality holds

Moreover, if f is convex, Assumption 4 holds and α k ≤ ¯ α < ℓ/ (2 dL 1 ) . Then

( α k ∥∇ f ( x k ) ∥ 2 ) k ∈ N ∈ ℓ 1 a.s.

( ∥ x k -x ∗ ∥ ) k ∈ N is a.s. convergent.

( α k ( f ( x k ) -f ( x ∗ ))) k ∈ N ∈ ℓ 1 a.s.

there exists a random variable ˆ x taking values in arg min f such that x k → ˆ x a.s. and lim k →∞ f ( x k ) = min f .

Proof. We have

Taking the conditional expectation,

For every k , set u k = ∥ x k -x ∗ ∥ . By Lemma 4,

Note that

-2 α k ⟨∇ f h k ( x k ) , x k -x ∗ ⟩ = 2 α k ⟨∇ f h k ( x k ) -∇ f ( x k ) , x ∗ -x k ⟩ -2 α k ⟨∇ f ( x k ) , x k -x ∗ ⟩ . Thus,

By the Cauchy-Schwarz inequality,

The first claim follows from Proposition 1. By Proposition 1 and Young's inequality with parameter τ k = α k h k , we get

Since f is convex, by Baillon-Haddad Theorem [3], we derive that

Note that ∆ > 0 . Thus, rearranging the terms

Since ρ k , C k ∈ ℓ 1 by Assumption 4, Robbins-Siegmund Theorem [43] ensures that ( u 2 k ) k ∈ N is convergent and ( α k ∥∇ f ( x k ) ∥ 2 ) k ∈ N ∈ ℓ 1 a.s. Since f is convex, it follows from (9) that

Robbins-Siegmund Theorem [43] implies that ( α k ( f ( x k ) -f ( x ∗ ))) k ∈ N ∈ ℓ 1 a.s. Assumption 4 implies that α k ̸∈ ℓ 1 therefore

By Lemma 8 and Assumption 4, we have that the sequence E [ f ( x k +1 ) -f ( x ∗ ) |F k ] -( f ( x k ) -f ( x ∗ )) is upper-bounded by a sequence in ℓ 1 . Thus, by Robbins-Siegmund Theorem [43], lim k ( f ( x k ) -f ( x ∗ )) exists a.s. Then, it follows from (10) that

Moreover, as we saw before, ( ∥ x k -x ∗ ∥ ) k ∈ N is convergent a.s. for every x ∗ ∈ arg min f . Then, by Opial's Lemma [40], there exists a random variable ˆ x taking values in arg min f such that x k → ˆ x a.s.

Lemma 10 (Gradient bound: convex smooth setting) . Suppose that Assumptions 3 and 4 hold, and assume f to be convex. Let ( x k ) k ∈ N be the sequence generated by Algorithm 1. Then, for every k ∈ N and every x ∗ ∈ arg min f ,

and where

Proof. By Lemma 9 we derive

By Baillon-Haddad Theorem and Assumption 4,

Let u k := √ E [ ∥ x k -x ∗ ∥ 2 ] . Taking the full expectation, by Jensen inequality we have

Summing the previous inequality from i = 0 , · · · , k , we get

Since u k is non-negative, the first claim of the lemma follows. Since ∆ > 0 , ρ k ≥ 0 , S k is non decreasing, and S k ≥ u 2 0 in (11), then

Thus, the (discrete) Bihari's Lemma [32, Lemma 9.8] yields

concluding the proof.

B Proofs of Main Results

B.1 Proof of Theorem 1

Rearranging the terms, taking the full expectation, and summing the first k iterations

Let ¯ x k := k ∑ i =0 α i x i / ( k ∑ i =0 α i ) . Dividing by k ∑ i =0 α i and observing that by convexity we have

we get the first claim. Under Assumption 2, the second claim holds by Lemma 6.

B.2 Proof of Corollary 1

By Theorem 1,

Replacing α k and h k with the sequences in the statement,

with

The second point of the corollary can be proved replacing α k = α and h k = h . Now, to prove the third point, fix ε ∈ (0 , 1) . Since we want E [ f (¯ x k ) -f ( x ∗ )] ≤ ε , we impose

Choosing h k = h ≤ ε 2 L 0 , to get the previous inequality it is sufficient to impose

Wefix a priori a number of iterations K and we minimize the left handside with respect to α , obtaining

Thus, for h k = h ≤ ε 2 L 0 , α as above and

we have E [ f (¯ x k ) -f ( x ∗ )] ≤ ε . Note that, since the computation of the surrogate requires 2 ℓ function evaluations, to ensure an error of ε we need to perform a number of function evaluations of the order

This concludes the proof.

B.3 Proof of Theorem 2

Taking the full expectation and rearranging the terms,

Next sum from i = 0 to i = k . By definition of f h , we have f h ( x ) ≥ min f for every x ∈ R d , thus,

The claim follows.

B.4 Proof of Corollaries 2 and 3

Due to the choice of α k = α ( k +1) -θ with θ ∈ (1 / 2 , 1) and α > 0 , we get

where

If we choose α k = α , we derive

If we fix a priori a number of iteration K and we minimize the right handside with respect to α , we get

Let ε ∈ (0 , 1) . Choosing α = ˆ α , we get η ( h ) K ≤ ε for

This concludes the proof of Corollary 2. To prove Corollary 3, we fix a maximum number of iterations K ∈ N and consider the random variable I of the statement. Let ∂ h f be the h -Goldstein subdifferential defined in Definition 1. It follows from [34, Theorem 3.1] that ∇ f h ( x I ) ∈ ∂ h f ( x I ) almost surely, therefore

In addition, Theorem 2 yields

Thus,

Hence, for α = ¯ α and K chosen s.t. inequality (14) holds, we have

This concludes the proof.

B.5 Proof of Theorem 3

By convexity,

Rearranging the terms and taking the full expectation,

Since E [ ∥ x ∗ -x k ∥ ] = E [ √ ∥ x ∗ -x k ∥ 2 ] , Jensen's inequality implies that

Denoting with u k = E [ ∥ x k -x ∗ ∥ 2 ] and taking the sum from i = 0 to i = k ,

where the last inequality holds by Assumption 4. Let ∆ := (1 /L 1 -( d/ℓ )¯ α ) . By Lemma 10, we have

Let ¯ x k := k ∑ i =0 α i x i / ( k ∑ i =0 α i ) . Dividing both sides by k ∑ i =0 α i , convexity yields

B.6 Proof of Corollary 4

In this proof, we use the same notation as the one in the proof of Theorem 3. By the choices of the parameters, we have

Thus, using these inequalities in Theorem 3, we get

with

Dividing by k ∑ i =0 α i , we get

Note that by Assumption 4, α < ℓ/ ( dL 1 ) , thus 1 /α > ( dL 1 ) /ℓ . The algorithm performs 2 ℓ function evaluations at each iteration. Thus, to guarantee E [ f (¯ x k ) -min f ] ≤ ε for ε ∈ (0 , 1) , the algorithm has to perform a number of function evaluations in the order of

Assuming, instead, α k ≤ ¯ α < ℓ/ (2 dL 1 ) , by Lemma 9 we get the last claim; i.e, there exists a random variable ˆ x taking values in arg min f s.t. x k → ˆ x a.s.

B.7 Proof of Theorem 4

Set C 1 = ( dL 1 ) / 2 . It follows from Lemma 8 that

Taking the full expectation and rearranging the terms, and recalling the definition of ∆ ,

Summing for i = 0 , · · · , k and observing that min f ≤ f ( x ) for every x ,

Divinding by ∆ k ∑ i =0 α i we get the claim.

B.8 Proof of Corollary 5

( i ) : From the choice of α k and h k , we have

It follows from Theorem 4 that

with C 1 = L 2 1 θ 4(2 θ -1) and C 2 = L 3 1 θ 2(2 θ -1) .

( ii ) : It follows directly from Theorem 4 taking into account that

and setting C 1 = L 2 1 / 8 and C 2 = L 3 1 / 4 .

C Experimental Details

In this appendix, we report details on the experiments performed. We implemented every script in Python3 (version 3.9.11) and used numpy (version 1.22.2) [27] and matplotlib (version 3.5.1) [29] libraries.

Machine used to perform the experiments. In the following table, we describe the features of the machine used to perform the experiments in Section 4.

Target Functions. We considered two synthetic target functions: a convex smooth function f 1 and a convex non-smooth function f 2 defined as follows

where A is a random Gaussian matrix (i.e. A i,j ∼ N (0 , 1) ) and ¯ v := [0 , 1 , · · · , d -1] ⊺ .

Choice of the number of directions. We report here the details of the first experiment of Section 4. For these experiments, we consider d = 50 and we use, for the smooth convex case, the following parameters

The constant L 1 is computed as the maximum eigenvalue of the matrix A ⊺ A . Note that this parameter choice satisfies Assumption 4. For the non-smooth target, we used

Note that this parameter configuration satisfies Assumption 2. The maximum number of function evaluations considered is 4000 . The direction matrices G k are generated with the QR method - see Appendix D.

Comparison with Finite-difference methods. In Section 4, we compare finite-difference method with different choice of directions. In order to make a fair comparison we consider only central finitedifferences. However, note that Algorithm 1 can be modified (in practice) considering computationally cheaper gradient estimators - see Remark 1. For these experiments, we consider d = 10 and ℓ = d for methods with multiple directions. The maximum number of function evaluations is 1000 for both smooth and non-smooth targets and the direction matrices G k for Algorithm 1 are generated with the QR method - see Appendix D. To solve the smooth problem we consider the following parameter choice for every method

where L 1 is computed taking the maximum eigenvalue of A ⊺ A . For Algorithm 1 and finite-difference with single and multiple spherical directions c = 0 . 99 while it is equal to c = 0 . 11 for finite-difference with single and multiple Gaussian directions. We made this choice since for finite-difference methods with Gaussian directions we observed divergence for larger choices of c - see Figure 3.

0

200

400

600

800

1000

function evaluations

10 -1

10 2

10 5

10 8

10 11

10 14

10 17

f

(

xk

) -

f

(

x

*

)

Smooth Convex Target

Single Gaussian

Single Spherical

Multi Gaussian

Multi Spherical

Our

0

200

400

600

800

1000

function evaluations

10 -2

10 -1

10 0

10 1

10 2

10 3

10 4

10 5

f

(

xk

) -

f

(

x

*

)

Smooth Convex Target

Single Gaussian

Single Spherical

Multi Gaussian

Multi Spherical

Our

For the non-smooth convex target, we considered the following parameter choice

For every method, we selected c = 0 . 65 except for the method with multiple Gaussian directions in which we selected c = 0 . 08 since it provided better performances - see Figure 4.

0

200

400

600

800

1000

function evaluations

10 0

10 1

f

(

xk

) -

f

(

x

*

)

NonSmooth Convex Target

Single Gaussian

Single Spherical

Multi Gaussian

Multi Spherical

Our

0

200

400

600

800

1000

function evaluations

10 0

10 1

f

(

xk

) -

f

(

x

*

)

NonSmooth Convex Target

Single Gaussian

Single Spherical

Multi Gaussian

Multi Spherical

Our

0

200

400

600

800

1000

function evaluations

10 0

10 1

f

(

xk

) -

f

(

x

*

)

NonSmooth Convex Target

Single Gaussian

Single Spherical

Multi Gaussian

Multi Spherical

Our

0

200

400

600

800

1000

function evaluations

10 0

10 1

f

(

xk

) -

f

(

x

*

)

NonSmooth Convex Target

Single Gaussian

Single Spherical

Multi Gaussian

Multi Spherical

Our

0

200

400

600

800

1000

function evaluations

10 0

10 1

f

(

xk

) -

f

(

x

*

)

NonSmooth Convex Target

Single Gaussian

Single Spherical

Multi Gaussian

Multi Spherical

Our

0

200

400

600

800

1000

function evaluations

10 0

10 1

f

(

xk

) -

f

(

x

*

)

NonSmooth Convex Target

Single Gaussian

Single Spherical

Multi Gaussian

Multi Spherical

Our

0

200

400

600

800

1000

function evaluations

10 0

10 1

f

(

xk

) -

f

(

x

*

)

NonSmooth Convex Target

Single Gaussian

Single Spherical

Multi Gaussian

Multi Spherical

Our

0

200

400

600

800

1000

function evaluations

10 0

10 1

10 2

f

(

xk

) -

f

(

x

*

)

NonSmooth Convex Target

Single Gaussian

Single Spherical

Multi Gaussian

Multi Spherical

Our

D Techniques to Generate Orthogonal Direction Matrices

In the literature, different algorithms were proposed to generate orthogonal matrices - see for instance [23, 38, 11, 28, 7, 2, 4, 44, 8] and references therein. Such methods can be used to generate the direction matrices G k required for the iteration proposed in Algorithm (1). In this appendix, we briefly discuss three of them.

QR factorization. As observed in [32, 42], a way to generate orthogonal consists in generating a random Gaussian matrix A ∈ R d × d with A i,j ∼ N (0 , 1) and perform the QR factorization i.e. A = QR . Then, the direction matrix is the truncation of the Q matrix i.e. QI d,ℓ .

Householder Reflection. To obtain a direction matrix, we can use a Householder reflector. This can be done by sampling a vector v from the unit sphere S d -1 . The direction matrix G is defined as a Householder reflector, given by

with I ∈ R d × d identity matrix. To obtain the desired matrix, we compute the product of G with I d,ℓ , i.e., we take the first ℓ columns. The (truncated) identity matrix can be generated and stored offline (note that since it is very sparse, it can be stored using a sparse format (e.g. the COO format proposed in scikit-learn library[9]). In this way, we can save resources in high-dimensional settings. In order to quantify the time-cost of this procedure, we compared the time of generating this kind of matrix with random matrices with different dimensions. For this experiment, we consider the ℓ = d case i.e. the most expensive. Matrices are computed in CPU and the details of the machine used are described in Appendix C. We report the mean and standard deviation of the time using 500 repetitions. In Figure 5, we compare the time-cost of generating orthogonal matrices with this procedure against generating random matrices while in Table 2 we report the mean and standard deviation of the results.

2

4

8

16

32

64

128

256

512

1024

2048

d

10 -7

10 -6

10 -5

10 -4

10 -3

10 -2

10 -1

Time (s)

Computational Time to generate matrices

random spherical

Householder

In Figure 5, we can observe that using this strategy we can limit the cost of generating random orthogonal matrices. In particular, for dimensions larger than 32 , our method is faster than random gaussian and spherical directions.

Moreover, if more computational resources are available, we can build m> 1 Householder reflectors G 1 , · · · , G m using m random vectors v 1 , · · · , v m sampled i.i.d from S d -1 and define the direction matrix as

It is important to note that when m = d , this procedure is equivalent to using the QR factorization.

Haar Butterfly matrices. We can build orthogonal matrices using Butterfly matrices [48]. Let G (0) := [1] , we can build an orthogonal matrix of dimension d = 2 n with the following recursion

where θ n is sampled uniformly in [0 , 2 π ] . Then we compute GI d,ℓ (we take the first ℓ columns). The construction of Haar butterfly matrices is faster than previous methods because it only requires simple operations. However, this procedure allows to build only matrices with d = 2 n for n ≥ 0 . In literature, different methods were proposed to cope with this limitation e.g. [23].

E Limitations

In this appendix, we discuss the main practical limitations of Algorithm 1. Like all finite-difference methods with multiple directions, O-ZD requires multiple function evaluations to execute a single step. In many practical applications, function evaluations can be time-consuming, leading to the use of a small number of directions ℓ . This may result in poor performance as observed in numerical experiments. As for the subgradient method, in O-ZD the step size significantly affects performance, and tuning it can be challenging. To address this limitation, an adaptive stepsize selection method could be proposed. Furthermore, decreasing the sequence h k too quickly can lead to numerical instability, as noted in [42].

F Other Experiments

We performed other experiments in minimizing convex functions. We considered the targets defined in Table 3 and, for each experiment, we reported the mean and standard deviation using 20 repetitions.

0

2500

5000

7500

10000

12500

15000

17500

20000

function evaluations

10 -1

10 0

10 1

10 2

10 3

f

(

xk

) -

f

(

x

*

)

Elastic Net

Single Gaussian

Single Spherical

Multi Gaussian

Multi Spherical

Ours

0

2500

5000

7500

10000

12500

15000

17500

20000

function evaluations

10 0

10 1

10 2

f

(

xk

) -

f

(

x

*

)

Total Variation

Single Gaussian

Single Spherical

Multi Gaussian

Multi Spherical

Ours

0

2500

5000

7500

10000

12500

15000

17500

20000

function evaluations

10 -14

10 -11

10 -8

10 -5

10 -2

10 1

f

(

xk

) -

f

(

x

*

)

Huber Loss

Single Gaussian

Single Spherical

Multi Gaussian

Multi Spherical

Ours

0

2500

5000

7500

10000

12500

15000

17500

20000

function evaluations

10 1

5×10 0

6×10 0

7×10 0

8×10 0

9×10 0

f

(

xk

) -

f

(

x

*

)

Infinity Norm

Single Gaussian

Single Spherical

Multi Gaussian

Multi Spherical

Ours

0

2500

5000

7500

10000

12500

15000

17500

20000

function evaluations

10 0

10 1

f

(

xk

)

Sparse Group Lasso

Single Gaussian

Single Spherical

Multi Gaussian

Multi Spherical

Ours

0

2500

5000

7500

10000

12500

15000

17500

20000

function evaluations

10 -14

10 -11

10 -8

10 -5

10 -2

10 1

f

(

xk

) -

f

(

x

*

)

L1 Norm

Single Gaussian

Single Spherical

Multi Gaussian

Multi Spherical

Ours

In Figure 6, we can observe that structured finite-difference performs better than unstructured methods.

In Table 3, we define the function used for the experiments. In particular:

Sparse Group Lasso: p is set to 3 and given an x ∈ R d , x ( β i ) is a vector obtained by taking 3 entries of x .

Huber Loss: δ is set to 0 . 5 .

Elastic Net: α, β are set to 0 . 5 .

Peer reviews:
comments: SUMMARY:
This paper presents a study on a structured finite-difference algorithm for non-smooth black box optimization. The authors successfully demonstrate that their finite-difference surrogate serves as an unbiased estimator of the gradient for the smoothing approximation of the target function. The proposed O-ZD method's convergence analysis is established under different assumptions.

STRENGTHS:
Strengths And Weaknesses:
Overall, the paper is well-structured and provides valuable insights into the structured finite-difference method for non-smooth black box optimization. The authors establish the optimal complexity in the non-smooth convex case and demonstrate convergence rates in the non-smooth nonconvex and smoothing settings. However, it is unclear to me what is the exact theoretical complexity improvement of this method compared to state-of-the-art methods in different scenarios. Additionally, the objective function used in the numerical experiments appears to be relatively simple.

Furthermore, I kindly request the authors to address the following questions for better understanding:

1. Could the authors please provide the exact formula of C in the Corollary 1? It seems that C should be somehow related to \theta.

2. In line 205, the authors claim that the complexity in terms of the number of iterations is better than [15] and [37]. However, I believe it would be more appropriate to compare the number of function evaluations since the iterations of O-ZD are more computationally expensive.

3. I suggest the authors include a table explicitly listing the number of function evaluations' complexity of this work and the related works, rather than just providing a discussion for each case separately.

WEAKNESSES:
See above

QUESTIONS:
See above

LIMITATIONS:
not applicable
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-17

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyzes the convergence rate of structured zeroth order optimization, Whose iterations’ descent direction Is chosen by random orthogonal group. It applies to the most general non smooth setting, and this paper gives the convergence rate of all specific settings of interest.\n\nSTRENGTHS:\nThis paper analyzes an important optimization method, and provides concrete mathematical analysis to prove the theories. The result is novel and comprehensive. This paper has a very detailed discussion of many settings, each of them has the convergence rate in its case.\n\nWEAKNESSES:\nNo major weaknesses, questions below.\n\nQUESTIONS:\nIn Eq(2), is there a motivation why you want to fix $l$ and vary $G$, is $h$ fixed or random? \n\nIn Algo. 1, I think it means sample $G_k$ i.i.d. from “a uniform distribution” on $O(d)$, is that correct?\n\nWhat structure do the Algo’s chosen direction satisfy? For example, the paper mentions “orthogonality [30,40]” as a type of structure, but are the directions in Algo. 1 orthogonal? What is the difference between 1) “sampling $G_k$ randomly” and 2) zeroth order GD when you just randomly choose a point nearby and estimate $f(x+dx) – f(x-dx)$? With this part clarified, I would be willing to increase the score.\n\nThere is another type of zeroth order method – although not always applicable in practice but worth mention. When the objective function is analytic, one can use complex number to make estimation variance smaller by estimating the gradient by \n$$ Im( f(x+yi) – f(x-yi) ) / 2y $$\nwhere $y$ is small. For example, if $f(x) = x^3$, then \n$$ Im( f(x+yi) – f(x-yi) ) / 2y = 3x^2 + O(y^2)$$\n$$ ( f(x+y) – f(x-y) ) / 2y = 3x^2 + O(y)$$\nWith complex number, the noise or variance on the real part can be ignored, with only high order noise kept.\n\n======\n\nRaised to 7.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a structured zeroth-order estimator for non-smooth optimization. The proposed algorithm using this estimator could achieve optimal convergence rate for non-smooth convex optimization, also achieve a convergence rate in terms of Goldstein stationarity for non-smooth non-convex optimization. Numerical experiments are provided to show the efficiency of the proposed algorithm.\n\nSTRENGTHS:\nThe theory is well-rounded with both theoretical and numerical evidences. The proposed structured zeroth-order estimator is the first one for non-smooth optimization. The use of the Goldstein stationary is pretty novel and an interesting direction to further explore.\n\nWEAKNESSES:\n(Please respond to the Questions section directly) The design of the zeroth-order estimator may cost more time, due to the fact that it requires sampling orthogonal matrices; The problem studied doesn’t cover stochastic situations; The numerical experiments are not adequate.\n\nQUESTIONS:\nMajor:\n1. Could the authors talk about possible applications of the proposed method? For example one could argue that zeroth-order smooth optimization can be applied to black-box attack of neural nets. But I’m not aware of the application of non-smooth situation.\n\n2. The design of the zeroth-order estimator may cost more time, due to the fact that it requires sampling orthogonal matrices (either using QR or other matrix operations). More specifically, I’m wondering how the proposed method behaves comparing to simply sample $\\ell$ vectors (with/without replacement) from the canonical orthogonal basis.\n\n3. Regarding the numerical experiments. The choice of $\\ell$ seem to be arbitrary. One could imagine that with a larger $\\ell$, the per-iteration convergence would be faster, but it would cost more time to construct the estimator. Can the author give some insight on how to choose $\\ell$ in an efficient manner? Another question related is that would the convergence and numerical behavior behave better if we vary $\\ell$ in each iteration?\n\n4. For the numerical experiment. I understand that due to page limit the authors moved a lot of details into the appendix, but it should be necessary to at least include the problem on which the experiments are conducted. Also since all the plots in the numerical experiments use “function evaluation” as ax-axis. I’m wondering what would the plot look like if we could the CPU time, since the proposed method could consume more time to construct the orthogonal matrix.\n\n5. This is a personal comment and the authors may consider this for future works. The proposed method and analysis are all for deterministic optimization, whereas modern machine learning is about stochastic optimization. It would be interesting to see the convergence behavior of the proposed method for stochastic optimization problems, where the smooth stochastic case has been studied[1]. For zeroth-order non-smooth stochastic optimization, Goldstein stationarity seems to be necessary.\n\n\nMinor:\n1. Please add some reference in the abstract, e.g. line 7 “Recently,… improve performance”.\n\nReferences:\n[1] Balasubramanian, Krishnakumar, and Saeed Ghadimi. \"Zeroth-order nonconvex stochastic optimization: Handling constraints, high dimensionality, and saddle points.\" Foundations of Computational Mathematics (2022): 1-42.\n\nLIMITATIONS:\nThe limitation is well stated in weakness and question sections.\n\nI’m not aware of any potential negative social impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nZeroth-order optimization is the sub-field of optimization concerned with solving \n$$ x^{\\star} \\in \\operatorname*{argmin}_{x \\in \\mathbb{R}^d}$$ \n_without_ using gradient information. This paper studies the most general case, where $f$ is assumed neither smooth nor convex. They propose a structured approach to the standard finite-difference gradient approximation, which means that the sampling directions are taken to be orthogonal.\n\nThe main contributions of this paper are theoretical; it provides a fine-grained analysis of the convergence rate of the proposed algorithm under a variety of assumptions. The theoretical claims are supported by two numerical experiments.\n\nSTRENGTHS:\n- The first 4 pages are easy to read, and position the problem under consideration (orthogonal sample directions for non-smooth optimization) nicely with respect to prior work\n- This paper is definitely a technical advancement on prior work.\n- The mathematical technique displayed in this paper is impressive. As an example, I enjoyed the use of the Goldstein subdifferential. The correct notion of \"approximate stationarity\" in this setting is quite subtle but you handle it very well.\n - In Corollaries 1, 2, and 4 I appreciated that you provided convergence guarantees in both the constant step-size and decreasing step-size settings. Both are useful\n\nWEAKNESSES:\n- Overall I found this paper to be a little too heavy on theory and a little too light on experiment. I would suggest relegating some of the parameter configurations of Corollaries 2, 4, and 5 to an appendix and adding an additional experiment (see next point).\n - I'd strongly suggest adding more experiments; in my opinion the two simple functions you tested do not provide enough data to draw any conclusions. It would be particularly interesting to see more non-smooth experiments. \n\nMinor points:\n - In line 235 \"the ball centered in 0\" should be \"the ball centered at x\"\n - In line 239 you write $\\partial f_h(x_I)$ but I think you mean $\\partial_h f(x_I)$. \n - Why is there a staircase pattern in the purple graph in Figure 1?\n\nQUESTIONS:\n1. In the related work section, you state that your method achieves the optimal dependence on dimension and then cite [1]. But this paper deals with stochastic zeroth-order optimization with a particular \"two-point\" query model, whereas in your paper you consider the non-stochastic/noise-free setting. \n\nI did look at [1] but could not find a clear statement of the optimal dependence on dimension in the noise-free setting (I believe it is $\\mathcal{O}(d)$). Could you make this a bit clearer in your exposition? Also, what is the authoritative reference for the optimal $d$-dependence for noise-free DFO?\n\n[1] J. C. Duchi, M. I. Jordan, M. J. Wainwright, and A. Wibisono. Optimal rates for zero-order convex optimization: The power of two function evaluations.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
This paper introduces and analyzes  a structured finite difference algorithm for non-smooth optimization problems. The algorithm is built on a smooth approximation of the nonsmooth loss function and a structured finite difference approximation. The convergence of the proposed algorithm in non-smooth convex, non-smooth non-convex, smooth convex and smooth non-convex cases are studied.

STRENGTHS:
1. A simple algorithm using structured finite difference approximation of the gradient is proposed.

2. The convergence behavior of the proposed algorithm in four cases are studied.

WEAKNESSES:
I think the relation between the convergence of the proposed method and the number of directions chosen is not clearly analyzed. The error in Theorem 1 is only linear in \ell. While numerical experiments suggests \ell has an impact on the convergence rate.

QUESTIONS:
1. In Theorem 1 and Corollary 1, the error is linear in \ell. The convergence rate seems only depends on \theta. Does \ell affect the convergence rate?

2. From the theories in this paper, setting \theta towards 1/2 provides better convergence rate. But what is the hurt by doing that?

LIMITATIONS:
The limitation is not discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-08