SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbb {E} [ f _ {h} (x _ {k + 1}) | \mathcal {F} _ {k} ] - f _ {h} (x _ {k}) \leq - \alpha_ {k} \| \nabla f _ {h} (x _ {k}) \| ^ {2} + c \frac {L _ {0} ^ {3} d \sqrt {d}}{\ell} \frac {\alpha_ {k} ^ {2}}{h}.
$$
Taking the full expectation and rearranging the terms,
$$
\alpha_ {k} \mathbb {E} [ \| \nabla f _ {h} (x _ {k}) \| ^ {2} ] \leq \mathbb {E} [ f _ {h} (x _ {k}) - f _ {h} (x _ {k + 1}) ] + c \frac {L _ {0} ^ {3} d \sqrt {d}}{\ell} \frac {\alpha_ {k} ^ {2}}{h}.
$$
Next sum from  $i = 0$  to  $i = k$ . By definition of  $f_{h}$ , we have  $f_{h}(x) \geq \min f$  for every  $x \in \mathbb{R}^{d}$ , thus,
$$
\sum_ {i = 0} ^ {k} \alpha_ {i} \mathbb {E} \left[ \| \nabla f _ {h} (x _ {i}) \| ^ {2} \right] \leq \mathbb {E} \left[ f _ {h} \left(x _ {0}\right) - \min  f \right] + c \frac {L _ {0} ^ {3} d \sqrt {d}}{\ell} \sum_ {i = 0} ^ {k} \frac {\alpha_ {i} ^ {2}}{h}. \tag {12}
$$
The claim follows.
B.4 Proof of Corollaries 2 and 3
By Theorem 2,
$$
\eta_ {k} ^ {(h)} \leq \left((f _ {h} (x _ {0}) - f (x ^ {*})) + c \frac {L _ {0} ^ {3} d \sqrt {d}}{\ell} \sum_ {i = 0} ^ {k} \frac {\alpha_ {i} ^ {2}}{h}\right) / \left(\sum_ {i = 0} ^ {k} \alpha_ {i}\right).
$$
Due to the choice of  $\alpha_{k} = \alpha (k + 1)^{-\theta}$  with  $\theta \in (1 / 2,1)$  and  $\alpha >0$  , we get
$$
\eta_ {k} ^ {(h)} \leq \frac {C _ {1}}{\alpha (k + 1) ^ {1 - \theta}} + \frac {C _ {2} d \sqrt {d} \alpha}{\ell h} \frac {1}{(k + 1) ^ {\theta}},
$$
$$
C _ {1} := \| x _ {0} - x ^ {*} \| ^ {2} (1 - \theta) \quad \text {a n d} \quad C _ {2} := \frac {c L _ {0} ^ {3} (1 - \theta)}{(1 - 2 \theta)}.
$$
If we choose  $\alpha_{k} = \alpha$ , we derive
$$
\eta_ {k} ^ {(h)} \leq \frac {f _ {h} \left(x _ {0}\right) - \operatorname* {m i n} f}{\alpha k} + \frac {c L _ {0} ^ {3} d \sqrt {d} \alpha}{\ell h}. \tag {13}
$$
If we fix a priori a number of iteration  $K$  and we minimize the right hand side with respect to  $\alpha$ , we get
$$
\hat {\alpha} = \sqrt {\frac {(f _ {h} (x _ {0}) - f (x ^ {*})) \ell h}{K c L _ {0} ^ {3} d \sqrt {d}}}.
$$
Let  $\varepsilon \in (0,1)$ . Choosing  $\alpha = \hat{\alpha}$ , we get  $\eta_K^{(h)}\leq \varepsilon$  for
$$
K \geq 4 \frac {\left(f _ {h} \left(x _ {0}\right) - f \left(x ^ {*}\right)\right) c L _ {0} ^ {3} d \sqrt {d}}{\ell h} \varepsilon^ {- 2}. \tag {14}
$$
This concludes the proof of Corollary 2. To prove Corollary 3, we fix a maximum number of iterations  $K \in \mathbb{N}$  and consider the random variable  $I$  of the statement. Let  $\partial_h f$  be the  $h$ -Goldstein subdifferential defined in Definition 1. It follows from [34, Theorem 3.1] that  $\nabla f_h(x_I) \in \partial_h f(x_I)$  almost surely, therefore
$$
\mathbb {E} _ {I} \min  [ \| \eta \| ^ {2}: \eta \in \partial_ {h} f (x _ {I}) ] \leq \mathbb {E} _ {I} \mathbb {E} [ \| \nabla f _ {h} (x _ {I}) \| ^ {2} ].
$$
In addition, Theorem 2 yields
$$
\begin{array}{l} \mathbb {E} _ {I} \mathbb {E} _ {G} [ \| \nabla f _ {h} (x _ {I}) \| ^ {2} ] = \left(\sum_ {j = 0} ^ {K - 1} \alpha_ {j} \mathbb {E} _ {G} [ \| \nabla f _ {h} (x _ {j}) \| ^ {2} ]\right) / \sum_ {j = 0} ^ {K - 1} \alpha_ {j} \\ \leq \mathbb {E} \left[ f _ {h} \left(x _ {0}\right) - \min  f \right] + c \frac {L _ {0} ^ {3} d \sqrt {d}}{\ell} \sum_ {i = 0} ^ {k} \frac {\alpha_ {i} ^ {2}}{h}. \\ \end{array}
$$
$$
\mathbb {E} _ {I} [ \| \eta \| ^ {2}: \eta \in \partial_ {h} f (x _ {I}) ] \leq \mathbb {E} _ {I} \mathbb {E} [ \| \nabla f _ {h} (x _ {I}) \| ^ {2} ] = \eta_ {k} ^ {(h)}.
$$
Hence, for  $\alpha = \bar{\alpha}$  and  $K$  chosen s.t. inequality (14) holds, we have
$$
\mathbb {E} _ {I} \left[ \| \eta \| ^ {2}: \eta \in \partial_ {h} f (x _ {I}) \right] \leq \varepsilon .
$$
B.5 Proof of Theorem 3
By Lemma 9,
$$
\begin{array}{l} \mathbb {E} \left[ \| x _ {k + 1} - x ^ {*} \| ^ {2} | \mathcal {F} _ {k} \right] - \| x _ {k} - x ^ {*} \| ^ {2} \leq \frac {2 d}{\ell} \alpha_ {k} ^ {2} \| \nabla f (x _ {k}) \| ^ {2} + 2 \alpha_ {k} \langle \nabla f (x _ {k}), x ^ {*} - x _ {k} \rangle \\ + \underbrace {L _ {1} d \alpha_ {k} h _ {k}} _ {=: \rho_ {k}} \| x ^ {*} - x _ {k} \| + \underbrace {\frac {L _ {1} ^ {2} d ^ {2}}{2 \ell} \alpha_ {k} ^ {2} h _ {k} ^ {2}} _ {=: C _ {k}}. \\ \end{array}
$$
By convexity,
$$
\begin{array}{l} \mathbb {E} \left[ \| x _ {k + 1} - x ^ {*} \| ^ {2} | \mathcal {F} _ {k} \right] - \| x _ {k} - x ^ {*} \| ^ {2} \leq \frac {2 d}{\ell} \alpha_ {k} ^ {2} \| \nabla f (x _ {k}) \| ^ {2} - 2 \alpha_ {k} \left(f (x _ {k}) - f (x ^ {*})\right) \\ + \rho_ {k} \| x ^ {*} - x _ {k} \| + C _ {k}. \\ \end{array}
$$
Rearranging the terms and taking the full expectation,
$$
\begin{array}{l} 2 \mathbb {E} [ \alpha_ {k} (f (x _ {k}) - f (x ^ {*})) ] \leq \mathbb {E} [ \| x _ {k} - x ^ {*} \| ^ {2} - \| x _ {k + 1} - x ^ {*} \| ^ {2} ] + \frac {2 d}{\ell} \alpha_ {k} ^ {2} \mathbb {E} [ \| \nabla f (x _ {k}) \| ^ {2} ] \\ + \rho_ {k} \mathbb {E} [ \| x ^ {*} - x _ {k} \| ] + C _ {k}. \\ \end{array}
$$
Since  $\mathbb{E}[\| x^{*} - x_{k}\| ] = \mathbb{E}[\sqrt{\|x^{*} - x_{k}\|^{2}} ]$ , Jensen's inequality implies that
$$
\begin{array}{l} 2 \mathbb {E} [ \alpha_ {k} (f (x _ {k}) - f (x ^ {*})) ] \leq \mathbb {E} [ \| x _ {k} - x ^ {*} \| ^ {2} - \| x _ {k + 1} - x ^ {*} \| ^ {2} ] + \frac {2 d}{\ell} \alpha_ {k} ^ {2} \mathbb {E} [ \| \nabla f (x _ {k}) \| ^ {2} ] \\ + \rho_ {k} \sqrt {\mathbb {E} [ \| x ^ {*} - x _ {k} \| ^ {2} ]} + C _ {k}. \\ \end{array}
$$
Denoting with  $u_{k} = \mathbb{E}\big[\| x_{k} - x^{*}\|^{2}\big]$  and taking the sum from  $i = 0$  to  $i = k$ ,
$$
\begin{array}{l} 2\sum_{i = 0}^{k}\alpha_{i}\mathbb{E}[f(x_{i}) - f(x^{*})]\leq \underbrace{u_{0}^{2} + \sum_{i = 0}^{k}C_{i}}_{=: S_{k}} + \frac{2d}{\ell}\sum_{i = 0}^{k}\alpha_{i}^{2}\mathbb{E}[||\nabla f(x_{i})||^{2}] + \sum_{i = 0}^{k}\rho_{i}u_{i} \\ \leq S _ {k} + \frac {2 d}{\ell} \bar {\alpha} \sum_ {i = 0} ^ {k} \alpha_ {i} \mathbb {E} [ \| \nabla f (x _ {i}) \| ^ {2} ] + \sum_ {i = 0} ^ {k} \rho_ {i} u _ {i}, \\ \end{array}
$$
where the last inequality holds by Assumption 4. Let  $\Delta \coloneqq (1 / L_1 - (d / \ell)\bar{\alpha})$ . By Lemma 10, we have
$$
\begin{array}{l} \sum_ {i = 0} ^ {k} \alpha_ {i} \mathbb {E} [ f (x _ {i}) - f (x ^ {*}) ] \leq \frac {1}{2} \left(S _ {k} + \frac {d \bar {\alpha}}{\Delta \ell} \left[ S _ {k} + \sum_ {i = 0} ^ {k} \rho_ {i} u _ {i} \right] + \sum_ {i = 0} ^ {k} \rho_ {i} u _ {i}\right) \\ = \frac {\ell \Delta + d \bar {\alpha}}{2 \ell \Delta} \left(S _ {k} + \sum_ {i = 0} ^ {k} \rho_ {i} u _ {i}\right) \\ \leq \frac {\ell \Delta + d \bar {\alpha}}{2 \ell \Delta} \left(S _ {k} + \sum_ {i = 0} ^ {k} \rho_ {i} \left(\sqrt {S _ {i}} + \sum_ {j = 0} ^ {i} \rho_ {j}\right)\right). \\ \end{array}
$$
Let  $\bar{x}_k := \sum_{i=0}^k \alpha_i x_i / (\sum_{i=0}^k \alpha_i)$ . Dividing both sides by  $\sum_{i=0}^k \alpha_i$ , convexity yields
$$
\mathbb {E} [ f (\bar {x} _ {k}) - \min  f ] \leq \frac {\sum_ {i = 0} ^ {k} \alpha_ {i} \mathbb {E} [ (f (x _ {i}) - f (x ^ {*})) ]}{\sum_ {i = 0} ^ {k} \alpha_ {i}}.
$$
B.6 Proof of Corollary 4
In this proof, we use the same notation as the one in the proof of Theorem 3. By the choices of the parameters, we have
$$
\sum_ {i = 0} ^ {k} \rho_ {i} \leq C _ {1} d \alpha h \quad \text {w i t h} \quad C _ {1} := \frac {L _ {1} \theta}{\theta - 1},
$$
$$
S _ {k} \leq \| x _ {0} - x ^ {*} \| ^ {2} + C _ {2} \frac {d ^ {2}}{\ell} \alpha^ {2} h ^ {2} \quad \text {w i t h} \quad C _ {2} := \frac {L _ {1} ^ {2} \theta}{2 \theta - 1}.
$$
Thus, using these inequalities in Theorem 3, we get
$$
\begin{array}{l} D _ {k} \leq \frac {\ell \Delta + d \bar {\alpha}}{2 \ell \Delta} \Big (\| x _ {0} - x ^ {*} \| ^ {2} + C _ {2} \frac {d ^ {2} \alpha^ {2} h ^ {2}}{\ell} + \sqrt {\| x _ {0} - x ^ {*} \| ^ {2}} C _ {3} d \alpha h \\ \left. + C _ {4} \frac {d \alpha h}{\sqrt {\ell}} + C _ {5} d ^ {2} \alpha^ {2} h ^ {2}\right), \\ \end{array}
$$
$$
C _ {3} := \frac {L _ {1} ^ {2} \theta}{\theta - 1}, \quad C _ {4} := \frac {L _ {1} ^ {2}}{\sqrt {2}}, \quad C _ {5} := \frac {L _ {1} ^ {2} \theta}{(\theta - 1) ^ {2}}.
$$
Dividing by  $\sum_{i=0}^{k} \alpha_{i}$ , we get
$$
\mathbb {E} \left[ f \left(\bar {x} _ {k}\right) - \min  f \right] \leq \frac {C}{\alpha k}.
$$
Note that by Assumption 4,  $\alpha < \ell /(dL_1)$ , thus  $1 / \alpha > (dL_{1}) / \ell$ . The algorithm performs  $2\ell$  function evaluations at each iteration. Thus, to guarantee  $\mathbb{E}[f(\bar{x}_k) - \min f] \leq \varepsilon$  for  $\varepsilon \in (0,1)$ , the algorithm has to perform a number of function evaluations in the order of
$$
\mathcal {O} (d \varepsilon^ {- 1}).
$$
Assuming, instead,  $\alpha_{k} \leq \bar{\alpha} < \ell / (2dL_{1})$ , by Lemma 9 we get the last claim; i.e., there exists a random variable  $\hat{x}$  taking values in  $\arg \min f$  s.t.  $x_{k} \to \hat{x}$  a.s.
B.7 Proof of Theorem 4
Set  $C_1 = (dL_1) / 2$ . It follows from Lemma 8 that
$$
\mathbb {E} [ f (x _ {k + 1}) | \mathcal {F} _ {k} ] - f (x _ {k}) \leq - \left(\frac {1}{2} - \frac {L _ {1} d}{\ell} \bar {\alpha}\right) \alpha_ {k} \| \nabla f (x _ {k}) \| ^ {2} + \frac {C _ {1} ^ {2} \alpha_ {k} h _ {k} ^ {2}}{2} + \frac {L _ {1} ^ {3} d ^ {2}}{4 \ell} \alpha_ {k} ^ {2} h _ {k} ^ {2}.
$$
Taking the full expectation and rearranging the terms, and recalling the definition of  $\Delta$ ,
$$
\Delta \alpha_ {k} \mathbb {E} [ \| \nabla f (x _ {k}) \| ^ {2} ] \leq \mathbb {E} [ f (x _ {k}) - f (x _ {k + 1}) ] + \frac {C _ {1} ^ {2} \alpha_ {k} h _ {k} ^ {2}}{2} + \frac {L _ {1} ^ {3} d ^ {2}}{4 \ell} \alpha_ {k} ^ {2} h _ {k} ^ {2}.
$$
Summing for  $i = 0, \dots, k$  and observing that  $\min f \leq f(x)$  for every  $x$ ,
$$
\Delta \sum_ {i = 0} ^ {k} \alpha_ {i} \mathbb {E} [ \| \nabla f (x _ {i}) \| ^ {2} ] \leq f (x _ {0}) - \min  f + \sum_ {i = 0} ^ {k} \frac {C _ {1} ^ {2} \alpha_ {i} h _ {i} ^ {2}}{2} + \frac {L _ {1} ^ {3} d ^ {2}}{4 \ell} \sum_ {i = 0} ^ {k} \alpha_ {i} ^ {2} h _ {i} ^ {2}.
$$
Divinding by  $\Delta \sum_{i=0}^{k} \alpha_{i}$  we get the claim.
B.8 Proof of Corollary 5
(i): From the choice of  $\alpha_{k}$  and  $h_k$ , we have
$$
\sum_ {i = 0} ^ {k} \alpha_ {i} h _ {i} ^ {2} \leq \frac {2 \theta \alpha h ^ {2}}{2 \theta - 1} \quad \sum_ {i = 0} ^ {k} \alpha_ {i} ^ {2} h _ {i} ^ {2} \leq \frac {2 \theta \alpha^ {2} h ^ {2}}{2 \theta - 1}.
$$
It follows from Theorem 4 that
$$
\eta_ {k} \leq \frac {1}{\Delta \alpha k} \bigg (f (x _ {0}) - \min f + C _ {1} d ^ {2} \alpha h ^ {2} + \frac {C _ {2} \alpha^ {2} h ^ {2} d ^ {2}}{\ell} \bigg),
$$
with  $C_1 = \frac{L_1^2\theta}{4(2\theta - 1)}$  and  $C_2 = \frac{L_1^3\theta}{2(2\theta - 1)}$
$(ii)$ : It follows directly from Theorem 4 taking into account that
$$
\sum_ {i = 0} ^ {k} \alpha_ {i} h _ {i} ^ {2} = k \alpha h ^ {2}, \quad \sum_ {i = 0} ^ {k} \alpha_ {i} ^ {2} h _ {i} ^ {2} = k \alpha^ {2} h ^ {2},
$$
and setting  $C_1 = L_1^2 /8$  and  $C_2 = L_1^3 /4$
C Experimental Details
In this appendix, we report details on the experiments performed. We implemented every script in Python3 (version 3.9.11) and used numpy (version 1.22.2) [27] and matplotlib (version 3.5.1) [29] libraries.
Machine used to perform the experiments. In the following table, we describe the features of the machine used to perform the experiments in Section 4.
Target Functions. We considered two synthetic target functions: a convex smooth function  $f_{1}$  and a convex non-smooth function  $f_{2}$  defined as follows
$$
\text {(C o n v e x S m o o t h)} \quad f _ {1} (x) := \frac {1}{2} \| A x \| ^ {2} \quad \text {w i t h} \quad A \in \mathbb {R} ^ {d \times d}
$$
$$
(\text {C o n v e x N o n - s m o o t h}) \quad f _ {2} (x) := \| x - \bar {v} \| _ {1}
$$
where  $A$  is a random Gaussian matrix (i.e.  $A_{i,j} \sim \mathcal{N}(0,1)$ ) and  $\bar{v} \coloneqq [0,1,\dots,d - 1]^{\top}$ .
Choice of the number of directions. We report here the details of the first experiment of Section 4. For these experiments, we consider  $d = 50$  and we use, for the smooth convex case, the following parameters
$$
\alpha_ {k} = 0. 9 9 \frac {\ell}{d L _ {1}} \quad \text {a n d} \quad h _ {k} = \frac {1 0 ^ {- 5}}{k + 1}.
$$
The constant  $L_{1}$  is computed as the maximum eigenvalue of the matrix  $A^{\mathrm{T}}A$ . Note that this parameter choice satisfies Assumption 4. For the non-smooth target, we used
$$
\alpha_ {k} = \sqrt {\frac {\ell}{d}} k ^ {- 1 / 2 - 1 0 ^ {- 5}} \qquad \text {a n d} \qquad h _ {k} = \frac {1 0 ^ {- 7}}{k + 1}.
$$
Note that this parameter configuration satisfies Assumption 2. The maximum number of function evaluations considered is 4000. The direction matrices  $G_{k}$  are generated with the QR method - see Appendix D.
Comparison with Finite-difference methods. In Section 4, we compare finite-difference method with different choice of directions. In order to make a fair comparison we consider only central finite-differences. However, note that Algorithm 1 can be modified (in practice) considering computationally cheaper gradient estimators - see Remark 1. For these experiments, we consider  $d = 10$  and  $\ell = d$  for methods with multiple directions. The maximum number of function evaluations is 1000 for both smooth and non-smooth targets and the direction matrices  $G_{k}$  for Algorithm 1 are generated with the QR method - see Appendix D. To solve the smooth problem we consider the following parameter choice for every method
$$
\alpha_ {k} = c \frac {\ell}{d L _ {1}} \quad \text {a n d} \quad h _ {k} = \frac {1 0 ^ {- 7}}{d ^ {2} (k + 1)},
$$
where  $L_{1}$  is computed taking the maximum eigenvalue of  $A^{\top}A$ . For Algorithm 1 and finite-difference with single and multiple spherical directions  $c = 0.99$  while it is equal to  $c = 0.11$  for finite-difference with single and multiple Gaussian directions. We made this choice since for finite-difference methods with Gaussian directions we observed divergence for larger choices of  $c$  - see Figure 3.
For the non-smooth convex target, we considered the following parameter choice
$$
\alpha_ {k} = c \frac {\ell}{d} k ^ {- 1 / 2 - 1 0 ^ {- 5}} \quad \text {a n d} \quad h _ {k} = \frac {1}{d ^ {2} (k + 1)}.
$$
For every method, we selected  $c = 0.65$  except for the method with multiple Gaussian directions in which we selected  $c = 0.08$  since it provided better performances - see Figure 4.
D Techniques to Generate Orthogonal Direction Matrices
In the literature, different algorithms were proposed to generate orthogonal matrices - see for instance [23, 38, 11, 28, 7, 2, 4, 44, 8] and references therein. Such methods can be used to generate the direction matrices  $G_{k}$  required for the iteration proposed in Algorithm (1). In this appendix, we briefly discuss three of them.
QR factorization. As observed in [32, 42], a way to generate orthogonal consists in generating a random Gaussian matrix  $A \in \mathbb{R}^{d \times d}$  with  $A_{i,j} \sim \mathcal{N}(0,1)$  and performing the QR factorization i.e.  $A = QR$ . Then, the direction matrix is the truncation of the  $Q$  matrix i.e.  $QI_{d,\ell}$ .
Householder Reflection. To obtain a direction matrix, we can use a Householder reflector. This can be done by sampling a vector  $v$  from the unit sphere  $\mathbb{S}^{d - 1}$ . The direction matrix  $G$  is defined as a Householder reflector, given by
$$
G := I - 2 v v ^ {\intercal},
$$
with  $I \in \mathbb{R}^{d \times d}$  identity matrix. To obtain the desired matrix, we compute the product of  $G$  with  $I_{d,\ell}$ , i.e., we take the first  $\ell$  columns. The (truncated) identity matrix can be generated and stored offline (note that since it is very sparse, it can be stored using a sparse format (e.g. the COO format proposed in scikit-learn library[9]). In this way, we can save resources in high-dimensional settings. In order to quantify the time-cost of this procedure, we compared the time of generating this kind of matrix with random matrices with different dimensions. For this experiment, we consider the  $\ell = d$  case i.e. the most expensive. Matrices are computed in CPU and the details of the machine used are described in Appendix C. We report the mean and standard deviation of the time using 500 repetitions. In Figure 5, we compare the time-cost of generating orthogonal matrices with this procedure against generating random matrices while in Table 2 we report the mean and standard deviation of the results.
In Figure 5, we can observe that using this strategy we can limit the cost of generating random orthogonal matrices. In particular, for dimensions larger than 32, our method is faster than random gaussian and spherical directions.
Moreover, if more computational resources are available, we can build  $m > 1$  Householder reflectors  $G_{1},\dots ,G_{m}$  using  $m$  random vectors  $v_{1},\dots ,v_{m}$  sampled i.i.d from  $\mathbb{S}^{d - 1}$  and define the direction matrix as
$$
G _ {1} G _ {2} \dots G _ {m} I _ {d, \ell}.
$$
It is important to note that when  $m = d$ , this procedure is equivalent to using the QR factorization.
Haar Butterfly matrices. We can build orthogonal matrices using Butterfly matrices [48]. Let  $G^{(0)} \coloneqq [1]$ , we can build an orthogonal matrix of dimension  $d = 2^n$  with the following recursion
$$
G ^ {(n)} = \left[ \begin{array}{c c} \cos (\theta_ {n}) G ^ {(n - 1)} & \sin (\theta_ {n}) G ^ {(n - 1)} \\ - \sin (\theta_ {n}) G ^ {(n - 1)} & \cos (\theta_ {n}) G ^ {(n - 1)} \end{array} \right]
$$
where  $\theta_{n}$  is sampled uniformly in  $[0,2\pi ]$ . Then we compute  $G I_{d,\ell}$  (we take the first  $\ell$  columns). The construction of Haar butterfly matrices is faster than previous methods because it only requires simple operations. However, this procedure allows to build only matrices with  $d = 2^{n}$  for  $n\geq 0$ . In literature, different methods were proposed to cope with this limitation e.g. [23].
In this appendix, we discuss the main practical limitations of Algorithm 1. Like all finite-difference methods with multiple directions, O-ZD requires multiple function evaluations to execute a single step. In many practical applications, function evaluations can be time-consuming, leading to the use of a small number of directions  $\ell$ . This may result in poor performance as observed in numerical experiments. As for the subgradient method, in O-ZD the step size significantly affects performance, and tuning it can be challenging. To address this limitation, an adaptive stepsize selection method could be proposed. Furthermore, decreasing the sequence  $h_k$  too quickly can lead to numerical instability, as noted in [42].
F Other Experiments
We performed other experiments in minimizing convex functions. We considered the targets defined in Table 3 and, for each experiment, we reported the mean and standard deviation using 20 repetitions.
In Figure 6, we can observe that structured finite-difference performs better than unstructured methods.
In Table 3, we define the function used for the experiments. In particular:
- Sparse Group Lasso:  $p$  is set to 3 and given an  $x \in \mathbb{R}^d$ ,  $x^{(\beta_i)}$  is a vector obtained by taking 3 entries of  $x$ .
- Huber Loss:  $\delta$  is set to 0.5.
- Elastic Net:  $\alpha, \beta$  are set to 0.5.