SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Nikita Kornilov
MIPT, SkolTech
kornilov.nm@phystech.edu
Ohad Shamir
ohad.shamir@weizmann.ac.il
Aleksandr Lobanov
MIPT, ISP RAS
lobanov.av@mipt.ru
Darina Dvinskikh
HSE University, ISP RAS
dmdvinskikh@hse.ru
Alexander Gasnikov
MIPT,
ISP RAS, SkolTech
gasnikov@yandex.ru
Innokentiy Shibaev
IITP RAS
innokenti.shibayev@phystech.edu
Eduard Gorbunov
MBZUAI
eduard.gorbunov@mbzuai.ac.ae
Samuel Horváth
samuel.horvath@mbzuai.ac.ae
In this paper, we consider non-smooth stochastic convex optimization with two function evaluations per round under infinite noise variance. In the classical setting when noise has finite variance, an optimal algorithm, built upon the batched accelerated gradient method, was proposed in [17]. This optimality is defined in terms of iteration and oracle complexity, as well as the maximal admissible level of adversarial noise. However, the assumption of finite variance is burdensome and it might not hold in many practical scenarios. To address this, we demonstrate how to adapt a refined clipped version of the accelerated gradient (Stochastic Similar Triangles) method from [35] for a two-point zero-order oracle. This adaptation entails extending the batching technique to accommodate infinite variance — a non-trivial task that stands as a distinct contribution of this paper.
In this paper, we consider stochastic non-smooth convex optimization problem
$$
\min  _ {x \in \mathbb {R} ^ {d}} \left\{f (x) \stackrel {\text {d e f}} {=} \mathbb {E} _ {\xi \sim \mathcal {D}} [ f (x, \xi) ] \right\}, \tag {1}
$$
where  $f(x,\xi)$  is  $M_2(\xi)$ -Lipschitz continuous in  $x$  w.r.t. the Euclidean norm, and the expectation  $\mathbb{E}_{\xi \sim \mathcal{D}}[f(x,\xi)]$  is with respect to a random variable  $\xi$  with unknown distribution  $\mathcal{D}$ . The optimization is performed only by accessing two function evaluations per round rather than sub-gradients, i.e., for any pair of points  $x,y\in \mathbb{R}^d$ , an oracle returns  $f(x,\xi)$  and  $f(y,\xi)$  with the same  $\xi$ . The primary motivation for this gradient-free oracle arises from different applications where calculating gradients is computationally infeasible or even impossible. For instance, in medicine, biology, and physics, the objective function can only be computed through numerical simulation or as the result of a real experiment, i.e., automatic differentiation cannot be employed to calculate function derivatives. Usually, a black-box function we are optimizing is affected by stochastic or computational noise. This noise can arise naturally from modeling randomness within a simulation or by computer discretization.
In classical setting, this noise has light tails. However, usually in black-box optimization, we know nothing about the function, only its values at the requested points are available/computable, so light tails assumption may be violated. In this case, gradient-free algorithms may diverge. We aim to develop an algorithm that is robust even to heavy-tailed noise, i.e., noise with infinite variance. In theory, one can consider heavy-tailed noise to simulate situations where noticeable outliers may occur (even if the nature of these outliers is non-stochastic). Therefore we relax classical finite variance assumption and consider less burdensome assumption of finite  $\alpha$ -th moment, where  $\alpha \in (1,2]$ .
In machine learning, interest in gradient-free methods is mainly driven by the bandit optimization problem [14, 2, 5], where a learner engages in a game with an adversary: the learner selects a point  $x$ , and the adversary chooses a point  $\xi$ . The learner's goal is to minimize the average regret based solely on observations of function values (losses)  $f(x, \xi)$ . As feedback, at each round, the learner receives losses at two points. This corresponds to a zero-order oracle in stochastic convex optimization with two function evaluations per round. The vast majority of researches assume sub-Gaussian distribution of rewards. However, in some practical cases (e.g., in finance [33]) reward distribution has heavy-tails or can be adversarial. For the heavy-tailed bandit optimization, we refer to [9].
Two-point gradient-free optimization for non-smooth (strongly) convex objective is a well-researched area. Numerous algorithms have been proposed which are optimal with respect to two criteria: oracle and iteration complexity. For a detailed overview, see the recent survey [15] and the references therein. Optimal algorithms, in terms of oracle call complexity, are presented in [10, 36, 3]. The distinction between the number of successive iterations (which cannot be executed in parallel) and the number of oracle calls was initiated with the lower bound obtained in [6]. It culminated with the optimal results from [17], which provides an algorithm which is optimal in both criteria. Specifically, the algorithm produces  $\hat{x}$ , an  $\varepsilon$ -solution of (1), such that we can guarantee  $\mathbb{E}[f(\hat{x})] - \min_{x \in \mathbb{R}^d} f(x)$  after
$$
\begin{array}{l} \sim d ^ {\frac {1}{4}} \varepsilon^ {- 1} \quad \text {s u c c e s s i v e i t e r a t i o n s}, \\ \sim d \varepsilon^ {- 2} \quad \text {o r a c l e c a l l s (n u m b e r o f} f (x, \xi) \text {c a l c u l a t i o n s)}. \\ \end{array}
$$
The convergence guarantee for this optimal algorithm from [17] was established in the classical setting of light-tailed noise, i.e., when noise has finite variance:  $\mathbb{E}_{\xi}[M_2(\xi)^2] < \infty$ . However, in many modern learning problems the variance might not be finite, leading the aforementioned algorithms to potentially diverge. Indeed, heavy-tailed noise is prevalent in contemporary applications of statistics and deep learning. For example, heavy-tailed behavior can be observed in training attention models [41] and convolutional neural networks [37, 20]. Consequently, our goal is to develop an optimal algorithm whose convergence is not hampered by this restrictive assumption. To the best of our knowledge, no existing literature on gradient-free optimization allows for  $\mathbb{E}_{\xi}[M_2(\xi)^2]$  to be infinite. Furthermore, convergence results for all these gradient-free methods were provided in expectation, that is, without (non-trivial) high-probability bounds. Although the authors of [17] mentioned (without proof) that their results can be formulated in high probability using [19], this aspect notably affects the oracle calls complexity bound and complicates the analysis.
A common technique to relax finite variance assumption is gradient clipping [31]. Starting from the work of [26] (see also [8, 19]), there has been increased interest in algorithms employing gradient clipping to achieve high-probability convergence guarantees for stochastic optimization problems with heavy-tailed noise. Particularly, in just the last two years there have been proposed
- an optimal algorithm with a general proximal setup for non-smooth stochastic convex optimization problems with infinite variance [39] that converges in expectation (also referenced in [27]),
- an optimal adaptive algorithm with a general proximal setup for non-smooth online stochastic convex optimization problems with infinite variance [42] that converges with high probability,
- optimal algorithms using the Euclidean proximal setup for both smooth and non-smooth stochastic convex optimization problems and variational inequalities with infinite variance [35, 30, 29] that converge with high probability,
- an optimal variance-adaptive algorithm with the Euclidean proximal setup for non-smooth stochastic (strongly) convex optimization problems with infinite variance [24] that converges with high probability.
None of these papers discuss a gradient-free oracle. Moreover, they do not incorporate acceleration (given the non-smooth nature of the problems) with the exception of [35]. Acceleration is a crucial
component to achieving optimal bounds on the number of successive iterations. However, the approach in [35] presumes smoothness and does not utilize batching. To apply the convergence results from [35] to our problem, we need to adjust our problem formulation to be smooth. This is achieved by using the Smoothing technique [27, 36, 17]. In the work [22] authors proposed an algorithm based on Smoothing technique and non-accelerated Stochastic Mirror Descent with clipping. However, this work also does not support acceleration, minimization on the whole space and batching. Adapting the technique from [35] to incorporate batching necessitates a substantial generalization. We regard this aspect of our work as being of primary interest.
Heavy-tailed noise can also be handled without explicit gradient clipping, e.g., by using Stochastic Mirror Descent algorithm with a particular class of uniformly convex mirror maps [39]. However, the convergence guarantee for this algorithm is given in expectation. Moreover, applying batching and acceleration is a non-trivial task. Without this, we are not able to get the optimal method in terms of the number of iterations and not only in terms of oracle complexity. There are also some studies on the alternatives to gradient clipping [21] but the results for these alternatives are given in expectation and are weaker than the state-of-the-art results for the methods with clipping. This is another reason why we have chosen gradient clipping to handle the heavy-tailed noise.
1.1 Contributions
We generalize the optimal result from [17] to accommodate a weaker assumption that allows the noise to exhibit heavy tails. Instead of the classical assumption of finite variance, we require the boundedness of the  $\alpha$ -moment: there exists  $\alpha \in (1,2]$  such that  $\mathbb{E}_{\xi}[M_2(\xi)^{\alpha}] < \infty$ . Notably, when  $\alpha < 2$ , this assumption is less restrictive than the assumption of a finite variance and thus it has garnered considerable interest recently, see [41, 35, 29] and the references therein. Under this assumption we prove that for convex  $f$ , an  $\varepsilon$ -solution can be found with high probability after
$$
\begin{array}{l} \sim d ^ {\frac {1}{4}} \varepsilon^ {- 1} \quad \text {s u c c e s s i v e i t e r a t i o n s}, \\ \sim \left(\sqrt {d} / \varepsilon\right) ^ {\frac {\alpha}{\alpha - 1}} \quad \text {o r a c l e} \\ \end{array}
$$
and for  $\mu$ -strongly convex  $f$ , the  $\varepsilon$ -solution can be found with high probability after
$$
\begin{array}{l} \sim d ^ {\frac {1}{4}} (\mu \varepsilon) ^ {- \frac {1}{2}} \quad \text {s u c c e s s i v e i t e r a t i o n s}, \\ \sim \left(d / (\mu \varepsilon)\right) ^ {\frac {\alpha}{2 (\alpha - 1)}} \quad \text {o r a c l e} \\ \end{array}
$$
In both instances, the number of oracle calls is optimal in terms of  $\varepsilon$ -dependency within the non-smooth setting [27, 39]. For first-order optimization under heavy-tailed noise, the optimal  $\varepsilon$ -dependency remains consistent, as shown in [35, Table 1].
In what follows, we highlight the following several important aspects of our results
- High-probability guarantees. We provide upper-bounds on the number of iterations/oracle calls needed to find a point  $\hat{x}$  such that  $f(\hat{x}) - \min_{x\in \mathbb{R}^d}f(x)\leq \varepsilon$  with probability at least  $1 - \beta$ . The derived bounds have a poly-logarithmic dependence on  $1 / \beta$ . To the best of our knowledge, there are no analogous high-probability results, even for noise with bounded variance.
- Generality of the setup. Our results are derived under the assumption that gradient-free oracle returns values of stochastic realizations  $f(x, \xi)$  subject to (potentially adversarial) bounded noise. We further provide upper bounds for the magnitude of this noise, contingent upon the target accuracy  $\varepsilon$  and confidence level  $\beta$ . Notably, our assumptions about the objective and noise are confined to a compact subset of  $\mathbb{R}^d$ . This approach, which differs from standard ones in derivative-free optimization, allows us to encompass a wide range of problems. This approach differs from standard ones in derivative-free optimization
- Batching without bounded variance. To establish the aforementioned upper bounds, we obtain the following: given  $X_{1},\ldots ,X_{B}$  as independent random vectors in  $\mathbb{R}^d$  where  $\mathbb{E}[X_i] = x\in \mathbb{R}^d$  and  $\mathbb{E}\| X_i - x\| _2^\alpha \leq \sigma^\alpha$  for some  $\sigma \geq 0$  and  $\alpha \in (1,2]$ , then
$$
\mathbb {E} \left[ \left\| \frac {1}{B} \sum_ {i = 1} ^ {B} X _ {i} - x \right\| _ {2} ^ {\alpha} \right] \leq \frac {2 \sigma^ {\alpha}}{B ^ {\alpha - 1}}. \tag {2}
$$
When  $\alpha = 2$ , this result aligns with the conventional case of bounded variance (accounting for a numerical factor of 2). Unlike existing findings, such as [40, Lemma 7] where  $\alpha < 2$ , the relation (2) does not exhibit a dependency on the dimension  $d$ . Moreover, (2) offers a theoretical basis to highlight the benefits of mini-batching, applicable to methods highlighted in this paper as well as first-order methods presented in [35, 29].
- Dependency on  $d$ . As far as we are aware, an open question remains: is the bound  $\left(\sqrt{d} / \varepsilon\right)^{\frac{\alpha}{\alpha - 1}}$  optimal regarding its dependence on  $d$ ? For smooth stochastic convex optimization problems using a  $(d + 1)$ -points stochastic zeroth-order oracle, the answer is negative. The optimal bound is proportional to  $d\varepsilon^{-\frac{\alpha}{\alpha - 1}}$ . Consequently, for  $\alpha \in (1,2)$ , our results are intriguing because the dependence on  $d$  in our bound differs from known results in the classical case where  $\alpha = 2$ .
1.2 Paper organization
The paper is organized as follows. In Section 2, we give some preliminaries, such as smoothing technique and gradient estimation, that are workhorse of our algorithms. Section 3 is the main section presenting two novel gradient-free algorithms along with their convergence results in high probability. These algorithms solve non-smooth stochastic optimization under heavy-tailed noise, and they will be referred to as ZO-clipped-SSTM and R-ZO-clipped-SSTM (see Algorithms 1 and 2 respectively). In Section 4, we extend our results to gradient-free oracle corrupted by additive deterministic adversarial noise. In Section 5, we describe the main ideas behind the proof and emphasize key lemmas. In Section 6, we provide numerical experiments on the synthetic task that demonstrate the robustness of the proposed algorithms towards heavy-tailed noise.
Assumptions on a subset. Although we consider an unconstrained optimization problem, our analysis does not require any assumptions to hold on the entire space. For our purposes, it is sufficient to introduce all assumptions only on some convex set  $Q \in \mathbb{R}^d$  since we prove that the considered methods do not leave some ball around the solution or some level-set of the objective function with high probability. This allows us to consider fairly large classes of problems.
Assumption 1 (Strong convexity) There exist a convex set  $Q \subset \mathbb{R}^d$  and  $\mu \geq 0$  such that function  $f(x, \xi)$  is  $\mu$ -strongly convex on  $Q$  for any fixed  $\xi$ , i.e.
$$
f (\lambda x _ {1} + (1 - \lambda) x _ {2}) \leq \lambda f (x _ {1}) + (1 - \lambda) f (x _ {2}) - \frac {1}{2} \mu \lambda (1 - \lambda) \| x _ {1} - x _ {2} \| _ {2} ^ {2},
$$
for all  $x_{1},x_{2}\in Q,\lambda \in [0,1]$
This assumption implies that  $f(x)$  is  $\mu$ -strongly convex as well.
For a small constant  $\tau > 0$ , let us define an expansion of set  $Q$  namely  $Q_{\tau} = Q + B_2^d$ , where  $+$  stands for Minkowski addition. Using this expansion we make the following assumption.
Assumption 2 (Lipschitz continuity and boundedness of  $\alpha$ -moment) There exist a convex set  $Q \subset \mathbb{R}^d$  and  $\tau > 0$  such that function  $f(x, \xi)$  is  $M_2(\xi)$ -Lipschitz continuous w.r.t. the Euclidean norm on  $Q_{\tau}$ , i.e., for all  $x_1, x_2 \in Q_{\tau}$
$$
\left| f \left(x _ {1}, \xi\right) - f \left(x _ {2}, \xi\right) \right| \leq M _ {2} (\xi) \| x _ {1} - x _ {2} \| _ {2}.
$$
Moreover, there exist  $\alpha \in (1,2]$  and  $M_2 > 0$  such that  $\mathbb{E}_{\xi}[M_2(\xi)^{\alpha}]\leq M_2^{\alpha}$ .
If  $\alpha < 2$ , we say that noise is heavy-tailed. When  $\alpha = 2$ , the above assumption recovers the standard uniformly bounded variance assumption.
Lemma 1 Assumption 2 implies that  $f(x)$  is  $M_2$ -Lipschitz on  $Q$ .
Randomized smoothing. The main scheme that allows us to develop batch-parallel gradient-free methods for non-smooth convex problems is randomized smoothing [13, 17, 27, 28, 38] of a non-smooth function  $f(x,\xi)$ . The smooth approximation to a non-smooth function  $f(x,\xi)$  is defined as
$$
\hat {f} _ {\tau} (x) \stackrel {\text {d e f}} {=} \mathbb {E} _ {\mathbf {u}, \xi} [ f (x + \tau \mathbf {u}, \xi) ], \tag {3}
$$
where  $\mathbf{u} \sim U(B_2^d)$  is a random vector uniformly distributed on the Euclidean unit ball  $B_2^d \stackrel{\mathrm{def}}{=} \{x \in \mathbb{R}^d : \|x\|_2 \leq 1\}$ . In this approximation, a new type of randomization appears in addition to stochastic variable  $\xi$ .
The next lemma gives estimates for the quality of this approximation. In contrast to  $f(x)$ , function  $\hat{f}_{\tau}(x)$  is smooth and has several useful properties.
Lemma 2 [17, Theorem 2.1.] Let there exist a subset  $Q \subset \mathbb{R}^d$  and  $\tau > 0$  such that Assumptions 1 and 2 hold on  $Q_{\tau}$ . Then,
1. Function  $\hat{f}_{\tau}(x)$  is convex, Lipschitz with constant  $M_2$  on  $Q$ , and satisfies
$$
\sup  _ {x \in Q} | \hat {f} _ {\tau} (x) - f (x) | \leq \tau M _ {2}.
$$
2. Function  $\hat{f}_{\tau}(x)$  is differentiable on  $Q$  with the following gradient
$$
\nabla \hat {f} _ {\tau} (x) = \mathbb {E} _ {\mathbf {e}} \left[ \frac {d}{\tau} f (x + \tau \mathbf {e}) \mathbf {e} \right],
$$
where  $\mathbf{e} \sim U(S_2^d)$  is a random vector uniformly distributed on unit Euclidean sphere  $S_2^d \stackrel{\text{def}}{=} \{x \in \mathbb{R}^d : \|x\|_2 = 1\}$ .
3. Function  $\hat{f}_{\tau}(x)$  is  $L$ -smooth with  $L = \sqrt{d} M_2 / \tau$  on  $Q$ .
Our algorithms will aim at minimizing the smooth approximation  $\hat{f}_{\tau}(x)$ . Given Lemma 2, the output of the algorithm will also be a good approximate minimizer of  $f(x)$  when  $\tau$  is sufficiently small.
Gradient estimation. Our algorithms will be based on randomized gradient estimates, which will then be used in a first order algorithm. Following [36], the gradient can be estimated by the following vector:
$$
g (x, \xi , \mathbf {e}) = \frac {d}{2 \tau} \left(f (x + \tau \mathbf {e}, \xi) - f (x - \tau \mathbf {e}, \xi)\right) \mathbf {e}, \tag {4}
$$
where  $\tau > 0$  and  $\mathbf{e} \sim U(S_2^d)$ . We also use batching technique in order to allow parallel calculation of gradient estimation and acceleration. Let  $B$  be a batch size, we sample  $\{\mathbf{e}_i\}_{i=1}^B$  and  $\{\xi_i\}_{i=1}^B$  independently, then
$$
g ^ {B} (x, \{\xi \}, \{\mathbf {e} \}) = \frac {d}{2 B \tau} \sum_ {i = 1} ^ {B} \left(f \left(x + \tau \mathbf {e} _ {i}, \xi_ {i}\right) - f \left(x - \tau \mathbf {e} _ {i}, \xi_ {i}\right)\right) \mathbf {e} _ {i}. \tag {5}
$$
The next lemma states that  $g^{B}(x, \{\xi\}, \{\mathbf{e}\})$  from (5) is an unbiased estimate of the gradient of  $\hat{f}_{\tau}(x)$  (3). Moreover, under heavy-tailed noise (Assumption 2) with bounded  $\alpha$ -moment  $g^{B}(x, \{\xi\}, \{\mathbf{e}\})$  has also bounded  $\alpha$ -moment.
Lemma 3 Under Assumptions 1 and 2, it holds
$$
\mathbb {E} _ {\xi , \mathbf {e}} [ g (x, \xi , \mathbf {e}) ] = \mathbb {E} _ {\{\xi \}, \{\mathbf {e} \}} [ g ^ {B} (x, \{\xi \}, \{\mathbf {e} \}) ] = \nabla \hat {f} _ {\tau} (x).
$$
$$
\mathbb {E} _ {\xi , \mathbf {e}} [ \| g (x, \xi , \mathbf {e}) - \mathbb {E} _ {\xi , \mathbf {e}} [ g (x, \xi , \mathbf {e}) ] \| _ {2} ^ {\alpha} ] \leq \sigma^ {\alpha} \stackrel {{d e f}} {{=}} \left(\frac {\sqrt {d} M _ {2}}{2 ^ {\frac {1}{4}}}\right) ^ {\alpha},
$$
$$
\mathbb {E} _ {\{\xi \}, \{\mathbf {e} \}} [ \| g ^ {B} (x, \{\xi \}, \{\mathbf {e} \}) - \mathbb {E} _ {\{\xi \}, \{\mathbf {e} \}} [ g ^ {B} (x, \{\xi \}, \{\mathbf {e} \}) ] \| _ {2} ^ {\alpha} ] \leq \frac {2 \sigma^ {\alpha}}{B ^ {\alpha - 1}}. \tag {6}
$$
3 Main Results
In this section, we present two our new zero-order algorithms, which we will refer to as ZO-clipped-SSTM and R-ZO-clipped-SSTM, to solve problem (1) under heavy-tailed noise assumption. To deal with heavy-tailed noise, we use clipping technique which clips heavy tails. Let  $\lambda > 0$  be clipping constant and  $g \in \mathbb{R}^d$ , then clipping operator clip is defined as
$$
\operatorname {c l i p} (g, \lambda) = \left\{ \begin{array}{l l} \frac {g}{\| g \| _ {2}} \min  \left(\| g \| _ {2}, \lambda\right), & g \neq 0, \\ 0, & g = 0. \end{array} \right. \tag {7}
$$
We apply clipping operator for batched gradient estimate  $g^{B}(x,\{\xi \} ,\{\mathbf{e}\})$  from (5) and then feed it into first-order Clipped Stochastic Similar Triangles Method (clipped-SSTM) from [18]. We will refer to our zero-order versions of clipped-SSTM as ZO-clipped-SSTM and R-ZO-clipped-SSTM for the convex case and strongly convex case respectively.
3.1 Convex case
Let us suppose that Assumption 1 is satisfied with  $\mu = 0$
Algorithm 1 ZO-clipped-SSTM  $\left(x^{0},K,B,a,\tau ,\{\lambda_{k}\}_{k = 0}^{K - 1}\right)$
Input: starting point  $x^0$  , number of iterations  $K$  , batch size  $B$  , stepsize  $a > 0$  , smoothing parameter  $\tau$  , clipping levels  $\{\lambda_k\}_{k = 0}^{K - 1}$    
1: Set  $L = \sqrt{d} M_2 / \tau$ $A_0 = \alpha_0 = 0$ $y^{0} = z^{0} = x^{0}$    
2: for  $k = 0,\dots ,K - 1$  do   
3: Set  $\alpha_{k + 1} = k + 2 / 2aL$ $A_{k + 1} = A_{k} + \alpha_{k + 1}$    
4:  $x^{k + 1} = \frac{A_ky^k + \alpha_{k + 1}z^k}{A_{k + 1}}.$    
5: Sample  $\{\xi_i^k\}_{i = 1}^B\sim \mathcal{D}$  and  $\{\mathbf{e}_i^k\}_{i = 1}^B\sim S_2^d$  independently.   
6: Compute gradient estimation  $g^{B}(x^{k + 1},\{\xi^{k}\} ,\{\mathbf{e}^{k}\})$  as defined in (5).   
7: Compute clipped  $\tilde{g}_{k + 1} = \mathrm{clip}\left(g^{B}(x^{k + 1},\{\xi^{k}\} ,\{\mathbf{e}^{k}\}),\lambda_{k}\right)$  as defined in (7).   
8:  $z^{k + 1} = z^{k} - \alpha_{k + 1}\tilde{g}_{k + 1}$    
9:  $y^{k + 1} = \frac{A_ky^k + \alpha_{k + 1}z^{k + 1}}{A_{k + 1}}.$    
10: end for   
Output:  $y^{K}$
Theorem 1 (Convergence of ZO-clipped-SSTM) Let Assumptions 1 and 2 hold with  $\mu = 0$  and  $Q = \mathbb{R}^d$ . Let  $\| x^0 - x^*\|^2 \leq R^2$ , where  $x^0$  is a starting point and  $x^*$  is an optimal solution to (1). Then for the output  $y^K$  of ZO-clipped-SSTM, run with batchsize  $B$ ,  $A = \ln^{4K} / \beta \geq 1$ ,  $a = \Theta(\max\{A^2, \sqrt{d}M_2K^{\frac{(\alpha + 1)}{\alpha}} A^{\frac{(\alpha - 1)}{\alpha}} /_{LRB}\frac{(\alpha - 1)}{\alpha}\})$ ,  $\tau = \varepsilon / 4M_2$  and  $\lambda_k = \Theta(R / (\alpha_{k+1}A))$ , we can guarantee  $f(y^K) - f(x^*) \leq \varepsilon$  with probability at least  $1 - \beta$  (for any  $\beta \in (0,1]$ ) after
$$
K = \widetilde {\mathcal {O}} \left(\max  \left\{\frac {M _ {2} \sqrt [ 4 ]{d} R}{\varepsilon}, \frac {1}{B} \left(\frac {\sqrt {d} M _ {2} R}{\varepsilon}\right) ^ {\frac {\alpha}{\alpha - 1}} \right\}\right) \tag {8}
$$
successive iterations and  $K \cdot B$  oracle calls. Moreover, with probability at least  $1 - \beta$  the iterates of ZO-clipped-SSTM remain in the ball  $B_{2R}(x^*)$ , i.e.,  $\{x^k\}_{k=0}^{K+1}, \{y^k\}_{k=0}^{K}, \{z^k\}_{k=0}^{K} \subseteq B_{2R}(x^*)$ .
Here and below notation  $\tilde{\mathcal{O}}$  means an upper bound on the growth rate hiding logarithms. The first term in bound (8) is optimal for the deterministic case for non-smooth problem (see [6]) and the second term in bound (8) is optimal in  $\varepsilon$  for  $\alpha \in (1,2]$  and zero-point oracle (see [27]).
We notice that increasing the batch size  $B$  to reduce the number of successive iterations makes sense only as long as the first term of (8) lower than the second one, i.e. there exists optimal value of batchsize
$$
B \leq \left(\frac {\sqrt {d} M _ {2} R}{\varepsilon}\right) ^ {\frac {1}{\alpha - 1}}.
$$
3.2 Strongly-convex case