SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Now we suppose that Assumption 1 is satisfied with  $\mu >0$ . For this case we employ ZO-clippedSSTM with restarts technique. We will call this algorithm as R-ZO-clipped-SSTM (see Algorithm 2). At each round R-ZO-clipped-SSTM call ZO-clipped-SSTM with starting point  $\hat{x}^t$ , which is the output from the previous round, and for  $K_{t}$  iterations.
Algorithm 2 R-ZO-clipped-SSTM
Input: starting point  $x^0$ , number of restarts  $N$ , number of steps  $\{K_t\}_{t=1}^N$ , batchesizes  $\{B_t\}_{t=1}^N$ , stepsizes  $\{a_t\}_{t=1}^N$ , smoothing parameters  $\{\tau_t\}_{t=1}^N$ , clipping levels  $\{\lambda_k^1\}_{k=0}^{K_1-1}, \dots, \{\lambda_k^N\}_{k=0}^{K_N-1}$
1:  $\hat{x}^0 = x^0$
2: for  $t = 1, \dots, N$  do
3:  $\hat{x}^t = \mathsf{ZO}$ -clipped-SSTM  $\left(\hat{x}^{t - 1},K_t,B_t,a_t,\tau_t,\{\lambda_k^t\}_{k = 0}^{K_t - 1}\right)$ .
4: end for
Output:  $\hat{x}^N$
Theorem 2 (Convergence of R-ZO-clipped-SSTM) Let Assumptions 1, 2 hold with  $\mu >0$  and  $Q = \mathbb{R}^d$ . Let  $\| x^0 -x^*\| ^2\leq R^2$ , where  $x^0$  is a starting point and  $x^{*}$  is the optimal solution to (1). Let also  $N = \lceil \log_2(\mu R^2 /2\varepsilon)\rceil$  be the number of restarts. Let at each stage  $t = 1,\dots,N$  of R-ZOclipped-SSTM, ZO-clipped-SSTM is run with batchsize  $B_{t}$ ,  $\tau_t = \varepsilon_t / 4M_2$ ,  $L_{t} = M_{2}\sqrt{d} /\tau_{t}$ ,  $K_{t} = \widetilde{\Theta} (\max \{\sqrt{L_{t}R_{t - 1}^{2} / \varepsilon_{t}},(\sigma R_{t - 1} / \varepsilon_{t})^{\frac{\alpha}{(\alpha - 1)}} / B_{t}\})$ ,  $a_{t} = \widetilde{\Theta} (\max \{1,\sigma K_{t}^{\frac{\alpha + 1}{\alpha}} / L_{t}R_{t}\})$  and  $\lambda_k^t = \widetilde{\Theta} (R / \alpha_{k + 1}^t)$  where  $R_{t - 1} = 2^{-\frac{(t - 1)}{2}}R$ ,  $\varepsilon_t = \mu R_{t - 1}^2 /4$ ,  $\ln^{4NK_t / \beta}\geq 1$ ,  $\beta \in (0,1]$ . Then to guarantee  $f(\hat{x}^N) - f(x^*)\leq \varepsilon$  with probability at least  $1 - \beta$ , R-ZO-clipped-SSTM requires
$$
\widetilde {\mathcal {O}} \left(\max  \left\{\sqrt {\frac {M _ {2} ^ {2} \sqrt {d}}{\mu \varepsilon}}, \left(\frac {d M _ {2} ^ {2}}{\mu \varepsilon}\right) ^ {\frac {\alpha}{2 (\alpha - 1)}} \right\}\right) \tag {9}
$$
oracle calls. Moreover, with probability at least  $1 - \beta$  the iterates of R-ZO-clipped-SSTM at stage  $t = 1,\ldots ,N$  stay in the ball  $B_{2R_{t - 1}}(x^{*})$ .
The obtained complexity bound (see the proof in Appendix C.2) is the first optimal (up to logarithms) high-probability complexity bound under Assumption 2 for the smooth strongly convex problems. Indeed, the first term cannot be improved in view of the deterministic lower bound [27], and the second term is optimal [41].
4 Setting with Adversarial Noise
Often, black-box access of  $f(x,\xi)$  are affected by some deterministic noise  $\delta (x)$ . Thus, now we suppose that a zero-order oracle instead of objective values  $f(x,\xi)$  returns its noisy approximation
$$
f _ {\delta} (x, \xi) \stackrel {\text {d e f}} {=} f (x, \xi) + \delta (x). \tag {10}
$$
This noise  $\delta (x)$  can be interpreted, e.g., as a computer discretization when calculating  $f(x,\xi)$ . For our analysis, we need this noise to be uniformly bounded. Recently, noisy «black-box» optimization with bounded noise has been actively studied [11, 25]. The authors of [11] consider deterministic adversarial noise, while in [25] stochastic adversarial noise was explored.
Assumption 3 (Boundedness of noise) There exists a constant  $\Delta >0$  such that  $|\delta (x)|\leq \Delta$  for all  $x\in Q$
This is a standard assumption often used in the literature (e.g., [17]). Moreover, in some applications [4] the bigger the noise the cheaper the zero-order oracle. Thus, it is important to understand the maximum allowable level of adversarial noise at which the convergence of the gradient-free algorithm is unaffected.
4.1 Non-smooth setting
In noisy setup, gradient estimate from (4) is replaced by
$$
g (x, \xi , \mathbf {e}) = \frac {d}{2 \tau} \left(f _ {\delta} (x + \tau \mathbf {e}, \xi) - f _ {\delta} (x - \tau \mathbf {e}, \xi)\right) \mathbf {e}. \tag {11}
$$
Then (6) from Lemma 3 will have an extra factor driven by noise (see Lemma 2.3 from [22])
$$
\mathbb {E} _ {\{\xi \}, \{\mathbf {e} \}} \left[ \left\| g ^ {B} (x, \{\xi \}, \{\mathbf {e} \}) - \mathbb {E} _ {\{\xi \}, \{\mathbf {e} \}} [ g ^ {B} (x, \{\xi \}, \{\mathbf {e} \}) ] \right\| _ {2} ^ {\alpha} \right] \leq \frac {2}{B ^ {\alpha - 1}} \left(\frac {\sqrt {d} M _ {2}}{2 ^ {\frac {1}{4}}} + \frac {d \Delta}{\tau}\right) ^ {\alpha}.
$$
To guarantee the same convergence of the algorithm as in Theorem 1 (see (8)) for the adversarial deterministic noise case, the variance term must dominate the noise term, i.e.  $d\Delta \tau^{-1} \lesssim \sqrt{d} M_2$ . Note that if the term with noise dominates the term with variance, it does not mean that the gradient-free algorithm will not converge. In contrast, algorithm will still converge, only slower (oracle complexity will be  $\sim \varepsilon^{-2}$  times higher). Thus, if we were considering the zero-order oracle concept with adversarial stochastic noise, it would be enough to express the noise level  $\Delta$ , and substitute the value of the smoothing parameter  $\tau$  to obtain the maximum allowable noise level. However, since we are considering the concept of adversarial noise in a deterministic setting, following previous work [11, 1] we can say that adversarial noise accumulates not only in the variance, but also in the bias:
$$
\mathbb {E} _ {\xi , \mathbf {e}} \langle [ g (x, \xi , \mathbf {e}) ] - \nabla \hat {f} _ {\tau} (x), r \rangle \lesssim \sqrt {d} \Delta \| r \| _ {2} \tau^ {- 1}, \quad \text {f o r a l l} r \in \mathbb {R} ^ {d}.
$$
This bias can be controlled by the noise level  $\Delta$ , i.e., in order to achieve the  $\varepsilon$ -accuracy algorithm considered in this paper, the noise condition must be satisfied:
$$
\Delta \lesssim \frac {\tau \varepsilon}{R \sqrt {d}}. \tag {12}
$$
As we can see, we have a more restrictive condition on the noise level in the bias (12) than in the variance  $(\Delta \lesssim \gamma^{M_2} / \sqrt{d})$ . Thus, the maximum allowable level of adversarial deterministic noise, which guarantees the same convergence of ZO-clipped-SSTM, as in Theorem 1 (see (8)) is as follows
$$
\Delta \lesssim \frac {\varepsilon^ {2}}{R M _ {2} \sqrt {d}},
$$
where  $\tau = \varepsilon /2M_2$  the smoothing parameter from Lemma 2.
Remark 1 ( $\mu$ -strongly convex case) Let us assume that  $f(x)$  is also  $\mu$ -strongly convex (see Assumption 1). Then, following the works [11, 22], we can conclude that the R-ZO-clipped-SSTM has the same oracle and iteration complexity as in Theorem 2 at the following maximum allowable level of adversarial noise:  $\Delta \lesssim \mu^{1/2} \varepsilon^{3/2} / \sqrt{d} M_2$ .
4.2 Smooth setting
Now we examine the maximum allowable level of noise at which we can solve optimization problem (1) with  $\varepsilon$ -precision under the following additional assumption
Assumption 4 (Smoothness) The function  $f$  is  $L$ -smooth, i.e., it is differentiable on  $Q$  and for all  $x, y \in Q$  with  $L > 0$ :
$$
\| \nabla f (y) - \nabla f (x) \| _ {2} \leq L \| y - x \| _ {2}.
$$
If Assumption 4 holds, then Lemma 2 can be rewritten as
$$
\sup _ {x \in Q} | \hat {f} _ {\tau} (x) - f (x) | \leq \frac {L \tau^ {2}}{2}.
$$
Thus, we can now present the convergence results of the gradient-free algorithm in the smooth setting. Specifically, if the Assumptions 2-4 are satisfied, then  $\widetilde{ZO}$ -clipped-SSTM converges to  $\varepsilon$ -accuracy after  $K = \widetilde{\mathcal{O}}\left(\sqrt{LR^2\varepsilon^{-1}}\right)$  iterations with probability at least  $1 - \beta$ . It is easy to see that the iteration complexity improves in the smooth setting (since the Lipschitz gradient constant  $L$  already exists,
i.e., no smoothing is needed), but oracle complexity remained unchanged (since we are still using the gradient approximation via  $l_{2}$  randomization (11) instead of the true gradient  $\nabla f(x)$ ), consistent with the already optimal estimate on oracle calls:  $\widetilde{\mathcal{O}}\left(\left(\sqrt{d} M_2R\varepsilon^{-1}\right)^{\frac{\alpha}{\alpha - 1}}\right)$ . And to obtain the maximum allowable level of adversarial noise  $\Delta$  in the smooth setting, which guarantees such convergence, it is sufficient to substitute the smoothing parameter  $\tau = \sqrt{\varepsilon / L}$  in the inequality (12):
$$
\Delta \lesssim \frac {\varepsilon^ {3 / 2}}{R \sqrt {d L}}.
$$
Thus, we can conclude that smooth setting improves iteration complexity and the maximum allowable noise level for the gradient-free algorithm, but the oracle complexity remains unchanged.
Remark 2 ( $\mu$ -strongly convex case) Suppose that  $f(x)$  is also  $\mu$ -strongly convex (see Assumption 1). Then we can conclude that R-ZO-clipped-SSTM has the oracle and iteration complexity just mentioned above at the following maximum allowable level of adversarial noise:  $\Delta \lesssim \mu^{1/2} \varepsilon / \sqrt{dL}$ .
Remark 3 (Upper bounds optimality) The upper bounds on maximum allowable level of adversarial noise obtained in this section in both non-smooth and smooth settings are optimal in terms of dependencies on  $\varepsilon$  and  $d$  according to the works [32, 34].
Remark 4 (Better oracle complexity) In the aforementioned approach in the case when  $f(x, \xi)$  has Lipschitz gradient in  $x$  (for all  $\xi$ ) one can improve oracle complexity from  $\widetilde{\mathcal{O}}\left(\left(\sqrt{d}M_2R\varepsilon^{-1}\right)^{\frac{\alpha}{\alpha - 1}}\right)$  to  $\widetilde{\mathcal{O}}\left(d(M_2R\varepsilon^{-1})^{\frac{\alpha}{\alpha - 1}}\right)$ . This can be done by using component-wise finite-difference stochastic gradient approximation [15]. Iteration complexity remains  $\widetilde{\mathcal{O}}\left(\sqrt{LR^2\varepsilon^{-1}}\right)$ . The same can be done for  $\mu$ -strongly convex case: from  $\widetilde{\mathcal{O}}\left((dM_2^2(\mu\varepsilon)^{-1})^{\frac{\alpha}{2(\alpha - 1)}}\right)$  to  $\widetilde{\mathcal{O}}\left(d(M_2^2(\mu\varepsilon)^{-1})^{\frac{\alpha}{2(\alpha - 1)}}\right)$ .
5 Details of the proof
The proof is built upon a combination of two techniques. The first one is the Smoothing technique from [17] that is used to develop a gradient-free method for convex non-smooth problems based on full-gradient methods. The second technique is the Accelerated Clipping technique that has been recently developed for smooth problems with the noise having infinite variance and first-order oracle [35]. The authors of [35] propose clipped-SSTM method that we develop in our paper. We modify clipped-SSTM by introducing batching into it. Note that due to the infinite variance, such a modification is interesting in itself. Then we run batched clipped-SSTM with gradient estimations of function  $f$  obtained via Smoothing technique and two-point zeroth-order oracle. To do this, we need to estimate the variance of the clipped version of the batched gradient estimation.
In more detail, we replace the initial problem of minimizing  $f$  by minimizing its smoothed approximation  $\hat{f}_{\tau}$ , see Lemma 2 In order to use estimated gradient of  $\hat{f}_{\tau}$  defined in (4) or (5), we make sure that it has bounded  $\alpha$ -th moment. For these purposes we proof Lemma 3. First part shows boundness of unbatched estimated gradient  $g$  defined in (4). It follows from measure concentration phenomenon for the Euclidean sphere for  $\frac{M_2}{\tau}$ -Lipschitz function  $f(x + \mathbf{e}\tau)$ . According to this phenomenon probability of the functions deviation from its math expectation becomes exponentially small and  $\alpha$ -th moment of this deviation becomes bounded. Furthermore, the second part of Lemma 3 shows that batching helps to bound  $\alpha$ -th moment of batched gradient  $g^B$  defined in (5) even more. Also batching allows parallel calculations reducing number of necessary iteration with the same number of oracle calls. All this is possible thanks to the result, interesting in itself, presented in the following Lemma.
Lemma 4 Let  $X_{1},\ldots ,X_{B}$  be  $d$  -dimensional martingale difference sequence (i.e.  $\mathbb{E}[X_i|X_{i - 1},\dots ,X_1] = 0$  for  $1 <   i\leq B)$  satisfying for  $1\leq \alpha \leq 2$
$$
\mathbb {E} \left[ \| X _ {i} \| ^ {\alpha} \mid X _ {i - 1}, \dots , X _ {1} \right] \leq \sigma^ {\alpha}.
$$
$$
\mathbb {E} \left[ \left\| \frac {1}{B} \sum_ {i = 1} ^ {B} X _ {i} \right\| _ {2} ^ {\alpha} \right] \leq \frac {2 \sigma^ {\alpha}}{B ^ {\alpha - 1}}.
$$
Next, we use the clipped-SSTM for function  $\hat{f}_{\tau}$  with heavy-tailed gradient estimates. This algorithm was initially proposed for smooth functions in the work [35]. The scheme for proving convergence with high probability is also taken from it, the only difference is additional randomization caused by smoothing scheme.
6 Numerical experiments
We tested ZO-clipped-SSTM on the following problem
$$
\min  _ {x \in \mathbb {R} ^ {d}} \| A x - b \| _ {2} + \langle \xi , x \rangle ,
$$
where  $\xi$  is a random vector with independent components sampled from the symmetric Levy  $\alpha$ -stable distribution with  $\alpha = 3/2$ ,  $A \in \mathbb{R}^{m \times d}$ ,  $b \in \mathbb{R}^m$  (we used  $d = 16$  and  $m = 500$ ). For this problem, Assumption 1 holds with  $\mu = 0$  and Assumption 2 holds with  $\alpha = 3/2$  and  $M_2(\xi) = \| A \|_2 + \| \xi \|_2$ .
We compared ZO-clipped-SSTM, proposed in this paper, with ZOSGD and ZO-SSTM. For these algorithms, we gridsearched batchsize  $B:5,10,50,100,500$  and stepsize  $\gamma :1e - 3,1e - 4,1e - 5,1e - 6$ . The best convergence was achieved with the following parameters:
ZO-clipped-SSTM:  $\gamma = 1e - 3,$ $B = 10,\lambda = 0.01$
- ZO-SSTM:  $\gamma = 1e - 5, B = 500$
- ZO-SGD:  $\gamma = 1e - 4, B = 100$ ,  $\omega = 0.9$ , where  $\omega$  is a heavy-ball momentum parameter.
Figure 1: Convergence of ZO-clipped-SSTM, ZO-SGD and ZO-clipped-SSTM in terms of a gap function w.r.t. the number of consumed samples.
The code is written on Pythone and is publicly available at https://github.com/ClippedStochasticMethods/Z0-clipped-SSTM. Figure 1 presents the comparison of convergences averaged over 15 launches with different noise. In contrast to ZO-clipped-SSTM, the last two methods are unclipped and therefore failed to converge under haivy-tailed noise.
7 Conclusion and future directions
In this paper, we propose a first gradient-free algorithm ZO-clipped-SSTM and to solve problem (1) under heavy-tailed noise assumption. By using the restart technique we extend this algorithm for strongly convex objective, we refer to this algorithm as R-ZO-clipped-SSTM. The proposed algorithms are optimal with respect to oracle complexity (in terms of the dependence on  $\varepsilon$ ), iteration complexity and the maximal level of noise (possibly adversarial). The algorithms can be adapted to composite and distributed minimization problems, saddle-point problems, and variational inequalities. Despite the fact that the algorithms utilize the two-point feedback, they can be modified to the one-point feedback. We leave it for future work.
Moreover, we provide theoretical basis to demonstrate benefits of batching technique in case of heavy-tailed stochastic noise and apply it to methods from this paper. Thanks to this basis, it is possible to use batching in other methods with heavy-tiled noise, e.g. first-order methods presented in [35, 29].
8 Acknowledgment
The work of Alexander Gasnikov, Aleksandr Lobanov, and Darina Dvinskikh was supported by a grant for research centers in the field of artificial intelligence, provided by the Analytical Center for the Government of the Russian Federation in accordance with the subsidy agreement (agreement identifier 000000D730321P5Q0002) and the agreement with the Ivannikov Institute for System Programming of dated November 2, 2021 No. 70-2021-00142.
[34] Andrej Risteski and Yuanzhi Li. Algorithms and matching lower bounds for approximately-convex optimization. Advances in Neural Information Processing Systems, 29, 2016.
[35] Abdurakhmon Sadiev, Marina Danilova, Eduard Gorbunov, Samuel Horváth, Gauthier Gidel, Pavel Dvurechensky, Alexander Gasnikov, and Peter Richtárik. High-probability bounds for stochastic optimization and variational inequalities: the case of unbounded variance. arXiv preprint arXiv:2302.00999, 2023.
[36] Ohad Shamir. An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. The Journal of Machine Learning Research, 18(1):1703-1713, 2017.
[37] Umut Simsekli, Levent Sagun, and Mert Gurbuzbalaban. A tail-index analysis of stochastic gradient noise in deep neural networks. In International Conference on Machine Learning, pages 5827-5837. PMLR, 2019.
[38] James C Spall. Introduction to stochastic search and optimization: estimation, simulation, and control. John Wiley & Sons, 2005.
[39] Nuri Mert Vural, Lu Yu, Krishna Balasubramanian, Stanislav Volgushev, and Murat A Erdogdu. Mirror descent strikes again: Optimal stochastic convex optimization under infinite noise variance. In Conference on Learning Theory, pages 65-102. PMLR, 2022.
[40] Hongjian Wang, Mert Gurbuzbalaban, Lingjiong Zhu, Umut Simsekli, and Murat A Erdogdu. Convergence rates of stochastic gradient descent under infinite noise variance. Advances in Neural Information Processing Systems, 34:18866-18877, 2021.
[41] Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit, Seungyeon Kim, Sashank J Reddi, Sanjiv Kumar, and Suvrit Sra. Why are adaptive methods good for attention models? Advances in Neural Information Processing Systems, 33, 2020.
[42] Jiujia Zhang and Ashok Cutkosky. Parameter-free regret in high probability with heavy tails. Advances in Neural Information Processing Systems, 35:8000-8012, 2022.
A Batching with unbounded variance
To prove batching Lemma 4 we generalize Lemma 4.2 from [7] not only for i.i.d. random variables with zero mean but for martingale difference sequence.
Lemma 5 Let  $g(x) = \text{sign}(x)|x|^{\alpha - 1} = \nabla \left(\frac{|x|^{\alpha}}{\alpha}\right)$  for  $1 < \alpha \leq 2$ . Then for any  $h \geq 0$
$$
\max _ {h} g (x + h) - g (x) = 2 ^ {2 - \alpha} h ^ {\alpha - 1} = 2 ^ {2 - \alpha} g (h).
$$
Proof. Consider  $l(x) = g(x + h) - g(x)$ . We see that  $l$  is differentiable everywhere except  $x = 0$  and  $x = -h$ . As long as  $x \neq 0, -h$ , we have
$$
l ^ {\prime} (x) = g ^ {\prime} (x + h) - g ^ {\prime} (x) = (\alpha - 1) \left(| x + h | ^ {\alpha - 2} - | x | ^ {\alpha - 2}\right).
$$
Since, we have  $\alpha > 1$ ,  $x = -\frac{h}{2}$  is local maxima for  $l(x)$ . Furthermore, note that  $l'(x) \geq 0$  for  $x \in (-\infty, -\frac{h}{2}) \setminus \{-h\}$  and  $l'(x) \leq 0$  for  $x \in \left(-\frac{h}{2}, \infty\right) \setminus \{0\}$ . Therefore,  $\frac{-h}{2}$  is global maxima.
Lemma 6 Let  $x_{1}, \ldots, x_{B}$  be one-dimensional martingale difference sequence, i.e.  $\mathbb{E}[x_i | x_{i-1}, \ldots, x_1] = 0$  for  $1 < i \leq B$ , satisfying for  $1 \leq \alpha \leq 2$
$$
\mathbb {E} [ | x _ {i} | ^ {\alpha} | x _ {i - 1}, \dots , x _ {1} ] \leq \sigma^ {\alpha}.
$$
We have:
$$
\mathbb {E} \left[ \left| \frac {1}{B} \sum_ {i = 1} ^ {B} x _ {i} \right| ^ {\alpha} \right] \leq \frac {2 \sigma^ {\alpha}}{B ^ {\alpha - 1}}.
$$
Proof. Everywhere below we will use the following notations.
$$
\mathbb {E} _ {<   i} [ \cdot ] \stackrel {\mathrm {d e f}} {=} \mathbb {E} _ {x _ {i - 1}, \ldots , x _ {1}} [ \cdot ], \qquad \mathbb {E} _ {| <   i} [ \cdot ] \stackrel {\mathrm {d e f}} {=} \mathbb {E} [ \cdot | x _ {i - 1}, \ldots , x _ {1} ].
$$
For  $\alpha = 1$  proof follows from triangle inequality for  $|\cdot|$ . When  $\alpha > 1$ , we start by defining
$$
S _ {i} = \sum_ {j = 1} ^ {i} x _ {j}, \qquad S _ {0} = 0, \qquad f (x) = | x | ^ {\alpha}.
$$
Then we can calculate desired expectation as
$$
\begin{array}{l} \mathbb {E} [ f (S _ {B}) ] = \mathbb {E} \left[ \sum_ {i = 1} ^ {B} f (S _ {i}) - f (S _ {i - 1}) \right] = \sum_ {i = 1} ^ {B} \mathbb {E} [ f (S _ {i}) - f (S _ {i - 1}) ] \\ = \sum_ {i = 1} ^ {B} \mathbb {E} \left[ \int_ {S _ {i - 1}} ^ {S _ {i}} f ^ {\prime} (x) d x \right] \\ = \sum_ {i = 1} ^ {B} \mathbb {E} \left[ x _ {i} f ^ {\prime} \left(S _ {i - 1}\right) + \int_ {S _ {i - 1}} ^ {S _ {i}} f ^ {\prime} (x) - f ^ {\prime} \left(S _ {i - 1}\right) d x \right]. \tag {13} \\ \end{array}
$$
While  $\{x_i\}$  is martingale difference sequence then  $\mathbb{E}[x_if'(S_{i - 1})] = \mathbb{E}_{< i}[\mathbb{E}_{| < i}[x_if'(S_{i - 1})]] = 0$  From (13) and Lemma 5  $(g(x) = f^{\prime}(x) / \alpha)$  we obtain
$$
\begin{array}{l} \mathbb {E} [ f (S _ {B}) ] = \sum_ {i = 1} ^ {B} \mathbb {E} \left[ x _ {i} f ^ {\prime} (S _ {i - 1}) + \int_ {S _ {i - 1}} ^ {S _ {i}} f ^ {\prime} (x) - f ^ {\prime} (S _ {i - 1}) d x \right] \leq 2 ^ {1 - \alpha} \sum_ {i = 1} ^ {B} \mathbb {E} \left[ \int_ {0} ^ {| x _ {i} |} f ^ {\prime} (t / 2) d t \right] \\ = \sum_ {i = 1} ^ {B} \mathbb {E} \left[ \int_ {0} ^ {| x _ {i} | / 2} 2 f ^ {\prime} (s) d s \right] = 2 ^ {2 - \alpha} \sum_ {i = 1} ^ {B} \mathbb {E} [ f (| x _ {i} | / 2) ] \\ = 2 ^ {2 - \alpha} \sum_ {i = 1} ^ {B} \mathbb {E} _ {<   i} \left[ \mathbb {E} _ {| <   i} [ f (| x _ {i} | / 2) ] \right] \leq 2 B \sigma^ {\alpha}. \tag {14} \\ \end{array}
$$
Now we are ready to prove batching lemma for random variables with infinite variance.
Lemma 7 Let  $X_{1},\ldots ,X_{B}$  be  $d$  -dimensional martingale difference sequence, i.e.,  $\mathbb{E}[X_i|X_{i - 1},\dots ,X_1] = 0$  for  $1 <   i\leq B$  satisfying for  $1\leq \alpha \leq 2$
$$
\mathbb {E} \left[ \| X _ {i} \| _ {2} ^ {\alpha} | X _ {i - 1}, \dots , X _ {1} \right] \leq \sigma^ {\alpha}.
$$
Let  $g \sim \mathcal{N}(0, I)$  and  $y_i \stackrel{\text{def}}{=} X_i^\top g$ . Firstly, we prove that  $\mathbb{E}[y_i^\alpha] \leq \mathbb{E}[\|X_i\|^{\alpha}]$ . Indeed, using conditional expectation we get
$$
\begin{array}{l} \mathbb {E} \left[ \left| y _ {i} \right| ^ {\alpha} \right] = \mathbb {E} \left[ \left(\left| X _ {i} ^ {\top} g \right|\right) ^ {\alpha} \right] = \mathbb {E} _ {X _ {i}} \left[ \mathbb {E} _ {g | X _ {i}} \left[ \left(\left| X _ {i} ^ {\top} g\right)\right) ^ {\alpha} \right] \right] \\ = \mathbb {E} _ {X _ {i}} \left[ \mathbb {E} _ {g | X _ {i}} \left[ \left(\left(| X _ {i} ^ {\top} g |\right) ^ {2}\right) ^ {\alpha / 2} \right] \right] \\ \end{array}
$$
$$
\begin{array}{l} \begin{array}{l} \text {J e s e n i n q} \\ \leq \end{array} \quad \mathbb {E} _ {X _ {i}} \left[ \left(\mathbb {E} _ {g | X _ {i}} \left[ (X _ {i} ^ {\top} g) ^ {2} \right]\right) ^ {\alpha / 2} \right] \\ = \mathbb {E} _ {X _ {i}} \left[ \left(\| X _ {i} \| _ {2} ^ {2}\right) ^ {\alpha / 2} \right] = \mathbb {E} [ \| X _ {i} \| _ {2} ^ {\alpha} ]. \tag {15} \\ \end{array}
$$
Next, considering  $X_{i}^{\top}g\sim \mathcal{N}(0,\| X\|^{2})$  and, thus,  $\mathbb{E}_g|X_i^\top g| = \| X\|$ , we bound desired expectation as
$$
\mathbb {E} _ {X} \left[ \left\| \sum_ {i = 1} ^ {B} X _ {i} \right\| _ {2} ^ {\alpha} \right] \quad = \quad \mathbb {E} _ {X} \left[ \left(\mathbb {E} _ {g} \left| \sum_ {i = 1} ^ {B} X _ {i} ^ {\top} g \right|\right) ^ {\alpha} \right] \tag {16}
$$
$$
\leq^ {\text {J e s s e n ' s i n q}} \mathbb {E} _ {X, g} \left[ \left| \sum_ {i = 1} ^ {B} X _ {i} ^ {\top} g \right| ^ {\alpha} \right] = \mathbb {E} _ {X, g} \left[ \left| \sum_ {i = 1} ^ {B} y _ {i} \right| ^ {\alpha} \right]. \tag {17}
$$
Finally, we apply Lemma 6 for  $y_{i}$  sequence with bounded  $\alpha$ -th moment from (15) and get
$$
\mathbb {E} _ {X} \left\| \left| \sum_ {i = 1} ^ {B} X _ {i} \right| \right\| _ {2} ^ {\alpha} \biggr ] \leq \mathbb {E} _ {X, g} \left[ \left| \sum_ {i = 1} ^ {B} y _ {i} \right| ^ {\alpha} \right] \leq 2 \sigma^ {\alpha} B.
$$
B Smoothing scheme
Lemma 8 Assumption 2 implies that  $f(x)$  is  $M_2$  Lipschitz on  $Q$ .
Proof. For all  $x_1, x_2 \in Q$
$$
\begin{array}{l} | f (x _ {1}) - f (x _ {2}) | \leq | \mathbb {E} _ {\xi} [ f (x _ {1}, \xi) - f (x _ {2}, \xi) ] | \leq \mathbb {E} _ {\xi} [ | f (x _ {1}, \xi) - f (x _ {2}, \xi) | ] (18) \\ \leq \mathbb {E} _ {\xi} \left[ M _ {2} (\xi) \right] \| x _ {1} - x _ {2} \| _ {2} \leq M _ {2} \| x _ {1} - x _ {2} \| _ {2}. (19) \\ \end{array}
$$