SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The following lemma gives some facts about the measure concentration on the Euclidean unit sphere for next proof.
Lemma 9 Let  $f(x)$  be  $M_2$  Lipschitz continuous function w.r.t  $\|\cdot\|$ . If  $e$  is random and uniformly distributed on the Euclidean sphere and  $\alpha \in (1,2]$ , then
$$
\mathbb {E} _ {\mathbf {e}} \left[ (f (\mathbf {e}) - \mathbb {E} _ {\mathbf {e}} [ f (\mathbf {e}) ]) ^ {2 \alpha} \right] \leq \left(\frac {b M _ {2} ^ {2}}{d}\right) ^ {\alpha}, b = \frac {1}{\sqrt {2}}.
$$
Proof. A standard result of the measure concentration on the Euclidean unit sphere implies that  $\forall t > 0$
$$
P r \left(| f (\mathbf {e}) - \mathbb {E} [ f (\mathbf {e}) ] | > t\right) \leq 2 \exp \left(- b ^ {\prime} d t ^ {2} / M _ {2} ^ {2}\right), \quad b ^ {\prime} = 2 \tag {20}
$$
(see the proof of Proposition 2.10 and Corollary 2.6 in [23]). Therefore,
$$
\begin{array}{l} \mathbb {E} _ {\mathbf {e}} \left[ (f (\mathbf {e}) - \mathbb {E} _ {\mathbf {e}} [ f (\mathbf {e}) ]) ^ {2 \alpha} \right] = \int_ {t = 0} ^ {\infty} P r \left(| f (\mathbf {e}) - \mathbb {E} [ f (\mathbf {e}) ] | ^ {2 \alpha} > t\right) d t \\ = \int_ {t = 0} ^ {\infty} P r \left(| f (\mathbf {e}) - \mathbb {E} [ f (\mathbf {e}) ] | > t ^ {1 / 2 \alpha}\right) d t \\ \leq \int_ {t = 0} ^ {\infty} 2 \exp \left(- b ^ {\prime} d \cdot t ^ {1 / \alpha} / M _ {2} ^ {2}\right) d t \leq \left(\frac {b M _ {2} ^ {2}}{d}\right) ^ {\alpha}. \tag {21} \\ \end{array}
$$
Finally, below we prove Lemma 3 which states that batched gradient estimation from (5) has bounded  $\alpha$ -th moment.
Proof of Lemma 3.
1. We will prove equality immediately for  $g^B$ . Firstly, we notice that distribution of  $\mathbf{e}$  is symmetric and by definition (5) we get
$$
\begin{array}{l} \mathbb {E} _ {\xi , \mathbf {e}} [ g ^ {B} (x, \xi , \mathbf {e}) ] = \left(\frac {d}{2 B \tau}\right) \sum_ {i = 1} ^ {B} \mathbb {E} _ {\xi_ {i}, \mathbf {e} _ {i}} \left[ f (x + \tau \mathbf {e} _ {i}, \xi_ {i}) \mathbf {e} _ {i} - f (x - \tau \mathbf {e} _ {i}, \xi_ {i}) \mathbf {e} _ {\mathrm {i}} \right] \\ = \frac {d}{B \tau} \sum_ {i = 1} ^ {B} \mathbb {E} _ {\mathbf {e} _ {i}} [ \mathbb {E} _ {\xi_ {i}} [ f (x + \tau \mathbf {e} _ {i}, \xi_ {i}) ] \mathbf {e} _ {i} ] \\ = \frac {d}{B \tau} \sum_ {i = 1} ^ {B} \mathbb {E} _ {\mathbf {e} _ {i}} [ f (x + \tau \mathbf {e} _ {i}) \mathbf {e} _ {i} ]. \tag {22} \\ \end{array}
$$
Using  $\nabla \hat{f}_{\tau}(x) = \frac{d}{\tau}\mathbb{E}_{\mathbf{e}}[f(x + \tau \mathbf{e})\mathbf{e}]$  from Lemma 2 we obtain necessary result.
2. By definition (4) of estimated gradient  $g$  we bound  $\alpha$ -th moment as
$$
\begin{array}{l} \mathbb {E} _ {\xi , \mathbf {e}} [ \| g (x, \xi , \mathbf {e}) \| ^ {\alpha} ] = \mathbb {E} _ {\xi , \mathbf {e}} \left[ \left| \frac {d}{2 \tau} (f (x + \tau \mathbf {e}, \xi) - f (x - \tau \mathbf {e}, \xi)) \mathbf {e} \right\rVert_ {2} ^ {\alpha} \right] \\ = \left(\frac {d}{2 \tau}\right) ^ {\alpha} \mathbb {E} _ {\xi , \mathbf {e}} \left[ \| \mathbf {e} \| _ {2} ^ {\alpha} | f (x + \tau \mathbf {e}, \xi) - f (x - \tau \mathbf {e}, \xi) | ^ {\alpha} \right]. \tag {23} \\ \end{array}
$$
Considering  $\| \mathbf{e}\| _2 = 1$  we can omit it. Next we add  $\pm \delta (\xi)$  in (23) for all  $\delta (\xi)$  and get
$$
\begin{array}{l} \mathbb {E} _ {\xi , \mathbf {e}} \left[ | f (x + \tau \mathbf {e}, \xi) - f (x - \tau \mathbf {e}, \xi) | ^ {\alpha} \right] \\ = \mathbb {E} _ {\xi , \mathbf {e}} [ | (f (x + \tau \mathbf {e}, \xi) - \delta) - (f (x - \tau \mathbf {e}, \xi) - \delta) | ^ {\alpha} ]. \\ \end{array}
$$
Using Jensen's inequality for  $|\cdot|^{\alpha}$  we bound
$$
\begin{array}{l} \mathbb {E} _ {\xi , \mathbf {e}} [ | (f (x + \tau \mathbf {e}, \xi) - \delta) - (f (x - \tau \mathbf {e}, \xi) - \delta) | ^ {\alpha} ] \\ \leq 2 ^ {\alpha - 1} \mathbb {E} _ {\xi , \mathbf {e}} \left[ | f (x + \tau \mathbf {e}, \xi) - \delta | ^ {\alpha} \right] + 2 ^ {\alpha - 1} \mathbb {E} _ {\xi , \mathbf {e}} \left[ | f (x - \tau \mathbf {e}, \xi) - \delta | ^ {\alpha} \right]. \\ \end{array}
$$
We note that distribution of  $\mathbf{e}$  is symmetric and add two terms together
$$
\begin{array}{l} 2 ^ {\alpha - 1} \mathbb {E} _ {\xi , \mathbf {e}} \left[ | f (x + \tau \mathbf {e}, \xi) - \delta | ^ {\alpha} \right] + 2 ^ {\alpha - 1} \mathbb {E} _ {\xi , \mathbf {e}} \left[ | f (x - \tau \mathbf {e}, \xi) - \delta | ^ {\alpha} \right] \\ \leq 2 ^ {\alpha} \mathbb {E} _ {\xi , \mathbf {e}} \left[ | f (x + \tau \mathbf {e}, \xi) - \delta | ^ {\alpha} \right]. \\ \end{array}
$$
Let  $\delta (\xi) = \mathbb{E}_{\mathbf{e}}[f(x + \tau \mathbf{e},\xi)]$ , then because of Cauchy-Schwartz inequality and conditional expectation properties we obtain
$$
\begin{array}{l} 2 ^ {\alpha} \mathbb {E} _ {\xi , \mathbf {e}} [ | f (x + \tau \mathbf {e}, \xi) - \delta | ^ {\alpha} ] = 2 ^ {\alpha} \mathbb {E} _ {\xi} [ \mathbb {E} _ {\mathbf {e}} [ | f (x + \tau \mathbf {e}, \xi) - \delta | ^ {\alpha} ] ] \\ \leq 2 ^ {\alpha} \mathbb {E} _ {\xi} \left[ \sqrt {\mathbb {E} _ {\mathbf {e}} \left[ | f (x + \tau \mathbf {e} , \xi) - \mathbb {E} _ {\mathbf {e}} [ f (x + \tau \mathbf {e} , \xi) ] | ^ {2 \alpha} \right]} \right]. \\ \end{array}
$$
Next, we use Lemma 9 for  $f(x + \tau \mathbf{e},\xi)$  with fixed  $\xi$  and Lipschitz constant  $M_2(\xi)\tau$
$$
\begin{array}{l} 2 ^ {\alpha} \mathbb {E} _ {\xi} \left[ \sqrt {\mathbb {E} _ {\mathbf {e}} \left[ | f (x + \tau \mathbf {e} , \xi) - \mathbb {E} _ {\mathbf {e}} [ f (x + \tau \mathbf {e} , \xi) ] | ^ {2 \alpha} \right]} \right] \leq 2 ^ {\alpha} \mathbb {E} _ {\xi} \left[ \sqrt {\left(\frac {2 ^ {- 1 / 2} \tau^ {2} M _ {2} ^ {2} (\xi)}{d}\right) ^ {\alpha}} \right] \\ = 2 ^ {\alpha} \left(\frac {\tau^ {2} 2 ^ {- 1 / 2}}{d}\right) ^ {\alpha / 2} \mathbb {E} _ {\xi} \left[ M _ {2} ^ {\alpha} (\xi) \right] \leq 2 ^ {\alpha} \left(\sqrt {\frac {2 ^ {- 1 / 2}}{d}} M _ {2} \tau\right) ^ {\alpha}. \\ \end{array}
$$
Finally, we get desired bound of estimated gradient
$$
\mathbb {E} _ {\xi , \mathbf {e}} [ \| g (x, \xi , \mathbf {e}) \| _ {2} ^ {\alpha} ] = \left(\frac {\sqrt {d}}{2 ^ {1 / 4}} M _ {2}\right) ^ {\alpha}. \tag {24}
$$
Now we apply Jensen inequality to
$$
\mathbb {E} _ {\xi , \mathbf {e}} [ \| g (x, \xi , \mathbf {e}) - \mathbb {E} _ {\xi , \mathbf {e}} [ g (x, \xi , \mathbf {e}) ] \| _ {2} ^ {\alpha} ] \leq 2 ^ {\alpha - 1} \left(\mathbb {E} _ {\xi , \mathbf {e}} [ \| g (x, \xi , \mathbf {e}) \| _ {2} ] + \mathbb {E} _ {\xi , \mathbf {e}} [ \| g (x, \xi , \mathbf {e}) \| _ {2} ]\right) \tag {25}
$$
And get necessary result.
For batched gradient  $g^{B}$  we use Batching Lemma 7 and estimation (25).
C Missing Proofs for ZO-clipped-SSTM and R-ZO-clipped-SSTM
In this section, we provide the complete formulation of the main results for clipped-SSTM and R-clipped-SSTM and the missing proofs.
Minimization on the subset  $Q$  In order to work with subsets of  $Q \subseteq \mathbb{R}^d$  we must assume one more condition on  $\hat{f}_{\tau}(x)$ .
Assumption 5 We assume that there exist some convex set  $Q \subseteq \mathbb{R}^d$ , constants  $\tau, L > 0$  such that for all  $x, y \in Q$
$$
\left\| \nabla \hat {f} _ {\tau} (x) \right\| _ {2} ^ {2} \leq 2 L \left(\hat {f} _ {\tau} (x) - \hat {f} _ {\tau} ^ {*}\right), \tag {26}
$$
where  $\hat{f}_{\tau}^{*} = \inf_{x\in Q}\hat{f}_{\tau}(x) > - \infty$
When  $Q = \mathbb{R}^d$  (26) follows from Lemma 2 as well. But in general case this is not true. In work [35] in Section "Useful facts" authors show that, in the worst case, to have (26) on a set  $Q$  one needs to assume smoothness on a slightly larger set.
Thus, in the full version of the theorems in which we can require much smaller  $Q$ , we will also require satisfying all three Assumptions 1, 2, 5.
C.1 Convex Functions
We start with the following lemma, which is a special case of Lemma 6 from [19]. This result can be seen the "optimization" part of the analysis of clipped-SSTM: the proof follows the same steps as the analysis of deterministic Similar Triangles Method [16], [12] and separates stochasticity from the deterministic part of the method.
Pay attention that in full version of Theorem 1 we require Assumptions 1, 2 to hold only on  $Q = B_{3R}(x^{*})$ , however we need to require one more smoothness Assumption 5 as well.
Theorem 3 (Full version of Theorem 1) Let Assumptions 1,2, 5 with  $\mu = 0$  hold on  $Q = B_{3R}(x^{*})$ , where  $R \geq \| x^0 - x^*\|$ , and
$$
a \geq \max  \left\{4 8 6 0 0 \ln^ {2} \frac {4 K}{\beta}, \frac {1 8 0 0 \sigma (K + 1) K ^ {\frac {1}{\alpha}} \ln^ {\frac {\alpha - 1}{\alpha}} \frac {4 K}{\beta}}{B ^ {\frac {\alpha - 1}{\alpha}} L R} \right\}, \tag {27}
$$
$$
\lambda_ {k} = \frac {R}{3 0 \alpha_ {k + 1} \ln \frac {4 K}{\beta}}, \tag {28}
$$
$$
L = \frac {M _ {2} \sqrt {d}}{\tau}, \tag {29}
$$
for some  $K > 0$  and  $\beta \in (0,1]$  such that  $\ln \frac{4K}{\beta} \geq 1$ . Then, after  $K$  iterations of ZO-clipped-SSTM the iterates with probability at least  $1 - \beta$  satisfy
$$
f \left(y ^ {K}\right) - f \left(x ^ {*}\right) \leq 2 M _ {2} \tau + \frac {6 a L R ^ {2}}{K (K + 3)} \quad a n d \quad \left\{x ^ {k} \right\} _ {k = 0} ^ {K + 1}, \left\{z ^ {k} \right\} _ {k = 0} ^ {K}, \left\{y ^ {k} \right\} _ {k = 0} ^ {K} \subseteq B _ {2 R} \left(x ^ {*}\right). \tag {30}
$$
In particular, when parameter  $a$  equals the maximum from (27), then the iterates produced by ZO-clipped-SSTM after  $K$  iterations with probability at least  $1 - \beta$  satisfy
$$
f \left(y ^ {K}\right) - f \left(x ^ {*}\right) \leq 2 M _ {2} \tau + \mathcal {O} \left(\max  \left\{\frac {L R ^ {2} \ln^ {2} \frac {K}{\beta}}{K ^ {2}}, \frac {\sigma R \ln^ {\frac {\alpha - 1}{\alpha}} \frac {K}{\beta}}{(B K) ^ {\frac {\alpha - 1}{\alpha}}} \right\}\right), \tag {31}
$$
meaning that to achieve  $f(y^{K}) - f(x^{*}) \leq \varepsilon$  with probability at least  $1 - \beta$  with  $\tau = \frac{\varepsilon}{4M_2} ZO$ -clipped-SSTM requires
$$
K = \mathcal {O} \left(\sqrt {\frac {M _ {2} ^ {2} \sqrt {d} R ^ {2}}{\varepsilon^ {2}}} \ln \frac {M _ {2} ^ {2} \sqrt {d} R ^ {2}}{\varepsilon^ {2} \beta}, \frac {1}{B} \left(\frac {\sigma R}{\varepsilon}\right) ^ {\frac {\alpha}{\alpha - 1}} \ln \left(\frac {1}{B \beta} \left(\frac {\sigma R}{\varepsilon}\right) ^ {\frac {\alpha}{\alpha - 1}}\right)\right) \quad i t e r a t i o n s. \tag {32}
$$
In case when second term in max in (31) is greater total number of oracle calls is
$$
K \cdot B = \mathcal {O} \left(\left(\frac {\sigma R}{\varepsilon}\right) ^ {\frac {\alpha}{\alpha - 1}} \ln \left(\frac {1}{\beta} \left(\frac {\sigma R}{\varepsilon}\right) ^ {\frac {\alpha}{\alpha - 1}}\right)\right).
$$
Proof. The proof is based on the proof of Theorem F.2 from [35]. We apply first-order algorithm clipped-SSTM for  $\frac{M_2\sqrt{d}}{\tau}$ -smooth function  $\hat{f}_{\tau}$  with unbiased gradient estimation  $g^{B}$ , that has  $\alpha$ -th moment bounded with  $\frac{2\sigma^{\alpha}}{B^{\alpha - 1}}$ . Additional randomization caused by smoothing doesn't affect the proof of the original Theorem.
According to it after  $K$  iterations we have that with probability at least  $1 - \beta$
$$
\hat {f} _ {\tau} (y ^ {K}) - \hat {f} _ {\tau} (x ^ {*}) \leq \frac {6 a L R ^ {2}}{K (K + 3)}
$$
and  $\{x^k\}_{k = 0}^{K + 1},\{z^k\}_{k = 0}^K,\{y^k\}_{k = 0}^K\subseteq B_{2R}(x^*)$
Considering approximation properties of  $\hat{f}_{\tau}$  from Lemma 2
$$
f (y ^ {K}) - f (x ^ {*}) \leq 2 M _ {2} \tau + \frac {6 a L R ^ {2}}{K (K + 3)}.
$$
Finally, if
$$
a = \max  \left\{4 8 6 0 0 \ln^ {2} \frac {4 K}{\beta}, \frac {1 8 0 0 \sigma (K + 1) K ^ {\frac {1}{\alpha}} \ln^ {\frac {\alpha - 1}{\alpha}} \frac {4 K}{\beta}}{B ^ {\frac {\alpha - 1}{\alpha}} L R} \right\},
$$
then with probability at least  $1 - \beta$
$$
\begin{array}{l} f (y ^ {K}) - f (x ^ {*}) \leq 2 M _ {2} \tau + \frac {6 a L R ^ {2}}{K (K + 3)} \\ = 2 M _ {2} \tau + \max  \left\{\frac {2 9 1 6 0 0 L R ^ {2} \ln^ {2} \frac {4 K}{\beta}}{K (K + 3)}, \frac {1 0 8 0 0 \sigma R (K + 1) K ^ {\frac {1}{\alpha}} \ln^ {\frac {\alpha - 1}{\alpha}} \frac {4 K}{\beta}}{K (K + 3) B ^ {\frac {\alpha - 1}{\alpha}}} \right\} \\ = 2 M _ {2} \tau + \mathcal {O} \left(\max  \left\{\frac {L R ^ {2} \ln^ {2} \frac {K}{\beta}}{K ^ {2}}, \frac {\sigma R \ln^ {\frac {\alpha - 1}{\alpha}} \frac {K}{\beta}}{(B K) ^ {\frac {\alpha - 1}{\alpha}}} \right\}\right), \\ \end{array}
$$
where  $L = \frac{M_2\sqrt{d}}{\tau}$  by Lemma 2.
To get  $f(y^{K}) - f(x^{*}) \leq \varepsilon$  with probability at least  $1 - \beta$  it is sufficient to choose  $\tau = \frac{\varepsilon}{4M_2}$  and  $K$  such that both terms in the maximum above are  $\mathcal{O}(\varepsilon)$ . This leads to
$$
K = \mathcal {O} \left(\sqrt {\frac {M _ {2} ^ {2} \sqrt {d} R ^ {2}}{\varepsilon^ {2}}} \ln \frac {M _ {2} ^ {2} \sqrt {d} R ^ {2}}{\varepsilon^ {2} \beta}, \frac {1}{B} \left(\frac {\sigma R}{\varepsilon}\right) ^ {\frac {\alpha}{\alpha - 1}} \ln \left(\frac {1}{B \beta} \left(\frac {\sigma R}{\varepsilon}\right) ^ {\frac {\alpha}{\alpha - 1}}\right)\right)
$$
that concludes the proof.
C.2 Strongly Convex Functions
In the strongly convex case, we consider the restarted version of ZO-clipped-SSTM (R-ZO-clipped-SSTM). The main result is summarized below.
Pay attention that in full version of Theorem 2 we require Assumptions 1, 2 to hold only on  $Q = B_{3R}(x^{*})$ , however we need to require one more smoothness Assumption 5 as well.
Theorem 4 (Full version of Theorem 2) Let Assumptions 1, 2, 5 with  $\mu > 0$  hold for  $Q = B_{3R}(x^{*})$ , where  $R \geq \| x^0 - x^*\|^2$  and R-ZO-clipped-SSTM runs ZO-clipped-SSTM  $N$  times. Let
$$
L _ {t} = \frac {\sqrt {d} M _ {2}}{\tau_ {k}}, \quad \tau_ {k} = \frac {\varepsilon_ {k}}{M _ {2}}, \tag {33}
$$
$$
K _ {t} = \left. \right.\left[ \max  \left\{1 0 8 0 \sqrt {\frac {L _ {t} R _ {t - 1} ^ {2}}{\varepsilon_ {t}}} \ln \frac {2 1 6 0 \sqrt {L _ {t} R _ {t - 1} ^ {2}} N}{\sqrt {\varepsilon_ {t}} \beta}, \frac {2}{B _ {t}} \left(\frac {1 0 8 0 0 \sigma R _ {t - 1}}{\varepsilon_ {t}}\right) ^ {\frac {\alpha}{\alpha - 1}} \ln \left(\frac {4 N}{B _ {t} \beta} \left(\frac {5 4 0 0 \sigma R _ {t - 1}}{\varepsilon_ {t}}\right) ^ {\frac {\alpha}{\alpha - 1}}\right)\right\}\right],
$$
$$
\varepsilon_ {t} = \frac {\mu R _ {t - 1} ^ {2}}{4}, \quad R _ {t - 1} = \frac {R}{2 ^ {(t - 1) / 2}}, \quad N = \left\lceil \log_ {2} \frac {\mu R ^ {2}}{2 \varepsilon} \right\rceil , \quad \ln \frac {4 K _ {t} N}{\beta} \geq 1, \tag {35}
$$
$$
a _ {t} = \max  \left\{4 8 6 0 0 \ln^ {2} \frac {4 K _ {t} N}{\beta}, \frac {1 8 0 0 \sigma \left(K _ {t} + 1\right) K _ {t} ^ {\frac {1}{\alpha}} \ln^ {\frac {\alpha - 1}{\alpha}} \frac {4 K _ {t} N}{\beta}}{B _ {t} ^ {\frac {\alpha - 1}{\alpha}} L _ {t} R _ {t}} \right\}, \tag {36}
$$
$$
\lambda_ {k} ^ {t} = \frac {R _ {t}}{3 0 \alpha_ {k + 1} ^ {t} \ln \frac {4 K _ {t} N}{\beta}} \tag {37}
$$
for  $t = 1,\dots ,\tau$  . Then to guarantee  $f(\hat{x}^{\tau}) - f(x^{*})\leq \varepsilon$  with probability  $\geq 1 - \beta$  R-clipped-SSTM requires
$$
\mathcal {O} \left(\max  \left\{\sqrt {\frac {M _ {2} ^ {2} \sqrt {d}}{\varepsilon \mu}} \ln \left(\frac {\mu R ^ {2}}{\varepsilon}\right) \ln \left(\frac {M _ {2} d ^ {\frac {1}{4}}}{\sqrt {\mu \varepsilon} \beta} \ln \left(\frac {\mu R ^ {2}}{\varepsilon}\right)\right), \left(\frac {\sigma^ {2}}{\mu \varepsilon}\right) ^ {\frac {\alpha}{2 (\alpha - 1)}} \ln \left(\frac {1}{\beta} \left(\frac {\sigma^ {2}}{\mu \varepsilon}\right) ^ {\frac {\alpha}{2 (\alpha - 1)}} \ln \left(\frac {\mu R ^ {2}}{\varepsilon}\right)\right) \right\}\right) \tag {38}
$$
oracle calls. Moreover, with probability  $\geq 1 - \beta$  the iterates of R-clipped-SSTM at stage  $t$  stay in the ball  $B_{2R_{t-1}}(x^*)$ .
Proof. The proof itself repeats of the proof Theorem F.3 from [35]. In this theorem authors prove convergence of restarted clipped-SSTM. In our case it is sufficient to change clipped-SSTM to ZO-clipped-SSTM and put results of Theorem 3 in order to guarantee  $\varepsilon$ -solution after  $\sum_{t=1}^{N} K_t$  successive iterations.
It remains to calculate the overall number of oracle calls during all runs of clipped-SSTM. We have
$$
\begin{array}{l} \sum_ {t = 1} ^ {N} B _ {t} K _ {t} = \\ = \mathcal {O} \left(\sum_ {t = 1} ^ {N} \max  \left\{\sqrt {\frac {M _ {2} ^ {2} \sqrt {d} R _ {t - 1} ^ {2}}{\varepsilon_ {t} ^ {2}}} \ln \left(\frac {\sqrt {M _ {2} ^ {2} \sqrt {d} R _ {t - 1} ^ {2}} N}{\varepsilon_ {t} \beta}\right), \frac {1}{B} \left(\frac {\sigma R _ {t - 1}}{\varepsilon_ {t}}\right) ^ {\frac {\alpha}{\alpha - 1}} \ln \left(\frac {N}{B \beta} \left(\frac {\sigma R _ {t - 1}}{\varepsilon_ {t}}\right) ^ {\frac {\alpha}{\alpha - 1}}\right) \right\}\right) \\ = \mathcal {O} \left(\sum_ {t = 1} ^ {N} \max  \left\{\sqrt {\frac {M _ {2} ^ {2} \sqrt {d}}{R _ {t - 1} ^ {2} \mu^ {2}}} \ln \left(\frac {\sqrt {M _ {2} ^ {2} \sqrt {d}} N}{\mu R _ {t - 1} \beta}\right), \left(\frac {\sigma}{\mu R _ {t - 1}}\right) ^ {\frac {\alpha}{\alpha - 1}} \ln \left(\frac {N}{\beta} \left(\frac {\sigma}{\mu R _ {t - 1}}\right) ^ {\frac {\alpha}{\alpha - 1}}\right) \right\}\right) \\ = \mathcal {O} \left(\max  \left\{\sum_ {t = 1} ^ {N} 2 ^ {t / 2} \sqrt {\frac {M _ {2} ^ {2} \sqrt {d}}{R ^ {2} \mu^ {2}}} \ln \left(2 ^ {t / 2} \frac {\sqrt {M _ {2} ^ {2} \sqrt {d}} N}{\mu R \beta}\right), \sum_ {t = 1} ^ {N} \left(\frac {\sigma \cdot 2 ^ {t / 2}}{\mu R}\right) ^ {\frac {\alpha}{\alpha - 1}} \ln \left(\frac {N}{\beta} \left(\frac {\sigma \cdot 2 ^ {t / 2}}{\mu R}\right) ^ {\frac {\alpha}{\alpha - 1}}\right) \right\}\right) \\ = \mathcal {O} \left(\max  \left\{\sqrt {\frac {M _ {2} ^ {2} \sqrt {d}}{R ^ {2} \mu^ {2}}} 2 ^ {N / 2} \ln \left(2 ^ {N / 2} \frac {\sqrt {M _ {2} ^ {2} \sqrt {d}}}{\mu R \beta} \ln \left(\frac {\mu R ^ {2}}{\varepsilon}\right)\right), \left(\frac {\sigma}{\mu R}\right) ^ {\frac {\alpha}{\alpha - 1}} \ln \left(\frac {N}{\beta} \left(\frac {\sigma \cdot 2 ^ {N / 2}}{\mu R}\right) ^ {\frac {\alpha}{\alpha - 1}}\right) \sum_ {t = 1} ^ {N} 2 ^ {\frac {\alpha t}{2 (\alpha - 1)}} \right\}\right) \\ = \mathcal {O} \left(\max  \left\{\sqrt {\frac {M _ {2} ^ {2} \sqrt {d}}{\varepsilon \mu}} \ln \left(\frac {\sqrt {M _ {2} ^ {2} \sqrt {d}}}{\sqrt {\varepsilon \mu} \beta} \ln \left(\frac {\mu R ^ {2}}{\varepsilon}\right)\right), \left(\frac {\sigma}{\mu R}\right) ^ {\frac {\alpha}{\alpha - 1}} \ln \left(\frac {N}{\beta} \left(\frac {\sigma}{\mu R}\right) ^ {\frac {\alpha}{\alpha - 1}} \cdot 2 ^ {\frac {\alpha}{2 (\alpha - 1)}}\right) 2 ^ {\frac {\alpha N}{2 (\alpha - 1)}} \right\}\right) \\ = \mathcal {O} \left(\max  \left\{\sqrt {\frac {M _ {2} ^ {2} \sqrt {d}}{\varepsilon \mu}} \ln \left(\frac {\sqrt {M _ {2} ^ {2} \sqrt {d}}}{\sqrt {\varepsilon \mu} \beta} \ln \left(\frac {\mu R ^ {2}}{\varepsilon}\right)\right), \left(\frac {\sigma^ {2}}{\mu \varepsilon}\right) ^ {\frac {\alpha}{2 (\alpha - 1)}} \ln \left(\frac {1}{\beta} \left(\frac {\sigma^ {2}}{\mu \varepsilon}\right) ^ {\frac {\alpha}{2 (\alpha - 1)}} \ln \left(\frac {\mu R ^ {2}}{\varepsilon}\right)\right) \right\}\right), \\ \end{array}
$$
which concludes the proof.