SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Blackbox approach. The key lies in the following lemma, which yields the near optimal  $\widetilde{O}(T^{\frac{2 - 2p}{3p - 2}})$  convergence rate of Clipped-SGD. In this case, we assume that the clipping parameters  $\lambda_t$  and the step sizes  $\eta_t$  are fixed. Note that the success probability is only  $1 - T\delta$ . This result uses Lemma 3.3 and Freedman's inequality (Theorem A.1) primarily as a blackbox to bound the error terms inductively by the initial function value gap to optimality.
Lemma 3.5. For  $1 \leq N \leq T + 1$ , let  $\eta_t = \eta$ ,  $\lambda_t = \lambda$  (the specific choices are omitted here for brevity) and  $E_N$  be the event that for all  $k = 1, \ldots, N$ ,
$$
L \eta^ {2} \sum_ {t = 1} ^ {k - 1} \| \theta_ {t} ^ {u} \| ^ {2} + (L \eta^ {2} - \eta) \sum_ {t = 1} ^ {k - 1} \left\langle \nabla f (x _ {t}), \theta_ {t} ^ {u} \right\rangle + \frac {3 \eta}{2} \sum_ {t = 1} ^ {k - 1} \left\| \theta_ {t} ^ {b} \right\| ^ {2} \leq \Delta_ {1}.
$$
Then  $E_N$  happens with probability at least  $1 - \frac{(N - 1)\delta}{T}$  for each  $N \in [T + 1]$ .
With the above lemma, we can obtain a near-optimal convergence rate. However, this rate is still suboptimal due to the use of  $T$  union bounds as part of the induction proof. We now discuss an improved analysis that closes the remaining gap.
Whitebox approach. Our whitebox approach defines a novel supermartingale difference sequence  $Z_{t}$  (shown below) and analyzes its moment generating function from first principles. The sequence is
designed to leverage the structure of the problem and Clipped-SGD via carefully chosen decreasing weights  $z_{t}$  (shown below).
$$
\begin{array}{l} Z _ {t} := z _ {t} \left(\frac {\eta_ {t}}{2} \| \nabla f (x _ {t}) \| ^ {2} + \Delta_ {t + 1} - \Delta_ {t} - \frac {3 \eta_ {t}}{2} \left\| \theta_ {t} ^ {b} \right\| ^ {2} - L \eta_ {t} ^ {2} \mathbb {E} \left[ \| \theta_ {t} ^ {u} \| ^ {2} \mid \mathcal {F} _ {t - 1} \right]\right) \\ - \left(3 z _ {t} ^ {2} L \eta_ {t} ^ {2} \Delta_ {t} + 6 L ^ {2} z _ {t} ^ {2} \eta_ {t} ^ {4} \lambda_ {t} ^ {2}\right) \mathbb {E} \left[ \| \theta_ {t} ^ {u} \| ^ {2} \mid \mathcal {F} _ {t - 1} \right] \\ \end{array}
$$
where  $z_{t}\coloneqq \frac{1}{2P_{t}\eta_{t}\lambda_{t}\max_{i\leq t}\sqrt{2L\Delta_{i}} + 8Q_{t}L\eta_{t}^{2}\lambda_{t}^{2}}$
for  $P_{t}, Q_{t} \in \mathcal{F}_{t-1} \geq 1$ . We also define  $S_{t} := \sum_{i=1}^{t} Z_{i}$ . Note that by selecting  $P_{t}, Q_{t}, \eta_{t}, \lambda_{t}$  appropriately so that  $P_{t} \eta_{t} \lambda_{t}$  and  $Q_{t} \eta_{t}^{2} \lambda_{t}^{2}$  are constants (see for example the proof of Proposition 3.7 in the Appendix), we can ensure that the sequence  $z_{t}$  is decreasing.
We now present Lemma 3.6 which is the main result for controlling the above martingale, whose proof will offer insights into the main technique in this paper. The technique to prove Lemma 3.6 is similar to the standard way of bounding the moment generating function in proving concentration inequalities, such as Freedman's inequality [4, 3]. The main challenge here is to find a way to leverage the structure of Clipped-SGD and choose the suitable coefficients  $z_{t}$ . Similarly to [18] where the authors analyze SGD with sub-Gaussian noise, we analyze the martingale difference sequence in a "whitebox" manner. In [18], however, thanks to the light-tailed noise, the weights  $z_{t}$  can be chosen depending only on the problem parameters and independently of the algorithm history. On the other hand, to use Lemma 2.2, we have to make sure that  $z_{t} \leq \frac{1}{R}$ , where  $R$  is an upper bound for the martingale elements. The key here is to choose  $z_{t}$  depending on the past iterates, and use the function value gaps  $\Delta_{t}$  to absorb the error incurred during the analysis. We give a proof sketch and defer the full version to the Appendix.
Lemma 3.6. For any  $\delta >0$ , let  $E(\delta)$  be the event that for all  $1\leq k\leq T$
$$
\begin{array}{l} \frac {1}{2} \sum_ {t = 1} ^ {k} z _ {t} \eta_ {t} \| \nabla f (x _ {t}) \| ^ {2} + z _ {k} \Delta_ {k + 1} \leq z _ {1} \Delta_ {1} + \log \frac {1}{\delta} + \sum_ {t = 1} ^ {k} \frac {3 z _ {t} \eta_ {t}}{2} \| \theta_ {t} ^ {b} \| ^ {2} \\ + \sum_ {t = 1} ^ {k} \left(\left(3 z _ {t} ^ {2} L \eta_ {t} ^ {2} \Delta_ {t} + 6 L ^ {2} z _ {t} ^ {2} \eta_ {t} ^ {4} \lambda_ {t} ^ {2} + z _ {t} L \eta_ {t} ^ {2}\right) \mathbb {E} \left[ \| \theta_ {t} ^ {u} \| ^ {2} \mid \mathcal {F} _ {t - 1} \right]\right). \\ \end{array}
$$
Then  $\operatorname*{Pr}\left[E(\delta)\right]\geq 1 - \delta$
Proof Sketch. Using Lemmas 3.3, 2.2, and the condition for  $z_{t}$ , we can show that  $\mathbb{E}\left[\exp (Z_t) \mid \mathcal{F}_{t - 1}\right] \leq 1$ . This then implies
$$
\mathbb {E} \left[ \exp \left(S _ {t}\right) \mid \mathcal {F} _ {t - 1} \right] = \exp \left(S _ {t - 1}\right) \mathbb {E} \left[ \exp \left(Z _ {t}\right) \mid \mathcal {F} _ {t - 1} \right] \leq \exp \left(S _ {t - 1}\right),
$$
which means  $(\exp (S_t))_{t\geq 1}$  is a supermartingale. By Ville's inequality, we have, for all  $k\geq 1$ $\operatorname *{Pr}\left[S_k\geq \log \frac{1}{\delta}\right]\leq \delta \mathbb{E}\left[\exp (S_1)\right]\leq \delta$  . In other words, with probability at least  $1 - \delta$  , for all  $k\geq 1$ $\sum_{t = 1}^{k}Z_{t}\leq \log \frac{1}{\delta}$  . Plugging in the definition of  $Z_{t}$  we obtain the desired inequality.
We now specify the choice of  $\eta_t$  and  $\lambda_t$ . The following lemma gives a general condition for the choice of  $\eta_t$  and  $\lambda_t$  that gives the right convergence rate in time  $T$ .
Proposition 3.7. We assume that the event  $E(\delta)$  from Lemma 3.6 happens. Suppose that for some  $\ell \leq T$ , there are constants  $C_1, C_2$  and  $C_3$  such that for all  $t \leq \ell$
$$
\begin{array}{l l} 1.   \lambda_ {t} \eta_ {t} \sqrt {2 L} \leq C _ {1}; & 2.   \frac {1}{L \eta_ {t}} \left(\frac {1}{\lambda_ {t}}\right) ^ {p} \leq C _ {2}; \quad 3.   \sum_ {t = 1} ^ {T} L \left(\frac {1}{\lambda_ {t}}\right) ^ {p} \lambda_ {t} ^ {2} \eta_ {t} ^ {2} \leq C _ {3}; \quad 4.   \| \nabla f (x _ {t}) \| \leq \frac {\lambda_ {t}}{2}. \end{array}
$$
Then for all  $t \leq \ell + 1$
$$
\frac {1}{2} \sum_ {i = 1} ^ {t} \eta_ {i} \| \nabla f (x _ {i}) \| ^ {2} + \Delta_ {t + 1} \leq \left(\sqrt {\Delta_ {1}} + 2 \sqrt {A} C _ {1}\right) ^ {2}
$$
for a constant  $A \geq \max \left\{64\left(\log \frac{1}{\delta} + \frac{60\sigma^p C_3}{C_1^2}\right)^2 + \frac{48\sigma^{2p} C_2 C_3 + 140\sigma^p C_3}{C_1^2}; 1\right\}$ .
Finally, the proof for Theorem 3.1 is a direct consequence of Proposition 3.7 where we defer the details to the Appendix.
Algorithm 2 Clipped-SMD
Parameters: initial point  $x_{1}$ , step sizes  $\{\eta_t\}$ , clipping parameters  $\{\lambda_t\}$ ,  $\psi$  is 1-strongly convex wrt  $\| \cdot \|$
for  $t = 1$  to  $T$  do
$$
\widetilde {\nabla} f (x _ {t}) = \min  \left\{1, \frac {\lambda_ {t}}{\| \widetilde {\nabla} f (x _ {t}) \| _ {*}} \right\} \widehat {\nabla} f (x _ {t})
$$
$$
x _ {t + 1} = \arg \min _ {x \in \mathcal {X}} \left\{\eta_ {t} \left\langle \widetilde {\nabla} f (x _ {t}), x \right\rangle + \mathbf {D} _ {\psi} (x, x _ {t}) \right\}
$$
4 Clipped Stochastic Mirror Descent for Convex Objectives
In this section, we present and analyze the Clipped Stochastic Mirror Descent algorithm (Algorithm 2) under heavy-tailed noise, with a general domain and arbitrary norm.
We define the Bregman divergence  $\mathbf{D}_{\psi}(x,y) = \psi (x) - \psi (y) - \langle \nabla \psi (y),x - y\rangle$  , where  $\psi :\mathbb{R}^d\to \mathbb{R}$  is a 1-strongly convex differentiable function with respect to the norm  $\| \cdot \|$  on  $\mathcal{X}$ . We assume for convenience that  $\mathrm{dom}(\psi) = \mathbb{R}^d$ . Algorithm 2 is a generalization of Clipped-SGD for convex functions to an arbitrary norm. The only difference from the standard Stochastic Mirror Descent algorithm is the use of the clipped gradient  $\widetilde{\nabla} f(x_{t})$  in place of the true stochastic gradient  $\widehat{\nabla} f(x_{t})$  when computing the new iterate  $x_{t + 1}$ .
Prior works such as [7] only consider the setting where the global minimizer lies in  $\mathcal{X}$ . Our algorithm and analysis does not require this restriction and instead only uses the following initial gradient estimate assumption from [22]:
(5) Initial gradient estimate: Let  $x_{1}$  be the initial point. We assume that we have access to an upperbound  $\nabla_{1}$  of  $\| \nabla f(x_{1}) \|_{*}$  i.e.  $\| \nabla f(x_{1}) \|_{*} \leq \nabla_{1}$ . This assumption is justified as follows. If the noise parameter  $\sigma$  defined in assumption (3) is known, we can use the procedure of [21] to estimate  $\| \nabla f(x_{1}) \|_{*}$ : we take  $O(\ln(1/\delta))$  stochastic gradient samples at  $x_{1}$ , and let  $g_{1}$  be the geometric median of these samples; we then set  $\nabla_{1} \coloneqq \| g_{1} \|_{*} + 10\sigma$ . It follows from [21] that  $\| \nabla f(x_{1}) \|_{*} \leq \nabla_{1}$  holds with probability at least  $1 - \delta$ . If the domain contains the global optimum  $x^{*} (\nabla f(x^{*}) = 0)$  and the initial distance  $\| x_{1} - x^{*} \|$  is known, we have the following alternative upper bound that follows from  $\nabla f(x^{*}) = 0$  and smoothness:  $\| \nabla f(x_{1}) \|_{*} = \| \nabla f(x_{1}) - \nabla f(x^{*}) \|_{*} \leq L \| x_{1} - x^{*} \|$ .
Convergence guarantees. We first state the convergence guarantee for this algorithm in Theorem 4.1 which works for an arbitrary norm and a general domain which may not include the global optimum. In this theorem, we assume that we know several problem parameters to show the main idea of our analysis. In Theorem 4.4, we remove the knowledge of the problem parameters.
Theorem 4.1. Assume that convex  $f$  satisfies Assumptions (1), (2), (3), (4) and (5). Let  $\gamma = \max \left\{\log \frac{1}{\delta}; 1\right\}$ ;  $R_{1} = \sqrt{2\mathbf{D}_{\psi}(x^{*}, x_{1})}$ , and assume that  $\nabla_{1}$  is an upper bound of  $\| \nabla f(x_{1}) \|_{*}$ . For known  $T$ , we choose  $\lambda_{t}$  and  $\eta_{t}$  such that
$$
\lambda_ {t} = \lambda = \max  \left\{\left(\frac {2 6 T}{\gamma}\right) ^ {1 / p} \sigma ; 2 \left(3 L R _ {1} + \nabla_ {1}\right) \right\}, a n d
$$
$$
\eta_ {t} = \eta = \frac {R _ {1}}{2 4 \lambda_ {t} \gamma} = \frac {R _ {1}}{2 4 \gamma} \min  \left\{\left(\frac {2 6 T}{\gamma}\right) ^ {- 1 / p} \sigma^ {- 1}; \frac {1}{2} (3 L R _ {1} + \nabla_ {1}) ^ {- 1} \right\}.
$$
$$
\frac {1}{T} \sum_ {t = 2} ^ {T + 1} \Delta_ {t} \leq 4 8 R _ {1} \max  \left\{2 6 ^ {\frac {1}{p}} T ^ {\frac {1 - p}{p}} \sigma \gamma^ {\frac {p - 1}{p}}; 2 \left(3 L R _ {1} + \nabla_ {1}\right) T ^ {- 1} \gamma \right\} = O \left(T ^ {\frac {1 - p}{p}}\right).
$$
Remark 4.2. This theorem shows that the convergence rate for the known time horizon case is  $O(T^{\frac{1 - p}{p}})$ . This rate is known to be optimal, matching the lower bounds shown in [26, 31]. The above guarantee is also adaptive to  $\sigma$ , i.e., when  $\sigma \to 0$ , we obtain the standard  $O(T^{-1})$  convergence rate of deterministic mirror descent.
Remark 4.3. The term  $\nabla_{1}$  in the above theorem comes from the inexact estimation of  $\| \nabla f(x_1)\|_*$ . If we assume that the global optimum lies in the domain  $\mathcal{X}$ , we can simply select  $\nabla_{1} = LR_{1}$  without using the estimation procedure, as discussed in (5).
In Theorem 4.1, we use the initial distance  $R_{1}$  to the optimal solution to set the step sizes and clipping parameters. This information is generally not available, but can be avoided. For example, for constrained problems where the domain radius is bounded by  $R$ , we can replace  $R_{1}$  in Theorem 4.1 by  $R$  without change in the dependency. However, for the general problem, we present Theorem 4.4, where we do not require knowledge of the constants  $T, \sigma, \delta$  or  $R_{1}$  to set the step sizes and clipping parameters. However, we still need the mild assumption of knowing an upper bound  $\nabla_{1}$  on  $\| \nabla f(x_{1}) \|_{*}$ . As discussed in (5),  $\nabla_{1}$  can be estimated with good accuracy when  $\sigma$  is known.
Theorem 4.4. Assume that convex  $f$  satisfies Assumption (1), (2), (3), (4) and (5). Let  $\gamma = \max \left\{\log \frac{1}{\delta}; 1\right\}$ ;  $R_{1} = \sqrt{2\mathbf{D}_{\psi}(x^{*}, x_{1})}$ , and assume that  $\nabla_{1}$  is an upper bound of  $\| \nabla f(x_{1}) \|_{*}$ . We choose  $\lambda_{t}$  and  $\eta_{t}$  such that
$$
\lambda_ {t} = \max  \left\{\left(5 2 t (1 + \log t) ^ {2} c _ {2}\right) ^ {1 / p}; 2 \left(L \max  _ {i \leq t} \| x _ {i} - x _ {1} \| + \nabla_ {1}\right); \frac {L c _ {1}}{6} \right\}, a n d
$$
$$
\eta_ {t} = \frac {c _ {1}}{2 4 \lambda_ {t}} = \frac {c _ {1}}{2 4} \min \left\{\left(5 2 t (1 + \log t) ^ {2} c _ {2}\right) ^ {- 1 / p}; \frac {1}{2 \left(L \max _ {i \leq t} \| x _ {i} - x _ {1} \| + \nabla_ {1}\right)}; \frac {6}{L c _ {1}} \right\},
$$
where the absolute constants  $c_{1}$  and  $c_{2}$  are to ensure the correctness of the dimensions. Then, with probability at least  $1 - \delta$ , we have
$$
\frac {1}{T} \sum_ {t = 2} ^ {T + 1} \Delta_ {t} \leq \frac {8}{T c _ {1}} \left(R _ {1} + \frac {c _ {1}}{3} \left(\gamma + \frac {2 \sigma^ {p}}{c _ {2}}\right)\right) ^ {2} \max  \left\{\left(5 2 T (1 + \log T) ^ {2} c _ {2}\right) ^ {1 / p}; \right.
$$
$$
4 R _ {1} L + \frac {2 c _ {1}}{3} L \left(\gamma + \frac {2 \sigma^ {p}}{c _ {2}}\right) + 2 \nabla_ {1}; \frac {L c _ {1}}{6} \Biggr \} = \widetilde {\cal O} \left(T ^ {\frac {1 - p}{p}}\right).
$$
Sketch of the analysis. In the remainder of this section, we provide a sketch of the analysis for Theorem 4.1, which starts with the following lemma.
Lemma 4.5. Assume that convex  $f$  satisfies Assumption (1), (2), (3), (4) and  $\eta_t \leq \frac{1}{4L}$ , the iterate sequence  $(x_{t})_{t \geq 1}$  output by Algorithm 2 satisfies the following:
$$
\begin{array}{l} \eta_ {t} \Delta_ {t + 1} \leq \mathbf {D} _ {\psi} (x ^ {*}, x _ {t}) - \mathbf {D} _ {\psi} (x ^ {*}, x _ {t + 1}) + \eta_ {t} \left\langle x ^ {*} - x _ {t}, \theta_ {t} ^ {u} \right\rangle + \eta_ {t} \left\langle x ^ {*} - x _ {t}, \theta_ {t} ^ {b} \right\rangle \\ + 2 \eta_ {t} ^ {2} \left(\| \theta_ {t} ^ {u} \| _ {*} ^ {2} - \mathbb {E} \left[ \| \theta_ {t} ^ {u} \| _ {*} ^ {2} \mid \mathcal {F} _ {t - 1} \right]\right) + 2 \eta_ {t} ^ {2} \mathbb {E} \left[ \| \theta_ {t} ^ {u} \| _ {*} ^ {2} \mid \mathcal {F} _ {t - 1} \right] + 2 \eta_ {t} ^ {2} \left\| \theta_ {t} ^ {b} \right\| _ {*} ^ {2}. \\ \end{array}
$$
Remark 4.6. In contrast to Remark 3.4, there is a mismatch between the gain  $\Delta_{t + 1}$  and the loss  $\langle x^{*} - x_{t},\theta_{t}\rangle$ . Since the distance  $\| x^{*} - x_{t}\|$  and the function value gap  $\Delta_t$  cannot be related in the general convex case, we do not obtain the same rate as in the nonconvex case.
We now define the following terms for  $t \geq 1$ :
$$
\begin{array}{l} Z _ {t} := z _ {t} \left(\eta_ {t} \Delta_ {t + 1} + \mathbf {D} _ {\psi} \left(x ^ {*}, x _ {t + 1}\right) - \mathbf {D} _ {\psi} \left(x ^ {*}, x _ {t}\right) - \eta_ {t} \left\langle x ^ {*} - x _ {t}, \theta_ {t} ^ {b} \right\rangle - 2 \eta_ {t} ^ {2} \left\| \theta_ {t} ^ {b} \right\| _ {*} ^ {2} \right. \\ - 2 \eta_ {t} ^ {2} \mathbb {E} \left[ \left\| \theta_ {t} ^ {u} \right\| _ {*} ^ {2} \mid \mathcal {F} _ {t - 1} \right] \Bigg) - \left(\frac {3}{8 \lambda_ {t} ^ {2}} + 2 4 z _ {t} ^ {2} \eta_ {t} ^ {4} \lambda_ {t} ^ {2}\right) \mathbb {E} \left[ \left\| \theta_ {t} ^ {u} \right\| ^ {2} \mid \mathcal {F} _ {t - 1} \right], \\ \end{array}
$$
$$
\mathrm {w h e r e} z _ {t} := \frac {1}{2 \eta_ {t} \lambda_ {t} \max _ {i \leq t} \sqrt {2 \mathbf {D} _ {\psi} (x ^ {*} , x _ {i})} + 1 6 Q \eta_ {t} ^ {2} \lambda_ {t} ^ {2}}
$$
for a constant  $Q \geq 1$ . We also define  $S_{t} \coloneqq \sum_{i=1}^{t} Z_{i}$ . We have the following lemma, which is analogous to Lemma 3.6 in the nonconvex case.
Lemma 4.7. For any  $\delta >0$ , let  $E(\delta)$  be the event that for all  $1\leq k\leq T$
$$
\begin{array}{l} \sum_ {t = 1} ^ {k} z _ {t} \eta_ {t} \Delta_ {t + 1} + z _ {k} \mathbf {D} _ {\psi} (x ^ {*}, x _ {k + 1}) \leq z _ {1} \mathbf {D} _ {\psi} (x ^ {*}, x _ {1}) + \log \frac {1}{\delta} + \sum_ {t = 1} ^ {k} z _ {t} \eta_ {t} \left\langle x ^ {*} - x _ {t}, \theta_ {t} ^ {b} \right\rangle \\ + 2 \sum_ {t = 1} ^ {k} z _ {t} \eta_ {t} ^ {2} \left\| \theta_ {t} ^ {b} \right\| _ {*} ^ {2} + \sum_ {t = 1} ^ {k} \left(\left(2 z _ {t} \eta_ {t} ^ {2} + \frac {3}{8 \lambda_ {t} ^ {2}} + 2 4 z _ {t} ^ {2} \eta_ {t} ^ {4} \lambda_ {t} ^ {2}\right) \mathbb {E} \left[ \| \theta_ {t} ^ {u} \| _ {*} ^ {2} \mid \mathcal {F} _ {t - 1} \right]\right). \tag {6} \\ \end{array}
$$
Then  $\operatorname*{Pr}[E(\delta)]\geq 1 - \delta$
Algorithm 3 Clipped-ASMD
Parameters: initial point  $y_{1} = z_{1}$ , step sizes  $\{\eta_t\}$ , clipping parameters  $\{\lambda_t\}$ , and mirror map  $\psi$  where  $\psi$  is 1-strongly convex wrt  $\| \cdot \|$ .
For  $t = 1$  to  $T$  do:
$$
\text {S e t} \alpha_ {t} = \frac {2}{t + 1}.
$$
$$
x _ {t} = \left(1 - \alpha_ {t}\right) y _ {t} + \alpha_ {t} z _ {t}.
$$
$$
\widetilde {\nabla} f (x _ {t}) = \min  \left\{1, \frac {\lambda_ {t}}{\left\| \widetilde {\nabla} f (x _ {t}) \right\| _ {*}} \right\} \widehat {\nabla} f (x _ {t}).
$$
$$
z _ {t + 1} = \arg \min  _ {x \in \mathcal {X}} \left\{\eta_ {t} \left\langle \widetilde {\nabla} f (x _ {t}), x \right\rangle + \mathbf {D} _ {\psi} (x, z _ {t}) \right\}.
$$
$$
y _ {t + 1} = \left(1 - \alpha_ {t}\right) y _ {t} + \alpha_ {t} z _ {t + 1}.
$$
We now specify the choice of  $\eta_t$  and  $\lambda_t$ . The following proposition gives a general condition for the choice of  $\eta_t$  and  $\lambda_t$  that gives the right convergence rate in time  $T$ .
Proposition 4.8. We assume that the event  $E(\delta)$  from Lemma 4.7 happens. Suppose that for some  $\ell \leq T$ , there are constants  $C_1, C_2, C_3$ , and  $A$  such that for all  $t \leq \ell$
$$
\begin{array}{l l} 1.   \lambda_ {t} \eta_ {t} = C _ {1}; & 2.   \sum_ {t = 1} ^ {\ell} \left(\frac {1}{\lambda_ {t}}\right) ^ {p} \leq C _ {2}; \quad 3.   \left(\frac {1}{\lambda_ {t}}\right) ^ {2 p} \leq C _ {3} \left(\frac {1}{\lambda_ {t}}\right) ^ {p}; \quad 4.   \| \nabla f (x _ {t}) \| _ {*} \leq \frac {\lambda_ {t}}{2}. \end{array}
$$
$$
\sum_ {i = 1} ^ {t} \eta_ {i} \Delta_ {i + 1} + \mathbf {D} _ {\psi} (x ^ {*}, x _ {t + 1}) \leq \frac {1}{2} (R _ {1} + 8 A C _ {1}) ^ {2}
$$
$$
f o r A \geq \max  \left\{\log \frac {1}{\delta} + 2 6 \sigma^ {p} C _ {2} + \frac {2 \sigma^ {2 p} C _ {2} C _ {3}}{A}; 1 \right\}.
$$
Theorem 4.1 follows from Proposition 4.8. Both proofs can be found in the Appendix.
5 Accelerated Stochastic Mirror Descent and Extensions
In Section D in the Appendix, we also show the convergence and its analysis for Clipped Accelerated Stochastic Mirror Descent (Algorithm 3). We require the following additional assumption:
(5') Global minimizer: We assume that  $\nabla f(x^{*}) = 0$
In other words, we assume that the global minimizer lies in the domain of the problem. This assumption is consistent with the works of [7, 28]. Our analysis readily extends to non-smooth settings, and more generally to functions that satisfy  $f(y) - f(x) \leq \langle \nabla f(x), y - x \rangle + G \| y - x \| + \frac{L}{2} \| y - x \|^2$ ,  $\forall y, x \in \mathcal{X}$ . This condition is satisfied by both Lipschitz functions (when  $L = 0$ ) and smooth functions (when  $G = 0$ ). The key step is to extend Lemma 4.5. The proof follows from [15] and can be found in the Appendix.
In this work, we propose a new approach to design and analyze various clipped gradient algorithms in the presence of heavy-tailed noise. Our analysis applies to various standard settings, including Clipped-SMD and accelerated Clipped-SMD for convex objectives with general domains and Clipped-SGD for nonconvex objectives, and gives optimal high probability rates in all settings. Our algorithms allow for setting step-sizes and clipping parameters when the time horizon and problem parameters such as the initial distance are unknown. For future work, since our algorithms have the limitation of still requiring the knowledge of parameters like  $L$  and  $p$ , it is of great interest to investigate the existence of a fully-adaptive method, like Adagrad, that converges under heavy-tailed noise without requiring the knowledge of any problem parameter. Finally, it would be interesting to extend our techniques to the setting of variational inequalities under heavy-tailed noise [6].
TDN and AE were supported in part by NSF CAREER grant CCF-1750333, NSF grant III-1908510, and an Alfred P. Sloan Research Fellowship. THN and HN were supported by NSF CAREER grant CCF-1750716 and NSF grant 2311649.
[1] Alina Beygelzimer, John Langford, Lihong Li, Lev Reyzin, and Robert Schapire. Contextual bandit algorithms with supervised learning guarantees. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pages 19-26. JMLR Workshop and Conference Proceedings, 2011.
[2] Ashok Cutkosky and Harsh Mehta. High-probability bounds for non-convex stochastic optimization with heavy tails. Advances in Neural Information Processing Systems, 34:4883-4895, 2021.
[3] Kacha Dzhaparidze and JH Van Zanten. On bernstein-type inequalities for martingales. Stochastic processes and their applications, 93(1):109-117, 2001.
[4] David A Freedman. On tail probabilities for martingales. the Annals of Probability, pages 100-118, 1975.
[5] Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization, 23(4):2341-2368, 2013.
[6] Eduard Gorbunov, Marina Danilova, David Dobre, Pavel Dvurechenskii, Alexander Gasnikov, and Gauthier Gidel. Clipped stochastic methods for variational inequalities with heavy-tailed noise. Advances in Neural Information Processing Systems, 35:31319-31332, 2022.
[7] Eduard Gorbunov, Marina Danilova, and Alexander Gasnikov. Stochastic optimization with heavy-tailed noise via accelerated gradient clipping. Advances in Neural Information Processing Systems, 33:15042-15053, 2020.
[8] Eduard Gorbunov, Marina Danilova, Innovokentiy Shibaev, Pavel Dvurechensky, and Alexander Gasnikov. Near-optimal high probability complexity bounds for non-smooth stochastic optimization with heavy-tailed noise. arXiv preprint arXiv:2106.05958, 2021.
[9] Mert Gurbuzbalaban, Umut Simsekli, and Lingjiong Zhu. The heavy-tail phenomenon in sgd. In International Conference on Machine Learning, pages 3964-3975. PMLR, 2021.
[10] Nicholas JA Harvey, Christopher Liaw, Yaniv Plan, and Sikander Randhawa. Tight analyses for non-smooth stochastic gradient descent. In Conference on Learning Theory, pages 1579–1613. PMLR, 2019.
[11] Elad Hazan and Satyen Kale. Beyond the regret minimization barrier: optimal algorithms for stochastic strongly-convex optimization. The Journal of Machine Learning Research, 15(1):2489-2512, 2014.
[12] Sham M Kakade and Ambuj Tewari. On the generalization ability of online strongly convex programming algorithms. Advances in Neural Information Processing Systems, 21, 2008.
[13] Ahmed Khaled and Peter Richtárik. Better theory for sgd in the nonconvex world. arXiv preprint arXiv:2002.03329, 2020.
[14] Guanghui Lan. An optimal method for stochastic composite optimization. Mathematical Programming, 133(1):365-397, 2012.
[15] Guanghui Lan. First-order and stochastic optimization methods for machine learning. Springer, 2020.
[16] Shaojie Li and Yong Liu. High probability guarantees for nonconvex stochastic gradient descent with heavy tails. In International Conference on Machine Learning, pages 12931-12963. PMLR, 2022.
A Freedman's inequality
Lemma A.1 (Freedman's inequality). Let  $(X_{t})_{t\geq 1}$  be a martingale difference sequence. Assume that there exists a constant  $c > 0$  such that  $|\bar{X}_t|\leq c$  almost surely for all  $t\geq 1$  and define  $\sigma_t^2 = \mathbb{E}\left[X_t^2\mid X_{t - 1},\ldots ,X_1\right]$ . Then for all  $b > 0$ ,  $F > 0$  and  $T\geq 1$