Title: Improved Convergence in High Probability of Clipped Gradient Methods with Heavy Tailed Noise
Venue: NeurIPS 2023
Accepted: yes
Keywords: convex optimization

non-convex optimization

high probability convergence

heavy-tailed noise

clipped stochastic gradient descent

clipped stochastic mirror descent
Abstract: In this work, we study the convergence in high probability of clipped gradient methods when the noise distribution has heavy tails, i.e., with bounded $p$th moments, for some $1<p\le2$. Prior works in this setting follow the same recipe of using concentration inequalities and an inductive argument with union bound to bound the iterates across all iterations. This method results in an increase in the failure probability by a factor of $T$, where $T$ is the number of iterations. We instead propose a new analysis approach based on bounding the moment generating function of a well chosen supermartingale sequence. We improve the dependency on $T$ in the convergence guarantee for a wide range of algorithms with clipped gradients, including stochastic (accelerated) mirror descent for convex objectives and stochastic gradient descent for nonconvex objectives. Our high probability bounds achieve the optimal convergence rates and match the best currently known in-expectation bounds. Our approach naturally allows the algorithms to use time-varying step sizes and clipping parameters when the time horizon is unknown, which appears difficult or even impossible using the techniques from prior works. Furthermore, we show that in the case of clipped stochastic mirror descent, several problem constants, including the initial distance to the optimum, are not required when setting step sizes and clipping parameters.

Improved Convergence in High Probability of Clipped Gradient Methods with Heavy Tailed Noise

Abstract

In this work, we study the convergence in high probability of clipped gradient methods when the noise distribution has heavy tails, i.e., with bounded p th moments, for some 1 < p ≤ 2 . Prior works in this setting follow the same recipe of using concentration inequalities and an inductive argument with union bound to bound the iterates across all iterations. This method results in an increase in the failure probability by a factor of T , where T is the number of iterations. We instead propose a new analysis approach based on bounding the moment generating function of a well chosen supermartingale sequence. We improve the dependency on T in the convergence guarantee for a wide range of algorithms with clipped gradients, including stochastic (accelerated) mirror descent for convex objectives and stochastic gradient descent for nonconvex objectives. Our high probability bounds achieve the optimal convergence rates and match the best currently known in-expectation bounds. Our approach naturally allows the algorithms to use time-varying step sizes and clipping parameters when the time horizon is unknown, which appears difficult or even impossible using existing techniques from prior works. Furthermore, we show that in the case of clipped stochastic mirror descent, several problem constants, including the initial distance to the optimum, are not required when setting step sizes and clipping parameters.

1 Introduction

Stochastic optimization is a well-studied area with many applications ranging from machine learning, to operation research, numerical linear algebra and beyond. In contrast to deterministic algorithms, stochastic algorithms might fail, and a pertinent question is how often does failure happen and how to increase the success rate. These questions are especially important in critical applications where failure is not tolerable, or when a single run is costly in time and resources. Fortunately, the standard stochastic gradient descent (SGD) algorithm has been shown to converge with high probability under a light-tailed noise distribution such as sub-Gaussian distributions [23, 12, 27, 14, 11, 10, 18], which gives strong guarantee on the success of single runs. However, recent observations in popular deep learning applications, such as training attention models [33] and convolutional networks [30], reveal a more challenging optimization landscape: the gradient noises follow heavy-tailed distributions, where the variance may be infinite [29, 33, 9], whereas

the standard light-tailed setting assumes that all the moments are bounded. Heavy-tailed gradient noises can cause algorithms like SGD to fail, and this mismatch between theory and practice has been suggested to be one of the reasons for the strong preference of adaptive methods like Adam over SGD in modern settings [33].

In this work, we consider the setting of heavy-tailed noise proposed by Zhang et al., (2020) [33], where the (unbiased) gradient noise only has bounded p th moments, for some p ∈ (1 , 2] . While standard SGD can fail to converge when the variance is unbounded, i.e. when p < 2 , [33] show that SGDwith appropriate clipping (or Clipped-SGD ) converges in expectation under heavy-tailed noise, where the convergence rate depends on O ( 1 δ ) if δ is the targeted maximum failure probability. It is more desirable, however, to obtain convergence results in high probability, where the convergence rate depends instead on O (log 1 δ ) , which gives better guarantees for single runs.

Recent follow-up works [2, 28, 19] show that variants of Clipped-SGD in fact converge with high probability. This is a pleasing result, extending the earlier work by [7] for p = 2 . However, there are several shortcomings of these results when compared with the corresponding bounds in the light-tailed setting. First, the clipped algorithm uses a fixed step size and a fixed clipping parameter depending on the number of iterations, which precludes results with unknown time horizons. Secondly, the convergence guarantees are worse than the light-tailed bounds by a log T factor, even for fixed step sizes and clipping parameters. These issues beg a qualitative question:

Is heavy-tailed noise inherently harder than light-tailed noise?

In this work, we answer the above question for Clipped-SGD and the general clipped (accelerated) stochastic mirror descent ( Clipped-SMD ) algorithm. We give an improved analysis framework that not only gives tighter bounds matching the light-tailed noise setting, but also allows for step sizes and clipping parameters for unknown time horizons. Furthermore, we show that this framework is applicable to various settings, from finding minimizers of convex functions with arbitrarily large domains using (accelerated) mirror descent, to finding stationary points for non-convex functions using gradient descent.

1.1 Contributions and Techniques

Our work addresses several open questions posed by previous works including handling general domains and dealing with an unknown time horizon under heavy-tailed noise. Qualitatively, we close the logarithmic suboptimality gap and achieve the optimal rate in several settings. More specifically:

-We demonstrate a novel approach to analyze clipped gradient methods in high probability that is general and applies to various standard settings. In the convex setting, we analyze Clipped-SMD and clipped stochastic accelerated mirror descent. In the non-convex setting, we analyze ClippedSGD. Using our new analysis, we show that clipped methods attain time-optimal convergence in high probability for both convex and nonconvex objectives under heavy-tailed gradient noise. In the convex setting, we obtain an O ( T 1 -p p ) convergence rate for arbitrary (not necessarily compact) convex domains for Clipped-SMD and ( 1 -p -2 ) σ In the nonconvex setting, we obtain the optimal convergence rate of

This bound is also time-optimal and matches the lower bound in [33]; in-expectation convergence of clipped-SGD provided by [33].

O T p σ + T for accelerated Clipped-SMD, where is the noise parameter. These rates are time-optimal and match the lower bounds in [26, 31]. O ( T 2 -2 p 3 p -2 ) for clipped-SGD. it also complements the

-Previous works for heavy-tailed noises follow the recipe of using Freedman-type inequalities [4, 3] as a blackbox and bound the iterates inductively for all iterations. This process incurs an additional log T dependency in the final convergence rate; in other words, the success probability goes from 1 -δ to 1 -Tδ . The step sizes and clipping parameters of this approach depend on the time horizon T to enable the union bound and induction across all iterations in the analysis, excluding the important case when the time horizon is unknown. Our whitebox approach forgoes the aforementioned induction, not only circumventing the log T loss but also allowing for an unknown time horizon. We further show that our analysis allows for a choice of step size and clipping parameters that do not depend on generally unknown parameters like the noise-parameter σ , the

failure probability δ , and the initial distance to the optimum, all of which appear impossible using only the techniques from prior works.

-Our whitebox approach analyzes the moment generating function of a well chosen martingale difference sequence to obtain tight rates for stochastic gradient methods. This approach is closest to the work of [18], which only work in the light-tailed noise setting. In contrast to the light-tailed noise setting where all the moments are well controlled, the heavy-tailed setting often requires algorithms to incorporate gradient clipping for controlling the possibly infinite moments. However, this makes the gradient estimate biased and requires more careful attention to control the bias propagating through the algorithm. Naively applying the technique in [18] is not enough to handle heavy-tailed noise. Rather, as will be shown in our analysis, we introduce a novel history-dependent weights for the martingale sequence that is able to cope with the propagating bias term of clipped methods for heavy-tailed noise across various settings.

1.2 Related Works

High probability convergence for light-tailed noises. Convergence in high probability of stochastic gradient algorithms has been established for sub-Gaussian noises in a number of prior works, including [23, 12, 27, 14, 11, 10] for convex problems with bounded domain (or bounded Bregman diameter) or with strong convexity. Other works [17, 20, 16] study convergence of variants of SGD for nonconvex objectives, where they consider sub-Gaussian and sub-Weibull noises. The most relevant to ours in this line of work is the one by [18], where a whitebox approach is employed to obtain tight rates for stochastic gradient methods in the light-tailed noise setting. However, their technique is not directly applicable in the heavy-tailed noise setting, where we need to introduce new ideas to handle the biases introduced by gradient clipping.

High probability convergence for noises with bounded variance and heavy tails. The design of new gradient algorithms and their analysis in the presence of heavy-tailed noises has drawn significant recent interest. Starting from the work [25] which propose Clipped-SGD to handle exploding gradients in recurrent neural networks, the recent works [30, 29, 33, 9] give new motivation for clipped methods in the context of convolutional networks and attention deep networks that attempts to explain the dominance of adaptive methods over SGD in practical modern scenarios.

While the convergence in expectation of vanilla SGD has been extensively studied [5, 23, 13, 18], only recently has the convergence of Clipped-SGD with heavy tailed noises been closely examined. There, [33] first show the convergence in expectation of Clipped-SGD for nonconvex functions and provide a matching lower bound. In the convex regime, several works with different clipping strategies for the case of p = 2 have shown high probability convergence for smooth problems with bounded domain [22, 24], smooth unconstrained problems [7], and non-smooth problems [8]. A variant of Clipped-SGD that utilizes momentum [2] has also been shown to converge with high probability for bounded p th moments gradient noise. However, the analysis in [2] requires a strong assumption which implies that the true gradients are bounded, a restrictive assumption that excludes objectives like quadratic functions.

More recently, [28, 19, 34] give nearly-optimal convergence rates for several Clipped-SGD variants. These works follow the recipe of using Freedman-type inequalities [4, 3] as a blackbox and bound the iterates inductively for all iterations, which incur an additional log T dependency in the final convergence rate. We show in our work that existing convergence rates can be tightened up and improved. Tight lower bounds for the optimal convergence rate have been shown by [26, 31] for convex objectives and by [33] for nonconvex settings. In both cases, our paper provides optimal convergence guarantees.

In a related but different line of work, [32] show that vanilla SGD can converge with heavy tailed noise for a special type of strongly convex functions, and [31] show that stochastic mirror descent converges in expectation for a special choice of mirror maps, although only for strongly convex objectives with bounded domains.

2 Preliminaries: Assumptions and Notations

We study the problem min x ∈X f ( x ) where f : R d → R and X is the domain of the problem. In the convex setting, we assume that X is a convex set but not necessarily compact. We let ∥·∥ be an

arbitrary norm and ∥·∥ ∗ be its dual norm. In the nonconvex setting, we take X to be R d and consider only the ℓ 2 norm.

2.1 Assumptions

Our paper works with the following assumptions:

Existence of a minimizer : In the convex setting, we assume that there exists x ∗ ∈ arg min x ∈X f ( x ) . We let f ∗ = f ( x ∗ ) .

(1') Existence of a finite lower bound : In the nonconvex setting, we assume that f admits a finite lower bound, i.e., f ∗ := inf x ∈ R d f ( x ) > -∞ .

Unbiased estimato r: We assume that our algorithm is allowed to query a stochastic first-order oracle that returns a history-independent, unbiased gradient estimator ̂ ∇ f ( x ) of ∇ f ( x ) for any x ∈ X . That is, conditioned on the history and the queried point x , we have E [ ̂ ∇ f ( x ) | x ] = ∇ f ( x ) .

Bounded p th moment noise : We assume that there exists σ > 0 such that for some 1 < p ≤ 2 and for any x ∈ X , ̂ ∇ f ( x ) satisfies E [ ∥ ̂ ∇ f ( x ) -∇ f ( x ) ∥ p ∗ | x ] ≤ σ p .

L -smoothness : We consider the class of L -smooth functions: for all x, y ∈ R d , ∥∇ f ( x ) -∇ f ( y ) ∥ ∗ ≤ L ∥ x -y ∥ .

2.2 Gradient Clipping Operator and Notations

We introduce the gradient clipping operator and its general properties used in Clipped-SMD (Algorithm 2) and Clipped-SGD (Algorithm 1). Let x t be the output at iteration t of an algorithm of interest. We denote by ̂ ∇ f ( x t ) the stochastic gradient obtained by querying the gradient oracle. The clipped gradient estimate ˜ ∇ f ( x t ) is taken as

where λ t is the clipping parameter used in iteration t . In subsequent sections, we let ∆ t := f ( x t ) -f ∗ denote the optimal function value gap at x t . We let F t = σ ( ̂ ∇ f ( x 1 ) , . . . , ̂ ∇ f ( x t ) ) be the natural filtration at time t and define the following notations for the stochastic error, the deviation, and the bias of the clipped gradient estimate at time t :

Note that θ u t + θ b t = θ t . Regardless of the convexity of the function f , the following lemma provides upper bounds for these quantities. These bounds can be found in prior works [7, 33, 19, 28] for the

Algorithm 1 Clipped-SGD

Parameters: initial point x , step sizes { η } , clipping parameters { λ }

1 t t for t = 1 to T do

special case of ℓ 2 norm. The extension to the general norm follows in the same manner, which we omit in this work.

Lemma2.1. For stochastic gradients ̂ ∇ f ( x t ) with bounded p th moment noise, the clipped gradients ˜ ∇ f ( x t ) satisfy the following properties:

Furthermore, if ∥∇ f ( x t ) ∥ ∗ ≤ λ t 2 then

Finally, we state a simple but important lemma that bounds the moment generating function of a zero-mean bounded random variable. The proof can be found in, for example, equation (3) of [1].

Lemma 2.2. Let X be a random variable such that E [ X ] = 0 and | X | ≤ R almost surely. Then for 0 ≤ λ ≤ 1 R

3 Clipped Stochastic Gradient Descent for Nonconvex Functions

In this section, we study the convergence of Clipped-SGD for nonconvex functions. Here, we consider the domain to be R d equipped with the standard ℓ 2 norm. We first outline a blackbox concentration argument to show convergence in high probability of Algorithm 1 and then followup with a more powerful whitebox approach that allows for a tight high probability convergence analysis.

Comparison to previous works. In the simple setting of known time horizon and without momentum for Clipped-SGD, the ˜ O ( T 2 -2 p 3 p -2 ) convergence rate has not been shown before to the best of our knowledge. The recent work by [28] study this case and only give a suboptimal rate of ˜ O ( T 1 -p p ) . Note that [2, 19] study other variants of Clipped-SGD with momentums incorporated. Although [2, 19] achieve the nearly-optimal time dependency of ˜ O ( T 2 -2 p 3 p -2 ) in the non-convex settings, they rely on using blackbox concentration inequalities which result in a suboptimal convergence rate that also requires a known time horizon.

We first present the guarantee for known time horizon T via our whitebox approach in Theorem 3.1 and defer the statement for unknown T in Theorem B.2 to the Appendix.

Theorem 3.1. Assume that f satisfies Assumption (1'), (2), (3), (4). Let γ := max { log 1 δ ; 1 } and ∆ 1 := f ( x 1 ) -f ∗ . For known time horizon T , we choose λ t and η t such that

Then with probability at least 1 -δ

Remark 3.2 . In comparison to the corresponding results in [28] (Theorem E.2), while our result achieves a poly T factor better rate when p < 2 , the dependency on log 1 δ in our result contains a dependency on p while the result in [28] does not. That term can dominate the convergence rate in the regime when δ is very small and p is very close to 1 . Hence, an open question is to remove such dependency on p for the log 1 δ term while still maintain the optimal rate on T .

Now, we turn to the analysis, starting with the key Lemma 3.3 (proof in the Appendix).

Lemma 3.3. Assume that f satisfies Assumption (1'), (2), (3), (4) and η t ≤ 1 L then for all t ≥ 1 ,

Remark 3.4 . In Lemma 3.3, we decompose the RHS into appropriate terms that allow us to define a martingale. This lemma helps us understand why we can achieve a better convergence rate O ( T 2 -2 p 3 p -2 ) (for minimizing the norm squared of the gradient) in comparison to the best rate of O ( T 1 -p p ) in the convex setting. We focus on the error term ⟨∇ f ( x t ) , θ t ⟩ = ⟨∇ f ( x t ) , θ u t ⟩ + 〈 ∇ f ( x t ) , θ b t 〉 on the RHS of (5). Since this error contains the gradient ∇ f ( x t ) , we leverage some of the gain ∥∇ f ( x t ) ∥ 2 on the LHS of 5: we use Cauchy-Schwarz to bound 〈 ∇ f ( x t ) , θ b t 〉 ≤ 1 2 ∥∇ f ( x t ) ∥ 2 + 1 2 ∥ θ b t ∥ 2 and use the some of the gain to absorb the first term. Then setting our parameters λ t , η t appropriately to balance the remaining terms helps us achieve the O ( T 2 -2 p 3 p -2 ) rate. Contrast this to the convex setting in the next section: the mismatch between the error term that contains the distance term ∥ x ∗ -x t ∥ and the gain term that contains the function value gap f ( x t ) -f ∗ prevents us from using the gain to absorb some of the error. Thus, this explains the convergence rate discrepancy between the convex case and the non-convex setting (see also Remark 4.6).

Before giving a sketch of our whitebox approach, we present a sketch of a blackbox argument that gives a nearly time-optimal convergence rate. This approach has an additional log T factor in the final rate but will serve as a point of comparison for our new techniques, which will close the logarithmic gap.

Blackbox approach. The key lies in the following lemma, which yields the near optimal ˜ O ( T 2 -2 p 3 p -2 ) convergence rate of Clipped-SGD. In this case, we assume that the clipping parameters λ t and the step sizes η t are fixed. Note that the success probability is only 1 -Tδ . This result uses Lemma 3.3 and Freedman's inequality (Theorem A.1) primarily as a blackbox to bound the error terms inductively by the initial function value gap to optimality.

Lemma 3.5. For 1 ≤ N ≤ T + 1 , let η t = η , λ t = λ (the specific choices are omitted here for brevity) and E N be the event that for all k = 1 , . . . N ,

Then E N happens with probability at least 1 -( N -1) δ T for each N ∈ [ T +1] .

With the above lemma, we can obtain a near-optimal convergence rate. However, this rate is still suboptimal due to the use of T union bounds as part of the induction proof. We now discuss an improved analysis that closes the remaining gap.

Whitebox approach. Our whitebox approach defines a novel supermartingale difference sequence Z t (shown below) and analyzes its moment generating function from first principles. The sequence is

designed to leverage the structure of the problem and Clipped-SGD via carefully chosen decreasing weights z t (shown below).

for P t , Q t ∈ F t -1 ≥ 1 . We also define S t := ∑ t i =1 Z i . Note that by selecting P t , Q t , η t , λ t appropriately so that P t η t λ t and Q t η 2 t λ 2 t are constants (see for example the proof of Proposition 3.7 in the Appendix), we can ensure that the sequence z t is decreasing.

We now present Lemma 3.6 which is the main result for controlling the above martingale, whose proof will offer insights into the main technique in this paper. The technique to prove Lemma 3.6 is similar to the standard way of bounding the moment generating function in proving concentration inequalities, such as Freedman's inequality [4, 3]. The main challenge here is to find a way to leverage the structure of Clipped-SGD and choose the suitable coefficients z t . Similarly to [18] where the authors analyze SGD with sub-Gaussian noise, we analyze the martingale difference sequence in a 'whitebox' manner. In [18], however, thanks to the light-tailed noise, the weights z t can be chosen depending only on the problem parameters and independently of the algorithm history. On the other hand, to use Lemma 2.2, we have to make sure that z t ≤ 1 R , where R is an upper bound for the martingale elements. The key here is to choose z t depending on the past iterates, and use the function value gaps ∆ t to absorb the error incurred during the analysis. We give a proof sketch and defer the full version to the Appendix.

Lemma 3.6. For any δ > 0 , let E ( δ ) be the event that for all 1 ≤ k ≤ T

Then Pr[ E ( δ )] ≥ 1 -δ .

Proof Sketch. Using Lemmas 3.3, 2.2, and the condition for z t , we can show that E [exp ( Z t ) | F t -1 ] ≤ 1 . This then implies

which means (exp ( S t )) t ≥ 1 is a supermartingale. By Ville's inequality, we have, for all k ≥ 1 , Pr [ S k ≥ log 1 δ ] ≤ δ E [exp ( S 1 )] ≤ δ . In other words, with probability at least 1 -δ , for all k ≥ 1 , ∑ k t =1 Z t ≤ log 1 δ . Plugging in the definition of Z t we obtain the desired inequality.

We now specify the choice of η t and λ t . The following lemma gives a general condition for the choice of η t and λ t that gives the right convergence rate in time T .

Proposition 3.7. We assume that the event E ( δ ) from Lemma 3.6 happens. Suppose that for some ℓ ≤ T , there are constants C 1 , C 2 and C 3 such that for all t ≤ ℓ

Then for all t ≤ ℓ +1

Finally, the proof for Theorem 3.1 is a direct consequence of Proposition 3.7 where we defer the details to the Appendix.

Algorithm 2 Clipped-SMD

Parameters: initial point x 1 , step sizes { η t } , clipping parameters { λ t } , ψ is 1 -strongly convex wrt ∥·∥

4 Clipped Stochastic Mirror Descent for Convex Objectives

In this section, we present and analyze the Clipped Stochastic Mirror Descent algorithm (Algorithm 2) under heavy-tailed noise, with a general domain and arbitrary norm.

We define the Bregman divergence D ψ ( x, y ) = ψ ( x ) -ψ ( y ) -⟨∇ ψ ( y ) , x -y ⟩ , where ψ : R d → R is a 1 -strongly convex differentiable function with respect to the norm ∥·∥ on X . We assume for convenience that dom( ψ ) = R d . Algorithm 2 is a generalization of Clipped-SGD for convex functions to an arbitrary norm. The only difference from the standard Stochastic Mirror Descent algorithm is the use of the clipped gradient ˜ ∇ f ( x t ) in place of the true stochastic gradient ̂ ∇ f ( x t ) when computing the new iterate x t +1 .

Prior works such as [7] only consider the setting where the global minimizer lies in X . Our algorithm and analysis does not require this restriction and instead only uses the following initial gradient estimate assumption from [22]:

(5) Initial gradient estimate : Let x 1 be the initial point. We assume that we have access to an upperbound ∇ 1 of ∥∇ f ( x 1 ) ∥ ∗ i.e. ∥∇ f ( x 1 ) ∥ ∗ ≤ ∇ 1 . This assumption is justified as follows. If the noise parameter σ defined in assumption (3) is known, we can use the procedure of [21] to estimate ∥∇ f ( x 1 ) ∥ ∗ : we take O (ln (1 /δ )) stochastic gradient samples at x 1 , and let g 1 be the geometric median of these samples; we then set ∇ 1 := ∥ g 1 ∥ ∗ +10 σ . It follows from [21] that ∥∇ f ( x 1 ) ∥ ∗ ≤ ∇ 1 holds with probability at least 1 -δ . If the domain contains the global optimum x ∗ ( ∇ f ( x ∗ ) = 0 ) and the initial distance ∥ x 1 -x ∗ ∥ is known, we have the following alternative upper bound that follows from ∇ f ( x ∗ ) = 0 and smoothness: ∥∇ f ( x 1 ) ∥ ∗ = ∥∇ f ( x 1 ) -∇ f ( x ∗ ) ∥ ∗ ≤ L ∥ x 1 -x ∗ ∥ .

Convergence guarantees . We first state the convergence guarantee for this algorithm in Theorem 4.1 which works for an arbitrary norm and a general domain which may not include the global optimum. In this theorem, we assume that we know several problem parameters to show the main idea of our analysis. In Theorem 4.4, we remove the knowledge of the problem parameters.

Theorem 4.1. Assume that convex f satisfies Assumptions (1), (2), (3), (4) and (5). Let γ = max { log 1 δ ; 1 } ; R 1 = √ 2 D ψ ( x ∗ , x 1 ) , and assume that ∇ 1 is an upper bound of ∥∇ f ( x 1 ) ∥ ∗ . For known T , we choose λ t and η t such that

Then with probability at least 1 -δ

Remark 4.2 . This theorem shows that the convergence rate for the known time horizon case is O ( T 1 -p p ) . This rate is known to be optimal, matching the lower bounds shown in [26, 31]. The above guarantee is also adaptive to σ , i.e., when σ → 0 , we obtain the standard O ( T -1 ) convergence rate of deterministic mirror descent.

Remark 4.3 . The term ∇ 1 in the above theorem comes from the inexact estimation of ∥∇ f ( x 1 ) ∥ ∗ . If we assume that the global optimum lies in the domain X , we can simply select ∇ 1 = LR 1 without using the estimation procedure, as discussed in (5).

In Theorem 4.1, we use the initial distance R 1 to the optimal solution to set the step sizes and clipping parameters. This information is generally not available, but can be avoided. For example, for constrained problems where the domain radius is bounded by R , we can replace R 1 in Theorem 4.1 by R without change in the dependency. However, for the general problem, we present Theorem 4.4, where we do not require knowledge of the constants T, σ, δ or R 1 to set the step sizes and clipping parameters. However, we still need the mild assumption of knowing an upper bound ∇ 1 on ∥∇ f ( x 1 ) ∥ ∗ . As discussed in (5), ∇ 1 can be estimated with good accuracy when σ is known.

Theorem 4.4. Assume that convex f satisfies Assumption (1), (2), (3), (4) and (5). Let γ = max { log 1 δ ; 1 } ; R 1 = √ 2 D ψ ( x ∗ , x 1 ) , and assume that ∇ 1 is an upper bound of ∥∇ f ( x 1 ) ∥ ∗ . We choose λ t and η t such that

where the absolute constants c 1 and c 2 are to ensure the correctness of the dimensions. Then, with probability at least 1 -δ , we have

Sketch of the analysis . In the remainder of this section, we provide a sketch of the analysis for Theorem 4.1, which starts with the following lemma.

Lemma 4.5. Assume that convex f satisfies Assumption (1), (2), (3), (4) and η t ≤ 1 4 L , the iterate sequence ( x t ) t ≥ 1 output by Algorithm 2 satisfies the following:

Remark 4.6 . In contrast to Remark 3.4, there is a mismatch between the gain ∆ t +1 and the loss ⟨ x ∗ -x t , θ t ⟩ . Since the distance ∥ x ∗ -x t ∥ and the function value gap ∆ t cannot be related in the general convex case, we do not obtain the same rate as in the nonconvex case.

We now define the following terms for t ≥ 1 :

for a constant Q ≥ 1 . We also define S t := ∑ t i =1 Z i . We have the following lemma, which is analogous to Lemma 3.6 in the nonconvex case.

Lemma 4.7. For any δ > 0 , let E ( δ ) be the event that for all 1 ≤ k ≤ T

Then Pr[ E ( δ )] ≥ 1 -δ .

Algorithm 3 Clipped-ASMD

Parameters: initial point y 1 = z 1 , step sizes { η t } , clipping parameters { λ t } , and mirror map ψ , where ψ is 1 -strongly convex wrt ∥·∥ . For t = 1 to T do:

We now specify the choice of η t and λ t . The following proposition gives a general condition for the choice of η t and λ t that gives the right convergence rate in time T .

Proposition 4.8. We assume that the event E ( δ ) from Lemma 4.7 happens. Suppose that for some ℓ ≤ T , there are constants C 1 , C 2 , C 3 , and A such that for all t ≤ ℓ

Then for all t ≤ ℓ +1

Theorem 4.1 follows from Proposition 4.8. Both proofs can be found in the Appendix.

5 Accelerated Stochastic Mirror Descent and Extensions

In Section D in the Appendix, we also show the convergence and its analysis for Clipped Accelerated Stochastic Mirror Descent (Algorithm 3). We require the following additional assumption:

(5') Global minimizer : We assume that ∇ f ( x ∗ ) = 0 .

In other words, we assume that the global minimizer lies in the domain of the problem. This assumption is consistent with the works of [7, 28]. Our analysis readily extends to non-smooth settings, and more generally to functions that satisfy f ( y ) -f ( x ) ≤ ⟨∇ f ( x ) , y -x ⟩ + G ∥ y -x ∥ + L 2 ∥ y -x ∥ 2 , ∀ y, x ∈ X . This condition is satisfied by both Lipschitz functions (when L = 0 ) and smooth functions (when G = 0 ). The key step is to extend Lemma 4.5. The proof follows from [15] and can be found in the Appendix.

6 Conclusion

In this work, we propose a new approach to design and analyze various clipped gradient algorithms in the presence of heavy-tailed noise. Our analysis applies to various standard settings, including Clipped-SMD and accelerated Clipped-SMD for convex objectives with general domains and Clipped-SGD for nonconvex objectives, and gives optimal high probability rates in all settings. Our algorithms allow for setting step-sizes and clipping parameters when the time horizon and problem parameters such as the initial distance are unknown. For future work, since our algorithms have the limitation of still requiring the knowledge of parameters like L and p , it is of great interest to investigate the existence of a fully-adaptive method, like Adagrad, that converges under heavy-tailed noise without requiring the knowledge of any problem parameter. Finally, it would be interesting to extend our techniques to the setting of variational inequalities under heavy-tailed noise [6].

References

Alina Beygelzimer, John Langford, Lihong Li, Lev Reyzin, and Robert Schapire. Contextual bandit algorithms with supervised learning guarantees. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , pages 19-26. JMLR Workshop and Conference Proceedings, 2011.

Ashok Cutkosky and Harsh Mehta. High-probability bounds for non-convex stochastic optimization with heavy tails. Advances in Neural Information Processing Systems , 34:48834895, 2021.

Kacha Dzhaparidze and JH Van Zanten. On bernstein-type inequalities for martingales. Stochastic processes and their applications , 93(1):109-117, 2001.

David A Freedman. On tail probabilities for martingales. the Annals of Probability , pages 100-118, 1975.

Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization , 23(4):2341-2368, 2013.

Eduard Gorbunov, Marina Danilova, David Dobre, Pavel Dvurechenskii, Alexander Gasnikov, and Gauthier Gidel. Clipped stochastic methods for variational inequalities with heavy-tailed noise. Advances in Neural Information Processing Systems , 35:31319-31332, 2022.

Eduard Gorbunov, Marina Danilova, and Alexander Gasnikov. Stochastic optimization with heavy-tailed noise via accelerated gradient clipping. Advances in Neural Information Processing Systems , 33:15042-15053, 2020.

Eduard Gorbunov, Marina Danilova, Innokentiy Shibaev, Pavel Dvurechensky, and Alexander Gasnikov. Near-optimal high probability complexity bounds for non-smooth stochastic optimization with heavy-tailed noise. arXiv preprint arXiv:2106.05958 , 2021.

Mert Gurbuzbalaban, Umut Simsekli, and Lingjiong Zhu. The heavy-tail phenomenon in sgd. In International Conference on Machine Learning , pages 3964-3975. PMLR, 2021.

Nicholas JA Harvey, Christopher Liaw, Yaniv Plan, and Sikander Randhawa. Tight analyses for non-smooth stochastic gradient descent. In Conference on Learning Theory , pages 1579-1613. PMLR, 2019.

Elad Hazan and Satyen Kale. Beyond the regret minimization barrier: optimal algorithms for stochastic strongly-convex optimization. The Journal of Machine Learning Research , 15(1):2489-2512, 2014.

Sham M Kakade and Ambuj Tewari. On the generalization ability of online strongly convex programming algorithms. Advances in Neural Information Processing Systems , 21, 2008.

Ahmed Khaled and Peter Richtárik. Better theory for sgd in the nonconvex world. arXiv preprint arXiv:2002.03329 , 2020.

Guanghui Lan. An optimal method for stochastic composite optimization. Mathematical Programming , 133(1):365-397, 2012.

Guanghui Lan. First-order and stochastic optimization methods for machine learning . Springer, 2020.

Shaojie Li and Yong Liu. High probability guarantees for nonconvex stochastic gradient descent with heavy tails. In International Conference on Machine Learning , pages 1293112963. PMLR, 2022.

Xiaoyu Li and Francesco Orabona. A high probability analysis of adaptive sgd with momentum. arXiv preprint arXiv:2007.14294 , 2020.

Zijian Liu, Ta Duy Nguyen, Thien Hang Nguyen, Alina Ene, and Huy Lê Nguyen. High probability convergence of stochastic gradient methods. International Conference on Machine Learning , 2023.

Zijian Liu, Jiawei Zhang, and Zhengyuan Zhou. Breaking the lower bound with (little) structure: Acceleration in non-convex stochastic optimization with heavy-tailed noise. arXiv preprint arXiv:2302.06763 , 2023.

Liam Madden, Emiliano Dall'Anese, and Stephen Becker. High probability convergence and uniform stability bounds for nonconvex stochastic gradient descent. arXiv preprint arXiv:2006.05610 , 2020.

Stanislav Minsker. Geometric median and robust estimation in banach spaces. Bernoulli , pages 2308-2335, 2015.

Alexander V Nazin, Arkadi S Nemirovsky, Alexandre B Tsybakov, and Anatoli B Juditsky. Algorithms of robust stochastic optimization based on mirror descent method. Automation and Remote Control , 80(9):1607-1627, 2019.

Arkadi Nemirovski, Anatoli Juditsky, Guanghui Lan, and Alexander Shapiro. Robust stochastic approximation approach to stochastic programming. SIAM Journal on optimization , 19(4):1574-1609, 2009.

Daniela A Parletta, Andrea Paudice, Massimiliano Pontil, and Saverio Salzo. High probability bounds for stochastic subgradient schemes with heavy tailed noise. arXiv preprint arXiv:2208.08567 , 2022.

Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. Understanding the exploding gradient problem. CoRR, abs/1211.5063 , 2(417):1, 2012.

Maxim Raginsky and Alexander Rakhlin. Information complexity of black-box convex optimization: A new look via feedback information theory. In 2009 47th Annual Allerton Conference on Communication, Control, and Computing (Allerton) , pages 803-510. IEEE, 2009.

Alexander Rakhlin, Ohad Shamir, and Karthik Sridharan. Making gradient descent optimal for strongly convex stochastic optimization. arXiv preprint arXiv:1109.5647 , 2011.

Abdurakhmon Sadiev, Marina Danilova, Eduard Gorbunov, Samuel Horváth, Gauthier Gidel, Pavel Dvurechensky, Alexander Gasnikov, and Peter Richtárik. High-probability bounds for stochastic optimization and variational inequalities: the case of unbounded variance. arXiv preprint arXiv:2302.00999 , 2023.

Umut ¸ Sim¸ sekli, Mert Gürbüzbalaban, Thanh Huy Nguyen, Gaël Richard, and Levent Sagun. On the heavy-tailed theory of stochastic gradient descent for deep neural networks. arXiv preprint arXiv:1912.00018 , 2019.

Umut Simsekli, Levent Sagun, and Mert Gurbuzbalaban. A tail-index analysis of stochastic gradient noise in deep neural networks. pages 5827-5837, 2019.

Nuri Mert Vural, Lu Yu, Krishna Balasubramanian, Stanislav Volgushev, and Murat A Erdogdu. Mirror descent strikes again: Optimal stochastic convex optimization under infinite noise variance. In Conference on Learning Theory , pages 65-102. PMLR, 2022.

Hongjian Wang, Mert Gurbuzbalaban, Lingjiong Zhu, Umut Simsekli, and Murat A Erdogdu. Convergence rates of stochastic gradient descent under infinite noise variance. Advances in Neural Information Processing Systems , 34:18866-18877, 2021.

Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit, Seungyeon Kim, Sashank Reddi, Sanjiv Kumar, and Suvrit Sra. Why are adaptive methods good for attention models? Advances in Neural Information Processing Systems , 33:15383-15393, 2020.

Jiujia Zhang and Ashok Cutkosky. Parameter-free regret in high probability with heavy tails. In Advances in Neural Information Processing Systems , 2022.

A Freedman's inequality

Lemma A.1 (Freedman's inequality) . Let ( X t ) t ≥ 1 be a martingale difference sequence. Assume that there exists a constant c > 0 such that | X t | ≤ c almost surely for all t ≥ 1 and define σ 2 t = E [ X 2 t | X t -1 , . . . , X 1 ] . Then for all b > 0 , F > 0 and T ≥ 1

B Missing Proofs from Section 3

Proof of Lemma 3.3. By the smoothness of f and the update x t +1 = x t -1 η t ˜ ∇ f ( x t ) we have

where the third inequality is due to ∥ θ t ∥ 2 ≤ 2 ∥ θ u t ∥ 2 + 2 ∥ ∥ θ b t ∥ ∥ 2 , and the last inequality is due to η t ≤ 1 L . Rearranging, adding, and subtracting E [ ∥ θ u t ∥ 2 | F t -1 ] , we obtain the lemma.

Detailed proof of Lemma 3.5. We state the following simple properties of the choice of η and λ in Theorem 3.1. We have

We will now prove by induction on N that E N happens with probability at least 1 -( N -1) δ T . For N = 1 , the event happens with probability 1 . Suppose that for some N ≤ T , Pr[ E N ] ≥ 1 -( N -1) δ T . We will prove that Pr[ E N +1 ] ≥ 1 -Nδ T .

Since the LHS of (5) is non-negative, for k ≤ N , we have, under the event E N ,

From the induction hypothesis and Lemma 3.3, we have that for all k ≤ N , ∆ k ≤ 2∆ 1 . Since the LHS of (5) is non-negative, by summing over t from 1 to N we have,

The bounds for B and D are straightforward from Lemma 2.1. First, with probability 1 , we have ∥ θ u t ∥ ≤ 2 λ . By the smoothness of f and the fact that f is bounded below, we have

Furthermore, when the event E N happens, we have

Thus, we can apply Lemma 2.1 and obtain ∥ ∥ θ b t ∥ ∥ ≤ 4 σ p λ 1 -p and E t [ ∥ θ u t ∥ 2 ] ≤ 40 σ p λ 2 -p .

Upperbound for B . By (3), when the event E N happens,

Upperbound for D . By 4, when the event E N happens,

To bound A and C , we use Freedman's inequality (Theorem A.1). We define, for t ≥ 1 , the following random variables

Thus ∥ Z t ∥ ≤ ∥∇ f ( x t ) ∥ ≤ 2 √ L ∆ 1 for all t .

Upperbound for A . Instead of bounding A = ( η -Lη 2 ) ∑ N t =1 ⟨-∇ f ( x t ) , θ u t ⟩ , we will bound A ′ = ( η -Lη 2 ) ∑ N t =1 ⟨ Z t , θ u t ⟩ . We check the conditions to apply Freedman's inequality. First E t [( η -Lη 2 ) ⟨ Z t , θ u t ⟩ ] = 0 . Further, with probability 1 , ∥ θ u t ∥ 2 ≤ 2 λ , and Z t ≤ 2 √ L ∆ 1 , thus ∣ ∣ ( η -Lη 2 ) ⟨ Z t , θ u t ⟩ ∣ ∣ ≤ ( η -Lη 2 ) ∥ Z t ∥ ∥ θ u t ∥ ≤ 4 √ L ∆ 1 ( η -Lη 2 ) λ ≤ 4 √ L ∆ 1 ηλ . Hence, {( η -Lη 2 ) ⟨ Z t , θ u t ⟩ } is a bounded martingale difference sequence. Therefore, for constant a and F to be chosen we have

We choose a such that which gives

If we choose F = 64 L ∆ 1 σ p λ 2 -p η 2 T , we can easily show that a ≤ 7∆ 1 12 . Therefore, with probability at least 1 -δ 2 T we have

Also notice that under the event E N , we have

Under E N , we have that Z t = -∇ f ( x t ) for all t ≤ N . Therefore, when E N ∩ E A happens, we have A = A ′ ≤ a.

Upperbound for C . We check the conditions to apply Freedman's inequality. First, E t [ Lη 2 ( ∥ θ u t ∥ 2 -E t [ ∥ θ u t ∥ 2 ])] = 0 . Further, with probability 1 , ∥ θ u t ∥ 2 ≤ 2 λ , thus ∣ ∣ ∣ Lη 2 ( ∥ θ u t ∥ 2 -E t [ ∥ θ u t ∥ 2 ])∣ ∣ ∣ ≤ Lη 2 ( 4 λ 2 +4 λ 2 ) = 8 Lλ 2 η 2 . Hence { Lη 2 ( ∥ θ u t ∥ 2 -E t [ ∥ θ u t ∥ 2 ])} is a bounded martingale difference sequence. Applying Freedman's inequality for constants c and G to be chosen, we have

We choose c such that which gives

If we choose G = 256 L 2 σ p λ 4 -p η 4 T , a simple calculation shows that c ≤ 7∆ 1 48 . we can show that with probability at least 1 -δ 2 T , the following event happens

Notice that when G = 256 L 2 σ p λ 4 -p η 4 T , under E N we have

Therefore, when E N ∩ E C happens, we have C ≤ c .

Finally, combining all the bounds for A,B,C,D using union bound and selecting λ and η appropriately to simplify the constants, we obtain the lemma.

Proof of Lemma 3.6. We have

For ( a ) we use Lemma 3.3. For ( b ) we use Lemma 2.2. Notice that

and since ∥ θ u t ∥ ≤ 2 λ t and ∥∇ f ( x t ) ∥ ≤ √ 2 L ∆ t for an L -smooth function, we have

Thus z t ≤ 1 2 η t λ t √ 2 L ∆ t +8 Lη 2 t λ 2 t . For ( c ) we use ( a + b ) 2 ≤ 2 a 2 + 2 b 2 and E [ ( X -E [ X ]) 2 ] E [ X 2 ] . For ( d ) , we use ∥∇ f ( x t ) ∥ 2 ≤ 2 L ∆ t and ∥ θ u t ∥ ≤ 2 λ t . We obtain E [exp ( Z t ) | F t -1 ] ≤ 1 .

Therefore

which means (exp ( S t )) t ≥ 1 is a supermartingale. By Ville's inequality, we have, for all k ≥ 1

In other words, with probability at least 1 -δ , for all k ≥ 1

Plugging in the definition of Z t we have

Note that we have z t is a decreasing sequence by construction (see the proof of Proposition 3.7 below). Hence, the LHS of the above inequality can be bounded by

We obtain the desired inequality.

Proof of Proposition 3.7. We will prove by induction on k that

The base case k = 0 is trivial. Suppose the statement is true for all t ≤ k ≤ ℓ . Now we show for k +1 . Recall that

Let us choose

We have

Now, note that ( z t ) t ≥ 1 is a decreasing sequence. By the induction hypothesis max i ≤ k √ ∆ i ≤ √ ∆ 1 +2 √ AC 1 . Hence:

By the choice of λ t , for all t ≤ k , ∥∇ f ( x t ) ∥ ≤ λ t 2 , we can apply the second part of Lemma 2.1 to obtain

Thus,

Since t ≤ 2 , we have

since

This concludes the proof.

Lemma B.1. The choices of η t and λ t in Theorem 3.1 satisfy the condition (1)-(3) of Proposition 3.7 for

Proof. We verify for the first case. The second follows exactly the same. First, we have p > 1 hence

For ( a ) , we use ( 1 λ t ) p ≤ C 2 Lη t and the induction hypothesis. For ( b ) , we use ∑ T t =1 L ( 1 λ t ) p λ 2 t η 2 t ≤ C 3 and Q t ≥ 1 . For ( c ) , we have

Finally, we have λ t ≥ 32 1 /p σT 1 3 p -2 hence

Therefore,

Proof of Theorem 3.1. Note that η ≤ T 1 -p 3 p -2 16 √ 90 Lγ ≤ 1 L . We have that with probability at least 1 -δ , event E ( δ ) happens. Conditioning on this event, we verify the conditions of Proposition 3.7. We select the following constants

We verify in Lemma B.1 that for these choice of constants, conditions (1)-(3) of Proposition 3.7 are satisfied. Furthermore, we have

We only need to show that, for all t , ∥∇ f ( x t ) ∥ ≤ λ t 2 . We will show this by induction. Indeed, for the base case we have ∥∇ f ( x 1 ) ∥ ≤ √ 2 L ∆ 1 ≤ λ 1 2 . Suppose that it is true for all t ≤ k . We will prove that ∥∇ f ( x k +1 ) ∥ ≤ λ k +1 2 . By Proposition 3.7 and the induction hypothesis

Thus, we get

as needed. From Proposition 3.7, we have

Therefore

Theorem B.2. Assume that f satisfies Assumption (1'), (2), (3), (4). Let γ = max { log 1 δ ; 1 } and ∆ 1 = f ( x 1 ) -f ∗ . For unknown T , we choose λ t and η t such that

Then with probability at least 1 -δ

We again verify the conditions of Proposition 3.7 for the choices of η t and λ t in Theorem B.2.

Lemma B.3. The choices of η t and λ t in Theorem B.2 satisfy the condition (1)-(3) of Proposition 3.7 for

The proof utilizes the following fact:

Fact B.4. We have ∑ ∞ t =1 1 2 t (1+log t ) 2 < 1 .

Proof. First, we have p > 1 hence

which gives

Finally, we have λ t ≥ 32 1 p σ ( 2 t (1 + log t ) 2 ) 1 3 p -2 , hence

Therefore,

Proof of Theorem B.2. Note that

Note that with Lemma B.3, verifying the conditions of Proposition 3.7 is identical to the proof of theorem 3.1. We have that with probability at least 1 -δ , event E ( δ ) from 3.7 happens. We have with probability at least 1 -δ :

Since η t is decreasing, we have

This means that

C Missing Proofs from Section 4

Lemma C.1. Suppose that η t ≤ 1 4 L and assume f satisfies Assumption (1), (2), (3) as well as the following condition

Then the iterate sequence ( x t ) t ≥ 1 output by Algorithm 2 satisfies the following:

Proof. By condition (14) and convexity,

By the optimality condition, we have

and thus

Note that

Thus

where we have used that D ψ ( x t +1 , x t ) ≥ 1 2 ∥ x t +1 -x t ∥ 2 by the strong convexity of ψ . Combining the two inequalities, and using the assumption that Lη t ≤ 1 4 , we obtain

This is what we want to show.

Proof of Lemma 4.7. We have

2

For ( a ) , we use Lemma 4.5. For ( b ) , we use Lemma 2.2. Notice that

and since ∥ θ u t ∥ ∗ ≤ 2 λ t , we have

Thus, z t ≤ 1 2 η t λ t √ 2 D ψ ( x ∗ ,x t )+16 η 2 t λ 2 t . For ( c ) , we use the inequalities ( a + b ) 2 ≤ 2 a 2 + 2 b 2 and E [ ( X -E [ X ]) 2 ] ≤ E [ X 2 ] . For ( d ) , we use the fact ∥ θ u t ∥ 2 ∗ ≤ 4 λ 2 t to get E [ ∥ θ u t ∥ 4 ∗ | F t -1 ] ≤ 4 λ 2 t E [ ∥ θ u t ∥ 2 ∗ | F t -1 ] . For ( e ) , we use the fact that ∥ θ u t ∥ ∗ ≤ 2 λ t and

We obtain E [exp ( Z t ) | F t -1 ] ≤ 1 . Therefore

which means (exp ( S t )) t ≥ 1 is a supermartingale. By Ville's inequality, we have, for all k ≥ 1

In other words, with probability at least 1 -δ , for all k ≥ 1

Plugging in the definition of Z t we have

Thus, we have

Note that we have z t is a decreasing sequence, hence the LHS of the above inequality can be bounded by

We obtain from here the desired inequality.

Proof of Proposition 4.8. We will prove by induction that on k

The base case k = 0 is trivial. We have D ψ ( x ∗ , x 1 ) = R 2 1 2 . Suppose the statement is true for all t ≤ k ≤ ℓ . Now, we show for k +1 . Recall that

Let us choose Q = A > 1 . By the induction hypothesis, we have max i ≤ t √ 2 D ψ ( x ∗ , x i ) ≤ R 1 +8 AC 1 , which implies

For an upperbound, since √ 2 D ψ ( x ∗ , x 1 ) = R 1 , we have:

Since z k is a decreasing sequence, we have

By the choice of λ t , for all t ≤ k , ∥∇ f ( x t ) ∥ ∗ ≤ λ t 2 , we can apply Lemma 2.1 and have

where for the last inequality we use ∑ k t =1 ( 1 λ t ) p ≤ C 2 and ( 1 λ t ) 2 p ≤ C 3 ( 1 λ t ) p . We obtain

Proof of Theorem 4.1. Note that our choice of η ensures η ≤ R 1 16 1 4 LR 1 ≤ 1 4 L . We have that with probability at least 1 -δ , event E ( δ ) happens. Conditioning on this event, in 4.8 we choose

We have

We only need to show that for all t

By 4.8 we have

We will show this by induction. Indeed, we have

Suppose that it is true for all t ≤ k . We prove that

Thus

as needed. Therefore from Lemma 4.7 we have

which gives

Theorem C.2. Assume that f satisfies Assumption (1), (2), (3), (4) and (5). Let γ = max { log 1 δ ; 1 } ; R 1 = √ 2 D ψ ( x ∗ , x 1 ) assume that ∇ 1 is an upper bound of ∥∇ f ( x 1 ) ∥ ∗ . For unknown T , we choose

Then with probability at least 1 -δ

Proof. We can follow the similar steps. Notice that ( η t ) is a decreasing sequence. We also use Fact B.4 to verify the second condition of Proposition 4.8. The proof is omitted.

Proof of Theorem 4.4. Note that η t ≤ 1 4 L . We have that with probability at least 1 -δ , event E ( δ ) happens. Conditioning on this event, in 4.8. We choose

We verify the conditions of Proposition 4.8

where we have 2 σ 2 p C 2 C 3 A ≤ 2 σ 2 p C 2 C 3 × c 2 2 σ p ≤ σ p c 2 . Also, note that

Therefore, from Lemma 4.7, we have

which gives

Note that

which gives us the final convergence rate.

D Clipped Accelerated Stochastic Mirror Descent

In this section, we extend the analysis of Clipped-SMD to the case of Clipped Accelerated Stochastic Mirror Descent (Algorithm 3). We will see that the analysis is basically the same with little modification. We present in Algorithm 3 the clipped version of accelerated stochastic mirror descent (see [15]), where the clipped gradient ˜ ∇ f ( x t ) is used to update the iterates in place of the stochastic gradient ̂ ∇ f ( x t ) .

We use the following additional assumption:

(5') Global minimizer : We assume that ∇ f ( x ∗ ) = 0 .

Theorem D.1. Assume that f satisfies Assumption (1), (2), (3), (4) and (5'). Let γ = max { log 1 δ ; 1 } ; and R 1 = √ 2 D ψ ( x ∗ , x 1 ) .

1. For known T , we choose a constant c and λ t and η t such that

Then with probability at least 1 -δ

2. For unknown T , we choose c t , λ t and η t such that

Then with probability at least 1 -δ

Remark D.2 . One feature of the accelerated algorithm is the interpolation between the two regimes: When σ is large, the algorithm achieves the O ( T 1 -p p ) convergence rate, which is the same as unaccelerated algorithms; however, when σ is sufficiently small, the algorithm achieves the accelerated O ( T -2 ) rate.

We also start the analysis of accelerated stochastic mirror descent with the following lemma.

Lemma D.3. Assume that f satisfies Assumption (1), (2), (3), (4) and η t ≤ 1 2 Lα t , the iterate sequence ( x t ) t ≥ 1 output by Algorithm 2 satisfies the following

By the optimality condition, we have and thus

Note that

Thus

where we have used that D ψ ( z t +1 , z t ) ≥ 1 2 ∥ z t +1 -z t ∥ 2 by the strong convexity of ψ . We have

Dividing both sides by α t η t and using the condition Lη t α t ≤ 1 2 , we have

as needed.

Similarly to the previous section, we define the following variables

where z t = 1 2 η t λ t max i ≤ t √ 2 D ψ ( x ∗ , x i ) + 16 Qη 2 t λ 2 t for a constant Q ≥ 1 . We also let S t = ∑ t i =1 Z i . Following the same analysis as in previous sections, we can obtain Lemma D.4 and Proposition D.5, for which we will omit the proofs here. The only step we need to pay attention to when showing Lemma D.4 is when we bound the sum

If we assume η t -1 α t -1 ≥ η t (1 -α t ) α t , since z t is a decreasing sequence and α 1 = 0 , we can lower bound the above sum by the last term z k η k α k ( f ( y k +1 ) -f ( x ∗ )) , which gives us the desired inequality.

Lemma D.4. Assume that for all t ≥ 1 , η t satisfies η t -1 α t -1 ≥ η t (1 -α t ) α t . For any δ > 0 , let E ( δ ) be the event that for all 1 ≤ k ≤ T

Furthermore

For k ≥ 1 we have α k +1 = 2 k +2 < 1 ; α k +1 1 -α k +1 = 2 k ≤ 4 k +2 ≤ 2 α t +1 and α t ≤ 3 2 α t +1 . Hence,

Then Pr[ E ( δ )] ≥ 1 -δ .

Finally, we state a general condition for the choice of η t and λ t , which follows exactly the same as in Proposition 4.8. The proof for Theorem D.1 is a direct consequence of this.

Proposition D.5. We assume that the event E ( δ ) from Lemma D.4 happens. Suppose that for some ℓ ≤ T , there are constants C 1 and C 2 such that for all t ≤ ℓ

Then for all t ≤ ℓ +1

for A ≥ max { log 1 δ +26 σ p C 2 + 2 σ 2 p C 2 C 3 A ; 1 } .

Proof of Theorem D.1. 1. Note that η t ≤ 1 2 cγ 2 Lα t ≤ 1 2 Lα t and

thus η t -1 α t -1 ≥ η t (1 -α t ) α t . We have that with probability at least 1 -δ , event E ( δ ) happens. Conditioning on this event, in 4.8 We choose

We can verify the conditions of Proposition D.5 similarly as in previous section for these choices of C 1 , C 2 , and C 3 .

We will show by induction that for all t ≥ 1 , ∥∇ f ( x t ) ∥ ∗ ≤ λ t 2 and max {∥ x t -x ∗ ∥ , ∥ y t -x ∗ ∥ , ∥ z t -x ∗ ∥} ≤ 2 R 1 .

For t = 1 , notice that x 1 = y 1 = z 1 . Thus, we have

Now assume that the claim holds for 1 ≤ t ≤ k . By Proposition D.5, we know that

which gives

2. Following the similar steps to the proof of Theorem D.1, and noticing that ( c t ) is a increasing sequence, we obtain the convergence rate.

as needed. Therefore, we have

Peer reviews:
comments: SUMMARY:
The author(s) proposed an improved convergence analysis of clipped gradient descent with heavey tail noise. By the improved analysis, the author(s) can either improved the logarithmic dependency of $T$ or relax the known time horizon assumption.

STRENGTHS:
- The author(s) proposed a new analysis framework for the high-probability error bound of clipped SGD with heavy-tail noise. The analysis is different from the classic Freedman concentration analysis, and can be used in more flexible settings compared to previous analysis, e.g., unknown time horizon, the new analysis also results in slightly better dependency of $T$.
- The writing is clear and easy to follow.
- Related works are well-addressed to my knowledge.
- Overall, I think this paper makes a decent contribution to the field if all other reviewers believe the proof is correct.

WEAKNESSES:
- It is better to include a table to compare with prior works in terms of the convegence rate and the assumptions being used. So readers can better understand the position of this work in the literature.

QUESTIONS:
- Ling 190, there is no $ \langle \nabla f(x_t), \theta_t \rangle $ in equation (5), is this a typo?

LIMITATIONS:
I do not find any nagative social impact of this work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the problem of deriving high-probability complexity bounds for the convex and non-convex optimization problems on closed convex sets under the assumption that the noise in the stochastic gradient has bounded $p$th moment for some $1<p\\leq 2$. The key feature of the results is that they have improved dependence on the failure probability $\\delta$ in comparison to the prior works. In the convex case, the authors also consider non-Euclidean norms, Clipped Stochastic Mirror Descent, and its accelerated version. Overall, the results are good and very relevant for the community.\n\nSTRENGTHS:\n1. **Improved dependence on $\\delta$ in the convex case.** Previous works addressing similar setups provide complexity results that depend on $\\delta$ as $\\log(1/(\\varepsilon\\delta))$ while the rates derived in this work (in the convex case, for known horizon) for Clipped-SMD and Clipped-ASMD imply the complexities that are proportional to $\\log(1/\\delta)$. Although this affects only the logarithmic term, I believe it is an important contribution to the stochastic optimization literature. In particular, this requires applying the proof technique (without induction argument) that differs from the existing approaches.\n\n2. **Constrained case for Clipped-SGD and non-Euclidean prox-structure.** This works provides high-probability convergence analysis for the constrained problems for the first time without the bounded variance assumption. Moreover, it provides an extension to the non-Euclidean case. This is an important step forward.\n\n3. **Horizon-independent results.** The authors also provide results that are independent of the time horizon $T$, while the previous works explicitly use $T$ to choose the parameters of the methods.\n\n4. The paper is well-written. I did not find any serious issues in the proofs.\n\nWEAKNESSES:\n1. **Deterministic term and logarithms in the results for the non-convex case.** Although the dominating terms (that contain $\\sigma$) has optimal dependence on $T$, the deterministic term (the one that does not depend on $\\sigma$) is $O(T^{\\frac{1-2p}{3p-2}})$ that is not optimal for the deterministic case: it becomes worse when $p \\to 1$, while should be $O(T^{-1})$ as for Gradient Descent.  Such a $O(T^{-1})$ deterministic term can be achieved (e.g., see Sadiev et al. (2023)). Next, the improvement of the logarithmic factor is not that evident: while under the logarithm, there is no $T$ anymore, the power of the logarithm increases. In particular, the first term in the rate from Theorem 3.1 is proportional to $\\gamma^{\\frac{p}{p-1}} \\geq \\gamma^2$. If one wants to achieve $\\frac{1}{T}\\sum_{t=1}^T \\|\\nabla f(x_t)\\|^2 \\leq \\varepsilon$, then according to Theorem 3.1, the complexity bound will have a term proportional to $O(\\frac{\\log^{\\frac{p(3p-2)}{2p-2}}\\frac{1}{\\delta}}{\\varepsilon^{\\frac{3p-2}{2p-2}}})$. In the best case (when $p=2$), this equals to $O(\\frac{\\log^{4}\\frac{1}{\\delta}}{\\varepsilon^2})$. The corresponding term from (Sadiev et al., 2023) is $O(\\frac{\\log\\frac{1}{\\delta\\varepsilon^2}}{\\varepsilon^2})$. When both $\\delta$ and $\\varepsilon$ are small, $\\log\\frac{1}{\\delta\\varepsilon^2}$ can be much smaller than $\\log^{4}\\frac{1}{\\delta}$. I believe this should be discussed in the paper, and a fair comparison should be provided.\n\n2. **Logarithms in the horizon-independent results.** When $T$ is unknown, the results become worse by a polynomial factor of $\\log T$ that also spoils the complexity even in the convex case (and worse than for the case of known $T$ in the prior works).\n\n3. **Some parts of the proofs are not finished.** The proof of Lemma 3.4 is not finished. Next, although the result is believable, the proof of Theorem B.2 is not complete as well. The authors should at least provide a more detailed sketch explicitly pointing to the places that will be changed and how.\n\n4. **Results for the accelerated method requires $\\nabla f(x^\\ast) = 0$.** When the gradient at the solution equals zero, then the problem becomes almost an unconstrained one. (in terms of the analysis). The authors should indicate in the main text that the accelerated algorithm is analyzed under this additional assumption.\n\nQUESTIONS:\n1. Line 39, \"where the convergence rate depends on...\": the work [32] does not have high-probability results. Do you mean that one can get such results from the results of [32] using Markov's inequality?\n\n2. Line 165, \" The proof can be found in, for example, Lemma 1 of [16]\": It seems that the reference is not accurate (I did not find this result in the mentioned paper).\n\n3. Lines 447-448: what is $A'$?\n\n4. Line 480: what is the first case?\n\n5. I believe authors should provide the missing proofs (I mentioned them in the Weaknesses section).\n\n### Minor\n\n1. Line 204: step sizes are $\\eta_t$ are fixed $\\to$ step sizes $\\eta_t$ are fixed\n\n2. Lemma 3.4: the summation is forgotten in the third term of the main inequality.\n\n3. Line 430: should be Lemma 3.2 instead of Lemma 5\n\n4. Line 439: the second inequality holds with probability strictly smaller than $1$\n\n5. Lines 484-485, second and third rows: should be $\\lambda$ instead of $\\lambda_t$.\n\n6. Line 487: Lemma 3.6 $\\to$ Proposition 3.6\n\n7. After line 488, definitions of $C_2$ and $C_3$: should be \"$=$\" instead of \"$\\leq$\".\n\n8. After line 493: after the first inequality the square is missing.\n\nLIMITATIONS:\nThe authors address the limitations of their work in detail.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the high probability guarantees for clipped stochastic gradient descent with heavy-tailed noise, in the setting of Zhang et al. (2020) where the unbiased gradient noise has finite p-th moments, where $p\\in(1,2]$. The bound obtained in the paper is time-optimal and in the non-convex case, matches the lower bound in the literature. The analysis is based on the novel whitebox approach that analyzes the generating function of a well chosen martingale difference sequence to obtain tighter rates for stochastic gradient methods.\n\nSTRENGTHS:\nThe bound obtained in the paper is time-optimal and in the non-convex case, matches the lower bound in the literature. \n\nThe analysis has technical novelty that involves analyzing the generating function of a well chosen martingale difference sequence to obtain tighter rates for stochastic gradient methods.\n\nWEAKNESSES:\nIn terms of weakness, I think the first weakness is that some of the model assumptions seem to be very strong. For example, it is assumed that $\\mathbb{E}[\\Vert\\hat{\\nabla}f(x)-\\nabla f(x)\\Vert_{\\ast}^{p}|x]\\leq\\sigma^{p}$. This assumption does not seem to hold in the simple setting of mini-batch. Is it possible to extend your analysis to allow the relaxation of this assumption to something like $\\mathbb{E}[\\Vert\\hat{\\nabla}f(x)-\\nabla f(x)\\Vert_{\\ast}^{p}|x]\\leq\\sigma^{p}(1+\\Vert x\\Vert^{q})$ where $q$ can be related to $p$?\n\nIn your main result, i.e. Theorem 3.1., you mainly discuss the dependence on time $T$. What you obtained is $O(T^{\\frac{2-2p}{3p-2}})$. However, there are other terms with explicit constants in your bound, and how does your bound depend on $p$? Does it have monotonic dependence on $p\\in(1,2]$ or not? It seems to me that as $p\\rightarrow 1$, the upper bound you have depends on whether $8\\gamma/\\sqrt{L\\Delta_1}}$ and $\\sigma$ are bigger than $1$ or smaller than $1$ and you will get very different results in these two cases. Is this something you expected and is there any intuition behind it?\n\nQUESTIONS:\n(1)\tOn page 3, you cited some works about high probability convergence for noises with bounded variance and heavy tails. Are there some works on unbounded variance that you can cite?\n(2)\tOn page 6, line 234, can you provide a reference to Ville’s inequality?\n\nLIMITATIONS:\nI did not see such discussions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides theoretical guarantees for clipped gradient methods in the presence of heavy-tailed gradient noise distributions, where the noise has a bounded $p$th moment for some $1 < p \\leq 2$. The convergence guarantees are established with high probability. The authors' approach distinguishes itself from previous work by deviating from the classical techniques rely on using concentration inequalities coupled with an inductive union bound argument to control the iterates across all iterations. Instead, they bound the moment generating function of a well chosen supermartingale sequence which allows them to enhance the convergence guarantees for a wide range of clipped gradient based algorithm. The authors analyze Clipped-SGD algorithm in the smooth non convex setting, Clipped-SMD and Clipped-ASMD in the smooth convex setting. Specifically the rates they obtain are time-optimal and align with the latest lower-bounds, i.e., respectively $O(T^{\\frac{2-2p}{3p-2}})$, $O(T^{\\frac{1-p}{p}})$ and $O(T^{\\frac{1-p}{p}}\\sigma + T^{-2})$.\n\nSTRENGTHS:\nThe paper have a well-written and well-structured presentation. The authors stresses the importance of their work by giving many intuitions on their new proving techniques and give comparisons with related works.  The authors provide guarantees and parameter values for the three algorithms studied in the paper, both for a known time horizon $T$, and an unknown time horizon. The theoretical analysis sounds clear and easy to follow backed with insightful intuitions. The \"whitebox\" technic based on a well chosen supermatingale exhibits potential reusability for addressing other problems.\n\nWEAKNESSES:\nThe paper appears to be quite incremental and technical lacking of substantial novel contributions. The three algorithms studied in the paper are already well-established methods, and aside from the application of the \"whitebox\" analysis technique, the convergence analysis for the algorithms follows conventional approaches commonly employed for clipped-gradient-based methods. Furthermore, the analysis of Clipped-SMD heavily depends on the hypothesis that there exists a bound $\\nabla_1$ on the gradient at the initial point i.e., $ \\lVert \\nabla f(x_1) \\rVert_* $ $ \\leq \\nabla_1$. If the optimal point $x_*$ does not lie in the domain $ \\lVert\\nabla f(x_1) \\rVert_*$  can be estimated w.h.p but requires the knowledge of $\\sigma$ which is quite restrictive.\n\nQUESTIONS:\nQ1) Is it possible to extend the \"whitebox\" technic to the analysis of the Clipped-SGD with momentum in the non-convex setting?\n\nQ2) How the studied algorithms compares practically with the parameters in the paper against existing state-of-the-art clipped gradient methods?\n\nLIMITATIONS:\nI think that the guarantees on the accelerated stochastic mirror descent algorithm should be added to the main body of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors consider clipped stochastic gradient methods: clipped-SMD in convex case (in non-convex case the y consider clipped-SGD), clipped-ASMD in convex case. They provided improved high-probability convergence analysis, which has some similar steps with analysis from (Gorbunov et.al 2020, Sadiev et.al 2023). Compare to this works, the authors provided analysis in non-Euclidean setup for clipped-SMD, clipped-ASMD. Also, they explain why the proposed approach works better than the previous one.\n\nSTRENGTHS:\n1. A good explanation of the difference between blackbox and whitebox approaches in a detailed way.\n2. The authors provide the convergence analysis of the proposed methods in a non-euclidian setup. \n\nGenerally, the paper is well-written and has good results.\n\nWEAKNESSES:\n1. There are some issues dealing with clipped-ASMD. At the beginning of the analysis, they consider a more general assumption on the smoothness of $f$. But in the end, they use only standard smoothness assumption. Also, compare to clipped-SMD, they assume that $f(x^*) = 0$.\n2. From my perspective, it would be better, if the proofs of all facts were provided in a more detailed way.\n\nQUESTIONS:\n1.  Could you explain why $z_k$ is decreasing sequence? It is not clear to me. \n2.  About extension to non-smooth setting: How exactly could it be done? According to the proof and the final statement of the main fact about complexity, there is no constant $G$ in the final complexity. Could you comment on it?\n3. The proof via blackbox approach is supported with a lack of explanation for some details, could you provide a full proof or the citation on the same proof technique? It is similar to the proof from the paper of Sadiev et.al, 2023. \n4. Please, could you provide a small explanation for the third inequality in the last chain of inequality on page 19? \n5. For inequality (d) on page 20, you did not write any explanation. Did you forget to do it?\n\nTypos:\n1. On line 204, the first 'are' is not needed.\n2. In eq. (6), it will look better, If '(6)' is on the same row as expression. \n3. On line 430, maybe, it would be better to write 'Lemma 3.2' or 'eq. (5)' instead of 'Lemma 5'\n4. On lines 545 and 612, 'Proposition' is missed before '4.8'.\n\nLIMITATIONS:\nThere are no limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}