Gradient Clipping Helps in Non-Smooth Stochastic Optimization with Heavy-Tailed Noise

Thanks to their practical efficiency and random nature of the data , stochastic1 first-order methods are standard for training large-scale machine learning models.2 Random behavior may cause a particular run of an algorithm to result in a highly3 suboptimal objective value , whereas theoretical guarantees are usually proved4 for the expectation of the objective value . Thus , it is essential to theoretically5 guarantee that algorithms provide small objective residual with high probability.6 Existing methods for non-smooth stochastic convex optimization have complexity7 bounds with the dependence on the confidence level that is either negative-power or8 logarithmic but under an additional assumption of sub-Gaussian ( light-tailed ) noise9 distribution that may not hold in practice , e.g. , in several NLP tasks . In our paper,10 we resolve this issue and derive the first high-probability convergence results with11 logarithmic dependence on the confidence level for non-smooth convex stochastic12 optimization problems with non-sub-Gaussian ( heavy-tailed ) noise . To derive our13 results , we propose novel stepsize rules for two stochastic methods with gradient14 clipping . Moreover , our analysis works for generalized smooth objectives with15 Hölder-continuous gradients , and for both methods , we provide an extension for16 strongly convex problems . Finally , our results imply that the first ( accelerated ) 17 method we consider also has optimal iteration and oracle complexity in all the18 regimes , and the second one is optimal in the non-smooth setting.19 1 Introduction20 Stochastic first-order optimization methods like SGD [ 32 ] , Adam [ 20 ] , and their various modifi-21 cations are extremely popular in solving a number of different optimization problems , especially22 those appearing in statistics [ 36 ] , machine learning , and deep learning [ 13 ] . The success of these23 methods in real-world applications motivates the researchers to investigate theoretical properties24 for the methods and to develop new ones with better convergence guarantees . Typically , stochastic25 methods are analyzed in terms of the convergence in expectation ( see [ 12 , 24 , 15 ] and references26 therein ) , whereas high-probability complexity results are established much rarely . However , as27 illustrated in [ 14 ] , guarantees in terms of the convergence in expectation have much worse correlation28 with the real behavior of the methods than high-probability convergence guarantees when the noise29 in the stochastic gradients has heavy-tailed distribution.30 Recent studies [ 35 , 34 , 41 ] show that in several popular problems such as training BERT [ 37 ] on31 Wikipedia dataset the noise in the stochastic gradients is heavy-tailed . Moreover , in [ 41 ] , the authors32 justify empirically that in such cases SGD works significantly worse than clipped-SGD [ 30 ] and33 Adam . Therefore , it is important to theoretically study the methods ’ convergence when the noise is34 heavy-tailed.35 Submitted to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Do not distribute . For convex and strongly convex problems with Lipschitz continuous gradient , i.e. , smooth convex and36 strongly convex problems , this question was properly addressed in [ 25 , 3 , 14 ] where the first high-37 probability complexity bounds with logarithmic dependence on the confidence level were derived38 for the stochastic problems with heavy-tailed noise . However , a number of practically important39 problems are non-smooth on the whole space [ 40 , 22 ] . For example , in deep neural network training,40 the loss function often grows polynomially fast when the norm of the network ’ s weights goes to41 infinity . Moreover , non-smoothness of the activation functions such as ReLU or loss functions such42 as hinge loss implies the non-smoothness of the whole problem . While being well-motivated by43 practical applications , the existing high-probability convergence guarantees for stochastic first-order44 methods applied to solve non-smooth convex optimization problems with heavy-tailed noise depend45 on the negative power of the confidence level that dramatically increases the number of iterations46 required to obtain high accuracy of the solution with probability close to one . Such a discrepancy in47 the theory between algorithms for stochastic smooth and non-smooth problems leads us to the natural48 question : is it possible to obtain high-probability complexity bounds with logarithmic dependence49 on the confidence level for non-smooth convex stochastic problems with heavy-tailed noise ? In this50 paper , we give a positive answer to this question . To achieve this we focus on gradient clipping51 methods [ 30 , 10 , 23 , 22 , 40 , 41 ] .52 1.1 Preliminaries53 Before we describe our contributions in detail , we formally state the considered setup.54 Stochastic optimization . We focus on the following problem55 min x∈Rn f ( x ) , f ( x ) = Eξ [ f ( x , ξ ) ] , ( 1 ) where f ( x ) is a convex but possibly non-smooth function . Next , we assume that at each point x ∈ Rn56 we have an access to the unbiased estimator∇f ( x , ξ ) of ∇f ( x ) with uniformly bounded variance57 Eξ [ ∇f ( x , ξ ) ] = ∇f ( x ) , Eξ [ ‖∇f ( x , ξ ) −∇f ( x ) ‖22 ] ≤ σ2 , σ > 0 . ( 2 ) This assumption on the stochastic oracle is widely used in stochastic optimization literature [ 11,58 12 , 19 , 21 , 26 ] . We emphasize that we do not assume that the stochastic gradients have so-called59 “ light tails ” [ 21 ] , i.e. , sub-Gaussian noise distribution meaning that P { ‖∇f ( x , ξ ) −∇f ( x ) ‖2 > b } ≤60 2 exp ( −b2/ ( 2σ2 ) ) for all b > 0.61 Level of smoothness . Finally , we assume that function f has ( ν , Mν ) -Hölder continuous gradients62 on a compact set Q ⊆ Rn for some ν ∈ [ 0 , 1 ] , Mν > 0 meaning that63 ‖∇f ( x ) −∇f ( y ) ‖2 ≤Mν‖x− y‖ν2 ∀x , y ∈ Q . ( 3 ) When ν = 1 inequality ( 3 ) implies M1-smoothness of f , and when ν = 0 we have that ∇f ( x ) 64 has bounded variation which is equivalent to being uniformly bounded . Moreover , when ν = 065 differentiability of f is not needed , and one can assume uniform boundedness of the subgradients of66 f . Linear regression in the case when the noise has generalized Gaussian distribution ( Example 4.467 from [ 2 ] ) serves as a natural example of the situation with ν ∈ ( 0 , 1 ) . Moreover , when ( 3 ) holds for68 ν = 0 and ν = 1 simultaneously then it holds for all ν ∈ [ 0 , 1 ] with Mν ≤M1−ν0 Mν1 [ 28 ] . As we69 show in our results , the set Q should contain the ball centered at the solution x∗ of ( 1 ) with radius70 2R0 = 2‖x0 − x∗‖2 , where x0 is a starting point of the method , i.e. , our analysis does not require ( 3 ) 71 to hold on Rn.72 High-probability convergence . For a given accuracy ε > 0 and confidence level β ∈ ( 0 , 1 ) we73 are interested in finding ε-solutions of problem ( 1 ) with probability at least 1− β , i.e. , such x̂ that74 P { f ( x̂ ) − f ( x∗ ) ≤ ε } ≥ 1 − β . For brevity , we will call such ( in general , random ) points x̂ as75 ( ε , β ) -solution of ( 1 ) . Moreover , by high-probability complexity of a stochastic methodM we mean76 the sufficient number of oracle calls , i.e. , number of∇f ( x , ξ ) computations , needed to guarantee that77 the output ofM is an ( ε , β ) -solution of ( 1 ) .78 high-probability complexity . Unlike all previous high-probability complexity results in this setup84 with ν < 1 ( see Tbl . 1 ) , our result depends only logarithmically on the confidence level β that85 is highly important when β is small . Moreover , up to the difference in logarithmic factors the86 derived complexity guarantees meet the known lower bounds [ 21 , 17 ] obtained for the problems87 with light-tailed noise . In particular , when ν = 1 we recover accelerated convergence rate [ 29 , 21 ] .88 That is , neglecting the logarithmic factors our results are unimprovable and , surprisingly coincide89 with the best-known results in the “ light-tailed case ” .90 • We derive the first high-probability complexity bounds for clipped-SGD when the objective91 functions is convex with ( ν , Mν ) -Hölder continuous gradient and the noise is heavy tailed. , i.e. , we92 derive93 O ( max { D2 , max { D1+ν , σ2R20 ε2 } ln D2 +D1+ν β } ) , D = M 1 1+ν ν R0 ε 1 1+ν high-probability complexity bound . Interestingly , when ν = 0 the derived bound for clipped-SGD94 has better dependence on the logarithms than the corresponding one for clipped-SSTM . Moreover,95 neglecting the dependence on ε under the logarithm , our bound for clipped-SGD has the same96 • Using restarts technique we extend the obtained results for clipped-SSTM and clipped-SGD to99 the strongly convex case ( see Tbl . 2 ) . As in the convex case , the obtained results are superior to all100 previous known results in the general setup we consider.101 • As one of the key contributions of this work , we emphasize that in our theoretical results it is102 sufficient to assume Hölder continuity of the gradients of f only on the ball with radius 2R0 =103 2‖x0 − x∗‖2 and centered at a solution of the problem . This makes our results applicable to much104 larger class of problems than functions with Hölder continuous gradients on Rn , e.g. , our analysis105 works even for polynomially growing objectives.106 • To test the performance of the considered methods we conduct several numerical experiments107 on image classification and NLP tasks , and observe that 1 ) clipped-SSTM and clipped-SGD108 show a comparable performance with SGD on the image classification task , when the noise109 distribution is almost sub-Gaussian , 2 ) converge much faster than SGD on the NLP task , when the110 noise distribution is heavy-tailed , and 3 ) clipped-SSTM achieves a comparable performance with111 Adam on the NLP task enjoying both the best known theoretical guarantees and good practical112 performance.113 1.3 Related work114 Light-tailed noise . The theory of high-probability complexity bounds for convex stochastic op-115 timization with light-tailed noise is well-developed . Lower bounds and optimal methods for the116 problems with ( ν , Mν ) -Hölder continuous gradients are obtained in [ 26 ] for ν = 0 , and in [ 11 ] for117 ν = 1 . Up to the logarithmic dependencies these high-probability convergence bounds coincide with118 the corresponding results for the convergence in expectation ( see first two rows of Tbl . 1 ) While not119 being directly derived in the literature , the lower bound for the case when ν ∈ ( 0 , 1 ) can be obtained120 as a combination of lower bounds in the deterministic [ 27 , 17 ] and smooth stochastic settings [ 11 ] .121 The corresponding optimal methods are analyzed in [ 4 , 6 ] through the lens of inexact oracle.122 Heavy-tailed noise . Unlike in the “ light-tailed ” case , the first theoretical guarantees with reasonable123 dependence on both the accuracy ε and the confidence level β appeared just recently . In [ 25 ] , the124 first such results without acceleration [ 29 ] were derived for Mirror Descent with special truncation125 technique for smooth ( ν = 1 ) convex problems on a bounded domain , and then were accelerated and126 extended in [ 14 ] . For the strongly convex problems the first accelerated high-probability convergence127 guarantees were obtained in [ 3 ] for the special method called proxBoost requiring solving auxiliary128 nontrivial problems at each iteration . These bounds were tightened in [ 14 ] .129 In contrast , for the case when ν < 1 and , in particular , when ν = 0 the best-known high-probability130 complexity bounds suffer from the negative-power dependence on the confidence level β , i.e. , have131 a factor 1/βα for some α > 0 , that affects the convergence rate dramatically for small enough132 β . Without additional assumptions on the tails these results are obtained via Markov ’ s inequality133 P { f ( x ) − f ( x∗ ) > ε } < E [ f ( x ) −f ( x∗ ) ] /ε from the guarantees for the convergence in expectation to134 the accuracy εβ , see the results labeled by ♣ in Tbl . 1 . Under an additional assumption on noise135 tails that P { ‖∇f ( x , ξ ) −∇f ( x ) ‖22 > sσ2 } = O ( s−α ) for α > 2 these results can be tightened [ 9 ] 136 when ν = 0 as O ( M20R 2 0 max { ln ( β−1 ) /ε2 , ( 1/βεα ) 2/ ( 3α−2 ) } ) without removing the negative-power137 dependence on the confidence level β . Different stepsize policies allow to change the last term in138 max to β− 1 2α−1 ε− 2α 2α−1 without removing the negative-power dependence on β.139 Gradient clipping . The methods based on gradient clipping [ 30 ] and normalization [ 18 ] are popular140 in different machine learning and deep learning tasks due to their robustness in practice to the noise141 in the stochastic gradients and rapid changes of the objective function [ 13 ] . In [ 40 , 22 ] , clipped-GD142 and clipped-SGD are theoretically studied in applications to non-smooth problems that can grow143 polynomially fast when ‖x − x∗‖2 → ∞ showing the superiority of gradient clipping methods144 to the methods without clipping . The results from [ 40 ] are obtained for non-convex problems145 with almost surely bounded noise , and in [ 22 ] , the authors derive the stability and expectation146 convergence guarantees for strongly convex under assumption that the central p-th moment of the147 stochastic gradient is bounded for p ≥ 2 . Since the authors of [ 22 ] do not provide convergence148 guarantees with explicit dependencies on all important parameters of the problem it complicates direct149 comparison with our results . Nevertheless , convergence guarantees from [ 22 ] are sub-linear and are150 given for the convergence in expectation , and , as a consequence , the corresponding high-probability151 convergence results obtained via Markov ’ s inequality also suffer from negative-power dependence on152 the confidence level . Next , in [ 41 ] , the authors establish several expectation convergence guarantees153 for clipped-SGD and prove their optimality in the non-convex case under assumption that the central154 α-moment of the stochastic gradient is uniformly bounded , where α ∈ ( 1 , 2 ] . It turns out that155 clipped-SGD is able to converge even when α < 2 , whereas vanilla SGD can diverge in this setting.156 2 Clipped Stochastic Similar Triangles Method157 In this section , we propose a novel variation of Clipped Stochastic Similar Triangles Method [ 14 ] 158 adjusted to the class of objectives with Hölder continuous gradients ( clipped-SSTM , see Alg . 1 ) .159 The method is based on the clipping of the stochastic gradients:160 clip ( ∇f ( x , ξ ) , λ ) = min { 1 , λ ‖∇f ( x , ξ ) ‖2 } ∇f ( x , ξ ) ( 4 ) where ∇f ( x , ξ ) = 1m ∑m i=1∇f ( x , ξi ) is a mini-batched stochastic gradient . Gradient clipping161 ensures that the resulting vector has a norm bounded by the clipping level λ . Since the clipped162 stochastic gradient can not have arbitrary large norm , the clipping helps to avoid unstable behavior of163 the method when the noise is heavy-tailed and the clipping level λ is properly adjusted.164 However , unlike the stochastic gradient , clipped stochastic gradient is a biased estimate of ∇f ( x ) :165 the smaller the clipping level the larger the bias . The biasedness of the clipped stochastic gradient166 Algorithm 1 Clipped Stochastic Similar Triangles Method ( clipped-SSTM ) : case ν ∈ [ 0 , 1 ] Input : starting point x0 , number of iterationsN , batchsizes { mk } Nk=1 , stepsize parameter α , clipping parameter B , Hölder exponent ν ∈ [ 0 , 1 ] . 1 : Set A0 = α0 = 0 , y0 = z0 = x0 2 : for k = 0 , . . . , N − 1 do 3 : Set αk+1 = α ( k + 1 ) 2ν 1+ν , Ak+1 = Ak + αk+1 , λk+1 = Bαk+1 4 : xk+1 = ( Akyk+αk+1zk ) /Ak+1 5 : Draw mini-batch mk of fresh i.i.d . samples ξk1 , . . . , ξ k mk and compute ∇f ( xk+1 , ξk ) = 1 mk ∑mk i=1∇f ( xk+1 , ξki ) 6 : Compute ∇̃f ( xk+1 , ξk ) = clip ( ∇f ( xk+1 , ξk ) , λk+1 ) using ( 4 ) 7 : zk+1 = zk − αk+1∇̃f ( xk+1 , ξk ) 8 : yk+1 = ( Akyk+αk+1zk+1 ) /Ak+1 9 : end for Output : yN complicates the analysis of the method . On the other hand , to circumvent the negative effect of167 the heavy-tailed noise on the high-probability convergence one should choose λ to be not too large.168 Therefore , the question on the appropriate choice of the clipping level is highly non-trivial.169 Fortunately , there exists a simple but insightful observation that helps us to obtain the right formula170 for the clipping level λk in clipped-SSTM : if λk is chosen in such a way that ‖∇f ( xk ) ‖2 ≤ λk/2171 with high probability , then for the realizations ∇f ( xk+1 , ξk ) of the mini-batched stochastic gradient172 such that ‖∇f ( xk+1 , ξk ) − ∇f ( xk+1 ) ‖2 ≤ λk/2 the clipping is an identity operator . Next , if the173 probability mass of such realizations is big enough then the bias of the clipped stochastic gradient is174 properly bounded that helps to derive needed convergence guarantees . It turns out that the choice175 λk ∼ 1/αk ensures the method convergence with needed rate and high enough probability.176 Guided by this observation we derive the precise expressions for all the parameters of clipped-SSTM177 and derive high-probability complexity bounds for the method . Below we provide a simplified version178 of the main result for clipped-SSTM in the convex case . The complete formulation and the full proof179 of the theorem are deferred to Appendix B.1 ( see Thm . B.1 ) .180 Theorem 2.1 . Assume that function f is convex and its gradient satisfy ( 3 ) with ν ∈ [ 0 , 1 ] , Mν > 0181 on Q = B2R0 = { x ∈ Rn | ‖x − x∗‖2 ≤ 2R0 } , where R0 ≥ ‖x0 − x∗‖2 . Then there exist such182 a choice of parameters that clipped-SSTM achieves f ( yN ) − f ( x∗ ) ≤ ε with probability at least183 1− β after O ( D ln 2 ( 1+ν ) 1+3ν D β ) iterations with D = M 2 1+3ν ν R 2 ( 1+ν ) 1+3ν 0 ε 2 1+3ν and requires184 O ( max { D ln 2 ( 1+ν ) 1+3ν D β , σ2R20 ε2 ln D β } ) oracle calls . ( 5 ) The obtained result has only logarithmic dependence on the confidence level β and optimal depen-185 dence on the accuracy ε up to logarithmic factors [ 21 , 17 ] for all ν ∈ [ 0 , 1 ] . Moreover , we emphasize186 that our result does not require f to have ( ν , Mν ) -Hölder continuous gradient on the whole space.187 This is because we prove that for the proposed choice of parameters the iterates of clipped-SSTM188 stay inside the ball B2R0 = { x ∈ Rn | ‖x − x∗‖2 ≤ 2R0 } with probability at least 1 − β , and,189 as a consequence , Hölder continuity of the gradient is required only inside this ball . In particular,190 this means that the better starting point leads not only to the reduction of R0 , but also it can reduce191 Mν . Moreover , our result is applicable to much wider class of functions than the standard class of192 functions with Hölder continuous gradients in Rn , e.g. , to the problems with polynomial growth.193 For the strongly convex problems , we consider restarted version of Alg . 1 ( R-clipped-SSTM , see194 Alg . 2 ) and derive high-probability complexity result for this version . Below we provide a simplified195 version of the result . The complete formulation and the full proof of the theorem are deferred to196 Appendix B.2 ( see Thm . B.2 ) .197 Theorem 2.2 . Assume that function f is µ-strongly convex and its gradient satisfy ( 3 ) with ν ∈ [ 0 , 1 ] ,198 Mν > 0 on Q = B2R0 = { x ∈ Rn | ‖x− x∗‖2 ≤ 2R0 } , where R0 ≥ ‖x0 − x∗‖2 . Then there exist199 Algorithm 2 Restarted clipped-SSTM ( R-clipped-SSTM ) : case ν ∈ [ 0 , 1 ] Input : starting point x0 , number of restarts τ , number of steps of clipped-SSTM in restarts { Nt } τt=1 , batchsizes { m1k } N1−1 k=1 , { m2k } N2−1 k=1 , . . . , { mτk } Nτ−1 k=1 , stepsize parameters { αt } τt=1 , clipping pa- rameters { Bt } τt=1 , Hölder exponent ν ∈ [ 0 , 1 ] . 1 : x̂0 = x0 2 : for t = 1 , . . . , τ do 3 : Run clipped-SSTM ( Alg . 1 ) for Nt iterations with batchsizes { mtk } Nt−1 k=1 , stepsize parameter αt , clipping parameter Bt , and starting point x̂t−1 . Define the output of clipped-SSTM by x̂t . 4 : end for Output : x̂τ such a choice of parameters that R-clipped-SSTM achieves f ( x̂τ ) − f ( x∗ ) ≤ ε with probability at200 least 1− β after201 N̂ = O ( D ln 2 ( 1+ν ) 1+3ν D β ) , D = max { ( Mν µR1−ν0 ) 2 1+3ν ln µR20 ε , ( M2ν µ1+νε1−ν ) 1 1+3ν } ( 6 ) iterations of Alg . 1 in total and requires202 O ( max { D ln 2 ( 1+ν ) 1+3ν D β , σ2 µε ln D β } ) oracle calls . ( 7 ) Again , the obtained result has only logarithmic dependence on the confidence level β and , as our203 result in the convex case , it has optimal dependence on the accuracy ε up to logarithmic factors204 depending on β [ 21 , 17 ] for all ν ∈ [ 0 , 1 ] .205 3 SGD with clipping206 In this section , we present a new variant of clipped-SGD [ 30 ] properly adjusted to the class of207 objectives with ( ν , Mν ) -Hölder continuous gradients ( see Alg . 3 ) .208 Algorithm 3 Clipped Stochastic Gradient Descent ( clipped-SGD ) : case ν ∈ [ 0 , 1 ] Input : starting point x0 , number of iterations N , batchsize m , stepsize γ , clipping parameter B > 0 . 1 : for k = 0 , . . . , N − 1 do 2 : Draw mini-batch of m fresh i.i.d . samples ξk1 , . . . , ξ k m and compute ∇f ( xk+1 , ξ k ) = 1 m ∑m i=1∇f ( xk+1 , ξki ) 3 : Compute ∇̃f ( xk , ξk ) = clip ( ∇f ( xk , ξk ) , λ ) using ( 4 ) with λ = B/γ 4 : xk+1 = xk − γ∇̃f ( xk , ξk ) 5 : end for Output : x̄N = 1N ∑N−1 k=0 x k We emphasize that as for clipped-SSTM we use clipping level λ inversely proportional to the stepsize209 γ . Below we provide a simplified version of the main result for clipped-SGD in the convex case . The210 complete formulation and the full proof of the theorem are deferred to Appendix C.1 ( see Thm . C.1 ) .211 Theorem 3.1 . Assume that function f is convex and its gradient satisfy ( 3 ) with ν ∈ [ 0 , 1 ] , Mν > 0212 on Q = B2R0 = { x ∈ Rn | ‖x− x∗‖2 ≤ 2R0 } , where R0 ≥ ‖x0 − x∗‖2 . Then there exist such a213 choice of parameters that clipped-SGD achieves f ( x̄N ) − f ( x∗ ) ≤ ε with probability at least 1− β214 after215 O ( max { D2 , D1+ν ln D2 +D1+ν β } ) , D = M 1 1+ν ν R0 ε 1 1+ν ( 8 ) iterations and requires216 O ( max { D2 , max { D1+ν , σ2R20 ε2 } ln D2 +D1+ν β } ) oracle calls . ( 9 ) As all our results in the paper , this result for clipped-SGD has two important features : 1 ) the217 dependence on the confidence level β is logarithmic and 2 ) Hölder continuity is required only on218 the ball B2R0 centered at the solution . Moreover , up to the difference in the expressions under219 the logarithm the dependence on ε in the result for clipped-SGD is the same as in the tightest220 known results for non-accelerated SGD-type methods [ 4 , 16 ] . Finally , we emphasize that for ν < 1221 the logarithmic factors appearing in the complexity bound for clipped-SSTM are worse than the222 corresponding factor in the complexity bound for clipped-SGD . Therefore , clipped-SGD has the223 best known high-probability complexity results in the case when ν = 0 and f is convex.224 For the strongly convex problems , we consider restarted version of Alg . 3 ( R-clipped-SGD , see225 Alg . 4 ) and derive high-probability complexity result for this version . Below we provide a simplified Algorithm 4 Restarted clipped-SGD ( R-clipped-SGD ) : case ν ∈ [ 0 , 1 ] Input : starting point x0 , number of restarts τ , number of steps of clipped-SGD in restarts { Nt } τt=1 , batchsizes { mt } τk=1 , stepsizes { γt } τt=1 , clipping parameters { Bt } τt=1 1 : x̂0 = x0 2 : for t = 1 , . . . , τ do 3 : Run clipped-SGD ( Alg . 3 ) for Nt iterations with batchsize mt , stepsize γt , clipping parame- ter Bt , and starting point x̂t−1 . Define the output of clipped-SGD by x̂t . 4 : end for Output : x̂τ 226 version of the result . The complete formulation and the full proof of the theorem are deferred to227 Appendix C.2 ( see Thm . C.2 ) .228 Theorem 3.2 . Assume that function f is µ-strongly convex and its gradient satisfy ( 3 ) with ν ∈ [ 0 , 1 ] ,229 Mν > 0 on Q = B2R0 = { x ∈ Rn | ‖x− x∗‖2 ≤ 2R0 } , where R0 ≥ ‖x0 − x∗‖2 . Then there exist230 such a choice of parameters that R-clipped-SGD achieves f ( x̄N ) − f ( x∗ ) ≤ ε with probability at231 least 1− β after232 O ( max { D 2 1+ν 1 ln µR20 ε , D 2 1+ν 2 , max { D1 ln µR20 ε , D2 } ln D β } ) iterations of Alg . 3 in total and requires233 O ( max { D 2 1+ν 1 ln µR20 ε , D 2 1+ν 2 , max { D1 ln µR20 ε , D2 , σ2 µε } ln D β } ) oracle calls , where 234 D1 = Mν µR1−ν0 , D2 = Mν µ 1+ν 2 ε 1−ν 2 , D = ( D 2 1+ν 1 +D1 ) ln µR20 ε +D2 +D 2 1+ν 2 . As in the convex case , for ν < 1 the log factors appearing in the complexity bound for R-clipped-235 SSTM are worse than the corresponding factor in the bound for R-clipped-SGD . Thus , R-clipped-236 SGD has the best known high-probability complexity results for strongly convex f and ν = 0.237 4 Numerical experiments238 We tested the performance of the methods on the following problems:239 • BERT fine-tuning on CoLA dataset [ 38 ] . We use pretrained BERT from Transformers library [ 39 ] 240 ( bert-base-uncased ) and freeze all layers except the last two linear ones.241 • ResNet-18 training on ImageNet-100 ( first 100 classes of ImageNet [ 33 ] ) .242 First , we study the noise distribution for both problem as follows : at the starting point we sample243 large enough number of batched stochastic gradients ∇f ( x0 , ξ1 ) , . . . , ∇f ( x0 , ξK ) with batchsize244 32 and plot the histograms for ‖∇f ( x0 , ξ1 ) −∇f ( x0 ) ‖2 , . . . , ‖∇f ( x0 , ξK ) −∇f ( x0 ) ‖2 , see Fig . 1.245 As one can see , the noise distribution for BERT + CoLA is substantially non-sub-Gaussian , whereas246 the distribution for ResNet-18 + Imagenet-100 is almost Gaussian.247 Next , we compared 4 different optimizers on these problems : Adam , SGD ( with Momentum ) ,248 clipped-SGD ( with Momentum and coordinate-wise clipping ) and clipped-SSTM ( with norm-249 clipping and ν = 1 ) . The results are presented in Fig . 2 . We observed that the noise distributions do250 not change significantly along the trajectories of the considered methods , see Appendix D. During251 the hyper-parameters search we compared different batchsizes , emulated via gradient accumulation252 ( thus we compare methods with different batchsizes by the number of base batches used ) . The base253 batchsize was 32 for both problems , stepsizes and clipping levels were tuned . One can find additional254 details regarding our experiments in Appendix D.255 Image classification . On ResNet-18 + ImageNet-100 task , SGD performs relatively well , and256 even ties with Adam ( with batchsize of 4× 32 ) in validation loss . clipped-SSTM ( with batchsize of257 2× 32 ) also ties with Adam and clipped-SGD is not far from them . The results were averaged from258 5 different launches ( with different starting points/weight initializations ) . Since the noise distribution259 is almost Gaussian even vanilla SGD performs well , i.e. , gradient clipping is not required . At the260 same time , the clipping does not slow down the convergence significantly.261 Text classification . On BERT + CoLA task , when the noise distribution is heavy-tailed , the methods262 with clipping outperform SGD by a large margin . This result is in good correspondence with the263 derived high-probability complexity bounds for clipped-SGD , clipped-SSTM and the best-known264 ones for SGD . Moreover , clipped-SSTM ( with batchsize of 8 × 32 ) achieves the same loss on265 validation as Adam , and has better accuracy . These results were averaged from 5 different train-val266 splits and 20 launches ( with different starting points/weight initializations ) for each of the splits , 100267 launches in total.268 References269 [ 1 ] George Bennett . Probability inequalities for the sum of independent random variables . Journal270 of the American Statistical Association , 57 ( 297 ) :33–45 , 1962.271 [ 2 ] Caroline Chaux , Patrick L Combettes , Jean-Christophe Pesquet , and Valérie R Wajs . A varia-272 tional formulation for frame-based inverse problems . Inverse Problems , 23 ( 4 ) :1495–1518 , jun273 2007.274 [ 3 ] Damek Davis , Dmitriy Drusvyatskiy , Lin Xiao , and Junyu Zhang . From low probability to275 high confidence in stochastic convex optimization . Journal of Machine Learning Research,276 22 ( 49 ) :1–38 , 2021.277 [ 4 ] Olivier Devolder . Exactness , inexactness and stochasticity in first-order methods for large-scale278 convex optimization . PhD thesis , PhD thesis , 2013.279 [ 5 ] Olivier Devolder , François Glineur , and Yurii Nesterov . First-order methods of smooth convex280 optimization with inexact oracle . Mathematical Programming , 146 ( 1 ) :37–75 , 2014.281 [ 6 ] Pavel Dvurechensky and Alexander Gasnikov . Stochastic intermediate gradient method for con-282 vex problems with stochastic inexact oracle . Journal of Optimization Theory and Applications,283 171 ( 1 ) :121–145 , 2016.284 [ 7 ] Kacha Dzhaparidze and JH Van Zanten . On bernstein-type inequalities for martingales . Stochas-285 tic processes and their applications , 93 ( 1 ) :109–117 , 2001.286 [ 8 ] David A Freedman et al . On tail probabilities for martingales . the Annals of Probability,287 3 ( 1 ) :100–118 , 1975.288 [ 9 ] Alexander Vladimirovich Gasnikov , Yu E Nesterov , and Vladimir Grigor ’ evich Spokoiny.289 On the efficiency of a randomized mirror descent algorithm in online optimization problems.290 Computational Mathematics and Mathematical Physics , 55 ( 4 ) :580–596 , 2015.291 [ 10 ] Jonas Gehring , Michael Auli , David Grangier , Denis Yarats , and Yann N Dauphin . Convolutional292 sequence to sequence learning . In Proceedings of the 34th International Conference on Machine293 Learning-Volume 70 , pages 1243–1252 . JMLR . org , 2017.294 [ 11 ] Saeed Ghadimi and Guanghui Lan . Optimal stochastic approximation algorithms for strongly295 convex stochastic composite optimization i : A generic algorithmic framework . SIAM Journal296 on Optimization , 22 ( 4 ) :1469–1492 , 2012.297 [ 12 ] Saeed Ghadimi and Guanghui Lan . Stochastic first-and zeroth-order methods for nonconvex298 stochastic programming . SIAM Journal on Optimization , 23 ( 4 ) :2341–2368 , 2013.299 [ 13 ] Ian Goodfellow , Yoshua Bengio , and Aaron Courville . Deep Learning . MIT Press , 2016.300 http : //www.deeplearningbook.org.301 [ 14 ] Eduard Gorbunov , Marina Danilova , and Alexander Gasnikov . Stochastic optimization with302 heavy-tailed noise via accelerated gradient clipping . In H. Larochelle , M. Ranzato , R. Had-303 sell , M. F. Balcan , and H. Lin , editors , Advances in Neural Information Processing Systems,304 volume 33 , pages 15042–15053 . Curran Associates , Inc. , 2020.305 [ 15 ] Robert Mansel Gower , Nicolas Loizou , Xun Qian , Alibek Sailanbayev , Egor Shulgin , and Peter306 Richtárik . Sgd : General analysis and improved rates . In International Conference on Machine307 Learning , pages 5200–5209 , 2019.308 [ 16 ] Vincent Guigues , Anatoli Juditsky , and Arkadi Nemirovski . Non-asymptotic confidence bounds309 for the optimal value of a stochastic program . Optimization Methods and Software , 32 ( 5 ) :1033–310 1058 , 2017.311 [ 17 ] Cristóbal Guzmán and Arkadi Nemirovski . On lower complexity bounds for large-scale smooth312 convex optimization . Journal of Complexity , 31 ( 1 ) :1–14 , 2015.313 [ 18 ] Elad Hazan , Kfir Levy , and Shai Shalev-Shwartz . Beyond convexity : Stochastic quasi-convex314 optimization . In Advances in Neural Information Processing Systems , pages 1594–1602 , 2015.315 [ 19 ] Anatoli Juditsky , Arkadi Nemirovski , et al . First order methods for nonsmooth convex large-316 scale optimization , i : general purpose methods . Optimization for Machine Learning , pages317 121–148 , 2011.318 [ 20 ] Diederik P Kingma and Jimmy Ba . Adam : A method for stochastic optimization . arXiv preprint319 arXiv:1412.6980 , 2014.320 [ 21 ] Guanghui Lan . An optimal method for stochastic composite optimization . Mathematical321 Programming , 133 ( 1-2 ) :365–397 , 2012.322 [ 22 ] Vien V Mai and Mikael Johansson . Stability and convergence of stochastic gradient clipping:323 Beyond lipschitz continuity and smoothness . arXiv preprint arXiv:2102.06489 , 2021.324 [ 23 ] Aditya Krishna Menon , Ankit Singh Rawat , Sashank J Reddi , and Sanjiv Kumar . Can gradient325 clipping mitigate label noise ? In International Conference on Learning Representations , 2020.326 [ 24 ] Eric Moulines and Francis R Bach . Non-asymptotic analysis of stochastic approximation327 algorithms for machine learning . In Advances in Neural Information Processing Systems , pages328 451–459 , 2011.329 [ 25 ] Aleksandr Viktorovich Nazin , AS Nemirovsky , Aleksandr Borisovich Tsybakov , and AB Judit-330 sky . Algorithms of robust stochastic optimization based on mirror descent method . Automation331 and Remote Control , 80 ( 9 ) :1607–1627 , 2019.332 [ 26 ] Arkadi Nemirovski , Anatoli Juditsky , Guanghui Lan , and Alexander Shapiro . Robust stochastic333 approximation approach to stochastic programming . SIAM Journal on optimization , 19 ( 4 ) :1574–334 1609 , 2009.335 [ 27 ] Arkadi Semenovich Nemirovsky and David Borisovich Yudin . Problem complexity and method336 efficiency in optimization . 1983.337 [ 28 ] Yu Nesterov . Universal gradient methods for convex optimization problems . Mathematical338 Programming , 152 ( 1-2 ) :381–404 , 2015.339 [ 29 ] Yurii E Nesterov . A method for solving the convex programming problem with convergence340 rate o ( 1/kˆ 2 ) . In Dokl . akad . nauk Sssr , volume 269 , pages 543–547 , 1983.341 [ 30 ] Razvan Pascanu , Tomas Mikolov , and Yoshua Bengio . On the difficulty of training recurrent342 neural networks . In International conference on machine learning , pages 1310–1318 , 2013.343 [ 31 ] Adam Paszke , Sam Gross , Francisco Massa , Adam Lerer , James Bradbury , Gregory Chanan,344 Trevor Killeen , Zeming Lin , Natalia Gimelshein , Luca Antiga , Alban Desmaison , Andreas345 Kopf , Edward Yang , Zachary DeVito , Martin Raison , Alykhan Tejani , Sasank Chilamkurthy,346 Benoit Steiner , Lu Fang , Junjie Bai , and Soumith Chintala . Pytorch : An imperative style , high-347 performance deep learning library . In H. Wallach , H. Larochelle , A. Beygelzimer , F. d'Alché-348 Buc , E. Fox , and R. Garnett , editors , Advances in Neural Information Processing Systems 32,349 pages 8024–8035 . Curran Associates , Inc. , 2019.350 [ 32 ] Herbert Robbins and Sutton Monro . A stochastic approximation method . The annals of351 mathematical statistics , pages 400–407 , 1951.352 [ 33 ] Olga Russakovsky , Jia Deng , Hao Su , Jonathan Krause , Sanjeev Satheesh , Sean Ma , Zhiheng353 Huang , Andrej Karpathy , Aditya Khosla , Michael Bernstein , Alexander C. Berg , and Li Fei-Fei.354 ImageNet Large Scale Visual Recognition Challenge . International Journal of Computer Vision355 ( IJCV ) , 115 ( 3 ) :211–252 , 2015.356 [ 34 ] Umut Şimşekli , Mert Gürbüzbalaban , Thanh Huy Nguyen , Gaël Richard , and Levent Sagun . On357 the heavy-tailed theory of stochastic gradient descent for deep neural networks . arXiv preprint358 arXiv:1912.00018 , 2019.359 [ 35 ] Umut Simsekli , Levent Sagun , and Mert Gurbuzbalaban . A tail-index analysis of stochastic360 gradient noise in deep neural networks . arXiv preprint arXiv:1901.06053 , 2019.361 [ 36 ] Vladimir Spokoiny et al . Parametric estimation . finite sample theory . The Annals of Statistics,362 40 ( 6 ) :2877–2909 , 2012.363 [ 37 ] Ashish Vaswani , Noam Shazeer , Niki Parmar , Jakob Uszkoreit , Llion Jones , Aidan N Gomez,364 Łukasz Kaiser , and Illia Polosukhin . Attention is all you need . In Advances in neural information365 processing systems , pages 5998–6008 , 2017.366 [ 38 ] Alex Warstadt , Amanpreet Singh , and Samuel R Bowman . Neural network acceptability367 judgments . arXiv preprint arXiv:1805.12471 , 2018.368 [ 39 ] Thomas Wolf , Lysandre Debut , Victor Sanh , Julien Chaumond , Clement Delangue , Anthony369 Moi , Pierric Cistac , Tim Rault , Rémi Louf , Morgan Funtowicz , Joe Davison , Sam Shleifer,370 Patrick von Platen , Clara Ma , Yacine Jernite , Julien Plu , Canwen Xu , Teven Le Scao , Sylvain371 Gugger , Mariama Drame , Quentin Lhoest , and Alexander M. Rush . Transformers : State-of-the-372 art natural language processing . In Proceedings of the 2020 Conference on Empirical Methods373 in Natural Language Processing : System Demonstrations , pages 38–45 , Online , October 2020.374 Association for Computational Linguistics.375 [ 40 ] Jingzhao Zhang , Tianxing He , Suvrit Sra , and Ali Jadbabaie . Why gradient clipping accelerates376 training : A theoretical justification for adaptivity . In International Conference on Learning377 Representations , 2020.378 [ 41 ] Jingzhao Zhang , Sai Praneeth Karimireddy , Andreas Veit , Seungyeon Kim , Sashank Reddi,379 Sanjiv Kumar , and Suvrit Sra . Why are adaptive methods good for attention models ? In380 H. Larochelle , M. Ranzato , R. Hadsell , M. F. Balcan , and H. Lin , editors , Advances in Neural381 Information Processing Systems , volume 33 , pages 15383–15393 . Curran Associates , Inc.,382 2020.383 Checklist384 1 . For all authors ... 385 ( a ) Do the main claims made in the abstract and introduction accurately reflect the paper ’ s386 contributions and scope ? [ Yes ] 387 ( b ) Did you describe the limitations of your work ? [ Yes ] Section 1.1 describes all assump-388 tions that we use389 ( c ) Did you discuss any potential negative societal impacts of your work ? [ No ] Our results390 are primarily theoretical , therefore , such a discussion is not applicable.391 ( d ) Have you read the ethics review guidelines and ensured that your paper conforms to392 them ? [ Yes ] 393 2 . If you are including theoretical results ... 394 ( a ) Did you state the full set of assumptions of all theoretical results ? [ Yes ] Section 1.1395 describes all assumptions that we use.396 ( b ) Did you include complete proofs of all theoretical results ? [ Yes ] Appendix B and C397 include the complete proofs of all the results we derive.398 3 . If you ran experiments ... 399 ( a ) Did you include the code , data , and instructions needed to reproduce the main experi-400 mental results ( either in the supplemental material or as a URL ) ? [ Yes ] See our code in401 the supplementary material.402 ( b ) Did you specify all the training details ( e.g. , data splits , hyperparameters , how they403 were chosen ) ? [ Yes ] See Appendix D.404 ( c ) Did you report error bars ( e.g. , with respect to the random seed after running exper-405 iments multiple times ) ? [ No ] Instead of it , we show the averaged trajectories of the406 methods ’ convergence.407 ( d ) Did you include the total amount of compute and the type of resources used ( e.g. , type408 of GPUs , internal cluster , or cloud provider ) ? [ Yes ] See Appendix D.409 4 . If you are using existing assets ( e.g. , code , data , models ) or curating/releasing new assets ... 410 ( a ) If your work uses existing assets , did you cite the creators ? [ Yes ] 411 ( b ) Did you mention the license of the assets ? [ No ] We use only publicly available412 resources.413 ( c ) Did you include any new assets either in the supplemental material or as a URL ? [ No ] 414 ( d ) Did you discuss whether and how consent was obtained from people whose data you ’ re415 using/curating ? [ No ] We use only publicly available resources.416 ( e ) Did you discuss whether the data you are using/curating contains personally identifiable417 information or offensive content ? [ N/A ] 418 5 . If you used crowdsourcing or conducted research with human subjects ... 419 ( a ) Did you include the full text of instructions given to participants and screenshots , if420 applicable ? [ N/A ] 421 ( b ) Did you describe any potential participant risks , with links to Institutional Review422 Board ( IRB ) approvals , if applicable ? [ N/A ] 423 ( c ) Did you include the estimated hourly wage paid to participants and the total amount424 spent on participant compensation ? [ N/A ] 425

Summary: This paper aims to obtain the high probability convergence for the convex stochastic optimization problem when the stochastic gradient of the objective function is not assumed to be sub-Gaussian. While this goal has been achieved in two recently proposed optimizers by exploiting the gradient clippings technique under the smoothness assumption on the objective, this paper considers the more general setting when corresponding gradient only satisfies a Holder continuity condition. This condition recovers the smooth and non-smooth settings when an extra parameter takes extreme values. In such a more general problem formulation, the authors propose new step-size schemes such that the convergence rate of the aforementioned optimizers (designed for smooth problems) can be smoothly extrapolated to the non-smooth setting.