Gradient Clipping Helps in Non-Smooth Stochastic Optimization with Heavy-Tailed Noise
Authors: Eduard Gorbunov
Marina Danilova
Innokentiy Andreevich Shibaev
Pavel Dvurechensky
Alexander Gasnikov
Venue: NeurIPS 2021 Submitted (2021)
Keywords: convex optimization
stochastic optimization
non-smooth optimization
heavy tails
clipping
SGD
high-probability bounds

Abstract: Thanks to their practical efficiency and random nature of the data, stochastic first-order methods are standard for training large-scale machine learning models. Random behavior may cause a particular run of an algorithm to result in a highly suboptimal objective value, whereas theoretical guarantees are usually proved for the expectation of the objective value. Thus, it is essential to theoretically guarantee that algorithms provide small objective residual with high probability. Existing methods for non-smooth stochastic convex optimization have complexity bounds with the dependence on the confidence level that is either negative-power or logarithmic but under an additional assumption of sub-Gaussian (light-tailed) noise distribution that may not hold in practice, e.g., in several NLP tasks. In our paper, we resolve this issue and derive the first high-probability convergence results with logarithmical dependence on the confidence level for non-smooth convex stochastic optimization problems with non-sub-Gaussian (heavy-tailed) noise. To derive our results, we propose novel stepsize rules for two stochastic methods with gradient clipping. Moreover, our analysis works for generalized smooth objectives with Holder-continuous gradients, and for both methods, we provide an extension for strongly convex problems. Finally, our results imply that the first (accelerated) method we consider also has optimal iteration and oracle complexity in all the regimes, and the second one is optimal in the non-smooth setting.

Gradient Clipping Helps in Non-Smooth Stochastic Optimization with Heavy-Tailed Noise
Anonymous Author(s)
Affiliation
Address
email
Abstract
Thanks to their practical efficiency and random nature of the data, stochastic first-order methods standard for training large-scale machine learning models. Random behavior may cause a particular run of an algorithm to result in a highly suboptimal objective value, whereas theoretical guarantees are usually proved for the expectation of the objective value. Thus, it is essential to theoretically guarantee that algorithms provide small objective residual with high probability. Existing methods for non-smooth stochastic convex optimization have complexity bounds with the dependence on the confidence level that either negative regative-power power or logarithmic but under an additional assumption of sub- Gab-Gausiinn (light-tailed) noise distribution that may not hold in practice, e.g.. in several NLP tasks. In our paper, we resolve this issue and derive the first high-probability convergence results with logarithmic dependence on the confidence level for non-smooth convex stochastic optimization problems with non-sub- uuu-aasssian (heavy-tailed) noise. To derive our results, we propose novel stepsize rules for two stochastic methods with gradient clipping. Moreover, our analysis works for generalized smooth objectives with Hölder-continuous gradients, and for both methods, we provide an extension for strongly convex problems. Finally, our results imply that the first (accelerated) method we consider also has optimal iteration and oracle complexity in all the regimes, and the second one is optimal in the non-smooth setting.
10
11
12
14
16
18
Introduction
20
Stochastic first-order optimization methods like SGD [32], Adam [20], and their various modifi- cations are extremely popular in solving a number different optimization problems, especially those appearing in statistics [36], machine learning, and deep learning [13]. The success of these methods in real-world applications motivates the researchers to investigate theoretical properties for the methods and develop new with better convergence guarantees. Typically, stochastic
22
24
25
26 methods are analyzed in terms of the convergence in expectation (see [12, 24, 15] and references
27 therein), whereas high-probability complexity results are established much rarely. However, as
28 illustrated in [14], guarantees in terms of the convergence in expectation have much worse correlation with the real behavior of the methods than high-probabee probability convergence guarantees when the noise
29
30 in the stochastic gradients has heavy-tailed distribution.
Recent studies [35, 34, 41] show that in several popular problems such training BERT [37] Wikipedia dataset the noise in the stochastic gradients is heavy-tailed. Moreover, in [41], the authors
32
justify empirically that in such cases SGD works significantly worse than clipped-SGI SGD [30] and
33
34 Adam. Therefore, it is important to theoretically study the methods' convergence when the noise is
35 heavy-tailed.
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

36
For convex and strongly convex problems with Lipschitz continuous gradient, i.e., smooth convex and strongly convex problems, this question was properly addressed in [25. 3. 14] where the first high- probability complexity bounds with logarithmic dependence on the confidence level were derived for the stochastic problems with heavy-tailed noise. However, number of practically important problems are non-smooth on the whole space [40, 22]. For example, in deep neural network training, the loss function often grows polynomially fast when the norm the network's ' weights goes to infinity. Moreover, non-smoothness of the activation functions such ReLU or loss functions such as hinge loss implies the non-smoothness of the whole problem. While being well-motivated by practical applications, the existing segh-prbbbbility convergence guarantees for stochastic first-order methods applied to solve non-smooth convex optimization problems with heavy-tailed noise depend on the negative power of the confidence level that dramatically increases the number of iterations required to obtain high accuracy of the solution with probability close to one. Such a discrepancy in the theory between algorithms for stochastic smooth and non-smooth problems leads to the natural question: it possible to obtain high-probability complexity bounds with logarithmic dependence on the confidence level for non-smooth convex stochastic problems with heavy-tailed noise? In this
37
38
39
40
41
43
45
46
47
49
51 paper, we give a positive answer to this question. To achieve this we focus on gradient clipping methods [30, 10, 23, 22, 40, 41].
52
53
1.1 Preliminaries
Before we describe our contributions in detail, we formally state the considered setup.
55
Stochastic optimization.
We focus on the following problem
min f(z), f(x) Es (f(x,,
#ER
58 where (xx) is a convex but possibly non-smooth function. Next, we assume that at each point e IR"
57 we have an access to the unbiased estimator (.f(,, of (=(x) with uniformly bounded variance
E& (
so² o a>0.
58
This assumption on the stochastic oracle is widely used in stochastic optimization literature [11,
59
12, 19, 21, 26]. We emphasize that we do not assume that the stochastic gradients have so-called
"light tails" [21], i.e., sub-Gaussian noise distribution meaning that [ [ V/(x)] > b} s
60
2 2e (oxp(=>/(2x3) for all b 0.
61
82 Level of smoothness. Finally, we assume that function has (v, A..)-Hölee continuous gradients on a compact set Q C R= for some v E [0, 1], M, >> 0 meaning that
63
[V f S M, | vIll Vz,y
(3)
64
When v 1 incquality (3) implies M -msoothhsss of f. and when v O we have that Vf(x)
85
has bounded variation which is equivalent to being uniformly bounded. Moreover, when v/ = 0 differentiability of f not needed, and one can assume uniform boundedness of the subgradients of f. Linear regression in the case when the noise has generalized Gaussian distribution (Example 4.4 from [2]) serves as a natural example of the situation with v E (0. 1). Moreover, when (3) holds for and
86
57
68
89
simultancously then holds for all show in our results, the set Q should contain the ball centered at the solution x² of (1) with radius
[0, 1] with M, S Mr M1-rr MM'"MP [28]. As we
70
2Rg = 2 12 x= 22,, where z is starting point of the method, i.e., our analysis does not require (3)
71
72 to hold on T..
73
High-probability convergence. For given accuracy and confidence level 3 (0, 1) we are interested in finding E e-solutions of problem (1) with probability at least 1 - B. i.e., such 2 that - B. For brevity, we will call such (in general, random) points T as (c, s)-outinn of (1). Moreover, by high-probability complexity of stochastic method M we mean
74
75 P( f(x) <) 2
77 the sufficient number of oracle calls, i.e., number of (x, g) computations, needed to guarantee that
the output of M is an (e, -oouution of (1).
2

---

Table 1: Summary of known and new High-probabilli complexity bounds for solving (1) with being convex and having (v, --j-Hölder continuous gradients. Columns: "Ref." reference, "Complexity" proobbpobbbilitty complexity (c accuracy, 3 confidence level, numerical constants and logarithmic factors are omitted), "HT" heavy-tailed noise, "UD" unbounded domain, "HCC" Hölder continuity of the gradient is required only the compact set. The results labeled by are obtained from the convergence guarantees expectation via Markov's inequality, Negative-power dependencies on the confidence level / are colored in red.
Method
Ref.
Complexity
HT?
UD?
HCC?
SGD
[26]
max
MA
8
0
x
/
x
AC-SA
[11, 21]
M1 R
o R
1
/
x
2(140)
SIGMA
[6]
max
Mr 433
Rg
1433
a Rg.
00.
1]
X
V
X
SGD
[26]* +
118
Mt Ró.
3"
R
15
0
x
X
AC-SA
[11, 211*
max
My RG
o? P
B"
2(1+2)
SIGMA
[6]
max
Mr 1430
R
1+30
R
[0, 1]
/
X
clipped-SSTM
[14]
max
M1 R o n
2
x
clipped-SGD
[14]
Mf R
R
1
/
2(1+2)
clipped-SSTM
Thm. 2.2
max
M +0
R
1430
[0. 1]
/
/
clipped-SGD
Thm. 3.1
max
M, 47
Rg
a² n
[0,
/
1.2
Contributions
We propose novel stepsize rules for clipped- SSTM [14] to handle the problems with Hölder continuous gradients and derive high-probability complexity guarantees for convex stochastic optimization problems without using "light tails" assumption, i.e., we prove that our version of clipped-SSTM
80
82
201
2(1+v) 1436
R2 g2 In 1) D
D
M MT 3
R
1936
max
Din
1
84 high-probability complexity. Unlike all previous high-probabiling complexity results in this setup with < (see Tbl. 1), our result depends only logarithmically the confidence level that is highly important when small. Moreover, up to the difference in logarithmic factors the derived complexity guarantees meet the known lower bounds [21, 17] obtained for the problems we recover accelerated convergence rate [29, 21].
86
with light-tailed noise. In particular, when That is, neglecting the logarithmic factors results unimprovable and, surprisingly coincide with the best-knoown results the "light-tailed case".
88
90
We derive the first gigbabbbbayiltty complexity bounds for clipped-SGi when the objective functions is convex with (n, M,))- Höölder continuous gradient and the noise is heavy tailed,, i.c., we derive
91
92
93
O max D2 max Ditv o² RD
D2 + +DItT DL ()
Ro
In
D
94 high-probability complexity bound. Interestingly, when v 0 the derived bound for clipped-SGD 85 has better dependence on the logarithms than the corresponding one for clipped-SSTM Moreover, neglecting the dependence under the logarithm, our bound for clipped-SGH SGD has the same

---

Table 2: Summary of known and new tigh-probbbilltyy complexity bounds for solving (1) with being ju-strongyy convex and having (v, -Hölder continuous gradients. Columns: "Ref." reference, "Complexity" nigh-probabiling complexity (c accuracy, 3 confidence level, numerical constants and logarithmic factors are omitted,, "HT heavy-tailed noise, "UD" unbounded domain, "HCC" Hölder continuity of the gradient is required only the compact set. The results labeled by are obtained from the convergence guarantees expectation via Markov's inequality, Negative-power dependencies on the confidence level B are colored in red.
Method
Ref.
Complexity
1
HT?
UD?
HCC²
SGD
[26]
max
M
0
x
/
x
AC-SA
[11, 211
max
1
/
x
f
JEE
InXX
N,
g²
SIGMA
[6]
[0, 1]
X
/
X
N= =
Mr
113P
M
1444
SGD
[26]*
max
MIlE?
Mi
JASIE
0
x
X
AC-SA
[1]
max
M1
JADE
1
x
N
BE,
SIGMA
[6]
[0,
4
x
Ñ
My
M2
R-clipped-SSD SSMpped-SSTM
[14]
max
1
x
R-clipped
[14]
max
Mj
o
1
/
x
N.
ps
R-clipped-SSTIM
Thm. 2.1
[0, 1]
/
/
N
Mr²
Ri.
(ulling
2
R-clipped-SGD
Thm. 3.2
max
MP
Mr
2
[0. 1]
v
R
149
97
dependence on the confidence level as the tightest known result in this case under the "light tails" assumption [16].
Using restarts technique we extend the obtained results for clipped-SSTM and clipped- to the strongly convex case (see Tbl. 2). As in the convex case, the obtained results are superior all previous known results in the general setup consider.
99
100
102 As one of the key contributions of this work, we emphasize that in our theoretical results it is sufficient assume Hölder continuity of the gradients of f only the ball with radius 2Rg 2|| x² ||2 and centered at solution of the problem. This makes our results applicable to much larger class of problems than functions with Hölder continuous gradients on R" e.g., our analysis works even for polynomially growing objectives.
103
104
105
106
107 To test the performance of the considered methods we conduct several numerical experiments on image classification and NLP tasks, and observe that 1) clipped-SSTM SSTM and dipped-SGD show a comparable performance with SGD on the image classification task, when the noise distribution almost abbGaausiian, 2) converge much faster than SGD on the NLP task, when the noise distribution is heavy-tailed, and 3) clipped-SSTM achieves comparable performance with Adam on the NI.P task enjoying both the best known theoretical guarantees and good practical performance.
108
109
110
112
1.3 Related work
115 Light-tailed noise. The theory high-probability complexity bounds for convex stochastic op- timization with light-tailed noise is well-developed. Lower bounds and optimal methods for the 117 problems with (v, --)-Hrr continuous gradients are obtained [26] for and in [11] for L. Up to the logarithmic dependencies these high nigh-probability convergence bounds coincide with

---

119
the corresponding results for the convergence in expectation (see first two rows of Tbl. 1) While not being directly derived in the literature, the lower bound for the case when V/ (0, 1) can be obtained as combination of lower bounds in the deterministic [27, 17] and smooth stochastic settings [11]. The corresponding optimal methods are analyzed in [4, 6] through the lens of inexact oracle.
120
122
Heavy-tailed noise. Unlike in the "light-tailed" case, the first theoretical guarantees with reasonable dependence on both the accuracy & and the confidence level 3 appeared just recently. In [25], the first such results without acceleration [29] were derived for Mirror Descent with special truncation technique for smooth (v = 1) convex problems on a bounded domain, and then were accelerated and extended in [14]. For the strongly convex problems the first accelerated high-pros proobbbbbbylltty convergence guarantees were obtained in [3] for the special method called proxBoost requiring solving auxiliary nontrivial problems at each iteration. These bounds were tightened in [14].
124
125
126
127
128
129
130 In contrast, for the case when v < 1 and, in particular, when y/ = 0 the best-known complexity bounds suffer from the negative-power dependence the confidence level B. i.e., have a factor 1/8- for some a > O, that affects the convergence rate dramatically for small enough 3. Without additional assumptions on the tails these results are obtained via Markov's inequality (()(x) - f(xx) >& }) < ([ 11/s from the guarantees for the convergence in expectation to the accuracy eB, see the results labeled by * in Tbl. 1. Under an additional assumption on noise 1 > so O(s for >> 2 these results can be tightened [9] when V = 0 as O MM RB max { In(a ")/e,(1/B))) (1/Be))//0 x)) "}) without removing the negative-power dependence on the confidence level B. Different stepsize policies allow to change the last term in max to 5 2m e za without removing the regative-power dependence on B.
132
133
134
135
136
tails that f <) f=)))
137
138
139
140
Gradient clipping. The methods based on gradient clipping [30] and normalization [18] are popular in different machine learning and deep learning tasks due to their robustness in practice to the noise in the stochastic gradients and rapid changes of the objective function [13]. In [40, 22], clipped-GD and SSD are theoretically studied in applications to non-smooth problems that can grow polynomially fast when || x= ||| + oc showing the superiority of gradient clipping methods to the methods without clipping. The results from [40] are obtained for non-convex problems with almost surely bounded noise, and in [22], the authors derive the stability and expectation convergence guarantees for strongly convex under assumption that the central p-th moment of the stochastic gradient is bounded for 2 2. Since the authors of [22] do not provide convergence guarantees with explicit dependencies on all important parameters of the problem it complicates direct comparison with our results. Nevertheless, convergence guarantees from [22] are sub-lines and are
142
143
144
145
146
147
148
149
150
151
given for the convergence in expectation, and, as consequence, the corresponding high-prou srobbbbbbbbility convergence results obtained via Markov's 's inequality also suffer from negative- power dependence on the confidence level. Next, in [41], the authors establish several expectation convergence guarantees for clipped- SGD and prove their optimality in the non-convex case under assumption that the central a-moment of the stochastic gradient uniformly bounded, where a E (1,2). turns out that clipped- SGD is able to converge even when a < 2, whereas vanilla SGD can diverge in this setting.
152
153
154
155
156
2
Clipped Stochastic Similar Triangles Method
157
158
In this section, propose novel variation of Clipped Stochastic Similar Triangles Method [14] adjusted to the class of objectives with Hölder continuous gradients Sclipedd-STTI see Alg. I).
159
160
The method is based on the clipping of the stochastic gradients:
(i,(V)(zzE),.) min {
(1,
A [V E)
(4)
where fz( = 1 vf(x, is a bani-batched stochastic gradient. Gradient clipping ensures that the resulting vector has norm bounded by the clipping level X. Since the clipped stochastic gradient cannot have arbitrary large norm, the clipping helps to avoid unstable behavior of the method when the noise is heavy-tailed and the clipping level À is properly adjusted.
161
162
163
164
185
However, unlike the stochastic gradient, clipped stochastic gradient is a biased estimate of (x)):: the smaller the clipping level the larger the bias. The biasedness of the clipped stochastic gradient
5

---

Algorithm 1 Clipped Stochastic Similar Triangles Method (clipped-SSTM); case E [0. 1]
Input: starting point z², number of iterations N, batchsizes {m& k=1 stepsize parameter a, clipping parameter B, Hölder exponent v E [0, 1].
N
1: Set Ao = ox 0, 5/ = x
2: for k ===00 o...., N do
3: Set 06+1 = alk: + 1) Akt1 = As + 01-1,, A6+1 B
004+1
4: ak+1 (Azv* +0/+1**\/M
5: Draw mini-batch my of fresh i.d. samples ef...., Smn, and compute 7f(**t1,E*) =
myy Em vf(ahtl, .4%)
1
Compute fzttt1 (oh clip fa {²), Ae+1) using (4)
6:
7: gh+1 = 2 ax+1 V ((att1,
8: g++1= (Axy" + 12 A/4111
9: end for
Output: y
167
complicates the analysis of the method. On the other hand, to circumvent the negative effect of the heavy-tailed noise on the nigh-probability convergence one should choose À to be not too large. Therefore, the question on the appropriate choice of the clipping level is highly non-trivial.
168
169
Fortunately, there exists simple but insightful observation that helps to obtain the right formula for the clipping level 0 in clipped-SS -SSTd--SS:: if Ag is chosen in such a way that V=(()")le S Av/2
171
with high probability, then for the realizations f((**1,, _h)) of the mini-batched stochastic gradient such that [V $*)
172
(ff(***>1]] < >a/2 the clipping is an identity operator. Next, if the probability mass of such realizations is big enough then the bias of the clipped stochastic gradient is
173
174
175
properly bounded that helps to derive needed convergence guarantees. It turns out that the choice
176
A
1/a, ensures the method convergence with needed rate and high enough probability.
177
Guided by this observation we derive the precise expressions for all the parameters of Hipped-SSTIE and derive high-probability complexity bounds for the method. Below we provide simplified version of the main result for clipped- -osdd--SS---- in the convex case. The complete formulation and the full proof of the theorem are deferred Appendix B.1 (see Thm. B.
178
179
180
Theorem 2.1. Assume that function f is convex and its gradient satisfy (3) with v E [0. 1]. M, 0 on Q = BRR. = fx= E R" || x 1=1|2 < 2Ro). where Rg N ||2 2211 Then there exist such
182
choice of parameters that clipped- -STM achieves f(y [(x²) S with probability at least
183
2012)
1 1 B after O (D
201
1438
iterations with D
M. TT.33 R. 113"
and requires
of (max max {
(
2 I
D o² R
2 In
D) oracle calls.
DJ 3
(5)
3
185
The obtained result has only logarithmic dependence on the confidence level 3 and optimal depen- dence on the accuracy E up to logarithmic factors [21, 17] for all [0, 1]. Moreover,
emphasize that our result does not require of to have (v, M, Höölerr continuous gradient on the whole space. This is because we prove that for the proposed choice of parameters the iterates of Clipped-SSTM stay inside the ball BRn, = {i E TR" || x² x=112 < 2 with probability at least
187
188
189
8, and, consequence, Hölder continuity of the gradient is required only inside this ball. In particular, this means that the better starting point leads not only to the reduction of Ro. but also it can reduce My. Moreover, our result is applicable to much wider class of functions than the standard class of functions with Hölder continuous gradients in R", e.g., to the problems with polynomial growth.
190
181
182
193
For the strongly convex problems, we consider restarted version of Alg. 1 (R-clipped-SSTT) Alg. 2) and derive high probabiliyy complexity result for this version. Below we provide a simplified version of the result. The complete formulation and the full proof of the theorem are deferred to Appendix B.2 (see Thm. B.2).
see
195
197
Theorem 2.2. Assume that function f is p-strongly convex and its gradient satisfy (3) with v E [o. 1). || x= Fl2 S 2Ro). where Ro 2 1| x" 2|2. Then there exist
198
M, 0 on Q B2R, {x Rn

---

Algorithm 2 Restarted clipped-SSTIN (R-clipped-SSSTM case [0.
Input: starting point zº, number of restarts number of steps of clipped-SSTM in restarts (~e
batchsizes {m k== {ml 21 Na-1
[f=k]== ..... {m]] N. -1 stepsize parameters {a') clipping pa-
==1 rameters [B:)t=1 Hölder exponent E [0, 1/.
1: 2 = 0
2: for f I,..,, do
3:
stepsize parameter ar, clipping parameter B,, and starting point ft-1, Define the output of clipped-SSTM by #t.
Run slipped-SSTIN (Alg. 1) for N, iterations with batchsizes {m, 1==1
N,,11
4: end for
Output: 8"
such a choice of parameters that R-clipped-SSTM achieves f(i") f(x*) S & with probability at 201 least 1 8 after
N = (D
(DmTT D D). D max
201
|RR/- (
Mr
1430
M2
143v
In
(6)
v
v
iterations of Alg. I in total and requires
o ( max {
2016) 1430
μs 3 B
0² In
Din
oracle calls.
3 8'
203
Again, the obtained result has only logarithmic dependence on the contidence level B and, our result in the convex case, it has optimal dependence on the accuracy up to logarithmic factors depending on 3 [21, 17] for all v E [0, 1].
204
205
3
SGD with clipping
206
207
In this section, we present new variant of clipped-SGD [30] properly adjusted to the class of objectives with (v, M2) --öderr continuous gradients (see Alg. 3).
208
Algorithm 3 Clipped Stochastic Gradient Descent Sripped-SGDi case [0, 1]
Input: starting point zo, number of iterations N, batchsize m, stepsize y, clipping parameter B > 0.
1: for k = O...., N - do
2: Draw mini-batch
m fresh i.i.d samples sf....., eh and compute f((*++1 ,g*) =
En V*(**+1 ,1)) ck)
3: Compute f((x". <*) clip(Vf(2* A using (4) with B/7
4: kht1 k Vxx )
5: end for
Output: N. EK= 1
209 We emphasize that as for Biipped-STMM we use clipping level inversely proportional to the stepsize
7. Below we provide simplified version of the main result for clipped-SGD in the convex case. The complete formulation and the full proof of the theorem are deferred to Appendix C. (see Thm. C.I).
211
Theorem 3.1. Assume that function f is convex and its gradient satisfy (3) with E [0, 1), M,
on Q B2n = (x E Rn | x=" x=1||2 < 2R),, where Ro 2 I 1/ Then there exist such
213
choice of parameters that clipped- SGD achieves {(xN) f(x E with probability least 1
215 after
o (max max { D², DitV Ditu In
D + D1+v DD+Ditv
M Ro
D
(8)
3
216 iterations and requires
O max D2, max { Dl+v
σ² R2
D2 + D )) oracle calls.
(9)
2
3

---

217
As all our results in the paper, this result for clipped-SGD has two important features: 1) the dependence on the confidence level / is logarithmic and 2) Hölder continuity required only on the ball B2n, centered at the solution. Moreover, up to the difference in the expressions under the logarithm the dependence on & the result for clipped-SGD is the same as in the tightest known results for non-accelerated SGD-type methods [4, 16]. Finally, we emphasize that for v < 1 the logarithmic factors appearing in the complexity bound for clipped-SSTI SSTM are worse than the corresponding factor in the complexity bound for clipped-SCR SGD. Therefore, clipped-SCC SGD has the best known high-probabillty complexity results in the case when v = 0 and f is convex.
218
220
222
224
225
For the strongly convex problems, we consider restarted version of Alg. (R-clipped-SGD. see Alg. 4) and derive high-probability complexity result for this version. Below we provide a simplified
Algorithm 4 Restarted clipped- SGD (R-clipped-See SGD): case E [0,
Input: starting point x., number of restarts т, number of steps of clipped-SGD in restarts (Na) batchsizes (ma M==1 stepsizes (m clipping parameters [B, )
1: _ = x00
2: fort= T...., do
3: Run clipped- SGD (Alg. 3) for N; iterations with batchsize my stepsize Yt, clipping parame-
ter Br, and starting point st a²-1, Define the output of clipped-SGD by &
4: end for
Output: 8"
version of the result. The complete formulation and the full proof of the theorem are deferred to Appendix C.2 (see Thm. C.2).
228
Theorem 3.2. Assume that function f is p-strongly convex and its gradient satisfy (3) with v E [o, 1],
M, > on Q B2ng = {ar R || xx ||2 < 2o), where Ro 2 ||x 1l2. Then there exist
230
such choice of parameters that R-clipped-Sc SGD achieves f( - 10 f(x") S E with probability at
232
least 3 after
2 D 14v
RR
D2 t4u max {
Dj uR
D2 In 3 1
max
In
233
iterations of Alg. total and requires
R.R
D2in RR
0²
D 1
max
D
max
D2,
In
oracle calls, where
μe
234
D1
M
D2
M, I
D= (D क. + Di)
RR
+ D2 + D 2
PR w
As in the convex case, for v < the log factors appearing in the complexity bound for R-clipped- SSTM are worse than the corresponding factor in the bound for R-clipped-SOR SGD. Thus, R-clipped-
235
236
237
SGD has the best known high high-probability complexity results for strongly convex and 0.
4
Numerical experiments
238
239
We tested the performance of the methods on the following problems:
240
BERT fine-tuning on CoLA dataset [38]. We use pretrained BERT from Transformers library [39] peet--base-nnaasee and freeze all layers except the last two linear ones.
241
242
ResNet-18 training on ImageNet- 100 (first 100 classes of ImageNet [33]).
243
First, we study the noise distribution for both problem as follows: at the starting point we sample
large enough number of batched stochastic gradients (x E...... V ff(x") {x) with batchsize
32 and plot the histograms for | { fx)"))]2
[V Ex) Vf(z see Fig. 1.
245
As
see, the noise distribution for BERT + CoLA is substantially non-sub- nan-sub-Gaussian. whereas the distribution for ResNet-18 + Imagenet- is almost Gaussian.
247
Next, we compared different optimizers these problems: Adam, SGD (with Momentum), clipped-SGD (with Momentum and coordinate- clipping) and clipped-SSTI (with norm- 1). The results presented in Fig. 2. We observed that the noise distributions do
249
clipping and
8

---

ResNett118 ImageNet-100, batch cound=-600
Bert + CoLA, batch counte-96t 96k
0.3
0.6
Gordla
deraity
0.2
0.4
0.2
0.0
nor
nolse norm
Figure 1: Noise distribution of the stochastic gradients for ResNet-18 on ImageNet-100 and BERT fine-tuning on the CoLA dataset before the training. Red lines: probability density functions with means and variances empirically estimated by the samples. Batch count is the total number of samples used to build a histogram.
251
not change significantly along the trajectories of the considered methods, see Appendix D. During the hyper-parameters search we compared different batchsizes, emulated via gradient accumulation (thus we compare methods with different batchsizes by the number of base batches used). The base 253
batchsize was 32 for both problems, stepsizes and clipping levels were tuned. One can find additional details regarding our experiments in Appendix D.
255
Train loss, ResNet:18
Validation loss, ResNett 18
Valication acouracy, ResNet-18
2.0
SGD
1.8
clpped-SSI SSMM
clpped SGD
1.6
accuracy
0
1.4
1.4
0.8
1.0
1.2
batch count
batch count
count
165
Train loss, BERT
Validation loss. BERT
Validation accursty, BERT
0.70
diipped
0.60
75
0.55
dipped-SCD
0.50
SGD
0.50
73
0.55
0.54
72
Figure 2: Train and validation loss + accuracy for different optimizers on both problems. Here, "batch count" denotes the total number of used stochastic gradients.
Image classification. On MasNet-188 + Imagellet- task, SGD performs relatively well, and even ties with Adam (with batchsize of 4 x 32) in validation loss. clipped-SSTM (with batchsize of 257
2 x 32) also ties with Adam and clipped-SGD is not far from them. The results were averaged from
259 5 different launches (with different starting points/weight initializations). Since the noise distribution
is almost Gaussian even vanilla SGD performs well, i.e., gradient clipping is not required. At the same time, the clipping does not slow down the convergence significantly.
281
262
Text classification. On BERT + CoLA task, when the noise distribution is heavy-tailed, the methods with clipping outperform SGD by a large margin. This result good correspondence with the derived -robbbbbbybylityy complexity bounds for clipped-SGD, clipped-SSTM and the best-known (with batchsize of 32) achieves the same loss on validation as Adam, and has better accuracy. These results were averaged from 5 different train-val
263
264
ones for SGD. Moreover, clipped-SSTI
265
257
splits and 20 launches (with different starting points/weight initializations) for each of the splits, 100 launches in total.

---

289
References
270
[1] George Bennett. Probability inequalities for the sum of independent random variables. Journal of the American Statistical A (7(297):33-45, 1962.
271
272
[2] Caroline Chaux, Patrick L Combettes, Jean-Christophe Pesquet, and Valérie R Wajs. A varia- tional formulation for frame-based inverse problems. Inverse Problems, 1354)1195--1518, jun 2007.
273
274
275
[3] Damek Davis, Dmitriy Drusvyatskiy, Lin Xiao, and Junyu Zhang. From low probability to high confidence in stochastic convex optimization. Journal of Machine Learning Research, 22(49):1-38, 2021.
278
277
278
[4] Olivier Devolder. Exactness, inexactness and stochasticity in first-ooord methods for large-scale convex optimization. PhD thesis, PhD thesis, 2013.
279
280
[5] Olivier Devolder, François Glineur, and Yurii Nesterov. First-order methods of smooth convex optimization with inexact oracle. Mathematiical Programming, 146(1):37-75, 2014.
281
282
[6] Pavel Dvurechensky and Alexander Gasnikov. Stochastic intermediate gradient method for con- vex problems with stochastic inexact oracle. Journal of Optimization Theory and Applications. 171(1):121-145, 2016.
283
285
[7]
Kacha Dzhaparidze and JH Van Zanten. On bernstein-type inequalities for martingales. Stochas- tic processes and their applications, 93(1):109-117, 2001.
286
287
88]
David A Freedman et al. On tail probabilities for martingales. the Annals of Probability, 3(1):100-118, 1975.
99
Alexander Vladimirovich Gasnikov, Yu E Nesterov, and Vladimir Grigor' Spokoiny. On the efficiency of a randomized mirror descent algorithm in online optimization problems. Computational Mathematics and Mathematical Physics, 55(4):580-58) 2015.
290
[10] Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. Convolutional sequence to sequence learning. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pages
293
JMLR. org, 2017.
[11] Saeed Ghadimi and Guanghui Lan. Optimal stochastic approximation algorithms for strongly convex stochastic composite optimization i: A generic algorithmic framework. SIAM Journal on Optimization, (44)::449-1492 2012.
296
297
[12] Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization, 23(4):2341-25 -32628,8, 2013.
299
300
[13] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016.
http:://www.ddepleerniiii
org.
302
[14]
Eduard Gorbunov, Marina Danilova, and Alexander Gasnikov. Stochastic optimization with heavy-tailed noise via accelerated gradient clipping. In H. Larochelle, M. Ranzato, R. Had- sell. M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 15053, Curran Associates, Inc., 2020.
303
304
305
306
[15] Robert Mansel Gower, Nicolas Loizou, Xun Qian, Alibek Sailanbayev, Egor Shulgin, and Peter Richtárik. Sgd: General analysis and improved rates. In International Conference on Machine Learning, pages 5200-5209, 2019.
307
308
309
[16] Vincent Guigues, Anatoli Juditsky, and Arkadi Nemirovski. Non-asymptotic confidence bounds for the optimal value of a stochastic program. Optimization Methods and Software, 32(5))1033- 1058, 2017.
310
312
[17] Cristóbal Guzmán and Arkadi Nemirovski. On lower complexity bounds for large-scale smooth convex optimization. Journal of Complexity, 31(1):1-14, 2015.
313
314
[18] Elad Hazan, Kfrr Levy, and Shai Shawav-Shwartz. Beyond convexity: Stochastic quasi-convex
optimization. In Advances in Neural Information Processing Systems, pages 1594-1602, 2015.
10

---

316
[19] Anatoli Juditsky, Arkadi Nemirovski, et al. First order methods for nonsmooth convex large- scale optimization, i: general purpose methods. Optimization for Machine Learning, pages 121-148, 2011.
317
318
319
[20] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint a4Xiv:1412.6900 26900, 2014.
320
321
[21] Guanghui Lan. An optimal method for stochastic composite optimization. Mathematical Programming, 133(1-2))365-39) 2012.
322
323
[22] Vien V Mai and Mikael Johansson. Stability and convergence of stochastic gradient elipping: Beyond lipschitz continuity and smoothness. arXiv preprint arXiv:2/02.06489, 2021.
324
[23] Aditya Krishna Menon, Ankit Singh Rawat, Sashank Reddi, and Sanjiv Kumar. Can gradient clipping mitigate label noise? In International Conference on Learning Representations, 2020.
326
327
[24]
Eric Moulines and Francis R Bach. Non-asymplotic analysis of stochastic approximation algorithms for machine learning. In Advances in Neural Information Processing Systems, pages 451-459, 2011.
329
330
[25] Aleksandr Viktorovich Nazin, AS Nemirovsky, Aleksandr Borisovich Tsybakov, and AB Judit- sky. Algorithms of robust stochastic optimization based mirror descent method. Automation and Remote Control, 80(9): 160- 2019.
332
335
[26] Arkadi Nemirovski, Anatoli Juditsky, Guanghui Lan, and Alexander Shapiro. Robust stochastic approximation approach to stochastic programming. SIAM Journal on optimization, 19(4):1574. 1 1609, 2009.
335
336
[27] Arkadi Semenovich Nemirovsky and David Borisovich Yudin. Problem complexity and method efficiency in optimization. 1983.
337
338
[28] Yu Nesterov. Universal gradient methods for convex optimization problems. Mathematical Programmingg 152(1-2):3,,444, 2015.
339
340
29
Yurii E Nesterov. A method for solving the convex programming problem with convergence rate o (1/k*) 2). In Dokl. akad. nauk Sssr, volume 269, pages 543-547, 1983.
341
342
30
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning, pages 1310-1318, 2013.
343
[3]] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito,,, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019.
345
346
347
348
349
350
[32] Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pages 400-407, 1951.
352
353
[33] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJC)), 115(3)22112555 2015.
355
356
[34] Umut Şimşekli, Mert Gürbüzbalaban, Thanh Huy Nguyen, Gaël Richard, and Levent Sagun. On the heavy-tailed theory of stochastic gradient descent for deep neural networks. arXiv preprint 1rXiv:1912.00018, 2019.
358
[35] Umut Simsekli, Levent Sagun, and Mert Gurbuzbalaban. A tail-index analysis of stochastic gradient noise in deep neural networks. arXiv preprint arXiv: 0901506053, 2019.
361
362
[36] Vladimir Spokoiny et al. Parametric estimation. finite sample theory. The Annals of Statistics, 4066)28877-2999 2012.
11

---

384
[37] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 99986600,, 2017.
385
366
367
[38] Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. arXiv preprint arXiv: 12471, 2018.
368
389
[39]
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 45, Online, October 2020.
371
373
375
Association for Computational Linguistics.
376
[40] Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaie. Why gradient clipping accelerates training: A theoretical justification for adaptivity. In International Conference on Learning Representations, 2020.
378
379
[41] Jingzhao Zhang, Sai Pranecth Karimireddy, Andreas Veit, Seungycon Kim, Sashank Reddi, Sanjiv Kumar, and Suvrit Sra. Why are adaptive methods good for attention models? In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 15383-15393, Curran Associates, Inc., 2020.
380
381
382
383
Checklist
384
385
1. For all authors...
386
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you deseribe the limitations of your work? [Yes] Section 1.1 describes all assump- tions that we use
389
390
(c) Did you discuss any potential negative societal impacts of your work? [No] Our results are primarily theoretical, therefore, such a discussion not applicable.
391
392
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
394
2. If you are including theoretical results...
395
(a) Did you state the full set of assumptions of all theoretical results? [Yes] Section 1.1 describes all assumptions that we use.
387
(b) Did you include complete proofs of all theoretical results? [Yes]] Appendix B and C include the complete proofs of all the results we derive.
398
399
3. If you ran experiments..
400
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URLY? [Yes] See our code in the supplementary material.
401
402
403
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix D.
(c) Did you report error bars (e.g., with respect to the random seed after rumning exper- iments multiple times)? [No] Instead of it, we show the averaged trajectories of the methods' convergence.
406
408
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix D.
409
410
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
411
(a) If your work uses existing assets, did you eite the creators? [Yes]
412
(b) Did you mention the license of the assets? [No] We use only publicly available resources.
12

---

414
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [No] We use only publicly available resources.
416
417
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5.
If you used crowdsourcing conducted research with human subjects...
420
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
421
422
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
425
13


Reviews:
[{"review_id": "rRlzV_DUog2", "reviewer": "Reviewer_fbjB", "summary": "The authors proposed  clipped-SSTM and its restart version for stochastic nonsmooth optimization with heavy tailed noise. A detailed convergence analysis shows the theoretical advantage of the proposed algorithm. Experiments on real models (Bert) show that clipped-SSTM outperforms standard SGD methods.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is very clearly structure, it has addressed an important issue in designing stochastic algorithms for heavy-tailed noise. However, although this paper makes some important theoretical contribution, I think it needs a major revision to justify the true advantage in non-smooth optimization. I would recommend rejection but I am also willing to hear the author's feedback.\n\n\n- The paper makes a technical contribution in showing high probability result of stochasitc algorithms when the  gradient has an biased (clipped) estimator. However, the 10 pages proof of (Theorem 2.1) is very technical, I really suggest providing a brief highlight of the proof of Theorem 2.1, which will substantially improve the readability of the theory.\n- In the experiment, the paper uses $\\nu=1$, which essentially assumes that the problem is smooth. This is unintuitive because the goal of this paper is for non-smooth optimization, but the results of the experiments is only able to corroborate the study [15] for smooth optimization. I understand that the assumption of Holder continuity is more general, but it is still somewhat misleading since the aim is for non-smooth optimization. I strongly suggest adding empirical study of non-smooth problems to justify the advantage of the proposed methods.\n- In the experiments, what are the loss functions used by Bert and Resnet, and are the models non-smooth?\n- It appears that Adam, without much tuning, already exhibits great performance. How are the parameters of Adam is chosen? I wonder what Adam performs if lr is tuned as a hyperparameter.\n- Another potential weakness is that the proposed algorithm has so many hyperparameters to tune. Can the authors ellaborate some intuition for tuning those parameters.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "NCDTWkGSUpv", "reviewer": "Reviewer_2WXq", "summary": "This paper aims to obtain the high probability convergence for the convex stochastic optimization problem when the stochastic gradient of the objective function is not assumed to be sub-Gaussian. While this goal has been achieved in two recently proposed optimizers by exploiting the gradient clippings technique under the smoothness assumption on the objective, this paper considers the more general setting when corresponding gradient only satisfies a Holder continuity condition. This condition recovers the smooth and non-smooth settings when an extra parameter takes extreme values. In such a more general problem formulation, the authors propose new step-size schemes such that the convergence rate of the aforementioned optimizers (designed for smooth problems) can be smoothly extrapolated to the non-smooth setting. \n\n\n\n", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work provides solid convergence analyses for the recently proposed clipped-SSTM and clipped-SGD under the more general Holder continuity condition. Interestingly, the convergence rate in the smooth setting is recovered by taking the parameter $\\nu$ of the Holder continuity condition to the limit of $1$. \n \nConcern:\nThe proposed step size scheme for the clipped-SSTM requires knowledge of the Holder parameter $\\nu$ which is in general unknown. As suggested by the experiment section of this paper, $\\nu$ is set to $1$ which recovers the previous clipped-SSTM. I am wondering how $\\nu$ affects the performance of clipped-SSTM when it is not set to $1$, as this sets apart this work and the previous work [15] algorithmically. For clipped-SGD, I have the same concern. I believe the more general Holder continuity condition introduces extra difficulty to the analysis of the algorithm, but if we will always use $\\nu=1$, the significance of the convergence under the more general condition is limited.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This is a theoretical work and I see no potential negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "HJm8sYP8Kms", "reviewer": "Reviewer_uN41", "summary": "\nThe submission presents convergence results for clipped stochastic gradient methods without the sub-Gaussian assumption.\n\nThe subject of the submission is of interest to the NeurIPS community. As we find that some of our most difficult problems exhibit heavy-tailed noise, how to handle it in stochastic optimization is becoming more and more relevant.\n\n\nMy understanding of the main contributions of the paper over existing work is that this submission considers $\\nu$-Hölder continuity (interpolating between Lipschitz functions and function with Lipschitz gradients) without assuming the stochastic gradients be sub-Gaussian (but still having bounded variance)\n- For the case $\\nu = 0$ (Lipschitz functions), recovers tighter bounds (logarithmic dependence on the confidence level $\\beta$ instead of polynomial) than naïve applications of Markov's inequality to results in expectation. \n- For the case of $\\nu = 1$ (Lipschitz gradient), the claimed contribution over the existing results of Gorbunov et al. [14] is that the problem need only be smooth in a ball around a minimum. \n\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Update:**\n\nI thank the authors for their response. It addressed some of my concerns and I have increased my score. \n\nFor a potential revision, I would recommend adding emphasis on the techniques used for the theoretical contribution (what are the key result needed to relax the assumptions, and what is the insight used in the proof), and discussing what practical problems can now be solved.\n\n\n\n---\n\nI think the contribution of the submission that is most likely to be of interest to the community is the $\\beta$-independent results on non-smooth functions. Given the assumptions and selected parameters, the results are believable although there is limited technical details in the main paper. The presentation in terms of Holder-continuity and the discussion of the need to be smooth only around the minimizer, although more general, might be of limited interest. The manuscript suffers from additional clarity issues around its motivations and does not make obvious what the new key idea or contribution is, beyond listing results. \n\n**Detailled comments**\n\n- Definitions of heavy-tails\n\n\tFrom the introduction, it is not clear what type of gradient noise/heavy-tail model the submission consider. The introduction motivates the papers of Simsekli et al. and Zhang et al. ([34, 35, 41]) who study heavy-tailed noise in NLP models. However, those papers use a definition of heavy-tailed noise that imply that the second-moment is unbounded, contrary to the assumption in Eq. (2). I assume the definition of Heavy-tailed used in this submission is any distribution that is not sub-Gaussian but still satisfies the bounded variance assumption? Could the authors could add an example of a problem with heavy-tailed distribution that fits the bounded variance criteria?\n\n- Missing key idea; \n\n\tThe main technical contribution is controlling the error induced by stochasticity, without light-tails, through the clipping operator and controlling its bias. While this is briefly mentioned in Section 2, p5, the technical details are left vague and relegated to the appendix.\n\n\tSimilarly, while smoothness (or Holder-contuinuity) on a ball is technically weaker than smoothness on the whole space, I do not see what new ideas or proof techniques are needed to drop this assumption. It seems like a distinction without a difference, but if new ideas are needed, those should be given to the reader. \n\n- Limitations and parameter settings\n\n\tFrom the appendix, it seems that the algorithm uses an increasing batch size. I understand there is a paragraph arguing that, given a target precision $\\epsilon$, one can choose parameters and a sufficiently small step-size to ensure batch-size of 1.\n\tHowever, this is not what I think of when I think of convergence guarantees as running the algorithm longer does not improve the error. \n\t\n\tA comment explaining those limitations in the main paper, and whether they are necessary and/or common in the high-probability litterature would help the reader. \n\n\tAlso, a clarification would help: I cannot figure out how to set the parameters of the method. For example, in the setting of Thm 2.1 with $\\nu = 1$, $N$ is taken to be dependent to a constant $a$ through $\\propto \\sqrt{a}$ in Eq. (39) (ignoring the $+1$ and $\\lceil\\cdot\\rceil$) but right after, in line 570, $a$ is taken to be $= 2^{14} \\log(4N/\\beta)$. It seems those two cannot hold at the same time?\n\t\n- Space issues\n\n\tI realise the above requests for additional details would take space. If the authors are looking for some, I would suggest moving the experimental section to the appendix. In its current state, it seems disconnected from the theoretical contributions of the submission. While they run the algorithms studied here, the experimental setting do not match the theoretical framework and I do not understand what the experiments are supposed to show beyond that the algorithms can be made to work with step-size tuning.\n\t\n\t\n\n\n**Minor comments**\n\n- Why does the entry for SGD in Tables 1 and 2, with and without ♣, have different constraints on the unbounded domain column? From my understanding, the heavy-tail version (♣) is obtained through a Markov inequality on the first\n- I do not understand how Alg. 3 is \"a new variant of clipped-SGD [30] properly adjusted to the class of objectives with $(\\nu, M_\\nu)$-Hölder continuous gradients\" (207). The algorithm looks just like standard clipping and does not depend on $\\nu$?\n- Please cite published works instead of arXiv submissions. [20] was published at ICLR'15, [35] at ICML'19, [38] in ACL'19.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "6flYR-GyEbp", "reviewer": "Reviewer_bkmn", "summary": "The paper tackles the important problem of high-probability convergence bounds for machine learning optimisation problems. Traditional analyses are done in expectation, which have been shown to not accurately represent all possible training outcomes; hence the need for tighter studies.\nIn this paper, in the specific setting of convex, non-smooth losses with heavy tailed noise, a new high-probability bound is introduced for two algorithms. What sets it apart from previous bounds is that there is no direct negative dependency on the confidence bound (only logarithmic ones); and that in some cases it matches the non-heavy tailed noise bound.\nNumerical experiments illustrating the theoretical results are presented.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is part of a rich literature tackling an important problem for machine learning practitioners: is my optimisation algorithm guaranteed to converge? The motivation is convincing, the related work appears to be thorough, and the results are nicely contrasted with previous work. The paper is nicely written, and the theoretical results represent a significant advance over previous work. One small note though is that given a whole new page for the submission, a small conclusion/discussion would not have gone amiss. As it stands, the paper ends rather abruptly.\n\nThe paper does present some weaknesses, including two serious ones. \n\nFirst, the problem setting is not as general as some elements of the presentation makes it sound. Crucially, the objective function has to be convex; which automatically exclude all but the simplest neural networks. Of course, this means the results are not as significant as they would be if they covered non-convex objectives (which is not to say that they're not significant; I do believe they are).\nThis restriction should be more prominently featured, e.g. in the title which mentions all other characteristics but this one. The insistence on neural network applications is also a bit problematic, as those problems are non-convex (e.g. BERT training is mentioned in the introduction, and it's only on the 9th page that it appears clearly that the BERT application is only for a convex finetuning problem). At the very least this would have to be plainly stated. As it stands, a careless reader might think that these theoretical results apply to their favorite neural network application, which they unfortunately do not.\nOn a similar note, insisting on non-smooth might not be the best idea since the analysis covers various amounts of smoothness including smooth functions.\n\nThe second, related, issue, is that the experimental section does not properly illustrate the theoretical results. We see that the methods seem to perform fine on a convex, somewhat artificial setting (only finetuning enough so that the problem remains convex) in NLP, and the vision application is not convex at all. The claim is then that the methods work well on heavy-tailed noise distribution and are less efficient on light-tailed ones, but it might also be that they simply work worse on non-convex problems? As it stands it's hard to disentangle these different aspects.\nWhat would ideally be presented is results on convex problems for both heavy and light-tailed noise distributions; together with potentially non-convex applications to illustrate that the methods can perform well there even if it's outside the scope of the theoretical analysis.\nOn a related point, the SGD used in the experiments uses momentum, contrary to the SGD that is analysed. Again, this gap in between theory and practice makes both results less relevant.\n\nA less serious issue is the fact that the strongly convex results still rely on a uniformly bounded gradient variance assumptions, which I think is contradictory unless the assumption only applies to a bounded set? While it might be the case that the assumption is only necessary on a bounded set, I haven't seen this clearly in the paper.\n\nAn interesting point is that in the analysed setup, the effective clipping constant goes up (as the learning rate typically goes down over time). This is rather counterintuitive: at the later stages of training one expects the gradient norms to go down rather than up, which means we could use smaller clipping factors, not larger.\nThis leads me to the following question: is this a practical recommendation, or rather an artefact of the proof? Having a larger clipping constant helps for the proof since it reduces the bias. But on the other hand, since we increase the clipping constant in the later stages of training, where we expect comparatively lower gradient norms, are we really clipping at all anymore?\n\nAll told, this is a nice if somewhat borderline paper. I believe it could be significantly improved by following the previous recommendations, and I am open to improving my score, should fixes materialise.\n\n\nThere are a number of smaller elements that should be fixed to improve the paper, which I list below\n- in terms of clarity it would be helpful to compare the results with fixed \\nu (either 0 or 1), because as it stands understanding the actual dependencies on various parameters is quite arduous. It would also make the message clearer.\n- l84, the sentence containing the equation is missing a verb?\n- the exponent on the logarithm (see equation l83 for instance) is a bit unclear. If it's indeed log(x)^y it'd be easier to read written this way explicitly rather than log^y(x), as the exponent is itself quite big.\n- l176 alpha_k is only defined in the algorithm box. It should also be defined in the main text\n- on a related topic, the step size is alpha for clipped-SSTM and gamma for clipped-SGD. Some consistency here would be appreciated by the reader, once again.\n- l175 that --> which\n- in algorithm 3, line 2 the gradient at x^{k+1} is computed but then l3 it's at x^k, which I'm assuming is a typo.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I see no particular potential negative impact of this work, besides those that are standard for theoretical machine learning papers. I don't think this deserves expanding upon.\nLimitations could probably use a little bit more work, as detailed in the main review.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "-w5ay9WJ8Z6", "reviewer": "Reviewer_BTy2", "summary": "This paper analyzes the clipped SGD and clipped SSTM algorithm on generalized smooth objectives with Hölder-continuous gradients. They consider the case where the assumption of sub-Gaussian noise is not needed. They modified the stepsize in the clipped SSTM algorithm to fit the Hölder-continuity assumption. Theoretical analysis is conducted with convergence rates presented. ", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is well-written. I have several comments below.\n\n1. They analyze the clipped-SSTM, clipped-SGD (with/without restart trick) algorithms. These algorithms are proposed in Gorbunov et al. (2020) , and the authors extend their assumption from Lipschitz continuous gradient ($\\nu=1$) to Hölder continuous gradients ($\\nu \\in [0,1]$). They only make one slight modification to the the clipped-SSTM algorithm which fits the Hölder continuous gradient assumption. From my perspective, this paper is a direct extension of Gorbunov et al. (2020) with the same algorithms and similar theoretical analysis. It is interesting to include the case when $\\nu=0$ and the objective function may not be differentiable, but the novelty of this paper is still questionable.\n\n2. The numerical experiments are irrelevant to the theoretical parts. In theorems, the assumption of convexity is needed where in the experiments, the authors consider two non-convex tasks. This paper is focused on the theoretical analysis with relaxed assumption for convex optimization. It seems irrelevant to run these existing algorithms on some non-convex optimization tasks, especially for the clipped-SGD algorithm, which is a classical method for deep learning tasks in the differential privacy literature. It would be better if the authors consider some tasks with convex  and possibly non-smooth objective function and check if the empirical convergence rate matches the one in their analysis.\n\nReferences mentioned above:\n\n1. Gorbunov, Eduard, Marina Danilova, and Alexander Gasnikov. \"Stochastic Optimization with Heavy-Tailed Noise via Accelerated Gradient Clipping.\" *Advances in Neural Information Processing Systems* 33 (2020): 15042-15053.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "None.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: There was a lot of discussion around this papers. The reviewers were happy with many points brought up by the authors, such as the way Adam was tuned. I also discussed the paper with the senior area chair. Ultimately, the main concern was the novelty of the results compared to Gorbunov et al. (2020). The paper at least needs a re-write to emphasize the contributions. It was also pointed that experiments were included, but weren't designed to isolate the novel assumptions. While it is true that papers can be accepted without experiments on the strength of their theory alone, the reviewers did not feel that the theoretical results were sufficiently-different for that to be the case here. 

Decision: Reject