Better Theory for SGD in the  Nonconvex World
Authors: Ahmed Khaled
Peter Richtárik
Venue: Accepted by TMLR ()

Abstract: Large-scale nonconvex optimization problems are ubiquitous in modern machine learning, and among practitioners interested in solving them, Stochastic Gradient Descent (SGD) reigns supreme. We revisit the analysis of SGD in the nonconvex setting and propose a new variant of the recently introduced \emph{expected smoothness} assumption which governs the behavior of the second moment of the stochastic gradient. We show that our assumption is both more general and more reasonable than assumptions made in all prior work. Moreover, our results yield the optimal  $\mathcal{O}(\epsilon^{-4})$ rate for finding a stationary point of nonconvex smooth functions, and recover the optimal $\mathcal{O}(\epsilon^{-1})$ rate for finding a global solution if the Polyak-Łojasiewicz condition is satisfied. We compare against convergence rates under convexity and prove a theorem on the convergence of SGD under Quadratic Functional Growth and convexity, which might be of independent interest. Moreover, we perform our analysis in a framework which allows for a detailed study of the effects of a wide array of sampling strategies and minibatch sizes for finite-sum optimization problems. We corroborate our theoretical results with experiments on real and synthetic data.

Published in Transactions Machine Learning Research (02/2023)
Better Theory for SGD in the Nonconvex World
Ahmed Khaled
alamed Shaneedddoale@@@ppeeeeoo....c.
Princeion University
Peter Richtárik
peter Wkaustt edu..s
King Abdullah University of Science and Technology
Reviewed OpenReview: https: // opwwrevee...e.eee net/ forum? id- gU4
Abstract
Largo-scale nonconvex optimization problems are ubiquitous in modern machine learning, and among practitioners interested in solving them, Stochastic Gradient Descent (SGD) reigns supreme. We revisit the analysis of SGD in the nonconvex setting and propose a new variant of the recently introduced erpected smoothness assumption which governs the behavior of the second moment of the stochastic gradient. We show that our assumption is both more general and more reasonable than assumptions made in all prior work. Moreover, our results yield the optimal O) rate for finding a stationary point of nonconvex smooth functions, and recover the optimal O==-l) rate for finding a global solution the condition satisfied. We compare against convergence rates under convexity and prove a theorem on the convergence of SGD under Quadratic Functional Growth and convexity, which might be of independent interest. Moreover, we perform our analysis in a framework which allows for detailed study of the effects of wide array of sampling strategies and minibatch sizes for finite-sum optimization problems. We corroborate our theoretical results with experiments on real and synthetic data.
Introduction
In this work we study the complexity of stochastic gradient descent (SGD) for solving unconstrained optimization problems of the form
min f(x).
(1)
ZER
where RRd + IR is possibly noncomeez and satisfies the following smoothness and regularity conditions. Assumption 1. Function f is bounded from below by an infimum finf e R, differentiable, and V is L--Lischitz:: for all v, y E RRd we have [[V f s L || - yll-
Motivating this problem is perhaps unnecessary. Indeed, the training of modern deep learning models reduces to nonconvex optimization problems, and the state-of-the-art methods for solving them are all variants of SGD (Salakhutdinov, 2014; Sun, 2020). SGD is randomized first-order method performing iterations of the form
Th+1 = 2x - -Txgg(xx),
(2)
where g(x) an unbiased estimator of the gradicnt Vz(x) (i.c., E -f())) and N an appropriately chosen learning rate. Since f can have many local minima and/or saddle points, solving (1) to global optimality intractable (Nemirovsky & Yudin, 1983; Vavasis, 1995). However, the problem becomes tractable one
scales down the roquirements on the point of interest from global optimality to some relaxed version thereof, such as stationarity or local optimality. In this paper are interested in the fundamental problem of finding an e-stationary point, i.., we wish to find a random vector & E Idd for which E [|V](=))? the expectation over the randomness of the algorithm.
< E², where IE [-]
1

---

Published in Transactions Machine Learning Research (02/2023)
1.1 Modelling stochasticity
Since unbiasedness alone is not enough to conduct complexity analysis of SGD, it is necessary to impart further assumptions on the connection between the stochastic gradient g(x)) and the true gradient V fxr). The most commonly used assumptions take the form of various structured bounds on the second moment of g(x). We argue (see Section 3) that bounds proposed in the literature are often too strong and unrealistic as they do not fully capture how randomness in g(æ) arises in practice. Indeed, existing bounds are primarily constructed in order to facilitate analysis, and their match with reality often takes the back seat. In order to obtain meaningful theoretical insights into the workings of SGD, very important model this randomness both correctly, that the assumptions we impart are provably satisfied, and accurately, so to obtain as tight bounds as possible.
1.2 Sources of stochasticity
Practical applications of SGD typically involve the training of supervised machine learning models via empirical risk minimization (Shalev-Shwartz & Ben-David 2014), which leads to optimization problems of finite-sum structure:
f(x) def Σ (1x)) f(())
min ZERd
(3)
In a single-machine setup, n the number training data points, and fi()) represents the loss of model x on data point i. In this setting, data access is expensive, and g(z) typically constructed via subsampling techniques such as minibatching (Dekel et al., 2012) and importance sampling (Needell et al., 2014). In the rather general arbitrary sampling paradigm (Gower et al., 2019), one may choose an arbitrary random subset S S [n]] of examples, and subsequently (zx) is assembled from the information stored in the gradients V (z) for i E S only. This leads to formulas of the form
g(z) = Σ vV)(),) ÈS
(4)
where v, are appropriately defined random variables ensuring unbiasodness.
In distributed setting, n corresponds to the number of machines (e.g., number of mobile devices in federated learning) and f(() represents the loss of model x all the training data stored on machine i. In this setting, communication is expensive, and modern gradient-type methods therefore rely on various randomized gradient compression mechanisms such as quantization (Gupla et al., 2015), sparsification (Wangni et al., 2018), and dithering (Alistarh et al., 2017). Given an appropriately chosen (unbiased) randomized compression map Q Rd + Rdd the local gradients (z( are first compressed to (x),, where Q, is an independent instantiation of Q sampled by machine in each iteration, and subsequently communicated to a master node, which performs aggregation (Khirirat et al., 2018). This gives rise to SGD with stochastic gradient of the form
g(x) - Σ Q(VA(())) n
In many applications, each f; has finite sum structure of its own, reflecting the empirical risk composed of the training data stored that device. In such satuasions, it is often assumed that compression is not applied to exact gradients, but to stochastic gradients coming from subsampling (Gupta et al., 2015; Ben-Nun & Hoefler, 2019; Horváth et al., 2019). This further complicates the structure of the stochastic gradient.
Contributions
The highly specific and elaborate structure of the stochastic gradient g(z) used in practice, such as that coming from subsampling as (4) or compression as in (5), raises questions about appropriate theoretical modelling of its second moment. As we shall explain in Section 3, existing approaches do not offer a satisfactory treatment.
2

---

Published in Transactions Machine Learning Research (02/2023)
(GC)
(M-SG) (F-SG)
(RG)
(ES)
(BV)
(SS)
Figure 1: Assumption hierarchy the second moment of the stochastic gradient of nonconvex functions. An arrow indicates implication. Our newly proposed ES assumption is the most general. The statement is formalized as Theorem 1.
Indeod, as we show through a simple example, none of the existing assumptions are satisfied even in the simple scenario of subsampling from the sum of two functions (see Proposition 1).
Our work is motivated by the need of more accurate modelling of the stochastic gradient for nonconvex optimization problems, which we argue would lead to a more accurate and informative analysis of SGD in the nonconvex world a problem of the highest importance in modern deep learning.
The key contributions of our work are:
Inspired by recent developments in the analysis of SGD in the (strongly) convex setting (Richtárik & Takáč, 2017; Gower et al., 2020; 2019), propose new assumption, which we call expected smoothness (ES), for modelling the second moment of the stochastic gradient, specifically focusing on nonconvex problems (sce Section 4). In particular, we assume that there exist constants A, B, C 2 0 such that
E [1[())]]]] S (A(x)) f'n() B[]
We show Scction 4.3 that (ES) the weakest, and hence the most general, among all assumptions in the existing literature we are aware of (see Figure 1), including assumptions such bounded variance (BV) (Ghadimi & Lan, 2013), maximal strong growth (M-SG) (Schmidt & Roux, 2013), strong growth (SG) (Vaswani et al., 2019), relaxed growth (RG) (Bottou et al., 2018), and gradient confusion (GC) (Sankararaman et al., 2020), which we review in Section 3.
Moreover, we prove that unlike existing assumptions, which typically implicitly assume that stochas- ticity comes from perturbation (see Scction 4.4), (ES) automatically holds under standard and weak assumptions made on the loss function in settings such as subsampling (see Section 4.5) and compression (see Section 4.6). In this sense, (ES) not an assumption but an inequality which provably holds and can be used to accurately and more precisely capture the convergence of SGD. For instance, the best of our knowledge, while the combination of gradient compression and subsampling is not covered by any prior analysis of SGD for nonconvex objectives, our results can be applied to this setting.
We recover the optimal O(e-4) rate for general smooth nonconvex problems and ((s rale under the PL condition (see Section 5). However, our rates are informative enough for us to be able to doduce, for the first time in the litcrature on nonconvex SGD, importance sampling probabilitics and formulas for the optimal minibatch size (see Section 6).
3 Existing Models of Stochastic Gradient
Ghadimi & Lan (2013) analyze SGD under the assumption that f is lower bounded, that the stochastic gradients g(x) are unbiased and have bounded variance
[|((=)) ff(x) s0²
(BV)
After writing this mamuscript, we became aware early work (Polyak & Tsypkin, 1973) that had a similar assumption. We discuss this Section
3

---

Published in Transactions Machine Learning Research (02/2023)
Note that due to unbiasedness, this is equivalent to
E [100=)]]]] = [[V +
(6)
With an appropriately chosen constant stepsize 7, their results imply O(E rate of convergence.
In the context of finite-sum problems with uniform sampling, where at each step i [n] is sampled uniformly and the stochastic gradient estimator used is g(x) Vf((z) for randomly selected index the maximal strong growth condition requires the inequality
Il.()))l S o: <
(M-SG)
to hold almost surely for some 2 0. Tseng (1998) used the maximal strong growth condition early to establish the convergence of the incremental gradient method, closely related variant of SGD. Schmidt & Roux (2013) prove the linear convergence of SGD for strongly convex objectives under (M-SG).
We may also assume that (M-SG) holds in expectation rather than uniformly, leading to expected strong growth:
E [|]())]]]] a [V]()]]
(E-SG)
for some a > O. Like maximal growth, variants of this condition have also been used in convergence results for incremental gradient methods (Solodov, 1998). Vaswani al. (2019) prove that under (E-SG), SGD converges to an e-stationary point in (e)-2) steps. This assumption quite strong and necessítates interpolation: fz()) 0, then g(x) = 0 almost surely. This is typically not true in the distributed, finite-sum case where the functions fi can be very different; e.g., (McMahan et al., 2017).
Bottou et al. (2018) consider the relaxed growth condition which is a version of (F-SG) featuring additive constant:
E [|o[))1|]] a [[ + 3.
(RG)
In view of (6), (RG) can be also seen a slight gencralization of the bounded variance assumption (BV). Bertsekas & Tsitsiklis (2000) establish the almost-sure convergence of SGD under (RG), and Bottou et al. (2018) give a (e convergence rate for nonconvex objectives to a neighborhood a stationary point of radius linearly proportional to B. Unfortunately, (RG) is quite difficult to verify practice and can be shown not hold for some simple problems, as the following proposition shows.
Proposition There is a simple finite-sum minimization problem with two functions for which (RG) is not satisfied.
The proof of this proposition and all subsequent proofs are relegated to the supplementary material.
In recent development, Sankararaman al. (2020) postulate gradient confusion bound for finite-sum problems and SGD with uniform single-element sampling. This bound requires the existence of in > 0 such that
(V..() Vzy(z)) -
(GC)
holds for all * j (and all x E Rd). For general nonconvex objectives, they prove convergence to a. neighborhood only. For functions satisfying the PL condition (Assumption 5), they prove linear convergence to a neighborhood of stationary point.
Lei et al. (2019) analyze SGD for of the form
f(x) Eş [f(())]
They assume that fg nonnegative and almost-surely Hölderr-ontinuuus in and then use g(z) fz()x) for & sampled i..d. from D. Specialized the case of L-smoothness, their assumption reads
(1g(x) - g(v)ll s L || vll and fe(x) 2
(SS)
4

---

Published in Transactions on Machine Learning Research (02/2023)
almost surely for all y, E RR. We term this condition sure-smoothness (SS). They establish the sample complexity log==)) Unfortunately, their results do not recover full gradient descent and their analysis is not easily extendable to compression and subsampling.
We summarize the relations between the aforementioned assumptions and our own (introduced in Section 4) in Figure 1. These relations are formalized as Thcorem
4 ES in the Nonconvex World
In this section, we first briefly review the notion of expected smoothness as recently proposed in several contexts different from ours, and use this development to motivate our definition of expected smoothness (ES) for nonconvex problems. We then proceed to show that (ES) is the weakest from all previous assumptions modelling the behavior of the second moment of stochastic gradient for nonconvex problems reviewed in Section 3, thus substantiating Figure 1. Finally, we show how (ES) provides correct and accurale model for the behavior of the stochastic gradient arising not only from classical perturbation, but also from subsampling and compression.
4.1 Brief history of expected smoothness: from convex quadratic to convex optimization
Our starting point is the work of Richtárik & Takáč (2017) who, motivated by the desire obtain deeper insights into the workings of the sketch and project algorithms developed by Gower & Richtárik (2015), study the behavior of SGD applied to a reformulation of a consistent linear system as stochastic convex quadratic optimization problem of the form
min f(x) def E [fe(x)] agR
The above problem encodes linear system in the sense that f(x) is nonnegative and equal to zero if and only if æ solves the linear system. The distribution behind the randomness in their reformulation (8) plays the role of a parameter which can be tuned in order to target specific properties of SGD, such as convergence rate or cost per iteration. The stochastic gradient g(x) = (z())) satisfies the identity [[la(x)]1] = 2f(x), which plays a key role in the analysis. Since their setting g(x+,) = 0 almost surely for any minimizer x, (which suggests that their problem is over-paraameterizel), the above identily can be written in the equivalent form
[ 2 (f(x) (f())=((xx)) f(x.))
Equation (9) the first instance of the expected smoothness property/inqquality we are aware of. Using tools from matrix analysis, Richtárik & Takáē (2017) are able to obtain identities for the expected iterates of SGD. Kovalev al. (2018) study the same method for spectral distributions and for some of these establish stronger identities of the form E ||ara - - 1/1 ==1 suggesting that the property (9) has the capacity to achieve a perfectly precise mean-square analysis of SGD in this setting.
Expected smoothness as an inequality was later used to analyze the JacSketch method (Gower et al., 2020), which general variance-reduced SGD that includes the widely-used SAGA algorithm (Defazio al., 2014) as special case. By carefully considering and optimizing over smoothness, Gower et al. (2020) obtain the currently best-known convergence rate for SAGA. Assuming strong convexity and the existence global minimizer x,, their assumption in our language reads
E [10(()) - g(2.)) s(x,))1 s 2A(((2)) ff(x)),
(C-ES)
where A 2 0, g(a)) is a stochastic gradient and the expectation is w.r.t. the randomness embedded in g- We refer to the above condition by the name conver expected smoothness (C-ES) as it provides a good model of the stochastic gradient of convex objectives. (C-ES) was subsequently used to analyze SGD for quasi strongly-convex functions by Gower et al. (2019), which allowed the authors to study a wide array of subsampling strategics with great accuracy as well as provide the first formulas for the optimal minibatch size
5

---

Published in Transactions Machine Learning Research (02/2023)
for SGD in the strongly convex regime. These rates tight up to non-problem specifie constants in the setting of convex stochastic optimization (Nguyen et al., 2019).
Independently and motivated by extending the analysis of subgradient methods for convex objectives, Grimmer (2019) also studied the convergence of SGD under a similar setting to (C-ES) (in fact, their assumption is (10) below). Grimmer (2019) develops quite general and clegant thoory that includes the use of projection operators and non-smooth functions, but the convergence rale they oblain for smooth strongly convex functions shows suboptimal dependence on the condition number.
4.2 Expected smoothness for nonconvex optimization
Given the utility of (C-ES), it is then natural to ask: can we ertend (C-ES) beyond converity? The first problem we are facod with that x, is ill-defined for nonconvex optimization problems, which may not have any global minima. However, Gower al. (2019) use (C-ES) through following direct consequence of (C-ES) only:
E [|o(2))1 - 20²
(10)
where 0² def = [[1a(x))1*]] We thus propose to remove I, and merely ask for a global lower bound fiuf on the function f rather than a global minimizer, dispense with the interpretation of o² as the variance at the optimum v, and merely ask for the existence any such constant. This yields the new condition
E [1s())]]] S - +C,
(11)
for some A,C 2 O. While (11) may be satisfactory for the analysis of convex problems, it does not enable us to easily recover the convergence of full gradient descent or SGD under strong growth in the case of nonconvex objectives. As we shall see further sections, the fix to add third term to the bound, which finally leads to our (ES) assumption:
Assumption 2 (Expected smoothness). The second moment of the stochastic gradient satisfies
E
[1(x))]]] (fA() - f + B (V)()
(ES)
for some A, B, C2 O and all Rd.
After we wrote this manuscript, we became aware the very early work (Polyak & Tsypkin, 1973). They considered similar assumption while analyzing pseudogradient algorithms, and they established an asymptotic convergence bound for variant of gradient descent. We arrived at this assumption independently and via the different motivation outlined above, and our proof techniques aimed towards finding non-asymptotic convergence rates. We believe the rediscovery of this assumpt ion is testimony to its utility, and our work significantly extends this early work with non-asymptotic convergence rates in several settings and other new contributions. Appendix gives a more detailed overview of the assumption used by Polyak & Tsypkin (1973).
In the rest of this section, we turn the generality of Assumption 2 and its use in correct and accurate modelling of sources of stochasticity arising in practice.
4.3 Expected smoothness as the weakest assumption
As discussed in Section 3, assumptions on the stochastic gradients abound in the literature on SGD. we hope for a correct and tighter theory, then we may ask that we at least recover the convergence of SGD under those assumptions. Our next theorem, described informally below and stated and proved formally in the supplementary material, proves exactly this.
Theorem (Informal). The expected hnest assumption (Assumption 2) the weakest condition among the assumptions in Scction 3.
6

---

Published in Transactions Machine Learning Research (02/2023)
4.4 Perturbation
One of the simplest models of stochasticity is the case of additive zero-mean noise with bounded variance, that is
g(x) Vf(x)) +.
where { is a random variable satisfying [<] = 0 and E [|E|]] S so?. Because the bounded variance condition (BV) is clearly satisfied, then by Theorem 1, our Assumption 2 is also satisfied. While this model can be useful for modelling artificially injected noise into the full gradient (Ge et al., 2015; Fang et al., 2019), it is unrcasonably strong for practical sources of noise: indeed, as we saw in Proposition 1, it does not hold for subsampling with just two functions. It. furthermore unable model sources of rather simple mulliplicatiive noise which arises in the case of gradient compression operators.
4.5 Subsampling
Now consider f having a finite-sum structure (3). In order to develop a general theory of SGD for a wide array of subsampling strategics, we follow the stochastic reformulation formalism pioneered by Richtárík & Takáë (2017); Gower et al. (2020) in the form proposed in (Gower et al., 2019). Given a sampling vector E _ drawn from some user-defined distribution D (where a sampling vector is one such that Ep [o:] = 1
for all i E [n]]), we define the random function fu(x) def 1 Σ v.f((()). Noting that ENwD [f(( = f(x), we
reformulate (3) as a stochastic minimization problem
1 Σ (uffic)) )):
min
EVnD [fo(z)] = EUND
(12)
ZER
where we assume only access to unbiased estimates of f(x)) through the stochastic realizations
1 (z(.).
Vf((z)=
(13)
That is, given current point x, we sample ~ D and set g(x) def (f(()). We will now show that (ES) is satisfied under very mild and natural assumptions the functions fo and the sampling vectors u.. In that sense, (ES) is not an additional assumplion, it is an inequalily that is automatically satisfied.
Assumption 3. Fach fi is bounded from below by siof and is L-smooth: That is, for all xr, Red have
L lly x11
f(()) - fl(z) + (Vf()), z)
2
To show that Assumption 2 is an automatic consequence of Assumption 3, we rely on the following crucial lemma. Lemma 1. Let f be function for which Assumption 1 is satisfied. Then for all Rd have
||]f())] s 2L (f(x)- finl).
(14)
This lemma shows up in several recent works and is often used in conjunction with other assumptions such as bounded variance (Li & Orabona, 2019) and convexity (Stich & Karimireddy, 2019). Lei et al. (2019) also use version of it to prove the convergence of SGD for nonconvex objectives, and compare our results against theirs in Section 5. Armed with Lemma 1, can prove that Assumption 2 holds for all non-degenerate distributions D.
Proposition 2. Suppose that Assumption 3 holds and that E [o]] is finite for all i. Let. A def fiul. Then A and Assumption 2 holds with A max; E [e]] and C 24Aini inf
- iin
7

---

Published in Transactions Machine Learning Research (02/2023)
The condition that E [u]] subsampling schemes in the literature. However, the generality of Proposition 2 comes at cost: the bounds are too pessimistic. By making more specific (and practical) choices of the sampling distribution D, we can get much tighter bounds. We do this by considering some representative sampling distributions next, without aiming to be exhaustive:
is finite is very mild condition D and is satisfied for virtually all practical
Sampling with replacement. An n-sided die is rolled a total of r > 0 times and the number of times the number shows up is recorded as S, We can then define
S.
2 Tq
(15)
where qi is the probability that the i-th side of the die comes up and Σ 9i 1. In this case, the number of stochastic gradients queried is always t though some of them may be repeated.
Independent sampling without replacement. We generate random subset S {1,,, ...,,,.. and define
LIES
U2
(16)
Pi
where lies = 1 if E S and otherwise, and = Prob (i S) > O. We assume that each number is included in S with probability ps independently of all the others. In this case, the number of stochastic gradients queried |S| is not fixed but has expectation E [[S[]] E
nine sampling without replacement. This is similar to the previous sampling, but we generate random subset S C {1, ...., n by choosing uniformly from all subsets of size for integer E [L, n]. We define U; as in (16) and it easy to see that p; = I for all i.
These sampling distributions were considered in the context of SGD for convex objective functions in (Gorbunov et al., 2020; Gower et al., 2019). We show next that Assumption is satisfied for these distributions with much better constants than the generic Proposition 2 would suggest. Proposition 3. Suppose that Assumptions and hold and let Ainf + D f!al). Then:
For independent sampling with replacement, Assumption 2 is satisfied with A = max; Tnge and inf
L.
B = 1 7
(ii) For independent sampling without replacement, Assumption 2 is satisfied with = max; (1-piLe 1, and C' 2AAinf
(iii) For 7-nice sampling without replacement, Assumption 2 is satisfied with 8-7 7(n-1)
max, Li,
-(r-1) and C = in.
T(=
4.6
Compression
We now further show that our framework is general enough to capture the convergence of stochastic gradient quantization or compression schemes. Consider the finite-sum problem (3) and let (xr)) be stochastic gradients such that E [g:(=)] = fi(x).. We construct an estimator g(x) via
g(x) Σ (x))(()),
(17)
where the Qi are sampled independently for all and across all iterations. Clearly, this generalizes (5). We
consider the class of w-compression operators:
Qe Rd + Rd is an w-compression operator if Assumption 4. We say that stochastic operator
Eg [[]( =z, Es [x))(()) ==1²] s w||x||?
(18)
8

---

Published in Transactions Machine Learning Research (02/2023)
Assumption is mild and is satisfied by many compression operators in the literature, including random dithering (Alistarh et al., 2017), random sparsification, block quantization (Horváth et al., 2019), and others. The next proposition then shows that if the stochastic gradients gi(x) themselves satisfy Assumption 2 with their respective functions fi. then g(z) also satisfies Assumption 2.
Proposition 4. Suppose that stochastic gradient estimator g(z) is constructed via (17) such that each º is wi-compressor satisfying Assumption 4. Supposc further that the stochastic gradient 9:(2) is such that E [x:] = V (x(r)) and that each satisfies Assumption 2 with constants (A,, B, C₃). Then there exists constants A, B, C 2 0 such that g(x) satisfies Assumption 2 with (A, B)C)-
To the best of our knowledge, the combination of gradient compression and subsampling is not covered by any analysis of SGD for nonconvex objectives. Hence, Proposition 4 shows that Assumption 2 is indeed versatile enough to model practical and diverse sources of stochasticity well.
SGD in the Nonconvex World
5.1 General convergence theory
Our main convergence result relies on the following key lemma.
Lemma 2. Suppose that Assumptions and 2 satisfied. Choose constant stepsize ? 0 such that
y S 1 BL
Then,
k
Wirk + WK-1 8R S
W-
LC 2
K-1
80 +
wk7
2
k=U
where Th def E [ [V
def
w for w-1 arbitrary, and oR def [[(x)))]] fint. (1+L)A))*
We
Lemma 2 bounds a weighted sum of stochastic gradients over the entire run of the algorithm. This idea of weighting different iterates has been used in the analysis of SGD in the convex case (Rakhlin et al., 2012; Shamir & Zhang, 2013; Stich, 2019) typically with the goal of returning weighted average of the iterates Fx at the end. In contrast, we only use the weighting to facilitate the proof.
Theorem 2. Suppose that Assumptions and 2 hold. Suppose that a stepsize 7 > 0 is chosen such that S LB. Letting do def f(xo) - fiul, we have
min E [IVI([N)1 ICy+ 2(1+ 80. 0sksK-1
gK
While the bound of Theorem 2 shows possible exponential blow-up, we can show that by carefully controlling the stepsize we nevertheless attain e-stationary point given O(E-4) stochastic gradient evaluations. This dependence is in fact optimal for SGD without extra assumptions on second-order smoothness or disruptiveness of the stochastic gradient noise (Drori & Shamir, 2020). We use similar stepsize to Ghadimi & Lan (2013).
Corollary 1. Fix & >> 0. Choose the stepsize > 0 as 7 min VI.AK LB 2fc) Then provided that
1 1
125
125g A 2C
K 2 max
B,
(19)
22
g2
we have E[ s.
As a start, the iteration complexity given by (19) recovers full gradient descent: plugging in B = 1 and A = C 0 shows that we require a total of 126,Le iterations in required to a reach an e-stationary point. This is the standard rate of convergence for gradient. descent on nonconvex objectives (Beck, 2017), up to absolute (non-problem-speeffic)) constants.
9

---

Published in Transactions Machine Learning Research (02/2023)
Plugging in A C 0 and B to be any nonnegative constant. recovers the fast convergence of SGD under strong growth (E-SG). Our bounds are similar to Lei et al. (2019) but improve upon them by recovering full gradient descent, assuming smoothness only in expectation, and attaining the optimal O(s rate without logarithmic terms.
5.2 Convergence under the Soyaakkojjsiewicz condition
One of the popular generalizations strong convexity in the literature the Polyak-Eojasicwicz (PL.) condition (Karimi et al., 2016; Lei al., 2019). We first define this condition and then establish convergence of SGD for functions satisfying it and (FS) assumption. In the rest of this section, we assume that. the function f has minimizer and denote f def min
Assumption 5. We say that differentiable function satisfies the Polyak-Lojasiesn condition if for all Rd
[+(()) 2((())
We rely on the following lemma where use the stepsize sequence recently introduced by Stich (2019) but without iterate averaging, as averaging in general may not make sense for nonconvex models.
Lemma 3. Consider sequence (rr); satisfying
T1+1 S (1 (1)a)) rI c72
(20)
where " < for all t 2 0 and a, 2 O with S b. Fix K > 0 and let ko 1. Then choosing the stepsize as
if K b
ko:
7
if K 2 b and t >> ko
a(a+t-Ro)
with 2b gives rK- exp ro
a
Using the stepsize scheme Lemma 3, we can show that SGD finds an optimal global solution at 1/K rate, where K is the total number of iterations.
Theorem 3. Suppose that Assumptions 1, 2, and 5 hold. Suppose that SGD is run for K iterations with the stepsize sequence (TK)& of Lemma with Ys S min (ztz (bz) for all k. Then
C 2μK
+ exp (-2x) 2x max (Ks, B} (f(20)-
K
[[f(xk) - ff
where Kj def L/p is the condition number of and Ks def A/u is the stochastic condition number.
The next corollary recovers the O(E-1) convergence rate for strongly convex functions, which is the optimal dependence on the accuracy (Nguyen et al., 2019).
Corollary 2. In the same setting of Theorem fix e > 0. Let ra def f(xo) f*. Then E [f(xx) f*] < as long as
2ro
2ro
9C
K 2 Ky max max 2ng log
2B log
2ue
While the dependence e is optimal, the siluation is dilferent when we consider the dependence problem constants, and in particular the dependence Ks: Corollary 2 shows possibly multiplicative dependence KyKs. This is different for objectives where we assume convexity, and we show this next. It is known that the PL-condition implies the quadratic functional grouth (QFG) condition (Necoara et al., 2019), and is in fact equivalent for convex and smooth objectives (Karimi et al., 2016). We will adopt this assumption in conjunction with the convexity of f for our next result.
10

---

Published in Transactions Machine Learning Research (02/2023)
Assumption 6. We say that convex function f satisfies the quadratic functional growth condition if
f(z) - 1 -
M
(21)
for all x E Id where f is the minimum value of f and where (अx) def arg mix || - yll is the projection
on the set of minima X def {uER* Bd -(v)=
There are only a handful results under QFG (Drusvyatskiy & Lewis, 2018; Necoara et al., 2019; Grimmer, 2019) and only one applies to our setting ( (Grimmer, 2019). For QFG in conjunction with convexity and expected smoothness, we can prove convergence in function values similar to Theorem 3,
Thcorem 4. Assume that Assumptions 2 and 6 hold with f convex. Choose S min { according to Lemma Then
4 A(BitAa)
[[f(IK) - fr 18x, Kf exp
(- M (f(xo) ff),
aK +
2
where M def = 8 max {4xy, 4Br + Ks).
Theorem 4 allows stepsizes (1/LL), which are much larger than the (()((K stepsizes in (Nguyen et al., 2019; Grimmer, 2019). Hence, it improves upon the prior results of Nguyen et al. (2019) and Grimmer (2019) in the context of finite-sum problems where the individual functions fi are smooth but possibly nonconvex, and the average ʃ is strongly convex or satisfies Assumption 6.
The following straightforward corollary of Theorem shows that when convexity is assumed, we can get dependence on the sum of the condition numbers Ky + Kg rather than their product. This is a significant difference from the nonconvex setting, and is not known whether is artifact of our analysis inherent difference.
Corollary 3. In the same setting of Theorem 4, fix & >>>0 0. Then [[((K)) s*] S e as long as
[4K
K max
.M log
μE
e
where M def 32Bxy Skg and ro f(xo) ft.
6 Importance Sampling and Optimal Minibatch Size
As an example application of our results, we consider importance sampling: choosing the sampling distribution to maximize convergence speed. We consider independent sampling with replacement with minibatch size Plugging the bound on A, B, C from Proposition 3 into the sample complexity from Corollary yields:
126
max (1 (1-T
D
L
K
max Tnq
(2
where D max (126,, 4 ]. Optimizing (22) over (n:)a yields the sampling distribution
(23)
The same sampling distribution has appeared in the literature before (Zhao & Zhang, 2015; Needell et al., 2014), and our work is the first to give it justification for SGD on nonconvex objectives. Plugging the distribution (23) into (22) and considering the total number of stochastic gradient evaluations x 7 we get,
KJ 2 120gL
max { { --1,
DL E²
where def 1 Σm= L, This is minimized over the minibatch size T whenever 7 1+ [DL=- Similar expressions for importance sampling and minibatch size can be obtained for other sampling distriiuttioss as in (Gower al., 2019).
11

---

Published in Transactions Machine Learning Research (02/2023)
SGD with uniform sampling SGD with importance sampling
60-3
SG0 with uniform sempling
SGD with importanca sampling
60
4
[|P
4x
3x
30
xx
10
0
1000
2000
3000
4000
5000
0
1000
2000
3000
4000
5000
Iterations
Iterations
Figure 2: Results on regularized linear regression with (right) and without (left) normalization. Normalization
means forcing ||æ]|| =
Table Fitted constants in the regularized logistic regression problem. Predicted constants are per Proposi- tion 3. The residual is the mean square error, see Section G.1 in the supplementary for more discussion of this table.
2A
B
Residual
ES, Predicted 9
0.994
6.113
ES, Fit
10.09
0.373
0.413
a
B
Residual
RG, Fit
0.38
2.09
0.57
7 Experiments
7.1 Linear regression with nonconvex regularizer
We first consider linear regression problem with nonconvex regularization to test the importance sampling scheme given in Section 6,
x + 2
min ZER
f(x)
4(x) Σ j=1
(24)
11
where 4(x) = (a,,, yill 01 a1....,0, On E Rd are generated, and = 0.1. We use n = 1000 and d = 50 and initialize x We sample minibatches of size T 10 with replacement and use y L.1/VLAK, where K 5000 is the number of iterations and A is in Proposition 3. Similar to Needell & Ward (2017), illustrate the utility of importance sampling by sampling a; from zero-mean Gaussian of variance i without normalizing the d, Since L; x ||a,]|?, this case. However, when we normalize, the two methods should not be very different, and Figure 2 (of a single evaluation run) shows this.
expect importance sampling outperform uniform sampling in
7.2 Logistic regression with a nonconvex regularizer
We now consider the regularized logistic regression problem from (Tran-Dinh et al., 2019) with the aim of testing the fit of our Assumption 2 compared to other assumptions. The problem has the same form as (24) but with the logistic loss 4(z) = log(1 + exp(-aT 2)) for a1,..., Og E Rd given and = 0.5. We run experiments on the a9a dataset (n = 32561 and d = 123) from LIBSVM (Chang Lin, 2011). We fix t = 1, and run SGD for K 500 iterations with a stepsize 7 = 1/VLAK as in the previous experiment. We use uniform sampling with replacement and measure the average squared stochastic gradient norm 1/n En [[V[((x))) every five iterations, addition to the loss and the squared gradient norm. We then run nonnegative lincar
12

---

Published in Transactions Machine Learning Research (02/2023)
least squares regression to fit. the data for expected smoothness (ES) and compare to relaxed growth (RG). We also compare with theoretically estimated constants for (ES). The result is in Table 1, where tight fit between our theory and the observed. The experimental setup and estimation details are explained more thoroughly in Section G.1 in the supplementary material.
Acknowledgments
Ahmed Khaled acknowledges internship support from the Optimization and Machine Learning Lab led by Peter Richtárik at KAUST.
References
Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. QSGD: commuication-effiiien SGD via gradient quantization and encoding. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp- 1709 1720, 2017. URL https::/tpsetteeee httpp//eeing..eeeeeeee neurips. cc/paper/2017/hashl http://////aaaaaaaaa202292999600 (66.3300288888808774111616778660
Abstract. html. 2, 9
Amir. Beck. First-Order Methods in Optimization. Society for Industrial and Applied Mathematics, 2017. doi:
10.1187/11781611974997 9
Tal Ben-Nun and Torsten Hoefler. Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis. ACM Comput. Surv., 52(4), 2019. ISSN 0360-0300. doi: 10.1145/3320060.
Dimitri P. Bertsckas and John N. Tsitsiklis. Gradient Convergence in Gradient methods with Errors. SIAM doi: 10.1137/1105266339733111
Journal on Optimization, 10(3):627-66 642, 2000.
Léon. Bottou, Frank F. Curtis, and Jorge. Nocedal. Optimization Methods for Large-Scale Machine Learning.
SIAM Review, 60(2):223 311, 2018. doi: 10.1137716M1080177.. 3, 4
Chih-Chung Chang and Chih-Jen Lin. LibSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology (TIST), 2(3):27, 2011. 12
Aaron Defazio, Francis R. Bach, and Simon Lacoste-Julien. SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. In Zoubin Ghahramani, Max Welling, Corinna Cortes, Neil D. Lawrence, and Kilian Q. Weinberger (eds.), Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebee, Canada, pp. 1646-1654, 2014. URL https: / http://cceeeenn.neeeeeee. nwwrips..pp...... cctpppps.cppppp///24////// Abaaeeat html. 5
Ofer Dekel, Ran Bachhaa, Ohad Shamir, and Lin Xiao. Optimal Distributed Online Prediction Using Mini-Batches J. Mach. Learn. Res., 13(mull):16--20,, 2012. ISSN 1532-4435. 2
Yoel Drori and Ohad Shamir. The complexity of finding stationary points with stochastic gradient descent. In Proceedings of the 37th Internalional Conference on Machine Learning, ICML, 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, 2658 2667. PMLR, 2020. URL
http://proceedingss mlr, ppwss/v119/drost
html. 9
Dmitriy Drusvyatskiy and Adrian S. Lewis. Error Bounds, Quadratic Growth, and Linear Convergence of Proximal Methods. Mathematics of Operations Research, 43(3):919 (9999999,-448 2018. 11
Cong Fang, Zhouchen Lin, and Tong Zhang, Sharp analysis for nonconvex SGD escaping from saddle points.
In Alina Beygelzimer and Daniel Hsu (ods.), Conference on Learning Theory, COLT 2019, 25-28 June 2019, Phoenir, AZ, USA, volume 99 of Proceedings of Machine Learning Research, pp. 1192- 1234. PMLR, 2019. URL
mlr, wtt.s/rree///aaegeeee/s angg/1 html.
13

---

Published in Transactions on Machine Learning Research (02/2023)
Rong Ge, Furong Huang, Chi Jin, and Yang Yuan. Escaping from saddle points online stochastic gradient for tensor decomposition. In Peter Grünwald, Elad Hazan, and Satyen Kale (eds.), Proceedings of The 28th Conference Learning Theory, COLT 2015, Puris, France, July 3-6, 2015, volume 40 of JMLR Workshop JMLR.org, 2015. URL http://proceediin mlr wwtp:s/w....p../eeeeeeeeeeee press/v40/
and Conference Proceedings, pp. 1e11..tml. 7
Saced Ghadimi and Guanghui Lan. Stochastic First- and Zeroth-Order Methods for Nonconvex Stochastic Programming. SIAM Journal on Oplimization, 23(4):2341 2268, 2013. doi:
3, 9
Eduard A. Gorbunov, Filip Hanzely, and Peter Richtárik. A unified theory of SGD: variance reduction, sampling, quantization and coordinate descent. In Silvia Chiappa and Roberto Calandra (cds.). The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, 26-28 August 2020, Online [Palermo, Sicily, Itayy] volume 108 of Proceedings of Machine Learning Research, pp. 680 6900 PMLR, 2020. URL attp://proceeding: mlr, wtt.:mmlr.pppp... watrss/v100//oobbboooe -tml.
Robert M Gower, Peter Richtárik, and Francis Bach. Stochastic quasi-gradient methods: variance reduction via Jacobian sketching. Mathematical Programming, pp. 1 1-58, 2020. ISSN 0025-5610. doi: 10.1007/ $10107-020-01506-0. 3, 5,
Robert Mansel Gower and Peter Richtárik. Randomized iterative methods for linear systems. SIAM Journal on Malris Analysis and Applications, 36(4):1660 1690, 2015. 5
Robert Mansel Gower, Nicolas Loizou, Xun Qian, Alibek Sailanbayev, Egor Shulgin, and Peter Richtárik. SGD: General Analysis and Improved Rates. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 5200-5209. PMLR, 2019. 2, 3, 5, 6, 7, 8,
Benjamin Grimmer. Convergence Rates for Deterministic and Stochastic Subgradient Methods without Lipschitz Continuity. SIAM Journal on Optimization, 235001335- 2019. ISSN 1095-7189. doi:
10.1137/18ml 111113//mmm17366x 17306x. 6, 11
Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In Francis R. Bach and David M. Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pp. gupta15. html. 2
JMLR.org, 2015. URL http://proceedings. mlr www.....e.//.............. press/v37/
Samuel Horváth, Dmitry Kovalev, Konstantin Mishchenko, Sebastian Stich, and Peter Richtárik. Stochastic distributed learning with gradient quantization and variance reduction. arXiv preprint or Xiv: Na::1944.0511 Abs/1194.005115, 2019. URL
o0t/////994/ 1011111199..11.....1111 2,
Hamod Karimi, Julie Nutini, and Mark Schmidt. Linear convergence of gradient and proximal-gradian methods under the Folyak-Lojasiewir condition. In European Conference on Machine Learning and Knowledge Discovery in Dotabases Volume 9851, ECML PKDD 2016, pp. 795-811. Springer-Verlag, 2016. 10
Sarit Khirirat, Hamid Reza Feyzmahdavian, and Mikael Johansson. Distributed learning with compressed gradients. arXiv preprint arXiv:1806.063 06573. 2
2018. URL http:////axvvvvo/raaaee
Dmitry Kovalev, Peter Richtárik, Eduard A. Gorbunov, and Elnur Gasanov. Stochastic spectral and conjugate descent methods. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NewrIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 3362 3371, 2018. URL https: / httpss////cccoee....moo www..ee....ppppeeeeeeeeeee 22221555888111553444722286688880 Abstract.. html. 5
14

---

Published in Transactions on Machine Learning Research (02/2023)
Yunwei Lei, Ting Hu, Guiying Tı, and Ke Tang. Stochastic Gradient Descent. for Nonconvex Learning Without Bounded Gradient Assumptions. IEEE Tronsactions Newral Networks and Learning Systems, pp. 1 7, 2019. ISSN 2162-2388. doi: 10.1109/TNNISS (WT.NNNSS.109...... 2019 2952219.. 4, 7, 10
Xiaoyu Li and Francesco Orabona. On the convergence of stochastic gradient descent with adaptive stepsizes. In Kamalika Chaudhuri and Masashi Sugiyama (eds.), The 22nd International Conference on Arlificial Intelligence and Statistics, AISTATS 2019, 16- 18 April 2019, Naha, Okinawa, Japan, volume 89 of Proceedings of Machine Learning Research, pp. 983 992. PMLR, 2019. URL http://proceeiing..tt press/v89/ http//999/111....// html. 7
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agũera y Arcas. Communicationn efficient learning of deep networks from decentralized data. In Aarti Singh and Xiaojin (Jerry) Zhu (eds.), Proceedings of the 20th International Conference Artificial Intelligence and Statistics, AISTATS 2017, 20- 22-22 April 2017, Fort Lauderdale, FL, USA, volume 54 of Proceedings of Machine Learning Research, pp. 1273 1282. PMLR, 2017. URL htttp://proceedingg mlr. presss/554/mcmmaaa//// http///mmmaaaaaaaaaaaaaaaaaaaaaa html. 4
Ion Necoara, Yurii Nesterov, and François Glineur. Linear Convergence of First Order Methods for Non- Strongly Convex Optimization. Mathematical Programming, (75(1-2):69- 107, 2019. ISSN 0025-5610. 10, 11, 29
Deanna Needell and Rachel Ward. Batched Stochastic Gradient Descent with Weighted Sampling. In Gregory E. Fasshauer and Larry L. Schumaker (eds.), Approzimation Theory XV: San Antonio 2016, pp. 279-306. Springer International Publishing, 2017. 12
Deanna Needell, Rachel Ward, and Nathan Srebro. Stochastic gradient descent, weighted sampling, and the randomized kaczmarz algorithm. In Zoubin Ghahramani, Max Welling, Corinna Cortes, Neil D. Lawrence, and Kilian Q. Weinberger (eds.), Advances in Newral Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pp. 1017-1025, 2014. URL https: https://proceed: http:///ooccee......ee/
wwweeeeiiinnnnnuuips.... www.ipe.c./ppaaeeeepee T20C118811178990000001724134 htw.://111111112144277770--aaaaa Awstract. 2, 11
Arkadi Nemirovsky and David B. Yudin. Problem Complexity and Method Efficiency in Optimization. Wiley, New York, 1983. ISBN 9780471103455. 1
Phnong Ha. Nguyen, Lam M. Nguyen, and Marten van Dijk. Tight dimension independent lower bound on the expected convergence rate for diminishing step sizes in SGD. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NewrIPS 2019, December 8-14, 2019, Vancowver, BC, Canada, pp. 3660 3669, 2019. URL https:// proceedings.neur/e wwtpcccpppppceeeeeeeeeee http:////seeaeeettteeoooototttan htt.apaaaaaaaeaaaoooaooaoottttta Abstract. 6, 10, 11
Boris Polyak and Y.Z. Tsypkin. Pseudogradient adaptation and training algorithms. Aulomation and Remole Control, 34:377 397, 01 1973. 3, 6, 18, 32
Alexander Rakhlin, Ohad Shamir, and Karthik Sridharan. Making gradient descent optimal for strongly convex stochastic optimization. In Proceedings of the 29th International Conference on Machine Learning, ICML 2012, Edinburgh, Scolland, UK, June 26- July 1, 2012. ieml.cc / Omnipress, 2012. URL http: //tccll.cc/012/////// ptt/0////pppppr/22..11 pdf.
Peter Richtárik and Martin Takáč. Stochastic Reformulations Linear Systems: Algorithms and Convergence Theory. arXiv preprint arXim: 01 108, abs/1706.0110s, 2017. URL https://arxtv.orrgatte 01108. 3, 5,
Ruslan Salakhutdinov. Deep learning. In Sofus A. Macskassy, Claudia Perlich, Jure Leskovec, Wei Wang, and Rayid Ghani (eds.), The 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '14, New York, NY, USA - August 24 - 27, 2014, pp. 1973. ACM, 2014. doi: 0t0/////2333.22330000 URL https://aol. http:///o..rrr/00..155222233.... /1t...111////.22250000000
15

---

Published in Transactions on Machine Learning Research (02/2023)
Karthik Abinav Sankararaman, Soham De, Zheng Xu, W. Ronny Huang, and Tom Goldstein. The impact of neural network overparameterization gradient confusion and stochastic gradient descent. In Proceedings of the 37th International Conference Machine Learning, ICML 2020, 13-18 July 2020, Värtual Event, URL http:
volume 119 of Proceedings of Machine Learning Research, pp. 8469 8479. PMLR, 2020.
//procesdings.nl mll.. press/vi19 watal//nnnaaaaaaaaaa22.. htmm.. 3,
Mark Schmidt and Nicolas Le Roux. Fast Convergence of Stochastic Gradient Descent under a Strong Growth Condition. arXiv preprint or Xiv::1308 6370, 2013. 3, 4
Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: from theory to algorithms. Cambridge University Press, 2014. ISBN 1107057132.
Ohad Shamir and Tong Zhang. Stochastic gradient descent for non-smooth optimization: Convergence results and optimal averaging schemes. In Proceedings of the 30th International Conference on Machine Learning, ICML. 2013, Atlanta, GA, USA, 16-21 June 2013, volume 28 of JMLR Workshop and Conference Proceedings, pp. 71 79. JMLR.org, 2013. URL http://prroceddings.... mlr wwwps/wesse/// shamir13. hhtml. 9
M.V. Solodov. Incremental Gradient Algorithms with Stepsizes Bounded Away from Zero Computational Optimization and Applications, 11(1):23-35, 1998. doi: 10.1023/2:101186600003
Sebastian U. Stich. Unified Optimal Analysis of the (Stochastic) Gradient Method. arXiv preprint 11199994724.2.. 2019. URL butps://arkee org/abs abs/1997.04222. 9, 10, 25, 27
arXix:1977.00 0423,,
Scbastian U. Stich and Sai Prancoth Karimireddy. The Frror-Foodback Framework: Better Rates for SGD with Delayed Gradients and Compressed Communication. arXiv preprinl arXiv:1909.068 05300, abss/190905533 2019. URL attps://arXivv org/abs
00350.0 7
Ruo- Yu Sun. Optimization for Deep Learning: An Overview. Journal of the Operalions Research Sociely of China, 8(2):249-29)) Jun 2020. ISSN 2194-6698. doi: 00.1007/440005-000- 000.000000000.6...0
Quoe Dinh, Nhan H. Pham, Dzung T. Phan, and Lam M. Nguyen. Hybrid Stochastic Gradient Descent Algorithms for Stochastic Nonconvex Optimization. arxiv preprint Xrv::190050000 059200 has/1995.05920, 2019.
URL httts:://arkaa org/ httpa/////..05.550 12
Paul Tseng, An Incremental Cradien Method with Momentum Term and Adaptive Stepsize Rule. SIAM Journal on Optimization, 8(2):506-531) 1998. doi: 10.1137//11032221552477777 4
Sharan Vaswani, Francis R. Bach, and Mark Schmidt. Fast and faster convergence of SGD for over- parameterized models and an accelerated perceptron. In Kamalika Chaudhuri and Masashi Sugiyama (eds.), The 22nd International Conference on Artificial Intelligence and Statistics, AISTATS 2019, 16-18 April 2019, Naha, Okinawa, Japan, volume 89 of Proceedings of Machine Learning Research, pp. 1195-1204. PMLR, 2019. URL http::/pproceediiieee mlr. w.l..press//ie /89/ss//////ee h/tp//vwwwwnnnee html. 3, 4
Stephen A. Vavasis. Complexity Issues in Global Optimization: A Survey. In Horst R. and Pardalos P. M. (eds.), Handbook of Global Optimization. Nonconvez Optimization and Its Applications, volume 2. Springer, 1995. doi: https://doi.org /t/////i.or101000011110// 10. 1-117777----11--2555
Jianqiao Wangni, Jialei Wang, Ji Liu, and Tong Zhang. Gradient sparsification for communication--efiiieee distributed optimization. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Resa-Bianchi,, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 1306 1316, 2018. URL https:// /ttp://///oceggss/////ee wwt.gg...urip... /01/// hash 2tta://28B2444555599999222222222 Abstraac.. html. 2
Peilin Zhao and Tong Zhang. Stochastic optimization with importance sampling for regularized loss minimiza- tion. In Francis R. Bach and David M. Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pp. 1-9. oMg, 2015. URL http::/ http://prroceeding. mlr,
www....ee......t......... press/v37 / 2haoa15 html. 11
16

---

Published in Transactions Machine Learning Research (02/2023)
Contents
Introduction
1.1 Modelling stochasticily
1.2 Sources of stochasticity
Contributions
Existing Models Stochastic Gradient
ES in the Nonconvex World
4.1
Brief history of expected smoothness: from convex quadratic to convex optimization
4.2
Expected smoothness for nonconvex optimization
4.3
Expected smoothness the weakest assumption
4.4
Perturbation
4.5
Subsampling
4.6
Compression
5
SGD in the Nonconvex World
5.1
General convergence theory
5.2
Convergence under the Polyak-Lojasiewicz condition
10
Importance Sampling and Optimal Minibatch Size
7 Experiments
12
7.1 Linear regression with a nonconvex regularizer
12
7.2 Logistic regression with a nonconvex regularizer
12
A Basic Facts and Notation
18
B Relations Between Assumptions
19
B.1 Proof of Proposition 1
19
B.2 Formal Statement and Proof of Theorem
20
C Proofs for Section 4.5 and 4.6
21
C.1 Proof of Lemma
C.2 Proof of Proposition
C.3 Proof of Proposition
22
C.4 Proof of Proposition 4
23
D Proofs for General Smooth Objectives
17

---

Published in Transactions Machine Learning Research (02/2023)
D.1 Proof of Lemma
24
D.2 A descent lemma
25
D.3 Proof of Theorem 2.
26
D.4 Proof of Corollary
26
E Proofs Under Assumption 5
27
E.1 Proof of Lemma
27
E.2 Proof of Theorem 3
28
F Proofs Under Assumption 6
29
F.1 A Lemma for Full Gradient Descent
29
F.2 Proof of Theorem 4.
30
G Experimental Details
31
G.1 Logistic regression with a nonconvex regularizer
31
H More on (Polyak & Tsypkin, 1973)
32
A Basic Facts and Notation
We will use the following facts from probability theory: If X is random variable and is a constant vector, then
E ||
x|] - E - [I
(25)
E
| - E [X]]
E [|x11]]
|I [X]]
(26)
A consequence of (26) is the following,
E [XX]I
< E [|X|1]
(27)
We will also make use of the following facts from linear algebra: for any a Red and any <>
2 (a, b) ||a|| + ||b|] - ||a b1|
(28)
|||| < (1 - <) l|a b1l + (1+ |
(29)
For vectors X1. X2...., Xn all in Rd, the convexity of the squared norm |1-112 and trivial application of Jensen's inequality yields the following inequality:
1
x2
n
X
|X
(30)
For an L-smooth function f we have that for all x, E Rd
f(x) < fly) + (Vf(y), v) + L - || v11 (V)(=) vf(1i) - L - y||
(31)
(32)
Given point æ nd and stepsize 7 > 0, we define the one step gradient descent mapping as,
TT(x) def
= x
(33)
18

---

Published in Transactions Machine Learning Research (02/2023)
B Relations Between Assumptions
B.1 Proof of Proposition
Proof. We define the function f: RR + IR as,
2²
if <
(x)
otherwise.
Then f is 1-smooth and lower bounded by 0. We consider using SGD with the stochastic gradient vector
Vf(z) + [ Vf(z) - V|2 with probability 1/2.
with probability 1/2,
(
Then suppose that (RG) holds, then there exists constants a and B such that,
E [|( s al[Vf(>)] + B.
Consider x max (1,2 (a + -)), then |a| > and hence Vf(x) by the definition of f. Specializing (RG) we get,
E ||({)))1 SB
(34)
On the other hand,
1/2 ((1 vz)² 2 (1 (1-v5))) E [1( 2
2
(1 + vz)
Mv
a = max {1, (1 8)}
(35)
We see that (34) and (35) are in clear contradiction. Tt follows that (RG) does not hold. We now show that (ES) holds: first, suppose that |x] 2 1, then
E [|(x))]]] = 2 ((11 [[l+) + (1-V1z)) )
(2+2|z) 1 |
3
2 (f(x) -'nt),
(36)
where in the last line we used that when |xc| have f(x) - fintt |o|| 1/2. Now suppose that | S then
E [[(
2 + v [r|]
+ +(x- v [x]] 2)
2² | < 1
(37)
Combining (36) and (37) then we have that for all E R,
[1a())][]] < f(x) - fiuf + 2.
follows that (ES) is satisfied with A 1/2, B 0, and 2.
19

---

Published in Transactions Machine Learning Research (02/2023)
B.2 Formal Statement and Proof of Theorem 1
Theorem 1 (Formal) Suppose that f R IR is I-smooth. Then the following relations hold:
1. The maximal strong growth condition (M-SG) implies the strong growth condition (F-SG).
2. The strong growth condition (E-SG) implies the relaxed growth condition (RG).
3. Bounded stochastic gradient variance (BV) implics the relaxed growth condition (RG).
4. The gradient confusion bound (GC) for finite-sum problems f f - Ss/n implies the relaxed growth condition (RG).
5. The relaxed growth condition implics the expected smoothness condition (ES).
6. If f has the expectation structure (7), then the sure-smoothness assumption implies the expected smoothness condition (ES).
Proof.
1. Suppose that (M-SG) holds. Then,
[11o())]]] S [olIV[(>)]*] = = [
E
Hence (E-SG) holds.
2. If (E-SG) holds, then plugging / = 0 shows that (RG) holds.
3. If (BV) holds, then plugging shows that (RG) holds.
4. We use the definition of [[Vf(=)]
f(
(z)
|V
n
n
n
(z) Vf,(,))
[+((2))
Σ Σ (Vf(()). V((aa)
n
1= =ij=lişi
E, [[V
11(12 n)
E, [|T[==))]]] - (1 1 [1
Rearranging we get,
E [+ Sn [V + (n (1) 1).
Hence (RG) holds with a = n and 8 (n - 1).
5. Putting A = 0, B = a and C = 3 shows that (FS) is satisfied.
6. Note that fg(x)) is almost surely bounded from below by O, and that by (SS), fe(x)) is almost-surely T-smooth. Then using Lemma 1 we have almost surely,
[[ s 22)fe(x).
Let pinf be the infimum of f (necessarily exists, since f(x)) F. [fe(x)) 2 0). Then,
||Vf{())] s 2L -fe(z)
- plnl) fL
20

---

Published in Transactions Machine Learning Research (02/2023)
Taking expectations and using that f(x) EEWD [fe(x)] that (FS) is satisfied with A 27 and C' = f
C Proofs for Section 4.5 and 4.6
C.1 Proof of Lemma 1
Proof. If we let 2+ VV(x), then using the L-smoothness of f we get
L 1|,0 1144-117 x/|
(()) + (V x)
Using that fint & f(x+) and the definition of x+ we have,
ii S f(x+) - |V 2L
1
(V(())1==-(( = 2T [V/())]
Rearranging we get the claim.
C.2 Proof of Proposition 2
Proof. We start out with the definition of f.(o.)) then use the convexity of the squared norm I1-112 and the linearity of expectation:
E [V
E
V (x)
(30)
22
fx)
M E [u
Vf((2)
We now use Lemma and the nonnegativity of fa(z) finf (since fint is lower bound on fi),
E [[AA]]]]
2
E [o]] (f((z) - s!!nt)
2 max; [,E ( n
Σ (f((z) f!!
i=1
2 max; (L,E [e]])
(f((z) ffint pint )
n
2 max (L,B [of] (f(x) fint) + 2 max (LiB [u?
plat - flot).
remains to notice that since f(x) = 1 ER f((x) 2 E finf then 1 Em fiul is a lower bound on f. and hence
A inf
1
piin pinl) = pint
rinf 0,
n
n
as piul is the infimum (greatest lower bound) of (f(x) x E IRd)) by definition.
21

---

Published in Transactions Machine Learning Research (02/2023)
C.3 Proof of Proposition 3
Proof. From the definition of V. and the linearity of expectation,
| =E
E [V =
Σ ((),, 1
b (f
(38)
We now consider each of the specific samplings individually: Independent sampling with replacement: we write
S
1/
Sykk
(39)
Tqi
TqI
k=1
where St, is defined by
{1
if the k-th dice roll resulted in i, otherwise,
0
and clearly E [S = qr- Then,
E [yyyy]
S,,.
(40)
g2
k=11=1
It straightforward to see by the independence of the dice rolls that
9:40
k I if = t, J. if k = t j.
E S5,
9:
0
Using this in (40) yields
+ J. otherwise.
[1 =
Plugging the last equality into (38), we gel.
E [[VVL(()))]]]
1 1- (
11)
|()((a))
1
1 ()))),, V(((0))
n
f(()2
(41)
n²
T/
(((()) Vf((z)
1
[+)12
+
n
n
74.
j=1
1 (1- [+(())]
1 |
+
n²
T4i
Using Lemma on each fú;
E [IVJ.(
2 n 22 (1- [[(=a)]
Σ
La
(f((z) f!a0)
ntq:
i=1
1 (1-
L (f((z) f!nn))
$
[f((z)]1 +2
max
nTq,
2=1
22

---

Published in Transactions Machine Learning Research (02/2023)
It remains to that 1 Ei= (fi()) l! f(x) -faat A proved in Proposition 2.
Ain and the nonnegativity of A was
iii Independent sampling without replacement: not difficult to see that,
i
[0:0y]
Pi i-
Using this in (38),
" Σ
n 1
E
(Vi(()) Vfz(a))
Σ Pi
[f((')]
n²a
j liift
1
Σ (1 -1 |
1
Vf((z). Vz)(2)) +
n
n ==1
Continuing similarly to the previous sampling (from (41) onward), we get the required claim.
(ii) r-nice sampling without replacement: it is not difficult to see by elementary combinatorics that
if
Prob (i E S and j E)
2(7-1)
otherwise.
a(n-1)
From this we can easily compute E [ez,y]. Substituting into (38) and proceeding as the previous two cases yields the proposition's claim.
C.4 Proof of Proposition 4
Proof. First, it is casy to see that g(xx)) is an unbiased estimator using the tower property of expectation and Assumption
[g(xx)]
1
E [[Qun(a(Ea))
Σ E ((,,, (xx)))
n
n
1
- -V (((x))
E (xx()
n
n we have for expectation conditional on satisfied,
Second, to show that Assumption (=())))(())) ..... In (xx):
E [[((=)))1]
= E
1=1
(26)
(p)() s((xa)) -
+
Since Q1,k, .2. On are independent by definition, the variance decomposes:
E
[1((=))1
E
[( (xx)))
+
Σ
(x
n²
n
(x)))
willg
+
g (xx))
(42)
2
23

---

Published in Transactions Machine Learning Research (02/2023)
We now take expectation with respect to the randomness in the g((x)) We consider the first term in (42), where once again the variance decomposes by the independence of 91 (Ix)) 92 (2(x)) (T))
n2
E
9 (Ix) Σ
(26)
E
ge(x))) V(((EA)
+ +[V[((*)]*
n
E [119((=a)) - VV- (xx()))) [++))
(27)
E
[112(1 + +
(43)
n
Combining (42) and (43),
E [1((=))1]]
< n
Σ (1 + wi) E [[[x(=)]]]] + +
(44)
For the first term (44), we have using Assumption 2 and then Lemma 1,
n 1a+ wi) E [[(([
(1 wi) (2A: (Ji(zN) - -:a + B[IV// +C),
n
i=1
n
Σ
(1 + wi) (2 + BBI,) (f((*)) simt) - Ca)
n
1 n2 Σ (1+
(2A (fi( -
n
= 2A (f(xx) finf) + +C.
(45)
where A def max; (1 + w.) (A,+ + B,I,) /n, and C del 24A inf + #/2 Em= (1 ww) Ci and where A defined
as in Proposition 2. Combining (45) with (44) we finally get,
E [[[(=))P]] < 2A (((m)) fint) + [[ +C,
which shows that Assumption satisfied.
D Proofs for General Smooth Objectives
D.1 Proof of Lemma
Proof. We start with the L-smoothness of J, which implies
L 2 || - 2x
f(xx+1) S f(xx) + ((xx)) -xx)
In?
=f(xx) - (F(())) +
2 1lo(
(46)
Taking expectations in (46) conditional on æk, and using Assumption 2, we get
T,²
E [[o(=a]]l]]
[[f(x++1) | mx] = f(xx) >11 (xx)))
(ES) v f(xx) (xx)
Ln²
2A (f(x)) - finnt) B[V 1 +C)
+
2
f(xx) 7 (1- LBBy)
|Vff(=))| + + (f(xx) + 19²C
2
24

---

Published in Transactions Machine Learning Research (02/2023)
Subtracting fint from both sides gives
2 E [f(+++1) | ma] finf 2 (1+ (J(IK) fim)) -2 (1-221 [[((x)]] + L5²C
2
taking expectation again, using the tower property and rearranging, we get
IE [](+++1)= - p + (1 LB>
- lسm + LF2C
[|
s (1-1)A))
2
Letting OF def E [f(xx) - finl] and rk def E
[f(x))]] we can rewrite the last incquality as
LBy
Ln²C
[
S (1 + A) 6% 8k+1 +
2
Our choice of stepsize guarantees that 1
LB* 2
2 1. As such,
L22C
7 2 5 (1+ (1) 6% 6k+1 +
(47)
2
We now follow Stich (2019) and define a weighting sequence wo, W1, W2..... WK that used to weight the
terms in (47). However, unlike Stich (2019), we are interested in the weighting sequence solely as a proof technique, and it does not show up in the final bounds. Fix w-1 0. Define wi I+4." for all k 2 0.
Multiplying (47) by wk/7,
Wk (1 + ITPA)
wk 8k+1 +
LCW
2 WkTk I
6N
2
Wk-1 6p
Wk dx+1
LCwxy
7 7
Summing up both sides as h = o,..... K
we have,
1 K
WkTk
<
W 6n
WK-
6x +
LC
K-1
Wky
2
2
k=0
k=0
Rearranging we get the lemma's statement.
D.2 descent lemma
Lemma 4. Under Assumption we have for any 7 > 0,
- ((T,(a) S ? (3I, 2) 2
(48)
where (x) def - VZxx)..
Proof. Using the L-smoothness of
f(z)-f(I,(0)
(T),
+ - TT (x)1
L
2
(33)
7 (VI(T, v)(2)) + In²
|v
2
(28)
7 (IVS(T,
(V/ -1V)(, v)(O))?
Tn² [VF(()))1
2
(29)
7 (SIISS(,,
v/<>) (2+ - + LA2 2 [V}('))
S
2
(32)
(I²çIIT,(x) (2+ + T,2 |V]())]
7
2
(33)
(1758+45-1 + +2+ (1) [
2
25

---

Published in Transactions Machine Learning Research (02/2023)
1. Put ( by
< (3I,+ ( (V/
D.3 Proof of Theorem 2
Proof. We start with Lemma 2,
K
K-1 1
WK
LC
K-1
WHTH S
Work +
do +
wk7
2
2
7
7
2
k=0
R-1 wk. Dividing both sides by WK we have,
Let WK
K
W-1 60
LO
min
Th S
wiTk S
+
(49)
2 oSAck-
WK
Wr W 7
2
k=0
Note that,
K-
K
Kw-1
WK = Σ
Wk 2
min 0Si
Wi = KwK-1
(50)
(1+ L2
k=0
k=0
Using this in (49),
(1 (1+I 1) LCY
min
Tk S
ôu +
OSASK-
gk
2
Dividing both sides by 1/2 yields the theorem's claim.
D.4 Proof of Corollary 1
Proof. From Theorem 2 under the condition that 7 7 1/(LL,
min E [[V](aw))]] s LCN+ 2(1 + A) K oskgk-1
((at)) TK
(51)
Using the fact that 1 + x S exp(x)) we have that
LI+ K S exp(L>>AAN AK) < exp(11 3,
where the second inequality holds because < 1/V LAK by assumption. Substituting in (51) we get,
min 0
E
[[V
< LCY+ + T (f(zo) - pluf).
(52)
To make the right hand side of (52) smaller than e², we require that the first term satisfies
2² S LC
22
S
2
2LC
Similarly for the second term, we get that the number of iterations must satisfy: Note that we have three requirements the stepsize so far:
66g kK
E²
126 722
K
(53)
8²
y S BL
7 S
2LC
VLAKK
26

---

Published in Transactions on Machine Learning Research (02/2023)
Plugging each of the previous bounds on the stepsize into (53),
BL
LC
K Z 1260 VLAK
K K2 g²
K
(54)
e
Since K appears in both sides of the last. requirement, by cancellation and squaring we simplify (54) to
BL
245,LC K 2
K2
K2 e
g²
e
Finally, we collect the terms into single bound:
128gL 22
2C
126
K
max
B,
62
8²
Proofs Under Assumption 5
E.1 Proof of Lemma 3
This proof loosely follows Lemma 3 in (Stich, 2019) without using the (sx), sequence as it is not relevant to our bounds and without using exponential averaging. K S b a: then starting from (20) we have for = 1
Proof.
TK- S (1- 722c.
Recursing the above inequality we get,
K-1
S (1- (1- K
To + 22
(1 a7)*
TK
(1 )7) K ru + 7
+ 7
exp
a-ayk) ro
a
aK
exp
ro ab
h
aK
s
exp
To +
(55)
b
a²K
where in the last line we used that K
< b. Tf
K > b
then first we have
aK
Tko
exp
ro +
(56)
2b
ab"
Then for t>> ko:
8 8 hi
4c
To S (1- a r-- + t
8 8 1 - hio
r
a²
(s +
ko) 2
Multiplying both sides by (s + t - -ko)²,
4c
(s+t ko)²2 re s
(s + - ku 2) (s+ + -ku)-
r a²
(ss + t - ho - -1)2 -1) 1 p:-1 +
4c
4c
S
(s / - kig - 11²² ri-1
+
27

---

Published in Transactions Machine Learning Research (02/2023)
Let. wy (s + -ko)² Then,
4c
WiTe & w
a²
Summing up for 1 t= ko + 1, + 2..... K and telescoping we get,
4c (K
-k)
8 + 4e (K ku)
WKTK S "ko +
a²
a²
Dividing both sides by w and using that since Ko then wK = (s + K - -ko)2 2 (K- - -ko)?
r
4c(K - ko)
8 - ko)
4c
TK
I
+
+ a²((K - ko)
(57)
WK
a²
By the definition of ko we have K - ko 2 K/2. Plugging this estimate into (57)
Ts
8c
Tk
+
(58)
K²
a²K
Using (56) and the fact that K²
62
4x2
we have,
a²
4
4s²
aK
K²
K2
exp
2b
ro
+
ab
(59)
4g2
ak'
4cs
exp
ro
+
(60)
K
2b
abK?
aK
4cs
exp
(61)
2b
abK?
For the second term in (61), note that since K b/a
4s²
482 2
45 a
(62)
bK
b
K
b
b
We substitute this into (61),
2
exp
aa'
ro
+
c
(63)
K2
2b
K²K
Substituting with (63) in (58),
ak
9c
TK S
exp
ro
(64)
2b
"²K
It remains to take the maximum of the two bounds (64) and (55).
E.2 Proof of Theorem 3
Proof. Using the L-smoothness of f.
f(xx+1) - fr S f(zx) - ft + (Vf(z)) 2z41 - Ix) + L
xx+1 - xxll
2
L
= f(zx) ff 7 - 7h (Vf(x=).9(())) +
[1((2x))1
2
Taking expectation conditional on I& and using Assumption 2,
E [f((+11) f1] f(xx) fr "x/lV((<*)) T E [[lo(ra)1]]
f(xx) - - (1- L.Bk (ES)
TnRC
2
2
28

---

Published in Transactions Machine Learning Research (02/2023)
Using 1 LBTe 3/ and Assumption 5, 2
37kll + InRA) (f(x)) - 5*) + L
[[ s 2 1
2
Our choice of stepsize implies LnkA S 21 hence
[[ f'll S (1-7 (f(#x) - f)) + L2R
2
Taking unconditional expectations and letting T E [f(xx) - - we have,
INRC
Th+1 (1-mp) Th 2
Applying Lemma 3 with n1 b- = max (2x A, 2LB), and = LC/2 yields
μK [[f(<k) - 14] S cxp max
) (f(2o) - -²) + 9LC
24²
Using the definition of Ky yields the theorem's statement.
F Proofs Under Assumption 6
F.1 A Lemma for Full Gradient Descent
Lemma 5. Under Assumption and for T, = (--xx-VV(() for y S t we have,
[T,(2) (<))|² < |r -1- L- (
(f(T f*).
(65)
Proof. This is the second to last step in the proof of Theorem 12 in (Necoara et al., 2019), and we reproduce it for completeness. Let E Rd. Then using the definition of (x),
[T, y1?
| (x) - x x yl
(66)
||
x||
||x v/1 [1++
z
(33)
|| - 27 -2) - x||
|> v1 24 (VVT()))T,,)) + 2 [T, x01² + ( L
[T,(()) - =nl?)
27 2
= || v/2 + (L - 1) (z) x1| -27 (Vf(z), TT(() - v) + 2
L
[T,(x) x1(² (67)
We isolate the third term in (67),
(VS(+), - v) + + x11=
x||²
- - a) 2 (17 xx|² (68)
Since ʃ is L-smooth, then
L + (T) - x|1
((T < (V)(s),
Using this in (68) and using convexity,
I. [T, (x) - x112 (V) v)
2 x) f(x)
2
z f(z)-(()) f - ((T,)))))) f(y). (69)
Using (69) in (67).
[T, (x) - 2|1²2 < | - -1 - (1 - I 1T) - x||² - 27 (z( - ((y))
remains to use that T,(x) - I =>>[((>) and to put y = x(x).
29

---

Published in Transactions Machine Learning Research (02/2023)
F.2
Proof of Theorem 4
Proof. For expectation conditional on Ik, we have
E ||
E || T(xx) - (xx())
I (f(x) xxx))l + 2 E -
(26) I [zxx)) (+x))1 22 (E [[l[[=o\10]- -1V
(60)
|| - (x=)]l - -22-LT) V f(x))) - 27x (F))) - 1*)
(70)
+ + E [11
We can decompose the function decrease term and then use Lemma 4 as follows,
-% (J(T (xx)) )
2tk (f(xk)) (*) 2T (f( (xx)))
(48) 2 (f(x)) - f*) th (3L + 2) (>x))]
(71)
Let rk =1a (xx))) Contimuing from (70),
S rk - (2 L(x) 12 [V[()) 2Tv (J(Tr, - - + o E
[11o(ra)1*]]
Th- 2nx (f(xx) - f*) + 4L [+)]( + 2h E [19(2x)11]]
(72)
Using Assumption 2 and Lemma applied on f,
E [1lo(a)]1]]
s 2A (f(#x) 1*) B [V
< (2A+ 2BL) - (M)
- C 22 (f(x)) 1*) + +C,
(73)
where p def A BL. We now use Lemma 1 to bound the squared gradicnt term:
E | [x
(72)
- 2% (f(zx) - f*) + 4L [[v](2))]] + + - E [[s(on)]]]
(14)
rk 27 (1 -1.277 (f(*x) f*) 9 E [1lo[x001]]
(73)
Tk - 27 (1 4 - 7up) (f(xx) - f*) + R.C.
(74)
Now note that our choice of 7 guarantees that - 4 YPP 2 2, using this and Assumption 6 in (74) we get,
E - - (xx))]] s 7/ (f(z)) f*)) + +220 S (1
7kff
2 rk + H2C.
We now use the property of projections that E ||oTat] オ (xx- 1)) |11 Specializing this with y =(xx) get,
E
|| y112
for any y X*.
E [ (((+))))]] 5 <(1- (1-2244)
Ih/l)
|| (2xx)12 +120.
2
Taking unconditional expectation we get,
E [r+t1 < (1- TKl
E [[rx]] + 72C.
2
Using Lemma 3 with a H, b= max {4L, 4BL + 4A), and O we recover
-NKK
36C
E |r < cxp
ra + K
(75)
2max (4L, 4BL + 4A}
30

---

Published in Transactions Machine Learning Research (02/2023)
60
WID werLere
sill Kampie
20-
640
1311
5 13
145
094
63
178
120
420
120
391
300 300
176
272
Figure 3: Results from SGD applied to the regularized logistic regression problem.
Equation (75) gives a convergence guarantee in terms of the distance of the iterates from the set of optina. We may convert this to convergence in function values using the following consequences of the quadratic functional growth property and smoothness: ro S _ (f(zo) - f*) and f(Ix) - f S _ Hence,
max {4L, 4BL + 4A}) (f(zo) ff 18K
-K
E [f(K)
2 2 exp
uK
G Experimental Details
G.1 Logistic regression with nonconvex regularizer
Experimental Sctup: The optimization problem considered is
min Σ log (1 + exp((affx)) + 1 x3
j=1
where 01, .2...... an E RR are from the a9a dataset with 32561 and d = 123. We set X = 1/2. We run SGD for K = 500 iterations with minibatch size 1 with uniform sampling.
Data Collected: We collect a snapshot once every five itcrates (so Zp, Ts, 215, 7500) consisting of the squared full gradient norm [+x()l the loss on the entire dataset fxx), and the average stochastic gradient norm E [|a())]]]] = 1/n E (z+))) These are plotted in Figure 3. The full gradient norms go to zero, as expected, but the stochastic gradient norms decrease and quickly however around 0.5, and the full loss shows similar pattern. We note that Lemma 2 can explain the plateau pattern, using it one show that provided the stepsize is set correctly, the loss at the last point does not exceed the initial loss in expectation.
Hyperparameter Estimation: We estimate the individual Lipschitz constants as I ~ 10.1" + 2X. We estimate the global Lipschitz constant L by their average L _ Li. We estimate A for independent sampling as in Proposition 3 by = max, Lı/(Tnq.) = max, Le as т = and 4 = 1/n for all i. We estimate finl as the minimum value encountered over the entire SGD run and estimate f!nff by running gradient descent for 200 iterations for each and reporting the minimum value encountered. The stepsize we use is then y = 1/VLKA
Data Fitting: For Assumption 2, the parameters A, B, C 0 model the average stochastic gradient norms in terms of the loss and the full gradient. To find the best such parameters empirically, we solve the following optimization problem, minimizing the squared residual error
{o(=) = I - -bl =ER0
min
(76)
where 2 = [ 22 2a] are the constants to be fit corresponding to 2A, B, and C' respectively. W is an m x matrix (for the number of snapshots, equal to 100 here) such that W., f(x,) -ind W. [xV 12 and Wr. 1. Morcover, b E T the vector of average stochastic gradient norms, i.c. by = 1/n Σi V (xx)
31

---

Published in Transactions Machine Learning Research (02/2023)
Problem (76) is nonnegative linear least squares problem and be solved efficiently by linear algebra. toolboxes. To model relaxed growth, solve similar problem but with only the parameters B and C (and with corresponding changes to the matrix W).
Discussion of Table 1: Table shows that (FS) can achieve a smaller residual error compared to (RG) by relying on the function values to model the stochastic gradients. The value of the first coefficient is quite close to the theoretically estimated value of Lmax- The value of the coefficient C, however, difficult to estimate. This is because according to our theory it depends on the global greatest lower bounds of the losses fo and f. In the case of convex objectives, however, this constant can be calculated well given knowledge of the minimizers of all the ft and of f.
H More (Polyak & Tsypkin, 1973)
Polyak & Tsypkin (1973) study minimizing function that is lower-bounded and whose gradient is L- Lipschitz. They study iterations of the form 2n 2n-1 7n8n: where Yn > 0 is stepsize and 8n is psoudogradient satisfying the following two incqualities:
5 (VI(xn-))) 1(oo))
E [|
s An + Kif(En-n) (V E[s.))
In addition, Polyak & Tsypkin (1973) assume the step sizes "n and the sequence An jointly satisfy 20 An <
oo and 1he co, and then additionally that either (a) E limn-soo 7n < 2 LK3 Under these assumptions, they prove that lim (V B[sn]} = 0. In the stochastic setting, we have that S, = g(En-1), and as such their assumptions reduce to:
(b) An 0, K₁ 0, and
(a) An assumption similar to (ES): E [19(2..1)111 s An + Kif(En-1) K2 Vflll
(b) 2 < and 1 7n 0o.
(c) Either (1) Men o or (2) An = 0, K1 = 0, and limn- 7/ < LKa
2
Part (a) would subsume our assumption if we could set A, to be a constant, but the conditions on the sequences An and 7/n in parts (b) and (c) are not the same in our work- we explicitly the stepsize instead, and obtain non-asymptotie results.
32


Reviews:
[{"review_id": "lVLFlM1NYP", "reviewer": "Reviewer_1v7k", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n* The proposed assumption can cover existing assumptions, and the convergence results can also be adapted accordingly.\n* The proposed assumption can cover certain stochastic gradients that cannot be described by prior works.\n\nWeakness:\n\n* The novelty may be limited. As claimed by the authors in the text after Assumption 2, the expected smoothness has already been discovered in early works. Therefore, such a \"rediscovery\" cannot be viewed as a significant novelty. Besides, a detailed version of this assumption made in [Polyak & Tsypkin, 1973] should also be presented for comparison.\n\n* The depth of this paper is also not enough. It has been widely known that variance-reduced stochastic gradients can significantly improve the convergence in both convex and nonconvex optimization problems. Then what can we get if the problem is characterized by such as new assumption proposed in this paper?\n\n* The authors have paid too much attention to the background and introduction of existing assumptions and stochastic gradients, while the authors only present the key theoretical results within 3 pages. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the convergence rate of SGD based on a general assumption of the objective function, that can cover multiple designs of stochastic gradients. \n\nIn particular, this paper proposes a new assumption that can be reduced to a set of existing assumptions. \n\nThen the authors show that the assumption can cover the stochastic gradients that are designed based on a combination of gradient compression and subsampling, which cannot be described by prior assumptions.\n\nFinally, the authors develop the convergence rate of SGD for general smooth nonconvex problems and the problems satisfying the PL condition, which recover the existing convergence results of SGD.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "1. Detailed comparison with prior work [Polyak & Tsypkin, 1973] should be added.\n2. Convergence rate of stochastic variance reduced gradients should be considered.\n3. The authors should reorganize the paper to pay more attention to the theoretical results.", "broader_impact_concerns": "No concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zMemg0I2fL", "reviewer": "Reviewer_VX3G", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**.\n\n- This paper is well-written and easy to read. \n- The introduced expected smoothness condition is quite nice. Indeed, there are several important examples (e.g., subsampling and compression) that satisfy this condition but do not satisfy other assumptions used in \npast works. \n\n**Weaknesses**.\n\n- The proofs are straightforward and not so novel, which are basically collections of the existing techniques. (But, I acknowledge the significance of the paper as commented in strength.) \n\n- Recently, a closely relevant paper [Gower et al. (2021)] was published. They introduced an expected residual (ER) condition which is also an extension of the standard expected smoothness, although ER is a special case of the condition proposed in this submission. \nMy concern is the difference between these conditions. Is there any example that does not satisfy ER but satisfies the proposed expected smoothness condition?\n\n[Gower et al. (2021)] Robert M. Gower, Othmane Sebbouh, and Nicolas Loizou. SGD for Structured Nonconvex Functions: Learning Rates, Minibatching and Interpolation. AISTATS, 2021.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes an expected smoothness assumption in a general form that is looser than other assumptions in the literature and studies the convergence of SGD for nonconvex problems. Specifically, this work shows the convergence to a stationary point for smooth nonconvex problems and convergence to the optimal value with an additional PL-condition or quadratic growth condition with better rates. Moreover, several interesting examples that satisfy an expected smoothness condition are provided.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "- The discussion about the relationship with the ER condition would be helpful as commented above.\n\n\nMinor comment:\n- Typo: The last equation of the proof of Theorem 4 on page 31. The exponential is missing.", "broader_impact_concerns": "Not applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1rQgaPfvCG", "reviewer": "Reviewer_kLwJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n1. The theory of the paper is relatively elegant, the flow of the paper is clear. The proof of the paper is kind of straightforward.\n2. The theory provides a more general conditions compared to existing works, also it recovers existing $\\mathcal{O}(\\epsilon^{-4})$ complexity results.\n\nWeakness:\n1. I may be a bit concerned on whether the results is that insighful (I understand such judgement is a bit subjective, and TMLR tries to avoid 'significance/impact/novelty' in review). Personally I may view the proposed condition as an aggregation of the bounded variance condition (BV) and C-ES, so I may view the analysis is also an aggregation of the analysis of SGD on BV and SGD on C-ES (assuming the transfer from convex to nonconvex problems here is not that hard), so the convergence should be expected. I may highly appreciate if authors can provide more insights on the analysis compared to existing literature, e.g., what specifically the gap that authors find exciting and successfully get through.\n2. With the new assumption, basically it requires users to know more problem parameters (e.g., Lipschitz constants of each $f_\\xi$, rather than a unified $L_{\\text{max}}$, as shown in the importance sampling example), which may not be that practical regarding the proposed more \"practical\" assumption (kind of trade-off?). With more information on the problem, getting a boost in the convergence with the importance sampling should be expected. So I hope to learn more about the importance and insight of the proposed new assumption.\n\nSome other questions:\n1. As a relaxation of the sure-smoothness (or each $f_\\xi$ is Lipschitz smooth), does the averaged smooth condition ($\\mathbb{E}_\\xi||\\nabla f_\\xi(x)-\\nabla f_\\xi(y)||^2\\leq L^2||x-y||^2$) in many literature (e.g., SPIDER) implies the ES condition?\n2. In the subsampling and compression examples, the problem of interest is reduced to the form of finite-sum (rather than the purely stochastic form), many results come with a form of $\\mathcal{O}(\\text{poly}(n)\\epsilon^{-2})$ (e.g., full GD will naturally attains an $\\mathcal{O}(n\\epsilon^{-2})$ complexity), rather than the $\\mathcal{O}(\\epsilon^{-4})$ here, which exhibits some advantages in terms of the $\\epsilon$-dependence. Would you mind commenting on the difference between these two complexity results?\n\nI appreciate authors' efforts on the work, please definitely indicate here if I misunderstand the details. Thank you very much.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, authors proposed a new condition, expected smoothness (ES), for gradient estimator in stochastic nonconvex optimization. Authors claimed that the ES conditon is the most general one compared to most existing literature, and works for subsampling and compression strategies. With the new condtion, authors recovered $\\mathcal{O}(\\epsilon^{-4})$ complexity for general nonconvex problems, and $\\mathcal{O}(\\epsilon^{-1})$ complexity with an extra PL condtion.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Generally I hope authors can add more details or discussion to further justify the insights of the proposed assumption and the results. See above questions for more details.", "broader_impact_concerns": "This is a purely theory paper, I think there is no emergent ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is well-written and gives a comprehensive overview of the existing literature elaborating on connections between the existing assumptions and the proposed model here. The authors do an excellent job of demonstrating the utility of their model with numerous applications to various practical settings. There was a bit of concern shared by reviewers regarding the novelty of the techniques but the overall picture the paper contributes in furthering our understanding of SGD for large scale nonconvex problems far outweighs those concerns. The response from authors did a good job of clarifying some of the questions that came up during the review process. 
Decision: Accept (poster)