Year: 2021

Abstract:
Stochastic gradient descent (SGD) is widely used for the nonlinear, nonconvex problem of training deep neural networks, but its behavior remains poorly understood. In this work, we construct example optimization problems illustrating that, when not in the regimes that the previous works often assume, SGD can exhibit many strange behaviors that are undesirable. Our constructions show that, with specific choices of learning rate and specific model (1) SGD may converge to local maxima, (2) SGD might only escape saddle points arbitrarily slowly, (3) SGD may prefer sharp minima over flat ones, and (4) AMSGrad may converge to local maxima. We also realize results in a minimal neural network-like construction. One crucial message that our work delievers is that the learning rate needs to be chosen and scheduled carefully to guarantee convergence.

heading: 1 INTRODUCTION
text: SGD is the main optimization algorithm for training deep neural networks, and understanding SGD is widely regarded as a key step on the path to understanding deep learning (Bottou, 2012; Zhang et al., 2018; Xing et al., 2018; Mori et al., 2021; Du et al., 2018; Allen-Zhu et al., 2018; Wojtowytsch, 2021a;b; Gower et al., 2021; Ziyin et al., 2021; Gurbuzbalaban et al., 2021; Zou et al., 2021; Li et al., 2021; Feng and Tu, 2021). Despite its algorithmic simplicity (describable with only two lines of equations), SGD is hard to understand. The difficulty is threefold: (1) SGD is discrete-time in nature, and discrete-time dynamics are typically much more complicated to solve than their continuous-time counterparts (May, 1976); (2) SGD noise is state-dependent (Ziyin et al., 2021; Hodgkinson and Mahoney, 2020); and (3) the loss landscape can be nonlinear and non-convex, involving many local minima, saddle points, and degeneracies (the Hessian is not full-rank) in the landscape (Xing et al., 2018; Wu et al., 2017). Each of these difficulties is so challenging that very few works attempt to deal with all of them simultaneously. Most previous works on SGD are limited to the cases when the loss landscape is strongly convex (Ziyin et al., 2021; Liu et al., 2021; Hoffer et al., 2017; Mandt et al., 2017), or when the noise is assumed to be Gaussian and time-independent (Zhu et al., 2019; Jastrzebski et al., 2017; Xie et al., 2021); for the works that tackle SGD in a non-convex setting, often strong conditions are assumed. The reliance on strong assumptions regarding each of the challenges means that our present understanding of SGD for deep learning could be speculative. This work aims to examine some commonly held presuppositions about SGD and show that when all the three challenging factors are taken together, many counter-intuitive phenomena may arise. Since most of these phenomena are potentially undesired, this work can also be seen as a worse case anaysis of SGD.
In this work, we study the behavior of SGD in toy landscapes with non-convex loss and multi-minima. We approach the problem of SGD convergence from a different angle from many of the related works: instead of studying when SGD will converge, our result helps answer the question of when SGD might fail. In particular, the problem setting considers discrete-time SGD close to saddle points, where the noise is due to minibatch sampling, and the learning rate is held constant throughout training. In the next section, we define the SGD algorithm and the necessary notations. In Sec. 3, we discuss the related works. A warmup example is provided in Sec. 4. Sec. 5 introduces the main results. Sec. 6 presents the numerical simulations, including a minimal example involving a neural network. We also encourage the readers to examine the appendix. Sec. A presents additional experiments. Sec. C presents a continuous-time analysis of the problems in the main text and is relevant for discussing the unique discrete-time features of SGD. Sec. D presents all the proofs.

heading: 2 BACKGROUND
text: Notation and Terminology. We use λ to denote the learning rate. w is the model parameter. L̂(w;x), with the caret symbol ⋅̂, denotes the sampled loss function, which is a function of the data point, and L ∶= Ex[L̂] is the true loss landscape averaged over the data distribution; for notational conciseness, we often hide the dependence of L̂ on x when the context is clear. The lowercase t denotes the time step of optimization. In this work, the terms learning rate and step size are used interchangeably. The phrases loss, objective, and potential are also used interchangeably.
Now, we introduce the minibatch SGD algorithm. The objective of SGD can be defined as a pair (L̂, p(x)) such that L̂ is a differentiable function and p(x) is a probability density. We aim at finding the minimizer of the following differentiable objective:
L(w) = Ex∼p(x)[L̂(w;x)], (1)
where x is a data point drawn from distribution p(x) and w ∈ RD is the model parameter. The gradient descent (GD) algorithm for L is defined as the update rule wt = wt−1 − λ∇wL(w,x) for some randomly initialized w0, where λ is the learning rate. The following definition defines SGD. Definition 1. The minibatch SGD algorithm computes the update to the parameter w with the following equation: wt = wt−1 − λĝt, where ĝt = ∇L̂(wt−1;xt), and xt is drawn from p(x) such that xi is independent of xj for i ≠ j.

heading: 3 RELATED WORKS
text: Escaping Saddle Points. Because neural networks are highly nonlinear, the landscape of neural networks is believed to have many local minima and saddle points (Du et al., 2017; Kleinberg et al., 2018; Reddi et al., 2018b). It is thus crucial for any optimization algorithm to be able to overcome saddle points efficiently and not become stuck in a suboptimal minimum. However, the majority of works on escaping saddle points assume no stochasticity or only artificially injected noise (Du et al., 2017; Jin et al., 2017; Ge et al., 2015; Reddi et al., 2018b; Lee et al., 2016). Some physics-inspired approaches take the stochasticity into account but often rely on continuous-time approximation and nontrivial assumptions regarding the stationary distribution of the parameter (Mori et al., 2021; Xie et al., 2021; Liu et al., 2021). In the context of stochastic non-convex optimization, one setting that is relatively well-understood is when the learning rate decreases to zero in the training (Pemantle, 1990; Vlaski and Sayed, 2019; Mertikopoulos et al., 2020). For example, Pemantle (1990) shows that stochastic approximation algorithms can avoid unstable stationary points; Vlaski and Sayed (2019) shows that SGD avoids saddle points in expectation, Mertikopoulos et al. (2020) shows that SGD avoids saddle points in probability; the later two also provide precise conditions under which the saddles points are avoided. These work, therefore, differ from the present work in that, while they assume that the learning rate decrease to zero, we only consider the case when the learning rate is held fixed. It is important to note that our result does not contradict these previous results. To the opposite, our result may be interpreted as reinforcing the crucial messages in these previous works, namely, that the learning rate needs to be chosen and scheduled carefully to guarantee convergence.
Role of Minibatch Noise. One indispensable aspect of SGD is its special stochasticity due to minibatch sampling (Wu et al., 2020; Ziyin et al., 2021; Zhu et al., 2019; Hodgkinson and Mahoney,
2020), whose strength and structure are determined by the model architecture and the distribution of the data points. In fact, it has been shown that the models trained with SGD significantly outperform the models directly trained with GD (Hoffer et al., 2017), which suggests that minibatch noise in SGD is beneficial for training. Understanding the role of minibatch noise is therefore of fundamental and practical value. However, in the previous works, the focus is on analyzing the behavior of SGD on the landscape specified by L; the unique structures due to the noise in SGD are often treated in an oversimplified manner without referring to the actual models in deep learning or the actual data distributions. For example, Daneshmand et al. (2018) assumes that the noisy gradient is negatively correlated with the direction of the Hessian with the smallest eigenvalue, the validity of which is unclear when the Hessian is not full rank. Kleinberg et al. (2018) assumes that the noise in SGD makes the landscape one-point strongly convex, which implies that the loss landscape before the convolution is already close to a strongly convex landscape. Another commonly assumed condition is that the loss-landscape satisfies the Lojasiewicz condition, which is non-convex but implies the unrealistic condition that there is no local minimum except the global minimum (Karimi et al., 2020; Wojtowytsch, 2021a; Vaswani et al., 2019), which is not sufficient to understand the complicated dynamics that often happen in a setting with many minima. In Sec. 6.3, we show that these assumptions are violated for a minimal toy nonlinear neural network with two layers and one hidden neuron. While our example may or may not be directly relevant for realistic settings in the deep learning practice, our work highlights the importance of analyzing the actual noise structure imposed by a deep neural network in future works.
Large Learning Rate Regime. Recently, it is realized that networks trained at a large learning rate have a dramatically better performance than networks trained with a vanishing learning rate (lazy training) (Chizat and Bach, 2018). Lewkowycz et al. (2020) shows that there is a qualitative difference between the lazy training regime and the large learning rate regime; the performance features two plateaus in testing accuracy in the two regimes, with the large learning rate regime performing much better. However, almost no previous theory exists for understanding SGD at a large learning rate. Our work is also relevant for understanding what happens at a large learning rate.

heading: 4 A WARM-UP EXAMPLE
text: This section studies a special example of the main results of our work. While the setting of this example is restricted, it captures the essential features and mechanisms of SGD that we will utilize to prove the main results. Consider the following loss function:
L̂(w) = x 2 w2, (2)
where x,w ∈ R and x is drawn from an underlying distribution p(x) at every time-step. We let E[x] = a and assume that Var[x] = σ2 is finite. This gives rise to a true, sample-averaged loss of L(w) = a
2 w2. The stationary point of this loss function is w = 0. When a > 0, the underlying
(deterministic) landscape is a minimum; the a > 0 case is now well understood in the discrete-time limit and when the underlying noise is state-dependent (Liu et al., 2021). When a < 0, the point w = 0 is a local maximum; one hopes that the underlying dynamics escape to infinity, and our goal is to understand the behavior of SGD in this case.
The following proposition and proof show that for the seemingly benign data distribution p(x) (zero-mean and bounded) above, SGD cannot escape from the local maximum when the learning rate is set to be 1.
Proposition 1. Let λ = 1 and a < 0 and p(x) be the distribution such that x = 1 and x = −1 + a with equal probability. Then, w converges to w = 0 with probability 1 with the SGD algorithm, independent of the initialization.
Proof. By the definition of the SGD algorithm, we have
wt+1 = { 0 with probability 0.5; (2 − a)wt otherwise.
(3)
Therefore, after t time steps, wt has at most 2−t probability of being non-zero. For infinite t, wt = 0 with probability 1. ◻
This simple toy example illustrates a few important points. First of all, it suggests that one cannot use the expected value of w to study the escape problem of SGD. The expectation value of wt is
Ex[wt] = (1 − a/2)Ex[wt−1] = (1 − a/2)tw0 = w0eln(1−a/2)t, (4) which is an exponentially fast escape from any initialization w0 for a < 0. In fact, ln(1− a) is exactly the escape time scale of the GD algorithm. This is counter-intuitive: SGD will converge to the local maximum with probability 1, even if its expected value escapes the local maximum exponentially fast.
Alternatively, one might also hope to use the expected loss or the norm Ewt[L(wt)] ∼ ∣∣wt∣∣22 as the metric of escape, but it is not hard to see that they suffer the same problem. At time t, because wt = 0 with probability 1− 2−t and wt = (2− a)tw0 with probability 2−t, we have E[L(wt)]∝ w20e 2t ln 2−a√ 2 , which also predicts an exponentially fast escape. Again, this fails to reflect the fact that SGD has only a vanishingly small probability of escaping the constructed local maximum. Statistically speaking, the norms of wt fail to be good metrics to measure the escape rate because these metrics are not robust against outliers. In our example, there is only 2−t probability for SGD to escape the local minimum, yet the speed at which this outlier escapes the maximum outweighs the decay in its probability through time, thus contributing more to the expected norm than any other events. This example also shows the difficulty and subtlety of dealing with the escape problem. This example also emphasizes the importance of showing convergence in probability for future works in non-convex optimization because convergence in expectation is not sufficient to guarantee a good empirical performance. The results in the rest of this work can be seen as extensions of this special case to less restrictive settings.

heading: 5 MAIN RESULTS
text: In this section, we present four main theoretical results: (1) SGD may converge to a local maximum; (2) SGD may escape a saddle point arbitrarily slowly; (3) SGD may prefer sharp minima over flat ones; and (4) AMSGrad may converge to a local maximum. Numerically, we show that these results are relevant for a minimal example involving a toy neural network, where some assumptions regarding the theoretical study of SGD are violated (Sec. 6.3).

heading: 5.1 SGD MAY CONVERGE TO A LOCAL MAXIMUM
text: We show that there are cases where SGD may fail to escape saddle points, while GD can successfully escape; this contradicts the suggestions in Kleinberg et al. (2018) that SGD always escapes a local minimum faster than GD. It is appropriate to begin with the definition of a saddle point. Definition 2. w is said to be a stationary point of L(w) if ∇wL(w) = 0. w is a minimum of L if, for all w′ in a sufficiently small neighborhood of w, L(w′) ≥ L(w). w is said to be a saddle point if w is a stationary point but not a local minimum.
This definition of saddle points, which includes local maxima, agrees with the common definition in the literature (Daneshmand et al., 2018). The following proposition generalizes our warmup example to a family of 1D problems in which SGD converges to a local maximum. Proposition 2. Let Eq. 2 be the loss function. Let λ > 0, and p(x) be such that µ ∶= E[ln ∣1 − λx∣] and s2 ∶= Var[ln ∣1 − λx∣]; then SGD converges to the local maximum in probability if µ < 0.
Proof Sketch. We first show that 1√ t ln ∣wt/w0∣ obeys a normal distribution as t→∞. This allows us to deduce the cumulative distribution function (c.d.f.) of w as t→∞. The c.d.f. has an bifurcative dependence on the sign of µ ∶= E[ln ∣1 − λx∣]. When µ < 0, the distribution of wt converges to 0 in probability. See Sec. D.2 for the detailed proof. ◻ Corollary 1. Let Eq. 2 be the loss function and p(x) = 1
2 δ(x−1)+ 1 2 δ(x+1−a); then SGD converges
to the local maximum in probability if
a a − 1 < λ < a −
√ a2 − 8a + 8
2(a − 1) . (5)
One special feature of this example is that the state-dependent noise vanishes at the saddle point. In this particular example, this is achieved by having an SGD noise proportional to the gradient thus vanishing at the saddle point. However, in general, a vanishing noise and state-dependent noise do
not always emerge together. In terms of minibatch sampling, this means the loss for all data points share a common stationary point and this is unlikely to be natural. Thus, what this example provides is a worst-case scenario: the state-dependent noise leads to vanishing noise at the saddle point. Note that the convergence is independent of the actual shape of the the distribution p(x), and so may apply to many kinds of distributions. For simplicity, we consider the case when p(x) is a Bernoulli variable. There are three different regimes/phases for this simple escaping problem (also see Fig. 1-Right). The escape regime λ > a− √ a2−8a+8
2(a−1) is also present when a > 0; it means that in this regime, SGD will also
escape w0 even if it is a local minimum. This regime is not present if we perform a continuous-time analysis (Sec. C), showing that this region is due to the instability of the discrete-time SGD algorithm. This kind of escaping is undesirable because, in this regime, the local gradient cannot provide any guidance for minimizing the loss. The region a
a−1 < λ < a−
√ a2−8a+8 2(a−1) is the trapped regime. This is
due to the special structure of the minibatch noise – a comparison with the continuous-time analysis suggests that SGD will not be trapped if the noise is not position dependent (Sec. C.1). The small learning rate regime λ < a
a−1 is the successful escape regime because the SGD noise is of order λ2
and becomes negligible at a small λ (Liu et al., 2021). This regime corresponds to the lazy learning regime of training: with negligible noise and vanishing gradient, SGD is known to optimize neural networks well (Du et al., 2018). This discussion also justifies our later choice for placing an upper bound to λ and focus only on the second and third regime in Sec. 5.3 and 5.4.
It is important to understand why the previous works that show that the previous guarantees of convergence does not apply to this particular example, for example, Ge et al. (2015). In comparison, the SGD algorithm is not guaranteed to escape the saddle point in this example because two crucial assumptions of Ge et al. (2015) is violated: (1) the gradient does not have a bounded fluctuation around its mean (and so even parameters very far away from the saddle can be brought back close to the saddle point due to the noise), and (2) we do not inject additive noise to the gradient and the actual noise decreases as we approach the saddle point (and so the closer we are to the saddle point, the less likely one can escape). The standard theoretical guarantee of convergence in Ge et al. (2015) thus does not apply to this example. This example shows why the state-dependent noise of SGD can strongly influence the convergence. We perform a continuous-time analysis in Section C of this example, and the continuous-time result shows that the convergence to local maxima occurs when the minibatch noise dominates the gradient, namely, when the signal to noise ratio becomes smaller than 1. This suggests the importance of minibatch noise in influencing the dynamics of SGD.

heading: 5.2 SGD MAY ESCAPE SADDLE POINTS ARBITRARILY SLOWLY
text: In the previous section, we have shown that one can adversarially construct a loss function for a fixed learning rate such that the SGD algorithm converges to a local maximum. This section shows that, even if one can tune the learning rate at will, there is a loss landscape where SGD can take an arbitrarily long time to escape, no matter how one chooses the learning rate. We begin by defining the escape rate. Definition 3. The asymptotic average escape rate of w at learning rate λ, γ(λ), is defined as γ ∶= limt→∞ 1tEwt[ln ∣wt∣]. The optimal escape rate is defined as γ
∗ ∶= supλγ(λ). Remark. This definition can be seen as a generalization of the Lyapunov exponent to a probabilistic setting. We note that this definition of escape rate differentiates this work from the standard literature on escaping saddles, where the focus is on the time scale for reaching a local minimum when saddle points exist (Ge et al., 2015; Daneshmand et al., 2018; Jin et al., 2017), while our definition focuses on the time scale of escaping a specified saddle point. These two time scales can be vastly different.
As discussed in the previous section, the undesired escape can be achieved by setting the learning rate to be arbitrarily large; however, this is not possible in practice because the learning rate of SGD is inherently associated with the stability of learning (Ziyin et al., 2021), and λ usually has to be much smaller than 1. Proposition 3. Let the loss function be Eq. (2), a < 0 and p(x) ∼ 1
2 δ(x − 1) + 1 2 δ(x − a + 1), where
δ is the Dirac delta function. Assuming that λ ≤ 1, SGD algorithm escapes the saddle point of Eq. (2) arbitrarily slowly, i.e., for any > 0, there exists loss function such that supλ γ(λ) < . Specifically, γ∗ = ln 2−a
2 √ 1−a
, and, for any > 0, γ∗ < if ∣a∣ ≤ 2 ∣−e √ e2 − 1 − e2 + 1∣.
Intuitively, one expects escaping to be easier as we increase the learning rate (Kleinberg et al., 2018), and that the escape rate should monotonically increase as we increase the learning rate. This example
conveys a novel message that the escaping efficiency is not monotonically increasing as the learning rate increases. For this example, the escape rate first decreases and starts to increase only when λ is quite large. This example also shows the subtlety in the escaping problem. On the one hand, one needs to avoid a too large step size to avoid training instability. On the other hand, one cannot use a too small learning rate because a small learning rate also makes optimization slow. This example suggests that there should be a tradeoff between learning speed and learning stability and that we may construct a general theory for finding the best learning rate that achieves the best tradeoff; however, this tradeoff problem is a sufficiently complex problem on its own and is beyond the scope of the present work.

heading: 5.3 SGD MAY PREFER SHARPER MINIMA
text: Modern deep neural networks are often overparametrized and capable of memorizing all the training data points quite easily. This means that the traditional metrics such as Rademacher complexity cannot be used to guarantee the generalization capability of SGD (Zhang et al., 2017). Nevertheless, neural networks are found to generalize very well. This inspired the hypothesis that SGD, the main training algorithm of neural networks, contains some implicit regularization effect such that it biases the neural network towards simpler solutions (Neyshabur et al., 2017; Soudry et al., 2018). One hypothesized mechanism of this regularization is that SGD selects flat minima over the sharp ones (Hochreiter and Schmidhuber, 1997; Meng et al., 2020; Xie et al., 2021; Liu et al., 2021; Mori et al., 2021; Smith and Le, 2018; Wojtowytsch, 2021b). In this work, we show that this may not be the case because the dynamics and convergence of SGD crucially depend on the underlying mini-batch noise, while the sharpness of the landscape is independent of the noise. Before introducing the results, let us first define the sharpness of a local minimum. Definition 4. Let w = w∗ be a local minimum of the loss function L(w); the sharpness s(w∗) of a local minimum w∗ is defined as s(w∗) ∶= Tr[∇2wL(w∗)]. We say that a minimum w∗1 is sharper than w∗2 if and only if s(w∗1) > s(w∗2).
In this definition, the sharpness is the trace of the Hessian of the quadratic approximation to the local minimum. Other possible definitions such as the determinant of the Hessian are essentially similar to this definition (Dinh et al., 2017).
For an explicit construction, consider the following objective on a 2-dimensional landscape:
L̂(w1,w2) = 1
2 [−(w1 −w2)2 − (w1 +w2)2 + (w1 −w2)4 + (w1 +w2)4 − 2aw22 + xbw21] . (6)
for a, b > 0 and p(x) = 1 2 δ(x− 1)+ 1 2 δ(x+ 1). The diagonal terms of the Hessian of the loss function is given by ⎧⎪⎪⎨⎪⎪⎩ ∂2 ∂w21 L̂(w1,w2) = −2 + 12w21 + 12w22; ∂2 ∂w22 L̂(w1,w2) = −2 − 2a + 12w21 + 12w22. (7) There are four local minima in total for this loss function. These minima and their sharpnesses are
⎧⎪⎪⎨⎪⎪⎩ (w1,w2) = (± 1√2 ,0) with s = 4; (w1,w2) = (0,± √ 1+a 2 ) with s = 4 + 4a. (8)
We see that for positive a, the minima at (0,± √
1+a 2 ) are sharper than the other two minima. We show that SGD will converge to these sharper minima. For this example, we assume that w1 and w2 are bounded because, if initialized sufficiently far from the origin, the fourth-order term will cause divergence. Proposition 4. Let the loss function be Eq. (6), ∣w1∣ ≤ 1 and ∣w2∣ ≤ 1. For any λ < c for some constant c = O(1), there exists some b such that if SGD converges in probability, it will converge to the sharper minimum at (w1,w2) = (0,± √ 1+a 2
). Remark. In this example, the noise is not full-rank, which is often the case in a deep learning setting for overparametrized networks (Wojtowytsch, 2021a). For example, when weight decay is used, the Hessian of the loss function should be full rank, while the rank of the noise covariance should be proportional to the inherent dimension of the data points, which is in general much smaller than the dimension of the Hessian in deep learning (Ansuini et al., 2019).
The proof is technical and given in the appendix Sec. D.5. In fact, it has been controversial whether finding a flat minimum can help generalization. For example, Dinh et al. (2017) shows that, for every flat minimum of a ReLU-based net, there exists a minimum that is arbitrarily sharper and generalizes as well. However, this work does not rule out the possibility that conditioning by using gradient-based optimization, the performance of the sharper minima that gradient descent finds is worse than the performance of the flatter minima. If this assumption is valid, then biasing gradient descent towards flatter minima can indeed help, and the stochasticity of SGD has been hypothesized to help in this regard. However, our construction shows that SGD, on its own, may be incapable of helping SGD converge to a flatter minimum. At least some other assumptions need to be invoked to show that SGD may help. For example, the definition of the neural networks may endow these models with a special kind of Hessian and noise structure that, when combined with SGD, results in a miraculous generalization behavior. However, no previous work has pursued this direction in sufficient depth to our knowledge, and future studies in this direction may be fruitful.
Moreover, it is worth commenting that, in the particular example we studied, SGD will converge to the local minimum at (0,± √
1+a 2 ) independent of its sharpness due to the special noise we constructed. This fact hints at the crucial role of noise in SGD optimization. 5.4 NON-CONVERGENCE OF ADAPTIVE GRADIENTS mt = β1mt−1 + (1 − β1)ĝt; (9) vt = β2vt−1 + (1 − β2)ĝ2t ; (10) v̂t = max(v̂t−1, vt); (11)
wt = wt−1 − λ√ v̂t mt. (12) Adam (Kingma and Ba, 2014) and its closely related variants such as RMSProp (Tieleman and Hinton, 2012) have been shown to converge to a local maximum even for some simple convex loss landscapes (Reddi et al., 2018a). The proposed fix, named AMSGrad, takes the maximum of all the previous preconditioners in Adam. This section shows that AMSGrad may also converge to a local maximum even in simple non-convex settings. Let xt ∼ p(x) and ĝt = ∇L̂(xt,wt−1), the Adam algorithm is given by Eq. (9)-(12), where v0 = m0 = 0. We have removed the numerical smoothing constant from the denominator, which causes no problem if w0 is initialized away from 0. Here, β1 is the momentum hyperparameter. vt is called the preconditioner, and β2 is the associated hyperparameter. The standard value for β1 is 0.9 and β2 is 0.999. In our theory, we only consider the case when β1 = 0. The experiment section shows that a similar problem exists when β1 > 0 (with additional interesting findings). Intuitively, this is easy to understand because AMSGrad behaves like SGD asymptotically, so we only have to wait for long enough, and the results in previous sections would apply. The construction below follows this intuition. Proposition 5. Let wt ∈ [−1,1], and w0 ≠ 0. For fixed λ < 1 and the loss function in Eq. (2) there exists a < 0 such that the AMSGrad algorithm converges in probability to a local maximum. Remark. The proof is given in Sec. D.6. Note that the example we construct is similar to the original construction in Reddi et al. (2018a) that shows that Adam converges to a maximum while AMSGrad succeeds in reaching the global minimum. In their example (with some rescaling), the gradient is a random variable
ĝt = { 1 with probability q ≈ 1; c0 < 0 with probability 1 − q ≪ 1;
(13)
such that the expected gradient is negative, and Adam is shown to converge to the direction opposite to the gradient descent (therefore to a maximum). In our example, the gradient is (roughly)
ĝt = { wt−1 with probability 1/2; −(1 + a)wt−1 with probability 1/2.
(14)
Therefore, our example can be seen as a minimal generalization of the Reddi et al. (2018a) example to a non-convex setting.

heading: 6 EXPERIMENTAL DEMONSTRATIONS
text: We perform experiments to illustrate the examples studied in this work. Due to space limitations, we illustrate the escape rate and the convergence to sharper minima example in Appendix Sec. A.

heading: 6.1 SGD CONVERGES TO A LOCAL MAXIMUM
text: Numerical results are obtained for the setting described in section 5.1. See Fig. 1-Left. The loss landscape is defined by L(w) = 1
2 aw2. In this numerical example, we set λ = 0.8 and a = −1, and
the histogram is plotted with 2000 independent runs. We see that the distribution converges to the local maximum at w = 0 as the theory predicts. For better qualitative understanding, we also plot the empirical phase diagram of this setting in Fig. 1-Right. We perform numerical evaluations for different combinations of a and λ on a a − λ plane, and for each combination, we plot the percentage of w that escapes the central bin (white is 100% and dark blue is 0%). For completeness, we also plot the case when a > 0, i.e., when the critical point at w = 0 is a local minimum. The orange line shows where the phase transition is expected to happen according to Eq. (5). The fact that the orange line agrees exactly with the empirical phase transition line confirms our theory.
We also numerically study a closely related fourth-order loss landscape, defined as L̂(w) = 1 2 xw2 + 1 4 w4, where the distribution p(x) is the same as before. The expected loss L(w) has two local
minima located as w = ±√a. We are interested in whether SGD can converge to these two points successfully; note that, without the fourth-order term, this loss is the same as the quadratic loss we studied. See Fig. 1-Middle. As before, a = −1 and λ = 0.8. We see that the distribution of w also concentrates towards the local maximum after training, and no w is found to converge to the two local minimum even if a significant proportion of w is initialized close to these two minima. A phase diagram analysis is given in the Sec. A.1.

heading: 6.2 NON-CONVERGENCE OF AMSGRAD
text: In this section, we illustrate the example of the convergence behavior of AMSGrad described in Sec. 5.4; for reference, we also plot the behavior of Adam and gradient descent for this example. See Fig. 2. The experiment is the average over 2000 runs, each with 5 × 104 steps. The uncertainty is reflected by the shaded region (almost invisibly small). The loss function is the same as discussed in Sec. 5.4 with a = −0.1. For illustration purposes, we set λ = 0.2 and β2 = 0.999 for both Adam and AMSGrad. When momentum is used, we set β1 = 0.9. GD is run with a learning rate of 0.01. Without momentum, both Adam and AMSGrad converge to the local maximum with almost the same speed. When momentum is added, AMSGrad still converges to the local maximum. Curiously, Adam is no longer attracted to the local maximum but also remains away from the local minimum. This suggests that Adam, with momentum, might have a better capability of escaping saddle points than AMSGrad and might be preferable to AMSGrad in a non-convex setting.

heading: 6.3 A TOY NEURAL NETWORK EXAMPLE
text: We have now illustrated several unexpected SGD phenomena in artificial settings, inspiring the natural question of whether they might be important for an actual neural network. Here we construct a neural-net-like optimization problem to show that the problems with convergence might indeed arise for a neural network. The toy neural network function is given by f(x) = w2σ(w1x), where w1,w2 ∈ R. σ is the nonlinearity, and we let σ(x) = x2 as a minimal example. This is the simplest kind of nonlinear feedforward network one can construct (2 layer with a single hidden neuron). We also pick a minimal dataset with a single data point: x = 1 with probability 1 and y ∈ {−1,2}, each with probability 0.5, i.e., the label has a degree of inherent uncertainty, which is often the case in real problems. We use mean squared error (MSE) as the loss function. Therefore, the expected loss is L(w1,w2) = 12(w2w 2 1 + 1)2 + 12(w2w 2 1 − 2)2. The global minimum L∗ = 2.25 is
degenerate, and is achieved when w1 = 1/ √
2w2. There is also a manifold of saddle points given by {(w1,w2)∣w1 = 0,w2 ≥ 0}, all with L = 2.5. Despite the simplicity of this example, it contains a few realistic features in a realistic problem, including (1) inherent uncertainty in the data, (2) a nonlinear hidden layer, and (3) a degenerate minimum with zero eigenvalues in the Hessian.
More importantly, most previous works on the convergence or escaping behavior of SGD are inapplicable to this example. One dominant assumption is the ρ-Hessian Lipschitz property (Jin
et al., 2017; Ge et al., 2015), which does not hold for this example in particular. Another common assumption is the PL condition (Karimi et al., 2020; Wojtowytsch, 2021a; Vaswani et al., 2019), which does not apply due to the existence of the saddle point. One recent assumption is that the loss function is 1-point convex for all points (Kleinberg et al., 2018), which is also ruled out because (0,0) is not 1-point strongly convex. Another relevant assumption is the correlated negative curvature assumption, which also does not hold for (0,0). We show in Sec. A.5 in detail why these conditions are violated.
See Fig. 3 for the experiment with this example. Here, w1 is initialized uniformly in [−1,1]; w2 is initialized uniformly in [0,1] to be closer to the global minimum and away from the saddle point (standard initialization such as initializing in [−1,1] does not change the conclusion). The left figure shows the stationary distribution of the model parameter at a small learning rate (λ = 0.001). All the parameters are located in the global-minimum valley as expected. In contrast, when the learning rate is large (λ = 0.1), the central figure shows that all models (1000 independent runs) converge to the saddle point at (0,0). The right figure investigates this change more systematically and shows the change in the average stationary training loss of the models as we increase the learning rate from 0.001 to 0.15. We see that for a small learning rate, the training loss is close to that of the global minima (L = 2.25), and for a significant range of large learning rates, the model invariably converges to a saddle point (L = 2.5). One additional interesting observation is that the model diverges at λ ≈ 0.11, and there is almost no sign of such divergence (such as increased fluctuation) when the learning rate is close to the divergent threshold. This example shows the relevance of our results to the study of neural networks. In fact, it has often been noticed that at convergence, many large-scale neural networks in real tasks exhibit negative eigenvalues in the Hessian (Alain et al., 2019; Granziol et al., 2019). The existence of negative eigenvalues at a late time suggests either the possibility that the algorithm has yet to escape or that it has indeed been attracted to such saddle points; if the latter is true, then our work offers an explanation. The emergence of negative eigenvalues at the end of training serves as indirect evidence that our result may be relevant for larger neural networks. Finally, we add the cautionary note that it remains an open question whether this toy example we offer is relevant for deep learning in practical settings or not, and it should be an important future work to investigate this problem.

heading: 7 DISCUSSION
text: We have shown that, when the learning rate is not carefully chosen or scheduled, SGD can exhibit many undesirable phenomena, such as convergence to local maxima or saddle points. The limitation of our work is clear. At best, all the constructions we made are minimal and simplistic toy examples that are far from practice, and investigating whether the discovered messages are relevant for deep learning or not is the one important immediate future step. Other relevant questions include: (1) Does convergence to saddle points help or hurt generalization? (2) If it hurts, how can we modify SGD to better avoid saddle points? We suspect changing learning rates, changing batch size, or injecting noise may help, but a convincing theoretical guarantee in a realistic setting is as yet lacking. (3) If saddle points do not have worse generalization, our results motivate for understanding why. This is especially relevant because we showed that using a large learning rate is more likely to converge to saddle points, while previous works have shown that using a large learning rate can improve generalization; combined, this may imply that certain saddle points may have intriguing but unknown regularization effects.

heading: A ADDITIONAL EXPERIMENTS

heading: A.1 PHASE DIAGRAM OF THE FOURTH-ORDER LOSS FUNCTION
text: The fourth-order loss function is a more realistic loss function than the one we considered in Sec. 5.1. We also performed one experiment in the main text (See Fig. 1). Here, we plot its phase diagram. We see that for this loss landscape also, there is some region such that for all λ, SGD converges to the local maximum. This loss landscape is very difficult to study in discrete time as it is known to lead to chaotic behavior at large step size (May, 1976). Therefore, we alternatively try to understand this landscape using continuous approximation. See Sec. C.

heading: A.2 ESTIMATING THE ESCAPE RATE
text: In the escape rate experiments, the empirical results are obtained from the proposed approximation. Here, we show that it is valid. See Fig. 5, where we plot the estimated γ as a function of the training step. We see that the estimated value converges within about 20 steps. We, therefore, estimate the escape rates at time step 100 in the main text.

heading: A.3 ESCAPE RATE EXPERIMENTS
text: This section illustrates the slow escape problem studied in Sec. 5.2. See Fig. 6-Right. γ is calculated by averaging the first 50 time steps across 2000 independent runs. We see that, as our theory predicts, as ∣a∣ gets closer to 0, the optimal escape rate decreases towards 0. This implies that there exists a landscape such that SGD is arbitrarily slow at learning, independent of parameter tuning. One additional observation is that the optimal learning rate is neither too large nor too small, i.e., there seems to be a tradeoff between the escape speed and escape probability (and between the speed and stability). See also the discussion at the end of Sec. 5.2.

heading: A.4 CONVERGENCE TO THE SHARPER MINIMUM
text: We use the same 2-dimensional loss landscape defined in Sec. 5.3. The experiment is run with 2000 independent simulations and learning rate λ = 0.05. See Fig. 7, where we overlap the underlying landscape with the empirical distribution (the heat map). We see that, even though the initialization overlaps significantly with the local flatter minimum, all points converge to the sharper minimum, as the theorem predicts.

heading: A.5 FAILURE OF STANDARD ASSUMPTION FOR THE NEURAL NETWORK EXAMPLE
text: In this section, we show that the standard assumptions in the previous literature do not hold for the example studied in Sec. 6.3. One common assumption is that the loss function is ρ-Hessian Lifschitz. Definition 5. A loss function L is said to be ρ-Hessian Lifschitz if, for some ρ > 0,
∀w,w′, ∣∣∇2L(w) −∇2L(w)∣∣ ≤ ρ∣∣w −w′∣∣. (15)
This certainly does not hold for the example we considered. Explicitly, let w2 = 0, the Hessian H(w) of the loss function is Hw1,w1 =Hw1,w2 =Hw2,w1 = 0, (16) and Hw2,w2 = −w21. (17) Let w1 →∞ violates the ρ-Hessian Lifschitz property. Another common assumption for the non-convex setting is the PL condition. Definition 6. Let L∗ be the value of L at the global minimum. A loss function L is said to satisfy the PL (Lojasiewitz) condition if
∀w, ∣∣∇L(w)∣∣2 ≥ µ(L(w) −L∗), (18) for some µ > 0.
This also does not hold due to the existence of the saddle point. Let (w1,w2) = (0,0). The gradient is zero, but the right-hand side is greater than 0.
Recently, the correlated negative curvature assumption has been proposed to study the escaping behavior of SGD.
Definition 7. Let vw be the eigenvector of the minimum eigenvalue of the HessianH(w). L satisfies the correlated negative curvature assumption if, for some γ > 0,
∀w,Ex[⟨vw,∇L(x,w)⟩2] > γ (19) where x is the data point.
This also does not hold. The point (0,0) violates this condition because ∇L(x,w) = 0 for all x at (w1,w2) = (0,0). In fact, it is exactly this point that violates this condition that the SGD converges to in this example.
Recently, the one point strongly convex assumption has been proposed to study the escaping behavior of SGD. Definition 8. A loss function is said to satisfy the c-one point strongly convex condition with respect to w∗ for all gradient noise , c > 0, define v =w − λ∇L(w), we have
⟨−∇E L(v − λw,w∗ − y) ≥ c∣∣w∗ − v∣∣22, (20) where is the random noise caused by minibatch sampling.
This assumption is equivalent to that the loss landscape is strongly convex after convoluting with the noise. This condition implies that there is only a single stationary point. However, this is not the case, consider any point with w1 = 0 and w2 ≤ 0. These points have zero gradient for w1 ≤ 0 and so = 0 with probability 1. Then, these points all have the same loss after the convolution:
E[L(v − λ )] = L(v), (21) which either imply that the landscape is not convex or that there is more than 1 stationary point, and so the system is not one point strongly convex.

heading: B ADDITIONAL THEORETICAL CONCERNS
text: Removed after update.

heading: C CONTINUOUS-TIME APPROXIMATION WITH FOKKER-PLANCK EQUATION
text: In this section, we further explore the role of multiplicative noise in SGD dynamics by investigating a continuous-time process. Compared with the discrete-time analysis, continuous-time analysis is usually more powerful in terms of calculation: it is able to deal with more complicated potentials. While the validity of this approximation may be questioned, we present numerical results to show that it is justified to study the example under consideration.

heading: C.1 FOKKER-PLANCK EQUATION AND ITS STATIONARY DISTRIBUTION
text: Recall that the SGD update takes the form (see Sec. 2)
∆wt = −λ∇L + λ √ Cηt, (22)
when λ < 1, the above equation may be approximated by a continuous-time Ornstein-Uhlenbeck process (Mandt et al., 2017)
dw(t) = −λ∇Ldt + λ√ S
√ C(w)dW (t), (23)
where λ is the learning rate; we have also introduced S as the batch size. dW (t) is a stochastic process satisfying
{dW (t) ∼ N (0, dtI),E[dW (t)dW (t′)]∝ δ(t − t′). (24)
By the definition of the underlying discrete-time process, the term dw(t) depends only on w(t) and has no dependence on w(t+ dt). Thus, the stochastic integration should be interpreted as Ito, leading to the following Fokker-Planck equation (Risken, 1996).
∂P [w, t∣w(0),0] ∂t = λ∇ ⋅ {(∇L)P [w, t∣w(0),0]} + λ 2 2S Tr(∇2{C(w)P [w, t∣w(0),0]}), (25) where Tr[⋅] is the trace. For simplicity, only one-dimensional version of the Fokker-Planck equation is considered, which is
∂P [w, t∣w(0),0] ∂t = λ ∂ ∂w {∂L ∂w P [w, t∣w(0),0]} + λ 2 2S ∂2 ∂w2 {C(w)P [w, t∣w(0),0]}. (26)
The probability P [w, t∣w(0),0] is normalized by definition, thus, the total probability is conserved. Expressing the conservation of probability in a microscopic manner:
∂P [w, t∣w(0),0] ∂t = − ∂ ∂w J(w, t∣w(0),0), (27)
where J[w, t∣w(0),0] is the probability flow. Comparing with the Fokker-Planck equation, the expression of J[w, t∣w(0),0] is obtained:
J[w, t∣w(0),0] = λ∂L ∂w P [w, t∣w(0),0] + λ 2 2S ∂ ∂w {C(w)P [w, t∣w(0),0]}. (28)
Assuming that the SGD dynamics is ergodic, there is a unique stationary distribution for w when t→∞, and it satisfies
0 = λ ∂ ∂w [∂L ∂w P (w)] + λ 2 2S ∂2 ∂w2 [C(w)P (w)]. (29)
The dependence on the initial condition could be dropped due to ergodicity. Integrating over w on both sides, one finds
J0 = λ ∂L ∂w P (w) + λ
2
2S
∂
∂w [C(w)P (w)], (30)
where J0 is the integral constant, which could be interpreted as the stationary flow of probability going from −∞ to ∞. When the loss function is convex, P (∞) = 0, leading to a trivial flow J0 = 0. In this case, the stationary solution takes a simple form:
P (w)∝ 1 C(w) exp [− 2S λ ∫ dw 1 C(w) ∂L ∂w ] . (31)
We will apply this equation to study the relevant problems in this work.

heading: C.2 A GENERAL CASE
text: In this section, we consider (a slightly more general version of) the fourth-order potential we studied in the main text. The average loss landscape in this model is
L(w) = 1 2 aw2 + 1 4 bw4, (32)
where b > 0, guarantees that the w is bounded regardless the value of a. In the limit of b → 0, this function reduces to saddle point problem we studied in Sec. 5.1. Besides the SGD noise, additive noise is also present in the dynamics. The variance of the additive noise has no dependence on w. When the additive noise is present, the update rule (equation of motion) is
dw(t) = −[λaw(t) + bw3(t)]dt + ληm(t) + ληa(t), (33) where both ηm(t) and ηa(t) are both Ornstein-Uhlenbeck process, denoting multiplicative noise and additive noise respectively. The w-dependent covariance of w is C(w) = w2. The SGD noise is thus a multiplicative noise. ηm(t) and has a variance w(t) 2 S dt, while the additive noise ηa(t) has a variance of σ2dt, where σ is a positive constant denoting the strength of the additive noise. If the multiplicative noise is seen as a part of the loss landscape, (2) coincides with the 2nd-order term in the loss landscape. There is no correlation between the additive noise and the SGD noise, i.e. E[ηm(t)ηa(t′)] = 0. The additive noise can be seen as the artificially injected noise, a technique sometimes used for aiding the escape or for Bayesian learning purposes (Jin et al., 2017; Welling and Teh, 2011).
The additive noise vanishes in the limit of σ → 0, and the model becomes a normal SGD with 4th-order loss function. It is always possible to define a noise η′(t), whose contribution is equivalent to the contribution of both the SGD and the additive noise, and the equation of motion becomes
dw(t) = −[λaw(t) + bw3(t)]dt + λη′(t). (34)
This transformed noise η′(t) thus has 0 mean and a variance of (w(t) 2
S + σ2)dt. Define η(t) =
η′(t)/ √ w(t)2
S + σ2, the equation of motion becomes
dw(t) = −λ[aw(t) + bw3(t)]dt + λ √
w(t)2 S + σ2η(t). (35)
Comparing with (23), one finds C(w) = w2 + Sσ2. (36)
The solution of the corresponding Fokker-Planck equation is
P (w)∝ 1 w2 + Sσ2 exp [− 2S λ ∫ aw + bw3 w2 + Sσ2 dw]
= (w2 + Sσ2)−1 exp ⎡⎢⎢⎢⎣ −2Sb λ ∫ (a b − Sσ2)w +w3 + Sσ2w w2 + Sσ2 dw ⎤⎥⎥⎥⎦
= (w2 + Sσ2)−1 exp [−2Sb λ ∫ wdw − 2Sb λ (a b − Sσ2)∫
w
w2 + Sσ2 dw]
= (w2 + Sσ2)−1 exp [−Sb λ w2 − (Sa λ − S
2σ2b
λ ) ln (w2 + Sσ2)].
Further simplification yields
P (w)∝ (w2 + Sσ2)−1−Saλ +S 2bσ2 λ exp [−Sb λ w2]. (37)
The function P (w) is finite everywhere and decays exponentially to 0 when w →∞, regardless of the values of the parameters a, b, σ, S. This indicates that ∫ ∞ −∞ dw(w2 + Sσ2)−1−Saλ +S 2bσ2 λ exp [−Sb λ w2] has a well-defined value. As a consequence, there is no concentration of measure. Thus, w(t) can be
fall into any interval with finite probability after big enough t, indicating that w(t) can be arbitrarily far away from 0. In practice, this means that w(t) escapes from the saddle point at w = 0 regardless of the value of the parameters.
The effect of additive noise is better understood when b = 0. In this case, which corresponds to the case shown in Fig.1, w neither concentrates at 0, nor escapes to ±∞. Instead, it stays near 0 without converging to it. The stationary distribution of w when b = 0, obtained by numerical simulation, is shown in Fig.8. The other settings are identical to the ones in Fig.1. The stationary distribution exists, but w does not concentrate at w = 0 in the stationary distribution. First of all, this figure shows that the additive noise helps SGD to escape saddle point. If the designed landscape is a part of a realistic landscape, having a broader distribution means having more chance of being attracted by another minimum. However, this figure also shows that w would stay in the neighborhood of w = 0 until the noise is big enough. Thus, having additive noise and long enough training time does not guarantee the efficiency of escape.
In the language of Bayesian inference, ln [P (w)] is the likelihood of w. ŵ, the most probable value of w, is defined by the relation
d ln [P (w)] dw ∣ w=ŵ = 0. (38)
which reads
0 = { d dw [(−1 − Sa λ + S 2bσ2 λ ) ln [(w2 + Sσ2)] − Sb λ w2]}∣
w=ŵ
= ⎧⎪⎪⎨⎪⎪⎩ 2w −1 − Sa λ + S2bσ2 λ w2 + Sσ2 − 2Sb λ w ⎫⎪⎪⎬⎪⎪⎭ RRRRRRRRRRRRw=ŵ
= {w [−1 − Sa λ + S 2bσ2 λ − Sb λ (w2 + Sσ2)]}∣ w=ŵ
= ŵ ( λ Sb + a b + ŵ2) .
The likelihood maximizer is
ŵ = ⎧⎪⎪⎨⎪⎪⎩ 0, a > − λ S ; ± √ − λ Sb − a b , a < − λ S .
(39)
When a > − λ S , the maximum likelihood parameter is w = 0. When a < − λ S , w = ± √ − λ Sb − a b
equally likely. This resembles the phase transition in Landau model and − λ
S is referred to as the critical a. For
the energy landscape, when a > 0 there is only one global minimum at w = 0, and when a < 0 there are two global minima at w = ± √ −a b
. Comparing with the maximum likelihood solutions, we see that the likelihood maximizer given by SGD is in fact a biased estimator of the underlying minima.
There is a bias term introduced by the SGD noise in both the critical a and the value of the most probable w. This term vanishes when S →∞, i.e. when the SGD noise vanishes. This indicates that, when the noise is state-dependent, there is no reason to expect SGD to be an unbiased or consistent estimator of the minimizer, as some works assume.
C.3 4-TH ORDER POTENTIAL WITH MULTIPLICATIVE NOISE
In this subsection, we study a 4-th order loss landscape in SGD dynamics without additive noise. The average loss function with 4th-order term is
L(w) = aw 2 2 + bw 4 4 ; (40)
the SGD update rule (i.e., the equation of motion) in this case is
dw(t) = −λ[aw(t) + bw3(t)]dt + λ 1√ S η(t)w(t), (41)
where the definition of η is identical to the previous example. The solution of corresponding stationary Fokker-Planck equation is
P (w)∝ w−2− 2Saλ exp [−Sb λ w2]. (42)
When
− Sa λ < 1, (43)
P (w) diverges at w = 0. The normalization factor, i.e. integral ∫ +∞ −∞ dww−2− 2Sa λ exp [−Sb λ w2], also diverges due to the divergence at w = 0. This solution could be seen as a limit of (37) when the additive noise vanishes. As the strength of the additive noise σ → 0, the normalization factor keep growing while the distribution remains normalized. For w ≠ 0, the function w−2− 2Saλ exp [−Sb
λ w2]
is always finite. Thus, in the limit σ → 0, P (w) approaches 0 everywhere except for at w = 0. It is straight forward to check that the normalization factor diverges slower that P (0) as σ → 0. As a consequence, when σ = 0, P (w) diverges at w = 0 and P (w) becomes a Dirac delta function. This corroborates with the result in Sec. 5.3 that a small a leads to a concentration of measure towards the local maximum.
Compared with (37), one finds that the additive noise prevents the concentration of measure. Thus, in practice, the additive noise helps SGD escape the local minimum or saddle point. However, the existence of additive noise does not change the critical a and the position of the peaks.

heading: C.4 QUADRATIC POTENTIAL
text: At last, we consider the continuous approximation of the model treated in the Sec. 5.1. Let the loss function be
L̂ = x 2 w2, (44)
and we have
{Ex[ĝt] = awt C(wt) = 1SE [∇`∇` T] − 1 S ∇; L(wt)∇L(wt)T = 1SE[x 2]w2t . (45)
The SGD update rule (i.e., the equation of motion) in 1d is
∆w(t) = −λ ∂ ∂w L̂ (46)
= −λxw(t). (47)
With the continuous approximation, the equation of motion becomes
dw(t) = −λaw(t)dt + λ 1√ S w(t)η(t), (48)
where η(t) ∼ N (0, √ dt). By comparing with the 1d Fokker-Planck equation,
{L(w) = a 2 w2;
B(w) = w. (49)
The stationary distribution of w is
P (w)∝ 1 w2 exp [−2S λ ∫ dw a w ]∝ w−2− 2Saλ . (50)
The only difference between the solution in this case and the one in the 4th-order-loss case is the exponential factor, showing that the 4th-order potential does nothing but keeping w(t) bounded. The function w−2− 2Sa λ diverges at w = 0 or w = ±∞ depending on the value of a. However, the stationary distribution, being a limit of (37), is well defined. In the case that it diverges at w = 0, P (w) becomes a delta function. On the contrary, there are two peaks infinitely far away from 0 when w−2− 2Sa λ diverges at w = ±∞. Being able to escape requires that the probability measure of w does not concentrate at 0, corresponding to
− Sa λ > 1. (51)
This straight line agrees with the boundary of the lower branch of the phase diagram in Fig. 9 when a → 0. This condition of escaping is worth interpretation. Note that a is the local curvature and reflects the strength of the gradient signal. In contrast, the strength of the SGD noise is proportional to λ/S.1 The term Sa/λ is thus the signal to noise ratio of this learning problem, and the escaping condition is exactly when the signal becomes larger than the noise. This analysis, therefore, pinpoints the cause of the convergence to saddle points to be the dominance of the SGD noise over the gradient.
Also, this example raises an interesting question regarding the mechanism of SGD that causes a convergence to the local maximum. From the perspective of the types of convergence, the SGD mechanism behind proposition 1 may be different from that of proposition 2. However, from the perspective of continuous-time analysis, there is really the same mechanism that is governing the SGD dynamics behind proposition 1 and 2 (namely, the fact that the noise has dominated the gradient).
1See Ziyin et al. (2021), for example.

heading: D DELAYED PROOFS

heading: D.1 TWO FREQUENTLY USED LEMMAS
text: We first prove the following lemma regarding the limiting distribution of ln ∣wt∣. Lemma 1. Let the loss function be L̂(w) = 1
2 xw2, x ∼ p(x) such that Var[x] = σ2 and Ex[x] = a <
0 and that p(x) is continuous in a δ-neighborhood of x = 1, and w0 ≠ 0. Then, for wt generated by SGD after t time steps,
1√ t (ln ∣wt/w0∣ − µ)→d N (0, s2) (52)
where µ = Ex[ln ∣1 − λx∣] and s2 = Var[ln ∣1 − λx∣], where λ > 0 is the learning rate.
Proof. After t steps of SGD, we have
wt = t
∏ i=1 [1 − λxi]w0. (53)
This leads to
ln ∣wt w0 ∣ = t ∑ i=1 ln ∣1 − λxi∣. (54)
Now, since p(1−x) is continuous in the neighborhood of 1 by assumption, we can apply Proposition 6 (appendix) to show that the first and second moment of ln ∣1 − λx∣ exists, whereby we can apply the central limit theorem to obtain
1√ t (ln ∣wt w0 ∣ − tE[ln ∣1 − λx∣])→d N (0, s2), (55)
where s2 = Var[ln ∣1 − λx∣]. This proves the lemma. ◻ Proposition 6. Ex∼p(x)[ln2 ∣x∣] is finite if the second moment of p(x) exists and if p(x) is continuous in the neighborhood of x = 0.
Proof. Since p(x) is continuous in the neighborhood of x = 0, then there exists δ > 0 such that for all ∣ ∣ < δ, ∣p(δ) − p(x)∣ ≤ c for some c > 0. This means that we can divide the integral over p(x) into three regions:
Ex∼p(x)[ln ∣x∣2] = ∫ ∞
−∞ dxp(x) ln2 ∣x∣ (56)
= ∫ −δ
−∞ dxp(x) ln2 ∣x∣ + ∫
δ
−δ dxp(x) ln2 ∣x∣ + ∫
∞
δ dxp(x) ln2 ∣x∣. (57)
The second term can be bounded as
∫ δ
−δ dxp(x) ln2 ∣x∣ ≤ [p(0) + c]∫
δ
−δ dx ln2 ∣x∣ (58)
= 2[p(0) + c]∫ δ
0 dx ln2 x (59)
= 2[p(0) + c]x(ln2 x − 2 lnx + 2)∣ δ
0
(60)
= 2[p(0) + c]δ(ln2 δ − 2 ln δ + 2) <∞. (61)
The first and the third term can also be bounded. Since ln ∣x∣ is a convex function in the regions x ≥ δ and x ≤ δ respectively. We can find linear functions ax+ b of x such that ax+ b ≥ lnx for x ≥ δ. This leads to
∫ ∞
δ dx p(x) ln2 ∣x∣ ≤ ∫
∞
δ dx p(x)(ax + b)2 (62)
≤ ∫ ∞
−∞ dx p(x)(ax + b)2 (63)
= ∫ ∞
−∞ dxp(x)(a2x2 + 2abx + b2) (64) = a2µ2 + 2abµ1 + b2, (65)
where we have used the notation E[x2] = µ2 and E[x1] = µ, which by assumption is finite. The case for x ≤ −δ is completely symmetric and can also be bounded by a2µ2 + 2abµ1 + b2. Therefore, we have shown that
Ex∼p(x)[ln ∣x∣2] ≤ 2(a2µ2 + 2abµ1 + b2) + 2(p(0) + c)δ(ln2 δ − 2 ln δ + 2) ≤∞. (66) This proves the proposition. ◻ Remark. Note that, the continuity in the neighborhood of 0 is not a necessary condition. For example, one can also prove that Ex[ln2 ∣x∣] is finite if p(x) is bounded and its second moment exists.

heading: D.2 PROOF OF PROPOSITION 2
text: Proof. The case when w0 = 0 is trivially true, we therefore consider the case w0 ≠ 0. By Lemma 1, we have
1√ t zt →d N (0, s2) (67)
where we have defined zt = 1√t(ln ∣ wt w0 ∣ − tµ), µ = E[ln ∣1 − λx∣] and s2 = Var[ln ∣1 − λx∣]. This means that
lim t→∞ p( 1√ t ln ∣wt w0 ∣) = lim t→∞ 1√ 2πs2 exp [− 1 2ts2 (ln ∣wt w0 ∣ − tµ)] . (68)
By definition, ∣wt∣ = ∣w0∣e √ tz+tµ. Therefore, we have
lim t→∞ P (∣wt∣ > ) = lim t→∞
P (∣w0∣e √ tz+tµ > ) = {0 if µ < 0;
1 if µ > 0, (69)
for all > 0. In other words, since µ and σ are t-independent, in the infinite t limit, the sign of µ becomes crucial. When µ > 0, the limiting distribution diverges to infinity; when µ < 0, ∣wt∣ converges to 0 in probability. This finishes the proof. ◻

heading: D.3 PROOF OF COROLLARY 1
text: Proof. By the definition of p(x), µ = E[ln ∣1 − λx∣] (70)
= 1 2 ln ∣1 − λ∣ + 1 2 ln ∣1 − λ(a − 1)∣ (71) = 1 2 ln ∣(1 − λ)(1 − λ(a − 1))∣, (72)
while s2 is indeed constant in time. Therefore, the asymptotic distribution of wt is solely dependent on the sign of µ. The above equation implies that µ ≥ 0 when
∣(1 − λ)[1 − λ(a − 1)]∣ > 1. (73) When λ ≤ 1, the above equation is solved by
λ < a a − 1 . (74)
When λ > 1, the above equation solves to
λ ≥ a − √ a2 − 8a + 8
2(a − 1) , (75)
only when the learning rate satisfies the above two conditions can it escape from the local maximum. Conversely, SGD cannot escape the local minimum when
a a − 1 ≤ λ ≤ a −
√ a2 − 8a + 8
2(a − 1) . (76)
We are done. ◻

heading: D.4 PROOF OF PROPOSITION 3
text: Proof. By Lemma 1, we have that 1
t E [ln ∣wt w0 ∣] = µ = 1 2 ln{(1 − λ)[1 − λ(a − 1)]}, (77)
where the second equality follows from the assumption that λ < 1. We differentiate with respect to λ to find the critical escape rate:
λ∗ = a 2(a − 1) . (78)
Since µ is convex in λ, it follows that this critical escape rate is the maximum escape rate. Plugging this into µ, we obtain that
γ∗ = µ(λ∗) = 1 2 ln (2 − a)2 4(1 − a) = ln 2 − a 2 √ 1 − a ≤ , (79)
i.e., the optimal escape rate can be made smaller than any if we set
∣a∣ ≤ 2 ∣−e √ e2 − 1 − e2 + 1∣ . (80)
This finishes the proof. ◻

heading: D.5 PROOF OF PROPOSITION 4
text: Proof. It suffices to show that w1 converges to 0 with probability 1 because, if this is the case, the
only possible local minimum to converge to is (w1,w2) = (0,± √ 1+a 2 ).
The SGD dynamics is
{∆w1,t = −λ(−2w1,t + 4w 3 1,t + 12w22,tw1,t + xtbw1,t);
∆w2,t = −λ(−2w2,t + 4w32,t + 12w21,tw2,t − 2aw2,t). (81)
We focus on the dynamics of w1. By the definition of the noise x, we have that
w1,t+1 = { w1,t[1 + λ(2 − b) − 4λw21,t − 12λw22,t] with probability 1/2; w1,t[1 + λ(2 + b) − 4λw21,t − 12λw22,t] with probability 1/2.
(82)
Equivalently,
∣w1,t+1 w1,t ∣ = {∣1 + λ(2 − b) − 4λw 2 1,t − 12λw22,t∣ with probability 1/2; ∣1 + λ(2 + b) − 4λw21,t − 12λw22,t∣ with probability 1/2. (83)
Since 0 ≤ 4λw21,t + 12λw22,t ≤ 16λ, we can define a new random variable rt:
rt ∶= r(xt) ∶= { max(∣1 + λ(2 − b)∣ , ∣1 + λ(2 − b) − 16λ∣) if xt ≥ 0; max(∣1 + λ(2 + b)∣ , ∣1 + λ(2 + b) − 16λ∣) if xt ≤ 0.
(84)
By construction, rt ≥ ∣w1,t+1/w1,t∣ for all values of xt. This implies that ∣w1,t/w1,0∣ ≤∏ti=1 rt, and so,
P (∣w1,t/w1,0∣ > ) ≤ P (∣ t
∏ i=1 rt∣ > ) , (85)
i.e., if∏ti=1 rt converges to 0 with probability 1, w1,t must also converge to zero with probability 1.
We let b = 1 λ − 6 (note that this is the value of b such that rt is minimized for both cases), and we obtain that
rt = { 8λ if xt ≥ 0; 2 max(∣1 − 2λ∣ , ∣1 − 10λ∣) if xt ≤ 0.
(86)
The rest of the proof follows from applying the central limit theorem to 1√ t ∑ti=1 ln rt as in Lemma 1. The result is that∏ti=1 ri converges to 0 with probabiliy 1 if
µ ∶= E[ln rt] = 1
2 ln[8λmax(∣1 − 2λ∣ , ∣1 − 10λ∣)] < 0. (87)
The above inequality solves to
λ ≤ 1 20
(1 + √
6) (88) which is of order O(1) as stated in the theorem statement. This proves the proposition. ◻

heading: D.6 PROOF OF PROPOSITION 5
text: Proof. First we note that, since ∣wt∣ ≤ 1 ∣ĝt∣ = ∣xtwt∣ ≤ 1 + a. (89)
By the definition of the AMSGrad algorithm, we have that
vt = β2vt−1 + (1 − β2)ĝ2t ; (90) v̂t = max(v̂t−1, vt). (91)
Since v0 = 0 and ∣ĝt∣ = ∣xtwt∣ ≤ 1 + a, we have that vt ≤ 1 + a (92)
for all t, and so v̂t ≤ max
t v̂t = max t vt ≤ 1 + a. (93)
Meanwhile, since v̂t is a monotonically increasing series and is upper bounded by 1 + a, it must converge to a constant 0 < c ≤ 1 + a. Now, as before, we want to upper bound the random variable
1√ t (ln ∣wt+1 wt ∣ − µ) = 1√ t
t
∑ i=1 (ln ∣1 − λ√ v̂t xt∣ − µ) , (94)
where µ = E[ln ∣wt+1 wt ∣]. Since v̂t converges to c, there must exists a positive integer N( ) such that for any > 0, v̂N ≤ c − . This allows us to divide the sum to two terms:
1√ t (ln ∣wt+1 wt ∣ − µ) = 1√ t
t
∑ i=1 (ln ∣1 − λ√ v̂t xt∣ − µ) (95)
= 1√ t
N ∑ i=1 (ln ∣1 − λ√ v̂t xt∣ − µ) + 1√ t t ∑ i=N+1 (ln ∣1 − λ√ c − kt xt∣ − µ) , (96)
where we introduced 0 ≤ kt ≤ . But the first term is of order O(1/ √ t) and converges to 0, and so for sufficiently large t, the first term is also smaller than arbitrary . This means that
1√ t (ln ∣wt+1 wt ∣ − µ) ≤ 1√ t
t
∑ i=1 (ln ∣1 − λ√ c − kt xt∣ − µ) + , (97)
We can now consider the random variable
∣1 − λ√ c − kt xt∣ = ⎧⎪⎪⎨⎪⎪⎩ ∣1 − λ√ v̂t ∣ with probability 1 2 ; 1 + λ√ v̂t (1 − a) with probability 1 2 .
(98)
We now define a new random variable
rt(xt) ∶= { m if xt = 1; 1 + λ√
c− (1 − a) if xt = −1 + a.
(99)
where we defined the constant m ∶= max(∣1 − λ/√c∣, ∣1 − λ/ √ c − ∣). One can easily check that
rt ≥ ∣1 − λ√c−ktxt∣. This means that
P (∣wt+1 w0 ∣ > α) ≤ P ( t ∏ i=0 ri > α +O(1/ √ t)) , (100)
i.e., if rt converges to 0 in probability, ∣wt+1w0 ∣ must also converge to 0 in probability. Proceeding as in the proof of Proposition 2. One finds that the condition for rt to converge in probability to 0 is
ln ∣m [1 + λ√ c − (1 − a)]∣ < 0, (101)
which is equivalent to
∣m [1 + λ√ c (1 − a)]∣ < 1. (102)
Since is arbitrary, we let → 0 and obtain
∣(1 − λ√ c )[1 + λ√ c (1 − a)]∣ ≤ 1. (103)
Denote λ/√c as λ′, this condition solves to
a a − 1 ≤ λ ′ ≤ a −
√ a2 − 8a + 8
2(a − 1) . (104)
Setting a such that the above condition is met, AMSGrad will converge to 0 in probability. For λ < 1, This completes the proof. ◻

Review summaries:
## Post-discussion reassessment  See relevant post below.  ## Summary of contributions  In this paper, the authors examine whether SGD with a large, constant step-size avoids local maximizers (or, more generally, undesirable saddle points of the underlying minimization problem). More precisely, they focus on the algorithm $$ w_{t+1} = w_t - \lambda \hat g_t $$ where $\lambda>0$ is the algorithm's step-size, and $\hat g_t$ is a (stochastic) gradient of the (stochastic) loss function $\hat L(w;x)$, with $x$ a random variable.  The paper's results can be summarized as follows:  1. If $\hat L(w;x) = (x/2) \cdot w^2$, the authors identify a range of values of $\lambda$ (which depends on the distribution of $x$) such that, in probability, $w_t$ converges to $0$ – which, under the specified distributional assumptions for $x$, is the global maximum of $L = \mathbb{E}[\hat L]$. This is made precise in Propositions 1 and 2, and Corollary 1.  2. They provide a quartic loss function under which SGD converges to the function's sharper minimizers (as measured by the trace of the Hessian at said points). [Proposition 4]  3. They provide a specific range of parameters under which the AMSGrad algorithm converges to the undesirable maximizer of item (1) above.  [In the supplement, the authors also provide an analysis of a gradient-like diffusion (Appendices B and C), which they discuss as a continuous-time model of (SGD). This part is not directly connected to the rest of the paper, so I am not including it in my evaluation below.]

This paper demonstrates on several fairly simple (e.g. 1-dimensional quadratic) objectives that stochastic gradient descent may easily have very poor behavior: it could converge to a maximum, or diverge even in convex settings if the learning rate is too high.  Specifically, it is shown that for any learning rate, there is is a distribution over quadratics whose expectation is $-rx^2$ for some $r\ge 0$ such that SGD will converge to the maximum at 0, and there is also a distribution whose expectation is $rx^2$ such that SGD still diverges even on this convex loss. Note that the distribution (and $r$) depend on the learning rate. Further, there are distributions such that SGD must escape a saddle point very slowly, and a distribution over 2-dimensional quartics such that SGD will converge to a "sharp" rather than a "flat" minimum. It is also shown that AMSgrad must converge to a local maximum on some non-convex distributions.  The results are proven by choosing a particular distribution over quadratic objectives that causes the logarithm SGD's iterate to be a sum of i.i.d. random variables. Then the central limit theorem provides an understanding of the limiting behavior of these iterates.  The results are augmented by empirical studies verifying the theorems directly, along with a very simple small neural network experiment.

Many theoretical works have studied SGD, but they commonly rely on restrictive and unrealistic assumptions about the noise. In this work, the authors construct example optimization problems illustrating that, if these assumptions are relaxed, SGD can exhibit many strange behaviors, including (1) SGD can converge to local maxima, (2) SGD may escape saddle points arbitrarily slowly, (3) SGD can prefer sharp minima over flat ones, and (4) AMSGrad can converge to local maxima. Therefore, the authors conclude that in the most general nonconvex case, many counter-intuitive phenomena of SGD may arise and contrast to the commonly held presumptions.

The paper provides several artificial examples on which SGD has an unintuitive behavior. This includes: (1) SGD converges to a local maximum; (2) If the learning rate is not fixed then SGD takes arbitrarily long time to escape saddle points; (3) SGD may prefer sharper minima (in contrast to several hypotheses regarding the implicit bias of GD); (4) Adaptive methods may also converge to a global maximum. Several experiments are made, varifying the theoretical results.