Eliminating Sharp Minima from SGD with Truncated Heavy-tailed Noise
Authors: Xingyu Wang
Sewoong Oh
Chang-Han Rhee
Venue: ICLR 2022 Poster (2022)
Keywords: Stochastic Gradient Descent
SGD
Heavy-Tails
Generalization

Abstract: The empirical success of deep learning is often attributed to SGD’s mysterious ability to avoid sharp local minima in the loss landscape, as sharp minima are  known to lead to poor generalization. Recently, empirical evidence of heavy-tailed gradient noise was reported in many deep learning tasks; and it was shown in (Simsekli et al., 2019a;b) that SGD can escape sharp local minima under the presence of such heavy-tailed gradient noise, providing a partial solution to the mystery. In this work, we analyze a popular variant of SGD where gradients are truncated above a fixed threshold. We show that it achieves a stronger notion of avoiding sharp minima: it can effectively eliminate sharp local minima entirely from its training trajectory. We characterize the dynamics of truncated SGD driven by heavy-tailed noises. First, we show that the truncation threshold and width of the attraction field dictate the order of the first exit time from the associated local minimum. Moreover, when the objective function satisfies appropriate structural conditions, we prove that as the learning rate decreases, the dynamics of the heavy-tailed truncated SGD closely resemble those of a continuous-time Markov chain that never visits any sharp minima. Real data experiments on deep learning confirm our theoretical prediction that heavy-tailed SGD with gradient clipping finds a flatter local minima and achieves better generalization.

Published as a conference paper at ICLR 2022
ELIMINATING SHARP MINIMA FROM TRUNCATED HEAVY-TAILED NOISE
SGD
WITH
Xingyu Wang' Sewoong Oh Chang-Han Rhee'
Northwestern University, University of Washington xi singyuwang20177 www.wwwwwwetteennnccmmmm edu
ABSTRACT
The empirical success of deep learning is often attributed SGD's mysterious ability to avoid sharp local minima in the loss landscape, as sharp minima are known to lead to poor generalization. Recently evidence of heavy-tailed gradient noise was reported in many deep learning tasks, and it was shown in (Simgekli) al. 2019a b) that SGD can escape sharp local minima under the presence of such heavy-tailed gradient noise, providing a partial solution to the mystery. In this work, we analyze a popular variant of SGD where gradients are truncated above a fixed threshold. We show that it achieves stronger notion of avoiding sharp minima: it can effectively eliminate sharp local minima entirely from its training trajectory. We characterize the dynamics of truncated SGD driven by heavy-tailed noises. First, we show that the truncation threshold and width the attraction field dictate the order of the first exit time from the associated local minimum. Moreover, when the objective function satisfies appropriate structural conditions, we prove that as the learning rate decreases the dynamics of the heavy-tailed truncated SGD closely resemble those of a continuous-tine Markov chain that never visits any sharp minima. Real data experiments on deep learning confirm our theoretical prediction that heavy-tailed SGD with gradient clipping finds a "flatter" local minima and achieves better generalization.
INTRODUCTION
Stochastic gradient descent (SGD) and its variants have seen unprecedented empirical successes in training deep neural networks. The training of deep neural networks is typically posed as non- convex optimization problem, and even without explicit regularization the solutions obtained by SGD often perform surprisingly well test data. Such unexpected generalization performance of SGD in deep neural networks are often attributed to SGD's ability to avoid sharp local minima [1) in the loss landscape, which tends lead poor generalization (Hochreiter & Schmidhuber) 1997 Keskar et al.) 2016, et al., 2018b) Jiang et al. 2019); see Appendix D for more details. Despite significant efforts to explain such phenomena theoretically, understanding how SGD manages avoid sharp local minima and end up with flat local minima within a realistic training time still remains as a central Recently, the heavy-tailed dynamies of SGD received significant attention,
mystery of deep learning. ? and it was suggested that the heavy tails in the stochastic gradients may be key ingredient that facilitates SGD's escape from sharp local minima: for example, Şimşekli et al. (2019a b) report the empirical evidence of heavy-tails in stochastic gradient noise in popular deep learning architectures (see also (Hodgkinson & Mahoney 2020, Srinivasan et al., 2021 Garg et al. 2021) and show that SGD escape sharp local minima in polynomial time under the presence of the heavy-taile gradient noise. More specifically, they view heavy-tailed SGDs as discrete approximations of Lévy driven Langevin equations and argue that the amount of time SGD trajectory spends in each local minimum is proportional to the width of the associated minimum according to the metastability theory (Pavlyukevich) 2007 Imkeller al. 2010a b) for such heavy-tailed processes.
We use the terminology sharpness in broad sense; we refer to Appendix for a detailed discussion.
TTo see detailed discussion on existing results on selection of local minima from the stability perspective and the novelty of our analysis, Appendix H

---

Published as a conference paper at ICLR 2022
(a]
Seary- lor raless propintt pphis
(a)
Xn
10a
[b)
Manry kg relee poggo
(8)
(c)
400
Jgtt teans, gusset
(c)
(d)
Xn
10r
b 0.28
0.5
oale teses, gradincassiog
clp
Cxip
(d)
10
caln"
(e)
Xn
Eydp
10
Cml
mm
1030000 2020000
3300000
0.01
0.001
Beraaion numberr
Leoming Rate
Figure I: (Left) Histograms of the locations visited by SGD. With truncated heavy-tailen noises, SGD hardly ever visits the two sharp minima m, and my. The objective function f is plotted at the bottom, and dashed lines are added as references for the locations of local minima. (Middle) Typical trajectories of SGD in different cases: (a) Heavy-tailed noises, no gradient clipping: (b) Heavy-tailed
noises, gradient clipping at b = 0.5; (c) Light-tailed noises, no gradient clipping; (d) Light-tailed noises, gradient clipping at 0.5. The objective function f is plotted at the right of each figure, and dashed lines are added as references for locations of the local minima. (Right) First Exit Time from S2g (-1.3, 0.2). Each dot represents the average of 20 samples of first exit time. Each dahsed line shows polynomial function c//11" where 3 is predicted by Theorem I and is chosen to fit the dots. The non-solid green dot indicates that for some the 20 samples of the termination threshold 5 x 10" was reached, and hence, it is an underestimation. Results in (Left) and (Middle) are obtained under learning rate = 0.001 and initial condition X₂ = 0.3.
In this paper, we study the global dynamics and long-run behavior of heavy-tailed SGD and its practical variant in depth. While in full generality the structure of gradient noises in SGD is state- dependent, in this work we focus on the role of noise magnitude and analyze the setting where each SGD update is perturbed by iid heavy-tailed noise. In particular, we consider an adaptive version of SGD, where the stochastic gradient is truncated above a fixed threshold. Such truncation scheme is often called gradient clipping and employed as default in various contexts (Eigssrrom et al. 2020 Merity et al. 2018 Graves) 2013, Pascanu et al. 2013 Zhang al. 2020 Gorbunov et al.) 2020). We uncover rich mathematical structure in the global dynamics of SGD under this scheme and prove that the asymptotic behavior of such SGD is fundamentally different from that of the pure form of SGD: in particular, under a suitable structural condition on the geometry of the loss landscape, gradient clipping completely eliminates sharp minima from the trajectory of SGDs. This provides a critical insight into how heavy-tailes dynamics of SGD can be utilized to find local minimum that generalizes better.
Figure (Left, Middle) clearly illustrates these points with the histograms of the sample trajectories of SGDs. Note first that SGDs with light-tailed gradient noise-(c) and (d) of Figure Middle)- never manages escape (sharp) minimum regardless of gradient clipping. In contrast, SGDs with heavy-tailed gradient noise- (a) and (b) of [ Middle) escapes from local minima. Moreover, there is clear difference between SGDs with gradient clipping and without gradient clipping. In (a) of Figure [ (Left), SGD without gradient clipping spends a significant amount of time at each of all four local minima (fm1, m2, m3, m1)). although it spends more time around the wide ones ((m2, m))) than the sharp ones ([m1, ma)). On the other hand, in (b) of Figure (Left), SGD with gradient clipping not only escapes from local minima but also avoids sharp minima (fmL, m) almost completely. This means that after we run SGD for long enough (more precisely, the required run length t t/1² is of polynomial order; see Theorem 2). it is almost guaranteed that it won't 't be at sharp minimum, effectively eliminating sharp minima from its training trajectories.
We also propose novel computational strategy that takes advantage of our newly discovered global dynamics of the heavy-tailed SGD. While the evidence of heavy tails were reported in many deep learning tasks (Simşekli et al. 2019b
Garg et al. 2021 Gurbuzbalaban al. 2020 Hodgkinson
2

---

Published as a conference paper at ICLR 2022
& Mahoney 2020;
Nguyen al. 2019: Mahoney & Martin 2019: Srinivasan et al. 2021 Zhang
et al. (2020). there seem to be plenty of deep learning contexts where the stochastic gradient noises light-tailed (Panigrahi et al. 2019) as well. [ Guided by our new theory. we propose an algorithm that injects heavy-tails to SGD by inflating the tail distribution of the gradient noise and facilitating the discovery of local minimum that generalizes better. Our experiments with image classification tasks, reported in Tables] [ illustrate that the tail-inflation strategy we propose here can indeed improve the generalization performance of the SGD predicted by our theory.
The rest of the paper is organized as follows. Section 2 formulates the problem setting and character- izes the global dynamics of the SGD driven by heavy-tailed noises. Section 3 presents numerical experiments that confirm our theory. Section proposses new algorithm that artificially injects heavy tailed gradient noise in actual deep learning tasks and demonstrate the improved performance.
We rigorously characterize the global behavior of the heavy-tailed SGD with gradient clipping. We first focus on the case where the loss function in RR with some simplifying assumptions its geometry. Even with such assumptions, our theorem involves substantial technical challenges since the traditional tools for analyzing SGD fail in our context due to the adaptive nature of its dynamics and non-Gaussiian distributional assumptions. For example, while the unclipped pure SGD can be analyzed by partitioning its trajectory at arrival times of large noises (as in Pavlyukevich 2005) and Imkeller al. (2010a)); ). such approach falls short in our context. Instead, we developed a set of delicate arguments for dealing with SGD's ' (near) regeneration structure and the return times to the local minima, as well as controlling the probability of atypical scenarios that would not arise in the unclipped case. Moreover, as evidenced by our Rd results in Appendix the approach developed here is critical in extending the analysis general loss landscapes.
Technical Contributions: 1)
2) We propose novel computational strategy for improving the generalization performance of SGD by carefully injecting heavy-tailes noise. We test the proposed algorithm with deep learning tasks and demonstrate its superiority with an ablation study. This also suggests that the key phenomenon we characterize in our theory- climination of sharp local minima-manite in real-world tasks.
2 THEORETICAL RESULTS
This section characterizes the global dynamics of SGD with gradient clipping when applied to non-convex objective function f. In Section 2.1 and 2.2 we make the following assumptions for the sake of the simplicity of analysis. However, as our multidimensional result in Section 2.3 the experiments in Section 3 and 4 suggest, the gist of the phenomena we a elimination of sharp local the stationary points are not necessarily strict local optima separated from one another.
peinnna--essstssss in general contexts where the domain of f is multi nulti-dimensional, and
Let f TR - TR be C2 function. There exist positive real I, > 0, positive integer Timin and an ordered sequence of real numbers m1, 81, m2, 82, .... - 1: Mmm.. such that (I) LL < mi
Assumption 1.
81 < m2 < 82 <... < Snan-1 Mmna < Li (2) ((( 0 iff e
{m1, $1, ... Smna 1, mmmm }: (3) For any x E {m1, M2, Mrnan } (((x) > 0: (4) For any {s1, "2. Sna fx( O.
As illustrated in Figure 2 (Left), the assumption above requires that has finitely many local minima (to be specific, the count IS mmin)) all of which contained in some compact domain [-L, L). Moreover, the points 81, ................................ *n... naturally partition the entire real line into different regions S2; = (8- 8i) (here we adopt the convention that 8p -00, 8ng. +c)). We call each region Ω2 the attraction field of the local minimum m,, as the gradient flow in 22, always points to m..
Throughout the optimization procedure, given any location E R assume that we have access to the noisy estimator f'(x)- Zn, of the true gradient f'(x), and f'(xx itself is difficult to evaluate. Specifically, in this work we interested in the case where the iid sequence of noises (Zm)x21 heavy-tailed. Typically, the taave- phenomena are captured by the concept of regular variation: for a measurable function Φ R4 IR,, we say that & is regularly varying at +0o with index (denoted as á E RVV.) if lim ((tx)/((r)) = 1 for all t >> 0. For details on the definition and properties of regularly varying functions, see, for example, chapter 2 Resnick (2007). In this paper,
For a detailed comparison existing works heavy-tailed phenomena is SGD, see Appendix
3

---

Published as a conference paper at ICLR 2022
work with the following distributional assumption on the gradient noise. Let
(z)x) - P( x), (x) e P(Z) < <2). H(x) - H+ (x) H- x).
Assumption 2. E. 0. Furthermore, there exists some E 00) such that function ((x) is regularly varying (at +00) with index -a. Besides, regarding the positive and negative tail for distribution of the noises, we have
H+(x) P+o lim H=(x) H(x)
lim 200
x000 H(x) =p- -P+
where p+ and p- are constants in interval (0,1).
Roughly speaking, Assumption 2 means that the shape of the tail for the distribution of noises Z, resembles a polynomial function x e", which is much heavier than the exponential tail of Gaussian distributions. Therefore, large values of Zn, are much more likely to be observed under Assumption compared to the typical Gaussian assumption. The index a of regular variation encodes the heaviness of the tail- the smaller the heavier- and are assuming that the left and right tails share the same index a. The purpose of this simplifying assumption is clarity of presentation, but our IR results in Appendix || relax such a condition and allow different regular variation indices in different directions.
Our work concerns a popular variant of SGD where the stochastic gradient is truncated. Specifically, when updating the SGD iterates with learning rate r7 > 0, rather than using the original noisy gradient descent step nf (fx( Z,.. we will truncate it at threshold > 0 and use you (n((f((,, (Xx,) - Zn)) instead. Here the truncation operator yp. (-(-) is defined as
(w) w min c//w// Vw E RR, C> 0.
Besides truncating the stochastic gradient, we also project the SGD into [-L, L] at each iteration; recall that is the constant in Assumption [ process (X]))20 driven by the following recursion
That is, the main object of our study is the stochastic
Xy - or. - - -x1))
(2)
The projection fr. and truncation 5hb here are common practices in many learning tasks for the purpose of ensuring that the SGD does not explode and drift to infinity. Besides, the projection also allows us to drop the sophisticated assumptions on the tail behaviors of f that are commonly seen in previous works (see, for instance, the dissipativity conditions in Nguyen al. (0119). For technical reasons, we make the following assumption about the truncation threshold b > O. Note that this assumption is very mild one, as it is obviously satisfied by (Lebesgue) almost every b >
Assumption 3. For each 1,2, Tomie min (1$: my/ |s:-1 mm13/ /b is not integer.
2.1 FIRST EXIT TIMES
Denote the SGD's first exit time from the attraction field 24 with (n) ف min(n 2 0: X P. In this section, we prove that o, (1) x) converges to an exponential distribution when scaled properly. To characterize such scaling, we first introduce few concepts. For each attraction field Ω, define (note that [x]] = min(n E Z n 2 x). [x] = max {n E Z S {x})
r
min [m, - s,-1l, [s, - m.))
[r/b].
(3)
Note that It' in fact depend on the the value of gradient clipping threshold even though this dependency is not highlighted by the notation. Here r; can be interpreted as the radius or the effective width of the attraction field, and 1 the minimum number of jumps required to escape Q2 when starting from m. Indeed, the gradient clipping threshold b dictates that single SGD step can travel more than b, and to exit 22 when starting from m. we can see that at least [r,/b] steps are required. We can interpret as the minimum effort required to exit pz. In this sense, is indicator of the width of the attraction field Ω, Theorem I states that 7 dictates the order of magnitude of the first exit time as well where the iterates X3 land at the first exit time. For each 2, define scaling (1/n)) (1s./))(((nnn))))
function (n) A (1/m)) ((1n) probability law when conditioning on X Jr, or simply write (x).
17-1 To stress the initial condition, we write P, for the
4

---

Published as a conference paper at ICLR 2022
h1
h2-1
12-1
m2
m;
m
m2
4,2-1 -1
12.
=
4,2
0.2:
0.6
+1
0.9 10.3
0.0.5
b-0.4
Figure 2: Typical transition graphs 9 under different gradient clipping thresholds b. (Left) The function illlustrated here has 3 attraction fields. For the second Q2 (81, 82), have $2 12 = 0.9, m2 $1 = 0.6. (Middle) The typical transition graph induced by b = 0.5. The entire graph G is irreducible since all nodes communicate with each other. (Right) The typical transition graph induced by b 0.4. When b = 0.4, since 0.6 < 2b and 0.9 > 26, the SGD can only exit Q2 from the left with only jumps started from m2- Therefore, on the graph 9 there are two communication classes: Gi = m)), G2 = {ma}; Gi is absorbing while G2 is transient.
Theorem 1. Under Assumptions 7 3 there exist constants qi > Vi E {1,2, [nminn and Fi 2 2Vjj 21,2, , [Rmin) } { such that
(i) Suppose that E S4 for some k E (1,2, Dmmn)) Under Pæ, the scaled first exit time qu (n) converges in distribution to (1) as
(ii) For k. {1,2, .... Pmiiie such that l + I, we have lim Px(X (x())) E 24) = 40a/qk-
The proof and discussion deferred to Appendix G The constants gi Iúj explicitly identified in terms of the gradient flows perturbed by Pareto jumps in Section C of Appendix. We note here that Theorem I limplies (i) for X to escape the current attraction field, say 9t it takes (11)x(11) time, and (ii) the destination is most likely to be reachable within 1 jumps from m,.
2.2 ELIMINATION OF SMALL ATTRACTION FIELDS
Under proper structural assumptions on the geometry of f, the sharp minima of f can be effectively eliminated from the trajectory of heavy-tailed SGD, facilitating the discovery of flat minima. This is somewhat surprising given that gradient clipping mechanism makes the SGD iterates move slower. The intuition behind this is that for narrow basins, applying gradient clipping has virtually no effect on the order of exit time; whereas for a wide basin that requires multiple jumps to escape under the clipping scheme, the clipping gradients significantly slows down the escape and makes SGD stay longer. In other words, gradient clipping only makes SGDs stay longer in the wider (better) basins.
Now, we introduce few new concepts. Similar to the the minimum number jumps 1 defined in we define the following as the minimum number of jumps to reach S2 from m, for any j + i:
3
( ([(m; s,)bb]
if i f i j
b
Recall that Theorem ! dictates that X is most likely to out of the current attraction field, say 2r to somewhere else after (1)) time steps, and the destination most likely to be reachable within jumps from m, Therefore, the transitions from Ω to 22 can be considered typical if S2 can be reached from mn with & jumps-that is, 4,5 = 1. Now we define the following directed graph that only includes these typical transitions.
Definition 1 (Typical Transition Graph). Given a function f satisfying Assumption Nand gradient clipping threshold b > 0 satisfying Assumption
3 directed graph G = (V, E) is the corresponding typical transition graph (1) V (m,,... M.n..: }: (2) An edge (m; - m2) is in E iffll.. 1ū
Naturally, the typical transition graph can be decomposed into different communication classes Gi.... Gk that are mutually exclusive by considering the equivalence relation associated with the existence of the (two-way) paths between and j. More specifically, for i * j. we say that
5

---

Published as a conference paper at ICLR 2022
i and j communicate if and only if there exists path (m, m Mk, my) well as a path (my, mxx"" :.. . m.) in G; in other words, by travelling through edges on G, m, can be reached from my and my can be reached from m4-
We say that communication class G is absorbing if there does not exist any edge (m, - m2) E E such that m, & G and mi & G. Otherwise, we say that G is transient. In the case that all m, 's communicate with each other on graph G. we say G is irreducible. See Figure (Middle) for the illustration of an irreducible case. When G is irreducible, we define the set of largest attraction fields M e (m, : 1,2
mmmin 1
(larg] where (luge max, 17; recall that 1 characterizes
the width of S2, Define the longest time scale Alare (1) (( (10)((H)))/))))))) (n)//)))))))))))))))))))))))//) " Note that this corresponds exactly to the order of the first exit time of the largest attraction fields; see Theorem [
The following theorem is the main result of this paper.
Theorem 2. Let Assumptions
7
hold and assume that the graph G is irreducible. For any t. > 0,
B>1 + (ax farge and x 1,, T.
1e/n
|t/mp'
1
Ja
X (x),()) (x)
U jimm &
23 +0
(5)
in probability as r + O.
The proof is deferred Appendix H Here we briefly discuss the implication of the result. Suppose that we terminate the training after a reasonably long time, say. [4/1]] iterations. Then the random variable that converges zero in eq. (5) is exactly the proportion of time that X spent in the attraction fields that are not wide. Therefore, by truncating the gradient noise of the heavy-tailed SGD, we can effectively eliminate small attraction fields from its training trajectory. In other words, it is almost guaranteed that SGD is in one of the widest attraction fields after sufficiently long training. Meanwhile, despite the asymptotic nature of Theorem 2 it has been confirmed in our simulation and deep learning experiments that the elimination effect can be observed under typical choices of v7.
Theorem is merely a manifestation the global dynamics of heavy-tailed SGD. The main messages of the next theorem are: (a) under clipped heavy-tailed noises, the dynamics of X for small r closely resemble that of a continuous toeeuuuuu-tmeeeuuee Markov chain; (b) this chain only visits local minima of the largest attraction fields of f, thus minima in small attraction fields are completely avoided.
Theorem 3. Let x E Qr for some i = 2, .... Temin- If Assumptions
3 hold and G is irreducible, then there exist continuous Markov chain Y on M large and random mapping æ satisfying
x (m) m m E Mlarge
* π (m) is random variable that only takes value in Mlarg if m & Miarg,
such that the scaled process {x"
(t/ Nlow (n)) (z) 2 0} converges to process (Y((T(m,)) :t2 in the sense of mnite-dimensiondl distributions: for any positive integer is and any 0 < f < <
< th. the random vector (X"L,, (20ear))) (x), (x), , , (t. (x) converges in distribution to
(Ye, (T(M))), ..... Yi. (T(,))) as 1) O.
In section D of Appendix, we detail the proof, the exact parametrization of the generator matrix of process Y, and the distribution of random mapping (-)). Here we add some remarks. Intuitively speaking, this result tells us that, regardless of where we initialize the SGD iterates, the dynamics of the clipped heavy-tailed SGD converge to a continuous-time Markov chain avoiding any local minima that not in the largest attraction fields. Second, under small learning rate vj > 0, if X²((x) is initialized at J. E Ω2 where Ω2 is NOT largest attraction field, then SGD will quickly escape f2, and arrive at some Ω2 that is indeed a largest one-i.c., my E M large; such transition is so quick that, under time scaling \1)3((n)) it is almost instantaneous as xxx() is actually initialized randomly at some of the largest attraction fields. This randomness is compressed in the random mapping n. In Section B we discuss how our characterization in Theorem 3 are more general and applicable in the machine learning context than metastability results cited in (Simşekli et al.. 2019b); in Section HH we see that the regularization effect of truncated heavy- noises described in Theorem 3 is of great generality and can still be observed locally when the irreducibility condition is removed.

---

Published as a conference paper at ICLR 2022
2.3 R EXTENSIONS OF THE THEORETICAL RESULTS
We focused on the I' case so far, for the clarity of the exposition. In this section, we informally reiterate that the same effect under truncated heavy-tailed noises persists in oigh-dimensions.. Rigorous statements are provided in Appendix ! We consider a setting similar to those in [mnkellrr et al.
(2010a) and analyze the first exit time o (1,xx) = minsk 2 0 : Xx (x: & G) from an open, bounded domain G with smooth boundary. For some f E + R. the SGD iterates (z) xx_ po (V f (x)) + nZz)) are subject to the standard Lg norm clipping with threshold b > O, iid noises Zn with heavy tails that resemble 1 1/20 with a > and initial condition X (a:
Assume that the origin O is the only attractor in G for the ODE (tt) = (z)((t)). Also, let 1/ be the minimum number of jumps required for the ODE axt) to escape from provided that x(0) 0 and the L2 norm of all jumps are less than b. The following informal version of Theorem 1.2 states that, under the presence of heavy-tailed noises and gradient clipping, the first exit time from a region in Rd is of order 0 (1//)+((-11)))))) Therefore, the order of first exit times from different regions in Ted are still dictated by the geometric characterization 1ğ. i.e. the minimum number of jumps required for escape, and those with largest 13. may dominate the SGD trajectory as 17 + 0. Furthermore. we extend the analysis to the generalized case where the distribution of noises Zn exhibits strong preference of certain directions and has different heavy-tailed indices a along different directions. We give the details of the R results in Section | and provide a proof in Section
Theorem (Informal), Under certain regularity conditions, for Lebesgue almost every b > 0, there exist q > 0 and X(7) that is regularly varying w.r.t 1) with index 1 + (a: 1)13 such that
X(1)o(1.2) = Exp(1) as 77 4 Vo Eg
3 SIMULATION EXPERIMENTS
We empirically demonstrate that, (a) as indicated by Theorem T the minimum jump number defined in (3) accurately characterizes the first exit times of the SGDs with clipped heavy-tailed gradient noises; (b) sharp minima can be effectively eliminated from such SGD; and (c) these properties are exclusive to heavy-tails. Under light-tailed noises, SGDs are trapped in sharp minima for long time. The test function f E (²(RR) is the same as in Fig. (Left,e). mi and my are sharp minima in narrow attraction fields, while m2 and my are flatter and located in larger attraction fields. Heavy-tailed noises have tail index a = 1.2, and light-tailed noises are NO, 1). See Appendix A for details.
First, we compare the first exit time of heavy-tailed SGD (when initialized at -0.7) from Q2 (-1.3, 0.2) under different clipping mechanism: (1) 0.28, where the minimum jump number required to escape is [" 3; (2) b 0.5, where [- = 2: (3) gradient clipping, where 1" 1 obviously. According to Theorem T the first exit times for the afforementioned 3 clipping mechanism of order (1/ (1/m)L.6 (1/n) as demonstrated in Figure [ (Rihht) Next, we investigate the global dynamics of heavy-tailed SGD, We compared the clipped case (with 0.5) against the case without clipping. Figure [ (Left, a, b) show the histograms of the empirical distributions of SGD, and Figure Middde, a.b) plots the SGD trajectories. Without gradient clipping, X,, still visits the two sharp minima ml, my. Under gradient clipping, the time spent at m1, my is almost completely eliminated and is negligible compared to the time X,, spent at m2, my in larger attraction fields. This matches the predictions of Theorems 3 the elimination of sharp minima with truncated heavy-tailed noises. We stress that the said properties are exclusive to heave- SGD. As shown in Figure Le),, and c,d), light-tailed SGD are easily trapped at sharp minima for extremely long time.
and (1/r) respectively. These theoretical predictions are accurate
Figure 3 illustrates the same phenomena in R², where f has several saddle points and infinitely many local minima-the local minima of 92 form a line segment, which is uncountably infinite set. Under clipping threshold b, attraction fields 22 and 22 are the larger ones since the escape from them requires at least two jumps. This suggests that the theoretical results from Section hold under more general contexts than Assumptions 3 In the next section, we provide experimental evidence that suggests that truncated heavy-tailed noise improves the generalization of SGD in deep learning.
7

---

Published as a conference paper at ICLR 2022
6 100
Q2
gracetp dp
gradient clipping
D4
no gradient dipppngg
04
Out
trahning Reration
Figure 3: Experiment result of htave- SGD when optimizing the modified Himmelblau function. (a) Contour plot of the test function f. (b) Different shades of gray are used to indicate the area of the four different attraction fields S21, 52g, 22, S2 of f. We say that a point belongs to an attraction field Ω if, when initializing at this point, the gradient descent iterates converge to the local minima in S2 which are indicated by the colored dots. The circles are added to imply whether the SGD iterates can escape from each S4 with one large jump not under clipping threshold b. (c) The time heavy-tailed SGD spent at different region. An iterate Xg is considered "visiting" Ω2 if its distance to the local minimizer of Ω is less than 0.5; otherwise we label X₂ "out". (d) The transition trajectories of heavy-tailed SGD. The dots represent the last "visited" attraction field at each iteration.
4 HEAVY-TAILED SGD IN DEEP LEARNING: AN ABLATION STUDY
In this section, we verify our theoretical results and demonstrate the effectiveness of clipped heavy- tailed noise in training deep neural networks. Contrary to the report (Şimşekli et al. 2019a), heavy- tailed noise may not be ubiquitous in image classification tasks. For instance, the non- nou-Sauss assumption on SGD noise is disputed by experiments in (Panigrahi et al.) 2019) for ResNet (see (He et al. 2016)). For tasks considered in this section, the gradient noise is not heavy-tailed when models are randomly initialized (see Appendix A Motivated by the absence of heavy-tailed noises in image classification, we make the SGI) noise heavy-tailed. Let 0 be the current model weight during training, (+) be the typical small-batch gradient, and 9GD (0) be the true (deterministic) gradient evaluated on the entire training dataset. Then by evaluating (SB(O) gGD we obtain a sample of the gradient noise. Due to the prohibitive cost of evaluating ( we instead use gsB (0) - gLB (8) ) as its approximation where JLB denotes the gradient evaluated on larger batch. This is justified by the unbiasedness in ELB [9)) = DGD((), For some heavy-tailes random variable Z, by multiplying Z with SGD noise, we obtain the following perturbed gradient:
Theany (0) = (0) + Z (gsB- (L)(
(6)
where SB and SBx are two mini batches that may or may not be identical. We use the following update recursion under gradient clipping threshold b: X = X 4pb (19hheayy (199heey(XX)) (X²) where pb is the truncation operator. We consider two different implementations: in our method / (labeled as "our I" in Table D SB and SB* are chosen independently, while in our method (labeled "our 2" in Table D). we use the same batch for SB and SB. In summary, by simply multiplying gradient noise with heavy-tailed random variables, we inject heavy-tailed noise into the optimization procedure.
We conduct an ablation study and benchmark the proposed clipped heavy-tailed methods against the following optimization methods. LB: large-batch SGD with X X ngLB (xX)): SB: amall-batch SGD with X X 1998 (XE); SB + Clip: the update recursion is X X - pa ( (X)); SB + Noise: Our method WITHOUT the gradient clipping mechanism, X 79heavy (X²).
leading to the update recursion X k
The experiment setting and choice of hyperparameters are adapted from (Zhu et al. 2019) We consider three different tasks: (1) LeNet (LeCun et al., 1990) corrupted FashionMNIST (iao et al. 2017). (. (2) VGG11 (Simonyan Zisserman 2014) on SVHN (Netzer et al.) 2011). (3) VGGIT CIFAR10 (Krizhevsky et a 2009) (see Appendix A for details). Here we highlight few points: First, within the same task, for all the 6 candidate methods will use the same 17, batch size, training iteration, and (when needed) the same clipping threshold and heavy-tailed RV Z for fair
8

---

Published as a conference paper at ICLR 2022
Table 1: 'est accuracy and expected sharpness of different methods across different tasks. The reported numbers are the averages over replications. For 95% CI, see Appendix A
Test accuracy
LB
SB
SB Clip
SB + Noise
Our 1
Our 2
FashionMNIST, LeNet
68.66%
69.20%
68.77%
64.43%
69.47%
70.06 %
SVHN, VGG11
82.87%
85.92%
85.95%
38.85%
88.42 %
88.37%
CIFARIO, VGGII
69.39%
74.42%
74.38%
40.50%
75.69%
75.87 %
Expected Sharpness
LB
SB
SB Clip
SB + Noise
Our 1
Our 2
MNIS,, LeNet
0.032
0.008
0.009
0.047
0.003
0.002
SVHN, VGGI1
0,694
0.037
0.041
0.012
0.002
0,005
CIFARIO, VGGII
2.043
0.050
0.039
2.046
0.024
0.037
Table 2: Our method's gain test accuracy persists even when applied with techniques such as data augmentation and scheduled learning rates. For 95% CI. see Appendix A
CIFARIO-VGGII
Rep 1
Rep 2
Rep
Rep 4
Rep 5
Average
SB+Clip
89.40%
89.41%
89.89%
89.52%
89.47%
89.54%
Our
90.76%
90.57%
90.49%
90.85%
90.79%
90.67% %
Our 2
90.67%
90.23%
90.52%
90.13%
90.70%
90.45%
CIFARI0O-VGGI6
Rep 1
Rep 2
Rep
Rep 4
Rep 5
Average
SB+Clip
55.76%
56.8%
56.38%
56.35%
56.32%
56.32%
Our
67.43%
65.12%
14%1
65.96%
63.57%
65.44%
Our
67.19%
61.17%
60.97%
64.75%
60.90%
62.99%
the training duration is long enough that LB and SB have attained 100% training accuracy and close-to-O training loss long before the end of training (the exception here is "SB + Noise" method; see Appendix A for the details); Second, to facilitate convergence local minima for our methods / and 2, we remove heavy-tailed noise for last final 5,000 iterations and run LB instead" [I
comparison;
Table shhows that in all 3 tasks both method I and our method 2 attain better test accuracy than the other candidate methods. Meanwhile, both methods exhibit similar test performance, implying that the implementation of the heavy-tailed method may not be the deciding factor. We also report the expected sharpness metric Eu-~N(o [L + +) - L(l")) ) used in Zhu et al. (2019); Neyshabur et al. 201 where (0, 1" is Gaussian distribution, 0" is the trained model weight and 7. is training loss. our experiment, we use 0.01 and the expectation is evaluated by averaging over 100 samples. We conduct 5 replications for each experiment scenario and report the averaged performance in Table [ Smaller sharpness of our methods and confirms that they encourage minimizers with a "flatter" geometry, thus attaining better test performances.
The ablation study in Table shows that both heavy-tailed noise and gradient clipping are necessary to find flat minima and hence achieve better generalization, which is predicted by our analyses. SB and SB + Clip achieve similar inferior performances, confirming that clipping does not help when noise is light-tailed. SB + Noise injects heavy-tailed noise without gradient clipping, which achieves an inferior performance. This poor serformance-eved after extensive parameter tuning and engineering (see Appendix Aopendix(A]forr for more (eeails)-dmmoostriiii front, especially when heavy-tailed noise is present yet little effort is put into controlling the highly volatile gradient noises. This is aligned with the observations in Zhang et al. (2020): Gorbunov et al. (2020) where adaptive gradient clipping methods are proposed to improve convergence of SGD in the presence of heavy- noises. This confirms that gradient clipping crucial for heavy-tailed SGD,
the difficulty on the optimization
Lastly, Table 2 shows that even in the more sophisticated settings with training techniques such as data augmentations and scheduled learning rates, truncated heavy-tailed SGD still manages to consistently find solutions with better test performance. For experiment details, see Appendix A In Table A.5 we also report the sharpness of the obtained solutions.
'The proposed method can be interpreted as a simplified version of GD + annealed heavy-tailed perturbation, where a detailed annealing is substituted by two-phase training schedule. In the first exploration phase the clipped heavy-tailed noises drive the iterates to explore the loss landscape and identify "wide" attraction fields. In the second exploitation phase, removing the artificial perturbation accelerates convergence to local minima.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENT
This work is partially supported by NSF awards DMS-2134012 and CCF-2019844 as part NSF Institute for Foundations of Machine Learning (IFML).
REFERENCES
Jeff Alstout, Ed Bullmore, and Dietmar Plenz. powerlaw: a python package for analysis of heavy-tailed distributions. PloS one, 9()):885777, 2014.
Patrick Billingsley. Convergence of probability measures. John Wiley Sons, 2013.
Aaron Clauset, Cosma Rohilla Shalizi, and Mark EJ Newman. Power-law distributions in empirical data. SIAM review, 51 (4):661-773, 2009.
Jeremy M Cohen, Simran Kaur, Yuanzhi Li, J Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs edge stability, arXiv preprint arXiv:2103 100000,, 2021.
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In International Conference on Machine Learning, pp. 1019-1028. PMLR, 2017.
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Alcksander Madry. Implementation matters in deep rl: A case study on ppo and trpo. In International Conference Learning Representations, 2020. URL http:ttsss//peereeee//
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXiv:2010.01412, 2020.
Saurabh Garg, Joshua Zhanson, Emilio Parisotto, Adarsh Prasad, Zico Kolter, Zachary Chase Lipton, Sivaraman Balakrishnan, Ruslan Salakhutdinov, and Pradeep Kumar Ravikumar. On proximal policy optimization's heavy-tailed gradients, 2021. URL httptst://ppeeeeettatee
met/forune www.:r/mmmddccee...o.. d
Eduard Gorbunov, Marina Danilova, and Alexander Gasnikov. Stochastic optimization with heavy-
tailed noise via accelerated gradient clipping. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 15042-15053, Curran Associates, Inc., 2020. URL htttps:: htt.://ewwd.inn............ep.pe nwwrips. cc/
Alex Graves. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0855 2013.
Mert Gurbuzbalaban, Umut Simsekli, and Lingjiong Zhu. The heavy-tail phenomenon in sgd. arXiv preprint arXiv:2006.044 240000000000040 2020.
Mert Gurbuzbalaban, Umut Simsekli, and Lingjiong Zhu. The heavy-tail phenomenon in sgd. In International Conference on Machine Learning, pp. 3964-3975, PMLR, 2021.
Haowei He, Gao Huang, and Yang Yuan. Asymmetric valleys: Beyond sharp and flat local minima.
arXiv preprint wxXiv:192.00074. 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and patlern recognition, pp. 770-778, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural computation, 9(1):1-42, 1997.
Liam Hodgkinson and Michael W Mahoney. Multiplicative noise and heavy tails in stochastic optimization. arXiv preprint arXiv:2006.0 06293, 2020.
Peter Imkeller, Ilya Pavlyukevich, and Michael Stauch. First exit times of non-linear dynamical systems d perturbed by multifractal Lévy noise. Journal of Statistical Physics, 141(1):94-11)
2010a.
10

---

Published as a conference paper at ICLR 2022
Peter Imkeller, llya Pavlyukevich, and Torsten Wetzel. The hierarchy of exit times of Lévy-driven Langevin equations. The European Physical Journal Special Topics, 191(1):211-222, 2010b.
Fabian Immler and Christoph Traut. The flow odes: Formalization of variational equation and poincaré map. Journal of Automated Reasoning, 62(2):215-255 22611,6--6666-- 2019
Stanislaw Jastrzebski, Maciej Szymezak. Stanislav Fort, Devansh Arpit, Jacek Tabor, Kyunghyun Cho, and Krzysztof Geras. The break-even point on optimization trajectories of deep neural networks. arXiv preprint arXiv:2002.09: 09572, 2020.
Yiding Jiang. Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic generalization measures and where to find them. arXiv preprint arXiv:1912.02278, 2019.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint 1609v 2016.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and In Kwon Choi. Asam: Adaptive sharpness- aware minimization for scale-invariant learning of deep neural networks. arXiv preprint arXiv:2102.11600, 2021.
Yann LeCun, Bernhard Boser, John Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. Handwritten digit recognition with back- pac-ppropagttion network. In Advances in neural information processing systems, pp. 396-404, 1990.
Dawei Li, Tian Ding, and Ruoyu Sun. Over-parameterizee deep neural networks have no strict local minima for any continuous activations. arXiv preprint 1rXiv:1812.11039 2018a.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 6391-6401, 2018b.
Filip Lindskog, Sidney I Resnick. Joyjit Roy, et al. Regularly varying measures metric spaces: Hidden regular variation and hidden jumps. Probability Surveys, 11:270-314, 2014.
Michael Mahoney and Charles Martin. Traditional and heavy tailed self regularization in neural network models. In International Conference Machine Learning, pp. 4284 4293..29.... PMLR, 2019.
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. Regularizing and optimizing LSTM language models. International Conference on Learning Representations, 2018. URL https
//openreview.co n aoppenrreveeev.ee..oooorrraaoee
Takashi Mori, Liu Ziyin, Kangqiao Liu, and Masahito Ueda. Logarithmic landscape and power-law escape rate of sgd. arXiv preprint arXiv:2105.09557, 2021.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nathan Srebro. Exploring general- ization deep learning. arXiv preprint arXiv:I70e 08947, 2017a
Behnam Neyshabur, Srinadh Bhojanapalli, David Mcallester, and Nati Srebro. Exploring gen- cralization in deep learning. In I. Guyon, U. V. Luxburg, S. Bengio. H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017b. URL UR._httpaa://prcceeiiiin neuriss.ee....... cc/
Than Huy Nguyen, Umut Simsekli, and Gaöl Richard. Non-asymptotic analysis of fractional langevin monte carlo for non-convex optimization. In International Conference on Machine Learning, pp- 4810-4819. PMLR, 2019.
Abhishek Panigrahi, Raghav Somani, Navin Goyal, and Pranceth Netrapalli. Non- Non-gaussianity of stochastic gradient noise. arXiv preprint arXiv: 1910.99666 2019.

---

Published as a conference paper at ICLR 2022
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning, Pp. 1310-1318. PMLR, 2013.
Hya Pavlyukevich. Metastable behaviour of small noise lévy-driven diffusion. arXiv preprini math/0601771, 2005.
Ilya Pavlyukevich. Cooling down lévy flights. Journal of Physics A: Mathematical and Theoretical, 40(41):12299,20000
Philip E Protter. Stochastic integration and differential equations. Springer, 2005.
Sidney I Resnick. Heavy-tail phenomena: probabilistic and statistical modeling. Springer Science & Business Media, 2007.
Chang- Manng-Hanha Rhee, Jose Blanchet, Bert Zwart, et al. Sample path large deviations for lévy processes and random walks with regularly varying increments. The Annals of Probability, 47(6):3551-3605 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 2014.
Umut Şimşekli, Mert Gürbüzbalaban, Thanh Huy Nguyen, Gaöl Richard, and Levent Sagun. On the heavy-tailed theory of stochastic gradient descent for deep neural networks. arXiv preprint arXiv: 8, 2019a.
Umut Şimşekli, Levent Sagun, and Mert Gurbuzbalaban. A tail-index analysis stochastic gradient noise in deep neural networks. In International Conference Machine Learning, pp. 5827-5837. PMLR, 2019b.
Vishwak Srinivasan, Adarsh Prasad, Sivaraman Balakrishnan, and Pradeep Kumar Ravikumar. Efficient estimators for heavy-tailed machine learning, 2021. URL hattps:/
pett/oorumid-335
Stephan Wojtowytsch. Stochastic gradient descent with noise of machine learning type. part ii:
Continuous time analysis. arXiv preprint arXiv:2106.02588, 2021.
Lei Wu, Chao Ma, et al. How sgd selects the global minima in over- per-paraaeteriized learning: A dynamical stability perspective. Advances in Neural Information Processing Systems, 31:8279- 8288, 2018
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: mnist: a novel image dataset for benchmarking machine learning algorithms, 2017.
Zeke Xie, Issei Sato, and Masashi Sugiyama. A diffusion theory for deep learning dynamics: Stochastic gradient descent exponentially favors flat minima. arXiv preprint arXiv:2002.03495 2020.
Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaie. Why gradient clipping accelerates training: A theoretical justification for adaptivity. In International Conference on Learning Representations, 2020. /tL/ittttss///peerrrre
net./ http://orummndddggno////o
Pan Zhou, Jiashi Feng, Chao Ma. Caiming Xiong, Steven Hoi, et al. Towards theoretically under- standing why sgd generalizes better than adam in deep learning, arXiv preprint arXiv:2010.05627, 2020.
Zhanxing Zhu, Jingfeng Wu, Bing Yu, Lei Wu, and Jinwen Ma. The anisotropic noise stochastic gradient descent: Its behavior of escaping from sharp minima and regularization effects. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.). Proceedings of the 36th International Conference on Machine Learning. volume 97 of Proceedings of Machine Learning Research. pp. 7654-7663, PMLR, 09-15 Jun 2019. URL http:pp//ppo..eeass..oo mwt.smmss.ppp..111.....
zhu htmll
12

---

Published as a conference paper at ICLR 2022
A DETAILS OF NUMERICAL EXPERIMENTS
A.1 DETAILS OF THE R SIMULATION EXPERIMENT
The function used in the experiments is
((x) = (x + (z + 1x3) - 0.2)) - -- -..66. ==)) 00
(1+ 0.01 4 0.5)2 [1
0.1 1(x + 1.5)2 (1
1
1
exp(-5(x 0z8) +088))
(A.1)
-L m;51
m2
$2
Ma
$3
m4
Q1
Q2
Q3
Q4
Figure A.I: Illustration of the test function f used in the R² experiment.
As shown in Figure A. the four isolated local minimizers of f are m, = -151,, 81 = -1.3, m2 = 1.32, and in experiment restrict the iterates
-0.66, 82 0.2, m3 0.49, 83 0.7, on [-I,, T.] with T, = 1.6. The heavy-tailed noises we used in the experiment were Zn = D. W,, where Wn, were sampled from Pareto Type II distribution (aka Lomax distribution) with shape parameter a = 1.2, and the signs Un were iid RVs such that PUU, = 1) = IP, -1) = 1/2.
m4 m
In the first exit time experiment, we tested three different settings: (a) b 0.28 (so that 1* 3); (b) b 0.5 (so that 1* = 2); (c) no gradient clipping (so that 15 = 1). For the first case, we tested learning rates {0.1, 0.05, 0.03, 0.02, 0.01, 0.005, 0.003, 0.001}. while for the other two cases, we tested learning rates 10.1, 0.03, 0.01, 0.003, 0.001, 0.0003, 0.0001 }. For each case, we ran the simulation 20 times and plotted the average of the 20 exit times Lastly, to prevent excessively long running time of the experiment, the simulation was terminated when the iteration number reached x 10 This threshold was reached only in the setting with = 0.001, = 0.28.
Next, we present extra sample paths of SGD when applied to function in eq. (A.1) in Figure A.2 and A.3 The blue curve on the right side of each plot shows I rotated by 90 degrees, and the dashed lines indicate the locations of local minima. For better readability of the figures, X,, for every 5,000 iterations. To generate these plots, we initialized the SGD iterates at 0.3 (so that it is in 92 (0.2, 0.7)) and fixed the learning rate as 1) 0.001. Again, gradient clipping (with b = 0.5) and without gradient clipping. Moreover, we also tested light-tailed noises where we use NN [0,0, 1) as the distribution for noises Zn. For each sample path of X,, run 10, 000, 000 iterations. In the left plots of Figure A.2] one can see that with clipped heavy-tailed stochastic gradients, the SGD iterates almost always stay around the wide attraction fields, and the sharp minima are almost completely eliminated from the trajectories of SGD. In comparison, in the right plots of Figure A.2 one can see that without gradient clipping, the heavy-tailed noises will drive SGD to spend substantial amount of time in all the different local minima, including the sharp ones. Lastly, from Figures one can see that under light- noises and small learning rates, SGD cannot escape a sharp minima once trapped there.
plotted
tested both with
A.2 DETAILS OF THE Rd SIMULATION EXPERIMENT
As illustrated in the contour plot in Figure (a), the function f in this experiment is a modified version of Himmelblau function, commonly used test function for optimization algorithm. The
13

---

Published as a conference paper at ICLR 2022
Keg-
Xn
Xeg
Xn
0.3
64
3.6
60
0A 0.
1.0
Figure A.2: Five sample paths of SGD under heavy-tailed noises with gradient clipping (left) and without gradient clipping (right). Note that in each case, SGD sample paths exhibit similar patters: SGD almost completely avoided sharp minima with gradient clipping, whereas SGD spent significant amount of time at the sharp minima without gradient clipping.
X
Xe
Reration number
Berazon number
1x7
Figure A.3: Five sample paths of SGD under light-tailed noises with gradient clipping (left) and without gradient clipping (right). Note that regardless of the use of gradient clipping, SGD never manages to escape the local minimum that it started from.
modifications serve two purposes. First, as shown in Figure (b), for the modified function the four attraction fields Q2 22 22g, 24 have different sizes; in particular, under gradient clipping threshold b 2.15, from the local minimizers of Ω and 22 (indicated by red dots in the corresponding area) at least two jumps are required to escape from the attraction field, while from the local minimizer in Ωg 24 it is possible to escape with one jump. Therefore, for the minimum jump number required
14

---

Published as a conference paper at ICLR 2022
= -== in this case. Second, for the modified test function f. the local minimizer in S2 is not a single point but connected line segment, which indicated by the dark line in bottom-left region in Figure (a) and the red line segment in in Figure (b). Therefore, the modification allows us to test the heavy-tailed SGD methods on a more general loss landscape.
escape, have 1 = = =
Now deseribe the construction of the test function Let h be the Himmelblau function with expression (1x y) = (x² + 11)2 + (a + y² 7)2. Next, define the following transformation for coordinates: 6(x, y) = [x(exp(co(== - c²) + 1)), y(exp(co(( c²) + + Let the composition be (2 y) = ((x - az.vl. v.). To create the connected region of local minimizers, define the following locally "cut" version of hég
i(x,8) = 1 {x E [bt, byl, by ayl < byb h² h"r,,) (1 i(z v) (1z v) + i(x, v) min(ho(I; cily c1ly-ayl'.) a.."..).
In other words, by taking minimum of the original he and a polynomial function w.r.t. around the original local minimizer of f2g, obtain function h" that attains local minimum entire line segment with = ay Lastly, the test function we use in the experiment is = 0.14", with a, 115,aa dy = 2.9, by -5.5, b,- 00.5, by 2.0, co 0.4, Ci 12.
In the experiment, we initialize the SGD iterates X at Xo = (2.9, 1.0), which is very close to the local minimizer in the small attraction field 123. For both the clipped and unclipped SGD, we perform updates for 3 x 102 steps, under learning rate 5 x 10- and heavy-tailed noise Z₁ 0.75W,, where the iid samples W₁ are isotropic and the law of [|Wl], the size of the noise, Pareto clipped SGD, we use threshold b = 2.15. To prevent the iterates from drifting to infinity, after each update X is projected back to the L2 ball centered at origin with radius 4.2 whenever X₃ leaves this ball.
For
-1
(a) with gradient cippping
(b) no grad ent clipping
Figure A.4: Heat map of SGD iterates when optimizing the modified Himmelblau function.
In Figure A.4 we use the x 10 steps of SGD iterates (for both the clipped and unclipped case) to create heat maps showing locations of SGD iterates. From this figure, two points can be made clear: first, the heavy-tailed SGD does spend much less time at the two small attraction fields when gradient clipping is applied; second, Q2 (the bottom-left attraction field) the SGD iterates frequent the entire connected region of local minima instead of certain point on this line segment.
A.3 DETAILS OF THE ABLATION STUDY
We first mention that the all experiments using neural networks are conducted on Nvidia GeForce GTX 1080 Ti. For the ablation study, the experiments and scripts are adapted from the ones in Zhu
ct al. 2019
In Figure A.S display the gradient noise distribution in the three tasks of the ablation study after the model is randomly initialized..
15

---

Published as a conference paper at ICLR 2022
Comppted FachionMNIST
VGG11
CIFARIO, VGG11
80-
50
30
40
40
20-
1.0
Figure A.5: Distribution of gradient noise in different tasks of the ablation study.
(a)
FashionMNiST
(b) SVHN, VGG11
CIFARID, V6G11
0.710
0.890
0.770
0.885
0.705
0.765
0.880
0.760
0.875
0.755
0.805
0.870
0.855
0.890
0.850
0.765
our methed
0.855
our methed
mechod
method2
0.880
0.850
0.735
0.685
Figure A.6: Test accuracy of the proposed clipped heavy-tailed methods test accuracy of vanilla SGD in the ablation study.
The experiment setting and choice of hyperparameters mostly adapted from the experiment in Zhu et al.] (2019). We consider three different tasks: (1) training LeNet on corrupted MNSH dataset; specifically, we use 1200-sample subset of the original FashionMNIST training dataset, and for 200 samples points in the training set we randomly assign label instead of using the correct ones; (2) VGG11 on SVHN dataset, where we use 25000-sample subset of the training dataset; (3) VGG11 on CIFARIO, where we use the entire training set. For all tasks we use the entire test dataset when evaluating test accuracy.
Table A.I: Test accuracy (percentage) and expected sharpness of different methods across different tasks. The reported numbers are the averages and 95%CI over 5 replications.
Test Accuracy
Corrupted FMNIST, LeNet
SVHN, VGGI1
CIFAR1O, VGGI1
LB
68.7.00.4
82.9-10.4
69.410.5
SB
69.2
85,9+0.2
74.4+0.4
SB Clip
68.8::0.6
85,9+002
74.4=0..
SB Noise
64.413,4
38.9+22.1
40.5+25.1
Our
69.5+00.8
88.4+-0.2
75.7+11.1
Our 2
70.1+0044
88.4++0.2
75.9+0.7
Expected Sharpness
Corrupted FMNIST, LeNet
SVHN, VGGI1
CIFARIO, VGGI1
LB
0.032
0.694
2.043-10.083
SB
0.008
0.037+0.007
0.050+10.013
SB Clip
0.00910.001
0.041
0.039:0.019
SB Noise
0.047+0.02
0.012+0.009
2.046+2.4
Our
0.0003
0.00270.0007
0.024+0.005
Our 2
0.0022
0.005
0.007
16

---

Published as a conference paper at ICLR 2022
Table A.2: Hyperparameters for training in the ablation study
Hyperparameters
FashionMNIST, LeNet
SVHN, VGGI1
CIFARIO, VGGI1
learning rate batch size for JSB
0.05
0.05
0.05
100
100
100
training iterations
10,000
30,000
30,000
gradient clipping threshold
20
20
0.5
0.5
0.5
a
1.4
1.4
1.4
Table A.3: Sharpness of different methods across different tasks. The reported numbers are the
averages over 5 replications.
PAC-Bayes Sharpness
Corrupted FMNIST, LeNet
SVHN, VGGII
CIFARIO, VGGII
LB
5.9 x 10a
2.97 10a
4.87 x 104
SB
3 x 10
6.9 X 10
7.2 x 10a
SB + Clip
3.3 x 103
7.3 x 103
6.8 x 103
SB + Noise
3.1 x 10a
7.76 104
6.74 X 10a
Our 1
1.9 x 10a
2.1 x 103
4.8 X 103
Our 2
1.6 x 10
2.3 x 103
5.8 x 10
Maximal Sharpness
Corrupted FMNIST, LeNet
SVHN, VGGI1
CIFARIO, VGGI1
LB
1.01 10
3.78 10a
5.46 x 101
SB
4.9 x 103
9.1 X 103
8.5 x 103
SB + Clip
5.4 x 103
9.3 X 103
8 x 103
SB + Noise
5.4 x 10a
1.19 100
1.18 X 10S
Our 1
3.2 x 103
2.5 X 103
5.8 X 103
Our 2
2.5 x 10
2.8 x 103
6.5 x 10
The heavy-tailed multipliers Zn used in this experiment, whenever heavy-tailed noise is needed, are Zn cW,, n where Wn n are iid Pareto(a) RVs. For each task, we first randomly initialize each model, and then run the 6 candidate methods in parallel starting from the same randomly initialized model weights for fair comparison.
The hyperparameters in training for each task are listed in Table A.2 The same set of hyperparameters used for all methods in the same task. Whenever gradient clipping scheme is applied, we clip the gradient if its L2 norm exceeds the threshold given in Table A.2 The exception here is the "SB + Noise" method: we use learning rate = 0.005; for FashionMNIST task we train for 100,000 iterations and the heavy-tailed noise is removed for the final 50,000 iterations; for SVHN and CIFARIO tasks, we train for 150,000 iterations and heavy-tailed noise removed for the last 70,000 iterations. Besides, for this method we always clip the model weights if its Loo norm exceeds 1. The reason for the extra tuning and extended training in "SB Noise" method is that, without the said modifications, in all three tasks we observed that the model weights quickly drift infinity and explodes; even with the weight clipping implemented, the model performance stays at random level with signs of improvements if do not tune down learning rate.
In Table A.3 we also report the sharpness of solutions under different shaprness metrics. First, the PAC-Baye Sharpness metric (see equation (53) in Jiang al. (2019) is defined as 1/o2 where is equal to the smallest 5 that induces a 0.1 expected sharpness, and reflects the sharpness/flatesss parameter used in studies on generalization gaps under the PAC-Bayes framework (see Neyshabur et al.)
(2017a)). Besides, the Maximal Sharpness metric (see equation (54) in Jiang et al. 2019)) is defined as 1/a² where equal to the smallest radius I that makes max / 55 [.(00" +
T(O))) | 2 0.1,
and metrics of form ma [( + v) - L((*)) ){ can be considered as a proxy for the spectral norm of the Hessian at the solution (see Dinh et al. (2017)). It worth noticing that, for all three sharpness metrics, the smaller the value is the "flatter" the loss landscape is around the solution. Lastly, for evaluation of the PAC-Bayes Sharpness and Maximal Sharpness metrics, conduct binary search as in Algorithm 2 of Jiang et al. (2019) with 6 = 0.01, 0, M, setting we always evaluate the training loss using one sweep of the entire training set, so M3 is
and M2 = 100: in our
17

---

Published as a conference paper at ICLR 2022
Table A.4: Results and 95% CI in the experiments with data augmentation.
Test Accuracy
SB + Clip
Our 1
Our 2
CIFARIO, VGGII
89.510.2
90.7
90.5+10.2
CIFARI00 VGGI6
56.3+0.3
6..4+11.2
63.0:12.5
Expected Sharpness
SB + Clip
Our I
Our
CIFARTO, VGGIT
0.17
0.004
0.10:E0.003
CIFAR100, VGG16
0.86+0000
0.05
0.48++0.07
Table A.5: Sharpness of solutions obtained by different methods in CIFARI0/100 tasks with data augmentation. Numbers reported here are the average of 5 replications.
CIFARIO-VGGI1
SB + Clip
Our
Our 2
Expected Sharpness
0.167
0.085
0.096
PAC-Bayes Sharpness
1.31 x 104
103
10t
Maximal Sharpness
1.66 x 10*
1.29 x 104
1.22 x 104
CIFAR CIFARIOO-VE VGG16
SB + Clip
Our
Our 2
Expected Sharpness
0.857
0.441
0.479
PAC-Bayes Sharpness
2.49 x 101
1.9 x 104
1.98 x 104
Maximal Sharpness
2.75 x 101
2.12 X 10a
2.16 10
case-specific and is equal to the number of batches of the training set under the batch size for the task at hand.
In Figure A.6 we plot the test accuracy of our method against that of the SGD for replications and tasks.
A.4 DETAILS CIFAR10/100 EXPERIMENTS WITH DATA AUGMENTATION
For both methods, we train the model for 300 epochs and set the initial learning rate 0.1. In our method, the training can be partitioned into two phases. In the first phase (the first 200 epochs), the learning rate is kept at constant. In the second phase, for every 30 epoch
reduce the learning rate by half. Also, an L2 weight decaying with coefficient 5 x 10-4 is enforced. As for parameters for heavy-tailed noises in eq. (6. in the second phase we remove heavy- noise and use SB to update weights.
0.5 and a 1.4 in the first phase, and
both methods for the small-batch direction Isn the batch size is 128, while for grn evaluate the gradient on a large sample batch of size 1,024. Under the epoch number 300 and batch size 128, the count of total iterations performed during training is 1.17 x 10a To augment the dataset, random horizontal flipping and cropping with padding size is applied for each training batch. Lastly, gradient clipping scheme is applied for both methods, and we fix 0.5. In other words, when the learning rate is (note that due to the scheduling of learning rates, 77 will be changing throughout the training), the gradient is clipped if its L2 norm is larger than b/n. The scripts are adapted from the ones in https://github.com/ece
These results are presented in Table Furthermore, in Table A.5 we see that our truncated heavy- tailed method also manages to find solutions with flatter geometry.
A.5 DISCUSSION ON GRADIENT NOISE DISTRIBUTIONS THE EXPERIMENTS
In this subsection, we (i) present the empirical evidence that supports our characterization of the baseline model-i.e. the absence of heavy tails the gradient noise-and and (ii) clarify that the emergence of heavy-tails in the $ sstationary distribution of SGD (argued in Hodgkinson & Mahoney (2020); Gurbuzbalaban et al. (2020)) does not contradict the observed absence of heavy tails in the "gradient noise of SGD. This allows noise separately from the choice of hyper-panc parameters.
study the impact of truncated heavy-tails in the gradient
We start with observations the noise distributions. In view of the well-established wisdom in heavy-tail literature that there is no single perfect tail estimator, we analyzed the stochastic gradient noise with four different methods: QQ plot, empirical mean residual life (EMRL., Hill plot, and
18

---

Published as a conference paper at ICLR 2022
PLFIT (Clauset et al. 2009). We applied these estimation/diagnosiii tools (i) at the beginning of the training, (ii) halfway through the training, (iii) at the end the training. Throughout all our experiments, consistently observe strong evidence that the gradient noises light-tailed, and even if (against all odds) the noises were from heavy-tailed distribution, the tail index should be far greater (hence, the resulting tail is much lighter) than the heavy-tails we inject (or the popular alpha stable assumption), and hence, the point we make with our tail-inflation experiment is still valid. We summarize the results follows.
First, QQ plots below (Fig. A.21 clearly show that the tails in noise distribution are always much lighter than the Pareto distributions with assumption, especially the alpha-stable distributions in Şimsekli et al. (2019b) (with a (0, 2)). seems far from the distribution of the actual data we obtained the image classification tasks. In fact, the tail of the noise distributions seems to be between that of lognormal and normal distributions, implying that is lighter than any power-law distribution.
or even 10. Therefore. the typical power-law
Pareto, a
Pareto, a
Pareto, ar Lognormal
Normal
Figure A.7: Ablation Study, Corrupted FMNIST & LeNet: At the beginning
Pareto, a = 2 Pareto, a = 5
Pareto, 10
Lognormal
Normal
Figure A.8: Ablation Study, Corrupted FMNIST LeNet: Half way through the training
Pareto, 2
Pareto,
Pareto, 10 Lognormal
Normal
Figure A.9: Ablation Study, Corrupted FMNIST & LeNet: At the end of training
Pareto, a
Pareto, a
Pareto,
Lognormal
Normal
Figure A.10: Ablation Study, SVHN & VGG11: At the beginning
Pareto, a = 2
Pareto, a = 5
Pareto, 10
Lognormal
Normal
Figure A.11: Ablation Study, SVHN & VGG11: Half way through the training
19

---

Published as a conference paper at ICLR 2022
Pareto, a
Pareto, a
Pareto, 10
Lognormal
Normal
Figure A. 12: Ablation Study, SVHN & VGGI1: At the end of training
Pareto, a
Pareto, a
Pareto, a
Lognormal
Normal
Figure A.13: Ablation Study, CIFARIO & VGGI1: At the beginning
Pareto, 2
Pareto,
Pareto, 10
Lognormal
Normal
Figure A.14: Ablation Study, CIFARIO & VGGI1: Half way through the training
Pareto, a = 2
Pareto, a = 5
Pareto, = 10
Lognormal
Normal
Figure A.15: Ablation Study, CIFARIO & VGG11: At the end of training
Pareto, a
Pareto, a
Pareto, 10
Lognormal
Normal
Figure A.16: Data Augmentation, CIFAR10 & VGG11: At the beginning
Pareto, a = 2
Pareto, a = 5
Pareto, 10
Lognormal
Normal
Figure A.17: Data Augmentation, CIFARIO & VGGI1: Half way through the training
Next, we plotted the empirical mean residual life (EMRL) of the gradient noise distributions in Fig. A.22 A.26 It is well known that the mean residual life blows up to infinity and only the distribution is heavy-tailed (more precisely, long-tailed) However, from the figures, none of the EMRL exhibits such a pattern in any case tested in our experiments. Instead, we see clear downward trends, which strongly suggest light tails, in all the tested cases.
see that
20

---

Published as a conference paper at ICLR 2022
Pareto, a
Pareto, a
Pareto, 10
Lognormal
Normal
Figure A.18: Data Augmentation, CIFARIO & VGG11: At the end of training
Pareto, a 2
Pareto, a 5
Pareto, a = 10
Lognormal
Normal
Figure A.19: Data Augmentation, CIFARIO0 & VGGI6: At the beginning
Pareto, a
Pareto, a
Pareto,
Lognormal
Normal
Figure A.20: Data Augmentation, CIFARI00 & VGG16: Halfway through the training
Pareto, a =
Pareto, a
Pareto, a 10
Lognormal
Normal
Figure A.21: Data Augmentation, CIFARI00 & VGG16: the end training
Beginning
Middle
End
125
62
6
10
Figure A.22: Plots of empirical mean residual life for noises in FMNIST&LeNe Task throughout training
Beginning
Middle
End
0007
1007
13
01009
Figure A.23: Plots of empirical mean residual life for noises in SVHN&VGG 11 Task throughout training
21

---

Published as a conference paper at ICLR 2022
Beginning
Middle
End
28
Figure A.24: Plots of empirical mean residual life for noises in CIFAR 10&VGC 11 Task throughout
training
Beginning
Middle
End
Figure A.25: Plots of empirical mean residual life for noises in dataAug, CIFAR 10& VGG 11 Task throughout training
Beginning
Middle
End
130
15
Figure A.26: Plots of empirical mean residual life for noises in dataAug, CIFAR 100& 11 Task throughout training
If we assume power-law tail, the Hill estimator is a classical tail index estimator with long history extreme value theory literature. A critical algorithmic parameter of the Hill estimator is the number of order statistics used in the estimation, and the Hill plot is popular exploratory tool for investigating the Hill estimators with different numbers of order statistics. Although it is well known that Hill estimators and Hill plots are fallible if the power-law assumption is not satisfied, (and hence, it is not suited for deciding whether a given set of samples are from light-tailed distribution or saavy-tailed distribution; in particular, the method will return some power-law tail index whether or not the samples are from a heavy-taid tailed distribution or a light-tailed one) we present the Hill plot see what would be the estimated tail indices if the gradient noises hypothetically followed a power law, In the Hill plots shown in Fig. A.27 A.31 we presented the rescaled version (altHill) of the Hill plots (see Chapter 4.4 [Resnick] (007)) for the following reason. Hill estimator is consistent estimator of the power-law index when the proportion of the samples used approaches O, and altHill plots allow us scrutinize the estimated indices under small proportion of samples.
particular, the points around the red dashed lines correspond to estimation using the top 1 1% of the samples. We can see that most Hill plots stay well above 10 for the most part and almost never drop below This strongly suggests that even if the gradient noises from a heavy-tailed distribution, it is likely to have a very high power-law index (implying relatively lighter tails), and hence, we cannot expect to observe a prominent heavy-tailed behavior from them.
A popular data-driven approach with statistical guarantees (again, under the assumption that the samples are indeed from heavy-tailed distribution) to select the number of order statistics in the Hill plot is PLFIT (Clauset et al.) 2009). We estimated the power-law indices using the python implementation (Alstott et al. 2014) of PL.FIT The numbers are presented in Table A.6 All the
22

---

Published as a conference paper at ICLR 2022
Beginning
Middle
End
Figure A.27: altHill Plots for noises in FMNIST&LeNet Task throughout training, Dashed Red Line:
Estimation based on the largest 1% data
Beginning
Middle
End
Figure A.28: altHill Plots for noises in SVHN& VGG 11 Task throughout training. Dashed Red Line:
Estimation based on the largest 1% data
Beginning
Middle
End
Figure A.29: altHill Plots for noises in CIFAR VO& Task throughout training. Dashed Red
Line: Estimation based on the largest 1% data
Beginning
Middle
End
Figure A.30: altHill Plots for noises in Data Augmentation, CIFAR 10& VGG 11 Task throughout training. Dashed Red Line: Estimation based on the largest 1% data
estimations are at least 5 for all cases tested in our experiments, and most of the time, the estimation above 10. Again, this means that even under the hypothetical assumption (against what the QQ plots and EMRLS suggest) that the gradient noises were from heavy-tailed distribution, the tail indices of the gradient noises should large, and hence, the gradient noises in our experiments have much lighter tails than any a-stable distribution (which requires noises we injected during tail inflation experiments (a = 1.4). In summary, extensive statistical analyses above suggest the absence of heavy tails in the gradient noises in experiments. Therefore, the characterization the vanilla SGD as the light-tailed taile (or at least lighter than the inflated tails) benchmark in our experiments valid, and our ablation study is well grounded.
2) or the heavy-tailed
23

---

Published as a conference paper at ICLR 2022
Beginning
Middle
End
16-
22-
96
Figure A.31: altHill Plots for noises in Data Augmentation. CIFAR 100& VGG 16 Task throughout training. Dashed Red Line: Estimation based on the largest 1% data
Table A.6: Power-law Indices Estimation throughout the Training, using PLFIT, All the estimations least 5 for all cases tested in experiments, and most of the times the estimation is above 10. This means that even under the assumption that the gradient noises were from heavy-tailed distribution, they should have much lighter tails than any a-stable distribution (which requires < 2) or the heavy-tailed noises we injected during tail inflation experiments (a 1.4).
Task
Beginning
Middle
End
FMNIST, LeNet SVHN, VGGI1
14.3
14.2
16.5
5.0
5.2
12.5
CIFARIO, VGGI1
9.2
6.6
7.0
dataAug. CIFARIO. VGGII
16.2
16.2
8.5
dataAug, CIFARI00, VGGI6
35.1
14.4
5.35
Next, we compare our work recent literature heavy-tailed phenomena in stationary distribution of SGD; see, for instance, HHodkkisson & Mahoney (2020); (uurbubalaaan et al. (220).. To be specific. Hodgkinson
Mahoney (2020): Gurbuzbalaban et al. (2020) show that heavy tails can arise in the stationary distribution of SGD through multiplicative dynamics, and the tail index of the resulting stationary distribution can be characterized by the learning rates and the magnitude of noises (through batch size). However, the results in Hodgkinson & Mahoney| (2020); Gurbuzbalaban et al. (2020) do not imply the existence heavy tails in the gradient noises. both papers (as well as other works in the literature), the "heaviness" of the tail of the gradient noise (e.g.. power-law index in heavy-tailed cases) is fixed in the model (same as our setting) and not entangled with the learning rate or batch size. For example, Bg in (5) of [Hddgkinson & Mahoney (2020) corresponds to the gradient noise, and its tail index doesn't depend on the learning rate or the batch size. In particular, the change of learning rate does not induce heavy tails in Bg. On the other hand, focus the impact of the heavy-tails *in the gradient noise* and the truncation scheme for (any) fixed batch size and small learning rates on the global dynamies of SGD. In view of this, it should be clear that our analysis can be decoupled from the choice of batch size or the impact of the learning rate on SGD's stationary distribution. Therefore, our observation and the design of the experiments are compatible with the aforementioned references.
B IMPLICATIONS OF THE THEORETICAL RESULTS
Systematic control of the exit times from attraction fields: In light of the wide minima folklore, one may want to find techniques to modify the sojourn time of SGD at each attraction field. Theorem I suggests that the order of the first exit time (w.rt. learning rate 17) is directly controlled by the gradient clipping threshold b. Recall that for an attraction field with minimum jump number i", Theorem [ tells us the exit time from this attraction field is roughly of order (a/m)=((a-11"" Given the width of the attraction field, its minimum jump number [" is dictated by gradient clipping threshold b. Therefore, gradient clipping provides us with very systematic method to control the exit time of each attraction field. For instance, given clipping threshold 6, the exit time from attraction field with width less than b is of order (1/n)", while the exit time from one larger than ó is at least (1/n)* 20- which dominates the exit time from smaller ones.
24

---

Published as a conference paper at ICLR 2022
The role of structural properties of G and f: Recall that in order for Theorem B to apply. the irreducibility of G is required. Along with the choice of b, the geometry of function f is a deciding factor of the irreducibility. For instance, we say that G is symmetric if for any attraction field Q2 such that = 2, 3, ... Ammin 1 (so that Q2 is not the leftmost or rightmost one at the boundary), we have that 9 is symmetric if and only if, for any i 2. Dmmin 1, [s m. m V/m 8 1 < 6, and symmetry is a sufficient condition for the irreducibility of G. The graph illustrated in Figure 2 Middle) symmetric, while the one in Figure Right) is not. As the name suggests, in the IR case the symmetry of G is more likely to hold if the shape of attraction fields in f is also nearly symmetric around its local minimum. If not, the symmetry (as well as irreducibility) of G can be violated as illustrated in Figure 2 especially when a small gradient clipping threshold b used.
-..i- > 0, 11,+1 0. One can
Generally speaking, our results imply that, even with the truncated heavy-tailed noises, the function needs to satisfy certain regularity conditions to ensure that SGD iterates avoid undesirable minima. This is consistent with the observations in Li et al. (2018b); 8b); the deep neural nets that are more trainable with SGD tend to have a much more regular structure in terms of the number and shape of local minima.
Heavy-tailed SGD without gradient clipping: It is worth mentioning that our results also charac- terize the dynamics of heavy-tailed SGDs without gradient clipping. For instance, since the reflection operation at + restricts the iterates on the compact set [-T., L.J. if we use a truncation threshold b that is large than 2I, then any SGD update that moves larger than b will definitely be reflected + L. Therefore, the dynamics are identical to that of the following iterates without gradient clipping:
aannnclipeed = sr.. [xyuundlpppdd - "/S" vx''(X (X2-1 unnclpp + +,12mn) n
(B.1)
The next result follows immediately from Theorem [and H.2
Corollary B.1. There exist constants qu > 0 Vi, 9b Vj + such that the following claims hold for any i and any E 94
1) Under P.. q.H((/n)o.()) converges in distribution to an Exponential random variable with rate 1 as 0:
2) For any j = 1,2, ... Umin such that +
lim P. (X"
(s(1)) E El2, 9t.5/%
3) Let Y be continuous-time Markov chain on {m1,... Mm.a) } with generator matrix Q parametrized by Qi,
9s Qú, = 9b.j- Then
X annneeepedd (x) + Yo(ms) as 40
in the sense of fimieddiieess distributions.
At first glance, Corollary B.I may seem similar to the results in Şimşekli et al. (2019)) and Pavlyuke- (207). However, the object studied in (Simçekii et al. (2019a); Pavlyukevicy (207)) is different: they study the following Langevin-tyre stochastic differential equation (SDE) driven by an a-stable Lévy process L, with scaling factor 17 > 0:
dY,"" = - (Y + ndLL.
In particular, Pavlyukevich (2007) studies the metastability of Y," and concludes that as 1 O, the first exit time and global dynamics of Y," admit a similar characterization
described in our Theorem B.I. Then Theorem 4 in (Şimşekli et al.) (2019a) argues that when the learning rate rj is sufficiently small, the distribution of the first exit time of the SGD X and that of the Lévy-driven Langevin SDE Y." are similar. However, the analysis of "Simçekli et al. (2019a) hinges critically on the assumption that T. is symmetric and a-stable. While such an assumption is convenient for their analysis, is a strong assumption. It implies that the gradient noise distribution belongs to very specific parametric family and excludes all the other heavy-tailed distributions. In particular, the assumption precludes analysis of any heavy tails with finite variance. On the contrary, our work directly analyzes the SGD X and reveals the heavy- SGD dynamics at a much greater level of generality. Specifically, allow the noise to have general regularly varying distributions with arbitrary tail index-wiin includes atsbable distributions as a (very) special case- and extend the characterization of global dynamics of heavy-tailed SGD to the adaptive versions of SGD where gradient clipping is applied.
25

---

Published as a conference paper at ICLR 2022
C GEOMETRIC CHARACTERIZATION I' AND EXISTING SHARPNESS METRICS
Throughout the paper, we have been using the term sharp minima when describing the elimination effect of truncated heavy-tailed SGD, while appealing to 1", the minimum number of jumps requires for escape defined in eq. (3), when rigorously presenting our theoretical results about dynamics of truncated heavy-tailed SGD. Despite the possible ambiguity of its use in the main paper, the terminology sharpness is meant to familiarize our rather technically involved notion of how heavy- tailed SGD behave. To resolve the potential confusions, in this section we provide a detailed discussion on the relationship between our geometric characterization (" and the existing sharpness metries.
In light of recent discovery that all local minima might be global minima for over- wverpaaraeetrieed deep neural nets (see et et al. (2018at), one popular explanation for the generalization gap between different local minima (for instance, between solutions found by GD and SGD) is that the geometry around the solution is closely related to its performance in the test setting. After the seminal work by Keskar et al.] (2016), myriad of attempts have been made provide empirical or theoretical evidences for the link between the sharpness of a local minimum and its generalization performance;
See, for instance, Xie et al.] (2020): Jiiang et al. (2019); He et al. (011)); Zhou et al. (2020). In summary, there exist at least four difterent classes of sharpness metrics among the current literature:
(a) spectral norm of the Hessian the local minimum, one surrogates of which is the maximal sharpness type of metrics; a variant used in Xie et al. (2020) based on corresponding large deviation theory is the eigenvalue of the Hessian along certain escape directions;
(b) expected sharpness type of metrics that evaluate the general fluctuation of the loss function within a Lp ball of centered at the local minimum (see Zhu et al. (2019))
(c) complexity metrics based on BAC-Bayes theory (see, for example, Neyshabur et al. (011aa);
(d) geometric property of a domain or the entire attraction field instead of the local minimum itself: see the mass of Radon measure m(W) over the domain W in Theorem in Zhou ct al. 2020).
Under such taxonomy, the ! characterization proposed in this paper falls into category (d). Indeed, describes the minimum effort required for escaping the domain rather than the geometry merely around a certain neighborhood of the local minimum. Here we have two remarks on the definition of l*. First, for a given loss landscape, this quantity 1* is dictated by the gradient clipping threshold b > O. In fact, this concept is tailored for the gradient clipping case, as in the unclipped case with heavy-tailed SGD, for any attraction field Z2 we always have 1; = Second, is analogous to the term mWW) in Zhou et al. 2020) in the sense that it reflects the volume of the attraction field (when compared to a given threshold b). Therefore, we stress that a more precise interpretation of 1*, as well as other metrics in class (d), is that they characterize how wide or narrow each minimum (attraction field) is, and a more a clear description of our theoretical results Theorem | 3 is that truncated heavy-tailed SGD effectively avoids all the narrow minima when learning rate is small enough.
While one can intuitively see that wide minima are more likely to be flat ones, we acknowledge the existence of counterexamples where a sharp minimum lies in a wide attraction field. Nevertheless, as the variety of definitions for sharpness keeps growing in existing literature, becomes rather unlikely for one geometric characterization to always agree with (or be equivalent to) other existing approaches. More importantly, the aim of this paper is not to establish 1" as the orthodox geometric property when studying generalization gap. Instead, as demonstrated in Table [ modern deep learning tasks the truncated heavy-tailed method (which prefers solutions with high I' as indicated by our theoretical results) obtains solutions that generalize better and exhibit flatter geometry when evaluated under different sharpness metrics. This observation is well aligned with the recent large-scale empirical study in Jiang et al. (2019), suggesting that in typical training setting it is beneficial to find wider minima (characterized by (") in order to achieve a flatter geometry and better generalization performance.
A.3 and A.5] in
26

---

Published as a conference paper at ICLR 2022
D ON THE RELATIONSHIP BETWEEN SHARPNESS AND GENERALIZATION
In principle, a sharp minimum does not necessarily lead to poor generalization in the sense that it is possible to construct pathological counterexamples in theory or in specific experiment settings; see, for example, Dinh et al. 2017); Neyshabur et al. (2017a). However, the loss geometry that arises in practice seems to exhibit strong correlation between the sharpness and the test error. For instance, sharpness aware optimization methods (Foret et al., 2020 Kwon et al.) 2021 improve generalization across various tasks and achieve state-of-the-ar performance on the CIFAR dataset. Also, one of the authors of the aforementioned paper Neyshabur et al. (2017a) continued investigating different complexity measures and conducted much larger scale experiments in Diang et al. (2019) Among more than 40 complexity measures from theoretical and empirical studies in the literature tested in Jiang et al. (2019) the sharpness metrics are the top ones for predicting the generalization performance. In our own experiments, we also show that sharpness and test accuracy are highly correlated.
E STABILITY-DRIVEN ANALYSES ON SGD
Wu et al. (2018) takes the perspective of linear stability and establishes conditions for SGD to be attracted to or avoid certain solutions based on learning rate, batch size, and the concept of non- uniformity of local minima. Inspired by (Wu et al. 2018), (Jastrzebski et al. 2020) analyzes the trajectory-wise stability of SGD and found that "break-even point" partitions the training procedure into two phases: the implicit regularization effects in SGD due to a larger learning rate becomes visible in the second phase after this "break-even point". Similarly, (Cohen et al. (202]) reports that typical GD trajectories in standard image classification tasks can be partitioned into two phases: in the first "progressive sharpening" phase the sharpness of the Hessian monotonically increases, while in the second phase we observe the "edge of stability" regime where the sharpness of Hessian hovers slightly above the critical value 2/n and the training loss slowly decrease in an oscillating fashion.
Compared to the aforementioned stability analyses, one major difference of our heavy-tailed regime that, even under gradient clipping, the trajectory cannot be partitioned into such "phases" and the traditional sense of stability around certain local minima is nullified by the constant basin hopping and exploration behaviors under heavy-tailed noises. In particular, the polynomial order of the exit time in heavy-tailed SGD dictates that, when compared to exponentially long exit time in vanilla SGD (see Xie et al. 22020), the iterates won't become "stabilized" and keep staying around a certain region in typical training procedure. Instead, our work characterizes the metastability of the truncated heavy-tailed SGD (i.e. constantly transitioning between different wide minima) that cannot be observed in GD or light-tailed SGD. More importantly, compared to the aforementioned stability related analyses, our work provides a very tight characterization of the global dynamics and distributions of the entire sample path.
F EXISTING ANALYSES ON HEAVY-TAILED PHENOMENA IN SGD
We start with one clarification: what has been established in Hodgkinson & Mahoney (020);
Gurbuzbalaban al. (202I); WWojtowytsch (2021); Mori et al. 102 is that the stationary distribution of the SGD (or the corresponding continuous-time SDE) can be heavy-tailed, and the heavy tailes therein can depend on various training hyperparameters. This, however, does not imply that the gradient noise is heavy-tailed, In particular, the heavy-tail index (or heavy-tailedness itself) of the gradient noise will not be affected by hyperparameters such as the learning rate.
To be specific, Hodgkinson & Mahoney 2020);; Gurbuzbalaban et al. (202T) show that heavy tails can arise in the stationary distribution of SGD through multiplicative dynamics, and the tail index of the resulting stationary distribution can be characterized by the learning rates and the magnitude of noises (through batch size). Similarly, the SDEs studied in Wootowytsch/ (2021): Mori et al. (202 I) are both driven by light-tailed (Gaussian) perturbations, and the authors show that even under light-tailed perturbations, heavy tails can arise in the stationary distribution through multiplicative dynamics, and the tail index of the resulting stationary distribution can be characterized by the learning rates. In comparison, we focus on the impact of the heavy-tails in the gradient noise and the truncation scheme on the global dynamics of SGD. Our results show that, under heavy-tailed noises, the power-law
27

---

Published as a conference paper at ICLR 2022
index for the tail in noises also characterizes the first exit time and global dynamics of SGD with no dependence on other training hyperparameters (at least in the asymptotic scheme). In other words, when the driving force of the dynamics is heavy-tailed, its "heavy-tailedness" (i.e., the same index (x) characterizes the "heavy-tailedness" of the entire SGI) without dependency on the other hyperparameters, and its effect the entire SGD path (rather than just the stationary distributions) admits a very clear and tight expression, as characterized in theoretical results in this work.
G PROOF OF THEOREM
1
This section proves Theorem 1 We first start with providing the definitions of q: and qi, that appear in the statement of the theorem. Let Leb,, denote the Lebesgue measure restricted on [0, oo), and define a (Borel) measure 1/a on R {0} as follows:
ap+ gatt +
Nr (dx) = 1 fa > 0}
11x
<
[0} |ora+I
where o, p-, and p+ are constants in Assumption 2. Define Borel measure μ4; on R x (R++) -1 as the product measure
[4 = (Ma)* x (Leb,)(i-1.
(G.1)
We also define mappings h2 as follows. For a real sequence w = (W1, W2. ".. ww; and a positive
real number sequence = (())]= define t = ti = and t, =t + t22 + +..... t'y for j = 2, :. ,47 Now we define a path & : [0, 20) + R as the solution to the following ODE with jumps:
x(0) = (m= + (ww)))
(G.2)
d'((t) = -f'(((t),, Vt E [t)-,, Vj = 2, ,4i
(G.3)
dt
*(1,) = sp (x(1) -) + (o(w))); Vj 2, .16-
(G.4)
( 41 + R as
Now we define the mappings h2 : RR x
hw(w,t) x(t1;).
It is easy to that h are continuous mappings. With these mappings, we define the following sets:
E a [(w, C CR x (R+) 55 tw,(w,t) #2):: Eij a
(G.5)
x (R+)
17-1 : (w, Eg)..
(G.6)
Lastly, the constant ys and qi are defined as follows:
q (F,), h. = (F()))) $
(G.7)
Before we move on to the proof of Theorem I we add a few remarks regarding the intuition behind it.
Suppose that X is started at the 7th local minimum m. of f. and consider the behavior of X over the time period H, 2 {1..... [t t//l] for a sufficiently large t. The heavy-tailed large deviations theory Rhee et al. (2019) and a heuristic application of the contraction fa/ni over this period will converge to the gradient flow of f. and the event that X" [n//)] escapes Ω2 within this period is (heavy-tailed) rare
principle implies that the path of
event. This means that the probability of such an event is of order (1/n)(a-1)1; Moreover, whenever it happens, it is almost always because X is shaken by exactly I large gradient noises of size (1/r), which translates to I jumps in X (///n] 's path, while the rest of its path closely resemble the deterministic gradient flow. Moreover, conditional on the event
28

---

Published as a conference paper at ICLR 2022
that X fails to escape from S2 within this period, the endpoint of the path is most likely to close to the local minima, i.e, X 2t/y] Re m,- This suggests that over the next time period
H2 - {[t/n] + 1, [t/n] + ... 2[t/n]] of length [t/n], X will behave similarly to its behavior over the first period H1. The same argument applies to the subsequent pe H3, .4....a as well. Therefore, over each time period of length [t/n]. there is (1 (1/m)(a-11; probability of exit. In view of this, the exit time should be of order (1/n)=+(a-11;, and resemble an exponential distribution when scaled properly.
Part of Theorem builds on this intuition and rigorously prove that the first exit time is indeed roughly of order (n) R (1/n)=((a-)1, and resembles exponential distribu- tion.
Given this, one would expect that X or)))) the location of SGD right at the time of exit, will hardly ever be farther than 17 b away from m,: the length of each update is clipped by b, and there will most likely be only 1द large SGD steps during this successful attempt. Indeed, from the definition of 9s.j above, one can see that qi,j > 0 if and only if inf infy,n, 1y m, < 17
Summarizing the three bullet points here, we see that the minimum number of jumps ? dictates how heavy-tailed SGD escapes attraction field, where the SGD lands upon its exit, and when the exit occurs.
Now we are ready to start proving Theorem First, note that Assumption implies the following:
There exist co 0, E (0, 1) such that for any æ E {m1, 81, Snain mani J y - x < ED.
If'(v) > coly z).
(G.8)
and for any y E [-L, L] such that ly - x| z Eo for all æ E {m11 .. 8 M }
we have
[f( >
co:
(G.9)
There exist constants E (0, oo), M o0) such that |mal < Ly Mm any x E [-L, LJ)
< L, and (for
[f'(x) s M, [f"()) M.
(G.10)
Recall that for the truncation operator was defined as
ve(w)) - p(w, e) a (w (w)c) Ai e
(G.11)
and the SGD iterates were defined
X oL X 9b (n(s1(((()) Zn+1)))-
(G.12)
Here 1 >> is the learning rate (step length) and b is the gradient clipping threshold. Also, recall that for any & E [Pmin], we let
(n) و min 2 0: X $ 94}
be the time that X exists from the k-th attraction field Qy. Meanwhile, given the gradient clipping threshold b, recall that
min m, 8,-1, 84 mc).
(G.13)
[r./b].
(G.14)
Intuitively speaking, 4 tells us the minimum number jumps with size no larger than 6 required in order to escape the attraction field if we start from the local minimum of this attraction field Ω.
Lastly, recall the definition of H--) P( and
H(1/1) 1-1
(x & ((1m))
1
29

---

Published as a conference paper at ICLR 2022
The proof of Theorem hhinoes the following two lemmas that characterize the behavior of X in two different phases respectively. Let & [Main] and I E Ω2. We consider the SGD iterates initialized at X = In the first phase, the SGD iterates return to [m, 2e, m + 2e). small neighborhood of the local minimizer in attraction field 94: in other words, it ends at
T Tean (1), e) A min(n 2 0: X [m, - m + 2e]].
(G.15)
During this phase, we show that for all learning rate 1) that is sufficiently small, is almost always the case that X would quickly return to [my- - 2e, my + 2c), and it never leaves O4 before Tri retum- Lemma G.1. There exists some E o0) such that for any ls E [Mmii]) the following claim holds for all e > small enough:
lim 940 yEI LJ (6( Py (XX E Og Vn E [ (n, e)], I/ei (n clog(1/0)
inf
=
"
During the second phase, X starts from somewhere [mg 2c, my + 2c] and tries to escape from Og meaning that the phase ends at (n). During this phase, we show that the distributions of the first exit time o (1) and the location X do converge to the ones described in Theorem as learning rate 1y tends to O.
ox(n)
Lemma G.2. There exist constants qi 0 Vi [Mmie]] and qrj 2 0 V. E [Mmiil] with i.j # i such that the following claim holds: given any > O, u > O and any k. E [Mmiee] with k * I, we have
lim sup 100
sup =E(m= 20mm2-22,m4-0 me,
P. (n)x(()) > w) S C + exp ( (- (-(1 C)u) (G.16)
2E[-L,L],
lim inf 740
inf mm(mm= - 26, 2c))
P (odA(n)o > zu) 2 -C + exp
(G.17)
lim n-0 wE[-L,L], xE(m= mc
sup
Pa X" Ok(1)
(x
ERi s Ux,l +
(G.18)
qk
Pr (x (c(()) EQ)
(x"
lim inf 900 =E[-L,L), xE(m- +2,
inf
(G.19)
9k
for all e > that are sufficiently small.
Now we are ready to show Theorem
Proof of Theorem Fix some E [Mminn and a E 24 n [-1,
L, L]. Let qu and qu, be the constants in
Lemma G.2
We first prove the weak convergence claim in Theorem It). Arbitrarily choose some u > 0 and
C'e (0, 1). It suffices to show that
lim ( > u) 2 + exp ( (- (1 CC)). n00
lim inf Pa (Q. w) 2 - 0) C exp coxp
(1
Recall the definition of the stopping time (i(A) in eq. G.15 Define event
[Tr(kt (17, )) Thth (n, e) S clog(1/e)
Ax E) A X S4g Vn
"1
where 0o is the constant in Lemma G.1 First, since E 24 (Sk-1, sk), it holds for all c 0 small enough that & E (Sk-1 + e,,88 8k +). Next, one can find some > 0 such that
(Due to Lemma G.I
Pa (=((((n.))))) < C. Vn sufficiently small;
(Due to eq.
G.16 eq.
G.17) and strong Markov property) For all r sufficiently small,
Pz (()de(()((()))
((()) (17, c)) > (1 C)u As(T,.) c) $ C + exp ( (1-C)u).
Pa
(andda - (()) (11, ,)) > a
Au(1,)) 2 -C exp ( (- C))+
30

---

Published as a conference paper at ICLR 2022
Fix such Lastly, for this fixed e, due to A RV -V-1-1(--1)(1) and a 1, we have qx)x(n)
clog(l/e) < Cu for all y sufficiently small. In summary, for all sufficiently small, we have
"
Ps (1( u)
<Px
(PA, + P ((90))((7))))))) nf n An(,,e)
SC + P. (9))4 > u) n A4 (7,
=C + P (mAc(()(((()) T (77 <)) (n) Trama (n. Ao(n.c) (P((A(())))
<C + P (o))c())((())) - T (1) s)) -C Au(1)
22C + cxp ( ((1 -()
and
(T) > u)
PP, ((ad)()))w(()) > u} n A (T,,
=P, (ada(()(((())) Tuki (11 <)) (x) T
ream (1.c) Ac(1, (A((A)()
PP (ada(())((())) Teai (17, <))
gu T (17, s)
A (1 (1-C)
2P, (ndd
I (17, <)) As(()))) (1-C)
2(1 (1-00 C) - exp ( (1 + +)++C
so this concludes the proof for Theorem
In order to prove claims in Theorem ii), we first observe that on event (1, e) we must have (n) Tr(k) (17, c). Next, arbitrarily choose some C E (0, 1), and note that it suffices to show that Pa (1) E 24) (11 C) 20.11, + gh.+C) holds for all r sufficiently small. Again, 9k
9x
find e > 0 such that
(Due Lemma G.I
Po ((AAA(n e))9) S C. Vn sufficiently small;
(Due to eq. GG.88 eq. (..99
and strong Markov property) For all r sufficiently small, (Xhem
9k.1
C
Pr X
e 24
Au(n,e)) < 9k.1 + C
In summary, for all r) sufficiently small, we have
Pa x" (n) 22
s Pz ([Ax(n,)))) + +P (1xx E EQI n Au())))
C + Pa
X"
Es Pa(n.e))Pa(Aaan
[k 2 +
< C +
4
and
Pg X2 (x"
E (ll
> P. {x" oz(n) E Eli n Au(,ee)
(1)
x X" ouin)
E Esu Ac(T, Aa(eee))((AAenn..))
0) Thit C
2
qk
and this concludes the proof.
31

---

Published as a conference paper at ICLR 2022
The rest of this section is devoted to the proofs of Lemma G.1 and Lemma G.2 Specifically, Lemma G. is an immediate Corollary of Lemma G.13 the proof of which will provided below. The proof of Lemma G.2 can be found at the end of this section.
G. PROOFS LEMMA G.I
G.2
The following three lemmas will be applied repeatedly throughout this section. The proofs are straightforward but provided in Section K for the sake of completeness.
Lemma G.3. Given two real functions R RR,, b: R+ + R4 such that a(e) 4,0, b(e) ass O, and a family of geometric RVs {U(e)= :e Of with success rate a(e) (namely, (P(U(()) > k) a(e))* for E N). for any c >> 1, there exists Eo > such that for any E E (0)
exp (-
(-c-001)) < & (((())) D(E)
exp (exx(pexx
a(E)
blte)
...b(()) c.b(e)
Lemma G.4. Given real functions R + R4, b: R_ RR_ such that a(E) + 0, & and
a(<)/b(() +0
as 4 0, and a family of geometric RVs (U()): c > 0} with success rate a(e) (namely, ((U(c)
k) = -1--00 =a())) for le E for any c > there exists some eo > such that for any E (0, co).
alc)/(c- b()) S P(U()) 11/(()) S (c- a(c)/b(cc
Lemma G.5. Suppose that function g: E + R (where E is an open set in Rd) is E (F) and |Vg(()) & C on its domain E for some constant C < cc. For a finite integer n, a sequence of vectors {z1,-.., , in Rd, and vectors 2, I E E, > O, consider two sequences {Ix k= and [ k== constructed as follows:
Fo
Ik = Ik-1 + V7Vg((=11) Tp2 Ak 1,2, ,,,
To
3g = - + [V((x=>) Vk- = 1,2......
we have that the line from Fa to Iy is contained in and || + + 2x 31|++11 for all k = 1,2,... for some ē > 0, then
ill S
Fnl ll S - exp(πCk) Ak = =1=12,
To facilitate the analysis below, we introduce some additional notations. First, we will group the noises Zn based on a threshold level 8 >> 0: let us define
2,56.n Zn 1 (1) Zml < 6},
(G.20)
2,6.1 Zn
1 Zm > 8}.
(G.21)
The former are viewed as small noises while the latter will be referred to as large noises or large jumps. Furthermore, for any j 2 define the jth arrival time and size large jumps as
(8) min(n > ( Znl > 8), ()) =0
(G.22)
(5) A Zry (8)
(G.23)
Next, for any e > 0, let (2(e) ) [m, m. + +e] be an e neighbooroood of the local minimum and Se() = [8: 6, 8; + <] be an e neighborroood of the local maximum 8;-
For most part of this section, we will zoom in on one of the local minima and its attraction field Ω = (s;-1, Without loss of generality, we assume m, = 0, and denote the attraction field as Ω- 34), (If m, happens to be the local minimum at the left or right boundary, then the attraction field is [-L, or (s_, L] where the SGD iterates will be reflected at +L.) Henceforth we will drop the dependency notation i when referring this specific attraction field until the very end of this section. Throughout the proof, the following (deterministic) dynamic systems will be used frequently as benchmark processes indicate the most likely location of the SGD iterates. Specifically, given
32

---

Published as a conference paper at ICLR 2022
any z S2, use (x) to indicate that the starting point is 2, namely Xu(x) x. Similarly,
consider the following ODE (t; as
x"; x)) = 2;; dx x) -ns' -ns((x((1:at) (x)():
(G.24)
(G.25)
dt
When we use update rate = 1, we will drop the dependency of ry and simply use x(t; to denote the process.
Based on Assumption 3, we know the existence of some constant E co) (note that eg is the
constant in eq. (G.8)) such that
r -े4 min s_, 8+h,
(G.26)
- [r/b],
(G.27)
(I" 1)6 + 1 < 100
(G.28)
r+ 1 < l'b 100
(G.29)
Here r can be understood as the effective radius of the said attraction field, Also, we fix such é small -f enough so that (Iet chC = - =
we have
0.9c - -f() 6 1.1ch Va E [-L, - + 100E
(G.30)
0.9c4 2 (x) 2 11ck Vo E [L 100E, L).
(G.31)
Similar the definition of ODE x", let consider the following construction of ODE X that can be understood as x" perturbed by l* shocks. Specifically, consider sequence or real numbers to- and real numbers " Wr- where (wyl S b for each j. Let t t [f1. ... to ). w = (W1. wr-) Based on these two sequences and rate rj > 0, define x (t;
0 t 12 t3
as
(0, x; w) pr + (p((w)));
(G.32)
dx x;; w) -n (1, w)) 1A & {4, ,,,,,,,,,, + (ww)) Vj ... 1
...
(G.33)
dl
x" (ty, x; w) P2
(G.34)
Again, when 1 we drop the notational dependency on 17 and use x to denote the process. Now (G.29) one can easily see the following fact: there exist constants f, 5 > 0 such that x(ti-, t, w (note that the starting point O, the local minimum) only (under the condition
from eq. G.28 eq.
that [wyl < Vj)
t 1 t-- S E
Vj 2,3, 10 8 Vj= 1,2, 1.".
(G.35)
(G.36)
The intuition is as follows: if the inter-arrival time between any of the 1" jumps is too long, then the path of =t;x))) will drift back the local minimum m, that the remaining (" shocks (whose sizes are bounded by b) cannot overcome the radius r which is strictly larger than ((* - 1)6; similarly, size of any of the shocks too small, then since all other jumps have sizes bounded by b, the shock created by the 12 jumps will be smaller than (1* 1)6 + 1002, which is strictly less than r. We fix these constants T, 8 throughout the analysis, and stress again that their values are dictated by the geometry of the function f. thus do not vary with the accuracy parameters and à mentioned earlier. In particular, choose such that 5 <.
In our analysis below, > 0 will be variable representing the level of accuracy in analysis. For instance, for small e, the chance that SGD iterates will visit somewhere that is e-close to s or s+ (namely, the boundary of the attraction filed) should be small. Consider some eg) where 6o is the constant in Assumption Due to eq. (G.8)eq. (G.9. one can see the existence of some
9o 0, < such that
|f'()) 2 go for any x E Ω such that lx 8- 6n, 60,1|x 84 > 60:
33

---

Published as a conference paper at ICLR 2022
Let (ove(x:)) 11) - minft 0: xt) [-c, be the time that the ODE returns to 6 --eeihhborhood of local minimum Ω when starting from æ. As proved in Lemma 3.5 of Pavlyukevich (2005), for any E S2 such that [x s_ > e, 1 s+ > we have
(ZDE 7) $ c1 log(1/c)
(G.37)
and we define the function
i(e) A ei log(1/e).
(G.38)
In short, given any accuracy level e, the results above give us an upper bound for how fast the ODE would return neighborhood of the local minimum, if the starting point is not too close to the boundary of this attraction field f2
For the first few technical results established below, show that, without large jumps, the SGD iterates (z) are unlikely to show significant deviation from the deterministic gradient descent process y (x) defined as
x0(x) = 2;
(G.39)
(x) = - - -n (x-
(G.40)
We are ready to state the first lemma, where we bound the distance between the gradient descent iterates y (y) and the ODE x" (t,, x) when the initial conditions x: are close enough.
Lemma G.6. The following claim holds for all n > for any > we have
sup <E(0..]
(s)
[u, & (2nM + |x vl) exp(nMt)
where M E (0, 00) is the constant in eq. G.10
Proof.
Define a continuous time process (s; y(z) (v). and note that
(x, = xxx,)) 1" (u, x))
[.0
x)([s] = 7
fx,
(1) - y-({s,)))) = y - n m 1/10 f' d).
[9]
Therefore, if we define function
b(u) = (", x) - (", y),
from the fact [f((()) < M, one see that [b(u) < nM + |r yl for any u E 1) and
[b(1] < 27M + |x yl. case that s 1, from the display above and the fact ())(() < M, we
now have
(x)((x) xs),))) - [b[[=])] + nM;
[b([s)) = nM J. [+]
[b(u)ldu.
From Gronwall's incquality (see Theorem 68, Chapter V Protter (0005), where we let function (u)) be a(u) = [b(u 1)]), we have
(x)(()) (x) (s( < (2nM |x v/) exp(nMt).
This concludes the proof.
34

---

Published as a conference paper at ICLR 2022
Now consider extension of the previous Lemma in the following sense: add perturbations to the gradient descent process and ODE, and show that, when both perturbed by 1" similar perturbations, the ODE and gradient descent process should still stay close enough. Analogous to the definition of the perturbed ODE 3" in eq. (G.32 -eq. (G.34). we can construct a process Y as a perturbed gradient descent process as follows. For a sequence of integers 0 = t < t2 < ... < tt- (let
= (tz))21) and sequence of real numbers W1.-.. tht- (let (1,) and y E I, define (for all n 1,2, (2)) the perturbed gradient descent iterates with gradient clipping at and reflection 7
(x: = PL (y%
(y: (yn-1 + 1po( - n1 (5 W)) +
(G.41)
with initial condition (o W) (p. + (pb(ūi)).
Corollary G.7. Given any > O, the following claim holds for all sufficiently small y > for any 2., y Ω2 and sequence of integers (4,)²= and any two sequence of real numbers (w] 13= w (WI)iz) such that
x yl <;
t O. and tj 1--1 S 22/n all j 2 where is the constant in eq. GG.35
wy أyو < for all j 1;
then we have
(t; - F1e (wit, De 1E(0,;-+
sup
where the constant (3 exp(nMt) 3)
Proof. Throughout this proof, fix some E (0, c/2M). We will show that for any r in the range the claim would hold.
First, on interval [0, t2), from Lemma G.6 one can see that (since 2M < c)
sup (t, Vie (ynt, W) < 3xxp(n, e. rE(0.t2)
The at / = 12, by considering the difference between w2 and W2, and the possible change due to one more gradient descent step (which bounded by nM E), have
sup te
Ix (t, w) - Y" (y: W) < (3 (3exp(nM + 2) -
Now we proceed inductively. For any j 3, /* assume that sup (1, x;; w) 3ly (v: w) < (3 (3xxp()MT) + 3)--1
tE(o.ty)
Then by focusing interval [t,, 1++1 and using Lemma G.6
show that X (t, w) 3 (y: W) - 2e ((3xxp(nM)) + +33-1 + 1) exp(n)Mt)
again,
sup
retts
< (3exp(n + +3
This concludes the proof.
In the next few results, show that the same be said for gradient descent iterates f,, and the SGD iterates X,, Specifically, our first goal is to show that before any large jump (see the definition in G.2Ip, it is unlikely that the gradient descent process yn would deviate too far from X Define the event
17, e, 12 (
max
7 + Za sc}
(G.42)
35

---

Published as a conference paper at ICLR 2022
and recall that arrival times (6) are defined in eq. (G.22
As a building block, we first study the case when the starting point a is close to the reflection boundary -L. The takeaway from the next result is that the reflection operator hardly comes into play, since the SGD iterates would most likely quickly move to somewhere far enough from L:; besides, throughout this procedure the SGD iterates would most likely stay pretty close the corresponding deterministic gradient descent process.
Lemma G.8. Given e (0, it holds for any sufficiently small e, 6, 1j > 0 that, if E ILLL, -L- +
and po(1 yi 96) < E, then on event A(n, 7), c, we have
(x) vv) & Po (|x - yl +c) Vk = 2, n (T) 1) A escupe (z)
where Tny Teecaee (x) m 2 0: (x) - L + and po = exp
2M 0.9c
is constant that does
not vary with our choice of e, 5,
Proof. For any & < (d), know that Zn - (thus 1/ Zxl 8). Also, recall that [/'(x)) < M for any x E 9z Therefore, as long as 1) and 5 are small enough, we will have that
(G.43)
so the gradient clipping operator in eq. G.12 has effect when k (6), and in fact the only possible time for the gradient clipping trick to work is at (0). Therefore, we can safely rewrite the SGD update as
(x) -
72 + R 4k (6)
where each Rs 2 0 and it represents the push caused by reflection at L.
First, choose small enough so that 9e Next, based on eq. (G.3I) we have the following lower bound:
x)(x) 2 x 0.9c e Vk < (5).
Let E) A min(n 2 0: (X - 2e}.
Due to the inequality above, know that
3e
(x c)
(G.44)
0.9c4.
One the other hand, given the current choice of if we choose n and 5 small enough, then using the same argument leading to eq. (G.43), we will have
x (a,c)
(I) s - + 2.1e 2 + 2.1e
tz 2 1 (namely x < L 2e).
Let us inspect the two scenarios separately. First, assume (xx, 2 1. For the deterministic gradient descent process y (v), we have the following bounds:
y 0.9c (v) 1.1c Vk S ( 1).
This gives us
(x(()) [1) < Jr vl +0.2c 2.16 VK tola, (TT
At time 1: = (z c), due to previous bound on (z c), we know that X
fo(2,e) (x) y% (or.e) (v) s
|x |x=y/ e/-y/+7e, nf (0) 1) < tu (z, <) then we have already shown the desired claim. Otherwise, starting from time (z, c), due to the definition of event A A(n, c, 5) in eq. (G.42). we know that the SGD iterates (x) will not touch the boundary -L afterwards. Therefore, by directly applying Lemma G.5 and notice that (x)( < M for any E [LL, L] and, we have
2M
[x(()) vx(1) < (|r - + 9c)
exp
Wk 5 (TT()) 1) A TTreupe
0.9c
36

---

Published as a conference paper at ICLR 2022
Indeed, it suffices to use Lemma G.5 for the next [22/ (0.9gc"]) steps show that (x should be smaller than e for the next [22 (0.99c )1 steps, while yV((v) will reach some where in (-L+ + 2E, -L +3)) 32) within [2E((0. ) steps so we must have
n A (T 1) A ( (T) A fo(T, E) to(x,,) e) S 25/
(G,45)
Lastly, in the case (a <) 0 (which means x: 2 - L + <), use Lemma did above and establish the same bound. This concludes the proof.
G.5 directly
Obviously, a similar result can be shown if x in the rightmost attraction field (8 L. and the approach is identical. We omit the details here. In the next Lemma, we consider the scenario where the starting point x is far enough from the boundaries.
Lemma G.9. Given any > the following holds for all sufficiently small 0: for any x. y E S and positive integer n such that |oc L| > 2c, |x + L > 2c, x s_1 > 2c, fr 8+1 > 2c and
|xc y <
TexpinMnn on event
A(n. 2 2cxp(nMm) 6) (2,
Vj = 1,2, ,M (T) - 11}
we have
( - -x) S Vm = 1.2, (8) 1).
Proof. For sufficiently small 77, we will have that the (deterministic) gradient descent iterates |y:] is monotonically decreasing in n, which ensures that y" always stays in the range that least c- c-away from +L or s s_, 84. We now show that the claim holds for any such 17.
On event { (1)(( 2. 1x7(()) E Ω Vj 1.2, A (T7(8) - 11}, we are able to apply Lemma G.5 inductively for any m [n] and obtain that
(((0)-x((x)) = (|æ vl + 2 2exp(namm) )xp(nnM) < = 11,, 1,,,...,,
and conclude the proof. The reason to apply the Lemma inductively for m 1,2, n, instead of directly at step n, is to ensure that SGD iterates X would not hit the boundary +L (so the reflection operator would not come into play on the time interval we are currently interested in), thus ensuring that Lemma G.5 is applicable.
Similar to the extension from Lemma G.6 to Corollary G.7 we can extend Lemma G.9 to show that, if we consider the gradient descent process that is only perturbed by large noises, then it should stay pretty close to the SGD iterates Xg. To be specific, let
Y (x) = x
(G.46)
(x) = PL Y (z) nf (z))
(G.47)
j21
be a gradient descent process (with gradient clipping at threshold b) that only shocked by large noises in (Zn)x21- The next corollary can be shown by an approach that is identical to Corollary G.7 (namely, inductively repeating Lemma G.9 at each jump time) so we omit the details here.
Corollary G.10. Given any e > the following holds for any sufficiently small v > 0: For any |x|| < 2e, on event Ao(E, n, 5) n Bo(e, n1 8), we have
(Y, - (7(x))x((<)) 7e Vn 1n=
where
An(e, n1, 2 { [ Vi 1,... 1' j-M (6) (5)+1.-- (11-11
max
2T (88+1 + +... 2215 2exp(2iM)
Bole, 11, 8) A Vjj 22 2,
()) 20.,0
< 21/1}
and p E (0, 00) is a constant that does not vary with n1 5,
37

---

Published as a conference paper at ICLR 2022
The next two results shows that the type of events A(n, 17, c. 6) defined in eq. GG.4 indeed very likely to occur, especially for small e. For clarity of the presentation, we introduce the following definitions that are slightly more general than the small and large jumps defined in eq. (.22 G.22 (for any c > 0)
T 7,. S c)
- 7, Z > c).
Lemma G.11. Define functions u(n) = 6/1/--, v(n)
enj² with e, 0. If real numbers
A, Å, e, ô and positive integers j, are such that the following conditions hold:
AE [0, a "
(G.48)
B (L, (2 24) A a(l A)),
(G.49)
A e [0, A
| Î << a(1 -) - B, N< (a(1 A) - $)
(G.50)
2
(G.51)
v(n) - jnn(n) 2 v21)/2 for all sufficiently small,
(G.52)
then
r/Zjsun) + zz 30(1)) o(1)"²)
PP
max k=1,2,- [1/n²]
as 17 O.
Proof.
From the stated range of the parameters, we know that
a - A) > 8,
(a(1 -) - B) N,
able to find y (0, 1) small enough such that
a(1 - A) 1 27) > 3, (a(1 (a(1-A((1 - 24) G)j > N.
(G.53)
(G.54)
Fix such 7 E (0, 1) for the rest of the proof, and let n(n) - [(1/1) a #6i E [n(n)]
1 > u(()) (-) Then
(
P
2 <u(n)
+... _u(oo)
v(n)
n(n)
1-1
su(n)
+ - n(n)
Su(n))
v(1), 1=1) + +P ( + ... + (,(0)
>01), 12
40
Note that since 2,5u(z) < u(n).
(1) < (n(1)) P p ((zg5u() +...+ zSu(n)
v(n) imp((n) (2,5 s u(n) vi [n(n) :i)
n(1)-=1
< <n(1)² P zu((())* +... zSu(n) n(n)-1
7
v(1) impu(T))
"
<n(n)² ( +
ZS(nn)
v(v)
(G.55)
2m
where the last inequality follows from eq.
G.52
First, since E. = 0, we have
EZZ
P(
> r)dr E RV (o-((()(1))) (1)))))))))))))
38

---

Published as a conference paper at ICLR 2022
Therefore, for all 17 > 0 that sufficiently small,
[BZ + .... +
n(n) -
In(n) 1 1--10111- 2)(1-21) < 2n (10-))(11)(1-29))
<(1/1) (1) due to eq. G.53
((1))
due to K/2 I A in G.50 and 1 - <
4n
we let Y, = Zn
_u(n)) EZ,
Su(n) and plug the bound above back into eq. (G.55 then (for
all 7 > 0 that are sufficiently small)
v(n)
(1) S n(1)) P(VV +
1n(n)-13
47
1/17 2
2-23
S <n(1)' exp
16
(G.56)
2(n(1) - i)
02-
(1/1) 8/12-3
where the last inequality is obtained from Bernstein's inequality. Note that from Karamata's theorem,
B = var (z (35(())²
S E Su(n))
u(T))²
2x 2xP([Zi) x)dr E RV RV---AVa--12--
Now note that
In case that a < 2, for all > that are sufficiently small, we have (using eq. G.50
2(n (1) - i) 12 < ((2-))+(2--)(--22 (111)2(1-2)
1/1 23
2(((1)) - i) 2 N 1/1
In case that for all that are sufficiently small,
2 2(n(T) 2 E| 1/7
and we know that 3 4
2 due to 2 - 3 2A and 2 s (see eq. G.48
eq. (G550);
Since y > ( and 2 < A, we know that
(1- Á) 2..
Therefore, is easy to see that the R.H.S. of eq. hence o(17*) N,, On the other hand,
G.56 decays at geometric rate as y) tends to zero,
(n(n)) P (17 Su(n)) > u(1)) -I..........
(II) < P(I
< <n(n)' P (1, Su(n) > u(n))--)²
which regularly varying w.r.t 1) with index (a(1 7) A) - -B Therefore, for all v sufficiently small,
(II < ( 20 < NM due to eq. G.55
Collecting results above, we have established that
+
Sm(n) +... [ > v(n) = N)
The conclusion of the lemma follows from Etemadi's theorem.
39

---

Published as a conference paper at ICLR 2022
Now consider the following setting. Let fix some positive integer N and 3 (1,2 A a). Then can find some positive integer j such that (a - B)j > N. Meanwhile, given any e > O, we will have
c- jō 2 c/2 for all 8 > 0 sufficiently small. Therefore, by applying Lemma G.11 with A = A 0 (hence u(n) = 6/m, v(n) = and B. N, c, 6 as described here, we immediately get the following result.
Lemma G.12. Given any B E (1, 2), 0, and N > O, the following holds for any sufficiently
small ó > 0:
1| + +... 23 256171 > >c =
max j=12,m,10/m 111/))
as 17 O.
Using results and arguments above, we are able to illustrate the typical behavior of the SGD iterates X in the following two scenarios. First, show that, when starting from most parts in the attraction field Ω the SGD iterates X will most likely return to the neighborhood of the local minimum within short period of time without exiting Ω. Given that there are only finitely many attraction fields on f. it is easy to see that the key technical tool Lemma G.] follows immediately from the next result.
Lemma G.13. For sufficiently small > 0, the following claim holds:
lim
sup
Pa X E E En S T Tremon(1) (1) annddd T Treturn (11, 6) S p(t)/n) =
where the stopping time involved defined as
Treumn (1), E) 2 min(n 0: 20: (x)) 2- 2c]}
the function i(e) is defined in eq. G.38), and the function p(-) is defined as p(t)
0.9c + 2i(e)
3
Proof. Throughout, we only consider small enough so that Lemma G.8 could hold. Also, fix some N 0, A. E (0, a - and a1, Let o (xx, 7)) min(n 2 0 $ ②}.
Without loss of generality, assume Ω = 84) and x < 0 (so reflection - L) is possibility, Any other case can be addressed similarly as shown below.
From Lemma
G.4 and the regular varying nature of H(-), we have, for any e. > 0.
( S p(c)/)) s sp --1-.
(G.57)
for any sufficiently small 17-
Let T excape (ar)
be the stopping time defined in Lemma
G.8
From eq. G.44
G.45 eq.
G.57
and
Lemma G.12 we know that
sup
TP Tin
(x)) co n), Telcape (x) 5
36
and
X Tam
(x)
7, - - +2]
0.9c
>1 N 0
(G.58)
for all sufficiently small 1).
Next, we focus on x E Ω such that |x- ----s-MM - s+1 1 > and x -L We start by considering the time it took for the (deterministic) gradient descent process y (1:) to return to [-1.5e, 1.5e]. From the definition of E(c) in eq. (G.38) and Lemma G.6 we know that for 7 small enough such that exp (2M((c)( < 0.5c, we have
minfn 2 0: y. |- 1.5c, 1.5c]) < 26(c)/n-
Now consider event A([(L/n)*1 4exp(2M 5) (see definition in eq. G.42 From Lemma G.12 we know that for any sufficiently small 6, we have
PP (((AA((1/))1). ccxp(2Mt(c)) 0))*) =(1
(G.59)
40

---

Published as a conference paper at ICLR 2022
Combining this result with eq. (G.57 eq. G.59 and Lemma G.9 get
sup xEf: 12 111-- L+7 2 1
Px (1) e) < ox, 11. T Teaum (1, S 20())/n)
(G.60)
(G.61)
for any sufficiently small To conclude the proof, we only to combine strong Markov property (at T scppp) with bounds in eq. eq.. (G.61).
In the next result, we show that, once entering a e e-small neighborhood of the local minimum, the SGD iterates will most likely stay there until the next large jump.
Lemma G.14. Given No > 0, the following claim holds for any e, ô > 0 that are sufficiently small:
P(3n < (6) s.t. Xx(x > 36 = o(17
sup x x 2c,
as 17 0.
Proof. Fix small enough such that 3s < (see Assumption for the constant en). Also, fix some Aa E (0, 1),8 (1, N a + Da + No. Due Lemma G.12 2 for any sufficiently small, will have
P
max j=122,--(0//)')
1 +
25.0
exp(2M)
)
(G.62)
Fix such 8 > O. We now show that the desired claim is true for the chosen 6.
First of all, from Lemma G.3 we know the existence of some >> 0 such that
(8) > 1/1 ((exp(-1/10)
(G.63)
Next, let us zoom in on the first [(1/))91 SGD iterates. For any small enough, we will have (x) E [-2e, for any 2 1 and y /m (x) [-e,e] given [-2e, 2e]. From now on
only consider such 7). Due Lemma G.9 we know that on event maxx==12. [(/m)) n + 25 exp(2MM) }. we have
[x(()) S 3E Vn [1/1 A (17
and on event { max [(1/m) have (8) (z) 2e, 2e]. Now by repeating the same argument inductively for [1/ times, we can show that on event
1 .... 2,5
exp(2M] n (TT > [(1/n)*1,, we
7,5. we have (x)) S 3e Vn < 1/yatta A (T) - 1). To conclude the proof, we only need to combine this fact with eq. G.62..
1
2a ((1m))))1 (T(1/))))11
7/
(1/m))1+5)
cxp(2MM
We introduce a few concepts that will be crucial in the analysis below. Recall the definition of perturbed ODE x in eq. (G. (G.34) (note that we will drop the notational dependency on learning rate y when we choose = 1). Consider the definition of the following two mappings from where
(Wi, wi-) a sequence of real numbers and (11, .., ,..)) with
0 = f < < tg < ....
h(w,t) = x(11,, (,t,
Next, define sets (for any E (-7,7)
E(e) (w.t) x F
h(w, [s_ c, +c]..
(G.64)
We add few remarks about the two types of sets defined above.
41

---

Published as a conference paper at ICLR 2022
Intuitively speaking, E(c) contains all the perturbations (with times and sizes) that send the ODE out the current attraction field (allowing for some error with size c);
From the definition of 8 in eq. G.3 GG.36) and Corollary G.7 one can easily see that for fixed E (-E,E),
w. t) E E(e) |w5| > 8,1, - 1--1 s
Lastly, E(e) are open sets due to f C².
Use Leb.; to denote the Lebesgue measure restricted on [0, oo), and define (Borel) measure Va with density on (0):
ap+ ga+1 1fx < 0} |
(dx) = 1xx 0}
where a > 1 is the regular variation index for the distribution of Z1 and p-, p+ E (0, are constants Assumption 2. Now we can define a Borel measure y on IR!", x R4
1'-1
as product measure
4= (va) (Ieb.."''1
(G.65)
Due to remarks above, one can see that for e E-E,E), we have u(E(e)) < co. We are now ready to analyze specific type of noise Zn.
Definition G.1. For any n 2 1 and any e (-E,E), (0,6A 0, we say that the jump Zn has (e, 6, n)-overflow
T/Zn| > d;
In the set {n + 1,... + 2[("tm/]) }, there are least (1* 1) elements (ordered as n < t2 < t3 < ... < tạ-) such that 1) Zr,1 > 8 for any i = 2....... ,,";
Lel 4 = and 11 = 1i - li-1 for any i = 2, ... 1", w; = T1 for any i = 1,... , , real sequence w = (wi, W2, .. wr- and a sequence of positive numbert i = (11(4-1) - -n)) we have
(w,r) E E(e).
Moreover, if Zn has (c, 5, overflow, then we call h(w,t) as (c, 1) -overfow endpoint.
Due to the iid nature of (Zj)s21, let us consider iid sequence (Vj)z20 where the sequence has the same law of Z1- Note that for any fixed n 2 1, the probability that Z, has (c. 6, n)) overflow is equal to the probability that Vo has(e, 8, -)voerftoow. More specifically, know that P(n/Vol > d) (8/n)) and now we focus on conditional probability admitting the following form:
ple, 6, 11) IP (Vo has (e, -overflow
1/ Vol > 8).
(G.66)
For any open interval A (a1, a2) such that A n [s_ + č, = 0, we also define
p(c, n7 A) P Vo has (c, 5, overfflw and the endpoint is in A n/Vol > =).
(G.67)
Lemma G.15. For any (-2,6), E (0, 8), and any open interval A (a1, a2) such that [01] A |a2) > r - and lail # L, 1,2a # L, we have
p(E, 6, mE A) lim nLo
(B(()nn
8a
H(1/m)
where μ the measure defined in eg. eq. G.67
G.65), and 5 A) the conditional probability defined in
Proof. Let us start by fixing some notations. Let Ti O, and define stopping times T min(n > Ty-1 1/ V.l > 8) and antev-arrival times T T T for any 1, and jump W, Vr, for any 2 O. Note that: first of all, the pair (T), W.) is independent of (T), W,) whenever # besides, W and T3 are independent for all j > 1.
42

---

Published as a conference paper at ICLR 2022
Define the following sequence (of random elements) (W1, wp- and t (t1,-.. .tt.) by
W , lj n,T,.
If Vo has (e, 6, -overflo,,, then the following two events must occur:
Tj s 28/n for any j = , 1
m|W;] > 8 for any j 1,
(w.t) (())
Therefore, for sufficiently small 1,, now have
p(c, 8,
= (P(T) s 22/n)) 1- / /I((W.)) E(())
P(n duy In Wil ......(nnW-- = dwr- du1-1n/Wi-13 > 8)
P(nTT = dralnT3s s 28) ... P(nTT = dt/- I1 S 28)
= =(P(T) s 25/m)
Qn (E(c)
(G.68)
1--1 induced by sequence
where Qn is the Beesa-mmas probability measure on TR'" x (R... of independent random variables (WI 6). ,,f (n. 8), I. (n, 8), T/. (n. 8)) such that
For any i 1, ,,", the distribution of W (n, 8) follows from P (nWi E /////11 > 6):
For any i = =, ... ,," the distribution of T (n, 6) follows from IP [I[i E
|TT S 28
(1, e.) = = (((())))P( ".a.,,),............ ".. (n, 6), nI ... 2
Now we study the weak convergence of W!, T
Due to the regularly varying nature of distribution of Zi (hence for Wi), we know that: for any x > 5,
ga lim P (nW2 > x n/Ww-1 26) = p+ 720 therefore, w/ (1. weakly converges to (randomly signed) Pareto RV that admits the density
8a
lim 900
n/ < <- n/Wh-1 6) p-
za
xa
a²a
asa
va,(ddr) 1fx 0)p+ gatt + 1fx 0}p-
4 0:
For any [0, 2t), since lim 10tpg/oo/m/m101 (b/ it is easy to show that
(1 H(5/n)))L///l 1;
lim
n40
[x/n]] ((//n)
therefore, we have (for any E (0,)
1 1- (1 (1- H(5/m))
P(TTi S x n < 28)
1- (1-H()/)) (25mr]
2
77 4 O. which implies that T converges weakly to uniform RV on 27
43

---

Published as a conference paper at ICLR 2022
Let us denote the weak limit of measure Q as M8,2i- In the discussion before the Lemma have
shown that, for any (w,t) t) E(e) (with 8 E (0, 8)), we have |my| 2 å and [t I 2f; since require
5 < 6, by definition of measures μu and M4822 we have
8at"
26.2 (E(e( = (28)1--1
For simplicity of notations, let E(c, A) = E(c) n h (A). By definition of the set E(c), we have (recall that A is an open interval (a1, 02) that does not overlap with [s_ + 8, - 2)
()e, h ((-00, u) (st +,, h ((a1.
h ((-00,, (s+ +c, n (2) = h (F(E, (2)
where F(c, (1,02) è ((-o0, u (s_ +c, (a1, 02). Meanwhile, it is easy to see that a continuous mapping, hence
(w,t) E (E((, -)= h(w,t) E (8- +,, e, a, a2))
Fix some with # =L, [s] > (1" - +). For any fixed real numbers t2. f1- W1. w s, then since X(4- [s b, 8 b), due to Assumption (in particular, there is no point on this interval with If [x)) co
if h(w1, Wt. ... -t,)
0; wi.... wt 22. t1- -1, t)
). there exists most one possible t that makes h(w,,.... ".......
wi-; b2, 14--1, s. Therefore,
let Wi be iid RVs from law Va,5 defined above, and (T) 1y22 be iid RVs from Unif(0, 2t), To
T Eド=2T Ti. By conditioning all W3 and all T " T we must have
PP (MWWT W.. T5
TTt) W dwn," ... Wi. dwr, T dd2,"" " TT- ddl1-11)
(G.69)
which implies
P (((WWi, ... WW. ... TT+) = =0
hence
p ((E((c,
By Portmanteau theorem (see Theorem 2.1 of Billingsley (203)) we have
lim 12 E(e, A)) M4.25 (Ec, A)) 700
Collecting the results we have and using eq. G.68 we can see that
p(e, 77, A)
lim 1/0
HH(1/)) 71
8a
(2t) 1" ple,, (1
88
P(T S 2t/m)
lim sup n00
sup
(2
H(1/n)/u
fal- (P(T 5 2F/T))
(2F) 11 .ple, 8, mi 1-1
8
P(T S 25/n) 1-11
< liimsup nJ0
lim sup n0o
jal. (P(T) < 2i/n)
2
H(L/n)/n
8a
P(T < 21/m)) 1-1
Sy(F(ee, A)) Iimm vio
2f
H(1nn)/
Fix some x > From Lemma G.4 and the regularly varying nature of function H, we get
80
P(T S 21/m)
8a (. 1-1
limsup
< K
lim sup
H(1/m)n = 1---1
27
H(1/n)/n
2F
44

---

Published as a conference paper at ICLR 2022
Due to the arbitrariness of K >> 1, we have established that
plc, 8, 77
lim sup 840
S u(E(e))
1-1 8"
H(1/m)
The lower bound can be shown by an argument symmetric the one for upper bound.
The following result is an immediate corollary of Lemma G.15 Corollary G.16. For any e e (-2,2), (0, b A 8), we have
p(c, m)
lim n40
1-1 = ((((c)) 8 H(110)
where μ is the measure defined in eq. G.65 and p(:,:))) :) is the conditional probability defined in eq. (G.66).
Define the following stopping times:
(n) min (n 0: X $ 2);
(G.70)
R(e, 6, min (n (6) : X E 1- 2e|}
(G.71)
indicate the time that the iterates escape the current attraction field, while R denotes the time the SGD iterates return small neighborhood of the local minimum after first exit from this small neighborhood. In the next few results, we study the probability of several atypical scenarios when SGD iterates make attempts to escape f2 or return to local minimum after the attempt fails. First, we show that, when starting from the local minimum, it is very unlikely to escape with less than /' big jumps.
Lemma G.17. Given e (0, N > O, the following claim holds for any sufficiently small 8 > 0:
Pr (a(()) R),,,), (n) (T) )(1²²)
sup (c]
as 17 O.
Proof. Based on the given e fix some 4nxp(2MM(e))
Recall the definition of f(e) in
eq. G.38.
First, using Lemma G.14 know that for sufficiently small 6, we have
sup =c]_2e,2c]
P(Ai ((A((c...n) o(n²²)
(G.72)
where
A7 (c. (,6, = = {3m < (T) s.l. [ 36 }
Define event
A 6,)) 2 {3j 2,. 2....
1/ -لك 16)+
+256n
+A >e]
s.l.
max k= (6) (5)
(8)
From Lemma G.12 we know that for sufficiently small 8 >
P P(A (E) N).
(G.73)
From now we only consider such ó that eq. G.72 eq. G.73 hold.
On event (AT u UAA n (a(()) < R(e, ( ((()n) > ())), we must have o(n) > (6) and
A)()) - (T) < 2f(c)/n
45

---

Published as a conference paper at ICLR 2022
Otherwise, due to Lemma G.6 and G.9 we know that step i (6) [t(c)/nl. we have
x < 2e, and XX S E Vn 2
for any sufficiently small 7). By repeating this argument inductively, we obtain the following result: define
J = min ..2,....:.0))) E
(0))),
then event (Af UA n ((((n) < R(E,n), o(1) < Tp (6)}. must have
a(() < (())/n Vj 2, ... ,J
(G.74)
Furthermore, using this bound and Lemma G.9 know that on event (A) u A n {o(n) R(c,n). (n)) T (6)).
X 11)66 $ 1 (6)
+
all = 2,3, 1,
xxx()) $ X ry (8)
+e+E
However, this implies
[Xm(m) S (1*(E+))++ (1" 1)b <r
and contradicts the definition of (1)) In summary,
sup Po (o(()) R((e,)) d(n) < (0) < mA o(7 =c]-2e.2e]
The following two results follow immediately from the proof above, especially the inductive argument leading to bound eq. (G.74). and we state them without repeating the deatils of the proof.
Corollary G.18. Given E (0, (0. O, the following claim holds for any sufficiently small 5 > 0:
sup Po #E| 2c]
(T <)))) A R(E, 7), and 3j 2.3, ,1' s.t. (8 177,1(6) 26(e)/n)
N)
O.
Corollary G.19. Given e E (0, E) > 0, the following claim holds for any sufficiently small 6 > 0:
P. ((e,,)) < (6) A R(E,n) (T) > 2(r((e)/n) =o(1²'
sup =e[-2c.2e]
as 7 O.
In the next result, we show that, if the inter-arrival time between some large jumps are too long, or some large jumps are still not large enough, then it is very unlikely that the SGD iterates could escape the time of -th large jump (or even get close enough to the boundary of the attraction field).
Lemma G.20. Given e (0, ε) and any N the following holds for all 8 that are sufficiently small:
(a (c, = =(T)²)
sup 2 2c,-2c,2]
where
(T7()))))) < d(n) A R(c,n)) n ( 2,3, s.t. (T) - (5) i//
or = = 1,2, ... s.l. (0W, <8} n n1 2r-
46

---

Published as a conference paper at ICLR 2022
Proof. Let Añ A2 be the events defined in the proof of Lemma G.17 Based the given c <>> 0, fix
some 4exp(2Mt(e))
Let J min = 2.3.... - (T))) f/n). On event A (c, 6, () 'nf
}*) from Lemma G.6 and G.9 and the definition of constant t, we know that
X XIz(0) 5 S|x,, (6) + +b+e++ for all - 2,3, J 1;
X x1y()) - 2c
X < - Vn - (6)
Now starting from step T7 (6), by using Lemma G.6 and G.9 again one can see that
( T7(
$ | Ty + beb j J + 1, ... ,
Combining these results, we have that XX 1p <r- E on event
ntss
}.
Next, define J' = min(j = 1,2,."..'lWW < 83. Similarly, on event (Af(e, A (E,
11),, using Lemma G.6 and G.9 again one can see that
X77700) S 11x7, ( + b + é for all j = 1,2, ,1,,,, J 17730) S |
Ty (6)
+ + e for all = 1,2, ... ..,jj
Since 5 E (0, we have | < - 7 on this event.
In summary, the following bound
sup Pr (B < (PAA(( = =(7)) #E[ 2e.22)
holds for any that is sufficiently small, which is established in Lemma G.17 This conclude the proof.
In the next lemma, we show that, starting from the local minimum, is unlikely that the SGD iterates will be right at the boundary of the attraction field after 1* large jumps. Recall that there are attraction fields on f. and excluding 8g DC, 8 co the remaining points 81, ... 8 are the boundaries of the attraction fields.
Lemma G.21. There exists function v1()) : E + Rt satisfying limeeo (E) 0 such that the following claim folds. Given E (0,e/(37 37 + 9)). we have
suPzel-2..20 Ps (B (,, 587(((
limsup 7/0
(H(L/n)/n)
for all i sufficiently small, where P and p are the constants defined in Corollary G.7 and G.10 and the event is defined as
B% 5,
=(T)(()) S ((()) A Mlem n 3K E [Mimin 1] such that X72 (8) E [8x 8k + e
47

---

Published as a conference paper at ICLR 2022
Proof. Let Añ A2 be the events defined in the proof of Lemma G.17 Based the given 0, fix some 4exp(2M Fix some N > al".
Choose å small enough so that claim in Lemma G.20 holds for the prescribed. Using the same arguments in Lemma G.20, we have the following inclusion of events:
B (c, (,, n (
vivi 2, .. ,,,,,,,,,,,. 17 - -)-1((0)) - t/m n { 1,2,3, .. ,",,", n/) > 8
Therefore, on event B3 (c, 1) (c, u 11) we can apply Corollary G.7 and G.10 and conclude that (0) has (-7, 11) overflow, and its (-č, 6, overrlow endpoint lies
(s 3 + + - 3)c)
for some [mmin- 1). Using Lemma G.15 and Corollary G.16 we have that (for any sufficiently small 17)
P ( (e,
<8n H(1 (
Tham (E((-nn ((5L -
8K + E))
where 3(p+
+ 7 3)c. Besides, established in the proof of Lemma G.17 we have
o(1²*)
for all sufficiently small 6. In conclusion, we only need choose
(e(e)
Σ ((B(-))) 3(p ++ 3(0 p 3)++)
To conclude the proof, just note that by combining the continuity of measure with the conditional probability argument leading to eq. (G.69) we can show that lim, (T()) =0
Lastly, we establish the lower bound for the probability of the most likely way for SGD iterates to exit the current attraction field: making I* large jumps in a relatively short period of time. Recall that E is the fixed constant in eq. (G.26 (G.29).
Lemma G.22, Given E (0, e/3), it holds for any sufficiently small 5 > O such that
inf m1 In (c,
lim inf
2 c₂
(H(1/n)n)) 1--1
where the event is defined as
Ae n) A {a(n) < R(e, o(n) = (0), X7x $ [s_ - 8, +5|
(T) 17-100 s
[1/m] Vj 2......117
2M
and the constant
1 1,1 1 ra 11
(
c, 2 2b
4M
is strictly positive and does not vary with c, 8.
Proof. Let A AA be the events defined in the proof Lemma G.17
Fix some such that
fix some (exp(2Mttie) Furthermore, choose 8 small enough so that eq. G.72 and eq. G.73) hold for the chosen N and e. Also, we only consider y) small enough so that MM A.
N > al". Based the given
We only consider 5 < M.
48

---

Published as a conference paper at ICLR 2022
Due to eq. (G.26)-eq. (G.29,, we can, without loss of generality, assume that 8+, and in this case we will have
b* 100 > 84 + 100
Under this assumption, we will now focus on providing a lower bound for the following event that describes the exit from the right side of Ω (in other words, by crossing 8+)
(e, (n(n) < R(e,n), o(n) = (; (6). X7 > s + te]
n (T) 1 S
/ Vj 2,,.....
2M
First, define event
A5(6, v) = (W] 2 26 avvj=,, =,, , - S 2M
and observe some facts on event A5(6, 7) (A
X S 3evn (5): (due to A not occurring)
(3) [b 3e, 3e]: (due to WT 2 and the effect of gradient clipping at step TT.
well as the fact that x [-3e, 3e] from the previous bullet point)
Due If(()) S M and M, one can see that (for any n 2 1)
Inf'
2 < 2nyM;
sup =E|-L,1]
this provides an upper bound for the change in SGD iterates at each step, and gives us
X E [b - 3e ē, b 3e + (8) < n < (6)
where also used ()) - (0) 2M [1/n]
Therefore, at the arrival time of the second large jump, we must have X 2g 2 2b 3e E;
By repeating the argument above inductively, we can show that (for all j = 1,2, .. ,,,
X E [(j - 1)b 3e (j - 1)2, (j 1)b + 36 + (j 1)e] VT S n < <T
x E [jb -jb- 3 (j- - 1 3e + (i 1 1)ē]:
In partieular, we know that X for any n T% (so the exit does not occur before T and at the arrival of the 1" -t large jump, we have (using 3G < ?)
X7,10 (n) 2 l'b 1e $t te.
In summary, we have shown that
A5 1) n (e, AA c (e,
To conclude the proof, just notice that (for sufficiently small 1))
P (g)(6.)) (4, (A
P(A2(5, 1)) (c, P(AT(,,))))) P(A (7, 5, 1))
2 P(Ag(5, 11) NN due to eq. G.72 and eq. G.73
H((b//)) " 4M H(b/n)/n)
due to Lemma G.4
H(8/n)
2 2c.8 (H(((()))/) "- - N for all sufficiently small, due to H E RV
(H((/))))))
49

---

Published as a conference paper at ICLR 2022
In order to present the main result of this section, need to take into account the loss landscape outside of the current attraction field Q. Recall that there are Remin attraction fields on f. For all the attraction fields different from Ω2 we call them (= where, for each & E [M - 1]. the attraction field f2 (sg xx)) with the corresponding local minimum located at myg. Also, recall that a (1)) is the first time X exits from Ω. Building upon these concepts, we can define a stopping time
T(n,c) e m 0: X
U
my 2c, my 2eI}
(G.75)
as the first time the SGD iterates visit a minimizer in an attraction field that is different from Z2 Besides, let index (n) be such that
Jo (n) = Xm E $2 Vj E [nemin 1].
(G.76)
In other words, it is the label of the attraction field that X escapes to. Lastly, define
A(1) H(111) (H(1/)//)) 1-1
(G.77)
0 m(E(0)),
(G.78)
1 A ((E(0) n no ((ix)) Vk E [Tmmin - 1].
(G.79)
For definitions of the measure μ4, set E, and mapping h, see eq. G.644) and eq. (G.65
Now ready to state Proposition G.23 the most important technical tool in this section. In eq. (G.80) and eq. (G.8I). we provide upper and lower bounds for the joint distribution of first exit time a and the label Ja indexing the attraction field we escape to; it worth noticing that the claims hold uniformly for all u > C. In eq. (G.82) and eq. (G.83). we provide upper and lower bounds for the joint distribution of when we first visit a different local minimum (which is equal to 7) and which one we visit (indicated by X²). The similarity between eq. (G.80) eq. (G.8T) and eq. G..22 eq. (G.3) suggests a strong correlation between the behavior the SGD iterates at time (1)) and that time r(n e), and this is corroborated by eq. (G.84) : show that it is almost always the case that is very close to o, and on the short time interval (T)) T(), c)] the SGD iterates stay within the same attraction field.
Proposition G.23, Given C > 0 and some k' E [Mmin 1), the following claims hold for all 0 that is sufficiently small:
limsup sup NiO ue(C,,0) [ell-e,,ee]
sup
PA (PP(())(())) u (n( =k)
3)3 5p + + vن
<2C
(G.80)
lim inf inf inf
Pa (()(())())) u, Jn(n) k')
n0o rE( 2r)20]
20 + expl -11 111 3) 1/ C
(G.81)
usl
limsup sup
sup
Pa (L)(())((n))) 6) > X%
E B(ixxr, 26))
100 vE(C, ré[ =
<4C + expl 1)3 28 +
(G.82)
lim inf inf
inf Ps (((((m)((ne)) > X E B(mx,
n00 ue(C, xe 2c]2e]
-C
2- -xcCccp(((1+07)) (1++))33)
(G.83)
lim
inf
Ps (X C, X k. Vn ( -(n, e))
2c,22
21 C.
(G.84)
50

---

Published as a conference paper at ICLR 2022
Before presenting the proof to Proposition G.23 we make some preparations. First, we introduce stopping times (for all k 2 1)
Te(e, 5, n) min(n 7 (,,,,,)) Zml > 6)
Tx(e, 5, 1)) = min(n 2 Tx 6,7) I - 2c}
with the convention that To(e, 5, 1) -e, n) 0. The intuitive interpretation is follows. For the fixed we treat [-2e,2r, as a small neighborhood of the local minimum of the attraction field Q. All the 2 partitioned the entire timeline into different attempts of escaping S. The interval [Tk-1, Fu] can be viewed as the k-th attempt. If for (()), the first exit time defined in eq. (G.70). we have (17) Tk. then we consider the k-th th attempt escape as a failure because the SGD iterates returned to this small neighborhood of the local minimum again without exiting the attraction field. On the other hand, the stopping times kk- indicate the arrival time of the first large jump during the k-th th attempt. The proviso that 7h: 2 Tk-1 can be interpreted, intuitively, as that an attempt considered failed only if, after some significant efforts to exit (for instance, a large jump) has been observed, the SGD iterates still returned to the small neighborhood [-2e, 2e). Regarding the notations, we add a remark that when there is no ambiguity we will drop the dependency on e, 6, 77 and simply write Tk. Tk-
To facilitate the characterization of events during each attempt, we introduce the following definitions. First, for all k 2 1, let
ju - #(n = 1 (, m), (c, + 1, (e, (o : > 6)
be the number of large jumps during the k th attempt. Two implications of this definition:
First, for any k with (1) < Fr. we have JR = O. Note that this proposition concerns the dynamics of SGD up until o (1)) the first time the SGD iterates escaped from 92, so there is no need to consider an attempt that is after (11), and we will not do so in the analysis below;
Besides, the random variable j& is measurable w.r.t. Frano()) the stopped alaggerra generated by the stopping time 73. A (m))
Furthermore, for each k: = 1, 2, .., let
(e, 8, m)) = Th- (E, 8, 17) A (1)) Te, 6, = min(n > Tx (c, 2 > 8) Ao(1)) A vj 2, We, 8) = (t, Vj 2
with the convention Te, 5, 11) 1
(c. 6, 1)) Note that for any k 2 1,j 2 1, Tk.j is stopping time. Besides, from the definition of JA. one can see that
-k- 1 S T Ao() Vj
lix]
(G.85)
and the sequences (Te.) de j=1 and (Wa.) are the arrival times and sizes of large jumps during the k- attempt, respectively. Again, when there no ambiguity we will drop the dependency on c, 6, n and simply write T and Wk..-
In order to prove Proposition G.23 we analyze the most likely scenario that the exit from Ω would happen. Specifically, we will introduce a series of events with superscript or o, where x indicates that the event is atypical or unlikely to happen and o means that is a typical event and likely to be observed before the first exit from the attraction field Ω. Besides, the subscript k indicates that the event in discussion concerns the dynamics of SGD during the k k-th attempt. Our goal is to show that for some event A (c, 5, 17) its probability becomes sufficiently small as learning rate 1) tends O, so the escape from Ω almost always occurs in the manner described by (A* x (c, 6, n)) In particular, the definition of this atypical scenario A involves the union of some atypical events A B that occur in the k th attempt. In other words, the intuition of A x is that something abnormal happened during one of the attempts before the final exit.
Here is one more comment for the general naming convention of these events. Events with label A often describe the "efforts" made in an attempt to get out of Ω (such as large noises), while those with label B concern how the SGD iterates return to [-2e, 2e] (and how this attempt fails). For instance,
51

---

Published as a conference paper at ICLR 2022
A discusses the unlikely scenario before Tk, the arrival of the -" large jump in this attempt, while B in general discusses the abnormal cases after Tk, and before the return to 2r, 2c]. On the other hand, A describes a successful escape during k-th th attempt, while Bg means that during this attempt the iterates return to without spending too much time.
Now we proceed and provide a formal definition and analysis of the aforementioned series of events. As building blocks, we inspect the process (XW)x21 even finer granularity, and bound the probability of some events (A )120 (422 detailing several cases that are unlikely to occur during the escape from or return to local minimum in the k-th attempt. First, for each le 2 1, define the event
&x, = {3i 0, 1, ,1" Jo s.t.
max
2., 7 1 Tr.++1
30 + 3p +3
(G.86)
j=T, (Ta, A0)))
Intuitively speaking, the event characterizes the atypical scenario where, during the k-th attempt, there is some large fluctuations (compared to ē) between any of the first 1 large jumps (or the first jæ large jumps in case that J. < 1*). Similarly, consider event (for all k 2 1)
A 8, n) {o(()
TR, J <('}
(G.87)
that describes the atypical case where the exit occurs during the th attempt with less than I* large jumps. Next, for all k 2 1 we have another atypical event (note that from eq. (G.85) we can see that, for any j 2 1. jg 2 j implies Tk. S (1) 7x)
J& 2 1" = 2,3, ... s.t. Tw.j- > 20(1/n)
(G.88)
representing the case where we have at least 1" large noises during the k-th attempt, but for some of the large noise (from the 2nd the -th,,, the inter-arrival time unusually long. Moving on, we consider the following events (defined for all 1 2 1)
k.3
jx 1", Fh a1)), - TR,1 23 (()mm)
(G.89)
that describes the atypical case where the k-th th attempt failed but the return to the small neighborhood 2c, 2c] took unusually long time.
The following event also concerns the scenario where there are at least 1* large noises during the k-th attempt:
j 2r 12 3j=
s.t. 1) Wr. I 8]:
(G.90)
specifically, describes the atypical case where, during this attempt, right after the I* large noise the SGD iterate is far enough from the local minimum yet some of the large noises are not that large. Lastly, by defining events
A x 6.5
ja
21", 1", Th. S (o(1) A X² E
c, +el}.
[s, o,
(G.91)
jE(Mai-11]
we analyze an atypical case where the SGD iterates arrive at somewhere too close to the boundaries of Ω at the arrival time of the 1" large noise during this attempt. As an amalgamation of these atypical scenarios, we let
A (c) &
U
(G.92)
2=0
Also, we analyze the probability of some events (B))z) Ja21 that concern the SGD dynamics after the 1' 1thth th large noise during the k-th th attempt. Let us define
i(e)
(,, A je 2 1", X% E { + +,,0 -c,, Thi - Th.j-1 S 2,3,
(e,,,) A ( Tk,-- > u(c)/n) (o(n)) Fr.
B (, é B nBx
(G.93)
52

---

Published as a conference paper at ICLR 2022
where p(-) is the function in Lemma G.13 From the definition of B in particular the inclusion of 22 one
that the intuitive interpretation of event B is that the SGD iterates did not return to local minimum efficiently (or simply escaped from the attraction field) after the -t large noise during the k-th attempt. In comparison, the following events will characterize what would typically happen during each attempt:
(e n) = Hi&21" ((n) = =T X $ [s_ 8, +el.
2f(c) vj = 2.3, ,.th
Tk.5 Tx.5-1 5
(G.94)
1
21 f() + p(e))
(a(n) > R3 Rh Tx 1
(G.95)
Intuitively speaking, A tells us that the exit happened right at TR.t. the arrival time of the -" large noise during the k-th th attempt, and Bi tells us that the first exit from Ω did not occur during the k-th attempt, and the SGD iterates returned local minimum rather efficiently. All the preparations above allow use to define
A (c,d,))
U A21
(A
n (AX UB
(G.96)
We need the next lemma in the proof of Proposition G.23 As mentioned earlier, the takeaway is that x is indeed atypical in the sense that we will almost always observe (A j².
Lemma G.24. Civen any C the following claim holds for all > 0, > sufficiently small:
lim sup sup PA(A (c,8,n)) <C
n0 1x|$2e
Proof. We fix some parameters for the proof. First, with out loss of generality only consider C' E (0, 1). and we fix some N > al", Next we discuss the valid range of e for the claim to hold. We only consider > 0 such that
ED
+ 6p+p++3
where p and " are the constants in Corollary G.7 and Corollary G.10 respectively, and is the constant in eq. (G.8.. Moreover, recall function V Lemma G.21 and the constant c, >> 0 Lemma G.22 Due to limejo (e) 0, it holds for all small enough such that
(e)
<C
(G.97)
In our proof we only consider small enough so the inequality above holds, and the claim in Lemma G.13 holds. Now we specify the valid range of parameter 5 that will be used below:
For all sufficiently small 8 > 0, the claim in Lemma G.14 will hold for the prescribed and with No N:
For all sufficiently small 0, the claims in Lemma G.17 and Lemma G. 20 will hold with the prescribed and N;
Corollary G.18
Corollary G.19
For all sufficiently small 8 > 0, the inequalities in Lemma G.2 and G.22 will hold for the we fixed at the beginning.
We show that the claim holds for any e, small enough to satisfy the conditions above. First, recall that
A (e, 6, = -3i==0
0,1, 0,1,--. 1.1.....,IAA. A s.t.
max j=T,, (Tat,,....TTTT (TT.
//
TT, +
Sh.n
25
3p + 3
3
53

---

Published as a conference paper at ICLR 2022
Due to choice of ò stated earlier and Lemma G.12 there exists some no 0 such that for all 77 E (0, no).
( 6. m)) S vk 2 1.
(G.98)
Similarly, recall that E,, 6, { (n)) < ix jx < 1- Let temporarily focus on the first
attempt (namely the case k = 1). From Lemma G.17 and our choice of c and 6, we know the existence of some ?/, 0 such that
sup |
P3 Pa (A (A,, (c, S 1n An E (0,
(G.99)
Next, for x - ji 2 3 2,3, 8.L. Tx.j TR.j1 20(e)/n). from Corollary G.18
and our choice of at the beginning, have the existence of some 722 such that
sup P2, (e, |e|$2e
S n An (0, 72).
(G.100)
Moving on, for X
(j&ct i o()), F - Tx.1 > 21 (()(())) due to Corollary G.19 and
our choice of e, 6, we have the existence of some 1/3 > 0 such that
sup P. (A,,3(,, 8, n)) - " An (0,
(G.
le|$2=
XX As for x A 2,, one can see the existence of 172 > such that
k..- 2 r- 1, ... , s.l. w/ s 8). from Lemma G.20
sup Pz (A (e, (e, m)) S n vn E (0, [0|$2e
(G.102)
Lastly, for A A 1i. Thp-- s o(n) X UgE(mm- -,
8, tel} s, from Lemma
G.2 we see the existence of mps > 0 such that
sup Pa (o(AA,s(e,,,)) m)) S (() (1(\/)/))* V E (0, |a|g2e
(G.103)
Also, for definitions of B A B. see eq. (G.93).e e... (G.95) respectively. Our next goal is to establish bounds regarding the probabilities of these events. First, if we consider the event -A UB ) BJ, then the inclusion of the (B))==1 implies that during the first attempts the SGD iterates have never left the attraction field, so
Recall that A (c, = U (c,ō,) m).
b n
UU
=
n
(N
(A) UB BB)) n > ix).
j=1
Next, note that
k-1
Pa (
n
j=1
=P. B
k
U
B
(N (A j=1
UB ) B3) n B
20(e)
k-1 r'
<Pr (jx 2 1" Th.
Tk Tkjj1 <
Vj = 2,3,
N U )
n
k-1
Pr k2
(A)
U ) n B n nBx
5=1
54

---

Published as a conference paper at ICLR 2022
From the definition of the events A BI Bj. one that N3= UB y Fr, (^a(())
and on this event i (A) U B ) n Bj we have a(n) > FR-1- So by applying strong Markov property stopping time Tk-1 A (m), we have
R-1 n (A) U
P. K
k-1
<P(T)(6) - (6) 17)110)))7)110
P2 B
(BFBB)))B) n nB ==1
k-1
(H(8/M(())/))
B
(N
(A)
BB) B B
j=1
(for all 17 sufficiently small due to Lemma G.4
<4(t(c) ((c)
H H11/m))"
1-11
h-1 ( (A) UBJ) n BB.1
77 Px
B
80
22
j=1
for all sufficiently small 7,, due to H E RV (1)).. Meanwhile, note that
(nf J n ( B E FT
on this event (N (A) UB y n BB) n B we have ((1) A F > Th and X Te. E [s_ C, 8+
c.
Therefore, using Lemma G.13 and strong Markov property again (at stopping time Tk, we know the following inequality holds for all 17 sufficiently small:
k-1
sup sup Pr
Bx k2
(N (A) U nB 1=1
#21 2x|
A-1 (A(A) (BJ)(BB)) nB
sup sup Pa (a(n)
> Tke Fr Tx. S p(c)/n)
#2 |x|
j=1
8a
<V((C
(((c))/00)"
Therefore, know the existence of some 77 > 0 such that
k-1
BB) < (c)8 H(1/n) 1--1
sup P. ( 2x
n UB (BBB)
An E (0. Vk 2
(G.104)
Similarly, can bound conditional probabilities of the form Pa (A be specific, recall that A = A we know the existence some 7/7 such that
n (A) UB y B To
x- By combining eq. G.98 -eq.
1.1
with Markov property,
A-1
(H(V/)))* 17
sup P. (A |x| j=1
s 5n + (c)
An E (0. Ak 2 1.
(G.105)
On the other hand, lower bound can be established for conditional probability involving A the event defined in eq. (G.94) describing the exit from Ω during attempt with exactly I" large noises.
55

---

Published as a conference paper at ICLR 2022
Using Lemma G.22 and Markov property of (X1)221 one see the existence of some 7/s such that
k-1
B 2 c H((lm)) r-1
inf 2
(A
n UB
Vn
18).
(G.106)
j=1
In order apply the bounds eq. (G.044 (G.106), make use of the following inclusion relationship:
k-1 (
n ( U S A B
(G.107)
To why this is true, let us consider decomposition of the event the L.H.S. of eq. (G.107 As mentioned above, on event n5= occurred and there are only three possibilities this event:
k-1 (A) UB nBJ we know that a ( >> - the k- th attempt
Ja < :":
ja 2 r" X Th.:- $ 92
ju 2 1, x h-.- EQ
Let us partition the said event accordingly and analyze them one by one.
. On (A) UB J n (especially A and Ax))) we can see that jæ 1", then we must have ( Ty and + - 7 S (()(() Therefore,
(A U U
) n [ik < r"), due to the exclusion of A
k-1
[A B (A L (jx <1)} C B
j=l
On (N)=1(A) UB F B (AX (jx 21', X7 $ R}. then the exclusion of A implies that Tkj Tkj-1 2f(c)/n for all 2, ... 1" and the exclusion of
A tells us that if X
Th & Ω then we have X%,,- $ [s_ - c, 8 c]. In summary,
( (ABBB)))))) UB [ix 21, X $ S AC.-
On (A) U J (A) UB ) r [js 1,, X2 Tr E [], the exclusion of A k.2 again implies that Tx.j T S 20())/n for all j = 2, ... hence Ta, - Tk.1 S 2r'((c//n Similarly, the exclusion of A tells that if X7 E Ω then have X7,,, +E, e Now since B did not occur (see the definition in eq. G.93 )
must have o(n) > F and Fr Tr, I p(e)/n, hence Th Tx.1 S p()) Therefore,
( (A) UU BB) U Uh 20, X EΩ)C C B
G.1077. Now we discuss some of its implications. First, from eq. 107 we immediately get that
Collecting results above, we have eq.
k-1
k-1
(N (A) UB ) )B) n (A ) f=1
n (A (A) U B UB UB (
==1
(G.108)
56

---

Published as a conference paper at ICLR 2022
Next, recall the definitions of A in eq. (G.44) and Bğ in eq. (G.95 and one that A and B are mutuaaly exclusive, since the former implies that the first exit occurs during the k-th attempt while the latter implies that this attempt fails. This fact and eq. (G.008) allow us conclude that
#
* n (A UB )
R-1
n (A U B UA;)= UA))
nB; B n
(n (A) (B n (A U UAR)..
(G.109)
Now use the results obtained so far to bound the probability of
A A (c,
U
u (A UB (A?)*) n (A UB
121
Using eq. (.109 we can see that (for any [-2e, 2c))
(A(A
8)))
(A) UA UA?)) n (AX
Σ
P
A21
Σ 121
P
(B
(A7
)((AUUB)))))))
h-1
Σ
P2
(A
nB
121
1-1
f-1
(AF UB)
j=1
Σ Ps
B
(B)
1>1
k-1
II
PP
(A)
n (A) UB uA3) ( ) nB;)
Σ 131
Ps
A
UU B
(A
BBBB)BBBB
k-1
(A U
uA n (AF " nB.)
1=1
1-1
b-1
1 ( (B
j-1
Pz
x
x
A U B (B
1 1 P. (A
121
j=1
57

---

Published as a conference paper at ICLR 2022
This allows to apply eq. (G.104)-ep eq. (G.106) and conclude that (here only consider
min : [8]}
sup PA (A, |
((1//)) ) : (1-c,8 (H(1/n) )
517d + (c)8a
17
H(1/m) 1-
5n + (2)
c,80r
((11n)
(c) + +5n"
for sufficiently small 7, due H E RV-a(1) and our choice of N al"
Co
3V(())
<C for all 1) small enough such that 5z)² < (t)...
The last inequality follows from our choice of in eq. G.97 This concludes the proof.
Having established Lemma G.24 we return to Proposition G.23 and give a proof. Recall that, aside from the attraction field Ω (s_, $4),, there Amin other attraction fields 22 (s s (for each & E [Miin 1)). Besides, the function X(-) and constants 1º 1/f are defined in eq. G.77
eq. G.79
Proof of Proposition G.23 we only need to consider E (0, 1). Next we discuss the valid range of for the claim to hold. We only consider >> 0 such that
We fix some parameters for the proof. First, with out loss of generality
4
6 3)
3
where P and " are the constants in Corollary G.7 and Corollary G.10 respectively, and eg is the constant in eq. (G.8). Due to continuity of measure it holds for all small enough such that (let ? = 3 3)cc
{(E(0))
<1/(1-0),
(G.110)
(()(2))
((E()))
>1/(1+0)
(G.111)
m(E( ()))
/1 h 16-1((8 22, + 22) U U 22, 84 + 28))
SC
(G.112)
m(E(-E))
11 ((F()) (55 é ot +0)
8 +C
(G.113)
M(E(E))
v
( (sp. + 28, st. -2E)
C
(G.114)
())(-))
18
In our proof we only consider small enough so the inequality above holds, and the claims in Lemma G.13 hold. Moreover, we only consider and 5 small enough so that Lemma G.24 hold and we have
lim sup (A x (c, C.
(G.115)
940 |z|52
We show that the desired claims hold for all e, sufficiently small that satisfy conditions above. First, in order to show eq. (G.84. we define event
A (c,5,n) n) (A (c, 9
{x()(((n.c) -((()) 2 C' or 3n o(1)) 1, (((ne) such that X2 & ÿg (m) 1
58

---

Published as a conference paper at ICLR 2022
Since À E RV -1--1-1(--1) and > 1, for the fixed the beginning of this proof, p(c) is fixed constant as well (the function is defined in Lemma G.13 and we have lim,zo (())p(e)/ / 0. Next, the occurrence of (A x (c, 6, n)) (in particular, the exclusion of all the A defined in eq- G.91))
we know that (n)) $ [$j, - E, + U [$], - c.
+ <| (recall that for any E [miin 11. we
have 22 = (.), ,87); for definition of Ja see eq. (G.6 ). Meanwhile, for all sufficiently small, we have e/ e/\(n) > p(e)/11 Therefore, using Lemma G. we can see that (for all vi sufficiently small)
sup Px A (e, (A (e, S C. $2
(G.116)
Lastly, observe that
(((AM)(((ne)) -())) 2 C or En onn) + , e)
such that X & $1.,(()) (m)))
SPa (A*))n (x())((())))) - - DC
or 3n = (n) + 1,, .(n e) such that X & (. }) + (A*)
so by combining eq. (G.115) with eq. G.116), we can obtain eq. (.84).
Moving on, we discuss the upper bounds eq. (G.80) and eq. G.82) Recall that the fixed constant
k' E [minn- 1] is prescribed in the description of this proposition. Let us observe some facts event
(A (A (e, 8))² n (n) k'}: If we let J(e, 8, n) e sup(k: 2 0 F < o(n)) be the number of attempts it took to escape, and
(e, A min 1: Ta, has (3 7 ++ 5, -overfffr) overflow),
then for all 17 sufficiently small, we must have J J on event (A x (c, n. 8)) n (n) = k3. To see this via proof of contradiction, let us assume that, for some arbitrary positive integer j. there exists some sample path on (A (A*) {Ja(1) = }}} such that J" = j
Then from the definition of (A x )), in particular the exclusion event A 3,00 (see the definition in eq. (.86 ). for all sufficiently small we are able to apply Corollary G. 10 and G.7 and conclude that X Tz,- & Ω: indeed, using Corollary 1.10 and G.7 we show that the distance between X2 and the perturbed ODE
(T, T3.1. 0; (0, T222 T3,, .. "T,,, - TT,,,,
(..,,............ ,n,,..
is strictly less than 3(p - 3) on the other hand, the definition of (3(p+++3) 5. overflow implies that
8" 03,,, Ty1, .T,.- T2,), (".",..... (,,...,,,,,....))
- 3)r, + +3 + p 3)c).
Therefore, must have X7 u. $ Q, which contradicts our assumption j J J. In summary, we have shown that, on (A consider
x n [Jo = K}, we have (e, 2 J(e, m). Similarly, if we
5e, min(k 2 1 Tx,1 has 1-3 8, 7) overflow),
then by the same argument above we can show that (c, S J(e,5,n) Now consider the following decomposition of events.
. On {J+ < Jt}, we know that for the first lo such that TE.1 has (- 3(p + P + 3)e, 6, 1) overflow, it does not have (3(p + ক + 3)e, - ovvefllo. Now analyze the probability that Zo does not have ( -vverflow conditioning on that it does have (-ê, 8, -vverflow (recall that we let = ++3)). Using Lemma G.15 and the bound
eq. (21122. we know that for all 1) sufficiently small,
sup P.. ( (J+
/ h ((s 22,8 -_ + 26) U (8_ - 166--2,,84 + +2)))
< C.
(G.117)
m(E(-E))
59

---

Published as a conference paper at ICLR 2022
On (A*) (Jo = k = J due to J J
know that Te (n))) is
the first among all Tg.1 to have (2, 6, n) (1))oveer -vverfoo.. Moreover, due to {Ja k}} and using Corollary G.10 and G.7 again as we did above, we know that the overflow endpoint of T((6))))1 is in (sp - 2, S + é) (recall that f4 = ( s))) In summary, for any n 2 (A (A*)
k (J+ J n} (Jo [J
()* > n) n TTt,a has overflow endpoint in (sh - 2, + }}
using Lemma G.15
we obtain that (for all sufficiently small)
sup Pa (A*) =k]nk (A)))N.=R3)t
= - =
2x|
p(e, (: é + +()
(A* >m))
< sup Pr
p(2,6,n)
|x|$2e
o +C
< sup P. 2r|
( m))
(G.118)
v
uniformly for any 0, 1, due to eq. G.113
On the other hand, on (A x yr, if y+,,1 has overflow endpoint in (s + 22, st. -22), then from Definition G.1 we know that Ty.,1 also has (é, -verfflo, hence J = J =
Moreover, using Corollary G.10 and G.7 again, we know that X7 E (8tr, + so
J, = k' summary, for any n
(A* (A*) =
_ > n}
2 (J Ts=,1 has overflow endpoint in (sw + 2ê, sx 22)
using Lemma G.15 we obtain that (for all n sufficiently small)
inf P (A*) (Jo = ( J - m1) |2
pl (1-2,8,7 q:: (S st. 22))
> inf Pr ((* m1)
0(-2,5,1)
1 -C
inf P3 (( > m²)
(G.119)
S2
uniformly for any = 0, 1, due to eq.
1.114..
Besides, the following claim holds event (A y
From eq. 10. the definition of Bi as well as the definition of event A (sce eq. (G.44) one can see that for any j = 1,2, J. we have
A TT < +
Now if we turn to the interval (7 Ty,1 (the time between the start of the j-th attempt
and the arrival of the first large noise during this attempt) for each j = 1,2, ... JJ, and the following sequence constructed by concatenating these intervals
S(c. 5, m) - (1,2, ... ,.,, + 1, + ... +2,.
R +1 2, ... 7o+1 + + 11, 1,...),
then the discussion above have shown that, for
min(n S(c, m) Zn has 38 7+ 3)c, 6. -overflow)
- 2 Tz1-
60

---

Published as a conference paper at ICLR 2022
Meanwhile, from the definition of overflow know that the probability that Z1 has (3 + ++3),,, overrfow is equal to
((66)))(3(p+p))) 3)c,
Therefore, if, within the duration of each attempt, we split the attempt into two parts at the arrival time of the first large jump (7x,1)<>1 at each attempt, and define (here the subscript before after indicates that an attempt)
counting the steps before after the first large jump in
Becfore (c, 5, 1)) - {n E S(c,8,)) n n n (1)), I (c. 17) 2 #Seoree (c, 6, m)). Sater (c, 1) - {n E 5(c, n n (n)), Ianer (c, 6, 11) _ #S 5, 1), then have o (n)
Janer- Moreover, the discussion above implies that latter S ((e) + p(c))
Thefore
Ibefore S min(n E S(e, n) Zn has (3)++ū+3)e...)) -overfow]
and on event (A )e.
Define geometric random variables with the following success rates
Ui Ui(E,6,n) ~ Geom (+3(0+5+3)),6677)
U2(e, n) ~ Geom (166)))p(((+++++) 3)c, d.
Using results above to bound Defore and Later separately on event (A ) we can show that (for all sufficiently small and any u > 0)
Pa 0 (()))(())) > u, Jo()) = =k)
sup zE|- 2c,2c
sup PA(A* ( (c,d,))) 1x|$2e
sup Pr [ xE(-2c,2c]
Jo)o ()
<C'+ +
sup Pe (()))()))( > ( (1 due to eq. G.115
=E]-2e,2c]
SC' sup Pr 2E(-
+ sup P7 aE(-2c.2c]
SC+ sup
Ps
2E(-2c,2
+P (n))) p(E) + 21 i(E)
(, > Ct)
n
<C
+ sup
Pr (1) -
=
xc]-2c.20]
Pa (=((A3))n +P (())))) ple) + +2(((e)
sup
2c
n
5 +C
<2C + PP (Om)(() (1 (1- C1-)
wl?
-P(D82(1) p(c) + 22((()) (c, Cn
(G.120)
n
61

---

Published as a conference paper at ICLR 2022
where the last inequality follows from eq. (G.117) and eq. [G.118) Now let analyze the probability
terms on the last row of the display above. For the first term, let a(1) - H(6/))/(6///////// (316/))0(((0++++)ee,, 6,
Due to Lemma G.15 we have (recall that 2 -: (((())
a(1)
lim
1.
710 X(n) (E( + 3)e))
Combining this with eq. (G.TTO) one can see that for all sufficiently small,
P P( (m))())s((e)) (1- - IP (eor)) > (1 (1- Vu > 0. Next, let b(17, u) = P P(a(n)Geom(a())) > (1 - C)²u) = P -P (Geeom((n))
(--)""u) For
g(y) = log(1 - y). we know the existence of some y/o such that for all y E (0, yo). we have log(1 y) -(1 -.. So one can see that for all 7 sufficiently small,
log bu, 11) < (1 - u log(1 a(n)) S -(1 -11-0(1
a(n)
b(,, 1)) S exp (1 (1- (1-C))
(G.121)
uniformly for all u > O. For the second probability term, if we only consider u 2 C, then
IP (f (n)) p(e) + 2/)(()) Ui(E, 8) Cu) s PP (m)))()) ple) + fl(((c) U,)(,,,,)) > C²).
" Using H E RRV-a(n) (1) with a > 1, we get
r33 7 3)e, 6, 1) U (e, 8, n) S Exp(1) as 4
due to the nature of the Geometric random variable U1. Besides, due to H E RV-a(1) with a > 1 and Lemma G.13 is easy to show that
A(n) 222)e)
lim 710
3 3),,, = 0.
Combining these results with Slutsky's 's theorem, we now obtain
((F(()))A()) p(e) + i()(()) U.le, n) 5. as 11 4.
Therefore,
lim sup sup TP (((((())))))) p(e) + 2r((())
(8, n) Cu 0.
(G.122)
x2C
7
Plugging eq. (G.12T) and eq.
G. 122 back into eq. (.120), we can establish the upper bound in
eq. G.80 To show eq. (G.22) note that for event
E(c, n) (v. ((n)r(n.c) > u, (7) E B(ms,, 20)),
we have (for definitions of r. see eq. (G.75)
E(c,n) 2( 2 (1)) > u, (n) = k/} n (X fmmm Vn E (o(n),
Ele, ()( (1)) (()))) > u, Jx(1) j] n (X E d. Vn en (() = = =. + #
Therefore, for all 17 sufficiently small,
sup P. (E((,)) 2n
< sup P. (A x) + sup Pa((A* n {XX & T. for some n E (o(n), e1
S2
2z|$2c
+ sup Pz (A f T (())((n) > Jo) 13) n {X. Zgo Vn E jo(n),,(nne)) r(n.))])
Sc
50 C
<4C + exp ( (- C)
62

---

Published as a conference paper at ICLR 2022
uniformly for all C, due to eq.
(G.115 eq. G.88 and eq. (G.120).
The lower bound can be shown by an almost identical approach. In particular, analogous to eq. can show that (for any 0)
G.
inf
P.
(1 (( (o)) = =k'
.rel fe(-22r,24 2r,2c]
inf
Pz ( u, = k2) n (A (c,
#E( 11-2e 2e.2el
inf
Pz (2_
u, (n) = k1 (c
sEl sel-22,22 2e,2el
Up +
2P 9 Xn))U?(,, 5,
C)u
2C
due to P) 2 P(E)- - F()) and eq.
G.115 eq.
G.117 eq. G.119 where
eq.
(e, 6, n) Geom
[((//))p(
3( + 7 3)c, 3)c, 6.
Using the similar argument leading to eq. G.11TT. conclude the proof.
able show eq. (G.8T) eq. (G.83) and
Recall that o (n) min(n 2 0 Xn $ 2i and that value of constants q, Gi, are specified via
eq. G.2 -eq. G.7 Define
T:minn (n, e) a min 2 o,(1)) XX E Ulm, 2e, my + 2e]},
(G.123)
J((n) =j
x" x² onin)
E 2 Vj [Mmiin]
(G.124)
The following result simply restatement of Proposition G.23 under the new system of notations. Despite the reiteration, we still state it here because this is the version that will be used to prove Lemma which is the key tool for establishing Theorem well as many other results in Section
H
Proposition G.25, Given C 0 and i.j [Temiu] such that + j. the following claims hold for all 6 > 0 that are sufficiently small:
lim sup sup
sup
Pz (g:di(1)oss()) X2 X² E EsA))
n00 uE(C,oo) 2E +2,)
xC+ (- (-11- C)) thd +C
lim inf inf inf
Pr (1:A1(1)000000 u, X" on(n) E EsAy
740 aE(C,,0) 2e 2e,
2 2-C ( (1++ J C
Ui
Pr (9.d((1))7711 (1, > X (n.)) ②2))
lim sup sup
sup
740 aE(C,oo) (m(m;- mc)
+xpe ( (- (-11 C')u Ui +C
lim inf inf
inf
Py (q:d((1)) (11, (1) > u, x" T (n.))
S2))
740 aE(C,o0) (m(m m,,, +2e)
2 2-C+exp (- (- Ji.j C
gi
lim inf
inf
P2 (a:Ae(n)))"" (1,c) az(n)) < <C,
n10 2E(m, 22c)
X E Pg,( Vn (o, 7. (17 (n,) > 1 - C.
Concluding this section, apply Proposition G.25 and prove Lemma G.2
63

---

Published as a conference paper at ICLR 2022
Proof of Lemma G.2 Fix some C' (0,1), u 0, and some k. [Mmial with & * I. Let qa, 9i, be the constants defined in eq. (G.7).
Fix some Co (0, C. A we have
C). Using Proposition G.25 we know that for all sufficiently small,
Roin
limsup N00
sup
Pa (1) u, X da)(t) E 02))
#E(m, -2
Co + exp C)) 9k.j Co
Vj
[mmin]]
Summing up the inequality above over all j E [miin we can obtain eq. (G.66). The lower bound eq. (G.17) can be established using an identical approach.
In order to show eq. G.19 note that we can find C1 (0, u) sufficiently small so that
[k.l C1
9k.1 C
-C1 exp ( (1 Ci) 201)
qh
Fix such C.. From Proposition G.25
we also know that for small enough, we have
lim inf 140
inf x =E(me m2, +2e)
Pz (9x)d(7))N(()) u, (m)(m) St)
Yhh1 -
exp ( (1+ 20g) - exp
qk
Then using P. eq. (G.19.
X (4()) E Ea) (())))()) u, X" xax(n) E 2u) conclude the proof for
Moving on, show eq. enough so that
G.18 in the following way. Note that can find C2 (0, u) small
9k.1 + C2
Ih.1 + C
202 +
(G.125)
Fix such C2. Since eq. small enough,
G.17) has been established already, we can find some u2 > 0 such that for all
Pa ()) s u2) C2
lim sup 750
sup =E(ms 2e,
(G.126)
Fix such u2. Mcanwhile, fix some C3 (0, C2 A u2). From Proposition G.25 we know that for all sufficiently small,
lim 740 =E(ma Te(ma-2e,ma422 +26)
sup
Pr (o))e())o > u2, X" ox(1) ERI)
(G.127)
$ C3 + exp ( Cz) 9k.1 + Cs
< C2 Ih.l + C2
(G.128)
qh
Lastly, observe the following decomposition of events (for any x 146)
P. X" (X E < P. 9x>x S u2) + PP, (u uz X2 EQI).
Combining this bound with eq. (G.125)-eg
G.128).
complete the proof.
H PROOFS FOR SECTION 2.2
In this section, show that gradient clipping scheme effectively partitions the entire optimization landscape into different regions based on the radius r; and minimum jump number i for each attraction field Q.. Furthermore, when staying in each region, the behavior of SGD iterates closely
64

---

Published as a conference paper at ICLR 2022
resembles Markov chain that only visits wider attraction fields in this region. We exclude the trivial case where 7mmn = and there only one attraction field.
This structure is as follows. First present some key lemmas that can be used prove the Theorem in the main paper. Then we devote the rest of the section to establish those lemmas. In order to prove Theorem
will make of the following lemma, where show that the type of claim in Theorem[] indeed valid if we look at a much shorter time interval. Then when we move onto the proof of Theorem 2 it suffices partition the entire horizon into pieces of these short time intervals, on each of which we analyze the dynamics of SGD respectively.
Lemma H.1. Assume the graph G is irreducible, and let e >> 0, a > 0 be any positive real numbers. For the following random variables (indexed by n)
10/A (ril
/ (n, 2
{ E
U
(H.1)
/tr (n))
jim & Mriie
the following claim holds for any sufficiently small t 0:
lim sup 740 zE|-L,L]
sup
Px (V c,,, > c) 1 56
Proof of Theorem we have
It suffices to show that for any > 0, 1+ (a (a-1))lurg 1) , 0,8 E (0,c).
lim sup Pz
(v,
> 3c) <
n40
for
18/1"]
V"(,,,,) A 1
If x
(x)
(x) E U fj
[1/10]
Jimy Mauu
Let us fix some é 0,6 E (0,c). First, let
1t/r""
N(1) = (1/2)ar8(()))
The regularly varying nature of H implies that ( RV (m Since >>
plarge (a- 1), we know that lim,oo (n) = oo. Next, due to Lemma H.l we can find to > 0 and
> 0 such that for any 7) E (0, 7))
sup (V small (n,, > <)< < -E| L.
(H.2)
For any le 2 define
k A
Vi((n)
[aj E
U
2 du.
(H.3)
[to/ Aarge (n)]
(K- L)tto)
jimy MiMe
is clear from its definition that V&8 stands for the proportion of time that the SGD iterates are outside
of large attraction fields on the interval [(k: 1) X J. { 2 From eq. H.2) and Markoy
property, one can see that for any 77 E (0,
sup (=(V((n) > e XT. X2 x (k- 8 2E[-L,1]
uniformly for all & 2 1. Now define ((n) ع #(n = ., .()) Va(n) > e). By a simple stochastic dominance argument, we have
sup P. (K()) (7) 2 5) < (N Vj = 1,2, xE[-L,1
65

---

Published as a conference paper at ICLR 2022
Meanwhile, strong law of large numbers implies the existence of some ? such that P(( Sinomial(,V(m))) 26) for all r7 (0, īT), thus
(())
sup IP (k)))/(()) 28) 5 Vn (0. |
Lastly, from the definition of ((n) and N(n), know that for all the (n) intervals [(k In 1) (y) ( ]l with & E (m)], only on ((1) of them did the SGD iterates spent more then proportion of time outside of the large attraction fields, hence
K(n) ('(nt,m)) sc N(n)
In summary, we now have
Pa (V" > 3c) <
for all 1y E (0, T1 A 1)) This concludes the proof.
When introducing Theorem in the main paper, stated that the results of eliminating sharp minima can be extended to the more general reducible case. Here we present the corresponding theoretical results in Theorem H.2 and H.3 below. The main message can be summarized follows: (a) SGD with truncated and heavy-tailed noise naturally partitions the entire training landscape into different regions; (b) In each region, the dynamics of X for small 1j closely resemble that of continuous-time Markov chain that only visits local minima; (3) In particular, any sharp minima within each region is almost completely avoided by SGD.
When the typical transition graph (see Definition[ in the main paper) is not irreducible, there will be multiple communication classes on the graph. Suppose that there are K communication classes G1, Gk. From now on, we zoom in on a specific communication class G E (G1, ........ Gx For this communication class G. define l2 - max(4, local minimum m, E G, call its attraction field Ω2 a large attraction field if 1 1. and a small attraction field 1 < l'g. We have thus classified all m, in into two groups: the ones in large large and the ones in small attraction fields mymall large mic
i 1,2, Temin; m, E G). For each
attraction fields my
mmmall Also,
define a scaling function \a associated with as \a H(11n) (H())) ((t/m)(((m))
15-1
Theorem H.2. Under Assumptions 3 G is absorbing. then there exists a continuous time Markov large
chain Y on {m1 large
large } such that for any I E 02,12 S L (where E (1,2, -. " }) with Mag
my G, and
X
Lt/ha(nn) (x) Ye(Tr((m)) as 4
in the sense of finite- dimenssina distributions, where TO is random mapping satisfying (1) (m()) = if m E {m] large
..... maa 1; (2) (m(m) is random variable that only takes value in {m large large if E {mmq mia
large
small
We stress that Theorem 3 in the main paper follows immediately from Theorem H.2 above.
Next, to state the corresponding result for transient communication class G, we introduce a couple of extra definitions. We consider version of X that killed when Xạ leaves G. Define stopping
time
T A min n 2 0 X &
S2)
U
2
(H.4)
d.
the first time the SGD iterates leave all attraction fields in G, and we use cemetery state t to construct the following process X1-" as a version of X with killing at T0:
X
if n < To(1). if n 2 Te(1).
Xt
(H.5)
t
66

---

Published as a conference paper at ICLR 2022
Theorem H.3. Under Assumptions if G is transient, then there exists a continuous-tine Markov large
chain Y with killing that has state space {m] large killed when it enters the absorbing cemetery state f) such that for any E 94 [x] L (where i E f1,2, ... Tmin 1) with m, E G. and
.... mia At} (we say the Markov chain Y is
X (()>0()))
(m(TT(mm) as 1] 1
the sense of finite-dimensionll distributions, where ag is random mapping satisfying (1) TG (m(m)= = m if m E {m]] (moe large
large Ji , ,mie (2) ma(m) is a rundom variable that only takes value in m/ mmalt)
{m (arge mia 11 if m E {mqquall
To show Theorem H.2 and H.3 we introduce the following concepts. First, we consider the case where the SGD iterates X initialized the communication class G and G is absorbing. For some > 0, 7 > 0, define (let (u, v)
+ vl)
of (n, A) - min (n 0::0::0 X2 E
B(m, 24)}
(H.6)
M.EG
/ (17, A) e m 2 /z (1. A)
X E
B(m;, 24)}
(H.7)
im;gGmml
(1 A) =j x² E B(mj 21),
To (n, A) =j
og E B(m,. 24)
(H.8)
of A) - min fn >  17, A):
x E
U
B(m;, 2 Ak 21
(H.9)
iam,EG, i
Th T. A) و min(n 2 1 A): X E U
B(m; 21)} Wk 21
(H.10)
() =j X% E B(mj, 2A), + +
X² E B(mj, 24) 2 1. A)
(H.11)
Intuitively speaking, at each G the SGD iterates visits minimizer that is not in a small attraction field on G, and we use I to mark the label of that large attraction field. Stopping time of is the first time that SGD visits a minimizer that is different from the one visited at T and G is the first time that minimizer not in small attraction field of G is visited again since of (and including o It is worth mentioning that, under this definition, we could have T T for any k 2 O. Meanwhile, define the following process that only keeps track of the updates on the labels (I )220 instead of the information of the entire trajectory of (Xz) m20:
A
mgg 0
3f 2 such that Th S < G
TR+1
(H.12)
otherwise
In other words, when n < TO we simply let Xg.A O, otherwise is equal to the latest "marker" for the last visited wide minimum up until step n. This marker process X jumps between the different minimizers of the large attractions in G. In particular, if for some n we have X E B Bmm,, 2A) for some j with my E Glarge then we must have X my. which implies that, in this case, X indeed indicates the location of X
Note that results in Theorem and concern a scaled version of X", Here we also define the corresponding scaled version of the processes
X
X
(H.13)
[t/Aa())
A
(H. 14)
[t/Aa()))*
a mapping T'(n, r)) e (n)(1) that translates a step the corresponding timestamp for the scaled processes, and the following series of scaled stopping times
T/, A) T (T (n. A), (n A) T (n))
(H.15)
67

---

Published as a conference paper at ICLR 2022
Before presenting the proof of Theorem H.2 and H.3 make several preparations. First, proof is inspired by ideas in Pavlyukevich (2005) and here we provide a briefing. At any time t 0, if we can show that X :1 is almost always in set Ulm (m, 2 (so the SGD iterates is almost
always close to a minimizer in a large attraction field), then the marker process X,"", is pretty accurate indicator of the location of X, so it suffices to show that the marker process X; X converges to a continuous-time Markov chain Y.
Second, we construct the limiting process Y and the random mapping aa before utilizing them in Theorem H.2 and H.3 As an important building block for this purpose, we start by considering the following discrete time Markov chain (DTMC) on the entire graph 9 = EV, Let PDTNe be a transition matrix with PDTMC (m, my) = as (Ea,))/ (s) (Ea) for all j * i, and DDMMC- = (YPPTMC)) )220 be the DTMC induced by the said transition matrix. Let
ToTMC& - min 20: y,INC Camull)
(H.16)
be the first time this DTMC visits large attraction field on the communication class G, or escapes from G. Lastly, define (for any such that mj & Gamaill
PTMC P.j - PV Terme
(m:)) = m;)
(H.17)
as the probability that the first large attraction field on G visited by Y-DMC is my when initialized at m,,
We add a comment regarding the stopping times TDD and probabilities Pt.j defined above. In the case that G absorbing, we have Y, y-DTMG (m²) G for all j 2 0f m, E G. Therefore, in this case,
given any with m, E G. we must have
DDTMC- min 20: DDTMC (m:) E Glare), TO
Σ
Pr. =
j: myE
On the contrary, when G is transient may have Ej: e Pi.j and Σj: vy Pij > 0. Lastly, whether G is absorbing or transient, we always have pi.j = 1 = j} if mi E Glane Next, consider the following definition of (continuous-time)ji jump processes.
Definition H.1. A contimuousttime process Y, on IR is (UU))20, (V))520) jump process if
Ye Yz= { {o 0 Ezz0 V/ U0+U1+-4t00
ift < Un otherwise
where (Ug)520 is sequence of non-negative random variables such that U, > OVj 2 1 almost surely, and (V,)i20 is sequence of random variables in IR.
Obviously, the definition above implies that Y, V, for any E (U,.
Now we are ready to construct the limiting continuous-time Markov chain Y. To begin with, we address the case where G is absorbing. For any m' Glarge let Y (m') be a ((Sk)k20: (Wi)kza)- jump process where So = O, Wo = mn and (for all k 2 0 and i, with my Glarge my & Gamall)
W h+ = my: Sk+1 >
W₂ mn, (WD (5
(H.18)
W₂ = m.) eexp(-q:t) Vi Vt >
=P
Wh+1 = my. S441 >
(H.
where
9: = (a(E.)
(H.20)
9:.j 1 + (F.))) +
Σ R: m&
Ms (Fa,k)))
(H.21)
and Pa. is defined in eq. (H.17). In other words, conditioning on Wt = m,, the time until next jump Se41 and the jump location W are independent, where S Exp(g:) and We+1 my with
68

---

Published as a conference paper at ICLR 2022
probability /9. First, is easy to see that Y is tint Markov chain. Second, under this definition Y is allowed to have some dummy jumps where W Wx+1; in this case the process Y, does not move to a different minimizer after the k + 1-th jump, and by inspecting the path of Y we cannot tell that this dummy jump has occurred. As result, that generator Q of this Markov chain admits the form (for all i with m,, mj E Glarge
Q
Σ
li,k, Qi, = Qi.j
Anskefi my EGluse
Moreover, define the following random function mo(+) such that for any my E G,
Ta(m)) = my with probability k../9 / if m, E Gsmall
(H.22)
m, if m, E Glare
By Y (Ie (m4) we refer to the version of the Markov chain Y where randomly initialize Wo (m())) The following lemma is the key tool for proving Theorem H.2
Lemma H.4. Assume that the communication class G is absorbing. Given any m, E G, S4 finitely many real numbers (4i)== such that 0 ti < f2 < ty,, and a sequence of strictly positive real numbers (Nn)a21 with lim,-
0, there exists a sequence of strictly positive real numbers (Aa)n21 with lim,, An such that
As n. tends to oC,
(8)
(
8 Am (x)) = (To()))))) ,Yo (TG(m.)))
(H.23)
For all [
lim Ps ( &
U EG
B(mj.Aw) =0.
(H.24)
n-000
Now address the case where G is transient, let t be real number such that L, L]. and we use i as the cemetery state since the processes X or X are restricted on [- L, L). Recall the definition of no defined in eq. (H.4). Analogous the process Xt in eq. (H.5). we can also define
x/!*"
[x," if t T
El.mA=
if t t TTc((( (H.25)
otherwise
otherwise,
Next, analogous TG. consider the stopping time
Tt min (t > 0: Y, $ G).
When G is transient, due to the construction of Y we know that Th < oc almost surely. The introduction of TO allows us to define
Yr
if i otherwise.
Y
(H.26)
t
The following Lemma will be used to prove Theorem H.3
Lemma H.S. Assume that the communication class G is transient. Given any my G, x Ω. finitely many real numbers (1i)'2 such that 0 li < <.... LRs, and a sequence of strictly positive real numbers (Nn)a11 with lim = 0, there exists a sequence of strictly positive real numbers (Da)a21 with lim, A. such that
As n tends to
(m (x),... X1 An + (To(m)))) Ytt, (TT(m.)))
(H.27)
For all [K'],
lim Pr X Ma o n=000
U
B(m,, And and Mi # -.
(H.28)
my EGiny
69

---

Published as a conference paper at ICLR 2022
Proof of Theorem H.2 and H.3
We first address the case where G is absorbing. Arbitrarily choose
some > 0, a sequence of strictly positive real numbers (7m)021 with lim, 7ho = 0, a positive
integer k', series of real numbers (t,)ī=1 with 0 < t < <... < tys, and sequence (my) with
Wj E Glare for all j E [K]]. It suffices to show that
lim P ***" E (wx A) A (K)) = P (Yi. (Ta(m))
Ak E [K1).
Using Lemmal H.4 wc can find a sequence of strictly positive real numbers (An)n>1 with lim,, An O such that eq. (H.23) and eq. (H.24) hold. From the weak convergence in eq. (H.23), we only need to show
lim P. (Xtt m $ B(x B. A)) - vk E [k]]
For all large enough, we have 2 <
A. For such large observe that
Pa Xtt $ B(X 2))
U
B(m,, 2An))
jam
(due to definition of the marker process X,, see eq. (H.6 eq. H.11 and eq. (H.12 (H.14)
<Pn dl
B(mj, Am))
jam
and by applying eq. (H.24)
we conclude the proof for Theorem H.2
The proof of Theorem H.3
is almost identical, with the only modification being that we apply Lemma In doing so, we able find sequence of [An)221 with lim, A, such that (H.27) and eq. (H.28) hold. Given the weak convergence claim in eq. (H.27). it suffices
H.5 instead of Lemma H.4
show that
lim Pa (X & B(K A)) Vk [A'].
1. = t. we must have X N.... = t as well. Therefore, for all n large enough so that 3A, A,
P x xmm & B(x B(Xli 2))
2 x to
$ B 3(tt:
A), xi +t)
<IP
x
mm
U
B(mj. 24n): xte.0m + t)
J: ECtue
SPP,
x tm
U
B(m,, An), Xt # t).
J: my m0tue
Apply eq. (H.28) and we conclude the proof.
H.1 PROOF OF LEMMA H.
First, we introduce another dichotomy for small and large noises. For any 3 > 0 and any learning rate 7/ > 0, we say that noise 2, is small if
1 Zn 17
and we say Z, is large otherwise. For this new classification of small and large noises, we introduce the following notations and definitions:
2,5 Zn.iinZni <n²},
(H.29)
7,, 7, > )²²)
(H.30)
(T) e min(n 21 1: 7/Zn| > 3
(H.31)
70

---

Published as a conference paper at ICLR 2022
Similar to Lemma G.12 the following result is direct application of Lemma G.11 and shows that is rather unlikely to observe large perturbation that are caused only by small noises. Specifically, since >> 1 always find
E (1 (1-1 1 -1- )
8 (2, (o(1-7)).
Now in Lemma G.11 if we = 7, = 2/2 and = 8 (in other words, u(n) = 1/m²- -" v(n) = ,), then for any positive integer j the condition eq. (G.52 is satisfied, allowing us draw the following conclusion immediately corollary from Lemma G.11
Lemma H.6. Given N > 0 and
E (0, (1 - 2) BE (
(2.21)))) A (a- af)).
we have (as $
P
max 1=12, [0/1))
+ +2
2,5 Rm > 17 1/2) =(11
The flavor of the next lemma is similar to that of Lemma G.13 Specifically, we show that, with high probability, the SGD iterates would quickly return the local minimum long they start from somewhere that are not too close the boundary of an attraction field (namely, the points $1, $2, ... -mmm) To this end, consider refinement of function f(-) defined in eq. (G.38). For any i = 1,2 ... 72min any E Ω4 and any 1 > 0, E (0, 1), we can define the return time to 17 neighboooodooo for the ODE x" as
(z. m) - min ( 2 0: (t, msl 5n
Given the bound in eq. G.37 (which is stated for specific attraction field) and the fact that there only exists finitely many attraction fields, we know the existence of some 02 < 0o such that for any i 1, mmins any y 0, any y (0,1) 1) and any Q such that | 8i| V [a 8 > 17,, we have
400 (z, m) S c2t log(1/n)//
and define function tt as
(n. - o27 log(1n)
Lastly, define the following stopping time for any 1, ... Miin any E Ω and any A 0
T (1) (17, A) min(n 2 0: (x) B(m, 24)}
where we adopt the notation (uu, v) [u v. u + 2) for the neighbooodod around point Lemma H.7. Given
7 (0, (0) E (0, 16Mcz 7
and any i = 1,2,
Romins any A 0, we have
lim inf
inf 950.-10--411MM22--1222
Pr (T(i) (17, A)
2027 1og(1/1)
X E Ωt Vn S xT(i) (n7, A))
7
11
Proof.
Throughout this proof, we only consider r) small enough such that
2027 log(1/n)/n < (, 7M s 5 2nj < A/2, 2n 2// < 17"
7/4
(H.32)
The condition above holds for all n7 > sufficiently small because 1, 25 1, and 7 < 7/4. Also, fix some B (1, 27) (a a7))
71

---

Published as a conference paper at ICLR 2022
Define the following events
A7 (n) max
2,900
17 n///2)
A2 (7)(7) & [1//))
(see eq.
HH.1
for definition of the stopping time involved)
and fix some N > 0. From Lemma H.6 we see that (for all sufficiently small 17)
P(A (n)) S 17
Besides, using Lemma G.4 constant 0 > 0 such that
logether with the fact that 3 < a(1 R),, know the existence of some
(() 1n
for all sufficiently small 1).
Now we focus on the behavior of the SGD iterates on event A (1) n (2 Let us arbitrarily
choose some x E S4 such that xx s,| V | we know that
s1-1| 2n17. First, from Lemma G.6 and eq. (H.32).
(x) Y(y (x)) s M exp log(1/))
s 27) 2M
S 2n Sn Vt S 202y log(//)nn
(H.33)
Next, from the definition of the function t and eq. H.33 , we know that for
A A) a min(n n 2 0 (x) E B(m,, + }
ToDaetm (2; (2;
(H.34)
2
we have
TeDaetur (2) A) I 2022 log////(1/n)// y 81-1 + 17", yi s; 17" Vn s 2027 log (1 1/n)/n-
(H.35)
(H.36)
Furthermore, event (A) n (n) due to Lemma and eq. HH32 we have that
(x) - y% s 1/ 7/2 exp (2Mc₂ log( (g/)) = n : 2M S 3//4 17 Vn S 2027 log(
Combining this with eq. H.35), eq. (H.36), we can conclude that (recall that due to eq. H.32 we have 207 2/2)
I (17, A) logy/ogl1/) X E Q Vn $ 201y log(1/n)/
on event (AN ( (m)) Therefore,
Ps (I(()) (17. A) < 2027 log(1/1)
lim inf
inf
X E Q2 Vn < I (17 A)
=E0,
22/27
17
lim IP (() 940
liminf 1 NTn 17² = 1. 750
This concludes the proof.
The takeaway of the next lemma that, almost always, the SGD iterates will quickly escape from the neighborhood of any 8;, the boundaries of each attraction fields.
Lemma H.8. Given any 7 E (0. 1), > O, we have
liminf inf Py min 2 0: 2::0 B(), n00 L,
H((/)) = =1.
t
72

---

Published as a conference paper at ICLR 2022
Proof. We only consider 17 small enough that
1+3
min
18: 8 > 31 -
2=23 Bai
n < 17"
Also, the claim is trivial if x & Us, 2n"), so without loss of generality we only consider the case where there is some [Mmin]] and æ [-L, L|, E B(s,,27) 2n)). Let define stopping times
T A min(n 2 1 n|Zn| > 5n
(H.37)
eecape min 2 0: X & U, 2n²)),
(H.38)
and the following two events
A² (n) 2 (T7 H(1/n)
A (n) (n|Zr=1 +
First, using Lemma G.3 and the regularly varying nature of H()) P(IZI) > +), we know the existence of some 0 Such that
(n)) S exp(-1/n")
for all rj > 0 sufficiently small. Next, by definition of 77, one can see that (for any 1) (0,
P(A (n) H(1n1
H(5/11
Again, due H RV-a and 7 > 0, know the existence of some 0₃ such that
P( (1)) < 17
for all > 0 sufficiently small. To conclude the proof, we only need to note the following fact on event (A/ (n) U A% (n)) There are only two possibilities on this event: Texcupe I T 1, or Tesape 2 T", Now we analyze the two cases respectively.
. On (A7 U A (n)) n (TTcape T - 1}}, we must have Texcape < T S st/H(1/n)
On (A (1) U A (n)) n (Tecape 2 T7 J. we know that at = T - 1, there exists an integer j E {1,2, 7mmi 1} such that X E ((8),2n7)) Now since n, < 17? and
1/ Zr> 577", we must have
Xx] X7 > 4nj X1 B(s,,2n2)
1 On the other hand, the exclusion of event A (1) tells us that XX X7 < 21 =
Due to eq. H.37), we then have X & (s;
In summary, (A (T) A (1)) S (Temum S {1/H(L/m) and this conclude the proof.
In the next lemma, we analyze the number of transitions needed to visit certain local minimizer in the loss landscape. In general, we focus on a communication class G and, for now, assume is absorbing. Next, we introduce the following concepts to record the transitions between different local minimum. To be specific, for any 17 > O and any A > 0 small enough so that B (m,,, A) n n2 = for all j. define
To(n, A) min 20: X E U, B(m,, 24)];
(H.39)
Io(n, A) iff X ()( B(m,, 2A);
(H.40)
Te(T, A) min(n > T A): X E U) 24)] Vk 2
(H.41)
44T, A) j iff (n.)) E B(m,, 24) Vk 1.
(H.42)
As mentioned earlier, the next goal is to analyze the transitions between attraction fields takes to visit my when starting from m; when m,, m, E G. Define
K2(1, A)
min 2 0: (n A) = ii.
73

---

Published as a conference paper at ICLR 2022
Lemma H.9. Assume that G is an absorbing communicatiin class on the graph G. Then there exists some constant > such that for any with m, E G. any e 0, and any > 0,
sup my EG; B(m, 23)
Pa (M A) Romin) < P(Geom(p) 2 u) +E Vu = 1,
Pa 3k E (Ki(n, 8.1. m)x(n,)) $ &G) se
sup 3: my eG; BBB
hold all sufficiently small.
Proof. The claim is trivial if, for the initial condition, we have a E B(m;, 24). Next, us observe the following facts.
Define (recall the definitions of measure 14 and sets Es, E in eq.
G. leq. G.5 eq. G.6
J(i) arg. min )
Vi
13 mg(B,,Gi)
p'
min 8: JaL MyEG my(E,)
From the definition of (() and the fact that there only finitely many attraction fields can see that p" > 0. Moreover, G being a communication class implies that
(J() - il < i Vj my E G.
Indeed, if < j. then since G is communication class and there some my E G with < j. we will at least have (E( > 0, so [(] - < i jl - 1; the case that i > can be approached analogously.
Now from the definition of J(j) and Proposition G.25 point, we know that for all r sufficiently small,
together with the previous bullet
inf P7 i 14 il 1, m4 E K1 A) > k, my E G) 2 p2/2
=E| L.
uniformly for all k 2 0.
Meanwhile, since p² > O, we are able to fix some small enough such that
Dimin 5 (p" /2) Pai
< 6.
On the other hand, for any J with my & G, by definition of the typical transition graph we must have My (E, for any j with my G. Then due to Proposition G.25 again, one can that for all 0 that sufficiently small,
sup Pa(m G Ki(n A) h. my E < =E[-L,1]
uniformly for all l: 2
Repeat this argument for 7min times, and we can see that for all 17 sufficiently small
inf Pa A) < k mmin (2 A) k, my, G) #E - L,L)
p
Thai
sup Pa E [nmin] s.t. Mla+1 G K2(n, A) k, my. G) & xE[-L.1]
uniformly for all k: 1.
74

---

Published as a conference paper at ICLR 2022
Lastly, apply the bounds established above, will make use of the following expression several probabilities. For any j with my E and æ E B(my, 2 and any v 1,2,"..,
(a E [uMmin] s.t. my & G, Ki(n), A)
u (mmin)
2- 1
Po Ek E [Meinn] s.t. Ma $ & Ki(n, A) > v. Omins Mi E G 1A S Um
1000
II 00 w=
Pa (Ki(T, > (w + Mmmn Ma- E G 4k E [[mmin]
K,(n. A) > w Mminn Mi. E G 1A S WD
Pa(m,, E G VV [ummin), Ki(n, A) > u Ammin)
P (n A) > (u+ 1)mmin m Ic4vran
G vk [Amin]
K,(n. A) > min mi G. th: [emminn)
In summary, now we can see that (for sufficiently small 17)
Px Ik E [Ki( A)] s... m.e(nA)) & &G)
sup J: m,EG se (m,
Ps 3u E [mminl such that Macuus $ G.
w= myEG ae Bm,,,22)))
Ki(n, A) > Wilmi mi, E G Vk E [W)7mmin]
sup k= j: EG; aE (1)
Pr Eu E [mminl such that M an $ G
Ki(n, A) > Wilmin my, E G Vk: E [MsIimin]
sup
P Ki(n, A) > (U+ 1)mmin M Ma E G Vk E [Mmii]]
D= ji m.EG; ae 2m)
Ki(n, A)
Wimin, M. E G Vk: E [UM
< Σ d 8 [1- (1-12
)
p
nam
u-1
7mmnn
Se
2
(p" /2)muin
u20
and
sup jam, EG, #m 24)
Pa (2 A) Mminn)
sup jim, EG, = (m, 22)
Pr (3l E [Wiimin] s.t. m. & G, K2(1, A) > u (mmi))
sup jmm, EG, zEB(m, 22)
Pa(m,, E GVV [ummmn], Ki A) u 71mmn)
mmin (1 -
pr
mmm
1-1
p" + (1
Temi
2
2
mm
mminn 3
(1
(p*) /2)mmni
2
Se +
P" Minia
)
uniformly for all 1,2, To conclude the proof, it suffices set p (g) Pain
75

---

Published as a conference paper at ICLR 2022
The proof above be easily adapted to the case when the communication class G is transient. Define
K. (n) A) min 2 O: [n,,, A) or mmxmn))) G).
Lemma H.10. Assume that G is transient communication class the graph G. Then there exists some constart > O such that for any i with my & G and any A E (0,E/3) E/3),
sup f: EG Bmm 24) hold for all n sufficiently small.
( (n, A) > Tomin) & P(Geom(p) 2 u) Vu 1.2,
(H.43)
Proof. The structure of this proof is analogous that of Lemma H.9 Again, the claim is trivial if, for the initial condition, we have x E B(ms, 2 Next, let us observe the following facts.
Define (recall the definitions of measure b4 and sets Ei, Big in eq.
G. leq. G.5 eq.
G.6
J(j) arg min 1 - Vi )P
min Ho(E,,66) 3. mGEG My(B,)
p'
From the definition of J(j) and the fact that there are only finitely many attraction fields we can see that p" > Moreover, G being a communication class implies that
(J()) <
Vj X my E G.
Indeed, if < j, then since G is communication class and there some my G with < j. we will at least have (E),-,)) > 0, so []( - I i - jl 1; the case that i >> can be approached analogously.
Now from the definition of J() and Proposition G.25 together with the previous bullet point, we know that for all sufficiently small,
inf P2(I241 il S 1 il 1, Mi++1 E G K (n, A) > k 2 p2/2 2
uniformly for all k 2 0.
Repeat this argument for Umin times, and we can see that for all 1) sufficiently small
inf PK( (v, A) s T'mn K A) > k) z
P
-E|- -
uniformly for all 1: 21
Lastly, for any j * i with my E G and E B(m,, 24) and any u = 1,2, ....,..
(K (1,, A) Memin)
P
K. (n. A) > (v + 1)Mmin Kg A) Remin
-=U
[1 P. K (1, A) < (o+ 1)mmin
ki A) Amin
v=0
In summary, now can see that (for suffciently small 1))
P. (K (17, A) 2 u Mmin) S 1--(p )mmm))
sup j.mjEG, af (mm,.22)
uniformly for all 1,2,.... To conclude the proof, it suffices set p (8)
76

---

Published as a conference paper at ICLR 2022
We now ready to prove Lemma H.T which, demonstrated earlier, is the key tool in proof of Theorem
Proof of Lemma H.1 some
The claim is trivial if (large so we focus on the case where Iluge 2. Fix
7 (0, - B (2 25) (a an)), 7 E (0, 16M
Let 9- = max, Hy(( We show that for any t (0, - the claim is true.
Now we only consider A E (0, E/3) and 1 small enough so that nM S 17 and r) < A. Consider the following stopping times
Te
min fn 2 0: X2 & U, 2n))]
T
min {n 2 0: X E U,B(m,,221 2n
First, from Lemma
H.8 we know that
sup P PT escape 1/H((In))<< 6/2 LE[-L.]]
for all 17 sufficiently small. Besides, by combining Lemma H.7 with Markov property (applied
Tecapee), we have
1
sup P. Teeuu L.(-
Texcuee
202y log(n/)/) T Te H(1/m)
<//2
for all 1) sufficiently small. Therefore, for all sufficiently small,
log(1 log(1/n) δ.
IP(Titum sup xE|-L,4]
H(1nn) + -222
(H.44)
Let be the unique index such that X E Et, Our next goal is to show that, almost always, the SGD iterates will visit the local minimum at some large attraction fields. Therefore, without loss of generality, we can assume that m. & M and define
T large
minfn 2 Tiem X2
U iam, M
Bim,, 2D)}
and introduce the following definitions: è Trum: Jo J
The A min(n (n > Tk-1 X j&
U
B(m,, 24)}
J/ = X 22 Wk 2
K a min(k 2 0: m.j. E Mluge..
In other words, the sequence of stopping times (Tx (2)) is the time that, starting from Ti the SGD iterates visited a local minimum that is different from the one visited at Tk-1, and (Ja)420 records the label of the visited local minima. The random variable K is the number of transitions required to visit a minimizer in large attraction field. From Lemma H.9 we know the existence of some p" such that (for all v) sufficiently small)
Pa(K 2 -mmin) S P (eeoee(p")) u) + 2
Vu 1,2,3,
where Geom(a) is Geometric random variable with success rate a E (0, 1). Therefore, one can find integer (8) such that (for all sufficiently small 7))
sup P(K > N(8) 5.. =E[-L,1
(H.45)
77

---

Published as a conference paper at ICLR 2022
Next, given results in Proposition G.25 and the fact that there only finitely many attraction fields, one can find a real number u (6) such that (for all sufficiently small 17)
u(6)
sup -,
PA(TA - Th-1 S
AJa-1 S 6/N(6)
(H.46)
uniformly for all = 1.2,
(6)). From eq. (H.44) eq.
H.45 .a H.46
we have
( m)
log 1/n)
sup xE[-L,4]
Pr X
U
B(m,, 24) Vn S (6))(())
large (n)
H(1mn)
+ 2027
1
y y Edd
$ 36
(H.47)
for any sufficiently small r7. To conclude the proof we just observe the following facts. First, due to H RV-a and plage 2 2. have
Alare (n)
log(1/1) > Jurge (n) 0.
lim (( 700
lim 940 H(1/n)
0, lim nj ")
Therefore, for sufficiently small we will have (note that 6 are fixed constants in this proof, so (6)), u()) are also fixed)
(O) H(L/n)/n (1)
W(1T) + 2027 log(L/n)
7
S
(H.48)
[t/Aluge (n)
Second, recall that we fixed some (0. iq-) where q" max, (E)). Also, choose some C small enough so that
C 8/2, 2(1 C)² <
From Proposition G.23 and the fact that there are only finitely many attraction fields, there exists some iju 0 such that for any 17 E (0, (T)) and any A 0 sufficiently small,
Po (ax(1) S
sup
sup
Alarge (1)
ian EMue ZE (mm +2A
Pz (E,))) large (n)o, s 9r))
sup im, Mlae zE|m - +221
sup
<C 2(1 C) < 28.
Combine this bound with Markov property (applied at TK), and we obtain that
sup eE(-L,E
Pr 3 E [L4/> large (x)]] s.1. X7
R+Tx
$
U
2))=
2.
< 28
for all 17 sufficiently small. Together with eq. (H.47 eqq.
H.48
we have shown that
sup 2E[-L,4]
Px
V /small (1, e,
< 58
holds for all sufficiently small.
H.2 PROOF OF LEMMA
H..5
We shall return to the discussion about the dynamics of SGD iterates communication class G. Recall that
Cargg
[m
large
large mi } Grsmall
{m= , .... mi
small
If X3 is initialized at some sharp minimum on G, then are interested in the behavior of X at the first visit to some large attraction fields on G. Define
Te(n, e min(n 0: X
U
B(m,, X Un m.e.li).
(H.49)
Not only is this definition of Ta analogous to the one for TDTMC in eq. (H.16). but, as illustrated in the next lemma, TG also behaves similarly as TG on a communication class G the following sense: the probabilities Pr, defined in eq. (H.17) govern the dynamics regarding which large attraction field on is the first one to be visited. Besides, TG is usually rather small, meaning that the SGD iterates would efficiently arrive large attraction field on G or simply escape from G.
78

---

Published as a conference paper at ICLR 2022
Lemma H.11. Given any 0 E (0, (a - 1)/2). CE (0.1), i.j E [mminl such that m, G my &
Charge the following claims hold all A > 0 that is sufficiently small:
np
lim sup n0o 2))().2A)
sup
- Po (T((,
Aa(1))
Xtc BB(m,,2)) S Pisj + +5e,
Tp
lim inf
inf 700 se 2())
Ps ())(n,)) - Aa(1))
Xto BB(m,,2)) 2 Pi. 5e,
Aa(1) x
1
lim sup 940
sup x (A,
Px (To(n, A)
S 2e.
hold (and the discussion to be meaningful), we must have lg Throughout the proof, we assume this is the case. Besides, we require that A E (0, so we have
Proof. For Gismlll *
B 34) n = 9 Vi [Rain]]
and the 3A ne-nniihboorhood of each local minimum will not intersect with each other. In this proof we will only consider A in this range.
From Lemma H.9 (if G is absorbing) or Lemma H.10 (if G is transient), we know the existence of some integer (e) such that for (see the definition Ig eq. (H.39 eq. (H.42))
No(n, A)
min(k 2 0: m) E G caarg or (), & G},
we have
sup Pa Na A) N(O) <c (E,,,
for all 17 sufficiently small Fix such (E). Next, from Proposition G.25
find u(e) E (0, 20)
and Á E (0,) such that for all E (0, A), we have
Pa (Th(n. A) - 1n. A) (1))A((4-1(74))))))) s <//N(e) Ak E [N())]
sup #E BEm,,,, 22)
for all 1) sufficiently small. Fix such uue) and A. Now note that on the event
A = No s N() n [T=(n, Tn-1 A) I u(c)/A(l--1(n 4).n) Ak [N(OI).
due to the choice of e (a, - 1)/2) and H E RV-aa we have (when 1) E (0, 1))
2
Ta(n, A) - Tx-1 () v An(n)
vk < A)
17
*To(n, A) TNo(na A) (e)u(()
(a(1)
For any 7 sufficiently small, we will have (e)u(() that for all A A),
"¹ xe(aj
In summary, we have established
Xa(m)
Pa To Po(TO> >
>c(n) )
178
lumsup sup
sup
< limsup n-O xe ((M,
sup P=(A9) < 2c.
(H.50)
9Jo se 2A)
Next, let
S(c) a {
(mi....... m' {m1.....
N(O) 3K [N(c)] s.t. m m
mm /
We can see that S(c) contains all the possible transition path for DTMC where the state my is visited within the first N(c) steps. Obviously, [S(e) cc. Let E e/(S(e)). If able to show the existence of some A 0 such that for all A E (0, A1), the following claim holds for any (eee)
(m)))= S(e):
Pe mm, Pa(m,, = m' Ak E (N(C1) - -P(Y (/P = m4 vk E (N(e))) < <11
sup sup Y1o re (m)
(H.51)
79

---

Published as a conference paper at ICLR 2022
then must have (for all A (0, A1))
X7o E B(mj, 24)) -Pid
limsup
sup (A))
-limsup
sup BE, 2
X7o E B(mj, 24); To ((()) + P, (% E B(m,, 23), To > N(e))
100
ToTMC mm)) m,j T TOTMC <N(O) PP ( m,j TDTMC MC > N(6) To
P
C
< limsup
sup (A)
Po X E B(mj. 24), To S N()))
DTMC - -P TTTM MO (m;)
mij TeTMe S SN())
+ lim sup m40
sup =AB(m,22)
P (T' N(e)) P(ICTMC (m²) N()))
<(S(e)le + lim sum sup no 2EB,m.2
sup
(T N(e) + +P N(e))
S3c.
To show that eq. HH.51 is true, we fix some (m)) S(c) and let (k(())) with (k)()) mi for each &s E [N(e)]. From the definition of YDTMC have (Iet k'()) i)
N(e) be the sequence
N(e)-1
Y DTMC (m)) = m4 Vk E [N(+))
II A=O
(K((K) (Fx(k))
On the other hand, using Proposition G.25 we know that for any arbitrarily chosen e' E (0, 1). we have
N(e)-1
Meaa (a (E((()))))))R) (1 E'),
lim sup sup ZE 20)
Px (mA(A))) = = m4 Ak E [N())) -
II
Meras (EW()))
n10
A=
N())-1
Mx (a) (BK()))))
lim inf nLo
inf we 2A)
Po (m)A())))= = m' Ak E (N()))
c))
(K (x)(
for all A O sufficiently small. The arbitrariness of c' us to see the existence of some A > O such that with E (0, eq. H.51 holds for any f E S(c). To conclude the proof, observe that
0, together with [S()) oc, allows
1/ E B((mj; (4)) Pa (T((T)))))
limsup sup 2A)
P X
X B(m,, 22))
Aa(n)'
< slimsup no0 2A),
sup
P.
To> TT
) <e
due to eq. H.50
Recall that continuous-time process X is the scaled version of X" defined in eq. H.13 and the mapping T' (n, 17) A ndx(1)) returns the timestamp t for X,""' corresponding to the unscaled step on the time horizon of X As an inverse mapping of T", we define the mapping N' (t, m) [n)) that maps the scaled timestamp back to the step number n for the unscaled process X",
In the next lemma, we show that, provided that X stays on communication class G before some time t, the scaled process X."" almost always in the largest attraction fields of communication class G.
Lemma H.12. Let G be a communication class on the graph G. Given any é1 > 0, >> O and any Ωt with m, E G, the following claim holds for all A small enough:
lim sup Pr (f xx," n10
U jom EGlay
B(my, (imy.3A)
n x e
U
Q4 Vs 10. < 2c1-
kt M.EG
80

---

certain that each Published as a conference paper at ICLR 2022
Proof. Let A (0,2/3) for the constant in eq-
G.228 leq.
G.29). J.-
B(,,, 2A) lies entirely in S2 and would not intersect with each other since
B(m; 32) n 2 = 0 vi E [[min]]
Besides, with e = A/3, we know the existence of some å > 0 such that claims in Lemma G.14 would hold of the chosen e, Fix such for the entirety of this proof. Lastly, fix some
7 E (0, (1 (1- ^ G).. 3 (1, - 25) ^ (a a))), 7 (0, 16M 4
a
The blueprint of this proof is as follows. We will define a sequence of stopping times (N₂) such that the corresponding scaled timestamps T = TT 1)) gradually approach t. By analyzing the behavior of X on a time interval [L م,, 4] that is very close to l (in particular, on the aforementioned stopping times Tÿ). we are able to establish the properties of a series of events
A 2 A2 2 A3. Moreover, we will show that A3 S [X, E Us: m,EGuu: B(m,, 32)), ) so the properties about events A, A2; A3 can used to bound the probability of the target event.
Arbitrarily choose some A, E (0, t). To proceed, let No - Nº A.. be the stopping time corresponding to timestamp t - A, for the scaled process. Using Lemma H.8 we know that for stopping time N₁ e min(n 2 No X $ U, 2n)")), we have
4./4
lim inf PN No < 14O aE[-1,1]
=1.
H(11))
Next, let N₂ - min(n N1: XI E Um, 24)). From Lemma H.7 oo (1H()))) we have
and H E RV-- (so that
P. PN -N 4./4 =1 H(1mn)
lim inf 9J0 2E|-L,1]
Collecting results above, we have
PN(N2 No < D./2
lim inf inf 14O LE(-L,
(H.52)
H(1n1)
Now note the following fact on the event {N₂ - No < A./2 J. The definition of the mapping
#11m)
T' implies that, for any pair of positive integers nı I 722 we have T T" 11) T ("1,
(n² n2 Ma(() & (n² m1) H1/n)). Therefore, on {N₂ No S
A./2 we have H(1/n)
A.
T (N2, n) T" (No, 7) < - n) <t 2
Besides, let T2 T (N2, 1)). Now we can see that for event
Ao 2 { X e
U k: mGEG
Q4 E [0.e] n { N2 - No <
A./2 H(1m)) }
we have
Ag S A A
T
<t
A f
U
2g v j0, Tı]
2
in mLEG
Meanwhile, from eq.
H.52 obtain that
lim sup
sup =E[-L,E)
P.
(A
e
U k: m&EG
Ω Vs [0. = O.
(H.53)
740
Moving on, we consider the following stopping times
Ng min 2 N₂: X2
U d: my EGmm
B(my, 24) X $ U 2,},
J: m,EG
T T (Ns. m)
81

---

Published as a conference paper at ICLR 2022
Using Lemma H.11
have
Ae/4
lim sup P. (Ns NN
A11) 61. Aa(11)
(H.54)
740
Meanwhile, event A n NNo N2 - Am(n) n {x.- E UR: myEo Ur Vs E [0, 10. have
2,/4
T3 T S 4./4, hence T3 t D.,t - A //. In summary,
A./4 \
An ( N3 2 \c(1)
n X U
Q vs E [
In mmEG
E A., 1..---////// xxx E
U 2x v (0, [o,Ta]
h: mLEG
Moreover, on event [x E Uk: my Ea S4 Vs E [0.t]]. if we let Ja be label of the local minimum visited at T3 such that J33 = = X E B(mj, 2A), then we must have m,js Glurge
Meanwhile, consider the following stopping times
T - min(s Tÿ: X;" $ 2222
From Proposition G.25
we know that
limsup Pa
T
Tī s A
1-8t-444 A,, - 1,,t- n nxx= U k: meEG
(ecit-,,tt-
24 Vs E [0. T3)
SeI + exp (- (1 +eegg""a))) A.)
(H.55)
where q- = max, H1 (Ez). Now we define the event
A₂ A n N3 N₂ v
A.4 Aa(1)
n - -T-
U l: mGEG
Q4 Vs (0, [0,1ā]}
Using eq. (H.53 eq. (L.5 we get
lim sup sup
Px A5 (x
U
2 Vs E [0.t]]) < 2E, exp ( (1 +c)g"" At).
L,
myEG
(H.56)
Furthermore, event A2, due to E D., - 4./4 as established above, we must have
X E f2 Vs E [T t].
Now let us focus on a timestamp T; = on event Ag. Next, define
0 H(1ln)
\G (7) and N₂ = NT m)) Obviously, T > Tī
N₂ a minfn 2 No: X E U (m,, 2A)}
Ts T (Ns, n).
Using Lemma H.7 and H.8 again did above when obtaining eq. H.52
show that
lim sum P.. (No- - N4 n40
A./16
H(1nn) 0.
(H.57)
On the other hand, on event A n No - No
A./16
we must have
H(1//3
Th - -T- S
A./16 H(1lm)
\e(n), so T5 E elt -t-
A./8
\o(n), A./16
H(1/)) Ac(1)];
H(Lm))
-x R. due to T - A..
82

---

Published as a conference paper at ICLR 2022
This implies that for event
2./8
4,/16 (11/n)
A Tē E jt H(1nn) Aa(1))
Ac(1)), Xig E aas J
e U
2 Vs E T5]).
k: mgEG
we have A₂ n (Ns N^
S A. Lastly, observe that m(10))
A./16
From Lemma G.4 we know that for Ne(0) min(n > Ns 7/ > 8} we have
sup P(N(()) N₂ s A,/H(1nn) - A,/8"
As stated at the beginning of the proof, our choice of á allows us to apply Lemma G.14 and show that
lupsup sup sup 750 DE
Pa(3n = No, ... ,No - s.t. X & Bmm,, 34) Ā) = O;
Combining the two bullet points above, we get
lim sup Pa (Es E [Ty t] such that X;"" & B(m,, A) s De/8".
(H.58)
n/o0
On the other hand,
Än (xg e B(m,, [Ty S (X
U
B(m 3A)}.
k: my EGles
In summary, for event
A//1 ) n H(1/n) ( x (im.,;34] Vs E [T 0
Ag - A2 1n No No S
we have Ag S (X, E Up maEGue B(m,, 3A)}. Besides, due to eq.
H.56 eq.
H.57 eq. H.58).
we get
lim sup nj0
sup
Ps Ag (5 (X E
U
Ωr vs E (0.4])
L,)
k: mLEG
A.
<2e +1 exp ( (1++1)4 A.)
ja
Remember that 61, q" are fixed constants while A, can be made arbitrarily small, so by driving A we can conclude the proof.
Recall the definition of jump processes in Definition H.I Central the proof of Lemma H.4 the next result provides a set of sufficient conditions for the convergence of a sequence such jump processes in the sense of finite dimensional distributions.
Lemma H.13. For a sequence of processes (Y"))z1 that, for each 2 1, y is (UF) )220, (Vz))200) jump process, and ((U))520, (V))j20) jump process Y, if
Uo = 0;
(Ug, Vr, UT, VR, UT VT,, ... converges in distribution to (0, Vo, U1 V1, U2 V2,...) ... as
n 20;
For any and any
P(U .... Un = =x)= =
For any 0.
lim P(U + Un + Un > x) 1
400
83

---

s the Published as a conference paper at ICLR 2022
then the finite dimensional distribution Y converges that of in the following sense: for any N and any < 11 < 12 < ... tk < oc, the random element (YT,)" ". ,Yt)) converges in distribution (Yrr 11,) as n - oc.
Proof. Fix some k E and 0 < ti < t2 <... < th < Jc. For notational simplicity, let t = ty- Let (ID, d) be the metric space where [. the space of all cådlåg functions in Ik the time interval [0, t). and d is the Skorokhod metric defined
d(Gi (2) - inf 1k 52 o All | Ill ΛEΛ
where A is the set of all nondecreasing homeomorphism from [0. onto itself, and [()) identity mapping. Also, we arbitrarily choose some E (0, 1) and some open set A C R².
From the assumption, we can find integer J(c) such that Σ
Me 1= U, st) < c, as well as an integer
(e) such that, for all 2 (e), we have P | U3 S <t) + P P(UU 24 < e. We fix such N(c) (we may abuse the notation slightly and simply write J, N when there is no ambiguity). Using Skorokhod's representation theorem, supports random variables (Up. Vin ... UT. VT)m21 and (Uo, Vo.... Ü, Vy) and satisfies the following conditions:
construct probability space (R,F, Q) that
L(U), VO : UU, V3) L(UB Vo,... Ü,g Vy) for all 2
L(Uo, Vo. "U,, V,) = L(Uo, Vo, ... Ü, Vy);
Uy any, Ug and V a.s. V, as n + for all j e [J].
Therefore, on (2, Q) we can define the following random elements (taking values in the space of càdlåg functions):
if 0g otherwise
y..W
Σjzo Vi [8g +ÜT+ t.rtu Uy. Ug+ür t....... +Üp,)) (8)
Y! VjL u. - U. Ü U (),)) Vs 202 0.
1=0
Note that (1) for the first jump time 4J we have U, = 0, hence Yo! Vo: (2) when defining yn.JJ we set its value on j0, U to be Vi instead of 0.
Since UT a Uy and V a.s. Vg
for all [J]. must have
lim d Y =
almost surely, which further implies that V Y.U
0o (ID, Now from
assumption that, for the jump times U1 + +U,. we have P = =x) = Va > 0,j 2 1, as well (13.3) in Billingsley (2013), then obtain
(Y y. (Y y!!')
(H.59)
as n - oc. Recall that the open set we arbitrarily chose the beginning of the proof, and e is also chosen arbitrarily. Now we observe the following facts.
Using eq. H.59
we can see that
lim inf Q((Y "... Ym yn... E A) 2 Q(.),,.... ,Y,,,, E A).
84

---

Published as a conference paper at ICLR 2022
The choice of (C)) and J(e) above implies that
J(c)
[Q(()))))) :.... Y y) E EA) P((Y,)... ,Y,) A) s SP
Ug
s <t.
f=1
10(0) ,.. ,,,,,,,,,l E EA)
-P(( , E EA)
11
J(c)
<P
Σ
UJ st)
+ Up +P 24) <c Vn N(c).
Collecting the two results above, we have established that
lim inf (((Y) Y A) P((Y,, Yi,) AA) -2c
From Portmanteau theorem, together with arbitrariness E > 0 and open set A, we can now conclude that (Yn: ( converges in distribution (Ytz:"". Y..).
The following lemma concerns the scaled version of the marker process * defined in eq. (H.13)
eq. (H.14). Obviously, it is jump process that complies with Definition H.1 When there
no ambiguity about the sequences (Mm)221 and (An)n21. let X!!)) a X A.annn From
eq. (H. (H.11) and eq. (H.15), know that for any n 2 1, ( is (Th,
Th- (Mm, An)) k200 (mrg(n....) x20 jump process (with the convention that - 0). Also, for clarity of the exposition, we let (for all n 2 1, 2 0)
(()) ( An) Tz- 1 An); s(") = (1h, An) - - (nn, An),
((") mTC(n
w(()) mmg((m,An))
Lastly, remember that Y is the continuous-time Markov chain defined in eq. H8.18 eq. H.221 and TG(+) is the random mapping defined in eq. (H.22) that is used to initialize Y. Besides, Y is ((S))a20, (Wx)x20) jump process under Definition H.l with So 0 and Wo (m)) (here 24 and X xr, is the index of the attraction field where the SGD iterate is initialized). The following result states that, given sequence of learning rates (Ta)221 that tend to 0, we are able to find a sequence of (An)x21 to parametrize X (n) X*mm...,, X X=.m..... several useful properties, one of which is that the jump times and locations of (") converges in distribuiton to those of Y (m))).
An so that they have
Lemma H.14. Assume the communication class G is absorbing. Given any my E G. x. E Qi finitely many real numbers (tr)k', such that 0 < t < 12 < <... < tw, and a sequence of strictly positive real numbers (1))221 with lim, th2 there exists sequence of strictly positive real numbers (An)22 with lim, A, O such that
* Unnde P2 (so X² = IT), as n. tends to xc,
(s("), w(n) ,(("), ((()), s(n) w ..... + (So, Wo. S1, W2, .......) (H.60)
(Recall the definition of Tg, I eq. (H.39 eg. (H.42)) Given any c > 0, the following claim holds for all sufficiently large:
sup P., (aj E [Ta An), Te(yn, An) s.l. X" & k20
U Ωt mIn EG) <
my EG
(H.61)
85

---

Published as a conference paper at ICLR 2022
Civen any O, the following claim holds for all n sufficiently large,
sup P (IIe & Glarge Vo [ummin]] mlc(mn E EG) k20
<P(Geomm(p)) 2 u) +e = 1,2,-...
(H.62)
For any LE [M],
lim x (xii $
U
B(mj. An), Xg E
U 2 vs E [0, =
n-+00
myEee
j: myeG
(H.63)
where p² > O is constant that does not vary with our choices of Qn or An.
Proof. Let
U/ - qj - -
Ui _ (5}EE,,)
so from the definition dj,k we have 9j 1 + kJ El: m.EGmall Pj,Pt,,-
In order specify our choice of (An)n21, (A()))z2; (7(6)))20 follows. Fix some 0 (0, 1)/2). Let A 7(0) 1. One can see the existence of some (A()))21 (f(1))s2 such that
consider construction of sequences
( E (o, 1)/2], 70) E (0, n - 1)/2] for all ju 121;
(Due Lemma H.7p for any j > 1, 7) E (0, 7()),, (remember that and are the fixed constants prescribed in the description of the lemma)
Pa (255 (1, A())) < <n 10 (()) = >
25
For definitions of oh G k. I TG see eq. H.6 HH.6-eq. H.11
(Due to Lemma H.11 for any j 2 1, 77 E (0. 7()).
Ps (1, (()) (7. A() n 1 (1. A()) =2 (n. A()) - - -
< 1/22
uniformly for all & 2 0 and all my Grmmll mig E Glunge Also, by definition of o* and T", we must have
P- The (1, (() - (n A()) = 0. 18 A()) = i (n. A())) = -1 =
for all A and mi, Glarge
(Due Proposition G.25 for any j 2 1. 1 E (0, 7()).
1
V/1.82 1
27 + exp (-(1
2) )0i, 11)
<P_ ok+1 A(j)
Tk (1.
A (j))
76 = is
(1, =ti)
u,
1
Vigia + -
2 + exp (-(1-
2 14, 2
4i.
uniformly for all k > all 1/21, and all mi, E Glarge mig mip E G.
86

---

Published as a conference paper at ICLR 2022
(Due to G being absorbing and, again, Proposition G.25 for any 7) E (0, mT)).. (Recall the definition of Th, Ia in eq. H.39 eq. (H.42)
1, for any 2
Px (I AG)) - i2 12 (1 A (i)) = in < 2
(H.64)
uniformly for all k my G,m & G.
(Due to Lemma H.9 There exists some p" > 0 such that for any j 2 1, for any j 2 1, 1) (0,),))),
Px m at Ia (v.06)) $ Glarg Ve E [ummin]] [n A()) = =(1)
<P(Geom(p") 2 u) + 1/25
(H.65)
uniformly for all l 2 0, 2 and mi, E G.
(Due Lemma H.12 for any j 2 for any j 2 17 E (0, 7()).
Pa X
B(mj, A()), X E U
2, vs E [0,4 < 1/22
my EGuv
3: myEG
(H.66)
uniformly for all & E [A]]
Fix such (A()))2o, Define a function J(.) :N + N as
J (n) 0 max 0: 77() 2 n))
with the convention that max 0o. Lastly, let
An = (J((n) Vn 2 1.
Note that, due to lim,, ?n = 0, we have lim, J(n) = oc, hence lim, A, = 0. Besides, the definition of J(-) tells that in case that (n) 2 1 (which will hold for all sufficiently large), the claims above holds with Th and (n). In particular, by combining lim, J(n) = oo with eq. (H.64heq. (H.65)eq. (F.66) respectively, we have eq. (H.6Teeq. (H.62)eq. (H.63).
Now it remains to prove eq. (H.60). To this end, it suffices to show that, for any positive integer K, we have (5(*), wr("), .... stn) w(()) converges in distribution (So, Wo.... SK. Wx) as n tends to infinity. In particular, note that Sp 0, Wo To (m:), so Wo m, with probability Pı.j if my E Grmaal and Wo = m, if my E Carrg
K
For clarity of the exposition, we restate some important claims above under the new notational system with 5(m) W(()) _(()) w(()) we introduced right above this lemma. Given any e > 0, the following & claims hold for all sufficiently large:
First of all,
Ps ((mm)
<// W/(") =m:) m.)
(H.67)
* For all k > 0 and all mi, Gsnaal mig Glaage
P (( 5(m) Thr w(n) mig
W((n) = ms)) - Piita <e.
(H.68)
For all k 2 0 and all m., Glargo
Ps ((m) ((m) = 0, W(n)) = min
( mmi =
(H.69)
87

---

Published as a conference paper at ICLR 2022
For all k 2 0, all mú, Glare, m4a G and all u >
(1 + c)q:, u) M -e
c+ exp
91
SPr ( s(()) u, W(n)
w((m) mss )
h+1
= =iiz
SPr (5%) s(1)) u u Wk+1 = mia
W =mis
vi +
Sc exp ( (1 e) c)
(H.70)
gi:
Here is one implication of eq. H..68) Since |G| I minn we have
Pz (5 ((())
2 a
W m4,
< Pomin
(H.71)
for all k 2 0 and mi, E Gamall
Note that for any may mi, E Glarge and any k: 2 0
Pa (()) k+1
(n)
5 u,
(n) k+1 Mi2 w(() = =mi,
1 # Jr
(())
5( > u, k+1 m2a w.()) =mm))
k+1
+ Σ
Pa (2(1)) k+1 - -5(mr) 2 (u - 8) vo, Wayr mia k = -mis)
is: may EGummll +20
Pa
(n) k+1
(x) (()) ds, k+1 mig
w mma
Fix some is with mig (amall On the one hand, due eq.
HH77
Pz
[n)
k+1 - 5(m)
k+1 2 u s,
Wi())
k++
mia W(()) = mis
-r
((m) s(") ds, W(n) 5+1 mis
w mis
k+1
S.minn.
On the other hand, by considering the integral on (u - nhe xc), we get
J Pr
(n) ((n)
k+1 -ktt1 2 (u 8) v 0, w(nn) k+1 mia
W((n) mis
(000
((()
s(()) = ds, W(nr) = mis
w(()) -mss
k+1
P2 Wh(n) mia W(n)) mis)
)e((u,0))
Pr
((x))
K41
((m) = ds, (())
k+1
= mis
((()) = min
2 2(P - c)
+ exp ( (1 + e)q,, 2)
Vir is -9)
gia
due to eq. H.68)) and eq. H.70 Meanwhile,
Po (n)) k+1
3 2 (u (c-ss) V) Wi(n) = mi
W = m4g
Pz (())
-((1) ((n) ds, Wi(m) mis
(-(1)) =mi,)
Vinis +
<(Amint + Pig.ia + e) E + exp ( - (1 e)gi, u)
41
due eq. HH68 eq. (.70 and eq. H.71
88

---

Published as a conference paper at ICLR 2022
Therefore, for any M41 mig E Glare and any k 2 0.
x k+1 _ u, w(()) = mbi W mis
lli2 + 81 J1si82 + Ligg mig EGuoall
<g(e) exp ( (- (1 e)qi, u)
4b
<g(c) exp ( (- (1 c)q,, u)
021.22 gi1
(H.72)
and
P7 ((m) 5k > u, w(() k+1 = mia W mis
llie + in i130isi8 + -igg mig @Qunall
2 g()) + -epp ( (1 + c)q,, u)
96
g(e()) + exp ( (- (1 e)g,, u)
(H.73)
where 9² = maxi di and
1tc
g(c) c qº
mmin + c)e
n'iin
+ +nmin (e + {
Mann (Teinn + E + 1)(1+ )
Note that lim g(() = O.
Now we apply the bounds in H.67 eqq
H.72 eq. (H.73) to establish the weak convergence claim
regarding (S())) w(n) ..... S(a) K
Fix some positive integer K, some strictly positive real numbers (sx) or sequence (w))k=o) E (Gluge) K+1 with wa mi, for each k. and some 0
W(()).
such that < ming-0,1,".. {8x). On the one hand, the definition of the CTMC Y implies that
P ( < tor Wo wo: S 88 and W₁ = wr Vk E (K])
K
-P(TT((m)) = wp) II
Sx > sy and Wt = Wk
WA. = Wk-1)
k=1
-(1(m= E Glarge io i} + {m; E Gimall piooois)
II exppp- 52) k=1
Ba- 1.88
Ti
On the other hand, using eq. (H.67 (H.72 eq. H.73) we know that for all sufficiently large,
P₂ (5(m) 80, w/()) (n) = wo; s(") sy and w( we Ak E [K])
2(1 <) 1 1mm, E Caagg to = i} + 1 {m, E Gamall G (Pto. - -c)
In 1d6
- g(t) - eexp(-(1 + e) 3k)
Pi-
k=1
and
P₂ ( su, w(()) = Ww: s(a)) sy and w(()) wh Ak E [K]
1 {m, E Clarge in = i) + {m= E (amall) (pions + s))
II k=1
((()) exp(-(1 18) Pig-1d0
Tie
89

---

Published as a conference paper at ICLR 2022
Since >> 0 can be arbitrarily small, now obtain
lim Far (5) < to, wor)) = wo; s(r)) > se. and W( = my Ak E [K])
1-+00
-P(S0 So < 80; Wo wo; S > ae and W₁- = wh vk E [K]),
and the arbitrariness of the integer K, the strictly positive real numbers (5k == and the sequence (ux) (wx) (Gtugg) K+1 allows to conclude the proof.
To extend the result above to the case where the communication class G' is transient, we revisit the definition of the yt eq. (H.26). Let G Glare U [t} and let mo = f (remember that all the local minimizers of f on [-L, L are m,, ... m.in). Meanwhile, using q: and qu in eq. (H.20 (H.21 we can define
-b.t 1 = 0}
if 1, j j21 if 0
Eje(naul @G 9i.j
if 2 j=
and 46 = d! q. Vi 2 1. Next, fix some i with m, E G and x Ω2 Define sequence of random variables (5 )220, (W2) such that S = 0 and Wo = (m)) Wi = t1(W, & Glarge] + Wol (Wo E Glare] (see the definition of random mapping म. in eq. H.22 and (for all k 2 and i,j with my, my E G)
PP [ k+1 = m,, S/41 > t w = mnj (Witz)) (5)
(H.74)
-P -P(Wi++ k+1 m. st > 1 W = m2) expl- 1) 2 11 0
(H.75)
Then is easy to see that Y (mm) defined in eq. H.26 is ((8) (Wi) jump process. In particular, from at any state that is not t (namely, any my with my E Glage), the probability that yt moves to t in the next transition equal to the chance that, starting from the same state, Y moves to state that is not in G. Once entering mo = t. the process yt will only make dummy jumps (with interarrival times being iid Exp( 1)): indeed, have % % 1 and % 0 for any 2
implying that, given W mo t. we must have W that Y is stuck the cemetery state after visiting it.
= t. These dummy jumps ensure
Similarly, we can characterize the jump times and locations of the jump process Att.m. (for the definition, see eq. (H.25). ). When there is no ambiguity about the sequences (y/n )n (An)n21, let Xtt((a) = At... and Xt.(n) = M.M....,, Also, recall that To defined in eq. (H.4) is the step when X exits the communication class G. Now let (Ex)x20 220 be a sequence of iid Exp(1) random variables that is also independent of the noises (Za)a2 (so they are independent from the SGD iterates X.). For all n 2 k 2 0, define (see eq. H..6 (H.II) and eq. (H.15) for definitions of the quantities involved)
(1h: An) A T (TG(TT) Tm) Th (Tms An) 0
if T An) < T (Ta(TT)) Tmd
(m))
k
otherwise
T (m, An) A T (n.)) nm) T (1n, An)
T ( A.) < T (Ta(Na); ma) otherwise
st (m))
Ex (m.inn)))))
if (n,, An) < T (To(TA); Tm)
wil(())
otherwise
if Th (m. An) < T' (TG((T)) m.) otherwise
w (m))
with the convention that r-, = O. Note that T (To(),), nm) is the scaled timestamp for (() = X
corresponding To())) hence T (TO()))) m)) = min(t 2 0: (m) U, myec 2z). One can
90

---

Published as a conference paper at ICLR 2022
(m) that Xt.(n) is a ((5) J220 )220 (WIx()) )=20) jump process. The next lemma is similar Lemma H.14 discusses the convergence of the jump times and locations X1.(m) on a communication class G in the transient case.
Lemma H.15. Assume that the communication class G is transient. Given any m. E G, E 94 finitely many real numbers (fi) such that 0 < 1 < l2 <... < tws and a sequence of strictly positive real numbers (1n)221 with lima-50 1/n 0, there exists a sequence of strictly positive real numbers (An)n21 with lim,, An = such that
Under Pz (so X² = x)), as n. tends to oc, (s).(m) w.l. " sss:(n), wish))
sh.("), wyl.(n),...) = (5) W WT, W ...) (H.76)
For any 1 E [
lim P2 (xt(()) &
U
B(my, An), x1.(0
U
2 Vs [0.til) =
1-000
J: myEGlu
J: myEG
(H.77)
Proof. Let
VjJk A (E,,),
Vj. 21, 1, j
Σ
Vmj E G
Pjt
PJ
3: my FG
Σ
Σ k: mauaa
PjAPP.. Vmy E Glarg
9j-1
Vj.k +
kam &G
In order to specify our choice of (An)n21, we consider a construction of sequences (1))) (m(5)) as follows. Fix some / E a0 - 1)/2). Let ( = -0) = 1. One can see the existence of some (A(j)) (7(1));2 such that
(j) e [Ú 1)/2]. 7() E (0, m 1)/2] for all j 2 1;
(Due to Lemma H.7) for any j 2 1, n E (0. n()),, (remember that x and are the fixed constants prescribed in the description of the lemma)
P. (o (11, A()) n ( A =1 =1)
1
2
G For definitions of oh Th 1 I see eq. H.6 cq. H.11
(Due to Lemma
H.11 for any j 2 1, n E (0. 7())}
Pg T ( A()) on ( n 1 (n. A()) =2 1 (1. A()) -
= -Pi,,
1/20
uniformly for all l 2 0 and all m,, Gasaal mia & Gamall Also, by definition of o* and 7", we must have
Pr T (1, A()) of (1, A()) 0, 0, 1 A()) = in (1. =11 =1
for all A 2 and mi, E Glange
(Due to Proposition G.25 for any j 2 1, E (0, 7()),,
20 + exp (- (-(1+ (1+ 2) 19i, u)
1/1.22 25
<P3
Oi (17, A(j)) - Th (n. (j))
> u, k+1
= iz
(1, A()) = =(1)
1
1
14, V4 12
+ 27
1
22 + +exp
(-(1 (1
23 14,14
gi
91

---

Published as a conference paper at ICLR 2022
uniformly for all k 2 1, all 1/2', and all ma E Glare mig G.
(Due Lemma H.12 for any j 2 for any j 2 (0, 7()),,
Pz (x(m) l
B(mj. (j)). x!m) E U Gy Vs E [0. < 1/2 (H.78)
&
myEGle
ji myEG
uniformly for all le E [M].
Fix such (2 Define a function J(.) : + N as
J(n) 0 max 0: nT) 2 mm
with the convention that max = - 00. Lastly, let
A (J(()) Vn 1.
Note that, due to lim, The = 0, we have lim, J(n co, hence lim, A, = 0. Besides, since
x-F (n)(n) xx() given X Ug: m,ea Ω2 for all [0, t). by combining lim, [nn oo with
eq. (H.78) obtain eq. H.77)
Now remains to prove eq. (H.76). To this end, it suffices to show that, for any positive integer K, we have (5)-0)) wri(()) ,.... () w converges in distribution (s), W SS W ) as n tends to infinity. In particular, due to introduction the dummy jumps, we know that for any with Th (mm, An) 2 Ta(M)) (in other words, ((.(()) has reached state t within the first l jumps) we have Sl.(n) Exp(1) and wr.(()) k+1 = t. Similarly, for any k with S +... St T have $441 ~ Exp(1) and W 1+1 = t. Therefore, it suffices to show that, for any positive integer K, any series of strictly positive real numbers (sx) (s*) any sequence ( (G) K+1 such that wi + f for any j < K, indices in such that wt Mi, for each we have
lim P2 (5%.6v) n=000
lo, w/. wos Sk St.(n) >
85. and W wR mk [K])
P $t < s0 Wt wo: st > 8k and : wy Vk E (K7)
(H.79)
Fix some (sk) (wx) k (c) K+1 and indices (ix k satisfying the conditions above. Besides, arbitrarily choose some é > O so that < mink=0,-. K 8g- To proceed, we start by translating the inequalities established above under the new system of notations.
First of all, for all n sufficiently large, (remember that z and are prescribed constants in the description of this lemma)
Ps St (a)
<nn Wil(()) =mi) > 1
(H.80)
For all l 2 0 and all m. E Gsmall mia E Glange it holds for all n sufficiently large that
Pz (tt, - -st. (t Thr wf((()) maa
W W = =ma,) -Pi
<
(H.81)
On the other hand, for all and all ms, Gamall it holds for all n sufficiently large that
P2 (m)
(n) o wt.(m)) = w1.600) =mi))
Σ
Sp un
Poz
f2r myg $G
Pr (st.(m) st.(m) nn
9 wi.()) =t whtt(n) mi.) -Por.t <E.
(H.82)
* For all k 0 and all mi, Glarg, it holds for all n that
P. st (t(())) gr(()) = 0, w = mi,
Wit
= mi, =1
(H.83)
W
92

---

Published as a conference paper at ICLR 2022
For all k 2 0, all mi1 E Glarg, mia G and all the following claim holds for all
sufficiently large:
Pi -
-e expe (1 e)q(, u)
qi
SP, (31.()) - -spp(n) u, W Whith miz
(/l( mi.)
<P3 (5tifl)) st.((a)
m: With)
Whi(()) =m;))
mig
0/ +
<c + exp ( - (1 - c)qi u)
(H.84)
On the other hand, for all h 2 0, all my E Glase, the following claim holds for all sufficiently large:
-E+ Eizi m.g $G V/1.22
c expexp exp (- (1+))q,,)) u)
9b1
SP.
(t.(m)
:)(((n)
> u, wi(l)) =t ( -mis) k+1
k+1
SP₂
(t.(m)
st.(1))
> nhe wi(n)
=t w min
k+1
k+1
Eis: myzEG Vi
Se + exp (- (c)(()q u)
(H.85)
Here one implication of eq. sufficiently large)
H.88leq. eqq H.82
Since |G| min have (when ni is
P2 c())
5
0 2
m/
W (m)
mia
< Mimin
(H.86)
for all and mi, E Gamall
Note that for any mi, m,, Glarge and any k 2 0
Pz (ttnm)
-st((n) u, w-t((a) mi,
wr(((n)) mis
(12 # 11 p St.(a) st.(m)) > u, wi(n) mig
wt.(0)) mis
k+1
Pr
Skt1 Sk+t1 > (u (u-3) 8) v0, Wi más
(t((n) ().
Wt.(0))
mis
ig: mig Gumall $20
(1)( k+1
sl.in) ds, Wil. mis
w.l.inn min
Pr
Fix some is with mi, E smalll Due to eq. H.86
f /. Ps (=(st.i)) k-1
st.(n)
-5t.(n) 2 u 4-
Wht(()) k+1
Wit(1)) = -ms
=mi,
nf]
PPx
(l.(m) -51.(n) do, Wh((w) mms W...1n) mma
k+1
S.minc.
93

---

Published as a conference paper at ICLR 2022
Meanwhile, by considering the integral on (u n/a oo), we get
S Pr
st.(m)
5-1 2 (u- 8) vO, W (m)
3t.(m)
wht((n) mis
k+1
k+1 =
Ps
(l.(n) k+1
(t.(n) h
= ds, W Whln) k-1
= m4s
w m4i
Po W (i(())
mii Wi.( mis)
)E(U,0))
k+1
(l.(() k+1
- 5 sl((m) = ds, W.l.in) 1-1
w/(((a) = mii
Pa
= mig
2(Pig,2 - () -c)(- (-) exp (1 e) u)
Vi
Ois
due to eq. (.8I and eq. (H.84). As for the upper bound,
Pr
(l.(m) k+1
k-1
2 (u 8) VO,
Wi.(m) = = k+1
Wh
mis
vE((unnh.on
Pa
(t.(a) k++1
st.(n) ds, Wi mig
w mii
Vin +
< < + Pig,12 + E)(
exp (- (1 c)gs, u)
9:
due to eq.
H88 eq. H.84 and eq. (H.86).
Therefore, for any mii; M2g E Glarge and any k 2 0,
(t)(m)) 12 k+1 st.(m) u, w (n))
w (x)
k+1
mia
Mis
1f22 F i1 i2 +
Eist mig EGunal P2
Sg(c) + exp ( ( (1 =c)g, u)
96
Sg(()= + -exp ( ( (1 <)gg, u)
961.2
(H.87)
9ir
and
Pz (st//1)) -s%/(n) > u, w wii((n) mii
w/(((n) =ma,)
liis & is i1 + Eis: mig &Gomall Pig Pig.22
2 g(t) + eexp ( (1 + c)q,, u)
96
2 - g(()) + exp ( (1 + e)(in
(H.88)
Oi1
where 9" = max; qi and
1t
g(t) A 2 0²"
Amin + te) r²
Mmmn
1 ). + Remi (e + ( + Aminn mmin((nmiin (mmine e + c + 1)(1 +
gr)
g
Note that lim g(c) =
On the other hand, for the case where the marker process t.) jumps the cemetery state t from some mi, E Gglarg, note that
P
st.(m)
(t.(n) u, w = wr((()) mi,))
=P =P3
Skt1
(n)
Sk
(n.(n) > u, wi.(n) =t wi((") mis
+
Pr
(tmm)) gl.nn)
k+1 -sk+1
> (u 8) v 0, Wi((m) =t (Pt.(n) =mis)
mis)
Ag mig @umall $>D
P
kk1
(n))
Sk
(l.(n) ds, Wi(()) mis wr.((n) =mi.)
94

---

Published as a conference paper at ICLR 2022
Arguing similarly did above by considering the integral [0, u, 1u--1 -//1 and (u - tu- 20) separately, and using eq. (H.82) and eq. (H.85). we then get (for all n sufficiently large)
Px st.in) s(.(m) > u, W =t w1.))) =mis>
k+1
Ligamin & Visi2 + Eigme, @Guail Vig Pi22.
Sg(e) + exp (- (1-e)q.s
4i1
=g(e) exp (1 E)q;,
(1-e)q,,)) u)
gintt
(H.89)
qia
and
Pr
sti(n) st((n)) k+1
> u, W =t W = -mis)
k+1
Eisiman FC V/2 + Eigim, EGuaal V V22it
Σ
g(t) + exp ( + +) u)
9611t
9 + exp (1 + c),, 2)
(H.90)
For simplicity of presentation, we also let 9A.0 gitt and Pj.0 Pist First of all, remember that have fixed some series of strictly positive real numbers (sk) k some sequence (w+) (G) K+1 such that w, + i for any j < K, and indices is such that wa = Mig for each k. The definition the continuous-time Markov chain y implies that
P ((So- So < to, Wo = woi Sk > sy and Wi = ww Ak E (K])
K
-P(TT((m) = wo)
II
Sx > and W₁ = m, W wk-1)
km l
-m1(m; Glarge to = 1} 1 [m, E Gamall Pio.
II
exp(- exp(-9in-18 8k) Piz-1
k=1
On the other hand, using eq.
H.87 eeq. (H.90). know that for all sufficiently large, P. (((a) so, W(()
(m) wp: s(") sk and w((a) ws Ak E [K])
21 - c) 1 [m+ E Clurge to i} {m; E Gremall (Pio. - v)
K
g(() + +exp(-( + e) 88) Ba-
Din-1
k=1
and
P (())
sp, w(n) wD; S(1)) s and W(()) me Vk. E (K])
1mm E Clarge in =i i) + 1im E Commall ( (Pionis + e))
K
II
((()) + exp(-(1 - -)qi2_1 8k) lin-1,62
lik-1
The arbitrariness of c > then allows us to establish eq. H.79) and conclude the proof.
Now we are ready to prove Lemma H.4 and Lemma H.5
Proof of Lemma H.4 From Lemma H.14 one can see the existence of some (An)a2 with lim, A, 0 such that eq. (.60 eq. (H63)) hold. For simplicity of notations, let (n)
mm.. (n) X and let [ tg
95

---

Published as a conference paper at ICLR 2022
Combine eq. (H.60) with Lemma H.13
and we immediately get eq. (H.23 In order prove
eq. (H.24), it suffices to show that for any
limsup Par
X (ra) th
U
B(mj, An)) S de bk [K]]
1: my EG
Fix c > O, and observe following bound by decomposing the events
P2x
x (n)
$
U
B(mj,
j: m
<P'
x
(m)
$
B(mj, An), x(")
U
a f (0.0)
j: M
j: j:myeG
+Pp 3t
[0, f such that x(") #
23)
j: myeG
Therefore, given eq. (H.63) suffices to prove
limsup Pa 3t E [0, E] such that x,
U
22
s 3c.
(H.91)
myEG
Let
Ty A min ( 2 0: x(() E U
B(m,, 24,))
janeEG
x(n) =j - x(a] B(m;, 2An)
Tyu)
Ty(a) A minft > I(m) x!"))
U
B(m,,2
EG, (+1(2)
(n) =j x (n) p B(m,, 2An).
Building upon this definition, we define the following stopping times and marks that only records the hitting time to minimizer in large attraction fields in G (with convention k (a))Jar (-))
-1, (a). 0, T(m) = 0)
k bn). (R:) A min (1 > k (a), (k: 1): my!ns E Glarge
r lar. A (m()) Trlr)
(1))))) ( (n), A ((m) (k))
Now by defining
((m) 2 #(k 2 0: T(a) S {1),
nin (t) A # 0: ((m), large t},
J(m A # 0: T(1) E (),
we use (L) to count the numbers of visits to local minima G, and sing (1) for the number of visits to minimizers the large attraction fields on G. ((m)
(s, t) counts the indices such that at
r(()) minimizer on G is visited and regarding the hitting time we have T.()) [s,
First all, the weak convergence result in eq. (H.60) implies the existence of some positive integer (e) such that
lim sup Pa (J(() (f) > N()) <c large
Fix such (e). Next, from eq, (H.62 we know the existence of some integer K(e) such that
lim P.,. (()) (T(f). (mrg..arge)) > K(() & </N((e).
R20
96

---

Published as a conference paper at ICLR 2022
Fix such (c) as well. From the results above, know that for event
(n) (Jinge (t) S N(e) n { ((m) (T Th Trn)), s K(e) Ak E [N(e])
we have lim sup, P2 (A))))))) S 2e. On the other hand, on event An()), we must have
(1) S (e))((())
Meanwhile, follows immediately from eq. (H.61) that
lim sup sup P 3t E [_ " such that x! $ U U2)= ((E)((e))
n
420
J: myEG
hence for event
A2(n) - (x
U
2 E (0, The Moxoo
J: myEG
we must have limsup,, Px ((A2(1))))
S. To conclude the proof, note that
(.(n) S (f) & ((e) (x(
( E
U
2 Vt E (0, mox
j: myEE
=[TMOK( 2 28 { x(
U
By Vt [0, ( Morkee
je j.meEG
c(x
U
22 vt E [0,
I myEG
we have established eq. H.99I.
Proof of Lemma II.5 From Lemma H. one can see the existence of some (An)a21 with lim, An 0 such that eq. (H.76) and eq. (H.77) hold. For simplicity of notations, we let Xt.(a)= Xt An, Xt.(n) X..*%% and let ty
Combine eq. H.76) with Lemma H.13 and we immediately get eq. H.27 In order to prove eq. (H.28), note that
x (m) 1k
B(mj, An) and x11(()) *t}
my Corrp
(x) bx
U
B(mj. An) and X(.(m))
U
2 vs E [0,
my ctrrp
J.
so the conclusion of the proof follows directly from eq. (H.77
I FIRST EXIT TIME OF TRUNCATED HEAVY-TAILED SGD IN Fd
1.1 MAIN RESULT
The object of interests is the following truncated heavy-tailed SGD iterates
(x) x²(x) po - [V) (z))=) Vk 20
where the initial condition is prescribed by Xo(x) = x, f is real-valued function on Rd 17 is the learning rate, k221 is the sequence of heavy-tailed noises, the standard gradient clipping operator is (v) = min 1, [|l]| denote the conditional law on {X] x).
with ||-|| being L2 norm. To ease notations, we also use P,, to
Specifically, we are interested in the first exit time of X from a domain G, i.e. the stopping time
a (n) min(n 0 X $ G}.
We work with following assumptions.
97

---

Published as a conference paper at ICLR 2022
Assumption 1.1. The region G is connected, bounded and open, and 0 G.
Assumption 1.2. The function f is smooth, i.e. f E (R
Assumption 1.3. The boundary set 0g is a (n -)-dimensio manifold of class Ca such that the vector field r(-) of the outer normals on 0g exists with
Vf(e) n(v) 2 co
Vu E ag
(L.1)
for some constant Co >
Assumption 1.4. For V f(0). the Hessian of f(-) at point all the eigenvalues are strictly positive.
For any x E G. let x (z) be the ODE flow with z0(x)) = solving
(x(x) - (xx(2() VE 200
Assumption I.S. W.L.O.G., the origin 0 is an attractor of the domain, iee. Vf(00) and asymptotically stable in G in the sense that
lim x xx(x)= =0 Vx E g. E-p00
We have the following assumption regarding the heavy-tailed noises (Zn)n21. For any x E R + define the following polar transformation
T(x) = || x/||||
(I.2)
with (x) = ||=1|, 1 To(x) = x/ ||=||. Also, let O = (0) Assumption 1.6. EZ1
Besides, there exist a positive integer m 2 1, a sequence < 01 02 < < < oo, sequence of slowly-varying functions (I,,... Imm). sequence of probubility
measures ... ..,, Sm) on the unit sphere Sd-1 with support F3 A supp(S,) as closed sets on Sd-1
such that
F,F = for any i + j:
For any j E [m]. define the cone E, e (F,((F)) U U in Id and measure p-) = PZ e
-n Ez). we have
1= .1)(1)
T dr x d0) - Vay (dr) x Sy(d0) as 1 + 00
(1.3)
in the sense of M(E (O).
Here Va is Borel measure defined on (0, oc) satisfying
It 00) = ao Vt 0:
For measure p(O) A p(Z E E1)) and any > 0,
1 pp)()) T (ll dr d0) - 0 as a-
(1.4)
in the sense of ([([)(O))
For any j E [m]. the measure Sy is absolutely continuous w.r.t spherical measure on Sd-1
function / R4 + + slowly varying (at oo) if limy ((tx)/(xx) holds for any æ For details on Mcconvergence and regular variation in general metric spaces, see Lindskog et al. (2014). Here we state one implication of the assumption. For any j = 0, 1.... m, let
H,(x) - P,, (1V CRRRR: | 2 x)).
(I.5)
The assumption above immediately implies that, for any j = L, .,, H. is regularly varying (at co) with index -a; (denoted as H, RV-a,) ) namely lim H,ttr)/H (t)) x ay vr 0. Meanwhile, for any a > 0, Ho(x) = o(1 /x") as x oc.
The asymptotically behavior of the first exit time hinges the following geometric character- ization of the domain G. For any integer & 2 1, a sequence of strictly positive real numbers (12, .4.) with i 0, sequence of non-zero vectors (wi,..., wk) and some " > 0, let
98

---

Published as a conference paper at ICLR 2022
((h) (t1, ,t,,,,,,;,)).........))) w()) (W1,'.. wx),, and define the ODE path with &t jumps (clipped size b) by ((k) (W1,... wk), define perturbed ODE path ""
xx (0; (8) wl) (o(())));; dā x; t(h, w())) dt
k
-1 - (1, (h1)) Ai & (4,4 41 12.
4
x (1, (8) (h)) 2t xz (0), w) (p(())
if Σ 4. for some
with the convention that g(t-) a lim (s)) for any function g. Also, when 7) we simply write E. Now we can assign a cost to each jump wi based on the direction using the following function:
aj 1 8 otherwise.
if w + 0, w Eg,
J(w)
(L.6
Given any perturbations described by (R) destination of the flow
we identify the E {0} Rt-1, wi(h E (R.
h(k:; (8), (k)))) 27 Lx (k) (k))))
Besides, define function (kw) (ii,.... {m) if # (i E [k:] E Ej} ij for all j E [m]] This allows us to define the following configuration sets
A(i ... 2m) - ww (R)
t I(w(()) = (11,... km). k
i5}
(I.7)
any (i,,-.. im) E Nmh i.e., some set of jumps (kk) is said to have configuration (i,, ... or belong to the configuration set A(t1, im) if the number of jumps in cone Ey is equal iz. We can also define the cost for each configuration as
1m)
J(i,, (im
Σ ag 1)1g-
(1.8)
Now can characterize the minimum cost to exit G:
Jg e min Σ J(w)): E (N) E {0} x R4 (K) (R)
1=1
to O; (K), (k)) g).
(I.9)
From the boundedness of G, one can see that Jg < co regardless of the actual value of the clipping threshold >> O. We need the following technical assumption regarding the configurations of jumps that can trigger the exit with the minimum cost.
Assumption 1.7. There exists only one array (f1,". im) E N such that Σグ 1) Jg and (for En im)
38 (0) x XR (k) E A(ii,... (m) C
O: (k) (1) G.
We use 8" = (ij + to denote the unique configuration in Assumption 1.7 and k = Σド 13. The implication is that, for set of jumps {(k), (*) with any other configuration, of the following must happen: (i) this set of jumps has cost strictly higher than Je: (ii) this set of jumps cannot send the ODE flow out of g.
99

---

Published as a conference paper at ICLR 2022
Mcanwhile, introduce the following concept the coverage of certain configuration set.
g( ,mm) - { O: tK) w(k)):
m
k = Σ b ((h) E x R wh) E A(in,-.. im), & [0, Σ L1}. ==1
(I.10)
5=1
It is easy to see that, for any configuration (i..... km) with cost (+1: ... im)) S Jg. the coverage (i1,, im) is closed set, and the following technical assumption holds for (Lebesgue) almost every b > O. Here the distance d(A, B) A,yE || yll for any A, B C R and A is the
interior of the set A.
Assumption 1.8. For any (i1, (m)) N one of the following must occur:
d(g(i,, km) 09) >
(g(i,... 2m) g #0
Note that with i and kt defined above, we can define the following mapping h from r
(Ti, rr=),, 0 (0, .kx) E (8u-1)#" = (1,," (x-) E {0} R$ such that
h(r, 0) = at L: 0; w)
where w (w:) with r,,;, Also, we introduce the concept of type for configuration 8". Specifically, define
j(h ... (mm) {(i,... ... . and for any j j(i"), we say that w (w1, ... (
E
#(n: jn = =k] in Ak E [m]]
) E A(i*) has type j if
wy E E,, Vi [k*].
In other words, based on the direction of each jump in w E A(i") group them into different types. Note that [i({") < co. Now for any type ji(),, define (Borel) measure on IR (gd-1) x
11 as
k"
Mg
Vag.
II
St.) 1=1
meeb --
where, for any the measure V/ is the Borel measure 00) with v. (I, 1/x1+o Lastly, define measure as = Σjej(t>)) Pg. As will be established in Lemma K.2 the following technical assumption is also very moderate one since it holds for (Lebesgue) almost every b > 0 under the current setting.
Assumption 1.9. The set h-1---1-- (BG) has mass under the measure p.
Having specified the problem setting, we are now ready to present Theorem I.1 the main result of this section. The implication of the theorem is clear: under proper scaling, the first exit time (n) converges in distribution to an Exponential random variable. Moreover, the scaling A(n) is roughly of order 17 J- implying that the first exit time (n) roughly of order 1/17 1+9 the learning rate 7) approaches 0.
Let Assumptions I.1 1.99hold hold. There exists a function X(1) that is regularly varying (as n 4 0) with index (1 + Jg) such that, for any and any > 0,
Theorem 1.1.
lim PP (a(())()))) > t) exp(-90 n40
where the conslant q ===(h>>(G9)
The proof is provided Section J As a concluding remark, we stress that the order of the first exit time is dictated by Jg. the minimum cost for exit introduced above. Bearing obvious similarity to
100

---

Published as a conference paper at ICLR 2022
the first exit time analysis of SDE driven by heavy-tailed Lévy processes in Imkeller et al. (2010a), our result can be viewed as a natural generalization when gradient clipping is applied and heavy- tailed noises may not always align with finite number of lines. In Imkeller et al. (2010a). the unclipped setting implies that the escape from the domain G can always be achieved with one big jump. However, big perturbation (in noise) along different directions may correspond to different heavy-tailed indices, i... induce different costs given our definition of cost function ((w) in eq. (L6). In Theorem of et al. (2000a), we see that the order of the first exit time is determined by oi, the smallest heavy-tailed index. The underlying reason is that the exit is almost always trigger by single big jump with the smallest cost. Similarly, in our setting where multiple jumps are required for escape due to the clipping mechanism, we see that the order of the first exit time is not dictated by the number of jumps or the accumulated distances of the jumps, but the smallest possible accumulated cost defined as the summation of (w,) where w, 's are the jumps the lead to escape from G. As detailed in the proof, this is because such jumps with the smallest costs Jg dictates the most likely way for exiting G.
1.2 A SPECIAL CASE: UNIFORM HEAVY TAIL INDEX ALONG ALL DIRECTIONS
As stated above, Theorem I.T deals with general case where the noise distribution is allowed to have different heavy-tailed indices along different directions (see Assumption [.6]. As a special case, is worth noticing that if a single heavy-tailed index a can be used to describe the tail behavior of noises along any direction (this can be easily guaranteed if the heavy-tailed noise is manually injected into light-tailed setting), then the minimum cost Jg will be equal to 1" (a 1) where I" is the minimum number of jumps required for escape. The readers can see that this is a natural extension of our R² results, implying that the same strong preference for "wide" minima still hold in I under truncated heavy-tailed SGD. To specific, we work with the following assumption about the noise distribution.
Assumption I.10. EZ = 0. Besides, there exists some a > 1, a slowly-varying functions I, probubility measures S on the unit sphere S with support Supp(S;) Sd-1 such that
For the measure v(.) = P(Z E. Ez), we have
..(((t) . T (t: dr x d0) + va (dr) x S(dB) as t - 00
(I.11)
in the sense of M(R))). Here vo is a Borel measure defined on (0, oc) satisfying
Vallt, 00)= t VI 0;
The measure S is absolutely continuous w.r.. spherical measure on Sd-1
Compared to Assumption L6 one can easily see that Assumption 1.10 is a stronger version with the specific proviso that single index a can describe the tail of the noise distribution along any direction in Rd. The first exit time results now admit simplified form. In particular, the cost of jumps will degenerate to the count of jumps in the sense that Σ J(w;) k(a 1) for any k 2 and any Moreover, now have Jg = l1 (a 1) where
(wi,.... ,w) E
ig a minfk E N: (h e (0) RÉ-! (k) E (R)
tu 0: (K5) (()) $ g}
s.t. 3
and the measure a can now be expressed as μ = I va) x
S) x m Leb following theorem is merely restatement of Theorem I. in this simplified setting. Still, present the result to highlight the role of the minimum jump number Jg in the first exit time.
m 5- Therefore, the
Theorem 1.2. Let Assumptions I.I L.5 1.10 and I.7 hold. There exists function X(n) that is regularly varying (as 17 1. 0) with index + 13 (a 1) such that, for any E G and any t > 0,
lim F2 (o > t) = exp(-9t) n00
where the constant q = ==hh=((9))
is clear that first exit time since the first exit time is roughly of order 1/17 +1-1+(11115 Therefore, even in the general Rdd case, the quantity 1g i.e. the minimum count of jumps to escape from a domain g.
101

---

Published as a conference paper at ICLR 2022
induces hierarchy of first exit time as the first exit time from the domain with largest lg (requiring most number of jumps to escape) will dominate the first exit time from other regions.
Lastly, if we work under the standard setting such the ones in Zhou et al. (2020) where local strong convexity of f in the domain G is assumed, the results can be further simplified and the minimum jump number ig will be directly tied the width of the each domain. Specifically, the quantity rg - supyeg lly|l can be interpreted the effective width radius of G, and we work with the following assumption.
Assumption I.11. The function f is strongly convex on the closed ball {y E R llvll S rg).
Two consequences follow immediately from this assumption. First of all, there exists some constant >>> 0 such that
||0, (x)] S ||=]| e Vo E {y: |Bll < rg}-
As a result, we must have 13 2 rg/b. Next, as long as rg/ is not an integer (which holds for Lebesgue almost every b > 0), for k = [bg/b]]
can find t2 > 0,... t1 > O and wn * 0.... uh * 0 where ((s, 0; (0, ... .(x), (wi,-.. ,wx) G for some
O. In fact, it is worth noticing that Assumption 1.8 now degenerates to the condition that r b is not an integer, and we now know that for any such b O, we have 1ng S [rg/b] In summary, we have established the following result indicating that the strong preference for wider minima under truncated heavy-tailed SGD still persists in IRd given proper convexity assumption on f.
Theorem I.3. Let Assumptions L.5 1.5
1.10 1.17 and 1.7 1.9 hold. There exists a function X(r) that regularly varying (as 17 4 0) with index 1 1+ ( 1) such that, for any x E G and any t > 0,
1.10
lim Fx (((()))))) > t) exp(-9t) 110
where the constant q pu (g)) and 1" [rg/b].
1.3 RELAXING THE TECHNICAL ASSUMPTIONS
In order to achieve the tightest characterization of the first exit time, some assumptions introduced above are slightly stronger than the ones in [mkeller et al. (2)10a). For instance, in Assumption I.3 we require ag to be of class C2 while the assumption (A3) in Imkeller et al. (2010a) only requires it to be a C manifold. Besides, Assumption [6 requires that Sy. the limiting distribution of the directions of each heavy-tailed component in noises, to be absolutely continuous w.r.t the spherical measure. As will be stressed in Section K. these conditions are only imposed to prove Lemma K.2. thus ensuring that Assumption [99 holds for almost every b > O. Briefly speaking, all the efforts to guarantee Assumption 1.9 allows us to conclude that the law of scaled first exit time ((1))(1) converges exactly to that of Exp(g) with = =+(h" ((gh)
Fortunately, the discussion below will show that, even when we relax all the extra technical assump- tions above (hence removing Assumption[[.) [.9). the order of the first exit time is still dictated by the minimum cost for escape Jg. and a similar result about the first exit time can be obtained where the distribution of the scaled first exit time a (1) will be asymptotically bounded by two Exponential RVs.
Specifically, we reiterate that the first half of Assumption 1.3 i.e. Og is a differential manifold, will only be applied prove Lemma K.2 The second half of the proof, namely the lower bound in eq. [.II, , only serves the ensure that for any I E G, we have 2z(x)= E G Vt 2 0 so that the gradient flow starting in will never leave this domain. Now let us focus on the open sets g and G. for some small e > 0 where
- (VER" d(y.9) <e), A E : d(y, 9 > e).
Note that n&g = U.G. 9 and for any positive reals c1, 62 such that e # cg and the two reals are small enough for G., and Ge, to be non-empty,
age n a = 0, ag., n dGe, = 0.
Given eq. (I.I). as well as the smoothness of the vector field V (sce Assumption 1.2) and the fact that 9 is connected, bounded and open and contains unique attractor (see Assumption 1.1 and
102

---

Published as a conference paper at ICLR 2022
it is easy to that for all > 0 sufficiently small, we will have xx(2)) E 9 Vt 2 0 for any E g and (x) E G. vt 2 0 for any x E ge.
will establish eq. (J.17) (which, again, does not require the C class assumption on äg or the absolute continuity assumption on the measures Sz). Note that eq. (.17) implies the existence of A > such that 1 (Ga)) < co. Therefore, for all but only countably many e E (0, A), we have pa (a(h-1(80)) (0g') = (OG.)) = 0. In other words, we can find a sequence en with lim,, En = 0 such that
Mcanwhile, in proof below
(ag = p(h (og.
An 2
Also, for a fixed > that ensures Assumption 1.7 and [.8] one can see that similar conditions will also hold for sets G and G. as long >> 0 is sufficiently small. In summary, if we consider first exit times
(n)) min 20: X g on(n) - min 2 0: X2 $ Ge,
with the obvious bounds that (n) S a(n) S (n(n) for all n 2 1, then it suffices to apply Theorem directly onto o., (n), (n)) Note that did not attempt to establish that ag aG, differential manifolds. However, we reiterate that this is not needed since the C2 manifold condition in Assumption I.3 only serves ensure that Assumption 1.9 holds for almost surely every b >> which we sidestep by picking proper sequence En to explicitly satisfy the condition. More formally speaking, consider the following relax ted assumptions.
Assumption 1.12. The boundary set 0g is (n - dimeessiil manifold of class C' such that the vector field n(.) of the outer normals on 0g exists with
f(())² n(v) co Yu ag
for some constant > O.
Assumption 1.13. EZ1 = 0. Besides, there exist a positive integer m 2 1, a sequence 0 < a1 <
a2 < < am < 00, sequence of slowiy-varying functions (4,,... .mm). ). sequence of probability measures (S,, , Sm) on the unit sphere Sd-1 with support F, e (Sp as closed sets on Sd-1 such that
F, = for any + j;
For any E (m], define the cone F.j - (Fg) in Ied and measure p()) P(Zz -n Eÿ). we have
t= .1((1) .P() o T dr x d0) - Vag (dr) x Sy(d0) as l + oe
in the sense of M (EA)O)) Here Va is Borel measure defined on (0, satisfying valt, 00) = a V1 0;
* For measure p(O) - P(Z E ((UEE Pij)) and any > 0,
t. p()) T (t dr d0) +0 as +
in the sense of M[(d O).
the of relaxed assumptions. The discussion above implies that the following result is an immediate consequence from Theorem[.] Theorem 1... Let the relaxed assumptions hold. There exists a function {(1) that is regularly varying (as n 4 0) with index (1 + Jg) such that, for any z E G and any t > 0,
We say Assumptions
exp(- qt) S lim inf P. (o > t) S lim sup (a (n)) > 1) S exp(-q"t)
110
940
with q =1)
J PROOF OF THEOREM D.
This section is devoted to establishing Theorem L.I For clarity of the exposition, we break down the proof into several major steps, each contained in subsection below.
103

---

Published as a conference paper at ICLR 2022
J.1. PICKING CONSTANTS E,
The goal of this subsection is to fix several important constants that characterize the typical sizes and inter-arrival times of large perturbations in SGD trajectory that can cause the escape from domain g. We will abuse the notations slightly when referencing certain constants, and quantities such as Co, C1 2g may not be equal to the ones used in assumptions above.
and First, from Assumption I.6 and the definition of Jg. we can define
1 [Jg/(as - 1)] +
(I.1)
and we must have oc 1 k" Meanwhile, from Assumption
L8 can find such that
d(g(i),... im), g > 1001"
(J.2)
for any (iI,...,, im) E N with Σ5= 11(03 - 1) < Jg, well as
Ei E G(i*) such that d( 9) 100
Furthermore, Assumption L.5 implies that ff(()= + for any v E 9 ((O). From Assumptions I.3
I.4
and (and by picking smaller of needed), one such that
the existence of some co > 0, é 0
|[x(())] S e not ||||| At 2 0, E B(0,)),
(J.3)
[V][())] 2 Vr E ((O,,)
(J.4)
d( d(xx(x), 2 d(x, 9") + cot if (x) E G and ((x,( s EY [0,
(J.5)
Here r(z,r) = fy E R a y/ < r is the open ball centered at x with radius r > 0. One
immediate consequence from eq. (J.5) that
d( (x).
2 min(d(a, ( 2") Vo E g, 9.t20. 0.
(J.6)
Besides, the boundedness of domain G and smoothness of f()) imply the existence of some M such that
0
G C M), [ < M Vz E G
(J.7)
(J.8)
V f())] M Vx E
(J.9)
where for the matrix norm we use spectral norm.
Recall the definitions of the configuration sets AAi ... m: ; the unique configuration i" and count of jumps k= to trigger the exit with minimum cost (see Assumption I.7 and the remark underneath). The following three results provide control on the sizes and inter-arrival times of typical jumps that can trigger the escape from G. The proofs involve analyses of the deterministic dynamical system TS and will be provided in Section K In particular, we introduce another concept type of jumps (wi,... , wx) that is similar configuration (ig, ... im)) and configuration sets A(i1, im). For any k E N and any (wi,- w.) ) E (IRd) y we say that the (w),.., wx) is of type-j for some j (ji," ....)) (1,2,... ... mm) iff
Wy E F. 1,....,
(J.10)
In other words, wt2 and 2² are of the same type j. then they have the same cardinality ww |wa [j] = k for some & E N; moreover, for any E [k], the jumps wi and w² are both in the cone Ej, so both jumps point at directions in cone E,, and have the same cost J(w!) = J(wf) = aj for some ji E [m]. Define the set
A
{w (R((O))L w is of type-j}-
The accumulated cost for any E AbTr is defined Jope (j) Elili - -1). Lastly, define the skip-one accumulated cost for any type
Joyp (i) max
Σ
1a 1,2, .b1..
1= B.ii
i.e., Tippe (j) is the highest possible accumulated cost if we remove element in w for Abe.
104

---

Published as a conference paper at ICLR 2022
Lemma J.1. There exist some (0, (0,00), E (0, 00), such that the following claim holds. Let k/ N and j (f1,- JRe) [1,2,-.. , be such that Jope < Jg. For any
g(k') = -1,, ((x) E (0) x IR -R (k(k) = (Wi, ww) E Alj the following set of conditions
t, <t =
k''
(I.11)
| vj 1,2, ... K.
(J.12)
is the necessary condition for inf ( 0; ((6) (k) g S e0-
Lemma J.2. Le 8, co be the positive constants prescribed in Lemmal J.1 There exists some 61 0 such that the following claim holds.
Let W E and j = (1.... .Jw) E f1,2,-..,,) , be such that Tope < Jg. For any E B(0,e)), any t(k) (t1,-.. ,tt) {0} x 1+--1 w(kk) (W1, www E Art the following set of conditions
< Vj = 2.3, ... ,
(J.13)
||y/ > 8/2 vj = 1,2, ... ,kk-
(J.14)
the necessary condition for inf 20 ( x; (k, w (k))), go) S co/2.
Lemma J.3. There exist some 6o 0, 60 such that
sup d (s x. (k) w (k))), co
#20
for all x E B(0, 6o), all (h, (hm E N with [(n, .. (im) < Jo and k: = Ey= ig. and all
wk( A(i im), (()) {0) R
Similar to the ODE path æ, we defined the following paths with rate 1 0:
(o = (x)((((())) V20,
(J.15)
(x) =. T.
(J.16)
In other words, ODE flow æ; is equivalent æ with rate 77 = The next result gives us an upper bound for the return time to neighborhood of the local minimum as for the gradient flow.
Lemma J.4. For (ODE(T,))) A min(t 0: x (x) B(O,e), there exists Cj (0, oc) such that for any e E (0, E),
(2, s Ci + c log(1/e)
Vr V
Proof. Due eq. J.4. we know that To a sup.ed f(x) (nf) f(x)
< oo and
8
sup (I, To.
zeg
Furthermore, from eq.
J.3 one can see that
sup (op(tet TODE( E) log(1/e) - logloo//// 1/E)
Cp
Now combining these two bounds with 1/1 time scaling, we have T T < + c log(1/e)
sup
zeg
for C = max(To + log(e) 1
With LemmaJJ11 J.1 J.4 we are able to pick some constants to facilitate the analysis below. By decreasing assume the existence of some T. < co, ē such that
and increasing M if needed,
105

---

Published as a conference paper at ICLR 2022
(Due to Lemma J.1 and J.2) For any & E N and j E {1, m3* such that Trppe() Jg. any æ E [300, 1). any g(h) (f1.-.., .tx) E {0} x R w(t) = (Wi, wxx E Aire with
i Ti ( (w)())), 99) 100/" d( x
it holds that
ts < t 2,3, ... and ||,, || 8 Vj = 1.2,
(J.17)
For any (m)) N with I (ii,,..... im) < Jg.
d ((i(i.." km) (9) 100
(J.18)
There exists some x E g( such that
d(x, 9) 100/";;
(J.19)
(Due to Lemma J.3 The constant E 0 is sufficiently small such that
sup u(pd(E(s,z) x; {(t,, (*))) > 100/" .20
(J.20)
for all x E B(0, 100 all (i1,-.. im) E N with J(i,, (mm)) < وg and k = Σド= ij. and all ((() E A(11, " ,m), ((h) (0} x R
(Due Lemma J.4) The constant i < oo is large enough that
sup TOpE (x, < t/n =eΦ
J.2 BOUNDING THE DISTANCE BETWEEN ODE FLOW AND SGD ITERATES
Moving on, we show that, without large jumps, the SGD iterates (x) are unlikely to show significant deviation from the deterministic gradient descent process y (z) defined as
xx)x)= = I, (x) = - (
(J.21)
(J.22)
We ready to state the first lemma, where bound the distance between the gradient descent iterates (y) and the ODE (t, x) when the initial conditions x, are close enough.
Lemma J.5. Given > 0 and x. 9 such that the line segment between (k: x) and y (y) lies in G for any A = 1,2, ... [t].
sup .E(0.r]
(x) -y
< M2n + || vl) exp(
where M E (0, 00) is the constant in eq. J.8 e.8. (I9
Proof. Define a continuous-time time process (s; y) - y(s) (v), and note that
'" V Vx(x" x))
(*, ("(({,,
[+]
x=((s],z) n7
(u,
[el
(v) = y = -
Vf(y" d),
Therefore, if we define function
b(u) = (3) (u,,))
106

---

Published as a conference paper at ICLR 2022
from the fact || f()Il M, one see that [[b())] I nM + || yll for any
00.1) and
[I(TT]] s 2n + || y||- In case that >> 1, from the display above and the fact [()]( s M,
we now have
(x)(()) (*( a)) - [[b(Ls})II + nM;
|b[ & nd / [+)]
[6(()]] du.
From Gronwall's inequality (see Theorem 68, Chapter V Protter (2005), where we let function a(u) be a(u) lb6(u + 1)I), we have
| (x)(()) x"s,,)) S (2n + || yl)) exp(n.tt..
This concludes the proof.
Now we consider an extension of the previous Lemma in the following sense: when both perturbed by most 1* similar perturbations, the ODE flow and gradient descent process should still stay close enough. Analogous to the definition of the perturbed ODE Ø",, we can construct a process ÿ" as perturbed gradient descent process as follows. For some integer 1 S I 1*, sequence of strictly positive integers (f2.... (t) (with convention
0 and let (() (4,,... (4)), sequence of non-zero vectors ( (W1,".. w)) and Rd, define the perturbed gradient descent iterates with gradient clipping at as
T. (v: (
1 1[n= - t tt, (I.23)
- -Kh (y: 10 wh) + Po( 1 -VVVV (,,")))))) Σ j=2
with initial condition (y: (0) ()))) = y + (1)
Corollary J.6. For any fixed > 0,4 > O, the following claim holds for all rj > sufficiently small:
given any z, E G, some integer 1 $ S l" sequence of strictly positive integers t!!) (tz) 1
(with convention t 0), and any two sequences of vectors w (wj)}=1, w(C (w,) such that (let tend m)= tz)
x x-yl< <
tj S ft/ all E [0];
my - أyyl e for all j [T]:
The line segment between 3" T 800 wil)) and (y: (()) ((()) (,,,,)))))))))))))))
lies in G for any k s
tend 1,
then we have
(" w() (u: ",000)] p(t)c
sup (0(
where the constant p (2 exp((MM) + 2)" In particular, p == p(t) constant where f is the constant in eg. (J.1I.
Proof. Throughout this proof, fix some v) E (0, (0,e/2M). We will show that for any 1 in the range the claim would hold.
First, interval [0, t2), from Lemma J.5
that (since 2M c)
n7 (8, x; (1) (0) ("r (1)) w ( 2exp(2MM)) c.
sup =E(0,(2)
107

---

Published as a conference paper at ICLR 2022
The at t 12, by considering the difference between w2 and W2 and the possible change due to one more gradient descent step (which bounded by 1/M < e), we have
x (s, s() ) T. (y: t(0 x (2exp(2Mt) +
sup e0(0,
Now proceed inductively. For any j ... 1-11 1, assume that
(s, 8 w(!) - V(s (y; 10) x - (2exp(2Mt) + 2)² e.
sup eE(0,4,
Then by focusing interval [tt, f++1 and using Lemma J.5 again,
show that
(s, ()) - [. (u: t0) x
- 2c+ ((2cxp(2Mt) - 2) 1) exp(2Mt)e S (2exp(2Mt) + 2)
sup xe]tsffrin)
This concludes the proof.
Using an almost identical approach, one can show the following results that bounds the distance between ODE flows with similar initial values and jumps.
Corollary J.7. There exists some consiant p" E (0, oo) such that the following claim holds: For
any fixed ε 0, any I,y E G, any integer 1 S l S 1", a sequence of strictly positive integers {(0) = (tz) (with convention t 0), and any two sequences of vectors w (wz)}=1 w.")
(ūr))/21 such that (let tend = tj)
<:
ty S for all j E [1];
Wj -أyl for all j [T]
The line segment between (s x; 10) w(() and (s' y ()))) lies in 9 for any s < tend
then have
(" x; (0)) (()) - (s, y )0, ( w
s'c.
sup SE (on)
In the next few results, we show that similar type of control can be obtained on the distance between gradient descent iterates T7 and the SGD iterates X Specifically,, our first goal show that before any large jump, it is unlikely that the (deterministic) gradient descent process yι would deviate too far from X To facilitate the analysis below, we introduce some additional notations. First, we will group the noises Zn based on a threshold level 5 > 0: let us define
2,560 7,, {17 2..1 S 5), 7,66. 7,, (n Z..1 > 5).
(J.24)
(J.25)
The former are viewed as small noises while the latter will be referred as large noises large jumps. Furthermore, for any j 2 1, define the jth arrival time, size, and direction of large jumps as
(7) e min(n (8); Zm 8). (6) O.
(J.26)
(7) A Zry
(J.27)
0)( A W/ (8)/ [+]]
(J.28)
The following event
A(n, ),
k= (0) 7 12 ... 2x S
(J.29)
describes a scenario where, before the first large jump, not much perturbation has been caused by noises.
108

---

Published as a conference paper at ICLR 2022
Lemma J.8. For any 2exp(nMn)' on event
0,7 O, any z. E and positive integer such that 1| yll
A(n, 1h 2 Mexp(nMn)
Ue
{ the line segment between (v) and (z lies in G Vj 00,,,, (..,,..,,,A(((()))))..) (27(
we have
1 (v) (x)l < Vm 1,2, , A (T) 1).
Proof. On the said event, able to apply Lemma G.5 and obtain that
X] - (1 yll + 2exp(nMn) exp(nMj) <e Vj 1.2. (8)
and conclude the proof.
Similar to the extension from Lemma J.3 to Corollary [.6] we can extend Lemma 1.8 show that, we consider the a gradient descent process that is only perturbed by large noises, then it should stay pretty close the SGD iterates X². To be specific, let
(x(x) (z) = (z) Y. pol (VS((2 11t==T(61nn22)
J21
be gradient descent process (with gradient clipping threshold b) that is only perturbed by large noises. The next corollary can be shown by an approach identical to the one for Corollary [.6] (namely, inductively repeating Lemma J.8 arrival time of each jump) so omit the details here.
Corollary J.9. There exists some function p: (0, oo) + (0, oo) such that the following claim hold for all > 0, > 0 and all sufficiently small > 0: For any a. E 9) with |æ - yll < e and any I [1"]. on event Ao(e, 7, 8, L, 1) Bu(e, m, 8, 1, Cole, 77, 6, L,t, we have
[V - XXxx) < p()) Vn 1,2, ... (8)
where
1, 1, ,1, max Ao(c, 8,1,,) & {Vi =
(8)+ Tt (8)-1 " ZTQ (6)41 + +... Zz S 2xxp(2tM) }
Bole, 5,1,t, 2 2, T7 < 2//m}.
Co(e, 5,1,1 A {the line segment between (v) and X7((x) lies in G Vj = 1. (T((0) 1}.
In particular, p = (() is a constant
The next result shows that the type of events A(n, 77. 6) defined in eq. (J29 indeed very likely to occur, especially for small For clarity of the presentation, introduce the following definitions that are slightly more general than the small and large jumps defined in eq. (I.24)eq. (.25) (for any
7. 7 S c}.
7,cc و Z, ZA c).
Lemma J.10. Given any t >> 0, e > 0, and N > 0, the following holds for any sufficiently small
8 >>00 0:
26/9 + ... + /56/1| > >6) ())=00 -
TP
max [t/m]]
17
as 17 O.
109

---

Published as a conference paper at ICLR 2022
Proof. For any C' (0, > 0,3 E (0, consider the following decomposition of the small noise -,55/ Here we adopt the convention that Eg I (UJ (For definitions and properties of the closed cones E.. Em see Assumption 1.6 and the remark underneath.)
2 - (2 E Eg}, 2 25C, A 2 /
s C
2, (n) N 1 - E (C,6/nl}
Based on this decomposition, we define the following iid random variable sequences
2 (m) 2,50 B2,0000
7 (n) (n) -
ZL.(1)
(n)
C,é.n
( (m) 2 .00
Meanwhile, define the projection operators RR + R with ( (0)) Us. First, recall that EZ1 0. So for all C' ssuffciently large, we have
EZ
EZ
sup 2
sup y2C
2t
Moreover, due to Hj E RV- +1) for all 1,2, ... and Ho(x) = o(1/2²)) for any a >> 0
(for definition of functions Hj, see eq.
(.55), know that E
z
for any 0, 0,, m,
implying that for all C large enough, we have (for any j = 0, 1, , m)
E $2.0)
sup v.=2C
<
(J.30)
2t 24 m +1
Fix such C. Besides, recall that in Assumption L.6 we have 1 < ai 02 am Therefore, are able fix some 3 E (0, 1) and 5 > 0 so that
Ba1 > 1,
(J.31)
12vm+1 +1' (N + 118 <
(J.32)
N
3a1 - +1)8
(J.33)
12vm mI
With the fixed C and d, note that eventually 8/11 > C as 4 O. Therefore, for all sufficiently small,
P
max
17
2 26/
+
>{) +)
1=122,.
max
17
Z
b/n
EZ
9)
n=1,2,- [t/0]]
Am
P
n= 1,2, (,22...t//))) [t/o]]
max
1
BZ
S0.0
1.0) (CC.s (k) + S700
2
E)
P
max
17
z sC,.)
EZ (0.0) +
4.00
(k) + +Z C0.0)
EZ
58/7.i)
1=1 1,2, 13/n1
j 0
2/m +1
m
Σ [k))
1,2,,".((//))
1,2,"
max
1/
6vm+ +1
(J.34)
1=0
Amml
[t/m]
+
Σ
700 (k)
(J.35)
6/m+1
0=
k=1
110

---

Published as a conference paper at ICLR 2022
We bound the two terms eq. (I.34)
and eq. (J35
respectively. First, observe that
d
eq.
J.34
max [t/ol
P. (2(0
6 + 1)d
Let 2 (k) = P. (k)) and let x(0) be an iid copy. By definition, |z!5| S 2C. Then from
Hoeffding's inequality.
eq. J.34
< 2(m + 1)d exp (-
1 -/ (
e
1/1/ )- o(1n)) 2C[1/n]
2 d/
As for term eq.
J.35
w.l.o.g. let us fix some j 0, 0, 1,. m and bound
[t/ml
P
(
(n)
(J.36)
6I
k-1
it/n
[t/n]
70) (n) 6/m+1 +1 1= + +P
Σ
70
c
I 12 k
6ym+1
+I
t=
n=1
n=1
(0)
((T)
where _ #(n = 1,2,
[t/n] (n) > 1/n and is some positive integer such that
kō
12vm+I' k(Ba1 1) N.
Note that we can find such due to our choice of constants eq. J.31 eq. J.33p.
For (I) and any i = 0, 1..... & - 1, observe that (the non-negative RVs (n) and (n) are defined at the beginning of the proof)
[e/nl
[t/ml]
8 Z0) (n) 11/ vn E [[t/n] i1)
(1)
P
Σ
(n)
in
6/m +1
1)
fe/n-i
$ st//n1 P
Σ (n))) > 12ym +1 n7
1
(n) - 1/11 Vn E [T//l] - -1)
==1
[t/m]
7500(1) (n) > 12 + 1)
- [t//n1 P
==1
Let (m) A 7 (m) R773² (n) and let WU) be iid copy. Due to eq. (.30 for all sufficiently small we will have that [t/n] E2 (1)1 24vini "f Therefore, for all n sufficiently small,
Tt/m]
(l) S [1/n]*
P
E 24 17 w6
(J.37)
Observe that [W) < 2/1² and
B(W = var Z(62 <B
/
2yP((Z > yddy 1' 2y)
Note that there are only two possibilities:
If j * 0 and ag S 2, then from Karamata's theorem, we have B(Z(0))2 RV RV- and note that ((2 a.) < 1;
Otherwise, one can find some > such that Hz(x) o(1/22+A), implying that E(ZZ)))2 s Jo 2yH
regardless of the actual value of 0.
111

---

Published as a conference paper at ICLR 2022
In any case, using Bernstein's inequality, conclude that
2 12 2 [t/m B(W + } /1n = =0())))
eq. U.37 S [t/n]'
2cxp (- -
On the other hand,
(II) S P( 2 k)
[t/ml) k
PP (m)
> 1/1 vn= =.::...... 1.....,, s [t/n1* (H5(1/19))*
which is upper bounded by function regularly varying w.r.t 7 with index k(B01 - 1) > N. Therefore, (II) o(17*) and conclude the proof.
J.3 ANALYZING DIFFERENT SCENARIOS BEFORE THE FIRST EXIT OR RETURN
Using results and arguments above, we are able to illustrate the typical behavior of the SGD iterates X in the following two scenarios. First, show that, when starting from anywhere in G, the SGD iterates X will most likely return to the neighborhood of the 0 within short period of time without exiting Q2
Lemma J.1. For any (0, E), the following claim holds:
Ps (x E vn S TTeamm (n); e)and T Treturn ( S p(c)/n)
lim inf 900 g
where the stopping time involved is defined as
T Treturn( (17, c)
min(r. 2 0: |X < 2c}
and the function p(c) c1 e1 log(2/c) with C1 (0, oo) being the constant specified Lemma J.4
Proof. Let = plc) = a + ci log(2/c) and n = [t/n] and arbitrarily choose some x with d(x,9" c. Also, fix some N 0. Due Lemma J.10 able to pick some > 0 such that (note that rpn < 2/ eventually as 0)
P
max It/ml
"7 2 //5 +... + 2,55/x
2 exxp(2mmM)
-0(1"
n=12
First, from Lemma J.5 we know that for any y small enough such that
M exp(2tM) c/2.
we must have
<0 (x) S se/2 Vs E [0,
Next, using Lemma J.8. we see that on event
A(n, 17, 2 2exp(mmM) 8) (T) > [t/r]},
we must have
(x)(()) (x(al <e =k= = 1. (.t........ [t/n/.
Combining these facts with the Lemma J.4 we have shown that, for all r7 E (0, is exprrr)))) [t/n]]
event A(n,7) 7) zexp(maay- 8) (TT(() Teeum (1) e) and Trecura e) S p(c)/n. In other words.
must have X g An 5
lim 7/0 )))
sup
Pz X & G for some n S Treuu (n. E) OR Treturn E) p(t)/n)
Sim P(A(n, 7h exp(nnMM) 8)9) + lim P(T) < [t/nl). 740
nio
112

---

Published as a conference paper at ICLR 2022
However, our choice of 6 at the beginning of the proof implies lim,uo P(A(n. 7,, explomd) 6) = 0. Meanwhile, from Assumption 1.6 we see that
that
[e/n]]
P(T 2 [t/n])
Σ
(1 (--0/(()))
H9(6/)))
Hz(0/n))
[t/ml
s(m+ (0/m)
Σ (1 (o(m)))
for all 1l sufficiently small (due to regularly varying nature of H2 and < an <... am). In particular, since Hi E RV-oy, for any fixed A E (0, - 1), we have (for all sufficiently small),
Pa/m] Σ
(6) & [t/n]) 5(m + 1)77
P(U()) S [t/n])
where Unn d Geom(n Lemma G.4 then tells that, for all 7) sufficiently small, P(U s
[t/nl) S -1++, thus implying
2(m + 1) 11 1 = 0 1-A
lim sup (8)) < [t/n]) & lim sup
140
n40 t
and concluding the proof.
In the next result, we show that, once entering a e emsllll neighborhood of the local minimum, the SGD iterates will most likely stay there until the next large jump.
Lemma J.12. Given No > the following claim holds for all E (0,t/3) and all d >> that is sufficiently small:
P (B3n < (8) s.t. |x 3c) = oo
sup #e (0.22))
as 17 O.
Proof. Let t = c1 + c1 log(1/e) where (0, oo) is the constant specified in Lemma J.4 and n = [t/n]- Fix some 3 > 1 + 0n and N B - + No. Due to Lemma 10 the following claim holds for all o that is sufficiently small:
max 1.2, (t///
"
1||25500
5660 .... + 2<6/11
2cxp(2nnMMT) = o(n
c/2
We fix one of such 8. Therefore, for event
1/m8
Ā 2
3j
s.t.
[t/m]
20/n
2,58/
2+(--1)(0/1 +... 2 z5611)
e/2 2 (2xp(2mmM))
max 1m112,." ",t,,,,,[[/)))) 17
110-1) +
the iid nature noises (Zn)22 implies that
P(A) 5(1+ bnp )P
1 1 25/n .... + b5 _-6/1)
c/2
max
1= [0/m] as 4.
2 (exp(2mmM)
=0( N- N-8+1)
Meanwhile, Note that T < (1) (0) min(n 2 0 2n1 > 6, 2n E Ei}. Therefore,
(O)"(() 1/m²² P(T (8) 1/nl). Specifically, note that (8) 1 (6/ and
H1 RV Lemma G.3 then tells us the existence of some > 0 such that
PPT7 (6) > 1/7²) of exp(-c/n))
113

---

Published as a conference paper at ICLR 2022
7 O. In summary, we have established that
P(ÄU (T > 1/n"1) = N07
Now let us focus on the complementary event (A) n (T) - 11/1g)) On this event, there must be
some j <
1/4"
such that + (j - 1)[t/1] - (0) S j/t/n]
Tt/ni
Let arbitrarily choose some E (30, 2e). j 1, then due to eq. (J.3) we know that
<s x) E (0, 2c) for all s 2 0. Also, from Lemma J.5 we know that for any 7) small enough such that M (xp(2)M) c/2, must have
(s,x) (()](t)| s/2 as 10. [t.nl]]]
Besides, on event (A) (2T( & n(j 1}. due to Lemma J.8
1}.
we must have
lxy - Xx((=)) < c/2 Ak. 0, 1, ())
with (8) < [t/n]. conclusion, in the case that j 1, we must have [X](()) TÍ (6).
<
3e for all
Now consider the case with j 2 2. Similarly, due to Lemma J.5 and J.8 we now have that
2 (8,z) y1, < e/2 Vs [0, [t/n]l
[1((x) (x) < c/2 Ak 0, 1,.,.... ,,,..,,t//]].
|=>"(,,)|| < 2e vs E [0, [t/v]]-
In particular, due to our choice of f the beginning of the proof and Lemma J.4 one can see that |æ> ([t/nl, x)]l < c. Collecting all the results, we now know that
[x) < 3c Ak [t/n],
< 2c.
Now suffices to apply the repeated apply the same arguments above. For instance, jf 2, then (2) and <= .11) using Lemma J.5 and J.S. one can show that (x)]l < 3e for all n < (6) (since in the case of j 2, have (0) ([t/n]). Otherwise, we have j 2 3 and the same arguments above can be
by letting 21 X² 7t/ (z) and bounding the gap between trajectory of X [t/n+
applied to show that
|X < 3e vk < 2(///l [xir)( (x) < 2e,,
| [2 (x) <Ec.
In summary, by proceeding inductively, we establish that for v E (0, TM exp(2TM)))
P < s.l. [x > 3e) SPAU 1/1²))
sup
and this concludes the proof.
We introduce a few concepts that will be crucial to the analysis below. Recall the definition of ODE with jumps E (note that we will drop the notational dependency on learning rate when we choose 7) 1). Recall the definitions of i* k (see Assumption 1.7 and I.8 and the remarks underneath).
We define the following mapping from (r1, . (61.... .0k)) E (8d-1)""
(f1....,44 .tx.) E {0} R such that
h(r 0,t) = x t.. 0: t, w)
(J.38)
where w ) with w, = r,0,, From the continuity of the ODE flow, we see that is continuous mapping. Also, we introduce the concept of type for configuration i". Specifically, define
j(1,, (i)) e 2(6 [(i..... ...) {0, 1,2, ... "
#[n: j. k} in vk E [m]] (J.39)
114

---

Published as a conference paper at ICLR 2022
and for any j(²"), we say that (w1, wx-) A(i*) has type if
wi E EJ, Vi E [k*]- In other words, based on the direction of each jump in E A(²") group them into different types.
Note that [j(i*)] < cc. Now for any type j(²",, define (Borel) measure I (S 1) x
I -1 as
Mj
Pag x II Sq Leb i=1
(J.40)
where, for any a > 0, the measure Va is the Borel measure on (0, 0o) with (n (c) = 1/20. We use the concepts above to characterize behavior of large noises in (7 Jn21.
Definition J.1. For any 7, 8 0, j(i") and any Borel set A C RRd, we say noise Zn is of type-( (ype-(A,6, (,,,11.)) iff
1 || > 0;
For ii n and i, _ min(k > T 1 7| > 8) for all i = 2,3, ... K., holds that
t - t,-1 < 2f for all = 2.3, ... ;
For t, _ - - (with t 0), w; n and
a (1/w || || : ..... ||00=- II), 0 = (wi/ ||0| |l, .... wk- || II),
(l1, ,Kx),
it holds that h(r, 0,t) A.
More generally, we say noise Zn is of type- (A, A,8,n) iff there exists j E j(i") such that Zn is of (Ape((A,)) 6, n).
Due to the iid nature of (Zz),21 let us consider an iid sequence (V3) where the sequence has
the same law of Z1. Note that for any fixed n 2 the probability that Z,, is of type- (A, n..) is equal to the probability that Vo is of type-(A, 6, n..)). More specifically, let H (x) = > Hz(z) = Zi]l > æ). Due Assumption [6] Hl E RV-o,, Besides, P( || Vo > 6) (8/T). Now focus on conditional probability admitting the following form:
p(A, n, j) P I(Vv is of (Ape-(A.6, n |Vol| > 6). (o
Lemma J.13. For any > any j E j( and any Borel set A S Rd such that
M(ri,". ". ,r. ,r,, 01, 0p- t2. ... .xx- E A = < 2F = 2, kk* and r; 8 Vi E [k*).
(J.41)
it holds that
p(A, 1,1.))
Mi (h- (A9)) < lim inf
gan I (Hal1/)) %h
H(I/1)/
p(A, 8, n.
< lim sup
S Mi (h-((A))
(Ha0)/))
8an
H
Proof. Let us start by fixing some notations. Let Ti 0, and define stopping times T, = min (n >
T5-1 :n [VV > 6} and inter-arrival times T = T, T for any j 2 2 (with T = 0), and large jump W, Vr, for any j 2 0. Note that: first, the pair (TV, W:) is independent of (T), W3) for any i { j: besides, W, and T3 are independent for all j 2 1.
Define the following sequence (of random elements) w = (Wi, wi.) and = (ti, ... (t-)) by W; r,W, = n/T.
and r = (T, (T,) 0 = (To (w; ( where operators T, (x) = |||||| Tx) = x/ ||>||| constitute the polar coordinate transform. By definition of type- (A, 5, n, j), know that
115

---

Published as a conference paper at ICLR 2022
T I 2t/n for any i 2..... kk=;
" || > for any = 1,2,
Eg, (i.e. 0g Fi) for all [A*];
h(r, 0,t) A. Therefore, (let Rí T, (W.). = To(W))
PA,, 8, n.
- -(P(T) S 22/nn)
k'-1
k'
P(W, E E,) P(nR; = dri, -; =d=).)
P( dt, ||T - 28)
Ry FWW e E.)] 2n. (h (A))
=(P(TT s 2f/m)
(J.42)
where Qn, is the probability measure on R x (Sd-1)k* x R induced by a sequence of (n),
random elements (n/8i.... .nl.. 0 nI
.nI (1)) such that
For any i 1, k*, we have R₁ TW, (n)), θ To (1)) where the distribution
of W (n) follows the law P ((V E 1 > Vo F..): ): Besides, (W?( is
independent sequence;
For any -.... 1", the distribution of T/ (n) follows from F (f) E (TT s 2F (for the definition of stopping times (n), see eq. (126 ) Besides, (7) (n)) is an
independent sequence, and is independent of (W]
Qnas()) (( RR., O O2, (I (n), nI (n)) e
As for weak convergence of (nR1,... ,n .,,... Ok- (T) T (1)), observe that
For any i [K*], from Assumption 1.6 one can see the regularly varying nature of distribution Vo on the cone Es, (hence for W (1) as well), thus yielding that (n, 9,) = (R) L where R₁ and O are independent, the law of O is Sj,, and the law of R; is the Pareto distribution with
80
P(R x) (xr 88.00.
For any [0, 2f), since lim, 1o x (m/m is easy to show that
1 - (1- (6/n))) 1;
lim n/0
therefore, we have (for any E (0,27)
1- ((66n)))o//n
P(TT(()) < (T) - 2t
+
T
1- (1-(1((8/)) [2//]]
2i
7) 4 O. which implies that T/ (1) converges weakly uniform RV [0,27].
116

---

Published as a conference paper at ICLR 2022
Together with the assumption set A in eq. of measure 2n, as Haj then (the measure My is defined in eq. (J.40))
J.41
that, if denote the weak limit
I 8t5.
Pa. (h (A)
(26) Ms (h (A)).
From the continuity of mapping h, set. Using Portmanteau theorem, we now have
that h (A) is closed set and h h ((A)) is open
I 8% (2f) My (A7))
inf 2n.5. (()(())) lim sup Qn (h (h- (A)) (A))
(J.43)
I 8's.
(2 -1 Mi (h (A)).
(J.44)
Moving on, we analyze the limit of the other terms in eq. (.42
For any i E [k*], note that
P(W; E.) Hj. (6/n)
((8mm)
Now due to the regularly varying nature of Hj and H,
I = P(W, E.)
5k
lim 7
(J.45)
H3, (1/n)/H(11n) 11
809
the other hand, from Lemma G.4 and the regularly varying nature of function H, for any fixed 1, we have
801
P(T2 S <2/nn)
-1-1
801
26H(6/n)/m
k--1
lim sup
S Sk lim sup
H(
=
2
H(1/n)/n
2i
H(L/n)/m
njo
n10
8a1
PPTT- S. 25/n))
k--1
8an
(6/(6/n)/T) W'-1
lim inf
2 (1/k)**
-1 lim inf n40
27.
H(1/n)/n
27
H(1/n)/n
=1/K)**'1
Due to arbitrariness of x > we yield that
4--1 (PPTT - 2t/n))
lim
k- 1. [1(1a)
(J.46)
27
391
Collecting all the limits in eq. J.44 leq. established that
(J46) and plugging them into eq. (I.42). we now have
n7.5)
S lim sup p(A,8, 1.5))
My (h (A9)) < lim inf IT. (1/m)
S uy (A))
80r I1, H2, (1/1)
501
#(1/7)a***
M(1/a)n**
To conclude the proof, recall that for any type j j(i"), we have # 1, ... 87 (i.e. for any j E [m]. the number of elements in (J,, . that are equal to j is exactly 17). thus II. H,, (1/n) IIm (H*(1/n))**
k' fr = =}}
More generally. for
PP(VV we know that p(A.8,n) Ejes(e.) p(A, 5,1.)). Also, define measure f1
(A,5,11) TP Vo is of (Ap,-(A,6
n 11Voll > 6).
(J.47)
Σ 14-
(J.48)
jEj(1")
The next result follows immediately from Lemma J.13 and the fact that [ī(}*)] < co. Recall that Jg is defined in eq. (I.9.
117

---

Published as a conference paper at ICLR 2022
Corollary J.14. There exists a function X(1) RV1+Jo--a (1) such that, given any 6 0 and any
Borel set A C R with Mrr," ... , 0,,,, 00,,12; 12, ... ,Kx-) EA = 4 < 2i = =
... ... kk*' and ri > Vi E [k*] :
(J.49)
it holds that
(A)) S lim 8,1,
< lim sup p(A,
S (A)).
8an {(1)
gmn. A(1)
In particular, the regularly varying function X(-) admits the form
X(1) I (H5(1/m)) 4
(J.50)
H(
Consider the following stopping times
(n) = min (n 2 0: X & 9):
R(e, 6, m7 = min (n 2 (6) ||xX|| S 3e).
Here a indicate the first time that the iterates exit domain G. while R denotes the time the SGD iterates return to small neighborhood of the attractor O after the first large jump. In the next few results, we study the probability of the different scenarios regarding the first exit time o (n) and first return time R(ee, 8, n). To this end, let
(5) _ max {k 2 0: (T()) S (()) A R(e, ó, n))
be the count of large jumps before the first exit or first return. Furthermore, we introduce the following concept as the accumulated cost of large jumps before the first exit or return. Let Th = 0 and
Tr.. & JHi + (8) vk S ())
where the cost function J(.) is defined in eq. (16). In other words, Te. indicates the total cost of the first & large jumps, if there are at least is large jumps, before the first exit first return. Similarly, can also define a step-wise accumulated cost as
TH (n) = max( (0) S n) Vn S (1) A 5,c,
In other words, Tr. (n) evaluates the total cost of large jumps up until step As preparation for our analyses below, we first discuss the following technical tools. For any e > 0, let i(e) = + log(1/e) where is the constant in Lemma J.4 Besides, define
E(E)
(J.51)
4 exp((iMM)
p(i(e))
p((())
where functions p(-),p() P(-) are defined in Corollary event A = Ax,
J.6 and 1.9 respectively. Furthermore, define an
(e, 6, 17) U AX (c, 1)) where
(e, 3n < (7) s.t. [[](()] > 3e
(J.52)
A5 (c, {3j 2, 1'' s.t.
" bT (0)+1 + ... + 2 {(0)
max k= 1T)
(J.53)
where the positive integer 1" is defined in eq. Lastly, define events (for any K [1-])
J.1 By definition of 1", we must have (8) < l".
(K) (A* (J < Jg, K S ()()))
nevj E [K], IT. (o()) A R(e, 6, - < 20(c)
7 }.
(K) A (A*) (T Jg. K (3))
n nB E [ s.t. (T) A ((n) A R(c,8, -6) 21(e)
118

---

Published as a conference paper at ICLR 2022
Recall that the minimum cost for exit Jg is defined in eq. (.9 Note that, in the definition of B (K)
and B2 (K) above, (0) (o(n)) A R(e,,, n) 1 only if = = (")6).
=
Lemma J.15. For any K E [17]. any c (0,7/3) and any n > sufficiently small, the following claim holds on event Bi (K):
X ((6)+17(0((z)) (z) T, [)] < 2e
sup :1 ((
where T (0) (5),) ( W ( and (X))))]))00)11
TR (0) (0)) w
Proof. We focus the distances between the following three objects: X²(
(x)
if < (T))): otherwise;
(²(x)) ())
Ezz1 1n==
and
z(s) = xz (s. (x) T, W) V4 20.
First of all, for x(2) = X" T. (d)-1 (a) = (07 (8)- (z), by definition event (A*)c we have
|| < 3e E. Then due to eq. (J.2 and the definition of event B (K) (in particular,
the fact that Jh: < Jg), we know that
d(=(s), 99) 1 Vs E [0, ( R(e,, -
In light of the definition of B (K) (i.e. the upper bound on T (0) - T (8)), by applying Corollary
J.6 we can show that
x(s) " [a] (1)+2 <E.
sup sE|0,T ((0))
particular, by applying Corollary (.. inductivll for all n S ( (8) R(e, 8, (5). we know that the line segment between 2s) and Уー (6) (x) is in & for all (8)
R(c, n) T (ō) Corollary J.6 can be further applied to + 1 and establish the inequality above.
Similarly, due to our choice of ē(c) in eq. (151) and the upper bound on + (5) TT (6) in definition
of event B,, (K), by applying Corollary J.9 inductively, we know that for all sufficiently small,
4 [el (11+ (2) - X2 (11 (x) <e
sup
,= TR 4
holds on event B (K) and this concludes the proof.
Lemma J.16. For any K [1*), any 0 and any 1j sufficiently small,
B (K)
Proof. The definition of event B (K) ensures that we can define some j as the smallest integer in (8)] such that for T 2 2 (5) + [2i(e)/n]. we have
TV. T < (8) A (r()) A R(c,5,
(J.54)
Analogous to the proof of the previous lemma, we focus on the pair-wise distances between the following three objects: (x)..
(²(x) (x) po (V E 1
if < (T))):
(x)
otherwise;
119

---

Published as a conference paper at ICLR 2022
and
x(s) = (s, T. W) vs 2 0. with = (6) - (6),
(6), ... TT - 7j (5) W
(W7(0), (O), " 5' (0)) and X X 17188 Again, using Corollary J.6 and J.9
one can see that for all 7y > 0 sufficiently small, we must have
x
111+2 (x) 2(s) < 2e
sup =E|0,7 -T"
on event BI (K). However, eq. JJ20 and |X01|| < <E < implies that d (=)(s), 9 100
sup se (6)-TT (6)
In the meantime, Lemma J.4 and our choice of ē(c) in eq. J.51 implies that
| - ( < c and || (x)]l < 3c,
thus dictating that R(e, < T and contradicting (.54). This concludes the proof.
Now we are ready to apply the tools above and analyze some atypical scenarios regrading the first exit and first return time. In the next result we show that, when starting from the local minimum, it is very unlikely to escape if the total cost of large jumps is less than Jg.
Lemma J.17. For any fixed (0,2/3), N 0, the following claim holds for any sufficiently small >
Pe Pa(ao() 8(e, Too(((n)) (J) o(n²*) as a 1.
sup
S2
Proof. Since the constant is fixed, one can see that = c1 + ej log(1/e) is also fixed where ei is the constant in Lemma J.4] Besides, we can fix some ē > 0 such that
7
cxxp(2iM) p()) <,, 7() <
where functions p(.), P(-) are defined in Corollary J6 and respectively.
Let us define an event A x = A U A where
A 3n T (8) s.t. | 3e).
A3 3j 2, ... s.l.
1=1,2, (TT(()) T3 (()-1))(2Em))) 1/ |||ZT, bry, (6)+1 +... + Ery (09++ }²
max
where the positive integer 1" is defined in eq. [.II. By definition l", we must have (7) < 1" on event (o()))())) < Jg).
(4*) (T))((()) < Jg). In particular, we Now let us analyze the dynamics of X on event (A*) n decompose it into two events
B1 (A*) (T,, (o(()) Jg} n (vj = 1,2, (0),
A (T) 2i/n),
B2 ( (T), (o)) < Jg]
n (3j = 1,2, (T) s.t. (5) A ( A R(c, -(ann)))7711 (B) > 2f/n).
Using Lemma J.15 and the fact that B₁ (Uk= BY (K) (K"(8) K}) 0 (JRa Jgh. one can see that for any > sufficiently small, we must have
d(x² (n) (x), > 100/ 2e > 2 - E
120

---

Published as a conference paper at ICLR 2022
event B1. In other words, event B1 must have R(c, o()).
other hand, Lemma J.16 the fact that B2 S UK-- Bi (K) n (K*(6) = K] implies that B2 0 whenever v > 0 is sufficienlly small. In summary, we have established that
sup P2 (o(()) < R(e,n). (o)((())) < <Jg) < sup P(AA).
Ja <2
|2
Lastly, from Lemma
1.10 and 1.12
that for any 5 > 0 that is sufficiently small,
P(A*) o(17))
sup as ||
and this concludes the proof.
Using an almost identical approach, we can establish the next two results and conclude that is also rather unlikely have scenarios where
The accumulated cost of large jumps exceeds Jg before the first return or exit,
Or the first return occurs before the first exit,
yet it takes rather long for the said event to occur.
Lemma J.18. Given e E (0,E/3), N > 0, the following claim holds for any sufficiently small 5 > 0:
Pr [K N.s. TK 2 Ig and (5) (6)= 2n(c)/nfor some j =2,3, ... K =EB(0,2))
sup
as 1] O where [(c)= c1 + cl (1e).
Given the fixed >> O, define i i= + ci log(1/e). Also, fix >>> O as the largest possible value such that
Proof.
E 4exp(2iM)) p()) Se p()E
where functions P(-) are defined in Corollary
J.6 and J.9 respectively.
Let us define event A A A where
A 3n < T (8) s.t. |X 3
A A
3j 2, ... s.l.
/ 1ry + +.... ZTy (0)+s > 2
max
k=1,2,- (TT (0) T. (8) (0)00)))((2))))))
where the positive integer 1" is defined in eq. [11 Furthermore, define event
2 for some j = 2, "
B 2 (A*) 3K JKEENS s.t. TR 2 Jg and T7()) (
Note that event B, the index
K = mk 2 1: Jrhh < Jg Vj = 1,2, --
well defined with < K" < 1" (due to the definition of 1*). As a consequence, we must have Th (5) (n) 8,1))
Now based on the exact value of K*, we can decompose the event B into B B(K where
2 = (A7) (K" K} n 6 < (o) (8) for some j E [K]}.
121

---

Published as a conference paper at ICLR 2022
By applying Lemma J.16 we know that for all rj 0 sufficiently small, B(K) -. Therefore, for
all sufficiently small
sup Pz E E s.t. The Jg and (8) Tj-1(6) > 2f(c)/n for some j 2. .K) 2)
< sup (A*).
=(,,0
Lastly, from Lemma
1.10 and
and
1.12
that for any 5 > 0 that is sufficiently small, P(A*) o(17 )
sup s: ||
and this concludes the proof. Lemma J.19. Given c E (0,2/3),N N > 0, the following claim holds for any sufficiently small 8 > sup ir (0,2cc)
PA [R(e, < <n)) J < JG.
(0) A R(c, 6, m) - T8) > 21(c)
for some j E [K)())) = o(n²) as 40
where cc + t log(1/c).
Proof. Given the fixed c > 0, define f = e1 + o log(1/c). Also, fix ē > O as the largest possible value such that
TS 4cxp(2iM) ((0 Se, p(ijEse
where functions p(-), (()) are defined in Corollary
666 and respectively.
Let us define an event A x = A A where
A 3n < T (8) s.t. |x 3E
A A ajj 2, s.t.
maxx km km1,,2. (T) (())-77-1) (0)0))77-0)-1111010000 (0)) 1/ Zr, 2ry (6)+1 +... + Er br (6)+s =]]
where the positive integer 1* is defined in eq. (.I.. By definition of l", we must have (7) < l' on event (R)))) Jg). Furthermore, define event
B ( n R(c, < <n)), Jk kn(B) < Jg.
21(e)
(0) (0) A R(E, n) - (0)
for some j E [0]]
"
Now observe that B C C B(K) where
B(K) (A*) n (6) = K} k (J < Jg}
n { (6)) s.t. (8) ^ A R(c, -T7 > 2f 17
By applying Lemma
J.16 we know that for all r sufficiently small, B(K) 0. Therefore, for
all sufficiently small
P. ( < <(()), Jk Jg.
sup
z (0022))
21(c)
(0) A R(c, 8. n) (8) >
for some E [K*(0))
< #c B(,26)
sup (^(*)).
n
122

---

Published as a conference paper at ICLR 2022
Lastly, from Lemma J.10 and J.12 one see that for any 5 > 0 that is sufficiently small, o o(
sup P(A") =
as 2=
and this concludes the proof.
Recall that in Lemma M have shown that it is rather unlikely to have the first exit with accumulated cost of large jumps less than Jg. In the next result, we show that even with large jumps of accumulated cost Jg. if some jumps still not large enough, or the inter-arrival times too long, then it is still very unlikely for the SGD iterates to even get close the the boundary. Define
Ky a min(k [K*(6)]= : The 2 Jz}
and let Ky, = do if K7(b) < Jg. Also, if J < oo, let
T (1, 6) - T KG
(6)
In other words, T (n, 8) is the first time that, prior first exit return, the accumulated cost of large jumps has reached Jg. Regarding the 1 (4-g-a+++)) term the next Lemma, we note that the function À defined in eq. (1.50) is regularly varying (as 4 0) with index 1 + Jg 011 Therefore, " 14 +0 o(((n)) as r approaches O.
143g
Lemma J.20. There exists some A > 0 such that the following claim holds for all e E (0,E/3) and all > 0 that is sufficiently small:
Px (E 8, m) U Be n)) = o(1) 1
sup 2: |2
as 0 where i(e)
(o(( and
B (e, 6, = {KY < o0) n I >
Jgh.
(2,,,(e, [KY x0
(nnd Jg}
Kgg
n n{3) [=j==2.0 = 2,3, Knd s.t.T,(0 (6) - TT, ( 2f/n or 3j =j=11 1,2, Ky% s.t. n [ s <ō}
of j=011.-..2220 122 (7,5)
min
d(x (x si).
Proof. From Assumption
188
we know the existence of some > satisfying the following
For any k E N and any j E {1.... ; mm* such that kk-1 (aj, -1) < Jg < > (ag. - -1) (note that there are only finitely many possible choices for such j), we have
condition:
Jg
(a.j. 1)
(J.55)
Fix such A > 0. Mcanwhile, given the fixed > O, we can define i i= + C1 log(1/e) and choose 7> 0 as the largest possible value such that
غ 4exp(2iM)) p()EESe, P(
where functions p(-), ((-) are defined in Corollary and J.9 respeciiveyl.. We stress that 7 fixed constant while depends the value of e, and for any sufficiently small will have > f.
Let us define an event A = A7 U A where
A
3n T (8) s.t. [X]+))> 3
A A 3j 2, s.t.
// | ITy (8)41 +...+ =}
max 1=1,2,- (TT (6)
123

---

Published as a conference paper at ICLR 2022
where the positive integer 1" is defined in eq. (.1.. By definition of 1", must have K) <I" on event (K% 20).
First, we analyze the following event
2/
B (A*) N [KT oo) (17(0
for some j 2,3, , KK..
"
In partieular, note that B
=UK= B(K) where
B(K) (A [K] K [] Jg]
28
(0) - (0) > "
for some j = 1,2,3 ... ,K) .k..
When K 1, the event B(0) 0 by definition (due event {T) (T)
2f for some j 1,2,3 K)) For K 1, Lemma J.16 implies that B(K) 0 for all sufficiently small. In summary, B = @ for all 7 sufficiently small.
?
Now we focus on the following two events
2 (8), < C (A*)on -T
km.6 { K n (Tr. 8
for all j 2,
k3(()) Jg),
7
21
D a (AX) (K) < <o) (T))
for all j = 2,3,
khho n {Jka(ss 6
Jg).
7
On the one hand, event C can be decomposed as follows. Let
k-1
k
- (1,2, .m..................... k= (j), =-] Σ 10
1)
Σ (aj, - 1)3
be the set that contains the types j such that the accumulated cost reach Jg if and only the last element is kept. One can see that there are only finitely many elements in Jt. Now we have C' = Ujes C(j) where (see the definition in eq. (.10))
2
C(j) A (A*) n (K <oo f () - 17) 5
for all j = 2,3, K
n ((WT(()) ((WT()))W"())) ... W"
k2. (0)) of type-jj.
For any j J" observe that (let ki ())
sup Pa(C(i)) 2
< 2/ for all i 2,,,, 2,3)))))KK))))
(8) E Eg Vi E [K]:"] and (7( T
7
by (2.(8/n) (P) < 28 4ー- H(3/7)
The last equality is due to independence of (,) and (0) (0). The regularly varying natures of functions Hj and imply that
ky lim 710
(8/m)
8kyar
B(6/m)
(1/ml
H(17))
Meanwhile, from Lemma G.4 and the regularly varying nature function Hy and H (in particular, repeating the same calculations that leads to eq. (I.46).. have
lim (P(
2f/m) k-
H(1/m)
28
301
124

---

Published as a conference paper at ICLR 2022
Therefore, have yielded that
sup [ells
1aeePx2CC (c(())
akya
-( )
limsup nJo
< 00
25(1)
6mn
where
(y(1) 1
Th Hg.(1/n) - (Jg (
due to eq. (1.5
Therefore, one can see that as 17 17.4.. O.
sup P=(CC) = o(n +Jg ||
Furthermore, the discussion above have shown that for all sufficiently small,
Px (c, S o(17 +Jg -)++++++++++++++++++))+)- + sup (A x: ||
sup
||
From Lemma 1.10 and J.12 one can see that for all N 1 Jg a1 + and all that is sufficiently small, sup,: n: || P(AX) )(T)". Fix such 8 >> 0. In conclusion, we now have that supz: || Par (B²
n)) o(1) -1
On the other hand, on event D, Assumption 1.7 shows that (see the definition in eq. (J39
(K% < <00) (Jkm(n) Jg} [KK] <0)
(WT (
W WT (0)) is of type-ý for some j E ji()).
kig
Therefore,
B B2
=(A8) (KT < oo) n (6) (T) ... W "" kg. (5)) is of type-j for some j E ji(")
n 2,3, ... K% (] - (0) s 2//1 n T2
min j=0,1,
d (X), 9e) [
a KJ s.l. ( (0) 27/n OR Яj 1,2, K s.l. W (d) s sōj.
In particular, using Lemma 1.15
that when 17 is sufficiently small, on event B (c. 6, n)
D we have |||||| < 3e for all T (6) and
XL6s - =²0(0] < 2e.
sup '6 (1.)))777))))
Here
2²) = 3" (xx T. W) Vs Vs20
with T (0) (8) (0), T (W)( Wi) W k7.
(8)
T (6)) W
k)))()))
ky.
(8)). and ( XT7((0 (8) -1 Besides, eq. J.17
dictates that, on event
(e, 6, 1) D, we must have inf .20 (s)) > 1001"7, hence d(x) g > 50/*..
min o0
However, this clearly contradicts the definition of event B (c, 8, m). In conclusion, we have established that (for all y sufficiently small) (,, 6, D 0 and
sup Px (B 6, < sup (A*) z: ||
z: S2
However, as per the argument above, our choice of sufficiently small 8 > ensures that supz: S2 P(AX)= + and concludes the proof.
125

---

Published as a conference paper at ICLR 2022
Furthermore, starting from the local minimum O, it is unlikely that the SGD iterates will be extremely close to ag when the accumulated cost of large jumps reaches Jg. The next lemma provides an upper bound for the probability of the said scenario. For any set A C Rd define its enlaageemen as the following open set A _ {x: d(z, A) < e).
Lemma J.21, For any E (0, + 3p*)) and any 8 > 0 that is sufficiently small,
supz: |2 Pa (t. d))
lim sup
- (())
X(n)
740
where B (c, = {K3
<oo}
n {d(x) (2,6))
0g) < c} and V(())
(1 ((19 with h defined in eq. (J38) and defined in eq. (J.48), and p" E (0, 0o) is
the constant provided in Corollary J.7
Proof. Again, given the fixed > O, we can define i= C1 + log(1/e) and choose E>> 0 as the
largest possible value such that
D()E Se, 5()E5Se
4 exx(2i
where functions p(-); P(-) are defined in Corollary and respectiivl, We stress that is fixed constant while depends on the value of e, and for any sufficiently small will have > T.
Let us define an event A = AA U A where
A n < T7 (8) s.t. | 3e
A 2 3j 2, s.t.
k=,,2,- (T) (6) (T) (1)-1)(((f1)) 1 TTy Zry
+... B Zr, (01 >
max
where the positive integer 1" is defined in eq. B5 (c, 5, n (A*)a:
J.1. Consider the following decomposition of event
Co A Bg d.) (Jn. Jg}. k7d Can (g ( ( Jg] n ({3]== 2,,3,, ,K s.l.
(nn
K3g
TT)) 28/1 āj 1,2, .K%, s.t. KK W|] oō}
Can e (Bă (e, m) (A* (Co, Cam)
Using Lemma J.20 we know that for all 8 > 0 sufficiently small,
sup |2 Px (Can U Can) 0.
lim 740
X(1)
As result,
sup |a 22c
Pa (B (.,8.)))
sup || (Ax)
|1 Pa (Cum)
lim sup n10
< limsup nJ
+ lim sup nJ0
X(7)
X(n)
X(n)
From Lemma I.10 and J.12 one can see that for all N > 1 + Jg - on and all 5 0 that sufficiently small, we have sup.:
|| P(A*) -() = o(x(n))
Moving on, we focus on bounding the probability of event Cam- In particular, note that
Cam (A*) (KJ < <0) n (T n
.g] n [d (2.0 (n.6)' 0g) < c} kg
n Vj [vv
2.3. .KKS (T) (5) (6) 2 and Vjj 1,2, ... KK3. ||] 20}.
126

---

Published as a conference paper at ICLR 2022
By applying Lemma J.15 event Cum, one can that when is sufficiently small, this event we have ||x < 3c for all k < (8) and
sup (0,)) (8)
X (a147100
="(s>| <cc.
Here
x"(ss = (s,
x(1; (s, T, W) 2
with T (6) TT (6), T 3g (W) (o), ... W kgg kyy (6)), and x(I) = ( To better control the location of ="(ss)
(6) T
(0)) W
ki
also construct
(s) x"' (s, 0; T. W) V8 20 0
where the only difference is that we substitute the initial value x with O. Note that on event Cam we have | < 3c. By applying Corollary J.7 (the condition about line segments contained in g have the following bound on event Cam:
verified due eq. (.18))
sup
(s) (s)l 3p'c.
<E(0.>
T.
Now recall Definition J.T Combining all the bounds we have obtained so far, we see that, when 1) sufficiently small, on event Can we have
KB.6
d x Σ T.), 0g) (2 <2 3p")e
and W is of type-j for some j j(t"). It then follows immediately from Corollary 4.14thar
sup al Px (Can))
< sum (2 (2433")<,6.7) ,,,,,))
lim sup sup 750
S 8m 1 1-- (8g (2+3))+)))
A(n)
X(1)
I 8m μ hh ( (3+30)))))
and this concludes the proof.
Lastly, we provide lower bound for the probability of the most likely way escape, i.e., due to large jumps with accumulated cost Jg and relatively short inter-arrival time less (when compared to t/m). We stress that, in the next result, the value of constant c. would not vary with the choice of parameters 8.
Lemma J.22. There exists some C, > O such that the following claim holds for any E (0,7/(4 3)*)) and any sufficiently small > 0:
inf |2 (e, n))
lim inf
> c
X(n)
where the event is defined as (e, (,)) a (K7g <
(1. (T) -T7)))))-100)
21 Vj 2,3, (8)
n
and pt E (0, 20) is the constant provided in Corollary J.7
Proof. First of all, Assumption 7 Timplies that
(K3) <
(K3, =k' =k) TV. (6)
127

---

Published as a conference paper at ICLR 2022
We also stress that K3 k- means that (()) 6(c, 2 T (6) (i.e. the arrival time of the - large jumps and that the accumulated cost of large jumps (6))), ... W (6) is exactly Jg-
Due to eq. J.19 and eq. (1.77. there exists some j E [("),, some 0 = (0), . with 0g E F3 Vi, some F = (71,"* :.. T such that Ti 2 8 Vi, some t = (f1, tx- such that t = 0 and f; E (0, t)
for all 2, k² such that
d(h(r,0,t) t). 9) 100
where h is the mapping defined in eq.
1.38 Let ū - h f. Moreover, the continuity of mapping
h implies the existence of some A 0 such that (r, 0, X C h-1 (B(,,)) E)) where the open domain X' is defined as
X - ((,,0,t) 8, & = 0 and
| ã. < A I - न. < A 11 Edl < A Vi E [k*]}.
In particular, the parameter A can be chosen small enough with
A min T. A < min FAA 1=2.-.
=1.....
so that for any 0,,,t)= E X, we have 4, > for all = 2. K and ri > for all i E [k*]. Fix such A > 0. Mcanwhile, from the definition of the measure n(()) in eq. (J.40 (J.48) and the fact that 0g Fj. where the closed set Fj, is the support of the probability measure Sy. on the unit sphere gd-1 (see Assumption [.6]. one can see that (XX) > O.
Let R (R.) θ (0+) T (Ta) with R2 n ||W" O (0) (I) and T; = (6) - Th (0)) for all i 2 2 and Ti = O. Now recall Definition J.1 and consider the
following event
B (R, E x3 ( is of typ)-()))),, o)}.
Corollary I.14 then gives the bound
P(B) X(n)
lim inf 11
> 8mh u(x) > 0.
From now on, we fix some c, E ()))). Given the fixed c > 0, we can define f= 11-11000 + Aog(1/e) and choose ē >> 0 as the largest possible value such that
غ 4exp(2iM) p(tt PiAESS
E
where functions p(-); P(-) are defined in Corollary J.6 and respeecttvv We stress that is fixed constant while depends on the value of e, and for any sufficiently small e we will have >>> T.
Let us define an event A = A U A where
A 3n < T (8) s.t. [x > 3
A A {jj 2, k s.t.
// Zyp (6)+1 + Z,, (6)+) >
max
km 1 < (T (1) (10)77--00 (7) (8) (0)
where the positive integer 1 is defined in eq.
[I]
Now we focus on the event B B On one hand, Lemma 1.15 on this event we have | 3E < E for all & (8) and
shows that, for all sufficiently small,
sup xE (6) (T]
X2
[1|+27(6
=>{s)
< <E
Here
2²) = ~5hh (s, T. W) 20
128

---

Published as a conference paper at ICLR 2022
with T (8) (6),,, - (o))),
(6)
(6), W
(W), (o), (6). and x(1) X -11
To better control the location of
=*)s,, also construct
x² (s, 0 T, W) 2 0
where the only difference is that we substitute the initial value (1) with 0.
First of all, from eq. U.20p. we see that
d( (s). 100 d(=](s), > 1000 As [0,
Tt) ΣT)
Besides, due to X C h (B 2)), we have that d 00000 Ei=1 T.), G) > 99(".. Next, using Corol-
lary J.7 we have that (0. Ti ||= - [+)] < 3p < é, which further implies that
R² d( Σ T.).9) > 196 - غ 2 9
Now we have the following facts regarding the distance between SGD iterates Xj and the domain G. First, on event A it holds that
X(X7.99 > 1 - E 2-66 Vj < TT.(6).
implying that ((n) 2 T.. (0).
Next, if R(c, 1)) < -h (6), then there exists some (8 S < To (6) such that ||x]]] S 2e hence
[2 T| < 2e + 2x < 700/". However, in light of eq. [.20 we know that on event (R(E,, 6, n)) ( (B) ). we have
d(z")
T.). 100
and yield a contradiction. Therefore, on event B\ B\A* we must have R(e, 8,1)) (5). Besides, bounding the gap between Xy. =²) and z)) at time j = T (ö) using results above, we can show that
d(x²) p. (8) 9) > 981 - 2 9.
Therefore, we must have (1) = - (5). In summary, we have shown that, for 7) sufficiently small, A* C A Combining all the bounds above, we have
inf inf (A (,,
lim inf P(B) 700
sup 21 P(Ax)
lim n,0
lim sup n0
X(n)
X(n)
A(n)
sup |= P(Ax)
80n - sumpup
140
A(n)
To conclude the proof, it suffices invoke Lemma 1.10 and
J.12
with some N > 1 Jg a1-
J.4 PROOF OF THE MAIN RESULT
Now we are ready to state the main result and provide upper and lower bounds to the distribution of the scaled first exit time from domain G. In particular, define scaling function
X(1) e ((1))(()nn)
(J.56)
where the function X(-) is defined in eq.
J.50 One can easily see that À E RV1+Jo (n). We show that the scaled first exit time (n)) converges in distribution to exponential random variable as 1+ term and order is 77 1 O. which implies that, in expectation, the first exit time is roughly 1/n dictated by the minimum cost for exit Jg.
129

---

Published as a conference paper at ICLR 2022
Proposition J.23. Given any C E (0,1),u 0. the following inequalities hold for all > 0 sufficiently small,
lim nio || I2c
limsup sup sup (2(((())(()) w) s 2C + exp(- -1xp(((1-
lim inf inf P2(O(1)A(1)) > u 2 CC + expl (11 C)qu)
140 || $2c
where q (0))
Before presenting the proof to Proposition I.23 we make some preparations. First, we introduce stopping times (for all k 2 1)
Te 6, 1) = min (n > 1 7 Z > 6)
(e, 6, n1) min(n
6,, || S 2c}
with the convention that (c 6, = =)) = 0. Intuitively, at each F& the SGD iterates have just returned to a small neighborhood of the local minimum Due to Markov property of X the SGD iterates almost regenerate at each 7x despite the previous trajectory, and the times (7x k22 partitions the entire timeline into different segments that can almost be interpreted as regeneration cycles, where Th--1 can be understood as the starting point of the k-th cycle. In light of the embedded (informal) regeneration process, one natural approach is to determine the dynamics and probability of making an exit on one (hence every) cycle, and this will be carried out with the help of technical results in the previous section. is worth noticing that Tk are defined under the proviso that Tk 2 Tk where тą is the first big jump during the k-th cycle. Considering results such Lemma I.12 it reasonable to expect that the SGD iterates would be trapped at local minimum until a large jump strikes. Therefore, define the (informal) regeneration points Ty in such a way that a cycle [7k-1, 1, Fx) ends only if we have observed at least one large jump already. Regarding the notations, we add remark that when there is no ambiguity we will drop the dependency on e, ö, 17 and simply write Tk. 7t.
Specifically, we are interested in the large jumps and the accumulated cost thereof during each cycle, which definitely entails some systematic bookkeeping. For all & 2 1, the random variable
j ف #(n Th-I(E, 6, n)), 1 6, 17) + 1,... (e, 6, 11) A (()) 7 / > 8}
can be understood as the count of large jumps during the k-th cycle. Here are two remarks on this definition.
First, for any ls with (n) < T.. we have j. = O. Recall that the object we study here is the first exit time (n)), there is virtually no need to keep track of the dynamics of the SGD after it leaves the domain g.
The random variable jA is measurable w.r.t. FrAna()). the stopped algebra generated by the stopping time Tk (n), which, intuitively, is stating that one should be able to determine the number of large jumps during the k-th cycle when this cycle ends.
Furthermore, for each le 1,2, ..., let Te, 6, 1)) Tk-1 6, 1) A (1). Te, 8, m)) min {n T :n 11 > 6) A Ao(1) A Vj 2 2, (e, 8, n) (c,,,((.bb)))) 1
with the convention (c, 5. 11) - (c, 6, 1). Note that for any k: 2 1,j 2 1, Tk.j is stopping
time. Besides, for any with 2 < <n))
t 1 S Tk.j Fa Am)) Vj lixl:
(J.57)
and the sequences (T )= and (WaJ)le
l j=1 are the arrival times and sizes of large jumps during the k-th cycle, respectively. Again, when there is no ambiguity we will drop the dependency on e, 6, and simply write T and Wk.s-
We are now able to keep track of the accumulated cost of jumps during each cycle. For any k > 1 > 1, we define
Teycke (i:e,, 2 J(Wn.(e,&,m)
130

---

Published as a conference paper at ICLR 2022
Moreover, define the following index
(kie,, a minfj 1,2, Je Je
(ie,,,,) Js) (e, 8) (n))
x0
otherwise
with the convention that min 0 0o. That is to say, if 2 (n) or (g)"((()) Jg. we define
j (k) as oo; otherwise j² (k) is the index for the first large jump during the k-th cycle that drives the accumulated cost of large jumps reach Jg. For clarity of the presentation below, also define
(2 e, 8,11 - 1( 19 (l) < x R + 1 (k) = jo
x (k: 8, n)
- 1 و Jo (k) xx (k))e,..)) + 1 [ (A) = 20 )0.
In other words, when j 2 (k) < oo, the random variable X2 (k; c, 6n) equal to X7 a the
location of the SGD iterates right when the accumulated cost on the k-th cycle reaches Jg. Notation wise, we will drop the dependency on 6, 6, again and simply write j (l:). x2 or Th (j) when there is no risk of ambiguity.
As have mentioned above, we want to zoom in on the each cycle and analyze the probability of each possible case. To be specific, want introduce a series of scenarios, formally defined as several events, that exhaust all the possibilities during a cycle. The events will be denoted in the form of A or 3 largely following the next few rules. First, we say an event/scenario is atypical if its probability rather small, and we assign with superscript x; otherwise we say the event typical and add superscript o, Besides, the subscript & indicates the cycle that the event concerns. Lastly, events with label (of type) A usually describe how the SGD iterates try escape from G, whereas events with label B focuses on how the SGD iterates return to the local minimum.
Now we proceed and formally define the said series of events. First, for each & 2 1, define the event
A le, n) & Ee E [1*] s.t.
17 Zr, + ................................ +.....+...... 2 -(]1)
max T= 11, 1+2, (Te,. I)A(TH.: 1t| 251e) )aF
||
3e for some 7r-1, 7- 1, 1,. .K.1 -1]
(J.58)
with f(c) 0 + ci log(1 1/c) and function Z(-) defined in eq. [.51] It is similar to the event A
defined in eq. (J.52 eq. (I.53) in the previous section, and its probability will be controlled using a
similar approach.
Next, consider event (for all k 2 1)
A (c, = (a(n) < Rr Teyol((j))
(jn) < Jo}
(J.59)
that describes an atypical case where the exit occurs during the k-th cycle yet the accumulated cost of all large jumps is less than Jg. Another atypical event is defined as
x A ( < oo n {ーj = 2,3, ... j= s.t. Tki - T 2i(e)/n). (J.60)
2
This event describes the case where the accumulated cost of large jumps in cycle has reached Js but the inter-arrival time between some large jumps are unusually long. We stress that, by definition of 22 (k:), we have {ز (R)) < co} = - < a()) n {J (jx) 2 Jg).
Moving on, we consider the following events (defined for all le 2 1)
A 3,3 A [Jeycle (je) < Jg. ih < <a(n)) {3) (fj= = 2,3. ... .s s.t. TE. - Th.j-1 > 2f(c)/)
(J.61)
that describes another similar case where the h th cycle ends with return to the local minimum but the inter-arrival time between some large jumps are unusually long.
The next event
A & (1) < 0o, Jh 52 (())))
1 (k)) s.t. m || [ =d}
(J.62)
131

---

Published as a conference paper at ICLR 2022
describes the case where the accumulated cost on the k-th cycle has hit Jg exactly at some point with SGD iterates being rather close to g yet some large jumps are not large enough when compared to the fixed constant d.
Meanwhile, with events
A - [J= 10 (k) < 00, Jeycke 1 (k) J63 [d(x220 09) [
<e)
(J.63)
we analyze the case where the SGD iterates reaches somewhere close the the boundary set 0g with large jumps of cost equal to Jg. Lastly, define event
A 6 (j (k) < oo, Jeyde (jyde(j22u > Jg}
(J.64)
for the case where the accumulated cost of large jumps on the k-th cycle exceeds Jg without hitting it, As an amalgamation of these atypical scenarios, we let
A dc) - U.
(J.65)
Next, we analyze the probability of some events (B )k21 that concern the behavior of SGD iterates during the k-th cycle after T21e (k). Let us define
B (,,(c,,,)) = (j (l) < <0, J Jzyole (l)) = Jg) n (d [k) 2 c
n [TTss - Thp S 2 f(e)
This
Vj =j 2,3, ... _²
B (x, 8,1) _ (Tx- T ( p(c)/)) (a(n) < 7
B (c, e B B 2
(J.66)
where pl-) is the function in Lemma 1.11 From the definition of B in particular the inclusion of B one can see that the intuitive interpretation of event B is that the SGD iterates did not return to local minimum efficiently or even escaped from G after large jumps with cost equal to Jg. In comparison, the following events will characterize what would typically happen during each attempt:
(e, 6, n)
1220 do (R:) < oo, Teycle (12 1s (I:) = Js3 n (a()) T2 (1))
n [Tis - Tk < 20(c) Vj = =2,3, (k))
(J.67)
8, A (a(n) > FR 3h Tk, < 21 21r((e) + p(t) } (Trycle (jx) S Jg).
(J.68)
"
Intuitively speaking, A tells us that the exit happened right at 72 (k)) with accumulated cost of large jumps in the cycle being exactly Jg. and Bž requires that that the first exit did not oceur during the k-th cycle, and the SGD iterates returned to local minimum rather efficiently. It is worth noticing that, by d (k) < 1" 22 (k)- < co.
Our immediate next goal to analyze the event
k-1 (A (AFU)) (A U (N
(e, 8,
U
(J.69)
A21
In particular, we show that its probability can be made arbitrarily small with proper e, 5, n, implying that we will almost always observe the event (A )s.
Lemma J.24, Given any C' > 0, the following claim holds for all e > 0,8 > 0 sufficiently small: lim sup 740
sup $2
(A (,, SC
Proof. We start by bounding for the probabilities of all the A events. Fix some N >>> 1 + Jg. For
A Ae, 5, = fii E [1*] 8.1.
1 [||2TT. + +... 2,|1 > [()))
max 3=T T= (Te. ()A(Tt.TT 261 (AT
u [{[11791
3c for some j 7 1,... Tk,1 - }
132

---

Published as a conference paper at ICLR 2022
using Lemma J.10 and J.12 one can show the existence of some eo (0, 2/3) such that for any e E (0, co). there is some (b(c) > 0 with
/ Pa Po(AN,ol6 0 Ag E (0, (o())-
lim sup
(J.70)
1/
Next, for (,, = (o(n)) < <kk, Tk (1.) < J3 it follows from Lemma J.17 that, for any c C (0, co), there exists some (c) S (c( such that
|UP||=1||22 52c (A (c, 6, m))
sup
0 vA E (0. (e))
(J.71)
1/
For event A = (b) < 00) n [3j = 2,3, (k) s.l. Tu.j - Ta.jii > 2((e)/n)
Lemma J.18 implies that, for any E (0, ca), there exists some 0 < ò2(() S ōi(c) such that
lim sup sup 52 Pa Pa
4g E (0, (c())-
(J.72)
For event A = ( (ix) < Jg, nu < {3j 2.3, ... . s.t. T -Tk,j-1 21(e)/1}.
thanks to Lemma 1.19 one can see that, for any (0,en). there exists some 0 < d3(t) S 62(e)
such that
Pa (A o 4g E (0)))(e)).
limsup S2
(I.73)
For event
A k,4
(k) < co, Jeycle (4)) (d) s [j22 (k)) s.l. v [W s8}.
(d(x2
using Lemma J.20 one can see the existence of some A such that, for all e E (0, co), there is some 0- < de(e) S de(e) such that
52c (A (c,
limsup
Aa E (0, (c))).
(J.74)
11 +1
7
(1) (k) < 0o, Jocchhss (k)) Jg) n (d(22o 0g) < <c) the plan As for event k.5 2j
to use Lemma J.21 to control its probability. But before that, we recall the definition of function (c) / h ((g) ((gg)3+))))) in Lemma J.21 and make several observations. First, note that ag is a closed set, so Nesoh 1 ((ag))) h- ha-((0)) for the mapping eq. (J.38). Then it follows from Assumption I.9 that lim.jo 1 h ((99)))) = 0, implying that for e > 0 sufficiently small, we must have
Cc. 4
((e) <
where C' > 0 is the fixed constant in the description of the Lemma, and c. > 0 is the constant in Lemma J.2 J.22 We stress that both of them would not vary with c, 8, 77. Combining this with Lemma
J.21
we know the existence of some Es E (0, col such that, for all e E (0, c5), one can find some
6s(e) < da(e) with
S2 ( (00(((E) < 8" Cc. vo E (()))
lim sup
sup
(J.75)
X(n)
Lastly, for event A = (K) co, ( (f220 (k)) Jgh. from Lemma J.20 again we know that (let eg = es) for all E (0,e6), one find some dg (e) dz(e) with
P. (A 6, m))
sup
lim
0 48 E (0. (e)).
(J.76)
1 a1 +A
133

---

Published as a conference paper at ICLR 2022
Recall that (e, Ul_o (c, Also, the events B Ag.. B3 defined in eq. (.66)eq. (J67 eeq. (J.68) respectively, Before bounding the conditional probability of events B we make several observations. First, if we consider the event n (A) U B n nBj, the inclusion of the (B)) implies that during the first & cycles the SGD iterates have never left G. so
nBJ - (A) (BJ)"BB)) n (o(()) > Fx).
UBJ)
j=1
j=1
Next, note that
R-1
Px (B
n (Aj
(B))
j=1
P2 (B
n (A}
UBJ)
Px B
(A u J BB
0
(
We start by analyzing term (I). From Assumption
one can see that on event B E it holds that
j2 = k", (Wk,1," ... W is of type-j for some j E ji")
Tk.s-1 < 2 [1] Vjj = 2,3, .k. Due to independence of Wa, and Tk.+11 Tki and This
have
sup (1 (0) < 20(() k0 -1.
Σ (((W7(0)) WP. (0)) is of type-j)
||
(Ef())))
(0) 21(c) 65- -1
II (5/n))
Σ
H(5/n)
$:
)
Due to the strong Markov property at stopping time 7 Lemma G.4 and the regularly varying nature of functions H,, H,
suP S2 (I)
lim sup (2f(e)/8m1)* 1- (2f(e))601)*(((()))) (H(L/))/)) k-
II (H.(L/n)) ((CU)) S.
[5(}))]
gk'as
In other words, for any ē > 0, there exists some C(ee, 8) < such that
52 (l)
lim sup a40
s (j((")(C(e,)).
80n X(n)
On the other hand, due Lemma J.11 and the strong Markov property stopping time TE for any E (0, E) and any > 0,
Cc./4
lim sup sup (II)
[(+)))())))
nJo ||
In summary, have established that for any E (0,7) and any 8>>0 O.
Pa (Bx
0==1(A) U BB B
lim sup njo
Cc./4Vk2 21 1.
(J.77)
80s X(n)
Similarly, we can bound conditional probabilities of the form P. (A
(A3
To with strong Markov property.
be specific, recall that A $ - k.c- Combining eq. (1.00 we know the for all E ea) and k 2 1.
J.76
suP Pa (AX
(A) SU (B
lim sup 740
Cc./4 V8 (0, (0)))
(J.78)
801 X(n)
134

---

Published as a conference paper at ICLR 2022
On the other hand, lower bound can be established for conditional probability of event Ag. the event defined in eq. (J.67). Due to Lemmal 1.22 and the strong Markov property at fk-1, one can see the existence of some 67 (0, Eg] such that, for any (7) there is some < 87(c) I óσ(E) such that
inf 2 P2 (A
U nB)) c. vg (7,))())). 2 1.
lim inf 940
(J.79)
80n X(n)
In order to apply the bounds eq.
J.77 eq. (.79), make use of the following inclusion relationship:
(A) U (UJ
Bj) n (A J C A L
(J.80)
To see why this true, let consider decomposition of the event on the L.H.S. of eq. d.80p. As mentioned above, on event (A) first h cycles the first exit does not occur and leaving us with only three possibilities on this event:
UB nB) we know that a (1) > Tk- implying that for the
(k)) = oo (on {o(n) > k this would happen if and only if Teyd Jg):
(k)) < co, x2 (k) $ 9:
(k)) < oo, x2 (l) Eg.
Let us partition the said event accordingly and analyze them one by one.
On (n) UB y n BB) (AX J (Teycle < Jg), due to the exclusion of A (especially A and A A we can see that if j, < 1", then we must have (n1) > 7k and Fr- T S 21" ((e (1)) In particular, note that on this event we must have la j by definition of 1r in eq. (J.). Therefore.
h-1
( (A] U U (A UB (k) = co) S B.. f=1
On (N
Jo (k) < 00, X2 (kk) & 9), the exclusion A implics that Th, - Tk - 2f(c)/n for all j 2, 2.... 12 (k), and the exclusion of K, tells us that ( (k)) Jg. Moreover, the exclusion of A U A tells us that if x2 (k) & G. we must have de (k), In summary,
1-1
(j= (k) < co. x2 $ ( CA. UU U (A U B
. On (A) ( ( ) ( (j22((())
< co, x2 (k) E 9). the same argument in the previous bullet point can be applied to show that Tx.j - -Tk S 2f(c)/n for all j = 2. 2 (k), Jeycle (2 (k)) = Jg, and d(x²Jg (k), > c. Now
since B did not occur, we must have (1)) > 7. and F T2 (k) < ple) hence Fre Tk.1 2() Therefore,
k-1
(
U ( By) n (A U B ()2 < 00,
< x2 E C B
1=1
With eq. (J80 established, we can immediately get that
k-1
4-1
(
A4 UB 3B) n (A U ) = (N (4) UB
n (A (
B
==1
(J.81)
135

---

Published as a conference paper at ICLR 2022
Next, recall the definitions of A in eq. (J67 and Bg in eq. (J.68), and events A and Bi are mutually exclusive, since the former implies that the first exit occurs during the k-th cycle whereas the latter implies the first exit does not occur in the first & cycles. This fact and eq. (J.8T allow us to conclude that
that the two
(A) UB. UA))*
n (A UU
B
( (A) (U
A(A UU UA!).. BBBBBB) (B,B
(J.82)
The last step is to use all the results so far to bound the probability of event
A (c,
U
n (A) (EUURUAA)) n (AA
A21
Using eq. JJ82 we can see that (for any x with ||>||| S 2e)
(A x (e, ó, n))
Pa
(A
B UA2)*
n (AX (AZUB))
121
1=1
k-1
Σ R21
P
(A ( (AA (B
h-1
k-1
II Pr (Ó UA)
j-1
Σ R21
Pa
AA
UB
n
(A UBB n BB)
n
B
(A UB
R-1
Σ
Po
A
UB
(A
UBB))
k21
A-1
j-1
P.
(A
(B))
U UAi)
n (A (B))
Σ
Po
A
UB
AA
nB;
II P. ((A) UA;)(
n (A
(B))
P R>1
UU
(A
B
B
II 1 1-
(A;
n (A UAN)))) UA
This allows us to apply eq. J.77 .qq. and all 1) >
J.79 and conclude that for all c C (0, (0, all d E (0, 8x(())
(801 X(m) -e)) (1 801 80 X(n) - -
sup
(A 8,)
an
8
9c X(n)
2 = C.
80)( c
2
when 77 is sufficiently small.
Having established Lemma J.24 we are ready to provide a proof for Proposition J.23
Proof of Proposition J.23
Define sets
Ex (2) - Rd (
( d(y.g9) (3+ 30")2)
136

---

Published as a conference paper at ICLR 2022
where p² is the fixed constant in Corollary J.7 From the continuity of measure, have
lim
(()) = = (())),
lim (E e.0
(g))
From Assumption 1.7 we know - f(h (g9)) O. From Assumption 1.9 also have
((h=(ag)) hence (n-'1((0 ((9))) as well. Therefore, together with Lemma J.24 we know
the existence of some 61 E (0, 3F3p such that, for any (0, (1), there some < (c) with
lup sup PA(A* (c, m)) <C
100 ||
and
C) <
s9 (e))) < (1+
Henceforth, we fix some and in this range and prove the inequalities in the Proposition.
To proceed, let Jc, 6. - sup 2 1: 7 <)())) We show that event (A* (e,8, we must have J S Jt where (recall Definition J.1
(e, n) a min Is 2 1: Zn. of type-
with E E d(v.9) > (3 + 3p")2} where p" is the fixed constant in Corollary J.7 To see this via a proof of contradiction, let us assume that J = < J for some integer j. Now we make some observations.
First of all, due to Jt= = and eq. (.17), on event have
we
Tj T3a-1 S 21/1 Vi 2,3, 2,3,
The definition of event x y especially the exclusion of event Aj.o defined in eq. J.58 now allows us to apply Lemma J.15 and show that
X [8|+T5,
->"(s) < 2c.
sup KE
Tú.1]
Here
=0))) = (s, x();; T, W) Vs 20
with T = (0, (0,T,22 - T2,1, - T.,,,.. - TJ
-1) w (W ... W and
1).
(1) - Moreover, the previous bullet point allows us apply Corollary J.7 and conclude that
sup E (.,T, N-
11² =3(s)) < 3p
T3.1
where =0()) = x (s, T, W
2 0 and p* is the fixed constant in Corollary J.7 As
result, |0 (T - T2,1) X²
< (2 3p")c.
However, due to Jt j, we have ( T,1),9) (3 (3+3p")e This immediately implies that X7 x7 & G and yields the contradiction J S.
Similarly, one can show that on event (A x we must have J 2 J for
J+ 8, e min k N 1: Zne, is of type- (Rae-(EE(0 (E)
with E- (=) ddd (v, Besides, the following claim holds event (A x ) (J oo).
(3 3p")pp) 3p*)z}.
137

---

Published as a conference paper at ICLR 2022
The definition of (A yo implies that during the J-th cycle the event A occurs whereas for any j < J we have Bj. Therefore, for any kt = 1, : we have
F (1)) Tk,, 21 ((e) + p(e)
7
In particular, the coefficient 1 on the R.H.S. can be justified follows: by definition A; and By we can see that Jh Jeycle (Jx) < Jg for all k S J (with j. = h due to the occurrence of A); the definition of I in eq. [.1 then implies that 1* J, for all k J.
Now we consider the following set
S(e. 6. U221 17--1 + 1, 1 2, ,T., 1,Th,13
the can be understood as the concatenation of all steps between any return time h and the first large jump time Tk during the - cycle. Our discussion above implies that, on event (A x ye, we have then the discussion above have shown that, for
min 8(c, Zn is of (ype-(Et (c), 2 Ty,1-
It worth noticing that the probability of Zi being of type- (e), 17, 6) (see the definition
in eq. (.47) (60/r)p(L E (c),8,n) n).
Therefore, for the following partition of the timeline
Spctore (e, 6, 7) {n E 6)) a a()),, Abctore d, 2 #S 6, m)) Satter (e, 6, 7)) {n S(e, n 5 o(])), Later 8, 1) a #Saner 6, n)), we have o11) = Thefore + Janer- Moreover, event (A )c , must have
Intter J ((e) p(e)) < Je (2( + p(e))/1
(,6,,,1 n) Z, of tyteeye-(e (aapeeeE""rre ())) Ihefore S min(n E 5(e,,,
Next, define geometric random variables with the following success rates
Ui(e, 8, Geom (1(6/))((E (e), (e),
U2(e, 8, Geom (t)
Given the results above for bounding Jucfore and Later on event (A )S, we have
sup Pz (x(()))())) >
|allS2e
< sup (((())))= + sup P. (A n ((n)( + -Iater) > u}
|
||
sup Po (a((A))))
sup Ps (AA*) (2(1) Loctore > (1 Cluu
|
|
+ sup P. n ( {a(n) Ou})
S
+P((()) < sup Pe ((A + P(((m) U1> (1- C'Ju) +P A(n) U2 21r + p(c)
Clu)
|e
A(D)
For term (D) recall that A(n) (m) Corollary J.14 and the regularly varying nature of H then imply that
lim H(8/n)p(E (c), (e). ((6mn) P(E E (e), -1 (1 CC). 740
X(1)
H(1/m)
8an X(m)
138

---

Published as a conference paper at ICLR 2022
From Lemma G.3
then have lim sup,1o (I) I exp(-(1 C)² For term (II, let C(c) -
fc) p(c) and note that
C X(n) 1 )
(I) = (1)
C(c)
M()/nn X(n)
Moreover, since
X(1) a
Mnn)/n +1 (1),
know that for any M >> O, have A(n)
O. Therefore, given any M > O, we have (II) - P P > M/X(n) for any 7) sufficiently small. Now from Lemma G.3
M eventually
X
lim supæyo (II) S cxp(-M) M) for any M > 0, and here we fix some M > such that exp( M) < C. In summary, we have shown
lim sup sup P2 (N())((n)) > w) s 2C exp(-(1 ( Q)²qu) n0 S2
and established the upper bound. The lower bound can be shown by an almost identical approach. In particular, since
PA (N(n))(1)) > zu) > inf P. ((A*) n (Mi)) (hhefore + Latter) > u²)
inf ||
||
> inf Pr (A* (MTr) > =w))
|
2 pP((n) Uf > u)
sup Pa (A (A))
||
where (e, n) ~ Geom (F)))) n)) The same calculation above for term (I) can be used here to provide a lower bound for lim inf P (((()) U3 > u) and conclude the proof.
Now we are ready to prove Theorem I.I the main theorem of this section.
Proof of Theorem L.
Recall that our choice of scaling function X(-) in eq. (.56)) is regularly varying
(w...t. 1)) with index
+ Jg. Fix some E G, > and E (0, 1). suffices to show that
lim sum (=(o())A(n) > t) < exp( (00001111000 - -) + +
n40
lim inf (o(())))) t) 2 exp(-q(1 + C)t) C.
700
First, we able to pick some (0. 1) sufficiently small such that ((),9") and Proposition I.23 is applicable. Due to Lemma J.11 for event
A A (Tretum (n, e) S p(e)/n, X E G Vn S Treumn (11 e)}
we have 1o (A) 1. Next, on event A, we have (n) T Tretum E) > 0 and X
X Tomes))))) s
2e. Moreover, by combining Proposition 1.23 with the strong Markov property at stopping time Tretum (7), e), have
lim sup P. ((o(n) - Treuum (1, c)) > (1 (1- C1-C)t A) 2C + exp(-(1-C)) 9J0
lim inf P. (no(n)- -Teeum > A) -C -xp((((11++)) 940
Observe that
< Pa (o(n) - Treamm (11, k)) ((T) (1 C). A) PA(AA
+ P. (Trewne (TT Ct. ct) - +Pa(A)).
139

---

Published as a conference paper at ICLR 2022
Besides, event A have (n), c) Σ O(1/n) as 17 4 O. Given that X(n) (o)))))((), have retumn (1) e)) & Cl. on event A for all 1 sufficiently small. Therefore, by applying the bounds above, we establish that lim sup, (n) > 1) 2C + exp(-(1 C qt). Similarly, in order to show the lower bound, observe that
(a(o(()A()) > 1) 2 P=((o(m)(()) > 13 n)
N PP (( Treuu (7, <)) A()))))))))))))()))))) >t t| (tt1a)P. Pa(A)) )
Taking lim inf on both sides yields lim inf, (=(o(1))()) > 1) 2 -C exp( C)q() and concludes the proof.
K PROOF OF TECHNICAL LEMMAS
Proof of Lemma
G.3
For any 6> 0.
PP P(U())> b(c)
1 = (1-a(e) (1/0()))
By taking logarithm on both sides, we have
IP U(e)) > b(e)
dte) = L1/b(e)] In - -((e())
[1/b(e] In a(e) -a(e) I/b(E) -a(E)
b(e)
Since lim In(1+x) 1, we know that for sufficiently small, we will have
a(e) b(e)
1 = < In InN((U())
a(E) s c. b(e)
(K.1)
By taking exponential on both sides, we conclude the proof.
Proof of Lemma G.4 To begin with, for any e > 0 we have
P(U(e) S b(e) ) = -1 (((())
b(e)
Using bound eq. K. we know that for sufficiently small, (E) > 1/b(e)) exp(-c- a(c)/b(c)) The upper bound follows from the generic bound exp(- (I) S I, Vz E R with -=cca(ce//l
Now we move onto the lower bound. Again, from bound eq. small 6, we will have
K. we know that for sufficiently
P(U(e) 1 b(e) 2 exp(
a(c)
Ve
b(e)
Due to the assumption that lim, ((e)/b(e) 0 and the fact that 1 exp(-x)) 2 Vo for x
sufficiently close to 0, will have (for small enough) IP (U(e)) I b(e
a(c) [(e)
Proof of Lemma G.3 Let ag
Tz. Using intermediate value theorem, one can easily see that
Vg(ā,-1) Vo(2)-1)) + n721 +... 2x) + x z;
ag
1=1
||ax|| < C/Co(ll +
||a- +
The desired bound then follows immediately from Gronwall's inequality.
140

---

Published as a conference paper at ICLR 2022
Proof of Lemma J.1 Due the finiteness of all types j' with (iy) < Jg, it suffices to fix one of such j' and prove the existence of constants Ł, 8, 60.
Let be the constants in eq.
J.3 For any r E (0, E), define the following stopping time
(x) e min(t 2 xx(a) E B(),r))
Due to Assumptions [I4 and [.5 as well as continuity in Poincaré Map (see Theorem 12 in Immler & Traut (2019)), for any fixed r > 0 we know that (-)() is continuous function on G. Note that, by B(0,7) r). Due & being compact and eq. J.3 we know
definition, have (2) 0 for any the existence of some constant T, E (0, oo) such that supzeg (.( S T, and me (x)x)) E B(0,r) for any 2: E g, 2 T.
Due to Assumption L8 and the finiteness of all types j' with Jyye((') < Jg, there exists (0, e/2 such that
sup d [ Oct, w) 99) > 260 220
(K.2)
for all j' with (i") < Jg. (0) x 1 E Ayre We fix such 0. Here is one implication that is worth mentioning. Recall that is fixed in the description of the Lemma and k' = [j). If
inf d 0; t(k)) w (()) 99) S co #20
for some (k4) {0} x B$ (t) E A.T then we must have
d( t, 0(00), (1''),, 09) S ED.
To show the existence of some 8 0 we appeal to proof by contradiction. (For clean pre- sentation, in and w we omit the (k') term in the superscript since the cardinality is fixed.) As- sume that we can find a sequence (t") w=)),, n21 such that min { [||W ... | || } s 1/n and
d @ D - 4, O; t", w").9) g") S for any n. Due to the truncation operator (p(()) in the definition of WG, without loss of generality in compact set. Define
replace all jumps wy by ypb (wx) to ensure that w/ is always
" = ( 0; t" w=) Vj = 1,2,
By picking subsequence when necessary, can further assume that
For any i 2 1, w
(n) converges some wt and yu converges some y particular, there exists some [k]] such that w =
Also, since G a closed set, we have d(y 9o) S cp.
For any = 2,3, ... ,., either converges to some finite 17 or lim, 1 = 0o.
Note that
lim 17 0 + y = 0
(K.3)
for the following reason: for any r (0, Z). our discussion about (x) the beginning of the proof implies that lim sup, |/,l1| <r.
Let i be the largest index i [A'] such that y 0. In case that we cannot find such index, let 0. We first consider the case where = The bullet points above then imply that, for any 2 we must have lim, to = < co. Now due to boundedness of E and the continuity of ODE flow, for
t (0,12, 1 ttr),
w (wi, W w ww))
141

---

Published as a conference paper at ICLR 2022
have
ac 4/ 0; t" w") y
Now recall that wj 0 for some E [K]. By removing the vacuous (size-zero) jumps in (t", (t",")) w"),
we now know the existence of some k' E E, it (B" t' E {0} x R such that d d( R'' & 0: ".)) I co- Meanwhile, the condition T (j) < Ig implies that, for the total cost jumps in after removal of size-zero jumps, Σ ( < Jg. However, this contradicts eq. K.2
Similarly, in the case with I 2 1, we have that for any i >> I, we must have lim, ty 1 From the boundedness of &ę and the continuity of ODE flow, for
t" (0, 1 742'" t
(K.4)
w - [wj, w i+1* w T+2" 21 w
(K.5)
we have
T Σ 0, w²) = yi.. i=I+1
(K.6)
In particular, i = 1, then for the index / with wi = O, we must have that > I, meaning that there is at least one vacuous jump in w". The same argument for the case I = 0 above can lead to the same contradiction with eq.
K.2 Otherwise, with I 2 2, we already know that the accumulated cost of all jumps in w* is strictly less than J/g, yet we still have d (((( 0, w²),,9) I Eg. This contradicts eq.
( again.
In summary, we have established the existence of the lower bound > 0 on jump sizes. We fix such 8 > The existence of < oo can be shown by an almost identical argument. In particular, if such I < oo does not exist, then by picking subsequence if needed we are able to find a converging sequence (t", (==)))))))))) such that eq. K. holds and I 2 due to inter-arrival time blowing up to infinity. In particular, since y wi and |
must have 7 > 2. By considering the K.5 one can see that the accumulated cost of jumps in w strictly less than Jg and yield a contradiction with eq. (K.2)
> 8 >
same t", w" pair in equation K.4 [K.4]and and equation
Proof of Lemma J.2 Due the finiteness of all types j' with Tnype < Jg. it suffices to fix one of such j' and prove the existence of the positive constant 61-
To proceed with proof by contradiction, the assume that such 6 0 does not exists. (For a clean presentation, in and w we omit the (k') term in the superscript since the cardinality is fixed.) As result, we able pick sequence (x=", tt,, w") w")21 such that one of the following two cases must occur:
lim, || 0; For any 2 1, inf .2 d(#(s, t" w,," I eu/2 and there is some in E [K'] such that 17 2 27:
lim, ||~||| 0; For any n 2 1. inf inf. d(ā(s,x", t", ",,w",, w,) < e and there is some
jn E [K]] such that |00, < 5/2.
We detail the analysis for the first case, as the second case can be addressed by an almost identical argument. First of all, due to the truncation operator sp((-) in the definition of Tt. without loss of generality we can replace all jumps wi by you (w()) to ensure that wy is always in a compact set. Define
V = के (C) w"; t", w") Vj = 1,2, By picking subsequence if necessary, further assume that
142

---

Published as a conference paper at ICLR 2022
(n) converges to some wj and yf converges some y.".
For any i 2 1, w
Also, since g is a closed set, we have d : 9") S cu/2.
For any i 2, .... k', either converges to some finite eú or lim, ff oc. In particular, there some I [k'] such that lim, 17 2 26.
Due to eq.
K.
lim, e = oo would imply y. O.
Let I max(i [k] lim, of E [27, x)) and /2 max [M] lim, 47 x).. If either of the two sets above is empty, let the corresponding I1 or Iz be O. The discussion above implies that least one of them must be non-zero. there are only two possibilities: (i) 5 1 < 12; (ii) 0 S 1 J1 We consider each scenario respectively.
First, if < < I2, then lim, Vi, vi, O, implying that I2 < K. Moreover, for any I2 + 1, 12 ., ,, lim, E tú sc. Using the boundedness t and the continuity of ODE flow, for
(0, 00,ti+ 12 12
) w" (wfz wi+ 12+ w.));
we have x 14 t 0:t", ,.) y with d 9") S co/2. However, this contradicts equa-
tion L2
Next, in scenario (ii) with 0 < 1 < T, we have lim, 17 = = 2 27. Moreover, for any i i= +, 1, 12 + 2, K, lim, E = t < co. Using the boundedness of t and the continuity of ODE flow, for
(0, 1, 2+12" 1 w" A (wf, w 12 12+12: wx))
rk' we have 2( O 11 t 0, w²) = yi with d (") S eo/2. However, bmw" is still of type j
with Tope (J) Jg. and there is some such that to 2 2ř. This would contradict Lemma J.1
In summary, we have established the existence of e1 > 0 such that L < 27 is necessary condition for any I (00,e)) e1) and any (f1..... (x) {0} x - w = (wl. ww) E A with inf,20 df (5(s, æ; (R), shown in an almost identical way. We omit the details here and conclude the proof.
w (+)), g S 22 As mentioned above, the necessity of ||w;] 8/2 can be
Li
Proof of Lemma J.3 Since there are only finitely many (i.,-.., im) with J(11. ......... im) < Jg. suffices to fix one of such (i1.-.. (mm) and establish the existence of the required eo, 80. (Henceforth, let = Σラ -
Assumption 188 together with the bound in eq. (.6). implies the existence of some E (0, E) such that
sup d (m(s 0: (8) w (+)) c1 200
(K.7)
for any w A(i im), (() {0) x RR- For éo = e1 /2, establish the existence of the prescribed 6n via a proof by contradiction. (Henceforth we drop the notational dependence on (k) when referencing sequences t(k) (k) since the cardinality is fixed.)
Assume the existence of sequence (#w)n21 in Rd,, sequence of real positives (Sn)n21, sequence (tn)n21 (t1, t4 m221 in {0} 184-1, and sequence (w,)m21 (wi,n:".. Wk.nn21 in A(ii, ... (m such that lim,, [x I| = 0 and
d ((((8,, Tni tn wx), 9") e0-
Due to existence of the clipping operator, all jumps wynn can be replaced by yo (w),)) without loss of generality to ensure that all winn are in compact set. Also, without loss of generality, all 8n can be chosen
Sn = inf 0: d(ø(s, Zni tn wn), 09) S Eo).
143

---

Published as a conference paper at ICLR 2022
From Assumption eq. J.6 see that Sn must be equal to Σi iinn for some in E [k]] i.e. must be the arrival time of some jump. Moreover, one can easily see that for
1/n (( Ini i bn, wn),
we always have d(yn 9) < b so all Yn are in a compact set well due to the boundedness of g. Therefore, by picking subsequence when necessary, we can further assume that
There exists some j" E [k] such that 8n, El lim for all 2 1, i.e. 8n always the arrival time of the -th jump:
For any j [ī*]. there exists some vj such that for yn.j #(Ei= ti Rni tin Wn) we have lim, Mn.j yj. In particular, for any j j we have d(yf, 99) 61 and Uly5..99 I EDi
For any j E [ki], there exists some w/ such that lim, wyn = wy. Moreover, note that v =w.
In particular, (wi, w A(ii, im):
For any j E [k], either there exists some tj < oo such that lim, tn,n = tj. or lim,, f, oo (in this case we let t = oo); in the latter case, due to the same argument in eq. (K.)). we
must have lim,, 2 DE ti Zzj tns wn.) O;
V3 =0} with the convention that Obviously, 2/- * Let it A max(j 0, . 36 = O, Vo.n = Zn. We must have i < < and
limm Vi..n Vn+ =0
For any j = jt + 1, + ., ,J", lim, Inn 17 < oo.
Now using the continuity of the ODE flow, must have that
limæ
Σ
ii Vi (fu+tims 41. 2,a ,ty mn) Wij++inn Wj. +2,n " " "
1=
Σ 0; (1 ...)) (w), W3 11 , = Vz..
1=
However, due to d(yj-: 9") S and recall our choice of = c1/2, for all sufficiently large, we must have
d @ Σ ti Vjumi (15 42 (t) (Wj +1,ns Wij +2mm "o. "."",)) ..) g c1 j=1+1
Meanwhile, using Gronwall's inequality repeatedly and supj=iit1,- of mi ty < co, by substitut- ing the initial condition Vi with O, we have
lim
iym Vinzi (tittim to +2. mn), (Wi++ Wj " .)).))
j=stt1
Σ
ty O: (1ut (attaassi2220 2,m. m)) (10g Wj +2,+" " ,,z",,))) 0.
j=1++1
This implies that, for all n sufficiently large.
"it42m""" Σ Ij,ns0; O; (11t21 ,4i+ (a)s (Wi +,,, "jj w...""")))))) S 1-1-
J=/++1
However, this contradicts eq. (K.7) This implies the existence of the required 6 and concludes the proof.
144

---

Published as a conference paper at ICLR 2022
K.1 SUFFICIENT CONDITIONS FOR ASSUMPTION L.
In this section, we show that under a proper set of regularity conditions on the boundary set ag and the distribution of noises Zn. Assumption I.9 will hold for (Lebesgue) almost every b >> 0. In particular, we stress that the C2 condition about manifold ag in Assumption L.3 as well as the
condition that measures S, are absolutely continuous w.r.t. the spherical measure or on gd-1, are only used to prove that p aa--111000 (aG))
and will only be applied in this section.
The key of our argument is the following geometric observation regarding the intersection of - ag and äB of some ball B. We stress that, as made evident by the proof, this lemma is essentially based on two assumptions: (I) As the boundary set of the connected bounded region G. ag is a closed set in R (II) As (d - -i-dennnsi manifold, ag of class C².
Lemma K.1. Let dz.b be the spherical measure on the sphere of the open ball B(x, b) for any E R b > 0. Under Assumptions 1.1 and I.3 it holds for (Lebesgue) almost every b > 0 that
Ju. [G n aB(r, = 0 Vx ER.
(K.8)
Proof. The fact that 0G is a subset of a separable metric space implies the existence of a countable atlas for this manifold. Therefore, we can find sequence (U1)221 that are bounded open subsets of Rd-1 containing 0, sequence (Vi)a21 that open sets in ag (in the metric space induced by Euclidean distance), and sequence of injective C2 mapping fi with f; Rd-1 Rd and fu(()) = Vi such that ag- = U.V.
Next, we zoom in on a specific chart (U,, Vi, fa) and observe the following facts. For any x U y E 1/2 with f((x)) = y. there exist some orthogonal matrix Qy E Rdxd such that Qyn(y) = (0, 0, ... 0, 1) where the vector field n(-) is the outer normal on ag. Besides, there is some vector ay E Idd such that Qυy + ay O. Moreover, there exist an open set on Ü, in Rd-1 containing 0, an open set y E Vy C Vi an open set V. & {Qy + ag : E V33. C2 function õy U2 + IR satisfying 9y (0) = 0 and
Ty (Wi, wa- 1) wa Yu (W1, ".., wa) Vy.
In other words, for any given y on this chart we simply rotate the chart to ensure that the tangent space at y after rotation is { 0) : x, E R E (d - 1]]. and reparametrize the C2 diffeomorphism associated to this chart around y so that the coordinates are simply the projection
onto the said tangent space. In this sense, the (rotated) manifold is also the graph of the C2 mapping Py. This allows to define (for any y aG)
A(v)
A1 9, (0)) do (V'),
A (v=( (0))
where, for any real symmetric (d - 1) x (d 1) matrix A, (A) 2 A² .... X are the ordered eigenvalues of the matrix, and V (-)())) is the Hessian gy. is worth noticing that A(-) a continuous function (on ag) due to the manifold being of class C²2
Restricting our discussion on some fixed chart (U,, Vs, ft) for now, for any b > 0, let
A,(b) {o E U,: for y = f1(x), 3j E [d 1] such that ( 9, (0)) = 1/b).
Note that for any > O. the set (b(b) is closed set (hence Borel measurable) since the continuity of A(-) implies that (b)) is an open set on U.. Furthermore, since misb (U+) < oc, there are most countably many > 0 such that miss mL.s(A, (b)) > O. Given the countability of the atlas, we know that
B
(b 0: 3i E N s.t. mla((A > 0}
contains at most countably many elements. In the rest of this proof, we show that eq. K. holds for any 6 such that b & B
Henceforth, we arbitrarily choose some b > 0 such that b & B*. We also arbitrarily choose some x E Rd and let B = B(x, b). To facilitate the discussion, we introduce a concept that is closely related to the set (b): let
(a) EUE V:: (w) E A.(b)) (u V. A(y) (b, b, b)) or (-b, -b.... ... -b))
145

---

Published as a conference paper at ICLR 2022
and let (b) = (b). Now consider the following decomposition of the sphere ÖB:
B2 B (b).
B1
Ø B2.
First, note that for any y E B1, one of the following four cases has to occur:
y $ ag;
y ag and the vector y x: lies in Ty0g, the tangent space of the manifold ag at y;
y ag; the vector y x is T, yet is not orthogonal T, either, i.c. y is not equal to c. :n(y for any E IR where n(y) is the outer normal at y;
. 0g; the vectors y and nly) are linearly dependent.
Our next goal is to show that, in any of these four cases, we can always find set y E O, that is open on the sphere ÖB such that a (O, n ngg = O. Note that this is obviously true when & ag, since both ag and êB are closed sets in R Now we consider the second case. If the vector lies in T, then after applying the affine transformation with orthogonal matrix Qy (recall that Qy = (0, ,0,,.,00.. (.))), we have
Qu(y x)) QyT,og- {(w1.-.. wd-1, w; E R Vi E [d 1]}.
Since Qyy ay 0, now know that after the affine transformation, the center of the ball B moves to
2σ + By Quy a, Qx(x y) [(wl,-.-.. wa-1, 0): wy E 1]}. Without loss generality, we can assume that Q ay = (b. 0, ... 0. 0). In other words, after the affine transformation, the ball becomes B, Q, + ag
B( 0,0,0,00..... -.. 0), 6). Moreover, for any
w (W1,"* .. wa) E aB.. we must have w1 2 and (wi b)2 + (w²)² + .... (wa)² 6².
Meanwhile, from the definition of the mapping Py, such that for any w E Un with w ag öB (hence = Qyw + dy V,, E ²B,,) such that
that there is open set U, around y
(w)² 6² (wi b)² w
W2-1
Let fwm wd-1) - (w., ... wd-1) w 2b + w + 2 Now now that F(0) = 0 and f())) = 0 for any w V, E BB,. Moreover, by definition we have (ū) 0 so 9_ [(0) 2b / O. Due implicit function theorem, we now know the existence of some open Tue
set U in Rdd containing O, some 2 function g" R - R such that for any V. n ÖB n
= g (W2,- wd-1) 1), wa = Ty (wi, ... Wd-1).
Therefore, within some open neighborhood Vi of such y, the set V ØB n ag is submanifold with dimension d - 2, so we must have
(V y n ØB agg = 0.
Next, we consider the case where E ag n B, but T,Ög. Similar the construction of the affine transformation with Qy ag above, one can find an orthogonal matrix 2 and vector ā such that Qyy + a 0 and (y - x) = (0, 0, .0,0),,),,))) b). Let B B((0,0, 0, bb), b). In other words, this time the rotation constructed ensures that, after rotation, the vector between Quy + ag = (on the sphere AB) and the center Q + ãy of the ball B is aligned with the d-th axis. Moreover, there is open set y E Vallt C Rd and C² function Tyl -gd-1 1 + with V (0) ef such that for any ÿ E al A Q2 Vall + āy. we have y
is neither in T, nor orthogonal to
Jd (ū Td-1)-
146

---

Published as a conference paper at ICLR 2022
By applying change of coordinates if necessary (which can be achieved by multiplying another orthogonal matrix), we can assume without loss generality that Vgy (0) = (0, 0, c) for some
+ O. Then for any w E y such that w E @Bnagg let = Q2 u ay and note that we must have
w w + + w + (b+ By (w1, ... w )) b²
In particular, Ty" (w', .M 1 = cud-1+ti (w) for some C function with r(0) = 0, Vr(0) = As a result, for function f(w) - wf + w + + w + (b + "g"""""" (wi,... wa- 2 we have 8 ((0) 26c + Using implicit function theorem again, one can see the existence of some Twa-i
open set 0 Vy IR and some C² function g" _ + R such that for any w A n ngg n ÔB, we have that (for to Qy + k.))
Wd-1 = 9 (W1.-.. Wd-2), wd Tyy(( (w1, wd-1).
Again, we have established that within some open neighhorhood Vi of such y, the set V noB nag is a submanifold with dimension d - 2, so we must have
x. (V n n& n 8g) = 0.
Lastly, consider the case where y E ag n B2 and the vectors y - x and ny) are linearly dependent. In other words, the tangent space TG is also the tangent space T, Since y & B,, we know that (w) w Ay + (w) where A, is a real symmetric matrix with no eigenvalue equal to 1 and rı a C function with [r] (w=) = o( [|=lll On the other hand, for any ú in the open set Q + By we have
Vy C OG, if we also have w E ÖB n ÖG, then for
ap = 9y (Wi,... wd-1) / 62 w/ w2
w2 b.
(K.9)
Also, note that /12 w w
w2 = - (ww, ... wa-1 1--1 (wi, Wd- 1²
wd-1) where r2 is also a C2 function with |r2(w)] = o(11w]13). Therefore, for any (Wi, wd-1) satisfying the equation eq. (..9,, we have
I
2 (W1,"" wa-1)
(w) wa-1)" -r (w1, wd-1) r2(w1, Wa-1). b
However, for the real symmetric matrix A, 1-1 b note that none of its eigenvalue is equal to 0, implying the existence of some e > 0 such that
Ia (w1.... (A,
(wa))"...a-))" 2 <(w? + ... + w.
2
b
For this fixed e > 0, we can also find such that
rı (Wi, wa-1) r2 r2(w1, wa- ))
E (w/ + ... w
2
for any w ... + w < 5. As result, the only solution eq. K.9 with w
w <
Wi = W2 = ... Wa-1 = 0. In summary, we have shown that there exists some set y E V that is open in R such that Vy n 0B ag = (v).
Collecting the results we have established so far, we now know that for any y E B, (recall that B, an open set on @B), there is an open set V containing and satisfying a nn ØB n agg)= 0. In particular, given the open cover UyER, V. B,, Lindelöf property then allows us to extract a countable open U121 V B₁ cover and conclude that
(B ngg) S Σ Zo. n Ø ag) 0.
22
Moving on, we evaluate Zr,6 (B,6(B2 n ag). For any y E ÖB, one of the four cases has to occur:
y & (b());
147

---

Published as a conference paper at ICLR 2022
y (b) and the vector y lies in Ty0g, the tangent space of the manifold 0g y;
y E (b); the vector x: is not in ,, yet it is not orthogonal to Gg either, i.e. y =
is not equal to c nny) for any E R where nty) is the outer normal at y;
y (b); the vectors y and (y) are linearly dependent.
Again, show that in any of these four cases, there is some set y V open in ÖB such that (z)6(A Vig) 0. In the first case, the fact that A (b) closed on ög immediately implies the existence of some Vy such that (b) n V = 0. For the second and third case, this can be shown using exactly the same implicit function argument above. For the last case where A (b) n OB and the vectors y J and nyy) are linearly dependent, from y E A n nB we know that y V, where (U,, V,, f&) is chart of OG and V; is open on aG. Moreover, recall the construction of open set y E Vy C Vi the beginning of the proof. It is worth noticing that V4 (6) Vin (b). Besides, due to the fact that the vectors y x and re(y) are linearly dependent, we know that the tangent space a, is also the tangent space O, Therefore, by definition of gy and Q2 ay. we know that Quy + by = O, and under the affine transformation, the ball becomes (( O, ...... Eb).b) b). Without loss of generality,
assume it is B & B(O0,0, 0, 6). b). Morcover, since & B", we have that
mLee ((w E Üy: of the C2 mapping Ty- Vy is the image of the mapping, and V, - {Qy + ay : w E Vy} is the image of Vy under the affine transformation. Therefore, for any i E A (b) nOB, let w = Q2 t6 + dy and we must have
A,(b)) 0 where, as defined the beginning of the proof, Üg y the domain
wd Bg (W1, Wd-1) = - 62 w w
w
(W1,,... wa-1) (w Uy A(()))
Now
let
V*
A
azy
v
(Wi, Wd-1, -b
V
682
w wi
w for some w E U,] and note that E Vi is an open set on the transformed sphere B and then
ØB. (Specifically, note that simply identify open perform the inverse transformation to move the set back to the original sphere @B.) Then it follows
immediately from mlob(fw (w Ü, A,())) = 0 that 2 (A4 n V.)) = 0.
In summary, for any E θB, can find set y E V open ²B such that o.,b (4)()) = 0. Lastly, by applying Lindelöf property again, we extract a countable open cover U221 V B2 cover and conclude that
orb (=)( S Σ ().((4(())))))) =0
and this concludes the proof.
As a result of Lemma K.I the following Lemma is essentially built upon three assumptions/facts: (I) As boundary set, 0g is closed set in Rd. (II) As (d 1)-dimensional manifold, ag is of class
C² (III) The measures Sy in Assumption I.6 are absolutely continuous w.r.t. the spherical measure O.
Lemma K.2. Under Assumptions I.]
1.3 and 1.6
it holds for (Lebesgue) almost every > 0 that
r h (09))
Proof. Fix some b satisfying the conditions in Lemma K.1 Recall that μ Ejes(s)) Mi (see
eq. J.40)). It suffices to show that μz [ (1-1(ag) = for some fixed E j(()). In particular,
148

---

Published as a conference paper at ICLR 2022
observe that
My h (8G)
1111200 PLE8A-1, r200 vie(
1h (ri,". .Kk -1, ,0..,,00--11 -1,2,- (tx) + po 0p.) E og}
U2. 18d-1,n 20
Sin. (d0 Vasy (drx)))
II
Pag, (dri) x (d.() x mm.oldlett)
(K.10)
where the function he defined as h" (11,
10.0..... -1,2,
0: (0, t2. ... -1), (r101, r202. - 110--10--1))
Let A h:" (ri, -1, 01,
On- - 42, /2, 4x Now by separating the two cases based
whether the truncation operator takes effect or not, we have
1/0 c8d-1 I Six. (d0 Mane (drx-)
1 (r1,-.. .T -1, 11,21 tx.) + (po(rr 0g}
1(2 + rk- Ok- { (d0k- (c0k=)aas,, (drk-)
(K.11)
Ba cgd-1 m2.
1f= + 60p. E 00) Sp. (d0k-)
Srb Masa (drk.).
(K.12)
12 E88-1
K.111. note that is equal to f I Ov.) E (=2+09)n0B00.)))53 (d0x. x Vase- (dri.)) where T (r, 0) r0 the inverse of the polar coordinate transform. Furthermore. since the set (-2+ ag) NBB( (, b) is either empty or is (d -idimnnnsinnal C submanifold (w.r.t B(0, b) when viewed as d dimensional manifold). In other words, it has zero mass under m For the measure v" - T (Sj.- Vass. ) due to Sj being absolutely continuous w.r.t. ar it is easy to see that v² is absolutely continuous w.r.t. m Therefore, we must have
For term eq.
v² v"((22 + ag) n (O, b)) = 0,
implying that the integral in eq. Seg-2 Vosc. (drk.) ) = 1/b
K.11 O. On the other hand, for term eq. 1.1))) we know that
oc. Besides,
J. 1(= b0 E Sg. Iga 0g (d0).
Then follows immediately from Lemma K. measure that the integral in term eq. K.12 is equal to 0. In summary, have shown that
and Sy being absolutely continuous w.r.t the spherical
and
Jo esu 1.r0.00 20
1f (ri, , -1,0, 0 12, ,44) + sp((r=00-)) E ag)
SJ (d0x. Vagg. (drk.) =0
for any (ri.... Tk- 01, ... -1, 1.. .4-). Plug this result back into eq.
K.10 and we
conclude the proof.
L NOTATIONS
Table [.1 lists the notations used in Section G
149

---

Published as a conference paper at ICLR 2022
Table L.1: Summary of notations frequently used in Section G
[k]
{1, .,,
77
Learning rate (gradient descent step size)
b
Truncation threshold of stochastic gradient
An accuracy parameter; typically used denote an neiihboorrooo of 85, my
8
A threshold parameter used to define large noises
a
A constant defined for eq. (G.28) G.29).. Since E < co. in eq. (G.8) the claim holds for Jaz yl E. Note that the value of the constant does not vary with choice of E,
M
Upper bound of [|] and |s"|
eq.
G.10
Radius of training domain
eq. G.10
The open interval (8); simplified notation for 22
8. Pe
(e(w) A p(w, c) V
truncation operator at level
2
"small" noise eq.
G.20
2260
Zm
"large" noise eq.
G.21
(6)
min(n > (6): : 7//2ml > 8
arrival time of j-th large noise eq.
G.22
Zr7(6)
size of j-th large noise eq.
G.23
x2(z)
X (x) = ppL (xx( (((e))))2+++))) Zntz))), x²(x)
SGD
(x)
(2 (=) nf (x%-(())))),, ()(x)
GD
(1)))
y perturbed by large noises (5), ())
GD + large jump perturbed GD
(x
y (r) perturbed by noise vector (t,
(t,2)
dx dx: (t; nf (x) dt, x";
ODE
x))
x² (t,z)
(1, w)
x (1, 2)) perturbed by noise vector (t, w)
perturbed ODE
A(n,n.c,0)
max 17 + Zμ Sej.
eq. G.42
kE (8)
a min - Effective radius of the attraction field Q.
1" A [r/b]. The minimum number of jumps required escape Q when starting from its local minimum m = 0.
h(w,t)
mapping defined as h(w, t) = x(tr- 0:t, w).
T,
Necessary conditions for h(w, t) to be outside of ?
eq.
G.35 eq. G.36
((E)
t(e) S с1 (og))/)). The quantity f(c)/1) provides an upper bound for the time it takes x to return to 2e neighhhbood of local minimum = 0 when starting from somewhere e-away from 8-, 84. See eq. (G.77).
El(e)
{ t) CRR RC h(w, (s_ - v (-L), (++ + +c) LJ}
6(e,6,n)
The probability that, for (T) 1) j=1 and
(o) we have Intuitively speaking, it characterizes the
(w, t) E(c) conditioning on (T = 1}. probability that the first 1" large noises alone can drive the ODE out of the attraction field. Defined in eq. (G.66).
Va
The Borel measure on R with density
apt pa+1 +1(z < <0
op_
Va (dx) 1 f(x 0}
|
where p-, P+ constants in Assumption 2 in the main paper. The product measure (va)" x (Leb.)" -1-1 min(n 2 0 X Ω}.
(1)
first exit time
150

---

Published as a conference paper at ICLR 2022
(xx)
P(|Z] > x) =
Tretum((,, ()
min(n 2 0: (x) E 2 2c}}
151

---

Published as a conference paper at ICLR 2022
M RESULTS ABOUT TAIL DISTRIBUTIONS OF NOISES IN OUR NUMERICAL EXPERIMENTS
M.1 QQ PLOTS
QQ plots below clearly show that the tails in noise distribution are always much lighter than the Pareto distributions with alpha = 2 or even 10. In fact, the tail of noise distributions seem to be between that of lognormal and normal distributions, implying that it lighter than any power-law
distribution.
Pareto, 2
Pareto, 5
Pareto, 10
Lognormal
Normal
Figure M.1: Ablation Study, Corrupted FMNIST & LeNet: At the beginning
Pareto, a 2
Pareto, a = 5
Pareto, 10
Lognormal
Normal
Figure M.2: Ablation Study, Corrupted FMNIST & LeNet: Half way through the training
Pareto, a = 2
Pareto, a = 5
Pareto, 10
Lognormal
Normal
Figure M.3: Ablation Study, Corrupted FMNIST & LeNet: At the end of training
Pareto, a
Pareto, a
Pareto, 10
Lognormal
Normal
Figure M.4: Ablation Study, SVHN VGGII: At the beginning
Pareto, 2
Pareto,
Pareto, 10
Lognormal
Normal
Figure M.5: Ablation Study, SVHN & VGG11: Half way through the training
152

---

Published as a conference paper at ICLR 2022
Pareto, a
Pareto, a
Pareto, a = 10
Lognormal
Normal
Figure M.6: Ablation Study, SVHN VGGII: At the end of training
Pareto, a
Pareto, a
Pareto,
Lognormal
Normal
Figure M.7: Ablation Study, CIFARIO VGG11: At the beginning
Pareto, a = 2
Pareto, a = 5
Pareto, = 10
Lognormal
Normal
Figure M.8: Ablation Study, CIFAR1O & VGG11: Half way through the training
Pareto, 2
Pareto,
Pareto, 10
Lognormal
Normal
Figure M.9: Ablation Study, CIFARIO VGG11: At the end of training
Pareto, a
Pareto, a
Pareto, a = 10
Lognormal
Normal
Figure M. Data Augmentation, CIFARIO & VGG11: At the beginning
Pareto, a = 2
Pareto, a = 5
Pareto, 10
Lognormal
Normal or
Figure M.11: Data Augmentation, CIFARIO & VGG11: Half way through the training
153

---

Published as a conference paper at ICLR 2022
Pareto, a
Pareto, a
Pareto,
Lognormal
Normal
Figure M.12: Data Augmentation, CIFARIO & VGG11: At the end of training
Pareto, a Pareto, a
Pareto, a = 10
Lognormal
Normal
Figure M.13: Data Augmentation, CIFARIO0 & VGGI6: At the beginning
Pareto, 2
Pareto, 5
Pareto, 10
Lognormal
Normal
Figure M. 14 Data Augmentation, CIFAR100 & VGG16: Halfway through the training
Pareto, 2
Pareto,
Pareto, 10
Lognormal
Normal
Figure M.15: Data Augmentation, CIFARI00 & VGG16: At the end of training
154

---

Published as a conference paper at ICLR 2022
M.2 EMPIRICAL MEAN RESIDUAL LIFE (EMRL) PLOTS
well known that the mean residual life blows up to infinity and only if the distribution is heavy-tailed (more precisely, long-tailed). However, from the figures below, one that none of the EMRL exhibits such a pattern in any case tested in our experiments. Instead, we see clear downward trends, which strongly suggests light tails in all cases tested.
Beginning
Middle
End
Figure M.16: Plots of empirical mean residual life for noises in FMNIST&LeNG Task throughout training
Beginning
Middle
End
14
1000
Figure M.17: Plots of empirical mean residual life for noises in SVHN& 11 Task throughout training
Beginning
Middle
End
34-
Figure M.18: Plots of empirical mean residual life for noises in CIFAR 10& VGG 11 Task throughout training
Beginning
Middle
End
Figure M.19: Plots of empirical mean residual life for noises dataAug, CIFAR VGG 11 Task throughout training
155

---

Published as a conference paper at ICLR 2022
Beginning
Middle
End
Figure M.20: Plots of empirical mean residual life for noises dataAug, CIFAR VO0&VGG 11 Task throughout training
M.3 HILL PLOTS
In the hill plots below, the estimated power-law indices using only the top 1% of samples (on the left hand side of the dashed lines) stay well above 10 for the most part and almost never drop below This strongly suggests that even if the gradient noises are from heavy-tailed distribution, it is likely have very high power law index (implying relatively lighter tails), and hence, we cannot expect to observe prominent heavy-tailed behavior from them.
Beginning
Middle
End
Figure M.21: aldHill Plots for noises in FMNIST&LeNet Task throughout training. Dashed Red Line:
Estimation based on the largest 1% data
Beginning
Middle
End
Figure M.22: aldHill Plots for noises in SGG 11 Task throughout training. Dashed Red Line:
Estimation based on the largest 1% data
156

---

Published as a conference paper at ICLR 2022
Beginning
Middle
End
Figure M.23: altHill Plots for noises in CIFAR VGG 11 Task throughout training. Dashed Red
Line: Estimation based on the largest 1% data
Beginning
Middle
End
22
90
Figure M.24: altHill Plots for noises in Data Augmentation, CIFAR 1O& 11 Task throughout training. Dashed Red Line: Estimation based on the largest 1% data
Beginning
Middle
End
25
Figure M.25: altHill Plots for noises in Data Augmentation, CIFAR 100& VGG 16 Task throughout training. Dashed Red Line: Estimation based on the largest 1% data
Table M.1: Power-law Indices Estimation throughout the Training, using PLFIT. All the estimations least 5 for all cases tested in experiments, and most of the times the estimation is above 10. This means that even under the assumption that the gradient noises were from heavy-tailed distribution, they should have much lighter tails than any a-stable distribution (which requires 2)
or the heavy-tailed noises we injected during tail inflation experiments (a 1.4).
Task
Beginning
Middle
End
FMNIST, LeNet
14.3
14.2
16.5
SVHN, VGG11
5.0
5.2
12.5
CIFARIO, VGGI1
9.2
6.6
7.0
dataAug. CIFARIO. VGGII
16.2
16.2
8.5
dataAug, CIFAR100, VGG16
35.1
14.4
5.35
157


Reviews:
[{"review_id": "wVi3UbJ1r6", "reviewer": "Reviewer_QXyo", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I believe that the topics of SGD-trained networks' generalization in general, and the role of heavy-tailed parameter statistics in this in specific are very timely and worthy of attention, since our understanding of the dynamics that lead to generalization is not on par with the empirical success of SGD-based methods. \n\nI find the authors' analyses and results interesting and well-presented. I think it has potential to improve our understanding of the generalization characteristics of SGD-trained networks. Although their analyses mostly focus on the univariate case, I believe that this is acceptable as they provide some seminal results regarding truncated-gradient SGD. \n\nHowever, I have a hard time understanding the authors' characterization of recent results in the literature (or lack thereof), which also informs their experiment design. My point can be most dramatically made by drawing attention to the choice of baseline that the authors present. While the recent theoretical and empirical findings in literature emphasize the relationship between learning rate, and tail index and/or generalization, the authors somehow base their methodology and experiments on the supposed absence of heavy tails in gradient noise in image classification tasks. This is not necessarily true, and this fact is well-documented. Given this fact, the fact that the authors present the baselines as thin-tailed noise algorithms (without any detailed analysis of learning rate / batch size and their effects on generalization), as well as not estimating the tail index of these noises is surprising. I think the authors need to take into account more recent results in the literature and possibly alter their experimental settings and discussion accordingly. This is especially important since the authors aim to analyse why a specific modification of SGD leads to improvements.\n\n- Hodgkinson, Liam, and Michael W. Mahoney. 2020. “Multiplicative Noise and Heavy Tails in Stochastic Optimization.” ArXiv:2006.06293 [Cs, Math, Stat], June. http://arxiv.org/abs/2006.06293.\n- Gurbuzbalaban, Mert, Umut Şimşekli, and Lingjiong Zhu. 2021. “The Heavy-Tail Phenomenon in SGD.” ArXiv:2006.04740 [Cs, Math, Stat], June. http://arxiv.org/abs/2006.04740.\n- Lewkowycz, Aitor, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, and Guy Gur-Ari. 2020. “The Large Learning Rate Phase of Deep Learning: The Catapult Mechanism,” March. https://arxiv.org/abs/2003.02218v1.", "summary_of_the_paper": "The authors analyse the behavior of SGD under gradient clipping. Their analysis in the univariate case shows that gradient clipping in the heavy-tailed gradient noise (almost) eliminates the algorithm's tendency to stay at sharp minima. The authors support their analysis with synthetic experiments. The authors then conduct experiments on real data where they add heavy tailed noise to the gradient, and clip it afterwards.", "summary_of_the_review": "The authors present interesting analyses and results regarding a modified version of SGD in optimization. Their characterization of the recent literature, and the experimental design based thereupon seems to need more attention.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "gVkpAXBWCL", "reviewer": "Reviewer_HGyL", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I appreciate the mathematical rigor of the work, but I am not convinced by its machine learning / deep learning relevance.\n\nSpecifically, I find the following points problematic.\n1. The title seems inappropriate. The word SGD is taken to mean a special kind of noise that is due to minibatch sampling. This work, however, only studies GD with injected power-law noise. I think it is misleading to say \"SGD\" in the title\n\n2. The assumptions seem too strong and deep learning irrelevant. One of my main objections is that the paper assumes finitely many minima, yet, neural networks, both underparametrized and overparametrized, should have infinitely many minima, and the fact that the minima of neural networks are degenerate makes it inappropriate to apply a transition graph analysis. This then makes me think that the theoretical analysis is not relevant for deep learning\n\n3. The main result in Theorem 2 only applies to the case when the learning rate is infinitesimal (and the limit needs to be taken under certain scaling conditions). This amounts to a continuous-time approximation, and I think is a crucial limitation of the theoretical results. The theorems reveal nothing about the behavior of SGD at a finite learning rate, which is the actual regime that SGD is run in practice. \n- Also, I feel that this continuous-time condition should be stated much earlier in the draft\n\n4. I am unconvinced by the experiment section. Both LeNet and VGG are outdated architectures because they lack the residual structure. I would ask for evaluation on at least a modern ResNet to demonstrate the effectiveness of the proposed method\n- Even if evaluated on ResNet, I still think it would not suffice because it is quite easy to improve a vanilla model; I would really want to see being able to improve some state-of-the-art results for the paper to be experimentally convincing\n\nOther important questions (but may not constitute reasons for rejection):\n1. It has been found that the power-law index of SGD noise crucially depends on the learning rate of SGD (see https://arxiv.org/abs/2106.02588, or https://arxiv.org/abs/2105.09557). How does this fact affect the theoretical results of this work?\n\n\nMinor question:\n1. page 3: what is \"regeneration structure\"?", "summary_of_the_paper": "The paper studies gradient descent with injected power-law tail noise. The work shows that in the infinitesimal learning rate regime, the heavy tail noise can cause GD to not to converge to sharp minima. \n\nBased on this theory, the paper proposes a technique to inject noise to GD to help training\n", "summary_of_the_review": "The following two reasons are the main weaknesses, based on which I recommend rejection:\n1. The theory seems irrelevant for machine learning / deep learning\n2. The experimental evaluation is weak because it uses outdated architecture and because the result only improves on badly performing vanilla training strategies\n\nTherefore, taken as a theoretical paper, I find the theory limited and irrelevant. Taken as an experimental/method paper, I find the improvement and methodology unconvincing. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "OcVE4ljYSVb", "reviewer": "Reviewer_kP7Y", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Pros\n\n- A beautiful theoretical analysis is provided for a one-dimensional landscape under some structure assumptions. \n- Very insightful synthetical experiments are provided to justify the intuitions and theory.\n- Inspired by the above analysis, the authors proposed two variants of SGD with injected heavy-tailed noise + gradient clipping. These modified SGDs are expected to converge to flatter minima, thereby generalizing better. \n- The experiments for deep nets on (Fashion)MNIST and CIFAR-10/CIFAR-100 are sufficient and show promising improvements over vanilla SGD.\n\n### Cons\n\nIt seems that all the analyses ignore the noise structure and are only concerned with the magnitude. For instance, for even a one-dimensional problem, the gradient noise of vanilla mini-batch SGD is state-dependent. However, gradient noises in the prototypical dynamics analyzed in this paper (see Eq. (2)) are iid random variables. As another example, Figure 1 can be misleading since the light-tailed SGDs only use state-independent noise. It is possible that SGD with structured noise can avoid those sharp minima (completely). The authors should disentangle the effect of noise magnitudes and noise structures, and state clearly what is concerned in this paper. \n\n\n\nFor the related work, previous work studying the stability-driven escaping from sharp minima is completely ignored. I would suggest comparing with them (see [1,2,3] and the reference therein). For example, the stability-driven escaping can tell us that, in Figure 1, GD with a relatively large learning rate never converge to the sharp minima: m1, m2.  \n\n\n### Other comments\n\n- In Figure 1, please make it clear in the caption that where the SGD starts from. \n\n- In the second paragraph of page 2, I am not sure why SGD with light-tailed noise never escape the sharp minima m3. I suppose that by adding large enough noise, SGD at least can escape from it, although it is as efficient as the heavy-tailed SGD. \n\n- \" if we stop training SGD at an arbitrary time point, it is almost guaranteed that it won’t be at a sharp minimum\". This claim seems wrong to me. Although heavy-tailed SGD with gradient clipping can avoid the sharp minima completely, it may take a very long time. \n\n- In Section 4, \"Contrary to the report in (S ̧ims ̧ekli et al., 2019a), heavy-tailed noise may not be ubiquitous in image classification tasks.\". Why is there this contradiction? In particular, Table 1 suggests that SGD+heavy-tailed noise performs very badly, even worse than large-batch SGD. Can you explain it? This also contradicts the synthetical experiments, where the heavy-tailed SGD converges to flat minima more likely than light-tailed SGD.\n\n- In paragraph above Section 5, \"event\"-> \"even\".\n\n  \n\n[1] Wu, Lei, et al.,  How sgd selects the global minima in over-parameterized learning: A dynamical stability perspective.\n\n[2] Jastrzebski et al. The Break-Even Point on Optimization Trajectories of Deep Neural Networks.\n\n[3] Cohen, et al. Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability", "summary_of_the_paper": "This paper study the long-time behavior of heavy-tailed SGD with gradient clipping. It is found that gradient clipping is crucial for heavy-tailed SGD to avoid sharp minima. The basic intuition is that the clipping operation reduces the distance moved by each SGD update. Therefore, for minima narrow than the threshold, the clipping does not change the first exit time. However, for wide minima, SGD is slowed down and takes more time to escape. Consequently, it is more likely that SGD locates in wide minima. \n\n", "summary_of_the_review": "This paper reveals that heavy-tailed noise+gradient clipping can help SGD eliminate the sharp minima. Beautiful theoretical analysis and insightful numerical experiments are provided. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Motivated by empirical observations that SGD performed on deep networks converge to regions of flatter loss curvature relative to large or full batch GD, the authors perform a theoretical analysis of trajectories of SGD with the presence of heavy tailed noise. The primary observation of the theory is that heavy tailed noise has a higher probability of "kicking" the current parameters to a new region of the input space, which has some probability of lying in a sharper region. However, it's important to note that in this analysis SGD with heavy tailed noise doesn't stay in the sharp regions, but will eventually be kicked back out of it back to other regions. In a sense, this defines a transition graph which predicts that the steady state distribution should spend some fraction of time in different regions of the input space (and different sharpness) while never "converging" anywhere. This is shown most clearly in Figure 1 top center where the heavy tailed SGD randomly jumps between different regions of the input space throughout the entire training trajectory. Experiments are then run on deep networks showing that heavy-tailed SGD with gradient clipping converges to regions of flatter curvature. 

Reviews of the work were generally positive, the theory is well presented and Figure 1 does a solid job demonstrating the main idea. The primary criticism was raised by reviewer HGyL, arguing that the results should be largely irrelevant to deep learning. Most of the debate between this reviewer and the authors centered around whether or not ReLU networks have minima which extend off to infinity. The AC will not dig into the details of the argument. It seems clear, however, that if there were a deep learning workload with heavy tailed noise that the authors results will have some relevancy, though the exact nature of the resulting transition graph may have a complicated dependence on the loss surface. Unfortunately the authors were unable to find a such a workload in image classification (there is some prior work suggesting the NLP models with rare tokens may be a better fit) and so needed to artificially induce heavy tailed noise to test their theory. This is a bit of a limitation, but given the clear writing and interesting experiments as noted by reviewers the work seems worth accepting. The AC strongly urges the authors though to include a more lengthy discussed of Wu. et. al. as that work seems to agree with experiment of the sharpness of stable regions selected by SGD when run on deep models without heavy tailed noise.
Decision: Accept (Poster)