Year: 2021

Abstract:
The empirical success of deep learning is often attributed to SGD’s mysterious ability to avoid sharp local minima in the loss landscape, as sharp minima are known to lead to poor generalization. Recently, evidence of heavy-tailed gradient noise was reported in many deep learning tasks, and it was shown in (Şimşekli et al., 2019a;b) that SGD can escape sharp local minima under the presence of such heavy-tailed gradient noise, providing a partial solution to the mystery. In this work, we analyze a popular variant of SGD where gradients are truncated above a fixed threshold. We show that it achieves a stronger notion of avoiding sharp minima: it can effectively eliminate sharp local minima entirely from its training trajectory. We characterize the dynamics of truncated SGD driven by heavy-tailed noises. First, we show that the truncation threshold and width of the attraction field dictate the order of the first exit time from the associated local minimum. Moreover, when the objective function satisfies appropriate structural conditions, we prove that as the learning rate decreases the dynamics of the heavy-tailed truncated SGD closely resemble those of a continuous-time Markov chain that never visits any sharp minima. Real data experiments on deep learning confirm our theoretical prediction that heavy-tailed SGD with gradient clipping finds a "flatter" local minima and achieves better generalization.

heading: 1 INTRODUCTION
text: Stochastic gradient descent (SGD) and its variants have seen unprecedented empirical successes in training deep neural networks. The training of deep neural networks is typically posed as a nonconvex optimization problem, and even without explicit regularization the solutions obtained by SGD often perform surprisingly well on test data. Such an unexpected generalization performance of SGD in deep neural networks are often attributed to SGD’s ability to avoid sharp local minima1 in the loss landscape, which tends to lead to poor generalization (Hochreiter & Schmidhuber, 1997; Keskar et al., 2016; Li et al., 2018b; Jiang et al., 2019); see Appendix D for more details. Despite significant efforts to explain such phenomena theoretically, understanding how SGD manages to avoid sharp local minima and end up with flat local minima within a realistic training time still remains as a central mystery of deep learning. Recently, the heavy-tailed dynamics of SGD received significant attention, and it was suggested that the heavy tails in the stochastic gradients may be a key ingredient that facilitates SGD’s escape from sharp local minima: for example, Şimşekli et al. (2019a;b) report the empirical evidence of heavy-tails in stochastic gradient noise in popular deep learning architectures (see also (Hodgkinson & Mahoney, 2020; Srinivasan et al., 2021; Garg et al., 2021)) and show that SGD can escape sharp local minima in polynomial time under the presence of the heavy-tailed gradient noise. More specifically, they view heavy-tailed SGDs as discrete approximations of Lévy driven Langevin equations and argue that the amount of time SGD trajectory spends in each local minimum is proportional to the width of the associated minimum according to the metastability theory (Pavlyukevich, 2007; Imkeller et al., 2010a;b) for such heavy-tailed processes.
In this paper, we study the global dynamics and long-run behavior of heavy-tailed SGD and its practical variant in depth. In particular, we consider an adaptive version of SGD, where the stochastic gradient is truncated above a fixed threshold. Such truncation scheme is often called gradient clipping and employed as default in various contexts (Engstrom et al., 2020; Merity et al., 2018; Graves, 2013;
1We use the terminology sharpness in a broad sense; we refer to Appendix C for a detailed discussion.
Pascanu et al., 2013; Zhang et al., 2020; Gorbunov et al., 2020). We uncover a rich mathematical structure in the global dynamics of SGD under this scheme and prove that the long-run behavior of such SGD is fundamentally different from that of the pure form of SGD: in particular, under a suitable structural condition on the geometry of the loss landscape, gradient clipping completely eliminates sharp minima from the trajectory of SGDs. This provides a critical insight into how heavy-tailed dynamics of SGD can be utilized to find a local minimum that generalizes better.
Figure 1 (Left, Middle) clearly illustrates these points with the histograms of the sample trajectories of SGDs. Note first that SGDs with light-tailed gradient noise—(c) and (d) of Figure 1 (Left, Middle)— never manages to escape a (sharp) minimum regardless of gradient clipping. In contrast, SGDs with heavy-tailed gradient noise—(a) and (b) of Figure 1 (Left, Middle)—easily escapes from local minima. Moreover, there is a clear difference between SGDs with gradient clipping and without gradient clipping. In (a) of Figure 1 (Left), SGD without gradient clipping spends a significant amount of time at each of all four local minima ({m1,m2,m3,m4}), although it spends more time around the wide ones ({m2,m4}) than the sharp ones ({m1,m3}). On the other hand, in (b) of Figure 1 (Left), SGD with gradient clipping not only escapes from local minima but also avoids sharp minima ({m1,m3}) almost completely. This means that if we stop training SGD at an arbitrary time point, it is almost guaranteed that it won’t be at a sharp minimum, effectively eliminating sharp minima from its training trajectories.
We also propose a novel computational strategy that takes advantage of our newly discovered global dynamics of the heavy-tailed SGD. While the evidence of heavy tails were reported in many deep learning tasks (Şimşekli et al., 2019b;a; Garg et al., 2021; Gurbuzbalaban et al., 2020; Hodgkinson & Mahoney, 2020; Nguyen et al., 2019; Mahoney & Martin, 2019; Srinivasan et al., 2021; Zhang et al., 2020), there seem to be plenty of deep learning contexts where the stochastic gradient noises are light-tailed (Panigrahi et al., 2019) as well. Guided by our new theory, we propose an algorithm that injects heavy-tails to SGD by inflating the tail distribution of the gradient noise and facilitating the discovery of a local minimum that generalizes better. Our experiments with image classification tasks, reported in Tables 1 and 2, illustrate that the tail-inflation strategy we propose here can indeed improve the generalization performance of the SGD as predicted by our theory.
The rest of the paper is organized as follows. Section 2 formulates the problem setting and characterizes the global dynamics of the SGD driven by heavy-tailed noises. Section 3 presents numerical experiments that confirm our theory. Section 4 proposes a new algorithm that artificially injects heavy tailed gradient noise in actual deep learning tasks and demonstrate the improved performance.
Technical Contributions: 1) We rigorously characterize the global behavior of the heavy-tailed SGD with gradient clipping. We first focus on the case where the loss function is in R1 with some simplifying assumptions on its geometry. Even with such assumptions, our theorem involves substantial technical challenges since the traditional tools for analyzing SGD fail in our context due to the adaptive nature of its dynamics and non-Gaussian distributional assumptions. For example, while the unclipped pure SGD can be analyzed by partitioning its trajectory at arrival times of large noises (as in Pavlyukevich (2005) and Imkeller et al. (2010a)), such an approach falls short in our context. Instead, we developed a set of delicate arguments for dealing with SGD’s (near) regeneration structure and the return times to the local minima, as well as controlling the probability of atypical scenarios that would not arise in the unclipped case. Moreover, as evidenced by our Rd results in Appendix G, the approach developed here is critical in extending the analysis to general loss landscapes.
2) We propose a novel computational strategy for improving the generalization performance of SGD by carefully injecting heavy-tailed noise. We test the proposed algorithm with deep learning tasks and demonstrate its superiority with an ablation study. This also suggests that the key phenomenon we characterize in our theory— elimination of sharp local minima—manifests in real-world tasks.

heading: 2 THEORETICAL RESULTS
text: This section characterizes the global dynamics of SGD with gradient clipping when applied to a non-convex objective function f . In Section 2.1 and 2.2, we make the following assumptions for the sake of the simplicity of analysis. However, as our multidimensional result in Section 2.3 and the experiments in Section 3 and 4 suggest, the gist of the phenomena we analyze—elimination of sharp local minima—persists in general contexts where the domain of f is multi-dimensional, and the stationary points are not necessarily strict local optima separated from one another. Assumption 1. Let f : R → R be a C2 function. There exist a positive real L > 0, a positive integer nmin and an ordered sequence of real numbers m1, s1,m2, s2, · · · , snmin−1,mnmin such that (1) −L < m1 < s1 < m2 < s2 < · · · < snmin−1 < mnmin < L; (2) f ′(x) = 0 iff x ∈ {m1, s1, · · · , snmin−1,mnmin}; (3) For any x ∈ {m1,m2, · · · ,mnmin}, f ′′(x) > 0; (4) For any x ∈ {s1, s2, · · · , snmin−1}, f ′′(x) < 0.
As illustrated in Figure 2 (Left), the assumption above requires that f has finitely many local minima (to be specific, the count is nmin), all of which contained in some compact domain [−L,L]. Moreover, the points s1, · · · , snmin−1 naturally partition the entire real line into different regions Ωi = (si−1, si) (here we adopt the convention that s0 = −∞, snmin = +∞). We call each region Ωi the attraction field of the local minimum mi, as the gradient flow in Ωi always points to mi.
Throughout the optimization procedure, given any location x ∈ R we assume that we have access to the noisy estimator f ′(x)− Zn of the true gradient f ′(x), and f ′(x) itself is difficult to evaluate. Specifically, in this work we are interested in the case where the iid sequence of noises (Zn)n≥1 are heavy-tailed. Typically, the heavy-tailed phenomena are captured by the concept of regular variation: for a measurable function φ : R+ 7→ R+, we say that φ is regularly varying at +∞ with index β (denoted as φ ∈ RVβ) if limx→∞ φ(tx)/φ(x) = tβ for all t > 0. For details on the definition and properties of regularly varying functions, see, for example, chapter 2 of Resnick (2007). In this paper, we work with the following distributional assumption on the gradient noise. Let
H+(x) , P(Z1 > x), H−(x) , P(Z1 < −x), H(x) , H+(x) +H−(x) = P(|Z1| > x).
Assumption 2. EZ1 = 0. Furthermore, there exists some α ∈ (1,∞) such that function H(x) is regularly varying (at +∞) with index −α. Besides, regarding the positive and negative tail for distribution of the noises, we have
lim x→∞
H+(x)
H(x) = p+, lim x→∞
H−(x)
H(x) = p− = 1− p+
where p+ and p− are constants in interval (0, 1).
Roughly speaking, Assumption 2 means that the shape of the tail for the distribution of noises Zn resembles a polynomial function x−α, which is much heavier than the exponential tail of Gaussian distributions. Therefore, large values of Zn are much more likely to be observed under Assumption 2 compared to the typical Gaussian assumption. The index α of regular variation encodes the heaviness of the tail—the smaller the heavier—and we are assuming that the left and right tails share the same index α. The purpose of this simplifying assumption is clarity of presentation, but our Rd results in Appendix G relax such a condition and allow different regular variation indices in different directions.
Our work concerns a popular variant of SGD where the stochastic gradient is truncated. Specifically, when updating the SGD iterates with a learning rate η > 0, rather than using the original noisy gradient descent step η(f ′(Xn)− Zn), we will truncate it at a threshold b > 0 and use ϕb ( η(f ′(Xn)− Zn) ) instead. Here the truncation operator ϕ·(·) is defined as
ϕc(w) , w ·min{1, c/|w|} ∀w ∈ R, c > 0. (1)
Besides truncating the stochastic gradient, we also project the SGD into [−L,L] at each iteration; recall that L is the constant in Assumption 1. That is, the main object of our study is the stochastic process {Xηj }j≥0 driven by the following recursion
Xηj = ∆ ϕL ( Xηj−1 − ϕb ( η(f ′(Xηj−1)− Zj) )) . (2)
The projection ϕL and truncation ϕb here are common practices in many learning tasks for the purpose of ensuring that the SGD does not explode and drift to infinity. Besides, the projection also allows us to drop the sophisticated assumptions on the tail behaviors of f that are commonly seen in previous works (see, for instance, the dissipativity conditions in Nguyen et al. (2019)). For technical reasons, we make the following assumption about the truncation threshold b > 0. Note that this assumption is a very mild one, as it is obviously satisfied by (Lebesgue) almost every b > 0. Assumption 3. For each i = 1, 2, · · · , nmin, min{|si −mi|, |si−1 −mi|}/b is not an integer.

heading: 2.1 FIRST EXIT TIMES
text: Denote the SGD’s first exit time from the attraction field Ωi with σi(η) = ∆ min{n ≥ 0 : Xηn /∈ Ωi}. In this section, we prove that σi(η, x) converges to an exponential distribution when scaled properly. To characterize such a scaling, we first introduce a few concepts. For each attraction field Ωi, define (note that dxe = min{n ∈ Z : n ≥ x}, bxc = max{n ∈ Z : n ≤ x} )
ri = ∆ min{|mi − si−1|, |si −mi|}, l∗i = ∆ dri/be. (3)
Note that l∗i ’s in fact depend on the the value of gradient clipping threshold b even though this dependency is not highlighted by the notation. Here ri can be interpreted as the radius or the effective width of the attraction field, and l∗i is the minimum number of jumps required to escape Ωi when starting from mi. Indeed, the gradient clipping threshold b dictates that no single SGD step can travel more than b, and to exit Ωi when starting from mi we can see that at least dri/be steps are required. We can interpret l∗i as the minimum effort required to exit Ωi. In this sense, l ∗ i is an indicator of the width of the attraction field Ωi. Theorem 1 states that l∗i dictates the order of magnitude of the first exit time as well as where the iterates Xηn land on at the first exit time. For each Ωi, define a scaling function λi(η) = ∆ H(1/η) ((1/η)H(1/η)) l∗i−1 . To stress the initial condition, we write Px for the probability law when conditioning on Xη0 = x, or simply write X η n(x). Theorem 1. Under Assumptions 1-3, there exist constants qi > 0 ∀i ∈ {1, 2, · · · , nmin} and qi,j ≥ 0 ∀j ∈ {1, 2, · · · , nmin} \ {i} such that
(i) Suppose that x ∈ Ωk for some k ∈ {1, 2, · · · , nmin}. Under Px, the scaled first exit time qkλk(η)σk(η) converges in distribution to Exp(1) as η ↓ 0.
(ii) For k, l ∈ {1, 2, · · · , nmin} such that k 6= l, we have limη→0 Px(Xησk(η) ∈ Ωl) = qk,l/qk.
The proof and discussion are deferred to Appendix E. The constants qi, qi,j are explicitly identified in terms of the gradient flows perturbed by Pareto jumps in Section C of Appendix. We note here that Theorem 1 implies (i) for Xηn to escape the current attraction field, say Ωi, it takes O ( 1/λi(η) ) time, and (ii) the destination is most likely to be reachable within l∗i jumps from mi.

heading: 2.2 ELIMINATION OF SMALL ATTRACTION FIELDS
text: Under proper structural assumptions on the geometry of f , the sharp minima of f can be effectively eliminated from the trajectory of heavy-tailed SGD, facilitating the discovery of flat minima. This is somewhat surprising given that gradient clipping mechanism makes the SGD iterates move slower. The intuition behind this is that for narrow basins, applying gradient clipping has virtually no effect on the order of exit time; whereas for a wide basin that requires multiple jumps to escape under the clipping scheme, the clipping of gradients significantly slows down the escape and makes SGD stay longer. In other words, gradient clipping only makes SGDs stay longer in the wider (better) basins.
Now, we introduce a few new concepts. Similar to the the minimum number of jumps l∗i defined in (3), we define the following as the minimum number of jumps to reach Ωj from mi for any j 6= i:
li,j = { d(sj−1 −mi)/be if j > i, d(mi − sj)/be if j < i.
(4)
Recall that Theorem 1 dictates that Xηn is most likely to move out of the current attraction field, say Ωi, to somewhere else after O ( 1/λi(η) ) time steps, and the destination is most likely to be reachable within l∗i jumps from mi. Therefore, the transitions from Ωi to Ωj can be considered typical if Ωj can be reached from mi with l∗i jumps—that is, li,j = l ∗ i . Now we define the following directed graph that only includes these typical transitions.
Definition 1 (Typical Transition Graph). Given a function f satisfying Assumption 1 and gradient clipping threshold b > 0 satisfying Assumption 3, a directed graph G = (V,E) is the corresponding typical transition graph if (1) V = {m1, · · · ,mnmin}; (2) An edge (mi → mj) is in E iff li,j = l∗i .
Naturally, the typical transition graph G can be decomposed into different communication classes G1, · · · , GK that are mutually exclusive by considering the equivalence relation associated with the existence of the (two-way) paths between i and j. More specifically, for i 6= j, we say that i and j communicate if and only if there exists a path (mi,mk1 , · · · ,mkn ,mj) as well as a path (mj ,mk′1 , · · · ,mk′n′ ,mi) in G; in other words, by travelling through edges on G, mi can be reached from mj and mj can be reached from mi.
We say that a communication class G is absorbing if there does not exist any edge (mi → mj) ∈ E such that mi ∈ G and mj /∈ G. Otherwise, we say that G is transient. In the case that all mi’s communicate with each other on graph G, we say G is irreducible. See Figure 2 (Middle) for the illustration of an irreducible case. When G is irreducible, we define the set of largest attraction fields M large =∆ {mi : i = 1, 2, · · · , nmin, l∗i = llarge} where llarge = maxj l∗j ; recall that l∗i characterizes the width of Ωi. Define the longest time scale λlarge(η) = H(1/η)((H(1/η)/η))l
large−1. Note that this corresponds exactly to the order of the first exit time of the largest attraction fields; see Theorem 1. The following theorem is the main result of this paper.
Theorem 2. Let Assumptions 1-3 hold and assume that the graph G is irreducible. For any t > 0, β > 1 + (α− 1)llarge and x ∈ [−L,L],
1
bt/ηβc ∫ bt/ηβc 0 1 { Xηbuc(x) ∈ ⋃ j:mj /∈M large Ωj } du→ 0 (5)
in probability as η → 0.
The proof is deferred to Appendix F. Here we briefly discuss the implication of the result. Suppose that we terminate the training after a reasonably long time, say, bt/ηβc iterations. Then the random variable that converges to zero in eq. (5) is exactly the proportion of time that Xηn spent in the attraction fields that are not wide. Therefore, by truncating the gradient noise of the heavy-tailed SGD, we can effectively eliminate small attraction fields from its training trajectory. In other words, it is almost guaranteed that SGD is in one of the widest attraction fields after sufficiently long training.
Theorem 2 is merely a manifestation of the global dynamics of heavy-tailed SGD. The main messages of the next theorem are: (a) under clipped heavy-tailed noises, the dynamics of Xηn for small η closely resemble that of a continuous-time Markov chain; (b) this chain only visits local minima of the largest attraction fields of f , thus minima in small attraction fields are completely avoided. Theorem 3. Let x ∈ Ωi for some i = 1, 2, · · · , nmin. If Assumptions 1-3 hold and G is irreducible, then there exist a continuous-time Markov chain Y on M large and a random mapping π satisfying
• π(m) ≡ m if m ∈M large;
• π(m) is a random variable that only takes value in M large if m /∈M large.
such that the scaled process {Xηbt/λlarge(η)c(x) : t ≥ 0} converges to process {Yt(π(mi)) : t ≥ 0} in the sense of finite-dimensional distributions: for any positive integer k and any 0 < t1 < t2 < · · · < tk, the random vector ( Xηbt1/λlarge(η)c(x), · · · , X η btk/λlarge(η)c(x) ) converges in distribution to(
Yt1(π(mi)), · · · , Ytk(π(mi)) ) as η ↓ 0.
In section D of Appendix, we detail the proof, the exact parametrization of the generator matrix of process Y , and the distribution of random mapping π(·). Here we add some remarks. Intuitively speaking, this result tells us that, regardless of where we initialize the SGD iterates, the dynamics of the clipped heavy-tailed SGD converge to a continuous-time Markov chain avoiding any local minima that is not in the largest attraction fields. Second, under small learning rate η > 0, if Xηn(x) is initialized at x ∈ Ωi where Ωi is NOT a largest attraction field, then SGD will quickly escape Ωi and arrive at some Ωj that is indeed a largest one—i.e., mj ∈M large; such a transition is so quick that, under time scaling λlarge(η), it is almost instantaneous as if Xηn(x) is actually initialized randomly at some of the largest attraction fields. This randomness is compressed in the random mapping π. In Section B, we discuss how our characterization in Theorem 3 are more general and applicable in the machine learning context than metastability results cited in (Şimşekli et al., 2019b); in Section F we see that the regularization effect of truncated heavy-tailed noises described in Theorem 3 is of great generality and can still be observed locally when the irreducibility condition is removed.
2.3 Rd EXTENSIONS OF THE THEORETICAL RESULTS
We focused on the R1 case so far, for the clarity of the exposition. In this section, we informally reiterate that the same effect under truncated heavy-tailed noises persists in high-dimensions. Rigorous statements are provided in Appendix G. We consider a setting similar to those in Imkeller et al. (2010a) and analyze the first exit time σ(η, x) = min{k ≥ 0 : Xηk (x) /∈ G} from an open, bounded domain G with smooth boundary. For some f : Rd 7→ R, the SGD iterates Xηk (x) = Xηk−1(x)− ϕb ( ∇f(Xηk−1(x)) + ηZk ) are subject to the standard L2 norm clipping with threshold b > 0, iid noises Zn with heavy tails that resemble 1/xα with α > 1, and initial conditionX η 0 (x) = x. Assume that the origin 0 is the only attractor in G for the ODE ẋ(t) = −∇f(x(t)). Also, let l∗G be the minimum number of jumps required for the ODE x(t) to escape from G provided that x(0) = 0 and the L2 norm of all jumps are less than b. The following informal version of Theorem G.2 states that, under the presence of heavy-tailed noises and gradient clipping, the first exit time from a region
in Rd is of order O(1/η1+(α−1)l∗G ). Therefore, the order of first exit times from different regions in Rd are still dictated by the geometric characterization l∗G , i.e. the minimum number of jumps required for escape, and those with largest l∗G may dominate the SGD trajectory as η ↓ 0. Furthermore, we extend the analysis to the generalized case where the distribution of noises Zn exhibits strong preference of certain directions and has different heavy-tailed indices α along different directions. We give the details of the Rd results in Section G and provide a proof in Section H. Theorem 4 (Informal). Under certain regularity conditions, for Lebesgue almost every b > 0, there exist q > 0 and λ(η) that is regularly varying w.r.t η with index 1 + (α− 1)l∗G such that
λ(η)σ(η, x)⇒ Exp(q) as η ↓ 0 ∀x ∈ G.

heading: 3 SIMULATION EXPERIMENTS
text: We empirically demonstrate that, (a) as indicated by Theorem 1, the minimum jump number defined in (3) accurately characterizes the first exit times of the SGDs with clipped heavy-tailed gradient noises; (b) sharp minima can be effectively eliminated from such SGD; and (c) these properties are exclusive to heavy-tails. Under light-tailed noises, SGDs are trapped in sharp minima for a long time. The test function f ∈ C2(R) is the same as in Fig. 1 (Left,e). m1 and m3 are sharp minima in narrow attraction fields, while m2 and m4 are flatter and located in larger attraction fields. Heavy-tailed noises have tail index α = 1.2, and light-tailed noises are N (0, 1). See Appendix A for details. First, we compare the first exit time of heavy-tailed SGD (when initialized at -0.7) from Ω2 = (−1.3, 0.2) under 3 different clipping mechanism: (1) b = 0.28, where the minimum jump number required to escape is l∗ = 3; (2) b = 0.5, where l∗ = 2; (3) no gradient clipping, where l∗ = 1 obviously. According to Theorem 1, the first exit times for the aforementioned 3 clipping mechanism are of order (1/η)1.6, (1/η)1.4 and (1/η)1.2 respectively. These theoretical predictions are accurate as demonstrated in Figure 1 (Right). Next, we investigate the global dynamics of heavy-tailed SGD. We compared the clipped case (with b = 0.5) against the case without clipping. Figure 1 (Left, a, b) show the histograms of the empirical distributions of SGD, and Figure 1(Middle, a,b) plots the SGD trajectories. Without gradient clipping, Xn still visits the two sharp minima m1,m3. Under gradient clipping, the time spent at m1,m3 is almost completely eliminated and is negligible compared to the time Xn spent at m2,m4 in larger attraction fields. This matches the predictions of Theorems 2-3: the elimination of sharp minima with truncated heavy-tailed noises. We stress that the said properties are exclusive to heavy-tailed SGD. As shown in Figure 1(Left,c,d) and Figure 1(Middle, c,d), light-tailed SGD are easily trapped at sharp minima for extremely long time.
Figure 3 illustrates the same phenomena in R2, where f has several saddle points and infinitely many local minima—the local minima of Ω2 form a line segment, which is an uncountably infinite set. Under clipping threshold b, attraction fields Ω1 and Ω2 are the larger ones since the escape from them requires at least two jumps. This suggests that the theoretical results from Section 2 hold under more general contexts than Assumptions 1-3. In the next section, we provide experimental evidence that suggests that truncated heavy-tailed noise improves the generalization of SGD in deep learning.

heading: 4 HEAVY-TAILED SGD IN DEEP LEARNING: AN ABLATION STUDY
text: In this section, we verify our theoretical results and demonstrate the effectiveness of clipped heavytailed noise in training deep neural networks. Contrary to the report in (Şimşekli et al., 2019a), heavytailed noise may not be ubiquitous in image classification tasks. For instance, the non-Gaussianity assumption on SGD noise is disputed by experiments in (Panigrahi et al., 2019) for ResNet (see (He et al., 2016)). For tasks considered in this section, the gradient noise is not heavy-tailed when models are randomly initialized (see Appendix A). Motivated by the absence of heavy-tailed noises in image classification, we make the SGD noise heavy-tailed. Let θ be the current model weight during training, gSB(θ) be the typical small-batch gradient, and gGD(θ) be the true (deterministic) gradient evaluated on the entire training dataset. Then by evaluating gSB(θ) − gGD(θ) we obtain a sample of the gradient noise. Due to the prohibitive cost of evaluating gGD(θ), we instead use gSB(θ)− gLB(θ) as its approximation where gLB denotes the gradient evaluated on a larger batch. This is justified by the unbiasedness in ELB [gLB(θ)] = gGD(θ). For some heavy-tailed random variable Z, by multiplying Z with SGD noise, we obtain the following perturbed gradient:
gheavy(θ) = gSB(θ) + Z ( gSB∗(θ)− gLB(θ) ) (6)
where SB and SB∗ are two mini batches that may or may not be identical. We use the following update recursion under gradient clipping threshold b: Xηk+1 = X η k − ϕb(ηgheavy(X η k )) where ϕb is the truncation operator. We consider two different implementations: in our method 1 (labeled as “our 1” in Table 1), SB and SB∗ are chosen independently, while in our method 2 (labeled as “our 2” in Table 1), we use the same batch for SB and SB∗. In summary, by simply multiplying gradient noise with heavy-tailed random variables, we inject heavy-tailed noise into the optimization procedure.
We conduct an ablation study and benchmark the proposed clipped heavy-tailed methods against the following optimization methods. LB: large-batch SGD with Xηk+1 = X η k − ηgLB(X η k ); SB: small-batch SGD with Xηk+1 = X η k − ηgSB(X η k ); SB + Clip: the update recursion is X η k+1 = Xηk − ϕb(ηgSB(X η k )); SB + Noise: Our method 2 WITHOUT the gradient clipping mechanism, leading to the update recursion Xηk+1 = X η k − ηgheavy(X η k ).
The experiment setting and choice of hyperparameters are adapted from (Zhu et al., 2019). We consider three different tasks: (1) LeNet (LeCun et al., 1990) on corrupted FashionMNIST (Xiao et al., 2017), (2) VGG11 (Simonyan & Zisserman, 2014) on SVHN (Netzer et al., 2011), (3) VGG11 on CIFAR10 (Krizhevsky et al., 2009) (see Appendix A for details). Here we highlight a few points: First, within the same task, for all the 6 candidate methods will use the same η, batch size, training iteration, and (when needed) the same clipping threshold b and heavy-tailed RV Z for a fair comparison; the training duration is long enough so that LB and SB have attained 100% training accuracy and close-to-0 training loss long before the end of training (the exception here is “SB + Noise” method; see Appendix A for the details); Second, to facilitate convergence to local minima for our methods 1 and 2, we remove heavy-tailed noise for last final 5,000 iterations and run LB instead2.
Table 1 shows that in all 3 tasks both our method 1 and our method 2 attain better test accuracy than the other candidate methods. Meanwhile, both methods exhibit similar test performance, implying that the implementation of the heavy-tailed method may not be a the deciding factor. We also report the expected sharpness metric Eν∼N (0,δ2I)|L(θ∗ + ν)− L(θ∗)| used in Zhu et al. (2019); Neyshabur et al. (2017b) where N (0, δ2I) is a Gaussian distribution, θ∗ is the trained model weight and L is training loss. In our experiment, we use δ = 0.01 and the expectation is evaluated by averaging over 100 samples. We conduct 5 replications for each experiment scenario and report the averaged
2The proposed method can be interpreted as a simplified version of GD + annealed heavy-tailed perturbation, where a detailed annealing is substituted by a two-phase training schedule. In the first exploration phase the clipped heavy-tailed noises drive the iterates to explore the loss landscape and identify “wide” attraction fields. In the second exploitation phase, removing the artificial perturbation accelerates convergence to local minima.
performance in Table 1. Smaller sharpness of our methods 1 and 2 confirms that they encourage minimizers with a “flatter” geometry, thus attaining better test performances.
The ablation study in Table 1 shows that both heavy-tailed noise and gradient clipping are necessary to find a flat minima and hence achieve better generalization, which is predicted by our analyses. SB and SB + Clip achieve similar inferior performances, confirming that clipping does not help when noise is light-tailed. SB + Noise injects heavy-tailed noise without gradient clipping, which achieves an inferior performance. This poor performance—even after extensive parameter tuning and engineering (see Appendix A for more details)—demonstrates the difficulty on the optimization front, especially when heavy-tailed noise is present yet little effort is put into controlling the highly volatile gradient noises. This is aligned with the observations in Zhang et al. (2020); Gorbunov et al. (2020) where adaptive gradient clipping methods are proposed to improve convergence of SGD in the presence of heavy-tailed noises. This confirms that gradient clipping is crucial for heavy-tailed SGD.
Lastly, Table 2 shows that even in the more sophisticated settings with training techniques such as data augmentations and scheduled learning rates, truncated heavy-tailed SGD still manages to consistently find solutions with better test performance. For experiment details, see Section A in the Appendix and the Table A.3 where we also report the sharpness of the obtained solutions.

heading: 5 CONCLUSION
text: We characterized the global dynamics of SGD with truncated heavy-tailed gradient noise and illustrated our theoretical results with numerical experiments. Our characterizations provide key insights into the global dynamics of SGD and reveal the strong regularization effects of truncated heavy-tailed noises. For the sake of analysis and presentation, we made some simplifying assumptions. However, as indicated by our numerical experiments, it is very likely that the same overall principle—elimination of sharp minima—persists in high-dimensional optimization landscapes that are much more sophisticated than our assumptions. We aim to address this limitation in future works and develop the theories of truncated heavy-tailed SGD at its full level of generality. Also, the conditions—irreducibility and symmetry of graph G—we impose on the geometry of f in Theorem 2 are likely to be stronger than necessary for eliminating sharp minima. In view of these, a natural next step is to investigate how the structures of state-of-the-art deep neural nets lend themselves to the SGD’s global dynamics under general conditions.

heading: A DETAILS OF NUMERICAL EXPERIMENTS

heading: A.1 DETAILS OF THE R1 SIMULATION EXPERIMENT
text: The function f used in the experiments is f(x) = (x+ 1.6)(x+ 1.3)2(x− 0.2)2(x− 0.7)2(x− 1.6) ( 0.05|1.65− x| )0.6 · ( 1 + 1 0.01 + 4(x− 0.5)2 )( 1 + 1 0.1 + 4(x+ 1.5)2 )( 1− 1 4 exp(−5(x+ 0.8)(x+ 0.8)) ) .
(A.1)
As shown in Figure A.1, the four isolated local minimizers of f are m1 = −1.51, s1 = −1.3,m2 = −0.66, s2 = 0.2,m3 = 0.49, s3 = 0.7,m4 = 1.32, and in our experiment we restrict the iterates on [−L,L] with L = 1.6. The heavy-tailed noises we used in the experiment were Zn = 0.1UnWn where Wn were sampled from Pareto Type II distribution (aka Lomax distribution) with shape parameter α = 1.2, and the signs Un were iid RVs such that P(Un = 1) = P(Un = −1) = 1/2. In the first exit time experiment, we tested three different settings: (a) b = 0.28 (so that l∗ = 3); (b) b = 0.5 (so that l∗ = 2); (c) no gradient clipping (so that l∗2 = 1). For the first case, we tested learning rates {0.1, 0.05, 0.03, 0.02, 0.01, 0.005, 0.003, 0.001}, while for the other two cases,
we tested learning rates {0.1, 0.03, 0.01, 0.003, 0.001, 0.0003, 0.0001}. For each case, we ran the simulation 20 times and plotted the average of the 20 exit times. Lastly, to prevent excessively long running time of the experiment, the simulation was terminated when the iteration number reached 5× 107. This threshold was reached only in the setting with η = 0.001, b = 0.28.
Next, we present extra sample paths of SGD when applied to function f in eq. (A.1) in Figure A.2 and A.3. The blue curve on the right side of each plot shows f rotated by 90 degrees, and the dashed lines indicate the locations of local minima. For better readability of the figures, we plotted Xn for every 5,000 iterations. To generate these plots, we initialized the SGD iterates at 0.3 (so that it is in Ω3 = (0.2, 0.7)) and fixed the learning rate as η = 0.001. Again, we tested both with gradient clipping (with b = 0.5) and without gradient clipping. Moreover, we also tested light-tailed noises where we use N(0, 1) as the distribution for noises Zn. For each sample path of Xn, we run 10, 000, 000 iterations. In the left plots of Figure A.2, one can see that with clipped heavy-tailed stochastic gradients, the SGD iterates almost always stay around the wide attraction fields, and the sharp minima are almost completely eliminated from the trajectories of SGD. In comparison, in the right plots of Figure A.2 one can see that without gradient clipping, the heavy-tailed noises will drive
SGD to spend substantial amount of time in all the different local minima, including the sharp ones. Lastly, from Figures A.3, one can see that under light-tailed noises and small learning rates, SGD cannot escape a sharp minima once trapped there.
A.2 DETAILS OF THE Rd SIMULATION EXPERIMENT
As illustrated in the contour plot in Figure 3 (a), the function f in this experiment is a modified version of Himmelblau function, a commonly used test function for optimization algorithm. The modifications serve two purposes. First, as shown in Figure 3 (b), for the modified function the four attraction fields Ω1,Ω2,Ω3,Ω4 have different sizes; in particular, under gradient clipping threshold b = 2.15, from the local minimizers of Ω1 and Ω2 (indicated by red dots in the corresponding area) at least two jumps are required to escape from the attraction field, while from the local minimizer in Ω3 or Ω4 it is possible to escape with one jump. Therefore, for the minimum jump number required to escape, we have l∗1 = l ∗ 2 = 2 > l ∗ 3 = l ∗ 4 = 1 in this case. Second, for the modified test function f , the local minimizer in Ω2 is not a single point but a connected line segment, which is indicated by the dark line in bottom-left region in Figure 3 (a) and the red line segment in in Figure 3 (b). Therefore, the modification allows us to test the heavy-tailed SGD methods on a more general loss landscape.
Now we describe the construction of the test function f . Let h be the Himmelblau function with expression h(x, y) = (x2 + y − 11)2 + (x + y2 − 7)2. Next, define the following transformation for coordinates: φ(x, y) = ( x(exp(c0(x− cx) + 1)), y(exp(c0(x− cx) + 1) ) . Let the composition
be hφ(x, y) = h ( φ(x − ax, y) ) . To create the connected region of local minimizers, define the
following locally “cut” version of hφ:
i(x, y) = 1{x ∈ [bl, br], |y − ay| < by}, h∗(x, y) = (1− i(x, y))hφ(x, y) + i(x, y) min{hφ(x, y), c1|y − ay|1.1}.
In other words, by taking minimum of the original hφ and a polynomial function w.r.t. y around the original local minimizer of Ω2, we obtain a function h∗ that attains local minimum on an entire line segment with y = ay. Lastly, the test function we use in the experiment is f = 0.1h∗, with ax = 1.5, ay = −2.9, bl = −5.5, br = −0.5, by = 2.0, c0 = 0.4, c1 = 12.
In the experiment, we initialize the SGD iterates Xk at X0 = (2.9, 1.0), which is very close to the local minimizer in the small attraction field Ω3. For both the clipped and unclipped SGD, we perform updates for 3× 107 steps, under learning rate 5× 10−4 and heavy-tailed noise Zk = 0.75Wk where the iid samples Wk are isotropic and the law of ‖Wk‖, the size of the noise, is Pareto(1.2). For clipped SGD, we use threshold b = 2.15. To prevent the iterates from drifting to infinity, after each update Xk is projected back to the L2 ball centered at origin with radius 4.2 whenever Xk leaves this ball.
In Figure A.4, we use the 3× 107 steps of SGD iterates (for both the clipped and unclipped case) to create heat maps showing locations of SGD iterates. From this figure, two points can be made clear: first, the heavy-tailed SGD does spend much less time at the two small attraction fields when gradient clipping is applied; second, in Ω2 (the bottom-left attraction field) the SGD iterates frequent the entire connected region of local minima instead of a certain point on this line segment.

heading: A.3 DETAILS OF THE ABLATION STUDY
text: We first mention that the all experiments using neural networks are conducted on Nvidia GeForce GTX 1080 Ti. For the ablation study, the experiments and scripts are adapted from the ones in Zhu et al. (2019).3.
In Figure A.5, we display the gradient noise distribution in the three tasks of the ablation study after the model is randomly initialized.
The experiment setting and choice of hyperparameters are mostly adapted from the experiment in Zhu et al. (2019). We consider three different tasks: (1) training LeNet on corrupted FashionMNIST
3https://github.com/uuujf/SGDNoise
dataset; specifically, we use a 1200-sample subset of the original FashionMNIST training dataset, and for 200 samples points in the training set we randomly assign a label instead of using the correct ones; (2) VGG11 on SVHN dataset, where we use a 25000-sample subset of the training dataset; (3) VGG11 on CIFAR10, where we use the entire training set. For all tasks we use the entire test dataset when evaluating test accuracy.
The heavy-tailed multipliers Zn used in this experiment, whenever heavy-tailed noise is needed, are Zn = cWn where Wn are iid Pareto(α) RVs. For each task, we first randomly initialize each model, and then run the 6 candidate methods in parallel starting from the same randomly initialized model weights for a fair comparison.
The hyperparameters in training for each task are listed in Table A.1. The same set of hyperparameters is used for all methods in the same task. Whenever gradient clipping scheme is applied, we clip the gradient if its L2 norm exceeds the threshold given in Table A.1. The exception here is the “SB + Noise” method: we use learning rate η = 0.005; for FashionMNIST task we train for 100,000 iterations and the heavy-tailed noise is removed for the final 50,000 iterations; for SVHN and CIFAR10 tasks, we train for 150,000 iterations and heavy-tailed noise is removed for the last 70,000 iterations. Besides, for this method we always clip the model weights if its L∞ norm exceeds 1.
The reason for the extra tuning and extended training in “SB + Noise” method is that, without the said modifications, in all three tasks we observed that the model weights quickly drift to infinity and explodes; even with the weight clipping implemented, the model performance stays at random level with no signs of improvements if we do not tune down learning rate.
In Table A.2, we also report the sharpness of solutions under different shaprness metrics. First, the PAC-Bayes Sharpness metric (see equation (53) in Jiang et al. (2019)) is defined as 1/σ2 where σ is equal to the smallest δ that induces a 0.1 expected sharpness, and reflects the sharpness/flatness parameter used in studies on generalization gaps under the PAC-Bayes framework (see Neyshabur et al. (2017a)). Besides, the Maximal Sharpness metric (see equation (54) in Jiang et al. (2019)) is defined as 1/σ2 where σ is equal to the smallest radius δ that makes max‖ν‖∞≤δ |L(θ
∗+ ν)−L(θ∗)| ≥ 0.1, and metrics of form max‖ν‖≤δ |L(θ∗+ν)−L(θ∗)| can be considered as a proxy for the spectral norm of the Hessian at the solution (see Dinh et al. (2017)). It worth noticing that, for all three sharpness metrics, the smaller the value is the "flatter" the loss landscape is around the solution. Lastly, for evaluation of the PAC-Bayes Sharpness and Maximal Sharpness metrics, we conduct binary search as in Algorithm 2 of Jiang et al. (2019) with d = 0.01, σ = 0,M1 = 10 and M2 = 100; in our setting we always evaluate the training loss using one sweep of the entire training set, so M3 is a case-specific and is equal to the number of batches of the training set under the batch size for the task at hand.
In Figure A.6, we plot the test accuracy of our method against that of the SGD for all 5 replications and 3 tasks.

heading: A.4 DETAILS OF CIFAR10/100 EXPERIMENTS WITH DATA AUGMENTATION
text: For both methods, we train the model for 300 epochs and set the initial learning rate as 0.1. In our method, the training can be partitioned into two phases. In the first phase (the first 200 epochs), the learning rate is kept at a constant. In the second phase, for every 30 epoch we reduce the learning rate by half. Also, an L2 weight decaying with coefficient 5 × 10−4 is enforced. As for parameters for heavy-tailed noises in eq. (6), we use c = 0.5 and α = 1.4 in the first phase, and in the second phase we remove heavy-tailed noise and use SB to update weights. In both methods for the small-batch direction gSB the batch size is 128, while for gLB we evaluate the gradient on a large sample batch of size 1,024. Under the epoch number 300 and batch size 128, the count of total iterations performed during training is 1.17× 105. To augment the dataset, random horizontal flipping and cropping with padding size 4 is applied for each training batch. Lastly, gradient clipping scheme is applied for both methods, and we fix b = 0.5. In other words, when the learning rate is η (note that due to the scheduling of learning rates, η will be changing throughout the training), the gradient is clipped if its L2 norm is larger than b/η. The scripts are adapted from the ones in https://github.com/chengyangfu/pytorch-vgg-cifar10.
These results are presented in Table 2. Furthermore, in Table A.3 we see that our truncated heavytailed method also manages to find solutions with a flatter geometry.

heading: A.5 DISCUSSION ON GRADIENT NOISE DISTRIBUTIONS IN THE EXPERIMENTS
text: In this subsection, we (i) present the empirical evidence that supports our characterization of the baseline model—i.e., the absence of heavy tails in the gradient noise—and (ii) clarify that the
emergence of heavy-tails in the *stationary distribution* of SGD (argued in Hodgkinson & Mahoney (2020); Gurbuzbalaban et al. (2020)) does not contradict the observed absence of heavy tails in the *gradient noise* of SGD. This allows us to study the impact of truncated heavy-tails in the gradient noise separately from the choice of hyper-parameters.
We start with our observations on the noise distributions. In view of the well-established wisdom in heavy-tail literature that there is no single perfect tail estimator, we analyzed the stochastic gradient noise with four different methods: QQ plot, empirical mean residual life (EMRL), Hill plot, and PLFIT (Clauset et al., 2009). We applied these estimation/diagnostic tools (i) at the beginning of the training, (ii) halfway through the training, (iii) at the end of the training. Throughout all our experiments, we consistently observe strong evidence that the gradient noises are light-tailed, and even if (against all odds) the noises were from a heavy-tailed distribution, the tail index should be far greater (hence, the resulting tail is much lighter) than the heavy-tails we inject (or the popular alpha stable assumption), and hence, the point we make with our tail-inflation experiment is still valid. We summarize the results as follows.
First, QQ plots below (Fig. A.7-A.21) clearly show that the tails in noise distribution are always much lighter than the Pareto distributions with α = 2 or even 10. Therefore, the typical power-law assumption, especially the alpha-stable distributions in Şimşekli et al. (2019b) (with α ∈ (0, 2)), seems far from the distribution of the actual data we obtained in the image classification tasks. In fact, the tail of the noise distributions seems to be between that of lognormal and normal distributions, implying that it is lighter than any power-law distribution.
Next, we plotted the empirical mean residual life (EMRL) of the gradient noise distributions in Fig. A.22-A.26. It is well known that the mean residual life blows up to infinity if and only if the
distribution is heavy-tailed (more precisely, long-tailed). However, from the figures, one can see that none of the EMRL exhibits such a pattern in any case tested in our experiments. Instead, we see clear downward trends, which strongly suggest light tails, in all the tested cases.
If we assume a power-law tail, the Hill estimator is a classical tail index estimator with a long history in extreme value theory literature. A critical algorithmic parameter of the Hill estimator is the number of order statistics used in the estimation, and the Hill plot is a popular exploratory tool
for investigating the Hill estimators with different numbers of order statistics. Although it is well known that Hill estimators and Hill plots are fallible if the power-law assumption is not satisfied, (and hence, it is not suited for deciding whether a given set of samples are from light-tailed distribution or
heavy-tailed distribution; in particular, the method will return some power-law tail index α whether or not the samples are from a heavy-tailed distribution or a light-tailed one) we present the Hill plot to see what would be the estimated tail indices if the gradient noises hypothetically followed a power law. In the Hill plots shown in Fig. A.27-A.31, we presented the rescaled version (altHill) of the Hill plots (see Chapter 4.4 in (Resnick, 2007)) for the following reason. Hill estimator is a consistent estimator of the power-law index when the proportion of the samples used approaches 0, and altHill plots allow us to scrutinize the estimated indices under a small proportion of samples. In particular, the points around the red dashed lines correspond to estimation using the top 1% of the samples. We can see that most Hill plots stay well above 10 for the most part and almost never drop below 2. This strongly suggests that even if the gradient noises are from a heavy-tailed distribution, it is likely to
have a very high power-law index (implying relatively lighter tails), and hence, we cannot expect to observe a prominent heavy-tailed behavior from them.
A popular data-driven approach with statistical guarantees (again, under the assumption that the samples are indeed from a heavy-tailed distribution) to select the number of order statistics in the Hill plot is PLFIT (Clauset et al., 2009). We estimated the power-law indices using the python implementation (Alstott et al., 2014) of PLFIT. The numbers are presented in Table A.4. All the estimations are at least 5 for all cases tested in our experiments, and most of the time, the estimation is above 10. Again, this means that even under the hypothetical assumption (against what the QQ plots and EMRLs suggest) that the gradient noises were from a heavy-tailed distribution, the tail
indices of the gradient noises should be large, and hence, the gradient noises in our experiments have much lighter tails than any α-stable distribution (which requires α < 2) or the heavy-tailed noises we injected during tail inflation experiments (α = 1.4). In summary, extensive statistical analyses above suggest the absence of heavy tails in the gradient noises in our experiments. Therefore, the characterization of the vanilla SGD as the light-tailed (or at least lighter than the inflated tails) benchmark in our experiments is valid, and our ablation study is well grounded.
Next, we compare our work to recent literature on heavy-tailed phenomena in stationary distribution of SGD; see, for instance, Hodgkinson & Mahoney (2020); Gurbuzbalaban et al. (2020). To be specific, Hodgkinson & Mahoney (2020); Gurbuzbalaban et al. (2020) show that heavy tails can arise in the stationary distribution of SGD through multiplicative dynamics, and the tail index of the resulting stationary distribution can be characterized by the learning rates and the magnitude of noises (through batch size). However, the results in Hodgkinson & Mahoney (2020); Gurbuzbalaban et al. (2020) do not imply the existence of heavy tails in the gradient noises. In both papers (as well as other works in the literature), the “heaviness” of the tail of the gradient noise (e.g., power-law index α in heavy-tailed cases) is fixed in the model (same as in our setting) and not entangled with the learning rate or batch size. For example, Bk in (5) of Hodgkinson & Mahoney (2020) corresponds to the gradient noise, and its tail index doesn’t depend on the learning rate or the batch size. In particular, the change of learning rate does not induce heavy tails in Bk. On the other hand, we focus on the impact of the heavy-tails *in the gradient noise* and the truncation scheme for (any) fixed batch size and small learning rates on the global dynamics of SGD. In view of this, it should be clear that our analysis can be decoupled from the choice of batch size or the impact of the learning rate on SGD’s stationary distribution. Therefore, our observation and the design of the experiments are compatible with the aforementioned references.
B IMPLICATIONS OF THE THEORETICAL RESULTS
Systematic control of the exit times from attraction fields: In light of the wide minima folklore, one may want to find techniques to modify the sojourn time of SGD at each attraction field. Theorem 1 suggests that the order of the first exit time (w.r.t. learning rate η) is directly controlled by the gradient clipping threshold b. Recall that for an attraction field with minimum jump number l∗, Theorem 1 tells us the exit time from this attraction field is roughly of order (1/η)1+(α−1)l ∗ . Given the width
of the attraction field, its minimum jump number l∗ is dictated by gradient clipping threshold b. Therefore, gradient clipping provides us with a very systematic method to control the exit time of each attraction field. For instance, given clipping threshold b, the exit time from an attraction field with width less than b is of order (1/η)α, while the exit time from one larger than b is at least (1/η)2α−1, which dominates the exit time from smaller ones.
The role of structural properties of G and f : Recall that in order for Theorem 3 to apply, the irreducibility of G is required. Along with the choice of b, the geometry of function f is a deciding factor of the irreducibility. For instance, we say that G is symmetric if for any attraction field Ωi such that i = 2, 3, · · · , nmin − 1 (so that Ωi is not the leftmost or rightmost one at the boundary), we have qi,i−1 > 0, qi,i+1 > 0. One can see that G is symmetric if and only if, for any i = 2, 3, · · · , nmin − 1, |si −mi| ∨ |mi − si−1| < l∗i b, and symmetry is a sufficient condition for the irreducibility of G. The graph illustrated in Figure 2(Middle) is symmetric, while the one in Figure 2(Right) is not. As the name suggests, in the R1 case the symmetry of G is more likely to hold if the shape of attraction fields in f is also nearly symmetric around its local minimum. If not, the symmetry (as well as irreducibility) of G can be violated as illustrated in Figure 2, especially when a small gradient clipping threshold b is used.
Generally speaking, our results imply that, even with the truncated heavy-tailed noises, the function f needs to satisfy certain regularity conditions to ensure that SGD iterates avoid undesirable minima. This is consistent with the observations in Li et al. (2018b): the deep neural nets that are more trainable with SGD tend to have a much more regular structure in terms of the number and shape of local minima.
Heavy-tailed SGD without gradient clipping: It is worth mentioning that our results also characterize the dynamics of heavy-tailed SGDs without gradient clipping. For instance, since the reflection operation at ±L restricts the iterates on the compact set [−L,L], if we use a truncation threshold b that is large than 2L, then any SGD update that moves larger than b will definitely be reflected at ±L. Therefore, the dynamics are identical to that of the following iterates without gradient clipping:
Xη,unclippedn = ϕL ( Xη,unclippedn−1 − ηf ′(X η,unclipped n−1 ) + ηZn ) . (B.1)
The next result follows immediately from Theorem 1 and F.2. Corollary B.1. There exist constants qi > 0 ∀i, qi,j > 0 ∀j 6= i such that the following claims hold for any i and any x ∈ Ωi.
1) Under Px, qiH(1/η)σi(η) converges in distribution to an Exponential random variable with rate 1 as η ↓ 0;
2) For any j = 1, 2, · · · , nmin such that j 6= i,
lim η↓0 Px(Xησi(η) ∈ Ωj) = qi,j/qi.
3) Let Y be a continuous-time Markov chain on {m1, · · · ,mnmin} with generator matrix Q parametrized by Qi,i = −qi, Qi,j = qi,j . Then
Xη,unclippedbt/H(1/η)c(x)→ Yt(mi) as η ↓ 0
in the sense of finite-dimensional distributions.
At first glance, Corollary B.1 may seem similar to the results in Şimşekli et al. (2019a) and Pavlyukevich (2007). However, the object studied in Şimşekli et al. (2019a); Pavlyukevich (2007) is different: they study the following Langevin-type stochastic differential equation (SDE) driven by an α-stable Lévy process Lt with scaling factor η > 0:
dY ηt = −f ′(Y η t−)dt+ ηdLt.
In particular, Pavlyukevich (2007) studies the metastability of Y ηt and concludes that as η ↓ 0, the first exit time and global dynamics of Y ηt admit a similar characterization as described in our Theorem B.1. Then Theorem 4 in Şimşekli et al. (2019a) argues that when the learning rate η is sufficiently small, the distribution of the first exit time of the SGD Xηn and that of the Lévy-driven Langevin SDE Y ηt are similar. However, the analysis of Şimşekli et al. (2019a) hinges critically on the assumption
that Lαt is symmetric and α-stable. While such an assumption is convenient for their analysis, it is a strong assumption. It implies that the gradient noise distribution belongs to a very specific parametric family and excludes all the other heavy-tailed distributions. In particular, the assumption precludes analysis of any heavy tails with finite variance. On the contrary, our work directly analyzes the SGD Xηn and reveals the heavy-tailed SGD dynamics at a much greater level of generality. Specifically, we allow the noise to have general regularly varying distributions with arbitrary tail index—which includes α-stable distributions as a (very) special case—and extend the characterization of global dynamics of heavy-tailed SGD to the adaptive versions of SGD where gradient clipping is applied.
C GEOMETRIC CHARACTERIZATION l∗ AND EXISTING SHARPNESS METRICS
Throughout the paper, we have been using the term sharp minima when describing the elimination effect of truncated heavy-tailed SGD, while appealing to l∗, the minimum number of jumps requires for escape defined in eq. (3), when rigorously presenting our theoretical results about dynamics of truncated heavy-tailed SGD. Despite the possible ambiguity of its use in the main paper, the terminology sharpness is meant to familiarize our rather technically involved notion of how heavytailed SGD behave. To resolve the potential confusions, in this section we provide a detailed discussion on the relationship between our geometric characterization l∗ and the existing sharpness metrics.
In light of recent discovery that all local minima might be global minima for over-parametrized deep neural nets (see Li et al. (2018a)), one popular explanation for the generalization gap between different local minima (for instance, between solutions found by GD and SGD) is that the geometry around the solution is closely related to its performance in the test setting. After the seminal work by Keskar et al. (2016), a myriad of attempts have been made to provide empirical or theoretical evidences for the link between the sharpness of a local minimum and its generalization performance; See, for instance, Xie et al. (2020); Jiang et al. (2019); He et al. (2019); Zhou et al. (2020). In summary, there exist at least four different classes of sharpness metrics among the current literature:
(a) spectral norm of the Hessian at the local minimum, one surrogates of which is the maximal sharpness type of metrics; a variant used in Xie et al. (2020) based on corresponding large deviation theory is the eigenvalue of the Hessian along certain escape directions;
(b) expected sharpness type of metrics that evaluate the general fluctuation of the loss function within a Lp ball of centered at the local minimum (see Zhu et al. (2019));
(c) complexity metrics based on PAC-Bayes theory (see, for example, Neyshabur et al. (2017a));
(d) geometric property of a domain or the entire attraction field instead of the local minimum itself; see the mass of a Radon measure m(W ) over the domain W in Theorem 1 in Zhou et al. (2020).
Under such taxonomy, the l∗ characterization proposed in this paper falls into category (d). Indeed, it describes the minimum effort required for escaping the domain rather than the geometry merely around a certain neighborhood of the local minimum. Here we have two remarks on the definition of l∗. First, for a given loss landscape, this quantity l∗ is dictated by the gradient clipping threshold b > 0. In fact, this concept is tailored for the gradient clipping case, as in the unclipped case with heavy-tailed SGD, for any attraction field Ωi we always have l∗i = 1. Second, it is analogous to the term m(W ) in Zhou et al. (2020) in the sense that it reflects the volume of the attraction field (when compared to a given threshold b). Therefore, we stress that a more precise interpretation of l∗, as well as other metrics in class (d), is that they characterize how wide or narrow each minimum (attraction field) is, and a more a clear description of our theoretical results Theorem 1-3 is that truncated heavy-tailed SGD effectively avoids all the narrow minima when learning rate is small enough.
While one can intuitively see that wide minima are more likely to be flat ones, we acknowledge the existence of counterexamples where a sharp minimum lies in a wide attraction field. Nevertheless, as the variety of definitions for sharpness keeps growing in existing literature, it becomes rather unlikely for one geometric characterization to always agree with (or be equivalent to) other existing approaches. More importantly, the aim of this paper is not to establish l∗ as the orthodox geometric property when studying generalization gap. Instead, as demonstrated in Table 1, A.2 and A.3, in
modern deep learning tasks the truncated heavy-tailed method (which prefers solutions with high l∗ as indicated by our theoretical results) obtains solutions that generalize better and exhibit flatter geometry when evaluated under different sharpness metrics. This observation is well aligned with the recent large-scale empirical study in Jiang et al. (2019), suggesting that in typical training setting it is beneficial to find wider minima (characterized by l∗) in order to achieve a flatter geometry and better generalization performance.

heading: D ON THE RELATIONSHIP BETWEEN SHARPNESS AND GENERALIZATION
text: In principle, a sharp minimum does not necessarily lead to poor generalization in the sense that it is possible to construct pathological counterexamples in theory or in specific experiment settings; see, for example, Dinh et al. (2017); Neyshabur et al. (2017a). However, the loss geometry that arises in practice seems to exhibit a strong correlation between the sharpness and the test error. For instance, sharpness-aware optimization methods (Foret et al., 2020; Kwon et al., 2021) improve generalization across various tasks and achieve state-of-the-art performance on the CIFAR dataset. Also, one of the authors of the aforementioned paper Neyshabur et al. (2017a) continued investigating different complexity measures and conducted much larger scale experiments in Jiang et al. (2019). Among more than 40 complexity measures from theoretical and empirical studies in the literature tested in Jiang et al. (2019), the sharpness metrics are the top ones for predicting the generalization performance. In our own experiments, we also show that sharpness and test accuracy are highly correlated.

heading: E PROOF OF THEOREM 1
text: This section proves Theorem 1. We first start with providing the definitions of qi and qi,j that appear in the statement of the theorem. Let Leb+ denote the Lebesgue measure restricted on [0,∞), and define a (Borel) measure να on R\{0} as follows:
να(dx) = 1{x > 0} αp+ xα+1 + 1{x < 0} αp− |x|α+1
where α, p−, and p+ are constants in Assumption 2. Define a Borel measure µi on Rl ∗ i × ( R+ )l∗i−1
as the product measure
µi = (να) l∗i × (Leb+)l ∗ i−1. (E.1)
We also define mappings hi as follows. For a real sequence w = (w1, w2, · · · , wl∗i ) and a positive real number sequence t = (t′j) l∗i j=2, define t1 = t ′ 1 = 0 and tj = t ′ 1 + t ′ 2 + · · ·+ t′j for j = 2, · · · , l∗i . Now we define a path x̂ : [0,∞) 7→ R as the solution to the following ODE with jumps: x̂(0) = ϕL ( mi + ϕb(w1) ) ; (E.2)
dx̂(t) dt = −f ′(x̂(t)), ∀t ∈ [tj−1, tj), ∀j = 2, · · · , l∗i ; (E.3)
x̂(tj) = ϕL ( x̂(tj−) + ϕb(wj) ) , ∀j = 2, · · · , l∗i . (E.4)
Now we define the mappings hi : Rl ∗ i × ( R+ )l∗i−1 7→ R as
hi(w, t) = x̂(tl∗i ). It is easy to see that hi’s are continuous mappings. With these mappings, we define the following sets:
Ei = ∆ {(w, t) ⊆ Rl ∗ i × ( R+ )l∗i−1 : hi(w, t) /∈ Ωi}; (E.5)
Ei,j = ∆ {(w, t) ⊆ Rl ∗ i × ( R+ )l∗i−1 : hi(w, t) ∈ Ωj}. (E.6)
Lastly, the constant qi and qi,j are defined as follows:
qi = µi(Ei), qi,j = µi(Ei,j) ∀i 6= j. (E.7)
Before we move on to the proof of Theorem 1, we add a few remarks regarding the intuition behind it.
• Suppose that Xηn is started at the i th local minimum mi of f , and consider the behavior of
Xηn over the time period H1 , {1, . . . , dt/ηe} for a sufficiently large t. The heavy-tailed large deviations theory Rhee et al. (2019) and a heuristic application of the contraction principle implies that the path of Xηdn/ηe over this period will converge to the gradient flow of f , and the event that Xηdn/ηe escapes Ωi within this period is a (heavy-tailed) rare event. This means that the probability of such an event is of order (1/η)(α−1)l ∗ i . Moreover, whenever it happens, it is almost always because Xηn is shaken by exactly l ∗ i large gradient noises of size O(1/η), which translates to l∗i jumps in X η dn/ηe’s path, while the rest of its path closely resemble the deterministic gradient flow. Moreover, conditional on the event that Xηn fails to escape from Ωi within this period, the endpoint of the path is most likely to be close to the local minima, i.e., Xηdt/ηe ≈ mi. This suggests that over the next time period H2 , {dt/ηe+ 1, dt/ηe+ 2, . . . , 2dt/ηe} of length dt/ηe, Xηn will behave similarly to its behavior over the first period H1. The same argument applies to the subsequent periods H3, H4, . . . as well. Therefore, over each time period of length dt/ηe, there is (1/η)(α−1)l ∗ i probability of exit. In view of this, the exit time should be of order (1/η)1+(α−1)l ∗ i and
resemble an exponential distribution when scaled properly. • Part (i) of Theorem 1 builds on this intuition and rigorously prove that the first exit time is
indeed roughly of order 1/λi(η) ≈ (1/η)1+(α−1)l ∗ i and resembles an exponential distribu-
tion. • Given this, one would expect that Xησi(η), the location of SGD right at the time of exit, will
hardly ever be farther than l∗i b away from mi: the length of each update is clipped by b, and there will most likely be only l∗i large SGD steps during this successful attempt. Indeed, from the definition of qi,j’s above, one can see that qi,j > 0 if and only if infy∈Ωj |y−mi| < l∗i b.
Summarizing the three bullet points here, we see that the minimum number of jumps l∗i dictates how heavy-tailed SGD escapes an attraction field, where the SGD lands on upon its exit, and when the exit occurs.
Now we are ready to start proving Theorem 1. First, note that Assumption 1 implies the following:
• There exist c0 > 0, 0 ∈ (0, 1) such that for any x ∈ {m1, s1, · · · , snmin−1,mnmin}, |y−x| < 0,
|f ′(y)| > c0|y − x|, (E.8)
and for any y ∈ [−L,L] such that |y − x| ≥ 0 for all x ∈ {m1, s1, · · · , snmin−1,mnmin}, we have
|f ′(y)| > c0; (E.9)
• There exist constants L ∈ (0,∞),M ∈ (0,∞) such that |m0| < L, |mnmin | < L, and (for any x ∈ [−L,L])
|f ′(x)| ≤M, |f ′′(x)| ≤M. (E.10)
Recall that for c > 0, the truncation operator was defined as
ϕc(w) , ϕ(w, c) = ∆ (w ∧ c) ∨ (−c), ∀w ∈ R, (E.11)
and the SGD iterates were defined as
Xηn = ϕL ( Xηn − ϕb ( η(f ′(Xηn)− Zn+1) )) . (E.12)
Here η > 0 is the learning rate (step length) and b > 0 is the gradient clipping threshold. Also, recall that for any k ∈ [nmin], we let
σk(η) = ∆ min{n ≥ 0 : Xηn /∈ Ωk}
to be the time that Xηn exists from the k−th attraction field Ωj . Meanwhile, given the gradient clipping threshold b, recall that
ri = ∆ min{mi − si−1, si −mi}, (E.13) l∗i = ∆ dri/be. (E.14)
Intuitively speaking, l∗i tells us the minimum number of jumps with size no larger than b required in order to escape the attraction field if we start from the local minimum of this attraction field Ωi. Lastly, recall the definition of H(·) = P(|Z1| > ·) and
λi(η) = ∆ H(1/η) (H(1/η) η )l∗i−1 .
The proof of Theorem 1 hinges on the following two lemmas that characterize the behavior of Xηn in two different phases respectively. Let k ∈ [nmin] and x ∈ Ωk. We consider the SGD iterates initialized at Xη0 = x. In the first phase, the SGD iterates return to [mk − 2 ,mk + 2 ], a small neighborhood of the local minimizer in attraction field Ωk; in other words, it ends at
T (k) return(η, ) = ∆ min{n ≥ 0 : Xηn ∈ [mk − 2 ,mk + 2 ]}. (E.15)
During this phase, we show that for all learning rate η that is sufficiently small, it is almost always the case that Xηn would quickly return to [mk − 2 ,mk + 2 ], and it never leaves Ωk before T (k) return.
Lemma E.1. There exists some c ∈ (0,∞) such that for any k ∈ [nmin], the following claim holds for all > 0 small enough:
lim η↓0 inf y∈[−L,L]: y∈(sk−1+ ,sk− )
Py ( Xηn ∈ Ωk ∀n ∈ [ T (k) return(η, ) ] , T (k) return(η, ) ≤ c log(1/ )
η
) = 1.
During the second phase, Xηn starts from somewhere in [mk − 2 ,mk + 2 ] and tries to escape from Ωk, meaning that the phase ends at σk(η). During this phase, we show that the distributions of the first exit time σk(η) and the location X
η σk(η) do converge to the ones described in Theorem 1 as learning rate η tends to 0.
Lemma E.2. There exist constants qi > 0 ∀i ∈ [nmin] and qi,j ≥ 0 ∀j ∈ [nmin] with i, j 6= i such that the following claim holds: given any C > 0, u > 0 and any k, l ∈ [nmin] with k 6= l, we have
lim sup η↓0 sup x∈[−L,L], x∈(mk−2 ,mk+2 )
Px ( qkλk(η)σk(η) > u ) ≤ C + exp ( − (1− C)u ) (E.16)
lim inf η↓0 inf x∈[−L,L], x∈(mk−2 ,mk+2 )
Px ( qkλk(η)σk(η) > u ) ≥ −C + exp ( − (1 + C)u ) (E.17)
lim sup η↓0 sup x∈[−L,L], x∈(mk−2 ,mk+2 )
Px ( Xησk(η) ∈ Ωl ) ≤ qk,l + C
qk (E.18)
lim inf η↓0 inf x∈[−L,L], x∈(mk−2 ,mk+2 )
Px ( Xησk(η) ∈ Ωl ) ≥ qk,l − C
qk (E.19)
for all > 0 that are sufficiently small.
Now we are ready to show Theorem 1.
Proof of Theorem 1. Fix some k ∈ [nmin] and x ∈ Ωk ∩ [−L,L]. Let qk and qk,l be the constants in Lemma E.2.
We first prove the weak convergence claim in Theorem 1(i). Arbitrarily choose some u > 0 and C ∈ (0, 1). It suffices to show that
lim sup η↓0
Px(qkλk(η)σk(η) > u) ≤ 2C + exp ( − (1− C)u ) ,
lim inf η↓0
Px(qkλk(η)σk(η) > u) ≥ (1− C) ( − C + exp ( − (1 + C)u )) .
Recall the definition of the stopping time T (k)return in eq. (E.15). Define event
Ak(η, ) = ∆ { Xηn ∈ Ωk ∀n ∈ [ T (k) return(η, ) ] , T (k) return(η, ) ≤ c log(1/ )
η } where c <∞ is the constant in Lemma E.1. First, since x ∈ Ωk = (sk−1, sk), it holds for all > 0 small enough that x ∈ (sk−1 + , sk − ). Next, one can find some > 0 such that
• (Due to Lemma E.1) Px ( (Ak(η, )) c ) ≤ C ∀η sufficiently small;
• (Due to eq. (E.16)eq. (E.17) and strong Markov property) For all η sufficiently small, Px ( qkλk(η) ( σ(η)− T (k)return(η, ) ) > (1− C)u ∣∣∣ Ak(η, )) ≤ C + exp (− (1− C)u), Px ( qkλk(η) ( σ(η)− T (k)return(η, ) ) > u
∣∣∣ Ak(η, )) ≥ −C + exp (− (1 + C)u). Fix such . Lastly, for this fixed , due to λk ∈ RV−1−l∗k(α−1)(η) and α > 1, we have qkλk(η) · c log(1/ )
η < Cu for all η sufficiently small. In summary, for all η sufficiently small, we have
Px(qkλk(η)σk(η) > u) ≤Px ( (Ak(η, )) c ) + Px ({ qkλk(η)σk(η) > u } ∩Ak(η, ) ) ≤C + Px ({ qkλk(η)σk(η) > u } ∩Ak(η, )
) =C + Px ( qkλk(η) ( σ(η)− T (k)return(η, ) ) > u− qkλk(η)T (k)return(η, )
∣∣∣ Ak(η, )) · Px(Ak(η, )) ≤C + Px ( qkλk(η) ( σ(η)− T (k)return(η, ) ) > (1− C)u
∣∣∣ Ak(η, )) ≤2C + exp ( − (1− C)u
) and
Px(qkλk(η)σk(η) > u) ≥Px ({ qkλk(η)σk(η) > u } ∩Ak(η, ) ) =Px ( qkλk(η) ( σ(η)− T (k)return(η, ) ) > u− qkλk(η)T (k)return(η, )
∣∣∣ Ak(η, )) · Px(Ak(η, )) ≥Px ( qkλk(η) ( σ(η)− T (k)return(η, ) ) > u− qkλk(η)T (k)return(η, )
∣∣∣ Ak(η, )) · (1− C) ≥Px ( qkλk(η) ( σ(η)− T (k)return(η, ) ) > u
∣∣∣ Ak(η, )) · (1− C) ≥(1− C) ( − C + exp ( − (1 + C)u
)) so this concludes the proof for Theorem 1(i).
In order to prove claims in Theorem 1(ii), we first observe that on event Ak(η, ) we must have σ(η) > T (k) return(η, ). Next, arbitrarily choose some C ∈ (0, 1), and note that it suffices to show that
Px(Xησ(η) ∈ Ωl) ∈ ( (1− C) qk,l−Cqk , C + qk,l+C qk ) holds for all η sufficiently small. Again, we can find > 0 such that
• (Due to Lemma E.1) Px ( (Ak(η, )) c ) ≤ C ∀η sufficiently small;
• (Due to eq. (E.18)eq. (E.19) and strong Markov property) For all η sufficiently small,
qk,l − C qk ≤ Px ( Xησk(η) ∈ Ωl ∣∣∣ Ak(η, )) ≤ qk,l + C qk .
In summary, for all η sufficiently small, we have Px ( Xησk(η) ∈ Ωl ) ≤ Px ( (Ak(η, )) c ) + Px ({ Xησk(η) ∈ Ωl } ∩Ak(η, ) ) ≤ C + Px ( Xησk(η) ∈ Ωl
∣∣∣ Ak(η, ))Px(Ak(η, )) ≤ C + qk,l + C
qk
and
Px ( Xησk(η) ∈ Ωl ) ≥ Px ({ Xησk(η) ∈ Ωl } ∩Ak(η, ) ) = Px ( Xησk(η) ∈ Ωl
∣∣∣ Ak(η, ))Px(Ak(η, )) ≥ (1− C)qk,l − C
qk
and this concludes the proof.
The rest of this section is devoted to the proofs of Lemma E.1 and Lemma E.2. Specifically, Lemma E.1 is an immediate Corollary of Lemma E.13, the proof of which will be provided below. The proof of Lemma E.2 can be found at the end of this section.

heading: E.1 PROOFS OF LEMMA E.1, E.2
text: The following three lemmas will be applied repeatedly throughout this section. The proofs are straightforward but provided in Section I for the sake of completeness. Lemma E.3. Given two real functions a : R+ 7→ R+, b : R+ 7→ R+ such that a( ) ↓ 0, b( ) ↓ 0 as ↓ 0, and a family of geometric RVs {U( ) : > 0} with success rate a( ) (namely, P(U( ) > k) = (1− a( ))k for k ∈ N), for any c > 1, there exists 0 > 0 such that for any ∈ (0, 0),
exp ( − c · a( )
b( )
) ≤ P ( U( ) > 1
b( )
) ≤ exp ( − a( ) c · b( ) ) .
Lemma E.4. Given two real functions a : R+ 7→ R+, b : R+ 7→ R+ such that a( ) ↓ 0, b( ) ↓ 0 and a( )/b( )→ 0 as ↓ 0, and a family of geometric RVs {U( ) : > 0} with success rate a( ) (namely, P(U( ) > k) = (1− a( ))k for k ∈ N), for any c > 1 there exists some 0 > 0 such that for any ∈ (0, 0),
a( )/(c · b( )) ≤ P(U( ) ≤ 1/b( )) ≤ c · a( )/b( )
Lemma E.5. Suppose that a function g : E 7→ R (where E is an open set in Rd) is g ∈ C2(E) and∥∥∇2g(·)∥∥ ≤ C on its domain E for some constant C < ∞. For a finite integer n, a sequence of vectors {z1, · · · , zn} in Rd, and vectors x, x̃ ∈ E, η > 0, consider two sequences {xk}k=0,...,n and {x̃k}k=0,...,n constructed as follows:
x0 = x
xk = xk−1 + η∇g(xk−1) + ηzk ∀k = 1, 2, · · · , n x̃0 = x̃
x̃k = x̃k−1 + η∇g(x̃k−1) ∀k = 1, 2, · · · , n If we have that the line segment from xk to x̃k is contained inE and η ‖z1 + · · ·+ zk‖+‖x− x̃‖ ≤ c̃ for all k = 1, 2, · · · , n for some c̃ > 0, then
‖xk − x̃k‖ ≤ c̃ · exp(ηCk) ∀k = 1, 2, · · · , n.
To facilitate the analysis below, we introduce some additional notations. First, we will group the noises Zn based on a threshold level δ > 0: let us define
Z≤δ,ηn = ∆ Zn1{η|Zn| ≤ δ}, (E.20)
Z>δ,ηn = ∆ Zn1{η|Zn| > δ}. (E.21)
The former are viewed as small noises while the latter will be referred to as large noises or large jumps. Furthermore, for any j ≥ 1, define the jth arrival time and size of large jumps as
T ηj (δ) = ∆ min{n > T ηj−1(δ) : η|Zn| > δ}, T η 0 (δ) = 0 (E.22) W ηj (δ) = ∆ ZTηj (δ). (E.23)
Next, for any > 0, let Ωi( ) = [mi − ,mi + ] be an −neighborhood of the local minimum mi, and Si( ) = [si − , si + ] be an −neighborhood of the local maximum si. For most part of this section, we will zoom in on one of the local minima mi and its attraction field Ωi = (si−1, si). Without loss of generality, we assume mi = 0, and denote the attraction field as Ω = (s−, s+). (Ifmi happens to be the local minimum at the left or right boundary, then the attraction field is [−L, s+) or (s−, L] where the SGD iterates will be reflected at ±L.) Henceforth we will drop the dependency on notation i when referring to this specific attraction field until the very end of this section. Throughout the proof, the following (deterministic) dynamic systems will be used frequently as benchmark processes to indicate the most likely location of the SGD iterates. Specifically, given any x ∈ Ω, we use Xn(x) to indicate that the starting point is x, namely X0(x) = x. Similarly, consider the following ODE xη(t;x) as
xη(0;x) = x; (E.24) dxη(t;x)
dt = −ηf ′
( xη(t;x) ) . (E.25)
When we use update rate η = 1, we will drop the dependency of η and simply use x(t;x) to denote the process.
Based on Assumption 3, we know the existence of some constant ̄ ∈ (0, 0) (note that 0 is the constant in eq. (E.8)) such that
r =∆ min{−s−, s+}, (E.26) l∗ =∆ dr/be, (E.27)
(l∗ − 1)b+ 100l∗̄ < r − 100l∗̄ (E.28) r + 100l∗̄ < l∗b− 100l∗̄. (E.29)
Here r can be understood as the effective radius of the said attraction field. Also, we fix such ̄ small enough so that (let cL− = −f ′(−L), cL+ = −f ′(−L)), we have
0.9cL− ≤ −f ′(x) ≤ 1.1cL− ∀x ∈ [−L,−L+ 100̄], (E.30) 0.9cL+ ≥ −f ′(x) ≥ 1.1cL+ ∀x ∈ [L− 100̄, L]. (E.31)
Similar to the definition of ODE xη, let us consider the following construction of ODE x̃η that can be understood as xη perturbed by l∗ shocks. Specifically, consider a sequence or real numbers 0 = t1 < t2 < t3 < · · · < tl∗ and real numbers w1, · · · , wl∗ where |wj | ≤ b for each j. Let t = (t1, · · · , tl∗),w = (w1, · · · , wl∗). Based on these two sequences and rate η > 0, define x̃η(t;x) as
x̃η(0, x; t,w) = ϕL(x+ ϕb(w1)); (E.32) dx̃η(t, x; t,w)
dt = −ηf ′
( x̃η(t, x; t,w) ) ∀t /∈ {t1, t2, · · · , tl∗} (E.33)
x̃η(tj , x; t,w) = ϕL ( x̃η(tj−, x; t,w) + ϕb(wj) ) ∀j = 2, · · · , l∗ (E.34)
Again, when η = 1 we drop the notational dependency on η and use x̃ to denote the process. Now from eq. (E.28)eq. (E.29) one can easily see the following fact: there exist constants t̄, δ̄ > 0 such that x̃(tl∗ , 0; t,w) /∈ Ω (note that the starting point is 0, the local minimum) only if (under the condition that |wj | ≤ b ∀j)
tj − tj−1 ≤ t̄ ∀j = 2, 3, · · · , l∗ (E.35) |wj | > δ̄ ∀j = 1, 2, · · · , l∗. (E.36)
The intuition is as follows: if the inter-arrival time between any of the l∗ jumps is too long, then the path of x̃η(t;x) will drift back to the local minimum mi so that the remaining l∗ − 1 shocks (whose
sizes are bounded by b) cannot overcome the radius r which is strictly larger than (l∗ − 1)b; similarly, if size of any of the shocks is too small, then since all other jumps have sizes bounded by b, the shock created by the l∗i jumps will be smaller than (l
∗ − 1)b+ 100̄, which is strictly less than r. We fix these constants t̄, δ̄ throughout the analysis, and stress again that their values are dictated by the geometry of the function f , thus do not vary with the accuracy parameters and δ mentioned earlier. In particular, choose δ̄ such that δ̄ < ̄.
In our analysis below, > 0 will be a variable representing the level of accuracy in our analysis. For instance, for small , the chance that SGD iterates will visit somewhere that is −close to s− or s+ (namely, the boundary of the attraction filed) should be small. Consider some ∈ (0, 0) where 0 is the constant in Assumption 1. Due to eq. (E.8)eq. (E.9), one can see the existence of some g0 > 0, c1 <∞ such that
• |f ′(x)| ≥ g0 for any x ∈ Ω such that |x− s−| > 0, ||x− s+| > 0; • Let t̂ODE(x, η) =
∆ min{t ≥ 0 : xη(t, x) ∈ [− , ]} be the time that the ODE returns to a −neighborhood of local minimum of Ω when starting from x. As proved in Lemma 3.5 of Pavlyukevich (2005), for any x ∈ Ω such that |x− s−| > , |x− s+| > , we have
t̂ODE(x, η) ≤ c1 log(1/ )
η (E.37)
and we define the function
t̂( ) =∆ c1 log(1/ ). (E.38)
In short, given any accuracy level , the results above give us an upper bound for how fast the ODE would return to a neighborhood of the local minimum, if the starting point is not too close to the boundary of this attraction field Ω.
For the first few technical results established below, we show that, without large jumps, the SGD iterates Xηn(x) are unlikely to show significant deviation from the deterministic gradient descent process yηn(x) defined as
yη0(x) = x, (E.39)
yηn(x) = y η n−1(x)− ηf ′ ( yηn−1(x) ) . (E.40)
We are ready to state the first lemma, where we bound the distance between the gradient descent iterates yηn(y) and the ODE xη(t, x) when the initial conditions x, y are close enough. Lemma E.6. The following claim holds for all η > 0: for any t > 0, we have
sup s∈[0,t]
|xη(s, x)− yηbsc(y)| ≤ (2ηM + |x− y|) exp(ηMt)
where M ∈ (0,∞) is the constant in eq. (E.10).
Proof. Define a continuous-time process yη(s; y) =∆ yηbsc(y), and note that
xη(s, x) = xη(bsc, x)− η ∫ s bsc f ′(xη(u, x))du
xη(bsc, x) = x− η ∫ bsc
0
f ′(xη(u, x))du
yηbsc(y) = y η(bsc, y) = y − η ∫ bsc 0 f ′(yη(u, y))du.
Therefore, if we define function
b(u) = xη(u, x)− yη(u, y),
from the fact |f ′(·)| ≤ M , one can see that |b(u)| ≤ ηM + |x − y| for any u ∈ [0, 1) and |b(1)| ≤ 2ηM + |x− y|. In case that s > 1, from the display above and the fact |f ′′(·)| ≤ M , we
now have |yηbsc(x)− x η(s, x)| ≤ |b(bsc)|+ ηM ;
|b(bsc)| ≤ ηM ∫ bsc
1
|b(u)|du.
From Gronwall’s inequality (see Theorem 68, Chapter V of Protter (2005), where we let function α(u) be α(u) = |b(u+ 1)|), we have
|yηbsc(x)− x η(s, x)| ≤ (2ηM + |x− y|) exp(ηMt).
This concludes the proof.
Now we consider an extension of the previous Lemma in the following sense: we add perturbations to the gradient descent process and ODE, and show that, when both perturbed by l∗ similar perturbations, the ODE and gradient descent process should still stay close enough. Analogous to the definition of the perturbed ODE x̃η in eq. (E.32)-eq. (E.34), we can construct a process Ỹ η as a perturbed gradient descent process as follows. For a sequence of integers 0 = t1 < t2 < · · · < tl∗ (let t = (tj)j≥1) and a sequence of real numbers w̃1, · · · , w̃l∗ (let w̃ = (w̃j)j≥1) and y ∈ R, define (for all n = 1, 2, · · · , tl∗ ) the perturbed gradient descent iterates with gradient clipping at b and reflection at ±L as
ỹηn(y; t, w̃) = ϕL ( ỹηn−1(y; t, w̃) + ϕb ( − ηf ′(ỹηn−1(y; t, w̃)) + l∗∑ j=2 1{n = tj}w̃j ))
(E.41)
with initial condition ỹη0(y; t, w̃) = ϕL ( y + ϕb(w̃1) ) . Corollary E.7. Given any > 0, the following claim holds for all sufficiently small η > 0: for any x, y ∈ Ω, and sequence of integers t = (tj)l ∗
j=1 and any two sequence of real numbers w = (wj)l ∗ j=1, w̃ = (w̃j)l ∗ j≥1 such that
• |x− y| < ;
• t1 = 0, and tj − tj−1 ≤ 2t̄/η for all j ≥ 1 where t̄ is the constant in eq. (E.35);
• |wj − w̃j | < for all j ≥ 1;
then we have sup
t∈[0,tl∗ ] |x̃η(t, x; t,w)− ỹηbtc(y; t, w̃)| ≤ ρ̄
where the constant ρ̄ = (3 exp(ηMt̄) + 3)l ∗ .
Proof. Throughout this proof, fix some η ∈ (0, /2M). We will show that for any η in the range the claim would hold.
First, on interval [0, t2), from Lemma E.6, one can see that (since 2Mη < ) sup
t∈[0,t2) |x̃η(t, x; t,w)− ỹηbtc(y; t, w̃)| ≤ 3 exp(ηMt̄) · .
The at t = t2, by considering the difference between w2 and w̃2, and the possible change due to one more gradient descent step (which is bounded by ηM < ), we have
sup t∈[0,t2]
|x̃η(t, x; t,w)− ỹηbtc(y; t, w̃)| ≤ (3 exp(ηMt̄) + 2) · .
Now we proceed inductively. For any j = 2, 3, · · · , l∗ − 1, assume that sup
t∈[0,tj ] |x̃η(t, x; t,w)− ỹηbtc(y; t, w̃)| ≤ (3 exp(ηMt̄) + 3) j−1 · .
Then by focusing on interval [tj , tj+1] and using Lemma E.6 again, one can show that
sup t∈[tj ,tj−1]
|x̃η(t, x; t,w)− ỹηbtc(y; t, w̃)| ≤ 2 + ( (3 exp(ηMt̄) + 3)j−1 + 1 ) exp(ηMt̄)
≤ (3 exp(ηMt̄) + 3)j · . This concludes the proof.
In the next few results, we show that the same can be said for gradient descent iterates ỹn and the SGD iterates Xn. Specifically, our first goal is to show that before any large jump (see the definition in E.21), it is unlikely that the gradient descent process yηn would deviate too far from Xηn. Define the event
A(n, η, , δ) = {
max k=1,2,··· ,n∧(Tη1 (δ)−1)
η|Z1 + · · ·+ Zk| ≤ }
(E.42)
and recall that arrival times T ηj (δ) are defined in eq. (E.22).
As a building block, we first study the case when the starting point x is close to the reflection boundary −L. The takeaway from the next result is that the reflection operator hardly comes into play, since the SGD iterates would most likely quickly move to somewhere far enough from±L; besides, throughout this procedure the SGD iterates would most likely stay pretty close to the corresponding deterministic gradient descent process. Lemma E.8. Given ∈ (0, ̄/9), it holds for any sufficiently small , δ, η > 0 that, if x ∈ [−L,−L+ ̄] and ρ0(|x− y|+ 9 ) < ̄, then on event A(n, η, , δ) we have
|Xηk (x)− y η k(y)| ≤ ρ0 · (|x− y|+ 9 ) ∀k = 1, 2, · · · , n ∧ (T η 1 (δ)− 1) ∧ T̃ ηescape(x)
where T̃ ηescape(x) = ∆ min{n ≥ 0 : Xηn(x) > −L+ ̄} and ρ0 =
∆ exp (
2M̄ 0.9cL−
) is a constant that does
not vary with our choice of , δ, η.
Proof. For any k < T η1 (δ), we know that Zk = Z ≤δ k (thus η|Zk| < δ). Also, recall that |f ′(x)| ≤M for any x ∈ Ωi. Therefore, as long as η and δ are small enough, we will have that
|η(−f ′(Xηn(x)) + Z ≤δ k )| ≤ b (E.43)
so the gradient clipping operator in eq. (E.12) has no effect when k < T η1 (δ), and in fact the only possible time for the gradient clipping trick to work is at T ηj (δ). Therefore, we can safely rewrite the SGD update as
Xηk (x) = X η k−1(x)− ηf ′(Xηk−1(x)) + ηZk +Rk ∀k < T η 1 (δ)
where each Rk ≥ 0 and it represents the push caused by reflection at −L. First, choose small enough so that 9 < ̄. Next, based on eq. (E.31) we have the following lower bound:
Xηk (x) ≥ x+ 0.9c L −ηk − ∀k < T η 1 (δ).
Let t̃0(x, ) = ∆ min{n ≥ 0 : Xηn(x) ≥ −L+ 2 }. Due to the inequality above, we know that
t̃0(x, ) ≤ 3
0.9cL−η . (E.44)
One the other hand, given the current choice of , if we choose η and δ small enough, then using the same argument leading to eq. (E.43), we will have
Xη t̃0(x, ) (x) ≤ −L+ 2.1 ≤ x+ 1.1cL−ηk + 2.1
if t̃0(x, ) ≥ 1 (namely x < −L+ 2 ).
Let us inspect the two scenarios separately. First, assume t̃0(x, ) ≥ 1. For the deterministic gradient descent process yηn(y), we have the following bounds:
y + 0.9cL−ηk ≤ y η k(y) ≤ y + 1.1c L −ηk ∀k ≤ t̃0(x, ) ∧ (T η 1 (δ)− 1).
This gives us
|Xηk (x)− y η k(y)| ≤ |x− y|+ 0.2c L −ηk + 2.1 ∀k ≤ t̃0(x, ) ∧ (T η 1 (δ)− 1).
At time k = t̃0(x, ), due to previous bound on t̃0(x, ), we know that |Xηt̃0(x, )(x)− y η t̃0(x, ) (y)| ≤ |x− y|+ 7 . If n∧ (T η1 (δ)− 1) ≤ t̃0(x, ) then we have already shown the desired claim. Otherwise,
starting from time t̃0(x, ), due to the definition of event A(n, η, , δ) in eq. (E.42), we know that the SGD iterates Xηn(x) will not touch the boundary −L afterwards. Therefore, by directly applying Lemma E.5, and notice that |f ′′(x)| ≤M for any x ∈ [−L,L] and, we have
|Xηk (x)− y η k(y)| ≤ (|x− y|+ 9 ) · exp ( 2M̄ 0.9cL− ) ∀k ≤ n ∧ (T η1 (δ)− 1) ∧ T̃ ηescape(x).
Indeed, it suffices to use Lemma E.5 for the next d2̄/(0.9ηcL−)e steps to show that |X η k (x)− y η k(y)| should be smaller than for the next d2̄/(0.9ηcL−)e steps, while y η k(y) will reach some where in (−L+ 2̄,−L+ 3̄) within d2̄/(0.9ηcL−)e steps so we must have
n ∧ (T η1 (δ)− 1) ∧ T̃ ηescape(x) ∧ t̃0(x, )− t̃0(x, ) ≤ 2̄/(0.9ηcL−) (E.45)
Lastly, in the case t̃0(x, ) = 0 (which means x ≥ −L+ ), we can use Lemma E.5 directly as we did above and establish the same bound. This concludes the proof.
Obviously, a similar result can be shown if x is in the rightmost attraction field (snmin−1 , L] and the approach is identical. We omit the details here. In the next Lemma, we consider the scenario where the starting point x is far enough from the boundaries. Lemma E.9. Given any > 0, the following holds for all sufficiently small η > 0: for any x, y ∈ Ω and positive integer n such that |x − L| > 2 , |x + L| > 2 , |x − s−| > 2 , |x − s+| > 2 and |x− y| < 2 exp(ηMn) , on event
A(n, η, 2 exp(ηMn) , δ) ∩
{ |yηj (y)| ∈ Ω, |X η j (x)| ∈ Ω ∀j = 1, 2, · · · , n ∧ (T η 1 (δ)− 1) } we have
|X̃ηm(y)−Xηm(x)| ≤ ∀m = 1, 2, · · · , n ∧ (T η 1 (δ)− 1).
Proof. For sufficiently small η, we will have that the (deterministic) gradient descent iterates |yηn| is monotonically decreasing in n, which ensures that yηn always stays in the range that are at least −away from ±L or s−, s+. We now show that the claim holds for any such η.
On event { |yηj (y)| ∈ Ω, |X η j (x)| ∈ Ω ∀j = 1, 2, · · · , n ∧ (T η 1 (δ) − 1) } , we are able to apply Lemma E.5 inductively for any m ∈ [n] and obtain that
|yηj (y)−X η j (x)| ≤ (|x− y|+ 2 exp(ηMn) ) exp(ηMj) < ∀j = 1, 2, · · · ,m
and conclude the proof. The reason to apply the Lemma inductively for m = 1, 2, · · · , n, instead of directly at step n, is to ensure that SGD iterates Xηn would not hit the boundary ±L (so the reflection operator would not come into play on the time interval we are currently interested in), thus ensuring that Lemma E.5 is applicable.
Similar to the extension from Lemma E.6 to Corollary E.7, we can extend Lemma E.9 to show that, if we consider the a gradient descent process that is only perturbed by large noises, then it should stay pretty close to the SGD iterates Xηn. To be specific, let
Y η0 (x) = x (E.46)
Y ηn (x) = ϕL ( Y ηn−1(x)− ϕb ( − ηf ′ ( Y ηn−1(x) ) + ∑ j≥1 1{n = T ηj (δ)}ηZn )) . (E.47)
be a gradient descent process (with gradient clipping at threshold b) that is only shocked by large noises in (Zn)n≥1. The next corollary can be shown by an approach that is identical to Corollary E.7 (namely, inductively repeating Lemma E.9 at each jump time) so we omit the details here. Corollary E.10. Given any > 0, the following holds for any sufficiently small η > 0: For any |x| < 2 , on event A0( , η, δ) ∩B0( , η, δ), we have
|Y ηn (x)−Xηn(x)| < ρ̃ ∀n = 1, 2, · · · , T η l∗(δ)
where
A0( , η, δ) = ∆ { ∀i = 1, · · · , l∗, max
j=Tηi−1(δ)+1,··· ,T η i (δ)−1
η|ZTηi−1(δ)+1 + · · ·+ Zj | ≤
2 exp(2t̄M)
} ;
B0( , η, δ) = ∆ { ∀j = 2, · · · , l∗, T ηj (δ)− T η j−1(δ) ≤ 2t̄/η } and ρ̃ ∈ (0,∞) is a constant that does not vary with η, δ, .
The next two results shows that the type of events A(n, η, , δ) defined in eq. (E.42) is indeed very likely to occur, especially for small . For clarity of the presentation, we introduce the following definitions that are slightly more general than the small and large jumps defined in eq. (E.20)eq. (E.21) (for any c > 0)
Z≤cn = ∆ Zn1{|Zn| ≤ c}, Z>cn = ∆ Zn1{|Zn| > c}.
Lemma E.11. Define functions u(η) = δ/η1−∆, v(η) = η∆̃ with , δ > 0. If real numbers ∆, ∆̃, β, , δ and positive integers j,N are such that the following conditions hold:
∆ ∈ [ 0, (1− 1
α ) ∧ 1 2
) , (E.48)
β ∈ ( 1, (2− 2∆) ∧ α(1−∆) ) , (E.49)
∆̃ ∈ [0, ∆ 2 ], ∆̃ < α(1−∆)− β, (E.50)
N < ( α(1−∆)− β ) j, (E.51)
v(η)− jηu(η) ≥ v(η)/2 for all η > 0 sufficiently small, (E.52) then
P (
max k=1,2,··· ,d1/ηβe
η ∣∣Z≤u(η)1 + · · ·+ Z≤u(η)k ∣∣ > 3v(η)) = o(ηN )
as η ↓ 0.
Proof. From the stated range of the parameters, we know that α(1−∆) > β,(
α(1−∆)− β ) j > N,
so we are able to find γ ∈ (0, 1) small enough such that α(1−∆)(1− 2γ) > β, (E.53)(
α(1−∆)(1− 2γ)− β ) j > N. (E.54)
Fix such γ ∈ (0, 1) for the rest of the proof, and let n(η) , d(1/η)βe, I , # { i ∈ [n(η)] :
|Z≤u(η)i | > u(η)1−γ } . Then
P (∣∣Z≤u(η)1 + · · ·+ Z≤u(η)n(η) ∣∣ > v(η))
= j−1∑ i=0 P (∣∣Z≤u(η)1 + · · ·+ Z≤u(η)n(η) ∣∣ > v(η), I = i)︸ ︷︷ ︸
,(I)
+P (∣∣Z≤u(η)1 + · · ·+ Z≤u(η)n(η) ∣∣ > v(η), I ≥ j)︸ ︷︷ ︸
,(II)
Note that since |Z≤u(η)i | < u(η), (I) ≤ ( n(η)
i ) · P (∣∣Z≤u(η)1 + · · ·+ Z≤u(η)n(η)−i∣∣ > v(η)− iηu(η)η , |Z≤u(η)i | ≤ u(η)1−γ ∀i ∈ [n(η)− i])
≤ n(η)i · P (∣∣Z≤u(η)1−γ1 + · · ·+ Z≤u(η)1−γn(η)−i ∣∣ > v(η)− iηu(η)η )
≤ n(η)i · P (∣∣Z≤u(η)1−γ1 + · · ·+ Z≤u(η)1−γn(η)−i ∣∣ > v(η)2η ) (E.55)
where the last inequality follows from eq. (E.52). First, since EZ1 = 0, we have∣∣EZ≤u(η)1−γ1 ∣∣ = ∣∣EZ>u(η)1−γ1 ∣∣ =
∫ ∞ u(η)1−γ P(|Z1| > x)dx ∈ RV(α−1)(1−γ)(1−∆)(η).
Therefore, for all η > 0 that are sufficiently small,
|EZ≤u(η) 1−γ 1 + · · ·+ EZ ≤u(η)1−γ n(η)−i |
≤n(η) · η(α−1)(1−∆)(1−2γ) ≤ 2η(α−1)(1−∆)(1−2γ)−β
≤(1/η)(1−∆)(1−2γ) due to eq. (E.53)
≤v(η) 4η due to ∆̃/2 ≤ ∆ in eq. (E.50) and 1− γ < 1.
If we let Yn = Z ≤u(η)1−γ n − EZ≤u(η) 1−γ
n and plug the bound above back into eq. (E.55), then (for all η > 0 that are sufficiently small)
(I) ≤ n(η)i · P(|Y1 + · · ·+ Yn(η)−i| > v(η)
4η )
≤ n(η)i exp ( −
2
16 · 1/η 2−2∆̃ 2 ( n(η)− i ) E|Y1|2 + 23δ1−γ · (1/η)(1−∆)(1−γ) · 4/η 1−∆̃
) (E.56)
where the last inequality is obtained from Bernstein’s inequality. Note that from Karamata’s theorem,
E|Y1|2 = var(Z≤u(η) 1−γ 1 ) ≤ E|Z ≤u(η)1−γ 1 |2
≤ ∫ u(η)1−γ
0
2xP(|Z1| > x)dx ∈ RV−(1−∆)(1−γ)(2−α)(η).
Now note that
• In case that α < 2, for all η > 0 that are sufficiently small, we have (using eq. (E.50)) 2 ( n(η)− i ) E|Y1|2 ≤ (1/η)β+(2−α)(1−∆) < (1/η)2(1−∆)
⇒ 1/η 2−2∆̃ 2 ( n(η)− i ) E|Y1|2 ≥ 1/η∆;
• In case that α ≥ 2, for all η > 0 that are sufficiently small, 2 ( n(η)− i ) E|Y1|2 < 1/ηβ+ ∆ 2
and we know that β + ∆2 < 2 − 2∆̃ due to 2 − β > 2∆ and 2∆̃ ≤ ∆ (see eq. (E.48)eq. (E.50));
• Since γ > 0 and 2∆̃ ≤ ∆, we know that
(1−∆)(1− γ) + (1− ∆̃) < 2− 2∆̃.
Therefore, it is easy to see that the R.H.S. of eq. (E.56) decays at a geometric rate as η tends to zero, hence o(ηN ). On the other hand,
(II) ≤ P(I ≥ j) ≤ ( n(η)
j
) · P ( |Z≤u(η)i | > u(η) 1−γ ∀i = 1, . . . , j )
≤ n(η)j · P ( |Z≤u(η)1 | > u(η)1−γ )j ,
which is regularly varying w.r.t. η with index ( α(1 − γ)(1 −∆) − β ) j. Therefore, for all η > 0 sufficiently small,
(II) ≤ η ( α(1−2γ)(1−∆)−β ) j < ηN due to eq. (E.54).
Collecting results above, we have established that P ( η ∣∣Z≤u(η)1 + · · ·+ Z≤u(η)n(η) ∣∣ > v(η)) = o(ηN ).
The conclusion of the lemma now follows from Etemadi’s theorem.
Now consider the following setting. Let us fix some positive integer N and β ∈ (1, 2 ∧ α). Then we can find some positive integer j such that (α− β)j > N . Meanwhile, given any > 0, we will have − jδ ≥ /2 for all δ > 0 sufficiently small. Therefore, by applying Lemma E.11 with ∆ = ∆̃ = 0 (hence u(η) = δ/η, v(η) = ) and β, j,N, , δ as described here, we immediately get the following result. Lemma E.12. Given any β ∈ (1, α ∧ 2), > 0, and N > 0, the following holds for any sufficiently small δ > 0:
P (
max j=1,2,··· ,d(1/η)βe
η|Z≤δ/η1 + · · ·+ Z ≤δ/η j | >
) = o(ηN )
as η ↓ 0.
Using results and arguments above, we are able to illustrate the typical behavior of the SGD iterates Xηn in the following two scenarios. First, we show that, when starting from most parts in the attraction field Ω, the SGD iterates Xηn will most likely return to the neighborhood of the local minimum within a short period of time without exiting Ω. Given that there are only finitely many attraction fields on f , it is easy to see that the key technical tool Lemma E.1 follows immediately from the next result. Lemma E.13. For sufficiently small > 0, the following claim holds:
lim η↓0 sup x∈Ω:|x−s−|∧|x−s+|>
Px ( Xηn ∈ Ω ∀n ≤ Treturn(η, ), and Treturn(η, ) ≤ ρ( )/η ) = 1
where the stopping time involved is defined as
Treturn(η, ) = ∆ min{n ≥ 0 : Xηn(x) ∈ [−2 , 2 ]}
the function t̂( ) is defined in eq. (E.38), and the function ρ(·) is defined as ρ( ) = 3̄ 0.9cL−∧cL+ + 2t̂( )
Proof. Throughout, we only consider small enough so that Lemma E.8 could hold. Also, fix some N > 0,∆α ∈ (0, α− 1) and β ∈ (1, α). Let σ(x, η) =∆ min{n ≥ 0 : Xηn /∈ Ω}. Without loss of generality, we assume Ω = [−L, s+) and x < 0 (so reflection at −L) is a possibility. Any other case can be addressed similarly as shown below.
From Lemma E.4 and the regular varying nature of H(·), we have, for any , δ > 0,
P(T η1 (δ) ≤ ρ( )/η) ≤ ηα−1−∆α (E.57)
for any sufficiently small η.
Let T̃ ηescape(x) be the stopping time defined in Lemma E.8. From eq. (E.44),eq. (E.45),eq. (E.57) and Lemma E.12, we know that
sup x∈[−L,−L+̄]
P ( T̃ ηescape(x) < σ(x, η), T̃ η escape(x) ≤ 3̄
0.9cL−η and Xη T̃ηescape (x) ∈ [−L+ ̄,−L+ 2̄] ) ≥1− ηN − ηα−1−∆α (E.58)
for all sufficiently small η.
Next, we focus on x ∈ Ω such that |x− s−| ∧ |x− s+| > and x ≥ −L+ ̄. We start by considering the time it took for the (deterministic) gradient descent process yηn(x) to return to [−1.5 , 1.5 ]. From
the definition of t̂( ) in eq. (E.38) and Lemma E.6, we know that for η small enough such that η exp(2Mt̂( )) < 0.5 , we have
min{n ≥ 0 : yηn(x) ∈ [−1.5 , 1.5 ]} ≤ 2t̂( )/η.
Now consider event A(d(1/η)βe, η, 4 exp(2Mt̂( )) , δ) (see definition in eq. (E.42)). From Lemma E.12, we know that for any sufficiently small δ, we have
P (( A(d(1/η)βe, η, 4 exp(2Mt̂( )) , δ) )c) = o(ηN ). (E.59)
Combining this result with eq. (E.57)eq. (E.59) and Lemma E.9, we get
sup x∈Ω:|x−s−|∧|x−s+|> ,x≥−L+̄
Px ( Treturn(η, ) < σ(x, η), Treturn(η, ) ≤ 2t̂( )/η ) (E.60)
≥ 1− ηN − ηα−1−∆α (E.61)
for any sufficiently small η. To conclude the proof, we only to combine strong Markov property (at T̃ ηescape) with bounds in eq. (E.58)eq. (E.61).
In the next result, we show that, once entering a −small neighborhood of the local minimum, the SGD iterates will most likely stay there until the next large jump.
Lemma E.14. Given N0 > 0, the following claim holds for any , δ > 0 that are sufficiently small:
sup x∈[−2 ,2 ]
P ( ∃n < T η1 (δ) s.t. |Xηn(x)| > 3 ) = o(ηN0)
as η ↓ 0.
Proof. Fix small enough such that 3 < 0 (see Assumption 1 for the constant 0). Also, fix some ∆α ∈ (0, 1), β ∈ (1, α), N > α+ ∆α− β +N0. Due to Lemma E.12, for any δ sufficiently small, we will have
P (
max j=1,2,··· ,d(1/η)βe
η|Z≤,δ1 + · · ·+ Z ≤,δ j | >
exp(2M)
) = o(ηN ). (E.62)
Fix such δ > 0. We now show that the desired claim is true for the chosen , δ.
First of all, from Lemma E.3, we know the existence of some θ > 0 such that
P(T η1 (δ) > 1/η α+∆α) = o(exp(−1/ηθ)). (E.63)
Next, let us zoom in on the first d(1/η)βe SGD iterates. For any η small enough, we will have yηn(x) ∈ [−2 , 2 ] for any n ≥ 1 and y η d(1/η)βe(x) ∈ [− , ] given x ∈ [−2 , 2 ]. From now on we
only consider such η. Due to Lemma E.9, we know that on event {
maxj=1,2,··· ,d(1/η)e η|Z≤,δ1 + · · ·+ Z≤,δj | > exp(2M) } , we have
|Xηn(x)| ≤ 3 ∀n ≤ d1/ηβe ∧ (T η 1 (δ)− 1)
and on event {
maxj=1,2,··· ,d(1/η)e η|Z≤,δ1 + · · ·+ Z ≤,δ j | > exp(2M) } ∩ {T η1 (δ) > d(1/η)βe}, we
have Xη Tη1 (δ) (x) ∈ [−2 , 2 ]. Now by repeating the same argument inductively for d1/ηα+∆α−βe times, we can show that on event
{∀i = 1, 2, · · · , d 1 ηα+∆α−β e, max j=1,··· ,d(1/η)βe η|Z≤,δ id(1/η)βe+1 + · · ·+ Z ≤,δ id(1/η)βe+j | > exp(2M) },
we have |Xηn(x)| ≤ 3 ∀n ≤ 1/ηα+∆α ∧ (T η 1 (δ) − 1). To conclude the proof, we only need to combine this fact with eq. (E.62).
We introduce a few concepts that will be crucial in the analysis below. Recall the definition of perturbed ODE x̃η in eq. (E.32)-eq. (E.34) (note that we will drop the notational dependency on learning rate η when we choose η = 1). Consider the definition of the following two mappings from where w = (w1, · · · , wl∗) is a sequence of real numbers and t = (t1, t2, · · · , tl∗) with 0 = t1 < t2 < t3 < · · · as
h(w, t) = x̃(tl∗ , 0; t,w).
Next, define sets (for any ∈ (−̄, ̄))
E( ) = {(w, t) ⊆ Rl ∗ × ( R+ )l∗−1 : h(w, t) /∈ [s− − , s+ + ]}. (E.64)
We add a few remarks about the two types of sets defined above.
• Intuitively speaking, E( ) contains all the perturbations (with times and sizes) that can send the ODE out of the current attraction field (allowing for some error with size );
• From the definition of t̄, δ̄ in eq. (E.35)eq. (E.36) and Corollary E.7, one can easily see that for a fixed ∈ (−̄, ̄),
(w, t) ∈ E( )⇒ |wj | > δ̄, tj − tj−1 ≤ t̄ ∀j;
• Lastly, E( ) are open sets due to f ∈ C2.
Use Leb+ to denote the Lebesgue measure restricted on [0,∞), and define (Borel) measure να with density on R\{0}:
να(dx) = 1{x > 0} αp+ xα+1 + 1{x < 0} αp− |x|α+1
where α > 1 is the regular variation index for the distribution of Z1 and p−, p+ ∈ (0, 1) are constants in Assumption 2. Now we can define a Borel measure µ on Rl∗ × ( R+ )l∗−1 as product measure
µ = (να) l∗ × (Leb+)l ∗−1. (E.65)
Due to remarks above, one can see that for ∈ (−̄, ̄), we have µ(E( )) <∞. We are now ready to analyze a specific type of noise Zn.
Definition E.1. For any n ≥ 1 and any ∈ (−̄, ̄), δ ∈ (0, b ∧ δ̄), η > 0, we say that the jump Zn has ( , δ, η)-overflow if
• η|Zn| > δ;
• In the set {n + 1, · · · , n + 2dl∗t̄/ηe}, there are at least (l∗ − 1) elements (ordered as n < t2 < t3 < · · · < tl∗ ) such that η|Zti | > δ for any i = 2, · · · , l∗;
• Let t1 = n and t′i = ti − ti−1 for any i = 2, · · · , l∗, wi = ηZi for any i = 1, · · · , l∗, for real sequence w = (w1, w2, · · · , wl∗) and a sequence of positive number t = (η(ti−n))l ∗
i=2, we have
(w, t) ∈ E( ).
Moreover, if Zn has ( , δ, η)−overflow, then we call h(w, t) as its ( , δ, η)−overflow endpoint.
Due to the iid nature of (Zj)j≥1, let us consider an iid sequence (Vj)j≥0 where the sequence has the same law of Z1. Note that for any fixed n ≥ 1, the probability that Zn has ( , δ, η)overflow is equal to the probability that V0 has( , δ, η)-overflow. More specifically, we know that P(η|V0| > δ) = H(δ/η), and now we focus on conditional probability admitting the following form:
p( , δ, η) = P ( V0 has ( , δ, η)-overflow ∣∣∣ η|V0| > δ). (E.66) For any open interval A = (a1, a2) such that A ∩ [s− + ̄, s+ − ̄] = ∅, we also define
p( , δ, η;A) = P ( V0 has ( , δ, η)-overflow and the endpoint is in A ∣∣∣ η|V0| > δ). (E.67)
Lemma E.15. For any ∈ (−̄, ̄), δ ∈ (0, b ∧ δ̄), and any open interval A = (a1, a2) such that |a1| ∧ |a2| > r − ̄ and |a1| 6= L, |a2| 6= L, we have
lim η↓0
p( , δ, η;A) δα ( H(1/η)
η )l∗−1 = µ(E( ) ∩ h−1(A)) where µ is the measure defined in eq. (E.65), and p(·, ·, ·;A) is the conditional probability defined in eq. (E.67).
Proof. Let us start by fixing some notations. Let T1 = 0, and define stopping times Tj = min{n > Tj−1 : η|Vn| > δ} and inter-arrival times T ′j = Tj − Tj−1 for any j ≥ 1, and large jump Wj = VTj for any j ≥ 0. Note that: first of all, the pair (T ′i ,Wi) is independent of (T ′j ,Wj) whenever i 6= j; besides, Wj and T ′j are independent for all j ≥ 1.
Define the following sequence (of random elements) w = (w1, · · · , wl∗) and t = (t1, · · · , tl∗) by wj = ηWj , tj = ηTj .
If V0 has ( , δ, η)-overflow, then the following two events must occur:
• T ′j ≤ 2t̄/η for any j = 2, · · · , l∗;
• η|Wj | > δ̄ for any j = 1, 2, · · · , l∗;
• (w, t) ∈ E( )
Therefore, for sufficiently small η, we now have p( , δ, η)
= ( P(T ′1 ≤ 2t̄/η) )l∗−1 · ∫ 1 { (w, t) ∈ E( ) } · P(ηW1 = dw1|η|W1| > δ) · · ·P(ηWl∗ = dwl∗ | η|Wl∗ | > δ) · P(ηT ′2 = dt′2|ηT ′2 ≤ 2t̄) · · ·P(ηT ′l∗ = dt′l∗ |ηT ′l∗ ≤ 2t̄)
= ( P(T ′1 ≤ 2t̄/η) )l∗−1 ·Qη,δ ( E( ) ∩ h−1(A) ) (E.68)
where Qη,δ is the Borel-measurable probability measure on Rl ∗ × ( R+ )l∗−1 induced by a sequence
of independent random variables (W ↑1 (η, δ), · · · ,W ↑ l∗(η, δ), T ↑ 2 (η, δ), · · · , T ↑ l∗(η, δ)) such that
• For any i = 1, · · · , l∗, the distribution ofW ↑i (η, δ) follows from P ( ηW1 ∈ · ∣∣∣ η|Wl∗ | > δ); • For any i = 2, · · · , l∗, the distribution of T ↑i (η, δ) follows from P ( ηT1 ∈ ·
∣∣∣ ηT1 ≤ 2t̄); • Qη,δ(·) = P ( (ηW ↑1 (η, δ), · · · , ηW ↑ l∗(η, δ), ηT ↑ 2 (η, δ), · · · , η ∑l∗ j=2 T ↑ j (η, δ)) ∈ · ) .
Now we study the weak convergence of W ↑1 , T ↑ 1 :
• Due to the regularly varying nature of distribution of Z1 (hence for W1), we know that: for any x > δ,
lim η↓0
P ( ηW1 > x ∣∣∣ η|Wl∗ | > δ) = p+ δα xα , lim η↓0 P ( ηW1 < −x ∣∣∣ η|Wl∗ | > δ) = p− δα xα ;
therefore, W ↑1 (η, δ) weakly converges to a (randomly signed) Pareto RV that admits the density
να,δ(dx) = 1{x > 0}p+ αδα
xα+1 + 1{x < 0}p−
αδα
|x|α+1 as η ↓ 0;
• For any x ∈ [0, 2t̄], since limη↓0bx/ηcH(δ/η) = 0, it is easy to show that
lim η↓0
1− (1−H(δ/η))bx/ηc
bx/ηcH(δ/η) = 1;
therefore, we have (for any x ∈ (0, 2t̄])
P(ηT1 ≤ x | ηT1 ≤ 2t̄) = 1− (1−H(δ/η))bx/ηc
1− (1−H(δ/η))b2t̄/ηc → x 2t̄
as η ↓ 0, which implies that T ↑1 converges weakly to a uniform RV on [0, 2t̄].
Let us denote the weak limit of measure Qη,δ as µδ,2t̄. In the discussion before the Lemma we have shown that, for any (w, t) ∈ E( ) (with δ ∈ (0, δ̄)), we have |wi| ≥ δ̄ and |t′i| ≤ 2t̄; since we require δ < δ̄, by definition of measures µ and µδ,2t̄ we have
µδ,2t̄ ( E( ) ∩ h−1(A) ) = δαl ∗ (2t̄)l∗−1 · µ ( E( ) ∩ h−1(A) ) .
For simplicity of notations, we let E( , A) =∆ E( ) ∩ h−1(A). By definition of the set E( ), we have (recall that A is an open interval (a1, a2) that does not overlap with [s− + ̄, s+ − ̄])
E( , A) = h−1 ( (−∞, s− − ) ∪ (s+ + ,∞) ) ∩ h−1 ( (a1, a2) ) = h−1 (( (−∞, s− − ) ∪ (s+ + ,∞) ) ∩ (a1, a2)
) = h−1 ( F ( , a1, a2)
) where F ( , a1, a2) = ∆ ( (−∞, s− − ) ∪ (s+ + ,∞) ) ∩ (a1, a2). Meanwhile, it is easy to see that h is a continuous mapping, hence
(w, t) ∈ ∂E( , A)⇒ h(w, t) ∈ {s− + , s+ − , a1, a2}.
Fix some s with s 6= ±L, |s| > (l∗−1)b+ ̄. For any fixed real numbers t2, · · · , tl∗−1, w1, · · · , wl∗ , if h(w1, · · · , wl∗ , t2, · · · , tl∗−1, t) = s, then since x̃(tl∗ − 1, 0;w1, · · · , wl∗ , t2, · · · , tl∗−1, t) ∈ [s− b, s+ b], due to Assumption 1 (in particular, there is no point x on this interval with |f ′(x)| ≤ c0 ), there exists at most one possible t that makes h(w1, · · · , wl∗ , t2, · · · , tl∗−1, t) = s. Therefore, let W ∗j be iid RVs from law να,δ defined above, and (T ∗,′ j )j≥2 be iid RVs from Unif[0, 2̄t], T ∗ 0 =
0, T ∗k = ∑k j=2 T ∗,′ j . By conditioning on all W ∗ j and all T ∗,′ 2 , · · · , T ∗,′ l∗−1, we must have
P ( h(W ∗1 , · · · ,W ∗j , T ∗2 , · · · , T ∗l∗) = s ∣∣∣W ∗1 = dw1, · · · ,W ∗l∗ = dwl∗ , T ∗,′2 = dt2, · · · , T ∗,′ l∗−1 = dtl∗−1 ) = 0 (E.69)
which implies
P ( h(W ∗1 , · · · ,W ∗j , T ∗2 , · · · , T ∗l∗) = s ) = 0
hence
µ ( ∂E( , A) ) = 0.
By Portmanteau theorem (see Theorem 2.1 of Billingsley (2013)) we have
lim η↓0 Qη,δ(E( , A)) = µδ,2t̄(E( , A)).
Collecting the results we have and using eq. (E.68), we can see that
lim sup η↓0 p( , δ, η;A)( H(1/η)
η
)l∗−1 δα
= lim sup η↓0
(2t̄)l ∗−1 · p( , δ, η;A) δαl∗ · ( P(T ′1 ≤ 2t̄/η)
)l∗−1 · (δα2t̄ · P(T ′1 ≤ 2t̄/η)H(1/η)/η )l ∗−1
≤ lim sup η↓0
(2t̄)l ∗−1 · p( , δ, η;A) δαl∗ · ( P(T ′1 ≤ 2t̄/η) )l∗−1 · lim sup η↓0
(δα 2t̄ · P(T ′ 1 ≤ 2t̄/η) H(1/η)/η )l∗−1
≤µ(E( , A)) · lim sup η↓0 (δα 2t̄ · P(T ′ 1 ≤ 2t̄/η) H(1/η)/η )l∗−1 .
Fix some κ > 1. From Lemma E.4 and the regularly varying nature of function H , we get
lim sup η↓0 (δα 2t̄ · P(T ′ 1 ≤ 2t̄/η) H(1/η)/η )l∗−1 ≤ κl ∗−1 lim sup η↓0 (δα 2t̄ · 2t̄H(δ/η)/η H(1/η)/η )l∗−1 = κl ∗−1.
Due to the arbitrariness of κ > 1, we have established that
lim sup η↓0 p( , δ, η;A)( H(1/η)
η
)l∗−1 δα ≤ µ(E( )).
The lower bound can be shown by an argument symmetric to the one for upper bound.
The following result is an immediate corollary of Lemma E.15. Corollary E.16. For any ∈ (−̄, ̄), δ ∈ (0, b ∧ δ̄), we have
lim η↓0
p( , δ, η) δα ( H(1/η)
η )l∗−1 = µ(E( )) where µ is the measure defined in eq. (E.65), and p(·, ·, ·) is the conditional probability defined in eq. (E.66).
Define the following stopping times:
σ(η) = min{n ≥ 0 : Xηn /∈ Ω}; (E.70) R( , δ, η) = min{n ≥ T η1 (δ) : Xηn ∈ [−2 , 2 ]}. (E.71)
σ indicate the time that the iterates escape the current attraction field, while R denotes the time the SGD iterates return to a small neighborhood of the local minimum after first exit from this small neighborhood. In the next few results, we study the probability of several atypical scenarios when SGD iterates make attempts to escape Ω or return to local minimum after the attempt fails. First, we show that, when starting from the local minimum, it is very unlikely to escape with less than l∗ big jumps. Lemma E.17. Given ∈ (0, ̄), N > 0, the following claim holds for any sufficiently small δ > 0:
sup x∈[−2 ,2 ]
Px ( σ(η) < R( , η), σ(η) < T ηl∗(δ) ) = o(ηN )
as η ↓ 0.
Proof. Based on the given > 0, fix some ̃ = 4 exp(2Mt̂( )) . Recall the definition of t̂( ) in eq. (E.38).
First, using Lemma E.14, we know that for sufficiently small δ, we have
sup x∈[−2 ,2 ]
P ( A×1 ( , δ, η) ) = o(ηN ) (E.72)
where
A×1 ( , δ, η) = { ∃n < T η1 (δ) s.t. |Xηn(x)| > 3 } .
Define event
A×2 (̃, δ, η) = ∆
{ ∃j = 2, · · · , l∗ s.t. max
k=1,2,··· ,Tηj (δ)−T η j−1(δ)−1
η|Z≤δ,η Tηj−1(δ)+1 +· · ·+Z≤δ,η Tηj−1(δ)+k
| > ̃ } .
From Lemma E.12, we know that for sufficiently small δ > 0, P ( A×2 (̃, δ, η) ) = o(ηN ). (E.73)
From now on, we only consider such δ that eq. (E.72)eq. (E.73) hold. On event ( A×1 ∪A × 2 )c ∩ {σ(η) < R( , η)} ∩ {σ(η) > T η1 (δ)}, we must have σ(η) > T η 1 (δ) and
T η2 (δ) ∧ σ(η)− T η 1 (δ) < 2t̂( )/η.
Otherwise, due to Lemma E.6 and E.9, we know that at step t̃ = T η1 (δ) + bt̂( )/ηc, we have
|Xη t̃ | < 2 , and |Xηn| ≤ ̄ ∀n ≤ t̃
for any sufficiently small η. By repeating this argument inductively, we obtain the following result: define
J = min{j = 1, 2, · · · : σ(η) ∈ [T ηj (δ), T η j+1(δ))},
then on event ( A×1 ∪A × 2 )c ∩ {σ(η) < R( , η), σ(η) < T ηl∗(δ)}, we must have
T ηj (δ) ∧ σ(η)− T η j−1(δ) ∧ σ(η) < 2t̂( )/η ∀j = 2, 3, · · · , J. (E.74)
Furthermore, using this bound and Lemma E.9, we know that on event ( A×1 ∪ A × 2 )c ∩ {σ(η) < R( , η), σ(η) < T ηl∗(δ)},
• |Xη Tηj (δ) | ≤ |Xη Tηj−1(δ) |+ b+ + ̄ for all j = 2, 3, J − 1,
• |Xησ(η)| ≤ |X η TηJ−1(δ) |+ + ̄
However, this implies
|Xησ(η)| ≤ l ∗(̄+ ) + (l∗ − 1)b < r
and contradicts the definition of σ(η). In summary,
sup x∈[−2 ,2 ]
Px ( σ(η) < R( , η), σ(η) < T ηl∗(δ) ) ≤ P ( A×1 ( , δ, η) ∪A × 2 (̃, δ, η) ) = o(ηN ).
The following two results follow immediately from the proof above, especially the inductive argument leading to bound eq. (E.74), and we state them without repeating the deatils of the proof.
Corollary E.18. Given ∈ (0, ̄), N > 0, the following claim holds for any sufficiently small δ > 0:
sup x∈[−2 ,2 ]
Px ( T ηl∗(δ) ≤ σ(η) ∧R( , η), and ∃j = 2, 3, · · · , l ∗ s.t. T ηj (δ)− T η j−1(δ) > 2t̂( )/η ) =o(ηN ) as η ↓ 0.
Corollary E.19. Given ∈ (0, ̄), N > 0, the following claim holds for any sufficiently small δ > 0:
sup x∈[−2 ,2 ]
Px ( R( , η) < T ηl∗(δ) ∧ σ(η), R( , η)− T η 1 (δ) > 2l ∗t̂( )/η ) = o(ηN )
as η ↓ 0.
In the next result, we show that, if the inter-arrival time between some large jumps are too long, or some large jumps are still not large enough, then it is very unlikely that the SGD iterates could escape at the time of l∗−th large jump (or even get close enough to the boundary of the attraction field). Lemma E.20. Given ∈ (0, ̄) and any N > 0, the following holds for all δ > 0 that are sufficiently small:
sup x∈[−2 ,2 ]
Px(B×2 ( , δ, η)) = o(η N ).
where
B×2 ( , δ, η) = {T η l∗(δ) ≤ σ(η) ∧R( , η)} ∩ { ∃j = 2, 3, · · · , l∗ s.t. T ηj (δ)− T η j−1(δ) > t̄/η
or ∃j = 1, 2, · · · , l∗ s.t. η|W ηj (δ)| ≤ δ̄ } ∩ {|Xη
Tη l∗ | ≥ r − ̄}.
Proof. Let A×1 , A × 2 be the events defined in the proof of Lemma E.17. Based on the given > 0, fix some ̃ = 4 exp(2Mt̂( )) .
Let J = min{j = 2, 3, · · · : T ηj (δ)−T η j−1(δ) > t̄/η}. On event ( A×1 ( , δ, η)∪A × 2 (̃, δ, η) )c ∩{J ≤ l∗}, from Lemma E.6 and E.9 and the definition of constant t̄, we know that
• |Xη Tηj (δ) | ≤ |Xη Tηj−1(δ) |+ b+ + ̄ for all j = 2, 3, J − 1;
• |Xη TηJ (δ) | ≤ 2̄
• |Xηn| < s− ̄ ∀n ≤ T η J (δ)
Now starting from step T ηJ (δ), by using Lemma E.6 and E.9 again one can see that
• |Xη Tηj (δ) | ≤ |Xη Tηj−1(δ) |+ b+ + ̄ for all j = J + 1, · · · , l∗.
Combining these results, we have that |Xη Tη l∗ | < r− ̄ on event
( A×1 ( , δ, η)∪A × 2 (̃, δ, η) )c ∩ {J ≤
l∗}. Next, define J ′ = min{j = 1, 2, · · · ; η|W ηj (δ)| ≤ δ̄}. Similarly, on event ( A×1 ( , δ, η) ∪
A×2 (̃, δ, η) )c ∩ {J > l∗} ∩ {J ′ ≤ l∗}, using Lemma E.6 and E.9 again one can see that
• |Xη Tηj (δ) | ≤ |Xη Tηj−1(δ) |+ b+ + ̄ for all j = 1, 2, · · · , l∗, j 6= J ′;
• |Xη TηJ (δ) | ≤ |Xη TηJ−1(δ) |+ δ̄ + + ̄ for all j = 1, 2, · · · , l∗, j 6= J ′.
Since δ̄ ∈ (0, ̄), we have |Xη Tη l∗ | < r − ̄ on this event.
In summary, the following bound
sup x∈[−2 ,2 ]
Px(B×2 ) ≤ P(A × 1 ( , δ, η) ∪A × 2 (̃, δ, η)) = o(η N )
holds for any δ that is sufficiently small, which is established in Lemma E.17. This conclude the proof.
In the next lemma, we show that, starting from the local minimum, it is unlikely that the SGD iterates will be right at the boundary of the attraction field after l∗ large jumps. Recall that there are nmin attraction fields on f , and excluding s0 = −∞, snmin =∞ the remaining points s1, · · · , snmin−1 are the boundaries of the attraction fields.
Lemma E.21. There exists a function Ψ(·) : R+ 7→ R+ satisfying lim ↓0 Ψ( ) = 0 such that the following claim folds. Given ∈ ( 0, ̄/(3ρ̄+ 3ρ̃+ 9) ) , we have
lim sup η↓0
supx∈[−2 ,2 ] Px(B × 3 ( , δ, η))( H(1/η)/η )l∗−1 ≤ δαΨ( )
for all δ sufficiently small, where ρ̄ and ρ̃ are the constants defined in Corollary E.7 and E.10, and the event is defined as
B×3 ( , δ, η) = { T ηl∗(δ) ≤ σ(η) ∧R( , η) } ∩ { ∃k ∈ [nmin − 1] such that XηTη
l∗ (δ) ∈ [sk − , sk + ]
} .
Proof. Let A×1 , A × 2 be the events defined in the proof of Lemma E.17. Based on the given > 0, fix some ̃ = 4 exp(2Mt̂( )) . Fix some N > αl∗.
Choose δ small enough so that claim in Lemma E.20 holds for the prescribed. Using the same arguments in Lemma E.20, we have the following inclusion of events:
B×3 ( , δ, η) ∩ ( A×1 ( , δ, η) ∪A × 2 (̃, δ, η) )c ⊆ { ∀j = 2, 3, · · · , l∗, T ηj (δ)− T η j−1(δ) ≤ t̄/η } ∩ { ∀j = 1, 2, 3, · · · , l∗, η|W η1 (δ)| > δ̄ } .
Therefore, on event B×3 ( , δ, η)∩ ( A×1 ( , δ, η)∪A × 2 (̃, δ, η) )c , we can apply Corollary E.7 and E.10
and conclude that ZTη1 (δ) has (−̄, δ, η)−overflow, and its (−̄, δ, η)−overflow endpoint lies
(sk − 3(ρ̄+ ρ̃+ 3) , sk + 3(ρ̄+ ρ̃+ 3) )
for some k ∈ [nmin − 1]. Using Lemma E.15 and Corollary E.16, we have that (for any sufficiently small η)
P ( B×3 ( , δ, η) ∩ ( A×1 ( , δ, η) ∪A × 2 (̃, δ, η) )c) ≤δα (H(1/η) η )l∗−1 · nmin−1∑ k=1 µ ( E(−̄) ∩ h−1 ( (sk − ̂, sk + ̂)
)) where ̂ = 3(ρ̄+ ρ̃+ 3) . Besides, as established in the proof of Lemma E.17, we have
P ( A×1 ( , δ, η) ∪A × 2 (̃, δ, η) ) = o(ηN )
for all sufficiently small δ. In conclusion, we only need to choose
Ψ( ) = nmin−1∑ k=1 µ ( E(−̄) ∩ h−1 ( (sk − 3(ρ̄+ ρ̃+ 3) , sk + 3(ρ̄+ ρ̃+ 3) ) )) .
To conclude the proof, just note that by combining the continuity of measure with the conditional probability argument leading to eq. (E.69), we can show that lim ↓0 Ψ( ) = 0.
Lastly, we establish the lower bound for the probability of the most likely way for SGD iterates to exit the current attraction field: making l∗ large jumps in a relatively short period of time. Recall that ̄ is the fixed constant in eq. (E.26)-eq. (E.29).
Lemma E.22. Given ∈ (0, ̄/3), it holds for any sufficiently small δ > 0 such that
lim inf η↓0 inf |x|≤2 Px(A◦( , δ, η))( H(1/η)/η )l∗−1 ≥ c∗δα where the event is defined as
A◦( , δ, η) =∆ { σ(η) < R( , η), σ(η) = T ηl∗(δ), X
η Tη l∗ /∈ [s− − , s+ + ] } ∩ { T ηj (δ)− T η j−1(δ) ≤ ̄
2M d1/ηe ∀j = 2, 3, · · · , l∗ } and the constant
c∗ = 1
2 (
1
2b )l ∗α(
̄
4M )l ∗−1
is strictly positive and does not vary with , δ.
Proof. Let A×1 , A × 2 be the events defined in the proof of Lemma E.17. Fix some N such that N > αl∗. Based on the given > 0, fix some ̃ = 4 exp(2Mt̂( ))
. We only consider δ < M . Furthermore, choose δ small enough so that eq. (E.72) and eq. (E.73) hold for the chosen N and . Also, we only consider η small enough so that ηM < b ∧ ̄. Due to eq. (E.26)-eq. (E.29), we can, without loss of generality, assume that r = s+, and in this case we will have
l∗b− 100l∗̄ > s+ + 100l∗̄. Under this assumption, we will now focus on providing a lower bound for the following event that describes the exit from the right side of Ω (in other words, by crossing s+)
A◦→( , δ, η) = ∆ { σ(η) < R( , η), σ(η) = T ηl∗(δ), X
η Tη l∗ > s+ + } ∩ { T ηj (δ)− T η j−1(δ) ≤ ̄
2M d1/ηe ∀j = 2, 3, · · · , l∗
} .
First, define event A◦3(δ, η) = { W ηj (δ) ≥ 2b ∀j = 1, · · · , l ∗, T ηj (δ)− T η j−1(δ) ≤ ̄
2M d1/ηe ∀j = 2, · · · , l∗
} ,
and observe some facts on event A◦3(δ, η) ∩ ( A×1 ( , δ, η) ∪A × 2 (̃, δ, η) )c .
• |Xηk | ≤ 3 ∀n < T η 1 (δ); (due to A × 1 not occurring)
• Xη Tη1 (δ) ∈ [b− 3 , b+ 3 ]; (due to W η1 ≥ 2b and the effect of gradient clipping at step T η 1 ,
as well as the fact that Xη Tη1 −1 ∈ [−3 , 3 ] from the previous bullet point)
• Due to |f ′(·)| ≤M and δ < M , one can see that (for any n ≥ 1) sup
x∈[−L,L] |ηf ′(x)|+ |ηZ≤δ,ηn | ≤ 2ηM ;
this provides an upper bound for the change in SGD iterates at each step, and gives us Xηn ∈ [b− 3 − ̄, b+ 3 + ̄] ∀T η 1 (δ) ≤ n < T η 2 (δ) where we also used T η2 (δ)− T η 1 (δ) ≤ ̄2M d1/ηe
• Therefore, at the arrival time of the second large jump, we must have Xη Tη2 (δ) ≥ 2b− 3 − ̄;
• By repeating the argument above inductively, we can show that (for all j = 1, 2, · · · , l∗) Xηn ∈ [(j − 1)b− 3 − (j − 1)̄, (j − 1)b+ 3 + (j − 1)̄] ∀T η j−1 ≤ n < T η j
Xη Tηj ∈ [jb− 3 − (j − 1)̄, jb+ 3 + (j − 1)̄];
In particular, we know that Xηn ∈ Ω for any n < T η l∗ (so the exit does not occur before T η l∗ ), and at the arrival of the l∗−th large jump, we have (using 3 < ̄) Xη Tη l∗ (δ) ≥ l∗b− l∗̄ > s+ + .
In summary, we have shown that
A◦3(δ, η) ∩ ( A×1 ( , δ, η) ∪A × 2 (̃, δ, η) )c ⊆ A◦→( , δ, η).
To conclude the proof, just notice that (for sufficiently small η)
P ( A◦3(δ, η) ∩ ( A×1 ( , δ, η) ∪A × 2 (̃, δ, η) )c) ≥ P(A◦3(δ, η))− P(A×1 ( , δ, η))− P(A × 2 (̃, δ, η))
≥ P(A◦3(δ, η))− ηN due to eq. (E.72) and eq. (E.73) ≥ (H(2b/η) H(δ/η) )l∗( ̄ 4M H(δ/η)/η )l∗−1 − ηN due to Lemma E.4 ≥ 2c∗δα(H(1/η)/η)l ∗−1 − ηN for all η sufficiently small, due to H ∈ RV−α ≥ c∗δα(H(1/η)/η)l ∗−1.
In order to present the main result of this section, we need to take into account the loss landscape outside of the current attraction field Ω. Recall that there are nmin attraction fields on f . For all the attraction fields different from Ω, we call them (Ω̃k)nmin−1k=1 where, for each k ∈ [nmin − 1], the attraction field Ω̃k = (s−k , s + k ) with the corresponding local minimum located at m̃k. Also, recall that σ(η) is the first time Xηn exits from Ω. Building upon these concepts, we can define a stopping time
τ(η, ) =∆ min{n ≥ 0 : Xηn ∈ nmin−1⋃ k=1 [m̃k − 2 , m̃k + 2 ]} (E.75)
as the first time the SGD iterates visit a minimizer in an attraction field that is different from Ω. Besides, let index Jσ(η) be such that
Jσ(η) = j ⇐⇒ Xησ(η) ∈ Ω̃j ∀j ∈ [nmin − 1]. (E.76)
In other words, it is the label of the attraction field that Xηn escapes to. Lastly, define
λ(η) =∆ H(1/η) ( H(1/η)/η )l∗−1 , (E.77)
νΩ =∆ µ ( E(0) ) , (E.78)
νΩk = ∆ µ ( E(0) ∩ h−1(Ω̃k) ) ∀k ∈ [nmin − 1]. (E.79)
For definitions of the measure µ, set E, and mapping h, see eq. (E.64) and eq. (E.65).
Now we are ready to state Proposition E.23, the most important technical tool in this section. In eq. (E.80) and eq. (E.81), we provide upper and lower bounds for the joint distribution of first exit time σ and the label Jσ indexing the attraction field we escape to; it is worth noticing that the claims hold uniformly for all u > C. In eq. (E.82) and eq. (E.83), we provide upper and lower bounds for the joint distribution of when we first visit a different local minimum (which is equal to τ ) and which one we visit (indicated by Xητ ). The similarity between eq. (E.80) eq. (E.81) and eq. (E.82)eq. (E.83) suggests a strong correlation between the behavior of the SGD iterates at time σ(η) and that of time τ(η, ), and this is corroborated by eq. (E.84): we show that it is almost always the case that τ is very close to σ, and on the short time interval [σ(η), τ(η, )] the SGD iterates stay within the same attraction field.
Proposition E.23. Given C > 0 and some k′ ∈ [nmin − 1], the following claims hold for all > 0 that is sufficiently small:
lim sup η↓0 sup u∈(C,∞) sup x∈[−2 ,2 ]
Px ( νΩλ(η)σ(η) > u, Jσ(η) = k ′ )
≤2C + exp(−(1− C)3u)ν Ω k′ + C
νΩ , (E.80)
lim inf η↓0 inf u∈(C,∞) inf x∈[−2 ,2 ]
Px ( νΩλ(η)σ(η) > u, Jσ(η) = k ′ )
≥− 2C + exp(−(1 + C)3u)ν Ω k′ − C νΩ , (E.81)
lim sup η↓0 sup u∈(C,∞) sup x∈[−2 ,2 ]
Px ( νΩλ(η)τ(η, ) > u, Xητ(η, ) ∈ B(m̃k′ , 2 ) ) ≤4C + exp(−(1− C)3u)ν Ω k′ + C
νΩ , (E.82)
lim inf η↓0 inf u∈(C,∞) inf x∈[−2 ,2 ]
Px ( νΩλ(η)τ(η, ) > u, Xητ(η, ) ∈ B(m̃k′ , 2 ) ) ≥− 4C + exp(−(1 + C)3u)ν
Ω k′ − C νΩ , (E.83)
lim inf η↓0 inf x∈[−2 ,2 ]
Px ( λ(η) ( τ(η, )− σ(η) ) < C, Xηn ∈ Ω̃Jσ(η) ∀n ∈ [σ(η), τ(η, )] ) ≥1− C. (E.84)
Before presenting the proof to Proposition E.23, we make some preparations. First, we introduce stopping times (for all k ≥ 1)
τk( , δ, η) = min{n > τ̃k−1( , δ, η) : η|Zn| > δ} τ̃k( , δ, η) = min{n ≥ τk( , δ, η) : |Xηn| ≤ 2 }
with the convention that τ0( , δ, η) = τ̃0( , δ, η) = 0. The intuitive interpretation is as follows. For the fixed we treat [−2 , 2 ] as a small neighborhood of the local minimum of the attraction field Ω. All the τ̃k partitioned the entire timeline into different attempts of escaping Ω. The interval [τ̃k−1, τ̃k] can be viewed as the k−th attempt. If for σ(η), the first exit time defined in eq. (E.70), we have σ(η) > τ̃k, then we consider the k−th attempt of escape as a failure because the SGD iterates returned to this small neighborhood of the local minimum again without exiting the attraction field. On the other hand, the stopping times τk−1 indicate the arrival time of the first large jump during the k−th attempt. The proviso that τ̃k ≥ τk−1 can be interpreted, intuitively, as that an attempt is considered failed only if, after some significant efforts to exit (for instance, a large jump) has been observed, the SGD iterates still returned to the small neighborhood [−2 , 2 ]. Regarding the notations, we add a remark that when there is no ambiguity we will drop the dependency on , δ, η and simply write τk, τ̃k.
To facilitate the characterization of events during each attempt, we introduce the following definitions. First, for all k ≥ 1, let
jk = ∆ #{n = τk−1( , δ, η), τk−1( , δ, η) + 1, · · · , τ̃k( , δ, η) ∧ σ(η) : η|Zn| > δ}
be the number of large jumps during the k−th attempt. Two implications of this definition:
• First, for any k with σ(η) < τ̃k, we have jk = 0. Note that this proposition concerns the dynamics of SGD up until σ(η), the first time the SGD iterates escaped from Ω, so there is no need to consider an attempt that is after σ(η), and we will not do so in the analysis below;
• Besides, the random variable jk is measurable w.r.t. Fτ̃k∧σ(η), the stopped σ−algebra generated by the stopping time τ̃k ∧ σ(η).
Furthermore, for each k = 1, 2, · · · , let Tk,1( , δ, η) = τk−1( , δ, η) ∧ σ(η), Tk,j( , δ, η) = min{n > Tk,j−1( , δ, η) : η|Zn| > δ} ∧ σ(η) ∧ τ̃k ∀j ≥ 2, Wk,j( , δ, η) = ZTk,j( ,δ,η) ∀j ≥ 1
with the convention Tk,0( , δ, η) = τ̃k−1( , δ, η). Note that for any k ≥ 1, j ≥ 1, Tk,j is a stopping time. Besides, from the definition of jk, one can see that
τ̃k−1 + 1 ≤ Tk,j ≤ τ̃k ∧ σ(η) ∀j ∈ [jk], (E.85) and the sequences ( Tk,j )jk j=1 and ( Wk,j )jk j=1
are the arrival times and sizes of large jumps during the k−th attempt, respectively. Again, when there is no ambiguity we will drop the dependency on , δ, η and simply write Tk,j and Wk,j .
In order to prove Proposition E.23, we analyze the most likely scenario that the exit from Ω would happen. Specifically, we will introduce a series of events with superscript × or ◦, where × indicates that the event is atypical or unlikely to happen and ◦ means that it is a typical event and is likely to be observed before the first exit from the attraction field Ω. Besides, the subscript k indicates that the event in discussion concerns the dynamics of SGD during the k−th attempt. Our goal is to show that for some event A×( , δ, η) its probability becomes sufficiently small as learning rate η tends to 0, so the escape from Ω almost always occurs in the manner described by (A×( , δ, η))c. In particular, the definition of this atypical scenario A× involves the union of some atypical events A×k ,B × k that occur in the k−th attempt. In other words, the intuition of A× is that something abnormal happened during one of the attempts before the final exit.
Here is one more comment for the general naming convention of these events. Events with label A often describe the “efforts” made in an attempt to get out of Ω (such as large noises), while those with label B concern how the SGD iterates return to [−2 , 2 ] (and how this attempt fails). For instance, A×k discusses the unlikely scenario before Tk,l∗ , the arrival of the l
∗−th large jump in this attempt, while B×k in general discusses the abnormal cases after Tk,l∗ and before the return to [−2 , 2 ]. On the other hand, A◦k describes a successful escape during k−th attempt, while B ◦ k means that during this attempt the iterates return to without spending too much time.
Now we proceed and provide a formal definition and analysis of the aforementioned series of events. As building blocks, we inspect the process (Xηn)n≥1 at a even finer granularity, and bound the probability of some events (A×k,i)i≥0, (B × k,i)i≥1 detailing several cases that are unlikely to occur during the escape from or return to local minimum in the k−th attempt. First, for each k ≥ 1, define the event
A×k,0( , δ, η) = ∆ { ∃i = 0, 1, · · · , l∗ ∧ jk s.t.
max j=Tk,i+1,··· ,(Tk,i+1−1)∧τ̃k∧σ(η)
η|Z≤δ,ηTk,i+1 + · · ·+ Z ≤δ,η j | >
3ρ̄+ 3ρ̃+ 3
} . (E.86)
Intuitively speaking, the event characterizes the atypical scenario where, during the k−th attempt, there is some large fluctuations (compared to ̃) between any of the first l∗ large jumps (or the first jk large jumps in case that jk < l∗). Similarly, consider event (for all k ≥ 1)
A×k,1( , δ, η) = ∆ { σ(η) < τ̃k, jk < l ∗ }
(E.87)
that describes the atypical case where the exit occurs during the k−th attempt with less than l∗ large jumps. Next, for all k ≥ 1 we have another atypical event (note that from eq. (E.85) we can see that, for any j ≥ 1, jk ≥ j implies Tk,j ≤ σ(η) ∧ τ̃k)
A×k,2 = ∆ { jk ≥ l∗, ∃j = 2, 3, · · · , l∗ s.t. Tk,j − Tk,j−1 > 2t̂( )/η } . (E.88)
representing the case where we have at least l∗ large noises during the k−th attempt, but for some of the large noise (from the 2nd to the l∗-th), the inter-arrival time is unusually long. Moving on, we consider the following events (defined for all k ≥ 1)
A×k,3 = ∆ { jk < l ∗, τ̃k < σ(η), τ̃k − Tk,1 > 2l∗t̂( )/η }
(E.89)
that describes the atypical case where the k−th attempt failed but the return to the small neighborhood [−2 , 2 ] took unusually long time. The following event also concerns the scenario where there are at least l∗ large noises during the k−th attempt:
A×k,4 = ∆ { jk ≥ l∗, |X η Tk,l∗ | ≥ r − ̄, ∃j = 1, 2, · · · , l∗ s.t. η|Wk,j | ≤ δ̄ } ; (E.90)
specifically, it describes the atypical case where, during this attempt, right after the l∗− large noise the SGD iterate is far enough from the local minimum yet some of the large noises are not that large. Lastly, by defining events
A×k,5 = ∆ { jk ≥ l∗, Tk,l∗ ≤ σ(η) ∧ τ̃k, X η Tk,l∗ ∈ ⋃
j∈[nmin−1]
[sj − , sj + ] } , (E.91)
we analyze an atypical case where the SGD iterates arrive at somewhere too close to the boundaries of Ω at the arrival time of the l∗ large noise during this attempt. As an amalgamation of these atypical scenarios, we let
A×k ( , δ, η) = ∆ 5⋃ i=0 A×k,i( , δ, η). (E.92)
Also, we analyze the probability of some events (B×k )k≥1 that concern the SGD dynamics after the l∗−th large noise during the k−th attempt. Let us define
B×k,1( , δ, η) = ∆ { jk ≥ l∗, X η Tk,l∗ ∈ [s− + , s+ − ], Tk,j − Tk,j−1 ≤ 2 t̂( ) η ∀j = 2, 3, · · · , l∗ } B×k,2( , δ, η) =
∆ {τ̃k − Tk,l∗ > ρ( )/η} ∪ {σ(η) < τ̃k} B×k ( , δ, η) = ∆ B×k,1 ∩ B × k,2 (E.93)
where ρ(·) is the function in Lemma E.13. From the definition of B×k , in particular the inclusion of B×k,2, one can see that the intuitive interpretation of event B × k is that the SGD iterates did not return to local minimum efficiently (or simply escaped from the attraction field) after the l∗−th large noise during the k−th attempt. In comparison, the following events will characterize what would typically happen during each attempt:
A◦k( , δ, η) = ∆ {jk ≥ l∗, σ(η) = Tk,l∗ , X η Tk,l∗ /∈ [s− − , s+ + ],
Tk,j − Tk,j−1 ≤ 2t̂( )
η ∀j = 2, 3, · · · , l∗}, (E.94)
B◦k( , δ, η) = ∆ {σ(η) > τ̃k, τ̃k − Tk,1 ≤
2l∗t̂( ) + ρ( )
η }. (E.95)
Intuitively speaking, A◦k tells us that the exit happened right at Tk,l∗ , the arrival time of the l∗−th large noise during the k−th attempt, and B◦k tells us that the first exit from Ω did not occur during the k−th attempt, and the SGD iterates returned to local minimum rather efficiently. All the preparations above allow use to define
A×( , δ, η) =∆ ⋃ k≥1 ( k−1⋂ i=1 ( A×i ∪ B × i ∪ A ◦ i )c) ∩ (A×k ∪ B×k ). (E.96) We need the next lemma in the proof of Proposition E.23. As mentioned earlier, the takeaway is that A× is indeed atypical in the sense that we will almost always observe (A×)c. Lemma E.24. Given any C > 0, the following claim holds for all > 0, δ > 0 sufficiently small:
lim sup η↓0 sup |x|≤2
Px(A×( , δ, η)) < C.
Proof. We fix some parameters for the proof. First, with out loss of generality we only consider C ∈ (0, 1), and we fix some N > αl∗. Next we discuss the valid range of for the claim to hold. We only consider > 0 such that
< ̄
6(ρ̄+ ρ̃+ 3) ∧ 0 3
where ρ̄ and ρ̃ are the constants in Corollary E.7 and Corollary E.10 respectively, and 0 is the constant in eq. (E.8). Moreover, recall function Ψ in Lemma E.21 and the constant c∗ > 0 in Lemma E.22. Due to lim ↓0 Ψ( ) = 0, it holds for all small enough such that
3Ψ( )
c∗ < C (E.97)
In our proof we only consider small enough so the inequality above holds, and the claim in Lemma E.13 holds. Now we specify the valid range of parameter δ that will be used below:
• For all sufficiently small δ > 0, the claim in Lemma E.14 will hold for the prescribed and with N0 = N ;
• For all sufficiently small δ > 0, the claims in Lemma E.17, Corollary E.18, Corollary E.19 and Lemma E.20 will hold with the prescribed and N ;
• For all sufficiently small δ > 0, the inequalities in Lemma E.21 and E.22 will hold for the we fixed at the beginning.
We show that the claim holds for any , δ small enough to satisfy the conditions above.
First, recall that
A×k,0( , δ, η) = ∆ { ∃i = 0, 1, · · · , l∗ ∧ jk s.t.
max j=Tk,i+1,··· ,(Tk,i+1−1)∧τ̃k∧σ(η)
η|Z≤δ,ηTk,i+1 + · · ·+ Z ≤δ,η j | >
3ρ̄+ 3ρ̃+ 3
} .
Due to our choice of δ stated earlier and Lemma E.12, there exists some η0 > 0 such that for all η ∈ (0, η0),
P ( A×k,0( , δ, η) ) ≤ ηN ∀k ≥ 1. (E.98)
Similarly, recall that A×k,1( , δ, η) = ∆ { σ(η) < τ̃k, jk < l∗ } . Let us temporarily focus on the first
attempt (namely the case k = 1). From Lemma E.17 and our choice of and δ, we know the existence of some η1 > 0 such that
sup |x|≤2
Px ( A×1,1( , δ, η) ) ≤ ηN ∀η ∈ (0, η1). (E.99)
Next, for A×k,2 = ∆ { jk ≥ l∗, ∃j = 2, 3, · · · , l∗ s.t. Tk,j − Tk,j−1 > 2t̂( )/η } , from Corollary E.18
and our choice of δ at the beginning, we have the existence of some η2 > 0 such that
sup |x|≤2
Px ( A×1,2( , δ, η) ) ≤ ηN ∀η ∈ (0, η2). (E.100)
Moving on, for A×k,3 = ∆ { jk < l∗, τ̃k < σ(η), τ̃k − Tk,1 > 2l∗t̂( )/η } , due to Corollary E.19 and
our choice of , δ, we have the existence of some η3 > 0 such that
sup |x|≤2
Px ( A×1,3( , δ, η) ) ≤ ηN ∀η ∈ (0, η3). (E.101)
As for A×k,4 = ∆ { jk ≥ l∗, |X η Tk,l∗ | ≥ r− ̄, ∃j = 1, 2, · · · , l∗ s.t. η|Wk,j | ≤ δ̄ } , from Lemma E.20, one can see the existence of η4 > 0 such that
sup |x|≤2
Px ( A×1,4( , δ, η) ) ≤ ηN ∀η ∈ (0, η4). (E.102)
Lastly, for A×k,5 = ∆ { jk ≥ l∗, Tk,l∗ ≤ σ(η)∧ τ̃k, X η Tk,l∗ ∈ ⋃ j∈[nmin−1][sj− , sj+ ] } , from Lemma E.21 we see the existence of η5 > 0 such that
sup |x|≤2
Px ( A×1,5( , δ, η) ) ≤ 2δαΨ( ) ( H(1/η)/η )l∗−1 ∀η ∈ (0, η5). (E.103)
Recall that A×k ( , δ, η) = ⋃5 i=0 A × k,i( , δ, η). Also, for definitions of B × k ,A ◦ k,B ◦ k, see eq. (E.93),eq. (E.94),eq. (E.95) respectively. Our next goal is to establish bounds regarding the probabilities of these events. First, if we consider the event ⋂k j=1(A × j ∪B × j )
c∩B◦j , then the inclusion of the (B◦j )kj=1 implies that during the first k attempts the SGD iterates have never left the attraction field, so
k⋂ j=1 (A×j ∪ B × j ) c ∩ B◦j = ( k⋂ j=1 (A×j ∪ B × j ) c ∩ B◦j ) ∩ {σ(η) > τ̃k}.
Next, note that
Px ( B×k | k−1⋂ j=1 (A×j ∪ B × j ) c ∩ B◦j )
=Px ( B×k,1 | k−1⋂ j=1 (A×j ∪ B × j ) c ∩ B◦j ) Px ( B×k,2 ∣∣∣ ( k−1⋂ j=1 (A×j ∪ B × j ) c ∩ B◦j ) ∩ B×k,1 ) ≤Px ( jk ≥ l∗, Tk,j − Tk,j−1 ≤ 2t̂( )
η ∀j = 2, 3, · · · , l∗ | k−1⋂ j=1 (A×j ∪ B × j ) c ∩ B◦j )
· Px ( B×k,2 ∣∣∣ ( k−1⋂
j=1
(A×j ∪ B × j ) c ∩ B◦j ) ∩ B×k,1 ) .
From the definition of the events A×j ,B × j ,B ◦ j , one can see that ⋂k−1 j=1 (A × j ∪B × j )
c∩B◦j ∈ Fτ̃k−1∧σ(η), and on this event ⋂k−1 j=1 (A × j ∪ B × j )
c ∩ B◦j we have σ(η) > τ̃k−1. So by applying strong Markov property at stopping time τ̃k−1 ∧ σ(η), we have
Px ( B×k | k−1⋂ j=1 (A×j ∪ B × j ) c ∩ B◦j )
≤P ( T ηj (δ)− T η j−1(δ) ≤ 2t̂( )/η ∀j ∈ [l ∗ − 1] ) · Px ( B×k,2 ∣∣∣ ( k−1⋂ j=1 (A×j ∪ B × j ) c ∩ B◦j ) ∩ B×k,1 )
≤2 ( H(δ/η)t̂( )/η )l∗−1 · Px ( B×k,2 ∣∣∣ ( k−1⋂ j=1 (A×j ∪ B × j ) c ∩ B◦j ) ∩ B×k,1 ) (for all η sufficiently small due to Lemma E.4)
≤4 ( t̂( ) δα )l∗−1(H(1/η) η )l∗−1 · Px ( B×k,2 ∣∣∣ ( k−1⋂ j=1 (A×j ∪ B × j ) c ∩ B◦j ) ∩ B×k,1 ) for all sufficiently small η, due to H ∈ RV−α(η). Meanwhile, note that
• (⋂k−1
j=1 (A × j ∪ B × j ) c ∩ B◦j ) ∩ B×k,1 ∈ FTk,l∗ ;
• on this event (⋂k−1
j=1 (A × j ∪ B × j ) c ∩ B◦j ) ∩ B×k,1 we have σ(η) ∧ τ̃k > Tk,l∗ and X η Tk,l∗
∈ [s− + , s+ − ].
Therefore, using Lemma E.13 and strong Markov property again (at stopping time Tk,l∗), we know the following inequality holds for all η sufficiently small:
sup k≥1 sup |x|≤2
Px ( B×k,2 ∣∣∣ ( k−1⋂
j=1
(A×j ∪ B × j ) c ∩ B◦j ) ∩ B×k,1 )
= sup k≥1 sup |x|≤2
Px ({ σ(η) > τ̃k, τ̃k − Tk,l∗ ≤ ρ( )/η }c ∣∣∣ ( k−1⋂ j=1 (A×j ∪ B × j ) c ∩ B◦j ) ∩ B×k,1 ) ≤Ψ( ) δ α
4 ( t̂( )/δα )l∗−1 . Therefore, we know the existence of some η6 > 0 such that
sup |x|≤2
Px ( B×k | k−1⋂ j=1 (A×j ∪ B × j ) c ∩ B◦j ) ≤ Ψ( )δα (H(1/η) η )l∗−1 ∀η ∈ (0, η6), ∀k ≥ 1. (E.104)
Similarly, we can bound conditional probabilities of the form Px ( A×k | ⋂k−1 j=1 (A × j ∪ B × j ) c ∩ B◦j ) . To be specific, recall that A×k = ∪5i=0A × k,i. By combining eq. (E.98)-eq. (E.103) with Markov property, we know the existence of some η7 > 0 such that
sup |x|≤2
Px ( A×k | k−1⋂ j=1 (A×j ∪ B × j ) c ∩ B◦j ) ≤ 5ηN + 2Ψ( )δα (H(1/η) η )l∗−1 ∀η ∈ (0, η7), ∀k ≥ 1.
(E.105)
On the other hand, a lower bound can be established for conditional probability involving A◦k, the event defined in eq. (E.94) describing the exit from Ω during an attempt with exactly l∗ large noises. Using Lemma E.22 and Markov property of (Xηn)n≥1, one can see the existence of some η8 > 0 such that
inf |x|≤2
Px ( A◦k | k−1⋂ j=1 (A×j ∪ B × j ) c ∩ B◦j ) ≥ c∗δα (H(1/η) η )l∗−1 ∀η ∈ (0, η8). (E.106)
In order to apply the bounds eq. (E.104)-eq. (E.106), we make use of the following inclusion relationship: ( k−1⋂
j=1
(A×j ∪ B × j ) c ∩ B◦j ) ∩ (A×k ∪ B × k ) c ⊆ A◦k ∪ B ◦ k. (E.107)
To see why this is true, let us consider a decomposition of the event on the L.H.S. of eq. (E.107). As mentioned above, on event ⋂k−1 j=1 (A × j ∪ B × j )
c ∩ B◦j we know that σ(η) > τ̃k−1, so the k−th attempt occurred and there are only three possibilities on this event:
• jk < l∗;
• jk ≥ l∗, X η Tk,l∗ /∈ Ω;
• jk ≥ l∗, X η Tk,l∗ ∈ Ω.
Let us partition the said event accordingly and analyze them one by one.
• On (⋂k−1
j=1 (A × j ∪ B × j ) c ∩ B◦j ) ∩ (A×k ∪ B × k ) c ∩ {jk < l∗}, due to the exclusion of A × k
(especially A×k,1 and A × k,3), we can see that if jk < l ∗, then we must have σ(η) > τ̃k and τ̃k − Tk,1 ≤ 2l∗t̂( )/η. Therefore,( k−1⋂
j=1
(A×j ∪ B × j ) c ∩ B◦j ) ∩ (A×k ∪ B × k ) c ∩ {jk < l∗} ⊆ B ◦ k.
• On (⋂k−1
j=1 (A × j ∪ B × j ) c ∩ B◦j ) ∩ (A×k ∪ B × k ) c ∩ {jk ≥ l∗, X η Tk,l∗
/∈ Ω}, then the exclusion of A×k,2 implies that Tk,j − Tk,j−1 ≤ 2t̂( )/η for all j = 2, · · · , l∗, and the exclusion of A×k,5 tells us that if X η Tk,l∗ /∈ Ω, then we have XηTk,l∗ /∈ [s− − , s+ + ]. In summary,
( k−1⋂ j=1 (A×j ∪ B × j ) c ∩ B◦j ) ∩ (A×k ∪ B × k ) c ∩ {jk ≥ l∗, X η Tk,l∗ /∈ Ω} ⊆ A◦k.
• On (⋂k−1
j=1 (A × j ∪ B × j ) c ∩ B◦j ) ∩ (A×k ∪ B × k ) c ∩ {jk ≥ l∗, X η Tk,l∗
∈ Ω}, the exclusion of A×k,2 again implies that Tk,j−Tk,j−1 ≤ 2t̂( )/η for all j = 2, · · · , l∗, hence Tk,l∗ −Tk,1 ≤ 2l∗t̂( )/η. Similarly, the exclusion of A×k,5 tells us that if X η Tk,l∗
∈ Ω, then we have XηTk,l∗ ∈ [s−+ , s+− ]. Now since B × k did not occur (see the definition in eq. (E.93)), we must have σ(η) > τ̃k and τ̃k − Tk,l∗ ≤ ρ( )/η, hence τ̃k − Tk,1 ≤ 2l ∗ t̂( )+ρ( )
η . Therefore,
( k−1⋂ j=1 (A×j ∪ B × j ) c ∩ B◦j ) ∩ (A×k ∪ B × k ) c ∩ {jk ≥ l∗, X η Tk,l∗ ∈ Ω} ⊆ B◦k.
Collecting results above, we have eq. (E.107). Now we discuss some of its implications. First, from eq. (E.107) we can immediately get that
( k−1⋂ j=1 (A×j ∪ B × j ) c ∩ B◦j ) ∩ (A×k ∪ B × k ) c = ( k−1⋂ j=1 (A×j ∪ B × j ) c ∩ B◦j ) ∩ (A×k ∪ B × k ) c ∩ (A◦k ∪ B ◦ k).
(E.108)
Next, recall the definitions of A◦k in eq. (E.94) and B ◦ k in eq. (E.95), and one can see that A ◦ k and B ◦ k are mutually exclusive, since the former implies that the first exit occurs during the k−th attempt while the latter implies that this attempt fails. This fact and eq. (E.108) allow us to conclude that
k⋂ i=1 ( A×i ∪ B × i ∪ A ◦ i )c = k⋂ i=1 ( A×i ∪ B × i )c ∩ B◦i = ( k−1⋂ i=1 ( A×i ∪ B × i )c ∩ B◦i) ∩ (A×k ∪ B×k ∪ A◦k)c. (E.109)
Now we use the results obtained so far to bound the probability of
A×( , δ, η) =∆ ⋃ k≥1 ( k−1⋂ i=1 ( A×i ∪ B × i ∪ A ◦ i )c) ∩ (A×k ∪ B×k ).
Using eq. (E.109), we can see that (for any x ∈ [−2 , 2 ]) Px(A×( , δ, η))
= ∑ k≥1 Px (( k−1⋂ i=1 ( A×i ∪ B × i ∪ A ◦ i )c) ∩ (A×k ∪ B×k )) = ∑ k≥1 Px (( k−1⋂ i=1 ( A×i ∪ B × i )c ∩ B◦i) ∩ (A×k ∪ B×k )) = ∑ k≥1 Px ( A×k ∪ B × k | k−1⋂ i=1 ( A×i ∪ B × i )c ∩ B◦i)
· k−1∏ j=1 Px ( j⋂ i=1 ( A×i ∪ B × i )c ∩ B◦i ∣∣∣ j−1⋂ i=1 ( A×i ∪ B × i )c ∩ B◦i)
= ∑ k≥1 Px ( A×k ∪ B × k | k−1⋂ i=1 ( A×i ∪ B × i )c ∩ B◦i)
· k−1∏ j=1 Px (( j−1⋂ i=1 ( A×i ∪ B × i )c ∩ B◦i ) ∩ (A×j ∪ B×j ∪ A◦j )c ∣∣∣ j−1⋂ i=1 ( A×i ∪ B × i )c ∩ B◦i)
= ∑ k≥1 Px ( A×k ∪ B × k | k−1⋂ i=1 ( A×i ∪ B × i )c ∩ B◦i)
· k−1∏ j=1 Px (( A×j ∪ B × j ∪ A ◦ j )c ∣∣∣ j−1⋂ i=1 ( A×i ∪ B × i )c ∩ B◦i)
≤ ∑ k≥1 Px ( A×k ∪ B × k | k−1⋂ i=1 ( A×i ∪ B × i )c ∩ B◦i) · k−1∏ j=1 ( 1− Px ( A◦j ∣∣∣ j−1⋂ i=1 ( A×i ∪ B × i )c ∩ B◦i) ) .
This allows us to apply eq. (E.104)-eq. (E.106) and conclude that (here we only consider η < min{ηi : i ∈ [8]} ),
sup |x|≤2
Px(A×( , δ, η))
≤ ∑ k≥1 ( 5ηN + 2Ψ( )δα (H(1/η) η )l∗−1) · ( 1− c∗δα (H(1/η) η )l∗−1)k−1
= 5ηN + 2Ψ( )δα
( H(1/η)
η )l∗−1 c∗δα ( H(1/η)
η )l∗−1 ≤2Ψ( ) + 5η α
c∗ for sufficiently small η, due to H ∈ RV−α(η) and our choice of N > αl∗
≤3Ψ( ) c∗ < C for all η small enough such that 5ηα < Ψ( ).
The last inequality follows from our choice of in eq. (E.97). This concludes the proof.
Having established Lemma E.24, we return to Proposition E.23 and give a proof. Recall that, aside from the attraction field Ω = (s−, s+), there are nmin − 1 other attraction fields Ω̃k = (s−k , s + k ) (for each k ∈ [nmin − 1]). Besides, the function λ(·) and constants νΩ, νΩk are defined in eq. (E.77)eq. (E.79).
Proof of Proposition E.23. We fix some parameters for the proof. First, with out loss of generality we only need to consider C ∈ (0, 1). Next we discuss the valid range of for the claim to hold. We
only consider > 0 such that
< ̄
6(ρ̄+ ρ̃+ 3) ∧ 0 3
where ρ̄ and ρ̃ are the constants in Corollary E.7 and Corollary E.10 respectively, and 0 is the constant in eq. (E.8). Due to continuity of measure µ, it holds for all small enough such that (let ̂ = 3(ρ̄+ ρ̃+ 3) )
µ(E(0)) µ ( E ( ̂ )) < 1/(1− C), (E.110)
µ(E(0)) µ ( E ( − ̂ )) > 1/(1 + C), (E.111)
µ ( h−1 ( (s− − 2̂, s− + 2̂) ∪ (s+ − 2̂, s+ + 2̂) )) µ(E(−̂)) ≤ C (E.112)
µ ( E(̂) ∩ ( s−k′ − ̂, s + k′ + ̂) ) µ(E(̂)) ≤ ν Ω k′ + C νΩ (E.113)
µ ( E(−̂) ∩ ( s−k′ + 2̂, s + k′ − 2̂) ) µ(E(−̂)) ≥ ν Ω k′ − C νΩ
(E.114)
In our proof we only consider small enough so the inequality above holds, and the claims in Lemma E.13 hold. Moreover, we only consider and δ small enough so that Lemma E.24 hold and we have
lim η↓0 sup |x|≤2
Px(A×( , δ, η)) < C. (E.115)
We show that the desired claims hold for all , δ sufficiently small that satisfy conditions above.
First, in order to show eq. (E.84), we define event
Ã × ( , δ, η) =∆ ( A×( , δ, η) )c∩{ λ(η) ( τ(η, )− σ(η) ) ≥ C or ∃n = σ(η) + 1, · · · , τ(η, ) such that Xηn /∈ Ω̃Jσ(η) } .
Since λ ∈ RV−1−l∗(α−1) and α > 1, for the we fixed at the beginning of this proof, ρ( ) is a fixed constant as well (the function ρ is defined in Lemma E.13) and we have limη↓0 λ(η)ρ( )/η = 0. Next, the occurrence of ( A×( , δ, η) )c (in particular, the exclusion of all the A×k,5 defined in eq. (E.91)), we know that Xησ(η) /∈ [s − Jσ − , s−Jσ + ] ∪ [s + Jσ − , s+Jσ + ] (recall that for any k ∈ [nmin − 1], we have Ω̃j = (s−j , s + j ); for definition of Jσ see eq. (E.76)). Meanwhile, for all η sufficiently small, we have /λ(η) > ρ( )/η. Therefore, using Lemma E.13 we can see that (for all η sufficiently small)
sup |x|≤2
Px ( Ã × ( , δ, η) | ( A×( , δ, η) )c) ≤ C. (E.116) Lastly, observe that
P ({ λ(η) ( τ(η, )− σ(η) ) ≥ C or ∃n = σ(η) + 1, · · · , τ(η, ) such that Xηn /∈ Ω̃Jσ(η) }) ≤Px ( (A×)c ∩ { λ(η) ( τ(η, )− σ(η) ) ≥ C
or ∃n = σ(η) + 1, · · · , τ(η, ) such that Xηn /∈ Ω̃Jσ(η) }) + Px(A×)
so by combining eq. (E.115) with eq. (E.116), we can obtain eq. (E.84).
Moving on, we discuss the upper bounds eq. (E.80) and eq. (E.82). Recall that the fixed constant k′ ∈ [nmin− 1] is prescribed in the description of this proposition. Let us observe some facts on event (A×( , η, δ))c ∩ {Jσ(η) = k′}: If we let J( , δ, η) =∆ sup{k ≥ 0 : τ̃k < σ(η)} be the number of attempts it took to escape, and
J↑( , δ, η) =∆ min{k ≥ 1 : Tk,1 has ( 3(ρ̄+ ρ̃+ 3) , δ, η ) −overflow},
then for all η sufficiently small, we must have J ≤ J↑ on event (A×( , η, δ))c ∩ {Jσ(η) = k′}. To see this via a proof of contradiction, let us assume that, for some arbitrary positive integer j, there exists some sample path on (A×)c ∩ {Jσ(η) = k′} such that J↑ = j < J . Then from the definition of (A×)c, in particular the exclusion of event A×j,0 (see the definition in eq. (E.86)), for all sufficiently small η, we are able to apply Corollary E.10 and E.7 and conclude that XηTj,l∗ /∈ Ω: indeed, using Corollary E.10 and E.7 we can show that the distance between XηTj,l∗ and the perturbed ODE
x̃η ( Tj,l∗ − Tj,1, 0; ( 0, Tj,2 − Tj,1, · · · , Tj,l∗ − Tj,1 ) , ( ηWj,1, · · · , ηWj,l∗ )) is strictly less than 3(ρ̄+ ρ̃+ 3) ; on the other hand, the definition of ( 3(ρ̄+ ρ̃+ 3) , δ, η ) −overflow
implies that x̃η ( Tj,l∗ − Tj,1, 0; ( 0, Tj,2 − Tj,1, · · · , Tj,l∗ − Tj,1 ) , ( ηWj,1, · · · , ηWj,l∗ )) /∈ [s− − 3(ρ̄+ ρ̃+ 3) , s+ + 3(ρ̄+ ρ̃+ 3) ].
Therefore, we must have XηTj,l∗ /∈ Ω, which contradicts our assumption j = J ↑ < J . In summary, we have shown that, on (A×)c ∩ {Jσ = k′}, we have J↑( , δ, η) ≥ J( , δ, η). Similarly, if we consider
J↓( , δ, η) =∆ min{k ≥ 1 : Tk,1 has ( − 3(ρ̄+ ρ̃+ 3) , δ, η ) −overflow},
then by the same argument above we can show that J↓( , δ, η) ≤ J( , δ, η). Now consider the following decomposition of events.
• On {J↓ < J↑}, we know that for the first k such that Tk,1 has ( − 3(ρ̄ + ρ̃ +
3) , δ, η ) −overflow, it does not have ( 3(ρ̄ + ρ̃ + 3) , δ, η ) −overflow. Now we analyze the probability that Z0 does not have (̂, δ, η)−overflow conditioning on that it does have (−̂, δ, η)−overflow (recall that we let ̂ = 3(ρ̄+ ρ̃+ 3)). Using Lemma E.15 and the bound eq. (E.112), we know that for all η sufficiently small,
sup |x|≤2
Px ( (A×)c ∩ {J↓ < J↑} )
≤ µ ( h−1 ( (s− − 2̂, s− + 2̂) ∪ (s+ − 2̂, s+ + 2̂) )) µ(E(−̂)) ≤ C. (E.117)
• On (A×)c ∩ {Jσ = k′} ∩ {J↑ = J↓}, due to J↑ = J↓ = J we know that TJ( ,δ,η),1 is the first among all Tk,1 to have (̂, δ, η)−overflow. Moreover, due to {Jσ = k′} and using Corollary E.10 and E.7 again as we did above, we know that the overflow endpoint of TJ( ,δ,η),1 is in (s − k′ − ̂, s + k′ + ̂) (recall that Ω̃k′ = (s − k′ , s + k′)). In summary, for any n ≥ 0
(A×)c ∩ {Jσ = k′} ∩ {J↑ = J↓ > n} ⊆ (A×)c ∩ {J↑ > n} ∩ { TJ↑,1 has overflow endpoint in (s − k′ − ̂, s + k′ + ̂) } so using Lemma E.15, we obtain that (for all η sufficiently small)
sup |x|≤2
Px ( (A×)c ∩ {Jσ = k′} ∩ {J↑ = J↓ > n} )
≤ sup |x|≤2
Px ( (A×)c ∩ {J↑ > n} ) · p ( ̂, δ, η; (s−k′ − ̂, s + k′ + ̂) ) p(̂, δ, η)
≤ sup |x|≤2
Px ( (A×)c ∩ {J↑ > n} ) · ν Ω k′ + C
νΩ . (E.118)
uniformly for any n = 0, 1, 2, · · · due to eq. (E.113).
• On the other hand, on (A×)c, if TJ↓,1 has overflow endpoint in (s−k′ + 2̂, s + k′ − 2̂), then
from Definition E.1 we know that TJ↓,1 also has (̂, δ, η)−overflow, hence J↓ = J↑ = J . Moreover, using Corollary E.10 and E.7 again, we know that XηT J↓,l∗ ∈ (s−k′ , s + k′) so Jσ = k ′. In summary, for any n ≥ 0,
(A×)c ∩ {Jσ = k′} ∩ {J↑ = J↓ > n} ⊇ (A×)c ∩ {J↓ > n} ∩ { TJ↓,1 has overflow endpoint in (s − k′ + 2̂, s + k′ − 2̂) } so using Lemma E.15, we obtain that (for all η sufficiently small)
inf |x|≤2
Px ( (A×)c ∩ {Jσ = k′} ∩ {J↑ = J↓ > n} )
≥ inf |x|≤2
Px ( (A×)c ∩ {J↓ > n} ) · p ( − ̂, δ, η; (s−k′ + 2̂, s + k′ − 2̂) ) p(−̂, δ, η)
≥ inf |x|≤2
Px ( (A×)c ∩ {J↓ > n} ) · ν
Ω k′ − C νΩ . (E.119)
uniformly for any n = 0, 1, 2, · · · due to eq. (E.114).
Besides, the following claim holds on event (A×)c.
• From eq. (E.109), the definition of B◦k as well as the definition of event A ◦ k (see eq. (E.94)),
one can see that for any j = 1, 2, · · · , J , we have
τ̃j ∧ σ(η)− Tj,1 ≤ 2l∗t̂( ) + ρ( )
η .
• Now if we turn to the interval (τ̃j−1, Tj,1] (the time between the start of the j−th attempt and the arrival of the first large noise during this attempt) for each j = 1, 2, · · · , J , and the following sequence constructed by concatenating these intervals
S( , δ, η) =∆ ( 1, 2, · · · , T1,1, τ̃1 + 1, τ̃1 + 2, · · · , T2,1, · · · , τ̃k + 1, τ̃k + 2, · · · , Tk+1,1, τ̃k+1 + 1, τ̃k+2 + 1, · · · ) ,
then the discussion above have shown that, for min{n ∈ S( , δ, η) : Zn has ( 3(ρ̄+ ρ̃+ 3) , δ, η ) −overflow} ≥ TJ,1. Meanwhile, from the definition of overflow we know that the probability that Z1 has( 3(ρ̄+ ρ̃+ 3) , δ, η ) −overflow is equal to
H(δ/η)p ( 3(ρ̄+ ρ̃+ 3) , δ, η ) .
• Therefore, if, within the duration of each attempt, we split the attempt into two parts at the arrival time of the first large jump (Tk,1)k≥1 at each attempt, and define (here the subscript before or after indicates that we are counting the steps before or after the first large jump in an attempt)
Sbefore( , δ, η) =∆ {n ∈ S( , δ, η) : n ≤ σ(η)}, Ibefore( , δ, η) =∆ #Sbefore( , δ, η), Safter( , δ, η) =∆ {n /∈ S( , δ, η) : n ≤ σ(η)}, Iafter( , δ, η) =∆ #Safter( , δ, η),
then we have σ(η) = Ibefore + Iafter. Moreover, the discussion above implies that Iafter ≤ J ( 2l∗t̂( ) + ρ( ) ) /η
Ibefore ≤ min{n ∈ S( , δ, η) : Zn has ( 3(ρ̄+ ρ̃+ 3) , δ, η ) −overflow}
and on event (A×)c.
Define geometric random variables with the following success rates U1( , δ, η) ∼ Geom ( p ( 3(ρ̄+ ρ̃+ 3) , δ, η )) U2( , δ, η) ∼ Geom ( H(δ/η)p ( 3(ρ̄+ ρ̃+ 3) , δ, η )) .
Using results above to bound Ibefore and Iafter separately on event (A×)c, we can show that (for all η sufficiently small and any u > 0)
sup x∈[−2 ,2 ]
Px ( vΩλ(η)σ(η) > u, Jσ(η) = k ′ )
≤ sup |x|≤2 Px(A×( , δ, η)) + sup x∈[−2 ,2 ]
Px ( {vΩλ(η)σ(η) > u, Jσ(η) = k′} ∩ ( A×( , δ, η) )c) ≤C + sup
x∈[−2 ,2 ] Px ( {vΩλ(η)σ(η) > u, Jσ(η) = k′} ∩ ( A×( , δ, η) )c) due to eq. (E.115)
≤C + sup x∈[−2 ,2 ]
Px ( {vΩλ(η)Ibefore( , δ, η) > (1− C)u, Jσ(η) = k′} ∩ ( A×( , δ, η) )c) + sup x∈[−2 ,2 ] Px ( {vΩλ(η)Iafter( , δ, η) > Cu} ∩ ( A×( , δ, η)
)c) ≤C + sup
x∈[−2 ,2 ] Px ( {vΩλ(η)Ibefore( , δ, η) > (1− C)u, Jσ(η) = k′} ∩ ( A×( , δ, η) )c) +P ( vΩλ(η) ρ( ) + 2l∗t̂( )
η · U1( , δ, η) > Cu ) ≤C
+ sup x∈[−2 ,2 ]
Px ( {vΩλ(η)Ibefore( , δ, η) > (1− C)u, Jσ(η) = k′} ∩ ( A×( , δ, η) )c ∩ {J↓ = J↑}) + sup x∈[−2 ,2 ] Px ( (A×)c ∩ {J↓ < J↑} ) + P ( vΩλ(η) ρ( ) + 2l∗t̂( ) η · U1( , δ, η) > Cu
) ≤2C + P ( vΩλ(η)U2( , δ, η) > (1− C)u )νΩk′ + C νΩ
+P ( vΩλ(η) ρ( ) + 2l∗t̂( )
η · U1( , δ, η) > Cu
) (E.120)
where the last inequality follows from eq. (E.117) and eq. (E.118). Now let us analyze the probability terms on the last row of the display above. For the first term, let a(η) = H(δ/η)p ( 3(ρ̄+ρ̃+3) , δ, η ) .
Due to Lemma E.15, we have (recall that νΩ = µ(E(0)))
lim η↓0
a(η) λ(η)µ ( E ( 3(ρ̄+ ρ̃+ 3) )) = 1. Combining this with eq. (E.110), one can see that for all η sufficiently small,
P ( vΩλ(η)U2( , δ, η) > (1− C)u ) ≤ P ( a(η)Geom ( a(η) ) > (1− C)2u ) ∀u > 0.
Next, let b(η, u) = P ( a(η)Geom ( a(η) ) > (1 − C)2u ) = P ( Geom ( a(η) ) > (1−C)
2u a(η)
) . For
g(y) = log(1 − y), we know the existence of some y0 > 0 such that for all y ∈ (0, y0), we have log(1− y) ≤ −(1− C)y. So one can see that for all η sufficiently small,
log b(u, η) ≤ (1− C) 2u
a(η) log(1− a(η)) ≤ −(1− C)3u ⇒ b(u, η) ≤ exp ( − (1− C)3u ) (E.121)
uniformly for all u > 0.
For the second probability term, if we only consider u ≥ C, then P ( vΩλ(η) ρ( ) + 2l∗t̂( )
η · U1( , δ, η) > Cu
) ≤ P ( vΩλ(η) ρ( ) + 2l∗t̂( )
η · U1( , δ, η) > C2
) .
Using H ∈ RV−α(η) with α > 1, we get p ( 3(ρ̄+ ρ̃+ 3) , δ, η ) U1( , δ, η) d−→ Exp(1) as η ↓ 0
due to the nature of the Geometric random variable U1. Besides, due to H ∈ RV−α(η) with α > 1 and Lemma E.15, it is easy to show that
lim η↓0
λ(η)ρ( )+2l ∗ t̂( )
η p ( 3(ρ̄+ ρ̃+ 3) , δ, η ) = 0.
Combining these results with Slutsky’s theorem, we now obtain
µ(E(0))λ(η) ρ( ) + 2l∗t̂( )
η · U1( , δ, η)
d−→ 0 as η ↓ 0.
Therefore,
lim sup η↓0 sup u≥C
P ( µ(E(0))λ(η) ρ( ) + 2l∗t̂( )
η · U1(δ, η) > Cu
) = 0. (E.122)
Plugging eq. (E.121) and eq. (E.122) back into eq. (E.120), we can establish the upper bound in eq. (E.80). To show eq. (E.82), note that for event
E( , η) = {νΩλ(η)τ(η, ) > u, Xητ(η, ) ∈ B(m̃k′ , 2 )}, we have (for definitions of τ , see eq. (E.75))
E( , η) ⊇{vΩλ(η)σ(η) > u, Jσ(η) = k′} ∩ {Xηn ∈ Ω̃Jσ(η) ∀n ∈ [σ(η), τ(η, )]},
E( , η)∩{vΩλ(η)σ(η) > u, Jσ(η) = j} ∩ {Xηn ∈ Ω̃Jσ(η) ∀n ∈ [σ(η), τ(η, )]} = ∅ ∀j 6= k ′.
Therefore, for all η sufficiently small, sup |x|≤2 Px(E( , η))
≤ sup |x|≤2 Px(A×) + sup |x|≤2
Px((A×)c ∩ {Xηn /∈ Ω̃Jσ(η) for some n ∈ [σ(η), τ(η, )]} )
+ sup |x|≤2
Px ( (A×)c ∩ {vΩλ(η)σ(η) > u, Jσ(η) = k′} ∩ {Xηn ∈ Ω̃Jσ(η) ∀n ∈ [σ(η), τ(η, )]} )
≤4C + exp ( − (1− C)3u )νΩk′ + C νΩ
uniformly for all u ≥ C, due to eq. (E.115), eq. (E.84) and eq. (E.120). The lower bound can be shown by an almost identical approach. In particular, analogous to eq. (E.120), we can show that (for any u > 0)
inf x∈[−2 ,2 ]
Px ( vΩλ(η)σ(η) > u, Jσ(η) = k ′ )
≥ inf x∈[−2 ,2 ]
Px ( {vΩλ(η)σ(η) > u, Jσ(η) = k′} ∩ ( A×( , δ, η) )c) ≥ inf x∈[−2 ,2 ] Px ( {vΩλ(η)Ibefore( , δ, η) > u, Jσ(η) = k′} ∩ ( A×( , δ, η)
)c) ≥P ( vΩλ(η)U ′2( , δ, η) > (1− C)u )νΩk′ + C νΩ − 2C
due to P(E\F ) ≥ P(E)− P(F ) and eq. (E.115)eq. (E.117)eq. (E.119) where U ′2( , δ, η) ∼ Geom ( H(δ/η)p ( − 3(ρ̄+ ρ̃+ 3) , δ, η )) .
Using the similar argument leading to eq. (E.121), we are able to show eq. (E.81), eq. (E.83) and conclude the proof.
Recall that σi(η) = min{n ≥ 0 : Xn /∈ Ωi} and that value of constants qi, qi,j are specified via eq. (E.2)-eq. (E.7). Define
τmini (η, ) = ∆ min{n ≥ σi(η) : Xηn ∈ ⋃ j [mj − 2 ,mj + 2 ]}, (E.123)
Ji(η) = j ⇐⇒ Xησi(η) ∈ Ωj ∀j ∈ [nmin]. (E.124)
The following result is simply a restatement of Proposition E.23 under the new system of notations. Despite the reiteration, we still state it here because this is the version that will be used to prove Lemma E.2, which is the key tool for establishing Theorem 1, as well as many other results in Section F.
Proposition E.25. Given C > 0 and i, j ∈ [nmin] such that i 6= j, the following claims hold for all > 0 that are sufficiently small:
lim sup η↓0 sup u∈(C,∞) sup x∈(mi−2 ,mi+2 )
Px ( qiλi(η)σi(η) > u, X
η σi(η)
∈ Ωj )
≤C + exp ( − (1− C)u )qi,j + C qi ,
lim inf η↓0 inf u∈(C,∞) inf x∈(mi−2 ,mi+2 )
Px ( qiλi(η)σi(η) > u, X
η σi(η)
∈ Ωj )
≥− C + exp ( − (1 + C)u )qi,j − C qi ,
lim sup η↓0 sup u∈(C,∞) sup x∈(mi−2 ,mi+2 )
Px ( qiλi(η)τ min i (η, ) > u, X
η τmini (η, )
∈ Ωj )
≤C + exp ( − (1− C)u )qi,j + C qi ,
lim inf η↓0 inf u∈(C,∞) inf x∈(mi−2 ,mi+2 )
Px ( qiλi(η)τ min i (η, ) > u, X
η τmini (η, )
∈ Ωj )
≥− C + exp ( − (1 + C)u )qi,j − C qi ,
lim inf η↓0 inf x∈(mi−2 ,mi+2 )
Px ( qiλi(η) ( τmini (η, )− σi(η) ) < C,
Xηn ∈ ΩJi(η) ∀n ∈ [σi(η), τ min i (η, )]
) ≥ 1− C.
Concluding this section, we apply Proposition E.25 and prove Lemma E.2.
Proof of Lemma E.2. Fix some C ∈ (0, 1), u > 0, and some k, l ∈ [nmin] with k 6= l. Let qi, qi,j be the constants defined in eq. (E.7).
Fix some C0 ∈ ( 0, Cnmin ∧ qk nmin C ) . Using Proposition E.25, we know that for all sufficiently small, we have
lim sup η↓0 sup x∈(mk−2 ,mk+2 )
Px ( qkλk(η)σk(η) > u, X
η σk(η)
∈ Ωj )
≤ C0 + exp ( − (1− C)u )qk,j + C0 qk ∀j ∈ [nmin].
Summing up the inequality above over all j ∈ [nmin], we can obtain eq. (E.16). The lower bound eq. (E.17) can be established using an identical approach.
In order to show eq. (E.19), note that we can find C1 ∈ (0, u) sufficiently small so that
−C1 + exp ( − (1 + C1) · 2C1 )qk,l − C1 qk ≥ qk,l − C qk .
Fix such C1. From Proposition E.25, we also know that for all small enough, we have
lim inf η↓0 inf x∈(mk−2 ,mk+2 )
Px ( qkλk(η)σk(η) > u,X
η σk(η)
∈ Ωl )
≥ −C1 + exp ( − (1 + C1) · 2C1 )qk,l − C1 qk .
Then using Px ( Xησk(η) ∈ Ωl ) ≥ P ( qkλk(η)σk(η) > u,X η σk(η) ∈ Ωl )
we conclude the proof for eq. (E.19).
Moving on, we show eq. (E.18) in the following way. Note that we can find C2 ∈ (0, u) small enough so that
2C2 + qk,l + C2 qk < qk,l + C qk . (E.125)
Fix such C2. Since eq. (E.17) has been established already, we can find some u2 > 0 such that for all small enough,
lim sup η↓0 sup x∈(mk−2 ,mk+2 )
Px ( qkλk(η)σk(η) ≤ u2 ) < C2 (E.126)
Fix such u2. Meanwhile, fix some C3 ∈ (0, C2 ∧ u2). From Proposition E.25 we know that for all sufficiently small,
lim sup η↓0 sup x∈(mk−2 ,mk+2 )
Px ( qkλk(η)σk(η) > u2, X
η σk(η)
∈ Ωl )
(E.127)
≤ C3 + exp ( − (1− C3)u2 )qk,l + C3 qk
≤ C2 + qk,l + C2
qk . (E.128)
Lastly, observe the following decomposition of events (for any x ∈ Ωk) Px ( Xησk(η) ∈ Ωl ) ≤ Px ( qkλk(η)σk(η) ≤ u2 ) + Px ( qkλk(η)σk(η) > u2, X η σk(η) ∈ Ωl ) .
Combining this bound with eq. (E.125)-eq. (E.128), we complete the proof.

heading: F PROOFS FOR SECTION 2.2
text: In this section, we show that gradient clipping scheme effectively partitions the entire optimization landscape of f into different regions based on the radius ri and minimum jump number l∗i for each attraction field Ωi. Furthermore, when staying in each region, the behavior of SGD iterates closely resembles a Markov chain that only visits wider attraction fields in this region. We exclude the trivial case where nmin = 1 and there is only one attraction field.
This structure is as follows. First we present some key lemmas that can be used to prove the Theorem 2-3 in the main paper. Then we devote the rest of the section to establish those lemmas. In order to prove Theorem 2, we will make use of the following lemma, where we show that the type of claim in Theorem 2 is indeed valid if we look at a much shorter time interval. Then when we move onto the proof of Theorem 2, it suffices to partition the entire horizon into pieces of these short time intervals, on each of which we analyze the dynamics of SGD respectively. Lemma F.1. Assume the graph G is irreducible, and let > 0, δ > 0 be any positive real numbers. For the following random variables (indexed by η)
V small(η, , t) =∆ 1
bt/λlarge(η)c ∫ bt/λlarge(η)c 0 1 { Xηbuc ∈ ⋃ j:mj /∈M large Ωj } du, (F.1)
the following claim holds for any sufficiently small t > 0:
lim sup η↓0 sup x∈[−L,L]
Px ( V small(η, , t) > ) ≤ 5δ
Proof of Theorem 2. It suffices to show that for any t > 0, κ > 1 + (α − 1)llarge, > 0, δ ∈ (0, ), we have
lim sup η↓0
Px ( V ∗(η, t, κ) > 3 ) < δ
for
V ∗(η, t, κ) =∆ 1
bt/ηκc ∫ bt/ηκc 0 1 { Xηbuc(x) ∈ ⋃ j:mj /∈M large Ωj } du.
Let us fix some > 0, δ ∈ (0, ). First, let
N(η) = d bt/η κc
bt/λlarge(η)c e.
The regularly varying nature of H implies that λlarge(η) ∈ RV1+llarge(α−1)(η). Since κ > 1 + llarge(α − 1), we know that limη↓0N(η) = ∞. Next, due to Lemma F.1, we can find t0 > 0 and η̄ > 0 such that for any η ∈ (0, η̄)
sup y∈[−L,L]
Py(V small(η, , t0) > ) < δ. (F.2)
For any k ≥ 1, define
Vk(η) = ∆ 1
bt0/λlarge(η)c ∫ kbt0/λlarge(η)c (k−1)bt0/λlarge(η)c 1 { Xηbuc ∈ ⋃ j:mj /∈M large Ωj } du. (F.3)
It is clear from its definition that Vk stands for the proportion of time that the SGD iterates are outside of large attraction fields on the interval [(k − 1)b t0λlarge(η)c, kb t0 λlarge(η)c]. From eq. (F.2) and Markov property, one can see that for any η ∈ (0, η̄)
sup x∈[−L,L]
Px(Vk(η) > | Xη0 , · · · , X η (k−1)bt0/λlarge(η)c) ≤ δ
uniformly for all k ≥ 1. Now define K(η) =∆ #{n = 1, 2, · · · , N(η) : Vk(η) > }. By a simple stochastic dominance argument, we have
sup x∈[−L,L]
Px(K(η) ≥ j) ≤ P(Binomial(N(η), δ) ≥ j) ∀j = 1, 2, · · · .
Meanwhile, strong law of large numbers implies the existence of some η̄1 > 0 such that P( Binomial(N(η),δ)N(η) > 2δ) < δ for all η ∈ (0, η̄1), thus
sup |x|≤L
Px(K(η)/N(η) > 2δ) ≤ δ ∀η ∈ (0, η̄1 ∧ η̄).
Lastly, from the definition of K(η) and N(η), we know that for all the N(η) intervals [(k − 1)b t0λlarge(η)c, kb t0 λlarge(η)c] with k ∈ [N(η)], only on K(η) of them did the SGD iterates spent more then proportion of time outside of the large attraction fields, hence
V ∗(η, t, κ) ≤ + K(η) N(η) .
In summary, we now have
Px(V ∗(η, t, κ) > 3 ) < δ
for all η ∈ (0, η̄1 ∧ η̄). This concludes the proof.
When introducing Theorem 3 in the main paper, we stated that the results of eliminating sharp minima can be extended to the more general reducible case. Here we present the corresponding theoretical results in Theorem F.2 and F.3 below. The main message can be summarized as follows: (a) SGD with truncated and heavy-tailed noise naturally partitions the entire training landscape into different
regions; (b) In each region, the dynamics ofXηn for small η closely resemble that of a continuous-time Markov chain that only visits local minima; (3) In particular, any sharp minima within each region is almost completely avoided by SGD.
When the typical transition graph (see Definition 1 in the main paper) is not irreducible, there will be multiple communication classes on the graph. Suppose that there are K communication classes G1, · · · , GK . From now on, we zoom in on a specific communication class G ∈ {G1, · · · , GK}. For this communication class G, define l∗G =
∆ max{l∗i : i = 1, 2, · · · , nmin; mi ∈ G}. For each local minimum mi ∈ G, we call its attraction field Ωi a large attraction field if l∗i = l∗G, and a small attraction field if l∗i < l ∗ G. We have thus classified all mi in G into two groups: the ones in large attraction fields mlarge1 , · · · ,m large iG and the ones in small attraction fields msmall1 , · · · ,msmalli′G . Also,
define a scaling function λG associated with G as λG(η) = ∆ H(1/η)
( H(1/η)
η
)l∗G−1 .
Theorem F.2. Under Assumptions 1-3, if G is absorbing, then there exists a continuous-time Markov chain Y on {mlarge1 , · · · ,m large iG } such that for any x ∈ Ωi, |x| ≤ L (where i ∈ {1, 2, · · · , nmin}) with mi ∈ G, and
Xηbt/λG(η)c(x)→ Yt(πG(mi)) as η ↓ 0
in the sense of finite-dimensional distributions, where πG is a random mapping satisfying (1) πG(m) ≡ m if m ∈ {mlarge1 , · · · ,m large iG }; (2) πG(m) is a random variable that only takes value in {mlarge1 , · · · ,m large iG } if m ∈ {msmall1 , · · · ,msmalli′G }.
We stress that Theorem 3 in the main paper follows immediately from Theorem F.2 above.
Next, to state the corresponding result for a transient communication class G, we introduce a couple of extra definitions. We consider a version of Xηn that is killed when X η n leaves G. Define stopping time
τG(η) = ∆ min{n ≥ 0 : Xηn /∈ ⋃ i:mi∈G Ωi} (F.4)
as the first time the SGD iterates leave all attraction fields in G, and we use a cemetery state † to construct the following process X†,ηn as a version of X η n with killing at τG:
X†,ηn = { Xηn if n < τG(η), † if n ≥ τG(η).
(F.5)
Theorem F.3. Under Assumptions 1-3, if G is transient, then there exists a continuous-time Markov chain Y with killing that has state space {mlarge1 , · · · ,m large iG
, †} (we say the Markov chain Y is killed when it enters the absorbing cemetery state †) such that for any x ∈ Ωi, |x| ≤ L (where i ∈ {1, 2, · · · , nmin}) with mi ∈ G, and
X†,ηbt/λG(η)c(x)→ Yt(πG(mi)) as η ↓ 0
in the sense of finite-dimensional distributions, where πG is a random mapping satisfying (1) πG(m) ≡ m if m ∈ {mlarge1 , · · · ,m large iG }; (2) πG(m) is a random variable that only takes value in {mlarge1 , · · · ,m large iG , †} if m ∈ {msmall1 , · · · ,msmalli′G }.
To show Theorem F.2 and F.3, we introduce the following concepts. First, we consider the case where the SGD iterates Xηn is initialized on the communication class G and G is absorbing. For some
∆ > 0, η > 0, define (let B(u, v) =∆ [u− v, u+ v])
σG0 (η,∆) = ∆ min{n ≥ 0 : Xηn ∈ ⋃ i: mi∈G B(mi, 2∆)} (F.6)
τG0 (η,∆) = ∆ min{n ≥ σG0 (η,∆) : Xηn ∈ ⋃ i:mi /∈Gsmall B(mi, 2∆)} (F.7)
IG0 (η,∆) = j ⇐⇒ X η τG0 ∈ B(mj , 2∆), ĨG0 (η,∆) = j ⇐⇒ X η σG0 ∈ B(mj , 2∆) (F.8)
σGk (η,∆) = ∆ min{n > τGk−1(η,∆) : Xηn ∈ ⋃ i:mi∈G, i 6=IGk−1 B(mi, 2∆)} ∀k ≥ 1 (F.9)
τGk (η,∆) = ∆ min{n ≥ σGk−1(η,∆) : Xηn ∈ ⋃ i:mi /∈Gsmall B(mi, 2∆)} ∀k ≥ 1 (F.10)
IGk (η,∆) = j ⇐⇒ X η τGk ∈ B(mj , 2∆), ĨGk (η,∆) = j ⇐⇒ X η σGk ∈ B(mj , 2∆)∀k ≥ 1.
(F.11)
Intuitively speaking, at each τGk the SGD iterates visits a minimizer that is not in a small attraction field on G, and we use IGk to mark the label of that large attraction field. Stopping time σ G k is the first time that SGD visits a minimizer that is different from the one visited at τGk , and τ G k+1 is the first time that a minimizer not in a small attraction field of G is visited again since σGk (and including σ G k ). It is worth mentioning that, under this definition, we could have IGk = I G k+1 for any k ≥ 0. Meanwhile, define the following process that only keeps track of the updates on the labels (IGk )k≥0 instead of the information of the entire trajectory of (Xηn)n≥0:
X̂η,∆n =
{ mIGk if ∃k ≥ 0 such that τ G k ≤ n < τGk+1
0 otherwise (F.12)
In other words, when n < τG0 we simply let X̂ η,∆ n = 0, otherwise it is equal to the latest “marker” for the last visited wide minimum up until step n. This marker process X̂ jumps between the different minimizers of the large attractions in G. In particular, if for some n we have Xηn ∈ B(mj , 2∆) for some j with mj ∈ Glarge, then we must have X̂η,∆n = mj , which implies that, in this case, X̂η,∆n indeed indicates the location of Xηn.
Note that results in Theorem F.2 and F.3 concern a scaled version of Xη. Here we also define the corresponding scaled version of the processes
X∗,ηt = ∆ Xηbt/λG(η)c (F.13) X̂∗,η,∆t = ∆ X̂η,∆bt/λG(η)c, (F.14)
a mapping T∗(n, η) =∆ nλG(η) that translates a step n to the corresponding timestamp for the scaled processes, and the following series of scaled stopping times
τ∗k (η,∆) = T ∗(τGk (η,∆), η), σ∗k(η,∆) = T∗(σGk (η,∆), η). (F.15)
Before presenting the proof of Theorem F.2 and F.3, we make several preparations. First, our proof is inspired by ideas in Pavlyukevich (2005) and here we provide a briefing. At any time t > 0, if we can show that X∗,ηt is almost always in set ⋃ i:mi∈Glarge B(mi, 2∆) (so the SGD iterates is almost always close to a minimizer in a large attraction field), then the marker process X̂∗,η,∆t is a pretty accurate indicator of the location of X∗,ηt , so it suffices to show that the marker process X̂ ∗,η,∆ t converges to a continuous-time Markov chain Y .
Second, we construct the limiting process Y and the random mapping πG before utilizing them in Theorem F.2 and F.3. As an important building block for this purpose, we start by considering the following discrete time Markov chain (DTMC) on the entire graph G = (V,E). Let PDTMC be a transition matrix with PDTMC(mi,mj) = µi(Ei,j)/µi(Ei) for all j 6= i, and Y DTMC = (Y DTMCj )j≥0 be the DTMC induced by the said transition matrix. Let
TDTMCG = ∆ min{j ≥ 0 : Y DTMCj /∈ Gsmall} (F.16)
be the first time this DTMC visits a large attraction field on the communication class G, or escapes from G. Lastly, define (for any j such that mj /∈ Gsmall)
pi,j = ∆ P ( Y DTMCTDTMCG (mi) = mj )
(F.17)
as the probability that the first large attraction field on G visited by Y DTMC is mj when initialized at mi.
We add a comment regarding the stopping times TDTMCG and probabilities pi,j defined above. In the case that G is absorbing, we have Y DTMCj (mi) ∈ G for all j ≥ 0 if mi ∈ G. Therefore, in this case, given any i with mi ∈ G, we must have
TDTMCG = min{j ≥ 0 : Y DTMCj (mi) ∈ Glarge}, ∑
j: mj∈Glarge pi,j = 1.
On the contrary, when G is transient we may have ∑ j: mj∈Glarge pi,j < 1 and ∑ j: mj /∈G pi,j > 0. Lastly, whether G is absorbing or transient, we always have pi,j = 1{i = j} if mi ∈ Glarge. Next, consider the following definition of (continuous-time) jump processes.
Definition F.1. A continuous-time process Yt on R is a ( (Uj)j≥0, (Vj)j≥0 ) jump process if
Yt = { 0 if t < U0∑ j≥0 Vj1[U0+U1+···+Uj , U0+U1+···+Uj+1)(t) otherwise ,
where (Uj)j≥0 is a sequence of non-negative random variables such that Uj > 0 ∀j ≥ 1 almost surely, and (Vj)j≥0 is a sequence of random variables in R.
Obviously, the definition above implies that Yt = Vj for any t ∈ [Uj , Uj+1). Now we are ready to construct the limiting continuous-time Markov chain Y . To begin with, we address the case where G is absorbing. For any m′ ∈ Glarge, let Y (m′) be a ( (Sk)k≥0, (Wk)k≥0 ) - jump process where S0 = 0,W0 = m′ and (for all k ≥ 0 and i, j with mi ∈ Glarge,mj /∈ Gsmall)
P ( Wk+1 = mj , Sk+1 > t ∣∣∣Wk = mi, (Wl)k−1l=0 , (Sl)kl=0) (F.18) =P ( Wk+1 = mj , Sk+1 > t ∣∣∣Wk = mi) = exp(−qit)qi,j qi ∀t > 0 (F.19)
where
qi = µi(Ei) (F.20) qi,j = 1{i 6= j}µi(Ei,j) + ∑
k: mk∈Gsmall µi(Ei,k)pk,j (F.21)
and pk,j is defined in eq. (F.17). In other words, conditioning on Wk = mi, the time until next jump Sk+1 and the jump location Wk+1 are independent, where Sk+1 is Exp(qi) and Wk+1 = mj with probability qi,j/qi. First, it is easy to see that Y is a continuous-time Markov chain. Second, under this definition Y is allowed to have some dummy jumps where Wk = Wk+1: in this case the process Yt does not move to a different minimizer after the k + 1-th jump, and by inspecting the path of Y we cannot tell that this dummy jump has occurred. As a result, that generator Q of this Markov chain admits the form (for all i 6= j with mi,mj ∈ Glarge)
Qi,i = − ∑
k:k 6=i, mk∈Glarge qi,k, Qi,j = qi,j .
Moreover, define the following random function πG(·) such that for any mi ∈ G,
πG(mi) = { mj with probability qi,j/qi if mi ∈ Gsmall mi if mi ∈ Glarge
(F.22)
By Y (πG(mi)) we refer to the version of the Markov chain Y where we randomly initialize W0 = πG(mi). The following lemma is the key tool for proving Theorem F.2.
Lemma F.4. Assume that the communication class G is absorbing. Given any mi ∈ G, x ∈ Ωi, finitely many real numbers (tl)k ′
l=1 such that 0 < t1 < t2 < · · · < tk′ , and a sequence of strictly positive real numbers (ηn)n≥1 with limn→0 ηn = 0, there exists a sequence of strictly positive real numbers (∆n)n≥1 with limn ∆n = 0 such that
• As n tends to∞,( X̂∗,ηn,∆nt1 (x), · · · , X̂ ∗,ηn,∆n tk′ (x) ) ⇒ ( Yt1(πG(mi)), · · · , Ytk′ (πG(mi)) ) (F.23)
• For all k ∈ [k′],
lim n→∞
Px ( X∗,ηntk /∈ ⋃ j: mj∈Glarge B(mj ,∆n) ) = 0. (F.24)
Now we address the case where G is transient, let † be a real number such that † /∈ [−L,L], and we use † as the cemetery state since the processes Xηn or X ∗,η t are restricted on [−L,L]. Recall the definition of τG defined in eq. (F.4). Analogous to the process X† in eq. (F.5), we can also define
X†,∗,ηt =
{ X∗,ηt if t < T ∗(τG(η), η)
† otherwise , X̂ †,∗,η,∆ t =
{ X̂∗,η,∆t if t < T ∗(τG(η), η)
† otherwise, . (F.25)
Next, analogous to τG, consider the stopping time
τYG = ∆ min{t > 0 : Yt /∈ G}.
When G is transient, due to the construction of Y we know that τYG < ∞ almost surely. The introduction of τYG allows us to define
Y †t = { Yt if t < τYG † otherwise. (F.26)
The following Lemma will be used to prove Theorem F.3. Lemma F.5. Assume that the communication class G is transient. Given any mi ∈ G, x ∈ Ωi, finitely many real numbers (tl)k ′
l=1 such that 0 < t1 < t2 < · · · < tk′ , and a sequence of strictly positive real numbers (ηn)n≥1 with limn→0 ηn = 0, there exists a sequence of strictly positive real numbers (∆n)n≥1 with limn ∆n = 0 such that
• As n tends to∞,( X̂†,∗,ηn,∆nt1 (x), · · · , X̂ †,∗,ηn,∆n tk′ (x) ) ⇒ ( Y †t1(πG(mi)), · · · , Y † tk′ (πG(mi)) ) (F.27)
• For all k ∈ [k′],
lim n→∞
Px ( X†,∗,ηntk /∈ ⋃ j: mj∈Glarge B(mj ,∆n) and X †,∗,ηn tk 6= † ) = 0. (F.28)
Proof of Theorem F.2 and F.3. We first address the case where G is absorbing. Arbitrarily choose some ∆ > 0, a sequence of strictly positive real numbers (ηn)n≥1 with limn ηn = 0, a positive integer k′, a series of real numbers (tj)k ′ j=1 with 0 < t1 < · · · < tk′ , and a sequence (wj)k ′
j=1 with wj ∈ Glarge for all j ∈ [k′]. It suffices to show that
lim n
Px ( X∗,ηntk ∈ B(wk,∆) ∀k ∈ [k ′] ) = P ( Ytk(πG(mi)) = wk ∀k ∈ [k′] ) .
Using Lemma F.4, we can find a sequence of strictly positive real numbers (∆n)n≥1 with limn ∆n = 0 such that eq. (F.23) and eq. (F.24) hold. From the weak convergence in eq. (F.23), we only need to show
lim n
Px ( X∗,ηntk /∈ B(X̂ ∗,ηn,∆n tk ,∆) ) = 0 ∀k ∈ [k′].
For all n large enough, we have 2∆n < ∆. For such large n, observe that Px ( X∗,ηntk /∈ B(X̂ ∗,ηn,∆n tk ,∆) )
≤Px ( X∗,ηntk /∈ ⋃ j:mj∈Glarge B(mj , 2∆n) )
(due to definition of the marker process X̂ , see eq. (F.6)-eq. (F.11) and eq. (F.12)-eq. (F.14)) ≤Px ( X∗,ηntk /∈ ⋃ j:mj∈Glarge B(mj ,∆n) )
and by applying eq. (F.24) we conclude the proof for Theorem F.2.
The proof of Theorem F.3 is almost identical, with the only modification being that we apply Lemma F.5 instead of Lemma F.4. In doing so, we are able to find a sequence of (∆n)n≥1 with limn ∆n = 0 such that eq. (F.27) and eq. (F.28) hold. Given the weak convergence claim in eq. (F.27), it suffices to show that
lim n
Px ( X†∗,ηntk /∈ B(X̂ †,∗,ηn,∆n tk ,∆) ) = 0 ∀k ∈ [k′].
If X†,∗,ηntk = †, we must have X̂ †,∗,ηn,∆n tk
= † as well. Therefore, for all n large enough so that 3∆n < ∆,
Px ( X†∗,ηntk /∈ B(X̂ †,∗,ηn,∆n tk ,∆) )
=Px ( X†∗,ηntk /∈ B(X̂ †,∗,ηn,∆n tk ,∆), X†∗,ηntk 6= † )
≤Px ( X†∗,ηntk /∈ ⋃ j: mj∈Glarge B(mj , 2∆n), X †∗,ηn tk 6= † )
≤Px ( X†∗,ηntk /∈ ⋃ j: mj∈Glarge B(mj ,∆n), X †∗,ηn tk 6= † ) .
Apply eq. (F.28) and we conclude the proof.

heading: F.1 PROOF OF LEMMA F.1
text: First, we introduce another dichotomy for small and large noises. For any γ̃ > 0 and any learning rate η > 0, we say that a noise Zn is small if
η|Zn| > ηγ̃
and we say Zn is large otherwise. For this new classification of small and large noises, we introduce the following notations and definitions:
Z≤,γ̃,ηn = Zn1{η|Zn| ≤ ηγ̃}, (F.29) Z>,γ̃,ηn = Zn1{η|Zn| > ηγ̃}, (F.30) T̃ η1 (γ̃) = ∆ min{n ≥ 1 : η|Zn| > ηγ̃}. (F.31)
Similar to Lemma E.12, the following result is a direct application of Lemma E.11, and shows that it is rather unlikely to observe large perturbation that are caused only by small noises. Specifically, since α > 1 we can always find
γ̃ ∈ (0, (1− 1 α ∧ 1 2 )
β ∈ ( 1, (2− 2γ̃) ∧ α(1− γ̃) ) .
Now in Lemma E.11, if we let ∆ = γ̃, ∆̃ = ∆/2 and = δ = 1 (in other words, u(η) = 1/η1−γ̃ , v(η) = ηγ̃/2), then for any positive integer j the condition eq. (E.52) is satisfied, allowing us to draw the following conclusion immediately as a corollary from Lemma E.11.
Lemma F.6. Given N > 0 and
γ̃ ∈ (0, (1− 1 α ) ∧ 1 2
), β ∈ ( 1, (2− 2γ̃) ∧ (α− αγ̃) ) ,
we have (as η ↓ 0) P (
max j=1,2,··· ,d(1/η)βe
η|Z≤,γ̃,η1 + · · ·+ Z ≤,γ̃,η j | > η
γ̃/2 )
= o(ηN ).
The flavor of the next lemma is similar to that of Lemma E.13. Specifically, we show that, with high probability, the SGD iterates would quickly return to the local minimum as long as they start from somewhere that are not too close the boundary of an attraction field (namely, the points s1, s2, · · · , snmin ). To this end, we consider a refinement of function t̂(·) defined in eq. (E.38). For any i = 1, 2, · · · , nmin, any x ∈ Ωi and any η > 0, γ ∈ (0, 1), we can define the return time to ηγ−neighborhood for the ODE xη as
t̂(i)γ (x, η) = ∆ min{t ≥ 0 : |xη(t, x)−mi| ≤ ηγ}.
Given the bound in eq. (E.37) (which is stated for a specific attraction field) and the fact that there only exists finitely many attraction fields, we know the existence of some c2 <∞ such that for any i = 1, 2, · · · , nmin, any η > 0, any γ ∈ (0, 1) and any x ∈ Ωi such that |x− si| ∨ |x− si−1| > ηγ , we have
t̂(i)γ (x, η) ≤ c2γ log(1/η)/η
and define function t↑ as
t↑(η, γ) =∆ c2γ log(1/η).
Lastly, define the following stopping time for any i = 1, 2, · · · , nmin, any x ∈ Ωi and any ∆ > 0
T (i) return(η,∆) = ∆ min{n ≥ 0 : Xηn(x) ∈ B(mi, 2∆)}
where we adopt the notation B(u, v) =∆ [u− v, u+ v] for the v−neighborhood around point u. Lemma F.7. Given
γ̃ ∈ (0, (1− 1 α ) ∧ (1 2 )), γ ∈ (0, γ̃ 16Mc2 ∧ γ̃ 4 ),
and any i = 1, 2, · · · , nmin, any ∆ > 0, we have
lim inf η↓0 inf x∈Ωi:|x−si−1|∨|x−si|≥2ηγ
Px ( T (i) return(η,∆) ≤ 2c2γ log(1/η)
η , Xηn ∈ Ωi ∀n ≤ T (i) return(η,∆) ) = 1.
Proof. Throughout this proof, we only consider η small enough such that
2c2γ log(1/η)/η < d(1/η)βe, ηM ≤ η2γ̃ , 2ηγ < ∆/2, 2ηγ̃/4 < ηγ . (F.32) The condition above holds for all η > 0 sufficiently small because β > 1, 2γ̃ < 1, and γ < γ̃/4. Also, fix some β ∈ ( 1, (2− 2γ̃) ∧ (α− αγ̃)
) Define the following events
A×1 (η) = ∆
{ max
j=1,2,··· ,d(1/η)βe η|Z≤,γ̃,η1 + · · ·+ Z ≤,γ̃,η j | > η
γ̃/2 }
A×2 (η) = ∆ {T̃ η1 (γ̃) ≤ d(1/η)βe} (see eq. (F.31) for definition of the stopping time involved)
and fix some N > 0. From Lemma F.6, we see that (for all sufficiently small η)
P(A×1 (η)) ≤ ηN . Besides, using Lemma E.4 together with the fact that β < α(1− γ̃), we know the existence of some constant θ > 0 such that
P(A×2 (η)) ≤ ηθ
for all sufficiently small η. Now we focus on the behavior of the SGD iterates on event ( A×1 (η) ∩A × 2 (η) )c . Let us arbitrarily choose some x ∈ Ωi such that |x− si| ∨ |x− si−1| > 2ηγ . First, from Lemma E.6 and eq. (F.32), we know that
|xηt (x)− y η btc(x)| ≤ 2ηM exp ( 2Mc2γ log(1/η) ) ≤ 2η2γ̃−2Mc2γ ≤ 2ηγ̃ ≤ ηγ ∀t ≤ 2c2γ log(1/η)/η (F.33)
Next, from the definition of the function t↑(·) and eq. (F.33), we know that for
TGD,return(x; η,∆) = ∆ min{n ≥ 0 : yηn(x) ∈ B(mi,
∆ 2 + ηγ)}, (F.34)
we have
TGD,return(x; η,∆) ≤ 2c2γ log(1/η)/η (F.35) yηn > si−1 + η γ , yηn < si − ηγ ∀n ≤ 2c2γ log(1/η)/η. (F.36)
Furthermore, on event ( A×1 (η) ∩A × 2 (η) )c , due to Lemma E.5 and eq. (F.32), we have that
|Xηn(x)− yηn(x)| ≤ ηγ̃/2 exp ( 2Mc2 log(1/η) ) = η γ̃ 2−2Mc2γ ≤ ηγ̃/4 < ηγ ∀n ≤ 2c2γ log(1/η)/η.
Combining this with eq. (F.35), eq. (F.36), we can conclude that (recall that due to eq. (F.32) we have 2ηγ < ∆/2)
T (i) return(η,∆) ≤ 2c2γ log(1/η)/η Xηn ∈ Ωi ∀n ≤ 2c2γ log(1/η)/η
on event ( A×1 (η) ∩A × 2 (η) )c . Therefore,
lim inf η↓0 inf x∈Ωi:|x−si−1|∨|x−si|≥2ηγ
Px ( T (i) return(η,∆) ≤ 2c2γ log(1/η)
η , Xηn ∈ Ωi ∀n ≤ T (i) return(η,∆) ) ≥ lim inf η↓0 P (( A×1 (η) ∩A × 2 (η) )c) ≥ lim inf η↓0 1− ηN − ηθ = 1.
This concludes the proof.
The takeaway of the next lemma is that, almost always, the SGD iterates will quickly escape from the neighborhood of any si, the boundaries of each attraction fields. Lemma F.8. Given any γ ∈ (0, 1), t > 0, we have
lim inf η↓0 inf x∈[−L,L]
Px ( min { n ≥ 0 : Xηn /∈ ⋃ i B(si, 2η γ) } ≤ t H(1/η) ) = 1.
Proof. We only consider η small enough so that
min i=2,3,··· ,nmin−1
|si − si−1| > 3η 1+γ 2 ,
ηM < ηγ .
Also, the claim is trivial if x /∈ ∪jB(sj , 2ηγ), so without loss of generality we only consider the case where there is some j ∈ [nmin] and x ∈ [−L,L], x ∈ B(sj , 2ηγ). Let us define stopping times
T γ =∆ min{n ≥ 1 : η|Zn| > 5ηγ}; (F.37) T γescape =
∆ min{n ≥ 0 : Xηn /∈ ∪jB(sj , 2ηγ)}, (F.38) and the following two events
A×1 (η) = ∆ {T γ > t
H(1/η) },
A×2 (η) = ∆ {η|ZTγ | > η 1+γ 2 }.
First, using Lemma E.3 and the regularly varying nature of H(·) = P(|Z1| > ·), we know the existence of some θ > 0 such that
P(A×1 (η)) ≤ exp(−1/ηθ)
for all η > 0 sufficiently small. Next, by definition of T γ , one can see that (for any η ∈ (0, 1))
P(A×2 (η)) = H(1/η
1−γ 2 )
H(5/η1−γ) .
Again, due to H ∈ RV−α and 1− γ > 0, we know the existence of some θ1 > 0 such that
P(A×2 (η)) < η θ1
for all η > 0 sufficiently small. To conclude the proof, we only need to note the following fact on event ( A×1 (η) ∪ A × 2 (η) )c . There are only two possibilities on this event: T γescape ≤ T γ − 1, or T γescape ≥ T γ . Now we analyze the two cases respectively.
• On ( A×1 (η) ∪A × 2 (η) )c ∩ {T γescape ≤ T γ − 1}, we must have T γescape < T γ ≤ t/H(1/η). • On ( A×1 (η) ∪ A × 2 (η)
)c ∩ {T γescape ≥ T γ}, we know that at n = T γ − 1, there exists an integer j ∈ {1, 2, · · · , nmin − 1} such that Xηn ∈ B(sj , 2ηγ). Now since ηM < ηγ and η|ZTγ | > 5ηγ , we must have
|XηTγ −X η Tγ−1| > 4η γ ⇒ XηTγ /∈ B(sj , 2η γ).
On the other hand, the exclusion of event A×2 (η) tells us that |X η Tγ − X η Tγ−1| < 2η 1+γ 2 . Due to eq. (F.37), we then have Xηn /∈ ∪iB(si, 2ηγ).
In summary, ( A×1 (η) ∪A × 2 (η) )c ⊆ {T γreturn ≤ t/H(1/η)} and this conclude the proof. In the next lemma, we analyze the number of transitions needed to visit a certain local minimizer in the loss landscape. In general, we focus on a communication class G and, for now, assume it is absorbing. Next, we introduce the following concepts to record the transitions between different local minimum. To be specific, for any η > 0 and any ∆ > 0 small enough so that B(mj ,∆) ∩ Ωcj = ∅ for all j, define
T0(η,∆) = min{n ≥ 0 : Xηn ∈ ∪jB(mj , 2∆)}; (F.39) I0(η,∆) = j iff X
η T0(η,∆) ∈ B(mj , 2∆); (F.40) Tk(η,∆) = min{n > Tk−1(η,∆) : Xηn ∈ ∪j 6=Ik−1(η,∆)B(mj , 2∆)} ∀k ≥ 1 (F.41) Ik(η,∆) = j iff X
η Tk(η,∆) ∈ B(mj , 2∆) ∀k ≥ 1. (F.42)
As mentioned earlier, the next goal is to analyze the transitions between attraction fields it takes to visit mj when starting from mi when mi,mj ∈ G. Define
Ki(η,∆) = ∆ min{k ≥ 0 : Ik(η,∆) = i}.
Lemma F.9. Assume that G is an absorbing communication class on the graph G. Then there exists some constant p > 0 such that for any i with mi ∈ G, any > 0, and any ∆ > 0,
sup j: mj∈G; x∈B(mj ,2∆)
Px(Ki(η,∆) > u · nmin) ≤ P(Geom(p) ≥ u) + ∀u = 1, 2, · · · ,
sup j: mj∈G; x∈B(mj ,2∆)
Px ( ∃k ∈ [Ki(η,∆)] s.t. mIk(η,∆) /∈ G ) ≤
hold for all η > 0 sufficiently small.
Proof. The claim is trivial if, for the initial condition, we have x ∈ B(mi, 2∆). Next, let us observe the following facts.
• Define (recall the definitions of measure µi and sets Ei, Ei,j in eq. (E.1)eq. (E.5)eq. (E.6))
J(j) =∆ arg min j̃:µi(Ej,j̃)>0 |i− j̃| ∀j 6= i
p∗ =∆ min j: j 6=i, mj∈G
µj(Ej,J(j))
µj(Ej) .
• From the definition of J(j) and the fact that there are only finitely many attraction fields we can see that p∗ > 0. Moreover, G being a communication class implies that
|J(j)− i| < |j − i| ∀j 6= i, mj ∈ G. Indeed, if i < j, then since G is a communication class and there are some mi ∈ G with i < j, we will at least have µj(Ej,j−1) > 0, so |J(j)− i| ≤ |i− j| − 1; the case that i > j can be approached analogously.
• Now from the definition of J(j) and Proposition E.25, together with the previous bullet point, we know that for all η sufficiently small,
inf x∈[−L,L]
Px(|Ik+1 − i| ≤ |Ik − i| − 1, mIk+1 ∈ G |Ki(η,∆) > k, mIk ∈ G) ≥ p∗/2
uniformly for all k ≥ 0.
• Meanwhile, since p∗ > 0, we are able to fix some δ > 0 small enough such that nminδ
(p∗/2)nmin < .
• On the other hand, for any j̃ with mj̃ /∈ G, by definition of the typical transition graph we must have µj(Ej,̃j) = 0 for any j with mj ∈ G. Then due to Proposition E.25 again, one can see that for all η > 0 that is sufficiently small,
sup x∈[−L,L] Px(mIk+1 /∈ G |Ki(η,∆) > k, mIk ∈ G) < δ
uniformly for all k ≥ 0.
• Repeat this argument for nmin times, and we can see that for all η sufficiently small
inf x∈[−L,L]
Px(Ki(η,∆) ≤ k + nmin |Ki(η,∆) > k, mIk ∈ G) ≥ (p∗
2 )nmin sup
x∈[−L,L] Px(∃l ∈ [nmin] s.t. mIk+l /∈ G |Ki(η,∆) > k, mIk ∈ G) ≤ nminδ
uniformly for all k ≥ 1.
• Lastly, to apply the bounds established above, we will make use of the following expression of several probabilities. For any j with mj ∈ G and x ∈ B(mj , 2∆) and any u = 1, 2, · · · ,
Px(∃l ∈ [unmin] s.t. mIl /∈ G, Ki(η,∆) > u · nmin)
= u−1∑ v=0 Px ( ∃k ∈ [nmin] s.t. mIk+vnmin /∈ G ∣∣∣Ki(η,∆) > v · nmin, mIl ∈ G ∀l ≤ vnmin) · v−1∏ w=0 Px ( Ki(η,∆) > (w + 1)nmin, mIk+wnmin ∈ G ∀k ∈ [nmin]
∣∣∣ Ki(η,∆) > w · nmin, mIl ∈ G ∀l ≤ wnmin
) Px(mIl ∈ G ∀l ∈ [unmin], Ki(η,∆) > u · nmin)
= u−1∏ v=0 P ( Ki(η,∆) > (v + 1)nmin, mIk+vnmin ∈ G ∀k ∈ [nmin] ∣∣∣ Ki(η,∆) > vnmin, mIk ∈ G ∀k ∈ [vnmin] )
In summary, now we can see that (for sufficiently small η)
sup j: mj∈G; x∈B(mj ,2∆)
Px ( ∃k ∈ [Ki(η,∆)] s.t. mIk(η,∆) /∈ G ) ≤ ∞∑ u=0 sup j: mj∈G; x∈B(mj ,2∆) Px ( ∃v ∈ [nmin] such that mIv+unmin /∈ G,
Ki(η,∆) > unmin,mIk ∈ G ∀k ∈ [unmin] )
≤ ∞∑ u=0 sup j: mj∈G; x∈B(mj ,2∆) Px ( ∃v ∈ [nmin] such that mIv+unmin /∈ G ∣∣∣ Ki(η,∆) > unmin,mIk ∈ G ∀k ∈ [unmin]
) · u−1∏ v=0 sup j: mj∈G; x∈B(mj ,2∆) Px ( Ki(η,∆) > (v + 1)nmin,mIk+vnmin ∈ G ∀k ∈ [nmin]
∣∣∣ Ki(η,∆) > vnmin,mIk ∈ G ∀k ∈ [vnmin]
) ≤ ∑ u≥0 nminδ ( 1− (p ∗ 2 )nmin )u−1 = nminδ (p∗/2)nmin ≤
and sup
j:mj∈G, x∈B(mj ,2∆) Px(Ki(η,∆) > u · nmin)
≤ sup j:mj∈G, x∈B(mj ,2∆) Px(∃l ∈ [unmin] s.t. mIl /∈ G, Ki(η,∆) > u · nmin)
+ sup j:mj∈G, x∈B(mj ,2∆) Px(mIl ∈ G ∀l ∈ [unmin], Ki(η,∆) > u · nmin)
≤ u∑ v=1 nminδ ( 1− (p ∗ 2 )nmin )v−1 + ( 1− (p ∗ 2 )nmin )u ≤ nminδ (p∗/2)nmin + ( 1− (p ∗ 2 )nmin
)u ≤ + ( 1− (p ∗
2 )nmin )u uniformly for all u = 1, 2, · · · . To conclude the proof, it suffices to set p = (p ∗
2 ) nmin .
The proof above can be easily adapted to the case when the communication class G is transient. Define
KGi (η,∆) = ∆ min{k ≥ 0 : Ik(η,∆) = i or mIk(η,∆) /∈ G}.
Lemma F.10. Assume that G is a transient communication class on the graph G. Then there exists some constant p > 0 such that for any i with mi ∈ G and any ∆ ∈ (0, ̄/3),
sup j: mj∈G; x∈B(mj ,2∆)
Px(KGi (η,∆) > u · nmin) ≤ P(Geom(p) ≥ u) ∀u = 1, 2, · · · (F.43)
hold for all η > 0 sufficiently small.
Proof. The structure of this proof is analogous to that of Lemma F.9. Again, the claim is trivial if, for the initial condition, we have x ∈ B(mi, 2∆). Next, let us observe the following facts.
• Define (recall the definitions of measure µi and sets Ei, Ei,j in eq. (E.1)eq. (E.5)eq. (E.6))
J(j) =∆ arg min j̃:µi(Ej,j̃)>0 |i− j̃| ∀j 6= i
p∗ =∆ min j: j 6=i, mj∈G
µj(Ej,J(j))
µj(Ej) .
• From the definition of J(j) and the fact that there are only finitely many attraction fields we can see that p∗ > 0. Moreover, G being a communication class implies that
|J(j)− i| < |j − i| ∀j 6= i, mj ∈ G.
Indeed, if i < j, then since G is a communication class and there are some mi ∈ G with i < j, we will at least have µj(Ej,j−1) > 0, so |J(j)− i| ≤ |i− j| − 1; the case that i > j can be approached analogously.
• Now from the definition of J(j) and Proposition E.25, together with the previous bullet point, we know that for all η sufficiently small,
inf x∈[−L,L]
Px(|Ik+1 − i| ≤ |Ik − i| − 1, mIk+1 ∈ G |KGi (η,∆) > k ≥ p∗/2
uniformly for all k ≥ 0.
• Repeat this argument for nmin times, and we can see that for all η sufficiently small
inf x∈[−L,L]
Px(KGi (η,∆) ≤ k + nmin |KGi (η,∆) > k) ≥ (p∗
2 )nmin uniformly for all k ≥ 1.
• Lastly, for any j 6= i with mj ∈ G and x ∈ B(mj , 2∆) and any u = 1, 2, · · · ,
Px(KGi (η,∆) > u · nmin)
= u−1∏ v=0 Px ( KGi (η,∆) > (v + 1)nmin ∣∣∣KGi (η,∆) > v · nmin) =
u−1∏ v=0 ( 1− Px ( KGi (η,∆) ≤ (v + 1)nmin ∣∣∣KGi (η,∆) > v · nmin))
In summary, now we can see that (for suffciently small η)
sup j:mj∈G, x∈B(mj ,2∆)
Px(KGi (η,∆) ≥ u · nmin) ≤ ( 1− (p ∗
2 )nmin )u uniformly for all u = 1, 2, · · · . To conclude the proof, it suffices to set p = (p ∗
2 ) nmin .
We are now ready to prove Lemma F.1, which, as demonstrated earlier, is the key tool in proof of Theorem 2.
Proof of Lemma F.1. The claim is trivial if llarge = 1, so we focus on the case where llarge ≥ 2. Fix some
γ̃ ∈ (0, (1− 1 α ) ∧ (1 2
)), β ∈ ( 1, (2− 2γ̃) ∧ (α− αγ̃) ) , γ ∈ (0, γ̃
16Mc2 ∧ γ̃ 4 ).
Let q∗ = maxj µj(Ej(0)). We show that for any t ∈ (0, δ4q∗ ) the claim is true.
Now we only consider ∆ ∈ (0, ̄/3) and η small enough so that ηM ≤ ηγ and ηγ < ∆. Consider the following stopping times
T γescape = ∆ min{n ≥ 0 : Xηn /∈ ∪jB(sj , 2ηγ)}; T γreturn = ∆ min{n ≥ 0 : Xηn ∈ ∪jB(mj , 2ηγ)}.
First, from Lemma F.8, we know that
sup x∈[−L,L]
Px(T γescape > 1/H(1/η)) < δ/2
for all η sufficiently small. Besides, by combining Lemma F.7 with Markov property (applied at T γescape), we have
sup x∈[−L,L]
Px ( T γreturn − T γescape > 2c2γ log(1/η)/η ∣∣∣ T γescape ≤ 1H(1/η)) < δ/2 for all η sufficiently small. Therefore, for all η sufficiently small,
sup x∈[−L,L]
P ( T γreturn >
1
H(1/η) + 2c2γ
log(1/η)
η
) < δ. (F.44)
Let J be the unique index such that Xη Tγreturn ∈ ΩJ . Our next goal is to show that, almost always, the SGD iterates will visit the local minimum at some large attraction fields. Therefore, without loss of generality, we can assume that mJ /∈M large, and define
T γlarge = ∆ min{n ≥ T γreturn : Xηn ∈ ⋃ i:mi∈M large B(mi, 2∆)}
and introduce the following definitions:
τ0 = ∆ T γreturn, J0 = ∆ J
τk = ∆ min{n > τk−1 : Xηn ∈ ⋃ j 6=Jk−1 B(mj , 2∆)}
Jk = j ⇔ Xητk ∈ Ωj ∀k ≥ 1 K =∆ min{k ≥ 0 : mJk ∈M large}.
In other words, the sequence of stopping times (τk)k≥1 is the time that, starting from T γ return, the SGD iterates visited a local minimum that is different from the one visited at τk−1, and (Jk)k≥0 records the label of the visited local minima. The random variable K is the number of transitions required to visit a minimizer in a large attraction field. From Lemma F.9, we know the existence of some p∗ > 0 such that (for all η sufficiently small)
sup x∈[−L,L]
Px(K ≥ u · nmin) ≤ P ( Geom(p∗) ≥ u ) + δ
2 ∀u = 1, 2, 3, · · · .
where Geom(a) is a Geometric random variable with success rate a ∈ (0, 1). Therefore, one can find integer N(δ) such that (for all sufficiently small η)
sup x∈[−L,L]
P(K ≥ N(δ)) ≤ δ. (F.45)
Next, given results in Proposition E.25 and the fact that there are only finitely many attraction fields, one can find a real number u(δ) such that (for all sufficiently small η)
sup x∈[−L,L]
Px(τk − τk−1 ≤ u(δ)
λJk−1(η) ) ≤ δ/N(δ) (F.46)
uniformly for all k = 1, 2, · · · , N(δ). From eq. (F.44), eq. (F.45),eq. (F.46), we now have
sup x∈[−L,L]
Px ( Xηn /∈ ⋃ j:mj∈M large B(mj , 2∆) ∀n ≤ N(δ)u(δ) H(1/η)/η λlarge(η) +
1
H(1/η) + 2c2γ
log(1/η)
η ) ≤ 3δ (F.47)
for any sufficiently small η. To conclude the proof we just observe the following facts. First, due to H ∈ RV−α and llarge ≥ 2, we have
lim η↓0 H(1/η)/η = 0, lim η↓0
λlarge(η) H(1/η) = 0, lim η↓0 log(1/η) η λlarge(η) = 0.
Therefore, for sufficiently small η, we will have (note that , δ are fixed constants in this proof, so N(δ), u(δ) are also fixed)
N(δ)u(δ)H(1/η)/ηλlarge(η) + 1 H(1/η) + 2c2γ log(1/η) η
bt/λlarge(η)c ≤ . (F.48)
Second, recall that we fixed some t ∈ (0, δ4q∗ ) where q ∗ = maxj µj(Ej). Also, choose some C > 0 small enough so that
C < δ/2, 2(1 + C)2 < 4.
From Proposition E.23 and the fact that there are only finitely many attraction fields, there exists some η̄0 > 0 such that for any η ∈ (0, η̄0) and any ∆ > 0 sufficiently small,
sup i:mi∈M large sup x∈[mi−2∆,mi+2∆]
Px ( σi(η) ≤
t
λlarge(η) ) ≤ sup i:mi∈M large sup x∈[mi−2∆,mi+2∆] Px ( µi(Ei)λ large(η)σi(η) ≤ q∗t ) ≤C + 2(1 + C)2q∗t ≤ 2δ.
Combine this bound with Markov property (applied at τK), and we obtain that
sup x∈[−L,L]
Px ( ∃n ∈ [ bt/λlarge(η)c ] s.t. Xηn+τK /∈ ⋃ i:mi∈M large Ωi ) ≤ 2δ
for all η sufficiently small. Together with eq. (F.47)eq. (F.48), we have shown that
sup x∈[−L,L]
Px ( V small(η, , t) > ) ≤ 5δ
holds for all η sufficiently small.

heading: F.2 PROOF OF LEMMA F.4, F.5
text: We shall return to the discussion about the dynamics of SGD iterates on a communication class G. Recall that
Glarge = {mlarge1 , · · · ,m large iG }, Gsmall = {msmall1 , · · · ,msmalli′G }.
If Xηn is initialized at some sharp minimum on G, then we are interested in the behavior of X η n at the first visit to some large attraction fields on G. Define
TG(η,∆) = ∆ min{n ≥ 0 : Xηn ∈ ⋃ i: mi∈Glarge B(mi, 2∆) or Xηn /∈ ∪i: mi∈GΩi}. (F.49)
Not only is this definition of TG analogous to the one for TDTMCG in eq. (F.16), but, as illustrated in the next lemma, TG also behaves similarly as TG on a communication class G in the following sense: the probabilities pi,j defined in eq. (F.17) govern the dynamics regarding which large attraction field on G is the first one to be visited. Besides, TG is usually rather small, meaning that the SGD iterates would efficiently arrive at a large attraction field on G or simply escape from G. Lemma F.11. Given any θ ∈ (0, (α− 1)/2), ∈ (0, 1), i, j ∈ [nmin] such that mi ∈ Gsmall, mj /∈ Glarge, the following claims hold for all ∆ > 0 that is sufficiently small:
lim sup η↓0 sup x∈B(mi,2∆)
Px ( TG(η,∆) ≤ ηθ
λG(η) , XηTG ∈ B(mj , 2∆)
) ≤ pi,j + 5 ,
lim inf η↓0 inf x∈B(mi,2∆)
Px ( TG(η,∆) ≤ ηθ
λG(η) , XηTG ∈ B(mj , 2∆)
) ≥ pi,j − 5 ,
lim sup η↓0 sup x∈B(mi,2∆)
Px ( TG(η,∆) > ηθ
λG(η)
) ≤ 2 .
Proof. For Gsmall 6= ∅ to hold (and the discussion to be meaningful), we must have l∗G ≥ 2. Throughout the proof, we assume this is the case. Besides, we require that ∆ ∈ (0, ̄/3) so we have
B(mi, 3∆) ∩ Ωci = ∅ ∀i ∈ [nmin]
and the 3∆-neighborhood of each local minimum will not intersect with each other. In this proof we will only consider ∆ in this range.
From Lemma F.9 (if G is absorbing) or Lemma F.10 (if G is transient), we know the existence of some integer N( ) such that for (see the definition of Ik in eq. (F.39)-eq. (F.42))
NG(η,∆) = ∆ min{k ≥ 0 : mIk(η,∆) ∈ G large or mIk(η,∆) /∈ G}, we have
sup x∈B(mi,2∆)
Px ( NG(η,∆) > N( ) ) <
for all η sufficiently small. Fix such N( ). Next, from Proposition E.25, we can find u( ) ∈ (0,∞) and ∆̄ ∈ (0, ̄/3) such that for all ∆ ∈ (0, ∆̄), we have
sup x∈B(mi,2∆)
Px ( Tk(η,∆)− Tk−1(η,∆) > u( )/Λ ( Ik−1(η,∆), η )) ≤ /N( ) ∀k ∈ [N( )]
for all η sufficiently small. Fix such u( ) and ∆̄. Now note that on the event A =∆ { NG ≤ N( ) } ∩ { Tk(η,∆)− Tk−1(η,∆) ≤ u( )/Λ ( Ik−1(η,∆), η ) ∀k ∈ [N( )] } ,
due to the choice of θ ∈ (0, (α− 1)/2) and H ∈ RV−α, we have (when η ∈ (0, 1))
Tk(η,∆)− Tk−1(η,∆) ≤ η2θ
λG(η) ∀k < NG(η,∆)
⇒TG(η,∆) = TNG(η,∆)(η,∆) ≤ N( )u( ) η2θ
λG(η) .
For any η sufficiently small, we will have N( )u( ) η 2θ
λG(η) < η
θ
λG(η) . In summary, we have established
that for all ∆ ∈ (0, ∆̄),
lim sup η↓0 sup x∈B(mi,2∆)
Px ( TG > ηθ
λG(η)
) ≤ lim sup
η↓0 sup x∈B(mi,2∆) Px(Ac) < 2 . (F.50)
Next, let S( ) =∆ {( m′1, · · · ,m′N( ) ) ∈ {m1, · · · ,mnmin}N( ) : ∃k ∈ [N( )] s.t. m′k = mj } .
We can see that S( ) contains all the possible transition path for Y DTMC where the state mj is visited within the first N( ) steps. Obviously, |S( )| < ∞. Let 1 = /|S( )|. If we are able to show the existence of some ∆̄1 > 0 such that for all ∆ ∈ (0, ∆̄1), the following claim holds for any (m′k) N( ) k=1 ∈ S( ):
lim sup η↓0 sup x∈B(mi,2∆) ∣∣∣Px(mIk = m′k ∀k ∈ [N( )])− P(Y DTMCk (mi) = m′k ∀k ∈ [N( )])∣∣∣ < 1, (F.51)
then we must have (for all ∆ ∈ (0, ∆̄ ∧ ∆̄1))
lim sup η↓0 sup x∈B(mi,2∆) ∣∣∣Px(XηTG ∈ B(mj , 2∆))− pi,j∣∣∣ = lim sup
η↓0 sup
x∈B(mi,2∆) ∣∣∣Px(XηTG ∈ B(mj , 2∆), TG ≤ N( ))+ Px(XηTG ∈ B(mj , 2∆), TG > N( )) − P ( Y DTMCTDTMCG (mi) = mj , T DTMC G ≤ N( ) ) − P ( Y DTMCTDTMCG (mi) = mj , T DTMC G > N( )
)∣∣∣ ≤ lim sup
η↓0 sup
x∈B(mi,2∆) ∣∣∣Px(XηTG ∈ B(mj , 2∆), TG ≤ N( )) − P ( Y DTMCTDTMCG (mi) = mj , T DTMC G ≤ N( )
)∣∣∣ + lim sup
η↓0 sup x∈B(mi,2∆) Px(TG > N( )) + P(TDTMCG (mi) > N( ))
≤|S( )| 1 + lim sup η↓0 sup x∈B(mi,2∆) Px(TG > N( )) + P(TDTMCG (mi) > N( ))
≤3 .
To show that eq. (F.51) is true, we fix some (m′k) N( ) k=1 ∈ S( ) and let (k ′(k)) N( ) k=1 be the sequence with mk′(k) = m′k for each k ∈ [N( )]. From the definition of Y DTMC we have (let k ′(0) = i)
P ( Y DTMCk (mi) = m ′ k ∀k ∈ [N( )] ) = N( )−1∏ k=0 µk′(k)(Ek′(k),k′(k+1)) µk′(k)(Ek′(k)) .
On the other hand, using Proposition E.25, we know that for any arbitrarily chosen ′ ∈ (0, 1), we have
lim sup η↓0 sup x∈B(mi,2∆)
Px ( mIk(η,∆) = m ′ k ∀k ∈ [N( )] ) ≤ N( )−1∏ k=0 µk′(k)(Ek′(k),k′(k+1)) µk′(k)(Ek′(k)) · (1 + ′),
lim inf η↓0 inf x∈B(mi,2∆)
Px ( mIk(η,∆) = m ′ k ∀k ∈ [N( )] ) ≥ N( )−1∏ k=0 µk′(k)(Ek′(k),k′(k+1)) µk′(k)(Ek′(k)) · (1− ′),
for all ∆ > 0 sufficiently small. The arbitrariness of ′ > 0, together with |S( )| < ∞, allows us to see the existence of some ∆̄1 > 0 such that with ∆ ∈ (0, ∆̄1), eq. (F.51) holds for any (m′k) N( ) k=1 ∈ S( ). To conclude the proof, observe that
lim sup η↓0 sup x∈B(mi,2∆) ∣∣∣Px(XηTG ∈ B(mj , 2∆))− Px(TG(η,∆) ≤ ηθλG(η) , XηTG ∈ B(mj , 2∆) )∣∣∣
≤ lim sup η↓0 sup x∈B(mi,2∆)
Px ( TG > ηθ
λG(η)
) <
due to eq. (F.50).
Recall that continuous-time process X∗,η is the scaled version of Xη defined in eq. (F.13), and the mapping T∗(n, η) =∆ nλG(η) returns the timestamp t for X∗,ηt corresponding to the unscaled step n on the time horizon of Xηn. As an inverse mapping of T
∗, we define the mapping N∗(t, η) = bt/λG(η)c that maps the scaled timestamp t back to the step number n for the unscaled process Xη . In the next lemma, we show that, provided that X∗,η stays on a communication class G before some time t, the scaled process X∗,ηt is almost always in the largest attraction fields of a communication class G. Lemma F.12. Let G be a communication class on the graph G. Given any 1 > 0, t > 0 and any x ∈ Ωi with mi ∈ G, the following claim holds for all ∆ > 0 small enough:
lim sup η↓0
Px ({ X∗,ηt /∈ ⋃ j:mj∈Glarge B(mj , 3∆) } ∩ { X∗,ηs ∈ ⋃ k: mk∈G Ωk ∀s ∈ [0, t] }) ≤ 2 1.
Proof. Let ∆ ∈ (0, ̄/3) for the constant ̄ in eq. (E.28)eq. (E.29), so we are certain that each B(mi, 2∆) lies entirely in Ωi and would not intersect with each other since
B(mi, 3∆) ∩ Ωci = ∅ ∀i ∈ [nmin].
Besides, with = ∆/3, we know the existence of some δ > 0 such that claims in Lemma E.14 would hold of the chosen , δ. Fix such δ for the entirety of this proof. Lastly, fix some
γ̃ ∈ (0, (1− 1 α ) ∧ (1 2
)), β ∈ ( 1, (2− 2γ̃) ∧ (α− αγ̃) ) , γ ∈ (0, γ̃
16Mc2 ∧ γ̃ 4 ).
The blueprint of this proof is as follows. We will define a sequence of stopping times (Nj)6j=1 such that the corresponding scaled timestamps T∗j = T
∗(Nj , η) gradually approach t. By analyzing the behavior of X∗,η on a time interval [t − ∆t, t] that is very close to t (in particular, on the aforementioned stopping times T∗j ), we are able to establish the properties of a series of events A1 ⊇ A2 ⊇ A3. Moreover, we will show that A3 ⊆ {X∗,η,∆t ∈ ⋃ i: mi∈Glarge B(mi, 3∆)}, so the properties about events A,A2, A3 can be used to bound the probability of the target event.
Arbitrarily choose some ∆t ∈ (0, t). To proceed, let N0 =∆ N∗(t − ∆t, η) be the stopping time corresponding to timestamp t − ∆t for the scaled process. Using Lemma F.8, we know that for stopping time N1 = ∆ min{n ≥ N0 : Xηn /∈ ∪jB(sj , 2ηγ)}, we have
lim inf η↓0 inf x∈[−L,L]
Px(N1 −N0 < ∆t/4
H(1/η) ) = 1.
Next, let N2 = ∆ min{n ≥ N1 : Xηn ∈ ∪jB(mj , 2∆)}. From Lemma F.7 and H ∈ RV−α (so that log(1/η)/η = o(H(1/η)), we have
lim inf η↓0 inf x∈[−L,L]
Px(N2 −N1 < ∆t/4
H(1/η) ) = 1.
Collecting results above, we have
lim inf η↓0 inf x∈[−L,L]
Px(N2 −N0 < ∆t/2
H(1/η) ) = 1. (F.52)
Now note the following fact on the event {N2 − N0 < ∆t/2H(1/η)}. The definition of the mapping T∗ implies that, for any pair of positive integers n1 ≤ n2, we have T∗(n2, η) − T∗(n1, η) = (n2 − n1)λG(η) ≤ (n2 − n1) ·H(1/η). Therefore, on {N2 −N0 ≤ ∆t/2H(1/η)} we have
T∗(N2, η)− T∗(N0, η) < ∆t/2⇒ T∗(N2, η) < t− ∆t 2 .
Besides, let T∗2 = T ∗(N2, η). Now we can see that for event
A0 = ∆ { X∗,ηs ∈ ⋃ k: mk∈G Ωk ∀s ∈ [0, t] } ∩ { N2 −N0 < ∆t/2 H(1/η) } ,
we have
A0 ⊆ A1 =∆ {
T∗2 < t− ∆t 2
} ∩ { X∗,ηs ∈ ⋃ k: mk∈G Ωk ∀s ∈ [0,T∗2] } .
Meanwhile, from eq. (F.52) we obtain that
lim sup η↓0 sup x∈[−L,L]
Px ( Ac1 ∩ {X∗,ηs ∈ ⋃ k: mk∈G Ωk ∀s ∈ [0, t]} ) = 0. (F.53)
Moving on, we consider the following stopping times
N3 = ∆ min{n ≥ N2 : Xηn ∈ ⋃ j: mj∈Glarge B(mj , 2∆) or Xηn /∈ ⋃ j: mj∈G Ωj}, T∗3 = ∆ T∗(N3, η).
Using Lemma F.11, we have
lim sup η↓0
Px ( N3 −N2 > ∆t/4
λG(η) ∣∣∣ A1) ≤ 1. (F.54) Meanwhile, on event A1 ∩ { N3 −N2 ≤ ∆t/4λG(η) } ∩ { X∗,ηs ∈ ⋃ k: mk∈G Ωk ∀s ∈ [0, t] } , we have T∗3 − T ∗ 2 ≤ ∆t/4, hence T ∗ 3 ∈ [t−∆t, t−∆t/4]. In summary,
A1 ∩ { N3 −N2 ≤ ∆t/4
λG(η)
} ∩ { X∗,ηs ∈ ⋃ k: mk∈G Ωk ∀s ∈ [0, t] }
⊆ { T∗3 ∈ [t−∆t, t−∆t/4] } ∩ { X∗,ηs ∈ ⋃ k: mk∈G Ωk ∀s ∈ [0,T∗3] }
Moreover, on event { X∗,ηs ∈ ⋃ k: mk∈G Ωk ∀s ∈ [0, t] } , if we let J3 be label of the local minimum visited at T∗3 such that J3 = j ⇐⇒ X ∗,η T∗3 ∈ B(mj , 2∆), then we must have mJ3 ∈ Glarge. Meanwhile, consider the following stopping times
Tσ =∆ min{s > T∗3 : X∗,ηs /∈ ΩJ3}. From Proposition E.25, we know that
lim sup η↓0
Px ( Tσ − T∗3 ≤ ∆t ∣∣∣ {T∗3 ∈ [t−∆t, t−∆t/4]} ∩ {X∗,ηs ∈ ⋃
k: mk∈G
Ωk ∀s ∈ [0,T∗3] })
≤ 1 + 1− exp ( − (1 + 1)q∗∆t ) (F.55)
where q∗ = maxj µj(Ej). Now we define the event
A2 = ∆ A1 ∩ { N3 −N2 ≤ ∆t/4
λG(η)
} ∩ { Tσ − T∗3 > ∆t } ∩ { X∗,ηs ∈ ⋃ k: mk∈G Ωk ∀s ∈ [0,T∗3] } .
Using eq. (F.53)-eq. (F.55), we get
lim sup η↓0 sup x∈[−L,L]
Px ( Ac2 ∩ {X∗,ηs ∈ ⋃ k: mk∈G Ωk ∀s ∈ [0, t]} ) ≤ 2 1 + 1− exp ( − (1 + 1)q∗∆t ) .
(F.56)
Furthermore, on event A2, due to T∗3 ∈ [t−∆t, t−∆t/4] as established above, we must have X∗,ηs ∈ ΩJ3 ∀s ∈ [T ∗ 3, t].
Now let us focus on a timestamp T∗4 = t− ∆t/8 H(1/η)λG(η) and N4 = N ∗(T∗4, η) . Obviously, T ∗ 4 > T ∗ 3 on event A2. Next, define
N5 = ∆ min{n ≥ N4 : Xηn ∈ ⋃ j B(mj , 2∆)} T∗5 = ∆ T∗(N5, η).
Using Lemma F.7 and F.8 again as we did above when obtaining eq. (F.52), we can show that
lim sup η↓0
Px ( N5 −N4 > ∆t/16
H(1/η)
) = 0. (F.57)
On the other hand, on event A2 ∩ { N5 −N4 ≤ ∆t/16H(1/η) } we must have
• T∗5 − T ∗ 4 ≤ ∆t/16 H(1/η)λG(η), so T ∗ 5 ∈ [t− ∆t/8 H(1/η)λG(η), t− ∆t/16 H(1/η)λG(η)];
• X∗,ηT∗5 ∈ ΩJ3 , due to T σ − T∗3 > ∆t.
This implies that for event Ã =∆ { T∗5 ∈ [t− ∆t/8
H(1/η) λG(η), t−
∆t/16
H(1/η) λG(η)], X ∗,η T∗5 ∈ ΩJ3
} ∩{X∗,ηs ∈ ⋃ k: mk∈G Ωj ∀s ∈ [0,T∗5]},
we have A2 ∩ { N5 −N4 ≤ ∆t/16H(1/η) } ⊆ Ã. Lastly, observe that
• From Lemma E.4, we know that for N6(δ) = ∆ min{n > N5 : η|Zn| > δ} we have
lim sup η↓0
P ( N6(δ)−N5 ≤ ∆t/H(1/η) ) ≤ ∆t/δα;
• As stated at the beginning of the proof, our choice of δ allows us to apply Lemma E.14 and show that
lim sup η↓0 sup x∈[−L,L]
Px ( ∃n = N5, · · · , N6 − 1 s.t. Xηn /∈ B(mJ3 , 3∆) | Ã ) = 0;
• Combining the two bullet points above, we get
lim sup η↓0
Px ( ∃s ∈ [T∗5, t] such that X∗,ηs /∈ B(mJ3 , 3∆) ∣∣∣ Ã) ≤ ∆t/δα. (F.58) On the other hand,
Ã ∩ {X∗,ηs ∈ B(mJ3 , 3∆) ∀s ∈ [T ∗ 5, t]} ⊆ {X ∗,η t ∈ ⋃ k: mk∈Glarge B(mk, 3∆)}.
In summary, for event
A3 = ∆ A2 ∩ { N5 −N4 ≤ ∆t/16
H(1/η)
} ∩ { X∗,ηs ∈ B(mJ3 , 3∆) ∀s ∈ [T ∗ 5, t] } ,
we have A3 ⊆ {X∗,ηt ∈ ⋃ k: mk∈Glarge B(mk, 3∆)}. Besides, due to eq. (F.56)eq. (F.57)eq. (F.58), we get
lim sup η↓0 sup x∈[−L,L]
Px ( Ac3 ∩ {X∗,ηs ∈ ⋃ k: mk∈G Ωk ∀s ∈ [0, t]} )
≤2 1 + 1− exp ( − (1 + 1)q∗∆t ) +
∆t δα .
Remember that δ, 1, q∗ are fixed constants while ∆t can be made arbitrarily small, so by driving ∆t to 0 we can conclude the proof.
Recall the definition of jump processes in Definition F.1. Central to the proof of Lemma F.4, the next result provides a set of sufficient conditions for the convergence of a sequence of such jump processes in the sense of finite dimensional distributions. Lemma F.13. For a sequence of processes (Y n)n≥1 that, for each n ≥ 1, Y n is a(
(Unj )j≥0, (V n j )j≥0
) jump process, and a ( (Uj)j≥0, (Vj)j≥0 ) jump process Y , if
• U0 ≡ 0;
• (Un0 , V n 0 , U n 1 , V n 1 , U n 2 , V n 2 , · · · ) converges in distribution to (0, V0, U1, V1, U2, V2, · · · ) as
n→∞;
• For any x > 0 and any n ≥ 1, P(U1 + · · ·+ Un = x) = 0;
• For any x > 0,
lim n→∞
P(U1 + U2 + · · ·Un > x) = 1,
then the finite dimensional distribution of Y n converges to that of Y in the following sense: for any k ∈ N and any 0 < t1 < t2 < · · · < tk < ∞, the random element (Y nt1 , · · · , Y n tk
) converges in distribution to (Yt1 , · · · , Ytk) as n→∞.
Proof. Fix some k ∈ N and 0 < t1 < t2 < · · · < tk < ∞. For notational simplicity, let t = tk. Let (D,d) be the metric space where D = D[0,t], the space of all càdlàg functions in R on the time interval [0, t], and d is the Skorokhod metric defined as
d(ζ1, ζ2) =∆ inf λ∈Λ ‖ζ1 − ζ2 ◦ λ‖ ∨ ‖λ− I‖
where Λ is the set of all nondecreasing homeomorphism from [0, t] onto itself, and I(s) = s is the identity mapping. Also, we arbitrarily choose some ∈ (0, 1) and some open set A ⊆ Rk.
From the assumption, we can find integer J( ) such that P (∑J( ) j=1 Uj ≤ t ) < , as well as an integer
N( ) such that, for all n ≥ N( ), we have P (∑J( )
j=1 U n j ≤ t
) + P ( Un0 ≥ t1 ) < . We fix such
J( ), N( ) (we may abuse the notation slightly and simply write J,N when there is no ambiguity).
Using Skorokhod’s representation theorem, we can construct a probability space (Ω,F ,Q) that supports random variables (Ũn0 , Ṽ n 0 , · · · , ŨnJ , Ṽ nJ )n≥1 and (Ũ0, Ṽ0, · · · , ŨJ , ṼJ) and satisfies the following conditions:
• L(Un0 , V n0 , · · · , UnJ , V nJ ) = L(Ũn0 , Ṽ n0 , · · · , ŨnJ , Ṽ nJ ) for all n ≥ 1;
• L(U0, V0, · · · , UJ , VJ) = L(Ũ0, Ṽ0, · · · , ŨJ , ṼJ);
• Unj a.s.−−→ Uj and V nj a.s.−−→ Vj as n→∞ for all j ∈ [J ].
Therefore, on (Ω,F ,Q) we can define the following random elements (taking values in the space of càdlàg functions):
Y n,↓Js =
{ Ṽ n0 if s < Ũ
n 0∑J
j=0 Ṽ n j 1[Ũn0 +Ũ n 1 +···+Ũnj , Ũn0 +Ũn1 +···+Ũnj+1)
(s) otherwise ,
Y ↓Js = J∑ j=0 Ṽj1[Ũ1+···+Ũj , Ũ1+···+Ũj+1)(s) ∀s ≥ 0.
Note that (1) for the first jump time of Y ↓J we have Ũ0 ≡ 0, hence Y ↓J0 = Ṽ0; (2) when defining Y n,↓J we set its value on [0, Ũn0 ) to be Ṽ n 0 instead of 0.
Since Unj a.s.−−→ Uj and V nj a.s.−−→ Vj as n→∞ for all j ∈ [J ], we must have
lim n
d(Y n,↓Js , Y ↓J s ) = 0
almost surely, which further implies that Y n,↓Js ⇒ Y ↓Js as n → ∞ on (D,d). Now from our assumption that, for the jump times U1 + · · ·+Uj , we have P(U1 + · · ·+Uj = x) = 0 ∀x > 0, j ≥ 1, as well as (13.3) in Billingsley (2013), we then obtain
(Y n,↓Jt1 , · · · , Y n,↓J tk )⇒ (Y ↓Jt1 , · · · , Y ↓J tk ) (F.59)
as n→∞. Recall that A is the open set we arbitrarily chose at the beginning of the proof, and > 0 is also chosen arbitrarily. Now we observe the following facts.
• Using eq. (F.59), we can see that
lim inf n
Q ( (Y n,↓Jt1 , · · · , Y n,↓J tk ) ∈ A ) ≥ Q ( Y ↓Jt1 , · · · , Y ↓J tk ) ∈ A ) .
• The choice of N( ) and J( ) above implies that
∣∣Q((Y ↓J( )t1 , · · · , Y ↓J( )tk ) ∈ A)− P((Yt1 , · · · , Ytk) ∈ A)∣∣ ≤ P( J( )∑ j=1 Uj ≤ t ) < ,
∣∣Q((Y n,↓J( )t1 , · · · , Y n,↓J( )tk ) ∈ A)− P((Y nt1 , · · · , Y ntk) ∈ A)∣∣ ≤P ( J( )∑ j=1 Unj ≤ t ) + P ( Un0 ≥ t1 ) < ∀n ≥ N( ).
Collecting the two results above, we have established that
lim inf n
P ( (Y nt1 , · · · , Y n tk ) ∈ A ) ≥ P ( (Yt1 , · · · , Ytk) ∈ A ) − 2 .
From Portmanteau theorem, together with arbitrariness of > 0 and open setA, we can now conclude that (Y nt1 , · · · , Y n tk ) converges in distribution to (Yt1 , · · · , Ytk).
The following lemma concerns the scaled version of the marker process X̂∗,η,∆ defined in eq. (F.13)-eq. (F.14). Obviously, it is a jump process that complies with Definition F.1. When there is no ambiguity about the sequences (ηn)n≥1 and (∆n)n≥1, let X̂ (n) t = ∆ X̂∗,ηn,∆nt . From
eq. (F.6)-eq. (F.11) and eq. (F.15), we know that for any n ≥ 1, X̂(n) is a (( τ∗k (ηn,∆n) −
τ∗k−1(ηn,∆n) ) k≥0, ( mIk(ηn,∆n) ) k≥0 ) -jump process (with the convention that τ∗−1 = 0). Also, for clarity of the exposition, we let (for all n ≥ 1, k ≥ 0)
S̃ (n) k = σ ∗ k(ηn,∆n)− τ∗k−1(ηn,∆n),
S (n) k = τ ∗ k (ηn,∆n)− τ∗k−1(ηn,∆n),
W̃ (n) k = mĨGk (ηn,∆n) ,
W (n) k = mIGk (ηn,∆n).
Lastly, remember that Y is the continuous-time Markov chain defined in eq. (F.18)-eq. (F.21) and πG(·) is the random mapping defined in eq. (F.22) that is used to initialize Y . Besides, Y is a( (Sk)k≥0, (Wk)k≥0 ) jump process under Definition F.1, with S0 = 0 and W0 = πG(mi) (here x ∈ Ωi and Xη0 = x, so i is the index of the attraction field where the SGD iterate is initialized). The following result states that, given a sequence of learning rates (ηn)n≥1 that tend to 0, we are able to find a sequence of (∆n)n≥1 to parametrize X̂(n) = X̂∗,ηn,∆n , X(n) = X∗,ηn,∆n so that they have several useful properties, one of which is that the jump times and locations of X̂(n) converges in distribuiton to those of Y (πG(mi)). Lemma F.14. Assume the communication class G is absorbing. Given any mi ∈ G, x ∈ Ωi, finitely many real numbers (tl)k ′
l=1 such that 0 < t1 < t2 < · · · < tk′ , and a sequence of strictly positive real numbers (ηn)n≥1 with limn→0 ηn = 0, there exists a sequence of strictly positive real numbers (∆n)n≥1 with limn ∆n = 0 such that
• Under Px (so Xη0 = x), as n tends to∞,
(S (n) 0 ,W (n) 0 , S (n) 1 ,W (n) 1 , S (n) 2 ,W (n) 2 , · · · )⇒ (S0,W0, S1,W1, S2,W2, · · · ) (F.60)
• (Recall the definition of Tk, Ik in eq. (F.39)-eq. (F.42)) Given any > 0, the following claim holds for all n sufficiently large:
sup k≥0
Px ( ∃j ∈ [Tk(ηn,∆n), Tk(ηn,∆n)] s.t. Xηj /∈ ⋃ l: ml∈G Ωl |mIk(ηn,∆n) ∈ G ) < ;
(F.61)
• Given any > 0, the following claim holds for all n sufficiently large,
sup k≥0
Pk ( mIk(ηn,∆n)+v /∈ G large ∀v ∈ [unmin] ∣∣∣mIk(ηn,∆n) ∈ G)
≤P(Geom(p∗) ≥ u) + ∀u = 1, 2, · · · ; (F.62)
• For any l ∈ [k′],
lim n→∞
Px ( X∗,ηntl /∈ ⋃ j: mj∈Glarge B(mj ,∆n), X ∗,ηn s ∈ ⋃ j: mj∈G Ωj ∀s ∈ [0, tk′ ] ) = 0
(F.63)
where p∗ > 0 is a constant that does not vary with our choices of ηn or ∆n.
Proof. Let
νj = ∆ qj = µj(Ej)
νj,k = ∆ µj(Ej,k)
so from the definition of qj,k we have qj,k = 1{j 6= k}νj,k + ∑ l: ml∈Gsmall νj,lpl,k.
In order to specify our choice of (∆n)n≥1, we consider a construction of sequences (∆̄(j))j≥0, (η̄(j))j≥0 as follows. Fix some θ ∈ (0, α − 1)/2). Let ∆̄(0) = η̄(0) = 1. One can see the existence of some (∆̄(j))j≥1, (η̄(j))j≥1 such that
• ∆̄(j) ∈ ( 0, ∆̄(j − 1)/2 ] , η̄(j) ∈ ( 0, η̄(j − 1)/2 ] for all j ≥ 1;
• (Due to Lemma F.7) for any j ≥ 1, η ∈ (0, η̄(j)], (remember that x and i are the fixed constants prescribed in the description of the lemma)
Px ( σ∗0 ( η, ∆̄(j) ) < ηθ, ĨG0 ( η, ∆̄(j) ) = i ) > 1− 1
2j .
For definitions of σGk , τ G k , I G k , Ĩ G k , see eq. (F.6)-eq. (F.11).
• (Due to Lemma F.11) for any j ≥ 1, η ∈ (0, η̄(j)],∣∣∣∣Px(τ∗k (η, ∆̄(j))− σ∗k(η, ∆̄(j)) < ηθ, IGk (η, ∆̄(j)) = i2 ∣∣∣ ĨGk (η, ∆̄(j)) = i1)− pi1,i2 ∣∣∣∣ < 1/2j
uniformly for all k ≥ 0 and all mi1 ∈ Gsmall,mi2 ∈ Glarge. Also, by definition of σ∗ and τ∗, we must have
Px ( τ∗k ( η, ∆̄(j) ) − σ∗k ( η, ∆̄(j) ) = 0, IGk ( η, ∆̄(j) ) = i1 ∣∣∣ ĨGk (η, ∆̄(j)) = i1) = 1 for all k ≥ 0 and mi1 ∈ Glarge.
• (Due to Proposition E.25) for any j ≥ 1, η ∈ (0, η̄(j)],
− 1 2j
+ exp ( − (1 + 1
2j )qi1u )νi1,i2 − 12j qi1
≤Px ( σ∗k+1 ( η, ∆̄(j) ) − τ∗k ( η, ∆̄(j) ) > u, ĨGk+1 = i2 ∣∣∣ IGk (η, ∆̄(j)) = i1) ≤ 1
2j + exp
( − (1− 1
2j )qi1u )νi1,i2 + 12j qi1
uniformly for all k ≥ 1, all u > 1/2j , and all mi1 ∈ Glarge,mi2 ∈ G.
• (Due to G being absorbing and, again, Proposition E.25) for any j ≥ 1, for any j ≥ 1, η ∈ (0, η̄(j)], (Recall the definition of Tk, Ik in eq. (F.39)-eq. (F.42))
Px ( Ik+1 ( η, ∆̄(j) ) = i2 | Ik ( η, ∆̄(j) ) = i1 ) < 1
2j (F.64)
uniformly for all k ≥ 0, mi1 ∈ G,mi2 /∈ G.
• (Due to Lemma F.9) There exists some p∗ > 0 such that for any j ≥ 1, for any j ≥ 1, η ∈ (0, η̄(j)],
Px ( m v+Ik ( η,∆̄(j) ) /∈ Glarge ∀v ∈ [unmin] ∣∣∣ Ik(η, ∆̄(j)) = i1) ≤P(Geom(p∗) ≥ u) + 1/2j (F.65)
uniformly for all k ≥ 0, u ≥ 1 and mi1 ∈ G.
• (Due to Lemma F.12) for any j ≥ 1, for any j ≥ 1, η ∈ (0, η̄(j)], Px ( X∗,ηntk /∈ ⋃ j: mj∈Glarge B(mj , ∆̄(j)), X ∗,ηn s ∈ ⋃ j: mj∈G Ωj ∀s ∈ [0, tk′ ] ) < 1/2j
(F.66)
uniformly for all k ∈ [k′].
Fix such (∆̄(j))j≥0, (η̄(j))j≥0. Define a function J(·) : N 7→ N as J(n) = 0 ∨max{j ≥ 0 : η̄(j) ≥ ηn}
with the convention that max ∅ = −∞. Lastly, let ∆n = ∆̄(J(n)) ∀n ≥ 1.
Note that, due to limn ηn = 0, we have limn J(n) = ∞, hence limn ∆n = 0. Besides, the definition of J(·) tells us that in case that J(n) ≥ 1 (which will hold for all n sufficiently large), the claims above holds with η = ηn and j = J(n). In particular, by combining limn J(n) = ∞ with eq. (F.64)eq. (F.65)eq. (F.66) respectively, we have eq. (F.61)eq. (F.62)eq. (F.63).
Now it remains to prove eq. (F.60). To this end, it suffices to show that, for any positive integer K, we have (S(n)0 ,W (n) 0 , · · · , S (n) K ,W (n) K ) converges in distribution (S0,W0, · · · , SK ,WK) as n tends to infinity. In particular, note that S0 = 0,W0 = πG(mi), so W0 = mj with probability pi,j if mi ∈ Gsmall, and W0 ≡ mi if mi ∈ Glarge. For clarity of the exposition, we restate some important claims above under the new notational system with S̃(n)k , W̃ (n) k , S (n) k ,W (n) k we introduced right above this lemma. Given any > 0, the following claims hold for all n sufficiently large:
• First of all,
Px ( S̃ (n) 0 < η θ n, W̃ (n) 0 = mi ) > 1− . (F.67)
• For all k ≥ 0 and all mi1 ∈ Gsmall,mi2 ∈ Glarge,∣∣∣∣Px(S(n)k − S̃(n)k < ηθn, W (n)k = mi2 ∣∣∣ W̃ (n)k = mi1)− pi1,i2∣∣∣∣ < . (F.68) • For all k ≥ 0 and all mi1 ∈ Glarge,
Px ( S (n) k − S̃ (n) k = 0, W (n) k = mi1 ∣∣∣ W̃ (n)k = mi1) = 1. (F.69) • For all k ≥ 0, all mi1 ∈ Glarge,mi2 ∈ G and all u > ,
− + exp ( − (1 + )qi1u )νi1,i2 − qi1
≤Px ( S̃ (n) k+1 − S (n) k > u, W̃ (n) k+1 = mi2 ∣∣∣W (n)k = mi1) ≤Px ( S̃ (n) k+1 − S (n) k > u− η θ n, W̃ (n) k+1 = mi2
∣∣∣W (n)k = mi1) ≤ + exp ( − (1− )qi1u )νi1,i2 + qi1
(F.70)
• Here is one implication of eq. (F.68). Since |G| ≤ nmin, we have Px ( S (n) k − S̃ (n) k ≥ η θ n ∣∣∣ W̃ (n)k = mi1) < nmin · (F.71) for all k ≥ 0 and mi1 ∈ Gsmall.
• Note that for any mi1 ,mi2 ∈ Glarge and any k ≥ 0 Px ( S (n) k+1 − S (n) k > u, W (n) k+1 = mi2 ∣∣∣W (n)k = mi1) =1{i2 6= i1}Px ( S̃ (n) k+1 − S (n) k > u, W̃ (n) k+1 = mi2
∣∣∣W (n)k = mi1) +
∑ i3: mi3∈Gsmall ∫ s>0 Px ( S (n) k+1 − S̃ (n) k+1 ≥ (u− s) ∨ 0, W̃ (n) k+1 = mi2 ∣∣∣ W̃ (n)k = mi3) · Px ( S̃ (n) k+1 − S (n) k = ds, W̃ (n) k+1 = mi3 ∣∣∣W (n)k = mi1).
Fix some i3 with mi3 ∈ Gsmall. On the one hand, due to eq. (F.71),∫ s∈(0,u−ηθn] Px ( S (n) k+1 − S̃ (n) k+1 ≥ u− s, W̃ (n) k+1 = mi2 ∣∣∣ W̃ (n)k = mi3) · Px ( S̃ (n) k+1 − S (n) k = ds, W̃ (n) k+1 = mi3
∣∣∣W (n)k = mi1) ≤nmin .
On the other hand, by considering the integral on (u− ηθn,∞), we get∫ s∈(u−ηθn,∞) Px ( S (n) k+1 − S̃ (n) k+1 ≥ (u− s) ∨ 0, W̃ (n) k+1 = mi2 ∣∣∣ W̃ (n)k = mi3) · Px ( S̃ (n) k+1 − S (n) k = ds, W̃ (n) k+1 = mi3
∣∣∣W (n)k = mi1) ≥ ∫ s∈(u,∞) Px ( W̃ (n) k+1 = mi2
∣∣∣ W̃ (n)k = mi3) · Px ( S̃ (n) k+1 − S (n) k = ds, W̃ (n) k+1 = mi3
∣∣∣W (n)k = mi1) ≥(pi3,i2 − ) ( − + exp ( − (1 + )qi1u )νi1,i3 − qi1
) due to eq. (F.68) and eq. (F.70). Meanwhile,∫
s∈(u−ηθn,∞) Px ( S (n) k+1 − S̃ (n) k+1 ≥ (u− s) ∨ 0, W̃ (n) k+1 = mi2 ∣∣∣ W̃ (n)k = mi3) · Px ( S̃ (n) k+1 − S (n) k = ds, W̃ (n) k+1 = mi3
∣∣∣W (n)k = mi1) ≤(nmin + pi3,i2 + ) ( + exp ( − (1− )qi1u )νi1,i3 + qi1
) due to eq. (F.68), eq. (F.70) and eq. (F.71).
• Therefore, for any mi1 ,mi2 ∈ Glarge and any k ≥ 0, Px ( S (n) k+1 − S (n) k > u, W (n) k+1 = mi2 ∣∣∣W (n)k = mi1) ≤g( ) + exp ( − (1− )qi1u )1{i2 6= i1}νi1,i2 +∑i3: mi3∈Gsmall νi1,i3pi3,i2 qi1
≤g( ) + exp ( − (1− )qi1u )qi1,i2 qi1
(F.72)
and Px ( S (n) k+1 − S (n) k > u, W (n) k+1 = mi2 ∣∣∣W (n)k = mi1) ≥− g( ) + exp ( − (1 + )qi1u )1{i2 6= i1}νi1,i2 +∑i3: mi3∈Gsmall νi1,i3pi3,i2 qi1
≥− g( ) + exp ( − (1 + )qi1u )qi1,i2 qi1
(F.73)
where q∗ = maxi qi and
g( ) =∆ +
q∗ + nmin(1 + ) +
1 +
q∗ nmin
+ nmin( + q∗ ) + nmin(nmin + + 1)(1 +
1
q∗ ) .
Note that lim ↓0 g( ) = 0.
Now we apply the bounds in eq. (F.67)eq. (F.72)eq. (F.73) to establish the weak convergence claim regarding (S(n)0 ,W (n) 0 , · · · , S (n) K ,W (n) K ). Fix some positive integer K, some strictly positive real numbers (sk)Kk=0, a sequence (wk) K k=0 ∈ ( Glarge )K+1 with wk = mik for each k, and some > 0 such that < mink=0,1,··· ,K{sk}. On the one hand, the definition of the CTMC Y implies that
P ( S0 < t0,W0 = w0; Sk > sk and Wk = wk ∀k ∈ [K] ) =P(πG(mi) = w0)
K∏ k=1 ( Sk > sk and Wk = wk ∣∣∣Wk−1 = wk−1) = ( 1{mi ∈ Glarge, i0 = i}+ 1{mi ∈ Gsmall}pi0,i1
) · K∏ k=1 exp(−qik−1sk) qik−1,ik qik−1 .
On the other hand, using eq. (F.67)eq. (F.72)eq. (F.73), we know that for all n sufficiently large, Px ( S (n) 0 < s0,W (n) 0 = w0; S (n) k > sk and W (n) k = wk ∀k ∈ [K] ) ≥(1− ) ( 1{mi ∈ Glarge, i0 = i}+ 1{mi ∈ Gsmall}(pi0,i1 − )
) · K∏ k=1 ( − g( ) + exp(−(1 + )qik−1sk) qik−1,ik qik−1
) and
Px ( S (n) 0 < s0,W (n) 0 = w0; S (n) k > sk and W (n) k = wk ∀k ∈ [K] ) ≤ ( 1{mi ∈ Glarge, i0 = i}+ 1{mi ∈ Gsmall}(pi0,i1 + )
) · K∏ k=1 ( g( ) + exp(−(1− )qik−1sk) qik−1,ik qik−1 ) .
Since > 0 can be arbitrarily small, we now obtain
lim n→∞
Px ( S (n) 0 < t0,W (n) 0 = w0; S (n) k > sk and W (n) k = mk ∀k ∈ [K] ) =P ( S0 < s0,W0 = w0; Sk > sk and Wk = wk ∀k ∈ [K] ) ,
and the arbitrariness of the integer K, the strictly positive real numbers (sk)Kk=0, and the sequence (wk) K k=0 ∈ ( Glarge )K+1 allows us to conclude the proof.
To extend the result above to the case where the communication class G is transient, we revisit the definition of the Y † in eq. (F.26). Let Ḡ = Glarge ∪ {†} and let m0 = † (remember that all the local minimizers of f on [−L,L] are m1, · · · ,mmin). Meanwhile, using qi and qi,j in eq. (F.20)eq. (F.21), we can define
q†i,j =  qi,j if i ≥ 1, j ≥ 1 1{j = 0} if i = 0∑ j∈[nmin],mj /∈G qi,j if i ≥ 1, j = 0.
and q†0 = 1, q † i = qi ∀i ≥ 1. Next, fix some i with mi ∈ G and x ∈ Ωi. Define a sequence of random variables (S†k)k≥0, (W † k )k≥0 such that S † k = 0 and W0 = πG(mi),W † 0 = †1{W0 /∈ Glarge} + W01{W0 ∈ Glarge} (see the definition of random mapping πG in eq. (F.22)) and (for all k ≥ 0 and i, j with mj ,ml ∈ Ḡ)
P ( W †k+1 = ml, S † k+1 > t ∣∣∣W †k = mj , (W †l )k−1l=0 , (S†l )kl=0) (F.74) =P ( W †k+1 = ml, S † k+1 > t
∣∣∣W †k = mj) = exp(−q†j t)q†j,l q†j ∀t > 0 (F.75)
Then it is easy to see that Y †(πG(mi)) defined in eq. (F.26) is a ( (S†k)k≥0, (W † k )k≥0 ) jump process. In particular, from at any state that is not † (namely, any mj with mj ∈ Glarge), the probability that Y † moves to † in the next transition is equal to the chance that, starting from the same state, Y moves to a state that is not in G. Once entering m0 = †, the process Y † will only make dummy jumps (with interarrival times being iid Exp(1)): indeed, we have q†0 = q † 0,0 = 1 and q † 0,j = 0 for any j ≥ 1, implying that, given W †k = m0 = †, we must have W † k+1 = m0 = †. These dummy jumps ensure that Y † is stuck at the cemetery state † after visiting it.
Similarly, we can characterize the jump times and locations of the jump process X̂†,∗,η,∆ (for the definition, see eq. (F.25)). When there is no ambiguity about the sequences (ηn)n≥1, (∆n)n≥1, let X̂†,(n) = X̂†,∗,ηn,∆n and X†,(n) = X∗,ηn,∆n . Also, recall that τG defined in eq. (F.4) is the step n when Xηn exits the communication class G. Now let (Ek)k≥0 be a sequence of iid Exp(1) random variables that is also independent of the noises (Zk)k≥1 (so they are independent from the SGD iterates Xηn). For all n ≥ 1, k ≥ 0, define (see eq. (F.6)-eq. (F.11) and eq. (F.15) for definitions of the quantities involved)
S̃ †,(n) k =
{ σ∗k(ηn,∆n) ∧ T ∗(τG(ηn), ηn)− τ∗k−1(ηn,∆n) if τ∗k−1(ηn,∆n) < T ∗(τG(ηn), ηn)
0 otherwise
S †,(n) k =
{ τ∗k (ηn,∆n) ∧ T ∗(τG(ηn), ηn)− τ∗k−1(ηn,∆n) if τ∗k−1(ηn,∆n) < T ∗(τG(ηn), ηn)
Ek otherwise
W̃ †,(n) k =
{ mĨGk (ηn,∆n) if σ∗k(ηn,∆n) < T ∗(τG(ηn), ηn)
† otherwise
W †,(n) k =
{ mIGk (ηn,∆n) if τ ∗ k (ηn,∆n) < T ∗(τG(ηn), ηn)
† otherwise
with the convention that τ∗−1 = 0. Note that T ∗(τG(ηn), ηn) is the scaled timestamp forX(n) = X∗,η corresponding to τG(ηn), hence T∗(τG(ηn), ηn) = min{t ≥ 0 : X(n) /∈ ⋃ j: mj∈G Ωj}. One can
see that X̂†,(n) is a ( (S †,(n) k )k≥0, (W †,(n) k )k≥0 ) jump process. The next lemma is similar to Lemma F.14 and discusses the convergence of the jump times and locations of X̂†,(n) on a communication class G in the transient case. Lemma F.15. Assume that the communication class G is transient. Given any mi ∈ G, x ∈ Ωi, finitely many real numbers (tl)k ′
l=1 such that 0 < t1 < t2 < · · · < tk′ , and a sequence of strictly positive real numbers (ηn)n≥1 with limn→0 ηn = 0, there exists a sequence of strictly positive real numbers (∆n)n≥1 with limn ∆n = 0 such that
• Under Px (so Xη0 = x), as n tends to∞,
(S †,(n) 0 ,W †,(n) 0 , S †,(n) 1 ,W †,(n) 1 , S †,(n) 2 ,W †,(n) 2 , · · · )⇒ (S † 0,W † 0 , S † 1,W † 1 , S † 2,W † 2 , · · · )
(F.76)
• For any l ∈ [k′],
lim n→∞
Px ( X †,(n) tl /∈ ⋃
j: mj∈Glarge B(mj ,∆n), X
†,(n) s ∈ ⋃ j: mj∈G Ωj ∀s ∈ [0, tl] ) = 0
(F.77)
Proof. Let
νj,k = ∆ µj(Ej,k) ∀j, k ≥ 1, j 6= k
pj,† = ∆ ∑ j̃: mj̃ /∈G pj,̃j ∀mj ∈ G small
qj,† = ∆ ∑ k:mk /∈G νj,k + ∑ k: mk∈Gsmall νj,kpk,† ∀mj ∈ Glarge.
In order to specify our choice of (∆n)n≥1, we consider a construction of sequences (∆̄(j))j≥0, (η̄(j))j≥0 as follows. Fix some θ ∈ (0, α − 1)/2). Let ∆̄(0) = η̄(0) = 1. One can see the existence of some (∆̄(j))j≥1, (η̄(j))j≥1 such that
• ∆̄(j) ∈ ( 0, ∆̄(j − 1)/2 ] , η̄(j) ∈ ( 0, η̄(j − 1)/2 ] for all j ≥ 1;
• (Due to Lemma F.7) for any j ≥ 1, η ∈ (0, η̄(j)], (remember that x and i are the fixed constants prescribed in the description of the lemma)
Px ( σ∗0 ( η, ∆̄(j) ) < ηθ, ĨG0 ( η, ∆̄(j) ) = i ) > 1− 1
2j .
For definitions of σGk , τ G k , I G k , Ĩ G k , see eq. (F.6)-eq. (F.11).
• (Due to Lemma F.11) for any j ≥ 1, η ∈ (0, η̄(j)],∣∣∣∣Px(τ∗k (η, ∆̄(j))− σ∗k(η, ∆̄(j)) < ηθ, IGk (η, ∆̄(j)) = i2 ∣∣∣ ĨGk (η, ∆̄(j)) = i1)− pi1,i2 ∣∣∣∣ < 1/2j
uniformly for all k ≥ 0 and all mi1 ∈ Gsmall,mi2 /∈ Gsmall. Also, by definition of σ∗ and τ∗, we must have
Px ( τ∗k ( η, ∆̄(j) ) − σ∗k ( η, ∆̄(j) ) = 0, IGk ( η, ∆̄(j) ) = i1 ∣∣∣ ĨGk (η, ∆̄(j)) = i1) = 1 for all k ≥ 0 and mi1 ∈ Glarge.
• (Due to Proposition E.25) for any j ≥ 1, η ∈ (0, η̄(j)],
− 1 2j
+ exp ( − (1 + 1
2j )qi1u )νi1,i2 − 12j qi1
≤Px ( σ∗k+1 ( η, ∆̄(j) ) − τ∗k ( η, ∆̄(j) ) > u, ĨGk+1 = i2 ∣∣∣ IGk (η, ∆̄(j)) = i1) ≤ 1
2j + exp
( − (1− 1
2j )qi1u )νi1,i2 + 12j qi1
uniformly for all k ≥ 1, all u > 1/2j , and all mi1 ∈ Glarge,mi2 ∈ G.
• (Due to Lemma F.12) for any j ≥ 1, for any j ≥ 1, η ∈ (0, η̄(j)], Px ( X (n) tk /∈ ⋃
j: mj∈Glarge B(mj , ∆̄(j)), X
(n) s ∈ ⋃ j: mj∈G Ωj ∀s ∈ [0, tk] ) < 1/2j (F.78)
uniformly for all k ∈ [k′].
Fix such (∆̄(j))j≥0, (η̄(j))j≥0. Define a function J(·) : N 7→ N as
J(n) = 0 ∨max{j ≥ 0 : η̄(j) ≥ ηn}
with the convention that max ∅ = −∞. Lastly, let
∆n = ∆̄(J(n)) ∀n ≥ 1.
Note that, due to limn ηn = 0, we have limn J(n) = ∞, hence limn ∆n = 0. Besides, since X †,(n) t = X (n) t given X (n) s ∈ ⋃ j: mj∈G Ωj for all s ∈ [0, t], by combining limn J(n) = ∞ with eq. (F.78) we obtain eq. (F.77).
Now it remains to prove eq. (F.76). To this end, it suffices to show that, for any positive integer K, we have (S†,(n)0 ,W †,(n) 0 , · · · , S †,(n) K ,W †,(n) K ) converges in distribution (S † 0,W † 0 , · · · , S † K ,W † K) as n tends to infinity. In particular, due to introduction of the dummy jumps, we know that for any k with τ∗k (ηn,∆n) ≥ τG(ηn) (in other words, X̂†,(n) has reached state † within the first k jumps) we have S†,(n)k+1 ∼ Exp(1) and W †,(n) k+1 ≡ †. Similarly, for any k with S † 0 + · · · + S † k ≤ τYG , we have
S†k+1 ∼ Exp(1) and W † k+1 ≡ †. Therefore, it suffices to show that, for any positive integer K, any
series of strictly positive real numbers (sk)Kk=0, any sequence (wk) K k=0 ∈
( Ḡ )K+1
such that wj 6= † for any j < K, indices ik such that wk = mik for each k, we have
lim n→∞
Px ( S †,(n) 0 < t0,W †,(n) 0 = w0; S †,(n) k > sk and W †,(n) k = wk ∀k ∈ [K] ) =P ( S†0 < s0,W † 0 = w0; S † k > sk and W † k = wk ∀k ∈ [K] ) (F.79)
Fix some (sk)Kk=0, (wk) K k=0 ∈
( Ḡ )K+1
, and indices (ik)Kk=1 satisfying the conditions above. Besides, arbitrarily choose some > 0 so that < mink=0,··· ,K sk. To proceed, we start by translating the inequalities established above under the new system of notations.
• First of all, for all n sufficiently large, (remember that x and i are prescribed constants in the description of this lemma)
Px ( S̃ †,(n) 0 < η θ n, W̃ †,(n) 0 = mi ) > 1− . (F.80)
• For all k ≥ 0 and all mi1 ∈ Gsmall,mi2 ∈ Glarge, it holds for all n sufficiently large that∣∣∣∣Px(S†,(n)k − S̃†,(n)k < ηθn, W †,(n)k = mi2 ∣∣∣ W̃ †,(n)k = mi1)− pi1,i2∣∣∣∣ < . (F.81) • On the other hand, for all k ≥ 0 and all mi1 ∈ Gsmall, it holds for all n sufficiently large that∣∣∣∣Px(S†,(n)k − S̃†,(n)k < ηθn, W †,(n)k = † ∣∣∣ W̃ †,(n)k = mi1)− ∑
i2: mi2 /∈G
pi1,i2 ∣∣∣∣ =
∣∣∣∣Px(S†,(n)k − S̃†,(n)k < ηθn, W †,(n)k = † ∣∣∣ W̃ †,(n)k = mi1)− pi1,†∣∣∣∣ < . (F.82) • For all k ≥ 0 and all mi1 ∈ Glarge, it holds for all n that
Px ( S †,(n) k − S̃ †,(n) k = 0, W †,(n) k = mi1 ∣∣∣ W̃ †,(n)k = mi1) = 1. (F.83) • For all k ≥ 0, all mi1 ∈ Glarge,mi2 ∈ G and all u > , the following claim holds for all n
sufficiently large: − + exp ( − (1 + )qi1u )νi1,i2 − qi1
≤Px ( S̃ †,(n) k+1 − S †,(n) k > u, W̃ †,(n) k+1 = mi2 ∣∣∣W †,(n)k = mi1) ≤Px ( S̃ †,(n) k+1 − S †,(n) k > u− η θ n, W̃ †,(n) k+1 = mi2
∣∣∣W †,(n)k = mi1) ≤ + exp ( − (1− )qi1u )νi1,i2 + qi1
(F.84)
• On the other hand, for all k ≥ 0, all mi1 ∈ Glarge, the following claim holds for all n sufficiently large:
− + exp ( − (1 + )qi1u )− +∑i2: mi2 /∈G νi1,i2 qi1
≤Px ( S̃ †,(n) k+1 − S †,(n) k > u, W̃ †,(n) k+1 = † ∣∣∣W †,(n)k = mi1) ≤Px ( S̃ †,(n) k+1 − S †,(n) k > u− η θ n, W̃ †,(n) k+1 = †
∣∣∣W †,(n)k = mi1) ≤ + exp ( − (1− )qi1u ) +∑i2: mi2 /∈G νi1,i2 qi1
(F.85)
• Here is one implication of eq. (F.81)eq. (F.82). Since |G| ≤ nmin, we have (when n is sufficiently large)
Px ( S (n) k − S̃ (n) k ≥ η θ n ∣∣∣ W̃ (n)k = mi1) < nmin · (F.86) for all k ≥ 0 and mi1 ∈ Gsmall.
• Note that for any mi1 ,mi2 ∈ Glarge and any k ≥ 0 Px ( S †,(n) k+1 − S †,(n) k > u, W †,(n) k+1 = mi2 ∣∣∣W †,(n)k = mi1) =1{i2 6= i1}Px ( S̃ †,(n) k+1 − S †,(n) k > u, W̃ †,(n) k+1 = mi2
∣∣∣W †,(n)k = mi1) +
∑ i3: mi3∈Gsmall ∫ s>0 Px ( S †,(n) k+1 − S̃ †,(n) k+1 ≥ (u− s) ∨ 0, W̃ †,(n) k+1 = mi2 ∣∣∣ W̃ †,(n)k = mi3) · Px ( S̃ †,(n) k+1 − S †,(n) k = ds, W̃ †,(n) k+1 = mi3
∣∣∣W †,(n)k = mi1). Fix some i3 with mi3 ∈ Gsmall. Due to eq. (F.86)∫
s∈(0,u−ηθn] Px ( S †,(n) k+1 − S̃ †,(n) k+1 ≥ u− s, W̃ †,(n) k+1 = mi2 ∣∣∣ W̃ †,(n)k = mi3) · Px ( S̃ †,(n) k+1 − S †,(n) k = ds, W̃ †,(n) k+1 = mi3
∣∣∣W †,(n)k = mi1) ≤nmin .
Meanwhile, by considering the integral on (u− ηθn,∞), we get∫ s∈(u−ηθn,∞) Px ( S †,(n) k+1 − S̃ †,(n) k+1 ≥ (u− s) ∨ 0, W̃ †,(n) k+1 = mi2 ∣∣∣ W̃ †,(n)k = mi3) · Px ( S̃ †,(n) k+1 − S †,(n) k = ds, W̃ †,(n) k+1 = mi3
∣∣∣W †,(n)k = mi1) ≥ ∫ s∈(u,∞) Px ( W̃ †,(n) k+1 = mi2
∣∣∣ W̃ †,(n)k = mi3) · Px ( S̃ †,(n) k+1 − S †,(n) k = ds, W̃ †,(n) k+1 = mi3
∣∣∣W †,(n)k = mi1) ≥(pi3,i2 − ) ( − + exp ( − (1 + )qi1u )νi1,i3 − qi1
) due to eq. (F.81) and eq. (F.84). As for the upper bound,∫
s∈(u−ηθn,∞) Px ( S †,(n) k+1 − S̃ †,(n) k+1 ≥ (u− s) ∨ 0, W̃ †,(n) k+1 = mi2 ∣∣∣ W̃ †,(n)k = mi3) · Px ( S̃ †,(n) k+1 − S †,(n) k = ds, W̃ †,(n) k+1 = mi3
∣∣∣W †,(n)k = mi1) ≤(nmin + pi3,i2 + ) ( + exp ( − (1− )qi1u )νi1,i3 + qi1
) due to eq. (F.81), eq. (F.84) and eq. (F.86).
• Therefore, for any mi1 ,mi2 ∈ Glarge and any k ≥ 0, Px ( S †,(n) k+1 − S †,(n) k > u, W †,(n) k+1 = mi2 ∣∣∣W †,(n)k = mi1) ≤g( ) + exp ( − (1− )qi1u )1{i2 6= i1}νi1,i2 +∑i3: mi3∈Gsmall νi1,i3pi3,i2 qi1
≤g( ) + exp ( − (1− )qi1u )qi1,i2 qi1
(F.87)
and Px ( S †,(n) k+1 − S †,(n) k > u, W †,(n) k+1 = mi2 ∣∣∣W †,(n)k = mi1) ≥− g( ) + exp ( − (1 + )qi1u )1{i2 6= i1}νi1,i2 +∑i3: mi3∈Gsmall νi1,i3pi3,i2 qi1
≥− g( ) + exp ( − (1 + )qi1u )qi1,i2 qi1
(F.88)
where q∗ = maxi qi and
g( ) =∆ 2 +
q∗ + nmin(1 + ) +
1 +
q∗ nmin
+ nmin( + q∗ ) + nmin(nmin + + 1)(1 +
1
q∗ ) .
Note that lim ↓0 g( ) = 0.
• On the other hand, for the case where the marker process X̂†,(n) jumps to the cemetery state † from some mi1 ∈ Glarge, note that
Px ( S †,(n) k+1 − S †,(n) k > u, W †,(n) k+1 = † ∣∣∣W †,(n)k = mi1) =Px ( S̃ †,(n) k+1 − S †,(n) k > u, W̃ †,(n) k+1 = †
∣∣∣W †,(n)k = mi1) +
∑ i3: mi3∈Gsmall ∫ s>0 Px ( S †,(n) k+1 − S̃ †,(n) k+1 ≥ (u− s) ∨ 0, W̃ †,(n) k+1 = † ∣∣∣ W̃ †,(n)k = mi3) · Px ( S̃ †,(n) k+1 − S †,(n) k = ds, W̃ †,(n) k+1 = mi3
∣∣∣W †,(n)k = mi1). Arguing similarly as we did above by considering the integral on [0, u−ηθn] and (u−ηθn,∞) separately, and using eq. (F.82) and eq. (F.85), we then get (for all n sufficiently large)
Px ( S †,(n) k+1 − S †,(n) k > u, W †,(n) k+1 = † ∣∣∣W †,(n)k = mi1) ≤g( ) + exp ( − (1− )qi1u )∑i2:mi2 /∈G νi1,i2 +∑i2:mi2∈Gsmall νi1,i2pi2,† qi1
=g( ) + exp ( − (1− )qi1u )qi1,† qi1
(F.89)
and
Px ( S †,(n) k+1 − S †,(n) k > u, W †,(n) k+1 = † ∣∣∣W †,(n)k = mi1) ≥− g( ) + exp ( − (1 + )qi1u )∑i2:mi2 /∈G νi1,i2 +∑i2:mi2∈Gsmall νi1,i2pi2,† qi1
=− g( ) + exp ( − (1 + )qi1u )qi1,† qi1
(F.90)
For simplicity of presentation, we also let qj,0 = qj,† and pj,0 = pj,†. First of all, remember that we have fixed some series of strictly positive real numbers (sk)Kk=0, some sequence (wk) K k=0 ∈ ( Ḡ )K+1 such that wj 6= † for any j < K, and indices ik such that wk = mik for each k. The definition of the continuous-time Markov chain Y † implies that
P ( S0 < t0,W0 = w0; Sk > sk and Wk = wk ∀k ∈ [K] ) =P(πG(mi) = w0)
K∏ k=1 ( Sk > sk and Wk = mk ∣∣∣Wk−1 = wk−1) = ( 1{mi ∈ Glarge, i0 = i}+ 1{mi ∈ Gsmall}pi0,i1
) · K∏ k=1 exp(−qik−1sk) qik−1,ik qik−1 .
On the other hand, using eq. (F.87)-eq. (F.90), we know that for all n sufficiently large, Px ( S (n) 0 < s0,W (n) 0 = w0; S (n) k > sk and W (n) k = wk ∀k ∈ [K] ) ≥(1− ) ( 1{mi ∈ Glarge, i0 = i}+ 1{mi ∈ Gsmall}(pi0,i1 − )
) · K∏ k=1 ( − g( ) + exp(−(1 + )qik−1sk) qik−1,ik qik−1
) and
Px ( S (n) 0 < s0,W (n) 0 = w0; S (n) k > sk and W (n) k = mk ∀k ∈ [K] ) ≤ ( 1{mi ∈ Glarge, i0 = i}+ 1{mi ∈ Gsmall}(pi0,i1 + )
) · K∏ k=1 ( g( ) + exp(−(1− )qik−1sk) qik−1,ik qik−1 ) .
The arbitrariness of > 0 then allows us to establish eq. (F.79) and conclude the proof.
Now we are ready to prove Lemma F.4 and Lemma F.5.
Proof of Lemma F.4. From Lemma F.14, one can see the existence of some (∆n)n≥1 with limn ∆n = 0 such that eq. (F.60)-eq. (F.63) hold. For simplicity of notations, we let X̂(n) = X̂∗,ηn,∆n , X(n) = X∗,ηn , and let t̄ = tk′
Combine eq. (F.60) with Lemma F.13, and we immediately get eq. (F.23). In order to prove eq. (F.24), it suffices to show that for any > 0,
lim sup n
Px ( X (n) tk /∈ ⋃
j: mj∈Glarge B(mj ,∆n)
) ≤ 4 ∀k ∈ [k′].
Fix > 0, and observe following bound by decomposing the events Px ( X (n) tk /∈ ⋃
j: mj∈Glarge B(mj ,∆n) ) ≤Px ( X (n) tk /∈ ⋃
j: mj∈Glarge B(mj ,∆n), X
(n) t ∈ ⋃ j: mj∈G Ωj ∀t ∈ [0, t̄] )
+Px ( ∃t ∈ [0, t̄] such that X(n)t /∈ ⋃ j: mj∈G Ωj ) Therefore, given eq. (F.63), it suffices to prove
lim sup n
Px ( ∃t ∈ [0, t̄] such that X(n)t /∈ ⋃ j: mj∈G Ωj ) ≤ 3 . (F.91)
Let
T (n) 0 = ∆ min{t ≥ 0 : X(n)t ∈ ⋃
j:mj∈G B(mj , 2∆n)}
I (n) 0 = j ⇐⇒ X (n)
T (n) 0
∈ B(mj , 2∆n)
T (n) k = ∆ min{t > T (n)k−1 : X (n) t ∈ ⋃ j:mj∈G, j 6=I(n)k−1 B(mj , 2∆n)}
I (n) k = j ⇐⇒ X (n)
T (n) k
∈ B(mj , 2∆n).
Building upon this definition, we define the following stopping times and marks that only records the hitting time to minimizer in large attraction fields in G (with convention k(n),large(−1) = −1, T (n),large−1 = 0, T (n) −1 = 0)
k(n),large(k) =∆ min{l > k(n),large(k − 1) : m I
(n) l
∈ Glarge}
T (n),large k = ∆ T (n) k(n),large(k), I (n),large k = ∆ I (n) k(n),large(k).
Now by defining
J (n)(t) =∆ #{k ≥ 0 : T (n)k ≤ t},
J (n) large(t) = ∆ #{k ≥ 0 : T (n),largek ≤ t},
J (n)(s, t) =∆ #{k ≥ 0 : T (n)k ∈ [s, t]},
we use J (n)(t) to count the numbers of visits to local minima on G, and J (n)large(t) for the number of visits to minimizers in the large attraction fields on G. J (n)(s, t) counts the indices k such that at T
(n) k a minimizer on G is visited and regarding the hitting time we have T (n) k ∈ [s, t].
First of all, the weak convergence result in eq. (F.60) implies the existence of some positive integer N( ) such that
lim sup n
Px(J (n)large(t̄) > N( )) < .
Fix such N( ). Next, from eq. (F.62), we know the existence of some integer K( ) such that
lim sup n sup k≥0
Px ( J (n) ( T (n),large k−1 , T (n),large k ) > K( ) ) ≤ /N( ).
Fix such K( ) as well. From the results above, we know that for event
A1(n) = ∆ {J (n)large(t̄) ≤ N( )} ∩
{ J (n) ( T
(n),large k−1 , T (n),large k
) ≤ K( ) ∀k ∈ [N( )] } ,
we have lim supn Px (( A1(n) )c) ≤ 2 . On the other hand, on event A1(n), we must have J (n)(t̄) ≤ N( )K( ).
Meanwhile, it follows immediately from eq. (F.61) that
lim sup n sup k≥0
P ( ∃t ∈ [T (n)k−1, T (n) k ] such that X (n) t /∈ ⋃ j: mj∈G Ωj ) < N( )K( ) ,
hence for event
A2(n) = ∆ { X (n) t ∈ ⋃ j: mj∈G Ωj ∀t ∈ [0, T (n)N( )K( )] } ,
we must have lim supn Px (( A2(n) )c) ≤ . To conclude the proof, note that A1(n) ∩A2(n) ⊆ {J (n)(t̄) ≤ N( )K( )} ∩ { X (n) t ∈
⋃ j: mj∈G Ωj ∀t ∈ [0, T (n)N( )K( )] }
= {T (n)N( )K( ) ≥ t̄} ∩ { X (n) t ∈ ⋃ j: mj∈G Ωj ∀t ∈ [0, T (n)N( )K( )] }
⊆ { X
(n) t ∈ ⋃ j: mj∈G Ωj ∀t ∈ [0, t̄] }
so we have established eq. (F.91).
Proof of Lemma F.5. From Lemma F.15, one can see the existence of some (∆n)n≥1 with limn ∆n = 0 such that eq. (F.76) and eq. (F.77) hold. For simplicity of notations, we let X̂†,(n) = X̂†,∗,ηn,∆n , X†,(n) = X†,∗,ηn , and let t̄ = tk′
Combine eq. (F.76) with Lemma F.13, and we immediately get eq. (F.27). In order to prove eq. (F.28), note that {
X †,(n) tk
/∈ ⋃
j: mj∈Glarge B(mj ,∆n) and X
†,(n) tk
6= † }
= { X †,(n) tk /∈ ⋃
j: mj∈Glarge B(mj ,∆n) and X†,(n)s ∈ ⋃ j:mj∈G Ωj ∀s ∈ [0, tk] }
so the conclusion of the proof follows directly from eq. (F.77).
G FIRST EXIT TIME OF TRUNCATED HEAVY-TAILED SGD IN Rd

heading: G.1 MAIN RESULT
text: The object of interests is the following truncated heavy-tailed SGD iterates
Xηk+1(x) = X η k (x) + ϕb ( − η∇f(Xηk (x)) + ηZk+1 ) ∀k ≥ 0
where the initial condition is prescribed by X0(x) = x, f is a real-valued function on Rd, η > 0 is the learning rate, (Zk)k≥1 is the sequence of heavy-tailed noises, the standard gradient clipping operator is ϕb(v) = min{1, b‖v‖} · v with ‖·‖ being L2 norm. To ease notations, we also use Px to denote the conditional law on {Xη0 = x}. Specifically, we are interested in the first exit time of Xηn from a domain G, i.e. the stopping time
σ(η) = min{n ≥ 0 : Xηn /∈ G}.
We work with following assumptions. Assumption G.1. The region G is connected, bounded and open, and 0 ∈ G. Assumption G.2. The function f is smooth, i.e. f ∈ C2(Rd). Assumption G.3. The boundary set ∂G is a (n− 1)-dimensional manifold of class C2 such that the vector field n(·) of the outer normals on ∂G exists with
∇f(v)Tn(v) ≥ c0 ∀v ∈ ∂G (G.1)
for some constant c0 > 0. Assumption G.4. For∇2f(0), the Hessian of f(·) at point 0, all the eigenvalues are strictly positive.
For any x ∈ Ḡ, let xt(x) be the ODE flow with x0(x) = x solving ẋt(x) = −∇f ( xt(x) ) ∀t ≥ 0,
Assumption G.5. W.L.O.G., the origin 0 is an attractor of the domain, i.e. ∇f(0) = 0 and 0 is asymptotically stable in G in the sense that
lim t→∞
xt(x) = 0 ∀x ∈ Ḡ.
We have the following assumption regarding the heavy-tailed noises (Zn)n≥1. For any x ∈ Rd, x 6= 0, define the following polar transformation
T(x) = (‖x‖, x/‖x‖) (G.2)
with Tr(x) = ‖x‖ ,Tθ(x) = x/ ‖x‖. Also, let O = {0} Assumption G.6. EZ1 = 0. Besides, there exist a positive integer m ≥ 1, a sequence 1 < α1 < α2 < · · · < αm <∞, a sequence of slowly-varying functions (l1, · · · , lm), a sequence of probability measures (S1, · · · , Sm) on the unit sphere Sd−1 with support Fj =∆ supp(Sj) as closed sets on Sd−1 such that
• Fi ∩ Fj = ∅ for any i 6= j;
• For any j ∈ [m], define the cone Ej =∆ T−1θ (Fj) ∪O in Rd and measure P(j)(·) = P(Z1 ∈ · ∩ Ej), we have
tαj · lj(t) · P(j) ◦ T−1 ( t · dr × dθ ) → ναj (dr)× Sj(dθ) as t→∞ (G.3)
in the sense of M(Ej\O). Here να is a Borel measure defined on (0,∞) satisfying να[t,∞) = t−α ∀t > 0;
• For measure P(0)(·) =∆ P ( Z1 ∈ · \(∪mj=1Ej) ) and any α > 0,
tα · P(0) ◦ T−1 ( t · dr × dθ ) → 0 as t→∞ (G.4)
in the sense of M(Rd\O);
• For any j ∈ [m], the measure Sj is absolutely continuous w.r.t. spherical measure on Sd−1.
A function l : R+ 7→ R+ is slowly varying (at∞) if limt→∞ l(tx)/l(x) = 1 holds for any x > 0. For details on M−convergence and regular variation in general metric spaces, see Lindskog et al. (2014). Here we state one implication of the assumption. For any j = 0, 1, · · · ,m, let
Hj(x) = ∆ Pj ( {y ∈ Rd : ‖y‖ ≥ x} ) . (G.5)
The assumption above immediately implies that, for any j = 1, · · · ,m, Hj(·) is regularly varying (at ∞) with index −αj (denoted as Hj ∈ RV−αj ), namely limt→∞Hj(tx)/Hj(t) = x−αj ∀x > 0. Meanwhile, for any α > 0, H0(x) = o(1/xα) as x→∞. The asymptotically behavior of the first exit time hinges on the following geometric characterization of the domain G. For any integer k ≥ 1, a sequence of strictly positive real numbers (t2, · · · , tk) with t1 = 0, a sequence of non-zero vectors (w1, · · · , wk) and some η > 0, let t(k) = (t1, · · · , tk),w(k) = (w1, · · · , wk), and define the ODE path with k jumps (clipped at size b) by w(k) = (w1, · · · , wk), we define perturbed ODE path x̃η as
x̃η(0; t(k),w(k)) = ϕb(ηw1);
dx̃η(t, x; t(k),w(k))
dt = −η∇f
( x̃η(t, x; t(k),w(k)) ) ∀t /∈ {t1, t1 + t2, · · · , k∑ j=1 tj}
x̃η(t, x; t(k),w(k)) = x̃η(t−, x; t(k),w(k)) + ϕb(ηwj) if t = j∑ i=1 ti for some j
with the convention that g(t−) =∆ lims↑t g(s) for any function g. Also, when η = 1 we simply write x̃. Now we can assign a cost to each jump wj based on the direction using the following function:
J(w) = { αj − 1 if w 6= 0, w ∈ Ej , ∞ otherwise. (G.6)
Given any set of perturbations described by t(k) ∈ {0} × Rk−1+ ,w(k) ∈ ( Rd\O )k , we identify the destination of the flow as
h(k; t(k),w(k)) = x̃( k∑ j=1 tk; t (k),w(k)).
Besides, define function I(w(k)) = (i1, · · · , im) if #{i ∈ [k] : wi ∈ Ej} = ij for all j ∈ [m]. This allows us to define the following configuration sets
A(i1, · · · , im) =∆ { w(k) ∈ ( Rd\O )k : I(w(k)) = (i1, · · · , im), k = m∑ j=1 ij }
(G.7)
for any (i1, · · · , im) ∈ Nm, i.e., some set of jumps w(k) is said to have configuration (i1, · · · , im) or belong to the configuration set A(i1, · · · , im) if the number of jumps in cone Ej is equal to ij . We can also define the cost for each configuration as
J (i1, · · · , im) = m∑ j=1 (αj − 1)ij . (G.8)
Now we can characterize the minimum cost to exit G:
JG = ∆ min{ k∑ j=1 J(wj) : ∃k ∈ N, t(k) ∈ {0} × Rk−1+ ,w(k) ∈ ( Rd\O )k s.t. x̃(
k∑ j=1 tk,0; t (k),w(k)) /∈ G}. (G.9)
From the boundedness of G, one can see that 0 < JG < ∞ regardless of the actual value of the clipping threshold b > 0. We need the following technical assumption regarding the configurations of jumps that can trigger the exit with the minimum cost.
Assumption G.7. There exists only one array (i1, · · · , im) ∈ Nm such that ∑m j=1 ij(αj − 1) = JG
and (for k = ∑m j=1 im)
∃t(k) ∈ {0} × Rk−1+ ,w(k) ∈ A(i1, · · · , im) s.t. x̃( k∑ j=1 tk,0; t (k),w(k)) /∈ G.
We use i∗ = (i∗1, · · · , i∗m) to denote the unique configuration in Assumption G.7 and k∗ = ∑m j=1 i ∗ j . The implication is that, for a set of jumps t(k),w(k) with any other configuration, one of the following must happen: (i) this set of jumps has a cost strictly higher than JG; (ii) this set of jumps cannot send the ODE flow out of G. Meanwhile, we introduce the following concept as the coverage of a certain configuration set.
G(i1, · · · , im) =∆ { x̃(s,0; t(k),w(k)) :
k = m∑ j=1 ij , t (k) ∈ {0} × Rk−1+ ,w(k) ∈ A(i1, · · · , im), s ∈ [0, k∑ j=1 tj ] } . (G.10)
It is easy to see that, for any configuration (i1, · · · , im) with cost J (i1, · · · , im) ≤ JG , the coverage G(i1, · · · , im) is a closed set, and the following technical assumption holds for (Lebesgue) almost every b > 0. Here the distance d(A,B) = infx∈A,y∈B ‖x− y‖ for any A,B ⊆ Rd, and A◦ is the interior of the set A.
Assumption G.8. For any (i1, · · · , im) ∈ Nm, one of the following must occur:
• d ( G(i1, · · · , im),Gc) > 0;
• ( G(i1, · · · , im) ∩ Gc )◦ 6= ∅ Note that with i∗ and k∗ defined above, we can define the following mapping h from r = (r1, · · · , rk∗),θ = (θ1, · · · , θk∗) ∈ (Sd−1)k ∗ , t = (t1, · · · , tk∗) ∈ {0} × Rk ∗ + such that
h(r,θ, t) = x̃( k∗∑ i=1 ti,0; t,w)
where w = (wi)k ∗ i=1 with wi = riθi. Also, we introduce the concept of type for configuration i ∗. Specifically, define
j(i1, · · · , im) =∆ { (j1, · · · , jk∗) ∈ {0, 1, 2, · · · ,m}k ∗ : #{n : jn = k} = ik ∀k ∈ [m] }
and for any j ∈ j(i∗), we say that w = (w1, · · · , wk∗) ∈ A(i∗) has type j if wi ∈ Eji ∀i ∈ [k∗].
In other words, based on the direction of each jump inw ∈ A(i∗) we group them into different types. Note that |j(i∗)| <∞. Now for any type j ∈ j(i∗), define a (Borel) measure on Rk∗+ × (Sd−1)k
∗ × Rk ∗−1
+ as
µj = ∆ ( k∗∏ i=1 ναji )× ( k∗∏ i=1 Sji)×mk ∗−1 Leb
where, for any α > 0, the measure να is the Borel measure on (0,∞) with να(x,∞) = 1/x1+α. Lastly, define measure µ as µ = ∑ j∈j(i∗) µj . As will be established in Lemma I.2, the following technical assumption is also a very moderate one since it holds for (Lebesgue) almost every b > 0 under the current setting. Assumption G.9. The set h−1 ( ∂G ) has zero mass under the measure µ.
Having specified the problem setting, we are now ready to present Theorem G.1, the main result of this section. The implication of the theorem is clear: under proper scaling, the first exit time σ(η) converges in distribution to an Exponential random variable. Moreover, the scaling λ(η) is roughly of order η1+JG , implying that the first exit time σ(η) is roughly of order 1/η1+G as the learning rate η approaches 0. Theorem G.1. Let Assumptions G.1-G.9 hold. There exists a function λ(η) that is regularly varying (as η ↓ 0) with index (1 + JG) such that, for any x ∈ G and any t > 0,
lim η↓0
Px(σ(η)λ(η) > t) = exp(−qt)
where the constant q = µ ( h−1(Gc) ) .
The proof is provided in Section H. As a concluding remark, we stress that the order of the first exit time is dictated by JG , the minimum cost for exit we introduced above. Bearing obvious similarity to the first exit time analysis of SDE driven by heavy-tailed Lévy processes in Imkeller et al. (2010a), our result can be viewed as a natural generalization when gradient clipping is applied and heavytailed noises may not always align with a finite number of lines. In Imkeller et al. (2010a), the unclipped setting implies that the escape from the domain G can always be achieved with one big jump. However, a big perturbation (in noise) along different directions may correspond to different heavy-tailed indices, i.e., induce different costs given our definition of cost function J(w) in eq. (G.6). In Theorem 1 of Imkeller et al. (2010a), we see that the order of the first exit time is determined by α1, the smallest heavy-tailed index. The underlying reason is that the exit is almost always trigger by a single big jump with the smallest cost. Similarly, in our setting where multiple jumps are required for escape due to the clipping mechanism, we see that the order of the first exit time is not dictated by the number of jumps or the accumulated distances of the jumps, but the smallest possible accumulated cost defined as the summation of J(wi) where wi’s are the jumps the lead to escape from G. As detailed in the proof, this is because such jumps with the smallest costs JG dictates the most likely way for exiting G.

heading: G.2 A SPECIAL CASE: UNIFORM HEAVY TAIL INDEX ALONG ALL DIRECTIONS
text: As stated above, Theorem G.1 deals with general case where the noise distribution is allowed to have different heavy-tailed indices along different directions (see Assumption G.6). As a special case, it is worth noticing that if a single heavy-tailed index α can be used to describe the tail behavior of noises along any direction (this can be easily guaranteed if the heavy-tailed noise is manually injected into a light-tailed setting), then the minimum cost JG will be equal to l∗(α − 1) where l∗ is the minimum number of jumps required for escape. The readers can see that this is a natural extension of our R1 results, implying that the same strong preference for “wide” minima still hold in Rd under truncated heavy-tailed SGD. To be specific, we work with the following assumption about the noise distribution. Assumption G.10. EZ1 = 0. Besides, there exists some α > 1, a slowly-varying functions l, a probability measures S on the unit sphere Sd−1 with support supp(Sj) = Sd−1 such that
• For the measure ν(·) = P(Z1 ∈ · ∩ Ej), we have tα · l(t) · ν ◦ T−1 ( t · dr × dθ ) → να(dr)× S(dθ) as t→∞ (G.11)
in the sense of M(Rd\O). Here να is a Borel measure defined on (0,∞) satisfying να[t,∞) = t−α ∀t > 0;
• The measure S is absolutely continuous w.r.t. spherical measure on Sd−1.
Compared to Assumption G.6, one can easily see that Assumption G.10 is a stronger version with the specific proviso that a single index α can describe the tail of the noise distribution along any direction in Rd. The first exit time results now admit a simplified form. In particular, the cost of jumps will degenerate to the count of jumps in the sense that ∑k i=1 J(wi) = k(α− 1) for any k ≥ 1 and any (w1, · · · , wk) ∈ (Rd\O)k. Moreover, we now have JG = l∗G · (α− 1) where
l∗G = ∆ min{k ∈ N : ∃t(k) ∈ {0} × Rk−1+ ,w(k) ∈
( Rd\O )k s.t. x̃( k∑ j=1 tk,0; t (k),w(k)) /∈ G}
and the measure µ can now be expressed as µ = ( ∏l∗G i=1 να)× ( ∏l∗G i=1 S)×m l∗G−1 Leb . Therefore, the following theorem is merely a restatement of Theorem G.1 in this simplified setting. Still, we present the result to highlight the role of the minimum jump number JG in the first exit time.
Theorem G.2. Let Assumptions G.1-G.5, G.10 and G.7-G.9 hold. There exists a function λ(η) that is regularly varying (as η ↓ 0) with index 1 + l∗G(α− 1) such that, for any x ∈ G and any t > 0,
lim η↓0
Px(σ(η)λ(η) > t) = exp(−qt)
where the constant q = µ ( h−1(Gc) ) .
It is clear that first exit time since the first exit time is roughly of order 1/η1+(α−1)l ∗ G . Therefore, even in the general Rd case, the quantity l∗G , i.e. the minimum count of jumps to escape from a domain G, induces a hierarchy of first exit time as the first exit time from the domain with largest l∗G (requiring most number of jumps to escape) will dominate the first exit time from other regions.
Lastly, if we work under the standard setting such as the ones in Zhou et al. (2020) where local strong convexity of f in the domain G is assumed, the results can be further simplified and the minimum jump number l∗G will be directly tied to the width of the each domain. Specifically, the quantity rG =
∆ supy∈G ‖y‖ can be interpreted the effective width or radius of G, and we work with the following assumption.
Assumption G.11. The function f is strongly convex on the closed ball {y ∈ Rd : ‖y‖ ≤ rG}.
Two consequences follow immediately from this assumption. First of all, there exists some constant c > 0 such that ‖xt(x)‖ ≤ ‖x‖ e−ct ∀x ∈ {y : ‖y‖ < rG}. As a result, we must have l∗G ≥ rG/b.Next, as long as rG/b is not an integer (which holds for Lebesgue almost every b > 0), for k = drG/be we can find t2 > 0, · · · , tk > 0 and w1 6= 0, · · · , wk 6= 0 where x̃ ( s,0; (0, t2, · · · , tk), (w1, · · · , wk) ) /∈ G for some s > 0. In fact, it is worth noticing that Assumption G.8 now degenerates to the condition that rG/b is not an integer, and we now know that for any such b > 0, we have l∗G ≤ drG/be. In summary, we have established the following result indicating that the strong preference for wider minima under truncated heavy-tailed SGD still persists in Rd given proper convexity assumption on f . Theorem G.3. Let Assumptions G.1-G.5, G.10-G.11 and G.7-G.9 hold. There exists a function λ(η) that is regularly varying (as η ↓ 0) with index 1 + l∗(α− 1) such that, for any x ∈ G and any t > 0,
lim η↓0
Px(σ(η)λ(η) > t) = exp(−qt)
where the constant q = µ ( h−1(Gc) ) and l∗ = drG/be.

heading: G.3 RELAXING THE TECHNICAL ASSUMPTIONS
text: In order to achieve the tightest characterization of the first exit time, some assumptions introduced above are slightly stronger than the ones in Imkeller et al. (2010a). For instance, in Assumption G.3 we require ∂G to be of class C2 while the assumption (A3) in Imkeller et al. (2010a) only requires it to be a C1 manifold. Besides, Assumption G.6 requires that Sj , the limiting distribution of the directions of each heavy-tailed component in noises, to be absolutely continuous w.r.t. the spherical measure. As will be stressed in Section I.1, these conditions are only imposed to prove Lemma I.2, thus ensuring that Assumption G.9 holds for almost every b > 0. Briefly speaking, all the efforts to guarantee Assumption G.9 allows us to conclude that the law of scaled first exit time λ(η)σ(η) converges exactly to that of Exp(q) with q = µ ( h−1(Gc) ) .
Fortunately, the discussion below will show that, even when we relax all the extra technical assumptions above (hence removing Assumption G.9), the order of the first exit time is still dictated by the minimum cost for escape JG , and a similar result about the first exit time can be obtained where the distribution of the scaled first exit time λ(η)σ(η) will be asymptotically bounded by two Exponential RVs.
Specifically, we reiterate that the first half of Assumption G.3, i.e. ∂G is a differential manifold, will only be applied to prove Lemma I.2. The second half of the proof, namely the lower bound in eq. (G.1), only serves the ensure that for any x ∈ G, we have xt(x) ∈ G ∀t ≥ 0 so that the gradient flow starting in G will never leave this domain. Now let us focus on the open sets G and G for some small > 0 where
G =∆ {y ∈ Rd : d(y,G) < }, G =∆ {y ∈ G : d(y,Gc) > }. Note that ∩ G = G, ∪ G = G and for any positive reals 1, 2 such that 1 6= 2 and the two reals are small enough for G 1 and G 2 to be non-empty,
∂G 1 ∩ ∂G 2 = ∅, ∂G 1 ∩ ∂G 2 = ∅. Given eq. (G.1), as well as the smoothness of the vector field ∇f (see Assumption G.2) and the fact that G is connected, bounded and open and contains a unique attractor 0 (see Assumption G.1 and G.5), it is easy to see that for all > 0 sufficiently small, we will have xt(x) ∈ G ∀t ≥ 0 for any x ∈ G and xt(x) ∈ G ∀t ≥ 0 for any x ∈ G . Meanwhile, in our proof below we will establish eq. (H.17) (which, again, does not require the C2 class assumption on ∂G or the absolute continuity assumption on the measures Sj). Note that eq. (H.17) implies the existence of ∆ > 0 such that µ ( h−1(G∆) ) <∞. Therefore, for all but only
countably many ∈ (0,∆), we have µ ( h−1(∂G ) ) = µ ( h−1(∂G ) ) = 0. In other words, we can find a sequence of n with limn n = 0 such that µ ( h−1(∂G n) ) = µ ( h−1(∂G n) ) = 0 ∀n ≥ 1
. Also, for a fixed b > 0 that ensures Assumption G.7 and G.8, one can see that similar conditions will also hold for sets G and G as long as > 0 is sufficiently small. In summary, if we consider first exit times
σn(η) =∆ min{k ≥ 0 : Xηk /∈ G n}, σn(η) =∆ min{k ≥ 0 : Xηk /∈ G n}
with the obvious bounds that σn(η) ≤ σ(η) ≤ σn(η) for all n ≥ 1, then it suffices to apply Theorem G.1 directly onto σn(η), σn(η). Note that we did not attempt to establish that ∂G or ∂G are differential manifolds. However, we reiterate that this is not needed since the C2 manifold condition in Assumption G.3 only serves to ensure that Assumption G.9 holds for almost surely every b > 0, which we sidestep by picking a proper sequence n to explicitly satisfy the condition. More formally speaking, consider the following relax ted assumptions. Assumption G.12. The boundary set ∂G is a (n− 1)-dimensional manifold of class C1 such that the vector field n(·) of the outer normals on ∂G exists with
∇f(v)Tn(v) ≥ c0 ∀v ∈ ∂G for some constant c0 > 0. Assumption G.13. EZ1 = 0. Besides, there exist a positive integer m ≥ 1, a sequence 0 < α1 < α2 < · · · < αm <∞, a sequence of slowly-varying functions (l1, · · · , lm), a sequence of probability measures (S1, · · · , Sm) on the unit sphere Sd−1 with support Fj =∆ supp(Sj) as closed sets on Sd−1 such that
• Fi ∩ Fj = ∅ for any i 6= j;
• For any j ∈ [m], define the cone Ej =∆ T−1θ (Fj) ∪O in Rd and measure P(j)(·) = P(Z1 ∈ · ∩ Ej), we have
tαj · lj(t) · P(j) ◦ T−1 ( t · dr × dθ ) → ναj (dr)× Sj(dθ) as t→∞
in the sense of M(Ej\O). Here να is a Borel measure defined on (0,∞) satisfying να[t,∞) = t−α ∀t > 0;
• For measure P(0)(·) =∆ P ( Z1 ∈ · \(∪mj=1Ej) ) and any α > 0,
tα · P(0) ◦ T−1 ( t · dr × dθ ) → 0 as t→∞
in the sense of M(Rd\O).
We say Assumptions G.1-G.2, G.12, G.4-G.5, G.13, G.7-G.8 are the set of relaxed assumptions. The discussion above implies that the following result is an immediate consequence from Theorem G.1. Theorem G.4. Let the relaxed assumptions hold. There exists a function λ(η) that is regularly varying (as η ↓ 0) with index (1 + JG) such that, for any x ∈ G and any t > 0,
exp(−qt) ≤ lim inf η↓0 Px(σ(η)λ(η) > t) ≤ lim sup η↓0 Px(σ(η)λ(η) > t) ≤ exp(−q↑t)
with q = µ ( h−1(Gc) ) , q↑ = µ ( h−1 ( (G)c )) .

heading: H PROOF OF THEOREM G.1
text: This section is devoted to establishing Theorem G.1. For clarity of the exposition, we break down the proof into several major steps, each contained in a subsection below.
H.1 PICKING CONSTANTS t̄, ̄, δ̄
The goal of this subsection is to fix several important constants that characterize the typical sizes and inter-arrival times of large perturbations in SGD trajectory that can cause the escape from domain G. We will abuse the notations slightly when referencing certain constants, and quantities such as c0, c1, c2 may not be equal to the ones used in assumptions above.
First, from Assumption G.6 and the definition of JG , we can define
l∗ = dJG/(α1 − 1)e+ 1 (H.1)
and we must have∞ > l∗ > k∗. Meanwhile, from Assumption G.8, we can find ̄ > 0 such that d ( G(i1, · · · , im),Gc) > 100l∗̄ (H.2)
for any (i1, . . . , im) ∈ Nm with ∑m j=1 ij(αj − 1) < JG , as well as
∃x ∈ G(i∗) such that d(x,G) > 100l∗̄.
Furthermore, Assumption G.5 implies that ∇f(v) 6= 0 for any v ∈ G\{0}. From Assumptions G.3, G.4 and G.5 (and by picking a smaller ̄ > 0 if needed), one can see the existence of some c0 > 0, ̄ > 0 such that
‖xt(x)‖ ≤ e−c0t ‖x‖ ∀t ≥ 0, x ∈ B(0, ̄), (H.3) ‖∇f(x)‖ ≥ c0 ∀x ∈ G\B(0, ̄), (H.4)
d ( xt(x),Gc ) ≥ d(x,Gc) + c0t if xs(x) ∈ G and d ( xs(x),Gc ) ≤ ̄ ∀s ∈ [0, t]. (H.5)
Here B(x, r) = {y ∈ Rd : ‖x− y‖ < r} is the open ball centered at x with radius r > 0. One immediate consequence from eq. (H.5) is that
d ( xt(x),Gc ) ≥ min{d(x,Gc), ̄} ∀x ∈ G, t ≥ 0. (H.6)
Besides, the boundedness of domain G and smoothness of f(·) imply the existence of some M > 0 such that
G ⊆ B(0,M), (H.7) ‖∇f(x)‖ ≤M ∀x ∈ G, (H.8)∥∥∇2f(x)∥∥ ≤M ∀x ∈ G (H.9)
where for the matrix norm we use spectral norm.
Recall the definitions of the configuration sets A(i1, · · · , im), the unique configuration i∗ and count of jumps k∗ to trigger the exit with minimum cost (see Assumption G.7 and the remark underneath). The following three results provide control on the sizes and inter-arrival times of typical jumps that can trigger the escape from G. The proofs involve analyses of the deterministic dynamical system x̃t and will be provided in Section I. In particular, we introduce another concept as type of jumps (w1, · · · , wk) that is similar configuration (i1, · · · , im) and configuration sets A(i1, · · · , im). For any k ∈ N and any (w1, · · · , wk) ∈ (Rd\O)k, we say that the (w1, · · · , wk) is of type-j for some j = (j1, · · · , jk) ∈ {1, 2, · · · ,m}k iff
wi ∈ Eji ∀i = 1, · · · , k. (H.10)
In other words, if w1 and w2 are of the same type j, then they have the same cardinality |w1| = |w2| = |j| = k for some k ∈ N; moreover, for any i ∈ [k], the jumps w1i and w2i are both in the cone Eji so both jumps point at directions in cone Eji and have the same cost J(w 1 i ) = J(w 2 i ) = αji − 1 for some ji ∈ [m]. Define the set
Atypej = {w ∈ (R d\O)|j| : w is of type-j}.
The accumulated cost for any w ∈ Atypej is defined as Jtype(j) = ∑|j| i=1(αji − 1). Lastly, define the skip-one accumulated cost for any type j as
J ↓type(j) = max{ ∑
i=1,··· ,|j|,i6=k
(αji − 1) : k = 1, 2, · · · , |j|},
i.e., J ↓type(j) is the highest possible accumulated cost if we remove one element in w for w ∈ A type j .
Lemma H.1. There exist some t̄ ∈ (0,∞), δ̄ ∈ (0,∞), 0 > 0 such that the following claim holds. Let k′ ∈ N and j = (j1, · · · , jk′) ∈ {1, 2, · · · ,m}k ′ be such that J ↓type(j) < JG . For any t(k ′) = (t1, · · · , tk′) ∈ {0}×Rk ′−1 + ,w (k′) = (w1, · · · , wk′) ∈ Atypej , the following set of conditions
tj < t̄ ∀j = 2, 3, · · · , k′ (H.11) ‖wj‖ > δ̄ ∀j = 1, 2, · · · , k′. (H.12)
is the necessary condition for infs≥0 d ( x̃(s,0; t(k),w(k)),Gc ) ≤ 0.
Lemma H.2. Let t̄, δ̄, 0 be the positive constants prescribed in Lemma H.1. There exists some 1 > 0 such that the following claim holds.
Let k′ ∈ N and j = (j1, · · · , jk′) ∈ {1, 2, · · · ,m}k ′ be such that J ↓type(j) < JG . For any x ∈ B(0, 1), any t(k ′) = (t1, · · · , tk′) ∈ {0} × Rk ′−1 + ,w (k′) = (w1, · · · , wk′) ∈ Atypej , the following set of conditions
tj < 2t̄ ∀j = 2, 3, · · · , k′ (H.13) ‖wj‖ > δ̄/2 ∀j = 1, 2, · · · , k′. (H.14)
is the necessary condition for infs≥0 d ( x̃(s, x; t(k),w(k)),Gc ) ≤ 0/2.
Lemma H.3. There exist some 0 > 0, δ0 > 0 such that
sup s≥0
d ( x̃(s, x; t(k),w(k)),Gc ) > 0
for all x ∈ B(0, δ0), all (i1, · · · , im) ∈ Nm with J (i1, · · · , im) < JG and k = ∑m j=1 ij , and all w(k) ∈ A(i1, · · · , im), t(k) ∈ {0} × Rk−1+ .
Similar to the ODE path x, we defined the following paths with rate η > 0: ˙(xηt )(x) = −η∇f ( xηt (x) ) ∀t ≥ 0, (H.15)
xη0(x) = x. (H.16)
In other words, ODE flow xt is equivalent to x η t with rate η = 1. The next result gives us an upper bound for the return time to a neighborhood of the local minimum as for the gradient flow.
Lemma H.4. For τηODE(x, ) = ∆ min{t ≥ 0 : xηt (x) ∈ B(0, )}, there exists c1 ∈ (0,∞) such that for any ∈ (0, ̄),
τηODE(x, ) ≤ c1 + c1 log(1/ )
η ∀x ∈ G.
Proof. Due to eq. (H.4), we know that T0 = ∆ supx∈Ḡ f(x)−infx∈Ḡ f(x)
c0 <∞ and
sup x∈Ḡ
τ1ODE(x, ̄) ≤ T0.
Furthermore, from eq. (H.3) one can see that
sup x∈Ḡ
τ1ODE(x, )− τ1ODE(x, ̄) ≤ log(1/ )− log(1/̄)
c0 .
Now combining these two bounds with a 1/η time scaling, we have
sup x∈Ḡ
τηODE(x, ) ≤ c1 + c1 log(1/ )
η
for c1 = max{T0 + log(̄)c0 , 1 c0 }.
With Lemma H.1-H.4, we are able to pick some constants to facilitate the analysis below. By decreasing ̄ and increasing M if needed, we can assume the existence of some t̄ <∞, δ̄ < ̄ such that
• (Due to Lemma H.1 and H.2) For any k ∈ N and j ∈ {1, 2, · · · ,m}k such that J ↓type(j) < JG , any x ∈ B(0, 100l∗̄), any t(k) = (t1, · · · , tk) ∈ {0}×Rk−1+ ,w(k) = (w1, · · · , wk) ∈ Atypej with
d ( x̃( k∑ j=1 tj , x; t (k),w(k)),Gc ) < 100l∗̄,
it holds that
tj < t̄ ∀j = 2, 3, · · · , k and ‖wj‖ > δ̄ ∀j = 1, 2, · · · , k. (H.17)
• For any (i1, . . . , im) ∈ Nm with J (i1, . . . , im) < JG , d ( G(i1, · · · , im),Gc) > 100l∗̄; (H.18)
• There exists some x ∈ G(i∗) such that d(x,G) > 100l∗̄; (H.19)
• (Due to Lemma H.3) The constant ̄ > 0 is sufficiently small such that
sup s≥0
d ( x̃(s, x; t(k),w(k)),Gc ) > 100l∗̄ (H.20)
for all x ∈ B(0, 100l∗̄), all (i1, · · · , im) ∈ Nm with J (i1, · · · , im) < JG and k =∑m j=1 ij , and all w
(k) ∈ A(i1, · · · , im), t(k) ∈ {0} × Rk−1+ ; • (Due to Lemma H.4) The constant t̄ <∞ is large enough so that
sup x∈Ḡ
τηODE(x, ̄) < t̄/η.

heading: H.2 BOUNDING THE DISTANCE BETWEEN ODE FLOW AND SGD ITERATES
text: Moving on, we show that, without large jumps, the SGD iterates Xηn(x) are unlikely to show significant deviation from the deterministic gradient descent process yηn(x) defined as
yη0(x) = x, (H.21)
yηn(x) = y η n−1(x)− η∇f ( yηn−1(x) ) . (H.22)
We are ready to state the first lemma, where we bound the distance between the gradient descent iterates yηn(y) and the ODE xη(t, x) when the initial conditions x, y are close enough. Lemma H.5. Given t > 0 and x, y ∈ G such that the line segment between xη(k, x) and yηk(y) lies in G for any k = 1, 2, · · · , btc,
sup s∈[0,t] ∥∥∥xη(s, x)− yηbsc(y)∥∥∥ ≤ (2ηM + ‖x− y‖) exp(ηMt) where M ∈ (0,∞) is the constant in eq. (H.8)eq. (H.9).
Proof. Define a continuous-time process yη(s; y) =∆ yηbsc(y), and note that xη(s, x) = xη(bsc, x)− η ∫ s bsc ∇f(xη(u, x))du
xη(bsc, x) = x− η ∫ bsc
0
∇f(xη(u, x))du
yηbsc(y) = y η(bsc, y) = y − η ∫ bsc 0 ∇f(yη(u, y))du.
Therefore, if we define function
b(u) = xη(u, x)− yη(u, y), from the fact ‖∇f(·)‖ ≤ M , one can see that ‖b(u)‖ ≤ ηM + ‖x− y‖ for any u ∈ [0, 1) and ‖b(1)‖ ≤ 2ηM + ‖x− y‖. In case that s > 1, from the display above and the fact
∥∥∇2f(·)∥∥ ≤M , we now have ∥∥∥yηbsc(x)− xη(s, x)∥∥∥ ≤ ‖b(bsc)‖+ ηM ;
‖b(bsc)‖ ≤ ηM ∫ bsc
1
‖b(u)‖ du.
From Gronwall’s inequality (see Theorem 68, Chapter V of Protter (2005), where we let function α(u) be α(u) = ‖b(u+ 1)‖), we have∥∥∥yηbsc(x)− xη(s, x)∥∥∥ ≤ (2ηM + ‖x− y‖) exp(ηMt). This concludes the proof.
Now we consider an extension of the previous Lemma in the following sense: when both perturbed by at most l∗ similar perturbations, the ODE flow and gradient descent process should still stay close enough. Analogous to the definition of the perturbed ODE x̃η, we can construct a process ỹη as a perturbed gradient descent process as follows. For some integer 1 ≤ l ≤ l∗, a sequence of strictly positive integers (t2, · · · , tl) (with convention t1 = 0 and let t(l) = (t1, · · · , tl)), a sequence of non-zero vectors w̃(l) = (w̃1, · · · , w̃l) and y ∈ Rd, define the perturbed gradient descent iterates with gradient clipping at b as
ỹηn(y; t (l), w̃(l))
=ỹηn−1(y; t (l), w̃(l)) + ϕb ( − η∇f(ỹηn−1(y; t(l), w̃(l))) + l∑ j=2 1{n = t1 + · · ·+ tj}w̃j ) (H.23)
with initial condition ỹη0 (y; t (l), w̃(l)) = y + ϕb(w̃1).
Corollary H.6. For any fixed > 0, t > 0, the following claim holds for all η > 0 sufficiently small: given any x, y ∈ G, some integer 1 ≤ l ≤ l∗, a sequence of strictly positive integers t(l) = (tj)lj=2 (with convention t1 = 0), and any two sequences of vectorsw(l) = (wj)lj=1, w̃ (l) = (w̃j) l j≥1 such
that (let tend = ∑l j=1 tj)
• |x− y| < ;
• tj ≤ 2t/η for all j ∈ [l];
• |wj − w̃j | < for all j ∈ [l];
• The line segment between x̃η(k, x; t(l),w(l)) and ỹηk(y; t (l), w̃(l)) lies in G for any k ≤
tend − 1,
then we have
sup s∈[0,tend] ∥∥∥x̃η(s, x; t(l),w(l))− ỹηbsc(y; t(l), w̃(l))∥∥∥ ≤ ρ̄(t) where the constant ρ̄ = (2 exp(2Mt) + 2)l ∗ . In particular, ρ̄ = ρ̄(t̄) is a constant where t̄ is the constant in eq. (H.11).
Proof. Throughout this proof, fix some η ∈ (0, /2M). We will show that for any η in the range the claim would hold.
First, on interval [0, t2), from Lemma H.5, one can see that (since 2Mη < )
sup s∈[0,t2) ∥∥∥x̃η(s, x; s(l),w(l))− ỹηbsc(y; t(l), w̃(l))∥∥∥ ≤ 2 exp(2Mt) · . The at t = t2, by considering the difference between w2 and w̃2, and the possible change due to one more gradient descent step (which is bounded by ηM < ), we have
sup s∈[0,t2] ∥∥∥x̃η(s, x; s(l),w(l))− ỹηbsc(y; t(l), w̃(l))∥∥∥ ≤ (2 exp(2Mt) + 2) · . Now we proceed inductively. For any j = 2, 3, · · · , l − 1, assume that
sup s∈[0,tj ] ∥∥∥x̃η(s, x; s(l),w(l))− ỹηbsc(y; t(l), w̃(l))∥∥∥ ≤ (2 exp(2Mt) + 2)j−1 · . Then by focusing on interval [tj , tj+1] and using Lemma H.5 again, one can show that
sup t∈[tj ,tj+1] ∥∥∥x̃η(s, x; s(l),w(l))− ỹηbsc(y; t(l), w̃(l))∥∥∥ ≤ 2 + ((2 exp(2Mt) + 2)j−1 + 1) exp(2Mt) ≤ (2 exp(2Mt) + 2)j · .
This concludes the proof.
Using an almost identical approach, one can show the following results that bounds the distance between two ODE flows with similar initial values and jumps.
Corollary H.7. There exists some constant ρ∗ ∈ (0,∞) such that the following claim holds: For any fixed > 0, any x, y ∈ G, any integer 1 ≤ l ≤ l∗, a sequence of strictly positive integers t(l) = (tj) l j=2 (with convention t1 = 0), and any two sequences of vectors w (l) = (wj) l j=1, w̃ (l) = (w̃j) l j≥1 such that (let tend = ∑l j=1 tj)
• |x− y| < ;
• tj ≤ 2t̄/η for all j ∈ [l];
• |wj − w̃j | < for all j ∈ [l];
• The line segment between x̃η(s, x; t(l),w(l)) and x̃η(s, y; t(l), w̃(l)) lies in G for any s < tend,
then we have
sup s∈[0,tend] ∥∥∥x̃η(s, x; t(l),w(l))− x̃η(s, y; t(l), w̃(l))∥∥∥ ≤ ρ∗ . In the next few results, we show that a similar type of control can be obtained on the distance between gradient descent iterates ỹηn and the SGD iteratesX η n. Specifically, our first goal is to show that before any large jump, it is unlikely that the (deterministic) gradient descent process yηn would deviate too far from Xηn. To facilitate the analysis below, we introduce some additional notations. First, we will group the noises Zn based on a threshold level δ > 0: let us define
Z≤δ,ηn = ∆ Zn1{η ‖Zn‖ ≤ δ}, (H.24)
Z>δ,ηn = ∆ Zn1{η ‖Zn‖ > δ}. (H.25)
The former are viewed as small noises while the latter will be referred to as large noises or large jumps. Furthermore, for any j ≥ 1, define the jth arrival time, size, and direction of large jumps as
T ηj (δ) = ∆ min{n > T ηj−1(δ) : η ‖Zn‖ > δ}, T η 0 (δ) = 0, (H.26) W ηj (δ) = ∆ ZTηj (δ), (H.27)
Θηj (δ) = ∆ W ηj (δ)/ ∥∥W ηj (δ)∥∥ . (H.28) The following event
A(n, η, , δ) = {
max k=1,2,··· ,n∧(Tη1 (δ)−1)
η ‖Z1 + · · ·+ Zk‖ ≤ }
(H.29)
describes a scenario where, before the first large jump, not much perturbation has been caused by noises.
Lemma H.8. For any > 0, η > 0, any x, y ∈ G and positive integer n such that ‖x− y‖ < 2 exp(ηMn) , on event
A(n, η, 2 exp(ηMn) , δ) ⋂
{ the line segment between yηj (y) and X η j (x) lies in G ∀j = 0, 1, · · · , n ∧ (T η 1 (δ)− 1) } ,
we have ‖yηm(y)−Xηm(x)‖ < ∀m = 1, 2, · · · , n ∧ (T η 1 (δ)− 1).
Proof. On the said event, we are able to apply Lemma E.5 and obtain that∥∥yηj (y)−Xηj (x)∥∥ ≤ (‖x− y‖+ 2 exp(ηMn) ) exp(ηMj) < ∀j = 1, 2, · · · , n ∧ (T η1 (δ)− 1) and conclude the proof.
Similar to the extension from Lemma H.5 to Corollary H.6, we can extend Lemma H.8 to show that, if we consider the a gradient descent process that is only perturbed by large noises, then it should stay pretty close to the SGD iterates Xηn. To be specific, let
Y η0 (x) = x
Y ηn (x) = Y η n−1(x)− ϕb
( − η∇f ( Y ηn−1(x) ) + ∑ j≥1 1{n = T ηj (δ)}ηZn ) .
be a gradient descent process (with gradient clipping at threshold b) that is only perturbed by large noises. The next corollary can be shown by an approach identical to the one for Corollary H.6 (namely, inductively repeating Lemma H.8 at arrival time of each jump) so we omit the details here.
Corollary H.9. There exists some function ρ̃ : (0,∞) 7→ (0,∞) such that the following claim hold for all > 0, t > 0 and all sufficiently small η > 0: For any x, y ∈ G) with ‖x− y‖ < and any l ∈ [l∗], on event A0( , η, δ, l, t) ∩B0( , η, δ, l, t) ∩ C0( , η, δ, l, t), we have
‖Y ηn (y)−Xηn(x)‖ < ρ̃(t) ∀n = 1, 2, · · · , T η l (δ)
where
A0( , η, δ, l, t) = ∆ { ∀i = 1, · · · , l, max
j=Tηi−1(δ)+1,··· ,T η i (δ)−1
η ∥∥∥ZTηi−1(δ)+1 + · · ·+ Zj∥∥∥ ≤ 2 exp(2tM)};
B0( , η, δ, l, t) = ∆ { ∀j = 2, · · · , l, T ηj (δ)− T η j−1(δ) ≤ 2t/η } ,
C0( , η, δ, l, t) = ∆ {the line segment between Y ηj (y) and X η j (x) lies in G ∀j = 0, 1, · · · , T η l (δ)− 1}.

heading: In particular, ρ̃ = ρ̃(t̄) is a constant
text: The next result shows that the type of events A(n, η, , δ) defined in eq. (H.29) is indeed very likely to occur, especially for small . For clarity of the presentation, we introduce the following definitions that are slightly more general than the small and large jumps defined in eq. (H.24)eq. (H.25) (for any c > 0)
Z≤cn = ∆ Zn1{‖Zn‖ ≤ c}, Z>cn = ∆ Zn1{‖Zn‖ > c}.
Lemma H.10. Given any t > 0, > 0, and N > 0, the following holds for any sufficiently small δ > 0:
P (
max n=1,2,··· ,dt/ηe
η ∥∥∥Z≤δ/η1 + · · ·+ Z≤δ/ηn ∥∥∥ > ) = o(ηN )
as η ↓ 0.
Proof. For any C ∈ (0,∞), δ > 0, β ∈ (0, 1), consider the following decomposition of the small noise Z≤δ/ηn . Here we adopt the convention that E0 = Rd\(∪mj=1Ej). (For definitions and properties of the closed cones E1, · · · , Em, see Assumption G.6 and the remark underneath.)
Z(j)n = ∆ Zn1{Zn ∈ Ej},
Z≤C,(j)n = ∆ Z(j)n 1{ ∥∥∥Z(j)n ∥∥∥ ≤ C}, Z ↓,(j) C,δ,η(n) = ∆ Z(j)n 1{ ∥∥∥Z(j)n ∥∥∥ ∈ (C, δ/η]}
Based on this decomposition, we define the following iid random variable sequences
z̃(j)(n) =∆ Z≤C,(j)n − EZ≤C,(j)n , Z̃(j)(n) =∆ ∥∥∥Z↓,(j)C,δ,η(n)∥∥∥ ,
Z̃ (j) β (n) = ∆ ∥∥∥Z≤1/ηβ ,(j)n ∥∥∥ . Meanwhile, define the projection operators Pi : Rd 7→ R with Pi(v1, · · · , vd) = vi. First, recall that EZ1 = 0. So for all C sufficiently large, we have
sup y≥C ∥∥∥EZ≤y1 ∥∥∥ = sup y≥C ∥∥EZ>y1 ∥∥ < 2t . Moreover, due to Hj ∈ RV−(αj+1) for all j = 1, 2, · · · ,m and H0(x) = o(1/xα) for any α > 0 (for definition of functions Hj , see eq. (G.5)), we know that E
∥∥∥Z(j)n ∥∥∥ <∞ for any j = 0, 1, · · · ,m, implying that for all C large enough, we have (for any j = 0, 1, · · · ,m)
sup y,z≥C ∥∥∥EZ≤y,(j)1 − EZ≤z,(j)1 ∥∥∥ < 12t · 24√m+ 1 . (H.30)
Fix such C. Besides, recall that in Assumption G.6 we have 1 < α1 < α2 < · < αm. Therefore, we are able fix some β ∈ (0, 1) and δ > 0 so that
βα1 > 1, (H.31)
(N + 1)δ < 12 √ m+ 1 , (H.32)
( N βα1 − 1 + 1 ) δ < 12 √ m+ 1 . (H.33)
With the fixed C and δ, note that eventually δ/η > C as η ↓ 0. Therefore, for all η sufficiently small, P (
max n=1,2,··· ,dt/ηe
η ∥∥∥Z≤δ/η1 + · · ·+ Z≤δ/ηn ∥∥∥ > )
≤P (
max n=1,2,··· ,dt/ηe η ∥∥∥∥∥ n∑ k=1 Z ≤δ/η k − EZ ≤δ/η k ∥∥∥∥∥ > 2)
=P (
max n=1,2,··· ,dt/ηe η ∥∥∥∥∥∥ m∑ j=0 n∑ k=1 Z ≤C,(j) k − EZ ≤C,(j) k + Z ↓,(j) C,δ,η(k) + EZ ≤C,(j) k − EZ ≤δ/η,(j) k ∥∥∥∥∥∥ > 2 )
≤ m∑ j=0 P ( max n=1,2,··· ,dt/ηe η ∥∥∥∥∥ n∑ k=1 Z ≤C,(j) k − EZ ≤C,(j) k + Z ↓,(j) C,δ,η(k) + EZ ≤C,(j) k − EZ ≤δ/η,(j) k ∥∥∥∥∥ >
2 √ m+ 1 ) ≤
m∑ j=0 P ( max n=1,2,··· ,dt/ηe η ∥∥∥∥∥ n∑ k=1 z̃(j)(k) ∥∥∥∥∥ > 6√m+ 1) (H.34) +
m∑ j=0 P ( η dt/ηe∑ k=1 Z̃(j)(k) > 6 √ m+ 1 ) . (H.35)
We bound the two terms eq. (H.34) and eq. (H.35) respectively. First, observe that
eq. (H.34) ≤ m∑ j=0 d∑ i=1 P ( max n=1,2,··· ,dt/ηe η ∣∣ n∑ k=1 Pi ( z̃(j)(k) )∣∣ > 6 √ (m+ 1)d ) .
Let z̃(j)i (k) = Pi ( z̃(j)(k) ) and let z̃(j)i be an iid copy. By definition, |z̃ (j) i | ≤ 2C. Then from Hoeffding’s inequality,
eq. (H.34) ≤ 2(m+ 1)d · exp ( − 1 2 · ( 6 √ d(m+ 1) )2 · 1/η2 2Cdt/ηe ) = o(ηN ).
As for term eq. (H.35), w.l.o.g. let us fix some j = 0, 1, · · · ,m and bound
P ( η dt/ηe∑ n=1 Z̃(j)(n) > 6 √ m+ 1 ) (H.36)
= k−1∑ i=0 P ( η dt/ηe∑ n=1 Z̃(j)(n) > 6 √ m+ 1 , I = i )
︸ ︷︷ ︸ ,(I)
+P ( η dt/ηe∑ n=1 Z̃(j)(n) > 6 √ m+ 1 , I ≥ k )
︸ ︷︷ ︸ ,(II)
where I =∆ #{n = 1, 2, · · · , dt/ηe : Z̃(j)(n) > 1/ηβ} and k is some positive integer such that
kδ < 12 √ m+ 1 , k(βα1 − 1) > N.
Note that we can find such k due to our choice of constants in eq. (H.31)-eq. (H.33).
For (I) and any i = 0, 1, · · · , k − 1, observe that (the non-negative RVs Z̃β(n) and Z̃(n) are defined at the beginning of the proof)
(I) ≤ ( dt/ηe i ) · P ( η dt/ηe−i∑ n=1 Z̃(j)(n) > 6 √ m+ 1 − iη · δ η , Z̃(j)(n) ≤ 1/ηβ ∀n ∈ [dt/ηe − i] )
≤ dt/ηei · P ( dt/ηe−i∑
n=1
Z̃(j)(n) > 12 √ m+ 1 · 1 η , Z̃(j)(n) ≤ 1/ηβ ∀n ∈ [dt/ηe − i]
)
≤ dt/ηei · P ( dt/ηe∑ n=1 Z̃ (j) β (n) > 12 √ m+ 1 · 1 η ) .
Let W̃ (j)(n) =∆ Z̃(j)β (n) − EZ̃ (j) β (n) and let W̃ (j) be an iid copy. Due to eq. (H.30), for all η sufficiently small we will have that dt/ηe|EZ̃(j)β (1)| < 24 √ m+1
· 1η . Therefore, for all η sufficiently small,
(I) ≤ dt/ηei · P ( | dt/ηe∑ n=1 W̃ (j)(n)| > 24 √ m+ 1 · 1 η ) . (H.37)
Observe that |W̃ (j)| ≤ 2/ηβ and
E(W̃ (j))2 = varZ̃(j)β ≤ E(Z̃ (j) β ) 2 = ∫ ∞ 0 2yP(Z̃(j)β > y)dy = ∫ 1/ηβ 0 2yHj(y)dy.
Note that there are only two possibilities:
• If j 6= 0 and αj ≤ 2, then from Karamata’s theorem, we have E(Z̃(j)β )2 ∈ RV−β(2−αj)(η) and note that β(2− αj) < 1;
• Otherwise, one can find some ∆ > 0 such that Hj(x) = o(1/x2+∆), implying that E(Z̃(j)β )2 ≤ ∫∞ 0 2yHj(y)dy <∞ regardless of the actual value of η > 0.
In any case, using Bernstein’s inequality, we can conclude that
eq. (H.37) ≤ dt/ηei · 2 exp ( −
1 2
(
12 √ m+1
)2 /η2
dt/ηeE(W̃ (j))2 + 13 ( 12 √ m+1 ) /η1−β
) = o(ηN ).
On the other hand, (II) ≤ P(I ≥ k) ≤ ( dt/ηe k ) · P ( Z̃(j)(n) > 1/ηβ ∀n = 1, . . . , k ) ≤ dt/ηek · ( Hj(1/η β) )k ,
which is upper bounded by a function regularly varying w.r.t. η with index k ( βα1 − 1 ) > N . Therefore, (II) = o(ηN ) and we conclude the proof.

heading: H.3 ANALYZING DIFFERENT SCENARIOS BEFORE THE FIRST EXIT OR RETURN
text: Using results and arguments above, we are able to illustrate the typical behavior of the SGD iterates Xηn in the following two scenarios. First, we show that, when starting from anywhere in G, the SGD iterates Xηn will most likely return to the neighborhood of the 0 within a short period of time without exiting Ω. Lemma H.11. For any ∈ (0, ̄), the following claim holds:
lim η↓0 inf x:d(x,Gc)>
Px ( Xηn ∈ G ∀n ≤ Treturn(η, )and Treturn(η, ) ≤ ρ( )/η ) = 1
where the stopping time involved is defined as Treturn(η, ) =
∆ min{n ≥ 0 : ‖Xηn‖ < 2 } and the function ρ( ) = c1 + c1 log(2/ ) with c1 ∈ (0,∞) being the constant specified in Lemma H.4.
Proof. Let t = ρ( ) = c1 + c1 log(2/ ) and n = dt/ηe and arbitrarily choose some x with d(x,Gc) > . Also, fix some N > 0. Due to Lemma H.10, we are able to pick some δ > 0 such that (note that ηn ≤ 2t eventually as η ↓ 0)
P (
max n=1,2,··· ,dt/ηe η ∥∥∥Z≤δ/η1 + · · ·+ Z≤δ/ηn ∥∥∥ > 2 exp(2ηnM)) = o(ηN )
First, from Lemma H.5 we know that for any η small enough such that
2ηM exp(2tM) < /2,
we must have ∥∥∥xη(s, x)− yηbsc(x)∥∥∥ ≤ /2 ∀s ∈ [0, n]. Next, using Lemma H.8, we see that on event
A(n, η, 2 exp(ηnM) , δ) ⋂ {T η1 (δ) > dt/ηe},
we must have
‖yηk(x)−X η k (x)‖ < ∀k = 0, 1, · · · , dt/ηe.
Combining these facts with the Lemma H.4, we have shown that, for all η ∈ (0, 4M exp(2tM) ), on event A(n, η, 2 exp(ηnM) , δ) ⋂ {T η1 (δ) > dt/ηe} we must have Xηn ∈ G ∀n ≤ Treturn(η, ) and Treturn(η, ) ≤ ρ( )/η. In other words,
lim η↓0 sup x:d(x,Gc)>
Px ( Xηn /∈ G for some n ≤ Treturn(η, ) OR Treturn(η, ) > ρ( )/η ) ≤ lim η↓0 P ( A(n, η, 2 exp(ηnM) , δ)c ) + lim η↓0 P(T η1 (δ) ≤ dt/ηe).
However, our choice of δ at the beginning of the proof implies that limη↓0 P ( A(n, η, 2 exp(ηnM) , δ) c ) = 0. Meanwhile, from Assumption G.6, we see that
P(T η1 (δ) ≤ dt/ηe) = dt/ηe∑ k=1 ( 1− m∑ j=0 Hj(δ/η) )k−1 · ( m∑ j=0 Hj(δ/η))
≤(m+ 1)H1(δ/η) dt/ηe∑ k=1 ( 1−H1(δ/η) )k−1 for all η sufficiently small (due to regularly varying nature of Hj and 1 < α1 < · · · < αm). In particular, since H1 ∈ RV−α1 , for any fixed ∆ ∈ (0, α1 − 1), we have (for all η sufficiently small),
P(T η1 (δ) ≤ dt/ηe) ≤(m+ 1)ηα1−∆ dt/ηe∑ k=1 (1− ηα1+∆)k−1 = m+ 1 η2∆ P(U(η) ≤ dt/ηe)
where U(η) =d Geom(ηα1+∆). Lemma E.4 then tells us that, for all η sufficiently small, P(U(η) ≤ dt/ηe) ≤ 2t η α1−1+∆, thus implying
lim sup η↓0 P(T η1 (δ) ≤ dt/ηe) ≤ lim sup η↓0
2(m+ 1)
t ηα1−1−∆ = 0
and concluding the proof.
In the next result, we show that, once entering a −small neighborhood of the local minimum, the SGD iterates will most likely stay there until the next large jump. Lemma H.12. Given N0 > 0, the following claim holds for all ∈ (0, ̄/3) and all δ > 0 that is sufficiently small:
sup x∈B(0,2 )
P ( ∃n < T η1 (δ) s.t. ‖Xηn(x)‖ > 3 ) = o(ηN0)
as η ↓ 0.
Proof. Let t = c1 + c1 log(1/ ) where c1 ∈ (0,∞) is the constant specified in Lemma H.4, and n = dt/ηe. Fix some β > 1 + α1 and N > β − 1 +N0. Due to Lemma H.10, the following claim holds for all δ that is sufficiently small:
P (
max k=1,2,··· ,dt/ηe η ∥∥∥Z≤δ/η1 + · · ·+ Z≤δ/ηk ∥∥∥ > /22 exp(2ηnM)) = o(ηN )
We fix one of such δ. Therefore, for event Ã =∆ { ∃j ≤ d 1/η β
dt/ηe e s.t.
max k=1,2,··· ,dt/ηe η ∥∥∥Z≤δ/η1+(j−1)dt/ηe + Z≤δ/η2+(j−1)dt/ηe + · · ·+ Z≤δ/ηk ∥∥∥ > /22 exp(2ηnM)},
the iid nature of noises (Zn)n≥1 implies that
P(Ã) ≤(1 + 1 tηβ−1
)P (
max k=1,2,··· ,dt/ηe η ∥∥∥Z≤δ/η1 + · · ·+ Z≤δ/ηk ∥∥∥ > /22 exp(2ηnM))
=o(ηN−β+1) as η ↓ 0.
Meanwhile, Note that T η1 (δ) ≤ T η,(1) 1 (δ) = ∆ min{n ≥ 0 : η|Zn| > δ, Zn ∈ E1}. Therefore, P(T η1 (δ) > 1/ηβ) ≤ P(T η,(1) 1 (δ) > 1/η β). Specifically, note that T η1 (δ) = d Geom(H1(δ/η)) and H1 ∈ RV−α1 . Lemma E.3 then tells us the existence of some c > 0 such that
P(T η1 (δ) > 1/η β) = o
( exp(−c/η) ) as η ↓ 0. In summary, we have established that
P(Ã ∪ {T η1 (δ) > 1/ηβ}) = o(ηN0).
Now let us focus on the complementary event (Ã)c ∩ {T η1 (δ) ≤ 1/ηβ}. On this event, there must be some j ≤ d 1/η β
dt/ηee such that 1 + (j − 1)dt/ηe ≤ T η 1 (δ) ≤ jdt/ηe.
Let us arbitrarily choose some x ∈ B(0, 2 ). If j = 1, then due to eq. (H.3), we know that xη(s, x) ∈ B(0, 2 ) for all s ≥ 0. Also, from Lemma H.5 we know that for any η small enough such that 2ηM exp(2tM) < /2, we must have∥∥∥xη(s, x)− yηbsc(x)∥∥∥ ≤ /2 ∀s ∈ [0, dt/ηe]. Besides, on event (Ã)c ∪ {T η1 (δ) ≤ 1/ηβ} ∩ {j = 1}, due to Lemma H.8, we must have
‖yηk(x)−X η k (x)‖ < /2 ∀k = 0, 1, · · · , T η 1 (δ)
with T η1 (δ) < dt/ηe. In conclusion, in the case that j = 1, we must have ‖Xηn(x)‖ < 3 for all n < T η1 (δ).
Now consider the case with j ≥ 2. Similarly, due to Lemma H.5 and H.8, we now have that∥∥∥xη(s, x)− yηbsc(x)∥∥∥ ≤ /2 ∀s ∈ [0, dt/ηe], ‖yηk(x)−X η k (x)‖ < /2 ∀k = 0, 1, · · · , dt/ηe,
‖xη(s, x)‖ < 2 ∀s ∈ [0, dt/ηe].
In particular, due to our choice of t at the beginning of the proof and Lemma H.4, one can see that ‖xη(dt/ηe, x)‖ < . Collecting all the results, we now know that
‖Xηk (x)‖ < 3 ∀k ≤ dt/ηe, ∥∥∥Xηdt/ηe(x)∥∥∥ < 2 .
Now it suffices to apply the repeated apply the same arguments above. For instance, if j = 2, then by letting x1 = X η dt/ηe(x) and bounding the gap between trajectory of X η bsc+dt/ηe(x) and x η(s, x1) using Lemma H.5 and H.8, one can show that ‖Xηn(x)‖ < 3 for all n < T η 1 (δ) (since in the case of
j = 2, we have T η1 (δ) < 2dt/ηe). Otherwise, we have j ≥ 3 and the same arguments above can be applied to show that
‖Xηk (x)‖ < 3 ∀k ≤ 2dt/ηe, ∥∥∥Xηdt/ηe(x)∥∥∥ < 2 , , ∥∥∥Xη2dt/ηe(x)∥∥∥ < 2 .
In summary, by proceeding inductively, we establish that for all η ∈ (0, 4M exp(2tM) ),
sup x∈B(0,2 )
P ( ∃n < T η1 (δ) s.t. ‖Xηn(x)‖ > 3 ) ≤ P(Ã ∪ {T η1 (δ) > 1/ηβ})
and this concludes the proof.
We introduce a few concepts that will be crucial to the analysis below. Recall the definition of ODE with jumps x̃ (note that we will drop the notational dependency on learning rate η when we choose η = 1). Recall the definitions of i∗, k∗ (see Assumption G.7 and G.8 and the remarks underneath). We define the following mapping h from r = (r1, · · · , rk∗),θ = (θ1, · · · , θk∗) ∈ (Sd−1)k ∗ , t = (t1, · · · , tk∗) ∈ {0} × Rk ∗ + such that
h(r,θ, t) = x̃( k∗∑ i=1 ti,0; t,w) (H.38)
where w = (wi)k ∗
i=1 with wi = riθi. From the continuity of the ODE flow, we see that h is a continuous mapping. Also, we introduce the concept of type for configuration i∗. Specifically, define
j(i1, · · · , im) =∆ { (j1, · · · , jk∗) ∈ {0, 1, 2, · · · ,m}k ∗ : #{n : jn = k} = ik ∀k ∈ [m] } (H.39)
and for any j ∈ j(i∗), we say that w = (w1, · · · , wk∗) ∈ A(i∗) has type j if
wi ∈ Eji ∀i ∈ [k∗].
In other words, based on the direction of each jump inw ∈ A(i∗) we group them into different types. Note that |j(i∗)| <∞. Now for any type j ∈ j(i∗), define a (Borel) measure on Rk∗+ × (Sd−1)k
∗ × Rk ∗−1
+ as
µj = ∆ ( k∗∏ i=1 ναji )× ( k∗∏ i=1 Sji)× Leb k∗−1 (H.40)
where, for any α > 0, the measure να is the Borel measure on (0,∞) with να(x,∞) = 1/xα. We use the concepts above to characterize behavior of large noises in (Zn)n≥1.
Definition H.1. For any η, δ > 0, j ∈ j(i∗) and any Borel set A ⊆ Rd, we say noise Zn is of type-(A, δ, η, j) iff
• η ‖Zn‖ > δ;
• For t̃1 = n and t̃i = ∆ min{k > t̃i−1 : η ‖Zk‖ > δ} for all i = 2, 3, · · · , k∗, it holds that
t̃i − t̃i−1 < 2t̄/η for all i = 2, 3, · · · , k∗;
• For ti = ∆ t̃i − t̃i−1 (with t1 = 0), wi = ηZt̃i , and
r =∆ (‖w1‖ , ‖w2‖ , · · · , ‖wk∗‖), θ = (w1/ ‖w1‖ , · · · , wk∗/ ‖wk∗‖), t = (t1, · · · , tk∗),
it holds that h(r,θ, t) ∈ A.
More generally, we say noise Zn is of type-(A, δ, η) iff there exists j ∈ j(i∗) such that Zn is of type-(A, δ, η, j).
Due to the iid nature of (Zj)j≥1, let us consider an iid sequence (Vj)j≥0 where the sequence has the same law of Z1. Note that for any fixed n ≥ 1, the probability that Zn is of type-(A, δ, η, j) is equal to the probability that V0 is of type-(A, δ, η, j). More specifically, let H(x) = ∑m j=0Hj(x) = P(‖Z1‖ > x). Due to Assumption G.6, H ∈ RV−α1 . Besides, P(η ‖V0‖ > δ) = H(δ/η). Now we focus on conditional probability admitting the following form:
p(A, δ, η, j) = P ( V0 is of type-(A, δ, η, j) ∣∣∣ η ‖V0‖ > δ). Lemma H.13. For any δ > 0, any j ∈ j(i∗), and any Borel set A ⊆ Rd such that h(r1, · · · , rk∗ , θ1, · · · , θk∗ , t2, · · · , tk∗) ∈ A =⇒ ti < 2t̄ ∀i = 2, · · · , k∗ and ri > δ ∀i ∈ [k∗],
(H.41)
it holds that
µj ( h−1(A◦) ) ≤ lim inf
η
p(A, δ, η, j)
δα1 · ∏m k=1
( Hk(1/η) )i∗ k
H(1/η)ηk∗−1
≤ lim sup η
p(A, δ, η, j)
δα1 · ∏m k=1
( Hk(1/η) )i∗ k
H(1/η)ηk∗−1
≤ µj ( h−1(Ā) ) .
Proof. Let us start by fixing some notations. Let T1 = 0, and define stopping times Tj = min{n > Tj−1 : η ‖Vn‖ > δ} and inter-arrival times T ′j = Tj − Tj−1 for any j ≥ 2 (with T ′1 = 0), and large jump Wj = VTj for any j ≥ 0. Note that: first, the pair (T ′i ,Wi) is independent of (T ′j ,Wj) for any i 6= j; besides, Wj and T ′j are independent for all j ≥ 1.
Define the following sequence (of random elements) w = (w1, · · · , wl∗) and t = (t1, · · · , tl∗) by wi = ηWi, ti = ηT ′ i .
and r = (Tr(wi))k ∗ i=1,θ = (Tθ(wi))k ∗
i=1 where operators Tr(x) = ‖x‖ ,Tθ(x) = x/ ‖x‖ constitute the polar coordinate transform. By definition of type-(A, δ, η, j), we know that
• T ′i ≤ 2t̄/η for any i = 2, · · · , k∗;
• η ‖Wi‖ > δ for any i = 1, 2, · · · , k∗;
• wi ∈ Eji (i.e. θi ∈ Fi) for all i ∈ [k∗];
• h(r,θ, t) ∈ A.

heading: Therefore, (let Ri = Tr(Wi),Θi = Tθ(Wi))
text: p(A, δ, η, j) = ( P(T ′2 ≤ 2t̄/η) )k∗−1 · ∫ 1 { (r,θ, t) ∈ h−1(A) } · ( k∗∏ i=1 P(Wi ∈ Ei) · P(ηRi = dri, Θi = dθi) ) · ( k∗∏ i=2 P(ηT ′i = dti | ηT ′i ≤ 2t̄) )
= ( P(T ′2 ≤ 2t̄/η) )k∗−1 · [ k∗∏ i=1 P(Wi ∈ Ei) ] ·Qη,δ,j ( h−1(A) ) (H.42)
where Qη,δ,j is the probability measure on Rk ∗ + × (Sd−1)k ∗ × Rk ∗−1 + induced by a sequence of random elements (ηR1, · · · , ηRk∗ ,Θ1, · · · ,Θk∗ , ηT ↑2 (η), · · · , ηT ↑ k∗(η)) such that
• For any i = 1, · · · , k∗, we haveRi = Tr(W ↑i (η)),Θi = Tθ(W ↑ i (η)) where the distribution of W ↑i (η) follows the law P ( V0 ∈ · ∣∣∣ η ‖V0‖ > δ, V0 ∈ Eji); Besides, (W ↑i (η))k∗i=1 is an independent sequence;
• For any i = 2, · · · , l∗, the distribution of T ↑i (η) follows from P ( ηT δ1 (η) ∈ · ∣∣∣ ηT δ1 (η) ≤ 2t̄) (for the definition of stopping times T δk (η), see eq. (H.26)); Besides, (T ↑ i (η)) k∗
i=1 is an independent sequence, and it is independent of (W ↑i (η)) k∗ i=1;
• Qη,δ,j(·) = P ( (ηR1, · · · , ηRk∗ ,Θ1, · · · ,Θk∗ , ηT ↑2 (η), · · · , ηT ↑ k∗(η)) ∈ · ) .
As for weak convergence of (ηR1, · · · , ηRk∗ ,Θ1, · · · ,Θk∗ , ηT ↑2 (η), · · · , ηT ↑ k∗(η)), observe that
• For any i ∈ [k∗], from Assumption G.6 one can see the regularly varying nature of distribution of V0 on the cone Eji (hence for W ↑ i (η) as well), thus yielding that
(ηRi,Θi) ⇒ (R∗i ,Θ∗i ) as η ↓ 0 where R∗i and Θ∗i are independent, the law of Θ∗i is Sji , and the law of R ∗ i is the Pareto distribution with
P(R∗i > x) = δαji
(x ∨ δ)αji ;
• For any x ∈ [0, 2t̄], since limη↓0bx/ηcH(δ/η) = 0, it is easy to show that
lim η↓0
1− (1−H(δ/η))bx/ηc
bx/ηcH(δ/η) = 1;
therefore, we have (for any x ∈ (0, 2t̄])
P(ηT δ1 (η) ≤ x | ηT δ1 (η) ≤ 2t̄) = 1− (1−H(δ/η))bx/ηc 1− (1−H(δ/η))b2t̄/ηc → x 2t̄
as η ↓ 0, which implies that T ↑i (η) converges weakly to a uniform RV on [0, 2t̄].
Together with the assumption on set A in eq. (H.41), we can now see that, if we denote the weak limit of measure Qη,δ,j as µδ,j , then (the measure µj is defined in eq. (H.40))
µδ,j ( h−1(A) ) = ∏k∗ i=1 δ αji (2t̄)k∗−1 µj ( h−1(A) ) .
From the continuity of mapping h, one can see that h−1(Ā) is a closed set and h−1(A◦) is an open set. Using Portmanteau theorem, we now have∏k∗
i=1 δ αji (2t̄)k∗−1 µj ( h−1(A◦) ) ≤ lim inf η Qη,δ,j ( h−1(A) ) ≤ lim sup η Qη,δ,j ( h−1(A) ) (H.43)
≤ ∏k∗ i=1 δ αji (2t̄)k∗−1 µj ( h−1(Ā) ) . (H.44)
Moving on, we analyze the limit of the other terms in eq. (H.42). For any i ∈ [k∗], note that
P(Wi ∈ Ei) = Hji(δ/η)
H(δ/η) .
Now due to the regularly varying nature of Hj and H ,
lim η
∏k∗ i=1 P(Wi ∈ Ei)∏k∗
i=1Hji(1/η)/H(1/η) =
δk ∗α1∏k∗
i=1 δ αji
. (H.45)
On the other hand, from Lemma E.4 and the regularly varying nature of function H , for any fixed κ > 1, we have
lim sup η↓0 (δα1 2t̄ · P(T ′ 2 ≤ 2t̄/η) H(1/η)/η )k∗−1 ≤ κk ∗−1 lim sup η↓0 (δα1 2t̄ · 2t̄H(δ/η)/η H(1/η)/η )k∗−1 = κk ∗−1,
lim inf η↓0 (δα1 2t̄ · P(T ′ 2 ≤ 2t̄/η) H(1/η)/η )k∗−1 ≥ (1/κ)k ∗−1 lim inf η↓0 (δα1 2t̄ · 2t̄H(δ/η)/η H(1/η)/η )k∗−1 = (1/κ)k ∗−1.
Due to arbitrariness of κ > 1, we yield that
lim η
( P(T ′1 ≤ 2t̄/η) )k∗−1( H(1/η)
η · 2t̄ δα1 )k∗−1 = 1. (H.46) Collecting all the limits in eq. (H.44)-eq. (H.46) and plugging them into eq. (H.42), we now have established that
µj ( h−1(A◦) ) ≤ lim inf
η
p(A, δ, η, j) δα1 · ∏k∗ i=1 Hji (1/η)
H(1/η)ηk∗−1
≤ lim sup η
p(A, δ, η, j) δα1 · ∏k∗ i=1 Hji (1/η)
H(1/η)ηk∗−1
≤ µj ( h−1(Ā) ) .
To conclude the proof, recall that for any type j ∈ j(i∗), we have #{k = 1, 2, · · · , k∗ : jk = j} = i∗j (i.e. for any j ∈ [m], the number of elements in (j1, · · · , jk∗) that are equal to j is exactly i∗j ), thus ∏k∗ i=1Hji(1/η) = ∏m k=1 ( Hk(1/η)
)i∗k . More generally, for
p(A, δ, η) = P ( V0 is of type-(A, δ, η) ∣∣∣ η ‖V0‖ > δ), (H.47) we know that p(A, δ, η) = ∑ j∈j(i∗) p(A, δ, η, j). Also, define measure µ as
µ = ∑
j∈j(i∗)
µj . (H.48)
The next result follows immediately from Lemma H.13 and the fact that |j(i∗)| <∞. Recall that JG is defined in eq. (G.9).
Corollary H.14. There exists a function λ̃(η) ∈ RV1+JG−α1(η) such that, given any δ > 0 and any Borel set A ⊆ Rd with
h(r1, · · · , rk∗ , θ1, · · · , θk∗ , t2, · · · , tk∗) ∈ A =⇒ ti < 2t̄ ∀i = 2, · · · , k∗ and ri > δ ∀i ∈ [k∗], (H.49)
it holds that
µ ( h−1(A◦) ) ≤ lim inf
η
p(A, δ, η) δα1 · λ̃(η) ≤ lim sup η p(A, δ, η) δα1 · λ̃(η) ≤ µ
( h−1(Ā) ) .
In particular, the regularly varying function λ̃(·) admits the form
λ̃(η) =
∏m j=1 ( Hj(1/η) )i∗j H(1/η)ηk∗−1
(H.50)
Consider the following stopping times
σ(η) = min{n ≥ 0 : Xηn /∈ G}; R( , δ, η) = min{n ≥ T η1 (δ) : ‖Xηn‖ ≤ 3 }.
Here σ indicate the first time that the iterates exit domain G, while R denotes the time the SGD iterates return to a small neighborhood of the attractor 0 after the first large jump. In the next few results, we study the probability of the different scenarios regarding the first exit time σ(η) and first return time R( , δ, η). To this end, let
Kη(δ) =∆ max{k ≥ 0 : T ηk (δ) ≤ σ(η) ∧R( , δ, η)}
be the count of large jumps before the first exit or first return. Furthermore, we introduce the following concept as the accumulated cost of large jumps before the first exit or return. Let J η,δ0 = 0 and
J η,δk = ∆ J η,δk−1 + J(W η k (δ)) ∀k ≤ K η(δ)
where the cost function J(·) is defined in eq. (G.6). In other words, J η,δk indicates the total cost of the first k large jumps, if there are at least k large jumps, before the first exit or first return. Similarly, we can also define a step-wise accumulated cost as
J ↓η,δ(n) = max{J η,δ k : T η k (δ) ≤ n} ∀n ≤ σ(η) ∧R( , δ, η).
In other words, J ↓η,δ(n) evaluates the total cost of large jumps up until step n. As a preparation for our analyses below, we first discuss the following technical tools. For any > 0, let t̂( ) = c1 + c1 log(1/ ) where c1 is the constant in Lemma H.4. Besides, define
̃( ) =∆ 4 exp(2t̂M) ∧ ρ̄(t̂( )) ∧ ρ̃(t̂( ))
(H.51)
where functions ρ̄(·), ρ̃(·) are defined in Corollary H.6 and H.9 respectively. Furthermore, define an event A× = A×1 ( , δ, η) ∪A × 2 ( , δ, η) where
A×1 ( , δ, η) = ∆ { ∃n < T η1 (δ) s.t. ‖Xηn(x)‖ > 3 } , (H.52)
A×2 ( , δ, η) = ∆
{ ∃j = 2, · · · , l∗ s.t.
max k=1,2,··· ,(Tηj (δ)−T η j−1(δ)−1)∧(2t̂( )/η)
η ∥∥∥ZTηj−1(δ)+1 + · · ·+ ZTηj−1(δ)+j∥∥∥ > ̃( )}
(H.53)
where the positive integer l∗ is defined in eq. (H.1). By definition of l∗, we must have Kη(δ) < l∗. Lastly, define events (for any K ∈ [l∗])
B×1 (K) = ∆ (A×)c ∩ {J η,δK < JG , K ≤ K η(δ)}
∩ {∀j ∈ [K], [T ηj+1(δ) ∧ σ(η) ∧R( , δ, η)]− T η j (δ) ≤
2t̂( )
η },
B×2 (K) = ∆ (A×)c ∩ {J η,δK ≤ JG , K ≤ K η(δ)}
∩ {∃j ∈ [K] s.t. [T ηj+1(δ) ∧ σ(η) ∧R( , δ, η)]− T η j (δ) >
2t̂( )
η }.
Recall that the minimum cost for exit JG is defined in eq. (G.9). Note that, in the definition of B×1 (K) and B×2 (K) above, T η j+1(δ) ∧ σ(η) ∧R( , δ, η) < T η j+1(δ) only if j = K = K η(δ). Lemma H.15. For any K ∈ [l∗], any ∈ (0, ̄/3) and any η > 0 sufficiently small, the following claim holds on event B×1 (K):
sup s∈[0,TηK+1(δ)∧σ(η)∧R( ,δ,η)−T η 1 (δ)] ∥∥∥Xηbsc+Tη1 (δ)(x)− x̃η(s,X(1)(x); T ,W )∥∥∥ < 2 where T = ( 0, T η2 (δ) − T η 1 (δ), T η 3 (δ) − T η 2 (δ), · · · , T η K+1(δ) − T η K(δ) ) , W =(
W η1 (δ),W η 2 (δ), · · · ,W η K+1(δ)
) , and X(1)(x) = Xη
Tη1 (δ)−1 (x).
Proof. We focus on the distances between the following three objects: Xηn(x),
Ỹ ηn (x) =
{ Xηn(x) if n < T η 1 (δ);
Ỹ ηn−1(x) + ϕb ( − η∇f ( Ỹ ηn−1(x) ) + ∑ j≥1 1{n = T η j (δ)}ηZn ) otherwise;
and
z(s) = x̃η ( s,X(1)(x); T ,W ) ∀s ≥ 0.
First of all, for X(1) = Xη Tη1 (δ)−1 (x) = Ỹ η Tη1 (δ)−1 (x), by definition of event (A×)c we have∥∥X(1)∥∥ < 3 < ̄. Then due to eq. (H.20) and the definition of event B×1 (K) (in particular, the fact that J η,δK < JG), we know that
d(z(s),Gc) > 100l∗̄ ∀s ∈ [ 0, T ηK+1(δ) ∧ σ(η) ∧R( , δ, η)− T η 1 (δ) ) .
In light of the definition of B×1 (K) (i.e. the upper bound on T η j+1(δ)−T η j (δ)), by applying Corollary H.6, we can show that
sup s∈[0,TηK+1(δ)∧σ(η)∧R( ,δ,η)−T η 1 (δ)] ∥∥∥z(s)− Ỹ ηbsc+Tη1 (δ)(x)∥∥∥ < . In particular, by applying Corollary H.6 inductively for all n ≤ T ηK+1(δ)∧σ(η)∧R( , δ, η)−T η 1 (δ), we know that the line segment between z(s) and Ỹ ηbsc+Tη1 (δ)(x) is in G for all s < T η K+1(δ)∧σ(η)∧ R( , δ, η)−T η1 (δ) so Corollary H.6 can be further applied to n+ 1 and establish the inequality above. Similarly, due to our choice of ̂( ) in eq. (H.51) and the upper bound on T ηj+1(δ)−T η j (δ) in definition of event B×1 (K), by applying Corollary H.9 inductively, we know that for all η sufficiently small,
sup j=0,1,2,··· ,TηK+1(δ)∧σ(η)∧R( ,δ,η)−T η 1 (δ) ∥∥∥Ỹ ηbsc+Tη1 (δ)(x)−Xηbsc+Tη1 (δ)(x)∥∥∥ < holds on event B×1 (K) and this concludes the proof.
Lemma H.16. For any K ∈ [l∗], any > 0 and any η > 0 sufficiently small,
B×2 (K) = ∅.
Proof. The definition of event B×2 (K) ensures that we can define some j ∗ as the smallest integer in [Kη(δ)] such that for T ∗ =∆ T ηj∗(δ) + d2t̂( )/ηe, we have
T ηj∗(δ) < T ∗ < T ηj∗+1(δ) ∧ σ(η) ∧R( , δ, η). (H.54)
Analogous to the proof of the previous lemma, we focus on the pair-wise distances between the following three objects: Xηn(x),
Ỹ ηn (x) =
{ Xηn(x) if n < T η 1 (δ);
Ỹ ηn−1(x) + ϕb ( − η∇f ( Ỹ ηn−1(x) ) + ∑ j≥1 1{n = T η j (δ)}ηZn ) otherwise;
and
z(s) = x̃η ( s,X(1); T ,W ) ∀s ≥ 0.
with T = ( 0, T η2 (δ) − T η 1 (δ), T η 3 (δ) − T η 2 (δ), · · · , T η j∗+1(δ) − T η j∗(δ) ) , W =(
W η1 (δ),W η 2 (δ), · · · ,W η j∗+1(δ)
) , and X(1) = Xη
Tη1 (δ) . Again, using Corollary H.6 and
H.9, one can see that for all η > 0 sufficiently small, we must have
sup s∈[0,T∗−Tη1 (δ)] ∥∥∥Xηbsc+Tη1 (δ)(x)− z(s)∥∥∥ < 2 on event B×2 (K). However, eq. (H.20) and
∥∥X(1)∥∥ < 3 < ̄ implies that sup
s∈[0,Tη j∗ (δ)−T η 1 (δ)]
d ( z(s), Gc ) > 100l∗̄.
In the meantime, Lemma H.4 and our choice of ̃( ) in eq. (H.51) implies that
‖z(T ∗ − T η1 (δ))‖ < and ‖X η T∗(x)‖ < 3 ,
thus dictating that R( , δ, η) < T ∗ and contradicting eq. (H.54). This concludes the proof.
Now we are ready to apply the tools above and analyze some atypical scenarios regrading the first exit and first return time. In the next result we show that, when starting from the local minimum, it is very unlikely to escape if the total cost of large jumps is less than JG . Lemma H.17. For any fixed ∈ (0, ̄/3), N > 0, the following claim holds for any sufficiently small δ > 0:
sup x: ‖x‖≤2
Px ( σ(η) < R( , δ, η), J ↓η,δ(σ(η)) < JG ) = o(ηN ) as η ↓ 0.
Proof. Since the constant > 0 is fixed, one can see that t̂ = c1 + c1 log(1/ ) is also fixed where c1 is the constant in Lemma H.4. Besides, we can fix some ̃ > 0 such that
̃ < 4 exp(2t̂M) , ρ̄(t̂)̃ < , ρ̃(t̂)̃ <
where functions ρ̄(·), ρ̃(·) are defined in Corollary H.6 and H.9 respectively.
Let us define an event A× = A×1 ∪A × 2 where
A×1 = ∆ { ∃n < T η1 (δ) s.t. ‖Xηn(x)‖ > 3 } ,
A×2 = ∆
{ ∃j = 2, · · · , l∗ s.t.
max k=1,2,··· ,(Tηj (δ)−T η j−1(δ)−1)∧(2t̂/η)
η ∥∥∥ZTηj−1(δ)+1 + · · ·+ ZTηj−1(δ)+j∥∥∥ > ̃}
where the positive integer l∗ is defined in eq. (H.1). By definition of l∗, we must have Kη(δ) < l∗ on event {J ↓η,δ(σ(η)) < JG}.
Now let us analyze the dynamics of Xηn on event (A ×)c ∩ {J ↓η,δ(σ(η)) < JG}. In particular, we decompose it into two events
B1 = ∆(A×)c ∩ {J ↓η,δ(σ(η)) < JG}
∩ {∀j = 1, 2, · · · ,Kη(δ), [T ηj+1(δ) ∧ σ(η) ∧R( , δ, η)]− T η j (δ) ≤ 2t̂/η},
B2 = ∆(A×)c ∩ {J ↓η,δ(σ(η)) < JG}
∩ {∃j = 1, 2, · · · ,Kη(δ) s.t. [T ηj+1(δ) ∧ σ(η) ∧R( , δ, η)]− T η j (δ) > 2t̂/η}.
Using Lemma H.15 and the fact that B1 = (⋃l∗ K=1B × 1 (K) ∩ {Kη(δ) = K} ) ∩ {J η,δKη(δ) < JG}, one can see that for any η > 0 sufficiently small, we must have
d(Xησ(η)∧R( ,δ,η)(x),G c) > 100l∗̄− 2 > 100l∗̄− ̄ > 0.
on event B1. In other words, on event B1 we must have R( , δ, η) < σ(η). On the other hand, Lemma H.16 the fact that B2 ⊆ ⋃l∗ K=1B × 2 (K) ∩ {Kη(δ) = K} implies that B2 = ∅ whenever η > 0 is sufficiently small. In summary, we have established that
sup x: ‖x‖≤2
Px ( σ(η) < R( , η), J ↓η,δ(σ(η)) < JG ) ≤ sup x: ‖x‖≤2 P(A×).
Lastly, from Lemma H.10 and H.12, one can see that for any δ > 0 that is sufficiently small,
sup x: ‖x‖≤2
P(A×) = o(ηN ),
and this concludes the proof.
Using an almost identical approach, we can establish the next two results and conclude that it is also rather unlikely to have scenarios where
• The accumulated cost of large jumps exceeds JG before the first return or exit, • Or the first return occurs before the first exit,
yet it takes rather long for the said event to occur. Lemma H.18. Given ∈ (0, ̄/3), N > 0, the following claim holds for any sufficiently small δ > 0:
sup x∈B(0,2 )
Px ( ∃K ∈ N s.t. J η,δK ≥ JG and T η j (δ)− T η j−1(δ) > 2t̂( )/η for some j = 2, 3, · · · ,K ) = o(ηN )
as η ↓ 0 where t̂( ) = c1 + c1 log(1/ ).
Proof. Given the fixed > 0, define t̂ = c1 + c1 log(1/ ). Also, fix ̃ > 0 as the largest possible value such that
̃ ≤ 4 exp(2t̂M) , ρ̄(t̂)̃ ≤ , ρ̃(t̂)̃ ≤
where functions ρ̄(·), ρ̃(·) are defined in Corollary H.6 and H.9 respectively.
Let us define an event A× = A×1 ∪A × 2 where
A×1 = ∆ { ∃n < T η1 (δ) s.t. ‖Xηn(x)‖ > 3 } ,
A×2 = ∆
{ ∃j = 2, · · · , l∗ s.t.
max k=1,2,··· ,(Tηj (δ)−T η j−1(δ)−1)∧(2t̂/η)
η ∥∥∥ZTηj−1(δ)+1 + · · ·+ ZTηj−1(δ)+j∥∥∥ > ̃}
where the positive integer l∗ is defined in eq. (H.1). Furthermore, define event
B =∆ (A×)c ∩ { ∃K ∈ N s.t. J η,δK ≥ JG and T η j (δ)− T η j−1(δ) > 2t̂
η for some j = 2, 3, · · · ,K
} .
Note that on event B, the index
K∗ = max{k ≥ 1 : J η,δj < JG ∀j = 1, 2, · · · , k − 1}
is well defined with 1 ≤ K∗ < l∗ (due to the definition of l∗). As a consequence, we must have T ηK∗(δ) ≤ σ(η) ∧R( , δ, η).
Now based on the exact value of K∗, we can decompose the event B into B = ∪l∗K=0B(K) where
B(K) =∆ (A×)c ∩ {K∗ − 1 = K} ∩ {J η,δK < JG} ∩ {T η j+1(δ)− T η j (δ) >
2t̂ η for some j ∈ [K]}.
By applying Lemma H.16, we know that for all η > 0 sufficiently small, B(K) = ∅. Therefore, for all sufficiently small η,
sup x∈B(0,2 )
Px ( ∃K ∈ N s.t. J η,δK ≥ JG and T η j (δ)− T η j−1(δ) > 2t̂( )/η for some j = 2, 3, · · · ,K ) ≤ sup x∈B(0,2 ) Px(A×).
Lastly, from Lemma H.10 and H.12, one can see that for any δ > 0 that is sufficiently small,
sup x: ‖x‖≤2
P(A×) = o(ηN ),
and this concludes the proof.
Lemma H.19. Given ∈ (0, ̄/3), N > 0, the following claim holds for any sufficiently small δ > 0:
sup x∈B(0,2 )
Px ( R( , δ, η) < σ(η), Jη,δKη(δ) < JG ,
T ηj+1(δ) ∧R( , δ, η)− T η j (δ) >
2t̂( )
η for some j ∈ [Kη(δ)]
) = o(ηN ) as η ↓ 0
where t̂( ) = c1 + c1 log(1/ ).
Proof. Given the fixed > 0, define t̂ = c1 + c1 log(1/ ). Also, fix ̃ > 0 as the largest possible value such that
̃ ≤ 4 exp(2t̂M) , ρ̄(t̂)̃ ≤ , ρ̃(t̂)̃ ≤
where functions ρ̄(·), ρ̃(·) are defined in Corollary H.6 and H.9 respectively.
Let us define an event A× = A×1 ∪A × 2 where
A×1 = ∆ { ∃n < T η1 (δ) s.t. ‖Xηn(x)‖ > 3 } ,
A×2 = ∆
{ ∃j = 2, · · · , l∗ s.t.
max k=1,2,··· ,(Tηj (δ)−T η j−1(δ)−1)∧(2t̂/η)
η ∥∥∥ZTηj−1(δ)+1 + · · ·+ ZTηj−1(δ)+j∥∥∥ > ̃}
where the positive integer l∗ is defined in eq. (H.1). By definition of l∗, we must have Kη(δ) < l∗ on event {Jη,δKη(δ) < JG}. Furthermore, define event
B =∆(A×)c ∩ { R( , δ, η) < σ(η), Jη,δKη(δ) < JG ,
T ηj+1(δ) ∧R( , δ, η)− T η j (δ) >
2t̂( )
η for some j ∈ [Kη(δ)]
} .
Now observe that B ⊆ ⋃l∗ K=1B(K) where
B(K) =∆(A×)c ∩ {Kη(δ) = K} ∩ {J η,δK < JG}
∩ {∃j ∈ [Kη(δ)] s.t. [T ηj+1(δ) ∧ σ(η) ∧R( , δ, η)]− T η j (δ) >
2t̂ η }.
By applying Lemma H.16, we know that for all η > 0 sufficiently small, B(K) = ∅. Therefore, for all sufficiently small η,
sup x∈B(0,2 )
Px ( R( , δ, η) < σ(η), Jη,δKη(δ) < JG ,
T ηj+1(δ) ∧R( , δ, η)− T η j (δ) >
2t̂( )
η for some j ∈ [Kη(δ)] ) ≤ sup x∈B(0,2 ) Px(A×).
Lastly, from Lemma H.10 and H.12, one can see that for any δ > 0 that is sufficiently small,
sup x: ‖x‖≤2
P(A×) = o(ηN ),
and this concludes the proof.
Recall that in Lemma H.17, we have shown that it is rather unlikely to have the first exit with accumulated cost of large jumps less than JG . In the next result, we show that even with large jumps of accumulated cost JG , if some jumps are still not large enough, or the inter-arrival times are too long, then it is still very unlikely for the SGD iterates to even get close the the boundary. Define
Kη,δJG = ∆ min{k ∈ [Kη(δ)] : J η,δk ≥ JG}
and let Kη,δJG =∞ if J η,δ Kη(δ) < JG . Also, if K η,δ JG <∞, let
T≥JG (η, δ) =∆ T η Kη,δJG (δ)
In other words, T≥JG (η, δ) is the first time that, prior to first exit or return, the accumulated cost of large jumps has reached JG . Regarding the o(η1+JG−α1+∆) term in the next Lemma, we note that the function λ̃ defined in eq. (H.50) is regularly varying (as η ↓ 0) with index 1 +JG −α1. Therefore, η1+JG−α1+∆ = o(λ̃(η)) as η approaches 0.
Lemma H.20. There exists some ∆ > 0 such that the following claim holds for all ∈ (0, ̄/3) and all δ > 0 that is sufficiently small:
sup x: ‖x‖≤2
Px ( B×2,(I)( , δ, η) ∪B × 2,(II)( , δ, η) ) = o(η1+JG−α1+∆)
as η ↓ 0 where t̂( ) = c1 + c1 log(1/ ) and
B×2,(I)( , δ, η) = {K η,δ JG <∞} ∩ {J η,δ Kη,δJG > JG}, B×2,(II)( , δ, η) = {K η,δ JG
<∞} ∩ {J η,δ Kη,δJG = JG}
∩ { ∃j = 2, 3, · · · ,Kη,δJG s.t.T η j (δ)− T η j−1(δ) > 2t̄/η or ∃j = 1, 2, · · · ,K η,δ JG s.t. η ∥∥W ηj (δ)∥∥ ≤ δ̄}
∩ {
min j=0,1,··· ,T≥JG (η,δ)
d ( Xηj ,G c ) ≤ ̄ } .
Proof. From Assumption G.8, we know the existence of some ∆ > 0 satisfying the following condition: For any k ∈ N and any j ∈ {1, · · · ,m}k such that ∑k−1 i=1 (αji−1) < JG < ∑k i=1(αji−1) (note that there are only finitely many possible choices for such j), we have
JG + ∆ < k∑ i=1 (αji − 1) (H.55)
Fix such ∆ > 0. Meanwhile, given the fixed > 0, we can define t̂ = c1 + c1 log(1/ ) and choose ̃ > 0 as the largest possible value such that
̃ ≤ 4 exp(2t̂M) , ρ̄(t̂)̃ ≤ , ρ̃(t̂)̃ ≤
where functions ρ̄(·), ρ̃(·) are defined in Corollary H.6 and H.9 respectively. We stress that t̄ is a fixed constant while t̂ depends on the value of , and for any sufficiently small we will have t̂ > t̄.
Let us define an event A× = A×1 ∪A × 2 where
A×1 = ∆ { ∃n < T η1 (δ) s.t. ‖Xηn(x)‖ > 3 } ,
A×2 = ∆
{ ∃j = 2, · · · , l∗ s.t.
max k=1,2,··· ,(Tηj (δ)−T η j−1(δ)−1)∧(2t̂/η)
η ∥∥∥ZTηj−1(δ)+1 + · · ·+ ZTηj−1(δ)+j∥∥∥ > ̃}
where the positive integer l∗ is defined in eq. (H.1). By definition of l∗, we must have Kη,δJG < l ∗ on event {Kη,δJG <∞}.
First, we analyze the following event B =∆ (A×)c ∩ {Kη,δJG <∞} ∩ { T ηj (δ)− T η j−1(δ) > 2t̂ η for some j = 2, 3, · · · ,Kη,δJG } .
In particular, note that B = ⋃l∗−1 K=0B(K) where
B(K) =∆(A×)c ∩ {Kη,δJG = K + 1} ∩ {J η,δ K < JG}
∩ { T ηj+1(δ)− T η j (δ) > 2t̂
η for some j = 1, 2, 3, · · · ,K
} .
When K = 0 < 1, the event B(0) = ∅ by definition (due to event { T ηj+1(δ) − T η j (δ) >
2t̂ η for some j = 1, 2, 3, · · · ,K
} ). For K = 1, · · · , l∗, Lemma H.16 implies that B(K) = ∅
for all η sufficiently small. In summary, B = ∅ for all η sufficiently small. Now we focus on the following two events
C =∆ (A×)c ∩ {Kη,δJG <∞} ∩ { T ηj (δ)− T η j−1(δ) ≤ 2t̂ η for all j = 2, 3, · · · ,Kη,δJG } ∩ {J η,δKη(δ) > JG},
D =∆ (A×)c ∩ {Kη,δJG <∞} ∩ { T ηj (δ)− T η j−1(δ) ≤ 2t̂ η for all j = 2, 3, · · · ,Kη,δJG } ∩ {J η,δKη(δ) = JG}.
On the one hand, event C can be decomposed as follows. Let
J↑ =∆ {j ∈ {1, 2, · · · ,m}k : k = |j|, k−1∑ i=1 (αji − 1) < JG < k∑ i=1 (αji − 1)}
be the set that contains all the types j such that the accumulated cost reach JG if and only if the last element is kept. One can see that there are only finitely many elements in J↑. Now we have C = ⋃ j∈J↑ C(j) where (see the definition in eq. (H.10))
C(j) =∆ (A×)c ∩ {Kη,δJG <∞} ∩ { T ηj (δ)− T η j−1(δ) ≤ 2t̂ η for all j = 2, 3, · · · ,Kη,δJG } ∩ {( W η1 (δ),W η 2 (δ), · · · ,W η
Kη,δJG
(δ) ) is of type-j } .
For any j ∈ J↑, observe that (let kj = |j|) sup ‖x‖≤2 Px(C(j))
≤P ( W ηi (δ) ∈ Eji ∀i ∈ [K η,δ JG ] and T ηi (δ)− T η i−1(δ) ≤ 2t̂ η for all i = 2, 3, · · · ,Kη,δJG ) = ( kj∏ i=1 Hji(δ/η) H(δ/η) ) · ( P(T η1 (δ) ≤ 2t̂/η)
)kj−1. The last equality is due to independence of W ηi (δ) and T η i (δ) − T η i−1(δ). The regularly varying natures of functions Hj and H imply that
lim η↓0
∏kj i=1 Hji (δ/η)
H(δ/η)∏kj i=1 Hji (1/η) H(1/η) = δkjα1 δ ∑kj i=1 αji .
Meanwhile, from Lemma E.4 and the regularly varying nature of function Hj and H (in particular, repeating the same calculations that leads to eq. (H.46)), we have
lim η↓0
( P(T η1 (δ) ≤ 2t̂/η) )kj−1(H(1/η) η · 2t̂ δα1
)kj−1 = 1. Therefore, we have yielded that
lim sup η↓0
sup‖x‖≤2 Px(C(j))
λ̃j(η) ≤ δ
kjα1
δ ∑kj i=1 αji · ( 2t̂ δα1 )kj−1 <∞ where
λ̃j(η) = ∆
∏kj i=1Hji(1/η)
ηkj−1H(1/η) = o(η1+JG−α1+∆)
due to eq. (H.55). Therefore, one can see that as η ↓ 0, sup ‖x‖≤2 Px(C) = o(η1+JG−α1+∆).
Furthermore, the discussion above have shown that for all η sufficiently small,
sup x: ‖x‖≤2
Px ( B×2,(I)( , δ, η) ) ≤ o(η1+JG−α1+∆) + sup
x: ‖x‖≤2 Px(A×).
From Lemma H.10 and H.12, one can see that for all N > 1 + JG − α1 + ∆ and all δ > 0 that is sufficiently small, supx: ‖x‖≤2 P(A×) = o(ηN ). Fix such δ > 0. In conclusion, we now have that supx: ‖x‖≤2 Px ( B×2,(I)( , δ, η) ) = o(η1+JG−α1+∆).
On the other hand, on event D, Assumption G.7 shows that (see the definition in eq. (H.39))
{Kη,δJG <∞} ∩ {J η,δ Kη(δ) = JG} = {Kη,δJG <∞} ∩ {( W η1 (δ),W η 2 (δ), · · · ,W η
Kη,δJG
(δ) ) is of type-j for some j ∈ j(i∗) } .
Therefore,
B×2,(II)( , δ, η) ∩D =(A×)c ∩ {Kη,δJG <∞} ∩ {( W η1 (δ),W η 2 (δ), · · · ,W η
Kη,δJG
(δ) ) is of type-j for some j ∈ j(i∗) }
∩ { ∀j = 2, 3, · · · ,Kη,δJG , T η j (δ)− T η j−1(δ) ≤ 2t̂/η } ∩ {
min j=0,1,··· ,T≥JG (η,δ)
d ( Xηj ,G c ) ≤ ̄ }
∩ { ∃j = 2, 3, · · · ,Kη,δJG s.t. T η j (δ)− T η j−1(δ) > 2t̄/η OR ∃j = 1, 2, · · · ,K η,δ JG s.t. η ∥∥W ηj (δ)∥∥ ≤ δ̄}.
In particular, using Lemma H.15, one can see that when η is sufficiently small, on event B×2,(II)( , δ, η) ∩D we have ‖X η k‖ < 3 for all k < T η 1 (δ) and
sup s∈[0,T≥JG (η,δ)−Tη1 (δ)] ∥∥∥Xηbsc − zη(s)∥∥∥ < 2 . Here
zη(s) = x̃η ( s,X(1); T ,W ) ∀s ≥ 0
with T = ( 0, T η2 (δ) − T η 1 (δ), T η 3 (δ) − T η 2 (δ), · · · , T η
Kη,δJG +1 (δ) − T η Kη,δJG (δ) ) , W =(
W η1 (δ),W η 2 (δ), · · · ,W η
Kη,δJG
(δ) ) , and X(1) = Xη
Tη1 (δ)−1 . Besides, eq. (H.17) dictates that, on
event B×2,(II)( , δ, η) ∩D, we must have infs≥0 d(zη(s),Gc) ≥ 100l∗̄, hence
min k∈[T≥JG (η,δ)]
d(Xηk ,G c) > 50l∗̄.
However, this clearly contradicts the definition of event B×2,(II)( , δ, η). In conclusion, we have established that (for all η sufficiently small) B×2,(II)( , δ, η) ∩D = ∅ and
sup x: ‖x‖≤2
Px ( B×2,(II)( , δ, η) ) ≤ sup x: ‖x‖≤2 Px(A×).
However, as per the argument above, our choice of sufficiently small δ > 0 ensures that supx: ‖x‖≤2 P(A×) = o(η1+JG−α1+∆) and concludes the proof.
Furthermore, starting from the local minimum 0, it is unlikely that the SGD iterates will be extremely close to ∂G when the accumulated cost of large jumps reaches JG . The next lemma provides an upper bound for the probability of the said scenario. For any set A ⊆ Rd, define its −enlargement as the following open set A =∆ {x : d(x,A) < }. Lemma H.21. For any ∈ ( 0, ̄/(3 + 3ρ∗) ) and any δ > 0 that is sufficiently small,
lim sup η↓0
supx: ‖x‖≤2 Px(B × 3 ( , δ, η))
λ̃(η) ≤ δα1Ψ( )
where B×3 ( , δ, η) = {K η,δ JG < ∞} ∩ {d(Xη T≥JG (η,δ) , ∂G) < } and Ψ( ) = µ ( h−1 ( (∂G)(3+3ρ∗) )) with h defined in eq. (H.38) and µ defined in eq. (H.48), and ρ∗ ∈ (0,∞) is
the constant provided in Corollary H.7.
Proof. Again, given the fixed > 0, we can define t̂ = c1 + c1 log(1/ ) and choose ̃ > 0 as the largest possible value such that
̃ ≤ 4 exp(2t̂M) , ρ̄(t̂)̃ ≤ , ρ̃(t̂)̃ ≤
where functions ρ̄(·), ρ̃(·) are defined in Corollary H.6 and H.9 respectively. We stress that t̄ is a fixed constant while t̂ depends on the value of , and for any sufficiently small we will have t̂ > t̄.
Let us define an event A× = A×1 ∪A × 2 where
A×1 = ∆ { ∃n < T η1 (δ) s.t. ‖Xηn(x)‖ > 3 } ,
A×2 = ∆
{ ∃j = 2, · · · , l∗ s.t.
max k=1,2,··· ,(Tηj (δ)−T η j−1(δ)−1)∧(2t̂/η)
η ∥∥∥ZTηj−1(δ)+1 + · · ·+ ZTηj−1(δ)+j∥∥∥ > ̃}
where the positive integer l∗ is defined in eq. (H.1). Consider the following decomposition of event B×3 ( , δ, η) ∩ (A×)c:
C(I) = ∆ B×3 ( , δ, η) ∩ (A×)c ∩ {J η,δ
Kη,δJG
> JG}.
C(II) = ∆ B×3 ( , δ, η) ∩ (A×)c ∩ {J η,δ
Kη,δJG
= JG} ∩ { ∃j = 2, 3, · · · ,Kη,δJG s.t.
T ηj (δ)− T η j−1(δ) > 2t̄/η or ∃j = 1, 2, · · · ,K η,δ JG s.t. η ∥∥W ηj (δ)∥∥ ≤ δ̄}
C(III) = ∆ ( B×3 ( , δ, η) ∩ (A×)c ) \ ( C(I) ∪ C(II) ) .
Using Lemma H.20, we know that for all δ > 0 sufficiently small,
lim sup η↓0
sup‖x‖≤2 Px ( C(I) ∪ C(II) ) λ̃(η) = 0.
As a result,
lim sup η↓0
sup‖x‖≤2 Px(B × 3 ( , δ, η))
λ̃(η) ≤ lim sup η↓0
sup‖x‖≤2 Px(A×)
λ̃(η) + lim sup η↓0
sup‖x‖≤2 Px(C(III))
λ̃(η) .
From Lemma H.10 and H.12, one can see that for all N > 1 + JG − α1 and all δ > 0 that is sufficiently small, we have supx: ‖x‖≤2 P(A×) = o(ηN ) = o(λ̃(η)).
Moving on, we focus on bounding the probability of event C(III). In particular, note that
C(III) = (A ×)c ∩ {Kη,δJG <∞} ∩ {J η,δ
Kη,δJG
= JG} ∩ {d(XηT≥JG (η,δ), ∂G) < }
∩ { ∀j = 2, 3, · · · ,Kη,δJG , T η j (δ)− T η j−1(δ) ≤ 2t̄/η and ∀j = 1, 2, · · · ,K η,δ JG , η ∥∥W ηj (δ)∥∥ ≥ δ̄}.
By applying Lemma H.15 on event C(III), one can see that when η is sufficiently small, on this event we have ‖Xηk‖ < 3 for all k < T η 1 (δ) and
sup s∈[0,T≥JG (η,δ)−Tη1 (δ)] ∥∥∥Xηbsc+Tη1 (δ)3 − zη(s)∥∥∥ < 2 . Here
zη(s) = x̃η ( s,X(1); T ,W ) ∀s ≥ 0
with T = ( 0, T η2 (δ) − T η 1 (δ), T η 3 (δ) − T η 2 (δ), · · · , T η
Kη,δJG (δ) − T η Kη,δJG−1 (δ) ) , W =(
W η1 (δ),W η 2 (δ), · · · ,W η
Kη,δJG
(δ) ) , and X(1) = Xη
Tη1 (δ)−1 . To better control the location of zη(s),
we also construct
zη0 (s) = x̃ η ( s,0; T ,W ) ∀s ≥ 0
where the only difference is that we substitute the initial value X(1) with 0. Note that on event C(III), we have ∥∥X(1)∥∥ < 3 . By applying Corollary H.7 (the condition about line segments contained in G is verified due to eq. (H.18)), we have the following bound on event C(III):
sup
s∈[0, ∑Kη,δJG i=1 Ti]
‖zη(s)− zη0 (s)‖ < 3ρ∗ .
Now recall Definition H.1. Combining all the bounds we have obtained so far, we see that, when η is sufficiently small, on event C(III) we have
d ( zη0 (Kη,δJG∑ i=1 Ti ) , ∂G ) < (2 + 3ρ∗)
andW is of type-j for some j ∈ j(i∗). It then follows immediately from Corollary H.14 that
lim sup η↓0
sup‖x‖≤2 Px(C(III))
λ̃(η) ≤ lim sup η
p ( (∂G)(2+3ρ∗) , δ, η ) λ̃(η) ≤ δα1µ ( h−1 ( (∂G)(2+3ρ∗) )) ≤ δα1µ ( h−1 ( (∂G)(3+3ρ ∗) ))
and this concludes the proof.
Lastly, we provide a lower bound for the probability of the most likely way of escape, i.e., due to k∗ large jumps with accumulated cost JG and relatively short inter-arrival time less (when compared to t̄/η). We stress that, in the next result, the value of constant c∗ > 0 would not vary with the choice of parameters , δ. Lemma H.22. There exists some c∗ > 0 such that the following claim holds for any ∈ (0, ̄/(4 + 3ρ∗)) and any sufficiently small δ > 0:
lim inf η↓0 inf‖x‖≤2 Px(A◦( , δ, η)) λ̃(η) ≥ c∗δα1
where the event is defined as A◦( , δ, η) =∆ {Kη,δJG <∞} ∩ { σ(η) = T≥JG (η, δ) } ∩ { T ηj (δ)− T η j−1(δ) ≤ 2t̄ η ∀j = 2, 3, · · · , k∗ } and ρ∗ ∈ (0,∞) is the constant provided in Corollary H.7.
Proof. First of all, Assumption G.7 implies that {Kη,δJG <∞} ∩ { σ(η) = T≥JG (η, δ) } = {Kη,δJG = k ∗} ∩ { σ(η) = T ηk∗(δ) } .
We also stress that Kη,δJG = k ∗ means that σ(η) ∧ R( , δ, η) ≥ T ηk∗(δ) (i.e. the arrival time of the k∗−th large jumps and that the accumulated cost of large jumps W η1 (δ), · · · ,W η k∗(δ) is exactly JG .
Due to eq. (H.19) and eq. (H.17), there exists some j ∈ j(i∗), some θ̃ = (θ̃1, · · · , θ̃k∗) with θ̃i ∈ Fi ∀i, some r̃ = (r̃1, · · · , r̃k∗) such that r̃i ≥ δ̄ ∀i, some t̃ = (t̃1, · · · , t̃k∗) such that t̃1 = 0 and t̃i ∈ (0, t̄) for all i = 2, · · · , k∗ such that
d ( h(r̃, θ̃, t̃), G ) > 100l∗̄
where h is the mapping defined in eq. (H.38). Let ỹ =∆ h(r̃, θ̃, t̃). Moreover, the continuity of mapping h implies the existence of some ∆ > 0 such that (r̃, θ̃, t̃) ∈ X ⊆ h−1 ( B(ỹ, ̄) ) where the open domain X is defined as
X =∆ {(r,θ, t) : t1 = 0 and ∥∥∥θi − θ̃i∥∥∥ < ∆, |ri − r̃i| < ∆, |ti − t̃i| < ∆ ∀i ∈ [k∗]}.
In particular, the parameter ∆ can be chosen small enough with
∆ < min i=2,··· ,k∗ t̃i, ∆ < min i=1,··· ,k∗
r̃i ∧ 1
so that for any (r,θ, t) ∈ X , we have ti > 0 for all i = 2, · · · , k∗ and ri > 0 for all i ∈ [k∗]. Fix such ∆ > 0. Meanwhile, from the definition of the measure µ(·) in eq. (H.40)eq. (H.48) and the fact that θ̃i ∈ Fji where the closed set Fji is the support of the probability measure Sji on the unit sphere Sd−1 (see Assumption G.6), one can see that µ(X ) > 0.
Let R = (Ri)k ∗ i=1,Θ = (Θi) k∗ i=1,T = (Ti) k∗ i=1 with Ri = η ‖W η i (δ)‖ , Θi = W η i (δ)/ ‖W η i (δ)‖ and Ti = η(T η i (δ)− T η i−1(δ)) for all i ≥ 2 and T1 = 0. Now recall Definition H.1 and consider the following event
B =∆ {(R,Θ,T ) ∈ X} = {ZTη1 (δ) is of type-(h(X ), η, δ)}.
Corollary H.14 then gives the bound
lim inf η↓0 P(B) λ̃(η) ≥ δα1µ(X ) > 0.
From now on, we fix some c∗ ∈ (0, µ(X )). Given the fixed > 0, we can define t̂ = c1 + c1 log(1/ ) and choose ̃ > 0 as the largest possible value such that
̃ ≤ 4 exp(2t̂M) , ρ̄(t̂)̃ ≤ , ρ̃(t̂)̃ ≤
where functions ρ̄(·), ρ̃(·) are defined in Corollary H.6 and H.9 respectively. We stress that t̄ is a fixed constant while t̂ depends on the value of , and for any sufficiently small we will have t̂ > t̄.
Let us define an event A× = A×1 ∪A × 2 where
A×1 = ∆ { ∃n < T η1 (δ) s.t. ‖Xηn(x)‖ > 3 } ,
A×2 = ∆
{ ∃j = 2, · · · , k∗ s.t.
max k=1,2,··· ,(Tηj (δ)−T η j−1(δ)−1)∧(2t̂/η)
η ∥∥∥ZTηj−1(δ)+1 + · · ·+ ZTηj−1(δ)+j∥∥∥ > ̃}
where the positive integer l∗ is defined in eq. (H.1).
Now we focus on the event B\A×. On one hand, Lemma H.15 shows that, for all η sufficiently small, on this event we have ‖Xηk‖ < 3 < ̄ for all k < T η 1 (δ) and
sup s∈[0,Tη
k∗ (δ)−T η 1 (δ)] ∥∥∥Xηbsc+Tη1 (δ − zη(s)∥∥∥ < 2 < ̄. Here
zη(s) = x̃η ( s,X(1); T ,W ) ∀s ≥ 0
with T = ( 0, T η2 (δ) − T η 1 (δ), T η 3 (δ) − T η 2 (δ), · · · , T η k∗(δ) − T η k∗−1(δ) ) , W =(
W η1 (δ),W η 2 (δ), · · · ,W η k∗(δ)
) , and X(1) = Xη
Tη1 (δ)−1 . To better control the location of
zη(s), we also construct
zη0 (s) = x̃ η ( s,0; T ,W ) ∀s ≥ 0
where the only difference is that we substitute the initial value X(1) with 0.
First of all, from eq. (H.20), we see that
d(zη(s),Gc) > 100l∗̄, d(zη0 (s),Gc) > 100l∗̄ ∀s ∈ [0, k∗∑ i=1 Ti).
Besides, due to X ⊆ h−1 ( B(ỹ, ̄) ) , we have that d ( zη0 (∑k∗ i=1 Ti ) ,G ) > 99l∗̄. Next, using Corol-
lary H.7, we have that sup∈[0,∑k∗i=1 Ti] ‖zη(s)− zη0 (s)‖ < 3ρ∗ < ̄, which further implies that d ( zη ( k∗∑ i=1 Ti ) ,G ) > 99l∗̄− ̄ ≥ 98l∗̄.
Now we have the following facts regarding the distance between SGD iterates Xηj and the domain G. First, on event B\A×, it holds that
d(Xηj ,G c) > 100l∗̄− ̄ ≥ 99l∗̄ ∀j < T ηk∗(δ),
implying that σ(η) ≥ T ηk∗(δ).
Next, if R( , δ, η) < T ηk∗(δ), then there exists some T η 1 (δ ≤ j < T η k∗(δ) such that ∥∥Xηj ∥∥ ≤ 2 hence ‖zη(j − T η1 )‖ ≤ 2 + 2 < ̄ < 100l∗̄. However, in light of eq. (H.20) we know that on event {R( , δ, η) < T ηk∗(δ)} ∩ (B\A×), we have
d ( zη ( k∗∑ i=1 Ti ) ,Gc ) > 100l∗̄
and yield a contradiction. Therefore, on event B\A× we must have R( , δ, η) ≥ T ηk∗(δ). Besides, bounding the gap between Xηj , z η(j) and zη0 (j) at time j = T η k∗(δ) using results above, we can show that
d(Xη Tη k∗ (δ) ,G) > 98l∗̄− ̄ ≥ 97l∗̄.
Therefore, we must have σ(η) = T ηk∗(δ). In summary, we have shown that, for all η sufficiently small, B\A× ⊆ A◦( , δ, η). Combining all the bounds above, we have
lim inf η↓0 inf‖x‖≤2 Px(A◦( , δ, η)) λ̃(η) ≥ lim inf η↓0 P(B) λ̃(η) − lim sup η↓0 sup‖x‖≤2 P(A×) λ̃(η)
≥ δα1c∗ − lim sup η↓0
sup‖x‖≤2 P(A×)
λ̃(η) .
To conclude the proof, it suffices to invoke Lemma H.10 and H.12 with some N > 1 + JG − α1.

heading: H.4 PROOF OF THE MAIN RESULT
text: Now we are ready to state the main result and provide upper and lower bounds to the distribution of the scaled first exit time from domain G. In particular, we define a scaling function
λ(η) =∆ λ̃(η)H(1/η) (H.56)
where the function λ̃(·) is defined in eq. (H.50). One can easily see that λ ∈ RV1+JG (η). We show that the scaled first exit time λ(η)σ(η) converges in distribution to an exponential random variable as η ↓ 0, which implies that, in expectation, the first exit time is roughly a 1/η1+JG term and its order is dictated by the minimum cost for exit JG . Proposition H.23. Given any C ∈ (0, 1), u > 0, the following inequalities hold for all > 0 sufficiently small,
lim sup η↓0 sup ‖x‖≤2
Px(σ(η)λ(η) > u) ≤ 2C + exp(−(1− C)2qu),
lim inf η↓0 inf ‖x‖≤2
Px(σ(η)λ(η) > u) ≥ −C + exp(−(1 + C)qu)
where q = µ ( h−1(Gc) ) .
Before presenting the proof to Proposition H.23, we make some preparations. First, we introduce stopping times (for all k ≥ 1)
τk( , δ, η) = min{n > τ̃k−1( , δ, η) : η ‖Zn‖ > δ} τ̃k( , δ, η) = min{n ≥ τk( , δ, η) : ‖Xηn‖ ≤ 2 }
with the convention that τ0( , δ, η) = τ̃0( , δ, η) = 0. Intuitively, at each τ̃k the SGD iterates have just returned to a small neighborhood of the local minimum 0. Due to Markov property of Xηn, the SGD iterates almost regenerate at each τ̃k despite the previous trajectory, and the times (τ̃k)k≥1 partitions the entire timeline into different segments that can almost be interpreted as regeneration cycles, where τ̃k−1 can be understood as the starting point of the k−th cycle. In light of the embedded (informal) regeneration process, one natural approach is to determine the dynamics and probability of making an exit on one (hence every) cycle, and this will be carried out with the help of technical results in the previous section. It is worth noticing that τ̃k are defined under the proviso that τ̃k ≥ τk where τk is
the first big jump during the k−th cycle. Considering results such as Lemma H.12, it is reasonable to expect that the SGD iterates would be trapped at local minimum until a large jump strikes. Therefore, we define the (informal) regeneration points τ̃k in such a way that a cycle [τ̃k−1, τ̃k) ends only if we have observed at least one large jump already. Regarding the notations, we add a remark that when there is no ambiguity we will drop the dependency on , δ, η and simply write τk, τ̃k.
Specifically, we are interested in the large jumps and the accumulated cost thereof during each cycle, which definitely entails some systematic bookkeeping. For all k ≥ 1, the random variable
jk = ∆ #{n = τk−1( , δ, η), τk−1( , δ, η) + 1, · · · , τ̃k( , δ, η) ∧ σ(η) : η ‖Zn‖ > δ}
can be understood as the count of large jumps during the k−th cycle. Here are two remarks on this definition.
• First, for any k with σ(η) < τ̃k, we have jk = 0. Recall that the object we study here is the first exit time σ(η), so there is virtually no need to keep track of the dynamics of the SGD after it leaves the domain G.
• The random variable jk is measurable w.r.t. Fτ̃k∧σ(η), the stopped σ−algebra generated by the stopping time τ̃k ∧ σ(η), which, intuitively, is stating that one should be able to determine the number of large jumps during the k−th cycle when this cycle ends.
Furthermore, for each k = 1, 2, · · · , let
Tk,1( , δ, η) = ∆ τk−1( , δ, η) ∧ σ(η),
Tk,j( , δ, η) = ∆ min { n > Tk,j−1( , δ, η) : η ‖Zn‖ > δ } ∧ σ(η) ∧ τ̃k ∀j ≥ 2,
Wk,j( , δ, η) = ∆ ZTk,j( ,δ,η) ∀j ≥ 1
with the convention Tk,0( , δ, η) = τ̃k−1( , δ, η). Note that for any k ≥ 1, j ≥ 1, Tk,j is a stopping time. Besides, for any k with τ̃k < σ(η)
τ̃k−1 + 1 ≤ Tk,j ≤ τ̃k ∧ σ(η) ∀j ∈ [jk], (H.57)
and the sequences ( Tk,j )jk j=1 and ( Wk,j )jk j=1
are the arrival times and sizes of large jumps during the k−th cycle, respectively. Again, when there is no ambiguity we will drop the dependency on , δ, η and simply write Tk,j and Wk,j .
We are now able to keep track of the accumulated cost of jumps during each cycle. For any k ≥ 1, i ≥ 1, we define
J cyclek (i; , δ, η) = ∆ i∑ l=1 J(Wk,i( , δ, η)).
Moreover, we define the following index j≥JG (k; , δ, η) =∆ { min{j = 1, 2, · · · , jk : J cycle k (j; , δ, η) ≥ JG} if τ̃k( , δ, η) < σ(η)
∞ otherwise
with the convention that min ∅ = ∞. That is to say, if τ̃k ≥ σ(η) or J cyclek (jk) < JG , we define j≥JG (k) as∞; otherwise j≥JG (k) is the index for the first large jump during the k−th cycle that drives the accumulated cost of large jumps to reach JG . For clarity of the presentation below, we also define
T≥JG (k; , δ, η) =∆ 1{j≥JG (k) <∞}Tk,j≥JG (k) + 1{j ≥JG (k) =∞}0,
X≥JG (k; , δ, η) =∆ 1{j≥JG (k) <∞}Xη T≥JG (k; ,δ,η) + 1{j≥JG (k) =∞}0.
In other words, when j≥JG (k) <∞, the random variable X≥JG (k; , δη) is equal to XηT k,j≥JG (k) , the location of the SGD iterates right when the accumulated cost on the k−th cycle reaches JG . Notation wise, we will drop the dependency on , δ, η again and simply write j≥JG (k), X≥JG (k) or J cyclek (j) when there is no risk of ambiguity.
As have mentioned above, we want to zoom in on the each cycle and analyze the probability of each possible case. To be specific, we want to introduce a series of scenarios, formally defined as several events, that exhaust all the possibilities during a cycle. The events will be denoted in the form of A◦k or B×j largely following the next few rules. First, we say an event/scenario is atypical if its probability is rather small, and we assign it with superscript ×; otherwise we say the event is typical and add a superscript ◦. Besides, the subscript k indicates the cycle that the event concerns. Lastly, events with label (of type) A usually describe how the SGD iterates try to escape from G, whereas events with label B focuses on how the SGD iterates return to the local minimum.
Now we proceed and formally define the said series of events. First, for each k ≥ 1, define the event
A×k,0( , δ, η) = ∆
{ ∃i ∈ [l∗] s.t. max j=Tk,i−1+1,Tk,i−1+2,··· ,(Tk,i−1)∧(Tk,i−1+d 2t̂( )η e)∧τ̃k∧σ(η) η ∥∥ZTk,i+1 + · · ·+ Zj∥∥ > ̃( )}
∪ {∥∥Xηj ∥∥ > 3 for some j = τ̃k−1, τ̃k−1 + 1, · · · , Tk,1 − 1} (H.58)
with t̂( ) = c1 + c1 log(1/ ) and function ̃(·) defined in eq. (H.51). It is similar to the event A× defined in eq. (H.52)eq. (H.53) in the previous section, and its probability will be controlled using a similar approach.
Next, consider event (for all k ≥ 1)
A×k,1( , δ, η) = ∆ { σ(η) < τ̃k, J cyclek (jk) < JG } (H.59)
that describes an atypical case where the exit occurs during the k−th cycle yet the accumulated cost of all large jumps is less than JG . Another atypical event is defined as
A×k,2 = ∆ {j≥JG (k) <∞} ∩ {∃j = 2, 3, · · · , j≥JG (k) s.t. Tk,j − Tk,j−1 > 2t̂( )/η}. (H.60)
This event describes the case where the accumulated cost of large jumps in a cycle has reached JG but the inter-arrival time between some large jumps are unusually long. We stress that, by definition of j≥JG (k), we have {j≥JG (k) <∞} = {τ̃k−1 < σ(η)} ∩ {J cyclek (jk) ≥ JG}. Moving on, we consider the following events (defined for all k ≥ 1)
A×k,3 = ∆ {J cyclek (jk) < JG , τ̃k < σ(η)} ∩ { ∃j = 2, 3, · · · , jk s.t. Tk,j − Tk,j−1 > 2t̂( )/η } (H.61)
that describes another similar case where the k−th cycle ends with return to the local minimum but the inter-arrival time between some large jumps are unusually long.
The next event
A×k,4 = ∆ {j≥JG (k) <∞, J cyclek (j ≥JG (k)) = JG} ∩ {d(X≥JG (k),Gc) ≤ ̄} ∩ { ∃j ∈ [j≥JG (k)] s.t. η ‖Wk,j‖ ≤ δ̄ } (H.62)
describes the case where the accumulated cost on the k−th cycle has hit JG exactly at some point with SGD iterates being rather close to Gc yet some large jumps are not large enough when compared to the fixed constant δ̄.
Meanwhile, with events
A×k,5 = ∆ {j≥JG (k) <∞, J cyclek (j ≥JG (k)) = JG} ∩ { d(X≥JG (k), ∂G) < } , (H.63)
we analyze the case where the SGD iterates reaches somewhere close the the boundary set ∂G with large jumps of cost equal to JG . Lastly, define event
A×k,6 = ∆ {j≥JG (k) <∞, J cyclek (j ≥JG (k)) > JG} (H.64)
for the case where the accumulated cost of large jumps on the k−th cycle exceeds JG without hitting it. As an amalgamation of these atypical scenarios, we let
A×k ( , δ, η) = ∆ 6⋃ i=0 A×k,i( , δ, η). (H.65)
Next, we analyze the probability of some events (B×k )k≥1 that concern the behavior of SGD iterates during the k−th cycle after T≥JG (k). Let us define
B×k,1( , δ, η) = ∆ {j≥JG (k) <∞, J cyclek (j ≥JG (k)) = JG} ∩ {d(X≥JG (k),Gc) ≥ }
∩ { Tk,j − Tk,j−1 ≤ 2 t̂( )
η ∀j = 2, 3, · · · , j≥JG (k) } B×k,2( , δ, η) = ∆ {τ̃k − Tk,j≥JG (k) > ρ( )/η} ∪ {σ(η) < τ̃k}
B×k ( , δ, η) = ∆ B×k,1 ∩ B × k,2 (H.66)
where ρ(·) is the function in Lemma H.11. From the definition of B×k , in particular the inclusion of B×k,2, one can see that the intuitive interpretation of event B × k is that the SGD iterates did not return to local minimum efficiently or even escaped from G after large jumps with cost equal to JG . In comparison, the following events will characterize what would typically happen during each attempt:
A◦k( , δ, η) = ∆ {j≥JG (k) <∞, J cyclek (j ≥JG (k)) = JG} ∩ {σ(η) = T≥JG (k)} ∩ { Tk,j − Tk,j−1 ≤ 2t̂( )
η ∀j = 2, 3, · · · , j≥JG (k)
} , (H.67)
B◦k( , δ, η) = ∆ {σ(η) > τ̃k, τ̃k − Tk,1 ≤
2l∗t̂( ) + ρ( )
η } ∩ {J cyclek (jk) ≤ JG}. (H.68)
Intuitively speaking, A◦k tells us that the exit happened right at T≥JG (k) with accumulated cost of large jumps in the cycle being exactly JG , and B◦k requires that that the first exit did not occur during the k−th cycle, and the SGD iterates returned to local minimum rather efficiently. It is worth noticing that, by definition, j≥JG (k) < l∗ if j≥JG (k) <∞. Our immediate next goal is to analyze the event
A×( , δ, η) =∆ ⋃ k≥1 ( k−1⋂ i=1 ( A×i ∪ B × i ∪ A ◦ i )c) ∩ (A×k ∪ B×k ). (H.69) In particular, we show that its probability can be made arbitrarily small with proper , δ, η, implying that we will almost always observe the event (A×)c. Lemma H.24. Given any C > 0, the following claim holds for all > 0, δ > 0 sufficiently small:
lim sup η↓0 sup x: ‖x‖≤2
Px(A×( , δ, η)) ≤ C.
Proof. We start by bounding for the probabilities of all the A×k,i events. Fix some N > 1 + JG . For
A×k,0( , δ, η) = { ∃i ∈ [l∗] s.t. max j=Tk,i−1+1,Tk,i−1+2,··· ,(Tk,i−1)∧(Tk,i−1+d 2t̂( )η e)∧τ̃k∧σ(η) η ∥∥ZTk,i+1 + · · ·+ Zj∥∥ > ̃( )}
∪ {∥∥Xηj ∥∥ > 3 for some j = τ̃k−1, τ̃k−1 + 1, · · · , Tk,1 − 1},
using Lemma H.10 and H.12, one can show the existence of some 0 ∈ (0, ̄/3) such that for any ∈ (0, 0), there is some δ0( ) > 0 with
lim sup η
sup‖x‖≤2 Px(A × 1,0( , δ, η))
ηN = 0 ∀δ ∈ (0, δ0( )). (H.70)
Next, for A×k,1( , δ, η) = { σ(η) < τ̃k, J cyclek (jk) < JG } , it follows from Lemma H.17 that, for any
∈ (0, 0), there exists some 0 < δ1( ) ≤ δ0( ) such that
lim sup η
sup‖x‖≤2 Px(A × 1,1( , δ, η))
ηN = 0 ∀δ ∈ (0, δ1( )). (H.71)
For event A×k,2 = {j≥JG (k) < ∞} ∩ {∃j = 2, 3, · · · , j≥JG (k) s.t. Tk,j − Tk,j−1 > 2t̂( )/η}, Lemma H.18 implies that, for any ∈ (0, 0), there exists some 0 < δ2( ) ≤ δ1( ) such that
lim sup η
sup‖x‖≤2 Px(A × 1,2( , δ, η))
ηN = 0 ∀δ ∈ (0, δ2( )). (H.72)
For event A×k,3 = {J cycle k (jk) < JG , τ̃k < σ(η)}∩ { ∃j = 2, 3, · · · , jk s.t. Tk,j−Tk,j−1 > 2t̂( )/η } , thanks to Lemma H.19, one can see that, for any ∈ (0, 0), there exists some 0 < δ3( ) ≤ δ2( ) such that
lim sup η
sup‖x‖≤2 Px(A × 1,3( , δ, η))
ηN = 0 ∀δ ∈ (0, δ3( )). (H.73)
For event
A×k,4 = {j ≥JG (k) <∞, J cyclek (j ≥JG (k)) = JG} ∩ {d(X≥JG (k),Gc) ≤ ̄} ∩ { ∃j ∈ [j≥JG (k)] s.t. η ‖Wk,j‖ ≤ δ̄ } ,
using Lemma H.20, one can see the existence of some ∆ > 0 such that, for all ∈ (0, 0), there is some 0 < δ4( ) ≤ δ3( ) such that
lim sup η
sup‖x‖≤2 Px(A × 1,4( , δ, η))
η1+JG−α1+∆ = 0 ∀δ ∈ (0, δ4( )). (H.74)
As for event A×k,5 = {j ≥JG (k) < ∞, J cyclek (j ≥JG (k)) = JG} ∩ { d(X≥JG (k), ∂G) < } , the plan is to use Lemma H.21 to control its probability. But before that, we recall the definition of function Ψ( ) = µ ( h−1 ( (∂G)(3+3ρ∗) )) in Lemma H.21 and make several observations. First, note that ∂G
is a closed set, so ∩ >0h−1 ( (∂G) ) = h−1(∂G) for the mapping h in eq. (H.38). Then it follows
from Assumption G.9 that lim ↓0 µ ( h−1 ( (∂G) )) = 0, implying that for > 0 sufficiently small,
we must have
Ψ( ) < Cc∗
4 .
where C > 0 is the fixed constant in the description of the Lemma, and c∗ > 0 is the constant in Lemma H.22. We stress that both of them would not vary with , δ, η. Combining this with Lemma H.21, we know the existence of some 5 ∈ (0, 0] such that, for all ∈ (0, 5), one can find some 0 < δ5( ) ≤ δ4( ) with
lim sup η
sup‖x‖≤2 Px(A × 1,5( , δ, η))
λ̃(η) ≤ δαΨ( ) < δαCc∗ 4 ∀δ ∈ (0, δ5( )). (H.75)
Lastly, for event A×k,6 = {j ≥JG (k) <∞, J cyclek (j ≥JG (k)) > JG}, from Lemma H.20 again we know that (let 6 = 5) for all ∈ (0, 6), one can find some 0 < δ6( ) ≤ δ5( ) with
lim sup η
sup‖x‖≤2 Px(A × 1,6( , δ, η))
η1+JG−α1+∆ = 0 ∀δ ∈ (0, δ6( )). (H.76)
Recall that A×k ( , δ, η) = ⋃6 i=0 A × k,i( , δ, η). Also, the events B × k ,A ◦ k,B ◦ k are defined in eq. (H.66)eq. (H.67)eq. (H.68) respectively. Before bounding the conditional probability of events B×k , we make several observations. First, if we consider the event ⋂k j=1(A × j ∪ B × j )
c ∩ B◦j , the inclusion of the (B◦j )kj=1 implies that during the first k cycles the SGD iterates have never left G, so
k⋂ j=1 (A×j ∪ B × j ) c ∩ B◦j = ( k⋂ j=1 (A×j ∪ B × j ) c ∩ B◦j ) ∩ {σ(η) > τ̃k}.
Next, note that
Px ( B×k | k−1⋂ j=1 (A×j ∪ B × j ) c ∩ B◦j )
=Px ( B×k,1 | k−1⋂ j=1 (A×j ∪ B × j ) c ∩ B◦j )
︸ ︷︷ ︸ =∆ (I)
·Px ( B×k,2 ∣∣∣ ( k−1⋂
j=1
(A×j ∪ B × j ) c ∩ B◦j ) ∩ B×k,1 ) ︸ ︷︷ ︸
=∆ (II)
.
We start by analyzing term (I). From Assumption G.7, one can see that on event B×k,1, it holds that
j≥JG (k) = k∗, (Wk,1, · · · ,Wk,k∗) is of type-j for some j ∈ j(i∗)
and Tk,j − Tk,j−1 ≤ 2 t̂( )η ∀j = 2, 3, · · · , k ∗. Due to independence of Wk,i and Tk,i+1 − Tk,i, we have
sup ‖x‖≤2
(I) ≤ ( P(T η1 (δ) ≤ 2t̂( ) η ) )k∗−1 · ∑
j: j∈j(i∗)
P (
(W η1 (δ), · · · ,W η k∗(δ)) is of type-j
)
= ( P(T η1 (δ) ≤ 2t̂( ) η ) )k∗−1 · ∑
j: j∈j(i∗) k∗∏ i=1 Hji(δ/η) H(δ/η) .
Due to the strong Markov property at stopping time τ̃k−1, Lemma E.4, and the regularly varying nature of functions Hi, H ,
lim sup η↓0 sup‖x‖≤2 (I)( 2t̂( )/δα1 )k∗−1( H(1/η)/η )k∗−1 · |j(i∗)| · δk∗α1 δk ∗+JG · (∏m q=1 ( Hi(1/η) )i∗q)/(H(1/η))k∗ ≤ 1. In other words, for any , δ > 0, there exists some C( , δ) <∞ such that
lim sup η↓0
sup‖x‖≤2 (I)
δα1 λ̃(η) ≤ |j(i∗)|C( , δ).
On the other hand, due to Lemma H.11 and the strong Markov property at stopping time Tk,k∗ , for any ∈ (0, ̄) and any δ > 0,
lim sup η↓0 sup ‖x‖≤2
(II) < Cc∗/4
|j(i∗)|C( , δ) .
In summary, we have established that for any ∈ (0, ̄) and any δ > 0,
lim sup η↓0
sup‖x‖≤2 Px ( B×k | ⋂k−1 j=1 (A × j ∪ B × j ) c ∩ B◦j )
δα1 · λ̃(η) < Cc∗/4 ∀k ≥ 1. (H.77)
Similarly, we can bound conditional probabilities of the form Px ( A×k | ⋂k−1 j=1 (A × j ∪ B × j ) c ∩ B◦j ) . To be specific, recall that A×k = ∪6i=0A × k,i. Combining eq. (H.70)-eq. (H.76) with strong Markov property, we know the for all ∈ (0, 6) and k ≥ 1,
lim sup η↓0
sup‖x‖≤2 Px ( A×k | ⋂k−1 j=1 (A × j ∪ B × j ) c ∩ B◦j )
δα1 · λ̃(η) < Cc∗/4 ∀δ ∈ (0, δ6( )). (H.78)
On the other hand, a lower bound can be established for conditional probability of event A◦k, the event defined in eq. (H.67). Due to Lemma H.22 and the strong Markov property at τ̃k−1, one can see the existence of some 7 ∈ (0, 6] such that, for any ∈ (0, 7), there is some 0 < δ7( ) ≤ δ6( ) such that
lim inf η↓0
inf‖x‖≤2 Px ( A◦k | ⋂k−1 j=1 (A × j ∪ B × j ) c ∩ B◦j )
δα1 λ̃(η) ≥ c∗ ∀δ ∈ (0, δ7( )), k ≥ 1. (H.79)
In order to apply the bounds eq. (H.77)-eq. (H.79), we make use of the following inclusion relationship:
( k−1⋂ j=1 (A×j ∪ B × j ) c ∩ B◦j ) ∩ (A×k ∪ B × k ) c ⊆ A◦k ∪ B ◦ k. (H.80)
To see why this is true, let us consider a decomposition of the event on the L.H.S. of eq. (H.80). As mentioned above, on event ⋂k−1 j=1 (A × j ∪B × j )
c ∩B◦j we know that σ(η) > τ̃k−1, implying that for the first k− 1 cycles the first exit does not occur and leaving us with only three possibilities on this event:
• j≥JG (k) =∞ (on {σ(η) > τ̃k−1}, this would happen if and only if J cyclek (jk) < JG);
• j≥JG (k) <∞, X≥JG (k) /∈ G;
• j≥JG (k) <∞, X≥JG (k) ∈ G.
Let us partition the said event accordingly and analyze them one by one.
• On (⋂k−1
j=1 (A × j ∪ B × j ) c ∩ B◦j ) ∩ (A×k ∪ B × k )
c ∩ {J cyclek (jk) < JG}, due to the exclusion of A×k (especially A × k,1 and A × k,3), we can see that if jk < l
∗, then we must have σ(η) > τ̃k and τ̃k − Tk,1 ≤ 2l∗t̂( )/η. In particular, note that on this event we must have l∗ > jk by definition of l∗ in eq. (H.1). Therefore,
( k−1⋂ j=1 (A×j ∪ B × j ) c ∩ B◦j ) ∩ (A×k ∪ B × k ) c ∩ {j≥JG (k) =∞} ⊆ B◦k.
• On (⋂k−1
j=1 (A × j ∪B × j ) c∩B◦j ) ∩(A×k ∪B × k )
c∩{j≥JG (k) <∞, X≥JG (k) /∈ G}, the exclusion of A×k,2 implies that Tk,j − Tk,j−1 ≤ 2t̂( )/η for all j = 2, · · · , j
≥JG (k), and the exclusion of A×k,6 tells us that J cycle k (j ≥JG (k)) = JG . Moreover, the exclusion of A×k,5 ∪ A × k,6 tells us that if X≥JG (k) /∈ G, we must have d(X≥JG (k),G) > . In summary,
( k−1⋂ j=1 (A×j ∪ B × j ) c ∩ B◦j ) ∩ (A×k ∪ B × k ) c ∩ {j≥JG (k) <∞, X≥JG (k) /∈ G} ⊆ A◦k.
• On (⋂k−1
j=1 (A × j ∪ B × j ) c ∩ B◦j ) ∩ (A×k ∪ B × k )
c ∩ {j≥JG (k) <∞, X≥JG (k) ∈ G}, the same argument in the previous bullet point can be applied to show that Tk,j − Tk,j−1 ≤ 2t̂( )/η for all j = 2, · · · , j≥JG (k), J cyclek (j
≥JG (k)) = JG , and d(X≥JG (k),Gc) > . Now since B×k did not occur, we must have σ(η) > τ̃k and τ̃k − T≥JG (k) ≤ ρ( )/η, hence τ̃k − Tk,1 ≤ 2l ∗ t̂( )+ρ( ) η . Therefore,
( k−1⋂ j=1 (A×j ∪ B × j ) c ∩ B◦j ) ∩ (A×k ∪ B × k ) c ∩ {j≥JG (k) <∞, X≥JG (k) ∈ G} ⊆ B◦k.
With eq. (H.80) established, we can immediately get that
( k−1⋂ j=1 (A×j ∪ B × j ) c ∩ B◦j ) ∩ (A×k ∪ B × k ) c = ( k−1⋂ j=1 (A×j ∪ B × j ) c ∩ B◦j ) ∩ (A×k ∪ B × k ) c ∩ (A◦k ∪ B ◦ k).
(H.81)
Next, recall the definitions of A◦k in eq. (H.67) and B ◦ k in eq. (H.68), and one can see that the two events A◦k and B ◦ k are mutually exclusive, since the former implies that the first exit occurs during the
k−th cycle whereas the latter implies the first exit does not occur in the first k cycles. This fact and eq. (H.81) allow us to conclude that k⋂ i=1 ( A×i ∪ B × i ∪ A ◦ i )c = k⋂ i=1 ( A×i ∪ B × i )c ∩ B◦i = ( k−1⋂ i=1 ( A×i ∪ B × i )c ∩ B◦i) ∩ (A×k ∪ B×k ∪ A◦k)c. (H.82)
The last step is to use all the results so far to bound the probability of event
A×( , δ, η) = ⋃ k≥1 ( k−1⋂ i=1 ( A×i ∪ B × i ∪ A ◦ i )c) ∩ (A×k ∪ B×k ). Using eq. (H.82), we can see that (for any x with ‖x‖ ≤ 2 )
Px(A×( , δ, η))
= ∑ k≥1 Px (( k−1⋂ i=1 ( A×i ∪ B × i ∪ A ◦ i )c) ∩ (A×k ∪ B×k )) = ∑ k≥1 Px (( k−1⋂ i=1 ( A×i ∪ B × i )c ∩ B◦i) ∩ (A×k ∪ B×k )) = ∑ k≥1 Px ( A×k ∪ B × k | k−1⋂ i=1 ( A×i ∪ B × i )c ∩ B◦i) · k−1∏ j=1 Px ( j⋂ i=1 ( A×i ∪ B × i )c ∩ B◦i ∣∣∣ j−1⋂ i=1 ( A×i ∪ B × i )c ∩ B◦i) = ∑ k≥1 Px ( A×k ∪ B × k | k−1⋂ i=1 ( A×i ∪ B × i )c ∩ B◦i)
· k−1∏ j=1 Px (( j−1⋂ i=1 ( A×i ∪ B × i )c ∩ B◦i ) ∩ (A×j ∪ B×j ∪ A◦j )c ∣∣∣ j−1⋂ i=1 ( A×i ∪ B × i )c ∩ B◦i)
= ∑ k≥1 Px ( A×k ∪ B × k | k−1⋂ i=1 ( A×i ∪ B × i )c ∩ B◦i) · k−1∏ j=1 Px (( A×j ∪ B × j ∪ A ◦ j )c ∣∣∣ j−1⋂ i=1 ( A×i ∪ B × i )c ∩ B◦i) ≤ ∑ k≥1 Px ( A×k ∪ B × k | k−1⋂ i=1 ( A×i ∪ B × i )c ∩ B◦i) · k−1∏ j=1 ( 1− Px ( A◦j ∣∣∣ j−1⋂ i=1 ( A×i ∪ B × i )c ∩ B◦i) ) .
This allows us to apply eq. (H.77)-eq. (H.79) and conclude that for all ∈ (0, 7), all δ ∈ (0, δ7( )) and all η > 0
sup |x|≤2 Px(A×( , δ, η)) ≤ ∑ k≥1 ( δα1 λ̃(η) · c∗C 2 ) · ( 1− δα1 λ̃(η) · c∗ 2 )k−1 = δα1 λ̃(η) · c∗C2 δα1 λ̃(η) · c∗2 = C.
when η is sufficiently small.
Having established Lemma H.24, we are ready to provide a proof for Proposition H.23.
Proof of Proposition H.23. Define sets
E+(z) =∆ {y ∈ Rd : d(y,G) > (3 + 3ρ∗)z}, E−(z) =∆ {y ∈ Rd : d(y,Gc) < (3 + 3ρ∗)z}
where ρ∗ is the fixed constant in Corollary H.7. From the continuity of measure, we have
lim ↓0
µ ( h−1(E+( )) ) = µ ( h−1((G)c) ) , lim ↓0 µ ( h−1(E−( )) ) = µ ( h−1(Gc) ) .
From Assumption G.7 we know q =∆ µ ( h−1(Gc) ) > 0. From Assumption G.9 we also have
µ ( h−1(∂G) ) = 0, hence µ ( h−1((G)c) ) = q as well. Therefore, together with Lemma H.24, we know the existence of some 1 ∈ (0, ̄3+3ρ∗ ) such that, for any ∈ (0, 1), there is some 0 < δ1( ) with
lim sup η↓0 sup ‖x‖≤2
Px(A×( , δ, η)) < C.
and (1− C)q < µ ( h−1(E+( )) ) ≤ q ≤ µ ( h−1(E−( )) ) < (1 + C)q.
Henceforth, we fix some and δ in this range and prove the inequalities in the Proposition.
To proceed, let J( , δ, η) =∆ sup{k ≥ 1 : τ̃k−1 < σ(η)}. We now show that on event (A×( , δ, η))c, we must have J ≤ J↑ where (recall Definition H.1)
J↑( , δ, η) =∆ min { k ≥ 1 : ZTk,1 is of type( E+( ), η, δ )} with E+(z) =∆ {y ∈ Rd : d(y,G) > (3 + 3ρ∗)z} where ρ∗ is the fixed constant in Corollary H.7. To see this via a proof of contradiction, let us assume that J↑ = j < J for some integer j. Now we make some observations.
• First of all, due to J↑ = j < J and eq. (H.17), on event (A×( , δ, η))c ∩ {J↑ = j < J} we have
Tj,i − Tj,i−1 ≤ 2t̄/η ∀i = 2, 3, · · · , k∗;
• The definition of event (A×)c, especially the exclusion of event A×j,0 defined in eq. (H.58) now allows us to apply Lemma H.15 and show that
sup s∈[0,Tj,k∗−Tj,1] ∥∥∥Xηbsc+Tj,1 − zη(s)∥∥∥ < 2 . Here
zη(s) = x̃η ( s,X(1); T ,W ) ∀s ≥ 0
with T = ( 0, Tj,2−Tj,1, Tj,3−Tj,1, · · · , Tj,k∗ −Tj,k∗−1 ) ,W = ( Wj,1, · · · ,Wj,k∗ ) , and X(1) = Xη−1+Tj,1 . Moreover, the previous bullet point allows us to apply Corollary H.7 and conclude that
sup s∈[0,Tj,k∗−Tj,1]
‖zη(s)− zη0 (s)‖ ≤ 3ρ∗
where zη(s) = x̃η ( s,0; T ,W ) ∀s ≥ 0 and ρ∗ is the fixed constant in Corollary H.7. As a
result, ∥∥∥zη0 (Tj,k∗ − Tj,1)−XηTj,k∗∥∥∥ < (2 + 3ρ∗) .
• However, due to J↑ = j, we have d(zη0 (Tj,k∗ − Tj,1),G) > (3 + 3ρ∗) . This immediately implies that XηTj,k∗ /∈ G and yields the contradiction J ≤ j.
Similarly, one can show that on event (A×)c we must have J ≥ J↓ for J↓( , δ, η) =∆ min { k ≥ 1 : ZTk,1 is of type( E−( ), η, δ )} with E−(z) =∆ {y ∈ Rd : d(y,Gc) < (3 + 3ρ∗)z}.
Besides, the following claim holds on event (A×)c ∩ {J <∞}.
• The definition of (A×)c implies that during the J−th cycle the event A◦J occurs whereas for any j < J we have B◦j . Therefore, for any k = 1, 2, · · · , J , we have
τ̃k ∧ σ(η)− Tk,1 ≤ 2l∗t̂( ) + ρ( )
η .
In particular, the coefficient l∗ on the R.H.S. can be justified as follows: by definition of A◦k and B◦k, we can see that J cycle k (jk) ≤ JG for all k ≤ J (with jJ = k∗ due to the occurrence of A◦J ); the definition of l∗ in eq. (H.1) then implies that l∗ > jk for all k ≤ J .
• Now we consider the following set
S( , δ, η) =∆ ∪k≥1 {τ̃k−1 + 1, τ̃k−1 + 2, · · · , Tk,1 − 1, Tk,1}
the can be understood as the concatenation of all steps between any return time τ̃k−1 and the first large jump time Tk,1 during the k−th cycle. Our discussion above implies that, on event (A×)c, we have then the discussion above have shown that, for
min{n ∈ S( , δ, η) : Zn is of type-(E+( ), η, δ)} ≥ TJ,1.
It is worth noticing that the probability ofZ1 being of type-(E+( ), η, δ) is (see the definition in eq. (H.47)) H(δ/η)p ( E+( ), δ, η ) .
Therefore, for the following partition of the timeline
Sbefore( , δ, η) =∆ {n ∈ S( , δ, η) : n ≤ σ(η)}, Ibefore( , δ, η) =∆ #Sbefore( , δ, η), Safter( , δ, η) =∆ {n /∈ S( , δ, η) : n ≤ σ(η)}, Iafter( , δ, η) =∆ #Safter( , δ, η),
we have σ(η) = Ibefore + Iafter. Moreover, on event (A×)c, we must have Iafter ≤ J ( 2l∗t̂( ) + ρ( ) ) /η ≤ J↑ ( 2l∗t̂( ) + ρ( ) ) /η
Ibefore ≤ min{n ∈ S( , δ, η) : Zn is of type-(E+( ), η, δ)}}.
Next, define geometric random variables with the following success rates U1( , δ, η) ∼ Geom ( H(δ/η)p ( E+( ), δ, η )) ,
U2( , δ, η) ∼ Geom ( p ( E+( ), δ, η )) .
Given the results above for bounding Ibefore and Iafter on event (A×)c, we have
sup ‖x‖≤2
Px ( λ(η)σ(η) > u ) ≤ sup ‖x‖≤2 Px ( (A×)c ) + sup ‖x‖≤2 Px ( (A×)c ∩ { λ(η)(Ibefore + Iafter) > u
}) ≤ sup ‖x‖≤2 Px ( (A×)c ) + sup ‖x‖≤2 Px ( (A×)c ∩ { λ(η)Ibefore > (1− C)u
}) + sup ‖x‖≤2 Px ( (A×)c ∩ { λ(η)Iafter > Cu
}) ≤ sup ‖x‖≤2 Px ( (A×)c ) + P ( λ(η) · U1 > (1− C)u
)︸ ︷︷ ︸ ,(I) +P ( λ(η) · U2 · 2l∗t̂( ) + ρ( ) η > Cu ) ︸ ︷︷ ︸
,(II)
.
For term (I), recall that λ(η) = H(1/η)λ̃(η). Corollary H.14 and the regularly varying nature of H(·) then imply that
lim η↓0
H(δ/η)p ( E+( ), δ, η ) λ(η) = H(δ/η) · p ( E+( ), δ, η ) H(1/η) δα1 · δα1 λ̃(η) = µ ( h−1(E+( )) ) > (1− C)q.
From Lemma E.3, we then have lim supη↓0 (I) ≤ exp(−(1 − C)2u). For term (II), let C( ) = ∆ 2l∗t̂( ) + ρ( ) and note that
(II) = P ( U2 > C
C( ) · λ̃(η) λ(η)/η · 1 λ̃(η)
) .
Moreover, since λ̃(η)
λ(η)/η ∈ RV−α1+1(η),
we know that for any M > 0, we have λ̃(η)λ(η)/η > M eventually as η ↓ 0. Therefore, given any M > 0, we have (II) ≤ P ( U2 > M/λ̃(η) ) for any η sufficiently small. Now from Lemma E.3,
lim supη↓0 (II) ≤ exp(−M) for any M > 0, and here we fix some M > 0 such that exp(−M) < C. In summary, we have shown
lim sup η↓0 sup ‖x‖≤2
Px ( λ(η)σ(η) > u ) ≤ 2C + exp(−(1− C)2qu)
and established the upper bound. The lower bound can be shown by an almost identical approach. In particular, since
inf ‖x‖≤2
Px ( λ(η)σ(η) > u ) ≥ inf ‖x‖≤2 Px ( (A×)c ∩ { λ(η)(Ibefore + Iafter) > u }) ≥ inf ‖x‖≤2 Px ( (A×)c ∩ { λ(η)Ibefore > u
}) ≥P ( λ(η) · U ′1 > u ) − sup ‖x‖≤2 Px ( (A×)c )
where U ′1( , δ, η) ∼ Geom ( H(δ/η)p ( E−( ), δ, η )) . The same calculation above for term (I) can be
used here to provide a lower bound for lim infη↓0 P ( λ(η) · U ′1 > u ) and conclude the proof.
Now we are ready to prove Theorem G.1, the main theorem of this section.
Proof of Theorem G.1. Recall that our choice of scaling function λ(·) in eq. (H.56) is regularly varying (w.r.t. η) with index 1 + JG . Fix some x ∈ G, t > 0 and C ∈ (0, 1). It suffices to show that
lim sup η↓0
Px(σ(η)λ(η) > t) ≤ exp(−q(1− C)3t) + 2C,
lim inf η↓0
Px(σ(η)λ(η) > t) ≥ exp(−q(1 + C)t)− C.
First, we are able to pick some ∈ (0, 1) sufficiently small such that d(x,Gc) > and Proposition H.23 is applicable. Due to Lemma H.11, for event
A =∆ {Treturn(η, ) ≤ ρ( )/η, Xηn ∈ G ∀n ≤ Treturn(η, )}, we have limη↓0 Px(A) = 1. Next, on event A, we have σ(η)− Treturn(η, ) > 0 and ∥∥∥XηTreturn(η, )∥∥∥ ≤ 2 . Moreover, by combining Proposition H.23 with the strong Markov property at stopping time Treturn(η, ), we have
lim sup η↓0
Px (( σ(η)− Treturn(η, ) ) λ(η) > (1− C)t | A ) ≤ 2C + exp(−(1− C)3qt),
lim inf η↓0
Px (( σ(η)− Treturn(η, ) ) λ(η) > t | A ) ≥ −C + exp(−(1 + C)qt).
Observe that
Px(σ(η)λ(η) > t) ≤ Px({σ(η)λ(η) > t} ∩A) + Px(Ac) ≤ Px (( σ(η)− Treturn(η, ) ) λ(η) > (1− C)t | A ) Px(A)
+ Px ({ Treturn(η, )λ(η) > Ct } ∩A ) + Px(Ac).
Besides, on event A we have Treturn(η, ) ≤ O(1/η) as η ↓ 0. Given that λ(η) ∈ RV(1+JG)(η), we have Treturn(η, )λ(η) ≤ Ct on event A for all η sufficiently small. Therefore, by applying the bounds above, we establish that lim supη↓0 Px(σ(η)λ(η) > t) ≤ 2C + exp(−(1 − C)3qt). Similarly, in order to show the lower bound, observe that
Px(σ(η)λ(η) > t) ≥ Px({σ(η)λ(η) > t} ∩A) ≥ Px (( σ(η)− Treturn(η, ) ) λ(η) > t | A ) Px(A).
Taking lim inf on both sides yields lim infη Px(σ(η)λ(η) > t) ≥ −C + exp(−(1 + C)qt) and concludes the proof.

heading: I PROOF OF TECHNICAL LEMMAS
text: Proof of Lemma E.3. For any > 0, P ( U( ) > 1
b( )
) = ( 1− a( ) )b1/b( )c .
By taking logarithm on both sides, we have lnP ( U( ) > 1
b( )
) = b1/b( )c ln ( 1− a( ) ) = b1/b( )c 1/b( ) ln ( 1− a( ) ) −a( ) −a( ) b( ) .
Since limx→0 ln(1+x)
x = 1, we know that for sufficiently small, we will have
−ca( ) b( )
≤ lnP ( U( ) > 1
b( )
) ≤ − a( )
c · b( ) . (I.1)
By taking exponential on both sides, we conclude the proof.
Proof of Lemma E.4. To begin with, for any > 0 we have P ( U( ) ≤ 1
b( )
) = 1− P ( U( ) > 1
b( )
) .
Using bound eq. (I.1), we know that for sufficiently small, P(U( ) > 1/b( )) ≥ exp(−c·a( )/b( )). The upper bound follows from the generic bound 1− exp(−x) ≤ x, ∀x ∈ R with x = c · a( )/b( ). Now we move onto the lower bound. Again, from bound eq. (I.1), we know that for sufficiently small , we will have
P ( U( ) ≤ 1
b( )
) ≥ 1− exp(− 1√
c · a( ) b( ) ).
Due to the assumption that lim ↓0 a( )/b( ) = 0 and the fact that 1 − exp(−x) ≥ x√c for x > 0 sufficiently close to 0, we will have (for small enough) P ( U( ) ≤ 1b( ) ) ≥ 1c · a( ) b( ) .
Proof of Lemma E.5. Let ak = ∆ xk − x̃k. Using intermediate value theorem, one can easily see that
ak = η k∑ j=1 ( ∇g(x̃j−1)−∇g(xj−1) ) + η(z1 + · · ·+ zk) + x− x̃;
⇒ ‖ak‖ ≤ ηC(‖a0‖+ · · · ‖ak−1‖) + c̃.
The desired bound then follows immediately from Gronwall’s inequality.
Proof of Lemma H.1. Due to the finiteness of all types j′ with Jtype(j′) < JG , it suffices to fix one of such j′ and prove the existence of constants t̄, δ̄, 0.
Let ̄ be the constants in eq. (H.3). For any r ∈ (0, ̄), define the following stopping time
τr(x) = ∆ min{t ≥ 0 : xt(x) ∈ B(0, r)}.
Due to Assumptions G.4 and G.5 as well as continuity in Poincaré Map (see Theorem 12 in Immler & Traut (2019)), for any fixed r > 0 we know that τr(·) is a continuous function on G. Note that, by definition, we have τr(x) = 0 for any x ∈ B(0, r). Due to G being compact and eq. (H.3), we know the existence of some constant Tr ∈ (0,∞) such that supx∈Ḡ τr(x) ≤ Tr and xt(x) ∈ B(0, r) for any x ∈ G, t ≥ Tr.
Due to Assumption G.8 and the finiteness of all types j′ with Jtype(j′) < JG , there exists a 0 ∈ (0, ̄/2) such that
sup s≥0
d ( x̃(s,0; t,w),Gc ) > 2 0 (I.2)
for all j′ with Jtype(j′) < JG , t ∈ {0} × R|j ′|−1 + ,w ∈ A type j′ . We fix such 0 > 0. Here is one implication that is worth mentioning. Recall that j is fixed in the description of the Lemma and k′ = |j|. If
inf s≥0
d ( x̃(s,0; t(k ′),w(k ′)),Gc ) ≤ 0
for some t(k ′) ∈ {0} × Rk ′−1 + ,w (t′) ∈ Atypej , then we must have
d ( x̃( k′∑ i=1 ti,0; t (k′),w(k ′)),Gc ) ≤ 0.
To show the existence of some δ̄ > 0 we appeal to a proof by contradiction. (For a clean presentation, in t and w we omit the (k′) term in the superscript since the cardinality is fixed.) Assume that we can find a sequence ( tn,wn ) n≥1 such that min { ‖wn1 ‖ , · · · , ‖wnk′‖ } ≤ 1/n and
d ( x̃ (∑k′
i=1 t n i ,0; t n,wn ) ,Gc ) ≤ 0 for any n. Due to the truncation operator ϕb(·) in the definition
of x̃t, without loss of generality we can replace all jumps wnj by ϕb ( wnj )
to ensure that wnj is always in a compact set. Define
ynj = x̃ ( tnj ,0; t n,wn ) ∀j = 1, 2, · · · , k′.
By picking a subsequence when necessary, we can further assume that
• For any i ≥ 1, w(n)j converges to some w∗i and yni converges to some y∗i . In particular, there exists some I ∈ [k′] such that w∗I = 0.
• Also, since Gc is a closed set, we have d(y∗k′ ,Gc) ≤ 0.
• For any i = 2, 3, · · · , k′, tni either converges to some finite t∗i , or limn tni =∞.
• Note that
lim n tni =∞⇒ y∗i = 0 (I.3)
for the following reason: for any r ∈ (0, ̄), our discussion about τr(x) at the beginning of the proof implies that lim supn ‖yni ‖ ≤ r.
Let Ĩ be the largest index i ∈ [k′] such that y∗i = 0. In case that we cannot find such index, let Ĩ = 0. We first consider the case where Ĩ = 0. The bullet points above then imply that, for any i ≥ 1, we must have limn tni = t ∗ i <∞. Now due to boundedness of t∗i and the continuity of ODE flow, for
t∗ =∆ (0, t∗2, t ∗ 2, · · · , t∗k′),
w∗ =∆ (w∗1 , w ∗ 2 , w ∗ 3 , · · · , w∗k′),
we have
x̃( k′∑ i=1 t∗i ,0; t ∗,w∗) = y∗k′ .
Now recall that w∗I = 0 for some I ∈ [k′]. By removing the vacuous (size-zero) jumps in (t∗,w∗), we now know the existence of some k̃′ ∈ N, w̃∗ ∈ (Rd\O)k̃′ , t̃∗ ∈ {0} × Rk̃′+ such that d ( x̃( ∑k̃′ i=1 t̃ ∗ i ,0; t̃ ∗, w̃∗),Gc ) ≤ 0. Meanwhile, the condition J ↓type(j) < JG implies that, for
the total cost of jumps in w̃∗ after removal of size-zero jumps, ∑k̃′ i=1 J(w̃ ∗ j ) < JG. However, this contradicts eq. (I.2).
Similarly, in the case with Ĩ ≥ 1, we have that for any i > Ĩ , we must have limn tni = t∗i <∞. From the boundedness of t∗i and the continuity of ODE flow, for
t∗ =∆ (0, t∗ Ĩ+1 , t∗ Ĩ+2 , · · · , t∗k′), (I.4)
w∗ =∆ (w∗ Ĩ , w∗ Ĩ+1 , w∗ Ĩ+2 , · · · , w∗k′), (I.5)
we have
x̃( k′∑ i=Ĩ+1 t∗i ,0; t ∗,w∗) = y∗k′ . (I.6)
In particular, if Ĩ = 1, then for the index I with w∗I = 0, we must have that I ≥ Ĩ , meaning that there is at least one vacuous jump inw∗. The same argument for the case Ĩ = 0 above can lead to the same contradiction with eq. (I.2). Otherwise, with Ĩ ≥ 2, we already know that the accumulated cost of all jumps in w∗ is strictly less than JG , yet we still have d ( x̃( ∑k′ i=Ĩ+1 t ∗ i ,0; t ∗,w∗),Gc ) ≤ 0. This contradicts eq. (I.2) again.
In summary, we have established the existence of the lower bound δ̄ > 0 on jump sizes. We fix such δ̄ > 0. The existence of t̄ <∞ can be shown by an almost identical argument. In particular, if such t̄ < ∞ does not exist, then by picking a subsequence if needed we are able to find a converging sequence ( tn,wn ) n≥1 such that eq. (I.6) holds and Ĩ ≥ 1 due to inter-arrival time blowing up to infinity. In particular, since y∗1 = w ∗ 1 and ‖w∗1‖ ≥ δ̄ > 0, we must have Ĩ ≥ 2. By considering the same t∗,w∗ pair in equation I.4 and equation I.5, one can see that the accumulated cost of jumps in w is strictly less than JG and yield a contradiction with eq. (I.2).
Proof of Lemma H.2. Due to the finiteness of all types j′ with Jtype(j′) < JG , it suffices to fix one of such j′ and prove the existence of the positive constant 1.
To proceed with a proof by contradiction, the assume that such 1 > 0 does not exists. (For a clean presentation, in t and w we omit the (k′) term in the superscript since the cardinality is fixed.) As a result, we are able to pick a sequence (xn, tn,wn)n≥1 such that one of the following two cases must occur:
• limn ‖xn‖ = 0; For any n ≥ 1, infs≥0 d ( x̃(s, xn; tn,wn),Gc ) ≤ 0/2 and there is some
jn ∈ [k′] such that tnjn ≥ 2t̄;
• limn ‖xn‖ = 0; For any n ≥ 1, infs≥0 d ( x̃(s, xn; tn,wn),Gc ) ≤ 0/2 and there is some
jn ∈ [k′] such that ∥∥wnjn∥∥ ≤ δ̄/2.
We detail the analysis for the first case, as the second case can be addressed by an almost identical argument. First of all, due to the truncation operator ϕb(·) in the definition of x̃t, without loss of generality we can replace all jumps wnj by ϕb ( wnj )
to ensure that wnj is always in a compact set. Define
ynj = x̃ ( tnj , x n; tn,wn ) ∀j = 1, 2, · · · , k′.
By picking a subsequence if necessary, we can further assume that
• For any i ≥ 1, w(n)j converges to some w∗i and yni converges to some y∗i .
• Also, since Gc is a closed set, we have d(y∗k′ ,Gc) ≤ 0/2.
• For any i = 2, 3, · · · , k′, tni either converges to some finite t∗i , or limn tni =∞. In particular, there is some I ∈ [k′] such that limn tnI ≥ 2t̄.
• Due to eq. (I.3), limn tni =∞ would imply y∗i = 0.
Let I1 = max{i ∈ [k′] : limn tni ∈ [2t̄,∞)} and I2 = max{i ∈ [k′] : limn tni = ∞}. If either of the two sets above is empty, let the corresponding I1 or I2 be 0. The discussion above implies that at least one of them must be non-zero, so there are only two possibilities: (i) 0 ≤ I1 < I2; (ii) 0 ≤ I2 < I1. We consider each scenario respectively. First, if 0 ≤ I1 < I2, then limn ynI2 = y ∗ I2
= 0, implying that I2 < k′. Moreover, for any i = I2 + 1, I2 + 2, · · · , k′, limn tni = t∗i <∞. Using the boundedness of t∗i and the continuity of ODE flow, for
t∗ =∆ (0, t∗1+I2 , t ∗ 2+I2 , · · · , t ∗ k′), w ∗ =∆ (w∗I2 , w ∗ 1+I2 , w ∗ 2+I2 , · · · , w ∗ k′),
we have x̃( ∑k′ i=1+I2 t∗i ,0; t ∗,w∗) = y∗k′ with d(y ∗ k′ ,Gc) ≤ 0/2. However, this contradicts equation H.2.
Next, in scenario (ii) with 0 ≤ I2 < I1, we have limn tnI1 = t ∗ I1 ≥ 2t̄. Moreover, for any i = I2 + 1, I2 + 2, · · · , k′, limn tni = t∗i <∞. Using the boundedness of t∗i and the continuity of ODE flow, for
t∗ =∆ (0, t∗1+I2 , t ∗ 2+I2 , · · · , t ∗ k′), w ∗ =∆ (w∗I2 , w ∗ 1+I2 , w ∗ 2+I2 , · · · , w ∗ k′),
we have x̃( ∑k′ i=1+I2 t∗i ,0; t ∗,w∗) = y∗k′ with d(y ∗ k′ ,Gc) ≤ 0/2. However, bmw∗ is still of type j with J ↓type(j) < JG , and there is some i such that t∗i ≥ 2t̄. This would contradict Lemma H.1.
In summary, we have established the existence of 1 > 0 such that tj < 2t̄ is a necessary condition for any x ∈ B(0, 1) and any t = (t1, · · · , tk′) ∈ {0} × Rk ′−1 + ,w = (w1, · · · , wk′) ∈ A type j with
infs≥0 d ( x̃(s, x; t(k),w(k)),Gc ) ≤ 0/2. As mentioned above, the necessity of ‖wj‖ > δ̄/2 can be shown in an almost identical way. We omit the details here and conclude the proof.
Proof of Lemma H.3. Since there are only finitely many (i1, · · · , im) with J (i1, · · · , im) < JG , it suffices to fix one of such (i1, · · · , im) and establish the existence of the required 0, δ0. (Henceforth, let k = ∑ j=1 im.)
Assumption G.8, together with the bound in eq. (H.6), implies the existence of some 1 ∈ (0, ̄) such that
sup t≥0
d ( x̃(s,0; t(k),w(k) ) > 1 (I.7)
for anyw(k) ∈ A(i1, · · · , im), t(k) ∈ {0} ×Rk−1+ . For 0 = 1/2, we establish the existence of the prescribed δ0 via a proof by contradiction. (Henceforth we drop the notational dependence on (k) when referencing sequences t(k),w(k) since the cardinality k is fixed.)
Assume the existence of a sequence (xn)n≥1 in Rd, a sequence of real positives (sn)n≥1, a sequence (tn)n≥1 = (t1,n, · · · , tk,n)n≥1 in {0}×Rk−1+ , and a sequence (wn)n≥1 = (w1,n, · · · , wk,n)n≥1 in A(i1, · · · , im) such that limn ‖xn‖ = 0 and
d ( x̃(sn, xn; tn,wn),Gc ) ≤ 0.
Due to existence of the clipping operator, all jumps wj,n can be replaced by ϕb(wj,n) without loss of generality to ensure that all wj,n are in a compact set. Also, without loss of generality, all sn can be chosen as
sn = inf{s ≥ 0 : d ( x̃(s, xn; tn,wn),Gc ) ≤ 0}.
From Assumption eq. (H.6), one can see that sn must be equal to ∑jn i=1 ti,n for some jn ∈ [k], i.e., it must be the arrival time of some jump. Moreover, one can easily see that for
yn = x̃(sn, xn; tn,wn),
we always have d(yn,G) ≤ b so all yn are in a compact set as well due to the boundedness of G. Therefore, by picking a subsequence when necessary, we can further assume that
• There exists some j∗ ∈ [k] such that sn = ∑j∗ i=1 ti,n for all n ≥ 1, i.e. sn is always the
arrival time of the j∗−th jump;
• For any j ∈ [j∗], there exists some y∗j such that for yn,j = x̃( ∑j i=1 ti,n, xn; tn,wn)
we have limn yn,j = y∗j ; In particular, for any j < j ∗ we have d(y∗j ,Gc) > 1 and d(y∗j∗ ,Gc) ≤ 0;
• For any j ∈ [k], there exists some w∗j such that limn wj,n = w∗j ; Moreover, note that y∗1 = w ∗ 1 .
• In particular, (w∗1 , · · · , w∗k) ∈ A(i1, · · · , im);
• For any j ∈ [k], either there exists some t∗j <∞ such that limn tj,n = t∗j , or limn tj,n =∞ (in this case we let t∗j = ∞); in the latter case, due to the same argument in eq. (I.3), we must have limn x̃( ∑j i=1 ti,n, xi; tn,wn) = 0;
Obviously, y∗j∗ 6= 0. Let j↓ = ∆ max{j = 0, 1, · · · , j∗ : y∗j = 0} with the convention that y∗0 = 0, y ∗ 0,n = xn. We must have j↓ < j ∗ and
• limn y∗j↓,n = y ∗ j↓ = 0
• For any j = j↓ + 1, j↓ + 2, · · · , j∗, limn tj,n = t∗j <∞.
Now using the continuity of the ODE flow, we must have that
lim n x̃ ( j∗∑ j=j↓+1 tj,n, y ∗ j↓,n ; (tj↓+1,n, tj↓+2,n, · · · , tj∗,n), (wj↓+1,n, wj↓+2,n, · · · , wj∗,n) )
=x̃ ( j∗∑ j=j↓+1 t∗j ,0; (t ∗ j↓+1 , t∗j↓+2, · · · , t ∗ j∗), (w ∗ j↓+1 , w∗j↓+2, · · · , w ∗ j∗) ) = y∗j∗ .
However, due to d(y∗j∗ ,Gc) ≤ 0 and recall our choice of 0 = 1/2, for all n sufficiently large, we must have d ( x̃ ( j∗∑ j=j↓+1 tj,n, y ∗ j↓,n ; (tj↓+1,n, tj↓+2,n, · · · , tj∗,n), (wj↓+1,n, wj↓+2,n, · · · , wj∗,n) ) ,Gc ) ≤ 5 8 1.
Meanwhile, using Gronwall’s inequality repeatedly and supj=j↓+1,··· ,j∗,n≥1 tj,n <∞, by substituting the initial condition y∗j↓,n with 0, we have
lim n ∣∣∣∣∣∣x̃( j∗∑ j=j↓+1 tj,n, y ∗ j↓,n ; (tj↓+1,n, tj↓+2,n, · · · , tj∗,n), (wj↓+1,n, wj↓+2,n, · · · , wj∗,n) )
−x̃ ( j∗∑ j=j↓+1 tj,n,0; (tj↓+1,n, tj↓+2,n, · · · , tj∗,n), (wj↓+1,n, wj↓+2,n, · · · , wj∗,n) )∣∣∣∣∣∣ = 0.
This implies that, for all n sufficiently large,
d ( x̃ ( j∗∑ j=j↓+1 tj,n,0; (tj↓+1,n, tj↓+2,n, · · · , tj∗,n), (wj↓+1,n, wj↓+2,n, · · · , wj∗,n) ) ,Gc ) ≤ 3 4 1.
However, this contradicts eq. (I.7). This implies the existence of the required δ0 and concludes the proof.

heading: I.1 SUFFICIENT CONDITIONS FOR ASSUMPTION G.9
text: In this section, we show that under a proper set of regularity conditions on the boundary set ∂G and the distribution of noises Zn, Assumption G.9 will hold for (Lebesgue) almost every b > 0. In particular, we stress that the C2 condition about manifold ∂G in Assumption G.3, as well as the
condition that measures Sj are absolutely continuous w.r.t. the spherical measure σ on Sd−1, are only used to prove that µ ( h−1(∂G) ) = 0 and will only be applied in this section.
The key of our argument is the following geometric observation regarding the intersection of C2−manifold ∂G and ∂B of some ball B. We stress that, as made evident by the proof, this lemma is essentially based on two assumptions: (I) As the boundary set of the connected bounded region G, ∂G is a closed set in Rd; (II) As a (d− 1)-dimensional manifold, ∂G is of class C2. Lemma I.1. Let σx,b be the spherical measure on the sphere of the open ball B(x, b) for any x ∈ Rd, b > 0. Under Assumptions G.1 and G.3, it holds for (Lebesgue) almost every b > 0 that
σx,b ( ∂G ∩ ∂B(x, b) ) = 0 ∀x ∈ Rd. (I.8)
Proof. The fact that ∂G is a subset of a separable metric space implies the existence of a countable atlas for this manifold. Therefore, we can find a sequence (Ui)i≥1 that are bounded open subsets of Rd−1 containing 0, a sequence (Vi)i≥1 that are open sets in ∂G (in the metric space induced by Euclidean distance), and a sequence of injective C2 mapping fi with fi : Rd−1 7→ Rd and fi(Ui) = Vi such that ∂G = ∪iVi. Next, we zoom in on a specific chart (Ui, Vi, fi) and observe the following facts. For any x ∈ Ui, y ∈ Vi with fi(x) = y, there exist some orthogonal matrix Qy ∈ Rd×d such that Qyn(y) = (0, 0, · · · , 0, 1)T where the vector field n(·) is the outer normal on ∂G. Besides, there is some vector ay ∈ Rd such that Qyy + ay = 0. Moreover, there exist an open set on Ũx in Rd−1 containing 0, an open set y ∈ Vy ⊂ Vi, an open set Ṽy =∆ {Qyw + ay : w ∈ Vy}, a C2 function g̃y : Ũx 7→ R satisfying g̃y(0) = 0 and
g̃y(w1, · · · , wd−1) = wd ∀w = (w1, · · · , wd) ∈ Ṽy. In other words, for any given y on this chart we simply rotate the chart to ensure that the tangent space at y after rotation is {(x1, · · · , xd−1, 0) : xi ∈ R ∀i ∈ [d − 1]}, and reparametrize the C2 diffeomorphism associated to this chart around y so that the coordinates are simply the projection onto the said tangent space. In this sense, the (rotated) manifold is also the graph of the C2 mapping gy . This allows us to define (for any y ∈ ∂G)
A(y) =∆ ( λ1 ( ∇2gy(0) ) , λ2 ( ∇2gy(0) ) , · · · , λd−1 ( ∇2gy(0) )) where, for any real symmetric (d− 1)× (d− 1) matrix A, λ1(A) ≥ λ2(A) ≥ · · · ≥ λd−1(A) are the ordered eigenvalues of the matrix, and∇2gy(·) is the Hessian of gy . It is worth noticing that A(·) is a continuous function (on ∂G) due to the manifold being of class C2. Restricting our discussion on some fixed chart (Ui, Vi, fi) for now, for any b > 0, let
Ai(b) =∆ {x ∈ Ui : for y = fi(x),∃j ∈ [d− 1] such that |λj ( ∇2gy(0) ) | = 1/b}.
Note that for any b > 0, the set Ai(b) is a closed set (hence Borel measurable) since the continuity of A(·) implies that (Ai(b))c is an open set on Ui. Furthermore, sincemLeb(Ui) <∞, there are at most countably many b > 0 such thatmLeb(Ai(b)) > 0. Given the countability of the atlas, we know that
B∗ =∆ {b > 0 : ∃i ∈ N s.t. mLeb(Ai(b)) > 0} contains at most countably many elements. In the rest of this proof, we show that eq. (I.8) holds for any b > 0 such that b /∈ B∗. Henceforth, we arbitrarily choose some b > 0 such that b /∈ B∗. We also arbitrarily choose some x ∈ Rd and let B = B(x, b). To facilitate the discussion, we introduce a concept that is closely related to the set Ai(b): let
AGi (b) = ∆ {y ∈ Vi : f−1i (y) ∈ Ai(b)} = {y ∈ Vi : A(y) = (b, b, · · · , b) or (−b,−b, · · · ,−b)}
and let AG(b) =∆ ∪i≥1AGi (b). Now consider the following decomposition of the sphere ∂B:
B2 =∆ ∂B ∩ AG(b), B1 =∆ ∂B\B2.
First, note that for any y ∈ B1, one of the following four cases has to occur:
• y /∈ ∂G;
• y ∈ ∂G and the vector y − x lies in Ty∂G, the tangent space of the manifold ∂G at y;
• y ∈ ∂G; the vector y − x is not in Ty∂G yet it is not orthogonal to Ty∂G either, i.e. y − x is not equal to c · n(y) for any c ∈ R where n(y) is the outer normal at y;
• y ∈ ∂G; the vectors y − x and n(y) are linearly dependent.
Our next goal is to show that, in any of these four cases, we can always find a set y ∈ Oy that is open on the sphere ∂B such that σx,b(Oy ∩ ∂G) = 0. Note that this is obviously true when y /∈ ∂G, since both ∂G and ∂B are closed sets in Rd. Now we consider the second case. If the vector y − x lies in Ty∂G, then after applying the affine transformation with orthogonal matrix Qy (recall that Qyn(y) = (0, · · · , 0, 1)), we have
Qy(y − x) ∈ QyTy∂G = {(w1, · · · , wd−1, 0) : wi ∈ R ∀i ∈ [d− 1]}.
Since Qyy + ay = 0, we now know that after the affine transformation, the center of the ball B moves to
Qyx+ ay = Qyy + ay +Qy(x− y) ∈ {(w1, · · · , wd−1, 0) : wi ∈ R ∀i ∈ [d− 1]}.
Without loss of generality, we can assume that Qyx+ ay = (b, 0, · · · , 0, 0). In other words, after the affine transformation, the ball becomes B̃y =∆ QyB + ay = B ( (b, 0, 0, · · · , 0), b ) . Moreover, for any
w = (w1, · · · , wd) ∈ ∂B̃y , we must have w1 ≥ 0 and
(w1 − b)2 + (w2)2 + · · ·+ (wd)2 = b2.
Meanwhile, from the definition of the mapping g̃y , one can see that there is an open set Uy around y such that for any w ∈ Uy with w ∈ ∂G ∩ ∂B (hence w̃ = Qyw + ay ∈ Ṽy ∈ ∂B̃y) such that
g̃y(w̃) 2 = b2 − (w̃1 − b)2 − w̃22 − · · · − w̃2d−1.
Let f̃(w1, · · · , wd−1) =∆ g̃2y(w1, · · · , wd−1) + w21 − 2bw1 + w22 + · · · + w2d−1. Now we now that f̃(0) = 0 and f̃(w̃) = 0 for any w̃ ∈ Ṽy ∈ ∂B̃y. Moreover, by definition we have ∇g̃y(0) = 0 so ∂ ∂w1
f̃(0) = −2b 6= 0. Due to implicit function theorem, we now know the existence of some open set U∗ in Rd containing 0, some C2 function g∗ : Rd−2 7→ R such that for any w ∈ Ṽy ∩ ∂B̃y ∩ U∗,
w1 = g ∗(w2, · · · , wd−1), wd = g̃y(w1, · · · , wd−1).
Therefore, within some open neighborhood V ∗y of such y, the set V ∗ y ∩ ∂B ∩ ∂G is a submanifold with dimension d− 2, so we must have
σx,b(V ∗ y ∩ ∂B ∩ ∂G) = 0.
Next, we consider the case where y ∈ ∂G ∩ B1 but y − x is neither in Ty∂G nor orthogonal to Ty∂G. Similar to the construction of the affine transformation with Qy, ay above, one can find an orthogonal matrix Q̃y and a vector ãy such that Q̃yy + ay = 0 and Q̃y(y − x) = (0, 0, · · · , 0, b). Let B̃ = B((0, 0, · · · ,−b), b). In other words, this time the rotation we constructed ensures that, after rotation, the vector between Q̃yy + ay = 0 (on the sphere ∂B̃) and the center Q̃yx+ ãy of the ball B̃ is aligned with the d−th axis. Moreover, there is an open set y ∈ V alty ⊂ Rd and a C2 function g̃alty : Rd−1 7→ R with∇g̃alty (0) 6= 0 such that for any ỹ ∈ Ṽ alty = ∆ Q̃yV alt y + ãy , we have
ỹd = g̃ alt y (ỹ1, · · · , ỹd−1).
By applying a change of coordinates if necessary (which can be achieved by multiplying another orthogonal matrix), we can assume without loss of generality that ∇g̃alty (0) = (0, 0, · · · , c) for some c 6= 0. Then for any w̃ ∈ V alty such that w ∈ ∂B∩ ∂G, let w = Q̃yw̃+ ãy and note that we must have
w21 + w 2 2 + · · ·+ w2d−1 + ( b+ g̃alty (w1, · · · , wd−1) )2 = b2.
In particular, g̃alty (w1, · · · , wd−1) = cwd−1+r(w) for someC2 function r with r(0) = 0,∇r(0) = 0. As a result, for function f̃(w) =∆ w21 + w 2 2 + · · · + w2d−1 + ( b + g̃alty (w1, · · · , wd−1) )2 . we have
∂ ∂wd−1 f̃(0) = 2bc 6= 0. Using implicit function theorem again, one can see the existence of some open set 0 ∈ V ∗y ∈ Rd and some C2 function g∗ : Rd−2 7→ R such that for any w̃ ∈ V ∗y ∩ ∂G ∩ ∂B, we have that (for w =∆ Q̃yw̃ + ãy)
wd−1 = g ∗(w1, · · · , wd−2), wd = g̃alty (w1, · · · , wd−1).
Again, we have established that within some open neighborhood V ∗y of such y, the set V ∗ y ∩ ∂B∩ ∂G is a submanifold with dimension d− 2, so we must have
σx,b(V ∗ y ∩ ∂B ∩ ∂G) = 0.
Lastly, consider the case where y ∈ ∂G ∩ B2 and the vectors y − x and n(y) are linearly dependent. In other words, the tangent space Ty∂G is also the tangent space Ty∂B. Since y /∈ B1, we know that g̃y(w) = 12w
TAyw + r1(w) where Ay is a real symmetric matrix with no eigenvalue equal to ±1/b and r1 is a C2 function with |r1(w)| = o(‖w‖2). On the other hand, for any w̃ in the open set Vy ⊂ ∂G, if we also have w̃ ∈ ∂B ∩ ∂G, then for w =∆ Qyw̃ + ay we have
wd = g̃y(w1, · · · , wd−1) = √ b2 − w21 − w22 − · · · − w2d−1 − b. (I.9)
Also, note that √ b2 − w21 − w22 − · · · − w2d−1 − b = − 1 2 (w1, · · · , wd−1) Id−1 b (w1, · · · , wd−1) T + r2(w1, · · · , wd−1) where r2 is also a C2 function with |r2(w)| = o(‖w‖2). Therefore, for any (w1, · · · , wd−1) satisfying the equation eq. (I.9), we have
1 2 (w1, · · · , wd−1)
( Ay −
Id−1 b
) (w1, · · · , wd−1)T = −r1(w1, · · · , wd−1) + r2(w1, · · · , wd−1).
However, for the real symmetric matrix Ay − Id−1b , note that none of its eigenvalue is equal to 0, implying the existence of some > 0 such that∣∣∣1
2 (w1, · · · , wd−1)
( Ay −
Id−1 b
) (w1, · · · , wd−1)T ∣∣∣ ≥ (w21 + · · ·+ w2d−1). For this fixed > 0, we can also find δ > 0 such that
| − r1(w1, · · · , wd−1) + r2(w1, · · · , wd−1)| ≤ 2 (w21 + · · ·+ w2d−1)
for any w21 + · · ·+w2d−1 < δ. As a result, the only solution to eq. (I.9) with w21 + · · ·+w2d−1 < δ is w1 = w2 = · · · , wd−1 = 0. In summary, we have shown that there exists some set y ∈ V ∗y that is open in Rd such that V ∗y ∩ ∂B ∩ ∂G = {y}. Collecting the results we have established so far, we now know that for any y ∈ B1 (recall that B1 is an open set on ∂B), there is an open set V ∗y containing y and satisfying σx,b(V ∗y ∩ ∂B ∩ ∂G) = 0. In particular, given the open cover ∪y∈B2V ∗y = B1, Lindelöf property then allows us to extract a countable open ∪i≥1V ∗yi = B1 cover and conclude that
σx,b(B1 ∩ ∂G) ≤ ∑ i≥1 σx,b(V ∗ yi ∩ ∂B ∩ ∂G) = 0.
Moving on, we evaluate σx,b(B2 ∩ ∂G). For any y ∈ ∂B, one of the four cases has to occur:
• y /∈ AG(b);
• y ∈ AG(b) and the vector y − x lies in Ty∂G, the tangent space of the manifold ∂G at y;
• y ∈ AG(b); the vector y−x is not in Ty∂G yet it is not orthogonal to Ty∂G either, i.e. y−x is not equal to c · n(y) for any c ∈ R where n(y) is the outer normal at y;
• y ∈ AG(b); the vectors y − x and n(y) are linearly dependent.
Again, we show that in any of these four cases, there is some set y ∈ V ∗y open in ∂B such that σx,b(AG(b)∩ V ∗y ) = 0. In the first case, the fact that AG(b) is closed on ∂G immediately implies the existence of some V ∗y such that AG(b) ∩ V ∗y = ∅. For the second and third case, this can be shown using exactly the same implicit function argument above. For the last case where y ∈ AG(b) ∩ ∂B and the vectors y − x and n(y) are linearly dependent, from y ∈ AG(b) ∩ ∂B we know that y ∈ Vi where (Ui, Vi, fi) is a chart of ∂G and Vi is open on ∂G. Moreover, recall the construction of open set y ∈ Vy ⊂ Vi at the beginning of the proof. It is worth noticing that Vi∩AG(b) = Vi∩AGi (b). Besides, due to the fact that the vectors y − x and n(y) are linearly dependent, we know that the tangent space Ty∂G is also the tangent space Ty∂B. Therefore, by definition of g̃y and Qy, ay , we know that Qyy + ay = 0, and under the affine transformation, the ball becomes B((0, 0, · · · ,±b), b). Without loss of generality, we assume it is B̃ =∆ B((0, 0, · · · , b), b). Moreover, since b /∈ B∗, we have that mLeb({w ∈ Ũy : x ∈ Ai(b)}) = 0 where, as defined at the beginning of the proof, Ũy is the domain of the C2 mapping g̃y, Vy is the image of the mapping, and Ṽy =
∆ {Qyw + ay : w ∈ Vy} is the image of Vy under the affine transformation. Therefore, for any w̃ ∈ AGi (b)∩ ∂B, let w = Qyw̃+ ay and we must have
wd = g̃y(w1, · · · , wd−1) = −b+ √ b2 − w21 − w22 − · · · − w2d−1,
(w1, · · · , wd−1) ∈ {w ∈ Ũy : x ∈ Ai(b)}.
Now let V ∗y = ∆
{ QTy (v − ay) : v = ( w1, · · · , wd−1,−b +√
b2 − w21 − w22 − · · · − w2d−1 ) for some w ∈ Ũx } and note that y ∈ V ∗y is an open set on
∂B. (Specifically, note that we simply identify an open set on the transformed sphere ∂B̃y , and then perform the inverse transformation to move the set back to the original sphere ∂B.) Then it follows immediately frommLeb({w ∈ Ũy : x ∈ Ai(b)}) = 0 that σx,b ( AG(b) ∩ V ∗y ) = 0.
In summary, for any y ∈ ∂B, we can find a set y ∈ V ∗y open on ∂B such that σx,b ( AG(b)∩V ∗y ) = 0. Lastly, by applying Lindelöf property again, we extract a countable open cover ∪i≥1V ∗yi = B2 cover and conclude that
σx,b(∂B ∩ AG(b)) ≤ ∑ i≥1 σx,b ( AG(b) ∩ V ∗yi ) = 0
and this concludes the proof.
As a result of Lemma I.1, the following Lemma is essentially built upon three assumptions/facts: (I) As a boundary set, ∂G is a closed set in Rd; (II) As a (d− 1)-dimensional manifold, ∂G is of class C2; (III) The measures Sj in Assumption G.6 are absolutely continuous w.r.t. the spherical measure σ. Lemma I.2. Under Assumptions G.1, G.3 and G.6, it holds for (Lebesgue) almost every b > 0 that
µ ( h−1 ( ∂G )) = 0.
Proof. Fix some b > 0 satisfying the conditions in Lemma I.1. Recall that µ = ∑
j∈j(i∗) µj (see eq. (H.40)). It suffices to show that µj ( h−1 ( ∂G ))
= 0 for some fixed j ∈ j(i∗). In particular, observe that
µj
( h−1 ( ∂G ))
= ∫ ti+1>0, θi∈Sd−1, ri>0 ∀i∈[k∗−1]
· (∫ θk∗∈Sd−1,rk∗>0 1 { h∗(r1, · · · , rk∗−1, θ1, · · · , θk∗−1, t2, · · · , tk∗) + ϕb(rk∗θk∗) ∈ ∂G } · Sjk∗ (dθk∗)ναjk∗ (drk∗) ) · k∗−1∏ i=1 ναji (dri)× Sji(dθi)×mLeb(dti+1) (I.10)
where the function h∗ is defined as
h∗(r1, · · · , rk∗−1, θ1, · · · , θk∗−1, t2, · · · , tk∗)
=∆ x̃ ( k∗−1∑ i=2 ti,0; (0, t2, · · · , tk∗−1), (r1θ1, r2θ2, · · · , rk∗−1θk∗−1) )
Let z =∆ h∗(r1, · · · , rk∗−1, θ1, · · · , θk∗−1, t2, · · · , tk∗). Now by separating the two cases based on whether the truncation operator takes effect or not, we have∫
θk∗∈Sd−1,rk∗>0 1 { h∗(r1, · · · , rk∗−1, θ1, · · · , θk∗−1, t2, · · · , tk∗) + ϕb(rk∗θk∗) ∈ ∂G } · Sjk∗ (dθk∗)ναjk∗ (drk∗)
= ∫ θk∗∈Sd−1,rk∗∈(0,b) 1 { z + rk∗θk∗ ∈ ∂G } Sjk∗ (dθk∗)ναjk∗ (drk ∗) (I.11)
+ ∫ θk∗∈Sd−1 1 { z + bθk∗ ∈ ∂G } Sjk∗ (dθk∗) · ∫ rk∗>b ναjk∗ (drk ∗). (I.12)
For term eq. (I.11), note that it is equal to ∫ 1 { T−1(rk∗ , θk∗) ∈ (−z+∂G)∩∂B(0, b) } Sjk∗ (dθk∗)× ναjk∗ (drk ∗) where T−1(r, θ) = rθ is the inverse of the polar coordinate transform. Furthermore, since the set (−z+ ∂G)∩ ∂B(0, b) is either empty or is a (d− 1)-dimensional C2 submanifold (w.r.t. B(0, b) when viewed as a d−dimensional manifold). In other words, it has zero mass undermdLeb. For the measure ν∗ =∆ T−1 ◦ (Sjk∗ × ναjk∗ ), due to Sj being absolutely continuous w.r.t. σ, it is easy to see that ν∗ is absolutely continuous w.r.t. mdLeb. Therefore, we must have
ν∗((−z + ∂G) ∩ ∂B(0, b)) = 0,
implying that the integral in eq. (I.11) = 0. On the other hand, for term eq. (I.12), we know that∫ rk∗>b
ναjk∗ (drk ∗) = 1/b1+αjk∗ <∞. Besides,∫
θk∗∈Sd−1 1 { z + bθk∗ ∈ ∂G } Sjk∗ (dθk∗) = ∫ Sd−1 1{z + bθ ∈ ∂G ∩ ∂B(z, b)}Sjk∗ (dθ).
Then it follows immediately from Lemma I.1 and Sj being absolutely continuous w.r.t. the spherical measure σ that the integral in term eq. (I.12) is equal to 0. In summary, we have shown that∫
θk∗∈Sd−1,rk∗>0 1 { h∗(r1, · · · , rk∗−1, θ1, · · · , θk∗−1, t2, · · · , tk∗) + ϕb(rk∗θk∗) ∈ ∂G } · Sjk∗ (dθk∗)ναjk∗ (drk∗) = 0
for any (r1, · · · , rk∗−1, θ1, · · · , θk∗−1, t2, · · · , tk∗). Plug this result back into eq. (I.10) and we conclude the proof.

heading: J NOTATIONS
text: Table J.1 lists the notations used in Section E.
Table J.1: Summary of notations frequently used in Section E
[k] {1, 2, . . . , k} η Learning rate (gradient descent step size) b Truncation threshold of stochastic gradient An accuracy parameter; typically used to denote an −neighborhood of si,mi δ A threshold parameter used to define large noises ̄ A constant defined for eq. (E.28)-eq. (E.29). Since ̄ < 0, in eq. (E.8) the claim holds
for |x− y| < ̄. Note that the value of the constant ̄ does not vary with our choice of η, , δ.
M Upper bound of |f ′| and |f ′′| eq. (E.10) L Radius of training domain eq. (E.10) Ω The open interval (s−, s+); a simplified notation for Ωi ϕ, ϕc ϕc(w) , ϕ(w, c) =
∆ (w ∧ c) ∨ (−c) truncation operator at level c > 0 Z≤δ,ηn Zn1{η|Zn| ≤ δ} “small” noise eq. (E.20) Z>δ,ηn Zn1{η|Zn| > δ} “large” noise eq. (E.21) T ηj (δ) min{n > T η j−1(δ) : η|Zn| > δ} arrival time of j-th large noise eq. (E.22) W ηj (δ) ZTηj (δ) size of j-th large noise eq. (E.23) Xηn(x) X η n+1(x) = ϕL ( Xηn(x)− ϕb ( η(f ′(Xηn(x))− Zn+1) )) , Xη0 (x) = x SGD yηn(x) yηn(x) = y η n−1(x)− ηf ′(y η n−1(x)), y η 0(x) = x GD
Y ηn (x) yηn(x) perturbed by large noises (Tη(δ),Wη(δ)) GD + large jump ỹηn(x; t,w) yηn(x) perturbed by noise vector (t,w) perturbed GD xη(t, x) dxη(t;x) = −ηf ′ ( xη(t;x) ) dt, xη(0;x) = x ODE
x(t, x) x1(t, x) x̃η(t, x; t,w) xη(t, x) perturbed by noise vector (t,w) perturbed ODE A(n, η, , δ) { max
k∈[n∧(Tη1 (δ)−1)] η|Z1 + · · ·+ Zk| ≤
} . eq. (E.42)
r r =∆ min{−s−, s+}. Effective radius of the attraction field Ω. l∗ l∗ =∆ dr/be. The minimum number of jumps required to escape Ω when starting from its local minimum m = 0. h(w, t) A mapping defined as h(w, t) = x̃(tl∗ , 0; t,w). t̄, δ̄ Necessary conditions for h(w, t) to be outside of Ω eq. (E.35)-eq. (E.36) t̂( ) t̂( ) =∆ c1 log(1/ ). The quantity t̂( )/η provides an upper bound for the time it takes xη
to return to 2 −neighborhood of local minimum m = 0 when starting from somewhere −away from s−, s+. See eq. (E.37).
E( ) { (w, t) ⊆ Rl∗ × Rl ∗−1 + : h(w, t) /∈ [(s− − ) ∨ (−L), (s+ + ) ∧ L] }
p( , δ, η) The probability that, for t = ( T ηj (δ) − 1 )l∗ j=1 and w = ( ηW ηj (δ) )l∗ j=1
, we have (w, t) ∈ E( ) conditioning on {T η1 (δ) = 1}. Intuitively speaking, it characterizes the probability that the first l∗ large noises alone can drive the ODE out of the attraction field. Defined in eq. (E.66).
να The Borel measure on R with density
να(dx) = 1{x > 0} αp+ xα+1 + 1{x < 0} αp− |x|α+1
where p−, p+ are constants in Assumption 2 in the main paper. µ The product measure µ = (να)l ∗ × (Leb+)l ∗−1.
σ(η) min{n ≥ 0 : Xηn /∈ Ω}. first exit time
H(x) P(|Z1| > x) = x−αL(x) Treturn( , η) min{n ≥ 0 : Xηn(x) ∈ [−2 , 2 ]}

heading: K RESULTS ABOUT TAIL DISTRIBUTIONS OF NOISES IN OUR NUMERICAL EXPERIMENTS

heading: K.1 QQ PLOTS
text: QQ plots below clearly show that the tails in noise distribution are always much lighter than the Pareto distributions with alpha = 2 or even 10. In fact, the tail of noise distributions seem to be between that of lognormal and normal distributions, implying that it is lighter than any power-law distribution.
Pareto, α = 2 Pareto, α = 5 Pareto, α = 10 Lognormal Normal
Figure K.1: Ablation Study, Corrupted FMNIST & LeNet: At the beginning
Pareto, α = 2 Pareto, α = 5 Pareto, α = 10 Lognormal Normal
Figure K.2: Ablation Study, Corrupted FMNIST & LeNet: Half way through the training
Pareto, α = 2 Pareto, α = 5 Pareto, α = 10 Lognormal Normal
Figure K.3: Ablation Study, Corrupted FMNIST & LeNet: At the end of training
Pareto, α = 2 Pareto, α = 5 Pareto, α = 10 Lognormal Normal
Figure K.4: Ablation Study, SVHN & VGG11: At the beginning
Pareto, α = 2 Pareto, α = 5 Pareto, α = 10 Lognormal Normal
Figure K.5: Ablation Study, SVHN & VGG11: Half way through the training
Pareto, α = 2 Pareto, α = 5 Pareto, α = 10 Lognormal Normal
Figure K.6: Ablation Study, SVHN & VGG11: At the end of training
Pareto, α = 2 Pareto, α = 5 Pareto, α = 10 Lognormal Normal
Figure K.7: Ablation Study, CIFAR10 & VGG11: At the beginning
Pareto, α = 2 Pareto, α = 5 Pareto, α = 10 Lognormal Normal
Figure K.8: Ablation Study, CIFAR10 & VGG11: Half way through the training
Pareto, α = 2 Pareto, α = 5 Pareto, α = 10 Lognormal Normal
Figure K.9: Ablation Study, CIFAR10 & VGG11: At the end of training
Pareto, α = 2 Pareto, α = 5 Pareto, α = 10 Lognormal Normal
Figure K.10: Data Augmentation, CIFAR10 & VGG11: At the beginning
Pareto, α = 2 Pareto, α = 5 Pareto, α = 10 Lognormal Normal
Figure K.11: Data Augmentation, CIFAR10 & VGG11: Half way through the training
Pareto, α = 2 Pareto, α = 5 Pareto, α = 10 Lognormal Normal
Figure K.12: Data Augmentation, CIFAR10 & VGG11: At the end of training
Pareto, α = 2 Pareto, α = 5 Pareto, α = 10 Lognormal Normal
Figure K.13: Data Augmentation, CIFAR100 & VGG16: At the beginning
Pareto, α = 2 Pareto, α = 5 Pareto, α = 10 Lognormal Normal
Figure K.14: Data Augmentation, CIFAR100 & VGG16: Halfway through the training
Pareto, α = 2 Pareto, α = 5 Pareto, α = 10 Lognormal Normal
Figure K.15: Data Augmentation, CIFAR100 & VGG16: At the end of training

heading: K.2 EMPIRICAL MEAN RESIDUAL LIFE (EMRL) PLOTS
text: It is well known that the mean residual life blows up to infinity if and only if the distribution is heavy-tailed (more precisely, long-tailed). However, from the figures below, one can see that none of the EMRL exhibits such a pattern in any case tested in our experiments. Instead, we see clear downward trends, which strongly suggests light tails in all cases tested.
Beginning Middle End
Figure K.16: Plots of empirical mean residual life for noises in FMNIST&LeNet Task throughout training
Beginning Middle End
Figure K.17: Plots of empirical mean residual life for noises in SVHN&VGG 11 Task throughout training
Beginning Middle End
Figure K.18: Plots of empirical mean residual life for noises in CIFAR 10&VGG 11 Task throughout training
Beginning Middle End
Figure K.19: Plots of empirical mean residual life for noises in dataAug, CIFAR 10&VGG 11 Task throughout training
Beginning Middle End
Figure K.20: Plots of empirical mean residual life for noises in dataAug, CIFAR 100&VGG 11 Task throughout training

heading: K.3 HILL PLOTS
text: In the hill plots below, the estimated power-law indices using only the top 1% of samples (on the left hand side of the dashed lines) stay well above 10 for the most part and almost never drop below 2. This strongly suggests that even if the gradient noises are from a heavy-tailed distribution, it is likely to have a very high power law index (implying relatively lighter tails), and hence, we cannot expect to observe a prominent heavy-tailed behavior from them.
Beginning Middle End
Figure K.21: altHill Plots for noises in FMNIST&LeNet Task throughout training. Dashed Red Line: Estimation based on the largest 1% data
Beginning Middle End
Figure K.22: altHill Plots for noises in SVHN&VGG 11 Task throughout training. Dashed Red Line: Estimation based on the largest 1% data
Beginning Middle End
Figure K.23: altHill Plots for noises in CIFAR 10&VGG 11 Task throughout training. Dashed Red Line: Estimation based on the largest 1% data
Beginning Middle End
Figure K.24: altHill Plots for noises in Data Augmentation, CIFAR 10&VGG 11 Task throughout training. Dashed Red Line: Estimation based on the largest 1% data
Beginning Middle End
Figure K.25: altHill Plots for noises in Data Augmentation, CIFAR 100&VGG 16 Task throughout training. Dashed Red Line: Estimation based on the largest 1% data
Table K.1: Power-law Indices Estimation throughout the Training, using PLFIT. All the estimations are at least 5 for all cases tested in our experiments, and most of the times the estimation is above 10. This means that even under the assumption that the gradient noises were from a heavy-tailed distribution, they should have much lighter tails than any α-stable distribution (which requires α < 2) or the heavy-tailed noises we injected during tail inflation experiments (α = 1.4).
Task Beginning Middle End
FMNIST, LeNet 14.3 14.2 16.5 SVHN, VGG11 5.0 5.2 12.5 CIFAR10, VGG11 9.2 6.6 7.0 dataAug, CIFAR10, VGG11 16.2 16.2 8.5 dataAug, CIFAR100, VGG16 35.1 14.4 5.35

Review summaries:
The authors analyse the behavior of SGD under gradient clipping. Their analysis in the univariate case shows that gradient clipping in the heavy-tailed gradient noise (almost) eliminates the algorithm's tendency to stay at sharp minima. The authors support their analysis with synthetic experiments. The authors then conduct experiments on real data where they add heavy tailed noise to the gradient, and clip it afterwards.

The paper studies gradient descent with injected power-law tail noise. The work shows that in the infinitesimal learning rate regime, the heavy tail noise can cause GD to not to converge to sharp minima.   Based on this theory, the paper proposes a technique to inject noise to GD to help training 

This paper study the long-time behavior of heavy-tailed SGD with gradient clipping. It is found that gradient clipping is crucial for heavy-tailed SGD to avoid sharp minima. The basic intuition is that the clipping operation reduces the distance moved by each SGD update. Therefore, for minima narrow than the threshold, the clipping does not change the first exit time. However, for wide minima, SGD is slowed down and takes more time to escape. Consequently, it is more likely that SGD locates in wide minima.