SlowGuess/NeurIPS_2023_OCR huggingface dataset:
**Deviation from gradient flow.** The difference with gradient flow is directly associated with the quantity  $\sum_{k} q(\gamma_{k} \nabla \mathcal{L}_{\mathcal{B}_{k}}(\beta_{k}))$ . Also, as the (stochastic) gradients converge to 0 and  $q(x) \stackrel{x \to 0}{\sim} x^{2}$ , one should think of this sum as roughly being  $\sum_{k} \nabla \mathcal{L}_{\mathcal{B}_{k}}(\beta_{k})^{2}$ : the larger this sum, the more the recovered solution differs from that of gradient flow. The full picture of how large step sizes and stochasticity impact the generalisation properties of  $\beta_{\infty}^{\star}$  and the recovery of minimum  $\ell_{1}$ -norm solution is nuanced as clearly seen in Fig. 1.
3.3 Convergence of the iterates
Theorem 1 provides the implicit minimisation problem but says nothing about the convergence of the iterates. Here we show under very reasonable assumptions on the stepsizes that the iterates indeed
converge towards a global optimum. Note that since the loss  $F$  is non-convex, such a convergence result is non-trivial and requires an involved analysis.
Theorem 2 (Convergence of the iterates). Let  $(u_{k},v_{k})_{k\geqslant 0}$  follow the mini-batch SGD recursion (3) initialised at  $u_0 = \sqrt{2}\alpha \in \mathbb{R}_{>0}^d$  and  $v_{0} = 0$ , and let  $(\beta_{k})_{k\geqslant 0} = (u_{k}\odot v_{k})_{k\geqslant 0}$ . Recall the "smoothness" parameter  $L$  on the minibatch loss defined in the notations. There exist  $B > 0$  verifying  $B = \tilde{\mathcal{O}} (\min_{\beta^{\star}\in S}\| \beta^{\star}\|_{\infty})$  and a numerical constant  $c > 0$  such that for stepsizes satisfying  $\gamma_k\leqslant \frac{c}{LB}$ , the iterates  $(\beta_{k})_{k\geqslant 0}$  converge almost surely to the interpolator  $\beta_{\infty}^{\star}$  solution of Eq. (5).
In fact, we can be more precise by showing an exponential rate of convergence of the losses as well as characterise the rate of convergence of the iterates as follows.
Proposition 1 (Quantitative convergence rates). For a uniform initialisation  $\alpha = \alpha \mathbf{1}$  and under the assumptions of Theorem 2, we have:
$$
\mathbb {E} \left[ \mathcal {L} \left(\beta_ {k}\right) \right] \leqslant \left(1 - \frac {1}{2} \gamma \alpha^ {2} \lambda_ {b}\right) ^ {k} \mathcal {L} \left(\beta_ {0}\right) \quad a n d \quad \mathbb {E} \left[ \left\| \beta_ {k} - \beta_ {\alpha_ {k}} ^ {\star} \right\| ^ {2} \right] \leqslant C \left(1 - \frac {1}{2} \gamma \alpha^ {2} \lambda_ {b}\right) ^ {k},
$$
where  $\lambda_{b} > 0$  is the largest value such that  $\lambda_{b}H \preceq \mathbb{E}_{\mathcal{B}}[H_{\mathcal{B}}]$ ,  $C = 2B(\alpha^{2}\lambda_{\min}^{+})^{-1}\left(1 + (4B\lambda_{\max})(\alpha^{2}\lambda_{\min}^{+})^{-1}\right)\mathcal{L}(\beta_{0})$  and  $\lambda_{\min}^{+}, \lambda_{\max} > 0$  are respectively the smallest non-null and the largest eigenvalues of  $H$ , and  $\beta_{\alpha_k}^{\star}$  is the interpolator that minimises the perturbed hypentropy  $h_k$  of parameter  $\alpha_k$ , as defined in Eq. (7) in the next subsection.
The convergence of the losses is proved directly using the time-varying mirror structure that we exhibit in the next subsection, the convergence of the iterates is proved by studying the curvature of the mirror maps on a small neighborhood around the affine interpolation space.
3.4 Sketch of proof through a time varying mirror descent
As in the continuous-time framework, our results heavily rely on showing that the iterates  $(\beta_k)_k$  follow a mirror descent recursion with time-varying potentials on the convex loss  $\mathcal{L}(\beta)$ . To show this, we first define the following quantities:
$$
\boldsymbol {\alpha} _ {k} ^ {2} := \boldsymbol {\alpha} _ {+, k} \odot \boldsymbol {\alpha} _ {-, k} \quad \text {a n d} \quad \phi_ {k} := \frac {1}{2} \operatorname {a r c s i n h} \left(\frac {\boldsymbol {\alpha} _ {+, k} ^ {2} - \boldsymbol {\alpha} _ {- , k} ^ {2}}{2 \boldsymbol {\alpha} _ {k} ^ {2}}\right) \in \mathbb {R} ^ {d},
$$
where  $\alpha_{\pm ,k}\coloneqq \alpha \exp \left(-\frac{1}{2}\sum_{i = 0}^{k - 1}q_{\pm}\bigl {(}\gamma_{\ell}\nabla \mathcal{L}_{\mathcal{B}_{\ell}}(\beta_{\ell})\bigr)\right)\in \mathbb{R}^{d}$ . Finally for  $k\geqslant 0$ , we define the potentials  $(h_k:\mathbb{R}^d\to \mathbb{R})_{k\geqslant 0}$  as:
$$
h _ {k} (\beta) = \psi_ {\boldsymbol {\alpha} _ {k}} (\beta) - \langle \phi_ {k}, \beta \rangle . \tag {7}
$$
Where  $\psi_{\alpha_k}$  is the hyperbolic entropy function defined Eq. (4). Now that all the relevant quantities are defined, we can state the following proposition which explicits the time-varying stochastic mirror descent.
Proposition 2. The iterates  $(\beta_{k} = u_{k}\odot v_{k})_{k\geqslant 0}$  from Eq. (3) satisfy the Stochastic Mirror Descent recursion with varying potentials  $(h_k)_k$  ..
$$
\nabla h _ {k + 1} \left(\beta_ {k + 1}\right) = \nabla h _ {k} \left(\beta_ {k}\right) - \gamma_ {k} \nabla \mathcal {L} _ {\mathcal {B} _ {k}} \left(\beta_ {k}\right),
$$
where  $h_k: \mathbb{R}^d \to \mathbb{R}$  for  $k \geqslant 0$  are defined Eq. (7). Since  $\nabla h_0(\beta_0) = 0$  we have:
$$
\nabla h _ {k} \left(\beta_ {k}\right) \in \operatorname {s p a n} \left(x _ {1}, \dots , x _ {n}\right). \tag {8}
$$
Theorem 1 and 2 and Proposition 1 follow from this key proposition: by suitably modifying classical convex optimization techniques to account for the time-varying potentials, we can prove the convergence of the iterates towards an interpolator  $\beta_{\infty}^{\star}$  along with that of the relevant quantities  $\alpha_{\pm ,k}$ ,  $\alpha_{k}$  and  $\phi_{k}$ . The implicit regularisation problem then directly follows from: (1) the limit condition  $\nabla h_{\infty}(\beta_{\infty}) \in \operatorname{Span}(x_1, \ldots, x_n)$  as seen from Eq. (8) and (2) the interpolation condition  $X\beta_{\infty}^{\star} = y$ . Indeed, these two conditions exactly correspond to the KKT conditions of the convex problem Eq. (5).
4 Analysis of the impact of the stepsize and stochasticity on  $\alpha_{\infty}$
In this section, we analyse the effects of large step sizes and stochasticity on the implicit bias of (S)GD. We focus on how these factors influence the effective initialisation  $\alpha_{\infty}$ , which plays a key role as shown in Theorem 1. From its definition in Eq. (6), we see that  $\alpha_{\infty}$  is a function of the vector  $\sum_{k} q(\gamma_k \nabla \mathcal{L}_{\mathcal{B}_k}(\beta_k))$ . We henceforth call this quantity the gain vector. For simplicity of the discussions, from now on, we consider constant step sizes  $\gamma_k = \gamma$  for all  $k \geqslant 0$  and a uniform initialisation of the weights  $\alpha = \alpha \mathbf{1}$  with  $\alpha > 0$ . We can then write the gain vector as:
$$
\operatorname {G a i n} _ {\gamma} := \ln \left(\frac {\boldsymbol {\alpha} ^ {2}}{\boldsymbol {\alpha} _ {\infty} ^ {2}}\right) = \sum_ {k} q (\gamma \nabla \mathcal {L} _ {\mathcal {B} _ {k}} (\beta_ {k})) \in \mathbb {R} ^ {d}.
$$
Following our discussion in section 3.1 on the scale and the shape of  $\alpha_{\infty}$ , we recall the link between the scale and shape of  $\mathrm{Gain}_{\gamma}$  and the recovered solution:
1. The scale of  $\mathrm{Gain}_{\gamma}$ , i.e. the magnitude of  $\| \mathrm{Gain}_{\gamma} \|_1$  indicates how much the implicit bias of (S)GD differs from that of gradient flow:  $\| \mathrm{Gain}_{\gamma} \|_1 \sim 0$  implies that  $\alpha_{\infty} \sim \alpha$  and therefore the recovered solution is close to that of gradient flow. On the contrary,  $\| \mathrm{Gain}_{\gamma} \|_1 \gg \ln(1 / \alpha)$  implies that  $\alpha_{\infty}$  has effective scale much smaller than  $\alpha$  thereby changing the implicit regularisation Eq. (5).
2. The shape of  $\mathrm{Gain}_{\gamma}$  indicates which coordinates of  $\beta$  in the associated minimum weighted  $\ell_{1}$  problem are most penalised. First recall from Section 3.1 that a uniformly large  $\mathrm{Gain}_{\gamma}$  leads to  $\psi_{\alpha_{\infty}}$  being closer to the  $\ell_{1}$ -norm. However, with small weight initialisation  $\alpha \rightarrow 0$ , we have,
$$
\psi_ {\alpha_ {\infty}} (\beta) \sim \ln \left(\frac {1}{\alpha}\right) \| \beta \| _ {1} + \sum_ {i = 1} ^ {d} \operatorname {G a i n} _ {\gamma} (\mathrm {i}) | \beta_ {\mathrm {i}} |, \tag {9}
$$
In this case, having a heterogeneously large vector  $\mathrm{Gain}_{\gamma}$  leads to a weighted  $\ell_{1}$  norm as the effective implicit regularisation, where the coordinates of  $\beta$  corresponding to the largest entries of  $\mathrm{Gain}_{\gamma}$  are less likely to be recovered.
4.1 The scale of  $\mathrm{Gain}_{\gamma}$  is increasing with the stepsize
The following proposition highlights the dependencies of the scale of the gain  $\| \mathrm{Gain}_{\gamma}\| _1$  in terms of various problem constants.
Proposition 3. Let  $\Lambda_b, \lambda_b > 0^3$  be the largest and smallest values, respectively, such that  $\lambda_b H \preceq \mathbb{E}_{\mathcal{B}}[H_{\mathcal{B}}^2] \preceq \Lambda_b H$ . For any stepsize  $\gamma > 0$  satisfying  $\gamma \leqslant \frac{c}{BL}$  (as in Theorem 2), initialisation  $\alpha \mathbf{1}$  and batch size  $b \in [n]$ , the magnitude of the gain satisfies:
$$
\lambda_ {b} \gamma^ {2} \sum_ {k} \mathbb {E L} \left(\beta_ {k}\right) \leqslant \mathbb {E} \left[ \| \operatorname {G a i n} _ {\gamma} \| _ {1} \right] \leqslant 2 \Lambda_ {b} \gamma^ {2} \sum_ {k} \mathbb {E L} \left(\beta_ {k}\right), \tag {10}
$$
where the expectation is over a uniform and independent sampling of the batches  $(\mathcal{B}_k)_{k\geqslant 0}$ .
The slower the training, the larger the gain. Eq. (10) shows that the slower the training loss converges to 0, the larger the sum of the loss and therefore the larger the scale of  $\mathrm{Gain}_{\gamma}$ . This means that the (S)GD trajectory deviates from that of gradient flow if the stepsize and/or noise slows down the training. This supports observations previously made from stochastic gradient flow [48] analysis.
The bigger the stepsize, the larger the gain. The effect of the stepsize on the magnitude of the gain is not directly visible in Eq. (10) because a larger stepsize tends to speed up the training. For stepsize  $0 < \gamma \leqslant \gamma_{\max} = \frac{c}{BL}$  as in Theorem 2 we have that (see Appendix G.1):
$$
\sum_ {k} \gamma^ {2} \mathcal {L} \left(\beta_ {k}\right) = \Theta \left(\gamma \ln \left(\frac {1}{\alpha}\right) \left\| \beta_ {\ell_ {1}} ^ {\star} \right\| _ {1}\right). \tag {11}
$$
Eq. (11) clearly shows that increasing the stepsize boosts the magnitude  $\| \mathrm{Gain}_{\gamma}\|_{1}$  up until the limit of  $\gamma_{\mathrm{max}}$ . Therefore, the larger the stepsize the smaller is the effective scale of  $\alpha_{\infty}$ . In turn, larger gap between  $\alpha_{\infty}$  and  $\alpha$  leads to a larger deviation of (S)GD from the gradient flow.
$^3 \Lambda_b, \lambda_b > 0$  are data-dependent constants; for  $b = n$ , we have  $(\lambda_n, \Lambda_n) = (\lambda_{\min}^+(H), \lambda_{\max}(H))$  where  $\lambda_{\min}^+(H)$  is the smallest non-null eigenvalue of  $H$ ; for  $b = 1$ , we have  $\min_i \|x_i\|_2^2 \leqslant \lambda_1 \leqslant \Lambda_1 \leqslant \max_i \|x_i\|_2^2$ .
Figure 2: Left: the scale of  $\mathrm{Gain}_{\gamma}$  explodes as  $\gamma \rightarrow \tilde{\gamma}_{\mathrm{max}}$  for both GD and SGD. Right:  $\beta_{\mathrm{sparse}}^{\star}$  is fixed, we perform 100 runs of GD and SGD with different feature matrices, and we plot the  $d$  coordinates of  $\mathrm{Gain}_{\gamma}$  (for GD and SGD) on the  $x$ -axis (which is in log scale for better visualisation). The shape of  $\mathrm{Gain}_{\gamma}^{\mathrm{SGD}}$  is homogeneous whereas that of GD is heterogeneous with much higher magnitude on the support of  $\beta_{\mathrm{sparse}}^{\star}$ . The shape of  $\mathrm{Gain}_{\gamma}^{\mathrm{GD}}$  is proportional to the expected gradient at initialisation which is  $(\beta_{\mathrm{sparse}}^{\star})^{2}$ .
Large step sizes and Edge of Stability. The previous paragraph holds for step sizes smaller than  $\gamma_{\mathrm{max}}$  for which we can theoretically prove convergence. But what if we use even bigger step sizes? Let  $(\beta_k^\gamma)_k$  denote the iterates generated with stepsize  $\gamma$  and let us define  $\tilde{\gamma}_{\mathrm{max}} := \sup_{\gamma \geqslant 0} \{\gamma \text{ s.t. } \forall \gamma' \in (0, \gamma), \sum_k \mathcal{L}(\beta_k^{\gamma'}) < \infty\}$ , which corresponds to the largest step size such that the iterates still converge for a given problem (even if not provably so). From Proposition 3 we have that  $\gamma_{\mathrm{max}} \leqslant \tilde{\gamma}_{\mathrm{max}}$ . As we approach this upper bound on convergence  $\gamma \rightarrow \tilde{\gamma}_{\mathrm{max}}$ , the sum  $\sum_k \mathcal{L}(\beta_k^\gamma)$  diverges. For such large step sizes, the iterates of gradient descent tend to "bounce" and this regime is commonly referred to as the Edge of Stability. In this regime, the convergence of the loss can be made arbitrarily slow due to these bouncing effects. As a consequence, as seen through Eq. (10), the magnitude of  $\mathrm{Gain}_\gamma$  can be become arbitrarily big as observed in Fig. 2 (left). In this regime, the recovered solution tends to dramatically differ from the gradient flow solution, as seen in Fig. 1.
Impact of stochasticity and linear scaling rule. Assuming inputs  $x_{i}$  sampled from  $\mathcal{N}(0,\sigma^2 I_d)$  with  $\sigma^2 > 0$ , we obtain  $\mathbb{E}\left[\| \mathrm{Gain}_{\gamma}\| _1\right] = \Theta \left(\gamma \frac{\sigma^2d}{b}\ln \left(\frac{1}{\alpha}\right)\| \beta_{\ell_1}^\star \| _1\right)$ , w.h.p. over the dataset (see Appendix G.3, Proposition 17). The scale of  $\mathrm{Gain}_{\gamma}$  decreases with batch size and there exists a factor  $n$  between that of SGD and that of GD. Additionally, the magnitude of  $\mathrm{Gain}_{\gamma}$  depends on  $\frac{\gamma}{b}$ , resembling the linear scaling rule commonly used in deep learning [22].
By analysing the magnitude  $\| \mathrm{Gain}_{\gamma}\| _1$  , we have explained the distinct behavior of (S)GD with large stepsizes compared to gradient flow. However, our current analysis does not qualitatively distinguish the behavior between SGD and GD beyond the linear stepsize scaling rules, in contrast with Fig. 1. A deeper understanding of the shape of  $\mathrm{Gain}\gamma$  is needed to explain this disparity.
4.2 The shape of  $\mathrm{Gain}_{\gamma}$  explains the differences between GD and SGD
In this section, we restrict our presentation to single batch SGD  $(b = 1)$  and full batch GD  $(b = n)$ . When visualising the typical shape of  $\mathrm{Gain}_{\gamma}$  for large step sizes (see Fig. 2 - right), we note that GD and SGD behave very differently. For GD, the magnitude of  $\mathrm{Gain}_{\gamma}$  is higher for coordinates in the support of  $\beta_{\ell_1}^{\star}$  and thus these coordinates are adversely weighted in the asymptotic limit of  $\psi_{\alpha_{\infty}}$  (per (9)). This explains the distinction seed in Fig. 1, where GD in this regime has poor sparse recovery despite having a small scale of  $\alpha_{\infty}$ , as opposed to SGD that behaves well.
The shape of  $\mathrm{Gain}_{\gamma}$  is determined by the sum of the squared gradients  $\sum_{k} \nabla \mathcal{L}_{\mathcal{B}_k} (\beta_k)^2$ , and in particular by the degree of heterogeneity among the coordinates of this sum. Precisely analysing the sum over the whole trajectory of the iterates  $(\beta_k)_k$  is technically out of reach. However, we empirically observe for the trajectories shown in Fig. 2 that the shape is largely determined within the first few iterates as formalized in the observation below.
Observation 1.  $\sum_{k}\nabla \mathcal{L}_{\mathcal{B}_k}(\beta_k)^2\lesssim \mathbb{E}[\nabla \mathcal{L}_{\mathcal{B}_k}(\beta_0)^2 ]$
In the simple case of a Gaussian noiseless sparse recovery problem (where  $y_{i} = \langle \beta_{\mathrm{sparse}}^{\star}, x_{i} \rangle$  for some sparse vector  $\beta_{\mathrm{sparse}}^{\star}$ ), we can control these gradients for GD and SGD (Appendix G.4) as:
$$
\nabla \mathcal {L} \left(\beta_ {0}\right) ^ {2} = \left(\beta_ {\text {s p a r s e}} ^ {\star}\right) ^ {2} + \varepsilon , \text {f o r s o m e} \varepsilon \text {v e r i f y i n g} \| \varepsilon \| _ {\infty} <   <   \left\| \beta_ {\text {s p a r s e}} ^ {\star} \right\| _ {\infty} ^ {2}, \tag {12}
$$
$$
\mathbb {E} _ {i _ {0}} \left[ \nabla \mathcal {L} _ {i _ {0}} \left(\beta_ {0}\right) ^ {2} \right] = \Theta \left(\| \beta_ {\text {s p a r s e}} ^ {\star} \| _ {2} ^ {2} \mathbf {1}\right). \tag {13}
$$
The gradient of GD is heterogeneous. Since  $\beta_{\mathrm{sparse}}^{\star}$  is sparse by definition, we deduce from Eq. (25) that  $\nabla \mathcal{L}(\beta_0)$  is heterogeneous with larger values corresponding to the support of  $\beta_{\mathrm{sparse}}^{\star}$ . Along with observation 1, this means that  $\mathrm{Gain}_{\gamma}$  has much larger values on the support of  $\beta_{\mathrm{sparse}}^{\star}$ . The corresponding weighted  $\ell_1$ -norm therefore penalises the coordinates belonging to the support of  $\beta_{\mathrm{sparse}}^{\star}$ , which hinders the recovery of  $\beta_{\mathrm{sparse}}^{\star}$  (as explained in Example 1, Appendix D).
The stochastic gradient of SGD is homogeneous. On the contrary, from Eq. (26), we have that the initial stochastic gradients are homogeneous, leading to a weighted  $\ell_1$ -norm where the weights are roughly balanced. The corresponding weighted  $\ell_1$ -norm is therefore close to the uniform  $\ell_1$ -norm and the classical  $\ell_1$  recovery guarantees are expected.
Overall summary of the joint effects of the scale and shape. In summary we have the following trichotomy which fully explains Fig. 1:
1. for small stepsizes, the scale is small, and (S)GD solutions are close to that of gradient flow;
2. for large stepsizes the scale is significant and the recovered solutions differ from GF:
- for SGD the shape of  $\alpha_{\infty}$  is uniform, the associated norm is closer to the  $\ell_1$ -norm and the recovered solution is closer to the sparse solution;
- for GD, the shape is heterogeneous, the associated norm is weighted such that it hinders the recovery of the sparse solution.
In this last section, we relate heuristically these findings to the Edge of Stability phenomenon.
5 Edge of Stability: the neural point of view
In recent years it has been noticed that when training neural networks with 'large' step sizes at the limit of divergence, GD enters the Edge of Stability (EoS) regime. In this regime, as seen in Fig. 3, the iterates of GD 'bounce' / 'oscillate'. In this section, we come back to the point of view of the weights  $w_{k} = (u_{k}, v_{k}) \in \mathbb{R}^{2d}$  and make the connection between our previous results and the common understanding of the EoS phenomenon. The question we seek to answer is: in which case does GD enter the EoS regime, and if so, what are the consequences on the trajectory? Keep in mind that this section aims to provide insights rather than formal statements. We study the GD trajectory starting from a small initialisation  $\alpha = \alpha 1$  where  $\alpha \ll 1$  such that we can consider that gradient flow converges close to the sparse interpolator  $\beta_{\mathrm{sparse}}^{\star} = \beta_{w_{\mathrm{sparse}}^{\star}}$  corresponding to the weights  $w_{\mathrm{sparse}}^{\star} = (\sqrt{|\beta_{\mathrm{sparse}}^{\star}|}, \mathrm{sign}(\beta_{\mathrm{sparse}}^{\star})\sqrt{|\beta_{\mathrm{sparse}}^{\star}|})$  (see Lemma 1 in [49] for the mapping from the predictors to weights for gradient flow). The trajectory of GD as seen in Fig. 3 (left) can be decomposed into up to 3 phases.
First phase: gradient flow. The stepsize is appropriate for the local curvature (as seen in Fig. 3, lower right) around initialisation and the iterates of GD remain close to the trajectory of gradient flow (in black in Fig. 3). If the stepsize is such that  $\gamma < \frac{2}{\lambda_{\mathrm{max}}(\nabla^2F(w_{\mathrm{sparse}}^\star))}$ , then it is compatible with the local curvature and the iterates can converge: in this case GF and GD converge to the same point (as seen in Fig. 1 for small stepsizes). For larger  $\gamma > \frac{2}{\lambda_{\mathrm{max}}(\nabla^2F(w_{\mathrm{sparse}}^\star))}$  (as is the case for  $\gamma_{\mathrm{GD}}$  in Fig. 3, lower right), the iterates cannot converge to  $\beta_{\mathrm{sparse}}^\star$  and we enter the oscillating phase.
Second phase: oscillations. The iterates start oscillating. The gradient of  $F$  writes  $\nabla_{(u,v)}F(w)\sim$ $(\nabla \mathcal{L}(\beta)\odot v,\nabla \mathcal{L}(\beta)\odot u)$  and for  $w$  in the vicinity of  $w_{\mathrm{sparse}}^{\star}$  we have that  $u_{i}\approx v_{i}\approx 0$  for  $i\notin \operatorname {supp}(\beta_{\mathrm{sparse}}^{\star})$  . Therefore for  $w\sim w_{\mathrm{sparse}}^{\star}$  we have that  $\nabla_uF(w)_i\approx \nabla_vF(w)_i\approx 0$  for  $i\notin \operatorname {supp}(\beta_{\mathrm{sparse}}^{\star})$  and the gradients roughly belong to  $\operatorname {Span}(e_i,e_{i + d})_{i\in \operatorname {supp}(\beta_{\mathrm{sparse}}^{\star})}$  . This means
Figure 3: GD at the  $EoS$ . Left: For GD, the coordinates on the support of  $\beta_{\mathrm{sparse}}^{\star}$  oscillate and drift towards 0. Right, top: The GD train loses saturate before eventually converging. Bottom: GF converges towards a solution that has a high hessian maximum eigenvalue. GD cannot converge towards this solution because of its large stepsize: it therefore drifts towards a solution that has a curvature just below  $2 / \gamma$ .
that only the coordinates of the weights  $(u_i, v_i)$  for  $i \in \operatorname{supp}(\beta_{\mathrm{sparse}}^{\star})$  can oscillate and similarly for  $(\beta_i)_{i \in \operatorname{supp}(\beta_{\mathrm{sparse}}^{\star})}$  (as seen Fig. 3 left).
Last phase: convergence. Due to the oscillations, the iterates gradually drift towards a region of lower curvature (Fig. 3, lower right, the sharpness decreases) where they may (potentially) converge. Theorem 1 enables us to understand where they converge: the coordinates of  $\beta_{k}$  that have oscillated significantly along the trajectory belong to the support of  $\beta_{\mathrm{sparse}}^{\star}$ , and therefore  $\mathrm{Gain}_{\gamma}(\mathrm{i})$  becomes much larger for  $i \in \operatorname{supp}(\beta_{\mathrm{sparse}}^{\star})$  than for the other coordinates. Thus, the coordinates of the solution recovered in the  $EoS$  regime are heavily penalised on the support of the sparse solution. This is observed in Fig. 3 (left): the oscillations of  $(\beta_{i})_{i \in \operatorname{supp}(\beta_{\mathrm{sparse}}^{\star})}$  lead to a gradual shift of these coordinates towards 0, hindering an accurate recovery of the solution  $\beta_{\mathrm{sparse}}^{\star}$ .
SGD in the  $EoS$  regime. In contrast to the behavior of GD where the oscillations primarily occur on the non-sparse coordinates of ground truth sparse model, for SGD we see a different behavior in Fig. 6 (Appendix A). For stepsizes in the  $EoS$  regime, just below the non-convergence threshold: the fluctuation of the coordinates occurs evenly over all coordinates, leading to a uniform  $\alpha_{\infty}$ . These fluctuations are reminiscent of label-noise SGD [2], that have been shown to recover the sparse interpolator in diagonal linear networks [50].