SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(S)GD over Diagonal Linear Networks: Implicit Bias, Large Stepsizes and Edge of Stability
Mathieu Even*  
Inria - ENS Paris
Scott Pesme* EPFL
Suriya Gunasekar  
Microsoft Research
In this paper, we investigate the impact of stochasticity and large step sizes on the implicit regularisation of gradient descent (GD) and stochastic gradient descent (SGD) over 2-layer diagonal linear networks. We prove the convergence of GD and SGD with macroscopic step sizes in an overparametrised regression setting and provide a characterisation of their solution through an implicit regularisation problem. Our characterisation provides insights on how the choice of minibatch sizes and step sizes lead to qualitatively distinct behaviors in the solutions. Specifically, we show that for sparse regression learned with 2-layer diagonal linear networks, large step sizes consistently benefit SGD, whereas they can hinder the recovery of sparse solutions for GD. These effects are amplified for step sizes in a tight window just below the divergence threshold, known as the "edge of stability" regime.
The stochastic gradient descent algorithm (SGD) [51] is the foundational algorithm for almost all neural network training. Though a remarkably simple algorithm, it has led to many impressive empirical results and is a key driver of deep learning. However the performances of SGD are quite puzzling from a theoretical point of view as (1) its convergence is highly non-trivial and (2) there exist many global minimums for the training objective which generalise very poorly [66].
To explain this second point, the concept of implicit regularisation has emerged: if overfitting is harmless in many real-world prediction tasks, it must be because the optimisation process is implicitly favoring solutions that have good generalisation properties for the task. The canonical example is overparametrised linear regression with more trainable parameters than number of samples: although there are infinitely many solutions that fit the samples, GD and SGD explore only a small subspace of all the possible parameters. As a result, it can be shown that they implicitly converge to the closest solution in terms of the  $\ell_2$  distance, and this without explicit regularisation [66, 24].
Currently, most theoretical works on implicit regularisation have primarily focused on continuous time approximations of (S)GD where the impact of crucial hyperparameters such as the stepsize and the minibatch size are ignored. One such common simplification is to analyse gradient flow, which is a continuous time limit of GD and minibatch SGD with an infinitesimal stepsize. By definition, this analysis does not capture the effect of stepsize or stochasticity. Another approach is to approximate SGD by a stochastic gradient flow [60, 48], which tries to capture the noise and the stepsize using an appropriate stochastic differential equation. However, there are no theoretical guarantees that these results can be transferred to minibatch SGD as used in practice. This is a limitation in our understanding since the performances of most deep learning models are often sensitive to the choice of stepsize and minibatch size. The importance of stepsize and SGD minibatch size is common knowledge in practice and has also been systematically established in controlled experiments [36, 42, 20].
Figure 1: Noiseless sparse regression with a diagonal linear network using SGD and GD, with parameters initialized at the scale of  $\alpha = 0.1$  (Section 2). The test losses at convergence for various step sizes are plotted for GD and SGD. Small step sizes correspond to gradient flow (GF) performance. We see that increasing the step size improves the generalisation properties of SGD, but deteriorates that of GD. The dashed vertical lines at step sizes  $\tilde{\gamma}_{\mathrm{max}}^{\mathrm{SGD}}$  and  $\tilde{\gamma}_{\mathrm{max}}^{\mathrm{GD}}$  denote the largest step sizes for which SGD and GD, respectively, converge. See Section 2 for the precise experimental setting.
In this work, we aim to expand our understanding of the impact of stochasticity and stepsizes by analysing the (S)GD trajectory in 2-layer diagonal networks (DLNs). In Fig. 1, we show that even in our simple network, there are significant differences between the nature of the solutions recovered by SGD and GD at macroscopic step sizes. We discuss this behavior further in the later sections.
The 2-layer diagonal linear network which we consider is a simplified neural network that has received significant attention lately [61, 57, 26, 50]. Despite its simplicity, it surprisingly reveals training characteristics which are observed in much more complex architectures, such as the role of the initialisation [61], the role of noise [48, 50], or the emergence of saddle-to-saddle dynamics [6, 49]. It therefore serves as an ideal proxy model for gaining a deeper understanding of complex phenomenons such as the roles of stepsizes and of stochasticity as highlighted in this paper. We also point out that implicit bias and convergence for more complex architectures such as 2-layer ReLU networks, matrix multiplication are not yet fully understood, even for the simple gradient flow. Therefore studying the subtler effects of large stepsizes and stochasticity in these settings is currently out of reach.
1.1 Main results and paper organisation
The overparametrised regression setting and diagonal linear networks are introduced in Section 2. We formulate our theoretical results (Theorems 1 and 2) in Section 3: we prove that for macroscopic step sizes, gradient descent and stochastic gradient descent over 2-layer diagonal linear networks converge to a zero-training loss solution  $\beta_{\infty}^{\star}$ . We further provide a refined characterization of  $\beta_{\infty}^{\star}$  through a trajectory-dependent implicit regularisation problem, that captures the effects of hyperparameters of the algorithm, such as step sizes and batch sizes, in useful and analysable ways. In Section 4 we then leverage this crisp characterisation to explain the influence of crucial parameters such as the step size and batch-size on the recovered solution. Importantly our analysis shows a stark difference between the generalisation performances of GD and SGD for large step sizes, hence explaining the numerical results seen in Fig. 1 for the sparse regression setting. Finally, in Section 5, we use our results to shed new light on the Edge of Stability (EoS) phenomenon [14].
Implicit bias. The concept of implicit bias from optimization algorithm in neural networks has been studied extensively in the past few years, starting with early works of Telgarsky [55], Neyshabur et al. [45], Keskar et al. [36], Soudry et al. [53]. The theoretical results on implicit regularisation have been extended to multiplicative parametrisations [23, 25], linear networks [34], and homogeneous networks [40, 35, 13]. For regression loss on diagonal linear networks studied in this work, Woodworth et al. [61] demonstrate that the scale of the initialisation determines the type of solution obtained, with large initialisations yielding minimum  $\ell_2$  norm solutions—the neural tangent kernel regime [30] and small initialisation resulting in minimum  $\ell_1$  norm solutions—the rich regime [13]. The analysis relies on the link between gradient descent and mirror descent established by Ghai et al. [21] and further explored by Vaskevicius et al. [56], Wu and Rebeschini [62]. These works focus on full batch gradient, and often in the infinitesimal stepsize limit (gradient flow), leading to general insights and results that do not take into account the effects of stochasticity and large step sizes.
The effect of stochasticity in SGD on generalisation. The relationship between stochasticity in SGD and generalisation has been studied in various works [41, 29, 11, 38, 64]. Empirically, models generated by SGD exhibit better generalisation performance than those generated by GD [37, 31, 27].
Explanations related to the flatness of the minima picked by SGD have been proposed [28]. Label noise has been shown to influence the implicit bias of SGD [26, 8, 15, 50] by implicitly regularising the sharp minimisers. Recently, studying a stochastic gradient flow that models the noise of SGD in continuous time with Brownian diffusion, Pesme et al. [48] characterised for diagonal linear networks the limit of their stochastic process as the solution of an implicit regularisation problem. However similar explicit characterisation of the implicit bias remains unclear for SGD with large step sizes.
The effect of stepsizes in GD and SGD. Recent efforts to understand how the choice of stepsizes affects the learning process and the properties of the recovered solution suggest that larger stepsizes lead to the minimisation of some notion of flatness of the loss function [52, 37, 44, 33, 64, 43], backed by empirical evidences or stability analyses. Larger stepsizes have also been proven to be beneficial for specific architectures or problems: two-layer network [39], regression [63], kernel regression [7] or matrix factorisation [59]. For large stepsizes, it has been observed that GD enters an Edge of Stability (EoS) regime [32, 14], in which the iterates and the train loss oscillate before converging to a zero-training error solution; this phenomenon has then been studied on simple toy models [1, 67, 12, 16] for GD. Recently, [2] presented empirical evidence that large stepsizes can lead to loss stabilisation and towards simpler predictors.
2 Setup and preliminaries
Overparametrised linear regression. We consider a linear regression over inputs  $X = (x_{1},\ldots ,x_{n})\in (\mathbb{R}^{d})^{n}$  and outputs  $y = (y_{1},\dots ,y_{n})\in \mathbb{R}^{n}$ . We consider overparametrised problems where input dimension  $d$  is (much) larger than the number of samples  $n$ . In this case, there exists infinitely many linear predictors  $\beta^{\star}\in \mathbb{R}^{d}$  which perfectly fit the training set, i.e.,  $y_{i} = \langle \beta^{\star},x_{i}\rangle$  for all  $1\leqslant i\leqslant n$ . We call such vectors interpolating predictors or interpolators and we denote by  $S$  the set of all interpolators  $S = \{\beta^{\star}\in \mathbb{R}^{d}$  s.t.  $\langle \beta^{\star},x_i\rangle = y_i,\forall i\in [n]\}$ . Note that  $S$  is an affine space of dimension greater than  $d - n$  and equal to  $\beta^{\star} + \mathrm{span}(x_1,\ldots ,x_n)^{\perp}$  for any  $\beta^{\star}\in S$ . We consider the following quadratic loss:  $\mathcal{L}(\beta) = \frac{1}{2n}\sum_{i = 1}^{n}(\langle \beta ,x_i\rangle -y_i)^2$ , for  $\beta \in \mathbb{R}^d$ .
2-layer linear diagonal network. We parametrisse regression vectors  $\beta$  as functions  $\beta_w$  of trainable parameters  $w \in \mathbb{R}^p$ . Although the final prediction function  $x \mapsto \langle \beta_w, x \rangle$  is linear in the input  $x$ , the choice of the parametrisation drastically changes the solution recovered by the optimisation algorithm [25]. In the case of the linear parametrisation  $\beta_w = w$  many first-order methods (SGD, GD, with or without momentum) converge towards the same solution and the choice of stepsize does not impact the recovered solution beyond convergence. In an effort to better understand the effects of stochasticity and large stepsize, we consider the next simple parametrisation, that of a 2-layer diagonal linear neural network given by:
$$
\beta_ {w} = u \odot v \text {w h e r e} w = (u, v) \in \mathbb {R} ^ {2 d}. \tag {1}
$$
This parametrisation can be viewed as a simple neural network  $x \mapsto \langle u, \sigma(\mathrm{diag}(v)x) \rangle$  where the output weights are represented by  $u$ , the inner weights is the diagonal matrix  $\mathrm{diag}(v)$ , and the activation  $\sigma$  is the identity function. In this spirit, we refer to the entries of  $w = (u,v) \in \mathbb{R}^{2d}$  as the weights and to  $\beta := u \odot v \in \mathbb{R}^d$  as the prediction parameter. Despite the simplicity of the parametrisation (1), the loss function  $F$  over parameters  $w = (u,v) \in \mathbb{R}^{2d}$  is non-convex (and thus the corresponding optimization problem is challenging to analyse), and is given by:
$$
F (w) := \mathcal {L} (u \odot v) = \frac {1}{2 n} \sum_ {i = 1} ^ {n} \left(y _ {i} - \langle u \odot v, x _ {i} \rangle\right) ^ {2}. \tag {2}
$$
Mini-batch SGD. We minimise  $F$  using mini-batch SGD: let  $w_{0} = (u_{0}, v_{0})$  and for  $k \geqslant 0$ ,
$$
w _ {k + 1} = w _ {k} - \gamma_ {k} \nabla F _ {\mathcal {B} _ {k}} \left(w _ {k}\right), \quad \text {w h e r e} \quad F _ {\mathcal {B} _ {k}} (w) := \frac {1}{2 b} \sum_ {i \in \mathcal {B} _ {k}} \left(y _ {i} - \langle u \odot v, x _ {i} \rangle\right) ^ {2}, \tag {3}
$$
where  $\gamma_{k}$  are stepsizes,  $\mathcal{B}_k\subset [n]$  are mini-batches of  $b\in [n]$  distinct samples sampled uniformly and independently, and  $\nabla F_{\mathcal{B}_k}(w_k)$  are minibatch gradients of partial loss over  $\mathcal{B}_k$ ,  $F_{\mathcal{B}_k}(w)\coloneqq \mathcal{L}_{\mathcal{B}_k}(u\odot v)$  defined above. Classical SGD and full-batch GD are special cases with  $b = 1$  and  $b = n$ , respectively. For  $k\geqslant 0$ , we consider the successive prediction parameters  $\beta_{k}\coloneqq u_{k}\odot v_{k}$  built from the weights
$w_{k} = (u_{k},v_{k})$ . We analyse SGD initialised at  $u_0 = \sqrt{2}\alpha \in \mathbb{R}_{>0}^d$  and  $v_{0} = \mathbf{0}\in \mathbb{R}^{d}$ , resulting in  $\beta_0 = \mathbf{0}\in \mathbb{R}^d$  independently of the chosen weight initialisation  $\alpha^2$ .
Experimental details. We consider the noiseless sparse regression setting where  $(x_{i})_{i\in [n]}\sim$ $\mathcal{N}(0,I_d)$  and  $y_{i} = \langle \beta_{\ell_{1}}^{\star},x_{i}\rangle$  for some  $s$  -sparse vector  $\beta_{\ell_1}^\star$  . We perform (S)GD over the DLN with a uniform initialisation  $\alpha = \alpha \mathbf{1}\in \mathbb{R}^{d}$  where  $\alpha >0$  Fig. 1 and Fig. 2 (left) correspond to the setup  $(n,d,s,\alpha) = (20,30,3,0.1)$  , Fig. 2 (right) to  $(n,d,s,\alpha) = (50,100,4,0.1)$  and Fig. 3 to  $(n,d,s,\alpha) = (50,100,2,0.1)$
Notations. Let  $H := \nabla^2\mathcal{L} = \frac{1}{n}\sum_i x_i x_i^\top$  denote the Hessian of  $\mathcal{L}$ , and for a batch  $\mathcal{B} \subset [n]$  let  $H_{\mathcal{B}} := \nabla^2\mathcal{L}_{\mathcal{B}} = \frac{1}{|\mathcal{B}|}\sum_{i\in \mathcal{B}}x_ix_i^\top$  denote the Hessian of the partial loss over the batch  $\mathcal{B}$ . Let  $L$  denote the "smoothness" such that  $\forall \beta, \|H_{\mathcal{B}}\beta\|_2 \leqslant L\|\beta\|_2, \|H_{\mathcal{B}}\beta\|_\infty \leqslant L\|\beta\|_\infty$  for all batches  $\mathcal{B} \subset [n]$  of size  $b$ . A real function (e.g., log, exp) applied to a vector must be understood as element-wise application, and for vectors  $u, v \in \mathbb{R}^d$ ,  $u^2 = (u_i^2)_{i \in [d]}$ ,  $u \odot v = (u_iv_i)_{i \in [d]}$  and  $u / v = (u_i / v_i)_{i \in [d]}$ . We write 1, 0 for the constant vectors with coordinates 1 and 0 respectively. The Bregman divergence [9] of a differentiable convex function  $h: \mathbb{R}^d \to \mathbb{R}$  is defined as  $D_h(\beta_1, \beta_2) = h(\beta_1) - (h(\beta_2) + \langle\nabla h(\beta_2), \beta_1 - \beta_2\rangle)$ .
3 Implicit bias of SGD and GD
We start by recalling some known results on the implicit bias of gradient flow on diagonal linear networks before presenting our main theorems on characterising the (stochastic) gradient descent solutions (Theorem 1) as well as proving the convergence of the iterates (Theorem 2).
3.1 Warmup: gradient flow
We first review prior findings on gradient flow on diagonal linear neural networks. Woodworth et al. [61] show that the limit  $\beta_{\alpha}^{*}$  of the gradient flow  $\mathrm{d}w_{t} = -\nabla F(w_{t})\mathrm{d}t$  initialised at  $(u_0,v_0) = (\sqrt{2}\alpha ,0)$  is the solution of the minimal interpolation problem:
$$
\beta_ {\boldsymbol {\alpha}} ^ {*} = \underset {\beta^ {\star} \in \mathcal {S}} {\operatorname {a r g m i n}} \psi_ {\boldsymbol {\alpha}} \left(\beta^ {\star}\right), \quad \text {w h e r e} \quad \psi_ {\boldsymbol {\alpha}} (\beta) = \frac {1}{2} \sum_ {i = 1} ^ {d} \left(\beta_ {i} \operatorname {a r c s i n h} \left(\frac {\beta_ {i}}{\alpha_ {i} ^ {2}}\right) - \sqrt {\beta_ {i} ^ {2} + \alpha_ {i} ^ {4}} + \alpha_ {i} ^ {2}\right). \tag {4}
$$
The convex potential  $\psi_{\alpha}$  is the hyperbolic entropy function (or hypentropy) [21]. Depending on the structure of the vector  $\alpha$ , the generalisation properties of  $\beta_{\alpha}^{\star}$  highly vary. We point out the two main characteristics of  $\alpha$  that affect the behaviour of  $\psi_{\alpha}$  and therefore also the solution  $\beta_{\alpha}^{\star}$ .
1. The Scale of  $\alpha$ . For an initialisation vector  $\alpha$  we call the  $\ell_1$ -norm  $\| \alpha \| _1$  the scale of the initialisation. It is an important quantity affecting the properties of the recovered solution  $\beta_{\alpha}^{\star}$ . To see this let us consider a uniform initialisation of the form  $\alpha = \alpha 1$  for a scalar value  $\alpha >0$ . In this case the potential  $\psi_{\alpha}$  has the property of resembling the  $\ell_1$ -norm as the scale  $\alpha$  vanishes:  $\psi_{\alpha}\sim \ln (1 / \alpha)\| .\|_{1}$  as  $\alpha \rightarrow 0$ . Hence, a small initialisation results in a low  $\ell_1$ -norm solution which is known to induce sparse recovery guarantees [10]. This setting is often referred to as the "rich" regime [61]. In contrast, using a large initialisation scale leads to solutions with low  $\ell_2$ -norm:  $\psi_{\alpha}\sim \| .\|_{2}^{2} / (2\alpha^{2})$  as  $\alpha \to \infty$ , a setting known as the "kernel" or "lazy" regime. Overall, to retrieve the minimum  $\ell_1$ -norm solution, one should use a uniform initialisation with small scale  $\alpha$ , see Fig. 7 in Appendix D for an illustration and [61, Theorem 2] for a precise characterisation.
2. The Shape of  $\alpha$ . In addition to the scale of the initialisation  $\alpha$ , a lesser studied aspect is its "shape", which is a term we use to refer to the relative distribution of  $\{\alpha_i\}_i$  along the  $d$  coordinates [3]. It is a crucial property because having  $\alpha \rightarrow 0$  does not necessarily lead to the potential  $\psi_{\alpha}$  being close to the  $\ell_1$ -norm. Indeed, we have that  $\psi_{\alpha}(\beta) \stackrel{\alpha \rightarrow 0}{\sim} \sum_{i=1}^{d} \ln \left( \frac{1}{\alpha_i} \right) |\beta_i|$  (see Appendix D), therefore if the vector  $\ln(1 / \alpha)$  has entries changing at different rates, then  $\psi_{\alpha}(\beta)$  is a weighted  $\ell_1$ -norm. In words, if the entries of  $\alpha$  do not go to zero "uniformly", then the resulting implicit bias minimizes a
2In Appendix C, we show that the (S)GD trajectory with this initialisation exactly matches that of another common parametrisation  $\beta_w = w_+^2 - w_-^2$  with initialisation  $w_{+,0} = w_{-,0} = \alpha$ . The second layer of our diagonal linear network is set to 0 in order to obtain results that are easier to interpret. However, our proof techniques can be applied directly to a general initialisation, at the cost of additional notations in our Theorems.
weighed  $\ell_1$ -norm. This phenomenon can lead to solutions with vastly different sparsity structure than the minimum  $\ell_1$ -norm interpolator. See Fig. 7 and Example 1 in Appendix D.
3.2 Implicit bias of (stochastic) gradient descent
In Theorem 1, we prove that for an initialisation  $\sqrt{2}\alpha \in \mathbb{R}^d$  and for arbitrary stepsize sequences  $(\gamma_k)_{k\geqslant 0}$  if the iterates converge to an interpolator, then this interpolator is the solution of a constrained minimisation problem which involves the hyperbolic entropy  $\psi_{\alpha_{\infty}}$  defined in (4), where  $\alpha_{\infty}\in \mathbb{R}^{d}$  is an effective initialisation which depends on the trajectory and on the stepsize sequence. Later, we prove the convergence of iterates for macroscopic step sizes in Theorem 2.
Theorem 1 (Implicit bias of (S)GD). Let  $(u_{k},v_{k})_{k\geqslant 0}$  follow the mini-batch SGD recursion (3) initialised at  $(u_0,v_0) = (\sqrt{2}\pmb {\alpha},\mathbf{0})$  and with step sizes  $(\gamma_k)_{k\geqslant 0}$ . Let  $(\beta_{k})_{k\geqslant 0} = (u_{k}\odot v_{k})_{k\geqslant 0}$  and assume that they converge to some interpolator  $\beta_{\infty}^{\star}\in S$ . Then,  $\beta_{\infty}^{\star}$  satisfies:
$$
\beta_ {\infty} ^ {\star} = \underset {\beta^ {\star} \in \mathcal {S}} {\operatorname {a r g m i n}} D _ {\psi_ {\alpha_ {\infty}}} \left(\beta^ {\star}, \tilde {\beta} _ {0}\right), \tag {5}
$$
where  $D_{\psi_{\alpha_{\infty}}}$  is the Bregman divergence with hyperentropy potential  $\psi_{\alpha_{\infty}}$  of the effective initialisation  $\alpha_{\infty}$ , and  $\tilde{\beta}_0$  is a small perturbation term. The effective initialisation  $\alpha_{\infty}$  is given by,
$$
\boldsymbol {\alpha} _ {\infty} ^ {2} = \boldsymbol {\alpha} ^ {2} \odot \exp \left(- \sum_ {k = 0} ^ {\infty} q \left(\gamma_ {k} \nabla \mathcal {L} _ {\mathcal {B} _ {k}} \left(\beta_ {k}\right)\right)\right), \tag {6}
$$
where  $q(x) = -\frac{1}{2}\ln \left((1 - x^2)^2\right)$  satisfies  $q(x)\geqslant 0$  for  $|x|\leqslant \sqrt{2}$ , with the convention  $q(1) = +\infty$ .
The perturbation term  $\tilde{\beta}_0\in \mathbb{R}^d$  is explicitly given by  $\tilde{\beta}_{0} = \frac{1}{2}\bigl (\pmb{\alpha}_{+}^{2} - \pmb{\alpha}_{-}^{2}\bigr)$ , where  $q_{\pm}(x) = \mp 2x-\ln ((1\mp x)^2)$ , and  $\pmb{\alpha}_{\pm}^{2} = \pmb{\alpha}^{2}\odot \exp (-\sum_{k = 0}^{\infty}q_{\pm}(\gamma_{k}\nabla \mathcal{L}_{\mathcal{B}_{k}}(\beta_{k})))$
Trajectory-dependent characterisation. The characterisation of  $\beta_{\infty}^{\star}$  in Theorem 1 holds for any stepsize schedule such that the iterates converge and goes beyond the continuous-time frameworks previously studied [61, 48]. The result even holds for adaptive stepsize schedules which keep the stepsize scalar such as AdaDelta [65]. An important aspect of our result is that  $\alpha_{\infty}$  and  $\tilde{\beta}_0$  depend on the iterates' trajectory. Nevertheless, we argue that our formulation provides useful ingredients for understanding the implicit regularisation effects of (S)GD for this problem compared to trivial characterisations (such as e.g.,  $\min_{\beta} \| \beta - \beta_{\infty}^{\star} \|$ ). Importantly, the key parameters  $\alpha_{\infty}, \tilde{\beta}_0$  depend on crucial parameters such as the stepsize and noise in a useful and analysable manner: understanding how they affect  $\alpha_{\infty}$  and  $\tilde{\beta}_0$  coincides with understanding how they affect the recovered solution  $\beta_{\infty}^{\star}$  and its generalisation properties. This is precisely the object of Sections 4 and 5 where we discuss the qualitative and quantitative insights from Theorem 1 in greater detail.
The perturbation  $\tilde{\beta}_0$  can be ignored. We show in Proposition 16, under reasonable assumptions on the stepsizes, that  $|\tilde{\beta}_0| \leqslant \alpha^2$  and  $\alpha_{\infty} \leqslant \alpha$  (component-wise). The magnitude of  $\tilde{\beta}_0$  is therefore negligible in front of the magnitudes of  $\beta^{\star} \in S$  and one can roughly ignore the term  $\tilde{\beta}_0$ . Hence, the implicit regularisation Eq. (5) can be thought of as  $\beta_{\infty}^{\star} \approx \mathrm{argmin}_{\beta^{\star} \in S} D_{\psi_{\alpha_{\infty}}}(\beta^{\star}, 0) = \psi_{\alpha_{\infty}}(\beta^{\star})$ , and thus the solution  $\beta_{\infty}^{\star}$  minimises the same potential function that the solution of gradient flow (see Eq. (4)), but with an effective initialisation  $\alpha_{\infty}$ . Also note that for  $\gamma_k \equiv \gamma \to 0$  we have  $\alpha_{\infty} \to \alpha$  and  $\tilde{\beta}_0 \to 0$  (Proposition 19), recovering the previously known result for gradient flow (4).