SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yiwen Kou* Zixiang Chen* Quanquan Gu
Los Angeles, CA 90095
{evankou,chenzx19,qgu}@cs.ucla.edu
The implicit bias towards solutions with favorable properties is believed to be a key reason why neural networks trained by gradient-based optimization can generalize well. While the implicit bias of gradient flow has been widely studied for homogeneous neural networks (including ReLU and leaky ReLU networks), the implicit bias of gradient descent is currently only understood for smooth neural networks. Therefore, implicit bias in non-smooth neural networks trained by gradient descent remains an open question. In this paper, we aim to answer this question by studying the implicit bias of gradient descent for training two-layer fully connected (leaky) ReLU neural networks. We showed that when the training data are nearly-orthogonal, for leaky ReLU activation function, gradient descent will find a network with a stable rank that converges to 1, whereas for ReLU activation function, gradient descent will find a neural network with a stable rank that is upper bounded by a constant. Additionally, we show that gradient descent will find a neural network such that all the training data points have the same normalized margin asymptotically. Experiments on both synthetic and real data backup our theoretical findings.
Neural networks have achieved remarkable success in a variety of applications, such as image and speech recognition, natural language processing, and many others. Recent studies have revealed that the effectiveness of neural networks is attributed to their implicit bias towards particular solutions which enjoy favorable properties. Understanding how this bias is affected by factors such as network architecture, optimization algorithms and data used for training, has become an active research area in the field of deep learning theory.
The literature on the implicit bias in neural networks has expanded rapidly in recent years (Vardi, 2022), with numerous studies shedding light on the implicit bias of gradient flow (GF) with a wide range of neural network architecture, including deep linear networks (Ji and Telgarsky, 2018, 2020; Gunasekar et al., 2018), homogeneous networks (Lyu and Li, 2019; Vardi et al., 2022a) and more specific cases (Chizat and Bach, 2020; Lyu et al., 2021; Frei et al., 2022b; Safran et al., 2022). The implicit bias of gradient descent (GD), on the other hand, is better understood for linear predictors (Soudry et al., 2018) and smoothed neural networks (Lyu and Li, 2019; Frei et al., 2022b). Therefore, an open question still remains:
What is the implicit bias of leaky ReLU and ReLU networks trained by gradient descent?
In this paper, we will answer this question by investigating gradient descent for both two-layer leaky ReLU and ReLU neural networks on specific training data, where  $\{\mathbf{x}_i\}_{i=1}^n$  are nearly-orthogonal
(Frei et al., 2022b), i.e.,  $\| \mathbf{x}_i\| _2^2\geq Cn\max_{k\neq i}|\langle \mathbf{x}_i,\mathbf{x}_k\rangle |$  with a constant  $C$ . Our main results are summarized as follows:
- For two-layer leaky ReLU networks trained by GD, we demonstrate that the neuron activation pattern reaches a stable state beyond a specific time threshold and provide rigorous proof of the convergence of the stable rank of the weight matrix to 1, matching the results of Frei et al. (2022b) regarding gradient flow.
- For two-layer ReLU networks trained by GD, we proved that the stable rank of weight matrix can be upper bounded by a constant. Moreover, we present an illustrative example using completely orthogonal training data, showing that the stable rank of the weight matrix converges to a value approximately equal to 2. To the best of our knowledge, this is the first implicit bias result for two-layer ReLU networks trained by gradient descent beyond the Karush-Kuhn-Tucker (KKT) point.
- For both ReLU and leaky ReLU networks, we show that weight norm increases at the rate of  $\Theta (\log (t))$  and the training loss converges to zero at the rate of  $\Theta (t^{-1})$ , where  $t$  is the number of gradient descent iterations. This improves upon the  $O(t^{-1 / 2})$  rate proved in Frei et al. (2022b) for the case of a two-layer smoothed leaky ReLU network trained by gradient descent and aligns with the results by Lyu and Li (2019) for smooth homogeneous networks. Additionally, we prove that gradient descent will find a neural network such that all the training data points have the same normalized margin asymptotically.
Implicit bias in neural networks. Recent years have witnessed significant progress on implicit bias in neural networks trained by gradient flow (GF). Lyu and Li (2019) and Ji and Telgarsky (2020) demonstrated that homogeneous neural networks trained with exponentially-tailed classification losses converge in direction to the KKT point of a maximum-margin problem. Lyu et al. (2021) studied the implicit bias in two-layer leaky ReLU networks trained on linearly separable and symmetric data, showing that GF converges to a linear classifier maximizing the  $\ell_2$  margin. Frei et al. (2022b) showed that two-layer leaky ReLU networks trained by GF on nearly-orthogonal data produce a  $\ell_2$ -max-margin solution with a linear decision boundary and rank at most two. Other works studying the implicit bias of classification using GF in nonlinear two-layer networks include Chizat and Bach (2020); Phuong and Lampert (2021); Sarussi et al. (2021); Safran et al. (2022); Vardi et al. (2022a,b); Timor et al. (2023). Although implicit bias in neural networks trained by GF has been extensively studied, research on implicit bias in networks trained by gradient descent (GD) remains limited. Lyu and Li (2019) examined smoothed homogeneous neural network trained by GD with exponentially-tailed losses and proved a convergence to KKT points of a max-margin problem. Frei et al. (2022b) studied two-layer smoothed leaky ReLU trained by GD and revealed the implicit bias towards low-rank networks. Other works studying implicit bias towards rank minimization include Ji and Telgarsky (2018, 2020); Timor et al. (2023); Arora et al. (2019); Razin and Cohen (2020); Li et al. (2021). Lastly, Vardi (2022) provided a comprehensive literature survey on implicit bias.
Benign overfitting and double descent in neural networks. A parallel line of research aims to understand the benign overfitting phenomenon (Bartlett et al., 2020) of neural networks by considering a variety of models. For example, Allen-Zhu and Li (2020); Jelassi and Li (2022); Shen et al. (2022); Cao et al. (2022); Kou et al. (2023) studied the generalization performance of two-layer convolutional networks on patch-based data models. Several other papers studied high-dimensional mixture models (Chatterji and Long, 2021; Wang and Thrampoulidis, 2022; Cao et al., 2021; Frei et al., 2022a). Another thread of work Belkin et al. (2020); Hastie et al. (2022); Wu and Xu (2020); Mei and Montanari (2019); Liao et al. (2020) focuses on understanding the double descent phenomenon first empirically observed by Belkin et al. (2019).
In this section, we introduce the notation, fully connected neural networks, the gradient descent-based training algorithm, and a data-coorrelated decomposition technique.
Notation. We use lower case letters, lower case bold face letters, and upper case bold face letters to denote scalars, vectors, and matrices respectively. For a vector  $\mathbf{v} = (v_{1},\dots ,v_{d})^{\top}$ , we denote by
$\| \mathbf{v}\| _2\coloneqq \left(\sum_{j = 1}^d v_j^2\right)^{1 / 2}$  its  $\ell_2$  norm. For a matrix  $\mathbf{A}\in \mathbb{R}^{m\times n}$ , we use  $\| \mathbf{A}\| _F$  to denote its Frobenius norm and  $\| \mathbf{A}\| _2$  its spectral norm. We use  $\mathrm{sign}(z)$  as the function that is 1 when  $z > 0$  and -1 otherwise. For a vector  $\mathbf{v}\in \mathbb{R}^d$ , we use  $[\mathbf{v}]_i\in \mathbb{R}$  to denote the  $i$ -th component of the vector. For two sequence  $\{a_k\}$  and  $\{b_k\}$ , we denote  $a_{k} = O(b_{k})$  if  $|a_{k}|\leq C|b_{k}|$  for some absolute constant  $C$ , denote  $a_{k} = \Omega (b_{k})$  if  $b_{k} = O(a_{k})$ , and denote  $a_{k} = \Theta (b_{k})$  if  $a_{k} = O(b_{k})$  and  $a_{k} = \Omega (b_{k})$ . We also denote  $a_{k} = o(b_{k})$  if  $\lim |a_k / b_k| = 0$ .
Two-layer fully connected neural newtork. We consider a two-layer neural network described as follows: its first layer consists of  $m$  positive neurons and  $m$  negative neurons; its second layer parameters are fixed as  $+1 / m$  and  $-1 / m$  respectively for positive and negative neurons. Then the network can be written as  $f(\mathbf{W},\mathbf{x}) = F_{+1}(\mathbf{W}_{+1},\mathbf{x}) - F_{-1}(\mathbf{W}_{-1},\mathbf{x})$ , where the partial network function of positive and negative neurons, i.e.,  $F_{+1}(\mathbf{W}_{+1},\mathbf{x}), F_{-1}(\mathbf{W}_{-1},\mathbf{x})$ , are defined as:
$$
F _ {j} \left(\mathbf {W} _ {j}, \mathbf {x}\right) = \frac {1}{m} \sum_ {r = 1} ^ {m} \sigma \left(\left\langle \mathbf {w} _ {j, r}, \mathbf {x} \right\rangle\right) \tag {3.1}
$$
for  $j \in \{\pm 1\}$ . Here,  $\sigma(z)$  represents the activation function. For ReLU,  $\sigma(z) = \max\{0, z\}$ , and for leaky ReLU,  $\sigma(z) = \max\{\gamma z, z\}$ , where  $\gamma \in (0, 1)$ .  $\mathbf{W}_j \in \mathbb{R}^{m \times d}$  is the collection of model weights associated with  $F_j$ , and  $\mathbf{w}_{j,r} \in \mathbb{R}^d$  denotes the weight vector for the  $r$ -th neuron in  $\mathbf{W}_j$ . We use  $\mathbf{W}$  to denote the collection of all model weights.
Gradient Descent. Given a training data set  $S = \{(\mathbf{x}_i, y_i)\}_{i=1}^n \subseteq \mathbb{R}^d \times \{\pm 1\}$ , instead of considering the gradient flow (GF) that is commonly studied in prior work on the implicit bias, we use gradient descent (GD) to optimize the empirical loss on the training data
$$
L _ {S} (\mathbf {W}) = \frac {1}{n} \sum_ {i = 1} ^ {n} \ell \left(y _ {i} \cdot f \left(\mathbf {W}, \mathbf {x} _ {i}\right)\right),
$$
where  $\ell(z) = \log(1 + \exp(-z))$  is the logistic loss, and  $S = \{(\mathbf{x}_i, y_i)\}_{i=1}^n$  is the training data set. The gradient descent update rule of each neuron in the two-layer neural network can be written as
$$
\mathbf {w} _ {j, r} ^ {(t + 1)} = \mathbf {w} _ {j, r} ^ {(t)} - \eta \cdot \nabla_ {\mathbf {w} _ {j, r}} L _ {S} (\mathbf {W} ^ {(t)}) = \mathbf {w} _ {j, r} ^ {(t)} - \frac {\eta}{n m} \sum_ {i = 1} ^ {n} \ell_ {i} ^ {\prime (t)} \cdot \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {j, r} ^ {(t)}, \mathbf {x} _ {i} \right\rangle\right) \cdot j y _ {i} \mathbf {x} _ {i} \tag {3.2}
$$
for all  $j \in \{\pm 1\}$  and  $r \in [m]$ , where we introduce a shorthand notation  $\ell_i^{\prime (t)} = \ell^{\prime}[y_i \cdot f(\mathbf{W}^{(t)}, \mathbf{x}_i)]$  and assume the derivative of the ReLU activation function at 0 is  $\sigma'(0) = 1$  without loss of generality. Here  $\eta > 0$  is the learning rate. We initialize the gradient descent by Gaussian initialization, where all the entries of  $\mathbf{W}^{(0)}$  are sampled from i.i.d. Gaussian distributions  $\mathcal{N}(0, \sigma_0^2)$  with  $\sigma_0^2$  being the variance.
In this section, we present our main theoretical results. For the training data set  $\mathcal{S} = \{(x_i,y_i)\}_{i=1}^n \subseteq \mathbb{R}^d \times \{\pm 1\}$ , let  $R_{\min} = \min_i \| x_i \|_2$ ,  $R_{\max} = \max_i \| x_i \|_2$ ,  $p = \max_{i \neq k} |\langle x_i, x_k \rangle|$ , and suppose  $R = R_{\max} / R_{\min}$  is at most an absolute constant. For simplicity, we only consider the dependency on  $t$  when characterizing the convergence rates of the weight matrix related quantities and the training loss, omitting the dependency on other parameters such as  $m, n, \sigma_0, R_{\min}, R_{\max}$ .
Theorem 4.1 (Leaky ReLU Networks). For two-layer neural network defined in (3.1) with leaky ReLU activation  $\sigma(z) = \max\{\gamma z, z\}, \gamma \in (0,1)$ . Assume the training data satisfy  $R_{\min}^2 \geq CR^2 \gamma^{-4} np$  for some sufficiently large constant  $C$ . For any  $\delta \in (0,1)$ , if the learning rate  $\eta \leq (CR_{\max}^2 / nm)^{-1}$  and the initialization scale  $\sigma_0 \leq \gamma \left(C R_{\max} \sqrt{\log(mn / \delta)}\right)^{-1}$ , then with probability at least  $1 - \delta$  over the random initialization of gradient descent, the trained network satisfies:
- The  $\ell_2$  norm of each neuron increases to infinity at a logarithmic rate:  $\| \mathbf{w}_{j,r}^{(t)}\| _2 = \Theta (\log (t))$  for all  $j\in \{\pm 1\}$  and  $r\in [m]$ .
- Throughout the gradient descent trajectory, the stable rank of the weights  $\mathbf{W}_j^{(t)}$  for all  $j \in \{\pm 1\}$  satisfies
$$
\lim _ {t \to \infty} \| \mathbf {W} _ {j} ^ {(t)} \| _ {F} ^ {2} / \| \mathbf {W} _ {j} ^ {(t)} \| _ {2} ^ {2} = 1,
$$
with a convergence rate of  $O(1 / \log (t))$ .
- Gradient descent will find  $\mathbf{W}^{(t)}$  such that all the training data points possess the same normalized margin asymptotically:
$$
\lim _ {t \to \infty} \left| y _ {i} f (\mathbf {W} ^ {(t)} / \| \mathbf {W} ^ {(t)} \| _ {F}, \mathbf {x} _ {i}) - y _ {k} f (\mathbf {W} ^ {(t)} / \| \mathbf {W} ^ {(t)} \| _ {F}, \mathbf {x} _ {k}) \right| = 0, \forall i, k \in [ n ].
$$
If we assume that  $\mathbf{W}^{(t)}$  converges in direction, i.e., the limit of  $\mathbf{W}^{(t)} / \| \mathbf{W}^{(t)}\| _F$  exists, denoted by  $\bar{\mathbf{W}}$ , then there exists a scaling factor  $\alpha >0$  such that  $\alpha \bar{\mathbf{W}}$  satisfies the Karush-Kuhn-Tucker (KKT) conditions for the following max-margin problem:
$$
\min  _ {\mathbf {W}} \frac {1}{2} \| \mathbf {W} \| _ {F} ^ {2}, \quad \text {s . t .} \quad y _ {i} f (\mathbf {W}, \mathbf {x} _ {i}) \geq 1, \forall i \in [ n ]. \tag {4.1}
$$
- The empirical loss converges to zero at the following rate:  $L_{S}(\mathbf{W}^{(t)}) = \Theta (t^{-1})$ .
Remark 4.2. In Theorem 4.1, we show that when using the leaky ReLU activation function on nearly orthogonal training data, gradient descent asymptotically finds a network with a stable rank of  $\mathbf{W}_j$  equal to 1. Additionally, we demonstrate that gradient descent will find a network by which all the training data points share the same normalized margin asymptotically. Moreover, if we assume the weight matrix converges in direction, then its limit will satisfy the KKT conditions of the max-margin problem (4.1). Furthermore, we analyze the rate of weight norm increase and the convergence rate of the stable rank for gradient descent, both of which exhibit a logarithmic dependency in  $t$ .
Theorem 4.3 (ReLU Networks). For two-layer neural network defined in (3.1) with ReLU activation  $\sigma(z) = \max\{0, z\}$ . Assume the training data satisfy  $R_{\min}^2 \geq CR^2 np$  for some sufficiently large constant  $C$ . For any  $\delta \in (0,1)$ , if the neural network width  $m \geq C \log(n / \delta)$ , learning rate  $\eta \leq (CR_{\max}^2 / nm)^{-1}$  and initialization scale  $\sigma_0 \leq \left(C R_{\max} \sqrt{\log(mn / \delta)}\right)^{-1}$ , then with probability at least  $1 - \delta$  over the random initialization of gradient descent, the trained network satisfies:
- The Frobenius norm and the spectral norm of weight matrix increase to infinity at a logarithmic rate:  $\| \mathbf{W}_j^{(t)}\| _F = \Theta (\log (t))$  and  $\| \mathbf{W}_j^{(t)}\| _2 = \Theta (\log (t))$  for all  $j\in \{\pm 1\}$
- Throughout the gradient descent trajectory, the stable rank of the weights  $\mathbf{W}_j^{(t)}$  for all  $j\in \{\pm 1\}$  satisfies,
$$
\lim _ {t \to \infty} \sup _ {\mathbf {\Phi} t \to \infty} \| \mathbf {W} _ {j} ^ {(t)} \| _ {F} ^ {2} / \| \mathbf {W} _ {j} ^ {(t)} \| _ {2} ^ {2} \leq c,
$$
where  $c$  is an absolute constant.
- Gradient descent will find a  $\mathbf{W}^{(t)}$  such that all the training data points possess the same normalized margin asymptotically:
Remark 4.4. For ReLU networks, we provide an example in the appendix concerning fully orthogonal training data and prove that the activation pattern during training depends solely on the initial activation state. Specifically, when training a two-layer ReLU network with gradient descent using such data, the stable rank of the network's weight matrix  $\mathbf{W}_j$  converges to approximately 2. It is worth noting that this stable rank value is higher than the stable rank achieved by leaky ReLU networks, which is 1.
Comparison with previous work. One notable related work is Lyu et al. (2021), which also investigates the implicit bias of two-layer leaky ReLU networks. The main distinction between our work and Lyu et al. (2021) is the optimization method employed. We utilize gradient descent, whereas they utilize gradient flow. Additionally, our assumption is that the training data is nearly-orthogonal,
while they assume the training data is symmetric. Our findings are more closely related to the work by Frei et al. (2022b), which investigates both gradient flow and gradient decent. In both our study and Frei et al. (2022b), we examine two-layer neural networks with leaky ReLU activations. However, they focus on networks trained via gradient flow, while we investigate networks trained using gradient descent. For the gradient descent approach, Frei et al. (2022b) provide a constant stable rank upper bound for smoothed leaky ReLU. In contrast, we prove that the stable rank of leaky ReLU networks converges to 1, aligning with the implicit bias of gradient flow proved in Frei et al. (2022b). Furthermore, they presented an  $O(t^{-1/2})$  convergence rate for the empirical loss, whereas our convergence rate is  $\Theta(t^{-1})$ . Another related work is Lyu and Li (2019), which studied smooth homogeneous networks trained by gradient descent. Our results on the rate of weight norm increase and the convergence rate of training loss match those in Lyu and Li (2019), despite the fact that we study non-smooth homogeneous networks. It is worth noting that Lyu and Li (2019); Lyu et al. (2021); Frei et al. (2022b) demonstrated that neural networks trained by gradient flow converge to a Karush-Kuhn-Tucker (KKT) point of the max-margin problem. We do not have such a result unless we assume the directional convergence of the weight matrix.
5 Overview of Proof Techniques
In this section, we discuss the key techniques we invent in our proofs to analyze the implicit bias of ReLU and leaky ReLU networks.
5.1 Refined Analysis of Decomposition Coefficient
Signal-noise decomposition, a technique initially introduced by Cao et al. (2022), is used to analyze the learning dynamics of two-layer convolutional networks. This method decomposes the convolutional filters into a linear combination of initial filters, signal vectors, and noise vectors, converting the neural network learning into a dynamical system of coefficients derived from the decomposition. In this work, we extend the signal-noise decomposition to data-correlated decomposition to facilitate the analysis of the training dynamic for two-layer fully connected neural networks.
Definition 5.1 (Data-correlated Decomposition). Let  $\mathbf{w}_{j,r}^{(t)}, j \in \{\pm 1\}, r \in [m]$  be the weights of first-layer neurons at the  $t$ -th iteration of gradient descent. There exist unique coefficients  $\rho_{j,r,i}^{(t)}$  such that
$$
\mathbf {w} _ {j, r} ^ {(t)} = \mathbf {w} _ {j, r} ^ {(0)} + \sum_ {i = 1} ^ {n} \rho_ {j, r, i} ^ {(t)} \cdot \| \mathbf {x} _ {i} \| _ {2} ^ {- 2} \cdot \mathbf {x} _ {i}. \tag {5.1}
$$
By defining  $\overline{\rho}_{j,r,i}^{(t)}\coloneqq \rho_{j,r,i}^{(t)}\mathbb{1}(\rho_{j,r,i}^{(t)}\geq 0)$ ,  $\underline{\rho}_{j,r,i}^{(t)}\coloneqq \rho_{j,r,i}^{(t)}\mathbb{1}(\rho_{j,r,i}^{(t)}\leq 0)$ , (5.1) can be further written as
$$
\mathbf {w} _ {j, r} ^ {(t)} = \mathbf {w} _ {j, r} ^ {(0)} + \sum_ {i = 1} ^ {n} \bar {\rho} _ {j, r, i} ^ {(t)} \cdot \| \mathbf {x} _ {i} \| _ {2} ^ {- 2} \cdot \mathbf {x} _ {i} + \sum_ {i = 1} ^ {n} \underline {{\rho}} _ {j, r, i} ^ {(t)} \cdot \| \mathbf {x} _ {i} \| _ {2} ^ {- 2} \cdot \mathbf {x} _ {i}. \tag {5.2}
$$
As an extension of the signal-noise decomposition first proposed in Cao et al. (2022) for analyzing two-layer convolutional networks, data-correlated decomposition defined in Definition 5.1 can be used to analyze two-layer fully-connected network, where the normalization factors  $\| \mathbf{x}_i\| _2^{-2}$  are introduced to ensure that  $\rho_{j,r,i}^{(t)}\approx \langle \mathbf{w}_{j,r}^{(t)},\mathbf{x}_i\rangle$  . This is also inspired by previous works by Lyu and Li (2019); Frei et al. (2022b), which demonstrate that  $\mathbf{W}$  converges to a KKT point of the max-margin problem. This implies that  $\mathbf{w}_{j,r}^{(\infty)} / \| \mathbf{w}_{j,r}^{(\infty)}\| _2$  can be expressed as a linear combination of the training data  $\{\mathbf{x}_i\}_{i = 1}^n$  , with the coefficient  $\lambda_{i}$  corresponding to  $\rho_{j,r,i}^{(t)}$  in our analysis. This technique does not rely on the strictly increasing and smoothness properties of the activation function and will serve as the foundation for our analysis. Let us first investigate the update rule of the coefficient  $\overline{\rho}_{j,r,i}^{(t)},\underline{\rho}_{j,r,i}^{(t)}$
Lemma 5.2. The coefficients  $\overline{\rho}_{j,r,i}^{(t)},\underline{\rho}_{j,r,i}^{(t)}$  defined in Definition 5.1 satisfy the following iterative equations:
$$
\bar {\rho} _ {j, r, i} ^ {(0)}, \underline {{\rho}} _ {j, r, i} ^ {(0)} = 0, \tag {5.3}
$$
$$
\bar {\rho} _ {j, r, i} ^ {(t + 1)} = \bar {\rho} _ {j, r, i} ^ {(t)} - \frac {\eta}{n m} \cdot \ell_ {i} ^ {\prime (t)} \cdot \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {j, r} ^ {(t)}, \mathbf {x} _ {i} \right\rangle\right) \cdot \left\| \mathbf {x} _ {i} \right\| _ {2} ^ {2} \cdot \mathbb {1} \left(y _ {i} = j\right), \tag {5.4}
$$
$$
\underline {{\rho}} _ {j, r, i} ^ {(t + 1)} = \underline {{\rho}} _ {j, r, i} ^ {(t)} + \frac {\eta}{n m} \cdot \ell_ {i} ^ {\prime (t)} \cdot \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {j, r} ^ {(t)}, \mathbf {x} _ {i} \right\rangle\right) \cdot \| \mathbf {x} _ {i} \| _ {2} ^ {2} \cdot \mathbb {1} \left(y _ {i} = - j\right), \tag {5.5}
$$
for all  $r \in [m]$ ,  $j \in \{\pm 1\}$  and  $i \in [n]$ .
To study implicit bias, the first main challenge is to generalize the decomposition coefficient analysis to infinite time. The signal-noise decomposition used in Cao et al. (2022); Kou et al. (2023) requires early stopping with threshold  $T^{*}$  to facilitate their analysis. They only provided upper bounds of  $4\log (T^{*})$  for  $\overline{\rho}_{j,r,i}^{(t)}, |\underline{\rho}_{j,r,i}^{(t)}|$  (See Proposition 5.3 in Cao et al. (2022), Proposition 5.2 in Kou et al. (2023)), and then carried out a two-stage analysis. To obtain upper bounds for  $\overline{\rho}_{j,r,i}^{(t)}, |\underline{\rho}_{j,r,i}^{(t)}|$ , they used an upper bound for  $|\ell_i^{(t)}|$  and directly plugged it into (5.4) and (5.5) to demonstrate that  $\overline{\rho}_{j,r,i}^{(t)}$  and  $|\underline{\rho}_{j,r,i}^{(t)}|$  would not exceed  $4\log (T^{*})$ , which is a fixed value related to the early stopping threshold. Therefore, dealing with infinite time requires new techniques. To overcome this difficulty, we propose a refined analysis of decomposition coefficients which generalizes Cao et al. (2022)'s technique. We first give the following key lemma.
Lemma 5.3. For non-negative real number sequence  $\{x_{t}\}_{t = 0}^{\infty}$  satisfying
$$
C _ {1} \exp (- x _ {t}) \leq x _ {t + 1} - x _ {t} \leq C _ {2} \exp (- x _ {t}), \tag {5.6}
$$
$$
\log \left(\exp \left(- x _ {0}\right) + C _ {1} \cdot t\right) \leq x _ {t} \leq \log \left(\exp \left(- x _ {0}\right) + C _ {2} \exp \left(C _ {2}\right) \cdot t\right). \tag {5.7}
$$
We can establish the relationship between (5.4), (5.5) and inequality (5.6) if we are able to express  $|\ell_i^{\prime (t)}|$  using coefficients  $\overline{\rho}_{j,r,i}^{(t)}$  and  $|\underline{\rho}_{j,r,i}^{(t)}|$ . To achieve this, we can first approximate  $\ell_i^{(t)}$  using the margin  $y_{i}f(\mathbf{W}^{(t)},\mathbf{x}_{i})$  and then approximate  $F_{j}(\mathbf{W}_{j}^{(t)},\mathbf{x}_{i})$  using the coefficients  $\rho_{j,r,i}^{(t)}$ . The approximation is given as follows: