SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zixiang Chen* Junkai Zhang* Yiwen Kou Xiangning Chen Cho-Jui Hsieh Quanquan Gu
{chenzx19,zhang,evankou,xiangning,chohsieh,qgu}@cs.ucla.edu
The challenge of overfitting, in which the model memorizes the training data and fails to generalize to test data, has become increasingly significant in the training of large neural networks. To tackle this challenge, Sharpness-Aware Minimization (SAM) has emerged as a promising training method, which can improve the generalization of neural networks even in the presence of label noise. However, a deep understanding of how SAM works, especially in the setting of nonlinear neural networks and classification tasks, remains largely missing. This paper fills this gap by demonstrating why SAM generalizes better than Stochastic Gradient Descent (SGD) for a certain data model and two-layer convolutional ReLU networks. The loss landscape of our studied problem is nonsmooth, thus current explanations for the success of SAM based on the Hessian information are insufficient. Our result explains the benefits of SAM, particularly its ability to prevent noise learning in the early stages, thereby facilitating more effective learning of features. Experiments on both synthetic and real data corroborate our theory.
The remarkable performance of deep neural networks has sparked considerable interest in creating ever-larger deep learning models, while the training process continues to be a critical bottleneck affecting overall model performance. The training of large models is unstable and difficult due to the sharpness, non-convexity, and non-smoothness of its loss landscape. In addition, as the number of model parameters is much larger than the training sample size, the model has the ability to memorize even randomly labeled data (Zhang et al., 2021), which leads to overfitting. Therefore, although traditional gradient-based methods like gradient descent (GD) and stochastic gradient descent (SGD) can achieve generalizable models under certain conditions, these methods may suffer from unstable training and harmful overfitting in general.
To overcome the above challenge, Sharpness-Aware Minimization (SAM) (Foret et al., 2020), an innovative training paradigm, has exhibited significant improvement in model generalization and has become widely adopted in many applications. In contrast to traditional gradient-based methods that primarily focus on finding a point in the parameter space with a minimal gradient norm, SAM also pursues a solution with reduced sharpness, characterized by how rapidly the loss function changes locally. Despite the empirical success of SAM across numerous tasks (Bahri et al., 2021; Behdin et al., 2022; Chen et al., 2021; Liu et al., 2022a), the theoretical understanding of this method remains limited.
Foret et al. (2020) provided a PAC-Bayes bound on the generalization error of SAM to show that it will generalize well, while the bound only holds for the infeasible average-direction perturbation instead of
Figure 1: Illustration of the phase transition between benign overfitting and harmful overfitting. The yellow region represents the regime under which the overfitted CNN trained by SGD is guaranteed to have a small excess risk, and the blue region represents the regime under which the excess risk is guaranteed to be a constant order (e.g., greater than 0.1). The gray region is the regime where the excess risk is not characterized.
practically used ascend-direction perturbation. Andriushchenko and Flammarion (2022) investigated the implicit bias of SAM for diagonal linear networks under global convergence assumption. The oscillations in the trajectory of SAM were explored by Bartlett et al. (2022), leading to a convergence result for the convex quadratic loss. A concurrent work (Wen et al., 2022) demonstrated that SAM can locally regularize the eigenvalues of the Hessian of the loss. In the context of least-squares linear regression, Behdin and Mazumder (2023) found that SAM exhibits lower bias and higher variance compared to gradient descent. However, all the above analyses of SAM utilize the Hessian information of the loss and require the smoothness property of the loss implicitly. The study for non-smooth neural networks, particularly for the classification task, remains open.
In this paper, our goal is to provide a theoretical basis demonstrating when SAM outperforms SGD. In particular, we consider a data distribution mainly characterized by the signal  $\mu$  and input data dimension  $d$ , and prove the following separation in terms of test error between SGD and SAM.
Theorem 1.1 (Informal statement of Theorems 3.2 and 4.1). Let  $p$  be the strength of the label flipping noise. For any  $\epsilon > 0$ , under certain regularity conditions, with high probability, there exists  $0 \leq t \leq T$  such that the training loss converges, i.e.,  $L_{S}(\mathbf{W}^{(t)}) \leq \epsilon$ . Besides,
1. For SGD, when the signal strength  $\| \pmb{\mu}\|_2 \geq \Omega(d^{1/4})$ , we have  $L_{\mathcal{D}}^{0-1}(\mathbf{W}^{(t)}) \leq p + \epsilon$ . When the signal strength  $\| \pmb{\mu}\|_2 \leq O(d^{1/4})$ , we have  $L_{\mathcal{D}}^{0-1}(\mathbf{W}^{(t)}) \geq p + 0.1$ .
2. For SAM, provided the signal strength  $\| \pmb{\mu}\| _2\geq \widetilde{\Omega} (1)$  , we have  $L_{\mathcal{D}}^{0 - 1}(\mathbf{W}^{(t)})\leq p + \epsilon$
- We discuss how the loss landscape of two-layer convolutional ReLU networks is different from the smooth loss landscape and thus the current explanation for the success of SAM based on the Hessian information is insufficient for neural networks.
- To understand the limit of SGD, we precisely characterize the conditions under which benign overfitting can occur in training two-layer convolutional ReLU networks with SGD. To the best of our knowledge, this is the first benign overfitting result for neural network trained with mini-batch SGD. We also prove a phase transition phenomenon for SGD, which is illustrated in Figure 1.
- Under the conditions when SGD leads to harmful overfitting, we formally prove that SAM can achieve benign overfitting. Consequently, we establish a rigorous theoretical distinction between SAM and SGD, demonstrating that SAM strictly outperforms SGD in terms of generalization error. Specifically, we show that SAM effectively mitigates noise learning in the early stages of training, enabling neural networks to learn features more efficiently.
Notation. We use lower case letters, lower case bold face letters, and upper case bold face letters to denote scalars, vectors, and matrices respectively. For a vector  $\mathbf{v} = (v_{1},\dots ,v_{d})^{\top}$ , we denote
by  $\| \mathbf{v}\| _2\coloneqq \left(\sum_{j = 1}^d v_j^2\right)^{1 / 2}$  its 2-norm. For two sequences  $\{a_k\}$  and  $\{b_k\}$ , we denote  $a_{k} = O(b_{k})$  if  $|a_{k}|\leq C|b_{k}|$  for some absolute constant  $C$ , denote  $a_{k} = \Omega (b_{k})$  if  $b_{k} = O(a_{k})$ , and denote  $a_{k} = \Theta (b_{k})$  if  $a_{k} = O(b_{k})$  and  $a_{k} = \Omega (b_{k})$ . We also denote  $a_{k} = o(b_{k})$  if  $\lim |a_k / b_k| = 0$ . Finally, we use  $\widetilde{O} (\cdot)$  and  $\widetilde{\Omega} (\cdot)$  to omit logarithmic terms in the notation. We denote the set  $\{1,\dots ,n\}$  by  $[n]$ , and the set  $\{0,\dots ,n - 1\}$  by  $[\overline{n} ]$ , respectively. The carnality of a set  $S$  is denoted by  $|S|$ .
2.1 Data distribution
Our focus is on binary classification with label  $y \in \{\pm 1\}$ . We consider the following data model, which can be seen as a special case of sparse coding model (Olshausen and Field, 1997; Allen-Zhu and Li, 2022; Ahn et al., 2022).
Definition 2.1. Let  $\mu \in \mathbb{R}^d$  be a fixed vector representing the signal contained in each data point. Each data point  $(\mathbf{x},y)$  with input  $\mathbf{x} = [\mathbf{x}^{(1)\top},\mathbf{x}^{(2)\top},\dots,\mathbf{x}^{(P)\top}]^\top \in \mathbb{R}^{P\times d},\mathbf{x}^{(1)},\mathbf{x}^{(2)},\dots,\mathbf{x}^{(P)}\in \mathbb{R}^d$  and label  $y\in \{-1,1\}$  is generated from a distribution  $\mathcal{D}$  specified as follows:
1. The true label  $\widehat{y}$  is generated as a Rademacher random variable, i.e.,  $\mathbb{P}[\widehat{y} = 1] = \mathbb{P}[\widehat{y} = -1] = 1 / 2$ . The observed label  $y$  is then generated by flipping  $\widehat{y}$  with probability  $p$  where  $p < 1 / 2$ , i.e.,  $\mathbb{P}[y = \widehat{y}] = 1 - p$  and  $\mathbb{P}[y = -\widehat{y}] = p$ .
2. A noise vector  $\pmb{\xi}$  is generated from the Gaussian distribution  $\mathcal{N}(\mathbf{0},\sigma_p^2\mathbf{I})$  , where  $\sigma_p^2$  is the variance.
3. One of  $\mathbf{x}^{(1)},\mathbf{x}^{(2)},\ldots ,\mathbf{x}^{(P)}$  is randomly selected and then assigned as  $y\cdot \mu$  , which represents the signal, while the others are given by  $\xi$  , which represents noises.
The data distribution in Definition 2.1 has also been extensively employed in several previous works (Allen-Zhu and Li, 2020; Jelassi and Li, 2022; Shen et al., 2022; Cao et al., 2022; Kou et al., 2023). When  $P = 2$ , this data distribution aligns with the one analyzed in Kou et al. (2023). This distribution is inspired by image data, where the input is composed of different patches, with only a few patches being relevant to the label. The model has two key vectors: the feature vector and the noise vector. For any input vector  $\mathbf{x} = [\mathbf{x}^{(1)}, \dots, \mathbf{x}^{(P)}]$ , there is exactly one  $\mathbf{x}^{(j)} = y\pmb{\mu}$ , and the others are random Gaussian vectors. For example, the input vector  $\mathbf{x}$  can be  $[y\pmb{\mu}, \pmb{\xi}, \dots, \pmb{\xi}], \dots, [\pmb{\xi}, \dots, y\pmb{\mu}, \pmb{\xi}]$  or  $[\pmb{\xi}, \dots, \pmb{\xi}, y\pmb{\mu}]$ : the signal patch  $y\pmb{\mu}$  can appear at any position. To avoid harmful overfitting, the model must learn the feature vector rather than the noise vector.
2.2 Neural Network and Training Loss
To effectively learn the distribution as per Definition 2.1, it is advantageous to utilize a shared weights structure, given that the specific signal patch is not known beforehand. When  $P > n$ , shared weights become indispensable as the location of the signal patch in the test can differ from the location of the signal patch in the training data.
We consider a two-layer convolutional neural network whose filters are applied to the  $P$  patches  $\mathbf{x}_1,\dots ,\mathbf{x}_P$  separately, and the second layer parameters of the network are fixed as  $+1 / m$  and  $-1 / m$  respectively, where  $m$  is the number of convolutional filters. Then the network can be written as  $f(\mathbf{W},\mathbf{x}) = F_{+1}(\mathbf{W}_{+1},\mathbf{x}) - F_{-1}(\mathbf{W}_{-1},\mathbf{x})$ , where  $F_{+1}(\mathbf{W}_{+1},\mathbf{x})$  and  $F_{-1}(\mathbf{W}_{-1},\mathbf{x})$  are defined as
$$
F _ {j} \left(\mathbf {W} _ {j}, \mathbf {x}\right) = m ^ {- 1} \sum_ {r = 1} ^ {m} \sum_ {p = 1} ^ {P} \sigma \left(\left\langle \mathbf {w} _ {j, r}, \mathbf {x} ^ {(p)} \right\rangle\right). \tag {1}
$$
Here we consider ReLU activation function  $\sigma (z) = \mathbb{1}(z\geq 0)z$ $\mathbf{w}_{j,r}\in \mathbb{R}^d$  denotes the weight for the  $r$ -th filter, and  $\mathbf{W}_j$  is the collection of model weights associated with  $F_{j}$  for  $j = \pm 1$ . We use  $\mathbf{W}$  to denote the collection of all model weights. Denote the training data set by  $S = \{(\mathbf{x}_i,y_i)\}_{i\in [n]}$ . We train the above CNN model by minimizing the empirical cross-entropy loss function
$$
L _ {\mathcal {S}} (\mathbf {W}) = n ^ {- 1} \sum_ {i \in [ n ]} \ell (y _ {i} f (\mathbf {W}, \mathbf {x} _ {i})),
$$
where  $\ell (z) = \log (1 + \exp (-z))$
2.3 Training Algorithm
Minibatch Stochastic Gradient Descent. For epoch  $t$ , the training data set  $S$  is randomly divided into  $H := n / B$  mini batches  $\mathcal{I}_{t,b}$  with batch size  $B \geq 2$ . The empirical loss for batch  $\mathcal{I}_{t,b}$  is defined
as  $L_{\mathcal{I}_{t,b}}(\mathbf{W}) = (1 / B)\sum_{i\in \mathcal{I}_{t,b}}\ell (y_if(\mathbf{W},\mathbf{x}_i))$ . Then the gradient descent update of the filters in the CNN can be written as
$$
\mathbf {w} ^ {(t, b + 1)} = \mathbf {w} ^ {(t, b)} - \eta \cdot \nabla_ {\mathbf {W}} L _ {\mathcal {I} _ {t, b}} (\mathbf {W} ^ {(t, b)}), \tag {2}
$$
where the gradient of the empirical loss  $\nabla_{\mathbf{W}}L_{\mathcal{I}_{t,b}}$  is the collection of  $\nabla_{\mathbf{w}_{j,r}}L_{\mathcal{I}_{t,b}}$  as follows
$$
\begin{array}{l} \nabla_ {\mathbf {w} _ {j, r}} L _ {\mathcal {I} _ {t, b}} \left(\mathbf {W} ^ {(t, b)}\right) = \frac {(P - 1)}{B m} \sum_ {i \in \mathcal {I} _ {t, b}} \ell_ {i} ^ {\prime (t, b)} \cdot \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {j, r} ^ {(t, b)}, \boldsymbol {\xi} _ {i} \right\rangle\right) \cdot j y _ {i} \boldsymbol {\xi} _ {i} \\ + \frac {1}{B m} \sum_ {i \in \mathcal {I} _ {t, b}} \ell_ {i} ^ {\prime (t, b)} \cdot \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {j, r} ^ {(t, b)}, \widehat {y} _ {i} \boldsymbol {\mu} \right\rangle\right) \cdot \widehat {y} _ {i} y _ {i} j \boldsymbol {\mu}, \tag {3} \\ \end{array}
$$
for all  $j \in \{\pm 1\}$  and  $r \in [m]$ . Here we introduce a shorthand notation  $\ell_i^{\prime (t,b)} = \ell^{\prime}[y_i \cdot f(\mathbf{W}^{(t,b)}, \mathbf{x}_i)]$  and assume the gradient of the ReLU activation function at 0 to be  $\sigma'(0) = 1$  without loss of generality. We use  $(t,b)$  to denote epoch index  $t$  with mini-batch index  $b$  and use  $(t)$  as the shorthand of  $(t,0)$ . We initialize SGD by random Gaussian, where all entries of  $\mathbf{W}^{(0)}$  are sampled from i.i.d. Gaussian distributions  $\mathcal{N}(0, \sigma_0^2)$ , with  $\sigma_0^2$  being the variance. From (3), we can infer that the loss landscape of the empirical loss is highly non-smooth because the ReLU function is not differentiable at zero. In particular, when  $\langle \mathbf{w}_{j,r}^{(t,b)}, \boldsymbol{\xi} \rangle$  is close to zero, even a very small perturbation can greatly change the activation pattern  $\sigma'(\langle \mathbf{w}_{j,r}^{(t,b)}, \boldsymbol{\xi} \rangle)$  and thus change the direction of  $\nabla_{\mathbf{w}_{j,r}} L_{\mathcal{I}_{t,b}}(\mathbf{W}^{(t,b)})$ . This observation prevents the analysis technique based on the Taylor expansion with the Hessian matrix, and calls for a more sophisticated activation pattern analysis.
Sharpness Aware Minimization. Given an empirical loss function  $L_{S}(\mathbf{W})$  with trainable parameter  $\mathbf{W}$ , the idea of SAM is to minimize a perturbed empirical loss at the worst point in the neighborhood ball of  $\mathbf{W}$  to ensure a uniformly low training loss value. In particular, it aims to solve the following optimization problem
$$
\min  _ {\mathbf {W}} L _ {S} ^ {\mathrm {S A M}} (\mathbf {W}), \quad \text {w h e r e} \quad L _ {S} ^ {\mathrm {S A M}} (\mathbf {W}) := \max  _ {\| \boldsymbol {\epsilon} \| _ {2} \leq \tau} L _ {S} (\mathbf {W} + \boldsymbol {\epsilon}), \tag {4}
$$
where the hyperparameter  $\tau$  is called the perturbation radius. However, directly optimizing  $L_S^{\mathrm{SAM}}(\mathbf{W})$  is computationally expensive. In practice, people use the following sharpness-aware minimization (SAM) algorithm (Foret et al., 2020; Zheng et al., 2021) to minimize  $L_S^{\mathrm{SAM}}(\mathbf{W})$  efficiently,
$$
\mathbf {W} ^ {(t + 1)} = \mathbf {W} ^ {(t)} - \eta \nabla_ {\mathbf {W}} L _ {S} (\mathbf {W} + \widehat {\boldsymbol {\epsilon}}), \quad \text {w h e r e} \quad \widehat {\boldsymbol {\epsilon}} = \tau \cdot \frac {\nabla_ {\mathbf {W}} L _ {S} (\mathbf {W})}{\| \nabla_ {\mathbf {W}} L _ {S} (\mathbf {W}) \| _ {F}}. \tag {5}
$$
When applied to SGD in (2), the gradient  $\nabla_{\mathbf{W}}L_S$  in (5) is further replaced by stochastic gradient  $\nabla_{\mathbf{W}}L_{\mathcal{I}_{t,b}}$  (Foret et al., 2020). The detailed algorithm description of SAM is shown in Algorithm 1.
Algorithm 1 Minibatch Sharpness Aware Minimization
Input: Training set  $\mathcal{S} = \cup_{i=1}^{n}\{(\mathbf{x}_i, \mathbf{y}_i)\}$ , Batch size  $B$ , step size  $\eta > 0$ , neighborhood size  $\tau > 0$ . Initialize weights  $\mathbf{W}^{(0)}$ .  
for  $t = 0, 1, \ldots, T-1$  do  
Randomly divide the training data set into  $H$  mini batches  $\{\mathcal{I}_{t,b}\}_{b=0}^{H-1}$ .  
for  $b = 0, 1, \ldots, H-1$  do  
We calculate the perturbation  $\widehat{\epsilon}^{(t,b)} = \tau \frac{\nabla_{\mathbf{W}} L_{\mathcal{I}_{t,b}}(\mathbf{W}^{(t,b)})}{\|\nabla_{\mathbf{W}} L_{\mathcal{I}_{t,b}}(\mathbf{W}^{(t,b)})\|_F}$ .  
Update model parameters:  $\mathbf{W}^{(t,b+1)} = \mathbf{W}^{(t,b)} - \eta \nabla_{\mathbf{W}} L_{\mathcal{I}_{t,b}}(\mathbf{W})|_{\mathbf{W} = \mathbf{W}^{(t,b)} + \widehat{\epsilon}^{(t,b)}}$ .  
end for  
Update model parameters:  $\mathbf{W}^{(t+1,0)} = \mathbf{W}^{(t,H)}$   
end for
3 Result for SGD
In this section, we present our main theoretical results for the CNN trained with SGD. Our results are based on the following conditions on the dimension  $d$ , sample size  $n$ , neural network width  $m$ , initialization scale  $\sigma_0$  and learning rate  $\eta$ .
Condition 3.1. Suppose there exists a sufficiently large constant  $C$ , such that the following hold:
1. Dimension  $d$  is sufficiently large:  $d \geq \widetilde{\Omega}\Big(\max\{nP^{-2}\sigma_p^{-2}\|\pmb{\mu}\|_2^2, n^2, P^{-2}\sigma_p^{-2}Bm\}\Big)$ .
2. Training sample size  $n$  and neural network width satisfy  $m, n \geq \widetilde{\Omega}(1)$ .
3. The 2-norm of the signal satisfies  $\| \pmb{\mu}\| _2\geq \widetilde{\Omega} (P\sigma_p)$
4. The noise rate  $p$  satisfies  $p \leq 1 / C$ .
5. The standard deviation of Gaussian initialization  $\sigma_0$  is appropriately chosen such that  $\sigma_0 \leq \widetilde{O}\Big(\big(\max\left\{P\sigma_p d / \sqrt{n},\|\pmb{\mu}\|_2\right\}\big)^{-1}\Big)$ .
6. The learning rate  $\eta$  satisfies  $\eta \leq \widetilde{O}\Big(\big(\max \left\{P^{2}\sigma_{p}^{2}d^{3 / 2} / (Bm),P^{2}\sigma_{p}^{2}d / B,n\| \mu \|_{2} / (\sigma_{0}B\sqrt{dm}),nP\sigma_{p}\| \boldsymbol {\mu}\|_{2} / (B^{2}m\epsilon)\right\} \big)^{-1}\Big)$ .
The conditions imposed on the data dimensions  $d$ , network width  $m$ , and the number of samples  $n$  ensure adequate overparameterization of the network. Additionally, the condition on the learning rate  $\eta$  facilitates efficient learning by our model. By concentration inequality, with high probability, the  $\ell_2$  norm of the noise patch is of order  $\Theta(d\sigma_p^2)$ . Therefore, the quantity  $d\sigma_p^2$  can be viewed as the strength of the noise. Comparable conditions have been established in Chatterji and Long (2021); Cao et al. (2022); Frei et al. (2022); Kou et al. (2023). Based on the above condition, we first present a set of results on benign/harmful overfitting for SGD in the following theorem.
Theorem 3.2 (Benign/harmful overfitting of SGD in training CNNs). For any  $\epsilon >0$ , under Condition 3.1, with probability at least  $1 - \delta$  there exists  $t = \widetilde{O} (\eta^{-1}\epsilon^{-1}mnd^{-1}P^{-2}\sigma_p^{-2})$  such that:
1. The training loss converges, i.e.,  $L_{S}(\mathbf{W}^{(t)})\leq \epsilon$
2. When  $n\| \pmb{\mu}\|_2^4 \geq C_1 dP^4 \sigma_p^4$ , the test error  $L_{\mathcal{D}}^{0-1}(\mathbf{W}^{(t)}) \leq p + \epsilon$ .
3. When  $n\| \pmb {\mu}\| _2^4\leq C_3dP^4\sigma_p^4$  , the test error  $L_{\mathcal{D}}^{0 - 1}(\mathbf{W}^{(t)})\geq p + 0.1.$
Theorem 3.2 reveals a sharp phase transition between benign and harmful overfitting for CNN trained with SGD. This transition is determined by the relative scale of the signal strength and the data dimension. Specifically, if the signal is relatively large such that  $n\|\pmb{\mu}\|_2^4 \geq C_1 d(P - 1)^4 \sigma_p^4$ , the model can efficiently learn the signal. As a result, the test error decreases, approaching the Bayes risk  $p$ , although the presence of label flipping noise prevents the test error from reaching zero. Conversely, when the condition  $n\|\pmb{\mu}\|_2^4 \leq C_3 d(P - 1)^4 \sigma_p^4$  holds, the test error fails to approach the Bayes risk. This phase transition is empirically illustrated in Figure 2. In both scenarios, the model is capable of fitting the training data thoroughly, even for examples with flipped labels. This finding aligns with longstanding empirical observations.
The negative result of SGD, which encompasses the third point of Theorem 3.2 and the high test error observed in Figure 2, suggests that the signal strength needs to scale with the data dimension to enable benign overfitting. This constraint substantially undermines the efficiency of SGD, particularly when dealing with high-dimensional data. A significant part of this limitation stems from the fact that SGD does not inhibit the model from learning noise, leading to a comparable rate of signal and noise learning during iterative model parameter updates. This inherent limitation of SGD is effectively addressed by SAM, as we will discuss later in Section 4.
3.1 Analysis of Mini-Batch SGD
In contrast to GD, SGD does not utilize all the training data at each iteration. Consequently, different samples may contribute to parameters differently, leading to possible unbalancing in parameters. To analyze SGD, we extend the signal-noise decomposition technique developed by Kou et al. (2023); Cao et al. (2022) for GD, which in our case is formally defined as:
Lemma 3.3. Let  $\mathbf{w}_{j,r}^{(t,b)}$  for  $j\in \{\pm 1\}$ ,  $r\in [m]$  be the convolution filters of the CNN at the b-th batch of t-th epoch of gradient descent. Then there exist unique coefficients  $\gamma_{j,r}^{(t,b)}$  and  $\rho_{j,r,i}^{(t,b)}$  such that
$$
\mathbf {w} _ {j, r} ^ {(t, b)} = \mathbf {w} _ {j, r} ^ {(0, 0)} + j \cdot \gamma_ {j, r} ^ {(t, b)} \cdot \| \boldsymbol {\mu} \| _ {2} ^ {- 2} \cdot \boldsymbol {\mu} + \frac {1}{P - 1} \sum_ {i = 1} ^ {n} \rho_ {j, r, i} ^ {(t, b)} \cdot \| \boldsymbol {\xi} _ {i} \| _ {2} ^ {- 2} \cdot \boldsymbol {\xi} _ {i}. \tag {6}
$$
Further denote  $\overline{\rho}_{j,r,i}^{(t,b)}\coloneqq \rho_{j,r,i}^{(t,b)}\mathbb{1}(\rho_{j,r,i}^{(t,b)}\geq 0)$ ,  $\underline{\rho}_{j,r,i}^{(t,b)}\coloneqq \rho_{j,r,i}^{(t,b)}\mathbb{1}(\rho_{j,r,i}^{(t,b)}\leq 0)$ . Then
$$
\mathbf {w} _ {j, r} ^ {(t, b)} = \mathbf {w} _ {j, r} ^ {(0, 0)} + j \gamma_ {j, r} ^ {(t, b)} \| \boldsymbol {\mu} \| _ {2} ^ {- 2} \boldsymbol {\mu} + \frac {1}{P - 1} \sum_ {i = 1} ^ {n} \bar {\rho} _ {j, r, i} ^ {(t, b)} \| \boldsymbol {\xi} _ {i} \| _ {2} ^ {- 2} \boldsymbol {\xi} _ {i} + \frac {1}{P - 1} \sum_ {i = 1} ^ {n} \underline {{\rho}} _ {j, r, i} ^ {(t, b)} \| \boldsymbol {\xi} _ {i} \| _ {2} ^ {- 2} \boldsymbol {\xi} _ {i}. \tag {7}
$$
Note that (7) is a variant of (6): by decomposing the coefficient  $\rho_{j,r,i}^{(t,b)}$  into  $\overline{\rho}_{j,r,i}^{(t,b)}$  and  $\underline{\rho}_{j,r,i}^{(t,b)}$ , we can streamline our proof process. The normalization terms  $\frac{1}{P - 1},\| \boldsymbol {\mu}\| _2^{-2}$ , and  $\| \pmb {\xi}_i\| _2^{-2}$  ensure that  $\gamma_{j,r}^{(t,b)}\approx \langle \mathbf{w}_{j,r}^{(t,b)},\boldsymbol {\mu}\rangle$  and  $\rho_{j,r}^{(t,b)}\approx (P - 1)\langle \mathbf{w}_{j,r}^{(t,b)},\pmb {\xi}_i\rangle$ . Through signal-noise decomposition, we characterize the learning progress of signal  $\pmb{\mu}$  using  $\gamma_{j,r}^{(t,b)}$ , and the learning progress of noise using  $\rho_{j,r}^{(t,b)}$ . This decomposition turns the analysis of SGD updates into the analysis of signal noise coefficients. Kou et al. (2023) extend this technique to the ReLU activation function as well as in the presence of label flipping noise. However, mini-batch SGD updates amplify the complications introduced by label flipping noise, making it more difficult to ensure learning. We have developed advanced methods for coefficient balancing and activation pattern analysis. These techniques will be thoroughly discussed in the sequel. The progress of signal learning is characterized by  $\gamma_{j,r}^{(t,b)}$ , whose update rule is as follows: