SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\ell_ {i} ^ {(t)} = \Theta \left(\exp \left(- y _ {i} f \left(\mathbf {W} ^ {(t)}, \mathbf {x} _ {i}\right)\right)\right) = \Theta \left(\exp \left(F _ {- y _ {i}} \left(\mathbf {W} _ {- y _ {i}} ^ {(t)}, \mathbf {x} _ {i}\right) - F _ {y _ {i}} \left(\mathbf {W} _ {y _ {i}} ^ {(t)}, \mathbf {x} _ {i}\right)\right)\right), \tag {5.8}
$$
$$
\left| F _ {j} \left(\mathbf {W} _ {j} ^ {(t)}, \mathbf {x} _ {i}\right) - \frac {1}{m} \sum_ {r = 1} ^ {m} \rho_ {j, r, i} ^ {(t)} \right| \leq \sum_ {i ^ {\prime} \neq i} \left(\frac {1}{m} \sum_ {r = 1} ^ {m} | \rho_ {j, r, i ^ {\prime}} ^ {(t)} | R _ {\min } ^ {- 2} p\right), \tag {5.9}
$$
From (5.9), one can see that we need to decouple  $\ell_i^{(t)}$  from  $|\rho_{j,r,i'}^{(t)}|(i' \neq i)$ . In order to accomplish this, we also prove the following lemma, which demonstrates that the ratio between  $\sum_{r=1}^{m} |\rho_{j,r,i}'|$  and  $\sum_{r=1}^{m} |\rho_{j,r,i'}^{(t)}|(i' \neq i)$  will maintain a constant order throughout the training process. Here, we present the lemma for leaky ReLU networks.
Lemma 5.4 (leaky ReLU automatic balance). For two-layer leaky ReLU network defined in (3.1), for any  $t \geq 0$ , we have  $\sum_{r=1}^{m} |\rho_{j,r,i}^{(t)}| \geq c\gamma^2 \sum_{r=1}^{m} |\rho_{j,r,i'}^{(t)}|$  for any  $j \in \{\pm 1\}$  and  $i, i' \in [n]$ , where  $c$  is a constant.
By Lemma 5.4, we can approximate the neural network output using (5.9). This approximation expresses the output  $F_{j}(\mathbf{W}_{j}^{(t)},\mathbf{x}_{i})$  as a sum of the coefficients  $\rho_{j,r,i}^{(t)}$ :
$$
F _ {j} \left(\mathbf {W} _ {j} ^ {(t)}, \mathbf {x} _ {i}\right) \approx \frac {1 \pm c \gamma^ {2} R _ {\operatorname* {m i n}} ^ {- 2} p n}{m} \sum_ {r = 1} ^ {m} \rho_ {j, r, i} ^ {(t)}. \tag {5.10}
$$
By combining (5.4), (5.5), (5.8), and (5.10), we obtain the following relationship:
$$
\frac {1}{m} \sum_ {r = 1} ^ {m} | \rho_ {j, r, i} ^ {(t + 1)} | - \frac {1}{m} \sum_ {r = 1} ^ {m} | \rho_ {j, r, i} ^ {(t)} | = \Theta \Big (\frac {\eta \| \mathbf {x} _ {i} \| _ {2} ^ {2}}{n m} \Big) \cdot \exp \Big (- \frac {1 \pm c \gamma^ {2} R _ {\mathrm {m i n}} ^ {- 2} p n}{m} \sum_ {r = 1} ^ {m} | \rho_ {j, r, i} ^ {(t)} | \Big).
$$
This relationship aligns with the form of (5.6), if we set  $x_{t} = \frac{1\pm c\gamma^{2}R_{\mathrm{min}}^{-2}pn}{m}\sum_{r = 1}^{m}|\rho_{j,r,i}^{(t)}|$ . Thus, we can directly apply Lemma 5.3 to gain insights into the logarithmic rate of increase for the average
magnitudes of the coefficients  $\frac{1}{m}\sum_{r = 1}^{m}|\rho_{j,r,i}^{(t)}|$ , which in turn implies that  $\| \mathbf{w}_{j,r}^{(t)}\| _2 = \Theta (\log t)$  and  $\| \mathbf{W}^{(t)}\| _F = \Theta (\log t)$ . In the case of ReLU networks, we have the following lemma that provides automatic balance:
Lemma 5.5 (ReLU automatic balance). For two-layer ReLU network defined in (3.1), there exists a constant  $c$  such that for any  $t \geq 0$ , we have  $|\rho_{y_i, r, i}^{(t)}| \geq c |\rho_{j, r', i'}^{(t)}|$  for any  $j \in \{\pm 1\}$ ,  $r \in S_i^{(0)} := \{r \in [m] : \langle \mathbf{w}_{y_i, r}^{(0)}, \mathbf{x}_i \rangle \geq 0\}$ ,  $r' \in [m]$  and  $i, i' \in [n]$ .
The automatic balance lemma guarantees that the magnitudes of coefficients related to the neurons of class  $y_{i}$ , which are activated by  $\mathbf{x}_i$  during initialization, dominate those of other classes. With the help of Lemma 5.5, we can get the following approximation for the margin  $y_{i}f(\mathbf{W}^{(t)},\mathbf{x}_{i})$ :
$$
F _ {y _ {i}} \left(\mathbf {W} _ {y _ {i}} ^ {(t)}, \mathbf {x} _ {i}\right) - F _ {- y _ {i}} \left(\mathbf {W} _ {- y _ {i}} ^ {(t)}, \mathbf {x} _ {i}\right) \approx \frac {1 \pm c R _ {\operatorname* {m i n}} ^ {- 2} p n}{m} \sum_ {r \in S _ {i} ^ {(0)}} \rho_ {y _ {i}, r, i} ^ {(t)}. \tag {5.11}
$$
By combining (5.4), (5.5), (5.8) and (5.11), we obtain the following relationship:
$$
\sum_ {r \in S _ {i} ^ {(0)}} | \rho_ {y _ {i}, r, i} ^ {(t + 1)} | - \sum_ {r \in S _ {i} ^ {(0)}} | \rho_ {y _ {i}, r, i} ^ {(t)} | = \Theta \Big (\frac {\eta \| \mathbf {x} _ {i} \| _ {2} ^ {2} | S _ {i} ^ {(0)} |}{n m} \Big) \cdot \exp \Big (- \frac {1 \pm c R _ {\mathrm {m i n}} ^ {- 2} p n}{m} \sum_ {r \in S _ {i} ^ {(0)}} \rho_ {y _ {i}, r, i} ^ {(t)} \Big),
$$
which precisely matches the form of (5.6) by setting  $x_{t} = \frac{1\pm cR_{\min}^{-2}pn}{m}\sum_{r\in S_{i}^{(0)}}\rho_{y_{i},r,i}^{(t)}$ . Therefore, we can directly apply Lemma 5.3 and obtain the logarithmic increasing rate of  $|\rho_{y_i,r,i}^{(t)}|$  for  $r\in S_i^{(0)}$ . Consequently, this implies that  $\| \mathbf{W}^{(t)}\| _F = \Theta (\log t)$ .
5.2 Analysis of Activation Pattern
One notable previous work (Frei et al., 2022b) provided a constant upper bound for the stable rank of two-layer smoothed leaky ReLU networks trained by gradient descent in their Theorem 4.2. To achieve a better stable rank bound, we characterize the activation pattern of leaky ReLU network neurons after a certain threshold time  $T$  in the following lemma.
Lemma 5.6 (leaky ReLU activation pattern). Let  $T = C\eta^{-1}nmR_{\max}^{-2}$ . For two-layer leaky ReLU network defined in (3.1), for any  $t \geq T$ , it holds that  $\mathrm{sign}(\langle \mathbf{w}_{j,r}^{(t)}, \mathbf{x}_i \rangle) = jy_i$  for any  $j \in \{\pm 1\}$  and  $r \in [m]$ .
Lemma 5.6 indicates that the activation pattern will not change after time  $T$ . Given Lemma 5.6, we can get  $\sigma'(\langle \mathbf{w}_{j,r}^{(t)}, \mathbf{x}_i \rangle) = \gamma$  for  $j \neq y_i$  and  $\sigma'(\langle \mathbf{w}_{j,r}^{(t)}, \mathbf{x}_i \rangle) = 1$  for  $j = y_i$ . Plugging this into (5.4) and (5.5) can give the following useful lemma.
Lemma 5.7. Let  $T$  be defined in Lemma 5.6. For  $t \geq T$ , it holds that
$$
\bar {\rho} _ {y _ {i}, r, i} ^ {(t)} - \bar {\rho} _ {y _ {i}, r, i} ^ {(T)} = \bar {\rho} _ {y _ {i}, r ^ {\prime}, i} ^ {(t)} - \bar {\rho} _ {y _ {i}, r ^ {\prime}, i} ^ {(T)}, \underline {{\rho}} _ {- y _ {i}, r, i} ^ {(t)} - \underline {{\rho}} _ {- y _ {i}, r, i} ^ {(T)} = \underline {{\rho}} _ {- y _ {i}, r ^ {\prime}, i} ^ {(t)} - \underline {{\rho}} _ {- y _ {i}, r ^ {\prime}, i} ^ {(T)},
$$
$$
\overline {{\rho}} _ {y _ {i}, r, i} ^ {(t)} - \overline {{\rho}} _ {y _ {i}, r, i} ^ {(T)} = (\underline {{\rho}} _ {- y _ {i}, r ^ {\prime}, i} ^ {(t)} - \underline {{\rho}} _ {- y _ {i}, r ^ {\prime}, i} ^ {(T)}) / \gamma ,
$$
for any  $i\in [n]$  and  $r,r^{\prime}\in [m]$
This lemma reveals that beyond a certain time threshold  $T$ , the increase in  $\rho_{j,r,i}^{(t)}$  is consistent across neurons within the same positive or negative class. However, for neurons belonging to the opposite class, this increment in  $\rho_{j,r,i}^{(t)}$  is scaled by a factor equivalent to the slope of the leaky ReLU function  $\gamma$ . From this and (5.1), we can demonstrate that  $\| \mathbf{w}_{j,r}^{(t)} - \mathbf{w}_{j,r'}^{(t)} \|_2 (r \neq r')$  can be upper bounded by a constant, leading to the following inequalities:
$$
\| \mathbf {W} _ {j} ^ {(t)} \| _ {F} ^ {2} \leq m \| \mathbf {w} _ {j, 1} ^ {(t)} \| _ {2} ^ {2} + m C _ {1} \| \mathbf {w} _ {j, 1} ^ {(t)} \| _ {2} + m C _ {2}, \| \mathbf {W} _ {j} ^ {(t)} \| _ {2} ^ {2} \geq m \| \mathbf {w} _ {j, 1} ^ {(t)} \| _ {2} ^ {2} - m C _ {3} \| \mathbf {w} _ {j, 1} ^ {(t)} \| _ {2} - m C _ {4}.
$$
Considering that  $\| \mathbf{w}_{j,r}^{(t)}\| _2 = \Theta (\log t)$ , the stable rank of  $\mathbf{W}_j^{(t)}$  naturally converges to a value of 1. For ReLU networks, we can partially characterize the activation pattern as illustrated in the following lemma.
Lemma 5.8. (ReLU activation pattern) For two-layer ReLU networks defined in (3.1), for any  $i \in [n]$ , we have  $S_i^{(t)} \subseteq S_i^{(t+1)}$  for any  $t \geq 0$ , where  $S_i^{(t)} := \{r \in [m] : \langle \mathbf{w}_{y_i, r}^{(t)}, \mathbf{x}_i \rangle \geq 0\}$ .
Lemma 5.8 suggests that once the neuron of class  $y_{i}$  is activated by  $\mathbf{x}_i$ , it will remain activated throughout the training process. Leveraging such an activation pattern, we can establish a lower bound for  $\| \mathbf{W}_j^{(t)}\|_2$  as  $\Omega(\log(t))$ . Together with the trivial upper bound for  $\| \mathbf{W}_j^{(t)}\|_F$  of order  $O(\log(t))$ , it provides a constant upper bound for the stable rank of ReLU network weight.
5.3 Analysis of Margin and Training Loss
Notably, Lyu and Li (2019) established in their Theorem 4.4 that any limit point of smooth homogeneous neural networks  $f(\mathbf{W}, \mathbf{x})$  trained by gradient descent is along the direction of a KKT point for the max-margin problem (4.1). Additionally, Lyu and Li (2019) provided precise bounds on the training loss and weight norm for smooth homogeneous neural networks in their Theorem 4.3 as follows:
$$
L _ {S} (\mathbf {W} ^ {(t)}) = \Theta \Big (\frac {1}{t (\log t) ^ {2 - 2 / L}} \Big), \qquad \| \mathbf {W} ^ {(t)} \| _ {F} = \Theta \big ((\log t) ^ {1 / L} \big),
$$
where  $L$  is the order of the homogeneous network satisfying the property  $f(c\mathbf{W},\mathbf{x}) = c^{L}f(\mathbf{W},\mathbf{x})$  for all  $c > 0$ ,  $\mathbf{W}$ , and  $\mathbf{x}$ . It is worth noting that the two-layer (leaky) ReLU neural network analyzed in this paper is 1-homogeneous but not smooth. In Section 5.1, we have already demonstrated that  $\| \mathbf{W}^{(t)}\|_F = \Theta (\log t)$ , and in this subsection, we will discuss the proof technique employed to show a convergence rate of  $\Theta (t^{-1})$  for the loss and establish the same normalized margin for all the training data points asymptotically. These results align with those presented by Lyu and Li (2019) regarding smooth homogeneous networks.
By the nearly orthogonal property, we can bound the increment of margin as follows:
$$
\frac {\eta}{5 n m} \cdot | \ell_ {i} ^ {\prime (t)} | \cdot \| \mathbf {x} _ {i} \| _ {2} ^ {2} \leq y _ {i} f \left(\mathbf {W} ^ {(t + 1)}, \mathbf {x} _ {i}\right) - y _ {i} f \left(\mathbf {W} ^ {(t)}, \mathbf {x} _ {i}\right) \leq \frac {3 \eta}{n m} \cdot \left| \ell_ {i} ^ {\prime (t)} \right| \| \mathbf {x} _ {i} \| _ {2} ^ {2}. \tag {5.12}
$$
Given (5.8) and (5.12), we can apply Lemma 5.3 and obtain
$$
\left| y _ {i} f \left(\mathbf {W} ^ {(t)}, \mathbf {x} _ {i}\right) - \log t - \log \left(\eta \| \mathbf {x} _ {i} \| _ {2} ^ {2} / n m\right) \right| \leq C _ {3}, \tag {5.13}
$$
where  $C_3$  is a constant. Utilizing (5.13) and the inequality  $z - z^2 / 2 \leq \log(1 + z) \leq z$  for  $z \geq 0$ , we can derive:
$$
\begin{array}{l} L _ {S} \left(\mathbf {W} ^ {(t)}\right) \leq \frac {1}{n} \sum_ {i = 1} ^ {n} \exp \left(- y _ {i} f \left(\mathbf {W} ^ {(t)}, \mathbf {x} _ {i}\right)\right) \\ \leq \frac {1}{n} \sum_ {i = 1} ^ {n} \exp \left(- \log t - \log (\eta \| \mathbf {x} _ {i} \| _ {2} ^ {2} / n m) + C _ {3}\right) = O (t ^ {- 1}), \\ \end{array}
$$
$$
L _ {S} (\mathbf {W} ^ {(t)}) \geq \frac {1}{n} \sum_ {i = 1} ^ {n} \exp \big (- y _ {i} f (\mathbf {W} ^ {(t)}, \mathbf {x} _ {i}) \big) - \exp \big (- 2 y _ {i} f (\mathbf {W} ^ {(t)}, \mathbf {x} _ {i}) \big) = \Omega (t ^ {- 1}).
$$
To demonstrate that all the training data points attain the same normalized margin as  $t$  goes to infinity, we first observe that (5.12) provides the following bounds for the increment of margin difference:
$$
\begin{array}{l} y _ {k} f \left(\mathbf {W} ^ {(t + 1)}, \mathbf {x} _ {k}\right) - y _ {i} f \left(\mathbf {W} ^ {(t + 1)}, \mathbf {x} _ {i}\right) \\ \leq y _ {k} f \left(\mathbf {W} ^ {(t)}, \mathbf {x} _ {k}\right) - y _ {i} f \left(\mathbf {W} ^ {(t)}, \mathbf {x} _ {i}\right) + \frac {3 \eta}{n m} \cdot \left| \ell_ {k} ^ {\prime (t)} \right| \cdot \| \mathbf {x} _ {k} \| _ {2} ^ {2} - \frac {\eta}{5 n m} \cdot \left| \ell_ {i} ^ {\prime (t)} \right| \cdot \| \mathbf {x} _ {i} \| _ {2} ^ {2}. \tag {5.14} \\ \end{array}
$$
Now, we consider two cases:
- If the ratio  $\left| {\ell }_{i}^{\left( t\right) }\right| /\left| {\ell }_{k}^{\left( t\right) }\right|$  is relatively large,then  ${y}_{k}f\left( {{\mathbf{W}}^{\left( t\right) },{\mathbf{x}}_{k}}\right)  - {y}_{i}f\left( {{\mathbf{W}}^{\left( t\right) },{\mathbf{x}}_{i}}\right)$  will not increase.
- If the ratio  $|\ell_i^{\prime (t)}| / |\ell_k^{\prime (t)}|$  is relatively small, then  $y_{k}f(\mathbf{W}^{(t)},\mathbf{x}_{k}) - y_{i}f(\mathbf{W}^{(t)},\mathbf{x}_{i})$  will also be relatively small. In fact, it can be bounded by a constant due to the fact that  $|\ell_i^{\prime (t)}| / |\ell_k^{\prime (t)}|$  can be approximated by  $\exp (y_kf(\mathbf{W}^{(t)},\mathbf{x}_k) - y_if(\mathbf{W}^{(t)},\mathbf{x}_i))$ . By (5.14), we can show that
Figure 1: Stable ranks and training loss for different leaky ReLU slopes  $\gamma$  across multiple runs. A slope of 1 corresponds to linear activation, while a slope of 0 corresponds to ReLU activation. Each line represents the mean stable rank or training loss for a given leaky ReLU slope, while the shaded regions indicate the variability of the values ( $\pm 3$  times the standard deviation) across the 5 runs.
$y_{k}f(\mathbf{W}^{(t + 1)},\mathbf{x}_{k}) - y_{i}f(\mathbf{W}^{(t + 1)},\mathbf{x}_{i})$  can also be bounded by a constant, provided that the learning rate  $\eta$  is sufficiently small.
By combining both cases, we can conclude that both  $|\ell_i^{\prime (t)}| / |\ell_k^{\prime (t)}|$  and  $y_{k}f(\mathbf{W}^{(t)},\mathbf{x}_{k}) - y_{i}f(\mathbf{W}^{(t)},\mathbf{x}_{i})$  can be bounded by constants. This result is formally stated in the following lemma.
Lemma 5.9. For two-layer neural networks defined in (3.1) with (leaky) ReLU activation, the following bounds hold for any  $t \geq 0$ :
$$
y _ {i} f \left(\mathbf {W} ^ {(t)}, \mathbf {x} _ {i}\right) - y _ {k} f \left(\mathbf {W} ^ {(t)}, \mathbf {x} _ {k}\right) \leq C _ {1}, \quad \ell_ {i} ^ {\prime (t)} / \ell_ {k} ^ {\prime (t)} \leq C _ {2}, \tag {5.15}
$$
for any  $i, k \in [n]$ , where  $C_1, C_2$  are positive constants.
By Lemma 5.9, which shows that the difference between the margins of any two data points can be bounded by a constant, and taking into account that  $\| \mathbf{W}^{(t)}\| _F = \Theta (\log t)$ , we can deduce the following result:
$$
\lim _ {t \rightarrow \infty} \left| y _ {i} f (\mathbf {W} ^ {(t)} / \| \mathbf {W} ^ {(t)} \| _ {F}, \mathbf {x} _ {i}) - y _ {k} f (\mathbf {W} ^ {(t)} / \| \mathbf {W} ^ {(t)} \| _ {F}, \mathbf {x} _ {k}) \right| = 0, \forall i, k \in [ n ].
$$
This demonstrates that gradient descent will asymptotically find a neural network in which all the training data points achieve the same normalized margin.
In this section, we present simulations of both synthetic and real data to back up our theoretical analysis in the previous section.
Synthetic-data experiments. Here we generate a synthetic mixture of Gaussian data as follows: Let  $\pmb{\mu} \in \mathbb{R}^d$  be a fixed vector representing the signal contained in each data point. Each data point  $(\mathbf{x}, y)$  with predictor  $\mathbf{x} \in \mathbb{R}^d$  and label  $y \in \{-1, 1\}$  is generated from a distribution  $\mathcal{D}$ , which we specify as follows:
1. The label  $y$  is generated as a Rademacher random variable, i.e.  $\mathbb{P}[y = 1] = \mathbb{P}[y = -1] = 1 / 2$ .
2. A noise vector  $\xi$  is generated from the Gaussian distribution  $\mathcal{N}(\mathbf{0},\sigma_p^2\mathbf{I}_d)$ . And  $\mathbf{x}$  is assigned as  $y\cdot \pmb {\mu} + \pmb{\xi}$  where  $\pmb{\mu}$  is a fixed feature vector.
Specifically, we set training data size  $n = 10$ ,  $d = 784$  and train the NN with gradient descent using learning rate 0.1 for 50 epochs. We set  $\pmb{\mu}$  to be a feature randomly drawn from  $\mathcal{N}(0,10^{-4}\mathbf{I}_d)$ . We then generate the noise vector  $\pmb{\xi}$  from the Gaussian distribution  $\mathcal{N}(\mathbf{0},\sigma_p^2\mathbf{I})$  with fixed standard deviation  $\sigma_{p} = 1$ . We train the FNN model defined in Section 3 with ReLU (or leaky-RelU) activation function and width  $m = 100$ . As we can infer from Figure 1, the stable rank will decrease faster for larger leaky ReLU slopes and have a smaller value when epoch  $t\to \infty$ .
Real-data experiments on MNIST dataset. Here we train a two-layer feed-forward neural network defined in Section 3 with ReLU (or leaky-ReLU) functions. The number of widths is
Figure 2: Stable ranks and test errors for different weight variances across multiple runs (ReLU Activation Function). Each line represents the mean stable rank or test accuracy for a given weight variance, while the shaded regions indicate the variability of the values ( $\pm 3$  times the standard deviation) across the 5 runs.
Figure 3: Stable ranks and test errors for different weight variances across multiple runs (leaky-ReLU Activation Function with slope 0.1). Each line represents the mean stable rank or test accuracy for a given weight variance, while the shaded regions indicate the variability of the values ( $\pm 3$  times the standard deviation) across the 5 runs.
set as  $m = 1000$ . We use the Gaussian initialization and consider different weight variance  $\sigma_0 \in \{0.00001, 0.00005, 0.0001, 0.0005, 0.001\}$ . We train the NN with stochastic gradient descent with batch size 64 and learning rate 0.1 for 10 epochs. As we can infer from Figures 2 and 3, the stable rank of ReLU or leaky ReLU networks will largely depend on the initialization and the training time. When initialization is sufficiently small, the stable rank will quickly decrease to a small value compared to its initialization values.
This paper employs a data-correlated decomposition technique to examine the implicit bias of two-layer ReLU and Leaky ReLU networks trained using gradient descent. By analyzing the training dynamics, we provide precise characterizations of the weight matrix stable rank limits for both ReLU and Leaky ReLU cases, demonstrating that both scenarios will yield a network with a low stable rank. Additionally, we present an analysis for the convergence rate of the loss function. An important future work is to investigate the directional convergence of the weight matrix in neural networks trained via gradient descent, which is essential to prove the convergence to a KKT point of the max-margin problem. Furthermore, it is important to extend our analysis to fully understand the neuron activation patterns in ReLU networks. Specifically, we will explore whether certain neurons will switch their activation patterns by an infinite number of times throughout the training or if the activation patterns stabilize after a certain number of gradient descent iterations.
We thank the anonymous reviewers and area chair for their helpful comments. YK, ZC, and QG are supported in part by the National Science Foundation CAREER Award 1906169 and IIS-2008981, and the Sloan Research Fellowship. The views and conclusions contained in this paper are those of the authors and should not be interpreted as representing any funding agencies.
LYU, K. and LI, J. (2019). Gradient descent maximizes the margin of homogeneous neural networks. arXiv preprint arXiv:1906.05890.
LYU, K., LI, Z., WANG, R. and ARORA, S. (2021). Gradient descent on two-layer nets: Margin maximization and simplicity bias. Advances in Neural Information Processing Systems 34 12978-12991.
MEI, S. and MONTANARI, A. (2019). The generalization error of random features regression: Precise asymptotics and double descent curve. arXiv preprint arXiv:1908.05355.
PHUONG, M. and LAMPERT, C. H. (2021). The inductive bias of relu networks on orthogonally separable data. In International Conference on Learning Representations.
RAZIN, N. and COHEN, N. (2020). Implicit regularization in deep learning may not be explainable by norms. Advances in neural information processing systems 33 21174-21187.
SAFRAN, I., VARDI, G. and LEE, J. D. (2022). On the effective number of linear regions in shallow univariate relu networks: Convergence guarantees and implicit bias. arXiv preprint arXiv:2205.09072.
SARUSSI, R., BRUTZKUS, A. and GLOBERSON, A. (2021). Towards understanding learning in neural networks with linear teachers. In International Conference on Machine Learning. PMLR.
SHEN, R., BUBECK, S. and GUNASEKAR, S. (2022). Data augmentation as feature manipulation: a story of desert cows and grass cows. arXiv preprint arXiv:2203.01572.
Soudry, D., Hoffer, E., Nacson, M. S., GUNASEKAR, S. and SREBRO, N. (2018). The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research 19 2822-2878.
TIMOR, N., VARDI, G. and SHAMIR, O. (2023). Implicit regularization towards rank minimization in relu networks. In International Conference on Algorithmic Learning Theory. PMLR.
VARDI, G. (2022). On the implicit bias in deep-learning algorithms. arXiv preprint arXiv:2208.12591.
VARDI, G., SHAMIR, O. and SREBRO, N. (2022a). On margin maximization in linear and relu networks. Advances in Neural Information Processing Systems 35 37024-37036.
VARDI, G., YEHUDAI, G. and SHAMIR, O. (2022b). Gradient methods provably converge to non-robust networks. arXiv preprint arXiv:2202.04347.
VERSHYNIN, R. (2018). High-Dimensional Probability: An Introduction with Applications in Data Science. Cambridge Series in Statistical and Probabilistic Mathematics, Cambridge University Press.
WANG, K. and THRAMPOULIDIS, C. (2022). Binary classification of gaussian mixtures: Abundance of support vectors, benign overfitting, and regularization. SIAM Journal on Mathematics of Data Science 4 260-284.
WU, D. and Xu, J. (2020). On the optimal weighted  $\ell_2$  regularization in overparameterized linear regression. Advances in Neural Information Processing Systems 33.
In this section, we conduct additional experiments on the nearly orthogonal and MINIST datasets.
A.1 Additional Experiment on Nearly Orthogonal Dataset
In this subsection, we conduct additional experiments on a nearly orthogonal dataset for long epochs to support our main Theorems 4.1 and 4.3.
Figure 4: (Left) Stable ranks for different leaky ReLU slopes  $\gamma$  across multiple runs. A slope of 1 corresponds to linear activation, while a slope of 0 corresponds to ReLU activation. Each line represents the mean stable rank for a given leaky ReLU slope, while the shaded regions indicate the variability of the values ( $\pm 3$  times the standard deviation) across the 20 runs. The red dashed line indicates a stable rank of 1. (Right) The difference between stable rank and 1 from the left figure is visualized on a semilog y-axis.
Figure 5: fully orthogonal data: (Left) Stable ranks for different leaky ReLU slopes  $\gamma$  across multiple runs. A slope of 1 corresponds to linear activation, while a slope of 0 corresponds to ReLU activation. Each line represents the mean stable rank for a given leaky ReLU slope, while the shaded regions indicate the variability of the values ( $\pm 3$  times the standard deviation) across the 20 runs. The red dashed line indicates a stable rank of 1. (Right) The difference between stable rank and 1 from the left figure is visualized on a semilog y-axis.
Figure 4: Under the same setting of the synthetic data introduced in Section 6, we train the NN with full batch gradient descent with a learning rate 0.1 for 1000 epochs. We set  $\pmb{\mu}$  to be a feature randomly drawn from  $\mathcal{N}(0,10^{-4}\mathbf{I}_d)$ . We then generate the noise vector  $\xi$  from the Gaussian distribution  $\mathcal{N}(\mathbf{0},\sigma_p^2\mathbf{I}_d)$  with fixed standard deviation  $\sigma_{p} = 1$ . We train the FNN model defined in Section 3 with ReLU (or leaky-ReLU) activation function and width  $m = 100$ . As we can see from Figure 4, the stable rank for the leaky ReLU network with large slopes  $\gamma$  will converge to 1 when epoch  $t\to \infty$ . In comparison, the stable rank for the ReLU network will not converge to 1.
Figure 5: To further illustrate the behavior of the ReLU network, we generate the synthetic training data with fully orthogonal input. Each data point  $(\mathbf{x},y)$  with input  $\mathbf{x}\in \mathbb{R}^d$  and label  $y\in \{-1,1\}$  is generated from a distribution  $\mathcal{D}$ , which we specify as follows:
1. The label  $y$  is generated as a Rademacher random variable, i.e.,  $\mathbb{P}[y = 1] = \mathbb{P}[y = -1] = 1 / 2$ .
2. Input  $\mathbf{x}$  is randomly generated from the basis  $\{\mathbf{e}_1,\mathbf{e}_2,\dots ,\mathbf{e}_d\}$ .