Title: Implicit Bias of Gradient Descent for Two-layer ReLU and Leaky ReLU Networks on Nearly-orthogonal Data
Venue: NeurIPS 2023
Accepted: yes
Keywords: ReLU Neural Networks

Implicit Bias

Deep Learning Theory
Abstract: The implicit bias towards solutions with favorable properties is believed to be a key reason why neural networks trained by gradient-based optimization can generalize well. While the implicit bias of gradient flow has been widely studied for homogeneous neural networks (including ReLU and leaky ReLU networks), the implicit bias of gradient descent is currently only understood for smooth neural networks. Therefore, implicit bias in non-smooth neural networks trained by gradient descent remains an open question. In this paper, we aim to answer this question by studying the implicit bias of gradient descent for training two-layer fully connected (leaky) ReLU neural networks. We showed that when the training data are nearly-orthogonal, for leaky ReLU activation function, gradient descent will find a network with a stable rank that converges to $1$, whereas for ReLU activation function, gradient descent will find a neural network with a stable rank that is upper bounded by a constant. Additionally, we show that gradient descent will find a neural network such that all the training data points have the same normalized margin asymptotically. Experiments on both synthetic and real data backup our theoretical findings.

Implicit Bias of Gradient Descent for Two-layer ReLU and Leaky ReLU Networks on Nearly-orthogonal Data

Abstract

The implicit bias towards solutions with favorable properties is believed to be a key reason why neural networks trained by gradient-based optimization can generalize well. While the implicit bias of gradient flow has been widely studied for homogeneous neural networks (including ReLU and leaky ReLU networks), the implicit bias of gradient descent is currently only understood for smooth neural networks. Therefore, implicit bias in non-smooth neural networks trained by gradient descent remains an open question. In this paper, we aim to answer this question by studying the implicit bias of gradient descent for training two-layer fully connected (leaky) ReLU neural networks. We showed that when the training data are nearly-orthogonal, for leaky ReLU activation function, gradient descent will find a network with a stable rank that converges to 1 , whereas for ReLU activation function, gradient descent will find a neural network with a stable rank that is upper bounded by a constant. Additionally, we show that gradient descent will find a neural network such that all the training data points have the same normalized margin asymptotically. Experiments on both synthetic and real data backup our theoretical findings.

1 Introduction

Neural networks have achieved remarkable success in a variety of applications, such as image and speech recognition, natural language processing, and many others. Recent studies have revealed that the effectiveness of neural networks is attributed to their implicit bias towards particular solutions which enjoy favorable properties. Understanding how this bias is affected by factors such as network architecture, optimization algorithms and data used for training, has become an active research area in the field of deep learning theory.

The literature on the implicit bias in neural networks has expanded rapidly in recent years (Vardi, 2022), with numerous studies shedding light on the implicit bias of gradient flow (GF) with a wide range of neural network architecture, including deep linear networks (Ji and Telgarsky, 2018, 2020; Gunasekar et al., 2018), homogeneous networks (Lyu and Li, 2019; Vardi et al., 2022a) and more specific cases (Chizat and Bach, 2020; Lyu et al., 2021; Frei et al., 2022b; Safran et al., 2022). The implicit bias of gradient descent (GD), on the other hand, is better understood for linear predictors (Soudry et al., 2018) and smoothed neural networks (Lyu and Li, 2019; Frei et al., 2022b). Therefore, an open question still remains:

What is the implicit bias of leaky ReLU and ReLU networks trained by gradient descent?

In this paper, we will answer this question by investigating gradient descent for both two-layer leaky ReLU and ReLU neural networks on specific training data, where { x i } n i =1 are nearly-orthogonal

̸

(Frei et al., 2022b), i.e., ∥ x i ∥ 2 2 ≥ Cn max k = i |⟨ x i , x k ⟩| with a constant C . Our main results are summarized as follows:

For two-layer leaky ReLU networks trained by GD, we demonstrate that the neuron activation pattern reaches a stable state beyond a specific time threshold and provide rigorous proof of the convergence of the stable rank of the weight matrix to 1, matching the results of Frei et al. (2022b) regarding gradient flow.

For two-layer ReLU networks trained by GD, we proved that the stable rank of weight matrix can be upper bounded by a constant. Moreover, we present an illustrative example using completely orthogonal training data, showing that the stable rank of the weight matrix converges to a value approximately equal to 2 . To the best of our knowledge, this is the first implicit bias result for two-layer ReLU networks trained by gradient descent beyond the Karush-Kuhn-Tucker (KKT) point.

For both ReLU and leaky ReLU networks, we show that weight norm increases at the rate of Θ(log( t )) and the training loss converges to zero at the rate of Θ( t -1 ) , where t is the number of gradient descent iterations. This improves upon the O ( t -1 / 2 ) rate proved in Frei et al. (2022b) for the case of a two-layer smoothed leaky ReLU network trained by gradient descent and aligns with the results by Lyu and Li (2019) for smooth homogeneous networks. Additionally, we prove that gradient descent will find a neural network such that all the training data points have the same normalized margin asymptotically.

2 Related Work

Implicit bias in neural networks. Recent years have witnessed significant progress on implicit bias in neural networks trained by gradient flow (GF). Lyu and Li (2019) and Ji and Telgarsky (2020) demonstrated that homogeneous neural networks trained with exponentially-tailed classification losses converge in direction to the KKT point of a maximum-margin problem. Lyu et al. (2021) studied the implicit bias in two-layer leaky ReLU networks trained on linearly separable and symmetric data, showing that GF converges to a linear classifier maximizing the ℓ 2 margin. Frei et al. (2022b) showed that two-layer leaky ReLU networks trained by GF on nearly-orthogonal data produce a ℓ 2 -max-margin solution with a linear decision boundary and rank at most two. Other works studying the implicit bias of classification using GF in nonlinear two-layer networks include Chizat and Bach (2020); Phuong and Lampert (2021); Sarussi et al. (2021); Safran et al. (2022); Vardi et al. (2022a,b); Timor et al. (2023). Although implicit bias in neural networks trained by GF has been extensively studied, research on implicit bias in networks trained by gradient descent (GD) remains limited. Lyu and Li (2019) examined smoothed homogeneous neural network trained by GD with exponentially-tailed losses and proved a convergence to KKT points of a max-margin problem. Frei et al. (2022b) studied two-layer smoothed leaky ReLU trained by GD and revealed the implicit bias towards low-rank networks. Other works studying implicit bias towards rank minimization include Ji and Telgarsky (2018, 2020); Timor et al. (2023); Arora et al. (2019); Razin and Cohen (2020); Li et al. (2021). Lastly, Vardi (2022) provided a comprehensive literature survey on implicit bias.

Benign overfitting and double descent in neural networks. A parallel line of research aims to understand the benign overfitting phenomenon (Bartlett et al., 2020) of neural networks by considering a variety of models. For example, Allen-Zhu and Li (2020); Jelassi and Li (2022); Shen et al. (2022); Cao et al. (2022); Kou et al. (2023) studied the generalization performance of two-layer convolutional networks on patch-based data models. Several other papers studied high-dimensional mixture models (Chatterji and Long, 2021; Wang and Thrampoulidis, 2022; Cao et al., 2021; Frei et al., 2022a). Another thread of work Belkin et al. (2020); Hastie et al. (2022); Wu and Xu (2020); Mei and Montanari (2019); Liao et al. (2020) focuses on understanding the double descent phenomenon first empirically observed by Belkin et al. (2019).

3 Preliminaries

In this section, we introduce the notation, fully connected neural networks, the gradient descent-based training algorithm, and a data-coorrelated decomposition technique.

Notation. We use lower case letters, lower case bold face letters, and upper case bold face letters to denote scalars, vectors, and matrices respectively. For a vector v = ( v 1 , · · · , v d ) ⊤ , we denote by

∥ v ∥ 2 := ( ∑ d j =1 v 2 j ) 1 / 2 its ℓ 2 norm. For a matrix A ∈ R m × n , we use ∥ A ∥ F to denote its Frobenius norm and ∥ A ∥ 2 its spectral norm. We use sign( z ) as the function that is 1 when z > 0 and -1 otherwise. For a vector v ∈ R d , we use [ v ] i ∈ R to denote the i -th component of the vector. For two sequence { a k } and { b k } , we denote a k = O ( b k ) if | a k | ≤ C | b k | for some absolute constant C , denote a k = Ω( b k ) if b k = O ( a k ) , and denote a k = Θ( b k ) if a k = O ( b k ) and a k = Ω( b k ) . We also denote a k = o ( b k ) if lim | a k /b k | = 0 .

Two-layer fully connected neural newtork. We consider a two-layer neural network described as follows: its first layer consists of m positive neurons and m negative neurons; its second layer parameters are fixed as +1 /m and -1 /m respectively for positive and negative neurons. Then the network can be written as f ( W , x ) = F +1 ( W +1 , x ) -F -1 ( W -1 , x ) , where the partial network function of positive and negative neurons, i.e., F +1 ( W +1 , x ) , F -1 ( W -1 , x ) , are defined as:

for j ∈ {± 1 } . Here, σ ( z ) represents the activation function. For ReLU, σ ( z ) = max { 0 , z } , and for leaky ReLU, σ ( z ) = max { γz, z } , where γ ∈ (0 , 1) . W j ∈ R m × d is the collection of model weights associated with F j , and w j,r ∈ R d denotes the weight vector for the r -th neuron in W j . We use W to denote the collection of all model weights.

Gradient Descent. Given a training data set S = { ( x i , y i ) } n i =1 ⊆ R d ×{± 1 } , instead of considering the gradient flow (GF) that is commonly studied in prior work on the implicit bias, we use gradient descent (GD) to optimize the empirical loss on the training data

where ℓ ( z ) = log(1 + exp( -z )) is the logistic loss, and S = { ( x i , y i ) } n i =1 is the training data set. The gradient descent update rule of each neuron in the two-layer neural network can be written as

for all j ∈ {± 1 } and r ∈ [ m ] , where we introduce a shorthand notation ℓ ′ ( t ) i = ℓ ′ [ y i · f ( W ( t ) , x i )] and assume the derivative of the ReLU activation function at 0 is σ ′ (0) = 1 without loss of generality. Here η > 0 is the learning rate. We initialize the gradient descent by Gaussian initialization, where all the entries of W (0) are sampled from i.i.d. Gaussian distributions N (0 , σ 2 0 ) with σ 2 0 being the variance.

4 Main Results

In this section, we present our main theoretical results. For the training data set S = { ( x i , y i ) } n i =1 ⊆ R d × {± 1 } , let R min = min i ∥ x i ∥ 2 , R max = max i ∥ x i ∥ 2 , p = max i = k |⟨ x i , x k ⟩| , and suppose R = R max /R min is at most an absolute constant. For simplicity, we only consider the dependency on t when characterizing the convergence rates of the weight matrix related quantities and the training loss, omitting the dependency on other parameters such as m,n,σ 0 , R min , R max .

̸

Theorem 4.1 (Leaky ReLU Networks) . For two-layer neural network defined in (3.1) with leaky ReLU activation σ ( z ) = max { γz, z } , γ ∈ (0 , 1) . Assume the training data satisfy R 2 min ≥ CR 2 γ -4 np for some sufficiently large constant C . For any δ ∈ (0 , 1) , if the learning rate η ≤ ( CR 2 max /nm ) -1 and the initialization scale σ 0 ≤ γ ( CR max √ log( mn/δ ) ) -1 , then with probability at least 1 -δ over the random initialization of gradient descent, the trained network satisfies:

The ℓ 2 norm of each neuron increases to infinity at a logarithmic rate: ∥ w ( t ) j,r ∥ 2 = Θ(log( t )) for all j ∈ {± 1 } and r ∈ [ m ] .

Throughout the gradient descent trajectory, the stable rank of the weights W ( t ) j for all j ∈ {± 1 } satisfies

with a convergence rate of O (1 / log( t )) .

Gradient descent will find W ( t ) such that all the training data points possess the same normalized margin asymptotically:

If we assume that W ( t ) converges in direction, i.e., the limit of W ( t ) / ∥ W ( t ) ∥ F exists, denoted by ¯ W , then there exists a scaling factor α > 0 such that α ¯ W satisfies the Karush-Kuhn-Tucker (KKT) conditions for the following max-margin problem:

The empirical loss converges to zero at the following rate: L S ( W ( t ) ) = Θ( t -1 ) .

Remark 4.2. In Theorem 4.1, we show that when using the leaky ReLU activation function on nearly orthogonal training data, gradient descent asymptotically finds a network with a stable rank of W j equal to 1 . Additionally, we demonstrate that gradient descent will find a network by which all the training data points share the same normalized margin asymptotically. Moreover, if we assume the weight matrix converges in direction, then its limit will satisfy the KKT conditions of the max-margin problem (4.1). Furthermore, we analyze the rate of weight norm increase and the convergence rate of the stable rank for gradient descent, both of which exhibit a logarithmic dependency in t .

Theorem 4.3 (ReLU Networks) . For two-layer neural network defined in (3.1) with ReLU activation σ ( z ) = max { 0 , z } . Assume the training data satisfy R 2 min ≥ CR 2 np for some sufficiently large constant C . For any δ ∈ (0 , 1) , if the neural network width m ≥ C log( n/δ ) , learning rate η ≤ ( CR 2 max /nm ) -1 and initialization scale σ 0 ≤ ( CR max √ log( mn/δ ) ) -1 , then with probability at least 1 -δ over the random initialization of gradient descent, the trained network satisfies:

The Frobenious norm and the spectral norm of weight matrix increase to infinity at a logarithmic rate: ∥ W ( t ) j ∥ F = Θ(log( t )) and ∥ W ( t ) j ∥ 2 = Θ(log( t )) for all j ∈ {± 1 } .

Throughout the gradient descent trajectory, the stable rank of the weights W ( t ) j for all j ∈ {± 1 } satisfies,

where c is an absolute constant.

Gradient descent will find a W ( t ) such that all the training data points possess the same normalized margin asymptotically:

The empirical loss converges to zero at the following rate: L S ( W ( t ) ) = Θ( t -1 ) .

Remark 4.4. For ReLU networks, we provide an example in the appendix concerning fully orthogonal training data and prove that the activation pattern during training depends solely on the initial activation state. Specifically, when training a two-layer ReLU network with gradient descent using such data, the stable rank of the network's weight matrix W j converges to approximately 2. It is worth noting that this stable rank value is higher than the stable rank achieved by leaky ReLU networks, which is 1 .

Comparison with previous work. One notable related work is Lyu et al. (2021), which also investigates the implicit bias of two-layer leaky ReLU networks. The main distinction between our work and Lyu et al. (2021) is the optimization method employed. We utilize gradient descent, whereas they utilize gradient flow. Additionally, our assumption is that the training data is nearly-orthogonal,

while they assume the training data is symmetric. Our findings are more closely related to the work by Frei et al. (2022b), which investigates both gradient flow and gradient decent. In both our study and Frei et al. (2022b), we examine two-layer neural networks with leaky ReLU activations. However, they focus on networks trained via gradient flow, while we investigate networks trained using gradient descent. For the gradient descent approach, Frei et al. (2022b) provide a constant stable rank upper bound for smoothed leaky ReLU. In contrast, we prove that the stable rank of leaky ReLU networks converges to 1 , aligning with the implicit bias of gradient flow proved in Frei et al. (2022b). Furthermore, they presented an O ( t -1 / 2 ) convergence rate for the empirical loss, whereas our convergence rate is Θ( t -1 ) . Another related work is Lyu and Li (2019), which studied smooth homogeneous networks trained by gradient descent. Our results on the rate of weight norm increase and the convergence rate of training loss match those in Lyu and Li (2019), despite the fact that we study non-smooth homogeneous networks. It is worth noting that Lyu and Li (2019); Lyu et al. (2021); Frei et al. (2022b) demonstrated that neural networks trained by gradient flow converge to a Karush-Kuhn-Tucker (KKT) point of the max-margin problem. We do not have such a result unless we assume the directional convergence of the weight matrix.

5 Overview of Proof Techniques

In this section, we discuss the key techniques we invent in our proofs to analyze the implicit bias of ReLU and leaky ReLU networks.

5.1 Refined Analysis of Decomposition Coefficient

Signal-noise decomposition , a technique initially introduced by Cao et al. (2022), is used to analyze the learning dynamics of two-layer convolutional networks. This method decomposes the convolutional filters into a linear combination of initial filters, signal vectors, and noise vectors, converting the neural network learning into a dynamical system of coefficients derived from the decomposition. In this work, we extend the signal-noise decomposition to data-correlated decomposition to facilitate the analysis of the training dynamic for two-layer fully connected neural networks.

Definition 5.1 (Data-correlated Decomposition) . Let w ( t ) j,r , j ∈ {± 1 } , r ∈ [ m ] be the weights of first-layer neurons at the t -th iteration of gradient descent. There exist unique coefficients ρ ( t ) j,r,i such that

By defining ρ ( t ) j,r,i := ρ ( t ) j,r,i 1 ( ρ ( t ) j,r,i ≥ 0) , ρ ( t ) j,r,i := ρ ( t ) j,r,i 1 ( ρ ( t ) j,r,i ≤ 0) , (5.1) can be further written as

As an extension of the signal-noise decomposition first proposed in Cao et al. (2022) for analyzing two-layer convolutional networks, data-correlated decomposition defined in Definition 5.1 can be used to analyze two-layer fully-connected network, where the normalization factors ∥ x i ∥ -2 2 are introduced to ensure that ρ ( t ) j,r,i ≈ ⟨ w ( t ) j,r , x i ⟩ . This is also inspired by previous works by Lyu and Li (2019); Frei et al. (2022b), which demonstrate that W converges to a KKT point of the max-margin problem. This implies that w ( ∞ ) j,r / ∥ w ( ∞ ) j,r ∥ 2 can be expressed as a linear combination of the training data { x i } n i =1 , with the coefficient λ i corresponding to ρ ( t ) j,r,i in our analysis. This technique does not rely on the strictly increasing and smoothness properties of the activation function and will serve as the foundation for our analysis. Let us first investigate the update rule of the coefficient ρ ( t ) j,r,i , ρ ( t ) j,r,i .

Lemma 5.2. The coefficients ρ ( t ) j,r,i , ρ ( t ) j,r,i defined in Definition 5.1 satisfy the following iterative equations:

it holds that

We can establish the relationship between (5.4), (5.5) and inequality (5.6) if we are able to express | ℓ ′ ( t ) i | using coefficients ρ ( t ) j,r,i and | ρ ( t ) j,r,i | . To achieve this, we can first approximate ℓ ( t ) i using the margin y i f ( W ( t ) , x i ) and then approximate F j ( W ( t ) j , x i ) using the coefficients ρ ( t ) j,r,i . The approximation is given as follows:

̸

̸

From (5.9), one can see that we need to decouple ℓ ( t ) i from | ρ ( t ) j,r,i ′ | ( i ′ = i ) . In order to accomplish this, we also prove the following lemma, which demonstrates that the ratio between ∑ m r =1 | ρ ( t ) j,r,i | and ∑ m r =1 | ρ ( t ) j,r,i ′ | ( i ′ = i ) will maintain a constant order throughout the training process. Here, we present the lemma for leaky ReLU networks.

̸

Lemma 5.4 (leaky ReLU automatic balance) . For two-layer leaky ReLU network defined in (3.1), for any t ≥ 0 , we have ∑ m r =1 | ρ ( t ) j,r,i | ≥ cγ 2 ∑ m r =1 | ρ ( t ) j,r,i ′ | for any j ∈ {± 1 } and i, i ′ ∈ [ n ] , where c is a constant.

By Lemma 5.4, we can approximate the neural network output using (5.9). This approximation expresses the output F j ( W ( t ) j , x i ) as a sum of the coefficients ρ ( t ) j,r,i :

By combining (5.4), (5.5), (5.8), and (5.10), we obtain the following relationship:

This relationship aligns with the form of (5.6), if we set x t = 1 ± cγ 2 R -2 min pn m ∑ m r =1 | ρ ( t ) j,r,i | . Thus, we can directly apply Lemma 5.3 to gain insights into the logarithmic rate of increase for the average

for all r ∈ [ m ] , j ∈ {± 1 } and i ∈ [ n ] .

To study implicit bias, the first main challenge is to generalize the decomposition coefficient analysis to infinite time. The signal-noise decomposition used in Cao et al. (2022); Kou et al. (2023) requires early stopping with threshold T ∗ to facilitate their analysis. They only provided upper bounds of 4 log( T ∗ ) for ρ ( t ) j,r,i , | ρ ( t ) j,r,i | (See Proposition 5.3 in Cao et al. (2022), Proposition 5.2 in Kou et al.

(2023)), and then carried out a two-stage analysis. To obtain upper bounds for ρ ( t ) j,r,i , | ρ ( t ) j,r,i | , they used an upper bound for | ℓ ′ ( t ) i | and directly plugged it into (5.4) and (5.5) to demonstrate that ρ ( t ) j,r,i and | ρ ( t ) j,r,i | would not exceed 4 log( T ∗ ) , which is a fixed value related to the early stopping threshold. Therefore, dealing with infinite time requires new techniques. To overcome this difficulty, we propose a refined analysis of decomposition coefficients which generalizes Cao et al. (2022)'s technique. We first give the following key lemma.

Lemma 5.3. For non-negative real number sequence { x t } ∞ t =0 satisfying

magnitudes of the coefficients 1 m ∑ m r =1 | ρ ( t ) j,r,i | , which in turn implies that ∥ w ( t ) j,r ∥ 2 = Θ(log t ) and ∥ W ( t ) ∥ F = Θ(log t ) . In the case of ReLU networks, we have the following lemma that provides automatic balance:

Lemma 5.5 (ReLU automatic balance) . For two-layer ReLU network defined in (3.1), there exists a constant c such that for any t ≥ 0 , we have | ρ ( t ) y i ,r,i | ≥ c | ρ ( t ) j,r ′ ,i ′ | for any j ∈ {± 1 } , r ∈ S (0) i := { r ∈ [ m ] : ⟨ w (0) y i ,r , x i ⟩ ≥ 0 } , r ′ ∈ [ m ] and i, i ′ ∈ [ n ] .

The automatic balance lemma guarantees that the magnitudes of coefficients related to the neurons of class y i , which are activated by x i during initialization, dominate those of other classes. With the help of Lemma 5.5, we can get the following approximation for the margin y i f ( W ( t ) , x i ) :

By combining (5.4), (5.5), (5.8) and (5.11), we obtain the following relationship:

which precisely matches the form of (5.6) by setting x t = 1 ± cR -2 min pn m ∑ r ∈ S (0) i ρ ( t ) y i ,r,i . Therefore, we can directly apply Lemma 5.3 and obtain the logarithmic increasing rate of | ρ ( t ) y i ,r,i | for r ∈ S (0) i . Consequently, this implies that ∥ W ( t ) ∥ F = Θ(log t ) .

5.2 Analysis of Activation Pattern

One notable previous work (Frei et al., 2022b) provided a constant upper bound for the stable rank of two-layer smoothed leaky ReLU networks trained by gradient descent in their Theorem 4.2. To achieve a better stable rank bound, we characterize the activation pattern of leaky ReLU network neurons after a certain threshold time T in the following lemma.

Lemma 5.6 (leaky ReLU activation pattern) . Let T = Cη -1 nmR -2 max . For two-layer leaky ReLU network defined in (3.1), for any t ≥ T , it holds that sign( ⟨ w ( t ) j,r , x i ⟩ ) = jy i for any j ∈ {± 1 } and r ∈ [ m ] .

̸

Lemma 5.6 indicates that the activation pattern will not change after time T . Given Lemma 5.6, we can get σ ′ ( ⟨ w ( t ) j,r , x i ⟩ ) = γ for j = y i and σ ′ ( ⟨ w ( t ) j,r , x i ⟩ ) = 1 for j = y i . Plugging this into (5.4) and (5.5) can give the following useful lemma.

Lemma 5.7. Let T be defined in Lemma 5.6. For t ≥ T , it holds that

for any i ∈ [ n ] and r, r ′ ∈ [ m ] .

This lemma reveals that beyond a certain time threshold T , the increase in ρ ( t ) j,r,i is consistent across neurons within the same positive or negative class. However, for neurons belonging to the oppose class, this increment in ρ ( t ) j,r,i is scaled by a factor equivalent to the slope of the leaky ReLU function γ . From this and (5.1), we can demonstrate that ∥ w ( t ) j,r -w ( t ) j,r ′ ∥ 2 ( r = r ′ ) can be upper bounded by a constant, leading to the following inequalities:

̸

Considering that ∥ w ( t ) j,r ∥ 2 = Θ(log t ) , the stable rank of W ( t ) j naturally converges to a value of 1. For ReLU networks, we can partially characterize the activation pattern as illustrated in the following lemma.

Lemma5.8. (ReLU activation pattern) For two-layer ReLU networks defined in (3.1), for any i ∈ [ n ] , we have S ( t ) i ⊆ S ( t +1) i for any t ≥ 0 , where S ( t ) i := { r ∈ [ m ] : ⟨ w ( t ) y i ,r , x i ⟩ ≥ 0 } .

Lemma 5.8 suggests that once the neuron of class y i is activated by x i , it will remain activated throughout the training process. Leveraging such an activation pattern, we can establish a lower bound for ∥ W ( t ) j ∥ 2 as Ω(log( t )) . Together with the trivial upper bound for ∥ W ( t ) j ∥ F of order O (log( t )) , it provides a constant upper bound for the stable rank of ReLU network weight.

5.3 Analysis of Margin and Training Loss

Notably, Lyu and Li (2019) established in their Theorem 4.4 that any limit point of smooth homogeneous neural networks f ( W , x ) trained by gradient descent is along the direction of a KKT point for the max-margin problem (4.1). Additionally, Lyu and Li (2019) provided precise bounds on the training loss and weight norm for smooth homogeneous neural networks in their Theorem 4.3 as follows:

where L is the order of the homogeneous network satisfying the property f ( c W , x ) = c L f ( W , x ) for all c > 0 , W , and x . It is worth noting that the two-layer (leaky) ReLU neural network analyzed in this paper is 1-homogeneous but not smooth. In Section 5.1, we have already demonstrated that ∥ W ( t ) ∥ F = Θ(log t ) , and in this subsection, we will discuss the proof technique employed to show a convergence rate of Θ( t -1 ) for the loss and establish the same normalized margin for all the training data points asymptotically. These results align with those presented by Lyu and Li (2019) regarding smooth homogeneous networks.

By the nearly orthogonal property, we can bound the increment of margin as follows:

Given (5.8) and (5.12), we can apply Lemma 5.3 and obtain

where C 3 is a constant. Utilizing (5.13) and the inequality z -z 2 / 2 ≤ log(1 + z ) ≤ z for z ≥ 0 , we can derive:

To demonstrate that all the training data points attain the same normalized margin as t goes to infinity, we first observe that (5.12) provides the following bounds for the increment of margin difference:

Now, we consider two cases:

If the ratio | ℓ ′ ( t ) i | / | ℓ ′ ( t ) k | is relatively large, then y k f ( W ( t ) , x k ) -y i f ( W ( t ) , x i ) will not increase.

If the ratio | ℓ ′ ( t ) i | / | ℓ ′ ( t ) k | is relatively small, then y k f ( W ( t ) , x k ) -y i f ( W ( t ) , x i ) will also be relatively small. In fact, it can be bounded by a constant due to the fact that | ℓ ′ ( t ) i | / | ℓ ′ ( t ) k | can be approximated by exp( y k f ( W ( t ) , x k ) -y i f ( W ( t ) , x i )) . By (5.14), we can show that

0

10

20

30

40

50

0

10

20

30

40

50

Stable Rank

Stable Ranks for Different Leaky ReLU Slopes

Leaky ReLU Slope = 0

Leaky ReLU Slope = 0.1

Leaky ReLU Slope = 0.5

Leaky ReLU Slope = 1.0

0

10

20

30

40

50

0.000

0.005

0.010

0.015

0.020

0.025

0.030

0.035

Training Loss

Training Loss for Different Leaky ReLU Slopes

Leaky ReLU Slope = 0

Leaky ReLU Slope = 0.1

Leaky ReLU Slope = 0.5

Leaky ReLU Slope = 1.0

Epoch

Epoch

y k f ( W ( t +1) , x k ) -y i f ( W ( t +1) , x i ) can also be bounded by a constant, provided that the learning rate η is sufficiently small.

By combining both cases, we can conclude that both | ℓ ′ ( t ) i | / | ℓ ′ ( t ) k | and y k f ( W ( t ) , x k ) -y i f ( W ( t ) , x i ) can be bounded by constants. This result is formally stated in the following lemma.

Lemma 5.9. For two-layer neural networks defined in (3.1) with (leaky) ReLU activation, the following bounds hold for any t ≥ 0 :

for any i, k ∈ [ n ] , where C 1 , C 2 are positive constants.

By Lemma 5.9, which shows that the difference between the margins of any two data points can be bounded by a constant, and taking into account that ∥ W ( t ) ∥ F = Θ(log t ) , we can deduce the following result:

This demonstrates that gradient descent will asymptotically find a neural network in which all the training data points achieve the same normalized margin.

6 Experiments

In this section, we present simulations of both synthetic and real data to back up our theoretical analysis in the previous section.

Synthetic-data experiments. Here we generate a synthetic mixture of Gaussian data as follows: Let µ ∈ R d be a fixed vector representing the signal contained in each data point. Each data point ( x , y ) with predictor x ∈ R d and label y ∈ {-1 , 1 } is generated from a distribution D , which we specify as follows:

The label y is generated as a Rademacher random variable, i.e. P [ y = 1] = P [ y = -1] = 1 / 2 .

A noise vector ξ is generated from the Gaussian distribution N ( 0 , σ 2 p I d ) . And x is assigned as y · µ + ξ where µ is a fixed feature vector.

Specifically, we set training data size n = 10 , d = 784 and train the NN with gradient descent using learning rate 0 . 1 for 50 epochs. We set µ to be a feature randomly drawn from N (0 , 10 -4 I d ) . We then generate the noise vector ξ from the Gaussian distribution N ( 0 , σ 2 p I ) with fixed standard deviation σ p = 1 . We train the FNN model defined in Section 3 with ReLU (or leaky-RelU) activation function and width m = 100 . As we can infer from Figure 1, the stable rank will decrease faster for larger leaky ReLU slopes and have a smaller value when epoch t →∞ .

Real-data experiments on MNIST dataset. Here we train a two-layer feed-forward neural network defined in Section 3 with ReLU (or leaky-ReLU) functions. The number of widths is

0

2

4

6

8

10

0

50

100

150

200

Stable Rank

Stable Ranks for Different Weight Variances

Weight Variance = 1e-05

Weight Variance = 5e-05

Weight Variance = 0.0001

Weight Variance = 0.0005

Weight Variance = 0.001

0

2

4

6

8

10

20

40

60

80

100

Test Accuracy (%)

Test Accuracy for Different Weight Variances

Weight Variance = 1e-05

Weight Variance = 5e-05

Weight Variance = 0.0001

Weight Variance = 0.0005

Weight Variance = 0.001

Epoch

Epoch

0

2

4

6

8

10

0

50

100

150

200

Stable Rank

Stable Ranks for Different Weight Variances

Weight Variance = 1e-05

Weight Variance = 5e-05

Weight Variance = 0.0001

Weight Variance = 0.0005

Weight Variance = 0.001

0

2

4

6

8

10

20

40

60

80

100

Test Accuracy (%)

Test Accuracy for Different Weight Variances

Weight Variance = 1e-05

Weight Variance = 5e-05

Weight Variance = 0.0001

Weight Variance = 0.0005

Weight Variance = 0.001

Epoch

Epoch

set as m = 1000 . We use the Gaussian initialization and consider different weight variance σ 0 ∈ { 0 . 00001 , 0 . 00005 , 0 . 0001 , 0 . 0005 , 0 . 001 } . We train the NN with stochastic gradient descent with batch size 64 and learning rate 0 . 1 for 10 epochs. As we can infer from Figures 2 and 3, the stable rank of ReLU or leaky ReLU networks will largely depend on the initialization and the training time. When initialization is sufficiently small, the stable rank will quickly decrease to a small value compared to its initialization values.

7 Conclusion and Future Work

This paper employs a data-correlated decomposition technique to examine the implicit bias of twolayer ReLU and Leaky ReLU networks trained using gradient descent. By analyzing the training dynamics, we provide precise characterizations of the weight matrix stable rank limits for both ReLU and Leaky ReLU cases, demonstrating that both scenarios will yield a network with a low stable rank. Additionally, we present an analysis for the convergence rate of the loss function. An important future work is to investigate the directional convergence of the weight matrix in neural networks trained via gradient descent, which is essential to prove the convergence to a KKT point of the max-margin problem. Furthermore, it is important to extend our analysis to fully understand the neuron activation patterns in ReLU networks. Specifically, we will explore whether certain neurons will switch their activation patterns by an infinite number of times throughout the training or if the activation patterns stabilize after a certain number of gradient descent iterations.

References

ALLEN-ZHU, Z. and LI, Y. (2020). Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. arXiv preprint arXiv:2012.09816 .

ARORA, S., COHEN, N., HU, W. and LUO, Y. (2019). Implicit regularization in deep matrix factorization. Advances in Neural Information Processing Systems 32 .

BARTLETT, P. L., LONG, P. M., LUGOSI, G. and TSIGLER, A. (2020). Benign overfitting in linear regression. Proceedings of the National Academy of Sciences .

BELKIN, M., HSU, D., MA, S. and MANDAL, S. (2019). Reconciling modern machine-learning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences 116 15849-15854.

BELKIN, M., HSU, D. and XU, J. (2020). Two models of double descent for weak features. SIAM Journal on Mathematics of Data Science 2 1167-1180.

CAO, Y., CHEN, Z., BELKIN, M. and GU, Q. (2022). Benign overfitting in two-layer convolutional neural networks. arXiv preprint arXiv:2202.06526 .

CAO, Y., GU, Q. and BELKIN, M. (2021). Risk bounds for over-parameterized maximum margin classification on sub-gaussian mixtures. Advances in Neural Information Processing Systems 34 .

CHATTERJI, N. S. and LONG, P. M. (2021). Finite-sample analysis of interpolating linear classifiers in the overparameterized regime. Journal of Machine Learning Research 22 129-1.

CHIZAT, L. and BACH, F. (2020). Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In Conference on Learning Theory . PMLR.

FREI, S., CHATTERJI, N. S. and BARTLETT, P. (2022a). Benign overfitting without linearity: Neural network classifiers trained by gradient descent for noisy linear data. In Conference on Learning Theory . PMLR.

FREI, S., VARDI, G., BARTLETT, P. L., SREBRO, N. and HU, W. (2022b). Implicit bias in leaky relu networks trained on high-dimensional data. arXiv preprint arXiv:2210.07082 .

GUNASEKAR, S., LEE, J. D., SOUDRY, D. and SREBRO, N. (2018). Implicit bias of gradient descent on linear convolutional networks. In Advances in Neural Information Processing Systems .

HASTIE, T., MONTANARI, A., ROSSET, S. and TIBSHIRANI, R. J. (2022). Surprises in highdimensional ridgeless least squares interpolation. The Annals of Statistics 50 949-986.

JELASSI, S. and LI, Y. (2022). Towards understanding how momentum improves generalization in deep learning. In International Conference on Machine Learning . PMLR.

JI, Z. and TELGARSKY, M. (2018). Gradient descent aligns the layers of deep linear networks. arXiv preprint arXiv:1810.02032 .

JI, Z. and TELGARSKY, M. (2020). Directional convergence and alignment in deep learning. Advances in Neural Information Processing Systems 33 17176-17186.

KOU, Y., CHEN, Z., CHEN, Y. and GU, Q. (2023). Benign overfitting for two-layer relu networks. arXiv preprint arXiv:2303.04145 .

LI, Z., ZHOU, Z.-H. and GRETTON, A. (2021). Towards an understanding of benign overfitting in neural networks. arXiv preprint arXiv:2106.03212 .

LIAO, Z., COUILLET, R. and MAHONEY, M. (2020). A random matrix analysis of random fourier features: beyond the gaussian kernel, a precise phase transition, and the corresponding double descent. In 34th Conference on Neural Information Processing Systems (NeurIPS 2020) .

LYU, K. and LI, J. (2019). Gradient descent maximizes the margin of homogeneous neural networks. arXiv preprint arXiv:1906.05890 .

LYU, K., LI, Z., WANG, R. and ARORA, S. (2021). Gradient descent on two-layer nets: Margin maximization and simplicity bias. Advances in Neural Information Processing Systems 34 1297812991.

MEI, S. and MONTANARI, A. (2019). The generalization error of random features regression: Precise asymptotics and double descent curve. arXiv preprint arXiv:1908.05355 .

PHUONG, M. and LAMPERT, C. H. (2021). The inductive bias of relu networks on orthogonally separable data. In International Conference on Learning Representations .

RAZIN, N. and COHEN, N. (2020). Implicit regularization in deep learning may not be explainable by norms. Advances in neural information processing systems 33 21174-21187.

SAFRAN, I., VARDI, G. and LEE, J. D. (2022). On the effective number of linear regions in shallow univariate relu networks: Convergence guarantees and implicit bias. arXiv preprint arXiv:2205.09072 .

SARUSSI, R., BRUTZKUS, A. and GLOBERSON, A. (2021). Towards understanding learning in neural networks with linear teachers. In International Conference on Machine Learning . PMLR.

SHEN, R., BUBECK, S. and GUNASEKAR, S. (2022). Data augmentation as feature manipulation: a story of desert cows and grass cows. arXiv preprint arXiv:2203.01572 .

SOUDRY, D., HOFFER, E., NACSON, M. S., GUNASEKAR, S. and SREBRO, N. (2018). The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research 19 2822-2878.

TIMOR, N., VARDI, G. and SHAMIR, O. (2023). Implicit regularization towards rank minimization in relu networks. In International Conference on Algorithmic Learning Theory . PMLR.

VARDI, G. (2022). On the implicit bias in deep-learning algorithms. arXiv preprint arXiv:2208.12591 .

VARDI, G., SHAMIR, O. and SREBRO, N. (2022a). On margin maximization in linear and relu networks. Advances in Neural Information Processing Systems 35 37024-37036.

VARDI, G., YEHUDAI, G. and SHAMIR, O. (2022b). Gradient methods provably converge to non-robust networks. arXiv preprint arXiv:2202.04347 .

VERSHYNIN, R. (2018). High-Dimensional Probability: An Introduction with Applications in Data Science . Cambridge Series in Statistical and Probabilistic Mathematics, Cambridge University Press.

WANG, K. and THRAMPOULIDIS, C. (2022). Binary classification of gaussian mixtures: Abundance of support vectors, benign overfitting, and regularization. SIAM Journal on Mathematics of Data Science 4 260-284.

WU, D. and XU, J. (2020). On the optimal weighted ℓ 2 regularization in overparameterized linear regression. Advances in Neural Information Processing Systems 33 .

A Additional Experiments

In this section, we conduct additional experiments on the nearly orthogonal and MINIST datasets.

A.1 Additional Experiment on Nearly Orthogonal Dataset

In this subsection, we conduct additional experiments on a nearly orthogonal dataset for long epochs to support our main Theorems 4.1 and 4.3.

Stable Ranks for Different Leaky ReLU Slopes

Epoch

0

200

400

600

800

1000

0

2

4

6

8

10

Stable Rank

Leaky ReLU Slope = 0

Leaky ReLU Slope = 0.5

Leaky ReLU Slope = 1.0

0

200

400

600

800

1000

10

2

10

1

10 0

10 1

Stable Rank - 1

Stable Ranks - 1 for Different Leaky ReLU Slopes (Semilogy)

Leaky ReLU Slope = 0

Leaky ReLU Slope = 0.5

Leaky ReLU Slope = 1.0

Epoch

0

200

400

600

800

1000

0

2

4

6

8

10

Stable Rank

Stable Ranks for Different Leaky ReLU Slopes

Leaky ReLU Slope = 0

Leaky ReLU Slope = 0.5

Leaky ReLU Slope = 1.0

Epoch

0

200

400

600

800

1000

10

1

10 0

10 1

Stable Rank - 1

Stable Ranks - 1 for Different Leaky ReLU Slopes (Semilogy)

Leaky ReLU Slope = 0

Leaky ReLU Slope = 0.5

Leaky ReLU Slope = 1.0

Epoch

Figure 4: Under the same setting of the synthetic data introduced in Section 6 , we train the NN with full batch gradient descent with a learning rate 0 . 1 for 1000 epochs. We set µ to be a feature randomly drawn from N (0 , 10 -4 I d ) . We then generate the noise vector ξ from the Gaussian distribution N ( 0 , σ 2 p I d ) with fixed standard deviation σ p = 1 . We train the FNN model defined in Section 3 with ReLU (or leaky-ReLU) activation function and width m = 100 . As we can see from Figure 4, the stable rank for the leaky ReLU network with large slopes γ will converge to 1 when epoch t →∞ . In comparison, the stable rank for the ReLU network will not converge to 1 .

Figure 5: To further illustrate the behavior of the ReLU network, we generate the synthetic training data with fully orthogonal input. Each data point ( x , y ) with input x ∈ R d and label y ∈ {-1 , 1 } is generated from a distribution D , which we specify as follows:

The label y is generated as a Rademacher random variable, i.e., P [ y = 1] = P [ y = -1] = 1 / 2 .

Input x is randomly generated from the basis { e 1 , e 2 , . . . , e d } .

Specifically, we set training data size n = 20 , d = 40 and train the NN with full batch gradient descent with a learning rate 0 . 1 for 1000 epochs. We train the FNN model defined in Section 3 with ReLU (or leaky-ReLU) activation function and width m = 10000 . As we can observe from Figure 5,

the stable rank for the leaky ReLU network with large slopes γ will converge to 1 when epoch t →∞ . In comparison, the stable rank for the large-width ReLU network will not converge to 1 but to 2 .

A.2 Additional Experiment on MINIST

Our focus is the training of a two-layer feed-forward neural network, as discussed in Section 3, utilizing either ReLU or leaky-ReLU activation functions. We examine different widths, specifically choosing from { 10 , 50 , 100 , 500 , 1000 } .

The network initialization process follows a Gaussian distribution, with a variance of σ 0 = 0 . 00001 . Training is executed using stochastic gradient descent, a batch size of 64 , and a learning rate of 0 . 1 , for a total of 10 epochs. As discerned from Figures 6 and 7, the stable rank of networks utilizing either ReLU or leaky ReLU is weakly influenced by the width. For an exceedingly small width such as 10, the weight matrix is low rank with a correspondingly small stable rank. However, this also results in low test accuracy as the network cannot effectively learn all necessary features. As the width increases, the test accuracy and final stable rank will increase. However, for sufficiently large widths, an increase in width no longer corresponds to stable rank or test accuracy increases.

Stable Ranks for Different Weight Variances

Epoch

0

2

4

6

8

10

0

50

100

150

200

Stable Rank

hidden size = 10

hidden size = 50

hidden size = 100

hidden size = 500

hidden size = 1000

0

2

4

6

8

10

20

40

60

80

100

Test Accuracy (%)

Test Accuracy for Different Weight Variances

Epoch

0

2

4

6

8

10

0

50

100

150

200

Stable Rank

Stable Ranks for Different Weight Variances

hidden size = 10

hidden size = 50

hidden size = 100

hidden size = 500

hidden size = 1000

0

2

4

6

8

10

20

40

60

80

100

Test Accuracy (%)

Test Accuracy for Different Weight Variances

Epoch

Epoch

B Preliminary Lemmas

In this section, we present some pivotal lemmas that illustrate some important properties of the data and neural network parameters at their random initialization and provide the update rule of coefficients from data-correlated decomposition.

Now turning to network initialization, the following lemma studies the inner product between a randomly initialized neural network neuron w (0) j,r ( j ∈ {± 1 } and r ∈ [ m ] ) and the training data. The calculations characterize how the neural network at initialization randomly captures the information in training data.

Lemma B.1. Suppose that d = Ω(log( mn/δ )) , m = Ω(log(1 /δ )) . Then with probability at least 1 -δ ,

for all r ∈ [ m ] , j ∈ {± 1 } and i ∈ [ n ] .

Proof of Lemma B.1. First of all, the initial weights w (0) j,r ∼ N ( 0 , σ 0 I ) . By Bernstein's inequality, with probability at least 1 -δ/ (4 m ) we have

Therefore, if we set appropriately d = Ω(log( m/δ )) , we have with probability at least 1 -δ/ 2 , for all j ∈ {± 1 } and r ∈ [ m ] ,

Under definition, we have ∥ x i ∥ 2 ≤ R max for all i ∈ [ n ] . It is clear that for each j, r , ⟨ w (0) j,r , µ ⟩ is a Gaussian random variable with mean zero and variance σ 2 0 ∥ x i ∥ 2 2 . Therefore, by Gaussian tail bound and union bound, with probability at least 1 -δ/ 2 ,

Next, we denote S (0) i as { r ∈ [ m ] : ⟨ w (0) y i ,r , x i ⟩ > 0 } . We give a lower bound of | S (0) i | in the following two lemmas.

Lemma B.2. Suppose that δ > 0 and m ≥ 50 log(2 n/δ ) . Then with probability at least 1 -δ ,

Proof of Lemma B.2. Note that | S (0) i | = ∑ m r =1 1 [ ⟨ w (0) y i ,r , x i ⟩ > 0] and P ( ⟨ w (0) y i ,r , x i ⟩ > 0) = 1 / 2 , then by Hoeffding's inequality, with probability at least 1 -δ/n , we have

Therefore, as long as m ≥ 50 log(2 n/δ ) , by applying union bound, with probability at least 1 -δ , we have

Now we give the update rule of coefficients from data-correlated decomposition. We will begin by analyzing the coefficients in the data-correlated decomposition in Definition 5.1. The following lemma presents an iterative expression for the coefficients.

Lemma B.3. (Restatement of Lemma 5.2) The coefficients ρ ( t ) j,r,i , ρ ( t ) j,r,i defined in Definition 5.1 satisfy the following iterative equations:

for all r ∈ [ m ] , j ∈ {± 1 } and i ∈ [ n ] .

Proof of Lemma B.3. First, we iterate the gradient descent update rule (3.2) t times and get

According to the definition of ρ ( t ) j,r,i , we have

Therefore, we have the unique representation

Now with the notation ρ ( t ) j,r,i := ρ ( t ) j,r,i 1 ( ρ ( t ) j,r,i ≥ 0) , ρ ( t ) j,r,i := ρ ( t ) j,r,i 1 ( ρ ( t ) j,r,i ≤ 0) and the fact ℓ ′ ( s ) i < 0 , we get

Writing out the iterative versions of (B.1) and (B.2) completes the proof.

C Coefficient Analysis of Leaky ReLU

In this section, we establish a series of results on the data-correlated decomposition for two-layer leaky ReLU network defined as

The results in Section C, D and G are based on Lemma B.1, which hold with high probability. Denote by E prelim the event that Lemma B.1 in Section B holds (for a given δ , we see P ( E prelim ) ≥ 1 -δ ). For simplicity and clarity, we state all the results in Section C, D and G conditional on E prelim .

̸

Denote β = max i,j,r {|⟨ w (0) j,r , x i ⟩|} , R max = max i ∈ [ n ] ∥ x i ∥ 2 , R min = min i ∈ [ n ] ∥ x i ∥ 2 , p = max i = k |⟨ x i , x k ⟩| and suppose R = R max /R min is at most an absolute constant. Here we list the exact conditions for η, σ 0 , R min , R max , p required by the proofs in this section.

where C is a large enough constant. By Lemma B.1, we can upper bound β by 2 √ log(12 mn/δ ) · σ 0 R max . Then, by (C.2) and (C.4), it is straightforward to verify the following inequality:

where c is a sufficiently small constant.

Suppose the conditions listed in (C.2) and (C.4) hold, we claim that for any t ≥ 0 the following property holds.

Lemma C.1. Under the same conditions as Theorem 4.1, for any t ≥ 0 , we have that

where c 1 is a constant.

To prove Lemma C.1, we divide it into two lemmas, each addressing a specific case: 0 ≤ t ≤ T 1 (Lemma C.2) when the logit | ℓ ( t ) i | = Θ(1) , and t ≥ T 1 (Lemma C.3) when the logit | ℓ ( t ) i | is smaller than constant order. Here, T 1 = C ′ η -1 nmR -2 max , and C ′ is a constant. For each case, we apply different techniques to establish the proof.

Lemma C.2 ( 0 ≤ t ≤ T 1 ) . Under the same conditions as Theorem 4.1, for any 0 ≤ t ≤ T 1 = C ′ η -1 nmR -2 max , where C ′ is a constant, we have that

where c 2 is a constant.

Proof of Lemma C.2. In this lemma, we first show that (C.8) hold for t ≤ T 1 = C ′ η -1 nmR -2 max where C ′ = Θ(1) is a constant. Recall from Lemma B.3 that

we can get and

Therefore, we have max j,r,i { ρ ( t ) j,r,i , | ρ ( t ) j,r,i |} = O (1) for any t ≤ T 1 and hence max i { F +1 ( W ( t ) +1 , x i ) , F -1 ( W ( t ) -1 , x i ) } = O (1) for any t ≤ T 1 . Thus there exists a positive constant ˜ c such that | ℓ ′ ( t ) i | ≥ ˜ c for any t ≤ T 1 . And it follows for any j ∈ {± 1 } , r ∈ [ m ] , i ∈ [ n ] that

On the other hand, by (C.10), (C.11) and (C.12), we have for any j ′ ∈ {± 1 } , r ′ ∈ [ m ] , i ′ ∈ [ n ] that

Dividing (C.14) by (C.13), we can get for any j, j ′ ∈ {± 1 } , r, r ′ ∈ [ m ] , i, i ′ ∈ [ n ] that

which indicates that the first bullet holds for time t ≤ T 1 as long as c 2 ≤ ˜ cR 2 min R -2 max .

Lemma C.3 ( t ≥ T 1 ) . Let T 1 be defined in Lemma C.2. Under the same conditions as Theorem 4.1, for any t ≥ T 1 , we have that

where c 3 = Θ(1) is a constant. Moreover, we also have the following increasing rate estimation of | ρ ( t ) y i ,r,i | , | ρ ( t ) -y i ,r,i | :

where c 4 , c 5 , c 6 are constants.

Proof of Lemma C.3. We prove this lemma by induction. By Lemma C.2, we know that (C.15) holds for time t = T 1 as long as c 3 ≤ c 2 . Suppose that there exists ˜ t > T 1 such that (C.15) holds for all time 0 ≤ t ≤ ˜ t -1 . We aim to prove that they also hold for t = ˜ t . For any 0 ≤ t ≤ ˜ t -1 , we have

̸

̸

where the first inequality is by σ ( z ) ≥ z ; the second equality is by (5.1); the third inequality is by triangle inequality and the definition of β, p, R min ; the fourth inequality is by the induction hypothesis (C.15). Besides, for any 0 ≤ t ≤ ˜ t -1 , we also have the following upper bound of

F y i ( W ( t ) y i , x i ) :

̸

where the first inequality is by triangle inequality and the definition of β, p, R min ; the second inequality is by the induction hypothesis (C.15). On the other hand, for any 0 ≤ t ≤ ˜ t , we can give following upper and lower bounds for F -y i ( W ( t ) -y i , x i ) by applying similar arguments like (C.16) and (C.17):

̸

and

̸

where the second inequality is by a property of leaky ReLU function that σ ( a + b ) ≤ σ ( a ) + σ ( b ) , ∀ a, b ∈ R .

̸

̸

̸

̸

Next, we can bound | ℓ ′ ( t ) i | for 0 ≤ t ≤ ˜ t -1 :

where the second inequality is by (C.16) and (C.19). And

where the first inequality is by (C.17) and (C.18); the last inequality is by 1 / (1 + exp( z )) ≥ exp( -z ) / 2 if z ≥ 0 . By (C.20), we can get for 0 ≤ t ≤ ˜ t -1 that

By (C.21) and γρ ( t ) y i ,r,i ≤ | ρ ( t ) -y i ,r,i | ≤ γ -4 ρ ( t ) y i ,r,i , we can get for 0 ≤ t ≤ ˜ t -1 that

By (5.4), (5.5) and σ ′ ∈ [ γ, 1] , we have for 0 ≤ t ≤ ˜ t -1 that

By plugging (C.22), (C.24), (C.23) and (C.25) into (C.26), we have for 0 ≤ t ≤ ˜ t -1 that

By applying Lemma H.1 to (C.27) and taking

we can get for 0 ≤ t ≤ ˜ t that

where c 4 := 1 -γ -2 c -1 3 R -2 min pn and the last inequality is by η ≤ ( CR 2 max /nm ) -1 and C is a sufficiently large constant.

By applying Lemma H.1 to (C.28) and taking

we can get for 0 ≤ t ≤ ˜ t that

where c 5 := 1 -γ -3 c -1 3 R -2 min pn and the last inequality is by η ≤ ( CR 2 max /nm ) -1 and C is a sufficiently large constant.

By applying Lemma H.2 to (C.29) and taking

we can get

where c 6 := 1 + γ -2 c -1 3 R -2 min pn .

By applying Lemma H.2 to (C.30) and taking

we can get

where c 6 := 1 + γ -2 c -1 3 R -2 min pn .

In order to apply Lemma H.4 (requiring b > a ), we loosen the bounds in (C.31), (C.32), (C.33) and (C.34) as follows:

where (C.35) is by Bernoulli's inequality that 1+ γ -1 x ≤ (1+ x ) γ -1 for every real number 0 ≤ r ≤ 1 and x ≥ -1 ; (C.37) is by Bernoulli's inequality that 1 + γx ≥ (1 + x ) γ for every real number 0 ≤ r ≤ 1 and x ≥ -1 . If R 2 min c 6 e -( γ +1) β ≥ γR 2 max c 5 e 2 β , we have

If R 2 min c 6 e -( γ +1) β < γR 2 max c 5 e 2 β , by Lemma H.4, we have

Therefore, we can get for 0 ≤ t ≤ ˜ t that

as long as

This condition holds under the following conditions:

This implies that induction hypothesis (C.15) holds for t = ˜ t .

Lemma C.4 (Implication of Lemma C.1) . Under the same condition as Theorem 4.1, if (C.8) hold for time t , then we have that

where c 1 is the same constant as defined in Lemma C.1.

Proof of Lemma C.4. By σ ′ ∈ [ γ, 1] , (5.4) and (5.5), we have

Thus, we have

Therefore, | ρ ( t ) j,r,i | ≥ γ | ρ ( t ) j ′ ,r ′ ,i | for any j, j ′ ∈ {± 1 } , r ′ , r ∈ [ m ] and i ∈ [ n ] , and hence

Plugging (C.41) back into (C.8) completes the proof.

Lemma C.5. Let T 1 be defined in Lemma C.2. Every neuron will never change its activation pattern after time T 1 , i.e.,

for any t ≥ T 1 , j ∈ {± 1 } and r ∈ [ m ] . Moreover, it holds that

for any t ≥ T 1 , j ∈ {± 1 } and r ∈ [ m ] .

Proof of Lemma C.5. For j = y i and t ≥ 0 , we have ρ ( t ) j,r,i = 0 , and so

̸

̸

where

where the first inequality is by triangle inequality; the second inequality is by | ρ ( t ) j,r,i ′ | ≤ γ -4 c -1 1 ρ ( t ) y i ,r,i from Lemma C.1 and Lemma C.4. By (C.13), we have for t ≥ T 1 that

Therefore, by (C.5), (C.6) and (C.43), we know that

and thus sign( ⟨ w ( t ) j,r , x i ⟩ ) = 1 for any r ∈ [ m ] , i ∈ [ n ] , j = y i .

̸

For j = y i and any t ≥ 0 , we have ρ ( t ) j,r,i = 0 , and so

̸

̸

where the first inequality is by triangle inequality; the second inequality is by | ρ ( t ) j,r,i ′ | ≤ γ -4 c -1 1 | ρ ( t ) -y i ,r,i | from Lemma C.1 and Lemma C.4. By (C.13), we have

Therefore, by (C.5), (C.6) and (C.44), we know that

̸

and thus sign( ⟨ w ( t ) j,r , x i ⟩ ) = -1 for j = y i , which completes the proof.

D Stable Rank of Leaky ReLU Network

In this section, we consider the properties of stable rank of the weight matrix W ( t ) found by gradient descent at time t , defined as ∥ W ( t ) ∥ 2 F / ∥ W ( t ) ∥ 2 2 , the square of the ratio of the Frobenius norm to the spectral norm of W ( t ) . Given Lemma C.5, we have following coefficient update rule for t ≥ T 1 where T 1 is defined in Lemma C.2:

Based on (D.1) and (D.2), we first introduce the following helpful lemmas.

Lemma D.1. Let T 1 be defined in Lemma C.3. For any r, r ′ ∈ [ m ] , i ∈ [ n ] and t ≤ T 1 ,

Proof of Lemma D.1. By (C.14), we can get

which completes the proof.

Lemma D.2. Let T 1 be defined in Lemma C.3. For any r, r ′ ∈ [ m ] , i ∈ [ n ] and t ≥ T 1 ,

Proof of Lemma D.2. By (D.1) and (D.2), we can get for any r ∈ [ m ] , i ∈ [ n ] and t ≥ T 1 that

Since ρ ( t ) y i ,r,i , ρ ( t ) y i ,r ′ ,i possess the same increment and ρ ( t ) -y i ,r,i , ρ ( t ) -y i ,r ′ ,i possess the same increment, we have

for t ≤ T 1 . Notice that

Notice that max i,r,r ′ {| ρ ( T 1 ) y i ,r,i -ρ ( T 1 ) y i ,r ′ ,i | , | ρ ( T 1 ) -y i ,r,i -ρ ( T 1 ) -y i ,r ′ ,i |} ≤ max i,r,r ′ {| ρ ( T 1 ) y i ,r,i | , | ρ ( T 1 ) -y i ,r,i |} ≤ C ′ , which completes the proof.

Lemma D.3. Let T 1 be defined in Lemma C.3. For t ≥ T 1 , it holds that

for any i ∈ [ n ] and r, r ′ ∈ [ m ] .

Proof of Lemma D.3. By Lemma C.3 about the activation pattern after time T 1 , we can get

for t ≥ T 1 . Recursively using (D.4) and (D.5) t -T 1 times, we can get

This indicates that for different r, r ′ ∈ [ m ] , ρ ( t ) y i ,r,i -ρ ( T 1 ) y i ,r,i and ρ ( t ) y i ,r ′ ,i -ρ ( T 1 ) y i ,r ′ ,i are the same, whereas γ ( | ρ ( t ) -y i ,r,i | - | ρ ( T 1 ) -y i ,r,i ) | and ρ ( t ) y i ,r ′ ,i -ρ ( T 1 ) y i ,r ′ ,i are the same, which completes the proof.

Now we are ready to prove the second bullet of Theorem 4.1.

Lemma D.4. Throughout the gradient descent trajectory, the stable rank of the weights W j satisfies,

with a decreasing rate of O ( 1 / log( t ) ) .

Proof of Lemma D.4. By Definition 5.1, we have

We first show that ∥ v ( t ) j,r ∥ 2 = Θ(log t ) .

̸

where the second last inequality is by triangle inequality; the last inequality is by | ρ ( t ) j,r,i | ≤ γ -4 c -1 1 | ρ ( t ) j,r,i ′ | from Lemma C.1 and Lemma C.4.

By the definition of v ( t ) j,r , we have

̸

where the first inequality is by Lemma D.1 and Lemma D.2.

̸

Now, we are ready to estimate the stable rank of W ( t ) . On the one hand, for ∥ W ( t ) j ∥ 2 F , we have

where the first inequality is by triangle inequality and Cauchy inequality; the last inequality is by Lemma D.1, Lemma D.2 and taking

On the other hand, for ∥ W ( t ) j ∥ 2 2 , we have

where the first inequality is by taking x = v ( t ) j, 1 / ∥ v ( t ) j, 1 ∥ 2 ; the second inequality is by Cauchy inequality; the third inequality by breaking v ( t ) j,r down into v ( t ) j, 1 + v ( t ) j,r -v ( t ) j, 1 and then expanding the

first term as well as applying triangle inequality to the last term; the fourth inequality is by Cauchy inequality; the last inequality is by Lemma D.1, Lemma D.2 and taking

By leverage the upper bound of ∥ W ( t ) j ∥ 2 F as well as the lower bound of ∥ W ( t ) j ∥ 2 2 , we can get

Since ∥ W ( t ) j ∥ 2 F / ∥ W ( t ) j ∥ 2 2 ≥ 1 , ∥ v ( t ) j, 1 ∥ 2 = Θ(log t ) and C 1 , C 2 , C 3 , C 4 are constants, it follow that

and

which completes the proof.

E Coefficient Analysis of ReLU

In this section, we discuss the stable rank of two-layer ReLU neural network, which is defined as

where σ ( z ) = max { 0 , z } is ReLU activation function.

These results are based on the conclusions in Section B, which hold with high probability. Denote by E ′ prelim the event that all the results in Section B hold (for a given δ , we see P ( E ′ prelim ) ≥ 1 -2 δ by a union bound). For simplicity and clarity, we state all the results in this and the following sections conditional on E ′ prelim .

̸

Denote β = max i,j,r {|⟨ w (0) j,r , x i ⟩|} , R max = max i ∈ [ n ] ∥ x i ∥ 2 , R min = min i ∈ [ n ] ∥ x i ∥ 2 , p = max i = k |⟨ x i , x k ⟩| and suppose R = R max /R min is at most an absolute constant. Here we list the exact conditions for η, σ 0 , R min , R max , p required by the proofs in this section:

where C is a large enough constant. By Lemma B.1, we can upper bound β by 2 √ log(12 mn/δ ) · σ 0 R max . Then, by (C.2) and (C.4), it is straightforward to verify the following inequality:

where c is a sufficiently small constant.

We first introduce the following lemma which characterizes the increasing rate of coefficients ρ ( t ) j,r,i .

Lemma E.1. For two-layer ReLU neural network defined in (E.1), under the same condition as Theorem 4.3, the decomposition coefficients ρ ( t ) j,r,i satisfy following properties:

where c 1 , c 2 , c 3 are constants. And the following activation pattern is also observed: S (0) i ⊆ S ( t ) i where S ( t ) i := { r ∈ [ m ] : ⟨ w ( t ) y i ,r , x i ⟩ ≥ 0 } , i.e., the on-diagonal neuron activated at initialization will remain activated throughout the training.

Proof of Lemma E.1. We first show that the first bullet and S (0) i ⊆ S ( t ) i hold for t ≤ T 1 = Cη -1 nmR -2 max where C = Θ(1) is a constant. Now we prove this by induction. When t = 0 , the two hypotheses hold naturally. Suppose that there exists time ˜ t ≤ T 1 such that the two hypotheses hold for all time t ≤ ˜ t -1 . We aim to prove they also hold for t = ˜ t . Recall from Lemma B.3 that

Therefore, max j,r,i { ρ ( t ) j,r,i , | ρ ( t ) j,r,i |} = O (1) for any t ≤ T 1 and hence max i { F +1 ( W ( t ) +1 , x i ) , F -1 ( W ( t ) -1 , x i ) } = O (1) for any t ≤ T 1 . Thus there exists a positive constant c such that | ℓ ′ ( t ) i | ≥ c for any t ≤ T 1 . By induction hypothesis, we have S (0) i ⊆ S ( t ) i for all 0 ≤ t ≤ ˜ t -1 and hence σ ′ ( ⟨ w ( t ) y i ,r , x i ⟩ ) = 1 for all 0 ≤ t ≤ ˜ t -1 . And it follows that for r ∈ S (0) i

On the other hand, by (E.8) and (E.9), we have

we can get

Dividing (E.10) by (E.11), we can get

which indicates that the first bullet holds for time t = ˜ t as long as c 1 ≤ ( cR 2 min ) /R 2 max . For r ∈ S (0) i , we have

̸

̸

where the second inequality is by (E.12). This implies that S (0) i ⊆ S ( t ) i holds for time t = ˜ t , which completes the induction. By then, we have already proved that the first bullet and S (0) i ⊆ S ( t ) i hold for t ≤ T 1 = Cη -1 nmR -2 max .

Next, we will prove by induction that the three bullets as well as S (0) i ⊆ S ( t ) i hold for any time t ≥ 0 . The second and third bullets are obvious at t = 0 as all the coefficients are zero. Suppose there exists ˜ t such that the three bullets as well as S (0) i ⊆ S ( t ) i hold for all time 0 ≤ t ≤ ˜ t -1 . We aim to prove that they also hold for t = ˜ t . We first prove that the second and third bullets hold for t = ˜ t . To prove this, we first provide more precise upper and lower bounds for | ℓ ′ ( t ) i | . For lower bound, we have

and

̸

̸

̸

where the first inequality is by triangle inequality; the second inequality is by the first induction hypothesis that ρ ( t ) y i ,r,i ≥ c 1 | ρ ( t ) y i ,r ′ ,i ′ | for r ∈ S (0) i and 0 ≤ t ≤ ˜ t -1 and hence | ρ ( t ) y i ,r ′ ,i ′ | ≤ 1 c 1 | S (0) i | ∑ r ∈ S (0) i ρ ( t ) y i ,r,i ; the third inequality is by

and hence ρ ( t ) y i ,r ′ ,i ≤ ρ ( t ) y i ,r,i , ∀ r ′ ∈ S ( t ) i \ S (0) i , r ∈ S (0) i for 0 ≤ t ≤ ˜ t -1 ; the last inequality is by | S ( t ) i | ≤ m ≤ c ′ | S (0) i | and c ′ can be taken as 2 . 5 by Lemma B.2. By plugging (E.14) back into (E.13), we can get

For upper bound of | ℓ ′ ( t ) i | , we first bound F y i ( W ( t ) y i , x i ) -F -y i ( W ( t ) -y i , x i ) as follows:

̸

̸

where the first inequality is by

̸

̸

and and hence

̸

the second inequality is by the first induction hypothesis that ρ ( t ) y i ,r,i ≥ c 1 | ρ ( t ) y i ,r ′ ,i ′ | , ρ ( t ) y i ,r,i ≥ c 1 | ρ ( t ) -y i ,r ′ ,i ′ | and hence | ρ ( t ) y i ,r ′ ,i ′ | ≤ 1 c 1 | S (0) i | ∑ r ∈ S (0) i ρ ( t ) y i ,r,i , | ρ ( t ) -y i ,r ′ ,i ′ | ≤ 1 c 1 | S (0) i | ∑ r ∈ S (0) i ρ ( t ) y i ,r,i for r ∈ S (0) i and 0 ≤ t ≤ ˜ t -1 ; the third inequality is by | S ( t ) i | ≤ m ; the fourth inequality is by m ≤ c ′ | S (0) i | . Therefore,

By the induction hypothesis, we know that S (0) i ⊆ S ( t ) i for all 0 ≤ t ≤ ˜ t -1 and hence σ ′ ( ⟨ w ( t ) y i ,r , x i ⟩ ) = 1 for all r ∈ S (0) i and 0 ≤ t ≤ ˜ t -1 . By (E.15) and (E.16), it follows that for all 0 ≤ t ≤ ˜ t -1

By applying Lemma H.2 to (E.17) and taking x t = c ′ (1+ R -2 min pn/c 1 ) m ∑ r ∈ S (0) i ρ ( t ) y i ,r,i , we can get

̸

̸

By applying Lemma H.1 to (E.18) and taking x t = 1 -2 c ′ R -2 min pn/c 1 m ∑ r ∈ S (0) i ρ ( t ) y i ,r,i , we can get for any 0 ≤ t ≤ ˜ t that

where the last inequality is by η ≤ ( CR 2 max /nm ) -1 , C is a large enough constant and hence (1 -2 c ′ R -2 min pn/c 1 ) η | S (0) i |∥ x i ∥ 2 2 e 2 β /nm 2 ≤ log 2 . Since S (0) i ⊆ S ( t ) i for all 0 ≤ t ≤ ˜ t -1 and hence σ ′ ( ⟨ w ( t ) y i ,r , x i ⟩ ) = 1 for all r ∈ S (0) i and 0 ≤ t ≤ ˜ t -1 , we have

Applying this to (E.19) and (E.20), we can get

By taking

the above inequalities indicates that the second and third bullets hold at time t = ˜ t . For the first bullet, it is only necessary to consider the situation where ˜ t ≥ T 1 = Cη -1 nmR -2 max . In order to apply Lemma H.4 (requiring b > a ), we loosen the bounds in (E.21) as follows:

where we use 0 . 4 m ≤ | S (0) i | ≤ 0 . 6 m .

Accordingly, it holds that

By applying Lemma H.4 to (E.22) and (E.23), we can get for any i, i ′ ∈ [ n ] , r ∈ S (0) i , r ′ ∈ S (0) i ′ and T 1 ≤ t ≤ ˜ t that

Notice that S (0) i ′ ⊆ S ( t ) i ′ for all 0 ≤ t ≤ ˜ t -1 and hence σ ′ ( ⟨ w ( t ) y i ′ ,r , x i ′ ⟩ ) = 1 for all r ∈ S (0) i ′ and 0 ≤ t ≤ ˜ t -1 , we have

and hence | ρ ( t ) j,r ′ ,i ′ | ≤ ρ ( t ) y i ′ ,r ′ ,i ′ for 0 ≤ t ≤ ˜ t . Therefore, as long as

the first bullet hold for time t = ˜ t . This condition holds as long as

Finally, we verify that S (0) i ⊆ S ( ˜ t ) i . For r ∈ S (0) i , we have

̸

̸

where the second inequality is by | ρ ( ˜ t ) y i ,r,i ′ | ≤ ρ ( ˜ t ) y i ,r,i /c 1 . This implies that S (0) i ⊆ S ( t ) i holds for time t = ˜ t , which completes the induction.

Next, we show that | ρ ( t ) -y i ,r,i ′ | will be much smaller than | ρ ( t ) y i ,r ′ ,i | as the training goes on.

Lemma E.2. There exists time T 2 and constant c such that for any time t ≥ T 2

where r ∈ [ m ] , r ′ ∈ S (0) i and i, i ′ ∈ [ n ] satisfying -y i = y i ′ .

Proof of Lemma E.2. First, we will prove by induction that for r ∈ [ m ] , r ′ ∈ S (0) i and i, i ′ ∈ [ n ] satisfying -y i = y i ′ it holds that

This result holds naturally when t = 0 since all the coefficients are zero. Suppose that there exists time ˜ t such that the induction hypothesis (E.24) holds for all time t ≤ ˜ t -1 . We aim to prove that (E.24) also holds for t = ˜ t . In the following analysis, two cases will be considered:

̸

̸

̸

For if | ρ ( ˜ t -1) y i ,r,i ′ | > β + ∑ k = i ′ | ρ ( ˜ t -1) y i ,r,k |∥ x k ∥ -2 2 p , then by the decomposition (5.1) we have

̸

and hence

Therefore, by induction hypothesis (E.24) at time t = ˜ t -1 , we have

̸

For if | ρ ( ˜ t -1) y i ,r,i ′ | ≤ β + ∑ k = i ′ | ρ ( ˜ t -1) y i ,r,k |∥ x k ∥ -2 2 p , by the first bullet in Lemma E.1, we have

̸

and thus

where the last inequality is by (E.25) and η ≤ ( CR 2 max /nm ) -1 with a sufficiently large constant C . This demonstrates that inequality (E.24) holds for t = ˜ t , thereby completing the induction process. By Lemma E.1, we know that there exists time T ′ such that

for any time t ≥ T ′ . Taking T 2 = T ′ and c = 2 /c 1 , we have

̸

which completes the proof.

Given Lemma E.2, the following corollary can be directly obtained.

Corollary E.3. There exists time T 3 and constant c such that for any time t ≥ T 3

F Stable Rank of ReLU Network

In this section, we consider the properties of stable rank of the weight matrix W ( t ) found by gradient descent at time t , defined as ∥ W ( t ) ∥ 2 F / ∥ W ( t ) ∥ 2 2 . Given Lemma E.1, we have following coefficient update rule for any t ≥ 0 , i ∈ [ n ] and r ∈ S (0) i :

where

Now we are ready to prove the first bullet of Theorem 4.3.

Lemma F.1. For two-layer ReLU neural network defined in (E.1), under the same condition as Theorem 4.3, the stable rank of W ( t ) j satisfies the following property:

where C = Θ(1) is a constant.

Proof of Lemma F.1. By decomposition (5.1), we have

and

Let a i ( t ) ⊤ be the i -th column of A t . It follows that

and

Accordingly, we have

On the other hand, we will give an lower bound for ∥ W ( t ) j -W (0) j ∥ 2 .

We first provide a lower bound for ∥ A t 1 n ∥ 2 . Note that

we need to bound ∑ n i =1 ρ ( t ) j,r,i ∥ x i ∥ -2 2 , r ∈ [ m ] . By Corollary E.3, there exists time T such that for any t ≥ T , | ρ ( t ) y i ,r,i ′ | ≤ cR -2 min pnρ ( t ) y i ,r,i , ∀ i ∈ S (0) r and ∀ i ′ ∈ [ n ] with y i ′ = -y i . Therefore, we have for t ≥ T that

̸

̸

where the second inequality is by Corollary E.3 and hence

the last inequality is by

Then, we have for t ≥ T that

where the second inequality is is by (F.2); the third inequality is by the second bullet of Lemma E.1. For ∥ X ⊤ ( XX ⊤ ) -1 1 n ∥ 2 , we have

̸

By the Gershgorin circle theorem, we know that λ min ( XX ⊤ ) lies within at least one of the Gershgorin discs D (( XX ⊤ ) ii , R i ) , i ∈ [ n ] where D (( XX ⊤ ) ii , R i ) is a closed disc centered at ( XX ⊤ ) ii = ∥ x i ∥ 2 2 with radius R i = ∑ i ′ = i | ( XX ⊤ ) ii ′ | = ∑ i ′ = i |⟨ x i , x i ′ ⟩| . Assume λ min ( XX ⊤ ) lies within D (( XX ⊤ ) ii , R i ) , then we can get following lower bound for λ min ( XX ⊤ ) :

̸

̸

Therefore, we have

Accordingly,

Therefore, we have for t ≥ T that

which completes the proof.

Next, we will provide an example of training data satisfying the condition in Theorem 4.3 and prove that the stable rank of W ( t ) j trained by gradient descent using such data will converge to 2 ± o (1) . We first provide the following lemma about the increasing rate of coefficients ρ ( t ) j,r,i .

Lemma F.2. If training data x 1 , · · · , x n are mutually orthogonal, the activation pattern after time T = Cη -1 R -2 min /nm is determined by the activation pattern at initialization, i.e.,

for any time t ≥ T . Besides, ρ ( t ) j,r,i satisfy the following properties:

Proof of Lemma F.2. Part 1. For if ⟨ w (0) j,r , x i ⟩ < 0 , we first prove by induction that

The result is obvious at t = 0 as all the coefficients are zero. Suppose that there exists ˜ t such that (F.3) holds for all time 0 ≤ t ≤ ˜ t -1 . We aim to prove that (F.3) also holds for t = ˜ t . Recall that by (5.4), (5.5) and with (F.3) at time ˜ t -1 , we have

By (5.1) and the orthogonality of training data, we can get

Therefore, (F.3) holds at time t , which completes the induction.

For if ⟨ w j,r , x i ⟩ ≥ 0 and j = y i , we will next prove by induction that

˜ (0)

The result is natural at t = 0 . Suppose that there exists ˜ t such that (F.4) hold for all time 0 ≤ t ≤ ˜ t -1 . By (5.4) and (F.4) at time ˜ t -1 , we have

and hence the orthogonality of training data, we can get

Therefore, (F.4) hold at time ˜ t , which completes the induction.

For if ⟨ w (0) j,r , x i ⟩ ≥ 0 and j = -y i , we first show that under the same condition as Theorem 4.3 it holds that

Since T = Cη -1 R -2 min /nm , we have max j,r,i {| ρ ( t ) j,r,i |} = O (1) for t ≤ T . Therefore, we know that F +1 ( W ( t ) +1 , x i ) , F -1 ( W ( t ) -1 , x i ) = O (1) . Thus there exists a positive constant c such that -ℓ ′ ( t ) i ≥ c for all i ∈ [ n ] . Here we use the method of proof by contradiction. Assume ρ ( T ) j,r,i ≥ -⟨ w (0) j,r , x i ⟩ . Since ρ ( T ) j,r,i ≤ ρ ( t ) j,r,i for 0 ≤ t ≤ T which can be seen from (5.5), we have ρ ( t ) j,r,i ≥ -⟨ w (0) j,r , x i ⟩ for all t ≤ T . Then, we can get

Therefore, by the non-negativeness of ⟨ w ( t ) j,r , x i ⟩ and (5.5), we can get

and hence

which is a contradiction. Therefore, ρ ( T ) j,r,i < -⟨ w (0) j,r , x i ⟩ . By (5.5), we have ρ ( t ) j,r,i ≤ ρ ( T ) j,r,i < -⟨ w (0) j,r , x i ⟩ for t ≥ T . Therefore,

Plugging this into (5.5) gives us

This completes the proof of the first half of the lemma about the activation pattern as well as the first two properties of ρ ( t ) j,r,i .

Part 2. Now we will show that

if ⟨ w ( t ) y i ,r , x i ⟩ ≥ 0 . By the activation pattern, we can get

Given this activation pattern, we can get for t ≥ 0 that

and

Therefore, we can get following upper and lower bounds for | ℓ ′ ( t ) i | :

And it follows that

By leveraging Lemma H.1 as well as Lemma H.2 and taking

we can get

Therefore, we have

Since ρ ( t ) y i ,r,i = ρ ( t ) y i ,r ′ ,i for any r = r ∈ S (0) i , we have

which completes the proof.

Lemma F.3. For two-layer ReLU neural network defined in (E.1), there exists mutually orthogonal data x 1 , · · · , x n such that stable rank of W ( t ) j will converge to 2 ± o (1) .

Proof of Lemma F.3. By (5.1), we have

Given the definition of v ( t ) j,r , we have the following representation of v ( t ) j,r and V ( t ) j .

Assume n is an even number and x 1 , · · · , x n/ 2 are with label +1 while x ( n/ 2)+1 , · · · , x n are with label -1 . And we take x 1 , · · · , x n as e 1 , · · · , e n . Given Lemma F.2, W ( t ) j = W (0) j + V ( t ) j and such selection of training data, we have

where

̸

Then, we can get the stable rank limits as follows:

Since x 1 = e 1 , · · · , x n = e n , we can get

Therefore, the entries of matrix C and matrix D can be regarded as i.i.d. random variables taking 0 or 1 with equal probability. For ∥ A ∥ F and ∥ B ∥ F , we have

By Lemma B.2, we have with probability at least 1 -δ that 0 . 4 m ≤ | S (0) i | ≤ 0 . 6 m . By Hoeffding's inequality, we have with probability at least 1 -2 δ that

Next, we estimate ∥ A ∥ 2 and ∥ B ∥ 2 . Let A = ˜ A + E [ A ] and B = ˜ B + E [ B ] . Assume G be the m × ( n/ 2) matrix with all entries equal to 1 / 2 . Then,

And the entries of matrix ˜ A and matrix ˜ B are independent, mean zero, sub-gaussian random variables. By Lemma H.3, we have with probability at least 1 -δ that

where C is a constant. Let 1 k denote the row vector with k entries equal to 1. Then for E [ A ] and E [ B ] , we have

and

By triangle inequality, we have

Notice that ∥ a ∥ 2 2 = Θ( n ) and ∥ b ∥ 2 2 = Θ( n ) , then with probability at least 1 -2 δ , we have

This leads to

For W ( t ) , we have the following lower bound

where the third inequality is by (5 / 3) √ n ≤ ∥ a ∥ 2 ≤ (5 / 2) √ n and (5 / 3) √ n ≤ ∥ b ∥ 2 ≤ (5 / 2) √ n due to 0 . 4 m ≤ | S (0) i | ≤ 0 . 6 m . And

where the third inequality is by (5 / 3) √ n ≤ ∥ a ∥ 2 ≤ (5 / 2) √ n and (5 / 3) √ n ≤ ∥ b ∥ 2 ≤ (5 / 2) √ n due to 0 . 4 m ≤ | S (0) i | ≤ 0 . 6 m . Therefore,

G Margin Results and Loss Convergence

In this section, we prove the convergence rate of training loss as well as the increasing rate of margin for both two-layer ReLU and leaky ReLU networks defined as

We first prove the following auxiliary lemma.

Lemma G.1. For both two-layer leaky ReLU and ReLU neural networks defined in (G.1), the following properties hold for any t ≥ 0 :

y i f ( W ( t ) , x i ) ≥ -c for any i ∈ [ n ] where c is a positive constant.

y i f ( W ( t ) , x i ) -y k f ( W ( t ) , x k ) ≤ C 1 for any i, k ∈ [ n ] where C 1 is a constant.

ℓ ′ ( t ) i /ℓ ′ ( t ) k ≤ C 2 for any i, k ∈ [ n ] where C 2 is a constant.

S ( t ) i ⊆ S ( t +1) i for any i ∈ [ n ] , where S ( t ) i := { r ∈ [ m ] : ⟨ w ( t ) y i ,r , x i ⟩ ≥ 0 } .

Proof of Lemma G.1. We prove this lemma by induction. When t = 0 , since

the first bullet holds as long as c ≥ 2 β . We also have

which verifies the second bullet at time t = 0 as long as C 1 ≥ 4 β . This leads to

as long as C 2 ≥ 2(1 + exp( C 1 )) . For any r ∈ S (0) i , we have

̸

̸

where the second equality is by ⟨ w (0) y i ,r , x i ⟩ ≥ 0 ; the first inequality is by triangle inequality; the second inequality is by | ℓ ′ (0) i ′ | / | ℓ ′ (0) i | ≤ C 2 and the condition that R 2 min ≥ Cnp , C is a sufficiently large constant. This verifies the fourth bullet at time t = 0 .

Now suppose there exists time ˜ t ≥ 0 such that these four hypotheses hold for any 0 ≤ t ≤ ˜ t . We aim to prove that these conditions also hold for t = ˜ t +1 . We first prove that y i f ( W ( ˜ t +1) , x i ) ≥ y i f ( W ( ˜ t ) , x i ) . We have

̸

By the fourth induction hypothesis at time t = ˜ t , we have S ( ˜ t +1) i ⊆ S ( ˜ t ) i and hence

for j = y i and r ∈ S ( t ) i . For σ ∈ { ReLU , leaky ReLU } , σ is non-decreasing and 1 -Lipschitz continuous, which gives

̸

Then, we have

where the first inequality is by (G.2), (G.3), σ ′ ∈ [0 , 1] and triangle inequality; the second inequality is by | ℓ ′ ( ˜ t ) i ′ | / | ℓ ′ ( ˜ t ) i | ≤ C 2 , | S ( ˜ t ) i | ≥ | S (0) i | ≥ 0 . 4 m and the condition that R 2 min ≥ Cnp , C is a sufficiently large constant. And

̸

where the first inequality is by (G.3), σ ′ ∈ [0 , 1] and triangle inequality; the second inequality is by | ℓ ′ ( ˜ t ) i ′ | / | ℓ ′ ( ˜ t ) i | ≤ C 2 , | S ( ˜ t ) i | ≥ | S (0) i | ≥ 0 . 4 m and the condition that R 2 min ≥ Cnp , C is a sufficiently large constant. Now, we obtain

which implies that y i f ( W ( ˜ t +1) , x i ) ≥ y i f ( W ( ˜ t ) , x i ) ≥ -c . This verifies the first bullet at time t = ˜ t +1 . By subtracting (G.5) from (G.4), we have

If | ℓ ′ ( ˜ t ) i | / | ℓ ′ ( ˜ t ) k | < 15 R 2 , then by Lemma H.5

and hence

Combining the two cases, we have

as long as C 1 ≥ max { 4 β, log(60 R 2 ) + 1 } , which verifies the fourth bullet at time t = ˜ t +1 . By Lemma H.5, this leads to

as long as C 2 ≥ 2(1 + exp( C 1 )) . For any r ∈ S ( ˜ t +1) i , we have

̸

̸

where the second equality is by ⟨ w ( ˜ t +1) y i ,r , x i ⟩ ≥ 0 ; the first inequality is by triangle inequality; the second inequality is by | ℓ ′ ( ˜ t +1) i ′ | / | ℓ ′ ( ˜ t +1) i | ≤ C 2 and the condition that R 2 min ≥ Cnp , C is a sufficiently large constant. This verifies the fourth bullet at time t = ˜ t +1 .

Notice that ∥ W ( t ) ∥ F = Θ(log t ) and considering the fact that the difference between any two margins can be bounded by a constant, the difference between any two margins can be bounded by a constant, we can derive the following lemma, which demonstrates that the normalized margin of all the training data points will converge to the same value.

Lemma G.2. For both two-layer ReLU and leaky ReLU neural networks, gradient descent will asymptotically find a neural network such that all the training data points possess the same normalized margin, i.e.,

for any i, k ∈ [ n ] .

By Lemma G.1, we can establish the subsequent lemma regarding the logarithmic rate of increase in margin. This lemma will be beneficial in demonstrating the convergence rate of the training loss in subsequent proofs.

Lemma G.3. There exists time T = Θ( η -1 R -2 min nm ) such that the following increasing rate of margin y i f ( W ( t ) , x i ) holds:

for any i ∈ [ n ] and t ≥ T , where C 3 , C 4 , C 5 are constants.

Proof of Lemma G.3. To prove this, we want to leverage Lemma H.1 and Lemma H.2. To achieve this, we need approximate | ℓ ′ ( t ) i | by y i f ( W ( t ) , x i ) . We have

and

by y i f ( W ( t ) y i , x i ) ≥ -c . Plugging the upper and lower bounds of | ℓ ′ ( t ) i | into (G.4) and (G.5), we obtain

By taking x t = y i f ( W ( t ) , x i ) -y i f ( W (0) , x i ) and applying Lemma H.1 to (G.8), we can get

As long as t ≥ e y i f ( W (0) , x i ) nm 3 η ∥ x i ∥ 2 2 exp ( -3 η ∥ x i ∥ 2 2 e y i f ( W (0) , x i ) nm ) = Θ( η -1 R -2 min nm ) , we have

where the last inequality is by y i f ( W (0) , x i ) ≤ 2 β and η ≤ ( CR 2 max /nm ) -1 , C is a sufficiently large constant. By taking x t = y i f ( W ( t ) , x i ) -y i f ( W (0) , x i ) and applying Lemma H.2 to (G.9), we can get

Since

Taking

completes the proof of the first equation. By plugging the margin upper bound into (G.6), we can get

By plugging the margin lower bound into (G.7), we can get

Therefore, taking C 4 = 1 /e C 3 (1 + e c ) and C 5 = exp( C 3 ) completes the proof.

Now we give the following lemma about the convergence rate of training loss.

Lemma G.4. For both two-layer ReLU and leaky ReLU networks defined in (G.1), we have the following convergence rate of training loss

Proof. Having obtained a lower bound for the margin in Lemma G.3, we can now use it to derive an upper bound for the loss function as follows:

where the first inequality is by log(1 + z ) ≤ z ; the second inequality is by Lemma G.3. Having obtained an upper bound for the margin in Lemma G.3, we can now use it to derive a lower bound for the loss function as follows:

where the first inequality is by log(1 + z ) ≥ z -z 2 / 2 for z ≥ 0 ; the second inequality is by Lemma G.3. This completes the proof.

In addition to the aforementioned lemmas, in the case of leaky ReLU, assuming convergence in direction, we can demonstrate that the directional limit corresponds to a Karush-Kuhn-Tucker (KKT) point of the max-margin problem. This result is presented in the following lemma.

Lemma G.5. For two-layer leaky ReLU network defined in (G.1), assume that W ( t ) converges in direction, i.e. the limit of W ( t ) / ∥ W ( t ) ∥ F exists. Denote lim t →∞ W ( t ) / ∥ W ( t ) ∥ F as ¯ W . There exists a scaling factor α > 0 such that α ¯ W satisfies Karush-Kuhn-Tucker (KKT) conditions of the following max-margin problem:

Proof. We need to prove that there exists λ 1 , · · · , λ n ≥ 0 such that for every j ∈ {± 1 } and r ∈ [ m ] we have

By (5.1), we know that

where the second equality is by ∥ W ( t ) ∥ F = Θ(log t ) and the last equality is by the existence of lim t →∞ W ( t ) / ∥ W ( t ) ∥ F as well as the uniqueness of data-correlated decomposition. By Lemma D.3 and ∥ W ( t ) ∥ F = Θ(log t ) , we can obtain that

for any j ∈ {± 1 } , r, r ′ ∈ [ m ] , i ∈ [ n ] , and

for any j ∈ {± 1 } , i, i ′ ∈ [ n ] with j = y i , j = -y i ′ and r, r ′ ∈ [ m ] . Define

By (G.12), we know λ ′ i is well defined and λ ′ i ≥ 0 . And by (G.13), we know that for any r ∈ [ m ] , i ∈ [ n ] ,

Then, we have

By Lemma C.5, it holds for any t ≥ T 1 that

This leads to

Thus, we can get

Taking λ i = λ ′ i ∥ x i ∥ -2 2 completes the proof of (G.11). On the other hand, by Lemma G.2 and the assumption of the existence of W ( t ) / ∥ W ( t ) ∥ F , we can get

for any i, k ∈ [ n ] . Taking α = 1 /y i f ( ¯ W , x i ) , we have

for any i ∈ [ n ] , which completes the proof.

H Auxiliary Lemmas

Lemma H.1. Let { x t } ∞ t =0 be an non-negative sequence satisfying the following inequality:

then we have

Evaluating the integral, we get:

Rearranging the inequality, we get:

then we have

Proof of Lemma H.1. Given the inequality x t +1 -x t ≤ C · e -x t for all t ≥ 0 , we want to prove that x T ≤ log( e x 0 + Ce C · T ) for T ≥ 0 . We start by manipulating the inequality as follows:

Summing the inequality from t = 0 to t = T -1 , we get:

Since e x is a monotone increasing function, we can approximate the above sum with an integral:

Taking the natural logarithm of both sides, we get:

Therefore, we have shown that x T ≤ log( e x 0 + Ce C · T ) , as required.

Lemma H.2. Let { x t } ∞ t =0 be an sequence satisfying the following inequality:

Proof of Lemma H.2. Given the inequality x t +1 -x t ≥ C · e -x t for all t ≥ 0 , we want to prove that x T ≥ log( e x 0 + C · T ) for T ≥ 0 . We start by manipulating the inequality as follows:

t (multiplying both sides by e x t ) .

Summing the inequality from t = 0 to t = T -1 , we get:

Since e x is a monotone increasing function, we can approximate the above sum with an integral:

Evaluating the integral, we get:

Rearranging the inequality, we get:

if b > a > 0 .

Proof of Lemma H.4. Let f ( t ) = log(1 + at ) / log(1 + bt ) , and we want to prove that f ′ ( t ) > 0 for all t > 0 . To find the derivative of f ( t ) , we use the quotient rule:

Next, we define the function g ( t ) = ( 1 b + t ) log(1 + bt ) -( 1 a + t ) log(1 + at ) , and we aim to show that g ′ ( t ) > 0 for all t > 0 . We start by computing the derivative of g ( t ) :

Since b > a and t > 0 , we have 1 + bt > 1 + at , which implies that log(1 + bt ) > log(1 + at ) . Therefore, we have g ′ ( t ) > 0 for all t > 0 . Note that g (0) = 0 , we then have g ( t ) > 0 for all t > 0 . Therefore, we have a (1 + bt ) log(1 + bt ) -b (1 + at ) log(1 + at ) > 0 for all t > 0 , which in turn implies that f ′ ( t ) > 0 for all t > 0 . Thus, we have shown that f ( t ) is increasing for t > 0 and hence f ( t ) > f ( s ) , which completes the proof.

Lemma H.5. Let g ( z ) = ℓ ′ ( z ) = -1 / (1 + exp( z )) , then we have that

and

Proof of Lemma H.5. We first prove the first inequality. For if z 1 ≤ 0 , we have

Taking the natural logarithm of both sides, we get:

Therefore, we have shown that x T ≥ log( e x 0 + C · T )

, as required.

Lemma H.3 (Theorem 4.4.5 in Vershynin (2018)) . Let A be an m × n random matrix whose entries a ij are independent, mean zero, sub-gaussian random variables. Then for any t > 0 we have

with probability at least 1 -2 exp( -t 2 ) . Here K = max i,j ∥ a ij ∥ ϕ 2 where ∥ · ∥ ϕ 2 is the sub-gaussian norm.

Lemma H.4. For t ≥ s > 0 , we have

For if z 1 > 0 , we have

Thus, for any z 1 ∈ R , we have

Now we prove the second inequality. We have

which completes the proof.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies convergence and implicit bias in two-layer (leaky) ReLU networks trained with gradient descent. They focus on the case where the initialization scale and the step size of GD are sufficiently small, and the training data are nearly orthogonal. They show two main results: \n-  For leaky ReLU networks, they prove convergence to stable-rank 1 of the weight matrices (they consider the matrix of the positive neurons and the matrix of the negative neurons separately), and also give a convergence rate analysis; \n-  For ReLU networks they bound the stable ranks by a constant, which in some special cases can be roughly 2. They analyze the convergence rate also here.\n\nSTRENGTHS:\nI think that it is a nice paper. The near-orthgonality assumption is quite strong, but I believe that it is a natural assumption, and that understanding the behavior of GD in this setting is helpful. For this reason, this assumption appears (either explicitly or implicitly) in several works in recent years.\n\nThis work extends the results from Frei et al. 2022b in several ways: \n- Frei et al. 2022b showed convergence to rank 1 of the matrices when using gradient flow and constant stable rank when using GD, and here it is extended to stable rank 1 using GD; \n- Frei et al. 2022b showed convergence to a constant stable rank when using GD with smooth leaky ReLU activation, and here it is extendedto leaky ReLU and to ReLU; \n- The convergence rate here is tighter than Frei et al. 2022b.\n\nI think that the question considered in the paper is interesting and the contribution is significant and non-trivial. Also, the paper is well-written.\n\nWEAKNESSES:\nI don’t see major weaknesses. Below I discuss some comments/questions that I have.\n\n- In the introduction the authors argue that “This finding suggests that ReLU networks possess superior learning ability compared to leaky ReLU networks”. I don’t see how this claim follows from the results. This sentence appears after mentioning the example where the stable rank becomes approximately 2 in ReLU networks, but this occurs also in leaky ReLU networks for this example. \n- Related to the above comment: Relating low stable rank to learning ability is not a trivial argument. While the exact rank can be used for bounding the complexity of the hypothesis class, this is not true in general for the stable rank. So I don’t see a direct argument that gives learning guarantees by only using the stable rank. (Still, I think that the stable rank is a natural notion and hence studying it is of interest).\n- The experiments on MNIST are conceptually similar to the experiments on CIFAR from Frei et al. 2022b. Please explain what is the contribution of this experiment compared to this prior work.\n- Typos:\n    - In Lemma 5.4 (line 215): should be $i,i’ \\in [n]$ instead of $r,r’ \\in [m]$.\n    - Line 286: “RelU”\n    - Line 759 (in Lemma F.2): \n        - In the first property should be $\\rho_{j,r,i}$\n        - The second property should be without the \\bar{}\n        - All conditions should consiter $w^{(0)}$ rather than $w^{(t)}$\n    - Last equation in page 42: should be $0_{m \\times(d-n)}$. Also in the equation before that.\n\nQUESTIONS:\nSee the “weaknesses” section.\n\nLIMITATIONS:\nThe authors discussed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates implicit bias, especially in terms of the stable rank of the weight matrix (which is the square of the ratio between the Frobenius and spectral norms), of gradient descent on the logistic loss for binary classification tasks and two-layer neural networks, where the first layer is trained and the second layer is fixed, the activation function is either ReLU or Leaky ReLU, and the training data are nearly orthogonal.  The main results are that, provided the learning rate and the initialisation scale are sufficiently small, as the number of training steps tends to infinity, the stable rank tends to 1 in the Leaky ReLU case, and is bounded by a constant in the ReLU case.  The authors also prove upper bounds on the convergence rate of the empirical loss, and tight bounds on the convergence rate of the norms of individual neurons in the Leaky ReLU case.  The theoretical results are supplemented by experiments on synthetic data generated by adding Gaussian noise, and on MNIST.\n\nSTRENGTHS:\nThe paper tackles one of the greatest current challenges in the theory of learning, namely understanding the implicit bias of gradient-based algorithms and its consequences for generalisation.  In contrast to most related works, it considers gradient descent rather than gradient flow, and is thus more readily applicable to practical settings.  The activation functions that it focuses on, namely ReLU and Leaky ReLU, are among the most popular.  The theoretical results are proved in detail in the extensive appendix, and sufficient details as well as the code for the experiments are provided.\n\nWEAKNESSES:\nThe paper is a follow-up to \"Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Data\" by Frei, Vardi, Bartlett, Srebro, and Hu (ICLR 2023), and shares most of the assumptions with the gradient descent part of that work, with the important difference that in the present paper the activation functions are ReLU and Leaky ReLU rather than a smooth variant of Leaky ReLU as in that work.  It is not very clear to me what difficulties exactly needed to be overcome to step up from that previous work to this paper, or in other words to what extent and in what ways the present paper is not a straightforward increment.  Another aspect that is similarly unclear to me is how essentially different are the ReLU and Leaky ReLU cases.  The theorems and proofs for them are presented in the main and in the appendix mostly separately, yet seem to follow similar patterns, so it would be good to clarify where the main underlying differences are.  A minor comment is that there is a typo saying that the signal-noise decomposition technique is introduced in the Preliminaries.\n\nQUESTIONS:\nWhat can you say about what happens if the second layer is trained as well, or if the network is deeper?\n\nLIMITATIONS:\nTraining only the first layer significantly simplifies the dynamics.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-15"}

comments: SUMMARY:
This paper studies training two-layer leaky-ReLU/ReLU networks with gradient descent. The authors show that for approximately orthogonal data, GD converges and the weight matrices in the leaky-ReLU network have a stable rank of one asymptotically under GD, while those in the ReLU network can have a stable rank larger than one.

STRENGTHS:
1. The convergence analysis of GD on two-layer neural networks seems novel
2. The characterization of the stable rank of the weight matrices is interesting

WEAKNESSES:
1. There is little discussion on the upper bound $c$ in Theorem 4.3. How large it can be? How does it depend on the training data?
2. For leaky-ReLU, the theorem only says it has a stable-rank one, meaning $W_j$ is approximately rank-one, is there a characterization of the left and right dominant singular vectors?

QUESTIONS:
See "Weakness"

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis study analyzes two-layer ReLU and leaky ReLU networks trained by gradient descent in a classification problem. It is shown that when training data is mutually nearly-orthogonal, by taking an initialization scale and a learning rate sufficiently small, a leaky ReLU network converges to a parameter with stable rank 1. They also show that a ReLU network converges to a parameter with a bounded stable rank, and it will equal 2 when the input is completely orthogonal. A similar analysis has already been provided by Frei et al. 2022 (with gradient flow and Leaky ReLU), this work takes another approach apart from the KKT condition, and a possibly faster convergence rate is obtained.\n\nSTRENGTHS:\n- Frei et al. (2022) have already shown a certain result on this topic, but it requires several non-practical modifications of the algorithm \n(such as considering gradient flow or smoothed leaky ReLU).\nThis study overcomes this shortcoming with a novel approach and obtains further results (extension for the ReLU activation and gradient descent without smoothing), which has an important novelty.\n\nWEAKNESSES:\n- The near-orthogonality assumption $R_{\\min}^2\\ge CR^2\\gamma^{-4}np$ seems to be strong (while Frei et al. (2022) also impose the same condition, but I realize that they assume high-dimensional settings).\n\n- It will be more interesting if the authors can characterize the direction to which the weight matrix converges. \n\n- The authors refer to the works on benign overfitting in the related work section, but the connection of this study to them needs to be made clearer. Does this study focus on high-dimensional settings? Or is the double descent curve provably obtained by this study?\n\n- Some terms are used without definition, such as $W_{y_i}^{(t)}$ in (5.8) and $w_{y_i,r}^{(0}$ in Lemma 5.5, and so on.\n\nQUESTIONS:\n- The authors conduct numerical experiments and observe the stable rank of trained networks, but I'm wondering about it precisely reflects the theoretical result. That is, whether the stable rank of the leaky ReLU network with large $\\gamma$ converges to 1 or not, and the stable rank of the ReLU network could converge to 2 by using fully-orthogonal training data.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}