Title: Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputs
Venue: NeurIPS 2023
Accepted: yes
Keywords: implicit bias

implicit regularization

training dynamics

ReLU networks

gradient flow

theoretical analysis
Abstract: We prove that, for the fundamental regression task of learning a single neuron, training a one-hidden layer ReLU network of any width by gradient flow from a small initialisation converges to zero loss and is implicitly biased to minimise the rank of network parameters.  By assuming that the training points are correlated with the teacher neuron, we complement previous work that considered orthogonal datasets.  Our results are based on a detailed non-asymptotic analysis of the dynamics of each hidden neuron throughout the training.  We also show and characterise a surprising distinction in this setting between interpolator networks of minimal rank and those of minimal Euclidean norm.  Finally we perform a range of numerical experiments, which corroborate our theoretical findings.

Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputs

Abstract

We prove that, for the fundamental regression task of learning a single neuron, training a one-hidden layer ReLU network of any width by gradient flow from a small initialisation converges to zero loss and is implicitly biased to minimise the rank of network parameters. By assuming that the training points are correlated with the teacher neuron, we complement previous work that considered orthogonal datasets. Our results are based on a detailed non-asymptotic analysis of the dynamics of each hidden neuron throughout the training. We also show and characterise a surprising distinction in this setting between interpolator networks of minimal rank and those of minimal Euclidean norm. Finally we perform a range of numerical experiments, which corroborate our theoretical findings.

1 Introduction

One of the grand challenges for machine learning research is to understand how overparameterised neural networks are able to fit perfectly the training examples and simultaneously to generalise well to unseen data [Zhang, Bengio, Hardt, Recht, and Vinyals, 2021]. The double-descent phenomenon [Belkin, Hsu, Ma, and Mandal, 2019], where increasing the neural network capacity beyond the interpolation threshold can eventually reduce the test loss much further than could be achieved around the underparameterised 'sweet spot', is a mystery from the standpoint of classical machine learning theory. This has been observed to happen even for training without explicit regularisers.

Implicit bias of gradient-based algorithms. A key hypothesis towards explaining the doubledescent phenomenon is that the gradient-based algorithms that are used for training are implicitly biased (or implicitly regularised ) [Neyshabur, Bhojanapalli, McAllester, and Srebro, 2017] to converge to solutions that in addition to fitting the training examples have certain properties which cause them to generalise well. It has attracted much attention in recent years from the research community, which has made substantial progress in uncovering implicit biases of training algorithms in many important settings [Vardi, 2023]. For example, for classification tasks, and for homogeneous networks (which is a wide class that includes ReLU networks provided they contain neither biases at levels deeper than the first nor residual connections), Lyu and Li [2020] and Ji and Telgarsky [2020] established that gradient flow is biased towards maximising the classification margin in parameter space, in the sense that once the training loss gets sufficiently small, the direction of the parameters subsequently converges to a Karush-Kuhn-Tucker point of the margin maximisation problem.

Insights gained in this foundational research direction have not only shed light on overparameterised generalisation, but have been applied to tackle other central problems, such as the susceptibility of networks trained by gradient-based algorithms to adversarial examples [Vardi, Yehudai, and Shamir, 2022] and the possibility of extracting training data from network parameters [Haim, Vardi, Yehudai, Shamir, and Irani, 2022].

Regression tasks and initialisation scale. Showing the implicit bias for regression tasks, where the loss function is commonly mean square, has turned out to be more challenging than for classification tasks, where loss functions typically have exponential tails. A major difference is that, whereas most of the results for classification do not depend on how the network parameters are initialised, the scale of the initialisation has been observed to affect decisively the implicit bias of gradient-based algorithms for regression [Woodworth, Gunasekar, Lee, Moroshko, Savarese, Golan, Soudry, and Srebro, 2020]. When it is large so that the training follows the lazy regime , we tend to have fast convergence to a global minimum of the loss, however without an implicit bias towards sparsity and with limited generalisation [Jacot, Ged, ¸ Sim¸ sek, Hongler, and Gabriel, 2021]. The focus, albeit at the price of uncertain convergence and lengthier training, has therefore been on the rich regime where the initialisation scale is small.

Considerable advances have been achieved for linear networks. For example, Azulay, Moroshko, Nacson, Woodworth, Srebro, Globerson, and Soudry [2021] and Yun, Krishnan, and Mobahi [2021] proved that gradient flow is biased to minimise the Euclidean norm of the predictor for one-hidden layer linear networks with infinitesimally small initialisation, and that the same holds also for deeper linear networks under an additional assumption on their initialisation. A related extensive line of work is on implicit bias of gradient-based algorithms for matrix factorisation and reconstruction, which has been a fruitful test-bed for regression using multi-layer networks. For example, Gunasekar, Woodworth, Bhojanapalli, Neyshabur, and Srebro [2017] proved that, under a commutativity restriction and starting from a small initialisation, gradient flow is biased to minimise the nuclear norm of the solution matrix; they also conjectured that the restriction can be dropped, which after a number of subsequent works was refuted by Li, Luo, and Lyu [2021], leading to a detailed analysis of both underparameterised and overparameterised regimes by Jin, Li, Lyu, Du, and Lee [2023].

For non-linear networks, such as those with the popular ReLU activation, progress has been difficult. Indeed, Vardi and Shamir [2021] showed that precisely characterising the implicit bias via a non-trivial regularisation function is impossible already for single-neuron one-hidden layer ReLU networks, and Timor, Vardi, and Shamir [2023] showed that gradient flow is not biased towards low-rank parameter matrices for multiple-output ReLU networks already with one hidden layer and small training datasets.

ReLU networks and training dynamics. We suggest that, in order to further substantially our knowledge of convergence, implicit bias, and generalisation for regression tasks using non-linear networks, we need to understand more thoroughly the dynamics throughout the gradient-based training. This is because of the observed strong influence that initialisation has on solutions, but is challenging due to the highly non-convex optimisation landscape. To this end, evidence and intuition were provided by Maennel, Bousquet, and Gelly [2018], Li et al. [2021], and Jacot et al. [2021], who conjectured that, from sufficiently small initialisations, after an initial phase where the neurons get aligned to a number of directions that depend only on the dataset, training causes the parameters to pass close to a sequence of saddle points, during which their rank increases gradually but stays low.

The first comprehensive analysis in this vein was accomplished by Boursier, Pillaud-Vivien, and Flammarion [2022], who focused on orthogonal datasets (which are therefore of cardinality less than or equal to the input dimension), and established that, for one-hidden layer ReLU networks, gradient flow from an infinitesimal initialisation converges to zero loss and is implicitly biased to minimise the Euclidean norm of the network parameters. They also showed that, per sign class of the training labels (positive or negative), minimising the Euclidean norm of the interpolator networks coincides with minimising their rank.

Our contributions. We tackle the main challenge posed by Boursier et al. [2022], namely handling datasets that are not orthogonal. A major obstacle to doing so is that, whereas the analysis of the training dynamics in the orthogonal case made extensive use of an almost complete separation between a turning phase and a growth phase for all hidden neurons, non-orthogonal datasets cause

considerably more complex dynamics in which hidden neurons follow training trajectories that simultaneously evolve their directions and norms [Boursier et al., 2022, Appendix A].

To analyse this involved dynamics in a reasonably clean setting, we consider the training of onehidden layer ReLU networks by gradient flow from a small balanced initialisation on datasets that are labelled by a teacher ReLU neuron with which all the training points are correlated. More precisely, we assume that the angles between the training points and the teacher neuron are less than π/ 4 , which implies that all angles between training points are less than π/ 2 . The latter restriction has featured per label class in many works in the literature (such as by Phuong and Lampert [2021] and Wang and Pilanci [2022]), and the former is satisfied for example if the training points can be obtained by summing the teacher neuron v ∗ with arbitrary vectors of length less than ∥ v ∗ ∥ / √ 2 . All our other assumptions are very mild, either satisfied with probability exponentially close to 1 by any standard random initialisation, or excluding corner cases of Lebesgue measure zero.

Our contributions can be summarised as follows.

Weprovide a detailed non-asymptotic analysis of the dynamics of each hidden neuron throughout the training, and show that it applies whenever the initialisation scale λ is below a precise bound which is polynomial in the network width m and exponential in the training dataset cardinality n . Moreover, our analysis applies for any input dimension d > 1 , for any n ≥ d (otherwise exact learning of the teacher neuron may not be possible), for any m , and without assuming any specific random distribution for the initialisation. In particular, we demonstrate that the role of the overparameterisation in this setting is to ensure that initially at least one hidden neuron with a positive last-layer weight has in its active half-space at least one training point.

We show that, during a first phase of the training, all active hidden neurons with a positive last-layer weight get aligned to a single direction which is positively correlated with all training points, whereas all active hidden neurons with a negative last-layer weight get turned away from all training points so that they deactivate. In contrast to the orthogonal dataset case where the sets of training points that are in the active half-spaces of the neurons are essentially constant during the training, in our correlated setting this first phase in general consists, for each neuron, of a different sequence of stages during which the cardinality of the set of training points in its active half-space gradually increases or decreases, respectively.

We show that, during the rest of the training, the bundle of aligned hidden neurons with their last-layer weights, formed by the end of the first phase, grows and turns as it travels from near the origin to near the teacher neuron, and does not separate . To establish the latter property, which is the most involved part of this work, we identify a set in predictor space that depends only on λ and the training dataset, and prove: first, that the trajectory of the bundle stays inside the set ; and second, that this implies that the directional gradients of the individual neurons are such that the angles between them are non-increasing.

We prove that, after the training departs from the initial saddle, which takes time logarithmic in λ and linear in d , the gradient satisfies a Polyak-Łojasiewicz inequality and consequently the loss converges to zero exponentially fast .

We prove that, although for any fixed λ the angles in the bundle of active hidden neurons do not in general converge to zero as the training time tends to infinity, if we let λ tend to zero then the networks to which the training converges have a limit: a network of rank 1 , in which all non-zero hidden neurons are positive scalings of the teacher neuron and have positive last-layer weights. This establishes that gradient flow from an infinitesimal initialisation is implicitly biased to select interpolator networks of minimal rank . Note also that the limit network is identical in predictor space to the teacher neuron.

We show that, surprisingly, among all networks with zero loss, there may exist some whose Euclidean norm is smaller than that of any network of rank 1 . Moreover, we prove that this is the case if and only if a certain condition on angles determined by the training dataset is satisfied. This result might be seen as refuting the conjecture of Boursier et al. [2022, section 3.2] that the implicit bias to minimise Euclidean parameter norm holds beyond the orthogonal setting, and adding some weight to the hypothesis of Razin and Cohen [2020]. The counterexample networks in our proof have rank 2 and make essential use of the ReLU non-linearity.

We perform numerical experiments that indicate that the training dynamics and the implicit bias we theoretically established occur in practical settings in which some of our assumptions are relaxed. In particular, gradient flow is replaced by gradient descent with a realistic learning rate,

the initialisation scales are small but not nearly as small as in the theory, and the angles between the teacher neuron and the training points are distributed around π/ 4 .

We further discuss related work, prove all theoretical results, and provide additional material on our experiments, in the appendix.

2 Preliminaries

Notation. We write: [ n ] for the set { 1 , . . . , n } , ∥ v ∥ for the Euclidean length of a vector v , v := v / ∥ v ∥ for the normalised vector, ∡ ( v , v ′ ) := arccos( v ⊤ v ′ ) for the angle between v and v ′ , and cone { v 1 , . . . , v n } := { ∑ n i =1 β i v i | β 1 , . . . , β n ≥ 0 } for the cone generated by vectors v 1 , . . . , v n .

One-hidden layer ReLU network. For an input x ∈ R d , the output of the network is

where m is the width, the parameters θ = ( a , W ) ∈ R m × R m × d consist of last-layer weights a = [ a 1 , . . . , a m ] and hidden-layer weights W ⊤ = [ w 1 , . . . , w m ], and σ ( u ) := max { u, 0 } is the ReLU function.

Balanced initialisation. For all j ∈ [ m ] let

where λ > 0 is the initialisation scale, z j ∈ R d \ { 0 } , and s j ∈ {± 1 } .

A precise upper bound on λ will be stated in Assumption 2.

We regard the initial unscaled hidden-layer weights z j and last-layer signs s j as given, without assuming any specific random distributions for them. For example, we might have that each z j consists of d independent centred Gaussians with variance 1 dm and each s j is uniform over {± 1 } .

We consider only initialisations for which the layers are balanced, i.e. | a 0 j | = ∥ w 0 j ∥ for all j ∈ [ m ]. Since more generally each difference ( a t j ) 2 -∥ w t j ∥ 2 is constant throughout training [Du, Hu, and Lee, 2018, Theorem 2.1] and we focus on small initialisation scales that tend to zero, this restriction (which is also present in Boursier et al. [2022]) is minor but simplifies our analysis.

Neuron-labelled correlated inputs. The teacher neuron v ∗ ∈ R d and the training dataset { ( x i , y i ) } n i =1 ⊆ ( R d \ { 0 } ) × R are such that for all i we have

In particular, since the angles between v ∗ and the training points x i are acute, each label y i is positive.

To apply our results to a network with biases in the hidden layer and to a teacher neuron with a bias, one can work in dimension d +1 and extend the training points to [ x i 1 ] .

Mean square loss gradient flow. For the regression task of learning the teacher neuron by the one-hidden layer ReLU network, we use the standard mean square empirical loss

Our theoretical analysis concentrates on training by gradient flow, which from an initialisation as above evolves the network parameters by descending along the gradient of the loss by infinitesimal steps in continuous time [Li, Tai, and E, 2019]. Formally, we consider any parameter trajectory θ t : [ 0 , ∞ ) → R m × R m × d that is absolutely continuous on every compact subinterval, and that satisfies the differential inclusion

where ∂L denotes the Clarke [1975] subdifferential of the loss function (which is locally Lipschitz).

We work with the Clarke subdifferential, which is a generalisation of the gradient, because the ReLU activation is not differentiable at 0 , which causes non-differentiability of the loss function [Bolte, Daniilidis, Ley, and Mazet, 2010]. Although it follows from our results that, in our setting, the derivative of the ReLU can be fixed as σ ′ (0) := 0 like in the orthogonal case [Boursier et al., 2022, Appendix D], and the gradient flow trajectories are uniquely defined, that is not a priori clear; hence we work with the unrestricted Clarke subdifferential of the ReLU. We also remark that, in other settings, σ ′ (0) cannot be fixed in this way due to gradient flow subtrajectories that correspond to gradient descent zig-zagging along a ReLU boundary (cf. e.g. Maennel et al. [2018, section 9.4]).

Basic observations. We establish the formulas for the derivatives of the last-layer weights and the hidden neurons; and that throughout the training, the signs of the last-layer weights do not change, and their absolute values track the norms of the corresponding hidden neurons. The latter property holds for all times t by continuity and enables us to focus the analysis on the hidden neurons.

Proposition 1. For all j ∈ [ m ] and almost all t ∈ [ 0 , ∞ ) we have:

The definition in part (i) of the vectors g t j that govern the dynamics is a membership because the subdifferential of the ReLU at 0 is the set of all values between 0 and 1 , i.e. ∂σ (0) = [ 0 , 1 ].

3 Assumptions

To state our assumptions precisely, we introduce some additional notation. Let

denote the sets of indices of training points that are, respectively, either inside or on the boundary or outside of the non-negative half-space of a vector v . Then let

̸

̸

be the sets of indices of hidden neurons that are initially active on at least one training point and whose last-layer signs are, respectively, positive or negative. Also let

denote the matrix whose columns are all the training points, and the sum of all training points whose indices are in a set I , weighted by the corresponding labels and divided by n .

Moreover we define, for each j ∈ J + ∪ J -, a continuous trajectory α t j in R d by

Thus, starting from the unscaled initialisation z j of the corresponding hidden neuron, α t j follows a dynamics obtained from that of w t j in Proposition 1 (i) and (ii) by replacing the vector g t j by γ I + ( α t j ) , which amounts to removing from g t j the network output terms and the activation boundary summands. These trajectories will be useful as yardsticks in our analysis of the first phase of the training.

(iii) x 1 , . . . , x n are distinct, the eigenvalues of 1 n XX ⊤ are distinct, and v ∗ does not belong to a span of fewer than d eigenvectors of 1 n XX ⊤ .

(ii) J + = ∅ , I 0 ( z j ) = ∅ for all j ∈ [ m ] , and ∡ ( z j , γ [ n ] ) > 0 for all j ∈ J -.

̸

(iv) | I 0 ( α t j ) | ≤ 1 for all j ∈ J + ∪ J -and all t ∈ [ 0 , ∞ ) .

̸

(v) For all j ∈ [ m ] and all 0 ≤ T < T ′ , if for all t ∈ ( T, T ′ ) we have I + ( w t j ) = I 0 ( w T ′ j ) = ∅ and I 0 ( w t j ) = I + ( w T ′ j ) = ∅ , then for all t ≥ T ′ we have w t j = w T ′ j .

This assumption is very mild. Part (i) excludes the trivial univariate case without biases (for univariate inputs with biases one needs d = 2 ), ensures that exact learning is possible, and fixes the length of the teacher neuron to streamline the presentation. Part (ii) assumes that, initially: at least one hidden neuron with a positive last-layer weight has in its active half-space at least one training point, no training point is at a ReLU boundary, and no hidden neuron with a negative last-layer weight is perfectly aligned with the γ [ n ] vector; this holds with probability at least 1 -(3 / 4) m for any continuous symmetric distribution of the unscaled hidden-neuron initialisations, e.g. z j i.i.d. ∼ N ( 0 , 1 dm I d ) , and the uniform distribution of the last-layer signs s j i.i.d. ∼U{± 1 } . Parts (iii) and (iv) exclude corner cases of Lebesgue measure zero; observe that 1 n XX ⊤ is positive-definite, and that (iv) rules out a yardstick trajectory encountering two or more training points in its half-space boundary at exactly the same time. Part (v) excludes some unrealistic gradient flows that might otherwise be possible due to the use of the subdifferential: it specifies that, whenever a neuron deactivates (i.e. all training points exit its positive half-space), then it stays deactivated for the remainder of the training.

Before our next assumption, we define several further quantities. Let η 1 > · · · > η d > 0 denote the eigenvalues of 1 n XX ⊤ , and let u 1 , . . . , u d denote the corresponding unit-length eigenvectors such that v ∗ = ∑ d k =1 ν ∗ k u k for some ν ∗ 1 , . . . , ν ∗ d > 0 . Also, for each j ∈ J + ∪ J -, let n j := | I -s j ( z j ) | be the number of training points that should enter into or exit from the non-negative half-space along the trajectory α t j depending on whether the sign s j is positive or negative (respectively), and let

̸

be the evolving angle between α t j and the vector governing its dynamics (if any). Then the existence of the times at which the entries or the exits occur is confirmed in the following.

Proposition 2. For all j ∈ J + ∪ J -there exist a unique enumeration i 1 j , . . . , i n j j of I -s j ( z j ) and unique 0 = τ 0 j < τ 1 j < · · · < τ n j j such that for all ℓ ∈ [ n j ] :

Finally we define two measurements of the unscaled initialisation and the training dataset, which are positive thanks to Assumption 1, and which will simplify the presentation of our results.

̸

̸

The quantity ε introduced here has no effect on the network training, but is a parameter of our analysis, so that varying it within the assumed range tightens some of the resulting bounds while loosening others. The assumed bound on the initialisation scale λ is polynomial in the network width m and exponential in the dataset cardinality n . The latter is also the case in Boursier et al. [2022], where the bound was stated informally and without its dependence on parameters other than m and n .

4 First phase: alignment or deactivation

We show that, for each initially active hidden neuron, if its last-layer sign is positive then it turns to include in its active half-space all training points that were initially outside, whereas if its last-layer

̸

sign is negative then it turns to remove from its active half-space all training points that were initially inside. Moreover, those training points cross the activation boundary in the same order as they cross the half-space boundary of the corresponding yardstick trajectory α t j , and at approximately the same times (cf. Proposition 2).

Lemma 3. For all j ∈ J + ∪ J -there exist unique 0 = t 0 j < t 1 j < . . . < t n j j such that for all ℓ ∈ [ n j ] :

The preceding lemma is proved by establishing, for this first phase of the training, non-asymptotic upper bounds on the Euclidean norms of the hidden neurons and hence on the absolute values of the network outputs, and inductively over the stage index ℓ , on the distances between the unit-sphere normalisations of α t j and w t j . Based on that analysis, we then obtain that each negative-sign hidden neuron does not grow from its initial length and deactivates by time T 0 := max j ∈ J + ∪ J -τ n j j +1 .

Lemma 4. For all j ∈ J -we have:

We also obtain that, up to a later time T 1 := ε ln(1 /λ ) / ∥ γ [ n ] ∥ , each positive-sign hidden neuron: grows but keeps its length below 2 ∥ z j ∥ λ 1 -ε , continues to align to the vector γ [ n ] up to a cosine of at least 1 -λ ε , and maintains bounded by λ 1 -3 ε the difference between the logarithm of its length divided by the initialisation scale and the logarithm of the corresponding yardstick vector length.

Lemma 5. For all j ∈ J + we have:

5 Second phase: growth and convergence

We next analyse the gradient flow subsequent to the deactivation of the negative-sign hidden neurons by time T 0 and the alignment of the positive-sign ones up to time T 1 , and establish that the loss converges to zero at a rate which is exponential and does not depend on the initialisation scale λ .

Theorem 6. Under Assumptions 1 and 2, there exists a time T 2 < ln(1 /λ )(4 + ε ) d ∆ 2 /δ 6 such that for all t ≥ 0 we have L ( θ T 2 + t ) < 0 . 5 ∆ 2 e -t · 0 . 4 δ 4 / ∆ 2 .

In particular, for ε = 1 / 4 and λ = ( ( mn n ) 9∆ 2 /δ 3 ) -3 /ε (cf. Assumption 2), the first bound in T < (ln m + n ln n ) d 17 27 ∆ 4 /δ 9 .

The proof of Theorem 6 is in large part geometric, with a key role played by a set S := S 1 ∪··· ∪ S d in predictor space, whose constituent subsets are defined as

Theorem 6 becomes 2 · ·

where the individual constraints are as follows (here η 0 := ∞ so that e.g. η 1 2 η 0 = 0 ):

Thus S is connected, open, and constrained by Ξ to be within the ellipsoid v ⊤ XX ⊤ ( v ∗ -v ) = 0 which is centred at v ∗ 2 , with the remaining constraints slicing off further regions by straight or curved boundary surfaces.

In the most complex component of this work, we show that, for all t ≥ T 1 , the trajectory of the sum v t := ∑ j ∈ J + a t j w t j of the active hidden neurons weighted by the last layer stays inside S , and the cosines of the angles between the neurons remain above 1 -4 λ ε . This involves proving that each face of the boundary of S is repelling for the training dynamics when approached from the inside; we remark that, although that is in general false for the entire boundary of the constraint Ξ , it is in particular true for its remainder after the slicing off by the other constraints. We also show that all points in S are positively correlated with all training points, which together with the preceding facts implies that, during this second phase of the training, the network behaves approximately like a linear one-hidden layer one-neuron network. Then, as the cornerstone of the rest of the proof, we show that, for all t ≥ T 2 , the gradient of the loss satisfies a Polyak-Łojasiewicz inequality ∥∇ L ( θ t ) ∥ 2 > 2 η d ∥ γ [ n ] ∥ 5 η 1 L ( θ t ) . Here T 2 := inf { t ≥ T 1 | ν t 1 /ν ∗ 1 ≥ 1 / 2 } is a time by which the network has departed from the initial saddle, more precisely when the first coordinate ν t 1 of the bundle vector v t with respect to the basis consisting of the eigenvectors of the matrix 1 n XX ⊤ crosses the half-way threshold to the first coordinate ν ∗ 1 of the teacher neuron.

The interior of the ellipsoid in the constraint Ξ actually consists of all vectors that have an acute angle with the derivative of the training dynamics in predictor space, and the 'padding' of λ ε/ 3 is present because the derivative of the bundle vector v t is 'noisy' due to the latter being made up of the approximately aligned neurons. The remaining constraints delimit the subsets S 1 , . . . , S d of the set S , through which the bundle vector v t passes in that order, with each unique 'handover' from S ℓ to S ℓ +1 happening exactly when the corresponding coordinate ν t ℓ exceeds its target ν ∗ ℓ . The non-linearity of the constraints Ψ ↑ k,k ′ is needed to ensure the repelling for the training dynamics.

6 Implicit bias of gradient flow

Let us denote the set of all balanced networks by

and the subset in which all non-zero hidden neurons are positive scalings of v ∗ , have positive last-layer weights, and have lengths whose squares sum up to ∥ v ∗ ∥ = 1 , by

̸

Our main result establishes that, as the initialisation scale λ tends to zero, the networks with zero loss to which the gradient flow converges tend to a network in Θ v ∗ . The explicit subscripts indicate the dependence on λ of the parameter vectors. The proof builds on the preceding results and involves a careful control of accumulations of approximation errors over lengthy time intervals.

Theorem 7. Under Assumptions 1 and 2, L ( lim t →∞ θ t λ ) = 0 and lim λ → 0 + lim t →∞ θ t λ ∈ Θ v ∗ .

7 Interpolators with minimum norm

To compare the set Θ v ∗ of balanced rank1 interpolator networks with the set of all minimum-norm interpolator networks, in this section we focus on training datasets of cardinality d , we assume the network width is greater than 1 (otherwise the rank is necessarily 1 ), and we exclude the threshold case of Lebesgue measure zero where M = 0 . The latter measurement of the training dataset is defined below in terms of angles between the teacher neuron and vectors in any two cones generated by different generators of the dual of the cone of all training points.

̸

̸

Assumption 3. n = d , m> 1 , and M̸ = 0 .

We obtain that, surprisingly, Θ v ∗ equals the set of all interpolators with minimum Euclidean norm if M < 0 , but otherwise they are disjoint.

10

-

8

10

-

6

10

-

4

10

-

2

10

0

10

2

10

-

6

10

-

4

10

-

2

10

0

10

2

max. angle b. active neurons

Centred training dataset

10

-

8

10

-

6

10

-

4

10

-

2

10

0

10

2

10

-

6

10

-

4

10

-

2

10

0

10

2

Uncentred training dataset

d

= 4

d

= 16

d

= 64

d

= 256

d

= 1024

Theorem 8. Under Assumptions 1 and 3:

(i) if M < 0 then Θ v ∗ is the set of all global minimisers of ∥ θ ∥ 2 subject to L ( θ ) = 0 ;

(ii) if M > 0 then no point in Θ v ∗ is a global minimiser of ∥ θ ∥ 2 subject to L ( θ ) = 0 .

For each of the two cases, we provide a family of example datasets in the appendix. We remark that a sufficient condition for M < 0 to hold is that the inner product of any two distinct rows χ k of the inverse of the dataset matrix X is non-positive, i.e. that the inverse of the Gram matrix of the dataset (in our setting this Gram matrix is positive) is a Z-matrix (cf. e.g. Fiedler and Pták [1962]). Also, if the training points were orthogonal then all the cos ∡ ( p , q ) terms in the definition of M would be zero and consequently we would have M < 0 ; this is consistent with the result that, per sign class of the training labels in the orthogonal setting, minimising the Euclidean norm of interpolators coincides with minimising their rank [Boursier et al., 2022, Appendix C].

8 Experiments

We consider two schemes for generating the training dataset, where S d -1 is the unit sphere in R d .

Centred: We sample µ from U ( S d -1 ) , then sample x 1 , . . . , x d from N ( µ , ρ d I d ) where ρ = 1 , and finally set v ∗ = µ . This distribution has the property that, in high dimensions, the angles between the teacher neuron v ∗ and the training points x i concentrate around π/ 4 . We exclude rare cases where some of these angles exceed π/ 2 .

Uncentred: This is the same, except that we use ρ = √ 2 -1 , sample one extra point x 0 , and finally set v ∗ = x 0 . Here the angles between v ∗ and x i also concentrate around π/ 4 in high dimensions, but the expected distance between v ∗ and µ is √ ρ .

For each of the two dataset schemes, we train a one-hidden layer ReLU network of width m = 200 by gradient descent with learning rate 0 . 01 , from a balanced initialisation such that z j i.i.d. ∼N ( 0 , 1 dm I d ) and s j i.i.d. ∼U{± 1 } , and for a range of initialisation scales λ and input dimensions d . 2

We present in Figure 1 some results from considering initialisation scales λ = 4 2 , 4 1 , . . . , 4 -12 , 4 -13 and input dimensions d = 4 , 16 , 64 , 256 , 1024 , where we train until the number of iterations

10

0

10

2

10

4

10

6

10

8

10

-

27

10

-

17

10

-

7

Training loss

0

5

·

10

6

1

·

10

7

1

.

5

·

10

7

2

·

10

7

10

-

22

10

-

15

10

-

8

10

-

1

Test loss

λ

= 4

-

2

λ

= 4

-

3

λ

= 4

-

4

λ

= 4

-

5

λ

= 4

-

6

λ

= 4

-

7

λ

= 4

-

8

reaches 2 · 10 7 or the loss drops below 10 -9 . The plots are in line with Theorem 7, showing how the maximum angle between active hidden neurons at the end of the training decreases with λ .

Figure 2 on the left illustrates the exponentially fast convergence of the training loss (cf. Theorem 6), and on the right how the implicit bias can result in good generalisation. The test loss is computed over an input distribution which is different from that of the training points, namely we sample 64 test inputs from N ( 0 , I d ) . These plots are for initialisation scales λ = 4 -2 , 4 -3 , . . . , 4 -7 , 4 -8 .

9 Conclusion

We provided a detailed analysis of the dynamics of training a shallow ReLU network by gradient flow from a small initialisation for learning a single neuron which is correlated with the training points, establishing convergence to zero loss and implicit bias to rank minimisation in parameter space. We believe that in particular the geometric insights we obtained in order to deal with the complexities of the multi-stage alignment of hidden neurons followed by the simultaneous evolution of their norms and directions, will be useful to the community in the ongoing quest to understand implicit bias of gradient-based algorithms for regression tasks using non-linear networks.

A major direction for future work is to bridge the gap between, on one hand, our assumption that the angles between the teacher neuron and the training points are less than π/ 4 , and the other, the assumption of Boursier et al. [2022] that the training points are orthogonal, while keeping a fine granularity of description. We expect this to be difficult because it seems to require handling bundles of approximately aligned neurons which may have changing sets of training points in their active half-spaces and which may separate during the training. However, it should be straightforward to extend our results to orthogonally separable datasets and two teacher ReLU neurons, where each of the latter has an arbitrary sign, labels one of the two classes of training points, and has angles less than π/ 4 with them; the gradient flow would then pass close to a second saddle point, where the labels of one of the classes have been nearly fitted but the hidden neurons that will fit the labels of the other class are still small. We report on related numerical experiments in the appendix.

We also obtained a condition on the dataset that determines whether rank minimisation and Euclidean norm minimisation for interpolator networks coincide or are distinct. Although this dichotomy remains true if the π/ 4 correlation bound is relaxed to π/ 2 , the implicit bias of gradient flow in that extended setting is an open question. Other directions for future work include considering multi-neuron teacher networks, student networks with more than one hidden layer, further non-linear activation functions, and gradient descent instead of gradient flow; also refining the bounds on the initialisation scale and the convergence time.

Acknowledgments and Disclosure of Funding

We acknowledge the Centre for Discrete Mathematics and Its Applications at the University of Warwick for partial support, and the Scientific Computing Research Technology Platform at the University of Warwick for providing the compute cluster on which the experiments presented in this paper were run.

References

Shahar Azulay, Edward Moroshko, Mor Shpigel Nacson, Blake E. Woodworth, Nathan Srebro, Amir Globerson, and Daniel Soudry. On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descent. In ICML , pages 468-477, 2021. 2

Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learning practice and the classical bias-variance trade-off. Proc. Natl. Acad. Sci. , 116(32):15849-15854, 2019. 1

Jérôme Bolte, Aris Daniilidis, Olivier Ley, and Laurent Mazet. Characterizations of Łojasiewicz inequalities: subgradient flows, talweg, convexity. Trans. Amer. Math. Soc. , 362(6):3319-3363, 2010. 5

Etienne Boursier and Nicolas Flammarion. Penalising the biases in norm regularisation enforces sparsity. CoRR , abs/2303.01353, 2023. Accepted to NeurIPS 2023.

Etienne Boursier, Loucas Pillaud-Vivien, and Nicolas Flammarion. Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputs. In NeurIPS , 2022. 2, 3, 4, 5, 6, 9, 10

Frank H. Clarke. Generalized gradients and applications. Trans. Amer. Math. Soc. , 205:247-262, 1975. 4

Damek Davis, Dmitriy Drusvyatskiy, Sham M. Kakade, and Jason D. Lee. Stochastic Subgradient Method Converges on Tame Functions. Found. Comput. Math. , 20(1):119-154, 2020.

Simon S. Du, Wei Hu, and Jason D. Lee. Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balanced. In NeurIPS , pages 382-393, 2018. 4

Matthias Englert and Ranko Lazi´ c. Adversarial Reprogramming Revisited. In NeurIPS , 2022.

Tolga Ergen and Mert Pilanci. Convex Geometry and Duality of Over-parameterized Neural Networks. J. Mach. Learn. Res. , 22(212):1-63, 2021.

Mathieu Even, Scott Pesme, Suriya Gunasekar, and Nicolas Flammarion. (S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stability. CoRR , abs/2302.08982, 2023. Accepted to NeurIPS 2023.

Miroslav Fiedler and Vlastimil Pták. On matrices with non-positive off-diagonal elements and positive principal minors. Czechoslovak Mathematical Journal , 12(3):382-400, 1962. 9

Spencer Frei, Yuan Cao, and Quanquan Gu. Agnostic Learning of a Single Neuron with Gradient Descent. In NeurIPS , 2020.

Spencer Frei, Gal Vardi, Peter L. Bartlett, and Nathan Srebro. Benign Overfitting in Linear Classifiers and Leaky ReLU Networks from KKT Conditions for Margin Maximization. In COLT , pages 3173-3228, 2023a.

Spencer Frei, Gal Vardi, Peter L. Bartlett, and Nathan Srebro. The Double-Edged Sword of Implicit Bias: Generalization vs. Robustness in ReLU Networks. CoRR , abs/2303.01456, 2023b. Accepted to NeurIPS 2023.

Spencer Frei, Gal Vardi, Peter L. Bartlett, Nathan Srebro, and Wei Hu. Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Data. In ICLR , 2023c.

Suriya Gunasekar, Blake E. Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro. Implicit Regularization in Matrix Factorization. In NeurIPS , pages 6151-6159, 2017. 2

Niv Haim, Gal Vardi, Gilad Yehudai, Ohad Shamir, and Michal Irani. Reconstructing Training Data From Trained Neural Networks. In NeurIPS , 2022. 2

Arthur Jacot, François Ged, Berfin ¸ Sim¸ sek, Clément Hongler, and Franck Gabriel. Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsity. CoRR , abs/2106.15933, 2021. 2

Arnulf Jentzen and Adrian Riekert. Convergence analysis for gradient flows in the training of artificial neural networks with ReLU activation. J. Math. Anal. Appl. , 517(2):126601, 2023.

Ziwei Ji and Matus Telgarsky. Directional convergence and alignment in deep learning. In NeurIPS , 2020. 1

Jikai Jin, Zhiyuan Li, Kaifeng Lyu, Simon S. Du, and Jason D. Lee. Understanding Incremental Learning of Gradient Descent: A Fine-grained Analysis of Matrix Sensing. In ICML , pages 15200-15238, 2023. 2

Sangmin Lee, Byeongsu Sim, and Jong Chul Ye. Magnitude and Angle Dynamics in Training Single ReLU Neurons. CoRR , abs/2209.13394, 2022.

Qianxiao Li, Cheng Tai, and Weinan E. Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundations. J. Mach. Learn. Res. , 20(40):1-47, 2019. 4

Zhiyuan Li, Yuping Luo, and Kaifeng Lyu. Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learning. In ICLR , 2021. 2

Kaifeng Lyu and Jian Li. Gradient Descent Maximizes the Margin of Homogeneous Neural Networks. In ICLR , 2020. 1

Kaifeng Lyu, Zhiyuan Li, Runzhe Wang, and Sanjeev Arora. Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Bias. In NeurIPS , pages 12978-12991, 2021.

Hartmut Maennel, Olivier Bousquet, and Sylvain Gelly. Gradient Descent Quantizes ReLU Network Features. CoRR , abs/1803.08367, 2018. 2, 5

Odelia Melamed, Gilad Yehudai, and Gal Vardi. Adversarial Examples Exist in Two-Layer ReLU Networks for Low Dimensional Data Manifolds. CoRR , abs/2303.00783, 2023. Accepted to NeurIPS 2023.

Hancheng Min, René Vidal, and Enrique Mallada. Early Neuron Alignment in Two-layer ReLU Networks with Small Initialization. CoRR , abs/2307.12851, 2023.

Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring Generalization in Deep Learning. In NeurIPS , pages 5947-5956, 2017. 1

Greg Ongie, Rebecca Willett, Daniel Soudry, and Nathan Srebro. A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Case. In ICLR , 2020.

Scott Pesme and Nicolas Flammarion. Saddle-to-Saddle Dynamics in Diagonal Linear Networks. CoRR , abs/2304.00488, 2023. Accepted to NeurIPS 2023.

Mary Phuong and Christoph H. Lampert. The inductive bias of ReLU networks on orthogonally separable data. In ICLR , 2021. 3

B.T. Polyak. Gradient methods for the minimisation of functionals. USSR Computational Mathematics and Mathematical Physics , 3(4):864-878, 1963. 8

Noam Razin and Nadav Cohen. Implicit Regularization in Deep Learning May Not Be Explainable by Norms. In NeurIPS , 2020. 3

Roei Sarussi, Alon Brutzkus, and Amir Globerson. Towards Understanding Learning in Neural Networks with Linear Teachers. In ICML , pages 9313-9322, 2021.

Pedro Savarese, Itay Evron, Daniel Soudry, and Nathan Srebro. How do infinite width bounded norm networks look in function space? In COLT , pages 2667-2690, 2019.

Lawrence Stewart, Francis Bach, Quentin Berthet, and Jean-Philippe Vert. Regression as Classification: Influence of Task Formulation on Neural Network Features. In AISTATS , pages 11563-11582, 2023.

Nadav Timor, Gal Vardi, and Ohad Shamir. Implicit Regularization Towards Rank Minimization in ReLU Networks. In ALT , pages 1429-1459, 2023. 2

Gal Vardi. On the Implicit Bias in Deep-Learning Algorithms. Commun. ACM , 66(6):86-93, 2023. 1

Gal Vardi and Ohad Shamir. Implicit Regularization in ReLU Networks with the Square Loss. In COLT , pages 4224-4258, 2021. 2

Gal Vardi, Gilad Yehudai, and Ohad Shamir. Learning a Single Neuron with Bias Using Gradient Descent. In NeurIPS , pages 28690-28700, 2021.

Gal Vardi, Gilad Yehudai, and Ohad Shamir. Gradient Methods Provably Converge to Non-Robust Networks. In NeurIPS , 2022. 2

Mingze Wang and Chao Ma. Early Stage Convergence and Global Convergence of Training Mildly Parameterized Neural Networks. In NeurIPS , 2022.

Yifei Wang and Mert Pilanci. The Convex Geometry of Backpropagation: Neural Network Gradient Flows Converge to Extreme Points of the Dual Convex Program. In ICLR , 2022. 3

Blake E. Woodworth, Suriya Gunasekar, Jason D. Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and Rich Regimes in Overparametrized Models. In COLT , pages 3635-3673, 2020. 2

Weihang Xu and Simon Du. Over-Parameterization Exponentially Slows Down Gradient Descent for Learning a Single Neuron. In COLT , pages 1155-1198, 2023.

Gilad Yehudai and Ohad Shamir. Learning a Single Neuron with Gradient Methods. In COLT , pages 3756-3786, 2020.

Chulhee Yun, Shankar Krishnan, and Hossein Mobahi. A unifying view on implicit bias in training linear neural networks. In ICLR , 2021. 2

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Commun. ACM , 64(3):107-115, 2021. 1

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors analyze the dynamics and implicit bias of gradient flow with the square loss when learning a single ReLU neuron using a one-hidden-layer ReLU network. They assume that the training data are correlated with the teacher neuron (the angles are smaller than $\\pi/4$), and that gradient flow starts from a small and balanced initialization. They give a non-asymptotic convergence analysis. In the limit where the initialization scale tends to zero, the resulting network has rank $1$. Namely, all non-zero neurons point in the direction of the teacher neuron. On the other hand, the resulting network might not have minimal Euclidean norm. Thus, there is an implicit bias for rank minimization but not for norm minimization.\n\nSTRENGTHS:\nUnderstanding convergence and implicit bias in overparameterized networks is an important question, that has attracted much interest in recent years. The paper gives a detailed analysis of the trajectory and implicit bias. The analysis is under strong assumptions (single-neuron teacher, correlated training data, small and balanced initialization, etc.), but these assumptions are present also in existing results, and the analysis of gradient flow is challenging even under such assumptions. Finally, the paper is well-written.\n\nWEAKNESSES:\nIn Assumption 1: \n- Item (iv): why is it a measure-zero event?\n- Items (iv) and (v): I think that the assumptions should specify properties of the training data and the training algorithm. Then, the properties of the trajectory should be shown using these assumptions. Can you specify items (iv) and (v) as assumptions on the data+algorithm?\n\nOther than that, I don’t have major concerns. An obvious limitation is the strong assumptions, and specifically the assumption on the correlated training data, but as I already mentioned, I think that it is reasonable here.\n\nQUESTIONS:\nSee the “weaknesses” section.\n\nLIMITATIONS:\nThe authors discussed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problem of learning a single ReLU using a 2-layer ReLU network using gradient flow on both layers. The main assumption is that the data is correlated with the target neuron, while other milder assumptions are also used (e.g. specific initialization and spectral assumption on the data matrix). The main result is a two-phase convergence to a global minimum. Several experiments are also given.\n\nSTRENGTHS:\n- The main convergence result is novel AFAIK, and shows an interesting convergence dynamic, where in the first phase the learned weights either align with the target neuron or deactivate, and in the second phase converge to a global minimum. \n- The connection between the global minima of the problem and the minimal norm solution in Section 7 is interesting and brings forward the question of whether minimizing the empirical loss results in a minimal norm solution, which was studied in previous works too (e.g. Vardi et al 2022).\n- The experimental part shows empirically the behavior of the angles between the learned weights and the target neuron.\n\nWEAKNESSES:\n- The assumption that the data is correlated with the target neuron is pretty strong. The motivation for taking an angle of at most \\pi/4 between each data point and the target neuron is also not clear. What changes if the angle is larger\\smaller? I think the authors should elaborate more on this assumptions, and what breaks if it is not assumed. To compare, other papers about studying a single neuron usually consider a data distribution spread in all directions (e.g. Frei et al. 2020, Yehudai & Shamir 2020).\n- The presentation of the main result is not clear. I think there should be a single Theorem stating the convergence result, with an explicit convergence rate. Currently, there is no single result just a lemma for each phase, and it is difficult to parse the main out-take of the paper. In such a paper, I think the convergence rate of the entire procedure is crucial to fully understand the quality of the result.\n- The paper is very technical, and in my opinion, doesn’t provide enough intuition to understand the quantities that are used. For example, the definition of \\delta in line 238. Can’t \\delta be zero or at least exponentially small (if the angle between two data points is very close to \\pi/2).\n- The result in Section 7 is interesting but not quite clear. What does the quantity M represents? What can we say about the dataset itself so that either option (i) or (ii) of Theorem 8 is applied?\n- I think that claiming that the implicit bias for the problem studied here is to minimize the norm is a bit misleading. As I understand it, all the learned weights either align with the target neuron or deactivate, this means that the solution converges to a specific form of rank-1 matrix (where each row is either v^* or 0).\n\nQUESTIONS:\n- What changes if we assume that the angle between the data samples and the target neurons is \\alpha, where \\alpha > \\pi/4?\n- What is the total convergence rate of gradient flow for learning a single neuron?\n- Is it possible to extend the result to gradient flow? or possibly SGD?\n- Is there an explicit condition on the dataset that can be given so that the set of interpolators also minimizes the norm of the predictor\n\nLIMITATIONS:\nThe authors do address adequately to the limitations of the paper, although I think it is important to elaborate more on the main assumption about the dataset.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies how a two-layer ReLU network can fit a single neuron. The authors consider the case where all \ntraining points are correlated with the teacher neuron and show that gradient flow from small initialization can \nconverge to a zero-loss solution. \nThey divide the training into two stages. In the first stage, the neurons are small and will align with the teacher \nneuron or deactivate, depending on the sign of the output weight, and in the second stage, the aligned neurons will grow \nand fit the target function. \nIn addition, they show that as the initialization scale goes to $0$, gradient flow converges to a rank-$1$ solution.\n\nSTRENGTHS:\n* Overall, the presentation is clear, and the main text is relatively easy to follow (see the weakness part of the \n  review for some minor issues.) \n* The use of the yardsticks $\\omega_j$ is interesting.\n* The geometric argument for the second phase seems novel and may be of independent interest. \n* Theorem 8, which shows the set of balanced rank-1 interpolating networks and the set of minimum-norm interpolating \n  networks can be the same or disjoint, depending on a certain quantity, is surprising.\n\nWEAKNESSES:\n* The presentation is overall clear, but the notations are cumbersome. \n  * For example, I don't think using $w$ and $\\omega$ simultaneously is a good idea, especially when they co-occur a lot \n    and represent two closely related objects. \n  * The detailed definitions of $\\delta$ and $S_l$ can be moved into the appendix, and define them informally in the \n    main text and maybe briefly explain how small can $\\delta$ be and the intuition behind $S_l$. \n* I personally don't like the $\\exp(-n)$ initialization scale, though I will accept it as it has been used in previous \n  works. I think it is somewhat cheating because, with it, you can make sure the norm of the neurons is sufficiently small \n  so that you can ignore them for any polynomially long time. \n* The setting is quite restricted and unrealistic as it requires the angle of all inputs and the teacher vector to be \n  smaller than $\\pi / 4$.\n\nQUESTIONS:\n* Is it possible to adapt your strategy to the Gaussian inputs case? This seems to be a more natural generalization to \n  the orthogonal input setting. \n* Could you intuitively explain the meaning of the conditions in the definition of $S_l$?\n\nLIMITATIONS:\nThis is a theoretical work and I cannot see any potential negative societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nConvergence and implicit bias of non-linear networks is an important open question in deep learning theory.  The paper studies these questions in the case of regression with ReLU networks with a single teacher neuron. It proves that at a vanishing initialization scale the student neurons align with the teacher (or gets deactivated). It also shows an interesting counter example such that the implicit bias as initialization tends to zero need not be a minimum norm interpolator.\n\nSTRENGTHS:\na) Going beyond the orthogonal data, the paper proposes an interesting setting which helps analyse the case of correlated inputs. \n\nb) The geometric technique to study the convergence after the alignment phase is novel. \n\nc) The scenario proposed where the implicit bias as $\\lambda \\to 0$ is a rank minimizing one instead of a minimum norm interpolator is a very interesting contribution.\n\nWEAKNESSES:\na) The setting is simplified: there is only a single teacher neuron and all the labels are only positive. It directly that neurons (at least yardstick neurons) with negative last layer decreases in norm. The assumption that the inputs are correlated further ensures that they are deactivated. This makes the analysis easier.  \n\nb) The technical analysis follows the same strategy as Boursier et. al. Some aspects are easier as there is only one saddle to escape.\n\nQUESTIONS:\na) It is mentioned in the contributions there exists a case where that the minimum norm interpolator is rank 2 instead of rank 1. Can the authors comment how the non-linearity plays a role here? Can the authors a simple example providing a more detailed illustration?\n\nLIMITATIONS:\na) The experiments are conducted with synthetic data and only in the case of single teacher neuron. It would interesting some empirical evidence in the case of more than one teacher if the phenomenon characterized in the paper holds in more generality.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}