Title: Rethinking Gauss-Newton for learning over-parameterized models
Venue: NeurIPS 2023
Accepted: yes
Keywords: implicit bias

gauss newton
Abstract: This work studies the global convergence and implicit bias of Gauss Newton's (GN) when optimizing over-parameterized one-hidden layer networks in the mean-field regime. 
We first establish a global convergence result for GN in the continuous-time limit exhibiting a faster convergence rate compared to GD due to improved conditioning. 
We then perform an empirical study on a synthetic regression task to investigate the implicit bias of GN's method.
While GN is consistently faster than GD in finding a global optimum, the learned model generalizes well on test data when starting from random initial weights with a small variance and using a small step size to slow down convergence. 
Specifically, our study shows that such a setting results in a hidden learning phenomenon, where the dynamics are able to recover features with good generalization properties despite the model having sub-optimal training and test performances due to an under-optimized linear layer. This study exhibits a trade-off between the convergence speed of GN and the generalization ability of the learned solution.

Rethinking Gauss-Newton for learning over-parameterized models

Abstract

This work studies the global convergence and implicit bias of Gauss Newton's (GN) when optimizing over-parameterized one-hidden layer networks in the mean-field regime. We first establish a global convergence result for GN in the continuous-time limit exhibiting a faster convergence rate compared to GD due to improved conditioning. We then perform an empirical study on a synthetic regression task to investigate the implicit bias of GN's method. While GN is consistently faster than GD in finding a global optimum, the learned model generalizes well on test data when starting from random initial weights with a small variance and using a small step size to slow down convergence. Specifically, our study shows that such a setting results in a hidden learning phenomenon, where the dynamics are able to recover features with good generalization properties despite the model having sub-optimal training and test performances due to an under-optimized linear layer. This study exhibits a trade-off between the convergence speed of GN and the generalization ability of the learned solution.

1 Introduction

Gauss-Newton (GN) and related methods, like the Natural Gradient (NG) can offer faster convergence rates compared to gradient descent due to the improved dependence on the conditioning of the problem [9, 21, 30, 31]. For these reasons, these methods have attracted attention in machine learning as an alternative to gradient descent when optimizing ill-conditioned problems arising from the use of vastly over-parameterized networks and large training sets [29, 45]. Unfortunately, GN's high computational cost per iteration, which involves solving an expensive linear system, restricts its applicability in large-scale deep learning optimization. Addressing this challenge has been a primary focus, with extensive efforts dedicated to crafting computationally efficient approximations for GN/NG methods where the aim is to strike a delicate balance between computational cost and optimization speed, thereby enhancing scalability [4, 9, 11, 21, 31, 42].

While substantial effort has been put into making GN scalable, little is known about the global convergence of these methods and their generalization properties when optimizing neural networks, notably compared to gradient descent. Understanding these properties is particularly relevant knowing that state-of-the-art machine learning methods rely on over-parameterized networks for which infinitely many solutions interpolating the data exist albeit exhibit varying generalization properties that depend on implicit bias of the optimization procedure [7, 39]. To date, the majority of studies on generalization in over-parameterized networks have primarily focused on analyzing the dynamics of gradient descent and gradient flows - their continuous-time limit -, making important progress in understanding the implicit bias of these optimization procedures. Specifically, [15]

showed that a gradient flow can run under two regimes which yield solutions with qualitatively different generalization properties: the kernel regime , where the internal features of a network remain essentially constant during optimization, often yields poor generalization, and the feature learning regime , which allows the internal features of a network to adapt to the data, seems behind the impressive generalization properties observed in practice. In addition, the same work precisely defines conditions for these regimes: The kernel regime systematically occurs, for instance, when taking the Neural Tangent limit (NTK), i.e. scaling the output of a network of width M by a factor 1 / √ M , while the feature learning regime can arise when instead taking the mean-field limit , i.e. scaling the output by a factor of 1 /M . However, these insights rest upon attaining a global solution, which is far from obvious given the non-convexity of the training objective. While global linear convergence of gradient flows is known in the kernel regime [1, 23], obtaining such convergence guarantees in the feature learning regime remains an active research area [8, 10, 12, 14]. Importantly, all these works have focused solely on gradient flows and gradient descent, given their prevalent usage when optimizing neural networks.

In contrast, GN/NG methods have received limited attention in the study of both generalization and global convergence for over-parameterized networks. The added complexity of these methods makes their analysis intricate, particularly in the stochastic setting where inaccurate estimates of the pre-conditioning matrix can alter the dynamics in a highly non-trivial manner [32, Section 12.2]. Recent works studied the convergence and generalization of GN in the kernel regime [11, 20, 26, 52], while others focused on linear networks [27]. However, the behavior of these methods for nonlinear over-parameterized models remains understudied in the feature learning regime which is a regime of interest as it is more likely to result in good generalization.

Contributions. The present work aims to deepen our understanding of the properties of GaussNewton (GN) methods for over-parameterized networks, building upon recent advancements in neural network optimization [15]. Unlike prior works that primarily addressed scalability issues of GN/NG or studied the impact of stochastic noise on optimization dynamics, we uniquely focus on studying the convergence and implicit bias of an exact GN method . Specifically, our research investigates GN in the context of a regression problem using over-parameterized one-hidden layer networks in a deterministic ( full-batch ) setting. By examining an exact, deterministic GN method, rather than an approximate or stochastic version, we eliminate potential regularization effects stemming from approximation and stochasticity that might influence convergence and generalization. This strategy, in turn, enables us to provide both theoretical and empirical insights, shedding light on the properties of GN methods when optimizing neural networks:

Weprovide a global convergence result in Proposition 3 with a linear rate for the continuous-time limit of GN's dynamics in the over-parameterized regime that holds under the mean-field limit. The achieved rate shows better conditioning than the gradient flow with the same objective, emphasizing the method's advantage from a pure optimization perspective. While similar rates can be achieved under strong non-degeneracy conditions as in [50], these conditions are never met over-parameterized networks. Instead, our result relies on a simple condition on the initial parameters of a similar nature as in [13, Thm 3.3]: (i) that the hidden weights are diverse enough so that an interpolating solution can be obtained simply by fitting the linear weights, and (ii) that the linear weights result from a simple fine-tuning procedure to reach near optimality. The result shows that GN can reach the data interpolation regime, often required for studying the implicit bias of optimization (Section 4).

The empirical study in Section 5 complements the convergence result in Proposition 3 by further investigating the implicit bias of GN on a student/teacher regression task using synthetic data. We show that, while GN attains an interpolating solution, as predicted by Proposition 3, such a solution generalizes more or less well depending on the regime under which the GN dynamics is running. Specifically, we show that the GN dynamics can exhibit both kernel and feature learning regimes depending on the choice of step size and variance of the weights at initialization. Our experiments also indicate that a hidden learning phenomenon occurs in GN dynamics where hidden features that generalize well are obtained even when both raw training and test performances are sub-optimal due to an under-optimized linear layer. Quite surprisingly, these features are found when using small step sizes and, sometimes, generalize better than those learned by gradient descent. This is contrary to the common prescription of using larger step sizes in gradient descent for better generalization [3]. Our results suggest a tradeoff between optimization speed and generalization when using GN which is of practical interest.

2 Related work

Convergence of Gauss-Newton's methods. Alarge body of work on inverse problems studied the local convergence of Gauss-Newton and its regularized versions such as the Levenberg-Marquardt method [25, 38]. Many such works focus on a continuous-time analysis of the methods as it allows for a simplified study [24, 41]. In this work, we also consider continuous-time dynamics and leave the study of discrete-time algorithms to future work. More recently, increasing attention was given to global convergence analysis. [50] studied the global convergence of GN with an inexact oracle and in a stochastic setting establishing linear convergence. The analysis requires that the smallest singular value of the Jacobian is positive near initialization, a situation that can never occur in the over-parameterized regime. [34] studied the convergence of the Levenberg-Marquardt dynamics, a regularized version of Gauss-Newton's method, in a non-convex setting under a cubic growth assumption and showed global albeit sub-linear convergence rates. It is unclear, however, if such cubic growth assumption holds in our setting of interest. Closest to our work is [11] which aims to accelerate optimization by solving a kernel regression in the NTK scaling limit. There, the authors show global convergence of a regularized version of GN in the kernel regime . We aim to investigate the behavior of GD in the feature learning regimes.

The implicit bias of gradient flows. In the over-parameterized setting, the training objective often possesses several global minima that perfectly interpolate the training data. The generalization error thus heavily depends on the solution selected by the optimization procedure. [39] highlighted the importance of understanding the properties of the selected solutions for a given initialization and several works studied this problem in the case of linear neural networks [6, 33, 40, 49, 51]. For nonlinear networks, there is still no general characterization of the implicit bias in the regression setting, although [10] recently made important progress in the case of a one-hidden layer network with ReLU activation and orthogonal inputs showing that gradient flows select a minimum norm interpolating solution when the initial weights are close to zero. Recently, the implicit bias of gradient flows was shown to play a crucial role in other optimization problems such as non-convex bi-level optimization [5, 47], therefore illustrating the ubiquity of such phenomenon in deep learning. The present work empirically studies the implicit bias of Gauss-Newton which is vastly understudied.

3 Setup and preliminaries

3.1 Regression using over-parameterized one-hidden layer networks

̸

Weconsider a regression problem where the goal is to approximate an unknown real-valued function f ⋆ defined over a subset X of R d from a pair of i.i.d. inputs/outputs ( x n , y n ) 1 ≤ n ≤ N where each x n is a sample from some unknown distribution P and y n = f ⋆ ( x n ) . We assume that f ⋆ belongs to L 2 ( P ) , the set of square-integrable functions w.r.t. P and note that f ⋆ always belongs to L 2 ( ˆ P ) , where ˆ P denotes the empirical distribution defined by the samples ( x 1 , . . . , x N ) . For simplicity, we assume the data are non-degenerate, meaning that x n = x n ′ whenever n = n ′ . In this case L 2 ( ˆ P ) is isomorphic to R N (i.e., L 2 ( ˆ P ) ∼ = R N ) and we can identify any function f ∈ L 2 ( ˆ P ) with the evaluation vector ( f ( x 1 ) , . . . , f ( x N )) . We are interested in approximating f ⋆ using a one-hidden layer network f w with parameter vector w belonging to some, possibly infinite, Hilbert space W . The network's parameters are learned by minimizing an objective of the form L ( f w ) where L is a non-negative, L -smooth, and µ -strongly convex function defined over L 2 ( ˆ P ) and achieving a 0 minimum value at f ⋆ . A typical example that we consider in Section 5 is the mean-squared error over the training set:

̸

For the convergence results, we will consider both finite-width one-hidden layer networks and their mean-field infinite-width limit . In the experiments, we will restrict to finite-width networks although, we will be using a relatively large number of units M compared to the size of the training data, therefore approximating the mean-field limit.

Finite-width one-hidden layer networks. Given a non-polynomial point-wise activation function γ and some positive integer M , these networks take the form:

where v ∈ R M is the linear weight vector while u ∈ R M × d is the hidden weight matrix. Popular choices for γ include ReLU [37] and its smooth approximation SiLU ( x ) = x (1 + e -βx ) -1 which can be made arbitrary close to ReLU by increasing β [19]. The above formulation can also account for a bias term provided the input vector x is augmented with a non-zero constant component.

Mean-field infinite-width limit. We consider some base probability µ measure with full support on X and finite second moment and denote by L 2 ( µ ) and L 2 ( µ, R d ) the set of square integrable functions w.r.t. µ and taking values in R and R d . Given a non-polynomial point-wise non-linearity, we define infinitely-wide one-hidden layer networks f w as:

Functions of the form (3) correspond to the mean-field limit of the network defined in (10) when the number of units M grows to infinity and appear in the Lagrangian formulation of the Wasserstein gradient flow of infinitely wide networks [48]. Existing global convergence results of GN methods were obtained in the context of the NTK limit [11] which does not allow feature learning . The meanfield limit we consider here can result in optimization dynamics that exhibit feature learning which is why we consider it in our global convergence result analysis. For completeness, we provide a brief discussion on mean-field vs NTK limits and kernel vs feature learning regimes in Appendix A.

Over-parameterization. We consider an over-parameterized setting where the network has enough parameters to be able to fit the training data exactly, thus achieving a 0 training error. To this end, we define the gram matrix G which is an N × N matrix taking the following forms depending on whether we are using a finite-width network ( G F ) or infinite-width network ( G I ) :

We say that a network is over-parameterized if G ( u 0 ) is invertible for some hidden-layer weight u 0 :

( A ) (Over-parameterization) G ( u 0 ) is invertible for an initial parameter w 0 =( v 0 , u 0 ) ∈W .

̸

When u 0 is sampled randomly from a Gaussian, a common choice to initialize a neural network, Assumption ( A ) holds for infinitely wide networks as long as the training data are non-degenerate (see [18, Theorem 3.1] for ReLU , and [17, Lemma F.1] for analytic non-polynomial activations). There, the non-degeneracy assumption simply means that the inputs are not parallel to each other (i.e. x i ∦ x j whenever i = j for any 1 ≤ i, j ≤ N ), a property that holds almost surely if the data distribution P has a density w.r.t. Lebesgue measure. In the case of finite-width networks with M > N , the result still holds with a high probability when u is sampled from a Gaussian [28].

3.2 Generalized Gauss-Newton dynamics

To solve (1), we consider optimization dynamics based on a generalized Gauss-Newton method. To this end, we denote by J w the Jacobian of ( f w ( x 1 ) , ..., f w ( x N )) w.r.t. parameter w which can be viewed as a linear operator from W to R N . Moreover, ∇L ( f w ) denotes the vector of size N representing the gradient of L w.r.t. the outputs ( f w ( x 1 ) , ..., f w ( x N )) . We can then introduce the Gauss-Newton vector field Φ : W → W defined as:

where H w is a symmetric positive operator on L 2 ( ˆ P ) and ε ( w ) is an optional non-negative (possibly vanishing) damping parameter. Starting from some initial condition w 0 ∈ W and for a given positive step-size γ , the Gauss-Newton updates and their continuous-time limit are given by:

The continuous-time limit is obtained by taking the step-size γ to 0 and rescaling time appropriately. When H w is given by the Hessian of L , the dynamics in (4) recovers the generalized GaussNewton dynamics in continuous time when no damping is used [44] and recovers the continuoustime Levenberg-Maquardt dynamics when a positive damping term is added [35]. When the matrix H w is the identity H w = I , the resulting dynamics is tightly related to the natural gradient [2, 30]. More generally, we only require the following assumption on H w :

( B ) H w is continuously differentiable in w with eigenvalues in [ µ H , L H ] for positive L H , µ H .

Optional damping. Amongst possible choices for the damping, we focus on ε ( w ) of the form:

where α is a non-negative number α ≥ 0 , and σ 2 ( w ) is the smallest eigenvalue of the Neural Tangent Kernel (NTK) A w := J w J ⊤ w which is invertible whenever J w is surjective. This choice allows us to study the impact of scalar damping on the convergence rate of the GN dynamics. While computing σ 2 ( w ) is challenging in practice, we emphasize that the damping is only optional and that the case when no damping is used, i.e. when α = 0 , is covered by our analysis.

4 Convergence analysis

4.1 Global convergence under no-blow up

Westart by showing that the continuous-time dynamics in (4) converge to a global optimum provided that it remains defined at all times.

Proposition 1. Let w 0 = ( v 0 , u 0 ) ∈ W be an initial condition so that u 0 satisfies Assumption ( A ). Under Assumption ( B ) and assuming that the activation γ is twice-continuously differentiable, there exists a unique solution to the continuous-time dynamics in (4) defined up to some maximal, possibly infinite, positive time T . If T is finite, we say that the dynamics blows-up in finite time. If, instead T is infinite, then f w t converges to a global minimizer of L . Moreover, denoting by µ the strong convexity constant of L and defining µ GN := 2 µ/ ( L H (1 + α/µ H )) , it holds that:

Proposition 1 is proved in Appendix B.2. It relies on the Cauchy-Lipshitz theorem to show that the dynamics are well-defined up to a positive time T and uses strong convexity of L to obtain the rate in (6). The result requires a smooth non-linearity, which excludes ReLU . However, it holds for any smooth approximation to ReLU , such as SiLU . As we discuss in Section 5, this does not make a large difference in practice as the approximation becomes tighter. The rate in (6) is only useful when the dynamics is defined at all times which is far from obvious as the vector field Φ can diverge in a finite time causing the dynamics to explode. Thus, the main challenge is to find conditions ensuring the vector field Φ remains well-behaved which is the object of Section 4.2.

Comparaison with gradient flow. The rate in (6) only depends on the strong-convexity constant µ of L , the smoothness constant of H , and damping strength α , with the fastest rate achieved when no damping is used, i.e. α = 0 . For instance, when choosing H to the identity and α = 0 , the linear rate becomes µ GN = 2 µ . This is in contrast with a gradient flow of w ↦→L ( f w ) for which a linear convergence result, in the kernel regime, follows from [15, Theorem 2.4]:

with µ GF := µσ ( w 0 ) / 4 where σ ( w 0 ) is the smallest singular value of the initial NTK matrix A w 0 . In practice, µ GF ≪ µ GN since the linear rate µ GF for a gradient flow is proportional to σ ( w 0 ) , which can get arbitrarily small as the training sample size N increases thus drastically slowing down convergence [43].

4.2 Absence of blow-up for almost-optimal initial linear layer

In this section, we provide a condition on the initialization w 0 = ( v 0 , u 0 ) ∈ W ensuring the GaussNewton dynamics never blows up. We start with a simple result, with a proof in Appendix B.1, showing the existence of a vector v ⋆ so that w ⋆ := ( v ⋆ , u 0 ) ∈ W interpolates the training data whenever the over-parameterization assumption Assumption ( A ) holds for a given u 0 .

Proposition 2. Let w 0 = ( v 0 , u 0 ) ∈ W be an initial condition so that u 0 satisfies Assumption ( A ). Define Γ( u ) := ( γ ( u ⊤ x n ) ) 1 ≤ n ≤ N and F ⋆ := ( f ⋆ ( x 1 ) , ..., f ⋆ ( x N )) and let v ⋆ be given by:

Then the vector w ⋆ = ( v ⋆ , u 0 ) ∈ W is a global minimizer of w ↦→ L ( f w ) and a fixed point of the Gauss-Newton dynamics, i.e. Φ( w ⋆ ) = 0 .

The above result ensures a global solution can always be found by optimizing the linear weights while keeping the hidden weights fixed. In practice, this can be achieved using any standard optimization algorithm such as gradient descent since the function v ↦→ l ( v ) = L ( f ( v,u 0 ) ) converges to a global optimum as it satisfies a Polyak-Lojasiewicz inequality (see Proposition 4 in Appendix B.1). However, this procedure is not of interest here as it does not allow us to learn the hidden features. Nonetheless, since w ⋆ = ( v ⋆ , u 0 ) is a fixed point of the Gauss-Newton dynamics, we can expect that an initial condition w 0 = ( v 0 , u 0 ) , where v 0 is close to v ⋆ , results in a well-behaved dynamics. The following proposition, proven in Appendix B.4, makes the above intuition more precise.

Proposition 3. Let w 0 = ( v 0 , u 0 ) ∈ W be an initial condition so that Assumption ( A ) holds, i.e. the gram matrix G ( u 0 ) is invertible, and denote by σ 2 0 the smallest eigenvalue of G ( u 0 ) . Assume that the activation γ is twice-continuously differentiable and that Assumption ( B ) holds. Let R be any arbitrary positive number and define C R = sup w ∈B ( w 0 ,R ) ∥ ∂ w J w ∥ op where ∥ . ∥ op denotes the operator norm. If the linear layer is almost optimal v 0 in the sense that:

then (4) is defined at all times, i.e. T = + ∞ , the objective L ( f w t ) converges at a linear rate to 0 according to (6) and the parameters w t remain within a ball of radius R centered around w 0 .

Proposition 3 essentially states that the dynamics never blow up, and hence, converge globally at the linear rate in (6) by Proposition 1, provided the hidden features are diverse enough and the initial linear weights optimize the objective well enough. As discussed in Section 3.1, the first condition on the hidden weights u 0 typically holds for a Gaussian initialization when the data are non-degenerate. Additionally, the near-optimality condition on the linear weights v 0 can always be guaranteed by optimizing the linear weights while keeping the hidden ones fixed.

Remark 1. In the proof of Proposition 3, we show that the occurrence of a finite-time blow-up is tightly related to the Neural Tangent Kernel matrix A w t becoming singular at t increases which causes the vector field Φ to diverge. When the NTK matrix A w t at time t is forced to remain close to the initial one A w 0 , as is the case in the NTK limit considered in [11], one can expect that the singular values of A w t remain bounded away from 0 since this is already true for A w 0 . Therefore, the main technical challenge is the study, performed in Propositions 7 to 9 of Appendix B.3, of the time evolution of the eigenvalues of A w t in the mean-field limit , where A w t is allowed to differ significantly from the initial NTK matrix A w 0 . The final step is to deduce that, while the weights w t are allowed to be in a ball centered in w 0 of arbitrarily large radius R , the singular values remain bounded away from 0 provided the initial linear weights v 0 satisfy the condition in (7).

Remark 2. Comparing Proposition 3 to the convergence results for gradient flows in the kernel regime [15, Theorem 2.4], our result also holds for an infinite dimensional space of parameters such as in the mean-field limit of one-hidden layer network. Moreover, while [15, Theorem 2.4] requires the norm w t to be within a ball of fixed radius R 0 determined by initial smallest singular value σ ( w 0 ) of A w 0 , our result allows w t to be within arbitrary distance R regardless of the initial σ ( w 0 ) provided the condition in (7) holds. Proposition 3 is more similar in flavor to the convergence result of the Wasserstein gradient flow for some probability functionals in [13, Theorem 3.3, Corollary 3.4] which requires the initial objective to be smaller than a given threshold.

While the results of this section guarantee global convergence of the training objective, they do not guarantee learning features that generalize well. Next, we empirically show that GN can exhibit regimes where it yields solutions with good generalization properties.

5 Empirical study of generalization for a Gauss-Newton algorithm

Weperform a comparative study between gradient descent (GD) and Gauss-Newton (GN) method in the context of over-parameterized networks. Since we observed little variability when changing the

seed, we have chosen to use a single seed for the sake of clarity in the figures when presenting the results of this section, while deferring the analysis for multiple seeds to Appendix C.1. Additional experiments using MNIST dataset [16] are provided in Appendix C.4. The results presented below were obtained by running 720 independent runs, each of which optimizes a network given a specific configuration on a GPU. The total time for all runs was 3600 GPU hours.

5.1 Experimental setup

Weconsider a regression task on a synthetic dataset consisting of N training points ( X n , Y n ) 1 ≤ n ≤ N . The objective is to minimize the mean-squared error L ( f w ) , defined in (1), over the parameters w of a model f w that predicts the target values Y n based on their corresponding input points X n .

Data generation. We generate N i.i.d. samples denoted as ( X n ) 1 ≤ n ≤ N from a standard Gaussian distribution of dimension d = 10 . Each corresponding target Y n is obtained by applying a predefined function f ⋆ , referred to as the teacher network , to the input X n (i.e., Y n = f ⋆ ( X n ) ). We choose f ⋆ to be a one-hidden layer network of the form in (10) with M ⋆ = 5 hidden units ( v ⋆ i , u ⋆ i ) 1 ≤ i ≤ M ⋆ drawn independently from a standard Gaussian distribution. Furthermore, we consider two nonlinearities γ when defining f ⋆ : the widely used ReLU activation [37] for the main results and its smooth approximation SiLU for additional ablations in Appendix C.3. This choice of target function f ⋆ results in a hidden low-dimensional structure in the regression task as f ⋆ depends only on 5 -dimensional linear projection of the input data. In most cases, we take the size of the training data to be N = 500 except when studying the effect of the training size. This choice allows us to achieve a balance between conducting an extensive hyper-parameter search and being able to compute precise GNupdates on the complete training data within a reasonable timeframe. Finally, we generate 10000 test samples to measure the generalization error.

Model. We consider a well-specified setting where the model w ↦→ f w , referred to as the student network , is also a one-hidden layer with the same activation function γ as the teacher network f ⋆ . Importantly, the student network possesses a number M = 5000 of hidden units, denoted as w := ( v i , u i ) 1 ≤ i ≤ M , which is much larger than the teacher's number of units M ⋆ = 5 and allows fitting the training data perfectly (over-parameterized regime). Following [15], we also normalize the output of the student f w by M to remain consistent with the mean-field limit as M increases.

Initialization. We initialize the student's hidden units according to a centered Gaussian with standard deviation (std) τ 0 ranging from 10 -3 to 10 3 . Varying the initial std τ 0 allows us to study the optimization dynamics in two regimes: the kernel regime (large values of τ 0 ) and the feature learning regime (small values of τ 0 ). Finally, we initialize the weights of the last layer to be 0 . This choice allows us to perform a systematic comparison with the minimum norm least squares solution obtained using random features as described next.

Methods and baselines. We consider three optimization methods for the model's parameters: a regularized version of Gauss-Newton (GN), gradient descent (GD), and optimizing the linear layer parameters alone, which can be viewed as a random features (RF) model.

(GN): We use the discrete GN updates in (4) with a constant step-size λ and H w = I . Each update is obtained using Woodbury's matrix identity by writing Φ( w k )= J ⊤ w k z k with z k the solution of a linear system ( J w k J ⊤ w k + ϵ ( w k ) I ) z k = ∇L ( f w k ) of size N . Here, we use the damping defined in (5) with α =1 and ensure it never falls below ϵ 0 = 10 -7 to avoid numerical instabilities. In practice, we found that small values of ϵ 0 had little effect on the results (see Figure 4 (Right) of Appendix C.1).

(GD): The model's parameters w are learned using gradient descent with a constant step-size λ . (RF): Instead of optimizing the entire parameter vector w = ( v, u ) of the model f w , we selectively optimize the parameter v of the linear layer while keeping the weights u of the hidden layer constant. This procedure corresponds to computing a minimal-norm least squares solution v RF for a random features (RF) model, where the features are obtained from the output of the hidden layer at initialization. Specifically, the solution v RF is obtained as

where Γ( u ) = ( γ ( u ⊤ X n )) 1 ≤ n ≤ N ∈ R M × N is the feature vector computed over the training data, and Y is a vector of size N consisting of the target values Y n . The choice of the un-regularized solution as a baseline is supported by recent findings [22], demonstrating its good generalization in the over-parameterized regime.

Stopping criterion. For both (GD) and (GN), we perform as many iterations as needed so that the final training error is at least below 10 -5 . Additionally, we stop the algorithm whenever the training error drops below 10 -7 or when a maximum number of iterations of K GD = 10 6 iterations for (GD) and K GN = 10 5 iterations for (GN) is performed. For (RF) we solve the linear system exactly using a standard linear solver.

5.2 Performance metrics

In addition to the training and test losses, we introduce two complementary metrics to assess the quality of the hidden features Γ( u ) learned by the student model.

Weighted cosine distance (WCD). Measuring proximity between the student and teacher's hidden weights ( u 1 , ..., u M ) and ( u ⋆ 1 , ..., u ⋆ M ⋆ ) requires being able to find a correspondance between each student's parameter u i and a teacher's one u ⋆ j . When using a positively homogeneous non-linearity such as ReLU , only the alignment between vectors is relevant. However, since the student's model is vastly over-parameterized, there could be vanishing weights u i = 0 which do not influence on the network's output, while the remaining weights are well aligned with the teacher's weights. We introduce the following weighted distance to account for these specificities:

Equation (9) finds a teacher's parameter u ⋆ j that is most aligned with a given student's parameter u i and downweights its cosine distance if u i has a small norm. In practice, we found this measure to be a good indicator for generalization.

Test loss after linear re-fitting (Test-LRfit). To evaluate the relevance of the hidden features Γ( u ) , we train a new linear model ˜ v on those features and measure its generalization error. In practice, we freeze the weights u of the hidden layer and fit the last layer v using the procedure (8) described above for the random features (RF) baseline. In our over-parameterized setting, this new model should always achieve perfect training loss, but its generalization error strongly depends on how the features were learned.

5.3 Numerical results

Implicit bias of initialization. Figure 1(Left) shows that GN and GD generalize more or less well compared to a random features model depending on the variance τ 0 of the weights at initialization. First, in the case of the RF model, changing τ 0 barely affects the final training and test error. This is essentially due to the choice of the non-linearity ReLU which is positively homogeneous. Second, for large τ 0 , the final test error of both GN and GD matches that of RF suggesting that the dynamics were running under the kernel regime/lazy regime [15, 23] where the final layer's weights are learned without changing the hidden features much. Finally, for small values of τ 0 , both GN and GD obtained a better test error than RF which can only be explained by learning suitable hidden features. These results are further confirmed when varying the size of the training set as shown in Appendix C.2. In Appendix C.3, we performed the same experiment using SiLU non-linearity instead of ReLU and observed the same transition between the kernel and feature learning regimes. While prior works such as [11] analyzed Gauss-Newton in the kernel learning regime, our results indicate that Gauss-Newton can also exhibit a feature learning regime for small values of τ 0 .

Implicit bias of the step size. Figure 1(Right) shows that increasing the step size in Gauss-Newton results in features that do not generalize well. This is unlike gradient descent where larger step sizes yield better-performing features 3 . This behavior is first observed in the top figures showing an increasing weighted cosine distance (WCD) between the student's hidden weights and the teacher's weights as the step size increases in the case of Gauss-Newton. On the contrary, the distance decreases with larger step sizes in the case of gradient descent, indicating that the algorithm learns better features. This effect is also confirmed when computing the test loss after refitting the linear layer to the training data exactly while keeping the learned hidden weights fixed (see linear re-fitting in Section 5.2). Several works, such as [3, 36], highlight the importance of taking large step-sizes

10

3

10

2

10

1

10 0

10 1

10 2

10

1

10 0

WCD

WCD/Loss vs std at initialization

10

3

10

2

10

1

10 0

10 1

10 2

Std of the initial weights

10

6

10

5

10

4

10

3

Loss

Test loss (Linear re-fitting)

Test loss

GN

RF

GN

RF

10

4

10

3

10

2

10

1

10 0

10 1

10 2

10 3

10

1

10 0

WCD

WCD/Loss vs step size (std=0.001)

10

4

10

3

10

2

10

1

10 0

10 1

10 2

10 3

Step size

10

8

10

6

10

4

10

2

Loss

Test loss (Linear re-fitting)

Test loss

Train loss

GN

RF

GN

RF

for gradient descent, to improve generalization, our results show that Gauss-Newton benefits instead of using smaller step-sizes. As shown in Appendix C.3, this behavior persists when using a different non-linearity, such as SiLU with different values for β ( 1 and 10 6 ). Interestingly, we observed that GNmight underperform GD as the parameter β defining SiLU decreases making SiLU nearly linear and less like ReLU . In all cases, while the final test loss remains large for small step sizes (in both GD and GN), the test loss after linear re-fitting is much lower in the case of Gauss-Newton, indicating that the algorithm was able to learn good features while the last layer remained under-optimized.

Hidden feature learning. Figure 2 (Left) shows the evolution of various metrics, including the WCD, the training, and test errors with the number of iterations for both GN and GD. For each method, results for the best step size and variance at initialization are presented ( ( τ 0 , λ ) = (10 -3 , 10 2 ) for GD and ( τ 0 , λ ) = (10 -3 , 10 -2 ) for GN). The bottom figure clearly indicates that the early iterations of GN essentially optimize internal features first while barely improving the last layer. This can be seen deduced from the training and test losses which remain almost constant at the beginning of optimization, while the test loss after linear re-fitting steadily decreases with the iterations. The last layer is learned only towards the end of training as indicated by a faster decrease in both train and test objectives. Gradient descent displays a different behavior with test loss before and after linear re-fitting remaining close to each other during optimization.

Better generalization of GN comes at the cost of a slower training. Figure 2 (Right) shows the evolution of the same quantities as in the left figures for τ 0 = 10 -3 as a function of time (in seconds) and for the three values of the step-size λ ( 10 -2 (bright colors) 1 (medium colors) and 10 2 (dark colors)). The figure shows that, while the training loss converges much faster for larger step sizes in the case of GN, it does not result in the best generalization error as measured by the test loss after linear re-fitting. Hence, better generalization comes at the cost of a slower convergence of the training loss. Again, the behavior is quite different in the case of gradient descent for which larger step sizes result in both faster optimization of the training objective and better generalization.

6 Conclusion

The results illustrate the importance of the initialization and choice of the step size for the generalization performance of GN when optimizing over-parameterized one-hidden layer networks. While

10 0

10 1

10 2

10 3

10 4

10 5

10 6

10 7

10

2

10

1

10 0

WCD

Loss/WCD vs Iteartions

10 0

10 1

10 2

10 3

10 4

10 5

10 6

10 7

Iterations

10

7

10

5

10

3

10

1

Loss

Train loss

Test loss

Test loss (Linear re-fitting)

GN

GN

10

1

10 1

10 3

10 5

10

2

10

1

10 0

WCD

Loss/WCD vs Time (s) for different

GN

GN

10

1

10 1

10 3

10 5

Time (s)

10

8

10

6

10

4

10

2

10 0

Loss

Train loss

Test loss (Linear re-fitting)

GN

GN

1

e

2

1

e

0

1

e

2

Step size

1

e

2

1

e

0

1

e

2

Step size

our theoretical analysis shows that GN can reach a global solution of the training objective when starting from a close-to-optimal initialization, this result does not imply anything about the generalization properties of the obtained solution or the quality of the learned features. Our empirical study instead shows that GN may favor feature learning (thus yielding improved generalization) at the cost of a slower optimization of the training objective due to the use of a smaller step size. Providing a theoretical analysis of these phenomena is an interesting direction for future work. Finally, our study shows that the training or test error may not always be a good indicator of the quality of the learned features due to an under-optimized final layer. Instead, a test/validation error after re-fitting the linear layer may be a better indicator for the quality of the learned features which can be of practical use.

Acknowledgments

The project leading to this work has received funding from the French National Research Agency (ANR-21-JSTM-0001) in the framework of the 'Investissements d'avenir' program (ANR-15IDEX-02). This work was granted access to the HPC resources of IDRIS under the allocation 2023-AD011013762R1 made by GENCI.

References

Z. Allen-Zhu, Y. Li, and Z. Song. 'A convergence theory for deep learning via overparameterization.' In: International Conference on Machine Learning . PMLR. 2019, pp. 242252.

S.-i. Amari. 'Natural Gradient Works Efficiently in Learning.' In: Neural Computation 10.2 (Feb. 1998), pp. 251-276. URL: https://www.mitpressjournals.org/doi/10.1162/ 089976698300017746 (visited on 08/22/2019).

M. Andriushchenko, A. Varre, L. Pillaud-Vivien, and N. Flammarion. 'SGD with large step sizes learns sparse features.' In: arXiv preprint arXiv:2210.05337 (2022).

M. Arbel, A. Gretton, W. Li, and G. Montufar. 'Kernelized Wasserstein Natural Gradient.' In: (Sept. 2019). URL: https://openreview.net/forum?id=Hklz71rYvS .

M. Arbel and J. Mairal. 'Non-Convex Bilevel Games with Critical Point Selection Maps.' In: Advances in Neural Information Processing Systems (NeurIPS) 2022 (2022).

S. Arora, S. Du, W. Hu, Z. Li, and R. Wang. 'Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks.' In: International Conference on Machine Learning . PMLR. 2019, pp. 322-332.

M. Belkin, A. Rakhlin, and A. B. Tsybakov. 'Does data interpolation contradict statistical optimality?' In: The 22nd International Conference on Artificial Intelligence and Statistics . PMLR. 2019, pp. 1611-1619.

G. Blanc, N. Gupta, G. Valiant, and P. Valiant. 'Implicit regularization for deep neural networks driven by an ornstein-uhlenbeck like process.' In: Conference on learning theory . PMLR. 2020, pp. 483-513.

A. Botev, H. Ritter, and D. Barber. 'Practical gauss-newton optimisation for deep learning.' In: International Conference on Machine Learning . PMLR. 2017, pp. 557-565.

E. Boursier, L. Pillaud-Vivien, and N. Flammarion. 'Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputs.' In: arXiv preprint arXiv:2206.00939 (2022).

T. Cai, R. Gao, J. Hou, S. Chen, D. Wang, D. He, Z. Zhang, and L. Wang. 'Gram-gaussnewton method: Learning overparameterized neural networks for regression problems.' In: arXiv preprint arXiv:1905.11675 (2019).

Z. Chen, E. Vanden-Eijnden, and J. Bruna. 'On feature learning in neural networks with global convergence guarantees.' In: arXiv preprint arXiv:2204.10782 (2022).

L. Chizat. 'Sparse Optimization on Measures with Over-parameterized Gradient Descent.' In: arXiv:1907.10300 [math, stat] (July 2019). arXiv: 1907.10300. URL: http://arxiv. org/abs/1907.10300 .

L. Chizat and F. Bach. 'On the Global Convergence of Gradient Descent for Overparameterized Models using Optimal Transport.' May 2018. URL: https : / / hal . archives-ouvertes.fr/hal-01798792 .

L. Chizat, E. Oyallon, and F. Bach. 'On lazy training in differentiable programming.' In: Advances in Neural Information Processing Systems 32 (2019).

L. Deng. 'The mnist database of handwritten digit images for machine learning research.' In: IEEE Signal Processing Magazine 29.6 (2012), pp. 141-142.

S. Du, J. Lee, H. Li, L. Wang, and X. Zhai. 'Gradient descent finds global minima of deep neural networks.' In: International conference on machine learning . PMLR. 2019, pp. 16751685.

S. S. Du, X. Zhai, B. Poczos, and A. Singh. 'Gradient descent provably optimizes overparameterized neural networks.' In: arXiv preprint arXiv:1810.02054 (2018).

S. Elfwing, E. Uchibe, and K. Doya. 'Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning. ArXiv e-prints (2017).' In: arXiv preprint arXiv:1702.03118 (2017).

J. R. Garcia, F. Freddi, S. Fotiadis, M. Li, S. Vakili, A. Bernacchia, and G. Hennequin. 'Fisher-Legendre (FishLeg) optimization of deep neural networks.' In: The Eleventh International Conference on Learning Representations . 2023.

R. Grosse and J. Martens. 'A Kronecker-factored Approximate Fisher Matrix for Convolution Layers.' In: Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48 . ICML'16. event-place: New York, NY, USA. JMLR.org, 2016, pp. 573-582. URL: http://dl.acm.org/citation.cfm?id=3045390.3045452 .

T. Hastie, A. Montanari, S. Rosset, and R. J. Tibshirani. 'Surprises in high-dimensional ridgeless least squares interpolation.' In: The Annals of Statistics 50.2 (2022), pp. 949-986.

A. Jacot, F. Gabriel, and C. Hongler. 'Neural Tangent Kernel: Convergence and Generalization in Neural Networks.' In: arXiv:1806.07572 [cs, math, stat] (June 2018). arXiv: 1806.07572. URL: http://arxiv.org/abs/1806.07572 .

B. Kaltenbacher, A. Neubauer, and A. Ramm. 'Convergence rates of the continuous regularized Gauss-Newton method.' In: Journal of Inverse and Ill-Posed Problems 10.3 (2002), pp. 261-280.

B. Kaltenbacher, A. Neubauer, and O. Scherzer. 'Iterative regularization methods for nonlinear ill-posed problems.' In: Iterative Regularization Methods for Nonlinear Ill-Posed Problems . de Gruyter, 2008.

R. Karakida and K. Osawa. 'Understanding approximate fisher information for fast convergence of natural gradient descent in wide neural networks.' In: Advances in neural information processing systems 33 (2020), pp. 10891-10901.

A. Kerekes, A. Mészáros, and F. Huszár. 'Depth Without the Magic: Inductive Bias of Natural Gradient Descent.' In: arXiv preprint arXiv:2111.11542 (2021).

C. Liu, L. Zhu, and M. Belkin. 'Loss landscapes and optimization in over-parameterized nonlinear systems and neural networks.' In: Applied and Computational Harmonic Analysis 59 (2022), pp. 85-116.

J. Martens. 'Deep Learning via Hessian-free Optimization.' In: Proceedings of the 27th International Conference on International Conference on Machine Learning . ICML'10. eventplace: Haifa, Israel. USA: Omnipress, 2010, pp. 735-742. URL: http://dl.acm.org/ citation.cfm?id=3104322.3104416 .

J. Martens. 'New insights and perspectives on the natural gradient method.' In: The Journal of Machine Learning Research 21.1 (2020), pp. 5776-5851.

J. Martens and R. Grosse. 'Optimizing Neural Networks with Kronecker-factored Approximate Curvature.' In: arXiv:1503.05671 [cs, stat] (Mar. 2015). arXiv: 1503.05671. URL: http://arxiv.org/abs/1503.05671 .

J. Martens and I. Sutskever. 'Training Deep and Recurrent Networks with Hessian-Free Optimization.' en. In: Neural Networks: Tricks of the Trade: Second Edition . Ed. by G. Montavon, G. B. Orr, and K.-R. Müller. Lecture Notes in Computer Science. Berlin, Heidelberg: Springer Berlin Heidelberg, 2012, pp. 479-535. URL: https://doi.org/10.1007/978-3-64235289-8_27 .

H. Min, S. Tarmoun, R. Vidal, and E. Mallada. 'On the explicit role of initialization on the convergence and implicit bias of overparametrized linear networks.' In: International Conference on Machine Learning . PMLR. 2021, pp. 7760-7768.

K. Mishchenko. 'Regularized Newton Method with Global O (1 /k ∗ ∗ 2) Convergence.' In: arXiv preprint arXiv:2112.02089 (2021).

J. J. Moré. 'The Levenberg-Marquardt algorithm: implementation and theory.' In: Numerical Analysis: Proceedings of the Biennial Conference Held at Dundee, June 28-July 1, 1977 . Springer. 2006, pp. 105-116.

R. Mulayoff, T. Michaeli, and D. Soudry. 'The implicit bias of minima stability: A view from function space.' In: Advances in Neural Information Processing Systems 34 (2021), pp. 17749-17761.

V. Nair and G. E. Hinton. 'Rectified linear units improve restricted boltzmann machines.' In: Proceedings of the 27th international conference on machine learning (ICML-10) . 2010, pp. 807-814.

Y. Nesterov. 'Modified Gauss-Newton scheme with worst case guarantees for global performance.' In: Optimisation methods and software 22.3 (2007), pp. 469-483.

B. Neyshabur, R. Tomioka, and N. Srebro. 'In search of the real inductive bias: On the role of implicit regularization in deep learning.' In: arXiv preprint arXiv:1412.6614 (2014).

S. Pesme, L. Pillaud-Vivien, and N. Flammarion. 'Implicit bias of sgd for diagonal linear networks: a provable benefit of stochasticity.' In: Advances in Neural Information Processing Systems 34 (2021), pp. 29218-29230.

A. G. Ramm. 'Dynamical systems method for solving operator equations.' In: Communications in Nonlinear Science and Numerical Simulation 9.4 (2004), pp. 383-402.

Y. Ren and D. Goldfarb. 'Efficient subsampled gauss-newton and natural gradient methods for training neural networks.' In: arXiv preprint arXiv:1906.02353 (2019).

D. A. R. Robin, K. Scaman, and M. Lelarge. 'Convergence beyond the over-parameterized regime using Rayleigh quotients.' In: Advances in Neural Information Processing Systems . Ed. by A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho. 2022. URL: https://openreview. net/forum?id=pl279jU4GOu .

N. N. Schraudolph. 'Fast curvature matrix-vector products for second-order gradient descent.' In: Neural computation 14.7 (2002), pp. 1723-1738.

I. Sutskever, J. Martens, G. Dahl, and G. Hinton. 'On the importance of initialization and momentum in deep learning.' en. In: International Conference on Machine Learning . Feb. 2013, pp. 1139-1147. URL: http://proceedings.mlr.press/v28/sutskever13.html .

A. B. Tsybakov. Introduction to Nonparametric Estimation . en. Springer Series in Statistics. New York: Springer-Verlag, 2009. URL: https://www.springer.com/gp/book/ 9780387790510 .

P. Vicol, J. Lorraine, D. Duvenaud, and R. Grosse. 'Implicit Regularization in Overparameterized Bilevel Optimization.' In: ICML 2021 Beyond First Order Methods Workshop . 2021.

S. Wojtowytsch. 'On the convergence of gradient descent training for two-layer relu-networks in the mean field regime.' In: arXiv preprint arXiv:2005.13530 (2020).

B. Woodworth, S. Gunasekar, J. D. Lee, E. Moroshko, P. Savarese, I. Golan, D. Soudry, and N. Srebro. 'Kernel and rich regimes in overparametrized models.' In: Conference on Learning Theory . PMLR. 2020, pp. 3635-3673.

N. Yudin and A. Gasnikov. 'Flexible modification of gauss-newton method and its stochastic extension.' In: arXiv preprint arXiv:2102.00810 (2021).

C. Yun, S. Krishnan, and H. Mobahi. 'A unifying view on implicit bias in training linear neural networks.' In: arXiv preprint arXiv:2010.02501 (2020).

G. Zhang, J. Martens, and R. B. Grosse. 'Fast convergence of natural gradient descent for over-parameterized neural networks.' In: Advances in Neural Information Processing Systems 32 (2019).

A Background: NTK vs mean-field limits, kernel vs feature learning regimes

We provide a brief discussion of the notions of NTK and mean-field limits as well as the related convergence regimes for a gradient flow. Given a non-polynomial point-wise and differentiable activation function γ and some positive integer M , consider the following one-hidden network:

where α ( M ) is a scaling factor that depends on the number of neurons M . For simplicity, we assume that the weights ( v i , u i ) are i.i.d. samples from a Gaussian. We are interested in the dynamics of a gradient flow when M increases to + ∞ and for specific choices of the scaling factor α ( M ) . First, let's introduce some notations. We denote by F ( w ) the vector of evaluation of the network on the training data F ( w ) = ( f w ( x 1 ) , . . . , f w ( x N )) and by J w the Jacobian of F ( w ) w.r.t. parameter w . Moreover, we introduce the NTK matrix A w := J w J ⊤ w which is an N × N symmetric positive semi-definite matrix. Finally, ∇L ( f w ) denotes the vector of size N representing the gradient of L w.r.t. F ( w ) . The gradient flow of an objective L ( F w ) is given by:

Additionally, it is often informative to track the time evolution of the vector F w t , which is given by the following ODE:

One can see from (11) that the NTK matrix A w t arises naturally when expressing the dynamics of the function F w t . We can even provide an explicit expression of A w t using the NTK kernel, which is a positive semi-definite kernel K w ( x, x ′ ) defined by the inner product:

The NTK matrix A w is then formed by collecting the values ( K w ( x i , x j )) 1 ≤ i,j ≤ N on the training data. It is easy to see that when α ( M ) = 1 √ M , then K w ( x, x ′ ) converges to an expectation under the distribution of the weights ( v i , u i ) 1 ≤ i ≤ M under mild conditions. In this case, A w depends on the choice of the distribution of the weights ( v i , u i ) 1 ≤ i ≤ M and not on the particular samples. On the other hand, when α ( M ) = 1 M , it is the function f w itself that converges to an expectation under the distribution of the weights. This distinction will lead to different qualitative dynamics for the gradient flows as we discuss next.

A.1 The Neural Tangent Kernel limit and the kernel regime ( α ( M ) = 1 √ M )

When choosing α ( M ) = 1 √ M and setting M → + ∞ , it is shown in [23] that the NTK matrix A w t remains constant in time and depends only on the distribution of the weights w 0 at initialization. Hence, F w t is effectively the solution of the ODE:

which converges at a linear rate to a global minimizer of L whenever A w 0 is invertible and L is strongly convex. The NTK limit results in a kernel regime, where the resulting solution is equivalent to the one obtained using a kernel method with the initial NTK kernel K w 0 ( x, x ′ ) [15, 23].

A.2 The mean-field limit and the feature learning regime ( α ( M ) = 1 M )

When choosing instead α ( M ) = 1 M , it is possible to view the limiting function as an expectation over weights drawn from some probability distribution ρ

In [48], it is further shown that such a formulation is essentially equivalent to the one considered in (3). As a result, the time evolution in (11) does not simplify as in the NTK limit, since the NTK matrix A w t (and NTK kernel K w t ) is allowed to evolve in time. This is a limit in which feature learning is possible since the dynamics do not necessarily simplify to a kernel method.

B Proofs

Notations. For an operator A between two Hilbert spaces, denote by ∥ A ∥ op its operator norm and by ∥ A ∥ F its Frobenius norm and let σ ⋆ ( A ) be its smallest singular value whenever it is well-defined. Denote by B ( w,R ) the open ball of radius R centered at an element w in a Hilbert space.

B.1 Over-parameterization implies interpolation.

Proof of Proposition 2. Denote by F ⋆ = ( f ⋆ ( x 1 ) , ..., f ⋆ ( x n )) . First, it is easy to see that the matrix ∂ v f w ∂ v f ⊤ w is exactly equal to the N × N matrix G ( u 0 ) considered in Assumption ( A ), which is assumed to be invertible. Moreover, ∂ v f w is independent of v since f ( v,u 0 ) is linear in v . Hence, we can simply choose:

Moreover, using again the linearity of f in v , we have that

We have therefore shown that w ⋆ = ( v ⋆ , u 0 ) is a global minimizer of w ↦→ L ( f w ) . Finally, since ∇L ( f w ⋆ ) = ∇L ( f ⋆ ) = 0 , it follows that ϕ ( w ⋆ ) = 0 .

Proposition 4. Assume that Assumption ( A ) holds for some vector u 0 . Then, the function v ↦→ l ( v ) = L ( f ( v,u 0 ) ) is convex and satisfies a Polyak-Lojasiewicz inequality:

with µ the strong convexity constant of L and σ 2 0 the smallest singular value of the matrix G ( u 0 ) appearing in Assumption ( A ).

Proof. It is easy to see that l ( v ) is convex as a composition of a strongly convex function L and a linear function v ↦→ f ( v,u 0 ) . We will show now that l satisfies a Polyak-Lojasiewicz (PL) inequality. To this end, we make the following calculations:

It is easy to see that the matrix ∂ v f w ∂ v f ⊤ w is exactly equal to the N × N matrix G ( u 0 ) considered in Assumption ( A ), which is assumed to be invertible. Hence, its smallest singular value σ 2 0 is positive and the following inequality holds:

On the other hand, since L is strongly convex and differentiable, then it satisfies the following PL inequality:

where µ is the strong convexity constant of L . Hence, we get the desired inequality by combining (13) and (14):

B.2 Global convergence under no blow-up: Proof of Proposition 1

In this section we prove the global convergence result under no blow-up stated in Proposition 1. We start by stating the smoothness assumption on the activation function γ as we will be using it in many places.

( C ) The non-linearity γ is twice-continuously differentiable.

As a first step, we need to prove a local existence and uniqueness of the solution to the ODE in (4) which results from Cauchy-Lipschitz theorem applied to the vector field Φ . To this end, a key step is to show that Φ is locally Lipschitz near initialization when the Jacobian J w 0 is surjective.

Proposition 5 (Regularity of Φ .) . Assume ( B ) and ( C ) and that w 0 satisfies Assumption ( A ). Then, there exists a neighborhood of w 0 so that Φ( w ) is locally Lipschitz in w .

Proof of Proposition 5. By application of the Woodbury matrix identity, Φ can be equivalently written as:

Moreover, by the smoothness assumption, J w , H w , ϵ ( w ) and ∇L ( f w ) are all differentiable in w . Additionally, since J w 0 is surjective, it must hold that σ 2 ( w ) > 0 in a neighborhood of w 0 so that J w J ⊤ w remains invertible. Finally, since H w is always positive, it holds that ( J w J ⊤ w + ε ( w ) H -1 w ) -1 and H -1 w are differentiable in a neighborhood of w 0 . Therefore, Φ is Locally Lipschitz in a neighborhood of w 0 .

The next proposition establishes the existence and uniqueness of the dynamics defined in (4).

Proposition 6 (Local existence) . Under the same conditions as Proposition 5, there exists a unique solution ( w t ) t ≥ 0 to (4) defined up to a, possibly infinite, maximal time T > 0 .

Proposition 6 is a direct consequence of the Cauchy-Lipschitz theorem which holds since the vector field Φ( w ) is locally Lipschitz in w in a neighborhood of w 0 by Proposition 5. The above local existence result does not exclude situations where the dynamics blow up in finite time T < + ∞ and possibly fail to globally minimize the objective L . We will show, later, that finite-time blow-ups never occur under additional assumptions on the initial condition.

Proof of Proposition 1. Existence and uniqueness of the continuous-time dynamics up to a maximal time T > 0 follows from Proposition 6. It remains to obtain the global convergence rate when T is infinite. For simplicity, we will ignore the dependence on w t in what follows and will write b := ∇L ( f w t ) . By differentiating the objective in time, we get the following expression:

Weused the Woodbury matrix inversion lemma to get the second line while the third and fourth lines use simple linear algebra. To get the fifth line, we used Lemma 3 to upper-bound the operator norm appearing in the fourth line. The last line follows by definition of ε ( w ) , the fact that H ≤ L H I and the fact that 1 2 ∥∇L ( f ) ∥ 2 ≥ µ L ( f ) by µ -strong convexity of L . This allows to directly conclude that L ( f w t ) ≤ L ( f w 0 ) e -µ GN t , with µ GN := 2 ηµL -1 H .

B.3 Control of the singular values of NTK matrix

In this section, we are interested in the evolution of the smallest singular value of the NTK matrix A w t := J w t J ⊤ w t . For any arbitrary positive radius R > 0 , define the constant C R and stopping time T R as follows:

̸

The stopping time T R characterizes the smallest-time when the dynamics becomes either degenerate ( σ ⋆ ( A w t ) = 0 ) or grows too far away from initialization ( w t = B ( w 0 , R ) ).

To control the singular values of A w t , we find it useful to consider the dynamics of the Frobenius norm of the pseudo-inverse J † w which we denote by a t := ∥ ∥ ∥ J † t ∥ ∥ ∥ F .

Indeed, simple linear algebra provides the following lower-bound on σ ⋆ ( A w t ) in terms of a t :

Note that, while the space of parameters can even be infinite-dimensional (mean-field limit), J w has a finite-dimensional range and thus always admits a pseudo-inverse J † w given by:

The next proposition provides a differential inequality for a t = ∥ ∥ J † w t ∥ ∥ F in terms of ∥ ˙ w t ∥ that holds for all times smaller than T R .

Proposition 7. Assume ( B ) and ( C ) and that w 0 satisfies Assumption ( A ), then for any R > 0 , the time T R is positive and smaller than the maximal time T , i.e. 0 < T R ≤ T . Moreover, for any t ∈ [0 , T R ) , a t satisfies:

Proof. Positive stopping time T R . By construction, T R must be smaller than T as it requires w t to be well-defined. Moreover, J w 0 is surjective by assumption and J w is continuous in w , therefore J w t must also be surjective for small enough positive times t > 0 . Additionally, it must hold that ∥ w t -w 0 ∥ ≤ R for small positive times, by continuity of t ↦→ w t . This allows shows that T R > 0 .

Dynamics of J † w t . By smoothness of J w it holds that J † w t is differentiable on the open interval (0 , T R ) and satisfies the following ODE obtained by direct differentiation:

In (17), we introduced notation J t := J w t and its time derivative ˙ J t for simplicity and denote by P t , the projector: P t := J ⊤ t ( J t J ⊤ t ) -1 J t .

Controlling | ˙ a t | . Taking the time derivative of a t and recalling the evolution (17) yields:

where we used that ( J † t ) ⊤ ( I -P t ) = 0 by Lemma 1 in Appendix B.5. Furthermore, taking the absolute value of the above equation, and recalling that the Frobenius norm is sub-multiplicative for the product of operators (Lemma 4 in Appendix B.5), we directly get:

By the chain rule, we have that ˙ J t = ∂ w J t ˙ w t therefore, ∥ ∥ ∥ ˙ J t ∥ ∥ ∥ F ≤ ∥ ∂ w J t ∥ op ∥ ˙ w t ∥ . Moreover, since t < T R , then it must holds that w t ∈ B ( w 0 , R ) so that ∥ ∂ w J t ∥ op ≤ C R . Henceforth, ∥ ∥ ∥ ˙ J t ∥ ∥ ∥ F ≤ C T ∥ ˙ w t ∥ and (18) yields the desired inequality after dividing by a t which remains positive all times t ∈ [0 , T R ) since J t does not vanish.

The next proposition provides an estimate of the time derivatives | ˙ a t | and ∥ ˙ w t ∥ up to time T R . For simplicity, we introduce the notation ∆ t = ∥∇L ( f w t ) ∥ .

Proposition 8. Assume ( B ) and ( C ) and that w 0 satisfies Assumption ( A ). Then, for any R > 0 , the time T R is positive and smaller than the maximal time T up to which (4) is defined, i.e. 0 < T R ≤ T . Moreover, define C := Lµ -1 µ -1 H ∆ 0 C R . Then a t and ∥ ˙ w t ∥ satisfy the following differential inequality on [0 , T R ) :

Proof of Proposition 8. By application of Proposition 7 to ( w t ) 0 ≤ t<T , we directly have that T R is positive and smaller than T and the following estimate holds for any t < T R :

It remains to upper-bound ∥ ˙ w t ∥ .

Controlling ∥ ˙ w t ∥ . Ignoring the dependence on w t in the notations for simplicity and setting B := H 1 2 J , we have, by definition of the dynamical system (4):

where the second line follows by the Woodbury matrix identity, and the third line follows by simple linear algebra. For the fourth line, we use the properties of the Frobenius norm. The fifth and last lines are direct consequences of Lemmas 2 and 4 in Appendix B.5.

Concluding. We can combine the upper-bound on | ˙ a t | and ∥ ˙ w t ∥ to get:

Finally, since t ≤ T , it follows from Proposition 1 that ∆ t ≤ L µ ∆ 0 e -µ GN 2 t which allows to conclude.

The next proposition controls a t and ∥ w t -w 0 ∥ in terms of T R , C R , and other constants.

Proposition 9. Consider the same conditions as in Proposition 8. Let ζ := 4 L µµ H µ GN and consider the (possibly infinite) time T + defined as:

with the convention that T + = + ∞ if 1 ≥ ζC R ∆ 0 a 2 0 . Then for any time t < T -:= min( T R , T + ) it holds that:

Proof of Proposition 9. First, by Proposition 8, we know that a t satisfies the following differential inequality:

with C := L µ H µ ∆ 0 C R . We will control a t by application of Grönwall's inequality. To this end, consider the ODE:

We know by Lemma 5 in Appendix B.5 that b t is given by:

for all times t < T + . Therefore, by Grönwall's inequality, it must hold that a t ≤ b t for all times t < min( T R , T + ) . Moreover, recalling now that ∥ ˙ w t ∥ ≤ Lµ -1 µ -1 H ∆ 0 a t e -µ GN 2 t by Proposition 8, we directly get that:

Integrating the above inequality in time yields the following estimate on ∥ w t -w 0 ∥ for any t < min( T R , T + ) :

where the second line follows by explicitly integrating the r.h.s. of (20) while the last line follows by the concavity of the square-root function.

Proposition 9 shows that the Frobenius norm of the J † t remains bounded at all times provided that ∆ 0 a 2 0 is small enough, a quantity that depends only on the initial conditions. Moreover, it also shows that making the product ∆ 0 a 0 small enough at initialization ensures that w t remains in a ball of radius R around w 0 . We exploit these two consequences to prove Proposition 3 in Appendix B.4.

B.4 Absence of blow-up for almost-optimal initial linear layer: proof of Proposition 3

To prove Proposition 3, we rely on the results of Appendix B.3 which allow controlling the evolution of the smallest singular value of the NTK matrix A w t . More precisely, Proposition 3 is a direct consequence of Proposition 9 with a particular choice for the initialization w 0 .

Proof of Proposition 3 . By assumption on w 0 , it holds that

Moreover, by definition of a 0 := ∥ ∥ ∥ J † 0 ∥ ∥ ∥ F and of σ 2 0 = σ min ( G ( u 0 )) , we know that a 2 0 < Nσ -2 0 . Hence, we get

where we introduced ζ := 4 L µµ H µ GN . This above inequality directly yields:

Therefore, we can use the above inequalities in the estimates (19) of Proposition 9 to get:

for any time t ∈ [0 , T R ) , where T R is defined as:

This implies, in particular, that w t never escapes the ball B ( w 0 , R ) . Moreover, using (16) we get that the smallest singular value σ ⋆ ( A ⊤ w t ) is always lower-bounded by a positive constant ˜ σ := a -2 0 ( 1 -ζC R ∆ 0 a 2 0 ) . Therefore, T R must necessarily be greater or equal to T , the maximal time over which w t is defined. Additionally, if T was finite, then w t must escape any bounded domain. This contradicts the fact that w t ∈ B ( w 0 , R ) . Therefore, T must be infinite and the inequality (6) applies at all times t ≥ 0 .

B.5 Auxiliary results

Lemma 1. Let J be a linear operator from a Hilbert space W to a finite-dimensional space H and assume that JJ ⊤ is invertible. Define the projector P = J ⊤ ( JJ ⊤ ) -1 J . Then the following relations hold:

Lemma 2. Let J be a linear operator from a Hilbert space W to a finite-dimensional space H and assume that JJ ⊤ is invertible. Then for any non-negative number ϵ , it holds that:

Lemma 3. Let W be a Hilbert space and R be a finite-dimensional euclidean space. Let J be a linear operator from W to R and H an invertible operator from R to itself. Further, assume that JJ ⊤ is invertible. Then the following holds for any non-negative number ϵ :

Lemma 4. Let J and K be two Hilbert-Schmidt operators between two Hilbert spaces. Then, it holds that:

Lemma 5. Let C and r be two positive constants. For a given initial condition b 0 > 0 , consider the following differential equation:

Then b t is defined for any time t < T + where T + is given by:

Moreover, b t is given by:

Proof. First note that a local solution exists by the Cauchy-Lipschitz theorem. Moreover, it must never vanish since otherwise, it would coincide with the null solution b t = 0 by uniqueness. However, this is impossible, since b 0 > 0 . Wecan therefore divide the ODE by b 3 t and explicitly integrate it which gives:

The solution is defined only for times t so that the r.h.s. is positive, which is exactly equivalent to having t < T + .

C Additional experimental results

C.1 Multi-seed experiments

We present result obtained for 5 independent runs. Each run uses a different initialization for the parameters of both student and teacher networks. Additionally, the training and test data are all generated independently for each run. All these results are obtained using SiLU non-linearity. It is clear from Figure 3, that the results display little variability w.r.t. the seed of the experiment, which justifies the single-seed setting chosen in the main paper.

Effect of over-parameterization Figure 4 (Left) shows the effect of over-parameterization of the network on generalization error. It appears that the generalization error remains stable with an increasing parameterization for both GN and GD, as soon as the network has enough overparameterization to fit the data exactly. On the other hand, the test error of RF improves with increasing over-parameterization which is consistent with [22].

Evolution of the smallest singular value of A w . Figure 4 (Right) shows that the smallest singular value σ ⋆ ( A w ) systematically increases during training, especially in the feature learning regime (bright red colors). Hence, the dynamics are far from blowing up even as the features change substantially. On the other hand, σ ⋆ ( A w ) remains nearly constant for large values of τ 0 (darker colors) which indicates that the features barely change in the kernel regime.

10

2

10

1

10 0

10 1

10 2

Initial std of the weights

10

4

10

3

10

2

Loss

Test loss at convergence

Random Features

1-HL: Gradient Descent

1-HL: Gauss-Newton

10 2

10 3

Training sample size

N

10

4

10

3

10

2

Loss

Test loss at convergence

Random Features

1-HL: Gradient Descent

1-HL: Gauss-Newton

2×10 3

3×10 3

4×10 3

Number of neurons/Rfs:

M

10

4

10

3

10

2

Loss

Test loss at convergence

Random Features

1-HL: Gradient Descent

1-HL: Gauss-Newton

10 0

10 1

10 2

10 3

10 4

10 5

10 6

10 7

Training iterations

10

7

10

6

10

5

10

4

10

3

10

2

(

Aw

)

100.0

10.0

5.0

1.0

0.5

0.1

0.01

0

C.2 Effect of the sample-size

Figure 5 shows the evolution of the test error as the training sample size N varies in { 100 , 200 , 500 , 1000 , 1500 } for two choices of activation functions: ReLU (Left) and SiLU (Right). In both cases, we display the results for two different values for the initial std τ 0 ( 10 -3 and 1 ). Results are reported for the best performing step size for each of GN and GD. The first observation is that the performance gap between random features RF and learned features (using either GN or GD) keeps increasing with the sample size. This observation suggests that the learned features use training samples more efficiently than random features. The second observation is the almost affine relation between the generalization error L test and training sample size in a logarithmic scale with a strong dependence of the slope on the variance at initialization and the optimization method. Such affine relation implies the following upper-bound on L test in terms of sample-size N :

for some positive constants C and α . The coefficient α controls the speed at which the estimator approximates f ⋆ as one accesses more training data and usually appears in learning theory to describe

the statistical convergence rate of a given estimator [46]. The consistent and strong dependence of α on the optimization methods (GN vs GD) and regime (kernel vs feature learning) further confirms the implicit bias of both initialization and optimization algorithms.

10 2

10 3

Training sample size N

10

9

10

8

10

7

10

6

10

5

10

4

10

3

10

2

10

1

Loss

Loss vs training sample size ('ReLU')

0 =1)

0 =10 3  )

GN (

0 =1)

GN (

0 =10 3 )

0 =1)

0 =10 3 )

0 =1)

0 =10 3  )

GN (

0 =1)

GN (

0 =10 3 )

0 =1)

0 =10 3 )

10 2

10 3

Training sample size N

10

9

10

8

10

7

10

6

10

5

10

4

10

3

10

2

10

1

Loss

Loss vs training sample size ('SiLU')

0 =1)

0 =10 3  )

GN (

0 =1)

GN (

0 =10 3 )

0 =1)

0 =10 3 )

0 =1)

0 =10 3  )

GN (

0 =1)

GN (

0 =10 3 )

0 =1)

0 =10 3 )

C.3 Choice of the non-linearity

Figure 6 shows the effect of step-size on the final performance of a SiLU network with two choices for the temperature parameter: β = 1 and β = 10 6 . Similarly to Figure 1, it is clear that increasing the step size in Gauss-Newton results in features that do not generalize well, while the opposite is observed in case of gradient descent. On the other hand, depending on the value of β , we observe that GN either outperforms GD ( β = 10 6 ) or the opposite ( β = 1 ). This variability suggests that the effectiveness of an optimization method in improving generalization is dependent on the specific characteristics of the problem at hand. Finally, note that for β = 10 6 , we recover almost the same results as those obtained for ReLU in Figure 1 (right). This is due to the fact SiLU becomes a tight smooth approximation to ReLU for large values of β . We also notice that when using a larger std for the initial weights (ex. τ 0 = 1 ), using either ReLU or SiLU with ( β = 1 ) yields similar results, as shown in Figure 7.

C.4 Experiments on MNIST

We have performed a series of experiments on MNIST to illustrate the behavior of GN on higher dimensional data (notably, with multidimensional targets). To stay as close as possible to the theoretical framework, we modify the original MNIST dataset. Specifically, we construct a student/teacher setup in which the trained network can achieve a zero-loss, which is an assumption made in Section 3.1.

Building datasets based on MNIST by using a teacher. In our theoretical framework, we restricted ourselves to cases where the minimum loss can be achieved on the considered model. To make this condition hold, we build a new dataset by using a teacher trained on the MNIST classification task [16]:

we construct a training set for the teacher which consists of D tr ∪D ts , where D ts is the original MNIST test dataset and D tr is a balanced subset of size 3000 of the original MNIST training set;

we then train a teacher neural network with one hidden layer of size 50 and ReLU activation function on training set D tr ∪ D ts using a cross-entropy objective;

10

4

10

3

10

2

10

1

10 0

10 1

10 2

10 3

10

1

10 0

WCD

WCD/Loss vs step size (beta=1)

10

4

10

3

10

2

10

1

10 0

10 1

10 2

10 3

Step size

10

8

10

6

10

4

10

2

10 0

Loss

Test loss (Linear re-fitting)

Test loss

Train loss

GN

RF

GN

RF

10

3

10

1

10 1

10 3

10

1

10 0

WCD

WCD/Loss vs step size (beta=1e6)

10

3

10

1

10 1

10 3

Step size

10

8

10

6

10

4

10

2

10 0

Loss

Test loss (Linear re-fitting)

Test loss

Train loss

GN

RF

GN

RF

10

3

10

1

10 1

10 3

10 0

9×10 1

9.5 × 10

1

1.05 × 10 0

1.1 × 10 0

WCD

WCD/Loss vs step size (std=1.0)

10

3

10

1

10 1

10 3

Step size

10

8

10

6

10

4

10

2

10 0

10 2

Loss

Test loss (Linear re-fitting)

Test loss

Train loss

GN

RF

GN

RF

10

3

10

1

10 1

10 3

10 0

8×10 1

9×10 1

WCD

WCD/Loss vs step size (std=1.0)

10

3

10

1

10 1

10 3

Step size

10

8

10

6

10

4

10

2

10 0

10 2

Loss

Test loss (Linear re-fitting)

Test loss

Train loss

GN

RF

GN

RF

3. given the trained teacher f T , we build new datasets ¯ D tr and ¯ D ts for the student network:

Training procedure. Once the pair of training and testing datasets ( ¯ D tr , ¯ D ts ) has been built, we use them to train and test a student neural network based on (10), with one hidden layer of size 5000 and a ReLU activation function. We use a similar initialization scheme for the student's network and fix the std of the initial weights to τ 0 = 0 . 001 . For training, we use a quadratic loss between the student and teacher's outputs and perform a maximum of 100000 iterations of either GN or GD. The training stops whenever the training loss goes below 10 -6 . We vary the step size λ on a logarithmic range [10 -3 , 10 3 ] for both methods and evaluate the test loss and test loss after linear re-fitting.

10

3

10

2

10

1

10 0

10 1

10 2

10 3

Step size

10 2

6×10 1

2×10 2

3×10 2

Loss

Test loss (Linear re-fitting)

Test loss

GN

GN

Results. Figure 8 shows the evolution of the test loss (both with or without re-fitting) as a function of the step size. We observe similar behavior as in the main experiments, with hidden learning occurring for GN when using small step sizes and a degraded generalization error as the step size increases. On the other hand, generalization improves for GD by increasing the step size up to the point where optimization becomes unstable.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies applying the Gauss-Newton method to train the neural network to solve a regression problem in the NTK and mean-field regime. The paper establishes a linear convergence rate on Gauss-Newton in this setting. The authors further provide experiments in the teacher-student network setting to study the effect of initialization and step size on convergence and generalization.\n\nSTRENGTHS:\nThe presentation of this paper is quite good. All the proofs and theorems and experimental results are well-documented. The proofs and experiment results in the appendix are very nicely structured and organized, making it very clear for the reviewer to understand what the authors are doing.\n\nWEAKNESSES:\nThe major weaknesses of this work is on its contribution. First of all, the Gauss-Newton method is a well-known classical method in convex optimization to achieve a faster super-linear convergence rate than first-order methods such as gradient descent which can only achieve linear convergence rate. Therefore, it is natural to apply this method to improve the optimization on training neural network. However, the challenging problem in training neural network is due to its non-convexity. The authors consider the NTK regime and mean-field regime. It is well-known that the optimization problem in the NTK regime is strongly-convex-like. Therefore, the results the authors established in the paper is non-surprising and further, proposition 1 is only able to establish a linear convergence rate whereas Gauss-Newton is able to achieve locally super-linear convergence rate in the classical convex setting, with some additional assumptions needed to be made. Note that if Gauss-Newton is only able to achieve linear convergence, then the advantage of Gauss-Newton is non-existent over gradient descent since gradient descent can also converge linearly and the cost per-step is much smaller than Gauss-Newton. Also, the goal of optimization in deep learning is for the model to generalize, and it is not clear to me how the Gauss-Newton method can affect the model's generalization. \n\nTherefore, currently, I am issuing a rejection for this submission.\n\nQUESTIONS:\nDue to the current scale of NeurIPS and the workload on the reviewer, I am unable to go through the appendix in detail. The authors claim their results also holds for the mean-field regime and the proofs in the appendix seems to only address the NTK regime. I encourage the authors to clarify their results. I think for the mean-field regime the authors may have made some assumptions like the initial point is close to the true minimizer which can circumvent the problem of dealing with non-convexity in the mean-field regime. However, this, again, returns back to the classical convex regime.\n\nLIMITATIONS:\nThe authors discussed the limitations in the conclusion section.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work theoretically and empirically studied learning dynamics and generalization properties of one-hidden-layer networks trained with the Gauss-Newton algorithm. The main theoretical result is that the authors provided a loss convergence guarantee, which is provably faster than the same networks trained with gradient descent. The authors then empirically studied generalization properties of the GN-trained NN in a student-teacher setting.\n\nSTRENGTHS:\nThe theoretical treatment of the GN in Sec. 4 is well appreciated. While I did not check the correctness of the proof, it appears intuitive that GN methods have better convergence.\n\nWEAKNESSES:\nMy primary concern is the relevance of the work. My impression is that GN method is virtually never used in NN training, and a brief literature search turned up only a few papers discussing this. This is simply due to the dramatically higher compute demand compared to GD, as the authors mentioned. The size of the Jacobian is N by N, where N is the number of trainable parameters. So training any modern NN with GN can be prohibitively expensive, since it involves pseudo-inversing the Jacobian at every step. While the author mentions other methods such as natural gradient descent, which are in spirit connected to GN, it is unclear how the theoretical/empirical results in this work generalize to those methods, which may be more relevant to this conference.\n\nMy second concern, which I am less confident of, is that the fact that GN converges faster than GD (at least in terms of the big O analysis) comes as no surprise. That GN converges faster than GD in a nonlinear least-squared problem appears to be well known (https://www.princeton.edu/~aaa/Public/Teaching/ORF363_COS323/F14/ORF363_COS323_F14_Lec9.pdf). I understand that additional technical requirements need to be proven in the one-hidden-layer NN case, but I'm not sure that they are of importance conceptually. \n\nFinally, the empirical part of the paper and the theoretical paper part are pretty unrelated, besides the fact that they both involve one-hidden layer networks.\n\nQUESTIONS:\n1. How can we generalize the results in this work, theoretically or empirically, to approximate second order methods (e.g. natural GD) that people use to train NNs?\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates theoretically and empirically the implicit biases of the Gauss-Newton (GN) optimization algorithm on over-parametrized one hidden layer-models (e.g. the capacity of the model is much higher than that of the ground-truth function to approximate). The main theoretical contribution of the paper pertains to global optimization:  a global convergence rate is derived when dynamics do not blow up in finite time, and to provide sufficient conditions on the initial state of the model to prevent this blow up. The experimental section explores generalization: it compares GN and Gradient Descent (GD) for various initial hidden weights variances and learning rates. The key take-away of this last section (and empirical prescription thereof) is that smaller learning rates are better for GN (while the converse holds for GD) and that GN under optimizes the linear layer (e.g. opposite of the lazy regime) while learning good hidden features.\n\nMore precisely:\n\n- Section 3.1 introduces the hypothesis at use on the ground-truth data, the learning objective (Eq. 1) and the model at use, in the finite width (Eq. 2) and mean-field infinite width (Eq. 3) cases. Overparametrization (A) is defined in terms of the invertibility of the feature kernel.\n\n- Section 3.2 defines the generalized GN vector field (Eq. 5) and defines associated discrete and continuous weight dynamics (Eq. 6). Hypothesis on the parameters coming into play in the GN dynamics are introduced (B and Eq. 5) in order to derive subsequent global convergence results.\n\n- Section 4.1 presents Proposition 1 which states that under the aforementioned assumptions, the GN dynamics either blow up or converge to a global minimizer of the loss with an explicit rate which only depends on specific constants characterizing the functions at play, which is in stark contrast with SGD convergence rate which is controlled by the small singular value of the Neural Tangent Kernel (NTK).\n\n- Section 4.2 presents Proposition 2 which states that it is sufficient, for the dynamics not to blow up and converge that the rate derived previously, that the linear layer is near optimal (e.g. $\\\\|\\nabla_v \\mathcal{L}(f(u_0, v_0))\\\\|$ sufficiently small at initialization).\n\n- Section 5.1 presents the teacher-student experimental setup, where the ground-truth data $X$ is Gaussian (with dimension $d=10$) with associated labels $Y$ generated by a shallow one hidden-layer model (5 hidden units) and the student model is a wide/overparametrized one hidden-layer model (5000), both with a ReLU activation. The three optimization algorithms at use are presented: GN, GD and random features (RF) whereby the hidden weights are random and the linear weights are taken as the minimal-norm least square solution (Eq. 9). \n\n- Section 5.2 presents the performance metrics, i.e. the weighted cosine distance (WCD) between teacher and student features (which allows for teacher and student features of different dimensions, promotes feature alignment and penalizes feature norm) and the test loss after linear re-fitting (Test-LRFit). Test-LRFit is particularly useful to disentangle the analysis of the whole model performance *versus* the quality of the learned hidden features, and subsequent identify when GD and GN operate in the kernel/lazy or feature-learning regime.\n\n- Section 5.3 presents and analyzes training experiments, with varying standard deviations of the visible-to-hidden weights (which we denote here $\\sigma_u$ for simplicity) and learning steps. Key results are as follows:\n   + Upon increasing values of $\\sigma_u$, GD and GN both exhibit a transition between the *feature learning regime* (performance is better than the RF baseline) and the *kernel regime* (performance matches that of the RF baseline).\n   + In the feature-learning regime (small $\\sigma_u$), GN operates best (in terms of the resulting test loss) with a small learning rate ($\\approx 10^{-3}$) with performance degrading onwards while GD operates best as the learning rate increases ($\\approx 10^{2}$). GN best performance is better than GD best performance.\n   + GN under-optimizes the output linear layer when employing small learning rates while learning good hidden features (as measured by the gap in test loss before and after Linear re-fitting).\n   + GN starts learning the internal features (as measured by test loss after linear re-fitting until iteration $\\approx 10^4$) and output features onwards (as measured by the test loss). Conversely, last layer and hidden layer are learned simultaneously with GD (i.e. test loss and test loss after linear re-fitting coincide throughout training).\n   + GN converges faster with larger learning rates, but achieves best generalization when learning rate is small. Therefore good generalization with GN comes at the cost of slower convergence.\n   + Finally, across all experiments, the WCD metric is a good proxy to the test loss.\n\nFinally, the Appendix provides very detailed mathematical proofs and extra experimental results (e.g. use of the SiLU activation function instead of ReLU, experiments on MNIST).\n\nSTRENGTHS:\n- This is a beautifully written paper that is readable for non-super experts of this literature (as I am).\n- The maths in Appendix are extremely rigorous and neat (I carefully checked the proofs).\n- The conditions under which the derived theoretical guarantees hold are apparently less stringent than equivalent studies on GD (e.g. any activation function, initial weight not constrained to lie within a radius determined by the smallest singular value of the NTK).\n- The experimental set up for the teacher-student study is sound and the RF baseline is extremely informative about what is going on.\n\nWEAKNESSES:\nI should state upfront that I am not an expert of this literature, so I may not be aware of the expected theoretical and experimental standards that lead to acceptance. This being said, I'm overall disappointed by the **experiments**:\n\n- **Most importantly**: the MNIST experiments (last page of the appendix) seem to contradict the main findings that GN has to operate in small learning rate regime to work best and that GN would fail to optimize the last (linear) layer -- the gap between test loss and test loss after linear re-fitting is even greater for GD than GN! In spite of this glaring observation in stark contrast with the results appearing in the main, the authors simply do not comment upon this phenomenon at all (L.744-747). \n\n- Apart from the choice of the model which sticks to the mean-field limit introduced in Section 3 (e.g. large number of hidden neurons, normalizing the model output by M), the theoretical and experimental parts are very disconnected. While I do understand that the intention was to cover optimization and generalization in a complimentary fashion (the former theoretically, the latter empirically), theory does not drive much the design of the experiments, and experiments are not crafted to validate the theory. I make some suggestions in the questions below.\n\n- For the experiments: while I do understand that the choice of a one-hidden layer teacher-student setting seems to be a standard in this literature, it lies in between something that could be even more theoretically tractable with controllable analytical quantities (e.g. a toy quadratic optimization problem with known curvature) and something more realistic (e.g. multiple hidden layers model). So I'm twice frustrated that the theory hasn't been checked in an even simpler setting where everything is analytically well controlled, nor empirically tested against a model that is more complicated, to see if the proposed empirical prescriptions hold beyond this simple setting.\n\n- I'm skeptic about the stopping criterion used for GN and worried that **it might skew all the experimental results**.\n\n- In spite of all the rigor and clarity of the proposed study (which I do appreciate a lot), there is unfortunately no concrete practical prescription to use the GN algorithm. Even more so because the results on MNIST contradict those obtained in the teacher-student setting.\n\nQUESTIONS:\nImportant questions/remarks:\n\n- My most important remark (hilighted above) is that I suspect your MNIST experiments are not included in the main because they directly contradict the analysis performed on the regression experiments: GD and GN exhibit the exact same trends, GN works best with large learning rates and optimizes the last layer even better than GD. If you have time by the end of the rebuttal phase, I would be very happy if you could \"fix\" the experiments, or make sense of the observed discrepancies between the regression problem and MNIST. \n\n- Could you please define precisely what you mean exactly by \"blow-up\"? You directly say in the main that if w is defined over a finite horizon ($T < \\infty$), then it is said to \"blow-up\". Then you say that blow up happens when the minimal singular value of the NTK is zero. My impression is that there is not enough intuition in the main as to why this is the case. Looking at the appendix, my own intuition is that when $\\sigma^\\star(A_w)$ approaches zero, $J_w^\\dagger$ diverges (Eq. 15 in the Appendix), and since the GN vector field is \"roughly\" $\\Phi(w) \\approx J_w^\\dagger \\cdot \\nabla \\mathcal{L}$, the GN vector field itself diverges. Is this intuition correct? Could you please include such intuition in the main to better define what \"blow up\" is precisely?\n\n- You say in the main that \"GN provably faster than GD\". Could you please write down explicitly the global convergence rate of GD to be able to compare it directly with that of GN derived in Proposition 1 (even if it is a well-known result)? However, the comparison does not seem this straightforward when reading L.196-197. I'm just wondering how much writing GD global convergence rate would enlighten this part. \n\n- I would have been very happy to see an empirical validation of Proposition 2 on a hand-crafted toy model, even simpler than the teacher-student setting, e.g. $\\mathcal{L}=\\frac{1}{2}(\\theta - \\theta_\\star)^\\top \\cdot Q \\cdot (\\theta - \\theta_\\star)$? On such problems the properties of the spectrum of $\\mathcal{L}$ could be well controlled. Taking $dim(\\theta) = 2$ and visualizing trajectories in case of convergence, divergence, and showing the trajectory of GD as well would have been nice to see!\n\n- L.223, you write: \"The near-optimality condition on v0 can always be guaranteed by optimizing the second variable v while keeping u0 fixed\". So why didn't you put it in practice in your own experiments? Especially because you observed that the last layer was not training so well. I'm not saying though that all GN experiments should have been performed with v-pretraining, but extra experiments could have been included. To make myself clearer, here is the gist of some experiment I would have been happy to see:\n  - identify a failure mode of GN in the student-teacher setting -- i.e. the weight dynamics diverge.\n  - show that by pre-training $v$, the problem vanishes (i.e. the weight dynamics converge to an optimum).\n  - demonstrate it on both the teacher-student problem, and on more complex problem like MNIST.\n\n- L. 297: why is the stopping criterion for GN lesser than that of GD ($K^{GD}=10^6$, $K^{GN}=1.5 \\times 10^5$)? Is this because you observed blow-up? This choice is glaring in Fig. 2, bottom left panel and looks very strange: the GN learning trajectory stops at $10^5$ iterations and as the loss was gently decreasing, I would have been happy to see it go even lower! The bottom line is that the stopping criterion for GN looks quite arbitrary and cherry-picked. Why not using an $\\epsilon-$relative-change-of-loss stopping criterion, namely: stop the simulation where the parameters have converged up to precision $\\epsilon$?\n\n- L.336: \"this is unlike GD where larger step size yield better performing features\". Looking closer at Fig. 1: indeed, the test loss decreases until $\\lambda=10^{-3}$, but what happens beyond? Here again, the cut seems arbitrary. And as we expect, when looking at the blue curves (GD) Fig. 6 in the Appendix, the test loss for GD does increase beyond $\\lambda=10^{-3}$. It is not a major issue, however I'm pointing out that the experimental analysis in the main is skewed again by arbitrary choices. \n\nMinor questions/remarks:\n- For non-experts like me: could you please an intuition why overparametrization (A) is defined this way? Perhaps this is something obvious for researchers of this community. \n-  You say in the main that your theoretical results hold both in the kernel and feature learning regime. However, I do not understand why - here I am certainly missing some subtleties. Perhaps it would have been helpful to add background knowledge in the appendix about the exact definition of the feature and kernel limit. Is this about normalizing the model output by $1/\\sqrt{M}$ (kernel) or $1/M$ (mean-field, as done in this paper)?\n- In the same vein, I'm confused about whether the kernel/feature-learning *regime* is the same as (or related to) the kernel/mean-field *limit*. For the paper to be self-contained, it would be useful (at least for me) to add more background knowledge in the Appendix. \n- L.207: could you explain the intuition?\n\nLIMITATIONS:\nThe main limitation of the paper is the quality of the presented experiments, the fact that they are not consistent across two different learning problems, and that the results presented in the paper do not lead to a concrete prescription. For the sake of having the proposed work prescribe concrete takeaway prescriptions to use the GN algorithm in practice, I would recommend accept by the end of the rebuttal phase, once:\n- the results you obtain on MNIST (on a single hidden layer architecture, consistently with the student-teacher setting) are consistent with the results presented in the main.\n- the choice of the stopping criterion for GN is justified.\n\n**POST-REBUTTAL UPDATE**: the authors have thoroughly addressed my concerns, so I increased my score to accept.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study the Gauss-Newton optimization algorithm in the overparameterized setting. They derive conditions for convergence of the Gauss-Newton algorithm with parameter-dependent damping in terms of the convexity of the loss, the smoothness of the loss Hessian, and the damping constant.\n\nThey then carry out an empirical study with a student-teacher, one hidden layer setup (overparameterized student), and show that Gauss Netwon in this setting requires small step size and small initialization to be successful. They also show that GN requires a larger number of steps but seems to induce more feature learning.\n\nSTRENGTHS:\nThe idea of studying GN in the feature learning limit and trying to understand its implicit biases is an interesting one. The convergence analysis is new to my knowledge, and the experiments focus nicely on key issues of learning dynamics from second-order methods.\n\nWEAKNESSES:\nThe algorithm as presented requires a parameter-dependent damping constant, which requires computation of the smallest non-zero NTK eigenvalues. These values are very difficult to compute in practice, and the algorithm is generally impractical. The theory consists of a convergence result for the aforementioned algorithm, and it’s not clear how relevant it is in practice.\n\nPart of the issue with second order methods like GN are their impracticality in large model, large data settings. The experiments are not on any realistic datasets, and don’t suggest any promising ways forward for development of GN methods.\n\nThere may be connections to other mean field regimes which are not discussed in the text.\n\nQUESTIONS:\nWhat is the connection of the mean field analysis to the regimes from these two papers?\n\nhttps://proceedings.neurips.cc/paper_files/paper/2022/hash/d027a5c93d484a4312cc486d399c62c1-Abstract-Conference.html\nhttps://proceedings.mlr.press/v139/yang21c\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}