Depth Without the Magic: Inductive Bias of Natural Gradient Descent

1 INTRODUCTION There is plenty of empirical evidence that the choice of network architecture is an important determinant of the success of deep learning ( He et al. , 2015 ; Vaswani et al. , 2017 ) . The empirical observations are now supported by theoretical work into the role that parameter-to-hypothesis mapping plays in determining inductive biases of gradient-based learning . Unregularized gradient descent can efficiently find low-rank solutions in matrix completion problems ( Arora et al. , 2019 ) , sparse solutions in separable classification ( Gunasekar et al. , 2018 ) or compressed sensing ( Vaškevičius et al. , 2019 ) . Valle-Pérez et al . ( 2018 ) studied deep neural networks and found evidence that the parameter-hypothesis mapping1 is biased towards simpler functions as measured by Kolmogorov complexity . Taken together , these observations and findings have lead the community to hypothesize that The parameter-to-hypothesis mapping influences the inductive biases of gradient-based learning and may play an important role in generalization . In parallel to improving architectures , considerable research was done to improve optimization algorithms for deep learning , with a focus on faster convergence and robustness to hyperparameters . Among the most advanced optimization methods are natural gradient descent ( NGD ) techniques . An intuitive motivation for NGD is that it improves convergence by implicitly lifting the problem from parameter-space , where the loss is non-convex and poorly behaved to the Riemannian manifold 1The mapping between the parameter space and the set of hypotheses as seen on Figure 1 of hypotheses , where the loss is better behaved . From the perspective of inductive biases , the most interesting aspect of NGD is its approximate invariance to reparametrization . Natural gradient descent eliminates the effect of parameter-to-hypothesis mapping . These two observations invite questions about the nature of inductive biases in NGD as well as the role of parametrization-dependence in generalization . The first , practical , implication is as follows : if the parameter-to-hypothesis mapping really does play an important role in generalization , then eliminating its influence on the optimization path may be undesirable , and consequently the pursuit of implementing exact NGD in deep architectures may be counterproductive . Secondly , studying the behaviour of NGD in various models and tasks may give us new insights about the importance of parametrization , and could perhaps offer a way to experimentally or theoretically test hypotheses . In this paper we study the inductive bias of natural gradient descent in deep linear models . These models are particularly suited for our analysis because ( a ) efficient algorithms exist to calculate exact natural gradients which is otherwise computationally intractable and ( b ) the inductive biases of Euclidean gradient descent ( EGD ) in these models have been thoroughly studied and understood . We make the following contributions : • In linear classification , we show that NGF is invariant under invertible transformations of data ( Theorems 1 & 2 ) and as a consequence it can not recover the ` p large margin solutions that EGD tends to converge to . • We further show that ( in case of separable classification ) when the number of parameters exceeds the number of datapoints , NGF interpolates training labels in a way similar to ordinary least squares or ridgeless regression ( Theorems 3 & 4 ) . • We demonstrate experimentally that there exist learning problems where NGD can not reach good generalization performance , while EGD with the right architecture can succeed . • To perform experiments , we extended the work of Bernacchia et al . ( 2018 ) to derive efficient and numerically stable algorithms for calculating exact natural gradients in diagonal networks ( Gunasekar et al. , 2018 ) and deep matrix factorization ( Arora et al. , 2019 ) . Before stating our main theoretical and experimental results we review some relevant background on parametrization-dependent implicit regularization and natural gradients . 2 BACKGROUND . 2.1 SEPARABLE CLASSIFICATION WITH DEEP LINEAR MODELS . In this article we consider binary classification datasets { ( xn , yn ) , n = 1 , . . . , N } separable by a homogeneous linear classifier with a positive margin ( i. e. ∃β∗ s.t . ynx > nβ ∗ ≥ 1 ∀n ) . ( We use the notation X = ( x1 · · ·xN ) > ) . In such situation β∗ is not unique and there may be many separating hyperplanes which all achieve 0 training loss - it is up to the inductive biases of the learning algorithm to select one . Soudry et al . ( 2017 ) studied the dynamics of unregularized Euclidean gradient descent on logistic loss and found that the iterate β ( t ) converges to the well-known ` 2 large margin classifier in direction , that is limt→∞ β ( t ) |β ( t ) | = β∗ ` 2 |β∗ ` 2 | where β∗ ` 2 = argmin β∈RD ||β||2 s.t . ynx > nβ ≥ 1 ∀n . Importantly , Gunasekar et al . ( 2018 ) later showed that this behaviour changes if the gradient descent is performed on a different parametrization . In this paper we will focus on L-layer linear diagonal networks ( Gunasekar et al. , 2018 ) , where β = w1 w2 . . . wL , using to denote elementwise product . When we adjust parameters w1 , . . . , wL through Euclidean gradient descent , β ( t ) converges to the ` 2 L large margin separator defined as limt→∞ β ( t ) |β ( t ) | = β∗diag |β∗diag| where β∗diag = argmin β∈RD ||β|| 2 L s.t . ynx > nβ ≥ 1 ∀n . A remarkable consequence of this is that unregularized gradient descent can find sparse classifiers , without any form of explicit regularization . In fact , this inductive bias is even more sparsity-seeking than the typically used ` 1 regularization ( see e. g. Koh et al. , 2007 ; Tibshirani , 1996 ) . Figure 2 illustrates this behaviour in a 2D example . 2.2 MATRIX COMPLETION VIA DEEP MATRIX FACTORIZATION . The task of matrix completion involves recovering an unknown matrix β∗ ∈ RD×D from a randomly chosen subset of observed entries2 . The problem is clearly underdefined : there are infinitely many matrices that match the observed entries . It is common to make additional assumptions about β∗ , most commonly that that it has low rank , under which it becomes identifiable . One approach to matrix completion under the low-rank assumption is based on explicit regularization ( e.g . nuclear norm ) which leads to a convex optimization problem . Another common approach is matrix factorization using an underparametrized representation β = UV where the sizes of U ∈ RD×R and V ∈ RR×D are restricted to ensure β ’ s rank is at most R. Learning then proceeds by minimizing the non-convex mean-squared reconstruction error in U , V via gradient descent . Remarkably , Gunasekar et al . ( 2017 ) showed that the gradient-based matrix factorization method tends to converge to low-rank solutions even in the overparametrized setting , i.e . when β = W1W2 where W1 and W2 are full square matrices , without any explicit regularization . This was later extended by Arora et al . ( 2019 ) , who studied the deep matrix product parametrization of the form β =W1W2 · · ·WL . Arora et al . ( 2019 ) ran experiments for different matrix completion tasks varying initialization , depth and number of observations and compared them to minimum nuclear norm solution . When the number of observed entries is large gradient descent in deep matrix factorization models tended to the minimum nuclear norm solution . However , in the interesting case of fewer observed entries , the behaviour was different . Gradient descent pre- ferred solutions with lower effective rank at the expense of higher nuclear norm . From the evolution of the singular values of β they also concluded that the implicit regularization is towards low rank that becomes stronger as depth grows . 2to simplify presentation we assume the matrices are square , but our arguments hold more generally . 2.3 NATURAL GRADIENT DESCENT . In the next section we briefly introduce some notation and key properties of natural gradient descent ( NGD , Amari , 1997 ; Pascanu & Bengio , 2013 ) . Intuitively , one can think of NGD as a gradient descent method , but not in the Euclidean space ( with the Euclidean metric ) of parameters , but instead on the Riemannian manifold of probabilistic models the parameters define ( equipped with a different metric ) . More specifically , let ’ s say that the parameter of interest is θ , where θ defines a probabilistic model p ( y|x , θ ) . We assume that we wish to minimize the log loss under this model , i. e. l ( θ , x , y ) = − log p ( y|x , θ ) and L ( θ ) = ∑N n=1 l ( θ , xn , yn ) . Then , NGD is usually defined as θ ( t+ 1 ) = θ ( t ) − ηF−1 ( θ ) ∇θL ( θ ) , where ( 1 ) F ( θ ) = EX [ EY |X ; θ [ ∇θL ( θ ) ∇ > θ L ( θ ) ] ] ( 2 ) is the average Fisher information matrix and η is the step size . In the above definition , EY |X ; θ is taken over the distribution specified by θ , but distribution with respect to which the expectation EX is calculated can be arbitrarily chosen . In this article we use the empirical distribution of training data , though other choices are possible ( Pascanu & Bengio , 2013 ) . We will also consider natural gradient flow ( NGF ) the continuous limit of NGD , analogously defined as θ̇ = −F−1 ( θ ) ∇θL ( θ ) . ( 3 ) We also note , that F ( θ ) is not generally invertible , and indeed it will not be in some of the cases we will consider . Therefore , it is more correct to define NGF as any trajectory θt which satisfies F ( θ ) θ̇ = −∇θL ( θ ) . ( 4 ) The natural gradient direction is thus only unique within the eigenspace of F ( θ ) . Of all natural gradient directions , one common choice is to use the Moore-Penrose pseudoinverse of F : θ̇ = −F+ ( θ ) ∇θL ( θ ) . ( 5 ) We have seen how in EGD , different parametrization of the same problem leads to drastically different trajectories and optima . However , NGD with infinitesimally small learning rate ( i. e. NGF ) always follows the same trajectory in model-space and this finds the same optimum , irrespective of how it is parametrized , provided that the parametrization is smooth and locally invertible . Below we formally state this property Amari ( 1997 ) , alongside a short proof for illustration . Statement ( Invariance of NGF under reparametrization ) . Let w and θ be two parameter vectors related by the mapping θ = P ( w ) and consider natural gradient flow in w. Assume that ( 1 ) the Jacobian J = ∂θt∂wt and ( 2 ) F ( θt ) are both full rank for all t. If wt follows natural gradient flow starting from w0 then θt = P ( wt ) follows NGF , i. e. it solves θ̇t = −F ( θt ) +∇θtL ( X , θt ) .

Summary: This paper studies the inductive bias of natural gradient flow in deep linear networks and in matrix factorization. They show that the solution to the empirical risk minimization problem using NGF is invariant to pixel permutation, and conclude that the desirable property of Euclidean gradient descent under logistic loss of finding the minimum l2 norm solution does not hold (theorem 1&2).  They also contribute an efficient NGD algorithm for deep diagonal linear networks.