Embedding Principle of Loss Landscape of Deep Neural Networks

1 Introduction . Understanding the loss landscape of DNNs is essential for a theory of deep learning . An important problem is to quantify exactly how the loss landscape looks like ( E et al. , 2020 ) . This problem is difficult since the loss landscape is so complicated that it can almost be any pattern ( Skorokhodov and Burtsev , 2019 ) . Moreover , its high dimensionality and the dependence on data , model and loss make it very difficult to obtain a general understanding through empirical study . Therefore , though it has been extensively studied over the years , it remains an open problem to provide a clear picture about the organization of its critical points and their properties . In this work , we make a step towards this goal through proposing a very general embedding operation of network parameters from narrow to wide DNNs , by which we prove an embedding principle for fully-connected DNNs stated intuitively as follows : Embedding principle : the loss landscape of any network “ contains ” all critical points of all narrower networks . ∗Corresponding author : zhyy.sjtu @ sjtu.edu.cn . †Part of this work is done when ZZ was an undergraduate student of Zhiyuan Honors Program at Shanghai Jiao Tong University . ‡Corresponding author : xuzhiqin @ sjtu.edu.cn . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . A “ narrower network ” means a DNN of the same depth but width of each layer no larger than the target DNN . The embedding principle slightly abuses the notion of “ contain ” since parameter space of DNNs of different widths are different . However , this inclusion relation is reasonable in the sense that , by our embedding operation , any critical point of any narrower network can be embedded to a critical point of the target network preserving its output function . Because of this criticality preserving property , we call this embedding operation the critical embedding . We conclude our study by a “ principle ” since the embedding principle is a very general property of loss landscape of DNNs independent of the training data and choice of loss function , and is intrinsic to the layer-wise architecture of DNNs . In addition , the embedding principle is closely related to the training of DNNs . For example , as shown in Fig . 1 ( a ) , the training of a width-500 two-layer tanh NN experiences stagnation around the blue dot presumably very close to a saddle point , where the loss decreases extremely slowly . As shown in Fig . 1 ( b ) , we find that the DNN output at this blue point ( red solid ) is very close to the output of the global minimum ( black dashed ) of the width-1 NN , indicating that the underlying two critical points of two DNNs with different widths have the same output function conforming with the embedding principle . Importantly , this example shows that the training of a wide DNN can indeed experience those critical points from a narrow DNN unraveled by the embedding principle . Moreover , it demonstrates the potential of a transition from a local/global minimum of a narrow NN to a saddle point of a wide NN , which may be the reason underlying the easy optimization of wide NNs . The embedding principle suggests an underlying mechanism to understand why heavily overparameterized DNNs often generalize well ( Breiman , 1995 ; Zhang et al. , 2017 ) as follows . Roughly , the overparameterized DNN has a large capacity , which seems contradictory to the conventional learning theory , i.e. , learning by a model of large capacity easily leads to overfitting . The embedding principle shows that the optima of a wide network intrinsically may be embedded from an optima of a much narrower network , thus , its effective capacity is much smaller . For example , as illustrated in Fig . 1 , training of a heavily overparametrized width-500 NN ( vs. 50 training data ) with small initialization first stagnated around a saddle presumably from width-1 NN and later converges to a global minimum presumably from width-3 NN , which clearly does not overfit . This implicit regularization effect unraveled by the embedding principle is consistent with previous works , such as low-complexity bias ( Arpit et al. , 2017 ; Kalimeris et al. , 2019 ; Jin et al. , 2020 ) , low-frequency bias ( Xu et al. , 2019 , 2020 ; Rahaman et al. , 2019 ) , and condensation phenomenon of network weights ( Luo et al. , 2021 ; Chizat and Bach , 2018 ; Ma et al. , 2020 ) . 2 Related works . The loss landscape of DNNs is complex and related to the generalization . Skorokhodov and Burtsev ( 2019 ) numerically show that the loss landscape can almost be any pattern . Keskar et al . ( 2017 ) visualize minimizers in a 1d slice and suggest that a flat minimizer generalizes better . Wu et al . ( 2017 ) find that the volume of basin of attraction of good minima may dominate over that of poor minima in practical problems . He et al . ( 2019 ) show that at a local minimum there exist many asymmetric directions such that the loss increases abruptly along one side , and slowly along the opposite side . Degeneracy is also an important property of minima . Cooper ( 2021 ) shows that global minima is typically a high dimensional manifold for overparameterized DNNs . Sagun et al . ( 2016 ) empirically shows that Hessian of the minimizer obtained by the training has many zero eigenvalues . Under strong assumptions , Choromanska et al . ( 2015 ) shows minima tend to be highly degenerate . This work demonstrates wide existence of highly degenerate critical points , including local or global minima and saddle points , in the loss landscape by the embedding principle . Lots of previous theoretical works focus on very wide DNNs , such as the phase diagram of two-layer ReLU infinite-width NNs ( Luo et al. , 2021 ) , NTK regime ( Jacot et al. , 2018 ; Arora et al. , 2019 ; Zhang et al. , 2020 ; Du et al. , 2019 ; Zou et al. , 2018 ; Allen-Zhu et al. , 2019 ; E et al. , 2019 ) , mean-field regime ( Mei et al. , 2018 ; Rotskoff and Vanden-Eijnden , 2018 ; Chizat and Bach , 2018 ; Sirignano and Spiliopoulos , 2020 ) . By the embedding principle , this work demonstrate the loss landscape similarity between a moderate-width NN and a very wide NN , that they share a set of critical points embedded from that of narrower NNs . Therefore , results about infinite-width NNs could provide valuable insights about training of finite-width NNs used in practice . The complexity of NN output increases during the training ( Arpit et al. , 2017 ; Xu et al. , 2019 , 2020 ; Rahaman et al. , 2019 ; Kalimeris et al. , 2019 ; Goldt et al. , 2020 ; He et al. , 2020 ; Mingard et al. , 2019 ; Jin et al. , 2020 ) . For example , the frequency principle ( Xu et al. , 2019 , 2020 ) states that DNNs often fit target functions from low to high frequencies during the training . In Zhang et al . ( 2021 ) , we make a comprehensive extension of this conference paper . In the long paper , we provide a mathematical definition of the critical embedding and propose a new class of general compatible embeddings , which is a much wider class of critical embeddings than composition embeddings in this work . These general compatible embeddings provide much richer details about the geometry of critical submanifolds of DNN loss landscape . Note that the composition embedding technique is also studied in Fukumizu et al . ( 2019 ) and Simsek et al . ( 2021 ) . 3 Main results . In this section , we intuitively summarize our key theoretical results about the embedding principle and empirically demonstrate its relevance to practice , starting from proposing an embedding operation as follows . Rigorous theoretical description and proofs are presented in the latter sections . 3.1 Characteristics of embedding principle . Consider a neural network fθ ( x ) , where θ is the set of all network parameters , x ∈ Rd is the input . We summarize assumptions and provide definitions needed for all our results in this work below . Assumption . ( i ) L-layer ( L ≥ 2 ) fully-connected NN . ( ii ) Training data S = { ( xi , yi ) } ni=1 , n ∈ Z+ ∪ { +∞ } . ( iii ) Loss function RS ( θ ) = ES ` ( fθ ( x ) , y ) . ( iv ) Loss function and activation function are differentiable . Note that , even for functions like ReLU or hinge loss , as long as we uniquely assign a subgradient to their non-differentiable points , all our results still hold . Definition 1 ( critical point ) . Parameter vector θ is a critical point of the landscape of RS if ∇θRS ( θ ) = 0 . Definition 2 ( critical submanifold/affine subspace ) . A critical submanifold or affine subspaceM is a connected subsubmanifold or affine subspace of the parameter space RM , such that each θ ∈M is a critical point of loss with the same loss value . Definition 3 ( degree of degeneracy ) . The degree of degeneracy of point θ in the landscape of RS is the corank of Hessian matrix∇θ∇θRS , i.e. , number of the zero eigenvalues . Remark . In the above definition of degree of degeneracy , we require twice differentiable activation function and twice differentiable loss to compute Hessian for convenience . For loss and activation functions with only first-order differentiability , we extend the definition of degree of degeneracy as follows : for any critical point θ belonging to a K-dimensional critical submanifoldM , its degree of degeneracy is at least K. We first introduce one-step embedding intuitively , and leave the rigorous definition latter . As shown in Fig . 2 , an one-step embedding is performed by splitting any hidden neuron , say the black neuron in the left network , into two neurons colored in blue and purple in the right network . The input weights of the two splitted neurons are the same as the input weights of the original black neuron . Each output weight of the original black neuron is splitted into two parts of fraction α and ( 1− α ) ( α ∈ R , a hyperparameter ) , respectively . The multi-step embedding is the composition of multiple one-step embeddings . Since each one-step embedding can add one neuron to a selected layer , parameter of any NN can be embedded to the parameter space of any wider NN through a multi-step embedding . The multi-step embedding operation leads to the following property readily . Proposition ( one-step embedding preserves network properties , informal Prop . 1 ) . For any point θnarr of a DNN , a point θwide of a wider DNN obtained from θnarr by one-step embedding satisfies ( i ) fθnarr ( x ) = fθwide ( x ) for any x ; ( ii ) representation of the wide DNN at θwide , i.e. , the set of all different response functions of neurons , is the same as representation of the narrow DNN at θnarr . The most important property of this embedding is criticality preserving as follows . Theorem ( criticality preserving , informal Theorem 1 ) . For any critical point θnarr of a DNN , a point θwide of a wider DNN obtained from θnarr by multi-step embedding is a critical point . The embedding operation explains the cause of a type of degeneracy in the loss landscape . Theorem ( degeneracy of embedded critical points , informal Theorem 2 ) . If output weights of neurons in each layer of a DNN at a critical point θnarr are not all zero , then , for any K-neuron wider DNN , θnarr can be embedded to a K-dimensional critical affine subspace . Remark . By above theorem , each step of embedding of a critical point in general is accompanied by an increased degree of degeneracy . Therefore , degenerate critical points in general widely exist in the loss landscape of a DNN , and non-degenerate critical points are rare because they often become degenerate once embedded to a wider DNN . In previous studies , degeneracy is often considered as a consequence of over-parameterization depending on the size of training data n. Specifically , Cooper ( 2021 ) proves that the degree of degeneracy of global minima is m − n for 1-d output , where m is the number of network parameters . However , we demonstrate by the above theorem that regardless of whether the NN is over-parameterized , degenerate critical points are prevalent in its loss landscape as long as narrower DNNs possess critical points .

Summary: This paper studies critical points of loss landscape of deep neural networks. The authors prove that all critical points of a narrow fully-connected network can be embedded to critical points of a wider network. Moreover, the degree of degeneracy of critical points of narrow networks increases under the described embedding. The critical embedding principle provides a new explanation of the existence of high-dimensional manifolds of critical points in the loss landscape of wide fully connected networks.  The embedding of a weight vector of a narrow DNN into the weight space of a wider DNN is defined as a composition of simple one-step embedding steps: at each step one neuron of a network is split into two with a specifically chosen assignment of input/output weights to both old and new neurons. The input weights for both new neurons are copied from the original neuron and output weights are re-parameterized via linear interpolation. The authors show that this one-step embedding preserves the output function of the network, criticality of a weight vector in the loss landscape, and increases the degree of degeneracy of the critical point.  The authors support the theoretical results with experiments on simple learning tasks (1D regression, classification on Iris and MNIST dataset). The experiments confirm the theoretical results: the embedding procedure preserves criticality and increases degeneracy.