Embedding Principle of Loss Landscape of Deep Neural Networks
Year: 2022

Abstract:
Understanding the structure of loss landscape of deep neural networks (DNNs) is obviously important. In this work, we prove an embedding principle that the loss landscape of a DNN “contains” all the critical points of all the narrower DNNs. More precisely, we propose a critical embedding such that any critical point, e.g., local or global minima, of a narrower DNN can be embedded to a critical point/affine subspace of the target DNN with higher degeneracy and preserving the DNN output function. Note that, given any training data, differentiable loss function and differentiable activation function, this embedding structure of critical points holds. This general structure of DNNs is starkly different from other nonconvex problems such as protein-folding. Empirically, we find that a wide DNN is often attracted by highly-degenerate critical points that are embedded from narrow DNNs. The embedding principle provides a new perspective to study the general easy optimization of wide DNNs and unravels a potential implicit low-complexity regularization during the training. Overall, our work provides a skeleton for the study of loss landscape of DNNs and its implication, by which a more exact and comprehensive understanding can be anticipated in the near future.

heading: 1 Introduction
text: Understanding the loss landscape of DNNs is essential for a theory of deep learning. An important problem is to quantify exactly how the loss landscape looks like (E et al., 2020). This problem is difficult since the loss landscape is so complicated that it can almost be any pattern (Skorokhodov and Burtsev, 2019). Moreover, its high dimensionality and the dependence on data, model and loss make it very difficult to obtain a general understanding through empirical study. Therefore, though it has been extensively studied over the years, it remains an open problem to provide a clear picture about the organization of its critical points and their properties.
In this work, we make a step towards this goal through proposing a very general embedding operation of network parameters from narrow to wide DNNs, by which we prove an embedding principle for fully-connected DNNs stated intuitively as follows:
Embedding principle: the loss landscape of any network “contains” all critical points of all narrower networks.
∗Corresponding author: zhyy.sjtu@sjtu.edu.cn. †Part of this work is done when ZZ was an undergraduate student of Zhiyuan Honors Program at Shanghai
Jiao Tong University. ‡Corresponding author: xuzhiqin@sjtu.edu.cn.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
A “narrower network” means a DNN of the same depth but width of each layer no larger than the target DNN. The embedding principle slightly abuses the notion of “contain” since parameter space of DNNs of different widths are different. However, this inclusion relation is reasonable in the sense that, by our embedding operation, any critical point of any narrower network can be embedded to a critical point of the target network preserving its output function. Because of this criticality preserving property, we call this embedding operation the critical embedding.
We conclude our study by a “principle” since the embedding principle is a very general property of loss landscape of DNNs independent of the training data and choice of loss function, and is intrinsic to the layer-wise architecture of DNNs. In addition, the embedding principle is closely related to the training of DNNs. For example, as shown in Fig. 1(a), the training of a width-500 two-layer tanh NN experiences stagnation around the blue dot presumably very close to a saddle point, where the loss decreases extremely slowly. As shown in Fig. 1(b), we find that the DNN output at this blue point (red solid) is very close to the output of the global minimum (black dashed) of the width-1 NN, indicating that the underlying two critical points of two DNNs with different widths have the same output function conforming with the embedding principle. Importantly, this example shows that the training of a wide DNN can indeed experience those critical points from a narrow DNN unraveled by the embedding principle. Moreover, it demonstrates the potential of a transition from a local/global minimum of a narrow NN to a saddle point of a wide NN, which may be the reason underlying the easy optimization of wide NNs.
The embedding principle suggests an underlying mechanism to understand why heavily overparameterized DNNs often generalize well (Breiman, 1995; Zhang et al., 2017) as follows. Roughly, the overparameterized DNN has a large capacity, which seems contradictory to the conventional learning theory, i.e., learning by a model of large capacity easily leads to overfitting. The embedding principle shows that the optima of a wide network intrinsically may be embedded from an optima of a much narrower network, thus, its effective capacity is much smaller. For example, as illustrated in Fig. 1, training of a heavily overparametrized width-500 NN (vs. 50 training data) with small initialization first stagnated around a saddle presumably from width-1 NN and later converges to a global minimum presumably from width-3 NN, which clearly does not overfit. This implicit regularization effect unraveled by the embedding principle is consistent with previous works, such as low-complexity bias (Arpit et al., 2017; Kalimeris et al., 2019; Jin et al., 2020), low-frequency bias (Xu et al., 2019, 2020; Rahaman et al., 2019), and condensation phenomenon of network weights (Luo et al., 2021; Chizat and Bach, 2018; Ma et al., 2020).

heading: 2 Related works
text: The loss landscape of DNNs is complex and related to the generalization. Skorokhodov and Burtsev (2019) numerically show that the loss landscape can almost be any pattern. Keskar et al. (2017) visualize minimizers in a 1d slice and suggest that a flat minimizer generalizes better. Wu et al. (2017) find that the volume of basin of attraction of good minima may dominate over that of poor minima in practical problems. He et al. (2019) show that at a local minimum there exist many asymmetric directions such that the loss increases abruptly along one side, and slowly along the opposite side.
Degeneracy is also an important property of minima. Cooper (2021) shows that global minima is typically a high dimensional manifold for overparameterized DNNs. Sagun et al. (2016) empirically shows that Hessian of the minimizer obtained by the training has many zero eigenvalues. Under strong assumptions, Choromanska et al. (2015) shows minima tend to be highly degenerate. This work demonstrates wide existence of highly degenerate critical points, including local or global minima and saddle points, in the loss landscape by the embedding principle.
Lots of previous theoretical works focus on very wide DNNs, such as the phase diagram of two-layer ReLU infinite-width NNs (Luo et al., 2021), NTK regime (Jacot et al., 2018; Arora et al., 2019; Zhang et al., 2020; Du et al., 2019; Zou et al., 2018; Allen-Zhu et al., 2019; E et al., 2019), mean-field regime (Mei et al., 2018; Rotskoff and Vanden-Eijnden, 2018; Chizat and Bach, 2018; Sirignano and Spiliopoulos, 2020). By the embedding principle, this work demonstrate the loss landscape similarity between a moderate-width NN and a very wide NN, that they share a set of critical points embedded from that of narrower NNs. Therefore, results about infinite-width NNs could provide valuable insights about training of finite-width NNs used in practice.
The complexity of NN output increases during the training (Arpit et al., 2017; Xu et al., 2019, 2020; Rahaman et al., 2019; Kalimeris et al., 2019; Goldt et al., 2020; He et al., 2020; Mingard et al., 2019; Jin et al., 2020). For example, the frequency principle (Xu et al., 2019, 2020) states that DNNs often fit target functions from low to high frequencies during the training.
In Zhang et al. (2021), we make a comprehensive extension of this conference paper. In the long paper, we provide a mathematical definition of the critical embedding and propose a new class of general compatible embeddings, which is a much wider class of critical embeddings than composition embeddings in this work. These general compatible embeddings provide much richer details about the geometry of critical submanifolds of DNN loss landscape. Note that the composition embedding technique is also studied in Fukumizu et al. (2019) and Simsek et al. (2021).

heading: 3 Main results
text: In this section, we intuitively summarize our key theoretical results about the embedding principle and empirically demonstrate its relevance to practice, starting from proposing an embedding operation as follows. Rigorous theoretical description and proofs are presented in the latter sections.

heading: 3.1 Characteristics of embedding principle
text: Consider a neural network fθ(x), where θ is the set of all network parameters, x ∈ Rd is the input. We summarize assumptions and provide definitions needed for all our results in this work below.
Assumption. (i) L-layer (L ≥ 2) fully-connected NN. (ii) Training data S = {(xi,yi)}ni=1, n ∈ Z+ ∪ {+∞}. (iii)Loss function RS(θ) = ES`(fθ(x),y). (iv) Loss function and activation function are differentiable. Note that, even for functions like ReLU or hinge loss, as long as we uniquely assign a subgradient to their non-differentiable points, all our results still hold.
Definition 1 (critical point). Parameter vector θ is a critical point of the landscape of RS if ∇θRS(θ) = 0. Definition 2 (critical submanifold/affine subspace). A critical submanifold or affine subspaceM is a connected subsubmanifold or affine subspace of the parameter space RM , such that each θ ∈M is a critical point of loss with the same loss value.
Definition 3 (degree of degeneracy). The degree of degeneracy of point θ in the landscape of RS is the corank of Hessian matrix∇θ∇θRS , i.e., number of the zero eigenvalues. Remark. In the above definition of degree of degeneracy, we require twice differentiable activation function and twice differentiable loss to compute Hessian for convenience. For loss and activation functions with only first-order differentiability, we extend the definition of degree of degeneracy as follows: for any critical point θ belonging to a K-dimensional critical submanifoldM, its degree of degeneracy is at least K.
We first introduce one-step embedding intuitively, and leave the rigorous definition latter. As shown in Fig. 2, an one-step embedding is performed by splitting any hidden neuron, say the black neuron in the left network, into two neurons colored in blue and purple in the right network. The input weights of the two splitted neurons are the same as the input weights of the original black neuron. Each output weight of the original black neuron is splitted into two parts of fraction α and (1− α) (α ∈ R, a hyperparameter), respectively. The multi-step embedding is the composition of multiple one-step embeddings. Since each one-step embedding can add one neuron to a selected layer, parameter of any NN can be embedded to the parameter space of any wider NN through a multi-step embedding. The multi-step embedding operation leads to the following property readily. Proposition (one-step embedding preserves network properties, informal Prop. 1). For any point θnarr of a DNN, a point θwide of a wider DNN obtained from θnarr by one-step embedding satisfies (i) fθnarr(x) = fθwide(x) for any x; (ii) representation of the wide DNN at θwide, i.e., the set of all different response functions of neurons, is the same as representation of the narrow DNN at θnarr.
The most important property of this embedding is criticality preserving as follows. Theorem (criticality preserving, informal Theorem 1). For any critical point θnarr of a DNN, a point θwide of a wider DNN obtained from θnarr by multi-step embedding is a critical point.
The embedding operation explains the cause of a type of degeneracy in the loss landscape. Theorem (degeneracy of embedded critical points, informal Theorem 2). If output weights of neurons in each layer of a DNN at a critical point θnarr are not all zero, then, for any K-neuron wider DNN, θnarr can be embedded to a K-dimensional critical affine subspace. Remark. By above theorem, each step of embedding of a critical point in general is accompanied by an increased degree of degeneracy. Therefore, degenerate critical points in general widely exist in the loss landscape of a DNN, and non-degenerate critical points are rare because they often become degenerate once embedded to a wider DNN.
In previous studies, degeneracy is often considered as a consequence of over-parameterization depending on the size of training data n. Specifically, Cooper (2021) proves that the degree of degeneracy of global minima is m − n for 1-d output, where m is the number of network parameters. However, we demonstrate by the above theorem that regardless of whether the NN is over-parameterized, degenerate critical points are prevalent in its loss landscape as long as narrower DNNs possess critical points.

heading: 3.2 Numerical experiments
text: Experimental setup. Throughout this work, we use two-layer fully-connected neural network with size d-m-dout. The input dimension d is determined by the training data. The output dimension dout is different for different experiments. The number of hidden neurons m is specified in each experiment. All parameters are initialized by a Gaussian distribution with mean zero and variance specified in each experiment. We use MSE loss trained by full batch gradient descent for 1D fitting
problems (Figs. 1, 3(a) and 4), and default Adam optimizer with full batch for others. The learning rate is fixed throughout the training. More details of experiments are shown in Appendix B.
Increment of degeneracy through embedding. We train a two-layer NN of width msmall = 2 to learn data of Fig. 1 shown in Fig. 3(a) or Iris dataset (Fisher, 1936) in Fig. 3(b) to a critical point. We first roughly estimate the possible interval of critical points by observing where the loss decays very slowly, and then take the point with the smallest derivative of the parameters (use L1 norm) as an empirical critical point. The L1 norm of the derivative of loss function at the empirical critical point is approximately 7.15× 10−15 for Fig. 3(a) and 3.72× 10−13 for Fig. 3(b), which are reasonably small. We then embed this critical point to networks of width m = 3 and m = 4 through an one-step or a two-step embedding, respectively. It is obvious from Fig. 3 that each step of embedding incurs one more zero eigenvalue in the Hessian matrix, which conforms with Theorem 2. Moreover, in Fig. 3(a), for m = 2, all eigenvalues are positive (red) indicating the critical point obtained by training is a local or global minimum. After embedding, this point becomes a saddle due to the emergence of negative eigenvalues (blue). Specifically, in both Fig. 3(a) and (b), we observe a steady increase of significant negative eigenvalues, e.g., from 0 to 1 to 2 in (a) and from 3 to 5 to 7 in (b), which implies reduced difficulty in escaping from the corresponding critical point in a wider NN during the training.
Empirical diagram of loss landscape. In Fig. 4, we present an empirical diagram of loss landscape of a width-3 two-layer tanh DNN to visualize a set of its critical points predicted by the embedding principle, i.e., critical points embedded from network of width-1 or -2 respectively as well as critical points that cannot be obtained through embedding. Through the training of width-1, -2, -3 network respectively on the training data presented in Fig. 1 for multiple trials, we discover 1 critical point for width-1 network, 2 critical points for width-2 network and 1 critical point for width-3 network that cannot be embedded from a narrower NN. Then, embedding all these four critical points to critical points/affine subspaces of loss landscape of the width-3 network, we obtain four sets of critical points with their loss values, output functions, degrees of degeneracy and width of network they embedded from illustrated in Fig. 4. This diagram immediately tells us what attracts the gradient-based training trajectory for a width-3 network. Specifically, if stagnation happens during the training, this diagram informs us the potential loss values and output functions at stagnation, which could help us better understand the nonlinear training process of not only a width-3 network but also much wider networks due to the embedding principle. Furthermore, as illustrated in Fig. 1 for the training process of a 500-neuron NN, saddle points of a wide NN, effectively local or global minima of narrow NNs, composes a trajectory, which may serve as a compass for achieving a global minimum from narrow NNs of low complexity.
Reduction of DNN at critical point. The embedding principle predicts a class of critical points of a NN embedded from much narrower NNs. At such a critical point, we shall be able to find neuron groups, within which neurons have similar orientation of input weights presumably originated from the same neuron of a narrow NN through embedding. This prediction is confirmed by the following experiment in Fig. 5. We train a width-400 two-layer ReLU NN fθ = ∑m k=1 akσ(w T k x̃)
empirical diagram of loss landscape
(x̃ = [xᵀ, 1]ᵀ) on 1000 training samples of the MNIST dataset with small initialization. At the blue dot in Fig. 5(a), the loss decreases very slowly, presumably very close to a saddle point. We then examine the orientation similarity between each pair of neuron input weights by computing the inner product of two normalized input weight. As shown in Fig. 5(b), there emerge 58 groups of neurons (neurons with very small amplitudes are neglected and later directly removed), where similarity between input weights in the same group is at least 0.9. For each group Ssimilar, we randomly select a neuron j, replace its output weight by ∑ k∈Ssimilar ak‖wk‖2/‖wj‖2, and discard all other neurons in the group. The parameter set before reduction is denoted by θori, and after reduction by θredu. Width of the NN is reduced from 400 to 58. We train the reduced NN from θredu as shown in Fig. 5, which stagnated after a few steps at the same loss value as the blue point in Fig. 5(a) marked by the blue dash and represented by the blue point in Fig. 5(c). We then compare the prediction between original model and the reduced model at the corresponding blue points on 10000 test data as shown in Fig.5(d). For each grid, color indicates the frequency of that prediction pair. Specifically, the highlight of diagonal element indicts high prediction agreement of two models (overall ∼ 98.5%). Therefore, this critical point of the reduced width-58 NN well matches the critical point of the original width-400 NN, clearly demonstrating the relevance of our embedding principle to real dataset training.

heading: 4 Preliminaries

heading: 4.1 Deep Neural Networks
text: Consider L-layer (L ≥ 2) fully-connected DNNs with a general differentiable activation function. We regard the input as the 0-th layer and the output as the L-th layer. Let ml be the number of neurons in the l-th layer. In particular, m0 = d and mL = d′. For any i, k ∈ N and i < k, we denote [i : k] = {i, i + 1, . . . , k}. In particular, we denote [k] := {1, 2, . . . , k}. Given weights W [l] ∈ Rml×ml−1 and bias b[l] ∈ Rml for l ∈ [L], we define the collection of parameters θ as a 2L-tuple (an ordered list of 2L elements) whose elements are matrices or vectors
θ = ( θ|1, · · · ,θ|L ) = ( W [1], b[1], . . . ,W [L], b[L] ) . (1)
where the l-th layer parameters of θ is the ordered pair θ|l = ( W [l], b[l] ) , l ∈ [L]. We may misuse
of notation and identify θ with its vectorization vec(θ) ∈ RM with M = ∑L−1 l=0 (ml + 1)ml+1.
Given θ ∈ RM , the neural network function fθ(·) is defined recursively. First, we write f [0]θ (x) = x for all x ∈ Rd. Then for l ∈ [L−1], f [l]θ is defined recursively as f [l] θ (x) = σ(W [l]f [l−1] θ (x)+b
[l]). Finally, we denote
fθ(x) = f(x,θ) = f [L] θ (x) =W [L]f [L−1] θ (x) + b [L]. (2)
For notational simplicity, we may drop the subscript θ in f [l]θ , l ∈ [0 : L]. We introduce the following notions for the convenience of the presentation in this paper.
Definition 4 (Wider/narrower DNN). We write NN({ml}Ll=0) for a fully-connected neural network with width (m0, . . . ,mL). Given twoL-layer (L ≥ 2) fully-connected neural networks NN({ml}Ll=0) and NN′({m′l}Ll=0), if m′0 = m0, m′L = mL, and for any l ∈ [L − 1], m′l ≥ ml and K =∑L−1 l=1 (m ′ l −ml) ∈ N+, then we say that NN
′({m′l}Ll=0) is K-neuron wider than NN({ml}Ll=0) and NN({ml}Ll=0)K-neuron narrower than NN ′({m′l}Ll=0).

heading: 4.2 Loss function
text: The training data set is denoted as S = {(xi,yi)}ni=1, where xi ∈ Rd, yi ∈ Rd ′ . For simplicity, here we assume an unknown function y satisfying y(xi) = yi for i ∈ [n]. The empirical risk reads as
RS(θ) = 1
n n∑ i=1 `(f(xi,θ),y(xi)) = ES`(f(x,θ),y). (3)
where the expectation ESh(x) := 1n ∑n i=1 h(xi) for any function h : Rd → R and the loss function `(·, ·) is differentiable and the derivative of `with respect to its first argument is denoted by∇`(y,y∗). Generally, we always take derivatives/gradients of ` in its first argument with respect to any parameter. We consider gradient flow ofRS as the training dynamics, i.e., dθ/dt = −∇θRS(θ) with θ(0) = θ0.
We define the error vectors z[l]θ = ∇f [l]` for l ∈ [L] and the feature gradients g [L] θ = 1 and g [l] θ = σ (1) ( W [l]f [l−1] θ + b [l] )
for l ∈ [L − 1]. Here σ(1) is the first derivative of σ. We call f [l]θ , l ∈ [L] feature vectors. The collections of feature vectors, feature gradients, and error vectors are Fθ = {f [l]θ }Ll=1,Gθ = {g [l] θ }Ll=1,Zθ = {z [l] θ }Ll=1. Using backpropagation, we can calculate the gradients as follows
z [L] θ = ∇`, z [l] θ = (W [l+1])ᵀz [l+1] θ ◦ g [l+1] θ , l ∈ [L− 1],
∇W [l]` = z [l] θ ◦ g [l] θ (f [l−1] θ ) ᵀ, ∇b[l]` = z [l] θ ◦ g [l] θ , l ∈ [L].
Here we use ◦ for the Hadamard product of two matrices of the same dimension.

heading: 5 Critical embedding
text: We introduce the one-step embedding for the DNNs which will lead us to general embeddings. Definition 5 (one-step embedding). Given a L-layer (L ≥ 2) fully-connected neural network with width (m0, . . . ,mL) and network parameters θ = (W [1], b[1], · · · ,W [L], b[L]) ∈ RM , for any l ∈ [L− 1] and any s ∈ [ml], we define the linear operators Tl,s and Vl,s applying on θ as follows Tl,s(θ)|k = θ|k, k 6= l, l + 1,
Tl,s(θ)|l =
([ W [l]
W [l] s,[1:ml−1]
] , [ b[l]
b [l] s
]) , Tl,s(θ)|l+1 = ([ W [l+1],0ml+1×1 ] , b[l+1] ) ,
Vl,s(θ)|k = ( 0mk×mk−1 ,0mk×1 ) , k 6= l, l + 1,
Vl,s(θ)|l = ( 0(ml+1)×ml−1 ,0(ml+1)×1 ) ,
Vl,s(θ)|l+1 = ([
0ml+1×(s−1),−W [l+1] [1:ml+1],s ,0ml+1×(ml−s),W [l+1] [1:ml+1],s ] ,0ml+1×1 ) .
Then the one-step embedding operator T αl,s is defined as for any θ ∈ RM
T αl,s(θ) = (Tl,s + αVl,s)(θ).
Note that the resulting parameter T αl,s(θ) corresponds to a L-layer fully-connected neural network with width (m0, . . . ,ml−1,ml + 1,ml+1, . . . ,mL).
An illustration of Tl,s, Vl,s, and T αl,s can be found in Fig. S1 in Appendix. Lemma 1. Given a L-layer (L ≥ 2) fully-connected neural network with width (m0, . . . ,mL), for any network parameters θ = (W [1], b[1], · · · ,W [L], b[L]) and for any l ∈ [L − 1], s ∈ [ml], we have the expressions for θ′ := T αl,s(θ) (i) feature vectors in Fθ′ : f [l′] θ′ = f [l′] θ , l ′ 6= l and f [l]θ′ = [ (f [l] θ ) ᵀ, (f [l] θ )s ]ᵀ ;
(ii) feature gradients inGθ′ : g [l′] θ′ = g [l′] θ , l ′ 6= l and g[l]θ′ = [ (g [l] θ ) ᵀ, (g [l] θ )s ]ᵀ ;
(iii) error vectors in Zθ′ : z [l′] θ′ = z [l′] θ , l
′ 6= l and z[l]θ′ = [ (z [l] θ ) ᵀ [1:s−1], (1− α)(z [l] θ )s, (z [l] θ ) ᵀ [s+1:ml] , α(z [l] θ )s ]ᵀ .
An illustration of Fθ and Zθ can be found in Fig. S2 in Appendix. Proposition 1 (one-step embedding preserves network properties). Given a L-layer (L ≥ 2) fully-connected neural network with width (m0, . . . ,mL), for any network parameters θ = (W [1], b[1], · · · ,W [L], b[L]) and for any l ∈ [L− 1], s ∈ [ml], the following network properties are preserved for θ′ = T αl,s(θ):
(i) output function is preserved: fθ′(x) = fθ(x) for all x;
(ii) empirical risk is preserved: RS(θ′) = RS(θ); (iii) the sets of features are preserved: {( f [l] θ′ ) i } i∈[ml+1] = {( f [l] θ ) i } i∈[ml]
and{( f [l′] θ′ ) i } i∈[ml′ ] = {( f [l′] θ ) i } i∈[ml′ ] for l′ ∈ [L]\{l}; Theorem 1 (criticality preserving). Given a L-layer (L ≥ 2) fully-connected neural network with width (m0, . . . ,mL), for any network parameters θ = (W [1], b[1], · · · ,W [L], b[L]) and for any l ∈ [L− 1], s ∈ [ml], if∇θRS(θ) = 0, then∇θRS(θ′) = 0. Lemma 2 (increment of the degree of degeneracy). Given a L-layer (L ≥ 2) fully-connected neural network with width (m0, . . . ,mL), if there exists l ∈ [L− 1], s ∈ [ml], and a q-dimensional manifoldM consisting of critical points of RS such that for any θ ∈ M, W [l+1][1:ml+1],s 6= 0, then M′ := {T αl,s(θ)|θ ∈ M, α ∈ R} is a (q + 1)-dimensional manifold consists of critical points for the corresponding L-layer fully-connected neural network with width (m0, . . . ,ml−1,ml + 1,ml+1, . . . ,mL).
Theorem 2 (degeneracy of embedded critical points). Consider two L-layer (L ≥ 2) fullyconnected neural networks NNA({ml}Ll=0) and NNB({m′l}Ll=0) which is K-neuron wider than NNA. Suppose that the critical point θA = (W [1], b[1], · · · ,W [L], b[L]) satisfiesW [l] 6= 0 for each layer l ∈ [L]. Then the parameters θA of NNA can be critically embedded to a K-dimensional critical affine subspace MB = {θB + ∑K i=1 αivi|αi ∈ R} of loss landscape of NNB . Here
θB = ( ∏K i=1 Tli,si)(θA) and vi = TlK ,sK · · · Vli,si · · · Tl1,s1θA.
Note that neuron-index permutation among the same layer is a trivial criticality invariant transform. More discussions about it, specifically for NNs of homogeneous activation functions like ReLU, can be found in Section A.1 in Appendix.

heading: 6 Conclusion and discussion
text: In this work, we prove an embedding principle that loss landscape of a DNN contains all critical points of all the narrower DNNs. This embedding principle unravels wide existence of highly degenerate critical points with low complexity in the loss landscape of a wide DNN, i.e., critical points with low-complexity output function and degenerate Hessian matrix, embedded from critical points of narrow DNNs. With such a loss landscape of DNN, the gradient-based training has the potential of getting attracted or even converging to a low complexity critical point as confirmed by above numerical experiments, which implies a potential implicit regularization towards low-complexity function of nonlinear DNN training dynamics.
Moreover, through critical embedding, a critical point in form of a common non-degenerate local minimum of a narrow DNN not only becomes degenerate in general, but also may become a saddle point as illustrated by numerical experiments. This may be the reason underlying the general easy optimization of wide DNNs observed in practice even beyond the linear/kernel/NTK regime (Chen et al., 2020; Trager et al., 2019; Geiger et al., 2020; Fort et al., 2020; Luo et al., 2021). We will perform more detailed analysis as well as numerical experiments specifically about this minimum-to-saddle transition later.
At the essence, the embedding principle results from the layer structure of a neural network model, which allows arbitrary neuron addition, input weight copying and output weight splitting within each layer. Therefore, though results in this work assume fully-connected NNs, these can be easily extended to other DNN architectures. Considering convolutional neural networks for example, the quantity that corresponds to width of fully-connected NNs is channel. Similar to one-step or multistep embedding, we can introduce a feature splitting operation, i.e., increase the number of channels by splitting all neurons sharing one convolution kernel with the same α, which can be proven to preserve the output function, representation as well as the criticality. Thereby, embedding principle holds in a sense that loss landscape of any CNN contains all critical points of all narrower CNNs whose number of channels in each layer is no more than that of the target CNN. Currently, depth serves as a preset hyperparameter in our analysis. Whether loss landscape of DNNs of different depth has certain embedding relation for specific DNN architectures such as ResNet is an interesting open problem.
Our embedding principle and experiments in Figs. 1 and 5 suggest that whenever training of a wide DNN is stagnated around a critical point, it potentially is embedded from a much narrower DNN. Therefore, many neurons with similar representation can be reduced to one neuron. How we can design efficient pruning algorithm to fully realize this potential and how it is related to existing pruning methods as well as the well-known lottery ticket hypothesis (Frankle and Carbin, 2018) are important problems for our future research.
We remark that our embedding principle applies for landscape of general loss functions. Although for loss functions like cross entropy, a meaningful finite critical point may not exist as its parameters diverge in general throughout the training, yet it is reasonable to expect that critical embedding may provide us certain approximate critical points from narrow NNs. Of course, how to properly define an approximate critical point is in itself a problem of interest. And we leave this problem for the future study.
Overall, our embedding principle provides the first clear picture about the general structure of critical points of DNN loss landscape, which is fundamental to the theoretical understanding of both training and generalization behavior of DNNs as well as the design of optimization algorithms. Of course, the
study of loss landscape of DNN is far from complete. This work serves as a starting point for a novel line of research, which finally leads to an exact and comprehensive theoretical description about loss landscape of DNNs as well as an understanding of its profound impact on training and generalization.
Acknowledgments and Disclosure of Funding
This work is sponsored by the National Key R&D Program of China Grant No. 2019YFA0709503 (Z. X.), the Shanghai Sailing Program, the Natural Science Foundation of Shanghai Grant No. 20ZR1429000 (Z. X.), the National Natural Science Foundation of China Grant No. 62002221 (Z. X.), the National Natural Science Foundation of China Grant No. 12101401 (T. L.), the National Natural Science Foundation of China Grant No. 12101402 (Y. Z.), Shanghai Municipal of Science and Technology Project Grant No. 20JC1419500 (Y.Z.), Shanghai Municipal of Science and Technology Major Project No. 2021SHZDZX0102, and the HPC of School of Mathematical Sciences and the Student Innovation Center at Shanghai Jiao Tong University.

Review summaries:
The paper discusses the question of organization of loss surfaces of neural networks, in particular the connection between wider and narrower networks with respect to the critical points. The proposed analysis introduces an embedding that projects a narrower network into a wider network (at least by one neuron in at least one hidden layer) without changing its function (output) and also saving all its critical points. It is shown, that the introduced procedure of the mapping narrower networks into wider increases degeneracy of the critical points, i.e., increasing amount of zero valued eigenvalues.

This paper studies critical points of loss landscape of deep neural networks. The authors prove that all critical points of a narrow fully-connected network can be embedded to critical points of a wider network. Moreover, the degree of degeneracy of critical points of narrow networks increases under the described embedding. The critical embedding principle provides a new explanation of the existence of high-dimensional manifolds of critical points in the loss landscape of wide fully connected networks.  The embedding of a weight vector of a narrow DNN into the weight space of a wider DNN is defined as a composition of simple one-step embedding steps: at each step one neuron of a network is split into two with a specifically chosen assignment of input/output weights to both old and new neurons. The input weights for both new neurons are copied from the original neuron and output weights are re-parameterized via linear interpolation. The authors show that this one-step embedding preserves the output function of the network, criticality of a weight vector in the loss landscape, and increases the degree of degeneracy of the critical point.  The authors support the theoretical results with experiments on simple learning tasks (1D regression, classification on Iris and MNIST dataset). The experiments confirm the theoretical results: the embedding procedure preserves criticality and increases degeneracy.