Embedding Principle of Loss Landscape of Deep Neural Networks
Authors: Yaoyu Zhang
Zhongwang Zhang
Tao Luo
Zhiqin Xu
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: deep neural network
embedding principle
loss landscape
critical points
degree of degeneracy

Abstract: Understanding the structure of loss landscape of deep neural networks (DNNs) is obviously important. In this work, we prove an embedding principle that the loss landscape of a DNN "contains" all the critical points of all the narrower DNNs. More precisely, we propose a critical embedding such that any critical point, e.g., local or global minima, of a narrower DNN can be embedded to a critical point/affine subspace of the target DNN with higher degeneracy and preserving the DNN output function. Note that, given any training data, differentiable loss function and differentiable activation function, this embedding structure of critical points holds.This general structure of DNNs is starkly different from other nonconvex problems such as protein-folding.Empirically, we find that a wide DNN is often attracted by highly-degenerate critical points that are embedded from narrow DNNs. The embedding principle provides a new perspective to study the general easy optimization of wide DNNs and unravels a potential implicit low-complexity regularization during the training.Overall, our work provides a skeleton for the study of loss landscape of DNNs and its implication, by which a more exact and comprehensive understanding can be anticipated in the near future. 

Embedding Principle of Loss Landscape of Deep Neural Networks
Zhongwaan Zhang' Tao Luo', Zhi-Qin John Xu't School of Mathematical Sciences, Institute of Natural Sciences, MOE-LSC and Qing Yuan Research Institute, Shanghai Jiao Tong University Shanghai Center for Brain Science and Brain-Inspired Technology /zhyy.sjtu, 0123zzw666, luotao41, xuzhiqin ) @ @jjtu.ddu.cn
Yaoyu 2 Zhongwang
Abstract
Understanding the structure of loss landscape of deep neural networks (DNNs) is obviously important. In this work, we prove embedding principle that the loss landscape of a DNN "contains" all the critical points of all the narrower DNNs. More precisely, point, e.g., local or global minima, of a narrower DNN can be embedded to a critical point/affine subspace of the target DNN with higher degeneracy and preserving the DNN output function. Note that, given any training data, differentiable loss function and diffecrentiable activation function, this embedding structure of critical points holds. This general structure of DNNs is starkly different from other nonconvex problems such as prodein-folding. Empirically, DNN is often attracted by highly-degenerate critical points that are embedded from narrow DNNs. The embedding principle provides a new perspective to study the general easy optimization of wide DNNs and unravels a potential implicit low-complexity regularization during the training. Overall, our work provides a skeleton for the study of loss landscape of DNNs and its implication, by which a more exact and comprehensive understanding can be anticipated in the near future.
propose critical embedding such that any critical
find that wide
Introduction
Understanding the loss landscape of DNNs is essential for theory of deep learning. An important problem is quantify exactly how the loss landscape looks like et al. 2020). This problem is difficult since the loss landscape is so complicated that it can almost be any pattern (Skorokhodoy and Burtsev 2019). Moreover, its high dimensionality and the dependence on data, model and loss make it very difficult to obtain general understanding through empirical study. Therefore, though it has been extensively studied over the years, remains an open problem to provide clear picture about the organization of its critical points and their properties.
In this work, we make step towards this goal through proposing very general embedding operation of network parameters from narrow to wide DNNs, by which we prove an embedding principle for fully-comnected DNNs stated intuitively as follows:
Embedding principle: the loss landscape of any network "contains" " all critical points of all narrower networks.
"Corespponiing author: zhyy.sjtu @ajjjuisttuedd.cnn
Part this work is done when ZZ was an undergraduate student Zhiyuan Honors Program at Shanghai Jiao Tong University.
1 CCoresponning author; xuzhiqin @ sjt..cdu.cnn
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

A "narrower network" means DNN of the same depth but width of each layer no larger than the target DNN. The embedding principle slightly abuses the notion of "contain" since parameter space of DNNs of different widths are different. However, this inclusion relation is reasonable in the sense that, by our embedding operation, any critical point of any narrower network can be embedded to a critical point of the target network preserving its output function. Because of this criticality preserving property, we call this embedding operation the critical embedding.
We conclude our study by "principle" since the embedding principle is a very general property of loss landscape of DNNs independent of the training data and choice of loss function, and is intrinsic to the layer-wise architecture of DNNs. In addition, the embedding principle is closely related to the training of DNNs. For example, as shown in Fig. aaa, the training of width-500 two-layer tanh NN experiences stagnation around the blue dot presumably very close to a saddle point, where the loss decreases extremely slowly. As shown Fig. [b), we find that the DNN output at this blue point (red solid) is very close to the output of the global minimum (black dashed) of the width-1 1 NN, indicating that the underlying two critical points of two DNNs with different widths have the same output function conforming with the embedding principle. Importantly, this example shows that the training of a wide DNN can indeed experience those critical points from a narrow DNN unraveled by the embedding principle. Moreover, it demonstrates the potential of transition from local/global minimum of a narrow NN to a saddle point of a wide NN, which may be the reason underlying the easy optimization of wide NNs.
The embedding principle suggests an underlying mechanism to understand why heavily overparame- terized DNNs often generalize well (Breiman, 1995; Zhang et al. (017) as follows. Roughly, the overparameterized DNN has a large capacity, which seems contradictory to the conventional learning theory, i.e., learning by a model of large capacity easily leads to overfitting. The embedding principle shows that the optima of a wide network intrinsically may be embedded from an optima of a much narrower network, thus, its effective capacity is much smaller. For example, as illustrated in Fig. D training of heavily overparametrized width-500 NN (vs. 50 training data) with small initialization first stagnated around a saddle presumably from w NN and later converges to a global minimum presumably from width-3 NN, which clearly does not overfit. This implicit regularization effect unraveled by the embedding principle is consistent with previous works, such low-complexity bias (Arpit et al., [2177] Kalimeris et al. 2019) Jin et al. 2020), low- www-freuency bias (Xu et al. 2019 2020
Rahaman al. (0199), and condensation phenomenon of network weights (Luo et al.
2021
Chizat
and Bach 2018: Ma et al. 2020).
Midh wid.1-1 trwinin dat
1.0
widlhnd
TOW ming dats
10
10--
training
20-3
10.t
-30
10
110
escch
(a)
(b)
(c)
Figure 1: (a) The training loss of two-layer tanh neural network with 500 hidden neurons. (b) (c) Red solid: the DNN output at training step indicated by (b) the blue dot or (c) the orange dot in (a); Black dashed: the output of the global minimum of (b) width-1 DNN or (c) width-3 DNN, respectively; Blue dots: training data.
2
Related works
The loss landscape of DNNs is complex and related to the generalization. [Skorokhodov and Burtsev (2019) numerically show that the loss landscape can almost be any pattern. Keskarr et al. (2017) visualize minimizers a 1d slice and suggest that a flat minimizer generalizes better. Wu et al. (2017) find that the volume of basin of attraction of good minima may dominate over that of poor minima in practical problems. He et al. (2019) show that at local minimum there exist many asymmetric
directions such that the loss increases abruptly along one side, and slowly along the opposite side.
2

---

Degeneracy is also an important property of minima. Cooper (2021) shows that global minima is typically a high dimensional manifold for overparameterized DNNs. Sagun et al. (2016) empirically shows that Hessian of the minimizer obtained by the training has many zero eigenvalues. Under strong assumptions, Choromanska et al. (2015) shows minima tend to be highly degenerate. This work demonstrates wide existence of highly degenerate critical points, including local or global minima and saddle points, in the loss landscape by the embedding principle.
Lots of previous theoretical works focus on very wide DNNs, such as the phase diagram of two-layer
ReLU infinite-wide NNs (Luo et al., (02T). NTK regime (Jacot al. 2018 Arora et al. 2019
Zhang et al., 2020 Du et al. [2199] Zou et al. 20188) Allen-Zhu et al. 2019 E et al. 2019)), mean- mean-field
regime (Mei al. 2018 Rotskoff and Vanden -iinden--iddenaaae 20188 Chizat and Bach, 2018: Sirignano
and Spiliopoulos 2020). By the embedding principle, this work demonstrate the loss landscape similarity between a moderate-widtt NN and a very wide NN, that they share set of critical points
embedded from that of narrower NNs. Therefore, results about anfinite-width NNs could provide valuable insights about training of finite-width NNs used in practice.
The complexity of NN output increases during the training (Arpit et al. 2017 Xu et al.
2019
2020
Rahaman al. 2019 Kalimeris al., 2019; Goldt et al. [020;] He
[2200] Mingard et al. 2019
Jin et al. 2020). For example, the frequency principle (Xu et al. 2019 2020) states that DNNs often
fit target functions from low high frequencies during the training.
In Zhang et al. (202I), we make a comprehensive extension of this conference paper. In the long paper, we provide mathematical definition of the critical embedding and propose new class of general compatible embeddings, which is a much wider class of critical embeddings than composition embeddings in this work. These general compatible embeddings provide much richer details about the geometry of critical submanifolds of DNN loss landscape. Note that the composition embedding technique is also studied in Fukumizu et al. (2019) and Simsek al. (202I)
3
Main results
In this section, we intuitively summarize our key theoretical results about the embedding principle and empirically demonstrate its relevance to practice, starting from proposing an embedding operation as follows. Rigorous theoretical description and proof' are presented in the latter sections.
3.1 Characteristics of embedding principle
Consider neural network f0(x), where 0 is the set of all network parameters, æ E Rd is the input. We summarize assumptions and provide definitions needed for all our results in this work below.
Assumption. (i) L-layer [L 2 2) fully fully-ooneetedd NN.
(ii) Training data S = {xx; y.) n Z Z (+0o).
Loss function R2s = Eg((S Es((fo(z))
(iv) Loss function and activation fanction are differentiable. Note that, even for functions like ReLU or hinge loss, as long as we uniquely assign subgradient to their non-difeerentialle points, all our results still hold.
Definition 1 (critical point). Parameter vector is a critical point of the landscape of Rş if
N Ro) =0)=
Definition 2 (critical submanifold/afiie subspace). A critical submanifold or affine subspace M
is a connected subsubmanifold or affine subspace of the parameter space R such that each 0 E M a critical point of loss with the same loss value.
Definition 3 (degree of degeneracy). The degree of degeneracy of point 0 in the landscape of Rs is the corank of Hessian matrix VoRS. i.e., number of the zero eigenvalues.
Remark. In the above definition of degree of degeneracy, we require twice differentiable activation function and twice differentiable loss to compute Hessian for convenience. For loss and activation functions with only first-order differentiability, we extend the definition of degree of degeneracy as follows: for any critical point 0 belonging to a K K-dimensional critical submanifold M, its degree of degeneracy is at least K.
3

---

split
1+1
one-step
embedding
1-1
Illustration of one-step embedding. The black neuron in the left network is splitted into the blue and purple neurons the right network. The red (green) output weight of the black neuron the left net is splitted into two red (green) weights in the right net with ratio and (1 ax), respectively.
Figure 2:
We first introduce one-step embedding intuitively, and leave the rigorous definition latter. As shown in Fig. 2 an one-step embedding is performed by splitting any hidden neuron, say the black neuron in the left network, into two neurons colored in blue and purple the right network. The input weights of the two splitted neurons are the same as the input weights of the original black neuron. Each output weight of the original black neuron splitted into two parts of fraction a and (1 - a) (a E R. hyperparameter), respectively. The multi-step embedding is the composition of multiple one-step embeddings. Since each one-step embedding can add one neuron to a selected layer, parameter of any NN can be embedded to the parameter space of any wider NN through multi-step embedding. The multi-step embedding operation leads to the following property readily.
Proposition (one-step embedding preserves network properties, informal Prop. D. For any
point fuae of DNN, a point wide of wider DNN obtained from Oum by one-step embedding satisfies
(i) (x) Sews (x) for any (ii) representation of the wide DNN at Owides i.e. the set of all different response functions of neurons, is the same as representation of the narrow DNN at O
The most important property of this embedding is criticality preserving as follows.
Theorem (criticality preserving, informal Theorem [] For any critical point O of DNN, point wide of a wider DNN obtained from O 0 by multi-step embedding is critical point.
The embedding operation explains the cause of type of degeneracy the loss landscape.
Theorem (degeneracy of embedded critical points, informal Theorem 2). If output weights of neurons in each layer of a DNN at a critical point 6 are not all zero, then, for any K-neuron wider K K-dimensional critical affine subspace.
DNN, f be embedded
Remark. By above theorem, each step of embedding of critical point in general is accompanied by an increased degree of degeneracy. Therefore, degenerate critical points in general widely exist in the loss landscape of a DNN, and non-degenerate critical points are rare because they often become degenerate once embedded to a wider DNN.
In previous studies, degeneracy is often considered as consequence of preetttraatttinntioonnnn depend- ing on the size of training data n. Specifically, Coopeer (202T) proves that the degree degeneracy of global minima is m for 1-d output, where is the number of network parameters. However, we demonstrate by the above theorem that regardless of whether the NN over- wver-parameterized, degenerate critical points prevalent in its loss landscape long narrower DNNs possess critical points.
3.2 Numerical experiments
Experimental setup. Throughout this work, we use two-layer fully-connected neural network with size d-m-dout- The input dimension determined by the training data. The output dimension dmnt is different for different experiments. The number of hidden neurons m is specified in each experiment. All parameters are initialized by a Gaussian distribution with mean zero and variance specified in each experiment. We use MSE loss trained by full batch gradient descent for ID fitting

---

problems (Figs. 3 a) and 4. and default Adam optimizer with full batch for others. The learning rate is fixed throughout the training. More details of experiments are shown in Appendix B.
Increment of degeneracy through embedding. We train two-layer NN of width Memall 2 to learn data of Fig. [ shown in Fig. Bfa) or Iris dataset (Fisher 1936) in Fig. b) to a critical point. We first roughly estimate the possible interval of critical points by observing where the loss decays very slowly, and then take the point with the smallest derivative of the parameters (use L1 norm) as an empirical critical point. The L1 norm of the derivative of loss function the empirical critical point is approximately 7.15 x 10-15 for Fig. 3/a) and 3.72 x 10-13 for Fig. (b), which are reasonably small. We then embed this critical point to networks of width m = and m through an one-step or two-step embedding. respectively. It is obvious from Fig. that each step of embedding incurs one more zero eigenvalue in the Hessian matrix, which conforms with Theorem 2 Moreover, in Fig. Bja) for m = 2. all eigenvalues are positive (red) indicating the critical point obtained by training is a local or global minimum. After embedding, this point becomes a saddle due to the emergence of negative cigenvalues (blue). Specifically, in both Fig. Bla) and (b), observe steady increase of significant negative cigenvalues, c.g., from 0 to to 2 in (a) and from 3 to 5 to in (b). which implies reduced difficulty in escaping from the corresponding critical point in wider NN during the training.
atstegg
10
3to/olgl abc/dgl
20-1
20-4
Index
Indox
(a) synthetic data
(b) Iris data
Figure 3: Eigenvalues of Hessian of NNs at the critical points embedded from the NN with width
Msmall = 2 for learning data of Fig. [ in (a) and for Iris dataset in (b). The value of m in each sub-figure is the NN width after embedding. The auxiliary dash line in each sub-figure is y = 10 We equally split one neuron of a width-2 two-layer NN at a critical point into neurons (k- 2, 3), whose input weights remain the same but output weights are 1/k of the original neuron.
Empirical diagram of loss landscape. In Fig. we present an empirical diagram of loss landscape of width-3 two-layer tanh DNN to visualize set of its critical points predicted by the embedding principle, i.e., critical points embedded from network of width-1 or -2 respectively as well as critical points that cannot be obtained through embedding. Through the training of width-1, -2, -3 network respectively on the training data presented in Fig. forr multiple trials, we discover critical point for width-1 network, 2 critical points for width-2 network and critical point for width-3 network that cannot be embedded from a narrower NN. Then, embedding all these four critical points to critical points/affine subspaces of loss landscape of the width-3 network, we obtain four sets of critical points with their loss values, output functions, degrees of degeneracy and width of network they embedded from illustrated in Fig. 4 This diagram immediately tells us what attracts the gradient-based training trajectory for width-3 network. Specifically, if stagnation happens during the training, this diagram informs us the potential loss values and output functions at stagnation, which could help us better understand the nonlinear training process of not only a width-3 network but also much wider networks due to the embedding principle. Furthermore, as illustrated in Fig. for the training process of 500-neuron NN, saddle points of wide NN, effectively local or global minima of narrow NNs, composes a trajectory, which may serve as a compass for achieving a global minimum from narrow NNs of low complexity.
Reduction of DNN at critical point. The embedding principle predicts a class of critical points of a NN embedded from much narrower NNs. At such a critical point, we shall be able to find neuron groups, within which neurons have similar orientation of input weights presumably originated from the same neuron of narrow NN through embedding. This prediction confirmed by the following experiment in Fig. 5 We train width-400 two-layer ReLU NN fo =In (w x)
5

---

empirical diagram loss landscape
m=3
loss
Embedded from
Ti==1
m
my=3
0.11
degree of degeneracy
0.07
NN output training data
10:5
Figure 4: Empirical diagram of loss landscape of a width-3 two-layer tanh NN, i.c. all critical points width-3 narrower NNs may get close to during the training under proper initialization. Each black dot at terminal represents a specific set of critical points of loss embedded from critical points of NNs of different widths (blue). These critical points have different loss values (ordinate), degrees of degeneracy (green) and output functions (red solid curves) as labelled in the figure. The blue dots represent the training data. We use the same equal splitting as Fig. 3 to embed critical points of width-1 or width-2 NN to critical points of the width-3 NN and compute the hessian to obtain the corresponding degree of degeneracy. Note that the degree of degeneracy of these critical points computed numerically in this problem coincides with their minimal degree of degeneracy mg in Theorem
( T) on 1000 training samples of the MNIST dataset with small initialization. At the blue dot in Fig. aa), the loss decreases very slowly, presumably very close to a saddle point. We then examine the orientation similarity between each pair of neuron input weights by computing the inner product of two normalized input weight. As shown in Fig. ((b), there emerge 58 groups of neurons (neurons with very small amplitudes are neglected and later directly removed), where similarity between input weights in the same group least 0.9. For each group Saimilars we randomly select neuron j. replace its output weight by k Ssigila 0g l///:///l///// ///o/l//|/w//ll and discard all other neurons in the group. The parameter set before reduction is denoted by O.ii and after reduction by Oeddu- Width of the NN is reduced from 400 58. We train the reduced NN from Predu as shown in Fig. [ which stagnated after a few steps at the same loss value as the blue point in Fig, ((a) marked by the blue dash and represented by the blue point in Fig. ((c). We then compare the prediction between original model and the reduced model at the corresponding blue points on 10000 test data as shown in Fig For each grid, color indicates the frequency of that prediction pair. Specifically, the highlight of diagonal element indicts high prediction agreement of two models (overall 98.5%). Therefore, this critical point of the reduced width-58 NN well matches the critical point of the original width-400 NN, clearly demonstrating the relevance of our embedding principle to real dataset training.
10
training
resucee net prediction
2000
selected point
10
Indox
200
triining loss selected paint
10
law amplinade
250
2000
4000
6000
400
100
200 index
300
400
250
500 spach
750
1000
epoch
original not prediction
(a) initial loss
(b) orientation similarity
(c) loss after reduction (d) prediction similarity
Figure 5: (a) The training loss of the initial network on MNIST. The blue point is selected for reduction. (b) The normalized inner product of input weights for different neurons. The abscissa and ordinate represent neuron index. Neurons "low amplitude" region has much lower amplitude than others, hence are removed. (c) The training loss of the reduced network. Blue dash indicates the same loss value as the blue dash in (a). The blue point is selected as representative for comparison. (d) Prediction similarity. For each grid, color indicates the frequency of that prediction pair.

---

Preliminaries
4.1 Deep Neural Networks
Consider L-layer (L 2 2) fully-connected DNNs with a general differentiable activation function. We regard the input as the 0-th layer and the output as the L-th layer. Let my be the number of neurons in the 1-th layer. In particular, my = d and my = d'. For any i, k E N and < k, we denote [i : k] = { + I..... k.. In particular, we denote [k] == {1, ... k}. Given weights w R x and bias 64d Rm for / [4], define the collection of parameters 0 2L-tuple (an ordered list of 2L elements) whose elements are matrices or vectors
= 0)) 10..... ...
.01z) ,
= (will, DP
Will B(.))
(1)
where the (-th layer parameters of 0 is the ordered pair 0/: (wll), b19) LE [4]. We may misuse of notation and identify with its vectorization vec(O) E TR with M = E1--1 (m + 1)m
Given 0 E RM the neural network function fa(-) is defined recursively. First, we write (xx) for all æ E Rd_ Then for E [L-1]. 1]. fo is defined recursively as f! (z) = o(wil!) -/l (x) + bll)). Finally, we denote
fo(x) = f(x = -
1//-11 (x) + blا)
(2)
For notational simplicity. we may drop the subscript in f/ 1 [0: L].
We introduce the following notions for the convenience of the presentation in this paper.
Definition 4 (Wider/narrower DNN), We write [N ( for fully- fully-connected neural network with width (mo..... mL). Given two L-layer (L 2 ffully-connecttr neural networks NN (N M and NN' (fm()=)) if my ma- m'y = m., and for any E [L - -L-1,, 11, mi 2 my and K = Et1 (m/ mi) N4. then we say that NN' (fm() (=m))))))))))))))))))))))) is K-neuron wider than NN((m) and NN((m))2o) K-neuron narrower than NN' ((m()
4.2 Loss function
The training data set is denoted as S = [(x; Bi) where E Rd. y, Rd For simplicity, here we assume an unknown function y satisfying (x;;) = y: for E [n]. The empirical risk reads
(0) = (f, v),)(a))) = Egl(f 0),
E h(x;) for any function _2d KK and the loss function where the expectation (gh(w)) Ei=
((---) is differentiable and the derivative of with respect to its first argument is denoted by Vi(y, y²). Generally, we always take derivatives/gradieet
of l in its first argument with respect to any parameter. We consider gradient flow of Rs as the training dynamics, i.e., d0/dt dt = V Rs (0) with O(0) = Og.
[!] We define the error vectors z
& for I E [L] and the feature gradients gø
[z]
and
| (1) (W/ll ff(-1) + b²)) for E [L 1--18 1]. Here (()) is the first derivative of a. We call f!!l
g
I E [L] feature vectors. The collections of feature vectors, feature gradients, and error vectors
Fo = 1T |_ Go = ( | Zo = (=/))_ Using backpropagation, we can calculate the
gradients follows
VE,
= ( 11
-y |+1]
go [++]
[L 1). 1 [L].
2
V win
1 20 go
[
Vome= 20 og
Here use for the Hadamard product of two matrices of the same dimension.
7

---

Critical embedding
We introduce the one-step embedding for the DNNS which will lead us to general embeddings. Definition 5 (one-step embedding). Given L-layer (L > fullycccnnecee neural network with width (ma.... ............)) ..., and network parameters 0 = (W blll, .... W bl41) E TR.M, for any 1 E [T. 1] and any E [mi], we define the linear operators Ti,s and Vi,s applying on A as follows
Ti.s (0)lk 0 los
k: +1,++1
W
1) b!d
b
-((=-(((ii ((wicts) Omm+ixi] blttt),
Ti,s (0)l:=
(i,(0))
11 1am
VL, (0)la (0mg Xm 0m
+ 1,1++11,
(O)) (0 (1(m++)xxma (mm++)xx1)
V1.((0)l++1 = ([0m)] w
[1:mp41l:st Oml x x(m- a)s W R+1]
[1mm41)s 0mlmixa)
Then the one-step embedding operator Tta defined as for any O E RRM
(T,(0) (Ti,s + (O).
Note that the resulting parameter (T,(()) corresponds fully neural network with width (mo,..., -4, my 1, m2)-
An illustration of Te. Vr and 7 can be found in Fig. S1 in Appendix.
Lemma 1. Given a L-laaye (I. 2 2) fully-connected neural network with width (ma,.. mL). for (W [1] 60 b0.,.... wi4,, bl41) and for any I E [L - 11. s E [mi]. we
any network parameters
have the expressions for 0' Tr. (0)
(i) feature vectors in Fg:: f/r fe l' + and f!! [(F!] YT, (f% {=]
feature gradients in Ger: 9ø gø l' #' and god [( (9.
(t')
(iii) error vectors in Ze: z = 2 11', +
=!| and zg-
11]
11:11p (1 - (z0)
||I) )a (z ) [m+1m az ).)
An illustration of Fo and Zg be found in Fig. S2 in Appendix.
Proposition 1 (one-step embedding preserves network properties). Given a L-layer (L 2 2) fully-comnecedd neural network with width (mo..... m,), for any network parameters O (Wili, 60 wiL,, blel) and for any ( [L 1]. s E [m:]. the following network properties are preserved for 0' TT
(i) outpul function is preserved: Jo (a)) = [o(#) for all x;;
(ii) empirical risk is preserved: Rs(0') = Rs(())
(iil) the sets features are preserved:
{(!! )
= (Ft!")
and
se(m)+1)
sE(mi]
((8") (.)"))) se(mr) = ={(s0" ) for I' E [T]\(T]::
Iremm
Theorem (criticality preserving). Given L-layer (L 2 2) fully-connected neural network with width (mo, m.). for any network parameters O = (W [1] blll ... W (1, bl and for any
I E [L 1]. E [mi), if (0) Lemma 2 (increment of the degree of degeneracy). Given L.layer (I. 2 2) fully-connectdd neural network with width (mo....,
then VoRs(0)) 0.
if there exists E [L. - 1), s E (mi), and q-dimensional manifold M consisting of critical points of Rs such that for any O E M, W [1mmpzll # 0, then M' := (T ( M, E R} (q dimeenssio manifold consists of critical points for the corresponding L-layer fully-connected neural network with width (mo....; m,-1, my 1. m m.).
8

---

Theorem 2 (degeneracy of embedded critical points), Consider two L-layer (L 2 2) fully- connected neural networks NN A (((m))22)) and NN ( ((m()m))))))) which K-neuron wider than A Suppose that the critical point 0 A = (Witl, 6t1 ..... will, bl6)) satisfies W + for each layer& 1 E [L]. Then the parameters 0^ of NN A can be critically embedded to K-dimensional critical offine subspace Mg {0B Σ K
04424 04 E [] of loss landscape of NN B. Here
OB = ( (0x)) and v; - Ti sx Vi... T
Note that neuron-index permutation among the same layer is trivial criticality invariant transform. More discussions about it, specifically for NNs of homogeneous activation functions like ReLU,, can be found in Section A.1 in Appendix.
6 Conclusion and discussion
In this work, we prove embedding principle that loss landscape of a DNN contains all critical points of all the narrower DNNs. This embedding principle unravels wide existence of highly degenerate critical points with low complexity in the loss landscape of a wide DNN, i.e., critical points with low-complexity output function and degenerate Hessian matrix, embedded from critical points of narrow DNNs. With such a loss landscape of DNN, the gradient-based training has the potential of getting attracted or even converging to a low complexity critical point as confirmed by above numerical experiments, which implies potential implicit regularization towards low-complexity function of nonlinear DNN training dynamics.
Moreover, through critical embedding. a critical point in form of a common non-degenerate local minimum of a narrow DNN not only becomes degenerate in general, but also may become a saddle point illustrated by numerical experiments. This may be the reason underlying the general easy optimization of wide DNNs observed in practice even beyond the linear/kerne//NKK regime (Chen) et al. 2020 Trager et al. 2019 Geiger al. 2020: Fort al. 2020; Luo al., 2021). We will perform more detailed analysis as well as numerical experiments specifically about this mininum-to--sddlee transition later.
At the essence, the embedding principle results from the layer structure of a neural network model, which allows arbitrary neuron addition, input weight copying and output weight splitting within each layer. Therefore, though results in this work assume fully-connected NNs, these can be casily extended to other DNN architectures. Considering convolutional neural networks for example, the quantity that corresponds to width of fully-connected NNs is channel. Similar to one-step or multi- step embedding, we can introduce feature splitting operation, i.e., increase the number of channels by splitting all neurons sharing one convolution kernel with the same a, which can be proven to preserve the output function, representation as well the criticality. Thereby, embedding principle holds in a sense that loss landscape of any CNN contains all critical points of all narrower CNNs whose number of channels in each layer is no more than that of the target CNN. Currently, depth serves as a preset hyperparameter in our analysis. Whether loss landscape of DNNs of different depth has certain embedding relation for specific DNN architectures such as ResNet is an interesting open problem.
Our embedding principle and experiments in Figs. I Tand 5 suggest that whenever training of wide DNN is stagnated around a critical point, it potentially is embedded from a much narrower DNN. Therefore, many neurons with similar representation can be reduced to neuron. How we can design efficient pruning algorithm to fully realize this potential and how it is related to existing pruning methods well as the well-known lottery ticket hypothesis (Frankle and Carbin 2018) are important problems for our future research.
We remark that our embedding principle applies for landscape of general loss functions. Although for loss functions like cross entropy, a meaningful finite critical point may not exist as its parameters diverge in general throughout the training. yet it is reasonable to expect that critical embedding may provide us certain approximate critical points from narrow NNs. Of course, how to properly define an approximate critical point is in itself problem of interest. And we leave this problem for the future study.
Overall, our embedding principle provides the first clear picture about the general structure of critical points of DNN loss landscape, which fundamental to the theoretical understanding of both training and generalization behavior of DNNs as well the design of optimization algorithms. Of course, the

---

study of loss landscape of DNN is far from complete. This work serves as starting point for a novel line research, which finally leads to an exact and comprehensive theoretical description about loss landscape of DNNs as well as understanding of its profound impact on training and generalization.
Acknowledgments and Disclosure of Funding
This work sponsored by the National Key R&D Program China Grant No. 2019YFA0709503
(Z. X.). the Shanghai Sailing Program, the Natural Science Foundation of Shanghai Grant No. 20ZR 1429000 (Z. X.). the National Natural Science Foundation of China Grant No. 62002221 (Z. X.), the National Natural Science Foundation of China Grant No. 12101401 (T. L.), the National Natural Science Foundation China Grant No. 12101402 (Y. Z.). Shanghai Municipal of Science and Technology Project Grant No. 20IC1419500 (Y.Z.), Shanghai Municipal of Science and Technology
Major Project No. 1021SHZ2X0002 and the HPC of School Mathematical Sciences and the Student Innovation Center Shanghai Jiao Tong University.
References
W. E, Ma, S. Wojtowytsch, L. Wu, Towards mathematical understanding of neural network-based machine learning: What we know and what we don't, arXiv preprint utXiv:2009.00113 (2020).
I. Skorokhodov, M. Burtsev, Loss landscape sightsecing with muilii-point optimization, arXiv preprint arXiv:1910.03861 (2019).
L. Breiman, Reflections after refereeing papers for nips. The Mathematics of Generalization XX (1995) 11-15.
C. Zhang. S. Bengio, M. Hardt, B. Recht, O. Vinyals, Understanding deep learning requires rethinking generalization, in: 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, OpenReview.net, 2017.
D. Arpit, S. Jastrzebski, N. Ballas, D. Krueger, E. Bengio, M. S. Kanwal, T. Maharaj. A. Fischer, A. C. Courville, Y. Bengio, S. Lacoste- A closer look memorization in deep networks, in: D. Precup. Y. W. Teh (Eds.), Proceedings of the 34th International Conference on Machine Learning. ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, PMLR, 2017, pp. 233-242.
D. Kalimeris, G. Kaplun, P. Nakkiran, B. L. Edelman, T. Yang, B. Barak, H. Zhang, SGD on neural networks learns functions of increasing complexity, in: H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Buc, E. B. Fox, R. Garnett (Eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, 2019, pp. 3491-3501.
P. Jin, L. Lu, Y. Tang, G. E. Karniadakis, Quantifying the generalization error in deep learning in terms of data distribution and neural network smoothness, Neural Networks 130 (2020) 85-99.
Z.-Q. J. Xu, Y. Zhang, Y. Xiao, Training behavior of deep neural network in frequency domain, International Conference on Neural Information Processing (2019) 264-274,
J. J. Xu, Y. Zhang, T. Luo, Y. Xiao, Z. Ma, Frequency principle: Fourier analysis sheds light on deep neural networks, Communications in Computational Physics 28 (2020) 1746-1767.
N. Rahaman, D. Arpit, A. Baratin, F. Draxler, M. Lin, F. A. Hamprecht, Y. Bengio, A. Courville, On the spectral bias of deep neural networks, International Conference on Machine Learning (2019).
Luo, Z.-Q. Xu, Z. Ma, Y. Zhang, Phase diagram for two-layer relu neural networks infinite- width limit, Journal of Machine Learning Research 22 (2021) 1-47.
I. Chizat, F. R. Bach, On the global convergence of gradient descent for over-parameterized models using optimal transport, in: NeurlPS 2018, 2018.
C. Ma, L. Wu, W. E, The quenching-activation behavior of the gradient descent dynamics for two-layer neural network models, arXiv preprint arXiv:2006. 14450 (2020).
10

---

N. S. Keskar. J. Nocedal, P. T. P. Tang, D. Mudigere, M. Smelyanskiy, On large- batch training for deep learning: Generalization gap and sharp minima, in: 5th International Conference on Learning Representations, ICLR 2017, 2017.
L. Wu, Z. Zhu, et al., Towards understanding generalization of deep learning: Perspective of loss landscapes, arXiv preprint arXiv: DeXXiv1706.1030 (2017).
H. He. G. Huang, Y. Yuan, Asymmetric valleys: Beyond sharp and flat local minima, arXiv preprint arXiv:1902.000 (2019).
Y. Cooper, Global minima of overparameterized neural networks, SIAM Journal on Mathematics of Data Science 3 (2021) 676-691. doi 10. 1137/1 9M1 308943
L. Sagun, L. Bottou, Y. LeCun, Singularity of the hessian in deep learning, arXiv preprint arXiv:1611.07476 (2016).
A. Choromanska, M. Henaff, M. Mathicu, G. B. Arous, Y. LeCun, The loss surfaces of multilayer networks, in: Artificial intelligence and statistics, PMLR, 2015, pp. 192-204.
A. Jacot, C. Hongler, F. Gabriel, Neural tangent kernel: Convergence and generalization in neural networks, in: NeurlPS 2018, 2018.
Arora, S. S. Du, W. Hu, Z. Li, R. Salakhutdinov, R. Wang, On exact computation with an infinitely wide neural net, in: NeurIPS 2019, 2019.
Y. Zhang, Z.-Q.J. J. Xu, T. Luo, Z. Ma, A type of generalization error induced by initialization in deep neural networks, in: MSML 2020, 2020.
S. Du, J. Lee, H. Li, L. Wang, X. Zhai, Gradient descent finds global minima of deep neural networks, in: International Conference on Machine Learning, PMLR, 2019, pp. 1675-1685.
D. Zou, Y. Cao, D. Zhou, Q. Gu, Stochastic gradient descent optimizes over-parameterized deep relu networks, arXiv preprint arXiv:1811 111181111188888 (2018).
Z. Allen-Zhu, Y. Li, Z. Song, A convergence theory for deep learning via over- parametrrization, in: International Conference on Machine Learning. PMLR, 2019, pp. 242-252.
W. E, C. Ma, L. Wu, A comparative analysis of the optimization and generalization property of two-layer neural network and random feature models under gradient descent dynamics, arXiv preprint arXiv: 1tt.1.944.4422 (2019).
S. Mei, A. Montanari, P.-M. Nguyen, mean field view of the landscape of two-layer neural networks, Proceedings of the National Academy of Sciences 115 (2018) E7665-E7671.
G. M. Rotskoff, E. Vanden-Eijnden Parameters as interacting particles: long time convergence and asymptotic error scaling of neural networks, in: NeurlPS 2018, 2018
Sirignano, K. Spiliopoulos, Mean field analysis of neural networks: A central limit theorem, Stochastic Processes and their Applications 130 (2020) 1820-1852.
S. Goldt, M. Mézard, F. Krzakala, I. Zdeborová, Modeling the influence of data structure on learning in neural networks: The hidden manifold model, Physical Review X 10 (2020) 041044.
S. He, X. Wang. S. Shi, M. R. Lyu, Z. Tu, Assessing the bilingual knowledge learned by neural machine translation models, arXiv preprint 1rXiv:2004.13770 (2020).
C. Mingard, Skalse, G. Válle-Pérez, D. Martinee-Rubio, Mikulik, A. A. Louis, Neural networks priori biased towards boolean functions with low entropy. arXiv preprint arXiv: 11522 (2019).
Y. Zhang, Li, Z. Zhang, T. Luo, Z. J. Xu, Embedding principle: a hierarchical structure of loss landscape of deep neural networks, arXiv preprint arXiv:2111.1555 (2021).
K. Fukumizu, S. Yamaguchi, Y.-i. Mototake, M. Tanaka, Semi-flat minima and saddle points by embedding neural networks to overparameterization, Advances in Neural Information Processing Systems 32 (2019) 13868-13876.
11

---

B. Simsek, F. Ged, A. Jacot, F. Spadaro, C. Hongler, W. Gerstner, J. Brea, Geometry of the loss landscape in overparametterized neural networks: Symmetries and invariances, Proceedings of the 38th International Conference on Machine Learning, PMLR, 2021, pp. 9722-9732.
R. A. Fisher, The use of multiple measurements in taxonomic problems, Annals of eugenics 7 (1936) 179-188.
Z. Chen, G. M. Rotskoff, J. Bruna, E. Vanden-Eijnden, A dynamical central limit theorem for shallow neural networks, in: NeurIPS, 2020.
M. Trager, Silva, D. Panozzo, D. Zorin, Bruna, Gradient dynamics of shallow univariate relu networks, in: Proceedings of the 33rd International Conference on Neural Information Processing Systems, 2019, pp. 8378-8387.
M. Geiger, S. Spigler, Jacot, Wyart, Disentangling feature and lazy training in deep neural networks, Journal of Statistical Mechanics: Theory and Experiment 2020 (2020) 113301.
S. Fort, G. K. Dziugaite, M. Paul, S. Kharaghani, D. M. Roy, S. Ganguli, Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel, Advances in Neural Information Processing Systems 33 (2020).
Frankle, M. Carbin, The lottery ticket hypothesis: Finding sparse, trainable neural networks, in: International Conference on Learning Representations, 2018.
12


Reviews:
[{"review_id": "qOEh_xQ45gU", "reviewer": "Reviewer_FvEo", "summary": "The paper discusses the question of organization of loss surfaces of neural networks, in particular the connection between wider and narrower networks with respect to the critical points. The proposed analysis introduces an embedding that projects a narrower network into a wider network (at least by one neuron in at least one hidden layer) without changing its function (output) and also saving all its critical points. It is shown, that the introduced procedure of the mapping narrower networks into wider increases degeneracy of the critical points, i.e., increasing amount of zero valued eigenvalues.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The overall work seems very interesting and shedding light on the organization of a loss surfaces of neural networks. Interesting future directions are discussed in section6, including the open question with depth (how depth affects the critical points of the loss surface) as well as identification of critical points.\n\nThere are still some points that raise questions, in particular:\n- It is never discussed if the theory is completely independent of the type of loss function and the activation functions.\n- I think the discussion about the degeneracy of the embedded critical points can be improved: in particular the remark in 3.1 does not seem straightforward conclusion from the informal Theorem2, while theorems in section5 are not discussed anymore. \n- Possibly some definitions are missing (e.g., critical hyperplane used in informal statement of theorem2).\n- I found hard to follow the diagram in Figure4 - how exactly it can be claimed that all the possible critical points for a neural network are found?\n- It is hard to understand the provided numerical experiments, in particular because the type of neural networks is not described; the definition of \"training till critical point\" is not given (I would suggest that the gradients are checked to be 0, but judging by the provided code it is very large amount of epochs). The provided code is not helpful - the images are build based on one of the random runs of the scripts and it is unclear how common is the observed picture.\n\nThe contribution of the paper is very interesting, but the introduced numerical experiments are raising more questions than contribute to the paper---this should be improved.\n\n-----\nAfter the rebuttal and editions I am raising the score to accept (7).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper discusses limitations; as a theoretical paper it does not have a direct societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "axQ2csI1gKx", "reviewer": "Reviewer_Vyz8", "summary": "The authors introduce a prove, in case of fixed-depth fully-connected networks, what they call an \"embedding principle\", stating that a network who's layers are wider than layers of narrower network contain all critical points of the narrower network.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall I like the paper and think it's an important contribution. I am not an expert on the subfield so it is possible that I missed some papers and cannot therefore vouch for its originality, but on the face value the paper addresses an important aspect of loss landscape analysis, delivers a theoretical results, and verifies it using empirical experiments.\n\nA few papers that I thought might be relevant:\nhttps://arxiv.org/abs/2010.15110 on the kernel and linear learning mentioned on line 268\n\nI am giving a weak accept since I'm not an expert on the specific subdomain but I'm happy to increase my score after discussing this with other reviewers and the AC + after the author' rebuttal.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "ReOb-mcbnE", "reviewer": "Reviewer_TiiQ", "summary": "This paper studies critical points of loss landscape of deep neural networks. The authors prove that all critical points of a narrow fully-connected network can be embedded to critical points of a wider network. Moreover, the degree of degeneracy of critical points of narrow networks increases under the described embedding. The critical embedding principle provides a new explanation of the existence of high-dimensional manifolds of critical points in the loss landscape of wide fully connected networks.\n\nThe embedding of a weight vector of a narrow DNN into the weight space of a wider DNN is defined as a composition of simple one-step embedding steps: at each step one neuron of a network is split into two with a specifically chosen assignment of input/output weights to both old and new neurons. The input weights for both new neurons are copied from the original neuron and output weights are re-parameterized via linear interpolation. The authors show that this one-step embedding preserves the output function of the network, criticality of a weight vector in the loss landscape, and increases the degree of degeneracy of the critical point.\n\nThe authors support the theoretical results with experiments on simple learning tasks (1D regression, classification on Iris and MNIST dataset). The experiments confirm the theoretical results: the embedding procedure preserves criticality and increases degeneracy. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "To the best of my knowledge the results presented in this paper are novel. While it is known from prior work that loss landscapes of neural networks have high-dimensional manifolds of critical points, this paper identifies one particular mechanism by which these manifolds are generated. \n\nThe theoretical results of the paper are clearly formulated and technically sound. The proofs are provided and correct (to the best of my understanding). The described experimental setups and results are adequate.\n\nIn my opinion, the main strength of the paper is that it identifies an interesting structure of critical points in the loss landscapes of neural networks. The further development of the framework described in this paper can help to identify new regular structures in the loss landscape and provide novel intuition on the optimization problems in deep learning.\n\nI have a few concerns about the paper:\n\n1) In the abstract (line 7), the authors claim that “the embedding structure of critical points is independent of loss function and training data”. Also, in Section 3.1 (line 129) it is stated that “the degeneracy of critical points is actually a key characteristic of the loss landscape of NNs independent of the size of training data”. These claims are not supported by formal arguments and I am not sure whether the claims are valid. If we increase the number of data points the loss landscape will change and points which were critical before, might no longer be critical (both for narrow and wide networks). All theoretical results in the paper assume that point \\theta_{narr} is a critical point of the current loss landscape. It is not clear to me how the structure of the critical points and the degrees of degeneracy can be independent of training data. \n\n2) In my opinion, the paper overemphasizes the importance of the embedding principle for explanation of phenomena observed in practice and the significance of the results seems limited to me. I agree with the authors that the embedding principle provides a new perspective on the structure of the critical points and the principle suggests that the training might be implicitly regularized towards functions represented by narrow neural networks. However, there is no clear demonstration of these regularization mechanisms in action and their connection to the embedding principle is vague. I think that the paper does not convincingly show that “the embedding principle provides an explanation for the general easy optimization” (abstract, line 11). I believe that more detailed experimental investigation is needed to demonstrate how the existence of embedded critical points in the loss landscape influences training.\n\n3) The embedding principle is based on a particular operation which increases the width of the network in such a way that the new wider network implements the same input-to-output mapping as the original network. This idea is closely related to model compression, network pruning and recent work on the lottery ticket hypothesis [1]. Unfortunately, this body of prior work is overlooked in the paper. I would encourage the authors to include the overview of these research areas and discuss the relation between the embedding principle and network pruning ideas.\n\n\nOther comments:\n* Line 170. Notation a_k, w_k is not defined.\n* Some details of experiments (used batch sizes, optimizers, learning rate schedules) are not provided in the text of the paper.\n* Shouldn’t the statement of Theorem 2 (Section 5) require that \\theta_A is a critical point? What does it mean (formally) that “$\\theta_A$ can be critically embedded to a critical hyperplane?”\n* The description of one step embedding procedure is presented for a neuron of a fully-connected layer. Does the same procedure generalize to convolutional layers? It would be helpful to include the discussion of this topic, since the description of the convolutional layers in terms of individual neurons is not trivial.\n\n\nReferences:\n\n[1] Jonathan Frankle, Michael Carbin, “The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks”, ICLR 2019\n\n\n====================================================================\n\nPost rebuttal update\n\n====================================================================\n\n\nAfter reading the author's response, I am satisfied with the revised version of the paper. I recommend the paper for acceptance. I change my original score from 6 (marginally above the acceptance threshold) to 7 (good paper accept).\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think the authors adequately addressed the limitations and societal impact of their work. I do not foresee any direct negative societal impact of this paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "N4gPO5YBVx-", "reviewer": "Reviewer_TrtW", "summary": "The paper proposes and proves an embedding principle that critical points of all narrow DNNs are embedded into the loss landscape of a wider DNN.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper opens up a fundamentally novel direction for understanding loss landscape of DNNs with implication to DNN training, generalization and robustness. The work provides a principled and elegant framework for characterizing loss landscapes of DNNs, which  connects DNN capacity with its generalization ability. Results of numerical experiments on three different datasets (synthetic, Iris, and Mnist) demonstrate relevance of the proposed embedding in DNN training. For detailed comments, please see below.\n\n(1) The details of how critical points of a narrow network are embedded to a wider DNN in numerical experiments are not provided in the paper. I would suggest authors to elaborate on this point.\n(2) Authors should add DNN training details (optimizer used, learning rate, number of epochs, etc) in the paper.\n(3) The paper is nicely written with a comprehensive related work review, though difficult to follow at certain sections.  I would suggest authors to revise the writing fro better readability and ease of following. Also, some of the figure captions can be improved for better reader comprehension, for example the caption of Figure 4. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Authors have detailed a number of future directions, which address some of the limitations of the current work. ", "review_text": "", "extra_scores": {"ethical_concerns": "NA", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies critical points in the loss landscape of deep neural networks and proves an "embedding principle", which the reviewers find novel and interesting. This could provide a framework towards deeper understanding of deep learning loss landscape. Most of the questions were raised regarding to the presentation and clarification. The reviewers are overall satisfied with the authors' responses and unanimously recommended acceptance. 
Decision: Accept (Spotlight)