Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes and proves an embedding principle that critical points of all narrow DNNs are embedded into the loss landscape of a wider DNN.
main_review: The paper opens up a fundamentally novel direction for understanding loss landscape of DNNs with implication to DNN training, generalization and robustness. The work provides a principled and elegant framework for characterizing loss landscapes of DNNs, which  connects DNN capacity with its generalization ability. Results of numerical experiments on three different datasets (synthetic, Iris, and Mnist) demonstrate relevance of the proposed embedding in DNN training. For detailed comments, please see below.

(1) The details of how critical points of a narrow network are embedded to a wider DNN in numerical experiments are not provided in the paper. I would suggest authors to elaborate on this point.
(2) Authors should add DNN training details (optimizer used, learning rate, number of epochs, etc) in the paper.
(3) The paper is nicely written with a comprehensive related work review, though difficult to follow at certain sections.  I would suggest authors to revise the writing fro better readability and ease of following. Also, some of the figure captions can be improved for better reader comprehension, for example the caption of Figure 4. 
limitations_and_societal_impact: Authors have detailed a number of future directions, which address some of the limitations of the current work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors introduce a prove, in case of fixed-depth fully-connected networks, what they call an "embedding principle", stating that a network who's layers are wider than layers of narrower network contain all critical points of the narrower network.
main_review: Overall I like the paper and think it's an important contribution. I am not an expert on the subfield so it is possible that I missed some papers and cannot therefore vouch for its originality, but on the face value the paper addresses an important aspect of loss landscape analysis, delivers a theoretical results, and verifies it using empirical experiments.

A few papers that I thought might be relevant:
https://arxiv.org/abs/2010.15110 on the kernel and linear learning mentioned on line 268

I am giving a weak accept since I'm not an expert on the specific subdomain but I'm happy to increase my score after discussing this with other reviewers and the AC + after the author' rebuttal.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper studies critical points of loss landscape of deep neural networks. The authors prove that all critical points of a narrow fully-connected network can be embedded to critical points of a wider network. Moreover, the degree of degeneracy of critical points of narrow networks increases under the described embedding. The critical embedding principle provides a new explanation of the existence of high-dimensional manifolds of critical points in the loss landscape of wide fully connected networks.

The embedding of a weight vector of a narrow DNN into the weight space of a wider DNN is defined as a composition of simple one-step embedding steps: at each step one neuron of a network is split into two with a specifically chosen assignment of input/output weights to both old and new neurons. The input weights for both new neurons are copied from the original neuron and output weights are re-parameterized via linear interpolation. The authors show that this one-step embedding preserves the output function of the network, criticality of a weight vector in the loss landscape, and increases the degree of degeneracy of the critical point.

The authors support the theoretical results with experiments on simple learning tasks (1D regression, classification on Iris and MNIST dataset). The experiments confirm the theoretical results: the embedding procedure preserves criticality and increases degeneracy. 

main_review: To the best of my knowledge the results presented in this paper are novel. While it is known from prior work that loss landscapes of neural networks have high-dimensional manifolds of critical points, this paper identifies one particular mechanism by which these manifolds are generated. 

The theoretical results of the paper are clearly formulated and technically sound. The proofs are provided and correct (to the best of my understanding). The described experimental setups and results are adequate.

In my opinion, the main strength of the paper is that it identifies an interesting structure of critical points in the loss landscapes of neural networks. The further development of the framework described in this paper can help to identify new regular structures in the loss landscape and provide novel intuition on the optimization problems in deep learning.

I have a few concerns about the paper:

1) In the abstract (line 7), the authors claim that “the embedding structure of critical points is independent of loss function and training data”. Also, in Section 3.1 (line 129) it is stated that “the degeneracy of critical points is actually a key characteristic of the loss landscape of NNs independent of the size of training data”. These claims are not supported by formal arguments and I am not sure whether the claims are valid. If we increase the number of data points the loss landscape will change and points which were critical before, might no longer be critical (both for narrow and wide networks). All theoretical results in the paper assume that point \theta_{narr} is a critical point of the current loss landscape. It is not clear to me how the structure of the critical points and the degrees of degeneracy can be independent of training data. 

2) In my opinion, the paper overemphasizes the importance of the embedding principle for explanation of phenomena observed in practice and the significance of the results seems limited to me. I agree with the authors that the embedding principle provides a new perspective on the structure of the critical points and the principle suggests that the training might be implicitly regularized towards functions represented by narrow neural networks. However, there is no clear demonstration of these regularization mechanisms in action and their connection to the embedding principle is vague. I think that the paper does not convincingly show that “the embedding principle provides an explanation for the general easy optimization” (abstract, line 11). I believe that more detailed experimental investigation is needed to demonstrate how the existence of embedded critical points in the loss landscape influences training.

3) The embedding principle is based on a particular operation which increases the width of the network in such a way that the new wider network implements the same input-to-output mapping as the original network. This idea is closely related to model compression, network pruning and recent work on the lottery ticket hypothesis [1]. Unfortunately, this body of prior work is overlooked in the paper. I would encourage the authors to include the overview of these research areas and discuss the relation between the embedding principle and network pruning ideas.


Other comments:
* Line 170. Notation a_k, w_k is not defined.
* Some details of experiments (used batch sizes, optimizers, learning rate schedules) are not provided in the text of the paper.
* Shouldn’t the statement of Theorem 2 (Section 5) require that \theta_A is a critical point? What does it mean (formally) that “$\theta_A$ can be critically embedded to a critical hyperplane?”
* The description of one step embedding procedure is presented for a neuron of a fully-connected layer. Does the same procedure generalize to convolutional layers? It would be helpful to include the discussion of this topic, since the description of the convolutional layers in terms of individual neurons is not trivial.


References:

[1] Jonathan Frankle, Michael Carbin, “The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks”, ICLR 2019


====================================================================

Post rebuttal update

====================================================================


After reading the author's response, I am satisfied with the revised version of the paper. I recommend the paper for acceptance. I change my original score from 6 (marginally above the acceptance threshold) to 7 (good paper accept).

limitations_and_societal_impact: I think the authors adequately addressed the limitations and societal impact of their work. I do not foresee any direct negative societal impact of this paper.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper discusses the question of organization of loss surfaces of neural networks, in particular the connection between wider and narrower networks with respect to the critical points. The proposed analysis introduces an embedding that projects a narrower network into a wider network (at least by one neuron in at least one hidden layer) without changing its function (output) and also saving all its critical points. It is shown, that the introduced procedure of the mapping narrower networks into wider increases degeneracy of the critical points, i.e., increasing amount of zero valued eigenvalues.
main_review: The overall work seems very interesting and shedding light on the organization of a loss surfaces of neural networks. Interesting future directions are discussed in section6, including the open question with depth (how depth affects the critical points of the loss surface) as well as identification of critical points.

There are still some points that raise questions, in particular:
- It is never discussed if the theory is completely independent of the type of loss function and the activation functions.
- I think the discussion about the degeneracy of the embedded critical points can be improved: in particular the remark in 3.1 does not seem straightforward conclusion from the informal Theorem2, while theorems in section5 are not discussed anymore. 
- Possibly some definitions are missing (e.g., critical hyperplane used in informal statement of theorem2).
- I found hard to follow the diagram in Figure4 - how exactly it can be claimed that all the possible critical points for a neural network are found?
- It is hard to understand the provided numerical experiments, in particular because the type of neural networks is not described; the definition of "training till critical point" is not given (I would suggest that the gradients are checked to be 0, but judging by the provided code it is very large amount of epochs). The provided code is not helpful - the images are build based on one of the random runs of the scripts and it is unclear how common is the observed picture.

The contribution of the paper is very interesting, but the introduced numerical experiments are raising more questions than contribute to the paper---this should be improved.

-----
After the rebuttal and editions I am raising the score to accept (7).
limitations_and_societal_impact: The paper discusses limitations; as a theoretical paper it does not have a direct societal impact.
review_title: null

Meta-review: This paper studies critical points in the loss landscape of deep neural networks and proves an "embedding principle", which the reviewers find novel and interesting. This could provide a framework towards deeper understanding of deep learning loss landscape. Most of the questions were raised regarding to the presentation and clarification. The reviewers are overall satisfied with the authors' responses and unanimously recommended acceptance. 
Decision: Accept (Spotlight)