Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper studied the landscape of linear neural networks using algebraic geometry tools. They introduced a distinction between "pure" and "spurious" critical points. They showed that for quadratic loss, there are no spurious local minimum in both the filling and non-filling case. For other convex loss, there are no spurious local minimum in the filling case, but there are spurious local minimum in the non-filling case. They gave a precise description of the number of topologically connected components of the variety of global minima. 

My concern to this paper is that the landscape of linear neural networks, which is the subject of this paper, has already been analyzed in the literature. The final results of this paper, though derived using a different approach, are not new. Another limitation of this paper is, the approach of algebraic geometry used in the analysis seems hard to be generalized to non-linear multi-layers neural networks. 

A contribution of this paper is that the viewpoint of pure and spurious critical points made the landscape results of linear neural networks more intuitive. However, I don't have the expertise to assess whether this viewpoint was implicitly contained in the proof of previous results. Given this, I am not sure the contribution of this paper is enough for ICLR. 

I don't hold a strong opinion, since there could potentially be great ideas inside the algebraic geometry tools. 

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper studies the critical locus of loss functions of deep linear networks. More specifically, this paper introduces a distinction between pure critical points and spurious critical points. The authors prove that all non-global local minima are always pure for convex losses and provide a precise description of the number of topologically connected components of the global minima.

Generally speaking, this paper is well written. However, it is a little bit messy in Section 3.2, where the authors present 4 propositions, one corollary and one remark in less than one page.

My main concern is the contribution and importance of this paper as the landscape of the training loss function of deep linear networks has been well studied. This paper indeed provides some new techniques and insights in this research direction, but the authors do not discuss the possible extension to a more complicated and practical case. I would like to raise my score if the authors can provide some reasonable and convincing discussions regarding the extension to the nonlinear case.

Besides, there is no comparison with the existing results. The properties of critical points of deep linear networks have been studied in many literatures, the authors should provide detailed comparison and discussion between the derived results and these related work in the surrounding text of theorems. 

 Another question is whether this analysis can be generalized to more practical settings, such as deep nonlinear networks. Besides, I believe that the critical points of general deep linear network can have an exact mapping to the critical points of normalized deep linear networks. Again, it is more interesting to explore the benefit of reparameterization for nonlinear networks. 

The last question is whether the landscape analysis in this paper can shed some light on the training dynamics. This paper gives some results regarding the connected/disconnected components of global minima. Is it possible to show which global minima are more likely to be found by optimization algorithms?



review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: The authors use tools from algebraic geometry to study the critical points that arise in linear neural networks under a variety of loss functions.  In particular, they study linear maps that are parameterized (or overparameterized) as the product of multiple matrices.  The authors introduce a distinction between two types of critical points (for the linear function composed with a provided loss): there are those that are due to the actual manifold of functions that are provided to the loss (pure critical points), and there are those that are due to parameterizations of this manifold (spurious critical points).  The authors show that in the case that the linear function has maximal rank (is "filling"), then there are no non-global minima in the loss landscape.  In the case of lower rank ("nonfilling") linear functions, this property does not hold generally, but does hold for quadratic losses.   The authors show that for this framework, there can be exponentially many disconnected components in parameter space corresponding to global minima.  

This paper provides important insight into the loss landscape of neural networks, and it exhibits nuanced observations even in the linear case.  Naturally, there are many important follow-up studies that can be done in the case of non-linear networks.  It is vital to have a thorough understanding before that work can be satisfactorily undertaken.

It would benefit the authors to provide more summary/commentary in the main results section.  Currently, that section comes across as a list of lemmas, theorems, propositions, and corollaries, with relatively little narration in between.  It was challenging for me to identify which of these items were the main results, as summarized on in the first paragraph of the conclusions section.  I recommend that any theorem/corollary/proposition that is the primary basis for a claim made in the conclusion should receive significantly more commentary in Section 3.

Minor comment:
In the first paragraph of the conclusion, it says that "... the absence of ... is a rather general property ...".  The word "rather" should be clarified.

review_title: Official Blind Review #3

Meta-review: This paper studies the landscape of linear networks and its critical point. The authors utilize geometric properties of determinantal varieties to derive interesting results on the landscape of linear networks. The reviewers raised some concerns about the fact that many of the results stated here can already be achieved using other techniques and therefore had some concerns about the novelty of these results. The authors provided a detailed response addressing these concerns. One reviewer however still had some concerns about the novelty. My own understanding of the paper is that while some of these results can be obtained using other approaches the proof techniques (brining ideas from algebraic geometry) is novel and could be rather useful. While at this point it is not clear that the techniques generalize to the nonlinear case I think algebraic geometry perspective have a good potential and provide some diversity in the theoretical techniques. As a result I recommend acceptance if possible.
Decision: Accept (Poster)