Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The motivation of this paper is training deep neural network seems to not suffer from local minima, and it tries to explain this phenomenon by showing that all local minima of deep neural network is global minima. The paper shows that for any convex differentiable loss function, a deep linear neural network has no so called spurious local minima, which to be specific, are local minima that are not global minima, as long as it is true for two-layer Neural Network. The motivation is that combining with existing result that no spurious local minima exists for quadratic loss in two-layer Neural Network, this relation connecting between two-layer and deeper linear neural network immediately implies an existing result that all local minima are global minima, removing all assumptions. The result also holds for general “multi-tower” linear networks. 

Overall, this paper could be an improvement of existing results. It is well written and the proof step is clear in general. However, there’re some weakness need clarifications on the results, especially on the novelty. Given reasonable clarifications in response, I would be willing to change my score.

For novelty, it is unclear if the results from Lemma 1 to Theorem 1 and 2 are both being stated as novel results. The first part of proof of Theorem 1 is obvious and straightforward, and the other direction has been used before for multiple times as claimed in the paper, what is your novelty exactly here? For the key technical claim of Lemma 1, it looks like this perturbation technique already exists in (Laurent & Brecht, 2018), why do you claim it as a novel argument? 

Besides novelty, there are also some other unclear pieces in this paper needs clarification:
1)	Is the main result which is “no spurious local minima for deep neural network” holds for any differentiable convex loss other than quadratic loss? How will Theorem 1 help us understand the mystery of neural network? 
2)	How does the result help us understand non-linear deep neural network, which is commonly use in practice?
3)	The paper should give some explanations about why the results help training neural networks.

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The paper shows an interesting result: deep linear NN has introduced no more spurious local minima than two layer NN and provides an intuitive and short proof for the results, which improve and generalize the previous results under milder assumptions. Overall, the paper is well written and clear in comparison and explanation. 

The weakness is that the main theoretical contribution seems to be merely Lemma 1, and all other theorems are a direct corollary. Also, it would be of great interest to see concrete results on non-linear neural networks, since that is exactly what is used in common practice.

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: Summary: 

The paper shows a deep linear network has no spurious local minima as long as it is true for the two layer case for any convex differentiable loss. 

Comments: 

1) I understand that there exists some work on deep linear network recently. However, they seem to be only for theoretical purpose. Most of the current practical problems do not consider this kind of network for training. If it has high impact in practice, then people are starting to use it. Could you please provide more reasons why we need to care about this impractical network? 

2) It is still unclear about the contributions of the paper. Why “deep linear network has no spurious local minima as long as it is true for the two layer case” is important? And what we can take any advantage from here? What if there exist some spurious local minima for the two layer case (which is widely true)? 

3) The paper looks like a technical report and seems not to be ready. 

The results are quite incremental from the existing ones. The contributions of this work to the deep learning community are still ambiguous. 

review_title: Official Blind Review #1

Meta-review: Paper shows that the question of linear deep networks having spurious local minima under benign conditions on the loss function can be reduced to the two layer case. This paper is motivated by and builds upon works that are proven for specific cases. Reviewers found the techniques used to prove the result not very novel in light of existing techniques. Novelty of technique is of particular importance to this area because these results have little practical value in linear networks on their own; the goal is to extend these techniques to the more interesting non-linear case. 
Decision: Reject