Venue: ICLR 2017 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper studies the energy landscape of the loss function in neural networks.  It is generally clearly written and nicely provides intuitions for the results.  One main contribution is to show that the level sets of the loss becomes connected as the network is increasingly overparameterized.  It also quantifies, in a way, the degree of disconnectedness possible in terms of the increase in loss that one must allow to find a connected path.  It would seem that this might have some implications for the likelihood of escaping local minima with stochastic gradient descent.  The paper also presents a simple algorithm for finding geodesic paths between two networks such that the loss is decreasing along the path.  Using this they show that the loss seems to become more nonconvex when the loss is smaller.  This is also quite interesting.

The work does have some significant limitations, which is not surprising given the difficulty of fully analyzing the network loss function.  However, the authors are quite clear about these limitations, which especially include not yet analyzing deep networks and analyzing only the oracle loss, and not the empirical loss.  I would have also appreciated a little more practical discussion of the bound in Theorem 2.4.  It is hard to tell whether this bound is tight enough to be practically relevant.

review_title: Interesting analysis

final_score: rating: null
review_content: I'd like to sincerely thank all the authors for clarifications. Unfortunately I still think that it is a strong reject paper. Authors' responses did not solve main problems that I pointed out. This paper is badly written, the results are incremental and presented in the unclear way. The proofs are presented in such a way that I cannot really verify whether proposed theoretical claims (which to the best of my understanding constitute the core of the paper) are correct even though I consider myself to be very familiar with the mathematical tools used here.
The authors should consider submitting this paper to a journal, but after a major revision.
review_title: final evaluation - strong rejection

final_score: rating: 2: Strong rejection
review_content: This is an incremental result (several related results that the authors of the paper mentioned here were already published). The authors claim that they can get rid of the technical assumptions from the previous papers but the results they propose are significantly weaker and also quite technical. The main theoretical result - Theorem 2.4 is not convincing at all. Furthermore, the paper is badly written. No theoretical intuition is given, the experimental section is weak and in some places the formatting is wrong.
review_title: incremental result on the loss surface of deep neural networks

final_score: rating: null
review_content: Can you elaborate more on the comparison with previous results regarding loss surface of deep neural nets ?
review_title: comparison with previous results

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This work contributes to understanding the landscape of deep networks in terms of its topology and geometry. The paper analyzes the former theoretically, and studies the latter empirically. Although the provided contributions are very specific (ReLU nets with single hidden layer, and a heuristic to calculate the normalized geodesic), the results are original and of interest. Thus, they could potentially be used as stepping stones for deeper developments in this area.

Pros:
1. Providing new theory about existence of "poor" local minima for ReLU networks with a hidden unit that relies on input distribution properties as well as the size of the hidden layer.
2. Coming up with a heuristic algorithm to compute the normalized geodesic between two solution points. The latter reflects how curved the path between the two is. 

Cons:
The results are very specific in both topology and geometry analysis.
1. The analysis is performed only over a "single" hidden layer ReLU network. Given the importance of depth in deep architectures, this result cannot really explain the kinds of architectures we are interested in practically. 
2. The normalized geodesic criterion is somewhat limited in representing how easy it is to connect two equally good points. For example, there might exist a straight line between the two (which is considered as easy by the geodesic criterion), but this line might be going through a very narrow valley, challenging gradient based optimization algorithms (and thus extremely difficult to navigate in practice). In addition, the proposed algorithm for computing the normalized geodesic is a greedy heuristic, which as far as I can tell, makes it difficult to know how we can trust in the estimated geodesics obtained by this algorithm.

With all cons said, I stress that I understand both problems tackled in the paper are challenging, and thus I find the contributions valuable and interesting.
review_title: Insightful Results

final_score: rating: null
review_content: I am a bit confused with the presentation in section 2.3.2. The beginning of the section uses matrices W1 and W2 (e.g. Eq (6)) as the weights. This continues to be the case in the following paragraph W=W1 and beta=W2. However, in the third paragraph, the authors switch to vectors w1 and w2. How these vectors relate to the original weight marices W1 and W2?
review_title: Switching from matrix weights to vector weights?

final_score: rating: null
review_content: At some point, would one expect overparameterization to lead to overfitting?  Can anything be said about whether the amount of overparameterization needed to reduce problems from local minima would still allow one to avoid overfitting?
review_title: Overparameterization

Meta-review: The paper presents an analysis of deep ReLU networks, and contrasts it with linear networks. It makes good progress towards providing a theoretical explanation of the difficult problem of characterizing the critical points of this highly nonconvex function. 

I agree with the authors that their approach is superior to earlier works based on spin-glass models or other approximations that make highly unrealistic assumptions. The ReLU structure allows them to provide concrete analysis, without making such approximations, as opposed to more general activation units.

The relationship between smoothness of data distribution and the level of model overparamterization is used to characterize the ability to reach the global optimum. The intuition that having no approximation error (due to overparameterization) leads to more tractable optimization problem is intuitive. Such findings have been seen before (e.g. in tensor decomposition, the global optima can be reached when the amount of noise is small). I recommend the authors to connect it to such findings for other nonconvex problems.  The paper does not address overfitting, as they themselves point out in the conclusion. However, I do expect it to be a very challenging problem.
Decision: Accept (Poster)