Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Summary: This paper studies the landscape of deep neural networks with piecewise-linear activation functions. The paper showed that under very mild assumptions, the loss surface admits infinite spurious local minima. Further, it is shown that the loss surface is partitioned into many multilinear cells. If the network is two-layer with two-piece linear activations, it is proved that within each cell every local minimum is global. 

Pros:
  --Constructed spurious local minima for piecewise linear activations, for a broader setting than previous papers. 
  --The paper is well written, with detailed explanation of proof skeleton. 

Cons: 
The significance of the results are not clear. Details are given below. 

1.	This paper only considers piecewise linear activation, which is a special type of non-linear activation. The major examples are ReLU and leaky ReLU. However, related results for ReLU have been studied for a few previous works mentioned in the last two paragraphs of Sec. 3.2. In particular, Yun et al. 2019b already proves a similar result for 1-hidden-layer neural-net with ReLU activation. I think extending the construction to broader settings (any depth, any piecewise linear and more losses) is mathematically nice, but the motivation of this extension is somewhat unclear to me. One motivation is that this is helpful for the purpose of understanding a “big picture” of the landscape, which I will discuss next.

2.	The second major result is Theorem 2, on the “big picture” with ReLU-like activations. However, Theorem 2 is somewhat trivial to prove, and the link to Theorem 1 is rather weak.
   (a) The main message of Thm 2 is the partition of the surface into multiple pieces, and each piece has good property. This partition is somewhat straightforward, and has been studied before, in, e.g., [R1].
   For a global “big picture”, partitioning itself is not very interesting. Theorem 2 mainly describes the property of each region separately for 2-layer network, which is weaker than [R1]. 
   (b) Theorem 2 seems easy to prove. The 1st, 3rd and 4th property are all straightforward. The 2nd property “local analogous convexity” was given a 2-page proof in the paper. However, I don’t understand why not use the following simple argument: for each region, the network behaves like a deep linear network, thus directly applying existing result shall imply “every local minimum in the region is the global minimum of the region”, right? If not, what is the difficulty?
   (c) The 3rd property says “some local minima are concentrated as a valley in some cell”. What are the formal definitions of “concentrated” and “valley” in this sentence?
   (d) The link to Theorem 1 is weak: the link is the 3rd property of Theorem 2 that “some local minima are in a valley”. It is just about some special local minima and weakly related to the other properties on the “global view”. In addition, the fact that “some of them are in a valley” may be due to the very special construction, thus it is not surprising and does not reveal anything interesting about the “big picture”. 


3.	Other issues:
a) While ReLU-type activations are popular, there are still commonly-used activation functions are not piece-wise linear, e.g., tanh, swish. It is not proper to claim that "this paper presents how nonlinearities in activations substantially shape the loss surface" and "almost every practical neural network ....". I suggest replacing "nonlinearity" with "piecewise linearity" in both the title and the abstract, and modifying the over-statements. 
   b) In Property 1 of Theorem 2, “smooth and multilinear partition” might be a bit misleading. The loss surface should be fractional in general, where multilinear cells are separated by non-smooth boundaries. “Smooth partition” seems to imply that the boundaries are smooth or the partition method is smooth in some sense.
   c) The name “analogous convexity” is not appropriate. Analogous convexity is not formally defined in the paper. According to Sec. 4.3 third paragraph, “the property of analogous convexity that the local minima wherein are equally good”. It seems that “analogous convexity” is just “all local minima are good”, which is very different from convexity. It is a weaker property than quasi-convexity, star-convexity, etc, and thus it is better not to call it “analogous convexity”.
    d) Property 3 of Theorem 2 is very far from “mode connectivity”. The proof of Property 3 relies on a special construction of Theorem 1, and the latter is for two arbitrary global minima. 


[R1] Soudry and Hoffer. "Exponentially vanishing sub-optimal local minima in multilayer neural networks." arXiv preprint arXiv:1702.05777 (2017).


Conclusion:  I think this paper is studying an important and interesting question, and the efforts of constructing local minima and understanding big picture are both interesting to me. However, I’m afraid the current form of the paper does not meet the standard of the conference. That being said, it would be a nice paper if the big picture can be explored deeper, and the link to the spurious local minima can be built stronger. 

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper studies the theoretical property of neural network's loss surface. The main contribution is to prove that the loss surface of every neural network (with arbitrary depth) with piecewise linear activations has infinite spurious local minima. Moreover, the paper further characterizes the partition of the local minima. More precisely, the loss surface is partitioned into multiple smooth and multilinear open cells and within each cell, the local minima are equally good. This result can also explain the linear neural network case where there is only one cell, implying that all local minima are global. 

On one hand, I find the paper very clear and the result very clean, which unites a lot of existing results. On the other hand, with a reasonable initialization in practice, we will not attain the local minima constructed in the paper since it requires all the activations to be positive. This limits the plausible implication from this theoretical study. Overall, I am very positive of the paper, the following are some detailed comments. 

a. Please be more precise in the abstract that the activation function need to be piecewise linear. 
The current sentence "the loss surface of every neural network has infinite spurious local minima" does not include this specification. Moreover, if the activation is differentiable, is the claim still hold? It seems to me from the middle of page 3 that Li et al 2018 shows a non-local minima result in this case.

b. How different is the analysis comparing to existing result?
I have only go through the skeleton of the proof and have not read into the details. It seems to me the construction of the local minima is very similar to [1], since the main idea is to consider the linear region by activating all the neurons. Could you summarize the main difficulty to extend their results to multi-layer cases? (Maybe it would be good to illustrate with a simple case like 3 layers few neurons per layer)
Moreover, when considering the local convexity, is it sufficient to say that locally in each cell it is a linear network and then the results on linear network transfers to it locally?

[1] Small nonlinearities in activation functions create bad local minima in neural networks, Yun et al, 2019
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper focus on how activation functions’ nonlinearities shape the loss surface of neural networks. The authors first show why the loss surface of every neural network has infinite spurious local minima. Secondly, the authors prove one theorem to show four properties of the loss surfaces of nonlinear neural networks.
Although this paper is generally easy to follow, and the motivation about nonlinearities and the loss surface is clear, the insight of this paper is somehow shortcoming. Though this work can prove such properties within different preconditions, whether other works’ conditions are inconvenient or not may remain further discussions. This work is established based on several preconditions, while it is hard to assert that most kinds of neural networks can satisfy them perfectly. For instance, this work mentions “Deep learning without poor local minima (NeurIPS2016)”, which requires full-rank and conditional independence of each node. It could be feasible when training a stacked network with particular limitations. This work requires all hidden layers are wider than the output layer, which may not be suitable for image segmentation, generative tasks or super-resolution, etc. Besides, it is laudable to prove fundamental rules in neural networks, while showing or inspiring researchers about how to implement or approximate such results to improve neural networks might be more helpful.
Some questions:

1. The authors assert that “the loss surface of *every* neural network has infinite spurious local minima” in the abstract, while in chapter 3 line 2, authors mention, “We find that *almost all* practical neural networks have infinitely many spurious local minima.” Which one is correct? Based on the following description in this paper, I guess this result is conditionally tenable.

2. In lemma 3, authors construct the local minima by adding very negative biases and show they are spurious. However, it is less likely to learn such negative biases in the real case. Besides, some networks require biases equal to zero to achieve some specific target. My question is: if biases are conditioned on real-world data distribution, will lemma 3 and 4 still work in this case?

3. This paper mentions “infinite” many times. Based on the reference, I believe that the “neural network” in this work refers to the “artificial neural network,” which is majorly stored within float tensors. So the number of combinations of parameters is finite. So why use “infinite” instead of “many”? Finite means I can train a small scale of networks with fewer precisions and check the global minima with a fixed dataset.

All in all, I believe this paper can be significantly improved if more details and experiments are provided.
review_title: Official Blind Review #2

Meta-review: Quoting R3: "This paper studies the theoretical property of neural network's loss surface. The main contribution is to prove that the loss surface of every neural network (with arbitrary depth) with piecewise linear activations has infinite spurious local minima."

There were split reviews, with two reviewers recommending acceptance and one recommending rejection.  During a robust rebuttal and discussion phase, both R2 and R3's appreciation for the work was strengthened.  The authors also provided a robust response to R1, whose main concerns included (i) that the paper's analysis is limited to piecewise linear activation functions, (ii) technical questions about the difficulty of proving theorem 2, which appear to have been answered in the discussion, and (iii) concerns about the strength of the language employed.

On the balance, the reviewers were positively impressed with the relevance of the theoretical study and its contributions.  Genuine shortcomings and misunderstandings were systematically resolved during the rebuttal process.
Decision: Accept (Poster)