Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents an analysis of convolutional neural networks from the perspective of how the rank of the features is affected by the kinds of layers found in the most popular networks. Their analysis leads to the formulation of a certain theorem about the global minima with respect to parameters in the latter portion of the network.

The authors ask important questions, but I am not sure that they obtain important answers. On the plus side, I'm glad that people are trying to further our understanding our neural networks, and I think that their investigation is worthy of being published.

They present a collection of assumptions, lemmas, and theorems. They have no choice but to have assumptions, because they want to abstract away the "data" part of the analysis while still being able to use certain properties about the rank of the features at certain layers.

Most of my doubts about this paper come from the feeling that equivalent results could be obtained with a more elegant argument about perturbation theory, instead of something like the proof of Lemma A1. That being said, it's easy to voice such concerns, and I'm willing to believe that there might not exist a simple way to derive the same results with an approach more along the line of "whatever your data, pick whatever small epsilon, and you can always have the desired properties by perturbing your data by that small epsilon in a random direction". Have the authors tried this ?

I'm not sure if the authors were the first to present this approach of analyzing the effects of convolutions from a "patch perspective", but I think this is a clever approach. It simplifies the statement of some of their results. I also like the idea of factoring the argument along the concept of some critical "wide layer".

Good review of the literature.

I wished the paper was easier to read. Some of the concepts could have been illustrated to give the reader some way to visualize the intuitive notions. For example, maybe it would have been interesting to plot the rank of features a every layer for LeNet+MNIST ?

At the end of the day, if a friend asked me to summarize the paper, I would tell them :

"Features are basically full rank. Then they use a square loss and end up with an over-parametrized system, so they can achieve loss zero (i.e. global minimum) with a multitude of parameters values."


Nitpicking :

"This paper is one of the first ones, which studies CNNs."
This sentence is strange to read, but I can understand what the authors mean.

"This is true even if the bottom layers (from input to the wide layer) and chosen randomly with probability one."
There's a certain meaning to "with probability one" when it comes to measure theory. The authors are using it correctly in the rest of the paper, but in this sentence I think they simply mean that something holds if "all" the bottom layers have random features.
review_title: Interesting direction.

final_score: rating: 7: Good paper, accept
review_content: This paper analyzes the expressiveness and loss surface of deep CNN. I think the paper is clearly written, and has some interesting insights.
review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper analyzes the loss function and properties of CNNs with one "wide" layer, i.e., a layer with number of neurons greater than the train sample size. Under this and some additional technique conditions, the paper shows that this layer can extract linearly independent features and all critical points are local minimums. I like the presentation and writing of this paper. However, I find it uneasy to fully evaluate the merit of this paper, mainly because the "wide"-layer assumption seems somewhat artificial and makes the corresponding results somewhat expected. The mathematical intuition is that the severe overfitting induced by the wide layer essentially lifts the loss surface to be extremely flat so training to zero/small error becomes easy. This is not surprising. It would be interesting to make the results more quantitive, e.g., to quantify the tradeoff between having local minimums and having nonzero training error. 
review_title: The loss surface and expressivity of deep convolutional neural networks

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper presents several theoretical results on the loss functions of CNNs and fully-connected neural networks. I summarize the results as follows:

(1) Under certain assumptions, if the network contains a "wide“ hidden layer, such that the layer width is larger than the number of training examples, then (with random weights) this layer almost surely extracts linearly independent features for the training examples.

(2) If the wide layer is at the top of all hidden layers, then the neural network can perfectly fit the training data.

(3) Under similar assumptions and within a restricted parameter set S_k, all critical points are the global minimum. These solutions achieve zero squared-loss.

I would consider result (1) as the main result of this paper, because (2) is a direct consequence of (1). Intuitively, (1) is an easy result. Under the assumptions of Theorem 3.5, it is clear that any tiny random perturbation on the weights will make the output linearly independent. The result will be more interesting if the authors can show that the smallest eigenvalue of the output matrix is relatively large, or at least not exponentially small.

Result (3) has severe limitations, because: (a) there can be infinitely many critical point not in S_k that are spurious local minima; (b) Even though these spurious local minima have zero Lebesgue measure, the union of their basins of attraction can have substantial Lebesgue measure; (c) inside S_k, Theorem 4.4 doesn't exclude the solutions with exponentially small gradients, but whose loss function values are bounded away above zero. If an optimization algorithm falls onto these solutions, it will be hard to escape.

Overall, the paper presents several incremental improvement over existing theories. However, the novelty and the technical contribution are not sufficient for securing an acceptance.


review_title: Review of "The loss surface and expressivity of deep convolutional neural networks"

Meta-review: Dear authors,

While I appreciate the result that a convolutional layer can have full rank output, this allowing a dataset to be classified perfectly under mild conditions, the fact that all reviewers expressed concern about the statement is an indication that the presentation sill needs quite a bit of work.

Thus, I recommend it as an ICLR workshop paper.
Decision: Invite to Workshop Track