Venue: ICLR 2017 conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper provides some theoretical guarantees for the identity parameterization by showing that 1) arbitrarily deep linear residual networks have no spurious local optima; and 2) residual networks with ReLu activations have universal finite-sample expressivity. This paper is well written and studied a fundamental problem in deep neural network. I am very positive on this paper overall and feel that this result is quite significant by essentially showing the stability of auto-encoder, given the fact that it is hard to provide concrete theoretical guarantees for deep neural networks.

One of key questions is how to extent the result in this paper to the more general nonlinear actuation function case. 

Minors: one line before Eq. (3.1), U \in R ? \times k


review_title: a good paper

final_score: rating: null
review_content: How different is the result when you learn the top layer instead of random projection? Could it be the main source of under-fitting for ImageNet simply because you have many more classes? Also, have you tried optimizing without weight decay?
review_title: Random projections

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper investigates the identity parametrization also known as shortcuts where the output of each layer has the form h(x)+x instead of h(x). This has been shown to perform well in practice (eg. ResNet). The discussions and experiments in the paper are interesting. Here's a few comments on the paper:

-Section 2: Studying the linear networks is interesting by itself. However, it is not clear that how this could translate to any insight about non-linear networks. For example, you have proved that every critical point is global minimum. I think it is helpful to add some discussion about the relationship between linear and non-linear networks.

-Section 3: The construction is interesting but the expressive power of residual network is within a constant factor of general feedforward networks and I don't see why we need a different proof given all the results on finite sample expressivity of feedforward networks. I appreciate if you clarify this.

-Section 4: I like the experiments. The choice of random projection on the top layer is brilliant. However, since you have combined this choice with all-convolutional residual networks, it is hard for the reader to separate the affect of each of them. Therefore, I suggest reporting the numbers for all-convolutional residual networks with learned top layer and also ResNet with random projection on the top layer.

Minor comments:

1- I don't agree that Batch Normalization can be reduced to identity transformation and I don't know if bringing that in the abstract without proper discussion is a good idea.

2- Page 5 above assumption 3.1 : x^(i)=1 ==> ||x^(i)||_2=1

 
review_title: null

final_score: rating: null
review_content: Can you please clarify or discuss those  points?

- Would the results in Theo 2.1 and 2.2 hold when input and output (x and y) have different dimensions? 

- Theo 3.2 ensures overfitting the training set under the assumption that all points can not be too close. can you say anything about f(x+delta) how stable is  your construction, or any generalization or stability statement?

- did you do any experimentation with your construction on real or synthetic data under your data assumption?
 would be interesting to see some empirical results using your construction and not back propagation.
would be interesting to see on those data when model underfits or overfits.


review_title: output set of different dimension/ stability in Relu

final_score: rating: 5: Marginally below acceptance threshold
review_content: Paper Summary:

Authors investigate identity re-parametrization in the linear and the non linear case. 

Detailed comments:

— Linear Residual Network:

The paper shows that for a linear residual network any critical point is a global optimum. This problem is non convex it is interesting that this simple re-parametrization leads to such a result. 

 — Non linear Residual Network:

Authors propose a construction that maps the points to their labels via a resnet , using an initial random projection, followed by a residual block that clusters the data based on their label, and a last layer that maps the clusters to the label. 

1- In Eq 3.4  seems the dimensions are not matching q_j in R^k and e_j in R^r. please clarify 

2- The construction seems fine, but what is special about the resnet here in this construction? One can do a similar construction if we did not have the identity? can you discuss this point?
In the linear case it is clear from a spectral point of view how the identity is helping the optimization. Please provide some intuition.  

3-   Existence of a network in the residual  class that overfits does it give us any intuition on why residual network outperform other architectures? What does an existence result of such a network tell us about its representation power ? 
A simple linear model under the assumption that points can not be too close can overfit the data, and get fast convergence rate (see for instance tsybakov noise condition).

4- What does the construction tell us about the number of layers? 

5- clustering the activation independently from the label, is an old way to pretrain the network. One could use those centroids as weights for the next layer (this is also related to Nystrom approximation see for instance https://www.cse.ust.hk/~twinsen/nystrom.pdf ). Your clustering is very strongly connected to the label at each residual block.
I don't think this is appealing or useful since no feature extraction is happening. Moreover the number of layers in this construction
does not matter. Can you weaken the clustering to be independent to the label at least in the early layers? then one could you use your construction as an initialization in the training. 

— Experiments : 

- last layer is not trained means the layer before the linear layer preceding the softmax?

Minor comments:

Abstract:  how  the identity mapping motivated batch normalization?

review_title: null

Meta-review: The paper begins by presenting a simple analysis for deep linear networks. This is more to demonstrate the intuitions behind their derivations, and does not have practical relevance. They then extend to non-linear resnets with ReLU units and demonstrate that they have finite sample expressivity. They formally establish these results. Inspired by their theory, they perform experiments using simpler architectures without any batch norm, dropout or other regularizations and fix the last layer and still attain competitive results. Indeed, they admit that data augmentation is a form of regularization and can replace other regularization schemes. 
 I think the paper meets the threshold to be accepted.
Decision: Accept (Poster)