Title: Identity Matters in Deep Learning
Authors: Moritz Hardt
Tengyu Ma
Venue: ICLR 2017

Abstract: An emerging design principle in deep learning is that each layer of a deep
artificial neural network should be able to easily express the identity
transformation. This idea not only motivated various normalization techniques,
such as batch normalization, but was also key to the immense success of
residual networks.

In this work, we put the principle of identity parameterization on a more 
solid theoretical footing alongside further empirical progress. We first
give a strikingly simple proof that arbitrarily deep linear residual networks
have no spurious local optima. The same result for feed-forward networks in
their standard parameterization is substantially more delicate.  Second, we
show that residual networks with ReLu activations have universal finite-sample
expressivity in the sense that the network can represent any function of its
sample provided that the model has more parameters than the sample size.

Directly inspired by our theory, we experiment with a radically simple
residual architecture consisting of only residual convolutional layers and
ReLu activations, but no batch normalization, dropout, or max pool. Our model
improves significantly on previous all-convolutional networks on the CIFAR10,
CIFAR100, and ImageNet classification benchmarks.


TL;DR: Emerging theory explaining residual networks alongside new empirical progress

Review discussion:
Can you please clarify or discuss those  points?
- Would the results in Theo 2.1 and 2.2 hold when input and output (x and y) have different dimensions?
- Theo 3.2 ensures overfitting the training set under the assumption that all points can not be too close. can you say anything about f(x+delta) how stable is  your construction, or any generalization or stability statement?
- did you do any experimentation with your construction on real or synthetic data under your data assumption?
would be interesting to see some empirical results using your construction and not back propagation.
would be interesting to see on those data when model underfits or overfits.
Thanks for the questions.
Question 1: The result in Thm 2.1 and 2.2 holds generally when the dimension of y is less than or equal to the dimension of x in the following sense: suppose dim y = k and dim x = d and k < d. We can choose the last layer weight matrix (with dimension k\times d) so that it contains arbitrary orthogonal vectors as rows. We fix the last layer throughout the training. The rest of layers consist of weight matrices with dimension d by d. We can show that i) for any choice of the last layer with orthogonal rows, there exist weight matrices for the rest of layers with small norms that achieve the global minimum error (the minimum possible error achievable by any choices of all of the weight matrices) ; ii) when the last layer is fixed, all the critical points with small norm weights are global minima.
Question 2: Our construction is fairly robust to the perturbation in the input. Concretely, when the weighted are constructed in Thm. 3.2, we can perturb any training data point x with Euclidean norm \delta < (1-\rho)/4, so that the output on the perturbed data point is still approximately the correct label. (Concretely, suppose without the perturbation, the output is e_j where j is the correct label, then after the perturbation, the output is \alpha e_i where \alpha is approximately equal to 1.)
Question 3: We haven't tested it but we will do it soon. We believe the empirical result should match the theoretical prediction though.
Regarding the question about experimentation with our construction, we did the following experiments:
We take the CIFAR-10 dataset and construct a residual network of 100 layers using the construction theoretically analyzed in the paper. Each layer is of dimension k=500 and the nets are fully-connected. The training error of the constructed network has error 0.5%, whereas the test data has error 89% --- the net indeed overfits to the real training data, and cannot be used as an initialization of the training. It serves mainly as theoretical evidence that the network is powerful in its residual parameterization.
Thanks!
How different is the result when you learn the top layer instead of random projection? Could it be the main source of under-fitting for ImageNet simply because you have many more classes?
Also, have you tried optimizing without weight decay?
The results with and without learned projection were fairly similar. On CIFAR10 it was less than 1% difference in accuracy, with a slight advantage for the fixed final layer (that we reported on). On CIFAR100, the advantage was also for a fixed final layer, suggesting that the improvement is robust to an increase in the number of classes. On ImageNet, we don't yet have a systematic comparison between the two, but this is on our todo list.
This paper investigates the identity parametrization also known as shortcuts where the output of each layer has the form h(x)+x instead of h(x). This has been shown to perform well in practice (eg. ResNet). The discussions and experiments in the paper are interesting. Here's a few comments on the paper:
-Section 2: Studying the linear networks is interesting by itself.
However, it is not clear that how this could translate to any insight about non-linear networks.
For example, you have proved that every critical point is global minimum. I think it is helpful to add some discussion about the relationship between linear and non-linear networks.
-Section 3: The construction is interesting but the expressive power of residual network is within a constant factor of general feedforward networks and I don't see why we need a different proof given all the results on finite sample expressivity of feedforward networks. I appreciate if you clarify this.
-Section 4: I like the experiments. The choice of random projection on the top layer is brilliant.
However, since you have combined this choice with all-convolutional residual networks, it is hard for the reader to separate the affect of each of them.
Therefore, I suggest reporting the numbers for all-convolutional residual networks with learned top layer and also ResNet with random projection on the top layer.
Minor comments:
1- I don't agree that Batch Normalization can be reduced to identity transformation and I don't know if bringing that in the abstract without proper discussion is a good idea.
2- Page 5 above assumption 3.1 : x^(i)=1 ==> ||x^(i)||_2=1
Thank you for your valuable suggestions and comments that we address below.
Section 2:  That’s certainly a valid concern for all results on linearized neural nets.
However, we show that the norm of the weights for the linear network can be very small if the number of layers is deep.
When the weights are small, we can hope that the non-linear case can be approximated by the linear version via Taylor expansion.
However, there are technical difficulties to make this intuition precise and rigorous, and these are the major bottleneck of the theoretical understanding of deep learning. One concrete thing we can say about the benefit of re-parameterization in the non-linear case is this: With standard feed-forward networks, all zero weights form an obvious bad critical point.
However, with the re-parameterization, such an obvious bad critical point no longer exists. This may be seen as mild evidence that our result may be true more generally. Adding this discussion is a good suggestion!
Section 3:  As far as we know, the proof for finite sample expressivity for feedforward nets requires the network to have a very large width (the width needs to be larger than the number of examples). The benefit of ResNets is that it allows such construction to use a network with a small width. We think this is a more natural construction, which is also both more similar to the nets used in practice, and exemplifies the appealing properties of ResNets.
Section 4: Great suggestion. We will run all experiments with and without batch-norm to tease out the effect of random projections on the standard architecture.
Paper Summary:
Authors investigate identity re-parametrization in the linear and the non linear case.
Detailed comments:
— Linear Residual Network:
The paper shows that for a linear residual network any critical point is a global optimum. This problem is non convex it is interesting that this simple re-parametrization leads to such a result.
— Non linear Residual Network:
Authors propose a construction that maps the points to their labels via a resnet , using an initial random projection, followed by a residual block that clusters the data based on their label, and a last layer that maps the clusters to the label.
1- In Eq 3.4  seems the dimensions are not matching q_j in R^k and e_j in R^r. please clarify
2- The construction seems fine, but what is special about the resnet here in this construction? One can do a similar construction if we did not have the identity? can you discuss this point?
In the linear case it is clear from a spectral point of view how the identity is helping the optimization. Please provide some intuition.
3-   Existence of a network in the residual  class that overfits does it give us any intuition on why residual network outperform other architectures? What does an existence result of such a network tell us about its representation power ?
A simple linear model under the assumption that points can not be too close can overfit the data, and get fast convergence rate (see for instance tsybakov noise condition).
4- What does the construction tell us about the number of layers?
5- clustering the activation independently from the label, is an old way to pretrain the network. One could use those centroids as weights for the next layer (this is also related to Nystrom approximation see for instance https://www.cse.ust.hk/~twinsen/nystrom.pdf ). Your clustering is very strongly connected to the label at each residual block.
I don't think this is appealing or useful since no feature extraction is happening.
Moreover the number of layers in this construction
does not matter. Can you weaken the clustering to be independent to the label at least in the early layers?
then one could you use your construction as an initialization in the training.
— Experiments :
- last layer is not trained means the layer before the linear layer preceding the softmax?
Minor comments:
Abstract:  how  the identity mapping motivated batch normalization?
Thank you for your valuable suggestions and comments that we address below.
1. There is a typo in the equation 3.4 (and at several related places). There shouldn’t be q_j in Eqn 3.4.
In other words, the last layer doesn’t use identity parameterization because we need to change the dimension to match the number of labels. We will update the paper very soon to correct these typos.
2. As far as we know, the proof for finite sample expressivity for feedforward nets requires the network to have a very large width (the width needs to be larger than the number of examples). The benefit of ResNets is that it allows such construction to use a network with a small width. We think this is a more natural construction, which is also both more similar to the nets used in practice, and exemplifies the appealing properties of ResNets.
3.
Indeed, finite sample expressivity result does show that representational power of the net is great, though it doesn’t imply overfitting. (As a matter of fact, such nets don’t overfit).
Moreover, as argued before, such expressivity result is not easily obtained for other architecture like feed-forward neural nets or linear model (specifically, with a linear model, one can use effectively at most d (=dimension) parameters, and we mainly consider the case when n is much larger than d).
4. Our construction suggests that if we want the width of the net to be smaller, it would be better to use deeper nets.
5. We don’t expect our construction to be used as an initialization of the training.
Moreover, we don’t expect this construction is be close to what is found by the SGD. It serves mainly as theoretical evidence that the network is powerful in its residual parameterization.
Regarding experiment: yes, the linear layer before softmax is not trained — this is motivated by our construction where the last layer is independent of the data, and it helps improves the result by roughly 1%.
This paper provides some theoretical guarantees for the identity parameterization by showing that 1) arbitrarily deep linear residual networks have no spurious local optima; and 2) residual networks with ReLu activations have universal finite-sample expressivity. This paper is well written and studied a fundamental problem in deep neural network. I am very positive on this paper overall and feel that this result is quite significant by essentially showing the stability of auto-encoder, given the fact that it is hard to provide concrete theoretical guarantees for deep neural networks.
One of key questions is how to extent the result in this paper to the more general nonlinear actuation function case.
Minors: one line before Eq. (3.1), U \in R ? \times k
Thanks for the comments! We will fix the typo. (It should be U \in R^{k\times k})

Meta-review:
role: PC
data: comment: The paper begins by presenting a simple analysis for deep linear networks. This is more to demonstrate the intuitions behind their derivations, and does not have practical relevance. They then extend to non-linear resnets with ReLU units and demonstrate that they have finite sample expressivity. They formally establish these results. Inspired by their theory, they perform experiments using simpler architectures without any batch norm, dropout or other regularizations and fix the last layer and still attain competitive results. Indeed, they admit that data augmentation is a form of regularization and can replace other regularization schemes. 
 I think the paper meets the threshold to be accepted.

Decision: Accept (Poster)