Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: 1- Why could one conclude "non-linearities are not the causes of hard learning" based on a single plot on MNIST dataset?
2- Did you try to get similar results to figure 4 for non-linear networks?
review_title: Linear vs non-linear, are they really similar?

final_score: rating: 4: Ok but not good enough - rejection
review_content: I think the write-up can be improved. The results of the paper also might be somewhat misleading. The behavior for when weights are 0 is not revealing of how the model works in general. I think the work also underestimates the effect of the nonlinearities on the learning dynamics of the model.
review_title: not ready yet

final_score: rating: null
review_content: Hi, 

 Sorry for the delay in posting my question. Before a question a comment. I think you need to explain Lemma 1 and Lemma 2 better. I think is quite hard (at least it was for me) to understand what they claimed (not necessarily the proof itself). Being a bit more verbose could help. I'm also thinking that the proof should be in the main text as it is the main contribution imho of the work. But I can see space is tight for that. Also regarding writing style, I find epithets like superb maybe not necessary and I would use \emph instead of bold face. 


Regarding the claims. I think the paper is interesting .. there are a few things though that I don't agree with. The only reasons this thing works with initialization 0, is because there are only 2 layers skipped with an identity, which results in an stable point from which you can escape. If you skip over more than 2 and initialize with 0 you wouldn't be able to learn! I think this needs to made clearer in the text. This is an unusual operation regime !

Secondly I don't think the results say anything about how hard an optimization problem we have due to the nonlinearity. Anecdotal evidence is not enough to say that the main difficulty of training is not from the nonlinearity. To be honest I would not even know how to compare the amount of difficulty cause by A or B. I think that sentence is misleading. 

Also I think looking at the Hessian at a fixed point might also be misleading. How does the condition number deteriorates as I move from this initialization? How fast? Also just because this number doesn't depend on depth at that point (but it does anywhere in vicinity) doesn't necessary mean that the condition number is better than the one obtain by say Xavier initialization or orthogonal initialization. Condition number for orthogonal initialization (at initialization) is also 1 and doesn't depend on depth, right?  I know there are empirical evidence .. I just I'm not sure, it feels there are a few things missing from the argument. 

review_title: Pre-review questions

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper studies the optimization issue of linear ResNet, and shows mathematically that for 2-shortcuts and zero initialization, the Hessian has condition number independent of depth. I skimmed through the proof but have not checked them carefully. 

This result is a nice observation for training deep linear networks.  But I do not think the paper has fully resolved the linear vs nonlinear issue. Some question:

1. Though the revision has added some results using ReLU units, it seems it is only added to the mid positions of the network (sec 5.3), is this how it is typically done in ResNet? Moreover, ReLU is not differentiable at zero point, which does not satisfy the condition you had in Theorem 1. Why not use differentiable activations like sigmoid or tanh?

2. From equation (22) in the appendix, it seems for nonlinear activations, the condition number depends on the derivative \sigma^\prime at 0. Therefore, if we use tanh which has derivative 1 at zero, the condition number is the same for linear and tanh activations. But this probably is not enough to explain the bit difference in performance or optimization for linear and nonlinear networks, or how the situations evolve after learning the 0 point.

3. As for the success of ResNet (or convnets in general) in computer vision, I believe there are more types of nonlinearity such as pooling? Can the result here generalizes to pooling as well?

Minor: 
- sec 1 last paragraph, low approximation error typically means more powerful model class and better training error, but not necessarily better test error
- sec 4.1 what do you mean by "zero initialization with small random perturbations"? why not exactly zero initialization, how large is the random perturbation?


review_title: this paper has nice observation

final_score: rating: null
review_content: I have so far taken a quick pass over the paper. Some questions:

1. In Theorem 1:  you showed condition number of 2-shortcut networks are depth-invariant, and gave the condition number in (13). How do the condition number behave for nonlinear activations, would it be larger or smaller than the linear case, what quantity of the activation does the condition number depends on?  I have not looked at the appendix, but it would be good to have some discussion in the main text.

2. Why only study the Hessian at zero initialization? I understand your point that it is easier to escape the saddle point for n=2 in this regime. But does it have implication on how it behaves after leaving this regime (and for vast majority of the time, we probably are not in that regime)?

3. You said in the end of section 2 that "nonlinearities are not the main cause of hard learning", then in your opinion, what are the causes?
review_title: preliminary questions

final_score: rating: 4: Ok but not good enough - rejection
review_content: ResNet and other architectures that use shortcuts have shown empirical success in several domains and therefore, studying the optimization for such architectures is very valuable. This paper is an attempt to address some of the properties of networks that use shortcuts. Some of the experiments in the paper are interesting. However, there are two main issues with the current paper:

1- linear vs non-linear: I think studying linear networks is valuable but we should be careful not to extend the results to networks with non-linear activations without enough evidence. This is especially true for Hessian as the Hessian of non-linear networks have very large condition number (see the ICLR submission "Singularity of Hessian in Deep Learning") even in cases where the optimization is not challenging. Therefore, I don't agree with the claims in the paper on non-linear networks. Moreover, one plot on MNIST is not enough to claim that non-linear networks behave similar to linear networks.

2- Hessian at zero initial point: The explanation of why we should be interested in Hessain at zero initial point is not acceptable. The zero initial point is not interesting because it is a very particular point that cannot tell us about the Hessian during optimization. 
review_title: This paper tries to explain the reason behind the success of ResNet architecture through theoretical arguments and experimental results. Unfortunately, theoretical arguments and experimental results are not enough to support the main claims of the paper.

Meta-review: This paper endeavors to offer theoretical explanations of the performance of ResNet. Providing better theoretical understanding of existing empirically powerful architectures is very important work and I commend the authors for tackling this. Unfortunately, this paper falls short in its current form: the particular choices and restrictions made (0 weights, linear regime) limit applicability to ResNet, and do not seem to offer insights sufficient to capture the causes of ResNet's performance.
Decision: Reject