Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The main contribution of this paper is a particular Taylor expansion of the outputs of a ResNet which is shown to be exact at almost all points in the input space.  This expression is used to develop a new layer called a “warp layer” which essentially tries to compute several layers of the residual network using the Taylor expansion expression — however in this expression, things can be done in parallel, and interestingly, the authors show that the gradients also decouple when the (ResNet) model is close to a local minimum in a certain sense, which may motivate the decoupling of layers to begin with.  Finally the authors stack these warp layers to create a “warped resnet” which they show does about as well as an ordinary ResNet but has better parallelization properties.

To me the analytical parts of the paper are the most interesting, particularly in showing how the gradients approximately decouple.  However there are several weaknesses to the paper (or maybe just things I didn’t understand).  First,  a major part of the paper tries to make the case that there is a symmetry breaking property of the proposed model, which I am afraid I simply was not able to follow.  Some of the notation is confusing here — for example, presumably the rotations refer to image level rotations rather than literally multiplying the inputs by an orthogonal matrix, which the notation suggests to be the case.  It is also never precisely spelled out what the final theoretical guarantee is (preferably the authors would do this in the form of a proposition or theorem).

Throughout, the authors write out equations as if the weights in all layers are equal, but this is confusing even if the authors say that this is what they are doing, since their explanation is not very clear.  The confusion is particularly acute in places where derivatives are taken, because the derivatives continue to be taken as if the weights were untied, but then written as if they happened to be the same.

Finally the experimental results are okay but perhaps a bit preliminary.  I have a few recommendations here:
* It would be stronger to evaluate results on a larger dataset like ILSVRC.  
* The relative speed-up of WarpNet compared to ResNet needs to be better explained — the authors break the computation of the WarpNet onto two GPUs, but it’s not clear if they do this for the (vanilla) ResNet as well.  In batch mode, the easiest way to parallelize is to have each GPU evaluate half the batch.  Even in a streaming mode where images need to be evaluated one by one, there are ways to pipeline execution of the residual blocks, and I do not see any discussion of these alternatives in the paper.
* In the experimental results, K is set to be 2, and the authors only mention in passing that they have tried larger K in the conclusion.  It would be good to have a more thorough experimental evaluation of the trade-offs of setting K to be higher values.

A few remaining questions for the authors:
* There is a parallel submission (presumably by different authors called “Residual Connections Encourage Iterative Inference”) which contains some related insights.  I wonder what are the differences between the two Taylor expansions, and whether the insights of this paper could be used to help the other paper and vice versa?
* On implementation - the authors mention using Tensorflow’s auto-differentiation.  My question here is — are gradients being re-used intelligently as suggested in Section 3.1?  
* I notice that the analysis about the vanishing Hessian could be applied to most of the popular neural network architectures available now.  How much of the ideas offered in this paper would then generalize to non-resnet settings?


review_title: Review of "Decoupling the Layers in Residual Networks"

final_score: rating: 7: Good paper, accept
review_content: Paper proposes a shallow model for approximating stacks of Resnet layers, based on mathematical approximations to the Resnet equations and experimental insights, and uses this technique to train Resnet-like models in half the time on CIFAR-10 and CIFAR-100. While the experiments are not particularly impressive, I liked the originality of this paper. 
review_title: Creative investigation of Resnets

final_score: rating: 6: Marginally above acceptance threshold
review_content: Motivated via Talor approximation of the Residual network on a local minima, this paper proposed a warp operator that can replace a block of a consecutive number of residual layers. While having the same number of parameters as the original residual network, the new operator has the property that the computation can be parallelized. As demonstrated in the paper, this improves the training time with multi-GPU parallelization, while maintaining similar performance on CIFAR-10 and CIFAR-100.

One thing that is currently not very clear to me is about the rotational symmetry. The paper mentioned rotated filters, but continue to talk about the rotation in the sense of an orthogonal matrix applying to the weight matrix of a convolution layer. The rotation of the filters (as 2D images or images with depth) seem to be quite different from "rotating" a general N-dim vectors in an abstract Euclidean space. It would be helpful to make the description here more explicit and clear.
review_title: warp operator

Meta-review: This paper proposes a “warp operator” based on Taylor expansion that can replace a block of layers in a residual network, allowing for parallelization. Taking advantage of multi-GPU parallelization the paper shows increased speedup with similar performance on CIFAR-10 and CIFAR-100. R1 asked for clarification on rotational symmetry. The authors instead removed the discussion that was causing confusion (replacing with additional experimental results that had been requested). R2 had the most detailed review and thought that the idea and analysis were interesting. They also had difficulty following the discussion of symmetry (noted above). They also pointed out several other issues around clarity and had several suggestions for improving the experiments which seem to have been taken to heart by the authors, who detailed their changes in response to this review. There was also an anonymous public comment that pointed out a “fatal mathematical flaw and weak experiments”. There was a lengthy exchange between this reviewer and the authors, and the paper was actually corrected and clarified in the process. This anonymous poster was rather demanding of the authors, asking for latex-formatted equations, pseudo-code, and giving direction on how to respond to his/her rebuttal. I don't agree with the point that the paper is flawed by "only" presenting a speed-up over ResNet, and furthermore the comment of "not everyone has access to parallelization" isn’t a fair criticism of the paper.
Decision: Accept (Poster)