Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a regularization technique for deep residual networks.  It is inspired by regularization techniques which disturb the training by applying multiplicative factors to the convolutional layer outputs e.g  Shake-Shake (Gastaldi '17) and PyramidDrop (Yamada '16).  The proposed approach samples a Bernoulli variable randomly to either follow the standard variant of Pyramid net, or applies a variant of shake-shake to pyramid net.

+ Experimental results on CIFAR-10 and CIFAR-100 well-exceed exceed the existing "vanilla" techniques + regularizers.  
- Clarity: some statements are not clear / not substantiated e.g. how does the proposed method overcome the memory problem that shake-shake has?  There are some minor issues wrt presentation, e.g. grammatical correctness of sentences, consistent usage of references, which can be fixed with more careful proofreading.
- Quality: even though the experimental results are compelling, the paper lacks thorough analysis in understanding the effects of the regularizer.  The two experiments looks at (1) the training error, which the paper openly states does not explain why the proposed regularization works and (2) variance of the gradients throughout learning; the larger variance of gradients is speculated to be the cause, but this is almost expected, given that the method is designed to allow larger fluctuations and perturbations during training.
review_title: Compelling experimental results, analysis not totally clear

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper proposes ShakeDrop regularization, which is essentially a combination of the PyramidDrop and Shake-Shake regularization. The procedure consists of essentially weighting the residual branch with a random weight, in the style of Shake-Shake, where the weight is sampled from a mixture of uniform distribution in [-1, 1] and delta at 1, such that the mixture of those two distributions varies linearly with layer depth, in the style of PyramidDrop. In the style of Shake-Shake, a different random weight (in [0, 1]) is used for the backward pass. The most surprising part is that that the forward weight can be negative thus inverting the output of a convolution. Apparently the goal is to "disturb" the training, and the procedure yields state-of-the-art results on CIFAR-10/100.

Positives:

- Results: state-of-the-art on CIFAR-10/100

Negatives:

1. No real motivation on why should this work. I guess the motivation is the mixture of PyramidDrop and Shake-Shake motivations, but the main surprising part (forward weight can be negative) is not motivated at all. There is a tiny bit of discussion at the very end, section 4.4, where authors examine the training loss (showing it's non-zero so less overfitting) and mean/variance of gradients (increased). However, this doesn't really satisfy me - it is clear that more disturbance will cause this behaviour, but that doesn't mean any disturbance is good, e.g. if I always apply the negative weight and make my model weights go in the wrong direction, I'm pretty sure training loss and gradients will be even larger, but it's a bad idea to do.

2. I'm concerned with the "weird trick that happens to work on CIFAR" line of work (not saying that this paper is the only offender) - are these methods actually useful and generalizable to other problems, or are we overfitting on CIFAR and creating MNIST v2.0 ? It would be nice to demonstrate that this regularization works in at least one more problem, maybe ImageNet, though maybe regularization is not needed there but just find one more dataset that needs regularization and test this on that.

3. The paper doesn't explain well what is the problem with Shake-Shake and memory. I see that the author of Shake-Shake has made a comment on this and that makes a lot of sense, i.e. there is no memory issue, just because there are 2x branches doesn't mean shake-shake needs 2x memory as it can use less capacity=memory to achieve the same performance. So it seems the main premise of the paper - "let's apply Shake-Shake to deeper models but we need to come up with a modified method because Shake-Shake cannot be applied due to memory problems" - seems wrong.

4. The writing quality is quite bad, it is very hard to understand what authors mean in parts of the text. E.g. at two places "it has almost the same residual block as Eqn. (1)" - how is it "almost"? Below equation 5, it is never specified that alpha and beta are sampled uniformly(?) from those ranges, one could think that alpha and beta are fixed constants that take a specific value that is in that range. There are also various grammatical errors such as "is expected to be powerful but slight memory overhead" or "which is introduced essence", etc.

Smaller comments:
- Isn't it surprising that alpha in [-1, 1] and beta in [0, 1] works well, but alpha in [0, 1] and beta in [-1, 1] works much worse? The two important cases, (alpha negative, beta positive) and (alpha positive, beta negative), seem to me like they are conceptually very similar.
- End of section 4.1, should it be b_l as p_L is a constant and b_l is what is sampled?
- I don't like that exactly the same text is repeated 3 times (abstract, end of intro, end of 1.1) and in very short distance from each other - repeating the same words 3 times doesn't make the reader understand it better, slight rephrasing is much more beneficial.

Overall:
Good to know that this method sets the new state of the art on CIFAR-10/100, so as such it should be of interest to the community to be available online (arXiv). But with fairly little novelty (is a combination of 2 methods), very little insights of why this should work at all (especially the negative scaling coefficient which is the only extra thing that one learns from this paper, since the rest is a combination of PyramidDrop and Shake-Shake), no idea on whether the method would work outside of the CIFAR-world, and bad quality of the text - I don't think the manuscript is sufficiently good for ICLR.


review_title: Review

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper proposes a new form of regularization that is an extension of "Shake-Shake" regularization (Gastaldi, 2017). The original "shake-shake" proposes using two residual paths adding to the same output (so x + F_1(x) + F_2(x)), and during training, considering different randomly selected convex combinations of the two paths (while using an equally weighted combination at test time). However, this paper contends that this requires additional memory, and attempt to achieve similar regularization with a single path. To do so, they train a network with a single residual path, where the residual is included without attenuation in some cases with some fixed probability, and  attenuated randomly (or even inverted) in others. The paper contends that this achieves superior performance than choosing simply a random attenuation for every sample (although, this can be seen as choosing an attenuation under a distribution with some fixed probability mass at 1). Experiments show improved generalization on CIFAR-10 and CIFAR-100.

I don't think the paper contains sufficiently novel elements to be accepted as a conference track paper at ICLR. While it is interesting that this works well (especially the "negative" weight on the residual), the proposed method is fundamentally a combination of prior work: dropout and "shake-shake" regularization. Moreover, the evaluation is somewhat limited---essentially, I feel there isn't conclusive proof that "shake-drop" is a generically useful regularization technique. For one, the method is evaluated only on small toy-datasets: CIFAR-10 and CIFAR-100. I think at the very least, evaluation on Imagenet is necessary. The proposed regularization is applied only to the "PyramidNet" architecture---which begs the question of whether the proposed regularization is useful only for this specific network architecture. It would have been more useful to see results with and without "shake-drop" on different architectures (the point being to show a consistent improvement with this regularization, rather than achieving 'state of the art' on CIFAR-10). Moreover, it would be interesting to see if the hyperparameter comparison shown in Tables 1 and 2 remained consistent across architectures.
review_title: Review

Meta-review: The paper proposes a regularisation technique based on Shake-Shake which leads to the state of the art performance on the CIFAR-10 and CIFAR-100 dataset. Despite good results on CIFAR, the novelty of the method is low, justification for the method is not provided, and the impact of the method on tasks beyond CIFAR classification is unclear.
Decision: Reject