Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: In this article, the authors offer a way to decrease the variance of the gradient estimation in the training of neural networks.
They start in the Introduction and Section 2 by explaining the multiple uses of random connection weights in deep learning and how the computational cost often restricts their use to a single randomly sampled set of weights per minibatch, which results to higher-variance gradient estimatos than could be achieved otherwise. In Section 3 the authors offer to get the benefits of multiple weights without most of the cost, when the distribution of the weights is symmetric and fully factorized, by multiplying sampled-once random perturbations of the weights by a rank-1 random sign matrix. This efficient mechanism is only twice as costly as a single random perturbation, and the authors show how to efficiently parallelize it on GPUs, thereby also allowing GPU-ization of evolution strategies (something so far difficult toachieve). Of note, they provide a theoretical analysis in Section 3.2, proving the actual variance reduction of their efficient pseudo-sampling scheme. In Section 4 they provide quite varied empirical analysis: they confirm their theoretical results on four architectures; they show its use it to regularise on language models; they apply it on large minibatch settings where high variance is a main problem; and on evolution strategies.

While it is a rather simple idea which could be summarised much earlier in the  single equation (3), I really like the thoroughness and the clarity of the exposure of the idea. Too many papers in our community skimp on details and on formalism, and it is a delight to see things exposed so clearly -- even accompanied with a proof.

However, the painful part: while I am convinced by the idea and love its detailed exposure, and the gradient variance reduction is made very clear, the experimental impact in terms of accuracy (or perplexity) is, sadly,  not very convincing. Nowhere in the text did I find a clear rationale of why it is beneficial to reduce the variance of the gradient. The numerical results in Table 1 and Table 2 also do not show a clear improvement: Flipout does not provide the best accuracy. The gain in wall clock could be a factor, but would need to be measured on the figures more clearly. And the validation errors in Figure 2 for Evolution strategies seem to be worse than backprop.The main text itself also only claims performance “comparable to the other methods”.  The only visible gain is on the lower part Figure 2.a on a ConvNet.

This makes me wonder if the authors could do a better job of putting forward the actual advantages of their methods on the end-results: could wall clock measure be put more forward, to justify the extra work? This would, in my mind, strongly improve the case for publication of this article.


A few improvement suggestions:
* Could put earlier more emphasis of superiority to Local Reparameterization Trick in terms of architecture, not wait until Section 2.2 and section 4.1
*Should also put more emphasis on limitations, not wait until 3.1.
* Proposition 1 is quite straightforward, not sure it deserves a proposition, but it’s elegant to put it forward.
* Footnote 1 on re-using the matrices is indeed practical, but also somewhat surprising in terms of bias risks. Could it be explained in more depth, maybe by the random permutations of the minibatches making the bias non systematic and cancelling out?
* Theorem 1: For readability could merge the expectations on the joint distribution as E_{x, \hat \delta W} , rather than separate expectations with the conditional distributions.
* Theorem 1: could the authors provide a clearer intuitive explanation of the \beta term alone, not only as part of \alpha + \beta, especially as it plays such a key role, being the only one that does not disappear? And how do they explain their empirical observation that \beta is close to 0? Any intuition on that?
* Experiments: I salute the authors for providing all the details in exhaustive manner in the Appendix. Very commendable.
* Experiments: I like the empirical verification of the theory. Very neat to see.

Minor typo:
* page 2 last paragraph, “evolution strategies” is plural but the verbs are used in singular (“is black box”, “It doesn’t”, “generates”)

review_title: A very pleasant article, but whose actual impact should be made clearer

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Typical weight perturbation algorithms (as used for e.g. Regularization, Bayesian NN, Evolution
Strategies) suffer from a high variance of the gradient estimates. This is caused
by sharing a weight perturbation by all training examples in a minibatch. More specifically
sharing perturbed weights over samples in a minibtach induces correlations between gradients of each sample, which can
not be resolved by standard averaging. The paper introduces a simple idea, flipout, to
perturb the weights quasi-independently within a minibatch: a base perturbation (shared
by all sample in a minibatch) is multiplied by a random rank-one sign matrix (different
for every sample). Due to its special structure it is possible to vectorize this
per-sample-operation such that only matrix-matrix products (as in the default forward
propagation) are involved. The incurred computational cost is roughly twice as much
as a standard forward propagation path. The paper also proves that this approach
reduces the variance of the gradient estimates (and in practice, flipout should
obtain the ideal variance reduction). In a set of experiments it is demonstrated
that a significant reduction in gradient variance is achieved, resulting
in speedups for training time. Additionally, it is demonstrated that
flipout allows evolution strategies utilizing GPUs.

Overall this is a very nice paper. It clearly lays out the problem, describes
one solution to it and shows both theoretically as well as empirically
that the proposed solution is a feasable one. Given the increasing importance
of Bayesian NN and Evolution Strategies, flipout is an important contribution.

Quality: Overall very well written. Relevant literature is covered and an important
problem of current research in ML is tackled.

Clarity: Ideas/Reasons are clearly presented.

Significance: The presented work is highly significant for practical applicability
of Bayesian NN and Evolution Strategies.
review_title: Flipout is an important contribution for weight-perturbation algorithms

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper is well written. The proposal is explained clearly. 
Although the technical contribution of this work is relevant for network learning, several key aspects are yet to be addressed thoroughly, particularly the experiments. 

Will there be any values of alpha, beta and gamma where eq(8) and eq(9) are equivalent. In other words, will it be the case that SharedPerturbation(alpha, beta, gamma, N) = Flipout(alpha1, beta1, gamma1, N1) for some choices of alpha, alpha1, beta, beta1, ...? This needs to be analyzed very thoroughly because some experiments seem to imply that Flip and NoFlip are giving same performance (Fig 2(b)). 
It seems like small batch with shared perturbation should be similar to large batch with flipout? 
Will alpha and gamma depend on the depth of the network? Can we say anything about which networks are better? 
It is clear that the perturbations E1 and E2 are to be uniform +/-1. Are there any benefits for choosing non-uniform sampling, and does the computational overhead of sampling them depend on the network depth/size. 

The experiments seem to be inconclusive. 
Firstly, how would the proposed strategy work on standard vision problems including learning imagenet and cifar datasets (such experiments would put the proposal into perspective compared to dropout and residual net type procedures) ?
Secondly, without confidence intervals (or significance tests of any kind), it is difficult to evaluate the goodness of Flipout vs. baselines, specifically in Figures 2(b,d). 
Thirdly, it is known that small batch sizes give better performance guarantees than large ones, and so, what does Figure 1 really imply? (Needs more explanation here, relating back to description of alpha, beta and gamma; see above). 

review_title: The paper presents a strategy to retain variance reduction while dropping weights rather than activations. It is an important idea that needs more work.

Meta-review: Thank you for submitting you paper to ICLR. The idea is simple, but easy to implement and effective. The paper examines the performance fairly thoroughly across a number of different scenarios showing that the method consistently reduces variance. How this translates into final performance is complex of course, but faster convergence is demonstrated and the revised experiments in table 2 show that it can lead to improvements in accuracy.  
Decision: Accept (Poster)