An empirical analysis of dropout in piecewise linear networks
Authors: David Warde-Farley, Ian Goodfellow, Aaron Courville, Yoshua Bengio
ICLR 2014

Abstract: The recently introduced dropout training criterion for neural networks has been the subject of much attention due to its simplicity and remarkable effectiveness as a regularizer, as well as its interpretation as a training procedure for an exponentially large ensemble of networks that share parameters. In this work we empirically investigate several questions related to the efficacy of dropout, specifically as it concerns networks employing the popular rectified linear activation function. We investigate the quality of the test time weight-scaling inference procedure by evaluating the geometric average exactly in small models, as well as compare the performance of the geometric mean to the arithmetic mean more commonly employed by ensemble techniques. We explore the effect of tied weights on the ensemble interpretation by training ensembles of masked networks without tied weights. Finally, we investigate an alternative criterion based on a biased estimator of the maximum likelihood ensemble gradient.


Peer review:
[Review]:
* A brief summary of the paper's contributions, in the context of prior work.
Paper experimentally verifies how relevant are anecdotal descriptions of dropout. 

* An assessment of novelty and quality. 
It is novel, and has a good quality. However, it doesn’t bring any new ideas. It rather presents experiments which were missed during initial development of dropout.

* A list of pros and cons (reasons to accept/reject).
pros:
- Verifies some of previously postulated intuition.
- Gives some indications what are the major building blocks of a good regularizer for neural networks.

cons:
- Experiment should be run on a larger datasets where not all possible masks are utilized but a large amount of them (e.g. not all 2^N, but e.g. 10^6 random one).
- All the comparisons should be with respect to networks without any dropout or model averaging. Maybe on the datasets which you considered regardless of dropout you would get the same results. Such setting should be compared on all the plots. It was unclear for me if such experiments were executed.
- Authors should write point-by-point what are all possible interpretations of dropout, and which one they are going to validate. It would be also good to have some suggestions how other anecdotal interpretations could be validated (e.g. co-adaptation).
- The same authors were working on max-out networks, which seems to play well with dropout. It should be explained here (or just experimentally compared) why maxout is a good architecture for dropout. 
- Figures 1, and 2 are hard to interpret. How should I know if relative difference of 0.1 is big or small.