An empirical analysis of dropout in piecewise linear networks
Authors: David Warde-Farley, Ian Goodfellow, Aaron Courville, Yoshua Bengio
ICLR 2014

Abstract: The recently introduced dropout training criterion for neural networks has been the subject of much attention due to its simplicity and remarkable effectiveness as a regularizer, as well as its interpretation as a training procedure for an exponentially large ensemble of networks that share parameters. In this work we empirically investigate several questions related to the efficacy of dropout, specifically as it concerns networks employing the popular rectified linear activation function. We investigate the quality of the test time weight-scaling inference procedure by evaluating the geometric average exactly in small models, as well as compare the performance of the geometric mean to the arithmetic mean more commonly employed by ensemble techniques. We explore the effect of tied weights on the ensemble interpretation by training ensembles of masked networks without tied weights. Finally, we investigate an alternative criterion based on a biased estimator of the maximum likelihood ensemble gradient.


Peer review:
[Review]:
This paper provided a very interesting analysis of dropout, which has recently shown great success in a variety of DNN applications. The paper is well written and the experimental analysis is strong. My comments are mainly to help improve the paper and clarify ambiguity in some places

•	Section 1, page1: when you give the equation f(x) = max(0,x) you should state that this is known as a recitified linear unit (ReLU) and provide the appropriate reference, which to my knowledge was first done here: Jarrett, K., Kavukcuoglu, K., Ranzato, M., and LeCun, Y. What is the best multi-stage architecture for ob- ject recognition? In Proc. International Conference on Computer Vision (ICCV’09). IEEE, 2009.
•	Section 2.2, page 3: why does ReLU work better than sigmoid when using dropout? You say this as well but providing some intuition would be good.
•	Section 3, page 3: The reason you have chosen small networks for your initial analysis is so that you can do an exhaustive enumeration. You state this in Section 4 but not 3. You should state this upfront in Section 3, otherwise the reader might think your analysis might not be generalizable to larger data sets.
•	Section 3, Page 4: Your training criterion (early stopping, etc) has been done in previous papers. Pls cite one of these so readers know that this is a commonly used approach
•	Section 4, Page 5: Pls switch the order of sentences “The overall result… and In order to make differences..” to make the flow easier to read. 
•	Section 4, Page 5:  Pls give references for Wilcoxon signed rank test and Bonferroni correction, as not all readers will be familiar with this.
•	Section 4, Page 5: There seem to be some outliers in Figure 1, but your significance test shows that this doesn’t matter. You might want to add a sentence stating that the outliers in Figure 1 are not really significant as shown by the test.
•	Section 5, Page 5: Can you comment on how expensive the arithmetic mean is?
•	Section 6, page 6: Pls provide reference for “utilized norm constraint regularization” → this is known as max-norm and can be found in Nitish Shrivastava’s thesis
•	Section 6. Page 6: You need a period after the “2” footnote.
•	Section 6, page 7: You should also state that the ensemble method required training 360 different networks which is computationally expensive compared to training just 1 network with dropout. 
•	Section 7, page 8: It is not clear to me how the dropout boosting injects the same amount of noise as dropout, and should be clarified.