An empirical analysis of dropout in piecewise linear networks
Authors: David Warde-Farley, Ian Goodfellow, Aaron Courville, Yoshua Bengio
ICLR 2014

Abstract: The recently introduced dropout training criterion for neural networks has been the subject of much attention due to its simplicity and remarkable effectiveness as a regularizer, as well as its interpretation as a training procedure for an exponentially large ensemble of networks that share parameters. In this work we empirically investigate several questions related to the efficacy of dropout, specifically as it concerns networks employing the popular rectified linear activation function. We investigate the quality of the test time weight-scaling inference procedure by evaluating the geometric average exactly in small models, as well as compare the performance of the geometric mean to the arithmetic mean more commonly employed by ensemble techniques. We explore the effect of tied weights on the ensemble interpretation by training ensembles of masked networks without tied weights. Finally, we investigate an alternative criterion based on a biased estimator of the maximum likelihood ensemble gradient.


Peer review:
[Review]:
We thank all of our reviewers for their very useful feedback. We are actively working on a revised version of the manuscript, which will be available by the end of the week.

Two of the reviewers mentioned difficulty interpreting Figures 1 and 2. We apologize that these were not as clear as they could be and are endeavouring to improve them. It seems likely we will simplify with a scatterplot of test error (approximate) versus test error (exact), and test error (arithmetic mean-exact) versus test error (geometric mean-exact). The main issue with these, especially in the case of Figure 1, is that the differences are barely noticeable and all points appear to fall on the y=x line.

Anonymous 3e6b:
- We have made efforts to increase the number of masks used in Section 6. We will endeavour to further increase this for the final copy. As far as larger datasets, while this is an interesting direction, we believe that since it is a regularization method, the problems solved by dropout (that of overfitting) are still very acutely present and studiable on small, relatively simple datasets, which have the added benefit of being amenable to the kinds of exhaustive enumeration of masks that we have carried out.

- The investigations in sections 4 and 5 specifically concern the properties of dropout-trained networks at test-time, therefore we felt comparisons to networks trained without dropout to be unwarranted as it is not clear what these comparisons would be aiming to demonstrate. Sections 6 concerns the performance of untied weight ensembles versus dropout; we can easily add a non-dropout baseline. Section 7’s results are all taken with respect to a network with identical hyperparameters trained without dropout; the test error without dropout is plotted on the X axis and test error with dropout (or dropout boosting) is plotted on the Y. We apologize that this was not clear, we can make this clearer in the axis labels.

- Thank you for the suggestion for a point-by-point summary of all possible (or at least all of the ones considered in the paper) interpretations of dropout, we will add this.

- While some of the conclusions presented in this paper undoubtedly apply to maxout as well, we felt that thoroughly addressing the very popular case of rectified linear networks was important. Had we repeated all of our analyses with maxout (bearing in mind the additional confounding factor of the pool size hyperparameter associated with maxout units) the paper would’ve extended well beyond the recommended length.

- We apologize that these were not clear. Differences are relative to the magnitude reported on the x axis. See our comments above.

Anonymous 925a:

- We appreciate the feedback on Figure 1 and 2. Reviewer 3e6b also found them confusing, so we will be addressing this (see comments above).

- Regarding Figure 3, we chose 120 in order to match Nitish Srivastava’s similar figure, but ran 360 because we would then have 3 independent ensembles with which we could report error bars. We have already increased the number of networks trained and will plot the curve all the way until the end, even without error bars.

Anonymous eb20:

Thank you for your constructive comments. We will address all of them in turn. A few specifics that warrant a particular reply:

- We do cite Jarrett et al in our Introduction, we will add a further citation where you suggest.
- We speculate that since the scaling is exact in the case of linear networks, locally linear networks (being “close” to linear, at least in a small region around training examples) will more closely approximate the ensemble geometric mean than with saturating nonlinearities. We will explicitly state this.
- The arithmetic mean is roughly as expensive as the geometric mean, but lacks a high quality tractable approximation like the weight scaling trick. We will explicitly state this.
- In terms of the stochastic estimator of the ensemble gradient, dropout boosting employs masks drawn from the same distribution as dropout bagging (a.k.a. ordinary dropout). Indeed, the first term of the dropout boosting update is simply the update utilized by dropout. The second term is the gradient of the log likelihood of the same randomly chosen submodel (i.e. same dropout mask) but substituting the true targets with the approximately-averaged ensemble prediction. Training proceeds in the same fashion as dropout, where one randomly selected subnetwork (from the same distribution over masks) is updated, but according to a more globally aware criterion. If the mask application is viewed merely as the addition of noise, both criteria employ identical noise, as the selection procedure and random distribution over masks is identical. We will endeavour to make the text clearer on this point.