An empirical analysis of dropout in piecewise linear networks
Authors: David Warde-Farley, Ian Goodfellow, Aaron Courville, Yoshua Bengio
ICLR 2014

Abstract: The recently introduced dropout training criterion for neural networks has been the subject of much attention due to its simplicity and remarkable effectiveness as a regularizer, as well as its interpretation as a training procedure for an exponentially large ensemble of networks that share parameters. In this work we empirically investigate several questions related to the efficacy of dropout, specifically as it concerns networks employing the popular rectified linear activation function. We investigate the quality of the test time weight-scaling inference procedure by evaluating the geometric average exactly in small models, as well as compare the performance of the geometric mean to the arithmetic mean more commonly employed by ensemble techniques. We explore the effect of tied weights on the ensemble interpretation by training ensembles of masked networks without tied weights. Finally, we investigate an alternative criterion based on a biased estimator of the maximum likelihood ensemble gradient.


Peer review:
[Review]:
For the experiments presented in Fig. 3, I think you guys should just train the tied-weight ensembles with a reduced number of masks too. Then you could directly expose the effect of weight tying on the performance of the induced ensemble, while conveniently avoiding the need to 'max-out' (heh) the size of the ensembles you train with untied weights. I.e., plot the expected performance of both tied-weight ensembles and untied-weight ensembles as a function of the number of distinct masks (i.e. ensemble members) used in training.

While the untied-weight ensembles would still be implicitly 'averaging an exponential number of models' at runtime, you would only be explicitly training as many of those models as are explicitly trained in the corresponding untied-weight ensemble.