On Fast Dropout and its Applicability to Recurrent Networks
Authors: Justin Bayer, Christian Osendorfer, Sebastian Urban, Nutan Chen, Daniela Korhammer, Patrick van der Smagt
ICLR 2014

Abstract: Recurrent Neural Networks (RNNs) are rich models for the processing of sequential data. Recent work on advancing the state of the art has been focused on the optimization or modelling of RNNs, mostly motivated by adressing the problems of the vanishing and exploding gradients.
The control of overfitting has seen considerably less attention.
This paper contributes to that by analyzing fast dropout, a recent regularization method for generalized linear models and neural networks from a back-propagation inspired perspective. We show that fast dropout implements a quadratic form of an adaptive, per-parameter regularizer, which rewards large weights in the light of underfitting, penalizes them for overconfident predictions and vanishes at minima of an unregularized training loss. One consequence of this is the absense of a global weight attractor, which is particularly appealing for RNNs, since the dynamics are not biased towards a certain regime. We positively test the hypothesis that this improves the performance of RNNs on four musical data sets and a natural language processing (NLP) task, on which we achieve state of the art results.


Peer review:
[Review]:
Dear reviewer b50c,

We will address your points one by one.

> This paper rederives the fast dropout method which is a deterministic alternative to the originally proposed stochastic dropout regularization for neural networks. Then the authors apply fast dropout to recurrent neural networks (RNN) on two different datasets.

The derivation is actually a minor part of the paper (about a single page out of nine) which is necessary to prepare common ground for the understanding of section 2.2.2, which is novel.

> The main focus of the paper is the derivation of the fast dropout training with almost no link to RNNs until the experiment sections, which makes the paper consists of two disconnected components.

While it is true that the analysis (Section 2.2.2.) is more general than RNNs, the consequence of it is particularly important for RNNs: the regulariser does not bias the dynamics. This is mentioned in the abstract, main text and partially in the conclusion, but probably not prominently enough.

> The authors could have presented experiments on feedforward NN without affecting the smoothness of the paper at all.

We did not include any experiments with standard feedforward MLPs since this has been done extensively in [1]; we do not feel that there is something that we could do better and do not see any novel hypothesis for MLPs we could test.

> The results are expected. Dropout, on small datasets, does better than baseline systems which doesn’t use any stochastic regularization and is as good as ones that does (Graves 2013). 

Without the analysis one cannot know about a bias of dynamics. Indeed, recent work [4] has shown that non-fast dropout might bias the dynamics due to its relation to L2 regularization. Thus we believe that the results are far form expected.
FD-RNNs improve upon the state of the art on Penn Treebank character prediction set in [5], but the result table probably needs to be clearer.

> In table 2, it is mentioned that assumptions made in RNN-NADE system is also applicable to FD. The authors are encouraged to add these assumptions to get better results.

Applying FD to RNN-NADE is certainly interesting by itself, but well beyond the scope of this paper.

> The results sections (3.1.2 and 3.2.2) still need more work. There are no analysis or discussion about the achieved results. Analysis of the effects of network structure, dropout rate, and other hyper-parameters are missing. The tables' order is reversed (table 3 is referenced first then table 2 and table 1).

We will add the actual hyper parameters obtaining the results to the appendix of an upcoming version. We will also add a few sentences on characteristics of the estimated parameters, such as the Eigenvectors of the transition matrix.

More specifically, we will show empirically that the transition matrices found during optimisation inhibit rather big Eigenvalues (>10) which is unlikely when classic regularisers such as weight decay are used.

> For the results in 3.1.2, please clarify if there is a test data on which you measure NLL or you measure performance on the training data only (there is no training/test division menitoned for the music datasets).

The results are obtained on the same splits as in the previous works by [2, 3]. We mention the use of test data for evaluation twice (in the table caption and the beginning of the section) but the information is probably not well placed. We will update the experimental section to be more clear.


Thanks for your efforts; we will address the issues in an upcoming version later this week.



[1] Wang, Sida, and Christopher Manning. 'Fast dropout training.' Proceedings of the 30th International Conference on Machine Learning (ICML-13). 2013.
[2] Bengio, Yoshua, Nicolas Boulanger-Lewandowski, and Razvan Pascanu. 'Advances in optimizing recurrent networks.' arXiv preprint arXiv:1212.0901(2012).
[3] Boulanger-Lewandowski, Nicolas, Yoshua Bengio, and Pascal Vincent. 'Modeling temporal dependencies in high-dimensional sequences: Application to polyphonic music generation and transcription.' arXiv preprint arXiv:1206.6392 (2012).
[4] Wager, Stefan, Sida Wang, and Percy Liang. 'Dropout training as adaptive regularization.' Advances in Neural Information Processing Systems. 2013.
[5] Graves, Alex. 'Generating sequences with recurrent neural networks.' arXiv preprint arXiv:1308.0850 (2013).