Title: Dropout with Expectation-linear Regularization
Authors: Xuezhe Ma
Yingkai Gao
Zhiting Hu
Yaoliang Yu
Yuntian Deng
Eduard Hovy
Venue: ICLR 2017
Keywords: Theory
Deep learning
Supervised Learning

Abstract: Dropout, a simple and effective way to train deep neural networks, has led to a number of impressive empirical successes and spawned many recent theoretical investigations. However, the gap between dropout’s training and inference phases, introduced due to tractability considerations, has largely remained under-appreciated. In this work, we first formulate dropout as a tractable approximation of some latent variable model, leading to a clean view of parameter sharing and enabling further theoretical analysis. Then, we introduce (approximate) expectation-linear dropout neural networks, whose inference gap we are able to formally characterize. Algorithmically, we show that our proposed measure of the inference gap can be used to regularize the standard dropout training objective, resulting in an explicit control of the gap. Our method is as simple and efficient as standard dropout. We further prove the upper bounds on the loss in accuracy due to expectation-linearization, describe classes of input distributions that expectation-linearize easily. Experiments on three image classification benchmark datasets demonstrate that reducing the inference gap can indeed improve the performance consistently.


Review discussion:
- In Theorem 5, what is the variable \eta ?
From Eq 22, it seems that lambda=0 is essentially the same as log-likelihood training in your latent variable network model. This should therefore be nearly equivalent to standard dropout according to the arguments you present.
Now comparing the results of Table 1&2 and  Fig. 1:
- for lambda=0 (which I assume corresponds to the leftmost points in every graph of Fig. 1) are consistent for Mnist.
- However for Cifar, the results seem better in Table 2 (11.18, 35.5 for standard and 10.86, 35.10 for EL) than in Fig.1 (12.8, 37.2 for normal and 12.2, 36.3).
You indicate at the page before that you used the same networks as Table 1 (but table 1 only concerns Mnist). Did you use the same networks as Table 2 for the Cifar graphs in Fig 1 ? If so, how would you account for the small but statistically significant difference ?
Additionally, how was lambda selected for the results in table 1 and 2 ?
- \eta, which is defined in Eq (21), is the parameter of the last layer (softmax layer) of the network.
- As mentioned in Section 5.2, the second term (lambda term) at the RHS of Eq (15) can be regarded as a regularization term, similar to other commonly used regularization terms like L-2 norm or Lasso.
When lambda=0, the loss function is exactly the same as log-likelihood. This regularizer pushes the model towards expectation-linearity, and lambda controls the strength of it. In Eq (22), lambda is reflected by \delta, with lambda=0 corresponding to \delta=\infty.
- Table 1 contains results of both MNIST and CIFAR (first 6 rows are for MNIST, 7th row is for CIFAR-10 and 8th row is for CIFAR-100).
Fig 1 is consistent with Table 1 (NOT Table 2). The architectures in Table 1 and Fig 1 are the same.
But Table 2 are for the experiments comparing with Dropout Distillation (Bulò et al., 2016).
So the architectures used in Table 2 are the same as those in (Bulò et al., 2016). We really appreciate your comments for the confusing part of this paper. We will revise this part (maybe it is better to switch Section 6.3 and 6.4 so that Fig 1 would be before Table 2).
- Following previous works, for each data set, we randomly select 10,000 images as the validation set. All the hyper-parameters are tuned on the validation set.
When they are fixed, we train the final model with all the training data, including the validation data.
This paper puts forward a not entirely new, but also not sufficiently understood interpretation of dropout regularization. The authors derive useful theorems that estimate or put bounds on key quantities that are of interest when analyzing dropout regularized networks from their perspective. They furthermore introduce an explicit regularization term that should have a well understood impact on these key quantities. In the experimental section they convincingly show that the proposed regularization indeed has the expected effect and that their perspective on dropout is therefore useful and meaningful.
Their proposed regularization also seems to have a positive impact on the models performance but they demonstrate this only on rel. small scale benchmark problems. I therefore don’t belief that this approach will have a large impact on how practitioner train models.
But their general perspective is well aligned with the recently proposed idea of “Dropout as a bayesian approximation” and the insights and theorems in this paper might enable future work in that direction.
Thanks so much for your review.
Yes, in this paper, we evaluated the proposed regularization method on small scale benchmark problems.
Evaluating the regularizer on large scale tasks and more complex networks (such as RNNs) is clearly one
of potential directions for future work.
This paper introduces dropout as a latent variable model (LVM). Leveraging this formulation authors analyze the dropout “inference gap” which they define to be the gap between network output during training (where an instance of dropout is used for every training sample) and test (where expected dropout values are used to scale node outputs).  They introduce the notion of expectation linearity and use this to derive bounds on the inference gap under some (mild) assumptions.
Furthermore, they propose use of per-sample based inference gap as a regularizer, and present analysis of accuracy of models with expectation-linearization constraints as compared to those without.
One relatively minor issue I see with the LVM view of dropout is that it seems applicable only to probabilistic models whereas dropout is more generally applicable to deep networks.
However I’d expect that the regularizer formulation of dropout would be effective even in non-probabilistic models.
MC dropout on page 8 is not defined, please define.
On page 9 it is mentioned that with the proposed regularizer the standard dropout networks achieve better results than when Monte Carlo dropout is used.  This seems to be the case only on MNIST dataset and not on CIFAR?
From Tables 1 and 2 it also appears that MC dropout achieves best performance across tasks and methods but it is of course an expensive procedure.  Comments on the computational efficiency of various dropout procedures - to go with the accuracy results - would be quite valuable.
Couple of typos:
- Pg. 2 “ … x is he input …” -> “ … x is the input …”
- Pg. 5 “ … as defined in (1), is …” -> ref. to (1) is not right at two places in this paragraph
Overall it is a good paper, I think should be accepted and discussed at the conference.
Thanks so much for your review.
Yes, currently our LVM framework is only applicable to probabilistic models.
But our regularizer can be applied to general deep networks.
It is an interesting direction for future work to development more general framework which is capable to analyze dropout on general deep networks.
For the comparison of performance between MC and standard dropout, the MC dropout did achieve better results on CIFAR.
So the statement of Page 9 should be only on MNIST. We will revise the paper to make it clearer.
We really appreciate your comments about typos and issues of writing. We will fix them.
summary
The paper explains dropout with a latent variable model where the dropout variable (0 or 1 depending on which units should be dropped) is not observed and is accordingly marginalised. Maximum likelihood under this model is not tractable but standard dropout then corresponds to a simple Monte Carlo approximation of ML for this model.
The paper then introduces a theoretical framework for analysing the discrepancy (called inference gap) between the model at training (model ensemble, or here the latent variable model), and the model at testing (where usually what should be an expectation over the activations over many models becomes the activation of one model with averaged weights).
This framework introduces several notions (e.g. expectation linearity) which allow the study of which transition functions (and more generally layers) can have a small inference gap. Theorem 3 gives a bound on the inference gap.
Finally a new regularisation term is introduced to account for minimisation of the inference gap during learning.
Experiments are performed on MNIST, CIFAR-10 and CIFAR-100 and show that the method has the potential to perform better than standard dropout and at the level of Monte Carlo Dropout (the standard method to compute the real dropout outputs consistently with the training assumption of an ensemble, of course quite expensive computationally)
The study gives a very interesting theoretical model for dropout as a latent variable model where standard dropout is then a monte carlo approximation. This is very probably widely applicable to further studies of dropout.
The framework for the study of the inference gap is interesting although maybe somewhat less widely applicable.
The proposed model is convincing although 1. it is tested on simple datasets 2. the gains are relatively small and 3. there is an increased computational cost during training because a new hyper-parameter is introduced.
p6 line 8 typo: expecatation
Thanks so much for your review.
For the typos you pointed out, we will revise this paper soon to fix them.
For the comment about evaluation on small scale benchmark problems (same comment also mentioned by Review 3), we leave the evaluation of the regularizer on large scale tasks and more complex networks (such as RNNs) to one of future work.
We made the following revisions:
1. We switched the section 6.3 and 6.4 to make the paper more clear.
2. We added the definition of MC dropout on page 8.
3. We fixed all the typos in the three reviewers' comments.

Meta-review:
role: PC
data: comment: This paper presents a theoretical underpinning of dropout, and uses this derivation to both characterize its properties and to extend the method. A solid contribution. I am surprised that none of the reviewers mentioned that this work is closely related to the uncited 2015 paper "Variational Dropout and the Local Reparameterization Trick" by Diederik P. Kingma, Tim Salimans, Max Welling.

Decision: Accept (Poster)