Pushing the bounds of dropout

Abstract: We push on the boundaries of our knowledge about dropout by showing theoretically that dropout training can be understood as performing MAP estimation concurrently for an entire family of conditional models whose objectives are themselves lower bounded by the original dropout objective. This discovery allows us to pick any model from this family after training, which leads to a substantial improvement on regularisation-heavy language modelling. The family includes models that compute a power mean over the sampled dropout masks, and their less stochastic subvariants with tighter and higher lower bounds than the fully stochastic dropout objective. The deterministic subvariant's bound is equal to its objective, and the highest amongst these models. It also exhibits the best model fit in our experiments. Together, these results suggest that the predominant view of deterministic dropout as a good approximation to MC averaging is misleading. Rather, deterministic dropout is the best available approximation to the true objective.

Reviews and author responses:

This paper proposes a new understanding of dropout on top of variational dropout, which shows that training with dropout equals to maximizing an empirical variational lower bound on the log-likelihood. This paper shows that the log posterior have the same lower bound when the inference model p(y|x) is defined by different methods, i.e., the arithmetic mean of predictions with different dropout masks, the geometric mean, and a power-mean family as an interpolation between these two cases. This indicates that with the same training objective, different inference methods have different gaps to the posterior lower bound. Intuitively, a smaller gap might lead to better performance. The paper then uses an existing result from Liao & Berg (2017) to show that the gap can be bounded by the variance of prediction probability. With empirical observations, the paper gives an unrigorous conclusion that the deterministic inference with dropout rate 0 achieves the smallest gap. However, this does not hold theoretically due to the extra bias on expectation. The optimality of deterministic inference does not hold empirically due to class imbalance or discrepancy between training and test sets. The paper then proposes two practical solutions for better inference: 1) tuning dropout rate, softmax temperature, and the power mean parameter; and 2) deterministic inference with tuned softmax temperature. By using the first inference solution, the performance on PTB and Wikitext2 LM can be improved by 2-3 on perplexity but is still slightly worse than the SOTA achieved by the mixture of softmaxes.

The idea of analyzing the gap to variational posterior lower bound for different dropout inference model is interesting. The derivations are correct. The organization is not perfect and readers might find it hard to follow here and there, but the main idea is understandable. Experiments show that the suggested tuning of inference hyperparameters can bring improvements to LM tasks, which is convincing. However, there are still major gaps between the theoretical analysis, the conclusion and the empirical solution (please see the detailed comments). Such gaps make the main contribution questionable and make it as a pure empirical paper on its value.

Detailed comments:

1) Reducing the variance of output prediction can reduce the gap on variational posterior, but how does the gap relate to the generalization error? The current paper only indicates that a small gap gives more consistency between the true objective and the optimized objective defined on the training set: they can be still far away from the expected posterior over data distribution. Hence, it is hard to directly relate "reducing the gap" and "improve the test-set performance".

2) As the author mentioned in Section 3.4, reducing the dropout rate causes a bias issue on the expectation. So it is not clear whether deterministic inference with zero dropout rate can achieve the smallest gap or not. In this way, the conclusion is only supported by the empirical observations but not the presented theoretical analysis.

3) One main contribution of this paper is the power-mean family of dropout. However, only one member (alpha=0.5) from the family has been evaluated in the experiments, and it does not achieve the best performance in most experiments. So this contribution seems not practically useful according to the empirical result.

4) It is not clear how the prediction variance is reduced gradually in order to generate the results in Figure 1(b). I guess reducing dropout rate is not the correct way to do so since it causes the bias issue and the tightness will be influenced.

Authors consider dropout as MAP for conditional model and consider different types of averaging to obtain predictive distribution p(y|x, theta) during inference stage. The paper proposes power mean family and shows that well-known types of MC averaging (arithmetic and geometric) are particular cases of proposed family. Authors show that power mean family objective is lower-bounded by the original dropout lower-bound. Therefore it is consistent to use original dropout on the training stage and do any kind of averaging from power mean family during inference stage.

Concerns:
1) In general, paper is hard to follow and main motivation of the accomplished work is not clearly stated in the paper.
2) One of the most confusing things about this paper is the analysis of the lower bound tightness. The authors state that the quality of fit for models in the power mean family depends on the Jensen gap ln(E[L]) - E[ln(L)] where L = p(y|x, w). This gap reflects the difference between the training objective and the objective which is used at the evaluation stage. From this perspective the expectation for the second term E[ln(L)] (which corresponds to the training objective and is the same in all settings) is fixed because we change the dropout scheme only in the inference. Therefore, the Eq. 10 for the gap from this paper is misleading. From this equality the authors derive that reducing the variance leads to decreasing the gap. However, from the Bayes inference it is known that the zero gap between the training objective and the one at evaluation will be reached when we use expectation with respect to the true posterior distribution on the model weights given training data. The authors however claim that the gap is zero when deterministic dropout (that is weight scaling rule) is used. This would be true if we used the same deterministic objective during training stage (that would correspond to no dropout). But we use expectation wrt non-degenerate noise distribution during training and at evaluation stage we take expectation wrt degenerate distribution (i.e. apply deterministic mode). Since the distributions are different we cannot conclude that the gap is zero. Moreover it can be even negative. Hence the statement about zero gap and justification of deterministic mode seems to be wrong. 
3) Experiments results are also hard to follow. In Tables 2, 3, 4 the differences in metric values are insignificant and there are no error bars. From such empirical results it is difficult to draw any conclusions. 

Overall, the motivation of the paper is not clear, the analysis for the lower bound tightness and following conclusions are misguided and experiments results are unconvincing. Therefore, I would suggest rejecting the current version.

Summary:
The paper aims to develop a more principled framework for choosing between different inference procedures in neural network models employing dropout as a stochastic regularizer. In particular, they posit a family of conditional models and show that the learning objectives of all these models are lower bounded by the usual dropout training objective with L2 regularization on weights. They proceed to show empirically that the deterministic inference procedure (multiplying the node's output by the droput rate) achieves the tightest lower bound. From this observation the authors conclude that deterministic inference should be seen as the best available approximation to the true dropout objective rather than an approximation to Monte Carlo averaging. 

Strengths: The paper builds on recent works viewing dropout as a Bayesian approximation to the predictive posterior distribution. Introducing a conditional model and showing that the dropout objective is akin to MAP estimation of the parameters of this model is interesting. The result that dropout simultaneously optimizes a lower bound to an entire family of conditional distributions is novel.

Weaknesses:
The writing is often not clear, ambiguous or misleading and needs improvement. For instance: 
In Section 2.1, comments on weaknesses of variational dropout seems out of place. It should either be ommitted or shifted to the previous section where variational dropout is introduced. 
In Section 2.1, "Consider a conditional model p(Y |X, Θ) as a crippled generative model with p(X) constant, X and Θ independent." Assuming p(X), the input features, to be constant is a very strong assumption. The variational lower bound is still true if p(X) is assumed arbitrary independent of Θ.
In Section 3.3, last paragraph, "Suppose we pick a base model from the power mean family and have a continuum of subvariants with gradually reduced variance in their predictions but the same expectation." It is not clear what the authors mean by continuum of subvariants? Is it the dropout rate?
Several statements are made without any citations or explanations. 
In Section 3. "While it is easy to argue in general that objectives of more than one model may share any given lower bound" How? More explanation needed.
In Section 3.1, "Notice how with SGD and multiple epochs, for each data point several dropout masks are encountered, and the approximating quantity is the geometric mean of the predicted probabilities". Citation needed.
In Section 3.2, "because M_α is monotonically increasing in α". Proof (in appendix) or citation needed. 
In Section 5, "The construction of a conditional model family with a common lower bound on their objectives is applicable to other latent variable models with similar structure and inference method." What general structure and inference method are the authors referring to?

Technical Concerns:
Section 3.3, last paragraph. The entire paragraph is extremely convoluted. It is not clear how one achieves deterministic dropout inference by reducing variance of the prediction y keeping its expectation constant. 
Section 3.3, "A similar argument based ... shows that Z monotonically increases... " How? The authors should deliberate more on this statement since, the Z term is also important in the difference between the true objective of each conditional model and the droput training objective.
In Section 5, "The gains reported in those works might be explained by reducing the bias of deterministic evaluation and also by encouraging small variance in the predictions and thus getting tighter bounds." Isn't this contradictory to Section 3.3, where from Eq. 10 and Eq. 11? If the bias is reduced and variance increases, according to Eq. 10, the lower bound would become looser.
How is Fig. 1a and 1b computed?
Conclusions drawn from experiments not convincing.
Section 3.4, last line. "Having trained a model with dropout, the best ﬁt is achieved by the deterministic model with no dropout. This result isolates the regularisation effects from the biases of the lower bound and the dropout family." How does this isolate the regularization effects? What biases of the lower bound? Do you mean the difference between the model's true objective and the dropout training objective?
Table 4 indicates that not only AMC, also power with alpha=0.5 is better than deterministic in some cases. Did the authors try other values of alphas? Deterministic seems to be good just for MNIST. This strongly refutes the most important claim of this paper, written in the abstract, "Together, these results suggest that the predominant view of deterministic dropout as a good approximation to MC averaging is misleading. Rather, deterministic dropout is the best available approximation to the true objective." 
Changing the dropout multiplier and adjusting the softmax temperature of the network output layer, to achieve comparable performance to AMC on several datasets seems to support the existing hypothesis that deterministic dropout is a good approximation to MC average, and not the other way around!

Summary: 
The paper introduces some interesting ideas about dropout but suffers from bad writing and presentation of results. One of the most important claims made in this paper, "dropout trains a deterministic model ﬁrst and foremost and a continuum of stochastic ones to various extents", is not well-motivated theoretically in Section 3.3. Consequently, this seems to be purely an empirical observation, which is contradicted by further experiments on linguistic datasets. Several conclusions made from experiments seem adhoc.


Meta-review: The reviewers have uniformly had significant reservations for the paper. Given that the authors did not even try to address them, this suggests the paper should be rejected.

Decision: reject