Title: Adjusting for Dropout Variance in Batch Normalization and Weight Initialization
Authors: Dan Hendrycks
Kevin Gimpel
Venue: ICLR 2017

Abstract: We show how to adjust for the variance introduced by dropout with corrections to weight initialization and Batch Normalization, yielding higher accuracy. Though dropout can preserve the expected input to a neuron between train and test, the variance of the input differs. We thus propose a new weight initialization by correcting for the influence of dropout rates and an arbitrary nonlinearity's influence on variance through simple corrective scalars. Since Batch Normalization trained with dropout estimates the variance of a layer's incoming distribution with some inputs dropped, the variance also differs between train and test. After training a network with Batch Normalization and dropout, we simply update Batch Normalization's variance moving averages with dropout off and obtain state of the art on CIFAR-10 and CIFAR-100 without data augmentation.

TL;DR: Batch Norm Incorrectly Estimates Variance When Dropout Is On

Review discussion:
Figure 1 shows exponential blowups and decay for Xavier initialization but the Log-loss decay over epochs from Figure 2 looks very smooth and similar to that of presented approach. Can you please comment on this?
Why does batch normalization variance needs to be updated before testing?
How re-estimation of batch normalization variance after finishing training help perform better on test set?
Thank you for the questions. The results in Figure 1 are from a deeper fully-connected network, while those from Figure 2 are from a shallower network trained on MNIST. Figure 1 illustrates that He and Xavier initializations can lead to exponential blowups and decays.
As a result, deeper networks may take longer to train because the network must learn to stabilize the signal or the network will not learn.
For this reason, Figure 3 shows that it takes longer for the He initialization to converge because several epochs are spent learning to stabilize the feedforward signal, and the He initialization failed to converge at more learning rates than when we used our initialization in the CIFAR-10 experiment.
We saw that the variance of a neuron's input grows when dropout is on inside the Weight Initialization derivation section.
Consequently, Batch Normalization's variance estimates are greater when dropout is on because a neuron's input variance is greater with dropout.
But dropout is off during testing, and the variance estimates BN normally uses are valid when dropout is on not off.
For this reason we re-estimate the batch normalization variance by simply turning dropout off and feeding forward the training data.
Then the variance re-estimates are accurate for when dropout is off, so the variance estimates are accurate for testing. We have added an expanded explanation in the manuscript thanks to your comment.
-Why don't you compare to also recent state of art methods like resnet variants or denseNet  for Section 2?
- The parameters are set to fix values of selected from small set of values.
However tuning with random search or bayesian optimization is the two common ways to obtain meaningful comparisons. At this stage, it is hard to see whether the difference is coming from proposed approaches since the parameters are not fine tuned reasonably.
- Are there any results on Imagenet?
Thank you for the questions. To our understanding, He and Xavier weight initializations are not at all suited for ResNets unless one uses Batch Normalization. This is because the argument behind the design of the He initialization assumes each layer depends only on the single previous one, but ResNets receive inputs from the main connection and the residual connection. Should anyone develop a weight initialization for ResNets, this work would still be of help because we specify how to adjust for an arbitrary nonlinearity's contractiveness and how to adjust for dropout.
We do not have ImageNet results due to ImageNet’s computational demands, and this is why the manuscript was not submitted to a more numbers-based conference.
The CIFAR-10 VGGNet inherited most of its hyperparameters (l2 decay, dropout, network structure, etc.) from this popular implementation: https://github.com/szagoruyko/cifar.torch/blob/master/models/vgg_bn_drop.lua But in light of your comment we now trained with 7 randomly selected learning rates and 3 determinsitically chosen rates per optimizer, and the results are in the updated manuscript. We have improved the manuscript thanks to your comment.
The paper presents an approach for compensating the input/activation variance introduced by dropout in a network.
Additionally, a practical inference trick of re-estimating the batch normalization parameters with dropout turned off before testing.
The authors very well show how dropout influences the input/activation variance and then scale the initial weights accordingly to achieve unit variance which helps in avoiding activation outputs exploding or vanishing. It is shown that the presented approach serves as a good initialization technique for deep networks and results in performances op par or slightly better than the existing approaches. The limited experimental validation and only small difference in accuracies compared to existing methods makes it difficult to judge the effectiveness of presented approach. Perhaps observing the statistics of output activations and gradients over training epochs in multiple experiments can better support the argument of stability of the network using proposed approach.
Authors might consider adding some validation for considering the backpropagation variance. On multiple occasions comparison is drawn against batch normalization which I believe does much more than a weight initialization technique. The presented approach is a good initialization technique just not sure if its better than existing ones.
Thank you for your analysis of our paper.
In light of your review, we added a figure showing the differences in gradient norms using distinct initializations.
The main observation made in the paper is that the use of dropout increases the variance of neurons. Correcting for this increase in variance, in the parameter initialization, and in the test-time statistics of batch normalization, improves performance, as is shown reasonably convincingly in the experiments.
This observation is important, as it applies to many of the models used in the literature. It's not extremely novel (it's been observed in the literature before that our simple dropout approximations at test time do not achieve the accuracy obtained by full Monte Carlo dropout)
The paper could use more experimental validation.
Specifically:
- I'm guessing the correction for dropout variance at test time is not only specific to batch normalization: Standard dropout, in networks without batch normalization, corrects only for the mean at test time (by dividing activations by one minus the dropout probability). This work suggests it would be beneficial to also correct for the variance. Has this been tested?
-  How does the dropout variance correction compare to using Monte Carlo dropout at test time? (i.e. just averaging over a large number of random dropout masks)
Thank you for your analysis of our paper.
In our experiments, we do not enable dropout during test time since, as we all know, dropout at test time hurts performance.
Then we do not need to correct for variance _during_ testing. We correct for the variance of dropout applied in training either by weight initialization used during training, or batch normalization parameter re-estimation after normal training but before testing. These corrections allow for better training (weight initialization correction) or fixes the trained parameters (batch normalization correction), which affect test time performance. If we are misunderstanding your last two points, let us know.
You adjust the batch normalization parameters after training, to account for the fact that you use dropout during training but not during testing. This can be seen as a more sophisticated version of the naive standard practice of multiplying weights by 1/p at test time, for dropout probability p. For the latter, we know that Monte Carlo dropout (using dropout at test time, but averaging over many random draws) works better: how does this compare to your method of adjusting the parameters?
> This can be seen as a more sophisticated version of the naive standard practice of multiplying weights by 1/p at test time.
We multiply by 1/p during training and not during testing, as it is the default in Tensorflow's and Lasagne's dropout implementation. Multiplying by 1/p during training corrects the mean but not the variance, as shown in our derivation section.
Thus before we adjust Batch Normalization variance parameters we already have the mean corrected, and this mean correction could have been accomplished with the naive 1/p rescale or Monte Carlo dropout.
Consequently, our variance adjustment is not serving as a sophisticated stand-in for a 1/p rescale since that rescale accounts the mean and our BN variance adjustment accounts for the variance.
This paper proposes new initialization for particular architectures and a correction trick to batch normalization to correct variance introduced by dropout.
While authors state interesting observations, the claims are not supported with convincing results.
I guess Figure 1 is only for mnist and for only two values of p with one particular network architecture, the dataset and empirical setup is not clear.
The convergence is demonstrated only for three dropout values in Figure 2 which may cause an unfair comparison.
For instance how does the convergence compare for the best dropout rate after cross-validation (three figures each figure has three results for one method with different dropouts [bests cv result for each one])?
Also how is the corresponding validation error and test iterations?
Also only mnist does not have to generalize to other benchmarks.
Figure 3 gives closer results for Adam optimizer, learning rate is not selected with random search or bayesian optimization, learning decay iterations fixed and regularization coefficient is set to a small value without tuning. A slightly better tuning of parameters may close the current gap.
Also Nesterov based competitor gives unreasonably worse accuracy compared to recent results which may indicate that this experiment should not be taken into account.
In Table 2, there is no significant improvement on CIFAR10. The CIFAR100 difference is not significant without including batch normalization variance re-estimation.
However there is no result for 'original with BN update' therefore it is not clear whether the BN update helps in general or not. SVHN also does not have result for original with BN update.
There should be baselines with batch normalizations for Figure 1,2 3 to support the claims convincingly.  The main criticism about batch normalization is additional computational cost by giving (Mishkin et al, 2016 ) as reference however this should not be a reason to not to compare the initialization to batch-normalization.
In fact, (Mishkin et al, 2016) performs comparison to batch normalization and also with and without data augmentation with recent state of art architectures.
None of the empirical results have data augmentation. It is not clear if the initialization or  batch normalization update will help or make it worse for that case.
Recent state of art methods methods like Res Net variant and Dense Net scale to many depths and report result for ImageNet.
Although the authors claim that this can be extended to residual network variants, it is not clear if there is going to be any empirical gain for that architectures.
This work requires a comprehensive and fair comparison.
Otherwise the contribution is not significant.
Thank you for your analysis of our paper.
We have added more detail to our synthetic experiment set-up in the new paper.
Following your suggestion about more random hyperparameter search, we re-ran the MNIST experiments tuning over {10^-3, 10^-4, 10^-5} and randomly drew 7 learning rates from 10^Unif[-1,-5]. We then selected the best curves based on validation results. The plots have been updated to include test log loss curves as well.
Your suggestion about tuning the learning rate with random hyperparameter search was indeed executed for the Adam and Nesterov experiments. Other parameters (depth, number of filters, etc.) were inherited from the popular VGG Net implementation. Note that He et al. [1] demonstrate that small weight initialization scalars leave some networks untrainable when there is no batch normalization (Figure 3), so to us it is not unreasonable to expect that different initializations in our Nesterov experiment have noticeably distinct convergence curves.
> However there is no result for 'original with BN update' therefore it is not clear whether the BN update helps in general or not. SVHN also does not have result for original with BN update.
This is because the "original with BN update" values are from the original DenseNet paper [2], and we do not have their model saved at the midpoint of training.
For that reason we re-ran their experiments with and without the Batch Normalization update all while using the setup described in their paper. The "Ours" rows show the consequence of using Batch Normalization re-estimation method and the test error without the re-estimation. Note that this lets us see the exact effects of Batch Normalization variance re-estimation because we show how the model performs on the test set and how that _same_ model performs if we only modify the model's Batch Normalization variance parameters keeping all other parameters the same.
> The CIFAR100 difference is not significant without including batch normalization variance re-estimation.
The CIFAR-100 experiment is designed to show the effect of batch normalization variance re-estimation. Without the batch normalization variance re-estimation, there are no appreciable differences between the original setup and our experiments. We are not testing our weight initialization in this section, only the batch normalization variance re-estimation.
Thank you for helping the paper become stronger.
Update: Anonymous Reviewer 2 edited their initial review on Jan 23rd to include new concerns.
> compare the initialization to batch-normalization
We compare our initialization to other initializations involving simple scalars, as many still use "Xavier" and "He" initializations. Batch Normalization significantly reduces the need for a good initialization, so we give the initialization and batch normalization dropout correction methods separate sections. It is still worthwhile to consider the simple scalar initializations because people may opt not to use Batch Normalization if they want to save computational time, do online learning, or train a ConvNet for reinforcement learning (since Batch Normalization introduces much noise to training in RL). This is why we show how to correct for dropout's influence against other initializations with simple scalars (Xavier, He), and then we show how to correct for dropout's influence in Batch Normalization separately.
> Dense Net scale to many depths, ... it is not clear if there is going to be any empirical gain for that
We trained on a 100-layer DenseNet with growth rate k=12, shown in Table 2, and this has 7 million parameters. Larger DenseNets are almost never more than twice this depth. Their larger DenseNet required 26GB of memory and to be trained on 4 Titan X's, according to a correspondence with Zhuang Liu, the DenseNet author. This was simply too bloated for us to train, and we were still able to surpass its performance with batch normalization variance re-estimation.
> Table 2 is not significant (paraphrase)
We respectfully disagree. Going from 5.77% to 5.38% (note we are approaching an accuracy ceiling) and 23.79% to 22.17% (both state of the art) with a simple, quick, and highly general trick is significant to us.
[1] Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. https://arxiv.org/abs/1502.01852
[2] Densely Connected Convolutional Networks. https://arxiv.org/pdf/1608.06993v1.pdf
Despite clarifications and revisions, the empirical analysis of the paper is still unsatisfactory. For initialization experiments, Figure 1 and 2 are restricted to mnist;  only considering VGG  Net for Figure 3 and 4 is restrictive. The initialization should be also compared to previous work   (Ioffe & Szegedy, 2015; Mishkin & Matas, 2015; Salimans & Kingma, 2016). For the normalization variance reestimation, Table 2 does not demonstrate significant performance gain;  ImageNet results could help.
Also results with data augmentation should be given for the sake of completeness. A comprehensive analysis like the one in (Mishkin et al, 2016 ) is required to judge whether proposed approaches give a significant gain in practice.
We have updated the paper. For the updated paper, we re-ran the MNIST experiments with random hyperparameter search and now plot the log-loss of the test set in addition to the training set log-loss. We also added more detail about the synthetic experiment set-up.
Last, we included a plot showing the Frobenius norm of the gradient as training progresses under different weight initializations.
Thank you all for your suggestions!

Meta-review:
role: PC
data: comment: This was a borderline paper. However, no reviewers were willing to champion the acceptance of the paper during the deliberation period. Furthermore, in practice, initialization itself is a hyperparameter that gets tuned automatically. To be a compelling empirical result, it would be useful for the paper to include a comparison between the proposed initialization and a tuned arbitrary initialization scale with various tuning budgets. Additionally, other issues with the empirical evaluation brought up by the reviewers were only partially resolved in the revisions. For these reasons, the paper has been recommended for rejection.

Decision: Reject