Title: Representing inferential uncertainty in deep neural networks through sampling
Authors: Patrick McClure
Nikolaus Kriegeskorte
Venue: ICLR 2017
Keywords: Deep learning
Theory
Applications

Abstract: As deep neural networks (DNNs) are applied to increasingly challenging problems, they will need to be able to represent their own uncertainty. Modelling uncertainty is one of the key features of Bayesian methods. Bayesian DNNs that use dropout-based variational distributions and scale to complex tasks have recently been proposed. We evaluate Bayesian DNNs trained with Bernoulli or Gaussian multiplicative masking of either the units (dropout) or the weights (dropconnect). We compare these Bayesian DNNs ability to represent their uncertainty about their outputs through sampling during inference. We tested the calibration of these Bayesian fully connected and convolutional DNNs on two visual inference tasks (MNIST and CIFAR-10). By adding different levels of Gaussian noise to the test images, we assessed how these DNNs represented their uncertainty about regions of input space not covered by the training set. These Bayesian DNNs represented their own uncertainty more accurately than traditional DNNs with a softmax output. We find that sampling of weights, whether Gaussian or Bernoulli, led to more accurate representation of uncertainty compared to sampling of units. However, sampling units using either Gaussian or Bernoulli dropout led to increased convolutional neural network (CNN) classification accuracy. Based on these findings we use both Bernoulli dropout and Gaussian dropconnect concurrently, which approximates the use of a spike-and-slab variational distribution. We find that networks with spike-and-slab sampling combine the advantages of the other methods: they classify with high accuracy and robustly represent the uncertainty of their classifications for all tested architectures.

TL;DR: Dropout- and dropconnect-based Bayesian deep neural networks with sampling at inference better represent their own inferential uncertainty than traditional deep neural networks.

Review discussion:
The results of this analysis are based on a very specific scenario of MNIST with dropout 0.5. Do these results hold across other, more complicated datasets?
Even though 0.5 is common for dropout, this is not always the value that is used. What do these results look like at different values of the dropout parameter?
Can you describe in more detail the calibration test that you apply?
1.	Do these results hold across other, more complicated datasets?
Thank you for this suggestion. It is indeed important to test that these results can be replicated for different training sets and architectures. We have now added a new Results section for a 14 layer CNN trained on CIFAR-10.
Overall, Monte-Carlo (MC) sampling at inference does lead to better calibrated models for both CIFAR-10 and for MNIST. One of the key differences between the MNIST and CIFAR-10 results was that the traditional dropout test-time procedure leads to very inaccurate CIFAR-10 networks (62.53% for Bernoulli Dropout and 75.90% for Gaussian Dropout) when dropout is performed at every layer. If the same networks were tested using MC sampling, they were significantly more accurate (89.81% for MC Bernoulli Dropout and 90.71% for MC Gaussian Dropout). This agrees with the findings of Gal and Gharhamani (https://arxiv.org/abs/1506.02158), who also found that using dropout at every layer can reduce accuracy if MC sampling is not used.
2.
Even though 0.5 is common for dropout, this is not always the value that is used. What do these results look like at different values of the dropout parameter?
In response to the reviewer’s question, we have now evaluated the change in classification error, entropy, and calibration when we varied the dropout parameter for several Bayesian fully-connected networks with different variational distributions.
Although setting the dropout parameter to 0.5 may not have been the optimal choice for every network, doing so led to reasonable results across all the networks. We chose to set the dropout parameter of all networks to the same value to make comparing between the different networks simpler. Plots for this analysis and a discussion of them have now been added to the beginning of the Results section.
3.	Can you describe in more detail the calibration test that you apply?
Sorry that we did not make this clear. We have now added the following to the beginning of the Results section.
We evaluated how calibrated a prediction was by the following procedure: (1) We binned test set predictions by predicted probability. (2) We calculated the percentage of predictions in each predicted-probability bin that correctly predicted a target label. Perfect calibration means that targets predicted with probability z are correct in z times 100% of the cases. We therefore (3) calculated the mean squared calibration error, i.e. the mean across bins of the squared deviations between the bin-mean predicted probability and the proportion of correct predictions in that bin.
The authors study the calibration of neural networks using different variants of dropout and weight noise. They find that sampling during training and testing improves calibration.
Most of the results are not novel and have been discussed previously by Yarin Gal and other authors. What this paper adds is a more systematic evaluation of multiple different variants of dropout.
- A comparison against bootstrapped uncertainty estimates would be useful
- Ian Osband had a paper at NIPS investigating exactly this kind of uncertainty representation in neural nets. He found that dropout represents the risk of the model, but not really the uncertainty. This difference becomes apparent when e.g. the target of the model is bi-modal.
Thank you very much for pointing us to Ian Osband’s NIPS paper, which is very interesting and relevant. A key difference between the previous (e.g. Yarin Gal’s and Ian Osband’s) work and ours is that they are attempting to estimate the uncertainty for the learned parameters given the training data.
However, we focus on evaluating how accurately a network distributes the probability mass in its output probability distributions, particularly for new examples.  This distribution of probability mass is often characterized by entropy, which measures the amount of uncertainty in a probability distribution. Analyzing the distribution of probability mass in the networks’ outputs instead of attempting to estimate the confidence in the parameters learned during training, as done by previous work, is one the contributions of our work. We will discuss the relationship of our results to those of Ian Osband’s paper in the Introduction section of our paper.
- Ian Osband had a paper at NIPS investigating exactly this kind of uncertainty representation in neural nets. He found that dropout represents the risk of the model, but not really the uncertainty. This difference becomes apparent when e.g. the target of the model is bi-modal.
In his NIPS 2016 workshop abstract (http://bayesiandeeplearning.org/papers/BDL_4.pdf), Osband noted the fact that the Bernoulli dropout weight distributions are not updated in a Bayesian manner, so the variance of the Bernoulli dropout weight distributions does not decrease as the amount of training data seen goes to infinity. This means that MC dropout does not model the uncertainty of each parameter estimate. We now discuss this observation in the introduction of our paper.
In MC Bernoulli dropout, two sources of variance are: (1) parameter variance (discussed above) and (2) process variance. For MC Bernoulli dropout, process variance is the variance between the outputs of sampled networks. In the ensemble view of dropout, each of the exponentially many Bernoulli sampled networks is trained on a sampled subset of the training data. If each sampled network, was trained to successfully predict all the training data then the process variance would be zero even though the parameter variance would not change.
Consequently, the “uncertainty” due to process variance does reduce to zero given infinite training data and time (assuming each sample network can learn the function and a solution can be found by optimization) even though the “uncertainty” of the parameters does not decrease. This is like the behavior of bootstrapped neural networks (Osband et al., 2016);
except that there is partial weight sharing between each of the Bernoulli sampled networks.
Per the universal approximation theorem, neural networks can represent arbitrary continuous functions.
From another perspective, neural networks with MC Bernoulli dropout can be viewed as an approximation of Gaussian processes (Gal and Ghahramani, 2015), which are distributions of functions. The “bi-modal” regression training set does not pass the “vertical line test” and is therefore not a function. If part of the data is made to pass the “vertical line test” by bootstrap sampling only one of the two vertically aligned points, the sampled “mode” becomes learnable by a neural network. In the example from Yarin Gal’s website, the MC Bernoulli dropout learns the expected regression value, in this case the mean of the two “modes”. Since the data points are far from this expected regression value, the network learned to have high uncertainty in this region of input space.
This paper presents an empirical evaluation of the effect of training with noise by dropout or dropconnect, on the predictive uncertainty of neural networks and convnets. The conclusion seems to be that applying both training and inference with noise can help the network produce better uncertainty estimates in terms of better calibrated predictions.
Although the experiments were thorough, the issue with an empirical paper like this is that it is very difficult to ensure that the lessons learned will generalize across problems and domains. This paper only investigated two simple image datasets and two neural network architectures on the task of classification. There are several ways in which I think this paper could be made stronger.
First, the problem is not well motivated: why do we care about uncertainty (although I believe we do)? A good application, or a motivation section would be beneficial here.
Next, what about investigating a different domain such as text or speech recognition? Or other problems such as regression? Do the results hold across different domains?
Finally, if we really care about uncertainty, there are a number of other techniques for inference in Bayesian neural networks such as stochastic variational inference using the reparameterization trick, or MCMC methods like stochastic gradient Langevin dynamics. The advantage of dropout is that it’s simple and fast, but do we lose anything by doing this in terms of calibration?
Other than the empirical comparison, there is little novelty to this paper, and therefore I think the conclusions drawn need to be more general or the motivation more compelling. Even addressing a subset of these suggestions would make the paper stronger in my opinion.
In Figure 4 the calibration MSE does not look so robust when MC sampling is not used. Do you have any hypothesis on why MC sampling is so important here?
In Figure 5, the legend says “CNN” when it should just be “NN”.
Also the caption says convolutional, when it should say fully connected.
In Table 3, why are the results so poor for non-MC-based dropout?
Although this agrees with Gal and Ghahramani, this seems to be in direct contradiction to Table 4 of the original JMLR dropout paper by Srivastava et al from 2014.
How is the noise in the test set applied for Figures 5-10? Is it Gaussian noise applied to the pixels?
-In Table 3, why are the results so poor for non-MC-based dropout?
Although this agrees with Gal and Ghahramani, this seems to be in direct contradiction to Table 4 of the original JMLR dropout paper by Srivastava et al from 2014.
In our experience, MC dropout is much more robust than non-MC dropout to changes in the dropout probability and architecture. Particularly for deeper networks with Bernoulli dropout, higher regularization (i.e. dropout probabilities closer to 0.5) results in non-MC dropout inference performing significantly worse than MC dropout for the same training error. The CIFAR-10 network used by Srivastava et al. (2014) contained three convolutional layers followed by two fully connected layers. Input units were kept with 0.9 probability, units in the first two convolutional layers were kept with 0.75 probability, and units in the third convolutional and two fully connected layers were kept with a probability of 0.5. The architecture that we used had thirteen convolutional layers and a fully connected layer with every hidden layer unit being kept with a 0.5 probability. This setup works well when MC sampling is used at inference, but leads to decreased accuracy for non-sampling dropout inference. We now mention this in the CIFAR-10 results section.
-For different dropout probabilities, the calibration MSE does not look so robust when MC sampling is not used. Do you have any hypothesis on why MC sampling is so important here?
One potential reason is that each dropout/dropconnect sampled network is trained to make good probabilistic predictions. MC sampling at inference may take advantage of this by averaging the outputs of dropout/dropconnect sampled networks during inference instead of approximating the ensemble of the trained networks by using the expectation of each layer as done in traditional dropout inference.
-How is the noise in the test set applied for Figures 5-10? Is it Gaussian noise applied to the pixels?
Yes, Gaussian noise was added to the image pixels in z-score space. We have clarified this in the revision.
This paper investigates whether the variational inference interpretation of dropout, as introduced in [Gal & Ghahramani (2016), and Kingma et al (2015)], can lead to good estimates of mode uncertainty outside of the training distribution. This is an area of research that indeed warrants more experimental investigation. One very interesting finding is that MC integration leads to much calibration, thus probably much better out-of-sample prediction, than the more usual.
Critique:
- As explained in Kingma et al (2015), when using continuous posterior distributions over the weights, the dropout rate can be optimized, leading to better regularization.
While the paper is cited in the introduction, this adaptive form of dropout is missing from experiments, without clarification.
- Only the dropout rate p=0.5 was used across experiments, while the optimal rate is problem dependent, as found by earlier published work.
- No new ideas are presented, and the analysis in the paper is quite limited. As it stands, this would be more appropriate for a workshop.
-Only the dropout rate p=0.5 was used across experiments, while the optimal rate is problem dependent, as found by earlier published work.
We agree that optimizing the dropout/dropconnect parameters, either as hyperparmeters or as learnable parameters during training, can lead to improved performance. We decided to use p = 0.5 for all experiments since it: (1) corresponds to maximum regularization in linear layers (Baldi and Sadowski, 2013), (2) it is the most commonly used dropout/dropconnect parameter value, and (3) it allows us to easily compare the different distributions without the added complication of vastly different dropout/dropconnect probabilities.
- As explained in Kingma et al (2015), when using continuous posterior distributions over the weights, the dropout rate can be optimized, leading to better regularization.While the paper is cited in the introduction, this adaptive form of dropout is missing from experiments, without clarification.
We now discuss adaptive dropout/dropconnect parameter learning (Kingma et al., 2015; Louizos, 2015; Gal 2016) in both the introduction and methods sections.
To address the overarching critique that the paper has limited novel results, we have now:
1. Implemented a multiplicative spike-and-slab dropout
2. Shown how (1) is equivalent to optimizing a lower bound of an objective function for spike-and-slab variational inference
3. Shown that multiplicative spike-and-slab dropout combines the advantages of the other methods in a way that is robust across the tested architectures, yielding good regularization in CNNs (like dropout methods) and good uncertainty estimates (like dropconnect methods).

Meta-review:
role: PC
data: comment: The reviewers unanimously recommend rejecting this paper.

Decision: Reject