Title: OMNIBUS DROPOUT FOR IMPROVING THE PROBA-BILISTIC CLASSIFICATION OUTPUTS OF CONVNETS

Abstract: While neural network models achieve impressive classification accuracy across different tasks, they can suffer from poor calibration of their probabilistic predictions. A Bayesian perspective has recently suggested that dropout, a regularization strategy popularly used during training, can be employed to obtain better probabilistic predictions at test time (Gal & Ghahramani, 2016a). However, empirical results so far have not been encouraging, particularly with convolutional networks. In this paper, through the lens of ensemble learning, we associate this unsatisfactory performance with the correlation between the models sampled with dropout. Motivated by this, we explore the use of various structured dropout techniques to promote model diversity and improve the quality of probabilistic predictions. We also propose an omnibus dropout strategy that combines various structured dropout methods. Using the SVHN, CIFAR-10 and CIFAR-100 datasets, we empirically demonstrate the superior performance of omnibus dropout relative to several widely used strong baselines in addition to regular dropout. Lastly, we show the merit of omnibus dropout in a Bayesian active learning application.

1. INTRODUCTION:

Deep neural networks (NNs) achieve state-of-the-art classification accuracy in many applications. However, in real world scenarios like medical diagnosis and autonomous driving, reliable probabilistic predictions are also crucial and need to be considered in assessing performance. Most modern NNs are trained with maximum likelihood to produce point estimates that are often over-confident (Guo et al., 2017) . Bayesian techniques can be used with neural networks to obtain well-calibrated predictions (MacKay, 1992; Neal, 2012) , yet they suffer from significant computational challenges. Thus, recent efforts have been devoted to making Bayesian neural networks more efficient (Blundell et al., 2015; Chen et al., 2014; Wu et al., 2018) . Monte Carlo (MC) dropout (Gal & Ghahramani, 2016a) , a cheap approximate inference technique which obtains uncertainty by performing dropout (Srivastava et al., 2014) at test time, is a popular Bayesian method to obtain uncertainty estimates for NNs.

Despite improvements over deterministic NNs, MC dropout can still produce over-confident predictions (Lakshminarayanan et al., 2017) , particularly with convolutional architectures. In this paper, we propose a simple yet effective solution to this problem. Inspired by the recent success of explicit ensembles of neural networks obtained using random initializations (Beluch et al., 2018) , we reiterate the original notion of dropout as "an extreme form of model combination with extensive parameter sharing" (Srivastava et al., 2014) , and interpret MC dropout as an ensemble of models. Borrowing machinery from ensemble learning, we then attribute the poor performance of MC dropout to its limited model diversity compared to that of explicit ensembles. This perspective reveals how structured dropout methods (Ghiasi et al., 2018; Tompson et al., 2015) can improve performance by promoting diversity. While the importance of diversity has been demonstrated by others, prior works consider explicit ensembles of different models. To the best of our knowledge, this is the first paper to examine structured dropout as a way to enhance diversity in an ensemble obtained from a single model. As discussed below, we also propose to combine different structured dropout methods, which we call omnibus dropout. We empirically verify that omnibus dropout can yield models with superior performance on the SVHN, CIFAR-10 and CIFAR-100 datasets compared to not only MC dropout, but also some of the most widely adopted baselines like deep ensembles (Lakshminarayanan et al., 2017) and temperature scaling (Guo et al., 2017) . Furthermore, we demonstrate the merit of better uncertainty estimates in a Bayesian active learning experiment (Gal et al., 2017b) .

2. RELATED WORK:

Dropout was first introduced as a stochastic regularization technique for NNs (Srivastava et al., 2014) . Inspired by the success of dropout, numerous variants have been proposed (Wan et al., 2013; Goodfellow et al., 2013; Tompson et al., 2015; Huang et al., 2016; Singh et al., 2016; Gastaldi, 2017; Ghiasi et al., 2018) . Unlike regular dropout, most of these methods drop parts of the NNs in a structured manner. For instance, DropBlock (Ghiasi et al., 2018) applies dropout to small patches of the feature map in convolutional networks, SpatialDrop (Tompson et al., 2015) drops out entire channels, Stochastic Depth Net (Huang et al., 2016) drops out entire ResNet blocks, and Swapout (Singh et al., 2016) combines the Stochastic Depth Net with regular dropout. These methods were proposed to boost test time accuracy. In this paper, we show that these structured dropout techniques can be successfully applied to obtain better uncertainty estimates as well.

Dropout can be thought of as performing approximate Bayesian inference (Gal & Ghahramani, 2016b) and offer estimates of uncertainty. Many other approximate Bayesian inference techniques have also been proposed for NNs (Kingma et al., 2015; Louizos & Welling, 2017) . However, these methods can demand a sophisticated implementation, are often harder to scale, and can suffer from sub-optimal performance (Blier & Ollivier, 2018) . Another popular alternative to approximate the intractable posterior is Markov Chain Monte Carlo (MCMC) (Neal, 2012) . More recently, stochastic gradient versions of MCMC were also proposed to allow scalability (Gong et al., 2019; Ma et al., 2015; Welling & Teh, 2011) . Nevertheless, these methods are often computationally expensive, and sensitive to the choice of hyper-parameters. A related approach, the SWA-Gaussian (Maddox et al., 2019) is another technique for Gaussian posterior approximation using the Stochastic Weight Averaging (SWA) algorithm (Izmailov et al., 2018) .

There are also non-Bayesian techniques to obtain calibrated confidence estimates. For instance, temperature scaling (Guo et al., 2017) has been empirically shown to be effective in calibrating the predictions. A related line of work uses an ensemble of several randomly-initialized NNs (Lakshminarayanan et al., 2017) . The method, known as deep ensembles, requires training and saving multiple NNs. It has also been demonstrated that an ensemble of snapshots of the trained model at different iterations can help obtain better uncertainty estimates (Geifman et al., 2019) . Compared to an explicit ensemble, this approach requires training only one model. Nevertheless, models at different iterations must all be saved in order to deploy the algorithm, which can be computationally demanding.

3. AN ANALYSIS OF THE PERFORMANCE OF MC DROPOUT:

3.1 MC DROPOUT AS ENSEMBLES OF DROPOUT MODELS Let's assume a dataset D = (X, Y ) = {(x i , y i )} n i=1 , where each (x i , y i ) ∈ (X × Y) is i.i.d.

We consider the problem of k-class classification, and let X ⊆ R d be the input space and Y = {1, • • • , k} be the label space1 . We restrict our attention to NN functions f w (x) : X → R k , where w = {W i } L i=1 corresponds to the parameters of a network with L-layers, and W i corresponds to the weight matrix in the i-th layer. We define a likelihood model p(y|x, w) = softmax(f w (x)). Maximum likelihood estimation can be performed to compute point estimates for w.

Recently, Gal & Ghahramani (2016a) proposed a novel viewpoint of dropout as approximate Bayesian inference (See Appendix A for a brief review). This perspective offers a simple way to marginalize out model weights at test time to obtain better calibrated predictions, which is called MC dropout:

EQUATION

where w (t) ∼ q(w|D train ) is assumed to be independently drawn layer-wise weight matrices:

W (t) i ∼ Ŵi • diag(Bernoulli(p))

, Ŵi is the parameter matrix learned during training, and p is the dropout rate.

In this paper, we view each dropout sample w (t) in Equation 1corresponding to an individual model in an ensemble, where MC dropout is performing (approximate Bayesian) ensemble averaging. The same analysis applies to structured dropout as well. Mathematically, using structured dropout in lieu of regular dropout amounts to only a change of the approximate distribution q(w|D train ) in Equation 1, so that we are performing Bayesian variational inference with a different class of approximate distributions. For instance, in the channel-level dropout, we sample one Bernoulli random variable for each channel.

3.2. DECOMPOSING THE PERFORMANCE OF ENSEMBLES:

First proposed by Krogh & Vedelsby (1995) , the error-ambiguity decomposition enables one to quantify the performance of ensembles with respect to individual models. Let {h t } T t=1 be an ensemble of T classifiers, and H(x) = t h t (x)/T is the ensemble prediction. In classification problems, h t (x) is often a probability vector such that h i t (x) = p(y = i|x, w t ). In MC dropout h t (x) = p(y|x, w (t) ). Model ambiguity can be then defined as:

α(h t |x) = ||h t (x) -H(x)|| 2

2 , which quantifies the difference between an individual model and the ensemble average. The Brier score measures both the accuracy and calibration of probabilistic classifications, and is proportional to mean squared error (MSE), which can be decomposed for an ensemble as:

EQUATION

where MSE(h t |x) = ||y -h t (x)|| 2 , y is the one-hot encoded vector of the correct label y,

MSE(h|x) = 1 T T t MSE(h t |x), and α(h|x) = 1 T T t α(h t |x)

correspond to the average MSE, and ensemble diversity (average ambiguity), respectively. Equation 2suggests that the more accurate and the more diverse the models, the better performance will be achieved by the ensemble. We use MSE instead of the negative log likelihood (NLL), another commonly used measure for quality of uncertainty estimates, due to mathematical convenience. The two metrics are closely related, and insights obtained from MSE carry over to NLL. In general, MSE or NLL can be seen as comprehensive measures influenced by both the accuracy and the calibration of the model. We give a brief discussion in Appendix B on the relationship between these metrics.

3.3. PERFORMANCE OF MC DROPOUT AND MODEL DIVERSITY:

The discussion of the previous section provides us with a potential recipe to enhance MC dropout.

To illustrate the importance of diversity, we conduct an experiment using ResNet-50 on CIFAR-10 to compare MC dropout with an explicit ensemble of five NNs (details can be found in Section 4).

As we see from Figure 1 , individual models in deep ensemble, on average, perform better than the ones in MC dropout, likely because of the reduced effective capacity of the latter. Furthermore, the performance of the ensembles improve with more models. Yet the improvement is larger for deep ensemble, mainly because of increased ensemble diversity, since we know from Equation 2 that the decrease in the Brier score in this analysis is attributable to the increase in ensemble diversity, as the average MSE does not change with the number of models.

The lack of diversity among MC dropout models is largely because neighboring pixel features are often correlated in convolutional layers. Thus, even with dropout, similar information propagates through the network in every iteration. Although we can encourage model diversity naively by increasing dropout rates, this can lead to reduced MSE of individual models, thereby hampering ensemble performance, as can be seen from Eq. 2. This is because higher dropout rates would lead to smaller effective model capacities given a fixed total number of model parameters.

3.4. OMNIBUS DROPOUT:

While model diversity can be promoted via explicit ensembles, they demand much more computational resources during training, which can be prohibitively expensive. Though typically more number of samples is needed for dropout based methods at test time, unlike deep ensembles, dropout uncertainty can be obtained sequentially, which has a lower memory requirement. Moreover, the number of samples needed can potentially be optimized with an adaptive sampling scheme (Inoue, 2019) .

In order to enhance diversity in an ensemble obtained from a single model, we examine the use of structured dropout, which drops information from contiguous regions of feature maps so that more divergent information is propagated to subsequent layers during training at each iteration. This enhancement in diversity of predictions can in turn lead to better performance. Specifically, we compare dropout at the patch-level which randomly drops out small patches of feature maps (Ghiasi et al., 2018) , the channel-level which drops out entire channels of feature maps at random (Tompson et al., 2015) , and layer-level which drops out entire layers of CNNs at random (Huang et al., 2016) .

We denote these as dropBlock, dropChannel and dropLayer respectively. We identify the test-time sampling of models trained with the aforementioned structured dropout methods as MC dropBlock, MC dropChannel, MC dropLayer.

As we empirically observe below, similar to increasing the dropout rate, the increased diversity of structured dropouts can come at the cost of reduced performance of individual models. Moreover, given considerable choices of dropout strategies available, it can be hard to pick the best one. Therefore, we propose a novel omnibus dropout strategy, which merely combines all the aforementioned methods. The implementation of omnibus dropout involves the sequential execution of the nested group of dropout methods: regular dropLayer, dropChannel, dropBlock and regular dropout. In our experiments, we use a constant dropout rate for all the dropout methods. Empirically we find this simple choice to mostly work well. As our results show, omnibus dropout yields good performance by promoting model diversity without hampering the performance of individual models.

4. EXPERIMENTS:

We empirically evaluate the performance of MC dropBlock, MC dropChannel, MC dropLayer and MC omnibus-dropout, and compare them to MC dropout, deep ensembles and temperature scaling2 . Unless otherwise stated, the following experimental setup applies to all of our experiments.

Model. Layer-level dropout requires skip connections so that there is still information flow through the network after dropping out an entire layer. Some of the examples include the FractalNet (Larsson et al., 2017) and the ResNet (He et al., 2016a) . We use the PreAct-Resnet (He et al., 2016b) Datasets. We conduct experiments using the SVHN (Netzer et al., 2011) , CIFAR-10 and CIFAR-100 (Krizhevsky, 2009) datasets with standard train/test-set split. Validation sets of 10000 and 5000 samples are used for SVHN and the CIFARs. To examine the performance of the proposed methods with models of different depth, we use the 18-, 50-and 101-layer PreAct-ResNet for SVHN, CIFAR-10 and CIFAR-100.

Training. We perform preprocessing and data augmentation using per-pixel mean subtraction, horizontal random flip and 32 × 32 random crops after padding with 4 pixels on each side. We used stochastic gradient descent (SGD) with 0.9 momentum, a weight decay of 10 -4 and learning rate of 0.01, and divided it by 10 after 125 and 190 epochs (250 in total) for SVHN and CIFAR-10, and after 250 and 375 (500 in total) for CIFAR-100.

Evaluation. All the results are computed on the test set using the model at the optimal epoch based on validation accuracy. We use the Brier score, negative log-likelihood (NLL), expected calibration error (ECE), and Classification accuracy to evaluate performance (see Appendix B for definitions). Following Naeini et al. (2015) , we partition predictions into 20 equally spaced bins and take a weighted average of the bins' accuracy and confidence difference to estimate ECE. To visualize calibration performance, we also plot the reliability diagrams (Maddox et al., 2019) , which are plots of the difference between accuracy and confidence against confidence. The closer the curve to the X-axis, the more calibrated the model predictions are.

4.1. ENSEMBLE DIVERSITY:

We first investigate model diversity achieved with dropout. For a fair comparison, we fix the dropout rate for all methods to 0.1 so that all models have the same effective number of parameters. There are numerous measures that quantify diversity of model ensembles (Zhou, 2012) . We use Interrater Agreement (IA) Kuncheva & Whitaker (2003) , defined as:

EQUATION

)

where T is the number of individual classifiers, n is the number of test samples, ρ(x k ) is the number of models that classify the k-th sample correctly, and p is average classification accuracy across classifiers. When all classifiers perfectly agree on the test set κ = 1, and smaller values indicate more diverse predictions. Figure 2 summarizes IA for sampled models trained on different datasets with different dropout methods. We also compare the results with deep ensemble. The number of models used to compute IA, T , is fixed to five for all approaches. In general, IA for MC dropout is much higher than structured dropout techniques. On the other hand, structured dropout can yield ensembles that are as diverse as the computationally expensive method of deep ensemble, confirming our expectation that dropping out correlated information can produce sampled models with more ambiguity. Note that the large IA for MC dropLayer on SVHN is likely caused by a relatively small model used for that problem -an 18-layer ResNet. Lastly, note that while MC omnibus-dropout yields models much more diverse than MC dropout, it is often not the most diverse one either.

The moderate diversity of MC omnibus-dropout, we believe, is the key to its effectiveness. To better understand its behavior, we study the performance metrics as a function of number of sampled models in the ensemble. Figure 3 shows the Brier score (left) and accuracy (right) against number of models for the CIFAR-10 dataset (Similar results observed for SVHN and CIFAR-100. See Appendix C). Firstly, as seen from Figure 3 (left), while the performance of individual models sampled from MC dropout is one of the best, the gain in Brier score with a larger number of test-time MC samples is much smaller compared to structured dropout techniques. On the other hand, though a larger diversity indeed leads to much sharper improvements as number of sampled models increases, the Brier scores (hence MSE) of individual models sampled from MC dropBlock, MC dropChannel and MC dropLayer are much larger than that of MC dropout, suggesting a trade-off between diversity and the performance of individual sample models. MC omnibus-dropout which enjoys the benefits from both structured and regular dropouts, is able to not only achieve good performance on one sampled model (with a Brier score close to MC dropout), but also good model diversity as evident by a significantly larger decrease in Brier score as number of models increases. Similar observations can be made from the accuracy plot of Figure 3 (right).

4.2. PERFORMANCE EVALUATION:

Table 1 summarizes the performance metrics produced by various models. To ensure a fair comparison, we treat the dropout rate as a hyper-parameter and conduct a linear grid search with 0.05 interval for optimal dropout rate based on NLL. The optimal dropout rates are shown in the table next to methods. Standard deviations are obtained on five models with random initializations for all dropout models, and bootstrapping the test sets for deep ensembles. As seen from Table 1 and Figure 4 , all forms of structured dropout models offer better uncertainty estimates than MC dropout in general. Overall, MC omnibus-dropout and deep ensembles are the best performing models. Remarkably, MC omnibus-dropout achieves even better uncertainty estimates on SVHN and CIFAR-10 and very comparable ones on CIFAR-100 compared to deep ensembles which requires five times more computational resource to train. Moreover, we also perform experiments with five explicit ensembles of models trained together with all types of dropout for a fair comparison, and most of the dropout models outperform deep ensembles trianed without dropout. Again, omnibus dropout is consistently one of the best methods (See Appendix C). Lastly, as evident from moderately increased classification accuracy over deterministic temperature scaling models, all types of dropout methods can be incorporated into architectures for uncertainty estimates with no accuracy penalty.

We believe the relatively good performance of MC dropout on SVHN compared to CIFARs is because the former task is easier so that the model can still predict accurately at an aggressive dropout rate of 0.35 at which even regular dropout can produce acceptably diverse sampled models. In contrast, as observed in our experiments, while using larger dropout rates for the more difficult CIFAR datasets can lead to more calibrated predictions, accuracy and NLL suffer due to drop in MSE of individual models (see Appendix C). Lastly, we believe the results for MC dropBlock can be improved by optimizing the choice of block size. A pre-fixed block size of 3 × 3 can be too small for the upstream convolutional layers where the size of feature maps are much larger than the block size, and too large for the last few downstream layers where the feature maps are comparable to the block size, as supported by sharp increases in NLL after the optimal dropout rate. 

4.3. BAYESIAN ACTIVE LEARNING:

To further demonstrate the merit of omnibus dropout, we consider the downstream task of Bayesian active learning on CIFAR-10. Active learning involves first training on a small amount of labeled data. Then, an acquisition function based on the outputs of models is used to select a small subset of unlabeled data so that an oracle can provide labels for these queried data. Samples that a model is the least confident about are usually selected for labeling, in order to maximize the information gain.

The model is then retrained with the additional labeled data that is provided. The above process can be repeated until a desired accuracy is achieved or the labeling resources are exhausted.

In our experiment, we train models with structured dropout at different scales using the identical setup as described in the beginning of this section, except that only 2000 training samples are used initially. To match up model capacity, the dropout rate is set to 0.1 for all methods. We also compare again a deterministic model. After the first iteration, we acquire 1000 samples from a pool of "unlabeled" data, and combine the acquired samples with the original set of labeled images to retrain the models. Following Gal et al. (2017b) process eight times so that in the last iteration, the training set contains 10000 images. To mimic a real world scenario in which number of labeled samples is small, we do not use a validation set, and the accuracies reported for this experiment correspond to the last-epoch accuracies. We repeat experiments five times for consistency.

Figure 5 shows the test accuracy against number of training samples for different models. In general, MC omnibus-dropout yields the best performance by far. Interestingly, MC omnibus-dropout is able to outperform all other methods consistently by a significant margin after the first iteration when all samples are randomly selected. In addition, it can be seen that, after the first iteration when all 2000 training images are randomly selected, the test accuracy using MC dropout is on par with that of other structured dropout methods. However, as more labeled data are added, the relative increase in accuracy is more significant for models using structured dropout compared to that of using regular dropout. This suggests that the uncertainty estimates obtained with structured dropout are more useful for assessing "what the model doesn't know", thereby allowing for the selection of samples to be labeled in a way that better helps improve performance. Note also that the comparative gain in accuracy by MC omnibus-dropout during the later stages of the learning process is not as large. We suspect this can be caused by the saturation effect on test accuracy.

5. CONCLUSION AND FUTURE WORK:

We reinterpret MC dropout as ensemble averaging strategy, and attribute its poor performance in convolutional neural networks to a lack of diversity of sampled models using the error-ambiguity decomposition of the Brier score (or MSE), a widely used performance metric that captures both accuracy and calibration of probabilistic outputs. As we demonstrate empirically, omnibus dropout, which is simple-to-implement and computationally efficient, strikes the right balance between model diversity among sampled models while retaining reasonable performance of individuals models, thereby consistently improving the quality of the ensemble's prediction.

We are interested in further exploring several directions. Firstly, we have only considered uniform individual dropout rates within the omnibus dropout strategy. Learning the optimal dropout rates for each type of dropout, possibly by building on the the work of (Gal et al., 2017a) , can potentially further improve the performance of omnibus dropout. Moreover, we used a constant dropout rate in our experiments, even though one can vary dropout rates across NNs (Huang et al., 2016) or incorporate dropout rate scheduling (Ghiasi et al., 2018) . How this impacts the quality of the probabilistic predictions is an open question. Lastly, we have only explored structured dropout in the context of CNNs, with application to computer vision tasks. We believe this idea can be extended beyond CNNs. which measures the expected difference between the true class probability and the confidence of the model (Kuleshov & Liang, 2015) . In addition to the error-ambiguity decomposition that we have discussed, MSE can also be decomposed as:

EQUATION

where E y [y|H(x)] corresponds to the true probability of y = 1 conditioned on H(x).

Var x [E y [y|H(x)]] measures the variation of the true class probabilities across the level-sets of the ensemble model H Kuleshov & Liang (2015) . Thus for this metric, the numeric values of H(x) are not important. It is minimized if H(x) is a constant and maximized when H(x) = f (y), for any bijective function f . One can therefore view Var x [E y [y|H(x)]] as a weak metric of accuracy that is not sensitive to calibration. Note Var x [y] does not depend on the models. Brier score thus can be seen as a metric that is influenced by both the accuracy and the ECE of the models. Similarly, NLL is a metric closely related Brier score on a log scale. Consequently, sometimes better uncertainty estimates in terms of NLL or Brier score can lead to slight drops in accuracy, as the reduction in calibration error outweighs increase in classification error. This phenomenon is indeed observed in practice as well. Figure 7 shows the plot of both NLL and accuracy against dropout rates for all dropout methods considered in the paper. For instance, it can be seen that while increasing the dropout rate for the MC dropout model on CIFAR-100 dataset from 0.1 to 0.2 leads to a reduction in NLL, there is also quite a significant dropout in classification accuracy. Similar trends can be seen for MC dropChannel on CIFAR-10 as well. Nevertheless, the trade-off is not always present. To exemplify, increasing dropout rate of MC dropout on the SVHN dataset also leads to an increase in accuracy as well. In conclusion, when tuning for the optimal dropout rate in practice, it can be beneficial to look at different metrics for a holistic consideration. As discussed in Section 4.1, patterns similar to the plots obtained on the CIFAR-10 dataset in Figure 3 are also observed consistently here. The only exception is to the MC dropLayer model on the SVHN dataset, which obtains better performance on individual model but much smaller improvements in both Brier score and test accuracy compared to the other the other dropout methods. We would like to highlight out that the seemingly contradictory results is likely caused by the shallow network used, an 18-layer ResNet. As no down-sampling layers are dropped out for layer dropout, the effective number of ResNet blocks that can be dropped is very small, leading to a much smaller dropout rate compared to ther other methods. This is not an issue with deeper models in which the number of downsampling blocks are much more than that of non-downsampling ones. Additional Results Using Explicit Dropout Ensembles. Smith & Gal (2018) demonstrated that explicit ensembles with MC dropout models produce better uncertainty estimates than that of deep ensembles. Thus we examine the effectiveness of ensembling multiple explicit models trained with structured dropout. This would serve as a fair comparison to deep ensemble. We use five explicit models, each trained with random initialization. drop_rate = 0.1 is used for all methods as it is impractical to tune for the optimal dropout rates for an ensemble of five models. Note that the optimal dropout rates found for individual models do not carry over to explicit ensembles of five models, as we observe in our experiments. At test time, we generate six sampled models from each dropout models (30 samples in total).

Results obtained are summarized in Table 3 . Similar to previous findings, ensembles with omnibus dropout consistently outperform all the rest. Moreover, ensembles with all types of structured dropout methods do better than MC dropout and deep ensembles. Deep ensembles does the worst in terms of uncertainty estimates.

We also investigate the sensitivity of the methods to the choice of dropout rate. To that end, we also report the results obtained with a single model for each method, with a fixed dropout rate of 0.1, a reasonable default value for dropout rate in general. The results are shown in Table 2 . Possibly due to the combination of all dropout method, omnibus dropout seems to be also relatively insensitive to the choice of dropout rate, performing well in all the experiments.

Results on Tuning the Dropout Rate. Figure 7 illustrates the plots of NLL and accuracy against dropout rate for all models on all of the datasets. As discussed in Appendix B, conflict between NLL and accuracy can occur sometimes. Interestingly, the NLL drastically increases after minima on all three datasets for dropBlock, suggesting the possibility that the block size for dropBlock may be too large towards later convolutional layers when the size of feature maps are comparable to that of block size.

:

Extension to regression tasks is straightforward but left out of this paper.

See Appendix C for results on explicit dropout ensembles.

In their experiments, ResNet blocks are only dropped out during testing, but not training.

Figures and Tables:

Figure 1: From left to right (1) Accuracy of MC dropout and deep ensemble (2) the relative improvements in accuracy of deep ensemble and MC dropout (3) Brier score of MC dropout and deep ensemble against number of models (4) the relative improvements in in Brier score of deep ensemble and MC dropout against number of models. Difference grows as number of models increases.

Figure 2: Interrater Agreement (IA) of models with different types of dropout with 0.1 dropout rate on the SVHN, CIFAR-10 and -100 datasets. The lower the IA, the more diverse the predictions of the models. Y-axis indicates different methods. MC dropout produces models with much larger IA, hence less model diversity, than structured dropout techniques in most of the cases.

Figure 3: Test Brier score (left) and accuracy (right) against number of models for ensemble prediction at test time on CIFAR-10. This corresponds to the number of different MC dropout instantiations at test time of the same model. The Model trained with omnibus dropout achieves the best in terms of accuracy and Brier score.

Figure 4: Reliability diagrams of predictions produced by difference models.

Figure 5: Left: Test accuracy against number of training samples for models with different methods of dropout and Variation Ratios as the acquisition function on CIFAR-10. Right: Relative improvements in test accuracy over that of the first iteration with different methods of dropout.

Figure 6: Test Brier score (left) and accuracy (right) against number of models for ensemble prediction at test time on SVHN and CIFAR-100. This corresponds to the number of different MC dropout instantiations at test time of the same model. The Model trained with omnibus dropout achieves the best in terms of accuracy and Brier score.

Figure 7: Plots of test time NLL (left) and accuracy (right) against dropout rate for models trained with different types of dropout on the SVHN, CIFAR-10 and CIFAR-100 datasets.

for all our experiments. We refer to the preAct-ResNet trained without dropout as a deterministic model. MC dropout, MC dropBlock and MC dropChannel models are implemented through inserting the corresponding dropout layers with a constant p before each convolutional layer. A block size of 3 × 3 is used for MC dropBlock. We followGhiasi et al. (2018) to match up the effective dropout rate of MC dropBlock to the desired dropout rate p. MC dropLayer is implemented through randomly dropping out entire ResNet blocks at a constant rate p. We empirically observe that, dropping out downsampling ResNet blocks during testing is harmful to the quality of uncertainty estimates. This is in agreement with experiments ofVeit et al. (2016) 3 . Hence, downsampling blocks are only dropped out during training. MC omnibus-dropout is implemented by including all types of aforementioned dropouts, each with the same dropout rate. For a full Bayesian treatment, we also insert a dropout

Table content: <table><tr><td>0.575 0.625 0.675 0.725 Interrater Agreement SVHN</td><td>0.50 0.55 0.60 0.65 Interrater Agreement CIFAR-10</td><td>0.65</td><td>0.70 Interrater Agreement 0.75 CIFAR-100</td><td>0.80</td><td>dropout dropBlock dropChannel dropLayer dropOmnibus deep ensemble</td></tr></table>

Results on benchmark datasets comparing accuracy and uncertainty estimates produced by different types of methods. The top-2 performing results for each metric are bold-faced. MC omnibus-dropout is consistently one of the best methods, outperforming even deep ensembles, which requires five times more computational resources, in many cases. The numbers in bracket next to dropout methods corresponds to the optimal drop_rate found by grid search using the NLL metric.

Table content: <table><tr><td/><td colspan="2">Datasets</td><td colspan="2">Methods</td><td/><td colspan="3">Accuracy ↑</td><td colspan="2">NLL ↓</td><td/><td colspan="6">Brier ↓ (×10 -3 ) ECE ↓ (×10 -2 )</td></tr><tr><td/><td/><td/><td colspan="2">Temp Scaling</td><td/><td/><td colspan="2">95.7 ± 0.1</td><td colspan="3">0.163 ± 0.002</td><td colspan="3">6.62 ± 0.10</td><td colspan="3">0.995 ± 0.160</td></tr><tr><td/><td/><td/><td colspan="3">Deep Ensemble</td><td/><td colspan="2">96.6 ± 0.1</td><td colspan="3">0.179 ± 0.009</td><td colspan="3">5.39 ± 0.16</td><td colspan="2">1.08 ± 0.08</td></tr><tr><td/><td>SVHN</td><td/><td colspan="3">Dropout (0.35)</td><td/><td colspan="2">96.7 ± 0.1</td><td colspan="3">0.128 ± 0.001</td><td colspan="3">5.11 ± 0.06</td><td colspan="3">0.934 ± 0.045</td></tr><tr><td/><td/><td/><td colspan="3">DropBlock (0.1)</td><td colspan="3">96.8 ± 0.1</td><td colspan="3">0.133 ± 0.002</td><td colspan="3">5.19 ± 0.07</td><td colspan="2">1.26 ± 0.14</td></tr><tr><td/><td/><td/><td colspan="3">DropChannel (0.2)</td><td/><td colspan="2">96.7 ± 0.1</td><td colspan="3">0.130 ± 0.001</td><td colspan="3">5.15 ± 0.06</td><td colspan="3">0.799 ± 0.032</td></tr><tr><td/><td/><td/><td colspan="3">DropLayer (0.25)</td><td/><td colspan="2">96.3 ± 0.1</td><td colspan="3">0.144 ± 0.002</td><td colspan="3">5.69 ± 0.05</td><td colspan="3">0.846 ± 0.250</td></tr><tr><td/><td/><td/><td colspan="9">Omnibus dropout (0.15) 96.9 ± 0.1 0.127 ± 0.001</td><td colspan="3">4.97 ± 0.09</td><td colspan="2">1.15 ± 0.06</td></tr><tr><td/><td/><td/><td colspan="2">Temp Scaling</td><td/><td/><td colspan="2">93.9 ± 0.1</td><td colspan="3">0.189 ± 0.002</td><td colspan="3">9.06 ± 0.08</td><td colspan="3">0.905 ± 0.114</td></tr><tr><td/><td/><td/><td colspan="3">Deep Ensemble</td><td colspan="6">95.2 ± 0.2 0.181 ± 0.009</td><td colspan="3">7.40 ± 0.28</td><td colspan="2">1.40 ± 0.16</td></tr><tr><td/><td colspan="2">CIFAR10</td><td colspan="2">Dropout (0.2)</td><td/><td/><td colspan="2">93.1 ± 0.1</td><td colspan="3">0.224 ± 0.003</td><td colspan="3">10.2 ± 0.1</td><td colspan="2">1.64 ± 0.07</td></tr><tr><td/><td/><td/><td colspan="3">DropBlock (0.1)</td><td/><td colspan="2">93.4 ± 0.1</td><td colspan="3">0.203 ± 0.003</td><td colspan="3">9.89 ± 0.10</td><td colspan="3">0.743 ± 0.116</td></tr><tr><td/><td/><td/><td colspan="3">DropChannel (0.15)</td><td/><td colspan="2">93.7 ± 0.1</td><td colspan="3">0.193 ± 0.002</td><td colspan="3">9.34 ± 0.9</td><td colspan="3">0.812 ± 0.104</td></tr><tr><td/><td/><td/><td colspan="3">DropLayer (0.1)</td><td/><td colspan="2">94.0 ± 0.2</td><td colspan="3">0.206 ± 0.001</td><td colspan="3">9.09 ± 0.17</td><td colspan="3">0.941 ± 0.068</td></tr><tr><td/><td/><td/><td colspan="9">Omnibus dropout (0.1) 94.4 ± 0.1 0.173 ± 0.001</td><td colspan="3">8.38 ± 0.10</td><td colspan="3">0.607 ± 0.078</td></tr><tr><td/><td/><td/><td colspan="2">Temp Scaling</td><td/><td/><td colspan="2">74.5 ± 0.3</td><td colspan="2">1.00 ± 0.01</td><td/><td colspan="3">3.57 ± 0.04</td><td colspan="2">4.02 ± 0.62</td></tr><tr><td/><td/><td/><td colspan="3">Deep Ensemble</td><td colspan="6">77.9 ± 0.4 0.922 ± 0.019</td><td colspan="3">3.12 ± 0.05</td><td colspan="2">5.10 ± 0.33</td></tr><tr><td colspan="3">CIFAR100</td><td colspan="2">Dropout (0.2)</td><td/><td/><td colspan="2">74.1 ± 0.4</td><td colspan="2">1.18 ± 0.01</td><td/><td colspan="3">3.71 ± 0.05</td><td colspan="2">9.18 ± 0.23</td></tr><tr><td/><td/><td/><td colspan="3">DropBlock (0.15)</td><td/><td colspan="2">73.7 ± 0.5</td><td colspan="2">1.04 ± 0.02</td><td/><td colspan="3">3.66 ± 0.05</td><td colspan="2">4.46 ± 0.97</td></tr><tr><td/><td/><td/><td colspan="3">DropChannel (0.15)</td><td/><td colspan="2">74.9 ± 0.5</td><td colspan="2">0.996 ± 0.02</td><td/><td colspan="3">3.46 ± 0.04</td><td colspan="2">3.17 ± 0.11</td></tr><tr><td/><td/><td/><td colspan="3">DropLayer (0.25)</td><td colspan="3">75.7 ± 0.2</td><td colspan="2">1.01 ± 0.01</td><td/><td colspan="3">3.42 ± 0.03</td><td colspan="3">2.90 ± 0.24</td></tr><tr><td/><td/><td/><td colspan="3">Omnibus dropout (0.25)</td><td/><td colspan="2">75.3 ± 0.2</td><td colspan="3">0.929 ± 0.005</td><td colspan="3">3.40 ± 0.02</td><td colspan="3">1.65 ± 0.21</td></tr><tr><td>Confidence -Accuracy</td><td>0.5 0.10 0.05 0.00 0.05 0.10 0.15 0.20 0.25</td><td colspan="2">0.6 deterministic 0.7 Confidence 0.8 SVHN tempScaling dropout dropBlock dropChannel dropLayer dropOmnibus Deep Ensemble</td><td>0.9</td><td>1.0</td><td>0.05 0.00 0.05 0.10 0.15 0.20</td><td>0.5</td><td>0.6</td><td>0.7 Confidence 0.8 CIFAR-10</td><td>0.9</td><td>1.0</td><td>0.1 0.0 0.1 0.2 0.3 0.4</td><td>0.2</td><td>0.4</td><td>0.6 Confidence CIFAR-100</td><td>0.8</td><td>1.0</td></tr></table>

Results comparing accuracy and uncertainty estimates obtained using a single model when drop_rate = 0.1 for all models. The top-2 performing results for each metric is bold-faced. MC omnibus-dropout is the best method in general.

Table content: <table><tr><td>Datasets</td><td>Methods</td><td>Accuracy ↑</td><td>NLL ↓</td><td colspan="2">Brier ↓ (×10 -3 ) ECE ↓ (×10 -2 )</td></tr><tr><td/><td>Temp Scaling</td><td>95.7 ± 0.1</td><td>0.163 ± 0.002</td><td>6.62 ± 0.10</td><td>0.995 ± 0.160</td></tr><tr><td/><td>Dropout</td><td>96.4 ± 0.1</td><td>0.179 ± 0.004</td><td>5.68 ± 0.07</td><td>1.34 ± 0.10</td></tr><tr><td>SVHN</td><td>DropBlock</td><td colspan="2">96.8 ± 0.1 0.133 ± 0.002</td><td>5.19 ± 0.07</td><td>1.26 ± 0.14</td></tr><tr><td/><td>DropChannel</td><td>96.5 ± 0.1</td><td>0.148 ± 0.002</td><td>5.41 ± 0.04</td><td>0.663 ± 0.050</td></tr><tr><td/><td>DropLayer</td><td>96.2 ± 0.1</td><td>0.154 ± 0.002</td><td>5.94 ± 0.10</td><td>1.13 ± 0.10</td></tr><tr><td/><td colspan="3">Omnibus dropout 96.8 ± 0.1 0.133 ± 0.003</td><td>5.07 ± 0.07</td><td>0.616 ± 0.077</td></tr><tr><td/><td>Temp Scaling</td><td>93.9 ± 0.1</td><td>0.189 ± 0.002</td><td>9.06 ± 0.08</td><td>0.905 ± 0.114</td></tr><tr><td/><td>Dropout</td><td>93.8 ± 0.1</td><td>0.226 ± 0.008</td><td>9.44 ± 0.10</td><td>2.30 ± 0.09</td></tr><tr><td>CIFAR10</td><td>DropBlock</td><td>93.4 ± 0.1</td><td>0.203 ± 0.003</td><td>9.89 ± 0.10</td><td>0.743 ± 0.116</td></tr><tr><td/><td>DropChannel</td><td>93.7 ± 0.1</td><td>0.196 ± 0.006</td><td>9.20 ± 0.136</td><td>0.970 ± 0.171</td></tr><tr><td/><td>DropLayer</td><td>94.0 ± 0.2</td><td>0.206 ± 0.001</td><td>9.09 ± 0.17</td><td>0.941 ± 0.068</td></tr><tr><td/><td colspan="3">Omnibus dropout 94.4 ± 0.1 0.173 ± 0.001</td><td>8.38 ± 0.10</td><td>0.607 ± 0.078</td></tr><tr><td/><td>Temp Scaling</td><td>74.5 ± 0.3</td><td>1.00 ± 0.01</td><td>3.57 ± 0.04</td><td>4.02 ± 0.62</td></tr><tr><td/><td>Dropout</td><td>74.8 ± 0.4</td><td>1.21 ± 0.01</td><td>3.71 ± 0.05</td><td>11.1 ± 0.4</td></tr><tr><td>CIFAR100</td><td>DropBlock</td><td>75.6 ± 0.2</td><td>1.04 ± 0.01</td><td>3.46 ± 0.02</td><td>6.98 ± 0.19</td></tr><tr><td/><td>DropChannel</td><td>75.3 ± 0.2</td><td>1.02 ± 0.01</td><td>3.43 ± 0.03</td><td>5.57 ± 0.08</td></tr><tr><td/><td>DropLayer</td><td>75.8 ± 0.3</td><td>1.04 ± 0.02</td><td>3.46 ± 0.04</td><td>7.42 ± 0.32</td></tr><tr><td/><td colspan="2">Omnibus dropout 76.3 ± 0.1</td><td>1.00 ± 0.01</td><td>3.37 ± 0.02</td><td>7.11 ± 0.20</td></tr></table>

Results showing accuracy and uncertainty estimates produced by different types of explicit ensembles when drop_rate = 0.1 for all models. five models are used for each ensemble. We generate 6 sampled models from each dropout models during evaluation (30 samples in total). The top-2 performing results for each metric is bold-faced.

Table content: <table><tr><td>Datasets</td><td>Methods</td><td>Accuracy ↑</td><td>NLL ↓</td><td colspan="2">Brier ↓ (×10 -3 ) ECE ↓ (×10 -2 )</td></tr><tr><td/><td>Deep Ensemble</td><td>96.6 ± 0.1</td><td>0.179 ± 0.009</td><td>5.39 ± 0.16</td><td>1.08 ± 0.08</td></tr><tr><td/><td>Dropout</td><td>97.0 ± 0.1</td><td>0.141 ± 0.006</td><td>4.82 ± 0.15</td><td>0.736 ± 0.077</td></tr><tr><td>SVHN</td><td>DropBlock</td><td colspan="2">97.2 ± 0.1 0.125 ± 0.004</td><td>4.79 ± 0.13</td><td>1.86 ± 0.09</td></tr><tr><td/><td>DropChannel</td><td>97.0 ± 0.1</td><td>0.129 ± 0.004</td><td>4.82 ± 0.14</td><td>0.949 ± 0.082</td></tr><tr><td/><td>DropLayer</td><td>96.8 ± 0.1</td><td>0.132 ± 0.005</td><td>4.91 ± 0.14</td><td>0.575 ± 0.077</td></tr><tr><td/><td colspan="3">Omnibus dropout 97.2 ± 0.1 0.122 ± 0.004</td><td>4.61 ± 0.13</td><td>1.05 ± 0.07</td></tr><tr><td/><td>Deep Ensemble</td><td>95.2 ± 0.2</td><td>0.181 ± 0.009</td><td>7.40 ± 0.28</td><td>1.40 ± 0.16</td></tr><tr><td/><td>Dropout</td><td>94.4 ± 0.2</td><td>0.176 ± 0.008</td><td>8.17 ± 0.29</td><td>1.04 ± 0.16</td></tr><tr><td>CIFAR10</td><td>DropBlock</td><td>94.0 ± 0.2</td><td>0.185 ± 0.006</td><td>9.12 ± 0.28</td><td>1.51 ± 0.18</td></tr><tr><td/><td>DropChannel</td><td>94.3 ± 0.2</td><td>0.174 ± 0.007</td><td>8.35 ± 0.29</td><td>0.900 ± 0.152</td></tr><tr><td/><td>DropLayer</td><td>94.8 ± 0.2</td><td>0.173 ± 0.006</td><td>7.88 ± 0.26</td><td>2.04 ± 0.17</td></tr><tr><td/><td colspan="3">Omnibus dropout 94.8 ± 0.2 0.160 ± 0.006</td><td>7.82 ± 0.27</td><td>0.953 ± 0.156</td></tr><tr><td/><td>Deep Ensemble</td><td>78.0 ± 0.4</td><td>0.923 ± 0.020</td><td>3.12 ± 0.05</td><td>5.12 ± 0.34</td></tr><tr><td/><td>Dropout</td><td>77.5 ± 0.4</td><td>0.931 ± 0.020</td><td>3.18 ± 0.05</td><td>4.51 ± 0.32</td></tr><tr><td>CIFAR100</td><td>DropBlock</td><td>77.3 ± 0.4</td><td>0.909 ± 0.019</td><td>3.17 ± 0.06</td><td>3.23 ± 0.31</td></tr><tr><td/><td>DropChannel</td><td>77.1 ± 0.4</td><td>0.871 ± 0.017</td><td>3.13 ± 0.04</td><td>2.24 ± 0.28</td></tr><tr><td/><td>DropLayer</td><td colspan="2">78.1 ± 0.4 0.855 ± 0.017</td><td>3.05 ± 0.04</td><td>2.14 ± 0.28</td></tr><tr><td/><td colspan="3">Omnibus dropout 78.0 ± 0.4 0.863 ± 0.017</td><td>3.08 ± 0.05</td><td>3.27 ± 0.29</td></tr></table>

Peer review (iclr2020):
PAPER SUMMARY: The paper argues that (ensembles of models with different types of dropout applied to each model) perform better than (ensembles in which the same type of dropout is applied to each model). They attribute this to increasing model diversity in the former case, and experimentally validate their claims. MAJOR COMMENTS: 1. Motivation Unclear: - Notational issues in (2): MSE(h_t | x) involves y, but y is not specified in the definition. As a result, later E_{x}[…] is evaluated disregarding the dependence on y, which is ambiguous. - Derivation of the second equality in (2) is not obvious, and needs a detailed proof. Notational issues exist in switching between H and h. - The above is used to argue that “the more diverse the models, the better performance achieved”. It is unclear how this follows from (2). 2. Unclear / Imprecise Writing - Before Sec. 3.4: “This is because higher Dropout rates lead to smaller effective network capacities” Needs reference. - “Dropout uncertainty can be obtained sequentially”. Unclear what this means. 3. The entire proposed method is described in one line, “we propose a novel omnibus dropout strategy, which merely combines all the aforementioned methods”. It is very unclear as to what the authors mean by combination. 4. Experiments - It seems that Omnibus dropout leads to a moderate “diversity”, and improves ensemble performance only for SVHN, out of the three datasets tested in Figure 4. For SVHN, the improvement is 0.1% accuracy, which is within the standard error (0.1). Hence, it seems that the proposed method provides no performance improvement. - A Similar statement is true for the other metrics (NLL, Brier Score, ECE). Score [Scale of 1-10]: 3: Reject The paper needs either more convincing experiments demonstrating the claims or theoretical analysis explaining the behavior for small networks. The paper needs to be rewritten to improve presentation, and state motivation, problem statement, and contributions clearly.