Omnibus Dropout for Improving The Probabilistic Classification Outputs of ConvNets
Authors: Zhilu Zhang
Adrian V. Dalca
Mert R. Sabuncu
Venue:  (2020)
Keywords: Uncertainty Estimation
Calibration
Deep Learning

Abstract: While neural network models achieve impressive classification accuracy across different tasks, they can suffer from poor calibration of their probabilistic predictions. A Bayesian perspective has recently suggested that dropout, a regularization strategy popularly used during training, can be employed to obtain better probabilistic predictions at test time (Gal & Ghahramani, 2016a). However, empirical results so far have not been encouraging, particularly with convolutional networks. In this paper, through the lens of ensemble learning, we associate this unsatisfactory performance with the correlation between the models sampled with dropout. Motivated by this, we explore the use of various structured dropout techniques to promote model diversity and improve the quality of probabilistic predictions. We also propose an omnibus dropout strategy that combines various structured dropout methods. Using the SVHN, CIFAR-10 and CIFAR-100 datasets, we empirically demonstrate the superior performance of omnibus dropout relative to several widely used strong baselines in addition to regular dropout. Lastly, we show the merit of omnibus dropout in a Bayesian active learning application. 

Under review as a conference paper at ICLR 2020
OMNIBUS DROPOUT FOR IMPROVING THE PROBA- BILISTIC CLASSIFICATION OUTPUTS OF CONVNETS
Anonymous authors Paper under double-blind review
ABSTRACT
While neural network models achieve impressive classification accuracy across different tasks, they can suffer from poor calibration of their probabilistic predic- tions. A Bayesian perspective has recently suggested that dropout, a regularization strategy popularly used during training, can be employed to obtain better proba- bilistic predictions at test time (Gal & Ghahramani 2016a). However, empirical results so far have not been encouraging, particularly with convolutional networks. In this paper, through the lens of ensemble learning, we associate this unsatisfac- tory performance with the correlation between the models sampled with dropout. Motivated by this, we explore the use of various structured dropout techniques to promote model diversity and improve the quality of probabilistic predictions. We also propose an omnibus dropout strategy that combines various structured dropout methods. Using the SVHN, CIFAR- 10 and CIFAR-100 datasets, we empirically demonstrate the superior performance of omnibus dropoul relative to several widely used strong baselines in addition to regular dropout. Lastly, we show the merit of omnibus dropout in Bayesian active learning application.
INTRODUCTION
Deep neural networks (NNs) achieve state-of-the-ar classification accuracy in many applications. However, in real world scenarios like medical diagnosis and autonomous driving, reliable probabilistic predictions are also crucial and need to be considered in assessing performance. Most modern NNs are trained with maximum likelihood to produce point estimates that are often over-confident (Guo iet al. 2017). Bayesian techniques can be used with neural networks to obtain well-calibrated predic- tions MacKay 1992 Neal 2012), yet they suffer from significant computational challenges. Thus, recent efforts have been devoted to making Bayesian neural networks more efficient (Blundell et al.) 2015; Chen et al. 2014; Wu et al. 2018). Monte Carlo (MC) dropout (Gal & Ghahramani 2016 cheap approximate inference technique which obtains uncertainty by performing dropout (Srivastaval ct al. 2014) at test time, a popular Bayesian method to obtain uncertainty estimates for NNs.
Despite improvements over deterministic NNs, MC dropout can still produce over-confident predic- tions (Lakshiinarayanan et al. (017)), particularly with convolutional architectures. In this paper, propose simple yet effective solution to this problem. Inspired by the recent success of ex- plicit ensembles of neural networks obtained using random initializations (Beluch et al. 2018), we reiterate the original notion of dropout "an extreme form of model combination with extensive parameter sharing" (Srivastava et al. 2014). and interpret MC dropout as an ensemble of models. Borrowing machinery from ensemble learning. we then attribute the poor performance of MC dropout to its limited model diversity compared to that of explicit ensembles. This perspective reveals how structured dropout methods (Ghiasi et al.) (2018) Tompson al. (2015) can improve performance by promoting diversity. While the importance of diversity has been demonstrated by others, prior works consider explicit ensembles of different models. To the best of our knowledge, this is the first paper to examine structured dropout model. As discussed below, we also propose to combine different structured dropout methods, which we call omnibus dropout. We empirically verify that omnibus dropout can yield models with superior performance on the SVHN, CIFAR-10 10 and CIFAR-100 datasets compared to not only MC dropout, but also some of the most widely adopted baselines like deep ensembles (Lakshminarayanan et al.) 2017) and temperature scaling (Guo et al. 2017). Furthermore, we demonstrate the merit of better uncertainty estimates in Bayesian active learning experiment (Gal al. 2017b)
way to enhance diversity in an ensemble obtained from a single

---

Under review as a conference paper at ICLR 2020
2
RELATED WORK
Dropout was first introduced stochastic regularization technique for NNs (Srivastava al. 2014). Inspired by the success of dropout, numerous variants have been proposed (Wan et al. 2013
Goodfellow al. 2013 Tompson al. 2015] Huang al.) 2016 /Sinhh et al., 2016 Gastaldi 2017: Ghiasi et al. 2018 Unlike regular dropout, most of these methods drop parts of the NNs in a structured manner. For instance, DropBlock (Ghiasi al. 2018) applies dropout to small patches of the feature map in convolutional networks, Srop (Tompson et al. 2015) drops out entire channels. Stochastic Depth Net (Huang et al.] (016) drops out entire ResNet blocks, and Swapout (Singh et al. 2016) combines the Stochastic Depth Net with regular dropout. These methods were proposed to boost test time accuracy. In this paper, we show that these structured dropout techniques can be successfully applied to obtain better uncertainty estimates as well.
Dropout can be thought of as performing approximate Bayesian inference (Gal & Ghahramani 20T6b) and offer estimates of uncertainty. Many other approximate Bayesian inference techniques have also been proposed for NNs (Kingma et al. Louizos & Welling 2017). However, these methods can demand sophisticated implementation,
often harder to scale, and can suffer from sub-optimal performance (Blier& & Ollivier 2018). Another popular alternative to approximate the intractable posterior is Markov Chain Monte Carlo (MCMC) (Neal) 2012). More recently, stochastic gradient versions of MCMC were also proposed to allow scalability (Gong et al., 2019: Ma et al.) 2015 Welling & Teh 2011). Nevertheless, these methods are often computationally expensive, and sensitive to the choice of hyper-parameters. A related approach, the SWA-Gaussian (Maddox al. 2019) is another technique for Gaussian posterior approximation using the Stochastic Weight Averaging (SWA) algorithm (Izmailov ct al. 2018).
There are also non-Bayesian techniques to obtain calibrated confidence estimates. For instance, temperature scaling (Guo et al.) 2017) has been empirically shown to be effective in calibrating the predictions. A related Tine of work uses an ensemble of several randomly- initialiin NNs (Lakshmi narayanan et al., 2017). The method, known deep ensembles, requires training and saving multiple NNs. It has also been demonstrated that an ensemble of snapshots of the trained model at different iterations can help obtain better uncertainty estimates (Geifman et al. 2019). Compared to explicit ensemble, this approach requires training only one model. Nevertheless, models at different iterations must all be saved in order to deploy the algorithm, which can be computationally demanding.
3
AN ANALYSIS OF THE PERFORMANCE OF MC DROPOUT
3.1 MC DROPOUT AS ENSEMBLES OF DROPOUT MODELS
Let's assume dataset D (X Y- {(x, 34) } where each (æ,, v1) E (X ))) is i.i.d. We consider the problem of k-class classification, and let X' C I2d be the input space and ) = (1..... .k] be the label space]] We restrict our attention to NN functions (x) : X' + R* where w = (w:) corresponds to the parameters of a network with L-layers, and W; corresponds to the weight matrix in the i-th layer. We define a likelihood model p(y| zz w) = softmax(fu. (z)). Maximum likelihood estimation can be performed to compute point estimates for w.
Recently, Gal & Ghahramani (2016a) proposed a novel viewpoint of dropout as approximate Bayesian inference (See Appendix A for brief review). This perspective offers simple way to marginalize out model weights at test time to obtain better calibrated predictions, which is called MC dropout:
T
ply = c,, Derain) = / p(u = clar, u w))(ur) DD triin ddw R
Σ Clyla,, un()),
where (t)) ~ g(w] (uraia) is assumed to be independently drawn layer-wise weight matrices: W:(") W. diag (Bernoulli(p)). W; is the parameter matrix learned during training, and p the dropout rate. In this paper, we view each dropout sample ((() in Equation corresponding to individual model in an ensemble, where MC dropout is performing (approximate Bayesian) ensemble averaging.
Extension to regression tasks is straightforward hut left out of this paper.
2

---

Under review as a conference paper at ICLR 2020
1.25
12
dispout
29455
5040
dopput
deep ensemble
1.20
020
0.75
211
desp crsamble
261.
0.00
42
Samner Modes
Warteer of Medel
Warber STMo3el:
humber of MAdiE
Figure 1: From left to right (1) Accuracy of MC dropout and deep ensemble (2) the relative im- provements accuracy of deep ensemble and MC dropout (3) Brier score of MC dropout and deep ensemble against number of models (4) the relative improvements in in Brier score of deep ensemble and MC dropout against number of models. Difference grows as number of models increases.
The same analysis applies to structured dropout as well. Mathematically. using structured dropout in lieu of regular dropout amounts only change of the approximate distribution g(w) ((w))uuii)) in Equation so that we are performing Bayesian variational inference with a different class of approximate distributions. For instance, the channel-level dropout, we sample one Bernoulli random variable for each channel.
3.2 DECOMPOSING THE PERFORMANCE OF ENSEMBLES
First proposed by Krogh Vedelsby 1995), the error-ambiguity decomposition enables one to quantify the performance of ensembles with respect to individual models. Let (he)i=1 be an ensemble of classifiers, and H(x)= = E, hy (a)/T 'T's is the ensemble prediction. In classification problems, (x()) is often probability vector such that (f(x) = ply = i|æ, w.). In MC dropout (xr))) = plylæ, (.)) Model ambiguity can be then defined as:
a(h, ()) [x))(x)) H (x)11
which quantifies the difference between an individual model and the ensemble average.
The Brier score measures both the accuracy and calibration of probabilistic classifications, and is proportional mean squared error (MSE), which be decomposed for an ensemble
MSE(H) = Ez, [MSE [=] = E., MSE(h [>)] - E [=<())=))
where MSE(N, (z)) yyy hh (=) y the one-hot encoded vector of the correct label y,
T
MSE(h) = - MSE(h; and (hh)z)
Σ
1 T Σ a (hel
correspond to the average MSE, and ensemble diversity (average ambiguity), respectively. Equation suggests that the more accurate and the more diverse the models, the better performance will be achieved by the ensemble. We MSE instead of the negative log likelihood (NLL), another commonly used measure for quality of uncertainty estimates, due to mathematical convenience. The two metrics or NLL can be seen as comprehensive measures influenced by both the accuracy and the calibration of the model. We give brief discussion in Appendix B the relationship between these metries.
closely related, and insights obtained from MSE carry
to NLL. In general, MSE
3.3 PERFORMANCE OF MC DROPOUT AND MODEL DIVERSITY
The discussion of the previous section provides us with potential recipe to enhance MC dropout. To illustrate the importance of diversity, we conduct an experiment using ResNet-50 on CIFAR- 10 to compare MC drepout with an explicit ensemble of five NNs (details can be found in Section 4 As we see from Figure individual models deep ensemble, on average, perform better than ones in MC dropout, likely because of the reduced effective capacity of the latter. Furthermore, the performance of the ensembles improve with more models. Yet the improvement is larger for deep ensemble, mainly because of increased ensemble diversity, since know from Equation 2 that the decrease in the Brier score in this analysis is attributable to the increase in ensemble diversity, as the average MSE does not change with the number of models.
The lack of diversity among MC dropout models is largely because neighboring pixel features are often correlated in convolutional layers. Thus, even with dropout, similar information propagates
3

---

Under review as a conference paper at ICLR 2020
through the network in every iteration. Although we can encourage model diversity naively by increasing dropout rates, this can lead to reduced MSE of individual models, thereby hampering ensemble performance, as can be seen from Eq. 2] This is because higher dropout rates would lead to smaller effective model capacities given a fixed total number of model parameters.
3.4 OMNIBUS DROPOUT
While model diversity can be promoted via explicit ensembles, they demand much more computational resources during training, which can be prohibitively expensive. Though typically more number of samples needed for dropout based methods at test time, unlike deep ensembles, dropout uncertainty can be obtained sequentially, which has a lower memory requirement. Moreover, the number of samples needed can potentially be optimized with an adaptive sampling scheme (Inouc 2019).
In order to enhance diversity in an ensemble obtained from single model, we examine the use of structured dropout, which drops information from contiguous regions of feature maps so that more divergent information is propagated to subsequent layers during training at each iteration. This enhancement in diversity of predictions can in turn lead to better performance. Specifically, we compare dropout at the patch-level which randomly drops out small patches of feature maps (Ghiasi eet al. 2018), the channel-leve which drops out entire channels of feature maps at random (Tompson al. 2015), and layer-level which drops out entire layers of CNNs random (Huang et al.) (016)) We denote these as dropBlock, dropChannel and dropLayer respectively. We identify the test-time sampling of models trained with the aforementioned structured dropout methods as MC dropBlock, MC dropChannel, MC dropLayer.
As we empirically observe below, similar to increasing the dropout rate, the increased diversity of structured dropouts can come at the cost of reduced performance of individual models. Morcover, given considerable choices of dropout strategies available, can be hard to pick the best one. There- fore, we propose novel omnibuus dropout strategy, which merely combines all the aforementioned methods. The implementation of omnibus dropout involves the sequential execution of the nested group of dropout methods: regular dropLayer, dropChannel, dropBlock and regular dropout. In our experiments, we use a constant dropout rate for all the dropout methods. Empirically we find this simple choice to mostly work well. As results show, omnibus dropout yields good performance by promoting model diversity without hampering the performance of individual models.
4 EXPERIMENTS
We empirically evaluate the performance of MC dropBlock, MC dropChannel, MC dropLayer and MC umnibus-dropout and compare them to MC dropout, deep ensembles and temperature scaling" Unless otherwise stated, the following experimental setup applies to all of our experiments.
Model. Layer-level dropout requires skip connections that there is still information flow through the network after dropping out an entire layer. Some of the examples include the FractalNet [Larsson et al.) 2017) and the ResNet (He al.) (016a)) We use the Rest-ttesnet (He et al.) (016b) for all our experiments. We refer to the mreAct-ReNeet trained without dropout as a deterministic model. MC dropout, MC dropBlock and MC dropChannel models are implemented through inserting the corresponding dropout layers with a constant p before each convolutional layer. A block size of 3 x 3 used for MC dropBlock. We follow Ghiasi al. (2018) match up the effective dropout rate of MC dropBlock to the desired dropout rate p. MC dropLayer is implemented through randomly dropping out entire ResNet blocks constant rate p. We empirically observe that, dropping out downsampling ResNet blocks during testing is harmful to the quality of uncertainty estimates. This is in agreement with experiments of Veit et al. 20116 Hence, downsampling blocks are only dropped out during training. MC omnibus-dropout is implemented by including all types of aforementioned dropouts, each with the same dropout rate. For a full Bayesian treatment, we also insert a dropout layer before the fully connected layer at the end of the NNs. For all our experiments, the dropout rate of this layer is set to be 0.1. To ensure a fair comparison, this layer was included for the "deterministic" models. For all models with dropout of all types, we sample 30 times at test-time for Monte Carlo estimation. We implement deep ensembles by training five NNs with random initializations. Although
"See Appendix C for results on explicit dropout ensembles. In their experiments, ResNet blocks are only dropped out during testing, but training.
4

---

Under review as a conference paper at ICLR 2020
SVHN
CIFAR-100
dropout
dropftock
dropCharnel
droplayer
drogOmnlbuz
deep ansemble
0.575
0.625
0.675
0.50
0.55
0.60
0.65
0.65
0.70 0.75
0.80
Interrater Agreement
Interrater Agreement
Interrater Agreement
Figure 2: Interrater Agreement (IA) of models with different types of dropout with 0.1 dropout rate on the SVHN, CIFAR-I0 and -100 datasets. The lower the IA, the more diverse the predictions of the models. Y-axis indicates different methods. MC dropout produces models with much larger IA, hence less model diversity, than structured dropout techniques in most of the cases.
used for training deep ensembles in the original paper, we find that adversarial training hampers both calibration and classification performance significantly, and thus do not incorporate it in our training.
Datasets. We conduct experiments using the SVHN (Netzer et al. 2011), CIFAR- 10 and CIFAR- 100 (Krizhevsky 2009) datasets with standard train/test-set split. Validation sets of 10000 and 5000 samples are used for SVHN and the CIFARs. To examine the performance of the proposed methods with models of different depth, we the 18-, 50- and 101-layer PreAct-ResNet for SVHN, CIFAR-10 and CIFAR-100.
Training. We perform preprocessing and data augmentation using per-pixel mean subtraction, horizontal random flip and 32 x 32 random crops after padding with 4 pixels on each side, We used stochastic gradient descent (SGD) with 0.9 momentum, a weight decay of 10- and learning rate of 0.01, and divided it by 10 after 125 and 190 epochs (250 in total) for SVHN and CIFAR-10. 10, and after 250 and 375 (500 in total) for CIFAR-100.
Evaluation. All the results are computed on the test set using the model at the optimal epoch based on validation accuracy. We use the Brier score, negative log- ligeeiidd (NLL), expected calibration eгror (ECE), and Classification accuracy to evaluate performance (see Appendix B for definitions). Following Naeini et al.] 2015), we partition predictions into 20 equally spaced bins and take a weighted average of the bins accuracy and confidence difference to estimate ECE. To visualize calibration performance, we also plot the reliability diagrams (Maddox et al. 2019). which are plots of the difference between accuracy and confidence against confidence. The closer the curve to the X-axis, the more calibrated the model predictions are.
4.1 ENSEMBLE DIVERSITY
We first investigate model diversity achieved with dropout. For a fair comparison, we fix the dropout rate for all methods to 0.1 so that all models have the same effective number of parameters. There numerous measures that quantify diversity of model ensembles (Zhou, 2012). We use Interrater Agreement (IA) Kuncheva & Whitaker (2003), defined as:
+ Ei= ((Tx)(TT p(xx))
n( 1)p(1 p)
where 7' is the number of individual classifiers, is the number of test samples, p(æx) is the number of models that classify the k-th sample correctly, and p is average classification accuracy across classifiers. When all classifiers perfectly agree on the test set 1, and smaller values indicate more diverse predictions. Figure (gaumaris IA for sampled models trained on different datasets with different dropout methods. We also compare the results with deep ensemble. The number of models used to compute IA, T, is fixed to five for all approaches. In general, IA for MC dropout is much higher than structured dropout techniques. On the other hand, structured dropout can yield ensembles that are as diverse as the computationally expensive method of deep ensemble, confirming our expectation that dropping out correlated information can produce sampled models with more ambiguity. Note that the large IA for MC dropLayer on SVHN is likely caused by a relatively small model used for that problem- an 18-layer ResNet. Lastly, note that while MC omnibus- -dropou yields models much more diverse than MC dropout, it is often not the most diverse either.
The moderate diversity of MC omnibus- dropout,, we believe, is the key to its effectiveness. To better understand its behavior, we study the performance metrics function of number of sampled models
5

---

Under review as a conference paper at ICLR 2020
0.015
drapout drepölock
0.014
2.64
drepChannel
Brier Brier Score
droplayer
0.012
drapOmmibus
0.011
dropout
0.010
2.42
dropBlock
dropChannel
0.009
dropLayer
dropOmnibus
0.008
0
10
15
20
25
30
25
Number of Models in the Ensemble
Number of Models the Ensemble
Figure Test Brier score (left) and accuracy (right) against number of models for ensemble prediction at test time on CIFAR-10. 10. This corresponds to the number of different MC dropout instantiations at test time of the same model. The Model trained with omnibus dropout achieves the best terms of accuracy and Brier score.
in the ensemble. Figure shows shows the Brier score (left) and accuracy (right) against number of models for the CIFAR-10 dataset (Similar results observed for SVHN and CIFAR- 100. See Appendix C). Firstly, as seen from (leftt, while the performance of individual models sampled from MC dropout is one of the best, the gain in Brier score with larger number of test-time MC samples much smaller compared to structured dropout techniques. On the other hand, though a larger diversity indeed leads to much sharper improvements as number of sampled models increases, the Brier scores (hence MSE) of individual models sampled from MC dropBlock, MC dropChannel and MC dropLayer are much larger than that of MC dropout, suggesting trade-off between diversity and the performance of individual sample models. MC omnibus-dropour which enjoys the benefits from both structured and regular dropouts, is able to only achieve good performance one sampled model (with a Brier score close to MC dropout), but also good model diversity as evident by significantly larger decrease in Brier score as number of models increases. Similar observations can be made from the accuracy plot of Figure (right).
4.2 PERFORMANCE EVALCATION
Table summarizes the performance metrics produced by various models. To ensure a fair comparison, we treat the dropout rate as a syper-paramecte and conduct linear grid search with 0.05 interval for optimal dropout rate based on NLL. The optimal dropout rates are shown the table next to methods. Standard deviations are obtained on five models with random initializations for all dropout models, and bootstrapping the test sets for deep ensembles. As seen from Table [ and Figure all forms of structured dropout models offer better uncertainty estimates than MC dropout in general. Overall, MC omnibus-dropout and deep ensembles are the best performing models. Remarkably, .C ammibus-dropon achieves even better uncertainty estimates SVHN and CIFAR- 10 and very comparable ones on CIFAR- 100 compared to deep ensembles which requires five times more computational resource train. Moreover, also perform experiments with five explicit ensembles of models trained together with all types of dropout for a fair comparison, and most of the dropout models outperform deep ensembles trianed without dropout. Again, omnibus dropout is consistently one of the best methods (See Appendix C). Lastly, as evident from moderately increased classification accuracy over deterministic temperature scaling models, all types of dropout methods can be incorporated into architectures for uncertainty estimates with no accuracy penalty.
We believe the relatively good performance of MC dropout on SVHN compared CIFARs because the former task is casier so that the model can still predict accurately at an aggressive dropout rate of 0.35 at which even regular dropout can produce acceptably diverse sampled models. In contrast, as observed in our experiments, while using larger dropout rates for the more difficult CIFAR datasets can lead to more calibrated predictions, accuracy and NLL suffer due to drop in MSE of individual models (see Appendix C). Lastly, we believe the results for MC dropBlock can be improved by optimizing the choice of block size. A pre-fixed block size x 3 can be too small for the upstream convolutional layers where the size of feature maps are much larger than the block size, and too large for the last few downstream layers where the feature maps are comparable to the block size, as supported by sharp increases in NLL after the optimal dropout rate.

---

Under review as a conference paper at ICLR 2020
Table 1: Results on benchmark datasets comparing accuracy and uncertainty estimates produced by different types of methods. The top-2 performing results for each metric are bold-faced. MC camibbssdrrpoue is consistently one of the best methods, outperforming even deep ensembles, which requires five times more computational resources, in many cases. The numbers in bracket next to dropout methods corresponds to the optimal drop_rale found by grid search using the NLL metric.
Dalasets
Methuds
Accuracy
NLL 1
Brier 1 (x10
ECE 1 (x10 3)
Temp Scaling
9..
0.163 1 0.002
6.62 1 0.10
0.995 = 0.160
SVHN Deep Ensemble
96.6 0.1
0.179 0.009
5.39 = 0.16
1.08 = 0.08
Dropout (0.35)
0.1
0.128 0.001
5.11 0.06
0.934 0.045
DropBlock (0.1)
96.8 + 0.1
0.133
5.19
1.26
DropChannel (0.2)
96.7
0.130
5.15
0.799 0.032
Layer (0.25)
96.3
0.144
5.69 0.05
0.846 0.250
Omnibus dropout (0.15)
96.9 - 0.1
0.127 + 0.001
4.97 0.09
1.15 0.06
Temp Scaling
93.9 0.1
0.189 + 0.002
9.06 + 0.088
0.905 0.TT14
Deep Ensemble
95.2 + 0.2
0.181 + 0.009
7.40 = 0.28
1,40+
CIFARIO
Dropout (0.2)
0.1
0.224 + 0.003
10.2 0.1
1.64 =
DropBlock (0.1)
93.4
+.0003
9.89 0.10
0.743 0.116
DropChannel (0.15)
9.1
+
9.3
0.812 = 0.104
Droplayer (0.1)
9.2
+
9.09
0.911 0.068
Omnibus dropout (0.1)
94.4 0.1
0.173 + 0.001
8.38 1.880.10
0.607 0.078
Temp Scaling
0.3
+
3.57 0.04
4.02 = 0.62
Deep Ensemble
77.9 0.4
0.922 0.019
3.12 0.05
5.10
CIFAR100
Dropout (0.2)
7..1
+
3.71 0.05
0.23
DrupBlock (0.15)
0.5
1.02
3.66
1.46
DropChannel (0.15)
0.5
0.996 + 0.02
3.46 0.04
3.17 0.11
DropLayer (0.25)
75.7 0.2
1.11 0.01
3
2.90 0.24
Omnibus dropout (0.25)
75.3 1.2 0.2
0.929 + 0.005
3.40
0.02
1.65 0.21
SVHN
CFAR-10
OFAR 100
0.4
0.20
SampSiiing
0.20
0.3
0.15
SropChanrel
0.25
0.30
Smpormsibue
Stiel spor
0.20
Jeep
0.1
0.25
0.0
0.00
-0.05
-0.10
0.5
0,7
1.0
0.7
1.0
1.5
Confidence
Confidance
Confidence
Figure 4: Reliability diagrams of predictions produced by difference models.
4.3 BAYESIAN ACTIVE LEARNING
To further demonstrate the merit of omnibus dropout, we consider the downstream task of Bayesian active learning on CIFAR-10. Active learning involves first training on a small amount of labeled data. Then, an acquisition function based on the outputs of models is used to select small subset of unlabeled data so that an oracle can provide labels for these queried data. Samples that a model is the least confident about are usually selected for labeling, in order to maximize the information gain. The model is then retrained with the additional labeled data that is provided. The above process can be repeated until desired accuracy is achieved or the labeling resources are exhausted.
In our experiment, we train models with structured dropout at different scales using the identical setup described in the beginning of this section, except that only 2000 training samples used initially. To match up model capacity, the dropout rate is set to 0.1 for all methods. We also compare again deterministic model. After the first iteration, "unlabeled" data, and combine the acquired samples with the original set of labeled images to retrain the models. Following Gal et al. (2017b), we consider three acquisition functions: Max Entropy, Hllyy æ, Dtrain] -.p ply ca Deraia) ) log p(y = c/æ, D Derain), ) the BALD metric (Bayesian and
acquire 1000 samples from pool of
Active Learning by Disagreement). I[y, w/x Dirain] = H[y/ xc, D Derainl -max, p(y, x. D Drrain). We repeat the acquisition
the Variation Ratios metric, variation-ratiolar
7

---

Under review as a conference paper at ICLR 2020
BALD
Entroyy
Variatin
0.90
6.90
0.90
0.85
0.90
0.75
detemmnizt::
dropout
0.70
dropslock
CrapCharnel
0.70
0.70
0.85
droplayeer
0.6
Number Training
lo4
Number Training Samples
106
Number Samplees
Figure 5: Left: Test accuracy against number of training samples for models with different methods of dropout and Variation Ratios as the acquisition function on CIFAR-10, Right: Relative improvements in test accuracy over that of the first iteration with different methods of dropout.
process cight times that in the last iteration, the training set contains 10000 images. To mimic a real world scenario in which number of labeled samples is small, we do not use a validation set, and the accuracies reported for this experiment correspond to the last-epoch accuracies. We repeat experiments five times for consistency.
Figure shows the test accuracy against number of training samples for different models. In general, MC omnibus-dropoul yields the best performance by far. Interestingly, MC onnibus-dropout is able outperform all other methods consistently by significant margin after the first iteration when all samples are randomly selected. In addition. it can be seen that, after the first iteration when all 2000 training images are randomly selected, the test accuracy using MC dropout on par with that of other structured dropout methods. However, as more labeled data are added, the relative increase in accuracy is more significant for models using structured dropout compared to that of using regular dropout. This suggests that the uncertainty estimates obtained with structured dropout are more useful for assessing "what the model doesn't know", thereby allowing for the selection of samples to be labeled in way that better helps improve performance. Note also that the comparative gain in accuracy by MC omnibus dropout during the later stages of the learning process is not as large. We suspect this can be caused by the saturation effect test accuracy.
CONCLUSION AND FUTURE WORK
We reinterpret MC dropout ensemble averaging strategy, and attribute its poor performance in convolutional neural networks to a lack of diversity of sampled models using the error-ambiguity decomposition of the Brier accuracy and calibration of probabilistic outputs. As we demonstrate empirically, omnibus dropout, which is couple-t-iimplement and computationally efficient, strikes the right balance between model diversity among sampled models while retaining reasonable performance of individuals models, thereby consistently improving the quality of the ensemble's prediction.
(or MSE), widely used performance metric that captures both
We are interested in further exploring several directions. Firstly, we have only considered uniform individual dropout rates within the omnibus dropout strategy. Learning the optimal dropout rates for each type of dropout, possibly by building on the the work of (Gal et al. (017a), can potentially further improve the performance of omnibus dropout. Moreover, we used a constant dropout rate in our experiments, even though one can vary dropout rates across NNs (Huang al.) (016)) or incorporate dropout rate scheduling (Ghiasi et al. 2018). How this impacts the quality the probabilistic predictions is open question. Lastly, we have only explored structured dropout in the context of CNNs, with application to computer vision tasks, We believe this idea can be extended beyond CNNs.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
William Beluch, Tim Genewein, Andreas Nürnberger, and Jan Köhler. The power of ensembles for active learning in image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9368-9377, 2018.
Léonard Blier and Yann Ollivier. The description length of deep learning models. In Advances in Neural Information Processing Systems, pp. 2216-2226, 2018.
Charles Blundell, Julien Coreeiie, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International Conference on Machine Learning, pp. 1613-1622, 2015.
Tianqi Chen, Emily Fox, and Carlos Guestrin. Stochastic gradient hamiltonian monte carlo. In International conference machine learning, pp. 1683-1691, 2014.
Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model uncertainty deep learning. In international conference on machine learning, pp. 1050-1059, 2016a.
Yarin Gal and Zoubin Ghahramani. A theoretically grounded application of dropout in recurrent neural networks. Advances in neural information processing systems, pp. 1019-1027, 2016b.
Yarin Gal, Jiri Hron, and Alex Kendall. Concrete dropout. In Advances in Neural Information Processing Systems, pp. 3581-3590, 2017a.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In
Proceedings of the 34th International Conference on Machine Learning-Volume Volume 70, pp. 1183-1192. JMLR. org, 2017b.
Xavier Gastaldi. Shake-shake regularization. arXiv preprint arXiv: 1rxXiv:7005.00 170.007857 2017.
Yonatan Geifman, Guy Uziel, and Ran EI- Vaniv.. Bias-reduced uncertainty estimation for deep neural classifiers. International Conference Learning Representations, 2019.
Golnaz Ghiasi, Tsung- Yi Lin, and Quoc V Le. Dropblock: A regularization method for convolutional networks. In Advances in Neural Information Processing Systems, pp. 10727-10737, 2018.
Wenbo Gong, Yingzhen Li, and José Miguel Hermández-Lobato. leta-learning for stochastic gradient meme. International Conference Learning Representattioss, 2019.
Ian J Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio. Maxout networks. International Conference on Machine Learning. 2013.
Alex Graves. Practical variational inference for neural networks. In Advances in neural information processing systems, pp. 2348-2356, 2011.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1321-1330. JMLR. org, 2017.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645. Springer. 2016b.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In European conference on computer vision, pp. 646-661. Springer, 2016.
Hiroshi Inoue Adaptive ensemble prediction for deep neural networks based on confidence level. In The 22nd International Conference Artificial Intelligence and Statistics, pp. 1284-1293, 2019.
Pavel Izmailov, Dmitrii Podoprikhín, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. Conference on Uncertainty in Artificial Intelligence, 2018.

---

Under review as a conference paper at ICLR 2020
Durk P Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameterizzttoon trick. In Advances in Neural Information Processing Systems, pp. 2575-2583 2015.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Anders Krogh and Jesper Vedelsby. Neural network ensembles, cross validation, and active learning. In Advances in neural information processing systems, pp. 231-238, 1995.
Volodymyr Kuleshov and Percy S Liang. Calibrated structured prediction. In Advances in Neural Information Processing Systems, pp. 3474-3482, 2015.
Ludmila Kuncheva and Christopher J Whitaker. Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Machine learning, (11::18112200 2003.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, pp. 6402-6413, 2017.
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Fractalnet: Ultra-deep neural networks without residuals. International Conference on Learning Representations, 2017.
Christos Louizos and Max Welling. Multiplicative normalizing flows for variational bayesian neural networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 2212211--22 2227. JMLR. org, 2017.
Yi-An Ma, Tianqi Chen, and Emily Fox. A complete recipe for stochastic gradient mcmc. In Advances in Neural Information Processing Systems. pp. 2917-2925, 2015.
David JC MacKay. A practical bayesian framework for backpropagation networks. Neural computa- tion, 4(3):488-4 472, 1992.
Wesley Maddox, Timur Garipov, Pavel Izmailov, Dmitry Vetrov, and Andrew Gordon Wilson. A simple baseline for bayesian uncertainty in deep learning. arXiv preprint arXiv:1902.024777 2019.
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated prob- abilities using bayesian binning. In Twenty-Ninth AAAI Conference on Artificial Intelligence. 2015.
Radford M Neal. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. 2.11.
Saurabh Singh, Derek Hoiem, and David Forsyth. Swapout:
Learning an ensemble of deep architec- tures. In Advances in neural information processing systems, pp. 28-36, 2016.
Lewis Smith and Yarin Gal. Understanding measures of uncertainty for adversarial example detection. arXiv preprint arXiv:1803.000 08833, 2018
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 152))1199-11958 2014.
Jonathan Tompson, Ross Goroshin, Arjun Jain, Yann LeCun, and Christoph Bregler. Efficient object localization using convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.
2015.
Andreas Veit, Michael J Wilber, and Serge Belongie. Residual networks behave like ensembles of relatively shallow networks. In Advances in neural information processing systems, pp. 550-558, 2016.
10

---

Under review as a conference paper at ICLR 2020
Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In International conference on machine learning, pp. 1058-1066, 2013.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11), pp. 681-688,
2011.
Anqi Wu, Sebastian Nowozin, Edward Meeds, Richard Turner, José Miguel Beemández-Lobatee and Alexander L Gaunt. Deterministic variational inference for robust bayesian neural networks. International Conference on Learning Representations, 2018.
Zhi-H Hua Zhou. Ensemble methods: foundations and algorithms. Chapman and Hall/CRC, 2012.

---

with Under review as a conference paper at ICLR 2020
APPENDIX A: BRIEF REVIEW OF DROPOUT AS BAYESIAN APPROXIMATION
Let us assume a dataset D = (X = ((x,, yi) ye) where each (æ.; Ui) E (X' x 3) iid. In this paper, we consider the problem k-class classification, and let X' S [2d be the feature space and ) = {1.... ke be the label space. A classifier is function that maps input feature space to the label space f X + R. We restrict our attention to functions that can be implemented as a DNN, and denote it by (a)), where and W; corresponds to the weight matrix in the i-th layer. We define a likelihood model p(yl 2,, w) = (f It is common practice to perform maximum likelihood compute point estimates for w. Uncertainty estimates can be obtained through Bayesian DNNs by first assuming a prior distribution on the weights, p(w). A common choice is the zero mean Gaussian (0, 7). Bayes Theorem can then be used to obtain the posterior p( w X, Y) = X w which inference can be carried out:
{W:) corresponds to the parameters of network with L-layers,
ply c/2r, Dirain) /
/Dly ply c/ar, w) w))pppp (Driii ddw.
The marginal distribution X)), and thus p(w|X, Y) are often intractable. Variational inference uses tractable family of distributions go (w) paramaterized by I to approximate the true posterior p( p(w) X, Y), by minimizing the Kullback-Leibler divergence KL(gol (w) Ip (ww X, Y)), which is equivalent to optimizing bound on the true objective Graves (2011). To interpret dropout as variational inference strategy Gal & Ghahramani (201 6a). the approximate distribution is defined as:
W; θ (
Zi.j Bernoulli(p:) for i 1, j- K;-1,
where 0 = (0.) are variational parameters be optimized and {p. are user-defined hyper- parameters that correspond to layerwise dropout rates. Minimizing the KL-divergence is mathemati- cally equivalent to maximizing the following objective:
(vi(0)=
Σ
f
qa (w) log p(y: kei, w)dw (L(gg()))l<())).
Using Monte Carlo integration with one sample w, (o(w) for each training datum (æ, y) to approximate the integral in the above equation, and optimizing over mini-batches of size m, the approximated objective becomes:
Lvi(0)
log plys w.) (l)(((())))p(())))
As shown in Gal & Ghahramani (0166)). there is direct correspondence between optimizing the above objective and regular dropout training for DNNs. Furthermore, uncertainty estimates can be obtained through marginalizing and performing Monte Carlo integration over the approximate distribution (o(w), This corresponds to dropout at test time:
ply = c/x Durai) ze / ply = c, w ())
T ply x, wt).
Σ
T
where we ~ qo (w) are dropout samples from the NN. This referred to as the MC dropout.
APPENDIX B: RELATIONSHIP BETWEEN DIFFERENT PERFORMANCE METRICS
Brier score, negative log- likellioloood (NLL) and the expected calibration error (ECE) are three of the most commonly used metrics for evaluating the quality of uncertainty estimates. In this section, we discuss the relationship between them.
As we noted in Section 3 the Brier score is equal to the normalized MSE in the context of classification. Recall, the ECE is defined as:
ECE(H)
E,
H1
(10)
12

---

Under review as a conference paper at ICLR 2020
0.0080
dropout
0.0075
dropBlock
dropChannel dropLayer
0.0070
Brier Score
0.0065
dropOOniibs
Accuracy
0.0060
dropout
dropBlock
0.0055
drepChennel
dropLayee
0.0050
drop@mmibus
10
15
20
25
30
Number of Models in the Ensemble
Number of Modéls in the Ensemble
CIFAR-100
CIFAR-100
drrppout
0.00450
dropBlock
dropChannel
Brier Score
0.00425
dropLayer
0.00400
dr dropOmmibus
Accurass
dropout
0.00375
dropBlock
dropChannel
0.00350
dropLayer
dropOmmiibus
0.00325
Number of Models the Ensemble
Number of Models the Ensemble
Figure 6: Test Brier score (left) and accuracy (right) against number of models for ensemble prediction test time on SVHN and CIFAR-100. This corresponds the number of different MC dropout instantiations at test time of the same model. The Model trained with omnibus dropout achieves the best in terms of accuracy and Brier score.
which measures the expected difference between the true class probability and the confidence of the model (Kuleshov & Liang, 2015). In addition to the error-ambiguity decomposition that we have discussed, MSE can also be decomposed as:
MSE(H) = (Iv H(x))²
(11)
= Exl(y - (()))(((())))11))))))))))))))))) ECE(()
(12)
Var Var [E, (x)]] + +ECE(H),
(13)
where (x)) corresponds to the true probability of y = 1 conditioned on (x)).
Var, [Eyly H (x)] measures the variation of the true class probabilities across the level-sets of
the ensemble model Kuleshov & Liang (015)). Thus for this metric, the numeric values of H(x) constant and maximized when H7x)) f(y), for any bijective function f. One can therefore view Var [Ey [yl H (x)]] as a weak metric of accuracy that is not sensitive to calibration. Note Var, [u] does not depend on the models. Brier score thus can be seen as metric that is influenced by both the accuracy and the ECE of the models. Similarly, NLL metric closely related Brier score on log scale. Consequently, sometimes better uncertainty estimates in terms of NLL or Brier score can lead to slight drops in accuracy, as the reduction in calibration error outweighs increase in classification error. This phenomenon is indeed observed in practice as well. Figure/ 7 shows the plot of both NLL and accuracy against dropout rates for all dropout methods considered in the paper. For instance, can be seen that while increasing the dropout rate for the MC dropout model CIFAR- 100 dataset from 0.1 to 0.2 leads to reduction in NLL,, there is also quite a significant dropout in classification accuracy. Similar trends can be seen for MC dropChannel exemplify, increasing dropout rate of MC drapout on the SVHN dataset also leads to an increase in accuracy as well. In conclusion, when tuning for the optimal dropout rate in practice, it can be beneficial to look at different metrics for a holistic consideration.
not important. It is minimized if H(x)
110 well. Nevertheless, the trade-off is not always present. To
APPENDIX C: ADDITIONAL RESULTS
Supplementary Results on Diversity of Dropout Models. In Figure 6, we show plots of Brier score and accuracy against number of models used for prediction on SVHN and CIFAR-100 datasets.
13

---

Under review as a conference paper at ICLR 2020
Table 2: Results comparing accuracy and uncertainty estimates obtained using single model when drop_rate 0.1 for all models. The top-2 performing results for each metric is bold-faced. MC onnibus-dropen is the best method in general.
Datasets
Methods
Accuracy 1
NLL
Brier à. (x10-))
ECE (X10
Temp Scaling
95.7 0.1
0,163 1 0.002
0.12:0.100
0.995 1 0.160
Dropout
9.1
0.179 0.004
5.68
1.34 0.10
SVHN
DropBlock
96.8 0.1
0.133
0.002
5.19
1.26
DropChannel
0.1
0.148 0.002
5.41
0.663 0.050
DropLayer
9.1
0.154 0.002
5.94
1.13 0.10
Omnibus dropout
96.8 - 0.1
0.133 0.003
5.07 0.07
0.616 0.077
Temp Scaling
0.1
0.189 0.002
9.06 = 0.08
0.905
Dropout
0.1
0.226 0.008
9.44 0.10
2.30 0.09
CIFARIO
DropBlock
0.1
0 0.003
9.10
0.743 1.880.111 0.116
DropChannel
931
0.196 0.006
9.20
0.970 0.171
DropLayer
94.0
0.206 0.001
9.09
0.941 + 0.068
Omnibus dropout
94.4
0.173 0.001
8.38
0.607 = 0.078
Temp Scaling
0.3
1.00 0.01
4.02 0.62
Dropout 74.8+
1.21 0.01
3.71
11.1
CIFAR100
DropBlock
0.2
1.04 0.01
3.46 3.02
6.98 + 0.19
DropChannel
75.3
1.02 0.01
0.03
5.57 0.08
DropLayer
75.8 0.3
1.04 0.02
3.46
7,42 0.32
Omnibus dropout
76.3 0.1
1.00
0.02
7.11 0.20
Table 3: Results showing accuracy and uncertainty estimates produced by different types of explicit ensembles when drop_rale 0.1 for all models. five models are used for each ensemble. We generate 6 sampled models from each dropout models during evaluation (30 samples in total). The top-2 performing results for each metric is bold-faced.
Datasets
Methods
Accuracy f
NLL L
Brier | (x10-*)
ECE (x10-²)
Deep Ensemble
96.6
0.179 0.009
5.39
1.08 E 0.08
Dropout
97.0 + 0.1
0.141 + 0.006
4.82
0.736 + 0.077
SVHN
DropBlock
97.2 = 0.1
0.125 + 0.004
4.19
1.86 1.86 0.09
DropChannel
9.1
0.129 0.004
4.82
0.949 0.082
DropLayer
9.18
0.132 0.005
4.11
0.575 0.077
Omnibus dropout
97.2 0.1
0.122 + 0.004
4.61
1.05 + 0.07
Deep Ensemble
95.2
0.181 0.009
1.40
Dropout
0.21+0.2
0.176 + 0.008
8.17
1.01
CIFARIO
DropBlock
9..0
0.185 0.006
9.12 0.28
1.51 0.18
DropChannel
9.2
0.174 0.007
8.35 0.29
0.900 + 0.152
Dropl ayer
9.2
0.173 + 0.006
7.88 0.26
2.04 0.17
Omnibus dropout
94.8 0.2
0.160 = 0.006
7.82
0.953 + 0.156
Deep Ensemble
7..0
0.923 1 0.020
1.1210005
5.12 1 0.34
Dropout
0.4
0.931 0.020
3.18 0.05
4.51 + 0.32
CIFARI00
DropBlock
0.4
0.909 0.019
3.17 0.06
3.23 - 0.31
DropChannel
77.1
0.871 0.017
3.13 0.04
2.24 + 0.28
DropLayer
78.1 - 0.4
0.855 + 0.017
3.05 0.04
2.14 + 0.28
Omnibus dropout
78.0 - 0.4
0.863 0.017
1.05
3.27 + 0.29
14

---

Under review as a conference paper at ICLR 2020
As discussed in Section 4.1 patterns similar to the plots obtained the CIFAR-10 dataset in Figure are also observed consistently here. The only exception is to the MC dropLayer model on the SVHN dataset, which obtains better performance individual model but much smaller improvements in both Brier score and test accuracy compared to the other the other dropout methods. We would like highlight out that the seemingly contradictory results is likely caused by the shallow network used, an 18-layer ResNet. As no down-sampling layers are dropped out for layer dropout, the effective number of ResNet blocks that can be dropped very small, leading to much smaller dropout rate compared to ther other methods. This is not an issue with deeper models in which the number of downsampling blocks are much more than that of -ondoowssamlingg ones.
97.0
96.8
dropout dropBlock chonChannel dropLayer dropomiibuss
Accuracy
96.6
dropout dropBlock
Test NLL
0.18
96.4
dropChannel draplayer dropOmribus
0.16
96.2
96.0
95.8
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
0.05
0.10
0.15
0.35 Dropout Rate
0.40 CIFAR-10
0.20
0.25
0.30
Dropout Rate
CIFAR-10
0.55
94
0.50
92
0.45
Accuracy
90
Test NIL
0.35
86
84
0.20
0.20
0.25
0.05
0.10
0.25
0.30
Dropout
Dropout
CIFAR-100
CIFAR
76
Accuracy
74-
Test
72
70
1.00
0.05
0.10
0.15
0.20
0.25
0.30
0.05
0.10
0.15
0,20
0.25
0.30
Drepout Rate
Drapout Rate
Figure 7: Plots of time NLL (left) and accuracy (right) against dropout rate for models trained with different types of dropout on the SVHN, CIFAR-10 and CIFAR-100 datasets.
Additional Results Using Explicit Dropout Ensembles. Smith Gal (2018) demonstrated that explicit ensembles with MC dropout models produce better uncertainty estimates than that of deep ensembles. Thus we examine the effectiveness of ensembling multiple explicit models trained with structured dropout. This would serve fair comparison to deep ensemble. We use five explicit models, each trained with random initialization. drop_radd rate = 0.1 is used for all methods as it is impractical tne for the optimal dropout rates for ensemble of five models. Note that the optimal
15

---

Under review as a conference paper at ICLR 2020
dropout rates found for individual models do not carry to explicit ensembles of five models, as we observe in our experiments. At test time, we generate six sampled models from each dropout models (30 samples in total).
Results obtained are summarized Table 3 Similar to previous findings, ensembles with omnibus dropout consistently outperform all the rest. Moreover, ensembles with all types of structured dropout methods do better than MC dropout and deep ensembles. Deep ensembles does the worst in terms of uncertainty estimates.
We also investigate the sensitivity of the methods to the choice of dropout rate. To that end, we also report the results obtained with single model for each method, with fixed dropout rate of 0.1, reasonable default value for dropout rate in general. The results are shown in Table Possibly due to the combination of all dropout method, omnibus dropout seems be also relatively insensitive the choice of dropout rate, performing well in all the experiments.
Results on Tuning the Dropout Rate. Figure illustrates the plots NLL and accuracy against dropout rate for all models on all of the datasets. As discussed in Appendix B, conflict between NLL and accuracy can occur sometimes. Interestingly, the NLL drastically increases after minima on all three datasets for dropBlock, suggesting the possibility that the block size for dropBlock may be too large towards later convolutional layers when the size of feature maps are comparable that of block size.
16


Reviews:
[{"review_id": "Hkl9f4ilir", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary of the paper:\nThe paper tackles the probabilistic classification problem which is interesting and important in deep learning. The paper proposes an approach called omnibus dropout. Omnibus dropout is a sequential execution of existing dropout techniques such as drop layer, drop channel, drop block and regular dropout. The experimental results suggest that omnibus dropout perform reasonably well in various datasets.\n\nPros:\n1. The proposed approach is reasonable which I found no surprising it produces reasonably well results.\n\nCons:\n1. The paper could be improved in writing, especially on justification why this kind of dropout combination gives better performance. \n2. The results are mixed and the improvements are not significant. So I am not convinced the proposed approach is always a better strategy.\n3. The proposed approach is simply a combination of the existing dropout methods. The contribution of the paper is very limited.\n\nQuestions:\n1. In table 1, different dropout rate is chosen for different method. I think it is also reasonable to compare different methods with the same dropout rate (basically this corresponds to the actual model capacity).", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1g0Jtez5S", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "PAPER SUMMARY: The paper argues that (ensembles of models with different types of dropout applied to each model) perform better than (ensembles in which the same type of dropout is applied to each model). They attribute this to increasing model diversity in the former case, and experimentally validate their claims.\n\nMAJOR COMMENTS: \n1. Motivation Unclear:\n- Notational issues in (2): MSE(h_t | x) involves y, but y is not specified in the definition. As a result, later E_{x}[…] is evaluated disregarding the dependence on y, which is ambiguous. \n- Derivation of the second equality in (2) is not obvious, and needs a detailed proof. Notational issues exist in switching between H and h.\n- The above is used to argue that “the more diverse the models, the better performance achieved”. It is unclear how this follows from (2).\n\n2. Unclear / Imprecise Writing\n- Before Sec. 3.4:  “This is because higher Dropout rates lead to smaller effective network capacities” Needs reference. \n- “Dropout uncertainty can be obtained sequentially”. Unclear what this means.\n\n3. The entire proposed method is described in one line, “we propose a novel omnibus dropout strategy, which merely combines all the aforementioned methods”. It is very unclear as to what the authors mean by combination.\n\n4. Experiments\n- It seems that Omnibus dropout leads to a moderate “diversity”, and improves ensemble performance only for SVHN, out of the three datasets tested in Figure 4. For SVHN, the improvement is 0.1% accuracy, which is within the standard error (0.1). Hence, it seems that the proposed method provides no performance improvement.\n- A Similar statement is true for the other metrics (NLL, Brier Score, ECE). \n\nScore [Scale of 1-10]:\n3: Reject\nThe paper needs either more convincing experiments demonstrating the claims or theoretical analysis explaining the behavior for small networks. The paper needs to be rewritten to improve presentation, and state motivation, problem statement, and contributions clearly. \n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "ByxUkcTRtS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed to use multiple structured dropout techniques to improve the ensemble performance of convolutional neural networks as well as the calibration of their probabilistic predictions. \n\nThe approach, which is termed omnibus dropout and combines multiple existing dropout methods, is reasonable and straightforward. \n\nThe paper presents extensive experimental results and analyses by mainly comparing with the explicit ensemble of multiple neural networks. The experiments reveal interesting properties of the learned networks and compelling results. \n\nMy main concern is about the technical novelty of the paper. It does not supply a new method in essence and instead provides careful experimental studies, from both accuracy and calibration perspectives, about a combination of existing dropout techniques. It reads like a very solid workshop paper in my opinion, but it is probably not a good fit to the main conference.\n\nQuestions:\n1. The reasoning in the first two paragraphs of the introduction is confusing or misleading. The first paragraph is mainly about the poorly calibrated probabilistic outputs of neural networks, while the second paragraph suddenly shifts to the performance of ensembled networks in terms of accuracy. \n\n2. Some of the comparisons with the \"deep ensemble\" may be unfair. There are only five networks in \"deep ensemble\", but 30 are used in test time for the proposed method. \n\n3. Where is \"deep ensemble\" in the active learning experiments? I could not find it in Figure 5. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper investigates how to improve the performance of dropout and proposes an omnibus dropout strategy to reduce the correlation between the individual models.

All the reviewers felt that the paper requires more work before it can be accepted. In particular, the reviewers raised several concerns about novelty of the method relative to existing methods, significance of performance improvements and clarity of the presentation. 

I encourage the authors to revise the draft based on the reviewers’ feedback and resubmit to a different venue.

Decision: Reject