Contextual Dropout: An Efficient Sample-Dependent Dropout Module
Authors: XINJIE FAN
Shujian Zhang
Korawat Tanwisuth
Xiaoning Qian
Mingyuan Zhou
Venue: ICLR 2021 Poster (2021)
Keywords: Efficient Inference Methods
Probabilistic Methods
Supervised Deep Networks

Abstract: Dropout has been demonstrated as a simple and effective module to not only regularize the training process of deep neural networks, but also provide the uncertainty estimation for prediction. However, the quality of uncertainty estimation is highly dependent on the dropout probabilities. Most current models use the same dropout distributions across all data samples due to its simplicity.  Despite the potential gains in the flexibility of modeling uncertainty, sample-dependent dropout, on the other hand, is less explored as it often encounters scalability issues or involves non-trivial model changes.  In this paper, we propose contextual dropout with an efficient structural design as a simple and scalable sample-dependent dropout module, which can be applied to a wide range of models at the expense of only slightly increased memory and computational cost. We learn the dropout probabilities with a variational objective, compatible with both Bernoulli dropout and Gaussian dropout. We apply the contextual dropout module to various models with applications to image classification and visual question answering and demonstrate the scalability of the method with large-scale datasets, such as ImageNet and VQA 2.0. Our experimental results show that the proposed method outperforms baseline methods in terms of both accuracy and quality of uncertainty estimation.

Published as a conference paper at ICLR 2021
CONTEXTUAL DROPOUT: AN EFFICIENT SAMPLE- DEPENDENT DROPOUT MODULE
Xinjie Fan".l, Fahuii,Suujaan Zhang*.,, Korawat Tanwisuth', Xiaoning Qian', Mingyuan Zhou' 'The University of Texas at Austin, Texas A&M University,
xfanêutexas edu,, szhangl 90utexas edu,, korawat www.istthhuteeeeeeeeee edu,, xqi anaanneeee tamu. mingyuan. zhou@mccombs utexas.
ABSTRACT
Dropout has been demonstrated simple and effective module to not only regu- larize the training process deep neural networks, but also provide the uncertainty estimation for prediction. However, the quality of uncertainty estimation is highly dependent on the dropout probabilities. Most current models use the same dropout distributions across all data samples due to its simplicity. Despite the potential gains in the flexibility of modeling uncertainty, sample-dependent dropout, on the other hand, is less explored as it often encounters scalability issues involves non-trivial model changes. In this paper, we propose contextual dropout with an efficient structural design as a simple and scalable sample-dependent dropout mod- ule, which can be applied to a wide range of models at the expense of only slightly increased memory and computational cost. We learn the dropout probabilities with a variational objective, compatible with both Bernoulli dropout and Gaussian dropout. We apply the contextual dropout module to various models with applica- tions to image classification and visual question answering and demonstrate the scalability of the method with large-scale datasets, such as ImageNet and VQA 2.0. Our experimental results show that the proposed method outperforms baseline methods in terms of both accuracy and quality of uncertainty estimation.
INTRODUCTION
Deep neural networks (NNs) have become ubiquitous and achieved state-of-the-an results in wide variety of research problems (LeCun et al. (0115). To prevent aver-parameterized NNs from overfit- ting, we often need to appropriately regularize their training. One way to do so is to use Bayesian NNs that treat the NN weights as random variables and regularize them with appropriate prior distributions (MacKay 1992 Neal 2012). More importantly, we can obtain the model's 's confidence tions by evaluating the consistency between the predictions that are conditioned on different posterior samples of the NN weights. However, despite significant recent efforts in developing various types of approximate inference for Bayesian NNs (Graves) 2011 Welling & Teh, 2011): Li et al.] 2016) Blundell al. 2015; Louizos Welling 2017: Shi et al. 2018), the large number NN weights makes it difficult to scale to real-world applications.
its predic-
Dropout has been demonstrated another effective regularization strategy, which can be viewed as imposing a distribution over the NN weights (Gal & Ghahramani] 2016). Relating dropout to Bayesian inference provides a much simpler and more efficient way than using vanilla Bayesian NNs to provide uncertainty estimation (Gal & Ghahramani] 2016). as there is no more need to explicitly instantiate multiple sets of NN weights. For example, Bernoulli dropout randomly shuts down neurons during training (Hinton et al. 2012 Srivastava et al.] 2014). Gaussian dropout multiplies the neurons with independent, and identically distributed (iid) Gaussian random variables drawn from (1,,,) where the variance is a tuning parameter (Srivastava al. 2014). Variational dropout generalizes Gaussian dropout by reformulating it under a Bayesian setting and allowing a to be learned under variational objective (Kingma al. 2015 Molchanov et al. 2017).
Equal contribution. Corresponding mi ngyuan, zhou@@ccommb. utexas.

---

Published as a conference paper at ICLR 2021
However, the quality of uncertainty estimation depends heavily the dropout probabilities (Gal et al. 2017). To avoid grid-search over the dropout probabilities, Gal et al. (2017) and Boluki et al. (2020) propose to automatically learn the dropout probabilities, which not only leads to faster experiment cycle but also enables the model to have different dropout probabilities for each layer, bringing greater flexibility into uncertainty modeling. But, these methods still impose the restrictive assumption that dropout probabilities are global parameters shared across all data samples. By contrast, we consider parameterizing dropout probabilities as function of input covariates, treating them as data-dependent local variables. Applying covariate-dependerr dropouts allows different data to have different distributions over the NN weights. This generalization has the potential to greatly enhance the expressiveness of Bayesian NN. However, learning covariate-dependent dropout rates challenging. Ba Frey (2013) propose standout, where binary belief network is laid over the original network, and develop a heuristic approximation to optimize free energy. But, as pointed out by Gal et al. (2017) it is not scalable due to its need to significantly increase the model size.
In this paper, we propose simple and scalable contextual dropout module, whose dropout rates depend on the covariates æ, as a new approximate Bayesian inference method for NNs. With a novel design that reuses the main network to define how the boosts the performance while only slightly increases the memory and computational cost. Our method greatly enhances the flexibility of modeling, maintains the inherent advantages of dropout over conventional Bayesian NNs, and is generally simple to implement and scalable to the large-scale applications. We plug the contextual dropout module into various types of NN layers, including fully connected, convolutional, and attention layers. On variety of supervised learning tasks, contextual dropout achieves good performance terms of accuracy and quality of uncertainty estimation.
dropout rates produced,
CONTEXTUAL DROPOUT
We introduce efficient solution for data-dependent dropout: (1) treat the dropout probabilities as sample-dependent local random variables, (2) propose an efficient parameterization of dropout probabilities by sharing parameters between the encoder and decoder, and (3) learn the dropout distribution with a variational objective.
2.1 BACKGROUND ON DROPOUT MODULES
Consider a supervised learning problem with training data D := {æ;, Vi M where we model the conditional probability Po (v: x;) using NN parameterized by 0. Applying dropout to a NN often means wiseey reweighing each layer with a data-specific Bernoulii/Gaussian distributed random mask 2,. which are vid drawn from prior (z)(2) parameterized by (Hinton et al. 2012 Srivastava et al., 2014). This implics dropout training can be viewed as approximate Bayesian inference Gal & Ghahramani 2016). More specifically,
may view the learning objective of supervised learning model with dropout as a marginal--likeiiieee
log f [ p(y: xt,; z z)p(2z ())))),
To maximize this often intractable log- -orrinaall,, it is common to resort to variational inference (Hoffmaan et al. [2013] Blei et al. 2017) that introduces a variational distribution (z) on the random mask z and optimizes an evidence lower bound (ELBO):
C(D) - Eg(z) log 11%,, roln i (=),(()
(EN Ex)))(s) [log (w: lm..,))) KL (I,))) (1)
[(z)
where (L(g(=)(Ppx<))
E.te) q(2) log p(=)] is Kullback-Leibler (KL) divergence based
regularization term. Whether the KL term is explicitly imposed is key distinction between regular
dropout (Hinton et al. 2012 Srivastava et al. 2014) and their Bayesian generalizations (Gal &
Ghahramani 2016 2016 Gal et al. 2017 Kingma et al. 2013 Molchanov et al. 2017: B et al. 2020).
2.2
WEIGHT UNCERTAINTY
In regular dropout, as shown in I while we make the dropout masks data specific during opti- mization, we keep their distributions the same. This implies that while the NN weights can vary from data to data, their distribution is kept data invariant. In this paper, we propose contextual dropout, in which the distributions of dropout masks z; depend on covariates z; for each sample (a,, y:). Specifically, we define the variational distribution as q (z. xx)) where & denotes its NN parameters. In the framework of amortized variational Bayes (Kingma & Welling 2013 Rezende
2

---

Published as a conference paper at ICLR 2021
et al.
view 94 as inference network (encoder) trying to approximate the posterior p Vo. z;) x ply: Z4, (zz)). Note as we have no access to y during testing, we parameterize encoder in way that it depends on æ; but not y/- From the optimization point of view, what we propose corresponds to the ELBO of log IT p(ya 2 encoder, which can be expressed as
2014).
given qs (z; ; (x,) as the
L(D) = E. L(a,,9)) v.), [( Mo) = Ea æ.) [log (u w., z.)] (L (oees/=))lll (z)))p(( (2)
This ELBO differs from that of regular dropout in ①T in that the dropout distributions for 2; are now parameterized by 2; and a single KL regularization term is replaced with the aggregation of N data-dependent KL terms. Unlike conventional Bayesian NNs, as 2; is now local random variable, the impact of the KL terms will not diminish as N increases, and from the viewpoint of uncertainty quantification, contextual dropout relies only on aleatoric uncertainty to model its uncertainty on yi given z, Like conventional BNNs, may add epistemic uncertainty by imposing prior distribution on and/or Φ, and infer their posterior given D. As contextual dropout with a point estimate on both O and Φ is already achieving sfate-of-the- art performance, leave that extension for future research. In what follows, we omit the data index for simplification and formally define its model structure.
Cross-layer dependence: For NN with L layers, we denote {(2 ..... z²,, with 2' represent- ing the dropout masks at layer I. As we expect x' to be dependent on the dropout masks in previous layers { Jist, we introduce an autoregressive distribution as ( x) = I g0(2" x''-), where x'-1, the output of layer 1 1, is function of {z',...,'' x))
Parameter sharing between encoder and decoder: We aim build an encoder by model- ing ( [ where æ may come from complex and highly structured data such as images and natural languages. Thus, extracting useful features from æ to learn the encoder distribu- tion ls itself becomes a problem as challenging as the original one, í.c., extracting discrimi- native features from æ to predict y. As intermediate layers in the decoder network pe are al- ready learning useful features from the input, we choose to reuse them in the encoder, instead of extracting the features from scratch. If we denote layer of the decoder network by d then the output of layer I, given its input x'-1, would be U = (x' Considering this as learned feature for as illustrated in Figure I we build the encoder this output as a' = =M,(U'), draw z' conditioning on a', and element-wised multiply z' with U (with broad- cast if needed) to produce the output of layer as æ'. In this way, we use {0, (p) to parameterize the encoder, which reuses parameters of the decoder. To produce the dropout rates of the encoder, we only need extra parameters ço, the added memory and computational cost of which are often insignificant in comparison to these of the decoder.
ga
x!-
U
x
hlp
broadcast
n
ac
z'
Encoder
Prior and decoder
regularize
Figure I: A contextual dropout module.
2.3 EFFICIENT PARAMETERIZATTON OF CONTEXTUAL DROPOUT MODULE
Denote the output of layer by multidimensional array (tensor) U - (x--1) E x where D' denotes the number of the dimensions of U and C²2 denotes the number of elements along dimension d E (1...., D'). For efficiency, the output shape of hLg is not matched to the shape of U'. Instead, make it smaller and broadcast the contextual dropout masks z' across the dimensions of U (Tompson et al. (205)). Specifically, we parameterize dropout logits a' of the variational distribution to have C22 elements, where d E {1,...., D') is specified dimension of U We sample z' from the encoder and broadcast them across all but dimension d of U We sample (er))(aa)) under contextual Bernoulli dropout, and follow Srivastava et al. (2014) to use ((1,a(a')/(
(aa)))) for contextual Gaussian dropout. To obtain a' R the average pooling of U across all but dimension d, with the output denoted as Favepoot, (U'), and
first take
then apply two fully-connected layers / and &b connected by FNI., a (Leaky) ReLU based nonlinear activation function,
a' = (((' = Tb/ (Fsi (Favepuoo, (U')))
3

---

Published as a conference paper at ICLR 2021
g/-1
df
Itx Ne X
Ha X.
My dlg
product
H
Query
SofttWax
H X
Eraader
G
M
1x1x
1x1xH
Pegairizz
Ixtac
Ixtec
Value
Regularioz
Convolutional Layer
Attention Layer
Figure 2: Left: Contextual dropout convolution layers. Right: Contextual dropout attention layers.
where o is linear transformation mapping from R RC//, while & is from RC5/ back to RC,, with ? being reduction ratio controlling the complexity of hig. Below we describe how to apply contextual dropout to three representative types of NN layers.
Contextual dropout module for fully-connectdd layers layer I is a fully-connected layer and U R x01x-xxppp Di, we set af E R R where D' is the dimension that the linear transformation is applied to. Note, if E R then a' RR,,, and F is identity map, a' & (Pvi. ( (U))))
Contextual dropout module for convolutional layers: Assume layer / is a convolutional layer with C2 as convolutional channels and U RC xC,, Similar to Spatial Dropout (Tompson et al. 2015), set a' E C and broadcast its corresponding z' spatially as illustrated in Figure Such parameterization is similar to the squeeze-and-exx -nxxexxttnnnniiitiionn unit for convolutional layers, which has been shown be effective in image classification tasks (Hu al., 2018). However, in squeeze-and- excitation, (aa)) is used as channel wise soft attention weights instead of dropout probabilities, therefore it serves as a deterministic mapping in model instead of a stochastic unit used in the inference network.
Contextual dropout module for attention layers: Dropout has been widely used in attention layers (Xu et al.] 2015b] Vaswani et al. [2117] Yu al.) 2019). For example, can be applied to multi-head attention weights after the softmax operation (see illustrations in Figure 2 The weights are of dimension NN, NNql. where H is the number of heads, Nx the number of keys, and Nq the number of queries. In this case, find that setting a' R gives good performance. Intuitively, this coincides with the choice channel dimension for convolutional layers, as heads in attention could be analogized channels in convolution.
2.4 VARIATIONAL. INFERENCE FOR CONTEXTUAL DROPOUT
In contextual dropout, we choose C(D) = E(Ew))D L(x, y) shown in (2) as the optimization objective. Note in our design, the encoder de reuses the decoder parameters 0 to define its own parameters. Therefore, we copy the values of θ into & and stop the gradient of 0 when optimizing lø- This is theoretically sound (Ba & Frey, 2013). Intuitively, the gradients to 0 from pe are less noisy than that from d as the training of po(y yy x, 2) is supervised while that of (p is unsupervised. As what we have expected, allowing gradients from ds to backpropagate to 0 is found adversely affect the training of pe in our experiments. We use a simple prior Pn, making the prior distributions for dropout masks the same within each layer. The gradients with respect v and O can be expressed as
v,, Lx, y) Ez a)) V,, log (=)], VeL(æ, y) = Ez~go (: V log po(y x, x)], (4)
which are both estimated via Monte Carlo integration, using single z ( | for each æ.
Now, we consider the gradient of with respect to sp the components of Φ {0, sp) not copied from the decoder. For Gaussian contextual dropout, we estimate the gradients via the reparameterization
Note that full-connected layers can be applied to multi multi-dimensional tensor long specify the dimension along which the summation operation is conducted Abadi al. 2015).
4

---

Published as a conference paper at ICLR 2021
trick (Kingma & Welling 2013). For x' ~
rewrite it as z' ((cad)))((((a))))
1 + Va(a))) )/(1 (a'))e' where el ~ ~N(O,T), I). Similarly, sampling a sequence of {21) from (z Φ) can be rewritten as føle, x)) where fo is deterministic differentiable mapping and are aid standard Gaussian. The gradient (x, y) can now be expressed as (see pseudo code of
Algorithm in Appendix)
Ve y) = EE-NN(. (Vy (z, fele, x)) log (e ]
(5)
log (J
For Bernoulli contextual dropout, backpropagating the gradient efficiently is not straightforward, as the Bernoulli distribution not reparameterizable, restricting the use of the reparameterization trick.
In this case, a commonly used gradient estimator is the REINFORCE estimator (Williams) 1992)
(see details in Appendix ()). This estimator, however, is known to have high Monte Carlo estimation variance. To this end, estimate V.p. with the augment-REINFORRC REINFFRCEE-mrge (ARM) estimator (Yin & Zhoul 2018), which provides unbiased and low-variance gradients for the parameters of Bernoulli distributions. We defer the details of this estimator Appendix A We note there exists an improved ARM estimator (Yin et al. 2020 the performance.
Dong et al. 2020), applying which could further improve
2.5 TESTING AND COMPLEXITY ANALYSIS
Testing stage: To obtain point estimate, we follow the common practice in dropout (Srivastava al. 2014) multiply the neurons by the expected values of random dropout masks, which means that we predict y with po(y) x, 2), where Σ = B m) under the proposed contextual dropout. When uncertainty estimation is needed, we draw K random dropout masks approximate Py x, z (()),, where
the posterior predictive distribution of y given æ using ply x) t Σk
(1) (K)
kid ~ (2 x)).
The added computation and memory of contextual dropout insignificant due to the parameter sharing between the encoder and decoder. Extra memory and computational cost mainly comes from the part of hip- where both the parameter size and number of operations of order (((C))/7)) where 7 is from 8 to 16. This is insignificant, compared to the memory and computational cost of the main network, which are of order larger than O((C)))). We verify the point by providing memory and runtime comparisons between contextual dropout and other dropouts on ResNet in Table 3 (see more model size comparisons in Table in Appendix).
Complexity analysis:
2.6 RELATED WORK
Data-dependent variational distribution: Deng et al. (2018) model attentions latent-alignment variables and optimize a tighter lower bound (compared to hard attention) using learned inference network. To balance exploration and exploitation for contextual bandits problems, Wang & 7hou (2019) introduce local variable uncertainty under the Thompson sampling framework. However, their inference networks of are both independent of the decoder, which may considerably increase memory and computational cost for the considered applications. Fan et al. (2000) propose Bayesian attention modules with efficient parameter sharing between the encoder and decoder networks. Its scope is limited to attention units as Deng et al. (2018), while we demonstrate the general applicability of contextual dropout fully connected, convolutional, and attention layers supervised learning models. Conditional computation (Bengio et al. 2015; 2013) Shazeer et al., 2017: Teja Mullapudi al. 2018) tries to increase model capacity without a proportional increase computation, where an independent gating network decides turning which part of a network active and which inactive for each example. contextual dropout, the encoder works much like gating network choosing the distribution of sub-networks for each sample. But the potential gain in model capacity is even larger, e.g., there are potentially ~ (((2))z) combinations of nodes for L fully-connected layers, where is the order of the number of nodes for one layer. Generalization of dropout: DropConnect (Wan) et al. 2013) randomly drops the weights rather than the activations so as to generalize dropout. The dropout distributions for the weights, however, are still the same across different samples. Contextual dropout utilizes sample-dependent dropout probabilities, allowing different samples to have different dropout probabilities.
5

---

Published as a conference paper at ICLR 2021
3 EXPERIMENTS
Our method can be straightforwardly deployed wherever regular dropout can be utilized. To test its general applicability and scalability, we apply the proposed method to three representative types of NN layers: fully connected, convolutional, and attention layers with applications on MNIST (LeCun al. 2010), CIFAR (Krizhevsky et al. 2009), ImageNet (Deng al. 2009), and VQA-v2 (Goyal 2017 To investigate the model's robustness to noise, we also construct noisy versions of datasets by adding Gaussian noises to image inputs (Larochelle et al., (007).
et al.
For evaluation, we consider both the accuracy and uncertainty on predicting y given æ. Many metrics have been proposed to evaluate the quality of uncertainty estimation. On one hand, researchers are generating calibrated probability estimates to measure model confidence (Guo et al. 2017 Naeini jtt al. 2015| Kuleshov al. 2018). While expected calibration error and maximum calibration error have been proposed to quantitatively measure calibration, such metrics do not reflect how robust the probabilities are with noise injected into the network input, and cannot capture epistemic or model uncertainty (Gal & Ghahramani] 2016). On the other hand, the entropy of the predictive distribution as well as the mutual information, between the predictive distribution and posterior over network weights, used metries to capture both epistemic and aleatoric uncertainty (Mukhoti & Gal 2018). However, it is often unclear how large the entropy or mutual information is large enough to be classified uncertain, so such metric only provides relative uncertainty measure.
Hypothesis testing based uncertainty estimation: Unlike previous information theoretic metrics, use statistical test based method estimate uncertainty, which works for both single-label and multi-label classification models. One advantage of using hypothesis testing over information theoretic metrics is that the p-value of the test can be more interpretable, making it easier to be deployed in practice to obtain a binary uncertainty decision. To quantify how confident our model is about this prediction, we evaluate whether the difference between the empirical distributions of the two most possible classes from multiple posterior samples is statistically significant. Please see Appendix D for a detailed explanation of the test procedure.
Uncertainty evaluation via PAvPU: With the p-value of the testing result and a given p- p-value thresh- old, we can determine whether the model certain or uncertain about one prediction. To evaluate the uncertainty estimates, we uses Patch Accuracy vs Patch Uncertainty (PAvPU) (Mukhoti & Gal 2018),
which is defined as PAvPU- = (nae + n//w// ( + neu + T&e + Ti:)). where Nlae Raus Tiee Reju are the numbers of accurate and certain, accurate and uncertain, inaccurate and certain, inacourate and uncertain samples, respectively. This PAvPU evaluation metric would be higher if the model tends to generate the accurate prediction with high certainty and inaccurate prediction with high uncertainty.
3.1 CONTEXTUAL DROPOUT ON FULLY CONNECTED LAYERS
We consider an MLP with two hidden layers of size 300 and 100, respectively, with ReLU activations. Dropout applied to the input layer and the outputs of first two full-connected layers. We use MNIST as the benchmark. We compare contextual dropout with MC dropout (Gal & Ghahramani
2016), concrete dropout (Gal et al. 2017), Gaussian dropout (Srivastava et al., 2014), and Bayes by Backprop (Blundell et al. 2015). Pl Please see the detailed experimental setting in Appendix C.I.
Table I: Results on noisy MNIST with MLP.
METHODS
ACCURACY
PAYPU(0.05)
LOG LIKEI IHOOD
MC BERNOULLI
86.36
85.63
-1.72
MC GAUSSIAN
86,31
85.64
-1.72
CONCRETE
86,52
86.77
-1.68
BAYES BACKPROP
86.55
87.13
-2.30
CONTEXTUAL GATING
86,20
-1.81
CONTEXTUAL EXTING+DROPOLS
86.70
87.01
-1.71
BERNOULLI CONTEXTUAL
87, 87.43=0.330
87.81+0.23
-1.41 20.01
GAUSSIAN CONTEXTUAL
87.35+0.33
87.72+0.29
-1.4340.01
Results and analysis: In Table 1 we show accuracy, PAvPU (p-value threshold equal 0.05) and, test predictive loglikelihood with error bars (5 random runs) for models with different dropouts under the challenging noisy data (added Gaussian noise with mean variance 1). Note that the uncertainty
Results original data is deferred to Table in Appendix

---

Published as a conference paper at ICLR 2021
if reccurcte accurcte
10.0
eccurate
uscurdda
7.5
Dantey
5.0
2.5
0.0
0
0.0
0.2
0.4
0.0
0.2 Dripout probats tien dits
Dropous probubilaas fat Loya
Dregous probubiitias
Drcpeual probrbbaltte leyer 3
Figure 3: Visualization of dropout probabilities of Bernoulli contextual dropout on the MNIST dataset: the learned dropout probabilities seem to increase we go higher-level layers, also observed in Gal et al. 2017). With contextual dropout, different samples own different dropout probabilities. Inaccurate ones often have higher dropout probabilities corresponding to higher uncertainties. On the further right figure, we compare the dropout distributions across 3 representative digits. The dropout probabilities are overall higher for digit 8 compared to digit 1, mcaning 1 is casier to classify. The distribution for 5 has longer tails than others showing there more variations in the uncertainty for digit 5.
Accuracy
PAVPU 0.05
08.25
88.5
00.00
88.0-
07.50
87.5
07:25
87.0
07.00
86.5
75
Contextual dropout
86.00
dropost
dropout
10
15
200
25
2.0
4.0
45
5.0
100
15
2.0
2.5
3.0
3.5
4.0
4.5
5.0
Number nets
Number nets
Figure 4: The performance of combining different dropouts with deep ensemble on noisy MNIST data.
results for p-value threshold 0.05 is in general consistent with the results for other p-value thresholds (see more in Table in Appendix). We observe that contextual dropout outperforms other methods in all metrics. Moreover, compared to Bayes by Backprop, contextual dropout is more memory and computationally efficient. As shown in Table in Appendix, contextual dropout only introduces 16% additional parameters. However, Bayes by Backprop doubles the memory and increases the computations significantly as we need multiple draws of NN weights for uncertainty. Due to this reason, we do not include it for the following large model evaluations. We note that using the output of the gating network to directly scale activations (contextual gating) underperforms contextual dropout, which shows that the sampling process important for preventing overfitting and improving robustness to noise. Adding a regular dropout layer on the gating activations (contextual gating + dropout) improves little, but still underperforms contextual dropout, demonstrating that how we use the gating activations matters. In Figure 3 we observe that Bernoulli contextual dropout learns different dropout probabilities for different samples adapting the sample-level uncertainty which further verifies our motivation and supports the empirical improvements. For cumple-dependen dropout, the dropout probabilities would not vanish to zero even though the prior for regularization is also learned, because the optimal dropout probabilities for each sample is not necessarily zero. Enabling different samples to have different network connections could greatly enhance the model's capacity. The prior distribution also plays a different role here. Instead of preventing the dropout probabilities from going to zero, the prior tries to impose some similarities between the dropout probabilities of different samples.
Combine contextual dropout with Deep Ensemble: Deep ensemble proposed by Lakshmi
narayanan al. (2017) is simple way to obtain uncertainty by ensembling models trained indepen- dently from different random initializations. In Figure 4 we show the performance of combining different dropouts with deep ensemble on noisy MNIST data. As the number of NNs increases, both accuracy and PAvPU increase for all dropouts. However, Bernoulli contextual dropout outperforms other dropouts by a large margin in both metrics, showing contextual dropout compatible with deep ensemble and their combination can lead to significant improvements. Out of distribution (OOD) evaluation: we evaluate different dropouts in an OOD setting, where we train our model with clean data but test it noisy data. Contextual dropout achieves accuracy of 78.08, consistently higher than MC dropout (75.22) or concrete dropout (74.93). Meanwhile, the proposed method is also better at uncertainty estimation with PAvPU of 78.49, higher than MC (74.61) or Concrete (75.49).
7

---

Published as a conference paper at ICLR 2021
3.2 CONTEXTUAL DROPOUT ON CONVOLUTIONAL LAYERS
We apply dropout to the convolutional layers in WRN ZZaoruyko & Komodakis 2016). In Figure in Appendix, we show the architecture of WRN, where dropout applied to the first convolutional layer in each network block; in total, dropout is applied to 12 convolutional layers. We evaluate on CIFAR-I0 10 and CIFAR-100. The detailed setting is provided in Appendix C.1
Table 2: Results on CIFAR-100 with WRN.
Dropout
Original Data
Noisy Data
Accuracy
PAvPU (0.05)
log likelihood
Accuracy
PAvPU (0.05)
log likelihood
Bernoulli
79.03
61.5t 14
-4.99
$2.01
54.25
.4.55
Gaussian
76.63
78.05
-3.93
$1.38
57.02
-4.23
Concrete 79.19
-4.50
51.58
56.61
-4.56
Berooulli Contexiual
80,85 10.05
81.5610.31
-3.56
13.44
58.6310.50
-3.73.10.04
Gaussian Contextual
80.93-0.18
81.6910.16
-3.43:10.07
53.72-0034
58.49110.3
-3.81 10.03
Results and analysis: We show the results for CIFAR- 100 in Table 2 (sce CIFAR-10 10 results in
Tables & in Appendix). Accuracies, PAvPUs, and test predictive loglikelihoods are incorporated for both the original and noisy data. We consistently observe that contextual dropout outperforms other models in accuracy, uncertainty estimation, and loglikelihood.
Uncertainty visualization: We conducted extensive qualitative analyses for uncertainty evaluation.
In Figures I in Appendix F.2 we visualize 15 CIFAR images (with true label) and compare
the corresponding probability outputs of different dropouts in boxplots. We observe (1) contextual dropout predicts the correct answer if is certain, (2) contextual dropout certain and predicts the correct answers on many images for which MC or concrete dropout is uncertain, (3) MC or concrete dropout uncertain about some easy examples or certain on some wrong predictions (see details in Appendix F22) (4) on an image that all three methods have high uncertainty, contextual dropout places a higher probability on the correct answer than the other two. These observations verify that contextual dropout provides better calibrated uncertainty.
Table 3: Results on ImageNet with ResNet- 18.
DROPOUT
ToP-1 Acc
PAVPU
PARAMS
SEC/STEP
WITHOUT
69.75
NA
11.70M
1,44
+GAUSSIAN
69
72.86
11.70M
1,50
CONTTENTUAL
70.03++0.07
74.68+0.08
11.88M
1.64
*CNTTEXTUAL (SCRATCH)
70.29::0.09
76.47+00.1
11.88M
1.64
Large-scale experiments with ImageNet: Contextual dropout is also applied to the convolutional layers in ResNet-18 18, where we plug contextual dropout into a pretrained model, and fine-tune the pretrained model on ImageNet. In Table we show it is even possible to finetune pretrained model with contextual dropout module, and without much additional memory or run time cost, it achieves better performance than both the original model and the one with regular Gaussian dropout. Training model with contextual dropout from scratch can further improve the performance. See detailed experimental setting in Appendix C.T
3.3 CONTEXTUAL DROPOUT ON ATTENTION LAYERS
We further apply contextual dropout to the attention layers of VQA models, whose goal is to provide an answer to a question relevant to the content of a given image. We conduct experiments on the commonly used benchmark, VQA-v2 (Goyal et al. 2017), containing answer (QA) pairs. There are three types of questions: Yes/No, Number, and Other. In Figure [5] we show one example for each question type. There are 10 answers provided by 10 different human annotators for each question (see explanation of evaluation metrics in Appendix C.2) As shown the examples, VQA is generally so challenging that there are often several different human annotations for a given image. Therefore, good uncertainty estimation becomes even more necessary.
question-
Model and training specifications: We use MCAN (Yu et al. 2019). state-of-the-art Transformer- like model for VQA. Self-attention layers for question features and visual features, as well as the dnestion-guidde attention layers of visual features, stacked one another to build a deep model. Dropout is applied in every attention layer (after the softmax and before residual layer (Vaswani et al. 2017)) and fully-connected layer to prevent overfitting Yu et al. 2019), resulting in 62 dropout layers
8

---

Published as a conference paper at ICLR 2021
Avergee
Questoon: this person havin hard time with the iste?
looking BGG wate' Water. water, latontive]
Questos: How many people snown?
Aanctation set fyes', yes' yes! yeat yes yes' yes"
eyes'
Astoration 2:
yes, Yoo yes?
MC answen postuan 010
Concrete
Dananite atsowen yeb. 010
Goussian aroseer: p-salue:
Gaussiaa answer pesalue
Gosassiion arsopert waler prostee: 0.04
Demoull Consonial brownn E-Vake
Berncutl Contacusl answar p-ilki 0.
Bemoou Connunual answeer
GHERGARN Conseciian prAssam B1
GRUSsIOn Contonual
Figure 5: VQA visualization: for each question type, present image-question pair along with human annotations. We manually classify each prediction by different methods hased their answers and p-values. For questions that have clear answer, we define the good as certain accurate, the average as uncertain & accurate or uncertain inaccurate, and the bad as certain & inaccurate. Otherwise, we define the good as uncertain & accurate, the average as certain & accurate or uncertain & inaccurate, and the bad as certain & inaccurate.
in total. Experiments are conducted using the code of Yu et al. (2019) as basis. Detailed experiment setting is in Appendix C.2
Table 4: Accuracy and PAvPU on visual question answering.
DROPOUT
ACCURACY
PAVPU ORIGINAL DATA
ORIGINAL DATA
NOISY DATA
NOISY DATA
BERNOULLI EUUET AL. 2019
67.2
MC BERNOULLI
66.95
61.45 62.75
70.04
66.11 42.42
MC GAUSSIAN
66.96
70.77
CONCRETE
66.82
61.47
71.02
65.94
BERNOULLI CONTEXTUAL
67.42:10.06
63. 63.73:10.08
71.65
68.57=0.11
GAUSSIAN CONTEXTUAL
67.35
63.82:00.05
71.62
68.64:0004
We compare different dropouts on both the original VQA dataset and a noisy visual features. In Tables we show the overall accuracy and uncertainty estimation. The results show that on the original data, contextual dropout achieves better accuracy and uncertainty estimation than the others. Moreover, on noisy data, where the prediction becomes more challenging and requires more model flexibility and robustness, contextual dropouts outperform their regular dropout counterparts by a large margin in terms of accuracy with consistent improvement across all three question types.
Results and analysis:
version, where Gaussian noise with standard deviation 5 is added
in Appendix F3 we visualize some mnage-question pairs, along with the human annotations and compare the predictions and uncertainty estimations of different dropouts. We show three of them in Figure 3 As shown in the plots, overall contextual dropout is more conservative on its wrong predictions and more certain on its correct predictions than other methods (see more detailed explanations in Appendix F.3
Visualization: In Figures 12 15
4 CONCLUSION
We introduce contextual dropout as simple and scalable data-dependent dropout module that achieves strong performance in both accuracy and uncertainty estimation on a variety of tasks including large scale applications. With an efficient parameterization of the coviariate-dependent variational distribution, contextual dropout boosts the flexibility of Bayesian neural networks with only slightly increased memory and computational cost. We demonstrate the general applicability of contextual dropout on fully connected, convolutional, and attention layers, and also show that contextual dropout masks are compatible with both Bernoulli and Gaussian distribution. On both image classification and visual question answering tasks, contextual dropout consistently outperforms corresponding baselines. Notably, on ImageNet, we find it is possible to improve the performance of pretrained model by adding the contextual dropout module during finetuning stage. Based on these results, we believe contextual dropout can serve as an efficient alternative to iata-independent dropouts in the versatile tool box of dropout modules.

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGEMENTS
The authors acknowledge the support of Grants IIS-1812699, IIS-1812641, ECCS-1952193, CCF- 1553281, and CCF-193490) from the U.S. National Science Foundation, and the Texas Advanced Computing Center for providing HPC resources that have contributed to the research results reported within this paper. M. Zhou acknowledges the support of a gift fund from ByteDance Inc.
REFERENCES
Martnn Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro. Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin. Sanjay Ghemawat, Ian Goodfellow, Andrew Harp. Geoffrey Irving. Michael Isard, Yangqing Jia. Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur. Josh Levenberg. Dan Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shiens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow; Large- scale machine learning on heterogeneous systems, 2015. URL BRLLattp://teesoof.it available from tensorflow.org.
Software
Jimmy Ba and Brendan Frey. Adaptive dropout for training deep neural networks. In Advances in Neural Information Processing Systems, pp. 3084-3092, 2013.
Emmanuel Bengio, Pierre-Lue Bacon, Joelle Pineau, and Doina Precup. Conditional computation in neural networks for faster models. arXiv preprint arXiv:1511.0629 2015.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville, Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 1308.333. 2013.
David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisticians, Journal of the American Statistical Association, 112(518):859-8777 8777999-777770 2017.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. arXiv preprint 1505:4450 2015.
Shahin Boluki, Randy Ardywibowo. Siamak Zamani Dadaneh, Mingyuan Zhou, and Xiaoning Qian. Learnable Bernoulli dropout for Bayesian deep learning. In Artificial Intelligence and Statistics, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and puttern recognition, pp. 248-255 Ieee, 2009.
Yuntian Deng, Yoon Kim, Justin Chiu, Demi Guo, and Alexander Rush. Latent aligmment and variational attention. In Advances in Neural Information Processing Systems, pp. 9712-9724, 2018.
Zhe Dong, Andriy Mnih, and George Tucker. DisARM: An antithetic gradient estimator for binary latent variables. In Advances in Neural Information Processing Systems 33, 2020.
Xinjie Fan, Shujian Zhang, Bo Chen, and Mingyuan Zhou. Bayesian attention modules. Advances in Neural Information Processing Systems, 33, 2020.
Yarin Gal and Zoubin Ghahramani. Dropout as Bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059, 2016.
Yarin Gal. Jini Hron, and Alex Kendall. Concrete dropout. In Advances in Neural Information Processing Systems, pp. 3581-3590, 2017.
Asghar Ghasemi and Saleh Zahediasl. Normality tests for statistical analysis: A guide for non-statisticians. International journal of endocrinology and metabolism, 10(2):86,, 2012.
Yash Goyal, Tejas Khot, Douglas Summers-Stay. Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6904 6913, 2017.
Alex Graves. Practical variational inference for neural networks. In Advances in neural information processing xystems, pp. 2348-2356, 2011.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference Machine Learning-Voole Volume 70, pp. 1321-1330. JMLR org, 2017.
10

---

Published as a conference paper at ICLR 2021
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. Proceedings of the IEEE international conference on computer vision, pp. 1026-10333 2015.
Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Improving neural networks by preventing co-adaptation feature detectors. arXiv preprint 1207.v:100.05800 2012.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 96)::1335-178., 1997.
Matthew D Hoffiman, David M Blei, Chong Wang, and John Paisley. Stochastic variational inference. The Journal of Machine Learning Research, 14(1):1303-133) 2013.
Jie Hu, Li Shen, and Gang Sun. Squeere-and-excitation networks. In Proceedings of the IEEE conference on computer vision and pottern recognition, pp. 7132-7141, 2018
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 2014.
Diederik P Kingma and Max Welling, Auto-encoding variational Bayes, arXiv preprint arXiv:1312.6114 2013.
Durk P Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameterization trick. In Advances in Neural Information Processing Systems, pp. 2575-2583, 2015.
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. arXiv preprint 1rXXiv:8807.0020 2018.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances Neural Information Processing Systems, pp. 6402-6413.
2017.
Hugo Larochelle, Dumitru Erhan, Aaron Courville, James Bergstra, and Yoshua Bengio. An empirical evaluation of deep architectures on problems with many factors of variation. In Proceedings of the 24th international conference on Machine learning, 473-480, 2007.
Yann LeCun, Corinna Cortes, and CJ Burges. MNIST handwritten digit database. AT&T Labs [Online]. Available: http://yunn. lecun. com/exdb/nniist, 2:18, 2010.
Yann LeCun, Yoshua Bengio, Geoffrey Hinton. Deep learning. Nature, 521(7553):436 444, 2015.
Chunyuan Li, Changyou Chen, David Carlson, and Lawrence Carin. Preconditioned stochastic gradient Langevin dynamics for deep neural networks. Thirtieth AAAI Conference Artificial Intelligence, 2016.
Yang Li and Shihao Ji. L0-ARM: Network sparsification via stochastic binary optimization. In The European Conference on Machine Learning (ECML), 2019.
Christos Louizos and Max Welling. Multiplicative normalizing flows for variational Bayesian neural networks. In Proceedings of the 34th International Conference on Machine Learning-Volie Volume 70, pp. 2212-22182222 2218-2227. JMLR. RR org, 2017.
David JC MacKay. practical bayesian framework for backpropagation networks. Neural computation, 4(3): 448 472, 1992.
Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Variational dropout sparsifies deep neural networks. In Proceedings of the 34th International Conference on Machine Learning-Voos Voluue 70, pp. 2498-2507. JMLR. org, 2017.
Jishnu Mukhoti and Yarin Gal. Evaluating bayesian deep learning methods for semantic segmentation. arXiv preprint a8Xiv: 12709, 2018.
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using Bayesian binning. In Twenty-Nint AAAI Conference on Artificial Intelligence, 2015.
Radford M Neal. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012.
Yurii E Nesterov. A method for solving the convex programming problem with convergence rate (1/k" 2), In Dokl. akad. nauk Sssr, volume 269, pp. 543-5477 1983.
11

---

Published as a conference paper at ICLR 2021
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods natural language processing (EMNLP), pp. 1532- 1223,15,,2220 2014.
Shaoqing Ren. Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pp. 91 91-99, 2015.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In ICML, pp. 1278 1786, 2014.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz. Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outragcously large neural networks: The sparsely-gater mixture-of-experts layer. arXiv preprint arXiv: (63.0688,,, 2017.
Jiaxin Shi, Shengyang Sun, and Jun Zhu. Kernel implicit variational inference. In International Conference on Learning Representations, 2018. URL
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1): 1929 1958. 2014.
Ravi Teja Mullapudi, William R Mark, Noam Shazeer, and Kayvon Fatahalian. Hydranets: Specialized dynamic architectures for efficient inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8080-8089, 2018
Damien Teney, Peter Anderson. Xiaodong He, and Anton Van Den Hengel. Tips and tricks for visual question answering: I. Learnings from the 2017 challenge. Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pp. 4223-4232, 2018.
Jonathan Tompson, Ross Goroshin, Arjun Jain, Yann LeCun, and Christoph Bregler. Efficient object localization using convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogmition, pp. 648-656, 2015.
Ashish Vaswani, Noam Shazcer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you nced. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In International conference machine learning, pp. 1058-1066, 2013.
Zhendong Wang and Mingyuan Zhou. Thompson sampling via local uncertainty. arXiv preprint 1933v:1910.16672. 2019.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference machine learning (ICML-I1), pp. 681-688, 2011.
Ronald J Williams. Simple statistical gradient-followeee following algorithms for connectionist reinforcement learning. In Reinforcement Learning, pp. 5-32. Springer, 1992.
Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. Empirical evaluation of rectified activations in convolutional network. arXiv preprint 1rXiv:5505.088: 2015a.
Kelvin Xu, Jimmy Ba. Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning, pp. 2048-2057, 2015b.
Mingzhang Yin and Mingyuan Zhou. ARM: Augment-REINFORREEEE REINFORCE gradient for discrete latent variable models. Preprint, May 2018
Mingzhang Yin, Nhat Ho, Bowei Yan, Xiaoning Qian, and Mingyuan Zhou. Probabilistic Best Subset Selection by Gradient- Bandient-Based Optimization. arXiv e-prints, 2020.
Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian. Deep modular co-attention networks for visual question answering. In Proceedings of the IEF. Conference on Computer Vision and Pattern Recognition, pp. 6281-6290, 2019.
Sergey Zagoruyko and Nikos Komodakis, Wide residual networks. arXiv preprint orXiv:1605.07145 2016.
12

---

Published as a conference paper at ICLR 2021
Appendix
A DETAILS OF ARM GRADIENT ESTIMATOR FOR BERNOULLI CONTEXTUAL DROPOUT
In this section, we will explain the implementation details ARM for Bernoulli contextual dropout. To compute the gradients with respect the parameters of the variational distribution, commonly used gradient estimator is the REINFORCE estimator (Williams
1992) as
V (a v) 1 (1) (z) [r
2, V log (0(= x)), r(æz, 2, v) log pa(y [ [x,))p((=) ( a))
This gradient estimator is, however, known have high variance (Yin Zhou 2018). To mitigate this issue, we use ARM to compute the gradient with Bernoulli random variable.
ARM gradient estimator: In general, denoting ((x) = 1/(1 + e=a) as the sigmoid function, ARM expresses the gradient of {()) = _ (er(s, )) [x]]] as
VaE(o=) = I Uniform(* 10.1 [9ARM (T), (TRM(TT) = [r(zinu) - r( ( (1/2221111111 T), (6)
where Zuve 1 [ and sudo 1 [=)](-))]] are referred to the true and pscudo actions, respectively, and 1(:] E {0, 1]K is indicator function.
Sequential ARM: Note that the above equation is not directly applicable to our model due to the cross-layer dependence. However, the dropout masks within each layer independent of each other conditioned on these of the previous layers, so we can break our expectation into a sequence and apply ARM sequentially. We rewrite L Ez (- am) [r(2, z, y)]. When computing V.pC, we can ignore the tp in r as the expectation of V log go x)) is zero. Using the chain rule, have VyL= Et= Vol VVLVVa'' With decomposition C Eg us ax) Eg' Berla (or))((o))) [r(((r 2nd .y)]. where r(x, 1:: ,y) = Egttiie 104tiem0o((1m... (m,, (r( (r), z, 1) we know
IIe Uniterm (1,, )} V C = Bgtht 1i (x) F ~
(TR'( [r( ld-1 2 mime y) r(x, - /)] n²),
where Lur := [xx'<a(aa)) and suii := [[m'>o(-a"]]] We estimate the gradients via Monte Carlo integration. We provide the pseudo code in Algorithm !
Implementation details: The computational complexity of sequential ARM is O(L) times that of the decoder computation. Although it is embarrassiggly parallelizable, in practice, with limited computational resource available, it maybe be challenging to use sequential ARM when fairly large. In such cases, the original non-sequential ARM can be viewed an approximation to strike good balance between efficiency and accuracy (see the pseudo code in Algorithm 2 in Appendix). In cases, for image classification models, is small enough for MLP, 12 for WRN) for to sequential ARM. For VQA, L is as large as 62 and hence we choose the non-sequential ARM.
To control the learning rate of the encoder, we use scaled sigmoid function: ae(a²) = T+exp(-tar where a larger / corresponding to larger learning rate for the encoder. This function also used in Li & Ji (2019) to facilitate the transition of probability between 0 and for the purpose of pruning NN weights.
13

---

Published as a conference paper at ICLR 2021
B ALGORITHMS
Below, we present training algorithms for both Berouuli and Gaussian contextual dropout.
Algorithm 1: Bernoulli contextual dropout with sequential ARM
Input: data D, r, ( {h 1/2 (1/) step size Output: updated 0, sp, repeat
Gy =0 0;
Sample æ. y from data D;
x
for = 1 to L
U' = So (x 1, a' (U')
Sample 't' from Uniform(0.1);
inn 1 (a)a))
sudo = (=)))(-
if z! sudo then
r sudo None;
else
xd!
sudo U O si,s
for l = I + 1 to L do
U = get x sudo audo = hi (U sudo)
Sample π tutoo from Uniform (0, I);
* sudo == 1 1 ))
z sudo U
Nuudo
end for
r sudo
roo y)
end if
x U¹ C zimue
end for Firue
r(xtuue y) for 1 1 to L do
i is not None then
G = GGp + l(Timee (112 v''
end if
end for
4 yp + sGGp. with step-size s;
0 0 + Bog
(e Po
80
7 = 17 + 8 lox (.L.
until convergence
14

---

Published as a conference paper at ICLR 2021
Algorithm 2: Bernoulli contextual dropout with independent ARM
Input: data D, (g6 (-1 (h Man step size Output: updated 0, sp, 17 repeat
-p=0; 0:
Sample y from data D; x = x
for 1 = 1--10 to L do
U = o (,(" a' = No (U')
Sample nO from Uniform(0.1); inm : [ (a)))
x' = U C zinu
end for Tine r(xx y)
sudo
for 1 1 to L do
Ui sndo 90 ( a'usoo = -h (U (imss))
z sudo = 1 Th 201(-a
suddo sudo G 2 sudo
end for
suddo
r(x y):
for / = 1 to L do
G = Gp + l(Tirue - T sudoo (1/2 - a'
end for
4 yp + sGip, with step-size s; m.). n + Bog (=LL
0 0 +
8 log
po(y po
until convergence
15

---

Published as a conference paper at ICLR 2021
Algorithm 3: Gaussian contextual dropout with reparamaterization trick
Input: data D, (g6 (=1 (/)) step size Output: updated 0, sp, 77 repeat
Sample æ; y from data D; x0 for = to L do
U' = ( (x ) a' (U9)
Sample e¹ from N/ 1);
oe
Uz 2
end for
6 yp + sV (log po x., 21:t)
lor (2 with step-size
log (21:2)
0 0 + Blog (y m.2...);
00 dn
7 1) a log (.
until convergence
C DETAILS OF EXPERIMENTS
All experiments are conducted using a single Nvidia Tesla V100 GPU.
Table 5: Model size comparison among different methods.
METHOD
MLP
WRN
MCAN
RESNET-18
MC OR CONCRETE
267K
36.5M
58M
11.6M
CONTEXTUAL
311K
36.6M
6IM
11.8M
BAYES BY BACKPROP
534K
Choice of hyper-parameters in Contextual Dropout: Contextual dropout introduces two additional hyperparameters compared regular dropout. One is the channel factor ? for the encoder network. In our experiments, the results are not sensitive to the choice of the value of the channel factor Any number from 8 16 would give similar results, which also observed in (Hu al. 2018). The other is the sigmoid scaling factor that controls the learning rate of the encoder. We find that the performance is not that sensitive to its value and often beneficial to make smaller than the learning rate of the decoder. In all experiments considered in the paper, which cover various noise levels and model sizes, we have simply fixed it at = 0.01.
C.I IMAGE CLASSIFICATION
We consider MLP with two hidden layers of size 300 and 100, respectively, and use ReLU activations. Dropout is applied to all three full-connected layers. We use MNIST as the benchmark. All models trained for 200 epochs with batch size 128 and the Adam optimizer (Kingma & Ba 2014) (3, = 0.9, B, 0.999). The learning rate is 0.001. We compare contextual dropout with MC dropout (Gal Ghahramani 2016) and concrete dropout (Gal al., 2017). For MC dropout, we use the hand-tuned dropout rate at 0.2. For concrete dropout, we inttaaiize the dropout rate at 0.2 for Bernoulli dropout and the standard deviation parameter 0.5 for Gaussian dropout, and set the Concrete temperature at 0.11 (Gal et al. (017). We initialize the weights in contextual dropout with He-initialization preserving the magnitude of the variance of the weights the forward pass (He) et al. 2015). We initialize the biases in the way that the dropout rate is 0.2 when the weights for contextual dropout are zeros. We also initialize our prior dropout rate at 0.2. For hyperparameter tuning, hold out 10, 000 samples randomly selected from the training set for validation. We use the chosen hyperparameters to train on the full training set (60, 000 samples) and evaluate on the (10,000 testing set (10, 000 samples). We use Leaky ReL.U (Xu et al.
MIP:
2015a) with 0.1 the non-linear
16

---

Published as a conference paper at ICLR 2021
operator in contextual dropout. The reduction ratio y is set as 10, and sigmoid scaling factor t 0.01.
For Bayes by Backprop, we use - log σ1 = 0. - log d2 = 6, π = 0.2 (following the notation in the original paper). For evaluation, set M 20.
WRN: We consider WRN Za & Komodakis 2016), including 25 convolutional layers. In 6 show the architecture of WRN, where dropout is applied the first convolutional layer in each network block; in total, dropout is applied to 12 convolutional layers. We use CIFAR-10 10 and CIFAR-I00 (Krizhevsky et al. 2009) as benchmarks. All experiments are trained for 200 epochs with the Nesterov Momentum optimizer (Nesterov) 1983). whose base learning rate is set as 0.1, with decay factor 1/5 at epochs 60 and 120. All other hyperparameters are the same as MLP except for Gaussian dropout, where we use standard deviation equal to 0.8 for the CIFAR 100 with no noise and 1 for all other cases.
We used ResNet- 18 as the baseline model. We use momentum SGD, with learning rate 0.1, and momentum weight 0.9. Weight decay is utilized with weight 1e-4 For models trained from scratch, we train the models with 90 epochs. For finetuning models, we start with pretrained baseline ResNet models and finetune for epoch.
ResNet:
Group Name
Layers
Ioniginal (161)
corv2
[Conv (roppuu1160; Original Conv x1601)
conva
[Conv + Dropout (320); Original Conv [3201]
corv4
[Conv + Dropoutt Original Conv [6401)
Figure 6: Architecture of the Wide Residual Network.
C.2 VQA
Dataset: The dataset is split into the training (80k images and 444k QA pairs), validation (40k images and 214k QA pairs), and testing (80k images and 448k QA pairs) sets. We perform evaluation on the validation set the true labels for the test set are publicly available (Deng et al. 2018).
the evaluation for VQA is different from image classification. The aceu- racy for a single answer could be a number between 0 and 1 (Goyal et al. 2017): Acc(ans) min (# humann that said ans) /3, 1}. We generalize the uncertainly evaluation accordingly:
Evaluation metric:
Mac Σ, Acc,Crr, neu EX( Ace,)(1 - Cer.) Mau Σ, Acc((1 Cer,). = Σ,(1 Accr)(Cer:)
where for the ith prediction Acc; is the accuracy and Cer, E {0, 1] is the certainty indicator.
Experimental setting: We follow the setting by Yu et al. 2019), where bottom-up features extracted from images by Faster R-CNN (Ren et al.) 2015) are used as visual features, pretrained word- embeddings (Pennington al. 2014) and LSTM (Hochreiter & Schmidhuber 1997) are used to extract question features. We adopt the encoder-decodee structure in MCAN with six co-attention layers. We use the same model hyperparameters and training settings in Yu al. (2019) as follows: the dimensionality of input image features, input question features, and fused multi-modal features set be 2048, 512, and 1024, respectively. The latent dimensionality in the multi-head attention is 512, the number of heads is set to 8, and the latent dimensionality for each head is 64. The size of the answer vocabulary is set to
3129 using the strategy in Teney et al. (2018). To train the MCAN model, we use the Adam optimizer (Kingma & Ba 2014) with 31 = 0.9 and B 0.98. The base learning rate is set to min(2.5te
1e +). where the current epoch number starting from 1. After 10 epochs, the learning rate is decayed by 1/5 every 2 epochs. All the models are trained up to 13 epochs with the same batch size of 64.
We only conduct training on the training set (no data augmentation with visual genome dataset), and evaluation on the validation set. For MC dropout, we use the dropout rate of 0.1 1 for Bernoulli dropout as in Yu et al. (2019) and the standard deviation parameter of 1/3 for Gaussian dropout. For concrete dropout, we initialize the dropout rate at 0.1 and set the Concrete temperature at 0.1 (Gal et al. 2017). For hyperparameter tuning, randomly hold out 20% of the training set for validation. After tuning, we train on the whole training set and evaluate on the validation set. We initialize the weights with He- He-initialization preserving the magnitude of the variance of the weights in the forward pass
(He
17

---

Published as a conference paper at ICLR 2021
al. 2015). We initialize the biases in the way that the dropout rate is 0.1 when the weights for
contextual dropout are zeros. We also initialize our prior dropout rate at 0.1. We use ReLU as the non-linear operator in contextual dropout. We use 8 for layers with C 8, otherwise y 1. We set Be for residual layers.
D STATISTICAL TEST FOR UNCERTAINTY ESTIMATION
Consider M posterior samples of predictive probabilities (pm J I where Pm is vector with the same dimension as the number of classes. For single-label classífication models, pm is produced by softmax layer and sums to one, while for multi-label classification models, Pm is produced by sigmoid layer and each element is between 0 and 1. The former output is used in most image classification models, while the latter often used in VQA where multiple answers could be true for single imput. In both cases, to quamtify how confident our model is about this prediction, whether the difference between the probabilities of the first and second highest classes is statistically significant with statistical test. We conduct the normality test the output probabilities for both image classification and VQA models, and find most of the output probabilities are approximately normal (we randomly pick some Q-Q plots (Ghasemi & Zahedias]
evaluate
2012) and show them in Figures 7 and This motivates us to use two-sample t-tes In the following, we briefly summarize the Iwo-sample t-test we use.
Two sample hypothesis testing is an inferential statistical test that determines whether there is statistically significant difference between the means in two groups. The null hypothesis for the t-test is that the population means from the two groups are equal: 141 = #2, and the alternative hypothesis is μl * P2. Depending on whether each sample in one group can be paired with another sample in the other group, we have either paired (-test or independent t-test. In our experiments, we utilize both types of two sample (-test. For single-label model, the probabilities are dependent between two classes due the softmax layer, therefore, multi-label model, the probabilities are independent given the logits of the output layer, so we use the independent two-sample (-test.
the paired two- wwo-sample t-test; for
For paired two-sample t-test, we calculate the difference between the paired observations calculate the i-statistic below:
Y 8/VN N
T
where the mean differenne between the paired observations, s is the standard deviation of the differences, and N is the number of observations. Under the null hypothesis, this statistic follows t-distribution with N 1 degrees of freedom the difference is normally distributed. Then, we use this (-statistic and I-distribution calculate the corresponding p-value.
For independent two-sample l-test, we calculate the I-statistic as below:
T 81 Y2
/ s²/N2
22 Σ(n P1) + [y2 -Y2) Ni + N2-2 -2
where N1 and N2 are the sample sizes, and Y1 and Yn the sample means. Under the null hypothesis, this statistic follows a t-distribution with N1 + N2 degrees freedom if both yı and V/ normally distributed. We calculate the p-value accordingly.
To justify the assumption of the two-sample t-test, we run the normality test on the output probabil- ities for both image classification and VQA models. We find most of the output probabilities approximately normal. We randomly pick some Q-Q plots (Ghasemi & Zahedias] 2012) and show them in Figures 7 and
18

---

Published as a conference paper at ICLR 2021
Table 6: Complete results on MNIST with MLP
ORIGINAL
NOISY
PAVPUr0.011
MC DROPOUT- BERNOULLS
96. 6.62
98. 25 98. 39 198
86.36 36
84, 85,63 86, 10
MC DROPOUT GAUSSIAN
9k.67
98 23 98 41 98 do
86.3]
$3,99 $5.64 $5.03
CONCRETE DROPOUT
98.6 61.61
98. 08.431900 98.
50
86.32
85,987 $5.77/ $5.92
CONTENTUAL
98.9200088)
GAUSSIAN CONTENTUAL DROPOUT
98. 98.92(0.09)
(0.71(0.02)
96.
(9.00000077
87.35(0.35)
87.6410.19) (2.7780032)
Table 7: Loglikelihood original MNIST with MLP.
LOG LIKELIHOOD
MC BERNOULLI
-1.480 +0. +0,0004
MC GAUSSIAN
-..482010.0003
CONCRETE
-1,4822 +0.0012
BAYES BY BACKPROP
-1.4806 +0.0007
BERNOULLI CONTEXTUAL
-1.4537 +0.0005
GAUSSIAN CONTEXTUAL
-1.458910.0005
Table Complete results on CIFAR- 10 with WRN
ORIGINAL
DAYA
NOISY DATA
ACCURACY
PAYPU /..00 (6.1)
ACCURACY
PAYPU
70.13
BERNOULLI
DROPOUT
91.50 93.24 24
N0.43/
CUNCRETE DROPOUT
94.60
73.51
78.41 $1.01
79.34
72.72/ 73.89 / 74.72
BERNOULLL Cun TENTUAL DEUPOUT
95 92,0.10)
95. 25:0.25)
/11.25//995.444 795.74/0. (2) 02/00/60//1161
$1.49:0.19)
$2.56cn.50) /33,280 /33.//11..0000
GAUSSIAN CONTEXTUAL DRGFUUT
96,04/0.1)
96.10.0.0))
81.64:0.31)
82. 38 (0.43) 92.60,0.36)
/ 63.40003366
Table 9: Complete log likelihood results on CIFAR-10 10 with WRN
CIFAR-10
ORIGINAL DATA
NOISY DATA
MC DROPOUT BERNOULLI MC DROPOUT GAUSSIAN
-1,91
-1.93
-1.54
-1.22
CONCRETE DROPOUT
-1.98
-2.0
BERNOULLI CONTEXTUAL DROPOUT
-1.24
-1.47
GAUSSIAN CONTEXTUAL DROPOUT
-1.19
-1.51
Table 10: Complete results on CIFAR- 100 with WRN
ORIGINAL
NOISY
ACCURACY
PAYPUU00.11
0.05/0.1 0.13
ACCURACY
PAYPU10.0E /0.05 /0.D)
MC DROPOUT BERNOUELI MC DROPOUT - GAUISSIAN
79.03
S6, 90 61. 54 64,
$2.01
$3.36 25 54.63
76.63
77.35
78.05
/ 78.26
$1.38
56.85 / 57,57..00 02 57.31
CONCRETE DROPOLT
79.19
59.15 64.141 66.63
$1,58
57.62 56.617 55.89
BERNOUTLE CONTENTUAL DROPOUT
80 8..0550005 05)
(1.004/0 28) /81 56/0. 31)/81 86/0:21)
(3.64(04455
58,29/0. 301 SUAKND. 50) 05..9/9..6644 40)
GAUSSIAN CONTEXTUAL DROPOUT
80.93 (0.15)
81.43/0.11/
81.490.166) 1L..90../6//22222...11 /2.02/0.111)
53.72/0.54)
58.01(0.6)
(8.44(0.45) 58.95(0.37)
19

---

Published as a conference paper at ICLR 2021
E TABLES AND FIGURES FOR P-VALUE 0.01, 0.05 AND 0.1
F QUALITATIVE ANALYSIS
In this section, we include the Q-Q plots of the output probabilities as the normality test for the assumptions of two-sample t-test. In Figure we test the normality of differences between highest probabilities and second highest probabilities on WRN model with contextual dropout trained on the orignal CIFAR-10 10 dataset. In Figure 8 we test the normality of highest probabilities and second highest probabilities (separately) on VQA model with contextual dropout trained on the original VQA-v2 dataset. We use 20 data points for the plots.
F.1 NORMALITY TEST OUTPUT PROBABILITIES
Prosstiity
Prodsti
FP
255
Thenetica quantier
Theretica quintie
Thaortic cumbiie
Frobabi it Poot
20
Thorstic
Thoretior
Thaandoo couniii
Thoandor courtto
Figure 7: QQ Plot for differences between highest probabilities and second highest probabilities on WRN model with contextual dropout trained on the orignal CIFAR-10 10 dataset.
Probulsiiky
Probubility Pict
Probubiiity Pott
Thecossoll QEATTI
Theardoo courino
Probabilit
Probobiltty Poot
Probabiltty Poot
Probabiltty
-38 -33 50 Theowstoal garrin
-05
+39 45 -0.
-29 -13 -20
Figure 8: QQ Plot for output probabilities of VQA models: top row corresponds to the probability dis- tributions of the class with the highest probability, and the bottom row corresponds to the probability distributions of the class with the second highest probability.
NNote that also tried nonparametric test, Wileoxon rank-sum test, and obtain similar results.
20

---

Published as a conference paper at ICLR 2021
F.2 BOXPLOT FOR CIFAR-10
In this section, we visualize 5 most uncertain images for each dropout (only include Bernoulli, Concrete, and Contextual Bernoulli dropout for simplicity) leading to 15 images in total. The true images with the labels are on the left side and boxplots of probability distributions of different dropouts are on the right side. All models are trained on the original CIFAR- 10 dataset. Among these 15 images, we observe that contextual dropout predicts the right answer if it is certain, and is certain and predicts the right answer on many images that MC dropout or concrete dropout is uncertain about (e.g, many images in Figure 2 10). However, MC dropout or concrete dropout is uncertain about some easy examples (images in Figures
10 or certain on some wrong predictions (images in Figure [1]. Moreover, an image that all three methods have high uncertainty, concrete dropout often places a higher probability on the correct answer than the other two methods (images in Figure IT).
21

---

Published as a conference paper at ICLR 2021
Clinse:
p-value:
p-value:
p-value:
Glase: hrisee
p-value: 1,0
p-valus: 0.0
p-valus: 0,0
22
Clues: culomabilc
p-value: 1.0
p-value: 0.1
p-value: 0.0
0.
08
04
02
62
62
500
p-value: 1.0
p-value: 0.26
p-valus:
10
10
03
05
085
08
20
0.4
62
50
10
Classs: dog
p-value:
p-value:
p-value:
0.
Image
(b) Concrete Dropout
MC Dropout
(d) Contextual Dropout
Figure 9: Visualization of probability outputs of different dropouts CIFAR-10. 5 plots that Concrete Dropout is the most uncertain are presented. Number to class map: (0: airplane, automobile, 2: bird, 3: cat, 4: deer, 5: dog, 6: frog, 7: horse, 8: ship, 9: truck.)
22

---

Published as a conference paper at ICLR 2021
Clanee
p-value:
p-value:
p-value:
03
03
Glasan: dag
p-value: 0.11
p-valus: 1,0
p-valus: 0,0
03
08
Chuaz:: dag
p-value: 0.93
p-value: 1.0
p-value: 0.08
04
02
50
day
p-value: 0.88
p-valus: 1,0
p-value:
05
03
08
04
50
Classs: dog
p-value:
p-valte:
03
03
644
Image
(b) Concrete Dropout
(c) MC Dropout
Contextual Dropout
Figure 10: Visualization of probability outputs of different dropouts on CIFAR-10. 5 plots that MC Dropout is the most uncertain are presented. Number class map: (0: airplane, 1: automobile, bird, 3: cat, 4: deer, 5: dog, 6: frog. 7: horse, 8: ship, 9; truck.)
23

---

Published as a conference paper at ICLR 2021
p-value:
p-value:
p-value:
69
69
Glass cal
p-valus: 0.0
p-value: 0.01
p-valus: 1,0
10
080
000
08
08
2
Class ship 8
p-value: 0.14
p-value: 0.0
p-value: 0.99
10
63
0.5
0.1
03
68
66
0.2
80
ship
p-value:
p-value:
p-value: 0.99
0.5
64
20
500
3.0
Class: cal
0.033
10
0.5
Image
(b) Concrete Dropout
MC Dropout
Contextual Dropout
Figure 11: Visualization of probability outputs of different dropouts CIFAR-10. 5 plots that Contextual Dropout the most uncertain are presented. Number to class map: {0: airplane, automobile, 2: bird, 3: cat, 4: deer, 5: dog, 6: frog, 7: horse, 8: ship, 9: truck.)
24

---

Published as a conference paper at ICLR 2021
F.3 VISUALIZATION FOR VISUAL QUESTION ANSWERING
In Figures 12 15 we visualize some image-question pairs, along with the human annotations (for simplicity, only show the different answers in the annotation set) and compare the predictions and uncertainty estimations of different dropouts (only include Bernoulli dropout, Concrete dropout, and contextual Bernoulli dropout) on the noisy data. We include 12 randomly selected image-question pairs, and 6 most uncertain image-question pairs for each dropout as challenging samples (30 in total). For each sample, we manually rank different methods by the general rule that accurate and certain is the most preferred, followed by accurate and uncertain, inaccurate and uncertain, and then inaccurate and certain. For each image-question pair, we rank three different dropouts based on their answers and p-values, and highlight the best performing one, the second best, and the worst with green, yellow, and red, respectively (tied ranks are allowed). As shown the plots, overall contextual dropout is more conservative its wrong predictions and more certain its correct predictions than other methods for both randomly selected images and challenging images.
Oveatin what hos of soor the extyway?
Outtsion Whole outinday 67
(chor---s-od wosddet door boad glans" soar. wesdoved Soor Jacee eoenway?
"ary's" souteoy ttaa Ne ond way. hol aawe bobse]
Annossion sot (ana)
Rtank
64
senveet sooe 20
Rani
Question What are above the fomil rackety?
MC SShotso" portaks' pectues)
Danovila matling 5-eakos: 0. Dimteoting pation poonice
Queshoc: What the Hd coking A
CundON Whwrs the rastionn sathook: (oiner)
borse" bnower Sasaprosr Varterr Sevear)
prollc
Conorste p-vabc::
Figure 12: VQA visualization: 6 plots that Contextual Dropout is the most uncertain are presented.
25

---

Published as a conference paper at ICLR 2021
Gueahon: Whet tem would bother someone the
Quedtonn Whher number on ainner shin? Atridation Earninn DesoaN p-60000
Anvotation uniopacctllll eesarg??
Queston: woman diossed conservativry?
Antocation set (ves)
MC wnnwer tnile prantem 8.0
Concril REEHan priclac
Dolttomnl RYSWYY paallo0001
Amronioon set (swnsring Teaaings Raying doent MC arewen ariving paskan 0.0 antmiing S-ente:
dbing?
Questton: What's in the sard?
Arcolatio 7oL whttr. MC arawerr bike produe 0.5 Diniissta Bil availaac
Concrate arsweo lootornt a-walus 1.0
Figure 13: VQA visualization: plots that Concrete Dropout is the most uncertain are presented.
oU: Seavy. Sabry:
vveiiton act sog
FANEESSO SURAANG 1000 3066S)
Cancies BTONN conorise Cala
CONCING wwwwer Prokec
ANiidation set 19:54 935 7.55)
Concrate SRNANI pisalal 8.8
Contediull BRONNN moom wililll
Questio:: Whar conceet the stow nying promete? books: Summssr New Bur, heashig beech: beash: halkea?
Question: aign?
Astotatio set (10' 111 Imary. 9/ (8 BesAne: Breakun
Ctancratt wordee
Bontestiiol ansiee words pasalie: 0.
Contedtull
Eanorita RESIVNTT chmenn pooit
Figure 14: VQA visualization: 6 plots that MC Dropout is the most uncertain presented.
26

---

Published as a conference paper at ICLR 2021
What
Ansolation magnoury '11 MGGansa
Annotation det fabover Womm polor. nnew city.
anteeer
ue: veguzs yome
Annosston ser (yea: no)
Currostioll ROsser pradiim 0.00
Concrote answer dnyy Mo
Donesite conerttratiinn anfaeie p-wokun
What syes dog have? Asmotation /broun". Saack?
Quesiion: How
Annoration set: (4)
Garazdo prsem: Incen,, enstion D.1
Concrein answer 2. p-istuer 0.0
peratue
Questio: Would tie this joo? (yea,
Concrena you. posstumn
p-tatuer
cats?
Queston: What i the pitk oqject?
MC aenueee: yas, p-rauum
Arnoeiton aee (ywar ho?
MC toothonush p-olue: 0.0 Concrete arewer toothbraa's p-va OLD
Commeoina
Commstion fowar posakon
Guestoon: How many people are there?
Annotation sor (red" bray: hid and gray?
Annotetion ast (4, 22 Y?. (0) MC entwer problier 84
Queston: Now
MC smsower grey, preatue 0.57
Concretl p.paalue:
Dontextus 2irauue
Figure 15: VQA visualization: 12 randomly selected plots presented.
27


Reviews:
[{"review_id": "uO0GEL-hnhp", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Sample-dependent dropout for prediction and confidence improvements\n\n**Quality:**\n\n_Pros._ The authors assume that the sample-dependent dropout rate is necessary to improve model performance in aspects of prediction and confidence. In the Bayesian framework, they propose a contextual dropout module that is carefully designed considering computational efficiency since it is prone to increase model capacity and computational cost.\n\n_Cons._ Figures and tables are intentionally resized to fit the smaller areas. Some text is hard to read. Please increase the font size of the text in the figures for a printed paper.\n\n**Clarity:**\n\n_Pros._ The manuscript is well-written and straightforward to understand the proposed method. \n\n_Cons._ In Section 2.3, why do we have a multi-dimensional tensor for a fully-connected layer, instead of two-dimensional U^l? (in \"Contextual dropout module for fully-connected layers\" paragraph.)\n\n**Originality:**\n\n_Pros._ It explores the context-aware dropout method, while the other dropout-related works do not, with a few extra parameters.\nIncluding the ImageNet and the VQA experiments is helpful to illustrate the applicability of the proposed method.\nThey show that the contextual dropout can successfully apply to attention networks.\n\n_Cons._ The sample-dependent dropout has a resemblance to the squeeze-and-excitation module (the authors also mentioned in Section 2.3). In the related work section, more discussion on it would be helpful to recognize its novelty.\n\nSample-dependent dropout inevitably needs a sub-network to model a conditional distribution q of z for a given x. For this matter, it would be fair to compare with the squeeze-and-excitation module or self-attention models (with a small hidden dimension for parsimony; with the weights from sigmoid function). \n\nIn the proposed work, if the output of a sigmoid function is used for the real-valued weight (without the ARM estimator), instead of binary-sampled weight, does it significantly underperforms? \n\n**What expected in rebuttal:**\n\n(1) I would like to know that the ARM estimator in the proposed work is a critical factor, where the real-valued output of a sigmoid function is readily used for the dropout mask (as a weight). One could see this work as a variant of self-attention networks instead of a new dropout technique. In this viewpoint, the authors should defend with more persuasive logic (and experiments if possible) for their originality. \n\n(2) For the VQA experiment, MCAN (Yu et al., 2019) gets 67.2 in the original paper with a standard dropout module with the same setting as described in the Appendix. To be fair, this report should be included. Then, the improvement from a standard dropout is reduced to 0.22 (Bernoulli contextual dropout gets 67.42 with extra parameters.)\n\n**Minor comments:**\nFootnote 1 has extra space before the period.", "extra_scores": {}, "extra_text": {}}, {"review_id": "sCemgA99YN1", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper proposes a way of estimating data-dependent dropout probabilities. This is done in each layer using a small auxiliary neural network which takes the data (on which dropout is going to be applied) as input and outputs dropout probabilities, which are sampled and multiplied into the data.\n\nPros:\n- The proposed model consistently leads to improvements in accuracy and uncertainty estimation over standard dropout for multiple network types and tasks (including CNNs and Transformers).\n- The experimental results are thorough and include relevant p-values and error bars.\n- The method is sound and well explained.\n\nCons:\n- The gains in accuracy and uncertainty estimation for most tasks are small.\n- A concern with learning the dropout probabilities on the training set is that the optimal value of dropout is zero, since that would remove regularization, thereby minimizing the training loss. It is not clear how this is avoided in the proposed approach. One thing that could prevent the dropout rates from going to zero is the prior ((\\ \\eta \\\\). However, this is also learned. So it would be good to explain what prevents the dropout rates from becoming zero.\n- Conflating of 2 different effects : There are at least 2 aspects of the proposed model that could be beneficial : (1) Increase in model capacity from having a multiplicative gating interaction in the network (in expectation, the states are gated by \\\\(\\sigma(\\alpha^l)\\\\)), and (2) a decrease in model capacity (regularization) due to dropout noise.   An ablation analysis can help tease apart the benefits coming from these two sources.  This could be done, for example, by removing stochastic sampling (so that only the gating aspect remains) and optionally adding a regular dropout layer on the gated data. The increase in model capacity due to gating (aspect (1)) could partly explain why dropout rates do not become zero.\n\nOverall, the model leads to consistent gains in performance with relatively extra low computational cost which makes this a good contribution. However, the significance is limited because the results are only moderately better.\n\nPost rebuttal\nThe authors addressed the concerns around having a deterministic gating only baseline. I will increase my score to 7", "extra_scores": {}, "extra_text": {}}, {"review_id": "gSjUB7GE9ar", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\n \nThe paper proposes contextual dropout as a sample-dependent dropout module, which can be applied to different models at the expense of marginal memory and computational overhead. The authors chose to focus on Visual Question Answering and Image classification tasks.  The results in the paper show the contextual dropbox can improve accuracy on ImageNet and VQA2.0 datasets. \n\n\n##########################################################################\n\nReasons for score: \n\n \nI vote for accepting. I like the idea of data-dependent dropbox and the authors showing its applicability to various neural network layers like fully connected, convolutional, and attention layers. My major concern is that it seems that the improvement seems marginal in most cases.  The clarity of the paper can also be improved. Hopefully, the authors can address my concern in the rebuttal period. \n\n \n##########################################################################\n\nPros: \n\n \n1. The paper develops a variant of dropout regularization which was shown to be effective in training deep neural nets in the past several years. \n \n2. The proposed modified version is novel and capture data dependency more naturally relying on aleatoric uncertainty to model the uncertainty on y given x ( less explored). The computational and memory overhead is also insignificant. It is also compatible with both Bernoulli dropout and Gaussian dropout. \n\n3.  Contextual dropout outperform alternative dropout variants and the gain is a little higher when noise is introduced \n\n\n \n##########################################################################\n\nCons: \n\n \n1. complexity an additional parameters even if not many might make it hard to tune. More studies on the hyper-parameter selection are needed especially based on the level of noise and the parameter \n\n2. More experiments on a recurrent neural network model would be helpful to evaluate the generality of contextual dropout.  For example, applying contextual dropout on LSTMS / Transformer models in language tasks or vision&language tasks like image captioning. \n\n3. contrast/comparison to Drop connect[R1], which was proposed as a generalization of dropout [R1].\n\n[R1] Regularization of Neural Networks using DropConnect\nLi Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, Rob Fergus ; Proceedings of the 30th International Conference on Machine Learning, PMLR 28(3):1058-1066, 2013\n\n4. \n\n minor\n-----------\nFig5 font is not so clear and colors can be better\nwhy gamma is between 8 and 16\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an input-dependent dropout strategy, using variational inference to infer the rates.  While the idea is a fairly straightforward variant of recent probabilistic dropout methods, the paper demonstrates consistent improvements across several types of NN layers (dense, convolutional, and attention) in large-scale experiments (e.g. ImageNet).  The reviewers unanimously agreed on accepting the paper.
Decision: Accept (Poster)