## 2 Methods
### Monte Carlo Dropout  
Dropout [53] is a stochastic regularization technique where during training, noise in the model is introduced by randomly setting a proportion of the nodes in the model to zero in each batch. Which nodes that are set to zero is determined by a Bernoulli distribution. During prediction dropout is turned off, resulting in a point estimate of class probabilities. MC dropout is basically the same; however, during prediction dropout is still turned on, randomly shutting a proportion of the nodes off. In this way, dropout generates a distribution over the model parameters by repeating the nodes sampling several times and predicting for each configuration. The process is similar to a bootstrap procedure [54]. MC dropout thus produces a posterior predictive distribution for the class probabilities. The process of dropping out units in the feature space (i.e., dropping out nodes) can be translated to the parameters space. We denote the dropped out weights \(\widehat{\mathbf{\omega}}=\{\widehat{\mathbf{W}}_{1},\widehat{\mathbf{W}}_{2}, \mathbf{b}_{1},\mathbf{b}_{2}\}\) which allow for a convenient representation of the MC dropout objective function. Here we also use data sub-sampling (mini-batches) with random index set \(S\) of size \(M\) and  
\[\widehat{\mathbf{\omega}}_{i}=\{\widehat{\mathbf{W}}_{1}^{i},\widehat{\mathbf{W}}_ {2}^{i},\mathbf{b}_{1},\mathbf{b}_{2}\}=\{\text{diag}(\widehat{\mathbf{e}}_{1}^{i })\mathbf{W}_{1},\text{diag}(\widehat{\mathbf{e}}_{2}^{i})\mathbf{W}_{2},\mathbf{ b}_{1},\mathbf{b}_{2}\},\]  
with \(\widehat{\mathbf{e}}_{1}^{i}\sim p(\widehat{\mathbf{e}}_{1}^{i})\) and \(\widehat{\mathbf{e}}_{2}^{i}\sim p(\widehat{\mathbf{e}}_{2}^{i})\), for \(1\leq i\leq N\) where \(p(\mathbf{e}_{1})\) and \(p(\mathbf{e}_{2})\) is Bernoulli distributions with probabilities \(p_{1}\) and \(p_{2}\) respectively. Thus, we define the objective function over the mini-batch sets such that  
\[ \begin{split}\widehat{\mathcal{L}}(\mathbf{\omega})=-\frac{1}{M} \sum_{i\in S}\log p(\mathbf{Y}_{i}|\mathbf{f}^{\widehat{\mathbf{e}}_{i}}(\mathbf{X} _{i}))\\ +\lambda_{1}\|\mathbf{W}_{1}\|_{2}^{2}+\lambda_{2}\|\mathbf{W}_{ 2}\|_{2}^{2}+\lambda_{3}\|\mathbf{b}_{1}\|_{2}^{2}+\lambda_{4}\|\mathbf{b}_{2 }\|_{2}^{2}.\end{split}  \tag{3}\]  
Though back-propagation and stochastic gradient decent we find optimal parameters for the model \(\mathbf{f}^{\mathbf{\omega}}\) as  
\[\operatorname*{arg\,min}_{\mathbf{\omega}}\widehat{\mathcal{L}}(\mathbf{\omega}).\]  
We must integrate over the weights to get the posterior predictive distribution, as shown in Equation (2). With weights optimized we can use the model to predict new classes given some new input \(\mathbf{x}^{*}\). To approximate the posterior predictive distribution we use a MC estimator to integrate over the weights. The weights \(\widehat{\mathbf{\omega}}_{t}\sim p(\mathbf{\omega}|\mathbf{X},\mathbf{Y})\) are generated from the posterior distribution such that,  
\[p(\mathbf{y}^{*}|\mathbf{x}^{*},\mathbf{X},\mathbf{Y})=\int p(\mathbf{y}^{*}| \mathbf{x}^{*},\mathbf{\omega})p(\mathbf{\omega}|\mathbf{X},\mathbf{Y})d\mathbf{\omega} \approx\frac{1}{T}\sum_{t=1}^{T}\mathbf{f}^{\widehat{\mathbf{\omega}}_{t}}( \mathbf{x}^{*})\xrightarrow[T\to\infty]{}\mathbb{E}[\mathbf{y}^{*}],\]  
which is an unbiased estimator and what we referred to as MC dropout. In [51] [PERSON] showed that variational inference could be approximated by MC dropout. This is the case if the derivatives of the two objective functions corresponding variational inference and MC dropout, with respect to the model parameters, are equal. This is true if the so-called KL condition is satisfied. In the same paper, [PERSON] showed that Gaussian priors over the model parameters satisfy this KL condition for large enough hidden units. So, with Gaussian priors, MC dropout approximates variational inference. While there could be some other priors that satisfy the KL condition, MC dropout and variational inference are not generally equivalent.