Title: BAYESIAN UNCERTAINTY ESTIMATION FOR BATCH NORMALIZED DEEP NETWORKS Anonymous authors Paper under double-blind review

Abstract: Deep neural networks have led to a series of breakthroughs, dramatically improving the state-of-the-art in many domains. The techniques driving these advances, however, lack a formal method to account for model uncertainty. While the Bayesian approach to learning provides a solid theoretical framework to handle uncertainty, inference in Bayesian-inspired deep neural networks is difficult. In this paper, we provide a practical approach to Bayesian learning that relies on a regularization technique found in nearly every modern network, batch normalization. We show that training a deep network using batch normalization is equivalent to approximate inference in Bayesian models, and we demonstrate how this finding allows us to make useful estimates of the model uncertainty. Using our approach, it is possible to make meaningful uncertainty estimates using conventional architectures without modifying the network or the training procedure. Our approach is thoroughly validated in a series of empirical experiments on different tasks and using various measures, showing it to outperform baselines on a majority of datasets with strong statistical significance.

1. INTRODUCTION:

Deep learning has dramatically advanced the state of the art in a number of domains, and now surpasses human-level performance for certain tasks such as recognizing the contents of an image (He et al., 2015) and playing Go (Silver et al., 2017) . But, despite their unprecedented discriminative power, deep networks are prone to make mistakes. Sometimes, the consequences of mistakes are minor -misidentifying a food dish or a species of flower (Liu et al., 2016) may not be life threatening. But deep networks can already be found in settings where errors carry serious repercussions such as autonomous vehicles (Chen et al., 2016) and high frequency trading. In medicine, we can soon expect automated systems to screen for skin cancer (Esteva et al., 2017) , breast cancer (Shen, 2017) , and to diagnose biopsies (Djuric et al., 2017) . As autonomous systems based on deep learning are increasingly deployed in settings with the potential to cause physical or economic harm, we need to develop a better understanding of when we can be confident in the estimates produced by deep networks, and when we should be less certain.

Standard deep learning techniques used for supervised learning lack methods to account for uncertainty in the model, although sometimes the classification network's output vector is mistakenly understood to represent the model's uncertainty. The lack of a confidence measure can be especially problematic when the network encounters conditions it was not exposed to during training. For example, if a network trained to recognize dog breeds is given an image of a cat, it may predict it to belong to a breed of small dog with high probability. When exposed to data outside of the distribution it was trained on, the network is forced to extrapolate, which can lead to unpredictable behavior. In such cases, if the network can provide information about its uncertainty in addition to its point estimate, disaster may be avoided. This work focuses on estimating such predictive uncertainties in deep networks (Figure 1 ).

The Bayesian approach provides a solid theoretical framework for modeling uncertainty (Ghahramani, 2015) , which has prompted several attempts to extend neural networks (NN) into a Bayesian setting. Most notably, Bayesian neural networks (BNNs) have been studied since the 1990's (Neal, 2012) . Although they are simple to formulate, BNNs require substantially more computational resources than their non-Bayesian counterparts, and inference is difficult. Importantly, BNNs do

MCBN MCDO CUBN CUDO:

Figure 1 : We propose a method to estimate uncertainty in any network using batch normalization (MCBN). Here, we show results on a toy dataset from networks with three hidden layers (30 units per layer). The solid line is the predictive mean of 500 stochastic forward passes. The outer area depicts the model's uncertainty as the 95% CI of the predictive distribution for each x value (inner shaded area is 50% CI). On the right, we show a similar plot using dropout to estimate uncertainty (MCDO) (Gal & Ghahramani, 2015) . The bottom row depicts a minimally useful baseline -the same networks but with a constant uncertainty (CUBN, CUDO).

not scale well and struggle to compete with modern deep learning architectures. Recently, Gal & Ghahramani (2015) developed a practical solution to obtain uncertainty estimates by casting dropout training in conventional deep networks as an approximate Bayesian model. They showed that any network trained with dropout is an approximate Bayesian model, and uncertainty estimates can be obtained by computing the variance on multiple predictions with different dropout masks.

This technique, called Monte Carlo Dropout (MCDO), has a very attractive quality: it can be applied to existing NNs without any modification to the architecture or the way the network is trained. Uncertainty estimates come (nearly) for free. However, in recent years dropout has fallen out of favor, limiting MCDO's utility. Google's Inception network, which won ILSVRC in 2014, did not use dropout (Szegedy et al., 2015) , nor did the ILSVRC 2015 winner, Microsoft's residual learning network (He et al., 2016) . In place of traditional techniques like dropout, most modern networks such as Inception and ResNet have adopted other regularization techniques. In particular, batch normalization (BN) has become widespread thanks to its ability to stabilize learning with improved generalization (Ioffe & Szegedy, 2015) .

An interesting aspect of BN is that the mini-batch statistics used for training each iteration depend on randomly selected batch members. We exploit this stochasticity and show that training using batch normalization, like dropout, is equivalent to approximate inference in Bayesian models1 . We demonstrate how this finding allows us to make meaningful estimates of the model uncertainty in a technique we call Monte Carlo Batch Normalization (MCBN) (Figure 1 ). The method we propose makes no simplifying assumptions on the use of batch normalization, and applies to any network using BN as it appears in practical applications.

We validate our approach by empirical experiments on eight standard datasets used for uncertainty estimation. We measure uncertainty quality relative to a baseline of fixed uncertainty, and show that MCBN outperforms the baseline on nearly all datasets with strong statistical significance. We also show that the uncertainty quality of MCBN is on par with that of MCDO. As a practical demonstration of MCBN, we apply our method to estimate segmentation uncertainty using a conventional segmentation network (Badrinarayanan et al., 2015) . Finally, as part of our evaluation, we make contributions to the methodology of measuring uncertainty quality by defining performance bounds on existing metrics and proposing a new visualization that provides an intuitive understanding of uncertainty quality.

2. RELATED WORK:

Bayesian models provide a natural framework for modeling uncertainty, and several approaches have been developed to adapt NNs to Bayesian reasoning. A common approach is to place a prior distribution (often a Gaussian) over each weight. For infinite weights, the resulting model corresponds to a Gaussian process (Neal, 1995) , and for a finite number of weights it corresponds to a Bayesian neural network (MacKay, 1992) . Although simple to formulate, inference in BNNs is difficult (Gal, 2016) . Therefore, focus has shifted to techniques to approximate the posterior distribution, leading to approximate BNNs. Methods based on variational inference (VI) typically rely on a fully factorized approximate distribution (Kingma & Welling, 2014; Hinton & Van Camp, 1993) but these methods do not scale easily. To alleviate these difficulties, Graves (2011) proposed a model using sampling methods to estimate a factorized posterior. Another approach, probabilistic backpropagation (PBP), also estimates a factorized posterior based on expectation propagation (Hernández-Lobato & Adams, 2015) .

Deep Gaussian Processes (DGPs) formulate GPs as Bayesian models capable of working on large datasets with the aid of a number of strategies to address scaling and complexity requirements (Bui et al., 2016) . The authors compare DGP with a number of state-of-the-art approximate BNNs, showing superior performance in terms of RMSE and uncertainty quality2 . Another recent approach to Bayesian learning, Bayesian hypernetworks, use a neural network to learn a distribution of paramaters over another neural network (Krueger et al., 2017) . Although these recent techniques address some of the difficulties with approximate BNNs, they all require modifications to the architecture or the way networks are trained, as well as specialized knowledge from practitioners.

Recently, Gal (2016) showed that a network trained with dropout implicitly performs the VI objective. Therefore any network trained with dropout can be treated as an approx. Bayesian model by making multiple predictions as forward passes through the network while sampling different dropout masks for each prediction. An estimate of the posterior can be obtained by computing the mean and variance of the predictions. This technique, referred to here as MCDO, has been empirically demonstrated to be competitive with other approx. BNN methods and DGPs in terms of RMSE and uncertainty quality (Li & Gal, 2017) . However, as the name implies, MCDO depends on dropout. While once ubiquitous in training deep learning models, dropout has largely been replaced by batch normalization in modern networks, limiting its usefulness.

3. METHOD:

The methodology of this work is to pose a deep network trained with batch normalization as a Bayesian model in order to obtain uncertainty estimates associated with its predictions. In the following, we briefly introduce Bayesian models and a variational approximation to it using Kullback-Leibler (KL) divergence following Gal & Ghahramani (2015) . We continue by showing a batch normalized deep network can be seen as an approximate Bayesian model. Then, by employing theoretical insights as well as empirical analysis, we study the induced prior on the parameters when using batch normalization. Finally, we describe the procedure we use for estimating uncertainty of batch normalized deep networks' output.

3.1. BAYESIAN MODELING:

We assume a finite training set D = {(x i , y i )} i=1:N where each (x i , y i ) is a sample-label pair.

Using D, we are interested in learning an inference function f ω (x, y) with parameters ω. In deterministic models, the estimated label ŷ is obtained as follows:

ŷ = arg max y f ω (x, y)

We assume f ω (x, y) = p(y|x, ω) (e.g. in soft-max classifiers), and is normalized to a proper probability distribution. In Bayesian modeling, in contrast to finding a point estimate of the model parameters, the idea is to estimate an (approximate) posterior distribution of the model parameters p(ω|D) to be used for probabilistic prediction:

p(y|x, D) = f ω (x, y)p(ω|D)dω

The predicted label, ŷ, can then be accordingly obtained by sampling p(y|x, D) or takings its maxima.

Variational Approximation In approximate Bayesian modeling, it is a common approach to learn a parametrized approximating distribution q θ (ω) that minimizes KL(q θ (ω)||p(ω|D)); the Kullback-Leibler (KL) divergence of posterior w.r.t. its approximation, instead of the true posterior.

Minimizing this KL divergence is equivalent to the following minimization while being free of the data term p(D)3 :

L VA (θ) := - N i=1 q θ (ω) ln f ω (x i , y i )dω + KL(q θ (ω)||p(ω))

Using Monte Carlo integration to approximate the integral with one realized ωi for each sample i4 , and optimizing over mini-batches of size M , the approximated objective becomes:

EQUATION

The first term is the data likelihood and the second term is divergence of the model prior w.r.t. the approximated distribution.

We now describe the optimization procedure of a deep network with batch normalization and draw the resemblance to the approximate Bayesian modeling in Eq (1).

3.2. BATCH NORMALIZED DEEP NETS AS BAYESIAN MODELING:

The inference function of a feed-forward deep network with L layers can be described as:

f ω (x) = W L a(W L-1 ...a(W 2 a(W 1 x))

where a(.) is an element-wise nonlinearity function and W l is the weight vector at layer l. Furthermore, we denote the input to layer l as x l with x 1 = x and we then set h l = W l x l . Parenthesized super-index for matrices (e.g. W (j) ) and vectors (e.g. x (j) ) indicates jth row and element respectively. Super-index u refers to a specific unit at layer l, (e.g. W u = W l,(j) , h u = h l,(j) ).5 

Batch Normalization Each layer of a deep network is constructed by several linear units whose parameters are the rows of the weight matrix W. Batch normalization is a unit-wise operation proposed in Ioffe & Szegedy (2015) to standardize the distribution of each unit's input. It essentially converts a unit's output h u in the following way:

ĥu = h u -E[h u ] Var[h u ]

where the expectations are computed over the training set6 . However, often in deep networks, the weight matrices are optimized using back-propagated errors calculated on mini-batches of data. Therefore, during training, the estimated mean and variance on the mini-batch B is used, which we denote by µ B and σ B respectively. This makes the inference at training time for a sample x a stochastic process, varying based on other samples in the mini-batch.

Loss Function and Optimization:

Training deep networks with mini-batch optimization involves a (regularized) risk minimization with the following form:

L RR (ω) := 1 M M i=1 l(ŷ i , y i ) + Ω(ω)

Where the first term is the empirical loss on the training data and the second term is a regularization penalty acting as a prior on model parameters ω. If the loss l is cross-entropy for classification or sum-of-squares for regression problems (assuming i.i.d. Gaussian noise on labels), the first term is equivalent to minimizing the negative log-likelihood:

L RR (ω) := - 1 M τ M i=1 ln f ω (x i , y i ) + Ω(ω).

with τ = 1 for classification. In a batch normalized network the model parameters are

{W 1:L , γ 1:L , β 1:L , µ 1:L B , σ 1:L B }.

If we decouple the learnable parameters θ = {W 1:L , γ 1:L , β 1:L } from the stochastic parameters ω = {µ 1:L B , σ 1:L B }, we get the following objective at each step of the mini-batch optimization of a batch normalized network:

L RR (θ) := - 1 M τ M i=1 ln f {θ, ωi} (x i , y i ) + Ω(θ) (2)

where ωi is the mean and variances for sample i's mini-batch at a certain training step. Note that while ωi formally needs to be i.i.d. for each training example, a batch normalized network samples the stochastic parameters once per training step (mini-batch). For a large number of epochs, however, the distribution of sampled batch members for a given training example converges to the i.i.d. case.

Comparing Eq. (1) and Eq. (2) reveals that the optimization objectives are identical, if there exists a prior p(ω) corresponding to Ω(θ) such that ∂ ∂θ KL(q θ (ω)||p(ω)) = N τ ∂ ∂θ Ω(θ). In a batch normalized network, q θ (ω) corresponds to the joint distribution of the normalization parameters µ 1:L B , σ 1:L B , as implied by the repeated sampling from D during training. This is an approximation of the true posterior, where we have restricted the posterior to lie within the domain of our parametric network and source of randomness. With that we can use a pre-trained batch normalized network to estimate the uncertainty of its prediction using the inherent stochasticity of BN. Before that, we briefly discuss what Bayesian prior is induced in a typical batch normalized network.

3.3. PRIOR p(ω):

The purpose of Ω(θ) is to reduce variance in deep networks. L2-regularization, also referred to as weight decay (Ω(θ) = λ l=1:L ||W l || 2 ), is a popular technique in deep learning. The induced prior from L2-regularization is studied in Appendix 6.5. Under some approximations as outlined in the Appendix, we find that BN for a deep network with FC layers and ReLU activations induce Gaussian distributions over BN unit's means and standard deviations, centered around the population values given by D (Eq. ( 6), details in Appendix 6.3). Factorizing this distribution across all stochastic parameters and assuming Gaussian priors, we find the approximate corresponding priors:

p(µ u B ) = N (0, J l-1 x 2 2N τ λ l ) p(σ u B ) = N (µ p , σ 2 p )

where J l-1 is the dimensionality of the layer's inputs and x is the average input over D for all input units. In the absence of scale and shift transformations from the previous BN layer, it converges towards an exact prior for large training datasets and deep networks (under the assumptions of the factorized distribution). The mean and variance for the BN unit's standard deviation, µ p and σ 2 p , have no relevance for the reconciliation of the optimization objectives of Eq. ( 1) and (2).

3.4. PREDICTIVE UNCERTAINTY IN BATCH NORMALIZED DEEP NETS:

In the absence of the true posterior we rely on the approximate posterior to express an approximate predictive distribution:

p * (y|x, D) := f ω (x, y)q θ (ω)dω

Following Gal & Ghahramani (2015) we estimate the first and second moment of the predictive distribution empirically (see Appendix 6.4 for details). For regression, the first two moments are:

E p * [y] ≈ 1 T T i=1 f ωi (x) Cov p * [y] ≈ τ -1 I + 1 T T i=1 f ωi (x) f ωi (x) -E p * [y] E p * [y]

where each ωi corresponds to sampling the net's stochastic parameters ω = {µ 1:L B , σ 1:L B } the same way as during training. Sampling ωi therefore involves sampling a batch B from the training set and updating the parameters in the BN units, just as if we were taking a training step with B. Recall that from a VA perspective, training the network amounted to minimizing KL(q θ (ω)||p(ω|D)) wrt θ. Sampling ωi from the training set, and keeping the size of B consistent with the mini-batch size used during training, ensures that q θ (ω) during inference remains identical to the approximate posterior optimized during training.

After each update of the net's stochastic parameters, we take a forward pass with input x, producing output f ωi (x). After T such stochastic forward passes, we compute the mean and sample variance of outputs to find the mean E p * [y] and variance Cov p * [y] of the approximate predictive distribution. Note that Cov p * [y] also requires addition of constant variance from observation noise, τ -1 I.

The network is trained just as a regular BN network. The difference is in using the trained network for prediction. Instead of replacing ω = {µ 1:L B , σ 1:L B } with population values from D, we update these parameters stochastically, once for each forward pass . 7The form of p * can be approximated by a Gassuian for each output dimension (for regression). We assume bounded domains for each input dimension, wide layers throughout the network, and a unimodal distribution of weights centered at 0. By the Liapounov CLT condition, the first layer then receives approximately Gaussian inputs (a proof can be found in Lehmann (1999)). Having sampled µ u B and σ u B from a mini-batch, each BN unit's output is bounded. CLT thereby continues to hold for deeper layers, including f ω (x) = W L x L . A similar motivation for a Gaussian approximation of Dropout has been presented by Wang & Manning (2013) .

The actual form of p * is likely to be highly multimodal, as can be seen immediately from f ω (x) = W L x L with elements in x L normalized, scaled and shifted differently. Gal & Ghahramani (2015) note the multimodality as well, since MCDO implies a bimodal variational distribution over each weight matrix column.

4. EXPERIMENTS AND RESULTS:

We assess the uncertainty quality of MCBN quantitatively and qualitatively. Our quantitative analysis relies on eight standard regression datasets, listed in Table 1 . Publicly available from the UCI Machine Learning Repository (University of California, 2017) and Delve (Ghahramani, 1996) , these datasets have been used to benchmark comparative models in recent related literature (see Hernández-Lobato & Adams (2015) , Gal & Ghahramani (2015) , Bui et al. (2016) and Li & Gal (2017) ). We report results using standard metrics, and also propose useful upper and lower bounds to normalize these metrics for a more meaningful interpretation in Section 4.2. In cases where the raw datasets contain more than one target feature, the feature used is specified by target feature.

Our qualitative results consist of three parts. First, in Figure 1 we demonstrate that MCBN produces reasonable uncertainty bounds on a toy dataset in the style of (Karpathy, 2015) . Second, we develop a new visualization of uncertainty quality by plotting test errors sorted by predicted variance in Figure 2 . Finally, we apply MCBN to SegNet (Kendall et al., 2015) , demonstrating the benefits of MCBN in an existing batch normalized network.

4.1. METRICS:

We evaluate uncertainty quality based on two metrics, described below: Predictive Log Likelihood (PLL) and Continuous Ranked Probability Score (CRPS). We also propose upper and lower bounds for these metrics which can be used to normalize them and provide a more meaningful interpretation.

Predictive Log Likelihood (PLL) Predictive Log Likelihood is a widely accepted metric for uncertainty quality, used as the main uncertainty quality metric for regression (e.g. (Hernández-Lobato & Adams, 2015), (Gal & Ghahramani, 2015) , (Bui et al., 2016) and (Li & Gal, 2017) ). A key property is that PLL makes no assumtions about the form of the distribution. The measure is defined for a probabilistic model f ω (x) and a single observation (y i , x i ) as:

PLL(f ω (x), (y i , x i )) = log p(y i |f ω (x i ))

where p(y i |f ω (x i )) is the model's predicted PDF evaluated at y i , given the input x i . A more detailed description is given in Appendix 6.4. The metric is unbounded and maximized by a perfect prediction (mode at y i ) with no variance. As the predictive mode moves away from y i , increasing the variance tends to increase PLL (by maximizing probability mass at y i ). While PLL is an elegant measure, it has been criticized for allowing outliers to have an overly negative effect on the score (Selten, 1998) .

Continuous Ranked Probability Score (CRPS) Continuous Ranked Probability Score is a less sensitive measure that takes the full predicted PDF into account. A prediction with low variance that is slightly offset from the true observation will receive a higher score form CRPS than PLL.

In order for CRPS to be analytically tractable, we need to assume a Gaussian unimodal predictive distribution. CRPS is defined as

CRPS(f ω (x i ), (y i , x i )) = ∞ -∞ F (y) -1(y ≥ y i ) 2 dy

where F (y) is the predictive CDF, and 1(y ≥ y i ) = 1 if y ≥ y i and 0 otherwise (for univariate distributions) (Gneiting & Raftery, 2007) . CRPS is interpreted as the sum of the squared area between the CDF and 0 where y < y i and between the CDF and 1 where y ≥ y i . A perfect prediction with no variance yields a CRPS of 0; for all other cases the value is larger. CRPS has no upper bound.

4.2. BENCHMARK MODELS AND NORMALIZED METRICS:

In order to establish a lower bound on useful performance for uncertainty estimates, we define a baseline that predicts constant variance regardless of input. This benchmark model produces identical point estimates as MCBN, which yield the same predictive means. 1 .

For a probabilistic model f , an upper bound on uncertainty performance can also be defined for CRPS and PLL. For each observation (y i , x i ), a value for the predictive variance T i can be chosen that maximizes PLL or minimizes CRPS8 . Using CUBN as a lower bound and the optimized CRPS score as the upper bound, uncertainty estimates can be normalized between these bounds (1 indicating optimal performance, and 0 indicating performance on par with fixed uncertainty). We call this normalized measure CRPS =

CRPS(f,(yi,xi))-CRPS(f CU ,(yi,xi))

min T CRPS(f,(yi,xi))-CRPS(f CU ,(yi,xi)) × 100, and the PLL analogue PLL = PLL(f,(yi,xi))-PLL(f CU ,(yi,xi)) max T PLL(f,(yi,xi))-PLL(f CU ,(yi,xi)) × 100. This normalized measure gives an intuitive understanding of how close a Bayesian model is to estimating the perfect uncertainty for each prediction.

We also evaluate CRPS and PLL for an adaptation of the authors' implementation of Multiplicative Normalizing Flows (MNF) for variational Bayesian networks (Louizos & Welling, 2017) . This is a recent model specialized to allow a more flexible posterior what is achievable by e.g. MCDO's bimodal variational over weight columns. MNF uses auxillary variables on which the posterior is a latent. By applying normalizing flows to the auxillary variable such that it can take on complex distributions, the approximate posterior becomes highly flexible.

4.3. TEST SETUP:

Our evaluation of MCBN and MCDO is largely comparable to that of Hernández-Lobato & Adams (2015) , in that we use similar datasets and metrics. This setup was later also followed by Gal & Ghahramani (2015) , where we in comparison implement a different hyperparameter selection, allow for a larger range of dropout rates, and use larger networks with two hidden layers.

With the exception of Protein Tertiary Structure9 , all our models share a similar architecture: two hidden layers with 50 units each, using ReLU activations. Input and output data were normalized during training. Results were averaged over five random splits of 20% test and 80% training and cross-validation (CV) data. For each split, 5-fold CV by grid search with a RMSE minimization objective was used to find training hyperparameters and optimal n.o. epochs. For BN-based models, the hyperparameter grid consisted of a weight decay factor ranging from 0.1 to 1 -15 by a log 10 scale, and a batch size range from 32 to 1024 by a log 2 scale. For DO-based models, the hyperparameter grid consisted of the same weight decay range, and dropout probabilities in {0.2, 0.1, 0.05, 0.01, 0.005, 0.001}. DO-based models used a batch size of 32 in all evaluations.

The model with optimal training hyperparameters was used to optimize τ numerically. This optimization was made in terms of average CV CRPS for MCBN, CUBN, MCDO, and CUDO respectively, before evaluation on the test data.

All estimates for the predictive distribution were obtained by taking 500 stochastic forward passes through the network, throughout training and testing. The implementation was done with Tensor-Flow. The Adam optimizer was used to train all networks, with a learning rate of 0.001. The extensive part of the experiments (i.e. training and cross validation) was done on Amazon web services using 3000 machine-hours. All code necessary for reproducing both the quantitative and qualitative results is released in an anonymous github repository (https://github.com/iclr-mcbn/mcbn).

4.4. TEST RESULTS:

A summary of the results measuring uncertainty quality of MCBN, MCDO and MNF are provided in Table 2 . Tests are run over eight datasets using 5 random 80-20 splits of the data with 5 different random seeds each split. We report CRPS and PLL, expressed as a percentage, which reflects how close the model is to the upper bound. The upper bounds and lower bounds for each metric are de- scribed in Section 4.2. We check to see if the reported values of CRPS and PLL significantly exceed the lower bound models (CUBN and CUDO) using a one sample t-test, where the significance level is indicated by *'s. Further details from the experiment are available in Appendix 6.6.

In Figure 2 , we provide a novel visualization of uncertainty quality visualization in regression datasets. Errors in the model predictions are sorted by estimated uncertainty. The shaded areas show the model uncertainty and gray dots show absolute prediction errors on the test set. A gray line depicts a running mean of the errors. The dashed line indicates the optimized constant uncertainty. In these plots, we can see a correlation between estimated uncertainty (shaded area) and mean error (gray). This trend indicates that the model uncertainty estimates can recognize samples with larger (or smaller) potential for predictive errors.

Qualitative results for Bayesian SegNet using MCBN was produced by using the main CamVid model in Kendall et al. (2015) . The pre-trained model was obtained from the online model zoo and was used without modification. 10 instances of mini-batches with size 6 were used to estimate the mean and variance of MCBN. Qualitative results can be found in Figure 3 depicting intuitive uncertainty at object boundaries. Quantitative measures on various segmentation datasets can be obtained and is beyond the scope of this work.

We provide additional experimental results in Appendix 6.6. In Tables 3 and 4 , we show the mean CRPS and PLL values for MCBN and MCDO. These results indicate that MCBN performs on par with MCDO across several datasets. In Table 6 we provide RMSE results of the MCBN and MCDO networks in comparison with non-stochastic BN and DO networks. These results indicate that the procedure of multiple forward passes in MCBN and MCDO show slight improvements in the predictive accuracy of the network.

5. DISCUSSION:

The results presented in Table 2 and Appendix 6.6 indicate that MCBN generates meaningful uncertainty estimates which correlate with actual errors in the model's prediction. We show statistically significant improvements over CUBN in the majority of the datasets, both in terms of CRPS and PLL. The visualizations in Figure 2 and in Appendix 6.6 show clear correlations between the estimated model uncertainty and actual errors produced by the network. We perform the same experiments using MCDO, and find that MCBN generally performs on par with MCDO. Looking closer, in terms of CRPS, MCBN performs better than MCDO in more cases than not. However, care must be used when comparing different models. The learned network parameters are different, leading to different predictive means which can confound direct comparison.

The results on the Yacht Hydrodynamics dataset seem contradictory. The CRPS score for MCBN is extremely negative, while the PLL score is extremely positive. The opposite trend is observed for MCDO. To add to the puzzle, the visualization in Figure 2 depicts an extremely promising uncertainty estimation that models the predictive errors with high fidelity. We hypothesize that this strange behavior is due to the small size of the data set, which only contains 60 test samples, or due to the Gaussian assumption of CRPS. There is also a large variability in the model's accuracy on this dataset, which further confounds the measurements for such limited data.

One might criticize the overall quality of the uncertainty estimates of MCBN and MCDO based on the magnitude of the CRPS and PLL scores in Table 2 . The scores rarely exceed 10% improvement over the lower bound. However, we caution that these measures should be taken in context. The upper bound is very difficult to achieve in practice (it is optimized for each test sample individually),

and the lower bound is a quite reasonable estimate for uncertainty. We have further compared against the recent work of Louizos & Welling (2017) , and find comparable results to their MNF-based variational technique specifically targeted to increase the flexibility of the approximate posterior.

Our approximation of the implied prior in Appendix 6.5 also provides a new interpretation of the empirical evidence that significantly lower λ should be used in batch normalized networks (Ioffe & Szegedy, 2015) . From a VA perspective, too strong a regularization for a given dataset size could be seen as constraining the prior distribution of BN units' means, effectively narrowing the approximate posterior.

In this work, we have shown that training a deep network using batch normalization is equivalent to approximate inference in Bayesian models. Using our approach, it is possible to make meaningful uncertainty estimates using conventional architectures without modifying the network or the training procedure. We show evidence that the uncertainty estimates from MCBN correlate with actual errors in the model's prediction, and are useful for practical tasks such as regression or semantic image segmentation. Our experiments show that MCBN yields an improvement over the baseline of optimized constant uncertainty on par with MCDO and MNF. Finally, we make contributions to the evaluation of uncertainty quality by suggesting new evaluation metrics based on useful baselines and upper bounds, and proposing a new visualization tool which gives an intuitive visual explanation of uncertainty quality. Finally, it should be noted that, over the past few years, batch normalization has become an integral part of most-if-not-all cutting edge deep networks which signifies the relevance of our work for estimating model uncertainty. 

6. APPENDIX:

6.1 VARIATIONAL APPROXIMATION Assume we were to come up with a faimly of distributions parametrised by θ in order to approximate the posterior, q θ (ω). Our goal is to set θ such that q θ (ω) is as similar to p(ω|D) as possible.

One strategy is to minimizing KL(q θ (ω)||p(ω|D)), the KL divergence of p(ω|D) wrt q θ (ω). Minimizing KL(q θ (ω)||p(ω|D)) is equivalent to maximizing the ELBO:

ω q θ (ω) ln p(Y|X, ω)dω -KL(q θ (ω)||p(ω))

Assuming i.i.d. observation noise, this is equivalent to minimizing:

L VA (θ) := - N n=1 q θ (ω) ln p(y i |f ω (x i ))dω + KL(q θ (ω)||p(ω))

Instead of making the optimization on the full training set, we can use a subsampling (yielding an unbiased estimate of L VA (θ)) for iterative optimization (as in mini-batch optimization):

LVA (θ) := - N M i∈B ω q θ (ω) ln p(y i |f ω (x i ))dω + KL(q θ (ω)||p(ω))

We now make a reparametrisation: set ω = g(θ, ) where is a RV. The function g and the distribution of must be such that p(g(θ, )) = q θ (ω). Assume q θ (ω) can be written q θ (ω| )p( )d where q θ (ω| ) = δ(ω -g(θ, )). Using this reparametrisation we get:

LVA (θ) = - N M i∈B p( ) ln p(y i |f g(θ, ) (x i ))d + KL(q θ (ω)||p(ω))

6.2. KL DIVERGENCE OF FACTORIZED GAUSSIANS:

If q θ (ω) and p(ω) factorize over all stochastic parameters:

KL(q θ (ω)||p(ω)) = - ω i q θ (ω i ) ln i p(ω i ) i q θ (ω i ) dω = - ω i q θ (ω i ) i ln p(ω i ) q θ (ω i ) i dω i = j - ω i q θ (ω i ) ln p(ω j ) q θ (ω j ) i dω i = j - ωj q θ (ω j ) ln p(ω j ) q θ (ω j ) dω j ω i =j i =j q θ (ω i )dω i = i - ωi q θ (ω i ) ln p(ω i ) q θ (ω i ) dω i = i KL(q θ (ω i )||p(ω i ))

(3) such that KL(q θ (ω)||p(ω)) is the sum of the KL divergence terms for the individual stochastic parameters ω i . If the factorized distributions are Gaussians, where q θ (ω i ) = N (µ q , σ 2 q ) and p(ω i ) = N (µ p , σ 2 p ) we get:

KL(q θ (ω i )||p(ω i )) = ωi q θ (ω i ) ln q θ (ω i ) p(ω i ) dω i = -H(q θ (ω i )) - ωi q θ (ω i ) ln p(ω i )dω i = - 1 2 (1 + ln(2πσ 2 q )) - ωi q θ (ω i ) ln 1 (2πσ 2 p ) 1/2 exp - (ω i -µ p ) 2 2σ 2 p dω i = - 1 2 (1 + ln(2πσ 2 q )) + 1 2 ln(2πσ 2 p ) + E q [ω 2 i ] -2µ p E q [ω i ] + µ 2 p 2σ 2 p = ln σ p σ q + σ 2 q + (µ q -µ p ) 2 2σ 2 p - 1 2 (4)

for each KL divergence term. Here H(q θ (ω i )) = 1 2 (1 + ln(2πσ 2 q )) is the differential entropy of q θ (ω i ).

6.3. DISTRIBUTION OF µ u:

B , σ u

B:

Here we approximate the distribution of mean and standard deviation of a mini-batch, separately to two Gaussians. For the mean we get:

µ B = Σ M m=1 W (j) x m M

where x m are the examples in the sampled batch. We will assume these are sampled i.i.d. 10 . Samples of the random variable W (j) x m are then i.i.d.. Then by central limit theorem (CLT) the following holds for sufficiently large M (often ≥ 30):

µ B ∼ N (µ, σ 2 M )

For standard deviation:

σ B = Σ M m=1 (W (j) x m -µ B ) 2 M Then √ M (σ B -σ) = √ M Σ M m=1 (W (j) x m -µ B ) 2 M - √ σ 2

We want to rewrite

Σ M m=1 (W (j) xm-µB) 2 M

. We take a Taylor expansion of f ) x m ] = µ and write

(x) = √ x around a = σ 2 . With x = Σ M m=1 (W (j) xm-µB) 2 M : √ x = √ σ 2 + 1 2 √ σ 2 (x -σ 2 ) + O[(x -σ 2 ) 2 ] so √ M (σ B -σ) = √ M 1 2 √ σ 2 Σ M m=1 (W (j) x m -µ B ) 2 M -σ 2 + O Σ M m=1 (W (j) x m -µ B ) 2 M -σ 2 2 = √ M 2σ 1 M Σ M m=1 (W (j) x m -µ B ) 2 -σ 2 + O √ M Σ M m=1 (W (j) x m -µ B ) 2 M -σ 2 2 = 1 2σ √ M Σ M m=1 (W (j) x m -µ B ) 2 -M σ 2 + O √ M Σ M m=1 (W (j) x m -µ B ) 2 M -σ 2 2 consider Σ M m=1 (W (j) x m -µ B ) 2 . We know that E[W (j

Σ M m=1 (W (j) x m -µ B ) 2 =Σ M m=1 ((W (j) x m -µ) -(µ B -µ)) 2 =Σ M m=1 ((W (j) x m -µ) 2 + (µ B -µ) 2 -2(W (j) x m -µ)(µ B -µ)) =Σ M m=1 (W (j) x m -µ) 2 + M (µ B -µ) 2 -2(µ B -µ)Σ M m=1 (W (j) x m -µ) =Σ M m=1 (W (j) x m -µ) 2 -M (µ B -µ) 2 =Σ M m=1 ((W (j) x m -µ) 2 -(µ B -µ) 2 ) then √ M (σ B -σ) = 1 2σ √ M Σ M m=1 ((W (j) x m -µ) 2 -(µ B -µ) 2 ) -M σ 2 + O √ M Σ M m=1 (W (j) x m -µ B ) 2 M -σ 2 2 = 1 2σ √ M Σ M m=1 (W (j) x m -µ) 2 -Σ M m=1 (µ B -µ) 2 -M σ 2 + O √ M Σ M m=1 (W (j) x m -µ B ) 2 M -σ 2 2 = 1 2σ √ M Σ M m=1 ((W (j) x m -µ) 2 -σ 2 ) -Σ M m=1 (µ B -µ) 2 + O √ M Σ M m=1 (W (j) x m -µ B ) 2 M -σ 2 2 = 1 2σ √ M Σ M m=1 ((W (j) x m -µ) 2 -σ 2 ) - 1 2σ √ M Σ M m=1 (µ B -µ) 2 + O √ M Σ M m=1 (W (j) x m -µ B ) 2 M -σ 2 2 = 1 2σ √ M Σ M m=1 ((W (j) x m -µ) 2 -σ 2 ) term A - √ M 2σ (µ B -µ) 2 term B + O √ M Σ M m=1 (W (j) x m -µ B ) 2 M -σ 2 2 term C

We go through each term in turn

Term A We have Term A = 1 2σ √ M Σ M m=1 ((W (j) x m -µ) 2 -σ 2 )

where ) x m ] = µ it holds that E[(W (j) x m -µ) 2 ] = σ 2 . Since (W (j) x m -µ) 2 is sampled approximately iid (by assumptions above), for large enough M by CLT it holds approximately that

Σ M m=1 (W (j) x m -µ) 2 is the sum of M RVs (W (j) x m -µ) 2 . Note that since E[W (j

Σ M m=1 (W (j) x m -µ) 2 ∼ N (M σ 2 , M Var((W (j) x m -µ) 2 )) where Var((W (j) x m -µ) 2 ) = E[(W (j) x m -µ) 2 * 2 ] -E[(W (j) x m -µ) 2 ] 2 = E[(W (j) x m -µ) 4 ] -σ 4 Then Σ M m=1 ((W (j) x m -µ) 2 -σ 2 ) ∼ N (0, M * E[(W (j) x m -µ) 4 ] -M σ 4 ) so Term A ∼ N (0, E[(W (j) x m -µ) 4 ] -σ 4 4σ 2 )

Under review as a conference paper at ICLR 2018

Term B:

We have

Term B = √ M 2σ (µ B -µ) 2 = 1 2σ √ M (µ B -µ)(µ B -µ) Consider (µ B -µ). As µ B p -→ µ when M → ∞ we have µ B -µ p -→ 0. We also have √ M (µ B -µ) = Σ M m=1 W (j) x m √ M - √ M µ

which by CLT is approximately Gaussian for large M . We can then make use of the Cramer-Slutzky Theorem, which states that if

(X n ) n≥1 and (Y n ) n≥1 are two sequences such that X n d -→ X and Y n p -→ a as n → ∞ where a is a constant, then as n → ∞, it holds that X n * Y n d -→ X * a.

Thus, Term B is approximately 0 for large M.

Term C:

We have

Term C = O √ M Σ M m=1 (W (j) x m -µ B ) 2 M -σ 2 2

Since E[(W (j) x m -µ) 2 ] = σ 2 we can make the same use of Cramer-Slutzky as for Term B, such that Term C is approximately 0 for large M.

Finalizing the distribution:

We have approximately

√ M (σ B -σ) ∼ N (0, E[(W (j) x m -µ) 4 ] -σ 4 4σ 2 ) so σ B ∼ N (σ, E[(W (j) x m -µ) 4 ] -σ 4 4σ 2 M ) 6.4 PREDICTIVE DISTRIBUTION PROPERTIES

This section provides derivations of properties of the predictive distribution p * (y|x, D) in section 3.4, following Gal (2016) . We first find the approximate predictive mean and variance for the approximate predictive distribution, then show how to estimate the predictive log likelihood, a measure of uncertainty quality used in the evaluation 4.

Predictive mean Assuming Gaussian iid noise defined by model precision τ , i.e.

f ω (x, y) = p(y|f ω (x)) = N (y; f ω (x), τ -1 I): E p * [y] = yp * (y|x, D)dy = y y ω f ω (x, y)q θ (ω)dω dy = y y ω N (y; f ω (x), τ -1 I)q θ (ω)dω dy = ω y yN (y; f ω (x), τ -1 I)dy q θ (ω)dω = ω f ω (x)q θ (ω)dω ≈ 1 T T i=1 f ωi (x)

where we take the MC Integral with T samples of ω for the approximation in the final step.

Predictive variance Our goal is to estimate:

Cov p * [y] = E p * [y y] -E p * [y] E p * [y]

We find that:

E p * [y y] = y y yp * (y|x, D)dy = y y y ω f ω (x, y)q θ (ω)dω dy = ω y y yf ω (x, y)dy q θ (ω)dω = ω Cov fω(x,y) (y) + E fω(x,y) [y] E fω(x,y) [y] q θ (ω)dω = ω τ -1 I + f ω (x) f ω (x) q θ (ω)dω = τ -1 I + E q θ (ω) [f ω (x) f ω (x)] ≈ τ -1 I + 1 T T i=1 f ωi (x) f ωi (x)

where we use MC integration with T samples for the final step. The predictive covariance matrix is given by:

Cov p * [y] ≈ τ -1 I + 1 T T i=1 f ωi (x) f ωi (x) -E p * [y] E p * [y]

which is the sum of the variance from observation noise and the sample covariance from T stochastic forward passes though the network.

Predictive Log Likelihood We use the Predictive Log Likelihood (PLL) as a measure to estimate the model's uncertainty quality. For a certain test point (y i , x i ), the PLL definition and approximation can be expressed as:

PLL(f ω (x), (y i , x i )) = log p(y i |f ω (x i )) = log f ω (x i , y i )p(ω|D)dω ≈ log f ω (x i , y i )q θ (ω)dω ≈ log T j=1 p(y i |f ωj (x i ))

where ωj represents a sampled set of stochastic parameters from the approximate posterior distrubtion q θ (ω) and we take a MC integration with T samples. For regression, due to the iid Gaussian noise, we can further develop the derivation into the form we use when sampling:

PLL(f ω (x), (y i , x i )) = log T i=1 N (y i |f ωj (x i ), τ -1 I) = logsumexp j=1,...,T - 1 2 ||y i -f ωj (x i )|| 2 + log T - 1 2 log 2π + 1 2 log τ

Note that PLL makes no assumption on the form of the approximate predictive distribution. The measure is based on repeated sampling ωj from q θ (ω), which may be highly multimodal (see section 3.4). we need to derive µ q and σ q , for both ω i = µ u B and ω i = σ u B . Starting with ω i = µ u B :

µ q = ∂ ∂W (j,m) x∈D W (j) x N = x(m) σ q = ∂ ∂W (j,m) x∈D (W (j) x -µ q ) 2 N M 1 2 = 1 2 x∈D (W (j) x -µ q ) 2 N M -1 2 x∈D 2(W (j) x -µ q )(x (m) -x(m) ) N M = 0

Using the result that σ q = 0, one can easily find for ω i = σ u B that µ q = 0 and σ q = 0, nullifying Eq. ( 7). We need only consider the partial derivatives of the KL divergence terms where ω i = µ u B . If we let µ p = 0, Eq. ( 7) reduces to:

∂ ∂W (j,m) KL(q θ (ω i )||p(ω i )) = µ q x(m) σ 2 p = x(m) W (j) x σ 2 p

for each BN unit and connecting weights from the previous layer, W (j,m) . Taking partial derivatives for all KL(q θ (ω i )||p(ω i )) components in the layer:

ω l i j m ∂ ∂W (j,m) KL(q θ (ω i )||p(ω i )) = m x(m) * j m W (j,m) x(m) σ 2 p (8)

We consider ReLU activations, such that for large N , x(m) > 0 ∀m. Note that this does not hold for the first layer (which is possibly normalized), but the effect of including these weights in the L2-regularization would be smaller the deeper the network. We assume most outputs from previous layer's BN units remain normalized through the scale and shift transformation, such that we can approximate x(m) by the average of all input units over test data, x in Eq. ( 8). With J l-1 units in the input layer, setting σ 2 p = J l-1 x 2 2N λ l , such that p(µ u B ) = N (0, J l-1 x 2 2N τ λ l ) would then reconcile Eq. ( 5), for any Gaussian p(σ u B ).

6.6. EXTENDED EXPERIMENTAL RESULTS:

Below, we provide extended results measuring uncertainty quality. In Tables 3 and 4 , we provide tables showing the mean CRPS and PLL values for MCBN and MCDO. These results indicate that MCBN performs on par or better than MCDO across several datasets. In Table 5 we provide the raw PLL and CRPS results for MCBN and MCDO. In Table 6 we provide RMSE results of the MCBN and MCDO networks in comparison with non-stochastic BN and DO networks. These results indicate that the procedure of multiple forward passes in MCBN and MCDO show slight improvements in the accuracy of the network.

In Figure 5 and Figure 6 , we provide a full set of our uncertainty quality visualization plots, where errors in predictions are sorted by estimated uncertainty. 

:

The possibility of using other stochastic regularization techniques is mentioned inGal (2016).

By uncertainty quality, we refer to predictive probability distributions as measured by PLL and CRPS.

achieved by constructing the Evidence Lower Bound, called ELBO, and assuming i.i.d. observation noise; details can be found in the appendix sec 6.1.

while a MC integration using a single sample is a weak approximation, in an iterative optimization for θ several samples will be taken over time.

For a (softmax) classification network, fω(x) is a vector with fω(x, y) = fω(x)(y) , for regression networks with i.i.d. Gaussian noise we have fω(x, y) = N (fω(x), τ -1 I).

6 It further learns an affine transformation for each unit using parameters γ and β, which we omit in favor of brevity:x(j) affine = γ (j) x(j) + β(j) .

As an alternative to using the training set D to sample ωi, we could sample from the implied q θ (ω) as modeled in Appendix 6.3. This would alleviate having to store D for use during prediction. In our experiments we used D to sample ωi however, and leave the evaluation of the modeled q θ (ω) for future research.

Ti can be found analytically for PLL, but must be found numerically for CRPS.

Where we used

units per hidden layer and 2-fold CV.

Although in practice with deep learning, mini-batches are sampled without replacement, stochastic gradient descent samples with replacement in its standard form.

The empirical distributions have been numerically checked to be linearly independent and the joint distribution is close to a bi-variate Gaussian.

Figures and Tables:

Figure 2: Errors in predictions (gray dots) sorted by estimated uncertainty on select datasets. The shaded areas show MCBN's (blue) and MCDO's (red) model uncertainty (light area 95% CI, dark area 50% CI). Gray dots show absolute prediction errors on the test set, and the gray line depicts a running mean of the errors. The dashed line indicates the optimized constant uncertainty. A correlation between estimated uncertainty (shaded area) and mean error (gray) indicates the uncertainty estimates are meaningful for estimating errors. See Appendix for complete results.

Figure 3: Results applying MCBN to Bayesian SegNet(Kendall et al., 2015). In the upper left, a scene from the CamVid driving scenes dataset. In the upper right, the Bayesian estimated segmentation. In the lower left, estimated uncertainty using MCBN for the car class. In the lower right, the estimated uncertainty of MCBN for all 11 classes.

Figure 4: Batch statistics used to train the network are normal. A one-sample Kolmogorov-Smirnov test checks that µ B and σ B come from a standard normal distribution. More examples are available in Appendix 6.7.

Figure 5: Errors in predictions (gray dots) sorted by estimated uncertainty on select datasets. The shaded areas show MCBN's (blue) and MCDO's (red) model uncertainty (light area 95% CI, dark area 50% CI). Gray dots show absolute prediction errors on the test set, and the gray line depicts a running mean of the errors. The dashed line indicates the optimized constant uncertainty. A correlation between estimated uncertainty (shaded area) and mean error (gray) indicates the uncertainty estimates are meaningful for estimating errors.

Figure 6: Errors in predictions (gray dots) sorted by estimated uncertainty on select datasets. The shaded areas show MCBN's (blue) and MCDO's (red) model uncertainty (light area 95% CI, dark area 50% CI). Gray dots show absolute prediction errors on the test set, and the gray line depicts a running mean of the errors. The dashed line indicates the optimized constant uncertainty. A correlation between estimated uncertainty (shaded area) and mean error (gray) indicates the uncertainty estimates are meaningful for estimating errors.

Figure 7: The distribution of means of mini-batches during training of one of our datasets. The distribution closely follows our analytically approximated Gaussian distribution. The data is collected for one unit of each layer and is provided for different epochs and for different batch sizes.

Figure 8: The distribution of standard deviation of mini-batches during training of one of our datasets. The distribution closely follows our analytically approximated Gaussian distribution. The data is collected for one unit of each layer and is provided for different epochs and for different batch sizes.

Properties of the eight regression datasets used to evaluate MCBN. N is the dataset size and Q is the n.o. input features. Only one target feature was used.

Table content: <table><tr><td>Dataset name</td><td>N</td><td colspan="2">Q Target Feature</td></tr><tr><td>Boston Housing</td><td>506</td><td>13</td><td/></tr><tr><td colspan="2">Concrete Compressive Strength 1,030</td><td>8</td><td/></tr><tr><td>Energy Efficiency</td><td>768</td><td>8</td><td>Heating Load</td></tr><tr><td>Kinematics 8nm</td><td>8,192</td><td>8</td><td/></tr><tr><td>Power Plant</td><td>9,568</td><td>4</td><td/></tr><tr><td>Protein Tertiary Structure</td><td colspan="2">45,730 9</td><td/></tr><tr><td>Wine Quality (Red)</td><td>1,599</td><td>11</td><td/></tr><tr><td>Yacht Hydrodynamics</td><td>308</td><td>6</td><td/></tr></table>

The variance is set to a fixed value that optimizes CRPS on validation data. This model reflects our best guess of constant variance on test data -any improvement in uncertainty quality from MCBN would indicate a sensible estimate of uncertainty. We call this model Constant Uncertainty BN (CUBN). Implementing MCDO as a comparative model, we similarly define a baseline for dropout, Constant Uncertainty Dropout (CUDO). The difference in variance modeling between MCBN, CUBN, MCDO and CUDO are visualized in plots of uncertainty bounds on toy data in Figure

Table content: <table/>

Uncertainty quality measured on eight datasets. MCBN, MCDO and MNF are compared over 5 random 80-20 splits of the data with 5 different random seeds each split. Reported values are uncertainty metrics CRPS and PLL normalized to a lower bound of constant variance and upper bound that maximizes the metric. CRPS and PLL are expressed as a percentage, reflecting how close the model is to the upper bound. We check to see if CRPS and PLL significantly exceed the baseline using a one sample t-test (significance level indicated by *'s). Best performer versus their baseline for each dataset and metric is marked by bold. See text for further details.

Table content: <table><tr><td/><td/><td>CRPS</td><td/><td/><td>PLL</td><td/></tr><tr><td>Dataset</td><td>MCBN</td><td>MCDO</td><td>MNF</td><td>MCBN</td><td>MCDO</td><td>MNF</td></tr><tr><td>Boston</td><td>8.50 ****</td><td>3.06 ****</td><td>8.30 ****</td><td>10.49 ****</td><td>5.51 ****</td><td>3.58 ***</td></tr><tr><td>Concrete</td><td>3.91 ****</td><td>0.93 *</td><td colspan="2">6.05 **** -36.36 **</td><td>10.92 ****</td><td>9.71 ****</td></tr><tr><td>Energy</td><td>5.75 ****</td><td>1.37 ns</td><td>3.45 ns</td><td colspan="2">10.89 **** -14.28 *</td><td>2.62 ns</td></tr><tr><td>Kin8nm</td><td>2.85 ****</td><td>1.82 ****</td><td>1.01 *</td><td>1.68 ***</td><td>-0.26 ns</td><td>-0.44 ns</td></tr><tr><td>Power</td><td>0.24 ***</td><td>-0.44 ****</td><td>-0.83 ***</td><td>0.33 **</td><td colspan="2">3.52 **** -1.38 ****</td></tr><tr><td>Protein</td><td>2.66 ****</td><td>0.99 ****</td><td>TBU</td><td>2.56 ****</td><td>6.23 ****</td><td>TBU</td></tr><tr><td>Wine (Red)</td><td>0.26 **</td><td>2.00 ****</td><td>TBU</td><td>0.19 *</td><td>2.91 ****</td><td>TBU</td></tr><tr><td>Yacht</td><td>-56.39 ***</td><td colspan="2">21.42 **** -54.18 ****</td><td colspan="2">45.58 **** -41.54 ns</td><td>71.18 ****</td></tr></table>

Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015. Irvine University of California. UC Irvine Machine Learning Repository, 2017. URL https: //archive.ics.uci.edu/ml/index.html.

Table content: <table><tr><td>Sida I Wang and Christopher D Manning.</td><td>Fast dropout training.</td><td>Proceedings of the</td></tr><tr><td colspan="3">30th International Conference on Machine Learning, 28:118-126, 2013. URL http://</td></tr><tr><td colspan="2">machinelearning.wustl.edu/mlpapers/papers/wang13a.</td><td/></tr></table>

The shaded areas show the model uncertainty and gray dots show absolute prediction errors on the test set. A gray line depicts a running mean of the errors. The dashed line indicates the optimized constant uncertainty. In these plots, we can see a correlation between estimated uncertainty (shaded area) and mean error (gray). This trend indicates that the model uncertainty estimates can recognize samples with larger (or smaller) potential for predictive errors. CRPS measured on eight datasets over 25 random 80-20 splits of the data. Mean values for MCBN and MCDO are reported along with standard error. A significance test was performed to check if CRPS significantly exceeds the baseline. The p-value from a one sample t-test is reported. Marked in bold is the best performing method versus its baseline.

Table content: <table><tr><td>CRPS</td></tr></table>

PLL measured on eight datasets over 25 random 80-20 splits of the data. Mean values for MCBN and MCDO are reported along with standard error. A significance test was performed to check if PLL significantly exceeds the baseline. The p-value from a one sample t-test is reported. Marked in bold is the best performing method versus its baseline.

Table content: <table><tr><td/><td>CRPS</td><td/><td/><td>PLL</td></tr><tr><td>Dataset</td><td>MCBN</td><td>MCDO</td><td>MCBN</td><td>MCDO</td></tr><tr><td>Boston Housing</td><td colspan="4">1.45±0.02 1.41±0.02 -2.38±0.02 -2.35±0.02</td></tr><tr><td>Concrete</td><td colspan="4">2.40±0.04 2.42±0.04 -3.45±0.11 -2.94±0.02</td></tr><tr><td>Energy Efficiency</td><td colspan="4">0.33±0.01 0.26±0.00 -0.94±0.04 -0.80±0.04</td></tr><tr><td>Kinematics 8nm</td><td colspan="3">0.04±0.00 0.04±0.00 1.21±0.01</td><td>1.24±0.00</td></tr><tr><td>Power Plant</td><td colspan="4">2.00±0.01 2.00±0.01 -2.75±0.00 -2.72±0.01</td></tr><tr><td colspan="5">Protein Tertiary Structure 1.95±0.01 1.95±0.00 -2.73±0.00 -2.70±0.00</td></tr><tr><td>Wine Quality (Red)</td><td colspan="4">0.34±0.00 0.33±0.00 -0.95±0.01 -0.89±0.01</td></tr><tr><td>Yacht Hydrodynamics</td><td colspan="4">0.68±0.02 0.32±0.01 -1.39±0.03 -2.57±0.69</td></tr><tr><td colspan="5">Table 5: CRPS and PLL measured on eight datasets over 25 random 80-20 splits of the data. Mean</td></tr><tr><td colspan="5">values and standard errors are reported for MCBN and MCDO. Marked in bold is the best performing</td></tr><tr><td>method for each metric.</td><td/><td/><td/><td/></tr><tr><td colspan="2">6.7 BATCH NORMALIZATION STATISTICS</td><td/><td/><td/></tr></table>

RMSE measured on eight datasets over 25 random 80-20 splits of the data. Mean values and standard errors are reported for MCBN and MCDO as well as conventional non-Bayesian models BN and DO. Marked in bold is the best performing method overall.of an actual training of Yacht dataset for one unit in the first hidden layer and the second hidden layer. Data is provided for different epochs and for different batch sizes.

Table content: <table/>

Peer review (iclr2018):
 The authors show how the regularization procedure called batch normalization, currently being used by most deep learning systems, can be understood as performing approximate Bayesian inference. The authors compare this approach to Monte Carlo dropout (another regularization technique which can also be considered to perform approximate Bayesian inference). The experiments performed show that the Bayesian view of batch normalization performs similarly as MC dropout in terms of the estimates of uncertainty that it produces. Quality: I found the quality to be low in some aspects. First, the description of what is the prior used by batch normalization in section 3.3 is unsatisfactory. The authors basically refer to Appendix 6.4 for the case in which the weight decay penalty is not zero. The details in that Appendix are almost none, they just say "it is thus possible to derive the prior...". The results in Table 2 are a bit confusing. The authors should highlight in bold face the results of the best performing method. The authors indicate that they do not need to compare to variational methods because Gal and Ghahramani 2015 compare already to those methods. However, Gal and Ghahramani*s code used Bayesian optimization methods to tune hyper-parameters and this code contains a bug that optimizes hyper-parameters by maximizing performance on the test data. In particular for hyperparameter selection, they average performance across (subsets of) 5 of the training sets from the 20x train/test split, and then using the tau which got the best average performance for all of 20x train/test splits to evaluate performance: https://github.com/yaringal/DropoutUncertaintyExps/blob/master/bostonHousing/net/experiment_BO.py#L54 Therefore, the claim that "Since we have established that MCBN performs on par with MCDO, by proxy we might conclude that MCBN outperforms those VI methods as well." is not valid. At the beginning of section 4.3 the authors indicate that they follow in their experiments the setup of Gal and Ghahramani (2015). However, Gal and Ghahramani (2015) actually follow Hernández-Lobato and Adams, 2015 so the correct reference should be the latter one. Clarity: The paper is clearly written and easy to follow and understand. I found confusing how to use the proposed method to obtain estimates of uncertainty for a particular test data point x_star. The paragraph just above section 4 says that the authors sample a batch of training data for this, but assume that the test point x_star has to be included in this batch. How is this actually done in practice? Originality: The proposed contribution is original. This is the first time that a Bayesian interpretation has been given to the batch normalization regularization proposal. Significance: The paper*s contributions are significant. Batch normalization is a very popular regularization technique and showing that it can be used to obtain estimates of uncertainty is relevant and significant. Many existing deep learning systems can use this to produce estimates of uncertainty in their predictions.