Title: PAC-BAYESIAN NEURAL NETWORK BOUNDS

Abstract: Bayesian neural networks, which both use the negative log-likelihood loss function and average their predictions using a learned posterior over the parameters, have been used successfully across many scientific fields, partly due to their ability to 'effortlessly' extract desired representations from many large-scale datasets. However, generalization bounds for this setting is still missing. In this paper, we present a new PAC-Bayesian generalization bound for the negative log-likelihood loss which utilizes the Herbst Argument for the log-Sobolev inequality to bound the moment generating function of the learners risk. We explore the generalization and calibration properties of the learned posterior on several image classification benchmarks, showing that the proposed approach provides better generalization and uncertainty estimates.

1. INTRODUCTION:

Deep neural networks are ubiquitous across disciplines and often achieve state of the art results (e.g., Krizhevsky et al. (2012) ; Simonyan & Zisserman (2014) ; He et al. (2016) ). Albeit neural networks are able to encode highly complex input-output relations, in practice, they do not tend to overfit (Zhang et al., 2016) . This tendency to not overfit has been investigated in numerous works on generalization bounds (Langford & Shawe-Taylor, 2002; Langford & Caruana, 2002; Bartlett et al., 2017a; 2019; McAllester, 2003; Germain et al., 2016; Dziugaite & Roy, 2017) . Indeed, many generalization bounds apply to neural networks. However, most of these bounds assume that the loss function is bounded (Bartlett et al., 2017a; Neyshabur et al., 2017; Dziugaite & Roy, 2017) . Unfortunately, this assumption excludes the popular negative log-likelihood (NLL) loss, which is instrumental to Bayesian neural networks that have been used extensively to calibrate model performance and provide uncertainty measures to the model prediction.

In this work we introduce a new PAC-Bayesian generalization bound for NLL loss of deep neural networks. Our work utilizes the Herbst argument for the logarithmic-Sobolev inequality (Ledoux, 1999) in order to bound the moment-generating function of the model risk. Broadly, our PAC-Bayesian bound is comprised of two terms: The first term is dominated by the norm of the gradients with respect to the input and it describes the expressivity of the model over the prior distribution. The second term is the KL-divergence between the learned posterior and the prior, and it measures the complexity of the learning process. In contrast, bounds for linear models or bounded loss functions lack the term that corresponds to the expressivity of the model over the prior distribution and therefore are the same when applied to shallow and deep models.

We empirically show that our PAC-Bayesian bound is tightest when we learn the mean and variance of each parameter separately, as suggested by Blundell et al. (2015) in the context of Bayesian neural networks (BNNs) . We also show that the proposed bound holds different insights regarding model architecture, optimization and prior distribution selection. We demonstrate that such optimization minimizes the gap between risk and the empirical risk compared to the standard Bernoulli dropout and other Bayesian inference approximation while being consistent with the theoretical findings. Additionally, we explore in-distribution and out-of-distribution examples to show that such optimization produces better uncertainty estimates than the baseline.

Generalization bounds for neural networks were explored in various settings. VC-theory provides both upper bounds and lower bounds to the network VC-dimension, which are linear in the number of network parameters (Bartlett et al., 2017b; 2019) . While VC-theory asserts that such a model should overfit as it can learn any random labeling (e.g., Zhang et al. (2016) ), surprisingly, neural networks generally do not overfit. Rademacher complexity allows to apply data dependent bounds to neural networks (Bartlett & Mendelson, 2002) . However, they rely on bounds of the loss function (Wan et al., 2013; Gao & Zhou, 2016) . Rademacher complexity bounds may also take into account the Lipschitz constant of the network. While this presents an improvement when its gradient-norms are bounded, these results also require the loss to be bounded. Stability bounds may be applied to unbounded loss functions and in particular to the negative log-likelihood (NLL) loss (Bousquet & Elisseeff, 2002; Rakhlin et al., 2005; Shalev-Shwartz et al., 2009; Hardt et al., 2015; Zhang et al., 2016) . However, stability bounds for convex loss functions, e.g., for logistic regression, do not apply to neural networks since they require the NLL loss to be a convex function of the parameters. Alternatively, Hardt et al. (2015) estimate the stability of stochastic gradient descent dynamics, which strongly relies on early stopping. This approach results in weaker bounds for the non-convex setting.

PAC-Bayesian bounds were recently applied to neural networks (Langford & Caruana, 2002; McAllester, 2013; Dziugaite & Roy, 2017; Neyshabur et al., 2017) . In contrast to our work, those related works all consider bounded loss functions. PAC-Bayesian bounds for the NLL loss in the online setting were put forward by Banerjee (2006) . The online setting does not to consider the whole sample space and therefore is simpler to analyze in the Bayesian setting. An excellent survey on PAC-Bayesian bounds was provided by Germain et al. (2016) . Alquier et al. (2016) introduced PAC-Bayesian bounds for linear classifiers with the hinge-loss. Germain et al. (2016) is closer to our setting and considers PAC-Bayesian bounds for the quadratic loss function, while proving that the quadratic loss function has exponential decay (sub-gamma) when the data is sampled from a Gaussian distribution. This result is a PAC-Bayesian generalization bound for the NLL loss of a regressor which assumes Gaussian model over a continuous label space y. Our work differs from this work in two important aspects: (1) our work does not consider linear classifiers and apply to any deep neural networks; (2) our bound considers the NLL loss for classification tasks, for which the label space y is discrete. PAC-Bayesian bounds for the NLL loss function are intimately related to learning Bayesian inference (Germain et al., 2016) . Recently many works applied various posteriors in Bayesian neural networks. Gal & Ghahramani (2015) ; Gal (2016) introduce a Bayesian inference approximation using Monte Carlo (MC) dropout, which approximates a Gaussian posterior using Bernoulli dropout. Srivastava et al. (2014) introduced Gaussian dropout which effectively creates a Gaussian posterior that couples between the mean and the variance of the learned parameters. Kingma et al. (2015) explored the relation of this posterior to log-uniform priors, while Blundell et al. (2015) suggests to take a full Bayesian perspective and learn separately the mean and the variance of each parameter. Our work uses the bridge between PAC-Bayesian bounds and Bayesian inference, as described by Germain et al. (2016) , to find the optimal prior parameters in PAC-Bayesian setting and apply it in the Bayesian setting.

Most of the literature regarding Bayesian modeling involves around a two-step formalism (Bernardo & Smith, 2009) : (1) a prior is specified for the parameters of the deep net; (2) given the training data, the posterior distribution over the parameters is computed and used to quantify predictive uncertainty. Since exact Bayesian inference is computationally intractable for neural networks, approximations are used, including MacKay (1992); Hernández-Lobato & Adams (2015); Hasenclever et al. (2017); Balan et al. (2015) ; Springenberg et al. (2016) . In this study we follow this two-step formalism, particularly we follow a similar approach to Blundell et al. (2015) in which we learn the mean and standard deviation for each parameter of the model using variational Bayesian practice. Our experimental validation emphasizes the importance of learning both the mean and the variance.

3. BACKGROUND:

Generalization bounds provide statistical guarantees on learning algorithms. They measure how the learned parameters w perform on test data given their performance on the training data S = {(x 1 , y 1 ), . . . , (x m , y m )}, where x i is the data instance and y i is its corresponding label. The performance of the learning algorithm is measured by a loss function (w, x, y). The risk of a learner is its average loss, when the data instance and its label are sampled from their true but unknown distribution D. We denote the risk by L D (w) = E (x,y)∼D (w, x, y). The empirical risk is the average training set loss L S (w) = 1 m m i=1 (w, x i , y i ). PAC-Bayesian theory bounds the risk of a learner E w∼q L D (w) when the parameters are averaged over the learned posterior distribution q. The parameters of the posterior distribution are learned from the training data S. In our work we focus on the following PAC-Bayesian bound: Theorem 1 (Alquier et al. (2016) ). Let KL(q||p) = q(w) log(q(w)/p(w))dw be the KLdivergence between two probability density functions p, q. For any λ > 0 and for any δ ∈ (0, 1] and for any prior distribution p, with probability at least 1 -δ over the draw of the training set S, the following holds simultaneously for any posterior distribution q:

EQUATION

PAC-Bayesian theory is intimately connected to Bayesian inference when considering the negative log-likelihood loss function (w, x, y) = -log p(y|x, w) and λ = m. Germain et al. (2016) proved that the optimal posterior in this setting is q(w) = p(w|S). Bayesian inference considers the posterior p(y|x, S) = p(w|S)p(y|x, w)dw, at test time for a data instance x, which corresponds to the risk of the optimal posterior. Unfortunately, the optimal posterior is rarely available, and PAC-Bayes relies on the approximated posterior q.

Coincidently, the approximated posterior and its KL-divergence from the prior distribution are instrumental to the evidence lower bound (ELBO), which is extensively used in Bayesian neural networks (BNNs) to bound the log-likelihood m i=1 log p(y i |x i ):

EQUATION

While the right hand side of a PAC-Bayesian bound, with the negative log-likelihood loss and λ = m, is identical to the right hand side of the ELBO bound in term of learning, they serve different purposes. One is used for bounding the risk while the other is used for bounding the marginal loglikelihood. Nevertheless, the same algorithms can be used to optimize BNNs and PAC-Bayesian intuitions and components can influence the practice of Bayesian neural networks.

4. PAC-BAYESIAN BOUNDS FOR THE NEGATIVE LOG-LIKELIHOOD LOSS:

It is challenging to derive a PAC-Bayesian bound for the negative log-likelihood (NLL) loss as it requires a bound on the log-partition function log E w∼p,S∼D m [e λ(L D (w)-L S (w)) ]. In cases where the loss function is uniformly bounded by a constant, e.g., the zero-one loss, the log-partition function is bounded as well. Unfortunately, the NLL loss is unbounded, even when y is discrete. For instance, consider fully connected case, where the input vector of the (k)-th layer is a function of the parameters of all previous layers, i.e., x k (W 0 , . . . , W k-1 ). The entries of x k are computed from the response of its preceding layer, i.e., W k-1 x k , followed by a transfer function σ(•), i.e.,

x k+1 = σ(W k-1 x k ). Then, since the NLL is define as -log(p(y|x, w)) = -(W k x k ) y +log( ŷ e (W k x k ) ŷ ),

if the rows in W k consist of the vector rx k then the NLL loss increases with r, and is unbounded when r → ∞.

Our main theorem shows that for smooth loss functions, the log-partition function is bounded by the expansion of the loss function, i.e., the norm of its gradient with respect to the data x. This property is appealing since these gradients often decay rapidly for deep neural networks, as we demonstrate in our experimental evaluation. Consequently deep networks enjoy tighter generalization bounds than shallow networks. Our proof technique follows the Herbst Argument for bounding the log-partition function using the Log-Sobolev inequality for Gaussian distributions (Ledoux, 2001) . Theorem 2. Assume (x, y) ∼ D and x given y follows the Gaussian distribution. Let (w, x, y) be a smooth loss function (e.g., the negative log-likelihood loss). For any δ ∈ (0, 1] and for any real number λ > 0, with probability at least 1 -δ over the draw of the training set S the following holds simultaneously for any posterior probability density function:

E w∼q [L D (w)] ≤ E w∼q [L S (w)]+ log E w∼p e 2λ 2 m E (x,y)∼D ∇x (w,x,y) 2 1 0 e -α (w,x,y) E (x, ŷ)∼D e α(-(w,x, ŷ)) dα + KL(q||p) + log(1/δ) λ . (3)

The Gaussian assumption for the data generating distribution D can be relaxed to any log-concave distribution, using Gentil (2005) , Corollary 2.5. We use the Gaussian assumption to avoid notational overhead.

Broadly, the proposed bound is comprised of two terms: The first term is the log-partition function which is dominated by the norm of the gradients with respect to the input, namely (w,x, ŷ)) dα , and it describes the expressivity of the model over the prior distribution. The second term is the KL-divergence between the learned posterior and the prior, and it measures the complexity of the learning process.

2λ 2 m E (x,y)∼D ∇ x (w, x, y) 2 1 0 e -α (w,x,y) E (x, ŷ)∼D e α(-

The proof starts with Eq. ( 1) and uses the Herbst Argument and the Log-Sobolev inequality to bound the moment-generating function E S∼D m [e λ(L D (w)-L S (w)) ]. Specifically, the proof consists of three steps. First we use the statistical independence of the training samples to decompose the moment generating function

E S∼D m [e λ(L D (w)-L S (w)) ] = e λE (x,y)∼D (w,x,y) E (x,ŷ)∼D [e λ m (-(w,x,ŷ)) ] m .

(4)

Then we use the Herbst argument to bound the function

M ( λ m ) E (x,ŷ)∼D [e λ m (-(w,x,ŷ))

] and obtain the following bound:

M λ m = e log M (0)+( λ m ) 2 1 0 αM (α)-M (α) log M (α) α 2 M (α)

dα .

(5)

Finally we use the log-Sobolev inequality for Guassian distributions,

EQUATION

The above theorem can be extended to settings for which x is sampled from any log-concave distribution, e.g., the Laplace distribution. The log-concave setting modifies the gradient norm and the log-Sobolev constant 2 in Eq. ( 6) that corresponds to Gaussian distributions, cf. Gentil (2005) . We avoid this generalization to simplify our mathematical derivations.

A detailed description of the proof can be found on Section 8.1 in the Appendix.

5. APPLICATION TO LOGISTIC REGRESSION:

The bound in Theorem 2 is favorable when applied to deep networks since their gradients w.r.t. data often decay rapidly. Nevertheless we can also apply our technique to shallow nets trained with NLL loss. We obtain PAC-Bayesian bounds for multi-class logistic regression.

The NLL loss for multiclass logistic regression takes the form:

-log p(y|x, w) = -(W x) y + log( ŷ e (W x) ŷ )

, where x ∈ R d is the data instance, y ∈ {1, . . . , k} are the possible labels, and W ∈ R k×d is the matrix of parameters. The bound in Theorem 2 takes the form:

Corollary 1. Assume (x, y) ∼ D and x ∈ R d given y follows the Gaussian distribution. Let (w, x, y) = -log p(y|x, w) be the negative log-likelihood loss for k-class logistic regression. For any δ ∈ (0, 1], for any λ > 0 and for any prior density function with variance σ 2 p ≤ m/16λ 2 , with probability at least 1 -δ over the draw of the training set S the following holds simultaneously for any posterior probability density function:

EQUATION

Full proof can be found on Section 8.2 in the Appendix, while we sketch the main steps of the proof below. The above corollary shows that PAC-Bayesian bound for classification using the NLL loss can achieve rate of λ = m. This result augments the PAC-Bayesian for regression using the NLL loss for regression, i.e., the square loss, of Germain et al. (2016) .

The PAC-Bayesian bound for logistic regression is derived by applying Theorem 2. We begin by realizing the gradient of log p(y|x, w) with respect to x. We denote by w y the y-th row of the parameter matrix W . Thus ∇ x log p(y|w, x) = ŷ p(ŷ|x, w)(w y -w ŷ ), and the gradient norm is upper bounded as follows: ∇ x log p(y|w, x) 2 ≤ 2 y w y 2 . Plugging this result into Eq. ( 18) we obtain the following bound:

EQUATION

Finally, whenever λσ p ≤ m/8 we derive the bound

E w∼p e 4λ 2 m ŷ w ŷ 2 ≤ m m -8λ 2 σ 2 p kd (9)

A detailed description of the proof can be found on Section 8.2 in the Appendix.

6. EXPERIMENTS:

In this section we study the derived bound empirically. We start with an ablation study of the proposed bound using classification and regression models. Next, we present our results for multiclass classification tasks using different datasets and different architectures. We conclude the section, with an analysis of the models' uncertainty estimates using for in-distribution examples and outof-distribution examples. All suggested models follows the a Bayesian Neural Networks (BNN) perspective, in which we learn the mean and standard deviation for each learnable parameter in the network where we define N (0, σ 2 p I) to be the prior over weights.

6.1 ABLATION Effect of σ p . We start by exploring the effect of σ p on the models' performance and the proposed generalization bound. For that, we trained several models using σ p ∈ {0.05, 0.1, 0.2, 0.3} using the MNIST (LeCun & Cortes, 2010) and Fashioin- MNIST Xiao et al. (2017) datasets. All results were obtained using fully connected layers with ReLU as non-linear activation function. We optimized the NLL loss function using Stochastic Gradient Descent (SGD) for 50 epochs with a learning rate of 0.01 and momentum of 0.9. For each model we compute the average train and test loss and accuracy together with the absolute difference between the training loss and the test loss, denoted as Generalization Loss. Moreover, we compute the generalization bound as stated in Eq. ( 18) for all settings. Results are summarized in Table 1 .

Although σ p = 0.2 reaches slightly better generalization bound on MNIST dataset, σ p = 0.1 performs better over all calculated metrics, i.e., average loss and accuracy, both on MNIST and Fashion-MNIST. Notice, for Fashion-MNIST we observed slightly better generalization gap while using σ p = 0.05, however, its loss and accuracy are worse comparing to σ p = 0.1.

Effect of λ. Recall, we bound the moment generating function using the norm of the functions' gradient with respect to the data x (Eq. ( 18)). To construct tighter generalization bounds, we would like to set λ → m. However, in Eq. ( 18) λ appears in both numerator and denominator. It is hence not clear whether the bound will converge, which depends on the model architecture, which is represented by the norm of its gradient. In other words, models with lower gradient norm could benefit from larger values of λ, hence tighter generalization bounds.

To further explore this property we trained five different models with different number of layers (1-5). We look into both classification models while optimizing the NLL loss function, and regression tasks while optimizing the Mean Squared Error (MSE) loss function. For classification we used MNIST and Fashion-MNIST datasets, while for regression we use the Boston Housing dataset (for the regression models, results were obtained using 5-fold cross validation). Except for the linear models, we force all models to have roughly the same numbers of parameters (∼80K for MNIST, ∼800K for Fashion-MNIST, ∼1500 for regression). For all models we set ReLU as non-linear activation functions. We optimize all models for 50 epochs using SGD with learning rate of 0.01 and momentum of 0.9. Based on results of the prior paragraph, in all reported settings we set σ p = 0.1.

Results are reported in Table 2 . It can be seen that deeper models produce tighter generalization bounds on all three datasets. When considering model performance on down-stream classification task we notice that in general, models with better generalization bounds perform slightly better in terms of loss and accuracy. One possible explanation is that deeper models have smaller gradients w.r.t. the input. To validate that we further computed the average squared gradient norm w.r.t. the input as a function of the model depth, for both MNIST and Fashion-MNIST datasets. It can be seen from Figure 1a that indeed the gradients decay rapidly as we add more layers to the network.

Next, we present in Figure 1b the generalization bound as a function of λ for MNIST models. We explored λ ∈ [ √ m, m] and stopped the plot once the bound can no longer be computed. Experiments using Fashion-MNIST produce similar plot and can be found on Section 8.7 in the Appendix.

Weights visualization. Since we consider Bayesian Neural Networks (BNNs) and optimize the KLdivergence between the prior and the posterior over the weights, we can visualize the average mean and standard deviation (STD) of the posterior as a function of the model depth. Figure 2a presents this for MNIST and Fashion-MNIST models using four and five depth levels. As expected, we can see that the average mean over the weights is zero for all layers while weights STD approaches 0.1. For the MNIST models (Figure 2a top row), we observed the standard deviation are ∼0.7 and not 0.1. We suspect this behaviour is due to fast optimization, hence the models do not have much signal to push the model towards the prior distribution. Notice, in all settings the average STD of the model weights decreases on the last layer. We observed a similar behavior also for the other models.

6.2 CLASSIFICATION Next, we compare BNN models against two commonly used baselines. The first baseline is a softmax model using the same architecture as the BNN while adding dropout layers. The second base- line is a Bayesian approximation using Monte Carlo Dropout (Gal & Ghahramani, 2015) , denoted as MC-Dropout, using different dropout rates and weight decay value of 1e-5.

To evaluate these approaches we conducted multi-class classification experiments using three classification benchmarks: MNIST, Fashion-MNIST, and CIFAR-10 ( Krizhevsky & Hinton, 2009) . We report train set and test set loss and accuracy, together with their generalization gaps (e.g., the difference between the test and training loss and accuracy). Notice, as oppose to Dziugaite & Roy (2017) our results are reported for multi-class classification and not for binary classification. For completeness, we report binary classification results on Section 8.4 in the Appendix. The premise beyond these type of experiments is to preset the benefit of learning the mean and STD separately for each of the models' parameters.

Results are reported in Table 7 . For BNN and MC-Dropout models, we sample 20 times from the posterior distribution and average their outputs to produce the model output. We also sampled more times, however, we did not see any significant differences. We observe that BNN models achieve comparable results to both baselines but with lower loss and accuracy generalization gaps. Throughout the experiments we use dropout value of 0.3 for Softmax and MC-Dropout models and σ p = 0.1 for BNN models. We chose these values after grid search over different dropout values for all baseline models. A detailed description of all implementation details together with results for more dropout rates can be found on Sections 8.4, 8.5, and 8.3 in the Appendix.

6.3. UNCERTAINTY ESTIMATES:

Lastly, we evaluated the uncertainty estimates of BNN models against softmax models and MC-Dropout models. We experimented with both in-distribution and out-of-distribution examples. The purpose of the following experiments is to demonstrate that following the Bayesian approach together with the carefully picked prior can lead to better uncertainty estimates. We apply a model trained using dataset A to OOD examples from dataset B. We trained models on MNIST, Fashion-MNIST and CIFAR-10 and assess prediction confidence using OOD examples from MNIST, Fashion-MNIST, NotMNIST (Cohen et al., 2017) , and SVHN (Netzer et al., 2011) .

Results are summarized in Table 8 . More OOD experiments using different dropout and prior rates can be found on Sections 8.3, 8.6 in the Appendix.

All models performed at the chance level (∼ 10% for 10 classes) for both OOD train and test sets.

When considering the loss, we observe significantly higher values for the softmax and MC-Dropout models. These two findings imply that the softmax and MC-Dropout models are overly confident and tend to output a high probability for the max label. Hence, we measure the average entropy for all models. We expect BNNs to have higher entropy, due to the fact that it produces better uncertainty estimates, i.e., its' predictions for OOD samples are closer to a uniform distribution. Indeed, results reported in Table 8 confirm this intuition.

7. DISCUSSION AND FUTURE WORK:

In the following study we present a new PAC-Bayesian generalization bound for learning a deep net using the NLL loss function. The proof relies on bounding the log-partition function using the squared norm of the gradients with respect to the input. Experimental validation shows that the resulting bound provides insight for better model optimization and prior distribution search. We demonstrate that learning the mean and STD for all parameters together with optimize prior over the parameters leads to better uncertainty estimates over the baselines and makes it harder to overfit.

8.1. PAC-BAYESIAN BOUNDS FOR THE NEGATIVE LOG-LIKELIHOOD LOSS -PROOF:

Proof. We begin by using the statistical independence of the training samples to decompose the following function:

EQUATION

= e λL D (w)

EQUATION

Next we represent the moment generating function

M ( λ m ) E (x,ŷ)∼D [e λ m (-(w,x,ŷ)) ] us- ing K(α) 1 α log M (α). The fundamental theorem of calculus asserts K( λ m ) -K(0) = λ/m 0 K (α)dα = λ/m 1 0 K (α)dα

, where the last equality follows from a change of integration variable and the integral limits. K (α) refers to the derivative at α. We then compute K (α) and K(0):

EQUATION

EQUATION

Concluding the Herbst argument we obtain the following equality:

EQUATION

Combining Eq. ( 12) with Eq. ( 15) we derive:

EQUATION

)

Finally we apply the log-Sobolev inequality for Gaussian distributions (cf. Ledoux (2001) , Chapter 2), as described in Eq. ( 6). To complete the proof we combine Eq. ( 6) with Eq. ( 17) to obtain:

EQUATION

8.2. APPLICATION TO LOGISTIC REGRESSION -PROOF:

Proof. To apply Theorem 2 we start by realizing the gradient of log p(y|x, w) with respect to x. We denote by w y the y-th row of the parameter matrix W . Thus ∇ x log p(y|w, x) = ŷ p(ŷ|x, w)(w y -w ŷ ). Using the convexity of the norm function we upper bound the gradient norm:

∇ x log p(y|w, x) 2 ≤ ŷ p(ŷ|x, w) w y -w ŷ 2 ≤ w y 2 + ŷ w ŷ 2 ≤ 2 ŷ w ŷ 2 (20)

Next we use the fact that the gradient norm upper bound is independent of x to simplify the moment generating function bound in Theorem 2. Since (w, x, y) = -log p(ŷ|x, w), we use the bound in Eq. ( 20):

E (x,y)∼D ∇ x (w, x, y) 2 1 0 e -α (w,x,y) E (x,ŷ)∼D e α(-(w,x,ŷ)) dα ≤ 2 ŷ w ŷ 2 1 0 E (x,y

)∼D e -α (w,x,y) E (x,ŷ)∼D e α(-(w,x,ŷ)) dα (21)

Thus we are able to simplify Theorem 2 as follows

EQUATION

Finally, we recall that p is the prior density function N (0, σ 2 p ). Since the parameters are statistically independent, this expectation decomposes to its kd parameters:

EQUATION

And the result follows from the fact

E v∼N (0,σ 2 p ) e 4λ 2 m v 2 = m m-8λ 2 σ 2 p : ∞ -∞ 1 √ 2πσ p e -v 2 2σ 2 p e 4λ 2 m v 2 dv = ∞ -∞ 1 √ 2πσ p e -v 2 1 2σ 2 p -4λ 2 m dv = ρ σ p ∞ -∞ 1 √ 2πρ e -v 2 2ρ 2 (24) for ρ 2 = σ 2 p m m-8λ 2 σ 2 p . Whenever λσ p ≤ m/8 then m m-8λ 2 σ 2 p ≤ √

2 and the result follows.

8.3. ARCHITECTURES:

The architechtures described in this sub-section are used for the multi-class, binary, and uncertainty estimates experiments. We use multilayer perceptrons for the MNIST dataset, while we use convolutional neural networks (CNNs) for both Fashion-MNIST and CIFAR-10. A detailed description of the architectures is available in Table 5 . We optimize the NLL loss function using SGD with a learning rate of 0.01 and a momentum value of 0.9 in all settings. We use mini-batches of size 128 and did not use any learning rate scheduling. For the MC-Dropout models we experienced with different weight decay values, however found that 1e-5 provides the best validation loss, hence choose this value. We follow the same setting in which we use the MNIST dataset, where we group digits [0, 1, 2, 3, 4] into label zero, and labels [5, 6, 7, 8, 9] into label one. All experiments in this subsection were conducted using multilayer perceptrons with one hidden layer consisting of 300 hidden neurons. We use the Rectified Linear Unit (ReLU) as our activation function (Glorot et al., 2011) . We optimize the negative log-likelihood loss function using stochastic gradient descent (SGD) with a learning rate of 0.1 and a momentum value of 0.9. We did not use any learning rate scheduling. SGD is run in mini-batches of size 128. Each model was trained for 20 epochs.

We compared BNN to softmax models with dropout (Srivastava et al., 2014) rates chosen from the set {0.0, 0.3, 0.5}. Hereby, a dropout with a rate of 0.0 means no dropout at all. In addition to the training set and test set loss and accuracy, we measure the generalization loss, while setting L D (w) to be the average test set loss. In the same manner, we measure the generalization accuracy, while using the zero-one loss instead of the negative log-likelihood loss. Table 6 summarizes the results. All models achieve comparable accuracy levels, however the softmax models suffer from larger generalization errors both in terms of loss and accuracy. Notice, as expected, using higher Bernoulli-dropout rates mitigates the generalization gap. 

:



Figures and Tables:

Figure 1: Analysis of the proposed bound as a function network depth. In (a) we show the squared gradient norm as a function of the layers of the model. In (b) we report the the generalization bound as a function of λ for different deep net depth levels using the MNIST dataset.

Figure 2: In (a): average mean and STD of model weights as a function of depth level. Top row is for MNIST, bottom row is for Fashion-MNIST. In (b): ECE and MCE metrics for Softmax, MC-Dropout and BNN models using MNIST, Fashion-MNIST, and CIFAR10 (lower is better).

Figure3: Analysis of the proposed bound as a function network depth. We report the the generalization bound as a function of λ for different deep net depth levels using the Fashion-MNIST dataset.

Different σ p values experiments. All models composed of five fully connected layers with ReLU activation functions.

Table content: <table><tr><td/><td>Model</td><td colspan="6">Train loss Test loss Train acc. Test acc. Gen. loss Bound</td></tr><tr><td>MNIST</td><td>σ p = 0.05 σ p = 0.1 σ p = 0.2 σ p = 0.3</td><td>0.07 0.02 0.03 0.09</td><td>0.08 0.06 0.08 0.11</td><td>98.48 99.53 99.24 97.2</td><td>97.73 98.27 97.9 96.66</td><td>0.02 0.03 0.05 0.02</td><td>0.2 0.08 0.08 0.67</td></tr><tr><td>Fashion</td><td>σ p = 0.05 σ p = 0.1 σ p = 0.2 σ p = 0.3</td><td>0.24 0.11 0.26 0.37</td><td>0.32 0.25 0.33 0.42</td><td>92.03 93.28 90.12 86.36</td><td>88.55 89.50 87.87 84.26</td><td>0.09 0.1 0.07 0.05</td><td>0.2 0.13 0.86 21.73</td></tr></table>

Different depth level experiments. For fair comparison all models have roughly the same number of parameters (∼80K for MNIST, ∼800K for Fashion-MNIST, ∼1500 for regression). All models composed of fully connected layers with ReLU activation function.

Table content: <table><tr><td/><td>Model</td><td colspan="6">Train loss Test loss Train acc. Test acc. Gen. loss Bound</td></tr><tr><td/><td>One layer</td><td>0.253</td><td>0.267</td><td>92.89</td><td>92.49</td><td>0.015</td><td>3.69</td></tr><tr><td>MNIST</td><td>Two layers Three layers Four layers</td><td>0.045 0.029 0.024</td><td>0.073 0.063 0.065</td><td>98.89 99.41 99.49</td><td>97.82 98.15 98.14</td><td>0.028 0.033 0.041</td><td>2.28 0.82 0.13</td></tr><tr><td/><td>Five layers</td><td>0.023</td><td>0.058</td><td>99.52</td><td>98.27</td><td>0.035</td><td>0.08</td></tr><tr><td/><td>One layer</td><td>0.38</td><td>0.46</td><td>86.86</td><td>84.2</td><td>0.07</td><td>5.68</td></tr><tr><td>Fashion</td><td>Two layers Three layers Four layers</td><td>0.21 0.21 0.21</td><td>0.31 0.31 0.30</td><td>92.52 92.34 92.49</td><td>88.9 88.7 88.8</td><td>0.1 0.1 0.09</td><td>4.9 1.08 0.22</td></tr><tr><td/><td>Five layers</td><td>0.19</td><td>0.29</td><td>93.3</td><td>89.6</td><td>0.1</td><td>0.13</td></tr><tr><td/><td>One layer</td><td>45.7</td><td>106.5</td><td>-</td><td>-</td><td>60.8</td><td>378.7</td></tr><tr><td>Boston</td><td>Two layer Three layers Four layers</td><td>8.6 8.2 10.4</td><td>15.7 12.9 14.1</td><td>---</td><td>---</td><td>7.1 4.7 3.7</td><td>286.3 105.3 45.9</td></tr><tr><td/><td>Five layers</td><td>12.0</td><td>12.1</td><td>-</td><td>-</td><td>0.1</td><td>31.9</td></tr></table>

Results for multi-class image classification benchmarks. We report both loss and accuracy metrics for the training set and test set, together with the generalization loss and accuracy. Train loss Train acc. Test loss Test acc. Gen. loss Gen. acc.

Table content: <table><tr><td/><td>Model</td><td/><td/><td/><td/><td/><td/></tr><tr><td>MNIST</td><td>Softmax MC-Dropout BNN</td><td>0.003 0.003 0.034</td><td>1.0 1.0 0.995</td><td>0.066 0.066 0.064</td><td>0.983 0.983 0.982</td><td>0.06 0.06 0.03</td><td>0.016 0.016 0.012</td></tr><tr><td>Fashion</td><td>Softmax MC-Dropout BNN</td><td>0.143 0.145 0.251</td><td>0.947 0.946 0.905</td><td>0.288 0.288 0.321</td><td>0.906 0.905 0.881</td><td>0.14 0.14 0.07</td><td>0.041 0.041 0.024</td></tr><tr><td>CIFAR-10</td><td>Softmax MC-Dropout BNN</td><td>0.492 0.493 0.453</td><td>0.831 0.829 0.844</td><td>0.541 0.544 0.479</td><td>0.817 0.818 0.836</td><td>0.05 0.05 0.02</td><td>0.013 0.011 0.007</td></tr></table>

Results for the out-of-distribution experiments. We report both loss and accuracy measures for the training set and test set, together with entropy values. We explore training on MNIST (MN), Fashion-MNIST (FMN), and CIFAR-10. Testing was done on MNIST (MN), Fashion-MNIST (FMN), NotMNIST, and SVHN. Reported losses and accuracies are computed using the out-ofdistribution dataset.

Table content: <table><tr><td>Model</td><td>In-dis.</td><td>Out-dis.</td><td colspan="6">Train loss Train acc. Test loss Test acc. Train ent. Test ent.</td></tr><tr><td>Softmax</td><td>FMN</td><td>MN</td><td>8.33</td><td>0.113</td><td>8.21</td><td>0.114</td><td>0.42</td><td>0.41</td></tr><tr><td>MC-Dropout</td><td>FMN</td><td>MN</td><td>7.84</td><td>0.112</td><td>7.74</td><td>0.112</td><td>0.45</td><td>0.45</td></tr><tr><td>BNN</td><td>FMN</td><td>MN</td><td>4.64</td><td>0.105</td><td>4.57</td><td>0.108</td><td>0.78</td><td>0.74</td></tr><tr><td>Softmax</td><td>FMN</td><td>NotMNIST</td><td>8.64</td><td>0.155</td><td>7.78</td><td>0.152</td><td>0.43</td><td>0.41</td></tr><tr><td>MC-Dropout</td><td>FMN</td><td>NotMNIST</td><td>8.39</td><td>0.123</td><td>7.82</td><td>0.087</td><td>0.54</td><td>0.55</td></tr><tr><td>BNN</td><td>FMN</td><td>NotMNIST</td><td>4.70</td><td>0.085</td><td>4.59</td><td>0.1</td><td>0.89</td><td>0.92</td></tr><tr><td>Softmax</td><td>CIFAR-10</td><td>SVHN</td><td>3.55</td><td>0.112</td><td>3.60</td><td>0.106</td><td>1.51</td><td>1.52</td></tr><tr><td colspan="2">MC-Dropout CIFAR-10</td><td>SVHN</td><td>3.55</td><td>0.112</td><td>3.60</td><td>0.105</td><td>1.51</td><td>1.52</td></tr><tr><td>BNN</td><td>CIFAR-10</td><td>SVHN</td><td>2.47</td><td>0.112</td><td>2.49</td><td>0.099</td><td>2.15</td><td>2.16</td></tr></table>

Model architectures. A description of the model architectures for MNIST, Fashion-MNIST and CIFAR-10. The numbers inside the parenthesis indicate the layer output dimension. For all models we use a ReLU activation function after each layer containing trainable weights.

Table content: <table><tr><td>Dataset</td><td>No. of layers</td><td>Architecture</td></tr><tr><td>MNIST</td><td>2</td><td>FC(300) → FC(10)</td></tr><tr><td>Fashion-MNIST</td><td>5</td><td>Conv(10) → MaxPool(2) → Conv(20) → MaxPool(2) → FC(120) → FC(82) → FC(10)</td></tr><tr><td>CIFAR-10</td><td>6</td><td>Conv(64) → Conv(64) → MaxPool(2) → Conv(64) → MaxPool(2) → Conv(64) → MaxPool(2) → FC(128) → FC(10)</td></tr><tr><td colspan="2">8.4 BINARY CLASSIFICATION</td><td/></tr><tr><td colspan="3">Experiments in this sub-section were conducted to show consistency with Dziugaite &amp; Roy (2017).</td></tr></table>

Results for experiments on a binary classification variant of MNIST. We report both loss and accuracy metrics for the training set and test set, together with the generalization loss and accuracy. Additionally, we report the KL-divergence value for the BNN models. We compare BNN to softmax models trained with Bernoulli dropout. Dropout rates and σ p values are in parenthesis next to model name, e.g., 'softmax (0.3)' refers to a softmax model with Bernoulli dropout of rate 0.3.Here we report results for multi-class classification for BNN and the baselines. Table7summarizes the results. The main purpose of these additional experiments is to explore more dropout and σ p values for different models.

Table content: <table><tr><td>Model</td><td colspan="4">Train loss Train acc. Test loss Test acc.</td><td>KL</td><td colspan="2">Gen. loss Gen. acc.</td></tr><tr><td>Softmax (0.0)</td><td>0.001</td><td>1.0</td><td>0.076</td><td>0.983</td><td>-</td><td>0.0746</td><td>0.017</td></tr><tr><td>Softmax (0.3)</td><td>0.011</td><td>1.0</td><td>0.055</td><td>0.983</td><td>-</td><td>0.0439</td><td>0.013</td></tr><tr><td>Softmax (0.5)</td><td>0.021</td><td>0.993</td><td>0.059</td><td>0.982</td><td>-</td><td>0.0379</td><td>0.010</td></tr><tr><td>BNN (0.1)</td><td>0.046</td><td>0.984</td><td>0.064</td><td>0.979</td><td>4950</td><td>0.0187</td><td>0.005</td></tr><tr><td>BNN (0.3)</td><td>0.047</td><td>0.983</td><td>0.081</td><td>0.975</td><td>4196</td><td>0.0331</td><td>0.008</td></tr><tr><td colspan="3">8.5 MULTI-CLASS CLASSIFICATION</td><td/><td/><td/><td/><td/></tr></table>

Results for multi-class image classification benchmarks. We report both loss and accuracy metrics for the training set and test set, together with the generalization loss and accuracy. Additionally, we report the KL-divergence for the BNN models. We compare BNN to softmax models and MC-Dropout models, trained with Bernoulli dropout. Dropout rates and σ p values are in parenthesis next to the model name, e.g., 'softmax (0.5)' refers to a softmax model with Bernoulli dropout of rate 0.5.Additional experimental results for out-of-distribution examples can be found in Table8.

Table content: <table><tr><td>Model</td><td colspan="4">Train loss Train acc. Test loss Test acc.</td><td>KL</td><td colspan="2">Gen. loss Gen. acc.</td></tr><tr><td/><td/><td/><td>MNIST</td><td/><td/><td/><td/></tr><tr><td>Softmax (0.0)</td><td>0.0003</td><td>1.0</td><td>0.0699</td><td>0.983</td><td>-</td><td>0.070</td><td>0.02</td></tr><tr><td>Softmax (0.1)</td><td>0.0008</td><td>1.0</td><td>0.0651</td><td>0.984</td><td>-</td><td>0.064</td><td>0.016</td></tr><tr><td>Softmax (0.5)</td><td>0.0103</td><td>1.0</td><td>0.0682</td><td>0.983</td><td>-</td><td>0.058</td><td>0.014</td></tr><tr><td>Softmax(0.7)</td><td>0.012</td><td>1.0</td><td>0.0658</td><td>0.983</td><td>-</td><td>0.054</td><td>0.013</td></tr><tr><td>Softmax(0.9)</td><td>0.1026</td><td>0.97</td><td>0.1385</td><td>0.962</td><td>-</td><td>0.036</td><td>0.008</td></tr><tr><td>MC-Dropout (0.1)</td><td>0.0009</td><td>1.0</td><td>0.0651</td><td>0.984</td><td>-</td><td>0.064</td><td>0.016</td></tr><tr><td>MC-Dropout (0.5)</td><td>0.0111</td><td>1.0</td><td>0.0678</td><td>0.982</td><td>-</td><td>0.057</td><td>0.015</td></tr><tr><td>MC Dropout(0.7)</td><td>0.0178</td><td>0.99</td><td>0.0781</td><td>0.979</td><td>-</td><td>0.0603</td><td>0.016</td></tr><tr><td>MC Dropout(0.9)</td><td>0.1373</td><td>0.96</td><td>0.1771</td><td>0.953</td><td>-</td><td>0.0398</td><td>0.007</td></tr><tr><td>BNN (0.3)</td><td>0.0448</td><td>0.986</td><td>0.0647</td><td>0.98</td><td>4251</td><td>0.02</td><td>0.006</td></tr><tr><td/><td/><td/><td>Fashion-MNIST</td><td/><td/><td/><td/></tr><tr><td>Softmax (0.0)</td><td>0.0159</td><td>0.995</td><td>0.6415</td><td>0.896</td><td>-</td><td>0.62</td><td>0.098</td></tr><tr><td>Softmax (0.1)</td><td>0.0899</td><td>0.966</td><td>0.3644</td><td>0.902</td><td>-</td><td>0.27</td><td>0.064</td></tr><tr><td>Softmax (0.5)</td><td>0.1941</td><td>0.927</td><td>0.3106</td><td>0.895</td><td>-</td><td>0.12</td><td>0.032</td></tr><tr><td>Softmax(0.7)</td><td>0.2361</td><td>0.915</td><td>0.333</td><td>0.892</td><td>-</td><td>0.097</td><td>0.023</td></tr><tr><td>Softmax(0.9)</td><td>0.8366</td><td>0.643</td><td>0.866</td><td>0.639</td><td>-</td><td>0.03</td><td>0.004</td></tr><tr><td>MC-Dropout (0.1)</td><td>0.0913</td><td>0.965</td><td>0.3587</td><td>0.901</td><td>-</td><td>0.27</td><td>0.064</td></tr><tr><td>MC-Dropout (0.5)</td><td>0.1973</td><td>0.926</td><td>0.3097</td><td>0.897</td><td>-</td><td>0.11</td><td>0.029</td></tr><tr><td>MC Dropout(0.7)</td><td>0.2535</td><td>0.913</td><td>0.3387</td><td>0.887</td><td>-</td><td>0.0852</td><td>0.026</td></tr><tr><td>MC Dropout(0.9)</td><td>0.908</td><td>0.608</td><td>0.9318</td><td>0.602</td><td>-</td><td>0.0238</td><td>0.006</td></tr><tr><td>BNN (0.3)</td><td>0.1841</td><td>0.932</td><td>0.2640</td><td>0.904</td><td>4869</td><td>0.08</td><td>0.028</td></tr><tr><td/><td/><td/><td>CIFAR-10</td><td/><td/><td/><td/></tr><tr><td>Softmax (0.0)</td><td>0.3361</td><td>0.881</td><td>0.4974</td><td>0.841</td><td>-</td><td>0.16</td><td>0.039</td></tr><tr><td>Softmax (0.1)</td><td>0.4170</td><td>0.858</td><td>0.5064</td><td>0.828</td><td>-</td><td>0.09</td><td>0.351</td></tr><tr><td>Softmax (0.5)</td><td>0.5299</td><td>0.821</td><td>0.5712</td><td>0.804</td><td>-</td><td>0.04</td><td>0.017</td></tr><tr><td>Softmax(0.7)</td><td>0.4366</td><td>0.852</td><td>0.5494</td><td>0.815</td><td>-</td><td>0.113</td><td>0.037</td></tr><tr><td>Softmax(0.9)</td><td>1.2778</td><td>0.495</td><td>1.3491</td><td>0.499</td><td>-</td><td>0.071</td><td>-0.004</td></tr><tr><td>MC-Dropout (0.1)</td><td>0.4189</td><td>0.856</td><td>0.5071</td><td>0.830</td><td>-</td><td>0.09</td><td>0.026</td></tr><tr><td>MC-Dropout (0.5)</td><td>0.5293</td><td>0.821</td><td>0.5711</td><td>0.807</td><td>-</td><td>0.04</td><td>0.014</td></tr><tr><td>MC Dropout(0.7)</td><td>0.4595</td><td>0.843</td><td>0.5878</td><td>0.811</td><td>-</td><td>0.128</td><td>0.033</td></tr><tr><td>MC Dropout(0.9)</td><td>1.3045</td><td>0.483</td><td>1.3935</td><td>0.482</td><td>-</td><td>0.089</td><td>0.001</td></tr><tr><td>BNN (0.3)</td><td>0.5643</td><td>0.807</td><td>0.5699</td><td>0.806</td><td>18237</td><td>0.005</td><td>0.001</td></tr></table>

Results for the out-of-distribution experiments. We report both loss and accuracy measures for the training set and test set, together with entropy values. We explore training on MNIST (MN), Fashion-MNIST (FMN), and CIFAR-10. Testing was done on MNIST (MN), Fashion-MNIST (FMN), NotMNIST, and SVHN. Reported losses and accuracies are computed using the out-ofdistribution dataset. Train loss Train acc. Test loss Test acc. Train ent. Test ent.

Table content: <table><tr><td>Model</td><td>In-dis.</td><td>Out-dis.</td><td/><td/><td/><td/><td/><td/></tr><tr><td>Softmax (0.0)</td><td>FMN</td><td>MN</td><td>10.32</td><td>0.136</td><td>10.04</td><td>0.146</td><td>0.34</td><td>0.34</td></tr><tr><td>Softmax (0.1)</td><td>FMN</td><td>MN</td><td>9.41</td><td>0.114</td><td>9.28</td><td>0.116</td><td>0.42</td><td>0.41</td></tr><tr><td>Softmax (0.5)</td><td>FMN</td><td>MN</td><td>8.16</td><td>0.114</td><td>7.97</td><td>0.114</td><td>0.42</td><td>0.41</td></tr><tr><td>MC-Dropout (0.1)</td><td>FMN</td><td>MN</td><td>9.12</td><td>0.113</td><td>8.99</td><td>0.113</td><td>0.44</td><td>0.43</td></tr><tr><td>MC-Dropout (0.5)</td><td>FMN</td><td>MN</td><td>7.68</td><td>0.113</td><td>7.50</td><td>0.113</td><td>0.46</td><td>0.45</td></tr><tr><td>BNN (0.3)</td><td>FMN</td><td>MN</td><td>5.31</td><td>0.096</td><td>5.22</td><td>0.098</td><td>0.79</td><td>0.75</td></tr><tr><td>Softmax (0.0)</td><td>MN</td><td>FMN</td><td>14.94</td><td>0.109</td><td>15.03</td><td>0.104</td><td>0.28</td><td>0.29</td></tr><tr><td>Softmax (0.1)</td><td>MN</td><td>FMN</td><td>16.27</td><td>0.098</td><td>16.33</td><td>0.092</td><td>0.29</td><td>0.29</td></tr><tr><td>Softmax (0.3)</td><td>MN</td><td>FMN</td><td>18.07</td><td>0.094</td><td>18.15</td><td>0.089</td><td>0.30</td><td>0.31</td></tr><tr><td>Softmax (0.5)</td><td>MN</td><td>FMN</td><td>18.78</td><td>0.077</td><td>18.89</td><td>0.075</td><td>0.38</td><td>0.38</td></tr><tr><td>MC (0.1)</td><td>MN</td><td>FMN</td><td>16.28</td><td>0.098</td><td>16.34</td><td>0.093</td><td>0.29</td><td>0.29</td></tr><tr><td>MC (0.3)</td><td>MN</td><td>FMN</td><td>18.09</td><td>0.093</td><td>18.16</td><td>0.090</td><td>0.31</td><td>0.31</td></tr><tr><td>MC (0.5)</td><td>MN</td><td>FMN</td><td>18.83</td><td>0.076</td><td>18.93</td><td>0.076</td><td>0.37</td><td>0.38</td></tr><tr><td>BNN (0.1)</td><td>MN</td><td>FMN</td><td>6.69</td><td>0.102</td><td>6.70</td><td>0.099</td><td>0.72</td><td>0.73</td></tr><tr><td>BNN (0.3)</td><td>MN</td><td>FMN</td><td>8.67</td><td>0.069</td><td>8.71</td><td>0.069</td><td>0.55</td><td>0.57</td></tr><tr><td>Softmax (0.0)</td><td>MN</td><td>NotMNIST</td><td>13.48</td><td>0.078</td><td>13.84</td><td>0.085</td><td>0.27</td><td>0.27</td></tr><tr><td>Softmax (0.1)</td><td>MN</td><td>NotMNIST</td><td>15.45</td><td>0.061</td><td>16.03</td><td>0.065</td><td>0.23</td><td>0.24</td></tr><tr><td>Softmax (0.3)</td><td>MN</td><td>NotMNIST</td><td>19.38</td><td>0.071</td><td>19.25</td><td>0.085</td><td>0.20</td><td>0.18</td></tr><tr><td>Softmax (0.5)</td><td>MN</td><td>NotMNIST</td><td>21.61</td><td>0.063</td><td>21.89</td><td>0.048</td><td>0.18</td><td>0.18</td></tr><tr><td>MC (0.1)</td><td>MN</td><td>NotMNIST</td><td>15.46</td><td>0.061</td><td>16.05</td><td>0.065</td><td>0.23</td><td>0.24</td></tr><tr><td>MC (0.3)</td><td>MN</td><td>NotMNIST</td><td>19.44</td><td>0.072</td><td>19.23</td><td>0.076</td><td>0.20</td><td>0.18</td></tr><tr><td>MC (0.5)</td><td>MN</td><td>NotMNIST</td><td>21.73</td><td>0.062</td><td>21.92</td><td>0.053</td><td>0.19</td><td>0.19</td></tr><tr><td>BNN (0.1)</td><td>MN</td><td>NotMNIST</td><td>5.67</td><td>0.087</td><td>5.52</td><td>0.085</td><td>0.79</td><td>0.74</td></tr><tr><td>BNN (0.3)</td><td>MN</td><td>NotMNIST</td><td>7.71</td><td>0.060</td><td>7.99</td><td>0.070</td><td>0.57</td><td>0.57</td></tr><tr><td>Softmax (0.0)</td><td>FMN</td><td>NotMNIST</td><td>12.46</td><td>0.152</td><td>11.92</td><td>0.13</td><td>0.29</td><td>0.27</td></tr><tr><td>Softmax (0.1)</td><td>FMN</td><td>NotMNIST</td><td>9.71</td><td>0.080</td><td>9.71</td><td>0.074</td><td>0.32</td><td>0.38</td></tr><tr><td>Softmax (0.5)</td><td>FMN</td><td>NotMNIST</td><td>8.79</td><td>0.122</td><td>8.22</td><td>0.087</td><td>0.5.</td><td>0.52</td></tr><tr><td>MC-Dropout (0.1)</td><td>FMN</td><td>NotMNIST</td><td>9.45</td><td>0.079</td><td>9.45</td><td>0.076</td><td>0.34</td><td>0.4</td></tr><tr><td>MC-Dropout (0.5)</td><td>FMN</td><td>NotMNIST</td><td>8.23</td><td>0.156</td><td>7.47</td><td>0.157</td><td>0.46</td><td>0.44</td></tr><tr><td>BNN (0.3)</td><td>FMN</td><td>NotMNIST</td><td>4.74</td><td>0.123</td><td>4.72</td><td>0.11</td><td>0.79</td><td>0.81</td></tr><tr><td>Softmax (0.0)</td><td>CIFAR-10</td><td>SVHN</td><td>4.93</td><td>0.115</td><td>5.13</td><td>0.109</td><td>0.94</td><td>0.96</td></tr><tr><td>Softmax (0.1)</td><td>CIFAR-10</td><td>SVHN</td><td>3.78</td><td>0.123</td><td>3.86</td><td>0.115</td><td>1.51</td><td>1.54</td></tr><tr><td>Softmax (0.5)</td><td>CIFAR-10</td><td>SVHN</td><td>3.11</td><td>0.121</td><td>3.19</td><td>0.131</td><td>1.73</td><td>1.75</td></tr><tr><td colspan="2">MC-Dropout (0.1) CIFAR-10</td><td>SVHN</td><td>3.79</td><td>0.123</td><td>3.87</td><td>0.115</td><td>1.50</td><td>1.53</td></tr><tr><td colspan="2">MC-Dropout (0.5) CIFAR-10</td><td>SVHN</td><td>3.10</td><td>0.122</td><td>3.17</td><td>0.132</td><td>1.74</td><td>1.77</td></tr><tr><td>BNN (0.3)</td><td>CIFAR-10</td><td>SVHN</td><td>3.45</td><td>0.091</td><td>3.54</td><td>0.094</td><td>1.71</td><td>1.75</td></tr><tr><td>8.7 FIGURES</td><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td colspan="4">Additional figures for various experiments.</td><td/><td/><td/><td/><td/></tr></table>

Peer review (iclr2020):
This paper suggests a PAC-Bayesian bound for negative log-likelihood loss function. Many PAC-Bayesian bounds are provided for bounded loss functions but as authors point out, Alquier et al. (2016) and Germain et al. (2016) extend them to unbounded loss functions. I have two major concerns regarding this paper: 1- Technical contribution: Since Alquier et al. (2016) has already introduced PAC-Bayesian bounds for the hinge-loss, I think the technical contributions of this paper is not significant enough for the publication. Moreover, the particular format of the bound in Theorem 2 is problematic since the right hand side depends on the data-distribution. When presenting the generalization bound, we really want the right hand side to be independent of the distribution (given the training set) and that is the whole point of calculating the generalization bounds. In particular, I don*t see why inequality (1) is any better than inequality (2). 2- Experiments: The main issue with the correlation analysis done in Section 6 is that authors only change depth of the networks and then check the correlation of the generalization bound to the test error. The problem is that in all those networks deeper ones generalize better so it is not clear that the correlation is due to a direct relationship to generalization or a direct relationship to depth. For example, if we take 1/depth as a measure, it would correlate very well with generalization in all these experiments but 1/depth is definitely not the right complexity measure or generalization bound. To improve the evaluation, I suggest varying more hyperparameters to avoid the above issue. *************************** After rebuttals: Unfortunately, my concerns are not addressed adequately by authors. Therefore, my evaluation remains the same.