Title: COMPOUND DENSITY NETWORKS

Abstract: Despite the huge success of deep neural networks (NNs), finding good mechanisms for quantifying their prediction uncertainty is still an open problem. It was recently shown, that using an ensemble of NNs trained with a proper scoring rule leads to results competitive to those of Bayesian NNs. This ensemble method can be understood as finite mixture model with uniform mixing weights. We build on this mixture model approach and increase its flexibility by replacing the fixed mixing weights by an adaptive, input-dependent distribution (specifying the probability of each component) represented by an NN, and by considering uncountably many mixture components. The resulting model can be seen as the continuous counterpart to mixture density networks and is therefore referred to as compound density networks. We empirically show that the proposed model results in better uncertainty estimates and is more robust to adversarial examples than previous approaches.

1. INTRODUCTION:

Deep neural networks (NNs) have achieved state-of-the-art performance in many application areas, such as computer vision (Krizhevsky et al., 2012) and natural language processing (Collobert et al., 2011) . However, despite achieving impressive prediction accuracy on these supervised machine learning tasks, NNs do not provide good ways of quantifying predictive uncertainty. This is undesirable for many mission critical applications, where taking wrong predictions with high confidence could have fatal consequences (e.g. in medical diagnostics or autonomous driving).

A principled and the most explored way to quantify the uncertainty in NNs is through Bayesian inference. In the so-called Bayesian neural networks (BNNs) (Neal, 1995) , the NN parameters are treated as random variables and the goal of learning is to infer the posterior probability distribution of the parameters given the training data. Since exact Bayesian inference in NNs is computationally intractable, different approximation techniques have been proposed (Neal, 1995; Blundell et al., 2015; Hernández-Lobato & Adams, 2015; Ritter et al., 2018, etc.) . Given the (approximate) posterior, the final predictive distribution is obtained as the expected distribution under the posterior, where the expectation can be seen as an ensemble of an uncountably infinite number of predictors, where the prediction of each model is weighted by the posterior probability of the corresponding parameters.

Based on a Bayesian interpretation of dropout (Srivastava et al., 2014) , Gal & Ghahramani (2016) proposed to apply it not only during training but also when making predictions to estimate predictive uncertainty. Interestingly, dropout has been also interpreted as ensemble model (Srivastava et al., 2014) where the predictions are averaged over the different NNs resulting from different dropoutmasks. Inspired by this, Lakshminarayanan et al. (2017) proposed to use an simple NN ensemble to quantify the prediction uncertainty, i.e. to train a set of independent NNs using a proper scoring rule and defining the final prediction as the arithmetic mean of the outputs of the individual models, which corresponds to defining a uniformly-weighted mixture model. It is argued, that the model is able to encode two sources of uncertainty by calibrating the prediction uncertainty in each component and capturing the "model uncertainty" by averaging over the components. Note, that this should hold for any kind of predictive model that is defined in terms of a mixture distribution.

In this paper, we therefore aim at further investigating the potential that lies in using mixture distributions for uncertainty quantification. The flexibility of a mixture model can be increased by learning input-conditioned mixture weights, like it is done by mixture density networks (MDN) (Bishop, 1994) . Furthermore, one can consider uncountably many component distributions instead of a finite set, which turns the mixture distribution into a compound distribution. We combine both by deriving the continuous counterpart of MDNs, which we call compound density networks (CDNs). This model corresponds to a compound distribution in which both, component and mixing distribution, are parametrized based on NNs. We experimentally show that CDNs allow for better uncertainty quantification and are more robust to adversarial examples than previous approaches. This paper is organized as follows. In Section 2 we give a brief introduction to MDNs. We then formally define CDNs in Section 3. We review related work in Section 5 and present a detailed experimental analysis in Section 6. Finally we conclude our paper in Section 7.

2. MIXTURE DENSITY NETWORKS:

Let D = {x n , y n } N n=1 be a i.i.d dataset and let us define the following conditional mixture model

EQUATION

and an NN that maps x onto both the parameters π(x) of the mixing distribution and the parameters {φ k (x)} K k=1 of the K mixture components. The complete system is called mixture density network (MDN) and was proposed by Bishop (1994) . That is, an MDN is an NN parametrizing a conditional mixture distribution, where both the mixture components and the mixture coefficients depend on input x.1 MDNs can be trained by maximizing the log-likelihood of the parameters of the NN given the training set D using gradient-based optimizers such as stochastic gradient descent (SGD) and its variants.

MDNs belong to a broader class of models called mixture of experts (MoE) (Jacobs et al., 1991) which differ from standard mixture models by assuming that the mixture coefficients depend on the input.2 Because of its formulation as a mixture distribution, the predictive distribution of an MDN can handle multimodality better than a standard discriminative neural network. Furthermore, the mixture coefficients allow us to encode uncertainty about the prediction, namely by modelling the probability from which component distribution a data point was sampled.

3. COMPOUND DENSITY NETWORKS:

We aim at generalizing the MDN from a finite mixture distribution to a mixture of an uncountable set of components. The continuous counterpart of a conditional mixture distribution in eq. ( 1) is given by the conditional compound probability distribution

EQUATION

where φ(x) turns from a discrete into a continuous random variable.

We now want to follow the approach of MDNs to model the parameters of the components and the mixing distribution by NNs. To handle the continuous state space of φ(x) in the case of a compound distributions, the key idea is now to let φ(x) be given by a stochastic NN f (x; θ) = φ(x) with stochastic parameters θ. Since given x, f is a deterministic map from θ to φ(x), it is possible to replace the mixing distribution p(φ(x)|π(x)) = p(f (x; θ)|π(x)) by a distribution p(θ|π(x)) over θ. We further assume, that the parameters π(x) of the mixing distribution are given by some parametrized function g(x; ψ) = π(x) which can also be modelled based on NNs. In correspondence to MDNs, the complete system is called compound density network (CDN) and it is summarized by the following equation

p(y|x; ψ) = p(y|f (x; θ))p(θ|g(x; ψ)) dθ = E p(θ|g(x;ψ)) [ p(y|f (x; θ))] . (3) h x φ(x) W 1 ∼ p(W 1 |π 1 (x)) W 2 ∼ p(W 2 |π 2 (h)) π 1 (x) = g 1 (x; ψ 1 ) π 2 (h) = g 2 (h; ψ 2 )

Figure 1 : An example of a probabilistic hypernetwork applied to a two-layer MLP.

As for MDNs, CDNs training corresponds to maximizing its likelihood function, which is given by

EQUATION

As we now have to deal with an integral instead of a finite sum, this log-likelihood function is intractable. Fortunately, applying the reparametrization trick (Kingma & Welling, 2014) to θ ∼ p(θ|g(x; ψ)) enables Monte Carlo integration, while still being able to optimize the log-likelihood via backpropagation in conjunction with SGD.

To avoid overfitting, we can regularize the mixing distribution p(θ|g(x; ψ)) by penalizing its KLdivergence w.r.t. some simple distribution p(θ)3 , leading to the regularized objective function

EQUATION

where λ controls the regularization strength. We summarize the training procedure of CDNs in the pseudo-code provided in Appendix A. Note, that CDNs correspond to an abstract framework for modelling compound distributions with NNs. In the following section, we present a concrete example on how CDNs can be implemented.4 

3.1 PROBABILISTIC HYPERNETWORKS Ha et al. (2017) proposed to (deterministically) generate the parameters of an NN by another NN, which they call the hypernetwork. 5 We would like to follow this approach for modelling a CDN, that is, we aim at modelling the mixing distribution p(θ|g(x; ψ)) over network parameters by NNs. Since now the hypernetworks maps x to a distribution over parameters instead of a specific value θ, we refer to them as probabilistic hypernetworks. In the following we will describe this idea in more detail.

Let f be a multi-layer perceptron6 (MLP) with L-layers, parametrized by a set of layers' weight matrices7 θ = {W l } L l=1 , that computes the parameters φ(x) = f (x; θ) of the CDNs component distribution in eq. (3). Let h 1 , . . . , h l-1 denote the states of the hidden layers, and let us define h 0 = x, and h L = f (x; θ). We now assume the weight matrices {W l } L l=1 to be random variables and to be independent from each other given the state of the previous hidden layer. We define a series of probabilistic hypernetworks g = {g l } L l=1 (parametrized by ψ = {ψ l } L l=1 ), where g l maps h l-1 to the parameters of the distribution of W l , and let the joint distribution over θ be given by

EQUATION

)

An illustration of a stochastic two-layer network f (x; θ) computing φ(x), with parameters distributions given by probabilistic hypernetworks as defined in eq. ( 6) is given in Figure 1 . To make the definition of our model complete, we need to define the concrete statistical model we pick for p(W l |g l (h l-1 ; ψ)). This is done in the following section.

3.2. PROBABILISTIC HYPERNETWORKS WITH MATRIX VARIATE NORMALS:

A statistical model that was recently applied as the posterior over weight matrices in BNNs (Louizos & Welling, 2016; Sun et al., 2017; Zhang et al., 2018; Ritter et al., 2018) is the matrix variate normal (MVN) distribution (Gupta & Nagar, 1999 ). An MVN is parametrized by three matrices: a mean matrix M and two covariance factor matrices A and B. It is connected to the multivariate Gaussian by the following equivalence

EQUATION

where vec(X) denotes the vectorization of X. Due to the Kronecker factorization of the covariance, an MVN requires less parameters compared to a multivariate Gaussian, which motivates us to use it as the distribution over weight matrices in this work. Furthermore, we assume that the covariance factor matrices are diagonal matrices, following Louizos & Welling (2016) . That is, we choose the mixture distribution of the CDN to be

EQUATION

where g l maps the state h l-1 of the previous hidden layer onto the l-th MVN's parameters {M l , a l , b l }, defining the distribution over W l . Suppose W l ∈ R r×c , then the corresponding MVN distribution has rc + r + c parameters, which is more efficient compared to rc + rc parameters when modeling W l as fully-factorized Gaussian random variable.

In practice, the procedure of sampling θ and computing f (x; θ) = h L now can be described by

EQUATION

for l = 1, . . . , L, where σ is an arbitrary point-wise nonlinear activation function, which may differ for hidden and output layers.

Note that using the mixing distribution defined in eq. ( 8) allows us to apply the reparametrization trick (Appendix C.1). For the KL-divergence-based regularization, a straight forward choice for the simple distribution is p(θ) = L l=1 MN (W l |0, I, I), which corresponds to assuming that each layer's weight matrix is standard matrix normal distributed. In this case, the KL-term can be computed in closed form, as noted by Louizos & Welling (2016) and shown in Appendix C.2. Finally, we use a vector-scaling parametrization similar to the one used by Ha et al. (2017) and Krueger et al. (2017) for the mean matrices {M l } L l=1 , which we explain in detail in Appendix D.

4. CONNECTION TO BAYESIAN NEURAL NETWORKS AND VARIATIONAL INFORMATION BOTTLENECK:

Interestingly, eq. ( 2) could also be interpreted as integrating over the parameters of a Bayesian neural network, where p(φ(x)|π(x)) is an amortized approximate posterior. Performing variational inference in such a setting would lead to the following objective8 

EQUATION

Note, that this is a lower bound of eq. ( 5) (by Jensen's inequality) and that stochastic approximations of both objectives become equivalent when they are based on a single sample of θ and λ = 1. For arbitrary value of λ, the one-sample-approximation of eq. ( 5) is also equivalent to the approximated variational information bottleneck (VIB) objective (Alemi et al., 2017) . Following this objective, the proposed approach corresponds to performing VIB where the network parameters, instead of the hidden units, are considered as latent variables. We will analyze the effects of following these different objectives experimentally in Section 6.4.

5. RELATED WORK:

Various approaches for quantifying predictive uncertainty in NNs have been proposed. Applying Bayesian inference to NNs, i.e. treating the network parameters as random variables and estimating the posterior distribution given the training data based on Bayes' theorem, results in Bayesian neural networks (BNNs) (MacKay, 1992; Neal, 1995; Graves, 2011; Blundell et al., 2015; Louizos & Welling, 2016; Sun et al., 2017; Louizos & Welling, 2017; Ritter et al., 2018; Zhang et al., 2018, etc) . Since the true posterior distribution is intractable, BNNs are trained based on approximate inference methods such as variational inference (VI) (Peterson, 1987; Hinton & Van Camp, 1993; Graves, 2011; Blundell et al., 2015) , Markov Chain Monte Carlo (MCMC) (Neal, 1995) , or Laplace approximation (MacKay, 1992; Ritter et al., 2018) . The final prediction is then given by the expectation of the network prediction (given the parameters) w.r.t. the approximate posterior distribution. Louizos & Welling (2016) proposed to train BNNs based on VI with an MVN as the approximate posterior of each weight matrix (leading to a model they refer to as variational matrix Gaussian (VMG)). Multiplicative normalizing flow (MNF) (Louizos & Welling, 2017 ) models the approximate posterior as a compound distribution, where the mixing density is given by a normalizing flow. Zhang et al. (2018) also use an MVN approximate posterior and apply approximate natural gradient (Amari, 1998) based maximization on the VI objective, which results in an algorithm called noisy K-FAC. Meanwhile, the Kronecker-factored Laplace approximation (KFLA) (Ritter et al., 2018) extends the classical Laplace approximation by using an MVN approximate posterior with tractable and efficiently computed covariance factors, based on the Fisher information matrix.

There have been several concurrent works (Krueger et al., 2017; Louizos & Welling, 2017; Pawlowski et al., 2017; Sheikh et al., 2017) applying hypernetworks (Ha et al., 2017) to model the posterior distribution over network parameters in BNNs. In contrast to CDNs, the hypernetworks in these approaches are used to transform random noise drawn from a simple distribution into a random variable with a complicated distribution. That is, they use hypernetworks to sample from an implicit distribution of the parameters, without explicitly specifying the statistical model (like the MVN in our case). The approach by Sheikh et al. (2017) is the one most closely related to ours, since they use an objective analogous to eq. ( 4). Note, that the main difference between CDNs and these hypernet-BNNs is that the approximate posterior in a Bayesian setting does not depend on the current input point, while the mixing distribution of the CDN does. 

6. EXPERIMENTS:

We consider several standard tasks in our experimental analysis: 1D toy regression problems inspired by Hernández-Lobato & Adams (2015) and Depeweg et al. (2018) (Section 6.1), classification under out-of-distribution data (Section 6.2), and detection of and defense against adversarial examples (Szegedy et al., 2014 ) (Section 6.3). We consider the following recent models (described in Section 5) as the baselines for our CDN model: MNF, KFLA, noisy K-FAC, MC-dropout, and Deep Ensemble.9 

In all of the experiments, we consider λ ∈ {10 -n } 5 n=1 when training CDNs based on eq. ( 5), and pick the highest value of λ that still allows for high predictive power (e.g. an accuracy of > 0.97 on the validation set). We use this selection heuristic, based on our observation that in general, as λ increases, the accuracy is decreasing while the uncertainty estimate is increasing (see Appendix F.1 for an investigation of this behavior). The hyperparameters for the baselines are set to the values suggested by the respective original publications as outlined in Appendix F.

For BNNs and the CDN we estimate the predictive distribution p(y|x), based on 100 samples of network parameters from the posterior distribution or the mixing distribution, respectively. Unless stated otherwise, we use a single sample of θ to do Monte Carlo integration during training. For MNIST and CIFAR-10 experiments, we use mini-batches of size 200. We use Adam (Kingma & Ba, 2015) with default hyperparameters for optimization in all experiments and the implementations provided by Louizos & Welling (2017) 10 and Zhang et al. (2018) 11 for MNF and noisy K-FAC, respectively. The implementation of all models and experiments will be made available to the public once the review process is concluded.

6.1. TOY REGRESSION:

Following Hernández-Lobato & Adams (2015) and Depeweg et al. (2018) , we use two toy regression problems to demonstrate the capability of CDNs of quantifying predictive uncertainty. The datasets of those probems are generated as follows.

• Cubic (Hernández-Lobato & Adams, 2015): We sample 20 input points x ∼ U [-4, 4] and their target values y = x 3 + , where ∼ N (0, 3 2 ). • Mixture (Depeweg et al., 2018) : We sample 750 input points x ∼ p(x), where p(x) = 1 3 N (-4, 2 5 ) + 1 3 N (0, 0.9) + 1 3 N (4, 2 5 ) and generate their target values by y = sin x + 3|cos x 2 | , where ∼ N (0, 1).

We use a single layer MLP with 100 hidden units as the predictive network, while the hypernetworks (g 1 and g 2 ) are modeled with single layer MLPs with 10 hidden units each. Three samples from the mixing distribution are used to do Monte Carlo integration to approximate eq. ( 4). The results are presented in Figure 2 . We observe that our model is able to accurately quantify the heteroscedastic noise in the Mixture dataset. Comparing the results for the CDN with those of Bayesian models on the Cubic dataset (Figure 8 in the appendix) it becomes clear that the CDN is quantifying the noise applied in the data-generating process (aleatoric uncertainty) instead of epistemic uncertainty. A visualization of the learned mixing distributions for different input points is provided in Appendix G. images (of the same size and format as MNIST) of letters from the alphabet instead of handwritten digits. On such an out-of-distribution test set, the predictive distribution of an ideal model should have maximum entropy, i.e. it should have a value of ln 10 ≈ 2.303 which would be achieved if all ten classes are equally probable. The NN used for this experiment is an MLP with a 784-100-10 architecture.

We present the results in Figure 3 , where we plotted the cumulative distribution function (CDF) of the empirical entropy of the predictive distribution, following Louizos & Welling (2017) . A CDF curve close to the top-left corner of the figure implies that the model yields mostly low entropy predictions, indicating that the model is very confident. While one wishes to observe high confidence on data points similar to those seen during training, the model should express uncertainty when exposed to out-of-distribution data. That is, we prefer a model to have a CDF curve closer to the bottom-right corner on notMNIST, as this implies it makes mostly uncertain (high entropy) predictions, and a curve closer to the upper-left corner for MNIST. As the results show, our model has very high confidence on the test set of MNIST while having the lowest confidence on notMNIST compared to all baseline models. It is surprising that even though CDNs are designed to capture aleatoric uncertainty they outperforms other models in this experiment, which is rather designed for quantifying model uncertainty (Kendall & Gal, 2017) .

6.3. ADVERSARIAL EXAMPLES:

To investigate the robustness and detection performance of the CDN w.r.t. adversarial examples (Szegedy et al., 2014) , we apply the Fast Gradient Sign Method (FGSM) (Goodfellow et al., 2015) to a 10% fraction (i.e. 1000 samples) of the MNIST and CIFAR-10 test set. We do so, by making use of the implementation provided by Cleverhans (Papernot et al., 2018) . We use the same MLP (with 784-100-10 architecture) as before for experiments on MNIST, and the LeNet5 convolutional network (LeCun et al., 1998) for experiments on CIFAR-10. Out of implementation reasons, we only model the parameters of the fully-connected layers of LeNet5 as random variables for CDN and KFLA. The probabilistic hypernetworks are two-layer MLPs with 100 hidden units. Note, that we do not use adversarial training when training the Deep Ensemble in this experiment to allow for a fair comparison.

MNIST Figure 4 presents the accuracy and the average empirical entropy of the predictive distribution w.r.t. adversarial examples for MNIST with varying levels of perturbation strength (between 0 and 1). We observe that the CDN is significantly more robust to adversarial examples than all baseline models. The prediction entropy for adversarial examples is also higher for the CDN than for most other models (only MC-dropout shows higher uncertainty in the beginning). Furthermore, the prediction uncertainty of the CDN is steadily increases with increasing perturbation strength. We demonstrate how the performance of our model depends on the choice of regularization strength λ in Figure 5 . It is clearly visible that λ acts as a hyperparameter that balances the trade-off between detection (uncertainty) and robustness (accuracy). When allowing for lower uncertainty, the CDN gets surprisingly robust, showing a prediction accuracy of around 0.4 even for a perturbation strength of 1. Even when we increase the strength of the adversarial examples by computing them based on 10 different samples from our model, the CDN is still significantly more robust than all baseline models w.r.t. the weaker adversarial examples relying on a single sample.

CIFAR-10:

Figure 6 shows that, leading to second best results w.r.t. accuracy as well as uncertainty, our model is competitive to other state of the art models on CIFAR-10. Note, that for the CDN we do not use probabilistic hypernetworks for the convolution layers of LeNet5, i.e. we only treat the parameters of the fully-connected layers as random variables. Treating all parameters as random variables, as it is done by the BNNs in this experiment (except from KFLA) could potentially improve the performance of the CDN.

6.4. COMPARISON TO TRAINING BASED ON ELBO AND VIB OBJECTIVE:

In this section, we experimentally investigate the effects of (a) optimizing our proposed objective (eq. ( 5)), (b) treating our model as a BNN and train it with the ELBO objective (eq. ( 10)), and (c) following the VIB approach (which corresponds to optimizing the objective which results from introducing a hyperparameter λ in front of the KL-term in eq. ( 10)). As stated before, (a) and (c) Training based on the ELBO results in unsatisfactory performance as can be seen from the orange graphs and the indicated low validation accuracy. The CDN and the VIB objective both work well on the OOD classification task, where no performance is gained by using 10 instead of one sample (which makes both equivalent). This suggests that the input dependency of the distribution over θ (in addition to the weighting of the KL-term in the objective) plays a crucial role for the increased performance observed compared to baseline models for OOD classification. This hypothesis is also supported by the fact that the performance is increase compared to the VMG (Louizos & Welling, 2016) , which is a closely related BNN using a MVN distributions as approximate (input independent!) posterior.

While for the robustness against adversarial attacks the increased sample size improved the performance of models trained with the VIB as well as with the CDN objective, the model trained with the CDN objective clearly outperforms the others, reaching an surprisingly high accuracy about 0.9 even under huge perturbations.

7. CONCLUSION:

We introduce compound density networks (CDNs), a new framework that allows for better uncertainty quantification in neural network (NN), and corresponds to a compound distribution (i.e. mixture with uncountable components) in which both the component distribution and the mixing distribution are parametrized by NNs. CDNs are inspired by the success of recently proposed ensemble methods and represent a continuous generalization of mixture density networks (MDNs) (Bishop, 1994) . They can be implemented by using a hypernetwork to map the input to a distribution over the parameters of the target NN, that models a predictive distribution. An extensive experimental analysis showed that CDNs are able to produce promising results in terms of uncertainty quantification. Especially, when facing FGSM-based adversarial attacks, the predictions of our model are significantly more robust than those of previous models, while simultaneously providing a better chance of detecting the attack by showing increased uncertainty.

APPENDIX A PSEUDOCODE FOR TRAINING CDNS:

Algorithm 1 The training procedure of CDN.

Require::

Mini-batch size M , number of samples S used for Monte Carlo integration (eq. ( 4)), regularization strength λ, and learning rate α.

1: while the stopping criterion is not satisfied do 

APPENDIX B CDNS WITH LATENT VARIABLES:

In Section 3, we introduce CDNs as a compound distribution where the stochasticity of f (x; θ) comes from treating θ as random variables. Another approach to achieve stochasticity on f is by introducing latent variable z and letting the parameter θ to be fixed (learned with MLE). The CDN would then be descibed by (11) The log-likelihood function is therefore

EQUATION

and the objective function is

EQUATION

The training algorithm in Algorithm 1 only needs to be trivially modified to accommodate θ, i.e. in the gradient steps. In the following section, we present an instance of this framework.

B.1 A VARIANT OF ADAPTIVE DROPOUT:

This model relies on injecting noise into the hidden activations of the neural network f , similar to dropout (Srivastava et al., 2014) . However, in this model, we condition the dropout's Gaussian on the activations of the previous hidden layer, similar to adaptive Bernoulli dropout proposed by Ba & Frey (2013) .

Let z l ∼ p(z l |g l (h l-1 ; ψ l )) be the noise vector of layer l. We then multiplicatively apply this noise to the activations h l , i.e. h l z l , where denotes the Hadamard product. Let z := {z l } L-1 l=1 and let g(x; ψ) := {g l (h l-1 ; ψ l )} L l=1 . Let p(z|g(x; ψ)) =:

L-1 l=1 p(z l |g(h l ; ψ l )) be the mixing distribution. Each of the p(z l |g(h l ; ψ l )) can be any distribution as long as we can apply reparametrization trick. Then, we can immediately apply latent CDNs training procedure on this model. One example of p(z l |g l (h l-1 ; ψ l )) is N (z l |1, g l (h l-1 ; ψ l )), which then the model resembles Gaussian dropout (Srivastava et al., 2014) 

APPENDIX G VISUALIZATION OF THE LEARNED MIXING DISTRIBUTION:

To further understand the effect of conditioning the distribution over θ (i.e. the mixing distribution) we compute the mixing distribution p(θ|g(x i ; ψ)) for a set of samples x 1 , . . . x n and a CDN trained on a variant of the Cubic dataset, where we add random noise of different variance depending on x (that is, ∼ N (0, 15 2 ) if x < 0 and ∼ N (0, 3 2 ), otherwise). The results for two randomly selected weights w (l) i ∈ θ are shown in Figure 12 . We found that the mean and the variance of the distribution (which is Gaussian due to our model) varies depending on the value of the input x indicating that different mixture components get high probability for different inputs. 

:

For instance, as in the original publication, the mixture components could be K Gaussians, with φ k (x) being input-specific means and variances, and the mixture probabilities could be given by (applying the softmax function to the unnormalized) mixing weights π(x), both computed by one NN.

See(Bishop, 2006, ch. 5.6 and ch. 14.5.3) and(Murphy, 2012, ch. 11.2.4) for a detailed discussion of MDNs.

Note, that another option is L p regularization on ψ.

An alternative implementation corresponding to a form of adaptive dropout is described in Appendix B.

Specifically, they propose to apply a hypernetwork to compute the weight matrix of a recurrent NN at each time-step, given the current input and the previous hidden state.

Note, that probabilistic hypernetworks can analogously be applied to model the distribution over parameters of any other kind of network.

7 We assume that the bias parameters are absorbed into the weight matrix.

Usually the approximate posterior does not depend on the input and therefore the KL term is independent from x as well. When performing mini-batch optimization in this case, it is common practice to re-scale the KL-term with 1/B where B is the number of mini-batches as described byGraves (2011).

We also investigated the VMG, an MoE, and an MDN. Due to space restrictions, results for these methods are reported in Appendix E.

https://github.com/AMLab-Amsterdam/MNF_VBNN

https://github.com/gd-zhang/noisy-K-FAC

Available at http://yaroslavvb.blogspot.com/2011/09/notmnist-dataset.html.

https://github.com/AMLab-Amsterdam/SEVDL_MGP.

Figures and Tables:

Krueger et al. (2017) andLouizos & Welling (2017) use normalizing flows, while Pawlowski et al. (2017) and Sheikh et al. (2017) use arbitrary NNs as their hypernetworks.

Figure 2: Predictive distributions of CDN. Black lines corresponds to the true noiseless function, red dots correspond to samples, orange lines and shaded regions correspond to the empirical mean and the ±3 standard deviation of the predictive distribution, respectively.

Figure 3: CDFs of the empirical entropy of the predictive distribution of the models, where the y-axis denotes the fraction of predictions having entropy less than the corresponding value on the xaxis. Confident models should have its CDF close to the top-left corner of the figure, while uncertain models to the bottom-right. The number next to each model name indicates its test accuracy.

Figure 4: Prediction accuracy and average entropy of models trained on MNIST when attacked by FGSM-based adversarial examples(Goodfellow et al., 2015) with varying perturbation strength.

Figure 6: Prediction accuracy and average entropy of models trained on CIFAR-10 when attacked by FGSM-based adversarial examples with varying perturbation strength.

for m = 1, . . . , M ; s = 1, . . . , S do 4:θ ms ∼ p(θ|g(x m ; ψ))Use reparametrization trick 5:φ s (x m ) = f (x m ; θ ms ) [ p(θ|g(x m ; ψ)) p(θ))] 9: L(ψ) = log p(D|ψ) -λD KL 10: ψ ← ψ + α∇L(ψ)Update parameters ψ 11: end while

p(y|x) = p(y|x, z; θ)p(z|x)dz := p(y|f (x; z, θ))p(z|g(x; ψ))dz = E p(z|g(x;ψ)) [ p(y|f (x; z, θ))] .

Figure 10: Empirical CDF curve of uncertainty on MNIST (left) and notMNIST (right)

Figure 11: Accuracy (left) and uncertainty (right) w.r.t. adversarial examples.

Figure 12: Visualization of the distribution p(w (l) i )|g(x; ψ))of a randomly selected weight w (l) i ∈ θ for different samples of input x from the modified Cubic toy dataset. w (l) i denotes the i-th weight of the l-th layer of f (x; θ).

Gal & Ghahramani (2016) developed a theoretical framework that relates dropout training in NNs to approximate Bayesian inference and, as a result, proposed to approximate the predictive distribution by an average over the different networks resulting from independently sampled dropout-masks, a technique which they referred to as MC-dropout and which they applied to estimate the prediction uncertainty in NNs.Recently, Lakshminarayanan et al. (2017), proposed to use an ensemble of NNs in conjunction with a proper scoring rule and adversarial training to quantify the prediction uncertainty of deep NNs, leading to a model referred to as Deep Ensemble. The Deep Ensemble provides a non-Bayesian way to quantify prediction uncertainty, and is in this sense related to the approaches ofGuo et al. (2017) andHendrycks & Gimpel (2017).

Table content: <table/>

with adaptive scaling.F.1 HYPERPARAMETER SEARCHWe perform a hyperparameter search on λ for the training of CDNs and for τ for KFLA based on the MNIST validation accuracy, and pick the highest λ (the lowest τ ) that achieve > 0.97 validation accuracy. Validation accuracy on MNIST vs λ for CDN and τ for KFLA.

Table content: <table><tr><td colspan="2">λ</td><td>CDN</td><td/><td/></tr><tr><td>1</td><td/><td>0.107</td><td>τ</td><td>KFLA</td></tr><tr><td colspan="2">0.1</td><td>0.189</td><td>1</td><td>0.806</td></tr><tr><td colspan="2">0.01</td><td>0.111</td><td colspan="2">10 0.977</td></tr><tr><td colspan="2">0.001</td><td>0.948</td><td colspan="2">20 0.978</td></tr><tr><td colspan="2">0.0001</td><td>0.966</td><td colspan="2">30 0.978</td></tr><tr><td colspan="2">0.00001</td><td>0.972</td><td colspan="2">40 0.978</td></tr><tr><td colspan="3">0.000001 0.972</td><td/><td/></tr><tr><td>λ</td><td colspan="4">MNF n. K-FAC Deep Ens. MC-dropout</td></tr><tr><td>1</td><td colspan="2">0.981 0.974</td><td>0.207</td><td>0.162</td></tr><tr><td>0.1</td><td colspan="2">0.981 0.976</td><td>0.885</td><td>0.834</td></tr><tr><td>0.01</td><td colspan="2">0.978 0.981</td><td>0.965</td><td>0.932</td></tr><tr><td>0.001</td><td>0.98</td><td>0.981</td><td>0.983</td><td>0.971</td></tr><tr><td>0.0001</td><td>0.98</td><td>0.979</td><td>0.986</td><td>0.976</td></tr><tr><td colspan="2">0.00001 0.98</td><td>0.979</td><td>0.986</td><td>0.977</td></tr></table>

Validation accuracy on MNIST vs λ.

Table content: <table/>

Peer review (iclr2019):
 In this work the authors propose an extension of mixture density networks to the continuous domain, named compound density networks. Specifically the paper builds on top of the idea of the ensemble neural networks (NNs) and introduces a stochastic neural network for handling the mixing components. The mixing distribution is also parameterised by a neural network. The authors claim that the proposed model can result in better uncertainty estimates and the experiments attempt to demonstrate the benefits of the approach, especially in cases of having to deal with adversarial attacks. The paper in general is well written and easy to follow. I have some concerns regarding the presentation of the main objective and the lack of justification in certain parts of the methodology. Let me elaborate. First of all, I don’t understand how the main equation of the compound density network in Equation (3) is different from the general case of a Bayesian neural network? Can the authors please comment on that? I also find weird the way that the authors arrive to their final objective in Equation (5). They start from Equation (4) which is incorrectly denoted as the log-marginal distribution while it is the same conditional distribution introduced in Equation (3) with the extra summation for all the available data points. Then they continue to Equation (5) which they present as the combination of the true likelihood with a KL regularisation term. However, what the authors implicitly did was to perform variational inference for maximising their likelihood by introducing a variational distribution q(.theta) = p(.theta | g(x_n; .psi). Is there a reason why the authors do not introduce their objective by following the variational framework? Furthermore, in the beginning of Section 3.1 the authors present their idea on probabilistic hypernetoworks which “maps x to a distribution over parameters instead of specific value .theta.” How is this different from the case that we were considering so far? If we had a point estimate for .theta we would not require to take an expectation in Equation (3) in the first place. My biggest concern in the methodology, however, has to do with the selection of the matrix variate normal prior for the weights and the imposition of diagonal covariances (diag(a) and diag(b)). The Kronecker product between two diagonal matrices results in another diagonal matrix, i.e., diagonal covariance, which implies that the weights within a layer are given by an independent multivariate Gaussian. What is the purpose then for introducing the matrix variate Gaussian? I would expect that you would like to impose additional structure to the weights. I expect the authors to comment on that. Regarding the experimental evaluation of the model rather confusing. The authors have proposed a model that due to the mixing is better suited for predictions with heteroscedastic noise and can better quantify the aleatoric uncertainty. However, the selected experiments on the cubic regression toy data (Section 5.1) and the out-of-distribution classification (Section 5.2) are clear examples of system’s noise, i.e. epistemic uncertainty. The generative process of the toy data clearly states that there is no heteroscedastic noise to handle. The same applies for the notMNIST data which belong to a completely different data set compared to MNIST and thus out of sample prediction cannot benefit from the mixing; i.e., variations have to be explained by system’s noise. So overall I have the feeling that the authors have not succeeded to evaluate the model’s power with these two experiments and we cannot draw any strong conclusions regarding the benefit of the proposed mixing approach. To continue with the experimental evaluation, I found the plots with the predictive uncertainty in Figure 3 a bit confusing. The plot by itself, as I understood, quantifies the model’s uncertainty in in- and out-of sample prediction. While I agree with the authors that it is generally desirable for a model to be more confident when predicting in MNIST (since it has already seen samples of it) compared to when predicting in notMNIST (completely different data), these plots tells us nothing regarding the predictive power of the model. There is no value in being very confident if you are wrong and vice-versa, so unless there is an accompanying plot/table reporting the accuracy I see not much value from this plot alone. Finally, it is unclear how the authors have picked the best .lambda parameter for their approach? On page 5 they state that they “pick the value that results in a good trade-off between high uncertainty estimates and high prediction accuracy.” Does this mean that you get to observe the performance in the test in order to select the appropriate value for .lambda? If this is the case this is completely undesirable and is considered a bad practice.