Gaussian Process Meta-Representations Of Neural Networks
Authors: Theofanis Karaletsos
Thang Bui
Venue:  (2020)
Keywords: Bayesian Neural Networks
Representation Learning
Gaussian Processes
Variational Inference

Abstract: Bayesian inference offers a theoretically grounded and general way to train neural networks and can potentially give calibrated uncertainty. It is, however, challenging to specify a meaningful and tractable prior over the network parameters. More crucially, many existing inference methods assume mean-field approximate posteriors, ignoring interactions between parameters in high-dimensional weight space. To this end, this paper introduces two innovations: (i) a Gaussian process-based hierarchical model for the network parameters based on recently introduced unit embeddings that can flexibly encode weight structures, and (ii) input-dependent contextual variables for the weight prior that can provide convenient ways to regularize the function space being modeled by the NN through the use of kernels. 
Furthermore, we develop an efficient structured variational inference scheme that alleviates the need to perform inference in the weight space whilst retaining and learning non-trivial correlations between network parameters. 
We show these models provide desirable test-time uncertainty estimates, demonstrate cases of modeling inductive biases for neural networks with kernels and demonstrate competitive predictive performance of the proposed model and algorithm over alternative approaches on a range of classification and active learning tasks.

Under review as a conference paper at ICLR 2020
GAUSSIAN PROCESS META-REPPESSENTATION FOR NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Bayesian inference offers a theoretically grounded and general way to train neural networks and can potentially give calibrated uncertainty. It is, however, challenging to specify a meaningful and tractable prior over the network parameters. More crucially, many existing inference methods assume mean-field approximate poste- riors, ignoring interactions between parameters high-dimensional weight space. To this end, this paper introduces two innovations: (i) a Gaussian process-basse -based hierarchical model for the network parameters based on recently introduced unit embeddings that can flexibly encode weight structures, and (ii) input-dependent contextual variables for the weight prior that can provide convenient ways to regularize the function space being modeled by the network through the use of kernels. Furthermore, we develop an efficient structured variational inference scheme that alleviates the need to perform inference in the weight space whilst retaining and learning non-trivial correlations between network parameters. We show these models provide desirable test-time uncertainty estimates, demonstrate cases of modeling inductive biases for neural networks with kernels and demon- strate competitive predictive performance of the proposed model and algorithm over alternative approaches range of classification and active learning tasks.
INTRODUCTION
Bayesian neural networks (BNNs) (see e.g. MacKay 1992 Neal 1993 Ghahramani (016) are one of the research frontiers on combining Bayesian inference and deep learning, potentially offering flexible modelling power with calibrated predictive performance. In essence, applying probabilistic inference to neural networks allows all plausible network parameters, not just the most likely, to be used for predictions. Despite some earlier success in supervised learning tasks (Guyon et al.) 2005) Quunoneoo-Caaneeee
al.) (006). BNNs has enjoyed limited recent practical applications. This could be attributed to: (i) the need to choose an appropriate weight prior as commonly modelers care about functions priors rather than weight priors but require the power and expressivity of weight-based models, and (ii) recently developed approximate inference methods not able to capture complicated posterior correlations in large networks. As such, addressing these challenges is an active research area.
This paper attempts to remedy the aforementioned limitations for Bayesian neural networks by extending previous work unit-based representations for neural networks with unit-based Gaussian process hierarchical prior over network parameters, which flexibly models the correlations between weights in layer and across layers while forgoing the need for a parametric hyper-network. Moreover, input-dependent contextual variables are used for modulating the GP-function and in turn specifying input-dependent weight priors which draw from the tools from the kernel literature imbue neural networks with inductive biases. This may yield more desirable uncertainty estimates at test time, or restrict function spaces to generalize in particular ways. A structured variational inference approach is employed that side-steps the need to do inference in the weight space whilst retaining weight correlations the approximate posterior. These theoretical benefits translate to competitive practical performance range of supervised and active learning tasks. The paper is organized as follows: in the next section, we will briefly review mean proposed hierarchical model for BNNs. The proposed model and inference scheme are discussed in section 3 and section 4 respectively, followed by a discussion on related work in section [] We then validate the performance of the proposed model and algorithm in section 6
variational inference for BNNs and a recently

---

Under review as a conference paper at ICLR 2020
2 BACKGROUND
In this section, we lay out the groundwork for the paper by providing a concise summary of variational approaches to Bayesian neural networks and a recent approach to learning metaareprrsentaation of neural networks.
2.1 BAYESIAN NEURAL NETWORKS AND VARIATIONAL INFERENCE
Consider a training set comprising of N input-output pairs, D = (Xn, Vn Jn and a neural net- work parameterized by weights and biases, w. that describes the distribution over an output yr given an input Xa. P(Un | x... We follow Bayesian approach by placing a prior distribution over the network parameters, p(w), and obtaining the posterior distribution D() which in- volves calulation of the marginal likelihood p(D) f (dwp(w)p(P)) w). However, obtaining p(w(D) and p(D) exactly is intractable when N is large or when the network is large and as such, approximation methods are often required. In particular, mean-field Gaussian variational inference (MFVI) has recently become method of choice for approximate inference for Bayesian neural networks due to its simplicity and the recently popularized reparameterization trick (Sal- imans & Knowles [013) Kingma & Welling, 2013] TTitsias & Lázaro-Gredilla 2014, Blundell| al. 2015). MFVI sidesteps the intractability by positing a diagonal Gaussian approximation q(w) Mw diag(o²)) and optimising an approximate lower bound to the marginal likelihood (MFVI(g(w)) R [ll((w)ll (L]((w)])P(W))) and 6x ~ I). i.. We is a sample from q( (w)). Note that the mean-fiele variational Gaussian approximation with standard normal prior, presented in is often outperformed by point estimation in certain settings (Trippe & Turner 2018). Despite being practical and able to give reasonable uncertainty estimates, improving MFVI is still an active research area, and the main focuses of which are (i) improving the reparameterization gradient estimator to enable faster convergence (Miller et al. 2017: Wu al., 2018). (ii) replacing the typical standard Normal prior, p(w) N(:: 0, by a structured prior that better models the structures present in the weight a- --priori (Ghosh et al.) 2018 Neal 2012; Blundell al., 2015). and (iii) using structured variational approximations that can potentially capture weight correlations in the posterior (Louizos & Welling| [0166] Zhang et al.) 2017). This paper builds on the two latter themes and proposes a hierarchical model for the prior and a structured variational scheme that explicitly model and infer weight structures.
K k= Σn log p(y,, Wk, xx), where Wa μ + Ek
2.2
META-REPRESENNING WEIGHTS AND NETWORKS
Our work builds on and expands the class of hierarchical neural network models based on the concept of latent variables associated with units in a network as proposed in Karaletsos et al. (2018). In these models, each unit (visible or hidden) of the [-th layer of the network has a corresponding latent hierarchical variable 2t,i, possibly of high dimensions D., where denotes the index of the unit in layer. Note that these latent variables do not describe the activation of units, but rather constitute latent features associated with a unit. These latent variables are then used to construct the weights in the network. Their construction is judiciously chosen such that weight in the I-th weight layer, w1,dd is linked to the latent variables z's of the i-th input unit and the j-th output unit of the weight layer. We can summarize this relationship by introducing the concept of weight codes Cw. consisting of the set of weight encodings for each individual weight in the network Car,s., 1 Zt,j]: which can be deterministically constructed from the collection of unit latent variables z by concatenating them correctly. Mathematically. the probabilistic description of the relationship between the weight codes (summarizing the structured latent variables) and the weights w is:
L-1 Hi Hiss
p(wlz) = pww (z)) = II II p(wr. (alaa///L... [2,11, Z1,),
1=1
where denotes a visible or hidden layer and H, the number of units in that layer, and w denotes all the weights in this network.
The central motivations for this parameterization are three-fold. First, weights in a layer and across layers are explicitly correlated in the modelling stage, as they share latent variables if they are connected to the same unit, unlike the standard normal prior typically used as described in the last section. Second, the number of visible and hidden units in neural network is typically much smaller than the number of weights. For example, for the 1-th weight layer, there are H, x H141 weights compared to H; + H1+1 associated latent variables. This encourages the development of inference
2

---

Under review as a conference paper at ICLR 2020
schemes that work in the lower-dimensional hierarchical latent space directly without the need to infer the weights. Third, as hierarchical latent variables are associated with units in the network and have the same cardinalities, inferring these latent variables conditioning observed data could be thought of as a form of network representation learning.
In Karaletsos al. 2018), small parametric neural network regression model (conceptually a structured hyper-network) is chosen to map the latent variables to the weights. The network either assumes Gaussian noise model which factorizes over weights, ((u.,,g/21+1.;;11..
(wn,as i Mt,i. ) L. where (Mrss log (1,55) NNo[[Z Z1,))) or models dependence be- tween weights implicitly as (w)CCw(2)
p(c) NNo((Z Zi,j, d))de, where is a ran- 1.t.f
dom variate, wi,g = NNo((Z ()), and 0 denotes the parameters of the neural network mapping. We will call this network meta mapping. Note that given the collection of sampled unit variables z, the weights (or theirs mean and variance) can be obtained efficiently in parallel. A prior over the latent variables z, p(z) = (z: 0. I). completes the model specification. We can thus write down the joint density of the resulting hierarchical model follows,
i p(zi) (z),
II p(y)lw,xx))
ply, w, 2/
Variational inference was employed in prior work infer (and w implicitly), and to obtain point estimate of 0, as by-product of optimising the variational lower bound.
3 META-REPRESENTATIONN WITH GAUSSIAN PROCESS MAPPINGS AND CONTEXTUAL LATENT VARIABLES
In this section, present novel extensions of the hierarchical model in section 2.2 that aim to increase the robustness the small data settings and improve its out-of-sample uncertainty estimates.
Bayesian NN
BNNs with GP meta-prior
BNNs with GP metaa-prior local var.
- Liv Hi
X
X
w
Ve
W.
Figure Graphical depiction of various models: vanilla BNNs, BNNs with hierarchical GP- MetaPriors, and BNNs with hierarchical GP-MetaPriors and auxiliary variables.
3.1 META-REPPESSENTING WEIGITTS WITH GAUSSIAN PROCESSES
Notice that in section 2.2 the meta mapping from the hierarchical latent variables to the weights is a parametric non- non-linear function, specified by a neural network. We replace the parametric neural network by a probabilistic functional mapping and place a nonparametric Gaussian process (GP) prior over this function. That is,
plwrsslf c NWi.si If [[ 2i,l),, o p(fh)) (P( 0, ke (,, :n),,
where we have assumed a zero-mean GP, k,, (:.)) a covariance function and ς small set of prrperpprramttes. The representational effect is that the latent function introduces correlations for the individual weight predictions,
I- H44 Hi
P(w(z) =
= p(f)
pl (wraslf 2141,5 2(.) df.
(4)
3

---

Under review as a conference paper at ICLR 2020
Here, present homoscedastic noise model for the weights, but the
correlated g it), eqq. 10
diagonal g(wiz), eq. 11
model is readily adaptable to het- eroscedastic noise model which we omit for clarity. For more de- tailed discussion on GPs and their applications. please see Rasmussen
margiual cav w Ro (w w
& Williams (2005). It is worth not- ing that whilst the number of latent variables and weights can be large, the input dimension to the GP map- ping is only 2D,, where D3 is the di- mensionality of each latent variable z.
00 ~> Q(())
(2x)
cond. cov w/z corr. g(w 10
The GP mapping effectively performs one-dimensional regression from la- tent variables to individual weights while capturing their correlations. We will refer this mapping as GP- MetaPrior (metaGP).
Figure 2: Marginal and conditional covariance structures over weights in a 1x50x neural network. Sampling from the posterior of the hierarchical model reveals that even diago- nal GP approximation can capture off-diagonal correlations induced through unit correlations. Also note the off- -if-oiagnaal bands in the marginal plots above, which indicate the corre- lation structures induced by the latent variables of the hidden units connecting the layers.
In general, the form of the covariance function and the values of the hyper- parameters encapsulate prior knowl- edge about the unknown function. We note that GPs benefit from the fact that kernels are compositional and can thus be used to represent correlation structures explicitly. We exploit this fact and define the following factor- ized kernel the example of two weights in the network,
kew (Cuot Cu=) ) = k(Zz Zr g1) [2 (z)
(5) In this section and what follows, will use the popular exponentiated quadratic kernel with ARD lengthscales, k(x1. X2) = of exp
(oou 1,3) kin ",,,",,
(220 -(=1.a=2..a)) where { are the kernel lengthscales 213
and of is the kernel variance. For the model considered here, this admits the following separation in the kernel computation:
D2 Σ
D. - -2h 21st1.jodd
(2h
d)) 213
kwy, f1 ii fadd da = of exp
exp Σ
21
that is, the computation of the covariance matrix between all the weights can be computed efficiently product of two sub-covariance matrices that computed using the latent variables.
3.2 CONTEXTUAL VARIABLES FOR MODULATING FUNCTION PRIORS
We first note that while the hierarchical latent variables and meta mappings introduce non-trivial coupling between the weights a priori, the weights and latent variables are inherently global. That is, a function drawn from the model, represented by set of weights, does not take into account the inputs at which the function will be evaluated. Inspired by recent work on using contextual information or dataset summary to drive supervised learning predictions (Edwards & Storkey 2016: Garnelo et al.) 2018), we introduce the input variable into the weight codes Cora. = [141,65 Aijj 4 In turn, this yields input-conditional weight models ( Zi+1,65 Zi,j, x,). We again turn to compositional kernels and introduce new input kernel K₂ which we as follows,
kew (Cwr Cw²) = k((z( +1,51 Zr 5", 21), [2, 2, 52, x2))
kout (Z)) (1,++1+++ +1." 1,2) Ann (Zp)," z²) (2(
As result of having private contextual inputs to the meta mapping, the weights are also local to each data point. In effect, each function drawn from this model explicitly needs the input locations are interested in, in similar vein to how functions are drawn from Gaussian process.
4

---

Under review as a conference paper at ICLR 2020
What effects should we expect from such modulation? As usual, the modeler has to choose the input kernel. Consider the use of an exponentiated quadratic kernel: we would expect data which lies far away from training data to receive small kernel values from K,. This, in turn, would modulate the entire kernel Ke,, for that data point to small values, leading to a weight model that reverts increasingly to the prior. We would hope such model would help with modeling uncertainty by resetting weights to uninformative distributions away from training data. However, one may also want to use this mechanism to model inductive biases for the network, such adding structure to the weight prior that can be captured with a kernel function. This is potentially appealing avenue, as multiple useful kernels have been found in the GP literature that allow modelers to describe relationships between data, but was not tool that was previously accessible to neural network modelers. We consider this a novel form of functional regularization, as the entire network can be given structure that will constrain its function space. Unfortunately, we cannot easily apply this method directly to high dimensional inputs as commonly used kernel functions fail in this setting. To overcome this limitation, we introduce {e, J one for each training instance, which can be then used to as an extra input dimension for the meta mapping to generate weights instead of the actual inputs. In detail, each auxiliary input is obtained via (potentially nonlinear) transformation applied to an input: én - g( (V.,,,, where V E RDxxD= and D, and D2 are the dimensionality e, and Xn. respectively, and g()) is an arbitrary transformation. We may also layer these transformations in general. We typically set D. < D so this transformation could be thought of as dimensionality reduction operation. For low dimensional imputs, we can directly set E, = Xn, The auxiliary variable is then augmented to the input of the meta mapping as follows,
p(wnatssl.. f Z141,1; Ziij Xn, V) (N [( [(Z 1,,, Z1,j 6n)), oZ))
that is, the input dimension of the meta mapping is now 2D= + De. The covariance matrices can be decomposed and computed efficiently as Kronecker products Ke = Kout @ Kin Kg, K2 maps over data (N x N) instead of over weights (W x W) like the previous kernels and so their product requires efficient approximation avoid calculating an object of size (WN W [ Additionally, we also place a prior over the linear transformation: p( VV) = NVV; I).
4 INFERENCE AND LEARNING USING STOCHASTIC STRUCTURED VARIATIONAL INFERENCE
Performing inference is challenging due to the non-linearity of the neural network and the need to infer an entire latent function f. To address these problems, we derive a structured variational
inference scheme that makes use of innovations from inducing point GP approximation literature
(Titsias 2009 Hensman et al.) 2013
Junñonero-Candee & Rasmussen | Matthews et al. 2016
Bui et al., 2017) and previous work on inferring retressttrrsnnnnn (Karaletsos et al. [018 As
reminder, we write down the joint density of all variables in the model specified in section 3
ply, w, fI = p(z)p( V ()(())(w S, z, V, v))p(yyin (ooyy/ wx
N
= -p())((V)pl
II p(w.lf Cz, Xn: ())(yxlw,xx) x=1
We first partition the space 2 of inputs to the function ʃ into a finite set of M variables called inducing inputs Zu and the remaining inputs, Z {Zu. ] The function f is partitioned identically, f = {u, f J. where u- = fzzu). We can then rewrite the GP prior as follows, p(f) = p(fz.. u, zu )p(u u z..) The inducing inputs and outputs, (Zu. u}. will be used to parameterize the approximation. In particular, a variational approximation is judiciously chosen to mirror the form of the joint density:
q(w,,, f. Vix) = ((z))(())((()).) |u, zn)g(u)p(wlff V,x),
(8)
where the variational distribution over w is made to explicitly depend on remaining variables through the conditional prior, qz)) and v(V) chosen to be diagonal (mean-field) Gaussian densities, y(z) = (2
(2 Pz, ) and q(V) = -N(V: diag(o )). respectively, and g(u) is chosen to be a correlated multivariate Gaussian, (()) -Nu May En)). This approximation allows convenient
We can solve this in complexity dominated by the decomposition of the largest kernel in the product. "The conditioning on Zez, in f(f e., [1, zu) is made implicit here and in the rest of this paper.
5

---

Under review as a conference paper at ICLR 2020
cancellations yielding tractable variational lower bound as follows,
[
g(w, V p())p(( V WIl.e///ttzaaaa//aa//aa/a//aAA
(.metaGr())
(())q 222
W.J.JVV
KL
KL (L
K Σ
fla. v logply/w, w.f
where the last expectation has been partly approximated using simple Monte Carlo with the repa- rameterization trick, i.e. 24 ~ ((() and Va ((V). ). We will next discuss how to approximate the expectation F& sw., q(w, J Za, VA, x) log loyp(y/w p(y|w, (,,l,,),,. x). Note that we split into ft. and u, and
that we can integrate ft, out exactly to give, q( ((w)p Za: u, Vk, x) = N(w; A (b) ((7)).
A (k)
K (k) K. fauu
B (k)
(()
K(4) Squ
K K ufzu + +
(9)
I.fuufu
At this point, either (i) sample u from q(u), (ii) integrate out analytically. We opt for the second approach, which gives
q( (w Zk. Vx.x)= = N(w; A (k) Pa. (k))
(kk) + (k) EgA.(kk)
(10)
In contrast to GP regression and classification in which the likelihood term is factorized point-wise w.r.t. the parameters and thus their expectations only involve low dimensional integral, we have to integrate out w in this case, which is of much higher dimensions. When necessary or practical, we resort to Kronecker factored models or make additional diagonal approximation follows,
@(w/zs: Izk: VA. x) -Nw A (k) pu, dBag(B + A E (.(k))
(11)
Whilst the diagonal approximation above might look poor from the first glance, it is conditioned sample of the latent variables z; and thus the weights' correlations are retained after integrating out Such correlation is illustrated in fig. 2 where show the marginal and conditional covariance structures for the weights of a small neural network, separated into diagonal and full covariance models. The diagonal approximation above has been observed to give pathological behaviours in the GP regression case (Bauer et al.) 2016), but we did not observe these in practice. A comparison of this diagonal approximation to alternatives is provided in the appendix. F&s is approximated by Fu 2 Jww (i(l//. Zk. Vv. x) log p(y /w, x) which can be subsequently efficiently estimated using the local reparameterization trick (Kingma et al., 2015). The final lower bound is then optimized to obtain the variational parameters of (u)) ((z) and q( [), and estimates for the noise in the meta-GP model, the kernel
and the inducing inputs.
RELATED WORK
Bayesian neural networks have recently garnered resurgence of interests in the community. There is, however, long history of research on developing approximate Bayesian inference methods for these networks. Notable work include extended Kalman filtering (Singhal & Wu 1989), Laplace's
approximation (MacKay 1992), Hamiltonian Monte Carlo Neal 1993 [993|20122 2012), variational inference
(Hinton & Van Camp 1993 Barber & Bishop 1998 Graves 2011 Blundell et al.) 2015 Gal &
Ghahramani] 2016), and approximate expectation propagation (Hernández-Loban
& Adams 2015
Li et al. [015]] | Hereeeeez-
al. 2016). The work in this paper orthogonal these the model employs a hierarchical prior, and inference is done in a nower-dimensional latent space instead of the weight space. The variational approximation is chosen such that the marginal distribution over the weights is non-Gaussian and the correlations between weights are retained, in contrast to the popular mean-field Gaussian approximation. Additionally, a standard multivariate Normal distribution is typically used the prior over the network weights and biases. Imposing additional structures over the weights a prior or carefully choosing the right prior variances have been observed to improve the predictive performance (Ghosh et al. 2018 Neal 2012 Blundell et al. 2015). In this vein, the proposed model in this paper imposes a class of flexible structures over the network parameters and learn posterior over them conditioned on observed data. Another related theme hyper-networks, the core idea of which is to generate network parameters using another network (see e.g. Ha et al., 2016. Stanley et al. 2009). The proposed model could be thought of GP

---

Under review as a conference paper at ICLR 2020
hyper-network, but utilizes latent variables for nodes as introduced in (Karaletsos et al. 2018). thus avoiding over paraaeetttde, monolithic models over weight tensors. Instead, the model uses hierarchical modeling, conditional probabilities, and the unit nodes perform one-dimensional regression to weights maintaining modeling correlations.
EXPERIMENTS
In this section, we evaluate the proposed model and inference scheme on several toy and benchmark datasets. We compare the proposed approach primarily to maximum posteriori estimation (MAP) and MFVI, both of which use standard Normal prior over the weights. These were implemented using PyTorch (Paszke et al. (017)) and the code will be available upon acceptance. Additional results are included in the appendices.
use M 50 inducing points for all experiments in this section.
6.1 TOY REGRESSION AND CLASSIFICATION EXAMPLES
MFVI
metaNN
metaGP
metaGP
local
Figure 3: Illustration of the performance of Bayesian neural networks with standard Normal prior over the weights and a mean-field Gaussian variational approximation [MFVI], a meta-NN prior, meta-GP prior and a meta-GP prior with auxiliary inputs. Top: 1-D regression observations. The plots show the predictive mean and uncertainty as well as the training points. a toy four-class dataset of 100 examples. The
100 noisy
Middle and bottom: Multiclass classification background colour matches the entropy of the predictive probability distribution, e.g. darker is more certain. Each pair of plots show the behaviour around and away from the training points when zooming out. Best viewed in colour.
We first illustrate the performance of the proposed model on two toy examples, one regression and one classification. In both cases, a Bayesian neural network with one hidden layer of 50 hidden units used. Additionally, the hierarchical latent variables have two dimensions, For the regression case, we generate 100 data points from toy function suggested by Louizos al. (2019). For classification, four-class dataset of 100 data points used. Figure shows the predictive performance of the proposed models, MFVI and meta-NN on both tasks. Notably, the proposed model with contextual inputs (and thus input dependent network weights) gives uncertainty estimates that are reminiscent to that of GP regression model, despite being neural network under the hood. Following Bradshaw et al. (2017), we also show the uncertainty for data further from the training instances. The model with auxiliary inputs remains uncertain expected for these points whilst MFVI produces arguably overconfident predictions. We also attempt to summarize the distribution of the weights in the network in appendix A.7 in the appendices.
6.2 INPUT DEPENDENT NEURAL NETWORKS FOR UNCERTAINTY QUANTIFICATION
Motivated by the performance of the proposed metaGP model with auxiliary inputs in the toy examples in fig. 3 this section tests the ability of this model class to produce calibrated predictive uncertainty out-of-distribution samples. That for test samples that do come from the
7

---

Under review as a conference paper at ICLR 2020
same training distribution, robust and well-calibrated model should produce uncertain predictive distribution and thus high predictive entropy. Such a model could find applications in safety- tasks where the cost of making confident but wrong predictions is high, or in area where detecting unfamiliar inputs is crucial such as active learning or reinforecement learning. In this work, we first train neural network classifier with one hidden layer of 100 rectified linear units the MNIST dataset, and apply the metaGP prior only to the last layer of the network. The dimensions of the latent variables and the auxiliary inputs are both 2. After training, we compute the entropy of the predictions on various test sets, including notMNIST, fashionMNIST, Kuzushiji-MNISS and uniform and Gaussian noise inputs. Following (Lakshminarayann the CDFs of the predictive entropies for various methods are shown in fig. A calibrated classifier should give CDF that bends towards the top-left corner of the plot for in-distribution examples and, vice versa, towards the bottom-right corner of the plot for cut-rf-diistibbttion inputs. In most out-of- distribution sets considered, except Gaussian random noise, metaGP and metaGP with local auxiliary variables demonstrate competitive performance to Gaussian MFVI. Notably, MAP estimation, often deployed in practice, tends to give wildly poor uncertainty estimates on out-of-distribue samples. We illustrate this behaviour and that of other models and methods on representative inputs of the MNIST and Kuzushiji-MNIST datasets in fig.
al. al.) 2017 Louizos Welling (017).
O
Z
3
6
7
gauss
MPV
metaGP local
frnisst
0.0
0.0
0.5
1.0
1.5
2.0
entropy Jnst
Figure 4: The CDFs of predictive entropies on in-distribution and out-of-distribution test sets for various methods [Left] and the predictive class probability by these methods for representative samples from an in-distribution test set [Top right] and an out-of out-of-distribution test test [Bottom right]. Best viewed in colour.
6.3 MODELING INDUCTIVE BIASES FOR NEURAL NETWORKS WITH INPUT-DEPENDENT KERNELS
We further explore the utility of the contextual variable towards modeling inductive biases for neural networks and evaluate on predictive performance on regression example. In particular, we generate 100 training points from a synthetic sinusoidal function and create two test sets that contains in-sample inputs and oft-of-sample inputs, respectively. We test an array of models and inference methods, including BNN with MFVI, metaGP and metaGP with contextual variables. In particular, as noted in section 3.1 we can choose the covariance function to be used for the auxiliary variables encode our belief about how the weights should modulated by the input. pick exponentiated quadratic and periodic kernels (MacKay) 1998) in this example. The predictive performance measured by the root mean squared error (RMSE) and the log-likelihood (LL) on the test examples. Figure [ summarizes the results and illustrate the qualitative difference between models. It could be noted that the periodic kernel allows the model to discover and encode periodicity, allowing for more long-range confident predictions compared to that of the exponentiated quadratic kernel. In this example, MetaGP with contextual variables is superior to other methods, demonstrating good RMSE and LL on both in-distribution and out-of-distribution examples.
6.4
ACTIVE LEARNING
We next stress-test the performance of the proposed model in pool- mool-baeed active learning setting for real-valued regression, where limited training data is provided initially and the target is to sequentially select points from pool set to add to the training set. The criterion to select the next best point from
8

---

Under review as a conference paper at ICLR 2020
MPVI
MetaGP
MetaGP local RBF lornel
MetaCP local periodic kernel
Figure 5: Illustration of the effect of local variables and different kernels for the local variables. The underlying function is sinusoid. Each subplot shows the training points (black pluses), the in-distribution test points (magenta crosses), the out-of-distribttion test points (magenta filled circles), the predictive mean (connected line), two standard deviations (shaded area) and added observation noise (red dashed line). The title of the subplots include the predictive performance on the in- distribution and out-of-distribure test points, demonstrating the ability of model interpolate and extrapolate, correspondingly. Best viewed in colour.
boston
power
energy
yacht
mfvi-random
5.5
11
10
metaGF-random
5.0
metaGPractive
RMSE
4.5
4.0
2
2
2
7
learning steps
leamning steps
learning steps
steps
Figure 6: Active learning with BNNs using mean-field Gaussian variational inference [MFVI] and meta-GP hierarchical prior [MetaGP] on several UCI regression datasets. Each trace shows the root mean squared error (RMSE) the test set more data points selected and moved from the pool set to the training set, averaged over 40 runs. The objective function for selecting points from the pool set the predictive variance. Best viewed in colour.
the pool set based on the entropy of the predictive distribution, i.c. we pick one with the highest entropy. In practice, we approximate the predictive density by Gaussian density, which results in tractable entropy computation. Note that this selection procedure can be interpreted as selecting points that maximally reduce the posterior entropy of the network parameters (Houlsby et al. (001)) Four UCI regression datasets were considered, where each with 40 random train/test/pool splits. For each split, the initial train set has 20 data points, the test set has 100 data points, and the remaining points are used for the pool set, similar to the active learning set-up up in -Hernández-Lobar Lobato & Adams (2015). We compare the performance of the proposed model and inference scheme to that of Gaussian mean-field variational inference and show the average results in fig. 66] Across all runs, we observe that active learning is superior random selection and more crucially using the proposed model and inference scheme seems to yield comparable or better predictive errors with a similar number of queries. We also illustrate the procedure on toy classification dataset and visualize the predictive uncertainty as more points are picked and used for training in fig. [7 in the appendix. This simple setting quantitatively reveals the inferior performance of MFVI, compared to MAP and metaGP.
SUMMARY
We have developed flexible hierarchical neural network model and structured variational inference scheme that together (i) enable flexible modelling of the correlations between network parameters, (ii) allow weight structures to be retained in the approximate posterior without having to perform direct inference in the high-dimensional weight space, and (iii) allow input-dependent contextual variables be used to generate private weights per data point which translates more desirable

---

Under review as a conference paper at ICLR 2020
uncertainty estimates in practice and the ability to generalize out of sample with inductive biases such as periodicity. We plan to evaluate the performance of the model on more challenging decision making tasks such contextual bandits reinforcement learning, and to extend the inference scheme to handle continual learning.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
David Barber and Christopher M. Bishop, Ensemble learning Bayesian neural networks. In Neural Networks and Machine Learning, 1998.
Matthias Bauer, Mark van der Wilk, Carl Edward Rasmussen. Understanding probabilistic sparse Gaussian
process approximations. In Advances in Neural Information Processing Systems, pp. 1533-1541, 2016.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International Conference on Machine Learning, pp. 1613-1622, 2015.
John Bradshaw, Alexander G de G Matthews, and Zoubin Ghahramani. Adversarial examples, uncertainty, and transfer testing robustness in Gaussian process hybrid deep networks. arXiv preprint arXiv:170..02476 2017.
Thang D Bui, Josiah Yan, and Richard E Turner. A unifying framework for Gaussian process pseudo-point approximations using power expectation propagation. The Journal of Machine Learning Research, 18(1): 3649-3720, 2017.
Harrison Edwards and Amos Storkey. Towards neural statistician. arXiv preprint urXiv:100..02150 2016.
Yarin Gal and Zoubin Ghahramani. Dropout Bayesian approximation: Representing model uncertainty in deep learning. International Conference on Machine Learning, 2016.
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yec Whye Tch. Neural processes. arXiv preprint arXiv: 1807.01622, 2018.
Zoubin Ghahramani. A history of Bayesian neural networks. In NIPS Workshop Bayesian Deep Learning. 2016.
Soumya Ghosh, Jiayu Yao, and Finale Doshi- Structured variational learning of Bayesian neural networks with horseshoe priors. In International Conference Machine Learning, pp. 1739 1748, 2018.
Alex Graves. Practical variational inference for neural networks. In Advances in Neural Information Processing Systems, 2011.
Isabelle Guyon, Steve Gunn, Asa Ben-Hur, and Gideon Dror. Result amalysis of the NIPS 2003 feature selection challenge. In Advances in Neural Information Processing Systems, pp. 545-552, 2005.
David Ha, Andrew Dai, and Quoe Le. Hypernetworks. arXiv preprint arXiv: 2016.
James Hensman. Nicolo Fusi, and Neil Lawrence. Gaussian processes for big data. arXiv preprint arXiv:1309.6835, 2013.
José Miguel Hernández Lobato and Ryan P. Adams. Probabilistic backpropagation scalable learning of Bayesian neural networks. In International Conference on Machine Learning, 2015.
José Miguel Hemández-Loobrr Yingzhen Li, Mark Rowland, Daniel Hemándezz-Loarr Thang D. Bui, and Richard Turner. Black-box a-divergence minimization. In International Conference on Machine Learning.
2016.
Geoffrey Hinton and Drew Van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In Conference on Computational Learning Theory, pp. 5-13, 1993.
Neil Houlsby, Ferene Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning. arXiv preprint 11Xv:1112.57444 2011.
Thcofanis Karaletsos, Peter Dayan, and Zoubin Ghahramani. Probabilistic meta-representatiions of neural networks. arxiv preprint arXiv: 181000001 2018.
Diederik P Kingma and Max Welling. Auto-cncoding variational Bayes. arXiv preprint arXiv: 2013.
Durk P Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameteriizationn trick. In Advances Neural Information Processing Systems, pp. 2575-2583, 2015.
Balaji Lakshminarayanan. Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, pp. 6402-6413, 2017.
Yingzhen Li, José Miguel Memnánde--Lobate and Richard F. Turner. Stochastic expectation propagation. In Advonces Neural Information Processing Systems, pp. 2323-2331, 2015.
11

---

Under review as a conference paper at ICLR 2020
Christos Louizos and Max Welling, Structured and efficient variational deep learning with matrix Gaussian posteriors. In International Conference on Machine Learning, pp. 1708-1716, 2016.
Christos Louizos and Max Welling. Multiplicatiie normalizing flows for variational Bayesian neural networks. In International Conference on Machine Learning. pp. 2218-22222 22--2222,,,1111 2017.
Christos Louizos, Xiahan Shi, Klamer Schutte, and Max Welling. Functional neural processes. arXiv preprint arXiv:1906. 08324, 2019.
David J.C. MacKay, practical Bayesian framework for backpropagation networks. Neural Computation, 1992.
David J.C. MacKay. Introduction to Gaussian processes. NATO ASI series. Series F: computer and system sciences, pp. 133-165, 1998.
Alexander G. D. G. Matthews, James Hensman, Richard E Turner, and Zoubin Ghahramani. On sparse variational methods and the Kullback-Leibler divergence between stochastic processes. In International Conference on Artificial Intelligence and Statistics, 2016.
Andrew Miller, Nick Foti, Alexander D' Amour, and Ryan Adams. Reducing reparameterization gradient variance. In Advances in Neural Information Processing Systems, pp. 3708-3718, 2017.
Radford M. Neal. Bayesian learning via stochastic dynamics. In Advances in Neural Information Processing Systems, pp- 475-482, 1993.
Radford M Neal. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary DeVito, Zeming Lin, Alhan Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop, 2017.
Joaquin Cuidenero-Candela and Carl Edward Rasmussen, unifying view of sparse approximate Gaussian process regression. The Journal of Machine Learning Research, 6(Dec):1939-1959 2005.
Joaquin Sertfonnrr-Canndlle Carl Edward Rasmussen, Fahian Sinz,, Olivier Bousquet, and Bernhard Schölkopf. Evaluating predictive uncertainty challenge. In Machine Learning Challenges. Evaluating Predictive Uncer- lainty, Visual Object Classificatoo, and Recognising Tectual Entuilment, pp. 1-27. Springer, 2006.
Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning). The MIT Press, 2005. ISBN 026218253X.
Tim Salimans and David A Knowles. Fixed-form variational posterior approximation through stochastic linear regression. Bayesian Analysis, 8(4):837-882, 2013.
Sharad Singhal and Lance Wu. Training multilayer perceptrons with the extended Kalman algorithm. In Advances in Neurul Information Processing Systems, 1989.
Kenneth O Stanley, David B D' Ambrosio, and Jason Gauci. A hypercube- encoding for evolving large-scale neural networks. Artificial life, 15522288-221, 2009.
Michalis Titsias and Miguel In International Conference on Machine Learning, pp. 1971-1979, 2014.
Doubly stochastic variational Bayes for non-conjugate inference.
Michalis K. Titsias. Variational learning of inducing variables in sparse Gaussian processes. In International Conference on Artificial Intelligence and Statistics, pp. 567-574, 2009.
Brian Trippe and Richard Turner. Overpruning in variational Baycsian neural networks. arXiv preprint arXiv: (0to::8000.2200 2018
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric Xing. Deep kemee learning. In Artificial Intelligence ond Statistics, pp. 370-378, 2016.
Anqi Wu, Sebastian Nowozin, Edward Meeds, Richard E Turner, José Miguel Hernández- Lobato, and Alexan- der L. Gaunt. Fixing variational Bayes: Deterministic variational inference for Bayesian neural networks. arXiv preprint
2018
Guodong Zhang, Shengyang Sun, David Duvenaud, and Roger Grosse. Noisy natural gradient as variational inference. arXiv preprint 172.02171,.02299, 2017.
12

---

Under review as a conference paper at ICLR 2020
A EXTRA EXPERIMENTAL RESULTS
A.1 AN EMPIRICAL EVALUATION OF VARIOUS APPROXIMATIONS FOR g(w/zR, Vx,
this experiment, we analyze the impact different approximations to the covariance matrix of q( [ 2k, Vk, x):
@(wlz,, Vt, = NWw; A (4) pu, ( + A (4) A.AA)())))
If we use the exact, fully correlated Gaussian distribution above, it is necessary to sample from this distribution to evaluate the lower bound. This step costs (W") where W is the number parameters in the network.
The complexity can be greatly improved by making diagonal approximation B as follows,
(irre((/za, Vv, x) = (W: A (k) "uu, diagdiagBia (B) + A E,, T.(K)).
Sampling from this distribution can be done in O(W 2) where M is the number of pseudo-points. This can be further approximated by assuming diagonal covariance matrix,
qding (w] Zk, Vx, = NNw: A (k) Nu. diag (R) + A (k) E ATA))))))
The variational bound can then be evaluated by drawing samples from daing as in the above approximation, or by drawing activity samples by employing the local reparameterization trick (Kingma et al. 2015).
We evaluate the performance of using the exact and approximate conditional distributions above in range of toy regression and classification, and show representative results in figs. / and [ We note that the diagonal approximation is fast and gives qualitatively similar performance compared to more structured approximation or the exact case, in both cases where there is a singla GP for all weights in the network and there is multiple GPs, one for each weight layer in the network.
A.2 ROBUSTNESS IN VARIOUS DATA REGIMES FOR A TOY REGRESSION PROBLEM
In this experiment, evaluate the qualitative performance of various methods, including MFVI, NN-Metarep, GP-Metarep and Ge-Metarep with local, input-dependent kemel, on toy regression problem, in different data regimes. In particular, we considers 10, 20, and 50 training points respectively, and plot the predictions in fig. P GP- GP-metarep demonstrates consistent performance across all data regimes, comparable to that of NN-metarep. The input-dependent kernel helps the performance further in the out-of-distribution area.
A.3 EFFECT OF INPUT-DEPENDENT KERNEL AND APPLICATIONS TO FORWARD TRANSFER/FAST ADAPTATION/MULLTI
LEARNING
In this section, we investigate the effect of the hnput-dependent kernel by probing the learnt kernel hyper- parameters. In particular, we first train a model with an periodic ieputddependent kernel on a sinusoidal ID data set, as show in the main text. We then vary the period plyeer-arameee in the kernel whilst keeping other hyper- parameters and variational parameters fixed. The predictions for a few hyperparameters are shown in fig. [T] We note the variation/perict in the data is captured by weight modulation, governed by the input-dependent kernel. Changing the period hyperparameter affects how fast or slow the weights are changing wrt the input.
We further investigate using the proposed model for multi- task learning. In particular, the latent variable z and corresponding hyper-paranen parameters and variational parameters can be shared across different tasks whilst the meta mapping and the input-dependent kernel are private to each individual task. We first train the model on four regression tasks, each corresponds to sinusoid of particular frequency. At test time, novel test set is shown the model. The hyper-parameters of the input kemel and variational parameters corresponding to this new test set are optimised while other spermpprreeters and the latent variables are kept fixed. We evaluate the performance of the model on the novel test sets to see how the latent variables can be reused and shared across tasks to facilitate fast adaptation to new settings. The performance of the model on the tasks used for training and new tasks at test time is shown in fig. [T This result demonstrates the ability of the model trained with multiple similarly related tasks to faithfully and quickly adapt to new settings.
A.4 MNIST EXPERIMENT: COMPARISON TO DEEP KERNEL LEARNING AND FULL FIGURES
In this section, we include the full figures of the MNIST out-of distributit uncertainty experiment, as well as additional results using deep kernel learning (Wilson et al. 2016). In particular, we employ the same network architecture with the last layer being replaced by multiple independent GPs, one for each class (output dimension). As exact inference is intractable, variational inference based on inducing points is employed we used 50 inducing points for each output. The full results of all models/methods considered are shown in fig. [2 For clarify, the results of deep kernel learning and GP metarep are shown in fig. 13 GP-metarep with the input-dependent kernel shows good performance, outperforming deep kernel learning in all cases. In addition, we include the full figures for the predictive distributions representative examples in figs. 14 and T5
13

---

Under review as a conference paper at ICLR 2020
10a
10a
104
104
energy
10
Layer, Diag reparam.
10a
Layer, Diog reparam.
Layer, Diag
Layer, Diag
Diag rank
10a
Layer, Fuily
107
Layer, Fully
Global, Diag Local reparam.
Global, Diag Local reparam.
Globel Dieg
Global.
Global
10a
Global, Fully correlated
10a
Global, Fully correaaeed
10a
10a
10
10a
10
10
10
10a
10
time
Diag + Local reparam.
Diag
Diag + Low-rank
Full
Global
Diag + Local reparam.
Diag
Diag
Low-rank
Full
Layer
Figure 7: An evaluation of the covariance matrix approximations in toy regression example. Top: objective function during training vs epoch/time. Bottom: Predictions after training using one the approximations discussed in the text. Global: there is GP for all weights in the network. Layer: there are multiple GPs, one for each weight layer in the network. Best viewed colour.
A.5 ROBUSTNESS OF GP-METAREP WITH NETWORK ARCHITECTRRES
In this experiment, we compare the performance of GP-metarep for various numbers of hidden units (20, 50 and 100) and two activation functions (tanh and ReLU) on a toy regression problem. The observation noise is fixed in this experiment. We observe that the performance of the models is in general consistent across different activation functions and numbers of hidden units. We show the results fig. 16
A.6 A TOY ACTIVE LEARNING SETTING
Please see fig. 17 and the associated caption.
A.7
PROBABILITY DENSITIES WEIGHTS IN THE TOY ID REGRESSION EXPERIMENT
Please figs. 18 and 19 and the associated captions.
14

---

Under review as a conference paper at ICLR 2020
10
164
energy
endryy
Layer, Diag Local reparam.
Layer, Diag Locel reparam.
Layer, Glag
Layer, Diag
Layer, Diag
Layer,
Layer, Fully correlated
Layer. Fully correlated
Giobal, Diag Local reparam.
Global, Diag Local reparam.
Diag
Dieg
10f
Gotaal Disg + Low rank FITC
10
Global, Diag Low rank FITC
Global, Fully correaated
Global, Fully correlated
10
10a
10a
10
10
10a
10'
epoch
Diag + Local reparam.
Diag
Diag Low-rank
Full
Global
Diag Local reparam.
Diag
Diag Low-rank
Full
Layer
Figure 8: An evaluation of the covariance matrix approximations in a toy classification example. Top: objective function during training epoch/time. Bottom: Predictions after training using one the approximations discussed in the text. Global: there is one GP for all weights in the network. Layer: there are multiple GPs, one for each weight layer the network. Best viewed colour.
15

---

Under review as a conference paper at ICLR 2020
Figure 9: Performance of mean-field variational inference, NN-Metarep with variational inference and GP GP-Metarep with variational inference on tty regression problem with various number of training points. Best viewed colour.
50 data points
20 data points
10 data points
MFVI
NN-Metarep
GP-Metarep
GP- GP-Metarep local
16

---

Under review as a conference paper at ICLR 2020
T = period
-2
-3-
-3
period = 3*
-3
-2
period = T/2 2
Figure 10: We first train model with an input-dependent kernel on sinusoid data set (top) and then vary the period hyperparameter of the input-dependent kemel whilst keeping other hyperparameters and variational parametes fixed (middle and bottom). Best viewed colour.
17

---

Under review as a conference paper at ICLR 2020
Figure 11: Training on multiple related tasks and adaptation to novel tasks at test time. In this case, the latent variables (as well as weight code hyperparameters are shared across tasks while each individual has its own imputddependent kernel. test time, only the private parameters for the new task are re re-initialised and optimised. Best viewed in colour.
wwwm
wmmmi
wmw.mm
wwwm
Task 4
Task 8
m
www
www
w
Task 3
Task 7
M
Task 2
Task 6
H
Task 1
Task 5
TRAINING
TESTING
200 data points 50 data points
20 data points
81

---

Under review as a conference paper at ICLR 2020
MAP
metaGP local
mnist
uniform
MFVI
dkl
kmnist fmnist
gauss
metaGP
1.0
0.8
0.6
cdf
0.4
0.2
0.0
0.0
0.5
1.0
1.5
2.0
entropy /nat
Figure 12: Full results of the MNIST out-of-distribution uncertainty experiment. Best viewed in colour.
mnist kmnist fmnist
uniform
metaGP
dkl
gauss
metaGP + local
1.0
0.8
0.6
cdr
0.4
0.2
0.0
0.0
0.5
1.0
1.5
2.0
entropy /nat
Figure 13: For clarify, we show subset of the results of the MNIST out-of-distribution uncertainty experiment in fig. [2] for GP-metarep and deep kernel learning. Best viewed in colour.
19

---

Under review as a conference paper at ICLR 2020
0
/
2
6
1.0
0.5
1.0
NNV
0.0
IIMUU tocal mnnaaor
0.5
0.0
1.0
0.5
0.0
1.0
9 0
9
9
9 0
9 0
9
Figure 14: Predictive distribution for representative MNIST test examples by various methods. Best viewed colour.
b.
1.0
wA
0.5
0.0
0.0
Patuor hotal medar
0.5
0.0
1.0
0.5
0.0
1.0
9
0
9
9 0
9
0
Figure 15: Predictive distribution for representative KMNIST test examples by various methods. Best viewed in colour.
20

---

Under review as a conference paper at ICLR 2020
ReLU
Tanh
20 units
50 units
100 units
Figure 16: Performance of GP-metarep on a toy regression problem, with various numbers of hidden units and different activation functions. Best viewed in colour.
21

---

Under review as a conference paper at ICLR 2020
Step
Step
Step 2
Step 3
BNN-MAP
12
88
3.8
9.6
04
BNN-MetaGG BNN-MEVT
1330
1225
1375
120
1310
2.370
113
3.365
105
1300
3.100
1.00
08
0.5
64
94
Figure 17: Active learning with BNNs using maximum posteriori estimation [BNN-MAP], mean- field Gaussian variational inference [BNN-MFVI] and a meta-GP hierarchical prior [BNN-MetaGP] toy multi-class classification task. For each plot, the filled circle markers the current training points, with different colours illustrating different classes. The shaded crosses are the examples in the pool set, of which wish pick and evaluate be included in the training set. The unfilled circle markers are the examples from the pool set selected at step. The objective function for selecting points from the pool set the entropy of the predictive probability. Best viewed in colour.
22

---

Under review as a conference paper at ICLR 2020
15
51
Figure 18: Left: Training data and predictions using a Bayesian neural network with mean-field variational Gaussian approximation The network has one hidden layer of 50 rectified linear units. hence there are input weights and 50 output weights. Right: the marginal variational approximations over the weights after training. Most hidden units are pruned out after training, as observed in Trippe & Turner 2018). In this case, there are 12 active hidden units. We include the histogram of the weights in each layer in fig. 3
15 23
12 25 6O 15
15
60
23
25 O4
21
12
114
Input weights
Output weights
23

---

Under review as a conference paper at ICLR 2020
Figure 19: Middle: Training data and predictions using Bayesian neural network with an input- dependen meta GP prior. Similar to fig, [8, the network has one hidden layer of 50 units. The distributions of the weights the network, given an input, are included. Note that, for each weight, we first sample the latent variable 2. and then propagate this sample through the meta GP mapping obtain distribution of each weight. In each subplot, we plot 10 distributions corresponding to 10 different samples. On the left figures, as part of the input to the GP is far away from the training inputs, the weights become uncertain which leads to high uncertainty in the prediction. On the contrary, around the training points, the GP outputs are more certain and more diverse Note that, the marginal of each weight is an infinite mixture of Gaussian densities. Best viewed in colour.
AAA
V V
Inpus weights
Output wnights
Ontput wrights
Input wrights
24


Reviews:
[{"review_id": "rkxRezOpFH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an improvement over Probabilistic meta-representations of neural networks by replacing the NN parameterization of the network weights given latent variables by a probabilistic distribution whose mean is distributed by GP. Inference of the induced hierarchical model is achieved by variational inference and various\napproximations when needed.\n\nComments:\n\n1. The authors claim that the proposed method aims to increase the robustness in the small data settings\nand improve its out-of-sample uncertainty estimates. The second part is well justified. \nCould the author elaborate further on the high level intuition behind the first part?\n\n2. What exactly is the gain obtained by replacing a NN parameterization with a GP parameterization?\nIt seems like the proposed method gains the ability to model uncertainty, but potentially incurs performance trade-off \nfrom a series of approximation. \n\n3. Following from comment #2, I am a bit surprised by the lack of comparison against the work of \nKaraletsos although this work was built on top of it. I am interested to understand the mentioned\ntrade-off in practice.\n\n4. Regarding the practical consideration in Eq.(11), I think it kind of defeats the purpose of setting up\nthe latent variables so that \"weights in a layer and across layers are explicitly correlated in the \nmodelling stage\" (Section 2.2). Do all experiments presented in the paper employ this practical \nconsideration?\n\n5. For the text before Eq. (8) should the inducing inputs be xu instead of zu? It seems like a systematic\ntypo here because in the formulation for the lower bound it becomes p(u|xu) again instead of p(u|zu)\n\n6. Is Kuu in Eq.(9) computed by taking Kronecker product of K_in, K_out and K(xu,xu) or just K(xu,xu).\nI am under the impression that it is the former, in which case taking the inversion is costly (cubic in the number\nof latent variables). This would not permit large NN anyway, which kind of defeats the purpose. It explains the choice \nof small architecture in your experiments as well. \n\nOverall comment:\n\nI think the paper presents an interesting idea but I have questions regarding its practical significance as highlighted in my specific comments above. I hope the authors would clarify these so I can converge on a final rating.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1ly24CaKr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**: This paper proposes a hierarchical Bayesian approach to hyper-networks by placing a Gaussian process prior over the latent representation for each weight. A stochastic variational inference scheme is then proposed to infer the posterior over both the Gaussian process and the weights themselves. Experiments are performed on toy regression, classification, (edit: post rebuttal) and transfer learning tasks, as well as an uncertainty quantification experiment on MNIST.\n\npost rebuttal (noticed recently): Many apologies for updating the review the day before the deadline; however, I recently remembered that Kronecker inference is often used in variational methods - particularly within the vein of literature of deep kernel learning. Indeed, structure exploiting SVI was proposed in Stochastic Variational Deep Kernel Learning, https://arxiv.org/pdf/1611.00336.pdf, and this method is currently the default in Gpytorch: https://github.com/cornellius-gp/gpytorch/tree/master/examples/08_Deep_Kernel_Learning .\nFurthermore, Kronecker inference for non-Gaussian likelihoods for Laplace approximations was proposed back in 2015: http://proceedings.mlr.press/v37/flaxman15.pdf. \nI am not updating my score because it would be unfair; however, the record should be set somewhat straight here.\n\npost rebuttal: Thank you for the many clarifications and detailed responses. I'm now satisfied with their many changes and and tend to accept this paper despite the experimental results being somewhat limited. I would really encourage the authors to fix the color schemes (please less black and more brighter colors) on their decision boundaries plots however. \n\n**tldr**: While I appreciate the concept of this paper, I tend to reject this paper because I find the experimental results to be on too small scale of datasets. Specifically, I would like to see either a larger scale problem being solved with this kind of approach or a tough to model applied problem that is solved with this approach.\n\n**Originality**: As far as I can tell, this seems to be a novel approach to hyper-networks. Neural processes (Garnelo et al; 2018) propose a somewhat similar approach to training – with a latent process over some stored weight space. However, even that is quite distinct from the method proposed in this paper, and I tend to prefer this approach. \n\n**Quality**: I really appreciate the merging of neural network and Gaussian process methods; however, tragically, I do wonder if the proposed approach combines the worst of both worlds – the necessity of architecture search for neural networks with the choice of kernel function (as illustrated in Figure 5). \nIf the method is truly kernel dependent, is it also architecture dependent? That is, is it robust to different settings of nonlinearities and depths?\n\nActive learning experiment: While I appreciate the comparison here, it seems like here standard HMC should be trainable over well-designed priors on these architectures. So why not include a comparison instead of just MFVI?\n\n**Significance**: Unfortunately, I think that the experiments section is just a bit too limited to warrant acceptance right now. This is despite the fact that I really do appreciate the thoroughness and thoughtfulness of the experiments as they are. \n\nSpecifically, in Section 6.2 why is the metaGP prior only applied to the last layer of the network? If as I suspect, it is due to the complexity and difficulty of inference, that makes the method doubly tough to use in practice. With that being said, to only have experiments on the last layer implies that one should compare to Bayesian logistic regression and linear regression on the last layer of neural networks (e.g Perrone et al, 2018 and Riquelme et al, 2018). Experiments with other methods that combine Gaussian processes with representations on the final layer (e.g. Wilson et al, 2015) are also probably worth running. \n\nFigure 4 is a very well-done experiment, if a bit tough to read. I’d suggest that the out of distribution examples get their own figure, with the in distribution examples going into the appendix. I’d also suggest computing the expected calibration error (Naeini et al, 2015) for in and out of distribution examples on the test sets for both MNIST and K-MNIST in order to have quantitative results on the entire test set. \n\nTo recommend acceptance, I’d really have to see experiments on either a CIFAR sized dataset for classification or a larger scale regression experiment. A larger dataset on either transfer learning (after all you do have a meta-representation over functions that the NN can learn), a larger active learning experiment, or semi-supervised learning. \n\n**Clarity**: Overall, the paper is well-written and mostly easy to follow. The meat of the paper is found in Section 4, which I found a bit difficult to follow. \n\n(edit: post rebuttal. This concern is somewhat resolved due to the field not being well developed in this area, although it is a useful place to possibly extend the method in the future.) My primary concern here is that the prior ends up becoming Kronecker structured (after Eq. 7), so it isn’t clear to my why dense matrices and dense variational bounds have to be derived in this setting. Can one not follow the lead of the Gaussian process literature (e.g. Saatci 2012, Wilson & Nickisch, 2015) to exploit the Kronecker structure here to make computation of the log likelihoods fast?\n(edit: post rebuttal. This concern is somewhat resolved.) As a result, it’s not immediately clear to me why a diagonal approximation (Eq. 10) is even necessary? \nFurthermore, this may be a setting where iterative methods (e.g conjugate gradients and Lanczos decompositions as in Pleiss et al, 2018) for the predictive means and variances may shine and be fast.\nI do agree that the approximation in Figure 2 does seem to be relatively accurate, although I would ask the authors to compute a relative error for that plot if possible. Additionally, what is the strange high off diagonal correlations in the marginal covariances?\n\n(edit: post rebuttal. Thank you for the clarifications here.) Finally, I was a bit confused by the effect of adding the input dependent kernel in Section 3; this seems to make the weights much more complicated to model – now each data point has its own set of weights and therefore, we might have to store considerably more weight matrices over time. Could the authors perform a set of experiments showing the necessity of this kernel matrix in the rebuttal?\n\n**Minor Comments**: \n-\tAbove Eq. 9, “splitted” should be split.\n-\tFigure 3: could the data points be plotted in a brighter fashion? On a dark background, they are quick tough to see. Additionally, what is the difference between the two levels of classification plots?\n\n\nReferences:\n\nNaeini, et al. Obtaining Well Calibrated Probabilities by Bayesian Binning, AAAI, 2015. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4410090/\n\nPerrone, V, et al. Scalable Hyperparameter Transfer Learning, NeurIPS, 2018. http://papers.nips.cc/paper/7917-scalable-hyperparameter-transfer-learning\n\nPleiss, G, et al. Constant Time Predictive Distributions for Gaussian Processes, ICML, 2018. https://arxiv.org/abs/1803.06058\n\nRiquelme, C, Tucker, G, Snoek, J. Deep Bayesian Bandits Showdown, ICLR, 2018. https://arxiv.org/abs/1802.09127\n\nSaatci, Y. Scalable Inference for Structured Gaussian process models, PhD Thesis, U. of Cambridge, 2011. http://mlg.eng.cam.ac.uk/pub/pdf/Saa11.pdf\n\nWilson, AG and Nickisch, H. Kernel Interpolation for Scalable Structured Gaussian Processes, ICML, 2015. http://proceedings.mlr.press/v37/wilson15.pdf\n\nWilson, AG, et al. Deep Kernel Learning, AISTATS, 2015. https://arxiv.org/abs/1511.02222\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1xym1CyqS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents two models extended from a meta-presentation of neural networks (Karaletsos et al. 2018) in which neural network weights are constructed hierarchically from latent variables associated with each layer. The mappings from latent space to weight space are used Gaussian Processes instead of neural networks.\n\nThe proposed models include (1) MetaGP which directly replaces neural network assumption by Gaussian process prior and (2) MetaGP with contextual information which further takes into account input information via performing the multiplication between the kernel function over latent space and kernel function over input space.\n\nVariational inference is followed by the pseudo-inducing point approach.\n\nExperiments are conducted in both toy data sets and benchmark data sets, demonstrating several points i.e. uncertainty quantification, inductive bias. \n\nPros:\nThe paper is clearly written.\nIntroducing inductive bias or functional regularization for neural network\nInteresting capability of measuring the uncertainty of out-of-sample data. This can be one of the reasons that the method performed well in active learning.\n\nCons:\nThe approach is incremental or not-so-novel in terms of meta-representation for neural networks.\n\nComments and questions:\nThe prior distribution for latent variable $z$ is not specified. I assume the prior is independent Gaussian.\n$z$ is unknown beforehand. How are inducing locations for latent variable $z$ initialized?\n Do you think that there is a connection between contextual metaGP and residual nets? Can skip connections from the input layer to certain layers be considered to be similar to the idea of incorporate input kernel in the paper?\nCan you comment on the convergence of the estimation of the last term in the variational bound? The MCMC takes two stages of stochasticity: (1) sample $z_k$ and $V_k$ and (2) then estimate $F_k$ using reparameterization. This can make the convergence slow (https://arxiv.org/abs/1709.06181). \nMinor: a missing period in Sec. 6.3 “quadratic kernel In this example”", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose an approach to Bayesian deep learning, by representing neural network weights as latent variables mapped through a Kronecker factored Gaussian process. The ideas have merit and are well-motivated. Reviewers were primarily concerned by the experimental validation, and lack of discussion and comparisons with related work. After the rebuttal, reviewers still expressed concern regarding both points, with no reviewer championing the work.

One reviewer writes: "I have read the authors' rebuttal. I still have reservation regarding the gain of a GP over an NN in my original review and I do not think the authors have addressed this very convincingly -- while I agree that in general, sparse GP can match the performance of GP with a sufficiently large number of inducing inputs, the proposed method also incurs extra approximations so arguing for the advantage of the proposed method in term of the accurate approximate inference of sparse GP seems problematic."

Another reviewer points out that the comment in the author rebuttal about Kronecker factored methods (Saatci, 2011) for non-Gaussian likelihoods and with variational inference being an open question is not accurate: SV-DKL (https://arxiv.org/abs/1611.00336) and other approaches (http://proceedings.mlr.press/v37/flaxman15.pdf) were specifically designed to address this question, and are implemented in popular packages. Moreover, there is highly relevant additional work on latent variable representations for neural network weights, inducing priors on p(w) through p(z), which is not discussed or compared against (https://arxiv.org/abs/1811.07006, https://arxiv.org/abs/1907.07504). The revision only includes a minor consideration of DKL in the appendix. 

While the ideas in the paper are promising, and the generally thoughtful exchanges were appreciated, there is clearly related work that should be discussed in the main text, with appropriate comparisons. With reviewers expressing additional reservations after rebuttal, and the lack of a clear champion, the paper would benefit from significant revisions in these directions. 

Note: In the text, it says:
"However, obtaining p(w|D) and p(D) exactly is intractable when N is large or when the network is large and as such, approximation methods are often required."
One cannot exactly obtain p(D), or the predictive distribution, regardless of N or the size of the network; exact inference is intractable because the relevant integrals cannot be expressed in closed form, since the parameters are mapped through non-linearities, in addition to typically non-Gaussian likelihoods.
Decision: Reject