Deep Kernel Processes
Authors: Laurence Aitchison
Adam X. Yang
Sebastian W. Ober
Venue:  (2021)
Keywords: Gaussian process
doubly stochastic variational inference
variational Inference
Bayesian Inference

Abstract: We define deep kernel processes in which positive definite Gram matrices are progressively transformed by nonlinear kernel functions and by sampling from (inverse) Wishart distributions. Remarkably, we find that deep Gaussian processes (DGPs), Bayesian neural networks (BNNs), infinite BNNs, and infinite BNNs with bottlenecks can all be written as deep kernel processes. For DGPs the equivalence arises because the Gram matrix formed by the inner product of features is Wishart distributed, and as we show, standard isotropic kernels can be written entirely in terms of this Gram matrix (we do not need knowledge of the underlying features). We define a tractable deep kernel process, the deep inverse Wishart process and give a doubly-stochastic inducing-point variational inference scheme that operates on the Gram matrices, not on the features (as in DGPs). We show that the deep inverse Wishart process gives superior performance to DGPs and infinite BNN on standard fully-connected baselines.

Under review as a conference paper at ICLR 2021
DEEP KERNEL PROCESSES
Anonymous authors Paper under double-blind review
ABSTRACT
We define deep kernel processes in which positive definite Gram matrices are pro- gressively transformed by nonlinear kernel functions and by sampling from (in- verse) Wishart distributions. Remarkably, we find that deep Gaussian processes (DGPs), Bayesian neural networks (BNNs), infinite BNNs, and infinite BNNs with bottlenecks can all be written as deep kernel processes. For DGPs the equiv- alence arises because the Gram matrix formed by the inner product of features is Wishart distributed, and entirely in terms of this Gram matrix - we do not need knowledge the under- lying features. We define tractable deep kernel process, the deep inverse Wishart process, and give doubly-stochasee that operates the Gram matrices, not the features, in DGPs. We show that the deep inverse Wishart process gives superior performance to DGPs and infinite BNNs on standard fully-connected baselines.
show, standard isotropic kernels can be written
inducing-poiin variational inference scheme
INTRODUCTION
The deep learning revolution has shown us that effective performance on difficult tasks such as image classification (Krizhevsky et af., 2012) requires deep models with flexible lower-layers that learn task-dependent representations. Here, we consider whether these insights from the neural network literature can be applied to purely kernel-based methods. (Note that we do not consider deep Gaussian processes DGPs to be "fully kernel-based" as they use feature-based representation in intermediate layers).
Importantly. deep kernel methods (e.g. Cho & Saul 2009) already exist. In these methods, which are closely related to infinite Bayesian neural networks (Lee et al. 2017 Matthews et al., 2018 Garriga-Alonso al., 2018 Novak al., 2018), we take initial kernel (usually the dot product of the input features) and perform a series of deterministic, parameter- freee transformations to obtain an output kernel that we use in c.g. a support vector machine or Gaussian process. However, the deterministic, parameter-free nature of the transformation from input to output kernel means that they lack the capability to learn a top-layer representation, which believed be crucial for the effectiveness of deep methods (Aitchison) 2019).
To obtain the flexibility necessary to learn a task-dependent representation, we propose deep ker- nel processes (DKPs), which combine nonlinear transformations of the kernel, in Cho & Saul (2009) with a flexible learned representation by exploiting a Wishart or inverse Wishart process (Dawid 1981/Shal et al.. 2014). We find that models ranging from DGPs (Damianou & Lawrence 2013 Salimbeni & Deisenroth
2017) to Bayesian neural networks (BNNs: Blundell et al.) 2015 App. C.T). infinite BNNs (App. and infinite BNNs with bottlenecks (App.(C.> can be written as DKPs (i.c. only with kernel/Gram matrices, without needing features or weights). Practically, we find that the deep inverse Wishart process (DIWP), admits convenient forms for variational approx- imate posteriors, and we give novel scheme for doubly-stochastic variational inference (DSVI) with inducing points purely in the kernel domain (as opposed to (allibeni & Deisenroth 2017 who described DSVI for standard feature-based DGPs), and demonstrate improved performance with carefully matched models on fully-connected benchmark datasets.
2 BACKGROUND
We briefly revise Wishart and inverse Wishart distributions. The Wishart distribution is a general- ization of the gamma distribution that is defined over positive semidefinite matrices. Suppose that

---

Under review as a conference paper at ICLR 2021
Layer 1
Layer 2
Output Layer
X
K
G1
K2
22
Kg
Y
X
G
22
F3
Y
Generative models for two layer (L 2) deep GPs. (Top) Generative model for deep GP, with a kernel that depends on the Gram matrix, and with Gaussian-distriuuted features. (Bottom) Integrating out the features, the Gram matrices become Wishart distributed.
Figure/ 1:
we have collection of P-dimensional random variables x; with i {l...... N) such that then,
X iid
(0, V),
Σ = (VV.N)
(1)
has Wishart distribution with scale matrix V and N degrees of freedom. When N P 1, the density is,
1 S|NN exp (- Tr Tr (V-)))))
W (S; V = 2NP INPIVIP (2)
(2)
where r is the multivariate gamma function. Further, the inverse, S-1 has inverse Wishart distribu- tion, W-1 (V-1,N) The inverse Wishart is defined only for N P and also has closed-form density. Finally, we note that the Wishart distribution has mean NV while the inverse Wishart has mean V-1/(N- p 1) (for N > P 1).
DEEP KERNEL PROCESSES
We define kernel process to be set of distributions over positive definite matrices of different sizes, that are consistent under marginalisation (Dawid, 1981 Shah et al. (014). The two most common kernel processes the Wishart process and inverse Wishart process, which we write in slightly unusual form to ensure their expectation is K. We take G and G' to be finite dimensional marginals of underlying Wishart and inverse Wishart process,
G W (K NN, G W (K"/KN
G' W (ōK ,8 (P + +1)),
(3a)
G w-~ 8 (P" +1)),
(3b)
and where explicitly give the consistent marginal distributions over K* G" and G which P x P principal submatrices of the P x P matrices K, G and G' dropping the same rows and columns. In the inverse- -isshat distribution, 5 is positive parameter that be understood as controlling the degree of variability, with larger values for implying smaller variability in G'.
We define deep kernel process by analogy with DGP, composition of kernel processes, and show in App. that under sensible assumptions any such composition is itself a kernel process.
3.1 DGPs WITH ISOTROPIC KERNELS ARE DEEP WISHART PROCESSES
We consider deep GPs the form (Fig. top) with X E PxNN
XXT
for (,, otherwise,
K,
(4a)
K (G--1)
P (F) = INN .' (ES; O,)
(4b)
G No F.F.
(4c)
Here, F; E PxN, are the N₂ hidden features in layer E; À indexes hidden features so f is a single column F;, representing the value of the Xth feature for all training inputs. Note that K(-) is
Note that we leave the question of the full Kolmogorov extension theorem Kolmogorov 1933 for matrices to future work: for our purposes, it is sufficient to work with very large but ultimately finite input spaces as in practice, the input vectors are represented by elements of the finite set of 32-bit or 64-bit floating-poin numbers (Sterbenz 1974).
2

---

Under review as a conference paper at ICLR 2021
function that takes Gram matrix and returns kernel matrix; whereas K is a (possibly random) variable representing a kernel matrix. Note, we have restricted ourselves to kernels, that can be written as functions of the Gram matrix, Gr, and do not require the full set of activations, Fy. As we describe later, this is not too restrictive, as includes amongst others all isotropic kernels (i.e. those that can be written Note that we have a number of choices as to how to initialize the kernel in Eq. (4a). The current choice just uses a linear dot-product kernel, rather than immediately applying the kernel function K. This is both to ensure exact equivalence with infinite NNs with bottlenecks (App. CB and also to highlight an interesting interpretation of this layer as Bayesian inference over generalised lengthscale hyperparameters in the squared-exponential kernel (App. B e.g. Lalchand & Rasmussen 2020).
function of the distance between points Williams & Rasmussen 2006).
For DGP regression, the outputs, Y, are most commonly given by a likelihood that can be written in terms of the output features, F 1++ For instance, for regression, the distribution of the Äth output feature column could be
P ((yx(L41)) =N (yx:fx+t),
(5)
but our methods can be used with many other forms for the likelihood, including e.g. classification.
The generative process for the Gram matrices, GG, consists of generating samples from a Gaussian distribution (Eq. [Ab). and taking their product with themselves transposed (Eq. [4c). This exactly matches the generative process for Wishart distribution [). so we can write the Gram matrices, G2 directly in terms the kernel, without needing to sample features (Fig. 1 bottoom)
xxx) .Ni..
P (G = W
(6a)
Ni No
P G W (K (G,-1) /N,, No),
for E ( L},
(6b)
((F++1/)) -I (rk+) (,
(6c)
Except at the output, the model is phrased entirely in terms of nositive-definite kernels and Gram ma- trices, and is consistent under marginalisation (assuming a valid kernel) and is thus DKP. At a high level, the model can be understood alternatively sampling Gram matrix (introducing flexibility in the representation), and nonlinearly transforming the Gram matrix using kernel (Fig. 2
This highlights a particularly simple interpretation of the DKP as an autoregressive process. standard autoregressive process, we might propagate the current vector, x,, through a deterministic function, f(x.), and add zero-mean Gaussian noise, &.
X++1 f (x+) + +
such that
E X
[x]] = f (x:).
By analogy, the next Gram matrix has expectation centered on a deterministic transformation of the previous Gram matrix,
E G = -
(8)
so G2 can be written this expectation plus zero-mean random variable, E,, that can be inter- preted as noise,
G = K (G.1))
Note that E, is not in general positive definite, and may not have an analytically tractable distribu- tion. This noise decreases as N increases,
v -
1 N₂ (KT K2 (G. GG-1) (G-1))
(10)
Notably, N₂ tends to infinity, the Wishart samples converge on their expectation, and the noise disappears, leaving us with a series of deterministic transformations of the Gram matrix. Therefore, we can understand deep kernel process as alternatively adding "noise" the kernel by sampling e.g. a Wishart or inverse Wishart distribution (G2 and G3 in Fig. 2 and computing nonlinear transformation of the kernel (K (G2) and (G3)) in Fig. 2
Remember that we restricted to kernels that can be written function of the Gram matrix,
K. K (G+) = Keatures (Fe).
K l (F 1,11 F
(11)
3

---

Under review as a conference paper at ICLR 2021
K(G1)
22
K(G2)
G3
K(G3)
input index
Wishart
100
200
Inverse Wishart
0
input index
100
200
-1
0
100
200
100
200 0
100
200
100
200
100 input index
200
input index
input index
Input index
input index
Figure 2: Visualisations of single prior sample of the kernels and Gram matrices as they pass through the network. We use ID, equally spaced inputs with a squared exponential kernel. As we transition K(G{-1) GG, we add "noise" by sampling from a Wishart (top) or an inverse Wishart (bottom). As we transition from G₂ to K(Gç), we deterministically transform the Gram matrix using a squared-exponentee kernel.
where K (eeatures (-) takes matrix of features, Fz, and returns the kernel matrix, K2, and k is the usual kernel, which takes two feature vectors (rows of F,) and returns an element of the kernel matrix. This does not include all possible kernels because it is not possible to recover the features from the Gram matrix. particular, the Gram matrix is invariant to unitary transformations of the features: the Gram matrix is the same for F, and F` UF, where U is a unitary matrix, such that UU" I.
G: = Nr F N F.U.U = F.F.
(12)
Superficially, this might seem very limiting - leaving us only with dot-product kernels (Williams Rasmussen 2006) such as,
k(f, = f. +
(13)
However, in reality, far broader range of kernels fit within this class. Importantly, isotropic radial basis function kernels including the squared exponential and Matern depend only on the squared distance between points, It (Williams & Rasmussen 2006)
kf, f f') = k (Rt),
R= = p
(14)
These kernels can be written as a function of G, because the matrix squared distances, R, can be computed from G,
1 Rt N. Σi (FA P/A)= = N. IN ((FA)2-200 -2:))-2FF533 (13) G² -2C' + Gj
(15)
VARIATIONAL INFERENCE IN DEEP KERNEL PROCESSES
key part of the motivation for developing deep kernel processes was that the posteriors over weights in a BNN or over features in a deep GP are extremely complex and multimodal, with large number of symmetries that are not captured by standard approximate posteriors (MacKay) 1992 Moore 2016] Pourzanjani et al., 2017). For instance, in the Appendix we show that there are permutation symmetries in the prior and posteriors over weights in BNNs (App. D.I) and rotational symmetries in the prior and posterior over features in deep GPs with isotropic kernels (App. D.2). The inability to capture these symmetries standard variational posteriors may introduce biases in the parameters inferred by variational inference, because the variational bound is not uniformly tight across the state-space (Turner Sahani 2011). Intuitively, these symmetries arise in DGPs with isotropic kernels because the features at the next layer depend only the kemel matrix at
4

---

Under review as a conference paper at ICLR 2021
the previous layer, and this kernel is invariant to unitary transformations of the features (Eq. 12 As such, we can sidestep these complex posterior symmetries by working directly with the Gram matrices the random variables for variational inference.
We show that DGPs (Sec. 3.I and infinite NNs with bottlenecks (App. C are deep Wishart pro- cesses, natural approach would be to define an approximate posterior over the Gram matrices in the deep Wishart process. However, this turns out to be difficult, predominantly because the approx- imate posterior We would like to use, the non-central Wishart (App. [E) has a probability density function that is prohibitively costly and complex to evaluate in the inner loop of a deep learning model (Koev& Edelman 2006). Instead, we consider an inverse Wishart process prior, for which the inverse Wishart itself makes a good choice of approximate posterior.
4.1 THE DEEP INVERSE WISHART PROCESSES
By analogy with Eq.
66. our deep inverse Wishart processes (DIWPs) are given by (with G1 No + for l e ......L..
(22) w (6,1, + +1,
(16a)
G W- (Gg (G--1), P
D}.
(16b)
((F GL) IN ' ( ,,K,))
(16c)
remember that X PxN. G. E PxP and F. P Note that at the input layer, Ko No XXX" may be singular if there are more datapoints than features. Instead of attempting
to use singular Wishart distributions over G1, which would be complex and difficult to work with (Bodnar& & Okhrin 2008 [Bodnar et al. 2016), we instead define an approximate posterior over the full-rank No No matrix, $2, and use G1 No XΩX E PxPP.
Critically, the distributions in Eq. (16 are consistent under marginalisation as long as 6, is held constant (Dawid 1981), with P taken to be the number input points, or equivalently the size of K Further, the deep inverse Wishart process retains the interpretation as deterministic transformation of the kernel plus noise because the expectation is,
S.K (GL-1) GG (P (P+1+6)) - (P = K (G-11).
(17)
The resulting inverse Wishart process does not have a direct interpretation as e.g. deep GP, but does have more appealing properties for variational inference, as it is always full-rank and allows independent control over the approximate posterior mean and variance. Finally, it is important to note that Wishart and inverse Wishart distributions do not differ as much as one might expect; the standard Wishart and standard inverse Wishart distributions have isotropic distributions over the eigenvectors they only differ in terms of their distributions over eigenvalues, and these are often quite similar, especially if we consider a Wishart model with ResNet-like structure (App.
4.2 AN APPROXIMATE POSTERIOR FOR THE DEEP INVERSE WISHART PROCESS
Choosing an appropriate and effective form for variational approximate posteriors is usually a dif- ficult research problem. Here, we take inspiration from Ober & Aitchison (2020) by exploiting the fact that the inverse-Wishart distribution is the conjugate prior for the covariance matrix of a multivariate Gaussian. In particular, if we consider an inverse-Wishart prior over Σ E PxP with mean g. which forms the covariance of Gaussian-distribure matrix, V E PxP, consisting of columns Vx.
P ( = W- (E; P+1+
(18a)
P PVVE) = Nv ( (,
(18b)
P (EIV) = W- (Σ: + VVT, " P+1+8+NN)
(18c)
Inspired by this exact posterior that is available in simple models, we choose the approximate pos- terior in model to be,
Q (I2) = W- (2; V,VI +m (No + 1))
(19a)
Q (G. G.-1) = w- (G. (Ge-1) + V.V. 6r + (P 1)),
(19b)
Q (Fr+i/)L.) = - N (f(+): ExArAA, Ex); where Σx (K- (G²) Ax)
(19c)
5

---

Under review as a conference paper at ICLR 2021
and where 11 is learned No No matrix, (Vel-=2 are P x P learned matrices and {re 2=1 learned non non-negative real numbers. For more details about the input layer, see App. [F] At the output layer, take inspiration from the global inducing approximate posterior for DGPs from Ober & Aitchison 2020), with learned parameters being vectors, VA, and positive definite matrices, Ax (see App. G
In summary, the prior has parameters {5=) (which also appears the approximate posterior), and the posterior has parameters (Velé=1 and {relé=1 for the inverse- Wishart hidden layers, and No+1 and {Ax)x=r Na-1 at the output. In all our experiments, optimize all five parameters (foe, re 121) and ([vx,Ax)x) 1 and in addition, for inducing-point methods, we also opti- mize single set of "global" inducing inputs, X; E R.x,N, which are defined only at the input layer.
(vx)X=t)
4.3 DOUBLY STOCHASTIC INDUCING-POINT VARIATIONAL INFERENCE IN DEEP INVERSE WISHART PROCESSES
nigh-dimensional problems, we take inspiration from the DGP litera- ture (Salimbeni & Deisenroth, 2017) by considering doubly-stochastic inducing-pons poinng deep inverse Wishart processes. We begin by decomposing all variables into inducing and training (or test) points
For efficient inference
X, RR Nox
X
F FL F
G G
2
X
FL+1 =
G
(20)
where c.g. G is P and G is x P. where P; is the number of inducing points, and Pt is the number of testing/training points. Note that Ω does not decompose as is No x No. The full ELBO including latent variables for all the inducing and training points is,
P (2, (Ge) FL+I X) L = E log P (Y + log Q (S, (Gi) FL+1 (X)
(21)
where the expectation is taken over Q (2, (G)) FLt1 X). The prior is given by combining all terms in Eq. (I6) for both inducing and test/train inputs,
P (S2, (Ge (22,)
(22) [I' 2 (Ge/Ge-) P (FL+i|GL),
(22)
where the X-dependence enters on right because G: No XXXX" Taking inspiration from Salimbeni & Deisenroth (2017), the full approximate posterior is the product of an approximate posterior over inducing points and the conditional prior for train/test points,
Q (2, (Gi) F X)
Q (2, ( FX, ((G (G F (G Fft1, x). (23)
And the prior can be written in the same form,
P (2, (G Fr+i (X)
P (2, (G F (X)) (LG (G F (G F .X.
(24)
We discuss the second terms (the conditional prior) in Eq. 22 The first terms (the prior and
approximate posteriors over inducing points), are given by combining terms in Eq. (16) and Eq. 19
P (2, ( F X.) = P (S2
Th G
P P(F (
(25)
Q (2,
(G
F X,) Q (2)
II Q
Q (R(+IG)))
(26)
Substituting Eqs.
23 26
into the ELBO (Eq. 21
the conditional prior cancels and we obtain,
(S2
IT' Q (G Q (FH+)GH))
L =
log P (Y|E(+1)) + log
(27)
P (Ω
I P (GIG
P P (F

---

Under review as a conference paper at ICLR 2021
Importantly, the first term is summation across test/train datapoints, and the second term depends only on the inducing points, so as in (Salimbeni Deisenroth (2017) we can compute unbiased estimates of the expectation by taking only mimibatch of datapoints, and we never need to compute the density of the conditional prior in Eq. (28 we only need to be able to sample it.
Finally, to sample the test/training points, conditioned on the inducing points, we need to sample,
P (1G [22, P FL+1
F F x) P (F (F++ F t ITL
(G G G Ge-1). (28)
The first distribution, P (FL+1 F Gt). J. is a multivariate Gaussian, and can be evaluated using methods from the GP literature (Williams & Rasmussen 2006 Salimbeni & Deisenroth (017). The difficulties arise for the inverse Wishart terms, P [G G G G5-1). To sample this distribution, note that samples from the joint over inducing and train/test locations can be written,
G
((r)
L Lu
V L
Vi Vu
d8 + P. P.
where
ó.K (GL-1),
(29)
and where P1 the number of inducing inputs, and Pa the number of train/test inputs. Defining the Schur complements,
G
G
G (G
G
u:i Lu
L 2
Pu.
(30)
We know that G and (G. -1 G have distribution,
Eaton 1983)
Glill
G. G.-1 ~ W- (Vui, de + P. + P1+1,,,
(31a)
(G
G G G G MN (
(31b)
where MN is the matrix normal. Now, G and G can be recovered by algebraic manipulation. do not need to sample multiple of
Finally, because of the doubly stochastic form for the objective, jointly consistent samples for test points; instead, (and as in DGPs Salimbeni & Deisenroth (017) independently sample each test point (App- [ which dramatically reduces computational complexity.
We optimize using standard reparameterised variational inference (Kingma & Welling, 2013 Rezende et al., 2014) (Ober Aitchison 2020 for details on how to reparameterise samples from the Wishart, see).
5 COMPUTATIONAL COMPLEXITY
As in non-deep GPs, the complexity O(P3) for time and O(P2) for space for standard DKPs (the ((P3) time dependencies emerge e.g. because of inverses and determinants required for the inverse Wishart distributions). For DSVI, there is a P3 time and p? space term for the inducing points, because the computations for inducing points exactly the same in the non-DSVI case. As can treat each test/train point independently (App. []. the complexity for test/training points must scale linearly with P. and this term has p2 time scaling, e.g. due to the matrix products in Eq. 30) Thus, the overall complexity for DSVI O (Pp + P)) for time and O(P2 + P.P. for space which exactly the same as non-deep inducing GPs. Thus, and exactly as in non-deep inducing-GPa by using a small number of inducing points, we are able to convert a cubic dependence on the number of input points into linear dependence, which gives considerably better scaling.
Surprisingly, this is substantially better than standard DGPs, standard DGPs, we allow the ap- proximate posterior covariance for each feature to differ (Salimbeni & Deisenroth, 2017). in which case, we are in essence doing standard inducing-Gr GP inference over N hidden features, which gives complexity of O(NP3 + N P2 P.) for time and O(N P22 + P,P.P.) for space (Salimbeni & Deisen- roth 2017). is possible to improve this complexity by restricting the approximate posterior to have the same covariance for each point (but this restriction be expected to harms performance).

---

Under review as a conference paper at ICLR 2021
Table 1: Performance in terms of ELBO and predictive log-likelihood for three-layer (two hidden layer) DGP, NNGP and DIWP on UCI benchmark tasks. Errors are quoted as two standard errors in the difference between that method and the best performing method, as paired t-test. This is to account for the shared variability that arises due to the use of different test/train splits in the data (20 splits for all but protein, where 5 splits are used Gal & Ghahramani 2015) some splits are harder for all models, and some splits are easier. Because we consider these differences, errors for the best measure are implicitly included in emrors for other measures, and we cannot provide a comparable error for the best method itself.
metric
dataset
DGP
NNGP
DIWP
boston
1.30 + 0.02
-0.31 + 0.01
-0.29
concrete
-0.68 0.01
-0.40 + 0.00
-0.35
energy
0.59
1.47
1.47 = 0.00
kin8nm
0.05
-0.40
-0.33
ELBO
naval
0.16
1.38 1.882
1.44
power
0.00
0.00
0.01
protein
-1.07
-1.11
-1.09 + 0.01
wine
1.39+0.00
-1.17 0.00
-.16
yacht
0.39
1.62 +.02
1.66
boston 3.14
2.46 + 0.02
2.43
concrete
0.03
3.13 0.02
33.09
energy
0.00
-0.71
0.71 0.01
kin8nm
1.05
1.10 0.00
1.12
test LL
naval
2.80 0.12
5.74
5.73 0.21
power
-2.85
-2.33
-2.82
protein
-2.80
-2.8 0.01
- 0.01
wine
-1.18 0.03
0.96
-0.95
yacht
-2.45
-0.67
RESULTS
We began by comparing the performance of deep inverse Wishart process (DIWP) against infi- nite Bayesian neural networks (known as the neural network Gaussian process or NNGP) and DGPs. To ensure sensible comparisons against the NNGP, we used ReLU kernel in all models (Cho & Saul (009). For all models, we used three layers (two hidden layers and one output layer), with three applications of the kernel. In each case, we used a learned bias and scale for each input feature, and trained for 8000 gradient steps with the Adam optimizer with 100 inducing points, learning rate of 10-2 for the first 4000 steps and 10-3 for the final 4000 steps. evaluation, we used 100 samples from the final iteration of gradient descent, and for each training step we used 10 samples in the smaller datasets (boston, concrete, energy, wine, yacht), and sample in the larger datasets.
We found that DIWP usually gives better predictive performance and ELBOs. We expected DIWP to be better than (or the same as) the NNGP as the NNGP was special case of our DIWP (sending 6, - 00 sends the variance the inverse Wishart to zero, so the model becomes equivalent to the DIWP and NNGPs, and even past baselines on all datasets except protein (which is by far the largest). This is because we use a ReLU, rather than squared exponential kernel, as in (Salimbeni & Deisenroth, 2017). and because we used plain feedforward architecture for all models. In contrast, Salimbeni & Deisenroth (2017) found that good performance with DGPs on even UCI datasets required a complex architecture involving skip connections. Here, we used simple feedforward architectures, both to ensure fair comparison to the other models, and avoid the need for an architecture search. addition, inverse Wishart process is implicitly able to learn the network "width", dz. whereas in the DGPs, the width is fixed to be equal to the number of input features, following standard practice in the literature (c.g. Salimmeni & Deisenroth 2017).
NNGP). We found that the DGP performs poorly in comparison
Next, considered fully-connected networks for small image classification datasets (MNIST and CIFAR- 10). We used the same models as in the previous section, with the omission of learned bias and scaling of the inputs. Note that do not expect these methods perform well relative to
8

---

Under review as a conference paper at ICLR 2021
Table 2: Performance in terms of ELBO test log-likelihood and test accuracy for fully-connected three-layer (two hidden layer) DGPs, NNGP and DIWP on MNIST and CIFAR-10.
metric
dataset
DGP
NNGP
DIWP
ELBO
MNIST
-0.301 = 0.001
-0.268 0.001
-0.214 + 0.001
CIFAR-10
-1.735 + 0.002
-1.719 + 0.001
-1.659 + 0.001
test LL
MNIST
-0.130 0.001
-0.134 0.002
-0.122 0.001 -1.525 + 0.003
CIFAR-10
-..56 + 0.002
-1.539 + 0.002
test acc.
MNIST
96.5 = 0.1%
96.5 = 0.0%
96.9 = 0.0%
CIFAR-10
46.8 = 0.1%
47.4 0.1%
47.7 + 0.2%
standard methods (e.g. CNNs) for these datasets, as we are using fully-connected networks with only 100 inducing points (whereas e.g. work in the NNGP literature uses the full 60, 000 x 60, 000 covariance matrix). Nonetheless, as the architectures are carefully matched, provides another opportunity to compare the performance of DIWPs, NNGPs and DGPs. Again, we found that DIWP usually gave statistically significant but perhaps underwhelming gains in predictive performance (except for CIFAR- 10 seat-log-likelhhoeee gives very large improvements in the ELBO, with gains of 0.09 against DGPs for MNIST and 0.08 for CIFAR-10 (App. Kj. For MNIST, remember that the ELBO must be negative (because both the log-likelihood for classification and the KL-divergence term give negative contributions), so the improvement from -0 -0.301 to -0 -0.214 represents dramatic change.
where DIWP lagged by only 0.01). Importantly, DIWP
RELATED WORK
Our first contribution was the observation that DGPs with isotropic kernels can be written as deep Wishart processes the kernel depends only the Gram matrix. We then gave similar obser- vations for neural networks (App. C.T). infinite neural networks (App. C.2) and infinite network with bottlenecks (App. C.3] also see Aitchison 2019). These observations motivated us consider the deep inverse Wishart process prior, which is a novel combination of two pre-existing elements: nonlinear transformations of the kernel (e.g. Cho Saul 2009) and inverse Wishart priors over kernels (e.g. [Shah et al. 2014). Deep nonlinear transtormations of the kernel have been used in the infinite neural network literature (Lee al., [017)] Matthews et al., 2018) where they form de- terministic, parameter-fre kernels that do not have any flexibility in the lower-layers (Aitchison) 2019). Likewise, inverse-Wistee distributions have been suggested as priors over covariance matri- ces (Shah et 2014), but they considered a model without nonlinear transformations of the kernel. Surprisingly, without these nonlinear transformations, the inverse Wishart prior becomes equivalent to simply scaling the covariance with scalar random variable (App. L Shah al. 2014). Further linear (inverse) Wishart processes have been used in the financial domain to model how the volatility of asset prices changes over time (Philipov & Glickman 2006b a Asai & McAleer, 2009 Gourier- loux & Sufana| 2010)] Wilson & Ghahramani] 2010 Heaukulan & van der Wilk| Importantly, inference in these dynamical (inverse) Wishart processes often performed by assuming fixed, inte- ger degrees of freedom, and working with underlying Gaussian distributed features. This approach allows one leverage standard GP techniques (e.g. Kandemir & Hamprecht, 2015 Heaukulani & van der Wilk, 2019), but it is not possible to optimize the degrees of freedom and the posterior over these features usually has rotational symmetries (App. D.2) that are not captured by standard varia- tional posteriors. In contrast, we give novel doubly-stochastic variational inducing point inference method that operates purely on Gram matrices and thus avoids needing to capture these symmetries.
8 CONCLUSIONS
We proposed deep kernel processes which combine nonlinear transformations of the Gram matrix with sampling from matrix-variate distributions such as the inverse Wishart. We showed that DGPs, BNNs (App. C.I. infinite BNNs (App. C.2) and infinite BNNs with bottlenecks (App. C.3) are all instances of DKPs. We defined a new family of deep inverse Wishart processes, and give a novel stochss inducing point variational inference scheme that works purely in the space of Gram matrices. DIWP performed better than NNGPs and DGPs on UCI, MNIST and CIFAR-10 benchmarks.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Laurence Aitchison. Why bigger is not always better: on finite and infinite neural networks. arXiv preprint arXiv:1910.0000 2019.
Manabu Asai and Michael McAleer. The structure of dynamic correlations in multivariate stochastic volatility models. Journal of Econometrics, 150(2):182-13) 2009.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. arXiv preprint 5rXiv:/505.05424. 2015.
Taras Bodnar and Yarema Okhrin. Properties of the singular, inverse and generalized inverse parti- tioned wishart distributions. Journal of Multivariate Analysis, 99(10))2389-2444 2008.
Taras Bodnar, Stepan Mazur, and Krzysztof Podgórski. Singular inverse wishart distribution and its application to portfolio theory. Journal of Multivariate Analysis, 143:314-326, 2016.
Youngmin Cho and Lawrence K Saul. Kernel methods for deep learning. In Advances in neural information processing systems, pp. 342-350, 2009.
Andreas Damianou and Neil Lawrence. Deep gaussian processes. In Artificial Intelligence and Statistics, pp. 207-215, 2013.
A Philip Dawid. Some matrix-variate distribution theory: notational considerations and a Bayesian application. Biometrika, 68(1):265-274, 1981.
Morris L.. Eaton. Multivariate Statistics. A Vector Space Approach. -A Volume in the Wiley Series in Probability and Mathematical Statistics. Wiley-Intersciene 1983.
Yarin Gal and Zoubin Ghahramani. Dropout a Bayesian approximation: Representing model uncertainty in deep learning. 1rX.v:1506.002142, 2015.
Adrià Garriga-Aloso,, Carl Edward Rasmussen, and Laurence Aitchison. Deep convolutional net-
works shallow gaussian processes. arXiv preprint arXiv:1808.05555 2018
Pascal Germain, Francis Bach, Alexandre Lacoste, and Simon Lacoste-Julien. Pac-bayesian theory meets bayesian inference. In Advances in Neural Information Processing Systems, pp. 1884- 1892, 2016.
Christian Gourieroux and Razvan Sufana. Derivative pricing with wishart multivariate stochastic volatility. Journal of Business & Economic Statistics, 28(3):438 4518 2010.
Creighton Heaukulani and Mark van der Wilk. Scalable bayesian dynamic covariance modeling with variational wishart and inverse wishart processes. In Advances in Neural Information Processing Systems, pp. 4582-4592, 2019.
Thomas Hofmann, Bernhard Schölkopf, and Alexander Smola. Kernel methods in machine learn- ing. The annals of statistics, pp. 1171-1220, 2008
Melih Kandemir and Fred A Hamprecht. The deep feed-forward gaussian process: An effective
generalization to covariance priors. In Feature Extraction: Modern Questions and Challenges, pp. 145-159 159, 2015.
Diederik Kingma and Max Welling. arXiv:1312.61 14, 2013.
Auto-encoding variational bayes. arXiv preprint
Plamen Koev and Alan Edelman. The efficient evaluation of the hypergeometric function of matrix argument. Mathematics of Computation, 75(254):833 846, 2006.
AN Kolmogorov. Grundbegriffe der wahhscheinlichkerrreccnnug Ergebnisse der Mathematik, 1933.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.
Imagenet classification with deep convo-
lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
10

---

Under review as a conference paper at ICLR 2021
Vidhi Lalchand and Carl Edward Rasmussen. Approximate inference for fully bayesian gaussian process regression. In Symposium on Advances in Approximate Bayesian Inference, pp. 1-12. PMLR. 2020.
Jachoon Lee, Yasaman Bahri, Roman Novak, Samuel Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as gaussian processes. arXiv preprint 1rXiv:171.00065: 2017.
David JC MacKay, practical bayesian framework for backpropagation networks. Neural compu- tation, 4(3):448 472, 1992.
Alexander G de Matthews, Mark Rowland, Jiri Hron, Richard Turner, and Zoubin Ghahramani. Gaussian process behaviour in wide deep neural networks. arXiv preprint arXiv:1804.1222
2018.
David A Moore. Symmetrized variational inference. In NIPS Workshop on Advances in Approximate Bayesian Inferece, volume 4, pp. 31, 2016.
Roman Novak, Lechao Xiao, Jaehoon Lee, Yasaman Bahri, Greg Yang, Jiri Hron, Daniel A Abolafia, Jeffrey Pennington, and Jascha Sohl-Dickstein. Bayesian deep convolutional networks with many channels are gaussian processes. arXiv preprint drXiv:1810.05148 2018.
Sebastian W Ober and Laurence Aitchison. Global inducing point variational posteriors for bayesian neural networks and deep gaussian processes. arXiv preprint 2xxv:20005.08140, 2020.
Alexander Philipov and Mark E Glickman. Factor multivariate stochastic volatility via wishart processes. Econometric Reviews, 25(2-3):311-23333 2006a.
Alexander Philipov and Mark E Glickman. Multivariate stochastic volatility via wishart processes. Journal of Business & Economic Statisties, 24(3):313-323 2006b.
Arya A Pourzanjani, Richard M Jiang, and Linda R Petzold, Improving the identifiability of neural
networks for bayesian inference. In NIPS Workshop on Bayesian Deep Learning, volume 4, pp. 31, 2017.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference deep generative models. arXiv preprint 4rxiv:1401.4082. 2014.
Omar Rivasplata, Vikram Tankasali, and Csaba Szepesvari. Pac-bayes with backprop. arXiv preprint arXiv: 07738008.778,,...0 2019.
Hugh Salimbeni and Mare Deisenroth. Doubly stochastic variational inference for deep gaussian
processes. In Advances in Neural Information Processing Systems, pp. 1588-4599, 2017.
Amar Shah, Andrew Wilson, and Zoubin Ghahramani. Student-t processes alternatives to Gaus- sian processes. Artificial intelligence and statistics, pp. 877-885, 2014.
Muni S Srivastava et al. Singular wishart and multivariate beta distributions. The Annals of Statistics, 31(5):1537-1560 2003.
Pat Sterbenz. Floating- point computation. Prentice-Hall, 1974.
Richard Turner and Mancesh Sahani. Two problems with variational expectation maximisation for time-series models. In D Barber, T Cemgil, and S Chiappa (eds.), Inference and Learning in Dynamic Models. Cambridge University Press, 2011.
Harald Uhlig. On singular wishart and singular multivariate beta distributions. The Annals of Statis- tics, pp. 395-405, 1994.
Christopher Williams and Carl Edward Rasmussen. Gaussian processes for machine learning. MIT' press Cambridge, MA, 2006.
Andrew Gordon Wilson and Zoubin Ghahramani. Generalised Wishart processes. arXiv preprint 101.021 2010.
11

---

Under review as a conference paper at ICLR 2021
A DKPS ARE KERNEL PROCESSES
We define generic DKP to be (K), for random matrix K E RPP.P For instance, we could take,
K = W NK,
or ((K) = - (K, 8 (P+
(32)
where N is a positive integer and is a positive real number. A deep kernel process, D, is the composition of two (or more) underlying kernel processes, Ki and K2,
G1 (K), G2 D(K).
~ K2(G))
(33a)
(33b)
We define K", Gj and G as principle submatrices of K, G1 and G2 respectively, dropping the same rows and columns. To establish that consistent under marginalisation, we use the consis- tency under marginalisation of 11 and K2
G ~ (K'),
G K2 GGi),
(34a)
and the definition of the D as the composition of 1C and K2 (Eq.
G ~ D(K)).
(34b)
D is thus consistent under marginalisation, and hence is a kernel process.
Further, note that consider I. be deterministic distribution that gives mass only single G. In that case, K. can be thought of as a deterministic function which must satisfy corresponding consistency property,
G KCK).,
G K(KK).
(35)
and this indeed satisfied by all deterministic transformations kernels considered here. prac- tical terms, as long as G is always valid kernel, it is sufficient for the elements of Gūøz to depend only on Kij. K and K5; and for G to depend only on K,j. which is satisfied by e.g. the squared exponential kernel (Eq. [15 and by the ReLU kernel (Cho & Saul 2009).
B THE FIRST LAYER OF OUR DEEP GP AS BAYESIAN INFERENCE OVER A GENERALISED LENGTHSCALE
In our deep GP architecture, we first sample Fi PxN, from a Gaussian with covariance K₂- No XX" (Eq, 4ap. This might seem odd, as the usual deep GP involves passing the input, X e RPx,, directly to the kernel function. However, in the standard deep GP framework, the kernel (e.g. squared exponential kernel) has lengthscale hyperparameters which can be inferred using Bayesian inference. In particular,
Kparam| VNO X,, /N,x;) = exp (-2to 1 2No (x; ()) - x)))²
(36)
where Aipuram is a new squared exponential kernel that explicitly includes hyperparmeters Ω E I No No, and where x; is the ith row of X. Typically, in deep GPs, the parameter, Ω, is diagonal, and the diagonal elements correspond the inverse square of the lengthscale, 4, (i.e. Q2 1/17). However, in many cases it may be useful to have non-diagonal scaling. For instance, we could use,
S W ( I.N...
(37)
which corresponds to,
Ω = WWT.
where
Wi ~ N (0, ()) W NoXN,
(38)
Under our approach, we sample F = F, from Eq. 4b so F can be written as,
F= XW.
f, xx W.
(39)
12

---

Under review as a conference paper at ICLR 2021
Layer
Layer
Layer
Output Layer
W
W
W 3
W 4
L
1
J
X
F
2
3
F
Y
X
K
Fi
2₂
F2
K3
F3
K4
F4
Y
X
K
K2
K3
K
F1 P
Figure 3: A series of generative models for standard, finite BNN. Top. The standard model, with features, F;, and weights We (Eq. 41 Middle. Integrating out the weights, the distribution over features becomes Gaussian (Eq. [4 and we explicitly introduce the kernel, Kg, as a latent variable. Bottom. Integrating out the activations, Fy, gives deep kernel process, albeit one where the distri- butions P (K cannot be written down analytically, but where the expectation, [KKKKe-)
is known (Eq. 45
where f. is the ith row of F. Putting this into squared exponential kernel without lengthscale parameter,
10 f,, 1 f))
1 (f. f)) (f: =))"),
Noo
/No
exp
2No
2No (x; W x, W) (x, - -x,x))))
exp
2No (x: - x;) WWT (x: (xx)) x))")
exp
(x; x2) (x: x,)²)
exp
2No
kpprmm
X1 VNo" x,).
(40)
VNo
We find that a parameter-free squared exponential kernel applied to is equivalent to squared- exponential kernel with generalised lengthscale hyperparameters applied to the input.
C BNNS AS DEEP KERNEL PROCESSES
Here show that standard, finite BNNs, infinite BNNs and infinite BNNs with bottlenecks can be understood as deep kernel processes.
C.I STANARD FINITE BNNS (AND GENERAL DGPS)
Standard, finite BNNs are deep kernel processes, albeit ones which do not admit an analytic ex- pression for the probability density. In particular, the prior for standard Bayesian neural network
(Fig. 3 top) is,
P(W)) = IN O,INN-))))))
W IXN;
(41a)
[XW1 (Fe-1) W. otherwise,
for 1= 1,
FXN
(41b)
where wh. is the Äth column of Wt In the netuol- case, ó is a pointwise nonlinearity such as ReLU. Integrating out the weights, the features, Fr. become Gaussian distributed, they depend linearly on the Gaussian distributed weights, W..
P ((F = Nh / (F) = (F,,
(42)
where
K2 = 1 No_ ((F (FFt))
(43)
13

---

Under review as a conference paper at ICLR 2021
Crucially, Fy depends the previous layer activities, F only through the kernel, K₃ As such, we could write a generative model as (Fig. middle),
XX No.1 o FF F (F)=)))
for f otherwise,
K
(44a)
P (2) = II (,))
(44b)
where we have explicitly included the kernel, Kg, as latent variable. This form highlights that BNNs are deep GPs, in the sense that F are Gaussian, with a kernel that depends on the activations from the previous layer. Indeed note that any deep GP (i.e. including those with kernels that cannot be written as a function of the Gram matrix) as a kernel, Kg, is by definition matrix that can be written as the outer product of a potentially infinite number of features, ((Fe) where we allow ø to be much richer class of functions than the usual pointwise nonlinearities (Hofmann et al. 2008). We might now try to follow the approach we took above for deep GPs, and consider a Wishart- distributed Gram matrix, G2 N, F.F However, for BNNs encounter issue:
not able compute the kernel, Ke just using the Gram matrix, Gy: we need the full set of features, Fz.
Instead, we need an alternative approach to show that a neural network is a deep kernel process. In particular, after integrating out the weights, the resulting distribution is chain-structured (Fig. middle). so in principle we can integrate out F, to obtain a distribution over K₃ conditioned on Ke-1, giving the DKP model in Fig. 3 (bottom),
P ((K = dd 1PE--18 8n Kr (O (Fe-i(Ke-)).
(F
(45)
where P P(Fe-1(KE-)) is given by Eq. 44b) and åD is the Dirac-delta function, representing the deterministic distribution, P (Kr F (Eq. 44a). Using this integral to write out the generative process only in terms of Kr gives the deep kerne process in Fig. (bottom) While this distribution exists in principle, it cannot be evaluated analytically. But we can explicitly evaluate the expected value K2 given Kę-1 using results from Cho & Saul (2009). In particular, we take Eq. 44a write out the matrix- multipliccationn explicitly as series of vector outer products, and note that as f is IID across the empirical average is equal to the expectation of single term, which computed by Cho & Saul 2009).
E K N, E. E (2(2)) ((())) E (o(E))) ( (E))
= -f ar's N (, Ki) of o())0" (C)) K(Kr).
(46)
Finally, we define this expectation to be K(K₃) in the case of NNs.
C.2 INFINITE NNS
We have found that for standard finite neural networks, we were not able to compute the distribution over K2 conditioned on K (Eq. (45)). To resolve this issue, one approach is to consider the limit of an infinitely wide neural network. In this limit, the K2 becomes deterministic function of K,-1, as K, can written as the average of N₂ IID outer products, and as N₂ grows to infinity, the law of large numbers tells us that the average becomes equal its expectation,
lim K++1 lim 1 EN (F) E [o(()) (F) K. K(K)). Ne-000
(47)
No-
C.3 INFINITE NNS WITH BOTTLENECKS
In infinite NNs, the kernel is deterministic, meaning that there is no flexibility/variaaiiity and hence no capability for representation learning (Aitchison 2019). Here, we consider infinite networks with bottlenecks that combine the tractability of infinite networks with the flexibility of finite networks (Aitchison) 2019). The trick is separate flexible. finite linear "bottlenecks" from siffite-widit nonlinearities. We keep the nonlinearity infinite in order to ensure that the output kernel is determin- istic and can be computed using results from Cho & Saul 2009). In particular,
finite-widh
14

---

Under review as a conference paper at ICLR 2021
Layer
Layer
Output Layer
W
M
W₂
M2
W 3:
X
F
F
F2
F2
F3
Y
X
K
F1
G1
K2 -
F2
22
F
K3 F3
Y
X
K
G
K₂
G2
K3 F3
X
G
22
Figure 4: A series of generative models for an infinite network with bottlenecks. First row. The standard model. Second row. Integrating out the weights. Third row. Integrating out the features, the Gram matrices are Wishart-distributed, and the kernels are deterministic. Last row. Eliminating all deterministic random variables, we get a model equivalent to that for DGPs (Fig.
- Tbottom.
F2 E PxNx and infinite width Fy- E PRMM (we send My to infinity while leaving N₂ finite),
P (We) = W
Mo
No.
(48a)
F. [XWe
= otherwise,
(48b)
(
(M+) II N (mf:0,1/N,)
(48c)
F F,M..
(48d)
This generative process given graphically in Fig. (top).
Integrating the expansion weights, M. R R, and the bottleneck weights, W R MM-M the generative model (Fig. second row) can be rewritten,
No XX*
for f=
K
(49a)
(F
Tt (F/-1)
otherwise,
M
Kg IN OFS;0,K,)
O,
(49b)
1 N, F.F. G.
(49c)
P GF) = -I (1) 0, G.)
(49d)
Remembering that Ke+1 is the empirical mean of M, IID terms, as Me oo it converges its expectation
lim K lim 1 Mc-+00
Ex (F") o (f*) = E [o(t") of (r") G
K(G.).
(50)
Mc+00 Me
and we define the limit to be K(G2). Note if we use standard (e.g. ReLU) nonlinearities, we can use results from Cho & Saul 2009 to compute K(G.). Thus, we get the following generative process,
No. XXT for 1= otherwise,
K,
(51a)
G(-
P (G+) = W Gei N, L.KK Ne),
(51b)
Finally, eliminating the deterministic kernels, Kr, from the model, we obtain exactly the deep GP generative model in Eq. (Fig. fourth row).
D STANDARD APPROXIMATE POSTERIORS OVER FEATURES AND WEIGHTS FAIL TO CAPTURE S SYMMETRIES
We have shown that it is possible represent DGPs and variety of NNs deep kernel processes. Here, we argue that standard deep GP approximate posteriors are seriously flawed, and that working with deep kernel processes may alleviate these flaws.
15

---

Under review as a conference paper at ICLR 2021
In particular, show that the true DGP posterior has rotational symmetries and that the true BNN posterior has permutation symmetries that are not captured by standard variational posteriors.
D.1 PERMUTATION SYMMETRIES IN DNNS POSTERIORS OVER WEIGHTS
Permutation symmetries neural network posteriors were known in classical work on Bayesian neural networks (e.g. MacKay 1992). Here, we spell out the argument in full. Taking P to be permutation matrix (i.e. a unitary matrix with Pp" I with one in every row and column), have,
FF) P (FP).
(52)
ie. permuting the input to nonlinearity is equivalent to permuting its output. Expanding two steps of the recursion defined by Eq. 41)b).
F, - ((o((F
(53)
multiplying by the identity,
F, = (( PP'',,
(54)
where R. R applying Eq.
F; = W.
(55)
defining permuted weights,
W WP-IP
P We.
(56)
the output is the same under the original or permuted weights,
F2 = ((o(F-2)) W; = ((o(F W We
(57)
Introducing a different perturbation between every pair of layers we get a more general symmetry,
W W1P1,
(58a)
W. = PT_, W,P, for E (2.
(58b)
W W 4+1 = PP
(58c)
where P. E R. 1 As the output of the neural network is the same under any of these permutations the likelihoods for original and permuted weights are equal,
P W W...., (+) P (Y[X, W Wi...,, W2+1),
(59)
and as the prior over elements within a weight matrix is IID Gaussian (Eq, 41a). the prior probability density is equal under original and permuted weights,
(WI..... W L1) (Wi,...
Wi+1).
(60)
Thus, the joint probability is invariant to permutations,
P PY W..... ..... P WL+1) P P (Y|X, Wi...., W (W I...., W
(61)
and applying Bayes theorem, the posterior is invariant to permutations,
P (Wi, W Y, = P (W',-. Y,
(62)
Due in part to these permutation symmetries, the posterior distribution over weights is extremely complex and multimodal. Importantly, it is not possible to capture these symmetries using standard variational posteriors these symmetries if we work with Gram matrices and kernels, which are invariant to permutations (and other unitary transformations; Eq. 12
weights, such factorised posteriors, but it is not necessary to capture
16

---

Under review as a conference paper at ICLR 2021
D.2 ROTATIONAL SYMMETRIES IN DEEP GP POSTERIORS
To show that deep GP posteriors are invariant to unitary transformations, U, E RNNx,,,, where U.U = I, define transformed features, F"
= F.Ug.
(63)
To evaluate P FF we begin by substituting for F
P FF - -I O, K
No.1 F F F(
(64)
= =IINE 0, K
Nz-1 F U 1--U%_F F))).
(65)
= =IIN N
f 0, K
N Fr F(-FF)
(66)
= F
(67)
To evaluate P (F probability density,
substitute for F in the explicit form for the multivariate Gaussian
P (F Tr (F K F + const,
(68)
2 =- Tr (K f-1 FFF + const,
(69)
= Ir , F.U.U. + const,
(70)
--
1 Tr F(F,FFFF)) + const,
(71)
= P (Fr(Fe-1).
(72)
2-11 and the constant depends only on Fe-1- Combining these derivations, each of these conditionals invariant rotations of F, and Fe-1-
where K,-1 = N F
P (F (F) F--1) = FF-1))
(73)
The same argument can straightforwardly be extended the inputs, P (F)
P X = P X),
(74)
and to the final probability density, for output activations, FL+1 which is not invariant to permuta- tions,
P (FF++I P(F
(75)
Therefore, we have,
P (F), F2, Fr.1, = P PP f(P(FIFL))) (aan))))FF)) ((F)))),
(==
(76)
(I (FFIX)) (77)
P (Y(Fr11) PPPHIP))
P (F Fz, FLt1. Y
(78)
Therefore, applying Bayes theorem the posterior is invariant to rotations,
P (F FL, X, = FL. Fx,,
(79)
Importantly, these posterior symmetries are captured by standard variational posteriors with non-zero means (e.g. (Salimbeni & Deisenroth] 2017).
D.3 THE TRUE POSTERIOR OVER FEATURES IN A DGP HAS ZERO MEAN
We can use symmetry to show that the posterior of F; has zero mean. We begin by writing the expectation an integral,
F Fet1] dF FF
(80)
17

---

Under review as a conference paper at ICLR 2021
Changing variables in the integral to F" FF, and noting that the absolute value of the Jacobian is 1, we have
= =f aF (-F))((;:(-F)) Pett,
(81)
using the symmetry of the posterior,
/ dF' (FF)P(E-FFEE,,
(82)
=
E F Fet1),
(83)
the expectation is equal to minus itself, so it must be zero
E [F 1-1, Fe+1] = O.
(84)
E DIFFICULTIES WITH VI DEEP WISHART PROCESSES
The deep Wishart generative process is well-defined as long as we admit nonsingular Wishart dis- tributions (Uhlig 1994] Srivastava et al.) 2003). The issue comes when we try to form a variational approximate posterior over low-rank positive definite matrices. This is typically the case because the number of datapoints, is usually far larger than the number of features. In particular, the only convenient distribution over low-rank positive semidefinite matrices is the Wishart itself,
Q (Gr) = W (Ge: N₂ N,N,)
(85)
However, key feature of most variational approximate posteriors is the ability increase and decrease the variance, independent of other properties such as the mean, and in our case the rank of the matrix. For Wishart, the mean and variance are given by,
E GG] = v:
(86)
(G,)
V [ Nr + Q(C,)
(87)
Initially, this may look fine: we can increase or decrease the variance by changing N₁. However, remember that Ni is the degrees of freedom, which controls the rank of the matrix, G.. As such, N₂ is fixed by the prior: the prior and approximate posterior must define distributions over matrices of the same rank. And once Ne fixed, we no longer have independent control over the variance.
To go about resolving this issue, we need to find distribution over low-rank matrices with indepen- dent control of the mean and variance. The natural approach use non-central Wishart, defined as the outer product of Caussian-dissributee vectors with non-zero means. While this distribution easy to sample from and does give independent control over the rank, mean and variance, its probability density is prohibitively costly and complex to evaluate (Koev & Edelman 2006).
F SINGULAR (INVERSE) WISHART PROCESSES AT THE INPUT LAYER
In almost all cases of interest, our the kernel functions K(G) return full-rank matrices, so we can use standard (inverse) Wishart distributions, which assume that the input matrix is full-rank. However, this is not true at the input layer as Ko = No XXT will often be low-rank. This requires us to use singular (inverse) Wishart distributions which in general are difficult to work with (Uhlig 1994 Srrivastava al.) 2003 Bodnar & Okhrin| 2008 [odnar et al.) (0166). As such, instead we exploit knowledge of the input features to work with a smaller, full-rank matrix, Q E RR-ox.No where, remember, No is the number of input features in X. For a deep Wishart process,
1 XXXX = = W
N. Ko Ni),
where Ω ~ W
1 I,N.
(88)
and for a deep inverse Wishart process,
No xΩxx G W-1 (o, 81 + +P+ where Ω Ω~ W-1 (8 81 + No + 11. (89)
Now, we are able to the full-rank matrix, Ω rather than the low-rank matrix, G2 the random variable for variational inference. For the approximate posterior over 22 in deep inverse Wishart
process, we use
Q ( w- (&I+ V.VI 81 + m + (No 1))
(90)
Note in the usual case where there are fewer inducing points than input features, then the matrix Ko will be full-rank, and can work with G; the random variable usual.
18

---

Under review as a conference paper at ICLR 2021
G APPROXIMATE POSTERIORS OVER OUTPUT FEATURES
To define approximate posteriors over inducing outputs, we are inspired by global inducing point methods (Ober& & Aitchison 2020). In partieular, we take the approximate posterior to be the prior, multiplied by a "pseudo-likelioodd",
(91) This is valid both for global indueing inputs and (for small datasets) training inputs, and the key thing to remember is that in either case, for any given imput (e.g. an MNIST handwritten 2), there is a desired output (e.g. the class-lahel "2"), and the top-layer global inducing outputs, Va, express these desired outcomes. Substituting for the prior,
Q (FL+II)L) x (FL++I/GL Ni+1 N (vx:fx .A5').
Q (F L-1 & I N (fx+*) :0, K(G)) N (vx:ff+t) ,Ax'),
(92)
and computing this value gives the approximate posterior in the main text (Eq. T9p.
H
USING EIGENVALUES TO COMPARE DEEP WISHART, DEEP RESIDUAL WISHART AND INVERSE WISHART PRIORS
One might be concerned that the deep inverse Wishart processes in which we can easily perform inference are different to the deep Wishart processes corresponding to BNNs (Sec. [.T] and infinite NNs with bottlenecks (App. C.3). To address these concerns, we begin by noting that the (inverse) Wishart priors can be written in terms of samples from the standard (inverse) Wishart
G = LNLL,
G' = LLL'L
(93)
where K = LL" such that,
Ω ~ W I),,
S ~ W (NI, G' ~ W~ (NK, AN).
(94)
G G~W (H
(95)
Note that as the standard Wishart and inverse Wishart have uniform distributions over the eigenvec-
tors (Shah et al. (2014), they differ only in the distribution over eigenvalues of f and Ω. We plotted the eigenvalue histogram for samples from a Wishart distribution with N = P = 2000 (Fig. E top left). This corresponds to an IID) Gaussian prior over weights, with 2000 features in the input and output layers. Notably, there are many very small eigenvalues, which are undesirable as they eliminate information present in the input. To eliminate these very small eigenvalues, common approach is to use a ResNet-insppred architecture (which is done even in the deep GP literature, e.g. Sallimeni&& & Deisenroth, 2017). To understand the eigenvalues in residual layer, define ResW distribution by taking the outer product of a weight matrix with itself,
wWF Ω2 ResW (N,o),
(96)
where the weight matrix IID Gaussian, plus the identity matrix, with the identity matrix weighted as
W = vito (V*S+al)
6, Nf N(, 1).
(97)
With 1. there are still many very small eigenvalues, but these disappear compared these distributions to inverse Wishart distributions (Fig. S bottom) with varying degrees found that inverse Wishart distributions do not produce
increases. We
of freedom. For all degrees of freedom, very small cigenvalues, which would eliminate information. As such, these eigenvalue distributions resemble those for ResW with a larger than
DOUBLY STOCHASTIC VARIATIONAL INFERENCE IN DEEP INVERSE WISHART PROCESSES
Due to the doubly stochastic results in Sec. 4.3 we only need to compute the conditional distribution over single test/train point (we do not need the joint distribution over a number of test points). As such, we can decompose G and Psi as.
G g
1 A
G
V
(98)
9ú
19

---

Under review as a conference paper at ICLR 2021
W ()
ResWi(N 1)
Res 2)
ResW 3)
300
frequency
200
100
W WTM,. M
W- (M1,3)
W (MM 5N)
W (M. 10M)
300
frequency
200
100
0
2
3
2
3
eigenvalue
eigenvalue
eigenvalue
elgenvalue
Figure 5: Eigenvalue histograms for a single sample from the labelled distribution, with N 2000.
where G D RAxn ga E IR Rtimesl and 24 R column- and أú and shu scalars. Taking the results in Eq. (31) to the univariate case,
Ini Dü
gii Tt (G))
gi:
Visi vu 2// T L
4/a--
(99)
As gii is univariate, its distribution becomes Inverse Gamma,
Lu G ~ InverseGamma (a = 1/2 (òr + P P 1) ()ni))
(100)
As git a vector rather than a matrix, its distribution becomes Gaussian,
(G
8u Salgun G G ~ N (W, t Hi. V.
(101)
J SAMPLES FROM THE ID PRIOR AND APPROXIMATE POSTERIOR
First, we drew samples from one-layer (top) and two-layer (bottom) deep inverse Wishart process, with souaredd-xponenttee kernel (Fig. 6). We found considerable differences in the function family corresponding to different prior samples of the top-layer Gram matrix, GL (panels). While differ- ences across function classes in one-layer IW process inference over prior on the lengthscale, this is not true of the two-layer process, and to emphasise this, the panels for two-layer samples all have the same first layer sample (equivalent choosing lengthscale), but different samples from the Gram matrix at the second layer. The two-layer deep IW process panels use the same, fixed input layer, variability in the function class arises only from sampling Gg.
be understood as equivalent to doing
Next, we exploited kernel flexibilities in IW processes by training a one-layer deep IW model with a fixed kernel bandwidth on data generated from various bandwidths. The first row in Figure 7 shows posterior samples from one-layer deep IW processes trained on different datasets. For each panel, we first sampled five full G matrices using Eq. (11) and (31b). Then for each G1, we use Gaussian conditioning to get a posterior distribution on testing locations and drew one sample from the posterior plotted as single line. Remarkably, these posterior samples exhibited wiggling behaviours that were consistent with training data even outside the training range, which highlighted the additional kemel flexibility in IW processes. On the other hand, when model bandwidth was fixed, samples from vanilla GPs with fixed bandwidth in the second row displayed almost identical shapes outside the training range across different of training data.
20

---

Under review as a conference paper at ICLR 2021
one-layer GP
one-layer IW process
4
2
two-layer GP
two-layer process with fixed first layer
4
2
0
-2 2 4
2
0
2
-4
2
4
4
2
0
2
2
0
2
Figure 6: Samples from one-layer (top) and a two-layer (bottom) deep IW process prior (Eq, 16 On the far left, we have included set of samples from a GP with the same kernel, for comparison. This GP is equivalent to sending óu - oo in the one- one-layer deep IW process and additionally sending + oo the two-layer deep IW process. All the deep IW process panels use the same squared- exponential kernel with bandwidth 1. and ón ō1 O. For each panel, we draw a single sample of the top-layer Gram matrix, G then draw multiple GP-distributed functions, conditioned on that Gram matrix.
one-layer IW process
4
2
0
one-layer
2
0
-2
-10
0
10
-10
0
10
-10
0
10
-10
0
10
-10
0
10
Figure 7: The additional flexibility in one-layer deep IW process can be used to capture mismatch the kernel. We plot five posterior function samples from trained processes the first row, and samples from trained GPs below. We generate different sets of training data from GP with different kernel bandwidths (0.5, 1, 5, 10) across columns, while we keep the kernel bandwidth in all models being 1.
21

---

Under review as a conference paper at ICLR 2021
K WHY WE CARE ABOUT THE ELBO
While we have shown that DIWP offers some benefits in predictive performance, it gives much more dramatic improvements in the ELBO. While we might think that predictive performance is the only goal, there are two reasons to believe that the ELBO itself is also an important metric. First, the ELBO is very closely related to PAC-Bayesian generalisation bounds (e.g. Germain al.) 2016). In particular, the bounds are generally written as the average training log-likelihood, plus the Kl-divergence between the approximate posterior over parameters and the prior. This mirrors standard form for the ELBO,
C E log (x|z)] - DKL (Q PP) Q(x)
(102)
where x is all the data (here, the inputs, X and outputs, Y). and z are all the latent variables. Remarkably, Germain al. (e.g. 2016) present a bound on the sos-log-likelihon that is exactly the ELBO per data point, up to additive constants. As such, in certain circumstances, optimizing the ELBO equivalent to optimizing a PAC-Bayes bound the settllo-likelihooo Similar results are available in Rivasplata et al. (2019). Second, we can write down an alternative form for the ELBO as the model evidence, minus the KL-divergence between approximate and true posterior,
[ log P (xr) DKL (Q (=) (=|z)) log P (z:).
(103)
As such, for fixed generative model, and hence fixed value of the model evidence, log P (x)) the ELBO measures the closeness of the variational approximate posterior, Q (2) and the true pos- trying perform Bayesian inference, our goal should be to make the
terior, P (2|z). As approximate posterior as close as possible to the true posterior. If, for instance, we can set Q (2) to give better predictive performance, but be further from the true posterior, then that is fine in certain settings, but not when the goal is inference. Obviously, it is desirable for the true and approxi- mate posterior to as close as possible, which corresponds larger values of L (indeed, when the approximate posterior equals the true posterior, the KL-divergence is zero, and C = log P (x) ).
L DIFFERENCES WITH SHAH ET AL. (2014)
For one-layer deep inverse Wishart process, using our definition in Eq. (16)
Kg 1 Xx",
(104a)
P (G) Ko) = W- Ko, 81 + (P 1)),
(104b)
P (y) 0, (G:)).
(104c)
Importantly, we do the nonlinear kernel transformation after sampling the inverse Wishart, so the inverse-Wishart sample acts as a generalised lengthscale hyperparameter (App. B), and hence dra- matically changes the function family.
In contrast, for Shah al. (2014), the nonlinear kernel computed before, the inverse Wishart sampled, and the inverse Wishart sample is used directly as the covariance for the Gaussian,
Kg = K
1 xx")
(105a)
No
P (G, Kg) w-1 (6 Ko 81 (P +1).
(105b)
P (y) = (JA;
(105c)
This difference in ordering, and particular, the lack of nonlinear kernel transformation between the inverse-Ve inverse-Wishart and the output is why et al. (2014) were able to find trivial results in their model (that it is equivalent to multiplying the covariance by a random scale).
22


Reviews:
[{"review_id": "8pDr2N9xmVT", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "---\n##### Summary:\n\nThis paper proposes deep kernel processes (DKPs), which can be viewed as a specific kind of deep Gaussian processes where the kernel can be written as a function of the Gram matrix. The features in the intermediate layers are integrated out and the Gram matrix are Wishart distributed. A doubly stochastic variational inference method is proposed to learn DKPs. The idea looks novel to me. My major concern is about the writing.\n\n---\n##### Pros:\n\nThe paper is well-motivated and the idea is neat. For kernels that can be written as functions of the Gram matrix, DKP seems to be a better formulation than DGP. Due to difficulty in posterior inference, the authors propose to use inverse Wishart prior over Gram matrices. The use of inverse Wishart prior is explained and discussed in Appendix, which is satisfying. The authors also explain how the sparse variational inference method for DKP sidesteps the difficulties of capturing permutation and rotational symmetries in standard variational inference of BNNs or DGPs. The experimental results show the effectiveness of the proposed method. \n\n---\n##### Cons: \n\nThe paper is hard to read and understand. There are a lot of undefined or inconsistent notations, for example:\n(1) In the beginning of the Background section, the data dimension is sometimes denoted as uppercase $P$, sometimes denoted as lowercase $p$.\n\n(2) Following Eqn. (2), the inverse $\\mathbf{\\Sigma}^{-1}$ is not defined. What is $\\mathbf{\\Sigma}$? It seems that by $\\mathbf{\\Sigma}^{-1}$ the authors mean the inverse of $\\mathbf{S}$, but that is a weird notation.\n\n(3) In section 3, the notations $\\mathbf{F}_\\ell$ and $N_\\ell$ are not explained. Although it can be inferred from the equations (4a-4c) or from the DGP literature, it is still obscure. $\\mathbf{K}$ is sometimes used as a function that operates on the Gram matrix and sometimes used as the Kernel matrix, which need.\n\n(4) In Eqn. (8), $\\bm{G}_{\\ell}-1$ should be $\\bm{G}_{\\ell-1}$.\n\nFollowing that, the author mentioned “so $\\mathbf{G}_{\\ell+1}$ can be written as …” but used subscripts $\\ell$ and $\\ell-1$ in Eqn. (9).\n\n(5) In Eqn. (11), it is not clear what $\\mathbf{K}_{\\mathrm{features}}$ means.\n\nI just mentioned a few and there are many more. I suggest the authors refine the writing thoroughly.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "7wccsCMXpss", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a prior distribution over covariance matrices of kernels which is defined as a sequential graphical model where each variable is Wishart distributed and its scale matrix is a non-linear transformation of its predecesor variable on the graph. The paper begins by considering a DGP with isotropic kernels across the layers and realizes that the Gram matrices are Wishart distributed. Based on this, the paper proposes to bypass the inference of the features and sample the Gram matrices directly from Wishart distributions. This insight, in addition to the layered structure of DGPs, gives rise to the proposed prior distribution. Furthermore, given the restrictions of the Wishart distribution for modelling covariance matrices of arbitrary size [1], as well as the conjugacy properties of the inverse Wishart distribution, the paper uses the inverse Wishart distribution instead. Doubly stochastic variational inference is proposed for approximating the posterior distribution which includes the use of inducing points thanks to the marginalization properties of the inverse Wishart distribution. The experimental contribution consists of a comparison against DGP and Neural Network GP on the UCI, MNIST and CIFAR-10 dataset.\n\nIn terms of clarity, the readability of the paper is negatively affected by being too broad with the results presented. It seems that the equivalence between GPs with Wishart processes as priors and DGPs, BNN could be a paper on its own. Thus, no topic is presented with the depth needed. In regards to significance, it is not clear from the paper, how extensive is the family of kernels that can be generated from the prior distribution, given that the proposed model is equivalent to DGPs only in the case of isotropic kernels. In that sense, it is not clear from the paper why this approach would be beneficial over DGPs.\n\n[1] Amar Shah, Andrew Wilson, and Zoubin Ghahramani. Student-t processes as alternatives to Gaus-\nsian processes. In Artificial intelligence and statistics, pp. 877–885, 2014.", "extra_scores": {}, "extra_text": {}}, {"review_id": "4xo0C7MGOe2", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes the deep kernel processes, which can generalize several existing deep kernel methods, including deep Gaussian processes and Bayesian neural networks. I'm not very familiar with the existing literature on the deep kernel models, so I could not properly assess the novelty in this work. However, the work seems significant to me. This new formulation allows the authors to avoid the feature space, and form the deep structure entirely on the kernel space. Combined with the new inference scheme, the authors showed the proposed method has performance gain over DGP and NNGP on many applications.\n\nMy main concern about this work is about its practicality. I am not entirely familiar with the current development in this area, but I believe DGP as formulated in this paper is not very popular and scalability might be one of the main reasons. Could the author address some of the limitation on this side? How does it compare to some of the more scalable methods (such as deep kernel learning where only the last layer is GP)? How is the tradeoff between accuracy and runtime for different number of inducing points?", "extra_scores": {}, "extra_text": {}}, {"review_id": "3IEmpSfIze5", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\nSummary:\n \nThe manuscript proposes a deep kernel processes model, where gram matrices are transformed by non-linear kernel functions and assumed to follow Wishart distributions. \n    \n##########################################################################\nReasons for score: \n \nI think that method provides a new approach to implement deep kernel processes (it is fully kernel-based compared to Deep GPs which is based on the feature based representation). The method has all required features to make kernel based methods practical: it is still an inductive learning method (enable to make predictions for unseen test points) and provides doubly stochastic inducing-point based variational inference (to scale the model to large data set). However, the manuscripts could be improved in some aspects (please read my comments below).\n \n##########################################################################\nPros: \n \n1.\tI think that the method is carefully designed and equipped with all required features to make kernel based methods practical.\n2.\tThe method includes important models (DGPs, BNNs, and infinite BNNs) as special cases.  \n3.\tThe method shows superior performance over DGPs and infinite BNNs on the benchmark data sets. \n \n##########################################################################\nCons: \n \n1.\tSome parts are confusing or unclear. For example, in equation (19b) the dimensions of the matrices are not matched to each other: K(G_{l-1}) \\in R^{N_0 X N_0} but V_l V_l^T in R^{PXP}. In addition, in equation (22) please provide the exact form of Q(\\Omega, {G_{l}}^L_{l-2}, F_{L+1}|X_i). Equation 19 (a,b,c) do not really show how the method can reduce the computational complexity based on inducing points.    \n\n2.\tThe manuscript does not include computational (memory and time) complexities. I also think that the method should be compared to the previous approaches, including deep Gaussian processes with sparse approximation, in terms of these computational complexities. \n  \n#########################################################################\n\nMinor comments:\nI think that it would be better if the authors included more detailed descriptions what “task-dependent representation” in introduction. \nIn the sentence above equation (7), “the DKP as an autoregressive process (Fig. 2)”. I think that Fig. 2 is not about “autoregressive process”?\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper contributes to the literature of deep kernel learning by proposing a deep probabilistic model based on inverse Wishart distributions. This is an interesting addition to the literature, and the authors have provided some experimental evidence of the superiority of their method compared to state of the art on deep GPs and NNGPs. The major concerns with the paper, though, are related to the clarity of writing and its readability. Additionally, the experimental comparison is thin and lacks exploration for alternative configurations for DGPs and NNGPs. 
Decision: Reject