A variational approximate posterior for the deep Wishart process
Authors: Sebastian W. Ober
Laurence Aitchison
Venue: NeurIPS 2021 Poster (2021)
Keywords: deep Gaussian process
Bayesian inference
variational inference

Abstract: Recent work introduced deep kernel processes as an entirely kernel-based alternative to NNs (Aitchison et al. 2020). Deep kernel processes flexibly learn good top-layer representations by alternately sampling the kernel from a distribution over positive semi-definite matrices and performing nonlinear transformations. A particular deep kernel process, the deep Wishart process (DWP), is of particular interest because its prior can be made equivalent to deep Gaussian process (DGP) priors for kernels that can be expressed entirely in terms of Gram matrices. However, inference in DWPs has not yet been possible due to the lack of sufficiently flexible distributions over positive semi-definite matrices. Here, we give a novel approach to obtaining flexible distributions over positive semi-definite matrices by generalising the Bartlett decomposition of the Wishart probability density. We use this new distribution to develop an approximate posterior for the DWP that includes dependency across layers. We develop a doubly-stochastic inducing-point inference scheme for the DWP and show experimentally that inference in the DWP can improve performance over doing inference in a DGP with the equivalent prior.

A variational approximate posterior for the deep Wishart process
Sebastian W. Ober Department of Engineering University of Cambridge Cambridge, UK swo250cam. ac. uk
Laurence Aitchison
Department of Computer Science University of Bristol Bristol, UK
laurence www..aainsssnnssssssooocco ac.uk
Abstract
Recent work introduced deep kernel processes entirely kerne-baaeed alter- native to NNs (Aitchison al. 2020). Deep kernel processes flexibly learn good top-layer representations by alternately sampling the kernel from a distribution over positive semi-definite matrices and performing nonlinear transformations. particular deep kernel process, the deep Wishart process (DWP), is of particular interest because its prior can be made equivalent to deep Gaussian process (DGP) priors for kernels that can be expressed entirely in terms of Gram matrices. How- ever, inference in DWPs has not yet been possible due to the lack of sufficiently flexible distributions approach to obtaining flexible distributions over positive semi-definite matrices by generalising the Bartlett decomposition of the Wishart probability density. We use this new distribution to develop an approximate posterior for the DWP that includes dependency across layers. We develop doubly-stochastic inducing-point inference scheme for the DWP and show experimentally that inference in the DWP can improve performance over doing inference in DGP with the equivalent prior.
positive semi-definite matrices. Here, give a novel
1 Introduction
The successes of modern deep learning have highlighted that good performance tasks such as image classification (Krizhevsky et al., 2012) requires deep models with lower layers that have the flexibility learn good representations. Up until very recently, this was only possible in feature- based methods such as neural networks (NNs). Kernel methods did not have this flexibility because the kernel could be modified only using few kernel hyperparameters. However, with the advent of deep kernel processes (DKPs; Aitchison et al., 2021), we now have deep kernel methods that offer neural-network like flexibility the kernel top-layer representation. DKPs introduce this flexibility by taking the kernel from the previous layer, then sampling from Wishart or inverse Wishart centered on that kernel, followed by nonlinear transformation. The sampling and nonlinear transformation steps are repeated multiple times to form deep architecture. Remarkably, deep Gaussian processes (DGPs; Damianou Lawrence, 2013; Salimbeni & Deisenroth, 2017), standard Bayesian NNs. infinite-widde Bayesian NNs (neural network Gaussian processes NNGPs: Lee et al., 2018: Matthews et al., 2018: Novak et al., 2019; Garriga-Alone NNs with finite width bottlenecks (Aitchison, 2020) can be written as DKPs (Aitchison et al., 2021). Indeed, for kernels that can expressed terms operations on Gram matrices, Aitchison et al. (2021) showed that particular DKP, the deep Wishart process (DWP) has prior equivalent to a DGP's s prior. In a DGP, the random variables inferred in variational inference are the model's intermediate features, with kernels computed in DWP, there are no features at all: the only random variables are the positive semi-definite kernel
al., 2019) and infinite
function of these features at each layer. However,
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

matrices themselves, which are sampled directly from Wishart distributions: the DWP works entirely on the kernel matrices implied by the DGP's features.
Aitchison et al. (2021) argued that DWPs should have considerable advantages over related feature- based models, because feature-based models have pervasive symmetries in the true posterior, which are difficult to capture in standard variational approximate posteriors. For instance, in a neural network, it is possible permute rows and columns of weight matrices, such that the activations at given layer are permuted, but the network's 's overall input-output function remains the same (MacKay, 1992; Sussmann, 1992; Bishop et al., 1995). These permutations result in network weights with exactly the same probability density under the true posterior, but with very different probability densities under standard variational approximate posteriors. which are generally unimodal. However, these issues do not arise with DWPs, because all permutations of the hidden units correspond to the same kernel (see Appendix
Aitchison et (2021) for more details).
While Aitchison et al. (2021) showed the equivalence between DWPs and DGPs, they were not able to do inference in DWPs because they were not able to find a sufficiently flexible distribution over positive semi-definite matrices to form the basis of an approximate posterior. Instead, they were forced work with different DKP: the deep inverse Wishart processes (DIWPs), which was easier because the inverse Wishart itself forms a suitable approximate posterior. While the DIWP Bayesian model.
also avoids using features, it does not correspond directly to an already-establiite Therefore, the DWP is a more important model, as it allows us to directly compare feature- and kernel-based inference. In this work, we show how to create a sufficiently flexible approximate posterior for DWPs, thereby enabling us to compare directly to their equivalent DGPs. In particular, our contributions are:
We develop a new family of flexible distributions over positive semi-definite matrices by generalising the Bartlett decomposition (Sec. 3.1).
We use this distribution to develop an effective approximate posterior for the deep Wishart process which incorporates dependency across layers (Sec. 3.2).
We develop a doubly stochastic inducing-pon point inference scheme for the DWP. While the derivation mostly follows that for deep inverse Wishart processes (Aitchison al., 2021), we need to give novel scheme for sampling the test/training points conditioned on the inducing points, as this is very different in the DWP compared to the previous DIWP (Sec. 3.3).
We empirically compare DGP and DWP inference with the exact same prior. This was not possible in Aitchison et al. (2021) as they only derived an inference scheme for deep inverse Wishart processes, whose prior is not equivalent to DGP prior.
We provide reference implementation at https: htops:///ithult httpa///w...mmaaueene/////ttttte
2 Background
2.1 Wishart distribution
The Wishart, W (E, v), is a distribution over positive semi-definite P x P matrices, G, with positive parameter, v. The Wishart distribution is defined by taking & vectors nı R sampled from a zero-mean Gaussian. These vectors can be generated from standard Gaussian vectors, εx. by transforming them with the Cholesky, L of the scale parameter, Σ LL"",
definite scale parameter Σ E PxP and a positive, integer-valued degrees-offfreedon
LĒ = n ~ N E)
where &A ~ N IO,I)
Both n, and y can stacked to form P x v matrices, N and E.
N (n)
n2
nz)
- == (5, E
E2)
Wishart samples are defined by taking the sum of the outer products of the n, 's, which can be written as matrix multiplaaitii,,,.nnn.
Σ = - NN = LE=TL- LZLT = G ~ (E,,)
(1)

---

where Z = ==" is a sample from a standard Wishart (i.e. one with an identity scale parameter,)
Ex = =E" =z -~ W (L.v).
Note that therefore the Wishart has mean,
E [ = |nn = =
(3)
2.2 Bartlett decomposition
However, sampling 3\t be computationally expensive for very large values of Instead, it is possible to sample a Wishart by writing down the distribution over the Cholesky of z, denoted A (Bartlett, 1934). Taking Z AAT. the distribution over A is,
(A3))) Gamma (A)) a-====it a- i+1 1 ))
(4a)
P (AAsk) N/ (A,x; 0, 0)
(4b)
i.e. the square of the on-diagonal elements are Gamma distributed and the off-diagonal elements are IID standard Gaussian.
2.3 Deep Gaussian proceses (DGPs)
In DGP, we progressively sample features, Fe, from a Gaussian process, conditioned on features from the previous layer,
P F( W W (ff; O, Keatures (Fe-1))
with F. X,
(5a)
P (Y = W (yx; ff ,o²))
(5b)
where X E Pxvo is the input and F, E Px' are the features. We use P for the number of input points and ue for the width of layer E; thus vo is the number of inputs and VL+1 is the number of outputs. In addition, the features and targets be written stack of vectors, f E R and yA E R., i.c.
F F2= (ff
fg
Y
(y1 Y2
The function Keaaures (F (FF-1) takes the features at the previous layer and returns the corresponding P x P kernel matrix. We consider isotropic kernels such the squared exponential, which can be written function of R, the distance between input features and j.
K Kieatures;i = =k(R
R = 1
No ENe, (FA-1
2.4 Deriving equivalent deep Wishart processes
Following Aitchison al. (2021), we show how the DGP model of Eq. (5) can be expressed deep Wishart process. We first consider the P x P Gram matrices defined as
My PL M
G F.F.
((
where ff are IID and multivariate-Gauune distributed conditioned on the features at the previous layer (Eq. 5a). Thus, Ge follows the definition of the Wishart (Eq. 1), and we can sample G. directly,
P ((G W (Ges 1 Kentures (F (Fe-1),
To work entirely with Gram matrices rather than features, we need to be able to compute the kernel, K Keaatures (F (-))) as function of the Gram matrix the previous layer, Ge-1. Remarkably, this possible for a large family of practically relevant kernels (Aitchison et al., 2021), for instance

---

isotropic kernels and the arc-cosine (or ReLU) kernel. In particular, we focus on isotropic kernels, note that is possible recover distances from the Gram matrix:
- 2F ( - +Gz
Thus, since isotropic kernels depend only on the distance, it possible obtain K(-), which takes the Gram matrix from the previous layer and returns the same kernel matrix as that returned by applying Keatures to the features from the previous layer:
K features (Fe-1) = ((G(-1) = F
By using the equivalent kernel written as a function of the Gram matrix at the previous layer, we can entirely eliminate intermediate layer features, resulting in a deep Wishart process,
P (GG(GG=1) W (Ge: ((GG())) t,, = W (yx:tx+ oo11)
with Go
XX
(6a)
Mo
P(FLHI)E) = W IIX P
(6b)
(6c)
2.5 The DWP formulation captures true-posterior symmetries while DGP does not
We have two equivalent generative models: one phrased in terms of features, Fr and another phrased in terms of Gram matrices, Ge. Is there any reason to prefer one over the other? It turns that there In particular, consider transformation of the features, F` F,U where U is unitary matrix, such that UU transformations, in the sense that all unitary transformations of the underlying features have the exact same true-posterior probability density (see Aitchison al., 2021, Appendix D.2).
Remarkably, the true posterior is symmetric under these
P
X, Y) = (Fi,.... Fr., F1+1 (X,
would be desirable for variational approximate posteriors to capture these true posterior symmetries. However, the usual family of Gaussian approximate posteriors over features fails to capture these symmetries because they use non-zero means. Worryingly, the failure to capture these symmetries can bias variational inference focus low-mass areas of the true posterior (Moore, 2016; Pourzanjani et al., 2017).
In contrast, the deep Wishart process sidesteps this issue by phrasing posteriors entirely in terms of Gram matrices, G = 1 F.F.. Critically, the Gram matrix is invariant to unitary transformations of the features,
G F.F." =
TORT==LFUUR L.UUUUFF. = oy F....
As such, DWP approximate posteriors written in terms of G implicitly respect this unitary symmetry over the features.
2.6 Equivalent DWP posteriors will have better ELBOS and generalisation
Let consider the ELBOs implied by using DGP posterior Q ((F2)e) and the DWP posterior implied by this posterior, which for this section we denote Q ({ ((Ge)e). Using D = (X, Y):
CDGP = Eo P (Y KL(Q ((Fe)e) P| ((Fe)elD)),
CDwp E P (Y KL(Q ((G))) || ([Gr)
By the data processing incquality (see e.g. Thm 6.2 in Polyanskiy & Wu (2014)), since G, is a deterministic transformation of Fz, it is straightforward to show that Lowp 2 LDGP. This result is one of the main motivations of the recent trend towards function-space inference (Sun et al., 2018; Ma et al., 2019); for deeper theoretical understanding of this result we refer the reader to Burt et al. (2021), This fact can also be used to derive better PAC Bayes generalisation bounds (Section 6.1.3 of Alquier (2021)).
Note that this analysis relied on the assumption that Q ((Gz)) would the distribution implied by Q ((Fr)e) In practice, this is will not be the case, as we wish to directly specify our distribution terms of Gram matrices. However, given a sufficiently flexible posterior over Gram matrices, we should still see this advantages in practice. We now turn to developing such distribution.

---

Methods
As detailed in Aitchison al. (2021), the key difficulty in obtaining variational inference scheme for DWPs is the difficulty of providing sufficiently flexible approximate posterior. In particular, as are working with probabílistic process, the number of input points, P. can be arbitrarily large, and thus there is always the possibility that v < P and hence that our sampled Gram matrices are low-rank. We therefore need to form flexible variational approximate posteriors over rank Gram matrices. An obvious first choice is the Wishart distribution itself with degrees of freedom v, so as match the rank of matrices sampled from the prior. However, for fixed degrees of freedom the Wishart variance,
V [G, = ( + EnED,))
cannot be specified independently of the mean (Fq. 3), which is essential for variational approximate posterior that can flexibly capture potentially narrow true posteriors. An alternative approach would be work with a non-central Wishart, which is defined by taking E, which is IID standard Gaussian in the case of the Wishart, to have non-zero mean. However, the non-central Wishart has a probability density function that is too difficult evaluate in the inner loop of deep learning algorithm (Koev & Edelman, 2006). Instead, we develop new Generalised Singular Wishart distribution, based on the Bartlett decomposition, which modifies the Wishart give independent control over the mean and variance of sampled matrices.
3.1 The Generalised Singular Wishart
To define the Generalised Singular Wishart distribution, first need generalise the Bartlett construction to potentially singular matrices (i.e. those for which < P). Remembering that Z = AAT, in the singular case A is given by
A
A
Av1
Ave
(7a)
Ap1
Ap
P (A Gamma (43, "=i+1 1
(A N (A,;;
(7b)
Recalling that G LA and by applying the results of Appendices and C, have that
P II
min(Pw)
Gamma (A3, "-
P (G)
II
II (A,; $=/+1
minn(jv)
A L
P-3+1
jj
In Appendix D we prove that this corresponds to the known full rank and singular Wishart distribution. Equipped with the singular Bartlett,
now develop generalisation of the Wishart distribution:
(, σ), is a distribution over Definition The Generalised Singular Wishart, W (G; E, a,
positive semi-definite P matrices, G, with positive definite scale malrix Σ = LL" PRP. a positive, valued degrees-of- ffreeeoom parameter v, and Bartlett Bartlelt-generalinn generalising parameters a. B, O. These latter parameters modify the Bartlett decomposition as follows:
Q (A3,) Gamma (5)3;
for j S v,
Q (Aiss) = N (AgiiPij,
for j S
This implies a distribution over G LAATIT with density
P
min(P,v) P.u)
P 1 P-j+1 Gamma (A3zi B,) II N (Ayi *=+11
Q (G)=
II
II J=1
min(j.))
A
33
Note that we use Q (.) to reflect the fact that we will use the Generalised Singular Wishart the basis for our approximate posterior. The density is derived using the same transformations and Jacobians as for the singular Wishart above.
5

---

3.2 Full approximate posterior distribution
Unlike the deep inverse Wishart process in Aitchison et al. (2021), it is not possible to obtain an optimal last-layer posterior for the deep Wishart process. Therefore, we choose form that mimics the form of the DIWP posterior, allowing for similar across-layer dependencies:
Q (Gr - W (G: - 4)) ((G(-1(G++ + Pe, Ot Be Mc 01),
(9)
where the approximate posterior parameters are aV,, ae, 3, Me: oe, q6 qe) where qe < is scalar, and V. E "="'"_ Here, the learnable parameter qe allows us to trade the off influence on the scale matrix from the prior and the learned covariance Ve V which allows for similar across-layer dependencies as in (Aitchison et al., 2021). ve is fixed, as determines the width of the layer; the remaining parameters are simply the parameters from the Bartlett generalisation.
3.3 Doubly stochastic inducing-poing variational inference in deep inverse Wishart processes
For efficient inference in high-dimensional problems, we take inspiration from the DGP literature (Salimbeni & Deisenroth, 2017) by considering doubly-stochasti inducing-point deep Wishart processes. We begin by decomposing all variables into inducing and training (or test) points X; RAxNo and X, Ax where P is the number of inducing points, and Pt is the number of testing/training points,
X X
F
G G
FL+11
G.
F
where e.g. G is P. P. and G is x P. The full ELBO including latent variables for all the inducing and training points is
P ((GG) F
C= E log (Y + log Q (G
(10)
where the expectation is taken over Q ((Gr) ( The prior is given by combining all terms in Eq. (6) for both inducing and test/train inputs,
P ((Gr))=., F X) [T' P (G P (F (FL+1/G)).
where the X-dependence enters on the right because Go = _xx". Taking inspiration from Salimbeni & Deisenroth (2017), the full approximate posterior is the product of an approximate posterior over inducing points and the conditional prior for train/test points,
Q FL Q F ((G (G (G)H,, (11)
And the prior can be written in the same form,
P
P (IG IX, ((G (G)) P F(+1,, (12)
We discuss the second terms (the conditional prior) in Eq. (15). The first terms (the prior and approximate posteriors over inducing points), are given by combining terms in Eq. (6) and Eq. (9),
((G F IX)) [T/_ (G - P (F++I)HH)
(13)
Q ((G 1 (x)
(X)) = [1 Q (G G " Q (F |
(14)
Substituting Eqs. (11-14) into the ELBO (Eq. 10), the conditional prior cancels and we obtain,
I' (GIG
(
C E log (Y|F++1)) log
Ih P (G(G P (F(+IIGH)

---

The first term is summation across test/train datapoints, and the second term depends only on the inducing points, so as Salimbeni & Deisenroth (2017) we can compute unbiased estimates of the expectation by taking only minibatch of datapoints. We also never need to compute the density of the conditional prior Eq. (12), we only need to be able sample from it,
P (IG GUEE GEHIGGHE F.+1, X)
P (F GL) (G G Ge-1). (15)
The first distribution, P O Gr), is multivariate Gaussian, and can be evaluated using methods from the GP literature (Rasmussen Williams, 2006; Salimbeni & Deisenroth, 2017). Specifically, use the global inducing point scheme from Ober & Aitchison (2021). The second distribution is more difficult to sample from. To address this issue, we go back to features,
F-F vG W (K (G)-1),
(16)
with
F
K K
F
K (G--1)
K
where F2 E (R(+R))xL F, IR Rxvi and F, RRx. Our goal is to sample G and G. given G Our approach is to note that, F, conditioned F, is given by matrix normal, (Eaton et al., 2007, page 310),
P P(F = MN (K) K, F, Kni.))
(17)
where
Kui = K, K K K
Note that we sample each test/train point one-at-a- cin--at---iimmmmiddeeeeeeddl in which case, Pt = 1 and Ez2-1 scalar.
Then Gg, which includes G and G given by,
G
G G
F F.) F (F T
Ff (FF) T F. (F
F.F
G
For F,, we any value long G - F (Ft)² T
the resulting distribution over G2 arising from Eq. (16) does not depend on the specific choice of F; (App. E). Remembering that to sample Gū, we explicitly sample its potentially low-rank Cholesky, LeA,, we can directly use
F =
However, this only works if S. P. in which case, L,A, R In the unusual case where have fewer inducing points than degrees of freedom, P. < v, then L&A, E RXT, so we need to pad to achieve the required size of P, x Ve,
F (L, 0).
Finally, note that we can optimise all the variational parameters using standard reparameterised variational inference (Kingma & Welling, 2014; Rezende et al., 2014). For an algorithm, see Alg. 1.
3.4 Computational complexity
Recalling that 2/2 is the width the éth layer, P; is the number of inducing points, and P; the number of train or test points, the computational complexity of one DWP layer is given by O(p3 - P.P??) This decrease of a factor of Ut+1 over complexity for standard DGP inference, such as doubly stochastic variational inference (Salimbeni & Deisenroth, 2017), which has complexity O(ver1 P.)). The difference arises from the fact that in DGP, ve+1 Gaussian processes sampled in each layer, whereas for DWP sample single Gram matrix.
7

---

Algorithm 1 Computing predictions/ELBB for one batch
parameters: {vE)
Q parameters: (Ve, q6, aze, Be, M2, Tel X,
Inputs: X,; Targets: Y
combine inducing and test/train inputs
X (X, X,)
sample first Gram matrix and update ELBO
Go = Mn _XX
for é in (I....,4) do
sample inducing Gram matrix and its Cholesky, LeA, and update ELBO L, G ~ QG --1
C+ L log P
log Q G
sample full Gram matrix from conditional prior
Σ = 1(G
Σm En
Ed ii 2i Eit
F LEAr
F. MN
E E
Fr, I
G (FATT (F
G. =
end for
sample GP inducing outputs and update ELBO
FL+1 Q
(FL+IGHH)
L + log P (F G - log Q [
sample GP predictions conditioned on inducing points
F ~ ~Q (F F
add likelihood to ELBO
L + L C+llgg log P (Y
Results
The DWP prior is equivalent to DGP prior (Sec. 2.4) (Aitchison et al., 2021); the only difference is that in DGP, we use features as the latent variables, whereas in the DWP we use Gram matrices. Using Gram matrices as in the DWP should be beneficial as the true posteriors are expected to be simpler than in the DGP (Sec. 2.5). We perform comparison of the posterior intermediate-layee features implied by DWP and those of DGP on toy problem App. G; these show the benefits of being more able to model the true posterior.
For more quantitative experiments, trained DWP and DGP with the exact same generative model with a squared exponential kernel on the UCI datasets from Gal & Ghahramani (2015). We trained both models for 20000 gradient steps using the Adam optimizer Kingma & Ba (2015);
detail the exact experimental setup in Appendix F. We report EI ELBOSs and test log likelihoods for depth 5 in Table we report other depths and quote the relevant results from Aitchison al. (2021) for reference in Appendix H. We found that the DWP often outperformed the DGP model, especially evident we look at the ELBOs and smaller datasets (boston, concrete, energy, and yacht). On larger datasets, the benefits often disappear, accurate uncertainty modelling is less relevant. Note that we compared against the recently introduced DGP method based on global inducing points (Ober Aitchison, 2021). Global inducing point methods were particularly important in our setting because we use standard feedforward architecture without skip connections ensure equivalence between the DGP and DWP. Standard DSVI has considerable difficulties with optimizing the approximate posterior such models; to get optimization to work effectively Salimbeni & Deisenroth (2017) were forced to modify the prior to introduce skip connections.

---

Table ELBOs and log- likeliiheooos for UCI datasets from (Gal & Ghahramani, 2015) for a five-layer network. See Appendix H for other depths. Significantly better results are highlighted.
dataset
DWP
DGP
boston
-0.38 + 0.01
-0.45 + 0.01
concrete
-0.49 0.00
-0.50
energy
1.41 = 0.00
1.00
kin8nm
-0.14 0.00
-0.13
ELBO
naval
3.65 + 0.07
3.91
power
0.03 0.00
0.02 +.00
protein
-1.01 0.00
-1.00 0.00
wine
1
-1.19 0.00
yacht
1.65 0.01
1.37 1 0.03
boston
-2.39 1.39
-2.44 114400 0.04
concrete
-3.33
+.02 0.02
energy
-0.00
kin8nm
1.40
1.38 0.01
LL
naval
8.21
8.31
power
#.00
-2.88
protein
-2.73
-.03
wine
-0.66
-0.96
yacht
+.14
-0.88 0.08
4.1 Runtimes & training curves
In Sec. 3.4, showed that DWPs have lower computational complexity than DGPs, because of the need for DGPs to sample Vi features in each layer. whereas DWPs only need to sample one Gram matrix. Here, we briefly discuss the runtimes of our implementations. We show a plot of the training curves for one split of boston with a 5-layer DGP and DWP in Fig. 1, plotted against both runtime and epoch. From these plots, make observations. First, the DWP trains much than the DGP in terms of runtime. However, it seems to require slightly more epochs than the DGP converge (note that the spike at the start of the DGP curve an artifact of the tempering scheme we use). In Appendix H, we provide a table of time per epoch, which shows that we obtain faster runtime for protein and for shallower models, although the gains slightly more modest due the models being shallower and the fact that we run protein on GPU, as opposed to a CPU for boston.
quickly
Related Work
The DWP prior was introduced by Aitchison al. (2021). However, they were not able do variational inference with the DWP because they did not have a sufficiently flexible approximate posterior over positive semi-definite matrices. Instead, they were forced work with a deep inverse Wishart process, which is easier because the inverse Wishart itself is suitable approximate posterior. Here, we give flexible generalised Wishart distribution over positive remi-defnitt matrices which is suitable for use as variational approximate posterior in the DWP. As the deep Wishart process prior equivalent to DGP prior, we were able to directly compare DGP and DWP inference in models with the exact same prior. Such a comparison with equivalent priors was not possible in Aitchison et al. (2021), because their deep inverse Wishart process priors are not equivalent to DGP priors
There is an alternative line of work using generalised Wishart processes (Wilson & Ghahramani, 2011, opposed our deep Wishart processes). Note that the "generalised Wishart process" terminology does not seem to have spread as widely as it should, but it is very useful in our context. generalised Wishart process specifies a distribution over an infinite number of finite- dimensional Wishart- dynamical system, in which case there might an infinite number of such matrices, one for each time or location in the state-space (Wilson & Ghahramani, 201 I; Heaukulani & van der Wilk, 2019; Jorgensen al., 2020). In contrast, the Wishart process (Dawid, 1981; Bru, 1991) describes finite dimensional marginals of single, potentially infinite dimensional matrix. In our context, we stack
matrices. These matrices might represent e.g. the noise covariance

---

0.3
DGP
-0.4
(natg)
0.5
(mats)
0.55
0.7
0.7
0.8
-08-
-099
DGP boston boston
-1.0
-10
2800
5000
7500
10000
12500
5000
10000
15000
20000
Time (5)
Epoch
(a) ELBO time
ELBO vs. epoch
Figure 1: ELBO versus time and epoch for five-layer models one split of boston
) Wishart processes to form a deep Wishart process. Importantly, these generalised Wishart priors do not have the flexibility capture a DGP prior because the underlying features all locations are jointly multivariate Gaussian (Sec. in Wilson & Ghahramani, 2011) and therefore lack the required nonlincarities between layers. Further, not only do the underlying stochastic processes (deep vs generalised Wishart process) differ, inference is also radically different. In particular, work the generalised Wishart performs inference the underlying multivariate Gaussian feature vectors (Eq. e.g. Eq. 15-18 in Wilson & Ghahramani 2011, Eq. 12 in Heaukulani & van der Wilk 2019 or Eq. 24 in Jorgensen et al. 2020). Unfortunately, variational approximate posteriors defined over multivariate Gaussian feature vectors fail to capture symmetries in the true posterior (Sec. 2.5). In contrast, define approximate posteriors directly over the symmetric positive semi-definite Gram matrices themselves, which required us to develop new, more flexible distributions over these matrices.
6
Limitations
There are few limitations of our work. First, is only possible derive equivalent DWPs for certain kernels namely, those where matrices. While this holds for a large range of kernels, such as isotropic kernels and the arc-cosine kernel, it does not hold for some common kernels such automatic relevance determination (ARD) kernels. However, we note that we are able to use any kernel for the first layer, and that in practice we did not find that ARD kernels in intermediate layers significantly improved performance, all the features have a shared prior. The second limitation is that not currently possible to incorporate modifications the basic DGP model such skip connections (Duvenaud et al., 2014; Salimbeni & Deisenroth, 2017). Such modifications would require the use of the non-central Wishart distribution, which is difficult evaluate in the inner loop of deep learning algorithm (Koev & Edelman, 2006). Third, the generalisation models is non-trivial, and will be left future work. Finally, seems that the performance is not as competitive for larger datasets, where uncertainty representation is of more limited use we will explore improving this in future work.
skip the feature representation and work entirely in Gram
our work to more complex architectures such as convolutional
Conclusions
introduced a flexible distribution over positive semi-definite matrices which formed the basis of a variational approximate posterior for the deep Wishart process. We adapted the doubly stochastic variational inference scheme from Aitchison et al. (2021) to the deep Wishart process. Thus, we were able to directly compare the performance for inference in DWP DGP with exactly the same prior. This isolates the effects on performance the prior and the inference procedure. We found DWPs often have improved performance over their corresponding equivalent DGP models, particularly in the ELBOs, indicating improved uncertainty representation.
There are no anticipated social impacts the work is largely theoretical.
10

---

Acknowledgements
The authors thank David R. Burt, Mark van der Wilk, Samuel Power, and Adam X. Yang for helpful discussions, and would like to thank the reviewers for their comments.
Funding Transparency
SWO acknowledges funding from the Gates Cambridge Trust for his doctoral studies. The authors have no competing interests additional funding declare.
References
Laurence Aitchison. Why bigger is always better: on finite and infinite neural networks. Interna- tional Conference on Machine Learning (ICML), 2020.
Laurence Aitchison, Adam X Yang, and Sebastian W Ober. Deep kernel processes. International Conference on Machine Learning (ICML), 2021.
Pierre Alquier. User-friendly introduction to PAC-Bayes bounds, 2021.
Maurice Stevenson Bartlett On the theory of statistical regression. Proceedings of the Royal Society of Edinburgh, 53:260-283, 1934.
Christopher M Bishop et al. Neural networks for pattern recognition. Oxford university press, 1995.
Marie-France Bru. Wishart processes. Journal of Theoretical Probability, 4(4):725-751, 1991.
David Burt, Sebastian W Ober, Adrià Garriga-Alonso, and Mark van der Wilk. Understanding variational inference in function-space. In Symposium on Advances in Approximate Bayesian Inference (AABI), 2021.
Andreas Damianou and Neil Lawrence. Deep Gaussian processes. In Artificial Intelligence and Statistics (AISTA7S), 2013.
A Philip Dawid. Some mutrix-variiat distribution theory: notational considerations and a Bayesian application. Biometrika, 68(1):265--2) 1981.
David Duvenaud, Oren Rippel, Ryan P. Adams, and Zoubin Ghahramani. Avoiding pathologies in very deep networks. In Artificial Intelligence and Statistics (AISTATS), 2014.
Morris L Eaton et al. The Wishart distribution. In Multivariate Statistics, pp. 302-333 Institute of Mathematical Statistics, 2007.
Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model
uncertainty in deep learning. International Conference Machine Learning (ICML), 2015.
Adrià Garriga-Alonso, Carl Edward Rasmussen, and Laurence Aitchison. Deep convolutional networks as shallow Gaussian processes. International Conference on Learning Representations (ICLR), 2019.
Creighton Heaukulani and Mark van der Wilk. Scalable Bayesian dynamic covariance modeling with variational Wishart and inverse Wishart processes. In Advances in Neural Information Processing Systems, 2019.
Martin Jorgensen, Marc Deisenroth, and Hugh Salimbeni. Stochastic differential equations with variational Wishart diffusions. In International Conference on Machine Learning (ICML), 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations (ICLR), 2015.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. International Conference on Learning Representations (ICLR), 2014.

---

Plamen Koev and Alan Edelman. The efficient evaluation of the hypergeometric function of a matrix argument. Mathematics of Computation, 75(254):833-845 2006.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances in Neural Information Processing Systems, 2012.
Jachoon Lee, Yasaman Bahri, Roman Novak, Samuel Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as Gaussian processes. International Conference on Learning Representations (ICLR), 2018.
Chao Ma, Yingzhen Li, and José Miguel Hernández-Lobato. Variational implicit processes. In International Conference Machine Learning (ICML), 2019.
David JC. MacKay. A practical Bayesian framework for backpropagation networks. Neural Compula- tion, 4(3):488 472, 1992.
Arak M Mathai. Jacobians of matrix transformation and functions of matrix arguments. World Scientific Publishing Company, 1997.
Arakaparampil M Mathai and Hans Haubold. Special functions Sunciios for applied scientists. Springer, 2008.
Alexander G de G Matthews, Mark Rowland, Jiri Hron, Richard E Turner, and Zoubin Ghahramani. Gaussian process behaviour in wide deep neural networks. International Conference on Learning Representations (ICLR), 2018
David A Moore. Symmetrized variational inference. In NIPS Workshop on Advances in Approximate Bayesian Inference, volume 4. pp. 31, 2016.
Roman Novak, Lechao Xiao, Jaehoon Lee, Yasaman Bahri, Greg Yang. Jiri Hron, Daniel A Abolafia, Jeffrey Pennington, and Jascha Sohl-Dickstein. Bayesian deep convolutional networks with many channels are Gaussian processes. International Conference Learning Representations (ICLR), 2019.
Sebastian W Ober and Laurence Aitchison. Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processes. International Conference on Machine Learning (ICML), 2021.
Yury Polyanskiy and Yihong Wu. Lecture notes information theory. Lecture Notes for ECE563 (UIUC), 2014.
Arya A Pourzanjani, Richard M Jiang, and Linda R Petzold. Improving the identifiability of neural networks for Bayesian inference. In NIPS Workshop on Bayesian Deep Learning. volume 4, pp. 31, 2017.
Carl Edward Rasmussen and Christopher KI Williams. Gaussian processes for machine learning. MIT press Cambridge, MA, 2006.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. International Conference on Machine Learning (ICML), 2014.
Geoffrey Roeder, Yuhuai Wu, and David Duvenaud. Sticking the landing: Simple, lower-variance gradient estimators for variational inference. In Advances Neural Information Processing Systems, 2017.
Hugh Salimbeni and Mare Deisenroth. Doubly stochastic variational inference for deep Gaussian processes. In Advances in Neural Information Processing Systems, 2017.
Muni S Srivastava et al. Singular Wishart and multivariate beta distributions. The Annals of Statistics, pp. 1537-1560, 2003.
Shengyang Sun, Guodong Zhang, Jiaxin Shi, and Roger Grosse. Functional variational Bayesian neural networks. In International Conference Learning Representations (ICLR), 2018.
12

---

Héctor J Sussmann. Uniqueness of the weights for minimal feedforward nets with a given input-output map. Neural networks, 5(4):589-593, 1992.
Andrew Gordon Wilson and Zoubin Ghahramani. Generalised Wishart processes. Uncertainty in Arlificial Intelligence (UAI), 2011.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Though none were anticipated
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material as URL)? [Yes] (Code
supplementary material, and a reference implementation provided at https: // / thub. com/Laurence// b/aree////baysuunc
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A] UCI datasets have unclear licenses.
(c) Did you include any new assets either in the supplemental material URL? [Yes]
(Code in supplementary material
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] Standard datasets.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
13


Reviews:
[{"review_id": "ysCA7pOZW4-", "reviewer": "Reviewer_UZcb", "summary": "The authors introduce an approximate inference procedure for the deep Wishart process. In order to do so, they define a generalised singular Wishart process and use this as a variational posterior. The DWP appears to be a model consisting of layers of Wishart process, which are distributions over PSD matrices. This allows one to define a Wishart process whose parameters are the output of another Wishart process. When the a deep GP is constructed with a kernel that is isotropic, the, DGP is a DWP.  However, the usual approximate posterior used in the DGP fails to capture a certain orthogonal invariance property in the features that is present in the true posterior. The DWP on the other hand, deals with Gram matrices instead of features and this requirement from the approximate posterior is automatically satisfied. ", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The motivation for why the DWP is superior to the DGP seems to be that the variational posterior that it has access to can encode the symmetry constraint in the posterior. Excluding this initial motivation and the introduced methods, there is very little theory giving any kind of theoretical guarantees. I am also not convinced that this motivation is sufficient to justify an entirely new inference scheme, given that the current empirical evaluation is a bit limited. Are there any other reasons why we would prefer to do inference with a DWP versus a DGP?\n\n\nQuestions/Suggestions for improvement:\n1. As currently written, the paper seems to assume a certain familiarity with the original deep kernel processes in its motivation, which is not (yet) a mainstream model. Since the paper is specifically about the deep Wishart process, apparently a special case of the deep kernel process, in order to make the message more direct to the reader I would suggest only briefly mentioning deep kernel processes and then focussing on the DWP.\n2. The paper is not the easiest to read. I enjoyed section 2. In section 3, I would prefer a self-contained theorem characterising the generalised singular Wishart distribution, which for the reader doing their first pass, could be used as a abstract tool to understand the rest of the paper. Do all the equations need to be numbered? It seems like many are never referred to in the main text.\n3. Experiments. You only compared the DWP to a DGP with squared exponential kernel. Arguably you have removed the flexibility that the DGP has in that it can use non-isotropic kernels. For a fair comparison, the DGP should have access to non-isotropic kernels.\n4. I believe the authors should be more upfront about the requirement that the kernel is isotropic in the abstract, as this is a relatively strong constraint. The authors state that \"We mainly consider isotropic kernels\", but as far as I can tell, they exclusively consider isotropic kernels.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "bfLbMklY6rs", "reviewer": "Reviewer_uQwZ", "summary": "Deep kernel process (DKP, Aitchison et al., 2020) is a new probabilistic model that provides more flexible modelling power than a shallow kernel method. The deep Wishart process is equivalent to deep Gaussian process, but there is no inference algorithm in the original paper. This paper proposes a new variational inference algorithm for it. It proposes a generalised singular Wishart distribution to model the posterior distribution of the random Gram matrix in DWPs and apply the doubly-stochstic inducing-point scheme. This is a nice contribution to the progress of research along the line of DKPs.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Significance & originality:\n\nDKPs have a good property of preserving the symmetries in latent features that standard variational inference for DGPs does not. Earlier work developed variational inference method for deep inverse Wishart processes which has a different prior distribution from deep GPs. This work provides a variational inference method for deep Wishard processes so that one can compare the performance of DWPs with DGPs side by side. This is a nice contribution to the research in DKPs. The main novelty of this inference method is introducing the generalised Wishart distribution, while the application of doubly-stochstic inducing-point inference is mostly standard.\n\nTo better assess the significance of introducing the inference algorithm for DWPs, I think it would be useful to compare also the result of DIWPs. After all, we would like to find a flexible probabilistic model to have a high log-likelihood (or approximately ELBO) on train and test set.\n\n\nQuality:\n\nThe derivation of the generalised Wishart distribution based on Bartlett decomposition and its application in the variational inference algorithm looks good to me although I haven't checked the detailed derivation in the appendix.\n\nI have a question about the use of generalised Wishart distribution as the approximate posterior distribution. Do we know if the true posterior distribution of the Gram matrix has at most \\nu ranks or it could actually be higher than that? If the rank in the posterior could go higher, shall we consider the \\nu parameter in Q as another hyper-parameter to tune? Besides, could the authors comment on the capacity of the generalised Wishart distribution? How does the flexibility / capacity compare with the Gaussian approximation in DGPs besides its ability to model symmetries?\n\n\nClarity:\n\nThe authors provide sufficient background to understand deep Wishart processes and the proposed approximate posterior distribution. The methods section also provides an appropriate level of details in the main text to understand its algorithm.\n\nFor the experiment section, I would appreciate more analysis of the inference results beyond simply comparing the ELBO and test log-likelihood. It would be very helpful if the authors could show the difference in the inferred posterior using DWP and DGP. After all, the advantage of using DWPs versus DIWPs is that we can have a more direct comparison with DGPs but Table 1 doesn't give much information beside a better ELBO score.\n\nThe authors mentioned the computational complexity is lower than the for DGPs because it doesn't have to sample for each latent feature dimension separately. Does it mean the runtime of the proposed method is indeed lower than that for DGPs? If so, it would also be nice to show the improvement of the runtime in the experiments.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors answers \"Yes\" to question 1.b \"Did you describe the limitations of your work?\" but I don't see where the limitations are discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "Z7c3U_aDF-E", "reviewer": "Reviewer_qTiX", "summary": "The manuscript proposes a distribution over positive semi-definite matrices based on the singular Bartlett decomposition for a deep Wishart process (DWP), a special case of deep kernel learning [Aitchison et al., 2020] where gram matrices are used as the latent variables and are transformed through nonlinear kernel functions layer by layer. Based on this distribution, the authors developed a doubly stochastic inducing inference method for the DWP (previously inference in a DWP was not available) by adapting the corresponding variational inference method in [Aitchison et al., 2020]. \n\n", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n(Originality & quality) I think that the manuscript has certain contributions, one of which is developing the variational inference method for the DWP by proposing a distribution over positive semi-definite matrices based on the singular Bartlett decomposition. Although the propose method is not super new, the method does not seem to be trivial. I did not check every derivation but the method looks technically sound.\n\n(Significance)  However, I am concerned that this work might not attract broad interest. The manuscript proposes an inference method (which is not entirely different or new from previous work) to a specific model formulation. In addition, the experimental results section includes only toy datasets and a DGP as a comparison method. It is understandable that the manuscript attempts comparing the proposed method to the corresponding DGP model based on the connection between both formulations, but at the same time this approach would limit the significance of the proposed work. Readers cannot be convinced about the effectiveness of the proposed method to real-world problems.    \n\nMinor comments\nIt was at the first read a little bit difficult to understand how the proposed distribution (eq. 27) works since the scale parameter \\bSigma does not appear in the distribution. It would be helpful if there was a reminder that the Cholesky \\bL is calculated from the scale parameter around eq. 27.  ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Have the authors adequately addressed the limitations and potential negative societal impact of their work? Yes. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}, {"review_id": "GSNAZCqv0Lz", "reviewer": "Reviewer_ty8W", "summary": "The authors introduce effective inference strategies for deep Wishart processes (DWP). This class of deep models is of interest because it moves away from the featurized representation implicit in most deep models to one that is fully kernelized. Inference relies on a doubly stochastic inducing-point scheme familiar from variational Deep Gaussian Processes and makes use of a custom generalization of the Bartlett decomposition adapted to the DWP setting in order to define a flexible variational family. In experiments on UCI data the method shows small but consistent improvements over Deep Gaussian Processes.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is an interesting and technically sound submission. The clarity of exposition is generally high and the work is contextualized with respect to related lines of work in the literature. While the empirical benefits w.r.t. predictive performance of the proposed model and inference scheme are limited (when compared to Deep Gaussian Processes), what makes this approach particularly interesting is the fully kernelized approach that is taken. One might hope that further generalizations could pay larger predictive dividends. That the resulting scheme differs in a non-trivial way from more featurized representation commonly used in various approaches to deep Bayesian models can be read off, e.g., from the computational complexity, which does not depend on layer width (see Sec. 3.5).\n\nMore specific questions/comments:\n- I'm confused about some of the notation. Should lowercase p be changed to uppercase P everywhere (e.g. Sec 3.1)? Is V_\\ell defined correctly on line 163?\n- Is Sec. 3.1 the best use of space? It might make sense to move some of these details to the supplementary material and instead focus on one or more of: i) unpack choices made in Eqn 28a; ii) make the comparison to DGP inference more detailed.\n- Can you comment on the gradient variance of the sampling scheme in Sec. 3.4? In DGPs we can compute the distribution q(f) = \\int du p(f|u)q(u) in closed form and sample from the marginals. This has the nice benefit that the different f_i are iid. Alternatively we could sample u and then sample each f_i conditioned on u, which results in larger gradient variance. If I understand it correctly, you are effectively in this latter regime and thus presumably experience somewhat large gradient variance. Is that correct? I guess this is more or less a direct consequence of maintaining a kernelized approach? \n- You comment (line 230ff) that “to get optimization to work effectively Salimbeni & Deisenroth (2017) were forced to modify the prior to introduce skip connections.\" How do you know this is a problem with inference and not a problem with model specification? How can you disentangle the two? Shouldn't sufficiently deep DWPs exhibit some of the same pathologies explored in e.g. \"Avoiding pathologies in very deep networks\" by Duvenaud et. al.?\n\n## After author response\n\nI thank the authors for their response and I maintain my original score. While some of the concerns raised in the reviews have some merit (in particular w.r.t. the breadth of the experiments), I find some of the concerns raised by the more negative reviews less convincing (e.g. \"might not attract broad interest\"). While the topic explored in this submission is somewhat dry and technical, I believe that this contribution would make a valuable contribution to the conference. Formulating deep models in function space remains a somewhat underexplored (but potentially important) area, and for this reason (among others) I would argue that the solid technical contributions in this submission could be of interest to the NeurIPS community.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think it would be useful to better understand how the runtime compares e.g. to DGP inference. For example, if the scheme does in fact suffer from large gradient variance, then more optimization steps may be required, etc.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.25"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a variational approximation scheme for Deep Kernel Processes (DKP). Building on previous work, the DKP is an alternative representation of the Deep Gaussian Process (DGP), where the representation is over Gramm matrices rather than over function evaluations. 

Reviewers are in broad agreement that this work is a solid contribution in the development of DKPs, and in functional representation of Deep models in general. One reviewer raised a concern about the lack of theoretical guarantees that the new methodology brings over the DGP representation: in the rebuttal, the authors provide convincing arguments that the improvements in the ELBO due to the representation are directly connected to PAC bounds - I'd like to see a little discussion on this in the camera-ready version. 

Decision: Accept (Poster)