Deep Variational Implicit Processes
Authors: Luis A. Ortega
Simon Rodriguez Santana
Daniel Hernández-Lobato
Venue: ICLR 2023 poster (2023)
Keywords: Gaussian process
implicit process
variational implicit process
Bayesian inference
function-space inference
implicit process concatenation

Abstract: Implicit processes (IPs) are a generalization of Gaussian processes (GPs). IPs may lack a closed-form expression but are easy to sample from. Examples include, among others, Bayesian neural networks or neural samplers. IPs can be used as priors over functions, resulting in flexible models with well-calibrated prediction uncertainty estimates. Methods based on IPs usually carry out function-space approximate inference, which overcomes some of the difficulties of parameter-space approximate inference. Nevertheless, the approximations employed often limit the expressiveness of the final model, resulting, e.g., in a Gaussian predictive distribution, which can be restrictive. We propose here a multi-layer generalization of IPs called the Deep Variational Implicit process (DVIP). This generalization is similar to that of deep GPs over GPs, but it is more flexible due to the use of IPs as the prior distribution over the latent functions. We describe a scalable variational inference algorithm for training DVIP and show that it outperforms previous IP-based methods and also deep GPs. We support these claims via extensive regression and classification experiments. We also evaluate DVIP on large datasets with up to several million data instances to illustrate its good scalability and performance.

Published as a conference paper at ICLR 2023
DEEP VARIATIONAL IMPLICIT PROCESSES
Luis A. Ortega', Simón Rodriguez Santana?, Daniel Hernández-Lobate Universidad Autónoma de Madrid 'ICMAT-CSIC
(luis, ortega, daniel. hernandez | @uam. es, simon. renaaaoodrigggeee
ABSTRACT
Implicit processes (IPs) are a generalization of Gaussian processes (GPs), IPs may lack closed-form expression but are easy sample from. Examples include, among others, Bayesian neural networks or neural samplers. IPs can be used as priors over functions, resulting in flexible models with well-calibrated prediction uncertainty estimates. Methods based on IPs usually carry out function-space approximate inference, which overcomes some of the difficulties of parameter- space approximate inference. Nevertheless, the approximations employed often limit the expressiveness the final model, resulting, egg. a Gaussian predictive distribution, which can be restrictive. We propose here multi-layer generalization IPs called the Deep Variational Implicit process (DVIP). This generalization is similar to that of deep GPs over GPs, but is more flexible due to the use of IPs the prior distribution over the latent functions. We describe a scalable variational inference algorithm for training DVIP and show that it outperforms previous IP- based methods and also deep GPs. We support these claims via extensive regression and classification experiments. We also evaluate DVIP on large datasets with up several million data instances to illustrate its good scalability and performance.
INTRODUCTION
The Bayesian approach has become popular for capturing the uncertainty associated to the predictions made by models that otherwise provide woint-wise estimates, such as neural networks (NNs) (Gelman al., 2013; Gal, 2016; Murphy, 2012). However, when carrying out Bayesian inference, obtaining the posterior distribution in the space of parameters can become a limiting factor since it is often intractable. Symmetries and strong dependencies between parameters make the approximate inference problem much more complex. This is precisely the case in large deep NNs. Nevertheless, all these issues can be alleviated by carrying out approximate inference in the space of functions, which presents certain advantages due to the simplified problem. This makes the approximations obtained in this space more precise than those obtained in parameter-space as shown in the literature (Ma et al., 2019; Sun et al., 2019; Rodríguez Santana et al., 2022; Ma and Hernández-Lobato, 2021).
recent method for function-space approximate inference is the Variational Implicit Process (VIP) (Ma et al., 2019). VIP considers implicit process (IP) the prior distribution over the target function. IPs constitute a very flexible family of priors over functions that generalize Gaussian processes (Ma et al., 2019). Specifically, IPs are processes that may lack a closed-form expression, but that are casy-to-sample-fron from. Examples include Bayesian neural networks (BNN), neural samplers and warped GPs, among others (Rodríguez Santana al., 2022). Figure I (left) shows BNN, which is a particular case of an IP. Nevertheless, the posterior process of an IP is is intractable most of the times (except in the particular case of GPs). VIP addresses this issue by approximating the posterior using the posterior of a GP with the same mean and covariances as the prior IP. Thus, the approximation used VIP results in a Gaussian predictive distribution, which may too restrictive.
Recently, the concatenation of random processes has been used to produce models of increased flexibility. An example are deep GPs (DGPs) in which a GP is used as the input of another GP, systematically (Damianou and Lawrence, 2013). Based the success of DGPs, it is natural to consider the concatenation of IPs to extend their capabilities in a similar fashion to DGPs. Therefore, we introduce in this paper deep VIPs (DVIPs), multi-layer extension of VIP that provides increased expressive power, enables more accurate predictions, gives better calibrated uncertainty estimates, and captures more complex patterns in the data. Figure/ 1 (right) shows the architecture considered in

---

Published as a conference paper at ICLR 2023
Inputs
IP
IP
IF
IP
21
(x)
( (1)
(2 (2))
f (f 1a9)
Figure 1: (left) IP resulting from BNN with random weights and biases following Gaussian distribution. sample of the weights and biases generates a random function. (right) Deep VIP in which the input to an IP is the output of a previous IP. We consider a fully connected architecture.
DVIP. Each layer contains several IPs that are approximated using VIP. Importantly, the ficxibility of the IP-based prior formulation enables numerous models as the prior over functions, leveraging the benefits of, e.g., convolutional NNs, that increase the performance on image datasets. Critically, DVIP can adapt the prior IPs to the observed data, resulting in improved performance. When GP priors are considered, DVIP is equivalent to a DGP. Thus, can be seen as a generalization of DGPs.
Approximate inference in DVIPs is done via variational inference (VI). We achieve computational scalability in each unit using a linear approximation of the GP that approximates the prior IP, as in VIP (Ma et al., 2019). The predictive distribution of a VIP is Gaussian. However, since the inputs in the second and following layers are random in DVIP, the final predictive distribution is non-Gaussian. This predictive distribution is intractable. Nevertheless, one can easily sample from by propagating samples through the IP network shown in Figure 1 (right). This also enables Monte Carlo approximation of the VI objective which can be optimized using stochastic techniques, as in DGPs (Salimbeni and Deisenroth, 2017). Generating the required samples is straightforward given that the variational posterior depends only on the output of the the previous layers. This results in an iterative sampling procedure that can be conducted in scalable manner. Importantly, the direct evaluation of covariances are not needed in DVIP, further reducing its cost compared to that of DGPs. The predictive distribution is mixture of Gaussians (non-Gaussian more flexible than that of VIP.
We evaluate DVIP in several experiments, both in regression and classification. They show that DVIP outperforms single-layer VIP with a more complex IP prior. DVIP is also faster to train. We also show that DVIP gives results similar and often better than those of DGPs (Salimbeni and Deisenroth, 2017), while having lower cost and improved flexibility (due to the more general IP prior). Our experiments also show that adding more layers in DVIP does not over-fit and often improves results.
2 BACKGROUND
We introduce the needed background on IPs and the posterior approximation based on a linear model that will be used later on. First, consider the problem of inferring an unknown function f : R.M R given noisy observations = (y1,-.., yN ( at X = (xi...., xx). In the context of Bayesian inference, these observations are related to f (f(x)..... f(x))) f(xxx)).....(x via a likelihood, denoted as p(y f). IPs represent one of many ways to define a distribution over function (Ma et al., 2019).
Definition 1. An IP a collection of random variables f(-) such that any finite collection f {f(1), f(x)) ... [(NN] is implicitly defined by the following generative process
z ~ Pz(z) and f(xa) = gx z), Vn = 1,..,,
(ge(()))) with O its parameters, Ps source of noise, and ge (x., z) a function that given z and x,, outputs fxx,). go (x.. 2) can be, e.g., a NN with weights specified by and 0 using the reparametrization trick (Kingma and Welling, 2014). See Figure (left). Given z ~ Pz (z) and x, it is trraight-forward to generate a sample f(x) using go. i.e., f(x) = go(x, 2).
An IP is denoted as f(-)
Consider an IP the prior for unknown function and suitable likelihood p [f). In this context, both the prior x and the posterior X, y) are generally intractable, since the IP assumption does not allow for point-wise density estimation, except in the case of GP. To overcome this, in Ma
2

---

Published as a conference paper at ICLR 2023
et al. (2019) the model's joint distribution, ply, f(X), is approximated as
ply, ( R3 q(y, fX) = ply (f
(2)
where qgp is a Gaussian process with mean and covariance functions m(.) and K(,, .), respectively. These two functions are in turn defined by the mean and covariance functions of the prior IP,
m(x) (x)),,,
X[(1,, x2) =E ((f(x1) m(x1)) (f (x(2) m(x2))l
(3)
which can be estimated empirically by sampling from IP(ge(-.)) -), P2)
-).P2) (Ma et al., 2019). Using S ...o(.)))p)..... pa). I...., S. the mean and covariance functions are
Monte Carlo samples f-(-) (P(g)(:-)),
m (x) $ Ei fx(x,, X0 X2) "dxx))=((x))) (x(2))
(4)
((x,) = 1//S((((xx))) - m² mxx)),....... Js( m (x,))
Thus, the VIP's prior for f is simply a GP approximating the prior IP, which can be, e.g., BNN. Critically, the samples f. (x) keep the dependence w.r.t. the IP prior parameters 0 which enables prior adaptation to the observed data in VIP (Ma et al., 2019). Unfortunately, this formulation has the typical cost in ((N3) of GPs (Rasmussen and Williams, 2006). To solve this and also allow for mini-batch training, the GP is approximated using a linear model: f(x) = x(x))²a a m²(x) where a ~ (0, I). Under this definition, the prior mean and covariances of f(x) are given by (4). The posterior of a, p(aly), is approximated using Gaussian distribution, qu (a) - (aamm, S), whose parameters w = (m. S} (and other model's 's parameters) are adjusted by maximizing the a-energy
(5) where = 0.5, value that provides good general results (Hernández-Lobato et al., 2016; Ro- dríguez Santana and Hernández- Lobato, 2022). A Gaussian likelihood is also assumed with variance σf. Otherwise, 1-dimensional quadrature methods are required to evaluate (5). Importantly, (5) allows for mini-batch training to estimate w, 0 and x² from the data. Given, qu (a), it is straight-forward to make predictions. The predictive distribution for yn is, however, limited to be Gaussian in regression.
L (w, = -x 1 log Ee
KL (LL((a)))p(aa)
DEEP VARIATIONAL IMPLICIT PROCESSES
Deep variational implicit processes (DVIPs) are models that consider deep implicit process the prior for the latent function. A deep implicit process is a concatenation of multiple IPs, recursively defining an implicit prior over latent functions. Figure (right) illustrates the architecture considered, which is fully connected. The prior on the function at layer / and unit h, (-),),, is an independent IP whose inputs are given by the outputs of the previous layer. Let H2 be the 1-th layer dimensionality. Definition 2. A deep implicit process is a collection of random variables (hh :l = 1,...,., L A 1......y . such that each fh. fh (F.,))) ), with f! the outpur of the
h L..., Hi A previous layer in the network, i.e., f (f1,11 1......... ..... SH..... and each fh((-) an independent IP: fl())) ~ IP (-.z).p=). where f.). = X,, symbolizes the initial input features to the network.
As in VIP, we consider GP approximations for all the IPs in the deep IP prior defined above. These GPs are further approximated using linear model, as in VIP. This provides an expression for fh.m, given the previous layer's 's output f!-1 and ane the coefficients of the linear model for the unit h at layer I. Namely, fl 0% (1)"")) at, + -m (F,,)) where (-)))) and m,),(-) depend on the prior IP parameters 8% To increase the flexibility of the model, we consider latent Gaussian noise around each fl.n with variance of,, (except for the last layer 1 = L). That is, of {o H the noise variances layer I. Then, p(Sh [f- al,) is a Gaussian distribution with mean 2) (?'=!)"a,, + -m, (P(=)) and variance of.1- Let A = (a.,..., a and F' (f.,...., Cx)). The joint distribution of all the variables (observed and latent) in DVIP is
(6) where p(a()) = (aa, I, and we have omitted the dependence of fhon on f to improve readability. In (6). II, p(y, (f is the likelihood and I Ih p(f la pp(a)) is the deep IP prior. It may seem that the prior assumes independence across points. Dependencies are, however, obtained by marginalizing each ai, which is tractable since the model is linear in a/,
p (y, (F', A = -II P...I.... II I1 p(S.. (a()))
3

---

Published as a conference paper at ICLR 2023
We upproximate the posterior p ( ((F', A A' ly) using an approximation with fixed and tunable part, simplifying dependencies among layer units, but maintaining dependencies between layers:
d(FF', ( - IIT II II (lFh.nla/))
q(a))) = NN(a), (mm/..s). (7)
where the factors p( h.n ) are fixed to be the same factors as those in (6), and the factors g(a²) are the ones being specifically tuned. This approximation resembles that of Salimbeni and Deisenroth (2017) for DGPs, since the conditional distribution is fixed. However, we do not employ sparse GPs based on inducing points and use a linear model to approximate the posterior at each layer instead. Computing (f(,,.) is intractable. However, one can easily sample from it, as described next.
Using (7), we can derive a variational lower bound at whose maximum the Kullback-Leiblee (KL) divergence between ((FF,, A H2)) and p (F', A
3 ly) is minimized. Namely,
E., (1,,0 - E - [lgg Un [T, Et= Σ (L(g(a))lp(aa))
(8)
where we have used the cancellation of factors, and where 5 (m), S [ 1, ...,, LAh
1..... Hi} are the parameters of q and θ {0) 1 = 1..... L A h 1,..., Hi} are the DVIP prior parameters. Furthermore, (L(-I-) denotes the KL-divergence between distributions. (L(g(a), (L[g(a))p((()) )) involves Gaussian distributions and can be evaluated analytically. The expectations E, p(ynl [.]] are intractable. However, they can be approximated via Monte Carlo, using the
reparametrization trick (Kingma and Welling, 2014). Moreover, >a Egg [log p(yn lC4, ] can be approximated using mini-batches. Thus, (8) can be maximized w.r.t. Ω2 θ and (af) J=11 using stochastic optimization. Maximization w.r.t. θ allows for prior adaptation to the observed data, which a key factor when considering IP priors. Appendix A has all the details about the derivation of (8).
Sampling from the marginal posterior approximation. The evaluation of (8) requires samples from (F,,,) for all the instances in mini-batch. This marginal only depends on the variables of the inner layers and units fl,n, corresponding to the n-th instance. See Appendix B. Thus, we can sample from (((,,) by recursively propagating samples from the first to the last layer, using x,, as the input. Specifically, p(hh, , aa) is Gaussian with a linear mean in terms of a/. and q(a),) is Gaussian. ([=")=sp(ie p(Lh/ [ a/)((a))) day, is also Gaussian with mean and variance:
Thus, q(Sh. (m [
m/... (C, (f ) m/, + (C,,") (F = 6% ( + of (9)
where of,- and m and S,, the parameters of q(a()). Initially, let l = 1. We set f.p, xn and generate a sample from (fhh.. (f)) for h = l...., Hi. Let ft be that sample. Then, we use f' as the input for the next layer. This process repeats for / 2, ..... L. until we obtain f. ((P,
Making predictions for new instances. Let f.t be the values at the last layer for the new instance x.. We approximate g(.,,)) ) by propagating R Monte Carlo samples through the network. Then, ((P.t) R R E. He N Iritt, (Cl--1r) oh. (())-
(10)
where f.f- is the r-th sample arriving at layer L - and mf. (.) and (.) are given by (9). We note that (10) is a Gaussian mixture, which is expected to be more flexible than the Gaussian predictive distribution of VIP. We set R to 100 for testing and to 1 for training, respectively. Computing, p(y.) = Ey,p(y. (y. [f)))] is tractable in regression, and can be approximated using 1-dimensional quadrature in binary and multi-clase classification, in DGPs (Salimbeni and Deisenroth, 2017).
Input propagation. Inspired by the skip layer approach of, e.g. ResNet (He et al., 2016), and the addition of the original input to each layer in DGPs (Duvenaud et al., 2014; Salimbeni and Deisenroth, 2017), we implement the same approach here. For this, we add the previous input to the mean in (9) if the input and output dimension of the layer is the same, except in the last layer, where the added mean is zero. Namely, si (f--1) of (f!-1)1 "Tmm, + mi, 11]
(C + Mh for I = I, 1.
Computational cost. The cost at layer / in DVIP is in O(BS² H.), B S, with S the number of samples from the prior IPs, B the size of the mini-batch and H, the output dimension of the layer. This cost is similar to that of a DGP, which has squared cost in terms of M, the number of inducing
4

---

Published as a conference paper at ICLR 2023
points (Hensman al., 2013; Bui et al., 2016; Salimbeni and Deisenroth, 2017). In Salimbeni and Deisenroth (2017), the cost at each layer is H1)M2H)), if B > M. In our work, however, the number of prior IP samples S is smaller than the typical number of inducing points in DGPs. In our experiments we use 5 = 20, as suggested for VIP (Ma et al., 2019). Considering DVIP with T. layers, the total cost is (B,.5 +.... H))). of the prior IP samples, DVIP is faster than DGP, and the gap becomes bigger as L increases.
Our experiments show that, despite the generation
RELATED WORK
The relation between GPs and IPs has been previously studied. A 1-layer BNN with cosine activations and infinite width is equivalent to a GP with RBF kernel (Hensman et al., 2017). A deep BNN is equivalent to GP with a compositional kernel (Cho and Saul, 2009), as shown by Lee et al. (2017). These methods make possible to create expressive kernels for GPs. An inverse reasoning is used by Flam-Shepherd et al. (2017), where GP prior properties are encoded into the prior weights of a BNN. Fortuin (2022) provides an extensive review about prior distributions in function-space defined by BNNs and stochastic processes. They give methods of learning priors for these models from data.
VIP (Ma et al., 2019) arises from the treatment of BNNs as instances of IPs, For this, an approximate GP used to assist inference. Specifically, prior GP built with mean and covariance function given by the prior IP, BNN. VIP can make of the more flexible IP prior, whose paramcters can be inferred from the data, improving results over GPs (Ma et al., 2019). However, VIP's predictive distribution is Gaussian. DVIP overcomes this problem providing non-Gaussian predictive distribution. Thus, it is expected to lead to a more flexible model with better calibrated uncertainty estimates. DVIP differs from deep kernel learning (DKL) (Wilson et al., 201 where GP is applied to non-linear transformation of the inputs. Its predictive distribution is Gaussian, unlike that of DVIP and the non-linear transformation of DKL ignores epistemic uncertainty, unlike IPs.
There are other methods that have tried to make inference using IPs. Sun et al. (2019) propose the functional Bayesian neural networks (IBNN), where second IP is used to approximate the posterior of the first IP. This is a more flexible approximation than that of VIP. However, because both the prior and the posterior are implicit, the noisy gradient of the variational ELBO intractable and has to be approximated. For this, a spectral gradient estimator is used (Shi et al., 2018). To ensure that the posterior IP resembles the prior IP in data-free regions, fBNN relies on uniformly covering the input space. In high-dimensional spaces this can lead to poor results. Moreover, because of the spectral gradient estimator fBNN cannot tune the prior IP parameters to the data. In the particular case of a GP prior, IBNN simply maximizes the marginal likelihood of the GP w.r.t. the prior parameters. However, a GP prior implies a GP posterior. This questions using a second IP for posterior approximation. Recent works have employed first order Taylor approximation to linearize the IP, approximating their implicit distribution by a GP (Rudner et al., 2021; Immer et al., 2021). This leads to another GP approximation to an IP, different of VIP, where the computational bottleneck is located at computing the Jacobian of the linearized transformation instead of samples from the prior. More recent work in parameter-space considers using a repulsive term in BNN ensembles guarantee the diversity among the members, avoiding their collapse in the parameter space (D' D'Angelo and Fortuin, 2021).
Sparse implicit processes (SIPs) use inducing points for approximate inference in the context of IPs (Rodríguez Santana et al., 2022). SIP does not have the limitations of neither VIP nor IBNN It produces flexible predictive distributions (Gaussian mixtures) and it can adjust its prior parameters to the data. SIP, however, relies on a classifier to estimate the KL-term in the variational ELBO, which adds computational cost. SIP's improvements over VIP are orthogonal to those of DVIP over VIP and, in principle, SIP may also be used as the building blocks of DVIP, leading to even better results.
Functional variational inference (FVT) minimizes the KI.-divergence between stochastic process for approximate inference (Ma and Hernández-Lobato, 2021). Specifically, between the model's IP posterior and a second IP, as in IBNN. This is done efficiently by approximating first the IP prior using stochastic process generator (SPG). Then, second SPG is used efficiently approximate the posterior of the previous SPG. Both SPGs share key features that make this task easy. However, FVI is also limited, as IBNN, since it cannot adjust the prior to the data. This questions its practical utility.
As shown by Rodríguez Santana et al. (2022), adjusting the prior IP to the observed data is key for accurate predictions. This discourages using fBNN and FVI as building blocks of model using deep
5

---

Published as a conference paper at ICLR 2023
IP priors on the target function. Moreover, these methods do not consider deep architectures such as the one in Figure I (right). Therefore, we focus on comparing with VIP, as DVIP generalizes VIP.
To our knowledge, the concatenation of IPs with the goal of describing priors over functions has not been studied previously. However, the concatenation of GPs resulting in deep GPs (DGPs), has received lot of attention (Lawrence and Moore, 2007; Bui et al., 2016; Cutajar al., 2017; Salimbeni and Deisenroth, 2017; Havasi et al., 2018; Yu et al., 2019). In principle, DVIP is a generalization of DGPs in the same way as IPs generalize GPs. Namely, the IP prior of each layer's s unit can simply be a GP. Samples from such GP prior can be efficiently obtained using, e.g., a 1-layer BNN with cosine activation functions that is wide enough (Rahimi and Recht, 2007: Cutajar et al., 2017).
The posterior approximation used in DVIP is similar to that used in the context of DGPs by Salimbeni and Deisenroth (2017). Moreover, each prior IP is approximated by a GP DVIP However, in spite of these similarities, there are important differences between our work and that of Salimbeni and Deisenroth (2017). Specifically, instead of relying on sparse GPs to approximate each GP within the GP network, DVIP uses linear GP approximation that needs no specification of inducing points. Furthermore, the covariance function used in DVIP is more flexible and specified by the assumed IP prior. DGPs are, on the other hand, restricted to GP priors with specific covariance functions. DVIP has the extra ficxibility of considering a wider range of IP priors that need not be GPs. Critically, in our experiments, DVIP significantly outperforms DGPs in image-related datasets, where using specific IP priors based, e.g., on convolutional neural networks, can give a significant advantage over standard DGPs. Our experiments also show that DVIP is faster train than the DGP of Salimbeni and Deisenroth (2017), and the difference becomes larger as the number of layers L increases.
5 EXPERIMENTS
We evaluate the proposed method, DVIP, on several tasks. We use S = 20 and a BNN as the IP prior for each unit. These BNNs have 2 layers of 10 units each with tanh activations, as in Ma et al. (2019). We compare DVIP with VIP (Ma et al., 2019) and DGPs, closely following Salimbeni and Deisenroth (2017). We do not compare results with IBNN nor FVI, described in Section 4, because they cannot tune the prior IP parameters to the data nor they do consider deep architectures as the one in Figure 1 (right). An efficient PyTorch implementation of DVIP found in the supplementary material. Appendix C has all the details about the experimental settings considered for each method.
Regression UCI benchmarks. We compare each method on 8 regression datasets from the UCI Repository (Dua and Graff, 2017). Following common practice, we validate the performance using 20 different train/ / test splits of the data with 10% test size HHrráández-Looatt and Adams, 2015). We evaluate DVIP and DGP using 2, 3, 4 and layers. We compare results with VIP, which equivalent to DVIP with L = 1, and with VIP using a bigger BNN of 200 units per layer. We also compare results with a single sparse GP, which is equivalent to DGP for L 1, and with a Sparse Implicit Processes (SIP) with the same prior (Rodríguez Santana et al., 2022). Figure 2 shows the results obtained in terms of the negative test ligeeiiil.oood Results in terms of the RMSE and the exact figures are found in Appendix H. DVIP with at least 3 layers performs best on 4 out of the 8 datasets (Boston, Energy: Concrete and Power), having comparable results on Winered and Naval (all methods have zero RMSE on this dataset). DGPs perform best on 2 datasets (Protein and Kin8nm), but the differences are small. Appendix G shows that using a GP prior in DVIP in these problems performs better at higher cost. Adding more layers in DVIP does not lead to over-fitting and it gives similar and often better results (notably on larger datasets: Naval, P Protein, Power and Kin8nm). DVIP also performs better than VIP and SIP most of the times. By contrast, using more flexible BNN prior in VIP (i.e., 200 units) does not improve results. Figure 3 shows the training time in seconds of each method. DVIP is faster than DGP and faster than VIP with the 200 units BNN prior. Summing up. DVIP achieves similar results to those of DGPs, but at smaller cost.
Interpolation results. We carry out experiments on the CO2 time-series dataset (https: scrippsco2. ucsd. edu). This dataset has CO2 measurements from the Mauna Loa Observatory, Hawaii, in 1978. We split the dataset in five consecutive and equal parts, and used the 2nd and 4th parts as test data. All models are trained for 100, 000 iterations. Figure 4 shows the predictive distribution of DVIP and DGP with L = 2 on the data. DVIP captures the data trend in the missing gaps. For DVIP show samples from the learned prior, which are very smooth. By contrast,
6

---

Published as a conference paper at ICLR 2023
Boston
Energy
Concrete
Winared
D0P
100F
068
068 [
069
D6P
DVP
DMPS
Dvp
(0V14
DVP
(M/
DVP 2
DVit 2
Power
Navs!
Proteih
KinBhm
DCP
DGP
(GG
DEP
(069
DGP
DGF 2
DVP
DVP
(DVP
DVP
DVIP 2
5P
SP
200
VIP 200
VI7
-10
Single layer models
This work
Deep GP models
Figure 2: Negative test likelliodd results on regression UCI benchmark datasets over 20 splits. We show standard errors. Lower values (to the left) are better.
Boston
Energy
Concrete
Winared
DCP
DGP
DCP
1DGP
DGP
DGP
D09
DVP
(D17
DVP
DVP
DVt 2
SSP
VP
KinBrm
DCP
[GGP
DCP
(DGP
DGP 2
DoF 2
-S0P
DMM
DVD 2
DVF
5P
SP
200
VP 200
5000
15000
20030
30000
10032
20000
Single layer models
This work
Deep GP models
Figure 3: CPU training time (in seconds) on regression UCI benchmark datasets over 20 splits. We show standard errors. Lower values (to the left) are better.
DGP with RBF kernels fails to capture the data trend, leading to mean reversion and over-estimation of the prediction uncertainty (similar results for SGP shown in Appendix H). Thus, the BNN prior considered by DVIP could be a better choice here. This issue of DGPs can be overcome using compositional kernels (Duvenaud et al., 2014), but that requires using kernel search algorithms.
420
420
400
100
380
samppees
320
1965
1970
1930
1050
2000
2010
2020
1960
1970
1980
1030
2000
2010
2023
Figure 4: Missing values interpolation results on the CO2 dataset. Predictive distribution of DVIP (left) and DGP (right) with 2 layers each. Two times the standard deviation is represented.
Large scale regression. We evaluate each method on large regression datasets. First, the Year dataset (UCI) with 515, 345 instances and 90 features, where the original train/test splits are used.
7

---

Published as a conference paper at ICLR 2023
Second, the US flight delay (Airline) dataset (Dutordoir et al., 2020; Hensman et al., 2017), where following Salimbeni and Deisenroth (2017) we use the first 700, 000 instances for training and the next 100, 000 for testing. 8 features are considered: month, day of month, day of week, plane age, air time, distance, arrival time and departure time. For these two datasets, results are averaged over 10 different random seed initializations. Lastly, we consider data recorded on January, 2015 from the Taxi dataset (Salimbeni and Deisenroth, 2017). In this dataset 10 attributes are considered: time of day, day of week, day of month, month, pickup latitude, pickup longitude, drop-off longitude, drop-off latitude, trip distance and trip duration. Trips with a duration lower than 10 seconds and larger than hours are removed as in Salimbeni and Deisenroth (2017). leaving 12, 695, 289 instances. Results are averaged over 20 train/test splits with 90% and 10% of the data. Here, we trained each method for 500, 000 iterations. The results obtained are shown in Table 1. The last column shows the best result by DGP, which is achieved for T. = 3 on each dataset. We observe that DVIP outperforms VIP on all datasets, and on Air inn and T Taxi, the best method is DVIP. In Taxi a sparse GP and DGPs give similar results, while DVIP improves over VIP. The best method on Year, however, is DGP The difference between DVIP and DGP is found in the prior (BNN vs. GP), and in the approximate inference algorithm (DGP uses inducing points for scalability and DVIP linear model). Since DVIP generalizes DGP, DVIP using GP priors should give similar results to those of DGP on Year. Appendix G shows, however, that the differences in Year are not only due to the chosen prior (BNN vs. GP) but also due to the posterior approximation (linear model vs. inducing points). VIP using inducing points and GP prior gives similar results to those of SGP. The cost of approximately sampling from the GP prior in VIP is, however, too expensive to consider adding extra layers in DVIP.
Table I: Root mean squared error results on large scale regression datasets.
Single-Inye layer
Ours
DS-DGP
SGP
VIP
DVIP
DVIP
DVIP
DVIP
DGP
Year
9.15.10.01
10.27 10.01
9..0
9.34
9.30.00.03
9.27.10.03
8.9410.03
Airline
38.61++0.05
38.00+0.06
37.96-0..3
37.91+0..05
37.83+00.03
37.80-0.05
37.95-0004
Taxi
554.22.1..33
554.60.10.19
549.28
531.4211.59
147.3311.03
538.941.2.23
$52.9010.33
Image classification. We consider the binary classification dataset Rectangles es (Salimbeni and Deisenroth, 2017) and the multi-class dataset MNIST (Deng, 2012). Each dataset has 28 x 28 pixels images. The Rect dectangles dataset has 12, 000 images of (non-square) rectangle. The task is to determine if the height is larger than the width. Here, we used probit likelihood in each method. The MNIST dataset has 70, 000 images of handwritten digits. The labels correspond with each digit. Here, we used the robust-max multi-class likelihood in each method THeráández-Lobrr et al., 2011). In Rectangles, 20, 000 iterations are enough to ensure convergence. We employ the provided train-test splits for each dataset. Critically, here exploit DVIP's capability to use more flexible priors. In the first layer we employ a convolutional NN (CNN) prior with two layers of 4 and 8 channels respectively. No input propagation is used in the first layer. The results obtained are shown in Table 2, averaged over 10 random seed initializations. We report the best obtained results for DGP, which are obtained for L 3. We observe that DVIP obtains much better results than those of DGP and VIP in terms of accuracy. DVIP increases accuracy by 11% on Rectangl es compared to DGP, probably as a consequence of the CNN prior considered the first layer of the network being more suited for image-based datasets. Convolutional DGP can perform better than standard DGP in these tasks, however, the objective here to highlight that DVIP allows to easily introduce domain-specific prior functions that might not be easily used by standard GPs. Image classification is only example of this. Other examples may include using recurrent network architectures for sequential data.
Table 2: Results on image classification datasets.
Single-layer
Ours
DS-DGP
MNIST
SGP
VIP
DVIP
DVIP 3
DGP 2
DGP3
Accuracy (%)
96 25:0.04
97.9 03.99
98.39:00005
98. 3.200
97.75=20.04
97. 97.86:20.05
L.kelihood
4.1 2.14+00000
4I. 0.144++0.00
-0.074+0.000
00080+0000
-0.052.10.00
0.072:0.00
Sangle-layer
DS-DGP
Rectaugles
SGP
VIP
DVIP 2
DVIP3
DVIP 4
DVIP 5
DGP 3
Accuracy (9)
72.54+20.14
85.63-00.1 18
87,84=0.30
88.21:0.12
87.43:0.20
86.49:0.17
75.16:20.16
Likelibood
0.518.10.00
0.348.10.00
-0.30610.00
0.295510.00
-0.30910.00
-0.32010.00
-0.47010.000
AUC
0.828+0.000
0.930=00000
0.950=0.000
0.963++0.00
0.947:00.00
0.939=00000
0.858+00.00
8

---

Published as a conference paper at ICLR 2023
Large scale classification. We evaluate each method two massive binary datasets: SUSY and HIGGS, with 5.5 million and 10 million instances, respectively. These datasets contain Monte Carlo physics simulations to detect the presence of the Higgs boson and super-symmetry (Baldi et al., 2014). We use the original train/test splits of the data, and train for 500, 000 iterations. We report the AUC metric for comparison with Baldi et al. (2014); Salimbeni and Deisenroth (2017). Results are shown in Table 3, averaged over 10 different random seed initializations. In the case of DGPs, we report the best results, which correspond L = 4 and L 5, respectively. We observe that DVIP achieves the highest performance on SUSY (AUC of 0.8756) which is comparable to that of DGPs (0.8751) and to the best reported results in Baldi et al. (2014). Namely, shallow NNs (NN, 0.875), deep NN (DNN, 0.876) and boosted decision trees (BDT, 0.863). On HIGGS, despite seeing an steady improvement over VIP by using additional layers, the performance worse than that of DGP (AUC 0.8324). Again, we believe that GPs with an RBF kernel may be a better prior here, and that DVIP using inducing points and a GP prior should give similar results to those of DGP. However, the high computational cost of approximately sampling from the GP prior will make this too expensive.
Table 3: Results on large classification datasets.
Single-layer
Ours
DS-DGP
SUSY
VIP
DVIP2
DVIP
DVIP
DVIP
DGP
Accuracy (%)
79.75=0002
78.6820.02
80.11:00:33
80.13=0.01
80. N0.22:+0.01
50.24:0.02
80.06:20.01
Likelihood
0.436.00.00
4..45610.00
-0429110.00
0.429_0.00
-0.42710.00
-0.427.0.00
-0,43210.000
AUC
0.8727+0.000
0.857240.00
0.742440.00
0.874900.00
0.8755+0.00
0.8756+0.00
0.8751+0000
HIGGS
VIP
DVIP2
DVIP
DVIP
DVIP
DGP55
Accuracy (9)
69.95:00003
57.42:0.005
66.09:00:22
69 09.85:0.02
70.43++0.01
72.01
74.92+0.01
Likelihood
0.5733:0.00
-0.67210.000
-0.61110.00
00.555=0.00
40,56510.00
-0.54220.00
-0.50110.00
AUC
0.7693:20.00
0.6247:::00
0.719610.00
0.7704:0.00
0.778240.00
0.7962100000
0.6324110.00
Impact of the Number of Samples S and the Prior Architecture. Appendix E investigates the impact of the number of samples S' on DVIP's performance. The results show that one can get sometimes even better results in DVIP by increasing S at the cost of larger training times. Appendix F shows that changing the structure of the prior BNN does not heavily affect the results of DVIP.
6 DISCUSSION
Deep Variational Implicit Process (DVIP), a model based on the concatenation of implicit processes (IPs), is introduced as a flexible prior over latent functions. DVIP can be used on variety of regression and classification problems with no need of hand-tuning Our results show that DVIP outperforms or matches the performance of a single layer VIP and GPs. It also gives similar and sometimes better results than those of deep GPs (DGPs). However, DVIP has less computational cost when using a prior that is easy to sample from. Our experiments have also demonstrated that DVIP is wide range of tasks. DVIP does not seem to over-fit fit small datasets
both effective and scalable by increasing the depth, and on large datasets, extra layers often improve performance. We have also showed that increasing the number of layers is far more effective than increasing the complexity of the prior of a single-layer VIP model. Aside from the added computation time, which is rather minor, we see no drawbacks the use of DVIP instead of a single-layer VIP, but rather significant benefits.
The use of domain specific priors, such as CNNs in the first layer, has provided outstanding results in image-based datasets compared other GP methods. This establishes a new use of IPs with not-so- so-general prior functions. We foresee employing these priors in other domain specific tasks, such as forecasting or data encoding, as an emerging field of study. The prior flexibility also results in generalization of DGPs. As matter of fact, DVIP gives similar results to those of DGPs if a GP considered as the IP prior for each unit. Preliminary experiments in Appendix confirms this.
Despite the good results, DVIP presents some limitations: first of all, the implicit prior works as a black-box from the interpretability point of view. The prior parameters do not represent a clear property of the model in comparison to kemell parameters in standard GPs. Furthermore, even though using 20 samples from the prior has shown to give good results in some cases, there might be situations where this number must be increased, having a big impact in the model's 's training time. An unexpected result is that the cost of generating continuous samples from a GP prior in DVIP is too expensive. If GP prior is to be used, it is cheaper simply use DGP as the underlying model.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
Authors gratefully acknowledge the use the facilities of Centro de Computacion Cientifica (CCC) at Universidad Autónoma de Madrid. The authors also acknowledge financial support from Spanish Additional support was provided by the national
Plan Nacional I+D+i PID2019- 106827GB- 100, funded by MCIN/ AEI /10. 13039/39// 1009//5011000111 1033/ and FEDER, "Una manera de hacer Europa", as well as project T /10. 13039/500 10001 1033 and by the European Union NextGenerationFUU PRTR.
project
funded by MCIN/AEI
REFERENCES
P. Baldi, P. Sadowski, and D. Whiteson. Searching for exotic particles in high-energy physics with deep learning. Nature communications, 5(1):1-9, 2014.
T. Bui, D. Hernández-Lobato, J. Hernández- Y. Li, and R. Tumnr. Deep Gaussian processes for regression using approximate expectation propagation. In International conference machine learning, pages 1472-1481, 2016.
Y. Cho and L. Saul. Kernel methods for deep learning. Advances in neural information processing systems, 22, 2009.
K. Cutajar, V. Bonilla, Michiardi, and M. Filippone. Random feature expansions for deep Gaussian processes. In International Conference on Machine Learming, pages 884-893 2017.
A. Damianou and N. D. Lawrence. Deep Gaussian processes. In Artificial intelligence and statistics, pages 207-215, 2013.
D' Angelo and V. Fortuin. Repulsive deep ensembles are bayesian. Advances in Neural Information Processing Systems, 34:3451-3466 2021.
L. Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29:141-142, 2012.
D. Dua and C. Graff. UCI machine learning repository, 2017. URL http:://archive..tt uci, edu/ml.
V. Dutordoir, N. Durrande, and Hensman. Sparse Gaussian processes with spherical harmonic features. In International Conference Machine Learning, pages 2793-2802, 2020.
D. Duvenaud, O. Rippel, R. Adams, and 7. Ghahramani. Avoiding pathologies in very deep networks.
In Artificial Intelligence and Statistics, pages 202-210,, 2014.
D. Flam-Shepherd, J. Requeima, and D. Duvenaud. Mapping Gaussian process priors to Bayesian neural networks. NIPS Bayesian deep learning workshop, volume 3, 2017.
V. Fortuin. Priors in bayesian deep learning: review. International Statistical Review, 2022.
Y. Gal. Uncertainty in Deep Learning. PhD thesis, University of Cambridge, 2016.
A. Gelman, J. B. Carlin, H. S. Stern, D. B. Dunson, A. Vehtari, and D. B. Rubin. Bayesian data analysis. CRC press, 2013.
M. Havasi, J. M. HemándeezLoobtto and J. Murillo-Fuentes. Inference in deep Gaussian processes using stochastic gradient Hamiltonian Monte Carlo. In Advances in Neural Information Processing Systems, volume 31, 2018.
K. He, Zhang, Ren, and Sun. Identity mappings in deep residual networks. In European conference on computer vision, pages 630-645, 2016.
J. Hensman, N. Fusi, and N. D. Lawrence. Gaussian processes for big data. In Proceedings of the Tiitth-Ninth Conference on Uncertainty Artificial Intelligence, page 282-290, 2013.
Hensman, N. Durrande, A. Solin, et al. Variational fourier features for Gaussian processes. Journal of Machine Learning Research, 18(1):5537-5588. 2017.
10

---

Published as a conference paper at ICLR 2023
D. Hernández-Lobato, J. M. Hemándezz-Loato, and P. Dupont. Robust multi-class Gaussian process classification. In Advances in Neural Information Processing Systems, volume 24, 2011.
M. Hombndeez-oobto and R. Adams. Probabilistic backpropagation for scalable learning of Bayesian neural networks. In International conference machine learning, pages 1861-1869, 2015.
M. Hernández-Lobato, Li, M. Rowland, T. Bui, D. HemandezzLLoato,, and R. Turner. Black- box alpha divergence minimization. In International Conference on Machine Learning, pages 1511-1520, 2016.
A. Immer, M. Korzepa, and M. Bauer. Improving predictions of bayesian neural nets via local linearization. In International Conference on Artificial Intelligence and Statistics, pages 703-711. PMLR, 2021.
D. P. Kingma and J. Ba. ADAM: a method for stochastic optimization. In Intrernational Conference on Learning Representations, pages 1-15, 2015.
D. Kingma and M. Welling. Auto-encoding variational bayes. International Conference on Learning Representations, 2014.
N. D. Lawrence and A. J. Moore. Hierarchical Gaussian process latent variable models. In Proceed- ings of the 24th international conference on Machine learning, pages 481-488, 2007.
Lee, Bahri, Novak, S. Schoenholz, Pennington, and J. Sohl-Dickstein. Deep neural networks Gaussian processes. arXiv preprint arXiv:1711.00165 2017.
C. Ma and J. M. Hernández-Lobato. Functional variational inference based on stochastic process generators. In Advances in Neural Information Processing Systems, volume 34, pages 21795- 21807, 2021.
C. Ma, Li, and M. Hernández-Lobato. Variational implicit processes. In International Conference on Machine Learning, pages 1222-4233- 2019.
R. McAllister, Y. Gal, A. Kendall, M. Van Der Wilk, A. Shah, R. Cipolla, and A. Weller. Concrete problems for autonomous vehicle safety: Advantages of Bayesian deep learning. International Joint Conferences on Artificial Intelligence, Inc., 2017.
K. Murphy. Machine learning: a probabilistic perspective. MIT press, 2012.
A. Rahimi and B. Recht. Random features for large-scale kernel machines. In Advances in Neural Information Processing Systems, pages 1177-1184, 2007.
C. E. Rasmussen and C. K. I. Williams. Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning). The MIT Press, 2006.
S. Rodríguez Santana and D. Hemmáneez-Lobato Adversarial a-divergence minimization for Bayesian approximate inference. Neurocomputing, 471:260-2744 2022.
Rodríguez Santana, B. Zaldivar, and Hrrmández-Lobatee Function-space inference with sparse implicit processes. In International Conference Machine Learning, pages 18723-18740, PMLR, 2022.
G. Rudner, Z. Chen, W. Teh, and Gal. Tractable function-space variational inference in bayesian neural networks. In ICML 2021 Workshop Uncertainty and Robusiness in Deep Learning, 2021.
Sajda. Machine learning for detection and diagnosis of disease. Annu. Rev, Biomed. Eng., 537-565, 2006.
H. Salimbeni and M. Deisenroth. Doubly stochastic variational inference for deep Gaussian processes. In Advances in Neural Information Processing Systems, volume 30, 2017.
Shi, S. Sun, and J. Zhu. spectral approach to gradient estimation for implicit distributions. In International Conference Machine Learning, pages 4644 4653, 2018.

---

Published as a conference paper at ICLR 2023
P. N. Singh. Better application of Bayesian deep learning to diagnose disease. In 2021 5th Interna- tional Conference on Computing Methodologies and Communication (ICCMC), pages 928-934 IEEE, 2021.
S. Sun, G. Zhang, J. Shi, and R. Grosse. Functional variational Bayesian neural networks. In International Conference on Learning Representations, 2019.
A. G. Wilson, Z. Hu, R. Salakhutdinov, and E. P. Xing. Deep kernel learning. In Artificial intelligence and statistics, pages 370-378. PMLR, 2016a.
A. G. Wilson, Z. Hu, R. R. Salakhutdinov, and E. Xing. Stochastic variational deep kernel learning. Advances in Neural Information Processing Systems, 29, 2016b.
H. Yu, Y. Chen, B. K. H. Low, P. Jaillet, and Z. Dai. Implicit posterior variational inference for deep Gaussian processes. Advances in Neural Information Processing Systems, 32:14475-14486, 2019.
12


Reviews:
[{"review_id": "rh1ln4DpJb", "reviewer": "Reviewer_GNWm", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces Deep Variational Implicit Processes (DVIPs).  The main idea is to approximate the output of an implicit process (i.e., a random function from which it is easy to drawn samples) as a particularly simple Gaussian Process, and then stack several of these implicit processes with the output of one random function acting as the input to the next.  The authors present a variational inference objective for DVIPs and optimize this objective using sampling-based techniques.  The proposed framework generalizes Deep Gaussian Processes (DGPs) and also Variational Implicit Processes.  The authors apply DVIPs to a number of regression and classification experiments where the perform competitively with existing approaches like VIPs and DGPs.", "summary_of_the_review": "Overall, I think this paper presents an interesting approach that is for the most part clearly explained, shows promise on some preliminary benchmarks, and has well-documented code.", "strength_and_weaknesses": "Strengths:\n* The paper presents a comprehensive overview of related work and how the present work fits into this picture.\n* The idea of the DVIP itself is clean.  It is presented in a way where it is a natural extension of both deep gaussian processes and variational implicit processes.\n* The initial empirical experiments show that the approach could be useful in some settings.\n\nWeaknesses (roughly in order of importance):\n* At several points throughout the paper it is noted that DVIP is both a generalization of DGP and also that DVIP is substantially faster than DGP.  In some of the empirical results DGP outperforms DVIP (e.g., HIGGS) and the authors stat that using DVIP with GP priors should give results comparable to (but presumably faster) than DGP.  The authors should consider actually running these experiments (putting GP priors on DVIP) -- a faster method for performing accurate inference for DGPs would be of interest to the community.  If however, the accuracy is not as good, then it suggests that some of the approximations in the DVIP framework result in worse performance (e.g., sampling vs. using inducing points).  In any case, I think that the authors should either back up claims that DVIP can be a faster, but comparably accurate implementation of DGP, or remove such claims from the paper.\n* Figure 1 (right) was somewhat confusing to me.  The notation on the bottom suggests that each layer is a function of the inputs (x), but my understanding is that it is a function of the outputs of the previous layer (f^{(t-1)}).\n* minor comment: since the prior has hyperparameters that are learned from the data (the means and the variances of the weights in the BNNs), we might expect DVIP to be over-confident.  That is, with enough hyperparameters in the BNNs, DVIP could be overfit to the training data.  The authors mention that the prior mean and variance of the weights in each layer are constrained to be the same, and I suspect that this is to prevent overfitting.  It would be good to be more explicit about this possibility, and to include some results about regimes in which DVIP starts to become too overconfident. \n* minor comment: \"using 20 samples from the prior has shown to give remarkable results in most cases\" seems like too strong of a statement to use here. I do agree that it is interesting (remarkable even) that means and covariances can be estimated with sufficient accuracy using 20 samples, I think the phrasing here should be toned down (for the most part, the results are not _remarkably_ better than other approaches).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "YNbaRGOpqgP", "reviewer": "Reviewer_cZop", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces deep variational implicit processes (DVIP), a deep extension of the prior work \"variational implicit processes\" (VIP). It can be viewed as a direct generalization of deep Gaussian processes (DGP) by placing implicitly defined multivariate distributions over any finite multivariate marginal of the distribution over functions (e.g., Bayesian neural networks). The fundamental step is to approximate each layer of the implicit process with a GP whose mean and covariance functions are induced by the implicit process. Further linear approximation and reparametrization are adopted for practical training. Finally, the proposed method is validated against several standard benchmarks, with encouraging results.", "summary_of_the_review": "This paper presents an interesting extension of VIPs with solid technical parts and experimental results. The downside is that the novelty seems to be marginal.", "strength_and_weaknesses": "Strength:\n- The idea of extending the VIP to its deep variant is interesting and in the right direction.\n- Experimental results are thorough and encouraging.\n- The paper is well-written and easy to follow.\n\nWeaknesses:\n- Novelty and contribution: After reading the paper, I found the paper basically extends VIP to its deep version just like the way in the paper DSDGP (Salimbeni & Deisenroth, 2017). The derivations of DVIP are actually very similar to the derivations of DSDGP. It seems that incorporating the VIP priors into the framework of DSDGP is somewhat straightforward. It is unclear what is the main novel technical derivation or novel insights. (Nevertheless, this extension itself is also acceptable to me.)\n\nQuestions:\n- In Section 4,\n    > unlike that of DVIP and the non-linear transformation of DKL ignores epistemic uncertainty, unlike IPs.\n\n\n     This seems not true. DKL learns an NN-parametrized kernel function while maintaining a GP as the predictive distribution over functions, which represents epistemic uncertainty. In this sense, DKL and DVIP both learn parameters of prior so they can be viewed as empirical Bayesian methods. Have you considered these baselines besides DGPs?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QQxFpr2-KN1", "reviewer": "Reviewer_AATZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a deep extension of variational implicit processes (VIP, Ma et al. (2019)), in which an implicit process, defined as a (nonlinear) transformation of a collection of random variables, is approximated using a Gaussian process. Together with a Gaussian likelihood, this gives a Gaussian process posterior approximation for the implicit process. By extending VIPs to deep architectures, similarly to how deep GPs extend single GPs, the authors show that the resulting deep VIP (DVIP) model outperforms its shallow counterpart and achieves similar performance to a DGP, though with an impressive reduction in training time.", "summary_of_the_review": "The proposed model (DVIP) is a natural extension of the VIP, which could be a significant and potentially high-impact contribution to the field of Bayesian deep learning. The model shows good experimental performance, and the training time is particularly impressive. However, the model is simply a DGP (albeit more flexible than the traditional DGP) constructed from two well-known parts, VIPs and the doubly-stochastic DGP framework, limiting the novelty significantly. Further to this, the experiments do not always seem fair, which leaves one feeling somewhat sceptical about the claimed superiority of DVIP, which doesn't seem to be significantly better than a DS-DGP in terms of RMSE and NLL. Overall, I feel that the paper falls slightly below the acceptance threshold.", "strength_and_weaknesses": "### Strengths \nThe presented model (DVIP) is a very interesting contribution to the field of implicit processes (IPs) and Bayesian deep learning more broadly. It is a very natural extension of shallow IPs, and it is likely to be useful to a significant part of the community, in particular the more applied part. The performance in terms of RMSE and NLL is on par with DGPs, but DVIPs are much, much faster to train. In summary:\n- Significant contribution with potentially a high impact on the field.\n- Experimental results on par with DGPs, occasionally outperforming them.\n- Impressive computational performance.\n\n### Weaknesses \nWhile the contribution is certainly significant due to the impressive experimental results, the novelty appears limited. Despite the authors going through deriving the deep extension of VIPs, they are essentially just using the doubly-stochastic framework derived by Salimbeni & Deisenroth (2017). They can do so since VIPs (which the authors use for each node in the network) are simply GPs, only with an empirically determined mean vector and covariance matrix. This makes VIPs more flexible, but since they are simply GPs, they fit right into the doubly-stochastic DGP (DS-DGP) framework without needing any additional derivations. Thus, the two main components of the model, VIPs and the DS-DGP framework, are already fully developed and constructing the DVIP seems to be just a matter of combining the two. It is a good idea and a natural extension of VIPs, but I do not see any real novelty here. In section 4, the authors write that the DVIP is a generalisation of a DGP, but I disagree with that statement; the DVIP seems to be exactly a DGP. The argument seems to be (last paragraph of section 4) that the DVIP can use IP priors that are not necessarily GPs. While this is true (in the paper, the authors use a BNN as IP prior), the DVIP, as the name suggests, uses VIPs as nodes, which means that the model is still just a DGP; it just uses more flexible GPs as nodes rather than the usual sparse variational GPs.\n\nThe experiments are the other major weakness of the paper. While the experiments on the UCI datasets are very thorough, the remaining ones seem somewhat biased towards DVIP. In the interpolation experiment, the DVIP essentially achieves the same fit as an ordinary GP with a very simple covariance function (say, a squared exponential) would; adding such a baseline would be helpful. For the large scale regression experiment and the image classification experiment, only one (or two, in the case of MNIST) DS-DGP architecture is shown, whereas the authors typically show four architectures for the DVIP. Since the best DVIP architecture seems to vary between two and five layers, it doesn't seem fair only to allow the DS-DGP a single column in the tables. Furthermore, for the image classification experiment, the authors use a convolutional layer in their BNN, but they do not use a convolutional kernel (van der Wilk et al. (2017), Dutordoir et al. (2020)) for the DGP. This gives the DVIP an unfair advantage. Lastly, the authors mention that sparse IPs (SIPs) are more expressive than VIPs, being able to model multimodal outputs, though also more computationally demanding. I think a SIP would have made sense to include as an additional benchmark model to see if the deep structure of the DVIP is really necessary.\n\nIn summary:\n- Limited novelty (builds heavily on the DS-DGP framework).\n- The proposed model is essentially a DGP with a more expressive prior.\n- Experiments are not entirely fair or transparent. SIP would have been a good baseline to include.\n\nReferences:\n- Salimbeni & Deisenroth (2017): https://arxiv.org/abs/1705.08933\n- van der Wilk et al. (2017): https://arxiv.org/abs/1709.01894\n- Dutordoir et al. (2020): https://arxiv.org/abs/1902.05888", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Nv342okKYl9", "reviewer": "Reviewer_Lo8f", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a Deep Variational Implicit Process (DVIP) which is an extension of the existing VIP model in the same way that a Deep GP (DGP) is an extension of the GP. They show how to perform approximate inference in this model (combining ideas from DGP and VIP inference) and present the empirical performance of the model on a range of experiments.", "summary_of_the_review": "Overall, I think combining DGPs and VIPs is a nice idea and it is well presented here. Only the experiments could be done with a bit more care and discussed in more nuance, especially regarding the comparison to DGPs and using DVIPs with GP priors as a baseline. If the experiments would be improved, I'd be happy to recommend acceptance. For now, I will lean slightly toward rejection.\n\nUPDATE: Based on the changes made by the authors during the rebuttal, I have updated my score.", "strength_and_weaknesses": "Strengths:\n- Given that DGPs often work better than GPs and VIPs as well, combining them seems like a well-motivated idea.\n- The paper is overall clearly written.\n- The experiments are quite extensive.\n\nWeaknesses:\n- The experiments have mixed results and especially the comparison to the DGP could be done with more care.\n- The exact design of the right DVIP model for a given task could be discussed more, also related to prior work.\n\nMajor comments:\n- How principled is it in the model to fit the prior to the data? Surely, in a proper Bayesian model that wouldn't be allowed. It is said that overfitting is avoided by coupling the priors for different parts of the model, would there be a more principled way, e.g., using hyperpriors?\n- There are some related works on functional BNNs that could be mentioned, e.g. [1,2,3]\n- There are many works on priors regarding BNNs and stochastic processes, many of which should be mentioned (see [4] for an overview)\n- How much does width help in the DVIP model, especially given that the prior functions are the same for all units? Would it be similarly detrimental to the results in [4]?\n- I find Fig. 4 quite puzzling because this dataset is often used to show how the periodicity can be modeled. Even a simple GP with linear + periodic kernel can quite reliably do this. From this perspective, all the presented methods seem to miss the main feature of the data, which seems like a strong failure case.\n- Error bars would be useful in Tabs. 1 and 2, to judge the significance of the performance differences.\n- Contrary to the authors' claim, the DGP seems to yield a *better* likelihood on MNIST than the DVIP with CNN prior. Since the likelihood takes the full posterior into account and is a proper scoring rule, I would lend it more trust than the accuracy. How can the authors explain that the DGP outperforms the DVIP there?\n- It is claimed in many experiments where the DGP outperforms the DVIP that the DVIP could be used with GP prior to achieve the same results. I think this should be easy to try and would definitely warrant being included as a baseline. Particularly, I would be interested in the tradeoff between runtime and performance in those cases where both DGP and DVIP use the same GP priors.\n\nMinor comments:\n- Sec. 1: is is -> is\n- In eq. 6, why are there two products over n? I think there should only be one.\n- The notation is often quite laden with sub- and superscripts, maybe that could be simplified.\n- How sensible is it to train the model using only one mixture component (effectively a single Gaussian predictive, similar to the normal VIP or GP) and then use 100 components during testing? Would that not result in a stark shift of the kind of distributions that are predicted?\n\n[1] https://arxiv.org/abs/2008.08400\n\n[2] https://hudsonchen.github.io/papers/Tractable_Function_Space_Variational_Inference_in_Bayesian_Neural_Networks.pdf\n\n[3] https://arxiv.org/abs/2106.11642\n\n[4] https://arxiv.org/abs/2105.06868\n\n[5] https://arxiv.org/abs/2106.06529", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster