Neural Operator Variational Inference based on Regularized Stein Discrepancy for Deep Gaussian Processes
Authors: JIAN XU
Shian Du
Junmei Yang
Qianli Ma
Delu Zeng
Venue: Submitted to ICLR 2023 (2023)
Keywords: Deep Gaussian processes
Operator variational inference
Stein discrepancy

Abstract: A Deep Gaussian Process (DGP) model is a hierarchical composition of GP models that provides a deep Bayesian nonparametric approach to infer the posterior. Exact Bayesian inference is usually intractable for DGPs, motivating the use of various approximations. We theoretically demonstrate that the traditional alternative of mean-field Gaussian assumptions across the hierarchy leads to lack of expressiveness and efficacy of DGP models, whilst stochastic approximation often incurs a significant computational cost. To address this issue, we propose Neural Operator Variational Inference (NOVI) for Deep Gaussian Processes, where a sampler is obtained from a neural generator through minimizing Regularized Stein Discrepancy in L2 space between the approximate distribution and true posterior. Wherein a minimax problem is obtained and solved by Monte Carlo estimation and subsampling stochastic optimization. We experimentally demonstrate the effectiveness and efficiency of the proposed model, by applying it to a more flexible and wider class of posterior approximations on data ranging in size from hundreds to tens of thousands. By comparison, NOVI is superior to previous methods in both classification and regression.

Under review as a conference paper at ICLR 2023
NEURAL OPERATOR VARIATIONAL INFERENCE BASED ON REGULARIZED STEIN DISCREPANCY FOR DEEP GAUSSIAN PROCESSES
Anonymous authors Paper under double-bling brind review
ABSTRACT
A Deep Gaussian Process (DGP) model is a hierarchical composition GP mod- els that provides deep Bayesian nonparametric approach to infer the posterior. Exact Bayesian inference usually intractable for DGPs, motivating the use of various approximations. We theoretically demonstrate that the traditional alter- native of inean-ficld Gaussian assumptions across the hicrarchy leads to lack expressiveness and efficacy of DGP models, whilst stochastic approximation often incurs a significant computational cost. To address this issue, we propose Neural Operator Variational Inference (NOVI) for Deep Gaussian Processes, where a sampler is obtained from a neural generator through minimizing Regularized Stein Discrepancy in L2 space between the approximate distribution and true posterior. Wherein, a minimax problem is obtained and solved by Monte Carlo estimation and subsampling stochastic optimization techniques. We experimentally demonstrate the effectiveness and efficiency of the proposed model, by applying it to more flexible and wider class of posterior approximations on data ranging size from hundreds to tens of thousands. By comparison, NOVI is superior to previous methods in both classification and regression.
INTRODUCTION
Gaussian processes (GPs) Rasmussen & Williams (006) have proven to be extraordinarily effective as tool for statistical inference and machine learning, for example when combined with thresholding to perform classification tasks via probit models Rasmussen & Williams (2006); Neal (1997) or to find interfaces in Bayesian inversion Iglesias et al. (2016). However, the joint Gaussian assumption of the latent function values can be restrictive in number of circumstance: et al. (021I). This is due to at least two factors: first, not all prior information is purely expressible in terms of mean and covariance, and second, Gaussian marginals insufficient for many applications such as in the sparse data scenario, where the constructed probability distribution is far from posterior contraction. Thus, Deep Gaussian processes (DGPs) Damianou & Lawrence (2013) have been proposed to circumvent both of these constraints.
DGP model is a hierarchical composition of GP models that provides deep probabilistic nonpara- metric approach with sound uncertity quantification Ober & Aitchison (2021). The non-Gaussian distribution over composition functions yields both expressive capacity and intractable inference Dunlop et al. 2018). Previous work on DGP models utilized variational inference with a combination of sparse Gaussian processes Snelssn & Gharahmani (2005); Quunonero-Cannnne
& Rasmussen ; Gal
(2005) and mean-field Gaussian assumptions Hensman et al. (2015): Deisenroth & Ng (2015) (015)) al. (2014:; [Hensman et al. (2013): Hoang et al. (2015) 5| 2016); Titsias (2009b) for approximate posterior adjoint with stochastic optimization scale up DGPs to large datasets like (alimmeli Deisenroth| (2017). These strategies often incorporate a collection of inducing points (M < N) whose position is learned alongside the other model hyperparameters, reduicng the training cost to O (M?
While the field Gaussian assumptions of the approximate posterior simplifies the computation, these assumptions impose overly stringent constraints, potentially limiting the expressiveness and effectiveness of such deterministic approximation approaches for DGP models Havasi et al. (018);

---

Under review as a conference paper at ICLR 2023
Yu et al. (2019); Ustyuzhaninov et al. (2020): Lindinger et al. (0200) To solve the aforementioned problems, SGHMC Havasi et al. (2018) draws unbiased samples from the posterior belief using the stochastic approximation approach. However, due to its sequential sampling method, generating such samples is computationally expensive for both training and prediction, and its convergence is more challenging to evaluate in finite time Gao al. (202T). Despite previous literature Yu al.) (2019); Lindinger et al.] (2020) has discussed such issues, they all used different variants of the same KL-divergence--biie based variational bound, which not symmetric and usually not stable for oprimizttoorGooddelllow et al. 20: Huggins et al. (0018). In order to solve the above problems, we address the issue by operator variational inference Ranganath et al. (2016). Stein discrepancy based black-box algorithm that uses operators for optimizing any operator objective with data subsampling. where a minimax problem is obtained and solved by Monte Carlo estimation.
The main contributions are as follows:
We propose NOVI for DGPs, a novel variational framework based on Stein discrepancy and operator variational inference with a neural generator. It minimizes Regularized Stein Discrepancy in L2 space between the approximate distribution and true posterior to construct a more flexible and wider class of posterior approximations overcoming previous limitations caused by mean-fiel field Gaussian posterior assumptions and minimization of KL divergence.
We theoretically demonstrate that our training schedule is equivalent to optimizing the Fisher divergence between the approximation and the true posterior while the bias raised by our method can be bounded by Fisher divergence in Section 5
We experimentally demonstrate the effectiveness and efficiency of the proposed model on 8 UCI regression datasets and image classification datasets including MNIST, Feshion-MNIST and CIFAR-10 10, which outperforms state-of stute-of-the-ase art approximation methods. Additional ablation study demonstrates that our method is superior preventing overfitting.
Our code is publicly available at http://gtthbbcccmmm/uu///nninin0
PRELIMINARY
2.1 GAUSSIAN PROCESSES
Let random function f IRDD R map N training inputs X = [x to collection of noisy observed outputs y {yn J In general, a zero mean GP prior is imposed on the function f. i.e. f f~ GP(O, k) where & represents covariance function k R²D x IRD + R. Let (x } represent the latent function values at the inputs X. This assumption yields a multivariate Gaussian prior over the function values p(f) (f Kxx) where [K xxx]1 A xz)) In this work. we suppose y is contaminated by an i.i.d noise, thus pfy/f) = NNy f, o²I) where o² is the noise variance. The GP posterior of the latent output p (fly) has closed-form solution Rasmussen & Williams (006) but suffers from O(N3) computational cost and O(N2) storage requirement, thus limiting its scalability to big data.
Advanced sparse methods have been developed to set so-called inducing points Z (zm J (M & N) from the input space and the associated inducing outputs known as inducing variables: = (um f(zm) I Titsias (2009)): Snelson & Gharahmani (2005): Quiñonero-Canm -aandela & Rasmussen (2005), with a time complexity of O(NM)) In this Sparse GPs (SGPs) paradigm, inducing variables u share a joint multivariate Gaussian distribution with f: where the condition is specified as:
p(f, u) = p(f)
pfflu) N(KxzK zz Kxx
Kxzk KxzKzzkzn Kzx)
and p (u) = N (u K 2z) is the prior over the inducing outputs.
To solve the intractable posterior distribution of inducing variables p(uly), Sparse variational
GPs (SVGPs) Titsias (2009)); Hensman et al. (2015) reformulate the posterior inference problem as variational inference (VI) and confine the variational distribution to be q(f, (()
Hensman et al. (2013); Titsias (2009a); Gal et al. (2014); Salimbeni & Deisenroth (2017). This
method approximates (1) (m, Hensman al. (2015): 5); Deisenroth & (215)) Gal et al.
2

---

Under review as a conference paper at ICLR 2023
Hensman et al. y: Hoang al. (2015 2016); Titsias (2009b), then Gaussian marginal (2014);
is obtained by maximizing the evidence lower bound (ELBO) Hoffman et al. (2013).
2.2 DEEP GAUSSIAN PROCESSES
multi-layer DGP model is a hierarchical composition of GP models constructed by stacking the muti-output SGPs together Damianou & Lawrence (2013). Consider model with L layers and D independent random functions in layer used as an input to the é-th layer, i.e., F, {fr,1 = fe, (F,-1), Le ft,D, D. (Fa-1)). where
I.. ...... L such that output of the f-1-th layer Fę-1 is
ft.d GO ke) for d = 1, ..... Dj and Fo _ X. The inducing points and corresponding inducing variables for DGP layers are denoted by Z (Ze) and U {Ur) respectively where
U, {u,,1 fe,1 (Z)). ...., De Se.D, (Ze)-- Let F (Fe) the DGP model design
yields the following joint model density:
ply, F.U) = F FL) II (2 FF- (U)
(2)
t=1
Here we place independent GP priors within and across layers on U: p(U) = - p(U,) = IL De N (u, K 22) and in the same way as Equation ( the condition defined as: Dr
(F U2) = II d=1
II21 KF K Z, Ur.d KF KF,,, K K 2, K KZ
(3)
As an extension of Variational Inference with DGPs, DSVI Salimbeni & Deisenroth (017)) ap- proximates the posterior by requiring the distribution across the inducing outputs be p-sottriooii Gaussian and independent amongst distinct GPs to obtain an analytical ELBO (known as the mean- field assumption Opper & Saad (2001); Hoffman et (2013), 9 (U,,1:D,) = A (mi,1:D, St,.D,), ) where 1.,1 and St,1.D, are variational parameters. By iteratively sampling the layer outputs and utilizing the reparameterisation trick Kingma Welling (2013), DSVI enables scalability to big datasets.
As mentioned in Section [ while the mcan-fich field Gaussian assumptions of the variational posterior q(21) makes it simple to analytically marginalise out the inducing outputs, these assumptions impose overly stringent constraints, potentially limiting the expressiveness and effectiveness such deter- ministic approximation approaches for DGP models. In particular, according Bayes' Rule, the true posterior distribution can be written as:
3) (U) (
fp(Y, F.Y. oF (1)
(
Due to the fact that the latent functions F F1. FL-1 are inputs to the non-linear kernel function, the likelihood term p (y/U) in Equation (4) is intractable and p (U) y is often non- nan-Gassiia in reality. Moreover, the KL--based optimization often leads to unstable training Huggins et al. (2018).
To address this issue, we present and expressiveness based on Operator Variational Inference (OVI) Ranganath et al. (2016). while simultaneously learning preservable transformations and generating unbiased posterior samples constructed by neural networks, as detailed in Section and Section
variational family that provides both efficient computation
3 OVI AND STEIN DISCREPANCY
Definition 1. Let p(x) be a probability density supported on X C Id and &b : X + Rd be differentiable function, define Langevin-Stein Operator (LSO) Ranganath et al. (2016)
A, Va log p(x)² (bx)) ((x)).
Definition 2. (Stein's 's Discrepancy) Hu et al. (2018); Grathwohl et al. (2020); di Langosco al. (002)) Let p(x), q(x) be probability densities supported on X' C Rd. Stein discrepancy is defined by considering the maximum violation of Stein identity for in some proper function set F
S ( sup (e DeF
(6)
'The solution is given in App. A
3

---

Under review as a conference paper at ICLR 2023
Like previous methods Hu et al. (0018): Grathwohl et al. (2020), take the function space F in Stein discrepancy (6) to be the L2 space and parameterize & with neural network on as discriminator
LSD (g. P: n) A max (Bazg V~ log (xr) on (a:) Tr (V (x))))-
(7)
which is referred to the Learned Stein Discrepancy (LSD) Grathwohl et al. (2020). Neural networks as functions are not by definition square integrable, as they do not by default disappear at infinity. In
order to satisfy the conditions of Stein's 's identity[iuu & Wang (2016), La regularizer with strength
E IR+ is applied to LSD to gain a Regularized Stein Discrepancy (RSD)
RSD (q.p: m)) - max V (x) 6n (a) + Tr (VaΦn (z)) T Ag ( (x)) Φn (x)]}.
In Bayesian posterior inference, take p and qe the true posterior and approximate posterior, respectively, where e & and e is a set of variational parameters. Stein divergence in Equation (6) is usually used an objective of OVI Ranganaat et al. (2066), which is black-box algorithm that uses operators for optimizing any operator objective with data subsampling and a wider class of posterior approximations that does not require a tractable density. Given parameterizations of the variational family & and the discriminator Φny, OVI seeks to solve a minimax problem
r = arg inf sup Eez-go [Apon (a)) BEE
4 DEEP GAUSSIAN PROCESSES WITH NEURAL OPERATOR VARIATIONAL INFERENCE
Now we will discuss the algorithm design for the Bayesian inference problem of sampling the posterior p (UD) for DGPs. For consistency, we continue to use notation in Section 2.2 Let D {wn, Un ) represent the training dataset, U ( represent inducing variables and reprensent the DGP model hyperparameters including inducing points locations, kernel hyperparameters and noise variance.
4.1 NEURAL NETWORK AS GENERATOR
Let gn()) be the reference distribution that generates noise E Rdo, Let go represent our sampler, which a black-box generator parameterized by multi-layer neural network. Let qo (U) be the underlying density of the generated samples U go (e). In summary, our setup is follows:
go (E), Go (E) U qo (U)
Neural networks as a generator have high capacity and can well approximate almost any distribution by transforming simple ones such as Gaussian or uniform distribution with many applications in deep generative models Huszár (2017); Mescheder et al. (2017); Titsias & Ruiz (2019); Cybenko 1989); ; Lu & Lu 22020); (Perekrestekko et al. (2020); Yang et al. (022)). Since the generative distribution do (U) is implict, KL divergence is applicable measure between ge (U) and the true posterior
((U) in this case. Therefore, it is reasonable to use OVI and RSD to construct a better objective.
4.2 TRAINING SCHEDULE
In Section 3 we have reviewed OVI, method using Langevin-Ster operator and allowing for more flexible representation of the posterior geometry beyond the commonly used Gaussian distribution used in vanilla VI. We extend it to applications in inducing points posterior inference for DGP model by learning the parameters of neural network generator to best fit the data. Since our discriminator Φ,, is sufficiently expressive, we produce an objective whose expectation") is 0 if and only if the true posterior (D) and the approximate distribution g (U) are equivalent. During training, we will minimize
[( v) RSD (go (UD,(U
(10)
"The expectation doesn't 'itstiiinn include the regularization term.
4

---

Under review as a conference paper at ICLR 2023
with respect to f and jointly optimise the model hyperparameters by maximizing log- likelihood via Monte Carlo sampling. However, this procedure is difficult due to the supremum on r.h.s. of Equation (8). In order to obtain the optimized network parameters 0, we iteratively update the generator Go and the discriminator Φ,, in an alternating manner where the discriminator is trained to more accurately estimate the Stein Discrepancy and the generator is trained to minimize the estimation of the discrepancy. The proposed training algorithm is summarized in Algorithm which we refer to as Neural Operator Variational Inference (NOVI) for DGP.
Algorithm 1: NOVI for DGP
Input: training data = { [ In In-1> penalty parameter X, ne number of iterations for training the critic, learning rate a, 7,7, M batch size, sample number K Initialize discriminator 7), generator 0, DGP hyperparameters y repeat
N
for j = to n. do
Sample minibatch { y4 J: D
Generate i.i.d. noise inputs ..... EK from
Obtain fake sample ge (E1) ... 90 (Ek)
Compute empirical loss RSD(ga, p: Φn)
7 f- 7 V, RSD 2n))
end for Compute empirical loss (0,
0 - 3 V Vo (0,v)
V リ v - k Σ k= Vp loy 1'|p)
until 0, converge
In our implementation, we utilize Monte Carlo method to estimate the objective I0 and RSD(3)
RSD (go.P: P: Φb,)
Σ K
(Vulogr(U[DD v) u=ua (U*) Eu- (W" (U) سبا w))
K k=1
K Σ 1- (U*) (o,
[(0 RSS (m
(II)
where o, is the supremum of RSD estimate and the gradient with 0 and is computed via automatic differentiation. We use Hutchinson estimator Hutahiisnn (1989) to compute the expensive divergence of Φ. in Equation (11), which is simple yet effective way to obtain stochastic estimate of the trace of matrix. It can reduce the time complexity from O (D²) to O (D) where D is the dimensionality of the matrix. In Theorem [ we prove that the score function Vu log UD, v) can be evaluated by Monte Carlo method, which shows that RSD can be utilized as a reasonable objective to update the parameters of the generator network.
Theorem The score function Vu loggpt p( (U),.v) v) in Equation can be evaluated by Monte Carlo sampling (detailed proof can be seen in App. B):
A + log Σ S o= p())
Vu loogg(UUDD v)
(A
(12)
where
(K (KZ)) ZiZe ug, K K= Z. Ur ... K ZZ MEDDe) and
(())
N(K ((KK,,Z, Ku. KE,,f K 2, K 2.2, K 2,F,,,) for E = I,..., L, S is the
number of samples involved in estimation.
4.3 PREDICTION
Let D* = f y T be the test data, to predict its value, we sample from the optimized generator and convert the input locations to the test location x" in formula. We denote the function values at the test location as F.. To obtain the final layer density we use
9 (F
II
I
De p (T F u una)go (ux,a) dF
(13)
d=1
5

---

Under review as a conference paper at ICLR 2023
where 0* is the optimal of the generator and the first term of the integral aa 1 ug,d) is conditional Gaussian. We leverage this consequence draw samples from (F ]. and further perform the sampling using re-parameterization trick (Salimbeni & Deisenroth| (2017) Rezende et al.
(2014) Kingma al. (2015). Specifically, we first sample E¹ ~N, (Dt) and U qe (U), then recursively draw the sampled variables fi ~ p( Fi ue,a) for = ...... L as:
K K 2:2, u Ee Vdiag (KF; P/ K K 2,2, K 2
(14)
where the square root is element-wise. We define FÚj - X' for the first layer and use diag (-) to denote the vector of diagonal elements of a matrix. The diagonal approximation in Equation (4) holds since in DGP model, the i-th marginal of approximate posterior ((((,.)))))) depends only on the corresponding inputs x; Quinooero-Cander & Rasmussen (2005). our experiment, we concatenate
Ze and to generate U to avoid overfitting Yu et al. (2019).
5 CONVERGENCE GUARANTEES
Definition 3. The Fisher divergence Sriperumbudur et al. (2017) between two suitably smooth density functions is defined as
F (q. / || (x) logp ( (x) dx.
Theorem 2. Training the generator with the optimal discriminator corresponds to minimizing the fisher divergence between pa and The corresponding optimal loss is (detailed proof can be seen in App. C)
1
C
(go (U),,(U
4X
Theorem 3.
The bias of the estimation for prediction F7 in Equation 14) from the DGPs exact evaluation can be bounded by the square rool of the Fisher divergence between go U) and (U (U,,) up to multiplying a constant. (detailed proof can be seen in App. C)
Theoren shows algorithm is equivalent to minimizing Fisher divergence while Theorem guarantees a bounded bias of the estimation for prediction. Fisher divergence has proved useful in variety of statistics and machine learning applications Huggins et al.) (2018); Holmes & Walker (2017): Walker (2016). Connections between Fisher divergence and certain "rates of change" in KL divergence can seen de Bruijn's identity |Barron (1986); [Stam (1959) and Stein's 's identity Liu & Wang 6) Park et al. (2012). Under mild conditions, according to Sobolev inequality. Fisher divergence is stronger distance than KL divergence. fact, stronger than a lot of other distances between distributions, such total variation Chambolle (2004), Hellinger distance Beran (1977). Wasserstein distance Vallender (1974). etc Ley & Swan suitable Fisher divergence upper- upper-bounds bounds Wasserstein distance, which suggests that approximations based on minimizing the former, compared to KI. divergence, would lead to improved moment estimates.
et al. 2208)showw that a
RELATED WORKS
OVI and Stein Discrepancies Our method about the inference is inspired by OVI Ranganath et al.) (2016) and Stein Neural SampledHu et al. (2018) but the distinguish is ours concentrates on the DGP posterior and develop specific algorithms. Different from the general Bayesian model, the likelihood function of DGPs is not explicit, so we propose the stochastic gradient and Monte Carlo sampling method calculate the score function (Theorem I]. OVI Ranganaahh al.] (2016) introduces an objective for inference similar to RSD but utilize different class of discriminator and neither of the two methods Ranganath et al.) (2016); Hu al. (2018) applies many of state-of-the-an techniques we used for scalability such as Hutchinson estimator Hutchinson (189)..
Variational Inference Among the methods to address the limitations of mean-fieh variational inference, we can find methods that shares the same motivations ours for DGPs including adding

---

Under review as a conference paper at ICLR 2023
Boeton (N-506 D-13)
Energy (N-768 D-5)
Power (N-9568 D-f)
Concrete (N-1030 D-5)
DSVI
JPV DKCP
JPV 1X
JPVI DCP
JPVI DCP
SCEMC DCP
SCEIM DCP
SCIMC DCE
SCIMC 2
NOVI DOP
NOVI DGP4
NOVIESO DGP
NOVIDGP2
02 033 0.4
0.03 0.30 0.35
9.23 0.26
0.29
0225 0,30 0.35
Yacht (N-308 12m6)
Quar (N-546 12-8)
Protein (N-45730 D--9)
KinAom (N-8192 D-8)
DRVI
DRVI
DSV1
DSVI
SCHIM DCP
SCEIMC DCP
SCHMC DC2
SCIMC DXC
NOVI IDCP
NOVI D2
NOVI DGP
NOVIDGP2
0.03 0.66 0.99
0.45 0.50 0.55
0.67 0.75
038
0.25 0,30 0.35
Figure 1: Regression mean test RMSE results by NOVI method (bluc), SGHMC (orange), IPVI(pink) and DSVI (cyan) for DGPs on UCI benchmark datasets. Lower is better. The mean is shown with error bars of one standard error.
dependencies among the latent variables using structured variational family Fiiilyyinndi Lindinger al. (2020).
implicit distributions variational inference [ et al. (2017); Ma et al. (2019) Yu et al. (2019); [Sun al. (2019), and normalizing flowsReezende & Mohamed (2015); Yu et al. (2021).
Structured V!Lindinger et al. (220)) tends to design hand-crafted or expert solutions while the invertibility condition posts lots of constraints
the transformation form of normalizing flow:Dinh et al. (2016). The main issue of IPVI is density ratio estimation, which is often addressed using adversarial networks. Controlling the bias and variance density ratio estimation, however, in general a very difficult problem, especially in high dimensional setting BattingSSggiymaa al. (2012); Titsias) Ruiz (2019). In comparison, NOVI is more robust because it provides more accurate moment estimates Huggnas et al. (2018) than KL-bbased models, makes it feasible to handle higher dimensional latent variable SeeeibbttonnGrathwewe et al. (2020) and reaches convergence faster.
7 EXPERIMENTS
We empirically evaluate and compare the performance of our method with Doubly Stochastic VI (DSVI) (Salimbeni & Deisenroth (2017) for DGPs, which is implemented as our baseline model, Implicit Posterior VI (IPVI) for DGPs, which also constructs neural network to model posterior for approximate inference Yu et al. (2019) and state-of-the-art SGHMC model Havasi et al. (2018) using real-world datasets regression and classification tasks both in small and large data regimes. All our experiments were run with exactly the same hyper-parameters and initializations. Detailed training information can be seen in App. E.
7.1
UCI REGRESSION BENCIMARK
Our experiments are conducted on 8 UCI regression datasets with size ranging from 308 to 45730. The performance metric is average RMSE of the test data. The results are shown in Figure T (tabular version can be seen in App. D.3). On four the eight datasets, simply using 2-layer NOVI model achieves the best result and huge performance gap against other three methods. On larger datasets,
7

---

Under review as a conference paper at ICLR 2023
Data Sct
Model
Time (L=3)
Iter(I=3)
Acc (L=3)
Time (L=4)
Iter (L=4)
Acc (L=4)
DSVI
0.34s/iter
20K
0.54s/iter
20K
97.41
MNIST
IPVI
0,49s/iter
20K
0.62s/iter
20K
97.80
SGHMC
1.14s/iter
20K
1 1.22s/iter
20K
97.55
NOVI (ours)
0.38s/iter
10K
97.94
0,50s/iter
10K
98.01
DSVI
0.34s/iter
20K
0.50s/iter
20K
87.99
Aishion-MNIST
IPVI
0,48s/iter
20K
0.6ls/iter
20K
88.90
SGHMC
1 1.2ls/iter
20K
1 1.25s/iter
20K
87.08
NOVI (ours)
O, 0,40s/iter
10K
88.96
0,55s/iter
10K
89.15
DSVI
0,43s/iter
20K
0.66s/iter
20K
51.79
CIFAR-10
IPVI
0.62s/iter
20K
0.78s/iter
20K
53.27
SGHMC
8, 8.04s/iter
20K
8. 8.61s/iter
20K
52.81
NOVT (ours)
0.43s/iter
10K
53.32
0.52s/iter
10K
53.42
Table Mean accuracy (%) and training details achieved by DSVI, SGHMC and NOVI (ours)
DGP model for three image classification datasets. Batch size is set to 256 for all methods. L denotes the number of hidden layers. Our proposed method can also combined with convolution kernels Kumar et al. 2018 to obtain better result, for a fair comparison, we have not implemented here.
Type
DSVI
DSVI 3
DSVI
DSVI
Time (s)
0.835
0.903
0.965
1.339
Iteration
20K
20K
20K
20K
Type
IPVI 2
IPVI
IPVI 4
IPVI 5
Time (s)
0.117
0.162
0.211
0.260
Iteration
20K
20K
20K
20K
Type
SGHMC
SGHMC 3
SGHMC 4
SGHMC 5
Time (s)
0.630
1.000
1.490
1.870
Iteration
20K
20K
20K
20K
Type
NOVI 2
NOVI
NOVT 4
NOVI 5
Time (s)
0.391
0.613
0.863
1.123
Iteration
500
500
500
500
Table 2: Comparison of training time (s) of single iteration and total training iterations Energy dataset. Batch size is set to 1000 for all three methods. indicates that although IPVI takes less time per iteration, it requires larger training iteration converge, which more time-consuming than our method.
like 'Power', 'Concrete', 'Qsar' and 'Protein', the deepest NOVI model outperforms other methods. We attribute this phenomenon to the overfitting of the deep model on small data sets. Additional results for real-world regression datasets can be seen in App. D.5.
7.2 IMAGE CLASSIFICATION
We apply our method MNIST LeCun al. (1998), Fashion-MNIST Xiao al. (017) and CIFAR- 10 Krizhevsky et al. (2099) multic muliiclass lass classification problem. Both MNIST and Fashion-MNIST datasets are grey-scale images 28 x 28 pixels. The CIFAR-10 dataset consists of colored images of 32 x 32 pixels. Results are shown in Table F For all three datasets, NOVI outperforms other three methods with significant less training time and iterations. We also perform experiments using three UCT classification datasets and present results in App. D.1.
For 3-layer DSVI and SGHMC models, since they have not yet released the corresponding code to reproduce only the training time and report its iterations according the original paper.
8

---

Under review as a conference paper at ICLR 2023
Energy 08-756 D-6)
Powar (N-95588 D=4)
Boston (N-506 D-13)
Concrcte (K-1030 D-6)
1230
1,500
1.50
1:30
1335
11
11677
1167
1397
0335
0.53
0333
0335
0:51
0.90
0:36
059)
0311
0.33
6.33
033
0367
0.00
Figure 2: The mean RMSE comparison of NOVI (test: orange, train: red) with Monte Carlo log- likelihood maximization method (test: blue, train: cyan) using 2-layer DGP model on four UCI regression datasets.
7.3 COMPUTATIONAL COMPLEXITY
We have compared training efficiency with other three methods on a single GPU using Energy dataset. Results are shown in Table 2 It can be seen that when model takes less time per iteration than DSVI and SGHMC. Morever, we only need less than one-tenth of the number of iterations to converge compared with the other three methods. Also, as shown in Table [T] for high-dimensional image datasets, NOVI also requires significant less training time and iterations to converge, which shows that the proposed method scalable larger datasets. Comparison about numbers of inducing points can be seen in App. D.4.
7.4 ABLATION STUDY
To demonstrate the effectiveness of NOVI, we directly maximize log-likelihood with random initial- ized U and hyperparameters and compare with method using 2-layer DGP model. Results shown in Figure 2 For all datasets, it can be observed that NOVI yields lower test RMSE and higher train RMSE, hence indicating that our optimization method reduces overfitting. Although the loss fluctuation occurs during the training of our method, it is caused by the unique adversarial training and converges to stable value after only several hundred iterations. Additional results for ablation study on classification datasets can be seen in App. D.2.
8 CONCLUSION
This paper presented novel NOVI framework to incorporate Stein Discrepency with DGPs that can effectively model non-Gaussian and seraarhhy-related posterior, thus further enhancing the flexibility of DGP models. To achieve this, we generate inducing variables from a neural generator and optimize it jointly with variational parameters through adversarial training. Furthermore, we theoretically demonstrate that the bias raised by our method can bounded by Fisher divergence, which provides clear and concise tool to optimize the neural generator. Empirical evaluation shows that NOVI outperforms the state-of-art approximation methods both in regression and classification. The proposed method also requires significant less training time and iterations to converge, which shows that NOVI is more scalable to larger datasets. Due to the nature of adversarial training, NOVI will inevitably encounter fluctuations in loss during training, causing certain difficulties in optimization, but experimental results show that the fluctuations are greatly alleviated at convergence point. Future work includes implementing convolution structure to better extract features from images and utilizing Neural Architecture Search (NAS) method to obtain more suitable network architecture for practical applications.
REFERENCES
Andrew R Barron. Entropy and the central limit theorem. The Annals of probability, pp. 336-342, 1986.
We use Tesla V100 for all computations.
9

---

Under review as a conference paper at ICLR 2023
Rudolf Beran. Minimum hellinger distance estimates for parametric models. The annals of Statistics, pp. 443-466,11 1977.
Antonin Chambolle. An algorithm for total variation minimization and applications. Journal of Mathematical imaging and vision, 20(1):89-99) 2004.
George Cybenko.
Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems, 2(4):303-314, 1989.
Andreas Damianou and Neil Lawrence. Deep Gaussian processes. In Proc. AISTATS, pp- 207-215, 2013.
M. P. Deisenroth and W. Ng. Distributed Gaussian processes. In Proc. ICML, pp. 1481-1490, 2015.
Lauro Langosco di Langosco, Vincent Fortuin, and Heiko Strathmann. Neural variational gradient descent. arXiv preprint arXiv:2107.10731 2021.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv:1605.0880 2016.
Matthew M Dunlop, Mark A Girolami, Andrew M Stuart, and Aretha I. Teckentrup. How deep deep gaussian processes? Journal of Machine Learning Research, 19(54):1-46, 2018.
Vincent Dutordoir, James Hensman, Mark der Wilk, Carl Henrik Ek, Zoubin Ghahramani, and Nicolas Durrande. Deep neural networks as point estimates for deep gaussian processes. Advances 2021.
in Neural Information Processing Systems,
Y. Gal, M. van der Wilk, and Rasmussen. Distributed variational inference in sparse Gaussian process regression and latent variable models. In Proc. NeurIPS, pp. 3257-3265, 2014.
Xuefeng Gao, Mert Gürbüzbalaban, and Lingjiong Zhu. Global convergence of stochastic gradient hamiltonian monte carlo for nonconvex stochastic optimization: Nonasymptotic performance bounds and momentum-based acceleration. Operations Research, 2021.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Will Grathwohl, Kuan-Chieh Wang, Bern-Herrik Jacobsen, David Duvenaud, and Richard Zemel. Learning the stein discrepancy for training and evaluating energy-based models without sampling. In International Conference on Machine Learning, pp. 3732-3747. PMLR, 2020.
Marton Havasi, José Miguel Hernández-Lobato, and Juan José Kurillo-Fuentes Inference in deep Gaussian processes using stochastic gradient Hamiltonian Monte Carlo. In Proc. NeurIPS, pp. 7517-7527, 2018.
J. Hensman, N. Fusi, and N. Lawrence. Gaussian processes for big data. In Proc. UAI, pp. 282-290. 2013.
James Hensman, Alexander Matthews, and Zoubin Ghahramani. Scalable Variational Gaussian Process Classification. In Guy Lebanon and S. V. N. Vishwanathan (eds.), Proceedings of the Fighteenth International Conference on Artificial Intelligence and Statistics, volume 38 of Pro- ceedings of Machine Learning Research, pp. 351 351-360, San Diego, California, USA, 09-12 May 2015. PMLR. htth:tt:: Stctps://proceediee httpp//tceeeeinggnnecee mlr press v38/ htt//henmman11 .html
T. N. Hoang, Q. M. Hoang, and K. H. Low. A unifying framework of anytime sparse Gaussian process regression models with stochastic variational inference for big data. In Proc. ICMI, pp. 569-578, 2015.
N. Hoang, Q. M. Hoang, and K. H. Low. A distributed variational inference framework for unifying parallel sparse Gaussian process regression models. In Proc. ICML, pp. 382-391, 2016.
M. D. Hoffman, D. M. Blei, C. Wang, and Paisley. Stochastic variational inference. JMLR, 14(1): 1303-1347, 2013.
10

---

Under review as a conference paper at ICLR 2023
Chris C Holmes and Stephen G Walker. Assigning value to power likelihood in general bayesian model. Biometrika, 104(2):497-500 2017.
Tianyang Hu, Zixiang Chen, Hanxi Sun, Jincheng Bai, Mao Ye, and Guang Cheng. Stein neural sampler. arXiv preprint 1010:0081100354, 2018.
Jonathan H Huggins, Trevor Campbell, Mikotaj Kasprzak, and Tamara Broderick. Practical bounds on the error bayesian posterior approximations: nonasymptotic approach. arXiv preprint arXiv: 1809.09505 2018.
Ferenc Huszár. Variational inference using implicit distributions. arxiv:1702.082335 2017.
Michael F Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications Natistics-Simulltioon and Computation, 183)):1059-107,
1989.
Marco A Iglesias, Yulong Lu, and Andrew M Stuart. A bayesian level set method for geometric inverse problems. Interfaces free boundaries, 18(2):11-217, 2016.
Diederik P. Kingma and Max Welling. Auto-encoding variational Bayes. In Proc. ICLR, 2013.
Durk Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameterization trick. In Proc. NeurlPS, pp. 2575-2583, 2015.
Alex Krizhevsky, Geoffrey Hinton, Learning multiple layers of features from tiny images. 2009.
Vinayak Kumar, Vaibhav Singh, PK Srijith, and Andreas Damianou. Deep gaussian processes with convolutional kernels. arXiv preprint arXiv: 01655, 2018.
Yann LeCun, Corinna Cortes, and C Burges. Mnist handwritten digit database, 1998. URL htp://www.. research. att. com/ yann/ocr/mnist, 1998.
Christophe Ley and Yvik Swan. Stein's density approach and information inequalities. Electronic Communications Probability, 18:1-14, 2013.
Jakob Lindinger, David Reeb, Christoph Lippert, and Barbara Rakitsch. Beyond the mean field: Structured deep gaussian processes improve the predictive uncertainties. Advances in Neural Information Processing Systems, 1188498-8599 2020.
Qiang Liu and Dilin Wang. Stein variational gradient descent: A general purpose bayesian inference algorithm. Advances in neural information processing systems, 29, 2016.
Yulong Lu and Jianfeng
universal approximation theorem of deep neural networks for expressing probability distributions. Advances in neural information processing systems, 33: 3094-3105, 2020.
Chao Ma, Yingzhen Li, and José Miguel Hernández-Lobate Variational implicit processes. In International Conference on Machine Learning, pp. 4222-4233. PMLR, 2019.
Lars Mescheder, Sebastian Nowozin, and Andreas Geiger, Adversarial variational Bayes: Unifying variational autoencoders and generative adversarial networks. In Proc. ICML, pp. 2391-2400, 2017.
Radford M Neal. Monte carlo implementation of gaussian process models for bayesian regression and classification. arXiv preprint physics/9701026, 1997.
Sebastian W Ober and Laurence Aitchison. Global inducing point variational posteriors for bayesian neural networks and deep gaussian processes. In International Conference on Machine Learning, pp. 8248-8259, PMLR, 2021.
Manfred Opper and David Saad. Advanced mean field methods: Theory and practice. MIT press, 2001.
11

---

Under review as a conference paper at ICLR 2023
Sangwoo Park, Erchin Serpedin, and Khalid Qaraqe. On the equivalence between stein and de bruijn identities. IEEE Transactions on Information Theory, 58(12):7045-7067, 2012.
Dmytro Perekrestenko, Stephan Müller, and Helmut Bölcskei. Constructive universal high- dimensional distribution generation through deep relu networks. In International Conference on Machine Learning, pp. 7610-7619. PMLR, 2020.
J. Quiñonero-Candele and C. E. Rasmussen. A unifying view of sparse approximate Gaussian process regression. JMLR, 6: 6:999-19959 2005.
Rajesh Ranganath, Dustin Tran, Jaan Altosaar, and David Blei. Operator variational inference. Advances in Neural Information Processing Systems, 29, 2016.
C. E. Rasmussen and C. K. I. Williams. Gaussian processes for machine learning. MIT Press, 2006.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International conference on machine learning, pp. 1530-1538. PMLR, 2015.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proc. ICML, pp. 1278-1286, 2014.
Hugh Salimbeni and Mare Deisenroth. Doubly stochastic variational inference for deep Gaussian processes. In Proc. NeurIPS, pp. 4588-4599, 2017.
E. L. Snelson and Z. Gharahmani. Sparse Gaussian processes using pseudo-inputs. In Proc. NeurIPS, pp. 1257-1264, 2005.
Bharath Sriperumbudur, Kenji Fukumizu, Arthur Gretton, Aapo Hyvärinen, and Revant Kumar. Density estimation in infinite dimensional exponential families. Journal of Machine Learning Research, 18, 2017.
Aart J Stam. Some inequalities satisfied by the quantities of information of fisher and shannon. Information and Control, 2(2):101-112, 1959.
Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Density ratio estimation in machine learning. Cambridge University Press, 2012.
Shengyang Sun, Guodong Zhang, Jiaxin Shi, and Roger Grosse. Functional variational bayesian neural networks. arXiv preprint 1903vv/903.0555 2019.
M. K. Titsias. Variational learning of inducing variables in sparse Gaussian processes. Proc. AISTATS, pp. 567-574, 2009a.
M. K. Titsias. Variational model selection for sparse Gaussian process regression. Technical report, School of Computer Science, University of Manchester, 2009b.
Michalis K. Titsias and Francisco R. Ruiz. Unbiased implicit variational inference. In Proc. AISTATS, pp. 167-176, 2019.
Ivan Ustyuzhaninov, Ieva Kazlauskaite, Markus Kaiser, Erik Bodin, Neill Campbell, and Carl Henrik Ek. Compositional uncertainty deep gaussian processes. In Conference on Uncertainty in Artificial Intelligence, pp. 480 489. PMLR, 2020.
SS Vallender. Calculation of the wasserstein distance between probability distributions on the line. Theory of Probability & Its Applications, 18(4):784-786, 1974.
Stephen G Walker. Bayesian information in experiment and the fisher information distance. Statistics & Probability Letters, 1 11:55-9, 2016.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv 2017.
Yunfei Yang, Zhen Li, and Yang Wang. On the capacity of deep generative networks for approximating distributions. Neural Networks, 145:144-154 2022.
12

---

Under review as a conference paper at ICLR 2023
Haibin Yu, Yizhou Chen, Bryan Kian Hsiang Low, Patrick Jaillet, and Zhongxiang Dai. Implicit posterior variational inference for deep gaussian processes. Advances in Neural Information Processing Systems, 32, 2019.
Haibin Yu, Dapeng Liu, Bryan Kian Hsiang Low, and Patrick Jaillet. Convolutional normalizing flows for deep gaussian processes. In 2021 International Joint Conference on Neural Networks (IJCN), pp. 1-6, IEEE, 2021.
13


Reviews:
[{"review_id": "pqoL6Jy7FJJ", "reviewer": "Reviewer_vt6c", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose a novel framework to conduct approximate variational inference in deep Gaussian processes. It is based on minimizing the regularized Stein discrepancy between the approximate distribution and the true posterior through the newly proposed Neural Operator Variational Inference algorithm for DGPs, which iteratively updates a generator and the associated discriminator to estimate more accurately the target discrepancy. This leads to a better final approximation than the ones obtained through the regular mean-field Gaussian assumption usually employed in DGPs. Authors support this claimg by extensive regression experiments, in which they show also that the method presents important properties such as robustness to overfitting. ", "summary_of_the_review": "\nThe contribution seems correct and interesting to the research community both due to its implications in DGPs and VI. However, it may lack sufficient novelty to merit a higher score. The authors should make an effort to highlight in a stronger fashion how does this contribution differenciate from previous works. Some extensions to the current experimental setup could help the case here as well (it is fine already, but some simple additions would make it even stronger).      \n\n", "strength_and_weaknesses": "# Strengths: \n\n* The method is an interesting addition to the literature in Deep Gaussian Processes, a promising line of research that has layed out very interesting results for the past few years. \n\n* The approach is fairly easy to understand. Even though there are important details that must be covered in detail, for the most part it is pretty straight-forward. \n\n* The authors provide extensive experiments, accounting also for the scalability of the method and its applicability in different problems and datasets. This strenghtens the submission, as well as the convergence guarantees and the extra information provided in the supplementary material.\n\n* The paper is mostly well written and easily understandable, covering the most important points on previous works and providing a good description of the method's development.  \n\n\n# Weaknesses:\n\n* My main concern here is the novelty of the method, which I deem to be a bit lacking. Overall, I see the contribution as interesting to the community, and also potentially very useful. However, as far as I understand it, this method results from the combination of previously well-known previous contributions in a very direct manner. I may be mistaken, and in that case I am entirely open to change my opinion on this matter, but as it is I think the contribution may lack sufficient novelty.\n\n* I also suggest the authors to emphasize in a stronger note the differences between this method and previously existing contributions. This may help with the previous point as well.\n\n* The UCI experiments could be much more robust if other metrics were reported asides from the RMSE. Common choices are the negative log-likelihood or the continously ranked probability score, as a proper scoring rule. I suggest the authors to include some other metrics that help better assess the quality of the predictive distribution, since RMSE does not necessarily provide that information.\n\n* I would appreciate a more extensive experiment with bigger datasets than Energy (e.g. Taxi, Airlines or others similar to these ones). Also, it would help if the authors shed some light on the expected performance of the method in cases with high dimensionality.\n\n* (_minor_) The presentation of the tabulated results in the supplementary material should be revised.\n\n* (_minor_) The \"Related Work\" section seems limited, since there are several other approximate inference methods strongly related to this one. As an example, the NOVI setup proposed in section 4.2  strongly ressembles the construct made in articles like [5,6] that deal with implicit distributions for approximate inference. Moreover, formulation in the equation of section 4.1 makes me wish there was a discussion between the relation between these methods and function-space inference with implicit stochastic processes, since the two approaches actually have a lot in common. As suggestions, I see strong relationship with works such as [1,2,3,4]. \n\n* (_minor_) Figure 1 should be re-thought, its quality leaves some improvement to be desired to help understanding the content of the graphical models. \n\n\n## Other comments: \n\n* The language employed in section 4.2 makes it seem sometimes that the authors contribute here to the development of previous techniques, e.g. OVI (Ranganath et al. 2016). I suggest the authors to be clarify the language here to state what contributions are genuinely theirs.  \n\n* Please, clarify further the statement before Eq. 14 regarding the fact that the two distributions must be \"equivalent\"\n\n\n\n## References: \n\n\n[1] Ma, C., Li, Y., and Hernández-Lobato, J. M. (2019). “Variational implicit processes”.\nIn: International Conference on Machine Learning, pp. 4222–4233.\n\n[2] Sun, S., Zhang, G., Shi, J., and Grosse, R. (2019). “Functional variational Bayesian neural networks”. In: International Conference on Learning Representations.\n\n[3] Rodrı́guez Santana, S., Zaldivar, B., & Hernandez-Lobato, D. (2022). Function-space Inference with Sparse Implicit Processes. In International Conference on Machine Learning (pp. 18723-18740). PMLR.\n \n[4] Ma, C., & Hernández-Lobato, J. M. (2021). Functional variational inference based on stochastic process generators. Advances in Neural Information Processing Systems, 34, 21795-21807.\n\n[5] Mescheder, Lars, Sebastian Nowozin, and Andreas Geiger. \"Adversarial variational bayes: Unifying variational autoencoders and generative adversarial networks.\" International Conference on Machine Learning. PMLR, 2017.\n\n[6] Santana, S. R., & Hernández-Lobato, D. (2022). Adversarial α-divergence minimization for Bayesian approximate inference. Neurocomputing, 471, 260-274.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "dhyx5wAHg76", "reviewer": "Reviewer_C7ZF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new method for inference of the intractable posterior for deep Gaussian processes (DGPs). The idea is to minimize the Stein discrepancy between the approximated posterior distribution which is demonstrated by a generator and the true posterior distribution. The algorithm utilizes a discriminator to estimate the discrepancy between two distributions. It is claimed that a better approximation to the true posterior is recovered. To support the claim, extensive regression and classification experiments are conducted. ", "summary_of_the_review": "Given that the novelty and contribution of this paper are relatively incremental, I am inclined to reject this paper. Moreover, I would like to suggest the authors polish this paper in the direction of comparison of different inference methods for DGPs, the pros, and cons for each. ", "strength_and_weaknesses": "In fact, this paper demonstrates a high resemblance to Implicit Posterior Variational Inference for Deep Gaussian Processes [1]. The difference is that this work decides to minimize the Stein discrepancy while the work of [1] minimizes KL divergence. The design of the generator and discriminator, the iterative optimization procedure, and even the design to represent the inducing variables. \n\nThese two methods are not even compared in this paper, given all these resemblances. From my point of view, this comparison is compulsory. I am happy to raise my score if the authors can provide convincing reasons. \n\nIn the context of the NOVI algorithm, this paper is still far from satisfactory. The convergence guarantee is provided in Theorem 2 and Theorem 3 and yet no analysis or illustrations to prove this theorem. How would this method perform given a known true posterior distribution? This kind of synthetic experiment will make the claims more convincing than RMSE or classification accuracy. \n\n\n[1] https://arxiv.org/abs/1910.11998", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "cTZ8hzw8Xbp", "reviewer": "Reviewer_GwnC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper describes a learning and inference algorithm for deep Gaussian processes based on operator variational inference (OVI). \n\nParticularly, OVI (Ranganath et al., 2016, Hu et al. 2018, Grathwohl et al. 2020) generalizes Kernelized Stein Discrepancy (KSD) to situations where the RKHS space the smooth function (of the Langevin-Stein Operator) is replaced by the L2 space. Key to both OVI and KSD is the computation of the Stein Discrepancy (SD) which is parameterized by both the smooth operator and a surrogate of the true posterior that we wish to sample from. \n\nIn KSD, we can solve for SD in closed form which eases the optimization of the surrogate. In OVI, we parameterize the smooth function by a deep neural net and the SD can be estimated via sampling with re-parameterization & solving a minimax optimization task. Both OVI & SGD can be considered generic inference strategy for deep generative learning (including DGP). This paper extends OVI by also parameterizing the posterior with a neural net generator, which appears to be using the re-parameterization trick (Kingma & Welling, 2013). \n\nThis is demonstrated on deep Gaussian processes applied to 4 UCI regression datasets + 3 classification benchmark datasets (MNIST, Fashion-MNIST and CIFAR-10) showing marginal improvement over two baselines DSVI (Salimbeni & Deisenroth, 2017) and SGHMC (Havasi et al., 2018).", "summary_of_the_review": "Currently, my take of this paper is that it is an application of OVI to learning DGP. Despite the claim that it develops NOVI which is an extension of OVI via the neural net generator of the posterior in Section 4.1, this is not new. First, it is on high-level the same as re-parameterization which has been incorporated into KSD (a preliminary version of SD-based VI which OVI was built on). Second, even if we want to discuss it specifically in the context of stochastic inducing input, the same perspective has also been explored in a recent NeurIPS-19 paper -- http://web.mit.edu/~jaillet/www/general/neurips2019_main.pdf.\n\nGiven that no new algorithmic insight was introduced, I do not believe the current position of this paper merits an acceptance. But I do suggest another angle for this work (see my specific suggestions above) which is more meritable but that requires more extensive experiments, involving comparison with other families of inference algorithm for DGPs as well as detailed analysis of their modeling pros & cons (demonstrating explicitly deficiencies of existing approaches in the specific context of DGP; and showing how those could be remedied by using OVI). \n\n---\n\nThese are my preliminary assessment of this paper. I am of course happy to discuss if the authors disagree.", "strength_and_weaknesses": "Strengths:\n\n+ This paper is well-organized and has a good review section on important background of deep GP, Stein Discrepancy and SD-based inference techniques for generic deep Bayesian models such as KSD and OVI\n\n+ There are theoretical results characterizing the behavior of the optimal loss when the Stein Identity is satisfied; and an upper bound on the bias of the DGP predictor using the model learned via OVI.\n\nWeaknesses:\n\n- The contribution stance of this paper is somewhat problematic. It is in fact not clear to me what are the new algorithmic insights that the proposed algorithm (NOVI) provides in addition to what were found in OVI and KSD -- I will elaborate more on this in the Novelty section.\n\n- There is a lack of comparison with other relevant inference work for deep GP.\n\n- Despite the well-organized structure, this paper has quite a no. of typos -- some of which is quite problematic as it changes the semantic of the sentence into a contribution claim of an existing work -- more on this later.\n     ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WZ0d25XCo4L", "reviewer": "Reviewer_wts5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new method for approximating the posterior of deep gaussian processes, to allow for non-Gaussian posteriors.  The main idea is to train a generator network that approximates the posterior, while training a discriminator that learns a regularized version of the Stein discrepancy.  They also optimize hyperparameters and inducing point locations via SGD.  The authors also prove some theorems in the (unrealistic, but still interesting) regime of having infinitely expressive neural networks than can be perfectly optimized.", "summary_of_the_review": "The paper presents an interesting approach, but the key assumptions of its theorems are not met in practice (infinitely expressive neural networks, perfect optimization).  The impact of neural network architecture in particular, is not sufficiently explored.  I also found some parts of the paper to be presented in a way to appear rigorous, but are actually sort of pseudo-rigorous, at the expense of both clarity and accuracy (e.g., Theorem 1; repeated claims that neural networks with a finite number of hidden units are expressive enough to model any distribution).", "strength_and_weaknesses": "Strengths:\n* The approach is interesting, and the ability to learn complex posteriors in the DGP framework is promising.\n\nWeaknesses:\n* (minor) there are a number of typos throughout the manuscript. This did not interfere with my understanding, but the paper could use some thorough copy eding.   For example there are numerous citation typos (particularly missing spaces before citations in the text).  Also, e.g., \"In particular, according to Bayesian formula\" --> \"In particular, according to Bayes' Rule\";  \"a quick introduction to these concepts that forms the foundation of our method\" --> \"a quick introduction to these concepts that form the foundation of our method\"; and so on.\n* The theorems assume an infinitely expressive neural network architecture, but the role of network architecture on the empirical results is relatively unexplored (e.g., only number of layers, but not number of hidden units, convolutional architectures, etc...)\n* In table 1 in the appendix, why does the runtime increase only slightly with the number of inducing points?  If the runtime increases so slightly and accuracy increases substantially, then why not use substantially more inducing points.  E.g., spending an additional 0.011s (presumably per iteration?) on Concrete to go from 50 to 400 inducing points, but going from an RMSE of 0.28 to 0.19 seems like a very good tradeoff.  Why not go on to more inducing points?  When does accuracy plateau or when does the runtime become infeasible?\n* In equation (16), a non-random quantity on the left hand side is claimed to be equal to a random quantity on the right hand side.  This equality does not even hold in expectation (i.e., the gradient will be biased) as can be seen by applying Jensen's inequality to Equation (6) in the appendix.  And the authors are certainly aware that this is not a strict equality given the $\\approx$ in Equation (6) in the supplement.  As such, I think that calling this a \"theorem\" with a strict equality is somewhat disingenous. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject