Deep Neural Networks as Point Estimates for Deep Gaussian Processes
Authors: Vincent Dutordoir
James Hensman
Mark van der Wilk
Carl Henrik Ek
Zoubin Ghahramani
Nicolas Durrande
Venue: NeurIPS 2021 Poster (2021)
Keywords: Gaussian processes
Deep Gaussian processes
variational inference

Abstract: Neural networks and Gaussian processes are complementary in their strengths and weaknesses. Having a better understanding of their relationship comes with the promise to make each method benefit from the strengths of the other. In this work, we establish an equivalence between the forward passes of neural networks and (deep) sparse Gaussian process models. The theory we develop is based on interpreting activation functions as interdomain inducing features through a rigorous analysis of the interplay between activation functions and kernels. This results in models that can either be seen as neural networks with improved uncertainty prediction or deep Gaussian processes with increased prediction accuracy. These claims are supported by experimental results on regression and classification datasets.

Deep Neural Networks as Point Estimates for Deep Gaussian Processes
Vincent Dutordoir University of Cambridge Secondmind
James Hensman" Amazon
Mark van der Wilk Imperial College London
Carl Henrik Ek University of Cambridge
Zoubin Ghahramani University of Cambridge
Nicolas Durrande Secondmind
Google Brain
Abstract
Neural networks and Gaussian processes complementary in their strengths and weaknesses. Having better understanding of their relationship comes with the promise to make each method benefit from the strengths of the other. In this work, we establish an equivalence between the forward passes of neural networks and (deep) sparse Gaussian process models. The theory we develop is based on inter- preting activation functions interdomain inducing features through rigorous analysis the interplay between activation functions kernels. This results models that can cither be seen as neural networks with improved uncertainty predic- tion deep Gaussian processes with increased prediction accuracy. These claims are supported by experimental results on regression and classification datascts.
1 Introduction
Neural networks (NNs) [1] and Gaussian processes (GPs) [2] are well-cstablished frameworks for solving regression classification problems, with complementary strengths and weaknesses. NNs work well when given very large datasets, and are computationally scalable enough handle them. GPs, on the other hand, are challenging to scale to large datasets, but provide robust solutions with uncertainty estimates in low-data regimes where NNs struggle. Ideally, would have single model that provides best of both approaches: the ability to handle low and high dimensional inputs, and to make robust incertainty-awan predictions from the small to big data regimes.
Damianou and Lawrence [3] introduced the Deep Gaussian process (DGP) promising attempt to obtain such a model. DGPs replicate the structure of deep NNs by stacking multiple GPs as layers, with the goal of gaining the benefits of depth while retaining high quality uncertainty. Delivering this potential in practice requires efficient and accurate approximate Bayesian training procedure, which is highly challenging develop. Significant progress has been made recent years, which has led to methods outperform both GPs and NNs in various medium- diieum--iinniinal tasks [4. 5]. In addition, some methods [5. train DGPs in ways that closely resemble backpropagation which has also greatly improved efficiency compared to early methods [3]. However, despite recent progress, DGPs are still cumbersome train compared to NNs. The similarity between the training procedures sheds light on a possible reason: current DGP models are forced to choose activation functions that are known to behave poorly in NNs (e.g., radial basis functions).
NNs,
In this work we aim to further unify DGP and NN training, practices that are known to work for NNs can be applied in DGPs. We do this by developing a DGP for which propagating through
Work done while at Secondmind. Correspondence to ad3099cam.ac.c.
35th Conference Neural Information Processing Systems (NeurIPS 2021).

---

the mean of each layer is identical to the forward pass of a typical NN. This link provides practical advantages for both models. The training of DGPs can be improved by taking best practices from NNs, to the point where a DGP can even be initialised from a NN trained in its usual way. Conversely, NN solutions can be endowed with better uncertainty estimates by continued training with the DGP training objective.
2
Related Work
Many different relationships between GPs and NNs have been established over the years. These relationships mainly arise from Bayesian approaches to neural networks. Finding the posterior distribution neural network weights is challenging, closed-form expressions do not exist. As consequence, developing accurate approximations has been key goal since the carliest works on Bayesian NNs [7]. While investigating single-layer NN posteriors, Neal [8] noticed that randomly initialised NNs converged to a Gaussian process (GP) in the limit of infinite width. Like NNs, GPs represent functions, although they do not do so through weights. Instead, GPs specify function values at observed points, and kernel which describes how function values at different locations influence each other.
This relationship was of significant practical interest, as the mathematical properties of GPs could (I) represent highly flexible NNs with infinite weights, and (2) perform Bayesian inference without approximations. This combination was highly successful for providing accurate predictions with reliable uncertainty estimates [9, 10]. To obtain models with various properties, GP analogues were found for infinitely-wide networks with various activation functions [11. 12] including the ReLU [13]. More recently, Meronen al. [14] investigated the relationship in the opposite direction, by deriving activation functions such that the infinite width limit converges to given GP prior.
Since the growth of modern deep learning, relationships have also been established between in infinitely wide deep networks and GPs [15, 16, 17]. Given these relationships. one may wonder whether GPs can supersede NNs, particularly given the convenience of Bayesian inference in them. Empirically though, finite NNs outperform their GP analogues [16, 18, 19] on high-dimensional tasks such images. MacKay [12] explained this by noting that NNs lose their ability to learn features in the infinite limit, since every GP can be represented as single-layer NN, with fixed features [20, 21]. This observation justifies the effort of performing approximate Bayesian inference finite deep networks, so both feature learning and uncertainty can he obtained. Renewed effort in Bayesian training procedures has focussed computationally scalable techniques that take advantage of modern large datasets, and have provided usable uncertainty estimates [e.g., 22, 23, 24, 25].
However, questions remain about the accuracy of these approximations. For accurate Bayesian inference, marginal likelihoods are expected to be usable for hyperparameter selection [26, 27]. For most current approximations, there is either no positive or explicitly negative [233] evidence for this holding, although recent approaches do seem to provide usable estimates [28, 29].
DGPs [3] provide an alternative approach to deep NNs, which use GP layers instead of weight- based layers, in order to take advantage of the improved uncertainty estimates afforded by having an infinite number weights. The DGP representation is particularly promising because both early and recent work [3, 30, 31] shows that marginal likelihood estimates are usable for hyperparmeter selection, which indicates accurate inference. However, currently scalability and optimisation issues hinder widespread use.
In this work, we provide a connection between the operational regimes of DGPs and NN, deriving equivalence between the forward pass of NNs and propagating the means through the layers of a DGP. We share this goal with Sun et al. [32], who recently proposed the use of neural network inducing variables for GPs based on the decomposition of zonal kernels in spherical harmonics. However, compared to Sun al. [32], our method does not suffer from variance over-estimation, which fundamentally limits the quality of the posterior. Our approach also leads to proper GP models, as opposed to adding variance term NNs using Nystöm approximation, which allows the correct extension to DGPs and the optimisation of hyperparameters. These improvements are made possible by the theoretical analysis of the spectral densities of the kernel and inducing variable (Section 4.3).

---

a
xe
(o)
- (WW))
(x)))
Dal
by (ira(ha
Blc.(ha)
(a) Deep Gaussian process posterior
(b) Fully-connected deep neural net
Figure 1: visual representation of propagating the mean of each layer through the DGP (a) and DNN structure (b). The goal is to design basis functions Cu, (-) for the DGP that match the activation functions oW .) in the DNN.
3 Background
In this section, we review Deep Gaussian processes (DGPs), with a particular focus on the structure of the approximate posterior and its connection to deep NNs. We then discuss how inducing variables control the DGP activation function, which is key ingredient for our method.
3.1 Deep Gaussian Processes and Sparse Variational Approximate Inference
GPs are defined as an infinite collection of random variables (f(=)) such that for all n E IN, the distribution of any finite subset (f..)...... (xx)) } is multivariate normal. The GP is fully specified by its mean function j((x)) = [x)) and kernel k(ar,, æ²) = Cf ((zz)) fax')), shorthanded as f() ~ GP (P(((-), k(.,)) )). GPs are often used as priors over functions in supervised learning, as for Gaussian likelihoods the posterior f(-) D given a dataset = (x, E Rd, V. E R is tractable.
Surprisingly diverse properties of the GP prior can be specified by the kernel [2, 33, 34, 35, 36), although many problems require the flexible feature learning that deep NNs provide. DGPs [3] propose to solve this a fully Bayesian way by composing several layers of simple GP priors,
F(-) fL .... f2 f1,
where fe(.) GP (0, k))
(1)
intractable posterior F(.) DD We follow the variational approach by Salimbeni and Deisenroth [5] due to its similarity to backpropagation in NNs. They use an approximate posterior consisting of an independent sparse GP for each layer. Each sparse GP is constructed by conditioning the prior M inducing variables [37, 38, 39], commonly function values u, = {ur" = f((w"") } and then specifying their marginal distribution q(u))) = NN Ez). This results in the approximate posterior process:
Inference in DGPs is challenging the composition is longer GP, leading
a(fe())) =P GP Cu ke(:.") () + +c,, (.)C)L Mpn (E, Cu,u,) Cur ('))
(2)
where B, = Cuju, Mde, de the dimensionality the GP's output, Guy = Cov(fe(-). ux) and Curur Cov(u,, ug). It worth noting that, commonly used 'K', to denote the fact that these matrices contain covariances which are not necessarily simple kernel evaluations. The variational parameters μ4 E Mxx and Σ, E M (KL) divergence from the variational distribution
the symbol 'C', rather than the
are selected by reducing the Kullback- the true posterior. This is equivalent to maximising the Evidence Lower BOund (ELBO). Taking the evaluations of the GP h; = fe(he-1), the ELBO becomes [5]:
ELBO
(lo...)llogg( (loggoggo he,t)] Σ'e t=1
KL(g(ue) [(()) log
(3)
3.2 Connection between Deep Gaussian processes and Deep Neural Networks
Hensman and Lawrence [6] observed that the composite function of propagating an input through each layer's variational mean (Eq. (2)) of DGP equals:
o)] E-)f(()) = B,, (.) B, Cu, (:),
(4)

---

which resembles the forward pass through fully-connected NN layers with non- con-linearity o-)):
WW- Vg () V ((W.)
(5)
with We and Ve the pre-activation and output weights, respectively. Both models are visualised in Fig. 1. Indeed, if as a typical neural net activation function o oWW.) and set B, cqual to Vt, we obtain a formal mathematical equivalence between the forward pass of DNN and propagating the mean of cach layer through a DGP. This one of the main contributions this work. The remaining difference between the two models are then the so-called "bottleneck" layers in the DGP: h1 and h2 in Fig, la. This is consequence of the DGP explicitly representing the output each layer. However, while NN does not explicitly represent the outputs, low-rank structure in the matrices W and W3 V2 is typically found after training [40], which strengthens the connection between both models.
formulate an approximation that makes the covariance Cu,() (-) the same
3.3 Interdomain Inducing Features
The basis functions used in the approximate posterior mean (Eq. (2)) are determined by the co- variance between the inducing variables and other function evaluations [Cu(-)) (f)( um). Commonly, the inducing variables are taken to be function values un = f (Wm). which leads to the kernel becoming the basis function [ca(()] different properties of the GP (e.g. integral transforms um = f(x)
k(w..: -) Interdomain inducing variables [41] select
(z) gm (2 .dz), which modifies this covariance (see [42, 43J for an overview). and therefore gives control over the basis functions. Most current interdomain methods are designed to improve computational properties [44, 45, 46). Our aim is to control Cu2 (-) to he a typical NN activation function like a ReLU or Softplus. We share this goal with Sun et al. [32], who recently proposed the of NN inducing variables for GPs based on the decomposition of zonal kernels spherical harmonics.
3.4 The Are Cosine Kernel and its associated RKHS
The first order Are Cosine kernel mimics the computation of infinitely wide fully connected layers with ReLU activations. Cho and Saul [13] showed that for (t) max(0, t), the covariance between function values of f f(x) = o(w" x) for w ~ ~N 1/2Ig) and w E IR is given by
k(x,, x²) = Ew [o 2)) a a²)] = =|| ||||||||||
(V1 f + (( arccos t)).
(6)
radial
ungular (shupe function) s(t)
where f = m
|| || The factorisation of the kernel in a radial and angular factor leads to an RKHS
consisting of functions of the form f(z) | 9( | where 9(-) is defined on the unit hypersphere gd-1 {æ e Rd ||ا||| = 1) but fully determines the function on Rd.
The shape function can be interpreted as a kernel itself, since it is the restriction of k(-,)) .) to the unit hypersphere. Furthermore its expression only depends on the dot-produce between the inputs so it is zonal kernel (also known angular part of k(.,,) .) are the spherical harmonics on.j (we index them with a level n and an index within each level j E {I,..., Nd)) [46, 48]. Their associated eigenvalues only depend on n:
dot-product kernel [47]). This means that the eigenfunctions of the
[ -1) dt, clo0 (1)
Wd
Am
where Cia)( (-) the Gegenbauer polynomial of degree n. a d-2 wa is constant that depends on the surface area of the hypersphere. Analytical expressions of X, are provided in Appendix C. The above implies that i admits the Mercer representation:
x
k(æ, x²) = |
|
Anon
Φn.j
(8)
||00||
|
n=0
j=1
"See Appendix B for primer Gegenbauer polynomials and spherical harmonics.

---

(a) ReI.U: (t) = max(0.
(b) Softplus: s(t) log(1 + exp(3t))
Figure Activated inducing function Im (2) = ||^||| [wmll a (w x; wm ||=||) where (.)
correspond the ReLU (a) and Softplus (b). Although the input domain is R we only plot the value of the function on the unit circle @ (orange), and on the subspace that has an offset of in the 22 direction (blue). The linear radial component of 9m creates one-to-one mapping between the blue curve and the upper half of the orange one.
and that the inner product between any two functions g, Hy is given by:
In. In.g plmn.
(g.
(9)
14
An
n..
where fn.y and hay are the Fourier coefficients of g and h. i.c. g(x) = Emg (nsl|=/]on (
4 Method: Gaussian Process Layers with Activated Inducing Variables
The concepts introduced in the background section can now be used to summarise our approach: We consider DGP models with GP layers that have Are Cosine kernel and Variational Fourier Feature style inducing variables umm = (f(-), 9m (-)) Rt [44]. We then choose inducing functions gm(()) that have the same shape as neural network activation functions (Section 4.1). This yields basis functions for the SVGP model that correspond to activation functions (Section 4.2), and thus
model whose mean function can be interpreted as classic single-layer NN model. By stacking many of these SVGP models top of cach layers of DGP, obtain DGP for which propagating the mean of each layer corresponds to neural network. Section 4.3 covers the mathematical intricacies associated with the construction described above.
4.1 Activated Inducing Functions and their Spherical Harmonic Decomposition
The RKHS the Are Cosine kernel consists solely of functions that are equal to zero at the origin, i.e. Yf H4 f0) = 0. To circumvent this problem we artificially increase the input space dimension by concatenating a constant each input vector. In other words, the data space embedded in larger space with an offset such that it does not contain the origin anymore. This is analogous to the bias unit in multi-layer perceptron (MLP) layers in neural networks. For convenience will denote by (d-1) 1) the dimension of the original data space (i.e. the number of input variables), and by the dimension of the extended space on which the Arc Cosine kernel is defined.
The inducing functions (.)()) play an important role because they determine the shape of the SVGP's basis functions. Ideally they should be defined such that their restriction to the (d diddieensional original data space matches classic activation functions, such as the ReI.U or the Softplus, exactly. However, this results in angular component for (-) that not necessarily zonal. Since this property will be important later on, we favour the following alternative definition that enforces zonality:
9m Rd R,
E + |||||
a
(10)
with Wm. E Rd parameter, and o [-1, + R the function that determines the value of gm on the unit hypersphere. In Fig. 2, we show that choosing a(-) to be a ReLU (t max(0, t)) or

---

Softplus (a((t) = log(1 + exp(3t)) activation function leads to inducing functions that closely resemble the classic ReLU and Softplus the data space. In the specific case of the ReL.U it can actually be shown that the match exact, because the projection of the ReLU to the unit sphere leads to zonal function. The parameter Wm R determines the orientation and slope of the activation function- play same role as the pre-activation weights W a NN (cref Eq. (5)).
The zonality that we enforced in Eq. (10) particularly convenient when comes to representing Im (-) in the basis of the eigenfunctions of Ha. which is required for computing inner products. It indeed allows us make use of the Funk-Hecks theorem (see Appendix B) and obtain
No
8
Wm
Im (z)
||00||
1mmm
|/wmlll
On
x
n=0
j=1
(I))
Wil
S o(t) Clo (1 P)
= ddd dt.
where
On
C/aal (1)
Analytical expressions for on when (t) max(0. are given Appendix C.
4.2 Activated Interdomain Inducing Variables
We define our activated interdomain inducing variables as
Um (f(-), 9m (m)))mm.)))))
(12)
which is the projection of the GP onto the inducing function in the RKHS, as was done [32, 44, 46]. However, since the GP samples do not belong to the RKHS there mathematical subtleties associated with such a definition, which are detailed in Section 4.3. Assuming for now that they are indeed well defined, using these interdomain inducing variables as part of the SVGP framework requires access to quantities: (i) their pairwise covariance, and (ii) the covariance between the GP and the inducing variables. The pairwise covariance, which is needed to populate Cuu, is given by
Cov(Ume Uame) (99m())) 9m (-)) Me
oh +
C((a)
W W''
(13)
An
a
||mmm]]
The above is obtained using the RKHS inner product from Eq. (9), the Fourier coefficients from Eq. (11) and addition theorem for spherical harmonics from Appendix B. Secondly, the covariance between the GP and My,. which determines (cu(())m. is given by:
Cov(Um: f()) = (k(x, 9m (m) 9m (x)
(14)
as a result of the reproducing property the RKHS. becomes clear that this procedure gives rise to basis functions that are equal to our inducing functions. By construction, these inducing functions match neural network activation functions in the data plane, shown in Fig. 2. Using these inducing variables thus leads to an approximate posterior GP (Eq, (2)) which has a mean that is equivalent to a fully-connected layer with a non-linear activation function (e.g. ReLU, Softplus, Swish).
4.3 Analysis of the interplay between kernels and inducing functions
In this section describe the mathematical pitfalls that can be encountered [e.g., 32] when defining new inducing variables of the form of Eq. (12), and how we address them. We discuss two problems: I) the GP and the inducing function are not part of the RKHS, 2) the inducing functions are not expressive enough explain the prior. Both problems manifest themselves in an approximation that is overly smooth and over-estimates the predictive variance.
The Mercer representation of the kernel given in Eq. (8) implies that we have direct access to the Karhunen-Loove representation of the GP:
N
DC
f(x)
End A
where the En.j are i.i.d. A/(0 1).
(15)
||a0||

---

Exact
Apgroximation x²
Approximation A*
Approximation g
Betu Softolas
N=5
.... N=5
R=5
2-
.... R2=3
R-25
0.5
-1
0
2
-1
0
-1
0
17
-1
1
Figure 3: The ReLU and Softplus activation function and its approximation for different truncation levels and dimensions. These functions correspond to the orange fuunciion in Fig. 2 plotted on a line rather tham on the circle. Approximating the ReLU in larger dimensions becomes challenging.
Using this expression to compute the RKHS norm of a GP sample f()) yields 11-1 SR.j which is diverging series. This is clear indication that the GP samples do not belong to the RKHS [49], and that expressions such as 9.)()))())))))))))))))))())))))) g())) should be manipulated with care. According to the definition given in Eq. (9), the RKHS inner product is an operator defined over Hr H2 R. Since it is defined as a series, it is nonetheless mathematically valid to use the inner product expression for functions that are not in Hy provided that the series converges. Even if the decay of the Fourier coefficients of f()) is too slow make it an element of H4, if the Fourier coefficients of 9(-) decay quickly enough for the series En. Enggna/ VAn, converge then (f(-), (())))((())))))))) g())) M4, is well defined.
The above reasoning indicates that, for a given kernel k(:;)) .), some activation functions Im. (-) will result in inducing variables Umn = (f(-), (m)m))))) that well defined whereas other activation functions do not. For example, if we consider the case of the Arc Cosine kernel and the ReL.U inducing function, the decay rate of o, is proportional to the square root of An [47, 50]. This implies that the inner product series diverges and that this kernel and inducing variable cannot be used together. Alternatively, using smoother activation functionn for (t),, such as the Softplus, results in a faster decay of the coefficients o,, and can guarantee that inducing variables well defined.
An alternative to ensure the series convergence for any combination of kernel and activation function is to use a truncated approximation of the activation function 9m.(.) where all the Fourier coefficients above a given level N are set to zero, which basically turns the inner product series into a finite sum. Figure? 3 shows how the truc and truncated activation functions for the ReLU and Softplus compare. These correspond to the orange functions in Fig. 2, but are now plotted on line. In the low to medium dimensional regime, we see that even for small truncation levels we approximate the ReLU and Softplus well. In higher dimensions this becomes more challenging for the ReLU.
Unexpressive inducing variables through truncation (Fig. 4) The main concern with this trun- cation approach, however, comes from elsewhere: the larger N is. the closer (-) is to 9.m().. but the larger (lām (-)]l H. becomes (to the point where it may be arbitrarily large). Similarly to ridge regression where the norm acts as regulariser, using inducing functions with large norm in SVGP models comes with a penalty which enforces more smoothness the approximate posterior and limits its expressiveness. Figure 4 shows how the norm of our ReLU inducing functions grow in the RKHS. So by making N' larger such that we approximate the ReLU better, we incur greater penalty in the ELBO for using them. This leads to unexpressive inducing variables, which can be seen by the growing predictive uncertainty. The Sofiplus, which is part of the RKHS, does not suffer from this.
Unexpressive inducing variables through spectra mismatch (Fig. 5) Any isotropic stationary kemnel whose inputs æ, E gd-1 are restricted to the unit hypersphere is a zonal kernel (i.e.. the kernel only depends on the doo-pro. This means that we are not limited to only the Arc Cosine because we can replace the shape function in Eq. (6) by any stationary kernel, and our approach would still hold. For example, we could use the Matérn-5/2 shape function with (t) = (1 + V5t + 5t2 /3) cxp(-v5t). v5t). However, in Fig. 5 we compare the fit of an SVGP model using a Matérn-5/2 kernel (left)
model using an Arc Cosine kernel (right). While both models use our Softplus inducing variables, we clearly observe that the Matérn kernel gives rise to a worse posterior model (lower ELBO and an over-estimation of the variance). In what follows we explain why this is the case.
In the bottom panel in Fig. 5 we see that for the Matérn-5/2, the Nyström approximation Off = C C fu u. is unable to explain the prior Kyf, imposed by the kernel. This leads

---

Softguus
Hatém (E.80
(EL90 -12.8))
0-:00
0--3
-2
-3
10-
Kr-Or
Ke- Ohe 0u50,r
115./1lls
0.6-
50ur
0.4
0.2
0.0
30
10 30
Figure 4: Top: Predictive variance an SVGP fit on a synthetic dataset using our ReLU (left) and Softplus (right) inducing variables for N 10, 15 and 25. Bottom: the norm of the inducing function Zm (-) as fuuccion of N.
Predictive mean and variance for an SVGP model using our Softplus inducing variables for the Matérn-5/2 (left) and Are Co- sine (right) kernel. Bottom: The two terms that constitute the predictive variance.
Figure 5: Top:
to the overestimation of the predictive variance in the top plot. The reason for this problem is the mismatch between the eigenvalues An (Eq. (7)) of the Matérn and the Fourier coefficients on (Eq. (11)) of the Softplus inducing function. As shown in Fig. C.2. the Matérn kernel has a full spectrum (i.e., An * O, Vn E IN,, whereas the coefficients for the Softplus are zero at levels n = 3, 5, .... We thus trying approximate our prior kernel, containing all levels, by a set of functions that is missing many. This problem does not occur for the combination of Softplus (or ReI.U) inducing functions and the Are Cosine kernel (right-hand side) because their spectral decomposition match. In other words, the Are Cosine kernel has zero coefficients for the same levels as our activated inducing functions.
5 Experiments
The premise of the experiments is to highlight that (i) method leads valid inducing variables, (ii) our initialisation improves DGPs terms accuracy, and (iii) we are able to improve on simple Bayesian neural networks [24, 51] in terms of calibrated uncertainty. We acknowledge that the NNs we benchmark against are the models for which we can build an equivalent DGP. While this leads a fair comparison, it excludes recent improvements such as Transformer and Batch Norm layers.
5.1 Initialisation: From Neural networks to Activated DGPs
We have shown that we can design SVGP models with basis functions that behave like neural net activations. This leads mean of the SVGP posterior which is of the same form single, fully-connected NN layer. Composing several of these SVGPs hierarchically inlo a DGP gives rise to the equivalent DNN. This equivalence has the practical advantage that we can train the means of the SVGP layers in DGP if they are NN model. We enforce the one-to-one map between the NN and DGP, by parameterising the weight matrices of the NN to have low-rank. That is, we explicitly factorise the weight matrices W, V
(-1 (see Fig. 1b) such that we can use them initialise the DGP as follows: B, C 14g V; and W; used for the directions wm, of the inducing functions 9m Once the mean of the DGP is trained,
further optimise the remaining model hyper- and variational parameters w.r.t the ELBO, which is a more principled objective [52]. For this we initialise the remaining parameters of the DGP, Σg to le - 5 as recommended by Salimbeni and Deisenroth [5]. This approach allows us to exploit the benefit of both, the efficient training of the DNN in combination with the principled uncertainty estimate of the DGP. For all SVGP models we use the Arc Cosine kerne and inducing variables obtained with the Softplus activation (Section 4.1) with = 20, for the reasons explained in Section 4.3.

---

Neural Not Fit
Activated DCP Fir
Noural Net Fit
Activated DGP Fit
X1
X1
X1
X1
1-Layer DGP
3-Layer DGP
Figure 6: We compare the fit of a single-layer and threc-layer DNN optimised using binary cross- entropy and it's equivalent DGP trained using the ELBO. The DNNs very confident, even far away from the data. We used the DNNs to initialise the DGP before optimising the ELBO, which in both situations leads to model exhibiting more calibrated uncertainly.
Illuttrative example: Banana classification Figure 6 shows the difference in predictive probabil- ity p(y x) for DNN and our activated DGP, the single-layer and three-layer case. We configure the models with a Softplus activation function and set the number of both inducing variables of the GP and hidden units of the DNN to 100. In this experiment, the first step w.r.t. the binary cross-entropy objective, upon convergence we initialise the DGP with this solution and resume training of the DGP w.r.t. the ELBO. Especially in the single-layer case, we notice how the sharp edges from the NN relaxed by the GP fit, and how the GP expresses uncertainty away from the data by letting p(y x) R 0.5. This is due to the ELBO, which balances both data fit and model complexity, and simultancously trains the uncertainty.
optimise the DNN
5.2 Regression on UCI benchmarks
We compare a series of models on a range of regression problems. The important aspect is that we keep the model configuration and training procedure fixed across all datasets. We three-layered models with 128 inducing variables (or, equivalently, hidden units). In each layer, the number of output heads is equal to the input dimensionality of the data. The Activated DGP (ADGP) and neural network approaches (NN, NN+Dropout, NN Ensembles and NN+TS) use Softplus activation functions. The Dropout bascline [24] uses rate of 0.1 during train and test. The NN baseline is a deterministic neural net that uses the training MSE the empirical variance during prediction. The NN+TS baseline uses temperature scaling (TS) on held-out validation set to compute a variance for prediction [51]. The NN Ensemble baseline uses the mean of 5 independently trained NN models.
The DGP and ADGP both use the Are Cosine kernel. The main difference is that the DGP has standard inducing points Um f(2m). whereas ADGP makes use our activated inducing variables Um = (f, Bm)... The ADGP is trained in two steps: we first train the mean of the approximate posterior w.r.t. the MSE, and then optimise all parameters w.r.t. the ELBO, explained in Section 5.1.
In Fig. we see that in general our ADGP model is more accurate than its neural network initialisation (NN) in terms of RMSE. This is a result of the second stage of training in which we use the ELBO rather than the MSE, which is especially beneficial to prevent overfitting the smaller datasets. When it comes to NLPD, ADGP shows improvements over its NN initialisation for datasets out of 7 and consistently outperforms classic DGPs.
5.3 Large scale image classification
In this experiment we measure the performance of our models under dataset shifts [53]. For MNIST and Fashion-MNIST the out-of-distribution (OOD) test sets consist of rotated digits from ('.i.. the original test set) to 180% For CIFAR-10 we apply four different types of corruption to the test images with increasing intensity levels from 0 to 5. For MNIST and FASHION-MNIST the models consist of two convolutional and max- max-pooling layers, followed by two dense layers with 128 units and output heads. The dense layers are either fully-connected neural network layers using a Softplus activation function (NN, NN+Dropout NN+TS), or our Activated GP layers using the Are Cosine

---

Yacht (308, 6)
Energy (768, 8)
Concrete (1030,
WineRed (1599, 11)
WineWhite (4898, 11)
Kin8mn (8192, 8)
Power (9568, 4)
rank
NLPD RMSF
0.55
0.85
0.40
M
0.50
0.50
0.25
0.78
0.45
0.75
0.75
0.30
0.25
2.00
0.30
1.00
1.30
0.60
1.00
0.20
0.80
1.30
0.40
0.10
0.00
0.10
0.60
1.20
1.20
0.20
0.05
ADGP (ours)
NN+Dropout
NN+TS
DGP
NN Ensemble
Figure 7: UCL. Root Mean Squared Error (RMSE) and Negative Log Predictive Density (NLPD) with 25 % and 75% quantile error bars based on 5 splits. Dataset size and dimension given in parentheses.
MNIST
FASHION-MNIS MNNIST
CIFAR-10
-
ADCP (ours)
NN-Dropout
0.50
NN-TS
0.5
-5
-10
Rotation (degrees)
Retation Idegrees)
Cornuption Bleee)
Figure 8: Results on the rotated MNIST, FASHION-MNIST and corrupted CIFAR-10, showing the mean and std. dev. of the accuracy (top), and test ligeeihkkl (TLL) (bottom).
kernel and Softplus inducing variables (ADGP). For the CIFAR-10 models, we use the residual convolutional layers from ResNet [54] to extract useful features before passing them to our dense GP or NN layers. Details of the model architectures are given in Appendix E. As previously, the ADGP model initialised to the solution of the NN model, and training is then continued using the ELBO. In Fig. 8 we observe that the models perform very similar in terms of prediction accuracy, but that ADGP better account uncertainty as evidenced by the Test Log Likelihood metric.
6 Conclusion
In this work, we establish connection between fully-connected neural networks and the posterior deep sparse Gaussian processes. We use specific flavour of interdomain inducing variables based the RKHS inner product obtain basis functions for the SVGP that match activation functions from the neural network literature. By composing such SVGPs together, we obtain a DGP for which forward pass through the mean of each layer equivalent address important mathematical subtleties ensure the validity of the approach and to gain insights on how to choose the actívation function. As demonstrated in the experiments, being able to interpret the mathematical expression either deep neural network as the mean of deep Gaussian process can benefit both approaches. On the one hand, allows us to improve the prediction accuracy and the uncertainty representation of the neural network by regularising it with the ELBOO obtained from GP prior. On the other hand, it enables by initialising them with a pre-trained neural network. We believe that by providing an equivalence between the models. not in the infinite limit but in the operational regime, opens many future research directions allowing beneficial knowledge transfer between the two domains.
forward pass in DNN. We also
to better optimise deep Gaussian process model
10

---

References
[1] Ian Goodfellow, Yoshua Bengio, and Aaron Courville, Deep Learning. The MIT Press, 2016.
[2] Carl E. Rasmussen and Christopher K. I. Williams, Gaussian Processes for Machine Learning. MIT Press, 2006.
[3] Andreas Damianou and Neil D. Lawrence. "Deep Gaussian Processes". In: Proceedings of the 16th International Conference on Artificial Intelligence and Statistics (AISTATS), 2013.
[4] Thang Bui, Daniel Hermandez-Lobate Jose Hernandez- Turner. "Deep Gaussian Processes for Regression using Approximate Expectation Propaga- tion". In: Proceedings of The 33rd International Conference on Machine Learning (ICML). 2016.
Yingzhen Li, and Richard E.
[5] Hugh Salimbeni and Mare P. Deisenroth. "Doubly Stochastic Variational Inference for Deep Gaussian Processes". In: Advances in Neural Information Processing Systems 30 (NIPS). 2017.
[6] James Hensman and Neil D. Lawrence, "Nested Variational Compression in Deep Gaussian Processes". In: arXiv preprint arXiv:1412.1370 (2014).
[7] David J. C. MacKay. "A Practical Bayesian Framework for Backpropagation Network". In:
Neural Computation (1992).
[8] Radford M. Neal. Bayesian Learning for Neural Networks. Springer, 1995.
[9] Christopher KI Williams and Carl Edward Rasmussen. "Gaussian processes for regression". In: (1996).
[10]
Carl Edward Rasmussen. "Evaluation of Gaussian processes and other methods for non- non-linear regression". PhD thesis. University of Toronto Toronto, Canada, 1997.
[11] Christopher I. Williams. "Computing with Infinite Networks". In: Advances in Neural Information Processing Systems 11 (NIPS 1997) (1998).
[12] David J. C. MacKay. "Introduction to Gaussian Processes". In: (1998).
[13] Youngmin Cho and Lawrence K. Saul. "Kernel Methods for Deep Learning... In: Advances in Neural Information Processing Systems 22 (NIPS). 2009.
[14] Lassi Meronen, Christabella Irwanto, and Amo Solin. "Stationary Activations for Uncertainty Calibration Deep Learning". In: Advances in Neural Information Processing Systems 33
(NeurIPS), 2020.
[15] Alexander G. de G. Matthews, Mark Rowland, Jiri Hron, Richard E. Turner, and Zoubin Ghahramani. "Gaussian process behaviour wide deep neural networks". In: Proceedings of the 6th International Conference on Learning Representations (ICLR) (2018).
[16] Jachoon Lee, Yasaman Bahri, Roman Novak, Samuel S. Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. "Deep neural networks as Gaussian processes". In: 6th International Conference Learning Representation (ICLR). 2018.
[17] Greg Yang. "Wide feedforward or recurrent neural networks of any architecture are Gaussian processes". In: Advances in Neural Information Processing Systems 32 (NeurIPS). 2019.
[18] Adrià Garriga-Alonso, Carl E. Rasmussen, and Laurence Aitchison. "Deep Convolutional Networks as shallow Gaussian Processes". In: Proceedings of the 7th International Conference on Learning Representations (ICLR) (2018).
[19] Roman Novak, Lechao Xiao, Jachoon Lee, Yasaman Bahri, Greg Yang, Jiri Hron, Daniel A. Abolafia, Jeffrey Pennington, and Jascha Sohl-Dickstein. "Bayesian deep convolutional networks with many channels are Gaussian processes". In: Proceedings of the 7th International Conference Learning Representations (ICLR) (2019).
[20] James Mercer. "Functions positive and negative type, and their connection the theory of integral equations". In: Philosophical transactions of the royal society of London. Series A, containing papers of a mathematical or physical character (1909).
[21] Carl E. Rasmussen and Christopher K. Williams. In: Gaussian Processes for Machine Learning. 2006. Chap. Regression ($2.1 1 & $2.2).
[22] Durk Kingma, Tim Salimans, and Max Welling. "Variational Dropout and the Local Repa- rameterization Trick". In: Advances in Neural Information Processing Systems 28 (NIPS). 2015.

---

[23] Charles Blundell, Julien Corneiiee, Koray Kavukcuoglu, and Daan Wierstra. "Weight Uncer- tainty in Neural Network". In: Proceedings of The 32nd International Conference on Machine Learning (ICML). 2015.
[24] Yarin Gal and Zoubin Ghahramani. "Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning Zoubin Ghahramani". In: Proceedings of The 33rd International Conference on Machine Learning (ICML). 2016.
[25] Christos Louizos and Max Welling. "Structured and efficient variational deep learning with matrix gaussian posteriors", In: Proceedings The 33rd International Conference on Machine Learning. 2016.
[26] David J. C. MacKay. "Bayesian Model Comparison and Backprop Nets". In: Advances in Neural Information Processing Systems (NIPS 1991). 1992.
[27] David J. C. MacKay. Information Theory, Inference and Learning Algorithms. Cambridge University Press, 2003.
[28]
Scbastian W. Ober and Laurence Aitchison. Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processes. 2020.
[29] Alexander Immer, Matthias Bauer, Vincent Fortuin, Gunnar Rätsch, and Mohammad Emtiyaz Khan. Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning. 2021.
[30] Andreas Damianou. "Deep Gaussian Processes and Variational Propagation of Uncertainty".
PhD thesis. University of Sheffield, 2015.
[31] Vincent Dutordoir, Mark van der Wilk, Artem Artemev, and James Hensman. "Bayesian Image Classification with Deep Convolutional Gaussian Processes". In: Proceedings of the
23th International Conference on Artificial Intelligence and Statistics (AISTATS). 2020.
[32] Shengyang Sun, Jiaxin Shi, and Roger B. Grosse. "Neural Networks as Inter-Domain Inducing Points". In: 3rd Symposium Advances in Approximate Bayesian Inference. 2021.
[33] David Duvenaud, James Lloyd, Roger Grosse, Joshua Tenenbaum, and Ghahramani Zoubin, "Structure discovery in nonparametric regression through compositional kernel scarch". In: International Conference Machine Learning. PMLR. 2013.
[34] Andrew Wilson and Ryan Adams. "Gaussian process kernels for pattern discovery and extrap- olation". In: International conference on machine learning. PMLR. 2013.
[35] David Ginsbourger, Nicolas Durrande, and Olivier Roustant. "Kernels and designs for mod- elling invariant functions: from group invariance to additivity". In: mODa 10-Advances in Model-Oriented Design and Analysis. 2013.
[36] Mark van der Wilk, Matthias Bauer, S. T. John, and James Hensman. "Learning Invariances using the Marginal Likelihood". In: Advances in Neural Information Processing Systems 31 (NeurIPS). 2018.
[37] Michalis K. Titsias. "Variational Learning of Inducing Variables in Sparse Gaussian Processes". In: Proceedings of the 12th International Conference on Artificial Intelligence and Statistics (AISTATS). 2009.
[38] James Hensman, Nicolo Fusi, and Neil D. Lawrence. "Gaussian Processes for Big Data". In: Proceedings of the 29th Conference on Unceriainty in Artificial Intelligence (UAI). 2013.
[39] Alexander G. de G. Matthews, James Hensman, Turner F. Richard, and Zoubin Ghahramani. "On Sparse Variational Methods and the Kullback-Leille Divergence between Stochastic Processes". In: Proceedings of the 19th International Conference on Artificial Intelligence and Statisties (AISTATS). 2016.
[40] Misha Denil, Babak Shakibi, Laurent Dinh, Marc textuuutixtt Aurelio Ranzato, and Nando de Freitas. "Predicting Parameters in Deep Learning.. In: Advances in Neural Information Processing Systems 26 (NIPS). 2013.
[41] Miguel Låzaro-Gredill and Aníbal Figueiras- Figueiras-Vidal. "Inter-domain Gaussian Processes for Sparse Inference using Inducing Features". In: Advances in Neural Information Processing Systems 22 (NIPS). 2009.
[42] Mark van der Wilk, Vincent Dutordoir, S. T. John, Artem Artemev, Vincent Adam, and James Hensman. "A Framework for Interdomain and Multioutput Gaussian Processes". In: arxiv preprint ar arXiv:2003.01115 (2020).
12

---

[43] Felix Leibfried, Vincent Dutordoir, S. T. John, and Nicolas Durrande. "A Tutorial on Sparse Gaussian Processes and Variational Inference", In: arXiv preprint arXiv:2012. 1.3962 (2020).
[44] James Hensman, Nicolas Durrande, and Amo Solin. "Variational Fourier Features for Gaussian Processes". In: Journal of Machine Learning Research (2018).
[45] David R Burt, Carl Edward Rasmussen, and Mark van der Wilk. "Variational orthogonal features". In: arXiv preprint arXiv:2006. 13170 (2020).
[46] Vincent Dutordoir, Nicolas Durrande, and James Hensman. "Sparse Gaussian Processes with Spherical Harmonic Features". In: Proceedings of the 37th International Conference on Machine Learning (ICML.. 2020.
2020.
[47] Alberto Bietti and Francis Bach. "Deep Equals Shallow for ReLU Networks in Kernel Regimes".. In: arXiv preprint arXiv:2009. 14397 (2020).
[48]
Holger Wendland. Scattered Data Approximation. Cambridge University Press, 2005. Motonobu Kanagawa, Philipp Hennig, Dino Sejdinovic, and Bharath K Sriperumbudur: "Gaus- sian Processes and Kemnel Methods: A Review on Connections and Equivalences". arXiv preprint arXiv: 02,58882 (2018).
[49]
[50] Francis Bach. "Breaking the Curse of Dimensionality with Convex Neural Networks". In: Journal of Machine Learning Research (2017).
[51] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. "On calibration of modern neural networks". In: Proceedings of the 34th International Conference on Machine Learning (ICML). 2017.
[52] Edwin Fong and Chris Holmes. "On the marginal likelihood and cross-validation".. In: arXiv preprint arXiv: 190.087755 (2019).
[53] Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snock. "Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift". In: Advances in Neural Information Processing Systems 32 (NeurIPS). 2019.
[54] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. "Deep residual learning for image recognition". Proceedings of the IEFE conference on computer vision and pattern recognition. 2016.
[55] Feng Dai and Yuan Xu. Approximation Theory and Harmonic Analysis on Spheres and Balls. Springer, 2013.
[56] Costas Efthimiou and Christopher Frye. Spherical Harmonics in Dimensions. World Scien- tific Publishing, 2014.
[57] Alexander G. de G. Matthews, Mark van der Wilk, Tom Nickson, Keisuke. Fujii, Alexis Boukouvalas, Pablo León-Vill Villagrá, Zoubin Ghahramani, and James Hensman. "GPΩlo: A Gaussian process library using TensorFlow". In: Journal of Machine Learning Research (2017).
[58] Vincent Dutordoir, Hugh Salimbeni, Eric Hambro, John McLeod, Felix Leibfried, Artem Artemev, Mark van der Wilk, James Hensman, Mare Deisenroth, and ST John. "GPflux: A Library for Deep Gaussian Processes". In: arXiv preprint arXiv:2003.01 20t::000300011155 (2021). [59] Tilmann Gneiting and Adrian F. Raftery. "Strictly Proper Scoring Rules, Prediction, and Estimation". . In: Journal of the American Statistical Association (2007).
[60] Durk Kingma and Jimmy Ba. "Adam: A Method for Stochastic Optimization". In: Proceedings the 2nd International Conference on Learning Representations (ICLR). 2014.
13


Reviews:
[{"review_id": "hF-A8MpDcrW", "reviewer": "Reviewer_AwCU", "summary": "This paper explores the connection between neural networks and Gaussian processes (GP). Specifically, this paper interprets the neural network activations as the inter-domain inducing variables for a GP. Specifically, this paper adopts the proposed approach in deep gaussian processes, proposing initialize a DGP by a pre-trained neural network based on the connection. They investigated the effectiveness of the approach by multiple experiments.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Method\n### The Problem and the Approach\nNeural networks are state-of-art models in deep learning, while GPs are standard in Bayesian learning. Despite some connections are drawn for infinite-width neural networks and GPs [1, 2], their developments are in general disjoint. How to combine the merits in both models, which is tackled in this paper, is an important problem. Specifically, this paper proposes the Activated Inducing Variables, which interprets the neural network activations as the inter-domain inducing variables for a GP. In consequence, a one-hidden-layer neural network is equivalent to the predictive mean of a respected variational Gaussian process.\n\n### Comparisons to [3]\nThe activated inducing variables in this paper is similar to the approach in [3], which limits the novelty of the proposed approach. However, the Deep Gaussian Processes and the ArcCosine kernels are not investigated in [3]. Therefore, it is helpful to present the differences and contributions clearer.\n\n### Deep Gaussian Processes\nThis paper explores the connections between deep neural networks and deep gaussian processes in the experiments. This connection is further validated by the regression and classification experiments. Therefore, it might be helpful to discuss the connection more clearly. \n\nSpecifically, the background Section 3.2 shows the equivalence between propagating the predictive means in a DGP and a DNN with low-rank matrices $\\mathbf{W} \\mathbf{V}$. The paper proposes to initialize a DGP with a pre-trained DNN.\n1) How is the rank chosen ? \n2) How to decompose a single weight matrix into $\\mathbf{W} \\mathbf{V}$ to match the DGP ? \nGiven that there are infinite options for approaching the low-rank decomposition, are there justifications on how to choose it ? \n\nIn general, I think the connections between deep NNs and deep GPs deserve more discussions in the main text.\n\n### Benefit against Standard Inducing Points\nADGP adopts the \"activation inducing variables\" for posterior inference in DGPs. However, it is not clear why the \"activation inducing variables\" could provide a better variational approximation than standard inducing points. In fact, the standard inducing points can be understood as the \"activation inducing points\" as well by letting the activation to the kernel function: \n$$\n\\sigma(w^T x) := |w| |x| \\hat{k}(t) = k(w, t) \n$$\nwhere $t$ is the inner product between normalized $w$ and $t$, $\\hat{k}$ is the function applied to $t$ in Eq(6).\nMoreover, using the \"standard\" \"activation inducing points\" also prevents the spectral mismatch mentioned in Sec 4.3. Therefore, it is beneficial to discuss the potential differences between them.  \n\nOn the other hand, it might be the pretraining procedure that leads to the empirical improvements. Could the authors conduct a similar pretraining phase for the standard inducing points ?\n\n## Clarity\nThis paper is well-written. The method is clearly presented.\n\n## Experiments\nThis paper demonstrates promising experimental results, where the proposed ADGP outperforms both standard DGPs and NNs. \n\n1) Although a vanilla NN usually leads to poor uncertainty estimates, the simple temperature scaling (TS) using the validation set usually improves the performance by a large margin [4]. In Figure 7, could the authors also compare with the NN+TS in terms of the test log likelihoods ? \n2) Similar to the temperature scaling in classification, the single observation variance can be treated as the hyperparameter and tuned using the validation set. Could a similar comparison be conducted in the regression experiment as well ? \n\n\n## Other Comments\n1) Under the proposed formulation, each NN layer is the predictive mean of one GP layer. However, the whole DNN is not necessarily the predictive mean of the whole DGP. Therefore, it is helpful to make the claim clearer. For example, in Line 32-33.  \n2) Line 158: why the Softplus function is not $ \\log (1 + \\exp(t))$ ? \n3) Are the arguments in Line194-198 supported by a theorem or existing literatures? \n4) Line 222: I think it should be \"any isotropic stationary kernels\".\n5) Line 259: do the \"single-layer\" and \"three-layer\" mean \"hidden layer\" or \"any layer\" ? \n\n### References\n[1] Radford M. Neal. Bayesian Learning for Neural Networks. Springer, 1995.\n\n[2] Jacot, Arthur, Franck Gabriel, and Clément Hongler. \"Neural Tangent Kernel: Convergence and Generalization in Neural Networks.\"\n\n[3] Shengyang Sun, Jiaxin Shi, and Roger B. Grosse. “Neural Networks as Inter-Domain Inducing Points”. In: 3rd Symposium on Advances in Approximate Bayesian Inference. 2021.\n\n[4] Guo, Chuan, et al. \"On calibration of modern neural networks.\" International Conference on Machine Learning. PMLR, 2017.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "To further improve the paper, I think the following are potential directions,\n1) discuss the differences with [3]\n2) present the connections between DNNs and DGPs clearer, both methodologically and experimentally.\n3) discuss the comparisons with standard inducing variables\n4) demonstrate the advantage of the proposed approach against NNs more clearly by adding the NN+TS baseline\n\nThis paper does not lead to negative societal impacts.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "Z5FCu2iwb6b", "reviewer": "Reviewer_3kns", "summary": "This paper proposes to establish an equivalence between the forward passes of neural networks and  deep Gaussian process (DGP). Authors further propose models that can either be seen as neural networks with improved uncertainty or deep Gaussian processes with increased prediction accuracy. This is achieved by developing a DGP that has an identical forward pass to a neural net. The training of DGPs can be improved by taking practices from NNs, e.g., a DGP can be initialised in the same way as neural  net without the specialized initialization in DGP literature. Experiments are performed on UCI benchmarks, MNIST, Fashion-MNIST and CIFAR-10. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes to establish an equivalence between the forward passes of neural networks and  deep Gaussian process (DGP). Authors further propose models that can either be seen as neural networks with improved uncertainty or deep Gaussian processes with increased prediction accuracy. This is achieved by developing a DGP that has an identical forward pass to a neural net. The training of DGPs can be improved by taking practices from NNs, e.g., a DGP can be initialised in the same way as neural  net without the specialized initialization in DGP literature. Experiments are performed on UCI benchmarks, MNIST, Fashion-MNIST and CIFAR-10. \n\nOverall, this is an interesting paper that further connects the best practices of Deep GP and neural net literature. I especially like the practical models that authors propose to bring about better training  of deep GP and improve the  accuracy. These are important and interesting  contributions to the machine learning community. Below are some questions and suggestions.\n\n1. Though the code is provided in GPFlow, given the huge popularity of pytorch these days, authors should also release a Pyro based code to maximize the impact.\n\n2. In Figure 6, the proposed model performs worse than DGP on a couple datasets. Is there a general reason? Can you further improve the model to outperform DGP?\n\n3. How correlated are the predictions of the proposed model, DGP, and NN on the tested datasets? \n\n4. Authors should discuss how the perspective from DGP and this paper differs from neural tangent kernel (NTK). I understand that NTK, DGP are different in terms of how they compare to NN. But how does this paper make a difference in this context?\n\n5. Good to summarize more high level intuition why we can improve training without initialization, because  the mean initialization can simply be viewed as skip connections. How does your model connect to and compare to such skip connections?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "None", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "WQaZT3HfXI", "reviewer": "Reviewer_oinS", "summary": "This work extends the approach of Sun et al. to more than two layers of neural networks by connecting weights to sparse approximations of deep Gaussian processes. \n", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths & Weaknesses:\n\nIn general this is an interesting paper that brings the idea that connects NN and inducing points further. These contributions include\n* extending the approach of Sun et al. to more than two layers of neural networks by considering deep GPs.\n* proposing to use the arccosine kernel and softplus activation function.\n* careful analysis of the interaction between kernels and activation functions, which provides insights on the importance of matching their spectral properties.\n* using the connection to provide pre-training of deep GPs (via fitting a neural network and use it to initialize the mean function), and post-training of NNs (via the ELBO of deep GPs) to improve uncertainty estimation.\n\nHowever, there are still plenty of details missing to fully understand the model used: \n\nAlthough the central idea of inter-domain inducing points is well-explained in section 4, this mostly has appeared in Sun et al. and seems not the main contribution of the paper. Surprisingly, the main contribution (a deep extension) is not discussed at all in the main text. it is unclear how the GP layers are stacked to recover the deep NN, e.g., what are the correspondence between $W_1, B_1, W_2, B_2$ and $W_1, V_1, W_2, V_2$, how the deep GP variational distribution is initialized from the deep NN.\n\nThe latter seems mysterious to me since the forward pass of deep GP is not equal to propagating through the mean functions of each layer (note that there are nested expectations on the left side of eq. (4)). I wonder how the authors manage to \"develop a DGP that has an identical forward pass to a typical ReLU NN\" since identical copy layer-by-layer is not going to make it.\n \nAnother thing here is that I don't think the claim with ReLU is true, the final model is only tested on softplus and there are problems of ReLU discussed in section 4.3)\n\nThese details are important since these are the major differences from the approach of Sun et al., where only neural networks with a single hidden layer are considered. \n\nRelated work:\n\nI'm aware of the work of Sun et al. so I can quickly get a understanding of the improvements made in the paper. However, for others who aren't I feel that this relationship is unclear and not even discussed. The general framework of section 4 has large overlap with their approach but none of these is mentioned. I'd suggest to properly cite the ideas developed in previous work and at the same time point out what new contributions are made in this work. The current form reads like the authors invented activated inducing points, while the (14) is the clearly the main idea of the previous work.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper shows an equivalence between forward passes of a neural network and deep Gaussian processes.  It received a positive review, and low-borderline reviews that were appreciative of the method but concerned about novelty over a workshop paper appearing this year.
This paper was discussed extensively with the AC, Senior AC, and an external expert who provided an additional opinion on the paper.  Quoting from the communication with the external expert:

I would accept this paper. The points the authors make at the very top
are significant by themselves:

- addressing the issue of variance starvation
- the model is proper and not degenerate
- training of the model is possible

The AC and Senior AC are in agreement that there is significant additional novelty and development over Sun et al. [45] to warrant publication of this paper.  The authors should take into account the detailed comments in the reviews pointing out the relationship between the works, and to appropriately include the relationship in the camera ready version.  The paper provides a valuable contribution to unifying DGPs and DNNs, and shows significant novel theoretical and practical results.
Decision: Accept (Poster)