Deep Neural Networks as Point Estimates for Deep Gaussian Processes

1 Introduction . Neural networks ( NNs ) [ 1 ] and Gaussian processes ( GPs ) [ 2 ] are well-established frameworks for solving regression or classification problems , with complementary strengths and weaknesses . NNs work well when given very large datasets , and are computationally scalable enough to handle them . GPs , on the other hand , are challenging to scale to large datasets , but provide robust solutions with uncertainty estimates in low-data regimes where NNs struggle . Ideally , we would have a single model that provides the best of both approaches : the ability to handle low and high dimensional inputs , and to make robust uncertainty-aware predictions from the small to big data regimes . Damianou and Lawrence [ 3 ] introduced the Deep Gaussian process ( DGP ) as a promising attempt to obtain such a model . DGPs replicate the structure of deep NNs by stacking multiple GPs as layers , with the goal of gaining the benefits of depth while retaining high quality uncertainty . Delivering this potential in practice requires an efficient and accurate approximate Bayesian training procedure , which is highly challenging to develop . Significant progress has been made in recent years , which has led to methods outperform both GPs and NNs in various medium-dimensional tasks [ 4 , 5 ] . In addition , some methods [ 5 , 6 ] train DGPs in ways that closely resemble backpropagation in NNs , which has also greatly improved efficiency compared to early methods [ 3 ] . However , despite recent progress , DGPs are still cumbersome to train compared to NNs . The similarity between the training procedures sheds light on a possible reason : current DGP models are forced to choose activation functions that are known to behave poorly in NNs ( e.g. , radial basis functions ) . In this work we aim to further unify DGP and NN training , so practices that are known to work for NNs can be applied in DGPs . We do this by developing a DGP for which propagating through ⇤Work done while at Secondmind . Correspondence to vd309 @ cam.ac.uk . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . the mean of each layer is identical to the forward pass of a typical NN . This link provides practical advantages for both models . The training of DGPs can be improved by taking best practices from NNs , to the point where a DGP can even be initialised from a NN trained in its usual way . Conversely , NN solutions can be endowed with better uncertainty estimates by continued training with the DGP training objective . 2 Related Work . Many different relationships between GPs and NNs have been established over the years . These relationships mainly arise from Bayesian approaches to neural networks . Finding the posterior distribution on neural network weights is challenging , as closed-form expressions do not exist . As a consequence , developing accurate approximations has been a key goal since the earliest works on Bayesian NNs [ 7 ] . While investigating single-layer NN posteriors , Neal [ 8 ] noticed that randomly initialised NNs converged to a Gaussian process ( GP ) in the limit of infinite width . Like NNs , GPs represent functions , although they do not do so through weights . Instead , GPs specify function values at observed points , and a kernel which describes how function values at different locations influence each other . This relationship was of significant practical interest , as the mathematical properties of GPs could ( 1 ) represent highly flexible NNs with infinite weights , and ( 2 ) perform Bayesian inference without approximations . This combination was highly successful for providing accurate predictions with reliable uncertainty estimates [ 9 , 10 ] . To obtain models with various properties , GP analogues were found for infinitely-wide networks with various activation functions [ 11 , 12 ] including the ReLU [ 13 ] . More recently , Meronen et al . [ 14 ] investigated the relationship in the opposite direction , by deriving activation functions such that the infinite width limit converges to a given GP prior . Since the growth of modern deep learning , relationships have also been established between infinitely wide deep networks and GPs [ 15 , 16 , 17 ] . Given these relationships , one may wonder whether GPs can supersede NNs , particularly given the convenience of Bayesian inference in them . Empirically though , finite NNs outperform their GP analogues [ 16 , 18 , 19 ] on high-dimensional tasks such as images . MacKay [ 12 ] explained this by noting that NNs lose their ability to learn features in the infinite limit , since every GP can be represented as a single-layer NN , with fixed features [ 20 , 21 ] . This observation justifies the effort of performing approximate Bayesian inference in finite deep networks , so both feature learning and uncertainty can be obtained . Renewed effort in Bayesian training procedures has focussed on computationally scalable techniques that take advantage of modern large datasets , and have provided usable uncertainty estimates [ e.g. , 22 , 23 , 24 , 25 ] . However , questions remain about the accuracy of these approximations . For accurate Bayesian inference , marginal likelihoods are expected to be usable for hyperparameter selection [ 26 , 27 ] . For most current approximations , there is either no positive or explicitly negative [ 23 ] evidence for this holding , although recent approaches do seem to provide usable estimates [ 28 , 29 ] . DGPs [ 3 ] provide an alternative approach to deep NNs , which use GP layers instead of weight-based layers , in order to take advantage of the improved uncertainty estimates afforded by having an infinite number of weights . The DGP representation is particularly promising because both early and recent work [ 3 , 30 , 31 ] shows that marginal likelihood estimates are usable for hyperparmeter selection , which indicates accurate inference . However , currently scalability and optimisation issues hinder widespread use . In this work , we provide a connection between the operational regimes of DGPs and NN , deriving an equivalence between the forward pass of NNs and propagating the means through the layers of a DGP . We share this goal with Sun et al . [ 32 ] , who recently proposed the use of neural network inducing variables for GPs based on the decomposition of zonal kernels in spherical harmonics . However , compared to Sun et al . [ 32 ] , our method does not suffer from variance over-estimation , which fundamentally limits the quality of the posterior . Our approach also leads to proper GP models , as opposed to adding a variance term to NNs using a Nystöm approximation , which allows the correct extension to DGPs and the optimisation of hyperparameters . These improvements are made possible by the theoretical analysis of the spectral densities of the kernel and inducing variable ( Section 4.3 ) . 3 Background . In this section , we review Deep Gaussian processes ( DGPs ) , with a particular focus on the structure of the approximate posterior and its connection to deep NNs . We then discuss how inducing variables control the DGP activation function , which is a key ingredient for our method . 3.1 Deep Gaussian Processes and Sparse Variational Approximate Inference . GPs are defined as an infinite collection of random variables { f ( x ) } x2Rd such that for all n 2 N , the distribution of any finite subset { f ( x1 ) , . . . , f ( xn ) } is multivariate normal . The GP is fully specified by its mean function µ ( x ) = E [ f ( x ) ] and its kernel k ( x , x0 ) = Cov ( f ( x ) , f ( x0 ) ) , shorthanded as f ( · ) ⇠ GP ( µ ( · ) , k ( · , · ) ) . GPs are often used as priors over functions in supervised learning , as for Gaussian likelihoods the posterior f ( · ) | D given a dataset D = { xi 2 Rd , yi 2 R } Ni=1 is tractable . Surprisingly diverse properties of the GP prior can be specified by the kernel [ 2 , 33 , 34 , 35 , 36 ] , although many problems require the flexible feature learning that deep NNs provide . DGPs [ 3 ] propose to solve this in a fully Bayesian way by composing several layers of simple GP priors , F ( · ) = fL . . . f2 f1 , where f ` ( · ) ⇠ GP 0 , k ` ( · , · ) . ( 1 ) Inference in DGPs is challenging as the composition is no longer a GP , leading to an intractable posterior F ( · ) | D. We follow the variational approach by Salimbeni and Deisenroth [ 5 ] due to its similarity to backpropagation in NNs . They use an approximate posterior consisting of an independent sparse GP for each layer . Each sparse GP is constructed by conditioning the prior on M inducing variables [ 37 , 38 , 39 ] , commonly function values u ` = { um ` = f ` ( wm ` ) } Mm=1 , and then specifying their marginal distribution as q ( u ` ) = N ( µ ` , ⌃ ` ) . This results in the approximate posterior process : q ( f ` ( · ) ) = GP ⇣ B > ` cu ` ( · ) ; k ` ( · , ·0 ) + c > u ` ( · ) C 1 u ` u ` ( ⌃ ` Cu ` u ` ) C 1 u ` u ` cu ` ( · 0 ) ⌘ , ( 2 ) where B ` = C 1u ` u ` µ ` 2 R M⇥d ` , d ` the dimensionality of the GP ’ s output , cu ` ( · ) = Cov ( f ` ( · ) , u ` ) and Cu ` u ` = Cov ( u ` , u ` ) . It is worth noting that , we use the symbol ‘ C ’ , rather than the more commonly used ‘ K ’ , to denote the fact that these matrices contain covariances which are not necessarily simple kernel evaluations . The variational parameters µ ` 2 RM⇥d ` and ⌃ ` 2 Rd ` ⇥M⇥M are selected by reducing the Kullback-Leibler ( KL ) divergence from the variational distribution to the true posterior . This is equivalent to maximising the Evidence Lower BOund ( ELBO ) . Taking the evaluations of the GP as h ` = f ` ( h ` 1 ) , the ELBO becomes [ 5 ] : ELBO = XN i=1 Eq ( hi , L ) [ log p ( yi |hi , L ) ] XL ` =1 KL [ q ( u ` ) k p ( u ` ) ] log p ( y ) . ( 3 ) 3.2 Connection between Deep Gaussian processes and Deep Neural Networks . Hensman and Lawrence [ 6 ] observed that the composite function of propagating an input through each layer ’ s variational mean ( Eq . ( 2 ) ) of a DGP equals : Eq [ fL ( · ) ] . . . Eq [ f1 ( · ) ] = B > LcuL ( · ) . . . B > 1 cu1 ( · ) , ( 4 ) which resembles the forward pass through fully-connected NN layers with non-linearity ( · ) : V > L ( WL · ) . . . V > 2 ( W2 · ) V > 1 ( W1 · ) ( 5 ) with W ` and V ` the pre-activation and output weights , respectively . Both models are visualised in Fig . 1 . Indeed , if we can formulate an approximation that makes the covariance cu ` ( · ) the same as a typical neural net activation function ( W ` · ) and set B ` equal to V ` , we obtain a formal mathematical equivalence between the forward pass of a DNN and propagating the mean of each layer through a DGP . This is one of the main contributions of this work . The remaining difference between the two models are then the so-called “ bottleneck ” layers in the DGP : h1 and h2 in Fig . 1a . This is a consequence of the DGP explicitly representing the output at each layer . However , while a NN does not explicitly represent the outputs , low-rank structure in the matrices W2V > 1 and W3V > 2 is typically found after training [ 40 ] , which strengthens the connection between both models .

Summary: This paper proposes to establish an equivalence between the forward passes of neural networks and  deep Gaussian process (DGP). Authors further propose models that can either be seen as neural networks with improved uncertainty or deep Gaussian processes with increased prediction accuracy. This is achieved by developing a DGP that has an identical forward pass to a neural net. The training of DGPs can be improved by taking practices from NNs, e.g., a DGP can be initialised in the same way as neural  net without the specialized initialization in DGP literature. Experiments are performed on UCI benchmarks, MNIST, Fashion-MNIST and CIFAR-10.