Deep Neural Networks as Point Estimates for Deep Gaussian Processes

1 Introduction . Neural networks ( NNs ) [ 1 ] and Gaussian processes ( GPs ) [ 2 ] are well-established frameworks for solving regression or classification problems , with complementary strengths and weaknesses . NNs work well when given very large datasets , and are computationally scalable enough to handle them . GPs , on the other hand , are challenging to scale to large datasets , but provide robust solutions with uncertainty estimates in low-data regimes where NNs struggle . Ideally , we would have a single model that provides the best of both approaches : the ability to handle low and high dimensional inputs , and to make robust uncertainty-aware predictions from the small to big data regimes . Damianou and Lawrence [ 3 ] introduced the Deep Gaussian process ( DGP ) as a promising attempt to obtain such a model . DGPs replicate the structure of deep NNs by stacking multiple GPs as layers , with the goal of gaining the benefits of depth while retaining high quality uncertainty . Delivering this potential in practice requires an efficient and accurate approximate Bayesian training procedure , which is highly challenging to develop . Significant progress has been made in recent years , which has led to methods outperform both GPs and NNs in various medium-dimensional tasks [ 4 , 5 ] . In addition , some methods [ 5 , 6 ] train DGPs in ways that closely resemble backpropagation in NNs , which has also greatly improved efficiency compared to early methods [ 3 ] . However , despite recent progress , DGPs are still cumbersome to train compared to NNs . The similarity between the training procedures sheds light on a possible reason : current DGP models are forced to choose activation functions that are known to behave poorly in NNs ( e.g. , radial basis functions ) . In this work we aim to further unify DGP and NN training , so practices that are known to work for NNs can be applied in DGPs . We do this by developing a DGP for which propagating through ⇤Work done while at Secondmind . Correspondence to vd309 @ cam.ac.uk . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . the mean of each layer is identical to the forward pass of a typical NN . This link provides practical advantages for both models . The training of DGPs can be improved by taking best practices from NNs , to the point where a DGP can even be initialised from a NN trained in its usual way . Conversely , NN solutions can be endowed with better uncertainty estimates by continued training with the DGP training objective . 2 Related Work . Many different relationships between GPs and NNs have been established over the years . These relationships mainly arise from Bayesian approaches to neural networks . Finding the posterior distribution on neural network weights is challenging , as closed-form expressions do not exist . As a consequence , developing accurate approximations has been a key goal since the earliest works on Bayesian NNs [ 7 ] . While investigating single-layer NN posteriors , Neal [ 8 ] noticed that randomly initialised NNs converged to a Gaussian process ( GP ) in the limit of infinite width . Like NNs , GPs represent functions , although they do not do so through weights . Instead , GPs specify function values at observed points , and a kernel which describes how function values at different locations influence each other . This relationship was of significant practical interest , as the mathematical properties of GPs could ( 1 ) represent highly flexible NNs with infinite weights , and ( 2 ) perform Bayesian inference without approximations . This combination was highly successful for providing accurate predictions with reliable uncertainty estimates [ 9 , 10 ] . To obtain models with various properties , GP analogues were found for infinitely-wide networks with various activation functions [ 11 , 12 ] including the ReLU [ 13 ] . More recently , Meronen et al . [ 14 ] investigated the relationship in the opposite direction , by deriving activation functions such that the infinite width limit converges to a given GP prior . Since the growth of modern deep learning , relationships have also been established between infinitely wide deep networks and GPs [ 15 , 16 , 17 ] . Given these relationships , one may wonder whether GPs can supersede NNs , particularly given the convenience of Bayesian inference in them . Empirically though , finite NNs outperform their GP analogues [ 16 , 18 , 19 ] on high-dimensional tasks such as images . MacKay [ 12 ] explained this by noting that NNs lose their ability to learn features in the infinite limit , since every GP can be represented as a single-layer NN , with fixed features [ 20 , 21 ] . This observation justifies the effort of performing approximate Bayesian inference in finite deep networks , so both feature learning and uncertainty can be obtained . Renewed effort in Bayesian training procedures has focussed on computationally scalable techniques that take advantage of modern large datasets , and have provided usable uncertainty estimates [ e.g. , 22 , 23 , 24 , 25 ] . However , questions remain about the accuracy of these approximations . For accurate Bayesian inference , marginal likelihoods are expected to be usable for hyperparameter selection [ 26 , 27 ] . For most current approximations , there is either no positive or explicitly negative [ 23 ] evidence for this holding , although recent approaches do seem to provide usable estimates [ 28 , 29 ] . DGPs [ 3 ] provide an alternative approach to deep NNs , which use GP layers instead of weight-based layers , in order to take advantage of the improved uncertainty estimates afforded by having an infinite number of weights . The DGP representation is particularly promising because both early and recent work [ 3 , 30 , 31 ] shows that marginal likelihood estimates are usable for hyperparmeter selection , which indicates accurate inference . However , currently scalability and optimisation issues hinder widespread use . In this work , we provide a connection between the operational regimes of DGPs and NN , deriving an equivalence between the forward pass of NNs and propagating the means through the layers of a DGP . We share this goal with Sun et al . [ 32 ] , who recently proposed the use of neural network inducing variables for GPs based on the decomposition of zonal kernels in spherical harmonics . However , compared to Sun et al . [ 32 ] , our method does not suffer from variance over-estimation , which fundamentally limits the quality of the posterior . Our approach also leads to proper GP models , as opposed to adding a variance term to NNs using a Nystöm approximation , which allows the correct extension to DGPs and the optimisation of hyperparameters . These improvements are made possible by the theoretical analysis of the spectral densities of the kernel and inducing variable ( Section 4.3 ) . 3 Background . In this section , we review Deep Gaussian processes ( DGPs ) , with a particular focus on the structure of the approximate posterior and its connection to deep NNs . We then discuss how inducing variables control the DGP activation function , which is a key ingredient for our method . 3.1 Deep Gaussian Processes and Sparse Variational Approximate Inference . GPs are defined as an infinite collection of random variables { f ( x ) } x2Rd such that for all n 2 N , the distribution of any finite subset { f ( x1 ) , . . . , f ( xn ) } is multivariate normal . The GP is fully specified by its mean function µ ( x ) = E [ f ( x ) ] and its kernel k ( x , x0 ) = Cov ( f ( x ) , f ( x0 ) ) , shorthanded as f ( · ) ⇠ GP ( µ ( · ) , k ( · , · ) ) . GPs are often used as priors over functions in supervised learning , as for Gaussian likelihoods the posterior f ( · ) | D given a dataset D = { xi 2 Rd , yi 2 R } Ni=1 is tractable . Surprisingly diverse properties of the GP prior can be specified by the kernel [ 2 , 33 , 34 , 35 , 36 ] , although many problems require the flexible feature learning that deep NNs provide . DGPs [ 3 ] propose to solve this in a fully Bayesian way by composing several layers of simple GP priors , F ( · ) = fL . . . f2 f1 , where f ` ( · ) ⇠ GP 0 , k ` ( · , · ) . ( 1 ) Inference in DGPs is challenging as the composition is no longer a GP , leading to an intractable posterior F ( · ) | D. We follow the variational approach by Salimbeni and Deisenroth [ 5 ] due to its similarity to backpropagation in NNs . They use an approximate posterior consisting of an independent sparse GP for each layer . Each sparse GP is constructed by conditioning the prior on M inducing variables [ 37 , 38 , 39 ] , commonly function values u ` = { um ` = f ` ( wm ` ) } Mm=1 , and then specifying their marginal distribution as q ( u ` ) = N ( µ ` , ⌃ ` ) . This results in the approximate posterior process : q ( f ` ( · ) ) = GP ⇣ B > ` cu ` ( · ) ; k ` ( · , ·0 ) + c > u ` ( · ) C 1 u ` u ` ( ⌃ ` Cu ` u ` ) C 1 u ` u ` cu ` ( · 0 ) ⌘ , ( 2 ) where B ` = C 1u ` u ` µ ` 2 R M⇥d ` , d ` the dimensionality of the GP ’ s output , cu ` ( · ) = Cov ( f ` ( · ) , u ` ) and Cu ` u ` = Cov ( u ` , u ` ) . It is worth noting that , we use the symbol ‘ C ’ , rather than the more commonly used ‘ K ’ , to denote the fact that these matrices contain covariances which are not necessarily simple kernel evaluations . The variational parameters µ ` 2 RM⇥d ` and ⌃ ` 2 Rd ` ⇥M⇥M are selected by reducing the Kullback-Leibler ( KL ) divergence from the variational distribution to the true posterior . This is equivalent to maximising the Evidence Lower BOund ( ELBO ) . Taking the evaluations of the GP as h ` = f ` ( h ` 1 ) , the ELBO becomes [ 5 ] : ELBO = XN i=1 Eq ( hi , L ) [ log p ( yi |hi , L ) ] XL ` =1 KL [ q ( u ` ) k p ( u ` ) ] log p ( y ) . ( 3 ) 3.2 Connection between Deep Gaussian processes and Deep Neural Networks . Hensman and Lawrence [ 6 ] observed that the composite function of propagating an input through each layer ’ s variational mean ( Eq . ( 2 ) ) of a DGP equals : Eq [ fL ( · ) ] . . . Eq [ f1 ( · ) ] = B > LcuL ( · ) . . . B > 1 cu1 ( · ) , ( 4 ) which resembles the forward pass through fully-connected NN layers with non-linearity ( · ) : V > L ( WL · ) . . . V > 2 ( W2 · ) V > 1 ( W1 · ) ( 5 ) with W ` and V ` the pre-activation and output weights , respectively . Both models are visualised in Fig . 1 . Indeed , if we can formulate an approximation that makes the covariance cu ` ( · ) the same as a typical neural net activation function ( W ` · ) and set B ` equal to V ` , we obtain a formal mathematical equivalence between the forward pass of a DNN and propagating the mean of each layer through a DGP . This is one of the main contributions of this work . The remaining difference between the two models are then the so-called “ bottleneck ” layers in the DGP : h1 and h2 in Fig . 1a . This is a consequence of the DGP explicitly representing the output at each layer . However , while a NN does not explicitly represent the outputs , low-rank structure in the matrices W2V > 1 and W3V > 2 is typically found after training [ 40 ] , which strengthens the connection between both models .

Summary: This paper explores the connection between neural networks and Gaussian processes (GP). Specifically, this paper interprets the neural network activations as the inter-domain inducing variables for a GP. Specifically, this paper adopts the proposed approach in deep gaussian processes, proposing initialize a DGP by a pre-trained neural network based on the connection. They investigated the effectiveness of the approach by multiple experiments.