Deep Kernel Processes

1 INTRODUCTION . The deep learning revolution has shown us that effective performance on difficult tasks such as image classification ( Krizhevsky et al. , 2012 ) requires deep models with flexible lower-layers that learn task-dependent representations . Here , we consider whether these insights from the neural network literature can be applied to purely kernel-based methods . ( Note that we do not consider deep Gaussian processes or DGPs to be “ fully kernel-based ” as they use a feature-based representation in intermediate layers ) . Importantly , deep kernel methods ( e.g . Cho & Saul , 2009 ) already exist . In these methods , which are closely related to infinite Bayesian neural networks ( Lee et al. , 2017 ; Matthews et al. , 2018 ; Garriga-Alonso et al. , 2018 ; Novak et al. , 2018 ) , we take an initial kernel ( usually the dot product of the input features ) and perform a series of deterministic , parameter-free transformations to obtain an output kernel that we use in e.g . a support vector machine or Gaussian process . However , the deterministic , parameter-free nature of the transformation from input to output kernel means that they lack the capability to learn a top-layer representation , which is believed to be crucial for the effectiveness of deep methods ( Aitchison , 2019 ) . To obtain the flexibility necessary to learn a task-dependent representation , we propose deep kernel processes ( DKPs ) , which combine nonlinear transformations of the kernel , as in Cho & Saul ( 2009 ) with a flexible learned representation by exploiting a Wishart or inverse Wishart process ( Dawid , 1981 ; Shah et al. , 2014 ) . We find that models ranging from DGPs ( Damianou & Lawrence , 2013 ; Salimbeni & Deisenroth , 2017 ) to Bayesian neural networks ( BNNs ; Blundell et al. , 2015 , App . C.1 ) , infinite BNNs ( App . C.2 ) and infinite BNNs with bottlenecks ( App . C.3 ) can be written as DKPs ( i.e . only with kernel/Gram matrices , without needing features or weights ) . Practically , we find that the deep inverse Wishart process ( DIWP ) , admits convenient forms for variational approximate posteriors , and we give a novel scheme for doubly-stochastic variational inference ( DSVI ) with inducing points purely in the kernel domain ( as opposed to Salimbeni & Deisenroth , 2017 , who described DSVI for standard feature-based DGPs ) , and demonstrate improved performance with carefully matched models on fully-connected benchmark datasets . 2 BACKGROUND . We briefly revise Wishart and inverse Wishart distributions . The Wishart distribution is a generalization of the gamma distribution that is defined over positive semidefinite matrices . Suppose that we have a collection of P -dimensional random variables xi with i ∈ { 1 , . . . , N } such that xi iid∼ N ( 0 , V ) , then , ∑N i=1xix T i = S ∼ W ( V , N ) ( 1 ) has Wishart distribution with scale matrix V and N degrees of freedom . When N > P − 1 , the density is , W ( S ; V , N ) = 1 2NP |V|ΓP ( N 2 ) |S| ( N−P−1 ) /2 exp ( − 12 Tr ( V−1S ) ) , ( 2 ) where ΓP is the multivariate gamma function . Further , the inverse , S−1 has inverse Wishart distribution , W−1 ( V−1 , N ) . The inverse Wishart is defined only for N > P − 1 and also has closed-form density . Finally , we note that the Wishart distribution has mean NV while the inverse Wishart has mean V−1/ ( N − P − 1 ) ( for N > P + 1 ) . 3 DEEP KERNEL PROCESSES . We define a kernel process to be a set of distributions over positive definite matrices of different sizes , that are consistent under marginalisation ( Dawid , 1981 ; Shah et al. , 2014 ) . The two most common kernel processes are the Wishart process and inverse Wishart process , which we write in a slightly unusual form to ensure their expectation is K. We take G and G′ to be finite dimensional marginals of the underlying Wishart and inverse Wishart process , G ∼ W ( K /N , N ) , G′ ∼ W−1 ( δK , δ + ( P + 1 ) ) , ( 3a ) G∗ ∼ W ( K∗/N , N ) , G′∗ ∼ W−1 ( δK∗ , δ + ( P ∗ + 1 ) ) , ( 3b ) and where we explicitly give the consistent marginal distributions over K∗ , G∗ and G′∗ which are P ∗ × P ∗ principal submatrices of the P × P matrices K , G and G′ dropping the same rows and columns . In the inverse-Wishart distribution , δ is a positive parameter that can be understood as controlling the degree of variability , with larger values for δ implying smaller variability in G′ . We define a deep kernel process by analogy with a DGP , as a composition of kernel processes , and show in App . A that under sensible assumptions any such composition is itself a kernel process . 1 3.1 DGPS WITH ISOTROPIC KERNELS ARE DEEP WISHART PROCESSES . We consider deep GPs of the form ( Fig . 1 top ) with X ∈ RP×N0 K ` = { 1 N0 XXT for ` = 1 , K ( G ` −1 ) otherwise , ( 4a ) P ( F ` |K ` ) = ∏N ` λ=1N ( f ` λ ; 0 , K ` ) , ( 4b ) G ` = 1 N ` F ` F T ` . ( 4c ) Here , F ` ∈ RP×N ` are the N ` hidden features in layer ` ; λ indexes hidden features so f ` λ is a single column of F ` , representing the value of the λth feature for all training inputs . Note that K ( · ) is a 1Note that we leave the question of the full Kolmogorov extension theorem ( Kolmogorov , 1933 ) for matrices to future work : for our purposes , it is sufficient to work with very large but ultimately finite input spaces as in practice , the input vectors are represented by elements of the finite set of 32-bit or 64-bit floating-point numbers ( Sterbenz , 1974 ) . function that takes a Gram matrix and returns a kernel matrix ; whereas K ` is a ( possibly random ) variable representing a kernel matrix . Note , we have restricted ourselves to kernels , that can be written as functions of the Gram matrix , G ` , and do not require the full set of activations , F ` . As we describe later , this is not too restrictive , as it includes amongst others all isotropic kernels ( i.e . those that can be written as a function of the distance between points Williams & Rasmussen , 2006 ) . Note that we have a number of choices as to how to initialize the kernel in Eq . ( 4a ) . The current choice just uses a linear dot-product kernel , rather than immediately applying the kernel function K. This is both to ensure exact equivalence with infinite NNs with bottlenecks ( App . C.3 ) and also to highlight an interesting interpretation of this layer as Bayesian inference over generalised lengthscale hyperparameters in the squared-exponential kernel ( App . B e.g . Lalchand & Rasmussen , 2020 ) . For DGP regression , the outputs , Y , are most commonly given by a likelihood that can be written in terms of the output features , FL+1 . For instance , for regression , the distribution of the λth output feature column could be P ( yλ|FL+1 ) = N ( yλ ; f L+1 λ , σ 2I ) , ( 5 ) but our methods can be used with many other forms for the likelihood , including e.g . classification . The generative process for the Gram matrices , G ` , consists of generating samples from a Gaussian distribution ( Eq . 4b ) , and taking their product with themselves transposed ( Eq . 4c ) . This exactly matches the generative process for a Wishart distribution ( Eq . 1 ) , so we can write the Gram matrices , G ` , directly in terms of the kernel , without needing to sample features ( Fig . 1 bottom ) , P ( G1|X ) =W ( 1 N1 ( 1 N0 XXT ) , N1 ) , ( 6a ) P ( G ` |G ` −1 ) =W ( K ( G ` −1 ) /N ` , N ` ) , for ` ∈ { 2 , . . . L } , ( 6b ) P ( FL+1|GL ) = ∏NL+1 λ=1 N ( fL+1λ ; 0 , K ( GL ) ) . ( 6c ) Except at the output , the model is phrased entirely in terms of positive-definite kernels and Gram matrices , and is consistent under marginalisation ( assuming a valid kernel ) and is thus a DKP . At a high level , the model can be understood as alternatively sampling a Gram matrix ( introducing flexibility in the representation ) , and nonlinearly transforming the Gram matrix using a kernel ( Fig . 2 ) . This highlights a particularly simple interpretation of the DKP as an autoregressive process . In a standard autoregressive process , we might propagate the current vector , xt , through a deterministic function , f ( xt ) , and add zero-mean Gaussian noise , ξ , xt+1 = f ( xt ) + σ 2ξ such that E [ xt+1|xt ] = f ( xt ) . ( 7 ) By analogy , the next Gram matrix has expectation centered on a deterministic transformation of the previous Gram matrix , E [ G ` |G ` −1 ] = K ( G ` −1 ) , ( 8 ) so G ` can be written as this expectation plus a zero-mean random variable , Ξ ` , that can be interpreted as noise , G ` = K ( G ` −1 ) + Ξ ` . ( 9 ) Note that Ξ ` is not in general positive definite , and may not have an analytically tractable distribution . This noise decreases as N ` increases , V [ G ` ij ] = V [ Ξ ` ij ] = 1N ` ( K2ij ( G ` −1 ) +K 2 ii ( G ` −1 ) K 2 jj ( G ` −1 ) ) . ( 10 ) Notably , as N ` tends to infinity , the Wishart samples converge on their expectation , and the noise disappears , leaving us with a series of deterministic transformations of the Gram matrix . Therefore , we can understand a deep kernel process as alternatively adding “ noise ” to the kernel by sampling e.g . a Wishart or inverse Wishart distribution ( G2 and G3 in Fig . 2 ) and computing a nonlinear transformation of the kernel ( K ( G2 ) and K ( G3 ) in Fig . 2 ) Remember that we are restricted to kernels that can be written as a function of the Gram matrix , K ` = K ( G ` ) = Kfeatures ( F ` ) , K ` ij = k ( F ` i , : , F ` j , : ) . ( 11 ) where Kfeatures ( · ) takes a matrix of features , F ` , and returns the kernel matrix , K ` , and k is the usual kernel , which takes two feature vectors ( rows of F ` ) and returns an element of the kernel matrix . This does not include all possible kernels because it is not possible to recover the features from the Gram matrix . In particular , the Gram matrix is invariant to unitary transformations of the features : the Gram matrix is the same for F ` and F′ ` = UF ` where U is a unitary matrix , such that UU T = I , G ` = 1 N ` F ` F T ` = 1 N ` F ` U ` U T ` F T ` = 1 N ` F′ ` F ′T ` . ( 12 ) Superficially , this might seem very limiting — leaving us only with dot-product kernels ( Williams & Rasmussen , 2006 ) such as , k ( f , f ′ ) = f · f ′ + σ2 . ( 13 ) However , in reality , a far broader range of kernels fit within this class . Importantly , isotropic or radial basis function kernels including the squared exponential and Matern depend only on the squared distance between points , R , ( Williams & Rasmussen , 2006 ) k ( f , f ′ ) = k ( R ) , R = |f − f ′|2 . ( 14 ) These kernels can be written as a function of G , because the matrix of squared distances , R , can be computed from G , R ` ij = 1 N ` ∑N ` λ=1 ( F ` iλ − F ` jλ ) 2 = 1N ` ∑N ` λ=1 ( ( F ` iλ ) 2 − 2F ` iλF ` jλ + ( F ` jλ ) 2 ) = G ` ii − 2G ` ij +G ` jj . ( 15 )

Summary: This paper proposes a prior distribution over covariance matrices of kernels which is defined as a sequential graphical model where each variable is Wishart distributed and its scale matrix is a non-linear transformation of its predecesor variable on the graph. The paper begins by considering a DGP with isotropic kernels across the layers and realizes that the Gram matrices are Wishart distributed. Based on this, the paper proposes to bypass the inference of the features and sample the Gram matrices directly from Wishart distributions. This insight, in addition to the layered structure of DGPs, gives rise to the proposed prior distribution. Furthermore, given the restrictions of the Wishart distribution for modelling covariance matrices of arbitrary size [1], as well as the conjugacy properties of the inverse Wishart distribution, the paper uses the inverse Wishart distribution instead. Doubly stochastic variational inference is proposed for approximating the posterior distribution which includes the use of inducing points thanks to the marginalization properties of the inverse Wishart distribution. The experimental contribution consists of a comparison against DGP and Neural Network GP on the UCI, MNIST and CIFAR-10 dataset.