Deep Kernel Processes

1 INTRODUCTION . The deep learning revolution has shown us that effective performance on difficult tasks such as image classification ( Krizhevsky et al. , 2012 ) requires deep models with flexible lower-layers that learn task-dependent representations . Here , we consider whether these insights from the neural network literature can be applied to purely kernel-based methods . ( Note that we do not consider deep Gaussian processes or DGPs to be “ fully kernel-based ” as they use a feature-based representation in intermediate layers ) . Importantly , deep kernel methods ( e.g . Cho & Saul , 2009 ) already exist . In these methods , which are closely related to infinite Bayesian neural networks ( Lee et al. , 2017 ; Matthews et al. , 2018 ; Garriga-Alonso et al. , 2018 ; Novak et al. , 2018 ) , we take an initial kernel ( usually the dot product of the input features ) and perform a series of deterministic , parameter-free transformations to obtain an output kernel that we use in e.g . a support vector machine or Gaussian process . However , the deterministic , parameter-free nature of the transformation from input to output kernel means that they lack the capability to learn a top-layer representation , which is believed to be crucial for the effectiveness of deep methods ( Aitchison , 2019 ) . To obtain the flexibility necessary to learn a task-dependent representation , we propose deep kernel processes ( DKPs ) , which combine nonlinear transformations of the kernel , as in Cho & Saul ( 2009 ) with a flexible learned representation by exploiting a Wishart or inverse Wishart process ( Dawid , 1981 ; Shah et al. , 2014 ) . We find that models ranging from DGPs ( Damianou & Lawrence , 2013 ; Salimbeni & Deisenroth , 2017 ) to Bayesian neural networks ( BNNs ; Blundell et al. , 2015 , App . C.1 ) , infinite BNNs ( App . C.2 ) and infinite BNNs with bottlenecks ( App . C.3 ) can be written as DKPs ( i.e . only with kernel/Gram matrices , without needing features or weights ) . Practically , we find that the deep inverse Wishart process ( DIWP ) , admits convenient forms for variational approximate posteriors , and we give a novel scheme for doubly-stochastic variational inference ( DSVI ) with inducing points purely in the kernel domain ( as opposed to Salimbeni & Deisenroth , 2017 , who described DSVI for standard feature-based DGPs ) , and demonstrate improved performance with carefully matched models on fully-connected benchmark datasets . 2 BACKGROUND . We briefly revise Wishart and inverse Wishart distributions . The Wishart distribution is a generalization of the gamma distribution that is defined over positive semidefinite matrices . Suppose that we have a collection of P -dimensional random variables xi with i ∈ { 1 , . . . , N } such that xi iid∼ N ( 0 , V ) , then , ∑N i=1xix T i = S ∼ W ( V , N ) ( 1 ) has Wishart distribution with scale matrix V and N degrees of freedom . When N > P − 1 , the density is , W ( S ; V , N ) = 1 2NP |V|ΓP ( N 2 ) |S| ( N−P−1 ) /2 exp ( − 12 Tr ( V−1S ) ) , ( 2 ) where ΓP is the multivariate gamma function . Further , the inverse , S−1 has inverse Wishart distribution , W−1 ( V−1 , N ) . The inverse Wishart is defined only for N > P − 1 and also has closed-form density . Finally , we note that the Wishart distribution has mean NV while the inverse Wishart has mean V−1/ ( N − P − 1 ) ( for N > P + 1 ) . 3 DEEP KERNEL PROCESSES . We define a kernel process to be a set of distributions over positive definite matrices of different sizes , that are consistent under marginalisation ( Dawid , 1981 ; Shah et al. , 2014 ) . The two most common kernel processes are the Wishart process and inverse Wishart process , which we write in a slightly unusual form to ensure their expectation is K. We take G and G′ to be finite dimensional marginals of the underlying Wishart and inverse Wishart process , G ∼ W ( K /N , N ) , G′ ∼ W−1 ( δK , δ + ( P + 1 ) ) , ( 3a ) G∗ ∼ W ( K∗/N , N ) , G′∗ ∼ W−1 ( δK∗ , δ + ( P ∗ + 1 ) ) , ( 3b ) and where we explicitly give the consistent marginal distributions over K∗ , G∗ and G′∗ which are P ∗ × P ∗ principal submatrices of the P × P matrices K , G and G′ dropping the same rows and columns . In the inverse-Wishart distribution , δ is a positive parameter that can be understood as controlling the degree of variability , with larger values for δ implying smaller variability in G′ . We define a deep kernel process by analogy with a DGP , as a composition of kernel processes , and show in App . A that under sensible assumptions any such composition is itself a kernel process . 1 3.1 DGPS WITH ISOTROPIC KERNELS ARE DEEP WISHART PROCESSES . We consider deep GPs of the form ( Fig . 1 top ) with X ∈ RP×N0 K ` = { 1 N0 XXT for ` = 1 , K ( G ` −1 ) otherwise , ( 4a ) P ( F ` |K ` ) = ∏N ` λ=1N ( f ` λ ; 0 , K ` ) , ( 4b ) G ` = 1 N ` F ` F T ` . ( 4c ) Here , F ` ∈ RP×N ` are the N ` hidden features in layer ` ; λ indexes hidden features so f ` λ is a single column of F ` , representing the value of the λth feature for all training inputs . Note that K ( · ) is a 1Note that we leave the question of the full Kolmogorov extension theorem ( Kolmogorov , 1933 ) for matrices to future work : for our purposes , it is sufficient to work with very large but ultimately finite input spaces as in practice , the input vectors are represented by elements of the finite set of 32-bit or 64-bit floating-point numbers ( Sterbenz , 1974 ) . function that takes a Gram matrix and returns a kernel matrix ; whereas K ` is a ( possibly random ) variable representing a kernel matrix . Note , we have restricted ourselves to kernels , that can be written as functions of the Gram matrix , G ` , and do not require the full set of activations , F ` . As we describe later , this is not too restrictive , as it includes amongst others all isotropic kernels ( i.e . those that can be written as a function of the distance between points Williams & Rasmussen , 2006 ) . Note that we have a number of choices as to how to initialize the kernel in Eq . ( 4a ) . The current choice just uses a linear dot-product kernel , rather than immediately applying the kernel function K. This is both to ensure exact equivalence with infinite NNs with bottlenecks ( App . C.3 ) and also to highlight an interesting interpretation of this layer as Bayesian inference over generalised lengthscale hyperparameters in the squared-exponential kernel ( App . B e.g . Lalchand & Rasmussen , 2020 ) . For DGP regression , the outputs , Y , are most commonly given by a likelihood that can be written in terms of the output features , FL+1 . For instance , for regression , the distribution of the λth output feature column could be P ( yλ|FL+1 ) = N ( yλ ; f L+1 λ , σ 2I ) , ( 5 ) but our methods can be used with many other forms for the likelihood , including e.g . classification . The generative process for the Gram matrices , G ` , consists of generating samples from a Gaussian distribution ( Eq . 4b ) , and taking their product with themselves transposed ( Eq . 4c ) . This exactly matches the generative process for a Wishart distribution ( Eq . 1 ) , so we can write the Gram matrices , G ` , directly in terms of the kernel , without needing to sample features ( Fig . 1 bottom ) , P ( G1|X ) =W ( 1 N1 ( 1 N0 XXT ) , N1 ) , ( 6a ) P ( G ` |G ` −1 ) =W ( K ( G ` −1 ) /N ` , N ` ) , for ` ∈ { 2 , . . . L } , ( 6b ) P ( FL+1|GL ) = ∏NL+1 λ=1 N ( fL+1λ ; 0 , K ( GL ) ) . ( 6c ) Except at the output , the model is phrased entirely in terms of positive-definite kernels and Gram matrices , and is consistent under marginalisation ( assuming a valid kernel ) and is thus a DKP . At a high level , the model can be understood as alternatively sampling a Gram matrix ( introducing flexibility in the representation ) , and nonlinearly transforming the Gram matrix using a kernel ( Fig . 2 ) . This highlights a particularly simple interpretation of the DKP as an autoregressive process . In a standard autoregressive process , we might propagate the current vector , xt , through a deterministic function , f ( xt ) , and add zero-mean Gaussian noise , ξ , xt+1 = f ( xt ) + σ 2ξ such that E [ xt+1|xt ] = f ( xt ) . ( 7 ) By analogy , the next Gram matrix has expectation centered on a deterministic transformation of the previous Gram matrix , E [ G ` |G ` −1 ] = K ( G ` −1 ) , ( 8 ) so G ` can be written as this expectation plus a zero-mean random variable , Ξ ` , that can be interpreted as noise , G ` = K ( G ` −1 ) + Ξ ` . ( 9 ) Note that Ξ ` is not in general positive definite , and may not have an analytically tractable distribution . This noise decreases as N ` increases , V [ G ` ij ] = V [ Ξ ` ij ] = 1N ` ( K2ij ( G ` −1 ) +K 2 ii ( G ` −1 ) K 2 jj ( G ` −1 ) ) . ( 10 ) Notably , as N ` tends to infinity , the Wishart samples converge on their expectation , and the noise disappears , leaving us with a series of deterministic transformations of the Gram matrix . Therefore , we can understand a deep kernel process as alternatively adding “ noise ” to the kernel by sampling e.g . a Wishart or inverse Wishart distribution ( G2 and G3 in Fig . 2 ) and computing a nonlinear transformation of the kernel ( K ( G2 ) and K ( G3 ) in Fig . 2 ) Remember that we are restricted to kernels that can be written as a function of the Gram matrix , K ` = K ( G ` ) = Kfeatures ( F ` ) , K ` ij = k ( F ` i , : , F ` j , : ) . ( 11 ) where Kfeatures ( · ) takes a matrix of features , F ` , and returns the kernel matrix , K ` , and k is the usual kernel , which takes two feature vectors ( rows of F ` ) and returns an element of the kernel matrix . This does not include all possible kernels because it is not possible to recover the features from the Gram matrix . In particular , the Gram matrix is invariant to unitary transformations of the features : the Gram matrix is the same for F ` and F′ ` = UF ` where U is a unitary matrix , such that UU T = I , G ` = 1 N ` F ` F T ` = 1 N ` F ` U ` U T ` F T ` = 1 N ` F′ ` F ′T ` . ( 12 ) Superficially , this might seem very limiting — leaving us only with dot-product kernels ( Williams & Rasmussen , 2006 ) such as , k ( f , f ′ ) = f · f ′ + σ2 . ( 13 ) However , in reality , a far broader range of kernels fit within this class . Importantly , isotropic or radial basis function kernels including the squared exponential and Matern depend only on the squared distance between points , R , ( Williams & Rasmussen , 2006 ) k ( f , f ′ ) = k ( R ) , R = |f − f ′|2 . ( 14 ) These kernels can be written as a function of G , because the matrix of squared distances , R , can be computed from G , R ` ij = 1 N ` ∑N ` λ=1 ( F ` iλ − F ` jλ ) 2 = 1N ` ∑N ` λ=1 ( ( F ` iλ ) 2 − 2F ` iλF ` jλ + ( F ` jλ ) 2 ) = G ` ii − 2G ` ij +G ` jj . ( 15 )

Summary: This paper proposes deep kernel processes (DKPs), which can be viewed as a specific kind of deep Gaussian processes where the kernel can be written as a function of the Gram matrix. The features in the intermediate layers are integrated out and the Gram matrix are Wishart distributed. A doubly stochastic variational inference method is proposed to learn DKPs. The idea looks novel to me. My major concern is about the writing.