Marginalised Gaussian Processes with Nested Sampling

1 Introduction . Gaussian processes ( GPs ) represent a powerful non-parametric and probabilistic framework for performing regression and classification . An important step in using GP models is the specification of a covariance function and in turn setting the parameters of the chosen covariance function . Both these steps are jointly referred to as the model selection problem [ Rasmussen and Williams , 2006 ] . The parameters of the covariance function are called hyperparameters as the latent function values take the place of parameters in a GP . The bulk of the GP literature addresses the model selection problem through maximisation of the GP marginal likelihood p ( y|θ ) . This approach called ML-II2 typically involves using gradient based optimisation methods to yield point estimates of hyperparameters . The posterior predictive distribution is then evaluated at these optimised hyperparameter point estimates . Despite its popularity this classical approach to training GPs suffers from two issues : 1 ) Using point estimates of hyperparameters yield overconfident predictions , by failing to account for hyperparameter uncertainty , and 2 ) Non-convexity of the marginal likelihood surface can lead to poor estimates located at local minima . Further , the presence of multiple modes can affect the interpretability of kernel hyperparameters . This work proposes a principled treatment of model hyperparameters and assesses its impact on the quality of prediction uncertainty . Marginalising over hyperparameters can also be seen as a robust approach to the model selection question in GPs . The form of the kernel function influences the geometry of the marginal likelihood surface . For instance , periodic kernels give rise to multiple local minima as functions with different periodicities can be compatible with the data . Expressive kernels which are derived by adding/multiplying together primitive kernels to encode different types of inductive biases typically have many hyperparameters , exacerbating the local minima problem . ∗Equal contribution 2where ‘ ML ’ stands for marginal likelihood and ‘ II ’ denotes the second level of the hierarchy pertaining to hyperparameters 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . The spectral mixture ( SM ) kernel proposed in Wilson and Adams [ 2013 ] is an expressive class of kernels derived from the spectral density reparameterisation of the kernel using Bochner ’ s Therorem [ Bochner , 1959 ] . The SM kernel has prior support over all stationary kernels which means it can recover sophisticated structure provided sufficient spectral components are used . Several previous works [ Kom Samo and Roberts , 2015 , Remes et al. , 2017 , 2018 , Benton et al. , 2019 ] have attempted to further enhance the flexibility of spectral mixture kernels , such as the introduction of a time-dependent spectrum . However , we postulate that the key limitation in the SM kernel ’ s performance lies not in its stationarity or expressivity , but in the optimisation procedure . The form of the SM kernel gives rise to many modes in the marginal likelihood , making optimisation especially challenging . It therefore presents an excellent opportunity to test out nested sampling ’ s capabilities , as it is an inference technique which is highly effective at navigating multimodal distributions . We note that aside from being successfully applied to graphical models by Murray et al . [ 2006 ] , nested sampling has been largely overlooked in the machine learning literature . Below we provide a summary of our main contributions : • Highlight some of the failure modes of ML-II training . We provide insights into the effectiveness of ML-II training in weak and strong data regimes . • Present a viable set of priors for the hyperparameters of the spectral mixture kernel . • Propose the relevance of Nested Sampling ( NS ) as an effective means of sampling from the hyperparameter posterior [ see also Faria et al. , 2016 , Aksulu et al. , 2020 ] , particularly in the presence of a multimodal likelihood . • We demonstrate that in several time series modelling tasks ( where we predict the future given past observations ) , incorporating hyperparameter uncertainty yields superior prediction intervals . 2 Background . This section provides a brief account of marginalised Gaussian process regression , followed by an outline of the spectral mixture kernel [ Wilson and Adams , 2013 ] which we shall use in the numerical experiments . Marginalised Gaussian Processes . Given some input-output pairs ( X , y ) = { xi , yi } Ni=1 where yi are noisy realizations of latent function values fi with Gaussian noise , yi = fi+ i , i ∼ N ( 0 , σ2n ) , we seek to infer some as-yet unseen values y∗ . Let kθ ( xi , xj ) denote a positive definite kernel function parameterized with hyperparameters θ . Following the prescription of Lalchand and Rasmussen [ 2020 ] , the marginalised GP framework is given by , Hyperprior : θ ∼ p ( θ ) , GP Prior : f |X , θ ∼ N ( 0 , Kθ ) , Likelihood : y|f ∼ N ( f , σ2nI ) , ( 1 ) where Kθ denotes the N ×N covariance matrix , ( Kθ ) i , j = kθ ( xi , xj ) . The predictive distribution for unknown test inputs X ? integrates over the joint posterior3 , p ( f ? |y ) = ∫∫ p ( f ? |f , θ ) p ( f |θ , y ) p ( θ|y ) dfdθ ( 2 ) For Gaussian noise , the integral over f can be treated analytically as follows p ( f ? |y ) = ∫ p ( f ? |y , θ ) p ( θ|y ) dθ ' 1 M M∑ j=1 p ( f ? |y , θj ) = 1 M M∑ j=1 N ( µ ? j , Σ ? j ) , where θ is dealt with numerically and θj correspond to draws from the hyperparameter posterior p ( θ|y ) . The distribution inside the summation represents the standard GP posterior predictive for a fixed hyperparameter setting θj . Thus , the final form of the posterior predictive in marginalised GPs is approximated by a mixture of Gaussians ( see Rasmussen and Williams [ 2006 ] for a review of GPs ) . Throughout this work we shall adopt a Gaussian likelihood , hence the only intractable integrand we need to consider is the hyperparameter posterior p ( θ|y ) , p ( θ|y ) ∝ p ( y|θ ) p ( θ ) ( 3 ) 3Implicitly conditioning over inputs X , X ? for compactness . For non-Gaussian likelihoods , one would have to approximate the joint posterior over f and θ . In related work , Filippone and Girolami [ 2014 ] compared a range of MCMC algorithms for marginalised Gaussian processes . Murray and Adams [ 2010 ] used a slice sampling scheme for general likelihoods specifically addressing the coupling between f and θ. Titsias and Lázaro-Gredilla [ 2014 ] presented an approximation scheme using variational inference . Jang et al . [ 2018 ] considered a Lévy process over the spectral density . Most recently , Lalchand and Rasmussen [ 2020 ] demonstrated marginalisation with HMC and variational inference . Spectral Mixture Kernels . The spectral mixture kernel is of particular interest in this work , due to its complex likelihood surface , thereby offering a difficult challenge for sampling methods and optimisation methods alike . The kernel is characterised by a spectral density S ( ν ) , defined as a mixture of Q Gaussian pairs [ Wilson and Adams , 2013 ] : S ( ν ) = Q∑ i=1 wi 2 [ G ( ν , µi , σi ) +G ( ν , −µi , σi ) ] . ( 4 ) Here the weight wi specifies the variance contributed by the ith component whileG ( ν , µi , σi ) denotes a Gaussian function with mean µi and standard deviation σi . To avoid confusion with other standard deviations , and convey its physical significance , we shall refer to σi as the bandwidth . The formalism of ( 4 ) is readily expanded to higher dimensions . If the power spectrum is represented by a sum of multivariate Gaussians , each with a diagonal covariance , with entries collected in the D dimensional vector σi , the corresponding kernel k ( τ = x− x′ ) takes the form , k ( τ ) = Q∑ i=1 wi cos ( 2πτ · µi ) D∏ d=1 exp ( −2π2τ2dσ2i ( d ) ) , ( 5 ) where τd , µ ( d ) i , σ ( d ) i are the d th elements of the D dimensional vectors τ , µi and σi respectively . The vector of kernel hyperparameters θ = { wi , µi , σi } Qi=1 is typically unknown , we account for this uncertainty by treating them as random variables . Sampling and Symmetries . Fig . 1 shows the marginal likelihood surface for a 2-component SM kernel , given two datasets of different size . One of the striking features of these surfaces lies in their symmetry : this is due to the kernel ’ s invariance to the ordering of its components . The marginal likelihood of a SM kernel with Q spectral components possesses Q ! identical regions of parameter space . A naive attempt to explore the full posterior distribution of a spectral mixture kernel would try to quantify probability mass across these degenerate regions , a much more computationally intensive task than is necessary . One solution is to only sample from one region , and ignore its symmetric counterparts . To achieve this , we can adopt an approach known as forced identifiability [ Buscicchio et al. , 2019 ] to ensure that the components are defined in sequential order with respect to their frequency µ . While sampling from degenerate ( symmetric ) modes will not improve performance , in low data regimes the modes are not always well-identified as the LML surface favours several local optima . Hence , a sampling scheme that is capable of sampling from multiple modes will give rise to a diversity of functions compatible with the data . 3 Applying Nested Sampling to Gaussian Processes . This section summarises our proposal for how Nested Sampling ( NS ) can be used to marginalise the hyperparameters of Gaussian Processes , while simultaneously yielding an estimate of the model evidence . Nested Sampling . The nested sampling algorithm was developed by Skilling [ 2004 ] ( see also Skilling et al . [ 2006 ] ) as a means of estimating the model evidence Z = ∫ ψ ( θ ) π ( θ ) dθ . Here π ( θ ) denotes the prior , and ψ is the likelihood , which for our purposes is given by the GP marginal likelihood p ( y|θ ) . Irrespective of the dimensionality of θ , the integral can be collapsed to a onedimensional integral over the unit interval , Z = ∫ 1 0 ψ ( X ) dX . Here X is the quantile function associated with the likelihood : it describes the prior mass lying below the likelihood value ψ . The extreme values of the integrand , ψ ( X=0 ) and ψ ( X=1 ) , therefore correspond to the minimum and maximum likelihood values found under the support of the prior π ( θ ) . Sampling proceeds in accordance with Algorithm 1 . It begins with a random selection of a set of live points drawn from the prior . The number of live points , nL , dictates the resolution at which the likelihood surface will be explored . At each iteration , a new sample is drawn from the prior and replaces the live point with the lowest likelihood , under the strict conditions that it possesses a higher likelihood . By following this iterative procedure the set of live points gradually converge upon regions of higher likelihood . The prior mass enclosed by these points shrinks exponentially . By the jth iteration , the expectation of X has reduced to 〈Xj〉 = [ nL/ ( nL + 1 ) ] jX0 ≈ e−j/NX0 . Higher values of nL therefore come at the cost of a slower convergence rate . While the sequence of samples provides an estimate of Z , an invaluable quantity in the context of Bayesian model selection , they also represent importance weighted samples of the posterior . If we wished to perform inference in a low-dimensional setting , then a uniform sampling strategy would suffice . However since we wish to explore higher dimensions , we employ the PolyChord algorithm [ Handley et al. , 2015 , Handley et al. , 2015 ] , which performs slice sampling [ Neal , 2003 ] at each iteration . Unless otherwise stated , we use 100 live points , which are bounded in a set of ellipsoids [ Feroz et al. , 2009 ] . These bounding surfaces allow the macroscopic structure of the likelihood contours to be traced , enabling a much more efficient sampling process . This approach has proven particularly adept at navigating multi-modal likelihood surfaces [ Allison and Dunkley , 2014 ] . These attractive properties have motivated numerous scientific applications , including the detection of gravitational waves [ Veitch et al. , 2015 ] , the categorisation of cosmic rays [ Cholis et al. , 2015 ] , and the imaging of a supermassive black hole [ Akiyama et al. , 2019 ] . We implemented4 the above algorithm via a combination of the DYNESTY [ Speagle , 2020 ] and GPflow [ De G. Matthews et al. , 2017 ] software packages . For the former , we make use the ‘ rslice ’ sampling option provided by DYNESTY , along with the default number of five slices , and adopt the default criterion for convergence . This is defined as the point at which the estimated posterior mass contained within the set of live points falls below 1 % of the total , specifically log ( Ẑ + ∆Ẑ ) − log ( Ẑ ) < 0.01 , where ∆Ẑ ' ψmaxXi .

Summary: The paper suggests using the nested sampling algorithm to marginalize hyperparameters of Gaussian processes with spectral mixture kernels. The motivation is that the posterior over hyperparameters of such kernels is highly multimodal and nested sampling is known to work well with multimodal distributions. Furthermore, practical priors for the parameters of spectral mixture kernels are suggested. The proposed nested sampling driven marginalised Gaussian process is compared to NUTS driven marginalised Gaussian process and the Gaussian process with hyperparameters estimated by maximum likelihood on a set of low-dimensional benchmark problems. The comparison is in terms of negative log predictive density. It is shown that the proposed method usually outperforms other methods while also performing faster than NUTS.