Details on Gaussian Process Regression (GPR) and Semi-GPR Modeling

This report tends to provide details on how to perform predictions using Gaussian process regression (GPR) modeling. In this case, we represent proofs for prediction using non-parametric GPR modeling for noise-free predictions as well as prediction using semi-parametric GPR for noisy observations. 1. GAUSSIAN PROCESS REGRESSION Gaussian processes (GPs) are widely used for modeling a phenomenon based on the observed spatiotemporal data [12]. A GP can be used as a tool for either classification or regression [1, 2, 12]. GPs have been used for decades as a supervised learning tool for regression problems known as Gaussian process regression (GPR) models [1, 2], and are also referred to as kriging, named after the mining engineer D.G. Krige in the geostatistics literature [3–5]. GPR models and kriging methods are applicable to a variety of problems such as the prediction and estimation of temperature, precipitation, missing pixel and un-mixing of pixels in hyperspectral imaging (HSI), human head pose estimation, concentration of carbon dioxide in the atmosphere, etc. [2, 6–11]. In GP modeling, it is assumed that the phenomenon of interest (PoI) can be evaluated via an unknown and probably nonlinear function, which we denote by f(·). The arguments of the function comprise a variable set u referred to as the input data. For example, u can be defined as u= [ux, uy, uz, t] , where (ux, uy, uz) and t denote the spatial and temporal information about the measurements, respectively. Unlike parametric models such as linear regression, GP is non-parametric. In GP one defines a probability distribution function as a prior over the unknown function f(·), directly. In other words, GP defines a distribution over functions in the function space and the inference is performed directly in this space [2]. This is more general than a parametric model such as Bayesian linear regression, where the prior distribution is defined over the space of parameters. The GP model treats any observation as an outcome of a Gaussian random variable, and all of these random variables are jointly Gaussian. With this setting, any well-defined GP model only needs a mean accompanied with a positive definite covariance function. Under this assumption, GP provides a posterior distribution over the unknown function f once data are observed. Therefore, for any set of N observations with the input data set {u1, . ,uN}, GP assumes that the distribution p(f(u1), . , f(uN)) is jointly Gaussian with some mean μ(U) and a covariance matrix K(U), where U :=[u1, . ,uN ]. The entry in row i and column j of K(U) is denoted by [K(U)]ij = κ(ui,uj), where κ(., .) is a positive definite kernel function. The kernel function specifies This work is supported by the Grant NASA NNX13 AD 39 A. the covariance between the pairs of random variables at the corresponding data points. The GP model is defined as follows [13]: f(U) ∼ GP(μ(U), K(U)), (1) where f(U) :=[f(u1), . , f(uN)] and μ(U) :=[μ(u1), . , μ(uN)] . For the regression purposes, GPR predicts the behavior of the PoI at the unseen data points using the available training data set. The GPR model can also handle noisy observations. Suppose we have access to a set of N noisy observations y= f(U)+ , where ∼N (0, σ nIN) and yn=f(un)+ n,∀n=1, . , N . The pair (un, yn) is the nth training data. Using the GPR modeling, the goal then becomes predicting the underlying function f evaluated at some other input data set U? , i.e., inferring f(U? ), where U? := [u? ,1, . ,u? ,M ] T . The set U? serves as the input test data set. Based on GP modeling, the prior joint distribution between the training and test data can be expressed as [13] [ y f? ] = N ([μy μf? ] , [ K(U,U) K(U? , U) K(U,U? ) K(U? , U? ) ]) , (2) where f? denotes f(U? ) and [K(U,U? )]nm = κ(un,u? ,m). The predictive distribution over the test data, using the existing rules for conditioning Gaussian distributions, is expressed as follows f? |U,y, U? ∼ N (μf? ,Σf? ), (3) where μf? =μy +K(U? , U) ( K(U,U) + σ nI )−1 (y − μy) Σf? =K(U? , U? )−K(U? , U) ( K(U,U)+σ nI )−1 K(U,U? ). (4) Therefore, the point estimate for f(U? ) is the mean μf? and the amount of uncertainty in the estimations is represented by the covariance Σf? in (4). Design of the covariance function requires incorporating some prior knowledge about the behavior of the PoI as it determines the amount of correlation between any pair of data points [1]. Some of the most widely used covariance functions are squared exponential kernel (κSE(u, u′) = exp {− (u−u ′)2 2l2 }) and rational quadratic (κRQ(u, u′) = (1+ (u−u ′)2 2αl2 )−α), where l and α are hyperparameters [2]. These kernels fall in the category of stationary covariance functions. Once the structure of the covariance function is selected, the corresponding hyperparameters in the model can be chosen either empirically or using some quantified statistical methods. In the empirical approach, the selection of hyperparameters is usually achieved using the empirical features obtained from the observed data such as the smoothness or periodic behavior of the samples. Remark 1: Although GPs are powerful tools for regression and classification problems, they suffer from high computational complexity as the sample size of the training data set increases. This problem occurs because the estimation of the test data involves inverting the covariance matrix of the training data which grows as more data are collected. Regarding the complexity of GPs, there exist some approaches such as the one for truncated covariance matrices in GPs [14], online sparse matrix GPs (OSMGP) algorithm [8], sparse greedy GP (SGGP) approximation method [15], and reduced rank GP (RRGP) [16]. Furthermore, there exist some studies on estimating the covariance matrix instead of an experimentally designed kernel function. For instance, Xu and Choi provided an approach to estimate and improve the quality of covariance function for anisotropic spatio-temporal GP using mobile sensor networks [17]. The suggested sampling method for such problem is based on minimizing the information-theoretic cost function of the Fisher information [17]. 2. PREDICTION USING NON-PARAMETRIC GPR FOR NOISE-FREE OBSERVATIONS Suppose we observe a training data set D = { (xi, fi), i = 1, . , N } and fi = f(xi), where xi and fi denote the ithe set of inputs and the corresponding output, respectively. Given a test set X∗ of size N∗ ×D, the goal is to predict the set of outputs collected into the vector f∗. By definition of the GP, the joint distribution has the following form [ f f∗ ] = N ([ μ μ∗ ] , [ K K∗ K ∗ K∗∗ ]) , where K = κ(X,X) is N ×N , K∗ = κ(X,X∗) is N ×N∗, K∗∗ = κ(X∗, X∗) is N∗ ×N∗, and κ(,̇)̇ is a kernel function. Then, the posterior distribution over f∗ becomes [2, 13] p(f∗|X∗, X, f) = N (f∗|μf∗ ,Σf∗), (5) where { μf∗ = μ∗ +K T ∗ K −1(f − μ(X)), μ∗ := μ(X∗) Σf∗ = K∗∗ −K ∗ KK∗. Below, we provide details on how to derive (5) borrowed from [18]. Remark 2: As a brief review, the inversion of a matrix in block form can be represented as [ A B C D ]−1 = [ E F G H ] , (6) where  E = A−1 + A−1B(D − CA−1B)−1CA−1 F = −A−1B(D − CA−1B)−1 G = −(D − CA−1B)−1CA−1 H = (D − CA−1B)−1. Using Remark 2, the logarithm of the posterior distribution over f∗ in (5) is proportional to log {p(f∗|X∗, X,y)} ∝ − ([ (f − μ) (f∗ − μ∗) ] [ E F G H ] [ f − μ f∗ − μ∗ ]) , where [ E F G H ] = [ K K∗ K ∗ K∗∗ ]−1 .