## 3 Review of Gaussian Process Regression Models
### GP and Sparse GP Regression  
Gaussian processes (GP) are a set of methods that generalize the multivariate normal to infinite dimensions. Not only do GPs have a flexible non-parametric form, GP methods are also attractive because they offer principled uncertainty quantification via a predictive distribution. For supervised learning problems, \(\mathcal{GP}\) prior models have the distinct advantage of allowing the user to automatically learn the correct functional form linking the input space \(\mathcal{X}\) to the output space \(\mathcal{Y}\). This is achieved by specifying a prior over the distribution of functions, which then allows the derivation of the posterior distribution over these functions once data have been observed. Throughout this paper, we use the notation \(f(\cdot)\sim\mathcal{GP}(\mu(\cdot),k(\cdot,\cdot))\) to denote a generic GP prior with mean function \(\mu(\cdot)\) and covariance function \(k(\cdot,\cdot)\). Let \(y_{i}\in\mathbb{R}\) denote an observed response and \(\mathbf{x}_{i}\) and \(\mathbf{x}_{j}\) be two distinct input vectors in \(\mathbb{R}^{p}\). The GP regression model is defined as  
\[y_{i} =f(\mathbf{x}_{i})+\varepsilon_{i},\;\varepsilon_{i}\sim\mathcal{ N}(0,\sigma_{y}^{2}), \tag{1}\] \[f(\cdot) \sim\mathcal{GP}(\mu(\cdot),k(\cdot,\cdot)), \tag{2}\]  
where the mean function and the covariance kernel are, respectively,  
\[\mu_{\mathbf{x}_{i}} =E[f(\mathbf{x}_{i})], \tag{3}\] \[k_{\mathbf{x}_{i},\mathbf{x}_{j}} =E[(f(\mathbf{x}_{i})-\mu(\mathbf{x}_{i}))(f(\mathbf{x}_{j})-\mu (\mathbf{x}_{j}))], \tag{4}\]  
where \(\mathbf{X}=(\mathbf{x}_{1},\ldots,\mathbf{x}_{n})\) are fixed predictors, and \(\mathbf{f}=(f(\mathbf{x}_{1}),\ldots,f(\mathbf{x}_{n}))^{\prime}\) is an \(n\)-dimensional vector. Given observed responses \(\mathbf{y}=(y_{1},\ldots,y_{n})^{\prime}\in\mathbb{R}^{n}\), it follows that the Gaussianprocess in (2) has a multivariate normal distribution with mean vector \(\mathbf{\mu}=\left(\mu_{\mathbf{x}_{1}},\dots,\mu_{\mathbf{x}_{n}}\right)^{\prime}\) and variance-covariance matrix \(\mathbf{K}_{\mathbf{x}\mathbf{x}}=\left\{k_{\mathbf{x}_{i},\mathbf{x}_{j}}\right\} \in\mathbb{R}^{n\times n}\). Standard GP regression provides convenient closed forms for posterior inference. The posterior distribution \(p(f(\mathbf{x}_{i})|\mathbf{X},\mathbf{y})\) is Gaussian with mean and variance, respectively, given by  
\[\mu_{i} =\mu_{\mathbf{x}_{i}}+\mathbf{k}_{i}^{\prime}\Big{[}\mathbf{K}_{ \mathbf{x}\mathbf{x}}+\sigma^{2}\mathbf{I}_{n}\Big{]}^{-1}(\mathbf{y}-\mathbf{\mu }),\] \[k_{i} =k_{\mathbf{x}_{i},\mathbf{x}_{i}}-\mathbf{k}_{i}^{\prime}\Big{[} \mathbf{K}_{\mathbf{x}\mathbf{x}}+\sigma^{2}\mathbf{I}_{n}\Big{]}^{-1} \mathbf{k}_{i}, \tag{5}\]  
where \(\mathbf{k}_{i}=(k_{\mathbf{x}_{i},\mathbf{x}_{j}},j=1,\dots,n)^{\prime}\). Given a new set of inputs \(\mathbf{X}_{*}\in\mathbb{R}^{p\times m}\), the joint distribution of the observed response \(\mathbf{y}\) and the GP prior \(f(\cdot)\) and the posterior predictive evaluated at new input set \(\mathbf{X}_{*}\) are  
\[ \begin{bmatrix}\mathbf{y}\\ \mathbf{f}_{*}\end{bmatrix} \sim\mathcal{N}_{n+m}\bigg{(} \begin{bmatrix}\mathbf{ \mu}_{t}\\ \mathbf{\mu}_{*}\end{bmatrix} ^{\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\! which in turn leads to approximating \(p(\mathbf{f})\) by \(q(\mathbf{f})\).