# Gaussian Processes for Signal Processing and Representation in Control Engineering
## 3 Computational Methods of Gaussian Process  
### Modelling and Computation of Hyperparameters  
The first problem, which comes to computing the GP is the hyperparameters of chosen kernel. Dealing with setting up those parameters mostly comes from complex Bayesian inference in complex hierarchical model with three levels of inference: posterior over parameters, posterior over hyperparameters and at the top posterior of model [1]. All levels can be described with formulas, where \(\mathbf{w}\) is a set of parameters, \(\boldsymbol{\theta}\) are hyperparameters and \(\mathcal{H}_{i}\) as set of possible model structures. Equation (11) (posterior over parameters) is the first level:  
\[p(\mathbf{w}\mid\mathbf{y},X,\boldsymbol{\theta},\mathcal{H}_{i})=\frac{p( \mathbf{y}\mid X,\mathbf{w},\mathcal{H}_{i})p(\mathbf{w}\mid\boldsymbol{ \theta},\mathcal{H}_{i})}{p(\mathbf{y}\mid X,\boldsymbol{\theta},\mathcal{H}_ {i})}, \tag{11}\]  
where \(p(\mathbf{y}\mid X,\boldsymbol{\theta},\mathcal{H}_{i})\) is independent of the parameters and calledmarginal likelihood. Marginal likelihood is given by equation number (12):  
\[p(\mathbf{y}\mid X,\boldsymbol{\theta},\mathcal{H}_{i})=\int p(\mathbf{y}\mid X,\mathbf{w},\mathcal{H}_{i})p(\mathbf{w}\mid\boldsymbol{\theta},\mathcal{H}_ {i})d\mathbf{w}. \tag{12}\]  
The second level equation (posterior over hyperparameters) is given by:  
\[p(\boldsymbol{\theta}\mid\mathbf{y},X,\mathcal{H}_{i})=\frac{p(\mathbf{y}\mid X,\boldsymbol{\theta},\mathcal{H}_{i})p(\boldsymbol{\theta}\mid\mathcal{H}_{i} )}{p(\mathbf{y}\mid X,\mathcal{H}_{i})}, \tag{13}\]  
where normalizing constant is:  
\[p(\mathbf{y}\mid X,\mathcal{H}_{i})=\int p(\mathbf{y}\mid X,\boldsymbol{ \theta},\mathcal{H}_{i})p(\boldsymbol{\theta}\mid\mathcal{H}_{i})d\boldsymbol{ \theta}. \tag{14}\]The top level (posterior of model) is given by:  
\[p(\mathcal{H}_{i}\mid\mathbf{y},X)=\frac{p(\mathbf{y}\mid X,\mathcal{H}_{i})p( \mathcal{H}_{i})}{p(\mathbf{y}\mid X)}. \tag{15}\]  
This approach can be difficult even to prepare, because it requires not only knowledge about the problem, but also the necessary Bayesian inference skills. Therefore, a fully Bayesian approach is a much less common solution in practice, although it may bring concrete results. Such modeling may result in a higher accuracy of the model and its easier adaptation in the case of small amounts of data. The shown integrals may not be analytically tractable, so idea is to use an analytical approximations or Markov chain Monte Carlo (MCMC) methods of sampling from a probability distribution. MCMC itself requires as to use more advanced methods like building whole model in Stan. This method, however, clearly makes GP more suitable to the certain problem with no harm to its overall flexibility in specific field. Drawback of such approach is computational burden of heavy computation requirements, so it may appear a necessity of approximation. Another way to accomplish the task of getting appropriate hyperparameters is called ML-II (type II maximum likelihood), which is based on maximizing marginal likelihood from Equation (12). Then the integral from Equation (14) can be approximated with use of a local expansion around the maximum (the Laplace approximation). In fact we should be careful, with such optimization approach, because of its tendency to overfitting (especially with more number of hyperparameters). In general ML-II has a lesser accuracy then method involving MCMC, but is easier in implementation and use in practical examples, especially because of lesser computation performance demand [1].