# Gaussian Processes for Signal Processing and Representation in Control Engineering
## 2 Gaussian Process  
[PERSON] [1] came up with the following definition, which captures the essence of Gaussian Process:  
A Gaussian process is a collection of random variables, any Gaussian process  
finite number of which have a joint Gaussian distribution. If we define mean function \(m(x)\) and the covariance function mean function \(k(x,x^{\prime})\) of a real process \(f(x)\) as:  
\[m(x)=E[f(x)] \tag{1}\]  
\[k(x,x^{\prime})=E[f(x)-m(x))(f(x^{\prime})-m(x^{\prime}))]. \tag{2}\]  
Then we can define GP as:  
\[f(x)\sim GP(m(x),k(x,x^{\prime})). \tag{3}\]  
The mean and covariance functions completely characterize the GP prior, which controls the full behavior of function \(f\) prior. If we assume that \(\mathbf{f}=\{f(\mathbf{x}_{n})\}_{n=1}^{N}\) then the prior distribution of \(\mathbf{f}\) is a multivariate Gaussian distribution \(\mathbf{f}\sim\text{Normal}(\mathbf{\mu},\mathbf{K})\), where mean is \(\mathbf{\mu}=\{\mu(\mathbf{x}_{n})\}_{n=1}^{N}\), and \(\mathbf{K}\) is covariance matrix, where \(K_{i,j}=k(\mathbf{x}_{i},\mathbf{x}_{j})\) The joint distribution of \(f\) and a new \(\tilde{f}\) is also a multivariate Gaussian with assumption of mean equal 0 is presented by the Formula (4):  
\[p(\mathbf{f},\tilde{f})=\text{Normal}\Bigg{(}\left[ \begin{array}{c}\mathbf{f}\\ f^{*}\end{array} \right]\Bigg{|}\mathbf{0}, \begin{bmatrix}\mathbf{K}_{f,\mathbf{f}}&\mathbf{k }_{f,\tilde{f}}\\ \mathbf{k}_{f,\tilde{f}}&\mathbf{k}_{f,\tilde{f}}\end{bmatrix} \Bigg{)} \tag{4}\]where \(k_{f,f}\) is the covariance between \(f\) and \(\vec{f}\), and \(k_{f},\vec{f}\) is the prior variance of \(\vec{f}\)[9]. On such a prior distribution, we then make an inference from the GP model, by adding the data from the process we would like to model. In general, GP is a stochastic process used for modeling data, which were observed over time, space or both [10]. Main thing that can characterise GP is that is a kind of generalization of normal probability distributions, where each of them describes a random variable (scalar or vector if we deal with multivariate distribution). This kind of generalization can be defined as follows: let us assume a set of index \(T\), then GP on set \(T\) is a collection of random variables indexed by a continuous variable, that is:  
\[\{f(t):t\in\mathcal{T}\}, \tag{5}\]  
which has a property that for  
\[\forall n\in N,\forall t_{1},\ldots,t_{n}\in\mathcal{T}, \tag{6}\]  
the marginal distribution of any finite subset of random variables  
\[\{f(t_{1}),f(t_{2}),\ldots,f(t_{N})\}, \tag{7}\]  
is a multivariate Gaussian distribution [9; 11]. GP can be fully determined by only declaring mean and covariance functions [1; 10; 12]. Mean in most cases is set to value \"0\", because such a setting can be useful, simplifies matters and is not a difficult requirement to fulfill. Of course, there are examples where we would like to change the mean e.g., for better model interpretability or the specification of our prior [1; 12]. Covariance function (also called kernel function) represents a similarity between data points [13]. Note that usually covariance is chosen from the set of already defined functions. One should at least pick one, which represents the prior beliefs of the problem. However, in fact, covariance function can be any function, which has the property of generating a positive definite covariance matrix [14]. Anyway, creating and defining new covariance functions, which will simultaneously be correct and have a practical usage, can be really difficult. The most basic and common kernel function is the Radial Basis Function (RBF), which is also called Squared Exponential (SE) and is defined with the formula (8):  
\[k\big{(}x_{i},x_{j}\big{)}=\exp\left(-\frac{d\big{(}x_{i},x_{j}\big{)}^{2}}{2 l^{2}}\right). \tag{8}\]  
Its main property is that its value is usually only dependent on the distance from the specified point. The parameter which RBF kernel uses is \(l\) as a characteristic length scale and \(d\big{(}x_{i},x_{j}\big{)}\) is the Euclidean distance between points.