# Prediction of Flotation Deinking Performance: A Comparative Analysis of Machine Learning Techniques
## 2 Materials and Methods
### Machine Learning Techniques for Prediction of Flotation Deinking Performance
#### 2.1.3 Gaussian Process Regression  
Suppose a training dataset \(D=\{(x_{i},y_{i}),i=1,\ldots,N\}\) which consists of N training pairs, where \(x_{i}\in R^{n}\) is the n-dimensional vector denoting model's inputs and \(y_{i}\in R\)are the observed responses to these inputs (model's outputs). Aggregating the column vectors of inputs in matrix X and the responses in the vector y, a training dataset becomes \(D=(X,y)\). The goal of GPR is to predict the value of response, given the new (unseen) input vector and the training data, i.e., to determine the conditional distribution of the responses given the inputs [24]. Consider a standard linear regression model with Gaussian noise [24]:  
\[y=f(X)+\varepsilon=X^{T}\omega+\varepsilon \tag{8}\]  
where \(X\) is the input matrix, \(\omega\) is the vector of weights, \(y\) is the vector of observed responses and \(f(X)=X^{T}\omega\) is the function value which differs from the observed response \(y\) by error \(\varepsilon\), that follows an independent, identically distributed Gaussian distribution with zero mean and variance \(\sigma^{2}\), i.e., \(\varepsilon\)-N(0,\(\sigma^{2}\)). The weights \(\omega\) and the error variance \(\sigma^{2}\) are estimated from the data [24]. Applying the Bayes' rule, the posterior distribution over the weights can be determined as [24,25]:  
\[p(\omega|X,y)=\frac{p(y|X,\omega)p(\omega)}{p(y|X)} \tag{9}\]  
where \(p(y|X)\) is the normalizing term which is independent of weights \(\omega\) and can be neglected. Assuming a zero mean Gaussian prior of the weights \(p(\omega)\) with the covariance matrix \(\Sigma\), i.e., \(\omega\sim N(0,\Sigma)\), one obtains the posterior distribution \(p(\omega|X,y)\) as a Gaussian with mean \(\overline{\omega}\) and covariance matrix \(A^{-1}\)[24]:  
\[p(\omega|X,y)\sim N\bigg{(}\overline{\omega}=\frac{1}{\sigma^{2}}A^{-1}Xy,A^{ -1}\bigg{)} \tag{10}\]  
where \(A=\sigma^{-1}XX^{T}+\Sigma^{-1}\). To make a prediction for a new, unseen, test input \(x^{*}\), one can average over all possible parameter values, weighted by their posterior probability, i.e., the distribution \(f(x^{*})\) at \(x^{*}\) is again Gaussian, with a mean given by the posterior mean of the weights multiplied by the test input [24,25]:  
\[p(f(x^{*})|x^{*},X,y)\sim N\bigg{(}\frac{1}{\sigma^{2}}{x^{*}}^{T}A^{-1}Xy,{x^ {*}}^{T}A^{-1}x^{*}\bigg{)} \tag{11}\]  
When the linear regression model is not adequate, the input data points \(x_{i}\) can be mapped to a higher-dimensional space using the transform \(\Phi:x_{i}\rightarrow\varphi(x_{i})\). The model is further derived, same as in the linear case, substituting \(x\) everywhere with \(\varphi(x)\). Equation (10) then becomes [24]:  
\[p(f(x^{*})|x^{*},X,y)\sim N\bigg{(}\frac{1}{\sigma^{2}}{\varphi(x^{*})}^{T}A^{ -1}\varphi(X)y,\varphi(x^{*})^{T}A^{-1}\varphi(x^{*})^{T}\bigg{)} \tag{12}\]  
This model represents a GPR model. Hence, a Gaussian process is completely defined by its mean function and covariance function. The choice of an adequate covariance function for a given dataset is very important. In our experiments, we use different covariance functions, such as exponential, squared exponential, Matern and rational quadratic. Each of these covariance functions depend on the hyperparameters whose values also need to be tuned. For some of the covariance functions, hyperparameters are easy to interpret and can be used to also combine learning with automatic feature selection, i.e., to determine which inputs (features) are relevant and to exclude all the irrelevant ones from the learning process. For example, consider the covariance function [24]:  
\[K(x_{i},x_{j})=\beta exp\Bigg{[}-\frac{1}{2}\sum_{n=1}^{N}\bigg{(}\frac{x_{i} ^{n}-x_{j}^{n}}{r_{n}}\bigg{)}^{2}\Bigg{]} \tag{13}\]where \(r_{n}\) denotes the length-scale of the covariance function along the input dimension \(n\). It is obvious that if \(r_{n}\) is very large, the covariance function becomes independent of the \(n\)-th input; therefore, it can be considered irrelevant and can be removed from the inference.