# Prediction of Flotation Deinking Performance: A Comparative Analysis of Machine Learning Techniques
## 2 Materials and Methods
### Machine Learning Techniques for Prediction of Flotation Deinking Performance
#### 2.1.2 Regression Tree Ensembles  
Linear regression represents a global model, where a single formula describes the relations between the inputs and the outputs of the model over the entire data space. When there are many features interacting in nonlinear ways, it is very hard to design a single global model. An alternative approach is to divide the data space into smaller partitions, where the modelling of these interactions is easier to achieve. These partitions can be further divided into even smaller regions, until finally one obtains the data space cells where simple models can be applied. This is called recursive partitioning [21]. Regression trees use the tree to represent the recursive partition. It splits the input data space in partitions and assigns a prediction value to each partition. The terminal nodes of the tree, denoted as leaves, represent these partition cells. In order to determine to which leaf the input data belong to and to assign it the prediction value, the algorithm starts from the root node and asks successive binary questions. Depending on the outcome of the question, the sub-branch of the tree is chosen. Eventually, the algorithm arrives at the leaf node, where the prediction is made. This prediction is found as an average of all training data instances which reach that leaf node [21]. Suppose a training dataset \(D=\{(x_{i},y_{i}),i=1,\ldots,N\}\) which consists of N training pairs, where \(x_{i}\in R^{n}\) is the n-dimensional vector denoting model's inputs and \(y_{i}\in R\) are the observed responses to these inputs (model's outputs). Suppose further a division of the input data space into \(M\) partitions \(R_{i}\); \(i=1,2,\ldots,M\), where the response is modelled as a constant \(c_{i}\) in each partition:  
\[f(x)=\sum_{i=1}^{M}c_{i}I\{x\in R_{i}\} \tag{7}\]  
where \(I\{x\in R_{i}\}\) is a binary function that takes the value 0 or 1 depending on the outcome of the question at the tree's split point [21; 22]. The constant \(c_{i}\) can be determined as the average of the responses \(y_{i}\) in the region \(R_{i}\). The greedy algorithm is used in order to determine the split point [21; 23], which is very efficient, but might lead to poor decisions, especially in the lower tree branches, due to unreliable estimates based on the small number of samples. To overcome this issue, more regression trees can be combined in an ensemble, which represents a predictive model composed of a weighted combination of multiple regression trees. Different algorithms exist for ensemble learning, such as bagging, random forests and boosting [21]. Boosting, which is an ensemble technique where the predictors are created sequentially, is used in this paper. The rationale behind this is that each subsequent predictor learns from the mistakes committed by the previous predictors [21; 22]. When gradient boosting is applied to regression tree ensembles, the first regression tree is the one that maximally reduces the loss function for the selected tree structure and the given training dataset. The residual (prediction error) is then calculated, which represents the mistake committed by the predictor model (the first regression tree). At the next step, a new tree is fitted to the residuals of the first tree. At each step, a new tree is added to the model, which is fitted to the residuals of the previous one. The residual values are usually multiplied by the learning rate (value less than 1) to avoid overfitting. The final model obtained by boosting is simply a linear combination of all trees (usually hundreds or thousands of trees). The main idea of boosting is that, instead of using a complex single regression tree, which is easily over fitted, a much better fit is produced if many simple regression trees are trained iteratively, each of them improving the prediction performance of the previous ones [22]. Boosting algorithms play a crucial role in dealing with bias variance trade-off.