## 3 Materials and Methods
### Background
#### 3.1.3 Ensemble Learning  
Ensemble learning is a machine learning technique where multiple models are combined to produce a stronger and more accurate model. They can be broadly categorized into two sections: bagging and boosting. Both bagging and boosting use decision trees(DTs) as base learners due to their simplicity. A _decision tree_ is a non-parametric supervised machine learning algorithm used for classification and regression tasks. The DT aims to create a model that predicts the target value by learning from data features. Being able to handle both categorical and numerical features makes it a versatile method for a wide range of predictive tasks. It consists of a root node, interior nodes, and terminal nodes, where the nonterminal nodes are linked to decisions and the terminal nodes represent the final classifications [53; 54]. The bagging (bootstrap aggregating) method is an ensemble learning technique designed to stabilize high-variance weak learners. The main idea is to generate multiple subsets of the training dataset using random sampling with replacement (bootstrap sampling) and to train an independent base model \(f_{b}(x)\) on each subset. The predictions of these individual models are then aggregated [54]. For classification tasks, the final prediction is obtained through majority voting, whereas for regression tasks, it is determined by averaging the outputs, as shown in Equations (1) and (2):  
For regression,  
\[\hat{f}(x)=\frac{1}{B}\sum_{b=1}^{B}f_{b}(x) \tag{1}\]  
For classification,  
\[\hat{f}(x)=\text{mode}\{f_{1}(x),f_{2}(x),\dots,f_{B}(x)\} \tag{2}\]  
A prominent example is the Random Forest. The Random Forest [55] is an ensemble learning method used for classification and regression problems. It operates by building multiple decision trees, each trained with different subsets of data. These trees are later merged to obtain more accurate and stable predictions. Due to inherent randomness and feature selection, it is robust against overfitting and noise. Another example of bagging is extra trees (extremely randomized trees). Extra trees [56] is an extension of the Random Forest algorithm, introducing additional randomness to the process of tree-building. Like the Random Forest, it selects a random subset of features to train each base estimator [57], but with one difference. The Random Forest identifies the optimal split point for each selected feature. However, extra trees randomly selects the split point without evaluating the quality. Furthermore, Random Forest uses bootstrap replicas of the dataset to train each tree, whereas extra trees uses the entire dataset for training [58]. Boosting is a method that aims to build a strong ensemble model by training a series of weak learners in succession. Each new model learns by taking into account the errors of the previous model, thus reducing the prediction errors step by step. This structure increases the accuracy while reducing the model bias [59]. The basic formulation of boosting is seen in Equation (3):  
\[F_{m}(x)=F_{m-1}(x)+\eta\cdot h_{m}(x) \tag{3}\]  
where \(F_{m}(x)\) represents the ensemble model at the \(m\)-th iteration, \(h_{m}(x)\) represents the weak learner trained at the \(m\)-th iteration to address the errors of the previous ensemble, and \(\eta\) represents the learning rate controlling the contribution of each new weak learner. At each step, the new model is trained on negative gradients of the loss function according to the predictions of the existing ensemble model, as shown in Equation (4):  
\[r_{im}=-\left[\frac{\partial L(y_{i},F(x_{i}))}{\partial F(x_{i})}\right]_{F(x )=F_{m-1}(x)} \tag{4}\]  
Here, \(r_{im}\) denotes the pseudo-residual for the \(i\)-th training example at iteration \(m\), indicating both the direction and magnitude of the correction needed. \(L(y_{i},F(x_{i}))\) representsthe loss between the true target value \(y_{i}\) and the model prediction \(F(x_{i})\). Common loss functions include squared error for regression and log-loss for classification.