## 3 Results and Discussion
### Tree-Based Algorithms: Random Forest and Gradient-Boosted Tree Models
#### 3.3.2 Gradient-Boosted Tree-Based Models
In an attempt to reduce possible overfitting, the minimal cost-complexity pruning algorithm [61] was also used with the GBR algorithm. Figure 17 shows the influence of the value of the pruning parameter. In Figure 17a, a minimum of two cases for parent nodes and one for child nodes were considered, while Figure 17b shows a minimum of 20 cases for parent nodes and 10 for child nodes. In both cases, similar performance was obtained for a pruning parameter of 0.00001 or less, and this was comparable to the performance of the model when no pruning was carried out (i.e., a pruning parameter of zero). However, pruning parameters greater than 0.00001 caused underfitting of the models. Table 10 summarises several goodness-of-fit measures for the best configurations of hyperparameters found. Different proportions of samples were also used to train each base learner. The best gradient-boosted models achieved a mean RMSE of between 3.15 and 3.2979 and outperformed RF regardless of the proportion of samples extracted from the training dataset to train each base estimator (see Tables 9 and 10). The gradient-boosted tree models discussed above were created using the Scikit-learn API. With the aim of optimising these models, we also used the XGBoost library, an optimised distributed gradient boosting library that allowed us to apply some regularisation techniques such as L1 and L2 regularisation [62] and dropout techniques [63]. The booster parameter sets the type of learner used in the algorithm: the gbtree booster allows for L1 and L2 regularisation, while the dart booster inherits the gbtree booster, but adds dropout techniques, allowing some trees to be dropped to solve the possible problem of overfitting. Both learners gave the same results when no dropout was added, and all instances of the training dataset were employed to train each base learner (\(p=1\)). For \(p<1\), the dart learner slightly outperformed the gbtree learner, although the differences were not significant. Note that XGBoost does not support the hyperparameters related to the minimum number of samples required to split an internal or a leaf node; instead, the min_child_weight parameter can be controlled. This parameter represents the minimum sum of the instance weights needed for a child node. If the tree partitioning step results in a leaf node with a sum of the instance weights of less than min_child_weight, the building process will cease any further partitioning. In linear regression tasks, this simply corresponds to the minimum number of instances required for each node. This parameter was varied from 0 to 20, and the best performance was obtained for values of zero and one, which gave the same results. Figure 18 illustrates the behaviour of XGBoost with varying numbers of trees and a varying percentage of samples used to train each base learner. Neither L1 and L2 regularisation nor dropout techniques are included in these results. Under these conditions, XGBoost behaved analogously to the previous gradient-boosted tree models, in which the nodes could be expanded until all the leaves were pure. That is, both algorithms gave a similar performance as the number of trees increased. More specifically,