## 3 Results and Discussion
### Tree-Based Algorithms: Random Forest and Gradient-Boosted Tree Models
#### 3.3.2 Gradient-Boosted Tree-Based Models  
The RF regressors studied in Section 3.3.1 are ensemble methods from the family of averaging methods. By contrast, in boosting methods, the base estimators are built sequentially, such that each decision tree attempts to reduce the error of the previous tree. In this section, we explore the use of two well-known boosting ensemble methods: GBR and extreme GBR (or XGBoost). To obtain gradient-boosted tree models using GBR, both the squared error and the absolute error were used as loss functions. In addition, the splitting criteria were the MSE and the MSE with [PERSON]'s improvement score. When the absolute error loss function was used, the best splitting criterion was the Friedman MSE; however, the best results were obtained with the squared error loss function, which gave the same performance with both splitting criteria. Gradient-boosted tree models were obtained with varying numbers of boosting stages, from 10 to 1000. The maximum depth of the individual regression estimators was chosen as between one and five, and the best results were obtained with a value of three. For this maximum depth, Figure 15 illustrates the behaviour of the RMSE and MAE as the number of boosting iterations (added trees) increases. The learning rate was set to 0.02 in Figure 15a,b and to 0.05 in Figure 15c,d. The learning rate and the  
Figure 13: Behaviour of RF models for varying maximum depths; proportion of samples for training each base estimator = 0.75. (**a**) RMSE, minimum of two cases for parent nodes and one for child nodes; (**b**) RMSE, minimum of 20 cases for parent nodes and 10 for child nodes. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Model** & **St. Measures** & **Median** & **Mean** & **Confidence Interval (95\%)** \\ \hline (a) & MAE & 2.9328 & 2.9530 & [2.8378, 3.0682] \\  & MAPE & 16.2147 & 16.2340 & [15.4197, 17.0483] \\  & RMSE & 3.8316 & 3.8239 & [3.6988, 3.9490  
number of estimators are two critical hyperparameters for this method [73]. These hyperparameters interact with each other in such a way that lower values of the learning rate require larger numbers of estimators to maintain similar RMSE and MAE values [55]. For a learning rate of 0.02, good results were obtained with 250 weak learners or trees (see Figure 15a,b). With this learning rate, a higher number of trees could lead to overfitting the model. A slight improvement was obtained with a learning rate of 0.05 and 130 weak learners. In this case, clear overfitting occurs with more than 200 added trees (see Figure 15c,d). In contrast, a small number of weak learners or a lower value of the learning rate could lead to underfitting of the model (see Figures 15 and 16). In addition, as the number of trees increases, the values of the learning rate that produce strong underfitting decrease. For example, with 250 weak learners, a learning rate of 0.005 or less caused underfitting of the models for our problem (see Figure 16c,d). With 130 weak learners, strong underfitting occurred for values less than or equal to 0.01 (see Figure 16a,b), while a learning rate greater than 0.1 produced overfitting in both cases. Figure 14: Behaviour of RF models for varying numbers of minimum cases for parent and child nodes; maximum depth = 5, percentage of samples for training each base estimator = 0.75. (**a**) Minimum of two cases for parent nodes and one for child nodes; (**b**) minimum of 10 cases for parent nodes and five for child nodes; (**c**) minimum of 20 cases for parent nodes and 10 for child nodes; (**d**) minimum of 40 cases for parent nodes and 20 for child nodes; (**e**) minimum of 50 cases for parent nodes and 25 for child nodes. From the above figures, we see that the best results were obtained with 130 boosting iterations (weak learners or trees) and a learning rate of between 0.04 and 0.06; a mean RMSE of 3.799, a mean MAE of 2.9338 and a mean MAPE of 16.2570% were achieved for the test datasets with a learning rate of 0.05.