## 3 Results and Discussion
### Tree-Based Algorithms: Random Forest and Gradient-Boosted Tree Models
#### 3.3.2 Gradient-Boosted Tree-Based Models
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Model** & **Measures** & **Mean of Test Sets** & **With Validation (Mean of 10 Models)** \\ \hline (a) & \(R^{2}\) & 0.8185 & 0.8762 \\ \(p=0.75\) & MAE & 2.9183 & 2.4472 \\  & MAPE & 16.2005 & 13.6766 \\  & RMSE & 3.7921 & 3.1528 \\ \hline (b) & \(R^{2}\) & 0.8204 & 0.8646 \\ \(p=0.75\) & MAE & 2.8988 & 2.5283 \\  & MAPE & 16.1394 & 14.1537 \\  & RMSE & 3.7725 & 3.2979 \\ \hline (a) & \(R^{2}\) & 0.8182 & 0.8765 \\ \(p=0.9\) & MAE & 2.9209 & 2.4396 \\  & MAPE & 16.2251 & 13.6751 \\  & RMSE & 3.7959 & 3.1500 \\ \hline (b) & \(R^{2}\) & 0.8201 & 0.8590 \\ \(p=0.9\) & MAE & 2.9032 & 2.5825 \\  & MAPE & 16.2201 & 14.4576 \\  & RMSE & 3.7747 & 3.3650 \\ \hline (a) & \(R^{2}\) & 0.8178 & 0.8732 \\ \(p=1.0\) & MAE & 2.9338 & 2.4690 \\  & MAPE & 16.2570 & 13.8063 \\  & RMSE & 3.7990 & 3.1907 \\ \hline (b) & \(R^{2}\) & 0.8175 & 0.8632 \\ \(p=1.0\) & MAE & 2.9318 & 2.5466 \\  & MAPE & 16.3024 & 14.2430 \\  & RMSE & 3.8021 & 3.3145 \\ \hline \hline \end{tabular}
\end{table} 
Table 10: Statistical measures for gradient-boosted tree models; learning rate \(=\) 0.05, number of estimators \(=\) 130, maximum depth \(=\) 3, \(p=\) proportion of samples for training each base learner; (a) minimum of two cases for parent nodes and one for child nodes, (b) minimum of 20 cases for parent nodes and 10 for child nodes. for a learning rate of 0.05, both models gave good results for numbers of base estimators of between 90 and 130. However, XGBoost produced more underfitting for small numbers of trees. When setting the best configurations of hyperparameters, we found that XGBoost slightly outperformed the previous gradient-boosted tree model. For example, with 130 weak learners, a learning rate of 0.05 and 75% of samples used to train each base estimator, XGBoost achieved a mean RMSE of 3.1402, a mean MAE of 2.4375 and a mean MAPE of 13.6244%, i.e., better results than those shown in Table 10 (see model (a) in Tables 10 and 11). To analyse the prediction accuracy of XGBoost depending on the type of regularisation applied, the parameters used to control the L1 and L2 regularisation (\(\alpha\) and \(\lambda\), respectively) were varied in the set \(\{0,0.0001,0.001,0.01,0.1,0.2\}\) and the dropout rate was varied in the set \(\{0,0.01,0.05,0.1\}\). The best results were obtained for \(\alpha=0\), \(\lambda=0.001\), and a dropout rate of zero (see Figures 19 and 20). The prediction accuracy of these models was comparable to the results obtained without these types of regularisation. More specifically, under the above conditions, XGBoost achieved a mean RMSE of 3.1396, a mean MAE of 2.4371 and a mean MAPE of 13.6238%. For the test datasets, the mean values were 3.7812, 2.9020 and 16.0921%, respectively (see Table 11). Note that both the gradient-boosted tree models and the extreme gradient-boosted tree models outperformed RF. Figure 17: Gradient-boosted tree models with varying pruning parameters; learning rate \(=0.05\), proportion of samples for training each base learner \(=1\). (**a**) RMSE. Minimum of two cases for parent nodes and one for child nodes; (**b**) RMSE. Minimum of 20 cases for parent nodes and 10 for child nodes.