## 3 Results and Discussion
### Tree-Based Algorithms: Random Forest and Gradient-Boosted Tree Models  
#### 3.3.1 Random Forests  
To construct the RF models, the dataset was also partitioned into training and test sets with a ratio of 75:25. This randomised process was repeated 10 times. Each of the random training datasets was then used to create an RF model, using bootstrap samples to build the trees. A specific percentage of samples was drawn from the training datasets to \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Number of Neurons, \(\alpha\)** & **Measures** & **Mean of Test Sets** & **With Validation (Mean of 10 Models)** \\ \hline \multirow{3}{*}{\(8,\alpha=0.01\)} & \(R^{2}\) & 0.8014 & 0.8043 \\  & MAE & 3.0467 & 3.0404 \\  & MAPE & 16.7511 & 16.6353 \\  & RMSE & 3.9617 & 3.9628 \\ \hline \multirow{3}{*}{\(30,\alpha=0.1\)} & \(R^{2}\) & 0.8045 & 0.8290 \\  & MAE & 2.9960 & 2.8415 \\  & MAPE & 16.3343 & 15.4097 \\  & RMSE & 3.9328 & 3.7050 \\ \hline \multirow{3}{*}{\(40,\alpha=0.1\)} & \(R^{2}\) & 0.8032 & 0.8317 \\  & MAE & 2.9536 & 2.7956 \\ \cline{1-1}  & MAPE & 15.9433 & 15.0795 \\ \cline{1-1}  & RMSE & 3.9437 & 3.6765 \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Statistical measures for MLP networks; ReLU activation function, L-BFGS solver. train each base estimator. Once a random forest model had been obtained, its predictive accuracy and generalisability were analysed using the corresponding test dataset. Repeated \(k\)-fold cross-validation was also applied, and no appreciable differences between the two cross-validation procedures were found. To explore the influence of the number of samples used to train the base estimator, different percentages for the training set were considered. The results showed similar performance for percentages greater than or equal to 75% (see Figure 11). In addition, each RF model was created with varying numbers of trees, from two to 1000. No major differences were found when a number of trees greater than or equal to 10 was used (see Figure 12a). Moreover, as expected, no overfitting was detected as the number of trees increased. In the rest of the results discussed in this section, we use 100 trees to estimate each RF, as this number of trees gave optimal RF models (see Figure 12b). The splitting criteria used were the MSE and the MAE. Both gave the same performance. To detect overfitting, we varied not only the maximum depth of the trees but also the minimum number of cases needed for parent and child nodes. Figure 13 illustrates the behaviour of the RMSE when the maximum depth of the trees was varied. In Figure 13a, the nodes could be expanded until all leaves are pure, while in Figure 13b, a minimum of 20 cases for parent nodes and 10 for child nodes are required to augment the branches. As can be seen from these figures, good results were obtained with a maximum depth of five. Figure 14 compares the RMSE for the training and test datasets of the 10 models with varying values of the minimum number of cases for parent and child nodes needed to increase the branches of the tree. Table 8 summarises several goodness-of-fit measures for the models shown in Figure 14. Acceptable results were obtained from pre-pruning the trees, where a minimum of 20 samples was required to split an internal node and 10 samples in the leaf or terminal nodes (model (c)). This model achieved a mean RMSE of 3.8434 and a mean MAE of 2.9809 for the test data. Although models (a) and (b) in Table 8 seem a priori to outperform model (c), decreasing the required cases for parent and child nodes could lead to overfitting; see Figure 14a with a minimum of two cases for internal nodes and one for terminal nodes and Figure 14b with a minimum of 10 cases for internal nodes and  
Figure 10: Relative error of the MLP networks for varying numbers of neurons in the hidden layer; number of iterations \(=2000\), ReLU activation function, L-BFGS solver. (**a**) \(\alpha=0\) (without L2 regularisation); (**b**) \(\alpha=0.01\); (**c**) \(\alpha=0.1\); (**d**) \(\alpha=2\). five for terminal nodes.