## 3 Models' Development and Results
### Bagging and Boosting Models  
This study developed six models, including standard ANN, standard DT (CART), bagged ANN, boosted ANN, bagged DT (random forest), and boosted DT (extreme gradient boosting tree). These models are developed using the transformed and selected inputs in the previous steps. For developing these models, a 10-fold cross-validation technique was employed to avoid overfitting. In k-fold cross-validation, smaller collections are created by dividing the training dataset into k folds. Then, the training data, including k-1 folds, are trained. The unused part of the data is employed to validate the resulting model. Next, the mean of the values calculated in the circle is adopted to report the k-fold cross-validation performance. Figure 5 shows the process of a 10-fold cross-validation used in this study. The team behind this research employed various hyperparameters controlling the learning process to develop these models. They also used a grid-search strategy to optimize these parameters. The process of a grid-search in this study is shown in Figure 6. As previously mentioned, the single DT developed in this study is CART. Its optimized hyperparameters included levels under root = 5; highest replacements = 5; lowest variation in impurity = 0.0; lowest records in parent branch = 2%; and smallest records in child branch = 1%. Concerning XGBT model, the optimized hyperparameters were: boost rounds = 10; maximum tree depth = 6; minimum child weight = 1.0; objective = reg:linear; sub sample = 1.0; Eta = 0.3; Colsample by tree = 1.0; Colsample be level = and 1.0; Lambda. For RF, the best hyperparameters comprised: number of trees to build = 10 and minimum leaf node size = 1. Concerning the standard ANN model, the hyperparameters were: NN model = MLP; hidden layer 1 = 1; hidden layer 2: 0; and overfit prevention set = 30%. For boosted and bagged models: number of component models for boosting = 10 and combining rule = mean. Once the data were prepared and models' hyperparameters were tuned, the models were executed. Figure 7 shows a comparison between the actual and predicted values of \"E\" by the models developed in this study. The performance of the models was evaluated using two well-known criteria, including linear correlation (R) and mean absolute error (MAE). The calculation formulas of these criteria are shown by Equations (1) and (2). \[R=\frac{\sum_{i=1}^{n}(e_{i}-\overline{e}_{i})(m_{i}-\overline{m}_{i})}{\sqrt {\sum_{i=1}^{n}(e_{i}-\overline{e}_{i})^{2}(m_{i}-\overline{m}_{i})^{2}}} \tag{1}\]  
Figure 5: Tenfold cross-validation process. \[MAE=\frac{\sum_{i=1}^{n}\lvert e_{i}-m_{i}\rvert}{n} \tag{2}\]  
where, \(e_{i}\) and \(m_{i}\) denote \(n\)th real and predicted values, respectively; \(\tilde{e}_{i}\) and \(\overline{m}_{i}\) signify the average values of real and predicted values, respectively; and \(n\) stands for the number of samples in the dataset. Tables 4 and 5 show the performance of each model developed in this study. Concerning standard models, the ANN possessed a better R and MAE than the CART model. RF outperformed its peer, ANN, when the bagged models are concerned. For boosted models, XGBT had a better performance than the boosted ANN. Among all models developed in this study, boosted models outperformed the bagged and standard models. Finally, the boosted DT, which, in fact, was an XGBT, outperformed other models developed in this study. \begin{table}
\begin{tabular}{l c c c c} \hline \hline  & \multicolumn{2}{c}{**DT**} & \multicolumn{2}{c}{**ANN**} \\ \hline  & **R** & **MAE** & **R** & **MAE** \\ \hline Standard model & 0.476 & 21.111 & 0.566 & 18.928 \\ Bagged model & 0.91 & 9.726 & 0.849 & 12.551 \\ Boosted model & 0.994 & 3.836 & 0.972 & 5.343 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Models’ performance. Figure 6: Grid-search process in this study. In addition to R and MAE, a lift chart also was used for models' evaluation. The lift chart is from the family of cumulative charts.