## 4 Results and Discussion
### Hyperparameter Tuning of Machine Learning Models  
The purpose of hyperparameter tuning is two-fold: (i) to determine the optimum values of hyperparameters of ML models for the problem considered and (ii) to ensure that the ML models do not overfit or underfit the training data. The key hyperparameters of ML models are optimized by minimizing the testing MAPE obtained using repeated five-fold cross validation tests on the training dataset. Figure 10 presents the results of hyperparameter tuning of ML models in the form of average testing of MAPE versus the variation of corresponding major hyperparameters of the models. Note that each MAPE value in Figure 10 is the average of 15 different MAPE values resulting from repeated five-fold cross validation tests.  
Results presented in Figure 10a show that the average testing MAPE of the KNN model first decreases as the number of nearest neighbors (k) increases, indicating an increase in accuracy. However, when k increases further (k > 3), the accuracy of the model decreases. This indicates that the critical value of k is 3, in order to avoid overfitting (k < 3) or underfitting (k > 3) the training data. Based on this observation, the optimum value for k in the KNN algorithm is chosen to be 3. Similar to the KNN model, the average testing MAPE of the SVR model decreases as the penalty parameter C increases (Figure 10b), indicating that relatively smaller values of C would underfit the training data. Though it is not very apparent from  
Figure 9: Results of feature importance scores based on the prediction of AAR obtained from three DTR-based ML models (RFR, ABR and GBR).  
Figure 10c, relatively larger values of C would overfit the training data. Based on the results obtained and to be consistent with the previously developed ML models related to this topic (performance prediction of rocking foundations), the optimum value for C is chosen to be 1.0.  
It is well known that deep DTR models tend to overfit the training data, while shallow DTR models tend to underfit the training data [39]. Based on the results shown in Figure 10c and to be consistent with the previously developed DTR-based models related to this topic, the optimum value for the maximum depth is set at 6 for base DTR models in the ensembles. Figure 10d shows that, for all three DTR-based ensemble models (RFR, ABR and GBR), the accuracy of the models increases as the number of trees increases (this is more apparent for the GBR model). The number of random features (maximum) to be considered is kept at 4 for the RFR model, and the learning rate is kept at 0.1 for both boosting models (ABR and GBR). When the number of trees in the ensembles increases beyond 100, the average testing MAPE of the models does not decrease any further. This indicates that the minimum number of trees required in DTR-based ensemble models is 100. This is remarkably consistent for all three DTR-based ensemble models. Table 1 summarizes the key hyperparameters chosen for five nonlinear ML models developed in this study.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Machine Learning Model** & **Hyperparameter** \\ \hline k-nearest neighbors regression (KNN) & k = 3 \\ Support vector regression (SVR) & weight = inverse distance \\  & C = 1.0 \\ epsilon = 0.1 & epsilon = 0.1 \\  & mapping function = RBF 1 \\ Random forest regression (RFR) & max. depth = 6 \\  & max. features = 4 \\  & number of trees = 100 \\ Boosting models (ABR and GBR) & max. depth = 6 \\  & learning rate = 0.1 \\  & number of trees = 100 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Optimum values chosen for major hyperparameters of machine learning models.  
Figure 10: Variation of the average MAPE of machine learning models with their major hyperparameters from k-fold cross validation tests of models using training data for (**a**) KNN, (**b**) SVR, (**c**) DTR and (**d**) DTR-based ensemble models.