## 2 Materials and Methods
### Experiments  
Experiments were conducted using Python 3.7 with the python-weka-wrapper3 package version 0.1.7 running on the MacOS 10.15 operating system. The scripts can be downloaded from Supplementary Material. Figure 1 illustrates the process of the classifier building. Two-layer nested cross-validation was used to build and evaluate the classifiers. In the outer loop, the dataset was split into 10 folds containing a training set and a holdout test set in a 9:1 ratio using stratified random sampling. The process of data splitting was repeated three times by varying the random seed, thus generating 30 training and test set pairs. In the inner loop, the training set was used to build classifiers, for which another 10-fold cross-validation was used to find the optimal hyperparameters (Table 1). The classifiers with the optimal hyperparameters were tested on the holdout test set. Then the evaluation metrics from the 30 training and test set pairs were averaged to estimate the generalization performance of classifiers. This approach ensures that the training, validation, and evaluation data are completely separated.  
Feature selection is a common ML technique. It has the potential of improving training efficiency, result comprehensibility, and prediction performance. Therefore, during the experiments, feature selection was applied to the training sets mainly to filter out redundant and/or irrelevant features from the original data to build a more explainable model. In this study, the correlation-based feature subset selection (CfsSubsetEval module in Weka) with the BestFirst search method was used to perform the feature selection procedure. A correlation-based feature selection method was used to evaluate the correlations between feature subsets and the dependent variable. The optimal feature subset contains features that are highly correlated with the dependent variable, but uncorrelated with each other [28]. The best first search strategy was used to find an optimal feature subset from the feature space. This approach does not specify a threshold to determine the most important features. Instead, the search terminates when the limit of the number of fully expanded subsets that result in no improvement is reached [28].  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Techniques** & **Hyperparameters** & **Range** & **Increment** \\ \hline NB & NA & NA & NA \\ LR & Ridge value & Default & NA \\ RF & Number of trees & 10–200 & 10 \\ \(k\)NN & Number of neighbors & 1–20 & 1 \\ SVM & Complexity & 0.5–1.0 & 0.5 \\ Kernel & PolyKernel, RBFKernel & NA \\ C4.5 & Confidence factor & 0.20–0.45 & 0.05 \\  & Minimum of number of instances per leaf & 2–20 & 1 \\ CART & Minimum number of instances per leaf & 2–20 & 1 \\ MLP & Learning rate & 0.1–0.5 & 0.2 \\  & Momentum & 0.1–0.3 & 0.1 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Machine learning (ML) techniques and hyperparameter tuning. NA = not applicable.  
Figure 1: The process of classifier building. In the outer loop, 10-fold cross-validation (CV) was used to estimate the performance of the classifiers. In the inner loop, another 10-fold CV was used to find the optimal hyperparameters.