## 4 Empirical Study
### Ensemble Models Construction and Validation
Given the fact that non-defective cases make up the vast majority of software defect datasets, we did not employ any sampling techniques (e.g., SMOTE) to balance the dataset. We employed a stratified 10-fold cross-validation method [49], repeated 10 times, to validate the performances of ensemble models. Each dataset was divided into k equal-sized folds, with one-fold serving as a testing dataset for the model's evaluation and the remaining k-1 folds being used for training the model. This technique was performed ten times (i.e., k = 10), with each fold serving as a testing dataset precisely once. To get a final estimate, the detection performance from the k iterations was averaged. Each fold in stratified cross-validation has a proportion of instances belonging to each class that is roughly equal to the proportion in the original dataset. The key benefit of this method is that it uses all instances for both model training and testing. The whole process was repeated 10 times, resulting in 100 independent runs for each investigated model. Finally, the reported model performance was the average of the 100 runs. Repeating the 10 fold cross validation ten times provides a model performance estimate with little bias and low variance [42,50]. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Ensemble** & **Hyperparameter Name** & **Description** & **Optimized Value Range** \\ \hline \multirow{4}{*}{Random Forest (RF) Extra Trees (ET)} & n\_estimators & Number of trees in the forest. & [100, 50, 40, 30] \\  & max\_depth & Maximum depth of the tree. & [1, 4, 8] \\  & min\_samples\_leaf & Minimum number of samples at a leaf node. &
\begin{tabular}{c} RF = [1, 10, 5] \\ ET = [1, 2, 4] \\ \end{tabular} \\  & criterion & Measure the quality of a split. & [gini, entropy] \\ \hline \multirow{2}{*}{AdaBoost (Ada)} & n\_estimators & Number of estimators at which boosting is terminated. & [50, 100, 1000] \\  & learning\_rate & The step size of the loss function. & [1, 0.1, 0.001, 0.01] \\ \hline \multirow{4}{*}{Gradient Boosting (GB)} & n\_estimators & Number of estimators at which boosting is terminated. & [100, 50, 500, 1000] \\  & learning\_rate & The step size of the loss function. & [0, 1, 0.001, 0.01] \\  & min\_samples\_leaf & Minimum number of samples at a leaf node. & [1, 10, 5] \\  & max\_depth & Maximum depth of individual tree. & [3, 7, 9] \\  & loss & The loss function to be optimized in the boosting process. & [deviance, exponential] \\ \hline \multirow{4}{*}{Hist Gradient Boosting (HGB)} & max\_iter & Number of iterations of the boosting process. & [100, 50, 1000] \\  & learning\_rate & The step size of the loss function. & [0, 1, 0.001, 0.01] \\  & min\_samples\_leaf & Minimum number of samples at a leaf node. & [20, 10, 5] \\  & max\_depth & Maximum depth of individual tree. & [None, 1, 3, 5] \\  & loss & The metric to use in the boosting process. & [auto, binary crossentropy, categorical crossentropy] \\ \hline \multirow{4}{*}{XGBoost (XGB)} & n\_estimators & Number of gradient boosted trees. & [100, 50, 1000] \\  & learning\_rate & The step size of the loss function. & [0.3, 0.1, 0.001, 0.01] \\  & max\_depth & Maximum tree depth for base learners. & [6, 3, 4, 5] \\ \hline \multirow{4}{*}{CatBoost (CAT)} & n\_estimators & Number of estimators at which boosting is terminated. & [100, 100, 50, 500] \\  & loss\_function & The metric to use in the boosting process. & [Logloss, MultiClass] \\  & learning\_rate & The step size of the loss function. & [0.03, 0.001, 0.01, 0.1] \\ \cline{1-1}  & depth & Depth of the tree. & [1, 5, 10] \\ \cline{1-1}  & min\_data\_in\_leaf & Minimum number of samples at a leaf node. & [1, 10] \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Hyperparameters space investigated in this empirical study.