## 5 Results and Discussion
### Effects of Hyperparameter Optimization on Tree-Based Ensembles  
In order to find the magnitude of change in the prediction performance, we used the Wilcoxon size effect estimate. Table 5 shows the categories of the impact of hyperparameter optimization on the studied tree-based ensembles using the numbers of ensembles thus affected. RF ensembles were highly influenced by hyperparameter optimization, as it tended to have medium to large impact on 50% of the defect datasets. On the contrary, HGB was slightly affected by hyperparameter optimization; it had negligible impact on more than 50% of the ensembles. On average, the magnitude of change was considered negligible to small in the other tree-based ensembles. The effect size estimates the magnitude of the change; however, it does not determine whether the change is positive or negative. Therefore, we further expanded previous results to show whether hyperparameter optimization increased or decreased the prediction performance. Figure 2 shows the effect size for each tree-based ensemble along with the positive (i.e., in blue) or negative (i.e., in red) impact. Negligible = 0, small = 1, medium = 2 and large = 3. For each ensemble, the total number of positive and negative represent the number of medium and large effect size. Overall, none of the tree-based ensembles showed a positive impact on any datasets. We observed that ET and RF had a negative impact in most of the datasets. Additionally, as shown in the figure, we note that hyperparameter optimization had the lowest impact on HGB, as it had negligible impact across 13 datasets. Clearly, the impact is related to the dataset's characteristics; for example, the prediction performances on Debug, Eclipse 2.1, Eclipse PDE UI, Lucene 2.4 and Prop 1 datasets \begin{table}
\begin{tabular}{l c c c c} \hline
**Classifier** & **Negligible** & **Small** & **Medium** & **Large** \\ \hline Ada & 7 & 7 & 5 & 2 \\ CAT & 2 & \(11\) & 6 & 2 \\ ET & 8 & 4 & 8 & 1 \\ GB & 7 & 6 & 7 & 1 \\ HGB & 13 & 3 & 2 & 3 \\ RF & 9 & 1 & 7 & 4 \\ XGB & 7 & 9 & 2 & 3 \\ \hline \end{tabular}
\end{table} 
Table 5: Wilcoxon effect size for each of the studied tree-based ensembles. deteriorated for all tree-based ensembles after applying hyperparameter optimization. Similarly, on other datasets, including Eclipse 2.0, JM1, Prop 3 and Xalan 2.5, applying hyperparameter optimization increased the prediction performances for all ensembles. Computational cost is another important aspect to be considered when performing hyperparameter optimization. For each tree-based ensemble, we computed the amount of time required to optimize the model for each of the studied datasets. Each hyperparameter combination in the grid space was evaluated by 10 independent runs (i.e., 5 x 2-fold cross-validation) to select the tuned model with the best cross-validation score.The computational cost of applying grid search hyperparameter optimization is presented in Table 6. Clearly, CatBoost required a lot of computational effort compared to the other tree-based ensembles. The bagging tree-based ensembles (i.e., RF and ET) took the least amount of time for hyperparameter optimization. Focusing on boosting ensembles, we observe that long computational time was related to the size of the datasets in most cases. However, for the bagging ensembles, we notice that larger datasets, such as JM1, Prop1 and Prop 5, took the least amount of time.