## 2 Materials and Methods
### Experimental Protocol  
The same protocol used in the original study [40] protocol was replicated here. However, a distinct ML classifier type was used in the present study to support its goals. Like in [40], a cost-sensitive approach was used in training to compensate for the harmful effect of the unbalanced dataset in generating the high FN ratios. Therefore, distinct cost values were assigned as a penalty to the FNs to reduce the ML model bias towards the most represented class (non-defective) without changing the cost of false positives (FP), which creates relative costs (RC) effect between the FN and FP penalty values. That aims to understand the effect of assigning different cost values to the quality of the final model predictions and lower the FN rate. However, a larger range of RC values assigned to FN ([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 20, 30, 40, 50, 100]) was experimented to broaden the investigation of the effects of the cost-sensitive training in the present domain. Since, as in [76,77], distinct, and a larger number of ML classifiers (RF) models were induced when compared to [40]. The dataset composition and research protocol are depicted in Figure 2.  
As in the benchmark study [40], for each distinct cost value assigned for the FN, distinct ML classifiers (RF) were induced using a 10 \(\times\) 10-fold cross-validation strategy [78]. The 10 \(\times\) 10-fold cross-validation supports a more reliable validation of the proposed technique. Among the arguments by [79], the 10-fold was used rather than the leave-one-out cross-validation because it yields better results for the size of the dataset and results in less variance, which helps to compare the performance of distinct ML models induced as the FN assigned cost is increased. It smooths out the extreme effects of the luckiest and unluckiest data selection for training and testing, which leads to more realistic conclusions. Moreover, compared to vanilla train/test dataset split strategies, it reduces problems like underfitting and overfitting and helps to estimate better how accurately the model will perform in practice. For all the executions, a unitary cost (1) was assigned to true positive (TP), true negative (TN), and FP. That is, the default penalty value (=1) was kept for TP, TN, and FP to train all the ML models in this study. Moreover, the default FN penalty value (=1) was also maintained for the experiments investigating RC = 1 (baseline). However, for all other RC values, the FN cost (penalty value) was set equal to the corresponding investigated RC value. Thus, in the setup for the experiment to investigate RC = 2, FN cost was set to 2. In the setup for the experiment to investigate RC = 3, FN cost was set to 3, and so on, until FN was set to 100 for RC = 100. Since 20 values of RC were investigated, 20 executions of 10 \(\times\) 10-fold cross-validation were executed in total, as illustrated in Figure 2. Therefore, for each value of RC explored, 10 repetitions of 10-fold cross-validation were performed,  
Figure 2: Protocol used to create distinct models to predict defective software modules using different RC values to support decision-making. resulting in 100 ML models trained and validated for each distinct cost assigned to FN. Finally, for each one of the 20 executions, the average value of the evaluation metrics (detailed in the next subsection) from the 100 RF was computed and further analyzed (as presented in Section 3). Since 20 distinct FN cost values were used, a total of 2000 RFs were generated considering all the executions. Following the 10-fold cross-validation protocol, for each execution, the NASA MDP CROSS-PROJECTS DATASET was split into 10 equal-size subsets. Then, all 10 possible combinations of 9 subsets were used to train the RF, and each subset left aside per combination was used to evaluate the performance of the ML model. Since a 10 \(\times\) 10-fold cross-validation was used, that process was repeated 10 times per value of RC evaluated.