## 4 Experimental Setup
### Performance Metrics  
Table 5 shows the confusion matrix utilized to define these performance evaluation indicators.  
For performance evaluation, balance, probability of detection (PD), and probability of false alarm (PF), which are widely used, were employed for the performance evaluation. The balance is a good indicator when considering class imbalance [37].  
* Probability of detection (PD) is defined as percentage of defective modules that are classified correctly. PD is defined as follows: PD = TP/(TP + FN)
* Probability of false alarm (PF) is defined as proportion of non-defective modules misclassified within the non-defect class. PF is defined as PF = FP/(FP + TN)  
For analyzing cost effectiveness, we defined new performance metric, commit inspection reduction (CIR), derived from file inspection reduction (FIR) and line inspection  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Group** & **Metric** & **Definition** \\ \hline \multirow{3}{*}{Size} & LA & Lines of code added \\  & LD & Lines of code deleted \\  & LT & Lines of code in a file before the changes \\ \hline \multirow{3}{*}{Diffusion} & NS & Number of modified subsystems \\  & ND & Number of modified directories \\  & NF & Number of modified files \\  & Entropy & Distribution of modified code across each file \\ \hline Purpose & FIX & Whether the change is a defect fix or not \\ \hline \multirow{3}{*}{History} & NDEV & Number of developers that changed the modified files \\  & AGE & Average time interval between the last and the current change \\  & NUC & Number of unique changes to the modified files \\ \hline \multirow{3}{*}{Experience} & EXP & Developer experience \\  & REXP & Recent developer experience \\  & SEXP & Developer experience on a subsystem \\ \hline \hline \end{tabular}
\end{table}
Table 4: Fourteen change-level metrics [7].  
\begin{table}
\begin{tabular}{c c c c} \hline \hline  & **Confusion** & \multicolumn{2}{c}{**Predicted Class**} \\  & **Matrix** & \multicolumn{2}{c}{**Buggy (P)**} & **Clean (N)** \\ \hline \multirow{2}{*}{**Actual**} & **Buggy** & True Positive (TP) & False Negative (FN) \\  & **Clean** & False Positive (FP) & True Negative (TN) \\ \hline \hline \end{tabular}
\end{table}
Table 5: Confusion matrix.  
reduction (LIR) [37; 38]. CIR is defined as how much effort reduce the effort for code inspection in commit-level compared to random selection inspection.
* The commit inspection reduction (CIR) is defined as the ratio of reduced lines of code to inspect using the proposed model compared to a random selection to gain predicted defectiveness (PrD): CIR = (PrD \(-\) CD)/PrD, where CI and PrD are defined below.
* The commit inspection (CI) ratio is defined as the ratio of lines of code to inspect to the total lines of code for the reported defects. Lines of code (LOC) in the commits that were true positives is defined as TP\({}_{\text{LOC}}\) and similarly with TN\({}_{\text{LOC}}\), FP\({}_{\text{LOC}}\), and FN\({}_{\text{LOC}}\). CI is defined as follows: CI = (TP\({}_{\text{LOC}}\) + FP\({}_{\text{LOC}}\))/(TP\({}_{\text{LOC}}\) + TN\({}_{\text{LOC}}\) + FP\({}_{\text{LOC}}\) + FN\({}_{\text{LOC}}\))
* The predicted defectiveness (PrD) ratio is defined as the ratio of the number of defects in the commits predicted as defective to the total number of defects. The number of defects in the commits that were true positives is defined as TP\({}_{\text{D}}\) and similarly with TN\({}_{\text{D}}\), FP\({}_{\text{D}}\), and FN\({}_{\text{D}}\). PrD is defined as follows: PrD = TP\({}_{\text{D}}\)/(TP\({}_{\text{D}}\) + FN\({}_{\text{D}}\))