# Credibility Based Imbalance Boosting Method for Software Defect Proneness Prediction
## 4 Experimental Design
### Performance Measures  
In this paper, four overall performance measures, including AUC, \(F_{1}\), adjusted F-measure (AGF), and MCC are used to evaluate the prediction performance. According to [80; 81], some performance measures such as Accuracy and Precision are not suitable to evaluate the prediction performance for highly imbalanced datasets. Therefore, for overall measures including AUC, \(F_{1}\), MCC, and AGF are used in this study, which has been widely used in previous studies [3; 82; 83]. The larger these measures are, the better the prediction performance. The measures discussed above (i.e., AUC, MCC, and AGF) can be calculated according to the confusion matrix, as shown in Table 2. In the SDP field, the defect-prone module is generally regarded as the positive class (aka. minority class) sample and the non-defect-prone one as the negative class (majority class) sample. PD (aka. recall, true positive rate, or sensitivity) and PF (aka. false positive rate) are separately defined, as follows:  
\[PD=\frac{TP}{TP+FN};PF=\frac{FP}{FP+TN} \tag{6}\]  
\(F_{1}\) is a weighted harmonic mean of Precision and PD, which is defined, as follows:  
\[F_{1}=\frac{2\times Precision\times PD}{Precision+PD} \tag{7}\]  
where \(Precision=TP/\left(TP+FP\right)\). Adjusted F-measure (AGF) [84] is an adjusted version of F-Measure that suffers from the well-known problem of without considering the TNR. According to [PERSON] et al. [84], AGF is defined as:  
\[AGF=\sqrt{F_{2}\times InvF_{0.5}} \tag{8}\] \begin{table}
\begin{tabular}{c c c c} \hline \hline  & & \multicolumn{2}{c}{**Predicted**} \\ \cline{3-4}  & & **Positive** & **Negative** \\ \hline \multirow{2}{*}{Actual} & Positive & \(TP\) & \(FN\) \\ \cline{2-4}  & Negative & \(FP\) & \(TN\) \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Confusion Matrix. where \(F_{2}\) can be calculated according to \(F_{\alpha}=\frac{\left(1+\alpha^{2}\right)Precision*PD}{\alpha^{2}\left(Precision+PD \right)}\) when \(\alpha\)=2 and \(Precision=TP/\left(TP+FP\right)\); \(InvF_{0.5}\) is calculated as standard \(F_{0.5}\) after switching the class label of samples in order to construct a new confusion matrix. AUC is the area under of receiver operating characteristic (ROC) curve and ranges from 0 to 1, which is first used to evaluate the machine learning algorithm by [PERSON] [85]. The ROC curve is obtained by plotting PF on the X-axis and PD on the Y-axis. AUC is a widely used measure because it is rarely affected by class imbalance [82]. The AUC value of a random prediction model is 0.5. Given a testing dataset having 20 samples, the actual label and the scores, i.e., the predicted probability of being positive class, are shown in Table 3. In the table, P and N represent the positive and negative classes, respectively. We first sort scores in descending order and take each source as the threshold successively. For a sample, if its score is not smaller than _threshold_, then its label is predicted as positive class (i.e., P). For example, when the score of 3-th sample is used as the threshold, then both 1-th and 2-th samples are classified as positive class and the rest samples are predicted as negative class. For each threshold, we can construct a confusion matrix, as shown in Table 2, and then we can obtain a two-tuple of (PD and PF). Because the testing dataset has 20 samples, we can obtain 20 groups of (PD and PF). We then construct a two-dimensional (2-D) coordinate system where x-axis and y-axis present PF and PD, respectively. We plot 20 point of (PD and PF), then the ROC curve is obtained as Figure 1. As shown in the figure, the area under the ROC curve is called AUC whose value is equal to the sum of area of multiple rectangles under the curve. In this study, we directly use MATLAB build-in function _perfcurve_ to calculate AUC.