# Identify High-Impact Bug Reports by Combining the Data Reduction and Imbalanced Learning Strategies
## 4 Experimental Design
### Evaluation Metrics  
We use precision, recall, the F-measure and Area Under the Curve (AUC) as our evaluation metrics. These metrics are commonly used measures for evaluating classification performance [24; 50]. They can be derived from the confusion matrix, which captures all four possible classification results, as presented in Table 4. The number of true positives (TP) is the number of low impact bug reports that are correctly divided into low impact bug reports. The number of false positives (FP) is the number of high impact bug reports that are incorrectly divided into low impact bug reports. The number of false negatives (FN) is the number of low impact bug reports that are incorrectly divided into high impact bug reports. The number of true negatives (TN) is the number of high impact bug reports that are correctly divided into high impact bug reports. Where, the sum of TP and TN is the number of correctly classified bug reports and the sum of FP and FN is the number of correctly classified bug reports. Based on the values of TP, FP, FN, and TN, the precision, recall, F-measure and AUC are calculated as follows. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{**Configuration Matrix**} & \multicolumn{5}{c}{**Actual**} \\ \cline{2-7}  & \multicolumn{2}{c}{**Low Impact**} & \multicolumn{2}{c}{**High Impact**} \\ \hline \multirow{2}{*}{Predicted} & low impact & TP:true positives & FP:false positives & \multirow{2}{*}{TN:true negatives} \\  & high impact & FN:false negatives & & & \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Confusion matrix, which can be used to calculate many evaluation metrics. \begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline \multirow{2}{*}{**Project**} & **CiPin5** & \multicolumn{5}{c}{**Training Sets (8 Copies)**} & \multicolumn{2}{c}{**Test Sets (2 Copies)**} \\ \cline{3-8}  & **(Row*Column)** & **The Proportion of Bugs Deleted** & _Severity_ & _Nonsereity_ & _Severity_ & _Nonsereity_ \\ \hline Mozilla & 2702 \(\times\) 2285 & 85.62\% & 1222 & 938 & 306 & 236 \\ GCC & 2734 \(\times\) 3292 & 80.42\% & 1720 & 464 & 429 & 121 \\ Eclipse & 7305 \(\times\) 4697 & 82.20\% & 4090 & 1750 & 983 & 482 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: The preprocessed datasets for Mozilla, GCC, and Eclipse. \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline
**Algorithm** & **NP** & **T** & **a** & **Cross\_L** & **Cross\_H** & **Variation\_Ratio** & **Variation\_Num** \\ \hline GA(FS) & 36 & 50 & one-sixth & 0.5 & 1 & 0.5 & The total number of features / 100 \\ \hline GA(IS) & 30 & 50 & one-sixth & 0.5 & 1 & 0.5 & The total number of instances / 100 \\ \hline GA(FS+IS) & 36 & 50 & one-sixth & 0.5 & 1 & 0.5 & — \\ \hline \hline \end{tabular}
\end{table} 
Table 3: The parameters of the GA(FS), GA(IS), GA(FS+IS). _Accuracy:_ The accuracy of the model is the number of correct classifications divided by the total number of classifications. The accuracy is defined as follows:  
\[Accuracy\ =\ \frac{TP+TN}{TP+FP+TN+FN}\times 100\% \tag{5}\]  
_Precision:_ The percentage of bug reports that are predicted to be either nonhighimpact or highimpact and are correctly predicted. We thus consider a separate precision for each severity. For a bug report severity of either nonhighimpact or highimpact, we define the precision more formally as follows:  
\[Precision_{low}\ =\ \frac{TP}{TP+FP}\times 100\% \tag{6}\]  
\[Precision_{high}\ =\ \frac{TN}{TN+FN}\times 100\% \tag{7}\]  
_Recall:_ The percentage of all bug reports that are actually nonhighimpact or highimpact and are correctly predicted to be nonhighimpact or highimpact, respectively. As for the precision, we also consider a separate recall for each severity. For a bug report severity of either nonhighimpact or highimpact, we define the recall more formally as follows:  
\[Recall_{low}\ =\ \frac{TP}{TP+FN}\times 100\% \tag{8}\]  
\[Recall_{high}\ =\ \frac{TN}{TN+FP}\times 100\% \tag{9}\]  
_F-measure:_ Usually, the precision and recall are not discussed in isolation.