# A Deep Ensemble Learning Method for Effort-Aware Just-In-Time Defect Prediction
## 4 Experimental Evaluation
### Result and Discussion  
For evaluation of this model, we used the 10 cross fold validation for the performance testing of different unsupervised models. As shown in Figure 3, the correlation between different attributes is unbalanced, in order to handle the unbalanced data; we obtain the balance between two classes. Furthermore, the correlation between the balance attributes is shown in Figure 4. The improved correlation between attributes is obtained by applying normalization to the attributes. \begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline \multirow{2}{*}{**Project**} & \multirow{2}{*}{**Period**} & \multirow{2}{*}{**No.**} & **Avg LOC** & **No. Modified** & **No.** & **No. Dev Per File** \\  & & **Changes** & **File** & **Change** & **Files/Change** & **Changes/Day** & **Max** & **Avg.** \\ \hline
**Buggilla** & 08/1998–12/2006 & 4620 & 389.8 & 37.5 & 2.3 & 1.5 & 37 & 8.4 \\
**Columba** & 11/2002–7/2006 & 4455 & 125.0 & 149.4 & 6.2 & 3.3 & 10 & 1.6 \\
**Eclipse JDT** & 5/2001–12/2007 & 35,386 & 260.1 & 71.4 & 4.3 & 14.7 & 19 & 4.0 \\
**Eclipse** & 5/2001–12/2007 & 64,250 & 231.6 & 72.2 & 4.3 & 26.7 & 28 & 2.8 \\
**Platform** & 1/200–12/2006 & 98,275 & 360.2 & 106.5 & 5.3 & 38.9 & 155 & 6.4 \\
**PostgreSQL** & 7/1996–5/2010 & 20,431 & 563.0 & 101.3 & 4.5 & 4.0 & 20 & 4.0 \\
**OSS-Median** & – & 27,909 & 310.1 & 86.7 & 4.4 & 9.4 & 24 & 4.0 \\
**C-1** & 10/2000–12/2009 & 4096 & – & 16.4 & 2.0 & 1.2 & - & - \\
**C-2** & 10/2000–12/2009 & 9277 & – & 19.2 & 2.4 & 2.8 & - & - \\
**C-3** & 7/2000–12/2009 & 3586 & – & 16.6 & 2.0 & 1.3 & - & - \\
**C-4** & 12/2003–12/2009 & 5182 & – & 12.9 & 1.8 & 2.4 & - & - \\
**C-5** & 10/1982–12/1995 & 10,961 & 303.0 & 39.0 & 4.8 & 2.3 & - & - \\
**Com-Median** & – & 5182 & – & 16.6 & 2.0 & 2.3 & - & - \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Statistic of the projects included in the dataset [15]. Figure 3: Imbalance correlation matrix of attributes. Distribution in the dataset is extremely important when the pattern learning is considered. In our case, as shown in Figure 5 there are many outliers, present in the current dataset, which can create bottleneck at the time of learning. These outliers need to be dealt with correctly, or must be discarded to make the distribution in the dataset even. When checking the dataset distribution with the target it is evident that there are too many outliers, these can create a bottleneck when it comes to pattern learning. So, they must be dealt with properly or they should be discarded. Figure 6 shows the distribution of three unsupervised model, i.e., exp, rexp and sexp with the target. All these three attributes show maximum correlation with the target as compared to the rest of the attributes like npt, pd, ndev, fbs, etc. Figure 4: Subsample correlation matrix of attributes. Figure 5: Distribution of dataset. Figure 6: Distribution of exp, rexp and sexp with the target. In Table 2, we report the performance of deep neural network along with other supervised and unsupervised learning methods. For performance reporting, we used the following measures: accuracy, p-score, f1-score, sensitivity and specificity. According to our evaluation, Random forest followed by XGBoost outperforms other classification methods and shows 71% accuracy. In general, supervised models outperformed unsupervised models when we have labelled data and the dataset is large enough for model training. This table also reports results on the whole project cross validation by using different datasets. In both cases high sensitivity and specificity is observed for all learning methods. [PERSON] et al., [1] state the importance of using the unsupervised model instead of the supervised model for learning. In their paper, they proposed that the removal of highly correlated attributes by computing the reciprocal of raw metrics and discarding the LA and LD helps in ranking the values in descending order.