# A New Framework Consisted of Data Preprocessing and Classifier Modelling for Software Defect Prediction

Authors: Haijin Ji; Song Huang
Journal: Mathematical Problems in Engineering
 (2018)

---

## Abstract

Different data preprocessing methods and classifiers have been established and evaluated earlier for the software defect prediction (SDP) across projects. These novel approaches have provided relatively acceptable prediction results for different software projects. However, to the best of our knowledge, few researchers have combined data preprocessing and building robust classifier simultaneously to improve prediction performances in SDP. Therefore, this paper presents a new whole framework for predicting fault-prone software modules. The proposed framework consists of instance filtering, feature selection, instance reduction, and establishing a new classifier. Additionally, we find that the 21 main software metrics commonly do follow nonnormal distribution after performing a Kolmogorov-Smirnov test. Therefore, the newly proposed classifier is built on the maximum correntropy criterion (MCC). The MCC is well-known for its effectiveness in handling non-Gaussian noise. To evaluate the new framework, the experimental study is designed with due care using nine open-source software projects with their 32 releases, obtained from the PROMISE data repository. The prediction accuracy is evaluated usingF-measure. The state-of-the-art methods for Cross-Project Defect Prediction are also included for comparison. All of the evidences derived from the experimentation verify the effectiveness and robustness of our new framework.


---

## Body

## 1. Introduction

In software engineering, software defect prediction (SDP) refers to estimating precisely the faulty modules and helping software engineers allocate limited testing resources to the most fault-prone modules prior to the testing and maintenance phases. During the phase of predicting defects, the most important step is to establish a robust defect prediction model (also known as the defect predictor), which has motivated many researchers to design and establish various predictors which cover different aspects of software quality [1–7]. These proposed models, in general, could be divided into two groups: Within-Project Defect Prediction (WPDP) and Cross-Project Defect Prediction (CPDP). WPDP refers to the predictors that are trained from data of historical releases in the same project and estimates the most fault-prone modules in the new releases [8]. Zimmermann et al. [9] argued that the defect predictors would perform well within projects as long as there are sufficient data available to train the models. However, it is not always able to collect such kind of historical data for new projects. Thus, WPDP may be restricted to some cases where the size of historical releases is limited.An alternative to predicting defects in new projects lack of historical dataset is to utilize data from different projects as training samples, i.e., CPDP. More formally, CPDP aims to detect the most fault-prone modules in a project using predictors trained by the historical data across projects [2]. One of the key components for the predictors is how to appropriately build machine learning algorithms of SDP. Thus, there are two main factors, i.e., selecting the training dataset appropriately and evolving the learning algorithm, considered by researchers to boost the effectiveness of CPDP.For the first issue, many researchers have found that the performance of predictors is seriously affected by the quality of training data [10–13]. Data quality commonly suffers the challenges of superfluous instances across projects and redundant unnecessary features (also known as software metrics), respectively. It has been proven that instance filtering (denoting similar dataset selection) and feature selection can efficiently handle with the aforementioned problems affecting data quality, respectively [14, 15]. To the best of our knowledge, however, there are few literatures discussing how to perform instance filtering and feature selection simultaneously to improve training data quality in SDP.For improving the learning algorithm, first it should be noted that the main task of CPDP is to classify modules of the new software projects into two classes: non-fault-prone (NFP) or fault-prone (FP). Hence, various classifiers (i.e., the learning algorithms) have been employed to establish CPDP, such as Logistic Regression (LR), Naïve Bayes (NB), Random Tree (RT), and Support Vector Machine (SVM) [5–7]. Compared with LR, RT, and SVM (more detailed information is shown in Section 4.3.3), Menzies et al. [16] argued that NB may be more suitable than the other classifiers [17]. The implementation of NB is based on an assumption that datasets of our interest are normally distributed. However, after we conducted a robust normal distribution hypothetical test (so-called Kolmogorov-Smirnov test [18–20]) on 32 different software releases (the referenced datasets varying from small to large size are described in Table 5 and available in the PROMISE data repository (available at http://openscience.us/repo)), we found that the 21 metrics (see Table 6 including 20 feature attributes and 1 labeled attribute) most commonly used in software tend to follow nonnormal distribution. Therefore, it is not appropriate to select NB as the classifier for CPDP since NB is built on the hypothesis that is inconsistent with the results obtained from the Kolmogorov-Smirnov test (the associated details are referred to in Table 7 and [21]).To overcome the two main existing shortcomings mentioned above in CPDP, we develop a new CPDP (see Figure1). First, as proposed by Turhan et al. [22], Euclidean distance is effective in measuring similarity between the given releases and historical datasets across projects and helps select proper training data automatically. The classick-means algorithm [23] is well-known for its efficiency in clustering datasets based on the Euclidean distance norm. Therefore, we attempt to introducek-means algorithm in combination with correlation analysis and random sampling for instance filtering, feature selection, and instance reduction in data preprocessing, respectively.Figure 1
The framework of the four-stage approach.Second, we attempt to establish a new classifier based on maximum correntropy criterion (MCC) [24], which is well-known for its efficiency in handling non-Gaussian or nonnormal noise, along with an l2 norm penalty. MCC has been verified to obtain robust analysis in face recognition [25], image processing [26], and regression analysis [24], to name a few. However, it is quite difficult to solve the optimization problem of the correntropy combined with l2 norm penalty. Thus, Half-Quadratic (HQ) optimization framework in a proper manner is used to optimize the objective function.In a word, this paper aims to integrate the advantages ofk-means and MCC to boost the potential of CPDP. For simplicity, the newly proposed framework for identifying fault-prone modules is denoted as KMP fork-means and MCC-based predictor. To investigate the efficiency of KMP, we conduct an experiment on the aforementioned 9 software projects and find answers to the following research questions.RQ1: Is the MCC-based classifier better to obtain prediction results compared with other commonly used learning algorithms?This question attempts to evaluate the performance of MCC-based classifier for the prediction in the form of predicting the software module of our interest being faulty or nonfaulty.RQ2: How do KMP perform in comparison to the state-of-the-art method?He et al. [8] proposed a method of finding training dataset across projects by testing tremendous combinations of historical releases from other software projects. This novel approach with a simplified software metric set has been also validated by He et al. [27] and even outperforms the initial one equipped with all metrics in most cases. Therefore, we would like to include the new He et al.’s method as a yardstick to investigate whether KMP can obtain better prediction results.The rest of this paper is structured as follows. Section2 gives a brief overview of CPDP,k-means algorithm, and MCC. Section 3 presents a new proposed methodology for predicting software defects in software systems. Section 4 designs two research questions and the associated experimental settings. The prediction results are shown in Section 5 and discussed in Section 6. Finally, we close this paper with some summaries in Section 7.


## 2. Preliminaries

To facilitate readers’ easy understanding, we review the related work of CPDP, some basic concepts ofk-means algorithm, and maximum correntropy criterion, respectively, in this section.


### 2.1. Related Work about CPDP

Brian et al. [28] may be the first pioneers who published an exploratory literature associated with CPDP to the public. They utilized a simple but rigorous algorithm to analyze an open-source project (called Jwriter) based on another software project (called Xpose) and obtained a relatively accurate fault-prone class ranking. Then the significance of CPDP motivated many researchers to develop various novel models for software fault prediction. Turhan et al. [22] successfully proposed a nearest-neighbor filtering approach to selecting robust cross-project data for training prediction models. Rahman et al. [29] implemented a cost-sensitive analysis of CPDP on 9 large Apache Software Foundation projects with 38 releases. They revealed that the cost-sensitive CPDP performance was not worse than WPDP performance. In recent years, a method for the ensemble of different fault prediction techniques, such as Poisson regress, multilayer perception, and genetic programming, has been proven to be very effective in SDP for it takes the advantage of each participating algorithm and can provide more acceptable prediction results compared to the individual technique [30, 31]. From previous empirical studies on CPDP, we can find that relatively little attention has been paid to propose a predictor concerning both training data selection and learning algorithm optimization simultaneously.


### 2.2. K-Means Algorithm

k-means algorithm is one of the most used clustering techniques, which minimizes the following objective function with respect to cluster center Vi:(1)Jm=∑i=1km∑j=1ndxj,Vi,where km is the number of clusters and d·, in general, is the Euclidean distance norm. xj denotes each data point of the interest (e.g., an instance of software projects), xj∈X. Let C=C1,C2,…,Ckm be a partition of X, ⋃l=1kmCl=X. Thus, the k-means algorithm is executed in the following steps [23]:(1)
Initialize the centerVi of cluster Ci, i=1,2,…,km(2)
Assignxj to the cluster Cl if(2)l=argmini=1kmdxj,Vi(3)
Update the centerVi by(3)Vi=∑xi∈ClxiCl(4)
Repeat steps (2) and (3) until the value ofJm is no longer decreasingThe computational complexity ofk-means algorithm is very low because of its quick convergence and the low cost of the distance computation. Additionally,k-means algorithm measures the similarity between each pair of data point mainly based on the Euclidean distance, whose efficiency in software fault prediction has been demonstrated by Turhan et al. [22]. Therefore, it is reasonable to employk-means algorithm to fast select or filter instances from other software projects as training dataset for identifying fault-prone modules of the new project.


### 2.3. Maximum Correntropy Criterion

Correntropy is well-known in information theoretic learning for its ability of processing non-Gaussian noise [25, 26]. Liu et al. [24] derived it from a correlation function in random processes and stated that correntropy has a close relationship with the information potential (IP) of Renyi’s quadratic entropy andm-estimators. Based on IP, correntropy is defined as a roust local similarity between two arbitrary random variables A and B:(4)VσA,B=EkσA-B,where E· denotes the expectation operator and kσ· is a kernel function that satisfies Mercer’s theory [32]. From (4), it can be seen that correntropy is different from classic kernel functions since it works with each pair of samples independently. Additionally, it can make use of the kernel technique that nonlinearly maps the input data to a higher dimensional space [25, 26]. Hence, correntropy owns several important properties with the robust theoretic foundation, such that it is symmetric, bounded, and positive [24].In practice, we often only obtain a finite number of pairwise dataai,bii=1n, of which the joint probability is unknown. This leads to the sample estimator of correntropy:(5)V^n,σA,B=1n∑i=1nkσai-bi.The Gaussian kernel gx≜exp-x2/2σ2 is generally selected as the kernel function kσ·. Thus, we can rewrite (5) as(6)V^n,σA,B=1n∑i=1ngai-bi.Then we define the error ei in adaptive systems as ei=ai-bi, and the maximum of error ei in (6),(7)Jθ=maxθ1n∑i=1ngej,is called the maximum correntropy criterion (MCC) [24], where parameter θ will be specified in the later sections. Compared with the mean square error, a global metric, the MCC is local, which means that it is faster to find the optima of correntropy value over the search domain, i.e., along the line A=B. Moreover, MCC is of capability in Gaussian or non-Gaussian noise environment [25, 26], which indicates that MCC may be an efficient tool for SDP since most of the variables (such as software metrics) in software engineering follow the nonnormal distributions (see Table 7).


## 2.1. Related Work about CPDP

Brian et al. [28] may be the first pioneers who published an exploratory literature associated with CPDP to the public. They utilized a simple but rigorous algorithm to analyze an open-source project (called Jwriter) based on another software project (called Xpose) and obtained a relatively accurate fault-prone class ranking. Then the significance of CPDP motivated many researchers to develop various novel models for software fault prediction. Turhan et al. [22] successfully proposed a nearest-neighbor filtering approach to selecting robust cross-project data for training prediction models. Rahman et al. [29] implemented a cost-sensitive analysis of CPDP on 9 large Apache Software Foundation projects with 38 releases. They revealed that the cost-sensitive CPDP performance was not worse than WPDP performance. In recent years, a method for the ensemble of different fault prediction techniques, such as Poisson regress, multilayer perception, and genetic programming, has been proven to be very effective in SDP for it takes the advantage of each participating algorithm and can provide more acceptable prediction results compared to the individual technique [30, 31]. From previous empirical studies on CPDP, we can find that relatively little attention has been paid to propose a predictor concerning both training data selection and learning algorithm optimization simultaneously.


## 2.2. K-Means Algorithm

k-means algorithm is one of the most used clustering techniques, which minimizes the following objective function with respect to cluster center Vi:(1)Jm=∑i=1km∑j=1ndxj,Vi,where km is the number of clusters and d·, in general, is the Euclidean distance norm. xj denotes each data point of the interest (e.g., an instance of software projects), xj∈X. Let C=C1,C2,…,Ckm be a partition of X, ⋃l=1kmCl=X. Thus, the k-means algorithm is executed in the following steps [23]:(1)
Initialize the centerVi of cluster Ci, i=1,2,…,km(2)
Assignxj to the cluster Cl if(2)l=argmini=1kmdxj,Vi(3)
Update the centerVi by(3)Vi=∑xi∈ClxiCl(4)
Repeat steps (2) and (3) until the value ofJm is no longer decreasingThe computational complexity ofk-means algorithm is very low because of its quick convergence and the low cost of the distance computation. Additionally,k-means algorithm measures the similarity between each pair of data point mainly based on the Euclidean distance, whose efficiency in software fault prediction has been demonstrated by Turhan et al. [22]. Therefore, it is reasonable to employk-means algorithm to fast select or filter instances from other software projects as training dataset for identifying fault-prone modules of the new project.


## 2.3. Maximum Correntropy Criterion

Correntropy is well-known in information theoretic learning for its ability of processing non-Gaussian noise [25, 26]. Liu et al. [24] derived it from a correlation function in random processes and stated that correntropy has a close relationship with the information potential (IP) of Renyi’s quadratic entropy andm-estimators. Based on IP, correntropy is defined as a roust local similarity between two arbitrary random variables A and B:(4)VσA,B=EkσA-B,where E· denotes the expectation operator and kσ· is a kernel function that satisfies Mercer’s theory [32]. From (4), it can be seen that correntropy is different from classic kernel functions since it works with each pair of samples independently. Additionally, it can make use of the kernel technique that nonlinearly maps the input data to a higher dimensional space [25, 26]. Hence, correntropy owns several important properties with the robust theoretic foundation, such that it is symmetric, bounded, and positive [24].In practice, we often only obtain a finite number of pairwise dataai,bii=1n, of which the joint probability is unknown. This leads to the sample estimator of correntropy:(5)V^n,σA,B=1n∑i=1nkσai-bi.The Gaussian kernel gx≜exp-x2/2σ2 is generally selected as the kernel function kσ·. Thus, we can rewrite (5) as(6)V^n,σA,B=1n∑i=1ngai-bi.Then we define the error ei in adaptive systems as ei=ai-bi, and the maximum of error ei in (6),(7)Jθ=maxθ1n∑i=1ngej,is called the maximum correntropy criterion (MCC) [24], where parameter θ will be specified in the later sections. Compared with the mean square error, a global metric, the MCC is local, which means that it is faster to find the optima of correntropy value over the search domain, i.e., along the line A=B. Moreover, MCC is of capability in Gaussian or non-Gaussian noise environment [25, 26], which indicates that MCC may be an efficient tool for SDP since most of the variables (such as software metrics) in software engineering follow the nonnormal distributions (see Table 7).


## 3. Ak-Means and Maximum Correntropy Criterion Based Predictor (KMP)

In this section, we propose a new CPDP called KMP, which consists of four stages including instance filtering byk-means, feature selection using correlation analysis, instance reduction based on random sampling, and building a MCC-based classifier to categorize modules of new software projects into FP class or NFP one (see Figure 1). The details of each stage in KMP are explained in the following subsections.


### 3.1. Instance Filtering

According to the principle of the analogy-based learning,k-means algorithm is used to select similar datasets between the new software project and historical ones across project. More specifically, let each module (or instance) in dataset of a new software project (DNSP) and historical dataset of other projects (HDP) be xj=f1,f2,…,fkTj=1,2,…,n without the attribute of class label, where f represents the feature attributes and k=20 denotes the number of features. We performk-means algorithm following the step by step procedure shown in Section 2.2. Then let C=C1,C2,…,Ckm be the final partition of Xxj∈X. Thus each module xj in DNSP is assigned to its own cluster Cq. In this stage, we utilizek-means algorithm to classify each module to its nearest cluster measured by Euclidean distance, which has been verified by Turhan et al. [22] as an effective distance norm, for instance, filtering. Therefore, the cluster Cq is selected as the potential training datasets for detecting the fault of xj in next stages.We take the releaseIvy-1.1(see Table 5) as an example to show how to implement instance filtering in detail. All the modules xjf1,…,fkj=1,…,n1 with all software metrics and the unknownBug attribute in the releaseIvy-1.1 and historical cross-project datasets xj′f1,…,fk,lj=1,…,n2 with the knownBugl from the other 31 releases are employed to implementk-means algorithm. We set the number of clustersk to be τn as suggested by Jian and Cheng [33] (τ∈0.33,0.50 is found to be experimentally fine). When τ=0.50, k=56 in this paper since the size of modules is 12505. Table 1 presents the clustering results.Table 1
Clustering results for the releaseIvy-1.1 (n1=111).
Module (No.) Cluster (No.) Module (No.) Cluster (No.) Module (No.) Cluster (No.) 1 29 38 54 75 38 2 8 39 24 76 35 3 15 40 26 77 10 4 41 41 28 78 30 5 38 42 51 79 23 6 24 43 40 80 28 7 4 44 42 81 15 8 13 45 4 82 53 9 53 46 13 83 29 10 29 47 20 84 20 11 51 48 15 85 17 12 40 49 5 86 26 13 10 50 20 87 17 14 24 51 29 88 23 15 20 52 29 89 10 16 4 53 24 90 17 17 21 54 51 91 13 18 55 55 14 92 40 19 37 56 10 93 34 20 35 57 37 94 46 21 13 58 41 95 14 22 51 59 28 96 55 23 23 60 23 97 9 24 16 61 37 98 29 25 23 62 55 99 23 26 37 63 48 100 29 27 25 64 9 101 23 28 53 65 40 102 37 29 4 66 40 103 24 30 35 67 29 104 23 31 21 68 40 105 10 32 6 69 20 106 40 33 51 70 10 107 23 34 4 71 28 108 5 35 56 72 51 109 4 36 37 73 38 110 46 37 40 74 20 111 55


### 3.2. Feature Selection and Instance Reduction

As suggested by He et al. [27], a simplified feature set can improve the performance of SDP and prevent the fault prediction from a time-consuming process. Therefore, in the second stage of KMP, we attempt to remove irrelevant or redundant features in training data derived from the first stage. Considering that we aim to develop a robust linear classifier based on MCC in Section 3.3, the correlation analysis is employed here to implement feature selection. We measure the correlation between 20 features and theBug (denoting the number of bugs detected in the module) for each cluster obtained fromk-means algorithm to find relevant features and reduce redundant ones. Specifically, a module xi in DNSP is assigned to the cluster Cq with the historical dataset Xq=xu∣fu1,fu2,…,fuk,luTu=1,2,…,m across projects, where l denotes theBug and m the number of historical samples. Then we perform correlation analysis between each feature and theBug within Xq. Only features f1,f2,…,fp with statistics significance at the 95% confidence level are accepted as relevant features. Moreover, if the size of f1,f2,…,fp, i.e., p, is quite large, say more than 6, the selected features have to be simplified further and ranked according to their own correlation coefficients. Lastly, we select the subset from the top-K of the ranking list as the most relevant feature set for the cluster Cq, where K=k·lgk recommended by Liu et al. [34] and Song et al. [35]. It should be noted that the capital K is different from the lowercase k in this paper. Considering the close relationship between feature selection and instance reduction, we describe the approach to the latter one in the same subsection.In general, the number of NFP instances in each software project is much greater than the one of FP instances. Therefore, we should appropriately reduce the NFP instances to control the quality of training data with the relevant feature set. Although it bears the risk of information loss by removing valuable instances, random sampling is still one of the most widely applied techniques for its efficiency in instance reduction [34, 36]. We randomly select the NFP and FP instances in each cluster obtained from the first two stages, respectively, assuming that all the modules are generated from the same uniform distribution. The NFP/FP ratio is set to 65%/35% as the terminal condition of random sampling as suggested by Khoshgoftaar et al. [37].Subsequent to the example shown in Section3.1, we show how to implement feature selection and instance reduction in detail. From Table 1, it can be seen that the first module ofIvy-1.1 has been assigned to the 29th cluster. Thus, we perform correlation analysis between each software feature attribute with theBugl using historical datasets xi′f1,f2,…,fk,l in this cluster. The correlation coefficients are presented in Table 2. Unfortunately, none of the metrics is significant at 95% confidence level. Thus we directly rank the features according to their own absolute value of the correlation. Then top-Kof the ranking list are selected as the selected feature set for the 29th cluster. Here, we adopt the recommendation from [35] that K=21×lg⁡21≈6 for 20 software metrics and 1 Bug attribute. Thus, the selected feature subset is {WMC,AMC,AVG_CC,CBO,LCOM3,MOA}. Lastly, we randomly select the NFP and FP instances from HDP in the 29th cluster with the setup of NFP/FP ratio to 65%/35% as the terminal condition.Table 2
Correlation between each software metric and bug. The selected features in the top-K are highlighted.
WMC AMC AVG_CC CA CAM  Bug 0.2177 0.1798 0.1798 0.0789 0.0923  CBM CBO CE DAM DIT  Bug 0.1000 0.1782 0.1001 -0.0566 0.0828  IC LCOM LCOM3 LOC MAX_CC  Bug 0.0333 -0.0457 0.1526 0.0587 0.1186  MFA MOA NOC NPM RFC  Bug 0.0775 0.1667 -0.0287 -0.0145 0.1186


### 3.3. A Robust Classifier Based on MCC

For software fault prediction, letL=l1,…,lmT be a vector denoting theBug attribute of each instance from HDP assigned to the cluster Cq and vector E=∑ifi1βi,…,∑ifimβiT be a linear estimator of the software defect set L. We attempt to find a proper weight vector β=β1,…,βpT so that E becomes as correlated to L as possible under MCC. Thus, we obtain the following MCC-based model:(8)JKMP=maxβ∑j=1mglj-∑i=1pfijβi+λβ22,where g· denotes a Gaussian kernel function and ·2 is the l2-norm. It is difficult to directly optimize (8) for the objective function of KMP is nonlinear. To solve such an optimization problem, the half-quadratic (HQ) technique [38] is utilized in this subsection. The rationale to applying HQ method is its effectiveness in solving information theoretical learning optimization problem [25, 26]. Then, according to the theory of convex conjugated functions, (8) can be solved by the following proposition associated with HQ technique.Proposition 1 (see [25, 26]).
There must be a convex conjugate functionφ of gx so that(9)gx=maxp′p′x2σ2-φp′,And, for a fixed x, the maximum is reached at p′=-gx.Proof.
It can be referred to in He et al. [38].Substituting (9) into (8), we obtain a new objective function in the multiplicative HQ form:(10)JKMP=maxβ,p∑j=1mpjlj-∑i=1pfijβi2-φpj+λβ22,where p=p1,…pmT denotes the auxiliary variables brought by HQ optimization. Thus we convert maximizing JKMPβ to maximizing the new objective function (10), i.e., JKMPβ,p. According to Proposition 1, pj can be computed in an iterative way (t denotes the ordinal number of iterations):(11)pjt+1=-glj-∑i=1pfijβi.When pjt+1 is given, the analytic solution of β in (10) can be determined by(12)βt+1=FTPt+1F+λI-1FTPt+1L,where F=fiji=1:p,j=1:m is the (most) relevant feature set of cluster Cq and Pt+1 is a diagonal matrix whose element Pjj=pj. If there are larger errors lj-∑i=1pfijβi in (11), the maximization function JKMPβ,p will control these errors by giving proper weights in (12) so that the errors provide small contributions in (10). As a result, the optimal solution of JKMPβ,p is robust to large errors. Next, we demonstrate the convergence of JKMPβ,p.Proposition 2.
The sequenceJKMPβt+1,pt+1,t=1,2,… derived from (11)-(12) converges.Proof.
Based on Proposition1 and (11)-(12), we obtain(13)JKMPβt,pt≤JKMPβt,pt+1≤JKMPβt+1,pt+1,which means that the sequence JKMPβt+1,pt+1,t=1,2,… is nondecreasing at each iteration. Additionally, it has been proven that Jθ (i.e., (7)), where θ is identical to β in (10), is bounded in [24]. Moreover, according to Proposition 1, we have(14)maxθJθ+λβ22=maxβJβ+λβ22=maxβ,pJβ,p+λβ22=maxβ,pJKMPβ,p.Then it can be obviously concluded that JKMPβ,p is also bounded with the fixed β. Consequently we demonstrate that the sequence JKMPβt+1,pt+1,t=1,2,… converges.Then we illustrate how to classify an unknown module from DNSP. For each clusterCq, let F=f1,…,fpT be the (most) relevant feature set and β=β1,…,βpT be its associate weight vector derived from (12) in an iterative way. We set λ=1 of (10) throughout the paper, and the kernel size σ is determined using Silverman’s rule(15)σ=1.06×minσE,R1.34×m-1/5,where σE denotes the standard deviation of the error; i.e., lj-∑i=1pfijβi, and R is the error interquartile range. The module xi=fi1,…,fipT of our interest will be categorized into the FP class if(16)xiTβ>α,where α is a constant (α=0.5 is found to be experimentally fine); otherwise, we classify xi to the NFP class.Subsequent to the example presented in previous sections, we show how to perform prediction using MCC-based classifier here. The randomly selected training dataset with the feature subset{WMC,AMC,AVG_CC,CBO,LCOM3,MOA,Bug} (see Section 3.2) is utilized to train the MCC-based classifier based on (10)-(12). We can obtain the weight vector using an iterative way:(17)β=0.0298,0.0394,0.0297,-0.0409,0.0015,0.0015T.The first module ofIvy-1.1 with the filtered feature set {WMC: 1.0986, AMC: 0.6931, AVG_CC: 0.0, CBO: 2.0794, LCOM3: 0.0, MOA: 0.0} is applied as the input of (16):(18)xiTβ=-0.0250<α=0.5.Therefore, we categorize the first module into NFP class, which is consistent with the fact.


### 3.4. Pseudocode of KMP

For readers’ easy understanding, we present the pseudocode of the whole four-stage framework, i.e., KMP, in Algorithm1.Algorithm 1:KMP algorithm.
Input: The dataset xif1,f2,…,fki=1,2,…,n1 in DNSP, historical cross-project datasets xi′f1,f2,…,fk,l with the known Bug  l and the sample sizen2, and let X=xi∪xi′f1,f2,…,fk. Begin: // Instance Filtering // 1: Obtain a partitionC=Cj∣⋃j=1kmCj=X provided by k-means algorithm. // Feature Selection // 2:for  each cluster Cj  do perform correlation analysis using historical datasets xi′f1,f2,…,fk,l belonging to Cj; select the features f1,f2,…,fp with statistics significance at the 95% confidence level; 3:if  p>6  then Rank f1,f2,…,fp according to their own correlation coefficients; The top-K of the ranking list acts as the final selected features for Cj; 4:else features f1,f2,…,fp acts as the final selected features for Cj; 5:end if 6: the historical datasetsxi′ with filtered features is denoted as Sj. 7:end for // Instance Reduction // 8: Randomly select instances fromSj for each cluster Cj until the FP/NFP ratio is 65%/35%. // Train the Prediction Model with the Training Dataset and Predict // 9:for each module xif1,f2,…,fk in DNSP do Use training datasets Sj belonging to the same cluster Cj with xi to train the MCC-based regress according to Eqs. (10)-(15) in an iterative way; Predict whether the module xi is FP or NFP based on Eq. (16); 10:end for


## 3.1. Instance Filtering

According to the principle of the analogy-based learning,k-means algorithm is used to select similar datasets between the new software project and historical ones across project. More specifically, let each module (or instance) in dataset of a new software project (DNSP) and historical dataset of other projects (HDP) be xj=f1,f2,…,fkTj=1,2,…,n without the attribute of class label, where f represents the feature attributes and k=20 denotes the number of features. We performk-means algorithm following the step by step procedure shown in Section 2.2. Then let C=C1,C2,…,Ckm be the final partition of Xxj∈X. Thus each module xj in DNSP is assigned to its own cluster Cq. In this stage, we utilizek-means algorithm to classify each module to its nearest cluster measured by Euclidean distance, which has been verified by Turhan et al. [22] as an effective distance norm, for instance, filtering. Therefore, the cluster Cq is selected as the potential training datasets for detecting the fault of xj in next stages.We take the releaseIvy-1.1(see Table 5) as an example to show how to implement instance filtering in detail. All the modules xjf1,…,fkj=1,…,n1 with all software metrics and the unknownBug attribute in the releaseIvy-1.1 and historical cross-project datasets xj′f1,…,fk,lj=1,…,n2 with the knownBugl from the other 31 releases are employed to implementk-means algorithm. We set the number of clustersk to be τn as suggested by Jian and Cheng [33] (τ∈0.33,0.50 is found to be experimentally fine). When τ=0.50, k=56 in this paper since the size of modules is 12505. Table 1 presents the clustering results.Table 1
Clustering results for the releaseIvy-1.1 (n1=111).
Module (No.) Cluster (No.) Module (No.) Cluster (No.) Module (No.) Cluster (No.) 1 29 38 54 75 38 2 8 39 24 76 35 3 15 40 26 77 10 4 41 41 28 78 30 5 38 42 51 79 23 6 24 43 40 80 28 7 4 44 42 81 15 8 13 45 4 82 53 9 53 46 13 83 29 10 29 47 20 84 20 11 51 48 15 85 17 12 40 49 5 86 26 13 10 50 20 87 17 14 24 51 29 88 23 15 20 52 29 89 10 16 4 53 24 90 17 17 21 54 51 91 13 18 55 55 14 92 40 19 37 56 10 93 34 20 35 57 37 94 46 21 13 58 41 95 14 22 51 59 28 96 55 23 23 60 23 97 9 24 16 61 37 98 29 25 23 62 55 99 23 26 37 63 48 100 29 27 25 64 9 101 23 28 53 65 40 102 37 29 4 66 40 103 24 30 35 67 29 104 23 31 21 68 40 105 10 32 6 69 20 106 40 33 51 70 10 107 23 34 4 71 28 108 5 35 56 72 51 109 4 36 37 73 38 110 46 37 40 74 20 111 55


## 3.2. Feature Selection and Instance Reduction

As suggested by He et al. [27], a simplified feature set can improve the performance of SDP and prevent the fault prediction from a time-consuming process. Therefore, in the second stage of KMP, we attempt to remove irrelevant or redundant features in training data derived from the first stage. Considering that we aim to develop a robust linear classifier based on MCC in Section 3.3, the correlation analysis is employed here to implement feature selection. We measure the correlation between 20 features and theBug (denoting the number of bugs detected in the module) for each cluster obtained fromk-means algorithm to find relevant features and reduce redundant ones. Specifically, a module xi in DNSP is assigned to the cluster Cq with the historical dataset Xq=xu∣fu1,fu2,…,fuk,luTu=1,2,…,m across projects, where l denotes theBug and m the number of historical samples. Then we perform correlation analysis between each feature and theBug within Xq. Only features f1,f2,…,fp with statistics significance at the 95% confidence level are accepted as relevant features. Moreover, if the size of f1,f2,…,fp, i.e., p, is quite large, say more than 6, the selected features have to be simplified further and ranked according to their own correlation coefficients. Lastly, we select the subset from the top-K of the ranking list as the most relevant feature set for the cluster Cq, where K=k·lgk recommended by Liu et al. [34] and Song et al. [35]. It should be noted that the capital K is different from the lowercase k in this paper. Considering the close relationship between feature selection and instance reduction, we describe the approach to the latter one in the same subsection.In general, the number of NFP instances in each software project is much greater than the one of FP instances. Therefore, we should appropriately reduce the NFP instances to control the quality of training data with the relevant feature set. Although it bears the risk of information loss by removing valuable instances, random sampling is still one of the most widely applied techniques for its efficiency in instance reduction [34, 36]. We randomly select the NFP and FP instances in each cluster obtained from the first two stages, respectively, assuming that all the modules are generated from the same uniform distribution. The NFP/FP ratio is set to 65%/35% as the terminal condition of random sampling as suggested by Khoshgoftaar et al. [37].Subsequent to the example shown in Section3.1, we show how to implement feature selection and instance reduction in detail. From Table 1, it can be seen that the first module ofIvy-1.1 has been assigned to the 29th cluster. Thus, we perform correlation analysis between each software feature attribute with theBugl using historical datasets xi′f1,f2,…,fk,l in this cluster. The correlation coefficients are presented in Table 2. Unfortunately, none of the metrics is significant at 95% confidence level. Thus we directly rank the features according to their own absolute value of the correlation. Then top-Kof the ranking list are selected as the selected feature set for the 29th cluster. Here, we adopt the recommendation from [35] that K=21×lg⁡21≈6 for 20 software metrics and 1 Bug attribute. Thus, the selected feature subset is {WMC,AMC,AVG_CC,CBO,LCOM3,MOA}. Lastly, we randomly select the NFP and FP instances from HDP in the 29th cluster with the setup of NFP/FP ratio to 65%/35% as the terminal condition.Table 2
Correlation between each software metric and bug. The selected features in the top-K are highlighted.
WMC AMC AVG_CC CA CAM  Bug 0.2177 0.1798 0.1798 0.0789 0.0923  CBM CBO CE DAM DIT  Bug 0.1000 0.1782 0.1001 -0.0566 0.0828  IC LCOM LCOM3 LOC MAX_CC  Bug 0.0333 -0.0457 0.1526 0.0587 0.1186  MFA MOA NOC NPM RFC  Bug 0.0775 0.1667 -0.0287 -0.0145 0.1186


## 3.3. A Robust Classifier Based on MCC

For software fault prediction, letL=l1,…,lmT be a vector denoting theBug attribute of each instance from HDP assigned to the cluster Cq and vector E=∑ifi1βi,…,∑ifimβiT be a linear estimator of the software defect set L. We attempt to find a proper weight vector β=β1,…,βpT so that E becomes as correlated to L as possible under MCC. Thus, we obtain the following MCC-based model:(8)JKMP=maxβ∑j=1mglj-∑i=1pfijβi+λβ22,where g· denotes a Gaussian kernel function and ·2 is the l2-norm. It is difficult to directly optimize (8) for the objective function of KMP is nonlinear. To solve such an optimization problem, the half-quadratic (HQ) technique [38] is utilized in this subsection. The rationale to applying HQ method is its effectiveness in solving information theoretical learning optimization problem [25, 26]. Then, according to the theory of convex conjugated functions, (8) can be solved by the following proposition associated with HQ technique.Proposition 1 (see [25, 26]).
There must be a convex conjugate functionφ of gx so that(9)gx=maxp′p′x2σ2-φp′,And, for a fixed x, the maximum is reached at p′=-gx.Proof.
It can be referred to in He et al. [38].Substituting (9) into (8), we obtain a new objective function in the multiplicative HQ form:(10)JKMP=maxβ,p∑j=1mpjlj-∑i=1pfijβi2-φpj+λβ22,where p=p1,…pmT denotes the auxiliary variables brought by HQ optimization. Thus we convert maximizing JKMPβ to maximizing the new objective function (10), i.e., JKMPβ,p. According to Proposition 1, pj can be computed in an iterative way (t denotes the ordinal number of iterations):(11)pjt+1=-glj-∑i=1pfijβi.When pjt+1 is given, the analytic solution of β in (10) can be determined by(12)βt+1=FTPt+1F+λI-1FTPt+1L,where F=fiji=1:p,j=1:m is the (most) relevant feature set of cluster Cq and Pt+1 is a diagonal matrix whose element Pjj=pj. If there are larger errors lj-∑i=1pfijβi in (11), the maximization function JKMPβ,p will control these errors by giving proper weights in (12) so that the errors provide small contributions in (10). As a result, the optimal solution of JKMPβ,p is robust to large errors. Next, we demonstrate the convergence of JKMPβ,p.Proposition 2.
The sequenceJKMPβt+1,pt+1,t=1,2,… derived from (11)-(12) converges.Proof.
Based on Proposition1 and (11)-(12), we obtain(13)JKMPβt,pt≤JKMPβt,pt+1≤JKMPβt+1,pt+1,which means that the sequence JKMPβt+1,pt+1,t=1,2,… is nondecreasing at each iteration. Additionally, it has been proven that Jθ (i.e., (7)), where θ is identical to β in (10), is bounded in [24]. Moreover, according to Proposition 1, we have(14)maxθJθ+λβ22=maxβJβ+λβ22=maxβ,pJβ,p+λβ22=maxβ,pJKMPβ,p.Then it can be obviously concluded that JKMPβ,p is also bounded with the fixed β. Consequently we demonstrate that the sequence JKMPβt+1,pt+1,t=1,2,… converges.Then we illustrate how to classify an unknown module from DNSP. For each clusterCq, let F=f1,…,fpT be the (most) relevant feature set and β=β1,…,βpT be its associate weight vector derived from (12) in an iterative way. We set λ=1 of (10) throughout the paper, and the kernel size σ is determined using Silverman’s rule(15)σ=1.06×minσE,R1.34×m-1/5,where σE denotes the standard deviation of the error; i.e., lj-∑i=1pfijβi, and R is the error interquartile range. The module xi=fi1,…,fipT of our interest will be categorized into the FP class if(16)xiTβ>α,where α is a constant (α=0.5 is found to be experimentally fine); otherwise, we classify xi to the NFP class.Subsequent to the example presented in previous sections, we show how to perform prediction using MCC-based classifier here. The randomly selected training dataset with the feature subset{WMC,AMC,AVG_CC,CBO,LCOM3,MOA,Bug} (see Section 3.2) is utilized to train the MCC-based classifier based on (10)-(12). We can obtain the weight vector using an iterative way:(17)β=0.0298,0.0394,0.0297,-0.0409,0.0015,0.0015T.The first module ofIvy-1.1 with the filtered feature set {WMC: 1.0986, AMC: 0.6931, AVG_CC: 0.0, CBO: 2.0794, LCOM3: 0.0, MOA: 0.0} is applied as the input of (16):(18)xiTβ=-0.0250<α=0.5.Therefore, we categorize the first module into NFP class, which is consistent with the fact.


## 3.4. Pseudocode of KMP

For readers’ easy understanding, we present the pseudocode of the whole four-stage framework, i.e., KMP, in Algorithm1.Algorithm 1:KMP algorithm.
Input: The dataset xif1,f2,…,fki=1,2,…,n1 in DNSP, historical cross-project datasets xi′f1,f2,…,fk,l with the known Bug  l and the sample sizen2, and let X=xi∪xi′f1,f2,…,fk. Begin: // Instance Filtering // 1: Obtain a partitionC=Cj∣⋃j=1kmCj=X provided by k-means algorithm. // Feature Selection // 2:for  each cluster Cj  do perform correlation analysis using historical datasets xi′f1,f2,…,fk,l belonging to Cj; select the features f1,f2,…,fp with statistics significance at the 95% confidence level; 3:if  p>6  then Rank f1,f2,…,fp according to their own correlation coefficients; The top-K of the ranking list acts as the final selected features for Cj; 4:else features f1,f2,…,fp acts as the final selected features for Cj; 5:end if 6: the historical datasetsxi′ with filtered features is denoted as Sj. 7:end for // Instance Reduction // 8: Randomly select instances fromSj for each cluster Cj until the FP/NFP ratio is 65%/35%. // Train the Prediction Model with the Training Dataset and Predict // 9:for each module xif1,f2,…,fk in DNSP do Use training datasets Sj belonging to the same cluster Cj with xi to train the MCC-based regress according to Eqs. (10)-(15) in an iterative way; Predict whether the module xi is FP or NFP based on Eq. (16); 10:end for


## 4. Experimental Setup

This section presents the description of the primary work which has to be done before testing the effectiveness of our new predictor. First, we put forward two research questions (RQ) concerning the MCC-based classifier and KMP, respectively. Second, we illustrate how to measure the performance in our experimentation. Last we overview He et al.’s approach [27] to SDP across projects. The rationale to select this method is that it has produced more acceptable results than other CPDPs [8, 22, 27].


### 4.1. Research Questions

Our experimentation is implemented using nine software projects with their 32 releases, more details of which are shown in Table5. It should be noted here that the selected datasets have removed their unique ID and zero lines. Additionally, we have utilized a “log-filter” to all numeric values μ in these software instances with lnμ+1 for data preprocessing, which is suggested by Song et al. [39]. Then we attempt to accumulate empirical evidences to answer the following RQs.RQ1:Is the MCC-based classifier better to obtain prediction results compared with other commonly used learning algorithms?First, we evaluate the performance of the MCC-based classifier for SDP and compare it with conventional classifiers such as LR, NB, RT, and SVM under the same scenario, where training datasets are obtained using He et al.’s approach [27]. The rationale to design the experimentation like this is that we have to control variables in the whole study. In other words, if we directly select the training dataset derived from the first three stages of KMP (see Sections 3.1-3.2) to test different classifiers, it may be difficult to judge whether the prediction results are improved (deteriorated) by the efficiency of classifiers or the techniques of data preprocessing (e.g., instance filtering byk-means algorithm). Additionally, the first three stages of KMP are designed mainly for the MCC-based classifier. They may be unsuitable for training other classic classifiers and even deteriorate their prediction performance.RQ2:How do KMP perform in comparison to the state-of-the-art method?Second, there are two main factors of affecting prediction accuracy for CPDP: the selected training data and learning techniques (i.e., classifiers) applied [27]. Therefore, we investigate the performance of the proposed KMP including training data selection and learning algorithm modelling. He et al.’s [27] approach combined with conventional classifiers are also included for comparison.


### 4.2. Evaluation Measures

Different measures have been proposed for performance evaluation of CPDPs during the calibration, validation, and application phases of software models. In this study, we follow the suggestion from He et al. [8, 27] to utilizeF-measure to evaluate the effectiveness of different predictors for it integratesRecall andPrecision in a single indicator.Recall andPrecisionare commonly used accuracy indicators:(19)Recall=TPTP+FP,Precsion=TPTP+FN,where TP (i.e., true positive) refers to correctly classified buggy classes and FP (i.e., false positive) means nonbuggy classes are wrongly classified to be faulty, while FN (i.e., false negative) denotes that buggy classes are wrongly classified to be nonfaulty. Considering that values ofPrecision andRecall are, in fact, mutually exclusive [8], we should employF-measureto investigate the overall prediction performance:(20)F=2×Precsion×RecallPrecsion+Recall.The closer the value of F is to 1, the better the model performance is. More details of such measures can be found in [27].


### 4.3. He Et al.’s Approach Included for the Experimentation

#### 4.3.1. Training Data Selection

The training datasets derived from He et al.’s approach are the most suitable training data [8] for each testing releases in Table 5. We take the releaseivy-1.1 as an example of explaining how to select its most suitable training data. Each combination of datasets across the Ivy project is employed as potential training data (e.g., <ant-1.3>, <camel-1.2>, <ant-1.3, camel-1.2>…, <jedit-3.2, lucene-2.2, poi-1.5>, <synapse-1.0, xalan-2.6, xerces-1.2> …). If the use of <ant-1.4, lucene-2.2, velocity-1.5> as the training data provides the best prediction result for releaseivy-1.1, we select <ant-1.4, lucene-2.2, velocity-1.5> as the most suitable training dataset forivy-1.1 in the cross-project context. It should be noted that the best prediction result recommended by He et al. [27] means yielding the results with the highestF-measure value (see Section 4.3).


#### 4.3.2. Feature Selection

As shown in Table5, each instance consists of two parts: 20 static code metric attributes and a labeled variableBug. As suggested by He et al. [27], we transform theBug into a binary classification in the following experiments. An instance is NFP only if its number of bugs is equal to 0; otherwise, it is FP. A defect prediction model typically categorizes each instance as either buggy or nonbuggy. Besides this data preprocessing, He et al. [27] recommended using the top-K representative metric attributes to predict defects for all projects. The optimalK is determined by the number of occurrences for each software metric. In our experimentation, the occurrences of the top-K metrics are computed by the algorithm proposed in Section 3.3.1 from He et al. [27]. There are 5 software attributes with the number of occurrences being more than 15 compared with the total number 32: CBO (22), RFC (21), LOC(20), LCOM (18), and CE (17). Therefore, these five representative software metrics constitute a general feature subset used both in Experiment 1 of Section 5 and by He et al.’s method compared with KMP in Experiment 2.


#### 4.3.3. Machine Learning Algorithms Utilized for SDP

The most suitable training datasets with the Top-5 software feature subset are used to train four different classifiers, i.e., LR, SVM, RT, and NB, respectively. The rationale to select these machine learning algorithms is that their efficiency has been widely verified in SDP [8, 22, 27]. Then we illustrate the setup of these prediction models as follows.LR is a probabilistic statistical regression model for SDP by fitting data to a logistic curve [40]. It can be applied as a binary predictor to predict a binary response. In CPDP, the labeled variable (either FP or NFP) is binary; therefore, it is suitable for solving the problem. We implemented LR for software fault prediction with a “glmfit” function and “glmval” in Matlab environment in this paper.SVM is a type of supervised learning model with robust learning algorithms that is typically applied for regression analysis and classification by identifying the optimal hyperplane that appropriately separates samples into two different classes. We performed SVM for SDP with a “svmtrain” function and “svmclassify” in Matlab.RT is also a supervised learning algorithm [41] based on the simplest hypothesis spaces. A Random Tree has two key factors: a body, which is a set of labeled instances, and a schema, which is a set of features. We implemented RT for software fault prediction with a “ClassificationTree.fit” function and “predict” in Matlab.NB is one of the most commonly used classifier based on conditional probability [42–44]. Each feature in this classifier is assumed to be independent, and that is why the classifier is termed as ‘naïve’. In practice, the Naïve Bayes classifier can provide better prediction results than more sophisticated classifiers, although the independence assumption is often violated [16, 45]. The prediction model constructed by this classifier is a set of probabilities. The probability that a new instance is FP or NFP is estimated using the product of the individual conditional probabilities of the feature values for each class. We performed NB for SDP with a “NaiveBayes.fit” function and “predict” in Matlab.The parameters of all these learning algorithms were determined after repeating experiments for many times, and only the best classification result was selected for comparison.


## 4.1. Research Questions

Our experimentation is implemented using nine software projects with their 32 releases, more details of which are shown in Table5. It should be noted here that the selected datasets have removed their unique ID and zero lines. Additionally, we have utilized a “log-filter” to all numeric values μ in these software instances with lnμ+1 for data preprocessing, which is suggested by Song et al. [39]. Then we attempt to accumulate empirical evidences to answer the following RQs.RQ1:Is the MCC-based classifier better to obtain prediction results compared with other commonly used learning algorithms?First, we evaluate the performance of the MCC-based classifier for SDP and compare it with conventional classifiers such as LR, NB, RT, and SVM under the same scenario, where training datasets are obtained using He et al.’s approach [27]. The rationale to design the experimentation like this is that we have to control variables in the whole study. In other words, if we directly select the training dataset derived from the first three stages of KMP (see Sections 3.1-3.2) to test different classifiers, it may be difficult to judge whether the prediction results are improved (deteriorated) by the efficiency of classifiers or the techniques of data preprocessing (e.g., instance filtering byk-means algorithm). Additionally, the first three stages of KMP are designed mainly for the MCC-based classifier. They may be unsuitable for training other classic classifiers and even deteriorate their prediction performance.RQ2:How do KMP perform in comparison to the state-of-the-art method?Second, there are two main factors of affecting prediction accuracy for CPDP: the selected training data and learning techniques (i.e., classifiers) applied [27]. Therefore, we investigate the performance of the proposed KMP including training data selection and learning algorithm modelling. He et al.’s [27] approach combined with conventional classifiers are also included for comparison.


## 4.2. Evaluation Measures

Different measures have been proposed for performance evaluation of CPDPs during the calibration, validation, and application phases of software models. In this study, we follow the suggestion from He et al. [8, 27] to utilizeF-measure to evaluate the effectiveness of different predictors for it integratesRecall andPrecision in a single indicator.Recall andPrecisionare commonly used accuracy indicators:(19)Recall=TPTP+FP,Precsion=TPTP+FN,where TP (i.e., true positive) refers to correctly classified buggy classes and FP (i.e., false positive) means nonbuggy classes are wrongly classified to be faulty, while FN (i.e., false negative) denotes that buggy classes are wrongly classified to be nonfaulty. Considering that values ofPrecision andRecall are, in fact, mutually exclusive [8], we should employF-measureto investigate the overall prediction performance:(20)F=2×Precsion×RecallPrecsion+Recall.The closer the value of F is to 1, the better the model performance is. More details of such measures can be found in [27].


## 4.3. He Et al.’s Approach Included for the Experimentation

### 4.3.1. Training Data Selection

The training datasets derived from He et al.’s approach are the most suitable training data [8] for each testing releases in Table 5. We take the releaseivy-1.1 as an example of explaining how to select its most suitable training data. Each combination of datasets across the Ivy project is employed as potential training data (e.g., <ant-1.3>, <camel-1.2>, <ant-1.3, camel-1.2>…, <jedit-3.2, lucene-2.2, poi-1.5>, <synapse-1.0, xalan-2.6, xerces-1.2> …). If the use of <ant-1.4, lucene-2.2, velocity-1.5> as the training data provides the best prediction result for releaseivy-1.1, we select <ant-1.4, lucene-2.2, velocity-1.5> as the most suitable training dataset forivy-1.1 in the cross-project context. It should be noted that the best prediction result recommended by He et al. [27] means yielding the results with the highestF-measure value (see Section 4.3).


### 4.3.2. Feature Selection

As shown in Table5, each instance consists of two parts: 20 static code metric attributes and a labeled variableBug. As suggested by He et al. [27], we transform theBug into a binary classification in the following experiments. An instance is NFP only if its number of bugs is equal to 0; otherwise, it is FP. A defect prediction model typically categorizes each instance as either buggy or nonbuggy. Besides this data preprocessing, He et al. [27] recommended using the top-K representative metric attributes to predict defects for all projects. The optimalK is determined by the number of occurrences for each software metric. In our experimentation, the occurrences of the top-K metrics are computed by the algorithm proposed in Section 3.3.1 from He et al. [27]. There are 5 software attributes with the number of occurrences being more than 15 compared with the total number 32: CBO (22), RFC (21), LOC(20), LCOM (18), and CE (17). Therefore, these five representative software metrics constitute a general feature subset used both in Experiment 1 of Section 5 and by He et al.’s method compared with KMP in Experiment 2.


### 4.3.3. Machine Learning Algorithms Utilized for SDP

The most suitable training datasets with the Top-5 software feature subset are used to train four different classifiers, i.e., LR, SVM, RT, and NB, respectively. The rationale to select these machine learning algorithms is that their efficiency has been widely verified in SDP [8, 22, 27]. Then we illustrate the setup of these prediction models as follows.LR is a probabilistic statistical regression model for SDP by fitting data to a logistic curve [40]. It can be applied as a binary predictor to predict a binary response. In CPDP, the labeled variable (either FP or NFP) is binary; therefore, it is suitable for solving the problem. We implemented LR for software fault prediction with a “glmfit” function and “glmval” in Matlab environment in this paper.SVM is a type of supervised learning model with robust learning algorithms that is typically applied for regression analysis and classification by identifying the optimal hyperplane that appropriately separates samples into two different classes. We performed SVM for SDP with a “svmtrain” function and “svmclassify” in Matlab.RT is also a supervised learning algorithm [41] based on the simplest hypothesis spaces. A Random Tree has two key factors: a body, which is a set of labeled instances, and a schema, which is a set of features. We implemented RT for software fault prediction with a “ClassificationTree.fit” function and “predict” in Matlab.NB is one of the most commonly used classifier based on conditional probability [42–44]. Each feature in this classifier is assumed to be independent, and that is why the classifier is termed as ‘naïve’. In practice, the Naïve Bayes classifier can provide better prediction results than more sophisticated classifiers, although the independence assumption is often violated [16, 45]. The prediction model constructed by this classifier is a set of probabilities. The probability that a new instance is FP or NFP is estimated using the product of the individual conditional probabilities of the feature values for each class. We performed NB for SDP with a “NaiveBayes.fit” function and “predict” in Matlab.The parameters of all these learning algorithms were determined after repeating experiments for many times, and only the best classification result was selected for comparison.


## 4.3.1. Training Data Selection

The training datasets derived from He et al.’s approach are the most suitable training data [8] for each testing releases in Table 5. We take the releaseivy-1.1 as an example of explaining how to select its most suitable training data. Each combination of datasets across the Ivy project is employed as potential training data (e.g., <ant-1.3>, <camel-1.2>, <ant-1.3, camel-1.2>…, <jedit-3.2, lucene-2.2, poi-1.5>, <synapse-1.0, xalan-2.6, xerces-1.2> …). If the use of <ant-1.4, lucene-2.2, velocity-1.5> as the training data provides the best prediction result for releaseivy-1.1, we select <ant-1.4, lucene-2.2, velocity-1.5> as the most suitable training dataset forivy-1.1 in the cross-project context. It should be noted that the best prediction result recommended by He et al. [27] means yielding the results with the highestF-measure value (see Section 4.3).


## 4.3.2. Feature Selection

As shown in Table5, each instance consists of two parts: 20 static code metric attributes and a labeled variableBug. As suggested by He et al. [27], we transform theBug into a binary classification in the following experiments. An instance is NFP only if its number of bugs is equal to 0; otherwise, it is FP. A defect prediction model typically categorizes each instance as either buggy or nonbuggy. Besides this data preprocessing, He et al. [27] recommended using the top-K representative metric attributes to predict defects for all projects. The optimalK is determined by the number of occurrences for each software metric. In our experimentation, the occurrences of the top-K metrics are computed by the algorithm proposed in Section 3.3.1 from He et al. [27]. There are 5 software attributes with the number of occurrences being more than 15 compared with the total number 32: CBO (22), RFC (21), LOC(20), LCOM (18), and CE (17). Therefore, these five representative software metrics constitute a general feature subset used both in Experiment 1 of Section 5 and by He et al.’s method compared with KMP in Experiment 2.


## 4.3.3. Machine Learning Algorithms Utilized for SDP

The most suitable training datasets with the Top-5 software feature subset are used to train four different classifiers, i.e., LR, SVM, RT, and NB, respectively. The rationale to select these machine learning algorithms is that their efficiency has been widely verified in SDP [8, 22, 27]. Then we illustrate the setup of these prediction models as follows.LR is a probabilistic statistical regression model for SDP by fitting data to a logistic curve [40]. It can be applied as a binary predictor to predict a binary response. In CPDP, the labeled variable (either FP or NFP) is binary; therefore, it is suitable for solving the problem. We implemented LR for software fault prediction with a “glmfit” function and “glmval” in Matlab environment in this paper.SVM is a type of supervised learning model with robust learning algorithms that is typically applied for regression analysis and classification by identifying the optimal hyperplane that appropriately separates samples into two different classes. We performed SVM for SDP with a “svmtrain” function and “svmclassify” in Matlab.RT is also a supervised learning algorithm [41] based on the simplest hypothesis spaces. A Random Tree has two key factors: a body, which is a set of labeled instances, and a schema, which is a set of features. We implemented RT for software fault prediction with a “ClassificationTree.fit” function and “predict” in Matlab.NB is one of the most commonly used classifier based on conditional probability [42–44]. Each feature in this classifier is assumed to be independent, and that is why the classifier is termed as ‘naïve’. In practice, the Naïve Bayes classifier can provide better prediction results than more sophisticated classifiers, although the independence assumption is often violated [16, 45]. The prediction model constructed by this classifier is a set of probabilities. The probability that a new instance is FP or NFP is estimated using the product of the individual conditional probabilities of the feature values for each class. We performed NB for SDP with a “NaiveBayes.fit” function and “predict” in Matlab.The parameters of all these learning algorithms were determined after repeating experiments for many times, and only the best classification result was selected for comparison.


## 5. Experimental Results

In this section, we present the experimental results so as to validate the efficiency and robustness of both MCC-based classifier and KMP.


### 5.1. Experiment 1: Is the MCC-Based Classifier Better to Obtain Prediction Results Compared with Other Commonly Used Learning Algorithms?

In this experiment, we employ the most suitable dataset for each testing software release to train different classifiers. Figure2 presents the results of box-plot analysis. Each part of box-plots represents the minimum, first quartile, median, third quartile, and maximum of the prediction results, respectively. The line in the middle denotes the median. According to Figure 2, we can obtain that (1) except for MCC-based classifier, NB has produced the higher median value and higher minimum value than other learning algorithms, which is consistent with the conclusion derived from [8, 22, 27], (2) MCC-based classifier is superior to other classifiers inF-measure since it has produced the highest median value and highest minimum value, although it obtained the lowest maximum value, (3) the accuracy of prediction results derived from MCC-based classifier is acceptable for it meets the threshold ofF value greater than 0.5 [8, 22, 27], and (4) MCC-based classifier has the narrow box compared to other conventional algorithms, which means that it has the lowest variance in prediction results. The reason why the MCC-based classifier outperforms others may be attributed to the effectiveness of correntropy in handling variables in non-Gaussian distributions.Figure 2
Box-plot analysis of different algorithms for answering RQ1.Moreover, we have utilized Wilcoxon rank sum test to investigate whether these conventional learning algorithms have performed significantly different from MCC-based classifier or not. All theF values derived from different SDP techniques for each release are used to implement the test. Table 3 presents the results that we can find a statistically significant difference at 95% level for all the pairs. In summary, these results validate that the MCC-based classifier is able to perform SDP with acceptable accuracy.Table 3
Wilcoxon rank sum test forF-measure (Highlighted values are statistically significant at the 95% level).
F-measure Groups p values RT vs MCC-based classifier 0.0495 SVM vs MCC-based classifier 0.0478 LR vs MCC-based classifier 0.0097 NB vs MCC-based classifier 0.0498


### 5.2. Experiment 2: How Do KMP Perform in Comparison to the State-of-the-Art Method?

Based on the step by step procedure shown in Sections 3.1–3.3, we can obtain the prediction results of all the other modules in the 32 software releases. The results measured by theF-measure are shown in Figure 3. RT, SVM, LR, NB, and MCC mean that these classifiers were trained by the training data derived from He et al.’s method. From Figure 3, we can see that (1) KMP is superior to He et al.’s method inF-measure for it has produced the highest median value and highest minimum value and (2) the MCC-based classifier, combined with thek-means, correlation analysis, and random sampling, can provide more robust prediction results than the one in coordination with He et al.’s method since it has the narrower box in Figure 3.Figure 3
Box-plot analysis of different algorithms for answering RQ2. RT, SVM, LR, NB, and MCC mean that these classifiers were trained by the training data derived from He et al.’s method. KMP denotes the MCC-based classifier was trained by the framework proposed in Section3.Additionally, we performed Wilcoxon rank sum test to see whether He et al.’s method has significant difference from KMP or not.F-measure values provided by different prediction techniques for each release were used to perform the tests. Table 4 shows that all pairs are significantly different at 95% confidence level. Overall, it may be safe to say that KMP is a robust and effective tool of SDP.Table 4
Wilcoxon rank sum test forF-measure (highlighted values are statistically significant at the 95% level). RT, SVM, LR, NB, and MCC mean that these classifiers were trained by the training data derived from He et al.’s method. KMP denotes the MCC-based classifier was trained by the framework proposed in Section 3.
F-measure Groups p values RT vs KMP 0.0444 SVM vs KMP 0.0040 LR vs KMP 0.0003 NB vs KMP 0.0487 MCC vs KMP 0.0491Table 5
Descriptions of the selected datasets.
No. Datasets #Instances #FP Instances %FP 1 Ant-1.3 125 20 0.160 2 Ant-1.4 178 40 0.225 3 Ant-1.5 293 32 0.109 4 Ant-1.6 351 92 0.262 5 Ant-1.7 745 166 0.223 6 Camel-1.2 608 216 0.355 7 Camel-1.4 872 145 0.166 8 Camel-1.6 965 188 0.195 9 Ivy-1.1 111 63 0.568 10 Ivy-1.4 241 16 0.066 11 Jedit-3.2 272 90 0.331 12 Jedit-4.0 306 75 0.245 13 Lucene-2.0 195 91 0.467 14 Lucene-2.2 247 144 0.583 15 Lucene-2.4 340 203 0.597 16 Poi-1.5 237 141 0.595 17 Poi-2.0 314 37 0.118 18 Poi-2.5 385 248 0.644 19 Poi-3.0 442 281 0.636 20 Synapse-1.0 157 16 0.102 21 Synapse-1.1 222 60 0.270 22 Synapse-1.2 256 86 0.336 23 Synapse-1.4 196 147 0.750 24 Synapse-1.5 214 142 0.664 25 Synapse-1.6 229 78 0.341 26 Xalan-2.4 723 110 0.152 27 Xalan-2.5 803 387 0.482 28 Xalan-2.6 885 411 0.464 29 Xerces-init 162 77 0.475 30 Xerces-1.2 440 71 0.161 31 Xerces-1.3 453 69 0.152 32 Xerces-1.4 588 437 0.743Table 6
Descriptions of software metrics.
Metrics Implication WMC Weighted Methods per Class AMC Average Method Complexity AVG_CC Mean values of methods in the same class CA Afferent couplings CAM Cohesion Among Methods of class CBM Coupling Between Methods CBO Coupling Between Object classes CE Efferent couplings DAM Data Access Metric DIT Depth of Inheritance Tree IC Inheritance Coupling LCOM Lack of Cohesion in Methods LCOM3 Normalized version of LCOM LOC Lines Of Code MAX_CC Maximum values of methods in the same class MFA Measure of Function Abstraction MOA Measure of Aggregation NOC Number of Children NPM Number of Public Methods RFC Response for a Class  Bug Number of bugs detected in the classTable 7
The results for the Kolmogorov-Smirnov test. The logical value h=1 if it rejects the null hypothesis that the data of the interest follow normal distribution at the 0.05 significance level and h=0 if it cannot.
Metric n=40% n=70% n=95% h=0 h=1 h=0 h=1 h=0 h=1 WMC 3 9997 2 9998 1 9999 DIT 9 9991 5 9995 3 9997 LCOM 10 9990 7 9993 3 9997 RFC 4 9996 5 9995 2 9998 CBO 12 9988 6 9994 2 9998 NOC 7 9993 6 9994 4 9996 CA 8 9992 5 9995 3 9997 CE 9 9991 5 9995 1 9999 DAM 13 9987 6 9994 2 9998 NPM 9 9991 9 9991 3 9997 MFA 4 9996 3 9997 2 9998 CAM 7 9993 7 9993 4 9996 MOA 11 9989 4 9996 2 9998 IC 5 9995 2 9998 3 9997 CBM 11 9989 8 9992 4 9996 AMC 5 9995 4 9996 5 9995 LCOM3 13 9987 6 9994 4 9996 MAX_CC 7 9993 5 9995 2 9998 AVG_CC 14 9986 7 9993 3 9997 LOC 5 9995 5 9995 2 9998  Bug 4 9996 6 9994 3 9997


## 5.1. Experiment 1: Is the MCC-Based Classifier Better to Obtain Prediction Results Compared with Other Commonly Used Learning Algorithms?

In this experiment, we employ the most suitable dataset for each testing software release to train different classifiers. Figure2 presents the results of box-plot analysis. Each part of box-plots represents the minimum, first quartile, median, third quartile, and maximum of the prediction results, respectively. The line in the middle denotes the median. According to Figure 2, we can obtain that (1) except for MCC-based classifier, NB has produced the higher median value and higher minimum value than other learning algorithms, which is consistent with the conclusion derived from [8, 22, 27], (2) MCC-based classifier is superior to other classifiers inF-measure since it has produced the highest median value and highest minimum value, although it obtained the lowest maximum value, (3) the accuracy of prediction results derived from MCC-based classifier is acceptable for it meets the threshold ofF value greater than 0.5 [8, 22, 27], and (4) MCC-based classifier has the narrow box compared to other conventional algorithms, which means that it has the lowest variance in prediction results. The reason why the MCC-based classifier outperforms others may be attributed to the effectiveness of correntropy in handling variables in non-Gaussian distributions.Figure 2
Box-plot analysis of different algorithms for answering RQ1.Moreover, we have utilized Wilcoxon rank sum test to investigate whether these conventional learning algorithms have performed significantly different from MCC-based classifier or not. All theF values derived from different SDP techniques for each release are used to implement the test. Table 3 presents the results that we can find a statistically significant difference at 95% level for all the pairs. In summary, these results validate that the MCC-based classifier is able to perform SDP with acceptable accuracy.Table 3
Wilcoxon rank sum test forF-measure (Highlighted values are statistically significant at the 95% level).
F-measure Groups p values RT vs MCC-based classifier 0.0495 SVM vs MCC-based classifier 0.0478 LR vs MCC-based classifier 0.0097 NB vs MCC-based classifier 0.0498


## 5.2. Experiment 2: How Do KMP Perform in Comparison to the State-of-the-Art Method?

Based on the step by step procedure shown in Sections 3.1–3.3, we can obtain the prediction results of all the other modules in the 32 software releases. The results measured by theF-measure are shown in Figure 3. RT, SVM, LR, NB, and MCC mean that these classifiers were trained by the training data derived from He et al.’s method. From Figure 3, we can see that (1) KMP is superior to He et al.’s method inF-measure for it has produced the highest median value and highest minimum value and (2) the MCC-based classifier, combined with thek-means, correlation analysis, and random sampling, can provide more robust prediction results than the one in coordination with He et al.’s method since it has the narrower box in Figure 3.Figure 3
Box-plot analysis of different algorithms for answering RQ2. RT, SVM, LR, NB, and MCC mean that these classifiers were trained by the training data derived from He et al.’s method. KMP denotes the MCC-based classifier was trained by the framework proposed in Section3.Additionally, we performed Wilcoxon rank sum test to see whether He et al.’s method has significant difference from KMP or not.F-measure values provided by different prediction techniques for each release were used to perform the tests. Table 4 shows that all pairs are significantly different at 95% confidence level. Overall, it may be safe to say that KMP is a robust and effective tool of SDP.Table 4
Wilcoxon rank sum test forF-measure (highlighted values are statistically significant at the 95% level). RT, SVM, LR, NB, and MCC mean that these classifiers were trained by the training data derived from He et al.’s method. KMP denotes the MCC-based classifier was trained by the framework proposed in Section 3.
F-measure Groups p values RT vs KMP 0.0444 SVM vs KMP 0.0040 LR vs KMP 0.0003 NB vs KMP 0.0487 MCC vs KMP 0.0491Table 5
Descriptions of the selected datasets.
No. Datasets #Instances #FP Instances %FP 1 Ant-1.3 125 20 0.160 2 Ant-1.4 178 40 0.225 3 Ant-1.5 293 32 0.109 4 Ant-1.6 351 92 0.262 5 Ant-1.7 745 166 0.223 6 Camel-1.2 608 216 0.355 7 Camel-1.4 872 145 0.166 8 Camel-1.6 965 188 0.195 9 Ivy-1.1 111 63 0.568 10 Ivy-1.4 241 16 0.066 11 Jedit-3.2 272 90 0.331 12 Jedit-4.0 306 75 0.245 13 Lucene-2.0 195 91 0.467 14 Lucene-2.2 247 144 0.583 15 Lucene-2.4 340 203 0.597 16 Poi-1.5 237 141 0.595 17 Poi-2.0 314 37 0.118 18 Poi-2.5 385 248 0.644 19 Poi-3.0 442 281 0.636 20 Synapse-1.0 157 16 0.102 21 Synapse-1.1 222 60 0.270 22 Synapse-1.2 256 86 0.336 23 Synapse-1.4 196 147 0.750 24 Synapse-1.5 214 142 0.664 25 Synapse-1.6 229 78 0.341 26 Xalan-2.4 723 110 0.152 27 Xalan-2.5 803 387 0.482 28 Xalan-2.6 885 411 0.464 29 Xerces-init 162 77 0.475 30 Xerces-1.2 440 71 0.161 31 Xerces-1.3 453 69 0.152 32 Xerces-1.4 588 437 0.743Table 6
Descriptions of software metrics.
Metrics Implication WMC Weighted Methods per Class AMC Average Method Complexity AVG_CC Mean values of methods in the same class CA Afferent couplings CAM Cohesion Among Methods of class CBM Coupling Between Methods CBO Coupling Between Object classes CE Efferent couplings DAM Data Access Metric DIT Depth of Inheritance Tree IC Inheritance Coupling LCOM Lack of Cohesion in Methods LCOM3 Normalized version of LCOM LOC Lines Of Code MAX_CC Maximum values of methods in the same class MFA Measure of Function Abstraction MOA Measure of Aggregation NOC Number of Children NPM Number of Public Methods RFC Response for a Class  Bug Number of bugs detected in the classTable 7
The results for the Kolmogorov-Smirnov test. The logical value h=1 if it rejects the null hypothesis that the data of the interest follow normal distribution at the 0.05 significance level and h=0 if it cannot.
Metric n=40% n=70% n=95% h=0 h=1 h=0 h=1 h=0 h=1 WMC 3 9997 2 9998 1 9999 DIT 9 9991 5 9995 3 9997 LCOM 10 9990 7 9993 3 9997 RFC 4 9996 5 9995 2 9998 CBO 12 9988 6 9994 2 9998 NOC 7 9993 6 9994 4 9996 CA 8 9992 5 9995 3 9997 CE 9 9991 5 9995 1 9999 DAM 13 9987 6 9994 2 9998 NPM 9 9991 9 9991 3 9997 MFA 4 9996 3 9997 2 9998 CAM 7 9993 7 9993 4 9996 MOA 11 9989 4 9996 2 9998 IC 5 9995 2 9998 3 9997 CBM 11 9989 8 9992 4 9996 AMC 5 9995 4 9996 5 9995 LCOM3 13 9987 6 9994 4 9996 MAX_CC 7 9993 5 9995 2 9998 AVG_CC 14 9986 7 9993 3 9997 LOC 5 9995 5 9995 2 9998  Bug 4 9996 6 9994 3 9997


## 6. Discussion

Through series of experiments designed by due care, all of the results have proven the effectiveness of KMP:(i)
For instance filtering, the k-means algorithm is proven useful in eliminating redundant instances while keeping relevant ones. The Euclidean distance norm is proved to be appropriate as the similarity functions used for clustering instances. This conclusion is consistent with the findings of Turhan et al. [22](ii)
For feature selection, the correlation analysis is effective for SDP. The feature is quite suitable when its correlation coefficient is accepted with statistics significance at the 95% confidence level. The features with the great values of the coefficient will be selected(iii)
For instance reduction, random sampling is proven useful in the experiments, which conforms to the proposition of Liu et al. [34]. The setup of NFP/FP ratio to 65%/35% is recommended as the terminal condition(iv)
For the MCC-based classifier, we observe that it is better than the classic models, i.e., NB, SVM, LR, and RT. Moreover, combined with k-means algorithm and proper data preprocessing approach, it is also comparable to the state-of-the-art method. Additionally, its effectiveness is not quite affected by the size of software datasets


## 7. Conclusion

In this paper, to combine data preprocessing and learning algorithm modelling simultaneously, we design a new four-stage cross-project defect prediction model, denoted as KMP. The new method includes instance filtering, software feature selection, instance reduction, and performing prediction of fault-prone class. For the stage of instance filtering, the classick-means algorithm is employed to find the relatively nearest-neighbor data of given software projects from historical cross-project data. The rationale to utilizek-means algorithm for training datasets selection is based on the principle of analogy-based learning. For the stage of feature selection, we use correlation analysis for each cluster derived fromk-means algorithm. For the third stage, we utilize random sampling to select training data with proper FP/NFP ratio. For the last stage, a new MCC-based classifier is built to perform software defect prediction. We use the HQ technique to obtain the associated parameters in an iterative way.To validate our approach, we systematically conduct verification experiments. The state-of-the-art methods are included for comparison. The final experimental results demonstrate the efficiency of MCC-based classifier and KMP in improving the performances of CPDP in terms of the designed research questions.In our future work, we attempt to optimize the four-stage method in several different ways. First, we aim to find the interrelations between the instance filtering and the feature selection in terms of some significant factors such as optimal number of clusters setting and the interval of instance sufficiency level. Second, we plan to employ many other real-life software projects to evaluate the efficiency of newly proposed CPDP. Finally, we will explore more efficient approaches to select robust training data for the MCC-based classifier.



---

*Source: 9616938-2018-09-06.xml*