Joint distribution matching model for distribution–adaptation‐based cross‐project defect prediction

IET SoftwareVolume 13, Issue 5 p. 393-402 Research ArticleOpen Access Joint distribution matching model for distribution–adaptation-based cross-project defect prediction Shaojian Qiu, School of Computer Science and Engineering, South China University of Technology, Guangzhou, 510000 People's Republic of ChinaSearch for more papers by this authorLu Lu, Corresponding Author lul@scut.edu.cn School of Computer Science and Engineering, South China University of Technology, Guangzhou, 510000 People's Republic of China Modern Industrial Technology Research Institute, South China University of Technology, Zhongshan, 528400 People's Republic of ChinaSearch for more papers by this authorSiyu Jiang, School of Software Engineering, South China University of Technology, Guangzhou, 510000 People's Republic of ChinaSearch for more papers by this author Shaojian Qiu, School of Computer Science and Engineering, South China University of Technology, Guangzhou, 510000 People's Republic of ChinaSearch for more papers by this authorLu Lu, Corresponding Author lul@scut.edu.cn School of Computer Science and Engineering, South China University of Technology, Guangzhou, 510000 People's Republic of China Modern Industrial Technology Research Institute, South China University of Technology, Zhongshan, 528400 People's Republic of ChinaSearch for more papers by this authorSiyu Jiang, School of Software Engineering, South China University of Technology, Guangzhou, 510000 People's Republic of ChinaSearch for more papers by this author First published: 01 October 2019 https://doi.org/10.1049/iet-sen.2018.5131Citations: 4AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onEmailFacebookTwitterLinked InRedditWechat Abstract Using classification methods to predict software defect is receiving a great deal of attention and most of the existing studies primarily conduct prediction under the within-project setting. However, there usually had no or very limited labelled data to train an effective prediction model at an early phase of the software lifecycle. Thus, cross-project defect prediction (CPDP) is proposed as an alternative solution, which is learning a defect predictor for a target project by using labelled data from a source project. Differing from previous CPDP methods that mainly apply instances selection and classifiers adjustment to improve the performance, in this study, the authors put forward a novel distribution–adaptation-based CPDP approach, joint distribution matching (JDM). Specifically, JDM aims to minimise the joint distribution divergence between the source and target project to improve the CPDP performance. By constructing an adaptive weight vector for the instances of the source project, JDM can be effective and robust at reducing marginal distribution discrepancy and conditional distribution discrepancy simultaneously. Extensive experiments verify that JDM can outperform related distribution–adaptation-based methods on 15 open-source projects that are derived from two types of repositories. Notations source/target project number of source/target project instances d number of feature space dimension marginal distribution of source/target project conditional distribution of source/target project joint distribution of source/target project maximum mean discrepancy between distributions weight vector for the instances of the source project 1 Introduction Predicting the defects of a software project at an early phase of the software lifecycle will effectively help quality-assurance teams search for potential issues and assign test resources [1]. Based on this, many researchers pay close attention to defect-prediction technology and try to detect defect-prone modules or files by machine-learning methods. Most of the previous methods [2-6] operate primarily under a within-project setting, meaning that prediction models are trained and applied on files or modules from the same software project. Nevertheless, in real-world scenarios, due to the frequent software iterations and expensive data labelling, there are little to no within-project training data for a new defect prediction task. Fortunately, different companies and organisations share their public data repositories (e.g. AEEEM [7] and PROMISE [8]) regarding software defects. By taking advantage of these public data, many cross-project defect-prediction (CPDP) approaches have been proposed. These CPDP methods can be categorised into two sub-settings [9]: predicting with only cross-project data [10-15] and predicting with a limited amount of labelled within-project data [16-19]. In this study, we focus on the former (CPDP with only cross-project data), which is more challenging than the latter because it needs to handle the cold-start case. The cold-start case is a well-known issue for recommender systems [20] and it also exists in the early stage of a project that has no defect-labelled data. To verify the feasibility of the CPDP with only cross-project data, He et al. [11] investigate 34 datasets and propose a method that selects suitable training data to build a prediction model automatically. A series of experimental results show that the performance of the improved CPDP method is comparable to within-project defect prediction approaches. In recent years, various researchers have focused on improving the performance of CPDP. Turhan et al. [10] assume that dissimilar data between source and target project easily bring negative effects, so they apply a nearest neighbour filter (filter) that selects analogous data to fine-tune CPDP models. However, the discarded data may contain useful information for model training. In the CPDP area, the data gravitation (DG) method [21] is usually used to re-weight source instances to weaken the impact of irrelevant source data [12, 14]. Ma et al. [12] propose a transfer naive Bayes (TNB) model that first re-weights source instances via DG, then trains a naive Bayes model on these reweighted data. Using the same DG process, Ryu et al. [14] also utilise the re-weighted source instances to build a value-cognitive boosting model with a support vector machine (VCB-SVM). These CPDP approaches mentioned above focus primarily on instances selection and classifiers adjustment. In fact, reducing distribution discrepancy between the source and target projects is also an effective way to improve the performance of CPDP. We call this type of approach the distribution–adaptation-based CPDP method. To explore the effectiveness of the distribution-adaptation-based CPDP method, Nam et al. [13] propose an approach called transfer component analysis plus (TCA+), which extends a transfer learning approach, TCA [22], with some customised normalising rules. Jing et al. [15] also apply the semi-supervised TCA (SSTCA) method when they conduct the CPDP task. Specifically, TCA aims to find a feature transformation that reduces marginal probability distribution discrepancy to facilitate CPDP. However, in practice, just considering the discrepancy of the marginal probability distribution is not enough. Instead, it is necessary to consider the challenging scenario in which the source and target projects are different in both marginal and conditional probability distributions [23]. Let us use a toy example to demonstrate this scenario. As Fig. 1 shows, we display the data of two real-world projects, Eclipse JDT Core (JDT) and Apache Lucene (LC). For effective demonstration and comparison, two-dimensional features, ‘number of code lines’ and ‘number of methods’, are chosen to plot on the X-axis and Y-axis, respectively. We normalise these data by min–max scaling and only show the main distribution in coordinates. The blue (red) points indicate the defective (clean) instances of projects. In this figure, the discrepancies of marginal probability distribution and conditional probability distribution simultaneously exist between two projects ( and ). Fig. 1Open in figure viewerPowerPoint In our problem, both marginal distribution discrepancy and conditional distribution discrepancy exist between the source and target projects (a) Source project (JDT), (b) Target project (LC) To address this challenging scenario, we put forward a novel CPDP approach named joint distribution matching (JDM) that aims to adapt the marginal and conditional distributions jointly in a reproducing-kernel Hilbert space (RHKS), then use a non-parametric transfer learning method named kernel mean matching (KMM) [24] to construct an instance-weights vector for the source project. By integrating these two steps, JDM can simultaneously reduce the divergences in both the marginal distribution and conditional distribution between the source and target projects. In this study, our major contributions are: Unlike prior CPDP methods, which focus on filtering the instances or adjusting the classifiers, this study focuses on the up-to-date distribution–adaptation-based approach to improve CPDP performance. Specifically, this study proposes that, in CPDP task, it is necessary to consider the scenario in which the source and target projects are different in both marginal and conditional probability distributions. This study proposes a novel CPDP approach called JDM. It constructs the measurement for the divergences in both the marginal and conditional distributions between the source and target projects. Furthermore, JDM adopts an effective optimisation method to reduce divergences and improve CPDP performance. To validate the effectiveness of our proposed JDM, we compared it with seven distribution-adaptation-based CPDP methods by employing two base classifiers [logistic regression (LR) and random forest (RF)] and 15 real-world projects (five projects derived from the AEEEM repository and ten derived from the PROMISE repository). The experimental results, based on the Scott–Knott effect size difference (ESD) test, indicate that JDM can perform better than other methods examined. The rest of this paper is organised as follows. In Section 2, we review relevant works. In Section 3, we illuminate our JDM approach in detail. In Section 4, we provide our experimental setting, design, and results to validate the effectiveness of our approach. Finally, we discuss additional settings of JDM and threats to validity in Section 5. Section 6 concludes our work and outlook the future work. 2 Related work CPDP methods are used when a project (target project) has no or only very limited training data for a supervised defect-prediction task, but a similar project (source project) has sufficient training data. In previous researches, CPDP methods are categorised into two main types: predicting only using the data in source project [10-15, 25] and predicting with a small ratio of labelled data in target project [16-19]. In this study, we focus on the scenario in which the target project has no labelled data. In this section, we briefly review the existing CPDP approaches without labelled target-project data. In early studies of CPDP, Zimmermann et al. [25] apply 12 real-world software projects and 622 combinations to analyse the performance of the cross-project model and find that only 3.4% of results are acceptable. This experimental result means that CPDP is still a challenging task. In a more empirical investigation, He et al. [11] put forward a method that automatically selects suitable source data to build a prediction model for the target project. After conducting a series of experiments on 34 datasets obtained from ten open-source projects, the results show that the performance of improved CPDP methods is comparable to within-project defect-prediction approaches. More recently, various researchers have focused on how to improve the CPDP performance. Turhan et al. [10] analyse the applicability of cross-project data and design a nearest neighbour filter that selects analogy data to fine-tune CPDP models. Following instance-based CPDP principles, Ma et al. [12] develop a TNB model that uses the DG method to re-weight source instances and train the naive Bayes model on these weighted data. The experimental results indicate that TNB is more accurate in terms of area under curve (AUC). Using the same data DG method to re-weight source instances, Ryu et al. [14] propose an approach named VCB-SVM to address the class imbalance issue in the CPDP application. However, TNB or VCB-SVM is designed only for naive Bayes [26] or SVM [27]. Unlike the filter, TNB, and VCB-SVM which select or reweight source data, Nam et al. [13] propose a feature representation method named TCA+. TCA+ extends a transfer-learning approach of TCA [22] with customised normalising rules to transfer defect knowledge across projects. The experimental results for eight open-source projects show that TCA+ can improve the performance of CPDP. Jing et al. [15] apply the SSTCA [22] method when they conduct the CPDP task. However, TCA+ and SSTCA ignore to consider the discrepancy of the conditional probability distribution across projects. Besides, some studies on distribution adaptation [28] are also associated with this study. Huang et al. [24] put forward a non-parametric method, KMM, which generates resampling weights for training data without distribution estimation. The method works by adapting the marginal distributions between training and testing data in a RHKS. In this study, we extend the KMM method to match the joint probability distribution between the source and target projects. Long et al. [23] propose a distribution-adaptation-based approach, referred to as joint distribution adaptation (JDA), to adapt both the marginal distribution and conditional distribution of training and testing data. In this study, we apply the measuring approach in JDA to calculate the discrepancy of joint distribution and adjust the distribution–adaptation procedure. 3 Joint adaptation matching methodology In this section, we present the JDM approach in detail. The frequently used notations are shown in Nomenclature section. 3.1 Problem definition Given a labelled source project and unlabelled target project under the assumption that their feature spaces and binary label sets . Let denote the instances of the source project and denote the instances of the target project. and , respectively, represent the labels for and . In general, if two projects and are different, they may have a different marginal distribution and different conditional distribution . In this study, referring to [23], we use joint distribution to represent the composition of the marginal and conditional distributions. To address this challenging scenario, we design the JDM approach to minimise the divergences between (i) and ; (ii) and . There are two mainly steps in our proposed approach. The first step is to construct a divergence measurement that considers the marginal and conditional distributions in the meantime. For minimising the divergence, the second step is to provide a method of estimating instance weights to match the joint distribution of and . 3.2 Constructing the divergence measurement In the above problem setting, and are drawn from two projects whose distributions are different. Many criteria can be used to estimate the distance of distributions (e.g. Kullback Leibler divergence [29] and maximum mean discrepancy (MMD) [30]). To avoid building the parametric estimators and estimating intermediate density, we choose the non-parametric method, MMD, to estimate the distribution distance. According to Borgwardt et al. [30], the empirical estimate of MMD between and is (1)where is the RKHS norm and represents a feature mapping on the RKHS. When calculating the MMD between and , because the posterior probabilities and are quite involved, we turn to utilise the sufficient statistics of class-conditional distributions and instead. Besides, in our CPDP task, there are no labelled data in the target project, so cannot be directly calculated. To address this issue, we apply the predictor, trained by the labelled source data, to predict the pseudo-labels for unlabelled target data. Though some pseudo-labels may be incorrect due to the distribution discrepancy, we assume that the pseudo class centroids are not far from the true class centroids [23]. In our JDM, we select the combination of the distribution–adaptation method (e.g. TCA or KMM) and a base classifier (e.g. LR or RF) as the initial pseudo-label predictor. For further study, we show some experimental results to illuminate the influence of different initial pseudo-labels predictors in the discussion section of this study. Combining the adjustment of posterior probabilities and the operation of pseudo-labels mentioned above, the conditional distribution discrepancy for clean and defective classes can be written as (2). In this study, we mark clean as 0 and defective as 1, the label . (2)where and , and , respectively, represent the number of instances in and . To calculate the divergence between the joint distributions of the source and target projects, we apply the measurement function in [23] that simultaneously considers the MMDs of marginal and conditional distributions. The divergence can be formulated as (3)Incorporating (1), (2), and (3) leads to the divergence on joint distribution, computed as (4) 3.3 Matching on joint distribution In the distribution adaptation area, two types of methods are commonly used to reduce the above joint distribution divergence: minimising with feature representation (e.g. TCA [22], SSTCA [22] and JDA [23]) and minimising with instance reweighting (e.g. KMM [24] and KILEP [31]). Through experimental comparison, we find that the feature transformation functions, JDA, are difficult to converge and the performance is not stable in our CPDP datasets. Considering the alternative instance-based function, we adopt the KMM algorithm to resolve the minimising task. The KMM algorithm, which is a non-parametric method, directly infers weights for the source project without distribution estimation [24]. This algorithm utilises both source and target data to build a transferring bridge. Here, KMM also introduces MMD mentioned in the second part, which is used to measure the similarity of the two distributions. The main purpose of KMM is to calculate appropriate resampling weight for each instance of the source project to minimise the MMD in an ***RKHS. Using the matching operation for marginal distribution as the example, after space mapping and instance weighting, the MMD for the marginal distribution of source and target projects can be formulated as (5)where is an m-dimensional weight vector that we want to learn for the data of the source project, and . In this study, we use the Gaussian kernel function: , and is the parameter of function width. In the same way, we resolve (4) with the above KMM resolution. The formulation can be rewritten as (6)where is a weight vector for the instances whose label is l, and . Now we use all necessary ingredients to construct an optimisation problem to find a suitable to ensure the data distribution of reweighted source instances is close to the distribution of the target project. In reference to [24], we add three constraints to the formulation: the lower bound (LB) of elements in , the upper bound (UB) of elements in , and . A good choice of should be . The constrained optimisation problem can be formulated as (7)Formulation (7) is a standard constrained optimisation problem, which could be solved by many existing solvers and tools. In this study, we use the CVX toolbox (http://cvxr.com/cvx/) to tackle the minimisation process. Importantly, we generally can obtain more accurate pseudo-labels with JDM. Thus, in most cases, if we alternate using the new pseudo-labels to run JDM, we can enhance the prediction quality until convergence (all pseudo-labels are not changes in the next iteration). If researchers use some base classifiers involving randomness (e.g. RF), then JDM cannot converge quickly or may not converge. When using such base classifiers, we can relax the condition of the end of the iteration (e.g. 98% of pseudo-labels are not changed in the next iteration). This pseudo-label refinement procedure [23] is empirically effective in our CPDP datasets, whose details the discussion section presents. Algorithm 1 (Fig. 2) presents the pseudo-code of our JDM approach to perform CPDP. Fig. 2Open in figure viewerPowerPoint Algorithm 1: JDM 4 Experiments In this section, our JDM approach is implemented to perform CPDP empirically. We first introduce the experimental datasets and evaluation metrics. Then, to investigate the performance of the JDM approach, we design some comparative experiments and present the results. 4.1 Datasets To assess the JDM approach, we use two existing defect-prediction repositories (AEEEM [7] and PROMISE [8]). Both are widely-utilised in recent CPDP research [13, 16, 32, 33]. In our experiment, five projects are selected from the AEEEM repository (including 20 one–one CPDP combinations) and ten open-source projects are selected from the PROMISE repository (including 90 one–one CPDP combinations). Of the five AEEEM projects, each project consists of files that include 61 comprehensive attributes and a label (defective or clean). Of the ten PROMISE projects, each project consists of a collection of Java class that includes 20 static code attributes and a label (defective or clean). Tables 1 and 2, respectively, present the essential information of the two repositories including project name, time period (project versions), number of instances, and per cent of defects. To verify the generality of our approach, the repositories are composed of several projects with different sizes and defective rates. Tables 3 and 4, respectively, show the attributes of the AEEEM and PROMISE repositories. Since AEEEM includes 61 attributes, Table 3 only shows some representative attributes. For a detailed description of all the AEEEM attributes, please refer to [7]. Table 1. Five datasets selected from the AEEEM repository, sorted in order of the name. (Defect Rate: the percentage of defective instances) Project name Time period Instance count Defect rate Equinox (EQ) 2005.1–2008.6 324 39.8% Eclipse JDT (JDT) 2005.1–2008.6 997 20.7% Apache Lucene (LC) 2005.1–2008.10 691 9.3% Mylyn (ML) 2005.1–2009.3 1862 13.2% Eclipse PDE UI (PDE) 2005.1–2008.9 1497 14% Table 2. Ten datasets selected from the PROMISE repository Project name Project version Instance count Defect rate ant 1.7 745 22.3% arc 1.0 234 11.5% camel 1.6 965 19.5% e-learn 1.0 64 7.8% ivy 2.0 352 11.4% prop 6.0 660 10% synapse 1.2 256 33.6% system data 1.0 65 13.9% tomcat 6.0 858 9.0% velocity 1.6 229 34.1% Table 3. List of attributes selected from the AEEEM for presentation. The detail descriptions of attributes are referred to [13] Attribute Description ck oo cbo coupling between objects ck oo numberOfLinesOfCode number of lines of code numberOfBugsFoundUntil number of all bugs numberOfCriticalBugsFoundUntil number of bugs whose severity is critical and blocker cvsEntropy entropy of code changes cvsLogEntropy logarithmically decayed CvsEntropy LDHHcbo linearly decayed entropy of ck oo cbo LDHH numberOfLinesOfCode linearly decayed entropy of ck oo numberOfLinesOfCode WCHU cbo weighted churn of ck oo cbo WCHU numberOfLinesOfCode weighted churn of ck oo numberOfLinesOfCode Table 4. List of PROMISE attributes. The descriptions are referred to [16] Attribute Description dit maximum distance from a given class to the root of an inheritance tree noc no. of children of a given class in an inheritance tree cbo no. of classes that are coupled to a given class rfc no. of distinct methods invoked by code in a given class lcom no. of method pairs in a class that does not share access to any class attributes lcom3 another type of lcom metric proposed by Henderson–Sellers npm no. of public methods in a given class loc no. of lines of code in a given class dam the ratio of the no. of private/protected attributes to the total no. of attributes in a given class moa no. of attributes in a given class which are of user-defined types mfa no. of methods inherited by a given class divided by the total no. of methods that can be accessed by the member methods of the given class cam the ratio of the sum of the no. of different parameter types of every method in a given class to the product of the no. of methods in the given class and the no. of different method parameter types in the whole class ic no. of parent classes that a given class is coupled to cbm total no. of new or overwritten methods that all inherited methods in a given class are coupled to amc average size of methods in a given class ca afferent coupling, which measures the no. of classes that depend upon a given class ce efferent coupling, which measures the no. of classes that a given class depends upon max_cc maximum McCabe's cyclomatic complexity (CC) score of methods in a given class avg_cc arithmetic mean of the McCabe's CC scores of methods in a given class 4.2 Evaluation metrics To measure prediction performance, we use F-measure [34] and AUC [35], which are frequently used in recent CPDP researches [9, 12-16], as the evaluation metrics. These evaluation metrics are defined as follows. In a classification task, an instance may have four possible outcomes: classifying a truly defective instance into defective [true positive (TP)]; classifying a clean instance into defective [false positive (FP)]; classifying a truly defective instance into clean [false negative (FN)]; and classifying a truly clean instance into clean [true negative (TN)]. Based on these four outcomes, the F-measure, AUC, and their related metrics are defined below: Recall, also called probability of detection (pd), is a measure of completeness and defines the probabilities of true defective instances compared with the number of all defective instances Precision is a measure of exactness and defines the probabilities of true defective instances to the number of instances predicted as defective In fact, it is difficult to compare the performance of prediction models by using only precision or recall because there is a trade-off between them. To avoid this issue, we choose F-measure, which is a dual assessment of both precision and recall, to evaluate the various approaches in our experiment. The probability of false alarm (pf), also called FP rate, represents the probabilities of FP instances compared with the number of all clean instances A receiver-operating characteristics (ROC) graph, is a two-dimensional graph in which pd is plotted on the y-axis and pf is plotted on the x-axis, can be used to show the relative trade-off between benefit and cost. In this study, we also use a single scalar value, the area under the ROC curve (AUC), to compare the performances of the various prediction models. According to the meaning of AUC [35], it is in positive correlation with pd, while in negative correlation with pf. If AUC < 0.5, it indicates a prediction model with a low pd and a high pf. In the CPDP area, we seek the model with a high pd and a low pf. In other words, a higher AUC is better for our prediction task [12]. 4.3 Experiment design In this study, we compare the JDM approach with seven related distribution–adaptation and CPDP methods on the above 15 open-source projects (five projects are derived from the AEEEM repository, and ten projects are derived from the PROMISE repository). First, our algorithm adopts TCA [22] or KMM [24] approaches when building the initial predictor, so we compare JDM with TCA and KMM to observe the improvement with JDM process. Second, because SSTCA [22] is applied in the most up-to-date domain-adaptation CPDP method [15], we compare JDM with SSTCA algorithm. Third, considering that JDM is combined with JDA [23] and KMM, we also compare JDM with JDA, which are a renowned distribution–adaptation algorithm who handles the distribution difference. Fourth, because we utilise the instance-based method to handle the CPDP, we also compare two classic instance-based CPDP methods, DG [21] and filter [10]. Moreover, to make a fair comparison, we, respectively, use LR and RF as the base classifiers for all the above algorithms, so we also include LR and RF as the baselines for comparisons. Considering that RF involves a degree of randomness when the base classifier is RF, we ran the model ten times each and took the mean as the prediction performance. In the software defect prediction task, the parameter tuning of the model has received attention in recent years [36-38]. Good parameter selection will greatly improve model performance. Therefore, in the experiments this study describes, we have tuned the parameters for each method and extracted the combination of parameters for the first-ranking prediction result in the Scott–Knott ESD test [39]. Table 5 lists the tuned parameters in our experiment, and Table 6 presents the final parameter selection for each method.