Manifold embedded distribution adaptation for cross‐project defect prediction

IET SoftwareVolume 14, Issue 7 p. 825-838 Research ArticleOpen Access Manifold embedded distribution adaptation for cross-project defect prediction Ying Sun, Ying Sun School of Computer Science, Nanjing University of Posts and Telecommunications, Nanjing, 210023 People's Republic of ChinaSearch for more papers by this authorXiao-Yuan Jing, Corresponding Author Xiao-Yuan Jing jingxy_2000@126.com School of Computer Science, Nanjing University of Posts and Telecommunications, Nanjing, 210023 People's Republic of China School of Computer, Wuhan University, Wuhan, 430072 People's Republic of ChinaSearch for more papers by this authorFei Wu, Fei Wu School of Computer Science, Nanjing University of Posts and Telecommunications, Nanjing, 210023 People's Republic of ChinaSearch for more papers by this authorYanfei Sun, Yanfei Sun School of Internet of Things, Nanjing University of Posts and Telecommunications, Nanjing, 210003 People's Republic of China Jiangsu Engineering Research Center of HPC and Intelligent Processing, Nanjing University of Posts and Telecommunications, Nanjing, 210003 People's Republic of ChinaSearch for more papers by this author Ying Sun, Ying Sun School of Computer Science, Nanjing University of Posts and Telecommunications, Nanjing, 210023 People's Republic of ChinaSearch for more papers by this authorXiao-Yuan Jing, Corresponding Author Xiao-Yuan Jing jingxy_2000@126.com School of Computer Science, Nanjing University of Posts and Telecommunications, Nanjing, 210023 People's Republic of China School of Computer, Wuhan University, Wuhan, 430072 People's Republic of ChinaSearch for more papers by this authorFei Wu, Fei Wu School of Computer Science, Nanjing University of Posts and Telecommunications, Nanjing, 210023 People's Republic of ChinaSearch for more papers by this authorYanfei Sun, Yanfei Sun School of Internet of Things, Nanjing University of Posts and Telecommunications, Nanjing, 210003 People's Republic of China Jiangsu Engineering Research Center of HPC and Intelligent Processing, Nanjing University of Posts and Telecommunications, Nanjing, 210003 People's Republic of ChinaSearch for more papers by this author First published: 18 February 2021 https://doi.org/10.1049/iet-sen.2019.0389Citations: 1AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract Cross-project defect prediction (CPDP) technology refers to the constructing prediction model to predict the instance label of the target project by utilising labelled data from an external project. The challenge of CPDP methods is the distribution difference between the data from different projects. Transfer learning can transfer the knowledge from the source domain to the target domain with the aim to minimise the domain difference between different domains. However, most existing methods reduce the distribution discrepancy in the original feature space, where the features are high-dimensional and non-linear, which makes it hard to reduce the distribution distance between different projects. Moreover, previous works mainly consider marginal distribution or conditional distribution difference. In this study, the authors proposed a manifold embedded distribution adaptation (MDA) approach to narrow the distribution gap in manifold feature subspace. MDA maps source and target project data to manifold subspace and then joint distribution adaptation of conditional and marginal distributions is performed on manifold subspace. To evaluate the effectiveness of MDA, the authors perform extensive experiments on 20 public projects with three indicators. The experiment results show that MDA improves the average performance, but the improvement is not statistically significant in comparison to HYDRA (one of the baselines). 1 Introduction Software is commonly used in our life and software security is important [1-3]. Software defects are inevitable in the software product development process. Serious defects even lead to the failure of the product. Predicting the defects before releasing the software product is a crucial step. Software defect prediction (SDP) is an important step in software quality assurance and appears to be a hot research topic [4-8]. SDP analyses the history software resource and then defects the defect-prone instances in software. Most existing SDP methods build defect prediction models under within-project defect prediction (WPDP) [9, 10], i.e. training data on one project, and predicting the defects of new instances in the same project. In general, training a well prediction model needs sufficient labelled data. However, it is difficult to obtain sufficient data or label information. There are several reasons: (i) a new project usually has a few historical defect data; (ii) the collection of the label is time-consuming; (iii) some defect data is encrypted, and we cannot access these data. WPDP will be infeasible when limited labelled historical data are available. Fortunately, some labelled software data is available from open source projects and other external projects. Cross-project defect prediction (CPDP) [11-16] is an alternative solution. CPDP utilises the data of external projects (source project) to conduct the SDP model and predict defects on the target project. However, the source project and target project have different distributions. Zimmermann et al. [11] performed CPDP experiments on 622 cross-project pairs, and only 3.4% pairs achieved satisfactory precision, recall and accuracy values, i.e. the values of precision, recall, and accuracy are >0.75. Thus, CPDP is still challenging. When there are insufficient training data in one domain, transfer learning [17] is used to improve the learning ability by transferring knowledge from other domain by minimising the distribution differences between different domains. Transfer learning is widely used in many fields, such as computer vision [18, 19], natural language processing [20, 21], and SDP [22-24]. In SDP, transfer learning has been used in CPDP tasks [24, 25], which aims to minimise the distribution differences between different projects by transferring the knowledge from source project to the target project. There are different distribution differences in CPDP, the marginal distribution and conditional distribution differences [22, 23]. The marginal distribution is the marginal probability distribution of each instance, which considers the information of the features themselves. Also, the conditional distribution is the class-conditional probability for the instances, which considers the label information from the given instances. The key to successful transfer learning is to reduce the distribution difference [22, 23]. Feature matching is the main category, which performs subspace learning by exploring the similar geometrical properties of the features in manifold subspace [18, 26, 27], or distribution adaptation [28-30] to reduce marginal or conditional distribution divergence between different projects. Subspace learning [31, 32] can conduct subspace transformation to obtain a better feature representation, but it fails to perform feature alignment, thus the feature divergence usually will not be eliminated after subspace transformation. Distribution adaptation [28] methods usually reduce the distribution divergence in the original feature space, where features are high-dimensional and non-linear distributed, which can induce feature distortion and makes it hard to reduce the gap between different projects. In addition, most prior distribution adaptation methods only perform marginal distribution adaptation or conditional distribution adaptation independently [22, 23]. In practice, when the data sets are similar, aligning the conditional distribution is more dominant important. When the datasets are much more dissimilar, the marginal distribution is more important than the conditional distribution [33]. Merely performing one distribution adaptation is not sufficiently. It will limit the performance of CPDP. In this paper, we introduce manifold subspace learning and joint distribution adaptation to tackle these above challenges together in CPDP. Manifold subspace learning is a promising method of subspace learning techniques, which can exploit the subspace geometrical structure and learn a low-dimensional manifold space from the high-dimensional data. Joint distribution adaptation adapts both marginal and conditional distributions and introduces a balance factor to adjust the importance of these two distributions. Manifold subspace learning and joint distribution adaptation are both used to further exploit the advantages of them. The main contributions of this work are (i) Considering the poor feature represent theability of data in the original feature space, we utilise manifold subspace learning to solve the feature distortion in the feature transformation process. Manifold feature learning is capable of exploiting more latent feature properties. (ii) In view of different distribution difference between the source and target projects, we perform joint distribution adaptation to reduce the distribution gap between different projects. In addition, we add a Laplacian regularisation to further exploit the similar geometrical property in manifold space. (iii) Combining the manifold feature learning and joint distribution adaptation, we proposed a manifold embedded distribution adaptation (MDA) approach based on transfer learning to address the challenges of CPDP. We perform experiments on 20 projects of three data sets to evaluate the performance of MDA. Comprehensive experimental results show that MDA is effective for CPDP. The rest of the paper is organised as follows. We discuss the related work in Section 2. The details of the MDA approach are introduced in Section 3. The experimental setup is provided in Section 4. Section 5 discusses the experimental results. Section 6 analyses the threats to the validity of our work. Finally, we conclude our work in Section 7. 2 Related work 2.1 Cross-project defect prediction As far as we know, Briand et al. [34] did the first study of CPDP. They explored whether the CPDP was worth investigating. They found that CPDP is challenging and experimental results were poor. Zimmermann et al. [11] performed a defect prediction experiment on 622 cross-project pairs. Most of the experimental results were disappointing and only 3.4% pairs of all cross-project pairs achieved satisfactory performance in terms of precision, recall, and accuracy measures. The experiment showed that the selection of data, domain and process is important. Turhan et al. [35] proposed a filter used nearest-neighbour (NN) algorithm to select training instances. They selected similar cross-company instances based on within-company instances by using NN filter. They conducted a prediction model on filtered cross-company data. Then they proposed a mixed model [36] which can be used in both WPDP and CPDP. He et al. [25] investigated the selection of training data in CPDP by using the distributional characteristics. They found that the performance of CPDP is related to the distributional characteristics of data sets, which are valuable for training data selection. Xia et al. [14] developed a HYbrid moDel Reconstruction Approach (HYDRA) for CPDP. HYDRA considers the weights of the instances in training data and searches for the best weights on training data to perform instances selection. Ryu et al. [12] proposed a value-cognitive boosting with support vector machine (VCB-SVM) method using boosting SVM and resampling algorithm to address the class imbalance for CPDP. VCB-SVM uses a combination of asymmetric misclassification cost and similarity weights to guide the instance selection process. Nam et al. [37] studied the feature selection process based on metric matching by measuring whether the statistical information is similar, and the experiments showed that CPDP on heterogeneous metrics is feasible. Herbold et al. [38] evaluated 24 methods for CPDP. CamargoCruz method [39] always performs best for all results on data standardisation. Zhou et al. [6] investigated the progress which we made in CPDP by comparing the performance between existing CPDP methods and simple module size models. The simple models even perform more superior than most of the existing CPDP methods. Wu et al. [40] for the first time studied the cross-project semi-supervised defect prediction (CSDP) problem and proposed the cost-sensitive kernelised semi-supervised dictionary learning (CKSDL) method. They introduced semi-supervised dictionary learning into SDP field. CKSDL uses kernel mapping to enhance the separability and cost-sensitive technique to ease the misclassification problem. Liu et al. [41] proposed source project estimator that can choose similar instances from source projects with the highest distribution similarity compared with target project, and then they used TCA + [23] to build a prediction model based on selected instances from two source projects to improve the prediction performance. Wang et al. [42] proposed a cluster-based hybrid feature selection (CHIFS) method. CHIFS reduces the irrelevant features and selects the subsets of relevant features by calculating the feature quality coefficient to improve the prediction performance of CPDP. Chen et al. [43] reviewed the heterogeneous CPDP methods and investigated the feasibility of heterogeneous defect prediction in CPDP. 2.2 Transfer learning Machine learning methods have achieved significant success in different fields. Most machine learning methods have good effect under a common feature coherent distributed assumption. If the feature distribution does not accord with this assumption, the performance of the model will be poor. Transfer learning technologies can overcome this problem by transferring knowledge from a source project to the target project [44]. Nam et al. [23] proposed a transfer defect learning method named TCA +. TCA + first defines some normalisation rules and can select the best normalisation strategy, and then applies transfer component analysis for CPDP to make feature distributions in different projects similar. The experiment results show that TCA + achieves satisfying performance. Canonical correlation analysis (CCA) [17] is a powerful technology in transfer learning. Jing et al. [24] presented CCA + based on transfer learning for heterogeneous defect prediction. CCA + aims to maximise the correlation of different projects and achieves desirable prediction results. Nam et al. [37] proposed heterogeneous defect prediction (HDP) method to predict defect-proneness instances. The key of HDP is to transfer typical defect-proneness from source project to predict defects in the target project. HDP uses metric selection and then matches metrics that have similar distributions between different projects. In [37], extensive experiments were conducted and the experiment obtained expected results. Li et al. [45] proposed a transfer method named cost-sensitive transfer kernel CCA (CTKCCA) to mitigate the linearly inseparable problem and class imbalance problem in CPDP. The experiments on 28 public projects and CTKCCA can obtain desirable performance. In CPDP, the strategies based on manifold learning and probability distribution adaptation are important. These transfer learning methods for CPDP usually consider one distribution in the original space. In this work, we introduce the manifold subspace learning technique and make a step forward to consider two kinds of distribution differences simultaneously. 3 Methodology In this section, we elaborate our proposed approach MDA. First, we introduce the problem definition. Then we describe the details of the MDA approach. 3.1 Problem definition Suppose that S = X S , Y S is the source project. m is the number of instances of source project. Assume the data set as X S = x S 1 , x S 2 , . . . , x S m ∈ R m × d and label set as Y S = y S 1 , y S 2 , . . . , y S m , where x S i denotes the ith instance from the source project. An instance characterises a module in software with a vector, and each element of the vector represents a metric value. y S i denotes the corresponding class label of x S i , and the class represents whether the instance is buggy (defective) or clean (defect-free). Similarly, assume the target project as T = X T , Y T , n is the number of instances of the target project. X T = x T 1 , x T 2 , . . . , x T n ∈ R n × d is the data set, where x T i denotes the ith instance in X T , the labels of the target project are unknown. d is the dimension of instance feature in both S and T. We aim to find an optimal transformation matrix A for the source and target projects, to maximise the correlation between different projects. 3.2 Manifold embedded distribution adaptation MDA consists of two processes. First, MDA performs manifold feature learning to accommodate the feature distortion problem in the transformation process due to the nonlinear distribution of the high-dimensional data. Since the features from different projects in manifold have some similar geometrical structures and intrinsic representations, MDA can effectively exploit latent information from different projects by performing manifold feature learning. Second, to address the challenge of distribution difference from different projects, MDA performs joint distribution adaptation and considers both the importance of marginal and conditional distributions. Manifold feature learning and joint distribution adaptation are both used to further exploit the advantages of them. 3.2.1 Manifold feature learning Manifold feature learning plays an important role since the features in manifold space usually have a good geometric structure to avoid feature distortion. We map data from different projects into a common latent space while keeping the geometry structures of the input manifold, simultaneously the captured connections are mapped onto the common latent space. MDA learns the mapping function g ⋅ in the Grassmann manifold G d k . d k is the dimension of the subspaces of the different project data. We utilise the geodesic flow kernel (GFK) [18] to learn g ⋅ for its computational efficiency. G can be regarded as a collection of all d k -dimensional subspaces. Principal component analysis [18] is performed on the source project and the target project to obtain two corresponding orthogonal subspaces. Each original subspace corresponds to one point in G. The geodesic flow between two points can draw a path for the two subspaces. Constructing a geodesic flow from two points equals to transforming the original features into an infinite-dimensional feature space. The new features of data can be represented as z = g ( x ). From [18], the inner product of transformed features gives rise to a positive semi-definite GFK: z i , z j = x i T G x j (1) where G is a positive semi-definite matrix. The original data can be transformed into Grassmann manifold with z = g ( x ) = G x. G is just an expression form and cannot be computed directly, where x is an instance from source or target project, and X = { X S , X T }. This square root can be calculated by Denman–Beavers algorithm [46]. We use Z = G X as the manifold feature representation in the following sections. 3.2.2 Joint distribution adaptation Distribution adaptation explicitly minimises the predefined distance measures to reduce the distribution discrepancy between different projects. In the CPDP scenario, the data of source and target projects have different marginal distributions and different conditional distributions. To significantly reduce the distribution difference between different projects, joint distribution adaptation for CPDP considers the marginal and the conditional distribution adaptation at the same time. The distance between S and T can be represented as follows: d S , T = 1 − μ d P ( z S ) , P z T + μ d P ( y S | z S ) , P y T | z T (2) where P ( z S ) and P z T denote the marginal distribution. d ( P ( y S | z S ) and P y T | z T and denote conditional distribution. d P ( z S ) , P z T denotes the marginal distribution adaptation, and d P ( y S | z S ) , P y T | z T denotes the conditional distribution adaptation. μ ∈ 0 , 1 is a parameter that adjusts the importance of two kinds of distributions. The label set y T is not available in advance which leads to the calculation of the term P y T | z T is infeasible. We follow the method in [33] and P z T | y T approximately equal to P y T | z T . Thus we use a base classifier trained on S and then obtain the labels of z T . To calculate the divergence between source and target data distributions, the maximum mean discrepancy (MMD) is applied. MMD is an effective method and has been widely used in many methods [29, 47]. Then Formula (2) is formulated as follows: d S , T = 1 − μ 1 m ∑ i = 1 m z S i − 1 n ∑ j = 1 n z T j H 2 + μ ∑ c = 1 C 1 m c ∑ z S i ∈ S c z S i − 1 n c ∑ z T j ∈ T c z T j H 2 (3) where C denotes the number of classes of the label in CPDP. S c and T c denote the instances with the cth class label. m c and n c denote the number of instances of the cth class label from source and target projects. H denotes the reproducing kernel Hilbert space induced by the mapping in manifold space. Using the matrix tricks, we obtain the following formula that equals to Formula (3): min A T Z 1 − μ M 0 + μ M c Z T A + λ A F 2 s . t . A T Z H Z T A = I (4) where Z = Z S , Z T combines the source project and target project data. M 0 and M c denote the MMD matrices that we obtained. A denotes the transformation matrix, A F 2 is the Frobenius norm. I ∈ R ( m + n ) × ( m + n ) is an identity matrix, and H = I − ( 1 / ( m + n ) ) 1 is the centring matrix similar to this work [28]. The constraint condition ensures that A T Z preserves the inner attributes of the original data. The parameter λ is a regularisation term. The MMD matrix can be calculated by the following formulas: M 0 i j = 1 m if z i , z j ∈ S 1 n if z i , z j ∈ T − 1 m n otherwise (5) M c i j = 1 m c if z i , z j ∈ S c 1 n c if z i , z j ∈ T c − 1 m c n c , 1 m c z i ∈ S C , z j ∈ T c 1 n c z i ∈ T c , z j ∈ S c 0 otherwise (6) Additionally, to further exploit a similar relationship when learning transform matrices, we construct a Laplacian matrix by connecting nearby instances. It forces similar instances close to each other in the common space. We denote the inter-project similarity matrix as W i j = sim z i , z j z i ∈ N ( z j ) or z j ∈ N ( z i ) 0 otherwise (7) where sim ( ⋅ , ⋅ ) is a similarity matrix representing the similarity of two instances. N ( z i ) denotes the NNs of z i . Then we introduce the Laplacian matrix L = D − W, D i i = ∑ j = 1 m + n W i j . The final regularisation function can be formulated as R L = ∑ i , j = 1 m + n z i − z j 2 W i j = ∑ i , j = 1 m + n z i L i j z j = Z L Z T (8) The objective function of our MDA approach min A T Z 1 − μ M 0 + μ M c + β L Z T A + λ A F 2 s . t . A T Z H Z T A = I (9) where β is a balance parameter. To solve (9), we denote the Lagrange multipliers as Φ = Φ 1 , Φ 2 , . . . , Φ d l , then we rewrite formula (9) as L A = A T Z 1 − μ M 0 + μ M c + β L Z T A + λ A F 2 + I − A T Z H Z T A Φ (10) Set the derivative of A to 0, ∂ L A / ∂ A = 0. Then we compute the eigenvectors for the generalised eigenvector problem Z 1 − μ M 0 + μ M c + β L Z T + λ I A = Z H Z T A Φ(11) Finally, by solving formula (11), we find its d l small eigenvectors, and the optimal transformation matrix A is obtained. Algorithm 1 presents the details of MDA. Algorithm 1.The pseudo of MDA Input: Data matrices S = X S , Y S and T = { X T } from source and target projects, hyperparameters λ, β, μ. Output: The prediction label of the target project. 1. Learn the GFK G to obtain the feature representations in manifold space by Gong et al. [18]. 2. Get the transformed data representation Z = G X in manifold subspace. 3. Train a basic classifier using S, then get the pseudo label of target data. 4. Compute MMD matrices M 0 and M c by (5) and (6). 5. Using matrix tricks to rewrite (3) and obtain the joint distribution distance representation (4). 6. Construct a Laplacian matrix L = D − W by (7) and (8). 7. Construct objective function (9) by incorporating (4) and (10). 8. Construct Lagrange function (10) for problem (9). 9. Solve the generalised eigendecomposition problem in (11) and get d l eigenvalues. 10. Construct a transformation matrix A by the eigenvectors with respect to the d l eigenvalues. 11. Construct the transformed data of the source and target data by using A. 12. Obtain the prediction label of target data by using the logistic regression (LR) classifier. 4 Experiment In this section, we present the details of the experimental design. Fig. 1 presents the workflow of the CPDP process. To replicate the experiments, we publish the source code of MDA [https://github.com/anonymous-available/MDA]. Fig. 1Open in figure viewerPowerPoint Diagram of the experiment procedure 4.1 Datasets In an experiment, we employ 20 publicly projects from three defect data sets, including AEEEM [48], NASA [49], and PROMISE data sets [50, 51]. Table 1 shows the details about these data sets. Table 1. Details of dataset used in experiment Company Project No. of metrics No. of total instances No. of defective instances % of defective instances AEEEM EQ 61 324 129 39.81% JDT 61 997 206 20.66% LC 61 691 64 9.26% ML 61 1862 245 13.16% PDE 61 1497 209 13.96% NASA CM1 37 327 42 12.84% MW1 37 253 27 10.67% PC1 37 705 61 8.65% PC3 37 1077 134 12.44% PC4 37 1458 178 12.21% PROMISE ant1.7 20 745 166 22.28% camel1.6 20 965 188 19.48% ivy2.0 20 352 40 11.36% jedit4.1 20 312 79 25.32% lucene2.4 20 340 203 59.71% poi3.0 20 442 281 63.57% synapse1.2 20 256 86 33.59% velocity1.6 20 229 78 34.06% xalan2.6 20 885 411 49.83% xerces1.3 20 453 69 15.23% AEEEM data set is collected by D'Ambros et al. [48]. There are five projects, i.e. Equinox (EQ), Eclipse JDT Core (JDT), Apache Lucene (LC), and Eclipse PDE UI (PDE). It consists of 61 metrics, including seventeen source code metrics (i.e. object-oriented metrics and CK metrics), five previous defect metrics, five entropy-of-change metrics, seventeen entropy-of-source-code metrics, and seventeen churn-of-source code metrics. NASA data set is publicly for defect prediction [24, 37], which contains various static code metrics and corresponding defect labels. Static code metrics of NASA include McCabe complexity, Halstead complexity etc. In the NASA data set, we select five projects (CM1, MW1, PC1, PC3 and PC4) that share the same 37 metrics and have been preprocessed by Shepperd et al. [52]. These five projects are commonly used for evaluating the performance of CPDP in previous defect prediction literature [40, 45]. PROMISE data set is collected by Jureczko and Madeyski [50] from the PROMISE data repository [53]. In [38], we use ten handpicked public projects that are commonly used in CPDP studies [37, 54]. One instance is represented by 20 static code metrics including McCabe's cyclomatic metrics, CK metrics and other object-oriented metrics. 4.2 Performance measures In the experiment, we employ three commonly used measures in defect prediction experiment to evaluate the effectiveness of the MDA approach. The measures we used include F-measure, G-measure and AUC [45, 55]. F-measure is the harmonic mean of recall and precision. The recall is a measure defined as TP / TP + FN , where TP and FN mean true positive and false negative. The precision is defined as TP / TP + FP , where FP means false positive. Then F-measure can be defined as 2 × recall × precision recall + precision . G-measure is a harmonic mean of recall and specificity. The specificity is a statistical measure defined as TN / ( TN + FP ) , where TN means true negative. G-measure is defined as (12). TP, FN, FP and TN are defined in Table 2. G − measure = 2 × recall × specificity recall + specificity (12) AUC is used to measure the performance of the classification model. Receiver operating characteristic (ROC) curve is