# An Approach to Improving Homogeneous Cross-Project Defect Prediction by Jensen-Shannon Divergence and Relative Density

Authors: Jiajia Ren; Chunyu Peng; Shang Zheng; Haitao Zou; Shang Gao
Journal: Scientific Programming
 (2022)

---

## Abstract

Homogeneous cross-project defect prediction (HCPDP) aims to apply a binary classification model built on source projects to a target project with the same metrics. However, there is still room for improvement in the performance of the existing HCPDP models. This study has proposed a novel approach, including one-to-one and many-to-one predictions. First, we apply the Jensen-Shannon divergence to select the most similar source project automatically. Second, relative density estimation is introduced to choose the suitable instance of the selected source project. Third, one-to-one and many-to-one prediction models are trained by the selected instances. Finally, two benchmark datasets are used to evaluate the proposed approach. Compared to the state-of-the-art methods, the experimental results demonstrated that the proposed approach could improve the prediction performance in the F1-score, AUC, and G-mean metrics and exhibit strong adaptability to the traditional classifiers.


---

## Body

## 1. Introduction

Software defects are inevitable in the process of software development. Software defects affect user experience and bring economic losses to enterprises. Therefore, it is necessary to help software engineers accurately predict defects in projects. Recent years have seen an increasing trend in software defect prediction (Hall et al. [1]; Rahman et al. [2]; Yang et al. [3]; Ghotra et al. [4], Wen et al. [5]; Tang et al. [6]; Jiarpakdee et al. [7]). Previous research has demonstrated that machine-learning approaches are suitable for software defect prediction (Menzies et al. [8]; Zimmermann and Nagappan [9]; Hassan [10]). The project manager can use the defect prediction model to determine whether a module has defects. However, it is difficult to build an effective prediction model if not enough data are available. An alternative solution is cross-project defect prediction (CPDP) (He et al. [11]; Ma et al. [12]; Canfora et al. [13]; Herbold et al. [14]; Hosseini et al. [15]), which constructs the classifiers on existing projects with sufficient labeled data and predicts the defects of the target project. When the metrics (features) between the source and target projects are the same, CPDP is called homogeneous cross-project defect prediction (HCPDP).Watanabe et al. [16] introduced the metric compensation method, which revises the metrics of the target project based on the metrics of the source project. Turhan et al. [17] calculated the Euclidian distance of each instance and selected the nearest instances to complete defect prediction, but it was one-phase filtering. Herbold [18] calculated the feature vector of each metric and applied two strategies to select suitable source projects. Fukushima et al. [19] proposed an approach to consider the similarity between the source and target projects by calculating the Spearman values. Panichella et al. [20] applied ensemble learning to improve CPDP performance. Subsequently, two-phase filtering methods (He et al. [21]; He et al. [22]) were proposed to improve the work of Turhan. These pieces of work selected the source project and applied the filtering method to construct the prediction models. Due to the development of transfer learning, Nam et al. [23] applied transfer component analysis to search the feature mapping space for the source and target projects and constructed the prediction models. Liu et al. [24] proposed a two-phase transfer learning method, which used a source project estimator to select similar projects by measuring the Euclidian distance.Most prior studies focused on one-to-one defect prediction, and their findings have proved that selecting a suitable project is the primary solution for HCPDP. However, there is still room for improvement in the selection process, and many-to-one defect prediction is necessary for a natural environment. Therefore, this paper expands the Jensen-Shannon (JS) divergence (Zheng et al. [25]) to many-to-one defect prediction and applies a relative density strategy together to select the training instances. Unlike choosing the features in prior studies, JS divergence can keep all the features and calculate the similarity of two probability distributions for two projects directly. Moreover, prior work (Turhan et al. [17]; He et al. [22]; Nam et al. [23]) considered using distance or probability density to select the training instance, but it is difficult to choose when the metric dimension is high. This paper presents an alternative to solving this problem. Although it is not easy to exactly obtain the probability density of each instance, it is feasible to extract the proportional relation between the probability densities of any two instances. Therefore, the relative density is proposed to reflect the proportional relations for selecting instances in the projects. Finally, a novel weighting strategy in many-to-one defect prediction is designed. The experimental results show that the proposed approach can improve HCPD performance and be adaptable to different machine-learning algorithms.The main contributions of this study are summarized as follows:(1)
An approach for HCPDP based on the JS divergence with relative density is designed.(2)
An ensemble weighting strategy based on JS divergence is proposed to build a many-to-one prediction model.(3)
The proposed approach is not subject to change in the learning algorithms.To evaluate the proposed approach, two benchmark datasets from NASA (Shepperd et al. [26]) and PROMISE (Jureczko and Madeyski [27]) were selected. Compared to the previous studies, one-to-one prediction exhibited average improvements in the F1-score of 13%∼119% on NASA and 5%∼33% on PROMISE, in the AUC of 8%∼31% on NASA and 2%∼30% on PROMISE, in the G-mean by 10%∼68% on NASA and 3%∼56% on PROMISE; many-to-one prediction was averagely improved by 10%∼193% of the F1-score on NASA and 15%∼63% on PROMISE, 1%∼25% of the AUC on NASA and 3%∼14% on PROMISE, 2%∼75% of the G-mean on NASA and 10%∼34% on PROMISE. Moreover, five widely used methods, including Logistic Regression (LR), Naive Bayesian (NB), Support Vector Machine (SVM), K-Nearest Neighbor (KNN), and Random Forest (RF), were selected to test the proposed approach.The remainder of this paper is organized as follows: Section2 presents the preliminary results related to this study. The proposed approach is described in Section 3. Section 4 gives the experiment setup. The experiment results and analysis are shown in Section 5, and Section 6 introduces the threats to validity. Section 7 outlines the related work, and conclusion and future works are provided in Section 8.


## 2. Preliminary

This section introduces the preliminary result related to the proposed approach, including Jensen–Shannon divergence and the relative density estimation strategy.


### 2.1. Jensen-Shannon Divergence

Feature selection is often used in CPDP, and filtering, wrapper, and embedded methods are the main methods. The filtering method is very fast, but the selected features may not be useful for the model. The advantage of the wrapper method is that the selected features can improve the effect of the model, but the disadvantage is that the model needs to be trained many times to evaluate the effect of the features. The embedded method considers the advantages of the above two methods, but it is difficult to define a criterion for judging whether a feature is valid or not. Therefore, this paper considered keeping all the features and chose Jensen-Shannon (JS) divergence to measure the similarity of two probability distributions. In probability, JS divergence has the ability to measure the similarity of two distributions, which is based on Kullback–Leibler (KL) divergence (Lamberti and Majtey [28]). Compared with KL divergence, JS divergence has symmetry, which is suitable for cross-project defect prediction. For example, when A is a target project and B is a source project, JS (A||B) can be calculated. However, JS (B||A) is not calculated again because of the symmetry of JS divergence, that is, the two values are the same. Hence, JS divergence can reduce the analysis process of CPDP. Moreover, the range of JS divergence is [0,1], and its value is a constant, which discriminates the similarity of two projects more accurately.Suppose there are two distributions,P and Q, in which P is the true distribution and Q is the approximate distribution. Then the KL divergence is defined as(1)KLP‖Q=∑x∈DPxlogPxQx.However,KL divergence is asymmetric. In other words, KLP‖Q≠KLQ‖P. Then, the JS divergence is proposed, and it is represented by (2) as follows:(2)JSP‖Q=12KLP‖P+Q2+12KLQ‖P+Q2.Then MONTE CARLO is used to calculate JS value by (3) as follows:(3)DMCP‖Q=1n∑i=1nlogpxiqxi⟶DP‖Q.As described in work (Lamberti & Majtey [28]), The smaller the JS divergence, the more similar the source project is to the target project. In summary, JS divergence is used to avoid the selection randomness of source projects.


### 2.2. Relative Density Estimation Strategy

After a similar source project is selected, it is necessary to filter the suitable instances as the training data. If all the instances are used, noises and outliers will be drawn. The probability density can be calculated to distinguish significant instances from outliers. However, it is difficult to obtain the exact results when the metric dimension is high. Moreover, it is time-consuming as well.This subsection presents an alternative to solving this problem. It is not easy to exactly obtain the probability density of each instance, but it is feasible to extract the proportional relation of the probability densities between any two instances. In this study, the relative density reflects the proportional relation. To calculate the relative density, a K-nearest neighbors-based probability density estimation (KNN-PDE) (Fukunaga and Hostetler [29]; Mack and Rosenblatt [30]; Yu et al. [31]; Zheng et al. [32]) alike strategy is adopted. KNN-PDE calculates the Kth nearest neighbor distance of a single instance to measure its probability density distribution. When the number of instances tends to infinity, the results obtained from KNN-PDE converge to the actual probability density distribution.For each training instancexi, its Kth nearest neighbors are easy to find, and the distance between them is defined as diK. If diK is larger, xi will hold a lower density. For noise and outliers, they should appear in the region of low density, so diK can be used to measure the significance of each instance. To obtain a higher value for high-density instances and a lower value for noise and outliers, the reciprocal of diK can be used. The reciprocal of the K-nearest neighbors’ distance of each instance is named as its relative density. The relative density between any two samples is inversely proportional to the Kth nearest neighbor distance between them.(4)1/diK1/djK=djKdiK.According to (4), the selection of K is essential. If it is too small, noise and outliers are challenging to identify, but if it is too large, some small disjunctions will become blurred. The appropriate value for K was discussed in Section 5. After the estimation process was completed, all the instances were sorted, and then they were ready for the instance selection.


## 2.1. Jensen-Shannon Divergence

Feature selection is often used in CPDP, and filtering, wrapper, and embedded methods are the main methods. The filtering method is very fast, but the selected features may not be useful for the model. The advantage of the wrapper method is that the selected features can improve the effect of the model, but the disadvantage is that the model needs to be trained many times to evaluate the effect of the features. The embedded method considers the advantages of the above two methods, but it is difficult to define a criterion for judging whether a feature is valid or not. Therefore, this paper considered keeping all the features and chose Jensen-Shannon (JS) divergence to measure the similarity of two probability distributions. In probability, JS divergence has the ability to measure the similarity of two distributions, which is based on Kullback–Leibler (KL) divergence (Lamberti and Majtey [28]). Compared with KL divergence, JS divergence has symmetry, which is suitable for cross-project defect prediction. For example, when A is a target project and B is a source project, JS (A||B) can be calculated. However, JS (B||A) is not calculated again because of the symmetry of JS divergence, that is, the two values are the same. Hence, JS divergence can reduce the analysis process of CPDP. Moreover, the range of JS divergence is [0,1], and its value is a constant, which discriminates the similarity of two projects more accurately.Suppose there are two distributions,P and Q, in which P is the true distribution and Q is the approximate distribution. Then the KL divergence is defined as(1)KLP‖Q=∑x∈DPxlogPxQx.However,KL divergence is asymmetric. In other words, KLP‖Q≠KLQ‖P. Then, the JS divergence is proposed, and it is represented by (2) as follows:(2)JSP‖Q=12KLP‖P+Q2+12KLQ‖P+Q2.Then MONTE CARLO is used to calculate JS value by (3) as follows:(3)DMCP‖Q=1n∑i=1nlogpxiqxi⟶DP‖Q.As described in work (Lamberti & Majtey [28]), The smaller the JS divergence, the more similar the source project is to the target project. In summary, JS divergence is used to avoid the selection randomness of source projects.


## 2.2. Relative Density Estimation Strategy

After a similar source project is selected, it is necessary to filter the suitable instances as the training data. If all the instances are used, noises and outliers will be drawn. The probability density can be calculated to distinguish significant instances from outliers. However, it is difficult to obtain the exact results when the metric dimension is high. Moreover, it is time-consuming as well.This subsection presents an alternative to solving this problem. It is not easy to exactly obtain the probability density of each instance, but it is feasible to extract the proportional relation of the probability densities between any two instances. In this study, the relative density reflects the proportional relation. To calculate the relative density, a K-nearest neighbors-based probability density estimation (KNN-PDE) (Fukunaga and Hostetler [29]; Mack and Rosenblatt [30]; Yu et al. [31]; Zheng et al. [32]) alike strategy is adopted. KNN-PDE calculates the Kth nearest neighbor distance of a single instance to measure its probability density distribution. When the number of instances tends to infinity, the results obtained from KNN-PDE converge to the actual probability density distribution.For each training instancexi, its Kth nearest neighbors are easy to find, and the distance between them is defined as diK. If diK is larger, xi will hold a lower density. For noise and outliers, they should appear in the region of low density, so diK can be used to measure the significance of each instance. To obtain a higher value for high-density instances and a lower value for noise and outliers, the reciprocal of diK can be used. The reciprocal of the K-nearest neighbors’ distance of each instance is named as its relative density. The relative density between any two samples is inversely proportional to the Kth nearest neighbor distance between them.(4)1/diK1/djK=djKdiK.According to (4), the selection of K is essential. If it is too small, noise and outliers are challenging to identify, but if it is too large, some small disjunctions will become blurred. The appropriate value for K was discussed in Section 5. After the estimation process was completed, all the instances were sorted, and then they were ready for the instance selection.


## 3. Proposed Approach

As shown in Figure1, the proposed approach includes the training and prediction phases. First, the JS divergence values between source and target projects are calculated to select the most similar project. Then, the relative density information is estimated to select the high-density instances. Finally, the selected instances are used to train one-to-one classifiers or many-to-one classifiers. In the prediction phase, the trained classifiers are evaluated by the target project, and the prediction results are obtained. Since the proposed method does not divide the data, cross-validation is not required. The details are illustrated as follows.Figure 1
The framework of the proposed approach.


### 3.1. Training Phase

#### 3.1.1. JS divergence for Project Selection

Figure2 describes the project selection process based on JS divergence. The Gaussian mixture model (GMM = {GMMS1, GMMS2, ···, GMMSn, GMMT}) is generated first based on the source and target projects. Then, the JS divergence values are acquired based on equations (2) and (3). If the calculated JS divergence is the lowest, it denotes that the source project is the most similar to the target project.Figure 2
Selection process of the source project.


#### 3.1.2. Relative density estimation for instance selection

Since there are still noises and outliers in the source project, high-density instances are required to select. As shown in Figure3, the number of instances N is counted first. Second, the distance of each instance to its Kth nearest neighbor is calculated to estimate the relative density. Finally, the instances are sorted by percentage and then selected. The parameters of K and percent will be discussed in Section 5.Figure 3
Data selection in source project.


#### 3.1.3. One-to-One Classifier Training

Some HCPDP models are one-to-one predictions. They use one source project to construct the model and predict the target project. This process is not complex to understand. The learning algorithms can be selected to train the classifier based on the selected instances directly.


#### 3.1.4. Many-to-One Classifiers Training

Besides one-to-one prediction, many-to-one prediction methods aim to add training data by putting all the source projects together. In this paper, a dynamic ensemble voting strategy is proposed to construct classifiers based on JS divergence. As shown in Figure4, this paper trains the subclassifiers by the selected instances in similar source projects. Then, the percentage of the reciprocal of JS divergence is used as the dynamic voting weight for each subclassifier, that is, 1/JSi/1/JS1+1/JS2+…+1/JSnumbersofsourceprojects. By using this strategy, the model could guarantee that the selected source projects have the most significant weight, which can strengthen their performance.Figure 4
Many-to-one classifiers training.


### 3.2. Prediction Phase

For one-to-one prediction, we input the data of the target project into the trained classifier and obtain the prediction label. For many-to-one prediction, the target data is sent to each subclassifier, and then the prediction result is obtained by multiplying the voting strategy. Finally, the result of each classifier is added to get the prediction result. If the prediction result is higher than a threshold of 0.5 (Nam et al. [23]; Zhou et al. [33]; Wan et al. [34]), an instance is more likely to be highly defective.


## 3.1. Training Phase

### 3.1.1. JS divergence for Project Selection

Figure2 describes the project selection process based on JS divergence. The Gaussian mixture model (GMM = {GMMS1, GMMS2, ···, GMMSn, GMMT}) is generated first based on the source and target projects. Then, the JS divergence values are acquired based on equations (2) and (3). If the calculated JS divergence is the lowest, it denotes that the source project is the most similar to the target project.Figure 2
Selection process of the source project.


### 3.1.2. Relative density estimation for instance selection

Since there are still noises and outliers in the source project, high-density instances are required to select. As shown in Figure3, the number of instances N is counted first. Second, the distance of each instance to its Kth nearest neighbor is calculated to estimate the relative density. Finally, the instances are sorted by percentage and then selected. The parameters of K and percent will be discussed in Section 5.Figure 3
Data selection in source project.


### 3.1.3. One-to-One Classifier Training

Some HCPDP models are one-to-one predictions. They use one source project to construct the model and predict the target project. This process is not complex to understand. The learning algorithms can be selected to train the classifier based on the selected instances directly.


### 3.1.4. Many-to-One Classifiers Training

Besides one-to-one prediction, many-to-one prediction methods aim to add training data by putting all the source projects together. In this paper, a dynamic ensemble voting strategy is proposed to construct classifiers based on JS divergence. As shown in Figure4, this paper trains the subclassifiers by the selected instances in similar source projects. Then, the percentage of the reciprocal of JS divergence is used as the dynamic voting weight for each subclassifier, that is, 1/JSi/1/JS1+1/JS2+…+1/JSnumbersofsourceprojects. By using this strategy, the model could guarantee that the selected source projects have the most significant weight, which can strengthen their performance.Figure 4
Many-to-one classifiers training.


## 3.1.1. JS divergence for Project Selection

Figure2 describes the project selection process based on JS divergence. The Gaussian mixture model (GMM = {GMMS1, GMMS2, ···, GMMSn, GMMT}) is generated first based on the source and target projects. Then, the JS divergence values are acquired based on equations (2) and (3). If the calculated JS divergence is the lowest, it denotes that the source project is the most similar to the target project.Figure 2
Selection process of the source project.


## 3.1.2. Relative density estimation for instance selection

Since there are still noises and outliers in the source project, high-density instances are required to select. As shown in Figure3, the number of instances N is counted first. Second, the distance of each instance to its Kth nearest neighbor is calculated to estimate the relative density. Finally, the instances are sorted by percentage and then selected. The parameters of K and percent will be discussed in Section 5.Figure 3
Data selection in source project.


## 3.1.3. One-to-One Classifier Training

Some HCPDP models are one-to-one predictions. They use one source project to construct the model and predict the target project. This process is not complex to understand. The learning algorithms can be selected to train the classifier based on the selected instances directly.


## 3.1.4. Many-to-One Classifiers Training

Besides one-to-one prediction, many-to-one prediction methods aim to add training data by putting all the source projects together. In this paper, a dynamic ensemble voting strategy is proposed to construct classifiers based on JS divergence. As shown in Figure4, this paper trains the subclassifiers by the selected instances in similar source projects. Then, the percentage of the reciprocal of JS divergence is used as the dynamic voting weight for each subclassifier, that is, 1/JSi/1/JS1+1/JS2+…+1/JSnumbersofsourceprojects. By using this strategy, the model could guarantee that the selected source projects have the most significant weight, which can strengthen their performance.Figure 4
Many-to-one classifiers training.


## 3.2. Prediction Phase

For one-to-one prediction, we input the data of the target project into the trained classifier and obtain the prediction label. For many-to-one prediction, the target data is sent to each subclassifier, and then the prediction result is obtained by multiplying the voting strategy. Finally, the result of each classifier is added to get the prediction result. If the prediction result is higher than a threshold of 0.5 (Nam et al. [23]; Zhou et al. [33]; Wan et al. [34]), an instance is more likely to be highly defective.


## 4. Experiment Setup

### 4.1. Environment and Datasets

The experimental environment is a computer equipped with an Intel Xeon E3-1231 and 16 GB RAM, running Windows 10 (64 bit). Two benchmark datasets, NASA (Shepperd et al. [26]) and PROMISE (Jureczko and Madeyski [27]; Zhou et al. [33]; Xu et al. [35]), are collected. Table 1 shows the total number of instances and defects, and the percentage of defective instances in these datasets.Table 1
Properties of projects in two data sets.
DatasetProjectNumber of instancesNumber of defectsDefect%NASACM13344212.2KC1209532515.5KC32003618MC21254425.2MW12632710.3PC21493161.1PC4137917812.9PROMISEant1.774516622.3camel1.696518819.5ivy2.03524011.4jedit4.03067524.5log4j1.01353425.2lucene2.434020359.7poi3.044228163.6synapse1.22568633.6tomcat6.0858779.0velocity1.62297834.1xalan2.472311015.3xerces1.34536915.3


### 4.2. Evaluation Metrics

Three wide metrics are used in this study:


#### 4.2.1. F1-Score

Software defect prediction is recognized as a binary classification task. The final result may be True Positive (TP), which denotes the number of actually predicted defective modules; False Positive (FP) denotes the number of incorrectly predicted non-defective modules; True Negative (TN) represents the number of correctly predicted non-defective modules, and False Negative (FN) represents the number of incorrectly predicted defective modules. Based on the four results, precision is used to assess the correctness of a prediction model, Precision=TP/TP+FP, and recall is used to evaluate the possibility of correctly predicted defects, Recall=TP/TP+FN. In general, there is a trade-off between the two metrics. For example, by sacrificing precision, the recall value (Nam et al. [23]) may be improved. These trade-offs make it difficult to compare the performance of the prediction models using precision and recall (Kim et al. [36]; Xu et al. [35]; Wan et al. [34]). For this reason, this paper compares prediction results using F1-score values, that is, F1−score=2×Precision×Recall/Precision+Recall.


#### 4.2.2. AUC

AUC is used to estimate the area under the receiver operating characteristic curve, obtained by a set of (false positive rate and recall) pairs.


#### 4.2.3. G-Mean

G-mean can reflect the performance while the data is imbalanced,G−mean=TN/TN+FPTP/TP+FN.


## 4.1. Environment and Datasets

The experimental environment is a computer equipped with an Intel Xeon E3-1231 and 16 GB RAM, running Windows 10 (64 bit). Two benchmark datasets, NASA (Shepperd et al. [26]) and PROMISE (Jureczko and Madeyski [27]; Zhou et al. [33]; Xu et al. [35]), are collected. Table 1 shows the total number of instances and defects, and the percentage of defective instances in these datasets.Table 1
Properties of projects in two data sets.
DatasetProjectNumber of instancesNumber of defectsDefect%NASACM13344212.2KC1209532515.5KC32003618MC21254425.2MW12632710.3PC21493161.1PC4137917812.9PROMISEant1.774516622.3camel1.696518819.5ivy2.03524011.4jedit4.03067524.5log4j1.01353425.2lucene2.434020359.7poi3.044228163.6synapse1.22568633.6tomcat6.0858779.0velocity1.62297834.1xalan2.472311015.3xerces1.34536915.3


## 4.2. Evaluation Metrics

Three wide metrics are used in this study:


### 4.2.1. F1-Score

Software defect prediction is recognized as a binary classification task. The final result may be True Positive (TP), which denotes the number of actually predicted defective modules; False Positive (FP) denotes the number of incorrectly predicted non-defective modules; True Negative (TN) represents the number of correctly predicted non-defective modules, and False Negative (FN) represents the number of incorrectly predicted defective modules. Based on the four results, precision is used to assess the correctness of a prediction model, Precision=TP/TP+FP, and recall is used to evaluate the possibility of correctly predicted defects, Recall=TP/TP+FN. In general, there is a trade-off between the two metrics. For example, by sacrificing precision, the recall value (Nam et al. [23]) may be improved. These trade-offs make it difficult to compare the performance of the prediction models using precision and recall (Kim et al. [36]; Xu et al. [35]; Wan et al. [34]). For this reason, this paper compares prediction results using F1-score values, that is, F1−score=2×Precision×Recall/Precision+Recall.


### 4.2.2. AUC

AUC is used to estimate the area under the receiver operating characteristic curve, obtained by a set of (false positive rate and recall) pairs.


### 4.2.3. G-Mean

G-mean can reflect the performance while the data is imbalanced,G−mean=TN/TN+FPTP/TP+FN.


## 4.2.1. F1-Score

Software defect prediction is recognized as a binary classification task. The final result may be True Positive (TP), which denotes the number of actually predicted defective modules; False Positive (FP) denotes the number of incorrectly predicted non-defective modules; True Negative (TN) represents the number of correctly predicted non-defective modules, and False Negative (FN) represents the number of incorrectly predicted defective modules. Based on the four results, precision is used to assess the correctness of a prediction model, Precision=TP/TP+FP, and recall is used to evaluate the possibility of correctly predicted defects, Recall=TP/TP+FN. In general, there is a trade-off between the two metrics. For example, by sacrificing precision, the recall value (Nam et al. [23]) may be improved. These trade-offs make it difficult to compare the performance of the prediction models using precision and recall (Kim et al. [36]; Xu et al. [35]; Wan et al. [34]). For this reason, this paper compares prediction results using F1-score values, that is, F1−score=2×Precision×Recall/Precision+Recall.


## 4.2.2. AUC

AUC is used to estimate the area under the receiver operating characteristic curve, obtained by a set of (false positive rate and recall) pairs.


## 4.2.3. G-Mean

G-mean can reflect the performance while the data is imbalanced,G−mean=TN/TN+FPTP/TP+FN.


## 5. Results and Analysis

In this section, six one-to-one and seven many-to-one HCPDP methods are selected to compare against the proposed approach. Since all the compared methods use logistic regression (LR) (Fan et al. [37]) as the underlying classifier, LR is used to construct the classifier. All the methods were implemented by following their papers.


### 5.1. Six One-to-One Prediction Methods Are Described as Follows

(i)
ManualDown (Zhou et al. [33]). It is a simple unsupervised method, which performs better than most the existing works. ManualDown has been recognized as a new baseline method of CPDP.(ii)
Logistic regression (LR) (Fan et al. [37]). It only constructed an LR classifier to implement one-to-one defect prediction.(iii)
MT (Zhang et al. [38]). The authors proposed Multiple Transformations (MT) to improve prediction performance.(iv)
MT+ (Zhang et al. [38]). The authors improved MT to advance the prediction results.(v)
BDA (Xu et al. [36]). The authors proposed a balanced distribution adaptation (BDA) based transfer learning method to implement cross-project defect prediction.(vi)
CORAL (Sun et al. [38]). A method for domain adaptation was proposed by CORAL, which selected a similar source project with the target project by minimizing the distribution difference.


### 5.2. Seven Many-to-One Prediction Approaches Are Described as Follows

(i)
ManualDown (Zhou et al. [33]). A new baseline method is proposed by calculating the module size.(ii)
Logistic regression (Fan et al. [37]). The LR classifier was constructed to implement many-to-one defect prediction.(iii)
TCA+ (Nam et al. [23]). An algorithm was proposed to deal with the data pre-processing during the many-to-one prediction process.(iv)
TPTL (Liu et al. [24]). A source project estimator was given to select a similar source project and construct the two-phase prediction model.(v)
ISDA (Jing et al. [39]). The idea of subclass discriminant analysis (SDA) was introduced into cross-project defect prediction. The authors improved SDA to advance prediction performance.(vi)
TDS (Herbold et al. [18]). A method for training data selection was given, and the authors used Euclidean distance to measure the difference between the source and target projects.(vii)
DYCOM (Minku et al. [40]). It was a weighted sum of multiple pre-trained models from source projects and 10% of data from the target project.To check if the proposed approach over other methods was statistically significant, Friedman’s test (Friedman [41]) is a favorable statistical test to compare over two ways. At 5% significance level, Friedman’s test rejects the null hypothesis of “equal” performance among all the comparing methods. Then Nemenyi’s test (Demiar and Schuurmans [42]) is used to analyze which methods differ. Moreover, we also apply Cohen’s d (Cohen [43]) to calculate the effect size, which can quantify the difference among the methods. The calculation formula is shown in (5). Table 2 shows the effect grade corresponding to Cohen’s d.(5)Cohen′sd=M1−M2σ12+σ22/2.Table 2
Cohen’sd effectiveness level.
Cohen’s dEffectiveness leveld<0.2Negligible (N)0.2≤d<0.5Small (S)0.5≤d<0.8Medium (M)d≥0.8Large (L)


### 5.3. One-to-One Defect Prediction Results and Analysis

Firstly, Tables3 and 4 show the JS results of two projects on NASA and PROMISE. Since the JS result is smaller, the two projects are more similar. Then we can select the most similar source project for each target project. In the tables, the smallest JS value is boldface. For example, if CM1 is the target project, PC4 is selected as the source project.Table 3
The calculated JS values of two projects on NASA.
TargetSourceCM1KC1KC3MC2MW1PC2PC4CM100.68320.67950.67520.67820.68640.6418KC10.683200.68780.67650.68990.69090.6883KC30.67950.687800.61800.64990.68420.6696MC20.67520.67650.618000.68390.69280.6782MW10.67820.68990.64990.683900.67450.6835PC20.68640.69090.68420.69280.674500.6695PC40.64180.68830.66960.67820.68350.66950Table 4
The calculated JS values of two projects on PROMISE.
TargetSourceant1.7camel1.6ivy2.0jedit4.0log4j1.0lucene2.4poi3.0synapse1.2Tomcatvelocity1.6xalan2.4xerces1.3ant1.700.66990.68020.69290.68920.68390.68100.68710.69290.68900.66160.6858camel1.60.669900.68730.69280.68010.67310.68500.66320.69260.68200.63630.6908ivy2.00.68020.687300.69240.68410.66900.69190.68610.69300.68490.68350.6871jedit4.00.69290.69280.692400.69310.69160.69300.69310.69310.69300.69270.6911log4j1.00.68920.68010.68410.693100.67500.69150.67530.69290.67670.67750.6916lucene2.40.68390.67310.66900.69160.675000.68770.65690.69270.68730.67500.6898poi3.00.68100.68500.69190.69300.69150.687700.68730.69290.69210.69050.6913synapse1.20.68710.66320.68610.69310.67530.65690.687300.69270.69180.67020.6919Tomcat0.69290.69260.69300.69310.69290.69270.69290.692700.69310.69230.6931velocity1.60.68900.68200.68490.69300.67670.68730.69210.69180.693100.68260.6890xalan2.40.66160.63630.68350.69270.67750.67500.69050.67020.69230.682600.6880xerces1.30.68580.69080.68710.69110.69160.68980.69130.69190.69310.68900.68800After the source project is determined, instance selection could be completed by following relative density estimation. Then Tables5–7 present the experimental results (i.e., F1-score, AUC, and G-mean) of the proposed approach compared with the six baseline approaches on NASA and PROMISE. For each target project, the best results are highlighted in bold. The last row in the tables presents the improvement of our approach over other baseline approaches.Table 5
F1-score of the proposed approach and six one-to-one prediction methods on NASA and PROMISE.
TargetOursManualDownLRMTMT+BDACORALNASACM10.36090.30620.25510.19350.18800.29320.2549KC10.44510.41270.28470.19160.28310.40310.3526KC30.38300.34850.32540.22730.25450.38070.3810MC20.58820.52830.38670.25250.28570.46920.4214MW10.34430.26580.18330.13990.15240.32010.2446PC20.07320.03060.05500.02950.03220.07700.0571PC40.32860.30110.30690.11330.25540.29670.3015PROMISEant1.70.57140.53260.51020.34430.38310.52460.5100camel1.60.38490.34630.32920.28890.29400.34490.3273ivy2.00.34090.32710.35130.20100.21320.36660.3539jedit4.00.57450.49780.48970.36900.36670.51070.4931log4j1.00.53660.53470.52830.29360.32480.59560.5425lucene2.40.65060.62370.52250.67050.74210.57260.5437poi3.00.64270.70940.56100.69430.77730.53400.5774synapse1.20.59460.59430.56450.45660.47510.57820.5696Tomcat0.41790.26530.29380.16120.14560.32680.3058velocity1.60.50000.55210.43720.42230.48480.46050.4516xalan2.40.43450.41190.36250.28160.27170.39480.3669xerces1.30.40000.38780.39240.35450.42520.35680.3851Average0.45120.41980.37580.29920.33450.41080.3916Improvement3.1%7.5%15.2%11.7%4.0%6.0%Table 6
AUC of the proposed methods and six one-to-one prediction approaches on NASA and PROMISE.
TargetOursManualDownLRMTMT+BDACORALNASACM10.67480.65740.58270.51930.52500.61000.5824KC10.72250.71830.55470.51320.54030.65620.6254KC30.62800.59690.57230.51830.51150.62430.6202MC20.68530.60830.54120.52710.56210.61300.5665MW10.73210.65580.52370.54380.55750.68040.6075PC20.84040.60980.75920.56900.57930.79110.7698PC40.64600.63340.61480.55880.57230.60490.6095PROMISEant1.70.73380.73380.69610.51880.59460.70630.6959camel1.60.61680.57300.56630.51580.52090.58060.5654ivy2.00.70510.71470.69700.51430.53720.71380.7005jedit4.00.73230.66990.65790.52280.50130.67690.6612log4j1.00.69480.69900.68790.52090.53740.73910.7000lucene2.40.68940.59230.58240.50680.50370.63510.5995poi3.00.68920.68760.61360.51740.50000.62650.6335synapse1.20.69040.68100.66460.52070.52810.67930.6705Tomcat0.77720.70400.69470.50430.53200.73080.7117velocity1.60.62090.63780.57150.53810.60080.59570.5812xalan2.40.69900.72560.64330.55510.54160.67380.6477xerces1.30.66320.69430.66800.63420.72070.63930.6607Average0.69690.66280.62590.53260.55090.66200.6426Improvement3.4%7.1%16.4%14.6%3.5%5.4%Table 7
G-mean of the proposed approach and six one-to-one prediction methods on NASA and PROMISE.
TargetOursManualDownLRMTMT+BDACORALNASACM10.66680.64910.56120.46510.45430.58140.5627KC10.72250.70390.51140.33320.39610.62700.5947KC30.61490.59540.56050.45550.49660.60500.6083MC20.67520.60770.50780.37840.41400.57790.5382MW10.73070.64440.50620.42150.41620.67770.5966PC20.83480.60490.75520.53660.56170.79020.7654PC40.64300.62680.60170.31290.50210.57590.5942PROMISEant1.70.73380.73380.69610.51880.59460.70630.6959camel1.60.61680.57300.56630.51580.52090.58060.5654ivy2.00.70510.71470.69700.51430.53720.71380.7005jedit4.00.73230.66990.65790.52280.50130.67690.6612log4j1.00.69480.69900.68790.52090.53740.73910.7000lucene2.40.68940.59230.58240.50680.50370.63510.5995poi3.00.68920.68760.61360.51740.50000.62650.6335synapse1.20.69040.68100.66460.52070.52810.67930.6705Tomcat0.77720.70400.69470.50430.53200.73080.7117velocity1.60.62090.63780.57150.53810.60080.59570.5812xalan2.40.69900.72560.64330.55510.54160.67380.6477xerces1.30.66320.69430.66800.63420.72070.63930.6607Average0.69470.66030.61830.48800.51890.65430.6362Improvement3.5%7.7%20.7%17.6%4.0%5.9%Compared with the other six methods on two datasets, the one-to-one prediction method improved F1-score by 3.1%∼15.2% in Table5. In Table 6, AUC is improved by 3.4%∼16.4%, and G-mean is improved by 3.5%∼20.7% in Table 7. However, the authors ran the approach of MT+ on “poi3.0” many times, but the result was still 0. It may not be suitable to evaluate MT+, and the authors do not also consider this project in their paper.Next, Friedman’s and Nemenyi’s tests are used to analyze the performance. First, the correspondingp-values (all less than 0.05) under the Friedman test are given in Table 8, which refers to the differences among the seven methods from a global perspective. To further visually show the differences, CD diagrams were used in Figure 5. The average rank of each method is marked along the axis. This study connected them with a thick line if these approaches are not significantly different under the Nemenyi test. By investigating Figure 5 and Table 8, the proposed approach significantly improves the F1-score, AUC, and G-mean over all the other methods.Table 8
Thep-values under Friedman’s test among the seven methods.
Data setp-valueF1-scorep-valueAUCp-valueG-meanNASA7.99E-064.64E-062.10E-06PROMISE1.30E-046.12E-092.32E-08Figure 5
One-to-one comparison of average ranks in F1-score, AUC, and G-mean on two datasets. (a) NASA: F1-score. (b) NASA: AUC. (c) NASA: G-mean. (d) PROMISE: F1-score. (e) PROMISE: AUC. (f) PROMISE: G-mean.
(a)(b)(c)(d)(e)(f)In Table9, Cohen’s d is used to calculate effect size. The proposed one-to-one defect prediction obtained 14 “L” on NASA datasets, and 9 “L” on PROMISE datasets. It can be concluded that the proposed approach is significantly better than the other six one-to-one prediction methods.Table 9
Cohen’sd of our approach and six one-to-one prediction approaches on NASA and PROMISE.
DatasetMetricsManualdownLRMTMT+BDACORALNASAF1-score0.3316 (S)0.8382 (L)1.7424 (L)1.3012 (L)0.3113 (S)0.5680 (M)AUC1.1918 (L)1.6040 (L)3.4742 (L)3.1416 (L)0.7857 (M)1.2284 (L)G-mean1.2166 (L)1.6853 (L)4.0312 (L)3.8117 (L)0.9263 (L)1.3265 (L)PROMISEF1-score0.1913 (N)0.6076 (M)0.9499 (L)0.6422 (M)0.4063 (S)0.5259 (M)AUC0.3613 (S)1.0423 (L)4.1741 (L)2.7089 (L)0.5657 (M)0.8960 (L)G-mean0.3795 (S)0.9651 (L)2.6940 (L)1.7510 (L)0.5961 (M)0.8471 (L)Finally, the parameters of K and percent in Figure6 are analyzed. To decide the K value, we explore the parameter K based on the number of instances, and the range is K=⌈N/4⌉,⌈N/2⌉,⌈N⌉,⌈2N⌉,⌈4N⌉, where N is the number of instances. For the percent value, it represents the proportion chosen for sorting instances after relative density estimation, and its range was [0.1, 0.9]. Due to the space limitation, this paper takes CM1 in NASA and ant 1.7 in PROMISE as an example. Then the selected source projects based on JS values are PC4 and xalan2.4, respectively.Figure 6
The variance of the proposed approach performance with (K) and percent. (a) F1-score of PC4⇒CM1. (b) AUC of PC4⇒CM1. (c) G-mean of PC4⇒CM1. (d) F1-score of xalan2.4⇒ant1.7. (e) AUC of xalan2.4⇒ant1.7. (f) G-mean of xalan2.4⇒ant1.7.
(a)(b)(c)(d)(e)(f)Figure6 plots the variance of the performance with the variance of K and percent. On the X-axis, 0.1∼0.9 denotes the proportion of the selected sorting instances. On the Y-axis, 0.25∼4 indicates the choice of K, and the Z-axis indicates the values of F1-score, AUC, and G-mean, respectively. As investigated in Figures 6(a)–6(c), when K = 0.25, percent = 0.2, the F1-score, AUC, and G-mean of PC4CM1 were 0.3609, 0.6747, and 0.6668. In Figures 6(d)–6(f), when K = 4, percent = 0.7, the F1-score, AUC, and G-mean of xalan2.4 ant1.7 were 0.5714, 0.7338, 0.7289. Hence, the selection of K and percent are based on the choice of the project.


### 5.4. Many-to-One Defect Prediction Results and Analysis

Compared to one-to-one prediction, the proposed many-to-one prediction has the same process for source project selection and instance section. However, the exact number of source projects needed to discuss during the process of the many-to-one prediction, and the corresponding analysis will be given later.By the ensemble voting weighting strategy Tables10–12 show the F1-score, AUC, and G-mean of many-to-one prediction under the proposed approach compared with the seven many-to-one methods on NASA and PROMISE. The best results are still highlighted in bold, and the last row presents the improvement of the proposed approach over other approaches.Table 10
F1-score of the proposed approach and seven many-to-one prediction methods on two datasets.
TargetOursManualDownLRMTMT+BDACORALNASACM10.33570.32650.26360.31980.31210.37130.2222KC10.44710.27230.21540.39350.31740.34350.1991KC30.49410.32070.39580.32590.39320.44230.2449MC20.58140.30950.30000.44010.56880.60000.0000MW10.34150.33080.21850.29920.28210.28990.0588PC20.07850.34830.06060.06280.01320.07820.0625PC40.35350.34300.34400.20970.30300.25960.1494PROMISEant1.70.57420.48300.56730.54080.50700.49900.4990camel1.60.37740.49940.31030.32940.38620.38590.1017ivy2.00.43240.48970.39160.34480.31250.33140.4062jedit4.00.62150.48320.51500.49050.51850.42570.4567log4j1.00.67500.48260.59520.51670.54170.52750.3182lucene2.40.69230.46400.47300.52570.57140.61380.6322poi3.00.74740.43900.59020.52460.77130.80660.2310synapse1.20.67030.47810.58100.60690.60710.45500.3793Tomcat0.37180.51290.33850.30950.25900.21450.2136velocity1.60.55480.47990.50980.46540.50830.61450.1176xalan2.40.44690.49630.40850.40360.35730.28740.3290xerces1.30.52940.48790.40820.42960.39450.46360.4107Average0.49080.42350.39400.39680.41710.42160.2648Improvement6.7%9.7%9.4%7.4%6.9%22.6%Table 11
AUC of the proposed approach and seven many-to-one prediction methods on two datasets.
TargetOursManualDownLRMTMT+BDACORALNASACM10.65820.71650.58650.71680.69540.71340.5131KC10.71550.70970.53810.73600.67560.63160.5746KC30.70630.70890.63890.59920.63060.68220.5183MC20.67920.70310.51180.61300.67480.69870.6622MW10.73000.71320.57340.75210.73490.67760.5403PC20.84990.65230.81170.89660.70430.87240.6648PC40.66550.74690.65220.52050.65110.57640.5316PROMISEant1.70.74570.65490.74140.78640.65560.69520.7574camel1.60.61060.66400.55260.58420.63120.61710.5561ivy2.00.77820.65560.72980.70650.67940.68810.7683jedit4.00.76490.65850.68060.70500.66560.59140.6885log4j1.00.80300.65800.74390.76160.66220.68960.7312lucene2.40.72250.66570.58850.64940.53890.57040.6772poi3.00.74930.65320.66210.73490.69900.75040.7432synapse1.20.75160.65820.68170.77250.74760.57350.7201Tomcat0.76330.65960.73360.75530.74660.59930.7232velocity1.60.66310.65970.63080.64110.50330.70020.6825xalan2.40.72660.65730.68140.75190.66550.56740.7078xerces1.30.78580.65450.67660.75530.78240.73940.7834Average0.73000.67630.65350.70730.67070.66500.6602Improvement5.4%7.7%2.3%5.9%6.5%7.0%Table 12
G-mean of the proposed approach and seven many-to-one prediction methods on two datasets.
TargetOursManualDownLRMTMT+BDACORALNASACM10.65250.69840.55760.44940.43770.71300.4876KC10.71480.69300.42350.51040.43250.62840.6096KC30.69550.69580.62920.46280.50290.68090.6225MC20.67000.68910.43810.53930.61120.68160.0000MW10.72840.70080.56600.42250.40520.67460.3583PC20.84540.64610.80190.17960.08150.86300.2487PC40.66120.73060.64330.35160.42500.56240.5010PROMISEant1.70.74490.65020.74080.62580.58640.69270.5833camel1.60.60320.65910.53900.45710.48590.61670.4495ivy2.00.77790.65140.72920.53180.43300.68800.7050jedit4.00.76430.65370.67210.60270.59280.58010.6758log4j1.00.80300.65330.74380.60360.61110.68940.7408lucene2.40.70760.65940.53570.60000.57660.57040.6299poi3.00.74240.64720.62670.61370.71510.74950.5508synapse1.20.75100.65310.67740.67970.64620.56880.7250Tomcat0.76320.65570.73330.43230.38560.59790.6241velocity1.60.65360.65450.61710.56810.57720.70020.6924xalan2.40.72660.65310.67750.51720.46620.56200.4429xerces1.30.78580.65020.66960.54760.49570.73940.5233Average0.72590.66810.63270.51030.49830.66100.5353Improvement5.8%9.3%21.6%22.8%6.5%19.1%According to the results in Table10, the proposed many-to-one prediction improved the F1-score by 6.7%∼22.6%. In Table 11, AUC is improved by 2.3%∼7.7%, and G-mean is improved by 5.8%∼22.8% in Table 12 on NASA and PROMISE. In these tables, some results are still 0, the relevant project may not be suitable to evaluate, and the authors do not consider the projects in their papers.Table13 gives the p-values (all less than 0.05), which indicate that all the methods have significant performance differences between each other. Moreover, the CD diagrams in Figure 7 show that the proposed approach ranks first in all the evaluation metrics, except ranking second in G-mean on NASA and AUC on PROMISE. As a whole, many-to-prediction under the proposed approach significantly improved the F1-score, AUC, and G-mean over other many-to-one methods.Table 13
Thep-values under Friedman test of our approach and seven many-to-one prediction approaches.
Data setp-valueF1-scorep-valueAUCp-valueG-meanNASA2.38E-043.31E-028.94E-05PROMISE1.30E-045.21E-052.56E-06Figure 7
Many-to-one comparison of average ranks in F1-score, AUC, and G-mean on two datasets. (a) NASA: F1-score. (b) NASA: AUC. (c) NASA: G-mean. (d) PROMISE: F1-score. (e) PROMISE: AUC. (f) PROMISE: G-mean.
(a)(b)(c)(d)(e)(f)Similarly, Cohen’sd is used to evaluate effect size. The analysis results can be seen in Table 14 and the proposed many-to-one defect prediction obtained 10 “L” on NASA datasets, and 16 “L” on PROMISE datasets. The proposed approach is significantly better than the other seven many-to-one prediction methods, and the difference cannot be ignored.Table 14
Cohen’sd of our approach and seven many-to-one prediction approaches on NASA dataset.
DatasetMetricsManualDownLRMTMT+BDACORALNASAF1-score0.5137 (M)0.9428 (L)0.6253 (M)0.4204 (S)0.2374 (S)1.9947 (L)AUC0.1657 (N)1.2675 (L)0.2657 (S)0.7041 (M)0.2958 (S)2.3613 (L)G-mean0.3527 (S)1.3615 (L)3.2539 (L)2.5772 (L)0.3157 (S)2.0158 (L)PROMISEF1-score0.8531 (L)0.7585 (M)0.9331 (L)0.6151 (M)0.6386 (M)1.5802 (L)AUC2.1955 (L)1.1560 (L)0.3895 (S)1.1235 (L)1.5271 (L)0.4996 (S)G-mean2.1127 (L)1.1506 (L)2.7476 (L)2.4973 (L)1.4508 (L)1.5486 (L)Finally, Figure8 illustrates the variance of the evaluation metrics with the numbers of the source projects. It can be found that when the number of NASA source projects is 3 and the number of PROMISE source projects are 7, the result is optimal. For example, if “CM1” is the target project, the selected source projects are PC4, MC2, and MW1.Figure 8
The variance of the evaluation metrics with the numbers of the source projects. (a) F1-score of NASA. (b) AUC of NASA. (c) G-mean of NASA. (d) F1-score of PROMISE. (e) AUC of PROMISE. (f) G-mean of PROMISE.
(a)(b)(c)(d)(e)(f)


### 5.5. Different Learning Methods Analysis

This section aims to prove that the proposed is not subject to machine learning methods, that is, the prediction performance can be improved regardless of which learning method is selected. We first select four traditional learning methods that are often applied in CPDP, and they are LR, SVM, KNN, RF, and NB, respectively.Since AUC can reveal the classification performance, we take it as an example. Figure9 shows the comparison results, and it can be seen that AUC is improved regardless of which type of learning method is used to construct the classifier.Figure 9
Learning methods comparison results in AUC on two datasets. (a) NASA. (b) PROMISE.
(a)(b)To check the difference in learning methods, Figure10 graphically visualizes the average comparison results of the proposed approach with five learning methods on two datasets. When the classifier changes, the proposed approach still works well. It proves that the proposed approach has more robust adaptability compared to other learning methods. However, we cannot easily judge which learning method is better because it is affected by the datasets.Figure 10
Comparison of average ranks in F1-score on NASA and PROMISE. (a) One-to-one prediction on NASA. (b) Many-to-one prediction on NASA. (c) One-to-one prediction on PROMISE. (d) Many-to-one prediction on PROMISE.
(a)(b)(c)(d)


### 5.6. Discussion about the Time-Efficiency

To illustrate whether the running time is acceptable, Tables15 and 16 give the average running time of our approach and other approaches. Compared with TPTL and ISDA, the proposed approach needs less running time, but more than the other methods.Table 15
Average running time of one-to-one approaches on NASA and PROMISE.
Data setRunning time of approach (second)OursLRMTMT+BDACoralNASA228.64.429.536.270.55.0PROMISE528.27.758.279.2104.89.6Table 16
Average running time of many-to-one approaches on NASA and PROMISE.
Data setRunning time of approach (second)OursLRTCA+TPTLISDALtTDSDycomNASA524.22.78.12420.612495.70.20.24.1PROMISE924.01.714.11395.218830.40.40.35.9It is necessary to explain why our approach needs more time. Four components are contained in our approach, which includes JS calculation for source project selection, relative density estimation for instance selection, classifier building, and prediction. The running time of classifier building and prediction is less, and they can be ignored. In Table17, it can be seen that more time is used for JS calculation and the relative density estimation. As relative density estimation is based on the selected project, JS calculation is the main reason our approach needs more time. However, the calculation process does not need to be updated all the time. Even if a new project is added to the datasets, we only need to calculate the JS values of the new project with other projects, instead of recalculating all the JS values.Table 17
Average running time of JS calculation and relative density estimation on NASA and PROMISE.
ComponentOne-to-oneMany-to-oneNASAPROMISENASAPROMISEJS calculation163.7483.2163.7483.2Relative density estimation64.136.2359.8440.3Total227.8519.4523.5923.5


## 5.1. Six One-to-One Prediction Methods Are Described as Follows

(i)
ManualDown (Zhou et al. [33]). It is a simple unsupervised method, which performs better than most the existing works. ManualDown has been recognized as a new baseline method of CPDP.(ii)
Logistic regression (LR) (Fan et al. [37]). It only constructed an LR classifier to implement one-to-one defect prediction.(iii)
MT (Zhang et al. [38]). The authors proposed Multiple Transformations (MT) to improve prediction performance.(iv)
MT+ (Zhang et al. [38]). The authors improved MT to advance the prediction results.(v)
BDA (Xu et al. [36]). The authors proposed a balanced distribution adaptation (BDA) based transfer learning method to implement cross-project defect prediction.(vi)
CORAL (Sun et al. [38]). A method for domain adaptation was proposed by CORAL, which selected a similar source project with the target project by minimizing the distribution difference.


## 5.2. Seven Many-to-One Prediction Approaches Are Described as Follows

(i)
ManualDown (Zhou et al. [33]). A new baseline method is proposed by calculating the module size.(ii)
Logistic regression (Fan et al. [37]). The LR classifier was constructed to implement many-to-one defect prediction.(iii)
TCA+ (Nam et al. [23]). An algorithm was proposed to deal with the data pre-processing during the many-to-one prediction process.(iv)
TPTL (Liu et al. [24]). A source project estimator was given to select a similar source project and construct the two-phase prediction model.(v)
ISDA (Jing et al. [39]). The idea of subclass discriminant analysis (SDA) was introduced into cross-project defect prediction. The authors improved SDA to advance prediction performance.(vi)
TDS (Herbold et al. [18]). A method for training data selection was given, and the authors used Euclidean distance to measure the difference between the source and target projects.(vii)
DYCOM (Minku et al. [40]). It was a weighted sum of multiple pre-trained models from source projects and 10% of data from the target project.To check if the proposed approach over other methods was statistically significant, Friedman’s test (Friedman [41]) is a favorable statistical test to compare over two ways. At 5% significance level, Friedman’s test rejects the null hypothesis of “equal” performance among all the comparing methods. Then Nemenyi’s test (Demiar and Schuurmans [42]) is used to analyze which methods differ. Moreover, we also apply Cohen’s d (Cohen [43]) to calculate the effect size, which can quantify the difference among the methods. The calculation formula is shown in (5). Table 2 shows the effect grade corresponding to Cohen’s d.(5)Cohen′sd=M1−M2σ12+σ22/2.Table 2
Cohen’sd effectiveness level.
Cohen’s dEffectiveness leveld<0.2Negligible (N)0.2≤d<0.5Small (S)0.5≤d<0.8Medium (M)d≥0.8Large (L)


## 5.3. One-to-One Defect Prediction Results and Analysis

Firstly, Tables3 and 4 show the JS results of two projects on NASA and PROMISE. Since the JS result is smaller, the two projects are more similar. Then we can select the most similar source project for each target project. In the tables, the smallest JS value is boldface. For example, if CM1 is the target project, PC4 is selected as the source project.Table 3
The calculated JS values of two projects on NASA.
TargetSourceCM1KC1KC3MC2MW1PC2PC4CM100.68320.67950.67520.67820.68640.6418KC10.683200.68780.67650.68990.69090.6883KC30.67950.687800.61800.64990.68420.6696MC20.67520.67650.618000.68390.69280.6782MW10.67820.68990.64990.683900.67450.6835PC20.68640.69090.68420.69280.674500.6695PC40.64180.68830.66960.67820.68350.66950Table 4
The calculated JS values of two projects on PROMISE.
TargetSourceant1.7camel1.6ivy2.0jedit4.0log4j1.0lucene2.4poi3.0synapse1.2Tomcatvelocity1.6xalan2.4xerces1.3ant1.700.66990.68020.69290.68920.68390.68100.68710.69290.68900.66160.6858camel1.60.669900.68730.69280.68010.67310.68500.66320.69260.68200.63630.6908ivy2.00.68020.687300.69240.68410.66900.69190.68610.69300.68490.68350.6871jedit4.00.69290.69280.692400.69310.69160.69300.69310.69310.69300.69270.6911log4j1.00.68920.68010.68410.693100.67500.69150.67530.69290.67670.67750.6916lucene2.40.68390.67310.66900.69160.675000.68770.65690.69270.68730.67500.6898poi3.00.68100.68500.69190.69300.69150.687700.68730.69290.69210.69050.6913synapse1.20.68710.66320.68610.69310.67530.65690.687300.69270.69180.67020.6919Tomcat0.69290.69260.69300.69310.69290.69270.69290.692700.69310.69230.6931velocity1.60.68900.68200.68490.69300.67670.68730.69210.69180.693100.68260.6890xalan2.40.66160.63630.68350.69270.67750.67500.69050.67020.69230.682600.6880xerces1.30.68580.69080.68710.69110.69160.68980.69130.69190.69310.68900.68800After the source project is determined, instance selection could be completed by following relative density estimation. Then Tables5–7 present the experimental results (i.e., F1-score, AUC, and G-mean) of the proposed approach compared with the six baseline approaches on NASA and PROMISE. For each target project, the best results are highlighted in bold. The last row in the tables presents the improvement of our approach over other baseline approaches.Table 5
F1-score of the proposed approach and six one-to-one prediction methods on NASA and PROMISE.
TargetOursManualDownLRMTMT+BDACORALNASACM10.36090.30620.25510.19350.18800.29320.2549KC10.44510.41270.28470.19160.28310.40310.3526KC30.38300.34850.32540.22730.25450.38070.3810MC20.58820.52830.38670.25250.28570.46920.4214MW10.34430.26580.18330.13990.15240.32010.2446PC20.07320.03060.05500.02950.03220.07700.0571PC40.32860.30110.30690.11330.25540.29670.3015PROMISEant1.70.57140.53260.51020.34430.38310.52460.5100camel1.60.38490.34630.32920.28890.29400.34490.3273ivy2.00.34090.32710.35130.20100.21320.36660.3539jedit4.00.57450.49780.48970.36900.36670.51070.4931log4j1.00.53660.53470.52830.29360.32480.59560.5425lucene2.40.65060.62370.52250.67050.74210.57260.5437poi3.00.64270.70940.56100.69430.77730.53400.5774synapse1.20.59460.59430.56450.45660.47510.57820.5696Tomcat0.41790.26530.29380.16120.14560.32680.3058velocity1.60.50000.55210.43720.42230.48480.46050.4516xalan2.40.43450.41190.36250.28160.27170.39480.3669xerces1.30.40000.38780.39240.35450.42520.35680.3851Average0.45120.41980.37580.29920.33450.41080.3916Improvement3.1%7.5%15.2%11.7%4.0%6.0%Table 6
AUC of the proposed methods and six one-to-one prediction approaches on NASA and PROMISE.
TargetOursManualDownLRMTMT+BDACORALNASACM10.67480.65740.58270.51930.52500.61000.5824KC10.72250.71830.55470.51320.54030.65620.6254KC30.62800.59690.57230.51830.51150.62430.6202MC20.68530.60830.54120.52710.56210.61300.5665MW10.73210.65580.52370.54380.55750.68040.6075PC20.84040.60980.75920.56900.57930.79110.7698PC40.64600.63340.61480.55880.57230.60490.6095PROMISEant1.70.73380.73380.69610.51880.59460.70630.6959camel1.60.61680.57300.56630.51580.52090.58060.5654ivy2.00.70510.71470.69700.51430.53720.71380.7005jedit4.00.73230.66990.65790.52280.50130.67690.6612log4j1.00.69480.69900.68790.52090.53740.73910.7000lucene2.40.68940.59230.58240.50680.50370.63510.5995poi3.00.68920.68760.61360.51740.50000.62650.6335synapse1.20.69040.68100.66460.52070.52810.67930.6705Tomcat0.77720.70400.69470.50430.53200.73080.7117velocity1.60.62090.63780.57150.53810.60080.59570.5812xalan2.40.69900.72560.64330.55510.54160.67380.6477xerces1.30.66320.69430.66800.63420.72070.63930.6607Average0.69690.66280.62590.53260.55090.66200.6426Improvement3.4%7.1%16.4%14.6%3.5%5.4%Table 7
G-mean of the proposed approach and six one-to-one prediction methods on NASA and PROMISE.
TargetOursManualDownLRMTMT+BDACORALNASACM10.66680.64910.56120.46510.45430.58140.5627KC10.72250.70390.51140.33320.39610.62700.5947KC30.61490.59540.56050.45550.49660.60500.6083MC20.67520.60770.50780.37840.41400.57790.5382MW10.73070.64440.50620.42150.41620.67770.5966PC20.83480.60490.75520.53660.56170.79020.7654PC40.64300.62680.60170.31290.50210.57590.5942PROMISEant1.70.73380.73380.69610.51880.59460.70630.6959camel1.60.61680.57300.56630.51580.52090.58060.5654ivy2.00.70510.71470.69700.51430.53720.71380.7005jedit4.00.73230.66990.65790.52280.50130.67690.6612log4j1.00.69480.69900.68790.52090.53740.73910.7000lucene2.40.68940.59230.58240.50680.50370.63510.5995poi3.00.68920.68760.61360.51740.50000.62650.6335synapse1.20.69040.68100.66460.52070.52810.67930.6705Tomcat0.77720.70400.69470.50430.53200.73080.7117velocity1.60.62090.63780.57150.53810.60080.59570.5812xalan2.40.69900.72560.64330.55510.54160.67380.6477xerces1.30.66320.69430.66800.63420.72070.63930.6607Average0.69470.66030.61830.48800.51890.65430.6362Improvement3.5%7.7%20.7%17.6%4.0%5.9%Compared with the other six methods on two datasets, the one-to-one prediction method improved F1-score by 3.1%∼15.2% in Table5. In Table 6, AUC is improved by 3.4%∼16.4%, and G-mean is improved by 3.5%∼20.7% in Table 7. However, the authors ran the approach of MT+ on “poi3.0” many times, but the result was still 0. It may not be suitable to evaluate MT+, and the authors do not also consider this project in their paper.Next, Friedman’s and Nemenyi’s tests are used to analyze the performance. First, the correspondingp-values (all less than 0.05) under the Friedman test are given in Table 8, which refers to the differences among the seven methods from a global perspective. To further visually show the differences, CD diagrams were used in Figure 5. The average rank of each method is marked along the axis. This study connected them with a thick line if these approaches are not significantly different under the Nemenyi test. By investigating Figure 5 and Table 8, the proposed approach significantly improves the F1-score, AUC, and G-mean over all the other methods.Table 8
Thep-values under Friedman’s test among the seven methods.
Data setp-valueF1-scorep-valueAUCp-valueG-meanNASA7.99E-064.64E-062.10E-06PROMISE1.30E-046.12E-092.32E-08Figure 5
One-to-one comparison of average ranks in F1-score, AUC, and G-mean on two datasets. (a) NASA: F1-score. (b) NASA: AUC. (c) NASA: G-mean. (d) PROMISE: F1-score. (e) PROMISE: AUC. (f) PROMISE: G-mean.
(a)(b)(c)(d)(e)(f)In Table9, Cohen’s d is used to calculate effect size. The proposed one-to-one defect prediction obtained 14 “L” on NASA datasets, and 9 “L” on PROMISE datasets. It can be concluded that the proposed approach is significantly better than the other six one-to-one prediction methods.Table 9
Cohen’sd of our approach and six one-to-one prediction approaches on NASA and PROMISE.
DatasetMetricsManualdownLRMTMT+BDACORALNASAF1-score0.3316 (S)0.8382 (L)1.7424 (L)1.3012 (L)0.3113 (S)0.5680 (M)AUC1.1918 (L)1.6040 (L)3.4742 (L)3.1416 (L)0.7857 (M)1.2284 (L)G-mean1.2166 (L)1.6853 (L)4.0312 (L)3.8117 (L)0.9263 (L)1.3265 (L)PROMISEF1-score0.1913 (N)0.6076 (M)0.9499 (L)0.6422 (M)0.4063 (S)0.5259 (M)AUC0.3613 (S)1.0423 (L)4.1741 (L)2.7089 (L)0.5657 (M)0.8960 (L)G-mean0.3795 (S)0.9651 (L)2.6940 (L)1.7510 (L)0.5961 (M)0.8471 (L)Finally, the parameters of K and percent in Figure6 are analyzed. To decide the K value, we explore the parameter K based on the number of instances, and the range is K=⌈N/4⌉,⌈N/2⌉,⌈N⌉,⌈2N⌉,⌈4N⌉, where N is the number of instances. For the percent value, it represents the proportion chosen for sorting instances after relative density estimation, and its range was [0.1, 0.9]. Due to the space limitation, this paper takes CM1 in NASA and ant 1.7 in PROMISE as an example. Then the selected source projects based on JS values are PC4 and xalan2.4, respectively.Figure 6
The variance of the proposed approach performance with (K) and percent. (a) F1-score of PC4⇒CM1. (b) AUC of PC4⇒CM1. (c) G-mean of PC4⇒CM1. (d) F1-score of xalan2.4⇒ant1.7. (e) AUC of xalan2.4⇒ant1.7. (f) G-mean of xalan2.4⇒ant1.7.
(a)(b)(c)(d)(e)(f)Figure6 plots the variance of the performance with the variance of K and percent. On the X-axis, 0.1∼0.9 denotes the proportion of the selected sorting instances. On the Y-axis, 0.25∼4 indicates the choice of K, and the Z-axis indicates the values of F1-score, AUC, and G-mean, respectively. As investigated in Figures 6(a)–6(c), when K = 0.25, percent = 0.2, the F1-score, AUC, and G-mean of PC4CM1 were 0.3609, 0.6747, and 0.6668. In Figures 6(d)–6(f), when K = 4, percent = 0.7, the F1-score, AUC, and G-mean of xalan2.4 ant1.7 were 0.5714, 0.7338, 0.7289. Hence, the selection of K and percent are based on the choice of the project.


## 5.4. Many-to-One Defect Prediction Results and Analysis

Compared to one-to-one prediction, the proposed many-to-one prediction has the same process for source project selection and instance section. However, the exact number of source projects needed to discuss during the process of the many-to-one prediction, and the corresponding analysis will be given later.By the ensemble voting weighting strategy Tables10–12 show the F1-score, AUC, and G-mean of many-to-one prediction under the proposed approach compared with the seven many-to-one methods on NASA and PROMISE. The best results are still highlighted in bold, and the last row presents the improvement of the proposed approach over other approaches.Table 10
F1-score of the proposed approach and seven many-to-one prediction methods on two datasets.
TargetOursManualDownLRMTMT+BDACORALNASACM10.33570.32650.26360.31980.31210.37130.2222KC10.44710.27230.21540.39350.31740.34350.1991KC30.49410.32070.39580.32590.39320.44230.2449MC20.58140.30950.30000.44010.56880.60000.0000MW10.34150.33080.21850.29920.28210.28990.0588PC20.07850.34830.06060.06280.01320.07820.0625PC40.35350.34300.34400.20970.30300.25960.1494PROMISEant1.70.57420.48300.56730.54080.50700.49900.4990camel1.60.37740.49940.31030.32940.38620.38590.1017ivy2.00.43240.48970.39160.34480.31250.33140.4062jedit4.00.62150.48320.51500.49050.51850.42570.4567log4j1.00.67500.48260.59520.51670.54170.52750.3182lucene2.40.69230.46400.47300.52570.57140.61380.6322poi3.00.74740.43900.59020.52460.77130.80660.2310synapse1.20.67030.47810.58100.60690.60710.45500.3793Tomcat0.37180.51290.33850.30950.25900.21450.2136velocity1.60.55480.47990.50980.46540.50830.61450.1176xalan2.40.44690.49630.40850.40360.35730.28740.3290xerces1.30.52940.48790.40820.42960.39450.46360.4107Average0.49080.42350.39400.39680.41710.42160.2648Improvement6.7%9.7%9.4%7.4%6.9%22.6%Table 11
AUC of the proposed approach and seven many-to-one prediction methods on two datasets.
TargetOursManualDownLRMTMT+BDACORALNASACM10.65820.71650.58650.71680.69540.71340.5131KC10.71550.70970.53810.73600.67560.63160.5746KC30.70630.70890.63890.59920.63060.68220.5183MC20.67920.70310.51180.61300.67480.69870.6622MW10.73000.71320.57340.75210.73490.67760.5403PC20.84990.65230.81170.89660.70430.87240.6648PC40.66550.74690.65220.52050.65110.57640.5316PROMISEant1.70.74570.65490.74140.78640.65560.69520.7574camel1.60.61060.66400.55260.58420.63120.61710.5561ivy2.00.77820.65560.72980.70650.67940.68810.7683jedit4.00.76490.65850.68060.70500.66560.59140.6885log4j1.00.80300.65800.74390.76160.66220.68960.7312lucene2.40.72250.66570.58850.64940.53890.57040.6772poi3.00.74930.65320.66210.73490.69900.75040.7432synapse1.20.75160.65820.68170.77250.74760.57350.7201Tomcat0.76330.65960.73360.75530.74660.59930.7232velocity1.60.66310.65970.63080.64110.50330.70020.6825xalan2.40.72660.65730.68140.75190.66550.56740.7078xerces1.30.78580.65450.67660.75530.78240.73940.7834Average0.73000.67630.65350.70730.67070.66500.6602Improvement5.4%7.7%2.3%5.9%6.5%7.0%Table 12
G-mean of the proposed approach and seven many-to-one prediction methods on two datasets.
TargetOursManualDownLRMTMT+BDACORALNASACM10.65250.69840.55760.44940.43770.71300.4876KC10.71480.69300.42350.51040.43250.62840.6096KC30.69550.69580.62920.46280.50290.68090.6225MC20.67000.68910.43810.53930.61120.68160.0000MW10.72840.70080.56600.42250.40520.67460.3583PC20.84540.64610.80190.17960.08150.86300.2487PC40.66120.73060.64330.35160.42500.56240.5010PROMISEant1.70.74490.65020.74080.62580.58640.69270.5833camel1.60.60320.65910.53900.45710.48590.61670.4495ivy2.00.77790.65140.72920.53180.43300.68800.7050jedit4.00.76430.65370.67210.60270.59280.58010.6758log4j1.00.80300.65330.74380.60360.61110.68940.7408lucene2.40.70760.65940.53570.60000.57660.57040.6299poi3.00.74240.64720.62670.61370.71510.74950.5508synapse1.20.75100.65310.67740.67970.64620.56880.7250Tomcat0.76320.65570.73330.43230.38560.59790.6241velocity1.60.65360.65450.61710.56810.57720.70020.6924xalan2.40.72660.65310.67750.51720.46620.56200.4429xerces1.30.78580.65020.66960.54760.49570.73940.5233Average0.72590.66810.63270.51030.49830.66100.5353Improvement5.8%9.3%21.6%22.8%6.5%19.1%According to the results in Table10, the proposed many-to-one prediction improved the F1-score by 6.7%∼22.6%. In Table 11, AUC is improved by 2.3%∼7.7%, and G-mean is improved by 5.8%∼22.8% in Table 12 on NASA and PROMISE. In these tables, some results are still 0, the relevant project may not be suitable to evaluate, and the authors do not consider the projects in their papers.Table13 gives the p-values (all less than 0.05), which indicate that all the methods have significant performance differences between each other. Moreover, the CD diagrams in Figure 7 show that the proposed approach ranks first in all the evaluation metrics, except ranking second in G-mean on NASA and AUC on PROMISE. As a whole, many-to-prediction under the proposed approach significantly improved the F1-score, AUC, and G-mean over other many-to-one methods.Table 13
Thep-values under Friedman test of our approach and seven many-to-one prediction approaches.
Data setp-valueF1-scorep-valueAUCp-valueG-meanNASA2.38E-043.31E-028.94E-05PROMISE1.30E-045.21E-052.56E-06Figure 7
Many-to-one comparison of average ranks in F1-score, AUC, and G-mean on two datasets. (a) NASA: F1-score. (b) NASA: AUC. (c) NASA: G-mean. (d) PROMISE: F1-score. (e) PROMISE: AUC. (f) PROMISE: G-mean.
(a)(b)(c)(d)(e)(f)Similarly, Cohen’sd is used to evaluate effect size. The analysis results can be seen in Table 14 and the proposed many-to-one defect prediction obtained 10 “L” on NASA datasets, and 16 “L” on PROMISE datasets. The proposed approach is significantly better than the other seven many-to-one prediction methods, and the difference cannot be ignored.Table 14
Cohen’sd of our approach and seven many-to-one prediction approaches on NASA dataset.
DatasetMetricsManualDownLRMTMT+BDACORALNASAF1-score0.5137 (M)0.9428 (L)0.6253 (M)0.4204 (S)0.2374 (S)1.9947 (L)AUC0.1657 (N)1.2675 (L)0.2657 (S)0.7041 (M)0.2958 (S)2.3613 (L)G-mean0.3527 (S)1.3615 (L)3.2539 (L)2.5772 (L)0.3157 (S)2.0158 (L)PROMISEF1-score0.8531 (L)0.7585 (M)0.9331 (L)0.6151 (M)0.6386 (M)1.5802 (L)AUC2.1955 (L)1.1560 (L)0.3895 (S)1.1235 (L)1.5271 (L)0.4996 (S)G-mean2.1127 (L)1.1506 (L)2.7476 (L)2.4973 (L)1.4508 (L)1.5486 (L)Finally, Figure8 illustrates the variance of the evaluation metrics with the numbers of the source projects. It can be found that when the number of NASA source projects is 3 and the number of PROMISE source projects are 7, the result is optimal. For example, if “CM1” is the target project, the selected source projects are PC4, MC2, and MW1.Figure 8
The variance of the evaluation metrics with the numbers of the source projects. (a) F1-score of NASA. (b) AUC of NASA. (c) G-mean of NASA. (d) F1-score of PROMISE. (e) AUC of PROMISE. (f) G-mean of PROMISE.
(a)(b)(c)(d)(e)(f)


## 5.5. Different Learning Methods Analysis

This section aims to prove that the proposed is not subject to machine learning methods, that is, the prediction performance can be improved regardless of which learning method is selected. We first select four traditional learning methods that are often applied in CPDP, and they are LR, SVM, KNN, RF, and NB, respectively.Since AUC can reveal the classification performance, we take it as an example. Figure9 shows the comparison results, and it can be seen that AUC is improved regardless of which type of learning method is used to construct the classifier.Figure 9
Learning methods comparison results in AUC on two datasets. (a) NASA. (b) PROMISE.
(a)(b)To check the difference in learning methods, Figure10 graphically visualizes the average comparison results of the proposed approach with five learning methods on two datasets. When the classifier changes, the proposed approach still works well. It proves that the proposed approach has more robust adaptability compared to other learning methods. However, we cannot easily judge which learning method is better because it is affected by the datasets.Figure 10
Comparison of average ranks in F1-score on NASA and PROMISE. (a) One-to-one prediction on NASA. (b) Many-to-one prediction on NASA. (c) One-to-one prediction on PROMISE. (d) Many-to-one prediction on PROMISE.
(a)(b)(c)(d)


## 5.6. Discussion about the Time-Efficiency

To illustrate whether the running time is acceptable, Tables15 and 16 give the average running time of our approach and other approaches. Compared with TPTL and ISDA, the proposed approach needs less running time, but more than the other methods.Table 15
Average running time of one-to-one approaches on NASA and PROMISE.
Data setRunning time of approach (second)OursLRMTMT+BDACoralNASA228.64.429.536.270.55.0PROMISE528.27.758.279.2104.89.6Table 16
Average running time of many-to-one approaches on NASA and PROMISE.
Data setRunning time of approach (second)OursLRTCA+TPTLISDALtTDSDycomNASA524.22.78.12420.612495.70.20.24.1PROMISE924.01.714.11395.218830.40.40.35.9It is necessary to explain why our approach needs more time. Four components are contained in our approach, which includes JS calculation for source project selection, relative density estimation for instance selection, classifier building, and prediction. The running time of classifier building and prediction is less, and they can be ignored. In Table17, it can be seen that more time is used for JS calculation and the relative density estimation. As relative density estimation is based on the selected project, JS calculation is the main reason our approach needs more time. However, the calculation process does not need to be updated all the time. Even if a new project is added to the datasets, we only need to calculate the JS values of the new project with other projects, instead of recalculating all the JS values.Table 17
Average running time of JS calculation and relative density estimation on NASA and PROMISE.
ComponentOne-to-oneMany-to-oneNASAPROMISENASAPROMISEJS calculation163.7483.2163.7483.2Relative density estimation64.136.2359.8440.3Total227.8519.4523.5923.5


## 6. Threats to Validity

Four potential threats to the validity are described in the following:(1)
Accuracy of experiments. Most of the compared works do not provide codes of their methods. This study only analyses and implements their methods by following their papers.(2)
Bias of evaluation measures. In this work, the wide measures of F1-score, AUC, and G-mean are used to show the results of the prediction. Other measures, such as recall, precision, skewed F-measure, and Matthews correlation coefficient, are not considered.(3)
Bias of classifiers. Classification is a significant research topic, and many learning methods can be used to build classifiers. As investigated in previous studies, Logistic regression can achieve better performance. Therefore, this work also applied logistic regression to build the classifiers. Meanwhile, the convention learning methods of SVM, KNN, Random Forest, and Naïve Bayes are also tested to evaluate the performance.(4)
Bias of datasets. Several benchmark data sets are commonly used in the field of cross-project defect prediction, such as NASA (Shepperd et al. [26]), PROMISE (Jureczko and Madeyski [27]), AEEEM (D’Ambros et al. [44]), Relink (Wu et al. [45]), and SOFTLAB (Turhan et al. [17]), and some datasets contain different versions of each project. To evaluate the prediction performance, this work chose the widely used NASA and PROMISE as the datasets.


## 7. Related Work

Cross-project defect prediction has become one of the most important topics in software engineering (Rahman et al. [2]; Yang et al. [3]; Ghotra et al. [4]; Herbold et al. [14]; Wen et al. [5]). In practice, researchers have recognized it as a binary classification problem, that is, they model the defective data to train a machine learning model. Due to insufficient training data in the source projects, the researchers try to build a bridge from the source project to the target project. They utilize the information from the source projects and transfer it to the target project. The following will give the related studies only.For one-to-one defect prediction, Sun et al. [38] described an effective and efficient method for domain adaptation called CORAL, which minimized the distribution difference between source and target projects. Zhang et al. [46] applied different transformations to explore whether the cross-project defect prediction is affected and proposed Multiple Transformations (MT) and MT+ to improve prediction performance. Xu et al. [35] introduced a balanced distribution adaptation-based transfer learning method to implement defect prediction, which improved the existing methods’ performance. In this study, we compare those one-to-one models with our one-to-one approach.For many-to-one defect prediction, Herbold [18] utilized distance-based strategies to select training data in many-to-one defect prediction. The results show their method can achieve good performance. Nam et al. [23] extended their work to TCA+ with a preprocessing data method and conducted it on one-to-one and many-to-one predictions. Minku et al. [40] proposed a transfer learning model for Dycom based on the work (Nam et al. [23]). Dycom is a weighted sum of transferred models trained by various source projects. Jing et al. [39] proposed a method called subclass discriminant analysis, which can learn features from original metrics and make the distributions of source and target projects stable. Liu et al. [24] proposed a two-phase transfer learning model (TPTL) for CPDP. They introduced a source project estimator to choose similar source projects and built the two prediction models. By combining the prediction results of the two models, they further improved the final performance. In this study, we compare the above models with our many-to-one approach.Besides homogeneous defect prediction, heterogeneous defect prediction has recently become a great process. Gong et al. [47] utilized the thought of stratification embedded in the nearest neighbor to produce evolving training datasets with balanced data. Zou et al. [48] proposed a method named Joint Feature representation with double marginalized denoising auto-encoders to learn the global and local features, and they introduced local data gravitation between source and target domains to determine instance weight in the learning process (Zou et al. [49]). Jin et al. [50] used two support vector machines to implement domain adaptation to match data distribution. Mehta and Patnaik [51] used various ensemble machine learning techniques to improve classification performance. In the future, we will study heterogeneous defect prediction based on the state-of-the-art model.From the abovementioned literature, it can be concluded that there are two significant problems in CPDP research. The first problem is that the source and target project data usually exhibit significantly different distributions. Since the source and target projects might be implemented by different programming languages or companies, the same metrics in the source and target project data might have different distributions. The second problem is that there are no standard metrics between the source and target project data. It is difficult to predict the defects in the target project using conventional methods to build models on the source project data. Therefore, many efforts have been made to solve these two problems in recent years, and this study focuses on the solution to the first problem. In the future, standardized tools will be developed based on more experiments.


## 8. Conclusion and Future Works

This paper studies the limitations of prior studies and proposes a novel approach. First, to avoid the randomness of the source project selection, Jensen-Shannon divergence is used to measure the similarity between source and target projects automatically. Subsequently, relative density information is introduced to filter noise and outliers in similar source projects. Posteriorly, the one-to-one defect prediction model is constructed based on the most similar source project directly, and the many-to-one defect prediction model is built by a proposed ensemble weight strategy based on Jensen-Shannon divergence. Finally, the models predict the potential defects in the target project.Experimental results on the benchmark datasets of NASA and PROMISE indicate that our approach can improve the F1-score, AUC, and G-mean. Moreover, the results also prove that the proposed approach is adaptive regardless of the type of learning method.In further work, we will select more defect data sets to evaluate the effectiveness of the proposed approach. Moreover, the proposed approach will be improved, such as combing domain adaptation analysis, to implement heterogeneous defect prediction.



---

*Source: 4648468-2022-10-20.xml*