An empirical study on the effectiveness of data resampling approaches for cross‐project software defect prediction

IET SoftwareVolume 16, Issue 2 p. 185-199 ORIGINAL RESEARCHOpen Access An empirical study on the effectiveness of data resampling approaches for cross-project software defect prediction Kwabena Ebo Bennin, Kwabena Ebo Bennin orcid.org/0000-0001-9140-9271 Information Technology, Wageningen University and Research, Wageningen, the NetherlandsSearch for more papers by this authorAmjed Tahir, Amjed Tahir orcid.org/0000-0001-9454-1366 Massey University, Palmerston North, New ZealandSearch for more papers by this authorStephen G. MacDonell, Stephen G. MacDonell University of Otago and Auckland University of Technology, Dunedin, New ZealandSearch for more papers by this authorJürgen Börstler, Corresponding Author Jürgen Börstler jurgen.borstler@bth.se Blekinge Institute of Technology, Karlskrona, Sweden Correspondence Jürgen Börstler, Blekinge Institute of Technology, Karlskrona, Sweden. Email: jurgen.borstler@bth.seSearch for more papers by this author Kwabena Ebo Bennin, Kwabena Ebo Bennin orcid.org/0000-0001-9140-9271 Information Technology, Wageningen University and Research, Wageningen, the NetherlandsSearch for more papers by this authorAmjed Tahir, Amjed Tahir orcid.org/0000-0001-9454-1366 Massey University, Palmerston North, New ZealandSearch for more papers by this authorStephen G. MacDonell, Stephen G. MacDonell University of Otago and Auckland University of Technology, Dunedin, New ZealandSearch for more papers by this authorJürgen Börstler, Corresponding Author Jürgen Börstler jurgen.borstler@bth.se Blekinge Institute of Technology, Karlskrona, Sweden Correspondence Jürgen Börstler, Blekinge Institute of Technology, Karlskrona, Sweden. Email: jurgen.borstler@bth.seSearch for more papers by this author First published: 28 November 2021 https://doi.org/10.1049/sfw2.12052AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract Cross-project defect prediction (CPDP), where data from different software projects are used to predict defects, has been proposed as a way to provide data for software projects that lack historical data. Evaluations of CPDP models using the Nearest Neighbour (NN) Filter approach have shown promising results in recent studies. A key challenge with defect-prediction datasets is class imbalance, that is, highly skewed datasets where non-buggy modules dominate the buggy modules. In the past, data resampling approaches have been applied to within-projects defect prediction models to help alleviate the negative effects of class imbalance in the datasets. To address the class imbalance issue in CPDP, the authors assess the impact of data resampling approaches on CPDP models after the NN Filter is applied. The impact on prediction performance of five oversampling approaches (MAHAKIL, SMOTE, Borderline-SMOTE, Random Oversampling and ADASYN) and three undersampling approaches (Random Undersampling, Tomek Links and One-sided selection) is investigated and results are compared to approaches without data resampling. The authors examined six defect prediction models on 34 datasets extracted from the PROMISE repository. The authors' results show that there is a significant positive effect of data resampling on CPDP performance, suggesting that software quality teams and researchers should consider applying data resampling approaches for improved recall (pd) and g-measure prediction performance. However, if the goal is to improve precision and reduce false alarm (pf) then data resampling approaches should be avoided. 1 INTRODUCTION Defect prediction models can help to identify defective software components and thereby support managers in resource allocation. Previous studies have shown that defect prediction models can yield useful results [1, 2], but their reliability might be affected by the quality of the underlying datasets [3] or confounding variables affecting the measures used for creating the prediction models [4-6]. Several studies have proposed prediction models based on different statistical and machine-learning approaches [7-9]. However, the performance on these models largely depends on historical data, obtained either from the same project (in case historical data exist) or from projects that are very similar to the project under consideration regarding content and context. Zimmermann et al. [10] cautioned that defect prediction models perform well within projects as long as enough data for training of the prediction models exist. However, for new and unfamiliar projects, the lack of historical data becomes a challenge. A promising approach to handle this issue is to use a cross-company or cross-project defect prediction (CPDP), where data from other companies or projects are used for model training. To help in obtaining the most suitable training data for CPDP, different techniques have been proposed and validated, including data filtering techniques such as Nearest Neighbour (NN) filter [11], Double Transfer Boosting [12], and clustering [13, 14]. NN filter, a data filtering approach, has been shown to perform significantly better than several ensemble, boosting or transfer-learning-based approaches [15]. Hosseini et al. [16] and Turhan et al. [17] confirmed that the NN filter can have a positive impact on the performance of CPDP models. NN filter eliminates irrelevant data instances based on the characteristics of the target distribution, selecting only the more suitable defective and clean instances [11]. However, class imbalance is a prevalent problem in data mining and defect prediction [18, 19], where the majority of the instances are clean or not faulty [19, 20]. Consequently, the acquired dataset for cross-project model training will most likely be highly skewed towards one class (that is the non-defective or clean instance). Resampling approaches such as simple Random Over-Sampling (ROS), Random Under-Sampling (RUS), and synthetic methods such as SMOTE have been proposed to alleviate the negative effects of class imbalance on the performance of Within Project Defect Prediction (WPDP) models [21]. Resampling approaches aim to increase the number of minority class samples (defective modules), and they can significantly improve the performance (recall, G-mean) of defect prediction models [22]. The potential benefit of data resampling approaches in mitigating the negative effect of class imbalance on CPDP models has been investigated by few researchers. Previous studies [23, 12, 24, 25] considered the effect of very few data resampling approaches or evaluate the performances of data resampling approaches on CPDP using a few datasets. Other studies propose complex methods integrated with data resampling approaches to improve CPDP performances [26, 27]. A recent systematic literature review by Hosseini et al. [16] revealed that most studies in CPDP fail to use multiple performance measures and fail to apply robust statistical tests, including effect sizes, which resulted in an unfair comparison of the performances of CPDP to WPDP models. The motivation of this study is to augment the few existing studies and examine the practical benefits of data resampling approaches on CPDP models. We assess the impact of applying eight commonly used data resampling methods (MAHAKIL, SMOTE, Borderline-SMOTE, ADASYN, ROS, RUS, Tomek links and OSS) in the domain of CPDP after acquiring data from different projects (using the NN filter) and use the resampled data datasets for the training of the defect prediction model for a different project. Additionally, we conduct robust statistical tests on the results by testing for statistical significance using Brunner’s statistical test [28] and apply Cliff’s effect size to examine the practical benefits of the applied data resampling approaches. We selected the NN filter because it is easy to implement compared to other CPDP approaches and has been shown to improve CPDP performance in previous studies [29, 14, 16, 15]. To assess the impact of data resampling approaches on NN-filtered datasets, this study explores the following research questions. RQ1: What is the impact of data resampling approaches on NN-filtered datasets in CPDP? RQ2: What are the high-performing resampling approaches for NN-filtered datasets in CPDP? RQ3: Is the application of data resampling approaches practical for CPDP? The contributions of this paper are as follows: A benchmark experiment that shows that recall (pd) and g-mean performances of CPDP can be improved by applying data resampling approaches to NN-filtered datasets. A demonstration that oversampling and random undersampling methods always produce higher false alarms (pf) in CPDP. NN-filtered data with no resampling produces the best pf values. A python package for MAHAKIL—an easy-to-use tool for oversampling class imbalanced data. The remainder of this paper is structured as follows: Section 2 presents the related work. In Section 3, we discuss our methods and experimental settings. Our results and a discussion of the results are reported in Sections 5 and 6, followed by a discussion of potential threats to validity in Section 7. Finally, we present our conclusion from this study with potential future work directions in Section 8. 2 RELATED WORK 2.1 Cross-project defect prediction One of the first to attempt at developing CPDP models was Zimmermann et al. [10]. By conducting a large-scale experiment on 12 real-world datasets, 622 cross-project prediction models were analysed and investigated for the feasibility of cross-company defect prediction (CCDP) models. After observing a low success rate of 3.4%, they conclude that CCDP is still a challenge. Turhan et al. [11] proposed a practical defect prediction approach for organizations aiming to employ defect prediction but that lacks historical data. Applying the principles of analogy-based learning, they use the k-nearest neighbour algorithm to select 10 nearest data instances from other company data for every unlabelled test instance for CCDP. They demonstrated that even small data samples acquired using their approach could be used to build effective defect predictors. He et al. [30] conducted a large study using open-source projects to investigate the feasibility of CPDP, considering careful data selection approaches. The obtained results were similar to those achieved by previous studies [10, 11], indicating that CPDP works well in a few cases and carefully selecting the training data improves prediction performance though not necessarily selecting projects in the same domain. A recent benchmark study by Herbold et al. [15] complemented the positive impact of the data filtering approaches on CPDP models. Other studies, including [31, 12, 32, 33], have employed several transfer learning and data boosting techniques to improve CPDP performance. Considering two projects, Watanabe et al. [34] conducted an inter-project prediction and demonstrated that data characteristics had an impact on the success of cross-project predictions. They adapted CPDP by using a metric compensation method that adjusted the average values of each metric in both the training and test set to the same level and achieved high precision and recall values. Transfer learning techniques have also been applied in the domain of CPDP. In order to make the feature distribution of the source projects and target projects similar, Nam et al. [31] applied Transfer Component Analysis, which transforms the data based on the new feature representation discovered from both the source and target projects. They found that the prediction performance increased significantly after experimenting on eight open-source projects. Similarly, Yu et al. [32] proposed a novel semi-supervised clustering-based data filtering method that filters the data and adopts multi-source TrAdaBoost algorithm, an effective transfer learning method, into cross-company prediction to import knowledge not from one but from multiple sources to avoid a negative transfer. Poon et al. [35] proposed a Credibility theory-based Naive Bayes (CNB) classifier that uses a novel re-weighting mechanism to adapt the source data to the target data distribution simultaneously. The method ensures that the pattern of the source data and experimental results improved the performance over other CPDP methods. The study of Zhou et al. [36] investigated the performances of a number of CPDP techniques and models with simple size models and observed that simple size models in most cases outperformed the complex and recently proposed CPDP techniques. Asano et al. [37] applied bandit algorithms to help in selecting the most suitable projects for CPDP models. Our study aims to complement prior studies by aiming to improve the performance of existing CPDP models that adopts the NN filter approach. 2.2 Data resampling application in CPDP Although the class imbalance issue is known to be critical for defect prediction models, only a few studies have discussed the challenge of class imbalance in the context of CPDP. Ryu et al. [26, 27] proposed two methods, a boosting and instance weighting technique that uses transfer learning to solve the class imbalance issue. Jing et al. [38] employed a semi-supervised transfer component analysis to balance the source and target datasets before applying their proposed semi-supervised transfer component analysis and improved subclass discriminant analysis for prediction. Most of the previous studies' aim was to improve prediction performance by integrating data resampling approaches as part of their solutions. A study by Kamei et al. [25] on Just-In-Time (JIT) defect prediction models using 11 open-source cross-projects revealed that JIT models rarely improved in general, but did improve when selected training projects are similar to the testing data, larger set of training data is provided or using ensemble models. The random undersampling approach was applied to the training datasets before training the JIT models. Other studies have adopted the popular SMOTE approach in alleviating the negative effects of class imbalance. Chen et al. [12] used the SMOTE resampling approach to balance their training datasets before applying their proposed Double Transfer Boosting model. Recently, Limsettho et al. [29] proposed CDE-SMOTE that adopts the SMOTE data resampling approach to alleviate the negative effects of class distribution mismatch and imbalance between the training and test datasets. The authors estimate the proportion of each class of the target dataset using class distribution estimation and employ the SMOTE method to generate data samples equal to the estimated amount. Additionally, Goel et al. [23] observed that SMOTE improves the performance of CPDP models. Yu et al. [24] were the first to conduct an empirical study comprising of more than one data resampling approach. They applied six data resampling approaches on 15 open-source datasets and three CPDP models and observed that the performance improved when data resampling approaches are applied. More information on CPDP studies can be found in the recent literature review and meta-analysis study conducted by Hosseini et al. [16]. The comprehensive studies of Bennin et al. [21] and Tantithamthavorn et al. [39] provide more information on the impact of data resampling approaches on software defect prediction models. Previous studies aim to improve CPDP performance by using complex boosting and transfer learning techniques. Most studies [23, 12, 29, 25] investigated the effects of a single type of data resampling approach on CPDP performances. The study presented in this paper differentiates itself from the studies discussed above, and specifically, the study of Yu et al, [24], in which it applies eight data resampling approaches to the acquired data after applying an NN filter and empirically investigate the impact of data resampling approaches on the performance of CPDP. Compared to WPDP, filter-based CPDP approaches including the NN filter try to generate a project dataset that is similar to a target project dataset. This NN filter procedure is a simple approach that is independent of the prediction model, straightforward to implement and has been shown to have a positive impact on CPDP models [17, 15, 16]. 3 BACKGROUND In this section, we present a brief description of the NN filter and an overview of the data resampling approaches we employed in our experiments. The NN filter is selected because (1) it is easier to implement, compared to other similar approaches, (2) it has been widely (and successfully) used in previous studies [29, 14, 40] and (3) it is known to improve prediction performance in comparison to other data filtering techniques [16, 15]. 3.1 NN filter This approach proposed by Turhan et al. [11] is a relevancy filter that effectively selects the closest data instances with respect to the new target project from a collection of various projects based on the K-NN algorithm. It is a pre-processing approach that could be combined with the normal classification process. It was combined with a Naive Bayes classifier by the same authors. The procedure for NN filtering is as follows: Aggregate and merge all training data into one set of data. For each module in the target project, find k neighbours from the combined training data considering their pairwise Euclidean distances. Collect the selected neighbours and remove duplicated data instances to obtain the filtered dataset. 3.2 Oversampling approaches 3.2.1 Random oversampling (ROS) ROS can be considered as one of the most basic oversampling approaches. Minority instances are randomly selected and replicated, and thus, no new information is provided for the classifier. ROS is very easy to implement and has been widely used in several studies in defect prediction [41, 19, 21, 42]. 3.2.2 Synthetic Minority Oversampling TEchnique (SMOTE) A technique designed to alleviate the effect of class imbalance on prediction performance [43]. Several defect prediction studies [44-46] have adopted and applied SMOTE in the literature. SMOTE generates 'synthetic' minority samples by considering each minority class sample. Based on the parameter values (k), new synthetic samples are created along with the line segments that join k minority class NNs of each sample under consideration. 3.2.3 MAHAKIL A recently developed state-of-the-art oversampling technique that aims to generate diverse minority class samples in order to improve prediction performance and reduce false alarms [22]. Different from the conventional k NN methods, MAHAKIL utilizes the chromosomal theory of inheritance to generate diverse synthetic data instances. MAHAKIL alleviates the overgeneralization of prediction models since synthetic data instances created are not clustered into a specific region of the minority class of the dataset. The code used in this study (a python package) is provided online for future use and replication studies.11 https://bit.ly/3oVDji7 3.2.4 Adaptive Synthetic Sampling (ADASYN) By focusing only on the minority class samples that are difficult to classify, the ADASYN approach [47] assigns weights to the minority classes and dynamically adjusts the weights in a bid to reduce the bias in the imbalanced dataset by considering the characteristics of the data distribution. The ADASYN algorithm incorporates a density distribution in automatically deciding the number of synthetic samples needed for each minority class sample. In contrast to the SMOTE algorithm, which generates equal synthetic samples for each minority class, the ADASYN learning algorithm is induced to focus on the hard to learn examples within the minority class samples; therefore, samples generated are not equal for all samples. 3.2.5 Borderline-SMOTE This technique is a modification of the original SMOTE [43] that focuses more on the instances that are harder-to-classify, that is, instances on the borderline of the classifier [48]. Instances that are marked as hard-to-classify are labelled and considered for generating new synthetic data using SMOTE. Borderline-SMOTE aims to clearly set the decision boundary for the trained classifier for improved prediction performance. 3.3 Undersampling approaches 3.3.1 Random undersampling (RUS) RUS aims to balance the dataset by randomly selecting and deleting the majority of instances. RUS is easy and fast to implement. As a conventional technique, it is a widely used undersampling technique in several empirical studies [41, 21, 19, 42]. 3.3.2 Tomek links Tomek [49] observed that instances of different classes could be very close neighbours and overlap each other. These instances are closer to each other than they are to their own class samples, and they form what is referred to as Tomek links. The majority of instances that are part of the Tomek links are considered as noise and are deleted by this technique until all minimally distanced nearest-neighbour pairs are of the same class. 3.3.3 One-sided selection Kubat et al. [50] proposed the undersampling approach called One-sided selection. This technique aims to remove noisy and borderline majority instances by adopting the concept of Tomek links [49]. The technique works by randomly selecting a majority sample and combining it with all minority instances to create a new set S. The new set S is used to classify the original dataset and all misclassified instances are added to S. It considers majority instances to be redundant if these instances in S participate in Tomek links. 4 EVALUATION AND EMPIRICAL ANALYSIS The research questions and experimental design and execution are described in this section. 4.1 Research questions and experiment design The goal of this study is to empirically examine and assess the impact of using data resampling approaches on NN-filtered datasets used in CPDP models. To achieve our goal, we carried out two sets of experiments. The experiments are conducted and executed to answer the following three research questions. RQ1: What is the impact of data resampling approaches on NN-filtered datasets in CPDP? RQ2: What are the high-performing resampling approaches for NN-filtered datasets in CPDP? RQ3: Is the application of data resampling approaches practical for CPDP? The research questions are used to examine the impact of applying several data resampling approaches on the performance of the CPDP. The study considered only defect-proneness (predicting if a module is defective or not) and not defect counts. We follow the criteria used by Peters et al. [51] and Zimmermann et al. [10] to determine if resampling approaches are beneficial and practical for CPDP. The work of Zimmermann et al. [10] demonstrated that a strong predictor is the one that achieves g-mean, recall and accuracy above 75%. By using a simple criterion, the total count of test-sets with predictors (models) that produced g-mean, recall and auc above 75% are computed. Since we include pf measures, we set it as practical if pf is less than 15%. Detailed description on the dataset used, experimental setup and analysis is discussed below. 4.2 Datasets We obtained data from thirty-four (34) versions of twenty-two (22) open-source projects that are extracted from the PROMISE data repository, which are freely available.22 https://zenodo.org/communities/seacraft Dataset size, imbalance ratio and projects with more than one version all carry heavyweight in our selection criteria, though we keep to datasets widely used in previous studies. Table 1 shows a summary of the description of these datasets. These projects, donated by Jureczko and Madeyski [13] and Jureczko and Spinellis [52], were written in Java where a module corresponds to a Java file. We only consider top-level classes (where the class has the same name as the source file, i.e. inner classes are ignored). The datasets vary in size and imbalance ratios, and they provide an extensive domain for evaluating the impact of resampling approaches and have been used in several previous studies. Twenty-one (21) code metrics from the CK and other metrics suites (see Table 2 for details) are collected using the BugInfo and ckjm33 http://www.spinelis.gr/sw/ckjm tools from each module. A non-defective module is labelled as zero, and defective modules are labelled with the number of bugs present in the module. Projects that had more than one version available were all merged to create our training (source) datasets. Single-version datasets (the first 14 projects) were thus used as the test (target) dataset where we applied the NN filter to selectively produce the filtered training datasets for each of the test datasets. TABLE 1. Summary of 34 systems extracted from PROMISE data repository # Release No. of modules No. of defects Defects (%) (Target Data) 1 Arc 234 27 11.5 2 Berek 43 16 37.2 3 e-learning 64 5 8 4 Intercafe 27 4 14.8 5 Kalkulator 27 6 22.2 6 nieruchomosci 27 10 37 7 Pdftranslator 33 15 45.5 8 Redaktor 176 27 15.3 9 Serapion 45 9 20 10 Skarbonka 45 9 20 11 Systemdata 65 9 13.8 12 Tomcat 858 77 9 13 Workflow 39 20 51.3 14 Zuzel 29 13 44.8 (Source Data) 15 ant-1.3 125 20 16 16 ant-1.4 178 40 22.5 17 ant-1.5 293 32 10.9 18 ant-1.6 351 92 26.2 19 camel-1.0 339 13 3.8 20 camel-1.4 872 145 16.6 21 camel-1.6 965 188 19.5 22 ivy-1.1 111 63 56.8 23 ivy-1.4 241 16 6.6 24 ivy-2.0 352 40 11.4 25 jedit-4.0 306 75 24.5 26 jedit-4.1 312 79 25.3 27 jedit-4.2 367 48 13.1 28 jedit-4.3 492 11 2.2 29 pbeans1 26 20 76.9 30 pbeans2 51 10 19.6 31 synapse-1.0 157 16 10.2 32 xalan-2.4 723 110 15.2 33 xerces-1.2 440 71 16.1 34 xerces-1.3 453 69 15.2 TABLE 2. Description of static code metrics [30] Metrics Description Metrics Description WMC Weighted methods per class CAM Cohesion among methods of class DIT Depth of inheritance tree IC Inheritance coupling NOC Number of children CBM Coupling between methods CBO Coupling between objects AMC Average method complexity RFC Response for a class Ca Afferent couplings LCOM Lack of cohesion of methods Ce Efferent couplings LCOM3 Another form of LCOM CC McCabe's cyclomatic complexity NPM Number of public methods Max(CC) Max CC values of methods in class DAM Data access metric Avg(CC) Mean CC values of methods in class MOA Measure of aggregation LOC Lines of code MFA Measure of functional abstraction Defects Number of detected bugs in the class 4.3 Evaluation measures To assess our modules, we use four performance evaluation metrics: recall (i.e. probability of detection [pd]), Area Under the ROC Curve (AUC), g- measure and Probability of false alarm (pf). These performance metrics range between the values of 0%–100%. We decided not to include precision and F- measure as they have been refuted as unstable for assessing imbalanced datasets [53]. Recall (pd) measures how much of the defective modules were detected. In the general sense, a higher recall denotes better performance. The probability of false alarm (pf) measures the rate of wrongly predicted modules that were non-defective. A low or zero value for pf implies a better prediction model. For a highly imbalanced dataset, the AUC and g-measure (g-mean) performance metrics are preferred as they consider both the pf and pd values and do not value one metric over the other. AUC has also been recommended to be very stable for imbalance learning [21, 39]. For practical results, Zimmermann et al. [10] recommend pd values of 75% and above, which is maybe very challenging to achieve in CPDP studies. Herbold et al. [15] also observed that CPDP models rarely achieved a high recall of 75%. We use multiple performance measures to fully grasp the capabilities and dynamics of the examined prediction models and resampling methods as noted by the authors in [16]. The mathematical definitions of pd, pf and g-measure metrics are presented below. Computation of the AUC of ROC can be found elsewhere [54]. Re c a l l ( p d ) = T P T P + F N (1) p f = F P F P + T N (2) g − m e a s u r e = 2 ∗ p d ∗ ( 100 − p f ) p d + ( 100 − p f ) (3) 4.4 Experimental setup Two sets of empirical experiments are conducted. We first evaluate the performance of using an NN filter for CPDP, and we then investigate the influence data resampling approaches on the performance of CPDP models after filtering the training datasets. Implementation of the data resampling methods and the NN filter was executed using Imbalanced-learn [55], an open-source processing toolbox and the model construction and evaluation were executed using the open-source scikit learn [56] library available in python. We use all metrics from the datasets as suggested in [2]. For this study, five methods widely used in defect prediction studies were chosen. We considered Random Forests, Naive Bayes, K-NN algorithm, NNET and SVM [1]. Furthermore, we adopted a recently proposed boosting model called XGBoost [57]. XGBoost is an improved implementation of a gradient boosting framework optimized to be highly efficient, effective and better performance [57, 58]. The parameters used for the prediction models are displayed in Table 3. It should be noted that we predicted the presence of defects (binary) in a module and not the nu