Systematic literature review of preprocessing techniques for imbalanced data

IET SoftwareVolume 13, Issue 6 p. 479-496 Review ArticleOpen Access Systematic literature review of preprocessing techniques for imbalanced data Ebubeogu Amarachukwu Felix, Ebubeogu Amarachukwu Felix orcid.org/0000-0001-8389-4860 Faculty of Computer Science and Information Technology, University of Malaya, 50603 Kuala Lumpur, MalaysiaSearch for more papers by this authorSai Peck Lee, Corresponding Author Sai Peck Lee saipeck@um.edu.my Faculty of Computer Science and Information Technology, University of Malaya, 50603 Kuala Lumpur, MalaysiaSearch for more papers by this author Ebubeogu Amarachukwu Felix, Ebubeogu Amarachukwu Felix orcid.org/0000-0001-8389-4860 Faculty of Computer Science and Information Technology, University of Malaya, 50603 Kuala Lumpur, MalaysiaSearch for more papers by this authorSai Peck Lee, Corresponding Author Sai Peck Lee saipeck@um.edu.my Faculty of Computer Science and Information Technology, University of Malaya, 50603 Kuala Lumpur, MalaysiaSearch for more papers by this author First published: 01 December 2019 https://doi.org/10.1049/iet-sen.2018.5193Citations: 27AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Data preprocessing remains an important step in machine learning studies. This is because proper preprocessing of imbalanced data can enable researchers to reduce defects as much as possible, which, in turn, may lead to the elimination of defects in existing data sets. Despite the remarkable achievements that have been accomplished in machine learning studies, systematic literature reviews of imbalanced data preprocessing techniques are lacking. Consequently, there are a limited number of systematic literature review studies on imbalanced data preprocessing. In this study, the authors assess the existing literature to identify the key issues related to data quality and handling and to provide a convenient collection of the techniques used to address these issues when performing data preprocessing. They applied a systematic literature review method involving a manual search to select articles published from January 2010 to September 2018 for review. The qualities of the existing studies were assessed using certain quality assessment criteria. Of the 118 relevant studies found, only 2% were identified as having been conducted following systematic literature review guidelines. This study, therefore, calls for more systematic literature review studies on data preprocessing to improve the quality of the data applied in machine learning studies. 1 Introduction Data preprocessing is a crucial concern in machine learning research. It is performed before the construction of learning models to prepare reliable input data sets [1]. As a fundamental phase in machine learning studies, data preprocessing requires the understanding, identification, and specification of data-related issues as well as a knowledge-based approach that can be used to address these issues and thus make data more reliable for use in machine learning studies [2]. Notably, there is clear evidence that data preprocessing can impact the predictive performance of learning algorithms [3]. If data are properly preprocessed, researchers can accurately identify and report the number of defects found in their data, thereby making their data sets more suitable for enabling learning models to learn independently and accurately from unbiased data to produce reliable results. The credibility of defect prediction is one factor influencing data quality [4-7]. In addition, the performances of the prediction models themselves consistently depend on the quality of the data used to train them. Such models are trained using historical data to identify defect-prone software modules [8]. During data preprocessing activities, one of the most important characteristics of an existing data set is the imbalanced nature of the data it contains, which can be defined as the difference in volume between the defective (minority) and defect-free (majority) classes in the data set. This data characteristic, i.e. class imbalance, has received considerable attention, and attempts have been made to address this issue in various previous studies, such as [9-20]. As the attention paid to the imbalanced nature of data in defect prediction studies has increased, previous studies have focused on rebalancing the existing data sets used in such studies to improve the performance of learning algorithms. However, such rebalancing can also result in the alteration of a data set, which, in turn, may introduce more defects or cause the data to lose some valuable properties that could lead to knowledge discovery in machine learning. At the same time, the rebalancing of existing data sets does not have a significant impact on the performance of learning algorithms [21]. Although rebalancing can be used to achieve an increase in recall, it does not improve precision or enhance the area under the receiver operating characteristic curve, which serves as a standard measure for comparing the performances of prediction models. In addition, rebalancing can lead to unreliable prediction outcomes, which can result in concept drift [21]. In view of the fact that the results obtained via rebalancing can be misleading, as Tantithamthavorn et al. [8] confirmed, rebalancing can be performed to improve recall but should be avoided when new knowledge is to be discovered through learning algorithms. Thus, it is necessary to focus on eliminating defects from a data set by means of preprocessing rather than altering the original data set, which may lead to the introduction of additional defects. The removal of defects from an existing data set can be achieved through data preprocessing, which can be defined as the ability to make the original data set less defective and more suitable for use in training algorithms to make unbiased predictions. Although some previous studies have focused on data imbalance, as well as other data-related issues highlighted in this study, in order to improve the performance of learning algorithms, such issues remain unresolved in general throughout the broader machine learning community. In particular, one aspect of machine learning studies that often remains unclear is how researchers perform and report the approaches they apply for data preprocessing. This question, among others, is one of the research questions (RQs) raised in this systematic literature review (SLR). The objective of properly preprocessing and cleaning a data set is to enable learning algorithms to independently and accurately learn from unbiased data to produce reliable outcomes. If such models are instead trained with unprocessed and noisy data, they can produce unreliable and biased results [22-27]. Previous studies, such as [16, 18, 28, 29], have suggested that biased learning could result from training on imbalanced data. In contrast, we, like other researchers, such as the authors of [9, 30], take the view that data imbalance itself does not have a severe impact on the performance of learning algorithms. Rather, the research group conducting the study is more important than the chosen learning algorithm in determining performance [31]. As such, researchers should spend more time cleaning the data sets used in machine learning studies and should report in detail the approaches adopted in the cleaning process. Moreover, to ensure reliable data cleaning results, researchers themselves must participate in the data cleaning process [32]. Proper reporting of data cleaning approaches may reveal new knowledge that will enhance the learning outcomes in machine learning studies. As of now, a comprehensive and systematic record of the data preprocessing techniques adopted in machine learning studies is lacking. To address this lack, we conducted the current SLR study to uncover the descriptions of such methods, if any, in the existing literature. The purpose was to review the current techniques that have been applied in the preprocessing of data sets since January 2010 based on articles related to data preprocessing, with a particular focus on SLR articles. The reason for choosing the specified period of time is to assess the most recent articles and the most current techniques applied for addressing key data-related issues. Although some SLR studies were conducted prior to 2014, such as [4, 33], these prior studies suffer from the limitation that they did not focus explicitly on data preprocessing. As we embarked on this methodologically rigorous review process, our aim was not merely to aggregate the clear evidence regarding data preprocessing techniques available from the existing literature but also to encourage researchers to undertake further SLR studies on data preprocessing that can serve as a guide for data cleaning in machine learning studies. Hence, the current study presents lists of relevant publications regarding data preprocessing. Based on these relevant studies, we identify and aggregate a collection of key factors affecting the quality of the data applied in machine learning studies. In addition, we identify and aggregate a list of techniques applied to address these factors affecting data quality. With the current study, we wish to encourage high-quality SLRs regarding data preprocessing to avoid bias in research findings by means of quality evaluations. Based on the quality assessment (QA) applied in the current study, the average quality score of each relevant study by year of publication is presented. There is also a need to indicate which study followed the SLR guidelines as well as the preferred reporting items for systematic review and meta-analysis (PRISMA) guidelines. To that end, the current study presents the numbers of studies that followed both the SLR and PRISMA guidelines, their percentages and their mean quality scores. Notably, some of the reviewed studies are conference publications, while others are journal publications. In view of this fact, we aggregate these studies by both type and year of publication. The percentage distributions of the aggregated primary studies by publication year as well as their corresponding quality scores are presented in the current study. We describe our methodology in Section 2 and present our results in Section 3. In Section 4, we answer our four major RQs. We present our conclusions in Section 5. 1.1 Key data-related issues The list of identified issues related to the existing data sets, as reported by Hosseini et al. [34], includes class imbalance, data heterogeneity, high skewness, privacy, irrelevant and redundant features, continuous data, collinearity among metrics and noise in data. 1.1.1 Class imbalance One of the major characteristics of existing data sets is a class imbalance, which can be defined as the difference in data volume between the categories into which data are to be classified [13]. Class imbalance can also be regarded as a situation in which the numbers of defective (minority) and defect-free (majority) instances in a project are not equal. Class imbalance in a data set can affect the performance of classification algorithms, potentially leading to unfair and inaccurate evaluation outcomes when traditional assessment techniques are applied [14, 16, 18, 20]. Although approaches for overcoming the challenges posed by data imbalance have been proposed in many previous studies, such as [35-46], the issue of imbalanced data in machine learning studies still remains unresolved. In some of the primary studies selected in this SLR, such as [47-49], resampling techniques have been applied to address this problem. In addition, reweighting has been applied in previous studies, such as [50-52], to address the imbalance problem. Ryu et al. [53] applied a selective learning technique to address the issue of imbalanced data. 1.1.2 Data heterogeneity Variance in the attributes of existing data sets can lead to differences in the outcomes of different machine learning studies. This is because data samples from different sources that are applied in such studies exhibit different characteristics. The extent of the variations among data sources and characteristics can lead to greater or lesser heterogeneity in data sets. This is because data from different sources may exhibit different characteristics, may be expressed using different representations, may be stored in incompatible formats, and may otherwise be both inconsistent and interrelated [54]. In addition, most software projects are characterised by heterogeneity, with dissimilar metrics spread throughout the data sets that constitute these projects [55]. Furthermore, as reported by Yu and Mishra [56], data heterogeneity can be affected by various contextual factors, such as the domain, project size, and programming language used. Notably, due to the more homogeneous nature of the data involved, some machine learning algorithms are likely to perform well when applied for within-project defect prediction (WPDP) [34]. However, Nam et al. [57] confirmed that despite their success in WPDP due to the nature of the data, such algorithms may not be applicable for cross-project defect prediction. To address the issue of data heterogeneity, data transformation techniques have been applied in previous studies, such as [51, 57-60]. Filtering has also been applied to address data heterogeneity, as reported by the authors of [40, 61-64]. Data normalisation is another approach applied for this purpose, as reported by the authors of [39, 57, 65-67]. Nam et al. [68] reported the use of metric matching to address data heterogeneity, whereas the authors of [39, 55] addressed this issue by using clustering techniques. 1.1.3 High skewness Most of the time, the minority and majority classes in a data set are not normally distributed, which leads to data skewness. Data can be skewed either positively or negatively. Positively skewed data indicate a disadvantage to the right of the normal distribution line, whereas a negatively skewed data distribution shows a disadvantage to the left of the normal distribution line. Figs. 1-3 present normal, positively skewed and negatively skewed data distributions, respectively. The x-axis represents the frequency of the data, while the y-axis represents the data points for both the positive and negative classes. Fig. 1Open in figure viewerPowerPoint Normal skewness Fig. 2Open in figure viewerPowerPoint Positive skewness Fig. 3Open in figure viewerPowerPoint Negative skewness As shown in Fig. 1, a normal curve represents an almost perfectly symmetrical distribution. Spreading the data skewness evenly can improve the performance of learning algorithms [6]. Fig. 1 presents a normal distribution of data in which both the positive and negative classes are evenly distributed. In such a scenario, classification algorithms may show no bias when classifying both the positive and negative classes. In contrast, Figs. 2 and 3 present data with positive and negative skewness, respectively, which may cause classification algorithms to be biased towards the majority class. Although previous studies, such as [53, 61, 62, 65], have proposed means of addressing data skewness, this issue remains unresolved in general in machine learning studies. Data skewness can cause learning algorithms to produce poor and unreliable classification results [69]. In a normally distributed data set, the mode is typically equal to the median and mean. For a positively skewed data set, the mode is less than the median, which is less than the mean. In contrast, in a data set with a negative skew, the mode is greater than the median, and the median is greater than the mean. Logarithmic and automatic transformation schemes have been applied to address data skewness in previous studies, such as [39, 40, 61, 62, 65]. 1.1.4 Irrelevant and redundant features in data sets Most data sets contain irrelevant features, which can affect the performance of learning algorithms [70, 71]. Training such algorithms only on relevant features can yield optimal results in machine learning studies. By the same token, irrelevant features can affect the accuracy of learning algorithms. Therefore, to encourage unbiased prediction outcomes, classification algorithms should be trained on an ideal set of features characterised by relevant information, simplicity, and independence. Techniques for selecting appropriate features from existing data sets can be applied to identify the most suitable subset of features related to a particular problem [72]. In addition, feature selection offers the ability to minimise the dimensionality of the data [73]. Selecting the most relevant attributes thus simultaneously removes irrelevant features from a data set and enables better data management; for this reason, feature selection is essential when addressing imbalanced data [70, 71]. To address the issue of irrelevant and redundant features in data sets, feature selection approaches have been applied in many previous studies, for instance [10, 44, 56, 74-79]. 1.1.5 Continuous data Continuous data are data whose values can be measured in and divided into infinitely small increments, unlike discrete data, which take specific values. The majority of the existing data sets used in machine learning studies are characterised by continuous features, which can make it difficult to properly report these data in a discrete fashion [80]. Furthermore, the continuous nature of the existing data can lead to uncertainties in the results produced by learning algorithms. Thus, for most learning algorithms to produce more reliable classification outputs, a discrete feature space is required [81]. To address the issue of continuous data, Ching et al. [82] proposed a new information-theoretic data discretisation technique. Ma et al. [83] and Chen et al. [84] also applied discretisation techniques to continuous data. Discretisation transforms quantitative data into qualitative data by partitioning the features into a small number of non-overlapping intervals generated by implementing certain boundaries. Each feature value is assigned to its corresponding interval, thus making the data discrete [85]. 1.1.6 Data privacy Obviously, most project owners find it difficult to disclose their data sets as a result of the confidentiality of their projects [34, 86], although such projects may contain information that could be used to improve knowledge discovery in machine learning studies. Consequently, due to the privacy issues associated with these projects, such data sets are rarely made available to the machine learning community. Privacy concerns give rise to several controversial issues since the release of such information depends solely on the project owners or other responsible parties, who may not always be able to guarantee that their data sets are used for the intended purposes [87]. However, as privacy issues continue to hinder the release of relevant data sets, these concerns continue to pose a serious challenge in the big data mining community, which may sometimes require access to sensitive information in order to produce reliable results [54]. Consequently, encryption has been adopted by most researchers to ensure data privacy [88, 89]. In addition, multiparty data sharing has been applied to address the privacy issue, as reported by Peters et al. [90]. 1.1.7 Collinearity among metrics In many existing data sets, some of the attributes are collinear, as reported in [91-96]. In other words, these metrics are correlated with each other. Such metrics are commonly used in regression models for defect prediction [97]. In multiple regression, however, the variables are usually independent; for instance, in the expression (1)the variables and are treated as being independent of each other. However, multicollinearity will arise if and are correlated. In the above expression, Y represents the dependent variable and, as such, depends on and , which indicates that and contain relevant information about Y. If and are collinear, this situation can lead to redundancy and collinearity, which will hinder the ability to generate accurate prediction outcomes. When similar variables that are collinear are applied in a prediction model, the correlation that exists among the supposedly independent variables tends to reduce the accuracy of the model [34]. Kamei et al. [98] addressed the issue of collinearity among metrics by removing highly correlated metrics, whereas Jiarpakdee et al. [99] applied the combination of clustered variables and the variance inflation factor technique to address the collinearity issue. In addition, a principal-component-based approach for addressing the collinearity among metrics has been proposed, as reported by the authors of [100-104]. 1.1.8 Noise in data Noisy data will not produce meaningful outputs when they are used in prediction studies. Noisy data are usually associated with outliers. Outliers are data points that lie far away from the main cluster(s) of data. Outliers may occur in data sets as a result of measurement variations or may be a manifestation of experimental error. The removal of outliers can lead to a reduction in the noise found in a data set [53, 105, 106]. Noise filtering has also been reported to be an effective means of eliminating noise in a data set, as reported by Hosseini et al. [34] and Kang et al. [107]. 1.1.9 Missing values in data sets Most data sets used in machine learning studies tend to contain missing values. Missing values are data that were not stored or recorded as a result of faulty sampling, which often occurs due to human or computer error. Missing values are often difficult to avoid because they are identified only during data analysis, and as such, most researchers face difficulties when dealing with missing values [85]. If missing values are not adequately handled, they may lead to poor knowledge discovery or incorrect research findings. Notably, important information can be lost as a result of missing values [108]. Several approaches have been proposed to address the issue of missing values, as reported by Luengo et al. [109]. In addition, in the course of this SLR study, we uncovered an unpublished article that proposes an optimal decision technique capable of addressing the issue of missing values to aid in data preprocessing for machine learning studies. 2 Methodology This section presents the methodology applied in this SLR study. This study follows the original guidelines proposed by Kitchenham and Charters [110]. The phases of our methodology are as follows: Planning Formulation of RQs Search process QA Data collection Data analysis 2.1 Planning This subsection presents in detail the initial stage of our SLR study. During this starting phase, we identified the needs of the study; planned our search process; structured the scope of the study, including our search string and RQs; and selected the electronic databases to be included in the study. We also set the target years of publication, limiting the period of interest to January 2010 to September 2018. This planning phase formed the foundation of our research methodology. 2.2 Formulation of RQs This subsection presents the RQs addressed in this SLR study. Importantly, one of the objectives of this study is to investigate the techniques applied in data preprocessing. Thus, the RQs addressed in our study are as follows: RQ1. How many primary SLR studies on data preprocessing have there been since 2010? RQ2. How do researchers perform and report the techniques they apply when performing data preprocessing? RQ3. How does a data preprocessing technique address a specific data-related issue? RQ4. What are the limitations of current research? To address these RQs, we considered several factors. For instance, to address RQ1, we first identified SLR articles in specific topic areas related to data preprocessing and identified the year and the journal or conference proceedings in which each such article was published. To address RQ2, we focused specifically on articles that addressed data preprocessing issues and the procedures used to implement related techniques. With respect to RQ3, we focused more on the results of the individual studies selected for investigation in the current study. For RQ4, which concerns the limitations of the current state of research in this area, we considered the following specific questions: RQ4.1. Are the research topics addressed by SLR studies on data preprocessing limited in number? RQ4.2. Is there evidence that SLR studies on data preprocessing are limited due to a lack of primary studies? Motivation for RQs. Regarding RQ1, previous studies, such as [17, 20, 111-117], have indicated that the existing data sets are imbalanced. However, it has been argued that this class imbalance does not have a significant impact on the performance of learning algorithms and that instead, more emphasis should be placed on the quality of the data sets used [9, 30]. Therefore, it is important to investigate the number of SLR studies that have systematically reported data issues and preprocessing techniques for addressing such issues. RQ2 was formulated for the sake of knowledge discovery; we were motivated to investigate how researchers carry out data preprocessing activities as well as how they report such activities to gain a better understanding of the data preprocessing applied in machine learning studies. Relatedly, the need for a clear understanding of the data cleaning approaches existing in the literature and a desire to ascertain which approaches are more effective motivated RQ3. With respect to RQ4, we were motivated by the need for a clearer understanding of the availability of SLR studies on the approaches applied for data preprocessing and a desire to encourage other researchers to engage in more SLR studies concerning data preprocessing, cleaning, preparation, or sanitisation to make data sets more suitable for use in machine learning studies. 2.3 Search process This subsection describes how each article was identified in our study. Articles from journals and conference proceedings were extracted from electronic databases using the conventional manual search process. A list of the electronic databases searched is presented in Table 1. We also present a flow chart of the search process in Fig. 4. Table 1. List of electronic databases searched Database URL IEEE Explore www.ieeexplore.ieee.org SpringerLink www.link.springer.com ScienceDirect www.sciencedirect.com Web of Science www.webofknowledge.com Google Scholar www.scholar.google.com Wiley Online Library www.onlinelibrary.wiley.com ACM Digital Library www.dl.acm.org IET Software Digital Library www.digital-library.theiet.org Scopus www.scopus.com Fig. 4Open in figure viewerPowerPoint Search process flowchart Our search string was formulated to retrieve as many articles as possible related to data preprocessing, our topic of interest in this SLR study. We formulated and applied the following search string: (Data OR data set OR data sets OR imbalanced data OR imbalanced data set OR imbalanced data sets) AND (preprocessing OR cleaning OR cleansing OR preparation OR sanitisation OR purification) 2.3.1 Inclusion and exclusion criteria Here, we discuss the criteria used to determine whether an article would be included or excluded in our SLR study. Certain criteria were set for both inclusion and exclusion. For an article to be included in this study, it was required to have the necessary content and to be reported in the form of a publication written in standard English to ensure that we could clearly understand its content. Articles written in a different language than English would have been difficult for us to understand and, consequently, were not included in the current study. The inclusion and exclusion criteria for this SLR study are presented in Table 2. Table 2. Inclusion and exclusion criteria Inclusion criteria (paper must…) Exclusion criteria (paper must NOT…) (a) be a published article written in polished English (a) be written in a different language than English (b) have been published within the range of January 2010–September 2018 (b) fall outside the selected time period (c) address some data-related issue (c) focus on machine learning techniques, defect prediction models, effort estimation, or classification performance evaluation (d) report an approach for addressing data-related issues or an evidence-based or empirical study addressing the RQs (d) deviate from answering the RQs 2.4 Quality assessment (QA) This subsection presents the QA approach applied to evaluate the quality of each article selected. The essential purpose of such a QA is to ensure the high quality of SLRs concerning data preprocessing and to avoid biased research findings, thereby encouraging other researchers to strive for high-quality SLR articles. Through this QA approach, we were able to extract useful data from the articles selected in this SLR study. We rigorously adopted the University of York Centre for Reviews and Dissemination (CDR) Database of Abstracts of Reviews of Effects (DARE) criteria, which have been applied