1

Impact of Discretization Noise of the
Dependent variable on Machine Learning
Classifiers in Software Engineering
Gopi Krishnan Rajbahadur, Shaowei Wang, Yasutaka Kamei, and Ahmed E. Hassan
Abstract—Researchers usually discretize a continuous dependent variable into two target classes by introducing an
artificial discretization threshold (e.g., median). However, such discretization may introduce noise (i.e., discretization noise)due to ambiguous class loyalty of data points that are close to the artificial threshold. Previous studies do not provide aclear directive on the impact of discretization noise on the classifiers and how to handle such noise. In this paper, we
propose a framework to help researchers and practitioners systematically estimate the impact of discretization noise onclassifiers in terms of its impact on various performance measures and the interpretation of classifiers. Through a casestudy of 7 software engineering datasets, we find that: 1) discretization noise affects the different performance measuresof a classifier differently for different datasets; 2) Though the interpretation of the classifiers are impacted by thediscretization noise on the whole, the top 3 most important features are not affected by the discretization noise. Therefore,
we suggest that practitioners and researchers use our framework to understand the impact of discretization noise on theperformance of their built classifiers and estimate the exact amount of discretization noise to be discarded from the dataset
to avoid the negative impact of such noise.
Index Terms—Discretization noise, Discretization, Classifiers, Feature Importance Analysis, Performance, Random
Forest, Logistic Regression, Decision Trees, KNN.

F

1

I NTRODUCTIONM

A chine learning classifiers are widely used throughout software engineering studies. Some of the mostcommon uses of classifiers include predicting defects [23],[38], [52], [55], bug-fix times [33], understanding thefeatures that impact the defect proneness of a softwaresystem [9], [44], [48], [50].
Usually, classifiers are trained on labeled data points
and are used to predict the target class of the unlabeleddata points. In the absence of pre-defined class labels andthe availability of only the continuous dependent variable,
researchers usually discretize1 the continuous dependent
variable into artificial target classes. Such discretization
might be based on domain knowledge [25], a phenomenon

●

●

●

Gopi Krishnan Rajbahadur, Shaowei Wang and Ahmed E. Hassan
are with Software Analysis and Intelligence Lab (SAIL), School ofComputing, Queen’s University, Canada.
E-mail: {krishnan, shaowei, ahmed}@cs.queensu.ca
Yasutaka Kamei is with Principles of Software Languages (POSL)
Lab, Graduate School and Faulty of Information Science andElectrical Engineering, Kyushu University, Japan.
Email: kamei@ait.kyushu-u.ac.jp
Shaowei Wang is the corresponding author

1. Discretization is the process of turning numerical data into discrete
data with finite intervals [21]that the study wishes to observe [33], or in many cases when
there are no imposed target classes, an artificial discretization threshold is used to discretize the target variable into
binary (or n-ary) classes [13], [16], [67], [69].
However a plethora of prior studies note that the discretization of the continuous dependent variable could be
detrimental to the performance of classifier and may produce misleading results [10], [12], [15], [56]. One alternative approach to avoid such discretization noise would
be to train regression models on the continuous dependent
variable and then discretize the predicted outcome afterwards [55]. But as Rajbahadur et al. [55] observe, only
when there is a significant class imbalance in the dataset,classification through regression would yield better results.
Therefore, the discretization of continuous dependent variable with artificial discretization thresholds is still widelypracticed in software engineering as evidenced by [22], [25],[28], [31], [33], [59], [67], [69].
The other problem with such a discretization approach isthat the data points that are very close to the discretizationthreshold (e.g., median) get class labels that might not bereflective of the true class to which they belong. While many
previous studies explore the harmful impacts of discretizing
the continuous dependent variables on the performance ofdiscretization noise (both positively and negatively).
Therefore, we urge the researchers and practitioners
to use our framework to analyse if (and how much)discretization noise exists and how to address it.
2) Though the overall derived feature importance ranks
are impacted by discretization noise, the importanceranks of the top three important features are notaffected. Therefore, in absence of any impact on theinterpretation of the top x features detected by ourframework, one could include or discard the discretization noise in their datasets as recommended by ourframework. Especially without being worried about thediscretization noise’s impact on the interpretation of theclassifier.
Finally, we also provide the framework as an R package (and guidelines for using our framework) to provide
automated support to others who wish to revisit their prior
results or to consider discretization in the future studies.
Paper organization. Section 2 places our contribution
relative to prior work. Section 3 outlines our framework.
Section 4 presents the impact of discretization noise on theperformance of the classifiers and on the derived featureimportance ranks of several software engineering datasets.
Section 5 explores why discretization noise is sometimes
detrimental and sometimes useful. In Section 6 provide
a user guideline for our framework. We then discuss thethreats to the validity of our observations in Section 7.
Finally, we conclude our study in Section 8.

the classifiers [10], [12], [15], [56], the problem of data
points with ambiguous class labels and its impact on theclassifiers is completely unexplored. For instance, consider
the example of determining whether a bug was closed
fast or slow. A domain-expert might decide that any bug
closed within a week is a fast-closed bug. However, sucha discretization rule for the dependent variable would leadto noise for data points close to that 7-days threshold. For
instance, a bug that is closed within 7 days and 1 min would
be considered as a slow-closed bug. Such discretization
introduces noise in the data that is used for training theclassifiers. We define such noise as the discretization noise
and the data points whose ambiguous class labels thatgenerate the discretization noise as the noisy points.
In summary, discretization of continuous dependent
variable is both problematic and generates discretizationnoise. However, the practice of discretizing the continuous
dependent variable still remains a widely used practicein software engineering without any consideration to thegenerated discretization noise.
Therefore the main goal of our study is two-fold: first,
to introduce awareness among the software engineeringresearchers and practitioners about previously unexploreddiscretization noise. Second, we provide them with a framework - a systematic and rigorous method for exploring theimpact of discretization noise on their classifier of choice
for any given dataset.
We highlight the capability of our framework by conducting our study on four different types of software engineering datasets (Q&A websites data (4 websites), Linux
patch acceptance time data, bug-fix delay data, mobile app
ratings data), as a binary classification problem, where thediscretization noise is caused by the discretization of acontinuous dependent variable around artificial discretization thresholds. We applied our proposed framework tofour different families of classifiers (i.e., Random forestclassifier (RFCM), Logistic regression (LR), Classification
and Regression Trees (CART), and K-Nearest Neighbors(KNN)) to analyse the impact of discretization noise on
the various common performance measures (i.e., Accuracy,
Precision, Recall, Brier score, AUC, F-Measure, MCC). In
addition, we also analyse the impact of discretization noiseon the interpretation of classifiers in terms of the derived
feature importance ranks. The derived feature importanceranks is a rank list that reports the features of the dataset in
the order of their influence on the classification.
We highlight our findings and suggestions as follows:

2

R ELATED W ORK

In this section, we discuss two groups of prior work: efforts
that attempt to establish the impact of noise on classifiersand efforts that investigate problems with discretization onclassifiers.
2.1

Impact of Noise on Classifiers

Discretization noise while being similar to class noise [75],differs from it because class noise is caused by data points
with wrong class labels due to contradictory examples
and random misclassifications [75]. Whereas, we view discretization noise as data points with ambiguous class labels
due to their proximity to the discretization threshold. While
discretization noise is unexplored, a number of studies have
explored the impact of noise on classifiers [20], [35], [36],[60], [75]. For instance, Kim et al. [36] explored the impact
of class noise on defect classifiers and provided insights
on acceptable levels of noise in the data when buildingclassifiers and the sensitivity of the various classifiers toclass noise. Whereas Folleco et al. [20] explored the impact
of class noise on various classifiers when using software
quality data and found that the performance of the random
forest classifiers is the most robust. Similarly, studies outside software engineering have also found class noise to1) The impact of discretization noise is inconsistentacross multiple performance measures for different
datasets across all the studied classifiers. Though
the impact on Recall is the most pronounced (up to139%), other performance measures - Precision, Brierscore, F-Measure, and MCC are also impacted at least
up to 43.19% due to the inclusion or exclusion of2

variables or the class distribution of the dependent variable.
But in our study, we focus on studying the impact of
discretizing the dependent variable into meaningful target
classes based on itself and domain knowledge.

be detrimental to the performance of a classifier [54], [76].
On the contrary, Tantithamthavorn et al. [63] demonstrated
that the Precision of a classifier is not impacted by classnoise and the most influential features (i.e., features in the
top 3 ranks) remain the same irrespective of the amountof class noise that is present in the data. They suggested
that the insights from the constructed classifiers can be usedwithout worrying about the noise. However, they did notethat filtering the noise increases the Recall of the classifiers.
Different from prior studies, we are not concerned with
the impact of class noise and noise in the dataset that is
generated at random [26] or during data collection [49].
Rather, we provide a framework to analyse the impact of thediscretization noise that is generated due to the discretization of the dependent variable (in the absence of labeleddata) on the performance of classifiers and the derived
feature importance ranks of a classifier in a systematic andrepeatable manner.
2.2

2.2.2 Discretization of the dependent variable
Mockus [49] suggested that many of the software engineering datasets have poor data quality which affects the insightsthat are offered by the software analytical models. This
noise can arise from various sources in the case of labeled
data. But when the class labels for the data are not available
and a continuous dependent variable is discretized usinga threshold to generate class labels [13], [16], [67], [74],
we end up with discretization noise. But many researchers
actively have argued against the practice of discretization.
For instance, Altman et al. [5] and Cohen [10] pointed outthat discretization at the median of a continuous variableleads to a loss of information, and discretization at othercut points away from the center lead to a much greater
information loss. In addition, several prior studies argued
against the use of any data-driven cutpoint to discretize
dependent variable as it introduced noise and bias [12], [56],[57]. MacCallum [43] and DeCoster et al. [15] further statethat discretization is rarely ever justifiable and it is almost
always safer not to discretize. However, several researchers
also note that it is acceptable to discretize sometimes [14],[15], [17], [37]. In summary, though there are some benefits
associated with discretization, the majority of the research
agrees that it is not a safe practice and should mostly be
avoided.

Impact of Discretization on Classifiers

In software engineering, discretization has been used totransform both continuous independent and dependent variables into discrete classes [32], [42], [46], [47], [55], [74].
TABLE 1: Details of datasets used in the study
Dataset#Size#Features

Stack Overflow
Mathematics
Ask Ubuntu
Super User
Patch
Bug-delay
App-rating

55,85370,336
7,134
10,77620,000
2,434
7,365

28
27
26
27
22
23
22

R(dependent
variable)
0-9,981.40 mins
0-30,073.72 mins
0-31,638.55 mins
0-51,376.33 mins
0-1,266.92 days
-1,319.06*-1,990.27 days
1.41-4.97 stars

2.2.3 Discretization in software engineering
While a majority of the research community agrees that
discretization is not a safe practice as highlighted in the priorsection, discretization of the dependent variable still continues to be an accepted practice in software engineering andother fields. For instance, many software engineering studies
like [22], [25], [28], [31], [33], [59], [67], [69] discretize
the dependent variable to generate the outcome classes on
which machine learning classifiers are trained. Of these,some of the studies like [28], [67], [69] discard the dataaround the discretization threshold due to its ambiguousclass loyalties and use only the top and bottom x% to train
classifiers. But even while doing so, they do not provide any
clear theoretical or empirical reason for doing so. Whereassome other studies like [22], [25], [31], [33], [59] split thecontinuous dependent variable on the various criterion andinclude all the data points without any consideration for thenoise around the discretization threshold.
Therefore, our study is the first study in the field ofsoftware engineering to investigate the effect and impact of
discretization noise on the performance and interpretationof classifiers. For the researchers and practitioners who
continue to discretize the dependent variable artificially*The dependent variable has negative value since some developers started
fixing a bug before the bug was reported.
R(x) - Range(x)2.2.1 Discretization of the independent variable
Yang et al. [71] and Garcia et al. [21] have shown thatthe discretization of the independent variables improves
the performance of machine learning classifiers. True tothat result, many studies in software engineering discretize
the independent variables as part of data pre-processing
before constructing classifiers [42], [46]. For instance,
Jiang et al. investigated and found that while discretizationof independent variables improves the performance of someclassifiers, it did not universally benefit all classifiers [32].
Contrary to those studies, Nam and Kim used a medianbased discretization of independent variables to assign classlabels for un-labeled data points to train a cross-project
defect prediction model [53]. However, except for univariate
discretization techniques, the discretization techniques usedin the aforementioned studies cannot be used in our study,as they focus on discretizing an independent variable basedon the information that is contained in the other independent3

to build classifiers, we propose a framework. They canuse our proposed framework to analyse the impact thatthe generated discretization noise has on their classifiers.
Furthermore, we also shine the spotlight on the noise that is
generated due to the discretization of the dependent variable
for building classifiers - which generally is ignored in thefield of software engineering.

discretize the dependent variable with respect to an artificial threshold (a.k.a a cutpoint) into two response classes:“class1” and “class2”. We then assign the “class1” classlabel to all the data points with a dependent variable that has
a value that is less than or equal to the chosen discretizationthreshold (e.g., median). The remaining data points are
assigned the class label “class2”. The artificial thresholdfor such a discretization could be chosen in multiple ways.
The threshold could be domain specific and be defined by
the experts (e.g., ideal bug-fix time for a specific project as
defined by the software engineers working on the project).
Alternatively, in the absence of such an established domain
specific discretization threshold, many of the prior studies
have resorted to various heuristic, intuitive and alternate
thresholds for discretization [5], [13], [16], [67], [69]. But
irrespective of the choice of the discretization threshold,the data points close to the discretization threshold produce
discretization noise. Our framework analyses the impact of
discretization noise generated by any such discretizationthreshold.
In this study, to demonstrate the generalizability andapplicability of our framework, we use three artificial discretization thresholds.
Median based discretization Threshold (MT): Many priorstudies use median for discretizing the dependent variable
into binary classes [13], [16] and it is often used in theabsence of explicit domain knowledge about the classes ofa dependent variable [5].
Univariate Clustering based Threshold (CT): Univariate
clustering is an automated technique for discretization.
Univariate clustering splits the dependent variable into
multiple groups in an optimal fashion. We use Wang andSong’s implementation optimal k-means clustering in one
dimension (ckmeans.1d.dp2 ) here [68]. The ckmeans.1d.dp
divides data in one dimension into k clusters so that the sumof squares of within-cluster distances from each element to
its corresponding cluster mean is minimized [68]. We set kequal 2 since we wish to divide the dependent variable into
two classes.
CART based discretization Threshold (RTT): We use theregression tree approach as described by Breiman [7].3
Here, we use the continuous dependent variable of ourdataset as both the independent and the target variable forthe regression tree. We then use the generated regression
tree’s root node as the threshold for discretization since we
attempt to split the dependent variable into two classes.
The generated discretization threshold is used for discretizing the continuous dependent variable into binaryclasses.
Noisy area estimation: Once the dataset is discretized,
we need to define the area of the dataset which contains
discretization noise as the noisy area. Domain experts could

3 F RAMEWORK FOR U NDERSTANDING THE
I MPACT OF D ISCRETIZATION N OISE
An overview of our framework for understanding theimpact of discretization noise is presented in Figure 1. The
framework consists of six steps. Due to space constraints,
we detail the steps below (with more explanations where
needed) with a running example in Appendix B to better
demonstrate the use of our framework.
The individual steps of our approach are explained indetail below.
3.1

Step 1: Correlation & Redundancy Analysis

In this study, we collect data based on two criteria: 1) the
dependent variable is continuous; 2) the dependent variable
lacks a clear-cut threshold for discretization. Based on thesetwo criteria, we collect 4 types of data (7 datasets): Q&A
websites data [69], Linux patch acceptance time (Patch)data [33], Bug-fix delay time (Bug-delay) data [72], andMobile app rating (App-rating) data [67] (More details
about the datasets are given in Appendix A). Table 1
contains basic information about the number of data points
and the independent features of each of the studied datasets.
We perform correlation and redundancy analysis on the
independent features of a studied dataset to remove correlated and redundant features from the dataset, thereby notbiasing our feature importance results [62]. We do so insteadof using other common and state of the art dimensionality
reduction techniques like PCA, since, dimensionality reduction techniques like PCA combine and transform theoriginal features into principal components, which are nolonger directly interpretable. Finally, a recent MSR study
by Ghothra et al. [24] showed that correlation-based feature
selection is very robust for software engineering datasets.
Though we use and recommend correlation and redundancy
analysis, our framework supports the use of other methods.
We do not pre-process the independent features of thedataset any further in our study. However, practitioners canperform other data pre-processing steps like imputation if
required.
3.2

Step 2: Discretization

Threshold estimation: The primary objective of our framework is to understand the impact of discretization noise. We

2. https://cran.r-project.org/web/packages/Ckmeans.1d.dp/index.html
3. https://cran.r-project.org/web/packages/rpart/index.html

4

TABLE 2: Estimated discretization threshold, limits and % of data points in the noisy area for the datasets considered in
the study.
DatasetThreshold

SO
MA
AU
SU
PH
BD
AR

21.83 Mins
30.28 Mins
39.74 Mins
30.14 Mins
1.31 Days
0.67 Days
4.03 Stars

MT
Noisy
area (%)
29
41
38
31
10
6
43

Limit

Threshold
55
70
70
60
30
50
7

136.18 Mins
154.48 Mins
329.28 Mins
193.06 Mins
0.06 Days
0 Days3.86 Stars

CT
Noisy
area (%)
34
38
54
62
4
*
50

Limit

Threshold
85
85
95
95
40
*
10

214.81 Mins
502.98 Mins
338.93 Mins
262.53 Mins
9.48 Days
47.29 Days
3.74 Stars

RTT
Noisy
area (%)
53
44
53
56
22
54
63

Limit

step size
95
95
95
95
60
100
15

5
5
5
5
5
5
0.5

*The automated noisy area estimation algorithm found no data points in the noisy area
MT- Median based discretization Threshold, CT- Univariate Clustering based discretization Threshold, RTT- CART based discretization threshold
Datasets: SO- Stack Overflow, MA- Mathematics, AU- Ask Ubuntu, SU- Super User, PH- Patch, BD- Bug-delay, AR- App-rating

DatasetStep 4:
Performance
evaluation

Step 6: Inference validation

Step 1

Remove noisy area
in increments of
step_size until limit

Correlation
anlaysis

Redundancy
analysis

Threshold
Extremes
Noisy area

Training
data

PerformancemeasuresFeatureimportanceanalysis

FeatureimportanceClassiﬁer
Use the wholedatasetGeneratebootstrapsamplePerformance
calculation

Testingdata
Step 3: Classiﬁer construction

Step 2: Discretization

Step 5:
Featureimportancecalculation

Fig. 1: Overview of our framework.

determine a specific range of values around the discretization threshold to be noisy and this could be used as the noisy
area. But as we lack deep domain expertise of the datasets
considered in this study (which might be the case formany practitioners), we present Algorithm 1: an automatedalgorithm for estimating the noisy area in a given dataset.
We define the noisy area as the data points whoseclass loyalties are hard to discern due to their proximity
to the artificial discretization threshold. Such a hypothesis
follows from the rationale of prior studies where the data
points around the discretization threshold were discarded
to provide better class separation in the training data [1],[2], [34], [67], [69]. Algorithm 1 takes the dataset, cutpoint(i.e., the discretization threshold), and step size as input
parameters. step size controls the granularity of the analysis(i.e., the size of the increment from the cut point) - a
smaller value of the step size allows for a finer estimationof the noisy area, whereas a larger value provides a coarse
estimation of the noisy area. The step size used for all thedatasets in this study is given the Table 2.
Line 1 to 3 of the algorithm establishes the initial candidate noisy area, by selecting the area around the cutpoint.

More specifically, we consider the points within the area
cutpoint ± cutpoint ∗ 100% as the candidate noisy area.
We do so for two reasons: 1) most of the discretization noise
would be concentrated around the discretization threshold
due to its proximity to the threshold. 2) if we consider
more data, we might not be able to ascertain if the impactof the noisy area on the performance and interpretationof a classifier is due to discretization noise or the highvolume of data that is lost. Through line 4 to line 9 weincrementally subset the dataset into the quantum of size
given by cutpoint ± cutpoint ∗ setp size and computethe non-linearity of the quantum.
Non-linearity is one of the complexity measures definedby Ho and Basu [29]. Non-linearity score attempts to quantify how hard it might be for a classifier to classify the data
points (please refer Section 5 and Table 4 in the appendix for
more details about complexity measures). Once we establish
the non-linearity for all the quanta, we take the quantum
with the maximum non-linearity as the noisy area for ouranalysis and the step size that yielded the quantum as thelimit, which we use to demarcate the noisy area. We use
the maximum non-linearity to demarcate the noisy area as5

feature importance computation method as our framework
studies the impact of discretization noise on both the performance and feature importance. Therefore we demonstrate
the capability of our framework to analyse the impact of
discretization noise on random forest classifier (RFCM),
Logistic Regression (LR), Classification and Regression
Tree (CART), and K-Nearest Neighbour (KNN). All of thechosen classifiers are hyper parameter tuned to ensure thebest and stable performance. We used the method used by
Tantithamthavorn et al. [64] to hyper-parameter tune all of
our classifiers.
Though we use the four aforementioned classifiers, one
can use other classifiers instead of these classifiers without
any changes to the other steps in the framework.

cutpoint

Bottom x%

cutpointcutpoint*Limit

cutpoint+
cutpoint*Limit

Top x%
Data(sortedbased onthe valueof thedependent
variable)

Extremes

Noisy area Noisy area

Extremes

Fig. 2: Extremes and noisy area definitions of a dataset.

it indicates the quantum with the highest data complexity(thereby harder for the classifier). Figure 2 demonstrates
how the limit value is used to demarcate the noisy area
in a dataset. We present the limit that is generated for
demarcating the noisy area of all the studied datasets forvarious discretization thresholds in Table 2.
Extremes estimation: Finally we establish the data points
with the least discretization noise as the extremes. These
data points typically have high discriminative power as they
are the furthest away from the discretization threshold. Extremes are typically the data points that are associated withthe top and bottom x% of the sorted continuous dependent
variable. Prior studies usually consider the top and bottom
x% as data points that are devoid of noise and use themfor constructing the classifier [67], [69]. We use x as 10%
in this study. But the framework allows using any value
without any further change to the overall methodology.

3.4

Step 4: Performance Evaluation

In this step, the desired classifier performance evaluation measures are chosen. In this study, we observe andevaluate the performance of the constructed classifiers on
Accuracy, Precision, Recall, Brier score, Area Under thereceiver operator characteristic Curve (AUC), F-measure,and Mathew’s Correlation Coefficient (MCC), since manyprior studies studied the performance of classifiers using
these measures [6], [8], [73]. We calculate these measureswith “class 1” as the relevant (positive) class.
Though we demonstrate our framework on the aforementioned performance measures, our framework allows
users to use any performance evaluation measures (bythemselves or in combination with other measures).

Algorithm 1: Automated noisy area estimation
algorithm
3.5

Input: dataset, cutpoint, step size
Output: limit
Result: Estimates the noisy area in the data automatically bycomputing the limit
1 lower limit = cutpoint − cutpoint ∗ 100%
2 upper limit = cutpoint + cutpoint ∗ 100%
3 noisy area =SU BSET (dataset, lower limit,upper limit)
4 while ((cutpoint ± cutpoint ∗ step size) ≤upper limit AN D ≥ lower limit) do
5quanta = SUBSET(noisy area, cutpoint − cutpoint ∗step size, cutpoint + cutpoint ∗ step size)
6
nl score = COMPUTE NON LINEARITY(quanta)
7
results[step size] = (nl score)
8
step size += step size
9 end
10 limit = Indexof MAX(results)

3.3

Step 5: Feature Importance Calculation

We use the default feature importance calculation techniquethat is associated with each of the studied classifiers tocompute the feature importance for each classifier. We use
the variable importance computation method VarImp() ofcaret package to compute the feature importance of thestudied classifiers.

3.6

Step 6: Inference Validation

To ensure that the conclusions that we draw about ourclassifiers are statistically robust, we use the 100 outof-sample bootstrap validation technique, which yields an
optimal balance between the bias and variance as suggested
in the recent study of Tantithamthavorn et al. [65].
The out-of-sample bootstrap process is repeated 100
times. After the bootstrap validation, 100 performance measures and 100 lists of derived feature importance ranks are
generated. We carry out further analysis on these generated
performance measures and the derived feature importanceranks to investigate our research questions.

Step 3: Classifier construction

To study the impact of discretization noise, we constructa classifier on the whole dataset and on the dataset with
the noisy area removed. One can choose any classifier of
their choice in this step. In our study, we consider the 6classifiers considered by Rajbahadur et al. [55]. From the
6 classifiers, we choose the classifiers that have a default
6

3.7

Framework Deploymenton the performance of a chosen classifier. We specificallydraw attention to the classifier construction (step 3) of
the framework (see Figure 1). In order to ascertain theperformance impact of discretization noise on the classifiers,
we train the chosen classifier on data after excluding incremental amounts of discretization noise. More specifically,
we discard data points in windows which are defined as
cutpoint ± cutpoint ∗ x/100 and use the retained data as
the training data to build a classifier, where x varies from 0
to limit in increments of step size (as mentioned in Step
2 of our framework (See Section 3.2)). The step size canbe different for different datasets depending on the limit
used to define the noise area for a particular dataset. Table 2
presents the various limit and step size values used for
different datasets in our study. We perform this incremental
analysis (see Fig 1 of Appendix) on all the studied datasets
for the three different discretization thresholds (MT, CT,
and, RTT) considered in the study as given in Section 3.2
for all the four chosen classifiers (RFCM, LR, CART, and,KNN).
To measure whether the performance of a chosen classifier is impacted by removing data points in the noisy area
(data containing discretization noise), we use a Wilcoxon
signed-rank test [70], since it is a non-parametric test
without any assumptions about the underlying distribution.
Furthermore, to quantify the magnitude of the performancedifferences between the performance of the classifier withno data points removed and the classifier with data points
in the noisy area removed, we use Cohen’s d effect size
test [11]. The threshold for analyzing the magnitude is asfollows: ∣d∣ ≤ 0.2 means magnitude is negligible, ∣d∣ ≤ 0.5
means small, ∣d∣ ≤ 0.8 means medium and ∣d∣ > 0.8 means
large.
We perform these statistical tests between the performance measures of the classifier that is constructed on thewhole data and the classifier constructed on each step of
the incremental analysis (where noisy points in the noisy
area is incrementally removed). We do so to estimate howmuch data needs to be discarded to observe a statisticallysignificant impact on the performance of a classifier. The x
value (of the cutpoint ± cutpoint ∗ x/100 used for discarding data in the noisy area) for which different performancemeasures get significantly impacted is reported. If discarding the whole of the noisy area does not create a statisticallysignificant impact for a particular performance measure then0 is reported instead of x to signify that the discretizationnoise does not have a significant impact on that particularperformance measure for the studied classifier.
Results: The impact of discretization noise on the performance of different classifiers varies across datasets.
Similar performance impacts could be observed for the discretization noise generated by all the discretization thresholds considered in the study. Therefore, we only report
the impact of discretization noise generated by the median

We use our framework of 6 steps on any given dataset toanalyse the impact of discretization noise (as demonstrated
in Section 4) along with performance and interpretation on
the chosen classifier. Step 1 removes the correlation andredundancy among the features in a dataset, while step 2
is pivotal for estimating the noisy area and extremes for achosen discretization threshold. Steps 3 to 6 are repeatedby incrementally discarding data points in increments ofthe step size parameter (smaller step size enables finer
analysis and vice versa) in the noisy area around thethreshold until all of the data points in the noisy area are discarded. Such an incremental analysis helps the framework
identify the impact of discretization noise and determine
the exact amount of data from the noisy area that needs
to be discarded for a given dataset, discretization thresholdand classifier of choice. We also provide an R package4 ofour framework to enable others and practitioners automatedsupport to use our framework with trivial effort.

4

U NDERSTANDING THE I MPACT OF D IS -CRETIZATION N OISE ON THE P ERFORMANCEAND I NTERPRETATION OF A C LASSIFIER

4.1 Studying the Impact of Discretization Noiseon the Performance of a Classifier
Motivation: It is intuitive to expect that discretization noise
might impact the performance of a classifier. Ferri et al.
show that different performance measures are impacted
differently by different types of noise in a dataset [18].
Therefore, first, it is essential to establish if discretizationnoise impacts the performance of a classifier like othernoises. Second, if the discretization noise does impact theperformance of a classifier, we need to analyse how thediscretization noise in a dataset impacts the performanceof a classifier (either positively/negatively) in different performance measures. Finally, it is essential to establish howmuch data do we have to discard to avoid the impact of
discretization noise (as opposed to using only the top andbottom x%).
In order to enable researchers and practitioners to
perform such an analysis in a generalizable fashion, we
propose our framework. Our framework enables researchersand practitioners to examine the impact of discretizationnoise on the performance of various classifiers using avariety of software engineering datasets, across a multitudeof performance measures.
Approach: We employ our proposed framework (see Section 3) to perform an incremental analysis as mentioned in
Section 3.7 to estimate the impact of discretization noise
4. https://github.com/SAILResearch/suppmaterial-19-gopidiscretization noise impact

7

TABLE 3: Percentage of improvement in median performance of various classifiers with the noisy area removed overclassifiers with no data removed across various performance measures (The x value for which the performance impact firstoccurs for the given measure is also provided).
Classifier

RF

LR

CART

KNN

Dataset
SO
MA
AU
SU
PH
BD
AR
SO
MA
AU
SU
PH
BD
AR
SO
MA
AU
SU
PH
BD
AR
SO
MA
AU
SU
PH
BD
AR

ACC (%)
Mag
x
-0.65
50
-3.5
45
-7.76
40
-0.53§
0
-1.64
10
§
0.47
0
-0.78§
0
#
-0.76
55
-1.78
15
-5.54
50
-1.17
60
§
-0.08
0
-0.2
0
-0.36§
0
1.41
10
-1.67
10
§
-0.56
0
1.46
20
-1.12
10
1.2§
0
0.33
0
2.64
10
1.55
10
-0.15
0
1.78
20
-1.13
20
0.77
0
0.16
0

PRC (%)
Mag
x
5.07
15
8.96
15
12.09
30
5.88
25
1.36
25
0.72
0
-4.23
3
6.24
15
12.33
10
11.47
30
6.42
30
2.23
15
-0.39
0
-2.83
4
8.58
15
11.26
20
7.34
30
6.93
25
1.76
25
1.33§
0
-1.06#
6.5
6.02
10
10.45
10
4.12
50
4.28
20
-0.33
0
0.86
0
-0.62§0

RCL (%)
Magx-12.4215-32.2
15-99.02
25
-20.4
20
-6.64
10
0.75
0
13.17
2
-20.8120-46.87
10-139.29
25
-48.2
20
-4.68
10
§
1.63
0
10.82
2
-16.7720-37.1620-40.4930-21.36
45
-6.77
15
0.67
0
9.92
3.5
-9.88
25
-30.5120-46.1225-19.06
30
-4.09
15
0.44
0
11.54
2

BS (%)
Magx-8
5-11.77
5
-7.79
5
-7.82
5
-2.04
5
#
-0.66
40
-2.89
0.5
-7.73
5
-12.025-10.42
10
-5.87
10
-2.59
5
#
-0.78
40
-8.03
0.5
-2.77#
30
-6.54
60
-9.55
50
-7.32
55
§
-1.79
0
-0.77§
0-3.62§
0
-3.72
5
-6.74
5
-4.52
10
-3.04
15
-0.59
10
§
-0.22
0
-1.19
1.5

AUC (%)
Magx
00-1.23#
70
0
0
0§
45
-2.22
10
0§
0
0
0
§
0
0
0#
70
1.56§
0
0
0
§
0
0
0
0
0
0
3.85
20
0
30
0
50
2.99
25
-1.79
15
1.54§
0
1.64§
0
4.29
5
4.29
10
1.75#
55
1.69
15
-1.37
15
1.85
0
0
0

F-M (%)
Mag
x
-3.69
25
-11.6120-43.19
25
-7.59
30
-2.62
10
§
0.74
0
4.52
2
-7.04
25
-15.2815-59.7825-18.92
20
-0.97
15
0.98
0
3.95
2
-3.42
35
-11.9330-15.97
40
-7.05
50
-2.45
10
1.69§
0
4.35
3.5
-1.81
10
-9.91
10
-21.44
30
-7.01
40
-2.18
15
0.67
0
5.6
2

MCC (%)
Magx-0.054#55-6.38
50-18.64
45
0.36
0
-4.08
10
§
1.81
0
-2.93§
0
-0.83
0
-2.59
25
-16.3
60
-4.61
60
-1.28
20
-0.18
0
0.44
0
5.36
15
-1.27#
10
2.47§
0
9.52
20
-3.21
10
6.53§
0
3.79§
0
12.14
10
11.61
10
4.56§
0
17.31
20
-4.61
20
17.43
0
2.05
0

1. Performance Measures: ACC- Accuracy, PRC- Precision, RCL- Recall, BS- Brier Score, F-M- F-Measure
2. Datasets: SO- Stack Overflow, MA- Mathematics, AU- Ask Ubuntu, SU- Super User, PH- Patch, BD- Bug-delay, AR- App-rating
3. Mag- Magnitude of the performance impact |x- % of data of the noisy area when dropped starts statistically impacting the given performance measure
4. Cohen’s d effect size: Negligible - No formatting, Small -§ , Medium -# , Large - bold
5. ’−‘ indicates performance measure decreases due to removal of noisy area; ’+‘ indicates performance measure increases due to removal of noisy area
6. All the values with small, medium or large effect size are statistically significant with p ≤ 0.05
7. ’−‘in cases of Brier score indicates an actual increase in the Brier score and ’+‘ a decrease in Brier score (Lower the Brier score, the lesser the error)

based discretization threshold (MT) on various performancemeasures for all the four classifiers in Table 3 for brevity andspace constraints (See Table 1 and Table 2 of the appendix
for the performance impact due to other discretizationthresholds on the studied classifiers).

impacts the accuracy of the CART and KNN classifiers.
Such a varied impact on the different performance measures
for the different classifiers can be observed throughout (seeTable 3). Therefore we assert that different classifiers are
impacted differently (either positively or negatively) on
the studied performance measures and datasets.

Table 3 shows that the discretization noise impacts
different classifiers differently. For instance, in the case ofboth CART and KNN classifiers, for all the datasets except
for the patch dataset, the removal of discretization noise
improves the performance in terms of AUC. However, forthe patch dataset removal of discretization noise negatively
impacts the AUC measure. For the same classifiers, evenwhile the AUC and the Precision measures are positively
impacted by the removal of discretization noise for (5/7 for
CART and 6/7 for KNN), the Recall measure is negatively
impacted for 5/7 datasets. Similarly, for LR classifier, while
we observe no large impact on the AUC, we could observe
up to 139% impact on the Recall as we can observe from
Table 3. Furthermore, while the removal of discretizationnoise negatively impacts the accuracy of the RFCM andLR classifier, for the Stack Overflow dataset, it positively

Additionally, in Table 3 we also provide the x, whichtells us the percentage of data from the noisy area, that when
dropped, starts statistically impacting the given performancemeasure. This value aids the users of our framework to know
how much data they need to discard in order to avoid theperformance impact of the noise on a particular performancemeasure for a studied classifier.
We find that the removal of discretization noise has botha positive and negative impact on different performancemeasures for different classifiers. In addition, we do not
observe a generalizable trend in how the discretization noise
affects different performance measures. For instance, from
Table 3, we see that the removal of the noisy area fromdatasets negatively impacts the performance measures of an
RFCM for 6/7 datasets on Accuracy, Recall, Brier score,8

variable of all of the studied datasets is 12.99 and average
Kurtosis is 309.4). For example, let us consider the Ask
Ubuntu dataset with MT as the discretization threshold.
When the whole dataset is used (without any removal ofdata), the number of data points belonging to each classare equal. However, discarding the data in the noisy areaaccording to Table 2 for MT induces a class imbalance inthe dataset. The number of data points belonging to “class1”
only makes up 35% of the dataset. Such class imbalance
produced by discarding data induces a high degree of impact
on class-specific performance measures for the Ask Ubuntu
dataset as shown in Table 3.

F-measure, 2/7 datasets on AUC, and 5/7 datasets on MCC,
most of which in a statistically significant fashion with and alarge effect size. However, for 6/7 datasets, removal of noisy
area positively impacts the Precision in a statistically significant fashion with a large effect size (in most cases). Furthermore, while removal of noisy area negatively impacts
the performance measures for most datasets, it improves the
Accuracy, Precision, Recall, F-Measure, and MCC for the
App-rating dataset in a statistically significant fashion with alarge effect size. These varied impacts of the discretizationnoise on the different performance measures highlight theneed for our frameworks to study the peculiarities of eachcase study as the discretization noise affects the different
performance measures differently
Finally, we also note that magnitude of theperformance impact due to discretization noise
varies for different performance measures. For instance,in all of the classifiers, for most of the datasets, we could
see that while Recall is impacted heavily (up to 139% for
LR in Ask Ubuntu), other performance measures even if
impacted significantly, are not at the same magnitude (e.g,only -5.54% for accuracy in LR for Ask Ubuntu).

However, the more balanced measures like AUC and
MCC are more robust and insensitive to class distributions [39]. Therefore, AUC and MCC are impacted muchless severely by the discretization noise in the dataset. As
we can observe from Table 3, the magnitude to whichmeasures like AUC and MCC are impacted is lesser thanthe class-specific measures like Precision, Recall, and Fmeasure. For instance, while Recall is impacted by as muchas 139%, AUC is impacted only by at most 4.29% acrossall the datasets and classifiers. Even for the Ask Ubuntu
dataset, while the impact on class-specific performancemeasures is high, the AUC is seldom impacted. A similar
trend could be observed for MCC as they consider both
false positives and false negatives, even though they areslightly more sensitive to the discretization noise than AUC.
As Huang et al. [30] and Mossman [51] show balanced
measures like AUC are very stable and insensitive to noise
hence even a small impact in terms of magnitude (if theeffect size is large) could be significant. Therefore though
the absolute magnitude of the impact is small for some
performance measures like AUC and Accuracy, the true
nature of the impact needs to established by the practitioner.
Our framework seeks to provide a means for finding andmeasuring the impact. Finally, similar to other balanced
measures we observe that the impact on Brier score ismoderate (within 13%) when compared to class-specificperformance measures. Brier score being an error metriccalculates the mean squared difference between the actual
outcome and the assigned probability, which elucidates the
distance between the classifier’s predictions and the actualclasses in probability scale. Table 3 shows that Brier scoreis universally negatively impacted for a RFCM (signifying
an increase in Brier score). Because, though the noisy
points contain discretization noise, they also contain useful
information (see Section 5.1) and the removal of such information could universally affect the predicted probability
scores of classifiers, especially when they are robust to
noise [20]. However the Table 1 and Table 2 provided inAppendix C (for performance impact due to discretizationnoise that is generated by CTT and RT) shows a varied but
moderate impact (within 14%) for LR, CART and KNN,
which is similar to other balanced performance measures.

Discussion: We find that the magnitude of the impact
of some performance measures is much greater than otherperformance measures. Also, from Table 3 we note that fora given dataset and a classifier, some performance measures
are impacted even for small amounts of discretization noise(given by the x in the Table 3). Whereas, some otherperformance measures are more resilient. For instance, inthe case of the Mathematics dataset for the RFCM classifier,discarding 15% of the data in the noisy area significantly
impacts both Precision and Recall with a large magnitude
varying from 8.96% to -32.2%. Whereas even with a dropof 70% data from the noisy area the AUC gets impacted
marginally by -1.23%, which indicates that some performance measures are more resistant to discretizationnoise than others.
The different degrees to which different performancemeasures are impacted can be attributed to the different
nature of each performance measure and what they seek tocapture. For instance, Precision, Recall, and F-measure focus on capturing how good a classifier performs in predicting one of the classes [61]. Therefore, a potential imbalancein the dataset caused by discarding different amounts of datain the noisy area, even if it is discretization noise (as suchnoisy points contain useful information too), could impact
these measures greatly. Furthermore, Flach shows that thesemeasures are easily impacted by class imbalance [19]. These
explain the large impacts that we observe for the Precision,
Recall, and F-measures (as opposed to Accuracy whichtakes both the classes into consideration). Especially, withall of our datasets having a skewed distribution of thedependent variable (average Skewness of the dependent9In summary, unlike the class-specific performance mea- these classifiers (see step 3 of Section 3). We perform ansures, balanced measures are impacted moderately by incremental analysis on all the studied datasets for the three
the discretization noise.
different discretization thresholds (MT, CT, RTT) considTo conclude, discretization noise impacts different ered in the study in Section 3.2 (see Fig 1 of Appendix) for
performance measures differently for various datasets all the four chosen classifiers (RFCM, LR, CART, KNN).
with varying magnitudes for different classifiers (0We measure the derived feature importance values of139%) as shown in Table 3. We also note that balanced the chosen classifier trained on various data configurations.
performance evaluation measures are more resilient We use the Scott-Knott ESD test to rank [23], [41], [64],to the discretization noise whereas class-specific [65] the features with their feature importance values. To
performance evaluation measures are greatly impacted observe whether the derived feature importance ranks varyby the discretization noise in the dataset. Thus the significantly, we compare the derived feature importanceinclusion of discretization noise in the construction of any ranks of the classifier that is trained on the whole datachosen classifier could be either beneficial or detrimental and the classifier with x = limit (see Table 2) data points
depending on the performance measure of interest and the removed from the noisy area for each of studied dataset
dataset at hand. Such unpredictability demonstrates the across all the discretization thresholds. We compute theneed for our framework to better understand when difference between the derived feature importance ranks for
it is advisable to remove the noisy points, and how each feature in the dataset and compare them to the nullmuch of the data in the noisy area is to be removed to distribution (where the rank difference of each feature is
avoid impact on the studied performance measure for a zero) to see if removing the noisy area impacts the derived
chosen classifier.
feature importance ranks of a classifier [55].
For instance, lets consider the Stack Overflow dataset
The impact of noisy discretization data points is inconsiswith
MT as the discretization threshold when used for traintent across multiple performance measures for different
ing
an
RFCM. We wish to study the impact of discretization
datasets and different. Though the impact on classnoiseonthe interpretation of the constructed RFCM. For
specific measures (Precision, Recall,F-measure) is thesimplicity,
lets consider that the Stack Overflow dataset has
most pronounced (up to 139%) other performance meaonly
5
features.
For the RFCM that is intially trained onsures are also consistently impacted at least up to 60%
the
whole
dataset,a derived feature importance ranks isdue to the inclusion or exclusion of discretization noise
generatedforits
5
features(F whole = 3, 1, 5, 4, 2). Following(both positively and negatively). Hence, it is advisable towhich,
an
RFCM
istrainedon the Stack Overflow dataset
use our framework beforehand to carefully understanddevoid
of
the
noisy
area
(
F
noisy area removed = 2, 1, 3, 2, 4).
if the noisy discretization data points are to be used orThedifference
betweenF
whole and F noisy area removed is difdiscarded and how much of them needs to be discarded.
ference = 1, 0, 2, 2, 2. If the F whole and F noisy area removed
4.2 Studying the Impact of Discretization Noise are the same, then the difference generated would be zeroon the Interpretation of a Classifier
and would imply that the discretization noise does not
Motivation: Many prior studies use classifiers to understand impact the interpretation of a classifier. If the difference
the impact of features on the dependent variable [45], [66]. is not 0 (as in our example), we compare the difference
From Section 4.1, we observe that the data from the noisy against a zero distribution (null distribution= 0, 0, 0, 0, 0)
area sometimes impacts the performance of a classifier with a Wilcoxon-signed rank test and Cohen’s effect size
differently for different datasets. This could be because the test to determine whether if the difference is significant ordata in the noisy area contains useful information along otherwise.
In addition, even if the discretization noise impacts
with the discretization noise. Therefore removing/including
such data may also lead to a misleading interpretation the overall interpretation of a classifier, most researchersof a classifier. Therefore, we seek to observe how the and practitioners care only about the top x most important
discretization noise impacts the derived feature importance features [27], [40]. Therefore, in this section we present thewith our proposed framework to decide if such noisy data results of the top 3 features as a showcase. To measure how
points can be safely discarded or should they be included likely the rank of a feature could shift due to the removalof the discretization noise, we compute the likelihood ofnevertheless.
Approach: We demonstrate the capability of our framework rank shifts for top 3 ranks. The importance rank of a feature
to analyse the impact of discretization noise on the derived is ascertained by the median rank of the derived featurefeature importance ranks. We adopt the same incremental importance ranks for a feature of the dataset.
We use a bootstrap analysis to compute the likelihood
analysis approach that we adopted in the Section 4.1. But
instead of measuring the performance of the classifiers of rank shifts similar to prior study [64]. Figure 3 outlinesthat are constructed by incrementally discarding data from the process of estimation of the likelihood of a rank shift.
the noisy area, we note the feature importance values of The feature importance ranks that are generated from each10Features and theirderived featureimportance ranks

Incremental Feature analysis

Feature importance of aclassiﬁer constructed ondataset with 0% noise
removedScott-Knott ESD

Feature importance of aclassiﬁer constructed ondataset with 100 % (i.e.,
whole) noisy area
removedBootstrap analysis% of noise removed F1 F2 F3 ... Fn
0% noise
1 2 3
25
removed2% noise
1 2 1
26
removed...

... ... ...

...

Whole noisy area
removed2 3 326

Scott-Knott ESD

Ranks of the top 3 mostimportant features
F2
F3
1
2
2
3
...
...

F1
1
1
...
1

2

100

3

Bootstrap re-sampling
Estimation for the
likelihood of a rank shift

Generated ranks of all the features
for classiﬁers that are trained on
various data conﬁgurations

repeat 100 times

Fig. 3: The procedure for estimation of the likelihood of a rank shift.
TABLE 4: The likelihood of rank shifts in the top 3 mostimportant ranks (column A) and the comparison of thederived feature importance ranks of an RFCM trained on
the whole dataset (RankW ) and the dataset with the noisy
area removed (RankN R ) (column B).

classifier in the incremental feature analysis are taken as
the input. These derived ranks are then re-sampled withreplacement (i.e., Bootstrap re-sampling). The bootstrap resampled feature rank distribution is generated for all thefeatures of each of the studied datasets and they are reranked using the Scott-Knott ESD test as shown in the“Bootstrap analysis” part of Figure 3. This process is repeated 100 times. The intuition behind such a procedureis that the bootstrap re-sampling and re-ranking would
alleviate possible minor and insignificant fluctuations in
the derived feature importance ranks while highlighting
the pattern of significant fluctuations in the derived featureimportance ranks for a feature, and thereby bringing out
its true rank. Now we have 100 derived feature importanceranks for all the features in each of the studied datasets.
These 100 ranks (the output of the “Bootstrap analysis”
part of Figure 3) for a feature are used for estimating the
likelihood of a rank shift. The likelihood of a rank shift for
rank x feature is computed as the percentage of how many
ranks are not equal to x. For example, for rank 1 feature, outof 100 times 2 ranks are not equal to 1, then the likelihoodof a rank shift for that feature is 0.02. The estimation for the
likelihood of rank shifts is done for the features in the top
3 ranks of all the studied datasets. If the likelihood values
are high for a particular rank in a dataset, it indicates that
discretization noise impacts the derived feature importanceand the feature(s) reported at that rank should be interpretedwith caution in that dataset.
Results: The overall derived feature importance ranks
are impacted by the discretization noise for most of
the studied datasets. We only report the impact of discretization noise generated by median based discretizationthreshold (MT) on the derived feature importance ranks for
RFCM due to space constraints, However similar results
are noted on all the other classifiers (LR, CART, KNN)
on all the studies discretization thresholds (please refer toTable 3, 4, and 5 of Appendix C for other results). We show
a comparison of the derived feature importance ranks of an
RFCM that is trained on the dataset with and without the
noisy area removed in Table 4. The results highlight that

DatasetStack Overflow
Mathematics
Ask Ubuntu
Super User
Patch
Bug-delay
App-rating

Rank shift likelihood (A)Rank 1
Rank 2
Rank 3
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0

RankW vs. RankN R (B)p-value
Cohen’s d0-1.29 (L)0-2.14 (L)0-2.84 (L)
0.01
-0.74 (M)
0.03
-0.62 (M)0-1.39 (L)1-0.30 (S)for most of the studied datasets (6/7 datasets in the caseof the RFCM classifier and for all the datasets for otherclassifiers), the overall derived feature importance ranks ofall the features in a dataset, are significantly impacted (i.e.,p−value < 0.05) with a non-negligible effect size.
However, the derived feature importance ranks ofthe top 3 features are not impacted by the discretizationnoise. To specifically understand how much of an impactthat the discretization noise has on the most importantfeatures, we computed the likelihood of a rank shift for
the top 3 important features. We present the results inTable 4. For all the datasets, the ranks of the mostimportant features (i.e., all features at rank 1, rank
2 and rank 3) are not impacted due to discretizationnoise. We further observe that these trends are not specificto RFCM and the discretization noise generated by MT.
From Table 3, 4, and 5 (please see Appendix C) we observethat the most important features are not impacted by thediscretization noise generated in the dataset for any of thestudied classifiers.
In summary, though the overall ranks of derived featureimportance are impacted by the discretization noise in
the dataset, the top 3 (most important) features that mostresearchers and practitioners focus on [27], [40], [55], [63]are not impacted by the discretization noise. Therefore, wesuggest that the decision of either including or removing
11the data in the noisy area could be exclusively arrived
The noisy area contains useful information thatat from the results of Section 4.1 without being worried might help improve the performance of RFCMs. Frommuch about its impact on interpretation. Nevertheless Table 5, we observe that the RFCMs that are trained on thethese results might vary for other settings (e.g., other noisy area perform extremely well on the extremes for 5
datasets or classifiers) and our framework is able to provide out of the 7 studied datasets. Three datasets have an AUC
a case by case guidance.
that is larger than 0.95. For instance, in the Stack Overflow
dataset, the RFCMs that are trained on the noisy area have
The discretization noise (generated by our studied disan AUC of 0.96 when tested on the extreme areas.
cretization thresholds) does not impact the derived feaWhile the previous experiments show that noisy areature importance of any of the top 3 features yet it impacts
contains
useful information, we cannot conclusively estabthe overall derived feature importance ranks.
lish if the contained information in the noisy points amidst
the discretization noise could be successfully used by theclassifiers. To test if the studied classifiers can use the information contained in the noisy area, we construct classifiers
5 D ISCUSSION
that are trained with extremes and data from the noisy area
5.1 Why does a classifier trained on the whole (in contrary to the previous experiment, where we trained
dataset (with discretization noise) sometimes per- only on the noisy area) and then add increasing amounts ofform better than the classifier trained on data de- data from the noisy area. If the performance of a classifier
void of discretization noise?
does not degrade significantly (some degradation should
From the Section 4, we observe that excluding data from be expected) with the increased amount of noise, it may
the noisy area, i.e., data with discretization noise, some- indicate that the classifier is capable of capturing a signal astimes negatively impacts the performance of a classifier. long as there is enough information in the data. Therefore,
We seek to understand this counter intuitive phenomena. we could infer that despite the presence of discretization
Furthermore, in this section, we also remark about why the noise, the data points in the noisy area provide an additional
top 3 important features of a classifier are not impacted by signal to the classifier and the exclusion of the noisy area,
negatively impacts the performance of the classifier. On thediscretization noise.
We hypothesize that a classifier in some cases is able other hand, if there is a drastic and significant performance
to capture the signal from the noisy points in spite of degradation of the classifier, it would then invalidate ourthe discretization noise. Doing so, allows the classifier hypothesis that classifiers are able to learn an additional
to capture more information from the noisy data points, signal in spite of discretization noise.
in addition to information available from the clean data.
We perform the experiment by setting up a simulaTherefore, discarding those noisy points negatively impacts tion study with the help of our framework similar tothe performance of a classifier. To examine our hypothesis, the previous experiment, where we train all the classiwe start by constructing classifiers with data points from fiers on the extremes with different amount data fromthe noisy area (generated with MT) and test them on the the noisy area. We train our classifiers on four differdata from the extremes and noisy areas. Such an experiment ent data configurations which are given by (extremes +helps us understand whether the data with the discretization (noisyarea + over sample% ∗ (noisyarea)), where thenoise contains any useful information.
over sample takes values of 0, 100, 200, 300. We overWe follow the steps outlined in our framework (see sample different amounts of data from the noisy area
Section 3) to construct the classifiers on the noisy area and while keeping the amount of the data from extremes
generate the out-of-sample test sets from the extremes and constant. We build the classifiers on four data configuranoisy area separately. For this experiment, we construct an tions i.e., (extremes + (noisyarea + 0% ∗ (noisyarea)),
RFCM and observe its performance on the AUC measure. (extremes + (noisyarea + 100% ∗ (noisy area)),
We do so just on RFCM as our intention is only to analyse (extremes + (noisyarea + 200% ∗ (noisy area)), andif the noisy area contains useful information and our results (extremes + (noisyarea + 300% ∗ (noisy area)). For
on RFCM could help us test it succinctly. Furthermore, instance, in the Ask Ubuntu dataset from our study for MT,
in this section we report only the AUC measure as from the extremes have 1,427 data points and the noisy area has
Table 3 we could observe that AUC measure is the most 2,711 data points as shown in the Table 2. Therefore forresilient performance measure that has the least impact the first configuration, we would have 4,138 data points, ofacross all the studied classifiers and we wish to report which 65% is comprised of noisy points. Therefore for Ask
the impact on the most resilient measure. A high AUC Ubuntu dataset, our four data configuration consist of 65%,
would therfore objectively justify the presence of useful 79%, 85% and 88% noisy points respectively along with theinformation. However, all the other performance measures clean data from the extremes. (See Figure 3 of Appendix C
explaining the overall experimental setup)
follow the same trend.
12TABLE 5: Median performance (AUC) of the RFCMs on the different regions of the data.
Training data → Testing data
Noisy area → Extremes

Stack Overflow
0.96

Mathematics
0.96

Ask Ubuntu
0.86

Super User
0.86

Patch
0.98

Bug-delay
0.69

App-rating
0.76

TABLE 6: Performance comparison (in AUC) of classifiers
that are trained on different data configurations.

The performance of a classifier that is constructed on
the aforementioned data configurations is evaluated on theout-of-sample test data from the extremes. Note that theout-of-sample test data that is obtained from the extremes,
is not used in the training phase and is only used fortesting the constructed classifier. The experimental setup forconstructing a classifier is similar to that of our framework,as outlined in Section 3. Finally, we also capture the derived
feature importance ranks and observe the likelihood of rank
shifts for the top 3 most important features as outlined in
Section 3 and Section 4.2.
Adding data from the noisy area to the training
data does not greatly impact the performance of aclassifier. We report the results in Table 6. The columns
in Table 6 correspond to different data configurations that
we discussed earlier. We observe that the median AUC of aclassifier that is trained on the dataset without noise is quite
close to the AUC of a classifier that is trained on data with
300% noise. For RFCM, LR and CART classifiers, even an
addition of 300% of data from the noisy area only impacts
the AUC within 6% as we can observe from Table 6. Evenin the case of the KNN classifier, which is an instancebased classifier that is traditionally more sensitive to noisein the data [3], gets impacted only by 11% in terms of AUC
even with the addition of up to 300% data points from thenoisy area. These results signify that the performance ofthe constructed classifiers on the extremes does not degradesignificantly even when there is 300% (at least X% of thedata is noisy) data from the noisy area in addition to datafrom the extremes.
Furthermore, we also note that the likelihood of rank
shifts for top 3 ranks between classifiers that are trainedon the first configuration (0%) and the last configuration(300%) is 0. Which further reinforces the validity of ourhypothesis that the classifiers are able to capture the signal
in the noisy points despite the discretization noise and themost important features contributed by the true signal in
the underlying data are not perturbed by the discretizationnoise.
In summary, We establish that the noisy area does
contain some useful information. Further, we observe only
a maximum performance drop of 11% across 7 datasets forall of the studied classifiers (with less than 6% performancedrop for RFCM, LR and CART) with the addition of asmuch as 300% of data from the noisy area, where at least
more than 67% of the dataset is noisy. This suggests thata classifier is able to capture the information in the datain spite of the noise, thereby explaining why the exclusionof data points from the noisy area sometimes impacts theClassifier

RF

LR

CART

KNN

Dataset
SO
MA
AU
SU
PT
BD
AR
SO
MA
AU
SU
PT
BD
AR
SO
MA
AU
SU
PT
BD
AR
SO
MA
AU
SU
PT
BD
AR

0% Noise
0.96
0.96
0.86
0.86
0.99
0.69
0.76
0.92
0.93
0.78
0.79
0.98
0.68
0.72
0.89
0.87
0.74
0.72
0.94
0.64
0.64
0.80
0.80
0.62
0.69
0.83
0.55
0.60

100%
Noise
0.96
0.96
0.85
0.86
0.99
0.68
0.77
0.93
0.92
0.77
0.79
0.98
0.68
0.72
0.86
0.86
0.69
0.70
0.89
0.62
0.63
0.75
0.75
0.61
0.64
0.82
0.52
0.57

200%
Noise
0.96
0.96
0.84
0.87
0.99
0.66
0.76
0.92
0.92
0.76
0.78
0.98
0.66
0.71
0.84
0.87
0.67
0.69
0.90
0.60
0.62
0.71
0.72
0.59
0.62
0.81
0.51
0.56

300%
Noise
0.96
0.96
0.84
0.85
0.99
0.65
0.76
0.92
0.92
0.76
0.77
0.97
0.66
0.71
0.83
0.85
0.66
0.68
0.90
0.60
0.62
0.69
0.70
0.58
0.61
0.80
0.50
0.55

Datasets: SO- Stack Overflow, MA- Mathematics, AU- Ask Ubuntu, SU- Super
User, PH- Patch, BD- Bug-delay, AR- App-rating

performance of a classifier. In addition, the likelihood ofrank shift for the top 3 most important features is 0 for all
the classifiers, signifying that the discretization noise evenin such high quantities does not impact the interpretation of
the classifiers.
5.2 Why does inclusion of discretization noise
sometimes negatively impacts the performance ofa classifier?
Contrary to Section 5.1, in this section, we seek to understand why the inclusion of discretization noise negatively
impacts the performance of some classifiers. From Table 3
we observe that for all the studied classifiers, the inclusionof discretization noise sometimes negatively impacts theperformance of a classifier even though Section 5.1 showsthat data in the noisy area has useful information andclassifiers are capable of leveraging it. From Table 3 wealso observe that for all the studied classifiers and datasets,
at least one of the performance measure is negatively impacted. We hypothesize that such a negative impact could
be due to the high complexity (less discriminative power)of the noisy points around the discretization threshold,13

changes in different areas of the data without such compartmentalization of the data. The choice of using 5 quanta is
so that the compartmentalization is neither too granular nor
too encompassing. The 1st quantum contains most of the datafrom extremes and the 5th quantum contains most of the datafrom the noisy area, whereas 2nd to 4th quantum roughly
contain an equivalent amount of data points in between.
Finally, we compute the above-mentioned data complexity
metrics for the data points in each of these quantum andplot the results. (See Figure 2 of Appendix C explaining theoverall experimental setup)
From Figure 4, we observe that as we move from theextremes (1st quantum) towards the noisy area (5th quantum), we see a steady increase in data complexity acrossall complexity measures for all the datasets except for the
Bug-delay dataset. We see that all four complexity measures(i.e., Fischer’s discriminant ratio, linear separability, mixture
identifiability, and nonlinearity) are very high for the data
points in the noisy area compared to the data points in theextremes, and the inclusion of such complex data makes
it very hard for the classifiers to perform well. The steadyincrease in data complexity as we move across the quantum
can be attributed to the steady increase of the discretizationnoise in the dataset as we move from the 1st quantum tothe 5th quantum (the extremes to the noisy area). Therefore,
when the discretization noise (the data points in the noisy
area with high complexity) is discarded, the performance ofsome of the classifiers increases.
The lower complexity in the 2nd quantum for the Bugdelay dataset does not impact our findings. It is due to theway the dataset is split, the BoxCox transformation aims
to spread the dependent variable sufficiently so that theclass-wise binning yields data in all quantum. But for the
Bug-delay dataset, when we split the data into quantum,
we observe that the 2nd quantum has data points that only
belong to “class 2” and not “class 1” because the quantum
2 for the Bug-delay dataset contains only data points belonging to “class 2”, its complexity is very low, which isreflected in Figure 4. But this phenomenon has no bearing
on our findings that the quantum containing high volumes of
discretization noise (q5) is more complex than the quantum
containing extremes data (q1) and thereby discarding themsometimes improves the performance of the classifiers.
Hence, the presence of a high volume of discretizationnoise in the noisy area increases the data complexity, which
in turn results in the decreased performance of a classifierthat is trained with discretization noise, despite containing
useful information. Therefore, in some cases, the performance of a classifier benefits from discarding the data points
with from the noisy area.

L2

L2

0

0.0

0.2

5

0.4

10

F1

0.6

15

0.8

20

1.0

F1

3

4

5

1

2

3

QuantaQuanta

N2

N4

4

5

N4

0.0

Stack Overflow
Mathematics
Ask Ubuntu
Super User
Patch
Bug−delay
App−rating

0.0

−1.0

0.2

−0.5

0.4

N2

0.6

0.5

0.8

1.0

2

1.0

1

1

2

3
Quanta

4

5

1

2

3

4

5

QuantaFig. 4: Data complexity across quantum for the studied
datasets.
despite containing useful information. We arrive at sucha hypothesis as prior studies show that it is difficult forthe classifiers to perform well if the complexity of the datais high, irrespective of the contained information [4], [29].
Thus, we are interested in exploring if the negative impact
in the performance of a classifier due to the inclusion of
discretization noise is because of the high complexity of thedata points around the discretization threshold (noisy area).
Ho and Basu [29] provide complexity metrics to measure the complexity of data. We use these complexity
metrics to measure the complexity of different regions ofour data. From the multiple methods that are proposed by
Ho and Basu [29], we choose Fisher’s discriminant ratio(F1), linear separability (L2), mixture identifiability (N2)and nonlinearity (N4) as they are simple to explain and easy
to interpret. We briefly explain the metrics that we choosein the Table 6 of Appendix C. See the study by Ho andBasu [29] for more details about the computation of these
metrics.
We use the abovementioned measures to compute how
the complexity of the dataset changes across data points
as we move from the extremes to the noisy areas. We first
discretize the data into “class 1” and “class 2” classes as
outlined in Section 3.2. We then transform the continuous
dependent variable using the Box-Cox transformation [58]to alleviate the skew and increase the spread of the distribution of the dependent variable. We then split the data
into 5 quanta for each class using the bin function in R.
We do so to compartmentalize the data in relation to thecontinuous dependent variable and analyse the changes tothe complexity of the data points as we move closer tothe discretization threshold (we choose MT) for our casestudy. We would not be able to observe how the complexity

6

G UIDELINES FOR U SING O UR F RAME -

WORK

14

6.3

We explain in detail our framework in Section 3 anddemonstrate how it is used to study the impact of discretization noise on the performance and interpretation of aclassifier in Section 4. Furthermore this section, we provide
practical guidelines on how to use our framework and thebest practices to follow.
Figure 5 shows the involved steps, step-wise outputs,
user considerations at each step and the overall workflow
of our framework. A user can follow the steps one by one
when they are given a dataset to study.
6.1

In this section, we recommend the key best practices forothers to follow when they are discretizing the data using an artificial threshold. From Section 4.1 we note that
performance of all the classifiers is impacted across all
performance measures differently and that class-specificperformance measures (e.g., Precision and Recall) are moresensitive to discretization noise than others. Therefore, we
recommend the following best practices for the researchersand practitioners:
1) Irrespective of the choice of a classifier, if one intends todiscretize the continuous dependent variable into artificial classes, one should use our framework to analyse
if they should use the whole dataset or discard thediscretization noise.
2) Class-specific performance measures are more sensitive
to discretization noise. Therefore, instead of discarding a fixed amount of data like some of the previous
studies [28], [67], [69], we recommend the use of ourframework to estimate how much discretization noisethat one should discard to avoid any negative impacts on
their performance measure of choice. Hence, one could
avoid the unwanted loss of data.
3) If our framework reports a high likelihood of a rank
shift for one of the top n features, we recommend not
to trust the feature importance rank for that particularfeature and seek the opinion of the domain expert.
However, if our framework detects any impact in theoverall interpretation along with the performance, thenwe recommend the use of the interpretation of the best
performing model.

Performance impact estimation

A classifier constructed with increasing amounts of discretization noise being removed is constantly comparedagainst the classifier constructed on the whole dataset witha Wilcoxon signed-rank test and a Cohen’s effect size test
as outlined in Section 4.1. If for the chosen performancemeasure, the impact is statistically significant with nonnegligible effect size, then the amount of noisy points tobe discarded and the magnitude of the performance impact
due to the discretization noise is reported to the user. If thediscretization noise does not impact the chosen performancemeasure then our framework would output 0 (suggesting
no data needs to discarded) and recommend the use of thewhole dataset as outlined in the workflow of Figure 5.
However, the choice of the performance measure tofocus on and how much of an improvement/impact that one
should consider actionable depends entirely on the context.
For instance, in a dataset of 100 data points, if 90 data
points belong to “class 1” and 10 data points belong to“class 2”, then accuracy (w.r.t “class 1”) would be 90%
even if the classifier always predicts “class 1” for all the
100 data points. Therefore, other balanced performancemeasures like AUC might be required.
6.2

Best practices

7

T HREATS TO VALIDITY

External Validity Many of the prior studies highlight thatdifferent classifiers have different performance on the samedata [23], [55]. So the choice of classifiers might impact thefindings of our study, as we only use four classifiers (RFCM,
LR, CART, and KNN) in our analysis. However, the chosenclassifiers represent a diverse range of families: statisticalfamily, nearest neighbor family, Decision tree family andensemble family, i.e., 4/6 of the common classifier families
as outlined by Lessmann et al. [39]. We left out representative approaches from the neural networks family and the
support vector machine family. We did so, as classifiers
from these families typically do not have a default featureimportance measure.
Construct Validity Threats to construct validity pertains tothe suitability of the measures that are used in our study. In
our study, we study the impact of discretization noise that is
generated in the dataset by using three different discretization thresholds as mentioned in Section 3 and the results
might vary when another threshold is used. However, thethree chosen discretization threshold computation methods

Interpretation impact estimation

The derived feature importance ranks of the classifiers constructed on datasets with varying amounts of discretizationnoise being removed is computed. These computed ranks
are compared to see if the derived feature importance ranks
change between the classifiers that are constructed on thewhole dataset and the ones that are constructed on thedataset without discretization noise (please see Section 4.2
and Figure 5). Our framework then checks if the differences of the derived feature importance ranks between theclassifiers that are trained on the dataset with and without
the discretization noise are statistically significant. If they
are, our framework also calculates the likelihood of rank
shifts for the top n features (between the classifier trainedon the whole dataset and the data with the framework
recommended amount of data from noisy area removed).
In summary, our framework reports if there is an impact
of discretization noise on the overall interpretation and thelikelihood of rank shifts in for the top n features to the user.
15

Magnitude of performance
impact and x% of noise to
discardExtremes

Workﬂow

Cleaneddataset

Noisy
Area
Extremes

Step-wise
outputs

DatasetUser
considerations

Likelihood of a rank shift for
the top n features

100-out-of-sample-bootstrap until x
reaches limit (in increments of step size)
P<0.05 and EffectP<0.05 and EffectClassiﬁersize!= Negligible
size!= Negligible
constructed ondataset with x%data removed
Evaluate
Evaluate
statisticalstatisticalrobustness
robustness
Classiﬁer
P>0.05 and Effectconstructed on thesize= Negligible
whole dataset

Use wholedataset1. Threshold,extremes andnoisy area
2. Class labels

Constructedclassiﬁers

1. Magnitude ofperformance
impact
2. #noisy points
to be discarded

1. Likelihood ofrank shift (top x
features)2. Overall
impactInferences asvalidated over
100 out-ofsample
bootstrap

Correlation andredundancy
analysisDiscretization

Classiﬁer
construction

Performance
evaluation

Featureimportancecalculation

Validation with
100 out-ofsample bootstrap

Step 1

Step 2

Step 3

Step 4

Step 5

Step 6

1. Determine
discreizationthreshold
2. Estimate limit for
deﬁning noisy area
3. Deﬁne extremes

Choose theclassiﬁer to betested fordiscretizationnoise's impact

Choose therequired
performance
evaluation
measure(s)Choose therequired featureimportanceevaluation
measure(s)Decide if (and howmuch) data fromnoisy area should
be discarded fromthe dataset

Dataset without
correlated andredundant
featuresFig. 5: User considerations and workflow that are associated with each of the steps of our framework.

8(MT, CT, RTT) discretize the dependent variables differently and represent the most common ways of unsupervised
discretization of the dependent variable. Our framework
enables others to explore other discretization thresholds ina systematic manner.

C ONCLUSION

In this paper, we propose a framework to systematicallyand rigorously analyse the impact of discretization noise on
the performance and interpretation of a classifier within thecontext of their own domain. We perform a case study on avariety of software engineering datasets and we find that:
1) Discretization noise impacts the different performancemeasures of classifiers differently across the different
datasets. We observe that discretization noise leads toan up to 139% performance differences across various
performance measures across all the studied classifiers.
Hence it is very important for researchers and practitioners to use our framework to analyse the impact of
discretization noise on the classifier’s for before either
including or discarding it in their analysis.
2) When discretization noise negatively impacts the performance of a classifier, our framework provides asystematic and statistically robust way to estimate
exactly how much data should be discarded to avoid
discretization noise without incurring unwarranted data
loss.
3) Though discretization noise impacts the overall derived
feature importance ranks of a classifier, it does not
impact the derived feature importance ranks of the top
3 ranks for our case studies. Our framework provides aAnother construct validity in our study is the choice ofthe limit parameter for deciding the size of the noisy area
in each dataset. We used the limit values that are generated
by our automated noisy area estimation algorithm as given
in Section 3. Though such an algorithm estimates the noisy
area quantitatively based on a complexity measure, it might
not consider the inherent dataset characteristics and bias.
We acknowledge that it could be a potential threat and weurge researchers to explore various limits, or with limits thatare established by domain experts and ratify our findings.
Future studies should use our framework to test different
values for the limit.

Finally, in this section, we wish to reiterate to thereaders that our framework enables the researchers andpractitioners to fiddle with any components and try a varietyof combinations. We only define the needed analysis that is
to be done, so that the drawn observations are valid.
16

case by case guidance for others who wish to explore
its use for their own case studies.
R package and User Guideline: We provide an R package
to enable others to use our framework to analyse the impact
of discretization noise. Furthermore, we provide a user
guideline, a step-by-step walkthrough and the best practices
of using our framework in Section 6.

[25] P. J. Guo, T. Zimmermann, N. Nagappan, and B. Murphy, “Characterizing
and predicting which bugs get fixed: an empirical study of microsoft windows,” in Proceedings of the 32nd ACM/IEEE International Conference on
Software Engineering (ICSE-10), vol. 1. IEEE, 2010, pp. 495–504.
[26] F. E. Harrell, Missing Data. New York, NY: Springer New York, 2001, pp.
41–52.
[27] A. E. Hassan and R. C. Holt, “The top ten list: Dynamic fault prediction,” in
21st IEEE International Conference on Software Maintenance (ICSM’05).
IEEE, 2005, pp. 263–272.
[28] S. Hassan, C.-P. Bezemer, and A. E. Hassan, “Studying bad updates of
top free-to-download apps in the google play store,” IEEE Transactions on
Software Engineering, 2018.
[29] T. K. Ho and M. Basu, “Complexity measures of supervised classification
problems,” IEEE Transactions on pattern analysis and machine intelligence, vol. 24, no. 3, pp. 289–300, 2002.
[30] J. Huang and C. X. Ling, “Using auc and accuracy in evaluating learning algorithms,” IEEE Transactions on knowledge and Data Engineering, vol. 17,
no. 3, pp. 299–310, 2005.
[31] O. Jalali, T. Menzies, and M. Feather, “Optimizing requirements decisions
with keys,” in Proceedings of the 4th international workshop on Predictor
models in software engineering. ACM, 2008, pp. 79–86.
[32] Y. Jiang, B. Cukic, and T. Menzies, “Can data transformation help in thedetection of fault-prone modules?” in Proceedings of the 2008 workshop on
Defects in large software systems. ACM, 2008, pp. 16–20.
[33] Y. Jiang, B. Adams, and D. M. German, “Will my patch make it? andhow fast? case study on the linux kernel,” in Proceedings of the 10th IEEE
Working Conference on Mining Software Repositories (MSR-13). IEEE,
2013, pp. 101–110.
[34] T. Judd, K. Ehinger, F. Durand, and A. Torralba, “Learning to predict
where humans look,” in Computer Vision, 2009 IEEE 12th international
conference on. IEEE, 2009, pp. 2106–2113.
[35] T. M. Khoshgoftaar, S. Zhong, and V. Joshi, “Enhancing software quality
estimation using ensemble-classifier based noise filtering,” Intelligent Data
Analysis, vol. 9, no. 1, pp. 3–27, 2005.
[36] S. Kim, H. Zhang, R. Wu, and L. Gong, “Dealing with noise in defect prediction,” in Proceedings of the 33rd ACM/IEEE International Conference
on Software Engineering (ICSE-11), 2011, pp. 481–490.
[37] H. C. Kraemer, A. Noda, and R. O’Hara, “Categorical versus dimensional
approaches to diagnosis: methodological challenges,” Journal of psychiatricresearch, vol. 38, no. 1, pp. 17–25, 2004.
[38] R. Krishna, T. Menzies, and L. Layman, “Less is more: Minimizing code
reorganization using xtree,” Information and Software Technology, vol. 88,
pp. 53–66, 2017.
[39] S. Lessmann, B. Baesens, C. Mues, and S. Pietsch, “Benchmarking classification models for software defect prediction: A proposed framework andnovel findings,” IEEE Transactions on Software Engineering, vol. 34, no. 4,
pp. 485–496, 2008.
[40] C. Lewis, Z. Lin, C. Sadowski, X. Zhu, R. Ou, and E. J. Whitehead Jr, “Doesbug prediction support human developers? findings from a google casestudy,” in Proceedings of the 2013 international conference on Software
Engineering. IEEE Press, 2013, pp. 372–381.
[41] H. Li, W. Shang, Y. Zou, and A. E. Hassan, “Towards just-in-time suggestions for log changes,” Empirical Software Engineering, vol. 22, no. 4, pp.
1831–1865, 2017.
[42] Y. Ma, G. Luo, X. Zeng, and A. Chen, “Transfer learning for cross-company
software defect prediction,” Information and Software Technology, vol. 54,
no. 3, pp. 248–256, 2012.
[43] R. C. MacCallum, S. Zhang, K. J. Preacher, and D. D. Rucker, “On thepractice of dichotomization of quantitative variables.” Psychological methods, vol. 7, no. 1, p. 19, 2002.
[44] S. McIntosh, Y. Kamei, B. Adams, and A. E. Hassan, “The impact of code
review coverage and code review participation on software quality: A casestudy of the qt, vtk, and itk projects,” in Proceedings of the 11th IEEE/ACM
Working Conference on Mining Software Repositories (MSR-14). ACM,
2014, pp. 192–201.
[45] S. McIntosh, Y. Kamei, B. Adams, and A. E. Hassan, “An empirical study ofthe impact of modern code review practices on software quality,” Empirical
Software Engineering, vol. 21, no. 5, pp. 2146–2189, 2016.
[46] T. Menzies, C. Bird, T. Zimmermann, W. Schulte, and E. Kocaganeli,“The inductive software engineering manifesto: principles for industrial
data mining,” in Proceedings of the International Workshop on Machine
Learning Technologies in Software Engineering. ACM, 2011, pp. 19–26.
[47] T. Menzies, Z. Milton, B. Turhan, B. Cukic, Y. Jiang, and A. Bener, “Defect prediction from static code features: current results, limitations, newapproaches,” Automated Software Engineering, vol. 17, no. 4, pp. 375–407,
2010.
[48] T. Menzies, D. Owen, and J. Richardson, “The strangest thing about software,” Computer, vol. 40, no. 1, 2007.
[49] A. Mockus, Missing Data in Software Engineering. Springer, 2008, pp.
185–200.
[50] A. Mockus, “Organizational volatility and its effects on software defects,”
in Proceedings of the 18th ACM SIGSOFT international symposium on
Foundations of software engineering (FSE-10). ACM, 2010, pp. 117–126.
[51] D. Mossman, “Assessing predictions of violence: being accurate aboutaccuracy.” Journal of consulting and clinical psychology, vol. 62, no. 4, p.
783, 1994.
[52] J. Nam, W. Fu, S. Kim, T. Menzies, and L. Tan, “Heterogeneous defectprediction,” IEEE Transactions on Software Engineering, 2017.

R EFERENCES
[1]

[2]

[3]
[4]

[5]
[6]
[7]
[8]
[9]
[10]
[11]
[12]
[13]

[14]

[15]
[16]
[17]
[18]
[19]
[20]

[21]
[22]
[23]

[24]

W. Abdelmoez, M. Kholief, and F. M. Elsalmy, “Bug fix-time prediction
model using naı̈ve bayes classifier,” in Computer Theory and Applications(ICCTA), 2012 22nd International Conference on. IEEE, 2012, pp. 167–
172.
M. Abdelwahab and C. Busso, “Supervised domain adaptation for emotion recognition from speech,” in Acoustics, Speech and Signal Processing(ICASSP), 2015 IEEE International Conference on. IEEE, 2015, pp. 5058–
5062.
D. W. Aha and D. F. Kibler, “Noise-tolerant instance-based learning algorithms.” in IJCAI. Citeseer, 1989, pp. 794–799.
C. O. Alm, D. Roth, and R. Sproat, “Emotions from text: Machine learning
for text-based emotion prediction,” in Proceedings of the Conference on
Human Language Technology and Empirical Methods in Natural Language
Processing (HLT/EMNLP-05). Association for Computational Linguistics,
2005, pp. 579–586.
D. G. Altman and P. Royston, “The cost of dichotomising continuous
variables,” Bmj, vol. 332, no. 7549, p. 1080, 2006.
S. Boughorbel, F. Jarray, and M. El-Anbari, “Optimal classifier for imbalanced data using matthews correlation coefficient metric,” PloS one, vol. 12,
no. 6, p. e0177678, 2017.
L. Breiman, Classification and regression trees. Routledge, 2017.
G. W. Brier, “Verification of forecasts expressed in terms of probability,”Monthey Weather Review, vol. 78, no. 1, pp. 1–3, 1950.
M. Cataldo, A. Mockus, J. A. Roberts, and J. D. Herbsleb, “Software
dependencies, work dependencies, and their impact on failures,” IEEE
Transactions on Software Engineering, vol. 35, no. 6, pp. 864–878, 2009.
J. Cohen, “The cost of dichotomization,” Applied psychological measurement, vol. 7, no. 3, pp. 249–253, 1983.
J. Cohen, Statistical power analysis for the behavioral sciences . Hilsdale.
Hillsdale, N.J. : L. Erlbaum Associates, 1988, vol. 2.
N. V. Dawson and R. Weiss, “Dichotomizing continuous variables in statistical analysis a practice to avoid,” 2012.
M. A. de Almeida, H. Lounis, and W. L. Melo, “An investigation on the useof machine learned models for estimating correction costs,” in Proceedingsof the 20th IEEE/ACM international conference on Software engineering(ICSE-98). IEEE, 1998, pp. 473–476.
J. DeCoster, M. Gallucci, and A.-M. R. Iselin, “Best practices for using
median splits, artificial categorization, and their continuous alternatives,”
Journal of experimental psychopathology, vol. 2, no. 2, pp. jep–008 310,
2011.
J. DeCoster, A.-M. R. Iselin, and M. Gallucci, “A conceptual and empiricalexamination of justifications for dichotomization.” Psychological methods,
vol. 14, no. 4, p. 349, 2009.
K. El-Emam, D. Goldenson, J. McCurley, and J. Herbsleb, “Modelling
the likelihood of software process improvement: An exploratory study,”
Empirical Software Engineering, vol. 6, no. 3, pp. 207–229, 2001.
D. P. Farrington and R. Loeber, “Some benefits of dichotomization in
psychiatric and criminological research,” Criminal behaviour and mental
health, vol. 10, no. 2, pp. 100–122, 2000.
C. Ferri, J. Hernández-Orallo, and R. Modroiu, “An experimental comparison of performance measures for classification,” Pattern Recognition
Letters, vol. 30, no. 1, pp. 27–38, 2009.
P. A. Flach, “The geometry of roc space: understanding machine learning
metrics through roc isometrics,” in Proceedings of the 20th International
Conference on Machine Learning (ICML-03), 2003, pp. 194–201.
A. Folleco, T. M. Khoshgoftaar, J. Van Hulse, and L. Bullard, “Software
quality modeling: The impact of class noise on the random forest classifier,”in Proceedings of the IEEE Congress on Evolutionary Computation (CEC08). IEEE, 2008, pp. 3853–3859.
S. Garcia, J. Luengo, J. A. Saez, V. Lopez, and F. Herrera, “A survey of
discretization techniques: Taxonomy and empirical analysis in supervised
learning,” IEEE Transactions on Knowledge and Data Engineering, 2012.
G. Gay, T. Menzies, M. Davies, and K. Gundy-Burlet, “Automatically finding the control variables for complex system behavior,” Automated Software
Engineering, vol. 17, no. 4, pp. 439–468, 2010.
B. Ghotra, S. McIntosh, and A. E. Hassan, “Revisiting the impact of classification techniques on the performance of defect prediction models,” inProceedings of the 37th IEEE/ACM International Conference on Software
Engineering (ICSE-15). IEEE, 2015, pp. 789–800.
B. Ghotra, S. McIntosh, and A. E. Hassan, “A large-scale study of the
impact of feature selection techniques on defect classification models,” in
Mining Software Repositories (MSR), 2017 IEEE/ACM 14th International
Conference on. IEEE, 2017, pp. 146–157.

17

Gopi Krishnan Rajbahadur is currently
a Ph.D. student in the Software Analysis
and Intelligence Lab (SAIL) at Queen’s
University, Canada. He received his BE incomputer Science and Engineering fromSKR Engineering college, Anna University, India. He also spent close to five
years working as a data scientist in various software corporations in both India and Canada. His research interests
include interpretable machine learning,mining software repositories and safe data science. More information at: http://gopikrishnanrajbahadur.me/[53] J. Nam and S. Kim, “Clami: Defect prediction on unlabeled datasets (t),”in Automated Software Engineering (ASE), 2015 30th IEEE/ACM International Conference on. IEEE, 2015, pp. 452–463.
[54] D. F. Nettleton, A. Orriols-Puig, and A. Fornells, “A study of the effect ofdifferent types of noise on the precision of supervised learning techniques,”
Artificial intelligence review, vol. 33, no. 4, pp. 275–306, 2010.
[55] G. K. Rajbahadur, S. Wang, Y. Kamei, and A. E. Hassan, “The impact ofusing regression models to build defect classifiers,” in Proceedings of the
14th IEEE/ACM International Conference on Mining Software Repositories(MSR-17), 2017, pp. 135–145.
[56] P. Royston, D. G. Altman, and W. Sauerbrei, “Dichotomizing continuous
predictors in multiple regression: a bad idea,” Statistics in medicine, vol. 25,
no. 1, pp. 127–141, 2006.
[57] D. D. Rucker, B. B. McShane, and K. J. Preacher, “A researchers guideto regression, discretization, and median splits of continuous variables,”Journal of Consumer Psychology, vol. 25, no. 4, pp. 666–678, 2015.
[58] R. Sakia, “The box-cox transformation technique: a review,” The statistician, vol. 41, no. 2, pp. 169–178, 1992.
[59] J. Schumann, K. Gundy-Burlet, C. Pasareanu, T. Menzies, and T. Barrett,“Software v&v support by parametric analysis of large software simulation
systems,” in 2009 IEEE Aerospace Conference, 2009.
[60] C. Seiffert, T. M. Khoshgoftaar, J. Van Hulse, and A. Folleco, “An empirical
study of the classification performance of learners on imbalanced and noisy
software quality data,” Information Sciences, vol. 259, no. NA, pp. 571–595,
2014.
[61] M. Sokolova, N. Japkowicz, and S. Szpakowicz, “Beyond accuracy, f-score
and roc: a family of discriminant measures for performance evaluation,” inAustralasian joint conference on artificial intelligence. Springer, 2006, pp.
1015–1021.
[62] C. Tantithamthavorn and A. E. Hassan, “An experience report on defectmodelling in practice: Pitfalls and challenges,” in Proceedings of the 40thInternational Conference on Software Engineering: Software Engineeringin Practice. ACM, 2018, pp. 286–295.
[63] C. Tantithamthavorn, S. McIntosh, A. E. Hassan, A. Ihara, and K. Matsumoto, “The impact of mislabelling on the performance and interpretationof defect prediction models,” in Proceeding of the 37th IEEE/ACM International Conference on Software Engineering (ICSE-15), 2015, p. 812823.
[64] C. Tantithamthavorn, S. McIntosh, A. E. Hassan, and K. Matsumoto,“Automated parameter optimization of classification techniques for defectprediction models,” in Proceedings of the 38th IEEE/ACM International
Conference on Software Engineering (ICSE-16), 2016, pp. 321–332.
[65] C. Tantithamthavorn, S. McIntosh, A. E. Hassan, and K. Matsumoto, “Anempirical comparison of model validation techniques for defect prediction
models,” IEEE Transactions on Software Engineering, vol. 43, no. 1, pp.
1–18, 2017.
[66] P. Thongtanunam, S. McIntosh, A. E. Hassan, and H. Iida, “Revisitingcode ownership and its relationship with software quality in the scope ofmodern code review,” in Proceedings of the 38th IEEE/ACM International
Conference on Software Engineering (ICSE-16). IEEE, 2016, pp. 1039–
1050.
[67] Y. Tian, M. Nagappan, D. Lo, and A. E. Hassan, “What are the characteristics of high-rated apps? a case study on free android applications,”in Proceedings of the 2015 IEEE International Conference on Software
Maintenance and Evolution (ICSME-15). IEEE, 2015.
[68] H. Wang and M. Song, “Ckmeans. 1d. dp: optimal k-means clustering in one
dimension by dynamic programming,” The R journal, vol. 3, no. 2, p. 29,
2011.
[69] S. Wang, T.-H. Chen, and A. E. Hassan, “Understanding the factors forfast answers in technical q&a websites,” Empirical Software Engineering,
vol. PP, 2017.
[70] F. Wilcoxon, “Individual comparisons by ranking methods,” Biometrics
bulletin, vol. 1, no. 6, pp. 80–83, 1945.
[71] Y. Yang, G. I. Webb, and X. Wu, “Discretization methods,” in Data mining
and knowledge discovery handbook. Springer, 2009, pp. 101–116.
[72] F. Zhang, F. Khomh, Y. Zou, and A. E. Hassan, “An empirical study on
factors impacting bug fixing time,” in Proceedings of the 19th Working
Conference on Reverse Engineering (WCRE-12). IEEE, 2012, pp. 225–
234.
[73] F. Zhang, A. Mockus, I. Keivanloo, and Y. Zou, “Towards building a universal defect prediction model,” in Proceedings of the 11th Working Conferenceon Mining Software Repositories. ACM, 2014, pp. 182–191.
[74] Q. Zheng, A. Mockus, and M. Zhou, “A method to identify and correct
problematic software activity data: Exploiting capacity constraints and data
redundancies,” in Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, 2015, pp. 637–648.
[75] X. Zhu and X. Wu, “Class noise vs. attribute noise: A quantitative study,”
Artificial Intelligence Review, vol. 22, no. 3, pp. 177–210, 2004.
[76] X. Zhu, X. Wu, and Q. Chen, “Eliminating class noise in large datasets,”in Proceedings of the 20th International Conference on Machine Learning(ICML-03). AAAI, 2003, pp. 920–927.

Shaowei Wang is a postdoctoral
fellow in the Software Analysis andIntelligence Lab (SAIL) at Queens
University, Canada. He obtained his
PhD from Singapore Management
University, and BSc from Zhejiang
University. His research interests include
code mining and recommendation,
softwaremaintenance,developer
forum analysis, and mining software
repositories.
More
information
at:https://sites.google.com/site/wswshaoweiwang/

Yasutaka Kamei is an assistant professor at Kyushu University in Japan. He received his BE degree in informatics from
Kansai University, and PhD degree in information science from the Nara Instituteof Science and Technology. He was a
research fellow of the JSPS (PD) from
July 2009 to March 2010. From April 2010
to March 2011, he was a postdoctoral
fellow at Queens University in Canada.
His research interests include empiricalsoftware engineering, open source software engineering, andmining software repositories (MSR). He is a member of the IEEE
More information at http://posl.ait.kyushu-u.ac.jp/ kamei/

Ahmed E. Hassan is an IEEE fellow
and member, ACM influential educator,
an NSERC Steacie Fellow, a Canada Research Chair (CRC) in Software Analytics, and the NSERC/BlackBerry Software
Engineering Chair at the School of Computing at Queens University, Canada. His
industrial experience includes helping architect the Blackberry wireless platform,
and working for IBM Research at the Almaden Research Lab and the Computer
Research Lab at Nortel Networks. His research interests include
mining software repositories, empirical software engineering,load testing, and log mining. Dr. Hassan serves on the editorial
board of the IEEE Transactions on Software Engineering, theJournal of Empirical Software Engineering, and PeerJ Computer
Science. He spearheaded the organization and creation of the
Mining Software Repositories (MSR) conference and its researchcommunity. More information at https://sail.cs.queensu.ca/

18