Noname manuscript No.
(will be inserted by the editor)

The Impact of Feature Reduction Techniques on DefectPrediction Models
Masanari Kondo · Cor-Paul Bezemer ·Yasutaka Kamei · Ahmed E. Hassan · Osamu
Mizunothe date of receipt and acceptance should be inserted later

Abstract Defect prediction is an important task for preserving software quality. Mostprior work on defect prediction uses software features, such as the number of lines
of code, to predict whether a file or commit will be defective in the future. There areseveral reasons to keep the number of features that are used in a defect prediction
model small. For example, using a small number of features avoids the problem ofmulticollinearity and the so-called ‘curse of dimensionality’. Feature selection andreduction techniques can help to reduce the number of features in a model. Feature
selection techniques reduce the number of features in a model by selecting the mostimportant ones, while feature reduction techniques reduce the number of featuresby creating new, combined features from the original features. Several recent studies have investigated the impact of feature selection techniques on defect prediction.
However, there do not exist large-scale studies in which the impact of multiple feature
reduction techniques on defect prediction is investigated.
Masanari Kondo, Osamu Mizuno
Software Engineering Laboratory (SEL)
Kyoto Institute of Technology, Japan
E-mail: m-kondo@se.is.kit.ac.jp, o-mizuno@kit.ac.jp
Cor-Paul Bezemer
Department of Electrical and Computer Engineering
University of Alberta, Canada
E-mail: bezemer@ualberta.ca
Yasutaka Kamei
Principles of Software Languages group (POSL)
Kyushu University, Japan
E-mail: kamei@ait.kyushu-u.ac.jp
Ahmed E. Hassan
Software Analysis and Intelligence Lab (SAIL), School of Computing
Queen’s University
Kingston, Ontario, Canada
E-mail: ahmed@cs.queensu.ca

2

Kondo et al.

In this paper, we study the impact of eight feature reduction techniques on theperformance and the variance in performance of five supervised learning and five
unsupervised defect prediction models. In addition, we compare the impact of thestudied feature reduction techniques with the impact of the two best-performing feature selection techniques (according to prior work).
The following findings are the highlights of our study: (1) The studied correlation and consistency-based feature selection techniques result in the best-performing
supervised defect prediction models, while feature reduction techniques using neuralnetwork-based techniques (restricted Boltzmann machine and autoencoder) result inthe best-performing unsupervised defect prediction models. In both cases, the defectprediction models that use the selected/generated features perform better than thosethat use the original features (in terms of AUC and performance variance). (2) Neuralnetwork-based feature reduction techniques generate features that have a small variance across both supervised and unsupervised defect prediction models. Hence, we
recommend that practitioners who do not wish to choose a best-performing defectprediction model for their data use a neural network-based feature reduction technique.

1 Introduction
Software developers have limited time to test their software. Hence, developers need
to be selective when deciding where to focus their testing effort. Defect prediction
models help developers focus their limited testing effort on components that are themost likely to be defective. Because detecting defects at an early stage can considerably reduce the development cost [74], defect prediction models have received
widespread attention in software engineering research.
Many software features (e.g., software complexity features) can be used in defect prediction models [4, 11, 27, 36, 47]. However, it is important to carefully selectthe set of features that is used in such models, as using a set of features that is toolarge does not automatically result in better defect prediction. For example, priorstudies showed that reducing the number of features avoids the problem of multicollinearity [15] and the curse of dimensionality [5]. Hence, many of the existing defect prediction models used feature selection or reduction techniques [8, 11, 16, 18–
20, 23, 29, 33, 34, 48, 52–54, 61–64, 69, 76, 81].
Feature selection techniques reduce the number of features in a model by selecting the most important ones, while feature reduction techniques reduce the numberof features by creating new, combined features from the original features. Recent
studies [18, 76] investigated the impact of feature selection techniques on the performance of defect prediction models. However, this paper describes the first large-scale
study on multiple feature reduction techniques and their impact on a large number ofprediction models.
In this paper, we compared the impact of the original features, features that are
generated using traditional feature reduction techniques (i.e., PCA [11], FastMap [14],feature agglomeration [65], random projections [6], TCA [57] and TCA+ [54]), andfeatures that are generated using neural network-based feature reduction techniques
2The Impact of Feature Reduction Techniques on Defect Prediction Models

3(i.e., restricted Boltzmann machine (RBM) [70] and autoencoder (AE) [30]) on defectprediction models. In addition, we compared the impact of features that are generated using feature reduction techniques with features that are selected using the bestperforming feature selection techniques in prior work (correlation and consistencybased feature selection) [18, 76]. We compared the features along two dimensions:
their performance (area under the receiver operating characteristic curve (AUC)), andtheir performance variance (interquartile range (IQR)). The receiver operating characteristic curve is created by plotting the true positive rate against the false positive
rate, hence, the AUC represents the balance between the true positive rate and thefalse positive rate. The IQR represents the variance of a data distribution.
We conducted experiments on three publicly available datasets that contain software defects (the PROMISE [33], cleaned NASA [66], and AEEEM [11] datasets).
Our ultimate goal is to identify which feature reduction techniques yield new, powerful features that preserve the predictive power of the original features, and improve
the prediction performance compared to feature selection techniques. We studied theimpact of feature reduction techniques on five supervised learning and five unsupervised learning models for defect prediction in our experiments. In particular, we focus
on the following research questions:
RQ1: What is the impact of feature reduction techniques on the performance of
defect prediction models?
Motivation: Reducing the number of features in a model can address the multicollinearity problem [15] and the curse of dimensionality [5]. In this RQ, we
studied how feature reduction techniques impact the performance of supervisedand unsupervised defect prediction models.
Results: Feature agglomeration and TCA can reduce the number of features,
while preserving an AUC that is as good as that of the original features for supervised models. In addition, the AUC of unsupervised defect prediction models is significantly better when preprocessing the features with neural networkbased feature reduction techniques than other feature reduction techniques.
RQ2: What is the impact of feature reduction techniques on the variance of theperformance across defect prediction models?
Motivation: The AUC for a dataset can vary across defect prediction models.
Hence, it can be challenging for practitioners to choose the defect prediction
model that performs best on their data. If all defect prediction models havea small performance variance for a dataset, practitioners can avoid having to
make this challenging choice.
Results: Neural network-based feature reduction techniques (RBM and AE)
generate features that improve the variance of the performance across different
defect prediction models in many cases when used in a supervised or unsupervised defect prediction model. In addition, almost all feature reduction techniques (except PCA) generate features that improve the variance of the performance across different defect prediction models in many cases when used in an
unsupervised defect prediction model.
RQ3: How do feature selection techniques compare to feature reduction techniques when applied to defect prediction?
3

4

Kondo et al.

Motivation: Prior work [18, 76] showed that several feature selection techniques outperform the original models. In this RQ, we studied how feature selection techniques compare to feature reduction techniques.
Results: For the supervised defect prediction models, the studied feature selection techniques (correlation and consistency-based feature selection) outperform all the studied feature reduction techniques. However, for the unsupervised defect prediction models, the neural network-based feature reduction
techniques (RBM and AE) outperform the other studied feature selection/reduction techniques.
Our results provide practitioners with advice on which feature selection/reduction technique to use in combination with a defect prediction model. In particular, werecommend to use a feature selection technique when using a supervised defect prediction model, and a neural network-based feature reduction technique when using an
unsupervised defect prediction model, as these feature selection/reduction techniques
improve the variance across defect prediction models, while outperforming the otherfeature reduction techniques.
The organization of our paper is as follows. Section 2 introduces related work.
Section 3 presents our methodology. Section 4 presents experimental setup. Section 5
presents the results of our experiment. Section 6 discusses these results. Section 7
describes the threats to the validity of our findings. Section 8 presents the conclusion.

2 Related work
In this section, we discuss related work on defect prediction, and feature selection
and reduction.

2.1 Defect prediction
Defect prediction approaches can be divided in two categories: approaches that use
supervised learning and unsupervised learning. Supervised defect prediction models
need training data and test data. Usually, the training data and test data are collectedfrom the same, or very similar projects from the same organization (within-project
defect prediction). However, sometimes it is difficult to collect sufficient training datafrom the same project or organization (e.g., when the project is new).
In such cases, cross-project defect prediction can be a viable alternative solution.
Cross-project defect prediction uses training datasets that contain data that comesfrom multiple projects, or even from multiple organizations. Cross-project defectprediction offers a solution for the problem of within-project defect prediction sincesmall datasets can be extended with data from other projects. Hence, cross-project defect prediction is one of the important research themes in defect prediction research.
However, cross-project defect prediction has several challenges [3, 28, 29, 50, 53].
For instance, cross-project defect prediction has the problem of heterogeneous original features across datasets [53]. There exists one solution for this challenge [51],however; converging datasets collected from multiple companies to one dataset when
4The Impact of Feature Reduction Techniques on Defect Prediction Models

5

Table 1 Overview of prior work that uses feature selection or reduction techniques in combination withdefect prediction.
Technique

FamilyExample ofTechniques

References

Feature Selection

Filter-based Feature
Ranking
Filter-based Subset
Selection
Wrapper-based Subset
Selection
Greedy-based Feature
Selection
Distance-based Feature
Selection
State-of-the-Art
OthersChi-Square
Correlation
Correlation-based Feature
Subset Selection
Logistic
Regression
Greedy Forward
Selection Algorithm
EM Algorithm[16, 18–20, 33, 45, 48, 61, 69, 76]

MVS
Significance[52][34, 53, 69, 81]

PCA
State-of-the-Art

PCA
TCA

[8, 11, 18, 49, 50, 56, 61, 62, 76][54, 62]

Feature Reduction[8, 16, 18, 48, 61, 76][18, 61, 63, 64, 69, 76]
[48]
[29]

they contain different features still remains an open challenge and further investigation is needed.
One way to overcome the challenge of heterogeneity is to use unsupervised defectprediction models [2, 7, 52, 77, 78, 80]. Unsupervised models have the advantage thatthey do not need a training dataset, and therefore, are not affected by the problem ofheterogeneous features [2, 7, 52, 77, 78, 80]. Recently, Zhang et al. [78] summarized
the accuracy of several supervised and unsupervised models for defect prediction.
They concluded that connectivity-based unsupervised models have an accuracy thatis as good as that of supervised models. Therefore, unsupervised models are a viable
alternative to supervised models for defect prediction. Note that unsupervised models
do not need training data, and are therefore always within-project defect prediction
models.
2.2 Feature selection and reduction
Using feature selection or reduction technique has the advantage of addressing thecurse of dimensionality [5]. This problem is originally considered a dynamic optimization problem. However, machine learning models also need to consider thisproblem. The problem occurs when having a large number of features yet a small
sample size in machine learning models. In this case, the sample size is not large
enough to search the representations of all the combinations of features, and to generalize their parameters (resulting in overfitting) [31]. Hence, the prediction performance of these models for unseen data would be worse, and could lead to aclassification error. Feature selection or reduction techniques can address this problem by removing or combining redundant and irrelevant features. In addition, priorwork [45, 46] reported that the prediction performance of a model is often determinedby only a few features. Therefore, feature selection or reduction techniques can often
be applied without negatively affecting the prediction performance.
In this paper, we define feature selection and reduction as follows:5

6

Kondo et al.

– Feature selection techniques reduce the number of features by selecting a subsetof the original features.
– Feature reduction techniques reduce the number of features by combining original
features into new features.
Several researchers studied the impact of feature selection techniques on defectprediction models [8, 16, 18–20, 29, 33, 34, 45, 48, 52, 53, 61, 63, 64, 69, 76, 81] .
Table 1 gives an overview of prior work that uses feature selection or reduction techniques. For instance, Ghotra et al. [18] summarized the impact of feature selection
techniques for defect prediction. They reported that correlation-based feature selection outperforms the other feature selection techniques. In addition, they showed thatthe impact of feature selection techniques varies across the studied datasets. Xu etal. [76] also summarized the impact of feature selection techniques on defect prediction models. They also reported that the effectiveness of feature selection techniques
exhibits significant differences across studied datasets. Menzies et al. [45] used theinformation gain measure to rank features based on their importance in a defect prediction model. They found that defect prediction can often be done using only a small
set of features. For example, they reduced the number of features in the CM1 projectfrom 38 to 3 without affecting the prediction performance.
The impact of feature reduction techniques on defect prediction models has notbeen studied as extensively. Most researchers use principal component analysis (PCA) [8,11, 18, 49, 50, 56, 61, 62, 76], and only a few researchers use other feature reduction techniques [54, 62]. For instance, D’Ambros et al. [11] compared class-level
defect prediction models to present a benchmark for defect prediction. They used
PCA to avoid the problem of multicollinearity [15] among the independent variables.
Nagappan et al. [49] predicted post-release defects using complexity features. They
built prediction models using PCA to avoid the problem of multicollinearity. Neumann [56] proposed PCA-ANN which is a combination of PCA and artificial neuralnetworks. Neumann also used PCA to avoid the problem of multicollinearity. Challagulla et al. [8] compared several prediction models for identifying defects. In addition, they compared PCA with feature selection techniques such as feature subset
selection. They concluded that feature selection techniques are better than PCA, andthat PCA did not add any advantages for defect prediction. Rathore et al. [61] compared the performance of feature selection techniques. In this comparison, they also
used PCA. They found that PCA is one of the best-performing techniques in thiscomparison. Ren et al. [62] extended PCA to address class imbalance problem fordefect prediction. Nam et al. [54] applied transfer component analyses (TCA andTCA+) to training and test data to convert the data to be closer than the original data.
This process addressed the challenge of heterogeneity of training data and test data incross-project defect prediction. Nam’s approach significantly improved cross-project
defect prediction performance.
Despite the amount of prior work on feature reduction techniques (mostly on
PCA), no prior work has conducted a systematic study of the impact of feature reduction techniques on defect prediction models. In this paper, we provide such asystematic study of the impact of eight feature reduction and two best-performing
6

The Impact of Feature Reduction Techniques on Defect Prediction Models

7

Table 2 Description of studied projects
Studied Dataset

Project# ofEntities# ofDefective% Defective# ofFeatures∗# ofStudied Features∗

PROMISEAnt v1.7
Camel v1.6
Ivy v1.4
Jedit v4.0
Log4j v1.0
Lucene v2.4
POI v3.0
Tomcat v6.0
Xalan v2.6
Xerces v1.3

745
965
241
306
135
340
442
858
885
453

166
188
16
75
34
203
281
77
411
69

22.3
19.5
6.6
24.5
25.2
59.7
63.6
9.0
46.4
15.2

24
24
24
24
24
24
24
24
24
24

20
20
20
20
20
20
20
20
20
20

NASA

CM1
JM1
KC3
MC1
MC2
MW1
PC1
PC2
PC3
PC4
PC5

327
7,782
194
1,988
125
253
705
745
1,077
1,287
1,711

42
1,672
36
46
44
27
61
16
134
177
471

12.8
21.5
18.6
2.3
35.2
10.7
8.7
2.1
12.4
13.8
27.5

38
22
40
39
40
38
38
37
38
38
39

37
21
39
38
39
37
37
36
37
37
38

AEEEM

Eclipse JDT Core
EquinoxApache Lucene
Mylyn
Eclipse PDE UI

997
324
691
1,862
1,497

206
129
64
245
209

20.7
39.8
9.3
13.2
14.0

291
291
291
291
291

212
212
212
212
212

∗ We removed features that are not related to source code. For instance, the name of the file, name of the class andthe version. Hence, the number of studied features are different from the total number of features.

feature selection techniques on five supervised and five unsupervised defect prediction models.
Finally, Peters et al. [58] showed that often not only the features can be reduced
without negatively affecting the performance, but also the amount of the rows in thedata.

3 Methodology
In this section, we describe our methodology. In particular, we discuss our studied
datasets, defect prediction models, feature selection techniques, feature reduction
techniques, evaluation measure, our preprocessing steps, and our validation scheme.

3.1 Studied datasets
In our work, we used three publicly available datasets (the PROMISE [33], cleaned
NASA [55] and AEEEM [11] datasets) that were used by prior work [78] on supervised and unsupervised defect prediction models. Table 2 describes the studied
datasets. All datasets contain popular software features for measuring source code
complexity (see Table 3 for a summary of the used features). Each entity in a dataset
is labelled as defective or clean.
7

8

Kondo et al.

Table 3 Studied features
Studied Dataset
PROMISENASA
AEEEM

Features

Reference

CK
OO
McCabeHalstead
CK
OO
number of previous defects
change featurescomplexity code change features
churn of CK and OO
entropy of CK and OO

Chidamber et al. [9]Basili et al. [4]McCabe [43]Halstead [24]Chidamber et al. [9]Basili et al. [4]Kim et al. [36]Moser et al. [47]Hassan [27]D’Ambros et al. [11]D’Ambros et al. [11]The PROMISE dataset contains several types of projects. We chose the 10 projects
that were used by prior work [78], to ease the comparison of our results with priorwork. All studied PROMISE projects have the same number of features. The PROMISE
dataset contains the Chidamber and Kemerer (CK) features [9] and an additional setof object-oriented (OO) features [4].
The NASA dataset [55] contains 11 projects. Each project in the NASA dataset
has a different number of features. The NASA dataset contains McCabe features [43]and Halstead features [24]. We used the cleaned version [66] of the NASA dataset,because prior studies [59, 66] showed that the original version of the NASA dataset
contains inconsistent and mislabeled data.
The AEEEM dataset [11] contains five projects. All projects have the same number of features. Like the PROMISE dataset, the AEEEM dataset contains the CK andOO features. However, the AEEEM data also contains the number of previous defects [36], change features [47], complexity code change features [27], and the churnand entropy of the CK and OO [11] features.

3.2 Studied defect prediction models
We focused on defect prediction models that were used by prior work [78], to makeour results easier to compare. We studied five supervised models and five unsupervised models. Below we give a brief overview of the ideas behinds these models. Fora detailed overview, we refer the reader to the original papers in which these models
were introduced. We studied the following supervised defect prediction models:– Logistic Regression (LR) [44]: LR is one of the most commonly used models fordefect prediction. LR expresses the relationship between one or more independent
variables (i.e., the original features) and one dependent variable (i.e., defective or
clean) using a polynomial expression and a sigmoid function [25].
– Decision Tree (J48) [60]: J48 is a decision tree implementation in WEKA [21].
The decision tree uses a tree structure to decide on the dependent variable. In thistree, each node corresponds to one of the independent variables with a condition.
8

The Impact of Feature Reduction Techniques on Defect Prediction Models

9

J48 traverses the tree from the root to a leaf, while taking into account the input entity and the conditions in the tree. Each leaf corresponds to a value of thedependent variable.
– Random Forest (RF) [32]: RF is a popular ensemble learning model. RF builds
multiple decision trees based on subsets of training data that are randomly selected. RF decides on a value of the dependent variable by taking the result of amajority of the decision trees.
– Naive Bayes (NB) [79]: NB is a probability-based classifier that follows Bayes’
theorem. Bayes’ theorem describes the probability of an event, given knowledgeof conditions that could be related to the event.
– Logistic Model Tree (LMT) [39]: LMT is a classifier which combines a decision
tree and a logistic regression model. Like the decision tree, LMT follows a tree
structure. However, LMT uses logistic regressions instead of values in the leaves.
We used the caret library in R [38] to optimize the parameters of the supervised
models as suggested by Tantithamthavorn et al. [72].
We studied the following unsupervised defect prediction models:– Spectral Clustering (SC) [75]: SC labels entities using a graph that is based on
similarities across entities. In this graph, each node is an entity and each edge represents the similarity of the entities it connects. SC cuts sparse edges in this graphby classifying eigenvectors of the Laplacian matrix [75] of the graph. Following
this process, SC divides the entities into two groups.
– k-means (KM) [26]: KM is a popular clustering approach. KM classifies entities
based on the distances between entities and the center of a class (i.e., the meanof all entities in that class). In this paper, we used the Euclidean distance as the
distance metric.
– Partition Around Medoids (PAM) [35]: PAM is an approach that is similar to
KM. While KM uses the center of a class, PAM uses medoids. A medoid is anentity of which the sum of all the distances to the other entities in a class is at
its minimum. Because PAM uses the medoid instead of the center, PAM is lesslikely to be affected by outliers than KM.
– Fuzzy C-Means (FCM) [13]: FCM is also an approach that is similar to KM.
While KM classifies each entity to only one class in its process, FCM allows
entities to be a member of more than one class. The membership is expressed asa probability.
– Neural Gas (NG) [42]: NG is an approach that is similar to a self-organizing
map [37]. NG generates weighted points which have random features. Hence,
the weighted points are distributed across the feature space. For each learning
iteration, the features of the weighted points are updated by distances to closer
entities. Finally, the weighted points become the class centers.
We used the default parameters of the implementations for the unsupervised models.
We set the number of clusters to two, as defect prediction is a binary problem. Table 4 shows the libraries that we used for the implementation of the defect prediction
models.
Labeling technique in the unsupervised models: The unsupervised models
classify the data in two unlabeled clusters. We adopted the following heuristic to9

10

Kondo et al.

Table 4 Packages used for experiments
Defect Prediction Models
LR
RF
NB
J48
LMT
SC
KM
PAM
FCM
NG

Packages
The caret package in R
The caret package in R
The caret package in R
The caret package in R
The caret package in R
Zhang et al.’s implementation [78] (see app. A)
The cclust package in R
The cluster package in R
The e1071 package in R
The cclust package in R

Original feature N

Entities

Entities
New feature 1

FMFA

Original feature 1
PCA

Original
FeatureTest
data

Trainingdata⇥(O1 , O2 , ..., ON )An originalentity O , which
has N features

Training
data
Test
data

New
Feature

0

a11
B a21
B
B ..
@ .
aN 1

a12
a22
..
.

...
...
..
.

a1M
a2M
..
.

aN 2

...

aN M

1
C
C
C
A

N ⇥ M random-weight vectors matrix RN ⇥M .
Each column is a random-weight vector.

= (O · a + O · a + ... + O · a , ..., O · a + O · a + ... + O · a ) = X
1

11

2

21

N

N1

1

1M

2

2M

N

NM

RP
TCA/TCA+

Fig. 1 A visual overview of the core concepts of the traditional feature reduction techniques. The black
symbols represent the original features (or entities in FM and FA) and the purple symbols represent the
newly-generated features (or entities in FM and FA). RP (Random projection) transforms an original entity
to a new entity using M random-weight vectors.

identify the defective cluster: “For most features, software entities containing defects
generally have larger values than software entities without defects” [78]. In particular, we used the sum of row average of the normalized features in each cluster, todecide which cluster contains the defects [78]. To calculate the sum of row average,
we first summed the entity values in each cluster, respectively. Then, we calculated
the average values for each cluster. The cluster with the larger average value was
identified as the cluster with the defective entities.

3.3 Studied feature reduction techniques
In this subsection, we discuss the studied feature reduction techniques. We studied
two types of feature reduction techniques: traditional and neural network-based feature reduction techniques. We give a brief overview of the core concepts of each
10The Impact of Feature Reduction Techniques on Defect Prediction Models

11

0-1 Scale Original Features

V1

...

V2

H1

H2

Feature1

...

...

VN

HM

Feature M

Fig. 2 An overview of neural network-based feature reduction techniques (RBM and AE). RBM and AE
convert the original features (Vi ), which values must range between 0 and 1, into M new features (Hi ).
Note that the original input data may need to be preprocessed to satisfy the 0-1 range requirement.

feature reduction technique. For more precise details, we refer to the references thatare mentioned for each technique. Figure 1 shows a visualization of the traditionalfeature reduction techniques (PCA, FM, FA, TCA/TCA+, RP). Figure 2 gives an
overview of neural network-based feature reduction techniques (RBM and AE).
3.3.1 Traditional feature reduction techniques
We studied the following traditional feature reduction techniques.
– Principal Component Analysis (PCA): PCA is one of the most commonly usedfeature reduction techniques in defect prediction [8, 11, 18, 23, 50, 61, 62, 76].
PCA reduces the number of features by projecting the original set of features ona smaller number of principal components.
– FastMap (FM):For N original features, FM [14] first generates a (N-1)-dimensional
orthogonal hyper-plane of the line between two entities that are far from eachother. Second, FM projects the other entities on this hyper-plane. Because FM
projects the entities on the N − 1 orthogonal hyper-plane, we can reduce one feature from the original features. FM repeats this procedure until we get the required
number of new features. For instance, if we want three features to visualize ourdata from the N original features, we repeat the procedure N-3 times.
– Feature Agglomeration (FA): FA is a simple hierarchical clustering algorithm [65].
FA starts by creating a new feature from each original feature. Then, FA merges
the two nearest (based on their Euclidean distance) features into one feature, andrepeats this process until the desired number of features is reached.
– Transfer Component Analysis (TCA and TCA+): TCA [57] creates new featuresfrom the original features by projecting them on so-called transfer components(similar to PCA). However, the goal of TCA is not to reduce the number of features, but to reduce the gap between the distribution of the training and testing
data. During this process, the number of features is often reduced. Hence, TCA
11

12

Kondo et al.

can be used as a feature reduction technique. TCA+ [54] is an extension of TCA,
which optimizes the data using a preprocessing step according to the gap between
the distribution of the training and testing data, such as scaling the original features between 0 and 1 instead of using the z-score.
– Random Projection (RP):RP projects the original N-dimensional features onto M
generated features (M  N) using a N × M random-weight vectors matrix [6].
The equation of RP is as follows:
X = O × RN×M
where X is a generated M-dimensional vector entity, O = (O1 , O2 , ..., ON ) is an
original entity, and RN×M is a random-weight vectors matrix. For example, if wewant three features from N original features, we prepare three random-weightvectors with N random values in each of them. The random values are selected
such that X represents the original features.
3.3.2 Neural network-based feature reduction techniques
We studied the following neural network-based feature reduction techniques.
– Restricted Boltzmann Machine (RBM): An RBM [70] automatically extracts important information from the original features as weights and biases on a twolayered neural network. Each node in the first-layer corresponds to an original
feature, and each node in the second-layer corresponds to a new feature. We use
the output of the second-layer as the new features.
– Autoencoder (AE): AE [30] and RBM are similar, but trained differently. In RBM,
the network is trained based on a probability distribution. In AE, the network is
trained using the difference between the original and the generated features.
3.4 Studied feature selection techniques
We studied the correlation-based (CFS) and consistency-based feature selection techniques (ConFS). These techniques were reported as the best-performing feature selection techniques in prior studies [18, 76]. Below, we give a brief overview of thesetechniques.
– Correlation-based feature selection (CFS) [22]: CFS selects a subset of features
based on their correlation. The selected features have strong correlations with theclass label (clean or defective), while having a low correlation with each other.
– Consistency-based feature selection (ConFS) [12]: ConFS uses the consistencyof the class label across the entities instead of the correlation. For example, if file
A has a feature set (10, 20, 40, defective) and file B has a feature set (10, 20,
30, clean), we can identify the defective and clean entities using the third feature.
However, if a feature reduction technique removes the third feature, file A and file
B have the same feature set except for the class label. In that case, these entities
are inconsistent. Using this information, ConFS selects the best feature subset
from the original features.

12The Impact of Feature Reduction Techniques on Defect Prediction Models

13

3.5 Area under the receiver operating characteristic curve (AUC)
We used the Area Under the receiver operating characteristic Curve (AUC) as the performance measure since AUC is not affected by the skewness of defect data [71, 78].
The receiver operating characteristic (ROC) curve is created by plotting the false positive rate (on the x-axis) and the true positive rate (on the y-axis) at various thresholds.
In our experiment, the false positive rate is defined as the portion of clean entities thatare identified as defective; the true positive is defined as the portion of defective entities that are identified as defective. The threshold is used to label an entity as clean
or defective by checking whether its predicted probability is over the threshold. The
AUC is the area under the ROC curve. The values of the AUC range between 0 and
1; a perfect classifier has an AUC of 1, while a random classifier has an AUC of 0.5.

3.6 Preprocessing
Most feature reduction techniques require the data to be preprocessed. We detail the
preprocessing step below.
3.6.1 Preprocessing for traditional feature reduction techniques
The traditional feature reduction techniques require features that are normalized toa mean of 0 and a variance of 1 using the z-score [78]. The z-score is calculated asfollows:
Xorg − µ(1)Xz =σwhere µ is a mean of the value of the feature for all entities and σ is the standard
deviation of the value of the feature for all entities.
3.6.2 Preprocessing for neural network-based feature reduction techniques
The neural network-based feature reduction techniques require either binary features
or features that are between 0 and 1. Hence, we scale the original features as follows:
X scaled =Xorg − Xmin
Xmax − Xmin(2)

where Xorg is a vector of the value of a particular feature for all entities. Xmin is
the smallest value of the feature and Xmax is the largest value of the feature for all
entities [1].

3.7 Out-of-sample bootstrap sampling
Bootstrap sampling is a validation technique that is used to estimate the performanceof a model for unseen data. The technique is based on random sampling with replacement. Out-of-sample bootstrap sampling is a bootstrap sampling technique that13

14

Kondo et al.

estimates the future performance of a defect prediction model more accurately than across-validation scheme [71, 73]. Hence, we used the out-of-sample bootstrap sampling technique instead of a conventional validation technique such as 10-fold crossvalidation. The process of the out-of-sample bootstrap sampling is as follows:
1. Sample N data points following the distribution of the original dataset, with N
data points, while allowing for replacement.
2. Train a model using the sampled N data points, and test it using the data points
that were not sampled.
3. Repeat steps 1 and 2 M times.
4. Report the average/median performance as the performance estimate.
We used the out-of-sample bootstrap sampling under the condition where M = 100
and we report the median performance.

4 Experimental setup
In this section, we give an overview of the setup of our experiments. The results arepresented in Section 5. Figure 3 shows the steps of our experiments. We first conducted the out-of-sample bootstrap sampling on our studied datasets to generate andselect features using each of the studied feature reduction and selection techniques.
We then preprocessed the original features of each bootstrap sample as described in
Section 3.6. We generated eight new feature sets (one for each feature reduction technique) for each bootstrap sample. Hence, we generated 800 new feature sets using
feature reduction in total. Furthermore, the two studied feature selection techniques
selected two feature subsets (one for each feature selection technique) for each bootstrap sample. Hence, we selected 200 feature subsets using feature selection in total.
The smallest number of features in the studied datasets is 20 (i.e., in the PROMISE
dataset). Hence, to be able to observe the impact of a feature reduction technique,we configured each feature reduction technique to generate 10 features (H1–H10).
However, PCA uses variance to decide on the number of generated features [11, 18].
Therefore, each bootstrap sample results in a different number of generated featuresusing PCA. We configured PCA to retain 95% of the variance in the data [11, 18].
The median number of generated features by PCA in our experiments was 12 in the
PROMISE dataset, 10 in the NASA dataset and 34 in the AEEEM dataset.
The experimental setup for each RQ is discussed in the next section.

5 Results
In this section, we present the results of our experiments. For each RQ, we discuss
the motivation, approach and results.
14

The Impact of Feature Reduction Techniques on Defect Prediction Models

Generated

Selected

1 Features(reduction tech.)FeatureReduction/Selection
Tech.

Original
Features

For all files in abootstrap sample

2 Features(selection tech.)

H1

O1

H2

O2

H10

15

weights of the feature
3 All
reduction tech.

H1

On

Ori 1

Ori 2

Ori M

W 1,1

W 1,2

W 1,M

H2

W 2,1

W 2,2

W 2,M

H10

W 10,1

W 10,2

W 10,M

The out-of-sample bootstrap sampling.
We repeat the above procedure for each feature reduction tech.
First Step
RQ1: What is the impact of feature reduction tech. on the performance of defect prediction models?
RQ3: How do feature selection techniques compare to feature reduction techniques
when applied to defect prediction?

4
RQ1

1

RQ3

Model 1

Prediction
Models1 2Model 2Model 5

Project 1

AUC 1,1 AUC 1,2

AUC 1,5Calculate AUC for Project 2
each model foreach project forfeatures from each Project N
featurereduction/selection tech.

AUC 2,1 AUC 2,2

AUC 2,5

AUC N,1 AUC N,2

AUC N,5

4
Calculate the ratio
from the AUCs

Tech1 Tech2

5Model 1Model 2Model 5

Rat. 1,1

Rat. 1,2Rat. 1,5

Project 2

Rat. 2,1

Rat. 2,2Rat. 2,5

Project N

Rat. N,1 Rat. N,2Rat. N,5

Project 1

Conduct the
Scott-Knott
ESD test

Tech3 Tech4

PROMISE, NASA, AEEEM

Summarize as
boxplots foreach project.

Pro.1 Pro.2

Pro.9 Pro.10

PROMISE, NASA, AEEEMMedian5Compute medianratio values acrossthe models foreach project.

Project 1

Med. 1,1

Project 2

Med. 2,1

Project N

Med. N,1

Pro. 1

Summarize astables foreach project

Pro. 2

Pro. N

Tech 1

Med. 1,1 Med. 1,2

Med. 1,N

Tech 2

Med. 2,1 Med. 2,2

Med. 2,N

Tech M

Med. M,1 Med. M,2Med. M,N

RQ2: What is the impact of feature reduction tech. on the variance of the performanceacross defect prediction models?
RQ3: How do feature selection techniques compare to feature reduction techniques
when applied to defect prediction?

4

6

AUC (RQ1)
Model 1 Model 2

Model 5

Project 1

AUC 1,1 AUC 1,2

AUC 1,5Project 1

Value 1

Project 2

AUC 2,1 AUC 2,2

AUC 2,5Project 2

Value 2

Project N

AUC N,1 AUC N,2

AUC N,5Project N

Value N

IQR

Calculate IQR

Conduct the
Scott-Knott
ESD test

Tech1 Tech2

Tech3 Tech4

Supervised/Unsupervised

6

Summarize astables foreach feature
reduction/selection
tech.

Tech. 1

Tech. 2

Tech. M

Project 1

IQR. 1,1

IQR. 1,2

IQR. 1,M

Project 2

IQR. 2,1

IQR. 2,2

IQR. 2,M

Project N

IQR. N,1

IQR. N,2

IQR. N,M

Second Step
Discussion: Which features are generated by the feature reduction tech.?

1
Calculate correlation
between thegenerated featuresfrom each feature
reduction tech.

H1

H2

H10

H1

Cor 1,1

Cor 1,2

Cor 1,10

H2

Cor 2,1

Cor 2,2

Cor 2,10

H10Cor 10,1

Cor 10,2

3

Cor 10,10

Cluster 1
Cluster 2

Clustering results
using t-SNE forweights fromeach feature
reduction tech.

...
Cluster 10

Third Step

Fig. 3 Overview of our experimental design. We first generate/select 100 (the out-of-sample bootstrap)
feature sets using each feature reduction/selection technique for each studied dataset. The second step isdifferent for each RQ. We conduct correlation analysis and clustering analysis for discussion in the third
step.
15

16

Kondo et al.

5.1 RQ1: What is the impact of feature reduction techniques on theperformance of defect prediction models?
Motivation: Reducing the number of features that are used in a defect prediction
model can be beneficial for addressing the curse of dimensionality and multicollinearity of the model. There exist two ways to reduce the number of features in a model:(1) by selecting the most important features, and (2) by reducing the number of features by creating new, combined features from the original features. Prior work has
systematically studied the impact of feature selection techniques on defect prediction [18, 76], but no work has conducted a large-scale study of the impact of feature
reduction techniques on defect prediction. Hence, in this RQ, we studied the impact
of feature reduction techniques on the performance (AUC) of defect prediction models.
Approach: We used each feature set that was generated by a feature reduction
technique as input to the studied 5 supervised and 5 unsupervised defect prediction
models. We used the AUC as the performance measure. Because we calculated the
AUC of a defect prediction model using the out-of-sample bootstrap sampling 100
times for each feature reduction technique, each model has 100 AUC values. Hence,
we used the median value to represent the median performance of a defect prediction
model using a certain feature reduction technique. Because we studied 26 projects,
our experiments yielded 260 median AUC values for each feature reduction technique (5 supervised models*26 projects+5 unsupervised models*26 projects). For
comparison, we also calculated the performance of the studied defect prediction models without applying a feature reduction technique (indicated as ORG). Note that we
did within-project defect prediction in our experiments.
We used the Scott-Knott ESD test [73] (using a 95% significance level) to compare the median AUC values across feature reduction techniques. The Scott-Knott test
is a hierarchical clustering algorithm that ranks the distributions of values. In particular, distributions that are not statistically significantly different are placed in the samerank. The Scott-Knott ESD test is an extension of the Scott-Knott test, which not only
ranks based on significance, but also on Cohen’s d effect size [10]. The Scott-Knott
ESD test places distributions which are not significantly different, or have a negligible effect size, in the same rank. We used the ScottKnottESD R package1 that wasprovided by Tantithamthavorn [72].
Project-level analysis: the aforementioned procedure combines the results of all
projects. However, this procedure prevents us from understanding differences for eachproject. Hence, we also studied the performance at the project-level.
We compared the ratios of the AUCs (the median AUCs across all bootstrap samples) of each feature reduction technique to the original models. We calculated thisratio as follows:
AUC FR
ratio =AUCORG
Where AUCORG is the AUC of a prediction model using the original features, and
AUCFR is the AUC of a prediction model using the features that were generated by1

16

https://github.com/klainfo/ScottKnottESD

0.9

0.9

0.8

0.8

0.7

0.7

AUC

AUC

The Impact of Feature Reduction Techniques on Defect Prediction Models

17

●

0.6

0.6

A+
TC

P

FM

●
●
●
●

R

FA
TC
A

A

G
R
O

BM

PC

A

FM

PC

P
R

BM
R

AE

A

A+
TC

G

TC

FA

R
O

(a) The supervised models

AE

0.5

●

R

0.5

●
●
●
●
●
●
●
●

(b) The unsupervised models

Fig. 4 The Scott-Knott ESD test results for the supervised (logistic regression, random forest, naive Bayes,
decision tree, and logistic model tree) and the unsupervised (spectral clustering, k-means, partition aroundmedoids, fuzzy C-means, neural-gas) models. Each color indicates a rank: models in different ranks havea statistically significant difference in performance. Each boxplot has 130 median AUC values (5 defectprediction models times 26 projects). The x-axis refers to the feature reduction techniques; the y-axis refers
to the AUC values.

a particular feature reduction technique. Hence, a ratio larger than 1 indicates thatthe feature reduction technique improved the AUC compared to the original models,
while a ratio smaller than 1 indicates that the feature reduction technique reduced the
AUC. We computed the median ratio across the five studied supervised and unsupervised prediction models.
We used the aforementioned ratio to analyze performance at the project-level. The
project-level analysis shows the impact of the different feature reduction techniques inevery single project and dataset. Figure 6 shows the distributions of the ratios for eachstudied project for the supervised and unsupervised prediction models, respectively.
Each boxplot contains 40 ratio values (5 prediction models * 8 feature reduction
techniques). In addition, we show the median ratios for the best-performing feature
reduction techniques as tables for deeper analysis (Table 5). These median ratios werecomputed from five AUC values (one for each studied prediction model).
Results: FA and TCA can preserve the performance of the original defectprediction models, while at the same time reducing the number of features. Figure 4(a) and Figure 4(b) show that the performance of the supervised and unsupervised defect prediction models does not decrease when applying FA or TCA. Hence,these feature reduction techniques can safely be applied to reap the benefits of a reduced number of features. In particular, FA and TCA work well for supervised models. Interestingly, the performance of the supervised and unsupervised defect prediction models is significantly lower when using TCA+ (which is an extension of TCA),
compared to the original TCA.
The neural network-based feature reduction techniques (RBM and AE), significantly outperform traditional feature reduction techniques for the unsuper17

18

Kondo et al.
0.9

AUC

0.8

0.7
●

0.6

●
●
●
●
●
●
●

0.5

L_
FA
L_
O
R
G
SV
L_
T
C
SV
L_ A
TC
A
SV +
L_
A
SV E
L_
SV RP
L_
R
SV BM
L_
PC
U
SV A
L_
U
SV AE
L_
R
U
SV BM
L_
P
U
SV CA
L_
O
R
G
U
SV
L_
FA
SV
L_
U
SV FM
L_
TC
U
SV A
L_
R
U
SV P
L
U
SV _FM
L_
TC
A+

●

SV

SV

●
●
●
●

Fig. 5 The Scott-Knott ESD test results for both the supervised and unsupervised models. Each color
indicates a rank: models in different ranks have a significant difference in performance. Each boxplot has
130 median AUC values (5 defect prediction models times 26 projects). The x-axis refers to the feature
reduction techniques; the y-axis refers to the AUC values. In the x-axis, the “SVL ”-prefix refers to the
5 supervised defect prediction models; the “USVL ”-prefix refers to the 5 unsupervised defect prediction
models.

vised defect prediction models. Figure 4(b) shows the AUC values and the resultsof the Scott-Knott ESD test for the unsupervised models after applying the studied
feature reduction techniques.
The highest rank contains only the two studied neural network-based techniques:
RBM and AE. Hence, the neural network-based feature reduction techniques cansignificantly improve the AUC compared to the original models and other feature
reduction techniques. However, these neural network-based feature reduction techniques do not outperform ORG for the supervised models. In Section 6 we further
investigate why neural network-based feature reduction techniques work well for theunsupervised, but not for the supervised defect prediction models.
The supervised models with feature reduction techniques significantly outperform the unsupervised models with feature reduction techniques. Figure 5
shows the AUC after applying the feature reduction techniques to the supervised andunsupervised models. The supervised models significantly outperform the unsupervised models. Prior research [78] reported that spectral clustering (SC) is the only
studied unsupervised defect prediction model that outperforms the supervised models.
The reason that the unsupervised models perform worse than the supervised models in Figure 5 is that we consider all the unsupervised models together, to be able toprovide a more generic conclusion. However, as Figure 5 shows, some unsupervised
defect prediction models perform better than others.
In the AEEEM dataset, the feature reduction techniques improve the prediction performance of the supervised models for most projects. We observe thatthe feature reduction techniques did not improve the prediction performance in many
projects, as the median values of several boxplots in Figure 6 are lower than 1.0. How18

PROMISE

NASA

An
Ca t v1.
me 7
Ivyl v1.6
Jed v1.4i
Lo t v4.
g 0
Lu 4j v1
ce .0
n
POe v2To I v3..4
mc 0
Xa at v6la .
Xe n v2 0
rce .6
sv
1.3

CM
1
JM
1
KC
3
MC
1
MC
2
MW
1
PC
1
PC
2
PC
3
PC
4
PC
5

The Impact of Feature Reduction Techniques on Defect Prediction Models

19

AEEEM

1.4
1.2
1.0
0.8

Ec

lip

se

PD

EU

I

en
e
My
lyn

he
Lu
c

Ap

Ec
lip

ac

se

JDT

Co
r
Eq e
uin
ox

0.6

UI
DE
eP

Ec

Lu

Eq

he

lip
s

ce
ne
My
lyn

ox
ac
Ap

lip
s
Ec

uin

re

CM
1
JM
1
KC
3
MC
1
MC
2
MW
1
PC
1
PC
2
PC
3
PC
4
PC
5

AEEEM

Co

NASA

1.3
1.2
1.1
1.0
0.9
0.8
0.7
0.6

eJ
DT

PROMISEAn
Ca t v1.
me 7
Ivyl v1.6
Jed v1.4i
Lo t v4.
g 0
Lu 4j v1
ce .0
n
POe v2To I v3..4
mc 0
Xa at v6la .
Xe n v2 0
rce .6sv
1.3(a) The supervised models(b) The unsupervised models
Fig. 6 The ratios of the AUCs of the supervised and unsupervised prediction models. Each boxplot contains 40 ratio values (5 prediction models * 8 feature reduction techniques). The x-axis indicates the studied
projects. The y-axis indicates the ratio. The dashed blue line indicates a ratio of 1.0. A ratio larger than 1.0
indicates that the feature reduction technique improved the AUC compared to the original models, whilea ratio smaller than 1.0 indicates that the feature reduction technique reduced the AUC. A ratio of 1.0 (onthe dashed blue line) indicates that the prediction model that uses features that were generated by a feature
reduction technique had the same performance as the original model.

ever, the studied feature reduction techniques improved the prediction performanceof the supervised models for many projects in the AEEEM dataset (Figure 6(a)). We
further investigate this phenomenon in Section 5.3.1.
The neural network-based techniques improve the prediction performanceof the supervised/unsupervised prediction models for most projects. Table 5 shows
the median ratios for the feature reduction techniques. We observe that the neuralnetwork-based feature reduction techniques RBM and AE have the most gray cells
for the supervised/unsupervised prediction models in combination with the feature
reduction techniques.
However, almost all feature reduction techniques did not improve the prediction
performance in the NASA dataset except for FA with the unsupervised prediction
models. FA combined with the unsupervised prediction models improved over halfof the projects in the NASA dataset. We further investigate why the feature reduc19

20

Kondo et al.

0.3

0.3

0.2

IQR

IQR

●

●

●

0.1

0.2
●
●

0.1

●
●
●
●

●

●

●
●

●

A

G
R
O

PC

P

FM

FA

R

BM
R

A

AE

A+

(a) The supervised models

TC

TC

A

FM

PC

P

A+

R

TC

FA
TC
A
O
R
G

R

AE

0.0
BM

0.0

(b) The unsupervised models

Fig. 7 The Scott-Knott ESD test results for the IQR of the supervised and unsupervised models. Each color
indicates a rank: feature reduction techniques in different ranks have a significant difference in variance(IQR). Each boxplot has 26 IQR values (one for each project). The x-axis refers to the feature reduction
techniques; the y-axis refers to the IQR values.

tion techniques work well for the AEEEM dataset but not for the other datasets in
Section 5.3.1.

5.2 RQ2: What is the impact of feature reduction techniques on the variance ofthe performance across defect prediction models?
Motivation: A challenge in applying defect prediction for practitioners is to selectthe best-performing model for their data from many possible defect prediction models [17, 19]. In this RQ, we studied the variance in performance across defect prediction models of the studied feature reduction techniques for a particular dataset. If thisvariance is small, the practitioner does not need to worry about the choice at all, as
the models perform similarly across datasets.
Approach: We used the interquartile range (IQR) which captures the performancevariance across the studied defect prediction models for a given project and feature
reduction technique. We used the AUC values of all the studied supervised and unsupervised models for all bootstrap samples to conduct a new bootstrap sampling tocalculate the IQR for each reduction technique and each project. We calculated an
IQR value as follows:
1. Sample 100 AUC values at random from the 100 AUC values for each studied
supervised/unsupervised model allowing for replacement.
2. Compute the median AUC value across the sampled 100 AUC values.
3. Repeat steps 1 and 2 100 times.
4. Compute the IQR value for the 500 sampled median AUC values (5 supervised/unsupervised prediction models * 100 median AUC values) for each feature reduction/selection technique for each studied project.
Where the IQR values are computed as follows:
IQR = Q3 − Q1
where Q1 is the first quartile of the 500 sampled median AUC values, and Q3 is thethird quartile of the 500 sampled median AUC values. The first quartile is the median
20

The Impact of Feature Reduction Techniques on Defect Prediction Models

21

Table 5 The median AUC ratios of the feature reduction techniques. The table also contains the medianratios of the CFS and ConFS feature selection techniques, which are studied in RQ3. A ratio larger than
1 indicates that the feature reduction/selection technique improved the AUC compared to the originalmodels, while a ratio smaller than 1 indicates that the feature reduction/selection technique reduced the
AUC. The gray cells refer to the ratios that are greater than 1.0. The “Improved” row indicates the numberof projects for which a feature reduction/selection technique improved the performance.

AE

PCA

FM

FA

RP

TCA

TCA+

CFS

ConFS

PROMISEAnt v1.7
Camel v1.6
Ivy v1.4
Jedit v4.0
Log4j v1.0
Lucene v2.4
POI v3.0
Tomcat v6.0
Xalan v2.6
Xerces v1.3

1.015
0.973
1.065
1.012
1.047
1.016
0.932
1.014
0.861
1.032

1.004
0.979
1.081
0.996
1.034
0.956
0.899
1.016
0.912
1.022

0.983
0.921
1.005
0.971
1.000
0.940
0.955
0.947
0.987
0.969

0.896
0.880
1.005
0.775
0.960
0.804
0.739
0.875
0.698
0.817

0.995
0.996
1.026
1.011
1.012
0.978
0.993
0.997
0.992
1.019

0.956
0.962
0.999
0.921
0.895
0.886
0.946
0.925
0.970
1.017

0.962
0.948
1.036
0.952
0.961
0.962
0.965
0.985
0.993
1.034

0.962
0.948
1.020
0.947
0.954
0.955
0.971
0.990
0.985
1.033

1.012
0.962
1.062
0.984
0.993
0.990
1.016
1.045
1.012
1.005

1.006
0.956
1.037
0.996
1.002
0.978
1.003
1.026
0.998
1.012

NASA

CM1
JM1
KC3
MC1
MC2
MW1
PC1
PC2
PC3
PC4
PC5

1.004
1.004
0.910
0.956
1.019
1.005
0.906
1.039
0.944
0.793
0.999

0.979
1.001
0.944
1.021
1.014
1.016
0.830
1.019
0.971
0.806
0.982

0.939
0.997
0.901
0.932
0.955
0.949
0.936
0.931
0.985
0.931
0.994

0.984
0.923
0.923
0.901
0.947
0.984
0.972
0.994
0.921
0.651
0.798

0.974
0.999
1.010
0.999
1.037
1.019
1.013
1.020
0.997
0.906
0.990

0.995
1.003
0.924
0.915
1.014
0.991
0.976
0.999
0.988
0.862
0.977

0.949
1.008
0.905
0.980
0.976
0.952
0.971
0.935
1.001
0.866
0.989

0.940
0.956
0.874
0.932
0.972
0.963
0.925
0.933
0.968
0.870
0.986

1.028
1.000
1.025
1.001
0.973
1.016
1.004
1.169
1.026
1.003
0.986

0.981
1.001
0.985
0.973
0.968
1.015
1.021
1.028
0.995
0.986
0.997

AEEEM

(a) The supervised models
RBM

Eclipse JDT Core
EquinoxApache Lucene
Mylyn
Eclipse PDE UI

1.028
1.074
1.112
1.089
1.071

1.010
1.043
1.079
1.057
1.058

1.019
1.006
1.090
0.980
1.075

0.874
0.973
1.045
0.921
0.861

1.007
1.028
1.009
1.052
1.035

0.959
0.996
1.063
0.951
0.944

1.015
1.000
1.030
1.066
1.020

0.986
1.010
1.039
1.065
0.905

0.993
1.060
1.021
1.037
0.995

1.012
1.024
1.035
1.023
1.006

Improved

17

15

5

2

14

4

8

5

17

15

RBM

AE

PCA

FM

FA

RP

TCA

TCA+

CFS

ConFS

PROMISEAnt v1.7
Camel v1.6
Ivy v1.4
Jedit v4.0
Log4j v1.0
Lucene v2.4
POI v3.0
Tomcat v6.0
Xalan v2.6
Xerces v1.3

1.005
0.989
1.040
1.055
0.993
1.110
0.919
1.041
1.070
1.244

1.000
0.980
1.011
1.038
0.989
1.131
0.924
1.007
1.065
1.210

1.001
1.002
1.006
1.000
1.001
1.001
1.002
0.997
0.997
1.008

0.739
0.894
0.938
0.895
0.950
0.870
0.755
0.762
0.885
0.885

0.896
0.979
0.850
1.033
0.973
0.989
1.036
0.956
0.936
1.259

0.695
0.848
0.750
0.888
0.694
0.894
0.875
0.652
0.965
0.893

0.918
0.984
0.991
0.966
0.924
0.966
0.610
1.007
1.084
1.201

0.748
0.933
0.874
0.802
0.790
0.858
0.755
0.763
0.848
0.939

1.002
0.977
0.937
0.939
0.954
0.984
0.962
1.010
0.964
1.106

1.006
0.988
0.928
0.962
0.986
1.004
0.982
0.968
0.974
1.017

NASA

CM1
JM1
KC3
MC1
MC2
MW1
PC1
PC2
PC3
PC4
PC5

1.008
0.992
0.970
1.005
1.073
0.939
0.990
0.985
1.074
0.986
0.971

0.996
1.023
0.987
1.023
1.043
0.971
0.996
0.976
1.090
0.982
0.975

0.997
1.002
1.000
1.000
1.000
1.000
0.996
0.994
1.000
0.998
0.999

0.967
0.953
0.974
0.891
0.981
0.973
0.927
0.975
0.940
0.902
0.866

0.979
1.036
1.014
1.021
1.018
0.966
1.027
0.985
0.954
1.062
0.971

1.049
0.997
0.997
1.000
0.985
0.990
0.935
0.993
0.758
0.829
0.987

0.983
1.011
0.998
0.994
0.995
0.957
0.993
0.966
1.041
0.961
0.932

0.844
0.849
0.850
0.816
0.862
0.746
0.855
0.774
0.831
0.818
0.857

1.030
1.008
0.968
1.081
0.993
1.017
1.102
1.035
1.213
1.123
1.030

0.949
1.001
0.955
1.023
0.991
1.019
1.041
0.940
1.123
0.991
0.997

AEEEM

(b) The unsupervised models

Eclipse JDT Core
EquinoxApache Lucene
Mylyn
Eclipse PDE UI

1.121
1.029
1.031
1.006
1.015

1.089
1.105
1.053
1.015
1.024

0.997
1.000
1.000
1.001
1.000

0.888
0.913
0.863
0.829
0.827

0.957
1.069
0.989
0.996
0.980

0.995
1.033
1.006
0.892
0.977

1.107
1.073
1.024
0.994
0.993

0.805
0.900
0.740
0.827
0.793

1.017
1.025
0.956
1.009
1.003

1.007
1.033
0.981
0.996
1.024

Improved

16

15

9

0

10

3

8

0

16

11

21

22

Kondo et al.

between the smallest and the median of the 500 AUC values, and the third quartile isthe median between the median and the largest of the 500 AUC values. As we studied
26 projects, we have 26 IQR values for each feature reduction technique. We used theScott-Knott ESD test to compare the distributions of IQRs for each feature reduction
technique. Figure 7 shows the results of the Scott-Knott ESD test.
In addition, we compared the IQR values of the prediction models across thefeature reduction techniques for each project. Table 6 shows the results of the IQR
analysis at the project level. Each cell contains an IQR value that was computed from500 bootstrapped median AUC values of the supervised and unsupervised prediction
models.
Results: The neural network-based feature reduction techniques, RBM andAE, generate features that result in less variance across the supervised models
than the original features. Figure 7(a) shows that the original features (ORG) arein the second rank, and RBM and AE belong to the first rank. Hence, RBM and AE
significantly improve the variance of the performance across the supervised defectprediction models.
Almost all feature reduction techniques (except PCA) generate features thathave a significantly smaller performance variance across the unsupervised models than the original features. Figure 7(b) shows that the unsupervised models thatuse features that were generated by PCA, or the original features are in the lowest
rank. Hence, using the feature reduction techniques (except PCA) in combination
with an unsupervised defect prediction model results in a small performance variance, which is helpful for practitioners.
The neural network-based feature reduction techniques improve the performance variance of the supervised models for the largest number of projects. Table 6(a) shows that the features that were generated by the neural network-based feature reduction techniques (RBM and AE) improved the performance variance (IQR)
across the studied supervised prediction models for the largest number of projects
compared to the other feature reduction techniques, and the original models. RBM
and AE also belong to the first rank of the overall performance variance result (Figure 7(a)).
The neural network-based feature reduction techniques improve the performance variance of the unsupervised models for the largest number of projects.
Table 6(b) shows that the features that were generated by the neural network-based
feature reduction techniques (RBM and AE) improved the performance varianceacross the studied unsupervised prediction models for the largest number of projects.
Interestingly, in terms of overall performance variance, TCA and TCA+ belong to
the first and the second rank (Figure 7(b)). However, the difference with the neural network-based feature reduction techniques is only small (TCA+) and negligible(TCA), according to the Cliff’s delta effect size.
22

The Impact of Feature Reduction Techniques on Defect Prediction Models

23

Table 6 The IQR ratio for the feature reduction techniques in the studied supervised and unsupervised
prediction models. The gray cells refer to the ratios that are greater than 1.0. The “Improved” row indicatesthe number of gray cells in the column.

RBM

AE

PCA

FM

FA

RP

TCA

TCA+

CFS

ConFS

PROMISEAnt v1.7
Camel v1.6
Ivy v1.4
Jedit v4.0
Log4j v1.0
Lucene v2.4
POI v3.0
Tomcat v6.0
Xalan v2.6
Xerces v1.3

1.769
3.885
1.841
1.018
1.335
1.737
0.973
1.351
0.762
1.918

1.330
3.027
2.964
1.180
2.643
0.661
0.586
1.429
0.615
1.749

0.849
0.842
1.978
0.562
1.030
0.554
0.628
0.671
0.863
0.806

0.439
0.809
2.197
0.501
5.212
0.473
0.241
0.657
0.495
1.022

1.024
0.774
2.001
0.771
1.388
0.635
0.959
1.096
0.725
0.847

0.943
0.777
1.151
0.595
1.898
1.141
1.642
0.739
0.750
0.943

0.975
1.312
3.641
0.688
1.252
0.666
0.894
0.854
0.531
1.064

1.006
1.357
1.233
0.718
1.189
0.647
0.977
0.921
0.556
1.078

1.294
1.637
1.340
0.767
1.361
0.800
1.268
1.369
0.972
1.171

1.031
1.466
2.072
0.683
1.377
0.658
1.157
1.204
0.841
1.417

NASA

CM1
JM1
KC3
MC1
MC2
MW1
PC1
PC2
PC3
PC4
PC5

0.196
2.502
0.582
0.639
1.087
1.199
0.826
0.541
1.060
0.885
0.796

0.227
2.579
0.955
0.670
1.080
1.451
0.984
0.565
1.522
1.504
1.892

0.241
1.886
1.509
0.605
0.545
0.722
0.649
0.454
0.684
0.548
0.752

0.311
0.604
1.130
0.734
1.074
4.598
1.078
0.788
0.985
0.397
0.344

0.329
0.804
0.661
0.980
0.808
2.933
1.326
1.067
1.274
0.852
0.908

0.683
0.972
1.093
0.810
0.835
1.166
1.936
1.317
0.718
1.055
0.705

0.389
1.036
0.906
1.227
0.734
2.113
2.501
0.937
0.862
0.681
0.723

0.430
0.758
0.751
0.967
0.679
1.697
0.875
0.898
0.695
0.750
0.757

0.589
0.892
1.871
0.831
1.121
1.792
1.179
0.796
1.331
5.539
0.798

0.453
0.986
1.710
2.011
1.169
1.274
1.578
0.389
1.145
1.138
1.009

AEEEM

(a) The supervised models

Eclipse JDT Core
EquinoxApache Lucene
Mylyn
Eclipse PDE UI

1.332
1.989
1.373
5.981
1.574

1.433
2.927
1.844
4.024
1.208

1.246
1.543
0.936
0.940
1.096

0.434
0.821
0.608
0.946
0.224

1.556
1.076
0.989
5.923
0.392

0.998
0.822
1.678
1.155
0.374

1.373
1.075
1.014
1.289
0.434

0.970
1.129
0.993
1.282
0.384

1.072
1.438
0.881
1.992
0.475

1.393
1.337
1.022
2.406
0.452

Improved

17

18

7

7

11

11

12

8

16

19

AE

PCA

FM

FA

RP

TCA

TCA+

CFS

ConFS

PROMISEAnt v1.7
Camel v1.6
Ivy v1.4
Jedit v4.0
Log4j v1.0
Lucene v2.4
POI v3.0
Tomcat v6.0
Xalan v2.6
Xerces v1.3

2.693
1.224
2.881
3.221
3.400
0.460
0.764
2.993
2.028
8.508

1.772
1.715
4.073
3.229
3.445
0.982
0.582
1.985
3.317
6.308

1.035
1.331
0.924
1.012
1.221
0.653
1.152
1.116
0.763
1.132

2.539
0.961
2.096
1.704
1.346
0.796
7.759
2.326
2.339
11.297

3.228
1.659
3.690
4.387
0.542
0.577
1.425
3.566
1.363
6.729

3.217
0.778
3.628
2.213
0.977
1.865
7.589
4.079
3.747
10.673

1.447
0.503
3.317
3.346
1.562
0.945
4.332
1.512
0.829
2.089

4.320
0.680
3.076
4.496
0.783
0.560
2.842
5.814
0.907
10.157

0.335
0.209
2.809
0.671
0.762
0.176
0.415
0.630
0.362
4.720

0.245
0.301
2.153
0.790
0.793
0.163
0.927
0.904
2.860
0.980

NASA

CM1
JM1
KC3
MC1
MC2
MW1
PC1
PC2
PC3
PC4
PC5

2.032
1.206
0.920
5.284
1.377
2.059
2.413
2.058
2.293
4.032
5.900

2.820
5.044
0.753
3.674
2.056
1.207
2.298
1.900
2.230
3.431
11.856

1.074
0.988
0.936
1.035
1.839
0.952
0.996
1.014
1.069
0.917
1.025

0.694
0.983
1.280
0.576
0.778
1.043
1.047
2.441
2.318
4.188
18.438

1.274
1.360
0.639
3.833
1.670
0.897
1.405
5.610
4.411
1.266
4.839

1.535
1.304
0.780
1.261
1.919
1.008
0.866
1.035
7.025
3.476
2.096

1.573
39.761
0.946
5.611
2.709
1.325
3.256
5.055
13.1469.617
30.5033.922
154.852
3.250
3.635
4.491
1.520
3.491
7.867
13.20310.161
5.129

1.238
1.210
0.591
1.132
1.457
0.814
0.964
2.231
2.622
1.653
1.355

0.688
1.009
0.750
0.971
1.441
0.816
0.748
1.436
1.182
1.177
1.267

AEEEM

(b) The unsupervised models
RBM

Eclipse JDT Core
EquinoxApache Lucene
Mylyn
Eclipse PDE UI

15.4973.504
8.221
10.9403.237

15.262
7.741
6.507
9.597
2.806

0.973
0.684
0.989
1.038
1.068

1.727
3.573
0.521
8.177
0.690

1.290
1.155
4.423
1.099
1.235

3.088
1.420
1.610
1.036
0.465

20.6806.460
5.717
17.600
5.627

8.202
9.422
8.040
11.010
6.956

1.104
0.676
0.659
0.980
0.389

1.505
0.601
0.920
0.889
0.366

23

23

15

18

22

21

22

22

11

9

Improved

23

Kondo et al.

0.9

0.9

0.8

0.8

0.7

0.7

AUC

AUC

24

●

0.6

0.6

A+

TC

A
R
P
FM

TC

FA

FS

on

●
●
●
●

C

C
FS
PC
A
O
R
G

A
FM

PC

R
P
BM
R

R
G
TC
A
TC
A+
AE

O

FS

FA
C

on

FS
C

(a) The supervised models

AE
BM

0.5

●

R

0.5

●
●
●
●
●
●
●
●

(b) The unsupervised models

Fig. 8 The Scott-Knott ESD test results for the supervised (logistic regression, random forest, naive
Bayes, decision tree, and logistic model tree) and unsupervised (spectral clustering, k-means, partition
around medoids, fuzzy C-means, neural-gas) models. Each color indicates a rank: models in different ranks
have a statistically significant difference in performance. Each boxplot has 130 median AUC values (5defect prediction models times 26 projects). The x-axis refers to the feature reduction/selection techniques;
the y-axis refers to the AUC values.

5.3 RQ3: How do feature selection techniques compare to feature reduction
techniques when applied to defect prediction?
In this RQ, we compare feature reduction and selection techniques along two dimensions: the performance and the performance variance of the defect prediction models.
We study the correlation-based (CFS) and consistency-based (ConFS) feature selection techniques, as they performed best according to prior studies [18, 76].
Motivation: In RQ1 and RQ2, we found that several feature reduction techniques(FA, RBM and AE) outperform the original features (ORG) in terms of performanceor performance variance of the defect prediction models. Prior work [18, 76] showedthat several feature selection techniques outperform the original models as well. In
this RQ, we compare the performance (AUC) and the performance variance (IQR)of the feature reduction and selection techniques when applied to defect prediction
models.
Approach: The experimental procedure is the same as the procedures of RQ1 and
RQ2 (only we use the two feature selection techniques CFS and ConFS instead of thefeature reduction techniques).
Results: The feature selection techniques (correlation-based feature selection(CFS) and consistency-based feature selection technique (ConFS)) significantly
outperform the original features (ORG) in the supervised models, and perform
as well as the feature agglomeration (FA) reduction technique. Figure 8(a) shows
the AUC values and the results of the Scott-Knott ESD test for the supervised models
24

The Impact of Feature Reduction Techniques on Defect Prediction Models

0.3

0.3
●

0.2
●

●

IQR

IQR

25

●

0.1

0.2

●

●
●

0.1

●
●
●

●

●

●

●
●

●

(a) The supervised models

C

C
FS
on
FS

A

G
R

PC

O

FA
R
P
FM

A+
TC
A
AE
R
BM

TC

R
P
TC
A+
PC
A
FM

FA
TC
A
O
R
G

0.0
AE
C
FS
R
BM
C
on
FS

0.0

(b) The unsupervised models

Fig. 9 The Scott-Knott ESD test results for IQR in the supervised and unsupervised models. Each color
indicates a rank: feature reduction/selection techniques in different ranks have a significant difference invariance (IQR). Each boxplot has 26 IQR values (one for each project). The x-axis refers to the feature
reduction/selection techniques; the y-axis refers to the IQR values.

after applying the studied feature reduction and selection techniques.2 Each boxplot
shows the median AUC values for the projects using a certain feature reduction/selection technique. CFS, ConFS and FA are in the highest rank by themselves, whichindicates that the subsets of features that were selected by CFS or ConFS perform as
well as the feature sets that were generated by FA for the supervised models.
The neural network-based feature reduction techniques (RBM and AE) significantly outperform the feature selection techniques (CFS and ConFS) for theunsupervised defect prediction models. The highest rank contains only the twostudied neural network-based feature reduction techniques (Figure 8(b)). The studied
feature selection techniques (CFS and ConFS) belong to the second rank, together
with the original models (ORG). Hence, the studied feature selection techniques havea worse performance than the neural network-based feature reduction techniques forthe unsupervised defect prediction models.
In the supervised models, applying the neural network-based feature reduction techniques, RBM and AE, or the feature selection techniques, CFS and
ConFS, significantly outperforms the original models in terms of performancevariance. The original models (ORG) belong to the third rank (Figure 9(a)). The neural network-based feature reduction techniques and the feature selection techniques
belong to the first or second rank, hence they have a smaller performance variancethan the original models.
In the unsupervised models, all feature reduction techniques (except PCA)
significantly outperform the feature selection techniques in terms of performancevariance. The feature selection techniques belong to the worst rank together with theoriginal models (ORG) and PCA (Figure 9(b)). Hence, the studied feature selection
had a larger performance variance than almost all the studied feature reduction techniques for the unsupervised defect prediction models.
Our above findings for RQ3 are confirmed by our project-level analysis. Table 5 shows the median ratios of the performance of each feature reduction/selection
2 Note that the ranks are slightly different from Figure 4 due to the fact that Scott-Knott ESD is aclustering algorithm, and hence affected by the total set of input distributions. For more information see
https://github.com/klainfo/ScottKnottESD.

25

26

Kondo et al.

technique compared to the original models. We calculated this ratio as follows:
ratio =AUC FRSAUCORG

Where AUCORG is the AUC (the median AUC across all bootstrap samples) of a prediction model using the original features, and AUCFRS is the AUC of a prediction
model using the features that were generated/selected by a particular feature reduction or selection technique. We computed the median ratio across the five studied
supervised and unsupervised prediction models. Table 5 confirms our above findings
about the performance of the studied feature selection techniques compared to that of
the feature reduction techniques.
Table 6 shows the IQR ratio values of each feature reduction/selection technique.
We define this ratio as follows:
ratio =IQRFRSIQRORG

Where IQRORG is the IQR (the median IQR across all bootstrap samples) of a prediction model using the original features, and IQRFRS is the IQR of a prediction modelusing the features that were generated/selected by a particular feature reduction or
selection technique. We calculated the median IQR value for the supervised models
using bootstrap samples as follows:
1. Sample 100 values following the distribution of the 100 AUC values for eachstudied supervised model while allowing for replacement.
2. Compute the median AUC value across the sampled 100 values.
3. Repeat steps 1 and 2 100 times.
4. Compute the IQR value for the 500 sampled median AUC values (5 supervised
prediction models * 100 median AUC values) for each feature reduction/selection
technique for each studied project.
We repeated the above procedure for the unsupervised models. Table 6(a) showsthat the project-level results confirm our findings above, as the RBM and AE feature
reduction techniques and the CFS and ConFS feature selection techniques improve
the performance variance of most projects compared to the other techniques. In addition, Table 6(b) shows that all feature reduction techniques improve the performancevariance of more projects than the CFS and ConFS feature selection techniques.
5.3.1 Why do feature reduction techniques work well in the AEEEM dataset?
Motivation: We observed that the feature reduction techniques work better for the
projects in the AEEEM dataset than for the projects in the other datasets. Ghotra etal. [18] applied PCA to the data of each project to capture its richness. We use thesame analysis to investigate whether the dataset richness is an explanation of why
feature reduction techniques work better for the AEEEM dataset.
Approach: The idea behind Ghotra et al.’s analysis [18] is to generate featuresfrom a dataset using PCA that (together) retain at least 95% of the variance of the26

The Impact of Feature Reduction Techniques on Defect Prediction Models

27

Table 7 The number of generated features (principal components) that are needed to account for 95% of
the data variance.
StudiedDatasetStudied
Project# of Studied
Features# of Generated
Features% of Generated
Features

PROMISEAnt v1.7
Camel v1.6
Ivy v1.4
Jedit v4.0
Log4j v1.0
Lucene v2.4
POI v3.0
Tomcat v6.0
Xalan v2.6
Xerces v1.3

20
20
20
20
20
20
20
20
20
20

12
12
10
12
12
12
12
12
12
12

60.0
60.0
50.0
60.0
60.0
60.0
60.0
60.0
60.0
60.0

NASA

CM1
JM1
KC3
MC1
MC2
MW1
PC1
PC2
PC3
PC4
PC5

37
21
39
38
39
37
37
36
37
37
38

11
8
10
15
11
11
12
10
13
14
15

29.7
38.1
25.6
39.5
28.2
29.7
32.4
27.8
35.1
37.8
39.5

AEEEM

Eclipse JDT Core
EquinoxApache Lucene
Mylyn
Eclipse PDE UI

212
212
212
212
212

36
31
33
46
38

17.0
14.6
15.6
21.7
17.9

original dataset. Ghotra et al. reason that a larger number of generated features indicates a richer dataset. Likewise, they interpret that a small number of generated
features indicates redundancy in the original dataset. We applied PCA to each projectand counted the number of generated features.
Results: The PROMISE, NASA, and AEEEM datasets have different datarichness characteristics, however; the characteristics of the projects within eachdataset are consistent. Table 7 shows the number of generated features. While thenumber of generated features is approximately the same for the PROMISE and NASAprojects, the proportion of generated features compared to the number of original features is different. In addition, this proportion is even lower for the AEEEM projects.
Hence, we conclude that the datasets have different characteristics in terms of datarichness. However, within each dataset, the projects have approximately the samerichness characteristics.
The original features of the projects in the AEEEM dataset are more diverse
than the projects of the other datasets. We observe that 36 principal components
are needed to cover 95% of the variance of the Eclipse JDT Core project in Figure 10,compared to 12 components for the Ant project and 11 for the CM1 project. Hence,
the original features of the AEEEM dataset are much more diverse than those of the27

28

Kondo et al.

The cumulative proportionof the variance●

Ant v1.7 (PROMISE)

1.00
0.75

●

●

●

●

● ●

CM1 (NASA)

Eclipse JDT Core (AEEEM)● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

●
●

0.50

●
●

0.25
0.00
0

10
20
30
Number of features/principal components

Fig. 10 The number of principal components (features that were generated by PCA) that are needed toaccount for the original data variance for the Ant, CM1 and Eclipse projects in the PROMISE, NASAand AEEEM datasets. The x-axis indicates the number of principal components. The y-axis indicates the
cumulative proportion of the variance. The other projects of the datasets showed a similar pattern.

PROMISE and the NASA datasets. The diversity of the AEEEM dataset could be areason why feature reduction techniques improve the performance of this dataset.

5.3.2 Comparing feature selection and reduction techniques along the dimensions ofunderstandability and execution time
The understandability of the metrics in a defect prediction model is important, as
understandable metrics make the model, and its predictions, easier to explain [68].
Feature reduction techniques combine the original features into one or more newlygenerated features. Hence, these newly-generated features are by definition harder tounderstand than the features that are a subset (i.e., they were selected) of the original
features. We inspected the feature sets that were generated during our experiments,
and we observed that almost all generated feature sets consist of features that are
a complex combination of all available original features. Hence, defect prediction
models that are generated using feature reduction techniques are harder to understandthan those that use feature selection.
In addition, the execution time of a feature reduction/selection technique and a
defect prediction model is important – models that take too long to build or execute
are not very useful in practice. To conduct our experiments in a timely manner, we ranthem on a cluster of servers in parallel. Hence, it is difficult to compare the execution
time of the experiments. In general, the execution time of our feature reduction/selection techniques and defect prediction models was short (i.e., in the range of minutes).
Therefore, execution time is not a very problematic metric for practitioners who wish
to apply feature reduction or selection to their defect prediction models.
28

The Impact of Feature Reduction Techniques on Defect Prediction Models

29

6 Discussion: Which features are generated by the feature reduction
techniques?
In RQ1, RQ2, and RQ3, we observed that some feature reduction/selection techniques generate/select features that perform defect prediction better and less variancethan the features that were generated/selected by other feature reduction/selection
techniques. In particular, we found that RBM and AE outperform the other studied
feature reduction/selection techniques for the unsupervised models. However, RBM
and AE are less-performing feature reduction techniques than the original models(ORG) for the supervised models. In this discussion, we take a closer look at the generated features to investigate why neural network-based feature reduction techniques
perform well for the unsupervised defect prediction models, but not for the supervised models. In this section, we discuss possible explanations for the differences in
AUC and variance of the performance.
Approach: We focused our discussion on the RBM and AE feature reduction
techniques, as these techniques generate new features by assigning (combinationsof) weights to the original features. For example, a newly generated feature may begenerated by 0.5 times original metric 1 and 0.5 times original metric 2. These weightsets allow us to study how the new features are related to the original features, andto the features that were generated by other feature reduction techniques, and extract
possible explanations for the improved and small variance of the performance. RP
and PCA also generate new features by assigning weights to the original features,
however, the weights of RP are randomly generated, and PCA generates a different
number of features for each project, which makes them difficult to compare. Hence,
we focused our discussion on RBM and AE. As we generated 100 new feature setsof 10 features using RBM and AE, we generated 1,000 weight sets using these two
feature reduction techniques for each project. For each feature reduction technique,we randomly selected 10 (out of the 100) generated feature sets for our investigation.
We conducted correlation analysis and clustering analysis on the studied feature sets to study their similarity within and across projects. The correlation analysis
shows how independent the features that are generated for a project are. Highly correlated features can negatively impact the performance of regression models [15] andthis effect can affect our supervised models as well. We first calculated the Spearman
rank correlation [82] between the generated features in a feature set within a project.
Each generated feature was normalized using the z-score. We chose Spearman rankcorrelation because it is non-parametric, and therefore requires no assumption about
the distribution of the studied data.
To study the similarity of the generated feature sets across projects within adataset, we compared the weight sets of the generated features. First, we normalized all weights using z-score normalization. Second, we used k-means to cluster theweight sets of the features, and then we used t-distributed stochastic neighbour embedding (t-SNE) [41] to visualize the clustering results. t-SNE is commonly used forvisualizing high dimensional features in scatter plots [40]. In particular, t-SNE models high-dimensional objects (i.e., feature sets) by two- or three-dimensional points
such that similar objects are close, and dissimilar objects are further away from eachother.
29

30

Kondo et al.

1.000 1.000 1.000 1.000 1.000 1.000 1.000 0.999 0.999 1.000

H11.000 0.609 0.977 0.995 0.328 0.348 0.916 0.319 0.985 0.532

H1

1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000

H11.000 1.000 1.000 1.000 1.000 1.000 0.999 0.999 1.000

H2

1.000 0.486 0.616 0.851 0.825 0.819 0.840 0.534 0.914

H2

1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000

H2

1.000 1.000 1.000 1.000 1.000 1.000 0.999 1.000

H31.000 0.977 0.186 0.209 0.836 0.187 0.994 0.399

H3

1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000

H31.000 1.000 1.000 1.000 1.000 0.999 1.000

H4

1.000 0.328 0.342 0.913 0.320 0.982 0.531

H4

1.000 1.000 1.000 1.000 1.000 1.000 1.000

H4

1.000 0.999 1.000 0.999 0.998 1.000

H5

1.000 0.946 0.589 0.973 0.237 0.930

H51.000 1.000 1.000 1.000 1.000 1.000

H5

1.000 1.000 1.000 1.000 1.000

H6

1.000 0.611 0.894 0.256 0.904

H61.000 1.000 1.000 1.000 1.000

H6

1.000 1.000 0.999 1.000

H7

1.000 0.575 0.863 0.766

H7

1.000 1.000 1.000 1.000

H7

1.000 1.000 1.000

H8

1.000 0.235 0.911

H8

1.000 1.000 1.000

H8

1.000 1.000

H9

1.000 0.445

H91.000 1.000

H9

H9

H10

H10

H8

H7

H6

H5

H4

H3

H2

1.000

H1

H9

H10

H10

H8

H7

H6

H5

H4

H3

H2

1.000

H1

H9

H10

H10

H8

H7

H6

H5

H4

H3

H2

H1

1.000

(a) RBM-generated features in (b) RBM-generated features (c) RBM-generated features in
the Ant project in the PROMISE in the Eclipse project in the the Eclipse project using change
dataset.
AEEEM dataset.
features in the AEEEM dataset.
1.000 0.852 0.918 0.800 0.809 0.845 0.899 0.863 0.876 0.841

H1

1.000 0.306 0.343 0.230 0.343 0.331 0.533 0.462 0.579 0.216

H11.000 0.836 0.768 0.768 0.893 0.886 0.865 0.889 0.915

H2

1.000 0.058 0.173 0.631 0.001 0.604 0.148 0.463 0.030

H21.000 0.874 0.854 0.832 0.879 0.883 0.883 0.853

H3

1.000 0.562 0.273 0.275 0.205 0.265 0.442 0.605

H3

1.000 0.747 0.732 0.767 0.768 0.850 0.781

H4

1.000 0.067 0.074 0.425 0.087 0.404 0.640

H41.000 0.796 0.844 0.902 0.829 0.837

H5

1.000 0.475 0.404 0.343 0.546 0.041

H5

1.000 0.902 0.886 0.844 0.889

H6

1.000 0.079 0.376 0.333 0.034

H61.000 0.901 0.845 0.884

H7

1.000 0.035 0.768 0.249

H7

1.000 0.878 0.933

H8

1.000 0.374 0.167

H8

1.000 0.902

H91.000 0.235

H9

H10

H10

H10

H9

H8

H7

H6

H5

H4

H3

H2

1.000

H1

H10

H9

H8

H7

H6

H5

H4

H3

H2

H1

1.000

(d) AE-generated features in the (e) AE-generated features in the
Log4j project in the PROMISE Camel project in the PROMISE
dataset.
dataset.
Fig. 11 The Spearman rank correlation of generated features in the studied datasets. The darker colours
indicate a strong absolute correlation (close or equal to one). The lighter colours indicate a weak absolutecorrelation (close or equal to zero).

The goal of our clustering analysis is to find out how similar the generated features
are across projects within a dataset. Hence, we configured k-means to search for 10
clusters (as we are generating 10 new features). We visualized the clustering results
using the default settings of t-SNE.
Results: RBM generates feature sets in which all features are strongly correlated with each other. Figure 11(a) shows the Spearman rank correlation of one
set of RBM-generated features for the Ant project in the PROMISE dataset using aheatmap. We observe that all correlations are close to 1 (dark red), which means thatall RBM-generated features in the feature set are strongly correlated (and hence similar) to each other. We observe similar correlations for the other studied feature sets
for the PROMISE and NASA datasets. RBM generates weakly correlated features forseveral projects in the AEEEM dataset (e.g., for the Eclipse project: Figure 11(b)).
However, if we use a smaller set of original features from that dataset, such as only
the change features, RBM generates strongly correlated features for these projects aswell (e.g., for the Eclipse project: Figure 11(c)).
We observe similar correlations in feature sets that were generated by AE. However, the correlation within the AE-generated feature sets appeared to be linked tothe specific project. For example, AE generates sets of features that are strongly cor30

The Impact of Feature Reduction Techniques on Defect Prediction Models

30

30

20

20
t-SNE feature2

t-SNE feature2

31

10
0
10

10
0
10
20

20

30
20

10

0
10
t-SNE feature1

20

30

30

20

10 0
10
t-SNE feature1

20

30

40

(a) RBM-generated weight sets in the (b) RBM-generated weight sets in the NASA
PROMISE dataset.
dataset.

60

40

40

20

20

t-SNE feature2

t-SNE feature2

60

0
20
40
60

0
20
40
60

80

60

40

20 0
20
t-SNE feature1

40

60

40

30

20

10 0
10
t-SNE feature1

20

30

40

60

100
75
50
25
0
25
50
75
100

40

20
t-SNE feature2

t-SNE feature2(c) RBM-generated weight sets in the (d) RBM-generated weight sets using Change
AEEEM dataset.
features in the AEEEM dataset.

0

20

40

150

100

50 0
50
t-SNE feature1

100

150

60

40

20

0
20
t-SNE feature1(e) RBM-generated weight sets using com- (f) AE-generated
plexity code change features in the AEEEM PROMISE dataset.
dataset.

60

weightsets

020
t-SNE feature1

40

in

the

60

30
20

40

10

20

t-SNE feature2

t-SNE feature2

40

0
10

0
20
40

20

60

30
30

20

10

0
t-SNE feature1

10

20

40

20

(g) AE-generated weight sets in the NASA (h) AE-generated weight sets using Change
dataset.
features in the AEEEM dataset.
Fig. 12 k-means clustering results with t-SNE for generated weight sets in the studied datasets. Each shape
represents a project, and each color represents a cluster.
31

32

Kondo et al.

related to each other for the Log4j project (see Figure 11(d)), but features that are
not as strongly correlated for the Camel project (see Figure 11(e)). Hence, a possible
explanation of the reason for the small variance of the performance of features thatwere generated by the neural network-based feature reduction techniques (i.e., RBM
and AE) could be the strong correlation within the generated feature sets.
RBM mostly generates the same feature sets across projects. Figure 12(a)shows the k-means clustering result for one bootstrap sample of the RBM-generated
weight sets in the PROMISE dataset using t-SNE [40]. Each shape refers to a project,and each color refers to a cluster that was identified by k-means clustering. The xaxis and y-axis refer to the t-SNE features that were generated from the 260 RBMgenerated weight sets (10 weight sets on each project) by t-SNE. Hence, if there are
10 clearly identifiable groups of different shapes with the same colour in the t-SNE
plot, we can conclude that the generated weight sets (and hence the features) are thesame across projects. We observe that each colored cluster contains all shapes, whichindicates that each cluster contains all projects. Hence, in this bootstrap sample, RBM
generated the same feature sets across projects in the PROMISE dataset. We noticed
a similar pattern for the other projects in the PROMISE dataset for the other bootstrap
samples.
Figure 12(b) shows the result for the RBM-generated weight sets in the NASAdataset. The figure shows that RBM generated different feature sets across projects in
the NASA dataset. A possible reason is that the original features in the NASA dataset
are different in each project (Table 2).
Figure 12(c) shows the result for the RBM-generated weight sets in the AEEEM
dataset. The figure shows that RBM generated different feature sets across projects
in the AEEEM dataset. The explanation is similar to our observation during the correlation analysis. If we use one type of features, such as only the change features,
Figure 12(d) shows that RBM generates the same feature sets across projects. However, if we use complexity code change features, we observe that RBM generates
different feature sets across projects (Figure 12(e)). AE generated different features
across projects (Figure 12(f), 12(g) and 12(h)).
From the discussion results, we can extract several possible explanations for thefact that RBM and AE significantly improve the variance of the performance acrossthe unsupervised models, but not across the supervised models, and for why these feature reduction techniques perform well for the unsupervised models. As the studied
neural network-based feature reduction techniques appear to generate strongly correlated features for a project, these feature sets suffer from multicollinearity [15], which
is known to negatively affect the performance of the supervised models. However, as
the unsupervised defect prediction models do not need to be trained, these models
are not be affected by the multicollinearity problem. In addition, because RBM generates strongly correlated features for a project, the unsupervised models becomemuch simpler, which seems to improve the variance of the performance across theunsupervised defect prediction models.
32

The Impact of Feature Reduction Techniques on Defect Prediction Models

33

7 Threats to validity
7.1 External validity
With regards to the generalizability of our results, we applied our experiments to threepublicly available datasets. These studied datasets (PROMISE, NASA and AEEEM)
were all used in many prior defect prediction studies. The projects in these studied
datasets span different domains, include both open source and industrial projects andcontain different features. Future studies are necessary to investigate whether our
results generalize to other projects.
In addition, we studied only a subset of the many existing feature reduction andselection techniques and defect prediction models. We carefully selected techniquesand models that have been used before for defect prediction, and that have an implementation readily available. Without such an implementation, it is difficult and timeconsuming to ensure that the implementation matches the one used in prior studies.
Future studies are necessary to investigate whether our results apply to other feature
reduction/selection techniques and defect prediction models.

7.2 Internal validity
In our experiments, we used AUC as a performance measure. AUC is a popular performance measure for defect prediction, as it does not require a threshold. However,different software project teams may have different objectives. Hence, future studies
should investigate the impact of feature reduction techniques on other performancemeasures, while keeping in mind the possible pitfalls of studying threshold-dependent
performance measures [71].
When using the out-of-sample bootstrap sampling, we encountered computational errors in two bootstrap samples. The errors occurred because two bootstrap
samples violated requirements of the NB and SC models (e.g., a generated feature
had a variance of zero which violates a requirement of the NB model). To mitigate
this threat, we discarded these bootstrap samples and generated a new sample instead.
We provide all experimental scripts that we used in our study.3 This replication
package allows researchers and practitioners to replicate our experiments and confirm
our results.

8 Conclusion
In defect prediction, reducing the number of features is an important step when building defect prediction models [5, 15, 67, 69]. Prior studies indicated that reducing thenumber of features avoids the problem of multicollinearity [15] and the curse of dimensionality [5]. Feature selection and reduction techniques help to reduce the number of features in a model. Feature selection techniques reduce the number of featuresin a model by selecting the most important ones, while feature reduction techniques
3

https://sailhome.cs.queensu.ca/replication/featred-vs-featsel-defectpred/

33

34

Kondo et al.

reduce the number of features by creating new, combined features from the original
features.
Prior work [18, 76] studied the impact of feature selection techniques on defectprediction models. Our work is the first large-scale study on the impact of feature
reduction techniques on defect prediction models. In particular, we studied the impact of eight feature reduction techniques on five supervised and five unsupervised
defect prediction models. In addition, we compared the impact of feature reduction
techniques on defect prediction with the impact of the two best-performing feature
selection techniques (according to prior work).
We studied the impact of feature reduction/selection techniques on defect prediction models along two dimensions:
1. The defect prediction performance (AUC) of the features that are generated by
the feature reduction/selection techniques, to study whether feature reduction/selection techniques can improve the performance of defect prediction models.
2. The variance of the AUC across defect prediction models that use the features thatare generated by feature reduction or selected by feature selection techniques. It isdifficult to select the best performing model for each project, since the best model
may change per project [17, 19]. Hence, we studied whether feature reduction or
selection techniques can relieve the burden for practitioners of having to choose
the best performing defect prediction model for their data.
Below, we summarize the main recommendations that follow from our work.
Recommendation 1: For the supervised defect prediction models, use thecorrelation-based (CFS) or consistency-based (ConFS) feature selection techniques. Our experiments in RQ3 show that, for the supervised models, CFS and
ConFS outperform the feature reduction techniques (except feature agglomeration(FA)) and the original models. While FA has a similar performance, CFS and ConFS
have a smaller performance variance. Hence, using CFS or ConFS in combination
with a supervised defect prediction model allows practitioners to improve the performance of their defect prediction models, while making the choice for a particular
defect prediction model easier as well.
Recommendation 2: For the unsupervised defect prediction models, use aneural network-based technique (Restricted Boltzmann Machine (RBM) or autoencoder (AE)). Our experiments in RQs 1 and 3 show that the RBM and AE feature
reduction technique can significantly improve the performance of the unsupervised
defect prediction models compared to the other feature reduction/selection techniques
and the original models. In addition, we observed in RQs 2 and 3 that RBM andAE significantly improve the performance variance across the unsupervised models (except compared to the transfer component analyses (TCA and TCA+)). While
the transfer component analyses (TCA and TCA+) have the smallest performancevariance, they have a worse performance than RBM and AE. The effect size (Cliff’s
Delta) between the transfer component analyses and the neural network-based feature
reduction techniques is negligible or small for the performance variance in favourof the transfer component analyses, but small (TCA) or large (TCA+) for the performance in favour of the neural network-based techniques. Hence, using a neuralnetwork-based feature reduction technique to preprocess the data of the unsupervised
34

The Impact of Feature Reduction Techniques on Defect Prediction Models

35

defect prediction models can improve both their performance and relieve the burden
for practitioners of having to select the best-performing unsupervised defect prediction model for their project.
Recommendation 3: If a project has diverse data, the neural network-based
techniques (Restricted Boltzmann Machine (RBM) or autoencoder (AE)) arelikely to improve its defect prediction performance and performance variance.
Our experiments showed that RBM and AE consistently improve the AUC and IQRof projects in the AEEEM dataset, for both supervised and unsupervised models.
Practitioners should run PCA on their data to identify the diversity of their project’s
data (similar to what we did in Section 5.3.1). If the data turns out to be rich, RBM
and AE are good options to improve the defect prediction performance and variance.
Acknowledgment
This work was partially supported by JSPS KAKENHI Grant Number 16K12415.
References
1. Scikit learn: Minmaxscaler. http://scikit-learn.org/stable/modules/
generated/sklearn.preprocessing.MinMaxScaler.html. [Online; accessed 1-July-2018]
2. Abaei, G., Rezaei, Z., Selamat, A.: Fault prediction by utilizing self-organizing
map and threshold. In: Proceedings of the International Conference on Control
System, Computing and Engineering (ICCSCE), pp. 465–470. IEEE (2013)
3. Arora, I., Tetarwal, V., Saha, A.: Open issues in software defect prediction. Procedia Computer Science 46, 906–912 (2015)
4. Basili, V.R., Briand, L.C., Melo, W.L.: A validation of object-oriented design
metrics as quality indicators. IEEE Transactions on Software Engineering (TSE)
22(10), 751–761 (1996)
5. Bellman, R.: Dynamic programming. Princeton University Press (1957)
6. Bingham, E., Mannila, H.: Random projection in dimensionality reduction: applications to image and text data. In: Proceedings of the 7th International Conference on Knowledge Discovery and Data Mining, pp. 245–250. ACM (2001)
7. Bishnu, P.S., Bhattacherjee, V.: Software fault prediction using quad tree-based
k-means clustering algorithm. IEEE Transactions on Knowledge and Data Engineering 24(6), 1146–1150 (2012)
8. Challagulla, V.U.B., Bastani, F.B., Yen, I.L., Paul, R.A.: Empirical assessmentof machine learning based software defect prediction techniques. International
Journal on Artificial Intelligence Tools 17(02), 389–400 (2008)
9. Chidamber, S.R., Kemerer, C.F.: A metrics suite for object oriented design. IEEE
Transactions on Software Engineering (TSE) 20(6), 476–493 (1994)
10. Cohen, J.: Statistical power analysis for the behavioral sciences (1988)
11. D’Ambros, M., Lanza, M., Robbes, R.: An extensive comparison of bug prediction approaches. In: Proceedings of the 7th International Conference on Mining
Software Repositories (MSR), pp. 31–41. IEEE (2010)35

36

Kondo et al.

12. Dash, M., Liu, H.: Consistency-based search in feature selection. Artificial intelligence 151(1), 155–176 (2003)
13. Dunn, J.C.: A fuzzy relative of the isodata process and its use in detecting compact well-separated clusters. J. Cybernet 3, 32–57 (1973)
14. Faloutsos, C., Lin, K.I.: FastMap: A fast algorithm for indexing, data-mining andvisualization of traditional and multimedia datasets. In: Proceedings of the ACM
SIGMOD International Conference on Management of Data, pp. 163–174. ACM(1995)
15. Farrar, D.E., Glauber, R.R.: Multicollinearity in regression analysis: the problem
revisited. The Review of Economic and Statistics pp. 92–107 (1967)
16. Gao, K., Khoshgoftaar, T.M., Wang, H., Seliya, N.: Choosing software metrics
for defect prediction: an investigation on feature selection techniques. Software:
Practice and Experience 41(5), 579–606 (2011)
17. Ghotra, B., McIntosh, S., Hassan, A.E.: Revisiting the impact of classification
techniques on the performance of defect prediction models. In: Proceedings ofthe 37th International Conference on Software Engineering (ICSE), pp. 789–800.
IEEE Press (2015)
18. Ghotra, B., Mcintosh, S., Hassan, A.E.: A large-scale study of the impact offeature selection techniques on defect classification models. In: Proceedings ofthe 14th International Conference on Mining Software Repositories (MSR), pp.
146–157. IEEE Press (2017)19. Gray, A.R., Macdonell, S.G.: Software metrics data analysis–exploring the relative performance of some commonly used modeling techniques. Empirical Software Engineering 4(4), 297–316 (1999)
20. Guo, L., Cukic, B., Singh, H.: Predicting fault prone modules by the dempstershafer belief networks. In: Proceedings of the 18th International Conference on
Automated Software Engineering (ASE), pp. 249–252. IEEE (2003)
21. Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., Witten, I.H.: The
WEKA data mining software: an update. ACM SIGKDD explorations newsletter11(1), 10–18 (2009)22. Hall, M.A.: Correlation-based feature selection for machine learning. Ph.D. thesis, University of Waikato Hamilton (1999)
23. Hall, M.A., Holmes, G.: Benchmarking attribute selection techniques for discrete class data mining. IEEE Transactions on Knowledge and Data Engineering
15(6), 1437–1447 (2003)
24. Halstead, M.H.: Elements of software science, vol. 7. Elsevier New York (1977)
25. Han, J., Moraga, C.: The influence of the sigmoid function parameters on the
speed of backpropagation learning. In: Proceedings of the International Workshop on Artificial Neural Networks, pp. 195–201. Springer (1995)
26. Hartigan, J.A., Wong, M.A.: Algorithm as 136: A k-means clustering algorithm.
Journal of the Royal Statistical Society. Series C (Applied Statistics) 28(1), 100–108 (1979)
27. Hassan, A.E.: Predicting faults using the complexity of code changes. In: Proceedings of the 31st International Conference on Software Engineering (ICSE),
pp. 78–88. IEEE Computer Society (2009)36

The Impact of Feature Reduction Techniques on Defect Prediction Models

37

28. He, Z., Shu, F., Yang, Y., Li, M., Wang, Q.: An investigation on the feasibilityof cross-project defect prediction. Automated Software Engineering 19(2), 167–
199 (2012)
29. Herbold, S.: Training data selection for cross-project defect prediction. In: Proceedings of the 9th International Conference on Predictive Models in Software
Engineering, p. 6. ACM (2013)30. Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data withneural networks. Science 313(5786), 504–507 (2006)
31. Hira, Z.M., Gillies, D.F.: A review of feature selection and feature extraction
methods applied on microarray data. Advances in bioinformatics 2015 (2015)32. Ho, T.K.: Random decision forests. In: Proceedings of the 3rd International
Conference on Document Analysis and Recognition, vol. 1, pp. 278–282. IEEE
(1995)
33. Jureczko, M., Madeyski, L.: Towards identifying software project clusters withregard to defect prediction. In: Proceedings of the 6th International Conference
on Predictive Models in Software Engineering, p. 9. ACM (2010)
34. Kamei, Y., Fukushima, T., McIntosh, S., Yamashita, K., Ubayashi, N., Hassan,
A.E.: Studying just-in-time defect prediction using cross-project models. Empirical Software Engineering 21(5), 2072–2106 (2016)
35. Kaufman, L., Rousseeuw, P.J.: Finding groups in data: an introduction to cluster
analysis, vol. 344. John Wiley & Sons (2009)
36. Kim, S., Zimmermann, T., Whitehead Jr, E.J., Zeller, A.: Predicting faults fromcached history. In: Proceedings of the 29th International Conference on Software
Engineering (ICSE), pp. 489–498. IEEE Computer Society (2007)
37. Kohonen, T.: The self-organizing map. Proceedings of the IEEE 78(9), 1464–
1480 (1990)
38. Kuhn, M.: Caret: classification and regression training. Astrophysics Source
Code Library (2015)
39. Landwehr, N., Hall, M., Frank, E.: Logistic model trees. Machine Learning
59(1), 161–205 (2005)
40. van der Maaten, L.: Accelerating t-SNE using tree-based algorithms. Journal of
Machine Learning Research 15(1), 3221–3245 (2014)41. van der Maaten, L., Hinton, G.: Visualizing data using t-SNE. Journal of Machine Learning Research 9(Nov), 2579–2605 (2008)
42. Martinetz, T., Schulten, K., et al.: A “neural-gas” network learns topologies. Artificial Neural Networks pp. 397–402 (1991)
43. McCabe, T.J.: A complexity measure. IEEE Transactions on Software Engineering (TSE) (4), 308–320 (1976)
44. McDonald, J.H.: Handbook of Biological Statistics (3rd ed.). Sparky House
Publishing, Baltimore, Maryland. (2014)
45. Menzies, T., Greenwald, J., Frank, A.: Data mining static code attributes to learn
defect predictors. IEEE Transactions on Software Engineering (TSE) 33(1)(2007)
46. Menzies, T., Owen, D., Richardson, J.: The strangest thing about software. Computer 40(1) (2007)

37

38

Kondo et al.

47. Moser, R., Pedrycz, W., Succi, G.: A comparative analysis of the efficiency ofchange metrics and static code attributes for defect prediction. In: Proceedingsof the 30th International Conference on Software Engineering (ICSE), pp. 181–
190. IEEE (2008)
48. Muthukumaran, K., Rallapalli, A., Murthy, N.: Impact of feature selection techniques on bug prediction models. In: Proceedings of the 8th India Software
Engineering Conference, pp. 120–129. ACM (2015)49. Nagappan, N., Ball, T., Zeller, A.: Mining metrics to predict component failures.
In: Proceedings of the 28th International Conference on Software Engineering(ICSE), pp. 452–461. ACM (2006)
50. Nam, J.: Survey on software defect prediction. HKUST PhD Qualifying Examination, Department of Compter Science and Engineerning, The Hong Kong
University of Science and Technology, Tech. Rep (2014)
51. Nam, J., Fu, W., Kim, S., Menzies, T., Tan, L.: Heterogeneous defect prediction.
IEEE Transactions on Software Engineering (2017)
52. Nam, J., Kim, S.: CLAMI: Defect prediction on unlabeled datasets. In: Proceedings of the 30th International Conference on Automated Software Engineering(ASE), pp. 452–463. IEEE (2015)
53. Nam, J., Kim, S.: Heterogeneous defect prediction. In: Proceedings of the 10th
Joint Meeting on Foundations of Software Engineering (FSE), pp. 508–519.
ACM (2015)
54. Nam, J., Pan, S.J., Kim, S.: Transfer defect learning. In: Proceedings of the 2013
International Conference on Software Engineering (ICSE), pp. 382–391. IEEE
Press (2013)
55. NASA: Metrics data program. http://openscience.us/repo/defect/
mccabehalsted/. [Online; accessed 1-September-2016]
56. Neumann, D.E.: An enhanced neural network technique for software risk analysis. IEEE Transactions on Software Engineering (TSE) 28(9), 904–912 (2002)
57. Pan, S.J., Tsang, I.W., Kwok, J.T., Yang, Q.: Domain adaptation via transfer
component analysis. IEEE Transactions on Neural Networks 22(2), 199–210(2011)
58. Peters, F., Menzies, T., Gong, L., Zhang, H.: Balancing privacy and utility incross-company defect prediction. IEEE Transactions on Software Engineering
39(8), 1054–1068 (2013)
59. Petrić, J., Bowes, D., Hall, T., Christianson, B., Baddoo, N.: The jinx on theNASA software defect data sets. In: Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering, pp. 1–5. ACM(2016)
60. Quinlan, R.: C4.5: Programs for Machine Learning. Morgan Kaufmann Publishers (1993)
61. Rathore, S.S., Gupta, A.: A comparative study of feature-ranking and featuresubset selection techniques for improved fault prediction. In: Proceedings of the
7th India Software Engineering Conference, p. 7. ACM (2014)
62. Ren, J., Qin, K., Ma, Y., Luo, G.: On software defect prediction using machinelearning. Journal of Applied Mathematics 2014 (2014)38

The Impact of Feature Reduction Techniques on Defect Prediction Models

39

63. Rodrı́guez, D., Ruiz, R., Cuadrado-Gallego, J., Aguilar-Ruiz, J.: Detecting fault
modules applying feature selection to classifiers. In: Proceedings of the 2007
International Conference on Information Reuse and Integration, pp. 667–672.
IEEE (2007)
64. Rodriguez, D., Ruiz, R., Cuadrado-Gallego, J., Aguilar-Ruiz, J., Garre, M.: Attribute selection in software engineering datasets for detecting fault modules. In:
Proceedings of the 2007 EUROMICRO Conference on Software Engineeringand Advanced Applications, pp. 418–423. IEEE (2007)
65. Rokach, L., Maimon, O.: Clustering methods. In: Data mining and knowledge
discovery handbook, pp. 321–352. Springer (2005)
66. Shepperd, M., Song, Q., Sun, Z., Mair, C.: Data quality: Some comments on theNASA software defect datasets. IEEE Transactions on Software Engineering(TSE) 39(9), 1208–1215 (2013)
67. Shihab, E.: Practical software quality prediction. In: Proceedings of the 2014
International Conference on Software Maintenance and Evolution (ICSME), pp.
639–644. IEEE (2014)
68. Shihab, E., Jiang, Z.M., Ibrahim, W.M., Adams, B., Hassan, A.E.: Understanding
the impact of code and process metrics on post-release defects: A case study on
the Eclipse project. In: Proceedings of the International Symposium on Empirical
Software Engineering and Measurement (ESEM), pp. 4:1–4:10. ACM (2010)
69. Shivaji, S., Whitehead, E.J., Akella, R., Kim, S.: Reducing features to improvecode change-based bug prediction. IEEE Transactions on Software Engineering(TSE) 39(4), 552–569 (2013)
70. Smolensky, P.: Information processing in dynamical systems: Foundations ofharmony theory. Tech. rep., DTIC Document (1986)
71. Tantithamthavorn, C., Hassan, A.E.: An experience report on defect modellingin practice: Pitfalls and challenges. In: Proceedings of the 40th International
Conference on Software Engineering: Software Engineering in Practice Track(ICSE-SEIP), pp. 286–295. ACM (2018)
72. Tantithamthavorn, C., McIntosh, S., Hassan, A.E., Matsumoto, K.: Automatedparameter optimization of classification techniques for defect prediction models.
In: Proceedings of the 38th International Conference on Software Engineering(ICSE), pp. 321–332. ACM (2016)
73. Tantithamthavorn, C., McIntosh, S., Hassan, A.E., Matsumoto, K.: An empiricalcomparison of model validation techniques for defect prediction models. IEEE
Transactions on Software Engineering (TSE) 43(1), 1–18 (2017)
74. Tassey, G.: The economic impacts of inadequate infrastructure for software testing. National Institute of Standards and Technology, RTI Project 7007(011)(2002)
75. Von Luxburg, U.: A tutorial on spectral clustering. Statistics and computing
17(4), 395–416 (2007)
76. Xu, Z., Liu, J., Yang, Z., An, G., Jia, X.: The impact of feature selection on
defect prediction performance: An empirical comparison. In: Proceedings of the27th International Symposium on Software Reliability Engineering (ISSRE), pp.
309–320. IEEE (2016)

39

40

Kondo et al.

77. Yang, B., Yin, Q., Xu, S., Guo, P.: Software quality prediction using affinity
propagation algorithm. In: Proceedings of the International Joint Conference on
Neural Networks, pp. 1891–1896. IEEE (2008)
78. Zhang, F., Zheng, Q., Zou, Y., Hassan, A.E.: Cross-project defect prediction using a connectivity-based unsupervised classifier. In: Proceedings of the 38th
International Conference on Software Engineering (ICSE), pp. 309–320. ACM(2016)
79. Zhang, H.: The optimality of Naive Bayes. In: FLAIRS Conference. AAAI Press(2004)
80. Zhong, S., Khoshgoftaar, T.M., Seliya, N.: Unsupervised learning for expertbased software quality estimation. In: HASE, pp. 149–155. Citeseer (2004)
81. Zimmermann, T., Nagappan, N., Gall, H., Giger, E., Murphy, B.: Cross-project
defect prediction: a large scale experiment on data vs. domain vs. process. In:
Proceedings of the the 7th Joint Meeting of the European Software Engineering
Conference and the ACM SIGSOFT Symposium on The Foundations of Software Engineering (ESEC-FSE), pp. 91–100. ACM (2009)
82. Zwillinger, D., Kokoska, S.: CRC standard probability and statistics tables andformulae. Crc Press (1999)40