Leveraging Fault Localisation to Enhance DefectPrediction
Jeongju Sohn

Yasutaka Kamei

Shane McIntosh

Shin Yoo

KAIST
Daejeon, Korea
kasio555@kaist.ac.kr

Kyushu University
Kyushu, Japankamei@ait.kyushu-u.ac.jp

University of Waterloo
Waterloo, Canada
shane.mcintosh@uwaterloo.ca

KAIST
Daejeon, Korea
shin.yoo@kaist.ac.kr

Abstract—Software Quality Assurance (SQA) is a resource
constrained activity. Research has explored various means of supporting that activity. For example, to aid in resource investment
decisions, defect prediction identifies modules or changes that arelikely to be defective in the future. To support repair activities,
fault localisation identifies areas of code that are likely to requirechange to address known defects. Although the identificationand localisation of defects are interdependent tasks, the synergy
between defect prediction and fault localisation remains largely
underexplored.
We hypothesise that modifying code that was suspicious in
the past is riskier than modifying code that was not. To validate
our hypothesis, in this paper, we employ fault localisation, which
localises the root cause of a program failure. We compute the pastsuspiciousness score of code changes to each fault, and use thosescores to (1) define new features for training defect prediction
models; and (2) guide the next actions of developers for a commit
labelled as fix-inducing. An empirical study of three open-source
projects confirms our hypothesis. The new suspiciousness features
improve F1 score and balanced accuracy of Just-In-Time (JIT)
defect prediction models by 4.2% to 92.2% and by 1.2% to
3.7%, respectively. When guiding developer actions, past codesuspiciousness successfully guides developers to a defective file,inspecting two to nine fewer files on average, compared to thebaselines based on previous findings on past faults. These results
demonstrate the potential of synergies of fault localisation anddefect prediction, and lay the groundwork for explorations ofthat combined space.
Index Terms—defect prediction, fault localisation, search-based
software engineering, software quality assurance

I. I NTRODUCTION
Since exhaustive testing is impractical (if not impossible),software organizations must take pragmatic and cost-effective
approaches to Software Quality Assurance (SQA). To focus
SQA detection activities, defect prediction [1], [2] has been
proposed to prioritize modules by their likelihood of being defective. Modern so-called “Just-In-Time” (JIT) defect prediction models identify risky code changes, i.e., code changes in a
fix-inducing commit, rather than defective modules. Targetingrisky changes fits nicely within a change-based development
model, raising warnings about changes while the context ofthe changes is still fresh in the mind of developers [3].
To support SQA repair activities, fault localisation [4] hasbeen proposed to prioritise areas within the codebase thatare likely to require modification to address a defect. Fault
localisation ranks program elements by their likelihood of

being implicated in the fix for a given program failure (suspiciousness). Information Retrieval-based Fault Localisation(IRFL), one of the most actively studied approaches to fault
localisation, exploits the lexical similarity between code andbug reports to localise faulty code [5]–[7]. Unlike the mostdynamic FL techniques that require test execution, IRFL canbe applied as soon as bug reports are available.
Despite being interdependent tasks, the literature on (JIT)
defect prediction and fault localisation have largely developed
independently of one another. Indeed, both fault localisation and defect prediction strive to support quality assurance
activities with different timing—defect prediction and fault
localisation occur before and after program failures have been
identified, respectively. Exploiting synergies between the two
distinct areas may yield more accurate and actionable insights.
Early work by Sohn and Yoo [8], [9] shows that featuresthat have typically been used in defect prediction (i.e., codeand change features) can improve the performance of fault
localisation. In this paper, we set out to explore the inverse.
More specifically, we set out to address the following question:
Can insights from fault localisation be leveraged
to enhance JIT defect prediction?
To address this central question, we quantify the suspiciousness of past code using IRFL. Computing the past suspiciousness of code using fault localisation is a natural choice because
it shares a similar goal with JIT defect prediction, i.e., identifying risky code. We conjecture that code that was suspicious in
the past will remain suspicious for some time and warrants
additional scrutiny when being modified. We first evaluate
whether using past suspiciousness measures of code improve
the performance of JIT defect prediction models. Then, weinvestigate whether these past suspiciousness measures aid in
guiding the repair activities of developers for risky commits.
To evaluate our approach of using past suspiciousness of codein defect prediction, we perform an empirical study of threeopen-source projects. In addition to that empirical study, thispaper makes the following contributions:• We define new defect prediction features based on theprior suspiciousness of code according to fault localisation. Using these features, we observe that the F1 score of
JIT models improves by 4.2% to 92.2%. We also observea small increase in Balanced Accuracy of 1.2% to 3.7%.

•

We use the prior suspiciousness of code to guide the next
action of developers after a commit is predicted to berisky. The experimental results show that by inspecting
files modified in a commit in descending order of theirsuspiciousness, developers can examine the first defective
file, on average, two to nine files faster.

The remainder of this paper is organised as follows. Section II introduces our hypothesis on leveraging the past suspiciousness of code changes in a commit and the two approaches
that we took to verify this hypothesis. Section III presents
three research questions and describes the experimental setup
for the empirical study. Section IV discusses the results andSection V presents the threats to validity. Section VI describes
the related work, and lastly, Section VII concludes our paper.
II. FL2DP: FAULT L OCALISATION TO D EFECTP REDICTION
Defect Prediction (DP) aims to predict whether a givenchange is fix-inducing or not, before actually executing any
test. Fault localisation, on the other hand, aims to predict thelocation of a fault whose existence has been already confirmed
by a failing test execution. Although both techniques have been
widely studied, the interplay between two techniques has notbeen thoroughly investigated. Existing work has investigated
how features frequently used for defect prediction can improveaccuracy of fault localisation [8], [9], but the opposite directionof using fault localisation for better defect prediction has notbeen explored. This section motivates the way we make useof previous faults via fault localisation for better DP.
A. The Impact of Past Faults in Defect Prediction
Kim et al. showed the localities of faults: faults appear
together instead of individually [10]. The reasoning is thatdevelopers make fix-inducing changes due to the incorrectunderstanding of the changes, and therefore are likely tointroduce more than one fault in the area of misunderstanding,either directly or indirectly (e.g., via the propagation ofchanges). The authors introduced various types of localities:
temporal, spatial, and changed or new-entity. Despite theirdifferences, they all assume that the area that has been related
to faults, either was defective or changed along, is riskier.
Rahman et al. followed this up by comparing these localities toeach other and reported that temporal locality, which presumes
that recently fixed code is more defect-prone, performs the bestfor defect prediction [11].
Another way that previously defective elements become
more defect prone in the future is if fixing previous faults
actually resulted in new ones. Mockus and Weiss reported
that changes that fix defects are riskier than the other types ofchanges, such as new features [12]. Shihab et al. performed anindustrial case study on the risk of software changes [13]: theirstudy showed that the more bug-fixing changes were made ona file, the riskier the file is. Combined, these results suggestthat areas of source code that contained previous faults tend
to be more defect prone in the future.

We propose fault localisation techniques as a unified framework that can capture various aspects of previous faults studied
in the context of defect prediction. Instead of looking at binary
property of whether a file has been defective in the past or
not, or whether a file has been a target of bug fix or not, wemeasure how suspicious the changed file would be with respect
to previous faults, using a specific fault localisation technique.
This way, we capture degrees of suspiciousness instead of thebinary property of being faulty or not.
In the software life-cycle, there are two processes that aim
to locate the code related to program failure: Fault Localisation(FL) and Defect Prediction (DP). Both FL and DP are designed
to assist developers with testing and debugging. Their tasks
can be defined as follows:• Fault Localisation (FL): locate the source code thatcaused the observed program failure• Defect Prediction (DP): predict whether the changed
source code will result in a program failure
If DP is an a priori detection of faults, FL is an a posteriori
detection of faults, i.e., knowing where the fault is after thefact. In the software development timeline, these two tasks
will interleave with each other. Consequently, if there aretemporal and spatial locality in the occurrence of faults, weposit that FL results for previous faults can help with DP
for the current change. More specifically, we formulate thefollowing hypothesise:
Hypothesis: If a commit modifies code that was suspicious
in the past (FL), the commit is more likely to introduce
faults than those that were not (DP).
To validate this hypothesis, we define new features basedon the results of fault localisation on previous faults and investigate whether these features can improve the performanceof defect prediction. We also study how the suspiciousness ofcode computed from past fault localisation can make the DP
results more actionable for developers. The remainder of thissection introduces the new features and describes how we usethem to enhance the actionability of DP results.
B. IR based Fault Localisation as Defect Prediction Feature
Table I shows the input features used for our defect prediction model. The first 13 features are features that havebeen widely studied for defect prediction and are taken fromexisting work [3]. We have excluded three review features
studied by McIntosh and Kamei, as our choice of benchmark(see Section III-B) did not adopt an explicit code review
process. The remaining three features are based on IRFL.
While a diverse range of fault localisation techniques havebeen studied [5], [7]–[9], [14]–[17], we focus on IRFL becauseof two reasons: its non-dynamic nature, and its flexibility. The
intuition behind IRFL is that a bug report, and the part of thesource code that is the root cause of the reported bug, are likely
to share a strong lexical similarity with each other. Based on
this intuition, IRFL adopts various textual similarity metrics
developed in the field of Information Retrieval (IR) to rank

program elements according to their similarity to the given
bug report: the given bug report becomes the query, which
is used to search the documents that are program elements.
Consequently, it does not require any dynamic analysis, apartfrom the occurrence of the bug that resulted in the original bug
report. This makes IRFL an ideal candidate for a DP feature,since DP is typically performed before the test execution.
By flexibility of IRFL, we refer to the fact that textual
similarity can be computed between any bug report and any
program element, regardless of the timeline. Our hypothesisis that modifying code that was suspicious for historical faults
poses higher risk. However, if we were to compute the exact
suspiciousness of each program element at the time of eachhistorical fault, it would result in non-trivial analysis cost even
if the FL technique of choice is a non-dynamic one such as
IRFL. To avoid this, we approximate the past suspiciousness
by taking the textual similarity between historical bug reports
and the current program elements that have been modified andsubmitted for defect prediction. We posit that there will be
sufficient continuity in the set of lexical tokens that appear in
the same program element such as a file or a specific function.
This lexical continuity, coupled with the temporal and spatial
locality of fault [10], allows us to avoid the inherently noisy
and costly process of tracing change history from current
changes to the locations of previous faults.
TABLE I
16 FEATURES USED TO TRAIN DEFECT PREDICTION MODELS
Property

Description

Size

Lines Added
Lines Deleted

The number of added lines in a commit
The number of deleted lines in a commit

Diffusion

Subsystem
Directory
File
Entropy

The number of modified subsystems in a commit
The number of modified directory in a commit
The number of modified files in a commit
The spread of modified lines across files
in a commit

Changes

The number of changes made to the modified
files in past
The number of developers who have changed
the modified files in a commit in the past
The time interval to the last changes on themodified files

History
Developers
Age
Prior changes
Experience

Recent changes
Subsystemchanges
Awareness
sim2rsum

Bug report
sim2rmax
sim2rmean

The number of prior changes to the modified
files the authors participated
The number of prior changes to the modified
files the authors participated weighted by thetime interval between changes
The number of prior changes to the modified
directories the authors participated
The fraction of prior changes to the modified
directories that the authors participated
The sum of similarities between code changes
in a commit and recent bug reports
The maximum of similarities between codechanges in a commit and recent bug reports
The arithmetic mean of similarities between codechanges in a commit and recent bug reports

1) Vector Space Model for IRFL: We use IRFL techniquebased on Vector Space Model (VSM) [18] to compute thesimilarity between code changes made in a commit andprevious bug reports [4]. After the standard text preprocess-ing that includes text normalisation1 , stopword removal, andword stemming, both the query (i.e., the bug report) andthe documents (i.e., program elements, such as source files)
are represented as fixed-length weight vectors, where eachweight corresponds to a unique word (term). The typicalmeans of obtaining weights for each term is to compute term
frequency-inverse document frequency (tf-idf ). Given a termt, a document d, and the set of all documents, D, let ft,d be
the number of times t appears in d. tf-idf is defined as follows:tf (t, d) = log (1 + ft,d )idf (t, D) = log (|D|)|{d ∈ D|t ∈ d}|tf − idf (t, d, D) = tf (t, d) · idf (t, D)

(1)
(2)
(3)

Intuitively, tf (t, d) captures the degree to which a term t is
coupled with a document d (i.e., the frequency of appearanceof t in d), whereas idf (t, D) captures the degree of how
narrow the coupling between t and documents is (i.e., higher
idf (t, D) means most of documents in D contain t, whereas
lower idf (t, D) means only a few documents in D contain t).
A tf-idf weight is simply the product of TF and IDF of a given
term; a document d, or a query q, is represented as a fixedlength vector of tf-idf weights, each of which is the weight
for a term in the corpus D. Finally, the similarity between adocument and a query can be computed as the cosine similarity
between two tf-idf vectors.
2) Feature Engineering for IRFL Scores: For IRFL, wehave multiple documents (i.e., source code files) to matchagainst a single query (i.e., a bug report). However, for its use
in the context of Just-In-Time DP, we would like to measure
how similar the single commit under consideration is to arange of potentially related previous faults. This results in asingle document (i.e., the commit) and multiple queries (i.e.,
the previous faults that are potentially relevant). Both factors
affect how we convert an IRFL score (i.e., similarity) to a
defect prediction feature.
The single document situation means that the idf ’s only
information is whether a term is in documents or not, which
can be covered by tf . As a result, we fix the idf to 1, using
only the tf for actual computation of the similarity. Given
that each commit should ideally contain only a single change,
we believe that using term frequency to represent its topic is
an acceptable choice. Even if the commit under consideration
addresses multiple issues, we expect the range of topics to bemuch narrower and more focused compared to all possible
semantic topics in the entire system. Note that the single
document issue only applies when we try to convert the IRFL
score into a feature for defect prediction of a single commit
under consideration. If the commit is predicted to be defective,
we would like to localise the cause of concern for actionability,
for which we consider different files in the single commit as
separate documents. See Section II-C for more details.
The issue with multiple queries is that we cannot ascertain
which previous fault is the most relevant to the commit under
1 We normalise camelCase identifiers into its constituent subtokens.

consideration. To resolve this, we focus our analysis on the
bug reports that have been created within the predefined time
window (based on the temporal locality assumption). Since
there are often multiple bug reports within the time window,
multiple similarity values are calculated for the given commit.
We aggregate these similarity values using three different
methods and define one feature for each: arithmetic mean
(sim2rmean ), sum (sim2rsum ), and maximum (sim2rmax ).
Here, c denotes the commit under consideration, and r a
bug report submitted to the same project. By defining a time
window, we first identify the set of bug reports (R(c, w)) thathave been submitted during the time window as follows:

suspiciousness of each file modified by the commit c, in thecontext of actionability. We hereafter refer to susp(f, c, w, R)
as suspf for the sake of brevity.

R(c, w) = {r|timestamp(c) − timestamp(r) ≤ w}

RQ1. Feature Analysis: Do sim2rM features capture distinct
and useful features of fix-inducing commits? Before using
sim2rM features introduced in Section II-B2, we investigate
whether these features can capture distinct and useful properties of fix-inducing commits.
Firstly, to estimate the degree to which our features are
distinct, we perform Spearman correlation analysis on the
16 features in Table I. If sim2rM features capture new
dimensions that have not yet been explored, they will havea weak correlation to existing JIT DP features. Secondly, toshow that our features can be useful in defect prediction, we
perform Mann-Whitney U-tests on the sim2rM features of
fix-inducing commits and non-inducing commits (α = 0.05).
Since sim2rM features assume that defective changes are
more similar to past faults than non-defective changes are, we
expect fix-inducing commits to have larger values of sim2rM
features. We also compare the feature importance values in
the learnt defect prediction models between the sim2rM
features and the other 13 features to assess how useful the
sim2rM features are compared to the other JIT-DP features.

(4)Now, let cl(c) be the set of lines modified by c, and M be
one of the aggregation methods, {arithmetic mean, sum, max}.
The similarity based DP feature for the given commit c under
M is defined as follows:
sim2rM (cl(c), w, R) = M ({sim(cl(c), r)|r ∈ R(c, w)})

(5)

That is, sim2rM (cl(c), w, R) is either arithmetic mean,sum, or maximum of similarities between the changed lines in
the given commit and bug reports submitted within the given
time window from c. We use cosine similarity for sim.
C. Actionability
A defect prediction result is actionable if it can guidethe developer on how to handle the potentially fix-inducing
commit. In an ideal case, all changes in a commit will betightly coupled to each other, resulting in all files modified by a
fix-inducing commit being responsible for the induced defect.
In practice, however, this is not always the case: a fix-inducing
commit often contains changes unrelated to the induced fault.
As a result, without knowing which file is defective, developers
may waste their effort inspecting non-defective files. Given thisconsideration, we propose to use the similarity to previous bug
reports to guide the developer.
Our approach presumes the temporal locality of faults:
similar faults may appear again within a short time frame. We
exploit this for actionability by aiming to localise the fault
among the files modified by the given commit. Since there isno information about what exactly the induced fault will be at
the time of defect prediction, we use the previous bug reports
as an approximation, using the temporal locality assumption:
we run IRFL on the files changed by the given commit against
previous bug reports, i.e., R(c, w). The suspiciousness of a filef modified by a fix-inducing commit c is computed as follows:susp(f, c, w, R) =X

sim(cl(c, f ), r)(6)r∈R(c,w)

Here, cl(c, f ) denotes the set of lines in file f that havebeen changed by the commit c. The final suspiciousness score(susp) of a file f is the sum of the similarities between thechanges of the file and previous bug reports within the time
window w. Note that sim2rM is used to denote the defectprediction feature, which represents the commit under consideration, whereas susp(f, c, w, R) denotes the individual IRFL

III. E XPERIMENTAL S ETUP
In this section, we present our research questions and thecorresponding experimental setup for our empirical study.
A. Research Questions
We formulate three research questions to evaluate theutility of the past suspiciousness of code changes in a commit
when predicting its defect-proneness.

RQ2. Effectiveness: Can the use of sim2rM features improve
the performance of defect prediction models? To answer RQ2,
we compare the performance of defect prediction models
trained with and without sim2rM features. Before comparing
them, however, we first investigate the relationship between
the effectiveness of sim2rM features and the duration of
the time window. sim2rM features aggregate the similarities
between code changes and bug reports that are created withina fixed time window. Thus, the effectiveness of sim2rM
features can vary depending on the choice of the time window.
We use four different values for the time window, 30, 60, 90,
and 120 days, and inspect how the value of the time window
affects the benefit of using sim2rM features. We then go
back to the initial question and compare the performance ofmodels with and without sim2rM features, using the best
time window among the four. We run Mann-Whitney U-teston the pairs of the DP models with and without sim2rM
features and measure Cliff’s delta effect size for these pairs.
We deem the improvement from using sim2rM features isstatistically significant if the p-value of the U-test is ≤ 0.05,
and the effect size is at least medium (δ > 0.33) [19].

RQ3. Actionability: Can suspf guide the action of developers for fix-inducing commits? To validate the actionability of
past suspiciousness of code (suspf ), we compare the effort
spent before inspecting the first defective file when examining
files modified in a commit in descending order of their suspf
scores with the effort when examining the files randomly. If
suspf scores computed per file are truly actionable, they willallow developers to discover a defective file with less effort.
In addition to the random inspection, we use closed bug
count (cntCB ) as another baseline of suspf . Closed bug countof a file is the number of closed bugs that modified the file.
Here, closed bug denotes the bug that has been closed before
the commit. Previous studies used the closed bug count to sortfiles in order of their defect-proneness [11], [20]. Results ofthese studies show that simply ordering files in descendingorder of their closed bug counts could be as effective as moresophisticated defect prediction techniques.
Compared to previous work, which utilises only the groundtruth of defective files, we use past suspiciousness of files
that were non-defective in the past along with past suspiciousness of files that were defective. To verify that the
improvement gained from using suspf does not solely comefrom computing the suspiciousness scores of defective files,
but rather from taking both defective and non-defective files
into consideration, we select a variant of cntCB called simCB
as the last baseline. simCB stands for the similarity to pastclosed bugs; it adds up the suspiciousness scores of a file toeach past fault for which it was responsible instead of counting
them. Consequently, if the use of suspf also outperforms theuse of simCB , we can conclude that including files that were
non-defective in the past, in addition to past defective files
does contribute positively.
Unlike cntCB and simCB , suspf does not require bug
reports to be closed for computation, as it uses suspiciousness
scores of code. Nevertheless, we take the closed bugs as
inputs for suspf by default. To fully exploit suspf , weinvestigate how the effectiveness of suspf changes as more
reports become available for inspection. For this, we conduct
another experiment that compares the effectiveness of suspf
with different sets of reports: a set of both open and closed
bug reports, and a set of reports of all types.
TABLE IIT HREE OPEN - SOURCE BENCHMARKS : T HE TABLE ON THE LEFTDESCRIBES THE COLLECTED COMMITS , AND THE TABLE ON THE RIGHTSHOWS THE AVERAGE NUMBER OF BUG REPORTS COLLECTED USING
FOUR DIFFERENT TIME WINDOWS .

Project# ofcommits# of noninducing
commits# of fixinducing
commits

Collection
PeriodLang
MathClosure

3,929
4,810
1,948

3,697
4,325
1,848

232
485
100

2002/07 – 2019/10
2003/05 – 2019/10
2009/11 – 2013/12

Avg. # of bug reports
in diffrent time window (days)
30
60
90
120
3.15
4.44
7.46

5.50
8.12
14.60

7.59
11.78
21.64

9.74
15.43
28.51

v.1.5.0, which contains six open-source projects; these projects
manage their issue reports using different issue trackers.
Among the six projects, we select three projects, CommonsLang, Commons-Math, and Closure Compiler, for the evaluation. The main criterion of this selection is how structured anissue report is. If issue reports are written freely, there will bemany variables to consider when parsing them, e.g., whether toinclude comments. Thus, we prefer projects that use an issue
tracker with a strict format. Lang and Math adopt Jira [22], an
issue tracker that stores issues in a highly structured format
and is designed for use by agile development teams. Issues in
Closure that are covered by Defects4J are stored in Google
Code Archive. As issue reports managed by Git issue tracker
are less well structured, we exclude the projects that use theGit issue tracker, leaving only Lang, Math, and Closure.
We use the SZZ algorithm to identify fix-inducing commits [23]. Instead of implementing our own, we use the opensource implementation of SZZ algorithm [24]. To filter out
false-positives, we adopt six of the eight filters applied by
McIntoch and Kamei [3]. We omit the two filters that exclude
the commits that either fix or induce too many faults because
they were too aggressive, leading to several false-negatives,
when they were applied to our dataset. Table II shows the
final number of commits for each category, either fix or noninducing, and the associated commit collection period.
C. Training Defect Prediction Models
For training defect prediction models, we use Random
Forest (RF), which showed good-enough performance inprevious studies of defect prediction [25]–[30]. We employ
scikit-learn [31] for Random Forest and set the numberof trees as 200. For other parameters of RF, we use thedefault values provided in scikit-learn. We have decided to use
the default parameter setting, as our initial grid-search based
hyperparameter tuning has failed to improve performance
consistently and significantly across performance metrics andstudied subjects compared to the default settings. We would
recommend careful parameter tuning in case the proposed
technique is applied to a real project with stable history.
For training and test data sets, we sort the collected commits
in their chronological order, i.e., commit time, and sequentiallydivide them into five folds. We train defect prediction models
for each fold using all commits prior to the target fold. Since
the first fold, i.e., fold zero, does not have any preceding fold totrain defect prediction models, we exclude fold zero from theevaluation. Figure 1 describes how the fix-inducing commit
rate changes over time. As software becomes more mature,
the number of fix-inducing commits decreases dramatically.
For each fold, we repeat the training 30 times due to therandomness of Random Forest.
D. Evaluation Metrics

B. Subjects
We evaluate our approach of leveraging the past suspiciousness of code using Defects4J [21], a real-world fault dataset
frequently studied in fault localisation. We use Defects4J

To answer RQ2, we compare the performance of defectprediction models trained with and without sim2rM featuresand examine whether using sim2rM features improves theperformance of defect prediction models. We use five metrics

52

38

27

5
0

1

2

3

Fold

4

Math

159

0.16
0.14
0.12
0.10

76

0.08

70

0.06

37

0.04
0

1

2

3

Fold

0.125
0.100
0.075

21

0.050

9

7

0.025
0.000

4

Closure

61

0.150

143

Fix-inducing Rate

Lang

110

Fix-inducing RateFix-inducing Rate

0.14
0.12
0.10
0.08
0.06
0.04
0.02
0.00

0

1

2

2

3

Fold

4

Fig. 1. Fix-inducing rate per fold: the actual number of fix-inducing commits is written next to each point. For all three project, the fix-inducing rate decreases
gradually as the projects become more mature.

that are frequently adopted to evaluate the performance of
defect prediction models: ROC-AUC, Accuracy, Balanced
Accuracy, F1 score, and PR-AUC.
ROC-AUC measures the capability of a model distinguishing two classes, in our case, fix-inducing and non-inducing
commits. Accuracy is the overall rate of correctly classified
instances, while Balanced Accuracy is the average rate ofcorrectly classified instances of each class. F1 score is the
harmonic mean of Precision and Recall, and PR-AUC is thearea under the curve of Precision and Recall; it measures thetrade-off between these two as a threshold changes.
RQ3 concerns whether suspf can direct developers to
defective files. To answer this, we compare the effort ofdevelopers spent before meeting the first defective file. We
assume that the effort is proportional to the number of nondefective files examined before the first defective one anduse the number of those non-defective files as the effort. As
mentioned in Section III-A, we compare suspf with threebaselines: the random, the closed bug count (cntCB ), and theclosed bug similarity (simCB ). Among these baselines, therandom approach is inherently stochastic and requires to berepeated to generalise its outcome. Instead of repeating therandom approach, we compute the expected rank of the first
defective file and use it as an alternative. Assuming that thelikelihood of each file to be inspected is uniform, we define
the expected rank of the first defective file as below:
PM −N +1
exp(f, c) =rank=1Mrank, M = |Fc |, N = |GTc |(7)

As Equation (7) shows, the expected rank of the first
defective file is the arithmetic mean of all possible ranks ofdefective files. Here, M is the total number of files modified
in a commit c, and N is the number of those files whosechanges induce the fault. We also use this expected rank as
an alternative for the cases where all the modified files havethe same ranking by the other approaches.
In addition to the number of inspected files, we use acc@nmetric, which is often adopted for the evaluation of fault localisation, for the effort evaluation. acc@n counts the numberof fix-inducing commits whose ranking contains at least one
defective file within the top n. For example, if n = 5 andacc@n = 10, it means that at least one defective file has beensuccessfully located within the top five for ten fix-inducing
commits. For the random approach, we compute acc@n by
counting the number of fix-inducing commits in which the

expected rank is smaller or equal to n.
TABLE IIIM ANN -W HITNEY U- TEST ON sim2rM FEATURES BETWEENFIX - INDUCING AND NON - INDUCING COMMITS WITH A STATISTICALSIGNIFICANCE OF 0.05. P - VALUES SMALLER OR EQUAL TO 0.05 ARE IN
BOLD TEXT. F STANDS FOR FOLD .
Proj

Lang

Math

Closuresim2rsum
60
90

120

30

0.12
0.55
0.49
0.18
0.39
0.07

0.00
0.22
0.21
0.07
0.13
0.00

0.01
0.16
0.09
0.04
0.06
0.00

0.02
0.29
0.14
0.01
0.06
0.00

0
1
2
3
4
all

1.00
0.15
0.00
0.47
0.02
0.80

1.00
0.00
0.00
0.01
0.09
0.14

0.97
0.01
0.00
0.00
0.00
0.11

0
1
2
3
4
all

0.00
0.48
0.22
0.01
0.73
0.10

0.00
0.52
0.23
0.01
0.45
0.19

0.00
0.42
0.11
0.00
0.18
0.32

F

30

0
1
2
3
4
all

sim2rmean
60
90

120

30

sim2rmax
60
90

0.00
0.41
0.01
0.01
0.73
0.03

0.00
0.14
0.00
0.00
0.54
0.00

0.00
0.08
0.00
0.00
0.50
0.00

0.00
0.10
0.00
0.00
0.50
0.00

120

0.15
0.75
0.23
0.05
0.79
0.28

0.24
0.75
0.13
0.01
0.61
0.25

0.46
0.63
0.13
0.01
0.63
0.29

0.56
0.74
0.20
0.01
0.74
0.45

0.95
0.01
0.00
0.01
0.00
0.10

0.86
0.00
0.00
0.24
0.00
0.00

0.00
0.00
0.00
0.02
0.00
0.00

0.00
0.00
0.00
0.00
0.00
0.00

0.00
0.00
0.00
0.00
0.00
0.00

1.00
0.53
0.21
0.95
0.00
0.96

0.71
0.41
0.53
0.97
0.07
0.84

0.43
0.67
0.92
1.00
0.02
0.94

0.29
0.70
0.74
1.00
0.03
0.90

0.00
0.46
0.07
0.00
0.19
0.52

0.00
0.36
0.18
0.01
0.33
0.00

0.00
0.42
0.18
0.05
0.35
0.00

0.00
0.32
0.10
0.01
0.34
0.00

0.00
0.38
0.09
0.01
0.37
0.00

0.11
0.93
0.55
0.03
0.17
0.76

0.74
0.97
0.84
0.16
0.45
1.00

0.93
0.97
0.83
0.12
0.63
1.00

0.98
0.95
0.93
0.26
0.73
1.00

TABLE IVF EATURE I MPORTANCE .E ACH CELL CONTAINS THE RANKING OF THE
sim2rM FEATURE WHEN SORTING THE 16 FEATURES IN DESCENDING
ORDER OF THEIR AVERAGE FEATURE IMPORTANCE IN THE DP MODELS .
sim2rsum
60
90
120

30

sim2rmean
60
90
120

30

sim2rmax
60
90
120

project

fold

30

Lang

all

8

4

2

2

10

3

5

6

11

8

4

3

Math

all

8

7

4

4

11

4

2

2

12

10

8

8

Closure

all

4

4

4

4

9

7

5

5

6

5

6

7

IV. R ESULT AND A NALYSISThis section presents the results of our empirical evaluation
and responds to the research questions.
A. RQ1. Feature Analysis
Figure 2 presents the heatmaps of Spearman correlation
coefficients between the 16 features used to train our defectprediction models. Overall, for all three projects, our three
new features, sim2rsum , sim2rmax , and sim2rmean , areweakly correlated to the existing ones (lighter shade), while
being strongly correlated to each other (darker shade). Fromthese, we argue that sim2rM features do contain some extra
information that is not captured by the existing features. While
the figures show only the results of using a time window of
60 days, the other three windows also share the similar trend.

1

1
0.9

1

(a) Lang

ea
n

m2
r
sim sum2
sim rmax
2r
m

Lin
es
Lin Addees
De d
Dir leted
ect
ory
File
Su
bsy
ste
En m
tro
Ch py
an
De ges
velo
per
Ag s
e
Pir
or
Re Chgs
cent
Su Chgbsy s
sAw Chgsaresi ness

sim2rsum 0.51 0.42 0.6 0.57 0.6 0.46 0.56 0.5 0.45 -0.028 -0.028 0.47 0.45 1
sim2rmax 0.45 0.39 0.59 0.56 0.59 0.46 0.59 0.55 0.48 -0 -0 0.47 0.42 0.9
sim2rmean 0.5 0.42 0.62 0.6 0.62 0.4 0.6 0.57 0.49 -0.039 -0.043 0.46 0.39 0.83

0.0

0.4

0.2

1

sim2rsum 0.49 0.38 0.52 0.5 0.52 0.34 0.44 0.38 0.35 0.08 0.081 0.43 0.41 1
sim2rmax 0.37 0.34 0.48 0.45 0.48 0.35 0.45 0.41 0.39 0.05 0.047 0.4 0.36 0.86 1
sim2rmean 0.52 0.42 0.57 0.55 0.57 0.29 0.46 0.44 0.39 -0 -0 0.43 0.38 0.79 0.83

sim2rsum 0.29 0.2 0.27 0.29 0.27 -0.21 0.37 0.34
sim2rmax -0 -0 0.076 0 0.076 -0 0.2 0.22
sim2rmean 0.42 0.29 0.34 0.39 0.34 -0.26 0.33 0.23

0.0

1

0.6

0.3

0.0

1

0

0.25 0.25 0.25 0.14

00.17 0.17 0.16 0.062 0.64

1-00.099 0.1

0.59

0.13 0.13

1

0.3

0.8

1

(b) Math

ea
n

0.2

0.6

0.9

m2
r
sim sum2
sim rmax
2r
m

0.4

0.8

Closure, time window = 60 days

Lines Added 1
Lines Deleted 0.46 1
Directory 0.5 0.39 1
File 0.63 0.58 0.71 1
Subsystem 0.5 0.39 1 0.71 1
Entropy -0.45 -0.31 -0.47 -0.54 -0.47 1
Changes 0.23 0.29 0.38 0.49 0.38 -0.29 1
Developers 0.069 0.14 0.18 0.26 0.18 -0.17 0.87 1
Age 0.067 0.05 0.16 0.2 0.16 -0.067 0 0 1
Piror Chgs -0 0.051 0.096 0.068 0.096 -0.045 0.26 0.27 0.052 1
Recent Chgs -0 0.054 0.097 0.069 0.097 -0.047 0.27 0.28 0 1 1
Subsys Chgs 0 0.1 0.16 0.11 0.16 -0.08 0.33 0.33 0.054 0.89 0.89 1
Awareness 0.11 0.18 0.2 0.23 0.2 -0.11 0 -0.1 0 0.69 0.69 0.64

Lines
Lin Addees
De d
Dir leted
ect
ory
File
Su
bsy
ste
En m
tro
Ch py
an
De ges
velo
per
Ag s
e
Pir
or
Re Chgs
cent
Su Chgbsy s
sAw Chgsaresi ness

0.6

1.0

ea
n

0.8

Math, time window = 60 days

Lines Added 1
Lines Deleted 0.51 1
Directory 0.74 0.61 1
File 0.76 0.65 0.91 1
Subsystem 0.74 0.61 1 0.91 1
Entropy 0.2 0.25 0.21 0.14 0.21 1
Changes 0.51 0.65 0.7 0.71 0.7 0.31 1
Developers 0.44 0.59 0.61 0.58 0.61 0.42 0.82 1
Age 0.43 0.54 0.59 0.57 0.59 0.41 0.66 0.63 1
Piror Chgs 0.032 -0.046 0.042 0 0.042 0 0.028 -0.047 0 1
Recent Chgs 0.03 -0.05 0.036 0 0.036 0 0 -0.054 -0.027 0.99 1
Subsys Chgs 0.5 0.49 0.58 0.55 0.58 0.37 0.55 0.58 0.38 0.18 0.18 1
Awareness 0.45 0.42 0.47 0.48 0.47 0.43 0.38 0.31 0.31 0.26 0.26 0.74m2r
sim sum2
sim rmax
2r
m

1.0

Lin
es
Lin Addees
De d
Dir leted
ect
ory
File
Su
bsy
ste
En m
tro
Ch py
an
De ges
velo
per
Ag s
e
Pir
or
Re Chgs
cent
Su Chgbsy s
sAw Chgsaresi nessLang, time window = 60 days

Lines Added 1
Lines Deleted 0.56 1
Directory 0.79 0.6 1
File 0.79 0.62 0.95 1
Subsystem 0.79 0.6 1 0.95 1
Entropy 0.42 0.45 0.43 0.38 0.43 1
Changes 0.58 0.57 0.74 0.73 0.74 0.52 1
Developers 0.55 0.55 0.7 0.68 0.7 0.57 0.93 1
Age 0.56 0.53 0.69 0.69 0.69 0.55 0.67 0.68 1
Piror Chgs -0.065 -0 -0 -0.027 -0 -0 0 -0.044 -0 1
Recent Chgs -0.068 -0 -0.031 -0.039 -0.031 -0 0 -0.054 -0.057 0.99 1
Subsys Chgs 0.54 0.54 0.65 0.61 0.65 0.54 0.63 0.62 0.52 0.34 0.33 1
Awareness 0.54 0.51 0.59 0.59 0.59 0.57 0.45 0.4 0.46 0.34 0.33 0.82(c) Closure

Fig. 2. Heat maps of Spearman correlation coefficients between the 16 features. The coefficients whose p-values are greater than 0.05 are replaced with zero.

Table III shows the results of Mann-Whitney U-test between
sim2rM feature values from fix-inducing and non-inducing
commits. We have performed a two-tailed test with the alternative hypothesis that fix-inducing commits have higher
sim2rM values than non-inducing commits. Each cell in thetable contains the p-value from a specific U-test. Among three
sim2rM features, sim2rmax shows the weakest correlation
with its p-values rarely below 0.05, failing to reject thenull-hypothesis. sim2rmean shows the strongest correlation,
rejecting the null-hypothesis for all three projects; sim2rsum
sits in the middle. We suspect that these different correlations
may stem from how these features are aggregated. sim2rmax
takes the maximum similarity between bug reports and codechanges; it is more easily affected by any outlier bug reportcoincidently similar to the commit under consideration thanthe other features. In contrast, sim2rmean is the most stable
due to taking the arithmetic mean; sim2rsum is less sensitive
to outliers than sim2rmax , but susceptible to the number ofbug reports as it adds their similarities to the commit.
Table IV shows the rankings of the sim2rM features
when the 16 features used to train DP models are sorted indescending order of their feature importance values in thosemodels: if a feature is within the top four most importantfeatures (25%), its ranking is written in bold. Overall, at least
one variation of the sim2rM features is in the top four, further
emphasising the usefulness of sim2rM features.
Answer to RQ1: sim2rM features capture distinct anduseful properties of fix-inducing commits. Between three
sim2rM features, sim2rmean is the most useful one, followedby sim2rsum , and then sim2rmax .
B. RQ2. Effectiveness
Tables V to VII compare the performance of defect prediction with (denoted by IRFL) and without using sim2rM
features (denoted by base). Any performance metric value inbold text indicates that the use of sim2rM features outperforms the baseline; the values in percentage denote the degreeof relative improvement. The results are evaluated both foreach fold (fold 1 to 4) and for the union of these four folds(all). The row all in Tables V to VII, and the rightmost plotsof Figures 3a to 3f, show the summary of the results: overall,

TABLE VT HE EFFECTIVENESS OF sim2rM FEATURES IN L ANG . T HE VALUES INPERCENTAGE DENOTE THE RELATIVE IMPROVEMENT FROM USING THE
sim2rM FEATURES . T HE UNDERLINED VALUE MEANS THAT THE
RESPECTIVE IMPROVEMENT IS STATISTICALLY SIGNIFICANT (p ≤ 0.05)
AND HAS AT LEAST A MEDIUM EFFECT SIZE (δ ≥ 0.33).

F

1

2

3

4

all

30 days

IRFL: with sim2rM features
60 days
90 daysmetric

base

AUC
Acc
Accbal
F1
PR

0.698
0.929
0.540
0.146
0.195

0.671
0.929
0.535
0.132
0.203

-3.9%
0.0%
-0.9%
-9.6%
4.1%

0.658
0.929
0.541
0.149
0.204

-5.7%
0.0%
0.2%
2.1%
4.6%

0.662
0.928
0.537
0.137
0.205

-5.2%
-0.1%
-0.6%
-6.2%
5.1%

0.681
0.928
0.537
0.135
0.201

120 days
-2.4%
-0.1%
-0.6%
-7.5%
3.1%

AUC
Acc
Accbal
F1
PR

0.799
0.932
0.647
0.259
0.189

0.829
0.940
0.654
0.286
0.230

3.8%
0.9%
1.1%
10.4%
21.7%

0.842
0.937
0.632
0.252
0.202

5.4%
0.5%
-2.3%
-2.7%
6.9%

0.829
0.938
0.645
0.268
0.183

3.8%
0.6%
-0.3%
3.5%
-3.2%

0.833
0.926
0.640
0.237
0.176

4.3%
-0.6%
-1.1%
-8.5%
-6.9%

AUC
Acc
Accbal
F1
PR

0.682
0.900
0.592
0.195
0.163

0.692
0.907
0.582
0.187
0.167

1.5%
0.8%
-1.7%
-4.1%
2.5%

0.729
0.895
0.627
0.234
0.187

6.9%
-0.6%
5.9%
20.0%
14.7%

0.712
0.902
0.634
0.251
0.192

4.4%
0.2%
7.1%
28.7%
17.8%

0.717
0.890
0.633
0.234
0.186

5.1%
-1.1%
6.9%
20.0%
14.1%

AUC
Acc
Accbal
F1
PR
AUC
Acc
Accbal
F1
PR

0.866
0.925
0.757
0.092
0.058
0.644
0.921
0.584
0.177
0.110

0.874
0.926
0.751
0.090
0.055
0.649
0.926
0.580
0.176
0.113

0.9%
0.1%
-0.8%
-2.2%
-5.2%
0.8%
0.5%
-0.7%
-0.6%
2.7%

0.886
0.926
0.787
0.101
0.063
0.684
0.922
0.593
0.191
0.124

2.3%
0.1%
4.0%
9.8%
8.6%
6.2%
0.1%
1.5%
7.9%
12.7%

0.871
0.931
0.763
0.100
0.067
0.666
0.925
0.596
0.199
0.125

0.6%
0.6%
0.8%
8.7%
15.5%
3.4%
0.4%
2.1%
12.4%
13.6%

0.876
0.906
0.840
0.096
0.090
0.663
0.912
0.595
0.182
0.123

1.2%
-2.1%
11.0%
4.3%
55.2%
3.0%
-1.0%
1.9%
2.8%
11.8%

DP models using sim2rM features can often outperform thosethat do not. The underlined values in Tables V to VII showthat the improvement of the evaluation metric computed fromthe 30 repetitions is statistically significant (p ≤ 0.05) with amoderate effect size (δ > 0.33) in most cases.
As described in Figure 3, the impact of using sim2rM
features varies depending on the size of the time windowthat was used to collect bug reports. Overall, there does notexist the best value for the time window that outperforms all
the others, particularly since this value also changes during
the software development. For example, in Lang, 30 days isthe best choice for the time window in fold two, whereas,in fold three, 90 days is the better choice. To maximise theeffectiveness of sim2rM features, the time window should betuned for each project and adjusted dynamically throughout
the software development. Nevertheless, within these results,
the best value for the time window would be 90 days for

0.6
0.4
0.2
0.0
0.2

1

2

3

Fold

4

all

Math

0.5
0.4
0.3
0.2
0.1
0.0
0.1
0.2
0.3

1

2

(a) Lang: F1

0.1
0.0
0.1

2

3

Fold

all

4

1
0
1

1

2

3

Fold

all

30 dys
60 dys90 dys120 dys

0.05
0.00
0.05

30 dys
60 dys90 dys120 dys

0.10(d) Lang: Balanced Accuracy

2

3

Fold

4

all

Closure

0.3

0.10

1

4

30 dys
60 dys90 dys120 dys(c) Closure: F1

Math

0.2

1

4

30 dys
60 dys90 dys120 dys2(b) Math: F1
Rel. Imprv (Bal_Acc)

Rel. Imprv (Bal_Acc)

3

Fold

Lang

0.3

Closure3

allRel. Imprv (Bal_Acc)

0.4

30 dys
60 dys90 dys120 dysRel. Imprv (F1)Rel. Imprv (F1)0.8

Rel. Imprv (F1)

Lang

1.0

0.2
0.1
0.0
0.1
0.2

1

(e) Math: Balanced Accuracy

2

3

Fold

4

all

30 dys
60 dys90 dys120 dys(f) Closure: Balanced Accuracy

Fig. 3. Relative Improvements in F1 score and Balanced Accuracy: while the degree of improvement varies depending on both time window and fold, using
sim2rM features often improves the overall performance of defective prediction models (Rel.Imprvall > 0)

TABLE VI
T HE EFFECTIVENESS OF sim2rM FEATURES IN M ATH .
F

1

2

3

4

all

30 days

IRFL: with sim2rM features
60 days
90 days

TABLE VII
T HE EFFECTIVENESS OF sim2rM FEATURES IN C LOSURE .

metricbase

120 days

AUC
Acc
Accbal
F1
PR

0.787
0.826
0.656
0.414
0.429

0.797
0.834
0.668
0.436
0.447

1.3%
1.0%
1.8%
5.3%
4.2%

0.790
0.847
0.660
0.433
0.444

0.4%
2.5%
0.6%
4.6%
3.5%

0.794
0.852
0.652
0.424
0.442

0.9%
3.1%
-0.6%
2.4%
3.0%

0.799
0.847
0.677
0.457
0.459

1.5%
2.5%
3.2%
10.4%
7.0%

AUC
Acc
Accbal
F1
PR

0.833
0.896
0.714
0.430
0.475

0.844
0.894
0.746
0.459
0.490

1.3%
-0.2%
4.5%
6.7%
3.2%

0.849
0.882
0.763
0.454
0.459

1.9%
-1.6%
6.9%
5.6%
-3.4%

0.842
0.882
0.737
0.432
0.449

1.1%
-1.6%
3.2%
0.5%
-5.5%

0.842
0.896
0.741
0.459
0.464

1.1%
0.0%
3.8%
6.7%
-2.3%

AUC
Acc
Accbal
F1
PR

0.763
0.866
0.682
0.336
0.221

0.763
0.876
0.641
0.301
0.226

0.0%
1.2%
-6.0%
-10.4%
2.3%

0.759
0.863
0.662
0.312
0.203

-0.5%
-0.3%
-2.9%
-7.1%
-8.1%

0.766
0.844
0.667
0.300
0.197

0.4%
-2.5%
-2.2%
-10.7%-10.9%

0.758
0.835
0.685
0.310
0.203

-0.7%
-3.6%
0.4%
-7.7%
-8.1%

AUC
Acc
Accbal
F1
PR
AUC
Acc
Accbal
F1
PR

0.833
0.944
0.643
0.303
0.283
0.809
0.883
0.679
0.385
0.344

0.840
0.953
0.644
0.337
0.311
0.817
0.889
0.684
0.400
0.365

0.8%
1.0%
0.2%
11.2%
9.9%
1.0%
0.7%
0.7%
3.9%
6.1%

0.841
0.957
0.648
0.360
0.322
0.809
0.887
0.687
0.401
0.347

1.0%
1.4%
0.8%
18.8%
13.8%
0.0%
0.5%
1.2%
4.2%
0.9%

0.845
0.955
0.614
0.297
0.297
0.806
0.883
0.673
0.378
0.329

1.4%
1.2%
-4.5%
-2.0%
4.9%
-0.4%
0.0%
-0.9%
-1.8%
-4.4%

0.845
0.955
0.610
0.285
0.288
0.809
0.883
0.690
0.398
0.336

1.4%
1.2%
-5.1%
-5.9%
1.8%
0.0%
0.0%
1.6%
3.4%
-2.3%

Lang and Closure and 30 or 60 days for Math. With thesetime windows, sim2rM features improve F1 score by 12.4%,
4.2%, and 92.2% and Balanced Accuracy by 2.1%, 1.2%, and3.7%, respectively for Lang, Math, and Closure.
We observe that sim2rM features mainly improve F1 scoreand PR-AUC while increasing the other metrics evaluated overthe entire set, i.e., AUC, Accuracy, and Balanced Accuracy,
by a small degree, at least when using the best time window.
For example, in Lang, sim2rM features with the 90 days
window improve F1 and PR-AUC by 12.4% and 13.6%,
respectively; for the other metrics, the relative improvementsare 3.4%, 0.4%, and 2.1%, respectively for AUC, Accuracy,
and Balanced Accuracy. This increase in F1 score and PRAUC along with the small increase in the other metrics implies
that fix-inducing commits that were previously predicted as

F

1

2

3

4

all

30 days

IRFL: with sim2rM features
60 days
90 daysmetric

base

AUC
Acc
Accbal
F1
PR

0.671
0.941
0.519
0.078
0.127

0.669
0.936
0.533
0.120
0.140

-0.3%
-0.5%
2.7%
53.8%
10.2%

0.644
0.946
0.528
0.105
0.154

-4.0%
0.5%
1.7%
34.6%
21.3%

0.626
0.949
0.544
0.161
0.181

-6.7%
0.9%
4.8%
106.4%
42.5%

0.635
0.948
0.538
0.140
0.189

120 days
-5.4%
0.7%
3.7%
79.5%
48.8%

AUC
Acc
Accbal
F1
PR

0.815
0.982
0.547
0.156
0.326

0.833
0.984
0.559
0.189
0.417

2.2%
0.2%
2.2%
21.2%
27.9%

0.798
0.983
0.524
0.083
0.452

-2.1%
0.1%
-4.2%
-46.8%
38.7%

0.785
0.983
0.519
0.065
0.427

-3.7%
0.1%
-5.1%
-58.3%
31.0%

0.808
0.983
0.524
0.083
0.474

-0.9%
0.1%
-4.2%
-46.8%
45.4%

AUC
Acc
Accbal
F1
PR

0.684
0.964
0.495
0.004
0.073

0.679
0.962
0.507
0.034
0.088

-0.7%
-0.2%
2.4%
750.0%
20.5%

0.666
0.963
0.509
0.038
0.088

-2.6%
-0.1%
2.8%
850.0%
20.5%

0.649
0.966
0.547
0.128
0.116

-5.1%
0.2%
10.5%
3100.0%
58.9%

0.653
0.962
0.540
0.106
0.102

-4.5%
-0.2%
9.1%
2550.0%
39.7%

AUC
Acc
Accbal
F1
PR
AUC
Acc
Accbal
F1
PR

0.843
0.988
0.496
0.000
0.025
0.745
0.969
0.518
0.064
0.082

0.807
0.984
0.495
0.000
0.022
0.750
0.966
0.531
0.096
0.096

-4.3%
-0.4%
-0.2%
0.0%
-12.0%
0.7%
-0.3%
2.5%
50.0%
17.1%

0.791
0.988
0.496
0.000
0.021
0.726
0.970
0.521
0.074
0.095

-6.2%
0.0%
0.0%
0.0%
-16.0%
-2.6%
0.1%
0.6%
15.6%
15.9%

0.843
0.987
0.496
0.000
0.026
0.710
0.971
0.537
0.123
0.106

0.0%
-0.1%
0.0%
0.0%
4.0%
-4.7%
0.2%
3.7%
92.2%
29.3%

0.830
0.988
0.497
0.000
0.024
0.724
0.970
0.533
0.110
0.106

-1.5%
0.0%
0.2%
0.0%
-4.0%
-2.8%
0.1%
2.9%
71.9%
29.3%

non-inducing are correctly predicted as fix-inducing by using
sim2rM features, reducing the number of false-negatives,
which in our case, are fix-inducing commits predicted as
non-inducing. Bug reports explain bugs that were induced by
fix-inducing commits. We argue that by directly comparing
code changes to bug reports, sim2rM features capture theproperties of fix-inducing commits that are related to thecontext of induced bugs, which the previous features based on
the meta-data of fix-inducing commits were unable to capture.
Figure 3 shows that the degree of improvement varies
depending on folds. While there is no strong general trend, theimprovement is more notable in folds with more fix-inducing
commits, such as the first fold of Math and Closure.
Answer to RQ2: sim2rM features can improve the performance of defect prediction models, given sufficient fix-inducing commits to train and test. Benefits of sim2rM can befurther maximised by tuning the time window for each project.
C. RQ3. Actionability
Table VIII and Figure 4 compare the efforts spent until
encountering the first defective file between when following
the order of suspf , and when following the baselines, which
are random ordering, ordered by closed bug count (cntCB ),and ordered by closed bug similarity (simCB ). To focus on
the cases where developers can save their effort by knowing
defective files in advance, we consider only the fix-inducing
commits that also contain non-defective changes, for evaluation. As there are always more than one modified files in thesecases, the expected rank of the random approach is always
greater than one, resulting in acc@1 to be zero. Compared to
the random ordering, the use of suspf successfully increases
acc@1 of Lang, Math, and Closure from zero up to 42, 93,
and 28, respectively. While the degree of increase in acc@ndecreases as n grows, developers can still find the defective
file earlier using suspf , when compared to random inspection.
The effectiveness of suspf tends to increase as the time
window increases. For example, as the time window changes
from 30 to 60 days, acc@1 increases from 70 to 90 in Math.
However, an unnecessarily large time window may also harmthe effectiveness of suspf , since it becomes more likely thatthe used bug reports are now outdated: acc@1 decreases from93 to 89 in Math when the time window changes from 90 to
120 days. Similarly to our answer to RQ2, we may need totune the time window for each project to maximise the benefitof suspf . In general, Table VIII shows that a time window of
90 to 120 days achieves fair performance.
Inspecting files following the order of suspf can easily
outperform the other two baselines, cntCB and simCB . For
all three projects, acc@n with suspf is significantly higher:
for Lang and Math, acc@1 increases at least by 2600% and1067%, and by 800% and 592%, when compared to cntCBand simCB , respectively. For Closure, while both cntCBand simCB failed to rank any defective file at the top for
any fix-inducing commits (acc@1 = 0), suspf successfully
does so for at lest 27 commits (acc@1 = 27). Unlike therandom approach, cntCB and simCB do take past faults into
account, similarly to suspf . We argue that the difference inactionability comes from whether an approach takes files thatwere non-defective in the past into account: suspf does byusing past suspiciousness of files regardless of whether they
were defective in the past, whereas cntCB and simCB do
not by considering only the files that were defective in thepast. Consequently, from the improvement gained by using
suspf over cntCB and simCB , we suspect that the previously
suspicious files that were not actually defective may turn out
to be relevant to defective changes in subsequent commits that
we investigated.
Figure 4 visualises the complete comparison between suspf
and the baselines. Regardless of the choice of the time window,
suspf always reduces the inspection effort of developers by
successfully ranking defective files near the top.

Table IX presents how the effectiveness of suspf changes
as more reports are included for computation of suspf . Here,CB denotes closed bug repots, B denotes bug reports only withthe status of open or close, and A denotes all types of bug
reports. In Lang and Math, acc@n increases as more report
types are considered. For example, by switching from using
only CB to A, the degree of improvement in acc@1 increases
from at least 592% to at least 908% for Math with 60 days
window. While acc@n decreases for Closure by considering
open bug reports in addition (i.e., going from C to B), suspf
is the most effective when all types of bug reports are taken
into consideration (i.e., A). These results suggest that we canfurther improve actionability of DP results by leveraging more
bug reports. Table X reports how much effort (measured in
the number of files inspected) we can save by employing
suspf computed using all types of bug reports within the time
window. On average, developers can inspect two to nine fewer
files before meeting the first defective file.
Answer to RQ3: suspf can make DP results moreactionable by successfully guiding developers to defective files
based on the past suspiciousness. Comparison to the baselines
suggests that the previously suspicious, but non-defective files
can still be relevant to defective changes in the future commits.
V. T HREATS TO VALIDITY
The primary threat to validity of this study is whether thedata collection and model inference have been performed correctly. To mitigate this threat, we use the SZZ algorithm [23]that is widely used for defect prediction research [3], [13].
We use the open source implementation [24], with additional
filters from previous work [3] that are designed to reduce
false-positives. For model inference, we use the widely usedmachine learning package, scikit-learn [32].
Threats to external validity concerns any factors that affects
how generalisable the results of this study are. The biggest
factor is the fact that our results are based on Defects4J,
a benchmark of real-world faults [21]. Defects4J has been
extensively studied in the context of Fault Localisation (FL),and we use it to exploit the synergy between DP and FL.
However, further studies are needed to confirm our results forthe benchmark datasets in the area of defect prediction.
Threats to construct validity concerns whether our measurement actually reflects what we intend to measure. All
our evaluation metrics are standard metrics used for binaryclassification, which is how defect prediction is formulated.
VI. R ELATED W ORK
Our aim is to improve both accuracy and actionability of defect prediction results by considering previous faults via fault
localisation techniques. Historical faults have been studied inrelation to recent or future faults. Yu et al. generated a mathematical model that predicts the probability of a subsystemhaving a defect based on the correlation between the number ofpast and future faults [33]. Similarly, Graves et al. utilised thenumber of past defects to generate a model for predicting thenumber of future faults in a program module [34]. Exploiting

TABLE VIIIACTIONABILITY (acc@n): C OLUMN n DENOTES THE n IN acc@n. THE PARENTHESISED NUMBER IN COLUMN PROJECT IS THE TOTAL NUMBER OFEVALUATED FIX - INDUCING COMMITS . T HE OTHER PARENTHESISED VALUES ARE THE RELATIVE IMPROVEMENTS . F OR cntCB AND simCB , WE
COMPARED THEM WITH THE RANDOM (R). F OR suspf , THESE VALUSE ARE ARE THE RELATIVE IMPROVEMENTS OVER R, cntCB , AND simCB .
30 dayssimCB

60 dayssimCB

90 days
simCB

120 days
simCB

n

R

cntCB

suspf

cntCB

suspf

cntCB

suspf

cntCB

Lang (131)

1
3
5

0
78
98

1(—)
79 (0.01)
99 (0.01)

2(—)
79 (0.01)
99 (0.01)

27 ( — ,26.00,12.50)
94 (0.21, 0.19, 0.19)
109 (0.11, 0.10, 0.10)

1(—)
81 (0.04)
101 (0.03)

3(—)
81 (0.04)
101 (0.03)

33 ( — ,32.00,10.00)
93 (0.19, 0.15, 0.15)
108 (0.10, 0.07, 0.07)

1(—)
82 (0.05)
102 (0.04)

4(—)
82 (0.05)
102 (0.04)

36 ( — ,35.00, 8.00)
96 (0.23, 0.17, 0.17)
111 (0.13, 0.09, 0.09)

1(—)
82 (0.05)
102 (0.04)

4(—)
82 (0.05)
102 (0.04)

42 ( — ,41.00, 9.50)
98 ( 0.26, 0.20, 0.20)
110 (0.12, 0.08, 0.08)

Math (324)

1
3
5

0
6(—)
151 155 (0.03)
199 205 (0.03)

10 ( — )155 (0.03)
205 (0.03)

70 ( — ,10.67, 6.00)
201 (0.33, 0.30, 0.30)
230 (0.16, 0.12, 0.12)4(—)
152 (0.01)
205 (0.03)

13 ( — )153 (0.01)
205 (0.03)

90 ( — ,21.50, 5.92)
208 (0.38, 0.37, 0.36)
238 (0.20, 0.16, 0.16)

4(—)
151 (0.00)
204 (0.03)

12 ( — )152 (0.01)
204 (0.03)

93 ( — ,22.25, 6.75)
207 (0.37, 0.37, 0.36)
239 (0.20, 0.17, 0.17)

4(—)
151 (0.00)
204 (0.03)

12 ( — )152 (0.01)
204 (0.03)

89 ( — ,21.25, 6.42)
204 (0.35, 0.35, 0.34)
236 (0.19, 0.16, 0.16)

Closure (77)

1
3
5

0
38
51

0(—)
38 (0.00)
51 (0.00)

28 ( —- , —- , —- )55 (0.45, 0.45, 0.45)
66 (0.29, 0.29, 0.29)

0(—)
38 (0.00)
51 (0.00)

0(—)
38 (0.00)
51 (0.00)

27 ( —- , —- , —- )56 (0.47, 0.47, 0.47)
67 (0.31, 0.31, 0.31)

0(—)
38 (0.00)
51 (0.00)

0(—)
38 (0.00)
51 (0.00)

28 ( —- , —- , —- )59 (0.55, 0.55, 0.55)
67 (0.31, 0.31, 0.31)

0(—)
38 (0.00)
51 (0.00)

0(—)
38 (0.00)
51 (0.00)

27 ( —- , —- , —- )58 (0.53, 0.53, 0.53)
66 (0.29, 0.29, 0.29)

Lang

130
117
104
91
78
65
52
39
26
13
0
0

3

6

RANDOMSUSPF and 30 days
SUSPF and 60 days
SUSPF and 90 days
SUSPF and 120 days
CNTCB and 90 days
SIMCB and 90 days
9 12 15 18 21 24 27 30 33 36 39 42 45 48 51 54

Math

320
288
256
224
192
160
128
96
64
32
0
0

Effort (# of files)

3

6

9

RANDOMSUSPF and 30 days
SUSPF and 60 days
SUSPF and 90 days
SUSPF and 120 days
CNTCB and 60 days
SIMCB and 60 days
12 15 18 21 24 27 30 33 36 39 42 45 48

Effort (# of files)

suspf

Closure# of fix-inducing commits0(—)
38 (0.00)
51 (0.00)# of fix-inducing commits# of fix-inducing commits

Project

70
63
56
49
42
35
28
21
14
7
0
0

1

2

3

4

5

6

7

8

9

Effort (# of files)

10

11

RANDOMSUSPF and 30 days
SUSPF and 60 days
SUSPF and 90 days
SUSPF and 120 days
CNTCB and 90 days
SIMCB and 90 days
12 13 14 15

Fig. 4. The effort spent before meeting the first defective file: the x-axis is the number of non-defective files that are inspected, and the y-axis represents thenumber of fix-inducing commits in which at least one of their defective file is located in the top x + 1. For better visualisation, the graphs cover up to only
95% of the fix-inducing commits. For cntCB and simCB , we report only the best results.
,
TABLE IXC OMPARISON BETWEEN THREE TYPES OF suspf USING THREE DIFFERENTSETS OF REPORTS (CB, B, AND A). T HE ONE WITH THE BESTPERFORMANCE IS IN BOLD TEXT.
ProjectnCB

30 daysB

ACB

60 daysB

ACB

90 daysB

ACB

120 days
B

A

Lang (131)

1
3
5

27
94
109

49
101
111

49
101
111

33
93
108

51
99
111

51
99
111

36
96
111

53
96
113

53
96
113

42
98
110

57
99
111

57
99
111

Math (324)

1
3
5

70
201
230

88
206
238

123
233
261

90
208
238

94
204
242

131
243
266

93
207
239

100
204
247

134
238
264

89
204
236

95
206
244

125
237
263

Closure (77)

1
3
5

28
55
66

26
52
65

32
60
69

27
56
67

20
50
61

30
58
67

28
59
67

17
49
64

31
60
65

27
58
66

18
48
63

32
60
65

TABLE XE FFORT SAVED FROM USING suspf WITH ALL REPORTS (A) ( AVG . # OFFILES ): cnt AND sim ARE THE SHORT OF CNTCB AND SIM CB .
30 dayscnt
sim

R

60 dayscnt
sim

R

90 dayscnt
sim

R

120 dayscnt
sim

Project

R

Lang

4.5

4.5

4.5

4.7

3.9

4.0

4.7

3.3

3.3

5.1

3.8

3.8

Math

8.1

4.9

4.9

8.8

5.6

5.5

9.0

5.9

5.8

8.6

5.5

5.5

Closure

2.4

2.4

2.4

2.1

2.1

2.1

2.0

2.0

2.0

1.9

1.9

1.9

technique that uses the results of mutation testing on a target
program [25]. The results showed that the performance of
defect prediction could be improved by using the dynamic
analysis. While this work is similar in the sense that it exploits
information from other faults, it uses artificial mutants whereas
we directly leverage previous faults via fault localisation.
VII. C ONCLUSION
We focus on the observation that while both fault localisation and defect prediction try to assure a certain degree ofsoftware quality, the impact each process has on the other has
been rarely investigated. As the first step of exploring thismostly untouched area from the side of defect prediction, weleverage the past suspiciousness scores of code changes computed from fault localisation to improve defect prediction. The
experimental results on three open-source projects confirm thatthe past suspiciousness of code can enhance defect prediction
by improving the performance of JIT defect prediction models
and assisting developers on processing fix-inducing commits
via further directing them to defective files.
ACKNOWLEDGEMENT

the correlation even further, Hassan and Holt generated the listof the top ten subsystems that are most susceptible to havefaults based on their previous faults [35]. Zimmerman et al.
also showed that the number of pre-release and post-releasefaults are correlated [36]. While all these studies highlight thecorrelation between past and future faults, most of them stopat looking at the number and location of faults. We try toprovide richer information for defect prediction by adopting a
fault localisation technique.
Recently, Bowes et al. proposed a new defect prediction

Jeongju Sohn and Shin Yoo were supported by National Research Foundation of Korea (NRF) grant funded by the Korea
government (MEST) (Grant No. NRF-2020R1A2C1013629).
Yasutaka Kamei was partially supported by JSPS KAKENHI
Grant Numbers JP18H03222 and JSPS International Joint
Research Program with SNSF (Project ”SENSOR”).
R EFERENCES[1] C. Catal, “Software fault prediction: A literature review and current
trends,” Expert Systems with Applications, vol. 38, no. 4, pp. 4626 –
4636, 2011.

[2] Y. Kamei and E. Shihab, “Defect prediction: Accomplishments andfuture challenges,” in 2016 IEEE 23rd International Conference on
Software Analysis, Evolution, and Reengineering (SANER), vol. 5, 2016,
pp. 33–45.
[3] S. McIntosh and Y. Kamei, “Are fix-inducing changes a moving target? a longitudinal case study of just-in-time defect prediction,” IEEE
Transactions on Software Engineering, vol. 44, no. 5, pp. 412–428, May
2018.
[4] W. E. Wong, R. Gao, Y. Li, R. Abreu, and F. Wotawa, “A survey on
software fault localization,” IEEE Transactions on Software Engineering,
vol. 42, no. 8, pp. 707–740, 2016.
[5] T.-D. B. Le, R. J. Oentaryo, and D. Lo, “Information retrieval andspectrum based bug localization: Better together,” in Proceedings of the2015 10th Joint Meeting on Foundations of Software Engineering, ser.
ESEC/FSE 2015. New York, NY, USA: ACM, 2015, pp. 579–590.
[6] R. K. Saha, M. Lease, S. Khurshid, and D. E. Perry, “Improving bug localization using structured information retrieval,” in Automated Software
Engineering (ASE), 2013 IEEE/ACM 28th International Conference on.
IEEE, 2013, pp. 345–355.
[7] M. Wen, R. Wu, and S.-C. Cheung, “Locus: Locating bugs fromsoftware changes,” in Proceedings of the 31st IEEE/ACM International
Conference on Automated Software Engineering, ser. ASE 2016. New
York, NY, USA: Association for Computing Machinery, 2016, p.
262–273. [Online]. Available: https://doi.org/10.1145/2970276.2970359[8] J. Sohn and S. Yoo, “Fluccs: Using code and change metrics to improve
fault localisation,” in Proceedings of the International Symposium on
Software Testing and Analysis, ser. ISSTA 2017, 2017, pp. 273–283.
[9] J. Sohn and S. Yoo, “Empirical evaluation of fault localisation usingcode and change metrics,” IEEE Transactions on Software Engineering,pp. 1–1, 2019.
[10] S. Kim, T. Zimmermann, E. J. Whitehead Jr., and A. Zeller, “Predicting faults from cached history,” in 29th International Conference on
Software Engineering (ICSE’07), May 2007, pp. 489–498.
[11] F. Rahman, D. Posnett, A. Hindle, E. Barr, and P. Devanbu,“Bugcache for inspections: Hit or miss?” in Proceedings of the
19th ACM SIGSOFT Symposium and the 13th European Conference
on Foundations of Software Engineering, ser. ESEC/FSE ’11. New
York, NY, USA: ACM, 2011, pp. 322–331. [Online]. Available:
http://doi.acm.org/10.1145/2025113.2025157[12] A. Mockus and D. M. Weiss, “Predicting risk of software changes,” Bell
Labs Technical Journal, vol. 5, no. 2, pp. 169–180, 2000.
[13] E. Shihab, A. E. Hassan, B. Adams, and Z. M. Jiang, “An industrial
study on the risk of software changes,” in Proceedings of the
ACM SIGSOFT 20th International Symposium on the Foundationsof Software Engineering, ser. FSE ’12. New York, NY, USA:
Association for Computing Machinery, 2012. [Online]. Available:
https://doi.org/10.1145/2393596.2393670[14] M. Papadakis and Y. L. Traon, “Metallaxis-fl: mutation-based fault
localization,” Softw. Test., Verif. Reliab., vol. 25, no. 5-7, pp. 605–628,
2015. [Online]. Available: http://dx.doi.org/10.1002/stvr.1509[15] X. Li, S. Zhu, M. d’Amorim, and A. Orso, “Enlightened debugging,”
in Proceedings of the 40th International Conference on Software
Engineering, ser. ICSE ’18. New York, NY, USA: Association
for Computing Machinery, 2018, p. 82–92. [Online]. Available:
https://doi.org/10.1145/3180155.3180242[16] S. Yoo, X. Xie, F.-C. Kuo, T. Y. Chen, and M. Harman, “Human competitiveness of genetic programming in sbfl: Theoretical and empiricalanalysis,” ACM Transactions on Software Engineering and Methodology,
vol. 26, no. 1, pp. 4:1–4:30, July 2017.
[17] X. Li, W. Li, Y. Zhang, and L. Zhang, “Deepfl: Integrating multiple
fault diagnosis dimensions for deep fault localization,” in Proceedingsof the 28th ACM SIGSOFT International Symposium on Software
Testing and Analysis, ser. ISSTA 2019. New York, NY, USA:
Association for Computing Machinery, 2019, p. 169–180. [Online].
Available: https://doi.org/10.1145/3293882.3330574[18] G. Salton, A. Wong, and C. S. Yang, “A vector space model for
automatic indexing,” Commun. ACM, vol. 18, no. 11, pp. 613–620, Nov.
1975.
[19] J. Romano, J. Kromrey, J. Coraggio, and J. Skowronek, “Appropriate
statistics for ordinal level data: Should we really be using t-test andCohen’sd for evaluating group differences on the NSSE and othersurveys?” in annual meeting of the Florida Association of Institutional
Research, 2006, pp. 1–3.

[20] C. Lewis, Z. Lin, C. Sadowski, X. Zhu, R. Ou, and E. J. Whitehead,“Does bug prediction support human developers? findings from agoogle case study,” in 2013 35th International Conference on Software
Engineering (ICSE), May 2013, pp. 372–381.
[21] R. Just, D. Jalali, and M. D. Ernst, “Defects4j: A database of existingfaults to enable controlled testing studies for java programs,” in Proceedings of the 2014 International Symposium on Software Testing andAnalysis, ser. ISSTA 2014. New York, NY, USA: ACM, 2014, pp.
437–440.
[22] Atlassian. [Online]. Available: https://www.atlassian.com/software/jira[23] A. Z. Jacek Śliwerski, Thomas Zimmermann, “When do changes
induce fixes?” in Proceedings of the 2005 International Workshop
on Mining Software Repositories, ser. MSR ’05. New York, NY,
USA: Association for Computing Machinery, 2005, p. 1–5. [Online].
Available: https://doi.org/10.1145/1083142.1083147[24] M. Borg, O. Svensson, K. Berg, and D. Hansson, “Szz unleashed:
An open implementation of the szz algorithm - featuring example
usage in a study of just-in-time bug prediction for the jenkins
project,” in Proceedings of the 3rd ACM SIGSOFT International
Workshop on Machine Learning Techniques for Software Quality
Evaluation, ser. MaLTeSQuE 2019. New York, NY, USA: Association
for Computing Machinery, 2019, p. 7–12. [Online]. Available:
https://doi.org/10.1145/3340482.3342742[25] D. Bowes, T. Hall, M. Harman, Y. Jia, F. Sarro, and F. Wu, “Mutationaware fault prediction,” in Proceedings of the 25th International
Symposium on Software Testing and Analysis, ser. ISSTA 2016. New
York, NY, USA: ACM, 2016, pp. 330–341. [Online]. Available:
http://doi.acm.org/10.1145/2931037.2931039[26] J. Nam, W. Fu, S. Kim, T. Menzies, and L. Tan, “Heterogeneous defectprediction,” IEEE Transactions on Software Engineering, vol. 44, no. 9,
pp. 874–896, 2018.
[27] S. Lessmann, B. Baesens, C. Mues, and S. Pietsch, “Benchmarking
classification models for software defect prediction: A proposed framework and novel findings,” IEEE Transactions on Software Engineering,
vol. 34, no. 4, pp. 485–496, 2008.
[28] T. Mende and R. Koschke, “Effort-aware defect prediction models,”in 2010 14th European Conference on Software Maintenance andReengineering, 2010, pp. 107–116.
[29] S. Tabassum, L. L. Minku, D. Feng, G. G. Cabral, and L. Song, “Aninvestigation of cross-project learning in online just-in-time software
defect prediction,” in Proceedings of the ACM/IEEE 42nd International
Conference on Software Engineering, ser. ICSE ’20. New York,NY, USA: Association for Computing Machinery, 2020, p. 554–565.
[Online]. Available: https://doi.org/10.1145/3377811.3380403[30] J. Nam, W. Fu, S. Kim, T. Menzies, and L. Tan, “Heterogeneous defectprediction,” IEEE Transactions on Software Engineering, vol. PP, pp.
1–1, 06 2017.
[31] L. Buitinck, G. Louppe, M. Blondel, F. Pedregosa, A. Mueller, O. Grisel,
V. Niculae, P. Prettenhofer, A. Gramfort, J. Grobler, R. Layton, J. VanderPlas, A. Joly, B. Holt, and G. Varoquaux, “API design for machinelearning software: experiences from the scikit-learn project,” in ECML
PKDD Workshop: Languages for Data Mining and Machine Learning,
2013, pp. 108–122.
[32] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion,
O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, “Scikit-learn: Machine learning in Python,” Journal of Machine
Learning Research, vol. 12, pp. 2825–2830, 2011.
[33] T. . Yu, V. Y. Shen, and H. E. Dunsmore, “An analysis of several software
defect models,” IEEE Transactions on Software Engineering, vol. 14,
no. 9, pp. 1261–1270, 1988.
[34] T. L. Graves, A. F. Karr, J. S. Marron, and H. Siy, “Predicting
fault incidence using software change history,” IEEE Transactions on
Software Engineering, vol. 26, no. 7, pp. 653–661, 2000.
[35] A. E. Hassan and R. C. Holt, “The top ten list: dynamic fault prediction,” in 21st IEEE International Conference on Software Maintenance(ICSM’05), 2005, pp. 263–272.
[36] T. Zimmermann, R. Premraj, and A. Zeller, “Predicting defects for
eclipse,” in Third International Workshop on Predictor Models in
Software Engineering (PROMISE’07: ICSE Workshops 2007), 2007, pp.
9–9.