Evaluating Bug Prediction under Realistic Settings
Sho Ogino∗ , Yoshiki Higo∗ and Shinji Kusumoto∗∗ Graduate School of Information Science and Technology, Osaka University, Japan{s-ogino, higo, kusumoto}@ist.osaka-u.ac.jp

Abstract—Bug prediction is expected to reduce the cost ofquality assurance. To build a reliable bug prediction model, we
should use realistic settings that satisfy all three of the following
conditions. (1) We should build a dataset in a way that allows
us to evaluate the prediction performance of the model correctly.
(2) We should adopt the optimal granularity of bug prediction
to minimize the cost of quality assurance. (3) We should use
a dependent variable that correctly represents the presenceor absence of bugs in the software modules to be predicted.
However, no research has been conducted on bug prediction
models built under the above realistic settings. Consequently,
we established the following two objectives in this research.
(1) We experimentally evaluate the prediction performance ofbug prediction models built under realistic settings. (2) We
propose techniques to improve the prediction performance ofbug prediction models built under realistic settings. The firstobjective has now been achieved. Our experimental results showthat the F-Measure of the bug prediction models built underrealistic settings is only 0.19. Thus, there are still some issues
to be solved to build a high-performance bug prediction model
under realistic settings.
Index Terms—quality assurance; bug prediction; machinelearning

I. I NTRODUCTION
In recent years, the scale of software development has beensteadily increasing. Under this situation, techniques to reduce
development costs are indispensable. Techniques to reducethe cost of quality assurance such as reviews are especially
important because quality assurance occupies a large amountof the development cost [1].
Bug prediction is a technique to reduce the cost of quality
assurance. Bug prediction means predicting the presence orabsence of bugs in the software modules (e.g. source files).
Identifying buggy modules and intensively reviewing them canreduce the cost of quality assurance.
To properly evaluate the performance of a bug prediction
model, the dataset must be properly built. Traditionally, a
dataset has been built by calculating dependent variables
for software modules existing in a project at a certain time
point and then calculating independent variables using thedevelopment history from the beginning of the project to thetime point [2], [3].
However, Pascarella et al. claimed datasets that are built
in such a way as unrealistic because they contain data thatcannot realistically be obtained [4]. Pascarella et al. proposed
the release-by-release technique to build a dataset that avoids
the problem, and they investigated the prediction performancewith a realistic dataset. They concluded that low performance
was measured and that challenges remain in building a highperformance bug prediction model with a realistic dataset.

We consider that Pascarella et al.’s bug prediction models
built with the release-by-release technique are controversial.
The problem with their model is that the dependent variable
is not the indicator “whether a bug exists in the softwaremodule at that point in time (isBuggy)”, but the indicator“whether a bug has been fixed at least once in the past period(hasBeenFixed)”. This is because there can be a situation
where a software module is “false for hasBeenFixed but true
for isBuggy”, which means a model built with a dataset whose
dependent variable is hasBeenFixed (hasBeenFixed model)
may not predict bugs correctly.
As mentioned above, the prior studies built models under
unrealistic settings on dataset building and dependent variables. To build reliable bug prediction models, we should use
realistic settings. However, no research has been conductedon bug prediction models built under realistic settings. In
this study, the following two objectives were set up andinvestigated.
1) We experimentally evaluate the performance of bug prediction models built under realistic settings.
2) We propose a technique to improve the performance ofbug prediction models built under realistic settings.
The first objective has been achieved at this moment. Our
experimental results showed that the F-Measure of the bug
prediction model built under realistic settings is only 0.19,
and there are still some issues to be solved to build highperformance bug prediction models under realistic settings. In
the near future, we will propose a way to improve performance.
II. BACKGROUND
Various settings exist for building bug prediction models.
Herein, we introduce two settings, (1) granularity of prediction
target and (2) way of building a dataset, as items closely
related to this study.
A. Granularity of prediction target
There are three granularities of target that have been often
used in bug prediction models: source file, method, andcommit. In this study, we focus on method-level bug prediction because Pascarella et al. adopted method for predictive
granularity and one of the goals of this study is to confirm thetruth of Pascarella et al.’s conclusions.
B. Way of building a dataset
To properly evaluate the performance of a bug prediction
model, the dataset must be built in a proper way. Traditionally,

independent variables have been calculated using the development history from the beginning of the project to a certain
time point [2], [3]. Pascarella et al. claimed building a dataset
in this way as impractical because the dataset contains datathat cannot realistically be retrieved.
Pascarella et al. proposed the release-by-release technique
as a suitable way to build a dataset that avoids such a problem. Release-by-release calculates dependent and independent
variables for the software modules present at a given releaseof the project. The independent variables are calculated using
the development history from the previous release to the given
release. The procedure for building a dataset based on therelease-by-release is described in detail in section V-B.
III. R ESEARCH Q UESTIONSIn this paper, we investigate the following four Research
Questions.
RQ1. Is hasBeenFixed proper as a dependent variable forbuilding bug prediction models?
Through RQ1, we try to quantitatively judge whether hasBeenFixed is proper as a dependent variable for building bug
prediction models. If most of the buggy methods are notcaptured by hasBeenFixed, then models to predict hasBeenFixed (hasBeenFixed models) cannot correctly predict buggy
methods, which means that hasBeenFixed is inappropriate asa dependent variable for bug prediction models.
RQ2. Is the prediction performance of isBuggy models
lower than hasBeenFixed models’?
Through RQ2, we compare hasBeenFixed models and isBuggy models from the perspective of prediction performance.
If the performance of isBuggy models is lower than hasBeenFixed models’, there are still some issues in this research area.
RQ3. Under the realistic settings, which machine learning algorithm provides the best prediction performance?
Through RQ3, we identify the optimal machine learningalgorithm under the following realistic settings.
Granularity method
dataset building release-by-release
Dependent variable isBuggy

RQ4. Under the realistic settings, how does prediction
performance change just as the development history
accumulates?
Through RQ4, we try to obtain insights into whether themachine learning algorithms should be varied with the amountof development history.
IV. CONFIGURATIONSIn this Section, we describe the experimental configurations
common to the experiments described in the following sections.
A. Target projects
Target projects in this study are listed in Table I. Those
projects were selected for the following four reasons.
• Their Git repositories are available.
• They are written in Java.
• They adopt semantic versioning.
• There are at least four releases in their projects.
B. How to identify releases
In this study, we build datasets following the release-byrelease technique. It is necessary to identify the timing ofeach release to use the release-by-release technique. We follow
the technique proposed by Pascarella et al. [4] to identify thereleases of the projects employing semantic versioning as a
versioning scheme. Semantic versioning expresses versions in
the form of X.Y.Z (e.g., 1.2.1), and X represents the major
version. We identify only releases of major versions, which
are versions where both Y and Z are zero (e.g., 2.0.0).
C. How to calculate hasBeenFixed
For each method, hasBeenFixed is calculated as follows.
step1 We collect the project repository and bug reports whose
status is FIXED.
step2 Bug reports are issued for previously discovered bugs,
and every bug report has an ID. In this experiment, A
commit whose commit message contains bug report ID isregarded as a bug-fixing commit. We search the repository
for the bug-fixing commits.
step3 Suppose that a line of method A is changed in a bugfixing commit. If the commit falls within the interval fromthe (n − 1)-th release (Rn−1 ) to the n-th release (Rn ),then method A is true for hasBeenFixed at Rn .
step4 When the above operations are performed for all bug
reports, methods that are not true for hasBeenFixed are
judged to be false for hasBeenFixed.

TABLE ITARGET PROJECTS
Project Name
lucene-solr(LUC)
wicket(WIC)
cassandra(CAS)linuxtools(LIN)
egit(EGI)
jgit(JGI)
eclipse.jdt.core(ECL)
poi(POI)

development period
6,967 days
5,860 days
4,239 days
4,197 days
4,021 days
4,029 days
7,062 days
6,824 days

releases
9
7
4
8
5
5
4
4

commits
34,31920,972
25,671
10,7336,473
8,014
24,804
10,421

bug reports
6,232
2,871
5,402
2,201
2,529
700
9,716
2,620

methods
72,35622,961
28,56728,644
8,082
14,09823,91034,218Percentage of buggy methods
2.3 %
2.5 %
5.6 %
1.2 %
7.2 %
1.1 %
7.9 %
3.6 %

HasBeen Is
Fixed
Buggy

At

Method𝑅1

initialize

False

True

…

…

…

…

𝑅𝑛

update

False

False

𝑅𝑛

getName

True

True

𝑹𝒆𝒄𝒂𝒍𝒍𝑿𝒉𝒂𝒔𝑩𝒆𝒆𝒏𝑭𝒊𝒙𝒆𝒅 ∩ 𝑿𝒊𝒔𝑩𝒖𝒈𝒈𝒚=
𝑿𝒊𝒔𝑩𝒖𝒈𝒈𝒚

𝑿𝒄𝒐𝒏𝒅𝒊𝒕𝒊𝒐𝒏 :𝑚𝑒𝑡ℎ𝑜𝑑𝑠 𝑤ℎ𝑜𝑠𝑒 𝑐𝑜𝑛𝑑𝑖𝑡𝑖𝑜𝑛 𝑖𝑠 𝑡𝑟𝑢𝑒

Collect Data Calculate hasBeenFixed and isBuggy

Evaluate hasBeenFixed

Fig. 1. procedure of the experiment for RQ1

At

Collect Data

MethodDependent
Variable

Independent
variables
Commits…

𝑅1

initialize

0

3

…

…

…

…

…

…

𝑅𝑛

update

0

1

…

𝑅𝑛

getNeme1

2…

Build Dataset

Build Model

𝑷𝒓𝒆𝒄𝒊𝒔𝒊𝒐𝒏
𝑹𝒆𝒄𝒂𝒍𝒍𝑭 − 𝒎𝒆𝒂𝒔𝒖𝒓𝒆

Evaluate Model

Fig. 2. procedure of the experiment for RQ2

D. How to calculate isBuggy
We used the implementation of literature [5] of an SZZ
algorithm [6] to calculate isBuggy as follows.
step1 The same as step 1 of the hasBeenFixed calculation.
step2 The same as step 2 of the hasBeenFixed calculation.
step3 Suppose that a line of method A is changed in a bugfixing commit. We identify the commit that inserted thechanged line with the “git blame”, and regard the commit
as a bug-inducing commit.
step4 If a bug-fixing commit for method A was made after
Rn and the bug-inducing commit for the bug was madebetween Rn−1 and Rn , then method A is true for isBuggy
at Rn .
step5 When the above operations are performed for all bug
reports, methods that are not true for isBuggy are judged
to be false for isBuggy.
V. E XPERIMENTAL R ESULTS
A. RQ1In RQ1, we investigate whether hasBeenFixed accurately
captures buggy methods. An overview of the experiment isshown in Figure 1.
Procedure. We conducted the following steps to answer
RQ1: we (1) collected the repositories and the bug reports ofthe target projects, (2) calculated isBuggy and hasBeenFixedfor each target method, and (3) evaluated the properness of
hasBeenFixed quantitatively.
Evaluation metric. We use recall defined below to evaluate
the properness of hasBeenFixed as a dependent variable forbuilding bug prediction models.
Recall =|XhasBeenF ixed ∩ XisBuggy ||XisBuggy |XhasBeenF ixed is a set of methods that are true for hasBeenFixed, and XisBuggy is a set of methods that are true
for isBuggy. A method called ”initialize” that is shown in
Figure 1 is true for isBuggy but false for hasBeenFixed at
R1 . So ”initialize” existing at R1 is an element of XisBuggy ,but not an element of XhasBeenF ixed . Suppose the evaluatedrecall is less than 0.5. In that case, we judge hasBeenFixed to
be improper as a dependent variable for bug prediction models
because hasBeenFixed models cannot predict more than halfof the buggy methods.
Results. The results of the experiment are listed in Table II.
Table II lists recall, which is the percentage of the buggy methods captured by hasBeenFixed, for each project. The recalls
are all significantly below 0.5. We conclude that hasBeenFixed
is improper as a dependent variable for bug prediction models
because hasBeenFixed cannot capture more than half of the
buggy methods.
B. RQ2In RQ2, we build hasBeenFixed models and isBuggy models
to compare their performance. An overview of the experimentis shown in Figure 2.
Procedure. We conducted the following steps to investigate
RQ2: we (1) collected the repositories and the bug reports ofthe target projects, (2) built datasets based on the collected
data, (3) built bug prediction models based on the datasets, and(4) evaluated the performance of the bug prediction models.
Dependent variable. We adopted hasBeenFixed and isBuggy as dependent variables and built regression models for
TABLE IIR ECALL OF HAS B EEN F IXED FOR IS B UGGY
Project
Recall

LUC WIC CAS LIN
0.22 0.22 0.36 0.16

EGI
0.34

JGI
0.16

ECL
0.43

POI
0.20

ALL
0.28

𝑆1

𝑇1

𝑆2

𝑆2

𝑇2

𝑆𝑛−1

𝑆𝑛−1

𝑆𝑛−1

𝑅4

・
・
・

Development
Releasehistory
𝑅3
(𝑅0 )
𝑅1
𝑅2

𝑅𝑛−1 𝑅𝑛

A pair oftraining dataset& test dataset

学習
用

・
・
・

𝑆𝑛−1

𝑇𝑛−1

4) We over-sample instances in Sk which are true for dependent variable to address class imbalance problem [7].
Machine learning algorithm. The Random Forest (RF) [8]
was adopted as a machine learning algorithm because RF has
been commonly used in the previous studies [2]–[4] and thetime cost for building models is small.
Hyperparameter tuning. Hyperparameter tuning is essential to build high-performance models [9]. We performed 10hours of hyperparameter tuning for each model.
Evaluation metrics. We adopted the following measures toevaluate prediction performance.

Fig. 3. overview of datasets for each projecteach of them. The definitions of the metrics are given in
section IV-C and IV-D, respectively.
Independent variables. We adopt the product and process
metrics defined by Giger et al. [2] as independent variables.
The metrics are listed in Tables III and IV.
Building a dataset. This study follows release-by-releasetechnique to calculate pairs of a training dataset and a test
dataset. Based on this technique, we can make n − 1 pairs of
training dataset Sk and test dataset Tk from a project that has
been released for n times, as shown in Figure 3. The pairs arecalculated as follows.
1) We calculate pairs of Dependent and independent variables (instances) using the development history between
Rk−1 and Rk for each release Rk (R0 is the beginningof the development).
2) We build test dataset Tk of the instances calculated fromthe development history between Rk−1 and Rk .
3) We build training dataset Sk of the instances calculated
from the development history before Rk−1 .
TABLE IIIINDEPENDENT VARIABLES ( PRODUCT METRICS )Metrics nameFanIn
FanOutLocalVar
Parameters
CommentRatio
CountPath
Complexity
execStmt
maxNesting

Description
Number of methods that reference a given method
Number of methods referenced by a given method
Number of local variables in the body of a method
Number of parameters in the declaration of a method
Ratio of comments to source code (line based)
Number of possible paths in the body of a method
McCabe Cyclomatic complexity of a method
Number of executable source code statements
Maximum nested depth of all control structures
TABLE IVINDEPENDENT VARIABLES ( PROCESS METRICS )Metrics Name
MethodHistories
Authors
StmtAdded
MaxStmtAdded
AvgStmtAdded
StmtDeleted
MaxStmtDeleted
AvgStmtDeleted
Churn
MaxChurn
AvgChurn
Decl
Cond
ElseAdded
ElseDeleted

Description
Number of times a method was changed
Number of distinct authors that changed a method
Sum of all source code statements added
Maximum StmtAdded
Average of StmtAdded
Sum of all source code statements deleted
Maximum of StmtDeleted
Average of StmtDeleted
Sum of stmtAdded - stmtDeleted
Maximum churn for all method histories
Average churn per method history
Number of method declaration changes
Number of condition changes over all revisions
Number of added else-parts over all revisions
Number of deleted else-parts over all revisions

P recision=

Recall=F − M easure=|T P ||F P + T P ||T P ||F N + T P |2 × Recall × precision
Recall + precision

TP (True Positive) is a set of methods that have been classified
as buggy by a model and actually buggy. FN (False Negative)
is a set of methods that have been classified as not buggy bya model and actually buggy. FP (False Positive) is a set of
methods that have been classified as buggy by a model andactually not buggy.
Results. The results of the experiment are listed in Table V.
The table shows the average prediction performance for boththe hasBeenFixed models and the isBuggy models.
The F-Measure of the models adopting isBuggy as a realistic
dependent variable is about half that of hasBeenFixed. We
conclude that isBuggy models’ performance is quite lower thanhasBeenFixed models and there are still issues in this research
area.
C. RQ3In RQ3, we investigate which machine learning algorithm
provides the best performance under realistic settings. The
configurations to build and evaluate models are the same as
RQ2’s except for dependent variable and machine learningalgorithm.
Dependent variable. We set isBuggy as the dependent
variable. The definition is given in section IV-D.
Machine learning algorithm. We test Random Forest (RF)and Deep Neural Network (DNN) [10] as machine learningalgorithms. DNN is selected because DNN can compute
expressive models even though they are not always better thanRF models in terms of prediction performance.
Results. The results of the experiments are listed in Table VI. The table shows the average prediction performancefor both RF and DNN. We conclude that RF models are better
than DNN models from the perspective of F-Measure.
TABLE VP REDICTION PERFORMANCE FOR DEPENDENT VARIABLES
HasBeenFixed
IsBuggy

Precision
0.33
0.15

Recall0.75
0.61

F-Measure
0.40
0.19

TABLE VI
PREDICTION PERFORMANCE FOR MACHINE LEARNING ALGORITHMS

RF
DNN

Precision
0.15
0.13

Recall0.61
0.64

F-Measure
0.19
0.18

F-measure

0.3
RF

0.25

DNN

0.2
0.15
0.1
0.05
0
2

3

4

5

Number of releases

6

7

Fig. 4. performance transition as the development history accumulates

D. RQ4
We follow the release-by-release technique to build datasets.
Under this technique, the size of the training dataset is proportional to the length of development history (the number of
releases). In RQ4, we investigate how prediction performanceof models built under the realistic settings changes as thedevelopment history accumulates. The configurations to build
models are the same as RQ3’s.
Results.
Figure 4 shows the average performance of models to
predict buggy methods at the release. In the results of theexperiment, there are two points worthy of special mention.
Firstly, the prediction performance decays as the development history accumulates. This may be because the buggy
methods’ features change just as the development history
accumulates. However, this is a hypothesis, and we will test
it in the near future.
Secondly, DNN models are more powerful at an earlystage of development, and RF models are more powerful at alater stage of development. This may be because the buggy
methods’ features change just as the development history
accumulates, and DNN models strongly learned the buggy
methods’ features at the early stage of development.
We conclude that DNN is better for building bug prediction
models under realistic settings at the second release, and RF
at subsequent releases.
VI. T HREATS TO VALIDITY.
Herein, we describe possible threats to our study.
Dependent variable. In this study, we use bug reports
whose status is FIXED to calculate hasBeenFixed and isBuggy.
On the other hand, the target projects may include latent bugs,
which are not exposed yet and any bug reports for them havenot been made yet. The presence of latent bugs may harm theperformance of bug prediction models built in this study.
Hyperparameter tuning. Hyperparameter tuning is essential to build high-performance models using machine learningalgorithms. In this study, we performed 10 hours of hyperparameter tuning for each model. If models are tuned for alonger time, higher performance models may be built.
SZZ algorithm. We used the implementation of literature [5] of an SZZ algorithm [6] to calculate isBuggy.
However, the accuracy of the SZZ algorithm still has room
for improvement [11], [12]. This problem may harm theperformance of bug prediction models built in this study.

VII. C ONCLUSION
This paper is the first step to build reliable bug prediction models under realistic settings. We first investigated theproperness of the metric hasBeenFixed as a dependent variable
for bug prediction models. Then, we built bug prediction models under realistic settings and evaluated their performance.
As a result, hasBeenFixed was able to capture only 28%
of buggy methods. Thus, hasBeenFixed is improper as adependent variable for bug prediction models. The F-Measureof the bug prediction models built under realistic settings
was only 0.19, and it turned out to be challenging to build
practical bug prediction models under realistic settings. From
those results, we can say that there are still some issues to besolved to build high-performance bug prediction models underrealistic settings.
We are currently trying to construct a new technique toimprove bug prediction models’ performance under realistic
settings. Specifically, we are considering investigating how
prediction performance changes if new independent variables
are introduced.
ACKNOWLEDGMENT
This work was supported by MEXT/JSPS KAKENHI
20H04166.
R EFERENCES[1] T. Britton, L. Jeng, G. Carver, P. Cheak, and T. KKatzenellenbogen,“Increasing software development productivity with reversible debugging,” accessed 2020-10-16, https://undo.io/media/uploads/files/Undo
ReversibleDebugging Whitepaper.pdf.
[2] E. Giger, M. D’Ambros, M. Pinzger, and H. Gall, “Method-levelbug prediction,” in International Symposium on Empirical Software
Engineering and Measurement, 2012, pp. 171–180.
[3] H. Hata, O. Mizuno, and T. Kikuno, “Bug prediction based on finegrained module histories,” Proceedings - International Conference on
Software Engineering, pp. 200–210, 2012.
[4] L. Pascarella, F. Palomba, and A. Bacchelli, “Re-evaluating methodlevel bug prediction,” in 2018 IEEE 25th International Conference on
Software Analysis, Evolution and Reengineering (SANER), 2018, pp.
592–601.
[5] M. Borg, O. Svensson, K. Berg, and D. Hansson, “Szz unleashed: An
open implementation of the szz algorithm - featuring example usage
in a study of just-in-time bug prediction for the jenkins project,” inProceedings of the 3rd ACM SIGSOFT International Workshop on
Machine Learning Techniques for Software Quality Evaluation, 2019,
pp. 7–12.
[6] J. Sliwerski, T. Zimmermann, and A. Zeller, “When do changes induce
fixes?” SIGSOFT Software Engineering Notes, vol. 30, no. 4, pp. 1–5,
2005.
[7] N. V. Chawla, Data Mining and Knowledge Discovery Handbook.
Springer, 2010, pp. 875–886.
[8] A. Liaw and M. Wiener, “Classification and regression by randomforest,”
in R news, vol. 2, no. 3, 2002, pp. 18–22.
[9] J. Bergstra and Y. Bengio, “Random search for hyper-parameter optimization,” Journal of Machine Learning Research, vol. 13, p. 281–305,
2012.
[10] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification
with deep convolutional neural networks,” in Proceedings of the 25th
International Conference on Neural Information Processing Systems,
vol. 1, 2012, pp. 1097–1105.
[11] C. Williams and J. Spacco, “Szz revisited: Verifying when changes
induce fixes,” in Proceedings of the 2008 Workshop on Defects in Large
Software Systems, 2008, pp. 32–36.
[12] D. A. da Costa, S. McIntosh, W. Shang, U. Kulesza, R. Coelho, and A. E.
Hassan, “A framework for evaluating the results of the szz approach for
identifying bug-introducing changes,” IEEE Transactions on Software
Engineering, vol. 43, no. 7, pp. 641–657, 2017.