1The Ghost Commit Problem When Identifying
Fix-Inducing Changes: An Empirical Study ofApache Projects
Christophe Rezk, Student Member, IEEE, Yasutaka Kamei, Senior Member, IEEE,and Shane McIntosh, Member, IEEE
Abstract—The SZZ approach for identifying fix-inducing changes traces backwards from a commit that fixes a defect to those commitsthat are implicated in the fix. This approach is at the heart of studies of characteristics of fix-inducing changes, as well as the popular
Just-in-Time (JIT) variant of defect prediction. However, some types of commits are invisible to the SZZ approach. We refer to theseinvisible commits as “Ghost Commits.” In this paper, we set out to define, quantify, characterize, and mitigate ghost commits that impact
the SZZ algorithm during its mapping (i.e., linking defect-fixing commits to those commits that are implicated by the fix) and filtering
phases (i.e., removing improbable fix-inducing commits from the set of implicated commits). We mine the version control repositories of
14 open source Apache projects for instances of mapping-phase and filtering-phase ghost commits. We find that (1) 5.66%–11.72% of
defect-fixing commits of defect-fixing commits only add lines, and thus, cannot be mapped back to implicated commits; (2)1.05%–4.60% of the studied commits only remove lines, and thus, cannot be implicated in future fixes; and (3) that no implicatedcommits survive the filtering process of 0.35%–14.49% defect-fixing commits. Qualitative analysis of ghost commits reveals that 46.5%
of 142 addition-only defect-fixing commits add checks (e.g., null-ness or emptiness checks), while 39.7% of 307 removal-only commits
clean up (unused) code. Our results suggest that the next generation of SZZ improvements should be language-aware to connect
ghost commits to implicated and defect-fixing commits. Based on our observations, we discuss promising directions for mitigation
strategies to address each type of ghost commit. Moreover, we implement mitigation strategies for addition-only commits and evaluate
those strategies with respect to a baseline approach. The results indicate that our strategies achieve a precision of 0.753, improving
the precision of implicated commits by 39.5 percentage points.
Index Terms—SZZ, fix-inducing changes, defect-fixing changes

F

1

I NTRODUCTIONO

VER the lifetime of evolving software projects, defects

are inadvertently introduced during initial development [7], refactoring [1], or when fixing other defects [43].
Identifying changes that are likely to induce future fixes
could save developers’ time and effort. Additionally, deepening our understanding of these fix-inducing changes andrecognizing recurring patterns can help teams to anticipate
when changes are likely to induce fixes in the future.
The SZZ approach for identifying fix-inducing changes[32] mines Version Control Systems (VCSs) and Issue Tracking Systems (ITSs) to trace a defect-fixing change back topotential fix-inducing changes that are implicated in the fix.
The SZZ approach starts by identifying defect-fixing commits
by matching a bug report from the ITS to the commit thatfixes it. Defect-fixing commits are then mapped to implicatedchanges by extracting the set of removed lines and tracing
them through the VCS to the commit(s) that last modified•••

Christophe Rezk is with the Department of Electrical and Computer
Engineering, McGill University, Canada.
E-mail: christophe.rezk@mail.mcgill.ca
Yasutaka Kamei is with the Principles of Software Languages Group(POSL), Kyushu University, Japan.
E-mail: kamei@ait.kyushu-u.ac.jp
Shane McIntosh is with the David R. Cheriton School of Computer
Science, University of Waterloo, Canada.
E-mail: shane.mcintosh@uwaterloo.ca

them. Finally, potential fix-inducing commits are filtered toeliminate those that should not be implicated in the fix (e.g.,
implicated changes that appeared after the defect creationdate). The surviving implicated commits are labelled as fixinducing commits [32].
While the SZZ approach plays a pivotal role in understanding and predicting fix-inducing changes, it is notwithout limitations. At its core, the SZZ approach relies onheuristics to handle noisy software repository data; however, there are commits that these heuristics cannot detect.
We refer to these invisible commits as ghost commits, which
impact (at least) two phases of the SZZ approach. First,
Mapping Ghosts are commits that cannot be detected whenconnecting defect-fixing commits to potential fix-inducing
ones. Second, Filtering Ghosts are defect-fixing commits forwhich no fix-inducing change survives the filtering phase.
We perform an empirical study of ghost commits in 14
open source projects from the Apache Software Foundation.
First, we quantify the frequency at which ghost commits
occur. Second, we characterize mapping and filtering ghosts
to better understand their properties. Third, we propose
and evaluate mitigation strategies that address the additiononly ghost commits (the most frequently occurring ghosttype). Finally, we study the types of maintenance activities
in addition-only ghost commits to compare with recent
work on intrinsic/extrinsic bugs [27]. More specifically, we
contribute the following definitions and observations:© 2021 IEEE. Author pre-print copy. The final publication is available online at: https://doi.org/10.1109/TSE.2021.3087419

2

Mapping Ghost 1: Defect-fixing commits with no implicated commits (MG 1)
Definition: Defect-fixing commits that do not remove previously existing lines of code.
Motivation: The SZZ approach maps defect-fixing commits
to fix-inducing commits by locating the most recent commit to change the lines that were removed by the fixing
commit [32]. However, the SZZ approach cannot map linesthat were added during a defect-fixing commit back to fixinducing changes. In theory, a defect-fixing commit may beentirely comprised of line additions; yet these these commits
may have been induced by prior changes. Hence, gaining abetter understanding of defect fixes that only add lines isimportant for those who adopt the SZZ approach.
Quantification: 5.66%–11.72% of defect-fixing commits inthe subject systems only add lines, with a median of 7.64%.
Characterization: MG 1 most often contain new Checks(44.6%), i.e., new if conditions or try-catch blocks. Often,
such checks were omitted by prior changes, which ideallywould have been implicated in the corresponding fixes.
Mapping Ghost 2: Commits that cannot be implicated infuture fixes (MG 2)
Definition: Commits that consist of line removals only.
Motivation: Commits that only remove lines cannot be
implicated by SZZ in future defect-fixing activity, since nolines remain in the codebase to which future activities canbe mapped. In reality, these commits may be fix-inducing,since the removal of an incorrect line (or set of lines) canwreak havoc on a software system. Studying the frequency
and characteristics of removal-only commits will show themagnitude of their potential impact on SZZ-based analyses.
Quantification: 1.05%–4.60% of commits in the subject systems contain line removals only, with a median of 2.68%.
Characterization: Cleanup of unnecessary code is the mostfrequently occurring reason (39.7%) for MG 2. Such cleanup
activities are not risk-free. For example, the infamous leftpad incident,1 which caused numerous N ODE . JS applications to fail was caused by the removal of code.
Filtering Ghost: Defect-fixing commits with no implicatedcommits that survive filtering (FG)Definition: Defect-fixing commits where all implicated fixinducing commits are removed by the filtering phase.
Motivation: Since a series of filters are applied to the set of
potentially fix-inducing commits, there may be defect-fixing
commits for which all potentially fix-inducing commits
are filtered out. Since these defect-fixing commits are notassociated with any fix-inducing commits, it is important
for those who adopt SZZ in research and practice to better
understand their frequency and characteristics.
Quantification: 0.35%–14.49% of defect-fixing commits inthe subject systems are FG, with a median of 5.46%.
Characterization: FG commits are most often related to theissue report date filter (35%). Deeper analysis suggests thatthe date filter is too aggressive because follow-up fixes areoften linked to the same issue ID as the initial work.
Mitigation
Strategies: We propose mitigation strategies for MG 1 com1. https://www.theregister.co.uk/2016/03/23/npm left padchaos/

mits, which apply data flow analysis to the (set of) identifier(s) in the added lines, and then apply SZZ to the lines inthe data flow path of the (set of) identifier(s).
Comparative Analysis: Our approach complements a
syntax-based baseline approach [29], which applies SZZ tothe lines within the closest surrounding code block. Indeed,both approaches implicate identical commits 21.1% of thetime and share at least one commonly implicated commit in
73.2% of the remaining cases.
Precision Analysis: We manually analyze the implicatedcommits of both approaches to assess whether they truly
could have been fix inducing. We find the precision of theControl Flow approach to be 0.753, while the precision of thebaseline approach is 0.358. Indeed, the data flow approach
is likely more precise because it avoids implicated benign
lines that appear within the surrounding code block.
Maintenance TypeTo better understand the types of maintenance being performed within ghost commits, we classify our sample of MG
1 commits as corrective, adaptive, or perfective, according to
the taxonomy introduced by Swanson [33]. We find that thevast majority (92.4%) of MG 1 commits are corrective, while
5.7% are adaptive, and 2.1% are perfective. These observations share similarities with the recent work of RodriguezPerez et al. [27], who found that the fixes for bugs areoften extrinsic, i.e., do not have a fix-inducing change. The
fixes that we labeled as corrective maintenance are intrinsic
in nature, while perfective and adaptive maintenance areoften extrinsic. This indicates that 7.6% of ghost commits
are extrinsic in nature, which falls within the range of rates
reported by Rodriguez-Perez et al. [27].
Our findings suggest that the SZZ approach for detecting fix-inducing changes currently overlooks a considerable
amount of commit activity. While these observed proportions are not exceedingly large, they still represent a sizeable
proportion of defect-fixing activity for which current SZZ
solutions do not apply. Future studies that rely on SZZshould explore context-aware enhancements to create moreaccurate and reliable SZZ data sets.

2

BACKGROUNDThis section describes the stages of the SZZ approachand how ghost commits can impact SZZ-based analyses.
Figure 1 contains an overview of the SZZ approach, which(i) merges issues and commits to identify defect-fixing commits (Section 2.1); (ii) maps defect-fixing commits back toprior changes that are implicated by the fix (Section 2.2); and(iii) applies a series of filters to remove implicated changes
that are unlikely to have induced the fix (Section 2.3).
2.1

Identifying Defect-Fixing Commits

The first stage identifies which commits in the VCS are
defect fixing. The assumption being that the occurrence ofa fix implies the existence of a defect prior to the fix.
(I 1) Merge Issues & Conflicts: To identify defect-fixing commits, VCS entries must be linked with issue reports in the
ITS. Issue reports in the ITS track the development activity
backlog for a project. These reports contain rich (meta)data

3
Identifying Defect-Fixing
Commits

Mapping(I 1)
Merge Issues& Commits(M 1)
Map BugFixing
Commits to
ImplicatedChangesSZZ Input
Data

GC 0

Filtering

SZZ
MappedData(F 1)
Apply Issue
Report Date
Filter(F 2)
ApplyContent
Filters

SZZ
Filtered
Data 1SZZ
Filtered
Data 2(F 3)
Apply
Suspiciousness
Filters

FG

MG 1,2

Fig. 1. An overview of the phases of the SZZ approach. Mapping Ghosts (MG 1, MG 2) are identified in the Mapping phase, while Filtering Ghosts(FG) are identified in the Filtering phase. These phases are described in Sections 2.1–2.3.
Data Extraction

ApacheOS Projects

Issue
Tracking
SystemVersionControl
System(DE 1)
Extract Issue
Properties(DE 2)
Extract
CommitProperties

IssueID Type RepDate ...
1
2

SZZ Output
Data

...

CommitID IssueID Date ...
1
2...

Data Analysis

SZZ(DE 3)
Remove noncode changes(SZZ)
Apply SZZ
Algorithm(DA 1)
Analyze GC
Frequency

GC

Freq

MG 1

7.64%

MG 2

2.68%

FG 1

5.46%

Ghost Commit
Categories

MG 1,2(DA 2)
Analyze GC
Root Cause

FG

Fig. 2. An overview of the Quantification and Characterization phases of our case study: extracting the data needed for SZZ, applying SZZ, andanalyzing the ghost commits detected. These phases are described in Section 4.2 and Section 4.3.

about development tasks, including a Type field, which
may be “Defect” or “Enhancement,” for example.
In a nutshell, commits that are linked to issue reportsof type “Defect” are considered to be defect fixing. Unfortunately, the links between VCS and ITS entries are
not explicitly enforced by either tool by default. Recent
integrated toolsets, such as GitLab, offer workflows2 thatenforce linking between issues and commits; however, forprojects that have not fully adopted a toolset like GitLab,
it is a common practice for developers to manually record
links between commits and issue reports in commit messages. For example, commit ae90791 from the P IG project
includes the message “PIG-5118 Script fails with Invalid dag
containing 0 vertices rohini” to indicate that the commit
is associated with the issue report PIG-5118. Thus, toidentify defect-fixing commits, one must identify the VCSITS linkage practices of the subject systems, then recover the
VCS-ITS links using repository mining scripts.
Ghost Commit 0 (GC 0): Only the VCS commits thathave a recovered link to an ITS record of type “Defect” areincluded in the SZZ data set. Since link recording practices
are rarely enforced, developers may omit necessary links
without receiving a warning from the VCS or ITS. Thus, SZZ
implementations may miss defect-fixing commits wherelinks were omitted. We refer to such missing defect-fixing
commits as Ghost Commit 0 (GC 0).
GC 0 has been defined and studied in prior work [3,21, 39]. Bird et al. [3] report that linkage bias in datasetscompromises the validity of software models built using
those datasets. Nguyen et al. [21] find that linkage bias indatasets exists even when strict guidelines are enforced on
the development process. Wu et al. [39] propose Relink, a
VCS-ITS link recovery tool to rebuild missing links andmitigate linkage bias. Given that GC 0 is already wellunderstood, we do not investigate it further in this paper.

2. https://docs.gitlab.com/ee/topics/gitlab flow.html

2.2

Mapping

After VCS-ITS links have been recovered, commits thatare implicated in defect-fixing commits can be identified.
This mapping step produces a database, which stores links
between defect-fixing and potential fix-inducing commits.
(M 1) Map Defect-Fixing Commits to Implicated Changes:
For each defect-fixing commit, its removed lines are selectedusing the diff command. Next, the parent commit(s) ofeach removed line are identified using the blame command.
Note that modifying a line registers as a removal and an
addition. Thus, analyzing removed lines covers cases whencode is removed or modified.
Mapping Ghost 1 (MG 1): Since the mapping step traceslines that previously existed, defect-fixing commits that do
not remove or modify lines cannot be mapped to implicatedcommits. We refer to defect-fixing commits that do not remove or modify lines as Mapping Ghost type 1 (MG 1). For
example, commit 4adc8e43 from the A CTIVE MQ project
fixes a defect by interrupting the socketHandlerThread
to cleanly shut down an embedded broker.
Mapping Ghost 2 (MG 2): Invoking the blame command on removed lines cannot implicate past commitsthat only remove lines. However, commits that remove thewrong lines can induce future fixes. We refer to commits that
do not add any new lines of code as Mapping Ghost type 2(MG 2). For example, in commit c10e8d24 from the H BASE
project, the removal of the createWriterInTmp method
may lead to a defect fix in the future.
2.3

Filtering

Next, a series of filters are applied to remove commitsthat could not or are unlikely to have led to the future fix.
This filtering stage reduces the sets of implicated commits
to those that are likely to be fix inducing.
(F 1) Apply Issue Report Date Filter: Implicated commitsthat appear after a defect has been reported are unlikely
3. https://github.com/apache/activemq/commit/4adc8e4/
4. https://github.com/apache/hbase/commit/c10e8d2/

4

to have induced the fix. To mitigate such noise, researchers
apply filters to discard such implicated commits (f1 ) [32]. To
do so, we use the --after:<date> flag of the blame command,
where <date> is the defect creation date. However, if nomodifications were made to a line after the specified date,the ∧ character is prepended to the output.
(F 2) Apply Content Filters: Implicated commits that modify whitespace or comments do not alter system behaviour
and are unlikely to induce a future fix. To mitigate suchnoise, researchers apply content filters to ignore implicatedcommits that update comments (f2 ) or whitespace (f3 ) [14].
Routine maintenance updates often modify a large number of files or lines of code (e.g., updates to coding style).
Such commits are another source of noise in SZZ data. To
mitigate the impact of this, researchers often filter out largecommits. For example, McIntosh and Kamei [16] filter out
commits that change more than 100 files or 10,000 lines (f4 ).
(F 3) Apply Suspiciousness Filters: Commits that fix a large
number of issues or induce a large number of fixes add noise
to SZZ data. Developers routinely address issues one at atime, which is why multiple issues being fixed by a single
commit is suspicious. Similarly, a commit that induces alarge number of future fixes is suspicious, since it is unlikelythat one change would be so problematic.
To filter out these suspicious commits, da Costa et al. [7]propose a framework. Their implementation of the framework for Apache projects uses the project-specific thresholdsof the upper Median Absolute Deviation (MAD) [9] of thenumber of issues that a commit fixes (f5 ) and the upper
MAD of the number of fixes a change induces (f6 ).
Filtering Ghost (FG): It is possible that, for a given
defect-fixing commit, no implicated commits survive thefiltering stages (f1 –f6 ). We refer to these defect-fixing commits as Filtering Ghosts (FGs). FGs are problematic because
no implicated commits can be associated with them. Thus,
models that are trained using SZZ data will not be able toidentify the commits that induce them.

3

R ELATED W ORK

Fix-inducing changes have been the subject of considerable research. Since teams have limited resources, identifying changes are likely to be buggy can help with timeand effort allocation. The SZZ approach [32] plays a crucialrole in such allocation efforts. Below, we present the related
work on fix-inducing changes and the limitations of SZZ.
3.1

Fix-Inducing Changes

The SZZ approach has been used to study properties of
fix-inducing changes in several settings. For example, theseminal paper [32] used SZZ to study the day of the week
when fix-inducing changes tended to appear. Eyolfson etal. [8] used SZZ to study the hour of the day when fixinducing changes tended to appear. SZZ has also been used
to detect and characterize defect-fix patterns [22], to study
how long defects survive [4, 12], and to study the links
between fix-inducing changes and (a) code authorship [25],(b) code clones [24], and (c) faulty defect fixes [41].
SZZ is also at the core of Just-In-Time defect prediction,a term coined by Kamei et al. [11], which describes a

popular variant of change-level defect prediction. Mockus
and Weiss [18] used various change properties to predictrisky code changes at Bell Labs. Kim et al. [13] and Kamei
et al. [11] expanded upon the set of metrics, including thosethat were computed using VCS and ITS data, and analyzed abroader set of projects, including swaths of open source andproprietary projects. Kononenko et al. [15] further expanded
upon the metric set to include code reviewing data. JIT
defect prediction has been deployed in industrial settingsat Cisco [34], Blackberry [30], and Avaya [18] to name a few.
In recent years, as improvements to machine learningtechnology have appeared, JIT defect prediction has also
been improved. To address the cold-start problem for software analytics, Kamei et al. [10] studied the efficacy of crossproject JIT defect prediction. Yang et al. [42] propose Deeper,
which uses deep learning techniques to train JIT models.
While the prior work has made important contributions,
it is built upon the underlying SZZ approach, which classifies changes as fix-inducing or clean. In this paper, we focus
on risks to the completeness of SZZ data, quantifying andcharacterizing that risk in 14 open source ASF projects.
3.2

Limitations of the SZZ ApproachThis paper is not the first to propose improvements to
the SZZ approach. Table 1 presents an overview of pastwork studying SZZ limitations and improvements. Kim
et al. [14] introduced an improvement to SZZ that uses
annotation graphs as opposed to the annotate command.
Moreover, the approach filters out style changes. Williams
and Spacco [37, 38] proposed adding weights to the SZZmapping technique, as well as using the DiffJ5 tool to disregard formatting changes when comparing code files. Neto
et al. [20] proposed an SZZ implementation that ignores
refactoring changes, since those are unlikely to introduce
defects. Contributing to this line of work, we propose several language-aware improvements to SZZ (see Section 6) toimprove the completeness (recall) of SZZ-generated data.
Past work has also raised concerns about the risks ofmodelling defect data using SZZ. For example, da Costa etal. [7] evaluated several variants of the SZZ approach using
suspiciousness filters based on the earliest defect appearance,
the future impact of a change, and the realism of defect introduction. This work differs from that of da Costa et al. in that we
focus on increasing the recall of SZZ data by defining ghostcommits, and studying strategies to capture them.
Rodriguez-Perez et al. [26, 27, 28] introduce the conceptof extrinsic defects to describe defects that should not havean implicated code change. There is an interesting interplay
between the extrinsic/intrinsic classification proposed byRodriguez-Perez et al. [26], which focus on the nature of
the defects being fixed, and the ghost commit concept we
propose in this work, which focuses on the commits thatcurrently slip through the mapping and filtering stagesof the SZZ algorithm. We study the relationship between
extrinsic/intrinsic defects and ghost commits in Section 6.

4

S TUDY D ESIGN

The goal of our study is to better understand the extent
to which the ghost commit problem impacts SZZ data of
5. http://www.incava.org/projects/diffj

5

TABLE 1
An overview of past work addressing SZZ Limitations.
SZZ Limitation Addressed

Paper

Contribution

Kim et al. [14]Non-semantic changes are identified as defect-fixing
Use an automated approach with annotation graphs
Williams and Spacco [38] Annotation graphs are imprecise at tracking lines
Use a weighted line mapping approach to track unique lines
Non-semantic changes are inaccurately identified
Use the DiffJ tool to ignore only non-semantic changes
Neto et al. [20]Refactoring changes are flagged as defect-inducing
Introduce a refactoring-aware SZZ implementation
Da Costa et al. [7]Techniques to evaluate SZZ-generated data are limited Introduce a framework to evaluate SZZ-generated data
Perez et al. [26, 27, 28]
Not all defects are introduced by a specific commit
Introduce an approach to explore different causes of defects
This paper
Certain (ghost) commits are overlooked by SZZQuantify, characterize, and mitigate ghost commits

Implicating(M1)
Apply Control
Flow Analysis
MG 1
Sample(M2)
ApplyBaseline
ApproachControl Flow
ImplicatedCommits

Baseline
ImplicatedCommits(M3)
Perform
comparative
analysis

Identical 21.1%
Common 73.2%(M4)
Perform
precision
analysis

Control Flow 75.3%(MT1)
Perform
maintenancetype analysis

Corrective 91.7%
Adaptive 5.6%
Perfective 2.8%

ApproachBaseline

Prec
35.8%

Fig. 3. The mitigation and maintenance type analyses performed for MG
1 commits

real software projects. In working towards these goals, weformulate three concrete research objectives:
Objective 1: Quantification. Our first objective is to measure how often ghost commits occur. While Section 2
defines ghost commits, it is unclear if they occur often
enough to be of concern for users of SZZ.
Objective 2: Characterization. Our second objective is to
study the properties of ghost commits. Specific development activities may be disproportionately responsible for generating ghost commits. Knowing these tendencies may help researchers to propose solutions toand practitioners to avoid generating ghost commits.
Objective 3: Mitigation. Our final objective is to propose
strategies to mitigate the ghost commit problem. Ideally, these will be extensions to the SZZ approach itself.
To tackle these objectives, we conduct an empirical studyof repository data from open source projects. Figure 2 provides an overview of our study approach for Objectives 1
and 2, and Figure 3 provides an overview of our approachfor Objective 3. Below, we present our rationale for selecting
our subject projects (Section 4.1), as well as our approaches
to data extraction and analysis (Sections 4.2 and 4.3), andghost commit mitigation (Section 4.4).

4.1

Corpus of Software Projects

We study 14 projects from the Apache Software Foundation (ASF). Similar to Munaiah et al. [19], we identify criteriathat must be satisfied by our subject projects.
Criterion 1: Replicability. We want to ensure that ourstudy can be replicated (and extended) by researchers.
To reduce barriers to access of the raw data, we selectsubject projects whose software repositories (VCS, ITS)
are freely and openly available for download. To further

enable replicability, we have made our data extraction
and analysis scripts publicly available.6
Criterion 2: System Size and Activity. We want to study
large, actively maintained projects, since such projects
stand to benefit the most from SZZ analyses.
Criterion 3: VCS-ITS Linkage. Like all SZZ-based studies, a key concern is the quality of the links between
commits (VCS) and issue reports (ITS). Thus, we selectsoftware projects where a large proportion of commits
are explicitly linked to issue reports.
To satisfy Criterion 1, we select projects from the Apache
Software Foundation (ASF). The ASF provides resources tosupport the development of software for the public good.
The VCS7 and ITS8 of Apache are publicly available. Selecting ASF projects for analysis satisfies many of Munaiah
et al.’s Community, Documentation, and License criteria for
selecting engineered software repositories for analysis.
To satisfy Criterion 2, we select 14 of the most actively developed ASF projects for analysis. These 14 subject
projects have been studied in prior work [7, 17]. Table 2
provides an overview of the 14 subject projects, and showsthat the size of the projects ranges between 0.087 MLOC
and 4.3 MLOC. Satisfying Criterion 2 also satisfies Munaiah
et al.’s History criterion. Moreover, the selected ASF projects
include unit tests (Munaiah et al.’s Unit Tests criterion) and
use a Cloudbees (Jenkins) instance to perform continuous
integration9 (Munaiah et al.’s CI criterion).
To ensure that Criterion 3 is satisfied, we study the
VCS-ITS linkage practices of ASF projects. We find thatASF developers tend to record the issue ID within commit
messages following a clear pattern. For example, below is
the commit message that accompanies commit ae90791
from the Apache P IG project:“PIG-5118 Script fails with Invalid dag [...]”We use regular expressions to extract these issue ID references. Thus, we compute the VCS-ITS linkage rate, i.e., thepercentage of commits that are associated with issue reports.
Table 2 shows that we select a VCS-ITS linkage rate
threshold of 50%. This threshold helps to satisfy Munaiah
et al.’s Issues criterion. A sensitivity analysis shows that athreshold of 60% would result in five fewer projects, whilea threshold of 40% would only add one project. Thus, we
believe that the impact of this threshold choice is minimal.
6. http://doi.org/10.5281/zenodo.4558395
7. https://git.apache.org/
8. https://issues.apache.org/
9. https://builds.apache.org/

6

4.2

Data Extraction(DE 1) Extract Issue Properties: The ASF uses the JIRA ITS.
We use the JIRA REST API10 to extract the identifier (IssueID), type (Type), and reported date (RepDate) for eachreferenced issue of the subject projects.
(DE 2) Extract Commit Properties: We first collect a copy of
the Git VCS archive of each subject system. In the past, theASF used Subversion as its primary VCS,11 while providing
read-only Git mirrors for convenience. Nowadays, several
Apache projects now use Git as their primary VCS.
Next, we extract (meta) data about the commits thatappear on the trunk branch. We focus on the trunk branchbecause it is the main development branch in ASF projects.
For each commit on a trunk branch, we extract three
key properties: (1) the CommitID; (2) the commit message(to detect whether there is an IssueID encoded within it, andextract it if it exists); and (3) the list of modified files.
(DE 3) Remove Non-Code Changes: Since we want to study
defects in subject system behaviour, we focus our analysis
on commits to source code files. Thus, we filter out commitsthat only modify .txt, .xml, and CHANGELOG files.
4.3

Data Analysis(DA 1) Analyze GC Frequency: To analyze the frequency ofeach ghost type, we compute: (1) the proportion of additiononly defect-fixing commits among all of the defect-fixing
commits (MG 1); (2) the proportion of removal-only commits among all of the commits (MG 2); and (3) the proportion of defect-fixing commits whose fix-inducing commits
are entirely discarded by the filtering phase (FG).
(DA 2) Analyze GC Root Cause: We then set out to better
understand the types of changes that are associated witheach type of ghost commit. To determine the types ofchanges that appear in ghost commits, we apply an open
coding approach [5] to classify examples of each type ofghost commit. Since an understanding of the context of thestudied system is required to code changes, we choose to
select one project from our corpus of studied projects toperform open coding on rather than a broad sample ofchanges from several projects. We analyze a project witha “typical” rate of ghost commits, i.e., a project with arate close to the median rate in our corpus. To obtain the
categories used to categorize the MG, the first author independently classified the MG, defined the taxonomy based on
observed patterns, and shared this taxonomy with the otherauthors, who provided feedback. To estimate the degreeof subjectiveness in our classification process, the second
author independently classified the same ghost commits
using the revised taxonomy. We then use Cohen’s Kappa,
a coefficient that measures inter-rater reliability, to computean agreement score between the codes of the first andsecond authors [6]. Finally, cases where coders disagreed
were discussed in a follow-up meeting until a consensus
could be reached. In those meetings, the third author would
cast the tie-breaking vote if necessary.
10. https://developer.atlassian.com/server/jira/platform/
rest-apis/
11. http://www.apache.org/dev/version-control.html

4.4

Mitigation Analysis(M 1) Apply Data Flow Analysis: For MG 1, we apply the
mitigation strategy (see Section 6.1) to the added lines to
identify a list of lines to be mapped (and filtered) by SZZ.
If the commit that last modified a line is a refactoringchange, we continue to trace backwards to find the commit
that last made a non-refactoring change to the line, following the RA-SZZ approach introduced by Neto et al. [20].
We do not currently have an approach for handlingbreaking changes [35]; however, they did not present an issue for us in our analysis. We suspect that this issue was notprevalent because we focus our analysis on recent changes.
Indeed, Tufano et al. find that breaking changes tend to be
most prevalent in old commits, where dependencies on old
versions of libraries and tools may present issues.
(M 2) Apply Baseline Approach: The baseline approach wecompare to is a modified version of A-SZZ, introduced by
Sahal and Tosun [29]. A-SZZ considers the lines between“the first left bracket above and the first right bracket below”
the added lines as a code block, then runs the log commandon all the functional lines of the block to implicate commits.
In cases where the syntactic A-SZZ definition of a code block
cuts into another method or loop, we instead consider theblock to be the first enclosing method or loop.
(M 3) Perform Comparative Analysis: After applying boththe Data Flow Analysis and the Baseline Approach, we
compare the sets of potentially bug-inducing commits implicated for each defect-fixing commit to count the instances where both techniques yielded the same results, andwhether there are implicated commits in common for thecases where they yielded different results.
(M 4) Perform Precision Analysis: For each pair of fixinducing and defect-fixing commits from both approaches,
we take a deeper look at the fix-inducing commit to assess
whether it should have been implicated in the fix. Since
we are not subject matter experts in the studied systems,
we take a conservative approach to labelling the implicatedcommits. We assume that implicated commits are correctly
labelled (i.e., true positives) unless it is evident that they
could not have contributed to the bug (i.e., false positives).
tp
We use this data to compute the precision score ( tp+fp ) for
each technique. Note that we exclude New Entity changes
from this analysis due to their inherent ambiguity.
4.5

Maintenance Type Analysis(MT 1) Perform Maintenance Type Analysis: To study theinterplay between intrinsic/extrinsic defects [26] and ghostcommits, we classify ghost commits by Swanson’s maintenance categories [33], where: (1) corrective maintenancerectifies a processing, performance, or implementation failure; (2) adaptive maintenance responds to changes in thedata or processing environments; and (3) perfective maintenance improves non-functional properties (e.g., performance, maintainability). Corrective maintenance maps onto
the concept of intrinsic defects, while adaptive and perfective maintenance are likely due to extrinsic defects.

5

Q UANTIFICATION AND C HARACTERIZATION

Below, we present the quantification and characterization results for the three types of ghost commits.

7

TABLE 2
An overview of the subject projects and Ghost Commits’ frequency.
Project

Size Commits Issues Linkage MG 1 MG 2
Issues
(%)
(%)
(LOC)
Commits

FG(%)

A CTIVE MQ 0.087 mil
A MBARI
3.8 mil
C AMEL
2.6 mil
C AYENNE 0.563 mil
D ERBY
1.4 mil
H BASE
1.3 mil
H IVE
2.6 milJ ACKRABBIT 4.3 mil
K ARAF
0.281 mil
O PEN JPA 0.837 mil
P IG
0.581 mil
Q PID
0.246 mil
S LING
1.1 mil
T HRIFT
0.466 mil

9,945 5,138
23,901 23,346
31,726 17,072
5,897 3,248
8,180 6,791
14,099 12,701
12,548 12,132
8,517 5,563
7,042 4,689
4,861 3,231
3,152 2,932
14,181 7,659
21,668 12,168
5,305 3,340

51.66% 7.55% 2.35% 5.58%
97.68% 7.73% 1.55% 13.59%
53.81% 9.44% 2.14% 6.64%
55.08% 5.91% 2.70% 5.43%
83.02% 7.74% 4.60% 9.41%
90.08% 7.42% 2.18% 6.43%
96.68% 9.23% 1.67% 14.49%
65.32% 7.33% 3.36% 0.35%
66.59% 10.45% 1.80% 1.77%
66.47% 6.56% 1.72% 9.02%
93.02% 8.57% 1.05% 5.49%
54.01% 7.42% 3.56% 2.13%
56.16% 5.66% 2.38% 1.34%
62.96% 11.72% 2.68% 1.06%

Mean
Median12,216 8,572
9,231 6,177

70.90% 8.05% 2.41% 5.25%
65.89% 7.64% 2.68% 5.46%

1.4 mil
997 K

TABLE 3The categories of MG 1. Percentages are of the overall sample unless
indented to indicate category values. Definitions appear in Section 5.1.
Category

Number Percentage

New Entity
New Class
New SubClassNew Interface
Check
If Check
Null Check
Try/Catch Check
Configuration
Override
Logging
Expanding Class

6 4.2%
1
16.7%
4
66.7%
1
16.7%
66 46.5%
54
81.8%
25
37.9%
17
25.8%
7 4.9%
18 12.7%
12 8.5%64 45.1%

TABLE 4The categories of MG 2. Percentages are of the overall sample unless
indented to indicate category values. Definitions appear in Section 5.2.
Category
Cleanup
UnusedUnused Method
Unused Configuration
Unused Dependency
Unused Class
Unused Variable
Redundant
Duplicate
DeprecatedRenaming
Refactoring
Dead CodeEntire File
Undo
Revert Entire Commit
Partial Revert
Update Settings
Configuration
Framework
LoggingDocumentation
Fix Race Condition
Miscellaneous

Number Percentage
122 39.7%
38
31.4%
9
23.7%
4
10.5%
7
18.4%
9
23.7%
9
23.7%
4
3.3%
14
11.5%
8
6.6%
3
2.5%
6
4.9%
3
2.5%
46
37.7%
35 11.4%
14
40.0%
21
60.0%
25 8.1%
20
80.0%
5
20.0%
25 8.1%
11 3.6%
5 1.6%
84 27.4%

5.1

Defect Fixes with No Implicated Commits (MG 1)

Quantification: Mapping Ghost 1 is not uncommon among
the studied projects. Table 2 shows that 5.66%–11.72% of all
defect-fixing commits are of type MG 1 (i.e., contain only
added lines), with a median of 7.64%. Current implementations of SZZ cannot map MG 1 defect-fixing commits backto commits that are implicated in the fix.
Characterization: To gain insight into the characteristics of
MG 1 defect fixes, we manually code changes from theA CTIVE MQ project. We select A CTIVE MQ because its proportion of MG 1 fixes is 7.55%, which is closest to the medianvalue (7.64%). After the first and second authors initiallyclassify all 148 of the instances of MG 1 in A CTIVE MQ,
we obtained an agreement score of κ = 0.314, which isconsidered to be fair agreement. In our follow-up meetings,
several patterns of disagreements emerged, which werelargely due to initial misunderstandings of the classification
types. After the meetings, the coders came to a consensus
on 145 commits, only requiring a tie-breaking vote for threecommits. This suggests that the true agreement score ismuch greater than the one reported above.
Table 3 provides an overview of the categories of MG 1
that we discovered. New Entity changes involve either the
addition of a New Class, New Subclass, or a New Interface.
Check changes consist of branching upon checking certain
conditions using if statements, try/catch statements, and/or
assertions. We also record which of these changes are checks
for special values like null. Configuration changes are thosethat update settings, such as changes to .properties files.
Override changes involve overriding a method inheritedfrom a superclass in a subclass. Logging changes add or edit
code being used to log execution behaviour. Expanding Classchanges add new functionality to an existing class.
Within our sample, we observe that Check-type changes
occur the most. Of these, most (81.8%) consisted of if
branching statements. Most often, these commits would add
a check for null to improve the robustness of a method. For
example, commit d92d3a8 fixes issue AMQ-3782 by adding
a check for null of reconnectTask.
5.2

Commits that Cannot be Mapped to Fixes (MG 2)

Quantification: Although MG 2 commits are less common
than MG 1 commits, MG 2 commits still account for a considerable proportion of the change activity. Table 2 showsthat 1.05%–4.60% of all commits are of type MG 2 (i.e.,
contain removed lines only), with a median of 2.68%.
Since MG 2 commits do not add lines that future changes
can improve upon, current SZZ implementations cannot
implicate MG 2 commits in future fixes. Similar to MG 1,
extensions to the SZZ algorithm that enable implicating MG
2 commits would likely improve the recall of the approach.
Characterization: To characterize MG 2 commits, we manually code commits from the H BASE project. We select H BASE
because its proportion of MG 2 commits is 2.18%, whichis the closest to the median (2.68%). After the first andsecond author independently classified all 307 instances of
MG 2, the initial agreement score was κ = 0.567, which isconsidered to be moderate agreement. During the followup meeting, all disagreements were resolved due to clarifications without requiring a tie-breaking vote.

8

TABLE 5
The filtering ghosts removed by each step of the filtering process.

No Filter

Issue Date Filter
Date (f1 )BFC

Content Filters
Comments (f2 )Whitespace (f3 )Size (f4 )BFC

FG Drop %BFC

FG Drop %BFC

FG Drop %BFC

FG Drop %A CTIVE MQ
A MBARIC AMELC AYENNE
D ERBY
H BASEH IVE
J ACKRABBITK ARAFO PEN JPA
P IG
Q PID
S LING
T HRIFT

1,720
9,904
2,635
368
1,520
4,181
4,631
1,112
847
776
1,057
2,206
2,093
1,226

1,644
9,892
2,499
367
1,412
4,010
4,622
1,110
844
716
1,012
2,200
2,089
1,225

79.1%
2.75%
77.71%
4.76%
75.52%
63.57%
1.34%
50%
20%
85.71%
77.59%
12.77%
14.29%
7.69%

1,644
9,887
2,499
364
1,412
4,006
4,619
1,110
844
716
1,010
2,198
2,087
1,223

0
1.14%
0%
14.29%
0%
1.49%
0.04%
0%
0%
0%
3.45%
4.26%
7.14%
15.38%

1,628
9,797
2,465
357
1,394
3,980
4,592
1,109
833
711
1,006
2,181
2,072
1,216

16.67%
20.59%
19.43%
33.33%
12.59%
15.38%
4.02%
25%
73.33%
7.14%
6.9%
36.17%
53.57%
53.85%

1,624
9,467
2,460
347
1,377
3,912
3,960
1,108
832
706
999
2,159
2,065
1,213

4.17%
75.51%
2.86%
47.62%
11.89%
40.24%
94.19%
25%
6.67%
7.14%
12.07%
46.81%
25%
23.08%Median

1,620

1,528

35%

1,528

0.59%

1,511

20.01%

1,500.5

24.04%

Table 4 provides an overview of the categories of MG
2. Cleanup changes remove code that is not needed. As
the name suggests, Deleting Entire File changes remove files
from the VCS. Unused changes remove artifacts and code
elements that are unused. Other types of Cleanup changes
include Redundant, Duplicate, Deprecated, Renaming, Refactoring, and Dead Code. These Cleanup changes may induce future fixes if they are too aggressive, removing code that was
still needed. Undo changes either Revert Entire Commits orPartially Revert a commit. Revert Entire Commit changes areunlikely to be fix-inducing, since these changes usually refer
to undoing commits that were initially problematic. However, Partial Revert changes may induce future fixes, sinceundoing part of a commit is likely done by hand and may
be prone to error. Updating Settings changes are the same as
Configuration changes described under MG 1. Logging andDocumentation changes are unlikely to induce future fixes,
since they do not impact core system functionality. Race
Condition changes, such as attempts to resolve deadlocks,
may induce future fixes due to incomplete or incorrect fix
attempts. We also consider Miscellaneous changes as unlikely
to induce future fixes.
Table 4 shows that the largest proportion of MG 2 commits are Cleanup. Most often, these commits remove codethat is no longer needed. For example, commit e5123cc removes the startCatalogJanitorChore method, which
is believed to be unused.
Prior work [31, 40] studied revert commits in a variety ofopen source and proprietary settings, reporting that 1%–5%of commits are revert commits. We find a larger proportion(11.4%) of commits undo prior commits in our sample ofMG 2 commits. We suspect that this is because 60% of our
undo commits are not explicitly labelled as reverted (i.e.,they were not produced using the git revert command).
Since the prior work focuses on explicitly labelled revert
commits, the most comparable figure in our study would be
the 4.56% (= 40% × 11.4%) of MG 2 commits that Revert
Entire Commits, which falls within the range of prior work.
This suggests that the scope of revert commits is broader

Bug ReportCreatedIC

Comment on
Bug Report

BFC

Sep 3

Sep 19Sep 23

Sep 25

Fig. 4. An example of a FG.

than previously analyzed. An analysis of non-explicit revert
commits might be an interesting direction for future work.
We observe that 1.95% of MG 2 commits are refactorings.
This rate is similar to that of Tufano et al. [36], who observedthat only 1.8% (= 9% × 20%) of removed instances of code
smells were removed through refactorings.
We also observe that 9.4% of MG 2 commits remove
entire classes and 9.4% of MG 2 type commits remove
entire methods. In the context of removal of Self Admitted
Technical Debt (SATD), Zampetti et al. [44] found that on
average, 30.2% and 14.0% of SATD is removed by deleting
entire classes and methods, respectively. We attribute thisdifference in the rates of entire class and method removals
to our differing study contexts (i.e., all removal-only commits vs. SATD-removing commits). However, we believethat the results are complementary enough to indicate thatlarge removal operations occur frequently enough to justify
dedicated analysis approaches.
5.3 Defect-Fixing Commits with No Implicated Commits That Survive Filtering (FG)
Quantification: Filtering Ghosts make up a considerable
proportion of changes among the studied projects. Table
2 shows that 0.35%–14.49% of defect-fixing commits areof type FG (i.e., have all of their implicated fix-inducing
commits filtered out), with a median of 5.46%. Current
implementations of the SZZ algorithm filter out all commitsthat are implicated by FG defect-fixing commits. Extensions
to the SZZ algorithm that enable pinpointing other fixinducing commits that could lead to FG defect fixes would
again improve the recall of the approach.

9Characterization: To characterize FG commits, we computehow many FG defect fixes are being removed by each filter
f1 –f4 (none of the FG commits in the studied projects are
due to f5 or f6 ). Table 5 shows that across all studied
projects, the largest proportion of FG commits is due to thedate filter (f1 ), with a median of 35%.
To investigate why so many FG commits are being
removed by the date filter, we conduct a deeper inspection.
We initially suspected that many of these FG would be due
to inconsistencies in time-keeping between the VCS and ITS;
however, this was not the case. Figure 4 provides an example of a FG (commit 3a356b5) from the P IG project. TheSZZ approach implicates one potential fix-inducing commit
IC (f22c685) in the future fix in commit BFC. However,
the issue report that documents the defect (PIG-942) thatis associated with BFC was created on Sept. 3rd , while thepotentially implicated commit IC appeared later on Sept.
19th . Thus, IC is filtered out of the set of implicated commits
for BFC. However, a comment on the issue report from
Sept. 23rd explains that the initial fix attempt in commit IC
contains problems that the later BFC commit addresses. In
this case, the comment points out that IC introduces thepotential for a null pointer exception, which is certainly adefect that matters for SZZ-based analyses.
Mapping and filtering ghosts are not uncommon in ASF
projects. Future SZZ implementations will likely benefit
from mitigation of ghost commits.

6

E XPLORING M ITIGATION S TRATEGIESIn the prior section, we discovered that mapping andfiltering ghosts do occur in the repositories of large andactive ASF projects. In this section, we present the resultsof our mitigation analysis for MG 1 commits (the mostfrequently occurring type of ghost commit) and propose
strategies for mitigating other types of ghost commits thatshould be explored in future work.
6.1

MG 1 Mitigation Analysis

Comparative Analysis: We find that data flow analysis
implicated exactly the same commits as the baseline
approach [29] in 15 of the 71 MG 1 commits fromthe ActiveMQ project (21.1%). A deeper examination ofthese implicated commits reveals that they mostly occur when the entire enclosing method was last modified by the same commit. For example, commit f7c7993
adds an if-check if (from.equals(to)). Our control flow analysis blames line 192 containing the enclosing method declaration public static Converter
lookupConverter(Class from, Class to), while ASZZ blames all the lines in the method (192–206). In thiscase, the implicated commit is the same since the entire
method was added by the same commit (1802116).
In cases where the lines immediately surrounding theadded lines were last modified by different commits thanthe method/class declaration, the two techniques yield different results. Yet at least one common commit is implicatedby both techniques in 41 of the remaining 56 cases (73.2%).
We are unable to implicate commits for non-Check New
Entity changes and for 50% of Override changes. Nonetheless, our data flow analysis also reveals that at least one

refactoring commit is incorrectly implicated as fix-inducing
46.5% of the time. This is due to an inherent shortcomingof SZZ and stresses the importance of implementing an
automated Refactoring Aware SZZ implementation [20].
Precision Analysis: We find that our data flow analysis has
a precision of 0.753, while A-SZZ has a precision of 0.358.
One reason for this difference in precision is the data flow
approach’s ability to implicate lines outside the code block
immediately surrounding the added lines. For example,
commit d92d3a8 adds a null check for reconnectTask onlines 148–150. The data flow approach traces line 129, which
updates reconnectTask’s value. This line is outside thetry block surrounding the null check.
Another reason for the difference in precision is that thelines in the code block are often unrelated to the defect being
fixed. This results in a higher rate of false positives.
A context-aware, data flow based approach implicates commits more precisely than a purely syntactic approach.
6.2

MG 1 Mitigation Strategies

Broadly speaking, the proposed mitigation strategies
require language-aware extensions to the SZZ approach.
Below, we describe our approach to mitigate each of thecategories of MG 1 from Table 3.
Algorithm 1 Null Check Mitigation
1: nullCheckV ariable = variable being null checked
2: range = additionLineN umber ± scanSize
3: linesT oT race = {}4: for line in range do
5:
if line contains nullCheckV ariable then
6:
Append line to linesT oT race
7:
end if
8: end for
9: return szz(linesT oT race)

Check: For each Check-type MG 1 commit, we first locate the identifier being checked and identify the line(s)that introduce or modify its value. For example, commit
4adc8e4 from the A CTIVE MQ project adds a null-checkfor the socketHandlerThread identifier on lines 470–473.
Data flow analysis reveals that socketHandlerThread
was introduced on line 451, which we add to the list of lines
to be processed by SZZ. In cases where the line introducing
the variable are not in scope, we follow the A-SZZ approach,tracing the surrounding block.
A key limitation of the approach is its reliance upon a(heavyweight) data flow analysis rather than solely mining
the software repositories. Semantic knowledge of the subject
system (i.e., a context-aware approach) is required whenanalyzing a change and deciding which change introduced
the identifier being checked, e.g., when blaming the method
declaration instead of lines surrounding the modified code.
Different SZZ users may have different needs depending on
the cost of false negatives (i.e., the importance of mitigating
ghosts) and false positives (i.e., the rate of false alarms).
Maintenance type analysis reveals that all Check-type
MG 1 commits are corrective, which is expected since theaddition of a check implies addressing an intrinsic defect.
Algorithm 1 outlines our mitigation strategy for null checks,
with a computational cost proportional to the breadth of thescanned area.

10

Algorithm 2 New Entity Mitigation
1: ref LineN umber = line referring to new entity
2: range = ref LineN umber ± scanSize
3: linesT oT race = {}4: for line in range do
5:Append line to linesT oT race
6: end for7: return szz(linesT oT race)

New Entity: We propose an SZZ-inspired sub-approach,where other classes, methods, and variables which referto the new entity refer are first mapped to the new entity
through static analysis of the source code and then filteredbased on their likelihood of leading to a defect. SZZ could
then be applied to the filtered set of other entities to identify potential fix-inducing changes. For example, commit
f6a5c7b adds the class XBeanFileResolver to help convert relative paths by verifying whether a provided path isa URL to an XBean file (boolean isXBeanFile(String
configUri)). Our proposal would apply SZZ to call sitesof this method. Algorithm 2 shows our mitigation strategy
for New Entity changes, with a computational cost computational cost proportional to the breadth of the scanned area.
While this direction is exciting, a key limitation of thismitigation strategy is that the implementation would require an in-depth parse of the source code of a project. Current SZZ implementations only rely on lightweight parses ofproject source code (e.g., to identify irrelevant comment andwhitespace changes). Adding this layer of complexity may
be too costly to justify the benefits for all projects; however,for projects where the implications of false negatives aresevere (e.g., safety critical systems), it may be worthwhile.
Another, less immediately concerning limitation is thatthe solution does not account for dynamic language features, such as reflection and dependency injection. Like
any static analysis, the proposed solution would inherit theclassic static analysis limitations. Hybrid static and dynamic
analyses could be used to address these limitations, butwould impose an even higher analysis cost.
When manually exploring this strategy in our sample,
we find that four of the six New Entity commits also involve
the addition of a check. In the example above, the new
XBeanFileResolver class is immediately used by an if
check in the same commit, which we can implicate using
Algorithm 1. Maintenance type analysis reveals that fiveof the six New Entity changes are corrective (intrinsic),
while the remainder are adaptive. Approaches to mitigate
extrinsic defects [26] are important for New Entity changes.
Configuration: For Configuration changes, we must convey an even deeper understanding of the context of thechange to the SZZ algorithm. For instance, an understanding of the properties being updated and/or the external
tool/framework being called is needed to implicate changes
in this category. To illustrate, consider commit 9c75fe7,
which updates the JMSXUSER_ID message property so thatthat it appears when browsing the message via JMX. A
deeper understanding of the JMX API would be required to
implicate fix-inducing commits for this defect-fixing change.
This type of ghost commit requires deep investmentin project-specific details, which may not transfer to otherprojects. Indeed, Configuration changes can be so specific toa niche that investigating them would require a complete

understanding of the studied projects.
Complicating matters further, maintenance type analysis
reveals that six of the seven Configuration changes are
adaptive. This suggests that the bulk of configuration fixes
do not have a commit to implicate. Given their relative infrequency and low rates of corrective maintenance, mitigationof Configuration ghosts are unlikely to yield much value.
Algorithm 3 Override Mitigation
1: overriddenM ethod = method being overridden
2: range = class hierarchy threshold
3: linesT oT race = {}4: for class in range do
5:
if class is superclass of overriddenM ethod then
6:
Append overridden method declaration to linesT oT race
7:
end if
8: end for
9: return szz(linesT oT race)

Override: We propose applying SZZ to the superclass variant of the method being overridden. For example, commit 51ef021 overrides the getPercentUsage() method,which belongs to the StoreUsage subclass, to fix a bug.
Our proposal would apply SZZ to the superclass variant
from Usage. Algorithm 3 outlines our mitigation strategy
for Override changes, with a computational cost proportional to the depth of the class hierarchy being searched.
A key concern with this solution is how quickly the setof implicated commits may grow. For complex hierarchies
with several variants of an overridden method, the set oflines being fed to SZZ may quickly grow, essentially trading
a false negative problem for a false positive one. To counter
this, the range setting can constrain the search space.
In the example above, applying our strategy leads
us to implicate commit 6d8e2c5, which originallyadded getPercentUsage() in the superclass. The
method was later overridden in commit 51ef021 to
add percentUsage = caclPercentUsage(), which refreshes the setting when retrieved over JMX.
Ten of the 18 Override changes also involve the additionof a check, where Algorithm 1 applies. In the example above,a null check of store is added to the overridden method.
Turning to the maintenance type, we find that six of theeight non-Check Override changes are corrective, and theremaining two are perfective. This suggests that extrinsic
defects are not a large concern for Override ghost commits.
Algorithm 4 Logging Mitigation
1: loggingV ariable = variable being logged
2: range = loggingLineN umber ± scanSize
3: linesT oT race = {}4: for line in range do
5:
if line contains loggingV ariable then
6:
Append line to linesT oT race
7:
end if
8: end for
9: return szz(linesT oT race)

Logging: We propose applying data flow analysis to determine where the value being logged, or the method containing the exception being logged, was last updated. Algorithm
4 outlines our mitigation strategy for Logging changes, witha computational cost proportional to the number of logging
variables of interest and the breadth of the scanned area.
For example, commit 56bed30 adds a logging statement to log a start failure exception LOG.trace("Error

11on start: ", e);. Applying our strategy to the catchstatement where e is caught implicates commit 082fdc5,
where the catch block was added without logging. Similar
to Algorithm 1, Algorithm 4 increases the complexity of SZZ
by increasing the amount of static analysis required.
Ten of the twelve logging changes are contained within
a Check change. In such cases, we implicate commits using
Algorithm 1, and consider them to be corrective (intrinsic).
Algorithm 5 Expanding Class Mitigation
1: linesT oT race = {}2: for line within expandedClass do
3:
if line last updated expandedClass then
4:
Append line to linesT oT race5:
end if
6: end for7: return szz(linesT oT race)

Expanding Class: An initial attempt may implicate the commit that last updated the expanded class as potentially fix
inducing. For example, commit 24f73a5 adds the method
testReceipts to the StompTest class. The intuition behind our approach is that a limitation in the initial implementation or last update to the class may be implicated inthis future fix. Algorithm 5 outlines our mitigation strategy
for Expanding Class changes, with a computational cost
proportional to the size of the expanded class.
This approach is naı̈ve, since the last change to a class
may not be responsible for the expansion. Yet this same
limitation is at the core of SZZ, i.e., the last edit to a line
may not be truly responsible for introducing the defect [28].
We find that all studied Expanding Class changes arecorrective. These changes fix defects by adding functionality that should have been added when the surrounding
block was last updated. For example, commit 5f7a81f
creates a copy of datasequence to fix a race condition in
the decompress method. We implicate commit 44bb9fb,
which adds this method without accounting for the race
condition. Commit c391321 fixes a null pointer exception
by adding return statements, while commit 4d0e572 fixes
a defect that is caused by the doRecoverNextMessages
method not breaking out loops by adding break statements. What is striking about these examples is how distinct
they are. An even finer grained analysis may be needed topropose mitigation strategies for each of these changes.
6.3

Promising Directions for Future Work on MG 2

To implicate MG 2 commits in future fixes, we propose
to track of program elements that were removed in a lookup
table. This lookup table can be checked during the SZZmapping phase. If program elements that were removed
are re-added later, the lookup table can map defect-fixing
commits to the commits where the elements were removed.
A key limitation of this approach is the cost of creating
and traversing the lookup table; however, we envision that asimple hash-like data structure could be efficient. Perhaps ofgreater concern is the risk of false positives, when commitsthat reintroduce a program element have done so as acoincidence rather than an intentional resurrection of theprevious code. To mitigate this risk, more heavyweightmatching techniques (e.g., clone detection [2]) could beapplied. This would increase the analysis cost (since entire

program elements would need to be tracked and not just theidentifier), but would likely reduce the false positive rate.
6.4

Promising Directions for Future Work on FG

While in theory, the null pointer exception discussed
in Section 5.3 and its fix should have been tracked under
an independent issue ID, in our experience, this reuse ofissue IDs is common developer behaviour. Indeed, Miuraet al. [17] found that 5%–62% (median 29%) of work items
across 14 studied systems are composed of two or morecommits. Moreover, Park et al. [23] found that 22%–33% ofresolved defects across three studied systems required morethan one fix attempt. Future SZZ extensions should takesuch behaviour into account to mitigate filtering ghosts.
Such filtering ghosts happen due to the inherent limitations of SZZ. A potential strategy to address multiple fix
attempts being linked to a single issue ID would be to relax
the date-cutoff in the filtering stage of SZZ by specifying adate range, within which commits may be implicated. Thisway, commits made after the bug report creation date, but
discussed in bug report comments may be considered. Thisapproach would increase the total number of commits tobe analyzed, and thus further increases the complexity ofapplying SZZ. A trade-off between the recall of SZZ andthe resources needed to analyze the extra commits could be
explored by varying the threshold of the date range.

7

T HREATS TO VALIDITY

Construct Validity: Construct threats to validity are associated with how closely our measurements reflect what we
set out to measure. When linking VCS commits to ITS
reports, we rely on developers recording the issueID within
the commit message. However, developers may mistype oromit the issueID, which would introduce linkage bias [3] into
our datasets. To mitigate the risk of linkage bias, we select asample of projects where the linkage rate exceeds 50%.
We characterize ghost commits using an open coding approach. Since we are not developers of the studied projects,
our understanding of the studied projects is limited. Thissurface understanding of the projects could introduce misclassification in our results. To mitigate this risk, two authors
independently coded the samples.
Internal Validity: Internal threats to validity emerge whenalternative hypotheses may also explain our observations.
We argue that addition-only fixes (MG 1) and removal-onlycommits (MG 2) present a risk for current SZZ implementations. However, it may be that these commits do not accountfor enough data to be of practical consequence. On the other
hand, we observe that ghost commits account for a considerable proportion of the fixes and the commits in the studied
projects. Since their mitigation will improve the recall of SZZ
approaches, the relative importance of addressing the ghostcommit problem may depend on the importance of false
negatives for the project(s) under analysis.
Developers may not create a new issue report for every
defect. As we observed in our analysis of the filtering ghosts,
follow-up work (e.g., minor defects in an initial patch) may
be tacked onto the same issue ID as the initial commit.
Future work should investigate how the SZZ approach canaccount for these patterns of use of issue trackers.

12

External Validity: External threats to validity are concernedwith the generalizability of our results. We studied 14 open
source projects from the Apache Software Foundation. Since
these projects are primarily written in Java, our results
may not generalize to other organizations or programming
languages. However, the studied projects are of varying
sizes (0.087 MLOC–4.3 MLOC) and span multiple domains(e.g., database management systems, content repositories).

8

C ONCLUSIONS

Defects are introduced during software development.
Identifying commits that are at risk of inducing future fixes
can help teams to allocate quality assurance effort more
effectively. To aid in identifying risky commits, the popular
SZZ approach for identifying fix-inducing commits is used;however, the SZZ approach is not without limitations. In
this paper, we focus on three types of ghost commits, i.e.,commits that cannot connect to or from other commits.
We conduct an empirical study of 14 Apache open source
projects to quantify and characterize these ghost commits,
observing that they occur regularly and share several common properties. Based on that characterization, we propose
context-aware directions for the community to improveupon the SZZ approach to mitigate ghost commits.

ACKNOWLEDGEMENTS
Kamei was supported by JSPS KAKENHI Japan (Grant
Numbers: JP21H04877, JP18H03222) and JSPS International
Joint Research Program with SNSF (Project “SENSOR”).

R EFERENCES
[1]

[2]

[3]

[4]

[5]
[6]

[7]

G. Bavota, B. D. Carluccio, A. De Lucia, M. Di Penta,
R. Oliveto, and O. Strollo, “When Does a Refactoring
Induce Bugs? An Empirical Study,” in Proc. of the Int’lWorking Conf. on Source Code Analysis and Manipulation,
2012, pp. 104–113.
I. D. Baxter, A. Yahin, L. Moura, M. Sant’Anna, and
L. Bier, “Clone Detection Using Abstract Syntax Trees,”in Proc. of the Int’l Conf. on Software Maintenance, 1998,
pp. 368–377.
C. Bird, A. Bachmann, E. Aune, J. Duffy, A. Bernstein,
V. Filkov, and P. Devanbu, “Fair and Balanced?: Biasin Bug-Fix Datasets,” in Proc. of the Joint Meeting of theEuropean Software Engineering Conf. and the Symposiumon The Foundations of Software Engineering, 2009, pp.
121–130.
G. Canfora, M. Ceccarelli, L. Cerulo, and M. Di Penta,“How Long Does a Bug Survive? An Empirical Study,”
in Proc. of the Working Conf. on Reverse Engineering.
IEEE, 2011, pp. 191–200.
K. Charmaz, Constructing Grounded Theory. Sage, 2014.
J. Cohen, “A Coefficient of Agreement for Nominal Scales,” Educational and Psychological Measurement,
vol. 20, no. 1, pp. 37–46, 1960.
D. A. da Costa, S. McIntosh, W. Shang, U. Kulesza,
R. Coelho, and A. E. Hassan, “A Framework for Evaluating the Results of the SZZ Approach for Identifying
Bug-Introducing Changes,” IEEE Transactions on Software Engineering, vol. 43, no. 7, pp. 641–657, 2017.

[8]

J. Eyolfson, L. Tan, and P. Lam, “Do Time of Day andDeveloper Experience Affect Commit Bugginess?” in
Proc. of the Working Conf. on Mining Software Repositories,
2011, pp. 153–162.
[9] D. C. Howell, “Median Absolute Deviation,” Encyclopedia of Statistics in Behavioral Science, pp. 1193–1193, 2005.
[10] Y. Kamei, T. Fukushima, S. McIntosh, K. Yamashita,
N. Ubayashi, and A. E. Hassan, “Studying Just-InTime Defect Prediction using Cross-Project Models,”
Empirical Software Engineering, vol. 21, no. 5, pp. 2072–
2106, 2016.
[11] Y. Kamei, E. Shihab, B. Adams, A. E. Hassan,
A. Mockus, A. Sinha, and N. Ubayashi, “A Large-Scale
Empirical Study of Just-in-Time Quality Assurance,”
IEEE Transactions on Software Engineering, vol. 39, no. 6,
pp. 757–773, 2013.
[12] S. Kim and E. James Whitehead, “How Long Did it
Take to Fix Bugs?” in Proc. of the Int’l Workshop on
Mining Software Repositories, 2006, pp. 173–174.
[13] S. Kim, E. James Whitehead, and Y. Zhang, “Classifying
software changes: Clean or buggy?” IEEE Transactions
on Software Engineering, vol. 34, no. 2, pp. 181–196, 2008.
[14] S. Kim, T. Zimmermann, K. Pan, and E. James Whitehead, “Automatic Identification of Bug-introducing
Changes,” in Proc. of the Int’l Conf. on Automated Software Engineering, 2006, pp. 81–90.
[15] O. Kononenko, O. Baysal, L. Guerrouj, Y. Cao, andM. W. Godfrey, “Investigating Code Review Quality:
Do People and Participation Matter?” in Proc. of the Int’l
Conf. on Software Maintenance and Evolution, 2015, pp.
111–120.
[16] S. McIntosh and Y. Kamei, “Are Fix-Inducing Changes
a Moving Target? A Longitudinal Case Study of Just-inTime Defect Prediction,” IEEE Transactions on Software
Engineering, vol. 44, no. 5, pp. 412–428, 2017.
[17] K. Miura, S. McIntosh, Y. Kamei, A. E. Hassan, andN. Ubayashi, “The Impact of Task Granularity on Coevolution Analyses,” in Proc. of the Int’l Symposium on
Empirical Software Engineering and Measurement, 2016,
pp. 47–57.
[18] A. Mockus and D. M. Weiss, “Predicting Risk of Software Changes,” Bell Labs Technical Journal, vol. 5, no. 2,
pp. 169–180, 2000.
[19] N. Munaiah, S. Kroh, C. Cabrey, and M. Nagappan,“Curating Github for Engineered Software Projects,”
Empirical Software Engineering, vol. 22, no. 6, pp. 3219–
3253, 2017.
[20] E. C. Neto, D. A. da Costa, and U. Kulesza, “The Impactof Refactoring Changes on the SZZ Algorithm: An
Empirical Study,” in Proc. of the Int’l Conf. on Software
Analysis, Evolution and Reengineering, 2018, pp. 380–390.
[21] T. H. Nguyen, B. Adams, and A. E. Hassan, “A Case
Study of Bias in Bug-Fix Datasets,” in Proc. of theWorking Conf. on Reverse Engineering, 2010, pp. 259–268.
[22] K. Pan, S. Kim, and E. James Whitehead, “Toward an
Understanding of Bug Fix Patterns,” Empirical Software
Engineering, vol. 14, no. 3, pp. 286–315, 2009.
[23] J. Park, M. Kim, B. Ray, and D.-H. Bae, “An Empirical
Study of Supplementary Bug Fixes,” in Proc. of theWorking Conf. on Mining Software Repositories, 2012, pp.
40–49.

13[24] F. Rahman, C. Bird, and P. Devanbu, “Clones: What isthat Smell?” Empirical Software Engineering, vol. 17, no.
4-5, pp. 503–530, 2012.
[25] F. Rahman and P. Devanbu, “Ownership, Experienceand Defects: A Fine-Grained Study of Authorship,” inProc. of the Int’l Conf. on Software Engineering, 2011, pp.
491–500.
[26] G. Rodrı́guez-Pérez, M. Nagappan, and G. Robles,“Watch out for Extrinsic Bugs! A Case Study of their
Impact in Just-In-Time Bug Prediction Models on the
OpenStack project,” IEEE Transactions on Software Engineering, 2020.
[27] G. Rodrı́guez-Pérez, G. Robles, A. Serebrenik, A. Zaidman, D. M. Germán, and J. M. Gonzalez-Barahona,“How Bugs Are Born: A Model to Identify How Bugs
Are Introduced in Software Components,” Empirical
Software Engineering, vol. 25, no. 2, pp. 1294–1340, 2020.
[28] G. Rodrı́guez-Pérez, A. Zaidman, A. Serebrenik,
G. Robles, and J. M. González-Barahona, “What if a Bug
has a Different Origin? Making Sense of Bugs Without
an Explicit Bug Introducing Change,” in Proc. of theInt’l Symposium on Empirical Software Engineering and
Measurement, 2018, pp. 1–4.
[29] E. Sahal and A. Tosun, “Identifying Bug-Inducing
Changes for Code Additions,” in Proc. of the Int’l Symposium on Empirical Software Engineering and Measurement,
2018, pp. 1–2.
[30] E. Shihab, A. E. Hassan, B. Adams, and Z. M. Jiang, “AnIndustrial Study on the Risk of Software Changes,” inProc. of the Int’l Symposium on the Foundations of Software
Engineering, 2012, pp. 62–73.
[31] J. Shimagaki, Y. Kamei, S. McIntosh, D. Pursehouse,
and N. Ubayashi, “Why are Commits being Reverted?
A Comparative Study of Industrial and Open Source
Projects,” in Proc. of the Int’l Conf. on Software Maintenance and Evolution, 2016, pp. 301–311.
[32] J. Śliwerski, T. Zimmermann, and A. Zeller, “When do
Changes Induce Fixes?” in Proc. of the Int’l Workshop on
Mining Software Repositories, 2005, pp. 1–5.
[33] E. B. Swanson, “The Dimensions of Maintenance,” inProc. of the Int’l Conf. on Software Engineering, 1976, pp.
492–497.
[34] M. Tan, L. Tan, S. Dara, and C. Mayeux, “Online DefectPrediction for Imbalanced Data,” in Proc. of the Int’l
Conf. on Software Engineering, vol. 2, 2015, pp. 99–108.
[35] M. Tufano, F. Palomba, G. Bavota, M. Di Penta,
R. Oliveto, A. De Lucia, and D. Poshyvanyk, “Thereand Back Again: Can You Compile That Snapshot?”
Journal of Software: Evolution and Process, vol. 29, no. 4,
p. e1838, 2017.
[36] M. Tufano, F. Palomba, G. Bavota, R. Oliveto, M. Di
Penta, A. De Lucia, and D. Poshyvanyk, “When and
Why Your Code Starts to Smell Bad,” in Proc. of the Int’l
Conf. on Software Engineering, 2015, pp. 403–414.
[37] C. C. Williams and J. W. Spacco, “Branching and Merging in the Repository,” in Proc. of the Int’l Working Conf.
on Mining Software Repositories, 2008, pp. 19–22.
[38] C. C. Williams and J. W. Spacco, “SZZ Revisited: Verifying When Changes Induce Fixes,” in Proc. of theWorkshop on Defects in Large Software Systems, 2008, pp.
32–36.

[39] R. Wu, H. Zhang, S. Kim, and S.-C. Cheung, “Relink:
Recovering Links Between Bugs and Changes,” in Proc.
of the ACM SIGSOFT Symposium and the European Conf.
on Foundations of Software Engineering, 2011, pp. 15–25.
[40] M. Yan, X. Xia, D. Lo, A. E. Hassan, and S. Li, “Characterizing and Identifying Reverted Commits,” Empirical
Software Engineering, vol. 24, no. 4, pp. 2171–2208, 2019.
[41] H. Yang, C. Wang, Q. Shi, Y. Feng, and Z. Chen, “BugInducing Analysis to Prevent Fault Prone Bug Fixes.”
in Proc. of the Int’l Conf. on Software Engineering andKnowledge Engineering, 2014, pp. 620–625.
[42] X. Yang, D. Lo, X. Xia, Y. Zhang, and J. Sun, “DeepLearning for Just-in-Time Defect Prediction,” in Proc. of
the Int’l Conf. on Software Quality, Reliability and Security,
2015, pp. 17–26.
[43] Z. Yin, D. Yuan, Y. Zhou, S. Pasupathy, and L. Bairavasundaram, “How do Fixes Become Bugs?” in Proc. of
the Symposium and the European Conf. on Foundations ofSoftware Engineering, 2011, pp. 26–36.
[44] F. Zampetti, A. Serebrenik, and M. Di Penta, “Was SelfAdmitted Technical Debt Removal a Real Removal?
An In-Depth Perspective,” in Proc. of the Int’l Conf. on
Mining Software Repositories, 2018, pp. 526–536.

Christophe Rezk is a Master’s student at McGill
University. In his research, Christophe studies
Just-in-Time (JIT) defect prediction, focusing on
the SZZ approach for identifying fix inducingchanges, evaluating its shortcomings, and exploring ways to improve it.

Yasutaka Kamei is an associate professor at
Kyushu University in Japan. He has been a research fellow of the JSPS (PD) from July 2009
to March 2010. From April 2010 to March 2011,
he was a postdoctoral fellow at Queen’s University in Canada. He received his Ph.D. fromthe Nara Institute of Science and Technology.
His research interests include empirical software
engineering, open source software engineeringand Mining Software Repositories (MSR).

Shane McIntosh is an Associate Professor at
the University of Waterloo. Previously, he wasan assistant professor at McGill University. He
received his Ph.D. from Queen’s University. In
his research, Shane uses empirical methods
to study software build systems, release engineering, and software quality: https://rebels.cs.
uwaterloo.ca/.