2020 IEEE International Conference on Software Maintenance and Evolution (ICSME)

SBFL-Suitability:
A Software Characteristic for Fault Localization
Yui Sasaki∗† , Yoshiki Higo∗ , Shinsuke Matsumoto∗ and Shinji Kusumoto∗∗ Graduate School of Information Science and Technology, Osaka University, Japan† The Japan Research Institute, Limited, Japan{s-yui, higo, shinsuke, kusumoto}@ist.osaka-u.ac.jp
degree of efﬁciency of diagnosing the causes of failures or
identifying parts to be modiﬁed.
There are various implementation ways in high-level programming languages such as C/C++ and Java, and thus developers choose how to implement the required functionality
according to their preferences or project policies. When an
implementation of a functionality gets changed, the execution
paths of its test cases may vary. This change would lead todifferences in each statement’s suspiciousness and ranking.
Therefore, we believe that source code itself has a characteristic that indicates the efﬁciency of localizing faults.
In this paper, we deﬁne SBFL-Suitability as the efﬁciency
of localizing faults using SBFL techniques. SBFL-Suitability
can be a part of Maintainability or Analyzability. Considering
SBFL-Suitability as a part of software quality characteristics
allows us to conduct the following activities.

Abstract—Spectrum-Based Fault Localization (in short, SBFL)
is one of the popular techniques to localize faulty code fragmentsof a given program. SBFL utilizes the information about whichstatements are executed by each of the success or failure test
cases. There are various implementation ways for the samefunctionality if we use high-level programming languages. The
authors consider that differences in these implementation ways
may affect the efﬁciency of localizing faults using SBFL. In thispaper, we deﬁne a characteristic to what extent a program issuitable for SBFL as SBFL-Suitability, and we propose a technique for measuring SBFL-Suitability. The proposed techniquegenerates many slightly-variant programs from a given programwith Mutation Testing, and then it measures how accurately SBFL
detects the changed program statements in the variant programs.
We conducted an experiment to investigate how SBFL-Suitability
differs depending on the differences in source code structures.
As a result, we found that (1) the fewer statements in the same
nesting level, the higher SBFL-Suitability tends to be, and (2) the
presence of Early Return improves SBFL-Suitability.
Index Terms—Spectrum-Based Fault Localization, Mutation
Testing, Software Quality

We can ﬁnd how reliable the SBFL results are for a given
program. If it is reliable, we can debug the program with
the information on SBFL.
• We can conduct refactoring to a given program from theviewpoint of improving its SBFL-Suitability.
•

I. I NTRODUCTION
In software development, debugging is a highly laborintensive and costly task [1]. Developers spend almost half
or more their programming time on debbuging [2]. For thisreason, there are a variety of studies on supporting debugging.
Fault Localization is one of the promising techniques that automatically localize faulty code fragments of a given program.
Recently, Spectrum-Based Fault Localization (in short, SBFL)
has been actively studied [3]. SBFL techniques calculate thelikelihood of a fault (henceforth, suspiciousness) for eachstatement in a given faulty program using test results andthe information about which statements are executed by eachtest case (henceforth, execution paths). We can efﬁciently
identify a faulty statement of a faulty program by checking
the statements in the program in the descending order of
their suspiciousness. The applied SBFL technique is presumed
to work the most efﬁciently for the program if the faulty
statement has the highest suspiciousness. In other words, thegiven program and test cases are well suited for SBFL.
Software quality consists of various viewpoints. ISO/IEC
25010 [4] deﬁnes the quality model for software products,
which comprises the eight quality characteristics and subcharacteristics derived from each of them. Maintainability,
which is one of the quality characteristics, includes Analysability as one of its sub-characteristics. Analysability indicates the2576-3148/20/$31.00 ©2020 IEEE
DOI 10.1109/ICSME46990.2020.00076

We also propose a technique for measuring SBFL-Suitability
for a given program. More concretely, ﬁrstly, the proposed
technique generates many slightly-variant programs by changing a target program intentionally with Mutation Testing [5].
The generated programs can be considered as faulty programs.
Secondly, the technique applies SBFL to each of the generated
programs. Finally, the technique calculates SBFL-Suitability
by measuring how accurately SBFL localizes the changed
program statement of each of the generated programs.
In this paper, we conducted an experiment to investigate
how the differences in source code structures affect SBFLSuitability. As a result, we found that some source codecharacteristics, such as a small number of statements at thesame nesting level or the presence of Early Return, improve
SBFL-Suitability.
The main contributions of this paper are the following.
We introduce a novel software quality characteristic,
SBFL-Suitability.
• We propose a concrete technique for measuring SBFLSuitability utilizing Mutation Testing.
• We ﬁnd that SBFL-Suitability varies with source code
structures.
•702

We reveal the characteristics of source code structuresthat improve SBFL-Suitability.

program(a)
Before Refactoring•

II. R ELATED W ORK(b)
After Refactoring

SBFL is one of the most popular techniques in the ﬁeld offault localization and it has been actively studied recently [6]–[8]. SBFL techniques calculate suspiciousness of the program
statements in a program (i.e., the probability that the statement
includes a fault) using the test results and the execution pathsof each test case. The execution paths are information about
which statements in the source code are executed by eachtest case. Intuitively, a statement that is executed by many
failed test cases can be considered more likely to contain afault. Abreu et al. compared several SBFL techniques, andthey concluded that the Ochiai’s formula [9] was the mosteffective one [10].
Mutation Testing [5] is a technique to evaluate given tests
using generated faulty programs, called mutants. A mutant isgenerated by applying small changes (e.g., replacement of inﬁx
operators) from the original program that behaves correctly.
The key idea of the mutation testing is to measure to what
extent given tests can identify the mutants as faulty programs.
Note that mutants might not be faulty because the appliedchanges might not alter the behavior. There are various tools
for mutation testing [11]–[13]. A recent study reported that
mutation testing was useful for improving the veriﬁcation ofindustry software programs [14].

susp

s 1 : boolean result = false;s 2 : if (0 < a)s 3 : result = true; s 4 : if (0 <= b) // correct: 0 < bs 5 : result = true;s 6 : return result;s' 2 : if (0 < a)s' 3 : return true;
 s' 4 : if (0 <= b) // correct: 0 < bs' 5 : return true;s' 6 : return false;

test case (input: a, b)t 1 (1,1) t 2 (1,0) t 3 (0,1) t 4 (0,0)

0.50
0.50
0.00
0.50
0.50
0.50





































0.50
0.00




















P

F

0.71
0.71
-

test results:

P

P

Fig. 1. SBFL results compared with different source code structuresat its statement. Those results show that faulty statements 𝑠4
and 𝑠4 have different suspiciousness.
We consider a situation that a developer who does not know
the faulty locations in those methods tries to identify thefaulty statements on the basis of the SBFL results. Then, s/hechecks each of the statements in the descending order of theirsuspiciousness. In the case of Figure 1(a), the suspiciousnessof faulty statement 𝑠4 is the highest though there are a totalof ﬁve statements with the same highest suspiciousness. Thus,in the worst case, s/he has to check the ﬁve statements to
identify the fault. In contrast, in Figure 1(b), s/he can identify
the fault by only checking at most two statements. Therefore,
the method of Figure 1(b) is more efﬁcient in identifying thefault than the method of Figure 1(a). Note that this scenario
is not depending on the SBFL formula (i.e., Ochiai) because
their execution paths itself are changed by refactoring.

III. SBFL-Suitability
The authors consider that source code itself has characteristics of how well it is suited to SBFL. Even if two different
source code have the same functionality and test cases, theirdifferent source code structures may cause differences in the
efﬁciency of identifying faults using SBFL. In this paper, wedeﬁne to what extent a given source code is suitable for SBFL
as SBFL-Suitability.
Herein, we show a concrete example of how SBFLSuitability differs depending on source code structures. Figure 1 shows two Java methods that take two input numbers andreturn true if and only if at least one of them is positive. Figure 1(b) shows the method that a refactoring has been applied
to the method in Figure 1(a). More concretely, the method of
Figure 1(a) returns the value at the end of the method with thetemporary variable result, which is assigned inside of eachif-block, whereas the method of Figure 1(b) directly returns
inside of each if-block. Both of the methods contain the same
fault. Test case 𝑡4 fails because the conditional expression of𝑠4 in Figure 1(a) and the one of 𝑠4 in Figure 1(b) behave
incorrectly when the variable b is 0.
Figures 1(a) and 1(b) show test results (P: Passed, F: Failed),
the execution path for each test case, and the suspiciousness1(labeled as ‘susp’) of each program statement calculated by the
Ochiai’s formula. Suspiciousness takes a value between 0 and
1. Value 1 means the highest likelihood that the fault locates

IV. M EASURING SBFL-Suitability
We propose a technique to measure SBFL-Suitability of agiven program. Our basic idea is to generate multiple faulty
programs from the original program and measure how efﬁciently SBFL can identify which statement is faulty. To generate faulty programs, we utilize mutation testing techniques.
The proposed technique measures how accurately SBFL can
localize the faulty statements in each of the generated mutants
while mutation testing essentially measures how accurately atest suite can identify the generated mutants as faulty.
A. Factors Affecting SBFL-Suitability
SBFL-Suitability depends on not only the source code
structure of a program but also the following two factors: atest suite and a mutant generator.
As an example, we consider the situation where the test
suite includes only 𝑡3 and 𝑡 4 in Figure 1(b). Since the execution
paths for the two test cases are identical, the suspiciousness
calculated from each of the execution paths are the same for atleast statements 𝑠2 , 𝑠4 , and 𝑠5 that are executed by those test
cases. In the case of the original test suite, statements 𝑠4 and 𝑠5
are more likely to be fault than 𝑠2 . In this way, how efﬁciently
SBFL works varies depending on the given test suite.
The proposed technique uses several mutation operators,
which are transformation rules for generating mutants. This

1 The prepared test suite does not execute 𝑠 in Figure 1(b), which means
6
that the suspiciousness is not calculated for 𝑠6 .

703

program pmutant m1
1. generating mutants

faultystatement

than susp𝑇 (𝑠). For example, if there are two statements withsuspiciousness 1.0 and one statement with 0.9, both statementswith 1.0 are in the second place, and the statement with 0.9
is in the third place.
A rank of suspiciousness has different worthiness dependingon the total number of statements. For example, the 10th placeout of 100 statements is more valuable than the 10th place outof 10 statements. Thus, we normalize a rank of suspiciousness
to a range between 0 and 1. A normalized rank rScore𝑇 (𝑠)indicates how high it appears in the total number of statements
and is calculated as follows:mutant mn…



2. applying SBFL
on each mutant
mutant generator G

test suite T

SBFLScoreT,G(p)rScoreT(m1)…

rScoreT(mn)

3. calculating an SBFLScore of p

Fig. 2. Process of the SBFLScore CalculationrScore𝑇 (𝑠) = 1 −paper calls a set of mutation operators as a mutant generator.
SBFL-Suitability depends on a mutant generator.

rank𝑇 (𝑠) − 1
totalStatements𝑇 − 1

where totalStatements𝑇 is the number of statements executedby a test suite 𝑇. The value 1 is the most valuable.
Let rScore𝑇 (𝑚) be the normalized rank of the faulty statement in mutant 𝑚. rScore𝑇 (𝑚) is unique for each mutant
because each mutant contains only a single faulty statement.
Let 𝑠 𝑚 fault be the faulty statement in a mutant 𝑚. rScore𝑇 (𝑚)is the unique normalized rank of statement 𝑠 𝑚 fault as follows:

B. Calculating SBFLScore
We deﬁne an indicator of the degree of SBFL-Suitability as
SBFLScore. SBFLScore takes a value between 0 and 1, and thecloser to 1, the higher SBFL-Suitability. Let SBFLScore𝑇 ,𝐺 ( 𝑝)be an SBFLScore of a program 𝑝 with a test suite 𝑇 anda mutant generator 𝐺. Figure 2 shows the process of the
SBFLScore calculation. We calculate an SBFLScore of 𝑝 withthe following three steps:
1) generating multiple mutants from 𝑝,2) applying SBFL to each mutant and then calculating
suspiciousness ranking of their faulty statements, and3) calculating an SBFLScore of 𝑝 from the suspiciousness
ranking of each of the generated mutants.
Step 1. Generating Mutants: The proposed technique generates multiple mutants using a mutant generator 𝐺 for a
program 𝑝. Each mutant has only a single different statementfrom the original program. We treat such a statement as afaulty statement. Note that, at this point, we do not knowwhether all of the mutants are faulty or not.
Step 2. Applying SBFL for Each Mutant: For each of thegenerated mutants, we apply SBFL using a test suite 𝑇, andthen, we calculate a suspiciousness for each statement. We
exclude mutants that are identiﬁed as non-faulty. Let 𝑀 𝐺 ( 𝑝)be a set of obtained mutants for a program 𝑝 with a mutant
generator 𝐺. We deﬁne the following values for statement 𝑠
included in mutant 𝑚 ∈ 𝑀 𝐺 ( 𝑝):• susp𝑇 (𝑠): the suspiciousness of 𝑠 when executing 𝑇,𝑇• rank (𝑠): the rank of susp𝑇 (𝑠), and𝑇
𝑇• rScore (𝑠): the normalized rank (𝑠) between 0 and 1.
We use Ochiai’s formula, known to be effective, to calculate
suspiciousness. susp𝑇 (𝑠) is calculated by the Ochiai’s formula
as follows:rScore𝑇 (𝑚) = rScore𝑇 (𝑠 𝑚 fault )Step 3. Calculating SBFLScore: SBFLScore is the averagevalue of the normalized rank of each mutant’s faulty statement
generated from 𝑝. We deﬁne it by the following formula.
1
rScore𝑇 (𝑚)SBFLScore𝑇 ,𝐺 ( 𝑝) =𝐺
|𝑀 ( 𝑝)|
𝐺
𝑚∈𝑀

( 𝑝)

V. E XPERIMENT
We implemented a tool based on the proposed techniquefor Java. We conducted an experiment to investigate how thedifferences in source code structures affect SBFL-Suitability.
A. Experimental Settings
We used PIT [15], an open-source mutation testing tool,
as a reference for mutation operators. PIT has published thetransformation rules of mutation operators, and it deﬁnes
groups of operators on its web site. We selected all of theeleven mutation operators included in group ‘DEFAULTS’.
We implemented a new mutant generator tool for source codethat supports these operators because PIT (and other existing
mutation testing tools) generate bytecode mutants instead ofsource code ones. Table I shows the target mutation operators.
In this paper, we focus on refactoring for a differencein source code structures. Table II shows the ﬁve types ofrefactoring. We selected2 refactorings classiﬁed as ‘Symplifying Conditional Expression’ [16]. The reason is that theexecution path of each test case is changed because statements
in conditional blocks are changed by these refactorings, whichmight affect their suspiciousness. We manually and thoughtfully created target programs by reference to program sampleof each refactoring pattern [16] as many mutation operators

fail𝑇 (𝑠)susp𝑇 (𝑠) = 
totalFail𝑇 ∗ (fail𝑇 (𝑠) + pass𝑇 (𝑠))where fail𝑇 (𝑠) is the number of failed test cases that cover 𝑠,
pass𝑇 (𝑠) is the number of passed test cases that cover 𝑠, andtotalFail𝑇 is the total number of failed test cases.
Next, we calculate rank𝑇 (𝑠), which indicates the numberof statements whose suspiciousness is equal to or greater

2 We excluded refactorings across multiple Java methods because we apply
SBFL and then calculated a ranking of the statements for each method.

704

rScoremutant: m 1 m 2 m 3 m 4 m 5 m 6 m 7
mutation operator: NC CB INC NC CB INC PR
s 1 : if (0 < n)0.67 0.50 0.50 0.50 0.25 0.25 0.75
0.33 1.00 1.00 0.00 0.00 0.00 0.25
s 2 : n--;0.00 0.00 0.00 1.00 0.75 0.75 0.00
s 3 : else if (n < 0)0.00 0.00 0.25 0.75 1.00 0.25
s 4 : n++;s 5 : return n;
0.67 0.50 0.50 0.50 0.25 0.25 0.75

could be applied as possible. We also manually created test
cases with satisfying the condition coverage for all the mutants
generated from each program.
B. Results and Discussion
We applied our tool to the prepared programs. Note thatin all target programs, the generated mutants failed at least
one or more test cases; in other words, they were faulty.
Table II shows the results of the SBFLScore measurement.
SBFLScore was decreased by the refactoring in Cases 1–3
whereas SBFLScore was increased in Cases 4 and 5. Due tospace limitations, we discuss only Cases 1 and 5.
Figures 3 and 4 shows the programs of Cases 1 and 5. In
both ﬁgures, (a) and (b) show the code fragments before andafter refactoring, respectively. We describe a pair of statementswith the same subscript number before and after refactoringsuch 𝑠2 and 𝑠2 in Figure 3 as 𝑠2 , 𝑠2 . Such a pair denotes thatthey have a correspondence relation between before and after
refactoring. For example in Figure 3, 𝑠2 has not been changed
 and 𝑠  havefrom 𝑠2 through the refactoring whereas 𝑠1𝑎1𝑏
been split from 𝑠1 through the refactoring. These relations are , and 𝑠 , 𝑠  . Figures 3 anddescribed as 𝑠2 , 𝑠2 , 𝑠1 , 𝑠1𝑎
1 1𝑏
4 also show rScore of each statement included in each of
the generated mutants. Each rScore is also represented by ahorizontal bar chart whose range is from 0 to 1. rScore of thefaulty statement in each mutant is shown in bold. For example,
𝑚 1 in Figure 3(a) is the mutant where mutation operator NC
has been applied to statement 𝑠1 , and its rScore is 0.67.
In the following description, we focus on a pair of mutants
with the same subscript number before and after refactoring,such as 𝑚 1 and 𝑚 1 . We describe such a pair as 𝑚 1 , 𝑚 1 . Sucha pair of mutants denotes that they have been generated byapplying the same mutation operators to corresponding state ).
ments between before and after the refactoring (e.g., 𝑠1 , 𝑠1𝑎
By comparing rScore of each pair of mutants, we investigate
how different the ease of localizing the corresponding faulty
statement between before and after the refactoring.
1) Case 1: ‘Decompose Conditional’ is a refactoring thatextracts conditional expressions to meaningfully named methods. We prepared the target program whose conditional expressions were extracted to variables instead of methods because
our technique measures SBFL-Suitability for each method.

(a) Before Refactoring (SBFLScore =0.81)
rScoremutant: m '1 m '2 m '3 m '4 m '5 m '6 m '7
mutation operator: NC CB INC NC CB INC PR
s' 1a : boolean f1 = (0 < n);0.40 0.33 0.33 0.33 0.17 0.17 0.50
0.40 0.33 0.33 0.33 0.17 0.17 0.50
s' 3a : boolean f2 = (n < 0);0.40 0.33 0.33 0.33 0.17 0.17 0.50
s' 1b : if (f1)
0.20 1.00 1.00 0.00 0.00 0.00 0.17
s' 2 : n--;0.00 0.00 0.00 1.00 0.83 0.83 0.00
s' 3b : else if (f2)0.00 0.00 0.17 0.83 1.00 0.17
s' 4 : n++;s' 5 : return n;0.40 0.33 0.33 0.33 0.17 0.17 0.50(b) After Refactoring (SBFLScore =0.53)
Fig. 3. Case 1: Decompose Conditional

SBFLScore was decreased by the refactoring. rScore of thefaulty statements were the same for 𝑚 3 , 𝑚 3  and 𝑚 6 , 𝑚 6 
whereas they were decreased for the others. For example,
rScore of the faulty statements in 𝑚 1 , 𝑚 1  were the highest
in each mutant. There were two statements with the same
highest rScore in 𝑚 1 (i.e., 𝑠1 and 𝑠5 ) whereas there were four , 𝑠  , 𝑠  and 𝑠  ). Those statements
statements in 𝑚 1 (i.e., 𝑠1𝑎
3𝑎 1𝑏
5
are at the nesting level 1. We checked the calculation process
of rScore, and then we found that susp of all of the statements
were 1.00. This observation implies that the increase in thenumber of the same nesting level leads to the decrease ofrScore; as a result, SBFL-Suitability gets worsened.
The fewer statements at the same nesting level, the higher
SBFL-Suitability tends to be.
2) Case 5: ‘Replace Nested Conditional with Guard
Clauses’ is a refactoring that returns early from a method bychecking conditions that are satisﬁed not to execute the mainprocess of the method. The refactoring is effective to preventthe source code from being deeply nested. To simplify theexperiment, we prepared the target program where a returnstatement has been inserted in each conditional block. Such a
coding style is called ‘Early Return’ [17].
SBFLScore was increased by the refactoring. rScore of

TABLE IM UTATION O PERATORS
Transformation example
Before
After(CB) Conditional Boundary a < b
a <= b(INC) Incrementsn++n-(INV) Invert Negatives-nn(MA) Math
a + b
a - b
(NC) Negate Conditionals a < ba >= b(VM) Void Method Calls
method();;(PR) Primitive Returns
return 5;
return 0;(ER) Empty Returns
return "str"; return "";(FR) False Returns
return true;
return false;(TR) True Returns
return false; return true;(NR) Null Returns
return object; return null;
Mutation operator

TABLE IITARGET R EFACTORINGS
Case
Case 1Case 2
Case 3
Case 4
Case 5705

Refactoring pattern
Decompose Conditional
Consolidate Conditional Expression
Consolidate Duplicate Conditional Fragments
Remove Control Flag
Replace Nested Conditional with Guard Clauses

SBFLScore
Before After
0.81 0.53
0.95 0.72
0.69 0.53
0.61 0.67
0.83 0.95

rScoremutant: m 1 m 2 m 3 m 4 m 5 m 6 m 7 m 8mutation operator: NC CB INV NC CB INV INV PR
s 1 : int result = 0; 0.67 0.50 0.50 0.33 0.33 0.33 0.33 0.67
s 2 : if (x > 0)0.67 0.50 0.50 0.33 0.33 0.33 0.33 0.67
s 3 : result = -10; 0.50 1.00 1.00 0.00 0.00 0.00 0.00 0.17
s 4 : else if (y > 0) 0.33 0.00 0.00 1.00 0.83 0.83 0.83 0.50
s 5 : result = -20; 0.00 0.00 0.00 0.83 1.00 1.00 0.00 0.00
else








s 6 : result = -30; 0.17 0.00 0.00 0.17 0.00 0.00 1.00 0.33
s 7 : return result; 0.67 0.50 0.50 0.33 0.33 0.33 0.33 0.67

the characteristic as SBFL-Suitability. Besides, we proposeda technique to measure SBFL-Suitability utilizing mutation
testing techniques. We applied the proposed technique toseveral refactoring situations, and observed the difference of
SBFL-Suitability between before and after the refactoring.
As a result, we found the characteristics of the source code
structures that improve SBFL-Suitability.
In the future, we are going to conduct experiments for real
and large-scale programs to generalize our proposed technique.
Besides, we are going to propose a technique to convert
source code structures to improve SBFL-Suitability of a given
program. If we convert source code structures to a higher
degree of SBFL-Suitability before applying SBFL, we canidentify faulty location more efﬁciently.

(a) Before Refactoring (SBFLScore =0.83)
rScoremutant: m '1 m '2 m '3 m '4 m '5 m '6 m '7 m '8 m '9 m '10
mutation operator: NC CB INV NC CB INV INV PR PR PR
s' 2 : if (x > 0)
1.00 0.75 0.75 0.50 0.50 0.50 0.50 0.50 0.75 0.50
s' 3 : return -10; 0.75 1.00 1.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00
0.50 0.00 0.00 1.00 0.75 0.75 0.75 0.75 0.00 0.75
s' 4 : if (y > 0)s' 5 : return -20; 0.00 0.00 0.00 0.75 1.00 1.00 0.00 0.00 0.00 1.00
s '6 : return -30; 0.25 0.00 0.00 0.25 0.00 0.00 1.00 1.00 0.00 0.00

ACKNOWLEDGMENT(b) After Refactoring (SBFLScore =0.95)

This study has been supported by Grants-in-Aid for Scientiﬁc Research (B) (18H03222), Scientiﬁc Research (B)(20H04166) from the Japan Society for the Promotion of
Science.

Fig. 4. Case 5: Replace Nested Conditional with Guard Clauses

R EFERENCES[1] B. Hailpern and P. Santhanam, “Software debugging, testing, andveriﬁcation,” IBM Systems Journal, vol. 41, no. 1, pp. 4–12, 2002.
[2] T. Britton, L. Jeng, G. Carver, and P. Cheak, “Reversible debugging
software quantify the time and cost saved using reversible debuggers ,”2013.
[3] W. E. Wong, R. Gao, Y. Li, R. Abreu, and F. Wotawa, “A survey on
software fault localization,” IEEE TSE, vol. 42, no. 8, pp. 707–740,
2016.
[4] ISO/IEC 25010:2011, Systems and software engineering ― Systems andsoftware Quality Requirements and Evaluation (SQuaRE) ― System andsoftware quality models, Std., 2011.
[5] Y. Jia and M. Harman, “An analysis and survey of the development ofmutation testing,” IEEE TSE, vol. 37, no. 5, pp. 649–678, 2011.
[6] J. A. Jones, M. J. Harrold, and J. Stasko, “Visualization of test information to assist fault localization,” in Proc. ICSE, 2002, pp. 467–477.
[7] M. Y. Chen, E. Kiciman, E. Fratkin, A. Fox, and E. Brewer, “Pinpoint:
Problem determination in large, dynamic internet services,” in Proc.
DSN, 2002, pp. 595–604.
[8] V. Dallmeier, C. Lindig, and A. Zeller, “Lightweight defect localization
for Java,” in Proc. ECOOP, 2005, pp. 528–550.
[9] A. da Silva Meyer, A. A. F. Garcia, A. P. de Souza, and C. L.
de Souza Jr., “Comparison of similarity coefﬁcients used for cluster
analysis with dominant markers in maize (zea mays l),” Genetics andMolecular Biology, vol. 27, no. 1, pp. 83–91, 2004.
[10] R. Abreu, P. Zoeteweij, and A. J. C. van Gemund, “On the accuracyof spectrum-based fault localization,” in Proc. TAIC PART, 2007, pp.
89–98.
[11] Y.-S. Ma, J. Offutt, and Y.-R. Kwon, “MuJava: A mutation system for
Java,” in Proc. ICSE, 2006, pp. 827–830.
[12] R. Just, “The major mutation framework: Efﬁcient and scalable mutation
analysis for Java,” in Proc. ISSTA, 2014, pp. 433–436.
[13] H. Coles, T. Laurent, C. Henard, M. Papadakis, and A. Ventresque, “PIT:
A practical mutation testing tool for Java (demo),” in Proc. ISSTA, 2016,
pp. 449–452.
[14] R. Ramler, T. Wetzlmaier, and C. Klammer, “An empirical study on theapplication of mutation testing for a safety-critical industrial softwaresystem,” in Proc. SAC, 2017, pp. 1401–1408.
[15] H. Coles. PIT. [Online]. Available: https://pitest.org/[16] M. Fowler, Refactoring: Improving the Design of Existing Code.
Addison-Wesley, 1999.
[17] D. Boswell and T. Foucher, The Art of Readable Code: Simple andPractical Techniques for Writing Better Code. O’Reilly Media, 2011.

the faulty statements were the same or increased after the
refactoring except for 𝑚 5 , 𝑚 5 . 𝑠1 , 𝑠2 , and 𝑠7 were at thenesting level 1 and these rScore were always the same in eachmutant. In contrast, 𝑠2 , 𝑠4 , and 𝑠6 were also at the nestinglevel 1 while these rScore were different from each other. The
reason was that the number of test cases that executed thosestatements was different due to the inserted return statements.
This observation implies that applying Early Return reduces
the number of statements with the same suspiciousness; as aresult, SBFL-Suitability is improved.
The presence of Early Return improves SBFL-Suitability.
VI. T HREATS TO VALIDITY
Our experiment selected the ﬁve types of refactorings listed
in Table II. The target programs were simple, which weremanually created as examples of the refactoring patterns. If
we conduct experiments with other types of refactorings orreal programs, we may obtain different results and discover
new characteristics of source code structures.
SBFLScore depends on a test suite and a mutant generator,
as mentioned in Section IV. We selected the eleven mutation
operators listed in the Table I as the mutant generator. If weuse more mutation operators, we may obtain different results.
We also created manually test suites that satisﬁed the condition
coverage of each of the programs and their mutants. There aretest cases that take the same execution path when executing
the test for each mutant. It is possible that such test cases
affect the results of our experiment.
VII. C ONCLUSION
In this paper, we considered that source code has characteristics of how well it is suited to SBFL, and then we deﬁned

706