Revisiting Program Suitability for Fault
Localization with Large Dataset and Various
Mutation Operators
Hikaru Kubo∗ , Yoshiki Higo∗ and Shinji Kusumoto∗∗ Graduate School of Information Science and Technology, Osaka University, Japan{ hkr-kubo@ist., higo@ist., kusumoto.shinji.ist@ } osaka-u.ac.jp

Abstract—Spectrum-Based Fault Localization (in short,
SBFL) is one of the popular techniques to localize faulty
statements of a given program. SBFL utilizes the information
about which statements are executed in each of the successful
or failed test cases. Even if multiple programs have the samefunctionality, the accuracy of SBFL can differ due to theirstructural variations. Thus, changing program structures tobe suitable for SBFL may improve the accuracy of fault
localization while maintaining functionality. In previous
research by Sasaki et al., SBFL-Score was proposed to discoverprogram structures suitable for SBFL. SBFL-Score is one of themetrics used to evaluate how well a program is suitable for
SBFL. Furthermore, the previous research measured SBFLScores for pairs of programs with different structures but thesame functionality, and obtained a program structure suitablefor SBFL. However, a small number of programs and asmall number of mutation operators used in the experiments
were shortcomings. Thus, in this study, we conducted anexperiment with approximately 36 times more programs andabout 2.5 times more mutation operators than in the previous
research. As a result of this experiment, we identified four
new program structures suitable for SBFL.
Index Terms—Spectrum-based Fault Localization, Mutation
Testing, Software Quality

I. Introduction
In software development, debugging is a highly laborintensive and costly task. There is a report suggesting thatdebugging accounts for over half of the costs involved insoftware development [1], [2]. For this reason, there aremany studies supporting debugging. One area of researchin debugging support is Fault Localization, which aims tolocalize faulty statements in a program. In recent years,
Spectrum-Based Fault Localization (SBFL) has been actively
studied [3]. SBFL techniques calculate the likelihood of a fault(henceforth, suspiciousness) for each program statement in agiven faulty program using test results and the information
about which program statements are executed in each test
case (henceforth, execution paths). The basic idea behind
SBFL is that statements executed in many failed tests are
more likely to be faulty, while statements executed in manysuccessful tests are less likely to be faulty.
The accuracy of SBFL is influenced by various factors, suchas types of defects [4] and types of tests [5]. In these factors,
the previous research by Sasaki et al. [6] focused on program
structure.

The previous research proposed SBFL-Suitability as a quality indicator of how well a given program is suitable for
SBFL [6]. Furthermore, the previous research also proposed
SBFL-Score as an evaluation metric for SBFL-Suitability. The
basic idea behind measuring SBFL-Score is to create artificial
defects for a program that passes all test cases intentionally.
By measuring how accurately SBFL can identify these artificial defects, the SBFL-Suitability of the target program canbe evaluated.
The previous research suggested that the accuracy of SBFL
is affected by the program structures, and it is possible toimprove the accuracy of SBFL by changing the program
structure temporarily before executing SBFL. Building on
this concept, the previous study conducted an experimentto identify program structures that are suitable for SBFL
by measuring SBFL-Scores using five pairs of programs withdifferent structures but the same functionality. However, thenumber of programs used for measurement was only ten,and the program structures that are suitable for SBFL were
not sufficiently investigated. Additionally, only 11 types ofmutation operators were used to intentionally create artificial
defects in programs. Due to the small-size experiment, weconsidered that the measured SBFL-Scores are inappropriate
as an evaluation metric of SBFL-Suitability.
To address these shortcomings, we utilize a dataset [7]consisting of a large number of program pairs with different
structures but the same functionality. In this study, we aim
to discover new program structures that are suitable for
SBFL by measuring SBFL-Scores using 365 programs in thedataset. Additionally, we define 16 new mutation operators
to increase the number of mutants. This allows 27 mutation
operators applicable to the programs.
As a result of adding new mutation operators, the numberof mutants increased in approximately 98.6% of the programs.
Moreover, by measuring the SBFL-Scores of the 365 programsand visually checking the results, we identified four new
program structures that are suitable for SBFL.
II. BackGroundA. Spectrum-Based Fault Localization (SBFL)
Fault Localization is one of the techniques used to support debugging, which aims to localize faulty statements ina program. Spectrum-Based Fault Localization (henceforth,test case (input :a,b)(a)
Befor e Refactoring

program(b)
After Refactoring

SBFL) is one of the automated fault localization techniquesthat utilize tests. In SBFL, the spectrum is the execution pathinformation that indicates which statements were executedby each test case. The basic idea behind SBFL is that statements executed by many failed tests are more likely to befaulty, while statements executed by many successful tests
are less likely to be faulty.
We explain the process of identifying faulty statements
using SBFL. First, we execute all test cases and the success or
failure of each test case and the execution path informationis recorded. Next, using this information, suspiciousness iscalculated for each statement. There are various techniques
to calculate suspiciousness. Abreu et al. evaluated the effectiveness of calculation formulas used in SBFL and concluded
that Ochiai’s [8] is the most effective [9].
Formula (1) shows how suspiciousness susp(s) is calculated in Ochiai. Here, f ail(s) represents the number of failed
tests executing statement s, pass(s) represents the numberof successful tests executing statement s, and totalF ail
represents the total number of failed tests.

s 1 : boolean result = false;s 2 : if (0 < a)s 3 : result = true;! s 4 : if (0 <= b) //correct: 0 < bs 5 : result = true;s 6 : return result;s' 2 : if (0 < a)s' 3 : return true;! s' 4 : if (0 <= b) //correct: 0 < bs' 5 : return true;s' 6 : return false;

f ail(s)(1)
totalF ail × (f ail(s) + pass(s))
The suspiciousness is calculated for all statements s; thehigher the value, the more likely it to be faulty.
In calculating suspiciousness, the execution path information of failed tests is the most important factor. That is
because which statements were executed and which were
not in failed tests are major clues to localizing the faulty
statement. This is why the numerator of Ochiai’s formula isf ail(s), emphasizing the importance of the execution pathinformation of failed tests.
B. SBFL-Suitability
SBFL-Suitability was proposed in the previous research [6].
SBFL-Suitability is one of the quality characteristics of aprogram, indicating how well a given program is suitablefor SBFL. Even if two programs have the same functionality
and test suite, differences in program structure can lead tovariations in the accuracy of fault localization using SBFL.
We explain how changes in SBFL-Suitability occur due todifferences in program structure with an example. Programs(a) and (b) shown in Figure 1 have the same functionality but
different structures. When SBFL is applied to both programs
using the test (c) shown in Figure 1, suspiciousness iscalculated for each statement.
In program (a), there are four statements with the same
suspiciousness as the faulty statement. On the other hand,in program (b), there is only one statement with the same
suspiciousness as the faulty statement. The fewer statementswith the same suspiciousness as the location of the defect,
the fewer statements need to be checked. This means higher
accuracy in fault localization using SBFL. Therefore, program(b) has higher SBFL-Suitability than program (a).

t 1 (1,1)t 2 (1,0)t 3 (0,1)

t 4 (0,0)

0.50
0.50
0.00
0.50
0.50
0.50

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓
✓
✓

✓
✓
✓

✓
✓
✓

✓
✓
✓

0.50
0.00

✓
✓

✓
✓

✓

✓

✓

✓

✓

✓

P

F

0.71
0.71
-

test results:

P

P

Fig. 1: SBFL results compared with different program structures
Set of mutants 𝑴𝑮 (𝑷)Mutant 𝑚!

...

Mutant 𝑚"

Generate mutants
Program 𝑃
Mutant generator 𝐺

Execute SBFL foreach mutant

𝑟𝑆𝑐𝑜𝑟𝑒 $ 𝑚%

Test suite 𝑇

𝑺𝑩𝑭𝑳𝑺𝒄𝒐𝒓𝒆𝑻,𝑮 (𝑷)SBFL executer 𝐸

susp(s) = p

susp

𝑟𝑆𝑐𝑜𝑟𝑒 $ (𝑚& )Calculate SBFL score

Fig. 2: Calculation process of the SBFL-Score
C. SBFL-Score
In addition to SBFL-Suitability, we explain the SBFL-Score
proposed in the previous research [6]. The SBFL-Score is
one of the evaluation metrics for SBFL-Suitability. Figure 2
shows the calculation process of SBFL-Score. The basic idea
behind measuring the SBFL-Score is to utilize mutation testing
techniques [10] to intentionally change a given program andcreate artificial defects as many as possible. Intentionallychanged programs are called mutants. By measuring how
accurately SBFL can identify these artificial defects, we canassess how well the entire program is suitable for SBFL.
D. Experiment in the Previous Research
The previous research conducted an experiment to investigate how SBFL-Scores vary due to the differences in program
structures [6].
1) Overview of Experiment: The previous research conducted an experiment to measure SBFL-Scores for five program pairs by using an SBFL-Score measurement tool. Two
programs in each program pair have the same functionality
but different program structures. They compared the SBFLScores of the two programs in each pair and assumed thata program having a higher SBFL-Score is more suitable for
SBFL than the other program in each pair. By inspecting
the executing path of each program, they considered thereason why SBFL-Scores differ from the difference in program
structure.
2) Experimental Targets and Test Suites: In the previous
research, the experimental target was five method pairs andtest suites. The five method pairs were created based on
refactoring patterns by Sasaki et al. The refactoring patterns

selected for this experiment were from the category of ‘Simplifying Conditional Expression’ as classified by Fowler [11].
The test suite for each method was created by Sasaki etal. to satisfy the following three criteria:• each test suite in a method pair has the same test cases,• each mutant fails in any test case, and• condition coverage reaches 100 %.
3) SBFL-Score measurement tool: In the previous researchby Sasaki et al., they developed an SBFL-Score measurement
tool. It consists of a mutant generator and an SBFL executer.
They implemented 11 types of mutation operators listed
in Table I in the mutant generator. These operators wereselected from the default mutation operators provided by
PIT [12], an open-source mutation testing tool. PIT is widelyused in mutation testing for generating mutants [13].
Their tool has a functionality to execute SBFL for mutants
generated by the mutant generator. Their tool uses Ochiai’s
formula (Formula (1)) to calculate the suspiciousness valueof each line in a mutant. Their tool also has a functionality
to save the logs of SBFL such as execution path informationof each mutant and SBFL-Score.
4) Result: The previous research revealed that the fewer
statements exist at the same nesting level, the higher SBFLSuitability tends to be.
They also got two transformation ways of program structure to improve SBFL-Suitability below:• Using return statement to terminate method early.
• Reduce the number of statements in the same nestinglevel.
III. Experimental Setup
A. Chenges from previous research
In this research, we conduct an experiment similar to theprevious research by Sasaki et al. [6].
To reveal new program structures that are suitable for
SBFL, we made some changes to the experimental setup ofprevious research.
1) Definition of Program Structure: In the previous researchby Sasaki et al., they did not define the precise meaning of
program structure. In this research, we define the program
structure of the method as the execution paths for thesame test suite. Even if methods are written differently, if
the execution paths of the two methods are the same, theprogram structures of the two methods are regarded as thesame.
TABLE I: Mutation operators used in the previous research
Mutation Operator
Conditional Boundary
Increments
Invert Negatives
Math
Negate Conditionals
Void Method Calls
Primitive Returns
Empty Returns
False Returns
True Returns
Null Returns

Previous
a<b
n++
-n
a+b
a==b
method();return 5;
return "str";
return true;
return false;
return object;

Later
a<=b
n-n
a-b
a!=b
;
return 0;
return "";return false;
return true;
return null;

s1
s2
s3
s4
s5

Program (Input: a)
if (a == 0)
return 0;
else if (a > 0)
return 1;
return -1;

t1
✓

t2
✓

✓
✓

✓
✓

t3
✓
✓

s′1
s′2
s′3
s′4
s′5

✓

Program (Input: a)
if (a > 0)
return 1;
else if (a < 0)return -1;
return 0;

(a)

t1
✓
✓

t2
✓

t3
✓

✓
✓

✓
✓

(b)

Fig. 3: A method pair with different execution paths

s′1
s′2

Program (Input: a)

t1

t2

t3

if (a>0)
return 1;

✓
✓

✓

✓

✓
✓

✓

Program (Input: a)

t1

t2

t3

s1
s2

if (a>0)
return 1;

✓
✓

✓

✓

s3
s4

if (b<0&&a!=0)
return -1;

✓
✓

✓

s′3
s′4

else if (a<0&&b<0)
return -1;
else

s5

return 0;✓

s′5

return 0;

(a)

✓

(b)

Fig. 4: A method pair with same execution paths

The term "same execution paths" refers to two methods
satisfying the following three conditions, where si represents
the i-th statement in the first method, and s′i represents thei-th statement in the second method.
• The number of all statements in the two methods is thesame.
• The number of statements executed in the same test case(m) is the same in the two methods.
• For all test cases in the test suite, for range in 1 ≤ i ≤ m,when statement si is executed, s′i is also executed.
Figure 3 shows an example of a method pair with different
execution paths, while Figure 4 shows an example of amethod pair with the same execution paths. For the method
pairs with the same execution paths, applied mutation operators are only the factors that affect SBFL-Scores. We considered such method pairs inappropriate for the experiment inthis research.
2) Experimental target and test suite: The previous researchused a dataset consisting of only ten methods that were created manually, which were part of five method pairs. In short,the size of the dataset is small. Thus, the generalizability of
the research result is low. The previous research also has not
thoroughly investigated program structures contributing tohigher SBFL-Scores.
To address this shortcoming, we utilize a dataset [7] that
includes methods with the same functionality implemented
in Java. The dataset consists of 728 methods and test suites.
Each method in this dataset was collected from open-source
software. The methods in the dataset are categorized into
276 groups based on their functionality, with each groupcontaining two to twelve methods. The test suite for eachmethod was automatically generated using Evosuite [14].
We set four conditions for the experimental target.
Cond. 1: The methods within the same group share the samefunctionality.
Cond. 2: The test suite attached to the method reaches 100%of the instruction and condition coverages for the methods.

Cond. 3: The test suite attached to the method does notcontain Flaky Test[15].
Cond. 4: The program structures in the same group are
different from one another.
The Reasons for setting Cond. 2 to 4 are as follows:
Regarding Cond. 2: Ensuring 100% instruction coverageis necessary to calculate suspiciousness values from all thestatements. Ensuring 100% condition coverage is necessary
to attach different suspiciousness to different program statements as much as possible.
Regarding Cond. 3: A Flaky Test is a test case that exhibits
both a passing and a failing result with the same code. If
Flaky Tests exist in a test suite, SBFL-Score of a given method
may vary every time we execute the test suite.
Regarding Cond. 4: The purpose of this research is toreveal what kind of program structures are more suitable for
SBFL, which means that it is not necessary to treat multiple
methods that have the same structure.
The procedure for obtaining methods satisfying all theconditions is as follows:
Step 1: Add some test cases manually.
The test suite for each method was automatically generated using Evosuite [14]. However, there are some cases
where the automatically generated test suites do not achieve
100% instruction and condition coverage. Therefore, we manually add the test cases to ensure 100% instruction coverageand condition coverage.
Step 2: Exclude methods that do not meet the conditions.
We execute the test suite to investigate whether each target
method satisfies Cond. 1 to Cond. 4. As a result, 363 methods
within 145 groups were excluded from the experimental
target. We use 365 methods within 131 groups that satisfy
the four conditions as the experimental target.
3) Mutant generator: The previous research used 11 typesof mutation operators. They implemented the ten target
programs so that these mutation operators could be appliedin many parts of the code, resulting in a sufficient numberof mutants generated from a method. However, it may not
be possible to generate a sufficient number of mutants whenusing other programs. In such cases, SBFL-Score values areless reliable.
To address this shortcoming, we visually inspected theexperimental target and extracted locations where existing
mutation operators cannot be applied. We defined additional
mutation operators shown in Table II to mutate to thelocations we described above. The reasons for selecting thosemutation operators are below.
• Applicable to multiple methods.
• Not necessary to record identifiers to implement amutation operator.
Newly defined mutation operators may generate the same
mutants as existing ones. For example, if there is a statement return false; in a program, both the Change
Boolean Literal and True Return operators cangenerate the same mutant. Thus, we modified the mutant

generator to ensure the same mutant is not duplicately
generated.
To evaluate that the SBFL-Score values become more reliable, we measure the number of mutants generated byadditional mutation operators for each method in the experimental target. It is necessary to consider the size of themethod because as the size of the method increases, thenumber of mutants tends to increase. To evaluate the increasein mutants compared to the method size, we calculate the
AMPL (the number of Additional Mutants Per LOC) metricfor each method. Equation (2) is the definition of AMPL. The
term "Logical LOC" in Equation (2) refers to the number oflines in the method excluding empty lines, comment lines,
and lines containing only brackets. A higher value of AMPL
indicates improved accuracy of SBFL-Scores.
AMPL =The number of additional mutants
Logical LOC of a method(2)

We implemented the 11 mutation operators mentioned
in Section II-D and the additional 16 mutation operators
mentioned in Section III-A3 within the mutant generator. To
measure the AMPL value of each target program, we also
implemented the functionality to switch mutation operators
used. It allows us to record the difference in the number ofmutants before and after adding new operators.
B. The procedure of experiment
We evaluate the increase in mutants and find factors
contributing to higher SBFL-Scores in the following steps:
Step A: Measure SBFL-Scores and AMPLs of the target
methods.
For all the target methods, we measure the SBFL-Scoresand AMPLs using a measurement tool that we developed.
To obtain the execution path information necessary forcalculating SBFL-Scores, we utilized kGenProg [16], an APR
tool. As mentioned in II-D3, the execution path informationof running SBFL for each mutant is saved to files. It allows
us to analyze the factors contributing to higher SBFL-Scoresin Step C.

TABLE II: Additional mutation operators we implemented. If
code exists in the "Later" column separated by commas, itmeans the mutation operator in this row generates multiple
mutants.
Mutation Operator
Change String Literal
Change Instanceof
Nonvoid Method Calls
Constructor Calls
Compound Operator
Change Numeric Literal
Change Boolean Literal
Change Unary Operator
Add Not Operator
More Specific If
Less Specific If
Break and Continue
Null Assignment
Empty Assignment
Primitive Assignment
Change Throw Statement

Previous"String"a instanceof A
a=method()
a=new A()a+=1
if(x<0)true!ismethod()
if(b)
a&&b
a||b
break;
Object x=y;s=a.toString();
int a=b;throw new A;

Later"String1"
a instanceof B
a=nulla=nulla-=1
if(x<1)
false
ismethod()if(!b)
a,b,a||ba,b,a&&b
continue;
Object x=null;s="";
int a=0;
throw new B;

Times
96
192
26
446
90
988
225
13
54
222
180
74
83
91
232
132

Number of Methods

60
50
40
30
20
10
0 ≤ (, ] (, ] (, ] (, ] (, ] (, ] (, ] (, ] (, ] (, ] (, ] (, ] (, ] (, ] (, ] (, ] (, ] (, ] (, ] (, ] (, ] (, ] (, ] (, ] (, ] (, ] (, ] (, ] (, ] (, ]
0
1
2
3
4
5
6
AMPL

Fig. 5: Histogram of AMPL values of the experimental target

Step B: Exclude groups where no differences in SBFLScores from the subject of consideration.
There are some groups that have methods with thesame SBFL-Scores, even though the program structures differ
within the groups. Those groups are not necessary to analyze
factors contributing to higher SBFL-Scores. We exclude thosegroups from Step C.
Step C: Classify factors contributing to higher SBFLScores and analyze the reasons.
We visually inspect all methods in each group and record
the differences in program structure. Then, we classify
the differences in program structure into several structuregroups. Next, we check the significance of the difference in
SBFL-Scores for each classified structure group by using astatistical approach. We use Wilcoxon’s signed-rank test [17]because SBFL-Scores do not follow a normal distribution.
Finally, we visually inspect the execution path informationfor each mutant. By comparing the SBFL-Scores and execution
path information among the methods, we analyze the reasons
why the factors contribute to higher SBFL-Scores.
IV. Results
A. The increase in mutants
Fig. 5 shows the Histogram of AMPL values of the experimental target. Out of the 365 methods, 359 methods have
AMPL values greater than 0. The "Times" column in Table IIshows the number of times each additional mutant is applied.
According to these results, it is considered that the numberof mutants increased overall because of additional mutation
operators.
B. Factors contributing to higher SBFL-Scores
Out of the 131 groups, 119 groups have differences in SBFLScores in the same group.
There are 12 groups that have no differences in SBFLScores. The reasons are as follows:• There are no conditional expressions in any methods inthe group.
• Only the difference between the methods in the samegroup is that multiple conditions of if statements
appear in a different order.
By inspecting the 119 method pairs manually, we revealed
the following four new factors contributing to higher SBFLScores:

Factor 1: Using control statements instead of ternary operators or lambda expressions.
Factor 2: Using additional if statement for applying Early
Return [18].
Factor 3: Using multiple if statements instead of an if
statement with a logical operator.
Factor 4: No possibility that exceptions occur in the firststatement.
There are some groups that satisfy the factor discoveredin the previous research, which is having a low number ofstatements in the same nesting level.
The factors contributing to higher SBFL-Scores and thenumber of groups that satisfy the factors are listed inTable III.
C. Significance of the difference in SBFL-Scores
Figure 6 (a)-(f) shows box-and-whisker plots with individual data points showing the differences in SBFL-Scores
between the presence and the absence of each of the fivefactors, including the four newly discovered factors and one
factor discovered in the previous research. Lines between
points indicate the difference of SBFL-Scores in the samegroup. To check the significance of the differences in SBFLScore, we used Wilcoxon’s signed-rank test and calculated pvalues. As can be seen from Figure 6, for factors other thanFactor 4, which has a small number of samples, the differencein SBFL-Scores between the presence and the absence of thefactor was significant (p < 0.05).
D. Factors contributing to higher SBFL-Scores
We explain why each factor contributes to higher SBFLScores with examples below.
Factor 1: Using control statements instead of ternary
operators or lambda expressions.
There are several ways to write conditionals and loops.
For example, in Figure 7, method (b) includes an if
statement. As a result, there are differences in the execution
paths when running a test suite, leading to a non-zero SBFLScore. On the other hand, method (a) contains a ternary
operator and does not contain any control statements. In suchcases, there are no variations in the suspiciousness of eachstatement. Consequently, it leads SBFL-Score of 0. Therefore,
by using control statements instead of ternary operators orlambda expressions, SBFL-Scores are improved.
Factor 2: Using additional if statement for applying
Early Return.
TABLE III: The number of groups classified as each factor
Factor
Factor 1
Factor 2
Factor 3
Factor 4
Factor discovered in the previous research
Unknown

Number of Groups
20
8
11
4
63
13

0.00

0.25

0.50
SBFLScore

0.75

yes
no

1.00

0.4

0.5

0.6
SBFLScore

0.7

0.8

A

no

no
0.25

0.50
SBFLScore

no
0.6

0.7
SBFLScore

0.8

0.9
C

0.75

conditionno(c) Factor 3

yes

7.8e−11

yes

0.00

yes

0.5

(b) Factor 2

yes

Condition

condition

yes

B

0.12

Condition(a) Factor 1

no

0.014

no

condition

yes

0.0058yesnoCondition

condition

yes

Condition

no0.00032

Condition

condition

yes
no
0.00

0.25

0.50
SBFLScore

0.75

1.00

D

(d) Factor 4E(e) Factor discovered in the previous research

Fig. 6: Distribution of the differences in SBFL-Scores between the presence and the absence of each factor

1 return a > b ? 1 : 0;

1 if(a > b)
2
return 1;
3 else4
return 0;

(a)

s1
s2
s3
s4

Program (Input: p)
int s = 0;
for (int i=0; i<L; i++)s -= p[i]; //bug here
return 0;

t1
✓
✓
✓
✓

t2
✓
✓
✓
✓

t3 susp rScore✓ 1.000.33✓ 1.00
0.33
0.58
0.00
✓ 1.00
0.00

t1
✓
✓

t2
✓
✓

t3
✓
✓
✓

✓
✓
✓

✓
✓
✓

(b)
(a)

Fig. 7: A method pair indicating the difference between thepresence and absence of control statement

1 int s = 0;
2 for (int i = 0; i < L; i++)3s += p[i];4 return s;

(a)

1
2
3
4
5
6

if (L < 1)
return 0;
int s = 0;
for (int i = 0; i < L; i++)s += p[i];return s;

(b)

s1
s2
s3
s4
s5
s6

Program (Input: p)
int s = 0;
if (L < 1)
return 0;
for (int i=0; i<L; i++)s -= p[i]; //bug here
return s;(b)Fig. 9: Example of differences in execution path between thepresence and absence of Factor 2.

Fig. 8: A method pair indicating the difference between thepresence and absence of Early Return

In some cases, the return value of a method can be determined based on the input before executing all the program
statements in the method.
For example, in Figure 8, method (a) contains a for loop toprocess the contents of an array. On the other hand, method(b) also contains an if statement to return 0 if the array
length is 0.
By using if statement to apply Early Return, some test
cases execute the newly added if statement and do notreach for loop. Therefore, if a faulty statement exists infor loop, the number of successful tests that execute faulty
statement decreases. This leads to an increase in the rScoreand an increase in the SBFL-Score (Figure 9(b)). Therefore, byusing if statement to apply Early Return, SBFL-Scores are
improved.
Factor 3: Using multiple if statements instead of an
if statement with a logical operator.
There are two ways to write a conditional branch withmultiple conditional expressions: connecting conditional expressions with the logical operator or dividing conditional
expressions into separate if statements.
For example, in Figure 10, method (a) contains a single
if statement with two conditional expressions connected by
the logical OR operator (||). On the other hand, method(b) contains two if statements with a single condition

susp rScore
0.82
0.20
0.82
0.20
0.00
0.00
1.00
0.60
1.00
0.60
1.00
0.60

1 if (i < s || i > e)2
return -1;
3 return 0;(a)

1 if (i < s)2
return -1;
3 else if (i > e)4
return -1;
5 return 0;(b)Fig. 10: A method pair indicating the difference between thepresence and absence of a logical OR operator

expression.
Even if there is a fault in the right side of conditional
expressions connected by the logical OR operator, a test casethat checks the left side of conditional expression becomesa successful test if the left side of conditional expression
is correct. Thus, the statement is executed frequently insuccessful tests. As a result, the suspiciousness of the faulty
statement decreases, leading to an increase in the ranking of
suspiciousness for other statements that are not faulty (Figure11(a)). This leads to a decrease in rScore and a decrease in
the SBFL-Score. Therefore, by separating condition expressions instead of connecting them using logical operators,
SBFL-Scores are improved (Figure 11(b)).
Factor 4: No possibility that exceptions occur in thefirst statement.
There are test cases that expect the occurrence of an
exception [19]. These test cases may be used to execute
SBFL. The SBFL-Score measurement tool used in this study

t1
✓
✓
✓

Program (Input: s,e)
int i = 0;
if (i < s || i <= e) // bug here
return -1;
return 0;

s1
s2
s3
s4

t2
✓
✓

t3
✓
✓
✓

✓

t4
✓
✓
✓

susp rScore
0.71
0.66
0.71
0.66
0.47
0.00
0.50
0.33

(a)
s1
s2
s3
s4
s5
s6

Program (Input: s,e)
int i = 0;
if (i < s)return -1;
else if (i <= e) // bug here
return -1;
return 0;

t1
✓
✓

t2
✓
✓

✓
✓

✓

t3
✓
✓
✓

t4
✓
✓
✓

✓

susp rScore
0.71
0.60
0.71
0.60
0.00
0.00
1.00
1.00
0.50
0.20
0.50
0.20

Thus, when running test cases that expect an exception, it isdetermined that line 2 is executed.
If there is a fault after line 3, the test case providing
s=null always results in an exception occurring on line
3. In this case, test cases expecting an exception become
successful tests. As a result, the number of successful tests
executing a faulty statement decreases (Figure 13(b)). This
leads to an increase in rScore and an increase in SBFL-Score.
V. Threat to Validity
A. Internal threats(b)

Fig. 11: Example of differences in execution path between thepresence and absence of Factor 3.
1 String t(String s, Object... args) {2
StringBuilder b = new StringBuilder(s.length() + 16 * args.length);//Exception occurs when s=null
3
...
4 }

(a)
1 String t(String s, Object... args) {2
s = String.valueOf(s);3StringBuilder b = new StringBuilder(s.length() + 16 * args.length);//Exception occurs when s=null
4
...
5 }

(b)

Fig. 12: A method pair indicating the difference of thelocation of statements where the exception occurs

s1
s2
...
s7

Program (Input: s,t)StringBuilder b = new StringBuilder(s.length() + 16 * args.length);
int templateStart = 1; //bug here...
return b.toString();

t1

t2
✓

t3 susp rScore✓ 1.00
0.00

✓
...
✓

✓ 1.00...
...
✓ 1.00t2✓✓

t3 susp rScore✓ 0.870.00✓ 1.00
0.14

✓
...
✓

✓ 1.00...
...
✓ 1.00

0.00
...
0.00

(a)
s1
s2
s3
...
s8

Program (Input: s,t)s = String.valueOf(s);StringBuilder b = new StringBuilder(s.length() + 16 * args.length);
int templateStart = 1; //bug here...
return b.toString();

t1
✓

0.14
...
0.14

There are some kinds of granularities, such as statements
and code blocks, used in SBFL [20]. In this study, weused statements as a granularity of SBFL. However, if we
use another granularity, there is a possibility of producing
different results.
Measurement results of SBFL-Scores are influenced by the
choice of test suites and a mutant generator. Thus, using
different test suites and a mutant generator can lead tovariations in SBFL-Scores and produce different results.
The code coverage tool used in this study determinesthat when an exception occurs, the statement where theexception occurs is treated as unexecuted. If a different tool isused for collecting code coverage information, the statement
where the exception occurs might be treated as unexecuted,
potentially negating the experimental results, in particular
Factor 4 in Section IV-B.
B. External threats
In this research, we used a dataset created by Higo etal. [7]. This dataset includes small programs containing
a single method. If we use programs that have different
characteristics from the ones used in this study, such as large
programs containing multiple methods, there is a possibilityof producing different results.
As described in Section III-A, we set four conditions forthe experimental target and excluded programs that did notmeet these conditions. As a result, approximately half of the
programs were excluded from the experimental targets. Thus,
there is a lack of generalizability in the experimental results.
VI. Related Work(b)

Fig. 13: Example of differences in execution path between thepresence and absence of Factor 4.

determines that when an exception occurs, the statement
where the exception occurs is treated as unexecuted. Thus,
this specification affects SBFL-Scores.
For example, assume the case of the two methods shown in
Figure 12. In Method (a), if the input is s=null, an exception
occurs in line 2. Thus, when running test cases that expect
an exception, it is determined that none of the statementsin the method are executed. On the other hand, in method(b), if the input is s=null, an exception occurs in line 3.

SBFL is one of the most popular techniques in fault
localization and it has been actively studied in recent years[20], [21].
In this Research, we revealed four program structures toimprove the accuracy of SBFL. There is some research about
the quality including the accuracy of SBFL. Abrew et al.
researched some influences on the accuracy of SBFL [22].
They showed that Ochiai’s formula consistently outperforms
the other formulas. Furthermore, they showed that a limited
number of failing tests is optimal, and additional failing test
cases do not affect the accuracy of SBFL. Golagha et al.
introduced a technique of predicting the quality of SBFL
by using 70 static, dynamic, test suite, and fault-related
metrics [23]. Their study showed that it is not necessary toexecute all test cases before applying SBFL. However, theyalso found that applying SBFL right after the first failed testis less effective than applying it after executing all tests for
multi-location bugs, which is contrary to the single-location
bug study.
There is also some research about improving the accuracyof SBFL. Zhang et al. introduced a technique of improving theaccuracy of SBFL by using the PageRank algorithm [24]. The
experimental results of this research demonstrate that theirtechnique can outperform state-of-the-art SBFL techniques
significantly. Hongdou He et al. introduced a technique for
improving the accuracy of SBFL by using Fault Influence
Propagation [25]. They conducted an experiment on the realworld fault dataset Defects4J [26] with 33 raw spectrumbased fault locators, which proves that the proposed approach
improves the baseline 14.9% average at the top-5 position.
VII. Conclusions and Future WorkIn this study, we addressed the shortcomings of previous
research [6], namely the small number of programs investigated and the small types of mutation operators. We conducted experiments to measure SBFL-Scores and investigatedprogram structures that are suitable for SBFL. As a result of
the investigation, we identified four new program structuresthat are suitable for SBFL.
There are two future challenges below:
Development of an Automated Program Transformation Tool
Both the previous research [6] and our study have identified program structures with high SBFL-Suitability. Thus, it
is possible to develop a tool that automatically transforms
programs to improve their SBFL-Suitability. Furthermore,by using the developed tool, it is possible to conduct anexperiment to determine if the accuracy of fault localization
improves when applying the transformations to programswith existing defects.
Investigation of the Relationship between SBFLSuitability and Other Quality Characteristics
While improved SBFL-Suitability, there may be a decreasein maintainability. Actually, the program in Figure 11(b)that is suitable for SBFL contains code clones (Line 2 andLine 4). Therefore, it is a significant challenge to investigate
a relationship between SBFL-Suitability and other quality
characteristics, such as maintainability.
Acknowledgement
This research was supported by JSPS KAKENHI Japan(JP21K18302, JP21H04877, JP22H03567, JP22K11985).
References[1] B. Hailpern and P. Santhanam, “Software debugging, testing, andverification,” IBM Systems Journal, vol. 41, no. 1, pp. 4–12, 2002.
[2] T. Britton, L. Jeng, G. Carver, and P. Cheak, Quantify the time and costsaved using reversible debuggers. Technical report, Cambridge Judge
Business School, 2012.
[3] W. E. Wong, R. Gao, Y. Li, R. Abreu, and F. Wotawa, “A survey on
software fault localization,” IEEE Transactions on Software Engineering,
vol. 42, no. 8, pp. 707–740, 2016.

[4] F. Keller, L. Grunske, S. Heiden, A. Filieri, A. van Hoorn, and D. Lo, “A
critical evaluation of spectrum-based fault localization techniques on alarge-scale software system,” in Proc. IEEE International Conference on
Software Quality, Reliability and Security, 2017, pp. 114–125.
[5] B. Vancsics, T. Gergely, and A. Beszedes, “Simulating the effect of test
flakiness on fault localization effectiveness,” in Proc. IEEE Workshop on
Validation, Analysis and Evolution of Software Tests, 2020, pp. 28–35.
[6] Y. Sasaki, Y. Higo, S. Matsumoto, and S. Kusumoto, “Sbfl-suitability: A
software characteristic for fault localization,” in Proc. IEEE International
Conference on Software Maintenance and Evolution, 2020, pp. 702–706.
[7] Y. Higo, S. Matsumoto, S. Kusumoto, and K. Yasuda, “Constructing
dataset of functionally equivalent java methods,” in Proc. the International Conference on Mining Software Repositories, 2022, pp. 682–686.
[8] R. Abreu, P. Zoeteweij, and A. J. Van Gemund, “An Evaluation of
Similarity Coefficients for Software Fault Localization,” in Proc. Pacific
Rim International Symposium on Dependable Computing, 2006, pp. 39–
46.
[9] R. Abreu, P. Zoeteweij, R. Golsteijn, and A. J. C. van Gemund, “A
Practical Evaluation of Spectrum-based Fault Localization,” Journal of
Systems and Software, vol. 82, no. 11, pp. 1780–1792, 2009.
[10] Y. Jia and M. Harman, “An analysis and survey of the development ofmutation testing,” IEEE TSE, vol. 37, no. 5, pp. 649—678, 2011.
[11] F. Martin, Refactoring: Improving the Design of Existing Code. AddisonWesley, 1999.
[12] H. Coles, T. Laurent, C. Henard, M. Papadakis, and A. Ventresque,“PIT: A Practical Mutation Testing Tool for Java (Demo),” in Proc.
International Symposium on Software Testing and Analysis, 2016, pp.
449–452.
[13] J. Xuan and M. Monperrus, “Test case purification for improving
fault localization,” in Proc. International Symposium on Foundations ofSoftware Engineering, 2014, pp. 52–63.
[14] G. Fraser and A. Arcuri, “EvoSuite: automatic test suite generationfor object-oriented software,” in Proc. Symposium and the European
Conference on Foundations of Software Engineering, 2011, pp. 416–419.
[15] Q. Luo, F. Hariri, L. Eloussi, and D. Marinov, “An Empirical Analysis
of Flaky Tests,” in Proc. International Symposium on Foundations ofSoftware Engineering, 2014, pp. 643–653.
[16] Y. Higo, S. Matsumoto, R. Arima, A. Tanikado, K. Naitou, J. Matsumoto,
Y. Tomida, and S. Kusumoto, “kgenprog: A high-performance, highextensibility and high-portability apr system,” in Proc. the Asia-Pacific
Software Engineering Conference, 2018, pp. 697–698.
[17] F. Wilcoxon, “Individual comparisons by ranking methods,” in Breakthroughs in Statistics: Methodology and Distribution. Springer, 1992,
pp. 196–202.
[18] M. A. Saca, “Refactoring improving the design of existing code,” inProc. IEEE Central America and Panama Convention, 2017, pp. 1–3.
[19] F. Dalton, M. Ribeiro, G. Pinto, L. Fernandes, R. Gheyi, and B. Fonseca,“Is exceptional behavior testing an exception? an empirical assessment
using java automated tests,” in Proc. Int’l Conf. on Evaluation andAssessment in Software Engineering, 2020, pp. 170–179.
[20] Q. I. Sarhan and A. Beszedes, “A survey of challenges in spectrum-based
software fault localization,” IEEE Access, vol. 10, pp. 10 618–10 639, 2022.
[21] H. A. de Souza, M. L. Chaim, and F. Kon, “Spectrum-based software
fault localization: A survey of techniques, advances, and challenges,”
arXiv preprint arXiv:1607.04347, 2017.
[22] R. Abreu, P. Zoeteweij, and A. J. van Gemund, “On the accuracy ofspectrum-based fault localization,” in Testing: Academic and Industrial
Conference Practice and Research Techniques, 2007, pp. 89–98.
[23] M. Golagha, A. Pretschner, and L. C. Briand, “Can we predict the qualityof spectrum-based fault localization?” in IEEE International Conference
on Software Testing, Validation and Verification, 2020, pp. 4–15.
[24] M. Zhang, X. Li, L. Zhang, and S. Khurshid, “Boosting spectrum-based
fault localization using pagerank,” in Proc. ACM SIGSOFT International
Symposium on Software Testing and Analysis. Association for Computing Machinery, 2017, pp. 261–272.
[25] H. He, J. Ren, G. Zhao, and H. He, “Enhancing spectrum-based fault
localization using fault influence propagation,” IEEE Access, vol. 8, pp.
18 497–18 513, 2020.
[26] R. Just, D. Jalali, and M. D. Ernst, “Defects4j: A database of existingfaults to enable controlled testing studies for java programs,” inProceedings of the international symposium on software testing andanalysis, 2014, pp. 437–440.