Does Refactoring Break Tests and to What Extent?
Yutaro Kashiwa∗ , Kazuki Shimizu∗ , Bin Lin†
Gabriele Bavota† , Michele Lanza† , Yasutaka Kamei∗ , Naoyasu Ubayashi∗ ,∗ Kyushu University, Japan — † Software Institute – USI, Lugano, Switzerland

Abstract—Refactoring as a process is aimed at improving thequality of a software system while preserving its external behavior. In practice, refactoring comes in the form of many specificand diverse refactoring operations, which have different scopes
and thus a different potential impact on both the production
and the test code. We present a large-scale quantitative study
complemented by a qualitative analysis involving 615,196 test
cases to understand how and to what extent different refactoring
operations impact a system’s test suites. Our findings show thatwhile the vast majority of refactoring operations do not or very
seldom induce test breaks, some specific refactoring types (e.g.,“R ENAME ATTRIBUTE” and “R ENAME C LASS”) have a higher
chance of breaking test suites. Meanwhile, “A DD PARAMETER”
and “C HANGE R ETURN T YPE” refactoring operations often
require additional lines of changes to fix the test suite they break.
While some modern IDEs provide features to automatically apply
these two types of refactoring operations, they are not always able
to avoid test breaks, thus demanding extra human efforts.

I. I NTRODUCTION
Refactoring is a good practice, leading to improved code
quality, higher developer productivity, and less error prone
code [1] [2]. This intuition is supported by previous studies [3][4]. By definition, refactoring does not change theexternal behavior of a system. However, refactoring can break
test suites because it can modify the internal behavior whichis tested in the test suites.
Test suites can reach a considerable size in modern softwaredevelopment. Fixing them can be an extensive and timeconsuming task [5]. Vassallo et al. [6] surveyed continuous
refactoring barriers and indicated that one of the reasonsdevelopers hesitate to refactor code is the possibility to break
test suites. The problem is exacerbated by the recent adoptionof continuous integration and deployment practices and ingeneral fierce market competition [7].
Many studies [8][9][10][11] investigated the impact of
refactoring on test code. Rachatasumrit et al. [8] claimed that
half of the broken test code involves refactoring edits. Still,
it is not clear how refactoring breaks test suites because they
did not conduct a qualitative analysis. Indeed, in their work,
they indicate that the refactoring operations involved in thefailed tests might not be the cause. To unveil the impact of
refactoring on test suites, this study sets out to address thefollowing central question:“Does refactoring break tests and to what extent?”
We conducted a conceptual replication[12][13] of the study
by Rachatasumrit and Kim [8], exploiting state-of-the-art
mining techniques to run a more fine-grained analysis. Specifically, we use commit histories rather than released versions.

Such a change in granularity reduces the size of each change
we need to inspect, which makes it easier to identify thecause of the failures during qualitative analysis. However,
applying dynamic trace tools as in the mentioned study isexpensive since it requires long execution time, heavy memory
usage, etc. [14]. Running such an analysis to every revision ineach studied repository further worsens the issue. Leveraging
modern technology, we exploit Kubernetes [15] to run our
analyses through several computational instances in parallel.
Our experiments involve 615,196 test methods taken from8 projects, and a total of 22,000 computation hours in thecontainers. The research questions and results are as follows:
RQ1 : What types of refactoring break test suites?
Most of the refactoring types, as expected, are not observed
to break test methods but Compiler-Errors can be triggeredby refactoring operations that modify method signatures (e.g.,
M OVE C LASS, A DD PARAMETER). Failures are rarely causedby refactoring, but sometimes they do occur when C HANGE
R ETURN T YPE and A DD PARAMETER are performed. A previous study [8] reported that many test methods are broken byrefactoring operations. We found that only 2.5% of refactoring
operations break test suites.
RQ2 : What is the magnitude of the fix triggered bydifferent types of refactoring on test methods?
We observed that the efforts needed to fix test breaks
triggered by refactoring vary for different refactoring types.
However, while A DD PARAMETER and C HANGE R ETURNT YPE seldom break tests, they are more likely to have a wider
impact than other types of refactoring, requiring 4-12 lines ofchanges to fix the tests they break.
Discussion: Can IDEs help in reducing the chances ofbreaking tests during refactoring activities?
IDEs automate the mechanical part of refactoring by, forexample, consistently changing all references to a variable
when its identifier is renamed. We found that they can help
to avoid test breaks triggered by refactoring involving rename
or move actions. However, they did not prevent test breaks
triggered by C HANGE R ETURN T YPE, C HANGE PARAMETERT YPE, or A DD PARAMETER.
Paper Organization. Section II introduces our researchquestions and discuss the related literature. Section III describes our study design, while Section IV discusses theachieved results, thus answering our research questions. In
Section V we conduct additional analyses and discuss thebroader implications of our results. Finally, Section VI summarizes our findings and discusses future work.

II. BACKGROUND AND R ELATED W ORKWhen developers modify their source code, they run theirtest suites to detect regressions. Past studies [16][17] havedeveloped approaches to detect the test methods affected bychanges in the source code since exercising the whole test
suite can be time-consuming [18][19][20]. These approaches
can precisely identify test methods that should be run, which
reduces the time for running test suites. However, the modified
source code is not always in a state so that it can be built. It
has been reported that changes in the production code break
test suites in many cases [8][21].
Tang et al. [21] examined what percentage of bug-fixing
changes cause test failures. They reported that 48.7% ofbug-fixing changes would break regression test suites. Thus,
developers will recognize that they need time to fix test suites
after modifying the production code, which disrupts theirdevelopment schedule.
Even refactoring, which should not change the behavior ofa system, is known to break test suites. Rachatasumrit andKim [8] examined what proportion of failed tests are relevant
to refactoring edits. They showed that half of the failed tests
involve refactoring edits in the production code. The previous
studies deepen the understanding of the impact that changes
have on test suites, which fosters the development of automatic
repair tools for test suites. However, there are still missing
pieces of evidence that we aim at building in our study:1) It is uncertain how refactoring breaks test suites:
Rachatasumrit and Kim [8] found that refactoring operations
might lead to failed tests. It is however unclear whether
the refactoring edits are the real causes of the test failures,
since they are often part of tangled commits also involving
other types of changes. To identify the real cause, a manual
inspection would be needed. However, such an inspection
would be challenging on those data since the refactoring edits
have been extracted between releases, and the numbers of both
refactoring and non-refactoring related code changes involving
a test method are tremendous.
To overcome this issue, we mine the complete change
history of the studied systems at commit level, thus obtaining
much smaller changes. Also, we study the impact of a single
and isolated refactoring operation on the test methods exercising the refactored production code. This has been possible
thanks to novel mining tools developed in the last years andallows our study to be more fine-grained and involving ahigher number of data points. Also, the precise data extracted(i.e., a single refactoring operation impacting tests) allow fora simpler manual inspection that can validate the impact of
refactoring on the test.
This can be summarized by the following research question:
RQ1 : What types of refactoring break test suites?
2) The fixing cost of a broken test is not a constant:
The tests broken by different refactoring types may trigger adifferent fixing cost.

For example, A DDING A PARAMETER to a method may
require the writing of several new lines in the test method toprepare the additional input represented by the new argument.
Instead, a R ENAME C LASS refactoring may only impact asingle line in the test method (i.e., the change to the classname).
A study by Elish and Alshayeb [10] estimated the impactof refactoring operations on the testing effort required by the
refactored code components. Such an impact is estimated interms of internal quality metrics that are measured before andafter the refactoring and is based on the findings by Bruntink
and van Deursen [22] showing that changes in internal quality
metrics result in changes in the testing effort. Elish andAlshayeb found that E NCAPSULATE F IELD and E XTRACTM ETHOD increase the testing effort. Differently from ourstudy, they do not assess the size of the fix triggered in testcode by refactoring operations.
Besides the work by Elish and Alshayeb [10], it is worth
mentioning the several previous studies analyzing the impactof refactoring on quality attributes. Stroggylos and Spinellis [23] mined refactoring operations from version control
system logs of three open-source libraries with the goal of
studying their impact on the values of nine object-oriented
quality metrics. Their results show the possible negative effects
that refactoring can have on some quality metrics. Similarly,Chávez et al. [24] and Cedrim et al. [25] reported thatrefactoring not always result in an increase of code quality.
Such a research question has also been investigated by Szoke
et al. [26], who reported that small refactoring operations
performed in isolation rarely impact software quality, whilea high number of refactorings performed in a block can resultin notable code quality improvement.
Moser et al. [27] conducted a case study in a close-to
industrial environment to investigate the impact of refactoringon the productivity of an agile team. The achieved results showthat in the context of mobile apps development, refactoring
increases developers’ productivity.
Previous work also analyzed the extent to which refactoring
activities induce faults [28]. Authors showed that refactorings involving hierarchies (e.g., Push Down Method) induce
faults more frequently than others that are likely to beharmless in practice. This finding highlights the strong bond
between refactoring and (regression) testing, that also pushed
researchers to propose techniques aimed at recommending
refactoring solutions (e.g., to remove antipatterns) by keeping
low the testing effort required after refactoring [29]. On a
related research thread, it is also worth mentioning the study
by Sabané et al. [30], that showed the higher testing cost ofclasses affected by antipatterns.
Related to the above-described work, we formulate thefollowing research question for our study:
RQ2 : What is the magnitude of the fix triggeredby different types of refactoring on test methods?

TABLE I
S UMMARY OF THE SYSTEMS UNDER STUDY.
Project

DomainC OMMONS - IO
S PRINGJ ODA - BEANS
J SOUPS PARKL ITTLE P ROXYR XJAVA - JDBCS POONIO functional library
SQL mapping framework
Code generator framework
HTML library
Web framework
HTTP proxy
Database client library
Program analysis library

Commits
2,740
1,615
840
1,386
1,062
1,003
850
3,278

History
Refactoring
commits
444
155
279
289
309
252
186
878

III. S TUDY D ESIGN
The goal of our study is to investigate the extent to which
refactoring operations break the related test suites, by answering the two research questions we formulated in Section II.
The basic idea behind our study is to simulate a regression
testing scenario during a refactoring process: Once the codeis refactored, developers can check whether bugs have been
introduced by running (part of) the test suite. To reproduce
such a scenario, we take the production code refactored incommit ci and we run on it the test suite before any change
triggered by the refactoring has been implemented by thedeveloper. To do so, we run on the production code in ci thetest suite in the previous revision (ci−1 ). In particular, as what
was done in the previous studies [8][21], we first checkout thetwo snapshots ci and ci−1 . Then, we replace the test directoryin ci with the one in ci−1 . Finally, we identify tests broken(i.e., tests that fail or that exhibit compilation errors) by the
refactoring operation.
Below we detail our study design, explaining how we collectand analyze the data to answer our research questions.
A. Context Selection
We selected as context for our study eight projects from
GitHub (summarized in Table I) satisfying the following
criteria.
(Criterion 1) Java system: In this study, we use a state-ofart refactoring mining tool [31] that only works on Java
code.
(Criterion 2) JUnit: There are many testing frameworks forJava systems, such as JUnit [32] and TestNG [33]. Wetarget projects that use JUnit 4 or 5, which is an opensource testing framework and a de facto standard for Javaprojects [34].
(Criterion 3) Maven project: We must compile production
and test code and run test code for each commit, which iscomputationally expensive. In this study, we target maven
projects to easily automate these processes. Maven [35]is a popular build tool widely adopted for Java systems.
(Criterion 4) Simple Project: Projects sometimes have subprojects depending on each other. When a sub-project isbroken, the project build might fail. To avoid this, we
decided to exclude projects that contain sub-projects.

Refactoring
instances
4,064
782
3,515
1,358
1,789
1,616
849
9,564

LOC
(Production)
13,736
1,924
17,068
12,570
6,253
4,180
4,611
64,038

Latest snapshot
LOC(Test)
24,714
3,933
34,185
9,201
5,045
4,665
3,330
44,960Test
methods
1,310
120
444
725
320
55
74
1,938

B. Data Collection
To determine which test methods are affected by eachrefactoring operation performed in a software repository, it isnecessary to verify if each test method exercises the production
code where refactoring operations have been performed. Sucha procedure, depicted in Figure 1, consists of three main steps
that we detail in the following: “Refactoring Detection”, “TestRun”, and “Impact Analysis”.
Refactoring Detection. We detect refactoring operations
performed in the production code of each commit of thestudied repositories. This has been done by using RefactoringMiner (ver. 2.0.3) [31], the state-of-the-art refactoring detector.
RefactoringMiner provides, together with each detected refactoring, also exact information about the code lines impacted byit. Note that the location of the refactoring can be a single file(e.g., in the case of rename operations) as well as multiple
files (e.g., a method moved from a source to a target file).
RefactoringMiner has been shown to be able to detect 55 typesof refactoring operations with higher precision and recall thanother tools [31][36].
We run RefactoringMiner on every commit in all branches1of the cloned repositories to obtain the type and location of theperformed refactorings. Note that common commits present in
multiple branches are considered only once. Merge commits
are excluded since they generate duplicated results.
Test Run. To determine whether a specific test method was
broken by refactoring edits, we must know which lines in the
production code are exercised by the test method. Indeed, in
our study, we consider a refactoring as responsible for breaking
a test method t only if there is an overlap between the lines of
production code impacted by the refactoring (as reported by
RefactoringMiner) and the ones exercised by t. To establish
such a link, we employ a dynamic execution trace method thatprovides us with the production code paths executed by eachtest method.
To run tests with Maven, the Surefire plugin is needed. Mostof the maven projects have it, but some projects do not (theytest the product on their local environments without continuous
integration). We inserted the plugin setting in the build files(pom.xml) to automate testing.
1 We decided to use all branches because Kovalenko et al. [37] claimed
that using only partial history distorts results.

①
Refactoring
Detection＋ーRevision X-1

diff files

Location
BeforeAfter
Change Change

Refactoring information

Revision X

Commit History

testMethodA() [Line 1]…productionMethodA [Line 1]…productionMethodA [Line 2]②
Run Tests…

Repository

Refactoring
type

Run Tests

testMethodA() [Line 1]…productionMethodA [Line 1]……extractedMethod [Line 1]…productionMethodA [Line 2]

Snapshots(Revision X-1)③
ImpactAnalysis

Trace log…

ImpactAnalysis…

Snapshots(Revision X)

testMethodA() :Extract Method
testMethodX() :Change Return type

Trace log

Outputs
Test methods andRefactoring typeFig. 1. Overview of the study design. Goal: Identifying if each test method in a repository is affected by a refactoring operation. Our approach extracts two
snapshots and runs tests on them, using a dynamic execution analyzer, and examines if there are refactoring edits impacting the traces.

By default, maven testing stops if a test method fails. Since
we need to run all tests even if a test method fails, we enabled
the “testFailureIgnore” option in the Surefire plugin.
Also, running the tests requires to identify them in theanalyzed repositories. If the pom file of a project did notcontain specified production and test directories, maven default production and test directories (i.e., “src/main/java” and“src/test/java”) are used. Test methods are identified using the@Test JUnit annotation. We regard these test methods as test
cases and do not distinguish unit tests or system tests.
Dynamic trace execution methods are known for being
costly. To reduce the required time, memory, and disk space we
use SELogger [38]. SELogger collects most of the execution
paths but omits redundant paths (i.e., for/while loops). We add
the dependency on SELogger into the build file of the studied
projects so that SELogger can be executed during test runs.
With the updated file, we first compile the test code for eachcommit on which we detected at least one refactoring. Tests arerun both on the snapshot before and after the refactoring (i.e.,ci and ci−1 ). If a test method causes compilation errors, weremove it and the remaining test methods get compiled again.
The removed methods are recorded and this information isused to count compiler errors in RQ1 . We repeat this removing
and compiling process until all of the tests are successfullycompiled. After that, we execute the tests remaining in test
suites. More specifically, we run each test method in isolation
rather than the whole test suite to avoid Out of Memory
Exceptions during log trace collection.
We skip commits that include errors in production code(e.g., due to dependency problems, compiler errors). On theremaining commits, we executed 615,196 test methods. Then,we excluded test methods failing in ci−1 : If a refactoringhas been performed in commit ci and the related tests in therevision ci−1 were failing even before the refactoring, thosemust be excluded, since we want to identify test breaks causedby the refactoring operation, and not already present in thesystem. This filter removed 3,411 test methods.
Impact Analysis. After running tests, the execution tracefor each revision is generated, which provides us with the lines

the tests exercise in the production code. In addition, RefactoringMiner provides us with the location of the production code
where refactoring is performed. Matching the two outputs, we
identify whether refactoring edits impacted any of the paths
exercised by any of the test methods. Figure 2 outlines how
we match refactoring edits to test case execution paths.
Production Code
Test Code1 class Calc{
1 class TestCalc{
2 int i = 10;
2 @Test3 int j = 3;
3 public void testAdd(){
4 int add(){
Calc c = new Calc();
4
assertEquals( 13, c.add());
5
return i + j;
5
6 }
6 }
7 int divide(){7 @Testif (j == 0)88 public void testDivide(){Calc c = new Calc();
throw new ArithmeticException(); 99assertEquals( 3, c.divide());
return i / j;
10
10
}
}
11
11
[Chage Return Type (CR)]
12 }
12 }

CR

int -> double

Fig. 2. Specifying refactoring impact: A red shade represents a refactoring.
The refactored line is used only by testDivide(). After refactoring, the assertionin testDivide should be modified from the int value of 3 to the double valueof 10/3. The arrows represent the order of statements invoked by each test
method.

In Figure 2, a C HANGE R ETURN T YPE refactoring is performed in Calc.divide(). Since the refactored line is exercisedby testDivide(), the method signature (i.e., package, class,method name, parameter types, and line number) is recordedin the trace log. Thus, testDivide(), at line 10, is determined
to be affected by the refactoring, whereas testAdd() is not
because it does not invoke divide(). If attributes are impacted
by a refactoring, we only identify as impacted test methods
that use those attributes in the execution traces to reduce false
positives.
We apply this approach to the two snapshots before andafter changes because several types of refactoring (e.g., A DD
M ETHOD A NNOTATION) are recorded only in one of the twosnapshots by RefactoringMiner as they do not modify anylines in one snapshot. In contrast, the changes caused by otherrefactoring types (e.g., I NLINE VARIABLE) are recorded by
RefactoringMiner for both snapshots.

We identify them as the same refactoring by using their hashvalues given by RefactoringMiner.2 Thanks to the performed
analysis, we know for each commit the lines in the test
methods affected by the detected refactoring operations.
Once this information is known, we also excluded from ourstudy test methods affected by multiple refactoring operations,because we cannot identify the exact refactoring operationthat possibly resulted in a broken test. We keep the cases
in which a single refactoring impacted multiple lines in the
production code, since we can still identify, in these cases, thespecific refactoring operation causing the test to break. Thisfilter removed 6,314 test methods.
Our data collection process requires considerable computational resources (about 3 hours per commit). For this reason,
we employed Kubernetes to parallelize the computation. We
deployed Kubernetes on 30 nodes of m5.2xlarge instances (8vcpu and 32GB memory) of Amazon Web Services and 10
nodes on our local servers, running our pipeline for about one
month. The program used for data collection is available on
GitHub (https://github.com/posl/DoesRefactoringBreakTest).
C. Data Analysis
To answer our research questions, we replace the test
directory in ci with the one in ci−1 . Finally, we identify
tests broken (i.e., tests that fail or that exhibit compilation
errors) by the refactoring operation using the previously extracted information. Then, two of the authors manually andindependently inspected each test method that resulted in test
breaks (i.e., compilation error or test failure) to make surethey were actually caused by the refactoring. The two authorsclassified in the same way 83.6% of the inspected test breaks,
with a Cohen’s kappa coefficient of 0.66, which demonstrates a
substantial agreement [39]. To solve the disagreements, a third
author inspected the conflicting cases, discussing them withthe first two authors and deciding on the final outcome (i.e.,
the test was or was not broken by the refactoring operation).
Differently from previous studies [8] [21], we also classify
the test breaks into three subcategories:
Compiler-Error: The production code after the refactoring
can be built, but the test code (related test methods)
cannot compile (e.g., new arguments have been added
to the tested method).
Runtime-Error: The test method suddenly terminated during
its running (e.g., NullPointerException).
Failure: The test method received unexpected values from theproduction code after refactoring.
For each category of test break, we report the number oftest breaks that are involved in each type of refactoring. In
addition, the rates (i.e., compiler-error rate, runtime-error rate,
and failure rate) are calculated to identify the refactoring types
that are more likely to involve test breaks (if any).
2 RefactoringMiner’s Java API provides a hash value for each refactoring
instance to identify each of them, based on the refactoring type name, locations
before and after refactoring, etc. )For example, the failure rate for test methods involving
refactoring α is calculated as follows:
failures involving refactoring α
F ailureRateα =all test methods affected by refactoring α
While the above data analysis is sufficient to answer RQ1 ,to address RQ2 we compute the median of changed lines inbroken test methods. For each refactoring type, we calculate
the following metric.
Changed lines: We extract file diffs with the “git diff”
and sum the numbers of added and deleted lines in eachtest method impacted by refactoring and other relevant
methods in test code (e.g., util methods). We used thehistogram algorithm to generate diffs because Nugroho
et al. [40] showed that the algorithm can precisely assess
the impacted lines.
IV. R ESULTS
Below we detail our following findings with examples:• No test suites involving refactoring of local variables
failed.
• Refactoring operations changing method signatures andreturn types are the most likely to trigger CompilerErrors.
• No Runtime-Error caused by refactoring operations could
be identified.
• It is rather rare for refactoring operations to directly
introduce Failures.
• The number of lines needed to fix test breaks triggeredby refactoring varies for different refactoring types, with
CHANGE RETURN TYPE being the most likely to require
demanding fixes.
A. RQ1 : What types of refactoring break test suites?
Table II shows the number of test methods involved in ourstudy for each type of refactoring and the frequency withwhich each refactoring operation type results in passed or
broken tests. The refactoring types are grouped into three
categories – “Local Variable”, “Method”, and “Class” – basedon the entity on which a refactoring operation is performed.
Since, as any automated procedure, our data extraction pipeline
can result in false positives, all data in our study has been
manually validated. In particular, the manual inspection aimed
at removing cases in which the breaking test change was
not the refactoring operation. The manual validation was
performed by three authors: Each instance was independently
inspected by two authors and a third one was in charge ofsolving conflicts (16.4% of cases). The manual validation
resulted in the exclusion of seven false positives, one related
to Compiler-Errors and six to Failures. These false positives
are already excluded from Table II. In addition to that, ten test
methods encountered Runtime-Errors for which we could not
identify the root cause of the error. We also removed themfrom our analysis, leading to no remaining Compiler-Error.
We also exclude the errors that are caused by other factors(e.g., errors due to runtime-environments, invalid dependencies, flakiness [41]).

TABLE II
RQ1: T HE NUMBER OF TEST METHODS THAT ARE INVOLVED IN REFACTORING AND RESULT IN EACH TYPE OF TEST BREAKS .
T HE REFACTORING OPERATIONS THAT APPEAR LESS THAN 10 TIMES ARE NOT SHOWN DUE TO SPACE LIMITATIONS . T HE LAST ROW INCLUDES THOSEINSTANCES OMITTED IN THE TABLE , THUS DOES NOT EQUAL THE SUM OF PREVIOUS ROWS .
Category
Local
Variable

MethodClass

Refactoring TypeE XTRACT VARIABLE
R EPLACE VARIABLE WITH ATTRIBUTIONR ENAME VARIABLE
C HANGE VARIABLE T YPEC HANGE R ETURN T YPEE XTRACT M ETHODE XTRACT AND M OVE M ETHODC HANGE PARAMETER T YPER ENAME M ETHODA DD PARAMETERR ENAME PARAMETER
A DD M ETHOD A NNOTATIONM OVE C LASSA DD C LASS A NNOTATIONR ENAME ATTRIBUTE
E XTRACT ATTRIBUTE
C HANGE ATTRIBUTE T YPER ENAME C LASS
All

#methods
1,068
608
458
145
498
318
312
301
276
195
114
75
567
171
125
99
57
32
5,478

Finding 1. No test suites involving refactoring of local
variables failed. From Table II we can see that while over
2,000 test methods involving refactoring were performed on
local variables, none of them led to any error or failure. Thisfinding is partially in contrast with what has been reported inprevious work [8], in which refactoring operations involving
local variables were reported as likely to introduce bugs.
Clearly, a bug could still be introduced without breaking
changes in the test suite (e.g., in cases in which the test suite,
despite exercising the lines of code involving the variable, isparticularly weak in terms of asserted behavior). However, apossible explanation for such a different finding is also thelack of manual inspection and the use of changes between
releases in the previous work, which may classify a refactoring
as responsible for introducing a bug despite that other changes
tangled to the refactoring might be the bug-triggering event.
Finding 2. Refactoring operations changing method
signatures and return types are the most likely to trigger Compiler-Errors. Seven types of refactoring operations,
including C HANGE R ETURN T YPE, C HANGE PARAMETERT YPE, R ENAME M ETHOD, A DD PARAMETER, M OVE C LASS,R ENAME ATTRIBUTE, and R ENAME C LASS, resulted inCompiler-Errors. Most of them, except C HANGE R ETURNT YPE and R ENAME ATTRIBUTE, change method signatures(i.e., package name, file name, class name, method name,
and/or parameter types) [42]. Changing the method signature
is indeed an obvious breaking change when it comes tolocations of the code (such as tests) invoking the refactored
method. Moreover, changes of return types can also lead totype inconsistency, while not impacting method signatures. An
example of test broken by C HANGE R ETURN T YPE can beseen in Snippet 1: A developer changed the return type of themethod “removeSdchEncoding” from “HttpHeaders”
to “void”, while no other changes were made.

#PASS
1,068
608
458
145
490
318
312
300
251
180
114
75
526
171
92
99
57
22
5,343

#COMPILER ERROR
0 (0.0%)0 (0.0%)0 (0.0%)0 (0.0%)7 (1.4%)0 (0.0%)0 (0.0%)1 (0.3%)25 (9.1%)
13 (6.7%)
0 (0.0%)0 (0.0%)41 (7.2%)
0 (0.0%)33 (26.4%)
0 (0.0%)0 (0.0%)10 (31.2%)
132 (2.4%)#FAILURE
0 (0.0%)0 (0.0%)0 (0.0%)0 (0.0%)1 (0.2%)0 (0.0%)0 (0.0%)0 (0.0%)0 (0.0%)2 (1.0%)0 (0.0%)0 (0.0%)0 (0.0%)0 (0.0%)0 (0.0%)0 (0.0%)0 (0.0%)0 (0.0%)3 (0.1%)As the original test method still expected to receive an instance of “HttpHeaders”, a CompilerError occurred. Therefore, the developer had to perform“removeSdchEncoding” outside of the assertion statement
and use the “headers” in the assertion instead (Snippet 2).
Snippet 1. “Change Return Type” refactoring causing a compiler error
1 - public static HttpHeaders removeSdchEncoding(
2 HttpHeaders headers) {3 + public static void removeSdchEncoding(
4 + HttpHeaders headers) {Snippet 2. Fix for compile errors due to “Change Return Type” in test code
1 - ...
2 - assertEquals(expectedEncodings,
3 ProxyUtils.removeSdchEncoding(headers)
4 .get(HttpHeaders.Names.ACCEPT_ENCODING));
5 + ...
6 + ProxyUtils.removeSdchEncoding(headers);
7 + assertEquals(expectedEncodings,
8 +headers.getAll(HttpHeaders.Names.ACCEPT_ENCODING));
9 ...

Finding 3. No Runtime-Error caused by refactoring
operations could be identified. Throughout our study, only
ten Runtime-Errors occurred and we could not identify thecause of the error. Other than that, we did not find any
Runtime-Errors related to refactoring. The reason might be that
Runtime-Errors were resolved by developers when modifying
production code. After developers modified the production
code, they might ran the test suites. If any Runtime-Error was
spotted during the exercise of test methods, they might tend
to further modify the production code until no RuntimeError occurred before committing the changes. However, suchcases could only be identified through a more fine-grained
analysis capturing code editing in the IDE. The commit-level
granularity of our study does not allow to unveil them.

Snippet 3. Example of changes causing failures
1 2 +

30
25
#Changed Lines

Finding 4. It is rather rare for refactoring operations to
directly introduce Failures. Failures were observed in only
three test methods, with all of the corresponding to refactoring
operations at the method level (i.e., C HANGE R ETURN T YPEand A DD PARAMETER). An example of test failure caused by
C HANGE R ETURN T YPE can be found in Snippet 3.

public List<Element> previousElementSiblings() {public Elements previousElementSiblings() {A developer changed the return type of themethod“previousElementSiblings()”
from“List<Element>” to “Elements”. As “Elements”
extends “List<Element>” and it was already implemented
before the refactoring, no Compiler-Error was raised.
However, for an instance of the type “Elements”, themethod “previousElementSiblings()” will produce
a list where siblings are ordered by its distance from thecurrent node, while “List<Element>” will generate alist of siblings keeping their original order as child nodesof their common parent. Therefore, the different behaviorof the method “previousElementSiblings()” after
refactoring resulted in the Failure. Interestingly, developers
only fixed this issue in a later commit after the refactoring
got merged into the codebase.
RQ1 : While a previous study [8] showed that many
refactoring-related commits break tests, our manual inspection indicates that refactoring operations only account for
2.5% of the real cause of test breaks. Refactoring operations changing method signatures are more likely tocause Compiler-Errors in test code, and most of the time
refactoring does not directly trigger Failures.
B. RQ2 : What is the magnitude of the fix triggered by different
types of refactoring on test methods?
Figure 3 depicts, for each refactoring type for which weobserved broken tests, the distribution of the numbers ofchanged lines in each test method as a consequence of
refactoring operations in relevant production code. In these
violin plots, the thickness of the outer layer represents theprobability density of the plotted values. In the center of eachviolin plot, the white dot represents the median, while the thickblack bar represents the interquartile range.

15
10
5
0

Snippet 4. Example of test methods detecting failures
1 @Test2 public void testPreviousElementSiblings() {3Document doc = Jsoup.parse(
4"<ul id=’ul’>" +5"<li id=’a’>a</li>" +6"<li id=’b’>b</li>" +7...
8 - List<Element> elementSiblings =9 element.previousElementSiblings();
10 + Elements elementSiblings =11 +element.previousElementSiblings();
12
...
13
assertEquals("a", elementSiblings2.get(0).id());
14
assertEquals("b", elementSiblings2.get(1).id());

20

ChangeReturn Type

Add
Parameter

RenameMethodRefactoring Type

Move
Class

RenameClass

Fig. 3. Distribution of changed lines in test suites to fix test breaks introduced
by refactoring operations. We do not show C HANGE PARAMETER T YPE here
as it only occurred once in our study.

Finding 5. The number of lines needed to fix test
breaks triggered by refactoring varies for different refactoring types, with CHANGE RETURN TYPE being the most
likely to require demanding fixes. As it can be seen from
Figure 3, most of the issues caused by refactoring can be
fixed within 15 changed lines in a test method. The median
number of changed lines needed for C HANGE R ETURN T YPE
is 7, which is the highest among all types of refactoring.
C HANGE R ETURN T YPE often requires developers to modify
the lines around the refactored methods, which is illustrated
in Snippet 6. In this case, similarly to the one previously
discussed, developers changed the return type of the method“nextElementSiblings” from “List<Element>” to“Elements” (Snippet 5), thus a different approach is needed
to verify if the element has any next sibling.
Snippet 5. Example of Change Return Type refactoring
1 2 +public List<Element> previousElementSiblings() {public Elements previousElementSiblings() {Snippet 6. Example of fixes required by Change Return Type, modifying theapproach to assert values
1 List<Element> elementSiblings4 = div.
nextElementSiblings();
2 - try {3 Element elementSibling = elementSiblings4.get(0);4 fail("This element should has no next siblings");
5 - } catch (IndexOutOfBoundsException e) {6 - }7 + assertEquals(0, elementSiblings4.size());

Most test breaks caused by A DD PARAMETER can be fixedby modifying around five lines of code in test methods. Whenmanually inspecting how these broken tests were resolved, we
found that 73.3% (11/15) of them were fixed by adding nullvalues to the parameter list, while for the remaining 26.7% ofcases, developers had to manually select the correct argument
to pass. The majority of the test breaks caused by R ENAME
M ETHOD requires less than five changed lines for the fix.

As test methods usually invoke the renamed methods several
times, during our manual inspection, we noticed in almost
all cases (∼96%) the lines changes in the test methods justrequired to rename the method to invoke.
M OVE C LASS and R ENAME C LASS are least likely torequire changes in test code: the medians of changed lines are
0 for these two types of refactoring. However, in some rarecases, significant changes are needed for M OVE C LASS. To
better understand how the impact of different refactoring types
differs from each other, we applied the Mann-Whitney U-test
with a Bonferroni correction to all the pairs of distributions ofchanged lines. We also measured effect size r from the z-scoreof the U-test. As a result, statistically significant differences (pvalue < 0.01/10) can be found between the pairs of C HANGE
R ETURN T YPE and R ENAME M ETHOD with a large effect size
(r=0.63), as well as between R ENAME M ETHOD and M OVEC LASS with a medium effect size (r=0.43). The result is inline with our findings from Figure 3.

We reproduced the refactoring in the production code by automating the process using the two IDEs. Finally, we evaluatedwhether test suites were correctly modified in consequence ofthe refactoring by comparing with the obtained tests with thosethat have been fixed by developers.
Results. Table III shows what percentage of test breaks
prevented by IDEs.

RQ2 : A DD PARAMETER and C HANGE R ETURN T YPE seldom break tests but can have a substantial impact on the test
code, requiring 4-12 lines of changes to fix the test suite.

The two selected IDEs can support all the refactoring operations that resulted in breaks in RQ1 . Both IDEs managed to
avoid all the test breaks triggered by refactoring that involves
the renaming or moving of code entities. Instead, all the test
breaks caused by A DD PARAMETER, C HANGE R ETURN T YPEand C HANGE PARAMETER T YPE refactoring are still observed
even by automating the refactoring with the IDEs.

V. D ISCUSSION
In the following we detail, complemented by examples, thefollowing findings:• IDEs do not help to avoid test breaks when the methods inproduction code are subject to ADD PARAMETER refactoring, since an additional input is required by the refactored
method, triggering changes in the test code that must be
manually implemented.
• When CHANGE RETURN TYPE or CHANGE PARAMETER TYPE
is performed, the IDEs are unable to examine whether
the impacted instance still has valid callees or has acompatible data type as a parameter, thus not helping
to avoid test breaks.
A. Discussion: Can IDEs help in reducing the chances ofbreaking tests during refactoring activities?
Modern IDEs provide great support for refactoring sourcecode. While they are not meant to recommend refactoring
operations, IDEs such as IntelliJ and Eclipse can help for
example in consistently updating the code base when a rename
method/class/variable refactoring is performed, thus lowering the chances of introducing bugs and/or breaking tests.
Given the results of our study, showing that some of these
refactorings can indeed break tests, we investigate whether
those breaks could have been avoided by performing these
refactorings through the IDE.
Approach. We selected two popular IDEs, InteliJ and
Eclipse3 , which account for 89% of the market share as of2020 [43]. Next, we selected 135 test methods in which test
breaks happened in RQ1 . Then, we cloned the repositories
in which these refactorings have been performed and checkedout the revision right before the refactoring implementation.
3 We used InteliJ 2021.1 and Eclipse 2021-03.

TABLE IIIP ERCENTAGE OF THE TEST BREAKS PREVENTED BY IDE S
Refactoring TypeM OVE C LASSR ENAME ATTRIBUTER ENAME M ETHODA DD PARAMETERR ENAME C LASSC HANGE R ETURN T YPEM OVE A ND R ENAME C LASSC HANGE PARAMETER T YPECOMPILER ERROR
100% (41/41)
100% (33/33)
100% (25/25)
0% ( 0/13)
100% (10/10)
0% ( 0/ 7)
100% ( 2/ 2)
0% ( 0/ 1)

FAILURE
0% ( 0/ 2)
0% ( 0/ 1)-

Finding 6. IDEs do not help to avoid test breaks
when the methods in production code are subject to
ADD PARAMETER refactoring, since an additional input isrequired by the refactored method, triggering changes
in the test code that must be manually implemented.
When an A DD PARAMETER is performed on a method, thetwo IDEs ask the user to provide as input the type andname of the new parameter. The user can assign a default
value for the new parameter so that IDEs insert them in the
invocations to the refactored method. However, all the test
breaks (i.e., 13 Compiler-Errors and 2 Failures) triggered by
A DD PARAMETER were still observed in our study. One major
reason for such a finding is that different values for the added
parameter are often required by different locations of code
invoking the refactored method. Even the same test method
invoking multiple times a method in the production code may
require different parameters’ values for the multiple invocations. However, both IDEs only allow to input one predefinedvalue. Snippets 7 and 8 show the difference between changes
required in the production and test code as a consequence ofan A DD PARAMETER.
Snippet 7. Changes triggered by Add Parameter in the production code1

public FileVisitResult visitFile(final Path file, final
BasicFileAttributes attributes) throws IOException {2 - if (Files.exists(file) &&3 pathFilter.accept(file)
4 == FileVisitResult.CONTINUE) {5 + if (Files.exists(file) &&6 +pathFilter.accept(file, attributes)
7 +== FileVisitResult.CONTINUE) {Snippet 8. Example of changes triggered by Add Parameter in the test code
1 @Test2 public void testDeprecatedWildcard() throws Exception {3 ...
4 - assertEquals(FileVisitResult.CONTINUE,
5 listFilter.accept(txtPath));
6 + assertEquals(FileVisitResult.CONTINUE,
7 +listFilter.accept(txtPath, null));In this example, we applied A DD PARAMETER refactoringon the method accept and set the default value of the
newly added parameter to the variable “attributes”. Consequently, the IDE appended the value “attributes” to theparameter list of all the occurrences of accept. However, as
this variable was never declared in the test case (Snippet 8), an
error occurred. Therefore, we had to manually inspect this caseand choose an appropriate value. If we wanted to automate
this process, the IDE should consider the context of eachrefactoring. Modern IDEs are still incapable of performing
such a task and more efforts can be devoted into this direction.
Finding 7. When CHANGE RETURN TYPE or CHANGE
PARAMETER TYPE is performed, the IDEs are unable toexamine whether the impacted instance still has valid
callees or has a compatible data type as a parameter, thus
not helping to avoid test breaks. While the two IDEs support
the C HANGE R ETURN T YPE refactoring, seven CompilerErrors and one Failure have been observed when we reproduced the refactorings causing breaking tests in RQ1 .
Although the IDEs modified some of the lines that include thecaller or callee of the refactored method, some lines had to
be manually modified to solve Compiler-Errors and Failures.
Snippet 9 shows the test code immediately after a C HANGE
R ETURN T YPE is performed with the IDE.
Snippet 9. Test method after a C HANGE R ETURN T YPE applied with IDE
1 - final PathCounts pathCounts =2 - PathUtils.deleteFile(tempDirectory.resolve(fileName));3 + final Counters.PathCounters pathCounts =4 + PathUtils.deleteFile(tempDirectory.resolve(fileName));5
Assertions.assertEquals(0,6pathCounts.getDirectoryCount());
7
Assertions.assertEquals(1, pathCounts.getFileCount());
8
Assertions.assertEquals(0, pathCounts.getByteCount());

Snippet 10. Example of refactoring C HANGE PARAMETER T YPE
1 - public String asString(Document doc, Properties
properties) {2 + public static String asString(Document doc, Map<String, String> properties) {3...
4transformer.setOutputProperties(properties);

The IDE modifies the lines in the test code (i.e., line 1 and
2). However, the test code has compilation errors in line 58 because the class after refactoring (i.e., PathCounters)
does not implement the three methods that PathCounts has.
As for the other cases, the IDEs do not help to avoid test
breaks involving C HANGE R ETURN T YPE from specific types
to void, and any return type using generics.
While the IDEs might avoid test breaks involving several
class types that can be applied by autoboxing (e.g., int toInteger), we found none of these cases in our study.

Similar scenarios also apply to CHANGE PARAMETER TYPE
refactoring, which is illustrated in Snippet 10. In thisexample, we applied C HANGE PARAMETER T YPE refactoring on the method asString to modify the typeof parameter “properties” from “Properties” to“Map<String, String>”, which is incompatible with themethod “setOutputProperties”. As the IDEs are notable to detect or fix this issue, a Compiler-Error occurred.
Our analysis indicates that IDEs can help to avoid test
breaks caused by specific refactoring operations. For our future
work, we plan to assess whether these test breaks can be solved
through automatic test-suite repair techniques [44][45].
Additional Analysis: IDEs are helpful to avoid test breaks
triggered by refactoring involving rename or move, but they
do not prevent test breaks triggered by C HANGE R ETURNT YPE, C HANGE PARAMETER T YPE, or A DD PARAMETER.
B. Summary and Implications
This study examined the impact of refactoring on test suites,
measuring the broken tests and fixed lines in consequenceof refactoring. Additionally, this study investigated whether
modern IDEs can prevent test breaks caused by different
refactoring types. In this section, we map the findings across
RQ1 , RQ2 , and the additional analysis performed with IDEs,
and then discuss which refactoring requires more effort to fix
test suites when developers refactor production code.
RQ1 and RQ2 showed that R ENAME M ETHOD, M OVE
M ETHOD and R ENAME C LASS are likely to induce CompilerErrors (Finding 2). Also, R ENAME M ETHOD is the third interms of magnitude of the requried fix in the test code (Finding 5). However, our additional analysis showed that refactoring tools in the IDE can prevent breaking test code bymodifying the test code according to the change in the production code. Still, many developers conduct such refactorings
manually [1][3][46], being more prone to break tests.
Take away for practitioners: When possible, the automated refactoring feature in the IDE must be exploited
to minimize the chance of introducing breaking changes(and possibly bugs). This is especially true when performing refactorings such as R ENAME M ETHOD, M OVE
M ETHOD or R ENAME C LASS, for which modern IDEs
provide an excellent support.
As for A DD PARAMETER, C HANGE PARAMETER T YPE,
and C HANGE R ETURN T YPE, these refactoring types change
the “communication interface” of the impacted methods (i.e.,their input/output), which often induces Compiler-Errors but
rarely Failures (Findings 2 and 4). In addition, IDEs are oflittle help in avoiding breaking changes when it comes to theautomation of these refactoring types (Findings 6 and 7).
Thus, developers must manually select the correct argument
or properly handle the new return type and fix the code location invoking the refactored method since compilers tell only
the locations of the errors. Again, previous studies show thatthese refactorings are mostly conducted manually [1][3][46].

In addition, these refactoring types require relatively larger
fixes than the other types (Finding 5) and these fixes are
not trivial to perform since, for example, the developer has
to select the appropriate values for the additional argumentin the different locations in which the method is invoked.
Thus, these types of refactoring requires a high effort fordevelopers to fix the broken tests. Researchers could explore
approaches to avoid test breaks when these types of refactoring are performed. For example, it could be helpful fordevelopers to recommend the most appropriate input values in
the invocations of the refactored method. This could be done,by comparing trace logs generated by various inputs for the
method refactored with A DD PARAMETERS with the original
trace logs (i.e., before refactoring).
Take away for researchers: Researchers and IDE developers should improve the support provided when implementing A DD PARAMETER, C HANGE PARAMETERT YPE, and C HANGE R ETURN T YPE refactoring, especially when these refactoring impact multiple locationsin the test/production code.
C. Threats to Validity
Internal Validity. Internal validity threats concern factors
internal to our study that could influence our results. To
investigate the inherent impact of refactoring, we decided to
exclude from our study test methods that were affected bymultiple refactoring edits. For example, in one commit wefound a single test method that exercised the production code
locations impacted by 19 refactoring edits. Our choice wasdriven by the will of isolating to the extent possible the impactof the different refactoring types. However, we acknowledge
that our analysis may miss interesting test breaking cases
resulting from combinations of refactoring.
Construct Threats. Construct validity threats concern therelationship between theory and observation, and are mainly
related to sources of imprecision in our analyses. We employeddynamic analysis in an effort to establish precise links between
production and test code.
Compared with FaultTracer [47] used in the previous
work [8], our approach can detect line-level refactoring.
However, it might produce more false positives. Thus, we
manually validated the data used in our study to increase theconfidence in our findings. As for manual inspection, to avoid
subjectiveness bias, all the manual analyses we performed
were run by two authors, and a third author was involvedin case of conflicts. Still, we cannot exclude imprecisions insuch a process.

Still, there are many factors negatively impacting the generalizability of our findings. First, we only focused on Javasystems using Maven. Second, we only employed projects
that do not include in their repository any sub-project to avoid
dependency issues. Third, the number of projects (8) is clearly
too low to claim any generalizability of our findings. However,
we want to highlight the costly analysis performed to collectthe data used in our experiments.
VI. C ONCLUSIONS AND F UTURE W ORKThis study measured the impact of refactoring on test
suites. We conducted a large-scale and fine-grained study using
dynamic analysis and manual inspection of the collected data.
We employed RefactoringMiner, the state-of-the-art refactoring detection tool to identify refactoring edits performed ineight open source projects. We checked whether test methods
exercise location in the production code subject to refactoringrelated edits. This was done by collecting and parsing execution log traces. Although collecting this data on every commit
requires a formidable amount of time, we overcame this issue
by expoloiting Kubernetes and numerous docker instances.
Throughout our empirical study we ran more than615,196 test methods, and we observed that most types ofrefactoring operations do not break test suites and, even in
the rare occasions when they do, small fixes are required to
the related tests. A few refactoring operations are more likely
to break test suites. For example, A DD PARAMETER resultsin compilation errors for 6.7% of test methods and failuresfor 1.0% of them. Also, several lines of code must usually be
fixed in test methods as a result of this refactoring. Finally,
we observed that for refactoring renaming and moving code
entities, exploiting the automated support of IDEs can help inavoiding test breaks. Instead, this is not the case for A DD
PARAMETER, C HANGE PARAMETER T YPE, and C HANGE
R ETURN T YPE refactoring.
Overall, based on the evidence we found through our study,
the question of whether refactoring breaks tests is to be answered with a sound “no, it does not, and it should not”. It does
not, as by definition refactoring is a behavior-preserving codetransformation process, thus contradicting previous findings
by Rachatasumrit and Kim [8], who however did not heed theissue of tangled commits. We phrased this finding as a takeaway for practitioners. However, for refactoring operations thatmodify the method signatures and do induce test breaks wealso found a gap in terms of IDE support, which we phrased
as a take-away for researchers.
Our future work fill focus on (i) the research challenges
we listed in Section V, and (ii) corroborating our findings byrunning our study on a larger set of systems.
ACKNOWLEDGMENT

External Threats. External validity threats concern thegeneralizability of our findings. We selected eight projects
for our quantitative study, paying attention to select projects
having a different size and belonging to different application
domains. The number of projects is comparable with that ofsimilar previous studies [8][21].

We thank Dr. Ishio and his colleagues for developing
SELogger and giving us helpful advice. We also gratefully
acknowledge the financial support of JSPS and SNSF forthe project “SENSOR” (No. 183587, JPJSJRP20191502), and
JSPS for the KAKENHI grants (JP21H04877, JP21K17725).

R EFERENCES[1] D. Silva, N. Tsantalis, and M. T. Valente, “Why we refactor? confessionsof github contributors,” in Proceedings of the 24th ACM International
Symposium on Foundations of Software Engineering (FSE), 2016, pp.
858–870.
[2] J. Pantiuchina, F. Zampetti, S. Scalabrino, V. Piantadosi, R. Oliveto,
G. Bavota, and M. D. Penta, “Why developers refactor source code: A
mining-based study,” ACM Transactions on Software Engineering andMethodology (TOSEM), vol. 29, no. 4, pp. 29:1–29:30, 2020.
[3] M. Kim, T. Zimmermann, and N. Nagappan, “A field study of refactoring
challenges and benefits,” in Proceedings of the 20th ACM Symposiumon the Foundations of Software Engineering (FSE), 2012, p. 50.
[4] A. C. Bibiano, V. Soares, D. Coutinho, E. Fernandes, J. L. Correia,
K. Santos, A. Oliveira, A. Garcia, R. Gheyi, B. Fonseca, M. Ribeiro,
C. Barbosa, and D. Oliveira, “How does incomplete composite refactoring affect internal quality attributes?” in Proceedings of the 28th
International Conference on Program Comprehension (ICPC), 2020, pp.
149–159.
[5] A. Vahabzadeh, A. M. Fard, and A. Mesbah, “An empirical study of bugs
in test code,” in Proceedings of the 2015 IEEE International Conference
on Software Maintenance and Evolution (ICSME), 2015, pp. 101–110.
[6] C. Vassallo, F. Palomba, and H. C. Gall, “Continuous refactoring in
CI: A preliminary study on the perceived advantages and barriers,” inProceedings of the 2018 IEEE International Conference on Software
Maintenance and Evolution (ICSME), 2018, pp. 564–568.
[7] B. Adams and S. McIntosh, “Modern release engineering in a nutshell- why researchers should care,” in Proceedings of the 2016 IEEE
23rd International Conference on Software Analysis, Evolution, andReengineering (SANER), 2016, pp. 78–90.
[8] N. Rachatasumrit and M. Kim, “An empirical investigation into theimpact of refactoring on regression testing,” in Proceedings of the 28th
IEEE International Conference on Software Maintenance (ICSM), 2012,
pp. 357–366.
[9] C. Vassallo, F. Zampetti, D. Romano, M. Beller, A. Panichella, M. D.
Penta, and A. Zaidman, “Continuous delivery practices in a large
financial organization,” in Proceedings of IEEE International Conference
on Software Maintenance and Evolution (ICSME), 2016, pp. 519–528.
[10] K. O. Elish and M. Alshayeb, “Investigating the effect of refactoring on
software testing effort,” in Proceedings of the 16th Asia-Pacific Software
Engineering Conference (APSEC), 2009, pp. 29–34.
[11] N. Tsantalis, V. Guana, E. Stroulia, and A. Hindle, “A multidimensional
empirical study on refactoring activity,” in Proceedings of the 2013
Conference of the Center for Advanced Studies on Collaborative, 2013,
pp. 132–146.
[12] F. Shull, J. C. Carver, S. Vegas, and N. J. Juzgado, “The role ofreplications in empirical software engineering,” Empirical Software
Engineering (EMSE), vol. 13, no. 2, pp. 211–218, 2008.
[13] O. S. Gómez, N. J. Juzgado, and S. Vegas, “Understanding replicationof experiments in software engineering: A classification,” Informationand Software Technology (IST), vol. 56, no. 8, pp. 1033–1048, 2014.
[14] B. Cornelissen, L. Moonen, and A. Zaidman, “An assessment methodology for trace reduction techniques,” in Proceedings of the 24th IEEE
International Conference on Software Maintenance (ICSM), 2008, pp.
107–116.
[15] Kubernetes. https://kubernetes.io/.
[16] X. Ren, B. G. Ryder, M. Störzer, and F. Tip, “Chianti: a change impact
analysis tool for java programs,” in Proceedings of the 27th International
Conference on Software Engineering (ICSE), 2005, pp. 664–665.
[17] M. J. Harrold, J. A. Jones, T. Li, D. Liang, A. Orso, M. Pennings,
S. Sinha, S. A. Spoon, and A. Gujarathi, “Regression test selection forjava software,” in Proceedings of the 16th ACM conference on Objectoriented programming, systems, languages, and applications (OOPSLA),
2001, pp. 312–326.
[18] H. Do, S. Mirarab, L. Tahvildari, and G. Rothermel, “An empirical
study of the effect of time constraints on the cost-benefits of regression
testing,” in Proceedings of the 16th ACM International Symposium on
Foundations of Software Engineering (FSE), 2008, pp. 71–82.
[19] A. K. Onoma, W. Tsai, M. H. Poonawala, and H. Suganuma, “Regression testing in an industrial environment,” Communications of the ACM,
vol. 41, no. 5, pp. 81–86, 1998.
[20] A. Srivastava and J. Thiagarajan, “Effectively prioritizing tests in development environment,” in Proceedings of the International Symposium
on Software Testing and Analysis (ISSTA), 2002, pp. 97–106.

[21] X. Tang, S. Wang, and K. Mao, “Will this bug-fixing change break
regression testing?” in Proceedings of the 2015 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement(ESEM), 2015, pp. 215–224.
[22] M. Bruntink and A. van Deursen, “An empirical study into classtestability,” The Journal of Systems and Software (JSS), vol. 79, no. 9,
pp. 1219–1232, 2006.
[23] K. Stroggylos and D. Spinellis, “Refactoring–does it improve software
quality?” in Proceedings of the 5th International Workshop on Software
Quality (WoSQ), 2007, p. 10.
[24] A. Chávez, I. Ferreira, E. Fernandes, D. Cedrim, and A. Garcia,“How does refactoring affect internal quality attributes?: A multi-project
study,” in Proceedings of the 31st Brazilian Symposium on Software
Engineering, 2017, pp. 74–83.
[25] D. Cedrim, A. Garcia, M. Mongiovi, R. Gheyi, L. Sousa, R. de Mello,
B. Fonseca, M. Ribeiro, and A. Chávez, “Understanding the impact of
refactoring on smells: A longitudinal study of 23 software projects,” inProceedings of the 2017 11th Joint Meeting on Foundations of Software
Engineering (ESEC/FSE), 2017, pp. 465–475.
[26] G. Szoke, G. Antal, C. Nagy, R. Ferenc, and T. Gyimóthy, “Bulk fixing
coding issues and its effects on software quality: Is it worth refactoring?”
in Proceedings of the 2014 IEEE 14th International Working Conference
on Source Code Analysis and Manipulation (SCAM), 2014, pp. 95–104.
[27] R. Moser, P. Abrahamsson, W. Pedrycz, A. Sillitti, and G. Succi, “A casestudy on the impact of refactoring on quality and productivity in an agile
team,” in Proceedings of the 2008 Balancing Agility and Formalism inSoftware Engineering, 2008, pp. 252–266.
[28] G. Bavota, B. D. Carluccio, A. D. Lucia, M. D. Penta, R. Oliveto, andO. Strollo, “When does a refactoring induce bugs? an empirical study,”
in Proceedings of the 12th IEEE International Working Conference on
Source Code Analysis and Manipulation (SCAM), 2012, pp. 104–113.
[29] R. Morales, A. Sabane, P. Musavi, F. Khomh, F. Chicano, and G. Antoniol, “Finding the best compromise between design quality and testing
effort during refactoring,” in Proceedings of the IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering(SANER), 2016, pp. 24–35.
[30] A. Sabane, M. D. Penta, G. Antoniol, and Y. Guéhéneuc, “A study on
the relation between antipatterns and the cost of class unit testing,” inProceedings of the 17th European Conference on Software Maintenanceand Reengineering (CSMR), 2013, pp. 167–176.
[31] N. Tsantalis, A. Ketkar, and D. Dig, “Refactoringminer 2.0,” IEEE
Transactions on Software Engineering (TSE), 2020.
[32] JUnit. https://junit.org/.
[33] TestNG. https://testng.org/.
[34] J. Farley and W. Crawford, Java Enterprise in a Nutshell. O’ Reilly& Associates, Inc., 2005.
[35] Maven. https://maven.apache.org/.
[36] G. Soares, R. Gheyi, E. R. Murphy-Hill, and B. Johnson, “Comparing
approaches to analyze refactoring activity on software repositories,” The
Journal of Systems and Software (JSS), vol. 86, no. 4, pp. 1006–1022,
2013.
[37] V. Kovalenko, F. Palomba, and A. Bacchelli, “Mining file histories:
should we consider branches?” in Proceedings of the 33rd ACM/IEEEInternational Conference on Automated Software Engineering (ASE),
2018, pp. 202–213.
[38] K. Shimari, T. Ishio, T. Kanda, and K. Inoue, “Near-omniscient debugging for java using size-limited execution trace,” in Proceeding of
the 2019 IEEE International Conference on Software Maintenance andEvolution (ICSME), 2019, pp. 398–401.
[39] J. R. Landis and G. G. Koch, “The measurement of observer agreement
for categorical data,” Biometrics, vol. 33, pp. 159–174, 1977.
[40] Y. S. Nugroho, H. Hata, and K. Matsumoto, “How different are different
diff algorithms in git?” Empirical Software Engineering (EMSE), vol. 25,
no. 1, pp. 790–823, 2020.
[41] Q. Luo, F. Hariri, L. Eloussi, and D. Marinov, “An empirical analysis
of flaky tests,” in Proceedings of the 22nd ACM SIGSOFT International
Symposium on Foundations of Software Engineering (FSE), 2014, pp.
643–653.
[42] Oracle. https://docs.oracle.com/javase/tutorial/java/javaoo/methods.html.
[43] CODEAHOY. https://codeahoy.com/java/top-5-java-ides/.
[44] B. Daniel, D. Dig, T. Gvero, V. Jagannath, J. Jiaa, D. Mitchell, J. Nogiec,
S. H. Tan, and D. Marinov, “Reassert: a tool for repairing broken unit
tests,” in Proceedings of the 33rd International Conference on Software
Engineering (ICSE), 2011, pp. 1010–1012.

[45] X. Li, M. d’Amorim, and A. Orso, “Intent-preserving test repair,”in Proceedings of the 12th IEEE Conference on Software Testing,
Validation and Verification (ICST), 2019, pp. 217–227.
[46] E. R. Murphy-Hill, C. Parnin, and A. P. Black, “How we refactor, andhow we know it,” IEEE Transactions on Software Engineering (TSE),
vol. 38, no. 1, pp. 5–18, 2012.
[47] L. Zhang, M. Kim, and S. Khurshid, “Faulttracer: a change impact andregression fault analysis tool for evolving java programs,” in Proceedingsof the 20th ACM Symposium on the Foundations of Software Engineering(FSE), 2012, p. 40.