An Empirical Study on the Impact of Refactoring Activities
on Evolving Client-Used APIs
Raula Gaikovina Kulaa,b,, Ali Ounia,c , Daniel M. Germand , Katsuro Inouea
a Osaka University, Japanb Nara Institute of Science and Technology, Japanc Ecole de Technologie Superieure Montreal, Canada

arXiv:1709.09474v2 [cs.SE] 28 Sep 2017d University of Victoria, Canada

Abstract
Context: Refactoring is recognized as an effective practice to maintain evolving software systems. For software
libraries, we study how library developers refactor their Application Programming Interfaces (APIs), especially when
it impacts client users by breaking an API of the library.
Objective: Our work aims to understand how clients that use a library API are affected by refactoring activities. We
target popular libraries that potentially impact more library client users.
Method: We distinguish between library APIs based on their client-usage (refereed to as client-used APIs) in order tounderstand the extent to which API breakages relate to refactorings. Our tool-based approach allows for a large-scale
study across eight libraries (i.e., totaling 183 consecutive versions) with around 900 clients projects.
Results: We find that library maintainers are less likely to break client-used API classes. Quantitatively, we findthat refactoring activities break less than 37% of all client-used APIs. In a more qualitative analysis, we show two
documented cases of where non-refactoring API breaking changes are motivated by other maintenance issues (i.e.,
bug fix and new features) and involve more complex refactoring operations.
Conclusion: Using our automated approach, we find that library developers are less likely to break APIs and tend to
break client-used APIs when addressing these maintenance issues.
Keywords: Refactoring, API Breakages, Software Libraries, Software Evolution

1. Introduction
Software libraries are constantly evolving, either responding to client needs, patching bug fixes or addressing
other maintainability concerns. Refactoring is a controlled and widely-used technique for improving the design of an
existing software, especially with modern and large-scale software systems that depend on a large number of thirdparty libraries. Fowler recommends refactoring to improve software readability and reusability, while increasing the
speed at which developers can write and maintain their code base [1, 2].
The Application Programming Interface (API) are specifications that govern interoperability between a client
application and a library. External APIs refer to the APIs available for client usage. Since clients solely rely on
APIs for ‘blackbox’ access to the library’s functionality, API backward compatibility is an important consideration
for both client and library developers. Clients migrating to a newer library version would be particularly concernedwith whether previously invoked external APIs in an older version will continue to be invoked without error. Thisis known as preserving API compatibility1 . Hence, any API change between two library versions that violates thislinkage is known as an API breakage. From a library viewpoint, a developer refactoring an external APIs may notconsider the effect its has in affecting a client’s chances of adopting the latest version. Conversely, negligence to

Email addresses: raula-k@is.naist.jp (Raula Gaikovina Kula), ali@ist.osaka-u.ac.jp (Ali Ouni), dmg@uvic.ca (Daniel M.
German), inoue@ist.osaka-u.ac.jp (Katsuro Inoue)
1 Java standards documentation at http://docs.oracle.com/javase/specs/jls/se8/html/jls-13.html
Preprint submitted to Elsevier

October 2, 2017

refactor the code base may increase the complexity and maintainability efforts (Lehman’s 2nd law), leading up to an
eventual degradation in software quality (Lehman’s 7th law) [3].
In this work, we conduct an empirical study to explore the relationship between API refactorings and breakages
based on actual API usage by clients. We distinguish between library APIs based on their client-usage (refereed to asclient-used APIs) in order to get a deeper understanding on the extent to which API breakages can be related to refactoring activities. Our investigation covers over 9,700 breaking classes and around 12,900 refactoring operations fromeight popular Java libraries, with each library having around 10∼38 consecutive releases. We observe the following:(i) library maintainers are less likely to break client-used APIs compared to other classes of the library, (ii) detectedrefactoring operations only breaking less than 37% of client-used APIs, qualitatively finding that the (iii) rest (63%)
API breakages are motivated by maintenance issues that are likely to involve more complex refactorings. Finally, we
find that (iv) simple refactorings (i.e., move method, rename method, move field) were less frequently applied toclient-used API classes compared to other classes.
Our main contributions of this paper are three-fold and can be summarized as follows: (1) our study involves
the investigation of APIs that are used by actual client, (2) using automated tooling, we conducted a large scale
empirical study to investigate API breakages and refactorings and (3) we present a large dataset of API breakages andrefactorings which is publicly available as a replication package at: http://sel.ist.osaka-u.ac.jp/people/
raula-k/APIBreakage/
The rest of the paper is organized as follows. Section 2 describes the background and definitions. Section 3
presents our approach we use in the empirical study. Section 4 details the research questions and what method is usedin the study. We then show our results in Section 5, with discussion of implications and threats of the study in Section
6. Section 7 surveys related work. Finally, Section 8 concludes the paper and presents future research directions.
2. Basic Concepts & Definitions
This section provides the necessary background and concepts that are prerequisites to understand the conductedstudy.
2.1. Backward Compatibility of APIs
The precise definition of backward compatibility depends in part on the Java language’s notion of binary compatibility2 :“binary compatible with (equivalently, does not break binary compatibility with) pre-existing binaries if
pre-existing binaries that previously linked without error will continue to link without error.”
Importantly, a class or interface should treat its accessible members (method and fields) and constructors, their existence and behavior, as a contract with its users.
In this paper, we define that any changes violating this contract are said to cause an API breakage between thelibrary and its client user. We show two examples of API breakages. The first example of an API breakage is when amethod name is modified (i.e., renamed or deleted method). For instance, the removal of the method in a class could
break the API linkage, resulting in a NoClassDefFound exception error to the client application. Conversely, adding
parameters (i.e., adding new fields, methods, or constructors) to an existing class or interface usually does not breakan API.
The second example of an API breakage is when third-parties cause an API breakage to the library, which thenindirectly breaks the client. In many cases, a library is also a client user of other libraries within their environment.
For instance, any changes to the library’s environment such as an update to the Java Development Kit (JDK) may
break a method in the library, and therefore ripples its effect to any client user of this library API.
2 documentation at http://docs.oracle.com/javase/specs/jls/se8/html/jls-13.html

2

all classes
API classes

non API
classesclient-used APIclasses≃ external API classes
non client-used APIclasses≃ internal API classes

Figure 1: A conceptual composition of all library class types. The venn diagram shows the relationship between (a) client-used API, (b) non
client-used API and (c) non API class types.

2.2. Refactoring Activities and API breakages
Refactoring is a disciplined engineering practice that restructures an existing code by altering its internal structure
without changing its external behavior [1]. Fowler discusses around seventy various refactorings, which can be either
simple or become quite complex. In this paper, we determine if any of the API breakages is related to a refactoring
activity. Formally, we define a Refactoring Operation (Ropt ) as an atomic refactoring change applied between two
library versions.
2.3. API Categorization Based on Client Usage
In this paper, we are interested in the APIs actually used by a client application, assuming that a code change
between a client-used API will cause a breakage to that contract between library and client user. To investigate theextent of which developers are breaking their APIs, we must first define the usage dimension of an API. In reality, notall public entities (APIs) are intended for client usage. Based on a developer’s intended use, an API of a library caneither be external or internal.
• External APIs - are APIs designed by library maintainers for usage by clients.
• Internal APIs - are APIs intended only for internal usage by the library code itself.
An internal API may exist for several reasons. For instance, the Finalizer class within base.internal packageof the google-guava17.0 documented [4]:While this class is public, we consider it to be *internal* and not part of our published API. It is public
so we can access it reflectively across class loaders in secure environments.
In an ideal world, internal APIs are never used by any client. However, in reality internal APIs may be subjected toclient usage. For instance, Businge et al. found that a large proportion of plugins used the Eclipse framework internal
APIs [5]. Moreover, concepts such as the Application Binary Interfaces (ABIs) [6] and the OSGi framework [7] havebeen proposed to differentiate between the two API types. However, unless explicitly documented, it is extremely
difficult to distinguish between external or internal APIs.
As shown in Figure 1, we describe the different class categories of a library. To distinguish between external andinternal APIs, we propose a method to approximate external API classes by mining actual usage by clients, defined as3

client-used API classes. Details of the method are explained in the subsequent methodology subsection. All libraryclass categories are defined as follows:• API class - is a class that has at least one public entity (i.e., method and field members) and accessible by any
client user.
• non API class - is a class that contains no API entities, i.e., private or protected.
• client-used API class (clientUse) - is an API class that is used by at least one client. It is an approximationof the external APIs. The set of client-used API classes should ideally cover all external APIs. However, there
exist cases when a client uses an internal API.
(i.e., client-used API classes ' external API classes)• non client-used API class (non clientUse) - is an API class that is not used by any client. The set of all non
client-used API classes should cover all internal APIs.
(i.e., non client-used API classes ' internal API classes)
Henceforth, we classify our API breakages at the class-level. Classes are then classified as either:• breaking class - is a changed class that is breaking its API in either class, method or field levels such asrename/move/delete changes.
• non breaking class - a changed class that does not affect API compatibility.
We then explore the extent to which breaking changes to client-used API are caused by refactoring activities.
As defined in the Section 2, Ropt is an atomic refactoring operation applied between two library versions. We now
introduce the following terminologies related to Ropt :• Ref class - is a changed API class where at least one Ropt has been applied to any of its elements. (i.e., field,
methods or class attributes).
• Ropt density - refers to the number of Ropt applied per class.
3. Approach
In this section, we first present our case study libraries and methodology used in the empirical study. Our method
includes (1) categorization of API based on client usage, (2) API breakage detection and (3) API refactoring detection.
Table 1: Studied libraries showing the releases range, number of versions, time period, and the range of number of classes per library (min-max).

Library

Release range#Versions

Releases Time Period

guava
httpclient
javassistjdomjoda-time
log4j
slf4j
xercesr03 ∼ 18.0
4.0 ∼ 4.5
2.5.1 ∼ 3.19.0
1.1 ∼ 2.0.6
0.95 ∼ 2.8
1.1.3 ∼ 1.2.17
1.1.0 ∼ 1.7.12
1.2.3 ∼ 2.11.0

22
25
28
10
22
17
38
21

Apr 10 ∼ Aug 14
Aug 09 ∼ May 15
Feb 06∼ Jan 15
Sept 04 ∼ Feb 15
Nov 05 ∼ May 15
Jun 01 ∼ May 12
Dec 06 ∼ Mar 15
Dec 00 ∼ Nov 10

4# Classes (min ∼ max)
727 ∼ 1763
230 ∼ 460
187 ∼ 334
73 ∼ 258
191 ∼ 246
242 ∼ 974
11 ∼ 28580 ∼ 1652

Table 2: Collected client-used API classes as shown in Figure 2# Collected Clients
Guava
httpclient
Javassist. JdomJoda-time
log4j
Slf4j
XercesAll clients

195
149
14
35
69
195
321
17
995

At Saturation Point (SP)used clients at SP client-used API classes at SP
98
67
11
16
20
36
20
15

184
87
30
26
27
46
9
47

3.1. Subject Libraries
We used a systematic method to select our subject libraries. Our selection of these libraries is based on thefollowing criteria: (1) have a large enough client-user API usage and (2) have sufficient evolution history. Additionally,
we required diverse libraries that (3) are from different application domains and (4) have been extensively studied inrelated work. This criteria was used to select libraries from a set of 2,500 client projects collected from GitHub.
Table 1 shows all 183 library versions from the eight selected libraries. For each library, we collected 10 to
38 different library versions. All libraries constitute a large client-base and are from different application domains.
Moreover, three out of the eight subject libraries were used in prior work [8, 9, 10]. The chosen studied libraries
range from being testing, logging, utilities and web-based libraries. As shown in the table, we selected guava [11],httpclient [12], javassist [13], jdom [14], joda-time [15], log4j [16], slf4j [17] and xerces [18] For all libraries, weonly selected consecutive version releases, ignoring release candidates. Only the official binaries and available sourcecode for each library were used in this study.
3.2. Client-Used API Extraction Method
Actual client usage is needed to distinguish between external or internal APIs. Specifically, we would need tocompile each individual client system to know what APIs are used by clients. To enable a large scale analysis, we use
the jcabi-aether [19] library and JavaCompiler (ver.1.6) Eclipse compiler [20] to dynamically compile and log
all client-loaded classes. As a result, we are able to extract the fully qualified library class name of all external APIs
for many clients.
One of the main challenges to determine client-used API collection is the coverage of all external APIs. Hence,
our technique consists of continuously collecting client systems until full coverage is reached (i.e., no more APIs areused). We coin this coverage as the saturation point reached for a library version. So instead of trying to compile as
many clients are possible, we use the saturation point as a heuristic to show that enough clients have been collected.
Figure 2 and Table 2 shows the saturation point for our case studies. The saturation point is represented as a cumulative
count of client-used API classes (x-axis) represented as a function over the number of client projects (y-axis), withthe coasting of the curve assuring confidence that a stable number of client-used API classes have been reached. For
example, of the 195 collected clients, guava reached a saturation with 98 client systems to cover 184 API classes. It
is important to note that each project was selected at random, making the formation of the curve unintentional. The
table also summarizes the number of client GitHub projects that we mined for each of the eight subject libraries (total
code base size of 600GB). To ensure maturity and quality of the client projects, the projects dataset only includes
java projects that had at least 100 commits. We ran experiments for about 30 days. The process of client-used API
collection of a single project took between 10 min ∼ 3 hours.
3.3. API Breakage Detection Method
In recent times, state–of–the–art API breakage detection tools [21, 22, 23, 24, 25] have been extensively used byboth researchers [26], [27] and practitioners [11], [12] alike, especially for a systematic comparison of API checking

5

50

100

150

80
60
40
20

Cummulative sum of# client-used API

150
100
50

Cummulative sum of# client used API

0

200

0

50

# Client Projects

100

150

# Client Projects

5

Cummulative sum of# client used API
10
15
20
25

(b) httpclient# client used API
15 20 25 30(a) guava

4

6

8

10

12

14

0

15

20

(d) jdom

20

30

40

50

30

35

15 20 25 30 35 40 45(c) javassist

25

60

0

50

100

# Client Projects# Client Projects(e) Joda-time(f) log4j

150

200

20

# client used API
25 30 35 40 45

Cummulative sum of

7
6
5
4
3
2

Cummulative sum of# client used API

8

9

10

10

# Client Projects# client used API
5
15
25
0

5

# Client Projects

Cummulative sum of# client used API

2

0

50

100

150

200

250

300

5

10

# Client Projects# Client Projects(g) slf4j(h) xerces

15

Figure 2: Cumulative count of client-used API classes (x-axis) represented as a function over the number of client projects (y-axis). The saturation
function (coasting of the curve) indicates that a stable number of client-used API classes classes have been reached (See Table 2).

6

Ref classes

breaking classes

Overlap of Ref classes that are
breaking API classes
Figure 3: Venn diagram of the overlapping relations of refactored and breaking classes.

backward incompatibilities between library versions3 As noted by Raemaekers [27], these tools are underestimations–as all detected breaking API changes will definitely break an API but some binary compatible APIs could be semantically incompatible.
To identify the API differences between two library binaries, we use the Japi-cmp library [28]. Similar to othertools, Japi-cmp is able to detect and differentiate changes in instrumented and generated classes to determine binary
compatibility as well as public or private accessibility. Using the definitions in Section 2, we then map and label allclasses as either breaking or non-breaking. Overall, the resulting dataset consists of over 9,700 detected breaking
classes from the eight libraries.
3.4. Refactorings Detection Method
To automatically collect Ropt applied between the two versions, we use the state–of–the–art Ref-Finder [29]tool. Based on template logic rules, the tool identifies up to different 52 refactoring types between two versions. It
is important to note that the collected refactorings are structural, only detectable by mechanical transformations; “Ref-Finder does not include changes that may either require restricted conditions to be met, or to some degree ofadditional specification from a developer that could not be automatically inferred by a tool” [8]. As a result, ourdataset consists of 12,900 Ropt from all eight libraries.
3.5. Mapping Refactorings to API Breakages
The study involves a mapping between the collected Ref and breaking classes, where a Ref class contains
at least one Ropt . Figure 3 describes this mapping as an intersection between breaking classes and Ref classes. It is
important to note false positives, where the tools detect refactorings in unchanged classes. Upon manual inspection ofsome cases, we confirmed these were false positives as the classes were unchanged. As a result, we semi-automatically
identified and discarded 2,100 instances of such false positives, finally leaving us with 10,800 Ropt from all eight
libraries.
A simple example of a refactoring that breaks API can be seen with the com.google.common.collect.ImmutableMultisetof the Guava library4 . According to the API Diff report, the ImmutableMultiset<E> of(E[]) method (i.e., whichtakes E[] and returns an immutable multiset) was removed between version 11.0.02 and 12.0. In this example, ourapproach automatically detects this change as the remove method Ropt . The official Java documentation states that
3 For instance, developers of the google guava library, use JDiff to report changes between two versions, e.g., API changes from guava v18 tov19 are at http://google.github.io/guava/releases/19.0/api/diffs/
4 the API change at http://google.github.io/guava/releases/12.0/api/diffs/changes/com.google.common.collect.
ImmutableMultiset.html#methods

7

Table 3: Library Class Categories Incompatibility Matrix.

client-used API
non client-used API
non API

Compatible Change

Incompatible Changes (breakchange (Lv ))API compatible
API compatible
Not affect client

API Breaking code change
Incompatible change unintended for client
Incompatible change does not affect client‘deleting a method or constructor from a class breaks compatibility with any pre-existing binary that referenced thismethod or constructor; a NoSuchMethodError may be thrown when such a reference from a pre-existing binary
is linked. Such an error will occur only if no method with a matching signature and return type is declared in asuperclass’.
4. Empirical StudyIn this section, we present the goals and motivation, followed by the method used to address each research question.
4.1. Research Questions
Our motivation is to inspect the relationship between refactorings and API breakages. Related, Dig and Johnson[9] manually inspected library release notes for documented API changes to investigate the role of refactoring during
API evolution of a library. They cited two reasons why they preferred a manual analysis over the use of automatedtools: (1) ‘since most API changes follow a long deprecation replace-remove cycle, an obsolete API can coexist with
the new API for a long time’ and (2) some behavioral refactoring cases that ‘would have been misinterpreted by atool, but a human expert can easily spot’. In this study, we find that state–of–the–art tools are now able to detect
deprecations, thus negating the first reason. Additionally, we find that the automated approach is not as reliant on
documentation.
Our goal in this study is to use an automated approach to investigate how client usage-APIs are affected by the
refactoring activities. The automated approach has the benefit of reducing manual inspection and heuristic errors andenables a large-scale empirical study. We designed a rigorous quantitative empirical study, formulating the following
research questions:• (RQ1). To what extent are library maintainers breaking client-used APIs over time? We want to understand the
API breaking tendencies of library maintainers.
• (RQ2). To what extent are refactoring activities breaking client-used APIs? Sometimes API breakages areunavoidable, even for the more popular client-used APIs. Prior work indicates that refactoring is common with
API changes. Therefore, we want to understand how much of client-used API breakage is related to refactoring
activities.
In RQ2, we identified many API breakages not related to refactoring activities. We then formulated RQ3 and
RQ4 for a deeper analysis of the detected changes (both refactoring and non refactoring related) that break client-used
APIs:• (RQ3). What non-refactoring-related code changes are breaking client-used APIs? Specifically, our motivation
is to understand what API breaking changes are not related to refactorings.
• (RQ4). What refactoring-related code changes are breaking client-used APIs? From the perspective of all
refactoring activities, we would like to understand (i) how much and (ii) types of refactoring operations that are
breaking client-used APIs.

8

4.2. Research Method for RQ1
To answer RQ1, we followed two steps. First, we studied consecutive versions of a library to understand thelibrary evolution. The goal is to study how (i) client-used API classes, (ii) non client-used API classes and (iii) non
API classes evolve over several consecutive versions. Next, we investigate the number of code changes that lead toincompatibility with respect to the different class categories that we defined above. Since the tool is only able tocompare two versions at a time, we performed a side-by-side (i.e., each comparison is the current version against theimmediate successive library version). We introduce a normalized metric namely breakchange to describe the rate of
the number of breaking changes over all class changes at that version release as defined in Equation 1:
breakchange (Lv ) =|breaking classes||all changed classes|(1)

where Lv refers to a given library version and ranges from 0 6 breakchange 6 1 for each class category of Lv . Values
that are closer to 1 indicate that there are more breakages per class changes.
Table 3 shows the breakchange (Lv ) metric interpretation based on the class type. Hence, the breakchange (Lv ) metrichas different interpretations based on the class type. For instance, for non API classes, the metric shows significant
changes that do not affect clients. We believe that it is important to track which classes are more prone to incompatible
code changes. To assess the significance of breakages between the different library class categories, we use the Kruskal
Wallis and Mann-Whitney non-parametric test. The null hypothesis would state no statistical difference between theclass types. Furthermore, to assess the difference magnitude, we study the effect size based on Cohen’s d [30]. The
effect size is considered: (1) small if 0.2 6 d < 0.5, (2) medium if 0.5 6 d < 0.8, or (3) large if d > 0.8. For the effectsize, we use the Mann-Whitney tests with Bonferroni correction.
4.3. Research Method for RQ2
For RQ2, our method is to identify library refactorings that are applied to client-used API classes. We followed
two steps. To analyze the impact of the refactoring activities, we first identified for each library the (i) number ofRef classes and (ii) Ropt density. We then identified the Ref classes that are breaking. To map refactorings to API
breakages as described in Section 3.5, we introduce a normalized metric namely breaking–to–Ref rate as Equation 2:
breaking–to–Ref (Lv ) =|(Ref ∩ breaking classes)||Ref |(2)

where Lv refers to a given library version. The metric breaking–to–Ref rate(Lv ) returns a percentage that ranges from
[0..100%] for each class category of Lv . Values that are closer to 100% indicate that there are more refactorings thatare breaking each of the different class categories. Conversely, from an API breakage perspective, we now introduce
a normalized metric namely Ref–to–breaking rate to describe the ratio of overlap with respect to all breaking classes
as defined in Equation 3:
Ref–to–breaking (Lv ) =|(Ref ∩ breaking classes)||breaking classes|(3)

where Lv refers to a given library version. The metric Ref–to–breaking rate(Lv ) returns a score that ranges from [0..1]
for each class category of Lv . Values that are closer to 1 indicate that there are more breakages that are related torefactoring activities.
4.4. Research Method for RQ3
For RQ3, we used a qualitative approach to investigate the breaking APIs changes that were not detected in our
approach as refactoring operations. Results from the prior RQ2 (See Section 5.2 Table 5) indicate that three of thesix projects (Guava, HttpClient and xerces) have many client-used API breakages that were not related to refactoring
activities. We consulted related change logs of these three projects; Guava5 , HttpClient6 and Xerces 7 to understand
5 and example of Release 11 https://github.com/google/guava/wiki/Release11
6 https://archive.apache.org/dist/httpcomponents/httpclient/RELEASE_NOTES-4.5.x.txt
7 change logs at https://xerces.apache.org/xerces2-j/releases.html

9the reason why these API breakages were performed by the developer. We manually checked documented change logs
of each release to map an API breakage to either a bug fix issue or to accommodate a new enhancement (feature) in thelibrary. To reduce bias, a manual check was carried by a team of three researchers (one postdoctoral and two graduate
master students) persons with have an intermediate level of java programming and software development. Since teammembers do not posses any project-specific knowledge, we solely rely on keywords or issues links (i.e., issueID)in the change log comments to map each API breakage with a bug issue or new features. Xerces was later removed
from the analysis as there was too many ambiguous references with no clear linkage to the source code. Analysis will
include the aggregation of all API documented changes as either bug fixes or new features and show how many canbe mapped to the API breakages that did not involve any refactoring operations.
For a deeper analysis and validation, we will investigate and present some examples of these non-refactoring
related API breaking classes.
4.5. Research Method for RQ4
For RQ4, we identified what refactoring operations are breaking client-used APIs. We followed two steps in theanalysis. For a library, we aggregated the number of Ref instances where a certain Ropt (e.g., move method) has been
applied. In the second step, we used a normalized metric prsv to describe the ratio of overlapped breaking refactorings
between client-used API classes and non client-used API classes as defined in Equation 4.
P|client-used API ∩ breaking ∩ Re f classes|
Lv ∈L
(4)
prsv (L, Ropt ) = P|non client-used API ∩ breaking ∩ Re f classes|
Lv ∈L

where L refers to a given library, Ropt refers to a certain refactoring operation type.
Our hypothesis is that a prsv ratio less than 1 ( 0 6 prsv < 1) indicates that developers are applying less refactoring
operations to client-used API classes. Conversely, a high prsv ratio (prsv > 1 ) indicates that more refactoring
operations are applied to client-used API classes. A value of 1 indicates that the certain Ropt type is equally applied toboth client-used API classes and non client-used API classes.
5. Results
In this section, we present our results of the study by addressing each of the four research questions.
5.1. Findings for RQ1
Figures 4, 5 and 6 depict class category analysis of each consecutive library version. Each figure shows theevolution of (i) client-used API classes, (ii) non client-used API classes and (iii) non API classes over consecutive
library versions. From these figures, we summarize our findings with three observations (i.e., Figures 4a∼6b). First,
we observe that most libraries are composed of non client-used API classes categories (green line), showing thatlibraries usually have more non client-used API classes than client-used API classes. The exception is log4j, whichis shown in Figure 5c to have most APIs intended for external API usage. Interestingly, we see in Figure 4c that nonclient-used API classes of javassist disappears from the more recent libraries. Upon closer inspection, we noticed that
this was because developers had changed these non client-used API classes into non API classes. Second, we observea stable number of client-used API classes (red line) shown across all projects. From a client user viewpoint, thefindings indicate that developers of a library are less likely to expand their external APIs. The obtained results showthat the number of non API classes (blue line) is constantly changing (i.e., illustrated by various peaks) over time. We
find that some of the peak changes can be correlated to different events, such as a major or specially-named releases,
beta releases such as xercesImpl2.x.x jaxb and log4j1.3alpha6 , or modifying private non API classes into public APIs
such as in the case of httpclient.
Library maintainers are less likely to apply client-used API classes changes compared to other class categories.

10

11

javassist− 3.11.0.GA − 3.12.0.GA

non client-used API
httpclient− 4.5

httpclient− 4.4.1

httpclient− 4.4

httpclient− 4.3.6

httpclient− 4.3.5

httpclient− 4.3.4

httpclient− 4.3.3

httpclient− 4.3.2

httpclient− 4.3.1

httpclient− 4.3

non client-used API

javassist− 3.18.2_GA − 3.19.0_GA

javassist− 3.18.1_GA − 3.18.2_GA

javassist− 3.18.0_GA − 3.18.1_GA

javassist− 3.17.1_GA − 3.18.0_GA

javassist− 3.17.0_GA − 3.17.1_GA

javassist− 3.16.1_GA − 3.17.0_GA

javassist− 3.15.0_GA − 3.16.1_GA

javassist− 3.14.0_GA − 3.15.0_GA

javassist− 3.13.0_GA − 3.14.0_GA

javassist− 3.12.1.GA − 3.13.0_GA

javassist− 3.12.0.SP1 − 3.12.1.GA

httpclient− 4.2.6

httpclient− 4.2.5

httpclient− 4.2.4

httpclient− 4.2.3

httpclient− 4.2.2

httpclient− 4.2.1

httpclient− 4.2

httpclient− 4.1.3

httpclient− 4.1.2

httpclient− 4.1.1

httpclient− 4.1

httpclient− 4.0.3

httpclient− 4.0.2

httpclient− 4.0.1

client-used API

javassist− 3.12.0.GA − 3.12.0.SP1

client-used API

javassist− 3.10.0.GA − 3.11.0.GA

javassist− 3.9.0.GA − 3.10.0.GA

javassist− 3.8.1.GA − 3.9.0.GA

javassist− 3.8.0.GA_brew − 3.8.1.GA

javassist− 3.8.0.GA − 3.8.0.GA_brew

javassist− 3.7.1.GA − 3.8.0.GA

javassist− 3.6.0.GA − 3.7.1.GA

javassist− 3.5.0.GA − 3.6.0.GA

javassist− 3.4.GA − 3.5.0.GA

javassist− 3.3.GA − 3.4.GA

javassist− 3.3 − 3.3.GA

javassist− 3.1 − 3.3

javassist− 3.0 − 3.1

javassist− 2.5.1 − 3.0# of classes# of classes

guava− 18.0

guava− 17.0

guava− 16.0.1

guava−16.0

guava− 15.0

guava− 14.0.1

guava− 14.0

guava − 13.0.1

guava− 13.0_final

guava− 13.0

guava− 12.0.1

guava− 12.0

guava− 11.0.2

guava− 11.0.1

guava− 11.0

guava− 10.0.1

guava− 10.0

guava− r09

guava− r08

guava− r07

guava− r06

guava− r05# of classes
1250

1000

750

500

250

Library versions
non API(a) Guava

300

200

100

0

Library versions

non API(b) Httpclient

300

200

100

Library Versions(c) Javassist

Figure 4: An evolution of changed classes per class types for (a) guava, (b) httpclient, (c) javassist. These figures show the different # of classes
identified in chronological order of release.

client-used API

12non client-used API
joda_time− 2.7 − 2.8

joda_time− 2.6 − 2.7

joda_time− 2.5 − 2.6

joda_time− 2.4 − 2.5

joda_time− 2.3 − 2.4

joda_time− 2.2 − 2.3

joda_time− 2.1 − 2.2

joda_time− 2.0 − 2.1

joda_time− 1.6.2 − 2.0

non client-used API

log4j − 1.3alpha6

log4j− 1.2.17

log4j− 1.2.16

log4j− 1.2.15

log4j− 1.2.14

log4j− 1.2.13

joda_time− 1.6.1 − 1.6.2

joda_time− 1.6 − 1.6.1

joda_time− 1.5.2 − 1.6

joda_time− 1.5.1 − 1.5.2

joda_time− 1.5 − 1.5.1

joda_time− 1.4 − 1.5

joda_time− 1.3 − 1.4

joda_time− 1.2.1 − 1.3

joda_time− 1.2 − 1.2.1

joda_time− 1.1 − 1.2

joda_time− 1.0 − 1.1

joda_time− 0.95 − 1.0# of classesclient-used API

log4j− 1.2.12

log4j− 1.2.11

log4j− 1.2.9

log4j− 1.2.8

log4j− 1.2.7

log4j− 1.2.6

log4j− 1.2.5

log4j− 1.2.4# of classes

jdom2− 2.0.6

jdom2− 2.0.5

jdom2− 2.0.4

jdom− 2.0.3

jdom− 2.0.2

jdom− 2.0.1

jdom− 2.0.0

jdom− 1.1.3

jdom− 1.1.2# of classes
150

100

50

0

Library versions
non API(a) Jdom

200

150

100

50

0

Library Versions(b) Joda-time

600

400

200

0

Library versions

non API(c) log4j

Figure 5: An evolution of # of classes per class types for (a) jdom, (b) joda-time and (c) log4j libraries. Similar to Figure 4, these figures show thedifferent # of classes identified in chronological order of release.

client-used API

13

non client-used API
xercesImpl− 2.11.0

non client-used API

xercesImpl−2.10.0

xercesImpl− 2.9.1

xercesImpl− 2.9.0

xercesImpl− 2.8.1

xercesImpl− 2.8.0

xercesImpl− 2.7.1

xercesImpl − 2.6.2_jaxb

xercesImpl− 2.6.2

client-used API

xercesImpl− 2.6.1

xercesImpl− 2.6.0

xercesImpl− 2.5.0

xercesImpl− 2.4.0

xercesImpl− 2.3.0

xercesImpl− 2.2.1

xerces− 2.0.2

xercesImpl− 2.0.1

xerces− 2.0.0

xerces− 1.4.4

xerces− 1.4.0# of classes
slf4j_api− 1.2
slf4j_api− 1.3.0
slf4j_api− 1.3.1
slf4j_api− 1.4.0
slf4j_api−1.4.1
slf4j_api− 1.4.2
slf4j_api− 1.4.3
slf4j_api− 1.5.0
slf4j_api− 1.5.2
slf4j_api− 1.5.3
slf4j_api− 1.5.4
slf4j_api−1.5.5
slf4j_api− 1.5.6
slf4j_api− 1.5.7
slf4j_api− 1.5.8
slf4j_api− 1.5.10
slf4j_api− 1.5.11
slf4j_api−1.6.0
slf4j_api− 1.6.1
slf4j_api− 1.6.2
slf4j_api− 1.6.3
slf4j_api− 1.6.4
slf4j_api 1.6.5
slf4j_api− 1.6.6
slf4j_api− 1.7.0
slf4j_api− 1.7.1
slf4j_api−1.7.2
slf4j_api− 1.7.3
slf4j_api− 1.7.4
slf4j_api− 1.7.5
slf4j_api− 1.7.6
slf4j_api− 1.7.7
slf4j_api−1.7.8
slf4j_api− 1.7.9
slf4j_api−1.7.10
slf4j_api− 1.7.11
slf4j_api−1.7.12# of classes
15

10

5

Library versions

non API(a) Slf4j

750

500

250

0

Library versions

non API(b) Xerces

Figure 6: An evolution of # of classes per class types for (a) slf4j and (b) xerces libraries. Similar to Figure 4 and Figure 5, these figures show thedifferent # of classes identified in chronological order of release.

1.00

0.75

0.50

0.25

client-used API classes

Non-client-used API classes

xerces

slf4j

log4j

joda_time

jDom

javassist

httpclient

guava

0.00

non−API classes(a) For each of the eight libraries, we show the breakchange comparing (1) client-used API classes in red, (2) non
client-used API classes in green and (3) non API classes in blue.

# versions

breaking class instanceschanged class instances

Guava
httpclient
Javassist
Jdom
Joda-time
log4j
Slf4j
Xerces

22
25
28
10
22
17
38
21

2,215
113
1,017
106
1,097
583
21
4,622

9,973
1,426
2,572
445
2,922
3,051
235
7,796

Totals183

9,774

28,420(b) Corresponding to Figure 7a, this table shows the # of analyzed (1) library versions, (2)breaking and (3) changed classes collected.
Figure 7: Results of the breakchange rates for all eight libraries analyzed.

1.00
0.75
0.50
0.25
0.00
client-used API

non-client-used API

non API

Figure 8: Summary of breakchange comparing (1) client-used API classes in red, (2) non client-used API classes in green and (3) non API classes in
blue.

14

Table 4: The table reports (a) number of Ref classes and (b) Ropt density per Ref class ( x̄ = |Ropt |). Note that (–) represents no matches.

non
API

ient
Use

tUse

I

clien
tUse

non
cl

Ropt density(Median)guava
httpclient
Javassistjdom
Joda-time
log4j
slf4j
xerces

clien

guava
32
httpclient 3
Javassist
111
jdom
1
Joda-time
30
log4j
1
slf4j
–
xerces44

non breaking

nonAP|Re f |

143
11
8
1
–
–
1
244

44
–
6
–
11
–
–
31

31
6
29
–
29
2
2
23

24
16
10
3
5
–
1
104

139
7
2
–
12
3
–
66

8
5
5
1
–
–
1
6

5
–
2
–
10
–
–
4

4
2
1
–
1
2
3
4

4
3
1
4
1
–
22
4

8
2
1
–
1
2
–
5

non

clien
tUse

breaking classes

7
5
2
1
3
1
–
8

Figure 7a shows the breakchange rates for all eight libraries. From this figure, we observe that except for javassist
and joda-time, library developers are more likely to break non client-used API classes than client-used API classes.
Related, Figure 8 depicts the breakchange rates grouped by all class categories. The Figure shows that non clientused API classes are more prone to breakages than client-used API classes for all libraries. As shown in the Figure,non client-used API classes are reported to have the most breakages. A Kruskal Wallis test revealed a significant
differences between client-used API classes, non client-used API classes and non API classes values (p<0.01). The
post-hoc test using Mann-Whitney tests with Bonferroni correction proves the effect size to be medium (p<0.01, r =0.54) when comparing all class categories.
Findings show that incompatible API code changes are statistically more likely to occur in non client-used APIclasses compared to client-used API classes.

5.2. Findings for RQ2Table 4 presents a summary of Ref classes and their Ropt density. For instance, we identified 32 guava clientused API classes that were Ref classes. Out of the 32 Ref classes, we report a median of 7 Ropt that were applied
per Ref class. From this table, we can see that, in general, library maintainers applied more Ropt to non client-used
API classes and non API classes, as compared to client-used API classes, except for javassist. For example, thetable shows that for xerces, around 244 non client-used API classes were refactored, compared to 44 client-used APIclasses. In more detail, the results show that apart from slf4j, the median density of Ropt per class ranges from 1 to
10 Ropt at most. Interestingly, we find that slf4j had a high number of Ropt applied to one non-breaking client-used
API classes ( x̄=22). log4j, slf4j and jdom libraries reported only a few breaking classes matched to Ropt , which isconsistent with recent empirical studies conducted by Cossette et al. [8].
Table 5 reports the median values of Ropt that cause API breakages. We use this table to compare between clientused API classes and non client-used API classes. For guava, non client-used API classes ( x̄=9) were breaking due
to refactorings compared to client-used API classes ( x̄=2). From this table, we find that non refactoring changes are
15

brea
king
–to–
R

te
ng r
a

non
brea
king

o–b
reak
i

king
brea

2
1
2
1
15
0
−
3

2
1
1
0
2
1
1
2

166
46
3
1
65
87
41
24

251
74
1
16
45
75
136
16

1%
1%
37%
1%
6%
0%
0%
10%

53%
55%
75%
1%
48%
0%
0%
64%

guava
httpclient
Javassistjdom
Joda-time
log4j
slf4j
xerces

22
25
28
10
22
17
38
21

9
2
4
1
–
0
0
14

2
1
1
1
1
−
1
7

91
8
2
13
9
36
2
210

20
18
1
3
7
−
2
22

9%
14%
5%
1%
–
0%
0%
6%

86%
58%
44%
22%
–
−
0%
68%

# ve

Ref–
t

non
brea
k

22
25
28
10
22
17
38
21

ing

brea
k

guava
httpclient
Javassistjdom
Joda-time
log4j
slf4j
xercessrsionclientUse

ing|non Re f |(Median)non clientUse|Re f |
(Median)ef rateTable 5: Matrix that shows the median average # refactored API classes per library. For each library, we summarized the median values across all
library versions. Table includes median ( x̄) of matched refactored classes. 0 represents a value less than 0.01. (–) reports no matched classes.

more likely to break client-used API classes than non client-used API classes. Moreover, applied refactorings tend to
break more non client-used API classes compared to client-used API classes. The results show that many of the API
breakages are not mapped to the detected refactorings (i.e., non Ref classes). We find that more refactoring nonclient-used API classes are breaking compared to refactored client-used API classes, with the exception of Javassist.
Table 5 also shows the breaking–to–Ref and Ref–to–breaking rates. We report that the median Ref–to–breaking
rate for client-used API classes is up to 37% across all projects ( x̄=1%∼37%). Except for javassist, the result provides
evidence the detected API breakages could not be mapped to refactoring operations. Alternatively, the breaking–to–
Ref rates reported for client-used API classes in Table 5 indicates that breaking refactorings accounted for a medianrange of up to 75% of all Ropt . The highest breaking-Ref rate for non client-used API classes was 86%, reported forthe guava library.
Findings show that up to 75% refactored API classes are breaking their client-used APIs. However, these API
breaking refactorings account for less than 37% of all client-used API breakages.
5.3. Findings for RQ3
Table 6 shows results of the manual study of API breakages that did not map to any Ref Classes in developer
documentation (i.e., change logs). For instance, release 11 of the guava library listed 26 issues and 13 new features8 .
The table confirms our results that find these client-used API class breakages are not only related to refactoring activities (i.e., non Ref Class). We find that all four API breakages could be mapped to the API documented changes.
From the table, we were able to map 82% of non refactoring client-used API breakages to the API documentation forguava and 58% for httpclient. This finding indicates that many of the API breakages not involved in refactorings aremost likely motivated by maintenance issues such as bug fixes and for new feature enhancements.
8 The release notes are available at https://github.com/google/guava/wiki/Release1116

@@ -254,16 +258,28 @@ final Type resolve(final TypeVariable<?> var) {
254

258

255

259

*/

256

260

Type resolveInternal(TypeVariable<?> var, TypeTable forDependants) {* <p>Should only be called and overridden by {@link #resolve(TypeVariable)}.

257
261

-

Type type = map.get(var);+

Type type = map.get(new TypeVariableKey(var));

258

262

259

263

Type[] bounds = var.getBounds();

260

264

if (bounds.length == 0) {

261

265

262

266

if (type == null) {return var;
}

263

-

264

-

var.getGenericDeclaration(),265-

var.getName(),

266

-

267

return Types.newTypeVariable(

new TypeResolver(forDependants).resolveTypes(bounds));

267+

Type[] resolvedBounds = new TypeResolver(forDependants).resolveTypes(bounds);

268+// Starting from JDK 7u51, JDK built-in TypeVariable implementations aren't equal to custom

269+// implementations any more. So if a type variable isn't changed during resolution, we avoid

270+// constructing custom TypeVariable implementation.

271+// Only do this shortcut under the new JDK behavior. It doesn't work under the current (old)272+// JDK behavior because currently <T extends B> and <T extends String> are considered equal273+// when B resolves to String.

274+// That means even when the bounds compare equal, they may still have resolved to some275+// different types.

276+// In the new JDK, this shortcut works fine.

277+

if (Types.NativeTypeVariableEquals.NATIVE_TYPE_VARIABLE_ONLY

278

+

279

+

280

+

}

281

+

return Types.newArtificialTypeVariable(

282+&& Arrays.equals(bounds, resolvedBounds)) {return var;

var.getGenericDeclaration(), var.getName(), resolvedBounds);

283}(a) This code change in the method resolve was detected as breaking API compatibility for users of the older JDK.
4

guava/src/com/google/common/hash/ChecksumHashFunction.java

View@@ -85,9 +85,9 @@ public HashCode hash() {8585* cast won't lose any information and isnecessary to return a HashCode of the correct86* size.

86

87

*/

87

return HashCodes.fromInt((int) value);

88

88

-

89
90

*/
+

return HashCode.fromInt((int) value);} else {+return HashCode.fromLong(value);

91

}

92
93

90

return HashCodes.fromLong(value);91* size.

89} else {-* cast won't lose any information and isnecessary to return a HashCode of the correct

}

92

}

93

}

}
}

(b) This code change breaks API compatibily with the replacement HashCodes to HashCode in the method. This refactoring was
missed by the automated approach
Figure 9: We show two examples of API breaking changes that were not mapped to detected refactoring operations (i.e., non-Ref). We conjecturethat these changes are (a) in response to a complex defect in the code and (b) consist of a complex refactoring that is not captured by the automatedapproach.

17

Table 6: Shows for a library release (i) the number of issues and new features per version release analysis and (ii) the number of non-refactoring
related API breaking classes. We also show in the number of these API breakages mapped to the change log comments.

Library
Guava

Httpclient

Change Log # Issues # New Features
Release
v11
v12
v13
v14
v15
v16
v17
v18
v4.1.2v4.1.3v4.2
v4.2.1v4.2.2v4.2.3v4.2.4v4.2.5

26
43
26
64
53
19
11
21
263
5
4
15
8
8
21
6
6
73

13
24
28
10
11
8
5
7
106
4
9
4
17

|non Re f |∩ breaking∩ clientUse(# mapped to change logs)
4 (4)
4 (4)
7 (6)
5 (5)
7 (6)
5 (4)
6 (4)
3 (2)
41 (34) 82%
3 (2)
3 (2)
3 (1)
4 (2)
4 (2)
2 (2)
3 (1)
2 (2)
24 (14) 58%

From our manual analysis and similar to a study by Murphy-Hill et al. [31], we find that not all API changes
appear in the API change logs. Figure 9 does show two documented case examples of API breakages that are notmapped to a detected refactoring operation (i.e., non-Ref). These examples provide evidence that these many clientused API class breakages are: (a) motivated by a bug fix or new feature or (b) consists of a complex refactoring that is
not captured by the automated approach. In the first example (i.e., Figure 9a, we show an unavoidable API breaking
change, especially if it is used to fix a complex defect such as a third party library. This API breaking change wastriggered in response to an error reported by a client user ”JDK and Guava TypeVariable implementations are nolonger compatible under 1.7.0 51-b13” 9 It was widely reported to affect many client users of the library. Developers
found that a change in the standard Java library (JDK) causes guava to break API compatibility, as prior guava version
implemented an undocumented internal API of the JDK (i.e., Types.TypeVariable.newTypeVariable())10 . After much discussion among developers, the accepted API change was documented to ‘conditionally work only under
the new JDK’.
In the second example (i.e., Figure 9b), we acknowledge cases where the automated approach is unable to detect
more complex refactoring operations. Soares et al. [32] showed that Ref-Finder is unable to correctly detect all typesof refactoring operations, which is a validity threat and is discussed in detail (See Sections 6.3 and Section 6.4).
Moreover, this change is listed as a submitted enhancement issue11 related to ‘Move HashCodes static methods to
HashCode’ and involves 17 changed files (261 added and 219 deleted lines of code)12 .
Findings indicate that many client-used API breakages are likely to be motivated by other maintenance issues(i.e., bug fixes and new features) and involve more complex refactoring operations.

9 issue at https://github.com/google/guava/issues/1635 and fix at https://goo.gl/bqDpxU
10 A blogger discussions by users is at https://goo.gl/8tcHfY
11 urlhttps://github.com/google/guava/issues/1495
12 the code change is at https://goo.gl/JHVi5J

18

Table 7: Classification of Ropt for API classes with presver ratio. Note that one class may be classified under several refactoring types. Note (–) represents no matches. We also show the total of
all breakages (cu. + ncu.) and use to colors to highlight when prsv = low and prsv = high.

clientUse

non clientUse

non clientUse

prsv

clientUse(53) 1.12 20−(40) 0.25 3−−(9) 2.00 −(8) 1.00 661(76) 0.15 9(34) 0.08 8
1−−−(5) 5.00 1−(50) 1.50 16(82) 0.52 1(4) 1.00 12−−(50) 0.47 8
2
−
1
−
−

4
−
−
−
−
2
1
2
2
7
14
−
3
−
3
12
−
2
4
7
−
−
20
−
1
−
−
−

1
−
−
−
−
1
2
−
−
−
3
−
−
−
2
2
−
1
−
−
−
−
−
−
−
−
2
−

4
4
8.4 14.78

1.00 3
1.19 5.14

3
4.4

2
1
1.75 3.2

Median ( x̄)
Mean (µ)

non clientUse

non clientUse

25
−
32
−
1
3
4
10
−
66
19
−
3
2
2
1
−
20
54
2
−
9
34
−
−
6
−
−

non breaking

clientUse

clientUsechange parameter
28
cdcf*
−
extract method
83extract subclassextract superclass−6
inline method
inline temp
4
introduce explaining variable −−introduce null object
move field
10
15
move method
pull up constructor body−−pull up field
push down field
−
ratp*
−
remove control flag
5
1
remove middle man
remove parameter30
rename method28
rcwfm*
2
replace data with object
2
replace exception with test−
rmnwc*
16
rmwmo*
4
rncgc*−
replace temp with query−pull up method−
extract interfacea−

xercesbreaking

non clientUse

prsv

19

non clientUse

non breaking

clientUse

breaking

httpclient
breaking
non breaking
clientUse

guava

prsv

Classification of Ropt

10 (11) 0.10
1
1
−
−
−
−
−
−
4
6 (9) 0.50−−−−
1 (3) 2.00−
8 (9) 0.12
−
−
−
1
8
−
−
−
1 (3) 2.00
−

−
4
−
−
−
−
−
−
−
−
−
−
−
−
2
−
−
1
2
−
1
−
−
1
−
−
−
−

11
1
2
−
−
−
−

217
74
61
1
2
23
20
33
−
255
256
−
5
41
14
11
−
170
217
−
11
3
165
40
27
−
2
1

(273) 0.26
(118) 0.59(70) 0.15

−
−
7
−
−
−
5
5
−
−
−
−
−
−
17
−
−
−
−
−

56
44
9
−
−
7
8
8
−
15
12
−
−
−
2
1
−
45
146
−
4
−
49
4
9
−
5
−

−
8
5
−
−
1
−
2
−
7
16
−
−
−
−
1
−
−
1
−
−
−
9
1
−
−
−
9

19
48
47
−
−
1
7
21
−
5
12
−
−
−
9
6
−
18
19
−
2
4
117
5
14
1
−
−

0.50 10.95 1.71

5
6.17

9
17
24.9 62.70(30) 0.30(28) 0.40(41) 0.24(270) 0.06
(268) 0.05

(16) 0.14(12) 0.09(62) 0.26(363) 0.67
(15) 0.36(214) 0.30(44) 0.10(36) 0.33(7) 2.50

0.33 511
0.46 5.45 19.72∗ Note types abbreviations - cdcf = consolidate duplicate cond fragment, rcwfm = replace constructor with factory method, ratp = remove assignment to parameters, rmnwc =replace magic number with constant, rmwmo = replace method with method object, rncgc = replace nested cond guard clause5.4. Findings for RQ4Table 7 shows a classification of the applied Ropt for the three libraries guava, xerces, and httpclient in ourcollected dataset. As seen in the table, guava developers applied the change parameter Ropt 28 times to breaking
client-used API classes. Developers subsequently applied the same Ropt and broke 25 non client-used API classes during the library evolution of guava. We find that the guava and xerces libraries tend to refactor and break their versions
during evolution than httpclient. Our results align with the findings of Cossette et al. [8] on API transformations,where they also used the same libraries in their experiments. Results in Table 7 show that developers apply specific Ropt more frequently when evolving their libraries. For instance, Ropt such as move method (guava- 34 Ropt ),change parameter (httpclient- 11 Ropt ), and rename method (xerces- 363 Ropt ) were the most frequently applied that cause API breakages. For client-used API classes, remove parameter (guava- 30 Ropt ), move method(httpclient- 3 Ropt ) and rename method (xerces- 146 Ropt ) are reported as most frequent. Notably, move method(guava- 190 Ropt , xerces- 256 Ropt ), remove parameter (httpclient- 8 Ropt ) were applied to non client-used APIclasses.
Table 7 also reports the prsv ratio for each library. This metric measures the degree of likelihood to which librarydevelopers apply certain Ropt to client-used API classes compared to non client-used API classes (i.e., preserving
client-used API classes). We use color to highlight the prsv scores. Green highlights in the table represents a low
preservation of client-used API classes, while the red highlights indicates a high ratio of Ropt in non client-used APIclasses. For example, the library developers of both guava (prsv= 0.08) and xerces (prsv = 0.05) tend to apply lessmove method refactoring operations to client-used API classes. Our results shows library maintainers are less likely
to refactor (using the more frequent Ropt ) client-used API classes than non client-used API classes. For example, 5 out
of 10 Ropt in guava, 3 out of 5 Ropt types in httpclient, and 16 out of 17 Ropt types in xerces are less likely applied toclient-used API classes. We find that many high prsv ratios (depicted by red in the table) where by the rarely applied
Ropt types (e.g., remove control flag (guava- 5 Ropt , httpclient- 2 Ropt ) and pull up method (httpclient- 3 Ropt ,xerces- 7 Ropt ).
Findings show that library maintainers were more likely to refactor non client-used API classes compared toclient-used API classes.

6. Discussion
In this section, we first discuss the implications of results and then compare with related work. We then discuss
some challenges of our approach and finally present threats to the validity of our study.
6.1. Implications
Our results indicate that when evolving libraries, out of all code changes applied, maintainers are less likely to apply incompatible code changes to external API classes compared to the other classes during the library evolution. Thisimplies that library developers may understand the efforts by clients needed to update their libraries. Complementary
to this finding, Bloch mentions the growing awareness of library maintainers to APIs [33]. This is also reinforced by
Seo et al. [34] where they found that there are many cases where API breakage changes are only applied when unavoidable (i.e., in response to either vulnerabilities or needed bug fixes etc...). There are benefits to this awareness ofclient-used API breakages. In particular, the evolution of APIs encourages trust and reduce the latency of adoption byclient projects, which is currently being experienced as a problem by many OSS clients [35]. Larman [36] introduced
a notion of the Protected Variation (PV) pattern: identify points of predicted variation and create a stable interface
around them. This PV pattern could explain how contemporary developers build and evolve libraries in relation toclient-used APIs.
6.2. Comparison to Literature
It is important to understand that our work cannot be simply compared at face-value to prior studies. As outlined
in Section 4.1 there are obvious differences with our approach, compared to the studies of Dig and Johnson [9] andCossette et al [8]. Dig and Johnson used the change logs as heuristic to locate all API changes, and other considered
20

250

+

Iterable<Number> onlyPresent =Optional.presentInstances(optionals);

247

251

+

252

+

253

+

254

private static Optional<Integer> getSomeOptionalInt(){255

return Optional.of(1);

return Optional.of(1);

256

}

10

private static Optional<Integer> getSomeOptionalInt()
{

248
249

ASSERT.that(onlyPresent).hasContentsInOrder(2);}}

guava/src/com/google/common/base/Optional.java

View@@ ­207,19 +207,21 @@
207

207

* skipping over occurrences of {@link
Optional#absent}. Iterators are unmodifiable and are* skipping over occurrences of {@link
Optional#absent}. Iterators are unmodifiable and are

208* evaluated lazily.

208

209

*

209

* @since 11.0

210

*/

211

@Beta

212

public static <T> Iterable<T> presentInstances(final

213

+

214

+

210

­

211
212
213

­

* evaluated lazily.
*+* @since 11.0 (generics widened in 13.0)*/@Betapublic static <T> Iterable<T> presentInstances(

Iterable<Optional<T>> optionals) {final Iterable<? extends Optional<? extends T>>

optionals) {214

checkNotNull(optionals);

215

checkNotNull(optionals);

215

return new Iterable<T>() {216

return new Iterable<T>() {

216

218

217

@Override public Iterator<T> iterator() {

217
­

219

private final Iterator<Optional<T>> iterator =checkNotNull(optionals.iterator());

return new AbstractIterator<T>() {+private final Iterator<? extends Optional<?

extends T>> iterator =
220

219

+

checkNotNull(optionals.iterator());

221

220

222

@Override protected T computeNext() {

221
222

@Override public Iterator<T> iterator() {218

return new AbstractIterator<T>() {­@Override protected T computeNext() {223

while (iterator.hasNext()) {224

Optional<T> optional = iterator.next();

while (iterator.hasNext()) {+Optional<? extends T> optional =iterator.next();
223

if (optional.isPresent()) {

225

224

return optional.get();

226

225

227

}

if (optional.isPresent()) {return optional.get();}

0 commentson commit
254bde6
Figure10: Example
of a misidentified
detected refactoring-related API breaking change (Ref ∩ breaking class).Ref-Finder detects thiscode change between Guava version 12 and 13 as an add parameter Ropt , while the API breaking tool reports it as binary incompatible
modified method.
Write

Preview

public entities are APIs. In this study, we detect syntactic changes in classes to infer changes and determine client
usage to identify if the change has an effect to its users. As a result, our approach is unable to detect behavioral API
breakages. Dig and Johnson’s study included behavioral breakages, which we do not consider due to the limitationsof our approach. Our definition of an API does differ from prior work. Dig and Johnson considered all public entities.
Our work are more similar to the work of Cossette et al., in which we include the detection of protected entities. In
this work, we go further and use client usage to focus on API breakages the more popular APIs.
The usage of tools revealed more API breakages, some of which were not reported in the API change logs,
which was also consistent with the findings of Murphy-Hill et al [31]. These undocumented API changes could also
explain the disparity in results between manual (i.e., Dig and Johnson study) and machinery refactoring detection.
For mechanical refactoring detection, since Ref-Finder is template-based refactoring reconstructing approach, wewere only able to identify 23 out of 70 of Fowler’s catalog. In fact, Cossette et al. [8] also believed that tools would
miss some behavioral refactoring, saying that they ‘...do not believe that some changes would be easily handled bymechanical transformation tools; instead the API maintainer, or the client developer would need to craft some minimal
specification that would describe how to remap classes to accommodate these breaking changes.’ Another differencein our method that may have influenced results, is where we analyze API changes between consecutive versions, while
prior work analyzed versions that were not consecutive.

21

6.3. Challenges of the Automated Approach
Key threats to the automated approach accuracy is when: (i) refactorings are missed by our approach (i.e., RefFinder [32]), (ii) developers may not report all API changes [31] and (iii) misidentification of breaking APIs is reported
but it did not cause a breakage. Figure 10 presents an example of a misidentification reported by our automatedapproach 13 . In this example, Ref-Finder detects this change as an add parameter Ropt , that is also API breaking
since it has a change in the method signature. However, according to Java documentation, the superclass extends
indicates that the change of this special ‘type parameter of the class does not, in itself, have any implications for binary
compatibility’. We believe these limitations will encourage researchers to further investigate and help us understandhow developers evolve their libraries, especially in regards to avoidable API breakages.
6.4. Threats to Validity
Internal Threats: The most significant internal threat is correctness of the automated tools, especially
Ref-Finder. To mitigate this and as a sanity check, we randomly inspected a small sample of the results for validation. Mentioned earlier in the paper, an example of a false positive was when a unchanged file was reported to have a
refactoring identified. In the end, we understand that recall is not as obvious to investigate as ground truth is unknown.
Ref-Finder is the current state–of–the–art and actively used in research.
Another minor threat to our approach is that API breakages false positives caused by the class-level granularity ofanalysis. Theoretically, an external API class that has a breakage related to a private entity could be a false positive.
However, even with this assumption in mind, our analysis may be underestimations. It is true that the accuracyof the saturation point is fairly dependent on the sample size. We believe our sample clients are sufficient to atleast identify the most popular APIs that reside in the client-used API classes. Sometimes variations between the
refactored classpath (originating from source code) and API breakages class path (originating from binary code) may
cause a miss-match. To overcome this, we manually validated the consistency of file paths to ensure consistency
and completeness. Correct ordering of consecutive library releases is another minor threat. We therefore consider
Maven [37] as the ground truth to base our chronological ordering of the released versions of a library. Some of ourconclusions are based on the statistical analysis. We believe that due to outliers and nature of the data collected, nonparametric statistical tests were deemed appropriate.
External Threats: As an external threat, we understand that our collected clients and the six selected OSS libraries
are not necessary complete representations of the real world. However, we believe that the diverse nature (such as size,
domain, team) of the six libraries is enough to assume generalization. Although our approximations of external APIs
can only be justified through documentation and developers, we believe our method provides sufficient confidence of
external client coverage. Another important threat is selection of the more popular libraries. As a results, our findings
may not be applicable for less popular libraries. In this study, we consider that both library developers and users are
more concerned with popular APIs, as they tend to reach a larger client user-base. Moreover, the same libraries that
we study have been used in prior studies by researchers. As future work, we plan to expand our study to investigate
more frameworks and libraries. Since our study is focused only on java libraries, we cannot make generalizations
to other programming languages. We are confident that our research method is scalable and can be replicated withdifferent sets of clients and subject libraries in other languages.
7. Related WorkIn this section, we introduce literature related to API usage, library migration support and library evolution.
API usage. There has been different work that have collected clients API usage. For example, work such as De
Roover et al. [38] exploit API usage to understand popularity and usage patterns of clients. The data collected isvisualized to further explore to provide program comprehension as well as identify patterns in the code. Another setof research use the API usage as a measure of stability or popularity [39, 40]. Our previous work [41], among work
leveraged popularity to recommend when libraries are deemed safe to use by the masses. Other related work thatstudied the impact of API evolution on their clients on online forums such as Stack Overflow [42] and the Android
App [43], Pharo [44] and Smalltalk ecosystems [45].
13 commit can be found at goo.gl/CwXoBj and API change at https://goo.gl/VPPTIX

22

Library Migration Support. Much work has been in transformation of the client code to support migration oflibrary API changes. Work by Chow and Notkin [46] and Balaban et al. [47] use a change specification language.
There is work that provides the client with automatic tool support to accommodate changes made the APIs of a library.
For instance, SemDiff [48] recommends replacements for framework methods that were accessed by clients. Other
similar tools were proposed by Xing and Stroulia [49] and Schafer et al. [50]. Other work on reuse support is throughcode analysis. This area of work considers code clone detection techniques [51] to support which library version ismost appropriate candidate for migration. Godfrey et al. [52] proposed origin analysis to recover context of codechanges. Our previous work [53] tracked how code is reused cross-projects. Related works [10] focused on support
for clients migrating to a newer library version. Likewise, other works [39, 54, 55, 56] studied how library maintainers
balance API compatibility with an evolving library.
Library Evolution. There is similar work with respect to library maintenance and evolution. Cossette et al.
[8] manually illustrated the complexities of library changes and transformations. Other work such as Kim et al.
[57] studied the role of refactoring during software evolution. Recently, there has been large-scale empirical studies
conducted on library migrations and evolution. Empirical studies by Raemakers et al. [27, 58], Jezek et al. [26] andJoel et al. [59] studied in-depth how libraries that reside in the Maven Central super-repository evolve and break APIs.
8. Conclusions and Future Work
Refactorings is a key maintainability practice, even for library maintainers. When evolving code, we find thatlibrary developers are less likely to break APIs. However, we find that many of these API breaking changes relate tobug fixes and new features, with only up to 37% of client-used API breakages related to refactoring operations. The
study finds that there are still challenges to improving our tools. The study also reveals challenges faced by the tools.
As future work, we envision that this study encourages more research into automated refactoring detection techniques
to advance our understanding of refactoring activities on API breakages.
9. Acknowledgments
This work is supported by JSPS KANENHI (Grant Numbers JP25220003 and JP26280021) and the “Osaka University Program for Promoting International Joint Research.”.
References[1] M. Fowler, K. Beck, J. Brant, W. Opdyke, D. Roberts, Refactoring: improving the design of existing code, Addison-Wesley Longman
Publishing Co., Inc., 1999.
[2] W. F. Opdyke, Refactoring object-oriented frameworks, Ph.D. thesis, University of Illinois at Urbana-Champaign (1992).
[3] M. M. Lehman, Laws of software evolution revisited, in: Proceedings of the 5th European Workshop on Software Process Technology,
EWSPT ’96, Springer-Verlag, London, UK, UK, 1996, pp. 108–124.
URL http://dl.acm.org/citation.cfm?id=646195.681473[4] Source file example on google guava, accessed 2014-09-01, https://code.google.com/p/guava-libraries/source/browse/
guava/src/com/google/common/base/internal/Finalizer.java?r=9de0ce12f1648ee4f190e2ce706f26088727ff89.
[5] J. Businge, A. Serebrenik, M. G. J. van den Brand, Eclipse API usage: the good and the bad, Software Quality Journal 23 (1) (2015) 107–141.
[6] I. R. Forman, M. H. Conner, S. H. Danforth, L. K. Raper, Release-to-release binary compatibility in som, in: Proceedings of the Tenth Annual
Conference on Object-oriented Programming Systems, Languages, and Applications, OOPSLA ’95, 1995, pp. 426–438.
URL http://doi.acm.org/10.1145/217838.217880[7] J. Bauml, P. Brada, Automated versioning in osgi: A mechanism for component software consistency guarantee, in: Software Engineeringand Advanced Applications, 2009. SEAA ’09. 35th Euromicro Conference on, 2009, pp. 428–435.
[8] B. E. Cossette, R. J. Walker, Seeking the ground truth, Proc. of the ACM SIGSOFT Intrn. Symp. on the Foundations of Software Engineering- FSE ’12.
[9] D. Dig, R. Johnson, How do APIs evolve? A story of refactoring, Journal of Software Maintenance and Evolution 18 (2006) 83–107.
[10] P. Kapur, B. Cossette, R. J. Walker, Refactoring references for library migration, ACM SIGPLAN Notices 45 (10) (2010) 726.
[11] guava-libraries, accessed 2017-04-28, https://code.google.com/p/guava-libraries/.
[12] The hyper-text transfer protocol client, accessed 2017-04-28, https://hc.apache.org/httpcomponents-client-ga/.
[13] The java programming assistant, accessed 2017-04-28, http://jboss-javassist.github.io/javassist/.
[14] Jdom website, accessed 2017-04-28, http://www.jdom.org/.
[15] Joda-time for java, accessed 2017-04-28, http://www.joda.org/joda-time/.
[16] Apache log4j, accessed 2017-04-28, http://logging.apache.org/log4j/1.2/.

23

[17] Simple logging facade for java (slf4j), accessed 2017-04-28, http://www.slf4j.org/.
[18] The apache xerces project, accessed 2017-04-28, https://xerces.apache.org/.
[19] Aether adapter for maven plugins, accessed 2017-04-28, http://aether.jcabi.com/index.html.
[20] Javacompiler tool,accessed 2017-04-28,
http://docs.oracle.com/javase/7/docs/api/javax/tools/JavaCompiler.
CompilationTask.html.
[21] The japicmp tool, accessed 2016-04-08, https://siom79.github.io/japicmp/.
[22] The clirr tool, accessed 2016-04-08, http://clirr.sourceforge.net/.
[23] The java api compliance checker, accessed 2016-04-08, http://ispras.linuxbase.org/index.php/Java_API_Compliance_
Checker.
[24] Jdiff - an html report of api differences, accessed 2016-04-08, http://javadiff.sourceforge.net/.
[25] Revapi - fully featured api checker for java and beyond, accessed 2016-04-08, http://revapi.org/.
[26] K. Jezek, J. Dietrich, P. Brada, How Java APIs break - An empirical study, Information and Software Technology (2015) 129–146doi:
10.1016/j.infsof.2015.02.014.
[27] S. Raemaekers, A. van Deursen, J. Visser, Semantic versioning versus breaking changes: A study of the maven repository (2014) 215–224.
[28] japicmp tool and library, accessed 2017-04-28, https://github.com/siom79/japicmp.
[29] K. Prete, N. Rachatasumrit, N. Sudan, M. Kim, Template-based reconstruction of complex refactorings, in: Software Maintenance (ICSM),
2010 IEEE International Conference on, IEEE, 2010, pp. 1–10.
[30] J. Cohen (Ed.), Front Matter, revised edition Edition, Academic Press, 1977.
[31] E. Murphy-Hill, C. Parnin, A. P. Black, How we refactor, and how we know it, in: 2009 IEEE 31st International Conference on Software
Engineering, IEEE, 2009, pp. 287–297. doi:10.1109/ICSE.2009.5070529.
URL http://ieeexplore.ieee.org/document/5070529/[32] G. Soares, R. Gheyi, E. Murphy-Hill, B. Johnson, Comparing approaches to analyze refactoring activity on software repositories, Journal of
Systems and Software 86 (4) (2013) 1006–1022. doi:10.1016/j.jss.2012.10.040.
URL http://linkinghub.elsevier.com/retrieve/pii/S016412121200297X[33] J. Bloch, How to design a good api and why it matters, in: Companion to the 21st ACM SIGPLAN Symposium on Object-oriented Programming Systems, Languages, and Applications, OOPSLA ’06, ACM, New York, NY, USA, 2006, pp. 506–507.
[34] H. Seo, C. Sadowski, S. Elbaum, E. Aftandilian, R. Bowdidge, Programmers’ build errors: A case study (at google), in: Proceedings ofthe 36th International Conference on Software Engineering, ICSE 2014, ACM, New York, NY, USA, 2014, pp. 724–734. doi:10.1145/
2568225.2568255.
URL http://doi.acm.org/10.1145/2568225.2568255[35] R. G. Kula, D. M. German, T. Ishio, K. Inoue, Trusting a library: A study of the latency to adopt the latest maven release, in: 22nd IEEE
International Conference on Software Analysis, Evolution, and Reengineering, SANER 2015, Montreal, Canada, March 2-6, 2015, 2015.
URL http://sel.ist.osaka-u.ac.jp/~raula-k/publications/latencyAdoptions2015ERA.pdf[36] C. Larman, Protected variation: the importance of being closed, Software, IEEE 18 (3) (2001) 89–91.
[37] The maven central repository, accessed 2014-09-01, http://search.maven.org/.
[38] C. De Roover, R. Lammel, E. Pek, Multi-dimensional exploration of API usage, IEEE International Conference on Program Comprehension(2013) 152–161.
[39] T. McDonnell, B. Ray, M. Kim, An empirical study of API stability and adoption in the android ecosystem, IEEE International Conference
on Software Maintenance, ICSM (2013) 70–79doi:10.1109/ICSM.2013.18.
[40] Y. M. Mileva, V. Dallmeier, M. Burger, A. Zeller, Mining trends of library usage, in: Proc. Intl and ERCIM Principles of Soft. Evol. (IWPSE)and Soft. Evol. (Evol) Workshops, IWPSE-Evol ’09, ACM, New York, NY, USA, 2009, pp. 57–62.
[41] R. G. Kula, C. D. Roover, D. M. German, T. Ishio, K. Inoue, Visualizing the evolution of systems and their library dependencies, Proc. ofIEEE Work. Conf. on Soft. Viz. (VISSOFT).
[42] M. L. Vásquez, G. Bavota, M. D. Penta, R. Oliveto, D. Poshyvanyk, How do API changes trigger stack overflow discussions? a study on the
android SDK, in: ICPC, ACM, 2014, pp. 83–94.
[43] G. Bavota, M. L. Vásquez, C. E. Bernal-Cárdenas, M. D. Penta, R. Oliveto, D. Poshyvanyk, The impact of API change- and fault-pronenesson the user ratings of android apps, IEEE Trans. Software Eng. 41 (4) (2015) 384–407.
[44] A. C. Hora, R. Robbes, N. Anquetil, A. Etien, S. Ducasse, M. T. Valente, How do developers react to API evolution? the pharo ecosystemcase, in: ICSME, IEEE Computer Society, 2015, pp. 251–260.
[45] R. Robbes, M. Lungu, D. Röthlisberger, How do developers react to API deprecation?: the case of a smalltalk ecosystem, in: SIGSOFT FSE,
ACM, 2012, p. 56.
[46] K. Chow, D. Notkin, Semi-automatic update of applications in response to library changes, in: Proceedings of the 1996 International Conference on Software Maintenance, ICSM ’96, IEEE Computer Society, Washington, DC, USA, 1996.
[47] I. Balaban, F. Tip, R. Fuhrer, Refactoring support for class library migration (2005) 265–279doi:10.1145/1094811.1094832.
URL http://doi.acm.org/10.1145/1094811.1094832[48] B. Dagenais, M. P. Robillard, Semdiff: Analysis and recommendation support for api evolution, in: Proceedings of the 31st International
Conference on Software Engineering, ICSE ’09, IEEE Computer Society, Washington, DC, USA, 2009, pp. 599–602.
[49] Z. Xing, E. Stroulia, API-evolution support with diff-catchup 33 (2007) 818–836. doi:10.1109/TSE.2007.70747.
[50] T. Schäfer, J. Jonas, M. Mezini, Mining framework usage changes from instantiation code, in: Proceedings of the 30th International Conference on Software Engineering, ICSE ’08, ACM, New York, NY, USA, 2008, pp. 471–480. doi:10.1145/1368088.1368153.
URL http://doi.acm.org/10.1145/1368088.1368153[51] T. Kamiya, S. Kusumoto, K. Inoue, CCFinder: a multilinguistic token-based code clone detection system for large scale source code, IEEE
Transactions on Software Engineering 28 (7) (2002) 654–670. doi:10.1109/TSE.2002.1019480.
[52] M. Godfrey, L. Zou, Using origin analysis to detect merging and splitting of source code entities, IEEE Transactions on Software Engineering
31 (2) (2005) 166–181.

24

URL http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1401931[53] N. Kawamitsu, T. Ishio, T. Kanda, R. G. Kula, C. D. Roover, K. Inoue, Identifying source code reuse across repositories using lcs-basedsource code similarity, in: Proc. of SCAM, 2014.
[54] G. Malpohl, J. J. Hunt, W. F. Tichy, Renaming detection, in: Proceedings of the 15th IEEE International Conference on Automated Software
Engineering, ASE ’00, IEEE Computer Society, Washington, DC, USA, 2000, pp. 73–.
URL http://dl.acm.org/citation.cfm?id=786768.786983[55] M. Mezini, Maintaining the consistency of class libraries during their evolution, SIGPLAN Not. 32 (10) (1997) 1–21. doi:10.1145/
263700.263701.
URL http://doi.acm.org/10.1145/263700.263701[56] P. Steyaert, C. Lucas, K. Mens, T. D’Hondt, Reuse contracts: Managing the evolution of reusable assets, SIGPLAN Not. 31 (10) (1996)
268–285. doi:10.1145/236338.236363.
URL http://doi.acm.org/10.1145/236338.236363[57] M. Kim, D. Cai, S. Kim, An empirical investigation into the role of API-level refactorings during software evolution, 2011 33rd International
Conference on Software Engineering (ICSE) (2011) 151–160.
[58] S. Raemaekers, A. van Deursen, J. Visser, Measuring software library stability through historical version analysis, in: Proc. of Intl. Comf.
Soft. Main. (ICSM), 2012, pp. 378–387.
[59] M. v. E. Joel Cox, Eric Bouwers, J. Visser, Measuring dependency freshness in software systems 2 (2015) 109–118.

25