# Roseau: Fast, Accurate, Source-based API Breaking Change Analysis in Java

Corentin Latappy, Thomas Degueule, Jean-Rémy Falleri, Romain Robbes, Lina Ochoa

► To cite this version:

Corentin Latappy, Thomas Degueule, Jean-Rémy Falleri, Romain Robbes, Lina Ochoa. Roseau: Fast, Accurate, Source-based API Breaking Change Analysis in Java. 41st IEEE International Conference on Software Maintenance and Evolution (ICSME 2025), Sep 2025, Auckland, New Zealand. pp.517-528, 10.1109/ICSME64153.2025.00053. hal-05176866v2

HAL Id: hal-05176866

https://hal.science/hal-05176866v2

Submitted on 23 Jul 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# ROSEAU: Fast, Accurate, Source-based
API Breaking Change Analysis in Java

Corentin Latappy,\* Thomas Degueule,\* Jean-Rémy Falleri,\*‡ Romain Robbes\* and Lina Ochoa†

\* Univ. Bordeaux, CNRS, Bordeaux INP, LaBRI, UMR 5800, Talence, France

Email: firstname.lastname@labri.fr

† Eindhoven University of Technology, Eindhoven, The Netherlands

Email: l.m.ochoa.venegas@tue.nl

‡ Institut Universitaire de France

**Abstract**—Understanding API evolution and the introduction of breaking changes (BCs) in software libraries is essential for library maintainers to manage backward compatibility and for researchers to conduct empirical studies on software library evolution. In Java, tools such as JAPICMP and REVAPI are commonly used to detect BCs between library releases, but their reliance on binary JARs limits their applicability. This restriction hinders large-scale longitudinal studies of API evolution and fine-grained analyses such as commit-level BC detection.

In this paper, we introduce ROSEAU, a novel static analysis tool that constructs technology-agnostic API models from library code equipped with rich semantic analyses. API models can be analyzed to study API evolution and compared to identify BCs between any two versions of a library (releases, commits, branches, etc.). Unlike traditional approaches, ROSEAU can build API models from source code or bytecode, and is optimized for large-scale longitudinal analyses of library histories.

We assess the accuracy, performance, and suitability of ROSEAU for longitudinal studies of API evolution, using JAPICMP and REVAPI as baselines. We extend and refine an established benchmark of BCs and show that ROSEAU achieves higher accuracy ($F_1$ = 0.99) than JAPICMP ($F_1$ = 0.86) and REVAPI ($F_1$ = 0.91). We analyze 60 popular libraries from Maven Central and find that ROSEAU delivers excellent performance, detecting BCs between versions in under two seconds, including in libraries with hundreds of thousands of lines of code. We further illustrate the limitations of JAPICMP and REVAPI for longitudinal studies and the novel analysis capabilities offered by ROSEAU by tracking the evolution of Google's Guava API and the introduction of BCs over 14 years and 6,839 commits, reducing analysis times from a few days to a few minutes.

**Index Terms**—software library evolution, API, breaking change

## I. INTRODUCTION

Like any other software, software libraries continuously evolve to introduce new features, fix bugs, enhance security and performance, and implement refactorings. Library users must keep their dependencies up to date to take advantage of these improvements and prevent technical debt build-up [1]. However, when a library evolves, it may introduce breaking changes (BCs) that disrupt backward compatibility with existing clients. Syntactic BCs, in particular, affect the Application Programming Interface (API) of libraries, manifesting as compilation and linking errors in client code. Consequently, clients may delay upgrading their dependencies, raising security concerns and complicating future updates [2, 3].

Automatic BC detection tools have been proposed to help library developers identify and document these changes [4]. In particular, tools such as JAPICMP and REVAPI are actively used in the development of popular Java libraries and frameworks such as Apache Commons [5], Spring Framework [6], Google Gson [7], and Neo4j [8]. These tools also enable research contributions and empirical studies of software evolution. Notably, they have been used to identify BCs in large software repositories [9, 10], analyze breaking dependency updates [11], construct safe migration plans for outdated dependencies [12, 13], analyze software versioning schemes [14], and assist code reviews in pull requests [15].

However, existing BC detection tools can only consume binary JAR archives and cannot directly analyze the source code of software libraries. Using them requires compiling the library versions of interest, which significantly hinders their use in several key scenarios. For practitioners, this implies that backward compatibility checks must be delayed until after a binary is built, which can be unnecessarily costly in continuous integration. This also prevents their use for immediate feedback in IDEs and for quickly identifying commits or pull requests that introduce backward-incompatible changes [15]. For researchers, empirical evidence suggests that nearly half of Java software projects hosted on GitHub cannot be built automatically, restricting the libraries they can analyze with these tools [16, 17, 10]. Even when projects can be successfully built, compilation is costly and impractical for large-scale longitudinal studies of API evolution. Consequently, researchers have primarily focused on analyzing library releases readily available in package managers, overlooking research questions at the finer granularity of individual changes and commits.

In this paper, we present a new approach to API analysis and syntactic BC detection that reifies API models as first-class entities. API models are designed to be self-contained and technology-independent, allowing extraction from any suitable source, such as source code, bytecode, or even IDE-specific internal code representations [18]. BCs between two library versions can be inferred by comparing their respective API models using straightforward detection rules. We implement our approach in a new tool, ROSEAU, which emphasizes accuracy and performance to enable large-scale and fine-grained
longitudinal studies of API evolution. We evaluate the accuracy, performance, and suitability of ROSEAU for longitudinal studies using JAPICMP and REVAPI as baselines.

Specifically, this paper makes the following contributions:

* We present a novel approach for inferring accurate API models from source code or bytecode, and for detecting BCs by comparing API models;

* We present an implementation of this approach in ROSEAU, a new BC detection tool optimized for accuracy and performance;

* We extend and refine an established benchmark of BCs proposed by Jezek and Dietrich [4] with new metrics, stronger oracles, and a streamlined evaluation pipeline;

* We evaluate ROSEAU, JAPICMP, and REVAPI on the extended benchmark, and find that ROSEAU achieves higher precision and recall ($F_1 = 0.99$) than JAPICMP ($F_1 = 0.86$) and REVAPI ($F_1 = 0.91$);

* We evaluate ROSEAU's runtime performance and find that it matches JAPICMP and surpasses REVAPI when analyzing binary JARs. When analyzing source code and commit histories, ROSEAU outperforms both by two orders of magnitude, as it does not require compiling source code;

* We showcase ROSEAU's suitability for large-scale longitudinal studies of API evolution by analyzing 14 years and 6,839 commits of the popular Guava library, reducing analysis times from a few days to a few minutes.

The remainder of this paper is organized as follows. We review related work and identify the limitations of current BC detection tools in Section II. We present API models and BC detection rules in Section III, and their optimized implementation in ROSEAU in Section IV. We evaluate ROSEAU's accuracy, performance, and suitability for large-scale longitudinal studies in Section V, using JAPICMP and REVAPI as baselines. Finally, we conclude the paper in Section VI.

## II. RELATED WORK AND MOTIVATION

Modern software system development typically involves the use of third-party software libraries. Application Programming Interfaces (APIs) govern the interactions between a library and its clients, constituting the primary friction point during evolution. Library developers must carefully design and evolve their APIs so that clients can benefit from new features, bug fixes, or performance improvements without having to constantly rewrite their code. In particular, introducing BCs in an API burdens client developers, who may hesitate to upgrade their dependencies, raising security concerns as they might miss security patches, and making future upgrades even more difficult as client updates accumulate.

In this section, we review some background notions on backward compatibility and BCs and discuss the importance of BCs in both software engineering practice and research.

### A. Backward Compatibility and Breaking Changes

Whenever a library evolves, it may break the contract previously established with its clients by introducing BCs. BCs are language-specific and can either be syntactic or semantic in

<figure><img src="image_2.png" /><figcaption>Fig. 1: An example BC propagating to clients.</figcaption></figure>

nature [10]. Syntactic changes affect the library's API, i.e., the signatures of exported symbols accessible from client code (types, methods, etc.). Semantic (or behavioral) changes affect the run-time behavior of a library and can alter client functionalities or trigger run-time errors [19]. Changes that affect neither the API nor the behavior of software libraries are backward-compatible.

In Java, syntactic BCs are further categorized into source-incompatible and binary-incompatible changes. Source-incompatible changes occur when recompiling an existing client against a new version of a library results in compilation errors. Binary-incompatible changes occur when linking a pre-existing client binary with a new library binary results in class loading or linking errors [20, §13.2]. Some BCs are only source-incompatible (e.g., a method throwing a new type of checked exception), some others are only binary-incompatible (e.g., changing the return type of a method covariantly), and some are both source- and binary-incompatible (e.g., removing a method) [21].

Identifying changes that affect APIs is not always straightforward. While some BCs are overt (such as removing a public type from an API), others are more subtle. Figure 1 depicts an example of the latter, highlighting the importance of accurately delimiting the scope of an API. In this example, the abstract class B extends an internal class A. An external client extends B via the concrete class C. The library maintainers add an abstract method to A that leaks into C, raising a missing implementation error, although there is no direct use of the modified internal class A. In this case, maintainers may implement this change without realizing its potential impact on clients. Conversely, maintainers may hesitate to make changes even when they would not break backward compatibility. To alleviate these problems, several tools have been proposed to assist maintainers in making informed decisions when evolving their APIs.
## B. Breaking Changes in Software Engineering Practice

Although most ecosystems and library maintainers aim for stability and compatibility throughout evolution, their policies regarding BCs management can vary [22]. For instance, maintainers of the popular Apache Commons libraries prohibit breaking binary compatibility in patch and minor releases. In case of binary-incompatible major releases, they recommend changing the package name and Maven coordinates of the library [23]. Google's Guava attempts to maintain backward compatibility across releases, except for APIs explicitly marked as unstable [24]. Some libraries also use various methods to mark specific APIs as unstable, such as using `@Beta` and `@Experimental` annotations or placing these APIs in `*.internal.*` packages.

To implement these policies, different ecosystems offer (breaking) change detection tools, for instance, in Rust (cargo-semver-checks), Go (gocompat and gorelease), and Python (AexPy [25]). The Java ecosystem, in particular, boasts a rich set of BC detection tools, inventoried and evaluated by Jezek and Dietrich [4]. Although many have been abandoned over time, JAPICMP<fnref n="1" /> and REVAPI<fnref n="2" /> are still actively maintained. JAPICMP and REVAPI take as input two versions of a library as binary JARs and diff them to identify both source-incompatible and binary-incompatible changes. They produce reports in various formats and can be integrated into continuous integration through dedicated build manager plug-ins. These tools are used in the development of numerous libraries such as Apache's commons-lang [5], Spring's spring-framework [6], and Google'sAPOIson [7]. Other libraries, such as Apache's Hadoop and Google's Guava use simpler API differencing tools, such as JDiff, to list API differences without distinguishing backward-compatible changes and BCs [26].

The aforementioned tools are well-suited for documenting BCs between releases. However, their reliance on binary JARs introduces several limitations. They incur a substantial cost in continuous integration, as they require compiling the library and delaying BC detection until after binary construction. These steps could be avoided if backward compatibility could be verified at the source code level. Furthermore, they typically use a published version of the library as a reference, which is compared against the current development version. Therefore, branch comparisons or pull request reviews become arduous, as the reference development version is usually unreleased and thus not easily referenced. This highlights the need for an efficient source-level approach to detecting BCs.

## C. Breaking Changes in Software Engineering Research

API evolution is a thriving research area that has garnered considerable interest in the literature, particularly through empirical studies on software ecosystems [27]. BCs are central to API evolution, and several studies have investigated their prevalence, causes, and impacts, particularly in Java ecosystems (e.g., [28, 29, 9, 10, 11]), with more recent research extending

to `npm` [30] and the Golang ecosystem [31]. Other studies have focused on facilitating client updates while prioritizing security, minimizing technical lag, and preserving backward compatibility [12, 32]. In the same line, Jaime et al. model a client update as a multi-objective optimization problem accounting for potential BCs [13].

Overall, the above-mentioned studies identify BCs, often relying on JAPICMP, and analyze their introduction and impact at the granularity of library releases. Brito et al., however, study API evolution at the finer granularity of individual commits using the APIDiff tool [33]. APIDiff can detect common refactorings in APIs (such as method pull-up) and certain BCs, but the list of supported BCs is limited compared to state-of-the-art tools such as JAPICMP or REVAPI [34]. Ochoa et al. identified the need to analyze the introduction of BCs in individual pull requests instead of waiting for releases [15]. However, their tool, Maracas, still relies on JAPICMP and comes at the cost and sometimes impossibility of compiling the two branches.

The reliance of current tools on binary JARs is unsurprising, as Java bytecode is significantly more stable and less sensitive to changes in the Java language than source code, while also being straightforward and efficient to parse. However, JARs are not always readily available, which severely hinders empirical research on the evolution of Java libraries. Various authors report that only around half of Java projects on GitHub can be built automatically [16, 17, 10], which prevents the analysis of individual commits in the wild. Even when the project successfully compiles, the compilation phase is too costly and impractical for large-scale studies. Thus, researchers have focused on analyzing releases of popular libraries that are already readily available, eluding research questions that would require analyzing individual commits, branches, or pull requests. This highlights the need for an approach that enables researchers to analyze source code directly and scales BC analysis on large datasets of commit histories.

# III. DETECTING BREAKING CHANGES WITH API MODELS

Syntactic BCs are changes that affect the APIs of software libraries—contracts established between these libraries and their clients. While modifications to internal implementations can introduce semantic BCs, they do not affect APIs. Similarly, changes in APIs explicitly documented as internal or beta-access are outside the scope of BC detection as they are usually exempt from compatibility guarantees. Therefore, an essential first step is to precisely delimit the scope of a library's API. Identifying BCs between two versions then involves comparing the two APIs, cataloging their differences, and determining whether these differences compromise backward compatibility at the source or binary levels.

In our approach, we reify API models as self-contained first-class entities that can be reasoned about to track API evolution (Section III-A). API models can then be compared to infer a list of BCs using straightforward detection rules expressed directly at the API level (Section III-B). Throughout

<footnote n="1">¹https://siom79.github.io/japicmp/</footnote>

<footnote n="2">²https://revapi.org/revapi-site/main/</footnote>
this section, we reference the specification of the latest long-term support version of Java (JLS 21 [20]) when appropriate.

## A. API Models

Similar to prior work on syntactic usage models [35], we consider that a software library declares a set of symbols. Each symbol has a unique identifier within the library and an associated declaration that specifies its properties: kind, visibility, modifiers, location, etc. The JLS 21 defines the following symbol kinds relevant to API definitions: type declarations (classes, interfaces, enumerations, records, and annotation interfaces), executables (constructors and methods), and fields [20, §6.1]. For instance, Java's standard library declares the symbol `public class String` of kind `class` and visibility `public`, located in the file `String.java`. It also declares a field `private int hash` of type `int` and visibility `private`, and a method `public char charAt(int index)` of visibility `public` that takes an `int` as an argument and returns a `char` value.

Type declarations and fields are uniquely identified by their fully qualified name such as `java.lang.String` and `java.lang.String.hash`. Methods are uniquely identified by the fully qualified erasure of their signatures such as `java.lang.String charAt(int) [20, §8.4.2].

Symbols can be *exported*, allowing client code outside the library to access them. Given a function *exported*: $S \to \{\top, \bot\}$ that indicates whether a given symbol $s \in S$ is exported, the API of a library is the set $\mathcal{A} \subseteq S$ such that $\mathcal{A} = \{s \in S: \text{exported}(s) = \top\}$. In Java, a symbol is considered exported (or accessible) if every symbol in the visibility chain—from the compilation unit to the symbol itself—is exported [20, §6.6.1].<fnref n="3" /> More specifically, the following rules apply:

* **Public symbols:** (transitively-)`public` type declarations, as well as `public` methods and fields within `public` type declarations, can be accessed from anywhere without restriction;

* **Protected symbols:** `protected` fields, methods, and type declarations within effectively extensible `public` type declarations can be accessed through subtyping or by code located in the same package. An effectively extensible type declaration is one that is neither `final` nor `sealed` and that, in the case of classes, possesses a `public` or `protected` constructor that subclasses can access;

* **Package-private symbols:** package-private symbols (Java's default visibility in the absence of a visibility modifier) can be accessed by code located in the same package.

Note that the accessibility of an inherited member is assessed in the context of the inheriting type, not the original declaring type. This is, for instance, the case in Figure 1 where the method `n()` is “re-exported” to clients through the subclass B.

The only legal visibilities for top-level type declarations are `public` and package-private, so library developers must use the latter to prevent clients from accessing them. In our model,

we consider all `public` and `protected` symbols as exported. Package-private symbols are not considered exported because client code intentionally using a package with the same name as the library's to access its package-private symbols is a pathological case that most likely breaches the API's intent.

Symbols are organized in a hierarchy with kind-specific properties. Each symbol is associated with its unique identifier, visibility, modifiers, and annotations. Different symbol kinds accept different legal modifiers. Classes can, for example, be declared `abstract` or `sealed`, while methods can be declared `default` or `native`. Type declarations hold fields and methods and can contain other type declarations (*nested* or *inner*). They can also declare a list of type variables (aka. formal type parameters) and implemented interfaces. In addition, classes can declare a unique superclass and a set of constructors. Executables hold a list of parameters, a return type, a list of type variables, and potentially thrown exception types. Fields have a specific type. References to type declarations (e.g., a thrown exception, a super-class, a return type) are modeled as type references that associate names to declarations.

An API model is thus a graph of API symbols with a spanning containment tree representing the natural containment relationships between type declarations, fields, and executables, and cross-references linking type references to type declarations. Note that the information contained in an API model is purely syntactic and can be trivially inferred from source code (e.g., by visiting an abstract syntax tree) or bytecode (e.g., by parsing class files). However, raw information extracted directly from source code or bytecode is insufficient for accurate analysis. Some properties of symbols are not always explicitly set in source code or bytecode (e.g., implicit constructors) while others must be derived (e.g., all supertypes of a type declaration). For instance, interfaces nested within other types are implicitly static, even if the `static` modifier is not explicitly present. A class is considered *effectively final* (respectively *effectively abstract*) if (i) it is explicitly declared as `final` or `sealed` (respectively *abstract*), or (ii) none of its constructors are accessible to subclasses. Inferring the precise methods or fields that can be invoked or accessed on a type declaration involves resolving all supertypes in its hierarchy while accounting for Java's overriding, shadowing, and hiding semantics [20, §8.4.8].

API models automatically infer this missing and derived information that is not readily available in source code or bytecode to ensure accurate analysis, accounting for differences between source code and bytecode, as compilers typically infer certain implicit properties during compilation. The underlying tool responsible for visiting the abstract syntax trees or class files can thus focus solely on syntax and only extract the information explicitly set in the source.

## B. Detecting Breaking Changes

Detecting BCs between two library versions involves comparing their APIs $\mathcal{A}$ and $\mathcal{A}'$ to identify changes that disrupt backward compatibility, either at the source or binary levels.

Our approach revolves around matching symbols pairwise across compared versions to analyze their differences. We use

<footnote n="3">³Recent versions of Java offer a *module* construct that enables finer control over exports but is beyond the scope of this paper.</footnote>
<table><caption>TABLE I: BC detection rules implemented in ROSEAU. $s \sim s'$ denotes that symbols $s$ and $s'$ have matching fully qualified names or erasures. $\checkmark/\xmark$ denotes whether a change is source- or binary-breaking ($\xmark$) and source- or binary-compatible ($\checkmark$). Executable-related BCs apply to both methods and constructors. Annotation-related BCs are omitted for the sake of conciseness.</caption><thead><tr><th>Kind</th><th>Detection Rule</th><th>Binary</th><th>Source</th></tr></thead><tbody><tr><td>Type removed</td><td>T &isin; A &and; &not;T' &isin; A' | T &sim; T'</td><td>x</td><td>x</td></tr><tr><td>Type now protected</td><td>T &sim; T' &and; public(T) &and; protected(T')</td><td>x</td><td>x</td></tr><tr><td>Type kind changed</td><td>T &sim; T' &and; typeKind(T) &ne; typeKind(T')</td><td>x</td><td>x</td></tr><tr><td>Supertype removed</td><td>T &sim; T' &and; &exist; S &isin; superTypes(T) | exported(S) &and; T' &not;subseteq; S</td><td>x</td><td>x</td></tr><tr><td>Type variable removed</td><td>T &sim; T' &and; |typeVars(T')| &gt; |typeVars(T')|</td><td>✓</td><td>x</td></tr><tr><td>Type variable added</td><td>T &sim; T' &and; |typeVars(T')| &gt; |typeVars(T')| &gt; 0</td><td>✓</td><td>x</td></tr><tr><td>Type variable changed</td><td>T &sim; T' &and; &exist; i &isin; [1, |typeVars(T')|], &exist; b' &isin; bounds(typeVar<sub>s</sub>(T')), &forall; b &isin; bounds(typeVar<sub>s</sub>(T)) | b &not;subseteq; b'</td><td>✓</td><td>x</td></tr><tr><td>Type now final</td><td>T &sim; T' &and; -effectivelyFinal(T) &and; effectivelyFinal(T')</td><td>x</td><td>x</td></tr><tr><td>Type now abstract</td><td>T &sim; T' &and; -effectivelyFinal(T) &and; effectivelyFinal(T')</td><td>x</td><td>x</td></tr><tr><td>Nested type now static</td><td>T &sim; T' &and; -static(T) &and; static(T')</td><td>x</td><td>x</td></tr><tr><td>Nested type no longer static</td><td>T &sim; T' &and; static(T) &and; -static(T')</td><td>x</td><td>x</td></tr><tr><td>Class now checked exception</td><td>T &sim; T' &and; uncheckedException(T) &and; checkedException(T')</td><td>✓</td><td>x</td></tr><tr><td>Abstract method added to type</td><td>T &sim; T' &and; &exist;m' &isin; allMethods(T') | abstract(m') &and; &not;m' &isin; allMethods(T) | m &sim; m'</td><td>✓</td><td>x</td></tr><tr><td>Executable removed</td><td>e &isin; executables(T) &and; &not;e' &isin; executables(T') | e &sim; e'</td><td>x</td><td>x</td></tr><tr><td>Executable now protected</td><td>e &sim; e' &and; public(e) &and; protected(e')</td><td>x</td><td>x</td></tr><tr><td>Executable checked exception removed</td><td>e &sim; e' &and; &exist;t &isin; thrown(e) | &not;t' &isin; thrown(e'), t' &not;subseteq; t</td><td>✓</td><td>x</td></tr><tr><td>Executable checked exception added</td><td>e &sim; e' &and; &exist;t &isin; thrown(e') | &not;t &isin; thrown(e'), t' &not;subseteq; t</td><td>✓</td><td>x</td></tr><tr><td>Executable parameter generics changed</td><td>e &sim; e' &and; &exist;i &isin; [1, |params(e)|] | |typeArgs(param<sub>i</sub>(e))| &ne; |typeArgs(param<sub>i</sub>(e'))|</td><td>✓</td><td>x</td></tr><tr><td>Executable type variable removed</td><td>e &sim; e' &and; |typeVars(e)| &gt; |typeVars(e')| &and; (isMethod(e) &or; |typeVars(e)| &gt; 1)</td><td>✓</td><td>x</td></tr><tr><td>Executable type variable added</td><td>e &sim; e' &and; |typeVars(e')| &gt; |typeVars(e)| &gt; 0</td><td>✓</td><td>x</td></tr><tr><td>Executable type variable changed</td><td>e &sim; e' &and; &exist;i &isin; [1, |typeVars(e)|], &exist;b' &isin; bounds(typeVar<sub>i</sub>(e')), &forall; b &isin; bounds(typeVar<sub>i</sub>(e)) | b &not;subseteq; b'</td><td>✓</td><td>x</td></tr><tr><td>Method now final</td><td>m &sim; m' &and; -effectivelyFinal(m) &and; effectivelyFinal(m')</td><td>x</td><td>x</td></tr><tr><td>Method now static</td><td>m &sim; m' &and; -static(m) &and; static(m')</td><td>x</td><td>✓</td></tr><tr><td>Method no longer static</td><td>m &sim; m' &and; -static(m) &and; -static(m')</td><td>x</td><td>x</td></tr><tr><td>Method now abstract</td><td>m &sim; m' &and; -abstract(m) &and; abstract(m')</td><td>x</td><td>x</td></tr><tr><td>Method return type changed</td><td>m &sim; m' &and; returnType(m) &ne; returnType(m')</td><td>x</td><td>x</td></tr><tr><td>Field removed</td><td>f &isin; fields(T) &and; &not;f' &isin; fields(T') | f &sim; f'</td><td>x</td><td>x</td></tr><tr><td>Field now protected</td><td>f &sim; f' &and; public(f) &and; protected(f')</td><td>x</td><td>x</td></tr><tr><td>Field now final</td><td>f &sim; f' &and; -final(f) &and; final(f')</td><td>x</td><td>x</td></tr><tr><td>Field now static</td><td>f &sim; f' &and; -static(f) &and; static(f')</td><td>x</td><td>✓</td></tr><tr><td>Field now longer static</td><td>f &sim; f' &and; static(f) &and; -static(f')</td><td>x</td><td>x</td></tr><tr><td>Field type changed</td><td>f &sim; f' &and; type(f) &ne; type(f')</td><td>x</td><td>x</td></tr></tbody></table>

the notation $s \sim s'$ to indicate that $s \in A$ and $s' \in A'$ share the same unique identifier in $A$ and $A'$. If there is no symbol $s' \in A'$ matching $s \in A$ then $s$ has been removed from the API, which is always a BC. If there is no symbol $s \in A$ matching $s' \in A'$ then $s'$ is a new symbol. Inserting new symbols into an API is backward-compatible unless the new symbol is an abstract method, which requires clients to implement it. Note that renaming a symbol, in terms of backward compatibility, constitutes a BC and is represented as both a removal (of the previous symbol) and an addition (of the new symbol). Finally, when two symbols of the same kind match, their properties are checked for BCs using straightforward detection rules. These detection rules are derived from the JLS 21 [20], and we evaluate their accuracy in Section V. Naturally, they are largely aligned with those found in JAPICMP and REVAPI, with the adjustments presented hereafter. Table I lists the BCs that can be detected atop API models, their associated detection rules, and whether they break binary or source compatibility.

A key difference between our approach and state-of-the-art tools such as JAPICMP and REVAPI is that BCs are expressed in terms of the API, not in terms of the underlying source. For instance, when the visibility of a field is decreased, JAPICMP identifies the BC field less accessible and REVAPI the BC field visibility reduced. In contrast, because API models can only represent public and protected fields, the corresponding BC in our approach is called field now protected. When a field goes from protected to private, it is instead reported as a field removed since the field is no longer part of the API, which

better reflects how the API has evolved. While both JAPICMP and REVAPI offer some support for detecting certain non-BCs (such as introducing a new default method in an interface), we focus strictly on BCs.

Although they are not explicitly referenced in detection rules, API-related constructs recently introduced in Java, such as record and sealed, are fully supported. A record class is an implicitly final class that holds final fields known as record components [20, §8.10]. A sealed class or interface specifies all its direct subtypes when it is declared and cannot be extended beyond the closed world of the library. These constructs are governed by the same rules as regular classes and interfaces with a few additional constraints, are affected by the same kinds of BCs, and require no further treatment. We further evaluate the accuracy of our detection rules and their implementation in Section V.

## IV. ROSEAU: FAST AND ACCURATE BC DETECTION

We have implemented our approach in a new static analysis tool, ROSEAU. To address the scenarios discussed in this paper, ROSEAU is designed with three primary objectives: high accuracy, excellent performance, and the ability to handle both bytecode and source code.

In a nutshell, ROSEAU consists of three main components. API extractors are responsible for turning library code, either in source code or bytecode form, into API models. New extractors can be easily contributed by leveraging existing parsing technologies to build API models that conform to the
rules expressed in Section III. A diff component is responsible for comparing two API models and inferring a list of syntactic BCs following the rules listed in Table I. Finally, a report component formats the list of BCs into a machine-readable or user-readable report in JSON, CSV, or HTML.

## A. Backends

ROSEAU currently supports inferring API models from either source code or bytecode using two independent backends. Each backend is responsible for parsing and visiting the source to extract the syntactic information necessary to build API models:

* **JDT**: The JDT parser is part of Eclipse's *Java Development Tools* and is a popular solution for efficiently parsing Java source code.<fnref n="4" /> As it offers excellent performance, the JDT backend is best suited for large-scale analyses;

* **ASM**: OW2's ASM is a bytecode analysis and manipulation framework focused on performance.<fnref n="5" /> It is a widely-used framework, including as part of the OpenJDK and the Groovy and Kotlin compilers, and the preferred backend for bytecode analysis.

API models are self-contained and independent of the underlying source, so ROSEAU automatically dumps the source ASTs or class files once the API model is constructed. This significantly lowers the memory footprint of ROSEAU, as this intermediate information can quickly fill up memory for large libraries [36]. API models are lightweight and can be maintained in memory for further analyses while maintaining a minimal memory footprint (e.g., using them as a pivot when walking through a list of commits).

When parsing the source library, some references in API models might be dangling when ROSEAU cannot resolve the corresponding type declaration. This occurs when the API refers to types not provided in the classpath, e.g., when the API uses other third-party libraries. To mitigate this issue, ROSEAU natively resolves all symbols in Java's standard library and accepts a user-defined classpath as input when necessary to complete the resolution process.

APIs can be compared regardless of what they are extracted from: an API inferred from a JAR release on Maven Central can be transparently checked for BCs against an API inferred from the source code of the latest commit fetched from GitHub. APIs inferred from source code and from the equivalent compiled bytecode are identical, except for potentially missing information in bytecode depending on the compiler's configuration, which does not affect BC detection (e.g., parameter names, precise position of the symbol in the source file, etc.). Information about generic types remains accessible in bytecode despite erasure (using signatures [37, §4.7.9]) and is incorporated in the API models by the ASM backend. API models can also be serialized and desnakelized as JSON when needed, enabling their reuse in subsequent analyses. This is particularly handy for tagging a specific API

version as a reference (e.g., the latest release) and reusing it at virtually no cost.

Finally, to accurately infer the API of a software library, ROSEAU implements the API delimitation mechanisms presented in Section II. Specifically, users can set up regex-based exclusions for packages (e.g., `*.internal.*`), choose to ignore symbols marked with user-defined annotations (such as Google's `@Beta` or Apache's `@Internal`), and determine whether `@Deprecated` symbols should be included in backward compatibility checks.

Our implementation of API models offers several benefits over other tools. For instance, JAPICMP constructs an explicit *change model* from two JAR files, serving as a thin layer of abstraction still closely tied to the underlying bytecode parser Javassist. However, this model captures the differences between two APIs, not the API itself. As such, it is specific to the comparison of two versions and does not allow reusing the API extracted from a given version for further analyses (e.g., against multiple other versions). In contrast, our API model is fully self-contained and technology-independent, allowing an easier evolution with new versions of Java, the transparent definition of new backends, and the ability to maintain API models in memory or on disk for further analyses.

## B. Performance

ROSEAU is designed with a strong focus on performance, implementing several optimizations to enable large-scale studies of API evolution.

*Diet parsing*: In all backends, ROSEAU prioritizes performance by parsing input source code or bytecode in "diet" mode. This entails ignoring method bodies, expressions, and statements (such as field initializers) in source files, as these elements do not influence the API. Similarly, code, debug, and frame attributes in bytecode are neither analyzed nor visited. The JDT and ASM backends are configured to enable this "diet" mode.

*Type references and immutability*: API models use type references to link names to type declarations (classes, interfaces, type parameters, primitive and array types, etc.). In ROSEAU, these type references enjoy several properties. Type references are strongly typed. For instance, references to superclasses are type-guaranteed to refer to *class* symbols, not any other kind of type declaration. They are lazily resolved only when needed for efficiency. Additionally, they are unique within the typing scope of an API, meaning that there is only one single reference toward a given type declaration shared by all referencers. A given name is thus only resolved once. Finally, API models are implemented as fully immutable data structures, allowing full parallelization of the evaluation of the BC detection rules depicted in Table I.

*Incremental updates*: When comparing two versions of the same API, it is likely that they share most of their symbols. This is typically the case when analyzing two subsequent releases or all commits within a given branch. ROSEAU is specifically designed to handle these scenarios by supporting incremental updates of API models. This allows API models

<footnote n="4">⁴https://wiki.eclipse.org/JDT_Core/</footnote>

<footnote n="5">⁵https://asm.ow2.io/</footnote>
to be partially updated without parsing and inferring the API from scratch whenever the underlying source changes (e.g., a source tree or a JAR file). API models can be updated incrementally to reflect only the changes between two library versions, drastically improving performance in scenarios where multiple versions share a large portion of their symbols.

Every symbol in the API model is linked to a physical location that identifies its source (a `.java` or `.class` file). When the source is updated (e.g., when checking out a new commit), `ROSEAU` computes the minimal set of files that have been added, deleted, or changed between the two versions. This information can be extracted in various ways, for instance, by using hash-based comparisons, modification-time checks, or commit information from a version control system.

Once the changed files are identified, the update proceeds as follows:

* *Added files:* new files are parsed, and their corresponding symbols are integrated into the API model;

* *Deleted files:* symbols originating from deleted files are removed from the API model. Assuming that the source is consistent (the source code compiles or the JAR file is valid), any reference to a deleted symbol must also have been updated in dependent files. Therefore, if a symbol is deleted, other files referencing it will have been modified and thus re-parsed to reflect the change;

* *Changed files:* modified files are parsed again, and the corresponding symbols in the API model are updated accordingly. Since the API model uses name-based type references, any change to a symbol is automatically visible to all other symbols referencing it.

Note that since API models are immutable in `ROSEAU`, unchanged symbols are first copied from the original API into the new version. By leveraging incremental updates, our approach minimizes unnecessary re-parsing, ensuring that only symbols affected by changes in the source are recomputed. As demonstrated in the next section, this drastically improves performance, especially when conducting large-scale longitudinal studies of library histories.

# V. EVALUATION

In our evaluation, we aim to assess the accuracy and runtime performance of `ROSEAU` in detecting BCs, including in the context of large-scale longitudinal studies. Throughout the experiments, we compare `ROSEAU` (version 0.1.0) against two baselines. Originally, Jezek and Dietrich inventoried nine BC detection tools for Java libraries [4]. However, only two have been updated in the past five years and are still actively used in practice (cf., Section II): JAPICMP (version 0.23.1) and REVAPI (version 0.15.0). To the best of our knowledge, no other tool has emerged since Jezek and Dietrich's inventory.

We evaluate each tool through three experiments. The first evaluates their ability to accurately identify BCs (Section V-A). The second measures their runtime performance on popular Java libraries (Section V-B). The third uses Guava as a case study to evaluate their suitability for analyzing large commit histories (Section V-C).

All experiments in this section were conducted on an Intel® Xeon® W-2125@4.00 GHz with 32 GB of RAM on Linux 6.1.0-generic using Java HotSpot 21.0.5 (2024-10-15 LTS). The source code, scripts, and data discussed in this section are available in our replication package [38].

## A. Accuracy

*Protocol:* To evaluate the accuracy of BC detection tools, we refine and extend the original benchmark of Jezek and Dietrich, which has already been used for accuracy evaluation in prior work (e.g., [9]). This benchmark follows a systematic methodology to list API evolution cases. It considers possible changes along three dimensions: what is changing (e.g., a modifier), where the change is applied (e.g., a method), and how it is changed (e.g., it is removed), resulting in 224 possible combinations corresponding to changes (breaking and non-breaking) that may happen in real-world APIs [4]. Concretely, the benchmark comprises a first version of a synthetic API and a second version with the changes applied. A third component, the client, contains a set of Java files. Each file contains a main() method using a symbol that compiles successfully against the first API. To build a ground truth, the benchmark automatically (i) compiles all files in the client against the second API version to detect source-breaking changes and (ii) links the existing client binary against the binary of the second API version to detect binary-breaking changes. Finally, it runs BC detection tools on both versions of the APIs and compares their reports with the ground truth using a simple grep-based search on the client's file name to determine whether they accurately identify BCs. The benchmark calculates the accuracy of each tool, defined as the ratio of correctly diagnosed changes over the total number of changes.

This benchmark is a solid foundation for evaluating BC detection tools. However, it suffers from several drawbacks. First, changes are sometimes intertwined, with the same file in the first version declaring several symbols all affected by different changes in the second version. A single client sometimes uses several symbols: if a linking error happens on the first, the linker aborts and subsequent errors are not evaluated for binary compatibility. Following Ochoa et al.'s approach, we refactor the benchmark so that only a single symbol is changed between the two versions of the same file, and each client only uses a single symbol of the API [9].

Second, the use of shell scripts and grep to determine whether a tool's report matches the ground truth is brittle. We find several instances where cases are mixed because they share a common prefix (e.g., *genericsIfazeTypeAdd* and *genericsIfazeTypeAddSecond*), which produces unsound results. We address this problem by implementing a Java pipeline that programmatically runs each case independently, solving the unsoundness issues in the benchmark.

Third, the accuracy metric computed in the original benchmark confounds false positives and false negatives, and does not evaluate the tools' ability to distinguish source-breaking and binary-breaking changes, hindering the interpretability of the results. In our refined benchmark, we evaluate the tools'
precision and recall independently and report separate scores for their ability to distinguish source-breaking changes and binary-breaking changes.

Finally, we realize that the client code used as an oracle in the original benchmark is sometimes too weak. When a client fails to compile or link with the second version, the diagnosis is clear. When a client successfully compiles and links, however, then either the change is indeed backward-compatible, or it is breaking but the client was unable to trigger the incompatibility. For example, the *genericsIfazeTypeBoundsAdd* case involves adding a bound to an interface's type variable in the API (Lib <A> to Lib<A extends Number>). However, the corresponding client uses it with an *Integer* bound (Lib<Integer>), which does not trigger any compilation error when the client migrates to the second version, as *Integer* complies with the added bound. This change is thus misclassified as a binary-compatible change. A client using the interface with a *String* bound, for instance, triggers the incompatibility. Thus, we manually review all clients in the benchmark to transform them into stronger oracles. Notably, we addressed numerous cases related to type bounds and introduced new clients that extend types and override methods—in addition to simply instantiating and invoking them—to simulate inversion-of-control uses. Our final benchmark contains 267 cases. 86 are backward-compatible, 81 are source-breaking, 21 are binary-breaking, and 79 are both source- and binary-breaking.

<table><caption>TABLE II: Precision, recall, and $F_1$-score of JAPICMP, REVAPI, and ROSEAU.</caption><thead><tr><th rowspan="2"></th><th colspan="3">All</th><th colspan="3">Source</th><th colspan="3">Binary</th></tr><tr><th>Prec.</th><th>Rec.</th><th>F<sub>1</sub></th><th>Prec.</th><th>Rec.</th><th>F<sub>1</sub></th><th>Prec.</th><th>Rec.</th><th>F<sub>1</sub></th></tr></thead><tbody><tr><td>JAPICMP</td><td>0.90</td><td>0.82</td><td>0.86</td><td>0.78</td><td>0.80</td><td>0.79</td><td>0.90</td><td><strong>0.98</strong></td><td><strong>0.94</strong></td></tr><tr><td>REVAPI</td><td>0.85</td><td>0.97</td><td>0.91</td><td>0.77</td><td>0.96</td><td>0.85</td><td>0.89</td><td>0.95</td><td>0.92</td></tr><tr><td>ROSEAU</td><td><strong>0.98</strong></td><td><strong>0.99</strong></td><td><strong>0.99</strong></td><td><strong>0.88</strong></td><td><strong>1.00</strong></td><td><strong>0.94</strong></td><td><strong>0.91</strong></td><td><strong>0.98</strong></td><td><strong>0.94</strong></td></tr></tbody></table>

**Results:** We use our benchmark to compute the accuracy of ROSEAU, JAPICMP, and REVAPI. As the results obtained on the benchmark using the two backends of ROSEAU are identical, we do not distinguish them in this experiment.

Table II shows the precision, recall, and $F_1$-score obtained by the three tools. Overall, all three obtain decent scores on the benchmark. In all cases, ROSEAU achieves slightly higher scores than the other tools, only tied in recall and $F_1$-score by JAPICMP for binary-BCs. While all tools are nearly as accurate on binary-BCs, the benefits of ROSEAU are more apparent for source-BCs. This might be explained by the original focus of JAPICMP and REVAPI on binary-BCs. JAPICMP, for instance, was originally focused on detecting the binary incompatibilities specified in the JLS [20, chap. 13] but has gradually evolved to support source incompatibilities. Indeed, JAPICMP achieves higher scores than in the original benchmark of Jezek and Dietrich: at the time, it was not able to detect any BC related to generics; this support was since implemented in the version used in our experiments. As ROSEAU was originally designed to analyze both source code and bytecode, we always considered both kinds of compatibility. Overall, we find that ROSEAU

outperforms other tools in accuracy, with an overall $F_1$-score of 0.94 for source- and binary-breaking changes, and 0.99 for all changes—where the ability to differentiate source- and binary-breaking changes is not evaluated.

**Accuracy:** ROSEAU achieves higher overall accuracy ($F_1$ = 0.99) than REVAPI ($F_1$ = 0.91) and JAPICMP ($F_1$ = 0.86), primarily due to improved detection of source-breaking changes.

## B. Performance

In this experiment, we evaluate the runtime performance of the tools in a scenario where two versions of a library are compared for BCs. This represents the typical use of BC detection tools as part of build managers or continuous integration. As there is no past version of the library to consider, this experiment does not evaluate ROSEAU's support for incremental updates, which we address in the last experiment.

The choice of backend has a significant impact on ROSEAU's runtime performance. The time spent building an API model and comparing two API models is negligible compared to the time spent parsing and visiting the source files or class files. In this experiment, we thus consider two variants of ROSEAU: ROSEAUASM, which uses the ASM backend, and ROSEAUJDT, which uses the JDT backend.

**Protocol:** To measure the performance of each tool in realistic conditions, we first curate a dataset of popular libraries. First, we collect the top-100 artifacts with the most uses on Maven Central from https://mvnrepository.com. We discard from this set all artifacts that are not libraries or do not contain any source code (e.g., Spring Boot starter projects) or that are implemented in other JVM languages (e.g., the standard libraries of Kotlin, Clojure, and Scala). The resulting set contains 60 popular Java libraries, including h2, Spring, jackson-databind, assertj, commons-collections, etc. Finally, we include Java-21's standard library: 69 standard modules (e.g., java.lang, java.io, etc.) totaling 4M lines of code. We retrieve each library's source code and binary JAR from Maven Central, selecting only the most recent stable version.

We evaluate each tool's performance by comparing each library's last stable version with itself. Although no BC should be detected in this scenario, it represents the worst-case scenario where the tools must traverse the entire API to thoroughly check for differences. This prevents possible optimizations such as automatically marking all methods in a class as deleted when the class itself is deleted, without visiting them. To ensure a fair comparison of the tools, we created a functionally equivalent Java wrapper for each. This wrapper runs the analysis without generating a report, thereby avoiding any I/O beyond loading and analyzing the libraries.

We evaluate the runtime performance of ROSEAUASM, JAPICMP, and REVAPI (using the library's JAR as input and ignoring the potential compilation time necessary to build it), as well as of ROSEAUJDT (using the library's sources as input).
This experiment does not exercise ROSEAUJDT's incremental updates, which we evaluate in the next experiment.

For accurate and stable measurements, we use a JMH harness<fnref n="6" /> configured with 5 fresh JVM forks and 10 measurements in single-shot time mode to obtain 50 measurements per library and tool. We set the warm-up iterations to 10 as we empirically verified that all tools reach stable performance at that point.

**Results:** Figure 2 shows the relation between the tools' runtime performance and the size of the analyzed libraries (lines of code as reported by the `cloc` utility). In all cases, there is a clear linear relation between library size and the tools' runtime performance, although `REVAPI`'s data points are more scattered. When analyzing JARs, `JAPICMP` and `ROSEAUASM` perform similarly, inferring BCs for all libraries up to 250k lines of code in less than a second. `REVAPI`, however, is visibly slower than the other tools.

ROSEAUJDT is the only tool able to analyze source code directly. Surprisingly, it can analyze all 60 libraries in less than 2 seconds, mainly thanks to the optimized “diet” parsing implemented in `ROSEAU`. In scenarios where only source code is available, `ROSEAUJDT` largely surpasses the other tools as they would require compiling the source code first. We evaluate and confirm this intuition in the next experiment.

Regarding the standard library of Java 21 and its 4M lines of code (not included in Figure 2 for the sake of readability), the mean analysis time is 5,157 ms for `JAPICMP`, 148,535 ms for `REVAPI`, 4,834 ms for `ROSEAUASM`, and 18,990 ms for `ROSEAUJDT`. Here again, `JAPICMP` and `ROSEAUASM` perform similarly when analyzing the JAR version of the JDK, with `REVAPI` lagging behind. `ROSEAUJDT` is able to parse the 4M lines of code, build the API model, and conduct the analysis in 19 seconds, showcasing its potential for very large libraries.

This demonstrates that `ROSEAU` can analyze libraries with excellent performance regardless of their size, both as JARs or in their source code form. `ROSEAU` can thus serve as a drop-in replacement for `JAPICMP` and `REVAPI` in continuous integration and build managers with better accuracy, and opens new analysis capabilities that we evaluate in the next experiment.

**Performance:** When analyzing JARs, `ROSEAU` performs similarly to `JAPICMP` in terms of execution times, analyzing libraries up to 250k lines of code in less than a second. When analyzing source code, `ROSEAU` largely outperforms `JAPICMP` and `REVAPI` (which require compilation), analyzing all libraries in less than 2 seconds. `ROSEAU` parses and analyzes the source code of Java 21's entire standard library (4M lines) in under 19 seconds.

## C. Case Study

In this case study, we aim to evaluate the tools' suitability to conduct large-scale longitudinal studies using Google's Guava library as the subject. Guava is a very popular library with a large number of contributors and, at the time of writing, a rich

<figure><img src="image_3.png" /><figcaption>Fig. 2: Execution time of BC detection tools across 60 popular Java libraries.</figcaption></figure>

history of 6,839 commits spanning 14 years. In this case study, our goal is to analyze sequentially all commits in Guava's main branch throughout its evolution. We limit our analysis to Guava for brevity, and refer the reader to our replication package for a similar analysis of `commons-lang` and `h2database`.

**Protocol:** To illustrate the suitability of each tool for this task, we first conduct a preliminary assessment restricted to the latest 100 commits on Guava's main branch. Our objective is to illustrate the amount of time required by `ROSEAU`, `JAPICMP`, and `REVAPI` to analyze these 100 commits. Retrieving them is straightforward, but they only exist in the form of source code. As `JAPICMP` and `REVAPI` require binary JARs as input, we first measure the time necessary to compile and package these 100 commits as JAR files without running any tests. Then, we measure the time taken by `ROSEAUJDT`, with and without incremental updates, to build the API model of the commit. In both cases, when comparing commits $c_{i-1}$ and $c_i$, the API model of $c_i$ is kept in memory to be reused as a pivot when comparing $c_i$ and $c_{i+1}$. To estimate the time each tool would have taken to analyze Guava's entire commit history, we calculate the average time required to analyze each of the 100 commits and multiply it by the total number of commits.

In a second step, we use the most efficient approach, `ROSEAU` with incremental updates, to analyze all commits in Guava's history. We present the extracted information, focusing on the evolution of its API and the introduction of BCs over time. We detail some key events in its history, showcasing the suitability of `ROSEAU` for conducting empirical studies of library histories.

<table><caption>TABLE III: Mean time and estimated total time to build a JAR or extract an API model from a sample of 100 Guava commits.</caption><thead><tr><td>Method</td><td>Mean commit time (ms)</td><td>Total time (minutes)</td></tr></thead><tbody><tr><td>Packaging JAR</td><td>68,750</td><td>7,836</td></tr><tr><td>Non-incremental ROSEAU</td><td>1,065</td><td>121</td></tr><tr><td>Incremental ROSEAU</td><td>81</td><td>9</td></tr></tbody></table>

**Results:** Table III depicts the results of our preliminary experiment. On average, packaging Guava's source code

<footnote n="6">⁶https://openjdk.org/projects/code-tools/jmh/</footnote>
<figure><img src="image_4.png" /><figcaption>Fig. 3: Evolution of Guava's API and introduction of BCs over time. Major versions are depicted as vertical dashed lines.</figcaption></figure>

to obtain a JAR takes 69 seconds per commit. Analyzing the history of Guava would thus take nearly five and a half days using JAPICMP or REVAPI. This highlights their limitations for conducting large-scale longitudinal studies. Without incremental updates, ROSEAU takes about a second to build an API model from each commit and roughly two hours to analyze all commits. Finally, ROSEAU performs best when incremental updates are activated, as it minimizes the amount of source code to parse at each step. In this configuration, ROSEAU takes 81 ms per commit, or 9 minutes for all commits.

Finally, we end the preliminary experiment and use ROSEAU with incremental updates to analyze all 6,839 commits in Guava's history and compute the BCs between each pair of subsequent commits. On average, ROSEAU analyzes a new commit in 117 ms—this varies with the number of added, changed, and removed files in each commit due to incremental updates. This is slower than in the preliminary experiment, as this includes the time taken to identify BCs and compute metrics. ROSEAU analyzes all 6,839 commits in Guava in 799 seconds (approximately 13 minutes): 208 seconds walking from one commit to the next using git checkout (26%); 284 seconds parsing source code and inferring the API model (36%); 247 seconds diffing two APIs to identify BCs (31%); and 60 seconds computing metrics and storing the results. One key benefit of ROSEAU is that intermediate API models can be reused in the next step: for each commit, it compares the new version with the previous one stored in memory. This demonstrates that, unlike existing tools, ROSEAU enables large-scale longitudinal studies of API evolution and BCs.

Figure 3 shows the information ROSEAU extracts at each analysis step and highlights key moments in Guava's API evolution. Commits are represented on the X-axis. For each commit, the plot shows the number of API symbols (including types, executables, and fields), the number of @Beta and

@Deprecated symbols, and the amount of BCs the commit introduces. This analysis is possible thanks to ROSEAU's reified API models that can not only detect BCs, but also track API evolution, in contrast with JAPICMP and REVAPI. Without claiming to conduct an in-depth qualitative analysis, we discuss some interesting moments in Guava's history to showcase ROSEAU's value for empirical research on API evolution.

A first observation is that Guava's API has steadily grown over time, doubling the number of exported API symbols between versions 10.0 and 33.0. From the start, a significant portion of Guava's API was exposed as @Beta, following the maintainers' guidelines. In version 32.0, however, Guava's maintainers decided to freeze most of their beta APIs: "*the '@Beta' annotation is removed from almost every class and member. This makes them officially API-frozen.'*,"<fnref n="7" /> as clearly shown in Figure 3. Regarding BCs, most were introduced in earlier versions of the library, with very few BCs introduced after 2018, highlighting Guava's maturity. On Feb 2, 2016, shortly after the release of version 19.0, commit 2067e7 introduced 270 BCs. This commit rolled back a set of overloads introduced in com.google.common.base.Preconditions that were introduced to avoid autoboxing and numeric conversions. The rollback commit thus removed 270 methods from this class. Version 20.0 and 31.0 also included a notable number of BCs, the release notes indeed discussing removed APIs.

**Large-scale longitudinal studies:** ROSEAU analyzes all 6,839 commits of the Guava library, spanning 14 years, in 13 minutes. Its reified API models enable commit-level tracking of API evolution and breaking changes, making it ideal for large-scale empirical studies of API evolution.

<footnote n="7">⁷https://github.com/google/guava/releases/tag/v32.0.0</footnote>
## D. Threats to Validity

**Construct Validity:** In Section V-A, we refine the clients to strengthen their ability to detect BCs in the two API versions. We manually review all changes and corresponding clients in the benchmark, but we cannot guarantee that no change was mislabeled. While the original benchmark was constructed systematically, exhibiting most of the possible changes to Java APIs, we cannot guarantee that it is comprehensive, especially when considering multiple interacting changes occurring concurrently. Therefore, it is possible that some changes that could happen in real-world libraries are not covered in the benchmark.

**Internal Validity:** Accurate runtime measurements are challenging and can be affected by many external factors [39]. We used a dedicated benchmark framework, JMH, and a dedicated machine with only the required dependencies that was not performing any other task during the experiments. In Section V-C, we assume that the time taken to compile and package the last 100 commits of Guava can be used to extrapolate compilation times throughout its history. While this is an over-approximation (earlier versions were smaller and likely to compile faster), this does not jeopardize our conclusions.

**External Validity:** In Section V-B, we curate a set of 60 popular libraries for performance measurements. While this set contains libraries built by diverse contributors and focusing on diverse domains, they are not a random sample of Java libraries. The results may vary in another set of libraries. Similarly, our case study uses Guava as the sole subject and may not be representative of the results obtained for other libraries.

## VI. CONCLUSION

In this paper, we have presented ROSEAU, a new static analysis tool for Java libraries that infers accurate API models from either source code or bytecode to support API evolution analysis. API models of two library versions can be compared to identify syntactic breaking changes between them. Using an established benchmark of breaking changes, we have shown that ROSEAU is more accurate than the state-of-the-art tools JAPICMP and REVAPI. We have evaluated the performance of ROSEAU and shown that it matches their performance when analyzing bytecode, and outperforms them by two orders of magnitude when analyzing source code, showcasing the suitability of ROSEAU for large-scale longitudinal studies of API evolution, taking minutes instead of days.

As future work, we will incorporate ROSEAU as part of IDEs to connect it to their internal code representation and provide live feedback on API design and evolution to library designers. We will leverage the capabilities of ROSEAU to conduct large-scale empirical studies of popular libraries to better understand the dynamics of client-library co-evolution and inform code reviews by providing maintainers with detailed API reports.

## DATA AVAILABILITY

ROSEAU is hosted on GitHub (https://github.com/alien-tools/roseau). This paper's experiments use version 0.1.0 (https:

//github.com/alien-tools/roseau/releases/tag/v0.1.0). The replication package on Zenodo [38] contains the extended version of Jezek and Dietrich's benchmark used in the accuracy evaluation, the 60 popular Java libraries used in the performance evaluation, the full experimentation pipeline, the raw results presented in Section V, and the associated plots and analyses.

## ACKNOWLEDGMENTS

This work was partially funded by the French National Research Agency through grant ANR ALIEN (ANR-21-CE25-0007). The authors thank Yasmine Hamdaoui for her initial work on ROSEAU and the anonymous referees for their precious comments.

## REFERENCES

[1] A. Zerouali, T. Mens, J. M. González-Barahona, A. Decan, E. Constantinou, and G. Robles, "A formal framework for measuring technical lag in component repositories - and its application to rpm," *J. Softw. Evol. Process.*, vol. 31, no. 8, 2019.

[2] R. G. Kula, D. M. Germán, A. Ouni, T. Ishio, and K. Inoue, "Do developers update their library dependencies? - an empirical study on the impact of security advisories on library migration," *Empir. Softw. Eng.*, vol. 23, no. 1, pp. 384-417, 2018.

[3] S. Mirhosseini and C. Parnin, "Can automated pull requests encourage software developers to upgrade out-of-date dependencies?" in *Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering, ASE*, 2017, pp. 84-94.

[4] K. Jezek and J. Dietrich, "API evolution and compatibility: A data corpus and tool evaluation," *J. Object Technol.*, vol. 16, no. 4, pp. 2:1-23, 2017.

[5] JApiCmp report for Apache. [Online]. Available: https://commons.apache.org/proper/commons-io/japicmp.html

[6] JApiCmp report for Spring. [Online]. Available: https://docs.spring.io/spring-framework/docs/6.0.6_to_6.0.7/spring-beans.html

[7] JApiCmp config for Gson. [Online]. Available: https://github.com/google/gson/blob/3488c30/pom.xml#L476-L505

[8] JApiCmp config for Neo4j. [Online]. Available: https://github.com/neo4j/cypher-dsl/blob/e76e9de/pom.xml#L577-L666

[9] L. Ochoa, T. Degueule, J. Falleri, and J. J. Vinju, "Breaking bad? Semantic versioning and impact of breaking changes in Maven Central," *Empir. Softw. Eng.*, vol. 27, no. 3, p. 61, 2022.

[10] D. Jayasuriya, S. Ou, S. Hegde, V. Terragni, J. Dietrich, and K. Blincoe, "An extended study of syntactic breaking changes in the wild," *Empir. Softw. Eng.*, vol. 30, no. 2, p. 42, 2025.

[11] F. Reyes, B. Baudry, and M. Monperrus, "Breaking-good: Explaining breaking dependency updates with build analysis," in *IEEE International Conference on Source Code Analysis and Manipulation, SCAM*, 2024, pp. 36-46.
[12] A. Dann, B. Hermann, and E. Bodden, "UCPY: safely updating outdated dependencies," in *45th IEEE/ACM International Conference on Software Engineering, ICSE*, 2023, pp. 233-244.

[13] D. Jaime, P. Poizat, J. E. Haddad, and T. Degueule, "Balancing the quality and cost of updating dependencies," in *Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, ASE*, 2024, pp. 1834-1845.

[14] L. Zhang, C. Liu, Z. Xu, S. Chen, L. Fan, B. Chen, and Y. Liu, "Has my release disobeyed semantic versioning? Static detection based on semantic differencing," in *37th IEEE/ACM International Conference on Automated Software Engineering, ASE*, 2022, pp. 51:1-51:12.

[15] L. Ochoa, T. Degueule, and J. Falleri, "BreakBot: Analyzing the impact of breaking changes to assist library evolution," in *44th IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results ICSE (NIER)*, 2022, pp. 26-30.

[16] F. Hassan, S. Mostafa, E. S. L. Lam, and X. Wang, "Automatic building of Java projects in software repositories: A study on feasibility and challenges," in *2017 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM*, 2017, pp. 38-47.

[17] M. Sulír, M. Bacíková, M. Madeja, S. Chodarev, and J. Juhár, "Large-scale dataset of local Java software build results," *Data*, vol. 5, no. 3, p. 86, 2020.

[18] Program Structure Interface (PSI), JetBrain's Intellij. [Online]. Available: https://plugins jetbrains com/docs/Intellij/psi.html

[19] D. Jayasuriya, V. Terragni, J. Dietrich, and K. Blincoe, "Understanding the impact of APIs behavioral breaking changes on client applications," *Proc. ACM Softw. Eng.*, vol. 1, no. FSE, pp. 1238-1261, 2024.

[20] *The Java Language Specification, Java SE 21 Edition*, 2023. [Online]. Available: https://docs.oracle.comjavase/specs/jls/se21/jls21.pdf

[21] Kinds of compatibility from OpenJDK. [Online]. Available: https://wiki.openjdk.org/display/csr/Kinds+of+Compatibility

[22] C. Bogart, C. Kästner, J. D. Herbsleb, and F. Thung, "When and how to make breaking changes: Policies and practices in 18 open source software ecosystems," *ACM Trans. Softw. Eng. Methodol.*, vol. 30, no. 4, pp. 42:1-42:56, 2021.

[23] How Apache Commons handle binary compatibility. [Online]. Available: https://garygregory.wordpress.com/2020/06/14/how-we-handle-binary-compatibility-at-apache-commons/

[24] Guava important warnings. [Online]. Available: https://github.com/google/guava/blob/0fad76f/README.md

[25] X. Du and J. Ma, "AexPy: Detecting API breaking changes in python packages," in *IEEE 33rd International Symposium on Software Reliability Engineering, ISSRE*, 2022, pp. 470-481.

[26] JDiff report for Guava. [Online]. Available: https://guava.dev/releases/33.4.0-jre/api/diffs/

[27] M. Lamothe, Y. Guéhéneuc, and W. Shang, "A systematic review of API evolution literature," *ACM Comput. Surv.*, vol. 54, no. 8, pp. 171:1-171:36, 2022.

[28] L. Xavier, A. Brito, A. C. Hora, and M. T. Valente, "Historical and impact analysis of API breaking changes: A large-scale study," in *IEEE 24th International Conference on Software Analysis, Evolution and Reengineering, SANER*, 2017, pp. 138-147.

[29] D. Jayasuriya, V. Terragni, J. Dietrich, S. Ou, and K. Blincoe, "Understanding breaking changes in the wild," in *Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA*, 2023, pp. 1433-1444.

[30] D. Venturini, F. R. Cògo, I. Polato, M. A. Gerosa, and I. S. Wiese, "I depended on you and you broke me: An empirical study of manifesting breaking changes in client packages," *ACM Trans. Softw. Eng. Methodol.*, vol. 32, no. 4, pp. 94:1-94:26, 2023.

[31] W. Li, F. Wu, C. Fu, and F. Zhou, "A large-scale empirical study on semantic versioning in Golang ecosystem," in *38th IEEE/ACM International Conference on Automated Software Engineering, ASE*, 2023, pp. 1604-1614.

[32] L. Zhang, C. Liu, Z. Xu, S. Chen, L. Fan, L. Zhao, J. Wu, and Y. Liu, "Compatible remediation on vulnerabilities from third-party libraries for Java projects," in *45th IEEE/ACM International Conference on Software Engineering, ICSE*, 2023, pp. 2540-2552.

[33] A. Brito, M. T. Valente, L. Xavier, and A. C. Hora, "You broke my code: understanding the motivations for breaking changes in APIs," *Empir. Softw. Eng.*, vol. 25, no. 2, pp. 1458-1492, 2020.

[34] A. Brito, L. Xavier, A. C. Hora, and M. T. Valente, "APIDiff: Detecting API breaking changes," in *25th International Conference on Software Analysis, Evolution and Reengineering, SANER*, 2018, pp. 507-511.

[35] G. Monce, T. Couturou, Y. Hamdaoui, T. Degueule, and J. Falleri, "Lightweight syntactic API usage analysis with UCov," in *Proceedings of the 32nd IEEE/ACM International Conference on Program Comprehension, ICPC*, 2024, pp. 426-437.

[36] Q. L. Dilavrec, D. E. Khelladi, A. Blouin, and J. Jézéquel, "HyperAST: Enabling efficient analysis of software histories at scale," in *37th IEEE/ACM International Conference on Automated Software Engineering, ASE*, 2022, pp. 54:1-54:12.

[37] *The Java Virtual Machine Specification, Java SE 21 Edition*, 2023. [Online]. Available: https://docs.oracle.comjavase/specs/jvms/se23/jvms23.pdf

[38] C. Latappy, T. Degueule, J.-R. Falleri, R. Robbes, and L. Ochoa, "Replication kit - Roseau," 2025. [Online]. Available: https://doi.org/10.5281/zenodo.15536417

[39] D. Costa, C. Bezemer, P. Leitner, and A. Andrzejak, "What's wrong with my benchmark results? Studying bad practices in JMH benchmarks," *IEEE Trans. Software Eng.*, vol. 47, no. 7, pp. 1452-1467, 2021.